[HN Gopher] A Cookbook of Self-Supervised Learning
___________________________________________________________________
A Cookbook of Self-Supervised Learning
Author : ZunarJ5
Score : 79 points
Date : 2023-04-25 16:13 UTC (6 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| cs702 wrote:
| Nice. It looks like it will be useful... although best practices
| are likely going to continue to evolve. The biggest question in
| my mind is whether we can come up, at some point in the future,
| with a kind of universal self-supervised learning objective that
| works well in practice for any task.
| nmaley wrote:
| The NFL theorem means nothing if all the learning tasks have a
| common underlying structure. In the real world, they do. The
| laws of physics and chemistry create emergent causal
| relationships. Any SSL learning algorithm that learns to
| exploit causal relationships will consistently perform well
| over a variety of real world tasks.
| medo-bear wrote:
| no such thing as free lunch -
| https://en.m.wikipedia.org/wiki/No_free_lunch_theorem
| tomrod wrote:
| We really need to revise this to say "there is no global free
| lunch."
|
| You can and often do get local free lunches.
| medo-bear wrote:
| the question was about "objective that works well in
| practice for any task". thats pretty global in my books
| karpierz wrote:
| If you take "any task" to mean literally any conceivable
| task, then sure.
|
| If you take "any task" to mean "any practical task" or
| "any task a human would conceivably want to have done",
| then no free lunch doesn't apply.
| tbalsam wrote:
| Another way of looking at karpierz's comment is through
| the incompressibility of pure noise at scale.
|
| As soon as some infinitely generated sort of noise is
| from some subset of possible noise, there is indeed
| (AFAIK) some kind of an ideal estimator that
| appropriately compresses that noise source with no bias
| and less entropy that the full space of possible noise.
|
| I hope this shines an additional alternative light on the
| topic.
| tbalsam wrote:
| The most common statement I make is about the misapplication
| of the NFL.
|
| This is not an appropriate use of the NFL.
|
| The original commenter is asking about a general solution
| that will work well for all problems presented at it. The NFL
| details fine-grained tradeoffs in _ideal solutions_ for
| specific traits in certain areas. Additionally, we're not
| operating in an unbiased space here, so trivially there is a
| best estimator without bias. So by that very fact alone the
| NFL is invalidated in terms of the method it is being applied
| in here.
|
| This is something I feel frustrated seeing a lot of younger
| people entering the field do (not saying you are young or
| new, it's just the trend). If this was the case in this kind
| of a way we never would have gotten beyond MLPs.
|
| Yes, indeed there is in fact a set of general solutions that
| works roughly well over everything and is biased towards the
| situations where people will need it the most.
|
| No, it will likely not be the technically best performing
| solution.
|
| What the OP is looking for I believe is convenience,
| stability, and reliably good performance.
|
| Hence, the NFL is not applicable or relevant to this matter
| in the way it is being used.
___________________________________________________________________
(page generated 2023-04-25 23:00 UTC)