[HN Gopher] Contrastive Representation Learning ___________________________________________________________________ Contrastive Representation Learning Author : gk1 Score : 78 points Date : 2022-08-19 14:09 UTC (8 hours ago) (HTM) web link (lilianweng.github.io) (TXT) w3m dump (lilianweng.github.io) | mountainriver wrote: | another incredible article by Lilian Weng, never ceases to | impress and enlighten | hadrianpaulo wrote: | Are there techniques for contrastive learning that's also | applicable to tabular data? | cs702 wrote: | Nice job! This is a fantastic resource for anyone interested in | using contrastive methods for inducing AI/ML models to learn to | embed data in a space such that samples considered similar stay | close to each other (e.g., as measured by cosine or Euclidean | distance) while dissimilar ones stay far apart. Self-supervised | contrastive methods, in particular, can be remarkably useful when | none of the samples in your data are labeled and you want your | model to discover structure. | fxtentacle wrote: | I'm surprised that this doesn't mention cross-entropy, the | contrastive loss function used by Facebook's wav2vec2 XLS-R | pretraining paper and by OpenAI's CLIP. | canjobear wrote: | Contrastive losses arise from using methods like NCE (mentioned | in the post) to approximate cross entropy loss when the | partition function is intractable. | roknovosel wrote: | Great read, thanks for sharing. Would love to see the natural | language + code mixed in there :) | | I've been interested in contrastive learning for a while, mainly | as a means to train semantic code search models. OpenAI released | a great paper on this topic called Text and Code Embeddings by | Contrastive Pre-Training[1] that outlines the approach. I've used | it as a base to build https://codesearch.ai [2] with pretty good | results. | | [1] https://arxiv.org/pdf/2201.10005.pdf [2] | https://sourcegraph.com/notebooks/Tm90ZWJvb2s6MTU1OQ== ___________________________________________________________________ (page generated 2022-08-19 23:01 UTC)