[HN Gopher] Early Days of AI
___________________________________________________________________
Early Days of AI
Author : todsacerdoti
Score : 72 points
Date : 2023-08-21 18:01 UTC (4 hours ago)
(HTM) web link (blog.eladgil.com)
(TXT) w3m dump (blog.eladgil.com)
| mdp2021 wrote:
| > _it is worth thinking of this as an entirely new era_
|
| In this case, we could considered the <<Early Days of AI>> as not
| having happened yet. It is absurd to forget a past that worked to
| celebrate a present that largely does not, to sensible
| understanding of the goal. Tools must be reliable.
|
| > _and discontinuity from the past_
|
| Let us hope this is a bump on the road, a phase, better if
| organic and eventually productive of something good.
| eladgil wrote:
| Definitely not my intention to forgot or denigrate the past.
| Obviously all this exists due to deep learning and prior
| architectures. What I have been running into is many people and
| companies are interpreting this as "just more of the same" for
| prior ML waves, when really this is an entirely new capability
| set.
|
| To the (bad) analogy on cars versus planes - both have wheels
| and can drive on the ground, but planes open up an entirely new
| dimension / capability set that can transform transportation,
| logistics, defense and other areas that cars were important,
| but different enough in.
| simonw wrote:
| I'm finding the comparison to the previous wave of ML
| absolutely fascinating.
|
| I tinkered with ML for a few years, and it took a LOT of work
| to get anything useful out of it at all.
|
| Now with LLMs I can literally type out a problem in English
| and there's a reasonably good chance I'll get a useful
| result!
|
| It really does feel like an entirely new set of capabilities
| to me.
| andy99 wrote:
| That's the big difference in this round. Before you had to
| have the ML expertise and the expertise to understand the
| implication of say a MNIST classifier example. Now anyone
| can "get" it because you're prompting and getting inference
| back in English. Underneath the fundamentals aren't all
| that different though, it has the same novelty factor and
| the same limitations apply.
| simonw wrote:
| I think the fundamentals are radically different, just
| due to the ease of applying this stuff.
|
| I used to be able to train and deploy a ML model to help
| solve a problem... if I put aside a full week to get that
| done.
|
| Now I tinker with LLMs five minutes at a time, or maybe
| for a full hour if I have something harder - and get
| useful results. I use them on a daily basis.
| joewferrara wrote:
| The author has more experience in the field than me, so gotta
| defer to him for the most part, and while I generally agree with
| the post, I disagree strongly with one point. The author frames
| this new era of AI about using transformer models (and diffusion
| models) but transformer models have been around for a while and
| have been useful from before GPT3, the model the author claims as
| the starting point for this new AI. BERT is a transformer model
| that came out in 2018 and is a very useful transformer style
| model, which showed the promise of transformers before GPT3.
|
| Edit: Going back through the post, the author's slide has
| Transformers labeled as 2017, so he is aware of the history and
| he's just emphasizing that GPT3 was the first transformer model
| that he thinks had something interesting and related to the
| current AI explosion. I think BERT style models would be worth a
| mention in the post as the first transformer models found to be
| widely useful.
| whimsicalism wrote:
| From the article,
|
| > The biggest inklings that something interesting was afoot
| came kicked with GPT-3 launching in June 2020
|
| Yes, I agree with you that if you were in this field, this is
| quite late to the realization. Most of my grad seminar in NLP
| in 2018 imagined ChatGPT-style tech would be possible as
| "language modeling is essentially world modeling."
| eladgil wrote:
| 100% agree the theory on AI is old and actually dates back to
| the early days of "cybernetics". But the real difference is
| at what point do we considered it sufficiently reduced to
| practice? I chose GPT-3 but undoubtedly people can point to
| earlier examples as glimpses of what was coming for sure.
| dvt wrote:
| > There is enormous potential for this new wave of tech to impact
| humanity. For example, Google's MedPaLM2 model outperforms human
| physicians to such a strong degree that having medical experts
| RLHF the model makes it worse (!).
|
| Wow, what a ridiculously disingenuous cherry-picked claim. If you
| actually _read_ the paper you 'll find this gem: "However, for
| one of the axes, including inaccurate or irrelevant information,
| Med-PaLM 2 answers were not as favorable as physician answers."
| Typical AI hype blog post donning the HN front page. At this
| point, I'm ready to put on my tinfoil hat and say that a16z, etc.
| is heavily pushing all these narratives because the next round of
| investments for the great majority of AI startups will almost
| certainly be the bagholder round.
| gumby wrote:
| I think in five years, LLMs will be like expert systems: largely
| considered not really "AI" but sitting around in the back end of
| all sorts of random systems.
| eladgil wrote:
| The bar for what is "AI" keeps moving. For example plane
| autopilots would be "AI" in the 1980s, the ability for a
| machine to win at chess, go, and other games etc.
| [deleted]
| TMWNN wrote:
|
| disagree with the legions of experts who last year denounced
| Blake Lemoine and his claims. I know enough to know, though,
| of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a
| longstanding tradition/bad habit of advances being dismissed
| by those in the field itself as "not real AI". Anyone, expert
| or not, in 1950, 1960, or even 1970 who was told that before
| the turn of the century a computer would defeat the world
| chess champion would conclude that said feat must have come
| as part of a breakthrough in AGI. Same if told that by 2015
| many people would have in their homes, and carry around in
| their pockets, devices that can respond to spoken queries on
| a variety of topics.
|
| To put another way, I was hesitant to be as self-assuredly
| certain about how to define consciousness, intelligence, and
| sentience--and what it takes for them to emerge--as the
| experts who denounced Lemoine. The recent GPT breakthroughs
| have made me more so.
|
| I found this recent Sabine Hossenfelder video interesting.
| <https://www.youtube.com/watch?v=cP5zGh2fui0>
| calderknight wrote:
| I don't think those things have ever been widely considered
| AI. It may have widely thought that those things would
| require AI, though.
| beambot wrote:
| Chess is featured in Peter Norvig's "Artificial
| Intelligence: A Modern Approach" dating back to the 1st
| edition (1995) and at least up until the 3rd edition
| (2009). Algorithms such as alpha-beta pruning were
| definitely considered AI at the time.
| calderknight wrote:
| Yeah AI has multiple denotations, those things have never
| been considered "real" AI tho
| [deleted]
| robotresearcher wrote:
| The MIT AI Group, including Marvin Minsky, were the
| mainstream of AI more than 50 years ago, and begat the MIT
| AI Lab. They and everyone else at the time called their
| work AI.
| TMWNN wrote:
|
| disagree with the legions of experts who last year denounced
| Blake Lemoine and his claims. I know enough to know, though,
| of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a
| longstanding tradition/bad habit of advances being dismissed
| by those in the field itself as "not real AI". Anyone, expert
| or not, in 1950, 1960, or even 1970 who was told that before
| the turn of the century a computer would defeat the world
| chess champion would conclude that said feat must have come
| as part of a breakthrough in AGI. Same if told that by 2015
| many people would have in their homes, and carry around in
| their pockets, devices that can respond to spoken queries on
| a variety of topics.
|
| To put another way, I was hesitant to be as self-assuredly
| certain about how to define consciousness, intelligence, and
| sentience--and what it takes for them to emerge--as the
| experts who denounced Lemoine. The recent GPT breakthroughs
| have made me more so.
|
| I found this recent Sabine Hossenfelder video interesting.
| <https://www.youtube.com/watch?v=cP5zGh2fui0>
| TMWNN wrote:
|
| disagree with the legions of experts who last year denounced
| Blake Lemoine and his claims. I know enough to know, though,
| of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a
| longstanding tradition/bad habit of advances being dismissed
| by those in the field itself as "not real AI". Anyone, expert
| or not, in 1950, 1960, or even 1970 who was told that before
| the turn of the century a computer would defeat the world
| chess champion would conclude that said feat must have come
| as part of a breakthrough in AGI. Same if told that by 2015
| many people would have in their homes, and carry around in
| their pockets, devices that can respond to spoken queries on
| a variety of topics.
|
| To put another way, I was hesitant to be as self-assuredly
| certain about how to define consciousness, intelligence, and
| sentience--and what it takes for them to emerge--as the
| experts who denounced Lemoine. The recent GPT breakthroughs
| have made me more so.
|
| I found this recent Sabine Hossenfelder video interesting.
| <https://www.youtube.com/watch?v=cP5zGh2fui0>
| TMWNN wrote:
| As a non-expert in the field I was hesitant at the time to
| disagree with the legions of experts who last year denounced
| Blake Lemoine and his claims. I know enough to know, though,
| of the AI effect <https://en.wikipedia.org/wiki/AI_effect>, a
| longstanding tradition/bad habit of advances being dismissed
| by those in the field itself as "not real AI". Anyone, expert
| or not, in 1950, 1960, or even 1970 who was told that before
| the turn of the century a computer would defeat the world
| chess champion would conclude that said feat must have come
| as part of a breakthrough in AGI. Same if told that by 2015
| many people would have in their homes, and carry around in
| their pockets, devices that can respond to spoken queries on
| a variety of topics.
|
| To put another way, I was hesitant to be as self-assuredly
| certain about how to define consciousness, intelligence, and
| sentience--and what it takes for them to emerge--as the
| experts who denounced Lemoine. The recent GPT breakthroughs
| have made me more so.
|
| I found this recent Sabine Hossenfelder video interesting.
| <https://www.youtube.com/watch?v=cP5zGh2fui0>
| ShamelessC wrote:
| You should check out the WaPo article that originally
| published his concerns. He frequently makes many errors
| audibly with a reporter who is trying rather hard to see
| his point of view. I'm not trying to be rude, but he came
| off like kind of a sucker that would fall for a lot of
| scammer tactics. There were usually some form of
| strangeness such as him deciding when the content limit of
| the conversation began and ended. Further, he asks only
| leading questions, which would be fine if transformers
| didn't specifically train to output the maximum likelihood
| text tokens from the distribution of their training set,
| which was internet text created by humans.
|
| He was frequently cited as an engineer but I don't think he
| actually had a strong background in engineering but rather
| in philosophy.
| Animats wrote:
| > LLMs will be like expert systems
|
| No. LLMs actually work. Expert systems were a flop.
|
| I went through Stanford CS in the mid-1980s, just when it was
| becoming clear that expert systems were a flop. The faculty was
| in denial about that. It was sad to see.
|
| We're at the beginning of LLMs, and systems which use LLMs as
| components. This is the fun time for the technology. Ten years
| out, it will be boring, like Java.
|
| The next big thing is figuring out the best ways to couple LLMs
| to various sources of data, so they can find and use more
| information specifically relevant to the problem.
|
| And someone has to fix the hallucination problem. We badly need
| systems that know what they don't know.
| dragonwriter wrote:
| > LLMs actually work. Expert systems were a flop.
|
| Expert systems actually work and, while we don't normally
| even call them "expert systems" any more, are important in
| basically every domain - "business rules engines" are
| generalized expert system platforms, and are widely used in
| business process automation.)
|
| Despite how well they work, and early optimism resulting from
| that about how much further they'd be able to go, they ran
| into limits; it is not implausible that the same will turn
| out to be true for LLMs (or even transformer-based models
| more generally.)
|
| > We're at the beginning of LLMs, and systems which use LLMs
| as components. This is the fun time for the technology. Ten
| years out, it will be boring, like Java.
|
| ...and expert systems. (And, quite possibly, by then they
| will have revealed their fundamental, intractable
| llimitations, like expert systems.)
| whimsicalism wrote:
| > I went through Stanford CS in the mid-1980s, just when it
| was becoming clear that expert systems were a flop. The
| faculty was in denial about that. It was sad to see.
|
| It's interesting how this appears to be a recurring cycle -
| when I attended school, it appeared that the faculty were in
| denial about the death of probabilistic graphical models and
| advanced bayesian techniques in favor of simple linear
| algebra with unsupervised learning. Even when taught about
| deep ML, there was heavy emphasis on stuff like VAE which had
| fun bayesian interpretations.
| vasilipupkin wrote:
| ML in general has been in production in many places/companies for
| a while now. Specifically GPt-4 is useful as a coding assistant
| and a reference tool. However, it's hard to know whether what
| it's telling you is accurate or fake and if you want to be
| thorough you need to double check it. So, it's already useful but
| in a limited way and it remains to be seen how much better it's
| likely to get in the near future.
|
| But let's not confuse that in general with AI or Machine Learning
| which is already used heavily in lots of places.
|
| the specific type of architecture that gpt-4 uses for next word
| prediction is not the only possible architecture and is not
| what's used for many real world tasks. There a lot of different
| problems being addressed by ML and next word prediction is just
| one of them, although quite important.
| eladgil wrote:
| Agreed on ML being in production in a lot of places and has
| been quite valuable, particularly for large incumbents. I write
| about this a bit here: https://blog.eladgil.com/p/ai-startup-
| vs-incumbent-value
|
| I think the difference this time is the types of capabilities
| provided by transformers vs prior waves of AI are sufficiently
| different to allow many more types of startups to emerge, as
| well as big changes in some types of enterprise software by
| incumbents - in ways that were not enabled by pre-existing ML
| approaches.
| galaxytachyon wrote:
| While I largely agree with this blog, I would like to point out
| one thing that often come up in HN: an inherent weakness or flaw
| in the current AI architecture that prevent it from widespread
| adoption. Right now it is either scaling hardware or
| hallucination. But there can be something deeper that we yet to
| see in public, for example, the inability to adapt to some
| specific, but critical, reasoning logic.
|
| I don't see the post address this possibility even though it is
| very likely. Microsoft promised AI powered Office a while ago and
| we are still waiting. GPT4 is supposed to be able to look at
| images and solve problems but we still haven't seen that yet.
| Something is preventing these big companies from implementing
| these features and this is supposed to be a solved problem. How
| can we be sure that there is no serious roadblocks in the future
| that plunge the field into another AI winter?
| alooPotato wrote:
| It's been like 4 months
| galaxytachyon wrote:
| In term of AI research and development, that was a long time
| ago. Microsoft is also pivoting hard into AI. They treat it
| as a cornerstone technology and I am sure they are not
| skimping on the cost of implementing it into their main
| products. Yet the only thing they have is Bing. Mind you they
| have early access to this tech. Bing uses GPT4 before the
| paper for GPT4 even came out so it is not like they only had
| it for 4 months.
| phillipcarter wrote:
| What's really special about this new era of ML is its
| accessibility.
|
| Every business had problems with probabilistic solutions. ML is
| the way to do that. But the barrier to entry has been so high for
| so long. And so you had to be in big tech or a highly specialized
| shop to play.
|
| Now all you need is an API key and one line of code to call the
| most powerful models in the world.
| joe_the_user wrote:
| I think this view, that present day AI is something absolutely
| new, is actually the dominant view.
|
| I don't believe the "absolutely new" view is very enlightening
| very often. Notably, it seems like "the dawn of a new era of
| tech" offers little insight to the process of change (but much
| hype). Even something like the explosion of the Internet is
| usefully compared to earlier technologies and what gave it's
| uniqueness wasn't incomparability but an explosion of scale.
| abatilo wrote:
| The highlight of the fact that large org adoption is slow feels
| really valuable here. Especially when we just recently had all of
| the articles that claimed that ChatGPT was "over" when it appears
| that this is mostly because of things like summer vacations with
| students. The adoption and understanding of these products and
| the risk involved with hallucinations is something that will take
| time to understand.
|
| I recently joined one of the LLM provider companies and watching
| these phases over the next few years will be really interesting.
| Especially combined with what's going on with regulation and the
| like.
|
| Random aside, hi Elad! I think you're reading some of these
| comments. I just left Color after ~2.5 years. I hope to get to
| formally introduce myself to you one day.
___________________________________________________________________
(page generated 2023-08-21 23:00 UTC)