[HN Gopher] ChatGPT's API is so good and cheap, it makes most te...
___________________________________________________________________
ChatGPT's API is so good and cheap, it makes most text generating
AI obsolete
Author : minimaxir
Score : 286 points
Date : 2023-03-11 18:20 UTC (4 hours ago)
(HTM) web link (minimaxir.com)
(TXT) w3m dump (minimaxir.com)
| d23 wrote:
| Off topic, but how'd you get that amazing header banner Max? I
| tried a web-based ControlNet + Stable Diffusion combo[1], but the
| quality is nothing near what you have there.
|
| [1] - https://stablediffusionweb.com/ControlNet
| minimaxir wrote:
| https://news.ycombinator.com/item?id=35112641
| zibzob wrote:
| Does anyone know if there's a way to use this technology to help
| understand a large codebase? I want a way to ask questions about
| how a big unfamiliar codebase works. It seems like ChatGPT isn't
| trained on open source code, so it can't answer questions there.
| When I asked it how something works in the Krita source, it just
| hallucinated some gibberish. But if there's a way to train this
| AI on a specific codebase, maybe it could be really useful. Or is
| that not possible with this type of AI?
| freezed88 wrote:
| This is what we've designed LlamaIndex for!
| https://github.com/jerryjliu/gpt_index. Designed to help you
| "index" over a large doc corpus in different ways for use with
| LLM prompts.
| roflyear wrote:
| No, not large understanding. But if you are unfamiliar with
| specific language features, or there is confusing code it can
| help you figure things out. But no it is not good for any large
| corpus of text, and you can't give it new stuff and teach it
| anything.
| scarface74 wrote:
| It "knows" the AWS API, CloudFormation and from what others
| have told me the CDK pretty well. I've asked it to write plenty
| of 20-30 line utility scripts and with the proper prompts, it
| gets me 95% of the way there.
|
| I assume it would "understand" more popular open source
| frameworks.
| vorticalbox wrote:
| In this case you can feed it bits of code you're interested in
| and ask it to explain, the API has a limit of 4096 tokens
| (which is a good chunk of text).
|
| I actually built a slack bot for work and daily ask it to
| refactor code or "write jsdocs for this function"
| zibzob wrote:
| Yeah, and this is pretty useful for small bits of code, but
| what I want is a way to ask questions about large projects.
| It would be nice to ask something like "which classes are
| responsible for doing X", or "describe on a high level how Y
| works in this code". But I'm not sure if that is actually
| possible with the current technology.
| roseway4 wrote:
| It's possible to do this either by fine-tuning an existing
| model or using an existing chat model prompts enriched by a
| vector search for relevant code. See my comment elsewhere.
| roseway4 wrote:
| OpenAI Codex understands code. Though it's primary use case is
| code completion, it might be to do Q&A well given a prompt with
| context.
|
| https://platform.openai.com/docs/guides/code
|
| I'd you're interested in trying the very cheap models behind
| ChatGPT, you may want to have a look at langchain and
| langchain-chat for an example of how to build a chatbot that
| uses vectorized source code to build context-aware prompts.
| zibzob wrote:
| Thanks for the links, I'll take a look at this and see if
| it's something I could reasonably achieve.
| Jiocus wrote:
| Have you checked out Copilot Labs, the experimental version of
| Copilot? It's bundled with ability to explain and document
| source code, among other things.
|
| https://githubnext.com/projects/copilot-labs/
| zibzob wrote:
| That looks promising! But I think it only works on small
| snippets of code and doesn't have an overview of the whole
| codebase...still, maybe it's coming down the line as they
| improve it.
| rocauc wrote:
| There's good work happening in this area, e.g. Sourcegraph is
| working on "Cody" to understand and search your code base
| https://twitter.com/beyang/status/1614895568949764096
| bcrosby95 wrote:
| ChatGPT does not understand your code, does not have the same
| mental model as you do of your code, and from my experiments
| does not have the ability to connect related but spatially
| disconnected concepts across even small codebases which will
| cause it to introduce bugs.
|
| Asking it about these things sounds like it would result in
| questionable, at best, responses.
| zibzob wrote:
| I see, that's what I was worried about. It would be really
| helpful if it could answer high-level questions about a big
| confusing codebase, but maybe it's not just a matter of
| showing it the code and having it work.
| skissane wrote:
| ChatGPT has a published context window of 4096 tokens.
| Although, I saw someone on Twitter saying the real figure,
| based on experiments, was closer to 8192 tokens. [0] Still,
| that's an obvious roadblock to "understanding" large code
| bases - large code bases are too big to fit in its "short-
| term memory", and at runtime its "long-term memory" is
| effectively read-only. Some possible approaches:
|
| (A) wait for future models that are planned to have much
| longer contexts
|
| (B) fine tune a model on this specific code base, so the
| code base is part of the training data not the prompt
|
| (C) Break the problem up into multiple invocations of the
| model. Feed each source file in separately and ask it to
| give a brief plain text summary of each. Then concatenate
| those summaries and ask it questions about it. Still
| probably not going to perform that well, but likely better
| than just giving it a large code base directly
|
| Another issue is that, even the best of us make mistakes
| sometimes, but then we try the answer and see it doesn't
| work (compilation error, we remembered the name of the
| class wrong because there is no class by that name in the
| source code, etc). OOTB, ChatGPT has no access to
| compilers/etc so it can't validate its answers. If one gave
| it access to an external system for doing that, it would
| likely perform better.
|
| [0] https://mobile.twitter.com/goodside/status/159887467420
| 46187...
| panarky wrote:
| Saying a machine cannot understand the way humans understand
| is like saying airplanes cannot fly the way birds fly.
| Dudeman112 wrote:
| Which is correct and a big reason for why early flight
| machines had no chance at all of working
|
| Of course, that doesn't tell you whether the machine
| understanding will be useful or not
| sandkoan wrote:
| I'm actually building this very thing--shoot me an email at
| govind <dot> gnanakumar <at> outlook <dot> com if you'd like to
| be a beta tester.
| xyz_ielh wrote:
| [dead]
| recuter wrote:
| [flagged]
| superkuh wrote:
| This might be true for the type of business and institutional
| uses that can operate under the extremely puritanical filters
| that are bolted onto gpt3.5-turbo. But for most human person uses
| the earlier text completion models like gtp3 davinci are
| incomparibly better and more responsive. But also 10x as pricey.
| Still, it's worth it compared to the lackluster and recalcitrant
| non-output of gpt3.5-turbo.
|
| I think over the next couple months most human people will switch
| away from gpt3.5-turbo in openai's cloud to self-hosted LLM
| weights quantized to run on consumer GPU (and even CPU), even if
| they're not quite as smart.
| abraxas wrote:
| I have a hard time imagining anything that comes even close to
| ChatGPT being able to run on consumer hardware in the next
| couple of years
| yieldcrv wrote:
| I could perceive something like in 1 or less.
|
| M3 Macbook with eGPU functionality restored in conjunction
| with more efficient programming would mean having enough
| memory available to all the processors. This would definitely
| count as consumer hardware.
|
| Custom built GPU-like devices with tons of RAM could become
| vogue. Kind of like the Nvidia A100 but even more purpose
| built for running LLMs or whatever models come next.
| superkuh wrote:
| That's what I thought 2 weeks ago. I figured it'd be ~5 years
| before I could do anything at home. But already people have
| the leaked facebook llama weights running on CPU w/under 32
| GB of system ram doing a token a second or so.
| riku_iki wrote:
| that llama is likely much smaller than chatgtp
| mattnewton wrote:
| I can certainly imagine it after seeing
| https://github.com/ggerganov/llama.cpp
|
| Still a couple years out but moving way faster than I would
| have expected.
| abraxas wrote:
| ChatGPT has 175B weights if I'm not mistaken. Llama 7B
| would not be in any way comparable.
| v64 wrote:
| One finding in the LLaMA paper [1] is that our current
| large models are undertrained. LLaMA with 13B params
| outperforms GPT-3 175B (not ChatGPT), but an "instruct"
| version of LLaMA was finetuned over the 65B model and did
| quite well.
|
| [1] https://arxiv.org/pdf/2302.13971.pdf
| superkuh wrote:
| For people who think the number of parameters determines
| LLM coherence, well, that's a good rule of thumb. But
| there's an optimal training set data size to parameters
| count and gpt3 was trained on too little data. The LLM
| coming out now trained on more data with fewer parameters
| and achieve something close.
|
| Sure, the 7 billion parameter can't do long outputs. But
| the 13 billion one is not too bad. They're not a full
| replacement by any means but for many use cases a local
| service that is stupider is far preferable to a paid
| cloud service.
| KeplerBoy wrote:
| It's crazy, but it seems to be happening already. Granted,
| that's probably still a far-cry from Chat-GPT, but it seems
| inevitable a few years down the line.
|
| https://news.ycombinator.com/item?id=35100086
| zamnos wrote:
| Moore's law isn't quite beaten yet, so the (hypothetical
| future) RTX 5090 and 6090 is gonna be _insane_. Combined with
| software optimization and refinement of the techniques, along
| with training != inference, means I think we 'll see
| something better, runnable locally, in a couple of years. The
| leaps and bounds Stable Diffusion has gone is insane.
| Facebook's LLaMA is also seeing a similar growth from just
| having the model available.
| echelon wrote:
| llm-nasty will find a way.
|
| Stable Diffusion broke free of the shackles and was pushed
| further than DALL-E could have ever hoped for.
|
| Just wait. People's desires for LLMs to say spicy things and
| not be controlled by a single party will make this happen yet
| again. And they'll be more efficient and powerful. Half the
| research happening is from "waifu" groups anyway, and they'll
| stop at nothing.
| isatty wrote:
| Exactly, there is so much money to be made by generating porn
| that it'll be done by this year.
| yieldcrv wrote:
| seriously. entertain the humans.
| [deleted]
| echelon wrote:
| VCR, cable, internet, usenet, web, streaming, VR...
|
| There are so many technologies that were propelled forward
| because of it, not in spite of it.
|
| Twitter, Reddit, Tumblr...
|
| Tumblr learned a hard lesson when they tried to walk away.
| zirgs wrote:
| Feeding the AI lots of porn is one way to fix broken fingers.
| Spivak wrote:
| Or just keep using davinci because is it's also really cheap
| all things considered. I was excited about getting 1/10th the
| cost but also came to the same conclusion as you as turbo can't
| actually _do_ anything. I could care less about getting it to
| write porn or nazi propaganda but good lord it can't even write
| code, do line edits or follow instructions more complicated
| than simple call /response.
| superkuh wrote:
| My use case is IRC bots. If you just have the bot responding
| to only a single line and not knowing any of the chat
| history, yeah, it can be fairly cheap. But once you try to
| start giving it short term memory by feeding in the prior
| ~handful of lines you blow through that $18 of free credit in
| a couple weeks. Something that costs $25/mo is not cheap for
| a human person.
|
| I am not happy with your implication that gpt3.5-turbo only
| doesn't respond to "nazi" stuff and that my users are such
| people. But I guess getting Godwin'd online isn't new. It
| literally won't even respond to innocuous questions.
| Spivak wrote:
| What kinda volume are you pushing though because I also do
| that and even have it ingest whole pdfs/word docs as
| conversation context and I get charged like $3/mo on
| average.
|
| Edit: I'm literally agreeing with you and describing
| innocuous questions that it doesn't respond to. I'm saying
| that if all it refused to do was write hate and erotica it
| would be fine and I would use it but the filter catches
| things like code.
| superkuh wrote:
| With gpt3 davinci we were doing about ~30 requests for
| text completion per hour (at peak activity) each having
| ~11 lines of chat history (including up to 190 token
| responses from davinci) which added up to about ~1000 to
| 5000 tokens each. So 30*3000 at $0.0200/1000 tokens
| equals a few dollars per day.
| tracyhenry wrote:
| what is an example that you can do with Davinci, but not
| chatgpt? In my limited experience with prompting you can ask
| chatgpt to do a lot of things
| superkuh wrote:
| gpt3.5-turbo fails the turing test due to it's constant
| butt covering. Davinci can pass for a human. I am speaking
| only of the API responses. The "chatgpt" web interface is
| something different.
| LeoPanthera wrote:
| It doesn't matter that the older model will happily generate
| text to make your grandmother blush. The usage policy
| specifically says you can't do that. They even provide an API
| endpoint for checking whether the input, and output, is allowed
| or not.
|
| There's nothing stopping you from ignoring it, except for the
| certainty that OpenAI will simply block you.
| faizshah wrote:
| Its actually good for most human person uses too like writing
| or learning. I've never encountered it refusing to do a task in
| my actual work.
|
| I would guess the risk to their brand vs the number of actual
| applications of the unfiltered ai makes it an obvious trade
| off.
|
| I mean who turns off google safe search when writing an essay
| or lyrics?
| zamnos wrote:
| Wait, there are people that aren't children that actually
| have SafeSearch turned on as more than an accident? Not
| trying to be insulting, I just genuinely have it turned off
| in my settings and haven't noticed any of my search results
| being particularly NSFW and assumed everyone else did too.
| [deleted]
| bilbo0s wrote:
| If the default is off, most will have it off.
|
| If the default is on, most will have it on.
|
| All of which to say, no one cares, and google very likely
| knows that. Google will only care if enough of their users
| care. And they will probably operate in a fashion that
| keeps the maximum number of their users in the "don't care"
| camp. It's just business.
| kayodelycaon wrote:
| If the default is on, most people would have it on.
| tracyhenry wrote:
| A couple months might be too soon imho. But I hope that in 2-3
| years there will be a model with similar performance but much
| smaller size, small enough to run incredibly fast inference +
| training on my laptop. OpenAI might need to rethink their moat
| in case that happens.
|
| Think about all the smart ML researchers in academia. They
| can't afford training large models on large datasets, and their
| decades of work is made obsolete by OpenAI's bruteforce
| approach. They've got all the motivation in the world to work
| on smaller models.
| soulofmischief wrote:
| I actually don't think that we will make significant
| advancements in reducing model size before we make
| significant advances in increasing available power and
| compute.
|
| One reason is that the pressure is still on for models to be
| bigger and more power hungry, as many believe compute will
| continue to be the deciding factor in model performance for
| some time. It's not a coincidence that OpenAI's CEO, Sam
| Altman, also runs a fusion energy r&d company.
| flangola7 wrote:
| But processing hardware has been seeing diminishing returns
| for years. My CPU from 2013 is still doing what I need; a
| 1993 processor in 2003 would have been useless.
|
| Where do you see hardware improvements coming from?
| alpark3 wrote:
| Everyone loves to hate on OpenAI and talk about how they're
| really ClosedAI and an evil corporation vying for power, but the
| opposite way is also interesting to think about. I think it's
| fair to say that majority of scientists at OpenAI wouldn't be
| working there if they knew they were working for an evil
| corporation. These are some of the brightest people on the
| planet, yet I've only heard good things about OpenAI leadership,
| especially Sam Altman, and their commitment to actually guiding
| AI for the better.
|
| I'm not saying that OpenAI is benevolent, but let's assume so for
| the sake of argument. They definitely would need real-world
| experience running commercial AI products, for the organizational
| expertise as well as even more control over production of safe
| and aligned AI technologies. A hypothetical strategy, then, would
| be to a) get as much investment/cash as needed to continue
| research productively (Microsoft investment?) b) with this cash,
| do research but turn that research into real-world product as
| fast as possible c) and price these products at a loss so that
| not only are they the #1 product to use, other potentially
| malevolent parties can't achieve liftoff to dig their own niche
| into the market
|
| I guess my point is that a company who truly believes that AI is
| potentially a species-ending technology and requires incredible
| levels of guidance may aim for the same market control and
| dominance as a party that's just aiming for evil profit. Of
| course, the road to hell is paved with good intentions and I'm on
| the side of open source(yay Open Assistant), but it's
| nevertheless interesting to think about.
| Sol- wrote:
| > and their commitment to actually guiding AI for the better
|
| I think the Silicon Valley elite's definition of "for the
| better" means "for the better for people like us". The
| popularity of the longtermism and transhumanism cult among them
| also suggests that they'd probably be fine with AI wiping out
| much of humanity1, as long as it doesn't happen to them - after
| all, they are the elite and the future of humanity, with the
| billions of (AI-assisted) humans of that will exist!
|
| And they'll think it's morally right too, because there's so
| many utility units to be gained from their (and their
| descendants') blessed existence.
|
| (1 setting aside whether that's a realistic risk or not, we'll
| see)
| recuter wrote:
| > These are some of the brightest people on the planet, yet
| I've only heard good things about OpenAI leadership, especially
| Sam Altman, and their commitment to actually guiding AI for the
| better.
|
| Hear hear. It ought to be remembered that there is nothing more
| difficult to take in hand, more perilous to conduct, or more
| uncertain in its success than to take the lead in the
| introduction of a new order of things.
| wpietri wrote:
| > These are some of the brightest people on the planet, yet
| I've only heard good things about OpenAI leadership
|
| This is a deeply ahistorical take. Lots of technically bright
| people have been party to all sorts of terrible things.
| Don't say that he's hypocritical Rather say that he's
| apolitical "Vunce ze rockets are up, who cares vere zey
| come down "Zats not mein department!" says Werner von
| Braun
| ben_w wrote:
| While "smart people do terrible things" is an absolutely fair
| point, it's also the kind of thing I hear AI researchers say,
| even with similar references.
|
| Sometimes they even say this example in the context of "why
| human-level AI might doom us all".
| masfuerte wrote:
| I think you're agreeing. The "yet" implied a contrast.
| edgyquant wrote:
| I think it's safe to say people wouldn't be working for any
| company if they thought it was evil, so your whole point is
| moot.
| soulofmischief wrote:
| The conclusion derived from this argument, that there are no
| evil companies, doesn't seem to match up with empirical data
| scotty79 wrote:
| > I think it's safe to say people wouldn't be working for any
| company if they thought it was evil
|
| Did you read what you wrote?
| ben_w wrote:
| Hah no.
|
| Lots of people work for organisations they actively think are
| evil because it's the best gig going; plenty of other people
| find ways to justify how their particular organisation isn't
| evil despite all it does so they can avoid the pain of
| cognitive dissonance and keep getting paid.
|
| My _current_ approval of OpenAI is conditional, not certain.
| (I don 't work there, and I at least _hope_ I will be "team-
| think-carefully" rather than "team OpenAI can't possibly be
| wrong because I like them").
| al2o3cr wrote:
| Similarly, if you feel the need to fart it COULD be a monkey
| trying to escape - sure, it's been eggy gases every single time
| before but THIS TIME COULD BE DIFFERENT!
|
| Don't hold it in, the monkeys need your help!
| croes wrote:
| Ever read The Physicists from Durrenmatt?
|
| Or let me quote Dr. Ian Malcolm:
|
| "Your scientists were so preoccupied with whether they could,
| they didn't stop to think if they should."
| [deleted]
| avereveard wrote:
| yeah remember when a lot companies based themselves on the bing
| search api and then the price increase 3x-10x depending on usage?
| thanks, but no thanks.
| [deleted]
| vintermann wrote:
| One kind of text generation AI it already makes obsolete, is
| specialized translation models. It's no surprise it outdoes
| Google Translate, that feels like it hasn't been updated in a
| while. But it also outdoes Deepl now, and Deepl is good.
|
| And it seems to handle translating from low-resource languages
| extremely well. Into them, it's a bit harder to judge.
|
| It handles translation between closely related languages such as
| Swedish and Norwegian extremely well. Google Translate goes via
| English and accumulates pointless errors.
| DuckFeathers wrote:
| The biggest problem with ChatGPT (and alternatives) is the risk
| of being coopted for generating the content someone gets in
| trouble for. Someone very important will get in BIG BIG trouble
| and try to blame OpenAI for it... and the series of lawsuits that
| will follow will kill them.
|
| While other such models will be impacted, hopefully, there will
| be significant variations in alternatives so that we don't lose
| this technology over giant corporations trying to get out of
| their trouble by suing their service providers.
|
| There will also be companies that will use modified versions of
| open source alternatives... to make them much more conservative
| and cautious, so that they don't get in trouble. There will be
| these variations that will be shared by certain industries.
|
| So, while the generative AI is here to stay, there will be a LOT
| of variations... and ChatGPT will have to change a lot if they
| want to stay alive and relevant over time.
| freedomben wrote:
| would you consider OpenAI (in its current iteration) to be
| conservative?
| sebzim4500 wrote:
| You may be right that some of the smaller AI players could be
| overwhelmed by lawsuits but OpenAI has a nearly $2 trillion
| company bankrolling them so they can hire every lawyer in the
| US if necessary.
| FuckShadowBans wrote:
| [dead]
| margorczynski wrote:
| What's the catch? How do they plan to make money out of it? Or
| maybe the plan is to use the massive amount of data gathered to
| make it better for e.g. Bing search? Cut out the competition
| before it has a chance to flourish?
|
| Companies, especially giant publicly traded ones like MS (the de
| facto owner of OpenAI) don't give out freebies.
| m3kw9 wrote:
| OpenAI wins by innovating faster than everyone because a lot of
| these models inner workings are known and can be trained to
| meet ChatGPTs metrics. so all they have to do is hire the best
| and move faster, as long as they have on par or better, people
| won't be switching
| dragonwriter wrote:
| > What's the catch?
|
| The catch is its a tactic to discourage investment in competing
| technologies, enabling OpenAI to build their lead to the point
| it is insurmountable.
|
| > How do they plan to make money out of it?
|
| Altman's publicly-stated plan for making money from OpenAI is
| (I'm completely serious) [0]:
|
| (1) Develop Artificial General Intelligence under the control
| of OpenAI.
|
| (2) Direct the AGI to find a way to make a return for
| investors.
|
| [0] https://techcrunch.com/2019/05/18/sam-altmans-leap-of-
| faith/
| guiriduro wrote:
| That idea might actually work. If a startup is a build-
| measure-learn loop, then coming up with ballpark viable
| ideas, devising experiments to test them and optimising for
| traction/profit should be a cinch for AGI. So just train it
| to build a business for itself.
| overcast wrote:
| The API is a paid service, like all the other APIs.
| alpark3 wrote:
| I believe his underlying assumption is that the API is so
| cheap that there's no way they're making money off of it. Yes
| it's paid, but doesn't matter if they're losing money on
| every API call.
| ianmcgowan wrote:
| If they're selling below cost, it doesn't matter. When you're
| selling a dollar for 90 cents, it's hard to make up for that
| in volume.
| scottLobster wrote:
| It's not like there are individual units of ChatGPT. With
| enough subscribers they could sell it for 1 cent per month
| and profit.
| supermatt wrote:
| Not sure what you mean by "individual units" but the
| suggestion is that it costs more than they charge. i.e
| it's not profitable, and the more they sell the more they
| lose.
| scottLobster wrote:
| My point was "making it up on volume" is largely
| irrelevant when it comes to mass market web-apps.
|
| Costs are relatively fixed outside of infrastructure, and
| potential customers are any number up to and including
| the internet-connected population of the world.
|
| The marginal cost of a new subscription is way less than
| they charge. The more they sell the less they lose, even
| if they're still losing overall to gain market-share.
| pixl97 wrote:
| This depends on the compute power quantum stepping....
|
| That is what is the upgrade cost to expand capacity as
| new customers are added. If for example adding 1 million
| new users requires $200,000k in hardware expenditure and
| $20k in yearly power expenditure, but your first year
| return on those customers is only going to be $50k,
| you're in a massive money losing endeavor.
|
| The point here is we really don't know the running and
| upkeep costs of these models at this point.
| chessgecko wrote:
| People are speculating that gpt3.5 turbo is actually much
| smaller and that they are very likely currently making a profit
| on it. It seems likely just given how quickly some of the 3.5
| turbo responses are from the api, and how much they push users
| to it. I haven't seen any really compelling theories of how
| they did it though, just the results...
| ethbr0 wrote:
| They wouldn't be the first business to have showroom halo
| products to attract customers, who instead but more
| profitable mass-market products. Auto industry 101.
| deeviant wrote:
| I don't know why everybody is asking themselves why they are
| offering it so cheaply, it seems rather obvious:
|
| 1. Get near every company to jump on the hype train and
| integrate openai api into their processes.
|
| 2. Get overwhelming market share.
|
| 3. Slowly reduce costs by increasing model and computation
| efficiency and raise prices.
|
| 4. Profit.
| sacred_numbers wrote:
| Alternatively:
|
| 1. Quickly reduce costs by increasing model and computation
| efficiency.
|
| 2. Massively reduce prices while still maintaining some gross
| margin.
|
| 3. Massively increase market size and take the vast majority
| of market share.
|
| 4. End up with a higher gross profit due to a much larger
| market size despite decreasing prices and gross margins.
|
| 5. Profit.
| theturtletalks wrote:
| Step 3 also includes raising prices once people have
| integrated the API. Google Maps was the "easy" and "cheap"
| way of integrating maps into apps until they got almost all
| the market share and raised prices through the roof.
| swatcoder wrote:
| The wildly successful public buzz draws internal and external
| money towards the project. Outsiders now see Microsoft as
| freshly repositioned against Google, and OpenAI as a rising
| rocket; internal budget is likewise drawn to related endeavors
| because everybody wants to claim a piece of whatever hits big.
|
| Meanwhile, yes, the preview provides both training data for the
| tooling, which has engineering value in AI, and usage data into
| how users think about this technology and what they intuitively
| want to do with it, which helps guide future product
| development.
|
| Both these reasons are also why they're (1) being so careful to
| avoid scandal, and (2) being very slow to clear up public
| misconceptions.
|
| An safe, excited public that's fully engaged with the tool
| (even if misusing and misunderstanding it) is worth a ton of
| money to them right now and so has plenty of justification to
| absorb investment. It won't last forever, but a new innovation
| door seems to have opened and we'll probably see this pattern a
| lot for a while.
| sebzim4500 wrote:
| It's also possible they have found a way to run the model
| extremely cheaply. To be fair, there has been many improvements
| to transformer inference since they initally set their prices
| (most notably flash attention), so if they were barely making a
| profit back then they could still be making a profit now.
|
| That's a big if, however, and no one really will give you
| figures on exactly what this costs at scale. Especially since
| we don't know for a fact how big GPT-3.5-turbo actually is.
| waboremo wrote:
| Yes your second guess is accurate. They will be changing
| pricing down the line when enough of the market is captured and
| competitors have been deterred. Most notably, Microsoft's
| largest competitor: Google.
| sourcecodeplz wrote:
| There is plenty of money from the 2012 - 2020 meteoric period
| that has not been spent yet. If I had plenty of money I would
| bet on Microsoft and OpenAI, as I am sure others are doing
| already. Thus they have enough to sustain this growth.
| bakugo wrote:
| They'll hike up the price by 10x once enough companies are
| relying on it to do business.
| politician wrote:
| We'll see AWS step in at that point with their own product
| offering.
| nico wrote:
| This is a market grab. They are moving fast to capture the
| market. Being cheap allows them to capture the market faster.
|
| The main customers won't be end users of ChatGPT directly, but
| instead companies with a lot of data and documents that are
| already integrating the apis with their systems.
|
| Once companies have integrated their services with OpenAIs
| apis, they are unlikely to switch in the future. Unless of
| course something revolutionary happens again.
| riku_iki wrote:
| > their services with OpenAIs apis, they are unlikely to
| switch in the future.
|
| why is that? If competitor release better or cheaper LLM, it
| is not that hard to switch API calls..
| nico wrote:
| Sure, that's the case if all your software does is make a
| couple of api calls and you have very few stake holders.
|
| But when you have built a big service around an external
| api, you have thousands or millions of users and thousands
| of employees - replacing an api is not just a big technical
| project, it's also a huge internal political issue for the
| organization to rally the necessary teams to make the
| changes.
|
| People hate change, they actively resist it. The current
| environment is forcing companies to adapt and adopt the new
| technologies. But once they've done it, they'll need an
| even bigger reason to switch apis.
| potatolicious wrote:
| > _" Being cheap allows them to capture the market faster."_
|
| I think it's worth remarking that this is IMO a smarter way
| of using price to capture market than what we've seen in the
| post decade (see: Uber, DoorDash) - in OpenAI's case there's
| every reasonable expectation that they can drop their
| operating costs well below the low prices they're offering,
| so if they are running in the red the expectation of
| temporariness is reasonable.
|
| What was unreasonable about the past tech cycle is that a lot
| of the expectations of cost reduction a) never panned out,
| and b) if subjected to even slight scrutiny would never have
| reasonably panned out.
|
| OpenAI has direct line-of-sight to getting these models
| _dramatically_ cheaper to run than now, and that 's a huge
| benefit.
|
| That said I remain a bit skeptical about the market overall
| here - I think the tech here is legitimately groundbreaking,
| but there are a few forces working against this as a
| profitable product:
|
| - Open source models and weights are catching up very
| rapidly. If the secret sauce is sheer scale, this will be
| replicated quickly (and IMO is happening). Do users need
| _ChatGPT_ or do they need _any decently-sized LLM_?
|
| - Productization seems like it will largely benefit incumbent
| large players (see: Microsoft, Google) who can afford to tank
| the operating costs _and_ additional R &D required on top to
| productize. Those players are also most able to train their
| own LLMs _and_ operate them directly, removing the need for a
| third party provider.
|
| It seems likely to me that this will break in three
| directions (and likely a mixture of them):
|
| - Big players train their own LLMs and operate them directly
| on their own hardware, and do not do business with OpenAI at
| any significant volume.
|
| - Small players lean towards undifferentiated LLMs that are
| open source and run on standard cloud configurations.
|
| - Small players lean towards proprietary, but non-OpenAI
| LLMs. There's no particular reason why GCP and AWS cannot
| offer a similar product and undercut OpenAI.
| boringg wrote:
| Until they raise prices. Classic venture playbook here - get
| everyone hooked on the product then raise rates.
|
| Also depends how you calculate cost. If its simply $ or if you
| are counting the externalities as 0.
| sourcecodeplz wrote:
| I don't think were going to reach winter before we can run our
| own ChatGPT locally with mundane hardware.
| vkou wrote:
| And yet, this is ChatGPT's attempt at generating a college essay:
|
| https://acoup.blog/2023/02/17/collections-on-chatgpt/
|
| Looking at the actual essay it produced, I don't need to know
| anything about Roman history to know that the essay sucks.
| Looking at the professor's markup of the essay, it becomes very
| clear that for someone who knows a lot about Roman history, the
| essay sucks - a _lot_.
|
| And it's not like it was prompted to write about an _esoteric_
| topic! According to the grader, the essay made 38 factual claims,
| of which 7 were correct, 7 were badly distorted, and 24 were
| outright bullshit. According to both myself, and the grader, way
| too much heavy lifting is done by vague, unsubstantiated, overly
| broad statements, that don 't really get expanded on further in
| the composition.
|
| But yes, if we're looking to generate vapid, low-quality, low-
| value content spam, ChatGPT is great, it will produce billions of
| dollars of value for advertisers, and probably net negative value
| for the people reading that drivel.
| photochemsyn wrote:
| What is you sequentially fed ChatGPT with samples of the course
| professor's own writing, and then asked it to write an essay on
| the subject of interest? As the article notes, optimization is
| possible:
|
| > "For example, high school and college students have been
| using ChatGPT to cheat on essay writing. Since current
| recognition of AI generated content by humans involve
| identifying ChatGPT's signature overly-academic voice, it
| wouldn't surprise me if some kids on TikTok figure out a system
| prompt that allow generation such that it doesn't obviously
| sound like ChatGPT and also avoid plagiarism detectors."
|
| A decent student might go to the trouble of checking all the
| factual claims produced in the essay in other sources, thus
| essentially using ChatGPT to write a rough draft then spending
| the time saved on checking facts and personalizing the style. I
| don't even know if that would count as serious cheating,
| although the overall structure of such essays would probably be
| similar. Running 'regenerate response' a few times might help
| with that issue, maybe even, 'restructure the essay in a novel
| manner' or similar.
| specproc wrote:
| I don't agree it's cheap. For generation at fairly small scale,
| sure, but generation is just the party trick. The real power for
| my use case lies in how much better it seems to do at traditional
| NLP tasks than an out-of-the-box model, with no further fiddling
| and faffing required.
|
| Say I've got a corpus of ~1m documents, each of 10+ paragraphs
| and I want to run quote extraction on them (it does this
| beautifully), vectorise them for similarity search, whatever.
| This gets pretty expensive pretty fast.
| andix wrote:
| What's the alternative? Hiring humans to do the job for you?
| Probably much more expensive.
| avibhu wrote:
| Tangential: you can finetune something like flan-ul2 to do
| quote extraction using examples generated from chatgpt. If you
| have a good enough GPU, it should help cut down costs
| significantly
| winddude wrote:
| Don't they have in the ToS you aren't allowed to use outputs
| for training downstream? Which is a little ridiculous,
| considering it's ToS.
|
| But yea, they cheap cost and lack of training is making me a
| take a long hard look at how I'm implementing more
| traditional NLP solutions.
| specproc wrote:
| Nice, that sounds like it's worth exploring. Much
| appreciated.
|
| Again though, it's the zero-effort part that's appealing. I'm
| on a very small team and getting that to close to the same
| standard will take time for a ham-fisted clod like myself.
| Worth giving a shot all the same though, thanks again.
| pfdietz wrote:
| It's interesting what you can do with ChatGPT with few shot
| learning. It generalizes at the drop of a hat, often
| correctly.
| leobg wrote:
| The zero shot ability is convenient. But for tasks that you
| need to get done millions of times, I'd much rather spend
| $10 on GPU compute and maybe a day of training data
| generation to train a T5 which I then "own".
|
| Also, running your own specialized model locally can be
| much faster than using someone's API.
| Ultimatt wrote:
| I suspect the author doesnt realise one request with hardly
| anything returned is many hundreds if not thousands of
| "tokens". It adds up very fast. Just some debug effort on a
| nonsense demo learning project cost $5 in a couple of hours.
| For maybe a hundred or so requests.
| carlosdp wrote:
| That's straight up not true, unless that "demo learning
| project" is feeding GPT the entire Bible or something.
|
| I have a project that uses davinci-003 (not even the cheaper
| ChatGPT API) like _crazy_ and I don 't come close to paying
| more than $30-120/month. With the ChatGPT API, it'll be 10x
| less...
| pharke wrote:
| You could have saved some money by writing tests. How much
| text were you sending at a time? I've been summarizing
| multiple 500 word chunks per query in my app as well as
| generating embeddings and haven't broken $10 over the course
| of a couple weeks.
| sebzim4500 wrote:
| It is not possible to pay anywhere close to $5 for a hundred
| requests, even if you used the max payload size every time.
|
| Is it possible you had a bug that caused you to send far more
| requests than you were intending to send? Or maybe you used
| the older models which are 10x more expensive?
| KyeRussell wrote:
| I can understand making a mistake on the Internet, but to say
| it with such snarky gusto is inexcusable.
|
| I've been playing with davinci pretty extensively and the
| only reason I've actually given OpenAI my credit card was
| because they won't let you do any fine-tuning with their free
| trial credit, or something like that. You're off by orders of
| magnitude, ESPECIALLY with the new 3.5 model.
| manmal wrote:
| I used it for dozens of requests yesterday and that amounted
| to less than 7 cents. I used MacGPT for that.
| celestialcheese wrote:
| 1000x this. Entity extraction from unstructured text with
| zero/few-shot is fantastic.
|
| I've got a use case where I need to extract model numbers from
| text - these LLMs are so good at it with very little work.
| manmal wrote:
| I'd wager it could cost anywhere between 1-10k to do that,
| which is a considerable amount of money. Might still be worth
| it though? If the alternative is mechanical turk, that would
| probably cost x1000-10000? Are there any ML alternatives that
| reliably produce useful results?
| ApolIllo wrote:
| When will they raise prices?
| nateburke wrote:
| When the PR gains from widespread adoption no longer cover the
| costs.
| kkielhofner wrote:
| As soon as there's a reliable base of foundational
| services/orgs/products/startups built on top of it.
|
| Especially with this edge, for now, it's Hotel California.
| lamontcg wrote:
| You think Google Search is polluted with AI written SEO'd trash
| already, well just wait for what it is in store when the chatbots
| attack whatever value is still contained in reddit-as-a-search-
| engine...
| Velc wrote:
| They already are. This week I uncovered a ChatGPT bot farm
| operating on reddit.
| sourcecodeplz wrote:
| Who uses Google anyway nowadays besides getting the address of
| a website you don't remember (assuming you know about ChatGPT).
| Toutouxc wrote:
| I can mostly tell when webpages from my search results are
| trying to bullshit me (and Kagi gives me some nice tools to
| suppress the bullshitting kind), but with ChatGPT I have no
| idea.
| darepublic wrote:
| Yes true if you trust the website then generally that trust
| can extend to all its content. You are putting your faith
| into the competence and consistency of a human being which
| is generally more trustworthy than the hit or miss results
| of a word predictor
| osigurdson wrote:
| I largely agree but I don't see how ChatGPT hits the same use
| cases as a fine-tuned model. Prompts can only have 8K tokens so
| any "in-prompt" fine tuning would have to be pretty limited. I'm
| not certain that the lack of ChatGPT fine tuning will be a
| permanent limitation however.
| riku_iki wrote:
| but they have API for fine tuning?..
| osigurdson wrote:
| They do, but not for gpt3.5 turbo (the ChatGPT model). See
| the following link for details:
| https://platform.openai.com/docs/guides/chat/is-fine-
| tuning-...
| riku_iki wrote:
| probably this may change soon..
| al2o3cr wrote:
| AI's main accomplishment so far is rendering its human shills
| indistinguishable from Markov generators.
| cs702 wrote:
| It will also make a _lot_ of simple machine-learning models
| obsolete. It 's just not that obvious yet.
|
| Imagine feeding a query akin to the one below to GPT4 (expected
| to have a 50,000-token context), and then, to GPT5, GPT6, etc.:
| query = f"The guidelines for approving or denying a loan are:
| {guidelines}. Here are sample application that
| were approved: {sample_approvals}. Here are
| sample applications that were denied: {sample_denials}.
| Please approve or deny the following loans: {loan_applications}.
| Write a short note explaining your decision for every
| application." decisions = LLM(query)
|
| Whether you like it or not, this kind of use of LLMs looks almost
| inevitable, because it will give nontechnical execs something
| they have always wanted: the ability to "read and understand" the
| machine's "reasoning." They machine will give them what they have
| always wanted: an explanation in plain English.
| logifail wrote:
| You've seen the movie The Big Short?
|
| Someone is likely coding:
|
| query = f"The guidelines for approving or denying a loan are:
| {guidelines}. Here are sample application that were approved:
| {sample_approvals}. Here are sample applications that were
| denied: {sample_denials}. Please write a loan application which
| is very likely to be approved. Provide necessary supporting
| details.
| cs702 wrote:
| Yeah, that sort of thing looks inevitable too.
| RC_ITR wrote:
| I imagined it and my theoretical supervised fine tuning bills
| are through the roof!
| vinni2 wrote:
| This would be a privacy nightmare. Banks would get into trouble
| if they send customer data to openAI. Unless they host their
| own LLM this is not yet practical.
| KyeRussell wrote:
| This is an entirely immaterial detail that could and would
| easily be addressed. I'm just going to assume that OpenAI's
| arm can be twisted wrt terms and conditions for Big Clients,
| as is standard practice. But even if it couldn't be, I've got
| no doubt that OpenAI will accept the literal shipping
| containers of money from a bank in exchange for an on-prem
| GPT-3 appliance.
| andix wrote:
| No, that probably won't work well. For such a task you need to
| train your model with thousands of samples, way too much for a
| simple prompt. But also you can't teach knowledge to a language
| model.
|
| The language model is trained for answering/completing text.
| You can do some additional training, but it will only pick up
| new words or new grammar. But it won't be able to learn how to
| calculate or how to draw conclusions.
| cs702 wrote:
| Your understanding is _very_ outdated. Go take a look at some
| of the things people are doing with LangChain to get a sense
| of what 's possible today and what will likely be possible in
| the very near future. LLMs are normally used in a zero-shot
| setting, without any kind of fine-tuning.
| loxias wrote:
| > Write a short note explaining your decision for every
| application
|
| Is there any evidence or reason to suspect that this would
| result in the desired effect? (explanations that faithfully
| correspond to the specifics of the input data resulting in the
| generated output)
|
| I suspect the above prompt _would_ produce _some_ explanations.
| I just don 't see anything tethering the explanations to the
| inner workings of the LLM. It would make some very convincing
| text that would convince a human... that would only be
| connected to the decisions by coincidence. Just like when
| ChatGPT hallucinates facts, internet access, etc. They look
| extremely convincing, but are hallucinations.
|
| In my unscientific experience, to the LLM, the "explanation"
| would be just more generation to fit a pattern.
| micromacrofoot wrote:
| A vast amount of the world is built on "close enough" and
| this is no different
| cs702 wrote:
| > Is there any evidence or reason to suspect that this would
| result in the desired effect?
|
| Yes.
|
| There's evidence that you can get these models to write
| chain-of-thought explanations that are consistent with the
| instructions in the given text.
|
| For example, take a look at the ReAct paper:
| https://arxiv.org/abs/2210.03629
|
| and some of the LangChain tutorials that use it:
|
| https://langchain.readthedocs.io/en/latest/modules/agents/ge.
| ..
|
| https://langchain.readthedocs.io/en/latest/modules/agents/im.
| ..
| Al-Khwarizmi wrote:
| Not to refute what you said, but what you describe is quite
| similar to what we humans call rationalization, and it has
| been argued (e.g. by Robert Zajonc) that most of the time we
| make decisions intuitively and then seek a rationalization to
| explain them.
|
| Also, good luck with human explanations in the presence of
| bias. No human is going to say that they refused a loan due
| to the race or sex of the applicant.
| pixl97 wrote:
| Well no smart humans, but it turns out there are plenty of
| dumb ones.
| roflyear wrote:
| Awful use of the language model.
| jcoc611 wrote:
| Probably should not fully automate this, but if you omit the
| "approve or deny" part then you got yourself a nice system that
| can pre-screen and surface statistical concerns with
| applications. You can still have a human making the final
| decisions
| cs702 wrote:
| Yes. In fact, I think that's how it will likely be used at
| first :-)
| jsemrau wrote:
| I have implemented an AI for credit decisioning in 13 countries
| on a multi-billion dollar portfolio. Here are my concerns about
| this elegant yet ineffective prompt:
|
| 1. LLMs in general are not build for quantitative analysis.
| Loan-to-value, income ratios, etc are not supposed to be
| calculated by such a model. Possible solution would be to
| calculate this beforehand and provide it to the model or train
| a submodel using a supervised learning approach to identify
| good/bad
|
| 2. Lending models are governed quarterly yet see relevant
| cohort changes only after a period of time after credit
| decision which can be many years. This prompt above does not
| take this performance of the cohort into consideration
|
| 3. Based on the governance companies adjust parameters and
| models regularly to adjust to changes in the environment. I.e.,
| a new car models comes out or the company is accessing a new
| customer segment. This process could not be covered well with
| this prompt since there would be no approvals/ denies for this
| segment.
|
| 4. Since transfer of personal-identififation data needs to be
| consented, it would likely be necessary to host an LLM like
| this internally or find a way to ensure there is no data
| leakage from the provider to other users on the platform.
|
| 5. Credit approval limits are not necessarily covered by this
| proceess. I.e., the credit decisions is unclear but would work
| with 5-10% more downpayment. Or the customer would be asked to
| lower the loan value or find someone in the company who can
| underwrite that loan volume. This person then has usually a
| bunch of additional questions (liquidity risk, interest risk
| ,etc) to ensure that the company is well protected and the
| necessary compliance checks are adhered to.
|
| 6. The discussions about this with regulators and auditors will
| be entertaining.
|
| Yet, I think it IS an elegant prompt which might provide some
| insights.
| cs702 wrote:
| There's evidence that you can get LLMs to write chain-of-
| thought explanations that are consistent with the
| instructions in the given text, including quantitative data,
| cohort performance, governance imperatives, qualitative
| considerations, etc. The models can even be given directions
| to write conditional approvals if necessary.
|
| To get a sense of what is and will be possible, take a look
| at the ReAct paper: https://arxiv.org/abs/2210.03629
|
| and some of the LangChain tutorials that use it:
|
| https://langchain.readthedocs.io/en/latest/modules/agents/ge.
| ..
|
| https://langchain.readthedocs.io/en/latest/modules/agents/im.
| ..
| mtlmtlmtlmtl wrote:
| Yes, let's put LLM in charge of loan applications. Definitely
| no financially devastating 2008-like slippery slope there.
|
| It'll be fine.
| thatwasunusual wrote:
| Yes, let's put people in charge of loan applications.
| Definitely no financially devastating 2008-like slippery
| slope there.
|
| It'll be fine.
| sebzim4500 wrote:
| You could check a random sample of them with expert humans to
| ensure there isn't a systematic issue causing you to issue
| large loans that you shouldn't be issuing.
|
| I doubt regulators would be happy with this though,
| especially since regulations are often a jobs program for
| former employees of regulators.
| psychphysic wrote:
| You mean you could ask a LLM to look at a sample of the
| loans and decide if there was a bias.
| NBJack wrote:
| The fun part is when a LLM hits that small probabiliy where
| they decide to go 'offscript'. It can result in a
| beautifully terrifying cascade of grammatically acceptable
| nonsense, and joe fun it would be in a legal document. We
| go from a loan for a home to a generous offer that includes
| a unicorn, a purple dishwasher, a unicorn, and a few
| dangling participles at the going market rate, all for the
| low low rate of 555.555.1212. [END TOKEN]--- WASHINGTON,
| D.C. President Trump today met with lawmakers to
| sebzim4500 wrote:
| I think the hope is that as LLMs get larger these issues
| will go away. Certainly there are issues with GPT-2 that
| completely went away when moving to larger models.
|
| Honestly, I haven't even seen GPT-3.5-turbo exhibit this
| behavior myself, although I am willing to believe it
| could happen. Llama 7B, however, goes off-script
| constantly.
| cs702 wrote:
| I laughed really hard -- _after_ trying to make sense of
| your comment!
|
| Thank you for posting this :-)
| rafram wrote:
| I don't think regulator nepotism is the main reason that
| the authorities would be uncomfortable with loan decisions
| being made by a system that definitionally reinforces
| existing biases and is incapable of thought. It's just a
| bad idea!
| credit_guy wrote:
| You don't need to check a random sample. You can have a
| policy where every single loan application is checked by a
| human, and you can add whatever affirmation is needed. It
| will still increase the productivity of those loan officers
| by a factor of 5. (Put it differently, banks would be able
| to lay off 80% of their loan officers).
| corbulo wrote:
| This is the exact kind of dystopic thinking that is feared
| with the use of AI.
|
| "We regularly take randomized samples and have not found
| error, your appeal has been denied."
|
| I mean come on, ethics anybody??
| sebzim4500 wrote:
| So long as it is better than the thing it replaces, I
| don't get the big deal.
| sangnoir wrote:
| Yep, can't wait for loan "hacks" like randomly name-dropping
| job titles and institutions in the loan application. "Our pet
| hamsters 'Stanford University' and 'Quantitative Analyst' are
| looking forward to having more room"
| KyeRussell wrote:
| We call him little quanty tables.
| [deleted]
| sposeray wrote:
| [dead]
| virtualjst wrote:
| Where is juniper notebook code for the chatgpt_generate function?
| virtualjst wrote:
| Where is the jupyter notebook code with the example
| chatgpt_generate function?
| minimaxir wrote:
| That function is just a wrapper over the base
| openai.ChatCompletion.create call from the documentation with
| no changes:
| https://twitter.com/minimaxir/status/1631044069483749377
| jonatron wrote:
| I pasted the article into a Markov Chain:
|
| exist generative text unfortunately with the current recognition
| of its creation which uses the chatgpt api which can confirm the
| media has weirdly hyped the upcoming surge of ai generated
| content its hard to keep things similar results without any
| chatgpt to do much better signalto-noise.
|
| --
|
| Is ChatGPT just an improved Markov Chain?
| bitL wrote:
| RLHF uses Markov chains as its backbone, at least theoretically
| (deep NN function approximations inside might override any
| theoretical Markov chain effect though).
| TechBro8615 wrote:
| I guess if you squint, it kinda is, in the sense that it
| generates one token at a time.
| theGnuMe wrote:
| It is in the sense that it is Markov chain with an 8k token
| memory and the "MCMC step" is the DNN.
| nerdponx wrote:
| It's not a Markov chain because by definition a Markov chain
| only looks at the previous word. ChatGPT looks at a long
| sequence of previous words. But the general idea is still
| broadly the same.
| PartiallyTyped wrote:
| That's not correct. In a Markov chain, the current state is
| a sufficient characteristic of the future. For all intents
| and purposes you can create a state with sufficiently long
| history to look at a long sequence of words.
| nerdponx wrote:
| Also fair, but then the "current" state would _also_ be a
| long window /sequence. Maybe that interpretation is valid
| if you look at the activations inside the network, but I
| wouldn't know about that.
| PartiallyTyped wrote:
| Yes, the state for both is a long window / sequence.
| Under this view, for the transformer we do not need to
| compute anything for the previous tokens as due to the
| causal nature of the model, the tokens at [0, ... N-1]
| are oblivious to the token N. For token N we can use the
| previous computations since they do not change.
| Accujack wrote:
| Actually, it looks at all the _meanings_ of the tokens
| within its window.
| xwdv wrote:
| Hence why you have to squint.
| jfengel wrote:
| To a first approximation, yes.
|
| The second approximation has significant differences, but
| that's an ok first pass at it.
| skybrian wrote:
| As a mathematical model, it's almost completely unhelpful,
| like saying that all computers are technically state machines
| because they have a finite amount of memory.
|
| Treating every combination of 4k tokens as a separate state
| with independent probabilities is useless for making
| probability estimates.
|
| Better to say that it's a stateless function that computes
| probabilities for the next token and leave Markov out of it.
| PartiallyTyped wrote:
| "just" is doing a lot of heavy lifting.
|
| ChatGPT needs a language model and a selection model. The
| language model is a predictive model that given a state
| generates tokens. For chatGPT it's a decoder model (meaning
| auto-regressive / causal transformer). The state for the
| language model is the fixed length window.
|
| For a Markov chain, you need to define what "state" means. In
| the simplest case you have a unigram where each next token is
| completely independent of all previously seen tokens. You can
| have a bi-gram model, where the next state is dependent on the
| last token, or an n-gram model that uses the last N-1 tokens.
|
| The problem with creating a markov chain with n-token state is
| that it simply doesn't generalize at all.
|
| The chain may be missing states and can't produce a probability
| distribution. e.g. since we use a fixed window for the state,
| our training data can have a state like "AA" that transitions
| to B, thus the sentence is "AAB". The model however may keep
| producing stuff, thus we need to get the new state, which is
| "AB". If "AB" is out of the dataset, well... tough luck, you
| need to improvise on how to deal with this. Approaches exist
| but nowhere near as good of a performance as a basic RNN let
| alone LSTMs and transformers.
| ar9av wrote:
| ChatGPT and Markov Chain are both text-generating models, but
| they use different approaches and technologies. Markov Chain
| generates text based on probabilities of word sequences in a
| given text corpus, while ChatGPT is a neural network-based
| model.
|
| Compared to Markov Chain, ChatGPT is more advanced and capable
| of producing more coherent and contextually relevant text. It
| has a better understanding of language structure, grammar, and
| meaning, and can generate longer and more complex texts.
| fancyfredbot wrote:
| OpenAI could probably make money offering the API for free at
| this point - the data they are getting is so valuable for them in
| building a competitive advantage in this space.
|
| Once they know use cases for the model they can make sure they
| are very good at those, and then they can consider hiking the
| price.
| eeegnu wrote:
| Charging a small amount is more optimal since it mitigates API
| spam without having to set a low rate limit. It also ties your
| users to a financial id, which is (probably) harder to get in
| bulk for nefarious purposes than just requiring a phone number
| to sign up.
| 1f60c wrote:
| I would love a breakdown of how they made the incredible header
| image (https://minimaxir.com/2023/03/new-chatgpt-
| overlord/featured....)
| preommr wrote:
| It says it in the caption: Stable diffusion + controlnet.
|
| If you haven't seen it already, controlnet has allowed for
| massive improvements in addint constraints to generated images.
|
| Here's an example of using a vector logo to make it semalessly
| integrate it in different environments:
| https://www.reddit.com/r/StableDiffusion/comments/11ku886/co...
| minimaxir wrote:
| Here's a few bonus OpenAI charcuterie:
| https://twitter.com/minimaxir/status/1633635144249774082
|
| 1. I used a ControlNet Colab from here based on SD 1.5 and the
| original ControlNet app:
| https://github.com/camenduru/controlnet-colab
|
| 2. Screenshotted a B/W OpenAI logo from their website.
|
| 3. Used the Canny adapter and the prompt: charcuterie board,
| professional food photography, 8k hdr, delicious and vibrant
|
| Now that ControlNet is in diffusers, my next project will be
| creating an end-to-end workflow for these types of images:
| https://www.reddit.com/r/StableDiffusion/comments/11bp30o/te...
| aqme28 wrote:
| It says it's made with ControlNet and Stable Diffusion.
| Probably SD'd "charcuterie board" over a drawing of the logo.
| napier wrote:
| Yes last week, but Llama 65B is as of today running on an M1 so
| yeah difficult to predict how centralised AI APIs will play out
| six months from now:
| https://twitter.com/lawrencecchen/status/1634507648824676353
|
| Still expecting OAI to be able to leverage a flywheel effect as
| they plough their recent funding injection into new foundation
| models and other systems innovations but there's also going to be
| increasing competition from other platform providers and also the
| open source community boosted by competitors open sourcing /
| leaking expensive to train model tech with the second order
| function of diffusing wind from sales.
| zirgs wrote:
| One of the biggest drawbacks of ChatGPT is that OpenAI knows
| everything that its users are doing with it. Every prompt and its
| answer are being logged. Hackers might breach OpenAI systems and
| leak its data.
|
| If you're Rockstar that's working on GTA 7 then you'll propbably
| want to keep all the AI written mission scripts, story ideas,
| concept art and other stuff like that on your own servers.
| altdataseller wrote:
| Isnt this the case for a lot of web products? Hackers can hack
| into Adobe and steal my prototypes. They can hack into my
| Dropbox and steal my files. They can hack into my Asana project
| and steal my roadmap
| BoiledCabbage wrote:
| > OpenAI retains API data for 30 days for abuse and misuse
| monitoring purposes.
|
| They just changed this. It is now only 30 day retention -
| https://openai.com/policies/api-data-usage-policies
| corbulo wrote:
| Data retention is kind of meaningless in this context since
| there's so many ways it is laundered/absorbed/analyzed while
| not technically violating whatever legalese they use this
| month.
| throwawayapples wrote:
| Does that only apply to API usage and not ChatGPT the web
| app?
|
| It would seem to, because the web app doesn't seem to expire
| your old chats.
| sebzim4500 wrote:
| I agree with you, but I do think that people are overstating
| the problem. It's no worse than sticking your data on the
| cloud, and a huge portion of companies are doing that willingly
| already.
| simonw wrote:
| ... which is one of the strongest arguments for being able to
| run a large language model on your own hardware instead!
| https://til.simonwillison.net/llms/llama-7b-m2
| Karrot_Kream wrote:
| Don't want to be vapid, but these are some cool guides! I
| know how to run these models but want to link my friends to
| guides to get started. Thanks!
| simonw wrote:
| Does anyone have a good feel for how likely it is that OpenAI
| might be running it at this price to get companies hooked, with
| plans to then raise the price later on once everyone is locked
| in?
|
| I'm personally much more excited about the LLaMA + llama.cpp
| combo that finally brings GPT-3 class language models to personal
| hardware. I wrote about why I think that represents a "Stable
| Diffusion" moment for language models here:
| https://simonwillison.net/2023/Mar/11/llama/
| pixl97 wrote:
| Depends how much competition ends up in this market. If there
| is plenty of competition that gives good results at a similar
| costs rising prices will be difficult. Now if it actually costs
| far more to run than the API cost is currently, we'll see it go
| up.
| minimaxir wrote:
| I pointed that out in the caveats since that happened with
| Google Maps, but in practice I don't think it'll happen (or if
| it happens it will only be a slight increase) since that would
| seriously upset its users. Especially since the low price was
| likely due to competition anyways.
|
| In the case of Google Maps it was effectively a monopoly.
| iampims wrote:
| Being a monopoly is what OpenAI is aiming for.
| minimaxir wrote:
| Specifically in the case of Google Maps it was a de facto
| monopoly, and thus has full control of pricing, due to the
| lack of _good_ competitors (OpenStreetMap doesn 't count).
|
| For LLMs, instead competition is very fierce which will
| pressure down prices such as here with the ChatGPT API.
| mirekrusin wrote:
| They want to take the widest possible share, which atm, without
| competition means bringing on people/companies that wouldn't
| otherwise consider it.
|
| The price will only go down when competition appears. They can
| only slow it down with the cheapest possible offering (to put
| market entry bar higher for competitors). They don't know what
| competition will do, but they know if they move fast they'll
| have very low chance of catching up anytime soon and that's all
| that matters.
|
| Competition will be interesting because interface is as simple
| as it can be (easy to switch to different provider).
|
| Providers can hook people though pre-training but I don't know
| if it's possible to do dedicated pre-training on large models
| like this. They may need to come up with something special for
| that.
| fullshark wrote:
| It's very likely, they are in a race and that's the tech
| playbook to win the market.
| fareesh wrote:
| How does one go about minimizing costs for certain situations?
|
| For example if I share a db schema and then ask it to generate
| some sql, I need to share that entire db schema for every single
| question that follows, is that right?
|
| Or is it possible for me to somehow pay and have it "retain" that
| schema knowledge for all subsequent queries without having to
| send the schema along with every single question?
| mirekrusin wrote:
| It's called fine tuning [0] but it's not yet available for
| chatgpt, only older models.
|
| [0] https://platform.openai.com/docs/guides/fine-tuning
| fareesh wrote:
| Yeah that seems to be the trouble - the pricing is great with
| ChatGPT but not so much with the older ones
| mirekrusin wrote:
| That may be partially because it's not possible to do pre-
| training on such a large model.
| fareesh wrote:
| On the web version is it doing that under the hood? I.e.
| if I have a 10+ follow up conversation does it start from
| the beginning each time?
| mirekrusin wrote:
| It keeps context of 8k tokens.
|
| In pre-training you're using much more examples and
| network is tuned around them.
___________________________________________________________________
(page generated 2023-03-11 23:00 UTC)