[HN Gopher] Using GPT-3 for plain language incident root cause f... ___________________________________________________________________ Using GPT-3 for plain language incident root cause from logs Author : stochastimus Score : 48 points Date : 2021-01-12 17:08 UTC (5 hours ago) (HTM) web link (www.zebrium.com) (TXT) w3m dump (www.zebrium.com) | bbu wrote: | This is pretty cool! However, these two samples are very simple | to solve. I'd love an "AI" to find root causes for problems that | are not obvious. Just throw the whole log collection at it and | let it solve all the issues. One can dream ;) | m463 wrote: | GPT-3-stackoverflow | deeeeplearning wrote: | But seriously, was Stackoverflow part of the training data | used to train GPT-3? Would definitely be an interesting fine | tuning experiment | stochastimus wrote: | From what I've read, the answer is "yes", stackoverflow was | crawled. EDIT: I looked and stackoverflow is included in | the Common Crawl dataset, which is one of the datasets on | which GPT-3 was trained. Having said that, it's not clear | to me the degree of coverage of that domain guaranteed by | that crawl... looks pretty comprehensive, though. | http://index.commoncrawl.org/CC- | MAIN-2020-24-index?url=*.sta... | stochastimus wrote: | Yeah, my experience so far has been that if I just pile a bunch | of logs in there, if it's not given salient lines, the language | model tends to either rat-hole on some particular detail that's | irrelevant, or else construct a non-factual narrative. But, | when Zebrium did the picking of these lines autonomously and | GPT-3 summarized them, we see a meaningful summary. Having said | that, I would also like to get GPT-3 more and more savvy with | larger and larger log-based prompts, and I'm hoping it's | possible with some tweaks to the prompt and some finetuning. | I'll keep the blog posted as we do more experiments. | a-dub wrote: | this is really cool! | [deleted] | brianjunyinchan wrote: | Super interesting. I wonder what other latent domain-specific | intelligence GPT-3 picked up during training, that is parseable | with text in and text out. Like a flash cards generator? | stochastimus wrote: | Hmm, I like this direction - so maybe, as the user is | navigating the incident, let them steer the model with | questions and/or additional lines. Is that sort of what you'd | envision? | sthatipamala wrote: | Polar (https://getpolarized.io/) has a GPT-3 based flash card | generator from text highlights. It's available to premium | subscribers. | king_magic wrote: | I'm fairly bearish on GPT-3, but this is actually a pretty cool | application. | jacques_chester wrote: | Is there a reason I'd use this approach over a process mining / | log mining system? I feel like it needs me to guess the right | question to get an answer. | stochastimus wrote: | Well, I've been trying really hard not to point it out because | I don't want this to be like a commercial. :) But, the idea | here is that the Zebrium ML picks the incident lines | unsupervised; then, the GPT-3 model creates the summary | unsupervised. So I guess the combination is what we've been | working on in a private beta, so that the user can get the best | of both worlds. | jacques_chester wrote: | Gotcha. I had understood it to be purely GPT-3 somehow, | rather than as a second step. | mckirk wrote: | That's cool and all, but I'm pretty sure what we really want to | see is | | "The expert described what had happened, in the form of a Haiku:" | stochastimus wrote: | I just tried this and I might stick with these settings! ;-) | For the postgresql example in the blog, I used your prompt. | Here's what I got: | | The logs were in a mess, But the expert could see, That the | database was in distress. | phaemon wrote: | I'm kind of surprised GPT-3 doesn't "understand" haiku. You'd | think it could extrapolate the rules? | | The logs are broken!, Sysadmin sweeps up the leaves, The | database cried | rictic wrote: | The encoding used by GPT-2 and GPT-3 greatly obscures many | of the textual properties of words. This at least partly | accounts for why it has so much trouble with meter, rhyme, | syllables, and some math. | | More info: https://www.gwern.net/GPT-3#bpes | stochastimus wrote: | Thanks for putting that info here! ___________________________________________________________________ (page generated 2021-01-12 23:00 UTC)