[HN Gopher] Using GPT-3 to explain how code works
___________________________________________________________________
Using GPT-3 to explain how code works
Author : simonw
Score : 100 points
Date : 2022-07-09 15:53 UTC (7 hours ago)
(HTM) web link (simonwillison.net)
(TXT) w3m dump (simonwillison.net)
| layer8 wrote:
| Tangential question: Is GPT-3 (and similar tools) being kept up
| to date with new information? Or would it basically have to be
| retrained from scratch to properly incorporate new data?
| simonw wrote:
| There's a mechanism you can use to add your own training data
| on top of GPT-3. This tutorial talks about using that to
| generate song lyrics (about half way through):
| https://wingedsheep.com/songwriting-with-gpt-3/
|
| I don't know how often OpenAI build new versions of the core
| GPT-3 language model.
| abirch wrote:
| What would be highly beneficial is for the AI to understand
| changes so when it spots an old pattern in my code, it could
| suggest the new pattern. I'd pay for a Copilot that did that.
| pornel wrote:
| This as an assistant/teacher could be incredibly empowering for
| people learning to program.
|
| I'm sceptical of AI writing programs for people, but I can easily
| imagine such AI guiding/unblocking non-programmers enough for
| them to actually program.
| layer8 wrote:
| It's a double-edged sword. From the article:
|
| _GPT-3 doesn't actually know anything about anything at all.
| It's a huge pattern generator. You can't trust anything it
| says, because all it does is group words together into
| convincing looking shapes based on text that it's seen before.
|
| Once again, I'm reminded that tools like GPT-3 should be
| classified in the "bicycles for the mind" category. You still
| have to know how to pedal!
|
| They're fantastic tools for thinking, but to actually use their
| output effectively requires VERY deep knowledge--both of the
| subject matter in question, and of the way that the AI tools
| themselves work._
| l33t2328 wrote:
| What is thinking if not pattern generating?
| trention wrote:
| It's not enough to state that the human mind is a pattern
| matcher as pretty much everyone of the "scale is all you
| need" crowd does. You have to prove it too.
|
| Otherwise it's just a variant of Russell's teapot.
| abrax3141 wrote:
| Even if you prove it, it's not useful.
| abrax3141 wrote:
| You aren't actually serious, are you?
| Swenrekcah wrote:
| We use far more variety of inputs than GPT-3.
|
| GPT-3 has more textual training data than any human could
| read in a thousand lifetimes, but we base our judgment on
| much more than that.
| layer8 wrote:
| I agree that in a sense it's probably only a matter of
| degree. However, take for example its explanation of the
| _xm_ modifiers, where it only explains the _x_ , but
| remains silent one the fact that it only explains the _x_.
| It's unclear whether it has any awareness that it provides
| an incomplete and potentially misleading explanation. It
| would be interesting to see how it reacts to "what you
| explained is only the _x_ , what about the _m_?".
| duskwuff wrote:
| And, in particular, GPT-3 will be _very_ easily misled by
| plausible-but-incorrect inputs -- like programs written by
| amateur developers. I definitely wouldn 't trust it to find
| subtle logical errors in a student's computer program, for
| example.
| Transfinity wrote:
| The types of error made by the model in the article are
| exactly the sort of error that I (senior engineer / decade of
| experience) would make if asked to describe how something
| works and I didn't want to spend too much time on the
| details. The difference is, I can usually give a decent
| estimate of how sure I am of something, and know how to zero
| in on the truth when it's important.
|
| Maybe a confidence level for a given explanation, along with
| some sort of "here's where you can go to learn more" would be
| useful? No idea if language models would be good at that kind
| of meta reasoning.
| loufe wrote:
| This is fantastic, I've been enjoying myself thoroughly with
| Dall-E 2 recently and similar to this love trying to imagine how
| the training method / training data must have led to the output.
| It's truly amazing times we live in, but it will take a lot more
| work than people 70 years ago thought to get to a level of
| polish.
| booleandilemma wrote:
| Could they do this for legal documents too?
|
| I'd hate to see just the programmers put out of work by GPT-3.
| Transfinity wrote:
| A buddy of mine works at a startup doing exactly this:
| https://www.advocat.ai/
|
| Unsure if they use GPT-3 specifically, but the core idea is the
| same.
| akamoonknight wrote:
| This is a fascinating use case that wouldn't have occurred to me
| at all, quite cool, even if it "doesn't know anything about
| anything at all".
|
| It even does pretty good with Verilog, which I assumed it would
| have issues with, admittedly some of this is just "google-able",
| but even so. It can even create a reasonable attempt at a
| testbench (even though it doesn't quite get it right), which is
| crazy to me.
|
| *Explain this code:
|
| module my_module(a,c,y); input a,c; output y; assign y= c ? a :
| 1'bz; endmodule*
|
| This code is a Verilog module that defines three inputs (a, c, y)
| and one output (y). The output y is assigned to the value of a if
| c is true, otherwise it is assigned to 1'bz.
|
| *What is 1'bz ?*
|
| 1'bz is a Verilog constant that represents a bit value of 'z',
| which is used to indicate a high impedance state.
|
| *Why is high impedance used ?*
|
| High impedance is used in digital circuits to allow multiple
| drivers to share a single bus without contention. When two or
| more drivers attempt to drive the same bus at the same time, a
| contention can occur and the bus voltage can become undefined. By
| using high impedance, drivers can avoid contention and allow the
| bus to remain stable.
|
| *Give me a testbench for this module*
|
| module testbench; reg a; reg c; wire y; my_module
| dut(a, c, y); initial begin a = 1'b0;
| c = 1'b0; #1; a = 1'b1; c = 1'b1;
| #1; a = 1'bz; c = 1'b1; #1; a =
| 1'bx; c = 1'b1; #1; $stop; end
|
| endmodule
|
| *Create the same testbench but add validation of the output.*
|
| module testbench; reg a; reg c; wire y; my_module
| dut(a, c, y); initial begin a = 1'b0;
| c = 1'b0; #1; a = 1'b1; c = 1'b1;
| #1; a = 1'bz; c = 1'b1; #1; a =
| 1'bx; c = 1'b1; #1; $stop; end
| always @(*) begin if (c == 1'b0) assert(y ==
| 1'b0); else if (c == 1'b1) assert(y == a);
| else if (c == 1'bz) assert(y == 1'bz); else if
| (c == 1'bx) assert(y == 1'bx); end
|
| endmodule
| punnerud wrote:
| This would be great for blind people, or you could program while
| biking.
| ctoth wrote:
| As a blind programmer I'm trying to understand how I should
| find this more useful than anyone else?
| netsec_burn wrote:
| I think what they're saying is, you can quickly understand
| the contents of a file by having it summarized for you
| audibly. Versus having each token read out to you. Then you
| can locate the part that you're trying to program faster.
| f38zf5vdt wrote:
| > As with everything GPT-3, this answer sounds very convincing!
| But is this actually true? I would expect the SQL query optimizer
| to be smart enough to optimize the LEFT JOIN to the point where
| it would run efficiently. I think GPT-3's answer to that question
| is actually very misleading.
|
| I really enjoy how well it gives wrong answers. The internet
| being polluted with GPT3-like text models is going to send us
| back to the time when verifying anything online as truthful is
| difficult, if it hasn't already.
| mwint wrote:
| Maybe my next move should be to start a consulting shop: "We
| Fix GPT-3 Code".
| abrax3141 wrote:
| Good luck with that. I don't think that these things are
| fixable. They're basically giant grammatical confabulation
| engines. When you want grammatically-correct random noise,
| great. Don't fly a plane with them, please...or at least not
| one I'm on!
| trention wrote:
| He meant fixing the output(s) not the model that produced
| them.
| jwilk wrote:
| > The (?xm) notation is a modifier that allows the regular
| expression to span multiple lines and to use comments.
|
| This is incorrect.
|
| (?xm) enables the VERBOSE (X) and MULTILINE (M) flags.
|
| From the docstring: M MULTILINE "^" matches
| the beginning of lines (after a newline) as
| well as the string. "$" matches the end of
| lines (before a newline) as well as the end
| of the string. X VERBOSE Ignore whitespace and
| comments for nicer looking RE's.
| xdfgh1112 wrote:
| The X flag also allows the written regex to span multiple
| lines, i.e. it counts newlines as whitespace. Interesting
| confusion there.
| simonw wrote:
| Hah yeah, good catch - I'd missed that incorrect "allows it to
| span multiple lines" statement.
___________________________________________________________________
(page generated 2022-07-09 23:00 UTC)