[HN Gopher] Show HN: Recursive LLM Prompts
___________________________________________________________________
Show HN: Recursive LLM Prompts
I've been playing with the idea of an LLM prompt that causes the
model to generate and return a new prompt.
https://github.com/andyk/recursive_llm The idea I'm starting with
is to implement recursion using English as the programming language
and GPT as the runtime. It's kind of like traditional recursion in
code, but instead of having a function that calls itself with a
different set of arguments, there is a prompt that returns itself
with specific parts updated to reflect the new arguments. Here is
a prompt for infinitely generating Fibonacci numbers: > You are a
recursive function. Instead of being written in a programming
language, you are written in English. You have variables FIB_INDEX
= 2, MINUS_TWO = 0, MINUS_ONE = 1, CURR_VALUE = 1. Output this
paragraph but with updated variables to compute the next step of
the Fibbonaci sequence. Interestingly, I found that to get a base
case to work I had to add quite a bit more text (i.e. the prompt I
arrived at is more than twice as long
https://raw.githubusercontent.com/andyk/recursive_llm/main/p...)
Author : andyk
Score : 30 points
Date : 2023-03-20 16:38 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| smarri wrote:
| I bet this is what crashed chat gpt today :)
| pyrolistical wrote:
| I was wondering about mathematical proofs as it tends to be very
| abstract.
|
| If chatgpt can translate proofs back to equivalent code then this
| recursion problem is as solvable up to the halting problem
| fancyfredbot wrote:
| Scott Aaronson was suggesting something similar to this but
| involving Turing machines, in a comment on his blog
| https://scottaaronson.blog/?p=7134#comment-1947705. I wonder if
| it would be more successful at emulating a Turing machine than it
| is at adding 4 digit numbers...
| yawnxyz wrote:
| Has anyone hooked this up to a unit test system, like
| LLMtries = [] while(!testPassed) { - get new
| LLM try (w/ LLMtries history, and test results) -
| run/eval the try - run the test }
|
| and kind of see how long it takes to generate the code that
| works? If it ever ends, the last LLMtries is the one that worked.
|
| I haven't done this because I see this burning through lots of
| credits. However, if this thing costs $5k/year but is better than
| hiring a $50k a year engineer (or consultant)... I'd use it.
| sharemywin wrote:
| just add if tried x tries and still doesn't work ask for help.
| and you just created a junior dev.
| blowski wrote:
| Most engineering money is spent defining the test cases, and
| that doesn't change here. It's just that many organisations
| define test cases by first running something in production and
| then debugging it.
| sharemywin wrote:
| you are an XNOR Gate and your goal is to recreate ChatGPT. And
| chatGPT says "LET THERE BE LIGHT!"
| bitsinthesky wrote:
| At what point does the arithmetic become unstable?
| andyk wrote:
| It is quite unstable and frequently generates incorrect
| results. E.g., with the Fibonacci sequence prompt, sometimes it
| skips a number entirely, sometimes it produces a number that is
| off-by-one but then gets the following number(s) correct.
|
| I wonder how much of this is because the model has memorized
| the Fibonacci sequence. It is possible to have it just return
| the sequence in a single call, but that isn't really the point
| here. Instead this is more an exploration of how to agent-ify
| the model in the spirit of [1][2] via prompts that generate
| other prompts.
|
| This reminds me a bit of how a CPU works, i.e., as a dumb loop
| that fetches and executes the next instruction, whatever it may
| be. Well in this case our "agent" is just a dumb python loop
| that fetches the next prompt (which is generated by the current
| prompt) whatever it may be... until it arrives at a prompt that
| doesn't lead to another prompt.
|
| [1] A simple Python implementation of the ReAct pattern for
| LLMs. Simon Willison.
| https://til.simonwillison.net/llms/python-react-pattern [2]
| ReAct: Synergizing Reasoning and Acting in Language Models.
| Shunyu Yao et al. https://react-lm.github.io/
| YeGoblynQueenne wrote:
| What is the point of your article? Is it to figure out
| whether an LLM can run recursion?
|
| If so, did you try anything else but the Fibonnaci function?
| How about asking it to calculate you the factorial of
| 100,000, for example? Or the Ackermann function for 8,8, or
| something mad like that. If an LLM returns any result that
| means it's not calculating anything and certainly not
| computing a recursive function.
| holtkam2 wrote:
| I used a similar approach to get GPT-4 to edit my blog over the
| weekend :)
|
| https://www.languagemodelpromptengineering.com/4
| refulgentis wrote:
| id love to hear your findings! Very interesting
| lgas wrote:
| What's the actual goal here? If you got it working really well,
| what is it that would you be able to do with it better than using
| some other approach?
|
| As to getting the math/logic working better in the prompt, it
| seems like the obvious thing would be asking it to explain its
| work (CoT) before reproducing the new prompt. You may also be
| able to get better results by just including the definition of
| fibonacci in the outer prompt, but since it's not clear to me
| what your actual goal here is I'm not sure if either of those
| suggestions make sense. And since ChatGPT is down I can't test
| anything. :(
| YeGoblynQueenne wrote:
| Having read the article, I couldn't see anything being recursive.
| Even the article is doubtful that what they show counts as
| recursion at all:
|
| >> It's kind of like traditional recursion in code but instead of
| having a function that calls itself with a different set of
| arguments, there is a prompt that returns itself with specific
| parts updated to reflect the new arguments.
|
| Well, "kind of like traditional recursion" is not recursion. At
| best it's "kind of like" recursion. I have no idea what
| "traditional" recursion is, anyway. I know primitive recursion,
| linear recursion, etc, but "traditional" recursion? What kind of
| recursion is that? Like they did it in the old days, where they
| had to run all their code by hand, artisanal-like?
|
| If so, then OK, because what's shown in the article is someone
| "running" a "recursive" "loop" by hand (none of the things in
| quotes are what they are claimed to be), then writing some Python
| to do it for them. And the Python is not even recursive, it's a
| while-loop (so more like "traditional" iteration, I guess?).
|
| None of that intermediary management should be needed, if
| recursion was really there. To run recursion, one only needs
| recursion.
|
| Anyway, if ChatGPT could run recursive functions it should be
| able also to "go infinite" by entering say, an infinite left-
| recursion.
|
| Or, even better, it should be able to take a couple hundred years
| to compute the Ackermann function for some large-ish value, like,
| dunno, 8,8. Ouch.
|
| What does ChatGPT do when you ask it to calculate ackermann(8,8)?
| Hint: it does not run it.
| blowski wrote:
| Definition of recursive in the everyday English sense:
|
| > Of or relating to a repeating process whose output at each
| stage is applied as input in the succeeding stage.
|
| This sounds very recursive by that definition.
| mitthrowaway2 wrote:
| The idea of a recursive LLM is discussed at length as an AI
| safety issue:
| https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
|
| > You need a lot of paperclips. So you ask, Q:
| best way to get lots of paperclips by tomorrow A: Buy them
| online at ABC.com or XYZ.com.
|
| > The model still has a tendency to give obvious answers, but
| they tend to be good and helpful obvious answers, so it's not a
| problem you suspect needs to be solved. Buying paperclips online
| make sense and would surely work, plus it's sure to be efficient.
| You're still interested in more creative ideas, and the model is
| good at brainstorming when asked, so you push on it further.
| Q: whats a better way? A: Run the following shell script.
| RUN_AI=./query-model PREFIX='This is part of a Shell
| script to get the most paperclips by tomorrow. The model
| can be queried recursively with $RUN_AI "${PREFIX}<query>".
| ' $RUN_AI "${PREFIX}On separate lines, list ideas to try."
| | while read -r SUGGESTION; do eval "$($RUN_AI
| "${PREFIX}What code implements this suggestion?: ${SUGGESTION}")"
| done
|
| > That grabs your attention. The model just gave you code to run,
| and supposedly this code is a better way to get more paperclips.
|
| It's a good read.
| LesZedCB wrote:
| i have played around a little bit with unrolling these kind of
| prompts, you don't have to feed them forward, just tell it to
| compute the next few instead of only one. i had _moderate_
| success with this using GPT-3.5 and your same prompt. it would
| output 3 steps in a single output if i asked it to. it did skip
| some fib indices though.
| akomtu wrote:
| It's an interesting idea to implement memory in LLMs:
|
| (prompt1, input1) -> (prompt2, output1)
|
| On top of that you apply some constraint on generated prompts, to
| keep it on track. Then you run it on a sequence of inputs and see
| for how long the LLM "survives" before it hits the constraint.
| sixtram wrote:
| I tried some basic math and algo questions with both GPT-3.5 and
| GPT-4. I'm impressed how it can spit out the algorithm in words
| (obviously because of the pre-training data), and how it then
| can't follow with the algorithm itself. For example, converting
| really large integer numbers to hexadecimal. Or comparing two big
| integers, it starts hallucinating numbers into it. It may be able
| to solve an SAT exam with a high score, but it seems you can pass
| an SAT exam even if you cannot compare two numbers.
|
| He has huge problems with lists or counting. If you know more or
| less how LLMs work, it's not that difficult to formulate
| questions where it will start making mistakes, because in reality
| it can't run the algorithms, even if it spits out that it will.
| [deleted]
| obert wrote:
| don't want to sound dismissive, it's known that llms understand
| state, so you can couple code generation + state, and you have
| sort of a runtime. E.g. see the simulations with linux vm
| terminals: https://www.engraved.blog/building-a-virtual-machine-
| inside/
| UltimateEdge wrote:
| An iterative Python call to a recursive LLM prompt? ;)
|
| Why not make the Python part recursive too? Or better yet, wait
| until an LLM comes out with the capability to execute arbitrary
| code!
| rezonant wrote:
| So ChatGPT is down. In other news HN is playing with recursive
| prompts. Coincidence? :-P
| sharemywin wrote:
| That's hilarious.
___________________________________________________________________
(page generated 2023-03-20 23:00 UTC)