[HN Gopher] Show HN: Magentic - Use LLMs as simple Python functions
___________________________________________________________________
Show HN: Magentic - Use LLMs as simple Python functions
This is a Python package that allows you to write function
signatures to define LLM queries. This makes it easy to mix regular
code with calls to LLMs, which enables you to use the LLM for its
creativity and reasoning while also enforcing structure/logic as
necessary. LLM output is parsed for you according to the return
type annotation of the function, including complex return types
such as streaming an array of structured objects. I built this to
show that we can think about using LLMs more fluidly than just
chains and chats, i.e. more interchangeably with regular code, and
to make it easy to do that. Please let me know what you think!
Contributions welcome. https://github.com/jackmpcollins/magentic
Author : jackmpcollins
Score : 165 points
Date : 2023-09-26 16:31 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| conor_f wrote:
| Looks super cool! A few questions:
|
| 1) Can you get the actual code output or will this end up calling
| OpenAI each function call? 2) What latency does it add? What
| about token usage? 3) Is the functionality deterministic?
| jackmpcollins wrote:
| 1) The OpenAI API will be queried each time a "prompt-function"
| is called in python code. If you provide the `functions`
| argument in order to use function-calling then magentic will
| not execute the function the LLM has chosen, instead it returns
| a `FunctionCall` instance which you can validate before
| calling.
|
| 2) I haven't measured additional latency but it should be
| negligible in comparison to the speed of generation of the LLM.
| And since it makes it easy to use streaming and async functions
| you might be able to achieve much faster generation speeds
| overall - see the Async section in the README. Token usage
| should also be a negligible change from calling the OpenAI API
| directly - the only "prompting" magentic does currently is in
| naming the functions sent to OpenAI, all other input tokens are
| written by the user. A user switching from explicitly defining
| the output schema in the prompt to using function-calling via
| magentic might actually save a few tokens.
|
| 3) Functionality is not deterministic, even with
| `temperature=0`, but since we're working with python functions
| one option is to just add the `@cache` decorator. This would
| save you tokens and time when calling the same prompt-function
| with the same inputs.
|
| ---
|
| 1) https://github.com/jackmpcollins/magentic#usage 2)
| https://github.com/jackmpcollins/magentic#asyncio 3)
| https://docs.python.org/3/library/functools.html#functools.c...
| BoorishBears wrote:
| I've personally found frameworks like this to get in the way of
| quality COT: It's rare for a prompt that takes great advantage of
| the LLM's reasoning to fit in the format these generators
| encourage
|
| A friend mentioned how terrible most cold email generators are at
| actually generating natural feeling emails. It just took asking
| him questions about how actual people in marketing come up with
| emails to come up with a chain of thought that produces
| intentionally uncanny emails for a wide range of inputs:
| https://rentry.co/54hbz
|
| It's not like you can't technically fit what I described into
| _bunch_ of comments (or an obnoxiously long multiline comment),
| but it 'd be bulky and not conducive to general happiness of
| anyone involved.
|
| I much prefer repurposing Nunjucks templates to keep all of that
| a separate document that's easy to manage with version control
| jackmpcollins wrote:
| With magentic you could do chain-of-thought in two or more
| steps: one function that generates a string output containing
| the chain-of-thought reasoning and answer, and a second that
| takes that output and converts it to the final answer object. I
| agree though that this is not encouraged or made obvious by the
| framework.
|
| The approach I'm encouraging with this is to write many
| functions to achieve your goal. So in the case of your email
| writing example you might have some of the following prompt-
| functions - write key bullet points for email about xyz ->
| list[str] - write email based on bullet points -> str -
| generate feedback for email to meet criteria abc -> str -
| update email based on feedback -> str - does email meet all
| criteria abc -> bool And between these you could have regular
| python code check things like blacklist/whitelist of keywords,
| length of paragraphs, and even add hardcoded strings to the
| feedback based on these checks.
| Difwif wrote:
| Looks great! I don't normally like these LLM libraries but this
| one sparks joy. I'll try it out on my next experiment.
|
| Could you highlight how you're parsing to structured objects and
| how it can fail? Ever since I discovered guidance's method of
| pattern guides I've been wanting this more and more (only works
| for local hugging face models though). Wish OpenAI offered a
| similar API.
| jackmpcollins wrote:
| Thanks! Currently magentic just uses OpenAI function-calling;
| it provides it a function schema that matches the structure of
| the output object. So it fails in the same ways as function-
| calling - struggles to match complex schemas, occasionally
| returns empty arrays, ...
| zainhoda wrote:
| Nice! I'm going to try it out and possibly integrate it into my
| Python package: https://vanna.ai
| jstarfish wrote:
| This looks really useful. Langchain is not my idea of a fun time.
|
| Love the examples too. Low-effort humor is the best:
|
| > create_superhero("Garden Man")
|
| > # Superhero(name='Garden Man', age=30, power='Control over
| plants', enemies=['Pollution Man', 'Concrete Woman'])
| cosmonoot wrote:
| Would check out https://www.askmarvin.ai/ if you're into this.
|
| I haven't downloaded 1.5 yet, but they released this last week:
| https://www.askmarvin.ai/prompting/prompt_function/
| brandall10 wrote:
| FWIW, at my last company we had a section in the developer
| guide encouraging using humor in tests - not only did it make
| them more fun to write, but it engaged the readership better.
| bogtog wrote:
| Really like how this is implemented with decorators. Everything
| just feels really smooth
| retrovrv wrote:
| Super cool! Looks quite intuitive, especially for function calls.
| smilingemoji wrote:
| The API looks very clean. Today I learned about "..." in Python
| bluecoconut wrote:
| Pretty cool, I made something similar (lambdaprompt[1]), with the
| same ideal of functions being the best interface for LLMs.
|
| Also, here's some discussion about this style of prompting and
| ways of working with LLMs from a while ago [2].
|
| [1] https://github.com/approximatelabs/lambdaprompt/ [2]
| https://news.ycombinator.com/context?id=34422917
| [deleted]
| cosmonoot wrote:
| Seems a lot like https://github.com/PrefectHQ/marvin?
|
| The prompting you do seems awfully similar to:
|
| https://www.askmarvin.ai/prompting/prompt_function/
| [deleted]
| pphysch wrote:
| This is neat. It makes it easy to prototype, and then you can
| just remove the decorator and write a specific implementation if
| you need to.
| jumploops wrote:
| I built a similar package for Typescript[0], with the goal of
| having type-safe LLM responses automagically.
|
| It's pretty fun, but I've found that having the LLM write code is
| often-times what I actually want most of the time.
|
| [0] https://github.com/jumploops/magic
| visarga wrote:
| Write some tests for those functions. It will be worth it. No, I
| am not kidding, especially for AI we need tests, but we should
| report accuracy instead of a hard fail/pass.
| [deleted]
| jedberg wrote:
| Curious as to why you chose to do it as a decorator instead of
| just a function call?
| dragonwriter wrote:
| Looking at the code, it looks like it is a way to support
| typing; just making it a function with the string template
| would let you return a dynamically-defined function but I think
| would make it harder to get static typing.
| jackmpcollins wrote:
| I found this was the most compact way to represent what I
| wanted to define, and makes it easy to keep the type hints for
| parameters. If you look inside `@prompt` it's creating a
| `PromptFunction` instance which I think would be a similar API
| to what you would end up with without using decorators
| https://github.com/jackmpcollins/magentic/blob/afdb22513385b...
| lachlan_gray wrote:
| This is also similar in spirit to LMQL
|
| https://github.com/eth-sri/lmql
| [deleted]
| js98 wrote:
| Very cool! At first the title reminded me of a project me and my
| colleague are working on called OpenAI-Functools [1], but your
| concept is quite the opposite, combining LLMs in your code rather
| seamlessly instead of the other way around. Quite cool, and
| interesting examples :)
|
| I'll definitely try to apply it in one of my pet projects. Good
| stuff
|
| [1] https://github.com/Jakob-98/openai-functools
| te_chris wrote:
| This is great. I hacked a smaller version of this together when I
| built an LLM app with Elixir. Honestly, the async by default of
| Ex is so much better suited to this stuff, especially as it's
| just api calls.
|
| Tempted to have a go at porting these ideas. Should be v doable
| with the macro system.
| avindroth wrote:
| We need a new language/DSL. Python is a lost cause for strings as
| first-class.
| [deleted]
| conor_f wrote:
| How so? What disadvantages does having strings as a first class
| Type have?
| ElectricalUnion wrote:
| Is it really LLMs (plural) when you only have OpenAPI
| integration?
| dragonwriter wrote:
| text-generation-webui offers an OpenAI API implementation,
| specifically to support OpenAI API clients, so you can get
| something more than just OpenAI support by just wrapping the
| OpenAI API.
|
| You could have more flexibility by abstracting out the
| underlying LLM APIs, but then you also have a bigger deal with
| supported features of different APIs, the same conceptual
| feature supported with very different parameter structures,
| etc., etc.
| msikora wrote:
| OpenAI offers a few different LLMs :)
| jackmpcollins wrote:
| Right now it just works with OpenAI chat models (gpt-3.5-turbo,
| gpt-4) but if there's interest I plan to extend it to have
| several backends. These would probably each be an existing
| library that implements generating structured output like
| https://github.com/outlines-dev/outlines or
| https://github.com/guidance-ai/guidance. If you have ideas how
| this should be done let me know - on a github issue would be
| great to make it visible to others.
| jackmpcollins wrote:
| Oh, and some companies offer APIs that match the OpenAI API
| and there are some open-source projects that do this for
| llama running locally. Since those would be compatible with
| the openai python package they will work with magentic too -
| though some of these do not support function calling.
|
| See for example Anyscale Endpoints
| https://app.endpoints.anyscale.com/landing and
| https://github.com/AmineDiro/cria
| AmazingTurtle wrote:
| I tried out guidance. Encountered endless bugs
| bl00p wrote:
| Are you familiar with https://github.com/PrefectHQ/marvin? This
| looks very similar
| jackmpcollins wrote:
| Yes, similar ideas. Marvin [asks the LLM to mimic the python fu
| nction](https://github.com/PrefectHQ/marvin/blob/f37ad5b15e2e77
| dd998...), whereas in magentic the function signature just
| represents the inputs/outputs to the prompt-template/LLM, so
| the LLM "doesn't know" that it is pretending to be a python
| function - you specify all the prompts.
| fredoliveira wrote:
| (Completely off-topic, but oh how I wish HN supported
| markdown)
| [deleted]
___________________________________________________________________
(page generated 2023-09-26 23:00 UTC)