[HN Gopher] Launch HN: Helicone.ai (YC W23) - Open-source loggin...
___________________________________________________________________
Launch HN: Helicone.ai (YC W23) - Open-source logging for OpenAI
Hi HN - Justin, Scott, and Barak here. We're excited to introduce
Helicone (https://www.helicone.ai) an open-source logging solution
for OpenAi applications. Helicone's one-line integration logs the
prompts, completions, latencies, and costs of your OpenAI requests.
It currently works with GPT, and can be integrated with one line of
code. There's a demo at https://www.helicone.ai/video. Helicone's
core technology is a proxy that routes all your OpenAI requests
through our edge-deployed Cloudflare Workers. These workers are
incredibly reliable and cause no discernible latency impact in
production environments. As a proxy, we offer more than just
observability: we provide caching and prompt formatting, and we'll
soon add user rate limiting and model provider back off to make
sure your app is still up when OpenAI is down. Our web application
then provides insights into key metrics, such as which users are
disproportionately driving costs and what is the token usage broken
down by prompts. You can filter this data based on custom logic and
export it to other destinations. Getting started with Helicone is
quick and easy, regardless of the OpenAI SDK you use. Our proxy-
based solution does not require a third party package--simply
change your request's base URL from https://api.openai.com/v1 to
https://oai.hconeai.com/v1. Helicone can be integrated with
LangChain, LLama Index, and all other OpenAI native libraries.
(https://docs.helicone.ai/quickstart/integrate-in-one-line-of...)
We have exciting new features coming up, one of which is an API to
log user feedback. For instance, if you're developing a tool like
GitHub Copilot, you can log when a user accepted or rejected a
suggestion. Helicone will then aggregate your result quality into
metrics and make finetuning suggestions for when you can save costs
or improve performance. Before launching Helicone, we developed
several projects with GPT-3, including airapbattle.com,
tabletalk.ai, and dreamsubmarine.com. For each project, we used a
beta version of Helicone which gave us instant visibility into user
engagement and result quality issues. As we talked to more builders
and companies, we realized they were spending too much time
building in-house solutions like this and that existing analytics
products were not tailored to inference endpoints like GPT-3.
Helicone is developed under the Common Clause V1.0 w/ Apache 2.0
license so that you can use Helicone within your own
infrastructure. If you do not want to self-host, we provide a
hosted solution with 1k requests free per month to try our product.
If you exceed that we offer a paid subscription as well, and you
can view our pricing at https://www.helicone.ai/pricing. We're
thrilled to introduce Helicone to the HackerNews community and
would love to hear your thoughts, ideas, and experiences related to
LLM logging and analytics. We're eager to engage in meaningful
discussions, so please don't hesitate to share your insights and
feedback with us!
Author : justintorre75
Score : 81 points
Date : 2023-03-23 18:25 UTC (4 hours ago)
| curo wrote:
| Happy Helicone customer here. It's a dead simple setup. It's
| great to have the extra charts and logging to debug issues and
| make sure all is running well.
|
| Congrats to the team!
| justintorre75 wrote:
| Thanks! Please reach out to the team on discord if you have any
| questions
| transitivebs wrote:
| Here's another open source alternative:
| https://github.com/6/openai-caching-proxy-worker
| transitivebs wrote:
| Aaand another even simpler one:
| https://github.com/egoist/openai-proxy
|
| This one uses Reflare under the hood:
| https://github.com/xiaoyang-sde/reflare
| justintorre75 wrote:
| These are great! we looked into using reflare as well,
| however reflare does not support streaming at the moment. We
| are looking to give back to the community and add it for
| them.
|
| I opened this issue up last week https://github.com/xiaoyang-
| sde/reflare/issues/443
| VWWHFSfQ wrote:
| Does this run afoul of OpenAI's terms of of service in any way?
| Using a commercial proxy/broker like this to access their API
| services instead of using directly.
| 3np wrote:
| More direct concern: PII leaking from end-users will be a
| problem if the usage of this service isn't made clear and
| consented with from each individual. I really think the
| responsible thing by providers like OP is to remind their
| customers about this.
|
| If you have end-users in for example EU then integrating a
| service like Helicone would usually require new informed
| consent to be legal.
| ttul wrote:
| I think you could dispense with this issue by just adding
| Helicone to your list of subprocessors in your GDPR
| disclosure.
| justintorre75 wrote:
| We see a lot of applications on the market essentially
| "proxying" traffic to OpenAI, moving to a non-proxy
| implementation is not too hard if they have any issues with us
| doing this.
| VWWHFSfQ wrote:
| I mean from your users' perspective, not yours. Since it's
| their OpenAI access that will get revoked for violating the
| TOS.
| Hansenq wrote:
| Congrats! We've been happy users of Helicone for the past few
| months--it literally helped us solve a bug with OpenAI's API
| where we didn't know why requests were failing and we failed to
| log some of their responses. Helicone helped us debug that it was
| a token limit issue really quickly, especially since the logging
| around hasn't been great.
|
| Love how easy it was to integrate too--just one line to swap out
| the OpenAI API with theirs.
| justintorre75 wrote:
| Thanks! Glad Helicone was able to help your workflow.
| nico wrote:
| Awesome.
|
| Is there a consumer version of this?
|
| Like an alternative ChatGPT client or chrome extension that will
| save my prompts/conversations, tell me which ones I liked more
| and let me search through them?
| zekone wrote:
| onboarded and started using it in literally 2 minutes. nice
| ayoKho wrote:
| we're glad you were able to get onboarded so quickly! we're
| always happy to incorporate any feedback you may have on the
| onboarding experience
| yawnxyz wrote:
| Wow so cool! Does this act kind of like a logger and would we be
| able to have access to the logs later on, or should we bring our
| own logger as well?
|
| (Also just curious, are you guys just using D1 or KV under the
| hood?)
| justintorre75 wrote:
| Thanks! :D No need to bring your own logger, just change one
| line of code and then all your logs are in Helicone.
|
| We are using CF Worker cache under the hood
| https://developers.cloudflare.com/workers/runtime-apis/cache...
|
| We might use D1 for some of our other features like rate
| limiting.
| StablePunFusion wrote:
| The entire company is based around the idea of providing metrics
| for one closed-source platform's API? Or is the "OpenAI
| application observability" just one part of what the company
| does? Otherwise it seems like taking the "all eggs in one basket"
| to the extreme.
| gregschlom wrote:
| "Solve one pain point for OpenAI's users, get some nice
| traction, get acquired by OpenAI in a couple of months" is
| probably the plan here. Not a bad one to turn a quick profit.
| quadcore wrote:
| _" all eggs in one basket" to the extreme._
|
| You couldn't describe better what a startup must do.
| throwaway280382 wrote:
| I guess they are betting that openAI will be such a big
| platform that many companies will need this capability. Not a
| bad idea I guess
| snacktaster wrote:
| It seems pretty brittle to me also. And their solution is to
| actually _proxy_ OAI's API wholesale which doesn't seem like a
| very good idea and might not even be permitted.
| justintorre75 wrote:
| OpenAI is just the start to capture most of the market and
| iterate on some ideas first :). We are adding multiple
| providers soon. ~Stay tuned~
| meghan_rain wrote:
| What providers apart from OpenAI exist?
| justintorre75 wrote:
| Here are a few...
|
| https://www.ai21.com/ https://www.anthropic.com/
| https://cohere.ai/ https://huggingface.co/bigscience/bloom
| https://goose.ai/ https://bard.google.com/
| https://ai.facebook.com/blog/large-language-model-llama-
| meta...
| version_five wrote:
| To be fair, there are companies based on providing metrics for
| AWS, e.g. Vantage, so it's not surprising to see VCs making
| bets in this area
| VWWHFSfQ wrote:
| AWS provides services and APIs to support that specific
| usage. This service is MITM'ing another company's API. I'm
| pretty sure AWS wouldn't tolerate that sort of thing.
| version_five wrote:
| One could think about it as consulting, but productized, on
| top of a service. Something like AWS is mind numbingly
| complicated. Amazon has solutions engineers but there's
| also a big 3rd party market. And some part of that market
| doesn't need fully bespoke solutions but can be well served
| by a SaaS that sits on top (of course entailing risk, but
| it's all risky).
|
| OpenAI isn't there yet, personally I'm bearish on
| generative AI and openAIs dominance, but they're the major
| player, and there's a version of the future where they
| dominate the landscape with a complex enterprise offering
| like AWS that leaves room for intermediaries. So it's not a
| bad idea to include bets on people who are building these
| intermediary products as part of a vc portfolio.
| 0xDEF wrote:
| AWS is a whole other beast than the few dozen REST and SSE
| endpoints at OpenAI.
| gmaster1440 wrote:
| It's a big basket.
| ianbicking wrote:
| I have _specifically_ thought of writing something just like
| this, so it's awesome to see it!
|
| One thing I would really like to be storing with my requests is
| the template and parameters that created the concrete prompt.
| (This gets a little confusing with ChatGPT APIs, since the prompt
| is a sequence of messages.) Custom Properties allow a little
| metadata, but not a big blob like a template. I see there's a way
| to have Helicone do the template substitution, but I don't want
| that, I have very particular templating desires. But I _do_ want
| to be able to understand how the prompt was constructed. There's
| probably some danger on the client side that I would send data
| that did not inform the prompt construction, and balloon storage
| or cause privacy issues, so there is some danger to this feature.
|
| Backoffs and other rate limiting sounds great. It would be great
| to put in a maximum cost for a user or app and then have the
| proxy block the user once that was reached as a kind of firewall
| of overuse.
|
| Note your homepage doesn't have a <title> tag.
| justintorre75 wrote:
| This is so great! Thanks for the feedback. This is something we
| have thought a lot about, and really want to make sure we are
| adding the "correct" features that don't disrupt the user
| experience.
|
| I kicked off a thread here
| https://github.com/Helicone/helicone/discussions/164 and would
| love your input!
|
| P.S. that for the <title> note
| ianbicking wrote:
| I'll note the jump in cost from Free to Starter will keep me
| from using this for my own projects, 1000 requests just isn't
| that much, and if I'm doing something more serious then 50,000
| requests also doesn't seem like a lot.
| justintorre75 wrote:
| stay tuned for updated pricing
| olliepop wrote:
| The reality is that virtually all of the public AI tech available
| right now is from OpenAI. Over time as more models are
| commercialised and generally available, it's probable that
| Helicone will serve them too.
|
| Congrats Justin and team! Excited for you.
| justintorre75 wrote:
| Couldn't agree more. Thanks! :)
| killthebuddha wrote:
| LLM infrastructure is _the spot to be right now_ for startups.
| antonok wrote:
| congrats! Helicone provides one of the biggest missing pieces
| from the the AI tool dev experience today, thanks for building
| this and sharing it with the rest of us!
| otterley wrote:
| What is the market for this solution? Whose pain point(s) are you
| solving? And what stops OpenAI from "Sherlocking"* you, i.e.,
| making whatever you're building a free included feature and
| extinguishing the market?
|
| *Or, to use a more modern analogy, "Evernoting"
| justintorre75 wrote:
| Great question! The short answer is nothing! We just have to
| out pace them and make sure we develop a better product
| experience.
|
| Two key advantages that we have over OpenAI. 1. We are open
| source and are trying to build a community of developers that
| can out run any single LLM provider. 2. We are completely
| provider agnostic, which allows us to aggregate and share
| common features across tools. (One analogy we like to use, is
| we want to emulate how dbt is database agonistic, and Helicone
| will grow to be provider agnostic)
| catsarebetter wrote:
| Great answer
| samstave wrote:
| This is great, however, I am concerned that the various AI
| 'ecosystem' of all the bolt-on, add-on, plug-ins etc... will be
| like a billion services all looking for payment - and any complex
| startup that needs a bunch of these services to build their own
| service/product/platform, it will be like the current state of
| streaming services.
|
| So, youll be trying to manage a ton of SLAs, contracts, payment
| requirements, limits on service access that may be out of your
| budget to pay for all the various services, API calls, etc.
|
| This is going to be an interesting cluster....
|
| So we need a company thats a single service to access all
| available AI connects and the multiple billing channels.
|
| However, then you have that as a single POF
___________________________________________________________________
(page generated 2023-03-23 23:00 UTC)