[HN Gopher] Show HN: I built an OSS alternative to Azure OpenAI ...
___________________________________________________________________
Show HN: I built an OSS alternative to Azure OpenAI services
Hey HN, I am proud to show you guys that I have built an open
source alternative to Azure OpenAI services. Azure OpenAI services
was born out of companies needing enhanced security and access
control for using different GPT models. I want to build an OSS
version of Azure OpenAI services that people could self host in
their own infrastructure. "How can I track LLM spend per API key?"
"Can I create a development OpenAI API key with limited access for
Bob?" "Can I see my LLM spend breakdown by models and endpoints?"
"Can I create 100 OpenAI API keys that my students could use in a
classroom setting?" These are questions that BricksLLM helps you
answer. BricksLLM is an API gateway that let you create API keys
with rate limit, cost control and ttl that could be used to access
all OpenAI and Anthropic endpoints with out of box analytics. When
I first started building with OpenAI APIs, I was constantly worried
about API keys being comprised since vanilla OpenAI API keys would
grant you unlimited access to all of their models. There are
stories of people losing thousands of dollars and the existence of
a black market for stolen OpenAI API keys. This is why I started
building a proxy for ourselves that allows for the creation of API
keys with rate limits and cost controls. I built BricksLLM in Go
since that was the language I used to build performative ads
exchanges that scaled to thousands of requests per second at my
previous job. A lot of developer tools in LLM ops are built with
Python which I believe might be suboptimal in terms of performance
and compute resource efficiency. One of the challenges building
this platform is to get accurate token counts for different OpenAI
and Anthropic models. LLM providers are not exactly transparent
with the way how they count prompt and completion tokens. In
addition to user input, OpenAI and Anthropic pad prompt inputs with
additional instructions or phrases that contribute to the final
token counts. For example, Anthropic's actual completion token
consumption is consistently 4 more than the token count of the
completion output. The latency of the gateway hovers around 50ms.
Half of the latency comes from the tokenizer. If I start utilizing
Go routines, might be able to lower the latency of the gateway to
30ms. BricksLLM is not an observability platform, but we do
provide integration with Datadog so you can get more insights
regarding what is going on inside the proxy. Compared to other
tools in the LLMOps space, I believe that BricksLLM has the most
comprehensive features when it comes to access control. Let me
know what you guys think.
Author : luyuanxin1995
Score : 83 points
Date : 2023-12-11 18:56 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| gigel82 wrote:
| No you didn't.
|
| The whole point of using Azure OpenAI over plain OpenAI is the
| fact that your data doesn't get donated to OpenAI for training
| (this solves "compliance" for enterprise customers that need it),
| which you're not solving (because you obviously can't run the
| OpenAI models in your own data center).
| Jayakumark wrote:
| +1
| j45 wrote:
| The Azure OpenAI API to me was more about reliability and
| minimizing downtime.
|
| The OpenAI API also allows opting out of training the OpenAI
| for training. If you are referring to something else with data,
| happy to learn.
|
| https://help.openai.com/en/articles/5722486-how-your-data-is...
|
| "API
|
| OpenAI does not use data submitted to and generated by our API
| to train OpenAI models or improve OpenAI's service offering. In
| order to support the continuous improvement of our models, you
| can fill out this form to opt-in to share your data with us. "
| droidno9 wrote:
| OpenAI also does not use API data to train its models.
|
| Source: https://openai.com/enterprise-privacy
| danenania wrote:
| According to OpenAI, they don't use data sent to the API for
| training and delete it after 30 days. So your concern is
| misplaced unless you think OpenAI is outright lying about their
| policies.
| happytiger wrote:
| Whoa there. That's only half the story.
|
| https://techcrunch.com/2023/03/01/addressing-criticism-
| opena...
|
| They said _only if they opt in_ not that they wouldn't do it.
|
| And they used to and could change those terms again after the
| market matures and developers have deeply integrated
| application investments.
|
| Plenty of companies have followed this strategy in the past.
|
| Let's not oversimplify the situation as it was a recent
| change as well...
| kristjansson wrote:
| That's only for ChatGPT. The APIs are governed by [0]
|
| > We will only use Customer Content as necessary to provide
| you with the Services, comply with applicable law, and
| enforce OpenAI Policies. We will not use Customer Content
| to develop or improve the Services.
|
| [0]: https://openai.com/policies/business-terms
| danenania wrote:
| Microsoft can also change their terms so I don't see much
| difference there.
|
| I can understand preferring the Azure endpoints, but let's
| be accurate about what the policies are.
| linkenaxx wrote:
| +1 i dont think Azure provides that much more in terms of
| data privacy unless youre saying we should believe
| Azure's policies but not OpenAI's...
| ajcp wrote:
| Many here think that I'm afraid of my companies data being used
| to train models, but that is only half-correct.
|
| The value proposition of Azure OpenAI to the enterprise is that
| it's bound by the Enterprise Agreement for not just data-use,
| but data access writ-large. I don't want my data to be
| accessible for *any reason* by a vendor that isn't covered
| explicitly in the EA.
|
| While OpenAI says they delete your data after 30-days I have no
| contract or agreement in place that ensures that, or that
| within those 30-days they don't have carte blanche to my data
| for whatever else they may want to do with it.
|
| The Azure OpenAI agreement *explicitly* states they only store
| my data for 30-days and can *only* access it if they suspect
| abuse of the service according to the ToC and our EA, which is
| no different than for other Azure services on my tenant. IIRC
| that's only after notifying me as well, and cannot exfiltrate
| that data even if they access it.
| gigel82 wrote:
| I suspect a lot of folks never worked with Enterprise / Gov
| customers and don't understand the restrictions and
| compliance requirements (like data residence, access control,
| FedRAMP, TISAX, reliability SLAs, etc. which you get with
| Azure but not with some "move fast and break things" startup
| like OpenAI).
|
| My comment is not dissing on the author, I'm just pointing
| out what most folks get from Azure is compliance (and maybe
| safety), and an OSS cannot solve that (unless they're running
| some other models in owned infrastructure or on-prem).
| quickthrower2 wrote:
| Even just doing SOC2 we prefer it
|
| Let alone if you are a SaaS your customers may demand it.
| robertlagrant wrote:
| > most folks get from Azure is compliance
|
| To be fair, this doesn't prevent problems. Paperwork
| doesn't plug any security gaps in any cloud provider.
| toomuchtodo wrote:
| It potentially plugs the contractual and liability risks,
| which might be more important (talk to your legal and
| compliance folks). None of your data is going to launch
| nuclear missiles, if it leaks it would be unfortunate,
| but not as much as the litigation and regulatory costs
| you could potentially incur.
|
| Everyone gets popped eventually. It's your job to show
| you operated from a commercially reasonable security
| posture (and potentially your third party dependency
| graph, depending on regulatory and cyber insurance
| requirements).
|
| (i report to a CISO, and we report to a board, thoughts
| and opinions are my own)
| ralph84 wrote:
| Compliance isn't about preventing problems. It's about
| identifying risks and determining who is responsible for
| mitigating those risks and who is on the hook for damages
| if the risks aren't sufficiently mitigated.
| ajcp wrote:
| Yup, and I agree with you.
|
| Even beyond enterprise thought people seem to think that
| the only thing their data is good for as far as OpenAI is
| concerned is for training-datasets, but that's just not the
| case.
|
| The authors of this have built something great, but it
| doesn't protect you from any of those non-training use-
| cases.
| 101008 wrote:
| How do you calculate usage limits if you don't tokenize inputs
| and outputs?
| luyuanxin1995 wrote:
| we do tokenize inputs and outputs. sorry if that is not clear
| in the post
| js4ever wrote:
| Congradulations (from the readme) :)
| luyuanxin1995 wrote:
| fixed. embarrassing on my part
| hluska wrote:
| You put in a lot of effort and built something cool - a
| spelling error is absolutely nothing to be embarrassed about.
| kristjansson wrote:
| This looks like a useful value-add, but I'd hesitate to call it a
| replacement for Azure OpenAI. Governance and observability
| features of Azure OpenAI are really secondary to the stability
| and reliability guarantees of Azure owning and operating the
| model instead of OpenAI...
| 0xDEF wrote:
| Azure OpenAI services also makes it possible to choose data
| centers outside the US.
| mdeeks wrote:
| Looks interesting! We're in the middle of building something
| similar right now for ourselves. We may look at this as an
| alternative for ourselves.
|
| By the way, I saw this after a quick glance poking through the
| code. This isn't encryption, it is hashing. Not sure where or how
| it is used but it is worth a rename at least:
| https://github.com/bricks-cloud/BricksLLM/blob/main/internal...
| luyuanxin1995 wrote:
| you are right. will update the language
| HellsMaddy wrote:
| This reminds me of an idea I had for an OpenAI proxy that
| transparently handles batching of requests. The use case is that
| OpenAI has rate limits not only on tokens but also requests per
| minute. By batching multiple requests together you can avoid
| hitting the requests limit.
|
| This isn't really feasible to implement if your app runs on
| lambda or edge functions, you'd need a persistent server.
|
| Here's a diagram I drew of a simple approach that came to mind:
| https://gist.github.com/b0o/a73af0c1b63fccf3669fa4b00ac4be52
|
| It would be awesome to see this functionality built into
| BricksLLM.
| computerex wrote:
| OpenAI API doesn't support batching afaik.
| HellsMaddy wrote:
| They do: https://platform.openai.com/docs/guides/rate-
| limits/batching...
| treprinum wrote:
| Embeddings can be batched.
| swyx wrote:
| how exactly are you intending to batch different prompts
| together in the openai api? its not like they accept an array
| of parallel inputs
| heyn05tradamu5 wrote:
| They've recently added this functionality to AWS Bedrock
| thankfully. Doesn't support OpenAI models, but does support
| Anthropic.
|
| https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-be...
| linkenaxx wrote:
| great idea for simple use cases, but not sure if it can load
| handle as well as Azure i like that its open sourced too since
| most solutions out there are black box and we can only rely on
| the "good will" of the company that they will do as they say
___________________________________________________________________
(page generated 2023-12-11 23:00 UTC)