[HN Gopher] OpenSearch: AWS fork of Elasticsearch and Kibana
___________________________________________________________________
OpenSearch: AWS fork of Elasticsearch and Kibana
Author : ke4qqq
Score : 477 points
Date : 2021-04-12 15:54 UTC (7 hours ago)
(HTM) web link (aws.amazon.com)
(TXT) w3m dump (aws.amazon.com)
| ignoramous wrote:
| OpenSearch was once was an initiative founded at A9, Amazon
| subsidiary, to create a personalized, cross-service, search
| engine: https://archive.is/PCKWq
|
| OpenSearch is from an era when Amazon and Google were covertly
| competitive. Google didn't get anywhere with Froogle and
| AppEngine; whilst Alexa and A9 didn't move any mountains.
|
| Code: https://github.com/dewitt/opensearch
| galaxyLogic wrote:
| I think this thread is much about shared source licenses like
| SSPL vs. "orthodox" open source licenses like GPL.
|
| Based on the link below it seems to me the difference is that
| SSPL etc. have a clause which prevents me from making money by
| selling the use of the licensed software over the network for
| instance.
|
| GPL puts some rather strict rules on users of copyleft software,
| mainly that you MUST distribute your modifications with the same
| license.
|
| What I don't quite get is why adding a rule that says "if you
| make this software usable over the network you must make it
| usable for free" would be considered categorically less ethical
| than GPL.
|
| GPL says you must give out your modifications for free. SSPL says
| you must also give out the rights to use that software for free
| as well.
|
| Isn't SSPL more ethical in the sense that it requires you to give
| out more for free?
|
| https://techcrunch.com/2018/11/29/the-crusade-against-open-s...
| znpy wrote:
| RIP Elastic
| jimmy2020 wrote:
| Is there any good that done by Amazon to support OSS? Like ever.
| They started with cloning MongoDB and now Elastic with actually
| zero contribution to the community regardless of their insane
| profits. This is a clear single. Amazon can always clone and
| redistribute any open source software then lock it in for AWS. If
| we've started to witness declining in OSS, well at least we know
| now who started the wave.
| benmller313 wrote:
| This seems like a much more positive response than the one they
| took with MongoDB. I agree, Amazon hasn't done much, but maybe
| this could be a start?
| jimmy2020 wrote:
| The start maybe done by open source some technology they use
| that can profit and help other startups. Maybe open source
| their own version of "React". That will be a good start.
| oscargrouch wrote:
| Maybe a dual-company like Mozilla would at least make it more
| clear that the big guy that are just taking advantage of the
| free lunch is doing more harm than good?
|
| Elastic should have a for-profit and a non-profit company
| taking donations that would actually control the open source
| code and hiring the core part of team working on it.
|
| I mean, we know how badly Amazon is behaving here, but at least
| they should have a option where they could realistically
| invest.
|
| Asking for a company to invest in a competitor that can grow
| and eat their lunch with the money being invested by them is
| not realistic. Even because the company investing the money
| would want to know if that money is actually being invested
| back in the open source software and not used by a competing
| company.
|
| If, giving this choice, they didn't invest back in the
| foundation, it would be much more clear that they are doing it
| in bad faith.
| unethical_ban wrote:
| You're asserting that they have forked OSS and then not
| provided back the source code for their own improvements.
|
| I guess we can check their github repos to see if that's the
| case.
| jimmy2020 wrote:
| As far as I know DocumentDB is closed source. Btw: I can name
| zero OSS projects from Amazon and this is not the same for
| Microsoft (VScode, TS) and FB (React, Jest).
| yandie wrote:
| Firecracker is a famous one that I can think of.
| invertedreversi wrote:
| Fork of Google crosvm
| jimmy2020 wrote:
| there's always a fork.
| Twirrim wrote:
| Firecracker, s2n, chalice, jsii.
|
| On top of that AWS has done a lot of contributions
| upstream, particularly around the linux kernel, https://git
| .kernel.org/pub/scm/linux/kernel/git/torvalds/lin...,
| (particularly the virtualisation stacks involved), some
| around OpenJDK at what looks like an increasing pace, and
| the like.
|
| To be clear: not an AWS employee.
| DetroitThrow wrote:
| I know several employees contributing to the Linux kernel,
| and I know they're hiring more Rust contributors. You're
| right though, I think Amazon lags compared to MSFT, FB, and
| G. Even Netflix or IBM/RedHat, for that matter.
| hintymad wrote:
| Do people assume that a company as large as AWS would
| automatically have a lot to contribute to the OSS? Maybe most
| of Amazon teams have not much to open source yet. Contributing
| to OSS is a bottom-up effort. An engineer needs to be motivated
| to generalize her project, to peel the code from Amazon's vast
| internal infrastructure, and to go through an approval process
| to open source her project. Given that many teams have razor-
| sharp focus on delivering features, for good or for bad, I was
| wondering how many engineers are really motivated enough to
| open source something internal.
| that_guy_iain wrote:
| I don't think it's a bottom up process. I feel often it's a
| top down process. Most companies with lots open source
| activity normally have management that have decided that is
| something they want to encourage and then it comes down to
| people making their code open sourcable.
| jimmy2020 wrote:
| It is both a top down process and the culture of the
| company itself. In the real world, we know how big tech
| open-sourced some of the most efficient technologies that
| empowered hundreds, if not millions, of startups.
| dariusj18 wrote:
| https://aws.amazon.com/opensource/
| jimmy2020 wrote:
| Yes exactly, you have actually to google it maybe you'll be
| able to find a useful link that makes Amazon looks good.
| mrnaught wrote:
| Why is this comment downvoted?
| dariusj18 wrote:
| I'm not sure what you mean.
| marricks wrote:
| Amazon will of course have a PR statement about how it
| cares about open source. It's not like Apple is known for
| it's commitment to open source but it has a similar page:
| https://developer.apple.com/opensource/
| rafaelturk wrote:
| TLDR: It's all Apache 2.0.
| williesleg wrote:
| Oh this will not end well.
| phd514 wrote:
| From the announcement: "You should consider the initial code to
| be at an alpha stage -- it is not complete, not thoroughly
| tested, and not suitable for production use. We are planning to
| release a beta in the next few weeks, and expect it to stabilize
| and be ready for production by early summer (mid-2021)."
|
| Given that Amazon announced the fork in January and they don't
| expect it to be production-ready until summer, I'm guessing
| they've underestimated the amount of work required to package and
| distribute a product as complex as Elasticsearch. Given that, I
| doubt they will be well-equipped to keep pace with new feature
| development.
| acdha wrote:
| I would question the assumption that this is "not suitable for
| production use" means "everything is broken and we're way
| behind" rather than, say, "we are being extremely conservative
| because our customers will expect support as soon as we say
| it's production ready and we need to test every upgrade
| scenario for our large number of existing customers". The AWS-
| managed ElasticSearch seems to be pretty popular and I would
| expect them to be as conservative about new offerings as they
| are with, say, RDS.
| z77dj3kl wrote:
| The fork announcement was announced as a response to the
| Elastic stuff. I don't think they made any predictions about
| when it'd be ready in that blog post, so I'm not sure why they
| would've underestimated anything?
| lornajane wrote:
| I'm not sure I agree with that assessment. Now that the fork is
| publicly available, others can contribute to get it ready,
| which wasn't possible until now.
| phd514 wrote:
| Yes, others can contribute, but significant feature
| development on large-scale OSS projects tends to be driven by
| developers paid to work on the project full-time and
| coordinated by an organized steering committee with clear
| governance (or company if the product is owned by a single
| company). I don't see any of that in place for OpenSearch and
| getting that all started up is not at all a trivial endeavor.
| busterarm wrote:
| Given how poorly of a job Elastic themselves did with keeping
| the full ecosystem of tools working in lockstep for YEARS, I'm
| sure Amazon will do fine.
|
| I remember all through Elasticsearch 5 where none of their
| packaged Kibana dashboards flippin' worked.
| colechristensen wrote:
| I would assume they're doing more than just packaging.
|
| A beginning is the time for taking the most delicate care that
| the balances are correct
| fastball wrote:
| 6 months from start to prod is... not bad at all? You must be a
| wizard programmer if that is your typical turnaround time.
|
| I don't remember AWS saying something like "it will be ready in
| weeks" in Jan...
| [deleted]
| retzkek wrote:
| > they've underestimated the amount of work required to package
| and distribute a product as complex as Elasticsearch
|
| The bulk of the work thus far has been to strip out the non-OSS
| components ("X-pack") and the many references to it, nothing to
| do with packaging, distributing, or even maintaining and
| developing features.
|
| https://discuss.opendistrocommunity.dev/t/preparing-opensear...
| zo1 wrote:
| I for one will be happy when those are taken out. So many
| headaches trying to get bloated Kibana to start as a docker
| container before realizing that some random x-pack-disable
| flag needs to be set for it to start without a random error.
| pmarreck wrote:
| What is the sell for ES over something like the fulltext search
| built into Postgres, considering that the cost of adding another
| dependency is not insignificant?
| Pirate-of-SV wrote:
| It's nice that they announce it and that there's some sort of
| future effort promised. From my perspective we might not upgrade
| the elastic-stack (with current Elastic projects) too far to not
| bacome accidentally incompatible in case we want to make a
| switch.
| [deleted]
| fakedang wrote:
| On a tangential note, how is Meilisearch compared to
| Elasticsearch?
| binarymax wrote:
| Meilisearch is a new niche engine for instant-search
| experiences, and is far less flexible and less mature than
| Elasticsearch
| parhamn wrote:
| > and we don't ask for a contributor license agreement (CLA)
|
| Makes me wonder what these are for (copyright transfer) and why
| they decided it's not needed. It also makes me wonder if this
| sort of thing has ever been taken/tested in court or if it's
| paranoid friction with little value add.
| iib wrote:
| Eric S. Raymond is against them, but also argues that they are
| harmful--as opposed to just useless--because if they ever got
| to court, a jurist would look at the practices of the community
| to decide whether such a thing is common enough that they
| should be required. [1]
|
| I know GNU does it (at least for Emacs) under the reason that
| the FSF can go after any GPL violation only if it is the clear
| copyright holder, but no such case exists, to my knowledge.
|
| [1] http://esr.ibiblio.org/?p=8287
| resoluteteeth wrote:
| > Makes me wonder what these are for (copyright transfer) and
| why they decided it's not needed. It also makes me wonder if
| this sort of thing has ever been taken/tested in court or if
| it's paranoid friction with little value add.
|
| Some companies/projects might use them purely to avoid possible
| future legal headaches (I think GNU does this), and I'm not
| sure to what degree that has actually been tested, but they can
| also allow re-licensing under a different license which is more
| clear cut and I think that's more the issue here
|
| Amazon is trying to say that they'll never relicense the code,
| so they have no need to take ownership over contributions.
| Boulth6 wrote:
| Indeed that is likely but I wonder why didn't they at least
| require a Developer's Cerificate of Origin [0] that
| kernel.org uses. This is really lightweight (just append one
| line to git commit message) and supposedly provides a minimum
| legal base for the change. IANAL.
|
| [0]: https://blog.chef.io/introducing-developer-certificate-
| of-or...
| conroy wrote:
| I just opened a pull request to fix a few typos in the
| README. They are requiring a Developer's Certificate of
| Origin.
| bombcar wrote:
| A copyright transfer would easily smell to people "Amazon is
| going to change the license at some point in the future to duck
| us over Elasticsearch-style". They're trying to avoid that
| smell.
| denysvitali wrote:
| Okay... and what's the difference here from Open Distro for
| ElasticSearch? I guess it's just a rebranding, isn't it?
| bbest123 wrote:
| Open Distro for ElasticSearch was not a fork rather an Apache
| 2.0-licensed distribution of Elasticsearch enhanced with
| enterprise security, alerting, SQL etc... OpenSearch is a
| community-driven, open source search and analytics suite
| derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana
| 7.10.2.
| hello_moto wrote:
| OpenDistro for ES is the surrounding tools for ES => plugins,
| index-state-management, basically a suite clone of ES
| Enterprise offerings (X-pack) because AWS can't ship AWS ES
| with X-Pack.
|
| OpenSearch is the ES (core) itself.
|
| If I'm not mistaken.
| bmcahren wrote:
| I feel Amazon took the feedback from the DocumentDB/MongoDB
| fiasco to heart and made positive change in their approach.
|
| DocumentDB is a closed source proprietary database created by
| Amazon to emulate the MongoDB API. Think Google's Dalvik runtime
| vs Sun/Oracle's JVM.
|
| This time around we have an open source fork of ES with big
| backers all contributing and very permissive licensing.
|
| In both cases, Amazon gets to implement AWS-specific upgrades to
| management to depend heavily on EBS replication rather than
| application-layer replication. Would it be nice to have that
| secret sauce that makes Aurora/DocumentDB so nice to use compared
| to self-hosting or RDS? Of course. Do we have to have it to
| consider using or contributing to the open source software? No.
| busterarm wrote:
| On the other hand, MongoDB is already sort of obsolete and
| trending towards death by the time that all ended up happening
| while ElasticSearch is hot and "new".
| rabbidruster wrote:
| Where do you get the impression MongoDB is trending towards
| death? Seems to be growing by some metrics; the stock price
| has more than doubled in the last year. Not a fan myself, but
| still seems a long way from death to me and seem to be doing
| something right in enterprise market.
| xvilka wrote:
| They also could sponsor Elasticsearch alternatives in Rust -
| Sonic[1] and Toshi[2]. Even more, integration[3] with Vector.
|
| [1] https://github.com/valeriansaliou/sonic
|
| [2] https://github.com/toshi-search/Toshi
|
| [3] https://github.com/timberio/vector/issues/988
| andreygrehov wrote:
| Does that mean ElasticSearch documentation won't be relevant
| soon?
| bombcar wrote:
| ElasticSearch will need to relatively quickly come out with a
| feature the OpenSearch doesn't replicate or people will just
| use the minimum that both support (see MySQL vs MariaDB).
| shankspeaks wrote:
| Imagine if they went after Mongo next?
|
| Atlas is a virtual monopoly for Mongo solely due to SSPL, and it
| has created a ridiculously overpriced ecosystem for hosted and
| managed services, and tooling around it.
|
| Parking the technical merits to one side, considering the sheer
| number of devs and early-stage products that are built on Mongo,
| I'd love for someone to go after them next.
| calmoo wrote:
| Amazon already have DocumentDB which clones the Mongo API. I
| don't think its forked though, they just use a barely mongo
| compatible wrapper around their own db engine.
| vio2 wrote:
| It's not nearly as compatible as you might think.
| Interestingly enough, MongoDB's CTO managed RDS at AWS.
| bgorman wrote:
| This already exists
|
| https://aws.amazon.com/documentdb/
| Graphguy wrote:
| Kinda blocked on the compatibility front after the 4.0 API
| though, eh?
| bmcahren wrote:
| Not if the recent Oracle vs Google supreme court ruling is
| to be acknowledged.
|
| DocumentDB is to MongoDB 4.0+ as Dalvik runtime was to JVM.
|
| Here is their 4.0 compatibility update: https://docs.aws.am
| azon.com/documentdb/latest/developerguide...
| Graphguy wrote:
| Time will tell...MongoDB 4.2 APIs are under SSPL
|
| "Amazon DocumentDB implements the Apache 2.0 open source
| MongoDB 3.6 and 4.0 APIs by emulating the responses that
| a MongoDB client expects from a MongoDB server, allowing
| you to use your existing MongoDB drivers and tools with
| Amazon DocumentDB."
|
| https://aws.amazon.com/documentdb/
|
| IANAL
| sdesol wrote:
| Will be interesting to see the resources that AWS will throw at
| this. You can get a sense of the resource that elastic.co is
| throwing at elasticsearch at
|
| https://public-001.gitsense.com/insights/github/repos?r=gith...
|
| I'm currently indexing the fork, so in about an hour or two, I'll
| provide the insights for the fork as well.
| sdesol wrote:
| I can't edit the comment anymore, but you can find the fork at
| https://public-001.gitsense.com/insights/github/repos?r=gith...
| binarymax wrote:
| Elastic spends so much time and effort on making sure that
| their search is performant (and they are not shy about
| deprecating and removing features that are slow). I think this
| is where Elastic will continue to shine. It's one thing to add
| features, it's another to make it so they work well and make
| sure the integrating product team doesn't shoot themselves in
| the foot.
| DetroitThrow wrote:
| I agree, I trust there will be value for some in openSearch
| not using SSPL and value for others in Elastic's
| performant/scaleable tendencies.
| sdesol wrote:
| What may hurt them though, is the number of customers that
| currently feel things are currently "good enough". I don't
| know what their sales engagement looks like, so I'm not sure
| if this will really hurt them or not.
| binarymax wrote:
| Perhaps! My money is on Elastic including an approximate
| nearest neighbor search in 8.0 which uses the new HNSW
| feature in Lucene 9, which is going to be hard to do in a
| distributed capacity and will be a significant feature if
| they can pull it off.
|
| PS - I've never seen gitsense before and it's really cool.
| I especially like the focus quadrant!
| sdesol wrote:
| Since search is your domain expertise, I'll take your
| word for it :-)
|
| As for GitSense, checkout the impact section and sort
| contributors by "First Commit". With this, you'll get a
| very good idea of the developer's expertise level.
| jillesvangurp wrote:
| I scrolled back through the commits. It looks like they've been
| removing traces of x-pack, Elastic branding, licensing checks,
| etc. since the beginning of March. So far it looks like one
| person is doing the bulk of all that work.
|
| If there are new features, I haven't seen any. The real
| question is do they have a team for new feature work that they
| are putting together or is this just a fork that is doomed to
| fall behind as Elastic's huge team continues to develop their
| code base fixing bugs that will never get fixed on the Amazon
| fork, adding features that will never get fixed, eventually
| releasing the 8.0 release that has been in the works for two
| years, etc.
|
| I don't see any evidence that they have that team so far.
| They're paying a few people to go through the moves of forking
| but I don't really see a grand vision beyond that so far.
| sdesol wrote:
| I really need to add a compare feature to my tool as it would
| make analysis a lot easier. Having said that, there is no
| denying there is a huge difference in work being done in both
| projects over the past 30 days.
|
| Amazon does have 16 open pull requests though, with about 7
| having 20 or more file changes, but I didn't dig into them to
| understand their significance. Maybe it's another feature
| I'll need to work on.
|
| If you look at the one year window for elasticsearch
|
| https://public-001.gitsense.com/insights/github/repos?q=wind.
| ..
|
| it's churn and activity has been extremely consistent and I'm
| not sure if this is an investment Amazon can and/or is
| willing to make.
|
| However, knowing enterprise, I'm not sure if this will make a
| huge difference as those making the decisions might not
| really care and they'll just accept whatever Amazon tells
| them.
| shawnz wrote:
| They have done extensive work in the Open Distro modules
| which I assume they will carry over. See:
| https://github.com/opendistro-for-elasticsearch/
| chatman wrote:
| If this project were to be governed by Apache Software
| Foundation, I'd have associated more credibility to this effort.
| bambam24 wrote:
| This is wonderful
| thayne wrote:
| I'm glad they changed the name. Although it's probably too much
| to ask that the AWS service be renamed to OpenSearch Service as
| well.
| ElijahLynn wrote:
| > We plan to rename our existing Amazon Elasticsearch Service
| to Amazon OpenSearch Service.
| thayne wrote:
| Oh, I missed that :)
| mgr86 wrote:
| Naive aside, but why would I want to use ElasticSearch or
| OpenSearch over Solr? Are ES and Solr not both based on Lucene?
| _tom_ wrote:
| Elasticsearch has done a good job of focusing on the log
| analytics space, and Kibana is a great tool.
|
| I'd probably use ES for any log analytics and Solr for things
| like website and ecommerce search.
|
| Of course, you could do both with either.
| seabrookmx wrote:
| ElasticSearch seems to have more mindshare. It can be easier to
| find resources online to help solve your problems.. though ES
| moves through versions fast (and they do break backwards
| compatibility on major version bumps) so sometimes this can
| still be an issue.
|
| Other benefit is that you don't have to rely on Zookeeper if
| you're horizontally scaling.
|
| I don't have a ton of experience with Solr but they seem pretty
| comparable.
| binarymax wrote:
| Great blog post on this subject:
| https://opensourceconnections.com/blog/2019/02/28/stop-worry...
| mdaniel wrote:
| The real vendor lock-in for ES (and now AWS OS) is the REST
| query API; if Solr implemented ES's API, I bet $1 a lot more
| people would have moved or at least considered moving over
|
| IIRC Solr also has some weird stuff about schemaless indices,
| whereas ES took the very Mongo-y approach of "yeah, just throw
| content at the index, don't worry about it" but then separately
| the approach of "I am angry with your new conflicting field in
| that document" and throws an exception; so you don't have to
| worry about the schema right up until you do have to worry
| about it
| elric wrote:
| In all three cases, Lucene is used as a "low level" (Java) API
| which provides search capabilities. OS, ES and Solr turn Lucene
| into a server, with features like horizontal scaling (ES
| Cluster, Solr Cloud). The major differences are in how well
| that all works, how easy it is to administer, how much caching
| and optimization is done on top of Lucene, etc.
|
| I haven't extensively used ES, but I've used Solr a lot (and
| contributed to), and I can say that it's a mess. The community
| is not one of the better ones I've seen. Bugs and stability
| issues are often ignored. Patches sit around gathering dust.
| There are some gems and very clever people in the community, of
| course, but it seems like there are too few of them to cope
| with the large beast that Solr has become. If I were starting a
| new project in the search space, Solr would not make my
| shortlist.
| busterarm wrote:
| I've used both and cannot think of a reason why I would use
| Solr again, besides licensing.
| vervas wrote:
| Not familiar with Solr but I believe the analogy to linux would
| be choosing the preferred distro while they all use the same
| kernel. There are a bunch of long comparison lists if you
| search for it.
| binarymax wrote:
| Hey All, if you're interested in getting a good understanding of
| this vs Elasticsearch, we invited the team to give a Haystack
| LIVE talk where they outlined the details and goals of the
| project: https://www.youtube.com/watch?v=J_6U1luNScg
| VWWHFSfQ wrote:
| I have mixed feelings about this server side license stuff that
| mongo db started. Imagine where the internet would be today if
| the creators of apache and mysql had tried to prevent shared
| hosting providers in the early days of the web from using their
| software
| asien wrote:
| > Imagine where the internet would be today if the creators of
| apache and mysql had tried to prevent shared hosting
|
| PHP + MySQL was the foundation of the Internet not so long ago
| , Wordpress is stil the backbone of lots of platform.
|
| With SS License definitely this would not have been possible.
| joking wrote:
| Imagine where elastic would be, as the whole success of elastic
| is based on a apache licensed project (lucene).
| madeofpalk wrote:
| In the same place as it would require elastic's changes to
| Lucerne to be made open source also?
| catern wrote:
| The server side license stuff doesn't prevent shared hosting
| providers from using software. It just requires hosting
| providers to open source their infrastructure.
|
| I think the internet would be even better today, if shared
| hosting providers had been sharing infrastructure technology
| since 25 years ago.
| VWWHFSfQ wrote:
| I think that's missing the forest for the trees. The license
| is designed to prevent hosting providers from selling the
| software as a service to their customers. The requirement to
| open source their entire infrastructure and operations is
| just a means to do that.
| antpls wrote:
| Technically it doesn't just prevent from selling SaaS. It
| prevents from selling SaaS without having the consent of
| the open source project first, which means negotiating a
| deal.
| brainless wrote:
| The time when Apache or MySQL started out was very different.
| Imagine where the internet would be if cloud computing itself
| didn't take off.
|
| Do you remember a time when there were hundreds of hosting
| providers? Do you remember WebHostingTalk where admins would go
| to check hosting offers from suppliers around the world?
|
| The monopolies finished that era. So I don't think that
| software companies trying to adapt now can be seen through the
| lens of what was 15 years ago.
| paxys wrote:
| There are more hosting providers today than in the era you
| are talking about. AWS has a "monopoly" simply because large
| companies are using it, and back in the day those companies
| would have run their own datacenters not used a shared PHP
| host. For a personal site or startup you have a thousand
| other options.
| that_guy_iain wrote:
| I wouldn't say AWS has a monopoly. There are tons of
| providers ranging from Digital Ocean to Google Cloud. AWS
| is just the largest because they've been around the
| longest. They're so deeply entrenched in Cloud that most
| people automatically think of them. I think things such as
| Netflix talking about their engineering and how they use
| AWS was a major boost at the start. Now a days people are
| literally studying to become AWS certified. We even have a
| AWS specialised working at my company. AWS and other cloud
| providers are also very smart in locking in start ups by
| offering them thousands upon thousands of free credit.
| Build your MVP on there and then end up vendor locked in
| but think it's a good thing.
|
| I heard a tidbit that I don't know if it's true or not but
| I would like to think it is. AWS has become so expensive
| for some companies that they're starting to migrate back to
| their own datacentres.
| darkwater wrote:
| I'm not even that sure this is true in absolute numbers,
| but I'm pretty sure it is not in relative figures. There
| are waaaay more customer/businesses on the Internet today
| than there were 15 years ago and not that many providers
| more.
| AtNightWeCode wrote:
| I have been perfectly happy with ES cloud services. Is this done
| by honest intentions from AWS or is it simply based on the fact
| that ES are making a lot of money of the cloud services?
| deknos wrote:
| i bet this will only really work in AWS. :D
| [deleted]
| TechBro8615 wrote:
| It's hard for me to know whether to feel bad for ES in this case.
| Did they bring it on themselves? Is Amazon too big and a bully?
|
| From my perspective, Amazon has made most of its profit price
| gouging consumers on bandwidth after vendor locking them into
| their ecosystem, where they bootstrap new services by wrapping
| open source software with some provisioning scripts, management
| dashboards and cookie-cutter API / console templates. Indeed,
| most of this is templated -- AFAIU, for example, each AWS service
| autogenerates its Boto bindings and parts of its console frontend
| via code generators. Amazon has really mastered the factory
| process of churning out new services, and when they find a
| popular one, they can invest more resources into developing it
| than the original team ever could.
|
| And therein lies the rub. If Amazon is improving the software in
| a way that the original team couldn't, it's hard to say that the
| community isn't benefiting. I think what strikes me the wrong way
| is that Amazon is not doing it for any altruistic reason. In
| fact, Amazon contributes very little to open source in general,
| considering how much they take from it. Compare them to Facebook
| (React, etc) or Google (tons of dev tools) or Microsoft (VSC,
| TypeScript). What does Amazon have? Firecracker, kind of? And now
| a fork of ES because that's the only way they could continue
| making money off it without violating the license a small startup
| put in place to stop them?
|
| Well, good for Amazon, I suppose, but I find myself instinctively
| disliking them for this. I'm not sure what the solution is.
| Hopefully technologies like Kubernetes and Terraform will
| encourage big customers to become at least cloud-agnostic, if not
| cloud-independent. At the very least it would be great if Amazon
| / Google / Microsoft stopped gouging bandwidth at such absurd
| margins. Or not. Maybe it will be their downfall as startups
| differentiate along those lines. That would be ironic, coming
| from the originators of "your margin is my opportunity."
|
| Personally I'm doing my part by not building anything with vendor
| lock-in. It's great to be able to deploy to any cloud, if you
| value either robustness or flexibility.
| raiflip wrote:
| It seems reductionist to say Amazon primarily wraps around open
| source. What about EC2? S3? Glue? DynamoDB? Many of the
| services that provide the most value are services Amazon has
| built out.
| sofixa wrote:
| Om top of these, many of the core services that AWS
| themselves rely on, like SQS, SNS, Kinesis, Lambda,
| Cloudfront, ECS, Fargate, Elastic Beanstalk are mostly
| homemade
| z77dj3kl wrote:
| My favorite "conspiracy theory" is that AWS intentionally
| creates stupidly verbose and numerous headers in all of their
| APIs just to up the bandwidth usage a few bytes per request at
| a time.
| cortesoft wrote:
| You don't pay for the bandwidth of API calls to AWS
| [deleted]
| [deleted]
| jrochkind1 wrote:
| What ES wants of course is for Amazon to give them a cut of
| revenue from hosting ES.
|
| We already know Amazon isn't interested in doing that (either
| at all, or at whatever price ES wanted, we don't know that).
|
| They had no legal requirement to when ES was open source. So ES
| changed the licensing to no longer be open source.
|
| So, Amazon could... a) decide to give ES a cut after all, b)
| decide to stop hosting ES, or c) fork the last open source
| version.
|
| I don't think anyone is surprised they chose c? Presumably ES
| isn't either? Maybe ES thinks this will be good for them/bad
| for Amazon anyway, because they are hoping potential customers
| will abandon the Amazon fork and stay on the original ES fork?
|
| Not sure why they'd be confident in that exactly. Maybe they
| know what they're doing.
|
| As users/customers, we would rather have a choice of hosted
| vendors/platforms, and that it remain un-forked (so we can
| use/write software compatible with either vendor/platform).
| Competition is good for us as users/customers, that's in fact
| one of the reasons we choose open source, so no one vendor can
| set the hosting price all on their own without competition. We
| want to be able to choose among competitors for hosting, based
| on price, customer service, performance, uptime, whatever.
|
| But ES didn't want that, they didn't want hosting competition
| to exist -- at least not without permission and agreed upon cut
| for them -- because, I guess, hosting was how they planned to
| make money as a company to fund development as well as profits
| for investors etc. So they changed their license to no longer
| allow it. So of the possible outcomes remaining... this one
| seems as good as any for the user/customer, I guess?
|
| So, when you say "I'm doing my part by not building anything
| with vendor lock-in" -- I'm not sure which course you are
| suggesting. In fact, between ElasticSearch and new OpenSearch
| fork.. it's _OpenSearch_ that is the one without vendor lock-
| in, right? OpenSearch is Apache licensed, and can be hosted by
| any vendor and still forked by anyone . It 's ElasticSearch
| that has a license limiting what vendors can host it (without
| permission of ES), it's the one with vendor lock-in, right? So
| not building anything with vendor lock-in means... ?
| galaxyLogic wrote:
| Good interesting points. Now Amazon will be the good guy
| because they will run open-source version, whereas
| ElasticSearch is not, if I understand you correctly.
|
| No single capitalist wants competitive markets. They want
| monopoly, for themselves. It is only when they don't have the
| monopoly or an easy way to get it that they cry for
| competitive markets. And that is good of course.
| dieters wrote:
| There are also ways out of vendor lock-in. Alternator comes to
| mind as a way of migrating DynamoDB workloads out to other
| cloud vendors or your own servers:
| https://docs.scylladb.com/using-scylla/alternator/
| prepend wrote:
| > In fact, Amazon contributes very little to open source in
| general, considering how much they take from it
|
| I don't think this is a fact. Amazon seems to contribute pretty
| significantly according to the pages [0,1] they put out that
| describes their contributions. Not to mention their membership
| in OSS foundations like Linux Foundation. [2]
|
| You have the caveat about in relation to benefit they gain, but
| that's pretty hard to measure. And I think isn't really a good
| measure.
|
| I'd like to learn more about why you make such an absolute
| claim and maybe you have some better measure.
|
| I remember back in the 90s when big orgs (Microsoft, IBM)
| didn't contribute to open source and can't even think of any
| big orgs today that don't contribute to open source. Even
| Oracle has big open source projects.
|
| [0] https://amzn.github.io/ [1]
| https://aws.amazon.com/opensource/ [2]
| https://www.linuxfoundation.org/en/join/members/
| aledalgrande wrote:
| Yeah and also what about projects like Firecracker?
|
| https://github.com/firecracker-microvm/firecracker/
| TechBro8615 wrote:
| The absolute claim, aside from being at home in a rant on HN,
| comes from a cursory glance at https://github.com/amzn,
| weighted by contributors and popularity, and compared to
| companies of similar size. Google, Microsoft and Facebook all
| build and maintain multiple open source projects that are
| hugely popular with people who use them outside of the
| company sandboxes. For example, people benefit from React
| without Facebook gaining much directly. ( _Facebook_! If
| Facebook has any redeeming qualities, it 's their open source
| contributions to the frontend ecosystem, although I promise
| you I could ascribe malicious intent to those as well...)
| Contrast that with Amazon. On their GitHub page, I see a few
| obscure projects amongst a bulbous array of AWS SDKs.
|
| To the sibling comment that asked about Firecracker -- I
| think Firecracker is awesome, and I did mention that in my
| original complaint. They even created it themselves! Well, a
| team of amazing engineers in Romania did. I have no personal
| insight into the matter, but it seems like they operate
| relatively independently from the AWS profit machine. Good
| for them too, it's incredible software. But I'm sure if they
| were to tell the story of how they got buy-in at Amazon to
| open source it, the same themes would come up -- how does
| Amazon benefit from this? In the case of Firecracker, the
| more people test it / harden it / run Doom on it, the more
| value Amazon can provide on its serverless platform. So
| again, unlikely to be purely altruistic intentions... but
| that's not to say there's anything wrong with that. I just
| find it all a bit distasteful in aggregate.
| dblock wrote:
| A better URL to look at is
| https://github.com/enterprises/amazon.
| whimsicalism wrote:
| That is a 404, so not better.
| ncann wrote:
| When I think of Google I think of Go, Angular, protobuf,
| Bazel, Dart, Flutter, Android, Chromium, Kubernetes,
| Tensorflow, etc.
|
| When I think of MS I think of C#, TypeScript, VSCode, .NET
|
| When I think of Facebook I think of React, Flux, Jest,
| PyTorch, GraphQL, Haxl
|
| When I think of Apple I think of Swift, WebKit
|
| When I think of Amazon...I can't really think of anything
| galaxyLogic wrote:
| I would add Kotlin to Google's credit.
|
| But I think it's important to note that these companies
| don't contribute to open source out of any moral
| obligation, do they?
|
| I think they do it to tie more developers and development
| around their eco-systems and products.
|
| Maybe Amazon should get smart and start doing something
| similar. Or maybe they don't need that. But in any case I
| don't hold it morally against them that they don't. I
| think a bigger issue is it seems they pay and have been
| paying very little or no taxes.
|
| https://www.cnbc.com/2020/02/04/amazon-had-to-pay-
| federal-in...
| reducesuffering wrote:
| No I would not add Kotlin to Google's credit. All the
| initial work and exponential adoption started with
| JetBrains. Google only greenlit it as an official Android
| dev language eventually (barring whatever OSS work
| they're doing on it only now).
| deanCommie wrote:
| I really don't understand this argument. Why do you think
| Facebook and Google have so many open source
| contributions? Is it really out of the goodness of their
| hearts? Or is it because that was part of their
| DELIBERATE STRATEGY to attract talent and use OSS as part
| of marketing outreach.
|
| Microsoft's core business is developer tooling. In the
| 90's and early 2000's that could be closed-source and
| proprietary. By the 2010s it was clear that the only way
| to operate with the kind of tools they have is to be open
| source, so they pivoted. But their goal is still
| business.
|
| Google built Kubernetes as a platform play to compete
| with AWS and Azure - brilliantly - by feeding engineer
| fears about "lock-in", giving them a set of tools that
| they could justify feeling "free", and then when the
| engineers invariably said "this is too complicated to
| build, maintain, and operate" they turned around and sold
| a GCP managed kubernetes solution! After all, who better
| to operate Kubernetes than the team that built it,
| amirite?
|
| Android is the same play just competing with closed-
| source iOS instead of AWS.
|
| Facebook built GraphQL for developers on THEIR Platform.
|
| Apple built Swift for developers on THEIR platform.
|
| Examples like this are just as cynical and capitalistic
| profit-driven as AWS "open sourcing" an SDK for
| interacting with AWS.
| dimitrios1 wrote:
| Facebook: Relay, Cassandra, React Native, RocksDB,
| Presto, Reason, btrfs, osquery.
|
| Apple made significant contributions to LLVM and Clang
|
| Hell IBM has more open source contributions (many through
| the Apache foundation and standards committees) than
| Amazon, which is saying something.
| cthalupa wrote:
| >Cassandra
|
| This is an interesting one in the context of this
| discussion. It likely does not exist as we know it
| without Amazon's Dynamo paper.
| amzn-throw wrote:
| > Well, a team of amazing engineers in Romania did. I have
| no personal insight into the matter, but it seems like they
| operate relatively independently from the AWS profit
| machine.
|
| Amazon and AWS is a massive multinational corporation with
| development teams around the world. Including Romania,
| where we had a dev center for a very long time:
| http://romania.amazon.com/#/
|
| There is no such thing as the AWS profit machine. All dev
| teams around the world operate with similar levels of
| autonomy and responsibility. It's just that some of them
| are working on super internal systems, some on super
| external, and some open source. Some make a ton of money,
| and some don't make any, but are beneficial to the overall
| developer experience/ecosystem, and so make sense.
| the_duke wrote:
| They didn't even create firecracker themselves. It's a fork
| of Googles crossvm.
| oblio wrote:
| https://github.com/aws
|
| https://github.com/awslabs
| cbushko wrote:
| To me it looks like they mostly contribute to projects that
| are SDKs to use AWS.
| manigandham wrote:
| People want to pay for services, not software and licenses.
| They want turn-key solutions that are available via API and
| GUI, instantly and on-demand. This is the fundamental reason
| why AWS is so successful and the demand is constantly proven
| with every new product launch.
|
| Elastic (and other vendors) complaining about this instead of
| using it for their own success is a problem of their own
| making. At least a few companies are finally learning.
| ampdepolymerase wrote:
| What's worse is the second order repercussions. Future open
| core/open source SaaSes will go straight to something like the
| Business Source License or the MongoDB license instead of
| traditional libre licenses. Amazon has done an incredible
| amount of damage to the open ecosystem.
| randoramax wrote:
| Good. So the VCs will go back to playing like in the old days
| of shareware software licenses. It'll be good.
| hintymad wrote:
| It sounds trivial to "wrap open source software", but
| surprisingly it is big value-add to thousands of companies. We
| can't just look at successful companies like Netflix to
| downplay the challenges of operating a service. Not every
| company knows how to operate complex systems under manageable
| cost. How many companies can really manage a Kafka cluster, let
| alone scaling it, for instance? Indeed, even companies that
| people deem powerful may screw up, if they don't get their
| culture or process right. Take Uber for example, for god damn
| five years, they still couldn't offer a service like EC2, let
| alone supporting persistent volumes. They still couldn't make
| their database provisioning on demand via an API. Their MySQL-
| based NoSQL solution was still based on FriendFeed's
| architecture and the APIs were hard to use. Yet they spent
| millions building a k8s replacement, building a GPU database,
| switching from mysql to postgres and back to mysql, etc and
| etc. So, yes, cloud companies like AWS buildd mere control
| planes to wrap open-source software, yet such seemingly mundane
| offering does bring values to many customers.
| cle wrote:
| This is the most important point, IMO. Amazon's value add is
| not the software itself, it's the operation of the software.
| That includes a LOT of stuff, not just making sure it's
| running. It's security modeling and patching, compliance,
| DDOS protection, etc. Amazon's product is an army of ops
| engineers working 24/7 to keep your stuff secure and online.
|
| With that in mind, their behavior here makes a lot more
| sense, and comparing it with companies who have dramatically
| different products, like Facebook and Google, takes a lot of
| effort to understand the differences and what impact they
| have.
| harshaw wrote:
| AWS just has engineers full stop. No specific ops
| engineers.
| stefan_ wrote:
| You can rent a car but nobody would suggest the car
| manufacturer be paid nothing for the privilege just cause
| the rental company _cracked_ car maintenance and how to
| fill the tank.
|
| The AWS "value add" is only value add in the context of
| being locked into AWS in the first place.
| soenkeliebau wrote:
| I'm not sure if I like your comparison to be honest.. not
| only is it not _just_ maintenance and filling the tank,
| if we stick with your picture, there is also things like
| buying the car, paying for it when no one uses it,
| insuring the car, repairing the car, general logistics of
| moving it around when someone has a one way rental, etc.
| - basically making it convenient for consumers to rent a
| car.
|
| Looking at what "as a service" providers of open source
| software do though, that is taking it a step further,
| since they wrap the software in a layer that _smoothes_
| out changes for the user. Going back to the rental
| company that would equate to the car manufacturer
| deciding the indicator needs to be on the right side of
| the steering wheel now and the rental company installing
| an adaptor so that it remains on the left for you, to
| keep the look and feel for the user the same.
|
| Not a perfect comparison, but they never are :)
| g9yuayon wrote:
| A key reason for Netflix to have an easy-to-operate
| infrastructure is that Netflix prioritizes productivity and
| scalability. They specifically did three things:
|
| 1. No fixed deadline, with a few exceptions of course, for
| platform-related projects.
|
| 2. Promotion/salary negotiation was not tied directly to
| release of external features.
|
| 3. A single engineer could be responsible for more than one
| service for the entire company, with 24x7 oncall.
|
| With Netflix establishing such incentives, engineers
| naturally focus on getting infrastructure right, to the point
| that oncall 24x7 is a non-issue.
|
| So, yeah, culture matters, big time.
|
| Edit: another incentive was that a service was measured by
| its adoption. The more people praised it, the more successful
| the service would be. Requiring meetings to get buy-in for a
| new service was considered a sign of potential failure. As a
| result, every single team focused on making the value
| proposition of their services obvious. Path of least
| resistance was a given instead of a debated topic.
| throwaway823882 wrote:
| > Is Amazon too big and a bully?
|
| Is Starbucks too big and a bully? Sure, they force Mom and Pop
| shops to close by out-competing them, and that sucks. But
| bullying? I believe that's just Capitalism.
|
| Or say you're 6'6 and weigh 230LBS and you join a football team
| full of people who are 5'9 and 175lbs. Are you a bully just
| because you're bigger?
|
| ES basically handed them a platter with a goose laying golden
| eggs and a sign that read "Free Goose" and hoped they wouldn't
| try to make money off the eggs.
| StavrosK wrote:
| > Is Starbucks too big and a bully? Sure, they force Mom and
| Pop shops to close by out-competing them, and that sucks. But
| bullying? I believe that's just Capitalism.
|
| I'd say it's more like a mom-and-pop coffee shop giving free
| coffee to patrons hoping to make money on cookies, and
| Starbucks coming in, taking the free coffee, opening a nice
| stand right next to the shop and selling the coffee they got
| for free.
|
| I'm having trouble justifying this ethically.
| throwaway823882 wrote:
| If they're giving you free coffee, and you sell the coffee
| you got for free... where's the harm? Starbucks is _giving
| it away_. Why would anyone buy your coffee rather than get
| it for free from the source?
|
| Now, assuming Starbucks charged you for the coffee, and you
| then re-sold it, this should also be fine. Starbucks is
| charging you presumably a rate with which they can recoup
| their costs. But if they are selling it _below_ cost, they
| are clearly putting themselves at risk. A lot of businesses
| take this kind of risk as a strategic part of their
| business, like with making the coffee free. But you have to
| do it in a way that a competitor can 't turn around to
| their advantage, or you're screwing yourself.
|
| Enter the concept of "not for resale". If a seller enters
| into a contract with a buyer, that contract can stipulate
| that the buyer can't resell the goods. Starbucks could
| theoretically require you to sign a contract saying you
| will not resell their coffee. That's pretty standard with
| licenses, even software licenses.
|
| ES must have known that their license did not forbid
| reselling. Yet they based their business model on this
| resellable coffee that they were giving away in order to
| make money on cookies.
|
| Is it Amazon's fault that ES chose a business model where
| they were selling coffee at a loss? Does Amazon have an
| ethical responsibility to keep ES's business afloat? Should
| we find any business unethical that tries to undercut the
| customer base of a rival, or take advantage of a rival's
| shaky business model?
|
| I think you have to come up with a whole framework for
| ethical competition, because one rule at a time isn't gonna
| capture it. (But I also think Capitalism is inherently
| unethical)
| nomel wrote:
| How is it unethical if ES chose to explicitly allowed it
| with the license they picked? The analogy would have to
| include a sign under the free coffee that literally said
| "Feel free to sell this free coffee for profit!", for it to
| be accurate.
|
| I don't see how it's unethical if you do something that
| someone said was perfectly fine to do. "Obvious chosen
| outcome" comes to my mind long before "unethical".
| notyourday wrote:
| As someone who tried buying services from ES and had to deal
| with their smug sales people that had a total disdain to those
| who wanted to give ES money, I am happy I will never need to
| deal with them in future.
| 015a wrote:
| One thing which surprised me: Elastic has a market
| capitalization of ~$11B.
|
| I think that changes some of the more floaty ethical concerns.
| This is not a David vs Goliath situation. This is Goliath vs
| Super-Goliath.
|
| At this point, I'm much less interested in the drama of which
| mega-corp is screwing over the other. I'm more interested in:
| how does it affect me? When the titans are done trampling over
| the rest of us, which side benefits me the most?
|
| Its too early to tell, but it seems like it'll be Amazon. The
| product is more open. They have a demonstrated history of great
| support. Yeah, they gouge us on networking and everything else,
| but at least they're the devil we know, and buying into the
| OpenSearch ecosystem has a greater probability of being the
| more open solution into the next decade.
| jimmy2020 wrote:
| This argument is part of Amazon's PR campaign to tell devs to
| not feel sorry for Elastic because it's now a big company and
| they make money in the market. So, if you built a successful
| OSS and start to make money then it's ethical to clone any
| OSS and pushes projects out of the market because now it is
| "Goliath vs Super-Goliath".
| manigandham wrote:
| Quite a few companies have proprietary products but have
| learned to make money selling services. One of them even
| has a 200B market cap and is called Oracle.
|
| Nobody needs to feel sorry for anyone.
| jimmy2020 wrote:
| Right. But the question here is not about companies value
| in the market. Why changing the subject? This should be
| about the ethics behind Amazon's aggressive actions
| against OSS and its effect on the OSS industry.
| manigandham wrote:
| The value in the market is the subject of the parent
| thread. Anyways, like I said, Elastic could've made money
| with a proprietary product but they chose OSS instead
| (and used Apache Lucene to build on).
|
| AWS is just offering customers what they want and there
| are many other companies doing the same thing (IBM's
| Compose, Aiven, Instaclustr, etc). How is this against
| OSS? This is the OSS industry operating as intended.
| jimmy2020 wrote:
| > The value in the market is the subject of the parent
| thread.
|
| As I said, it's a tactic to change the subject. Instead
| of focusing on actions with bad faith, change the subject
| by saying they have a lot of money so Amazon is allowed
| to be competitive and fork the code.
|
| > AWS is just offering customers what they want.
|
| No. it's not AWS improving some services. It's about a
| multi-billion dollar company launching a campaign against
| OSS. They did it with MongoDB and it continues. Some see
| these actions as justifiable. Because OSS should be MIT
| and maintainers should live with donations. Others,
| however, disagree. It can be OSS and profitable (without
| Amazon actions).
| pessimizer wrote:
| Forking OSS isn't bad faith. If you want to make people
| pay you, make your software proprietary. In that case,
| however, you wouldn't get the free ride into the market
| that being OSS gives you.
| jimmy2020 wrote:
| Yes. This is how you discourage people from doing OSS.
| "You can't stop me" argument leads to a slippery slope
| and if you care about the open-source you see it
| differently. Not because you can, but because of the
| consequences of your doing.
| aardvarkr wrote:
| Uhmmm I'm pretty sure David vs Goliath is talking about scale
| between competition. Saying that $11B is Goliath just because
| you're sitting at $1M doesn't mean they're not in a crazy
| mismatched fight against a $2T company. In the same way you
| could be in a David vs Goliath situation yourself if you with
| $1M in wealth tried to sue someone with $25K of wealth.
| Everything is relative. Doesn't mean it's not a crazy unfair
| mismatch that doesn't deserve sympathy and regret.
| ozim wrote:
| To be realistic and fair I don't believe part of Amazon
| that is doing fork of Elasticsearch is $2T, they have other
| stuff to do.
|
| Just like the guy that is $1M worth is not going to blow
| all of his money just to squash some guy. He probably has
| more powerful friends and maybe some connections but I
| don't see dumping $0.5m on lawyers just because he can take
| piece of cake from the other guy that will make him $10k a
| year.
|
| Imagine head of the department storming into Jeff Bezos
| office telling that he needs 20% of all Amazon worth to
| squash Elastic, that would be funny. Quick calc with 20% I
| assume would be $20B which would be only x2 of Elastic, so
| I don't see something like that happening.
| 20kleagues wrote:
| just a math correction: 20% of $2T is $400b - or about
| 40x of Elastic
| kodah wrote:
| Put another way, the $11B Goliath is .55% of Super Goliaths
| size.
| bombcar wrote:
| When there are two large entities (and I take this to mean
| way larger than you or your company) then you're better off
| rooting for the one that at least releases code you can
| use.
|
| If OpenSearch is truely open then in theory you can find
| another provider. But for ElasticSearch you're stuck with
| them.
| kfir wrote:
| > But for ElasticSearch you're stuck with them.
|
| I am a bit at a loss about that statement, you can run
| Elasticsearch on your on infrastructure
| virgilp wrote:
| That is not entirely true, of course. Not if you don't
| want to pay. As proof - just look at Amazon, they can't
| (that's why they forked it).
| mikepurvis wrote:
| I don't think it's the freedom to run it anywhere. It's
| the freedom to run it anywhere _and make changes to it
| that you don 't contribute back_:
|
| 1. Amazon wants to make private changes to the management
| layer for their cloud offering and not share those.
|
| 2. ES doesn't want that, so the 7.11+ license restricts
| it.
|
| 3. Amazon doesn't want to have to explain to their
| customers why their ES offering is stuck on v7.10, so
| they're changing the name of it.
|
| 4. Elastic was really hoping this wouldn't happen, but
| they overestimated the value of their brand and Amazon
| called their bluff.
|
| So yeah, nominally OpenSearch is unrestricted, but
| realistically few other entities are in a position to
| make or benefit from the private modifications Amazon
| will be making. For us normal people, ES and OS are
| equivalent today, so it's more about how they're going to
| diverge over time in terms of fixes, features, whatever.
| judge2020 wrote:
| The SSPL directly prohibits offering the software as a
| service without releasing the source code of your entire
| operation regardless of if you change it or not:
|
| > If you make the functionality of the Program or a
| modified version available to third parties as a service,
| you must make the Service Source Code available via
| network download to everyone at no charge...
|
| > "Service Source Code" means the Corresponding Source
| for the Program or the modified version, and the
| Corresponding Source for all programs that you use to
| make the Program or modified version available as a
| service, including, without limitation, management
| software, user interfaces, application program
| interfaces, automation software, monitoring software,
| backup software, storage software and hosting software,
| all such that a user could run an instance of the service
| using the Service Source Code you make available.
|
| AKA someone must be able to create aws.example.com if aws
| uses SSPL code. Not exactly the 'freedom to run it
| anywhere'.
|
| https://www.mongodb.com/licensing/server-side-public-
| license....
| mikepurvis wrote:
| Doesn't a lot of that hinge on what the "service" is?
| Like, if I run some generic knowledgebase website and use
| my own ES installation to provide user-facing search,
| does Elastic want to see my Ansible scripts? Or just the
| ones related to the search function?
|
| If this interpretation holds, I feel like it could be a
| pretty big problem for companies like GitLab, who not
| only have deep integration with ElasticSearch, but offer
| it as a premium-tier feature:
|
| https://docs.gitlab.com/ee/integration/elasticsearch.html
|
| Maybe this really is just a cash grab from the Elastic
| side, like, "it's too easy to self-host rather than
| paying for our SaaS offering, so now you need to pay us
| for many common self-hosting use cases also."
| judge2020 wrote:
| > Making the functionality of the Program or modified
| version available to third parties as a service includes,
| without limitation, enabling third parties to interact
| with the functionality of the Program or modified version
| remotely through a computer network, offering a service
| the value of which entirely or primarily derives from the
| value of the Program or modified version, or offering a
| service that accomplishes for users the primary purpose
| of the Program or modified version.
|
| This probably qualifies it as a service, but since it
| looks like gitlab doesn't install it or redistribute it
| Elastic isn't going to come after them (if they don't use
| SSPL code then SSPL doesn't apply). Plus, GitLab is
| effectively open-source, so they might meet the
| qualification for using SSPL anyways. Of course, I am not
| your/a lawyer.
| DetroitThrow wrote:
| There's several lawyers who believe the wording would
| imply the problematic interpretation - /dev/lawyer is one
| on the internet, one from my anecdotes is the lawyer who
| advised the company I used to work at.
|
| SSPL doesn't come up under free or open source licenses
| partly for this reason, actually.
| DetroitThrow wrote:
| Err, SSPL which Elasticsearch is licensed under isn't
| AGPL - it's a viral, proprietary license - not quite the
| copyleft we'd want if we wanted a fair playing field
| between all actors working on it.
| ocdtrekkie wrote:
| It's a viral open source license, it literally requires
| open sourcing code, there's nothing proprietary about it
| except that we allow a council of elitist snots to decide
| what is and isn't Open Source(TM), and they have decided
| Google and Amazon support is more important than viable
| businesses which are building open source businesses.
| busterarm wrote:
| The SSPL literally violates Freedom 0.
|
| > The freedom to run the program as you wish, for any
| purpose (freedom 0).
|
| And though we're talking about open source instead of
| free software, without Freedom 0, the software still
| might as well be proprietary.
|
| Edit: It also violates Rules 1, 5, 6 & 9 of the OSD. So
| no, let's not call it "open source"
| galaxyLogic wrote:
| Could you provide a link to explain how it violates
| "Rules 1, 5, 6 & 9 of the OSD"? Thanks
| busterarm wrote:
| Common freaking sense reading.
|
| https://opensource.org/osd
|
| If you can't use the software because you're a cloud
| provider, that's rules 1, 5 and 6 easy.
|
| If using it forces you to relicense all of your code
| under the SSPL that's rule 9.
|
| It is well known at this point that the SSPL withdrew
| their request for recognition to the OSI because they do
| not meet the rules of the OSD.
| DetroitThrow wrote:
| The principle of in the GNU manifesto would be that the
| software is available for anyone to use in the same way,
| such that Elastic isn't elevated to not disclose their
| closed source additions to the software.
|
| At this point, they are directly violating that core
| principle as well as the uncontroversial OSI directive 6
| which copyleftists like myself don't really have a
| problem with... So I'm not sure what the issue here would
| be other than you dislike the larger companies'
| involvement in OSS? I think myself and others would
| appreciate clarification.
| Macha wrote:
| If you think this is about Elastic caring about openness
| and freedom, ask yourself why they don't drop the CLA for
| a DCO and let themselves be beholden to the same terms
| mixedCase wrote:
| Quoting the comment you're replying to:
|
| > If OpenSearch is truely open then in theory you can
| find another provider
|
| Those are the key words. With an open source solution
| someone else can offer a turn-key solution that benefits
| from economies of scale.
| ForHackernews wrote:
| Uh...can they? Who's going to benefit from more
| "economies of scale" than Amazon, Alibaba?
| unreal37 wrote:
| Can Azure offer an OpenSearch service? Probably.
|
| So the idea is you can move from AWS to Azure if you
| wanted to with open source services underneath.
| 015a wrote:
| I don't root for David because he's smaller. I root for him
| because he's small.
| kordlessagain wrote:
| Let he who is without sin cast the first stone!
| adamdusty wrote:
| It may be relative but it isn't proportionate. A 11B market
| cap company can field a similarly competent legal
| department as a 2T market cap company can. I, with 25k,
| absolutely could not afford the same lawyer someone with
| 1M.
| ozim wrote:
| Have you considered that guy with $1M won't be willing to
| drop like $50k (2x your net worth) on a lawyer just to
| squash you?
|
| He must have possibility to earn $100k in process to do
| that. Well unless you really pissed him off.
|
| If you earn $25k a year and he spends $50k in one year to
| take your cake. Well unless he really is your competitor
| that can make use of your defeat he still needs 2 years
| to get even. Then it probably is not easy money because
| there is always a risk he will not win. Maybe he can find
| better ways to earn more money than squashing some $25k
| guys.
| reflexe wrote:
| It it quite funny, elasticsearch is also kind of wrapping an
| open source library (Apache Lucene) and selling it as their own
| product.
| downrightmike wrote:
| This is Amazon's playbook. Make a direct competitor and squeeze
| the originals out. They did that with jewelry early on and then
| anyone they couldn't buy out they would under cut until they
| capitulated like diapers.com The Everything Book by Brad Stone
| goes over this in detail. Clearly anti-competitive monopolistic
| actions are taken constantly by Amazon. The only reason they
| aren't trust busted is because the common line of reasoning is
| that consumers pay less for goods, but this is being looked at
| because shouldn't competition be lowering prices. IE if Amazon
| hadn't killed diapers.com, wouldn't diapers be cheaper overall?
| And the answer is they should be, but the government hasn't
| caught up. Once they start getting into the weeds, they'll see
| example after example of monopoly behavior destroying
| competitors and ultimately raising prices on consumers.
| amzn-throw wrote:
| > they can invest more resources into developing it than the
| original team ever could.
|
| I know this is a popular narrative, but as someone who works on
| AWS, I think you would be shocked by how small the individual
| dev teams are that build and maintain the services that
| everyone uses.
|
| I'm not going to downplay the network effects involved. Of
| course AWS has a tremendous advantage in being able to
| standardize the customer billing, IAM, and EC2 Usage.
|
| And there are economies of scale.
|
| But individual AWS service teams are: * incredibly lean and
| focused * still have to make a profit on their own terms based
| on the infrastructure they build and the fees they charge
| customers * laser customer obsessed to solve people's
| (developer's) direct needs.
|
| I understand the community's concern about AWS investment and
| approach to OSS. But I can assure you (though you have no
| reason to believe me) that the goal is never to embrace,
| extend, then extinguish. It's all in the service of going where
| the customers are, and solving problems that they tell us they
| have. The profits are a byproduct. The "working backwards"
| process is no joke. We spend a lot of time figuring out what is
| the right thing for customers to build, start building it, and
| THEN we think about how do we make money from it.
| g9yuayon wrote:
| In the world of OSS, code speaks. A small team may not help
| the cause of Open Search. This article nails how code
| influences leadership in an OSS project:
| http://hypercritical.co/2013/04/12/code-hard-or-go-home
| systemvoltage wrote:
| Could you please shed some light on how many people would be
| behind a product like AWS Lambda or AWS CloudWatch?
|
| As an outsider, I would guess huge swaths of developers with
| a massive hierarchy. Buildings full of folks working on AWS
| services. I have no idea and extremely curious.
| waynesonfire wrote:
| they have as many people dedicated to the capability as
| needed. at least the capabilities have owners, which from my
| experience is not trivial for an organization to achieve.
|
| curious if others have noticed this as well (capabilities
| without clear owners)? what does this depend on? time?
| company size? both?
| tssva wrote:
| "without violating the license a small startup put in place to
| stop them?"
|
| Elasticsearch was first released over a decade ago.
| ElasticSearch, now just Elastic, the company was founded over 9
| years ago and now is public. Are they still a "small startup"?
| If so when does a company graduate from that status?
| scarface74 wrote:
| There is absolutely nothing "cloud agnostic" about using
| Terraform. Every provisioner is specific to a cloud provider.
| If you are at any scale, moving a k8s cluster is the least of
| your issues.
| api wrote:
| You described the cloud lock in model very well, especially the
| bandwidth part. What they charge for outbound is nuts, and the
| other clouds are not much better. Inbound is free of course to
| make it easy to send your data in but costly to get it out.
| gsich wrote:
| No need to feel bad. The CEO sold a lot of his shares prior to
| the license change.
| busterarm wrote:
| As a longtime ElasticSearch cluster admin/developer and Elastic
| Cloud customer, I don't feel bad for Elastic in the slightest
| and I'm psyched about this fork.
|
| The way they operate their cloud service leaves a lot to be
| desired and encourages maximum spend if you end up wanting to
| use it for anything demanding in production.
| ProAm wrote:
| > The way they operate their cloud service leaves a lot to be
| desired and encourages maximum spend
|
| Pretty sure this is the definition of SAAS and IAAS
| busterarm wrote:
| They all give you levers. I'm more referring to being
| required to overspend to overcome Elastic's incompetence
| (i.e., you're already aware of the levers, they're maxed
| out and the provider doesn't have a way forward).
|
| Referring to my other example, in more detail: Elastic
| Cloud suggests operating your cluster across 3 datacenters
| for redundancy. This is a good idea. Then Elastic does
| maintenance in all three datacenters that your
| infrastructure is in at once and takes your clusters hard
| down. This is fucking stupid. Elastic's suggested solution
| to the problem: Operate a _duplicate_ cluster in 3
| datacenters in a different region/provider. No guarantees
| that they won't do the same there either so it's not
| actually a solution.
| cassianoleal wrote:
| Not to mention the support is pretty slow to respond and
| rarely respond adequately.
|
| At my current client we started with their cloud and in a few
| months deployed our own in Kubernetes using the official
| operator.
|
| It's a lot cheaper and gives us fewer headaches this way.
| busterarm wrote:
| My favorite moment as a customer dealing with all of the
| random times _they_ would take us down (even with triple
| redundancy) was their suggestion of operating a duplicate 3
| datacenter cluster in production as a hot spare.
|
| They must think their customers are cash machines.
| bombcar wrote:
| I mean if they were truely best of breed they wouldn't
| have needed a license change now would they?
| busterarm wrote:
| Indeed, although I'd say for a majority of situations,
| Amazon's hosted Elasticsearch service is a complete non-
| starter.
| damon_c wrote:
| Why is it a non-starter? I have a few ElasticSearch
| clusters that have been running for years with zero
| effort/downtime/hassle.
| x3n0ph3n3 wrote:
| It's pretty bad when you reach the 40-50 node scale with
| 10's or 100's of TB of data. I've had about half dozen
| calls with their service team about this over the last
| year.
| busterarm wrote:
| What the other guy said.
|
| Also my saying that is extremely colored by experiences
| with pre-ES6 versions, where AWS's offering didn't have
| many of the configuration knobs available that you really
| _need_ to operate a decent cluster.
| hospadar wrote:
| Personal experience is that AWS elasticsearch has often
| been missing some really useful stuff, index rebalancing,
| some of the utility endpoints that avoid me having to
| spend forever rebuilding indexes, etc. I'd be a little
| spooked to run something really huge and customer-facing
| (maybe it's possible and I'm a n00b, but for the money we
| pay, it should be n00b friendly).
|
| It's always great (really, it was quite easy to get
| started and usually works) until it's not (a couple times
| have had indexes break, or had to reindex to a fresh
| cluster to fix balancing problems).
|
| I'm sure LOTS of people use aws elasticsearch for big,
| user-facing stuff, but I often feel you'd be better off
| managing it yourself if it were truly critical.
| bogomipz wrote:
| This is really a shame to hear. There was once a an Elastic
| SaaS company from Norway called found.io that were pretty
| sharp and customer-centric. They were acquired be Elastic
| pretty early on[1]. I believe Elastic Cloud was built from
| this. I guess found.io's culture of delivering a good product
| didn't survive?
|
| https://www.elastic.co/about/press/elastic-acquires-
| elastics...
| dijit wrote:
| Honestly the amount of negativity towards Elastic in this
| thread is jarring. I've been an ES customer (self hosted,
| some cloud) for years and have only good things to say
| about them. Maybe I'm less a fan of the number of features
| they try to bake in and the direction of the company
| towards using logs for metrics (which causes heavy disk
| load instead of just storing metrics in time series) but.
| Yeah.
|
| I truly believe the negative voices are coming to the fore
| in this thread or they are paid Amazon folks (or they have
| a vested interest in AWS succeeding here).
| busterarm wrote:
| So everyone who doesn't share your experience is a paid
| actor? Not on throwaway accounts?
| bigtones wrote:
| Are you even a customer if you don't pay Elastic any
| money ? Your a user, sure, but not a customer.
|
| These other opinions and negative voices you reference
| come from actual customers who pay Elastic large sums of
| money, and they feel they don't get good value and
| service for that money they're forking over.
| dijit wrote:
| You cannot use the cloud systems without paying; how on
| earth do you figure I'm not a customer from what I wrote?
| bigtones wrote:
| Because you said you're "self hosted" on 'some cloud'.
| You don't need to pay Elastic if you're self-hosted and
| not using the Elastic Cloud - you can just download it
| and use it for free.
| narism wrote:
| They could be paying Elastic for gold/plat/etc licenses
| on their self hosted instances or "self hosted, some
| cloud" could mean they run mostly self hosted with some
| Elastic cloud usage.
| jnwatson wrote:
| How is it price gouging if the price is on the tin? It isn't
| like there's a "surprise" as to how much they charge, and it
| isn't like there aren't a dozen alternatives including DIY.
|
| I'm the first to say that AWS is too expensive, and I vote with
| my wallet (and the company I work for by proxy). But I'll never
| claim that there's any gouging involved.
| ballenf wrote:
| Price gouging is the practice of using outsized leverage in a
| particular market to charge excessive prices. Like snow
| shovels doubling in price after a snow storm. Or $10 water
| bottles after a hurricane.
|
| So for AWS the term is arguably correctly applied.
|
| But I'd be more worried about the market if AWS was
| artificially undercutting pricing because it would kill the
| incentive to create competitors or innovation in the space.
| sofixa wrote:
| > Price gouging is the practice of using outsized leverage
| in a particular market to charge excessive prices. Like
| snow shovels doubling in price after a snow storm. Or $10
| water bottles after a hurricane. > So for AWS the term is
| arguably correctly applied
|
| What outsized leverage have AWS had for a decade? There are
| multiple competitors at different levels, AWS are just
| better in terms of coverage/redundancy and amount of
| services.
| jen20 wrote:
| > So for AWS the term is arguably correctly applied.
|
| Can you elicit the argument by which this is correctly
| applied?
| marvindanig wrote:
| Coincidentally, AWS hasn't open-sourced anything that they use
| internally. Zilch. Nada. And yet they are using 'open source'
| developed by another firm (smaller is inconsequential here) to
| market themselves.
|
| IMO, FOSS licensing is completely broken. Its definitions (of
| what is free/open) are from a boomer's era that is no longer
| sustainable today.
| grumple wrote:
| Here are the main reasons to make open source software:
|
| To provide something for free, public use
|
| To get the world to help you maintain your software
|
| A reason to not make open source software:
|
| You want the exclusive right to offer that software as part of
| a paid service
|
| This is not the first instance of a company not understanding
| the last point there, and it won't be the last.
| fastball wrote:
| I think the claim that Amazon is winning through "vendor lock-
| in" is pretty silly. Honestly anyone who can't quickly migrate
| the stuff they're hosting on AWS onto one of the many other
| cloud platforms is pretty bad at DevOps. If you're using
| K8S/Docker/etc it should be _trivial_. But even if you 're not,
| the vast majority of AWS offerings were either built to be API-
| compatible with other existing tools (e.g. postgres-compat
| Aurora RDS), are literally identical to other services you can
| self-host (e.g. ElasticSearch) or _others_ have built services
| compatible with AWS services (e.g. DigitalOcean 's "Spaces" aim
| to be API identical to AWS S3 - you can literally use Amazon's
| S3 client libs to interact with various S3-compat services from
| other clouds).
|
| It's not "lock-in", it's providing a great all-in-one solution.
| You can host everything you want to host on AWS, which has good
| stability, good latency, etc. People are locked in because the
| DX around using AWS for everything your platform needs is just
| _better_ than other platforms / having different services on
| different cloud providers (at least for many people).
| StreamBright wrote:
| Do you know can I migrate my sophisticated security setup
| easily over? Including users, groups, roles, policies and
| instance policies.
| fastball wrote:
| Everything on AWS is accessible via API, which means you
| can easily automate the migration process. So do that?
|
| Obviously the more complex a system is the harder it will
| be to migrate, but that has nothing to do with Amazon
| trying to "lock you in" and everything to do with it just
| being a complex system that there is no industry standard
| solution for.
| real_joschi wrote:
| If you're not using any of the AWS services, that might be
| true but then you're also leaving a lot of potential on the
| table.
|
| If you're "cloud-agnostic" and could migrate away from AWS in
| the blink of an eye then you're paying for an overpriced VM
| offering and should probably migrate to a cheaper hosting
| provider immediately.
| busterarm wrote:
| > If you're "cloud-agnostic" and could migrate away from
| AWS in the blink of an eye then you're paying for an
| overpriced VM offering and should probably migrate to a
| cheaper hosting provider immediately.
|
| No this is what everyone suggesting this does not get. The
| offerings are not equivalent.
|
| There are a baseline of services that the cloud providers
| offer that can be made functionally-equivalent. It's not
| just EC2 but more like EC2,S3,Lambda,RDS,DynamoDB,ECS,EKS
| (plus some others and of course the other cloud's
| equivalents). The secret sauce is in the APIs and
| permisioning and all of these available within the same VPC
| (talking to each other without paying bandwidth costs).
|
| "Cloud-agnostic" has _never_ meant "just VMs". Some of
| these services are majorly hard to duplicate on your own
| VMs as well. Feel free to implement "cheap VM hosting + S3"
| and burn cash on transit costs.
|
| Cheaper hosting providers do not give you this by miles.
| Dylan16807 wrote:
| > all of these available within the same VPC (talking to
| each other without paying bandwidth costs).
|
| That's one of the real key issues, I think.
|
| If bandwidth is 10x cheaper, then maybe you decide it's
| fine if only half your inter-server bandwidth is free.
| busterarm wrote:
| It's always worth measuring/understanding your
| application because likely it's not "half" but skews
| heavily one way or the other (in vs out).
| JamesSwift wrote:
| The lock in is real, but they are also the best cloud option
| and its not even close in my opinion.
| fastball wrote:
| What would you say they lock you into? What is AWS doing
| specifically that makes it harder to move between providers
| than if they were not doing that thing?
| JamesSwift wrote:
| 1) A vast majority of users are relying on managed
| services to some degree.
|
| 2) Permissions as a whole are usually very ingrained into
| much of what orgs rely on.
| fastball wrote:
| What do you mean when you say managed services? Because
| to me managed services are definitely _not_ vendor lock-
| in because you can self-host most of those services and
| just migrate your data over.
|
| With regards to permissions, I don't feel like it's
| possible to permission across an entire platform and not
| make it difficult to migrate over - do you know of any
| provider or platform that allows for cross-platform
| authentication with permissions?
| JamesSwift wrote:
| I generally think of managed services as anything not run
| directly on EC2.
|
| I agree about permissions. I'm just illustrating where
| the lock in occurs.
| ukd1 wrote:
| I buy the main reason being the all-in-one solution; the
| comprehensiveness is attractive. However, I think you're
| underplaying the lock-in: migrating clouds is non-trivial -
| mostly due to stuff that's not running in k8s/docker/etc;
| stateful apps (Postgres, etc), and or just static data like
| s3. This takes time, and careful planning and sometimes
| downtime - and is mostly avoided due to it being hard.
| fastball wrote:
| I specifically talked about state and storage.
|
| You're conflating the difficulty of migrating a complex
| system to _anywhere_ with "vendor lock-in". It would be
| _harder_ to migrate an Aurora RDS Postgresql instance to an
| Aurora RDS MySQL instance than it would be to migrate from
| Aurora RDS (postgres) to a hosted postgres anywhere else.
|
| That to me screams "not vendor lock-in".
| minhazm wrote:
| Is it really price gouging for bandwidth? Or is bandwidth just
| really expensive in general? I honestly don't know. I would
| assume if it was actually much cheaper one of the cloud's would
| undercut the other to get customers.
| arkitaip wrote:
| Why would players in an oligopoly undercut each other when
| their implicit agreement around pricing makes all of them
| richer? Also, second tier cloud providers like Oracle give
| deep discounts and still can't compete with AWAZGO so pricing
| isn't necessarily a main competitive advantage.
| kmeisthax wrote:
| Oracle's problem is that nobody wants to work with Oracle.
| If I was managing a high-bandwidth service I would avoid
| Oracle Cloud purely to avoid legal risk to my clients.
| DetroitThrow wrote:
| While I agree that's why no one uses Oracle, corps not
| competing for price at the tier 1 provider level still
| isn't great for the consumer/market, which is what I
| think the original commenter was getting at.
| VectorLock wrote:
| I'm kind of surprised that people are this upset about how
| much AWS charges for bandwidth. They may charge more for
| bandwidth than a colo would but they're not a colo. A colo
| you get a network port and -thats it- you provide everything
| else yourself, with its attendant cost, and you roll that up
| into your total bill.
|
| If a colo provides you a 1 Gbps connection if you use less,
| you don't get a refund. And most of the time you don't get
| 24/7 saturation, or you get charged on some 95th percentile
| billing, and their networks are almost always oversubscribed
| anyways.
|
| AWS is trying to disincentivize using it as a dumb pipe. They
| want you to use it smartly and if you just want to push
| static data there are much more cost effective ways to do it,
| such as CDNs, which are more cost effective for both you AND
| AWS.
|
| Comparing AWS bandwidth costs and Colo and even other clouds
| like Oracle isn't fair because different things are
| associated with that cost.
| TechBro8615 wrote:
| It's absolutely price gouging. I'm not going to rant about
| this for the 100th time, but at least I'm in good company
| [0]. Do the math on the cost you pay if you saturate 1gbps
| for a month vs. the cost you pay for 1gbps IP transit at
| basically any colocation provider.
|
| Really this is the secret sauce of the cloud. Create new
| abstraction layers where you can charge for logical
| separation on a physical basis. First VMs, then containers,
| then serverless... Would be cool if somebody did it with
| bandwidth (looking at you, Cloudflare). Why can't I buy an
| elastically sized pipe? Why do I need to pay for the stuff I
| put through it instead of reserving a size for the time I'll
| need it?
|
| [0] https://twitter.com/eastdakota/status/1371252709836263425
| maximente wrote:
| boy howdy, with all the flak i hear about this and the
| awesome talent in tech, you'd figure an entrepreneur or
| 1000 would take a stab at this, make it better, charge
| less. apparently there's gazillions to be made by even
| charging 50 percent of what AWS does.
|
| so, when should we expect this gloriously efficient
| competitive market to kick in to action?
|
| my guess is that the AWS ecosystem, despite "price
| gouging", is simply the best and will be because this is
| really hard, non-glorious engineering, where solid
| reliability actually matters. anyone who wants to can go
| ahead and co-lo, so, whatever. people who want cloud will
| pay, and those who can't or won't, will not.
| Dylan16807 wrote:
| What competitive market?
|
| Nobody else can give you bandwidth out of Amazon data
| centers. Amazon's advantage is having a ton of services
| that work together, and they take advantage of it to
| price gouge on bandwidth.
|
| If you're buying a standalone CDN service you can get
| massively better rates.
| lostcolony wrote:
| Your point that "AWS has a bunch of other benefits to
| where people just accept the bandwidth costs so they can
| leverage those other benefits" doesn't actually counter
| the original claim that "the bandwidth costs are
| outrageously overpriced".
| toast0 wrote:
| There's tons of businesses that are happy to charge less
| for bandwidth; so it's clear Amazon (and some of the
| other high tier cloud services) are overcharging on this
| maybe by a factor of 10, although since transit bits are
| not all equal, someone with more detail could make a case
| that the overcharging is less.
|
| It's easyish to compete on bandwidth costs, but Amazon
| has a lot of other features many people want; it's harder
| to replicate all of those, especially the part about
| having a long history of operating such services and not
| making a lot of changes to make things more expensive or
| otherwise more difficult. Having to pay a much higher
| than market price for an easily replaced good in order to
| get a good that's less easily replaced is textbook anti-
| competive bundling.
|
| If your bandwidth usage is high enough, maybe it makes
| sense to send it all through AWS direct connect, and pay
| for transit yourself; although even then, the AWS direct
| pricing seems a bit high.
| VectorLock wrote:
| This is like going to a fancy restaurant and being upset
| that they charge so much for a steak when you can get
| beef at the supermarket for much less.
| stefan_ wrote:
| If you ask the fancy restaurant to cater a steak dinner
| for 1000 people they will charge you a lot closer to
| supermarket beef.
|
| The point is that if you charge absurd prices for what
| has basically no marginal cost your pricing model is
| broken and 1) you are excluding customers that are
| particularly sensitive to _this price_ or 2) you are
| liable to undercharge other customers that primarily use
| other services for which you are not charging what it
| costs you to provide.
|
| For AWS, given the generally inflated prices, it's
| probably a lot more of 1) than 2).
| VectorLock wrote:
| As I explained in another commented AWS wants to
| disincentivize dumb bandwidth usage. They want you to use
| your bandwidth for traffic that needs it to EC2, and you
| get much better rates for static data from CDNs, S3, etc.
| manigandham wrote:
| You're replying to a comment that includes a tweet from
| the CEO of Cloudflare, which is quite literally providing
| that competition with free bandwidth and an increasing
| suite of computing products.
|
| There are plenty of other platforms as well, like
| Digitalocean, that have much lower bandwidth pricing.
| sofixa wrote:
| That Twitter thread is only including pure bandwidth. What
| about all the highly redundant networking equipment (
| firewalls, routers, switches, Nitro offloading, DDoS
| protection, attack detection), software for all those
| abstractions you get ( VPCs, subnets, security groups, vpc
| peerings, Elastic IPs etc. ) and engineers? None of what i
| listed you pay for directly, and bandwidth seems to be the
| most reasonable product to lump it all in.
|
| It's like going to a restaurant and complaining about the
| price of steaks because beef should cost a lot less.
| There's a ton of other things involved, and yes, they
| probably have a decent margin, but not as much as you
| initially imply.
| stefan_ wrote:
| Keep in mind bandwidth gets cheaper as AWS gets bigger. If
| you are some random tiny colo provider, people don't
| necessarily care to peer with you unless you pay them for the
| privilege. If you are originating 20% of internet traffic,
| now people _need_ to peer with you or their customers won 't
| have a great experience.
| GordonS wrote:
| It _really_ is price gouging - bandwidth is actually cheap.
|
| A couple of comparisons:
|
| Oracle Cloud give you 10TB of bandwidth for free, with
| overage charged at around EUR7.5/TB.
|
| You can rent a VPS from the likes of Hetzner, and they will
| throw in 2-20TB of bandwidth for free, with overage charged
| at something like EUR1/TB - AWS charge an eye-watering EUR125
| for each TB!
|
| I think the reason the big 3 (AWS, Azure, GCP) still charge
| such huge amounts is that they profit so greatly from it, and
| there is more than enough business to go round.
| rodgerd wrote:
| I would say that it's two simple factors:
|
| 1. High bandwidth charges are an effective form of lock-in.
| Once your data is there, it's prohibitive to move it out
| again.
|
| 2. Bandwidth use is very poorly understood in many
| businesses, compared with simpler metrics like storage,
| memory, and CPU. AWS can run razor-thin margins on things
| that people easily compare to on-prem or VPS-style
| offerings, and then make the money back in areas like
| network traffic, fine-grained monitoring, and other items
| that as less obvious.
| martinald wrote:
| I totally agree, but was stunned to see DigitalOcean
| charging $0.10/GB for outbound transfer for their new
| (quite cool imo) apps service. You do get 40GB-100GB
| included, but it means it's unusable for bandwidth heavy
| apps. They include 1TB (which is pooled across all VMs) and
| 0.01/GB on their standard platform.
| GordonS wrote:
| Yeah, I think Digital Ocean are really taking the piss
| there - proven by their existing $0.01/GB pricing, as you
| mentioned.
|
| I reckon DO have seen the big providers getting away with
| it, and they want a piece. Why wouldn't they, when there
| is apparently so little pushback?
| martinald wrote:
| I suppose you have object storage which is still $0.01/GB
| (plus includes 1TB to start with), so for most apps your
| bandwidth will come out of that for most files, so your
| 0.1/GB price is only for html/api/etc transfer. But
| still, it's annoying and seems completely arbitatary.
| sofixa wrote:
| > I think the reason the big 3 (AWS, Azure, GCP) still
| charge such huge amounts is that they profit so greatly
| from it, and there is more than enough business to go round
|
| Or that their networking does vastly more and is of better
| quality? Hetzner can't provide you with everything around a
| VPC that AWS do. Just a tiny example - peering VPCs across
| regions, which is free.
| veeti wrote:
| You know you're greedy when Oracle is cheaper
| gnfargbl wrote:
| IP transit costs something like $350-$700/mo for a Gbit.
| Amazon are certainly getting better rates, so even with
| equipment costs I doubt they're spending much more than
| $0.005/GiB. Their pricing starts at $0.15/GiB. (Not to single
| out AWS, the other big providers are much the same.)
| ev1 wrote:
| I'm below $100/1Gbps 95% with more than one provider as a
| small player
| gnfargbl wrote:
| That's sounds like a _great_ deal: you 're paying less
| than half of Hurricane's advertised minimum price. And I
| had understood HE were a mid/low price carrier.
|
| Can I ask, are you in a data center? US or Europe?
| ev1 wrote:
| What do you mean? Hurricane's current special is
| $90/1Gbit/m.
|
| US.
|
| Some of the larger bandwidth transit resellers like FDC
| will do Cogent for $0.02/Mbps. They have a marketing site
| at epyc100gig.net (not an affiliate or employee; just a
| FDC customer).
| gnfargbl wrote:
| OK, either I'm getting pretty screwed somewhere or EU
| prices are well above the US. Probably the former. Thanks
| for the pointer.
|
| https://he.net/cgi-bin/ip_transit_quote is where I'm
| seeing the $200 number.
| ev1 wrote:
| On front page https://www.he.net/ there's a special in
| the top right :)
|
| I believe that kind of low-for-1G-commit pricing is for
| their fully owned FMT1/FMT2 US CA facility. You have a
| lot easier peering in EU (AMSIX, DECIX, etc) that will
| help compared to the US's love of commercial exchanges
| like Any2/Coresite/Equinix where peering costs
| practically more than transit.
| gatvol wrote:
| If you think AWS is expensive, give AZURE a go and be
| appalled.
| dariusj18 wrote:
| > Amazon is not doing it for any altruistic reason
|
| The beauty of OSS is that motives don't matter. If Amazon
| contributes and it's not detrimental in someway to the code,
| then it's a plus for anyone else who wants to use it.
| DetroitThrow wrote:
| Precisely! While their business itself may need to be broken
| up, a community governed OSS project isn't bad for OSS when
| the alternative is a proprietary license that gives a single
| corp the ability to not contribute back or be exposed to
| virality.
|
| All this being said, progressive corporate taxes seem more
| enticing year after year.
| InTheArena wrote:
| I suspect this is mostly AWS trying to stop using "ElasticSearch"
| in the title of something, probably for trademark reasons.
| mumblemumble wrote:
| It's been said that the best way to fortify your business is to
| use your clout to make the world inhospitable for adjacent
| businesses.
|
| As someone who is not currently in the cloud, that idea strikes
| me as being very pertinent to what's happening here. Increasingly
| many technologies are becoming cloud-only, or have non-cloud
| offerings that are decidedly second-class. Elastic offers on-prem
| support. I doubt Amazon will be doing the same with OpenSearch.
|
| It may be a subtle effect, but it's pushing the world in a
| direction that makes me uncomfortable. If it's harder for non-
| cloud-based companies to maintain non-cloud-based offerings, then
| that will push the industry even more toward being dominated by
| SaaS products. And these products often leave clients and users
| locked in, with limited control over their own data, and, by
| extension, reduced ability to control their own fates. What I
| worry about is that we may be witnessing a return of Embrace,
| Extend, Extinguish, only in a new form that's even more dangerous
| because it's harder to see.
|
| I appreciate the discomfort people have about the SSPL. It is a
| departure from the original ideas behind FOSS. But, at least as I
| see it, those open source principles were never an end in and of
| themselves. They're a means to a greater end: digital autonomy.
| To the extent that very large companies seem to be learning how
| to co-opt FOSS in order to re-assert control, FOSS's ability, in
| its current incarnation, to serve that end may be waning.
| busterarm wrote:
| Elastic's on prem support amounts to little more than an onsite
| where they explain to you how the Java Garbage Collector works.
| EVERY detail about tuning your clusters derives from keeping
| the Java GC from ruining your day.
|
| There's some index template optimizations but any semi-
| competent engineer or dba should be able to figure all that out
| (it's literally all in the documentation about what not to do).
|
| You'll still be able to pay a consultant to come help you --
| they don't have to be from Elastic.
|
| In fact, it seems like Amazon just created an industry for
| third party consultants here.
| ddevault wrote:
| I'm happy to see a couple of good choices made here:
|
| - Sticking with Apache 2.0
|
| - Asking for a Developer Certificate-of-Origin rather than a
| copyright assignment
|
| This bodes well for the future of this fork. Amazon also has the
| resources necessary to keep up consistent and quality maintenance
| of a project on this scale.
|
| Elastic would definitely like you to view AWS as the Big Bad
| here, but their response to the Elastic betrayal is very good,
| and I would like to see more like this in the future.
| fallat wrote:
| So as someone who has heard about Elasticsearch for years and
| years, and seen all this, right this moment I've decided to see
| what it really is.
|
| On their home page, "Why use Elastic search?", the reasons are
| basically:
|
| * It's fast!
|
| * It does a lot of stuff!
|
| * It has some tools to visualize data!
|
| * It's distributed!!
|
| I have to say this is not very appealing to me since it sounds
| like something any database could do.
| manigandham wrote:
| The Elastic website has a dedicated page to explain:
| https://www.elastic.co/what-is/elasticsearch
|
| _" Elasticsearch is a distributed, free and open search and
| analytics engine for all types of data, including textual,
| numerical, geospatial, structured, and unstructured.
| Elasticsearch is built on Apache Lucene and was first released
| in 2010..."_
|
| I suggest understanding what it is first before comparing it to
| other databases.
| fallat wrote:
| A lot of databases handle many types of data.
| [deleted]
| villasv wrote:
| You're starting from the wrong place if you're comparing
| Elasticsearch with a database. And you're also arriving at the
| wrong place if you think that any database can be distributed.
| WrtCdEvrydy wrote:
| Elastic gives you a lot of the fancy stuff that SQL kinda needs
| extras and hard work for... but it's just a document store with
| fancy weighting features.
| xapata wrote:
| > just a document store
|
| With a particular profile of efficiency choices and
| interfaces that might appeal to your project.
| api_or_ipa wrote:
| Elasticsearch is a no-sql database that optimizes for full-text
| searches, built atop Apache Lucene. If you're doing any kind of
| full-text search, for example, if you're trying to index a
| university library and make it searchable, then elasticsearch
| is for you. If you're not, I'd look elsewhere.
| xapata wrote:
| Devil is in the details.
| candiddevmike wrote:
| Will be interesting if other cloud providers (Google, Azure)
| offer this or you see other software companies offer support for
| it.
|
| Will also be an interesting case study if the community shifts to
| this project and it dwarfs elastic for features.
| galaxyLogic wrote:
| I wonder if ES had originally been AGPL licensed would that have
| helped them? If Amazon adapts AGPL code to integrate it with
| their own infra=structure doesn't that in fact mean that all of
| Amazons' software-based infra-structure would become AGPL as
| well, and thus easily reproduced by Google Cloud, MS Cloud,
| Oracle Cloud etc.? Or even inhouse? In other words wouldn't it
| mean it would be easy to replicate the Amazon Cloud-business (on
| a smaller scale)?
| lacker wrote:
| Amazon just wouldn't do that. They would either not offer it as
| a service, or make a clone from the beginning like they did
| with MongoDB. In general none of the cloud providers are
| actually willing to comply with the AGPL license.
| pessimizer wrote:
| Offering the unaltered software as a service, or forking it
| and releasing all of your changes under AGPL _does_ comply
| with AGPL.
| rurban wrote:
| They took over FreeRTOS for good, CBMC for good, with Xen they
| were a bit unlucky, but it still has much better security than
| KVM, and now they take over ElasticSearch.
|
| Good Open Source efforts, much better than until a few years ago.
| luke2m wrote:
| This is just sad, amazon playing the victim.
| whydoineedthis wrote:
| anyone know if OpenSearch still uses "/" as a special character?
| Largest pita when trying to use ES for logging web applications
| and quite frankly, made it near unusable.
|
| If Amazon fixed that, I would be firmly on their side. Also, any
| improvement over Kibana would be welcome.
| hit8run wrote:
| Perfect case for a megacorp destroying open source plus business
| models. I start to hate amazon with a passion. Craziest thing is
| they are not paying taxes in Europe though they dominate the
| market. Amazon needs be broken up. It's too big and too mighty.
| themolecularman wrote:
| cancel amazon prime?
|
| It's not too difficult to wait an extra day or two for
| products.
| kizer wrote:
| What about the thing that is already named OpenSearch?
| dewitt wrote:
| I'm the co-author and maintainer of the OpenSearch syndication
| protocol and I posted in support of reusing the name here:
| https://groups.google.com/g/opensearch/c/gi-iVJZgfdA
| kizer wrote:
| Turns out that was developed by Amazon according to Wikipedia.
| So maybe they're merging that usage into this offering (since
| that is a spec for search results)?
| carlfmeadows wrote:
| Yeah the original OpenSearch project is a different enough
| domain that I think confusion will be minimal. We have talked
| to the maintainer and he is supportive. We have also posted
| disambiguation in case anyone does get confused.
| https://opensearch.org/disambiguation.html
| jrochkind1 wrote:
| In general I'm supportive of an Amazon open source fork here.
|
| But the name re-use is unfortunate.
|
| Amazon's argument seems to be "Don't worry, we own the
| trademark for 'OpenSearch' cause it came out of Amazon
| originally, so it's cool!"
|
| That is really poor stewardship of the Intellectual Property
| of the trademark of a name that was part of a _standard_ that
| was meant to be an open multi-vendor standard. Amazon owned
| the trademark to protect it 's use under that standard, not
| to re-use it for something totally different harming the
| standard further.
|
| But it's just another indication that the original
| Opensearch, like the era of believing in open web standards
| for inter-operability that it was part of, is dead.
| DetroitThrow wrote:
| I agree, it speaks poorly for their stewardship
| tardyp wrote:
| There is also https://www.opensearchserver.com/ which is the
| first result when your search "opensearch docker".
| kizer wrote:
| It's what Chrome's tab to search utilizes. If you implement the
| protocol users can tab-to-search your site even with
| autocomplete.
| sunilkumarc wrote:
| Interesting!
|
| On a different note, recently I was looking to learn AWS concepts
| through online courses. After so much of research I finally found
| this e-book on Gumroad which is written by Daniel Vassallo who
| has worked in AWS team for 10+ years. I found this e-book very
| helpful as a beginner.
|
| This book covers most of the topics that you need to learn to get
| started:
|
| If someone is interested, here is the link :)
|
| https://gumroad.com/a/238777459/MsVlG
| [deleted]
| mekster wrote:
| Are people actually required to use ELK? What are your use cases?
|
| The interface is completely cluttered and it takes loads of
| resource and it feels like it's waiting to be replaced with
| lighter and more focused products.
|
| Graylog (though it uses Elasticsearch internally) does a decent
| job at log handling and creating all the visual items out of logs
| and Grafana/Loki can do quite good at it as well with a very
| small memory footprint.
|
| Besides, most of the "business intelligences" aren't actionable
| but just some visual arts you wouldn't need but to stare at when
| you're bored.
| mdaniel wrote:
| I recently learned that Graylog changed their license and it's
| now one of those vanity licenses:
| https://github.com/Graylog2/graylog2-server/blob/master/LICE...
| oever wrote:
| The new name clashes with the Open Search Foundation.
|
| https://opensearchfoundation.org/
| DetroitThrow wrote:
| That's extremely frustrating
| busterarm wrote:
| Amazon was using the term "OpenSearch" themselves (via former
| subsidiary A9) back in 2012.
| https://github.com/dewitt/opensearch/
| noitpmeder wrote:
| Like they care
___________________________________________________________________
(page generated 2021-04-12 23:00 UTC)