[HN Gopher] PlanetScale - Database for Developers ___________________________________________________________________ PlanetScale - Database for Developers Author : samlambert Score : 188 points Date : 2021-05-18 17:16 UTC (5 hours ago) (HTM) web link (www.planetscale.com) (TXT) w3m dump (www.planetscale.com) | cfors wrote: | > Developers want the durability, stability, and scalability of a | SQL database but do not want to be constrained by managing a | schema. It has been our goal to give both, not compromising on | the power of your datastore but making changes feel as easy as | deploying code. | | So is this a wrapper around managing Schemas powered by Vitesse? | (Btw, had to go to your github to figure that out) | | If you want to be the database for developers you should know | that developers do care about how you do this scaling. | halostatue wrote: | Amy I the only person reading this who doesn't think that the | marketing line in the middle about not managing a schema is super | scary? | | > Developers want the durability, stability, and scalability of a | SQL database but do not want to be constrained by managing a | schema. It has been our goal to give both, not compromising on | the power of your datastore but making changes feel as easy as | deploying code. | | Sorry. There's always a schema, and learning to manage it is a | _good_ thing. If you're not ready to manage and migrate, then | you're not in production. You're in something else. | [deleted] | Allstar wrote: | It's not entirely clear from the website or the documentation, | but this seems to be build exclusively for MySQL? | samlambert wrote: | Yes we are MySQL compatible. | rad_gruchalski wrote: | 100% compatible? Is there a document with a compatibility | matrix? Edit: I guess this is it: | https://vitess.io/docs/reference/compatibility/mysql- | compati.... | [deleted] | sorenbs wrote: | I have been using the beta version of PlanetScale for a while, | and it is extremely cool. It's using the mature technology that | powers Youtube to provide a developer experience for databases | similar to what Vercel and Netlify provide for hosting: It will | give you a database branch for each Git branch and help you | manage the workflows around it. | | And it is the first truly serverless relational database offering | that I am aware of. The cost scale to 0, so it is perfect for | small projects, but that same instance will scale to support | massive load when you need it. | asimpletune wrote: | So just wanted to add a few random thoughts about some of this | stuff. | | With sharding, where this stuff eventually gets you is when your | assumptions about shard keys no longer hold. | | Eg, you might have user initiated traffic to start with, so you | can easily and automatically shard everything by user id or | whatever. Then one day, those assumptions change because you | might have to accommodate event driven traffic, ie not | request/response, and the user's id can't be assumed to always be | present. For example let's say something in the real world causes | an event to be pushed onto your queue. That event could | correspond to a real user, but since it originated somewhere in | the real world, there's probably a separate id for how that user | is represented. So you can't rely on that user ID being present | to shard things by. | | Not sure, if that makes sense, but sharding can be hard. It's not | like free, and I still think it's important for engineers to | understand the mental model they're using, even with a tool like | vitess. | | Also, I saw claim either on planetscale or vitess that MySQL has | no native support for horizontal scaling with automatic sharding, | but I think they do? I think you just have to pay for that | though. | | Also, cross-shard transactions were mentioned as another | difficulty with sharding. They can be done with either sagas | (depends on the context but it's a design pattern), or 2PC which | is available in MySQL > 5.8 I believe in the form of XA | Transactions. | adambair wrote: | The amount of buzzwords on the site made me think this was some | kind of elaborate joke. I'm not yet convinced it isn't... | itwy wrote: | > The site is experiencing higher than normal traffic and we have | temporarily halted database creation. | | Ironic coming from the infinitely scalable database, isn't it? | paxys wrote: | Well that restriction will probably disappear if you just give | them your credit card number | golondon wrote: | Aren't schema changes are actually a big problem when there is a | huge amount of data in the table. Adding a column / index to a | table that has 500M rows in it is usually the pita. This is | definitely something on the right direction, but would love to | see this supporting this with the data itself. | lizztheblizz wrote: | That is exactly what we _do_ support. Our team includes the | original developer of gh-ost for MySQL, which has been built to | execute these kinds of massive changes at scale at GitHub. | We've integrated it tightly with Vitess. Once your branch is | ready to be merged with production, it executes that change in | a completely non-blocking way. | golondon wrote: | Ok, but most of the time you would like to test that change | on a copy or almost up to date replica, to see measure things | like how long it does take. If you were copying the data, | with things like PII filtering, to the branch development | database and allow folks to test it there it would be even | more amazing. Great start tho! | unknown_error wrote: | Can someone please explain how Vitess works, in plain English? | How does it magically make MySQL scale? | | And then what does PlanetScale add on top of Vitess hosted | anywhere else? | | Sorry, the linked blog post is both very abstract and assumes a | high level of preexisting knowledge about database scaling. | moshmosh wrote: | IME the answer to "how did they make [hard to scale thing] | easily scalable?" is _usually_ that they introduce limitations | in how you can use [hard to scale thing] so you can 't use it | in ways that are hard to scale, then automate scaling it in | well-known ways for use cases that are so-limited. Vitesse's | site mentions that it relies on horizontal sharding, so right | off the bat, my guess is that you can't use it in ways that are | sharding-unfriendly, or if you can then you'll be met with | restrictions on much of the "magic" of it if you do. | | _Rarely_ is it the case that someone 's actually discovered | e.g. novel math or something to make the hard part easier. | Better tools (to do well-understood things more easily for this | use case) and restrictions (so you don't use it in ways the | tools can't handle) are the usual way. | lizztheblizz wrote: | The "ease" we used to refer to in Vitess primarily relates to | its interaction with the application side, where it basically | presents itself as "one big MySQL datastore". It uses a | standard MySQL connector, and in general, once you have the | infrastructure up and running with a compatible schema | design, there's not too much to worry about from a coding | standpoint. Sharding happens transparently to the application | code, which generally translates to fewer code changes | required. | | Admittedly, that view left out the considerable challenge of | actually deploying and running the infrastructure, designing | and optimizing that schema, along with all the joys of | managing large cluster environments. | | That's what PlanetScale, the product, aims to solve. Dealing | with clustering infrastructure IS a hurdle for most teams to | overcome, and though Vitess' feature set and compatibility | has expanded greatly to accommodate some of the most | demanding use cases on the web today, a lot of its | functionality can still be out of reach for a developer just | trying to merge some code and a schema change. | | Abstracting as much of that complexity away from the end user | is the goal, as well as making their lives easier with a ton | of the functionality we've always wanted to see built with | Vitess. I can confirm that that is not an "easy" job on our | end. :) | paxys wrote: | Vitess is an additional layer on top of MySQL which all queries | pass though. Among other things, it implements its own query | parser which can then do stuff like split a query across shards | and join results etc. | | I wouldn't say there's too much "magic" in there, but it does a | lot of known difficult things (schema management, | sharding/resharding, connection pooling, query optimization, DB | administration, monitoring, backup/failover) which are | generally painful and expensive to do yourself. | motives wrote: | As I understand it, Vitess is basically a really powerful | sharding system, which goes a step further than typical | sharding solutions by basically making the shards one or more | unique databases. In the case of someone like slack, because | your tenant (e.g your company slack), is completely isolated | from other tenants, you can treat that basically as its own | database, and have a master for just that DB, allowing much | better scaling. The big limitation on Vitess is cross-shard | transactions, and the fact you have to make sure your schema | has a clear cut sharding key (like your tenant ID) that works | nicely with your application needs. The alternative for scaling | transactional SQL DBs in a multi-master fashion are the | "NewSQL" DBs like yugabyte and cockroachdb which are basically | document DBs with a partially implemented postgres frontend, so | don't have the full feature set of your SQL engine like Vitesse | does but don't require so much attention to sharding. These are | oversimplifications of the actual mechanisms, but give a basic | overview of the tradeoffs involved, please feel free to correct | me on any inaccuracies as I'm not an expert in DBs. | irfansharif wrote: | (crdb eng) I'm not sure what "document DB" means here, mind | elaborating? | aPoCoMiLogin wrote: | he is probably referring to the docdb document store in | yugabyte: | https://docs.yugabyte.com/latest/architecture/layered- | archit... | motives wrote: | Indeed I was referring to yugabyte, apologies for the | clumsy phrasing, I havent used crdb but I guess it is a | postgres frontend layered on a KV store instead of a | document store? | qeternity wrote: | As opposed to? Postgres is "just" a Postgres front end on | top of a kv store. | kmavm wrote: | I was Chief Architect at Slack from 2016 to 2020, and was | privileged to work with the engineers who were doing the work | of migrating to Vitess in that timeframe. | | The assumption that tenants are perfectly isolated is | actually the original sin of early Slack infrastructure that | we adopted Vitess to migrate away from. From some earlier | features in the Enterprise product (which joins lots of | "little Slacks" into a corporate-wide entity) to more post- | modern features like Slack Connect | (https://slack.com/help/articles/1500001422062-Start-a- | direct...) or Network Shared Channels | (https://slack.com/blog/news/shared-channels-growth- | innovatio...), the idea that each tenant is fully isolated | was increasingly false. | | Vitess is a meta-layer on top of MySQL shards that asks, per | table, which key to shard on. It then uses that information | to maintain some distributed indexes of its own, and to plan | the occasional scatter/gather query appropriately. In | practice, simply migrating code from our application-sharded, | per-tenant old way into the differently-sharded Vitess | storage system was not a simple matter of pointing to a new | database; we had to change data access patterns to avoid | large fan-out reads and writes. The team did a great write-up | about it here: https://slack.engineering/scaling-datastores- | at-slack-with-v... | motives wrote: | Definitely wasn't expecting the chief architect at Slack to | reply to that example, really appreciate the response, HN | is such a blessing in that regard :). The scaling | datastores at slack is a super interesting read aswell | thanks, does make me wonder if there was a fully 100% MySQL | compatible version of yugabyte/spanner etc if that would | have shifted the decision. | Rapzid wrote: | Something you really hit on here; there is no free lunch with | clustered databases. You have to design your application to | account for the sharding or you will run into data locality | related performance issues. | isatty wrote: | What is this? Reading this post does not explain it, but I figure | it some sort of new DB. | jaredcwhite wrote: | Can I install this locally? Will it work without an internet | connection? Is it fully open source? From what I can tell, the | answer to all three questions is no. | | Is it yet another example of vendor lock-in? Yes. | [deleted] | paxys wrote: | The underlying technology (https://vitess.io/) is fully open | source, so I imagine there will be very little vendor lock in. | sigg3 wrote: | If it is MySQL compatible it's not vendor lock in. | | It's more like a clever trap, first hit is free etc. | | Will be interesting to follow this. I'm not db savvy so I don't | mind leaving that job to someone else. | jaredcwhite wrote: | Vender lock-in isn't just about data formats, it's about the | overall makeup of your infrastructure and development | processes. If an org builds around the particular way | PlanetScale manages DB branches and other minutiae of their | service, they can't simply replace that with an in-house DB | server overnight. | piaste wrote: | True enough, but by that standard, every service would | constitute lock-in. A cleaning company isn't lock-in just | because you don't have in-house janitors. | | "Vendor lock-in" means that you're tied at a business- | critical level to some proprietary technology _that you | can't get anywhere else_. | | If you rent Linux servers, Postgres databases, or even k8s | setups from Azure, there's no shortage of alternative | vendors willing to sell you compatible product should you | tire of MS, and ideally you won't need to change much more | than a few endpoints. However, if your entire user base | lives in Azure AD, it's a very different story. | unknown_error wrote: | There is still value in proprietary solutions that offer | significant time/money savings vs open source. Not all of us | have infinite resources to develop and maintain every solution | in-house, whether that's an auto-scaling database or email or | Google Docs or whatever. | | For small/medium businesses, sometimes it's better just to let | the experts do their thing and build on top of it... | wiradikusuma wrote: | Can serverless solutions e.g. AWS Lamba or Google Cloud Run | connect to this? | samlambert wrote: | Yes absolutely! | jopsen wrote: | Where is it hosted? Do you have instances in all data centers | moving shards to where they are used? | | Using databases outside the same availability zone can be | slower... | aantix wrote: | I wonder what the latency would be to have a heroku insurance | pointed at a PlanetScale instance. | | Doesn't look like there's Postgres compatibility. | vira28 wrote: | Wondering will we see something like this for Postgres? | | I like the cool things but I can't migrate to MySQL just because | of this. | NAR8789 wrote: | looks like it's a ways off... planetscale says they're built on | vitess, and vitess has postgres compatibility at the bottom of | their "Medium Term" roadmap: | https://vitess.io/docs/resources/roadmap/ | Allstar wrote: | I've been following Citus Data, it might be interesting for | you. https://www.citusdata.com/ | samlambert wrote: | Sure you can! and we will make it worth it. | NAR8789 wrote: | I can't decide whether this joke is marketing folly or | strategic genius. | | On the one hand, making light of how big an undertaking a | _database engine migration_ would be makes you come off | sounding like the "mongodb is web scale" guy. That's a pretty | terrible look for a company proposing to take on critical | production infrastructure. | | On the other hand, adding postgresql support to vitess is | probably a _massive_ undertaking. Sure, it might make sense | for you to contribute to in the long run to open up that | customer segment, but at your current stage postgresql | customers are probably more of a product-distraction than | anything. In that light, driving us away for now is probably | the best policy. | | I'm gonna give you the benefit of the doubt. Well played, | sir. Well played. | | EDIT: nevermind, all your other comments so far in this | thread are variations on "Yes, we can do that!". You're | promising the moon, and that's fishy. | | Bit of unsolicited advice (you know what they say about | unsolicited advice, but here goes...): You'd be better off | admitting some weaknesses and discussing tradeoffs. For | examples, take a look at other HN threads where execs have | hopped on. What qualities do you seen in posts that get the | most positive responses? Now compare that to the confusion | and skepticism in this thread. Your messaging has landed off- | target. | | You're proposing to take over a piece of people's critical | production infrastructure. As a potential buyer considering | an infrastructure provider, seeing "buy our product and we'll | solve all your problems" just makes me skeptical. It makes | you sound like a used car salesman. Your landing page | delivers that basic message, and your comments in this thread | reinforce the resulting impression. | | Conversely, if in these comments you soberly acknowledged | your limitations, transitioned to a detailed treatment about | tradeoffs, and _then_ used that save to toot your horn some | more, that would give the impression you've considered this | problem deeply, help me figure out how closely your value | prop is aligned to my goals, and make me more inclined to | trust you, your company, and your product. | | As it is... | | I see your comment and it makes me think your company isn't | ready to take on my infrastructure (this is my database we're | talking about here... if you're not taking something as basic | as migration costs seriously, what other nasty surprises are | waiting for me down the line?). | | Then, I take a look at your about page (maybe this guy's just | an entry-level marketer in the people-pleaser phase), and I | see that you, Sam Lambert, are the Chief Product Officer. | That gives me some serious doubts about the future viability | of your company, because now I'm worried the CPO is | fundamentally out of touch with the userbase and can't | grapple with difficult details. | | You will not be touching my data any time soon, but I'll be | watching for changes, and I honestly wish you luck. I like | your vision, but your messaging execution needs work and your | product execution is yet-unproven. | | EDIT: kudos to lizztheblizz for giving more detailed answers | that inspire confidence. You've significantly repaired my | impressions of your company. | samlambert wrote: | We are definitely not out of touch with our user base. The | reason I feel a high degree of confidence is that we | currently have a major Postgres user moving to PlanetScale | at the moment and they are loving the experience. This is | v1 of our product. I have not seen a database product like | this, that cares about the daily lives of the users that | use it. I believe we can create a fundamental shift in how | databases think about their users. | | We concern ourselves with the productivity of our users. We | also solve hard problems at scale, for many large | companies. We have so many exciting developer focussed | features that I think people will love, all the while being | on a viable database. Vitess is by far the highest scale | open source database solution out there. | | Migration cost is a serious thing. We believe the immense | leap in productivity you will gain over time from our | product makes it more than worth it. | capableweb wrote: | Sounds interesting but there is not enough information for me to | get a picture of what this actually means. Thought I was gonna be | clever and install it, run it and see what it is all about but I | can't figure out how to get it. One link is "sign up" and the | other is "contact sales" but I'm not interested in either, I'm | interested in "Download" but I cannot find it anywhere. | | Am I stupid or misunderstanding something fundamentally here? Is | just a hosted DB or something like that? | moshmosh wrote: | It's a hosted DB, and as far as I can tell the Killer Feature | is that it makes schema updates less painful. How it does that | without significant performance trade-offs or caps is unclear | to me. | | [EDIT] on reading further in their docs, my suspicion is that | their "branching" concept is a hell of a lot more limited than | I believed at first. I initially took it to mean you could have | multiple active schemas working on your data at once--instead, | I think it's more like exporting _just_ the schema of your DB | and importing it to a fresh DB, which is nothing new and doesn | 't run into all kinds of operational and security issues the | other workflow would. I'm fairly sure all the actual _magic_ is | in the schema diffing, and the docs make me think even that isn | 't as fully-magical as one might hope. | lizztheblizz wrote: | It's closer to your original assumption than your second. | Branching relies on vreplication, and it does actually allow | you to develop multiple versions of your schema against your | full production dataset. The "magic" is a powerful | materialization engine that allows every branch to maintain | its own version, and it _does_ allow you to work with your | actual data. This is just scratching the surface of what that | can do, though. We have loads more features cooking. :) | ngrilly wrote: | Really great finally seeing a "serverless" SQL database based on | MySQL and Vitess that scales from 0 to n, even with a free tier! | | Where are you hosted? What latency should we expect from AWS, | GCP, DigitalOcean and fly.io? | | And also what degree of compatibility should we expect with | MySQL? The doc is quite sparse on this. | lizztheblizz wrote: | Vitess' compatibility with MySQL has made major leaps in the | past couple of versions and the team has started focusing on | locking in ongoing compatibility with various popular | development frameworks. You can find those here, and more are | getting added regularly: https://github.com/planetscale/vitess- | framework-testing/ | | The basics of MySQL compatibility are described in here, though | it's important to keep in mind that just because something | "works" doesn't always mean it's the best way to do things in a | sharded environment: | https://vitess.io/docs/reference/compatibility/mysql-compati... | 2color wrote: | This is wonderful. I've been developing with relational databases | for 12 years and couldn't be more excited about this. | | Scaling down to 0 opens up a world of opportunities when it comes | to team development workflows. The idea of quick environments per | pull request is within reach. | | Not to mention that it scales with you. | | The one thing I'm curious about is how it compares with | CockroachDB | machiaweliczny wrote: | Quick environment per pull request is available with SQLite now | :) | Rauchg wrote: | From the Vercel point of view, this promises to answer one of the | most frequent, interesting, and technically challenging questions | since we first launched our "immutable deploys". | | That is: how can I pair a brand new frontend preview deploy, with | a _serverless_ database with the specific schema my new feature | needs? | | This technology makes the whole serverless stack feel complete. | eurasiantiger wrote: | It doesn't solve the N+1 queries problem in a generic way. | | More to the point, it probably cannot solve it efficiently at | all, since it is not a graph database and thus cannot be paired | with a generic GraphQL resolver (would generate join queries | instead of lookups across edges) and a stack of generated | static queries in the backend (no need to allow generic | queries, just make it possible to write queries once and only | once). | adamfeldman wrote: | Branching wasn't mentioned in the linked blog post, but I | assume that is what the parent comment is excited about: | https://docs.planetscale.com/concepts/branching | Rauchg wrote: | Indeed, thank you! | qaq wrote: | With solutions like CockroachDB, YugabyteDB etc. gaining steam do | things like Vitess have a place long term? | k__ wrote: | I'm currently searching for a serverless database for my next | project and will look into it. | | I already played around with Upstash and Fauna. | | Somehow the signup shows an 422 error, then I get an confirmation | email that leads to a blank page. | paxys wrote: | If I understand correctly, PlanetScale = hosted Vitess? I assume | it was started by the creators/maintainers of the project | (similar to Confluent and such)? | samlambert wrote: | Yes! | satyrnein wrote: | It sounds like the database branch has copy of the production | schema, but what about the data itself? We've been using AWS | Aurora clones to quickly give developers copies of production, | can zero-copy clones be made as part of the branch? | gryzzly wrote: | is it not funny that something with a word "scale" in it disabled | creation of databases because of being under load? | christiansakai wrote: | will this support blob as well like S3? | pratio wrote: | The pricing gives me anxiety. | | $1.25/mo per 10GB storage | | $15/mo per 100 Million rows read | | $15/mo per 10 Million rows written | | But I won't lie, super excited to give this a try. | stalluri wrote: | Branching looks super dope! | | https://docs.planetscale.com/concepts/branching | olivierlacan wrote: | The boulder-sized caveat for all this admittedly really neat | stuff being: | | > PlanetScale's Non-Blocking Schema Changes' workflow doesn't | support FOREIGN KEYs in users' databases. | | > PlanetScale determined that the production safety that Non- | Blocking Schema Changes provide are worth this technical | tradeoff. Learn more. | | https://docs.planetscale.com/concepts/nonblocking-schema-cha... | lizztheblizz wrote: | To be clear, this is not a Vitess/PlanetScale-specific opinion | or choice. Foreign key constraints are a bit of a controversial | topic in large-scale MySQL environments in general, which is | the greater context in which this design decision was made by | the Vitess team. | | PlanetScale's (and Vitess') non-blocking schema changes rely on | open source tools for MySQL like pt-online-schema-change and | gh-ost, which are widely used in production environments | everywhere, and neither of them are too comfortable supporting | FK's, though pt-osc does accommodate them to some extent | (https://www.percona.com/doc/percona-toolkit/3.0/pt-online- | sc...). gh-ost's lack of support was discussed on HN previously | here: https://news.ycombinator.com/item?id=16983620 | | A good collection of resources on why they're considered | problematic and many companies designing large-scale MySQL | schemas tend to drop them can also be found here: | https://federico-razzoli.com/foreign-key-bugs-in-mysql-and-m... | CSDude wrote: | It looks great, congrats | nebulous1 wrote: | I have to admit that with the previous mocking of "web scale", I | initially assumed this was satire. | briandoll wrote: | Before GitHub launched, I built some large eCommerce sites, and | for a vcs we used CVS, and then Subversion. We had a person on | our team with the title of release manager, because branching in | Subversion, and merging back to make releases, was a specialist | effort that took time, patience, and managing a tremendous amount | of fighting between teams of what features and fixes could even | be merged together in order to ship this release. | | When I started using git, it broke my brain. I wasn't really sure | I understood it, but the idea of cheap local branches soon became | the most important thing to me in a vcs, and everything became | easier and so much faster. You could just work on code the way | real life happens, not in some methodical pre-planned release | schedule that always rubbed harshly against the reality of bug | fixes and ever changing minds. | | Planetscale is a lot like that transformation. We've been | thinking of databases as this specialist thing, where the arcane | knowledge required to make it perform well is a specialist role, | and something completely untouchable by mortal engineers. While | we've learned about the importance of good data modeling, we've | dealt with a layer in our stack that is essentially static. | Shipping features that require changes to a database schema sit | and languish, because the pain and coordination required can | often be too much for a team to want to deal with. | | What PlanetScale has done is solve two massive problems in one | platform. First, since it's built on Vitess, it Just Works At | Scale. You don't need to fiddle with knobs to get it to perform | well. You most definitely are not even in the top 10% of what is | already running on Vitess, so you don't really need to worry | about ever outgrowing PlanetScale. But the BIG innovation here is | what is possible now that a database works the way the SDLC has | evolved with git. Make a branch, change the schema, roll it out | with your code. Just like anything else, really. Use the | PlanetScale CLI to actually USE your database. Change the schema | because it will make your code better, and don't worry that | you're somehow going to do it wrong. | | PlanetScale just made databases useful for developers beyond a | nearly-static data store. It's a high-scale database that you can | change like code. It will break your brain a little at first, | just like git did. And then you'll wonder why the hell we waited | so long to have a database this good. | chris_st wrote: | Sounds great! | | One question about pricing -- I can't tell if the "free" tier | costs are for one month, or ongoing. That is, do you get the | free tier amounts for one month and then pay from then on, or | is that level of service free from then on? | samkottler wrote: | The free tier is free forever. | stronglikedan wrote: | > free forever | | or as long as a free tier exists ;-) | [deleted] | onionisafruit wrote: | Free until you decide it isn't free anymore. | briandoll wrote: | The free tier provides the stated usage levels at no cost, | indefinitely. So if you were to stay under the quotas of data | size, writes and reads, you don't pay at all. When you go | over, you just pay for what you use. No servers, no pre- | committed capacity, just simply pay for what you use. | kall wrote: | Does it mean you can write 10 million rows every month for | free or does it mean you can write ten million rows, | however many months that may take, and then pay for all use | after that. The first one is how pricing generally works, | but the page could be read as the second. | lizztheblizz wrote: | This is monthly. :) | kall wrote: | OK, i think a little /mo on the pricing page wouldn't | hurt. ___________________________________________________________________ (page generated 2021-05-18 23:01 UTC)