[HN Gopher] Postgres WASM
___________________________________________________________________
Postgres WASM
Author : kiwicopple
Score : 597 points
Date : 2022-10-03 14:25 UTC (8 hours ago)
(HTM) web link (supabase.com)
(TXT) w3m dump (supabase.com)
| netcraft wrote:
| I don't mean to take anything away from this post - its quite
| amazing and I can't wait to play with it more - but wanted to
| mention that there are sites out there that I believe solve the
| training portion maybe a little bit better, at least if all you
| want is to train on SQL not DBA type actions. My favorite is
| https://dbfiddle.uk/ - the ability to link and fork a set of
| statements is extremely handy.
| pistoriusp wrote:
| Totally, and what we're really trying to achieve here is to
| make this available to everyone, so that we can improve it as a
| collective.
|
| If you're building a product on Postgres and you want expose an
| evaluation version, or teach someone a Postgres functionality
| then build on top of this.
| burggraf wrote:
| So I've been using this for a couple weeks now on my mobile
| phone. There's really no reason to do this, other than I'm
| sitting in a car waiting for my wife to come out of the grocery
| store and I just feel like writing some some SQL code -- because
| I can. If you noticed, there's a little keyboard icon at the top
| of the demo that toggles on a mini keyboard with ^c, ^d and
| cursor keys. That's because the iPhone doesn't have those keys,
| and I'd start pinging a server and realize I can't break out of
| it and now I need to reset the whole emulator. Anyhow, that's why
| that little toggle is in there. :)
| _joel wrote:
| Completely naive question here, but could you cluster the WASM
| instances (with mods, I'd assume, more is it possible via the
| WASM abstractions?). Not sure why but seems like a interesting
| thing :)
| burggraf wrote:
| Yes, you probably could do that. It's something I've
| considered. Early on before I got the networking working
| through the proxy I found a way to use arp to connect multiple
| browser tab versions of this in a pseudo-private-network
| configuration. It only works locally and only works inside
| chrome, and only a single instance of chrome at that. But it
| was cool idea and the only use case I could think of was
| setting up a local cluster of Postgres instances within the
| browser. And my kids think playing video games is fun! Meh.
| steinlagems wrote:
| Scott from JavaScript Jam here. We are so excited that we are
| going to have Peter with us on Twitter Spaces this Wednesday to
| chat about Postgres WASM. Join us Live here:
| https://twitter.com/i/spaces/1yoKMZbgngkGQ See you Wednesday!
| Kye wrote:
| There have been a lot of new WASM things popping up on HN. Was
| there some major new change the enabled all this?
| lioeters wrote:
| I've also noticed a number of fun WASM projects this year. I
| think the underlying WASM features that enable all this have
| been around for a while, so the breakthroughs are probably
| coming from the community and ecosystem growing incrementally
| and synergistically, building on each other's work and
| collaborating.
| angelmm wrote:
| One of the things I love from this post is the collaboration on
| this kind of OSS projects.
|
| Making Postgres Wasm helped:
|
| - v86[0] to find a new bug
|
| - Providing a great deep-dive article that will trigger new ideas
| in the future
|
| - Showcase the possibilities of Wasm and how you can overcome the
| current challenges
|
| I really appreciate these projects are OSS :)
|
| Congratulations for the project!
|
| [0] https://github.com/copy/v86
| MuffinFlavored wrote:
| Could I hypothetically pass USB through to v86? Like libusb on
| the host into v86 on the guest? Or do USB over IP or something?
| simcop2387 wrote:
| Chrome supports a USB api that could potentially work for
| that,
| https://developer.chrome.com/docs/extensions/reference/usb/
|
| sadly though it's chrome only, not a standard of any kind (no
| other browsers support it that I know of). ESPHome can use it
| to program microcontrollers (along with the serial port
| support).
| OJFord wrote:
| Isn't that because it was explicitly rejected on grounds of
| security & privacy concerns, rather than 'oh if only others
| would be up with Google on implementing new features'?
| rjh29 wrote:
| > This API is part of the deprecated Chrome Apps platform.
| Learn more about migrating your app.
|
| Classic Google...
| lights0123 wrote:
| That's the old one. https://web.dev/usb/ is the supported
| one.
| MuffinFlavored wrote:
| What about this: https://wicg.github.io/webusb/
| burggraf wrote:
| I'm curious -- what's the intended use case for this? This is
| something we could try to add if gives some cool new
| functionality.
| Woberto wrote:
| Stephen Grider made a cool website[0] that allows people to make
| and use databases using PostgreSQL statements for his udemy
| course[1] about SQL. It's great for learning and testing and has
| been around a while so I feel like he should get some
| attention/credit too
|
| [0] pg-sql.com/
|
| [1] https://www.udemy.com/course/sql-and-postgresql/
| znpy wrote:
| I'd love to see udemy courses on postgresql administration
| beyond the basics... I'd love something to get you to, say, a
| junior-level dba. Focused on postgresql of course.
| yarg wrote:
| 30mb wouldn't be that bad if the browser was caching WASM
| dependencies.
|
| https://developer.mozilla.org/en-US/docs/WebAssembly/Caching...
| CSDude wrote:
| This is really cool. It'd also make a great teaching environment.
| jcuenod wrote:
| Hint: Don't use ctrl-w in their terminal. Turns out that has
| another meaning in Firefox...
|
| [EDIT]: I'm zero for three thus far. I give up.
| jlokier wrote:
| If you open the tab in its own window (i.e. no other tabs) _and
| also_ pin the tab, turns out ctrl-W doesn 't close the tab in
| Firefox.
|
| The key combo still appears to be intercepted, as the menu
| flashes but it doesn't close the tab. So I doubt you can use it
| in the terminal, but at least you won't lose your work.
|
| If you pin the tab but not in its own window, ctrl-W doesn't
| close the tab but it does switch to another tab.
|
| Source for the first point:
| https://www.reddit.com/r/firefox/comments/rs2bhn/comment/hqn...
| The rest is from me trying things out just now on MacOS, where
| really I used cmd-W, because ctrl-W doesn't do anything on
| MacOS. I'm assuming the corresponding behaviours will apply to
| ctrl-W on Windows and Linux, but you should test before relying
| on this.
| mdaniel wrote:
| That drives me _starkraving mad_ in AWS CloudShell, because
| muscle memory happens faster than thinking. I thought there was
| a web /dom method designed for gaming like "capture all input
| events" but maybe it has to be in full-screen to work or
| something
|
| The other one is pressing esc in some web ui if I've been
| vimming too recently and having it nuke whatever I've typed :-(
| (it was almost certainly in an Atlassian product, cause they're
| awesome like that)
| [deleted]
| r3tr0 wrote:
| Another implementation:
|
| https://xn--yv9h.fm/emu/?m=jbdf7ZqvfgR4dCizypXxMP
| burggraf wrote:
| This is cool! It's running debian and posgres 11. It's quite a
| bit larger (that's why we went with Buildroot, and frankly, we
| could do a lot better at trimming things down even more). But
| it's definitely a great implementation!
| r3tr0 wrote:
| yes. this is just a test dummy for something bigger I have.
| we should chat. Feel free to DM me.
| r3tr0 wrote:
| I originally went with buildroot too, but then I realized
| some things...
| burggraf wrote:
| Would love to chat! Love to get your ideas on this.
| r3tr0 wrote:
| what is the best way to contact you.
| burggraf wrote:
| You can email me at mark @ supabase dot io. (Yeah, I know
| there are probably robots that can parse that as valid
| email address, but I'm old-school :))
| r3tr0 wrote:
| got it.
| jd3 wrote:
| This is amazing! I used
| https://rextester.com/l/postgresql_online_compiler to rapidly
| iterate/prototype some complex jsonb queries for a project at
| work ~3 years ago, but it has since been locked behind a patreon
| paywall.
|
| While I've since switched to the native https://postgres.app/, it
| will be nice to be able to spin up a fresh postgres test db in
| the browser in the future.
| frankfrank13 wrote:
| One interesting use case that wasn't mentioned is edge computing.
| If you buy into Chrome-as-a-container that can run on an "edge"
| more easily than a data center, this is a step in that direction
| pistoriusp wrote:
| And those are very likely to have tcp sockets at some point in
| the future (or at least that is the hope!)
| tonetheman wrote:
| Funny thing is that Chrome used to have sockets. As an
| extension but they solidly worked. I wrote a socks proxy that
| ran in the browser. It worked really well.
| bluepnume wrote:
| Designed for browsers but... presumably can be run in Node? I'm
| thinking this would be perfect for tests, to avoid spinning up a
| standalone postgres server every time.
| tmpz22 wrote:
| I expect to see WASM tooling expose that Google's stewardship of
| Chromium has left many, many, features to be desired. You can
| blame it on the bureaucracy and legacy cruft of other companies
| like Microsoft and Apple, but at the end of the day there's just
| so much stuff you can't do with web apps (and oh wow just happens
| to align with Google's Ad and Cloud businesses).
| robertlagrant wrote:
| Can you list the features you mean that Firefox and Safari have
| that the Chromium project has left out because of Google's
| business interests? Be interesting to see.
| tmpz22 wrote:
| My comment doesn't assert Firefox or Safari are ahead on
| features. My comment is about Google/Chrome still being
| culpable as a market leader despite Apple (Safari) and
| Microsoft (IE legacy) likely slowing down browser development
| in general.
| robertlagrant wrote:
| I don't understand what you're comparing Chromium to if
| you're saying it doesn't have features for business
| reasons, when the other browsers also don't have those
| features.
| termcaps wrote:
| kaboomshebang wrote:
| The more I read about WASM the more magical it becomes...
| andersonmvd wrote:
| I was trying to find a way to make apps "data-leak resistant" and
| one step into this direction was to let the user store the data
| by bringing his/her own database. I even made a poc
| https://github.com/andersonDadario/byoda (explanation in the blog
| post link found on the readme) - but no user would manage his own
| database. Having a database in the browser opens more
| possibilities though. I will give it some thought. Looking
| forward to what else will be built on WASM.
| unilynx wrote:
| For that to work you won't even need a database in the user's
| browser - you could just store an encryption key as a cookie
| today which you use to encrypt all your serverside data when
| the user logs in
|
| The real problem is how you deal with the average user (who
| doesn't really backup properly) losing or crashing their device
| and thus their encryption key/data. You quickly end up with
| serverside storage and an email-based password reset again...
| andersonmvd wrote:
| It's complicated indeed. A simple "clear browsing data" would
| make him lose the data forever hehe. And if you keep the key
| in the server, it would be "hackable", which is what I was
| trying to avoid. I haven't looked at it but
| https://solid.mit.edu/ looked promising.
| jhd3 wrote:
| I was wondering if anyone had thought about using this to
| experiment with the planner.
|
| The engineering and support teams at Greenplum, a fork of
| Postgres, have a tool (minirepro[0]) which, given a sql query,
| can grab a minimal set of DDLs and the associated statistics for
| the tables involved in the query that can then be loaded into a
| "local" GPDB instance. Having the DDL and the statistics meant
| the team was able to debug issues in the optimizer (example [1]),
| without having access to a full set of data. This approach, if my
| understanding is correct, could be enabled in the browser with
| this Postgres WASM capability.
|
| [0] https://github.com/greenplum-
| db/gpdb/blob/6X_STABLE/gpMgmt/b...
|
| [1] https://github.com/greenplum-
| db/gpdb/issues/5740#issuecommen... (has an example output)
| pashkinelfe wrote:
| For me it's interesting can this work without the need of
| external proxy server which seems to be needed only to overcome
| browser connections limitations and nothing else. May there be
| some more "internal" way to implement/allow these connections?
|
| Overall this seems an inspiring thing. Thanks!
| pistoriusp wrote:
| Snaplet person here: Totally. It cannot, and it very likely
| will never be able to do that. Raw sockets were added, and
| subsequently removed, from Chromium.
|
| Do you mean, connect from 1 browser tab to another?
| burggraf wrote:
| Interestingly we can do this, sorta, but in Chrome only.
| Using arp you can open a network between multiple tabs of the
| same Chrome instance, putting each browser vm on a different
| private ip. I only see this useful, though, for testing
| clustering systems. There won't be any outside internet
| connectivity in a setup like this though.
| burggraf wrote:
| Supabase developer here.
|
| At this point, the proxy is necessary because all the major
| browsers block direct TCP/IP traffic. They allow websocket
| connections so that's how we're getting around it.
|
| There have been proposals to open up TCP/IP traffic but they've
| all been shot down so for the security implications.
| pistoriusp wrote:
| Peter from Snaplet here. A month ago I saw the CrunchyData post
| and wanted to play around with the code that made it happen, it
| wasn't OSS so I asked for help:
|
| > If anyone out there wants to work on an open source version of
| this full-time please reach out to me. [0]
|
| Paul reached out and we started working on it almost immediately.
| Check out the repo here: https://github.com/snaplet/postgres-wasm
|
| We have a blog post about some of the interesting technical
| challenges that we faced whilst building this:
| https://www.snaplet.dev/post/postgresql-in-the-browser
|
| Like most things, this is built on-top of the amazing open-source
| projects that made this possible, but special mention goes to
| v86.js and buildroot. We just glued it together.
|
| My hope is that we as a community can own this project and make
| PostgresQL, and the software that runs on it, accessible to a
| larger audience.
|
| ---
|
| [0] Request for collaboration:
| https://news.ycombinator.com/item?id=32500526
| justinclift wrote:
| Have you looked at compressing that ~16MB wasm file with
| something like Brotli?
|
| When I was actively doing stuff with wasm (~2019), Brotli was
| the best compression approach. eg 16MB uncompressed -> 2.4MB
| compressed
|
| https://github.com/golang/go/wiki/WebAssembly#reducing-the-s...
| pistoriusp wrote:
| Our demo is on Netlify, so I'm assuming that they've got gzip
| or something along those lines enabled.
|
| On my 50mbit/s (Germany :/) connection it's ~2 seconds
| stuaxo wrote:
| Nice, I wonder if anyone has got Django with postgres working
| in this environment?
| pistoriusp wrote:
| I don't see why it wouldn't work! You can exit pg and try it
| out :P
| sgt wrote:
| Didn't realize you guys are in SA. Hi from Stellenbosch. Cool
| to see more South Africans here and in the startup community in
| general.
| 101008 wrote:
| Now I only need Django WASM and I can distribute standalone
| applications :')
| pistoriusp wrote:
| Interesting, you're the second person to ask about this! Is
| there something in Django world that makes standalone
| application appealing?
| raitucarp wrote:
| Wow, the dream of decentralized web comes true, soon. With
| encryption and wasm, we are gonna conquer the goals toward true
| decentralized web.
| NoGravitas wrote:
| pistoriusp wrote:
| Oh man, sorry.
| nitsky wrote:
| This is a really exciting milestone for WASM! Has anyone tried
| compiling postgres for wasm32-wasi and implementing a wasi-
| compatible interface for the file system and other OS bits
| postgres needs? I think that would be a big improvement.
| burggraf wrote:
| Supabase developer here. I worked on this project with the guys
| over at Snaplet.
|
| One thing we are working on is putting postgres on an
| alternative filesystem using 9p. There's some really cool work
| by humphd that creates a filesystem inside IndexedDB[0]. We'd
| also like to maybe use the browser filesystem component to let
| you store the database on the host device in a path of your
| choosing. Not sure if these are possible yet, though.
|
| [0] https://humphd.github.io/browser-shell/
| nitsky wrote:
| Using IndexedDB, the forthcoming SQLite support in Chrome, or
| the browser filesystem components all might be good paths.
| Either way, what do you think about using WASI instead of
| Linux? I think it would reduce the overhead significantly.
| burggraf wrote:
| I'm not familiar with WASI but I'll take a look. We're
| using Buildroot now and it's nice and small, but we could
| make it smaller by optimizing the kernel. Now, if I only
| know how to optimize the kernel without breaking things :)
|
| Where's all those kernel hackers? Your help, we need. :)
| OOPMan wrote:
| This seems a step closer to the embeddable postgres I've wanted
| for ages.
| swyx wrote:
| why most it be postgres and not sqlite? just curious on what
| you were looking for out of it
| burggraf wrote:
| Why Postgres? (There's already a WASM version of SQLite, btw,
| and it's very cool.) We did this with Postgres for a number
| of reasons:
|
| 1. It's a slightly difficult task, and in doing this we hope
| to spur others to think about using WASM to run things they
| didn't think were possible before. Before Crunchy did this,
| nobody really knew this was possible. This project is a
| framework for you to port something new and exciting to run
| under WASM in the browser. What's that going to be?
|
| 2. We love Postgres. It's our favorite database and this tool
| gives us a quick and simple sandbox to try out new things
| that might mess up our production (or even dev) database. Got
| a crazy idea that might not work? Try it in the browser and
| if it doesn't work, refresh the page and start over.
|
| 3. My goal is to eventually have an entire version of
| Supabase running in the browser as a basic dev / experiment
| tool. This would make a great quick and easy way to try out
| Supabase, or even to do full scale development, after which
| you can migrate your data up to your staging or production
| databases.
| agustinbena wrote:
| I can not still agree to the usefulness to run this in a
| WebBrowser at 30MB. But for running in the backend over a wasm
| runtime, seems that the use case that make sense for the moment.
| rkagerer wrote:
| Aside from folks who already have Postgres infrastructure they
| want to replicate, what sort of applications would find this a
| better fit than using SQLite on the client side?
| aitchnyu wrote:
| I wish we could replace Docker for dev environments with asdf
| which examines your project and makes available language tooling
| for Ruby, Node etc and other tools with pinned versions. With
| this we could cd to our project and have a Postgres server ready
| for this project.
| iib wrote:
| I think `direnv` [1] may be closer to `asdf` than this. They
| also mention it, and the integration, `asdf-direnv` on their
| website [2].
|
| [1] https://direnv.net/ [2] https://asdf-
| vm.com/guide/introduction.html#direnv
| kiwicopple wrote:
| Hey HN, we're excited about this launch. This was a collaborative
| effort with the team at Snaplet [0].
|
| postgres-wasm is an embeddable Linux VM with Postgres installed,
| which runs inside a browser. It provides some neat features:
| persisting state to browser, restoring from pg_dump, logical
| replication from a remote database, etc.
|
| The idea was inspired by CrunchyData's HN post about a month ago
| [1]. We love the possibilities of Postgres+WASM, and so Supabase
| & Snaplet teamed up to create an open source version. The linked
| blog post explains the technical difficulties we encountered, and
| the architecture decisions we made.
|
| We're still working hard on this, but it's at a good "MVP" stage
| where you can run it yourself. Snaplet are working on a feature
| where you can drag-and-drop a snapshot into your browser to
| restore the state from any backup. Supabase are exploring ways we
| can run the entire Supabase stack inside the browser. You can
| find the Snaplet repo here [2], and the Supabase fork here [3].
| There's very little difference between these two, we just have a
| different browser UI.
|
| Both Supabase team and the Snaplet team will be in here
| commenting if you want to know anything else about the technical
| details.
|
| [0] Snaplet: https://www.snaplet.dev/
|
| [1] Crunchy post: https://news.ycombinator.com/item?id=32498435
|
| [2] Snaplet repo: https://github.com/snaplet/postgres-wasm
|
| [3] Supabase fork: https://github.com/supabase-
| community/postgres-wasm
| vineyardmike wrote:
| > an embeddable Linux VM with Postgres installed, which runs
| inside a browser.
|
| Wow! I feel like this is the lede. How much work was done
| supporting the VM and OS privatives (eg networking) vs PG
| specific work? I feel like a minimal Linux in the browser opens
| up a LOT more opportunities than just a database.
|
| When figma got bought out, a lot of articles were written about
| "where's the wasm applications", and I feel like throwing Linux
| into a browser really shows potential. One commenter already
| wondered if it could be used to compile microcontrollers (so
| creative, i now want that too), I wonder if it can be used
| similar to Repl.it, with packaging test environments.
|
| To be very, very clear, I would LOVE a write up about just the
| linux portion of this interesting project.
| kiwicopple wrote:
| > _How much work was done supporting the VM_
|
| All of the heavy lifting here is done by v86:
| https://github.com/copy/v86
|
| v86 can be used for a number of things besides Postgres -
| things like Repls or other entire applications are definitely
| achievable.
|
| Networking between Postgres and the internet was a lot of
| work, and Mark came up with a neat solution detailed in the
| blog post. This solution can be used for any other
| application. If you're looking to run a native application in
| the browser using v86, the repo & blog post is a good
| launching pad.
| Ericson2314 wrote:
| This is cool, but is also somewhat saddening too.
|
| The dream is a real wasm native postgres; in fact to get
| all of postgresql's cool shared mem proccess stuff and make
| it something like shared array buffer! The dream is also
| that WASI interface and new-school OS interfaces like
| memfd_create are increasingly aligned.
|
| Instead of rationalization our interfaces, however, it's
| just emulation layer on top of emulation layer, tech debt
| all the way down.
|
| ----
|
| I am not blaming you all in the slightest, to be clear.
| Obviously one needs to start somewhere. Just sighing at the
| state of things.
| gurjeet wrote:
| The links at the bottom are broken; presumably the post was
| written in Markdown, HN doesn't understand that.
| kiwicopple wrote:
| Thanks Gurjeet - fixed
| mockery wrote:
| _The first thing to point out is that our implementation isn 't
| pure WASM. We attempted to compile Postgres for WASM directly
| from source, but it was more complicated that we anticipated._
|
| _Crunchy 's HN post provided some hints about the approach they
| took, which was to virtualize a machine in the browser. We
| pursued this strategy too, settling on v86 which emulates an
| x86-compatible CPU and hardware in the browser._
|
| I'm out-of-domain but very curious about this part - it seems
| like a pretty extreme solution with a lot of possible downsides.
| Does this mean the "just compile native code to WASM" goal is
| still far off?
| fdr wrote:
| The problem is you need operating system features to run
| Postgres as-is, e.g. mapping memory, forking processes,
| manipulating files. What is missing is a WASM kernel that skips
| the x86 emulation but implements enough of the other stuff.
|
| For example, for just one of many hairy problems, consider that
| Postgres uses global variables in each backend for backend-
| local state (global state as such is in shared memory). How
| does this look in assembly, accounting for both the kernel and
| userspace components? This is the problem.
|
| A general way to convey this is: the more system calls a piece
| of software uses, the more difficult a WASM target without
| architecture emulation becomes. And Postgres doesn't even
| obligate that many obscure ones.
| mockery wrote:
| Thanks, those are specific requirements I could definitely
| see WASM struggling to meet.
|
| In my experience in a large+mature enough codebase
| (particularly one that is already multi-platform, like
| Postgres appears to be) many of those requirements are
| wrapped in an abstraction layer to allow targeting new
| platforms, but some requirements (like memory mapping) could
| definitely be dealbreakers if the target platform doesn't
| naturally support them.
|
| This solution still seems awfully complex (and probably not
| very efficient) but I certainly see why it's probably the
| "easiest" option.
| fdr wrote:
| I suppose, Postgres is portable, but it's portable to
| multi-tasking operating systems with virtual memory (which
| puts in a rather broad category of programs). This goes
| beyond wrapping how various system calls work on various
| platforms, but rather changing how accesses are generated,
| e.g. so backend 1 sees memory location 1 for its global
| field, backend 2 sees memory location 2 for that same
| global variable etc. Unlike functions that are frequently
| wrapped, there is no error code (save ones generated by a
| processor, e.g. segfault or bus error) or function called
| for loading an address.
|
| Long story short, I think the need to bypass MMU hardware
| emulation would prove among the most difficult problems. It
| will probably require assistance from the compiler, I don't
| know enough about WASM to guess how mature such relocations
| would be.
| burggraf wrote:
| Supabase developer here. I've tried compiling directly to WASM,
| but it did not go well. As I recall, there were features used
| by PostgreSQL that WASM didn't support yet. This is definitely
| something we'll revisit though, especially as WASM matures!
| danielvaughn wrote:
| Very impressive technically, but what are the practical use cases
| for this? We have IndexedDB and WebSQL, so apart from the other
| comments mentioning using it in tutorials, I'm not sure what the
| value add is.
|
| edit: as always, I should read the whole article first. The idea
| of using it as a dev environment is very cool.
| smallerfish wrote:
| There is room for serverless applications with a sane and
| reasonably complete data storage interface (which localstorage
| and indexeddb are not.) You could offer users the ability to
| backup data locally using the filesystem api (which is
| reasonably usable), or perhaps sync data using p2p using webrtc
| (which is not as usable as it should be).
| kiwicopple wrote:
| There are some things that are uniquely beneficial for Postgres
| beyond the data storage - the main one I'm excited about is
| using PostGIS inside the browswer.
|
| OSM + PostGIS in the browser has the potential to do for Maps,
| what Figma's WASM approach did for design.
| dahfizz wrote:
| But why is it relevant that PostGIS runs inside the browser?
| kiwicopple wrote:
| It's only relevant in that it "extends" the browser with a
| set of capabilities that it doesn't already have. You could
| also run everything via hosted database, but there may be
| situations where that is less-optimal than downloading the
| dataset once and leveraging the "offline compute" of a
| browser tab (for example, applications like ArcGIS Online)
| jdmdmdmdmd wrote:
| >We have [...] WebSQL
|
| Not any more - iirc it was depreciated.
|
| In the case of IndexedDB, I haven't looked into it, but Mozilla
| has the following to say about it:
|
| >Note: IndexedDB API is powerful, but may seem too complicated
| for simple cases. If you'd prefer a simple API, try libraries
| in See also section that make IndexedDB more programmer-
| friendly.[0]
|
| I suppose this project could make development easier by
| allowing developers to share server-side code? And it has the
| benefit of already having a large userbase.
|
| [0]https://developer.mozilla.org/en-
| US/docs/Web/API/IndexedDB_A...
| danielvaughn wrote:
| Oh wow, I never knew WebSQL was deprecated. And yeah
| IndexedDB is pretty complicated, I've tried working with the
| API a few times but always abandoned it for simpler
| solutions.
| bearjaws wrote:
| The use cases here are going to be really wide spread in my
| opinion, just a few ideas off the cuff. Obviously the 30mb size
| means it won't really be for regular consumer apps, but for
| enterprise or specific tasks it can make a lot sense.
|
| 1. Training websites
|
| 2. Interview challenges involving SQL
|
| 3. Client side tooling that loads data into your local machine
| and displays into a SaaS web app without the SaaS app ever having
| your data
|
| Appreciate the hard work from Supabase and Snaplet on this!
| alex_duf wrote:
| I can really appreciate the fun and technical challenge of
| running postgres in a browser. However the use cases are
| extremely far fetched.
|
| 1. training website: you can use a hosted PG, or use a sqlite
| wasm
|
| 2. same as above
|
| 3. if the use case is being offline, then the web browser isn't
| very relevant. If the use case is to avoid a load on the
| server, the sqlite in wasm will be just fine.
|
| It's only if you go into triggers and such that it might start
| being relevant, but then I'd start seriously questioning what
| on earth are you trying to do :D
|
| All of that to say: well done to the team that has done it,
| really fun and interesting work, I just can't see the use from
| where I stand.
| Bedon292 wrote:
| It is awesome to be able to do things isolated client side
| and not have to deal with permissions and resources for
| something like a training website. Which is all stuff you
| would have to deal with for a hosted version.
|
| And there are plenty of reasons why you may want to use PG
| over sqlite. Especially if you are trying to mimic a
| production environment which is PG. Personally I only ever
| use PG, and never have a reason to use sqlite.
| kiwicopple wrote:
| > _training website: you can use a hosted PG, or use a sqlite
| wasm_
|
| from a supabase POV (which is in the business of hosting
| Postgres databases), we will definitely be using this for
| training/tutorials. We have several thousand visitors to our
| docs every day, and hosting a database for every one of them
| is expensive.
|
| We can now provide a fresh database for every user, and they
| can "save/restore" it with the click of a button is huge.
|
| > _use case is being offline_
|
| The offline use-case is definitely far-fetched in the current
| iteration. but that's the beauty of technology - something
| that seems impossible today can be mainstream in a decade.
| ASalazarMX wrote:
| Without seeing the marketing, I think running a full RDBMS
| inside your browser is not a great idea. Just idling it becomes
| my most CPU intensive Firefox tab, out of dozens, according to
| about:performance.
|
| I shudder what performance a full-fledged application would
| demand. I know some people will embed this on an Electron app,
| for double the fun.
| burggraf wrote:
| Supabase developer here.
|
| I've used this to move data from a live Supabase database down
| to the browser for testing and playing around with things in a
| "sandbox" environment. Then I save snapshots along the way in
| case I mess things up.
|
| To move a table over from my Supabase-hosted postgres instance
| to the browser, I just exit out of psql and run something like
| this:
|
| pg_dump --clean --if-exists --quote-all-identifiers -t my_table
| -h db.xxxxx.supabase.co -U postgres | psql -U postgres
|
| Keep in mind if you try something like this, our proxy is rate
| limited for now to prevent abuse, so it might not be super
| fast. It's easy to remove rate limiting at the proxy, though.
| alex_duf wrote:
| Correct me if I'm wrong, but given your profile (I assume
| someone in the tech world), nothing stopping you from doing
| all of the above with a local pg. If installing is annoying
| you could run it in docker.
| burggraf wrote:
| You're absolutely right, you can use a local pg. This just
| makes it easier for me, as it's sort of a "sandbox"
| environment and I can easily take snapshots to do A/B
| testing or roll things back. I can also send a snapshot to
| a coworker so they can get my entire environment with all
| my data in a few seconds.
| aaronax wrote:
| People really do want to make the browser be the new OS.
| TheRealPomax wrote:
| But now instead of the annoyance of installing pg, you have
| the annoyance of installing docker. and then writing a
| dockerfile. And then bootstrapping docker. etc. etc. =)
| nicoburns wrote:
| Installing pg is hardly annoying. `brew install
| postgres`. Done.
| TheRealPomax wrote:
| Replied to the wrong person, but I'll take that bait
| anyway: "Ahahahaha, no".
|
| That only works if you live a blissful "all my hosted pg
| instances use the exact same version" world, which I've
| never seen be the case for even moderately sized
| projects. You're going to need multiple Postgres installs
| if you're going to need pg_dump/pg_restore, which you
| probably are.
|
| (How you solve that problem, of course, is not a one-
| size-fits-all, and Docker may be the answer... or it may
| not)
| burggraf wrote:
| Also, in case you're curious, PostgreSQL logical replication to
| the browser wasm instance DOES work. I've done it. :)
| NoGravitas wrote:
| > Obviously the 30mb size means it won't really be for regular
| consumer apps
|
| You know that it will end up being used for regular consumer
| apps. And once everyone is doing it, regular web pages being
| over 30MB and including an enterprise-grade SQL server engine
| will simply be accepted as normal, and everyone not doing it is
| a luddite.
| aaronax wrote:
| How long until WASM things become "installable" so that other
| websites can use the same egregious 30MB things?
| member_mini wrote:
| > 3. Client side tooling that loads data into your local
| machine and displays into a SaaS web app without the SaaS app
| ever having your data
|
| Member mini mongo?
|
| Supabase will be Meteor in no time.
| pier25 wrote:
| > _Supabase will be Meteor in no time._
|
| As in they will crash and burn?
| member_mini wrote:
| Nah I think Supabase will do well. Meteor also did well all
| things considered. My point is really that they are
| revisiting all the same technology and product decisions
| and coming to the same conclusions.
|
| There's only one way to architect a PaaS. Next.js and
| Vercel's offerings are, essentially, also the same.
|
| The real risk is having one person build it all. They have
| a team but not really. Personally I believe that's a good
| risk to take.
|
| But I think it will take a powerful psychological toll to
| operate this way, having to pretend to have a team (because
| investors like teams and not solo founders), having to
| pretend this isn't Meteor (because investors don't like
| being reminded of "losers"), etc. etc.
|
| Like downvote random Internet comments all you want, but
| actually I think it's a great idea to have one person do
| "Better Meteor," it's not my fault investors don't.
| kiwicopple wrote:
| > The real risk is having one person build it all
|
| > having to pretend to have a team
|
| In case you're talking specifically about supabase here,
| we're a full team: https://supabase.com/humans.txt
| member_mini wrote:
| I just want to say what you've built is really great and
| works really well.
| tmpz22 wrote:
| As someone who (unfortunately) used Meteor in the past I
| disagree. IMO from a dev perspective Meteor was just a poorly
| implemented promise at 0-effort real-time functionality on
| top of a database you were at the time (~2015) already
| interested in or using. It compounded all the problems of
| MongoDB with a non-perfect abstraction and javascript
| framework.
|
| Whatever Postgres in WASM ends up being used for there's no
| way it repeats all those circumstances - at minimum Postgres
| is just a more appropriate tool then MongoDB circa 2015.
| mrtksn wrote:
| IMHO the prime use case for all these WASM stuff is going to be
| platform independence. Web browsers are not that interesting
| because for regular use they already have ballooning resource
| use issues and making web apps even more resource intensive is
| not exactly inspiring, HOWEVER the web technologies are the
| only true multi-platform solution we have and it makes sense to
| use it to make everything with it and everything instantly
| becomes multi-platform.
|
| What I suspect may happen is, the rise of web browsers of a 3rd
| kind where these are not really for browsing the web but
| running code written for native domains. So instead of browsing
| web of linked text, we can have a web of algorithms to process
| data and requests.
| r3trohack3r wrote:
| Isn't that Deno and Node.js?
| mytherin wrote:
| Looks really nice, but I'm really curious about what hurdles you
| encountered in making a native WASM version of Postgres. It seems
| that both the performance and binary size suffer immensely from
| the VM running underneath. For example, if I compare to DuckDB-
| WASM [1] there is an immense difference in load time and query
| execution speed.
|
| Are there future plans at creating a native WASM version of
| Postgres? Making it run many times faster would certainly open up
| a lot more use cases.
|
| (Disclaimer: I work on DuckDB, but have not worked on the WASM
| version myself)
|
| [1] https://shell.duckdb.org
| kiwicopple wrote:
| congrats to the DuckDB team for their WASM version. I've been
| following it from afar and it's very impressive.
|
| > what hurdles you encountered in making a native WASM
|
| I'm sure Mark & Peter can jump in with specifics but mostly it
| was due to complexity - there it probably _can_ be done it 's
| just that we took the path of least resistance.
|
| > Are there future plans at creating a native WASM version of
| Postgres
|
| We'd like that. If anyone would like to collaborate with
| Supabase + Snaplet to create a more "native WASM" version then
| please reach out
| burggraf wrote:
| Supabase developer here. Yes we'd love to collaborate on a
| native WASM version. I tried this first before starting on
| this project and I didn't get very far. I'll have to go back
| to my tests on that, but it seemed like WASM wasn't yet
| capable of compiling Postgres due do some libraries not being
| supported. Hopefully we can find workarounds or push WASM
| into a new era of compatibility :)
| pistoriusp wrote:
| Snaplet person here: Having a native WASM version would be a
| huge win for everyone, especially is storage is decoupled from
| compute!
|
| At the moment the CPU and memory snapshot of the VM (with
| Postgres) is 12 MB, and subsequent reloads are cached. So yeah,
| not the worst, but not great.
|
| An optimization is that we're using 9P filesystem. So accessing
| anything on disk is lazily loaded over the network.
|
| > Are there future plans at creating a native WASM version of
| Postgres?
|
| Yup! I think that should be the goal, and we (Supabase &
| Snaplet) would be very happy to work with anyone that wants to
| build towards that.
| chrisjc wrote:
| > especially is storage is decoupled from compute!
|
| This would be amazing! I can imagine a situation where
| external tables are managed by some MPP, and a WASM compute
| engine (Postgres, DuckDB, etc) would be able to at least read
| subsets/partitions of the full external table.
|
| I wonder if the work required to make a native WASM Postgres
| would have to be split up into efforts for row-based vs
| column-based. Selfishly, I would love to have access to a
| column-based version first.
| pistoriusp wrote:
| We actually kinda see parts of this in
| https://github.com/superfly/litefs, albeit for SQLite.
|
| And here we see some ideas forming around "pluggable
| storage for PostgresQL": https://wiki.postgresql.org/wiki/F
| uture_of_storage#Pluggable...
|
| Seriously! If any of this sounds interesting to build,
| reach out, and we'll make it happen!
| fcoury wrote:
| Interesting. I have written a translation layer for the MongoDB
| wire protocol [0] that persists to PostgreSQL. I am wondering if
| I could make this layer also compile to wasm and run in the
| browser.
|
| ---
|
| [0] http://oxidedb.com
| burggraf wrote:
| Wow, this is awesome! So as I understand it, this uses Postgres
| and JSON fields to emulate the features of MongoDB, sort of as
| an abstraction?
| fcoury wrote:
| Thank you! It's exactly that: basically you don't need to
| keep an additional MongoDB instance if you already use and
| maintain a Postgres server somewhere.
|
| I translate the JSON-based query interface into the
| corresponding SQL statements, leveraging the excellent JSON
| support that PostgreSQL offers.
| vivekin wrote:
| This is very cool. This will open up a lot of different use
| cases.Congratulations!
| mradek wrote:
| I haven't messed around with wasm but I like postgres a lot. Just
| curious what are some use cases for this? Also not a supabase
| user even tho it looks cool.
|
| In my case, I use postgres along with postGIS for some of my
| services. Could this allow me to have some parity where the
| client can have a 1:1 table but populated and kept up-to-date
| with their own data to cut down on making network requests?
| spiderice wrote:
| There is a "why" section in the blog post that gives a number
| of examples. Namely:
|
| - Documentation: for tutorials and demos.
|
| - Offline data: running it in the browser for an offline cache,
| similar to sql.js or absurd-sql.
|
| - Offline data analysis: using it in a dashboard for offline
| data analysis and charts.
|
| - Testing: testing PostgresSQL functions, triggers, data
| modeling, logical replication, etc.
|
| - Dev environments: use it as a development environment -- pull
| data from production or push new data, functions, triggers,
| views up to production.
|
| - Snapshots: create a test version of your database with sample
| data, then take a snapshot to send to other developers.
|
| - Support: send snapshots of your database to support personnel
| to demonstrate an issue you're having.
|
| edit: formatting
| xani_ wrote:
| Huh, thought for a second that was a WASM support for stored
| procedures or something and was excited for a second
| supamichelp wrote:
| Well it's not what you were expecting, but your idea sounds
| interesting too! Maybe you could muster the same open source
| forces to make it a reality?
| kiwicopple wrote:
| I believe the team at Wasmer have worked on something like
| this:
|
| https://github.com/wasmerio/wasmer-postgres
|
| It only works for PG10, but I can't imagine it will take much
| effort to bring it up to the latest version
| michaelmior wrote:
| This was also what I was expecting. Another reply to your
| comment mentioned Wasmer, which I found after a quick Google
| search. Unfortunately it looks like the last commit was almost
| 2 years ago and there's an issue where people are inquiring
| about the status of the project that hasn't seen a reply from
| the maintainers in about the same amount of time.
| yangosoft wrote:
| Great! Really impressive.
|
| It seems that WASM could be an alternative container solution.
|
| This approach is like an intermediate step before recompiling it
| to wasm.
| encryptluks2 wrote:
| One that performs way worse then existing container solutions
| but that will of course be used to distribute binary blobs.
| yangosoft wrote:
| Agree, it is much slower.
|
| I would be interesting to see a comparison between a full
| recompiled version running on top of a wasm runtime vs some
| container solution but seems they found a lot of problems
| recompiling it.
| burggraf wrote:
| I love how you guys are already pushing us to think about
| performance issues on this project. Hacker news really
| pushes technology to the next level!
| encryptluks2 wrote:
| A good comparison might be something like Flatpak or
| gVisor.
| javajosh wrote:
| This is very cool. An important area of ignorance (for me) is how
| well browsers do with local data. A nightmare scenario is that a
| user happily uses your app, stores data locally, and then Chrome
| decides they don't need localstorage anymore. What's the status
| on local data reliability in browsers?
| Joel_Mckay wrote:
| Wise-men know better than to integrate google stuff into
| mission critical systems. Their behavior is purely governed by
| the information consolidation facet of any product or service
| they offer. Thus, given they already sample user traffic on 80%
| of the web, the company will continue to cull unpopular
| projects at astonishingly high rates.
|
| I've lost count of the number of projects I saw get burned by
| this common mistake.
___________________________________________________________________
(page generated 2022-10-03 23:00 UTC)