[HN Gopher] Show HN: PocketBase - Open Source realtime backend i...
___________________________________________________________________
Show HN: PocketBase - Open Source realtime backend in one file
Author : randomwebdev
Score : 383 points
Date : 2022-07-07 12:45 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| dugmartin wrote:
| Looks very nice.
|
| One thing I noticed is you are using server sent events for
| subscriptions. SSE have a lot of nice properties but do have one
| nasty limitation (from https://developer.mozilla.org/en-
| US/docs/Web/API/Server-sent...):
|
| "Warning: When not used over HTTP/2, SSE suffers from a
| limitation to the maximum number of open connections, which can
| be especially painful when opening multiple tabs, as the limit is
| per browser and is set to a very low number (6). The issue has
| been marked as "Won't fix" in Chrome and Firefox. This limit is
| per browser + domain, which means that you can open 6 SSE
| connections across all of the tabs to www.example1.com and
| another 6 SSE connections to www.example2.com (per
| Stackoverflow). When using HTTP/2, the maximum number of
| simultaneous HTTP streams is negotiated between the server and
| the client (defaults to 100)."
|
| This can lead to some weird bug reports from users.
| randomwebdev wrote:
| Yes, I'm aware of this limitation, but PocketBase uses HTTP2
| when combined with the --https flag (this is by go's default
| net/http implementation) so it shouldn't be too much of a
| concern.
| antholeole wrote:
| First time I've heard about the 6 connection limit.
|
| I'm not fully understanding the response - are you saying
| that the limit is not imposed if you use https? Or am I
| reading wrong?
| randomwebdev wrote:
| Go's default `net/http` package will serve HTTP2 when a
| https configuration is provided (also most web browsers
| don't support h2c, aka. HTTP2 without tls).
|
| The default limit of ~100 connections should be more than
| enough for most applications (additionally, the JS SDK
| client maintains a single SSE connection for a page no
| matter to how many things the user has subscribed, so that
| also helps).
| futhey wrote:
| HTTP/1.1 has a limit, set by each individual browser
| vendor, for the maximum number of connections between a
| client and a unique server domain. So, if you exceed 6
| simultaneous connections to that server (across multiple
| tabs and windows), it will move the request to a stalled
| state (like a queue) until one request is completed.
|
| Best solution today is to move to http2 on your server --
| which has an SSL (TLS 1.2) requirement.
|
| Looks like pocketbase implements this. If you use another
| server, like nginx, you have to enable this for each site.
| pbronez wrote:
| Sweet. How tightly is this coupled to SQLite? I'd like to embed
| Dolt [0] instead, specifically for the Dolthub-backed
| collaboration model.
|
| [0] https://github.com/dolthub/dolt
| randomwebdev wrote:
| Other than some json functions, PocketBase doesn't use too much
| SQLite specific features, but I don't plan adding support for
| other databases at the moment.
|
| Please also note that PocketBase is designed to be deployed
| only on a single server, so it cannot scale horizontally out of
| the box. Supabase/Nhost are better suited for that.
| catchmeifyoucan wrote:
| This looks great! Does this have support for offline usage on the
| client?
| jitl wrote:
| This looks really great. Have you thought about adding
| Operational Transform or CRDT primitives to your offering?
| Realtime subscriptions are nice - but they're even better with
| realtime collaboration. Because you're using SQLite directly,
| it's easy(er) for you to build in efficient CRDT mutations to
| your database rows because SQLite's BLOB column supports
| incremental IO: https://www.sqlite.org/c3ref/blob_open.html
|
| This could allow you to save a CRDT update operation without
| needing to read the entire CRDT blob into memory, parse it, and
| then save it all back to disk.
| ptudan wrote:
| This looks awesome! I will definitely give it a try. Right now I
| use postgres in a docker compose but its definitely a bit heavier
| for single machine prototyping.
|
| One thing that I like about my setup though is that there's a
| clear migration path to horizontally scale. Do you have a
| recommended way to move past pocketbase should an app get to that
| point?
| randomwebdev wrote:
| I guess there will be differences from use case to use case,
| but in general migrating from PocketBase to a more common
| horizontal supported stack is not different from rewriting your
| application.
|
| Internally, each Collection creates a standard SQLite table
| that holds the collection records, so migrating the data
| structure shouldn't be too troublesome. The only thing that may
| prove difficult to migrate could be reimplementing the access
| rules and filters.
|
| But in my opinion, when your application reaches that level of
| growth requiring multiple servers and services, your business
| use cases mostly likely will have changed several times already
| from your initial idea.
| codeptualize wrote:
| Looks amazing! Love the idea of a Firebase/Supabase like thing I
| can easily run locally and deploy as well.
|
| Also props for the dashboard design, looks and works very nice!
| 1xdevloper wrote:
| This looks perfect for storing the internal data of my side
| project! Love the website design too.
| oakesm9 wrote:
| How are the real time subscriptions with SQLite achieved? Is this
| a feature of SQLite itself or something that PocketBase itself
| implements on top of it?
| randomwebdev wrote:
| It's implemented in the application, it's not a feature of
| SQLite, but its very simple and robust - when a record is
| created/updated/deleted I'm broadcasting the change to all
| subscribers that are allowed to receive it (the access control
| is based on the colleciton's list and view rules).
|
| The realtime api implementation could be found at
| https://github.com/pocketbase/pocketbase/blob/master/apis/re...
| Norgie93 wrote:
| Reading through the code/docs it looks like you can subscribe
| to individual records (database rows) or collections
| (tables), is this correct? In your Records.getList method,
| you have filter functionality that would be nice to subscribe
| to (ie. subscribe to changes in "demo" collection where
| totalComments > 10). Subscribing to a specific result set
| seems like it could scale better than notifying subscribers
| of all table changes, which then get filtered in the
| subscribe() callback func in the SDK. Very cool work!
| randomwebdev wrote:
| Yes, you can subscribe to individual records or
| collections.
|
| User defined filters for the subscriptions are not
| supported, but it's a good idea and I may consider it in
| the future.
|
| Currently the subscriptions are filtered through the
| collection's list and view rules - they also act as "admin
| level filters", aka. filters that are always applied and
| regular users cannot modify (getList user defined filters
| are only appended to the search query together with the
| admin defined filters).
| oakesm9 wrote:
| That makes a lot of sense. Thanks for the link. I've not
| worked with Go before so I couldn't find exactly where the
| implementation was.
|
| This looks like a very interesting project and great for some
| of the small client projects which I currently use Firebase.
| I'll be sure to take a proper look in the future!
| codegeek wrote:
| Wow, super impressed. I have always wanted to build something
| like this where everything is embedded in 1 single file and I
| knew Go could be a great way to do it but always wondered how to
| handle the database part. Along comes SQLlite.
|
| One feedback which will probably deviate from everything in 1
| file concept but adding an option to upload files to an S3 URL
| will be awesome. Perhaps a pull request ? I am thinking of use
| cases like document management using this and if I can have the
| actual files in S3 (or S3 compatible), that would be a huge plus.
|
| EDIT: I spoke too soon. You already have an option for S3 as I
| tested the demo. Very cool.
|
| How much work went into this btw ? The github, documentation
| everything is well done. I envy work like this :)
| endofreach wrote:
| Uploaded files must deviate from ,,everything in one file"
| regardless of using S3, right? How (and why) would you store
| user uploads within the binary?
| codegeek wrote:
| yea it was a silly question. It does allow S3 anyway so we
| are good.
| randomwebdev wrote:
| Uploaded files and your database are stored outside of the
| binary.
| randomwebdev wrote:
| By default PocketBase uses the local filesystem, but you could
| also configure a S3 compatible storage (AWS, DigitalOcean,
| Vultr, etc.) from "Settings > Files Storage > Use S3 storage".
|
| I've started working on the project ~ 3-4 months ago. The last
| 4 weeks I've taken a longer break from my day-to-day job so
| that I could finalize the first release.
| randomwebdev wrote:
| PocketBase is a small open source project that I've been working
| for the last couple of months.
|
| You could think of PocketBase as a lightweight Firebase/Supabase
| alternative.
|
| In short, it is an open source Go backend (framework and app)
| consisting of:
|
| - embedded database (SQLite) with realtime subscriptions\
|
| - backed-in files and users management
|
| - convenient Admin dashboard UI
|
| - simple REST-ish API
|
| And all of this compiles to a single portable binary.
|
| It's still a little rough around the edges, but you could give it
| a try and share your feedback.
| masukomi wrote:
| feedback: re the site:
|
| i have no idea what you mean by a "backend". If it's got a
| "admin dashboard" it's got a UI which means it's got a front-
| end which means it's not a backend. I'm assuming "go backend"
| isn't specific to go but just indicating that it's written in
| go.
|
| it's got a example code block but doesn't explain the point of
| it. Why would an existing framework with a UI and everything
| need me to write a `main` method? Why doesn't it have one
| already? If it doesn't have one why doesn't it. If it does have
| one why am i not using it? My thinking here is that there are
| many existing patterns for registering plugins / extended
| functionality that don't require altering the main method of
| the thing you're hooking into. Why aren't you using that? Is
| this maybe intended to be like... a starter template for a
| Sinatra like thing with a pre-defined set of functionalty?
| Kind-of like how Django sets up model administration UIs for
| you?
|
| re your summary here and the site:
|
| i have no idea what "backed-in files" means and I've been doing
| web-dev for decades.
|
| "convenient Admin dashboard UI" ... to admin / do what?
|
| "simple REST-ish API" a) why "ish" and not actually REST b) ...
| that let's me do what?
|
| --- in summary.... this looks useful but i don't get what
| you're actually offering or what it's intended usage scenario
| is. This is compounded by the fact that you're using a very
| atypical definition of one word "backend" and a very uncommon
| term "backed-in files"
| sangnoir wrote:
| It's a lightweight replacement for Firebase. I That's why it
| is described as a backend, because it's primary feature is to
| act as a data store. If don't understand what a project does,
| and you're unfamiliar with the comparisons given (Firebase,
| Supabase), the right thing to do would bw to look up the
| comparisons.
| masukomi wrote:
| I know you have user management, a database, and do
| something with files. As noted before the "backed-in files"
| isn't a commonly used phrase so i'm left with 2 pieces of
| meaningful info, neither of which is particularly notable
| because every multi-user app has both of those.
|
| Firebase gives me a list of use cases
| https://firebase.google.com/use-cases and documentation
| whose left nav gives me a good insight into the high-level
| functionality it provides "authentication, storage,
| hosting, security rules, extensions, machine learning, etc"
|
| I think it's safe to assume you don't provide all of those,
| but i don't know what subset you DO provide. As a result,
| if i give you the benefit of the doubt, telling me it's a
| "replacement for firebase" is not meaningful because i
| don't know what parts it can replace unless it's just the
| few listed in which case it's absolutely _not_ a
| replacement.
| semireg wrote:
| I'm familiar with firebase and an electron developer and
| I still don't understand what this project does. I wish I
| did, because it seems it'd be interesting to me. Even the
| app that uses this (Presentator) is described in a way
| that leaves me grasping for "bit what does it DO?"
| resoluteteeth wrote:
| There is a bunch of software in this space like supabase,
| flat backend, and appwrite that attempts to be a generic
| self hosted backend for apps in a manner similar to
| firebase by providing data storage and authentication
| with a rest API in an integrated fashion, usually with a
| built in admin interface for schema design, user
| management, etc.
|
| You can also think of them as software like postgrest but
| with the addition of authentication and other app
| oriented features, and with a JavaScript client library
| out if the box.
|
| This program is doing the same thing in a more
| lightweight way as a single executable using sqlite.
|
| The idea is that by using this type of generic backend
| software for your database and authentication, you can
| basically just write the remainder of a typical app or
| webapp entirely as client side/frontend software, which
| is something that you can also do with firebase, but this
| allows you to easily self host it without having to
| create your own custom backend software.
| semireg wrote:
| Ok, so it's a self-hosted CRUD boilerplate with auth
| written in Go? Or a self-hosted Notion but with less
| view/UI customization out of the box?
| resoluteteeth wrote:
| > Ok, so it's a self-hosted CRUD boilerplate with auth
| written in Go? Or a self-hosted Notion but with less
| view/UI customization out of the box?
|
| I guess you could describe it as "CRUD boilerplate"
| although that could also make it sound like the main
| purpose is to provide database table/form views like
| Microsoft Access which isn't really accurate.
|
| It provides a database rest API like PostgREST with
| authentication so that you can write whatever frontend
| you want for it.
|
| Since you say that you're familiar with firebase, imagine
| you're writing an android app using react native or
| something that uses firebase as the backend to store
| users' posts (or whatever they are).
|
| The idea is that if you want to self host it instead of
| using firebase, you just run this type of software on
| your server and it will provide the basic functions that
| any client side app would need on the backend: storage
| for users' data as well as handling accounts.
|
| I don't think it's really that complicated; it's
| basically just an interlace to a database, but if you
| look at something like PostgREST that automatically
| creates a rest api for a database, you would probably
| think "wow, I could almost create an app just using this
| as the backend without having to create my own backend,
| but I would have to handle authentication/registration
| myself," so these types of platforms just go a little
| further and handle that as well.
| semireg wrote:
| Gotcha. Thank you for the explanation.
| sangnoir wrote:
| It can be used as a CRUD boilerplate for Go projects
| (where you extend it), _alternatively_ you may use it as
| a standalone REST(-ish) API backend that integrates auth
| and storing & retrieving state from some other app in
| any language. It's similar to DjangoREST which can bw
| used in both modes (as a base for a project written in
| the same language, or purely as data layer interfaced via
| REST)
| sangnoir wrote:
| I'm not OP; I wasn't aware of the project until I read
| the HN headline.
|
| "Lightweight replacement" also implies missing features,
| IMO. This project has - at the minimum - authentication,
| storage and hosting in a single binary that can be self-
| hosted. This absolutely can replace Firebase for _some_
| use-cases (e.g. on my LAN /homelab)
|
| Aside: your tone is weirdly - and needlessly combative.
| If you see no value in this project to yourself, that is
| fine, but it appears to be making you angry for some
| unfathomable reason.
| iampims wrote:
| You haven't tried really hard:
|
| > PocketBase could be used as a standalone app or as a Go
| framework/toolkit that enables you to build your own custom
| app specific business logic and still have a single portable
| executable at the end.
| masukomi wrote:
| if it's used a standalone app then what functionality is it
| providing beyond "user admin". Like, what problem does it
| solve in that use case?
|
| > a Go framework/toolkit that enables you to build your own
| custom app specific business logic
|
| that definition applies to literally every framework and
| toolkit. It tells me nothing.
| rpdillon wrote:
| > and still have a single portable executable at the end.
|
| You're quoting selectively in a way that elides
| meaningful parts of the sentence, and then stating that
| the description lacks meaning.
| resoluteteeth wrote:
| It provides an authenticated rest API (with a premade
| JavaScript client library) to an sqlite database, so you
| can store users' data similarly to how you might use
| firebase.
| Pr0ject217 wrote:
| This looks cool.
| Silica6149 wrote:
| Nice!
|
| It looks like you put a lot of work into this, what was the
| motivation behind this project? Did you not like the existing
| solutions available?
| randomwebdev wrote:
| Thanks! It was initially created to be the backend for the
| next version of my other open source project - Presentator v3
| (https://github.com/presentator/presentator).
|
| You could read more at
| https://github.com/presentator/presentator/issues/183, but to
| summarize - I wanted something that could be self hosted with
| almost no additional setup and no dependency on 3rd party
| services.
| swyx wrote:
| feedback: the "in one file" marketing confused me for a bit
| until i realized you meant "as a single binary". i would say
| that instead.
| mr-karan wrote:
| This is quite a cool project! Congrats on the public release.
| hernantz wrote:
| Very nice, whats planned next?
| giancarlostoro wrote:
| I'm confused about one detail:
|
| Does it render pages automatically based on Schema? I see your
| really simple example under examples, and its so minimal I want
| to think you just get all the tables and render views from
| that. Am I crazy? If so this is really neat.
|
| Did you ever consider using something like GORM? Because I
| could see this working with other databases and being extremely
| useful. I use Django to build really quick CRUD interfaces, but
| sometimes I need to be able to customize it more, and the admin
| UI is not as extensible as I would normally want, they even
| suggest you just build custom pages if its not doing what you
| need, which is a bummer because Django Admin covers the
| majority of my use cases.
| randomwebdev wrote:
| "Collections" stores a single records table meta data (eg.
| name, fields, validators, etc.), so when you create a new
| collection a new related table will be created (see https://g
| ithub.com/pocketbase/pocketbase/blob/3d07f0211dc747...).
|
| The admin UI just shows the collections through the web api
| (https://pocketbase.io/docs/api-collections/).
|
| I actually started with GORM, but its too complex and I ended
| up replacing it with a simpler query builder package (ozzo-
| dbx). While the query builder has abstraction for other
| databases, I'm not planning supporting them at the moment.
| giancarlostoro wrote:
| No worries, I've seen another project called UAdmin that
| kind of reminds me of this, my problem with UAdmin was that
| there were bugs and I didn't quite get why it worked how it
| worked, but having used Django since I realized what I was
| missing out on.
|
| Thank you for following up, I did see you had a package for
| extending the built-in Go package for SQL.
| lovelearning wrote:
| > backed-in files and users management
|
| Did you intend to say _baked_ -in? Like it's built-in? If so, I
| feel it's better to just use "built-in" because it's more
| familiar. The same typo's in the site too and confused me a
| bit.
| randomwebdev wrote:
| Yes, built-in sounds better. I'll update it.
| Xeoncross wrote:
| Very nice, I thought it was a basic wrapper around SQLite but
| then realized it actually had more features then that, such as
| it's own admin dashboard. Then I realized you built multiple auth
| designs into the system, then I realized there was a javascript
| SDK to interact with it, then I found your beautiful
| documentation site, then I realized there is an actual product
| inspiring it's development (https://presentator.io/)
|
| Each of these is another encouragement to use pocketbase. I went
| from "eh, I'll just keep my own wrapper" to "pocketbase has a lot
| of promise, I should try it"
| melony wrote:
| This looks amazing, congrats on shipping! Do you plan to add
| Firebase-style query optimizations too?
|
| https://news.ycombinator.com/item?id=31832266
| segphault wrote:
| This looks like a great solution for rapid prototyping scenarios
| where you want a basic CRUD backend with as little friction as
| possible. The API preview feature in the admin dashboard is a
| really nice touch.
| [deleted]
| roguas wrote:
| I would love it. But: postgres, +backups, +replica managment.
| Basically a managed database for postgres in a container.
| _query wrote:
| Check out https://thin.dev/ :) It's similar, supports self-
| hosting and uses postgres. Quick demo video here:
| https://www.youtube.com/watch?v=-jj19fpkd2c&t=3s
|
| (I'm founder of Thin)
| randomwebdev wrote:
| You may want to check Supabase or Nhost in that case.
|
| PocketBase was specifically designed to be self-contained and
| running only on a single server.
| 999900000999 wrote:
| Any client SDKs for Android, iOS, React, etc.
| gervwyk wrote:
| Well done! Really impressive, will give it a spin.
| ezekiel11 wrote:
| why is this written in Go , are there specific features that
| shine over Python3? I've been looking to learn Go, it seems lot
| of network intensive software use Go.
| rekwah wrote:
| If a selling point is run from "one file", that's quite
| difficult to do in python. There are things like pyinstaller
| but you end up shipping the entire interpreter in the bundle.
| mypalmike wrote:
| The pex tool lets you build single file python executables
| which are well suited for server deployments.
| ezekiel11 wrote:
| oh wow thanks i didn't know about this. in that case it
| would be super simple for me to wrap up a FastAPI server
| with SQLite or even some in-memory database that runs
| elsewhere, deliver as a single file!
| killingtime74 wrote:
| You could also just use Docker to wrap anything
| beardicus wrote:
| this looks really great! i had recently been thinking "boy it'd
| be really great if there were a 'supabase but sqlite' available"
| and here you are! now i just need a clever project as an excuse
| to use this.
|
| 1 question: are the docs in a repo anywhere? i don't see them in
| the main repo (i might just be missing them somehow) or anywhere
| in the org. if they are public somewhere would you be open to a
| docs polishing PR?
| randomwebdev wrote:
| The documentation is not auto generated and is part of the
| static landing site (built with SvelteKit).
|
| I didn't bother open source it because it is kinda messy and a
| little complicated to be edited by users not familiar with the
| codebase, but I'll try to find some time in the future and may
| publish it.
|
| In the meantime, if you find typos or think that some of the
| wording could be improved, feel free to open an issue or
| discussion in the main repo and I'll fix them.
| Comevius wrote:
| This here is some quality work top to bottom, including the
| documentation and the website.
| krat0sprakhar wrote:
| +1 .. Very amazing!
| Pr0ject217 wrote:
| Potentially silly question (I'll go out on a limb anyway).
|
| I would like to create a multi-platform app for non-technical
| users where their data is transparently self-contained/self-
| hosted. So, they open a single binary and it presents them with a
| UI, and the data created from within the app would be stored in
| their local PocketBase database. However, I wouldn't want the
| user to have to start a server directly (they wouldn't know what
| a server is).
|
| So, I think the binary would have to start both the PocketBase
| server and a front-end server, and then launch a browser (similar
| to Electron) pointed to their front-end server's port.
|
| Does this sound like a good approach? What could you recommend to
| orchestrate this seamlessly to make for a simple user experience
| when launching the app?
|
| Thank you.
| qbasic_forever wrote:
| Honestly I'd look at using Electron because you can take a hard
| dependency on the browser capabilities and state. If you rely
| on the user bringing their own browser you're going to have
| headaches of people on ancient browsers or just supremely
| jacked up browsers (messed up browser plugins, proxies, turned
| off javascript, etc.). Node can easily launch processes and
| there are even fancy process management libraries like pm2 you
| can add as a dependency should you need it, so the logic in
| your electron layer can be very simple and just launch some
| other processes and ferry requests to the view layer.
|
| Alternatively a webview-based framework or something like Lorca
| might be another option if you don't want to use Electron:
| https://maori.geek.nz/golang-desktop-app-webview-vs-lorca-vs...
| Pr0ject217 wrote:
| Thank you!
| randomwebdev wrote:
| Hm, there is nothing preventing you to start multiple services
| from a single binary.
|
| I'm not sure that PocketBase would be suitable for your use
| case, but it is distributed also as a Go package (see
| https://pocketbase.io/docs/use-as-framework/) and you could
| combine it with any other GUI Go package (fyne, go-astilectron,
| etc.)
| Pr0ject217 wrote:
| Thank you!
| abraxas wrote:
| Nice but my super anal quip is please don't call this thing
| "realtime". It doesn't make sense in this context. What is
| realtime about it? Are there timing guarantees or are timing
| constraints provided in API requests? Because that's what
| realtime systems are. Realtime does not mean "real fast". For
| example, a train track switching system might be getting run by a
| 1MHz CPU and still be real time. It's not about speed, it's about
| deadline guarantees.
| euroderf wrote:
| Suitable as the basis for a CMS ?
| vincentge wrote:
| This actually looks suuuupppppeeeerrrr cool!!!
___________________________________________________________________
(page generated 2022-07-07 23:00 UTC)