[HN Gopher] SQLite-loadable-rs: A framework for building SQLite ...
___________________________________________________________________
SQLite-loadable-rs: A framework for building SQLite Extensions in
Rust
Author : petercooper
Score : 105 points
Date : 2022-12-13 18:54 UTC (4 hours ago)
(HTM) web link (observablehq.com)
(TXT) w3m dump (observablehq.com)
| dfee wrote:
| Where can I get a good understanding of the limitations of SQLite
| for production use cases?
|
| I've used it frequently for prototyping, but the dialect is
| different enough from Postgres, that I usually switch over pretty
| early. But for production, it seems people are really advancing
| it as a solution for applications, which doesn't fit with the
| "When To Use" guide on the website.
| https://www.sqlite.org/whentouse.html
| simonw wrote:
| The idea that SQLite shouldn't be use for web applications is
| about a decade out-of-date at this point.
|
| But... actual guidance as to how to use it there is still
| pretty thin on the ground!
|
| Short version: use WAL mode (which is not the default). Only
| send writes from a single process (maybe via a queue). Run your
| own load tests before you go live. Don't use it if you're going
| to want to horizontally scale to handle more than 1,000
| requests/second or so (though vertically scaling will probably
| work really well).
|
| I'd love to see more useful written material about this. I hope
| to provide more myself at some point.
|
| Here are some notes I wrote a few months ago:
| https://simonwillison.net/2022/Oct/23/datasette-gunicorn/
| Thaxll wrote:
| > The idea that SQLite shouldn't be use for web applications
| is about a decade out-of-date at this point
|
| I think it's still very relevant. I mean you have service XYZ
| replicated two times behind a load balancer, how do you use
| SQLite out of the box? You can't, you have to use one of
| those service that will proxy / queue queries, so then why
| even trying to use SQLite.
|
| The fact that it's just a file on disk make it a bad solution
| for a lot of simple use cases.
| ok_dad wrote:
| You could partition your data to 2 parts, and direct your
| users to whichever service serves their partition of that
| data. Every database turns out to be just a set of files on
| disk, but for using SQLite you have to think differently
| than most databases due to the extreme locality of the
| data.
| emadda wrote:
| You only need a pool of servers behind a load balancer for
| extreme reliability or for scaling out.
|
| Most small to medium web applications can handle:
|
| - Only being reliable for 99.99% of the time (using a
| single cloud VM where the cloud provides the hardware level
| failover).
|
| - Scaling up instead of out (adding CPU, RAM and faster
| disk).
| dgb23 wrote:
| The listed use cases for applications in the link you provided
| are:
|
| - Application file format
|
| - Websites
|
| - Server-side database
|
| Under not recommended they essentially say, very high volume
| and highly concurrent access.
| senthilnayagam wrote:
| I love sqlite and rust extension which author has given gets me
| excited.
|
| Last week saw a similar project in rust to built Postgres
| extensions
|
| https://github.com/tcdi/pgx
|
| Hope I can play something interesting during vacations
| Kinrany wrote:
| SQLite has a big advantage over Postgres here if installing an
| extension is trivial.
| giovannibonetti wrote:
| You might want to have a look at Pgx:
|
| https://github.com/tcdi/pgx
| michaelmior wrote:
| I've never written an extension for Postgres, but for the ones
| I've installed, it's amounted to copying some files and running
| "CREATE EXTENSION" This feels pretty trivial to me already.
| maxpert wrote:
| Just in time! I've been planning a SQLite extension written in
| Rust for Marmot (https://github.com/maxpert/marmot) for
| evaluation purposes. I didn't wanted to touch C/C++ because I
| feel more comfortable with rust tooling. Thank you for doing
| this!
| alexgarcia-xyz wrote:
| Author here, happy to answer any questions! Here's a few links
| that may be useful:
|
| Direct GitHub link: https://github.com/asg017/sqlite-loadable-rs
|
| A soon-to-be-released sqlite-loadable extension that's the fasted
| CSV parser for SQLite, and rivals DuckDB's parser at non-
| analytical queries: https://github.com/asg017/sqlite-xsv
|
| A regular expression extension in Rust, the fastest SQLite regex
| implementation: https://github.com/asg017/sqlite-regex
|
| And in the near future, expect more extensions for postgres,
| parquet, XML, S3, image manipulation, and more!
| eatonphil wrote:
| I worked on a set of extensions for mattn/go-sqlite3, written in
| Go, inspired but a little further along (in functionality and
| test coverage) than SQLean. Useful if you want to see examples of
| this in Go or are using Go rather than Rust to embed SQLite.
|
| https://github.com/multiprocessio/go-sqlite3-stdlib
___________________________________________________________________
(page generated 2022-12-13 23:00 UTC)