[HN Gopher] Directly running DuckDB queries on data stored in SQ... ___________________________________________________________________ Directly running DuckDB queries on data stored in SQLite files Author : eatonphil Score : 23 points Date : 2022-03-25 12:57 UTC (10 hours ago) (HTM) web link (twitter.com) (TXT) w3m dump (twitter.com) | samwillis wrote: | Would love to see a benchmark comparing SQLite, DuckDB and this | SQLite/DuckDB combo with the same queries and data. | | Is this adapter copying the contents of the SQLite db into | DuckDBs own memory layout in order to optimise the queries or is | it just "proxying" to the SQLite library? | 1egg0myegg0 wrote: | It is just scanning the data (in parallel), not copying it into | DuckDB! | eatonphil wrote: | I'm excited to use duckdb in DataStation/dsq but the Go bindings | aren't yet super mature and I haven't had time to work on them | myself yet. | nattaylor wrote: | Can't wait to see what simonw comes up for using this with | datasette! | samwillis wrote: | This would be super useful for something I'm working on using | datasette, querying across three tables with over 1.5m rows. | Going to try the same queries on this tomorrow. | simonw wrote: | Setting up a plugin hook that lets you use alternative database | backends is getting more and more tempting... | wenc wrote: | Exciting stuff! | | I've been using duckdb instead of Pandas these days because it's | much faster on larger datasets plus I can write more compact and | complex SQL than I can Pandas constructs. The fact that duckdb | can query in memory Pandas dataframes faster than Pandas itself | is a plus. | | It's like having a local performant database engine that can | query and join across Parquet, CSV, Pandas and now SQLite. | | Kinda like a quick efficient local Spark. ___________________________________________________________________ (page generated 2022-03-25 23:00 UTC)