[HN Gopher] Building a database buffer pool in Zig using io_urin... ___________________________________________________________________ Building a database buffer pool in Zig using io_uring's new fixed- buffer mode Author : gavinray Score : 47 points Date : 2022-10-15 20:11 UTC (2 hours ago) (HTM) web link (gavinray97.github.io) (TXT) w3m dump (gavinray97.github.io) | cmrdporcupine wrote: | Aha! I was thinking about trying this exact same approach | recently, so I'm glad to see I wasn't insane! Very clever. | | I have been "in my spare time" (that is, never finishing) | building out a page buffer mgmt system based on the one in the | Umbra paper (see | https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf) | which has a kind of novel approach where they essentially | allocate the same physical address at multiple virtual addresses | but with different block/slab sizes so that they can reduce | fragmentation and still support dynamic page sizes. Anyways, it | seems this author is also on the same "page" (sorry) as me but | focusing more on exploiting `io_uring`. Nifty. | | I'm working in Rust, not Zig, but a lot of nice stuff to read | through here. | | I will have to read through this in more detail later when I have | more brain-space available. | gavinray wrote: | Jinx, I'm also working on this, as a tutorial series! | | I had the pleasure of hearing Neumann present the Umbra | architecture on the CMU DB series webinar recently. | | I'm slowly working through re-implementing the Coroutines async | buffer pool paper. Blocked on trying to properly implement and | understand "Morsel-Driven Parallelism": | | https://db.in.tum.de/~fent/papers/coroutines.pdf | | I have the async, io_uring buffer pool written in Kotlin using | it's native Coroutines but the scheduling and morsel stuff I'm | still trying to understand | cmrdporcupine wrote: | I watched the same CMU lecture, very cool. | | Are you getting paid to work on this kind of thing, anywhere? | Do you want to be? | gavinray wrote: | > Are you getting paid to work on this kind of thing, | anywhere? | | My dayjob is a fair bit higher level. I work on a sort of | query engine on the JVM that translates an internal query | IR/AST into dialect-specific SQL + other languages so you | can query anything with the same frontend language: | | https://techcrunch.com/2022/06/28/hasura-now-lets- | developers... | | I think it'd be neato to work on a database, but my dayjob | is also pretty interesting to me. | gavinray wrote: | > I'm working in Rust, not Zig, but a lot of nice stuff to read | through here. | | FYI, Datadog has a Rust library for scheduling things to run | thread-per-core with io_uring | | It'd be really useful for DB use cases: | | https://github.com/DataDog/glommio | cmrdporcupine wrote: | Nifty. I started from the part of the Umbra presentation that | initially struck me as most interesting -- the allocator | portion (where he talks about repeatedly MMAP_ANONYMOUS on | the same address range to reduce fragmentation, etc.) I | stopped working when I started pondering the bits about | pointer swizzling and how I'd make that work nicely in Rust, | and never got as far as the pieces that would actually do the | syncs to disk. | | I also came across another fellow who was inspired by the | optimistic locking btree presented in LeanStore Umbra: | https://github.com/NeowayLabs/bplustree | gavinray wrote: | One note: I mistakenly thought that fixed buffers were part of | the 5.19 changes for provided buffers | | Jens Axboe corrected me on this, fixed buffers have existed for a | while apparently! | | https://twitter.com/axboe/status/1581392910687567873?t=gx_tQ... ___________________________________________________________________ (page generated 2022-10-15 23:00 UTC)