[HN Gopher] ToyDB: Distributed SQL Database in Rust
___________________________________________________________________
ToyDB: Distributed SQL Database in Rust
Author : adamnemecek
Score : 168 points
Date : 2021-07-18 18:03 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| z3t4 wrote:
| So do the throughput decrease when you add more nodes? Or am I
| reading it wrong? This is a general problem with scaling
| horizontally, that the overhead can kill the performance, and
| single core/node performance is sacrificed.
| pas wrote:
| Likely you are reading it correctly. What you gain in possible
| availability (because now you have more nodes, which give some
| redundancy with regards to node failure) you have to pay the
| work necessary for constantly running the consistency protocol
| from your performance budget.
|
| How much you have to sacrifice, and how linear the scaling is,
| of course are important quality metrics of distributed systems.
| In multi-writer optimistic-locking sync-at-commit systems (eg
| Galera) in case of no conflict, it's possible to have the
| multi-node throughput exceed the throughput of the single-node
| version.
| void_mint wrote:
| https://github.com/erikgrinaker/toydb/blob/master/docs/refer...
|
| This is exactly what I was hoping to find. This is great!
| Ostrogodsky wrote:
| I concur, excellent choices of background material.
| mrtweetyhack wrote:
| hope this toy turns into a major player
| arpanetus wrote:
| i hope i aint being toxic
|
| but i wonder what if the same db was written in c++, would it get
| a post in HN?
| asdjlkadsjklads wrote:
| In this case i'd say yes. It's a DB purposefully written for
| learning, with good references and documentation.
|
| However if it was just another DB.. especially one competing in
| an area no one really feels deficient, prob not.
| [deleted]
| mathgladiator wrote:
| Well done, I'm excited about efforts like this and I hope you
| make progress towards some production traffic (I'm aware it is a
| toy, but toys are best when used by others :) )
| netsec_burn wrote:
| As a Rust developer (that's up to their neck in Rust projects)
| I'm really looking forward to a functioning sqlite clone. To
| the best of my knowledge, Sqlrite
| (https://github.com/joaoh82/rust_sqlite) was supposed to be
| that but it ended up diverging from sqlite. It'll make cross
| platform Rust builds easier.
| pm90 wrote:
| I'm curious to understand what you're looking to benefit from
| with a SQLite rust clone vs SQLite itself? Would it make
| writing apps easier? Or would it open up the internals of dB
| engines to rust developers? Or something else..? Honest
| question.
| netsec_burn wrote:
| I think I answered your question in my comment, no? It'll
| make cross platform builds easier instead of the current
| steps needed for FFI libraries. My experience with cross
| platform FFI builds are slim but last time I tried it, it
| was painful (versus Rust handling it all with the right
| toolchain).
| ofrzeta wrote:
| Can't be as hard as reimplementing SQLite, can it?
| wizzwizz4 wrote:
| But reimplementing SQLite can be done by somebody else,
| whereas dealing with FFI must be done by me.
| azornathogron wrote:
| I feel like I must be misunderstanding you... why would
| the FFI be done by you? The rusqlite crate provides
| wrappers you can use, and there are probably other
| wrapper crates that I haven't looked at.
| wizzwizz4 wrote:
| But I have to configure how it finds SQLite. Sure,
| _SQLite specifically_ is fairly easy - I only need to
| enable the "bundled" crate feature and make sure my
| Clang version matches the Rust version so LTO works - but
| it's a little arcane to use Cargo with any non-Rust
| package in the dependency chain. Cargo Just Works(tm).
| Cargo with C FFI _doesn 't_.
| azornathogron wrote:
| Thanks for the explanation!
| netsec_burn wrote:
| Not the parent comment, but I believe they're referring
| to the cross compilation aspects which wouldn't be
| resolved by just using the rusqlite crate (I use rusqlite
| and have this issue).
| fsloth wrote:
| Sqlite has been battletested for years.
|
| I utterly fail to see the point in reimplementing it in some
| other language.
|
| Innovate in databases - yes please - but "rewrite" - why?
|
| Not being able to use battletested libraries written in plain
| C (e.g. sqlite) sounds like a hugh mark against rust as a
| systems programming language.
|
| Is using C libraries an _actual_ issue or just an aesthetic
| grumble?
| pas wrote:
| Maintainability? Performance? Tighter integration into Rust
| projects?
|
| Also, with using SQLite's amazing test harness, it might
| even make sense to claim that the Rust re-implementation is
| of similar quality in terms of (lack of) bugs.
| zie wrote:
| only some of the test harness is open source, see:
| https://sqlite.org/testing.html
| simlevesque wrote:
| > battletested
|
| Is software that is always evolving battletested ? A new
| version could break something.
|
| I'm aware that sqlite has one of the best test suite in all
| of open source, I just wonder if something that was
| "battletested" is still battletested after a new release.
| lukevp wrote:
| I guess we can't speak about battle testing in a new
| release from a "running in production with real user
| workloads" but if the test suite is properly implemented
| and has good coverage and exercises many cases and
| failure modes, it's possible that a SQLite release is
| more battle tested on day zero than other software is
| when it's been running for months or years. What do you
| think? A new version presumably still runs through
| validation before release.
| ithkuil wrote:
| Here's where the asymmetry of security bugs and feature
| bugs comes into play.
|
| Introducing a buffer overflow in a small unused corner /
| edge case of the feature set allows an attacker to fully
| compromise the process, and it's thus a problem affecting
| a 100% of the users.
|
| Introducing a behavioral bug in the same unused corner /
| edge case, will affect only a very small number if people
| (possibly none)
| jasonwatkinspdx wrote:
| Freezing the software is no assurance of security either.
| I'll take ongoing maintenance of the quality of sqlite
| every time personally.
| mamcx wrote:
| Sqlite is probably the one C project I totally trust.
|
| But, yeah, "cloning" projects is losing proposition (even
| more if open source). I wish a sqlite-like on rust? yeah,
| but not a clone, but in spirit (exist so much we can
| improve with rdbms!)
| hxzhao wrote:
| the RIIR fallacy, again
| xpe wrote:
| AYATAINWK?
|
| Are You Aware That Acronym Is Not Widely Known?
| adamnemecek wrote:
| I've been building a prototype of a relational in-memory database
| in Rust to replace some involved ad-hoc data structures and in
| the process, I came across this.
|
| I think that all the state management problems and solutions in
| all languages/frameworks (react, swiftui, elm etc etc) would be
| better implemented as a database with changefeeds [0]/ change
| data capture [1].
|
| The advantage compared with something like Redux is that it's
| more general. You don't need to implement an enum with a case for
| each action and then have a big dispatcher.
|
| All your update logic would be a big switch statement on change
| feed from the database which would handle all the logic.
|
| The fundamental problem of state management is as follows, if a
| leaf view changes some data, how does some view higher in the
| hierarchy learn about this?
|
| I think that this solves the problem better than alternatives or
| the observer pattern.
|
| Ideally the API would be something like LINQ methods in order to
| avoid parsing SQL.
|
| I'm still trying to figure out if this would work.
|
| [0] https://rethinkdb.com/docs/changefeeds/ruby/
|
| [1] https://en.wikipedia.org/wiki/Change_data_capture
| lukevp wrote:
| > The advantage compared with something like Redux is that it's
| more general. You don't need to implement an enum with a case
| for each action and then have a big dispatcher. All your update
| logic would be a big switch statement on change feed from the
| database which would handle all the logic.
|
| You seem to have thought a lot about this, but I don't
| understand the distinction you're making and I'd appreciate
| help squaring this up in my mind.
|
| In the case of Redux, you have the current state, you have
| actions and selectors (from a CQRS perspective, an action = a
| command and a selector = query), and you have reducers that
| apply the actions to the state then update/notify the
| selectors. How is this different than what you stated? Are you
| just suggesting the removal of actions and directly changing
| the state, like reactive programming, with a changefeed being
| generated as a side effect of manipulating the state? In that
| case, how do you handle situations where the update requires
| knowledge of the current state? Won't you need to read in the
| current state, apply changes, and then write back? And at some
| point you will want to be able to pass parameters so you can
| combine some input (eg. the updated user's last name in a text
| box) and the current state (the user's first and last name) to
| update another state property (the user's full name). At that
| point you've essentially re-implemented the concept of actions
| and reducers.
| lukevp wrote:
| > The fundamental problem of state management is as follows, if
| a leaf view changes some data, how does some view higher in the
| hierarchy learn about this?
|
| At the end of the day, if you have a relational model and you
| want to allow subscriptions arbitrarily on the object graph,
| something somewhere will have to traverse the tree of the
| object's parents and apply a notification to them, right?
|
| Or you will have to only allow replacing object
| changes/subscriptions at a flat level, which is a document-
| based model rather than relational. In that case, you can
| filter the notification stream as an optimization for the
| clients, but the entire document would require replacement on
| every update. You can do an additional optimization and allow
| patches of the object, and use diffing to not notify if a
| specific property was unchanged after the state update, but
| these are all implementation details.
|
| I don't think this is a fundamental problem of state
| management, more-so a restriction of state management when
| dealing with state expressed as relational objects, and one of
| the reasons document databases are easier to reason about.
|
| Bringing this back around to state management frameworks like
| Redux, each Redux store is essentially a single document and
| selectors are filtered notifiers of changes.
|
| This project sounds interesting - you should make a blog post
| about your prototype database with some more details, I would
| love to read it and see some of the code!
| Wonnk13 wrote:
| Wow - I just started my Rust journey this weekend. Any good
| projects that are even simpler than this to learn. maybe a basic
| k/v store?
| adamnemecek wrote:
| Check out sled
|
| https://github.com/spacejam/sled
|
| It's surprisingly mature and the person behind it is very
| committed to the project.
| qchris wrote:
| +1 for sled--I was working with it this morning for a
| personal project, and it's frankly just really well built.
| Beside the great documentation and easy use, I kept finding
| myself looking for architectural tips for structuring Config
| structs, etc. because I could do a lot worse than end up
| coding something that is similarly nice to use.
___________________________________________________________________
(page generated 2021-07-18 23:00 UTC)