[HN Gopher] FoundationDB: A distributed unbundled transactional ...
___________________________________________________________________
FoundationDB: A distributed unbundled transactional key value store
Author : eatonphil
Score : 104 points
Date : 2022-03-11 15:19 UTC (7 hours ago)
(HTM) web link (muratbuffalo.blogspot.com)
(TXT) w3m dump (muratbuffalo.blogspot.com)
| tiffanyh wrote:
| I feel dumb asking, what's the common use case for FDB. And how
| does it compare to Redis?
|
| I've read lots about how fantastic FDB (and unique) but I'm
| struggling to understand the use case / problem to use it for.
| rapsey wrote:
| Fdb's main selling point was reliability. But with the
| proliferation of good raft/paxos implementations that I think is
| not that much of an advantage anymore.
| manigandham wrote:
| There's a lot more to reliability than just the consensus algo
| and implementation.
| frakkingcylons wrote:
| You say that like it's trivial to build a reliable OLTP key
| value store once you have a correct raft implementation. Never
| mind real world performance or reliability.
| [deleted]
| EGreg wrote:
| Is there a proven BFT version of that? RAFT aint it
| skyde wrote:
| For Anyone with experience with FoundationDB! How does it compare
| to CockroachDB ?
| eatonphil wrote:
| FoundationDB is just a key value store, so it's not comparable.
|
| There is a relational layer you can run though called
| Foundation Record Layer. But this doesn't support SQL AFAICT.
| The examples on Github just show a query builder interface in
| Java [0].
|
| There does seem to be a Lucene interface but it doesn't seem to
| be used [1].
|
| [0] https://github.com/FoundationDB/fdb-record-
| layer/blob/main/e...
|
| [1] https://forums.foundationdb.org/t/lucene-layer-on-
| foundation...
| StreamBright wrote:
| There used to be some SQL layer:
|
| https://forums.foundationdb.org/t/sql-layer-in-
| foundationdb/...
| eatonphil wrote:
| Neat, here [0] is a 3rd party one at the end of the thread
| but this isn't the one the thread is about. Doesn't seem
| like that one is public.
|
| [0] https://github.com/rustdesk/opentick
| tonyhb wrote:
| It's comparable based off of the fundamentals and I have the
| same question.
|
| They both use KVs using LSMs, do range queries, support ACID
| transactions, handle resharding and clustering for you.
|
| They're both based off of Spanner fundamentals. They both
| actually have an SQL layer - foundationdb's is the record
| layer. Just because one has a primary SQL interface doesn't
| mean we can't compare.
|
| I'd really like to know a comparison of the fundamentals
| including hotspot management, resharding, read/write
| performance, resharding etc. also and have been looking for
| this.
| eatonphil wrote:
| > They both actually have an SQL layer - foundationdb's is
| the record layer.
|
| Can you show me their SQL layer? I was looking for it.
| AlphaSite wrote:
| I don't think it ever got open sourced, although I do
| remember reading about it years back.
| tonyhb wrote:
| https://github.com/FoundationDB/fdb-record-layer is the
| SQL/orm like interface. This handles things in a DB like
| way - joins, indexes. I misspoke when I said SQL
| regarding this - sorry.
|
| There's a go and rust library floating around that's not
| as good. I've tried em :)
| eatonphil wrote:
| The readme says
|
| > Queries - The Record Layer does not provide a query
| language, however it provides query APIs with the ability
| to scan, filter, and sort across one or more record
| types, and a query planner capable of automatic selection
| of indexes.
|
| So I'm not sure how you can call that a SQL layer when
| SQL is a query language and FDB Record Layer says it
| doesn't have a query language?
| tonyhb wrote:
| Yeah, I did say "I misspoke when I said SQL regarding
| this - sorry.". I should have said "SQL-like", or at
| least relation-like.
|
| There were a couple SQL-ish things I found in other
| languages and got them mixed up. Sorry about that :)
| manigandham wrote:
| FoundationDB is a key/value store, and well-known for being
| fast, scalable and incredibly reliable. It passed many of the
| distributed database tests unlike many competitors. Here's a
| good HN thread when Apple open-sourced it a few years ago:
| https://news.ycombinator.com/item?id=16877395
|
| CockroachDB implements postgres-compatible SQL layer on top of
| their own key/value store called Pebble. This is similar to
| TiDB which implements mysql on top of their TiKV key/value
| store.
|
| The latter databases offer easier programming with SQL
| interfaces, but since you can model just about anything on top
| of key/value store, there are already "layers" that can
| implement similar abstractions on top of FDB like the Record
| Layer[1], Document Layer [2], and even a SQL layer [3] at one
| point.
|
| 1) https://www.foundationdb.org/blog/announcing-record-layer/
| 2) https://foundationdb.github.io/fdb-document-layer/ 3)
| https://github.com/jaytaylor/sql-layer
| atombender wrote:
| An important difference between CockroachDB and TiDB is that
| the former has an embedded key-value store; everything runs
| in the same process.
|
| TiDB is split into multiple parts: TiDB proper (stateless
| SQL/query plan layer), TiKV (the key-value store), and PD
| (placement driver, the main control plane and metadata
| layer). TiKV also isn't a pure key-value store, as it handles
| some particular query operations that can be "pushed down"
| from TiDB to reduce network round trips.
|
| FoundationDB's role similar to TiKV. It's a pure key-value
| store with replication built in.
| legojoey17 wrote:
| Adding clarifications to that, Pebble is a local K/V library
| akin to RocksDB/LevelDB and provides no services or
| distributed capabilities. Comparing the two does not make
| sense.
|
| On the other hand, TiKV is more comparable to the internal
| distributed transactional K/V layer to CockroachDB that its
| SQL layer is coupled to. That of course is not a usable
| external interface. You could utilize CockroachDB like a K/V
| store with simple table to forego the SQL functionality and
| planning (e.g. only primary col index, only key/value column,
| so on), but I am not sure what the practicality of that is as
| I have not kept up with CockroachDB development.
|
| (disclaimer: I interned at Cockroach several years back)
| manigandham wrote:
| The comparison is about providing a relational data model
| on top of a key/value store.
|
| Yugabyte also does something similar on top of its
| underlying DocDB document store.
| EGreg wrote:
| What is better, FoundationDB, CockroachDB, or Hypercore?
| vvern wrote:
| One of these is not like the others.
| jandrewrogers wrote:
| There is one caveat I would add. While you can always build
| just about any kind of database or data model on top of a
| logical KV store, the scalability and performance is almost
| always going to be qualitatively worse for queries than using
| a storage engine purpose-built for the intended type of
| database or data model. It is simpler to implement, which is
| why people do it, but you give up a lot of selectivity and
| locality on the query side.
|
| The target design case for KV stores is when all your queries
| are essentially trivial. Layering SQL on top of it implies
| the expected use case is complex queries of the type where a
| KV store won't perform well. Good for a compatibility
| interface, but not performant.
| skrtskrt wrote:
| though as with things like CAP theorem, there are always
| clever ways to to place a given implementation at various
| points along the spectrum, rather than just at one end or
| the other.
| sa46 wrote:
| Very curious, what's an alternative implementation strategy
| or more appropriate mental model?
|
| My mental model (I am not a database engineer) is that
| every SQL database is fundamentally backed by a key value
| store. For Postgres, each value is a row of a relation, the
| key is the relation's key, and the physical storage is the
| table file. Postgres builds b-trees on top to make it fast,
| but the b-tree itself is a relation, a key value store that
| Postgres works hard to keep sorted.
| mamcx wrote:
| Not expert, but as enthusiast I have read some about this
| (for https://tablam.org).
|
| KV is truly ill-suited for this.
|
| Is very easy to see if you put the data layout:
| --data pk city country 1 miami
| USA 2 bogota Colombia --As Kv
| (naive): pk1: 1 pk2: 2
| city1: miami city2: bogota --As
| btree, with "value" being N (as you say):
| Key Value 1 1 miami USA
| 2 2 bogota Colombia --As paged
| btree Page1 Low: 1 High:2
| //think this is a block 1 miami USA
| 2 bogota Colombia --As columnar:
| pk: 1 2 city: miami bogota
| --As PAX (hybrid columnar/row) Page1
| Low: 1 High:2 //think this is a block
| pk: 1 2 city: miami bogota
| travisd wrote:
| i'm also not a db engineer, but i think this is true-ish.
| however building and maintaining those index tables is
| hard and probably prone to issues if you can't update
| multiple as part of the same transaction.
|
| the other major thing you'd miss is smarter joins. the
| distributed databases do a lot of work to be able to push
| down predicates as far down as possible to the physical
| nodes storing the data.
|
| there's probably more as well.
| Gaelan wrote:
| > probably prone to issues if you can't update multiple
| as part of the same transaction
|
| IIRC one of FoundationDB's features is that it does
| support such transactions, so you can easily implement
| indexing on top of it.
| manigandham wrote:
| Yes, agreed, although a lot of storage modalities naturally
| map to key/value in some way.
|
| Ironically FDB uses SQLite for the storage servers.
| Abstractions all the way down.
| ledgerdev wrote:
| I was just looking into this, and found this really interesting
| post about foundationdb and sql.
|
| https://www.voltactivedata.com/blog/2015/04/foundationdbs-le...
|
| Also this, excellent comparison of distributed database
| consistency methods, mentions cockroach, but not foundation.
| http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst...
|
| This brings up the question, what method/protocol does
| foundation use for distributed consistency?
| chrisweekly wrote:
| FTA:
|
| > FoundationDB, a transactional key-value store that supports
| multi-key strictly serializable transactions across its entire
| key-space. FoundationDB (FDB, for short) is opensource. The paper
| says that: "FDB is the underpinning of cloud infrastructure at
| Apple, Snowflake and other companies, due to its consistency,
| robustness and availability for storing user data, system
| metadata and configuration, and other critical information."
|
| > The main idea in FDB is to decouple transaction processing from
| logging and storage. Such an unbundled architecture enables the
| separation and horizontal scaling of both read and write
| handling.
| [deleted]
| [deleted]
| dang wrote:
| Kind of interesting to list the past threads. Were there other
| major ones? I omitted a few trivial ones.
|
| _FoundationDB: A distributed unbundled transactional key value
| store_ - https://news.ycombinator.com/item?id=28740497 - Oct 2021
| (44 comments)
|
| _FoundationDB: A distributed, unbundled, transactional key value
| store [pdf]_ - https://news.ycombinator.com/item?id=27424605 -
| June 2021 (99 comments)
|
| _Swizzle Clogging_ -
| https://news.ycombinator.com/item?id=25176931 - Nov 2020 (7
| comments)
|
| _Redwood: The Future of the FoundationDB Storage Engine_ -
| https://news.ycombinator.com/item?id=20498836 - July 2019 (5
| comments)
|
| _Apple Open-Sources FoundationDB Record Layer_ -
| https://news.ycombinator.com/item?id=18910748 - Jan 2019 (32
| comments)
|
| _FoundationDB Record Layer_ -
| https://news.ycombinator.com/item?id=18906341 - Jan 2019 (78
| comments)
|
| _FoundationDB Summit: video roundup_ -
| https://news.ycombinator.com/item?id=18826096 - Jan 2019 (3
| comments)
|
| _FoundationDB Document Layer_ -
| https://news.ycombinator.com/item?id=18563179 - Nov 2018 (35
| comments)
|
| _FoundationDB 6.0 released, featuring multi-region support_ -
| https://news.ycombinator.com/item?id=18488879 - Nov 2018 (70
| comments)
|
| _FoundationDB Summit Program Announced_ -
| https://news.ycombinator.com/item?id=18304906 - Oct 2018 (23
| comments)
|
| _FoundationDB high contention allocator_ -
| https://news.ycombinator.com/item?id=17817946 - Aug 2018 (9
| comments)
|
| _FoundationDB community highlights, two weeks in_ -
| https://news.ycombinator.com/item?id=16994579 - May 2018 (49
| comments)
|
| _Snowflake Metadata powered by FoundationDB_ -
| https://news.ycombinator.com/item?id=16880379 - April 2018 (8
| comments)
|
| _Apple open-sources FoundationDB_ -
| https://news.ycombinator.com/item?id=16877395 - April 2018 (441
| comments)
|
| _FoundationDB 's Lesson: A Fast Key-Value Store Is Not Enough_ -
| https://news.ycombinator.com/item?id=9306297 - April 2015 (32
| comments)
|
| _Apple Acquires FoundationDB_ -
| https://news.ycombinator.com/item?id=9259986 - March 2015 (376
| comments)
|
| _FoundationDB and the New NoSQL_ -
| https://news.ycombinator.com/item?id=8745333 - Dec 2014 (19
| comments)
|
| _Databases at 14.4Mhz_ -
| https://news.ycombinator.com/item?id=8729420 - Dec 2014 (81
| comments)
|
| _The Future of NoSQL_ -
| https://news.ycombinator.com/item?id=8599198 - Nov 2014 (24
| comments)
|
| _On Lowered Expectations: Transactions, Scaling, and Honesty_ -
| https://news.ycombinator.com/item?id=7869623 - June 2014 (5
| comments)
|
| _FoundationDB vs. Jepsen_ -
| https://news.ycombinator.com/item?id=7467037 - March 2014 (23
| comments)
|
| _Google F1 vs. FoundationDB 's SQL Layer_ -
| https://news.ycombinator.com/item?id=6718282 - Nov 2013 (12
| comments)
|
| _Which modern databases support ACID transactions?_ -
| https://news.ycombinator.com/item?id=6636541 - Oct 2013 (54
| comments)
|
| _FoundationDB 1.0_ -
| https://news.ycombinator.com/item?id=6244631 - Aug 2013 (6
| comments)
|
| _FoundationDB marries NoSQL & SQL with Akiban acquisition_ -
| https://news.ycombinator.com/item?id=6057440 - July 2013 (3
| comments)
|
| _FoundationDB Fault Tolerance Demo Video_ -
| https://news.ycombinator.com/item?id=5739721 - May 2013 (15
| comments)
|
| _A NoSQL Database with ACID Transactions_ -
| https://news.ycombinator.com/item?id=5318307 - March 2013 (55
| comments)
|
| _FoundationDB -- Not Your Standard NoSQL Database_ -
| https://news.ycombinator.com/item?id=4503400 - Sept 2012 (17
| comments)
|
| _FoundationDB: A new generation of NoSQL database_ -
| https://news.ycombinator.com/item?id=4294719 - July 2012 (73
| comments)
| Timshel wrote:
| I would add this awesome talk which never got much traction
| (outside of comments in other threads).
|
| Testing Distributed Systems with Deterministic Simulation
| [video] - https://news.ycombinator.com/item?id=8345030 - Sept
| 20, 2014 (6 comments)
___________________________________________________________________
(page generated 2022-03-11 23:00 UTC)