[HN Gopher] ArcticDB: Why a Hedge Fund Built Its Own Database ___________________________________________________________________ ArcticDB: Why a Hedge Fund Built Its Own Database Author : todsacerdoti Score : 93 points Date : 2024-08-21 13:17 UTC (3 days ago) (HTM) web link (www.infoq.com) (TXT) w3m dump (www.infoq.com) | dang wrote: | Related: | | _ArcticDB: A high-performance, serverless Pandas DataFrame | database_ - https://news.ycombinator.com/item?id=35198131 - March | 2023 (1 comment) | | _Introducing ArcticDB: Powering data science at Man Group_ - | https://news.ycombinator.com/item?id=35181870 - March 2023 (1 | comment) | | _Introducing ArcticDB: A Database for Observability_ - | https://news.ycombinator.com/item?id=31260597 - May 2022 (31 | comments) | Nelkins wrote: | I don't think the last link is related. Different database. | silisili wrote: | Correct. They renamed FrostDB, here is the announcement - | | https://www.polarsignals.com/blog/posts/2022/06/16/arcticdb-. | .. | OutOfHere wrote: | https://github.com/man-group/arcticDB | stackskipton wrote: | Read the presentation. Answer was what I expected. We had unique | problem and because we make oil drums amount of cash, dipping a | bucket and taking that cash to solve the problem was easy | justification. | | These are really smart people solving problems they have but many | companies don't have buckets of cash to hire really smart people | to solve those problems. | | Also, the questions after presentation pointed out the data isn't | always analyzed in their database so it's more like storage | system then database. | | >Participant 1: What's the optimization happening on the pandas | DataFrames, which we obviously know are not very good at scaling | up to billions of rows? How are you doing that? On the pandas | DataFrames, what kind of optimizations are you running under the | hood? Are you doing some Spark? | | >Munro: The general pattern we have internally and the users | have, is that your returning pandas DataFrames are usable. | They're fitting in memory. You're doing the querying, so it's | like, limit your results to that. Then, once people have got | their DataFrame back, they might choose another technology like | Polars, DuckDB to do their analytics, depending on if they don't | like pandas or they think it's too slow. | datahack wrote: | This comment is underrated comedy gold. You clearly have worked | with big data. | primitivesuave wrote: | I skipped to the "why build a database" section and then | skipped another two minutes of his tangential thoughts - seems | like the answer is "because Moore's law"? | tda wrote: | I know there are tons of problems that are solved in excel while | they really shouldn't. Instead of getting the expert business | analyst to use a better tool (like pandas), money is spent to | "fix" excel. | | Apparently there is also a class of problems that outgrow pandas. | And instead of the business side switching to more suitable | tools, some really smart people are hired to build crutches | around pandas. | | Oh well, they probably had fun doing it. Maybe they get to work | on nogil python next | laweijfmvo wrote: | i'm so sorry someone built something that you don't agree with | beckingz wrote: | There's value in 'backwards compatibility' from a | process/skills perspective. I agree that companies usually pay | too high a premium on that, but there is value. | bdjsiqoocwk wrote: | Isn't it constrained to minutely timestamps or something like | that. | dnadler wrote: | Nope, it's fairly generic. I worked at Man for a number of | years and with Arctic a lot during that time. | | In my role I was mostly working with daily data, but also | higher and lower frequency data, and never had issues. | chirau wrote: | Two Sigma did a similar thing a few years back. It's called | Smooth Storage. | | https://www.twosigma.com/articles/smooth-storage-a-distribut... | dnadler wrote: | If it wasn't clear from the article, this is open source and | available on Man's GitHub page: | | https://github.com/man-group/arcticDB | | I used to work at man, so take this with a grain of salt, but I | really liked this and have spun it up at home for side projects | over the years. | | I'm not aware of other specialized storage options for | dataframes, but would be curious if anyone knows of any. | cgio wrote: | Need a licence apparently, source available. | camkego wrote: | The license is the Business Source License, which is not open | source: https://en.wikipedia.org/wiki/Business_Source_License | | The current license says the terms shall change to Apache after | two years a release has been available. | | Not for me, but some might find it interesting. | flockonus wrote: | Few doc choices are as annoying as using GIF to show code that | i should read in 2 seconds. | andrewstuart wrote: | Blog post from same company in two years: | | "How we switched from a custom database to Postgres". | Loughla wrote: | I don't understand your cynicism here. Should people/businesses | not try new things? | andrewstuart wrote: | >> Should people/businesses not try new things? | | Well yes, in the right context, like hobby/personal | programming. But things like "we built our own database" tend | to be really hard to justify and mostly represent technical | people playing toys when they actually have an obligation to | the business that is spending the money to not play toys but | to spend the money wisely and build a wise architecture, | especially for business critical systems. | | It's indulgent and irresponsible to do otherwise if standard | technology will get you there. The other point is that there | are few few applications in 2024 that would have data storage | requirements that cannot be met by Postgres or some other | common database and if not then perhaps the architecture | should be changed to do things in a way that is compatible | with existing data storage systems. | | Databases in particular as the core of a business system have | not only the get "put data in/get data out" requirement but | hundreds of sub requirements that relate to deployment and | operations and a million other things. You build a database | for that core set/get requirement and before you know it | you're wondering how to fulfill that vast array of other | requirements. | | This happens everywhere in corporate development that some | CTO who has a personal like for some technology makes the | business use that technology when in fact the business needs | nothing more than ordinary technology, avoiding all sorts of | issues such as recruiting. The CTO moves on, leaving behind | the project built with the technology flavor of the month and | a project that either needs to struggle to maintain it into | the future, or the need to replace it with a more ordinary | way of doing things. Likely even the CTO has now lost | interest in their playtime technology fad that the industry | toyed with and decided isn't a great idea. | | So I stand by my comment - they are likely to replace this | within a few years with something normal, likely Postgres. | ateng wrote: | I used to work at Man, and they have been using ArcticDB | (including the earlier iteration) for over 10 years now. | oxfordmale wrote: | They have tons of money, enough to support a development team | improving their database. In addition there is a long legacy, | both technically and politically, making it hard to even | propose to get rid of it. The only likely switch is a gradual | decline, with parts of the system moved to another database. | faizshah wrote: | I still didn't get why they built this, there's a better | explanation of the feature set in the FAQ comparison with | parquet: https://docs.arcticdb.io/latest/faq/ | | > How does ArcticDB differ from Apache Parquet?P | | > Both ArcticDB and Parquet enable the storage of columnar data | without requiring additional infrastructure. | | > ArcticDB however uses a custom storage format that means it | offers the following functionality over Parquet: | | > Versioned modifications ("time travel") - ArcticDB is | bitemporal. > Timeseries indexes. ArcticDB is a timeseries | database and as such is optimised for slicing and dicing | timeseries data containing billions of rows. > Data discovery - | ArcticDB is built for teams. Data is structured into libraries | and symbols rather than raw filepaths. > Support for streaming | data. ArcticDB is a fully functional streaming/tick database, | enabling the storage of both batch and streaming data. > Support | for "dynamic schemas" - ArcticDB supports datasets with changing | schemas (column sets) over time. > Support for automatic data | deduplication. | | The other answer I was looking for was why not kdb since this is | a hedge fund. | mianos wrote: | I think people are getting a little tired of being held ransom | to kdb/q and kdb consultants. Even if you have 'oil barrels' of | money, eventually it is annoying enough to look elsewhere. | blitzar wrote: | You don't get 'oil barrels' of money if you are in the habit | of giving big chunks of it to other people. | | Handing over seed capital levels of cash to external vendors | inevitably gives someone the idea to seed a competitor | instead. | porker wrote: | > Versioned modifications ("time travel") - ArcticDB is | bitemporal. > Timeseries indexes. | | That's a nice addition. | blitzar wrote: | Given how high a selling point this is, it is something that | I cannot recall ever using in ArcticDB (5+ years of using). | It's (financial) time series data, the past doesn't change. | | Back adjusting futures contracts or back adjusting for | dividends / splits come to mind as I write this, but I would | just reprocess these tasks from the raw data "as of date" if | needed. | jmakov wrote: | Is there any reason to use that instead of Delta lake? ___________________________________________________________________ (page generated 2024-08-25 09:01 UTC)