[HN Gopher] Data-Oriented Architecture
___________________________________________________________________
Data-Oriented Architecture
Author : Eyas
Score : 88 points
Date : 2020-03-08 19:27 UTC (3 hours ago)
(HTM) web link (blog.eyas.sh)
(TXT) w3m dump (blog.eyas.sh)
| janci wrote:
| Would it be possible make interactive systems with this
| architecture? The requester needs to be notified about result
| availability, so some communication must be targeted to specific
| component (the requester) going against the foundation idea of
| this pattern.
| Eyas wrote:
| I didn't cover it in the article, but the key to get
| interactivity would be to switch to a datastore layer that
| supports subscribable queries.
|
| I'm aware of Esper[1] which tries to do this. And
| maybe/arguably Firebase (?)
|
| [1]: http://www.espertech.com/esper/
| bradleyankrom wrote:
| I believe RethinkDB also has this functionality, but I'm not
| 100% certain. Will check the docs later and update...
| tyingq wrote:
| DOA isn't a great acronym :)
| dathinab wrote:
| Yes it's _Domain_ orientated architecture. Which is a less well
| known think since I think 10+ years ;=)
| evanmoran wrote:
| Well said, maybe Model Oriented Architecture (MOA) is better?
| awinter-py wrote:
| using schemas rather than APIs to share information between
| components seems right
|
| IMO one of the reasons CRUD is so hard now is that the schema is
| different at every layer of the product
|
| Slight differences between layers are necessary for permissions /
| privacy, but there are probably better ways to get that done than
| to reimplement the schema at every layer.
| atomicity wrote:
| I liked the clarity of the article and the reasoning is solid.
| Still, I'm not too convinced:
|
| - A key reason for splitting up a monolith into services is
| because collaboration becomes too costly with 100s to 1000s of
| developers working on the same code. Your build & test system
| can't handle the number of commits. Your code takes too long to
| build. Would the data-access layer not them become the
| development bottleneck in such scenarios, meaning that it doesn't
| scale as well as SOA?
|
| - What are the advantages and disadvantages of centralizing in
| the data-oriented layer over centralizing through a network-
| related tool like Envoy, Kubernetes, or Istio?
|
| - The database layer itself often becomes a performance
| bottleneck, which requires us to run a sharded database. Some
| companies go the extra distance by running in-memory databases,
| document databases, and time-series databases. In such cases,
| wouldn't the data access layer need to support federation, which
| is a hard problem according to database research?
|
| - Is O(N^2) really that big of a problem? It seems like the
| problem can be reduced to something simpler: developers cannot
| easily understand which services communicate with one another. If
| that is the case, would a visualization tool be sufficient?
| throwawaygo wrote:
| Cache rules everything around me.
| beigeoak wrote:
| Spectacular
| dathinab wrote:
| It's in a certain way a very roundabout way to describe a event
| based SOA system.
|
| Many of the problems described are problems from SOA systems
| which are using a number of common anti-patterns for SOA systems.
|
| Also the way this person describes DOA is prone to end up with a
| monolith in _data_ and a just the logic is not monolithic but
| might end up accidentally being quite tight coupled. (Through it
| does have some benefits).
|
| Also even with DOA you can end up with internal state coupling
| between services if you do it wrong. It's harder then in some bad
| designed SOA systems but IMHO roughly as likely as a SOA system
| communicating with events.
|
| ---
|
| - So use events if you do SOA (for inter service communication)!
| - Never rely on the internal state of another Service. - Make
| sure you don't send events to a specific service, instead "just"
| send events, then all services interested _in that event_ will
| receive it. (Make sure to subscribe for events independent of
| their source not services; E.g. use a appropriate event broker or
| service mech; Sometimes just broadcasting is fine; __Oh and
| naturally storing the event and making that trigger other
| services works, too. At which point we are back at DOA __)
|
| ----
|
| DOA is not bad just IMHO misguiding. If you want to use it look
| at common problems with event based systems for vectors of
| potential problems wrt. accidental internal state coupling as
| many of this will apply to DOA, too (if your system becomes
| complex enough).
|
| Note that I don't mean all problems caused by combining eventual
| consistency + high horizontal scaling with event systems. Sadly
| this is just very often mixed up.
| meowface wrote:
| What are some good articles/resources on event-based SOA? I've
| been hearing about event-driven architectures a lot in the past
| few years.
| discreteevent wrote:
| A central message broker that intermediates between services also
| solves the problems described in the "Problems of scale"
| paragraph. It might be worth mentioning this.
| bencollier49 wrote:
| Hmm. Does this not turn the data access layer into a service bus?
|
| This could get very messy at scale.
| apalmer wrote:
| This was the predominant architectural design about 20 years ago.
|
| There are many strong points for this the general challenges
| are...
|
| Databases don't/didn't have too much in the way of integration
| primitives...
|
| Databases can become a performance bottleneck that can only be
| scaled vertically...
|
| SQL language is not that great as an application programming
| language
| zmmmmm wrote:
| I like this architecture but I worry that in sufficiently complex
| systems it can lead to some significant complexity if there are
| entities within the database that have a lot of contention around
| them. For example, it may end up translating to deadlocks etc.
| when two services that are naive to each other start locking
| tables in interleaved sequences of interactions. It still seems
| wise even if you follow this pattern to segment areas of the
| database to management of particular services or consumers and
| then have those present APIs or message passing interfaces to
| each other. Which leads you half way back to SOA or
| microservices.
| Eyas wrote:
| Yeah, totally. The way I've seen this work is that a single
| service will "own" a record, so you never are running into a
| multi-writer situation for a given record.
|
| While this seems closer to SOA, the key difference here is that
| single Type or Table can still have multiple produces (of non-
| overlapping records). In a trading system, you'd have producers
| of RFQs from makretplace A, B, C, etc. but for each single row
| in that table, the same service "owns" it. So you still get the
| benefits of not caring about the DAG/callgraph or knowing about
| the individual service that calls you.
|
| Locking might be hard if you're doing some transactional
| change. Those become harder to do. But the half good news is
| that shifting to an "Event-based" programming mindset might
| mean you run into less of these.
|
| But yeah, there's a whole new set of drawbacks.
| nine_k wrote:
| But what's the point to keep them in the same table then?
|
| Won't having several independent (and maybe physically
| remote) tables, one per service, solve the problem better?
| You can still `union all` them for analytical purposes.
| Eyas wrote:
| Same schema, and the service doing the query doesn't need
| to worry about which tables to union. (Problem with union
| all is that it reintroduces addressability and a form of
| direct component interaction)
|
| But sure, you can probably implement that with views etc.
| also.
| thecleaner wrote:
| So on a high level redux but for backend (Store-reducers-
| components) ?
| ChicagoDave wrote:
| This is nothing new and generally how SOA began in the 2000's.
|
| One of the critical improvements to SOA is DDD (domain-Driven
| Design) where context matters and boundaries should include the
| service and its data.
|
| Data oriented architecture a bad idea. Period.
| macintux wrote:
| I am skeptical of blanket statements like that. I've yet to
| meet a (real) language or tool or architecture that doesn't
| have merit.
| Groxx wrote:
| Even "Big Ball of Mud"-design has some benefits:
| http://www.laputan.org/mud/
| stilisstuk wrote:
| Any good sources on DDD?
| stevetodd wrote:
| "Domain-Driven Design: Tackling Complexity in the Heart of
| Software" by Eric Evans seems to be the preeminent book on
| the subject.
| dzikimarian wrote:
| It's valuable, but very hard to read. I would recommend
| implementing DDD by Vaughn Vernon first. Also understanding
| event storming helps a lot.
| adamnemecek wrote:
| I thought this book was solid
|
| https://www.amazon.com/Data-oriented-design-engineering-
| reso...
| bestouff wrote:
| I thought it was what started the ECS architectures which are
| all the rage in game engines nowadays, and start bleeding in
| GUI and other fields too. Doesn't seem such a bad idea to me.
| TeMPOraL wrote:
| Nah, ECS probably got backported. On top of that, ECS as a
| term suffers from multiple personality disorder - there are
| different architectures with differing goals all bundled
| under the same name "ECS".
|
| See: https://news.ycombinator.com/item?id=21897824.
| Eyas wrote:
| While it's true that SOA generally began by breaking up
| services into separate binaries while usually keeping one
| centralized database, a lot of the specific principles of DOA
| are a _response_ to the distribution/fragmentation of state
| seen in SOA and micrservices in the mid 2000s.
|
| DOA is often implemented taking advantage of event-driven
| programming, pubsub, and message passing, which as generalized
| practices were not as prominent when SOA began, imo.
___________________________________________________________________
(page generated 2020-03-08 23:00 UTC)