[HN Gopher] System design and the cost of architectural complexi...
___________________________________________________________________
System design and the cost of architectural complexity (2013)
Author : damethos
Score : 309 points
Date : 2023-04-06 16:39 UTC (6 hours ago)
(HTM) web link (dspace.mit.edu)
(TXT) w3m dump (dspace.mit.edu)
| opportune wrote:
| In my experience this problem arises from a lack of consideration
| of two related things, and not doing a third thing:
|
| 1. Not considering how inevitable development issues and bugs
| will affect the entire system. That is, if a bug gets introduced
| here, how bad will the systemic effects be, and how do we prevent
| it? A severe case is if you start writing bad data from one
| component that then needs to be manually backfilled or reverted
| (especially if this then generates yet more bad data further down
| the line).
|
| 2. Not considering how entire-system failures will present
| themselves, and not having a good way to diagnose them.
|
| 3. Failure to develop testing (integration or whole-system
| blackbox) that catches the first case, failure to develop tooling
| (tracing, logging, synchronizing changes across components) that
| assists in the second case. Or instead adjusting the system so
| that the first or second cases are considered.
|
| It's easy to get stuck in a local optimum where a few old hats
| who understand the entire system are the only ones capable of
| predicting and diagnosing failures across components. It's also
| easy to say that less skilled or new engineers just need to put
| in the time to get good, but it's often the case that the old
| hats have a potentially-automatible procedure for narrowing down
| the problem or tracking where the issue came from, or that the
| benefits of separate components don't justify the increased rate
| of bugs and time spent tracking them down to fix them.
| r3trohack3r wrote:
| Having worked in and out of FAANG and with ex-FAANG in startups,
| FAANG engineers have a tendency to treat everything like a FAANG
| problem that demands FAANG solutions.
|
| You'll see these massive systems with event driven architectures
| and layers of nested microservices deployed to hand rolled K8S
| clusters with custom plugins and Cloud Native CI/CD systems
| strapped over the top of a complex repository setup and a complex
| metrics/tracing/logging stack to make sense of all the inter-
| dependencies and lifecycles of a request in this system. All of
| this for a system that is taking <100 RPS and likely won't see
| more than that for many years.
|
| I'm not exaggerating when I say you could host the entire company
| on a single R620 (+ a cold backup) with PID1 managing a couple of
| bash scripts and a few Python/Go/Java/JavaScript processes wired
| up to an SQLite db and a nightly backup.
|
| But their architecture choice does create jobs in our industry.
| These services need teams to maintain them. You have a K8S team,
| a DevTools team, a CI/CD team, teams for the API services, teams
| for the backend microservices, SREs to respond to incidents, etc.
| Those teams all need to be managed, coordinated, staffed, paid,
| etc. so you hire on the management and administrators to handle
| the size of the engineering organization responsible for this
| thing.
|
| So instead of two 4-5 figure servers and a few $100 a month in
| hosting, plus a handful of well compensated engineers who keep
| this thing running, your engineering department is burning 6-7
| figures per month in cloud costs and millions in headcount.
|
| To be clear, some companies need these setups. I do platform
| engineering consulting and help companies build these exact org
| structures and systems I'm talking about here. But I only take
| those contracts when the work is justified. There is a spectrum,
| and very few companies fall on the end of the spectrum that
| demands these solutions.
|
| Today, a majority of the companies in the American market should
| be on serverless offerings that scale to near-zero - simply to
| reduce the need to staff an engineering org to maintain the
| "infrastructure" under the system. You buy that engineering org
| from the serverless vendor with a support contract. I'm not
| setting a mom-and-pop shop up with an R620 because they'll be
| 100% dependent on a high-skill engineer for the rest of their
| existence, which isn't likely to yeild an ROI for them.
|
| Past that you start migrating to VMs or bare metal. That'll get
| you pretty far. Modern computers are Very Fast, you can serve a
| lot of traffic from a single 1U slot.
|
| Very few companies get to the scale where any of these FAANG
| architectures start to make sense.
|
| If I were to take a guess, I think a lot of the FAANG stuff is
| salary chasing. To justify a $250k - $500k salary folks think
| they need to build these fancy architectures to demonstrate their
| value, and the junior engineers and management are all in because
| they're chasing the salary ladder and need to get this experience
| to get into FAANG. But the reality is, with back of napkin math,
| you can save a company _millions_ per year with conservative
| architectures.
|
| For me at least, negotiating contract rates against that pool of
| savings is an unsolved problem.
| paulddraper wrote:
| > serverless offerings
|
| Serverless winds up being complex, in my experience.
| r3trohack3r wrote:
| I've been using Cloudflare and have found the opposite.
|
| You can accomplish an incredible amount for a small business
| with Pages and a single monolithic function. Once you outgrow
| that, IMHO, it's time to consider moving to a server - not to
| a more "complete" serverless offering like lambda.
|
| The migration path for this is pretty staightforward since
| the API surface area of a cloudflare worker is pretty small
| (depending on whether, and how, you are using Durable
| Objects, KV, D1, R2, etc. things can get a little harder). I
| can usually port a cloudflare worker to a Node.js service in
| < a day of work.
| kerneltime wrote:
| MIT has an amazing program for System Design and Management. Dan
| and I are both graduates of this program. Some of the courses I
| would recommend include System Architecture, System Safety, and
| System Dynamics. Most of the content is available on OCW.
|
| What is taught is not software-specific, but is entirely
| applicable to software, outside of the world of 'throw everything
| on the wall and see what sticks' as long as the venture
| capitalist can be shown growth at all costs. I wish more software
| developers were mindful of complexity and architecture.
| agomez314 wrote:
| Could you elaborate a bit more on what you got out of the
| program? I never heard about it before but i'm intrigued. Did
| you find a particular course memorable?
| kerneltime wrote:
| The tag line within SDM was that it is a program for those
| who want to lead engineering and not leave engineering (MBA)
| I think the meta framework for thinking and being able to
| step away from the madness of releasing a v1 product and
| having tools for thinking about the bigger picture. Also,
| MIT.. it is a very rewarding ecosystem to be in.
| waitingforset wrote:
| Sounds super interesting. Can you link to the specific OCW
| courses? I see a few from different departments.
| kerneltime wrote:
| ESD.34 ESD.342 16.863J 15.871 15.783J ESD.33 to name a few
| Also, 15.965 my favorite and offered by my thesis advisor
| Michael A. M. Davies: Based on which it is likely that OpenAI
| won't be walking away with the cake.
| uptownfunk wrote:
| Can you elaborate on the cake comment?
| [deleted]
| jen729w wrote:
| In the late 90s I travelled the world with a couple of CD-ROMs
| installing NT 4.0 on individual physical servers. I understood
| the entire stack. I was the only engineer and I was often very
| remote.
|
| In the mid 2000s we installed Server 2003/8 on a whole bunch of
| physical servers in a data centre. This was for one of
| Australia's 'big four' banks. Me and a team of less than ten,
| most of whom I still know, managed the entire thing. We were the
| '3rd level infrastructure' team. Most of us understood most of
| the stack end-to-end, although there were some specialities.
|
| In 2023 I work for an IT integrator. These days we use Azure,
| because Microsoft has fooled us in to thinking that it's cheaper.
|
| My current project is, essentially, a website migration. Not even
| a big or complex website. The project has been running for 6
| months. I only joined recently so I can't be sure but we've
| probably burned AU$2m. The schedule I'm managing has us doing the
| cutover in June. That is optimistic.
|
| The architects are currently trying to work out how Azure
| [details] connect to Azure [details] while still [security] and
| being able to [complex integration].
|
| Every day some new issue appears that the architects have to
| figure out how to work around.
|
| No single person has a goddamned clue how the end-to-end thing
| fits together. Not one, not a clue. That scares me.
|
| 'Architectural complexity' has crept down to the level of
| infrastructure. It's painful to see. I hope it stops soon but I
| am not hope _ful._
| throwawaaarrgh wrote:
| Automation moves complexity from one place to another. In this
| case the complexity moved into silos.
|
| Before, all the pieces of the architecture were in a couple
| hundred *.cpp files in one directory hierarchy. Now it's
| millions of files across thousands of directories each turned
| into a service run by a dedicated team, and none of them can
| see each other.
|
| You _can_ become an expert in all those pieces. But it requires
| actually using them all together, to discover the parts that
| don 't fit and how to work around them. This is why the modern
| ideal of completely independent APIs is a terrible design.
| There is literally no way to know if anything works with
| anything else until you try to run it in production. Monoliths
| are terrible at scaling, but easy to understand. SOA apps are
| great at scaling, but impossible to understand.
|
| Good luck on your migration. Changing the wheels on a moving
| tractor trailer always sucks.
| lifeisstillgood wrote:
| Isn't the point that there is a line somewhere that one app
| stops being one single app and becomes "eco system" - and you
| stop having deterministic understanding and start having
| "town planning" and "social expectations".
|
| I mean my use of bad metaphors kind of underlines the point
| we don't really understand his problem - but the large
| organisation that builds these is itself an example of such
| impossible to understand interactions - maybe we will learn
| from Azure etc and take those learnings into taking our own
| orgs.
| lifeisstillgood wrote:
| This is the essence - that a system needs to fit inside one
| persons head. All of it. There maybe a few people who
| understand all of a Boeing airplanes systems - there certainly
| were when they first flew 747s. Maybe a couple now.
|
| but once it stops fitting inside one persons head then the
| thing is literally only possible to design by committee - it
| cannot fit together and perhaps should stop being called a
| single system
| pixel_tracing wrote:
| Looks like we need a better system architecture for the MIT
| website hosting the material as it's overloaded with too many
| requests
| avi_vallarapu wrote:
| I believe that the great examples always arise from OpenSource
| projects. The design and modular code always play a great role in
| increasing the ability to customize or add features and also have
| an increase in collaboration from more volunteers. Developer life
| gets more interesting with a massively improved code quality when
| some tough decisions are taken much earlier.
| uptownfunk wrote:
| A wise engineer who I worked closely with told me "the job of the
| engineer is to manage complexity. Engineers don't like
| complexity".
|
| There are those hard-learned sutras that just make life so much
| easier down the line.
|
| Down the line always comes (unless you are playing career musical
| chairs and are willing to gamble that you won't be around to have
| to deal with the code mess when there's a sev2-, but as they
| say.. karma's a b..."
|
| Avoid premature optimization
|
| TDD
|
| Avoid complexity
|
| Readable code
|
| Documented code
|
| Probably helps avoid most problems.
| alphabetatheta wrote:
| The hardest part here is the tradeoff between architectural
| complexity as you build systems and speed of shipping product.
| Earlier stage companies will ship ship ship and ignore good
| architecture practice. At some point, it will come back to bite
| you if your company lives to see another day.
| manv1 wrote:
| This is why you need good people when you start: because good
| architecture is the difference between fast iteration and slow
| iteration.
|
| A good architecture will allow you to make changes easily. A
| bad one doesn't. It's actually pretty simple, conceptually
| speaking.
|
| If you believe that "late stage" companies make correct
| architecture choices you're probably incorrect. It's not about
| late stage or early stage, it's about knowing how to build
| software from scratch in a way that you don't hamstring
| yourself (and others) down the road.
| teacpde wrote:
| I would say it goes beyond early stage companies and extends to
| later stage product-driven companies, especially those that
| value time-to-market than anything else.
| jasondigitized wrote:
| Always a tradeoff. You can build a Ferrari but may end up
| caught in the garage while a competitor has paying customers
| doing laps around the feature track collecting $200 at every
| turn.
| duxup wrote:
| I would argue it's the later stage company who doesn't take the
| time to fix it / pay off that tech debt who fails.
|
| I'm not against picking up some tech debt here or there if you
| pay it off.
| [deleted]
| austin-cheney wrote:
| Complex means many, not challenging -
| https://www.dictionary.com/browse/complex
|
| The most practical measure of complexity is duplication. Are
| there two, or more, pieces of code accomplishing the same or a
| similar job? That is complex. The solution is to refactor the
| many parts into fewer parts.
| tjr wrote:
| _composed of many interconnected parts; compound; composite: a
| complex highway system._
|
| _characterized by a very complicated or involved arrangement
| of parts, units, etc.: complex machinery._
|
| I would say you could have a software system with many
| interconnected parts, and/or with a complicated or involved
| arrangement of parts, without having duplicate code or
| duplicated functionality.
| austin-cheney wrote:
| Duplication is just one of many potential measures. The end
| goal though is converting many to few.
| ryanklee wrote:
| No one uses the word complex to mean many.
|
| Further, sometimes 1 thing is overloaded in difficult-to-
| understand ways, and so there should be more things, not fewer.
| Sometimes there are many things that should be 1 thing.
|
| There isn't just one good measurement of complexity, as
| complexity isn't inherent to things or systems in themselvesas,
| rather complexity is a feature of perception, which gets
| confused in all sorts of irreducible ways.
| austin-cheney wrote:
| This popular language author invalidates your comment:
| https://www.infoq.com/presentations/Simple-Made-Easy/
| duxup wrote:
| I'm always thinking about "Can I (or anyone) get back into this
| easily 6 months from now?"
|
| In my situation, I probably will have to do that so there's a
| selfish reason there for sure.
|
| I recently had a whole series of frustrating situations where I
| dug through rediscovering how old code / systems work to make
| small changes or to find out the small change was enormous.
| Really deflating stuff. It's not my fault but it can be so
| demoralizing. Feels like a weight on you... I was done for the
| day after both of those horror shows.
|
| Then yesterday I had a 3 day project start and in 2 hours I ...
| did the thing. It was super flexible / powerful, handled errors
| gracefully, and easy to change / test. All because a year ago
| someone (well myself and anther person) took the time to simplify
| the original spaghetti code that originally existed and break it
| into more digestible functional-esque chunks. Dropping something
| "in between the chunks" (fancy technical terms here) was easy to
| do, test and read. Completely the opposite experience, it was
| energizing and fun.
| latchkey wrote:
| > I'm always thinking about "Can I (or anyone) get back into
| this easily 6 months from now?"
|
| As I age, my memory is getting worse and worse and I realize
| that quite clearly. Therefore, I always try to write
| documentation as I'm writing code, so that I can remember why I
| did something. It helps a lot so that 6 months later, I can do
| exactly that... but I also know that anyone else looking at my
| stuff will also realize why things are the way they are.
| duxup wrote:
| I'm the same way, notes, good documentation, etc.
|
| Sometimes I think I get some tasks done faster than when I
| was younger...
| bob1029 wrote:
| > All because a year ago someone (well myself and anther
| person) took the time
|
| I've been saying for half a decade or longer:
|
| "Going slower today means we can go faster tomorrow".
|
| It took a long time for some of my team members to process
| this, but I believe they've all taken it to heart by now. The
| aggressive, rapid nature of a startup can make it very
| difficult to slow down enough to consider more boring, pedantic
| paths. Thinking carefully about this stuff can really suck
| today, but when its 3am on Saturday and production is down, it
| will all begin to make a lot of sense.
|
| Having empathy for your future self/team is a superpower.
| RangerScience wrote:
| "Slow is smooth, smooth is fast."
| davidthewatson wrote:
| This has won countless races for just about every top F1
| driver you can name for decades, prolly WRC too. That old
| analog world transfers nicely to digital in video gaming.
| Sadly, it's not more widely accepted in software
| development though software design and software deployment
| seem to have caught on.
|
| As an old C++ hacker, I'm waiting for the day when modern
| C++ shops read Accelerated C++ from Koenig and Moo circa
| two decades ago. Then, I could rejoice in someone anywhere
| writing C++ code that more closely resembled the python-
| esque C++ masquerading as pseudo-code in that book.
|
| More sadly, I just keep seeing people emulate bit-twiddling
| from yesteryear when the compiler likely optimizes a fair
| bit of this.
|
| The cyclomatic complexity scores in the paper look off by
| an order-of-magnitude but they may be better than the laugh
| riot I've measured in the last few years and my math may be
| failing me at runtime.
| hbrn wrote:
| Racing is a very flawed analogy: the big difference
| between software development and racing is that:
|
| 1. F1 paths are known in advance.
|
| 2. The major unknowns in F1 are your competitor
| behaviors.
|
| Compare that to a typical startup: you're mostly riding
| in the dark on a track you see for the first time, and
| your major unknown is customer behavior.
| [deleted]
| cogman10 wrote:
| Yuup. Unfortunately, there's profit disincentives to this.
| Time to market for new features is a thing. Getting out
| features fast gets you kuddos from the suits. So you get a
| class of dev that spins out code like wickedly fast while at
| the same time leaving a mess for others to clean up.
|
| It's hard to correct that sort of behavior (without being an
| actual manager that knows code and can spot bad
| architecture).
| amw-zero wrote:
| Have you ever measured this alleged speedup when "tomorrow"
| comes?
| bob1029 wrote:
| Measured relative to what?
| temporallobe wrote:
| > I'm always thinking about "Can I (or anyone) get back into
| this easily 6 months from now?"
|
| People I work with get very annoyed with me because of this,
| but I am obsessive about documentation for this reason. Sure,
| it requires a lot of tedious writing and screenshots, etc., but
| it has saved me countless times. I still can easily get back
| into things years later thanks to documentation.
|
| The caveat is that when people who are not as passionate as you
| maintain a product and seemingly forget about documentation.
|
| In the old days, documentation was a very strict requirement on
| many of projects I was involved with. Now, in modern agile
| projects, it's an afterthought at best, despite having amazing
| documentation tools that we've never had before.
| bosie wrote:
| Do you mind expanding on which tools you are using for
| documentation (creating, maintaining etc) please?
| amw-zero wrote:
| What ways have you found for keeping the documentation in
| sync across frequent changes?
| r3trohack3r wrote:
| For my consulting, I primarily practice "reference first
| architectures."
|
| The idea is we identify the rough shape of what we are going to
| build and the components needed to deliver it (Linux?
| Terraform? K8S? HTML/CSS/JS? etc.).
|
| Next we measure up what we can "take for granted" for the
| engineering skillset the organization hires for. Then we pick
| books, official project documentation, etc. that will act as
| our "reference." We spend our upfront time pouring ourselves
| into this documentation and come away with a general
| "philosophy" of an approach to the architecture.
|
| Then we build the architecture, updating our philosophy with
| our learnings along the way.
|
| At the end of the project, we commit the philosophy to paper.
| We deliver the system we built, the philosophy behind it, and
| the stack of references used to build the system.
|
| What this means is I can take any engineer at the target level
| they hire for, hand them the deliverable and say "go spend a
| week reading these, you'll come back with sufficient expertise
| to own this system."
|
| It also acts as documentation for myself for future contracts
| if I get brought back in. Prior to starting the contract I can
| go back in and review all of those deliverables myself to hit
| the ground running once I'm back on the project.
| jrvarela56 wrote:
| This sounds like the right way to do it. For me it has been
| tough to come up with principles that don't sound like they
| apply to any system. You start off with a generic CRUD app
| but as it grows the default/usual web framework constructs
| tend to leave you with a ball of mud. You can couple anything
| in there together and since you're pressed for time, you tend
| to do it. Abstractions feel premature and when they start
| emerging there's lack of conviction to push through with them
| and clean up the whole thing.
|
| Do you have any starter resources to come up with principles
| for a system? Maybe something showing how certain principles
| lead you to implementation choices that would've been
| different under another philosophy.
| r3trohack3r wrote:
| For a 2 week Terraform audit, these are the high level
| philosophy points I put together. The final doc was 10
| pages. Each point lists the reason for choosing this
| approach and any trade-offs that come with it.
|
| * Small composable Terraform modules
|
| * Don't manage IaC declarations alongside code in a
| polyrepo
|
| * Direnv for managing env configs across repos
|
| * Manage k8s using k8s manifests and not terraform files
| (kubectl provider gives us this)
|
| * Delegate flux management to flux-cli
|
| * Auto-unseal Vault to capture and protect the vault token
|
| Then a list of recommended reading:
|
| * Terraform Up and Running
|
| * Building Microservices
|
| * Site Reliability Engineering
|
| This list is more tactical since we didn't build the
| system, we were auditing their current setup.
| mattacular wrote:
| > Don't manage IaC declarations alongside code in a
| polyrepo
|
| Can you elaborate on this one?
|
| (And thanks for the interesting comments)
| r3trohack3r wrote:
| > The source code for services should not be coupled to
| their deployments when managed in SCM. The lifecycle of
| changes for infrastructure are different than the
| lifecycles of artifacts for services. Any artifact should
| be able to be configured and deployed into any
| (supported) infrastructure configuration. For example,
| running git revert on a service should be able to yield a
| deployable artifact regardless of how the infrastructure
| is configured. By coupling these, you tie changes to
| infrastructure to changes in services. A rollback for
| your service can also unintentionally rollback how that
| service gets deployed - and avoiding that requires an
| engineer hold both the context of the infrastructure and
| the context of the service in their head whenever they
| are manging git history. It becomes difficult to deploy
| an older version of a service for testing. It also breaks
| git-bisect since, now, searching for a regression in
| software also changes how that software is getting
| deployed. This is an extension of managing IaC as small
| composable modules. The source code for a service should,
| itself, be viewed as a composable module. That module may
| take on a different format than other pieces of IaC (i.e.
| a .tar.gz, a .deb, or a docker image instead of a
| terraform module or a terraform provider) - but it's API
| contracts are still drawn around the unit of deployment
| and not the monolithic infrastructure stack that will be
| deployed with it.
|
| This, of course, does not apply to projects using a
| monorepo.
| perfmode wrote:
| how would one manage independent infra and services in a
| monorepo?
| [deleted]
| [deleted]
| amw-zero wrote:
| Sounds like an architecture decision record. Here's an
| example ADR template:
| https://github.com/joelparkerhenderson/architecture-
| decision....
| unixhero wrote:
| I cannot believe this is free. This is going straight to my read
| soon queue
| hinkley wrote:
| I've long had an allergy to accidental complexity but recently
| had some new aspects illuminated for me.
|
| I had a block of code that had been added to by others, making it
| a bit of a recursive mess with a handful of bugs that were hard
| to reproduce let alone fix. When I finally got tired of it all, I
| sat about to replacing the recursive code with some iteration,
| and ended up with DP code, which avoided a bunch of duplicate
| errors and the need for caching.
|
| Without all the DP terminology, it's what other people would call
| building a (programmatic) plan and then executing it, rather than
| a depth first search algorithm. I used to use this quite
| regularly, but have only used it a couple of times on my current
| gig. Strictly speaking planning an entire action before starting
| it might result in a bit extra data hanging around because of
| building the data structures ahead of time, but at the end of the
| day it allowed me to eliminate a whole lot of steps that seemed
| necessary, and also remove duplicate warning messages. It wasn't
| necessarily faster than it could be, but it was way faster than I
| ever expected it to be.
|
| Writing obvious code with clear actors and data flow steps makes
| it a lot easier to add features, and make performance
| improvements. Obscure code leads to more obscure code
| (qualitatively and quantitatively).
| nmstoker wrote:
| Should have (2013) in the title.
| dang wrote:
| Added. Thanks!
| beders wrote:
| Architectures need to be judged by their difficulty to make code
| and data changes.
|
| Changes in a stratified architecture are much simpler than in a
| layered design. (Changes in a layered design almost always mean
| changes to all the layers)
| germamme wrote:
| Software needs regular refactoring just like cars need regular
| maintenance. It is very difficult to determine how much time and
| effort to spend working on making it extendible when you're
| writing it the first time (even the product owner might not know
| how it will be used initially) but after a few months in
| production and a few feature requests, you'll get a better idea
| of what the pain points are and will be in a much better position
| to refactor. The problem is convincing the people who cut the
| check to allow engineers the time to do it
| discreteevent wrote:
| It's a balancing act:
|
| "There is no theoretical reason that anything is hard to change
| about software. If you pick any one aspect of software then you
| can make it easy to change, but we don't know how to make
| everything easy to change. Making something easy to change makes
| the overall system a little more complex, and making everything
| easy to change makes the entire system very complex. Complexity
| is what makes software hard to change."
|
| https://martinfowler.com/ieeeSoftware/whoNeedsArchitect.pdf
| eternalban wrote:
| Worth pointing out that this study does -not- equate
| "architectural complexity" with abstraction. Many consider use
| of "hierarchies, modules, abstraction layers" to be
| 'unnecessary complexity' where as the thesis clearly states
| they "play an important role in controlling complexity." OP is
| not a call to get rid of software architects, arguably it says
| ~'hire competent architects as bad architecture negatively
| impacts faults, staff turnover, and product success.'
|
| "Architecture controls complexity", and under-/poorly- designed
| architecture while superficially "simple" will give rise to un-
| intended complexity. Microservices are the du jour poster child
| here.
| ParetoOptimal wrote:
| Not a direct response, but some thoughts that come to mind
| without any direct conclusion:
|
| Good abstractions make simpler software. Leaky abstractions
| multiply the complexity of software by a lot.
|
| Some respond to this by making "simple" software that
| dispenses entirely with abstraction. This ends up in a lack
| of architecture where complexity still multiplies, though
| perhaps less than the typical mix of mostly leaky
| abstractions and a few sound ones.
|
| However, it's kind of the nihlism of software and throws away
| the opportunity for us to actually improve our craft... so
| I'm not all too interested in it.
| jeremyjh wrote:
| The uncomfortable truth is that software development is a
| series of decisions made day after day, and many people
| simply cannot reliably make one good decision after
| another. You see this in poker and chess too, there are
| many people who will never be good at those games (like
| me), and there are many people who will never be good at
| software. Demand for good developers outstrips supply and
| then you have the whole measurement problem on top of it.
| At least with poker and chess we know who is good at it,
| and who is not.
| tunesmith wrote:
| People regularly forget Rule Zero of patterns: Don't
| implement a pattern if it doesn't solve a problem. That's the
| difference between unnecessary complexity and controlling
| complexity.
| q7xvh97o2pDhNrh wrote:
| IMHO people aren't forgetting it -- it's pretty commonly
| the other way around.
|
| Anecdotally, the people I see advocating for spaghetti
| architecture are often _utterly convinced_ they 're solving
| some critical problem. Conversations with stakeholders then
| turn into fearmongering -- often helped by the fact that
| most stakeholders don't care about architectural nuance --
| and it becomes easy to wave off any competing simpler
| architectures by deriding them as "taking on tech debt." In
| general, it's surprisingly easy to play corporate politics
| to bring a convoluted architecture into reality.
| farrugp wrote:
| Can't wait to read this, really resonates with me as I'm dealing
| with this right now at a big FAANG company.
|
| In my experience the problem is that as with all things it's all
| about balance. We shouldn't throw away architecture entirely and
| write stupidly simple, quick solutions because they will be
| messy. But we also shouldn't over-abstract things so much that
| only the person/people who built the system can understand it and
| work with it. Both could have dire consequences for an
| organization, making building new features and delivering value
| to users slow, difficult and costly.
|
| Once I finish reading this MIT paper, I want to dig further into
| exactly what makes software 'complex'. In my experience:
|
| - too many layers of indirection - overly theoretical concepts
| not grounded in real world thinking - lack of documentation - bad
| naming - lack of testability - tight de-coupling - following
| 'best practices and patterns' without clear justification -
| trying to solve for problems before they exist
|
| We should be building systems that are grounded in concepts that
| are easily understandable - which is exactly why Object Oriented
| Programming has been so successful. We write programming
| languages as a means of communicating with each other about
| program logic, why not do it in terms that we as users already
| understand in the real world?
| neals wrote:
| I've gotta say, this study really hits home for me. As a
| developer who's worked on a few projects with some gnarly
| architectural complexity, I can totally see how that would lead
| to these kinds of costs. Just last year, I was part of a team
| that had to deal with a super convoluted codebase that felt like
| a patchwork of different styles and approaches, with no clear
| hierarchy or modularity. It was an absolute nightmare to navigate
| and make changes to, and our productivity took a nosedive. Not to
| mention the bugs that kept creeping in and the insane amount of
| time we spent debugging. I even saw a few of my colleagues jump
| ship because of the frustration. I wish management would've
| realized the potential benefits of investing in some proper
| refactoring efforts to improve the architecture. It might've
| saved them a lot of money (and headaches) in the long run!
| tunesmith wrote:
| Two things that get in the way of "long run" thinking.
|
| What's the timeframe that teams should adopt? If it will pay
| for itself in three years, is that too long? What's a good
| argument to make here that will appeal to the bean counters
| that are used to thinking in terms of quarterlies? I personally
| am comfortable with "eventually/infinite" but that is a tough
| sell.
|
| Capitalization vs Expense. Maintenance/refactoring is not
| capitalizable, and thus gets discouraged in businesses that
| care about P&L. New (capitalizable) projects/features are
| encouraged instead. What are some good ways to encourage
| maintenance/refactoring in this kind of environment?
| treis wrote:
| Stuff like this contributes massively to a culture of low
| expectations too. When it becomes accepted that simple things
| take a long time to do the normal reward system goes flying out
| the window. Both internally and organizationally.
|
| Internally because the sense of accomplishment from seeing the
| new bit of functionality is tiny compared to the effort it took
| to get done. Organizationally it becomes harder to reward
| productivity because with no sense of how long something should
| take there's no way of knowing if an engineer is fast or not.
|
| It's a productivity death spiral. Engineers slack off because
| there is no internal or external reward for doing good work.
| That slacking off slows development and then that velocity gets
| accepted as normal. Engineers continue to slack off against the
| new "normal" and establish an even slower normal to slack off
| against. That's how hours long tasks turn to days and then
| weeks and then months until the project dies.
| holoduke wrote:
| I still dont know what is better. Create quick throw away code
| you replace in a few years. Or build a well structured system
| that lasts longer. I have worked all my carreer in only the first
| 0-8 years of company startups. Most of the joy and success I got
| from 'quick hacking together working systems'. A lot of people
| arround me dont share that opinion and are better suited at
| structured companies. Maybe there is no better.
| bob1029 wrote:
| > Or build a well structured system that lasts longer.
|
| What if you had the ability to build a system that was
| initially structured well enough such that it could be made to
| last indefinitely? From a cost/benefit standpoint, is the Ship
| of Theseus approach not the most ideal for a business owner?
|
| Even for a developer, the notion that you have to constantly
| "trojan horse" your new product iterations into production via
| the existing paths means you will achieve mastery over clever
| workarounds and other temporary shims. Once you gain competence
| in this area, it is possible that you will never want for a new
| boat again. You start to get attached to the old one and all of
| the things it takes care of that you absolutely forgot about
| years ago.
| WJW wrote:
| > if you had the ability to build a system that was initially
| structured well enough such that it could be made to last
| indefinitely
|
| _If_ you could have that, it would obviously be amazing. Do
| you (or anyone) have that ability though? So far it seems the
| answer is no. People are just not smart enough to predict the
| future.
|
| As a more nuanced answer, a system that is scalable enough to
| grow to 1000x the current load is usually way too expensive
| to build with the resources you have "right now". The best
| you can do is build a system for 10-100x the current load and
| hope you haven't forgotten any edge cases, but usually you
| encounter them way sooner than 10x the load. Building so that
| you can easily refactor your current system is the way to go,
| but even then you will sooner or later run into problems your
| original design did not consider.
| bob1029 wrote:
| The point I am trying to make is that if your system can
| survive in production under realistic workload for _some_
| period of time (i.e. the boat floats & makes it out of the
| harbor), then what is preventing you from taking that
| success and incrementally moving it further into a
| preferred direction?
| smith7018 wrote:
| I've been a software engineer now for 12 years and I think
| there's an obvious answer: architecture. If you build software
| with a solid architecture from it's inception, then "hackily
| building features" will happen much faster and cleaner than a
| mess of a codebase. I wish more people realized that "build
| things fast now without caring about quality" translates to
| years of "God, I wish we could just throw all this out" and "no
| one knows how this part of the code base works but it does so
| we don't touch it." Plus, realistically speaking, adding that
| solid foundation shouldn't take _that_ much longer if you know
| what you're doing.
| jasondigitized wrote:
| When you have a savvy competitor that is taking the opposite
| tack and killing you in feature bake-off, you don't have this
| luxury. Your Ferrari is being built in the garage while
| paying customers are driving around in your competitor's
| Yugo.
| majgr wrote:
| Every messy project starts from 'architecture'. At the
| beginning it is clean, and super well organized. Then,
| instead of, these well thought out before extensions, some
| breaking feature needs to be implemented. And it is
| implemented in the most aligned to starting 'architecture'
| fashion. After couple of years of these implementations a
| mess is created.
|
| It is better to have simple solution at start, then, after
| each breaking feature, whole thing needs to be refactored,
| because initial assumptions about that software might have
| been changed.
| dasil003 wrote:
| This assumes you know where you are going. All code is a
| liability and all architecture decisions are tradeoffs.
| codr7 wrote:
| Speed is the main reason I do architecture.
|
| In the end it mostly comes down to reducing complexity, but the
| goal is always allowing new features to be added as fast/easy
| as possible.
|
| Because I'm lazy.
| hbrn wrote:
| No plan survives first contact with reality: "well structured
| systems" usually get thrown away in the same few years. The
| only difference is that they often get thrown away along with
| the company that built them.
|
| Quick throwaway code is orders of magnitude better (unless
| you're landing airplanes or building x-ray devices). Especially
| if you're consciously treating it as a throwaway.
|
| The only way to build well structured system that lasts long is
| to have vast domain expertise, meaning having done _literally_
| the same thing multiple times in the past. This rarely happens
| in general, and pretty much never happens if you 're
| innovating.
| iancmceachern wrote:
| "we found that differences in architectural complexity could
| account for 50% drops in productivity, three-fold increases in
| defect density, and order-of-magnitude increases in staff
| turnover."
|
| I think I can speak for many of us technical professionals when I
| say, been there, done that.
| pphysch wrote:
| Yep, working with well-architected software systems is the
| difference between a low-stress well-paying job and rapid
| burnout. No surprise that the latter results in massive
| turnover.
|
| It's remarkable how "add feature X" can be a 1 hour task or a 1
| month task depending on whether the system was designed to
| evolve and scale in that direction. But the non-technical
| management or customer just sees it as a simple request, and
| the (Nth) software developer is left to pick up the pieces.
| jkubicek wrote:
| This is _exactly_ why I think the "myth of the 10x engineer"
| is so obviously false.
|
| An average software engineer will create an overly complex
| system.
|
| If a skilled engineer can come in and create something that
| doubles team productivy, decreases bugs by 60% and improves
| retention across the org by 10x, that's huge! That's not a 10x
| engineer, that's a 100x engineer.
| contemplatter wrote:
| _That 's not a 10x engineer, that's a 100x engineer._
|
| You're speaking of ChatGPT!
| nerpderp82 wrote:
| Engineers that make simple systems are not rewarded is most
| organizations. In a ten person company, sure thing! In a
| large corporate org, no way.
| pixl97 wrote:
| This is why small organizations produce stuff, then large
| organizations buy it and sell it as a product.
|
| As an engineer you just have to ensure you have equity in
| the small company before it's purchased and/or hope the big
| company buys your company rather than just copying it and
| fighting you in court for a decade.
| travisgriggs wrote:
| Is this because as groups grow in size, high level decision
| making power tends to move up to less informed management
| individuals?
|
| You'll end up with a "I programmed a little bit back in the
| day" person who makes decisions based on how to currently
| keep their staff best utilized. Tools choices, boundary
| points, stacks, become chosen based on goals that are not
| rooted in design simplicity.
|
| I've watched this story play out again and again and again
| at companies. Upper management will promote these types of
| individuals into lower/middle management because
|
| a) they're deemed actually reassignable whereas a really
| talented Ux designer is obviously adding the most value
| doing that
|
| b) people buried/invested in tech stack/design details are
| harder for owner/operators to relate to than individuals
| more like themselves.
| api wrote:
| As groups grow in size it becomes increasingly hard to be
| informed as management. Eventually it becomes near
| impossible.
| kps wrote:
| At a place I worked, they explicitly rewarded 'solving
| complex problems'. It's actually surprising there weren't
| _more_ cobra farms.
| dustymcp wrote:
| Man the dead weight around development in corporations are
| mind numbing..
| nerpderp82 wrote:
| The institutionalized brain damage is real, I see bright
| new hires coming in, within 3 months they are towing all
| the lines, spouting tautologies, seeking alignment with
| corporate velocity vectors to gain momentum, generate
| impulse. Talking in non-specific platitudes that could
| literally mean anything and thus mean nothing.
|
| The biggest problem with a simple architecture and often
| why someone would complexify it up, is as a protection
| mechanism so you don't get some performance review
| motivated yokel doing a drive by to add in a feature and
| look like a rock star _because it was so damn easy_. NSS,
| that was the next step. You see this in open-source as
| well. Someone builds a beautiful foundation, and Steve
| rolls up with a submarine pull request to slam that
| cherry on top.
|
| A complex architecture ensures that you can adequately
| gate keep.
|
| The next aspect is you have to relentlessly say no to
| feature requests that literally destroy the system to add
| some new feature, the pressure to smash it to bits and
| make it a no longer simple system is such a perverse
| incentive that unless you have support from on high,
| every simple system will decay into a total mess, then
| the finger-painters will have moved on to some other
| system to predate on.
| a4isms wrote:
| FYI, it's "toeing" the line. It's an expression from
| track and field, where at the start of each race the
| competitors place the tip ("toe") of their foot on the
| start line when they're ready to run.
|
| Similar to the expression "Up to scratch." In a boxing
| ring, there is a line in the ring that boxers must come
| to to begin the fight. To be "up to scratch" is to be
| ready and worthy of the fight, and from that, to be of
| acceptable quality or capability.
| pksebben wrote:
| Minor correction for your correction:
|
| "Toe the line" most likely has it's roots in military
| tradition [1]. To toe the line is to stand at the line
| for inspection - it's still used that way today (I had to
| do it myself in boot camp).
|
| Relevant, as in this context, it means being obedient to
| the hierarchy.
|
| 1 - https://en.wikipedia.org/wiki/Toe_the_line#:~:text=Th
| e%20mos....
| a4isms wrote:
| Most interesting, I do believe you're right!
| piotr-yuxuan wrote:
| Interesting, I hadn't seen #:~:text=abc before. It does
| nothing on my browser (Firefox 109.0b9 on macOS) but is
| it inteded to highlight text?
| aidos wrote:
| It does on Chrome. I believe it's used from Google search
| results to link you to the relevant place in the page.
| johnmaguire wrote:
| > The next aspect is you have to relentlessly say no to
| feature requests that literally destroy the system to add
| some new feature, the pressure to smash it to bits and
| make it a no longer simple system is such a perverse
| incentive that unless you have support from on high,
| every simple system will decay into a total mess, then
| the finger-painters will have moved on to some other
| system to predate on.
|
| Would you rather have a simple system that does nothing,
| or a complex / difficult-to-work with system that solves
| customer needs? It's easy to go too far in either
| direction.
| whstl wrote:
| I never really had to make this choice. But I often had
| to choose between a system that is reasonable and
| actually solves the needs of customers, versus a system
| that is complex, made in a hurry and full of badly made
| features that only some project manager cares about. I
| know which one I prefer.
|
| Just today, for example, I removed several thousand lines
| of code from a very complex form that no customer ever
| really used other than for testing. My boss told me "just
| nuke it". Life goes on.
| PaulStatezny wrote:
| > Would you rather have a simple system that does
| nothing, or a complex / difficult-to-work with system
| that solves customer needs?
|
| With respect, this is clearly a false dichotomy.
|
| Creating the "simplest possible system" that solves
| customer needs is the entire point of this discussion.
| Avoiding as much complexity as possible brings enormous
| benefits. (Less bugs, less employee turnover, higher
| productivity, etc.)
|
| Never accept the notion that systems _must_ be
| "difficult-to-work with" in order to solve customer
| needs.
| hbrn wrote:
| In my experience, most of the software complexity is
| accidental, i.e. it doesn't directly emerge from customer
| needs.
|
| And even if your framework is to use customer needs as
| justification for complexity, people will still use
| something like Kubernetes for an app with 10 monthly
| users, because it "helps us deliver features faster, have
| zero-downtime deployments and be scalable".
|
| Most (non-product) engineers have a natural inclination
| to introduce more complexity, and will use mental
| gymnastics to justify it.
| babyshake wrote:
| > non-specific platitudes
|
| Anything specific that tends to be used often?
| whstl wrote:
| "We need to build something that scales" to justify
| virtually every single source of complexity.
| damiankennedy wrote:
| "Move fast and break things".
|
| At 48 I've stopped fixing other people's things.
| nerpderp82 wrote:
| It is org specific, I have already triggered myself and
| need to find a safe space.
| bradstewart wrote:
| Appealing to "best practice" without concrete, specific
| reasoning is a big one.
| waynesonfire wrote:
| I don't think it has to be so nefarious. What I've
| experienced is developers taking the path of least
| resistance and failing to perform proper analysis to
| arrive at a less complex architecture.
|
| Why are folks taking the path of least resistance becomes
| an interesting question.
| WakoMan12 wrote:
| i want to die
| niels_bom wrote:
| Reminds me of "I apologize for such a long letter - I didn't
| have time to write a short one."-- Mark Twain
| iancmceachern wrote:
| Exactly this. I've done well for myself by focusing on
| building things that work, and shipping them on time.
| Anything else is extra and not to be done at the expense of
| the first two things.
| kps wrote:
| > order-of-magnitude increases in staff turnover.
|
| I suspect this goes both ways. I have worked at a place
| characterized by frequent internal shuffles, and more than once
| spent ages reverse-engineering code when the original team
| could have handled the problem in a fraction of the time.
| jasmer wrote:
| This is the economic value of refactoring, right on paper.
|
| 'Complexity is the mind-killer' - Linus Atreides
___________________________________________________________________
(page generated 2023-04-06 23:00 UTC)