[HN Gopher] Why software ends up complex ___________________________________________________________________ Why software ends up complex Author : kiyanwang Score : 126 points Date : 2020-12-13 14:34 UTC (8 hours ago) (HTM) web link (alexgaynor.net) (TXT) w3m dump (alexgaynor.net) | dustingetz wrote: | 1) You can't see software, which combines with 2) a company with | bad sales but great engineering is a dead company, which results | in 3) decisions trickle down from sales but salesmen are blind to | the software. Proposed Solution: Programmers should learn sales. | (So it's diversity at the root of all problems !) | breck wrote: | Proposed Solution: make it so that you CAN "see software". this | is what I work on, but also things like Sublime's Minimap are | huge trends in this direction. | dustingetz wrote: | could work but we'll need to condense 50k LOC crud spa into a | flowchart without loss of essence. (that's what i work on). | what is your angle? i checked out treenotation.org , is there | more? flowchart is graph | breck wrote: | It's doesn't work with current languages. The shape of the | AST needs to be the same as the shape of the source. Once | you've done that though the program and visualization are | isomorphic. Check out the toHTMLcube demo in the Sandbox, | the language designer demo, or: | | https://v20.ohayo.computer/?filename=ohayo.ohayo-source- | code... | | That's all older stuff, but hints at some of the ideas. | reidjs wrote: | As a programmer currently learning sales by necessity, I agree | wholeheartedly. | dstick wrote: | Is this missing half the article? The page just stops on mobile | and there's no footer, nothing. | | I count 4 paragraphs in total. | nchelluri wrote: | I believe that you're seeing the whole article; it is short. | [deleted] | k__ wrote: | I had the impression the main problem was missing cognitive | flexibility of developers. | | Some people just do things like they always have. They don't | bother to understand what people before did and then bend | everything to their will. | throwaway201103 wrote: | Another twist on this is that some people are passive- | aggresively lazy. They may understand that what they are being | asked to do has longer term negative consequences, but they | don't want to invest the effort in trying to change any minds. | They just do what they are asked, and when things get | complicated they can say "I did what you asked for; you didn't | ask me to simplify anything" | | This is seen more in rigidly hierarchical organizations where | debate and ideas from the lower ranks tend to be quashed. Think | of the boss who says some variant of "you're paid to do what I | ask, not to talk about it" | SulfurHexaFluri wrote: | This downplays the difficulty of "bothering" to understand | things. Most developers work on applications of insane | complexity that they couldn't ever hope to understand. Almost | every change is done in some amount of ignorance of the | surroundings. | underdeserver wrote: | This is what project maintainers or code owners are for. It's the | maintainer's responsibility to reduce complexity, and because | it's hard to quantify the maintenance cost of additional | features, the maintainer should have the final say. | TobiasA wrote: | I have never heard of code ownership being a positive thing. | Isn't this more an issue of encouraging internal knowledge | sharing and confidence in refactoring? | [deleted] | beaconstudios wrote: | Here's my approach. I think many feature requests fall under the | X/Y problem. | | - view a new feature request as a new user capability | | - extend the model that the software implements, to encompass | that capability - regardless of how the feature was implemented | in the requester's head. | | - extend the software to match the new model. This may require | refactoring as the model may have had to undergo shifts to | encompass the new capability | | For example: | | I have a car. I model the car as four wheels, an engine, a | chassis, and a lever. The engine drives the wheels, the wheels | support the chassis, the chassis contains the engine. A lever in | the chassis sets the engine in motion. It's a simple model and is | capable of 1. sitting still and 2. moving forwards and backwards. | This is all the capabilities we've needed so far. | | A user requests a new feature where the wheels are instead | mecanum wheels (https://en.wikipedia.org/wiki/Mecanum_wheel). | | The default industry response is to either implement the change | as-requested, or reject it. I propose that the correct move | instead is to ask the user WHY they want mecanum wheels. They | reveal that they want the car to move in 2 dimensions, rather | than one. From that understanding you can extend the model of the | car to encompass the feature - you may add the mecanum wheels and | a mechanism to control them, you may add a steering wheel and | rack-and-pinion, you may do something completely different - | totally depending on how and why the user wants 2d movement | (depending on further questioning, ie "5 whys"). But you are | working to the capability, not the feature. By extending the | model, you can then change the software to match this new model. | | I think as software engineers we have a tendency to forget the | model and focus only on the code. A request for mecanum wheels | becomes a question of how to change the software to encompass | that feature. But we must always remember the existence of the | model, and the user's relationship to it. | karmakaze wrote: | The far greater source of complexity is the extra that has | resulted from features being analyzed and implemented | sequentially. Each one may have been economical in isolation, but | as a whole is not. This is why a rewrite often seems so | attractive, to analyze the entire known scope and use a smaller | set of mechanisms to do the same often allowing for additional | changes to fit more easily. Painting yourself in a corner is | either short-sightedness (technical/vertical, | voluntary/oblivious), a tech-debt choice, something that didn't | turn out as well as expected despite efforts for technical | reasons or unexpected changes. It also comes from trying too | hard, generalizing too early. So there are many reasons, and | pragmatically balancing them is something that is given far too | little weight. | carapace wrote: | One the one hand it's really hard to program well. I think anyone | who can solve a sudoku puzzle can learn to program, but only | people with freakish skill or determination can learn to program | really well. | | On the other hand, those folks who can program really well are | often what I call "complexity junkies": programming is their | sudoku, it's fun and exciting. It helps that you can get paid | well to do it. | | So you get things like Haskell and Rust. | KingOfCoders wrote: | Because there is no physical, 3dimensional limit. | hotcrossbunny wrote: | In my humble opinion, a lot of projects go quietly bad when they | experience some sort of new requirement that gets underestimated | in terms of its architectural impact by project management. At | such time senior Devs have either already moved on, or have their | eye off the ball such that new features get incorporated without | the necessary architectural support. These inflection points can | themselves introduce complexity but often become the gateway for | all sorts of subsequent small things that explode in size. In | short, don't miss architecture moments | thefourthchime wrote: | The other thing that happens, and really dooms a project is | when the senior people leave. Eventually you end up with a team | that doesn't really understand the code. | | This leads them to just tack on features while changing as | little as possible. This will grow into something truly | unmaintainable, virtually guaranteeing no competent work will | be done on the project again. | taeric wrote: | My favorite point on how software end up complex, is our industry | somehow thinks we are unique on this. | | Everything winds up complex. Just look at how many ingredients go | into simple store bought cookies. Look into the entire supply | chain around your flour, that lets your homemade cookies be | simple. | TheOtherHobbes wrote: | Unless you count obesity and diabetes, the public is not | noticeably inconvenienced by the cookie supply chain, because | it pretty much just works. So it's hard to find examples of | Cookie Technology Failure Modes with serious consequences. | | There's a huge difference between "hard to use" and "prone to | failure." | | Software is often hard to use, but it's _also_ extremely prone | to not working. | | This is not true of airliners [1], cars, large buildings, large | ships, and commodity rocket launch platforms - as well as | global supply chains of all kinds. | | [1] Unless there's software involved. | taeric wrote: | This is beside my point. I only picked cookies because I'm | making some. :) | | Look into the supply chain for lumber to your house. | Concrete. Electricity. Literally anything. Life is | complicated. | | Software just fools us by letting us more readily have a | blank slate sometimes. Though, even then, the complexity of | the tool chain to support such easy "hello world" programs is | insane. | | If you want to know frustration, try using any of the simple | tools you can buy at a dollar store for a time. Simple can | openers that will fall apart in no time. Tableware that will | bend and likely go unusable in months. | nradov wrote: | When the pandemic lockdown started some of the supply chain | abstractions leaked, and the US public was noticeably | inconvenienced by lack of toilet paper, cleaning supplies, | and (in some cases) meat. It turns out the supply chains are | actually rather fragile with many failure points. | Blikkentrekker wrote: | I'd say that the only reason that software seems too complex, | rather than as complex as it needs to be, is because every | programmer thinks he can rewrite it in a simpler way, but when | he's done, it's as complex that which he has rewritten. | | I've seen it happen so many times, and I've done it. It's the | very same principle that leads to almost every construction | project running behind schedule -- a man simply underestimates | the complexity of nigh every task he endeavors to complete. | nchelluri wrote: | I see your points, and I see the merit of "rewrite syndrome", | and lean strongly towards automated-test backed refactoring, | and all in all I disagree with your thesis. | | Sometimes, software patches and new features get tacked on and | tacked on and the system loses all semblance of cohesion or | integrity. Thinking of the system as a whole, iterating with | the confidence brought by tests of some sort, one can begin to | detangle all the unncessary intermixing and duplicate work and | begin to make the system sensible. | RivieraKid wrote: | Depends. In my experience, I've never regretted a rewrite and | always ended up with a better and simpler code. | | It can be very frustrating to modify low quality and ugly code | so I feel much better after a rewrite. | Aeolun wrote: | Depends. If you can re-evaluate the requirements when you start | your rewrite, you can likely consolidate a lot of the features. | | If the requirements stay exactly the same, then yeah, there's | no point. | nradov wrote: | That doesn't work in reality because for any large, complex | piece of software it's impossible to rediscover _all_ of the | requirements. There are always hidden requirements which were | never properly documented but somehow ended up as code in the | legacy system. | Blikkentrekker wrote: | Well, when _Wayland_ started they went in on the assumption | that they could cast away quite a bit that "no one was using" | or that "wasn 't necessary". | | And then they had to include more of what they cast away | because they underestimated the number of consumers of things | they personally weren't using. | | _libinput_ originally actually did not have a way to disable | and configure pointer acceleration I believe because the | developed thought there was no reason to ever turn it off. He | was not a gamer and was largely ignorant of how essential | being able to disable it is for the level of accuracy | required for video games. | SulfurHexaFluri wrote: | Now days Wayland compatible DEs seems to support everything | the average user needs while supporting quite a few modern | features that were apparently too difficult to support in | X11 like monitors with different DPI scales. | convolvatron wrote: | I completely disagree. certainly the chances that a rewrite in | a standard software organization are pretty good that the new | version will be just as broken, but in a new and different way | than the last. | | but I've taken several projects in the 100s of k-lines and | translated them into projects with equivalent functionality and | spanning between 1-2 decimal orders of magnitude less source | code. | | that's not an argument for rewriting in all circumstances - I | just think at least half of most mature software is just 'junk | dna' - useless boilerplate, unused paths, poor abstractions, | etc | AnonC wrote: | > This means that supporters can always point to concrete | benefits to their specific use cases, while detractors claim far | more abstract drawbacks. | | I don't agree with this. If you have a salesperson/accountant who | grew into being CEO and is just a bean counter who doesn't (want | to) understand the real costs of training new developers on | increased complexity in a system or being able to maintain a | complex system over decades, this could possibly be true. But in | that case, your engineering manager isn't a real engineering | manager either, and is just another bean counter in disguise. | | I get that senior management may not always (want to) understand | the nuances in building, maintaining and supporting a complex | system, but it's not abstract. It costs real money that all these | types can feel the pinch of. | | The real reason why software is complex or can get complex is | because the underlying domain and its requirements and | constraints are complex, combined with layers of complexity added | on the technical side to enable certain things (perhaps easy | configurability, scalability, reliability, etc.). There are self- | inflicted wounds too, where complexity is prematurely added. But | that's not the full story in all cases. | tonetheman wrote: | Software (like the universe) is entropy over time. | lhpz wrote: | Let me expand on this thermodynamical metaphor. Open systems | that may exchange matter and/or energy with their environment, | can maintain or even decrease their own entropy. This is how | living organisms exist, without violating the second law of | thermodynamics. | | If you see a software system as a living organism, then the | natural entropy increase (new feature requests?) may be | contained if enough energy (developer time?) is invested. Of | course this kind of investment may contravene some other | commercial software law, like being profitable. | khaledh wrote: | Complexity is a fact of the world we live in. The human body is | extremely complex, but all organs and systems in the human body | work in harmony and with astounding efficiency. | | We need to stop fighting complexity in software and embrace it | instead. The goal shouldn't be to pursue simplicity at the cost | of ignoring complexity of the domain being modelled, but to | acknowledge that we're modelling a complex system, and design it | such that subsystems work together in harmony. | | This article in particular touches on an important point which | resonates with me. It's what I call "displacement oriented | design". I view a code base as a puzzle that grows in size over | time; you can't grow a puzzle just by adding to it and leaving | all other pieces intact. As you expand the size of the puzzle you | need "displace" some of the existing pieces, reshape them, and | fit them with the new pieces. | nchelluri wrote: | I agree with some of your points and disagree with others. | | The human body is exceptionally awesome, but also so very | fragile. I think we can do better with well-thought-out design, | and a part of that that I embrace is the value of simplicity. | | My thoughts can be best said with a quote (especially the part | about a complexity budget): | | > I used to tolerate and expect complexity. Working on Go the | past 10 years has changed my perspective, though. I now value | simplicity above almost all else and tolerate complexity only | when it's well isolated, well documented, well tested, and | necessary to make things simpler overall at other layers for | most people. For example, the Go runtime is relatively complex | internally but it permits simple APIs and programming models | for users who then don't need to worry about memory management, | thread management, blocking, the color of their functions, etc. | A small number of people need to understand the runtime's | complexity, but millions of people can read & write simple Go | code as a result. More importantly, Go users then have that | much more complexity budget to work with to build their actual | application. I would've never built Perkeep had I needed to | fight both its internal complexity and the complexity imposed | on me by other contender languages/environments at the time. | | > | | > All that is to say, simplicity is not only refreshing, but it | also enables. Go made me feel productive in a way I hadn't felt | in many years where everything just felt like it was getting | more complex. Ever since finding Go, I've been regularly | hunting for other technologies that provide simplicity as a | feature. | | https://bradfitz.com/2020/01/30/joining-tailscale | khaledh wrote: | I agree on the distinction between internal vs. external | complexity; this particularly resonates with me: "the Go | runtime is relatively complex internally but it permits | simple APIs and programming models for users...". I probably | should've made it clear that the external interface of a | system (whether it's a UI or an API) should be simple and | abstracts away the internal complexity. | | To go back to the human body analogy, while the internals of | the human body are complex, the external interface is | extremely simple: you eat to get energy; you sleep to get | rest; you use your reproductive organs to reproduce; etc. You | don't have to learn how the internal systems work to operate | your body. | jameskilton wrote: | I feel that a lot of people misunderstand "complexity" vs | "complicated". There's nothing wrong with complex. It's the | nature of life that things are complex. Complicated though is | almost always a negative. Complex code is fine, it's probably | solving a real problem. Complicated code is not, it's just hard | to work with. | recursivedoubts wrote: | I agree with your comment. However, a good tool for controlling | complexity is deciding what your system is going to do. As I | said in a sibling comment, consider method overloading in java: | this is a real world feature, not uncommon in other languages. | There are arguments for and against it (I am against it.) | | The implementation of it may be amazing code, but none the less | it makes the java compiler and runtime far more complicated | that it would be if the feature were omitted. | | So, again, I agree with you, but I also agree with the articles | point that choosing features carefully is an important tool in | controlling complexity. | [deleted] | breck wrote: | Rich Hickey had a great talk on this that introduced me to the | word "complecting" (https://github.com/matthiasn/talk- | transcripts/blob/master/Hi...) | | Relatedly, I have a simple new maths for counting complexity, | so that you can compare two "complex" solutions and pick the | less "complicated" one: | https://github.com/treenotation/research/blob/master/papers/... | | SVG form: | https://treenotation.org/demos/complexityCanBeCounted.svg | bryik wrote: | This is a good point. I once worked on a system that checked | projects met various legal standards and rules before allowing | changes to be saved. This system was complex because the rules | were complex, the only way to make it simpler would have been | to convince the government to make the rules simpler. | bluetwo wrote: | "Complexity is a crutch" - I'm told Neil Gayman said this. Not | 100% sure but I totally agree. | ChrisMarshallNY wrote: | My experience, is that "complicated" vs. "complex," as you | define, changes, depending on who is looking at the code. | | If someone has a philosophical aversion to something like | abstraction, then they will label it "complicated," but I use | abstraction, all the time, to insert low-cost pivot points in a | design. I just did it this morning, as I was developing a | design to aggregate search results from multiple servers. My | abstraction will afford myself, or other future developers, the | ability to add more data sources in the future, in a low-risk | fashion. | | I also design frameworks in "layers," that often implement | "philosophical realms," as opposed to practical ones. Think OSI | layers. | | That can mean that adding a new command to the REST API, for | example, may require that I implement the actual leverage in a | very low-level layer, then add support in subsequent layers to | pass the command through. | | That can add complexity, and quality problems. When I do | something like that, I need to test carefully. The reason that | I do that, is so, at an indeterminate point in the future, I | can "swap out" entire layers, and replace them with newer tech. | If I don't think that will be important, then I may want to | rethink my design. | | That is the philosophy behind the OSI layers. They allow | drastically different (and interchangeable) implementation at | each layer, with clear interface points, above and below. | SulfurHexaFluri wrote: | And quite often the philosophy doesn't line up with reality. | The OSI layers have little relation to how networking | actually works and it would be next to impossible to replace | some of those layers. | PsylentKnight wrote: | I agree with your premise but I don't think you're using the | correct terms here. "Complex" and "complicated" are synonyms as | far as I can tell. | | What you're describing sounds like "essential complexity" vs. | "accidental complexity." See "No Silver Bullet." | | Sorry, this is pedantic, but using the incorrect terms adds | accidental complexity to a topic that is already essentially | complex. ;) | pansa2 wrote: | > _" Complex" and "complicated" are synonyms as far as I can | tell._ | | No. Complex is the opposite of simple. Complicated is the | opposite of easy. | | Simple and easy aren't synonyms - see the talk by Rich Hickey | linked in a sibling comment. | hotcrossbunny wrote: | Out of the tarpit anyone? | justapassenger wrote: | Biggest underlying reason for why software ends up complex, is | because real world domain in which software operates, is also | complex. | ratww wrote: | I don't know if I agree with that. | | Most of the complexity and bugs I see in software are not | because of the problem domain, but rather because of over- | abstraction, under-abstraction and abstraction leaks, and also | because of limitations and complexities introduced by the | programming model or environment. | | (unless you consider that "supporting five operating systems | and the language must be X" is part of "essential complexity") | | Of course, the more complex your domain is, the bigger is the | program. But the non-essential complexity that exists due to | the bureaucracy of languages/libraries/frameworks is a much | bigger factor in adding complexity, bug and lines of code. Some | examples: | | - Manual allocation and deallocation of memory is a good | example of something that we might think as essential, since | it's intertwined with our domain code, but turns out to be | unnecessary (even though the replacement has downsides). The | billion-dollar problem (nulls) is another one. | | - Supporting multiple environments/browsers/platforms. | Competition is good, but the cost is steep for third-party | developers. Using multiplatform frameworks partially solves but | also has drawbacks: performance, limitations, bugs, leaky | abstractions. If you need to be closer to metal, then different | OSs have different threading models or API styles. Sometimes | they don't even expose the main loop to you. You need to work | around those limitations. | | - In most environments we still don't have a nice way of | handling async operations without leaking the abstraction. The | current solution is adding "isLoading" data everywhere (or | creating some abstraction around both buttons and the fetching | mechanism). Concurrent Mode in React is probably the best thing | we have so far. | | - Most modern Javascript toolchains need multiple parsing | steps: in the transpiler (to convert to ES5), in the bundler | (to detect dependencies), in the linter, in the prettifier, and | in tests. Compatibility between them is not guaranteed, and you | might get conflicts which have to be solved by finding a middle | ground, and that sometimes take more time than writing | features. | | - Dogmatism is another issue. I remember in one workplace years | ago there was a "ORM only" rule and most of us would work the | SQL and then convert to Rails ActiveRecord (or worse: Arel). In | the end it was a complete waste of time and the results were | impossible to maintain. | | - I also think that the old Peter Norvig quote that "design | patterns are missing language features" still stands. Go has | proven that it's possible to have "dumb, simple code", but in | other languages our best practices involve adding non-essential | complexity to products. | | The only exception to that in my experience is SQL: if a query | is too big is not due to some bureaucracy of the language, but | rather due to the complexity of the domain itself. | justapassenger wrote: | For me what you're describing is more mess than complexity. I | see your point tho, but I'd also say that some of things you | describe here are direct result of real world problem domain | being complex, and changing over time. | | For example, wrong abstractions. As long as engineers writing | the software are competent (if they aren't - that's a totally | different story) they'll try to chose right level of | abstractions for current understanding of the problem. Years | down the road, a lot of their choices maybe end up to be a | mistake, very often because understanding of the problem | changed, or problems itself changed, and something that was | nice and clean solution isn't one anymore. If problem isn't | fully fixed and 100% understood, you'll never be able to make | all the right, future-proof decisions about right | abstractions. | ratww wrote: | Yes, it is "mess". But it's also additional complexity that | is non-essential to the problem. It's what Fred Brooks | called accidental complexity in his No Silver Bullet essay. | It is important for us to acknowledge and differentiate | between the two kinds of complexity, because we in the | software industry can and should aim at reducing the | accidental/non-essential type. | | About abstraction, even when it's good, it still depends. | For example: you can use a Visitor Pattern to solve a | collision detection problem, but if the language had | multiple dispatch you would have less complexity. So it is | definitely non-essential complexity. | pontus wrote: | That together with the fact that nobody gets promoted and | nobody collects revenue by simplifying or refactoring code. | ratww wrote: | Not directly. But simplifying and refactoring can help you | understand the code better than doing routine maintenance. | This helps you solve bugs and write new features faster and | with more stability, and also give better input during | meetings. | | So, indirectly, yes: you can get promoted and collect revenue | by simplifying and refactoring. | arbaal wrote: | Funny that I'm nobody again. I work for a big german | retailer. A big part of my work includes simplifying and | refactoring code, so that we have lower maintenance cost and | can move faster as a organisation unit. This was also a big | factor in my last promotion. | | We also increased profit by lowering our runtime cost by some | of those optimizations. | senko wrote: | If I were a betting man, I would bet that the main reason | why you're allowed to do that is because you lowered | runtime costs by some of those optimizations, not because | of the cleaner code you can move faster. | Aeolun wrote: | You can trade on previous achievements to spend time | improving your codebase (unless you have a micromanager | that tracks every hour). | ThePadawan wrote: | ? not parent, but I am really surprised by that mindset. | | A large part of my career has been telling my boss "we | should invest 5h of cleanup time here to make sure that | in the future we won't have those 5h worth of obstacles | here before building a new feature on top of it". | | In industries where requirements change on a weekly or | monthly basis, agility is worth a _lot_. | hugey010 wrote: | People do get promoted and paid for simplifying and | refactoring code. Heck, I got promoted and am currently being | paid to do just that. It often involves adding value in other | ways simultaneously, but what you said is just false. | pontus wrote: | I guess it's dangerous to use words like "nobody" or | "everyone" or "always" or "never" on the overly pedantic | Hacker News. I would hope that the sentiment of my point | came across either way. If not, let me clarify: it's | generally preferred to spend engineering effort on building | new features and optimizing code to improve performance | etc. over e.g. rewriting code from scratch to simplify it's | maintenance. I would venture to guess that an overwhelming | amount of engineering effort is spent on effectively adding | complexity to code. | nthj wrote: | Whoever is able to demonstrate the value of a sequence of | work to the business will get their work prioritized. | Product's job is to shape and predict the value of adding | new features. Customer complaints about slow reports help | Customer Support justify performance optimizations. | | If engineers want the business to pay them for | simplifying maintenance, they can identify a collection | of support tickets that have a compelling root cause, and | propose refactoring that system to eliminate the bugs and | maintenance issues. | | What I usually see is us engineers calling it "tech debt" | and complaining we never take time for it, which is about | as effective as the bookkeeper complaining about the | paperwork that debt payments create for him. Businesses | love debt. It's a useful financial tool. The business | doesn't get the issue. | airstrike wrote: | Alex Gaynor! I've missed this guy. I remember him back from his | work on Django. | | http://pyfound.blogspot.com/2019/02/the-steady-leader-of-pyt... | goblin89 wrote: | Software can be viewed as a function. | | A function is designed to be called in a certain way for certain | output. If the required use of the function changes, function's | signature would need to support more arguments, and its behavior | will increase in complexity accordingly. | | To avoid that, it is important to understand that a changed use | case calls for what is effectively _another function_ (possibly | more than one). It may not be immediately feasible to rewrite and | switch all callers over due to limited resources and lack of | control over aforementioned callers, but conceptually it is | another beast now--and existing implementation should move in | that direction or be put on a deprecation schedule, rather than | keep widening input and output spaces in a futile attempt to try | to be everything at once. | | Software is very similar in this regard, except it is much more | tempting (and often significantly easier) to "append" features | rather than rethink the fundamentals as time goes by and users | with different needs get on board. | | This is why, I believe, rigorously assessing[0] the scope and the | intended audience of a piece of software at early stages (and | constantly reassessing them afterwards) can go a long way against | unchecked rise in complexity over time. As can prudently | abstracting architecture pieces away into separate self- | containing focused pieces, which can be recombined in new ways | when context inevitably calls for what essentially is a different | piece of software--instead of having to rewrite everything or | horseshoe new features to support use cases that were not | originally envisioned (but have to be supported for business | reasons). | | [0] By having frank discussions and asking "why" many, many | times. | tedunangst wrote: | I like this essay, but I've also had success with "reject all | feature requests" and ended up with happy users, not no users. | tacitusarc wrote: | Within a problem space, there are two kinds of complexity: | inherent complexity, and accidental complexity. This article is | about accidental complexity. | | There is, as far as I can tell, and enormous amount of accidental | complexity in software. Far more than there is inherent | complexity. From my personal experience, this largely arises when | no time has been taken to fully understand the problem space, and | the first potential solution is the one used. | | In that case, the solution will be discovered as partially | deficient in some manner, and more code will simply be tacked on | to address the newfound shortcomings. I'm not referring here to | later expansion of the feature set or addressing corner cases, | either. I'm referring to code that was not constructed to | appropriately model the desired behavior and thus instances of | branching logic must be embedded within the system all over the | place, or perhaps some class hierarchy is injected, and | reflection is used in a attempt to make make the poor design | decisions function. | | I don't think adding features makes software more complex, unless | those features are somehow non-systemic; that is, there is no way | to add them into the existing representation of available | behaviors. Perhaps an example would be a set of workflows a user | can navigate, and adding a new workflow simply entails the | construction of that workflow and making it available via the | addition to some list. That would be a systemic feature. On the | other hand if the entirety of the behaviors embedded within the | workflow were instead presented as commands or buttons or various | options that needed to be scattered throughout the application, | that would be a non-systemic addition, and introduce accidental | complexity. | pydry wrote: | One things I've noticed about building software is that the | most appropriate contours of the problem space often only | become clear with hindsight. | | Even if you start off with the best intentions about not | putting in too many features it won't always help. | | This is why the second mover can also have an advantage in some | areas. If they recognize the appropriate contours they can | avoid the crufty features and more directly and effectively | tackle the main problem. | senko wrote: | In this regard, software is quite similar to law - more so than | to other STEM disciplines like mechanical or civil engineering or | math. | | A body of law is adapted and extended through many years, by | various groups having different priorities, and rarely does | someone dive in and refactor it to be simpler (while achieving | the same "business goals"). | | In software, at least we have an option of creating automated | tests to verify correctness or internal consistency, and using | stricter languages to avoid ambiguities. | | In light of this, we (as an industry) are actually doing quite | good! | LC_ALL wrote: | Building features on top of other features is often zero cost. | Code becomes a many layered cake consumed by the end user. In the | web development stack the simplest of features like text on a | screen is the achievement of decades of technological progress. | Text may be localized, shaped with HarfBuzz, run through libicu's | BIDI algorithm, encoded with a nontrivial encoding, wrapped in | markup language, nested inside of multiple layers of network | headers and corresponding metadata, sent over the wire as a | series of 0s and 1s, and then painstakingly unpacked in reverse | order. | | This is clearly complicated and clearly works. Many different | actors operating quasi-independently. You can imagine the | difficulty when one actor in a time crunch tries to design a | similarly complicated cake stitched together with parts homemade, | parts open sourced and parts paid for. | dimmke wrote: | Taking on the responsibility of pushing back hard on poorly | conceived new features is one of the important hidden skills of | being an effective software developer. Programmers who just do | whatever they get told like marching ants end up shooting an | organization in the foot long term. | | You have to develop an opinion of the thing you're | building/maintaining and what it should be able to do and not do. | You can't count on project managers to do that. | | The trick to doing this effectively is to find out the problem | the feature is _actually_ trying to solve and providing a better | solution. | | Usually the request is from end users of software and they have | identified the problem (we need to do xyz) and prescribed a | solution (put the ability to do xyz in a modal on this page.) But | if you can look to what other software has done, do a UX review | and find a way to add a feature in that solves their problem in a | way that makes sense in the larger context of the software, they | won't have a problem with it since it solves their problem and | the codebase will take less of a hit. | | Unfortunately, it's a lot easier to just add the modal without | complaint. | elb2020 wrote: | > You can't count on project managers to do that. | | This is one of my pet peeves when it comes to software | development. I _really_ think that software development project | managers ought to be able to spot the difference between a good | architectural decision and a bad architectural decision, a good | design decision and a bad design decision, a well implemented | function and a badly implemented function. It sinks my heart, | as a software development professional, having to work for | project managers who, in many cases, would be hard pressed to | explain what a byte is. It's just so wrong. | | It's like working for a newspaper editor who does not know how | to read or write. It does not mean that you cannot produce a | newspaper, but it depends upon the workers stepping in and | doing all the strategical technical decision behind the project | managers back. As an engineer you can live with it for some | time, but eventually it ends up feeling fake, and like a | masquerade. | | I'm much more in favor of hands on leadership types like | Microsoft's Dave Cutler, with the technical skills to actually | lead, and not just superficially 'manage'. | Rapzid wrote: | I'm not sure I would ever "work for" a project manager. Work | with, sure, but not for. | speedgoose wrote: | It's also faster to just add the modal. When you are asked to | do xyz ASAP because it has been sold to a customer and should | have been deployed a week ago, you don't feel the need to do a | UX review. | pc86 wrote: | Missing a deployment by a week (or more accurately, deploying | on that long of a timeframe), and only doing UX reviews when | you "feel the need" both speak to larger organizational | problems that probably aren't going to get solved by having a | senior dev push back on a poorly thought-out feature. | speedgoose wrote: | That's true. | officehero wrote: | > Taking on the responsibility of pushing back hard on poorly | conceived new features is one of the important hidden skills of | being an effective software developer. | | example from my current project: 1. Inherit half finished large | software with lots of features. 2. It contains bugs and is | impossible to effectively maintain/develop for the allocated | manpower. 3. Management still wants all features. 4. Be brave | and don't work on anything except essentials until they're | sorted out. Lie to management if you have to e.g. that you | found serious bugs that must be fixed (which is kind of true | but they wouldn't understand) | alkonaut wrote: | > Programmers who just do whatever they get told like marching | ants end up shooting an organization in the foot long term. | | This. This is why you want senior devs too. You want people who | can stand up to poorly conceived features. The most important | job is to say "no", or "how about X instead?". I get furious | when I see senior colleagues defend horrible design decisions | with "it was what was specified". Your job as a developer is to | take the spec from the product owner and suggest one that | actually fits the system in terms of maintainability, | performance etc. Blindly implementing a specification is a | horrible idea. | serial_dev wrote: | I'd like to add that it also depends on the culture of the | team. | | In team A, I challenged many ideas coming from the designer | and the product owner. I was also pushing back on technical | decisions coming from the CTO. They always listened: I would | change their minds a couple of times, sometimes I realized I | was wrong. Only a few times could we not resolve the issue, | but in the end I felt that they heard me and considered my | point of view. | | In team B, I started out with the same mindset and was trying | to challenge the validity of their product decisions. It was | superficially acknowledged, but 98% time my input was | basically ignored, and I felt like a party-pooper for | pointing out contradictions and mistakes in their thinking. | After months of trying to be listened to, I realized I'm here | to be a coding monkey, they don't want my input neither on | product, nor on technical problems. I learned to just nod and | smile, cheer them on on their bad decisions, they felt great | because they felt validated. It was also better for my | happiness short term, as it's not a great feeling to feel | that I'm bumming them out. | | Long term, I started looking for new positions, and since | then quit already. I still feel it's a shame as the "idea" | had great potential. | hypertele-Xii wrote: | What you describe is a lack of a designer/architect in the | loop. Devs are supposed to implement what is requested, as | requested. Designers and architects are supposed to figure out | _what_ to request to the devs, based on the customers ' needs. | And this indeed entails figuring out the customers' actual | problem, rather than parroting their solutions (which they are | almost always unqualified to design). | dbish wrote: | Your job description of devs is very limiting. Software | developers should be close to the customer problem, work to | understand it, and develop for it. When you silo them away | this way and expect them to be order takers you add bloat to | the team and inefficiencies. | pc86 wrote: | If you're working with a small team on some customer-facing | SaaS, _maybe_. But let 's be honest, there are a _lot_ of | software developers out there who either don 't have the | inclination or skill to push back against their boss's | boss's boss, or question why a particular abstraction is | request over another. They've got to work too, and for the | most part a senior or architect is just going to give them | tasks and wait for them to be completed. | | I've also noticed a distinct lack of "work[ing] to | understand [the customer problem]" from the "I'm going to | be a dev for 7.5 hours a day at work and not touch a | computer or think about programming for a single moment | more" subset of programmers. | dimmke wrote: | >there are a lot of software developers out there who | either don't have the inclination or skill to push back | against their boss's boss's boss, or question why a | particular abstraction is request over another. | | I definitely agree with you. There are a lot of people | who for various reasons (inherently lazy, not engaged | with their current role, etc...) would not do this. | | My problem is that I think this is a learned behavior in | a lot of environments. A lot of places try to treat | developers as code monkeys, where it's actually the | brilliant designers and business analysts making the | decisions and it's just the responsibility of these | programmers to execute their vision. | | This compartmentalization doesn't work well and it leads | to a lot of really bad software. Here's my theory as to | why. | | If you look at all the various jobs involved in creating | a piece of software: Design (UX/Screen Design), testing, | project management, business analyst/requirements | gathering, programmer. There's only one that's actually | _required_ to produce a functional piece of software: the | programmer. | | And as much as people want to pretend like this isn't the | case, the programmer often has to do a little bit of | every one of those other roles to effectively do their | job. In this comment thread, we've been discussing how | they have to put on their project management hat and help | shape requirements, but it's deeper than that. | | After all, what programmer hands off a feature for | testing without themselves testing that it worked? UX | might give you a wireframe, but there's almost always | edge cases not accounted for. As for project management, | if you're lucky you have a good one who actually makes | the project easier but more often you will have a bad one | who adds no value and only forwards emails to you and | bugs you about timelines, which effectively means you're | managing the work yourself. | | And this isn't to say that the people in those other | roles are necessarily bad at their job - but there's a | context you get as the person actually building the thing | that you can't get in these silo'd, standalone roles. You | see issues that other people can't and you have an | ownership over the end result that they don't. The | programmer is the most important piece of it all. They | tie it all together and the buck really stops with them. | Programmers who don't take ownership ultimately lead to | worse software. | | If companies understood this on a more fundamental level | and tried to select more for it in hiring than | whiteboarding questions, they'd probably be more | successful. | | The parent comment tried to make a distinction between | "architect" and "dev" - but this should start at the Jr. | levels. Teach them at the beginning of their career. | | To be clear, I do think those other roles are important | and valid. I just think it's very very rare for a team to | be staffed with people who are good in all of those areas | and when someone is deficient, it basically falls on the | programmer to make it up. | dbish wrote: | I mostly run large teams who have handled both internal | and consumer facing services. The software devs who don't | have an inclination to understand the customer problem | generally have not been highly effective and are not | people I would use as role models or for setting | expectations to others. I'll add to this that I do not | believe there is a place for an "architect" role that is | separate from a developer role. Good developers learn to | think about architecture and do so with larger impact as | they become more senior. "architects" who don't code tend | to be "architecture astronauts" who do not help you ship | value fast, at quality, to your customers. | dbish wrote: | Architecture astronaut: | https://www.joelonsoftware.com/2001/04/21/dont-let- | architect... | mirekrusin wrote: | There are complex things and there are complicated things. You | want to make complex things easier to understand by refining | abstractions and eliminate complicated things alltogether. | Constant refactoring and tabu word - intelligence - is the key | here. It doesn't mean you have to reshuffle everything every | week. It means spending maybe 5% - 20% of time on small things to | improve code based on current knowledge. This has to be pushed by | developers because semi-agile methodologies most companies use | have built-in forces to move those tasks out of sprints. | recursivedoubts wrote: | This is one vector for complexity, to be sure. Saying "no" to a | feature that is unnecessary, foists a lot of complexity on a | system, or has a low power to weight ratio is one of the best | skills a senior developer can develop. | | One of my favorite real world examples is method overloading in | Java. It's not a particularly useful feature (especially given | alternative, less complicated features like default parameter | values), interacts poorly with other language features (e.g. | varargs) and ends up making all sorts of things far more complex | than necessary: bytecode method invocation now needs to encode | the entire type signature of a method, method resolution during | compilation requires complex scoring, etc. The JVM language I | worked on probably had about 10% of its total complexity | dedicated to dealing with this "feature" of dubious value to end | users. | | Another vector, more dangerous for senior developers, is thinking | that abstraction will necessarily work when dealing with | complexity. I have seen OSGi projects achieve negative complexity | savings, while chewing up decades of senior man-years, for | example. | taeric wrote: | Your example is funny. Default parameters are far more | complicated than method overloading. | | That said, I fully agree on the OSGi point. Makes me worried | about a lot of the new features I hear are on the way. :( | recursivedoubts wrote: | > Default parameters are far more complicated than method | overloading. | | I've implemented both, I disagree. Unless you are talking | about default arguments in the presence of method | overloading, which is insane, and which I have also | implemented. | layer8 wrote: | The important thing with default parameters, if you have | separate compilation, is that they are fixed in the callee | and not in the caller. Otherwise, when the default value is | changed, previously compiled callers will use a different | value than declared by the callee. Compiling the default | value into the caller is a problem in C++, and (IIRC) in | Swift. In other words, (positional) default parameters | should act like syntactic sugar for equivalent method | overloads. And then actual overloads provide more | flexibility for the callee implementation because it | doesn't _have_ to represent the default case as a special | in-band value. | taeric wrote: | I'm taking about in use, mainly. Python, as an example, | really screws it up by not giving you a way to know if you | have the default or a value that equals it. Then, the more | default values you pile in, the more this fun facet piles | up. | | So, if you skip on that part of it, and you can while still | getting far, it is easier. With it, though, is stupid | complicated for little benefit. | dxdm wrote: | > Python, as an example, really screws it up by not | giving you a way to know if you have the default or a | value that equals it. | | It does: set a unique `object()` instance as the default. | It will only compare equal to itself, but it's even | clearer to read if you check for identity with `is`. | You'll need to define it outside of the function | definition. | | It's not the most concise, but it works, and it's an | established pattern used in many high quality open source | projects. | taeric wrote: | Thanks. This is kind of reinforcing my "not actually | simple to use" point, though. :) | pietrovismara wrote: | Why would you need to know if a value is the default or a | value the equals the default? I can't think of a reason | for that as I've never faced this problem in my career. | uryga wrote: | one example is `namedtuple`'s `._replace`: | >>> x = Foo(a=1, b=2, c=3) >>> x._replace(a=1000, | c=3000) Foo(a=1000, b=2, c=3000) | | in this case, you really need to know if the user passed | a replacement or not. | taeric wrote: | It is niche, but it does come up. Usually with boolean | parameters, but also on migrations. Makes it possible to | change defaults in a controlled way rather easily, since | you can very easily log all of the places you used the | default. | tener wrote: | Nah, you should be using another value if you care about | that. For example in C# a nullable bool would be a fine | choice (three values: true, false, null), or | alternatively Option types. | taeric wrote: | This feels like classic blaming the user. :) | | You can also have better behavior between multiple | defaults if you know whether someone gave a value or not. | | Basically, it is another tool compared to overloading. It | should be no surprise that there can be preference on how | to use between them. | [deleted] | tsimionescu wrote: | Common Lisp has a pretty simple fix for this: when you | declare a parameter with a default value, you can also | declare a variable that will be true iff the parameter | was actually passed. | | The function definition looks like this: | (defun foo (&optional (a 1 a-passed)) (if | a-passed (print a) (print "a not passed")) | > (foo 10) 10 > (foo 1) 1 > (foo) | a not passed | | This is still relatively easy to implement, and very easy | to use in my opinion. Of course, combining this with | named arguments is even better, and that is supported as | well (just replace &optional with &key, and then specify | the name when calling the function - (foo :a 1)). | OskarS wrote: | I always found it surprising that Java implemented method | overloading, but not operator overloading for | arithmetic/logical operators. It's such a useful feature for a | lot of types and really cleans up code, and the only real | reason it's hard to do is because it relies on method | overloading. But once you have that, why not just sugar "a + b" | into "a.__add(b)" (or whatever). | | You don't have to go all C++ insane with it and allow | overloading of everything, but just being able to do arithmetic | with non-primitive types would be very nice. | recursivedoubts wrote: | Yeah. | | One in-between option I have kicked around w/ people is | offering interfaces that allow you to implement operator | overloading (e.g. Numeric). Then you wouldn't have one-off or | cutesy operator overloading, but would rather need to meet | some minimum for the feature. (Or at least feel bad that you | have a lot of UnsupportedOperationExceptions) | | Java had/has a ton of potential, but they kept/keep adding | features that make no sense to me, making the language much | more complex without some obvious day-to-day stuff like list | literals, map literals, map access syntax, etc. | | Oh well. | J-Kuhn wrote: | C++ did set a bad example with operator overloading. | cout << "Hello World"; | | What the heck is that? Shift cout by "Hello World"? | | It looks like they did that just because they can. | monoideism wrote: | > Another vector, more dangerous for senior developers, is | thinking that abstraction will necessarily work when dealing | with complexity. | | I'm pretty good at fighting off features that add too much | complexity, but the abstraction trap has gotten me more than | once. Usually, a moderate amount of abstraction works great. | I've even done well with some really clever abstractions. | | Abstraction can be seductive, because it can have a big payoff | in _reducing_ complexity. So it 's often hard to draw the line, | particularly when working in a language with a type of | abstraction I've not worked much with before. | | Often the danger point comes when you understand how to use an | abstraction competently, but you don't yet have the experience | needed to be an expert at it. | bitwize wrote: | Yes, but remember Sanchez's Law of Abstraction[0]: | abstraction doesn't actually _remove_ complexity, it just | puts off having to deal with it. | | This may be a price worth paying: transformations that | _actually_ reduce complexity are much easier to perform on an | abstraction of a program than on the whole mess with all its | gory details. It 's just something to keep in mind. | | [0] https://news.ycombinator.com/item?id=22601623 | [deleted] | monoideism wrote: | You're echoing my comment: abstraction can be very useful, | but you have to take care. | | Also, it's pointless to claim that _transformations_ on an | abstraction can reduce complexity, but abstractions | themselves can not. Abstractions are required for that | reduction in complexity. | karmakaze wrote: | Optionals help, but what's really missing is union types for | arguments. | recursivedoubts wrote: | And named arguments. | | All three features, useful on their own, could be added to | java at maybe 10% of the complexity of method overloading. | With method overloading, they are all exponentially more | complicated. | | It's crazy how many places method overloading ends up rearing | its ugly head if you are dealing with the JVM. | karmakaze wrote: | I wish all the args always just came as one structured | object with some convenient syntax for accessing | destructured parts. That object can have named parts as | well as unions, optional, collections, and combinations of | them. | [deleted] | nradov wrote: | As a Java end user I'm really glad that method overloading | exists. The two largest libraries I ever built would have been | huge messes without overloading. But I take your point that | method overloading might be a net negative for the Java | platform as a whole. | recursivedoubts wrote: | Yes, java would be a mess without overloading (particularly | for telescoping args), but that's because it doesn't include | other, simpler features that address the same problems. | Namely: | | - default parameter values | | - union types | | - names arguments | | I would also throw in list and map literals, to do away with | something like varargs. | | All of these are much simpler, implementation-wise, than | method overloading. None would require anywhere near the | compiler or bytecode-level support that method overloading | does. It just has a very low power to weight ratio, when | other langauge features are considered. And, unfortunately, | it makes implementing all those other features (useful on | their own) extremely difficult. | layer8 wrote: | I don't really see a significant difference between | overloaded methods and a single method with a sum type | (where, in the general case, the sum is over the parameter- | list tuple types of the overloaded methods). One can | interpret the former as syntactic sugar for the latter. | recursivedoubts wrote: | Static vs dynamic dispatch, to start with. | | You need default values (and ideally named parameters) to | really make them equivalent, but yes, both address a | similar set of developer needs. | | The java implementation, where it bleeds into bytecode | and method scoring in insane ways, is particularly | unfortunate. | ozim wrote: | Well what I mostly experienced in my years in the field was | that developers, wether senior or not, feel obliged to create | abstract solutions. | | Somehow people feel that if they won't do a generic solution | for a problem at hand they failed. | | In reality the opposite is often true, when people try to make | generic solution they fail to make something simple, quick and | easy to understand for others. Let alone the idea that | abstraction will make system flexible and easier to change in | the future. Where they don't know the future and then always | comes a plot twist which does not fit into "perfect | architecture". So I agree with that idea that abstraction is | not always best response to complex system. Sometimes making | copy, paste and change is better approach. | BlargMcLarg wrote: | Kevlin Henney makes interesting points on this. We often | assume that abstractions make things harder to understand, at | the benefit of making the architecture more flexible. When | inserting an abstraction, it is supposed to do both, if at | all possible. Abstracting should not only help the | architecture, but also the understanding of the code itself. | If it doesn't do the latter, then you should immediately | question whether it is necessary, or if a better abstraction | exists. | | The take-away I took from it is that as developers, we love | to solve problems using technical solutions. Sometimes, the | real problem is one of narration. As we evolve our languages, | better technical abstractions become available. But that's | not going to prevent 'enterprisey' code from making things | look and sound more difficult. Just look at any other field | where abstractions aren't limited by technicalities: the same | overcomplicated mess forms. Bad narrators narrate poorly, | even when they are not limited. | diroussel wrote: | Sure over abstraction is a problem. And sometimes duplication | is better than depend y hell. But other times more as | traction is better. | | In true it's an optimisation problem where both under abs | over abstracting, or choosing a the wrong abstractions lead | to less optimal outcomes. | | To get more optimal out comes it helps to know what your | optimisation targets are: less code, faster compilation, | lower maintenance costs, performance, ease of code review, | adapting quirky to market demands, passing legally required | risk evaluations, or any number of others. | | So understand your target, and choose your abstractions with | your eyes open. | | I've dealt with copy paste hell and inheritance hell. Better | is the middle way. | wslh wrote: | I don't think most software development should end up complex | since most software development in the world is redundant. | | The problem is that we don't have the right high level | abstractions, low level robustness, and groups focusing on this | problem to build software faster. It is a matter of time, there | will be a silver bullet for most software needs. | | There is also some conflict of interest: imagine if Microsoft | gives you frameworks to build/integrate large software projects | with a few "parameters"? There will be fewer developers and | customers for them. | Aeolun wrote: | Making software that does everything by tweaking 'a few | parameters' is orders of magnitude more complex than the | software that would do it without. | | Since I'm now generally part of several year software projects, | we should be fine for the next few centuries or so. | codeulike wrote: | Its interesting that you use the term Silver Bullet. I assume | you are aware of the famous essay 'No Silver Bullet' | https://en.wikipedia.org/wiki/No_Silver_Bullet | wslh wrote: | Yes, it was intended. I think with enough redundancy there | are silver bullets in certain areas. | nchelluri wrote: | I agree wholeheartedly on the Brooks' "conceptual integrity" | thing. I really enjoyed his book The Design of Design. | | It takes effort to maintain cohesion and sound architecture, but | it pays off for future development. ___________________________________________________________________ (page generated 2020-12-13 23:01 UTC)