[HN Gopher] The collapse of complex software
___________________________________________________________________
The collapse of complex software
Author : feross
Score : 239 points
Date : 2022-06-09 15:18 UTC (7 hours ago)
(HTM) web link (nolanlawson.com)
(TXT) w3m dump (nolanlawson.com)
| loudmax wrote:
| I highly recommend Rich Hickey's "Simple Made Easy" talk as a way
| to think about software complexity. We all know that complex
| systems are hard to work with, but I've found his presentation
| very useful in understanding the nature of complexity.
| Understanding complexity helps being able to avoid or minimize
| it. For example, easy is not the same as simple, and simple
| changes to one part of a system can introduce great complexity at
| the macro level.
|
| https://www.youtube.com/watch?v=LKtk3HCgTa8
| vlunkr wrote:
| I hadn't heard that before, it's great.
|
| I like how he states that simpler software doesn't necessarily
| mean fewer components (roughly quoting). Where the article
| argues for fewer boxes and arrows. It's hard to argue
| specifics, because the article isn't getting into them, but
| sometimes more boxes and arrows makes things simpler. If you
| don't have multiple boxes, what do you have? One box of
| spaghetti?
|
| I also disagree with the article that engineers like
| complexity. I've maybe known a couple, but usually they are
| really junior, or really unpopular.
| appleflaxen wrote:
| I've watched that video at least three times, but I still can't
| articulate the exact distinction he is making between simple
| and easy :/
| aidenn0 wrote:
| Add two 10000 digit numbers by hand. That's quite simple to
| do, but it's certainly not easy.
| kortex wrote:
| A for loop is easy: everyone can conceptualize doing a
| process N times.
|
| A map is not as easy of a concept in some ways (it's more
| abstract, you are taking N elements of X to N elements of Y).
| But a map is _simpler_ than a for loop, because there is no
| state between steps: it 's completely symmetrical.
|
| The for-loop "complects" each iteration together, even if
| it's as little as incrementing an integer. It's a side
| effect, however trivial.
|
| To kick it up a notch, now imagine a 2-D loop. Now we have 2
| counters. If it's a 2-D array, each addressing operation
| interacts with 2 counters. But map just needs a second map
| chained on, and each map is completely "unaware" of the other
| map. You could map over N dimensions and each map is
| symmetrical, while the for loop needs an ever increasing
| number of counters. This slight difference in complexity
| shows how complexity compounds with other complexity.
| jacobsenscott wrote:
| This "software grows too complex" is certainly a culture issue,
| and not a software problem. It mostly comes from businesses
| throwing a bunch of under-skilled labor at the problem, and
| demanding more and more too quickly. Of course it will outgrow
| the ability of that team to work with, reason about, keep clean,
| and extend. They will get frustrated and move on, and a new group
| comes in, re-writes it from scratch, and so it goes.
|
| Look at software that has been around for decades, continues to
| improve, never "needs" a re-write, and will never be re-written.
| It exists and is not uncommon. Things like the linux kernel, all
| the GNU utils, emacs, vim, apache, nginx, postgres. The list goes
| on. We know how to write software that lasts for decades,
| continuously improves, and doesn't need constant re-writes. You
| give it to skilled people who care about it, and don't put
| artificial corporate constraints or impose fad architectures on
| them.
|
| I'm not saying the corporate model is "wrong" - it works. "Works"
| means it builds companies and generates profit. That is the goal.
| But it will not generate stable long lived well designed
| software. That's okay though because that isn't the goal. Sure,
| the poor quality cuts into the margins a bit, but not enough to
| matter.
| bigbillheck wrote:
| > emacs, ... don't put artificial corporate constraints
|
| The historical record disagrees. See for example RMS squashing
| better clang+emacs integration:
| https://lists.gnu.org/archive/html/emacs-devel/2014-01/msg01...
| jacobsenscott wrote:
| Weather you agree with this decision or not, it is consistent
| and clear. Something you generally won't get in corporate
| decision making. That is worth a lot. You know RMS isn't
| going to come back six months later and reverse it.
| dale_glass wrote:
| Many of those did need something approaching a rewrite and even
| got one of sorts, but just were big enough that there was
| plenty that was unaffected by whatever big change was needed.
|
| The Linux kernel for instance gets big changes. The BKL took a
| very long time to fully remove.
|
| The GNU utils needing a rewrite would be weird since most of
| them implement a pre-existing standard, so the need for a huge
| rewrite is very unlikely.
| bezospen15 wrote:
| Odd
|
| I thought it was caused by every little piece of software being
| tied to giant product teams full of MBA majors that want to add
| 50 features a week
| sethetter wrote:
| I'm curious how big of a factor a consistent core team is to
| having long-term sustainable and stable software. When
| companies have people rotating on projects constantly and the
| original vision and design principles of the product are lost
| or changed, I can only imagine that contributes to the problem.
| phendrenad2 wrote:
| I think that's a big factor. This is what RSUs and bonuses
| are supposed to solve: You entice employees to stay rather
| than job hop. However, companies have gotten too greedy and
| RSUs amount to a pittance, a vestigial perfunctory play act
| mimicking a bygone era.
| Gigachad wrote:
| The developers have also gotten too greedy and most of them
| will job hop every year for the maximum income.
| amenghra wrote:
| It's common for tech companies to give RSUs which vest over
| 4 years with annual refreshers being comparatively quite
| small. The financial incentives can dwindle after just 4-6
| years at a given job.
| regularfry wrote:
| The gotcha to that, of course, is that while you're
| enticing people to stay in the company, you're doing
| nothing _necessarily_ to keep them on a single project
| within that company.
| likortera wrote:
| Best comment I've ever read on HN.
| theteapot wrote:
| > Look at software that has been around for decades, continues
| to improve, never "needs" a re-write, and will never be re-
| written. It exists and is not uncommon. Things like the linux
| kernel, all the GNU utils, emacs, vim, apache, nginx, postgres.
|
| They don't get re-written because they can't be re-written
| without tearing down everything built on top. In every case
| though there is some newer software out there is inspired by
| but aiming to "fix" all the short comings of the predecessor.
| Glavnokoman wrote:
| Half of software you mentioned is just garbage inside. And it
| will never be rewritten simply because it happens to work
| good enough to discourage any serious investment in rewriting
| it.
| mike_hock wrote:
| Linux doesn't rewrite? Is writing a completely new I/O
| scheduler from scratch and replacing the old one not a rewrite?
| How much does a 5.14 kernel have in common with a 2.6.8 one?
|
| If anything, Linux isn't afraid to rewrite the parts that need
| to be rewritten, and that's one of its strengths.
|
| Apache, Vim, and Postgres have been "rewritten." Nginx _is_ an
| httpd written from scratch that sort of has the same features
| as Apache. Neovim exists. And newer database technologies that
| care about performance have been written.
| throwaway894345 wrote:
| > It mostly comes from businesses throwing a bunch of under-
| skilled labor at the problem, and demanding more and more too
| quickly.
|
| Even if you have plenty of skilled engineers, it doesn't mean
| that the business won't impose unrealistic timelines (and
| insist the timelines be met even if it means taking on loads of
| tech debt which never gets paid down) or change the
| requirements abruptly or even demand things that are inherently
| complex (without understanding the complexity).
|
| In other words, there are plenty of ways that management can
| create complexity, and in my experience management-induced
| complexity is the most common kind (note that I'm faulting
| "management" for things that include "communication between
| engineers and management" because management ultimately owns
| the software development process).
| lytefm wrote:
| Engineer induced complexity is also common. I've never heard
| about management pushing for a scalable, event-driven,
| microservice based architecture when a simple monolith
| serving a few thousand daily users would be just fine.
| throwaway894345 wrote:
| Maybe. I've never worked in an organization where engineers
| were driving the complexity, nor have I worked in an
| organization where microservices introduced complexity
| (I've heard people complain in vague terms about
| microservices, but I don't relate to their experiences).
| However, I _have_ worked in an organization that
| transitioned from monoliths to microservices, and the
| overall complexity was reduced because (I didn 't mean for
| this to turn into a monolith/microservices thread, but oh
| well):
|
| 1. an engineer under management pressure couldn't easily
| take an expedient dependency on the private internals of
| another system
|
| 2. teams could deploy their changes on their own cadence,
| so they didn't feel additional pressure to squeeze poorly-
| conceived things into one less-frequent release (many
| frequent releases are theoretically possible for monoliths,
| but much rarer for pretty understandable reasons)
|
| Moreover, reliability improves because of the
| aforementioned reduced complexity but also because the
| blast radius for a reliability issue is scoped to the
| service, and security improves for the same "blast radius"
| reason (an attacker who compromises a component can only
| access that component's secrets and can only talk to that
| component's collaborators).
|
| My suspicion is that microservices work very well when an
| organization has a strong architecture/SRE capability
| irrespective of management or product development
| capabilities, whereas a monolith depends on all
| capabilities firing on all cylinders.
| dasil003 wrote:
| "Microservices" (ie. SOA) absolutely adds complexity, the
| question is whether the benefits are worth the overhead.
| The answer varies according to domain, but is largely a
| function of team size.
| throwaway894345 wrote:
| There's definitely some complexity inherent in
| microservices which isn't inherent in monoliths, but that
| doesn't mean a _net_ increase in a given microservice
| relative to an equivalent monolith because microservices
| disincentivize other kinds of complexity (which monoliths
| don't disincentivize) as mentioned above.
|
| The _net complexity_ may well be (and often _is_ IME)
| less for a microservice than for an otherwise similar
| monolith. _In addition to_ this net complexity
| discussion, microservices also have security,
| scalability, and reliability benefits.
| miscaccount wrote:
| Well probably you have never met management that is keyword
| drive. We need 100 percent scalable system with zero
| downtime upgrade and support thousand customers update.
| Current number of customers : 0 Number of customers in
| pipeline: 0
| 331c8c71 wrote:
| Vim was rewritten after all (neovim) and not without a reason.
| turminal wrote:
| > Things like the linux kernel, all the GNU utils, emacs, vim,
| apache, nginx, postgres. The list goes on. We know how to write
| software that lasts for decades, continuously improves, and
| doesn't need constant re-writes.
|
| At least half of the items on that list are still here purely
| on momentum and reputation from the time there were no
| alternatives, not because they were really that marvelous.
|
| I use and even enjoy some of them, but they're far from great
| engineering.
| DoneWithAllThat wrote:
| I would disagree with that sentiment for every single item on
| that list. All those tools are amazing and have improved
| steadily over decades and I would absolutely put them in the
| "great engineering" category.
| throw10920 wrote:
| GNU utils are incredibly primitive and their text-driven
| APIs are fragile and cause tons of issues in shell
| scripting.
|
| I love Emacs, but it's a slow, crufty mess.
|
| Vim is a huge pile of code with a _lot_ of complexity in
| its codebase and hilariously bad design decisions like
| VimScript.
|
| Apache is a mess from a security perspective (1.7k listed
| CVEs[1]), and still slower than Nginx.
|
| Want to guess at the amount of lost productivity due to
| Apache vulnerabilities?
|
| [1] https://www.cvedetails.com/vulnerability-
| list/vendor_id-45/A...
| froh wrote:
| What would be counterexamples to that, i.e. great
| engineering? FOSS possibly?
| nextos wrote:
| seL4. Simple and formally verified.
|
| I think it's time to pour VC money on a fully verified
| stack.
|
| Perhaps not everything needs to be proven, integrity for
| some components can be ensured more informally with e.g.
| refinement types.
|
| Lots of governments and organizations need more robust
| software.
| turminal wrote:
| seL4 is great in theory, and formal verification is
| awesome, but I read some of their code recently the
| implementation leaves something to be desired.
| pclmulqdq wrote:
| Also, most of them have gotten a rewrite worth of upgrades
| during their life. The Linux kernel is a good example: most
| of the code you use today didn't exist in the kernel 15 years
| ago. A lot of code from back then still exists for
| compatibility purposes, but is rarely used in modern systems.
| regularfry wrote:
| But that's an interesting facet in itself. The more common
| refrain you hear in corporate settings is "we need a
| ground-up rewrite" - how did these projects manage not to
| need that? I have my suspicions, but it's a very
| interesting question nonetheless.
| pclmulqdq wrote:
| As a big fan of rewrites (but not as big a fan as some),
| I think Linux has avoided needing big rewrites with a lot
| of very clever modularity in its design. Many corporate
| software programs do not have that level of modularity.
| Modularity has performance costs, is slower to develop,
| and "looks ugly" on a block diagram, but it pays
| dividends for open source projects and projects that need
| upgrades over time.
|
| Individual parts of Linux go through relatively frequent
| rewrites (eg filesystems), but not the overall framework.
| analognoise wrote:
| Linux IS corporate. It has been corporate for a long time
| - the heaviest commits come from paid, professional
| developers.
| regularfry wrote:
| Not, I think, in the sense meant here. You're right, of
| course, but the distinguishing factor is that "corporate
| software" typically only involves a single corporation.
| There's no (or little) balancing of interests.
| int0x80 wrote:
| Final decisions are made by totally independent folks
| like Linus, Greg or Andrew. They set the rules and also
| the community, corporate has to adapt to them or go away.
|
| Also, what happens is that rewrites happen progressively,
| in small steps.
| astrange wrote:
| I think nginx is a rewrite of apache, just as busybox is a
| rewrite of coreutils. Linux tends to rewrite itself but just
| keep the userspace ABI.
| donatj wrote:
| > nginx is a rewrite of apache
|
| I mean not really. It does a tiny fraction of what Apache
| does, it just happens to do that specific fraction better.
| astrange wrote:
| That's disruption theory at work.
|
| https://en.wikipedia.org/wiki/Disruptive_innovation
| yyyk2 wrote:
| Busybox is not a rewrite of coreutils, it serves a completely
| different purpose.
| pablok2 wrote:
| Allowing projects to fail is important too, it just happens
| that complexity plays a big role in these failures.
| jinwoo68 wrote:
| In the corporate world, "leaders" are incentivized by having
| more people in their orgs. It makes them have more "power". So
| they constantly broaden the scope and try to hire more people.
| The leaders in software orgs never benefit from finishing
| projects early with less engineers. Nor do they benefit from
| maintaining stable and robust software. Making the project take
| longer with more engineers with complicated architectures makes
| them gain more power. Yeah, this makes me sad.
| [deleted]
| olah_1 wrote:
| Complexity is something that is underrated especially in the
| crypto-currency world.
|
| They claim to be building for the next hundreds of years, but you
| wouldn't know it.
| popcorncowboy wrote:
| tl;dr premise = society is complex, software is complex; complex
| societies collapse... therefore complex software collapses,
| question mark, waggling eyebrows, surely makes you think etc etc.
|
| > ...Opinions expressed in this blog are mine *and frequently
| wrong*.
|
| Emphasis added.
|
| Equating the complexities of software with the complexities of
| societies (the premise of this post) is a fun and provactive blog
| post. Which is all this is. I was really hoping for a serious
| treatment that meaningfully dug into this, but that's my problem.
| Forge36 wrote:
| OS/360?
|
| https://sudonull.com/post/111636-IBM-System-360-A-Failing-St...
|
| I think we've seen this with the rise and fall of hardware
| vendors. Most recently with the Mobile OS ecosystem (symbian,
| windows mobile)
| narag wrote:
| I don't buy the explanation of societies collapsing because
| they're too complex, Some software systems do.
|
| But not because some esoteric systemic property of complex
| systems, but just because working projects get deprived of
| maintainers and eventually there's nobody that understands WTF
| original programmers intended and how to fix the holes.
|
| So incomprehensible portions of the system are circumvented or
| replaced with something that just works. Actually the same
| happens in societies, unless some group sabotages the
| workaround.
| rglover wrote:
| The more abstract the mental model is for building _the thing_
| --at the tool/code level--the more likely it is that the thing
| being built will be a difficult to maintain mess, if not an
| absolute "burn it all down and start from scratch" failure.
|
| Fix the tools, and you _start_ to fix the code. Unnecessary
| terminology, confusing APIs, superfluous patterns...it 's like
| watching someone blindfolded in a dark room trying to justify the
| high price tag on their computer science degree.
|
| After years of running my mouth, I proved my rants about the
| unnecessary complexity by building my own JS framework [1] and
| I'll never look back. The emperor is bare-ass naked.
|
| [1] https://github.com/cheatcode/joystick
| kitanata wrote:
| This is crazy. I just gave a nearly identical presentation for a
| series of interviews. One at AKASA, one at Scribd, and one at
| Panorama. Like I gave this presentation, with the exact same
| title 2 weeks ago to a bunch of junior engineers at these various
| companies. It is super timely. I'm consistently surprised how
| multiple people can arrive to the same point, via the same
| inspirational sources at the same time and reach completely
| different conclusions. I'll have to write up a blog response to
| this article.
| LeonenTheDK wrote:
| Please do! I'd love to see more takes and discussion on the
| matter.
| commandlinefan wrote:
| > engineers like complexity
|
| Is that really true? If it's descriptive, then I'm not an
| engineer (and I may well not be). I don't like complexity at all,
| in any form. I want the least complex solution to any problem
| that will work. However, I've mostly given up on decomplexifying,
| because it's mostly a waste of time. Software becomes complex
| because getting something done _fast_ , _right now_ , "this
| time", because, "we're in a hurry this sprint" by duplicating
| code will always win over trying to modify something to be
| reusable. If you try to spend time fighting complexity, you won't
| just be fighting the complexity inherent in the existing system,
| you'll be fighting your coworkers who've given up on fighting
| complexity and just get their tickets into a done state.
| roansh wrote:
| This is a good talk by Jonathan Blow on/against software
| complexity https://www.youtube.com/watch?v=pW-SOdj4Kkk
|
| Also talks about the risk of losing knowledge, and a way to avoid
| that (make multiple copies).
| neoneye2 wrote:
| Great talk indeed. Just watched it. Thanks.
| nonrandomstring wrote:
| Very nicely written. A lot of gems in this short essay. What
| stood out for me was:
|
| > It takes a lot of discipline to resist complexity
|
| That can be taken in many ways.
|
| Self-discipline, to resist convenience, the omniscient and
| omnipotent desire to control everything and always have the right
| answer immediately. That takes discipline to let go of. Also to
| discipline others, to say no, push back, become the master and
| set firm boundaries against crushing 'demands' for complexity.
| And intellectual discipline, to see clearly, elegantly,
| eliminating what is unnecessary. Over-complexity is a failure of
| the ordering principle of an organism, a step on the road to
| chaos and entropy.
| daenz wrote:
| This is a problem I want to focus my life solving. I believe that
| software engineering _can_ be made so simple and clear that it
| can be wielded by an extremely small team of engineers. I believe
| that there are finite and fundamental classes of problems, that
| underpin the vast majority of problems, that have a distinct
| visual representation, and that by representing these problems
| visually, you let your "visual coprocessor" assist in seeing the
| solutions. I believe there are well-known strategies (taken from
| the real world) for managing hierarchies of complexity that ease
| the cognitive load of exploring any problem. We just need to get
| these approaches into a programming "langugage"!
| mamcx wrote:
| I think the same, but that requiere resources.
|
| I try to work on the side in a _spiritual_ successor of Foxpro
| /DBase (https://tablam.org).
|
| I consider a mix of relational/array model fill a lot of
| ergonomics for this (and it was proved to be right by the
| family of DBase langs).
|
| But what makes this much more complex today is the explosion on
| OS targets (Windows, Linux, MacOs, Android, iOS, Web), and the
| requirement to integrate with many other stuff, dealing with
| many formats (json, xml, ...), is harder to do UIs now and the
| base support is more inconsistent and mixed as ever...
|
| So, is possible to make a simpler tool, I certain of it, but
| then the developer/user will say "ah, ok, so how this connect
| to Redis, GraphQL and Amazon Web Services, run this on Android,
| Windows, parse CSV, ..."
|
| and that is what make this very hard at the end...
| astrange wrote:
| This is essentially Alan Kay/VPRI's old research program. I'm
| not sure why, but when people get into this they always think
| dataflow programming will solve everything - we already have
| Excel and it does do a lot, but it doesn't do everything.
| randomsearch wrote:
| I agree with you. I have other things to do, but good luck to
| you.
|
| There's an awful lot of cultural baggage in coding. Many of the
| concepts that seem essential - powerful text editors, devils
| tooling - can be completely removed. It requires rethinking
| from first principles and being willing to upset the Apple
| cart, but I believe it is possible.
| notatpeace wrote:
| May be you are looking at a much fine grained
| composition/decomposition tier to arrive at breaking down "all"
| software engineering towards simplicity, but anything real
| world usecase would end up being an aggregate so big, your
| composition/decomposition tier itself would add to complexity
| rather than reduce it.
|
| BPMN and all of the UI pallet, drag drop coding applications
| try to do this very thing, this is super successful at smaller
| scales but breaks the first real world application.
| davidgay wrote:
| > I believe that software engineering can be made so simple and
| clear that it can be wielded by an extremely small team of
| engineers.
|
| Not that I want to discourage you, but my view is that anything
| that makes software engineering simpler just leads us to
| tackling more complex problems until the complexity reaches the
| limit that people can handle.
|
| So in that view, you can't succeed at making software
| engineering always simple. Instead, you can make previously
| intractable problems tractable.
| daenz wrote:
| If history is a guide, then you are correct. But I believe
| complexity is fractal (I think we all know this: "turtles all
| the way down..."), and that you can structure and manage
| complexity with recursive rules, so that you only see the
| resolution that you care about, for the areas that you care
| about, and everything else is a low frequency representation.
|
| Frameworks approximately do this, but for specific domains:
| they let you organize complexity into well-defined areas and
| put a pin in them, easing the cognitive load. Then you can
| handle those abstractions more easily. But yes, to your
| point, because frameworks typically only tackle a few levels
| of complexity, you still get that complexity back when you
| use the framework to advance the problems to the edge of what
| your framework is designed to address.
|
| A recursive/fractal management of complexity will allow all
| levels of the hierarchy to feel similar, so you are never
| increasing complexity, only looking at a different
| resolution. I think the key to this is mapping out the
| fundamental organizational problems and how they relate to
| each other at different resolutions.
| nradov wrote:
| The complexities at different levels of the hierarchy are
| so qualitatively different that I doubt whether a unified
| set of abstractions could ever be practically useful. We're
| going all the way from CPU microcode to loosely-coupled
| distributed systems with multiple dependencies on third-
| party services. If you want to make any progress you'll
| need to narrow down your focus.
| ItsMonkk wrote:
| > tackling more complex problems
|
| This is the 'induced demand' argument. As perfection seems
| impossible, and mistakes inevitable, this seems likely.
|
| However, the more I look into issues, the more I realize
| that a very small number of errors introduced early on is
| what ultimately causes a plethora of them. Just fixing a
| very small amount of these mistakes would have untold
| effects on computing over the long-term. That is, if we can
| get over the initial switching costs.
|
| > small team of engineers
|
| It took 2 men to over-take thousands creating Unix over
| Multix. It took Linus Torvalds - just about alone - a week
| to create git. We have already seen this prophecy come
| true.
|
| As teams get bigger, communication sales factorially, and
| the more people you have the more mistakes you make, which
| increases the size exponentially with each further mistake.
| What Unix and git showed is that when you put everything
| into a small team of engineers heads, they can work through
| the complexity enough until they can do it themselves.
|
| > Frameworks
|
| One of the things I realized a while ago is that
| pure/impure and library/framework have a decent mapping
| between the two. If you have a framework, you give it code
| and it acts for you, just like impure code. And so the
| problem we keep running into, is instead of inverting the
| flow like 'hexagonal architecture' says, we keep piling
| impure onto impure onto impure. Hexagonal says to not do
| anything of substance, keep the adapter clean, but each
| framework sure is doing something. Each layer on the stack
| we go up, the harder it is to get down. So now we have OS
| and applications and containers and k8s running micro-
| services that ends up being run in a web browser, when all
| we really needed was microkernals.
| Gigachad wrote:
| This is true of almost everything. The inputs / resources
| dedicated are relatively fixed while the outputs are
| dependant.
| jandrewrogers wrote:
| This assumes that complex software problems are decomposable
| into discrete components defined and analyzable solely by their
| public interfaces. Complex software frequently cannot be
| modeled in this way -- it is what makes the software "complex".
| In reality, components have complex interactions far outside of
| what is captured in the component interfaces (see also: Hyrum's
| Law). Surfacing these implicit interactions at the component
| interface level just creates _insanely_ complex interface
| definitions, moving the complexity somewhere else.
|
| To tie it to something concrete, just about everyone thinks C++
| template metaprogramming is unreasonably difficult to reason
| about even aside from the syntax, but it exists to be extremely
| powerful at precisely expressing behavioral contracts of a
| software component that are difficult to do in other languages.
| Even then, it _barely scratches the surface_ of what would be
| useful to express for the interaction of those components to be
| "simple". The number of design parameters of a component that
| really matter in various contexts are astronomical. No one can
| deal with reasoning about that many component traits in
| practice, so the software engineers simply hide most of them --
| the complexity is still there and will manifest in unexpected
| ways because it is not visible at the interface.
|
| All systems engineering is complex for this reason. Any non-
| trivial software system has to be reasoned about as a monolith
| at some level to correctly address complexity, which creates an
| unavoidable cognitive load. This isn't a software engineering
| problem, it is a systems-thinking problem. In chemical
| engineering, for example, there are often complex system design
| problems that cannot be adequately addressed by decomposing
| them into a sequence of sub-problems, all that does is hide
| major complexity at the interface of the sub-problems.
| bigbonch wrote:
| There are many embellished, hand wavy statements in this
| comment.
|
| Could you give real world examples of: * "In
| reality, components have complex interactions far outside of
| what is captured in the component interfaces" * How
| template metaprogramming "precisely express[es] behavioral
| contracts... that are difficult to do in other languages"
|
| I think you contradict your poetry with "Any non-trivial
| software system has to be reasoned about as a monolith at
| some level to correctly address complexity". If _any_ non-
| trivial software system _can_ be reasoned about as a monolith
| then it is true there exists some model with sufficient
| simplicity that there are well-defined discrete components.
| gmfawcett wrote:
| That's a nice belief, but you've given no rational argument
| for it. Consider a monolith implemented as a single,
| unstructured function: 200,000 lines of unstructured code,
| written by a goto-loving maniac. Demonstrate why some
| equivalent model with well-defined, discrete components
| must exist.
| go_elmo wrote:
| I am sure every SE agrees that somehow things are always the
| same. How about making code pieces algebraic in order to reason
| and write with them much easier? Like using Haskell?
| nescioquid wrote:
| I attended a Common Lisp convention at MIT around the year they
| did away with the SICP course and replaced it with a course
| using Python to program robots. It came up in the Q&A of one of
| the talks and the question got kicked to Sussman who was in the
| audience and gave an explanation.
|
| He said that it used to possible, practically speaking, to
| treat software like discrete components so that you can reason
| about how to combine the units and predict the result, etc. By
| contrast, much programming today is bodging together a bunch of
| libraries whose behaviors are often poorly defined or otherwise
| opaque so that one often enough has to treat them like black
| boxes against which you have to apply scientific reasoning to
| understand how to use and integrate it. Hence Python and
| robots.
|
| Presumably the new course teaches skills around managing this
| complexity, but the shift in model advocated -- from the
| metaphor of well-behaved discrete components in composition to
| opaque units whose inputs and outputs (possibly depending on
| hidden state) must be discovered was interesting to me, and
| ultimately adds to the accidental complexity of doing software
| now.
|
| Do you think this sort of problem may have a distinct visual
| representation that would fundamentally reduce the complexity
| of library-dominated development? Or are you imagining a whole
| programming ecosystem built around your idea, so that the
| fundamental problem Sussmen described is somehow obviated? Or
| are you really trying to address essential, not accidental
| complexity?
| papito wrote:
| It's a solved problem, if you are solving the right one. Most
| companies out there are solving problems they don't have. The
| FAANG disciples descended upon the world to preach the gospel
| of Complexity, and everyone bought it - because everyone wants
| to be like Google.
|
| What, you think "microservices" is a new invention? We called
| it distributed systems, and we _really_ knew not to go there
| unless we wanted to decimate our productivity and sleep.
| michaelbuckbee wrote:
| What you're describing is one approach, the other is that you
| pay a bunch of SaaS/PaaS/IaaS companies to handle a lot of the
| workload for you so you and a small team can just focus on the
| business logic of your particular niche.
| yhoiseth wrote:
| Agree that this helps in some ways, but it also adds
| complexity. You can e.g. subscribe to a billing SaaS, but
| then you have to link your system to the billing system
| somehow
| AnimalMuppet wrote:
| I don't think you can solve it. (Not a comment on _you_ - I don
| 't think _anyone_ can solve it.) The problem is that software
| at least to some degree reflects the external environment.
|
| Let's say your software is in a financial company. Their
| software has to enable them to follow all the government
| financial regulations. Well, the government is following the
| larger societal "complexify to the point of collapse", and the
| financial regulations are certainly doing so. That means that
| the external behavior (the "business logic") of the software is
| insanely complex. You can't make that go away just by visual
| programming.
|
| But maybe you're not in the financial world. Maybe you're just
| writing programs for internal corporate processes at some
| generic company. Well, your software is still subject to the
| complexity that builds up in the company processes. Again, the
| programmers can't eliminate that complexity.
|
| Or maybe you're writing a customer-facing app - an external-
| facing web app, or an application that people actually install
| on their machines. Here you're at the mercy of the product or
| project manager trying to find new things for the app to do,
| and they still complexify the app to the point of collapse.
|
| The problem isn't that programming is too complicated. The
| problem is that _what we want programs to do_ is too
| complicated. Visual programming can 't save us from that.
| lbriner wrote:
| Good luck but your motivations will probably not line up with
| the masters who dictate things that become complexity.
|
| For example, I look at our software running on .net MVC and
| think as an Engineer, it simple and knowable but the Front-End
| Team are less worried about simplicity and more on flashy
| front-end stuff since that is their job. We end up bolting on a
| Front-end JS framework and complexity immediately ramps up by
| like 300%.
|
| Are they wrong for wanting a better and more flexible front-
| end? Not necessarily, I mean all the other companies have cool
| stuff and if we don't maybe our company dies.
|
| Ditto for lots of other examples...
| chasd00 wrote:
| this reminds me of the Head First series of books. I can still
| recall specific pages from the Head First Statistics book not
| because of the content but because every topic (and probably
| page) had a creative visualization.
| gjvnq wrote:
| I think that's an honourable but unattainable persuit.
|
| A huge part of the problem is the messiness of the real world
| and costs.
|
| While there may be a finite set of fundamental problems, the
| set of possible hardware and software platforms is ever
| increasing and each has it's own constraints and strengths.
|
| A "universal programming environment" would need something like
| a "universal hardware interface". That's why things like Java
| and the Web became so popular despite their poor design.
|
| Also, visual programming is a hard thing to make work and human
| language skills seem to be far greater than human visual
| skills.
|
| Perhaps the best that can be done is "starting from zero" and
| making sure everything, from the ICs in your hardware to the
| memory models behind your software is thoroughly tested and
| formally verified.
|
| This is insanely expensive with current technology. Perhaps
| some fancy math and AI advancements can make formal
| verification powerful enough for ubiquitous use. Until then, I
| see little hope for a "universal programming environment".
| gtowey wrote:
| > I believe that software engineering can be made so simple and
| clear that it can be wielded by an extremely small team of
| engineers.
|
| I think one thing missing from this line of thinking is that
| these are _people_. They have their own motivations and egos.
| They get sick, have families, take vacations, quit.
|
| I think part of the way large teams and large companies are
| structured is a hedge against changing teams with highly
| variable capacity. It's hard to overstate how much it can hurt
| a project to lose a someone who has a ton of experience locked
| up inside their head.
|
| The alternative is that I have worked for places that try to
| treat thier engineers like fungible tokens that can be shuffled
| and replaced at will. That environment feels extremely
| dehumanizing and demotivating.
| perlgeek wrote:
| I want to believe you, but there are just so many complicated
| problems IN REALITY that we have to model in software that I
| don't really see complexity going down.
|
| Just an example: around here, most people have a first (given)
| and a last (family) name. If I don't model that as separate, I
| have trouble interfacing with other software. If I do, I have
| trouble with people from other cultures that don't follow that
| convention. Storing both risks the data going out of sync.
| What's the "right" way to store person names? There doesn't
| seem to be a simple solution.
|
| Another example: we model physical cables (both for power grid
| and for data transmission) in our CMDB. All works fine, until
| you suddenly have a Y-shaped cable with three connectors that
| doesn't fit into your data model. The real world always has
| these 1% of cases that don't fit the general pattern; if you
| focus on the 99%, the 1% make trouble. If you focus on modeling
| _every_ case, you have 10x the complexity, even for the simple
| case.
|
| And then there are things that are moderately complex and
| security critical, like password recovery workflows. We haven't
| really found a way to reuse these among different technologies.
| Like, if you once figured out the perfect password reset
| workflow with Ruby on Rails, and your next job uses Python +
| Django, you're back to square one.
|
| If somebody has a good idea for how to tackle these problems,
| please let me know!
| nightski wrote:
| As you implied, I don't think the answer is to make the
| perfect abstractions that can handle any scenario.
|
| But where software does fail in my humble opinion is making
| it easy to pull in tried and true tested solutions to the
| problems that we do face, even if they are not as common.
| Because even though they may not seem common, I'm absolutely
| certain many face the same scenario.
|
| The amount of duplication solving the same problems is
| insane. But this is not an easy problem to solve and I don't
| intend to trivialize it.
|
| I think we need to come up with better tools. code sharing
| through libraries/repositories (like npm) is great, but it
| can't be the final solution.
|
| Back in the day in Haskell I dreamed of a system where you
| could type out a type signature and a fully tested rated
| implementation would be imported from an "open source"
| service. You could import modules, functions, data
| structures, anything. But that vision is still a long ways
| off.
| feoren wrote:
| The problem is "pulling in a tried and tested solution"
| means pulling in a slice of the whole stack: CSS,
| Javascript, client/server communication, API endpoints,
| data-flow logic, and database schema. People are very very
| bad at this because they believe that software engineering
| principles apply only to little tiny portions of this
| stack. If you actually apply your software engineering
| principles to the whole stack, you end up with something
| that looks very different than what everyone else is doing
| (so there's not a lot of tooling to help with). I have a
| better way to do this than anything else I've ever seen,
| but even with that I struggly very strongly with the JS/API
| side.
| notriddle wrote:
| > Back in the day in Haskell I dreamed of a system where
| you could type out a type signature and a fully tested
| rated implementation would be imported from an "open
| source" service.
|
| The type signature of GPT-3 is `string -> string`
| nightski wrote:
| Well I'd argue it's more Model => String => String if you
| are talking about using a fully pre-trained model. Even
| then you could probably expand on the type signature of
| the model to make it more useful.
|
| But if you look at like co-pilot for example, if that was
| given the ability to have type signatures serve as input
| you might get a lot more powerful results than what it
| does with raw text (which is very impressive).
|
| But this comes down to type signature design. You can
| encode any function using simple types like int -> int
| which aren't very useful. Where Haskell shines is when
| using types to limit the scope of inputs & outputs. What
| I am getting at is that you can still write uninformative
| type signatures in Haskell, but it also gives you the
| power to write more informative ones.
|
| I don't think Haskell is the answer, so please don't take
| that as what I am saying. I do think however using richer
| type systems could be a stepping stone towards a solution
| to this problem.
| astrange wrote:
| Type signatures (or ASTs) are structured data, and
| Copilot mainly works because GPT accepts any text and
| doesn't have to deal with your logical fully recursive
| concepts like ASTs.
|
| I feel like it wouldn't work if you could actually
| mathematically constrain the outputs to be syntactically
| correct either. That's probably one of those Godel
| things.
| a1369209993 wrote:
| > What's the "right" way to store person names?
| struct Screenname { Matches<String,regex"\w+"> _;
| // TODO: ensure it is illegal to ask for users' real names
| // for now we'll just do the right thing unilaterally
| }
| fmakunbound wrote:
| Works for email, phone numbers and addresses, too.
| a1369209993 wrote:
| Actually, it doesn't. All of those are, well, _addresses_
| , and have some mechanical structure necessary for
| routing (admittedly, for phone numbers, the structure
| amounts to "any sequence of 0-9, #, *, and possibly those
| four wierd extra symbols with the fourth DTMF column
| tone"). You probably don't (or least shouldn't) _care_
| about that structure as long as it routes correctly, but
| it does exist. (Ie, you 'd still use a string for that
| data, but it's not _inherently_ incorrect not to, just
| pointless and error-prone.)
| yellowapple wrote:
| > What's the "right" way to store person names?
|
| Store both a unified name field and separated given name /
| surname fields and let the user manage both. Yeah, this risks
| going out of sync, but that's the user's problem, not yours.
| Yeah, three fields are technically more complex than one or
| two, but it produces less complexity down the line.
| nradov wrote:
| Falsehoods programmers believe about names:
|
| https://www.kalzumeus.com/2010/06/17/falsehoods-
| programmers-...
|
| As a practical matter, the HL7 FHIR data model for names can
| work well enough for the vast majority of applications in
| almost any country.
|
| https://www.hl7.org/fhir/datatypes.html#HumanName
| pphysch wrote:
| > Names
|
| Don't try to force schemas onto schema-less data. Store the
| "name" as a JSON string/blob representing the various
| possible attributes (given, middle, family, title, etc) and
| provide a variety of functions for representing that data. IF
| you really need to do this at all (for an internal app, you
| probably don't).
|
| > Physical links
|
| Include an Hardware Asset FK in your Link M2M table. Model
| each binary link explicitly, so a Y cable = 2-3 different
| Links that point to the same cable Asset. Or you can have
| single Links with M2M inputs/outputs. But definitely don't
| model Y cables explicitly.
|
| > Password recovery
|
| What is so complicated about this, specifically? Python and
| Ruby are completely different languages with no guarantees
| for interop.
| corrral wrote:
| > We haven't really found a way to reuse these among
| different technologies. Like, if you once figured out the
| perfect password reset workflow with Ruby on Rails, and your
| next job uses Python + Django, you're back to square one.
|
| C and C++ libraries with light language-specific wrappers
| largely serve this purpose, in practice. It's plausible that,
| say, a PHP postgres client lib and a Node postgres client lib
| will share nearly all their code--probably as C or C++.
|
| This doesn't get leveraged much aside from interfacing with
| daemons and sometimes for extremely complex e.g. media or
| crypto libraries, though. No-one's doing this for high-level
| software workflow building-blocks, like a password reset
| flow.
|
| [EDIT] as for the "why", I suspect it's because the things
| it's used for are _far_ simpler than the things it isn 't. A
| password reset flow can potentially need to interface with
| _lots_ of different things, some of which may be custom to
| the project, some of which may vary with run-time input, et
| c. If you cut it down to only the parts that could truly be
| re-used anywhere, with all kinds of points where you can hook
| in as needed... you 've only made about 5% of the work re-
| usable, so it's just about pointless.
| rubyskills wrote:
| I'd say this is more true for C vs C++. It has been my
| experience that the interop between C and languages like
| Ruby is much easier, the tooling is usually much better
| (just use ffi!), and the complexity is less (since that is
| what we're aiming for right?)
| andrepd wrote:
| Can you elaborate and/or give examples? I read your comment and
| I can only see vague pseudo-profound statements.
| swatcoder wrote:
| This is an admirable and worthy pursuit, but keep in mind that
| any process you make streamlined:
|
| 1. becomes a post from which clever and ambitious people can
| build complex things. You invent docker to simplify application
| deployments and then somebody builds a n-dimensional
| microservice cloud on one side and starts commercializing new
| hardware architectures on the other. You don't remove the
| complexity, you just let it move around into new domains. (not
| a bad thing!)
|
| 2. is more temporal than you expect. The "finite and
| fundamental" problems of yesterday, today, and tomorrow are of
| different sets -- partly because (1) opens up new problem
| classes and partly because most problems are inescapably
| cultural and therefore subject to fashion cycles.
| likortera wrote:
| Don't want to discourage you...but... LOL
| rileyphone wrote:
| Me too, and I definitely agree that it is time to move past
| thinking in "languages" to a more holistic view of programming
| systems. The fact that as of now, only ~1% of people can access
| programming and wield the magic of the computer is a tragedy
| akin to the comparable levels of literacy in the medieval era.
| At the present, there seems to be a minor renaissance of
| thought along these lines stemming back to the pioneers of the
| field (McCarthy, Engelbart, Kay, etc). Everyone has a different
| take as to what the answer is, but I think "let 1000 flowers
| bloom" is the best way to find a solution to a complex and
| largely unknown problem such as software. Personally, I'm
| afraid that if we fail, the means of programming will become
| gated behind AIs and society will continue in its trance-like
| state of passivity instead of creative expression.
| wvenable wrote:
| > The fact that as of now, only ~1% of people can access
| programming and wield the magic of the computer is a tragedy
| akin to the comparable levels of literacy in the medieval
| era.
|
| I don't know why programmers think this way. We don't expect
| everyone to be a doctor, a botanist, a novelist, or a
| musician but for some reason we think anyone can be a
| programmer. That's just not the case -- programming is a
| skill like any other -- it takes some natural inclination,
| some training, and a lot of practice. Just like any other
| skill. A programming language is to a programmer what an
| instrument is to a musician.
| rileyphone wrote:
| No, but we expect everybody to know reading, writing, and
| arithmetic. Programming may be more intrinsically difficult
| than these, but most of the reason it's so hard today is
| because of solvable problems that beginners run into, to
| the point of half of expertise is just knowing to avoid all
| these pitfalls. The downside of the open source developer
| ecosystem today is that putting it all together is
| typically a painful process, which is what Replit is aiming
| to solve. There is a world of difference between a good IDE
| and a bad one for a beginner, and a lot of that comes down
| to integration with the language.
|
| Anyways, there's a reason programming should be made more
| widely accessible - it develops thinking skills and
| rationality when you learn it, as Mindstorms pointed out.
| Between the anecdotes in the book and things like [0], I
| think it's a travesty that we aren't pursuing this to its
| fullest extent, and instead try to teach children Python or
| JavaScript, two decent languages for software development
| but not exactly forgiving with beginners.
|
| [0] https://medium.com/@stevekrouse/goodbye-seymour-
| cb712757264f
| wvenable wrote:
| When I was a kid we also computer class were programming
| was taught. And music class. And drama. And woodworking.
|
| And like with these other things, you can make getting
| into them easier. With music, children start with simple
| instruments that no professional ever uses. And it's the
| same with programming, there are plenty of easier
| environments for children. But if you want to make
| programmers and you want to make musicians, eventually
| they have to use the real thing. My own son jumped
| straight into the deep end of Unity development knowing
| nothing because he wants to build something _real_. I
| neither encouraged or discouraged that environment and it
| 's pretty unforgiving.
|
| I don't think the solvable problems you speak of as are
| as solvable as you think they are. Also making software
| development out to be _special_ both in terms of it 's
| benefit to thinking skills and rationality and how it's
| merely some tools away from being professionally
| approachable to the masses is totally unfounded.
| notduncansmith wrote:
| We will have to contend with the last bit regardless.
| Programming requires attention to detail and tolerance for
| tedium, the erosion of which will only continue as AI grows.
| dahart wrote:
| A very worthy, laudable goal. In addition to some of the other
| good responses here, my reason for thinking it can't be solved
| with better tools alone is that, contrary to what the article
| suggests, engineering & tools alone are not the primary cause
| of our complexity, customers and consumers and management are.
| Business and the need to keep making money is what causes
| software to need to evolve and change, to support new features
| that weren't planned into the architecture, and to develop
| software in shorter timeframes than is necessary for clean
| solutions. This is what I've seen in a few decades of
| professional programming, that programmers are perfectly
| capable of solving hierarchical and cognitively complex
| problems. Customers or salespeople ask engineering to implement
| strange inconsistent features, people aren't happy with your
| software because your competitor does X, and we're always
| walking a balance between time and resources to meet those
| demands, and there's never enough of either. A lot of software
| business, for better or worse, is also driven by fads, so need
| complexity inducing maintenance just to not feel old and
| cruddy.
|
| The other reason complexity happens is dependencies and DRY
| thinking, which is often good, but dependencies, centralization
| of code, and sharing of code are also a risk. Avoiding
| repetition is what causes abstractions to grow. Using other
| people's libraries means that there are complex interactions
| you don't understand. Most of the time it's all fine, but
| occasionally sometimes it's not. This trade is made
| consciously, for the reason that it's _much_ faster to develop
| your app using existing libraries, and not reinvent all wheels,
| and nearly everyone is doing it. The reason that tooling is
| unlikely to solve this part of the problem is that failures are
| so often caused by incorrect expectations - someone using a
| library _hoping_ that it will do X when it only does Y. Even in
| theory, better tools could only tell us low-level information
| about what a library does, but I don't think it's possible for
| tools to tell you at a high level what a library can't do.
| gashmol wrote:
| I'm working on it too! Here is what I got so far (I don't claim
| any of these ideas as mine):
|
| Software development is the practice of producing descriptions.
| A requirement is a description of the application domain and
| the problems to be solved there. A program is a description of
| the system that solves that problem.
|
| There are many constraints that the descriptions must satisfy.
| The most important one is our limited cognitive capacity. We
| can't think of everything at once.
|
| The closer to each other the different descriptions are the
| easier it is to manipulate them. (This is just the principle of
| natural mapping from The Design of Everyday Things.)
|
| However, most real world descriptions are too complex to fit
| inside our working memory. We deal with this kind of complexity
| by a combination of decomposition (breaking up a complex
| description into a number of separate simple descriptions) and
| abstraction (ignoring some of the details of a complex
| description to make it simpler).
|
| The best example of decomposition is the original structured
| analysis and design where you'd derive the data-flow of the
| system in stages starting from the real world. Abstraction is
| best exemplified by OOP and FP.
|
| Good design methods encourage you to first write a description
| of the requirement. Then, derive or model the program after
| that description. The best example of such a method is data-
| structured design.
|
| To be able to assist you in solving a problem an effective
| method must make assumptions about the problem. It means that
| no one method will work for all the problems you encounter in
| real life. Thus, you must have several at your disposal and
| learn when to apply each one.
|
| I plan to use what I learned to tackle front end development
| because that's what pains me most in my day job.
| satyrnein wrote:
| I _think_ I understand what you 're getting at, but do you have
| any examples? Like you said, seeing helps understanding! :-)
| [deleted]
| meheleventyone wrote:
| Isn't this the dream since year dot? The problem is the real
| world is full of leaky abstractions, with different layers
| evolving at different rates and massive adoption inertia.
| pebcakID10T wrote:
| I have been developing software for over 20 years and leading
| teams of developers for the last 10. Please allow me to give
| you some insight from experience.
|
| To your first point, in my experience, the best software,
| including complex platforms that handle massive amounts of
| traffic and data, can be built and maintained by small teams. I
| mean less than 20 developers. The larger the number of
| developers working on an application or platform has an almost
| inverse correlation to the speed in which new features are
| built or bugs fixed.
|
| Software architecture has been done visually since perhaps its
| inception (tools like UML). At most places I've worked, every
| new project or large feature is diagrammed visually to use as a
| guide in breaking down the project into component parts.
|
| In my experience complexity arises, not from lack of tools or
| industry knowledge, but from 3 main causes:
|
| 1. Inexperienced developer asked to create project, who just
| starts building without planning beforehand.
|
| 2. The main one - business demands features built that were
| never expected or planned for, and built as fast as possible.
| This causes developers to take shortcuts, make inelegant and
| difficult to maintain design choices. And leads to often
| inscrutable code that becomes technical debt especially after
| the original developer leaves the company. This will always
| happen as long as software is used to make a business money.
|
| 3. High team turnover - I've seen places where developers came
| and went so often that there was a myriad of things half
| started and never finished.
|
| How I've solved or helped alleviate these issues: Make the
| business case to company owners or management that technical
| debt will be an ever increasing impediment to development
| velocity and the dev team will need a percentage of work in any
| given sprint to tackle tech debt issues (as opposed to having
| everybody work 100% on new features and bug fixes all the
| time).
|
| I have successfully taken a platform that was bug-ridden,
| difficult to maintain, and where new feature development had
| slowed to a crawl due to the over-complexity of the software,
| to a place of stability and ease of development, simply by
| allowing our team to chip away at tech debt over the course of
| several years. Tech debt issues were rated by level of
| complexity, risk to the business in change (regression bugs),
| and impact on team velocity. We worked on the highest impact,
| lowest risk items first and kept going until there wasn't much
| left on that list.
| eternalban wrote:
| > there are finite and fundamental classes of problems, that
| underpin the vast majority of problems
|
| We already have had, for a long time, a 'universal' model for a
| specific slice of information processing that consists of a
| very small set of components: Graphic User Interfaces.
|
| One problem is the diversity of domain semantics and all the
| associated nuances. In GUI programming, this is the tedious bit
| of naming the components and mapping them to processing
| elements.
|
| Another problem is engineering culture (du jour). A
| reductionist approach that attempts to leverage _structural_
| commonalities requires what is poo poo 'd as "boiler plate" and
| all the associated warts of component oriented programming
| (including factory-factories). A revisionist shift in mindset
| is required.
| kansface wrote:
| In addition to the comments below, complexity in software is
| just as often driven by politics/organizational structure.
| There is no solution in software to that problem.
| diamondap wrote:
| Yes. And that's the root of this problem:
|
| > Nobody can subtract from the system; everyone just adds.
| milkanic wrote:
| Have you looked at Clojure and REPL/data driven development?
| mathgladiator wrote:
| Welcome to the club!
|
| The fundamental problem of simplifying software is humans. Just
| consider date formats, time zones, and tax codes. Humans love
| to make complicated things.
|
| My philosophy on this has been to take the complex human stuff
| and stick it in a black box. A professional feather in cap with
| this approach is called BladeRunner (
| https://dl.acm.org/doi/10.1145/3477132.3483572 ) which
| radically simplified the distributed system aspect by putting
| all the gnarly glue and business logic in a V8 VM (JavaScript).
|
| My next thing is a focus on board games where I have invented a
| programming language and have started to evolve a platform.
| It's called Adama ( https://www.adama-platform.com/ ), and I
| think it is pretty cool. The interesting thing is that the
| complexity of board games is exceptional.
|
| I have a few clues to share. The first is that reactivity,
| which is found in excel, is a key to simplifying software as
| this makes the glue more automatic.
|
| Another clue is figuring out bidirectional communication which
| relates to reactivity as a two-way street. However, this is
| primarily hard because we don't have great things off the shelf
| to deal with this beyond TCP. For more of a deep dive, check
| out https://www.adama-platform.com/2021/12/22/woe.html which
| talks about WebSocket.
|
| My final clue is that you can't run away from state. So many
| people offload state because state is hard, and you have to
| contend with it. I'm building yet another database.
| nradov wrote:
| I am skeptical of visual programming in general. Some of us
| just aren't visual thinkers. I find it much easier to
| conceptualize complex concepts in terms of text and narrative.
| tasuki wrote:
| Don't worry - I _am_ a visual thinker and I 'm skeptical of
| visual programming. While it's cute for toy examples, I
| haven't seen it handle complexity well.
| throw827474737 wrote:
| How many clever computer scientist (including me ) had this
| idea and vision at least once? And how many attempts by
| languages/DSLs/visual languages have already been tried to
| create?
|
| I believe you are wrong, I'd wish you are right ;)
| jeffreygoesto wrote:
| Hey. Come to the automotive industry, where your code will live
| for whooping twenty years! =;-D
| parsley27 wrote:
| I like to believe that serverless technologies and cloud services
| reduce complexity for the org, but obviously that's at the cost
| of offloading that complexity to the cloud providers (and welding
| an org's software to that provider for years, if not decades).
|
| So different, but not less complex overall. But maybe there is
| value in having some of that complexity consistent across a small
| number of cloud providers.
| mark242 wrote:
| They absolutely do reduce complexity! For example, think about
| building, testing, and deployments.
|
| In pre-cloud Internet times, you'd have an untold number of
| extremely brittle bash scripts, cron jobs, rsync ssh key setup,
| fleets of build + test boxes to manually worry about disk
| space, pre-provisioned dev/QA database servers with also untold
| brittle sql startup/teardown scripts, and all of the requisite
| people whose job it was to solely maintain this infrastructure
| along with database tuning, build fleet monitoring, the list of
| menial tasks just goes on and on and on.
|
| Today, you have a yaml file in your .github/workflows
| directory.
|
| Now I agree that there are "different" requirements.
| Understanding the complexity of your workflows etc is no small
| feat-- but you're replacing such a huge amount of what used to
| be extremely expensive and brittle architecture with,
| basically, a text file or two. That's a huge cost savings.
| mark242 wrote:
| Another quick example - serving up product image assets to
| customers. Let's say you want to ship 1TB of images to
| customers per month.
|
| In pre-cloud times, you would spend untold piles of money
| spinning up racks of storage arrays, switches, firewalls,
| leasing out gigantic pipes for bandwidth, and again all of
| the requisite people in order to get that running. You'd rent
| space in multiple distributed global datacenters, so again
| you'd have an unreal amount of bash-script-file-sync services
| so that when someone uploads a new image it gets replicated
| all over the globe. Millions and millions of dollars. You'd
| probably have to write a custom resizing service with
| ImageMagick, hooked into your frontend, so that you were
| serving customers the correct size and not blowing through
| your bandwidth allocation. Just incredibly complex.
|
| Today, you click a button in your CDN provider's console;
| most of that functionality from above just comes for free.
|
| Again you have to do a little bit of munging your front-end
| to take into consideration the vagaries of your CDN provider,
| but overall it's such a huge savings of mental energy and
| time. Put a circa-1999 systems architect in front of the
| Cloudflare console from today and they wouldn't believe it
| was real.
| beardedetim wrote:
| Are you counting Chef/Ansible/Puppet/etc as things that came
| during the cloud era or that those things were not used to
| solve the problems you raised before the cloud era?
| mark242 wrote:
| Having used Puppet, the promise of Puppet was very
| different than the realities of Puppet and imo more
| accurately reflects the "different complexities" point. The
| point of those provision-bare-metal-on-the-fly frameworks
| just evaporated when, for example, you could launch a
| container on Fargate with, again, a single text file, or
| similar on Heroku.
| chasd00 wrote:
| idk, in the enterprise world all those things still exist but
| they're applied against AWS/other instead of your servers in
| the datacenter. Instead of "build+test boxes" you have
| "build+test instances". It takes the same level of
| redtape/approvals to get an AWS instance as it took to get a
| server in the old datacenter. All my enterprise clients have
| the dedicated infrastructure teams they always had, only now
| they're working in AWS/other and not the datacenter.
| mark242 wrote:
| I am going to go out on a limb and guess that this is
| almost entirely down to your finance department attempting
| to understand and control the monthly cloud spend. This is
| obviously fine, but rather than "oh, another 2xlarge, we
| have to add a couple hundred bucks to our budget" the
| reverse should be true; you have $x to spend per month on
| average with a reasonable growth plan built in.
|
| In the build and test example, the answer to "how much
| compute is running?" is based off developer velocity and so
| "it depends" is a fair answer. When nobody is shipping any
| builds, your cost should be $0. (I've found that this is
| kind of hard for enterprise-y finance departments to wrap
| their heads around and is why all those esoteric billing
| notifications AWS services even exist)
|
| Pre-cloud services days, finance departments had a much
| easier time. You had racks of physical boxes that had
| static costs attached to them, you had a static monthly
| bandwidth bill that let you run at a certain speed, and you
| had salary costs which are also pretty static month-to-
| month. The idea of "scale to 0" was completely unheard of.
| What do you mean your QA environment doesn't cost anything
| on the weekends when nobody's doing anything? etc etc etc,
| you get my point.
| hinkley wrote:
| Every time our operating costs go up (eg, cloud providers),
| something else in the organization gets more complicated in
| order to make up the shortfall. Or super simple when we go out
| of business.
|
| No free lunches here.
| parsley27 wrote:
| That assumes operating costs go up with cloud providers. Some
| orgs see savings, but obviously not all.
| hinkley wrote:
| Some developers are merely bad at math. Most are awful.
| Others take this flaw to epic proportions. Be afraid of the
| developer who confidently tells you the math works out.
|
| It's true that a lot of organizations had operational teams
| that have become money pits, or pushed back on all quality
| of life improvements because they don't have the talent,
| the budget, or the imagination to pull it off. When we move
| to the cloud we start aspiring to these things we didn't
| have _because they were expensive_. I don 't know of anyone
| who moved to the cloud and didn't move the goalposts. We
| were just talking in another thread, as we often do here,
| about how much YAGNI is going on out there. Yes, the price
| per feature goes down, but the overall price doesn't seem
| to. And I get stuck taking care of things someone else used
| to worry about, which is opportunity cost on top of it all.
|
| What is also true is that developers can learn a lot from
| their operational peers and avoid expensive mistakes. With
| cloud we have none of those peers. We have to learn
| everything first hand. By someone who is all too happy to
| let us wrap rope around our necks and then 'rescue' us from
| themselves. That's a perverse incentive and quite a setup
| for a fairly fucked up codependent relationship. At least
| with interdepartmental drama some of the money stays in the
| company longer before going to vendors.
| datavirtue wrote:
| We used to have devs bringing in frameworks and libraries
| without reading the manual. Now they spin up entire
| services or subsystems without reading the manual. You
| should read through the docs for MongoDB Atlas...which
| plainly illustrate that you now need a DBA unless you're
| cool with people pressing buttons in a panic during some
| type of self-induced performance issue.
| tomrod wrote:
| Having moved services from on prem servers to EC2 to OCP to
| EKS, I don't think these decrease complexity.
| eranation wrote:
| > Simplicity of design sounds great in theory, but it might not
| win you many plaudits from your peers. A complex design means
| more teams to manage more parts of the system, more for the
| engineers to do, more meetings and planning sessions, maybe some
| more patents to file. A simple design might make it seem like
| you're not really doing your job. "That's it? We're done? We can
| clock out?" And when promotion season comes around, it might be
| easier to make a case for yourself with a dazzling new design
| than a boring, well-understood solution
|
| I couldn't disagree more. The vast majority of companies I worked
| with recognized and rewarded simpler designs, the vast majority
| of software engineers I worked with would rather delete code than
| write code, rather throw away a complex design to a simpler one,
| and with some NIH here and there, YAGNI and KISS seems to be
| triumphant and what would get you promoted, not adding more
| arrows and boxes.
| a1369209993 wrote:
| > The vast majority of companies I worked with recognized and
| rewarded simpler designs, the vast majority of software
| engineers I worked with would rather delete code than write
| code
|
| What made up fantasy world do you live in, and are they
| accepting immigrants?
| mike00632 wrote:
| Haha, this makes me want to work at your company.
| groby_b wrote:
| Our industry's "solution" to complexity is somewhat unique: We
| sweep it under the carpet. The reason that actually works is that
| the marginal cost to reproduce an existing pile of complexity is
| zero.
|
| Once we've got a sufficiently complex pile that actually has
| promise, everybody duplicates the pile. And then puts an
| additional layer on top. These many pile+layer combos then co-
| evolve (and co-erode ;)
|
| At some point, something so useful has evolved that we start
| copying that pile... and put new layers on top. Rinse. Repeat.
|
| The complexity rarely collapses because those new layers can add
| support beams, if necessary. Worst case, a layer gets ablated,
| and we try again from one step below. We don't ever go back to
| the beginning.
|
| (This also explains, to me, why Enterprise software is what it is
| - the complexity piles can rarely be shared or reused, and so
| there's no co-evolution, little chance to settle on the best
| pile)
|
| And, to be fair, it's similar in other disciplines, but maybe a
| bit slower. Mechanical engineering has individual components,
| larger components built from those, and so on. It just is more
| expensive to copy the piles^W components we know to work.
|
| Total collapse is exceedingly rare.
|
| (That even holds for human societies - they, too, often "just"
| collapse down to simpler societal expressions. And we do co-
| evolve larger and larger forms of society. It's just that the
| erode-and-try-again process is uniquely painful for the humans
| involved)
| nonrandomstring wrote:
| "Don't worry. We'll fix it when it goes bad" -- Dolittle (Dark
| Star)
| jv22222 wrote:
| > It takes a lot of discipline to resist complexity, to say "no"
| to new boxes and arrows.
|
| If we refactor a very large function into smaller sub functions
| we are essentially adding new boxes... but are we creating
| complexity, or simplicity?
| protomyth wrote:
| I always thought it was because we really couldn't break the
| applications down without massive dependencies and the complexity
| of human procedures brought to the computer's logical world.
|
| It seems like folks who've built some form of common bus (shared
| data types and communications channels) do better scaling
| applications. If I can effectively split off a part of the
| application and plug a new part in its place, I can scale. OOP
| seems to love dependencies that accumulate until objects cannot
| be replaced because of all the connections. Clean separation is a
| hard problem.
|
| Computers are logical machines. Human laws and regulation are
| not. Modeling physics is an easier problem (programming not
| computing need wise) than calculating taxes. At one point in the
| US, every gallon of bio-diesel had to have its own serial
| number[1], this adds a level of complexity that goes beyond
| simple programs.
|
| My own thoughts (or my longtime hobby project) is that Agent
| oriented programming hasn't really been given a workout, and that
| decoupling data structures might lead to some improvements. I
| honestly think being able to replace parts of a running program
| in an orderly manner might be a indicator of a good direction.
|
| 1) https://mnbiofuels.org/media-mba/blog/item/1495-rins-101 - as
| a programming exercise, write a program to add, subtract gallons
| from a storage unit - it gets fun.
| danielovichdk wrote:
| I will tell you this.
|
| It's based on my 20+ years, 50 different teams and many hundreds,
| perhaps thousands of encounters with software developers.
|
| It's almost never the domain complexity that is high.
|
| Sending a rocket to Mars is complex. And perhaps complex every
| time.
|
| Dealing with people that has to work together can only be
| simplified when agreements are made and agreements are kept.
| Otherwise you cannot trust and have faith in people.
|
| It comes down to far and near organization leadership, mandate
| and empowerment in small teams with few communication links
| outwards, clear expectations from whomever is asking for
| something and process management.
|
| Writing code is rarely hard if you know the expectations and the
| boundaries of the task.
|
| Empowerment gives the team the possibility to act on what they
| discover and encounter.
|
| Don't look into the future. Because you cannot. And you are both
| naive and arrogant if you believe you can. Deal with what is
| known now!
|
| One thing is complexity in code. Another is complexity in
| business. Some things are not made for software to solve.
|
| Less experienced people are generally guided by their beliefs
| around their own ideas. If the mind is locked onto something it's
| hard not to thing it's a good idea. Usually in software there is
| simpler approach. Look for it
|
| I can go on because I help companies and teams for a living. But
| I will stop here
| _gabe_ wrote:
| > Don't look into the future. Because you cannot. And you are
| both naive and arrogant if you believe you can. Deal with what
| is known now!
|
| Yes! 100% this. We need less fortune tellers and more
| pragmatists.
| 0xFACEFEED wrote:
| > Don't look into the future. Because you cannot. And you are
| both naive and arrogant if you believe you can. Deal with what
| is known now!
|
| NO!
|
| This mentality is short sighted by definition and leads to
| stagnation. It may work within large financially healthy
| companies that aren't trying to rock to the boat.
|
| Making assumptions about the future is incredibly important for
| us to move forward. But with these caveats:
|
| 1) Acknowledge that you are taking a risk and that you may be
| wrong.
|
| 2) Validate your assumptions as quickly and as often as you
| can.
|
| 3) Be prepared to change course whenever your assumptions are
| invalidated.
|
| > Less experienced people are generally guided by their beliefs
| around their own ideas. If the mind is locked onto something
| it's hard not to thing it's a good idea. Usually in software
| there is simpler approach.
|
| Less experienced people are frequently the driving force behind
| new ideas precisely _because_ of their belief in it. As
| experienced people we should be encouraging them to carry
| forward responsibly instead of projecting our aged and jaded
| perspectives.
|
| > I can go on because I help companies and teams for a living.
|
| Honestly, that's a weird way to phrase what you do.
| wtetzner wrote:
| I think you and the parent are talking about two different
| things. Building products that are looking to the future is
| valuable, if risky.
|
| But I believe the parent comment was talking about software
| abstractions. i.e., don't build abstractions because you
| _might_ need them later. Instead, build the software in the
| simplest possible way to solve the problems it needs to
| handle now.
|
| This is because you'll likely be wrong about your assumptions
| about which abstractions you'll need, and ripping those
| abstractions out is going to be more expensive than just
| modifying simple code to do new things.
| jrochkind1 wrote:
| I appreciated this essay.
|
| Among other things, it suggests that there are some incentives
| for complexity: That engineers just _enjoy_ complexity, and that
| making simple designs might not be good for the respect of your
| peers /bosses/career: "A simple design might make it seem like
| you're not really doing your job."
|
| While that's true, I think the essay isn't clear enough about the
| fact that _simplicity is just plain hard_. Complexity is sort of
| the "natural" outcome of "adding things". It takes time, effort,
| and _skill_ to create software that can do what is asked for it
| while maintaining an internal simplicity. Even _if_ everyone
| involved _really wants_ it. I do not think software complexity
| comes only, or mainlly, from people _not wanting it enough_
| (explicitly or implicity), from incentives to complexity. It 's
| just plain hard. It's a mistake to think just _wanting it harder_
| (or having the right incentives) is sufficient.
|
| It may be the key challenge in software engineering -- how do we
| make software that does everything that's asked for it (powerful,
| complex things), while maintaining an internal simplicity? You
| can not accomplish it by cargo-culting any "design patterns" or
| other approaches. It's still a skilled craft. That comes from
| years of experience with the _intent_ of developing this skill
| (years of experience alone aren 't enough), looking at other
| people's code who are also trying to do it, and just some
| aptitude. And _domain knowledge_ , if you understand the nature
| of the problems you are working on better, you can design simple
| solutions for them better--contrary to some current practices
| treating developers as interchangeable.
|
| Yes, you need your organization to _allow_ you to do it
| (including taking the _more time_ it _will_ take), but that 's
| necessary but not sufficient for the outcome.
| mananaysiempre wrote:
| There is the well-worn Pascal quote about having no time to
| write a shorter letter[1]. There is also, I think, an important
| point hidden in the "domain knowledge" part: the freedom to
| _change the spec_ ; the effusive texts on Charles Moore's
| approach to Forth programming[2,3], however much scepticism
| they deserve otherwise, are very direct about that freedom
| being essential.
|
| But that means simplicity is a cross-cutting concern, like
| security, and does not particularly fit into any environment
| that would attempt to separate design and implementation. It
| needs "redraw all the module boundaries" to be a plausible (if
| improbable) response to "this one 300-line part of this one
| module is awkward and three times longer than it really needs
| to be". Aside from the obvious administrative problems,
| suggests that the complete edifice needs to fit into the mind
| of a couple of people at most or the iteration simply will not
| converge most of the time.
|
| I don't see a good solution to this.
|
| [1] https://quoteinvestigator.com/2012/04/28/shorter-letter/
|
| [2] http://www.ultratechnology.com/forth.htm
|
| [3] There was an example about changing a negative-feedback
| control loop, written into the spec by hardware engineers
| emulating analog circuitry, into a couple of fixed-point
| operations that behaved _differently_ but did the actual job
| just as well; I can't find the reference right now but this a
| _minor_ domain-specific change among the kind I have in mind.
| psychoslave wrote:
| >A software company with more modest aims, that has a stable
| customer base and doesn't change much over time (does such a
| thing exist?)
|
| Here is the choice, stagnate and disappear through incapacity or
| adapt and perish in burnout.
| henning wrote:
| > engineers like complexity. Admit it: as much as we complain
| about other people's complexity, we love our own
|
| No. Speak for yourself. I am happiest when my solution to a
| problem is simplest. I am least happy when a solution requires
| introducing new infrastructure or major new ideas.
| a_e_k wrote:
| I came here to say this.
|
| I have no idea who first said it, but one of my favorite
| sayings is: "Any idiot can build a bridge that stands, but it
| takes an engineer to build a bridge that barely stands."
|
| I strongly believe that this applies to computers as well. I
| love it when a system neatly solves its requirements in a way
| that looks effortless. When an algorithm looks like it just
| barely works for all the potential edge cases without any
| special handling because those edge cases just miraculously
| happen to work out as they fall through the happy path just
| like any other case. That's when a solution "feels" right.
|
| If I can present my solution/code/explanation to a junior
| engineer, a C.S. student, or even a non-technical person and
| they get it, then that's when I feel like I've really done my
| job. It's music to my ears if I can make a problem look so easy
| that someone asks "Is that it all it takes?" (until they've
| tried to solve it on their own :-)
| commandlinefan wrote:
| I feel the same way, but we both have to accept that we're
| tilting at windmills. Complexity can't be measured, so there's
| no way to demonstrate, much less prove, that the time you spent
| reducing complexity was time well spent. You can demonstrate a
| run time or memory performance improvement. You can demonstrate
| a new feature. Unless you can reduce complexity in zero time
| _and_ do so without introducing any unintentional side effects
| /bugs, you're just hurting yourself when you try.
| BenoitEssiambre wrote:
| In some sense this is the eternal question of life, the universe
| and everything, life being complex self reproducing software
| itself which adapts to environments and sometimes collapses when
| it changes. It's a fascinating subject, especially from the
| perspective of trying to push towards the limits of information
| and entropy. I'm not sure the root of the problem is solvable
| however.
| chrchang523 wrote:
| Note that Google's "Frequent rewrites" practice -- see section
| 2.11 of https://arxiv.org/pdf/1702.01715.pdf -- substantially
| reduces the severity of this problem (though it requires a good
| higher-level architecture).
| undershirt wrote:
| > 2.11. Frequent rewrites
|
| > Most software at Google gets rewritten every few years.
|
| > This may seem incredibly costly. Indeed, it does consume a
| large fraction of Google's resources. However, it also has some
| crucial benefits that are key to Google's agility and long-term
| success. In a period of a few years, it is typical for the
| requirements for a product to change significantly, as the
| software environment and other technology around it change, and
| as changes in technology or in the marketplace affect user
| needs, desires, and expectations. Software that is a few years
| old was designed around an older set of requirements and is
| typically not designed in a way that is optimal for current
| requirements. Furthermore, it has typically accumulated a lot
| of complexity. Rewriting code cuts away all the unnecessary
| accumulated complexity that was addressing requirements which
| are no longer so important. In addition, rewriting code is a
| way of transferring knowledge and a sense of ownership to newer
| team members. This sense of ownership is crucial for
| productivity: engineers naturally put more effort into
| developing features and fixing problems in code that they feel
| is "theirs". Frequent rewrites also encourage mobility of
| engineers between different projects which helps to encourage
| cross-pollination of ideas. Frequent rewrites also help to
| ensure that code is written using modern technology and
| methodology.
| c-smile wrote:
| Nothing new in principle, "transition from quantity to quality"
| (quantitative differences beyond a certain point pass into
| qualitative changes) was discovered by Hegel around 1800.
|
| So nice, working application after adding next 5,000 more lines
| here and there suddenly becomes too fragile to perform its main
| function.
|
| Usual solution for such problems is composition/modularization.
| Software shall be made of simple but composable modules.
|
| Like text editor (relatively simple) shall be composable with
| syntax analyzers to make IDE possible. The very same syntax
| analyzers should be usable as web server modules for the Web.
|
| That dream requires design of (again) simple but powerful
| architectures of composition/modularization.
|
| Seems like each generation of software philosophers approaching
| this task anew. Linux with its stdin/stdout modularization. MS
| Windows with OLE. COM and CORBA, and so on.
|
| Probably it's time to evolve something like COM and CORBA into
| something new?
| bitwize wrote:
| > One thing working in complexity's favor, though, is that
| engineers like complexity. Admit it: as much as we complain about
| other people's complexity, we love our own. We love sitting
| around and dreaming up new architectural diagrams that can
| comfortably sit inside our own heads
|
| Where does this yutz work? Because he doesn't speak for me.
| Generally, I write a complex program because I don't have time to
| write a simpler one; and when I have to deal with that complexity
| later I'm like _arrrggh_. Rube Goldberg machines are fun to
| watch, but no one wants to maintain, change, or build on one.
| chasd00 wrote:
| didn't Mark Twain say something like "i'm sorry this letter is
| so long, i didn't have time to write a shorter one". I feel
| like that with some of my code at times.
| bitwize wrote:
| I believe it was Blaise Pascal, to whose missive I was making
| a deliberate reference.
| tbwriting wrote:
| For those curious about what this looks like in the trenches:
| https://tylerberbert.substack.com/p/complexity
| bell-cot wrote:
| Back on the "Collapse of Complex _Societies_ " side, my
| impression is that a fair number of such collapses are now blamed
| on climate change, and/or resource depletion. The Anasazi thrived
| from ~700 to ~1130 due to an unusually rainy spell in SW North
| America. When the period of heavier rains ended, so did the
| agricultural productivity that supported their large population
| and complex society. Similar reasons may hold for the fall of the
| Khmer Empire ( https://en.wikipedia.org/wiki/Khmer_Empire ). A
| number of sharp, grim setbacks for European civilizations occur
| suspiciously close to major volcanic eruptions, which could have
| hit their agricultural foundations hard.
|
| ...and "yes", I read second ~half of the article. My point here
| is that the author's analogy works at least as well where he did
| _not_ bother to apply it as where he did.
| skadamat wrote:
| This was my favorite deep dive post on this exact topic from a
| few years back in the Atlantic:
|
| https://www.theatlantic.com/technology/archive/2017/09/savin...
| pugworthy wrote:
| The 911 overflow is an interesting bug. Kind of asks for a
| philosophy of, "Limit your build, but don't build in limits"
|
| Once I was on a research ship, and the data acquisition system
| crashed on the 256th day of the year. Care to guess the issue?
| :)
| throwaway23234 wrote:
| This is the truth: We get usefulness and simplicity from
| complexity. The more useful AND simple you need something to be,
| the more complex it actually is.
| waynesonfire wrote:
| one word... path of least resistance. this is why we can't have
| nice things. it takes WORK to reduce complexity.
| krm01 wrote:
| For the last 12 years, I've worked with 40+ different software
| teams (helping with UI/UX design [1]). From small scrappy
| startups to fortune 500.
|
| The teams that were great to work with and ended up having the
| biggest wins, were always small teams with a great cultural
| balance. They always strived for the simplest solutions.
|
| You could trace this balance of engineers in all of the cases
| back to GREAT hiring by the CEO.
|
| The teams that were the least productive and just burnt through
| VC money were the ones that had a handful of toxic engineers at
| their core. Finding complex and seemingly clever (over
| engineered) 'solutions' seemed to be their goal in every meeting.
|
| A way to find out if your team belongs to the second group,
| measure the amount of jargon that's being used in meetings.
|
| [1] www.fairpixels.pro
| elisharobinson wrote:
| it is a knapsack problem for sure , minimize ( tooling cost and
| complexity ) and maximize ( feature complexity aka "richness" or
| "usefulness" )
| js8 wrote:
| I feel like it's a bit backwards. The software complexity
| increases because people lack the patience to tackle the existing
| complexity, or learn why things are complex, and prefer to
| believe that it can be simple, or get job done somehow, and in
| order to do that, they (somewhat egoistically) go their own way,
| which actually - barring occasional good insight - mostly
| increases the total complexity.
|
| The same goes for societies. If people have more autonomy (like,
| not have external pressure to be productive), they might be more
| willing to contribute to the society as a whole, to fix it, and
| also take a more long-term view of the future. IME, in
| established open source software, there is less churn and
| accidental complexity.
|
| Complexity is also often the result of in-fighting, which again,
| is the symptom of a society where the trust is low.
|
| Though I am not sure, honestly, if there is a reliable way to
| increase the trust inside society rather than the actual
| collapse. That paradoxically often leads to increased
| cooperation.
| atoav wrote:
| I believe many software developers often tend to write software
| which has precisely the segree of complexity they can just
| barely manage. There is not many people who have the guts to
| build a simple straightforward step for step script. Software
| developers that fall into love with the beauty of their own
| abstractions can be a real problem.
| aylmao wrote:
| +1, plus I'll add that I think there's organizational factors
| too. Overall, organizations incentivize complexity, and I'd
| like to point out three ways in which they do:
|
| - One is that new systems tend to be rewarded, because people
| and organizations tend to associate "new" with "better".
| Related, people and organizations tend to associate high
| complexity systems with "hard work", and the transactional
| view of work tends to err on the side of rewarding "hard
| work".
|
| - Next is that complexity digs a moat of job security. I
| don't think this is a bad thing, job security is good. Again,
| unfortunately the transactional view of work can easily
| devolve into "we can pay someone less to do what you do" and
| building a complex system is a way to counter that.
|
| - Lastly, a story. At some point I worked on a team that
| quadrupled in size to tackle huge project, only to then fall
| into a sort of "identity crisis" because the project was done
| and there wasn't enough meaningful work afterwards for
| everyone.
|
| This insane growth, in my opinion, was also fueled by egos.
| The overarching team manager seemed pretty adamant in finding
| more people to hire, but I don't necessarily remember us
| feeling understaffed. They clearly expressed they wanted to
| use all the headcount the company had authorized, and the
| primary reason they always mentioned was that we'd "lose it
| if we didn't use it by the end of the year".
|
| This led to an "induced demand"-like situation. By the end of
| it, we had a huge team, the bigger meta-project was done, and
| everyone needed to find the next thing to work on and build.
| I saw plenty of proposals for solutions looking for a
| problem, and worked on some myself too before leaving.
| wtetzner wrote:
| > There is not many people who have the guts to build a
| simple straightforward step for step script
|
| I don't think it's that easy. Building simple software is
| very hard, and often takes several iterations to ensure you
| even understand the problem. Most developers are under time
| pressure, and stop as soon as there's a working solution,
| even if it's more complex than it should be.
| commandlinefan wrote:
| > precisely the segree of complexity they can just barely
| manage
|
| "Everyone knows that debugging is twice as hard as writing a
| program in the first place. So if you're as clever as you can
| be when you write it, how will you ever debug it?" -- Brian
| Kernighan
| commandlinefan wrote:
| > they (somewhat egoistically) go their own way,
|
| That's not necessarily all ego talking. If you have a task to
| accomplish and you're going to need to spend a week looking at
| it just to get a handle on how to _start_ getting it done the
| simplest overall way possible, your boss is just going to
| insist you do it the "quick and dirty" way "this one time"
| (every time). As Dilbert says, "our boss can't judge the
| quality of our work, but he knows when it's late". This is an
| inviolate law of software development: quick always beats good.
| Get on board, or you'll be replaced by somebody who is.
| agentultra wrote:
| _A legacy system exists: it's big, it's complex, and no one fully
| understands how it works. Architects are brought in to "fix" the
| system. They might wheel out a big whiteboard showing a lot of
| boxes and arrows pointing at other boxes, and inevitably, their
| solution is... to add more boxes and arrows. Nobody can subtract
| from the system; everyone just adds._
|
| In my experience this is the only _safe_ thing to do when you don
| 't fully understand the system and cannot _reason_ about it in a
| precise way. You create well defined abstractions over the messy
| parts, verify the abstractions, and write lots of tests.
| Eventually you can start deleting code. Although nobody ever ends
| up deleting code because they skip the "well defined" part and
| end up adding more indirection, afraid to delete code.
|
| The only real way to tame complexity is to reason about it with
| precise, mathematically-defined abstractions. You can't invent
| Paxos using boxes and arrows on a white-board and prove the
| minimum quorum required without showing your work. Fortunately we
| have languages and systems that can automate a lot of the effort
| in checking our work as long as we can express it in formal
| language.
|
| The other problem with this is that learning how to use these
| tools and techniques requires a lot of effort that isn't highly
| valued or rewarded in industry. Most businesses are fine with the
| trade off of errors and mistakes down the line if they're first
| to market. Ship it now, fix it later. We do a reasonably good job
| of preventing enough obvious errors with our tooling and
| processes. We can have _software that obviously has no errors_ or
| _software that has no obvious errors_. Time and again we chose
| the latter because the former is too difficult and doesn 't
| reward us with money _now_.
| commandlinefan wrote:
| > the only safe thing to do when you don't fully understand the
| system and cannot reason about it
|
| Well, the other option (the most sane one, in fact) would be to
| not mess with it _until_ you understand it and can reason about
| it - but in every corporate "how long is that going to
| take"/"what can we do to speed this up"/"why aren't you
| finished yet" quick-and-dirty-ocracy, that's not an option
| unless you can do that in about a day.
| tasuki wrote:
| I like that the author is working for Salesforce <3
| alexfromapex wrote:
| I have been thinking about this exact issue, with regards to the
| way clinical research is conducted.
| eldude wrote:
| Complexity in software is a lot like complexity in cuisine; most
| chefs want to stretch their abilities and create something
| memorable, transcendent even, but at the end of the day unless
| they're cooking for themselves or friends and family there's a
| business to be run.
|
| So complexity has a diminishing return for economical reasons.
| chasd00 wrote:
| i kind of like that analogy. And many times, a ham sandwich and
| a beer really hits the spot.
| Willish42 wrote:
| > A software company with more modest aims, that has a stable
| customer base and doesn't change much over time (does such a
| thing exist?) will be more like the humble tribe that follows the
| yearly migration of the antelope and focuses on sustainable,
| tried-and-true techniques. _(Whether such companies will end up
| like the hapless Gauls, overrun by Caesar and his armies, is
| another question.)_
|
| That last part is the key: with the strong preference for short
| term gains in an economic system where finances are considered
| and reconsidered quarterly, sustainability is an afterthought.
| "Growth at all costs" aligns with the dominant ideology of
| imperialism, and it's just as prevalent in corporations and
| organizations as it is in nation states. Literal millennia of
| evidence for that approach out-competing others is hard to beat,
| and I think if we knew how to survive a different way even at the
| micro level we'd probably live in a much better society.
| ozten wrote:
| Reminds me of the simplicity potential STEPS Toward the
| Reinvention of Programming (Oct 2012 Alan Kay and folks)
| https://news.ycombinator.com/item?id=11686325
| DenisM wrote:
| Business assets depreciate over time, nearly all of them, it's
| only the amalgam that can steadily appreciate, as assets are
| cycled in and out. Business is fine with that.
|
| The problem with software is that software deprecation cannot be
| readily modeled in Excel. We're back to the problem of
| quantifying Engineering Debt.
|
| The best I can come up with number of WTF moments per unit of
| cyclomatic complexity. The second best is aggregate cortisol
| level of developers divided by the business value produced during
| a release.
| lbriner wrote:
| An interesting problem is whether the complexity is unavoidable
| and if it is, we need to teach migration and proxying in Software
| Engineering 101.
|
| We have an old app: dotnet webforms, it's clearly hanging on for
| its life and needs replacing but the mythical rewrite is
| impossible if you intend to rebuild and switch, it just involves
| too much time away from your core product.
|
| Instead, we had to setup a proxy which could send all traffic
| between one of the two apps so we can move things a few at a time
| to the new system and all new work just gets built in the new
| system. It still takes a long time so I am now wondering whether
| as soon as we finish, we should immediately setup another new app
| alongside the current new one and repeat indefinitely.
|
| It certainly wasn't easy to do because old and new frameworks are
| rarely able to share cookie formats/sessions etc so some hand-
| cranking was required.
| 0xdeadbeefbabe wrote:
| > It takes a lot of discipline to resist complexity, to say "no"
| to new boxes and arrows. To say, "No, we won't solve that
| problem, because that will just introduce 10 new problems that we
| haven't imagined yet." Or to say, "Let's go with a much simpler
| design, even if it seems amateurish, because at least we can
| understand it." Or to just say, "Let's do less instead of more."
|
| Discipline and complexity are best friends. Laziness and
| procrastinating seem more valuable for avoiding complexity.
| jldugger wrote:
| > One thing working in complexity's favor, though, is that
| engineers like complexity.
|
| Honestly, I hate complex solutions. Even my own. What I like, and
| what impresses me, is the simple solution that takes a lot of
| work or math to prove it works. Like, emergent behavior type
| shit.
|
| Control theory, for example, runs systems with very simple
| runtime behavior. A few open loops with some parameters. The
| analysis to chose those parameters is where the engineering
| happens.
|
| Microservice practices are not yet that simple, but some day I
| hope they can be.
| an9n wrote:
| I seem to be in drastic opposition in that I _hate_ complexity,
| and will do anything to avoid it. I shudder with horror when I
| look at the teetering stacks all over the place these days.
| rektide wrote:
| It's hard to describe just how much of software engineering &
| it's enjoyability comes from attaining perspective, from reaching
| a point where you see & understand how a big pile of stuff works.
|
| > _One thing working in complexity's favor, though, is that
| engineers like complexity. Admit it: as much as we complain about
| other people's complexity, we love our own. We love sitting
| around and dreaming up new architectural diagrams that can
| comfortably sit inside our own heads_
|
| It's harder to acquire a sense of mastery & possession, of real
| understanding, when there's been multiple iterations of teams
| hacking on a piece of software, when it's many layered & has lost
| cohesion & concerted intentionality.
|
| Trying to describe how relishable, how enjoyable it can be to
| explore, to search & quest for meaning, to push designs that
| hopefully make sense, that hopefully grow, that hopefully
| wrangle: it's an under-sung & hard to tell story. It's
| interesting to me having such a big part of the world running off
| of a knowledge-working that is so under-described, so hard to
| communicate personally about.
| fbn79 wrote:
| Probably many huge codebases could be reduced to a handfull of
| good old SQL queries and nothing more... If you trash things
| like: framework boilerplate, complex graphical UI libs, devops
| complications, auto-testing ecc....
| [deleted]
| imbnwa wrote:
| The word 'simple' is hella overloaded in software circles
| p0nce wrote:
| I think Software education is in its infancy, instead of teaching
| people how to make debt decisions to keep complexity and costs
| low, we actively teach them to write disposable code. That is the
| content of most courses: code that will be thrown away, and that
| has to be written under tight time constraints.
|
| And when it comes to keeping complexity low, there is zero theory
| to back up anything, so it's not taught.
|
| Our trade is somewhat similar to reading/writing, and if you were
| to teach someone to read/write, you would probably insist on good
| writing, not just "finish your novel at this date, I won't read
| it anyway". And you would have good material for teaching to
| read, material that is currently missing.
| igammarays wrote:
| I'm waiting for the day that AWS collapses under its own weight.
| Heard horror stories from backend engineers working on the
| infrastructure/dashboard code there.
| danuker wrote:
| Judging by the price, it already has.
| WaxProlix wrote:
| If anything were going to do it, it'd be the labor crunch that
| we're seeing here. Without a constant influx of fresh-faced
| college coal lumps to burn in the tech debt engines Amazon
| would have a harder time of it. They've done a good job of
| pivoting though, and I think they'll weather the storm here
| (and be better off for it). See the increased comp to boost
| hiring rates - especially of non college hires - and an
| internal acknowledgement that addressing meaningful debt or
| internal pain is also a potential path to promotion.
|
| So, it'd be funny to see, but I don't think it's happening
| anytime soon.
| chrisweekly wrote:
| > "constant influx of fresh-faced college coal lumps to burn
| in the tech debt engines"
|
| made me literally LOL. That turn of phrase is gold.
| strix_varius wrote:
| Or even diamond, with enough pressure!
| stjohnswarts wrote:
| They're going to need diamond hands to hodl it together.
| greatpostman wrote:
| I think there's basically a zero percent chance of this
| happening. Aws may internally be full of technical debt, but
| for ten years now Amazon engineers have been automating the
| reliability issues of the platform. The capabilities are
| expanding, but the core components are some of the most rock
| solid engineering on the planet.
| angarg12 wrote:
| Preventing the Collapse of Civilization / Jonathan Blow
|
| [1] https://www.youtube.com/watch?v=ZSRHeXYDLko
___________________________________________________________________
(page generated 2022-06-09 23:00 UTC)