[HN Gopher] One Way Smart Developers Make Bad Strategic Decisions
       ___________________________________________________________________
        
       One Way Smart Developers Make Bad Strategic Decisions
        
       Author : ScottWRobinson
       Score  : 173 points
       Date   : 2022-03-08 14:35 UTC (8 hours ago)
        
 (HTM) web link (earthly.dev)
 (TXT) w3m dump (earthly.dev)
        
       | alephxyz wrote:
       | A well publicized example is EA mandating all its studios to use
       | the Frostbite engine
        
         | __turbobrew__ wrote:
         | I used to work on the Frostbite team at EA and it was quite the
         | train-wreck. Many game teams spent considerable time moving to
         | Frostbite only to fail and go back to the old game engine, the
         | game teams which managed to move to the frostbite engine were
         | unable to keep up with engine updates and got stuck on old
         | arcane engine versions, the frostbite engine team was split
         | between multiple geolocations and the teams in different
         | locations didn't get along well and ended up developing silos,
         | and finally there were about a million layers of management in
         | the frostbite team -- I heard from old timers that the team
         | used to be much more engineering focused.
        
         | fnbr wrote:
         | I have a bunch of friends who work for an EA studio (or who
         | used to work for an EA studio) and they all _hate_ Frostbite.
         | With a passion.
        
       | wnolens wrote:
       | Great short read and book recc.
       | 
       | I find myself often trying to articulate this idea (sometimes to
       | myself, often to a friend) - starting with the map and building a
       | territory from it --> generally bad.
       | 
       | While reading this article I wasn't even thinking about software.
       | I think of a lot of progressive and/or socialist politics. To
       | assert that the world should look as you one sees fit is flawed,
       | both because one is not representative but mostly because the
       | complexity of a system beyond toy example cannot be accurately
       | modeled enough when the risk is catastrophe.
        
       | twic wrote:
       | I haven't read Seeing Like A State. I have to say, i am
       | _extremely_ skeptical about the author 's fable about forestry.
       | Commercial foresters today still mostly uses monocultures with
       | evenly-spaced planting patterns, and i simply don't believe that
       | they would be doing that if there was a straightforwardly better
       | way to grow trees, even it was less "legible". This has a
       | powerful scent of Gladwell-esque insight porn - the sort of story
       | we love because it's counter-intuitive and makes us feel cleverer
       | than people who haven't heard it.
       | 
       | I don't suppose we have any foresters on this board who can
       | comment?
        
       | mathgladiator wrote:
       | While there is much to worry about over centralization, there are
       | benefits to at least trying if you can consolidate n-m efforts
       | when n is the total number efforts using a queue and m are the
       | snowflakes. It all depends on organization size and whether some
       | costs can be amortized.
       | 
       | Everyone having specialized everything is exceptionally expensive
       | at scale. Here, the key is tower of babel. What Tim should have
       | done is kick start the effort with one or two teams, then
       | organically grown it by reducing operational burden for new
       | teams.
       | 
       | The mistake here is trying to globally optimize rather than
       | seeking different vertical consolidation.
       | 
       | By the way, this is why Amazon has so many services.
        
       | steve76 wrote:
        
       | kerblang wrote:
       | In general, "unity" is something software developers routinely
       | pursue just for the sake of unity itself, failing to understand
       | that unity comes with significant tradeoffs. It is much harder to
       | build a unified solution than a localized, one-off solution.
       | Divide-and-conquer is often a much better engineering strategy:
       | DAC might create more work than unity, but the work is more
       | likely to succeed instead of falling apart because we failed to
       | anticipate all the use cases within the unified framework,
       | especially when we lack experience in the domain.
       | 
       | Also refer to Jeff Atwood's Rule of Threes (which he borrowed
       | from someone else) here.
        
       | kingdomcome50 wrote:
       | Counter example: Tom standardized a bunch of services... and it
       | worked! Everything is easier and more efficient now.
       | 
       | I agree with the thrust of this post: Changing something that is
       | not understood is a dubious undertaking. But the author fails to
       | make a compelling connection between the above and software
       | development. A poor solution _may_ be a result of not
       | understanding enough of the system as a whole, or it _may not_.
       | We simply can 't tell.
       | 
       | Standardization (i.e. simplification) is generally a good thing
       | in software development. How would Tim's system look if they had
       | opted for his approach from the start? How does the 3rd iteration
       | of the system compare to the 1st iteration? Maybe Tim's solution
       | is stepping-stone to something better. Impossible to tell.
        
         | EnKopVand wrote:
         | > Counter example: Tom standardized a bunch of services... and
         | it worked! Everything is easier and more efficient now.
         | 
         | I'm sorry, but that isn't really a counter point unless you
         | have some cases to back it up.
         | 
         | In my completely anecdotal experience standardisation never
         | really works. I say this as someone who's worked on enterprise
         | architecture at the national level in Denmark and has co-
         | written standardisations and principles on how to define things
         | from common building blocks.
         | 
         | The idea was that something like a journal of your health can
         | be defined as a model that can be used by everyone who ever
         | needs to define a journal for health data. And for some cases
         | it works well, it lets thousands of companies define what a
         | "person" is as an example and which parts are the person and
         | which parts are the employee and so on, and it lets them
         | exchange data between systems.
         | 
         | Until it doesn't. Because all of the sudden an employee is two
         | different things depending on what time of the day it is,
         | because a Nurse has different responsibilities while patients
         | are awake, in some hospitals, and not in others. But because
         | the "standardisation" doesn't account for this, 50 years of
         | enterprise architecture in the Danish public sector is yet to
         | really pay off.
         | 
         | Some of our best and most successful public sector projects are
         | the ones that didn't do fanatical standardisation but build
         | things with single responsibilities so that they could easily
         | be chained together to fit a myriad of unique needs.
         | 
         | Now, I'm not against standardisation in any way, but sometimes
         | it just doesn't make sense and sometimes it does. The issue is
         | that the standardisation approach tends to begin before anyone
         | knows which situation you are actually in.
        
           | stonemetal12 wrote:
           | As far as I can tell how well standardization works depends
           | on "how close to the humans" it is.
           | 
           | HTTP; TCP; json; xml: all standardize pretty well. Want to
           | standardize your micro services on nginx with data in json?
           | It will work swimmingly and save time because it is one less
           | decision to be made, and overtime everyone will become
           | familiar with how nginx is setup. Standardizing on what json
           | libs to use so that everyone can dig into the json
           | marshalling code without a lot of head scratching would be
           | another big win.
           | 
           | Trying to standardize people never works because they want to
           | do things their own way and view whatever standard you try to
           | impose as wrong.
        
             | Splizard wrote:
             | So the answer is, standardize communication/protocols but
             | leave procedure to the free market?
        
           | kingdomcome50 wrote:
           | > I'm sorry, but that isn't really a counter point unless you
           | have some cases to back it up.
           | 
           | My counter example is about exactly as detailed as the
           | author's example. Of course I was being tongue-and-cheek, but
           | _clearly_ standardization has worked in software.
           | 
           | You can toss your example right on top of all of the other
           | failed attempts at standardization. It in no-way supports the
           | conclusion that "standardization" is a problem. Like I said,
           | I agree with the author's argument, but their conclusion is
           | not supported by that argument. There are _many_ failure
           | modes to large projects.
        
       | jmull wrote:
       | Nice, interesting article.
       | 
       | But I would stress less that "Seeing Like a State" -- that is a
       | top-down, global solution -- was not the problem.
       | 
       | The problem was that "Tim" didn't really understand the problem
       | he was trying to solve (well, none of us truly understand very
       | much at all, but he didn't understand it better than many of the
       | teams associated with the individual services).
       | 
       | "Tim"'s proposal probably solved some problems but created
       | various other problems.
       | 
       | The best solution, though, (IMO) isn't that Tim should be smarted
       | and better informed than everyone else combined, nor that every
       | team should continue to create an independent solution. Instead
       | "Tim" could propose a solution, and the 100 micro service teams
       | would be tasked with responding constructively. Iterations would
       | ensue. You still really, really need "Tim", though, because
       | multiple teams, even sincere and proficient ones, will not arrive
       | at a coherent solution without leadership/direction.
       | 
       | > A global solution, by necessity, has to ignore local
       | conditions.
       | 
       | That's just flat wrong. A global solution can solve global
       | concerns and also allow for local conditions.
        
         | chasil wrote:
         | The scenario reminds me of this story:
         | 
         | https://mwl.io/archives/605
        
         | hosh wrote:
         | > That's just flat wrong. A global solution can solve global
         | concerns and also allow for local conditions.
         | 
         | Past a certain level of complexity, that's no longer true.
         | 
         | _Seeing Like a State_ is a great introduction to this, but I
         | think Carol Sanford's work goes much more into detail. The main
         | thing with the high-modernist view that James Scott was
         | critiquing is that it comes from what Sanford would call the
         | Machine World View. This is where the entire system can be
         | understood by how all of its parts interact. This view breaks
         | down at a certain level of complexity, of which James Scott's
         | book is rife with examples.
         | 
         | Sanford then proposes a worldview she calls the Living Systems
         | World View. Such a system is capable of self-healing and
         | regenerating (such as ecologies, watersheds, communities,
         | polities), and changing on its own. In such a system, you don't
         | affect changes by using direct actions like you do with
         | machines. You use indirect actions.
         | 
         | Kubernetes is a great example. If you're trying to map how
         | everything work together, it can become very complex. I've met
         | smart people who have trouble grasping just how Horizontal Pod
         | Autoscaling works, let alone understand its operational
         | characteristics in live environments. Furthermore, it can be
         | disconcerting to be troubleshooting something and then have the
         | HPA reverse changes you are trying to make ... if you are
         | viewing this through the Machine World View. But viewed through
         | Living Systems World View, it bears many similarities to
         | cultivating a garden. Every living thing is going to grow on
         | its own, and you cannot control for every single variable or
         | conditions.
         | 
         | For similar ideas (which I won't go into detail), there is
         | Christopher Alexander's ideas on Living Architecture. He is a
         | building architect that greatly influenced how people think
         | about Object Oriented Programming
         | (http://www.patternlanguage.com/archive/ieee.html) and Human-
         | Computer Interface design (what the startup world uses to great
         | affect in product design).
         | 
         | Another is the Cynefin framework
         | (https://en.wikipedia.org/wiki/Cynefin_framework). Cynefin
         | identifies different domains -- Simple, Complicated, Complex,
         | and Chaos. Engineers are used to working in the Complicated
         | domain, but when the level of complexity phase-shifts into the
         | Complex domain, the strategies and ways of problem-solving that
         | engineers are used to, will no longer work. This includes
         | clinging to the idea that for any given problem, there is a
         | global solution which will satisfy all local conditions.
        
           | salixrosa wrote:
           | Thanks for the reading recommendation! Learning about the
           | Cynefin framework and thinking about those kinds of problems
           | led me to James Scott and to Hayek, but I haven't come across
           | Sanford's work before.
        
           | hosh wrote:
           | Oh yeah, and I just remembered -- Go. It's a great way of
           | training strategic analysis and making decisions. After
           | moving past the basis, what one explores are global vs.
           | local, influence vs. territory, discerning urgent vs. big
           | moves, direction of play, and so forth. It is theoretically a
           | perfect-information game, but it is sufficiently complex
           | enough for humans that it simulates the fog of war and having
           | to make decisions in face of uncertainty.
        
             | jmull wrote:
             | "It is theoretically a perfect-information game"
             | 
             | Ha, ha! A concept for suckers.
        
               | samatman wrote:
               | Perfect information must not be confused with perfect
               | information processing.
               | 
               | GIGO does not imply its opposite.
        
           | tcgv wrote:
           | > This includes clinging to the idea that for any given
           | problem, there is a global solution which will satisfy all
           | local conditions.
           | 
           | The parent comment wasn't stating this. It was stating that
           | there could be a partial global solution that would benefit
           | all microservices, a solution which teams would have to adapt
           | for covering local conditions as well. A middle ground per
           | se.
           | 
           | Thanks for sharing the "Living Systems World View" btw, very
           | interesting!
        
           | jmull wrote:
           | Truly, a fascinating perspective, thank you.
           | 
           | Just for the context: I would say I'm a natural intuitive
           | bottom-upper, except that I can't help but reconsider
           | everything my intuitive self learns from a strongly
           | analytical top-down way.
           | 
           | From that perspective and 30+ years of experience (where I
           | like to think I'm at least open to being completely wrong
           | about anything and everything), I think top-down,
           | prescriptive solutions can be useful and effective, but need
           | to _understand_ and carve out the holes (and conservatively
           | large ones at that) for  "local" concerns - BTW, "local"
           | often typically just means lower, where there the lower level
           | itself can have "global" and "local" concerns.
           | 
           | Now, I know this often doesn't happen, so let's lay out how
           | it can work:
           | 
           | - there's a top-down person -- "Tim" in the article -- who
           | has responsibility for for developing a solution
           | 
           | - there are the separate teams, who are responsible for
           | communicating feedback on potential solutions.
           | 
           | Also, I wish I didn't need to point this out, but
           | "responsibility for" === "authority/control over".
           | 
           | (If that's not the case, then never mind: you essentially
           | have a "free-for-all" organization, and just better hope
           | someone who's not too crazy wins the cage-match and can hang
           | on long enough to be a net positive.)
        
             | aidenn0 wrote:
             | A point made that I think you are missing is that unless
             | all of the separate teams fully understand the potential
             | solution, then they can't provide useful feedback.
             | 
             | If team X doesn't know Kafka, then they can't tell you the
             | ways in which it's not as good as their existing
             | (potentially ad-hoc, but definitely working!) message
             | system. There may be things that their system does that the
             | team just automatically assumes all message-brokers will do
             | because it's "obvious" that it's needed.
             | 
             | If, on the other hand, someone on team X organically
             | considers Kafka as a local solution, learns it, tries it
             | out, all of this stuff becomes obvious immediately.
             | 
             | So the pure top-down approach has two possible solutions:
             | 
             | 1. It gets useless feedback "meh, seems fine"
             | 
             | 2. All N organizations actually take the time to try out
             | the solution before giving feedback, which means you spend
             | a lot of resources evaluating each top-down hypotheses
             | 
             | The suggested solution from TFA is to have a top-down
             | person embed in one team, find some improvements that work
             | locally, then embed in a second team and do the same. Only
             | then should one try to generalize from the shared
             | experiences of the team. It recognizes that good feedback
             | is expensive and bad feedback is likely, so just cut out
             | the whole "give feedback" stage and have the top-down
             | person learn from actually being embedded in a couple of
             | teams.
        
         | sokoloff wrote:
         | That depends a lot on the cardinality of the set of Tims.
         | 
         | If there's one Tim per team, you'll have 100 Tims proposing
         | different global improvements and 100 teams needing to respond
         | intelligently to those suggestions.
        
           | vlovich123 wrote:
           | That's when the business needs to assign one Tim, or a team
           | of Tims as dictator(s). Everyone else can provide feedback
           | but isn't the decision maker.
        
             | canalmighty wrote:
             | A Timtactorship
        
         | GiorgioG wrote:
         | > A global solution can solve global concerns and also allow
         | for local conditions.
         | 
         | Not if standardization is the priority.
        
           | CogitoCogito wrote:
           | > Not if standardization is the priority.
           | 
           | Is standardization always a priority?
        
             | [deleted]
        
         | ryukoposting wrote:
         | Through enough iteration, all problems can be solved. But, how
         | many iterations will be required to reach a solution that works
         | for everyone? At that point, is there a solid business case for
         | the project?
        
         | darkerside wrote:
         | I think the key difference is whether the local teams have the
         | choice to opt out or not, and my belief is that they should. If
         | they can, they can solve their own problem if the global
         | solution doesn't work. If the global solution wants to keep
         | them as consumers, they must adapt. If they can't leave, the
         | global team will almost certainly stop responding to their
         | needs over time. Like communism, a global solution is terrific
         | in theory, but human behavior causes it to break down in
         | practice.
         | 
         | Caveat, for small enough problems, good enough solutions, and
         | charismatic enough leaders, global solutions can work. But they
         | all break eventually.
        
         | phkahler wrote:
         | >> > A global solution, by necessity, has to ignore local
         | conditions.
         | 
         | >> That's just flat wrong. A global solution can solve global
         | concerns and also allow for local conditions.
         | 
         | So lets rephrase that. A global solution that ignores local
         | conditions will have problems and will likely fail.
        
         | bentcorner wrote:
         | Makes sense. In my work I've seen this when trying to get
         | developers on my team using certain patterns, styles, types,
         | conventions, or tools (or the inverse - deprecating them).
         | 
         | Suggestions are usually well grounded (e.g., "let's migrate to
         | this `std` class instead of this old home-rolled wrapper), but
         | sometimes there's some nuance to how something is currently
         | done and deep discussion of the proposal can work through these
         | bits.
        
       | NateEag wrote:
       | For anyone interested in social systems that help avoid this top-
       | down, centralized failure mode, I cannot recommend RFC 7282
       | enough:
       | 
       | https://datatracker.ietf.org/doc/html/rfc7282
       | 
       | A whole lot of wisdom is captured in that document, including a
       | deep understanding of the differences between unanimity, majority
       | rule, and consensus.
       | 
       | If you're involved in standardization efforts in any way, whether
       | it's deciding where your team will put braces in source code or
       | running software architecture for a Fortune 100, it will well
       | repay your reading time.
        
         | svilen_dobrev wrote:
         | interesting. For long time i've found that negative logic is
         | more powerful/overarching than positive one - #ifndef NOT_THIS
         | is more powerful than #if THIS .. and this article applies that
         | even to agreeing vs not-disagreeing.
        
       | yellowstuff wrote:
       | This article does a good job describing one failure mode that's
       | not understood well, but the opposite failure mode is much more
       | common in my experience- having lots of ways to do the same thing
       | can be very inefficient and brittle, even at small companies. The
       | right answer is not "never unify systems" or "always unify
       | systems", but develop judgement about when things should be
       | unified.
        
       | [deleted]
        
       | wvenable wrote:
       | I disagree with the conclusion although not necessarily with the
       | situation described.
       | 
       | A programmer only has so many hours in the day; if you want to be
       | a more efficient programmer you either have to learn to
       | type/think faster or you need to build frameworks, write
       | libraries, and codify common practices.
       | 
       | There are situations where that doesn't work but if your job is
       | to pump out code for an organization there's a good chance that
       | most of your applications will have the same authentication, the
       | same look and feel, etc. Putting effort into that core will pay
       | dividends later. But you can't be a slave to your own code; if it
       | doesn't fit in the box then don't force it.
        
         | sokoloff wrote:
         | > to be a more efficient programmer you either have to learn to
         | type/think faster or you need to build frameworks, write
         | libraries, and codify common practices.
         | 
         | That's focused on the writing/creating side of the equation. In
         | my side projects, I became a lot more efficient when I decided
         | to put effort into _using_ frameworks, _adopting_ libraries,
         | and _copying_ common practices.
        
           | wvenable wrote:
           | You should absolutely be using existing frameworks,
           | libraries, and common practices. But a popular framework will
           | allow you make any application imaginable. If you're not
           | making radically different applications every day, you can
           | extend and constrain existing frameworks with your own code
           | to make them specific to your needs.
           | 
           | For example, all my applications have the same look and feel,
           | corporate branding, and authentication features. We use a
           | popular framework but extended it to handle all of this stuff
           | so it's never repeated between projects.
        
       | travisgriggs wrote:
       | Lots of resonant points here. It's worth making it to the end.
       | 
       | I work at a company where there's a number of different little
       | less-than-one-man projects, and there's a lot of variety, and so
       | a couple of non-tech types, frustrated with resource allocation
       | (having the right kind of skills at the right place at the right
       | time in the right amount) wants to standardize and simplify.
       | 
       | What I've observed though is that when you tell your house
       | painters they can only work with black paint, they can only give
       | your customers black walls, and when your customer wants wood
       | panel, or textured fuschia, then you can't earn revenue from that
       | market demand.
        
       | galaxyLogic wrote:
       | As a whole the strategy of "Let's see what's common in all these
       | systems" is a good start to understanding the systems. There is a
       | limit to the complexity any single person can understand.
       | Unification is simplification. It helps understanding. But I
       | agree it is no good trying to make the landscape fit a simple map
       | when reality is much more complex. There's no Silver Bullet in
       | trying to combat complexity.
       | 
       | But rather than trying to unify everything think about micro-
       | services. Each service can be its own isolated solution. Of
       | course it needs to be optimized in terms of how well it works
       | when all the other services are running as well. But I think
       | isolation is the key to independently optimizing everything.
       | 
       | I like this sentence from the article: "Lots of it doesn't
       | matter, but some of it matters a lot".
        
         | Splizard wrote:
         | Except when Tim decides to standardize how each micro-service
         | is built, using a custom framework...
        
       | itsdrewmiller wrote:
       | This article is amazing and is perfectly timed for the news about
       | Sri Lanka's mass move to organic farming -
       | https://news.ycombinator.com/item?id=28443798
        
       | eternityforest wrote:
       | I've noticed that ALL beginners seem to have a reinvented global
       | solution phase.
       | 
       | Everyone who does electronics might say "Oh I'm going to use this
       | one connector for everything". And it's either ok, if it's a
       | standard connector, or a giant pile of crap that means they can't
       | use a lot of existing stuff because they insisted on this insane
       | DIY grand scheme.
       | 
       | Usually such things have an element of "I want to do Y, so I'll
       | build a modular kit X and use that to Y". And then X becomes the
       | real project and Y is never finished.
       | 
       | The insidious part is how the new product is often a tiny bit
       | better than what's out there. But it doesn't matter. The mediocre
       | standard solution is still way less trouble than the beautiful
       | perfect custom thing. I'd rather have Just Works tech than tech
       | that's Just Right. Anything that seems perfect and beautiful and
       | simple, I don't trust, because it was probably made for one
       | specific task, not to be a general standard you don't have to
       | think about.
       | 
       | I think of the failures with global solutions are because someone
       | did them on a small scale, or because they have to do with
       | natural systems.
       | 
       | Fully top down planning of manmade things by a giant industry
       | consortium is most of why tech is great. Otherwise we would have
       | no USB C, and 12 different CPU architectures.
       | 
       | Sometimes design by comittee protocols suck, but usually because
       | they didn't have _enough_ control, and instead of a protocol,
       | they deliver a description language for companies to make their
       | own protocol, with every feature optional so that compliance does
       | not necessarily mean compatibility.
       | 
       | When you do it internally it can suck because it's more effort
       | than it's worth to replace all your existing stuff.
        
       | ryukafalz wrote:
       | Some good points. At the same time, sometimes standardization is
       | good and necessary. Imagine if everyone had to reimplement TCP/IP
       | analogues to communicate over a network; we'd never get anything
       | done!
        
         | NateEag wrote:
         | And those types of universal standards arrive by a process not
         | unlike evolution - everyone who wants to has a crack at the
         | problem and the solutions compete for a painful decade or two.
         | Eventually a winner emerges from the top few frontrunners, and
         | by 2040 only networking historians remember ALOHAnet or token
         | ring networks.
         | 
         | However, you can't top-down design a universal standard.
         | 
         | With sufficient skill, patience, and wisdom, you may be able to
         | design a standard that's good enough to be widely applicable.
         | 
         | Humans have free will, though (in practice, at least, no matter
         | what you think about the philosophical question).
         | 
         | A standard only becomes universal when everyone chooses to
         | adopt it.
        
       | saila wrote:
       | The first part of the article describes the common situation
       | where we see similarities across projects and effort being
       | duplicated, apparently unnecessarily.
       | 
       | To improve understanding and efficiency, we come up with
       | proposals that involve some kind of standardization, such as a
       | shared library abstracting a service.
       | 
       | My team has recently embarked on such an effort, and I was really
       | hoping for a new take on how to avoid the various pitfalls
       | involved in attempting this kind of standardization.
       | 
       | Obviously, central planning works quite well for certain things
       | and, just as obviously, it hasn't worked well for other things.
       | In most situations, it seems that a combination of planning +
       | flexibility works best.
       | 
       | Further, what works or doesn't work for nation-states isn't
       | particularly applicable to software architecture. The analogy
       | feels quite tortured to me.
       | 
       | As to the example given in the article regarding trees, one could
       | just as easily choose an example from modern agriculture where
       | "central planning" seems to work quite well.
       | 
       | In the end, I feel like the article just boils down to: "Make
       | sure you understand the problem domain and the cost of large
       | scale change before you spend a lot of time and effort making
       | said changes."
        
         | memorythought wrote:
         | My reading of Seeing Like A State was not "central planning
         | bad". The author explicitly acknowledges that there are many
         | benefits to what they refer to as "high modernist" projects
         | which we all enjoy on a daily basis. My reading was more that
         | large scale projects designed to entirely change the way
         | something is done in order to make it legible to a central
         | authority necessarily throws away an enormous amount of local
         | information and this leads to a tradeoff. You end up being able
         | to build much bigger systems, but those systems are not as
         | flexible.
         | 
         | In the context of microservices that seems like a pertinent
         | point because the purpose of a microservice architecture in the
         | first place is often to allow an organisation to be more
         | flexible and accommodate local knowledge.
        
       | dre85 wrote:
       | Very interesting article! I was left wondering though in what
       | ways the queue abstraction solution failed?
        
         | dilatedmind wrote:
         | for this specific example, I think the shared library is not
         | the correct approach. Queues work for simple fifo behavior, but
         | in this case they also need fairness (and perhaps other logic,
         | like rate limiting per client, different priority for certain
         | clients etc)
         | 
         | For example, "Customer-A is clogging up the work queue and
         | starving other customers out". The solution to this could look
         | something like linux's completely fair scheduler, where every
         | client is allocated some amount of messages per interval of
         | time. This means messages need to be processed in a different
         | order then they are enqueued, and queues are not good at
         | reordering messages.
         | 
         | I would suggest implementing the queue abstraction as a rest or
         | grpc service, backed by a database like postgres, which holds
         | message payloads and everything related to message state (in
         | progress, retry after times, etc). Now we can implement all the
         | necessary scheduling logic within our queue service.
        
         | kerblang wrote:
         | They hinted at the fact that kafka is not actually a queue, and
         | it especially has problems with fairness if you try to use it
         | as a queue for unequally sized "jobs" and/or unequally sized
         | consumers. Kafka is for the exceptional case; actual message
         | queues are for the default/general case.
        
           | dre85 wrote:
           | That makes sense, but it also doesn't really support the
           | article's overall argument about the perils of
           | "standardization without understanding". If what you're
           | saying is true then it could very well be that the approach
           | to standardize the queuing would have worked perfectly well
           | had the senior dev chosen a different technology to underlay
           | his library (ie not Kafka), right?
        
             | kerblang wrote:
             | Yes technically the Kafka matter is a secondary to the
             | author's main point, which is _understanding each problem
             | first_. So yeah I would not waltz in and say,  "Oh, no need
             | for details, obviously you need to standardize on RabbitMQ
             | not Kafka," without a second thought, even though it's
             | likely that's _roughly_ where we 're gonna end up.
             | 
             | But even then, I am not going to write a custom library
             | when the drivers for either product are perfectly adequate.
             | Standardization works best when standards are flexible, and
             | trying to force my homemade library on every team is
             | inflexible (usually it's just arrogance).
        
       | sudhirj wrote:
       | This seems to be hallmark of a "Middle" developer. Not so junior
       | that they couldn't build a working solution that they assume
       | everyone should use, but not senior enough to think twice about
       | whether they should be building it.
       | 
       | The "we should make a common framework" for this line is the
       | dominant thought at this level. Never even a library. A
       | framework. Everyone must do it this way.
       | 
       | The more senior people share concepts and maybe libraries, and
       | allow the team to use them if they see fit.
        
         | 1123581321 wrote:
         | This kind of consequential decision can happen at high levels.
         | Obviously less often when a truly brilliant developer ends up
         | in a small organization (but that has its own risks.)
         | 
         | In the example, it was determined that they could not afford to
         | let each service solve its individual bottlenecks ad hoc. So a
         | corresponding strategic error was also made/forced at the
         | senior business level.
         | 
         | It's easy to speculate in hindsight, but in this case I could
         | imagine a globally enforced throughput mandate supported by a
         | widely visible and frequently reviewed dashboard, new
         | tools/libraries as needed, and an optional central queue
         | service required to 'compete for business' of the individual
         | service teams.
         | 
         | I can see potential problems with that too, though. In a sense,
         | failure has already happened when growth management is deemed
         | to be too important to be left to capable individuals on
         | decentralized teams.
         | 
         | Enjoyed the article.
        
         | Fiahil wrote:
         | I agree.
         | 
         | People use the most practical things at their disposal. If Tim
         | had opted to publish a repository of easy and _simple_ recipes
         | for managing kafka and postgres integrations, while retaining
         | the ability to use original libraries, then I see no reason why
         | it would not have gained traction.
        
         | shuntress wrote:
         | It's the large-scale version of taking "DRY" too literally.
         | 
         | Junior devs just repeat themselves because they don't know
         | better.
         | 
         | Middle devs rush into an incomplete abstraction by
         | overzealously not-repeating-themselves.
         | 
         | Senior devs just repeat themselves because they know they don't
         | understand what it would take to abstract out the solution.
         | 
         | Like everything... "It Depends". Don't Repeat Yourself _Too
         | Much_.
        
           | [deleted]
        
         | GiorgioG wrote:
         | I've worked at bigger companies and there are plenty of folks
         | much higher than 'middle dev' forcing these types of things
         | down the organization's throat.
        
           | lawn wrote:
           | Maybe even the CEO or CTO.
        
           | pc86 wrote:
           | You can certainly be a mid-level in skill but be a
           | senior/staff/principal at the company, or a
           | senior/staff/principal in _technical_ skill but middle or
           | junior in strategic or design skill.
        
             | [deleted]
        
       ___________________________________________________________________
       (page generated 2022-03-08 23:01 UTC)