[HN Gopher] Normalization of Deviance (2015)
       ___________________________________________________________________
        
       Normalization of Deviance (2015)
        
       Author : akshaykarthik
       Score  : 282 points
       Date   : 2023-02-14 15:52 UTC (7 hours ago)
        
 (HTM) web link (danluu.com)
 (TXT) w3m dump (danluu.com)
        
       | kerblang wrote:
       | Great stuff - I think this goes in the "required reading" list.
       | 
       | The tech industry tends to revolve around "I'm a super-rational
       | robotic genius" thinking that can't accept the existence of its
       | own irrational tendencies, to the point that it becomes
       | ridiculous.
        
         | [deleted]
        
         | SideburnsOfDoom wrote:
         | The standard "Required reading" text on the subject is "The
         | Field Guide to Understanding 'Human Error'" By Sidney Dekker
        
           | goostavos wrote:
           | Great book! (Although, it could have used a more aggressive
           | editor)
           | 
           | Reading it felt like a personal attack in many places.
           | However, reading it forever changed how I think about things.
           | It's a much more useful framing for everyone involved if you
           | start with the question of "why did they think this was the
           | right thing to do?" as opposed to "this person made a bad
           | choice / mistake". My (extraordinary) impatience naturally
           | predisposes me towards the latter, but the core argument of
           | the book is that that's _lazy_ -- you can hand wave away
           | anything and everything with  "operator error".
        
       | nilespotter wrote:
       | [flagged]
        
         | badrabbit wrote:
         | It is not.click.
        
         | dang wrote:
         | Can you please not post unsubstantive comments? It looks like
         | you've been doing it repeatedly, and we're trying for something
         | else here.
         | 
         | https://news.ycombinator.com/newsguidelines.html
        
       | dec0dedab0de wrote:
       | I'm still reading, but I just got to the part about flaky, and I
       | got annoyed because there are clear use cases for flaky or
       | pytest-retry.
       | 
       | If you have an integration test that relies on an unreliable
       | system you do not control. Sure you can mock it out for a unit
       | test, but if you want to make sure you catch breaking API
       | changes, you need to hit the actual system. And if it works after
       | retrying it a few times, then so be it. no need to throw shade.
        
         | JackFr wrote:
         | I'm going to push back and say that test is not a valuable
         | automatic test. The phrase "relies on an unreliable system"
         | captures that lack of value.
        
           | dec0dedab0de wrote:
           | When the code your testing is a client for some remote API,
           | and the sandbox/development/Testing version of that API
           | doesn't have the same resources and uptime guarantee as
           | production, then what are your options? as far as I can tell
           | they are:
           | 
           | Don't test it.
           | 
           | Only do unit tests with the connection mocked out.
           | 
           | Test against production.
           | 
           | Try it a few times with a delay, and if it works then you
           | know your code is good and you can move on with your
           | deployment. Which is what flaky and pytest-retry do.
           | 
           | Maybe I'm missing something, but out of those 4 options
           | retrying the test seems like the best one, with the big
           | caveat that it is only viable if the test does indeed work
           | after trying a few times. I really don't see any downside.
           | 
           | edit:
           | 
           | Maybe another option is to put the retry functionality
           | directly in the client code, which would make your code more
           | robust overall. but that is definitely more complex than
           | using one of these libraries just for testing.
        
             | salawat wrote:
             | You're on the right track. It's a perennial favorite of
             | devs to abhor flakyness, whereas after spending enough time
             | as a tester, you come to terms with the fact that you have
             | to take your tests as a statistical probe because most
             | places test systems are simply not that reliable;
             | sometimes, this is even a design feature.
             | 
             | This experience as a tester is in fact a normalization of
             | deviance from the ideal computation model of a developer.
             | Everything should work the first time everytime from their
             | point of view. The tester sees reality as it is. The
             | Emperor won't fund my test systems sufficiently to service
             | all my customers, so we make do ss best we can. Bonus
             | points in that we get to exercise the edge cases.
        
         | Aeolun wrote:
         | I think the issue is people using it when they're too lazy to
         | fix the test case.
        
           | namaria wrote:
           | I find quite interesting that people will prefer a highly
           | malleable language like Python, and then orgs have to adopt
           | testing to get around all the inconsistencies caused by
           | absent type system. And then people will write libraries to
           | get around the pesky tests to get their flexibility back.
           | 
           | It's fascinating really... Complex systems are always in
           | partial failure mode and that applies to collective
           | optimization challenges. Organizations will always be stuck
           | in local optima in most domains.
        
             | dec0dedab0de wrote:
             | Type systems do not replace testing, and if a test works
             | after retrying it then it is probably not something that a
             | type system would be able to catch.
        
       | somat wrote:
       | A thought experiment.
       | 
       | When is it "Normalization of Deviance"? and when is it a
       | "Efficiency Optimization"?
       | 
       | I mean, the difference is pretty clear after something has
       | failed, But very murky before.
        
         | jpollock wrote:
         | It is Efficiency Optimization when you know why the rule is
         | there, and having made an estimation of the risks, perform a
         | cost-benefit analysis.
         | 
         | aka "Chesterton's Fence"
         | 
         | Otherwise, it's "Normalization of Deviance":
         | 
         | * The build is broken again? Force the submit.
         | 
         | * Test failing? That's a flaky test, push to prod.
         | 
         | * That alert always indicates that vendor X is having trouble,
         | silence it.
         | 
         | Those are deviant behaviours, the system is warning you that
         | something is broken. By accepting that the signal/alert is
         | present but uninformative, we train people to ignore them.
         | 
         | vs...
         | 
         | * The build is always broken - Detect breakage cause and auto
         | rollback, or loosely couple the build so breakages don't
         | propagate.
         | 
         | * Low-value test always failing? Delete it/rewrite it.
         | 
         | * Alert always firing for vendor X? Slice vendor X out of that
         | alert and give them their own threshold.
        
           | manicennui wrote:
           | Unfortunately I don't find that most software engineers
           | understand the difference between actually determining costs
           | and benefits and choosing to make certain tradeoffs and
           | rationalizing whatever choice they already made.
        
             | jpollock wrote:
             | I think that's ok, For me, it's more about "change the
             | system, instead of ignoring it".
             | 
             | Once you change the system (document/rules/alerts/etc),
             | then if it breaks, you change it again and learn the
             | lesson. Both are conscious decisions by the org.
        
       | dang wrote:
       | Related:
       | 
       |  _Normalization of Deviance (2015)_ -
       | https://news.ycombinator.com/item?id=22144330 - Jan 2020 (43
       | comments)
       | 
       |  _Normalization of deviance in software: broken practices become
       | standard (2015)_ - https://news.ycombinator.com/item?id=15835870
       | - Dec 2017 (27 comments)
       | 
       |  _How Completely Messed Up Practices Become Normal_ -
       | https://news.ycombinator.com/item?id=10811822 - Dec 2015 (252
       | comments)
       | 
       |  _What We Can Learn From Aviation, Civil Engineering, Other
       | Safety-critical Fields_ -
       | https://news.ycombinator.com/item?id=10806063 - Dec 2015 (3
       | comments)
        
       | deanCommie wrote:
       | Reality: It is true that EVERY organization is broken in some way
       | or another.
       | 
       | You have to find the one that is broken in the way that is
       | tolerable to you.
       | 
       | Arguably the closest we know to a panacea in terms of engineering
       | culture and best practices is Google. And what are they now known
       | for? An inability to ship anything meaningful anymore. Spinning
       | around in circles launching and re-launching new chat apps.
       | 
       | These are not unrelated. High engineering standards are always in
       | tension with product delivery. As a security engineer once told
       | me, "the most secure system is the one that never gets launched
       | into production."
       | 
       | So while Dan is right, and all the examples are right, and things
       | like non-broken builds and a fast CI/CD pipeline are totally
       | achievable, don't learn the WRONG lesson from this which is that
       | when you arrive to a company and notice a bunch of WTFs, the
       | first thing you must do is start fixing them in spite of any old
       | timers who say "Actually that's not as bad as it seems".
       | Sometimes they're wrong. USUALLY, they're right.
        
       | stevehawk wrote:
       | This is a big term in aviation, because in most cases in order
       | for something catastrophic to happen it requires a lot of things
       | to have failed. And one way to ensure that enough things fail is
       | to start deviating from your maintenance, inspections, or general
       | responsibilities. Related: the swiss cheese models.
        
         | kurthr wrote:
         | It's also why things in aviation are so fixed and difficult to
         | change. Not having any new civil aviation planes for 30 years
         | worked... how about 40, 50? When will it break? Well, when
         | someone develops an easy to build, easy to fly, inexpensive
         | experimental craft and zillions of people do it all at once
         | (hasn't happened yet).
         | 
         | One of the more interesting things I've found is that a huge
         | number (easily a majority) of instructors are recently trained
         | flyers, because there is a pipeline to train them and they're
         | cheaper than using experienced pilots (esp for multi-engine and
         | more complex airplanes). They also know all the ins-outs of the
         | training and rule books (with recent changes) so they know how
         | to pass all the tests and how to teach that. Sooooo you have a
         | bunch of inexperienced pilots teaching all the new pilots...
         | there's likely a failure there, but it hasn't reared its head.
         | We still have a lot of ex-military folks around who didn't
         | learn that way.
         | 
         | Who do you want flying when things go bad? People who have
         | spent many hours with things about to go bad (military,
         | emergency/fire, sail plane pilots) who have experience dealing
         | with it. Those people can also be fun/terrifying to fly with,
         | because they will take risks.
        
           | neilv wrote:
           | > _They also know all the ins-outs of the training and rule
           | books (with recent changes) so they know how to pass all the
           | tests and how to teach that. Sooooo you have a bunch of
           | inexperienced pilots teaching all the new pilots..._
           | 
           | Sounds like there's some similarities with everyone focusing
           | on Leetcode interviews, and then one generation of that
           | filtering and then mentoring the next, and repeat.
           | 
           | The companies don't know what that's costing them, until
           | there's a problem that can't be ignored.
           | 
           | In the case of software engineering (poorly studied, relative
           | to aviation) the company will generally never learn whether a
           | non-Leetcode&promo-focused team could've avoided the problems
           | in the first place, nor whether non-Leetcode experience
           | could've handled a problem that happened anyway.
        
           | sokoloff wrote:
           | > Not having any new civil aviation planes for 30 years
           | worked...
           | 
           | The A220, A350, and A380 are all newer than 30 years old.
           | (A321 is barely younger than that and A330 barely older.)
           | Boeing has released the 777 and 787 in the last 30 years. The
           | Cirrus SR20 and SR22 are newer than that, as is the SF50 jet.
           | The Diamond DA40, DA42, and DA62 are newer. The Honda Jet is
           | newer. Cessna has a handful of business jets newer than that.
           | The Embraer Phenom 100 and 300 are newer than that. There are
           | variants of the CRJ newer than that (-700, -900, -1000).
           | 
           | That's a lot of new civil aviation aircraft designs in the
           | last 30 years.
        
           | pja wrote:
           | > Those people can also be fun/terrifying to fly with,
           | because they will take risks.
           | 
           | Risk homeostasis in action!
           | 
           | https://soaringeconomist.com/2019/10/30/experience-can-
           | kill-...
        
       | martopix wrote:
       | > Have you ever mentioned something that seems totally normal to
       | you only to be greeted by surprise?
       | 
       | Once some (foreigner) person was surprised at my dipping toast
       | with Nutella in my latte. I was equally surprised by his
       | surprise.
        
       | chaps wrote:
       | These stories ring so, so true. Once worked at a company whose
       | infrastructure issues were so deep and festering that after
       | fighting a fire, my boss told me, "If you go to the press about
       | this, the client will sue us and everyone who works here will
       | lose their jobs."
        
         | aliqot wrote:
         | That's what I never understood about this story; did you guys
         | have any suspicion it would dump radiation into the patient all
         | at once, or was this like a concurrency bug
        
           | whats_a_quasar wrote:
           | Huh? Are you assuming that the parent comment is about
           | someone programming a medical device?
        
             | lifeisstillgood wrote:
             | I'm think it's a meta joke - Therac-25
             | (https://en.m.wikipedia.org/wiki/Therac-25) was a
             | radiotherapy machine from the early 1980s and is (in)famous
             | for having software failed and killed I think dozens of
             | people. It's become a well known case study, but it's
             | highly unlikely anyone on HN worked on it - I think that's
             | the joke.
        
             | [deleted]
        
           | chaps wrote:
           | We weren't able to reliably install security daemons on a
           | client's machine because the entire automation system didn't
           | account for autoscaling. The issues were raised well before I
           | joined and the project head legitimately didn't understand it
           | as a problem that needed solving. The hosts were for a
           | presidential candidate's webserver, and they noticed the
           | webservers were missing security daemons days before the
           | election.
        
             | DiggyJohnson wrote:
             | Maybe I'm missing a joke, but was your client HRC's
             | campaign?(?!)
        
               | strbean wrote:
               | Did HRC's campaign website get hacked? I know her
               | mailserver was hacked, but that was when she was
               | secretary of state, no?
        
               | aliqot wrote:
               | That's not important and this ain't the place to ask
               | otherwise they'd have told us.
        
               | iosono88 wrote:
               | [dead]
        
               | DiggyJohnson wrote:
               | > this ain't the place to ask
               | 
               | Am I double-whooshing here?
               | 
               | How is a Hacker News comment thread not the right place
               | to respectfully ask questions in response to interesting
               | comments. I know I'm not entitled to an answer, nor do I
               | intend to start a flame war. Sheesh
        
               | aliqot wrote:
               | It is personal information that risks identifying them
               | more than they already had at the time of posting. It
               | took about two seconds to put everything together. I
               | don't have a dog in this fight politically one way or the
               | other, people don't need to identify themselves IRL here.
        
               | DiggyJohnson wrote:
               | Who are you to decide what others are comfortable sharing
               | on here. It is quite literally as simple as the person I
               | replied to choosing not reply to my comment. Why is this
               | issue a concern to you?
               | 
               | > I don't have a dog in this fight politically one way or
               | the other
               | 
               | Neither do I.
               | 
               | > people don't need to identify themselves IRL here
               | 
               | I don't think they do either. Why are you assuming I
               | "needed" this information?
        
               | chaps wrote:
               | Can you not? They're right.
        
               | lmm wrote:
               | There's nothing respectful about asking something that
               | someone has very blatantly made a deliberate decision to
               | leave out of their post, for completely understandable
               | reasons.
        
             | aliqot wrote:
             | jeez thats a rough spot to be in. did you stick around to
             | fix it or just get the hell out of dodge after that?
        
               | chaps wrote:
               | I did what I could with a handful of selenium scripts,
               | then hit a road block because we didn't have ssh access
               | to a chunk of the autoscaling hosts. Gave up after that,
               | told the customer rep to tell them we can't do it, and
               | gave my two week notice about a month later.
        
               | aliqot wrote:
               | Ouch, that has to be rough to endure. I'm glad you seem
               | to be in a better place now. Good on you for doing the
               | right thing and getting the hell out of there when your
               | options ran out.
        
             | outworlder wrote:
             | > security daemons
             | 
             | AKA compliance checkbox crap?
             | 
             | If infrastructure is immutable (which makes it work even
             | better for autoscaling), nothing new will get installed
             | unless you build a new image. Export whatever data you
             | require to ensure things you want to be running are
             | running. Monitor entry and exit points.
             | 
             | What is left for the "security deamons" to do?
        
             | riffic wrote:
             | are "security daemons" truly necessary though?
             | 
             | this whole thing sounds like a troll with enough convincing
             | language to seem plausible
        
               | chaps wrote:
               | We could debate security daemons until our minds bleed,
               | but.. man, I wish that all didn't happen.
        
       | Logans_Run wrote:
       | I'm not sure if this link has already been posted but have a look
       | at _How I Almost Destroyed a PS50 million War Plane and The
       | Normalisation of Deviance._
       | 
       | https://www.fastjetperformance.com/blog/how-i-almost-destroy...
        
       | superpope99 wrote:
       | Has Dan Luu ever explained why he doesn't put dates in his blog
       | posts?
        
         | Jtsummers wrote:
         | Most of his posts seem to be date-independent. To the extent
         | that it matters, you can check the homepage:
         | https://danluu.com/. There you will find month and year of the
         | posts.
        
       | aeturnum wrote:
       | If you enjoyed this - I highly recommend watching Adam Curtis'
       | _Can 't Get You Out of My Head_: https://thoughtmaybe.com/cant-
       | get-you-out-of-my-head/
        
       | pizzaknife wrote:
       | i routunely remind everyone,"we're all mercenaries."
       | 
       | i have marginal control over who i manage. The Product isnt
       | saving the world, but it is allowing us to live reasonably and
       | with a clear soul at the end of the sprint. The reason i say the
       | "mercenary" bit is simple: weigh your dreams against blood and
       | gold and compromise.
        
       | shadytrees wrote:
       | formatted for wide monitors https://ddanluu.com/wat
        
       | PaulHoule wrote:
       | I like the bit about                  Let's look at how the first
       | one of these, "pay attention to weak signals",        interacts
       | with a single example, the "WTF WTF WTF" a new person gives off
       | when the join the company.
       | 
       | and kinda wonder if a company that prioritized not getting this
       | reaction from new hires might find it is the most impactful thing
       | they can do in terms of culture.
        
         | lo_zamoyski wrote:
         | That doesn't seem like a healthy standard b/c it grounds
         | decision making in appearances rather than principles and
         | prudential judgements. Certainly, such feedback or opinions can
         | be worth considering as a way of getting at what principles are
         | being violated and deciding whether these violations are
         | tolerable or what ought to be done about them. A fresh pair of
         | eyes _could_ help. But an untrained pair of eyes might also not
         | be qualified to discern the right course of action.
        
           | PaulHoule wrote:
           | Yes and no. But 2/3 of the time the problem is that the
           | company has no documented build process or something obvious
           | like that . They have time to spend 2 years failing to
           | deliver a product because they don't know how to build it,
           | but when somebody asks "How do we build it?" the answer is
           | "Don't waste our time asking stupid questions." Of course
           | they have been wasting time not knowing how to build the
           | system, the guy who started the project might be able to hit
           | F5 in 15 different windows and get it to sorta kinda work,
           | but new hires they are hiring to work on it are quitting
           | right away and somehow they can never get it into production.
           | 
           | There should be no controversy at all that complete
           | instructions for installing everything required for a dev to
           | build the project and work on it should exist and it should
           | be possible to complete this task in hours, not the weeks
           | that it frequently takes. And, no, "docker" is not an answer
           | to this anymore than "The F5 Key is a Build Process"
           | 
           | https://blog.codinghorror.com/the-f5-key-is-not-a-build-
           | proc...
           | 
           | It is not "Docker" that solves the problem, it is the
           | discipline of scripting the image build process into a
           | dockerfile. If you know how to write a dockerfile you can
           | write a bash script that runs in 20 seconds as opposed to
           | having Docker spend 20 minutes downloading images and then
           | crash because of a typo.
           | 
           | You are right that a company might have good reasons for
           | doing things in an unobvious way, but most of the time when
           | nobody at a company claims to understand what the company is
           | doing except for the CEO and people aren't too sure about the
           | CEO, it is the fault of the company lacking alignment, not a
           | natural property of freshers.
        
             | skissane wrote:
             | > It is not "Docker" that solves the problem, it is the
             | discipline of scripting the image build process into a
             | dockerfile. If you know how to write a dockerfile you can
             | write a bash script that runs in 20 seconds as opposed to
             | having Docker spend 20 minutes downloading images and then
             | crash because of a typo.
             | 
             | The problem is the bash script may end up depending on
             | poorly understood aspects of the local setup (global config
             | files, installed packages, etc) - it might work fine now,
             | but then nobody runs it for 12 months and there's some
             | churn in personnel and suddenly people are trying to work
             | out why it crashes. Dockerfiles can avoid some of that
             | stuff, although not always (e.g. the common problem that if
             | you don't fix the versions of packages to be installed, an
             | updated package is released which then breaks the
             | Dockerfile)
        
         | chaps wrote:
         | Been told that before. When I spoke up, I was told that I was
         | new and shouldn't talk about things I know nothing about.
        
           | PaulHoule wrote:
           | It's not (1) "reacting to the reaction" which is the endpoint
           | but (2) not having that reaction. If (1) is important it as
           | because that is the path to (2).
           | 
           | I'd say a company that has accomplished (2) has cut the
           | workload in hiring employees by 30-50% in the sense that
           | every employee who has reaction (1) either internally or
           | externally is at risk for being disengaged or leaving soon.
           | Not only that but you are probably wasting your dev's time
           | and could get dramatically more productivity out of them if
           | you aren't WTFing them to death.
        
           | giobox wrote:
           | There's often a wise tradeoff between criticizing systems
           | you've just seen after being at the company 5 minutes and
           | actually spending some time at the company to learn the
           | historical context of _why_ the thing you think is insane
           | /shit is insane/shit before telling everyone who built it how
           | insane/shit it is.
           | 
           | People generally don't wake up in the morning and go into
           | work motivated to make insane/shit things - context, tech
           | debt and business realities all mount up and even the best of
           | us can end up making choices that in isolation look crazy.
           | 
           | There are of course companies who are really bad and you may
           | well be right, but so many times I have seen in my career a
           | young new hire storm in and think everything is shit without
           | paying heed to the context and historical pressures. The best
           | thing you can do in many cases is spend the first ~six months
           | at a new tech company trying to understand that context, and
           | indeed I think more mature engineers generally do.
        
             | a4isms wrote:
             | "Making good decisions, therefore, requires understanding
             | past decisions. Without knowing how things came to be, it's
             | easy to make things worse."
             | 
             | --https://thoughtbot.com/blog/chestertons-fence
        
               | chaps wrote:
               | I was once told this after raising the issue that "hey
               | maybe an API that responds with root ssh passwords is a
               | bad idea, and our clients are going to be pissed once
               | they find out." And.. I was right.
               | 
               | So often, citing Chesterton's fence is significantly more
               | naive than what it attempts to criticize.
        
               | a4isms wrote:
               | How is being right about wanting to make a change a
               | refutation of the idea that knowing why people made a
               | decision in the past is an excellent idea? Nothing about
               | Chesterton's Fence asserts that people in the past always
               | made good decisions, nor does it assert that perhaps what
               | was a good decision then is a bad decision today.
               | 
               | It simply asserts that understanding why a thing is the
               | way it is is valuable when making a decision to change
               | it.
               | 
               | That understanding could be as simple as--to take a real
               | world example that most readers here will remember--"They
               | chose to install a hidden web server on the user's
               | system, because they felt it was the best way to deliver
               | user convenience given the resources and time the team
               | had available."
               | 
               | We can still say it was a bone-headed choice to do that
               | because it opened a massive back door to every user's
               | system. And? What is the problem with looking into why
               | they made that choice before arguing that the choice
               | should be reversed with maximum prejudice?
               | 
               | Chesterton's Fence isn't a suggestion that no changes
               | should be proposed, or that if you look into the original
               | motivations you will change your proposal. Think of it as
               | insurance against the possibility that every once in a
               | while, you will discover a requirement that needs to be
               | addressed with your suggested change.
               | 
               | I don't see where you're coming from that quoting
               | Chesterton's Fence is even "criticism." It's a suggestion
               | to take out a little insurance by doing a little
               | homework.
        
               | chaps wrote:
               | We can still say it was a bone-headed choice to do that
               | because it opened a massive back door to every user's
               | system. And? What is the problem with looking into why
               | they made that choice before arguing that the choice
               | should be reversed with maximum prejudice?
               | 
               | Because I was suggesting a different method for the
               | specific task at-hand.                 I don't see where
               | you're coming from that quoting Chesterton's Fence is
               | even "criticism." It's a suggestion to take out a little
               | insurance by doing a little homework.
               | 
               | Every time I've heard someone quote Chesterton's Fence,
               | it's always been as a means to halt the conversation.
               | Essentially, "shutup" -- an indirect critique of critique
               | itself in dismissive form. There's possibly some meta
               | point here about you not knowing the full circumstances
               | of the situation to warrant bringing up Chesterton's
               | fence.
        
               | a4isms wrote:
               | > Every time I've heard someone quote Chesterton's Fence,
               | it's always been as a means to halt the conversation.
               | 
               | From here forward you can say that _almost_ every time
               | you 've heard someone quote Chesterton's Fence, it's
               | _almost_ always been as a means to halt the conversation.
               | 
               | Today, you've encountered a counter-example, and a very
               | firm counter-example, at that. To my mind, Chesterton's
               | Fence is explicitly NOT about shutting down a
               | conversation. It's an invitation to continue the
               | conversation with more information to validate your
               | suggested course of action.
               | 
               | No different than if an engineer suggests, "We should
               | rewrite this code to be faster." What team lead or
               | product manager wouldn't ask, "Is this a bottleneck? Have
               | you profiled it? Do we know there are users impacted by
               | this code's performance?"
               | 
               | Or if someone suggests building a bespoke feature flag
               | service. "Have you done a build vs. buy analysis? What
               | alternatives have you considered before choosing this
               | design? Are there any OSS solutions that are close enough
               | to our requirements?"
               | 
               | These kinds of responses shouldn't be uttered as a way of
               | shutting down a conversation. If that's someone's intent,
               | they are abusing their privilege.
               | 
               | The right way to use any of these patterns is to say them
               | in good faith, and then socialize amongst the team the
               | standard of preparation the team expects of someone
               | proposing a non-trivial change.
               | 
               | Over time, the need to say such things decreases because
               | the team internalizes what
               | preparation/rigor/justification is needed for proposing
               | changes, and does the work ahead of suggesting changes.
               | 
               | Whereas, if the tone and intent is to block change, the
               | team goes down a toxic path where people are discouraged
               | from suggesting improvements of any kind. If that's what
               | you've encountered, you have my sympathy and I can
               | complete understand why you might be wary of people
               | quoting Chesterton's Fence.
        
               | chaps wrote:
               | These kinds of responses shouldn't be uttered as a way of
               | shutting down a conversation. If that's someone's intent,
               | they are abusing their privilege.
               | 
               | To be perfectly honest, your quote came across in that
               | spirit. Anywho, peace!
        
               | aidenn0 wrote:
               | I can think of a dozen reasons why such an API might
               | exist. Chesterton's fence doesn't say "don't make
               | changes" it says "If you can't think of why something
               | might exist, you are not qualified to say that it
               | shouldn't exist."
        
               | chaps wrote:
               | I never said I didn't understand why the API existed.
        
               | aidenn0 wrote:
               | Then Chesterton's fence doesn't apply. People wrongly
               | using arguments to enforce blind conservatism doesn't
               | make the underlying arguments wrong...
        
               | chaps wrote:
               | Then we agree!!
        
               | justin_oaks wrote:
               | I agree. Chesterton's fence doesn't mean that if you
               | don't know why the fence is there, don't ever move it,
               | under any conditions. It means try to find out why it's
               | there before moving it.
               | 
               | In many cases in my career, I've seen code that doesn't
               | make sense or seems like a bad idea. The person who could
               | explain why it's there has long left the company. Am I
               | afraid and leave the screwy stuff there, while citing
               | Chesteron's fence? Hell no. I'll change it to do the
               | right thing. This results in either exposing the reason
               | why it's there, or showing that it really was
               | unnecessary/bad. If something breaks from the change then
               | it's good that I can finally document what wasn't
               | documented before. So either way it's a win.
        
               | Ma8ee wrote:
               | The problem is that the historical context of a decision
               | often becomes a defence of the status quo, even when most
               | people understand it is bad.
        
             | Buttons840 wrote:
             | > There's often a wise tradeoff between criticizing systems
             | you've just seen after being at the company 5 minutes and
             | waiting 6 months to understand the context.
             | 
             | I know this is reasonable advice, but it makes me deeply
             | cynical. After 6 months I will have learned to live in the
             | shit (to use your term), and so it still seems like I have
             | nothing to gain by speaking up or trying to fix things. A
             | culture that accepts shitty code probably isn't supper
             | demanding for an experienced developer who is accustom to
             | the mess, so I'll just coast through my time and hop jobs
             | after a few years.
             | 
             | If nobody wants to respectfully talk about my criticisms on
             | day one, then they wont really want to at 6 months either.
             | In the end I'm lead to believe I should have zero concern
             | for code quality and only worry about my personal
             | reputation.
        
               | AA-BA-94-2A-56 wrote:
               | This is exactly where I am at right now. I have a million
               | dollar mortgage and interest rates are going up. I'm
               | keeping my head down and perpetuating technical debt
               | until I can hop jobs for a higher salary.
               | 
               | Criticising the status quo is not a winning move for me,
               | especially when it's lead to the company's engineering
               | team tripling in size. If I'm asked, I'll pick some low
               | hanging fruit- remove reliance on legacy/redundant
               | JavaScript libraries such as jQuery, and spent time
               | writing better unit tests. But so far I haven't been
               | asked.
        
             | chaps wrote:
             | You (in fashion of the article) missed the point I made: I
             | was explicitly asked to speak up as the new employee and
             | when I did I was told to stop speaking up. When I brought
             | it up in the meeting I gave my two week notice, he admitted
             | to saying that and apologized.
        
               | dec0dedab0de wrote:
               | That's some bullshit. you did the right thing. When a new
               | employee joins and notices that something sucks the
               | answer should be something like these:
               | 
               | We know, but haven't had time to fix it, maybe we'll
               | assign that to you when you're caught up.
               | 
               | We didn't think of it that way, good catch, lets go into
               | detail later.
               | 
               | Yeah, but doing it this way makes this other thing
               | easier, we'll show you that when you're ready.
               | 
               | or even: I don't know, my brain is fried with this
               | project, can you ask again in a few months?
        
             | whats_a_quasar wrote:
             | I get what you're saying, but this is exactly how deviances
             | are normalized. When you've been with the company for a
             | long time and are familiar with the history, it's easy to
             | rationalize why things are the way they are and that they
             | can't be improved. You can explain something that's crazy
             | with context and historical pressure.
             | 
             | Dan's point is that sometimes the new person's judgement is
             | correct, and there actually is a real problem that's
             | invisible to people who have been with the project a long
             | time. But the new person's judgment is basically always
             | ignored, and that's a mistake - it ought to be weighted
             | heavily because they legitimately have a perspective that
             | insiders no longer have.
             | 
             | If instead you spend six months trying to understand the
             | context:
             | 
             | "new person joins
             | 
             | new person: WTF WTF WTF WTF WTF
             | 
             | old hands: yeah we know we're concerned about it
             | 
             | new person: WTF WTF wTF wtf wtf w...
             | 
             | new person gets used to it
             | 
             | new person #2 joins
             | 
             | new person #2: WTF WTF WTF WTF
             | 
             | new person: yeah we know. we're concerned about it."
             | 
             | I'm sympathetic because my first company was a mid-stage
             | startup with huge structural problems in the engineering
             | org structure and processes. When I joined I had frequent
             | "WTF" moments and had a similar experience where
             | experienced people would explain to me why things are the
             | way they are. So I trusted them, and put my head down, but
             | eventually got frustrated and left. A few months later the
             | company went bankrupt because they couldn't build product
             | fast enough, investors lost patience, and they couldn't
             | raise another round.
        
               | Buttons840 wrote:
               | > the new person's judgment is basically always ignored,
               | and that's a mistake
               | 
               | Remember, the new person has something that nobody else
               | on the team can ever learn, no matter how much they study
               | or how long they work. The new person has a fresh
               | perspective.
        
           | AnIdiotOnTheNet wrote:
           | To be fair, you really shouldn't. You know nothing of the
           | constraints that people are operating under, or the political
           | or cultural landscape you're dealing with, so you just come
           | off like a preachy academic.
        
             | chaps wrote:
             | See the other comment I made: I was explicitly asked to
             | speak up.
        
               | AnIdiotOnTheNet wrote:
               | If that's the case, then yeah fair enough. If someone
               | doesn't want your opinion they shouldn't ask for it.
        
         | jkaptur wrote:
         | Just to give a different, concrete, perspective (and push a hot
         | button HN issue), I've spent a fair amount of time working on
         | extremely large web applications, and _by far_ the #1  "WTF WTF
         | WTF" thing that new hires say is "what do you mean you aren't
         | using $TODAYS_HOT_JS_FRAMEWORK??"
         | 
         | Once you get away from "should we use version control" and into
         | actually difficult software engineering questions, it's not
         | clear how to balance a fresh perspective vs. an experienced
         | (normalized? tainted?) view. I wish the article went into this
         | more.
         | 
         | Like, how does the new hire (or anyone else) know the
         | difference between "learning the complexity of the new system"
         | and "internalizing/normalizing the deviance of this culture"?
        
           | 908B64B197 wrote:
           | > by far the #1 "WTF WTF WTF" thing that new hires say is
           | "what do you mean you aren't using
           | $TODAYS_HOT_JS_FRAMEWORK??"
           | 
           | To that speaks of the caliber of programmers hired. If all
           | they have seen is $TODAYS_HOT_JS_FRAMEWORK and wrote nothing
           | but a web app using $TODAYS_HOT_JS_FRAMEWORK they might not
           | grasp the fundamentals that would make then realize that
           | frameworks are just abstractions (and not that different from
           | one another).
           | 
           | I don't think any software engineer would even ask that
           | question, since the answer will almost always be
           | "$TODAYS_HOT_JS_FRAMEWORK didn't exist when the project
           | started, and it's not worth a re-write to port it over".
           | 
           | Now, that brings out a second important truth: a company
           | can't attract and retain a wide range of different caliber
           | employees. For instance, if a place still questions the
           | usefulness of source control (perhaps because they consider
           | git to be too complicated) there's no way they'll attract and
           | retain top performers. So the culture will select people that
           | agree that source control is a waste of time.
        
           | flappyeagle wrote:
           | If "what do you mean you aren't using
           | $TODAYS_HOT_JS_FRAMEWORK" is the first question they ask,
           | then you can end their employment right then and there. Hire
           | people whose "wtf" you take seriously.
        
             | evilduck wrote:
             | Both parties would feel like they're dodging a bullet.
             | 
             | People still griping about $TODAYS_HOT_JS_FRAMEWORK are
             | clearly out of touch in 2023. It was funny commentary in
             | 2013. It still rang a little true until around 2016. Now
             | it's just an indicator you are the one not to be taken
             | seriously.
             | 
             | It's React, Vue, or Angular and it's been that way for many
             | years now.
        
               | notduncansmith wrote:
               | React underlies a lot of the "new hotness" like Next.js
               | and Remix.run (I'm not sure if Vue, Angular, or Svelte
               | have equivalents but I wouldn't be surprised).
        
           | aidenn0 wrote:
           | > Like, how does the new hire (or anyone else) know the
           | difference between "learning the complexity of the new
           | system" and "internalizing/normalizing the deviance of this
           | culture"?
           | 
           | If a new hire can't checkout, build, and test the software on
           | the first day, then there is likely something either wrong
           | with the hire or the infrastructure. A sufficiently old and
           | arcane software system might take weeks before a new hire can
           | make even a simple change, but that shouldn't impact those
           | three items.
        
             | Jtsummers wrote:
             | Along with this "how long to spin up the new hire" issue,
             | one of my first (if not the first) questions when trying to
             | help people improve their processes related to software is:
             | 
             | > If the user/client asks you to make a small but not
             | _trivial_ change, how long would it take to update and
             | deploy the program?
             | 
             | I have had answers ranging from "A couple hours" to "A
             | year" (yes, they were serious). Most were in the 1-3 month
             | range, though, which is pretty bad for a small change. It
             | also makes it apparent why a bunch of changes get batched
             | together whether reasonable or not. If a single small
             | change, single large change, or collection of variable
             | sized changes all take a few months to happen, might as
             | well batch them all up. It becomes the norm for the team.
             | "Of course it takes 3 months to change the order of items
             | in a menu. Why would it ever be faster than that?"
        
           | ReflectedImage wrote:
           | That's not a "WTF". All front end developers trash their
           | predecessors work and rewrite in the last framework. That
           | just what they do.
        
           | jt2190 wrote:
           | Yes. From the article:
           | 
           | > "The thing that's really insidious here is that [once a
           | person buys] into the WTF idea... they can spread it
           | elsewhere for the duration of their career... Once people get
           | convinced that some deviation is normal, they often get
           | really invested in the idea."
           | 
           | > [H]ow does the new hire (or anyone else) know the
           | difference between "learning the complexity of the new
           | system" and "internalizing/normalizing the deviance of this
           | culture"?
           | 
           | The article implies that the new hire should pay close
           | attention to the things that are incentivized, and those that
           | are not.
        
           | whats_a_quasar wrote:
           | There's a whole section about how to balance a fresh
           | perspective, in "solutions". The best way is for an effective
           | VP to hear the WTFs of new hires, apply engineering
           | judgement, and make changes based on that signal. If
           | management is not the ones creating the deviance, they ought
           | to be able to tell what reactions are just unfamiliarity with
           | the system and what are signs of something actually being
           | broken. The article is arguing that most people ignore those
           | weak signals by default, and ought to pay more attention to
           | them, not that they're always reliable.
        
           | josephg wrote:
           | > it's not clear how to balance a fresh perspective vs. an
           | experienced (normalized? tainted?) view.
           | 
           | Having a healthy, balanced view comes from enough
           | experiences. Ideally from working in a bunch of places and
           | seeing enough things go sideways, and correctly understanding
           | and identifying the causal chain that led to failures.
           | 
           | Funnily enough its sort of like training an AI - you
           | essentially need a lot of correctly labelled data to learn.
           | Junior engineers don't have enough data points, and
           | unfortunately some "senior engineers" I've worked with took
           | (in my opinion) the wrong lessons from their experiences. (Eg
           | the CTO who thinks version control is too complex.)
           | 
           | The interesting cases are when smart, experienced people
           | disagree on what the best solution is. Should you keep your
           | team small and smart or have a varied team with more
           | mentorship and process? Is code review worth it in every
           | case? What is the right amount of tests for your software?
           | How often do we want to push to production?
           | 
           | When I was teaching programming my students would sometimes
           | ask juicy questions. My favorites were the questions I could
           | answer with "I'll tell you my opinion, but I've worked with
           | people I look up to who think I'm wrong about this..."
        
         | rr888 wrote:
         | The problem is when you join a high performing team and
         | organization. Is a WTF something they should fix your you need
         | to recalibrate what you think is normal?
        
         | GartzenDeHaes wrote:
         | In my experience, you cannot change an organization's culture
         | with rules, mission statements, listing values, or giving
         | speeches. They only way is to take down the old culture
         | bearers. Some of them may be managers, but more often they are
         | employees who have gained some organizational power. You'll
         | often find them at the center of sticky organizational spider
         | webs with approval processes such as purchasing and service
         | administrators.
         | 
         | To change the culture, these people have to go. Firing them may
         | not be feasible, but there are other options. Dethroning them
         | in the form of a promotion or even just physically moving them
         | can be effective. When people don't have to jump through their
         | hoops anymore, they lose their organizational power.
        
           | Ensorceled wrote:
           | > They only way is to take down the old culture bearers. ...
           | To change the culture, these people have to go.
           | 
           | I was briefly head of engineering at a company that had
           | several "old culture bearers" that made change impossible. I
           | was something like the 3rd or 4th engineering leader over the
           | space of a year. Apparently the person after me was actually
           | allowed to fire a few of these people and was able to turn
           | things around.
        
             | PaulHoule wrote:
             | More than once I had to leave in order for an employer to
             | take action against a problem employee, including my boss
             | once. In these cases the employer suddenly realized that it
             | made no sense to attempt to replace me without removing the
             | reason that drove me out.
        
               | Ensorceled wrote:
               | That is, unfortunately, very common. Though sometimes the
               | toxic people drive the company into the ground first.
        
         | e_i_pi_2 wrote:
         | This is something we actively try to take advantage of at my
         | company - we know that we've grown comfortable with
         | architecture that may not make a lot of intuitive sense, so
         | when we have new people join we try to make a list of confusing
         | concepts so we can try to clean them up. In the ideal case the
         | new person is able to do the cleanup, so we get a more
         | intuitive design and they learn the surrounding architecture
         | more along the way
        
       | AlbertCory wrote:
       | This guy needs to organize & format his writing better, since he
       | does have really interesting things to say.
        
         | albertop wrote:
         | [flagged]
        
           | justin_oaks wrote:
           | Probably being downvoted because the HN guidelines explicitly
           | say to not comment about such things:
           | 
           | "Please don't complain about tangential annoyances--e.g.
           | article or website formats, name collisions, or back-button
           | breakage. They're too common to be interesting."
           | 
           | https://news.ycombinator.com/newsguidelines.html
        
             | AlbertCory wrote:
             | As a matter of fact, it's in positive territory right now.
             | 
             | No, it's not a tangential annoyance. There are times when
             | form IS function.
        
           | AlbertCory wrote:
           | It's really pretty easy, and free, to use the Substack editor
           | to create a nicely formatted article. If you care at all
           | about your readers.
        
             | mixmastamyk wrote:
             | Talk about killing a fly with a sledgehammer. Simply needs
             | one or two lines of css, max-width and possibly font-size.
        
               | AlbertCory wrote:
               | I had two words there: "organize" and "format." This
               | addresses only the latter.
        
           | msm_ wrote:
           | Maybe because it's an absolute statement ("this person needs
           | to organise & format his writing better") for a something
           | mostly subjective.
           | 
           | In my opinion the website is readable, fast, lightweight, not
           | distracting and pleasant to read. It's also accessible for
           | people with disabilities, responsive, and works everywhere. I
           | understand that not everyone is as into minimalism as me,
           | just pointing out the problem in "this person needs (...)".
        
         | dang wrote:
         | " _Please don 't complain about tangential annoyances--e.g.
         | article or website formats, name collisions, or back-button
         | breakage. They're too common to be interesting._"
         | 
         | https://news.ycombinator.com/newsguidelines.html
        
         | burnished wrote:
         | This is one of the few sites I find immediately and highly
         | readable (on mobile). Does it not wrap lines on larger
         | displays?
        
         | sebstefan wrote:
         | Firefox has the "reader view" option toggleable with F9 for
         | when you stumble upon unreadable designs, if you want
        
           | jwilk wrote:
           | It's F9 on Windows, command-option-r on macOS, and ctrl-alt-r
           | elsewhere.
           | 
           | https://support.mozilla.org/en-US/kb/firefox-reader-view-
           | clu...
        
         | xnorswap wrote:
         | If you don't have a reader view in your browser, just paste
         | this into your global CSS:                   p          {
         | line-height: 1.7;           max-width: 60em;           font-
         | size: 1.2em;           margin-left: 5em;         }
         | 
         | It pretty much fixes the default readability which is
         | essentially zero on this site otherwise.
        
           | ryandrake wrote:
           | Yuck! I hate it when sites monkey around with max-width. I've
           | got a nice 27 inch monitor. I want to use all of it. It's
           | refreshing to see a site that _doesn 't_ insist on second-
           | guessing the width that I set my browser window.
        
             | atoav wrote:
             | So you look like you watch a tennis game when you read
             | text? Funny.
             | 
             | On a more serious note: A maximum length for text makes
             | sense ergonomically, which is why especially big prints
             | like newspapers or magazines work with columns. Columns
             | however haven't really cought on in the web, because they
             | do not combine very well with the whole scrolling thing.
        
             | mixmastamyk wrote:
             | Too hard to read. See newspapers. Multiple windows are a
             | thing.
        
               | ryandrake wrote:
               | So, leave the choice to the user. If I want to read text
               | in a small column, I can easily resize my window. Don't
               | try to force one way.
        
           | Sesse__ wrote:
           | Dan Luu talks a bit about this here:
           | https://twitter.com/danluu/status/1115707741102727168
        
             | AlbertCory wrote:
             | yes, if you ignore a millennium's worth of publishing
             | wisdom and think the world of text began anew in 1995, then
             | maybe you can claim that "the science is not settled."
             | 
             | He has every right to make his choice. As do we, in
             | deciding whether to read it.
        
             | robocat wrote:
             | Great thread!
             | 
             | Designer says: my opinion is right because A. Dan shows: A
             | is not factual. Designer says: my opinion is right because
             | of B.
             | 
             | Dan notices behaviours, and then he writes compellingly
             | about those behaviours.
        
             | RoddaWallPro wrote:
             | One of the more hilarious takes I've seen. "There are no
             | papers for this, and I choose to disregard the countless
             | number of people who say it is much easier for them to read
             | if the line lengths are constrained as they are in a book
             | or scroll or every other form of human writing ever put on
             | this earth, so I will not make my site easier to read. F
             | you."
        
               | whats_a_quasar wrote:
               | I really don't think that's what he's saying. You are
               | assuming a great deal of malice, rather than positive
               | intent. What he's saying is that there isn't hard
               | evidence that shorter lines are more readable, so he made
               | the style choice of longer lines. You're claiming that
               | most people prefer shorter line widths, but again present
               | no evidence that most people actually have that
               | preference, other than vague references to "countless
               | people". I actually think you're probably right, and if
               | you had data Dan might update his stylesheet. But in the
               | absence of evidence, you're just presenting your opinion
               | as fact, and assuming malice.
        
               | RoddaWallPro wrote:
               | Yeah, that's probably true. He also at least allows
               | people to set their own reading width by adjusting
               | browser.
               | 
               | My frustration stems from the fact that I find the
               | argument "there are no papers with sufficient evidence"
               | to be pedantic bullshit. Like yeah, sure, you aren't even
               | wrong, but absence of evidence is not evidence of
               | absence. I've never seen anyone claim to like 180 char
               | lines, whereas I've seen hordes of people who say it is
               | very difficult for them to read that line length, and
               | prefer something book-sized (lengthed?).
        
               | whats_a_quasar wrote:
               | Hmm, yeah I actually agree with that way of putting it.
               | The evidence that does exist are the anecdotes, and he
               | seems to ignore that evidence. Many of his readers,
               | myself included, seem to prefer fixed line lengths. So it
               | is a weird choice.
               | 
               | I was mostly reacting to the assumed malice in the parent
               | comment. Based on his blogging style, I think it's more
               | likely that Dan's just a pedantic guy implementing his
               | personal preferences on his personal blog :)
        
               | AlbertCory wrote:
               | "Not even wrong" FTW. There's even a blog with that
               | title:
               | 
               | https://www.math.columbia.edu/~woit/wordpress/
        
               | travisjungroth wrote:
               | This is why it's nice to get people's preferences vs
               | thoughts about preferences. Preferences aggregate,
               | thoughts about preferences do not. If one designer says
               | "Readers like narrow columns more, everyone knows that."
               | and another says "I like reading narrow columns.", I'd
               | give the person speaking about readers in general more
               | weight (even with them going the same direction). But, if
               | 100 designers spoke for all readers and 100 designers
               | spoke for themselves, I'm giving more weight to the
               | second _group_. Hearing 100 preferences is more valuable
               | than hearing one idea, 100 times.
        
           | sebstefan wrote:
           | I don't like fucking with the default CSS, my alternative is
           | this bookmark that injects Javascript in the page and
           | temporarily fixes the formatting at one click of a button
           | javascript:(function(){ var bod =
           | document.getElementsByTagName("body")[0]; bod.style.margin =
           | "40px auto"; bod.style.maxWidth = "40vw";
           | bod.style.lineHeight = "1.6"; bod.style.fontSize = "18px";
           | bod.style.color = "#444"; bod.style.padding = "0 10px"; })();
           | 
           | And for a dark mode:
           | javascript:(function(){ var body =
           | document.getElementsByTagName("body"); var html =
           | document.getElementsByTagName("html"); var img =
           | document.getElementsByTagName("img");
           | body[0].style.background = "#131313"; body[0].style.opacity =
           | "1.0"; html[0].style.filter = "brightness(115%) contrast(95%)
           | invert(1) hue-rotate(180deg)"; img[0].style.filter =
           | "contrast(95%) invert(1)"; })();
        
         | pasquinelli wrote:
         | that site is better than most, imho.
        
       | [deleted]
        
       | t3estabc wrote:
       | [dead]
        
       | bluedino wrote:
       | Write total shit for code, then look like a 'genius' for 'fixing'
       | bugs, only to have them come back again in the future (further
       | looking like a clown to the rest of the team)
        
       | irsagent wrote:
       | "rictus of horror" - What a set of words to describe a response.
        
       | kurthr wrote:
       | There's been a lot of work on reliability of complex systems and
       | how they operate. What has been found is that it is almost always
       | necessary to have failure (degraded operation) modes that prevent
       | system failure, and the more complex and more hazardous failure
       | is the more modes develop.
       | 
       | In these systems it is found that they are almost always
       | operating (or transitioning between) failure modes. Often
       | multiple operational failure modes are simultaneous. It becomes
       | very important to test the system in each of it's failure modes
       | and their combinations to maintain high up time.
       | 
       | https://how.complexsystems.fail/ is an example, but there are
       | many.
       | 
       | Human work, development, and maintenance is itself a system that
       | interacts with these critical systems. Frankly, failure to fail
       | causes failure (thus chaos monkey). The mythical man month is
       | almost a sub category of these failures as are HR hiring
       | processes and other BS. Being too successful and not having
       | competition (or similarly sclerotic competition) can be as much
       | of a hazard as "move fast, break things".
        
       | justin_oaks wrote:
       | I welcome others to share stories of the normalization of
       | deviance in their companies.
       | 
       | One company I worked had no unit tests, no infrastructure as
       | code, and no build server. This held strong for a while until
       | enough developers implemented some unit tests, infrastructure as
       | code (e.g. terraform), and a build server as skunkworks projects.
       | Eventually management tolerated them, but never endorsed them.
       | Some teams at the company still never embraced good practices
       | because it wasn't forced on them.
       | 
       | I guess I've never worked at a company that valued unit tests
       | across the whole of the engineering team. I introduced them and
       | implemented them on my own team, but others ignored it.
        
         | ctroein89 wrote:
         | > and no build server
         | 
         | Personal experience is that a build server normalizes deviance.
         | "But it works on the build server" we used to say, as, with
         | time, it become harder and harder to build locally. "Just fix
         | your environment!" we used to say, when it was the build system
         | that was actually at fault. "It's all so fragile, just copy
         | what we've done before!" we then said, repeating the mistakes
         | that made the build system so fragile.
         | 
         | Eventually, the build system moved into a Docker image, where
         | the smells where contained. But I'm still trying to refactor
         | the build system to a portable, modern alternative. If we
         | hadn't had a build server, we'd have fixed these core issues
         | earlier and wouldn't have built on such a bad foundation. Devs
         | should be building systems that work locally: the heterogeneity
         | forces better error handling, the limited resources forces
         | designing better scaleability, and most importantly, it
         | prevents "but it works on the build server!".
        
       | aj7 wrote:
       | "Let's say you notice that your company has a problem that I've
       | heard people at most companies complain about: people get
       | promoted for heroism and putting out fires, not for preventing
       | fires."
       | 
       | My first day at work at big-laser-company. Manufacturing engineer
       | for a laser (then) so complex, it required a PhD to solve
       | problems to get units out the door. The product was a ring laser.
       | What that means is that the laser beam travels around in a race
       | track pattern inside the laser before getting out, not a back-
       | and-forth bouncing between two mirrors. Now this laser could be
       | tuned to any wavelength by suitable setups and machinations, and
       | once there, would "scan" a small amount about this wavelength,
       | enabling scientists to study tiny spectral features in atoms and
       | molecules with great precision. I knew all this shit. I was a
       | Berkeley-trained physicist that built precision lasers out of
       | scrap metal for my thesis. First day of work. I walk into the
       | final test lab. The big laser was happily scanning away. The
       | bright yellow needle-like output beam was permitted to hit the
       | lab wall. As the laser scanned, the beam was MOVING on the wall.
       | Whereupon, first day of work, I exclaimed the most obscene four
       | words in manufacturing, for all to hear, "You can't ship that!"
       | ("Beam pointing instability" is detrimental to almost any laser
       | application. It turns out that during scanning, an optical
       | element was rotating, on a shaft, inside this laser. This
       | mechanical motion caused beam motion.") Well, I got an immediate
       | reputation as a negative guy. (You can tell it's deserved.) The
       | solution was to retrofit 28 lasers in the field, mostly in
       | Europe, with a component that cancelled the movement, on an
       | expensive junket by a service guy. Who was hailed as a "hero."
        
         | KMag wrote:
         | Interesting. I had an internship at a company that did inertial
         | navigation, mostly for defence applications. I only knew of
         | ring lasers for use in gyroscopes. (Send a laser around a loop
         | wave guide/fiberoptic, and any translational acceleration
         | cancels out going out and back, but any acceleration in
         | rotational velocity in the plane of the ring/rotation vector
         | perpendicular to the ring shows up as a Dopler shift. Tune the
         | laser to have a standing wave, and rotational acceleration
         | shifts the nodes of the standing wave around the ring.)
         | 
         | I had a colleague who got called up when a Trident missile MIRV
         | bus fell off a forklift and he had to do simulations to tell
         | the Navy if it was still good or needed to be brought back in
         | for rework/recalibration. My understanding is that either the
         | MRIV bus itself or its container has integral devices that
         | record peak 3-axis acceleration for just such a scenario. I
         | imagine they're as simple as a few precise weights on a few
         | wires with precise failure strains, so you can bracket the peak
         | acceleration by which wires broke and which survived.
         | 
         | On the one hand, it's great to have more accurate nukes, which
         | allow lower yields, smaller stockpiles, and presumably smaller
         | craters if everything goes sideways. On the other hand,
         | "surgical" nukes result in it more likely that one side will
         | use them and gamble that the other side won't massively
         | retaliate.
        
           | XorNot wrote:
           | You could look at it a different way: a more accurate nuke
           | means a nuke that's targeted at military facilities and not
           | sized 10x larger and aimed at "everything around that city
           | over there".
           | 
           |  _If_ it was ever used, that work saves lives.
        
             | KMag wrote:
             | More importantly, I think more accurate nukes along with
             | good satellite multispectral and signals intelligence means
             | that top generals carrying out orders for nuclear first
             | strikes can be more certain that they're signing their own
             | death warrants. Hopefully this results in any leader
             | ordering a nuclear first strike getting deposed by military
             | coup rather than starting a nuclear war.
        
               | tablespoon wrote:
               | > More importantly, I think more accurate nukes along
               | with good satellite multispectral and signals
               | intelligence means that top generals carrying out orders
               | for nuclear first strikes can be more certain that
               | they're signing their own death warrants.
               | 
               | How would you do that? In the event of a nuclear war, my
               | understanding is they'll mostly be flying around on
               | special command and control planes. I don't think nuclear
               | intercontinental SAMs are a thing. I'm not even sure if
               | they could even be possible (wouldn't they need active
               | guidance, which would be very hard on reentry).
        
             | throwbadubadu wrote:
             | Please just don't do that. (Or missed irony?)
        
         | geenew wrote:
         | I once encountered a situation with a very expensive field
         | laser. At one point, measurements started showing an increasing
         | amount of offset.
         | 
         | Over a period of days, the error became increasingly, comically
         | bad, until finally the system refused to boot.
         | 
         | A technician was called, and after hearing about the behaviour,
         | the first request was that a photo of the laser light exit port
         | be taken.
         | 
         | It was obvious why it wouldn't boot: a mirror in the light path
         | had fallen off.
         | 
         | The worst part was, the mirror had been held on by glue, and
         | had been slowly slipping out of place. The hot climate was
         | probably a factor.
         | 
         | They really should have had someone to say 'you can't ship
         | that' when the topic of glue to hold mirrors came up.
        
           | pm_me_your_quan wrote:
           | This exact problem happened with an optic in my lab in
           | graduate school. For two years the senior grad student and
           | postdoc blamed each other over the entire apparatus becoming
           | misaligned every couple of days. (It was a really toxic
           | environment.) Eventually, they both left, I was the only one
           | there, and it still became misaligned. In one day I tracked
           | it down to a prism from Thorlabs whose glue had gone bad
           | positioned at the very beginning of the laser line- it was
           | sliding in its mount.
           | 
           | I wish I had pushed more strongly about it. We spent probably
           | a full person-day of work every week on that.
        
             | whats_a_quasar wrote:
             | Oh man, this one hurts to read a little bit. It's crazy how
             | people cooperating poorly can eat up that much working
             | effort.
        
       | renewiltord wrote:
       | In _The Field Guide to Human Error Investigations_ by Sidney
       | Dekker, he quotes someone else saying something like:
       | 
       | > _Everything that can go wrong will go right._
       | 
       | Murphy's Law then manifests from escaping disaster through
       | repeated iterations of taking risks where most things play out
       | well anyway.
       | 
       | I have to laugh at the "append z to the end" strat at Google,
       | though. That's a good one.
        
       ___________________________________________________________________
       (page generated 2023-02-14 23:00 UTC)