[HN Gopher] Normalization of Deviance (2015) ___________________________________________________________________ Normalization of Deviance (2015) Author : akshaykarthik Score : 282 points Date : 2023-02-14 15:52 UTC (7 hours ago) (HTM) web link (danluu.com) (TXT) w3m dump (danluu.com) | kerblang wrote: | Great stuff - I think this goes in the "required reading" list. | | The tech industry tends to revolve around "I'm a super-rational | robotic genius" thinking that can't accept the existence of its | own irrational tendencies, to the point that it becomes | ridiculous. | [deleted] | SideburnsOfDoom wrote: | The standard "Required reading" text on the subject is "The | Field Guide to Understanding 'Human Error'" By Sidney Dekker | goostavos wrote: | Great book! (Although, it could have used a more aggressive | editor) | | Reading it felt like a personal attack in many places. | However, reading it forever changed how I think about things. | It's a much more useful framing for everyone involved if you | start with the question of "why did they think this was the | right thing to do?" as opposed to "this person made a bad | choice / mistake". My (extraordinary) impatience naturally | predisposes me towards the latter, but the core argument of | the book is that that's _lazy_ -- you can hand wave away | anything and everything with "operator error". | nilespotter wrote: | [flagged] | badrabbit wrote: | It is not.click. | dang wrote: | Can you please not post unsubstantive comments? It looks like | you've been doing it repeatedly, and we're trying for something | else here. | | https://news.ycombinator.com/newsguidelines.html | dec0dedab0de wrote: | I'm still reading, but I just got to the part about flaky, and I | got annoyed because there are clear use cases for flaky or | pytest-retry. | | If you have an integration test that relies on an unreliable | system you do not control. Sure you can mock it out for a unit | test, but if you want to make sure you catch breaking API | changes, you need to hit the actual system. And if it works after | retrying it a few times, then so be it. no need to throw shade. | JackFr wrote: | I'm going to push back and say that test is not a valuable | automatic test. The phrase "relies on an unreliable system" | captures that lack of value. | dec0dedab0de wrote: | When the code your testing is a client for some remote API, | and the sandbox/development/Testing version of that API | doesn't have the same resources and uptime guarantee as | production, then what are your options? as far as I can tell | they are: | | Don't test it. | | Only do unit tests with the connection mocked out. | | Test against production. | | Try it a few times with a delay, and if it works then you | know your code is good and you can move on with your | deployment. Which is what flaky and pytest-retry do. | | Maybe I'm missing something, but out of those 4 options | retrying the test seems like the best one, with the big | caveat that it is only viable if the test does indeed work | after trying a few times. I really don't see any downside. | | edit: | | Maybe another option is to put the retry functionality | directly in the client code, which would make your code more | robust overall. but that is definitely more complex than | using one of these libraries just for testing. | salawat wrote: | You're on the right track. It's a perennial favorite of | devs to abhor flakyness, whereas after spending enough time | as a tester, you come to terms with the fact that you have | to take your tests as a statistical probe because most | places test systems are simply not that reliable; | sometimes, this is even a design feature. | | This experience as a tester is in fact a normalization of | deviance from the ideal computation model of a developer. | Everything should work the first time everytime from their | point of view. The tester sees reality as it is. The | Emperor won't fund my test systems sufficiently to service | all my customers, so we make do ss best we can. Bonus | points in that we get to exercise the edge cases. | Aeolun wrote: | I think the issue is people using it when they're too lazy to | fix the test case. | namaria wrote: | I find quite interesting that people will prefer a highly | malleable language like Python, and then orgs have to adopt | testing to get around all the inconsistencies caused by | absent type system. And then people will write libraries to | get around the pesky tests to get their flexibility back. | | It's fascinating really... Complex systems are always in | partial failure mode and that applies to collective | optimization challenges. Organizations will always be stuck | in local optima in most domains. | dec0dedab0de wrote: | Type systems do not replace testing, and if a test works | after retrying it then it is probably not something that a | type system would be able to catch. | somat wrote: | A thought experiment. | | When is it "Normalization of Deviance"? and when is it a | "Efficiency Optimization"? | | I mean, the difference is pretty clear after something has | failed, But very murky before. | jpollock wrote: | It is Efficiency Optimization when you know why the rule is | there, and having made an estimation of the risks, perform a | cost-benefit analysis. | | aka "Chesterton's Fence" | | Otherwise, it's "Normalization of Deviance": | | * The build is broken again? Force the submit. | | * Test failing? That's a flaky test, push to prod. | | * That alert always indicates that vendor X is having trouble, | silence it. | | Those are deviant behaviours, the system is warning you that | something is broken. By accepting that the signal/alert is | present but uninformative, we train people to ignore them. | | vs... | | * The build is always broken - Detect breakage cause and auto | rollback, or loosely couple the build so breakages don't | propagate. | | * Low-value test always failing? Delete it/rewrite it. | | * Alert always firing for vendor X? Slice vendor X out of that | alert and give them their own threshold. | manicennui wrote: | Unfortunately I don't find that most software engineers | understand the difference between actually determining costs | and benefits and choosing to make certain tradeoffs and | rationalizing whatever choice they already made. | jpollock wrote: | I think that's ok, For me, it's more about "change the | system, instead of ignoring it". | | Once you change the system (document/rules/alerts/etc), | then if it breaks, you change it again and learn the | lesson. Both are conscious decisions by the org. | dang wrote: | Related: | | _Normalization of Deviance (2015)_ - | https://news.ycombinator.com/item?id=22144330 - Jan 2020 (43 | comments) | | _Normalization of deviance in software: broken practices become | standard (2015)_ - https://news.ycombinator.com/item?id=15835870 | - Dec 2017 (27 comments) | | _How Completely Messed Up Practices Become Normal_ - | https://news.ycombinator.com/item?id=10811822 - Dec 2015 (252 | comments) | | _What We Can Learn From Aviation, Civil Engineering, Other | Safety-critical Fields_ - | https://news.ycombinator.com/item?id=10806063 - Dec 2015 (3 | comments) | deanCommie wrote: | Reality: It is true that EVERY organization is broken in some way | or another. | | You have to find the one that is broken in the way that is | tolerable to you. | | Arguably the closest we know to a panacea in terms of engineering | culture and best practices is Google. And what are they now known | for? An inability to ship anything meaningful anymore. Spinning | around in circles launching and re-launching new chat apps. | | These are not unrelated. High engineering standards are always in | tension with product delivery. As a security engineer once told | me, "the most secure system is the one that never gets launched | into production." | | So while Dan is right, and all the examples are right, and things | like non-broken builds and a fast CI/CD pipeline are totally | achievable, don't learn the WRONG lesson from this which is that | when you arrive to a company and notice a bunch of WTFs, the | first thing you must do is start fixing them in spite of any old | timers who say "Actually that's not as bad as it seems". | Sometimes they're wrong. USUALLY, they're right. | stevehawk wrote: | This is a big term in aviation, because in most cases in order | for something catastrophic to happen it requires a lot of things | to have failed. And one way to ensure that enough things fail is | to start deviating from your maintenance, inspections, or general | responsibilities. Related: the swiss cheese models. | kurthr wrote: | It's also why things in aviation are so fixed and difficult to | change. Not having any new civil aviation planes for 30 years | worked... how about 40, 50? When will it break? Well, when | someone develops an easy to build, easy to fly, inexpensive | experimental craft and zillions of people do it all at once | (hasn't happened yet). | | One of the more interesting things I've found is that a huge | number (easily a majority) of instructors are recently trained | flyers, because there is a pipeline to train them and they're | cheaper than using experienced pilots (esp for multi-engine and | more complex airplanes). They also know all the ins-outs of the | training and rule books (with recent changes) so they know how | to pass all the tests and how to teach that. Sooooo you have a | bunch of inexperienced pilots teaching all the new pilots... | there's likely a failure there, but it hasn't reared its head. | We still have a lot of ex-military folks around who didn't | learn that way. | | Who do you want flying when things go bad? People who have | spent many hours with things about to go bad (military, | emergency/fire, sail plane pilots) who have experience dealing | with it. Those people can also be fun/terrifying to fly with, | because they will take risks. | neilv wrote: | > _They also know all the ins-outs of the training and rule | books (with recent changes) so they know how to pass all the | tests and how to teach that. Sooooo you have a bunch of | inexperienced pilots teaching all the new pilots..._ | | Sounds like there's some similarities with everyone focusing | on Leetcode interviews, and then one generation of that | filtering and then mentoring the next, and repeat. | | The companies don't know what that's costing them, until | there's a problem that can't be ignored. | | In the case of software engineering (poorly studied, relative | to aviation) the company will generally never learn whether a | non-Leetcode&promo-focused team could've avoided the problems | in the first place, nor whether non-Leetcode experience | could've handled a problem that happened anyway. | sokoloff wrote: | > Not having any new civil aviation planes for 30 years | worked... | | The A220, A350, and A380 are all newer than 30 years old. | (A321 is barely younger than that and A330 barely older.) | Boeing has released the 777 and 787 in the last 30 years. The | Cirrus SR20 and SR22 are newer than that, as is the SF50 jet. | The Diamond DA40, DA42, and DA62 are newer. The Honda Jet is | newer. Cessna has a handful of business jets newer than that. | The Embraer Phenom 100 and 300 are newer than that. There are | variants of the CRJ newer than that (-700, -900, -1000). | | That's a lot of new civil aviation aircraft designs in the | last 30 years. | pja wrote: | > Those people can also be fun/terrifying to fly with, | because they will take risks. | | Risk homeostasis in action! | | https://soaringeconomist.com/2019/10/30/experience-can- | kill-... | martopix wrote: | > Have you ever mentioned something that seems totally normal to | you only to be greeted by surprise? | | Once some (foreigner) person was surprised at my dipping toast | with Nutella in my latte. I was equally surprised by his | surprise. | chaps wrote: | These stories ring so, so true. Once worked at a company whose | infrastructure issues were so deep and festering that after | fighting a fire, my boss told me, "If you go to the press about | this, the client will sue us and everyone who works here will | lose their jobs." | aliqot wrote: | That's what I never understood about this story; did you guys | have any suspicion it would dump radiation into the patient all | at once, or was this like a concurrency bug | whats_a_quasar wrote: | Huh? Are you assuming that the parent comment is about | someone programming a medical device? | lifeisstillgood wrote: | I'm think it's a meta joke - Therac-25 | (https://en.m.wikipedia.org/wiki/Therac-25) was a | radiotherapy machine from the early 1980s and is (in)famous | for having software failed and killed I think dozens of | people. It's become a well known case study, but it's | highly unlikely anyone on HN worked on it - I think that's | the joke. | [deleted] | chaps wrote: | We weren't able to reliably install security daemons on a | client's machine because the entire automation system didn't | account for autoscaling. The issues were raised well before I | joined and the project head legitimately didn't understand it | as a problem that needed solving. The hosts were for a | presidential candidate's webserver, and they noticed the | webservers were missing security daemons days before the | election. | DiggyJohnson wrote: | Maybe I'm missing a joke, but was your client HRC's | campaign?(?!) | strbean wrote: | Did HRC's campaign website get hacked? I know her | mailserver was hacked, but that was when she was | secretary of state, no? | aliqot wrote: | That's not important and this ain't the place to ask | otherwise they'd have told us. | iosono88 wrote: | [dead] | DiggyJohnson wrote: | > this ain't the place to ask | | Am I double-whooshing here? | | How is a Hacker News comment thread not the right place | to respectfully ask questions in response to interesting | comments. I know I'm not entitled to an answer, nor do I | intend to start a flame war. Sheesh | aliqot wrote: | It is personal information that risks identifying them | more than they already had at the time of posting. It | took about two seconds to put everything together. I | don't have a dog in this fight politically one way or the | other, people don't need to identify themselves IRL here. | DiggyJohnson wrote: | Who are you to decide what others are comfortable sharing | on here. It is quite literally as simple as the person I | replied to choosing not reply to my comment. Why is this | issue a concern to you? | | > I don't have a dog in this fight politically one way or | the other | | Neither do I. | | > people don't need to identify themselves IRL here | | I don't think they do either. Why are you assuming I | "needed" this information? | chaps wrote: | Can you not? They're right. | lmm wrote: | There's nothing respectful about asking something that | someone has very blatantly made a deliberate decision to | leave out of their post, for completely understandable | reasons. | aliqot wrote: | jeez thats a rough spot to be in. did you stick around to | fix it or just get the hell out of dodge after that? | chaps wrote: | I did what I could with a handful of selenium scripts, | then hit a road block because we didn't have ssh access | to a chunk of the autoscaling hosts. Gave up after that, | told the customer rep to tell them we can't do it, and | gave my two week notice about a month later. | aliqot wrote: | Ouch, that has to be rough to endure. I'm glad you seem | to be in a better place now. Good on you for doing the | right thing and getting the hell out of there when your | options ran out. | outworlder wrote: | > security daemons | | AKA compliance checkbox crap? | | If infrastructure is immutable (which makes it work even | better for autoscaling), nothing new will get installed | unless you build a new image. Export whatever data you | require to ensure things you want to be running are | running. Monitor entry and exit points. | | What is left for the "security deamons" to do? | riffic wrote: | are "security daemons" truly necessary though? | | this whole thing sounds like a troll with enough convincing | language to seem plausible | chaps wrote: | We could debate security daemons until our minds bleed, | but.. man, I wish that all didn't happen. | Logans_Run wrote: | I'm not sure if this link has already been posted but have a look | at _How I Almost Destroyed a PS50 million War Plane and The | Normalisation of Deviance._ | | https://www.fastjetperformance.com/blog/how-i-almost-destroy... | superpope99 wrote: | Has Dan Luu ever explained why he doesn't put dates in his blog | posts? | Jtsummers wrote: | Most of his posts seem to be date-independent. To the extent | that it matters, you can check the homepage: | https://danluu.com/. There you will find month and year of the | posts. | aeturnum wrote: | If you enjoyed this - I highly recommend watching Adam Curtis' | _Can 't Get You Out of My Head_: https://thoughtmaybe.com/cant- | get-you-out-of-my-head/ | pizzaknife wrote: | i routunely remind everyone,"we're all mercenaries." | | i have marginal control over who i manage. The Product isnt | saving the world, but it is allowing us to live reasonably and | with a clear soul at the end of the sprint. The reason i say the | "mercenary" bit is simple: weigh your dreams against blood and | gold and compromise. | shadytrees wrote: | formatted for wide monitors https://ddanluu.com/wat | PaulHoule wrote: | I like the bit about Let's look at how the first | one of these, "pay attention to weak signals", interacts | with a single example, the "WTF WTF WTF" a new person gives off | when the join the company. | | and kinda wonder if a company that prioritized not getting this | reaction from new hires might find it is the most impactful thing | they can do in terms of culture. | lo_zamoyski wrote: | That doesn't seem like a healthy standard b/c it grounds | decision making in appearances rather than principles and | prudential judgements. Certainly, such feedback or opinions can | be worth considering as a way of getting at what principles are | being violated and deciding whether these violations are | tolerable or what ought to be done about them. A fresh pair of | eyes _could_ help. But an untrained pair of eyes might also not | be qualified to discern the right course of action. | PaulHoule wrote: | Yes and no. But 2/3 of the time the problem is that the | company has no documented build process or something obvious | like that . They have time to spend 2 years failing to | deliver a product because they don't know how to build it, | but when somebody asks "How do we build it?" the answer is | "Don't waste our time asking stupid questions." Of course | they have been wasting time not knowing how to build the | system, the guy who started the project might be able to hit | F5 in 15 different windows and get it to sorta kinda work, | but new hires they are hiring to work on it are quitting | right away and somehow they can never get it into production. | | There should be no controversy at all that complete | instructions for installing everything required for a dev to | build the project and work on it should exist and it should | be possible to complete this task in hours, not the weeks | that it frequently takes. And, no, "docker" is not an answer | to this anymore than "The F5 Key is a Build Process" | | https://blog.codinghorror.com/the-f5-key-is-not-a-build- | proc... | | It is not "Docker" that solves the problem, it is the | discipline of scripting the image build process into a | dockerfile. If you know how to write a dockerfile you can | write a bash script that runs in 20 seconds as opposed to | having Docker spend 20 minutes downloading images and then | crash because of a typo. | | You are right that a company might have good reasons for | doing things in an unobvious way, but most of the time when | nobody at a company claims to understand what the company is | doing except for the CEO and people aren't too sure about the | CEO, it is the fault of the company lacking alignment, not a | natural property of freshers. | skissane wrote: | > It is not "Docker" that solves the problem, it is the | discipline of scripting the image build process into a | dockerfile. If you know how to write a dockerfile you can | write a bash script that runs in 20 seconds as opposed to | having Docker spend 20 minutes downloading images and then | crash because of a typo. | | The problem is the bash script may end up depending on | poorly understood aspects of the local setup (global config | files, installed packages, etc) - it might work fine now, | but then nobody runs it for 12 months and there's some | churn in personnel and suddenly people are trying to work | out why it crashes. Dockerfiles can avoid some of that | stuff, although not always (e.g. the common problem that if | you don't fix the versions of packages to be installed, an | updated package is released which then breaks the | Dockerfile) | chaps wrote: | Been told that before. When I spoke up, I was told that I was | new and shouldn't talk about things I know nothing about. | PaulHoule wrote: | It's not (1) "reacting to the reaction" which is the endpoint | but (2) not having that reaction. If (1) is important it as | because that is the path to (2). | | I'd say a company that has accomplished (2) has cut the | workload in hiring employees by 30-50% in the sense that | every employee who has reaction (1) either internally or | externally is at risk for being disengaged or leaving soon. | Not only that but you are probably wasting your dev's time | and could get dramatically more productivity out of them if | you aren't WTFing them to death. | giobox wrote: | There's often a wise tradeoff between criticizing systems | you've just seen after being at the company 5 minutes and | actually spending some time at the company to learn the | historical context of _why_ the thing you think is insane | /shit is insane/shit before telling everyone who built it how | insane/shit it is. | | People generally don't wake up in the morning and go into | work motivated to make insane/shit things - context, tech | debt and business realities all mount up and even the best of | us can end up making choices that in isolation look crazy. | | There are of course companies who are really bad and you may | well be right, but so many times I have seen in my career a | young new hire storm in and think everything is shit without | paying heed to the context and historical pressures. The best | thing you can do in many cases is spend the first ~six months | at a new tech company trying to understand that context, and | indeed I think more mature engineers generally do. | a4isms wrote: | "Making good decisions, therefore, requires understanding | past decisions. Without knowing how things came to be, it's | easy to make things worse." | | --https://thoughtbot.com/blog/chestertons-fence | chaps wrote: | I was once told this after raising the issue that "hey | maybe an API that responds with root ssh passwords is a | bad idea, and our clients are going to be pissed once | they find out." And.. I was right. | | So often, citing Chesterton's fence is significantly more | naive than what it attempts to criticize. | a4isms wrote: | How is being right about wanting to make a change a | refutation of the idea that knowing why people made a | decision in the past is an excellent idea? Nothing about | Chesterton's Fence asserts that people in the past always | made good decisions, nor does it assert that perhaps what | was a good decision then is a bad decision today. | | It simply asserts that understanding why a thing is the | way it is is valuable when making a decision to change | it. | | That understanding could be as simple as--to take a real | world example that most readers here will remember--"They | chose to install a hidden web server on the user's | system, because they felt it was the best way to deliver | user convenience given the resources and time the team | had available." | | We can still say it was a bone-headed choice to do that | because it opened a massive back door to every user's | system. And? What is the problem with looking into why | they made that choice before arguing that the choice | should be reversed with maximum prejudice? | | Chesterton's Fence isn't a suggestion that no changes | should be proposed, or that if you look into the original | motivations you will change your proposal. Think of it as | insurance against the possibility that every once in a | while, you will discover a requirement that needs to be | addressed with your suggested change. | | I don't see where you're coming from that quoting | Chesterton's Fence is even "criticism." It's a suggestion | to take out a little insurance by doing a little | homework. | chaps wrote: | We can still say it was a bone-headed choice to do that | because it opened a massive back door to every user's | system. And? What is the problem with looking into why | they made that choice before arguing that the choice | should be reversed with maximum prejudice? | | Because I was suggesting a different method for the | specific task at-hand. I don't see where | you're coming from that quoting Chesterton's Fence is | even "criticism." It's a suggestion to take out a little | insurance by doing a little homework. | | Every time I've heard someone quote Chesterton's Fence, | it's always been as a means to halt the conversation. | Essentially, "shutup" -- an indirect critique of critique | itself in dismissive form. There's possibly some meta | point here about you not knowing the full circumstances | of the situation to warrant bringing up Chesterton's | fence. | a4isms wrote: | > Every time I've heard someone quote Chesterton's Fence, | it's always been as a means to halt the conversation. | | From here forward you can say that _almost_ every time | you 've heard someone quote Chesterton's Fence, it's | _almost_ always been as a means to halt the conversation. | | Today, you've encountered a counter-example, and a very | firm counter-example, at that. To my mind, Chesterton's | Fence is explicitly NOT about shutting down a | conversation. It's an invitation to continue the | conversation with more information to validate your | suggested course of action. | | No different than if an engineer suggests, "We should | rewrite this code to be faster." What team lead or | product manager wouldn't ask, "Is this a bottleneck? Have | you profiled it? Do we know there are users impacted by | this code's performance?" | | Or if someone suggests building a bespoke feature flag | service. "Have you done a build vs. buy analysis? What | alternatives have you considered before choosing this | design? Are there any OSS solutions that are close enough | to our requirements?" | | These kinds of responses shouldn't be uttered as a way of | shutting down a conversation. If that's someone's intent, | they are abusing their privilege. | | The right way to use any of these patterns is to say them | in good faith, and then socialize amongst the team the | standard of preparation the team expects of someone | proposing a non-trivial change. | | Over time, the need to say such things decreases because | the team internalizes what | preparation/rigor/justification is needed for proposing | changes, and does the work ahead of suggesting changes. | | Whereas, if the tone and intent is to block change, the | team goes down a toxic path where people are discouraged | from suggesting improvements of any kind. If that's what | you've encountered, you have my sympathy and I can | complete understand why you might be wary of people | quoting Chesterton's Fence. | chaps wrote: | These kinds of responses shouldn't be uttered as a way of | shutting down a conversation. If that's someone's intent, | they are abusing their privilege. | | To be perfectly honest, your quote came across in that | spirit. Anywho, peace! | aidenn0 wrote: | I can think of a dozen reasons why such an API might | exist. Chesterton's fence doesn't say "don't make | changes" it says "If you can't think of why something | might exist, you are not qualified to say that it | shouldn't exist." | chaps wrote: | I never said I didn't understand why the API existed. | aidenn0 wrote: | Then Chesterton's fence doesn't apply. People wrongly | using arguments to enforce blind conservatism doesn't | make the underlying arguments wrong... | chaps wrote: | Then we agree!! | justin_oaks wrote: | I agree. Chesterton's fence doesn't mean that if you | don't know why the fence is there, don't ever move it, | under any conditions. It means try to find out why it's | there before moving it. | | In many cases in my career, I've seen code that doesn't | make sense or seems like a bad idea. The person who could | explain why it's there has long left the company. Am I | afraid and leave the screwy stuff there, while citing | Chesteron's fence? Hell no. I'll change it to do the | right thing. This results in either exposing the reason | why it's there, or showing that it really was | unnecessary/bad. If something breaks from the change then | it's good that I can finally document what wasn't | documented before. So either way it's a win. | Ma8ee wrote: | The problem is that the historical context of a decision | often becomes a defence of the status quo, even when most | people understand it is bad. | Buttons840 wrote: | > There's often a wise tradeoff between criticizing systems | you've just seen after being at the company 5 minutes and | waiting 6 months to understand the context. | | I know this is reasonable advice, but it makes me deeply | cynical. After 6 months I will have learned to live in the | shit (to use your term), and so it still seems like I have | nothing to gain by speaking up or trying to fix things. A | culture that accepts shitty code probably isn't supper | demanding for an experienced developer who is accustom to | the mess, so I'll just coast through my time and hop jobs | after a few years. | | If nobody wants to respectfully talk about my criticisms on | day one, then they wont really want to at 6 months either. | In the end I'm lead to believe I should have zero concern | for code quality and only worry about my personal | reputation. | AA-BA-94-2A-56 wrote: | This is exactly where I am at right now. I have a million | dollar mortgage and interest rates are going up. I'm | keeping my head down and perpetuating technical debt | until I can hop jobs for a higher salary. | | Criticising the status quo is not a winning move for me, | especially when it's lead to the company's engineering | team tripling in size. If I'm asked, I'll pick some low | hanging fruit- remove reliance on legacy/redundant | JavaScript libraries such as jQuery, and spent time | writing better unit tests. But so far I haven't been | asked. | chaps wrote: | You (in fashion of the article) missed the point I made: I | was explicitly asked to speak up as the new employee and | when I did I was told to stop speaking up. When I brought | it up in the meeting I gave my two week notice, he admitted | to saying that and apologized. | dec0dedab0de wrote: | That's some bullshit. you did the right thing. When a new | employee joins and notices that something sucks the | answer should be something like these: | | We know, but haven't had time to fix it, maybe we'll | assign that to you when you're caught up. | | We didn't think of it that way, good catch, lets go into | detail later. | | Yeah, but doing it this way makes this other thing | easier, we'll show you that when you're ready. | | or even: I don't know, my brain is fried with this | project, can you ask again in a few months? | whats_a_quasar wrote: | I get what you're saying, but this is exactly how deviances | are normalized. When you've been with the company for a | long time and are familiar with the history, it's easy to | rationalize why things are the way they are and that they | can't be improved. You can explain something that's crazy | with context and historical pressure. | | Dan's point is that sometimes the new person's judgement is | correct, and there actually is a real problem that's | invisible to people who have been with the project a long | time. But the new person's judgment is basically always | ignored, and that's a mistake - it ought to be weighted | heavily because they legitimately have a perspective that | insiders no longer have. | | If instead you spend six months trying to understand the | context: | | "new person joins | | new person: WTF WTF WTF WTF WTF | | old hands: yeah we know we're concerned about it | | new person: WTF WTF wTF wtf wtf w... | | new person gets used to it | | new person #2 joins | | new person #2: WTF WTF WTF WTF | | new person: yeah we know. we're concerned about it." | | I'm sympathetic because my first company was a mid-stage | startup with huge structural problems in the engineering | org structure and processes. When I joined I had frequent | "WTF" moments and had a similar experience where | experienced people would explain to me why things are the | way they are. So I trusted them, and put my head down, but | eventually got frustrated and left. A few months later the | company went bankrupt because they couldn't build product | fast enough, investors lost patience, and they couldn't | raise another round. | Buttons840 wrote: | > the new person's judgment is basically always ignored, | and that's a mistake | | Remember, the new person has something that nobody else | on the team can ever learn, no matter how much they study | or how long they work. The new person has a fresh | perspective. | AnIdiotOnTheNet wrote: | To be fair, you really shouldn't. You know nothing of the | constraints that people are operating under, or the political | or cultural landscape you're dealing with, so you just come | off like a preachy academic. | chaps wrote: | See the other comment I made: I was explicitly asked to | speak up. | AnIdiotOnTheNet wrote: | If that's the case, then yeah fair enough. If someone | doesn't want your opinion they shouldn't ask for it. | jkaptur wrote: | Just to give a different, concrete, perspective (and push a hot | button HN issue), I've spent a fair amount of time working on | extremely large web applications, and _by far_ the #1 "WTF WTF | WTF" thing that new hires say is "what do you mean you aren't | using $TODAYS_HOT_JS_FRAMEWORK??" | | Once you get away from "should we use version control" and into | actually difficult software engineering questions, it's not | clear how to balance a fresh perspective vs. an experienced | (normalized? tainted?) view. I wish the article went into this | more. | | Like, how does the new hire (or anyone else) know the | difference between "learning the complexity of the new system" | and "internalizing/normalizing the deviance of this culture"? | 908B64B197 wrote: | > by far the #1 "WTF WTF WTF" thing that new hires say is | "what do you mean you aren't using | $TODAYS_HOT_JS_FRAMEWORK??" | | To that speaks of the caliber of programmers hired. If all | they have seen is $TODAYS_HOT_JS_FRAMEWORK and wrote nothing | but a web app using $TODAYS_HOT_JS_FRAMEWORK they might not | grasp the fundamentals that would make then realize that | frameworks are just abstractions (and not that different from | one another). | | I don't think any software engineer would even ask that | question, since the answer will almost always be | "$TODAYS_HOT_JS_FRAMEWORK didn't exist when the project | started, and it's not worth a re-write to port it over". | | Now, that brings out a second important truth: a company | can't attract and retain a wide range of different caliber | employees. For instance, if a place still questions the | usefulness of source control (perhaps because they consider | git to be too complicated) there's no way they'll attract and | retain top performers. So the culture will select people that | agree that source control is a waste of time. | flappyeagle wrote: | If "what do you mean you aren't using | $TODAYS_HOT_JS_FRAMEWORK" is the first question they ask, | then you can end their employment right then and there. Hire | people whose "wtf" you take seriously. | evilduck wrote: | Both parties would feel like they're dodging a bullet. | | People still griping about $TODAYS_HOT_JS_FRAMEWORK are | clearly out of touch in 2023. It was funny commentary in | 2013. It still rang a little true until around 2016. Now | it's just an indicator you are the one not to be taken | seriously. | | It's React, Vue, or Angular and it's been that way for many | years now. | notduncansmith wrote: | React underlies a lot of the "new hotness" like Next.js | and Remix.run (I'm not sure if Vue, Angular, or Svelte | have equivalents but I wouldn't be surprised). | aidenn0 wrote: | > Like, how does the new hire (or anyone else) know the | difference between "learning the complexity of the new | system" and "internalizing/normalizing the deviance of this | culture"? | | If a new hire can't checkout, build, and test the software on | the first day, then there is likely something either wrong | with the hire or the infrastructure. A sufficiently old and | arcane software system might take weeks before a new hire can | make even a simple change, but that shouldn't impact those | three items. | Jtsummers wrote: | Along with this "how long to spin up the new hire" issue, | one of my first (if not the first) questions when trying to | help people improve their processes related to software is: | | > If the user/client asks you to make a small but not | _trivial_ change, how long would it take to update and | deploy the program? | | I have had answers ranging from "A couple hours" to "A | year" (yes, they were serious). Most were in the 1-3 month | range, though, which is pretty bad for a small change. It | also makes it apparent why a bunch of changes get batched | together whether reasonable or not. If a single small | change, single large change, or collection of variable | sized changes all take a few months to happen, might as | well batch them all up. It becomes the norm for the team. | "Of course it takes 3 months to change the order of items | in a menu. Why would it ever be faster than that?" | ReflectedImage wrote: | That's not a "WTF". All front end developers trash their | predecessors work and rewrite in the last framework. That | just what they do. | jt2190 wrote: | Yes. From the article: | | > "The thing that's really insidious here is that [once a | person buys] into the WTF idea... they can spread it | elsewhere for the duration of their career... Once people get | convinced that some deviation is normal, they often get | really invested in the idea." | | > [H]ow does the new hire (or anyone else) know the | difference between "learning the complexity of the new | system" and "internalizing/normalizing the deviance of this | culture"? | | The article implies that the new hire should pay close | attention to the things that are incentivized, and those that | are not. | whats_a_quasar wrote: | There's a whole section about how to balance a fresh | perspective, in "solutions". The best way is for an effective | VP to hear the WTFs of new hires, apply engineering | judgement, and make changes based on that signal. If | management is not the ones creating the deviance, they ought | to be able to tell what reactions are just unfamiliarity with | the system and what are signs of something actually being | broken. The article is arguing that most people ignore those | weak signals by default, and ought to pay more attention to | them, not that they're always reliable. | josephg wrote: | > it's not clear how to balance a fresh perspective vs. an | experienced (normalized? tainted?) view. | | Having a healthy, balanced view comes from enough | experiences. Ideally from working in a bunch of places and | seeing enough things go sideways, and correctly understanding | and identifying the causal chain that led to failures. | | Funnily enough its sort of like training an AI - you | essentially need a lot of correctly labelled data to learn. | Junior engineers don't have enough data points, and | unfortunately some "senior engineers" I've worked with took | (in my opinion) the wrong lessons from their experiences. (Eg | the CTO who thinks version control is too complex.) | | The interesting cases are when smart, experienced people | disagree on what the best solution is. Should you keep your | team small and smart or have a varied team with more | mentorship and process? Is code review worth it in every | case? What is the right amount of tests for your software? | How often do we want to push to production? | | When I was teaching programming my students would sometimes | ask juicy questions. My favorites were the questions I could | answer with "I'll tell you my opinion, but I've worked with | people I look up to who think I'm wrong about this..." | rr888 wrote: | The problem is when you join a high performing team and | organization. Is a WTF something they should fix your you need | to recalibrate what you think is normal? | GartzenDeHaes wrote: | In my experience, you cannot change an organization's culture | with rules, mission statements, listing values, or giving | speeches. They only way is to take down the old culture | bearers. Some of them may be managers, but more often they are | employees who have gained some organizational power. You'll | often find them at the center of sticky organizational spider | webs with approval processes such as purchasing and service | administrators. | | To change the culture, these people have to go. Firing them may | not be feasible, but there are other options. Dethroning them | in the form of a promotion or even just physically moving them | can be effective. When people don't have to jump through their | hoops anymore, they lose their organizational power. | Ensorceled wrote: | > They only way is to take down the old culture bearers. ... | To change the culture, these people have to go. | | I was briefly head of engineering at a company that had | several "old culture bearers" that made change impossible. I | was something like the 3rd or 4th engineering leader over the | space of a year. Apparently the person after me was actually | allowed to fire a few of these people and was able to turn | things around. | PaulHoule wrote: | More than once I had to leave in order for an employer to | take action against a problem employee, including my boss | once. In these cases the employer suddenly realized that it | made no sense to attempt to replace me without removing the | reason that drove me out. | Ensorceled wrote: | That is, unfortunately, very common. Though sometimes the | toxic people drive the company into the ground first. | e_i_pi_2 wrote: | This is something we actively try to take advantage of at my | company - we know that we've grown comfortable with | architecture that may not make a lot of intuitive sense, so | when we have new people join we try to make a list of confusing | concepts so we can try to clean them up. In the ideal case the | new person is able to do the cleanup, so we get a more | intuitive design and they learn the surrounding architecture | more along the way | AlbertCory wrote: | This guy needs to organize & format his writing better, since he | does have really interesting things to say. | albertop wrote: | [flagged] | justin_oaks wrote: | Probably being downvoted because the HN guidelines explicitly | say to not comment about such things: | | "Please don't complain about tangential annoyances--e.g. | article or website formats, name collisions, or back-button | breakage. They're too common to be interesting." | | https://news.ycombinator.com/newsguidelines.html | AlbertCory wrote: | As a matter of fact, it's in positive territory right now. | | No, it's not a tangential annoyance. There are times when | form IS function. | AlbertCory wrote: | It's really pretty easy, and free, to use the Substack editor | to create a nicely formatted article. If you care at all | about your readers. | mixmastamyk wrote: | Talk about killing a fly with a sledgehammer. Simply needs | one or two lines of css, max-width and possibly font-size. | AlbertCory wrote: | I had two words there: "organize" and "format." This | addresses only the latter. | msm_ wrote: | Maybe because it's an absolute statement ("this person needs | to organise & format his writing better") for a something | mostly subjective. | | In my opinion the website is readable, fast, lightweight, not | distracting and pleasant to read. It's also accessible for | people with disabilities, responsive, and works everywhere. I | understand that not everyone is as into minimalism as me, | just pointing out the problem in "this person needs (...)". | dang wrote: | " _Please don 't complain about tangential annoyances--e.g. | article or website formats, name collisions, or back-button | breakage. They're too common to be interesting._" | | https://news.ycombinator.com/newsguidelines.html | burnished wrote: | This is one of the few sites I find immediately and highly | readable (on mobile). Does it not wrap lines on larger | displays? | sebstefan wrote: | Firefox has the "reader view" option toggleable with F9 for | when you stumble upon unreadable designs, if you want | jwilk wrote: | It's F9 on Windows, command-option-r on macOS, and ctrl-alt-r | elsewhere. | | https://support.mozilla.org/en-US/kb/firefox-reader-view- | clu... | xnorswap wrote: | If you don't have a reader view in your browser, just paste | this into your global CSS: p { | line-height: 1.7; max-width: 60em; font- | size: 1.2em; margin-left: 5em; } | | It pretty much fixes the default readability which is | essentially zero on this site otherwise. | ryandrake wrote: | Yuck! I hate it when sites monkey around with max-width. I've | got a nice 27 inch monitor. I want to use all of it. It's | refreshing to see a site that _doesn 't_ insist on second- | guessing the width that I set my browser window. | atoav wrote: | So you look like you watch a tennis game when you read | text? Funny. | | On a more serious note: A maximum length for text makes | sense ergonomically, which is why especially big prints | like newspapers or magazines work with columns. Columns | however haven't really cought on in the web, because they | do not combine very well with the whole scrolling thing. | mixmastamyk wrote: | Too hard to read. See newspapers. Multiple windows are a | thing. | ryandrake wrote: | So, leave the choice to the user. If I want to read text | in a small column, I can easily resize my window. Don't | try to force one way. | Sesse__ wrote: | Dan Luu talks a bit about this here: | https://twitter.com/danluu/status/1115707741102727168 | AlbertCory wrote: | yes, if you ignore a millennium's worth of publishing | wisdom and think the world of text began anew in 1995, then | maybe you can claim that "the science is not settled." | | He has every right to make his choice. As do we, in | deciding whether to read it. | robocat wrote: | Great thread! | | Designer says: my opinion is right because A. Dan shows: A | is not factual. Designer says: my opinion is right because | of B. | | Dan notices behaviours, and then he writes compellingly | about those behaviours. | RoddaWallPro wrote: | One of the more hilarious takes I've seen. "There are no | papers for this, and I choose to disregard the countless | number of people who say it is much easier for them to read | if the line lengths are constrained as they are in a book | or scroll or every other form of human writing ever put on | this earth, so I will not make my site easier to read. F | you." | whats_a_quasar wrote: | I really don't think that's what he's saying. You are | assuming a great deal of malice, rather than positive | intent. What he's saying is that there isn't hard | evidence that shorter lines are more readable, so he made | the style choice of longer lines. You're claiming that | most people prefer shorter line widths, but again present | no evidence that most people actually have that | preference, other than vague references to "countless | people". I actually think you're probably right, and if | you had data Dan might update his stylesheet. But in the | absence of evidence, you're just presenting your opinion | as fact, and assuming malice. | RoddaWallPro wrote: | Yeah, that's probably true. He also at least allows | people to set their own reading width by adjusting | browser. | | My frustration stems from the fact that I find the | argument "there are no papers with sufficient evidence" | to be pedantic bullshit. Like yeah, sure, you aren't even | wrong, but absence of evidence is not evidence of | absence. I've never seen anyone claim to like 180 char | lines, whereas I've seen hordes of people who say it is | very difficult for them to read that line length, and | prefer something book-sized (lengthed?). | whats_a_quasar wrote: | Hmm, yeah I actually agree with that way of putting it. | The evidence that does exist are the anecdotes, and he | seems to ignore that evidence. Many of his readers, | myself included, seem to prefer fixed line lengths. So it | is a weird choice. | | I was mostly reacting to the assumed malice in the parent | comment. Based on his blogging style, I think it's more | likely that Dan's just a pedantic guy implementing his | personal preferences on his personal blog :) | AlbertCory wrote: | "Not even wrong" FTW. There's even a blog with that | title: | | https://www.math.columbia.edu/~woit/wordpress/ | travisjungroth wrote: | This is why it's nice to get people's preferences vs | thoughts about preferences. Preferences aggregate, | thoughts about preferences do not. If one designer says | "Readers like narrow columns more, everyone knows that." | and another says "I like reading narrow columns.", I'd | give the person speaking about readers in general more | weight (even with them going the same direction). But, if | 100 designers spoke for all readers and 100 designers | spoke for themselves, I'm giving more weight to the | second _group_. Hearing 100 preferences is more valuable | than hearing one idea, 100 times. | sebstefan wrote: | I don't like fucking with the default CSS, my alternative is | this bookmark that injects Javascript in the page and | temporarily fixes the formatting at one click of a button | javascript:(function(){ var bod = | document.getElementsByTagName("body")[0]; bod.style.margin = | "40px auto"; bod.style.maxWidth = "40vw"; | bod.style.lineHeight = "1.6"; bod.style.fontSize = "18px"; | bod.style.color = "#444"; bod.style.padding = "0 10px"; })(); | | And for a dark mode: | javascript:(function(){ var body = | document.getElementsByTagName("body"); var html = | document.getElementsByTagName("html"); var img = | document.getElementsByTagName("img"); | body[0].style.background = "#131313"; body[0].style.opacity = | "1.0"; html[0].style.filter = "brightness(115%) contrast(95%) | invert(1) hue-rotate(180deg)"; img[0].style.filter = | "contrast(95%) invert(1)"; })(); | pasquinelli wrote: | that site is better than most, imho. | [deleted] | t3estabc wrote: | [dead] | bluedino wrote: | Write total shit for code, then look like a 'genius' for 'fixing' | bugs, only to have them come back again in the future (further | looking like a clown to the rest of the team) | irsagent wrote: | "rictus of horror" - What a set of words to describe a response. | kurthr wrote: | There's been a lot of work on reliability of complex systems and | how they operate. What has been found is that it is almost always | necessary to have failure (degraded operation) modes that prevent | system failure, and the more complex and more hazardous failure | is the more modes develop. | | In these systems it is found that they are almost always | operating (or transitioning between) failure modes. Often | multiple operational failure modes are simultaneous. It becomes | very important to test the system in each of it's failure modes | and their combinations to maintain high up time. | | https://how.complexsystems.fail/ is an example, but there are | many. | | Human work, development, and maintenance is itself a system that | interacts with these critical systems. Frankly, failure to fail | causes failure (thus chaos monkey). The mythical man month is | almost a sub category of these failures as are HR hiring | processes and other BS. Being too successful and not having | competition (or similarly sclerotic competition) can be as much | of a hazard as "move fast, break things". | justin_oaks wrote: | I welcome others to share stories of the normalization of | deviance in their companies. | | One company I worked had no unit tests, no infrastructure as | code, and no build server. This held strong for a while until | enough developers implemented some unit tests, infrastructure as | code (e.g. terraform), and a build server as skunkworks projects. | Eventually management tolerated them, but never endorsed them. | Some teams at the company still never embraced good practices | because it wasn't forced on them. | | I guess I've never worked at a company that valued unit tests | across the whole of the engineering team. I introduced them and | implemented them on my own team, but others ignored it. | ctroein89 wrote: | > and no build server | | Personal experience is that a build server normalizes deviance. | "But it works on the build server" we used to say, as, with | time, it become harder and harder to build locally. "Just fix | your environment!" we used to say, when it was the build system | that was actually at fault. "It's all so fragile, just copy | what we've done before!" we then said, repeating the mistakes | that made the build system so fragile. | | Eventually, the build system moved into a Docker image, where | the smells where contained. But I'm still trying to refactor | the build system to a portable, modern alternative. If we | hadn't had a build server, we'd have fixed these core issues | earlier and wouldn't have built on such a bad foundation. Devs | should be building systems that work locally: the heterogeneity | forces better error handling, the limited resources forces | designing better scaleability, and most importantly, it | prevents "but it works on the build server!". | aj7 wrote: | "Let's say you notice that your company has a problem that I've | heard people at most companies complain about: people get | promoted for heroism and putting out fires, not for preventing | fires." | | My first day at work at big-laser-company. Manufacturing engineer | for a laser (then) so complex, it required a PhD to solve | problems to get units out the door. The product was a ring laser. | What that means is that the laser beam travels around in a race | track pattern inside the laser before getting out, not a back- | and-forth bouncing between two mirrors. Now this laser could be | tuned to any wavelength by suitable setups and machinations, and | once there, would "scan" a small amount about this wavelength, | enabling scientists to study tiny spectral features in atoms and | molecules with great precision. I knew all this shit. I was a | Berkeley-trained physicist that built precision lasers out of | scrap metal for my thesis. First day of work. I walk into the | final test lab. The big laser was happily scanning away. The | bright yellow needle-like output beam was permitted to hit the | lab wall. As the laser scanned, the beam was MOVING on the wall. | Whereupon, first day of work, I exclaimed the most obscene four | words in manufacturing, for all to hear, "You can't ship that!" | ("Beam pointing instability" is detrimental to almost any laser | application. It turns out that during scanning, an optical | element was rotating, on a shaft, inside this laser. This | mechanical motion caused beam motion.") Well, I got an immediate | reputation as a negative guy. (You can tell it's deserved.) The | solution was to retrofit 28 lasers in the field, mostly in | Europe, with a component that cancelled the movement, on an | expensive junket by a service guy. Who was hailed as a "hero." | KMag wrote: | Interesting. I had an internship at a company that did inertial | navigation, mostly for defence applications. I only knew of | ring lasers for use in gyroscopes. (Send a laser around a loop | wave guide/fiberoptic, and any translational acceleration | cancels out going out and back, but any acceleration in | rotational velocity in the plane of the ring/rotation vector | perpendicular to the ring shows up as a Dopler shift. Tune the | laser to have a standing wave, and rotational acceleration | shifts the nodes of the standing wave around the ring.) | | I had a colleague who got called up when a Trident missile MIRV | bus fell off a forklift and he had to do simulations to tell | the Navy if it was still good or needed to be brought back in | for rework/recalibration. My understanding is that either the | MRIV bus itself or its container has integral devices that | record peak 3-axis acceleration for just such a scenario. I | imagine they're as simple as a few precise weights on a few | wires with precise failure strains, so you can bracket the peak | acceleration by which wires broke and which survived. | | On the one hand, it's great to have more accurate nukes, which | allow lower yields, smaller stockpiles, and presumably smaller | craters if everything goes sideways. On the other hand, | "surgical" nukes result in it more likely that one side will | use them and gamble that the other side won't massively | retaliate. | XorNot wrote: | You could look at it a different way: a more accurate nuke | means a nuke that's targeted at military facilities and not | sized 10x larger and aimed at "everything around that city | over there". | | _If_ it was ever used, that work saves lives. | KMag wrote: | More importantly, I think more accurate nukes along with | good satellite multispectral and signals intelligence means | that top generals carrying out orders for nuclear first | strikes can be more certain that they're signing their own | death warrants. Hopefully this results in any leader | ordering a nuclear first strike getting deposed by military | coup rather than starting a nuclear war. | tablespoon wrote: | > More importantly, I think more accurate nukes along | with good satellite multispectral and signals | intelligence means that top generals carrying out orders | for nuclear first strikes can be more certain that | they're signing their own death warrants. | | How would you do that? In the event of a nuclear war, my | understanding is they'll mostly be flying around on | special command and control planes. I don't think nuclear | intercontinental SAMs are a thing. I'm not even sure if | they could even be possible (wouldn't they need active | guidance, which would be very hard on reentry). | throwbadubadu wrote: | Please just don't do that. (Or missed irony?) | geenew wrote: | I once encountered a situation with a very expensive field | laser. At one point, measurements started showing an increasing | amount of offset. | | Over a period of days, the error became increasingly, comically | bad, until finally the system refused to boot. | | A technician was called, and after hearing about the behaviour, | the first request was that a photo of the laser light exit port | be taken. | | It was obvious why it wouldn't boot: a mirror in the light path | had fallen off. | | The worst part was, the mirror had been held on by glue, and | had been slowly slipping out of place. The hot climate was | probably a factor. | | They really should have had someone to say 'you can't ship | that' when the topic of glue to hold mirrors came up. | pm_me_your_quan wrote: | This exact problem happened with an optic in my lab in | graduate school. For two years the senior grad student and | postdoc blamed each other over the entire apparatus becoming | misaligned every couple of days. (It was a really toxic | environment.) Eventually, they both left, I was the only one | there, and it still became misaligned. In one day I tracked | it down to a prism from Thorlabs whose glue had gone bad | positioned at the very beginning of the laser line- it was | sliding in its mount. | | I wish I had pushed more strongly about it. We spent probably | a full person-day of work every week on that. | whats_a_quasar wrote: | Oh man, this one hurts to read a little bit. It's crazy how | people cooperating poorly can eat up that much working | effort. | renewiltord wrote: | In _The Field Guide to Human Error Investigations_ by Sidney | Dekker, he quotes someone else saying something like: | | > _Everything that can go wrong will go right._ | | Murphy's Law then manifests from escaping disaster through | repeated iterations of taking risks where most things play out | well anyway. | | I have to laugh at the "append z to the end" strat at Google, | though. That's a good one. ___________________________________________________________________ (page generated 2023-02-14 23:00 UTC)