[HN Gopher] A developer's guide to programatically overcome fear...
___________________________________________________________________
A developer's guide to programatically overcome fear of failure
Author : kiyanwang
Score : 248 points
Date : 2021-10-31 11:27 UTC (11 hours ago)
(HTM) web link (www.pagerduty.com)
(TXT) w3m dump (www.pagerduty.com)
| GDC7 wrote:
| We have to somehow hack ourselves and make our brains love
| winning/succeeding.
|
| I think we mostly don't love winning or succeeding per se, even
| when that happens it's not the joy of winning and succeeding as
| much as it is the relief of not having crashed and burned.
|
| The whole mental process is dominated by loss aversion, not by
| victory enthusiasm.
| crispyambulance wrote:
| I think whole swathes of the SWE community are behind the times
| when it comes to responding to crisis.
|
| There are models for how to handle this stuff in other
| industries. What comes to mind is "Tiger Teams" and "EMS
| services". Both of these involve the deliberate creation of
| special organizational structures that respond to crisis. For
| both of them that means contingency planning and, importantly,
| practice. It's largely about training for things that might
| happen and thus being ready when they do happen-- even if they're
| not exactly what was expected.
|
| Sadly, the model for crisis management in SWE (in most places) is
| just throwing around the word "accountability", putting people in
| the hot-seat, and hoping for the best. You end up with "hero-
| based" crisis management. In fiction, that works out great (James
| Bond types always win), but in reality you just end up with
| stressed-out people and flaky systems.
| JJMcJ wrote:
| Some companies, ones large enough to have serious SRE staffs,
| have 24 hour staffing of an operations center, and a War Room
| where people can huddle until a problem is fixed.
|
| If they are smart, they have ways to roll back offending code.
|
| But any company smaller than say a regional bank is going to
| have trouble affording this.
|
| Now, if you using a cloud provider, you can design in rollback.
|
| But of course there are surprises. A scenario I just thought
| up, when you discover on Black Friday that your order system
| can't handle more than 1048575 (that's 2 to the 20th minus 1,
| just to pick a slightly plausible number) orders in one day,
| and this limit is deep in your code and architecture.
| cactus2093 wrote:
| The trend I've seen in big tech is that microservices
| architectures kind of negate a lot of this benefit though.
| Yes, there might be SRE's who are on-call for existential
| company-wide outages, and ideally they have some general
| knowledge about some of the biggest systems and common causes
| of issues. But for any issues that are due to bugs in the
| code or configuration of one particular service (which is
| usually most issues), there is a separate on-call rotation
| for each service. These folks are mostly just regular
| engineers that are primarily responsible for building
| features and delivering under time pressure and don't have
| the luxury of a lot of training & practice dealing with
| incidents until they happen for real.
| lamontcg wrote:
| So after a year of two of oncall at Amazon (it was long
| enough ago that we were oncall for the whole thing, long
| before SRE roles were ever invented) the biggest thing I
| learned was that nobody was dying and it wasn't actually
| fire-fighting and we shouldn't be using analogies like War
| Rooms and all these other War/EMS/fire analogies.
|
| I've never worked in one of those jobs, but I've actually
| dragged dead bodies out of the water due to scuba diving
| accidents and analogizing IT outages to real life fatalities
| is almost sort of like how we think about holocaust/nazi
| comparisons and shouldn't use those casually. The industry
| really needs to chill out a bit about the impact, nobody is
| generally dying for the kinds of things we're talking about
| where the website is just down (obviously some IT really is
| life-affecting, and in finance people can be hauled off to
| jail by the SEC, but we're generally not talking about those
| industries here).
|
| There are some processes such as accident analysis from
| aviation and aerospace which are useful, but could be used a
| bit more dispassionately. Blameless postmortems are also
| better than blamestorming, although I think there is a bit of
| a limit (at some point you may really have a bad team
| producing poor architecture which is constantly knocking the
| website over and you need to kick someone out, that isn't the
| purpose of an engineering postmortem though, that should
| happen at a management level).
|
| I'm also weird and I don't really feel a lot of guilt (never
| raised religious might have something to do with it) and I
| just naturally understand how "shoulds" are bad and find it
| mildly appalling how Millennials love to throw the word
| "shame" around for situations where it really doesn't apply
| (your sportsball team shouldn't really be "shamed" if they
| lose). The broader population could do with a reassessment of
| shame and guilt and perfectionism and "shoulds".
| throwaway984393 wrote:
| Back in the day (and still today, sadly) SWEs had nothing to do
| with how their code was run. You paid Operations Engineers to
| staff a NOC and build an entire practice around preventing and
| responding to crises. The SWEs would commit their code, push a
| deploy button, and go home for the day (at 5PM on a Friday),
| oblivious to any chaos that might ensue.
|
| I still meet SWEs who seem to believe they have no
| responsibility for any failures resulting from their code if it
| "worked for them in dev". No responsibility to design their
| code to resist failures or security issues in production, nor
| re-design it if the current design is failure-prone. No
| responsibility to performance test, functional test, or check
| with other teams on what a change might impact at scale. No
| need to come up with a rollback plan or test it.
|
| If their product makes a ton of money, upper management enables
| this attitude, because nobody wants to kill the golden goose or
| risk losing its developers. It's a question of priorities. If
| the business does not require SWEs to care about risk, failure,
| reliability, etc, then they won't.
| dividedbyzero wrote:
| I think some orgs are getting there, with SRE teams and red
| team exercises and the like. But I agree the overall industry
| is still deep in the 19th century, as far as managing this
| specific set of risks goes.
|
| > What comes to mind is "Tiger Teams" and "EMS services"
|
| I know EMS services of course, but what are Tiger Teams? A
| cursory Google search didn't turn up anything useful. Any
| pointers where I could learn more about them?
| crispyambulance wrote:
| "Tiger Teams" are relatively common in large technical
| organizations (usually outside of computing).
|
| I have seen them at a linear accelerator facility and also in
| large factories. I think the first time the name was used was
| in the NASA space program (and looking at the wikipedia link
| someone provided that appears to be correct).
|
| Basically, they are teams that are put together to deal with
| ongoing critical cross-cutting problems in the organization.
| They are, by nature, cross-functional. Large orgs tend to put
| talent into silos. Silo'd talent can make the org vulnerable
| to weird problems where no one feels enough agency to act
| using the excuse-- "it's not my job". A Tiger Team is
| intended to work across silos.
| niea_11 wrote:
| It's the first time I hear about Tiger teams too. Here is a
| Wikipedia article : https://en.wikipedia.org/wiki/Tiger_team
| Luigilacorte wrote:
| Even with a framework like this, it's hard to get past the
| visceral fear if failure. It's only with practicing failure do we
| get accustomed to it.
|
| Tony Hsieh from Zappos famously does one thing a day that scares
| him. At a certain point, potential failure is as scary as just
| trying something new.
| fortran77 wrote:
| And it worked really well for Tony:
|
| https://www.nytimes.com/2021/01/26/technology/tony-hsieh-dea...
| bloodyplonker22 wrote:
| You're one of those people that never takes risks and never
| really does anything and just has a plain jane life, aren't
| you. But, obviously, you're quick to laugh at others'
| misfortunes for trying to do things.
| nonbirithm wrote:
| And seeking out or even opening yourself up to a chance of
| failure gives certain people a risk of falling into a spiral of
| depression. I am not even sure that going for desirable yet
| challenging goals is even an option for these people if too
| much failure will destroy their confidence about life in
| general, whether or not that is a consequence that logically
| follows.
|
| If you're weighing an option that opens you up to a higher risk
| of failure than other options, then depending on what type of
| person you are, you may have to evaluate whether you have
| enough of a support network or other "cushion" to absorb the
| mental blow of failure first. Failure is something that can be
| gotten used to, but before it is, you have no choice but to get
| past the damage to your mental health the failure causes, and
| some people are not capable of managing that on their own.
|
| This is why I have to sometimes frame my decisions as: "Because
| I will fail considerably more often if I choose this path, I'm
| giving myself a considerable risk of becoming a miserable wreck
| incapable of doing the things I'm capable of now, or worse. So
| for the sake of preserving my mental stability, I will have to
| do something else."
| prismatix wrote:
| When interviewing a while back, I was asked to reflect on my
| failures. Only then did I realize that failure was not in my
| mental model of my experiences. I had a hard time thinking of a
| "failure". It's not that I had never had any, it's just that I
| had been so accustomed to the idea of thinking of them as
| learning experiences rather than failures. I wouldn't say I'm the
| most accomplished person by any means, but I certainly have a
| tried-and-true way of getting through so-called "failures", and
| part of that has to do with not even including failure in my
| vocabulary of my experience.
| testemailfordg2 wrote:
| Probably a small learning for all of us here - test your code
| with the data you already hold in production.
| known wrote:
| "Courage is mastery of fear, not absence of fear" --Mark Twain
| (b. 1835)
| drawqrtz wrote:
| Apologies if this is not quite on topic, but this definition of
| shame really spoke to me:
|
| "When a person feels that he or she is becoming his or her
| undesired or feared self."
|
| I feel like a lot of my motivation to better myself and help
| improve my environment comes from that feeling. It's as if i have
| a very clear picture of who/what i dont want to become, but not
| so clear on what the "ideal me" would be.
|
| Is this something that motivates HN as well or do you try to
| realize an ideal(?) version of you through your thoughts/actions
| giaour wrote:
| Don't we all just chant the "Fear is the mind killer" Bene
| Gesserit litany until the fear of failure dissipates?
| quadrifoliate wrote:
| The fact that this is on a pager app company's site immediately
| sets off some alarm bells.
|
| "Systematically overcome your fear of failure" is a great
| principle, but why is PagerDuty of all companies promoting this?
| I can't help but think this is an attempt to market some kind of
| product, but I can't figure out what it is.
| toomanybeersies wrote:
| > why is PagerDuty of all companies promoting this?
|
| To drive traffic and increase sales of PagerDuty. It reads like
| pretty standard content marketing to me.
| zz865 wrote:
| My deep seated fear is that some fundamental infrastructure will
| fail, Database, dns, DDoS, there are so many things that can go
| wrong. Esp with big old code bases.
|
| Many developers really dont care what happens in production,
| leaving a subset to stress out. Perhaps the solution is just be
| like the people who just expect problems and accept it.
| pseudoramble wrote:
| I hear you, it isn't great when people are apathetic to what
| happens in prod, especially when there are things that can be
| done.
|
| That being said, here's another way to view it: of all of those
| deep seated fears you have, what do you know you can do to
| avoid certain issues and what can you actually not do?
|
| Let's take the database for example. What's the deep seated
| fear here? Maybe:
|
| > The disks crash, the data center bursts into flames and all
| the data is gone!
|
| A solution to those is perform backups and get the backups off-
| prem. You have to explain the risk and justifications to the
| company as best as you can. Once you've done that, what is
| left?
|
| Technically speaking, bugs in the backup script could exist, or
| maybe the backups get corrupted. Beyond that, there are the
| human ones. Maybe somebody else manages the script and breaks
| it, or maybe backups get unintentionally poisoned by malware
| getting into the backups. Or maybe even before that happens the
| company goes "eh, risk worth taking to save costs!" and you
| don't even get to implement the backups.
|
| Here's the thing: these things are _not_ in your control at
| this point! They are out of your hands, and with humans like us
| who make mistakes like us. So problems are going to be
| expected, especially ones you don't realize right now. But
| they're not in your control! And yet, you did your part, and
| things will continue on.
| rconti wrote:
| As someone with a background in infrastructure operations, it
| always felt like we were the ones who cared and stressed out,
| and development didn't (have to) care.
|
| These days, particularly in microservices, more and more
| developers are on call for their work, so they care more.
|
| Still, I find a radical misalignment between my mentality,
| where you CANNOT make mistakes because you're working without a
| net, and development, where typically there are more layers of
| safety. I'm far, far more cautious and anxious about making
| changes and breaking things; they've historically been paid to
| produce. If I try to have a conversation about it, they're
| unable to grok that their threshold for the acceptability of
| making mistakes is much higher than mine is.
|
| You NEED to be able to make mistakes in order to be able to
| produce, but it's a hard mentality shift.
| ChrisMarshallNY wrote:
| TDD starts with failure.
|
| I don't really subscribe to TDD, but they have a valid point:
| Start with something that doesn't work, and work on it, until it
| works.
|
| I tend to develop my software in a similar way, but I also start
| with a "fuzzy" design, which would give TDD people fits.
|
| I fail every day. Repeatedly. It's how I end up with some damn
| good software, at the end of the day.
| agumonkey wrote:
| failure + exploring solution space + minimizing said space <= I
| need a book about that :)
| ChrisMarshallNY wrote:
| I could probably write one, but writing books is _hard_. I am
| still smarting from the time I wrote a 400-page book, that
| was out of date before it could be published.
|
| I do have a blurb that discusses my design philosophy here:
| https://littlegreenviper.com/miscellany/evolutionary-
| design-...
| agumonkey wrote:
| "Vague" systemic exploration is interesting, having live
| prototypes to modify and refine gradually, to gain
| knowledge and maybe readjust the substructures.
| ChrisMarshallNY wrote:
| Here's an object example:
|
| I'm developing a fairly ambitious iOS app.
|
| It's a social media-like app, based on aggregate servers.
| I wrote the servers, myself, years ago. One is currently
| a world standard, and heavily in use. The other, was a
| server I wrote a few years ago as a "practice" project.
|
| The app, itself, has been under development for over a
| year.
|
| A big reason for that, is the team had only a vague idea
| of what it would do. It wasn't really feasible to get
| them to hash out a more detailed plan, so I set up a
| "breadboard" system, based on Apple's TestFlight beta-
| test system [0]. TestFlight is usually only engaged
| towards the very end of a project, but we've been using
| it since one month after my initial checkin.
|
| TestFlight requires a fairly "complete" app, that runs
| without tripping any alarms (like crashing). Apple vets
| each release.
|
| I can produce a fairly full-featured app, quite rapidly,
| so I was able to meet Apple's quality bar, pretty
| quickly.
|
| Because we are using TestFlight, we have some significant
| advantages:
|
| 1. The app is _constantly_ at "beta" quality. Even with
| incomplete functionality, what's there, works well. I
| cannot overstate the importance of this.
|
| 2. We can have non-technical stakeholders constantly
| "test driving" the UI, and providing relevant, useful
| feedback, on a continuous basis. They are also highly
| motivated and engaged.
|
| 3. Because Apple is constantly vetting the app, we know
| that we probably won't have any nasty surprises, when we
| finally release.
|
| During this year, we have explored a number of avenues
| that have resulted in "dead ends." I've tossed out
| months' worth of work.
|
| That's great. The best code I write, is the code I don't
| write. I'm happy to tear out huge chunks of code.
|
| The result is that we now have a clear, robust, ultra-
| high-quality app that is in the final stages of
| development. I still have a couple of areas of
| functionality to add, but we're done prevaricating. We
| know where we're going, and everyone is on board and
| psyched.
|
| The quality of the app is also, quite frankly, jaw-
| dropping. We're all thrilled to death.
|
| Using TestFlight has allowed us to work through a lot of
| stuff that is often encountered during early MVP stages;
| without the brand damage and public humiliation.
|
| [0] https://developer.apple.com/testflight/
| peterburkimsher wrote:
| TDD has some excellent philosophy behind it, but I struggle to
| implement it because I get discouraged by failure, and move on
| to other topics.
|
| If the TDD approach said "thank you" for the good parts (code
| structure, not breaking 500 other test cases), that would make
| the 13 errors less scary. One of the things I like about
| software is that it's "soft" - it should be possible to make
| changes without the fear of getting it wrong, and be able to
| fix them _now_ while I see the problem, rather than
| procrastinating until later when I 've forgotten about it.
|
| The same "thank you" system applies when working with external
| contractors (e.g. language translations). We can assume they
| tried their best, and therefore might not know what else to
| say. Reporting 19 problems will get us a reputation for being
| nitpicky, hostile, even toxic. Rather, we should thank them for
| the 730 that they got right, and suggest improvements for the
| 19 that we think might be better.
|
| With thankfulness and encouragement, TDD could really change
| the industry for the better.
| ChrisMarshallNY wrote:
| I like your attitude.
|
| I've always preferred carrots over sticks.
| dgb23 wrote:
| Who do you trust more?
|
| Someone who never makes mistakes or someone who deals with their
| mistakes openly and directly?
|
| We know the first one doesn't exist. Some people like to make us
| believe that, but it's bullshit.
|
| However, one big caveat: Owning mistakes, being open about them
| doesn't absolve us from making them, especially not in the
| future. We still _made_ the mistake. Should we dwell on that or
| get punished? Likely not. Severity, intention and so on matter
| too. But should we simply not feel bad about it? "Embrace
| mistakes"? Not so sure about that either.
|
| When we say "embrace mistakes", what we really mean is "embrace
| learning". Mistakes are opportunities to learn for ourselves and
| others indirectly.
|
| We shouldn't be victims of our bad feelings and beat ourselves
| up. I did that for years. It crushed my confidence and well-being
| into submission and years after I'm still recovering from that.
| But the solution is not to try and make us feel good about
| ourselves regardless. It's not _just_ a psychological problem.
|
| Mistakes happen for a reason, even if they are as simple as "I
| didn't write that down". They happen all the time, we can't
| prevent that, but what we can do is decide what kind of mistakes
| we want to face and ultimately what kind of problems we want to
| solve, be able to solve - for us and others.
|
| Don't get bogged down, don't beat yourself up and don't let
| others beat you up for your mistakes. But make yourself aware
| that you can actually learn from them and decide for yourself
| when you want or need to put the energy into doing that.
| ChrisMarshallNY wrote:
| I have a friend that used to drive trucks for mobsters, in New
| York.
|
| He tells me a story about how he drove his truck under a low
| bridge, and did about $20,000 worth of damage to it.
|
| When he got back, he was expecting to get fired (or worse).
|
| Instead, his boss asked him what he'd learned from the mistake,
| and gave him his next assignment. When my friend asked
| (incredulously) why he didn't get fired, his boss answered "I
| just spent twenty thousand bucks, training you to be careful of
| clearance. You think I'm gonna toss that out?"
|
| My friend says he learned a lot about management from these
| goombahs. They were managing _very_ dangerous people, who were
| big investments, and couldn 't be treated badly or
| disrespectfully.
| RustyConsul wrote:
| This story is an exact mob version of the story Horowitz told
| in hard things about hard things
| suketk wrote:
| Exactly. We tend to underestimate the power of compounding wrt
| learning from your mistakes. Two things that are important
| here: honesty (you can't fix what you don't know) and
| consistency (tight feedback loop = faster compounding).
|
| This extends far beyond software programming. It applies to any
| form of self-development. I actually developed a habit
| formation program framework [0] built off of these principles
| (and a couple others, e.g aligned incentives, bias towards
| action, writing is thinking).
|
| Here are examples of programs using it:
|
| * https://themoai.org/work-life-balance
|
| * https://themoai.org/intentional-technology
|
| [0] https://themoai.org
| webmobdev wrote:
| And sometimes even if you are an expert, you may still make a
| mistake if you aren't in an optimal frame of mind - for
| example, if you didn't sleep or eat properly or had an
| altercation with someone, or your normal routine got messed up
| ... sometimes it is just that - a shitty day in your life with
| nothing to learn but to just go through it.
| qsort wrote:
| I like this point of view.
|
| I get why people are saying things like "normalize mistakes"
| etc, living in some kind of police state environment that
| punishes every misstep is just miserable. _Especially_ if you
| 're working in software, where "no mistakes" is not a
| reasonable standard.
|
| OTOH it shouldn't become an excuse for complacency, and
| especially not an excuse to tolerate incompetence and/or
| totally unforced errors.
| notjustanymike wrote:
| Whenever I start something new, I accept failure immediately. It
| frees my brain to enjoy my mistakes; the more mistakes, the more
| I learn. The worst thing that can happen with something new is I
| do nothing wrong.
| greenyouse wrote:
| If you were working on a production system that had a failure,
| you'd probably want to dig into the why behind the failure and
| take some steps to keep it from happening in the future. Why not
| apply the same to your own failures and treat them as an
| opportunity to improve yourself? I like the idea of tracking my
| own personal performance to help mitigate future problems.
|
| There is probably only going to be a retro for the incident if it
| was something really bad. Even then it's not going to focus on
| you, it'll be more about cross team efforts and higher level
| issues.
|
| Having a personal tracker may help you understand your own weak
| spots, common sources of problems, and trends in your work. For
| each incident look at things like why it happened, what could
| have been done to avoid it, and the context that the mistake was
| made in (e.g. we had three issues going at the time and were
| under a product deadline for X which was due in the next two
| weeks).
|
| To measure performance you could classify the mistake type and
| plot it over time. It could be something general like the
| preventable/complex/innovative buckets stated in the article with
| technical details like: missed the bug during PR review,
| misunderstood the system architecture, missed handling an edge
| case, etc. Having the issues clearly stated for yourself should
| help you not make a similar mistake going forward. If you have a
| trend in the data you can focus your personal development on
| areas where you're weak. Keeping a reference of issues that have
| happened should help with understanding the problem history a
| year later too. That could be good for planning future work since
| you'll know where some of the hard spots might be.
| vesuvianvenus wrote:
| I think acceptance and comfortability with error messages (i.e.
| small failures nearly every step of the way) is the basis of
| one's career as a developer. We start knowing almost nothing,
| producing tons of errors as we go (which reduce relative to
| experience). And learning what those errors mean-- they guide us
| into proper development.
|
| Errors are good given that they lead to learning. Frustration is
| fun.
| MeinBlutIstBlau wrote:
| Learn to not care. Learn to not care about being fired, losing
| everything, and dying in general. For me, the moment was like a
| longer more drawn out experience like that of Office Space. But
| it became that. I didn't care what happened, who screwed what up,
| or what expectations were of me. Fire me if you don't like it.
| Now you're really gonna be screwed having to pick up the slack
| without me there.
|
| Once I unlocked this, the world became flat. I didn't care about
| the journey, just the goal. Whip something up. Learn how to
| socially manipulate expectations. Make it seem like you work.
| Half ass something and convince people there were significant
| difficulties.
|
| None of your code matters. You are paid to support. Not to
| reinvent the wheel. So support it with the easiest code possible.
| Quit trying to do something fancy. It doesn't help when layoffs
| come about. What does is cozying up to VP's and management who
| know you work and get things done. Your portfolio means fuck all
| if not a single person knows what you do, even if you are a
| better programmer.
___________________________________________________________________
(page generated 2021-10-31 23:00 UTC)