[HN Gopher] I Don't Want to Be on Call Anymore
___________________________________________________________________
I Don't Want to Be on Call Anymore
Author : kiyanwang
Score : 80 points
Date : 2021-11-28 12:38 UTC (10 hours ago)
(HTM) web link (www.honeycomb.io)
(TXT) w3m dump (www.honeycomb.io)
| habeebtc wrote:
| This model of oncall (what I currently operate off) is less
| abusive than how DevOps teams were run at my last company.
|
| I would get called on days off, evenings, weekends. As late as
| 1am. I once brought my laptop to a Cubs game, because I knew I
| would get called (and I did). It was mostly because Devs broke
| the build, and their managers refused to have them learn how to
| triage that themselves because it would be extra responsibility.
| jmmv wrote:
| Really liked the article. One thing I'd add (and that I wrote
| about before here: https://jmmv.dev/2021/07/principal-engineers-
| oncall.html) is that high-level engineers should be on-call.
| These engineers (along with management) are the people that can
| most easily make a difference in making the rotation better: and
| not because of their technical skills, but because if they
| "suffer through the pain", they may be able to set priorities to
| fix things.
|
| As some other people have alluded here, though, in some
| organizations engineers seem to "escape" on-call rotations the
| more senior they get, which is unfortunate...
| redisman wrote:
| Get the VPs and Directors on call and see how quickly we'll
| change priorities lol
| Buttons840 wrote:
| > It is engineering's job to own their software
|
| I'll accept this responsibility so long as it comes with the
| corresponding authority.
|
| Now tell me, do I really own the software?
| redisman wrote:
| I have denied your request for a new feature because I have
| ownership over the project and decided we should work on X. Oh
| I don't actually have any power but you think I'm an idiot who
| can't see that? Cool
| datavirtue wrote:
| Ahh the ownership argument, dashed upon the rocks.
| indymike wrote:
| At some point, there has to be someone to push the right button
| at 3AM. That can be done by shift work, it can also be handled by
| on call. In general, a job with on-call requirements should
| disclose that up front, and compensation should be better than
| the same position without the on-call requirement. For small
| teams, on-call may be the only way to do it because there isn't
| going to be a night shift.
| cushychicken wrote:
| Hell no, you're not a monster for not wanting to be on call.
| You're normal. Being on call is fundamentally not that fun.
|
| You're also totally normal if you respond to an incentive
| structure of offering senior level and above employees the perk
| of not having to carry a pager. Some companies use this to hire
| senior talent because, hey, it works.
|
| As a side note: there's a shitload of negative solidarity in this
| thread. That's the attitude of: "If I must eat shit, everyone
| else must eat some too". I think this article (while a little
| flawed) makes a good pivot towards positive solidarity, and
| asking the question: "Is there a way to achieve this goal where
| everyone eats less shit?"
| dmitrygr wrote:
| I ask every team I interview with whether they have any sort of
| on call. A "yes" answer terminates the interview, unless quickly
| followed by "but we can make an exception". Call me a monster if
| you wish, but my time is mine and my family's.
| snicker7 wrote:
| My team is distributed across the world. We ensure that we have
| multiple SME's for the most critical systems in each region. No
| one has to wake up at 3 AM.
| willcipriano wrote:
| My argument is that as the tech side of the organization takes on
| more and more of the operation of the business, it also needs to
| be taking on more of the rewards.
|
| For example if the expectation is that a engineering team
| designs, builds and maintains the machine twenty four hours a day
| that is responsible for the majority (if not all) of the firms
| profits then the majority of the rewards must flow there.
|
| One the other hand, traditional methods for doing this have
| fallen by the wayside. With performance overhangs and other
| chicanery, what was once a lottery ticket is now a lottery ticket
| that someone will almost certainly try to steal from you once it
| becomes worth something. Instead of rewarding engineers who keep
| the lights on, we make them look for another job every 2 years to
| keep with the market.
|
| People like to say tech is eating the world, tech is doing
| virtually all the killing but the hangers on seem to do the bulk
| of the eating.
| shoo wrote:
| Ah, capitalism! The company makes surplus profits that can be
| extracted by shareholders (and executives and employees,
| although these get labelled as expenses, not profits) if it is
| able to sell goods or services at a price point much higher
| than the cost of producing the service.
|
| The cost of keeping the machine running is an operational
| expense. There is supply and demand in the labour market. If
| the team keeping the machine running generates $100m of profits
| for the company and costs $5m in compensation, but it is
| possible to hire a similar team at market rates to do a
| comparable job for $5m , then it doesn't make much sense to pay
| the team more than $5m. The excess value of $95 m generated by
| the team's activities can be captured as profit.
|
| It's also important not to neglect the contributions of other
| roles. The machine is only able to produce revenue because it
| is matched with customers who are a good fit for the service
| the machine delivers and are willing to pay for it. If there
| are no customers then the machine and the engineering team that
| support it produce no revenue. So arguably the sales team that
| is able to identify potential customers and help them
| understand if renting access to the machine could help them
| also provide a large amount of value to the company.
|
| If you take the same high performing engineering team that
| builds and operates a valuable service in one company, and drop
| them into a different business context where there are simply
| much fewer paying customers, or the sales team isn't able to
| deliver, then the value the engineering team are able to
| generate in the new situation may be zero or negative. Maybe
| most of the value generated by the service isn't due to the
| team who built and maintain it, but the surrounding business
| context. Leverage is a big thing. The highest paying roles are
| often in situations with more leverage (in huge orgs or huge
| projects), not because the work is necessary more skilled or
| harder than work in other smaller scale situations.
|
| If some of the salespeople and engineers believe they are
| getting paid disproportionately little compared to the value
| they create, they could consider banding together and starting
| their own venture where they would be the owners, not the
| employees. Much easier said than done, and much easier to
| stomach if starting in a situation where they have the
| financial safety net and career connections so they'll be okay
| if the business fails.
| teeray wrote:
| > My best idea is 'pay people more,' but that doesn't sound so
| compelling as a pitch.
|
| I would not accept a position that paid for on-call, even if they
| paid more. That just legitimizes encroaching on personal time and
| likely will make it happen more often. Pay me instead in
| equivalent PTO (or greater) so that there is a stronger forcing
| function to reduce pages.
| b3morales wrote:
| I like this, but of course it collides with the trend towards
| only offering unspecified (so-called "unlimited") PTO.
| teeray wrote:
| I also add a mandatory annual minimum PTO rider when it's
| "unlimited"
| dilyevsky wrote:
| At google you could choose either 1/3-2/3 of your hourly pay
| (base) for each hour of being oncall outside of 9-5 or
| equivalent number of hours of additional pto. Best oncall comp
| structure I've seen ever since. Somehow people like to parrot
| only employee unfriendly practices from big corps...
| mindvirus wrote:
| I think the challenge that organizations run into with on call is
| that the costs of it are hidden and delayed - reduced
| productivity and lower retention. I think that because of this,
| organizations do not invest in keeping their systems in order.
|
| The question is, what do you do about it? At some point things do
| have to escalate to an engineer, and I don't think it's practical
| to have a 24/7 engineering team staffed for every service
| (although having 24/7 non-eng support is very reasonable).
|
| The best solution I've seen is having error budgets and on call
| compensation commensurate with required response times. Error
| budgets since they balance maintaining and expanding the service,
| compensation to respect people's time.
|
| What have you seen that works?
| supernovae wrote:
| If your business requires 24x7 coverage, you should pay for
| 24x7 coverage and not expect the same people working 8x5 are
| the ones who will do it "because it has to be done".
|
| I once worked for a tech company that said they had follow the
| sun rotation and i was dumb enough to take the job only to find
| out that the other countries had strict labor laws and follow
| the sun didn't mean the US folks would have their nights and
| weekends, but that the US folks were doing off hour support for
| the rest of the world because we don't have labor laws that
| protect our workers rights. I literally had to keep a seat warm
| from 8-5 my time to do the bulk of my work from 5-9pm and on
| weekends to "work around the customers" and they failed
| miserably at making tech work... (fin tech.. where they through
| money and soul sucking work at every problem)
| redisman wrote:
| Shift work has been a thing for decades. I would imagine hiring
| some percentage of your devs for the night shift wouldn't be
| that hard as many people enjoy that lifestyle. Then you already
| cover the majority of the 24 hours with normal work
| PicassoCTs wrote:
| On call is a dysfunctional company culture attacking its own
| workforce in a battle of attrition. Its the equivalent of
| rheumatic fever to a company process. If you need it, your
| reliability engineering is failing and the process that should
| mend it, is instead busy fixing symptoms.
|
| Such a company is already dieing and will do once the team "on
| call" is used up, burned out and gone.It will be replaced in the
| market by something more vital, still capable to value its
| workforce.
| ckdarby wrote:
| Are you saying, every fortune 100 tech company's reliability
| engineers are failing to do their jobs?
|
| MAANG all have on-call individuals even with the fact they're
| across all timezones and have been growing steady.
| nzmsv wrote:
| There are different ways of handling on-call and the
| practices at the companies you mention are very different, as
| is the on-call load.
|
| There is a world of difference between someone voluntarily
| accepting on-call as part of their duties in exchange for
| higher pay or a 4 day work week, and on-call being forced on
| everyone.
| supernovae wrote:
| He's saying we're all suckers... and sometimes, it takes a
| long time to find out how true that really may be.
| dopylitty wrote:
| This misses the key point that on-call itself is an abusive
| practice regardless of how well an organization practices it.
|
| If an organization thinks their systems should be available 24x7
| they should staff people 24x7.
|
| There is no situation where it's ok to contact someone to do work
| outside of their work hours.
| [deleted]
| echelon wrote:
| > There is no situation where it's ok to contact someone to do
| work outside of their work hours.
|
| This lacks perspective.
|
| Imagine that your system runs banking, healthcare, telecom,
| etc.
|
| Of course someone needs to be on call 24/7. The best people to
| hold the pager are the engineers that work on the system.
| shoo wrote:
| that's a pretty selective quote. you skipped over:
|
| > If an organization thinks their systems should be available
| 24x7 they should staff people 24x7.
|
| One way to achieve 24x7 coverage without anyone being
| structurally on-call outside of standard office hours could
| be "follow the sun" support -- have offices in multiple
| timezones around the world and run 3 x 8 hour shifts each day
| of support working their local office hours to provide 24/7
| coverage. Still need enough staff at each site to handle
| rotation for covering weekends and for people to go on
| holiday and get sick or quit for new jobs etc.
|
| I can understand why many companies would prefer not to pay
| for that if they can get away with cheaper alternatives, and
| why they might prefer to understaff and push the burden onto
| employees, but that's the company's problem, not the problem
| of the individual worker who trades their life by the hour in
| exchange for money.
| echelon wrote:
| If you're not a global empire with offices all over the
| world yet, the simpler thing is to pay your engineers well
| and tell them being oncall is one of the responsibilities.
| emerged wrote:
| It seems perfectly okay if you were hired at a job where on-
| call is a stated requirement. Is this a controversial opinion?
| I'll admit that I sometimes find myself out of phase with some
| of the dominant HN views on at-will employment, but it would be
| useful for me to know what the perceived flaw in my stance is.
| monksy wrote:
| No.
|
| If a job is expecting you to be oncall is the part of your
| job, they are looking for ways to not pay you for the work.
| (This is not talking about an occasional once a year
| expectation. ) This is a rotating or permnant basis of where
| you are expected to be ready to work and they're not paying
| you.
| nouveaux wrote:
| Doctors are on call. Firemen are on call.
|
| There are plenty of jobs that are not on call. Lots of
| enterprise positions do not require you to be on call. If you
| work on a site that requires 24/7 uptime, why is it so hard to
| accept that engineers have to be on call?
| hirundo wrote:
| "There is no situation where it's ok to contact someone to do
| work outside of their work hours."
|
| Here's one. I get paged for urgent fixes maybe three times per
| year. To staff all three shifts with senior engineers to cover
| for that would be enough to make the company unprofitable and
| lose many jobs. I don't much like being woken in the middle of
| the night for such things, but it's clearly necessary and not
| abusive.
| dopylitty wrote:
| If they can't afford to staff for 24x7 operation and maintain
| profitability that's their problem. They should either reduce
| their service hours or charge more.
|
| By working for free you're subsidizing your organization's
| unsustainable business model.
| jsolson wrote:
| Below a certain level I mostly agree with you. One company
| in my past didn't work this way, and one too many pages in
| the middle of the night for a business I wasn't personally
| invested in made it an easy job to walk away from.
|
| That said, there is a level of seniority at which "their
| problem" is "your problem." If you'd chuck the business out
| the door rather than getting paged a few times a year,
| you're not ready for that level yet. This varies by company
| and organization size, but fundamentally you can't (and
| shouldn't try to) anticipate everything up front. Sometimes
| you need the knowledge, judgement, or simply signing
| authority (literal or metaphorical) of someone specific.
| lightbendover wrote:
| Getting paid 200k+ for a job that had on-call as part of
| stated duties (and thus baked into the comp) prior to
| accepting the offer is now working for free? There is
| hyperbole and then there is whatever this is.
|
| Don't like it, quit and find a company that is moving
| slowly and doesn't value ownership from engineers, someone
| else will fill in the vacant post. Sounds like a reasonably
| sustainable business model to me, especially since it has
| worked like this for over decades at some companies.
| UncleMeat wrote:
| The number of pages is only part of the story. Even if you
| never get paged a single time, oncall still sucks because you
| have some response requirement. If I _could_ get paged and I
| need to respond within 30m then it is no hikes for me this
| weekend.
| [deleted]
| LandR wrote:
| Yeah,my issue with being on call is that i can't ever really
| let loose if I'm expected to be contactable anytime
| 24/7...what if I'm out with friends, drunk, high, on the side
| of a mountain? Etc.
|
| My hours are 9-5, after that is entirely me time and if you
| expect me to br contact able and in s state of contribute
| you're going to be disappointed.
|
| No amount of money is worth losing that me time.
| je42 wrote:
| Usually, being "on call". should mean no alcoho, and have
| inet + computer closed by i.e. 5-15min or so.
|
| which also means that this should be time that is paid on
| some hourly rate for the lower quality of live during the
| on-call time.
|
| Also, there should be a rotation, with enough people to
| make this bearable.
| wildrhythms wrote:
| Are you describing the current state of on-call? Because
| that is what it is.
|
| There should be no rotation; there should be no on-call
| in the first place. If hiring staff to work normal hours
| is too expensive for the company, then the company has
| bigger problems than a service going down.
| alistairSH wrote:
| The article explicitly states that's reasonable. It's the
| rotating pager duty that's unreasonable.
| [deleted]
| xtracto wrote:
| I am quit amazed at the general reaction of this forum to
| this standard industry practice.
|
| If you want a cushy 9-5 job, go work for a code factory like
| HCL, TCS and whatnot. That way you can build crap and wash
| your hands after handling it to the customer.
|
| In a startup environment, you gotta build your shit and
| maintain it to. It's an environment that's not for everyone.
|
| I've been in oncall roosters several times in the past. And
| it made freaking sure we wrote the best software we could.
|
| I also have been on the management side of it, and the way I
| set it up is that whoever was the oncall engineer, if he had
| to work for more than 2 hours after normal working hours, he
| would get an additional PTO day. It actually worked quite
| well.
| coldtea wrote:
| > _If you want a cushy 9-5 job, go work for a code factory
| like HCL, TCS and whatnot. That way you can build crap and
| wash your hands after handling it to the customer._
|
| You got it backwards: if you want a crappy quality of life
| (or have no life and you only identify with the company
| that sees you as a replaceable cog), and crappy products
| and customer service from a company that doesn't care (to
| plan right, to hire accordingly, to treat its staff right)
| go on call and overwork, producing sleep-derived crap.
|
| Craftsmen and artisans take their time and have boundaries.
|
| "Feature factories", startups, and mass market crap
| companies forego 9-to-5 and indoctrinate naive employees
| that they do something important by doing so.
|
| > _In a startup environment, you gotta build your shit and
| maintain it to. It 's an environment that's not for
| everyone._
|
| Yes. It's for starry-eyed naive youngsters right off the
| bus. The kind of people to believe they're "changing the
| world" by building a Facebook or Groupon.
| vasco wrote:
| > Craftsmen and artisans take their time and have
| boundaries.
|
| I've been an engineer on-call for 7 years since
| graduating. In the meantime I handled a handful of
| business critical situations, developed my ability to
| keep cool during crisis and put my skills to the test.
| When I started doing it, I wasn't even paid for on-call.
| Now I am. The money has no bearing on me having been
| exploited or not. I've done it because it's cool, and
| when I think it's too tiring I won't do it anymore.
|
| Am I less of an engineer, not a true craftsman (wtv that
| means), because of this? There's other opinions in the
| world, no need to be so close minded.
| yawaramin wrote:
| When you were doing on-call without being paid, that's
| called wage theft. It's the single biggest kind of theft
| in North America: https://www.epi.org/publication/wage-
| theft-bigger-problem-fo...
|
| It's also the kind of theft that insidiously convinces
| you that it's really cool and that you're actually OK
| with it because you're learning how to 'keep cool during
| crisis'. You know how else your employers could have
| taught you that? By providing actual training during
| working hours, while you were being paid.
| SpicyLemonZest wrote:
| I mean this with all the respect in the world, but I
| think you've gotten yourself caught in an ideological
| bubble here. To someone who doesn't already agree with
| you, this comment sounds like an argument against the
| concept of wage theft more than an argument against
| oncall. After all, you can't prove something's bad just
| by classifying it under a nasty-sounding label!
| yawaramin wrote:
| I purposefully did not make an argument against on-call,
| although I might have easily done that and others have in
| this thread. All I said is that doing on-call _without
| pay_ is wage theft. And this is not an ideological bubble
| thing, as anyone will quickly understand if they ask a
| plumber to do on-call for their house plumbing system
| without pay.
| foldr wrote:
| >In the meantime I handled a handful of business critical
| situations, developed my ability to keep cool during
| crisis and put my skills to the test.
|
| That's great, but you could still have done all those
| things if you were being compensated appropriately for
| the extra time you were putting in.
|
| It's like unpaid internships. I'm sure unpaid interns
| often do learn useful skills and make useful connections.
| But the practice is still exploitative.
| coldtea wrote:
| > _The money has no bearing on me having been exploited
| or not. I 've done it because it's cool_
|
| Convincing them of the "coolness" of it is the most
| common way to exploit the naive/fresh.
|
| The same people who do so to others, wouldn't even piss
| if they weren't compensated for it...
|
| (And being exploited or not is not a personal decision.
| If you aren't compensated for overtime, then you are
| exploited. It just means you're ok with it.).
| np- wrote:
| I think most experienced engineers realize on-call is a
| complete fool's game -- you will never get accolades for
| fixing shit at 2am. Literally never. Your only "reward"
| will be to become the escalation point to continue getting
| called at 2am. It's not only mentally demanding work, but
| also bottom-feeding work that gets no recognition. No one
| is doing their best work after being woken up in the middle
| of the night, it's all band-aid hacky fix shit that is
| being generated. Also in general the type of personality
| who would tolerate working 2+ hours beyond their working
| time is probably the type of person that isn't even going
| to use up all of their PTO to begin with, so I always find
| those kinds of rewards laughable.
| yawaramin wrote:
| You do understand that most startups fail, right? Clearly,
| on aggregate, their practices are not really working or
| worthy of imitation. For the ones that succeed, there's a
| huge element of luck. Otherwise people would make
| guarantees about success.
| whycombinatore wrote:
| On call typically means you gotta be online and respond
| within 15 minutes if paged. That's abusive. If you want a
| person to do that, then pay them for that. As an example, on
| call in Amazon works the way I described, and yes it is
| abusive.
| sdfhsdrhsd wrote:
| Unless you're working hourly or the on-call duties weren't
| disclosed during salary negotiations, you're already paid
| for it. I'll happily take a call after hours if it means
| I'm in a cushy job being paid 200k/year to solve puzzles.
| My partner manages a Safeway and makes well under half what
| I do for easily twice the work, and he's effectively on-
| call 24/7 for his store. Is either situation ideal? No, but
| I recognize that even with the injustice of having to ack
| an occasional alert at 2am, I'm in a crazy fucking comfy
| spot.
| ActorNightly wrote:
| They do pay people for that - the salaries at MAANG are
| inflated as is.
| syshum wrote:
| >>On call typically means you gotta be online and respond
| within 15 minutes if paged.
|
| No place I have ever worked has this kind of oncall policy.
|
| Most I have seen have a response time of 1hr, note
| resolution, but response.
|
| Amazon is large enough they should have people staffed 24/7
| just by staggering the timezone codes at the different
| offices. This is how Cisco TAC works, you call them you get
| what ever timezone code is "day time" at that time.
|
| So I would agree that 15min on call is abusive, luckily
| this is not my experience as "typical" in the industry,
| hell even fully staffed 24/7 call centers typically do not
| have a 15min response time
| conradfr wrote:
| In France you're necessarily compensated for being on call
| and then paid when you are paged.
| the_jeremy wrote:
| I interviewed at Brex, and they required getting online
| within 5 minutes of being paged. (I did not get the offer,
| but I would not have accepted the offer after learning
| that.) Where I currently work the response time is "ideally
| within 15 minutes", which is a lot more manageable, since
| plenty of places I like to go are within 15 minutes of my
| house. My job pays more than average because of on-call,
| it's 1 week out of 6, and there's probably an average of 1
| call out of business hours per week (mode is 0).
| sokoloff wrote:
| One argument is that you _are being paid_ for it; it's just
| not broken out into two lines, but the dollars are there.
|
| If you're making $200K/yr, maybe $150K of that is for your
| job and $50K is for being on-call. That seems like a more
| than fair price (or at least one that's in the ballpark)
| for periodic on-call service.
| kelnos wrote:
| In my experience, often people paid _more_ end up not
| having on-call duties. I know several people who have
| negotiated high salaries from the start of their
| employment along with a stipulation that they will never
| be on the on-call rotation.
|
| In my own situation, I removed myself from on-call duties
| during a leave of absence, and never went back on after I
| returned to work. Since then, I've still gotten the same
| raises I got before.
| lumost wrote:
| An unfortunate reality of on call is that companies care
| about projects delivered, not disasters averted.
|
| Unless your stepping in to right the ship when a company
| is on fire operationally, and can point to specific
| action/results you delivered to right the ship - getting
| pages 8x per week does nothing for your career.
| redisman wrote:
| If that was the case we could easily see it in the
| salaries of on-call versus not on-call positions. This is
| clearly not the case.
| [deleted]
| exsmelliarmus wrote:
| > If an organization thinks their systems should be available
| 24x7 they should staff people 24x7.
|
| I'm a software engineer. I'm paid well partly because things
| like oncall are necessary. It's priced into the compensation.
|
| > There is no situation where it's ok to contact someone to do
| work outside of their work hours.
|
| It sounds like you should find a company that agrees with this
| opinion. I don't agree.
| koheripbal wrote:
| I partially agree. It really depends on how often I'm called
| when on-call.
|
| There is a world of difference between being called twice per
| year, and twice per week.
|
| ...and it's fine for that to be priced in as long as those
| expectations are as clear as the salary when I take the job.
| coldtea wrote:
| > _I'm a software engineer. I'm paid well partly because
| things like oncall are necessary. It's priced into the
| compensation._
|
| Or you priced yourself down.
| JackFr wrote:
| Being on call is separable from your role as a software
| developer and can be compensated separately. My wife's (non-
| technical) organization has a requirement for a junior and a
| senior duty officer to be available 24/7/365. These shifts
| are compensated and available for people to sign up for. If
| shifts don't get filled, they fall to _senior_ managers in a
| rotation. You can be sure these managers somehow pay enough
| that they only very rarely need to take shifts themselves.
| [deleted]
| tragictrash wrote:
| Haha, That's why you still have to do on call.
| nzmsv wrote:
| This attitude is very common with software engineers and I
| find it baffling. Is it some kind of inferiority complex? I
| doubt Warren Buffett worries that he is overpaid and demands
| to be woken up in the middle of the night for some self-
| flagellation. So why do so many software engineers think "I
| am paid well so I deserve whatever the company throws at me"?
| You are selling something: your skills, expertise, and time
| on this planet. These things are limited and valuable.
| Negotiate the price and terms of the sale!
| rocgf wrote:
| I also find it baffling that people still think about
| things the way you do.
|
| It's not necessarily some kind of inferiority complex or
| anything else, it's about the free market.
| coldtea wrote:
| "Free" market just means people bound by monetary
| considerations...
| rsj_hn wrote:
| It means no one has to trade with you if they don't like
| what you have to offer in return. If you consider that
| "bound", then OK, but that language doesn't illuminate,
| it hides.
|
| It is a painful slap in the face when the market doesn't
| price what you have to offer as high a you like. I know a
| friend who was convinced that her calling was to be an
| artist, but always had a hard time selling her works, and
| often complained about the unfairness of life. I guess
| she was "bound" by monetary considerations by not being
| able to make ends meet as an artist. Eventually she gave
| up and became a hospital lab tech and is well
| compensated. So the market was telling her that her
| skills as a hospital lab tech were much more valuable to
| society than her skills as an artist, even if her own
| preferences were otherwise.
|
| At the same time, some other artist can buy an entire
| oceanside condo for one painting, because the market does
| value their output very highly.
|
| That's all that we're talking about here. It is "free"
| but that doesn't mean that you'll be able to get whatever
| you want in exchange for your own output.
| nzmsv wrote:
| The free market works the other way too though. If lab
| techs were in high demand (as they are), and your
| friend's employer was demanding an unreasonable schedule,
| she'd be free to jump ship to a better job. Right now
| software engineers have that kind of market power, so why
| not use it? If tomorrow the market decides we are
| overpaid, we'll either accept it or change jobs like your
| friend did. What I don't get is not using the power the
| free market gives you because?...
| staticassertion wrote:
| Lots of other fields have on calls or out of work
| expectations. That's part of the job, and you're
| compensated highly.
|
| It's not like on call is a shock or something, it's not
| "whatever they throw at me", it's just... part of the job.
| My sister is a dentist, she has to be on call sometimes,
| it's not a shocker or something you don't know when you're
| signing up for the job.
|
| You're implying that people aren't negotiating, but that's
| baseless. Software engineers are highly compensated because
| of these expectations, and we all negotiate accordingly.
| wildrhythms wrote:
| >we all negotiate accordingly
|
| This is quite a sweeping statement to make. My first
| engineering job salary was non-negotiable. I was told to
| either accept it, or they 'rapidly' move to another
| candidate.
| option_greek wrote:
| I would say its a superiority complex : thinking software
| engineers are paid well and hence need to be ready to
| sacrifice more to maintain this superior position.
|
| And of course also because the risks of bad or interrupted
| sleep is not recognised widely and mostly ignored.
| drewcoo wrote:
| Which is why you can also see the exec team paged in the
| middle of the night, right? Because they're paid well and
| need to sacrifice?
| joshuamorton wrote:
| Yes, and had their vacations cut short and other such
| things than rank and file employees would tolerate far
| less.
| SpicyLemonZest wrote:
| I've definitely seen my exec team get paged in the middle
| of the night, and they didn't give any indication that it
| was unreasonable or uncommon. If anything they seemed to
| enjoy it, which to be honest I do as well. Being woken up
| to solve a problem (on rare occasion) can be validation
| that you're an important person working on important
| things.
| gladinovax wrote:
| Clearly you don't think very highly of yourself. Sad.
| dang wrote:
| Can you please not create accounts to post flamewar
| comments to HN? It's not what this site is for, and it
| destroys what it is for.
|
| https://news.ycombinator.com/newsguidelines.html
| [deleted]
| redisman wrote:
| Gross
| strken wrote:
| Every oncall rotation I've been in for the past five
| years has had C-suite executives in it. Earlier the CTO,
| currently the CEO.
| [deleted]
| skybrian wrote:
| While I agree that in general they shouldn't feel bad about
| asking to be being Paid More (tm) you are making
| unwarranted assumptions about how happy they are with their
| current employment situation. You don't know them, don't
| know how much they're paid, and don't know how well or
| badly they deal with being on call.
|
| There is nothing immoral about deciding you're happy.
| nzmsv wrote:
| I never said it's wrong, I just said I don't understand
| it. I think my post sounds a lot less moralizing than the
| one I am replying to. On-call is not some kind of basic
| virtue. It is simply part of a job description, and
| that's a business contract.
|
| I personally think that a much healthier alternative
| would be to look for a position where one can maintain
| their physical and mental health, and give back extra
| income to worthy causes. There are many organizations
| much more worthy of my time and money than a bunch of
| managers who are too cheap to hire people for a follow-
| the-sun support org. But I'll accept a job with on-call
| if I think the rest of the deal outweighs the negatives.
|
| It's still fair to call out on-call as a negative though.
| It's something to be aware of when accepting a job and
| for companies to keep in mind when recruiting.
| coldtea wrote:
| > _There is nothing immoral about deciding you're happy._
|
| It is when it sends a signal (about the job, market, etc)
| that also affects others.
| shukantpal wrote:
| No it is not. Because your signal is the truth for you.
| dcow wrote:
| Under what moral framework?
| nzmsv wrote:
| Not moral, practical. The majority of places that do on-
| call do it because it's the default.
|
| That's the problem with on-call: it somehow took on moral
| undertones. And as a result it does not feel safe to
| speak out against it.
|
| If on-call was widely seen as a negative (that a few
| people like because it gives them a sense of importance,
| more power to them) then there would be far fewer
| companies pushing for it as the default. As it stands,
| most people suffer silently for lack of an alternative.
| And the first step towards change is to make it OK to
| publicly say that on-call's a negative, a health hazard,
| and other options exist (though they may cost more).
| skybrian wrote:
| If you believe that software engineers are not paid
| enough and it's morally important to do something about
| it then apparently I sent the best signal of all by
| retiring and thereby increasing demand for everyone else.
| But since this is absurd, I won't pat myself on the back
| too much.
|
| More generally, free markets are doing absurd things all
| the time (see Matt Levine) which makes it hard to extend
| moral reasoning very far without getting the equivalent
| of divide-by-zero errors. So I don't think we should be
| all that concerned about how it affects the job market in
| general when negotiating with an employer. You know what
| you want better than you know what anyone else wants. Ask
| to be Paid More (tm) if that's what you want, but if you
| don't want to, you don't have to and people saying there
| is some kind of moral imperative to try to become even
| more wealthy at a faster rate can be ignored.
| taormina wrote:
| Feel free to drive down your own value. This is the exactly
| what people are talking about when they say that they are
| looking for young and naive employees to exploit who won't
| know better. You're paid the big bucks because you have a
| very in-demand skill. Being dumb enough to answer a call at
| 3am is entirely orthogonal.
| abc_lisper wrote:
| Please heed this. This is the lesson you don't want to
| learn from experience
| philosopher1234 wrote:
| Why not?
| haswell wrote:
| Speaking from experience, the ensuing burnout and lack of
| sleep leading to other health challenges isn't worth the
| "life lesson".
| abc_lisper wrote:
| Because, you undervalue your time, you open yourself for
| abuse. May be that project that is behind can be done by
| you after work, as you sometimes do work then anyways. I
| have seen some managers who are acutely sensitive to this
| and repeatedly use this to their advantage while charming
| the underlings. Next, being on call means it is difficult
| to plan that long vacation, or a unplanned night of shit
| faced drinking with your buddies, or unable to sit with
| your kid in the night when she is sick. A perspective of
| life as unpredictable events helps reinforce this idea.
| You are, with out compensation making life predictable
| for others while making it less predictable to you.
| Third, your mental and physical health. Sleep disorders
| because of restlessness, groggy mornings and the stress
| that comes with normal working hours will take a toll
| over long term.
| willcipriano wrote:
| I was on a team that had lost a lot of it's members. We
| worked hard to keep up and were on track to meet
| deadlines. Some time went by, we were working nights and
| weekends and every time we asked we were assured that
| they were hiring but "couldn't find the right person". At
| the same time, every week we would hear about a new hire
| in management. They were magically able to find some new
| hires as soon as deadlines started to slip, had we not
| worked those nights and weekends I am certain that they
| would've found someone to hire much earlier (the first
| guy they did hire said the process took three months,
| they were clearly dragging their feet). Too bad I learned
| that lesson after already spending the day of my
| daughters birth in long meetings about testing
| environments.
|
| Why would they pay for something they can get for free?
| taormina wrote:
| Let's do the laziest back of the napkin math. Let's
| pretend you make $100k and work 2000 hours a year. You
| make $50/hr. Let's say you do 50 hours of week between
| trying to impress your boss and getting woken up to do
| call. Congratulations, you are now a $40/hr employee!
|
| Also, in my experience, the people who aren't getting
| woken up at 3am (aka. your boss) don't value the time and
| effort needed to stabilize these systems. So they want
| you working on the new shiny product feature that will
| get them promoted. Fixing the crappy data pipeline that
| shouldn't alert every other night isn't a priority when
| you aren't the one being woken up.
|
| Well, it should be a priority, but I also don't work at
| that previous job for this exact reason. I took a nice
| pay increase and have never had to be on-call. Don't
| settle.
|
| EDIT: Original 2000 hours comes from 40 hours a week * 50
| weeks. Like I said, back of the napkin math here.
| staticassertion wrote:
| > Congratulations, you are now a $40/hr employee!
|
| Another way of thinking about this is that you've
| invested 10 dollars of your paycheck into your career. I
| worked way more than 50 hours a week when I started my
| career. Subsequently, I was able to drop out of school
| early (by 2 years, saving 10s of thousands of dollars),
| get a full time job, increase my compensation by 50%
| within 2 years, and then by 200% the following year when
| I changed companies.
|
| By your math I'm sure I was getting paid 1/3rd of my
| actual compensation. But that has more than paid off in
| terms of the investment.
| Volundr wrote:
| > Also, in my experience, the people who aren't getting
| woken up at 3am (aka. your boss) don't value the time and
| effort needed to stabilize these systems.
|
| There's the rub. The issue isn't being on-call, it's not
| prioritizing making sure the system is robust so that on-
| call is boring.
|
| In my last job there was no formal on-call, but if shit
| was going bad I'd be expected to resolve it, or track
| down the right resources to do so. In my current there is
| a formal on-call rotation. In my 15 years at the previous
| job I probably got called out of bed 3-4 times (and due
| to my roll I was the first call, if anyone got woken up,
| I did, and then had to wake anyone else needed). In my 7
| months in the new job it hasn't happened yet.
|
| When everything is on fire, it feels obvious to me that
| asking your $x00k employee to put it out isn't
| unreasonable. What is unreasonable is making that the
| plan instead of having robust fire-prevention systems and
| making that the exception.
| skybrian wrote:
| Downvoted for insulting someone you don't know anything
| about.
| NullPrefix wrote:
| same
| coldtea wrote:
| Downvoted the above for hollier-than-thou preaching and
| missing the point.
| [deleted]
| gh0std3v wrote:
| > It sounds like you should find a company that agrees with
| this opinion. I don't agree.
|
| I agree that on-call is necessary in certain professions
| (e.g, doctors). I also agree that if an employee is willing
| to do on-call and they are compensated accordingly, then the
| practice is still ethical.
|
| However, to call someone to do work outside work hours is
| unreasonable. On-call is considered work time, so I am
| expecting to be contacted during that time. However, if I'm
| not on-call, then it is not time for me to work, and I
| shouldn't be contacted by my company and feel pressured to
| answer the call.
| UncleMeat wrote:
| But loads of software engineers don't have oncall. And these
| jobs don't pay less as a rule. If you aren't _at least_
| getting overtime pay, then you are being scammed.
| thrower123 wrote:
| Industrial companies that have an assembly line go down call
| people in to fix it at time and a half or double pay.
|
| Something is a little fucked with tech for doing it for free.
| booleandilemma wrote:
| The tech industry is filled with a special kind of person who
| thinks they are the best at what they do. They get paid a lot
| (for now) so they make the mistake of thinking they're at the
| same level of real professionals like doctors, lawyers, and
| engineers. The industry encourages this delusion by throwing
| around essentially made-up feel-good titles like "software
| engineer", "senior software engineer", "staff software
| engineer", etc.
|
| When these people begin to doubt themselves and their line of
| work, they're told they have "impostor syndrome" and
| everything is fine.
|
| The reason the industry does all of this is because it
| discourages their workers from forming unions. They trick
| their workers into thinking unions will slow them down.
|
| There will come a day when the industry is flooded with
| enough saps that their high salary is pulled out from under
| them and they'll understand the reality of the situation.
| They'll be left working for pennies and at odd hours of the
| day.
| Der_Einzige wrote:
| That day will literally never come. Software is too hard
| for average people to get good at in numbers that will
| displace our salaries.
|
| Covid remote jobs are incapable of lowering USA salaries
| despite the fears of outsourcing. Your fears of people
| thinking they are better than they actually are are wrong -
| and so is the implication that "we will get what we
| deserve" - well, unless what we deserve is another doubling
| of compensation for switching jobs every 2 years...
| vasco wrote:
| And at that point unions will make sense, like in any other
| low paid, undifferentiated type of work where the worker
| has a really hard time "just changing jobs" when things get
| abusive. That is not the situation in IT now.
| datavirtue wrote:
| They don't have a union so they hoover up the boss' arbitrary
| bullshit. Very simple. Most people just comply. They are the
| worse because they set unreasonable expectations for everyone
| else. Unions protect you from the other emoloyees' bad deals.
| dylan604 wrote:
| Submit an invoice for a minimum 1 hour charge for any after
| hours work/calls. Devs need to start billing like lawyers.
| HarryHirsch wrote:
| The "personal responsibility" narrative has seeped far to
| far into American life to be tolerable. Back then, we had
| industrial operations that were running 24 hours/day,
| things like factory assembly lines, steelmaking, the
| electric and gas utilities, and they had on-call rotas. But
| then we also had proper engineering practices, none of this
| "take ownership" crap, which reduced incidents, and we had
| unions that were in a position to push back on employer
| demands.
|
| Instead we have self-indulgent junk like that. Wonder what
| my uncle would say, he was chief engineer for the gas
| company back in the day.
| bengale wrote:
| I always managed to get out of doing any on call work by
| pushing this point. There was a price for being put in the
| list of numbers that would reach my phone during non work
| hours and then further charges if I had to answer a call.
| Was always a pretty tense discussion with managers but they
| never wanted to pay for it and I never worked for free.
| rubinelli wrote:
| You should also be compensated for the time you spend on-call
| (which means you can't take on other commitments you
| otherwise might.) Anything less and it's outright
| exploitation. "We pay you well" is a non-sequitur.
| ianai wrote:
| "We pay you well" just means "the labor market favors us so
| much that we can hire a replacement for you to do it."
| Charitably, it's because their pay is above a willingness
| to work threshold in the market. As a market participant,
| the supply side of labor typically is too large, not at all
| organized, and lacks information and any market power to
| price these things into valuations.
| ploxiln wrote:
| > "We pay you well" is a non-sequitur.
|
| Not at all. The choice might be between $60k salary plus
| around $20k for overtime in one career, vs another career
| with $200k+ salary plus crazy benefits (gym memberships, a
| variety of mental health services, $20k in fertility
| treatments (no joke), 3+ months paternity leave, over 10
| more things I can't even name they make no sense...)
|
| I don't even use any of the benefits, just the money (and
| the parental leave) but calling a few on-call nights per
| month "abuse" or "exploitation" is just ... without
| perspective. Nobody that does real work has it this cushy -
| not doctors, not teachers, not construction workers. Maybe
| aristocratic political appointees or something, but jeez,
| no field that is accessible to anyone with an old laptop
| and sufficient motivation. This is as cushy as it gets, for
| a _real_ job. Enjoy it while it lasts.
| credittw2021 wrote:
| Agreed.
|
| My brother makes about what I do, but he is an MBA and a
| plant manager with background in logistics.
|
| Sometimes he gets calls and has to go in to work. Hell
| there has been at least one instance where he had to drop
| everything and courier a bag of parts on a commercial
| flight so a line wouldn't stop.
|
| That said, I have worked in 'abusive' on call
| environments. I.e. we had a system that would break at
| least once a week between 3am and 6am (like, multiple
| times in that period). But because the oncall labor to
| remediate was 'free', fixing the problem was never a
| priority over the 2 years I worked there.
| [deleted]
| sidlls wrote:
| It's the "superhero" complex so many software engineers have.
| The dopamine hit they get when they resolve some outage or
| thorny bug or whatever (and the ensuing accolades from their
| peers) interferes with their ability to reason clearly about
| their compensation and the circumstances of their employment.
| SpicyLemonZest wrote:
| I guess I don't understand the line you're drawing here. If
| they enjoy the circumstances of their work, why _should_
| they go out of their way to "reason clearly" about how it's
| actually bad? I could understand if there's a collective
| action problem, but as many people in this thread have
| attested, people who want a job without oncall duty can get
| one.
| b3morales wrote:
| Remaining neutral on the on-call point, but as a
| practical/logical matter, things that _feel good and make
| you happy_ can still be bad for you personally,
| especially if they 're multiplied over the long term.
| Three or four gin and tonics every night feels great
| until your doctor reveals that your unexplained weight
| loss is from cirrhosis, or cancer.
| ebiester wrote:
| So, can this mythical 24x7 support the code you wrote? Did you
| write documentation that explains every case that could go
| wrong?
|
| If your team writes code that doesn't need to be supported by
| an engineer, then your team won't get called.
| pylua wrote:
| The truth is that other services break , the network breaks ,
| and someone will believe it's your software and it will be
| your burden to prove it at 2am in the morning .
| redisman wrote:
| Hah hilarious. Let me just go tell management our bar is now
| "never breaks" rather than "ship it".
| kayodelycaon wrote:
| I can't even be on-call if I wanted to. ADA requires a company
| justify any essential job functions. Justifying a programmer
| needing to be on call is difficult because it is a secondary
| role.
|
| Programmers aren't hired to do on-call. The company just decides
| they can skip hiring anyone for those roles by reusing "existing
| assets". Same with single sources of knowledge. They don't invest
| in having sufficient capacity.
|
| Basically, companies hire people who can do multiple jobs as
| cheaply as possible and with has as little support as possible.
| Which is not sustainable.
|
| ADA can really highlight dysfunctional systems.
|
| For anyone saying a small company can't afford this, ADA doesn't
| apply to companies under 15 employers or that can demonstrate
| undo-hardship given their resources. (By the way, trying to hire
| employees as contractors does not work.)
| JoshTko wrote:
| Not an SWE, but how common is on call duty, and are there no
| companies that pay spot bonuses per issue?
| SpicyLemonZest wrote:
| It's hard to quantify exactly how common it is. Common enough
| that you'd need to deliberately search for jobs that don't have
| it, but not so common that you won't find any.
|
| I've never heard of spot bonuses per issue, although I imagine
| it would create a perverse incentive to have noisy alerting
| that files a lot of issues. Flat bonuses per oncall shift exist
| (and I think they're a great idea) but they're relatively rare.
| varispeed wrote:
| Nothing wrong being on call as long as you are being paid to
| adequately. I always refuse being on-call if I am not being paid
| full hours plus extra for outside 9 to 5 for the entire duration
| regardless if anyone called.
| ckdarby wrote:
| Author lists off group of people who should not get pager duty
| and I think it is unfair to the rest of the team that individuals
| who fall into the groups outlined are able to escape the duties
| and responsibilities of the job.
|
| If the job posting or interviewing was upfront about the role
| will have on-call then the interviewee should make the decision
| to not apply or accept the offer if they're unable to fulfill the
| responsibilities.
| kayodelycaon wrote:
| Many programming jobs require on-call and overtime when they
| shouldn't. I wouldn't be able to have a job if that requirement
| was truly valid. It's usually not.
|
| I can't do on-call for medical reasons, nor can I work
| overtime. Neither of these are considered to be an essential
| job function for programmers, regardless of job descriptions,
| under ADA. A company must justify that job requirement. They
| can't.
|
| By the way, the "fairness" reason is explicitly excluded by
| law. It's the entire reason ADA exists.
| msrenee wrote:
| You think it's unfair for people who have young children to not
| be expected to also be on call 24/7? The author even specifies
| that they're talking about kids too young to sleep through the
| night. It's unfair for people with sleep disorders to not have
| to worry about getting called in at 2 am? In the anxiety case,
| it even sounds like the employee really wanted to be on call,
| but maybe needs to see a specialist to handle their reaction to
| stress.
|
| If people don't deserve a little slack due to personal
| circumstances, what happens when you find yourself in a
| situation where you can't pull your weight? I feel like people
| who think that not being able to do 110% of your job at all
| times makes you unqualified are in for a rude awakening when
| they find themselves in a similar situation.
| twic wrote:
| It's unfair for people who have young children to not be
| expected to also be on call 24/7 _and be rewarded in the same
| way as people who don 't, and are_.
| datavirtue wrote:
| There is no reward for OnCall work...except for a lack of
| respect. The OnCall people have always been viewed as less.
| I'm not doing anything on call unless our department head
| is on the bridge the entire time. OnCall is a pain that is
| inflicted during working hours.
| [deleted]
| barrkel wrote:
| That attitude leads directly to ageism, sexism and other
| discriminatory practices, and has second order effects on
| society which are contrary to the long term interests of
| capitalism - a stable consumer base with disposable income.
|
| Fairness isn't a straightforward concept, but your conception
| is more naive and - I'd guess - self-serving than most. How do
| you feel about progressive income taxes? Under a simplistic
| model of fairness, flat rates seem fair, but in reality the
| marginal ability to pay increases, and perceived pain of paying
| decreases, with increased income and wealth.
|
| Now expand that concept from the monetary domain to the time
| domain, and consider how fair it is for people who are
| particularly time poor - for non-selfish reasons - to need to
| spend more of their time on the job. Affordances can be made
| for inclusiveness and they can be fair.
| kayodelycaon wrote:
| Fairness, as used by the parent, means "if someone else can
| do something, I should be able to too."
|
| The other person's situation is never a factor. They see
| someone getting things for free and they think they should
| have them for free too.
| barrkel wrote:
| Do you feel the same way about maternity and paternity
| leave?
| ToddWBurgess wrote:
| I was on call for 5 years at an old job (non IT). Getting paged
| and yanked out of bed at all hours messes with you in so many
| ways. Being sleep deprived you turn to high sugar drinks and
| foods to keep you awake. The sleep deprivation and bad diets
| causes you to put on weight. Going to bed with the idea you could
| get paged does terrible things for your anxiety. On top of that
| all the sleep deprivation changes your personality.
|
| You become highly irritable and it really brings down your mood.
| Eventually I couldn't deal with it after 5 years and I pretty
| much walked off the job. I didn't realize the toll it was taking
| on me and what being on call was doing to me until I was no
| longer on call.
| Der_Einzige wrote:
| We need federal legislation that bans super strong Service level
| agreements in cloud tech. Amazon has the best cloud entirely
| because of their hellish on-call powered by their absurd SLAs.
| I'm sure this is one of the most important components behind why
| there is so much oncall.
|
| I'd love to see amazon and bezos get punished instead of rewarded
| just once for being slave drivers. Please pass federal
| legislation against "999" or other extremely absurd SLAs.
| yuppie_scum wrote:
| Extremely important article. I have been a part of some hell
| rotations and it is not sustainable at all.
| draw_down wrote:
| > I believe it's reasonable to expect to be woken up 2-3 times a
| year, as an engineer for that system.
|
| Reasonable that may be, but the problem is you don't know which
| 2-3 nights it will happen. No thanks.
| cletus wrote:
| So I've seen personally been involved in this at both Google and
| Facebook. They have very different approaches. One works
| reasonably well. One... doesn't.
|
| At Google, when you develop a service, you as a software team are
| responsible for maintaining it. This includes being oncall. But,
| depending on how critical the service is, you'll get bonus pay
| for this oncall time. This can amount to >$10k a year. And your
| oncall may not be that noisy. That is something and (IMHO)
| significant. Generally you'll be oncall for a week but this can
| vary.
|
| For sufficiently high profile services, oncall may end up being
| owned by an SRE team. That's not really something that can be
| thrown over the fence by SWE teams. SRE teams have to accept that
| ownership. To do this, it requires meeting standards like having
| an oncall runbook, a sufficiently long history (~6 months),
| adequate metrics and so on. At that point, SWEs will still be
| second level support for something SREs may need help resolving.
| You'll still get paid for this.
|
| SREs don't generally have week long oncalls. For the highest
| profile services, SRE support is global, meaning you'll have team
| members in 3 time zones such that whoever is oncall is in their
| normal working hours. So you might have an 8 hour oncall every
| week or something.
|
| At Facebook, generally as a software team you'll be responsible
| for that service forever. Some key services may be fully or
| partially supported by PEs (Production Engineers). PEs are less
| common than Google SREs.
|
| You don't get oncall pay at Facebook. It's simply expected.
| Depending on your management, you may be expected to do that
| oncall and everything else you're supposed to be doing.
|
| So here's why Google's system is better than Facebook's:
|
| 1. Oncalls, in my experience, tend to be a lot less noisy at
| Google. Services tend to be much more mature and stable at Google
| than Facebook;
|
| 2. Teams are generally larger at Google. This means that not
| everyone needs to be oncall. This is good. There is an optimum
| size for oncalls. IMHO it is about 8-12. Fewer than 8 and people
| are oncall too much. More than 12 and you lose skills by not
| using them often enough.
|
| 3. The pay is really important to (2) because it means that those
| who are oncall at least get compensated for it. At a minimum this
| sends the message that oncall is important and rewarded. It also
| means those oncall are less likely to resent those not oncall,
| which might well be the case if you're not compensated for the
| extra work;
|
| 4. There are a ton of orphan projects at Facebook (IME). By this
| I mean some product team (in particular) will develop something,
| ship it and then... forget about it. Onwership of source code
| trees and projects tends to be fairly strictly enforced at
| Google. These orphan projects tend to create a number of support
| issues and tasks that are a drain on oncall as there is no one to
| pass those tasks on to.
|
| 5. Culturally, Google seems to acknowledge and respect oncall
| work load more than Facebook does. For example, tasks at Facebook
| have SLAs and I saw plenty of games played to avoid dealing with
| tasks. Examples: changing priority to increase or remove the SLA,
| silently closing tasks, passing tasks back months later to the
| reporter asking "is this still an issue?", etc.
|
| So my point is that not wanting to be oncall is understandable
| but for many engineers, it's going to be part of your job. If so,
| you need to make sure your organization does it well. There's a
| world of difference between an oncall you don't get paid for
| that's 40 hours of work vs one where you get 2 tasks a week and
| you're getting paid for possibly having to answer an alert.
|
| Hating oncall is indicative of a bad engineering culture IME.
| Things like prioritizing shipping above all else, not rewarding
| fixing things, unreasonable management driven deadlines and so
| on. Oncall is the canary in the coal mine for all of that.
| supernovae wrote:
| Do you have kids? hobbies? friends? Do you like to go fishing
| on weekends? Do you like to go out on a date on weeknights? Do
| you like going to visit family on Christmas and Thanksgiving
| and leaving your laptop at home? It's great explaining to your
| 11 year old that you can't go to their recital because your on
| call isn't it?
|
| It boggles my mind that you suggest people lose skills by not
| using them often enough... these on call rotations aren't
| skills.
|
| What's weird, is that we default to look at the goodness or
| badness of something purely on "skills" and "money".
|
| Yes.. yes.. we've all read the SRE book and understand Googles
| approach to flexibility - but... it doesn't have to be that
| way.
|
| Ask yourself.. what's next? What are you looking to do after 25
| years of being on call? Will you still enjoy it?
|
| I'll never have those nights and weekends back that I spent
| being on call for products and services that don't exist
| anymore. Sure, I made some money - but it didn't have to be at
| such a great cost.
| vitus wrote:
| At least at Google, oncall shifts aren't set in stone. If you
| need to swap with a coworker (even for a few hours), you can.
| Devs should not be holding 15-minute (much less 5-minute)
| pagers; if your system is really that critical, you should be
| working to onboard an SRE team.
|
| > these on call rotations aren't skills.
|
| Troubleshooting live production issues is a skill.
|
| Triaging incidents to prioritize actions to minimize user
| harm is a skill.
|
| Assessing tradeoffs of mitigation strategies is a skill.
|
| The very tangible scenario of being woken up in the middle of
| the night builds empathy with your devops / SRE organization
| (which can manifest itself in designing more resilient
| systems, engaging in less risky behavior, improving
| documentation for the next oncaller, etc).
|
| Oncall isn't for everyone, and it's certainly not worthwhile
| if you don't take away any lessons from it. But devs are
| ultimately the closest ones to their code, and they should
| take advantage of the opportunity to understand how it
| behaves in production.
|
| > Ask yourself.. what's next? What are you looking to do
| after 25 years of being on call? Will you still enjoy it?
|
| I've had bad weeks of oncall as well as quiet weeks. If it
| ever gets to a point where bad weeks are much more frequent,
| and nobody's doing anything about it, then yeah, I won't want
| to do it anymore. But as long as it's still providing value
| beyond humans providing blood / sweat / tears, and as long as
| we're actually trying to make things better, I don't mind
| holding a pager a few weeks a year.
| cletus wrote:
| > Do you have kids? hobbies? friends? Do you like to go
| fishing on weekends?
|
| Do you like having a job that gives you money to pay for all
| those things as well as the basic ability to, you know, live?
| How far do you want to take this? Is working 40 hours a week
| taking time away from your children, friends or hobbies? Of
| course it is.
|
| But nobody owes you a living. A job is you trading your time
| and your skills to someone else in a way that provides that
| person value. There are going to be parts of that job you
| like and probably parts you don't.
|
| If you really want to avoid oncall then fine, do it. That
| will likely limit your career opportunities because some jobs
| will require it or your coworkers may be rewarded (monetarily
| and careerwise) because they're doing something you're not.
| There's nothing wrong with that choice but at the same time
| you don't get to complain because others benefit from doing
| something you won't.
|
| > but... it doesn't have to be that way.
|
| For 24/7 production services then who exactly supports those
| services?
|
| > I'll never have those nights and weekends back that I spent
| being on call for products and services that don't exist
| anymore.
|
| There's a pretty good chance that nothing you worked on will
| exist in 25 years. What's your point?
| joshuamorton wrote:
| I'm confused, the post you're replying to says that the
| Google model means you don't have nights OnCall.
|
| > . these on call rotations aren't skills.
|
| Incident response/management is absolutely a skill.
| quelltext wrote:
| > For example, tasks at Facebook have SLAs and I saw plenty of
| games played to avoid dealing with tasks. Examples: changing
| priority to increase or remove the SLA, silently closing tasks,
| passing tasks back months later to the reporter asking "is this
| still an issue?", etc.
|
| What kind of tasks are you referring to in the context of on
| call here? Tasks to remedy the root cause of repeated alarms
| paging folks?
| cletus wrote:
| Task is a general all-encompassing term here as well as the
| internal task tracking tool everyone uses (called "Tasks").
|
| Tasks could be filed by users of your service or external
| reports that get routed to your oncall or an alarm that fires
| because a metric moves outside of a "normal" range or
| whatever. Alarms ("pages") will come to you and need to be
| ACKed. Often they'll also generate a task. That task may
| require separate action or not. Many times those will just
| get closed or merged to existing tasks.
| cghendrix wrote:
| I have a question. Is this for all environments (dev, qa,
| staging or whatever the equivalent is at google/fb) or just
| production?
| joshuamorton wrote:
| SREs are (with a few exceptions) only responsible for
| production (or more precisely, things serving prod traffic,
| which may be multiple environments). Dev teams are usually
| responsible for keeping qa and ci green.
| hirundo wrote:
| There's a salutary effect on developer conscientiousness to put
| them on call for problems with their own code on production. If a
| missing nil check means that you get dragged out of bed at peak
| REM sleep, you check for missing nils harder. You test more. You
| design your whole project around the requirement of getting
| uninterrupted sleep.
|
| For years I was in this position for a trading company that had
| pre-market downloads at 4am my time. Bad data and my own bad code
| woke me up a lot ... which contributed a lot to making that code
| wake-me-up proof for further years.
|
| Of course you should get paid for that time, and there's a
| diminishing return to such character building exercises. But
| eating your own dog food is worthwhile particularly in the middle
| of the night.
| kayodelycaon wrote:
| This assumes your company is okay with spending time to write
| reliable systems.
| [deleted]
| l0b0 wrote:
| I and another person are taking over an old batch processing
| system at work. The system itself is fairly stable, but interacts
| with some other systems which keep falling over or changing
| things without adequate notification. Currently there are (mostly
| false) alarms almost every day, usually for stupid reasons like
| the network falling over for a few minutes or disk use going
| above a fairly arbitrary level. So when the subject of weekend
| on-call came up we just went hard "no", and we've been discussing
| solutions to _that_ ever since (adjusting cron scheduling, making
| alarms more forgiving, adjusting customer expectations, improving
| automation, etc).
|
| Importantly, we can do this because we live somewhere with
| employee protection, and our contracts do not mention being on
| call or working weekends. If you don't have that you're pretty
| much SOL.
| [deleted]
| throwaway55421 wrote:
| Most jobs I've had there was an implicit expectation that if your
| shit breaks, you fix it. This resulted in systems which didn't
| break because, well, that keeps work to working hours.
|
| A few years back I took a job with no mention of on call and then
| was told there would be a fortnightly 24 hour slot, which later
| turned into ~weekly as team members left.
|
| I left too.
|
| I have no idea why they felt it reasonable to not mention this
| before I joined, they effectively just wasted a few months pay on
| me on a gamble that I had no life outside of work.
|
| They seemed to find the idea that I don't take my laptop to the
| pub, or up a mountain, or on holiday, some sort of bizarre way of
| living. Sick system effect? Who knows.
| Wiseacre wrote:
| Sign of a wider lack of trust between employer and employee.
| Yet the idea of unionization is still very unpopular.
| throwaway55421 wrote:
| I don't follow.
|
| Most jobs I've had were high trust, this was the weird
| exception. I didn't need a union to fix that.
| Wiseacre wrote:
| It's not a rare phenomenon across the industry. Plenty of
| my colleagues and I have experienced something similar.
| 0x0000000 wrote:
| > Most jobs I've had there was an implicit expectation that if
| your shit breaks, you fix it. This resulted in systems which
| didn't break because, well, that keeps work to working hours.
|
| My experience was similar in my last long-term software role.
| We were not explicitly on call, but we
| developed/deployed/supported an app which was critical to a
| 24x7x365 business process. So we were careful in our testing,
| code reviews, and deployments.
|
| One time in 7 years I had to answer an out of hours call
| because of an issue in this app.
|
| I'll take that over semimonthly on-call rotas for suites of
| apps my team isn't isn't responsible for.
| cameronh90 wrote:
| > if your shit breaks, you fix it
|
| What if you're drunk?
| ozarkerD wrote:
| Good thing I was drunk when I made it!
| NikolaeVarius wrote:
| Stop deploying shit then. Also, I am very good at fixing
| systems even when drunk. Lots of practice.
| throwaway55421 wrote:
| You think about that when you design it.
|
| I used to work on trading systems which were managed in real
| time by a rotating team of traders (i.e. mathematically
| oriented people as opposed to software engineers).
|
| You design the system so that if it breaks there are at least
| workarounds that anyone can employ without needing to write
| code. For example, a big stop button, manual adjustments,
| manual trading, etc.
|
| Maybe it stops printing money overnight and you take an
| opportunity cost loss. That's fine, post mortem at work, fix
| it.
|
| In the worst case if you're not available then someone with
| ownership steps in like a CTO/founder level (who are of
| course always on call almost by definition, though they
| generally have the executive power to say - sod this, we'll
| just leave it down for a while).
| s0rce wrote:
| Seems like "if your shit breaks, you fix it" = on call
| 24/7... isn't it better to have a specific shift where you
| know you are responsible and then you aren't the rest of the
| time.
| throwaway55421 wrote:
| No, because in that case you are responsible for other
| people's systems you have no idea about and there is no
| incentive to fix them.
|
| I am always on call to secure my own house, so I have a
| good alarm system, cameras, locks etc which means I
| hopefully don't have to do much. It doesn't keep me up at
| night when I'm on holiday because, well, it's
| overwhelmingly likely that nothing will happen.
|
| If I were on call for the neighbourhood or general area
| once a week, I'd probably have to physically be there
| patrolling it because otherwise I have liability for things
| which are outside of my control.
| znpy wrote:
| I used to be on call, and now I'm not anymore (but I changed jobs
| in between).
|
| I'm not against being on-call but in my opinion if I get called
| tonight then tomorrow morning I want to be able to start a
| process that prevents the problem from happening again, and I
| want to be able to put EVERYTHING in discussion. I don't want to
| hear anything about development or operations, that thing must be
| root caused and it must whoever can implement a fix (either me,
| my team or the development team, either on our side or on
| customer side) must drop everything they're doing and implement a
| correction of error.
|
| TL;DR:
|
| If I can't fix stuff and prevent the problem from happening again
| then I don't want be on call.
| marcus_holmes wrote:
| > In a healthy engineering organization, there are no gaps in
| coverage. Every critical component is owned by a TEAM, not a
| person.
|
| I just joined a large organisation that practices this, and it's
| driving me nuts. Because no individual is taking ownership,
| nothing happens. There's no accountability, no drive to improve
| anything. We ask questions on Slack about "can someone deal with
| this problem, please?" and nothing happens. Rather than an alarm
| going off at 2am and the owner dealing with it (again), the alarm
| goes off and nothing happens, no-one fixes it. Stuff stays broken
| until it gets a ticket and assigned to a sprint (and even then,
| people self-assign to tickets, and there's no consequences to not
| finishing all the tickets on a sprint, so it's still relying on
| someone to be vaguely interested enough to pick it up and deal
| with it).
|
| I moved from being tech co-founder of a startup (where everything
| is my responsibility and it's all on me to fix it) to this. It's
| doing my head in. I think I actually prefer being "on-call" for
| code I built and have complete control over.
| DoneWithAllThat wrote:
| I used to not mind oncall as much, before managerialism and
| corporate attitudes took over tech. It was fun (I'd also
| stressful) to feel in the trenches so to speak, while being
| treated as someone mildly crazy for taking on the responsibility
| but also not held to the same silly and stupid bureaucratic
| attitudes that afflicted the rest of the company.
|
| Now I'm supposed to talk to businessspeak talk and walk the promo
| game walk while also supposed to answer the phone at one am and
| stay up all night saving the business from melting down.
|
| Yes I know I'm ranting. The industry has turned into a miserable
| corporate grind like finance or something, only with the added
| obligation of someone has to mop up to slop in the middle of the
| night and that someone is me.
| [deleted]
___________________________________________________________________
(page generated 2021-11-28 23:00 UTC)