[HN Gopher] DevOps: An idea so good, no one admits they don't do it
       ___________________________________________________________________
        
       DevOps: An idea so good, no one admits they don't do it
        
       Author : wagslane
       Score  : 167 points
       Date   : 2022-08-29 16:16 UTC (6 hours ago)
        
 (HTM) web link (wagslane.dev)
 (TXT) w3m dump (wagslane.dev)
        
       | oxfordmale wrote:
       | Agile is a philosophy, it is not sprints, story points or
       | whatever crazy methodology an expensive Agile Coaching company
       | has come up with. The same applies to DevOps. DevOps could just
       | be a bash script, written by the developer, to feit their
       | codebase. It doesn't have to be fancy or require an extensive
       | amount of investment.
        
         | mxuribe wrote:
         | You sound like Mr. Miyagi from the Karate Kid movies (which is
         | a good thing!)...specifically it remains me of that scene when
         | Daniel-San asks him what "belt" he is...and Mr. Miyagi responds
         | "Calvin Klein"! (At some point, Mr. Miyagi explains that his
         | exppertise of Karate is only for the need to survive
         | encounters, and not worry about which "belt" he had achieved.)
        
       | dec0dedab0de wrote:
       | I was a network engineer at my previous job, I was fed up with
       | doing things manually so I learned to code better. Now I'm a full
       | stack developer writing code for the network team of a different
       | company.
       | 
       | One thing that has changed drastically over the last 15 years is
       | the sharp divide between IT and development. It used to be that
       | developers were clueless how to deploy their own apps, and IT was
       | terrified of writing code preferring to do things manually, or
       | purchase something prepackaged. Now I see developers that
       | understand firewalls and load balancers, and IT folk that can
       | script stuff out for themselves. I think we have the push behind
       | DevOps to thank for that progress, and while there is still a
       | long ways to go, I think it's a good thing overall.
        
         | mxuribe wrote:
         | > ...writing code for the network team of a different
         | company...
         | 
         | Something like is the kind of job that i think i want to do
         | next. It need not be for a network team...but i'm headed more
         | for back-end work and these days i rather engage with internal
         | teams.
         | 
         | Separately, to your comment on sharp divide...The sharp divide
         | that i have seen is a lack of cohesion for the following 4 IT
         | groups: first-level support which gets sent to off-shore; the
         | folks who (honestly have the most face time with internal
         | clients who) roam around the office fixing desktops and
         | building network/wiring, and setting up/fixing A/V equipment in
         | conference rooms; sys admins; and developers. Furthermore, in
         | the last 2 companies that i worked for, there has been a push
         | to "centralize" certain functions like IT and HR to "shared
         | services"...but at the end, none of these groups seem
         | centralized! They all now work more separated than ever
         | before...and most of the interaction between each of these
         | groups - and even between the 4 subgroups i noted above in IT
         | alone - mostly communicate via tickets, or some other thick-
         | layered mechanism. Sure, they've been centralized in that these
         | people were pulled from some respective business unit, and
         | brought under a more single umbrella...but in the end, these
         | workers hate it, the effecicny drops, more bureacracy is
         | created (which creates no value)...and some executive gets to
         | lead the shared services...and of course some management
         | consultant company gets their payday. Oh, and yeah, IT at these
         | orgs definitely does not use any of the good parts of devops,
         | agilescrum whatever. Some orgs try and adopt some cookiecutter
         | approach only to have it create yet more bureacracy...but "hey,
         | we're agile!" /sigh
        
       | notabee wrote:
       | Corporate kayfabe. https://en.wikipedia.org/wiki/Kayfabe
        
       | manuelabeledo wrote:
       | It kind of feels like devs have been duped for the past twenty or
       | so years.
       | 
       | Want to save in project management? Just make devs responsible
       | for planning and estimating, and call it "agile".
       | 
       | Want to remove sysadmins and ops? Make devs write their own
       | configuration for the infrastructure, and say you do "devops".
       | 
       | Want to eliminate QA? Say that you do TDD.
        
         | 0x457 wrote:
         | > Want to eliminate QA? Say that you do TDD.
         | 
         | I don't think anyone ever thought of it like this. Well, some
         | dumb managers probably did. If you want to eliminate QA people,
         | then you need an enormous automated E2E test suite. To have
         | that, you need dedicated people who think of and write those
         | test...one would call it a QA team.
        
       | jbverschoor wrote:
       | Devops means let a non-admin developer have access to aws and
       | just buy everything he can find in order to try and fix his
       | shortcomings.
       | 
       | It'd be better and cheaper to follow a sysop course
        
       | vinaypai wrote:
       | Not directly related to the question, but I personally hate the
       | term "Sprint". Physiologically, a sprint involve by definition is
       | a short-term high effort activity that can't be sustained without
       | a break.
       | 
       | Does anyone use a different term that doesn't imply working
       | oneself to death?
        
         | duped wrote:
         | Cycle
        
         | 0x457 wrote:
         | Cycle or stint. I like "stint" more.
        
         | 0xbadcafebee wrote:
         | Just changing the word isn't going to change the practice.
         | You're supposed to take breaks between sprints. There needs to
         | be planning and alignment to an easily-obtained goal, which
         | requires time. But most people don't know how to do Scrum and
         | hand-waive the goal, the planning, the retro. The end result is
         | unsustainable work with no specific goal in mind that exceeds
         | the sprint cycle. Or 20 epics all in progress at once. Or
         | business goals that shift and put unobtainable expectations on
         | teams.
        
           | bot41 wrote:
           | What's a break period? Where is that described?
        
             | 0xbadcafebee wrote:
             | It's not explicitly described. "The break" (imo) is giving
             | enough time to planning at the beginning, and actually
             | dealing with the review & retro items. The length of the
             | sprint can be shortened so that there is more learning and
             | planning. Sprints are cancelled if the goal becomes
             | obsolete.
        
       | louwrentius wrote:
       | This is a content-free extremely meager blog post that doesn't
       | really belong on the front page of HN but the discussion in the
       | comments is quite illuminating.
       | 
       | The discussion shows how a lot of people have wildly different
       | ideas about what devops is and that in and of itself is
       | interesting.
        
       | wiremine wrote:
       | Recently I've been reading "When Agile Gets Physical" [1] and
       | "Extreme Programming Explained":
       | 
       | 1. "Extreme Programming Explained" explains the importance of
       | separating values from principles and practices.
       | 
       | 2. "When Agile Gets Physical" unpacks why using agile software
       | best practices are horrible for hardware. This is "duh" if you're
       | a hardware person, but the book goes back to agile/lean first
       | principles, and then builds up an approach for agile hardware.
       | 
       | It got me thinking that we often conflate values and practices.
       | Blinding adopting Scrum or DevOps without understanding some of
       | the underlying value and principles leads to problems. If you
       | understand the foundational principles, you're in a much better
       | position to adopt/reject certain practices for your given
       | situation.
       | 
       | In reality, most organizations just adopt a few agile practices,
       | while rejecting the basic values.
       | 
       | [1] https://rapidlearningcycles.com/when-agile-gets-physical/
       | 
       | [2] https://www.amazon.com/Extreme-Programming-Explained-
       | Embrace...
        
       | irrational wrote:
       | I hate devops. Mainly because I don't want to do it. I liked it
       | better when there was a dedicated infrastructure team instead of
       | pushing it down to developers and giving it a new name.
        
         | hedora wrote:
         | Yeah. As a developer, I need a competent QA and operations team
         | to support me. I'm not competent at those things, but 90% of
         | developers I've worked with are somehow worse at them.
         | 
         | I also need 1-2 months between major context switches, and the
         | ability to declare a subcomponent "ready for release".
         | 
         | Moving forward, I will actively avoid companies that claim to
         | use agile, scrum or devops.
         | 
         | Edit: Also, I want to work on a team that has a good release
         | manager.
        
       | GuB-42 wrote:
       | The problem with these fad methodologies is that it either works,
       | or you are not doing it properly. The methodology can't be wrong.
       | 
       | It is like saying that C programs never crash, because if they
       | crash, it means you are not using C properly. It is true, but no
       | one will pretend that, everyone knows that under real life
       | circumstances, C programs can crash, and there are plenty of
       | other programming languages that address this issue, at the
       | expense of other things, like performance or simplicity.
       | 
       | But for some reason those who push methodologies don't seem to
       | consider that if that methodology is difficult to implement, or
       | if it is not robust against things like conflicts of interest and
       | office politics, or whatever, then maybe it is a problem in their
       | methodology. It doesn't mean the methodology is bad, just that it
       | doesn't work in every case, and that some tweaking may be
       | necessary.
        
       | mekoka wrote:
       | _> We need better definitions for these terms _
       | 
       | When I point to the moon with my finger and say "moon", the fool
       | looks at my finger and calls it the moon.
       | 
       | Is it more important to mindlessly tag yourself "Agile", with all
       | the bells and whistles adorning the capital A, or would you
       | rather just _be_ agile according to the sought out goals and
       | philosophy, and adapted to your specific situation?
       | 
       | Is it more important to have a clearer definition of "DevOps", so
       | that you can more accurately implement all the prescribed
       | workflows, or would you rather understand the pursued ideals, so
       | that you can pick and choose what you need, according to _your
       | specific context_ , unencumbered by a slew of best, but
       | ultimately useless, practices?
        
       | vesinisa wrote:
       | The author is right that DevOps seems to often mean different
       | things to different people. I would have loved it if they had
       | taken more of an attempt at one definition than just that it's
       | when developers have an understanding of how and where their code
       | is deployed. That does sound sensible, but what other best
       | practices does DevOps usually entail?
        
       | giantg2 wrote:
       | I feel like DevOps is a trap.
       | 
       | Most companies that have sufficient talent and
       | organization/operational setup don't see a need for DevOps.
       | 
       | There seem to be a lot of other companies that are sucked in by
       | its promises and think that it will solve their issues.
       | Surprise... this didn't fix your shitty policies,
       | disorganized/unempowered org structure, or cheap talent (cause
       | you're unwilling to pay or train).
       | 
       | I assume there are a few places that can benefit from this. Maybe
       | some medium to large tech companies with great procedures,
       | documentation, and an empowered org structure. Especially if
       | they're paying enough for people who can manage it. The other is
       | probably for very small places wirh fairly simple tech where you
       | can't afford entire teams of people so you empower someone to
       | really own the system top to bottom (without ever calling it
       | DevOps).
        
         | turtlebits wrote:
         | IME, letting developers have free reign over service
         | architecture/deployment/CI/observability is a bad idea.
         | Especially when every team comes up with their own option.
         | 
         | You need some sort of Devops (best practices if not a team) to
         | provide guidance and help implement standards for these.
        
           | giantg2 wrote:
           | I agree, if we're talking about places big enough to have
           | multiple teams. The thing I was pointing out is that most
           | places that have multiple teams are terrible at setting up
           | best practices. They'd be better off with a centralized
           | "environments" team provisioning the server, network, etc so
           | the dev teams can work on the actually implementation. At
           | least it's easier to maintain standards, or at least
           | consistency, across apps if the low/mid level config is
           | handled by one team.
           | 
           | When I was talking about small places, I mean like tiny mom
           | and pop non-tech companies with an IT department of 1-5
           | people.
        
         | ryukoposting wrote:
         | Every company that has a software product also has DevOps,
         | whether or not they call it that. If someone manages your cloud
         | infrastructure, they're doing devops. If someone is coming up
         | with CI/CD setups, they're doing devops. If someone came up
         | with SOP for distributing new releases, they're doing devops.
         | 
         | Whether or not anyone's title has "DevOps" in it, devops tasks
         | are still devops tasks. By the same logic, even though I
         | watered the office plants this morning, I am still a firmware
         | engineer- not PlantOps.
         | 
         | The real trap of DevOps is the assumption that it's anything
         | new or non-essential. DevOps has existed since the notion of
         | "software as a product" came into existence, we just didn't
         | have a name for it yet.
         | 
         | None of this is to say you need a dedicated DevOps person (or
         | PlantOps, for that matter). DevOps is an integral part of any
         | software product, even if no specific member of the development
         | team is dedicated solely to DevOps tasks.
        
           | giantg2 wrote:
           | I don't think those are DevOps by themselves. You have to be
           | doing the Dev part of the system in addition to the Ops part.
           | The traditional format is that some team or tram member would
           | do ops work while others did the dev work.
        
       | ebiester wrote:
       | Or, alternatively, it might not be a great idea in all cases.
       | 
       | Maybe your development team shouldn't develop terraform the four
       | times a year they need it. It might even be faster to have two or
       | three individuals with a singular vision for the system in a
       | smaller team. (There may not be enough work for some teams to
       | have someone with specific DevOps skills. However, that ends up
       | in a partial-shared-service model where team members get called
       | off for organizational concerns.)
       | 
       | Maybe you need a subject matter expert that will develop
       | standards for teams and select a set of tools, setting those up
       | so that teams can build off those for their needs. (This is why
       | Ops are often shared services teams.)
       | 
       | Maybe it's not a great idea if the first team to encounter a
       | problem picks the tool that works best for them, and the
       | organization ends up with multiple tools because there was nobody
       | with both the time and expertise dedicated to the analysis. (This
       | is why Ops are often shared services teams.)
       | 
       | We need to break down barriers. We need to adopt valuable
       | practices. However, the parts of DevOps that don't make it to the
       | real world may just be because they don't solve the problems of
       | the organization.
        
       | nkotov wrote:
       | The mistake was turning DevOps into a job position which resulted
       | in silos being created more when the whole purpose was to
       | remove/prevent silos and to allow developers to operate at their
       | own pace.
        
       | jammycakes wrote:
       | In a previous job, I joined a team that was supposed to be
       | introducing DevOps to the organisation.
       | 
       | It started out well -- we spent a few months hacking with
       | Terraform, Docker, Vagrant, Kubernetes, and related technologies
       | to implement an infrastructure-as-code approach -- automating the
       | process of provisioning and decommissioning servers, and building
       | a setup where development teams could deploy updates with a
       | simple git push.
       | 
       | Unfortunately it all went downhill fairly rapidly. We ended up
       | spending the majority of our time manually applying security
       | patches to a bunch of snowflake servers that we'd lifted-and-
       | shifted from another hosting provider to AWS, and fielding
       | support requests from the development teams. Within a year, we
       | were being told in no uncertain terms by our project manager that
       | we were an operations team, not a development team.
       | 
       | It felt like a complete bait-and-switch. Within two years, I had
       | left the organisation in question and moved on to a new job
       | elsewhere doing actual development again. Last I heard, the
       | entire team had been disbanded.
       | 
       | It sounds like the author of this article must have had a very
       | similar experience. I wonder just how common it is. It seems that
       | in many places, "DevOps" is all Ops and no Dev.
        
         | mr_tristan wrote:
         | > It seems that in many places, "DevOps" is all Ops and no Dev.
         | 
         | This was definitely my experience at my last couple of jobs. At
         | my last one they "scaled out their DevOps team" by hiring tons
         | of juniors with next to no software development background. And
         | then they "empowered" teams by assigning the juniors to each
         | dev group. As a result, we ended up having to train them how to
         | do their core jobs, which... went about as well as you'd think.
         | 
         | Eventually, there was an attempt to shift everyone to
         | kubernetes. They had a special "DevOps" team build a layer on
         | top of it to handle the non-kubernetes aspects of deployment as
         | well, and somehow manage them together using Helm. If you're
         | wondering "what the hell does that mean", well, it turned out
         | nobody really knew. These "DevOps" engineers didn't really seem
         | to understand kubernetes core concepts, and just ended up
         | hacking away with some scripts on top of terraform delivered
         | via Helm until something got configured. It was incredibly slow
         | to deliver, hard to use, and I just stayed away from it until
         | some exec threw down the mandates. (And then everyone started
         | quitting because it was an absolute disaster.)
         | 
         | Ultimately, these are really stories about bad management, not
         | really anything to do with DevOps. But that's how these things
         | roll - some new hot concept comes to town, and bad managers try
         | to adopt the term, without really understanding it.
        
           | zo1 wrote:
           | It's not always bad managers or management. The fancy new
           | hotness gets pushed onto them by smooth-talking evangelists.
           | For every criticism or question they have a snazzy little
           | quip and retort that makes you look like an idiot for not
           | knowing "the obviousness" of your errors and how this new
           | fad/tech/framework/methodology solves it. And if that retort
           | doesn't work, they just tell you "it's standard in the
           | industry, dunno what you want me to tell you".
           | 
           | And from the outside, this all just looks like resume padding
           | and job-security. Devops is the new priesthood, subscribe or
           | be reduced to irrelevance by config warriors.
        
             | stcredzero wrote:
             | _The fancy new hotness gets pushed onto them by smooth-
             | talking evangelists._
             | 
             | Gets pushed onto them top-down. This means there is no real
             | competition, no real empiricism, no real comparative merit
             | involved in the switch.
             | 
             |  _How_ should never be imposed top-down. It 's only goals
             | and how success is measured which should be top-down. The
             | classic example of this was when Jeff Bezos mandated that
             | all Amazon software systems should be accessible by APIs
             | over the internal network, or else one would be fired.
             | 
             | Whenever I've seen _How_ imposed top down, I 've only seen
             | the lower level managers talk about how they could put off
             | and passive aggressively stymie the initiative.
        
         | hackernewds wrote:
         | Isn't it apparent in the name? MarketOps, ServerOps, PaymentOps
         | - all Operations
        
         | 0xbadcafebee wrote:
         | Here is how some companies "do DevOps":                 1. An
         | operations team with a different name.       2. A platform team
         | with a different name.       3. A development team with a
         | different name.       4. A "CI/CD team".       5. A role (ex.
         | "dev who automates ops", "ops who codes", "support specialist
         | who codes", all three in one).       6. A chart that the
         | delivery manager maintains.
         | 
         | Here is what DevOps should actually be:                 1.
         | Delivering rapidly and consistently with extremely high levels
         | of confidence.       2. The right people address problems
         | correctly, immediately, the first time, and fix it so it
         | doesn't happen again.       3. That's it.
        
         | spaceywilly wrote:
         | > We ended up spending the majority of our time ... fielding
         | support requests from the development teams
         | 
         | This has been my experience at 3 different small-medium
         | companies now. A too small DevOps team suddenly is in the
         | critical path for even the most trivial software task, then
         | engineering productivity grinds to a halt. I think a much
         | better pattern would be to enable dev teams to self serve. Set
         | up the required infrastructure and guard rails, then let teams
         | handle their own deployments and infrastructure. Give people
         | what they need to do it themselves instead of having to open a
         | support ticket for everything.
        
           | jammycakes wrote:
           | > I think a much better pattern would be to enable dev teams
           | to self serve. Set up the required infrastructure and guard
           | rails, then let teams handle their own deployments and
           | infrastructure.
           | 
           | I think that's how DevOps is actually supposed to be done in
           | the first place. You view Ops -- and the code used to manage
           | and support it -- as a product, and get a specialised team of
           | experienced Devs (and architects) to build it.
           | 
           | Once you've got the basic infrastructure and architecture in
           | place, you then train up the individual development teams to
           | customise it, extend it and troubleshoot it as they need to.
           | In much the same way as they do with any other software
           | product.
        
             | JamesBarney wrote:
             | My experience is what inevitably happens is the ops team
             | goes an writes a layer on top of Kubernetes, and now
             | instead of dealing with Kubernetes you're dealing with a
             | half baked poorly written abstraction with zero
             | documentation and no StackOverflow on top Kubernetes. So
             | you need to become an expert in both.
             | 
             | Most organizations don't have the resources, mindset, or
             | skills to support a software library product and should
             | only do it as a last resort.
        
           | lazide wrote:
           | Most developers won't know how to do it without footgunning
           | themselves constantly is the problem.
           | 
           | If the dev ops team is staffed enough to develop integrations
           | that won't allow that AND won't get in the way, and then
           | train folks and 'keep the line' enough to stop the scope
           | creep - they're probably not at a shitty small/medium sized
           | shop.
        
         | louwrentius wrote:
         | Sounds like the organization went full cargo cult with pets
         | (lift and shift) instead of a devops-ready environment.
        
           | sascha_sl wrote:
           | I've not seen one place that has escaped this problem though.
           | 
           | Something is either old and nobody feels like fixing it or
           | something doesn't fit into the current constraints of the
           | platform featureset. So they build something on their own,
           | probably undocumented and without consulting the supposed
           | DevOps team, but still using parts of the platform that have
           | not been declared an API. But when you want to exercise the
           | freedom you supposedly built with your platform, all these
           | edge cases fall back on you and inhibit change. "We can't
           | change the ingress controller, we rely on this implicit
           | behavior of it" "You can't change how database credentials
           | are provisioned, we pulled these from the Cloud SQL console
           | and comitted them to git". And facilitating any change as
           | soon as you _can_ cover the use-case is a fight with
           | stakeholders and POs that I usually have no nerve for.  "Why
           | do we need to do anything??? It works?". And then you get
           | blamed when it breaks. I love this job.
        
       | JohnFen wrote:
       | As a dev, the whole "rapid release" thing made a lot of sense to
       | me. As a user, though, I've turned 180 degrees on it. I hate
       | "rapid release" with a passion now. Which means I dislike Agile
       | by consequence.
       | 
       | Rapid release means I can no longer rely on the software that
       | engages in it. It's constantly shifting, with frequent changes in
       | UI and workflow, and new and exciting bugs constantly appearing.
       | 
       | I miss the days when I could plan ahead for new versions, so I
       | could time them to have the least possible disruption to my life.
        
         | sokoloff wrote:
         | "Everyone wants progress; no one wants change."
        
         | wiremine wrote:
         | Small, "rapid releases" don't need to all go to the end user at
         | once. That's an anti-pattern for solid UX design.
         | 
         | The classic "eBay color change" comes to mind:
         | 
         | https://kulor.medium.com/how-ebay-secretly-changed-their-bac...
        
           | alexb_ wrote:
           | Easy when it's a color change, but this attitude is horrible
           | when it comes to anything functional. You stop being able to
           | learn how things work because someone wanted to make a small
           | tweak, and now you don't even get notice it's happening.
        
             | lazide wrote:
             | Sometimes even a color change IS a major function change
             | too, especially if you have business customers
        
           | tremon wrote:
           | Yes, they do? I thought the entire idea behind "release
           | early, release often" was to shorten the feedback cycle
           | between code changes and user reports so that it's easier to
           | fix the issues?
           | 
           | However, not every code release needs to include a UX change.
           | I agree that changing the UX every week is a solid anti-
           | pattern.
        
             | philihp wrote:
             | Feature/UX changes should be behind feature flags, and
             | launched independently of a release. Releasing often let's
             | us identify and fix bugs quickly. I hear you that as users
             | we appreciate more stable UI, but to get to an intuitive
             | UI, you have to experiment and iterate.
             | 
             | While there is value in discrete and planned releases, we
             | value responding to change more.
        
               | Silhouette wrote:
               | _I hear you that as users we appreciate more stable UI,
               | but to get to an intuitive UI, you have to experiment and
               | iterate._
               | 
               | But there is no rule that says you have to do those
               | experiments in production and annoy your paying customers
               | who don't want to see them. If you didn't insist on
               | releasing every five minutes you'd have time to iterate
               | privately. Successful software was made this way for
               | decades before modern web applications came along and
               | frankly a lot of it had better UI/UX than a lot of the
               | junk that gets pushed out today.
               | 
               |  _While there is value in discrete and planned releases,
               | we value responding to change more._
               | 
               | I suppose the question is what change you're responding
               | to. In this situation it seems like a lot of that change
               | and the need to respond to it are self-inflicted damage.
        
               | yjftsjthsd-h wrote:
               | Surely feature flags _are_ a way to iterate privately
               | even while shipping the code?
        
               | Silhouette wrote:
               | That depends on who gets to decide when the flags get
               | turned on, doesn't it?
        
               | zmgsabst wrote:
               | Intuition is learned patterns.
               | 
               | You can't have an intuitive UI if it's constantly
               | changing so designers look productive in sprints: you're
               | constantly defying the intuition for your current
               | product, while giving users no stable target to learn the
               | behavior of.
        
         | MisterBastahrd wrote:
         | I hate it as a release engineer. The corners cut to do "rapid
         | releases" usually means sacrificing regression testing which
         | results in angry customers blowing up phones and inboxes.
        
         | barnabee wrote:
         | I like new releases when they mean new features and bug fixes.
         | Not so much when it's just a designer messing around with the
         | UI and graphic design or some "product manager" removing or
         | changing features on a whim, due to A/B testing or whatever.
         | 
         | New releases should almost always only add features or fix
         | bugs.
        
         | honkycat wrote:
         | rapid release is better because the less you release at a given
         | time, the less that can go wrong.
         | 
         | A lot of places just don't have the ability to maintain code
         | quality requisite to do the process properly.
        
         | dvfjsdhgfv wrote:
         | > I miss the days when I could plan ahead for new versions, so
         | I could time them to have the least possible disruption to my
         | life.
         | 
         | That's one of the many advantages of desktop apps: unless it's
         | a bad player that forces the update down your throat, it is you
         | who control if and when to upgrade, not the vendor, who may or
         | may not have their interests aligned with yours.
         | 
         | (It's a bit similar on the mobile if you disable automatics
         | updates, provided that you can ignore that obnoxious red dot
         | forever.)
        
         | MikePlacid wrote:
         | > Rapid release means I can no longer rely on the software that
         | engages in it. It's constantly shifting, with frequent changes
         | in UI and workflow, and new and exciting bugs constantly
         | appearing.
         | 
         | There's something terribly wrong in the picture you've
         | described. I do not know what Agile stands for, but our company
         | was doing "a new version each Friday" for like 15 years. One of
         | the axioms of such a fast turnout was "a new version should
         | work in the old environment _exactly_ the same as the old
         | version" - except bug fixes of cause, some of which were made
         | optional nevertheless.
        
           | Aloha wrote:
           | I think this is a great point. When agile and rapid release
           | is done correctly, it should work out as "kaizen for
           | software" not "we overhaul our UI and workflow every 3-6
           | months".
           | 
           | Small incremental change is easier to test in most cases, the
           | issue is, changing software that was designed as a whole over
           | to incremental feature improvements is hard, because often
           | the existing bulk of code cannot be meaningfully unit tested.
        
           | AdrianB1 wrote:
           | Your company is doing it right, but that is not Agile :)
        
         | crazygringo wrote:
         | Counterpoint: as a user, while you realize you hate the
         | frequent changes in UI and workflow -- you're not realizing
         | that you're using software with a much better UI now _overall_
         | than it would have been if not for the frequent experimentation
         | over the past e.g. 2 years.
         | 
         | Rapid release and experimentation means ultimately finding
         | better solutions for the users faster. If you go slow, things
         | don't change as much but they don't get better much either.
         | 
         | We tend to assume that all of the good changes would have
         | happened _anyways_ and all the bad changes are because of rapid
         | release.... but that 's not how it works.
        
           | plorkyeran wrote:
           | It is very unusual for software that's been around for a
           | while to actually have better UI than it did two years ago.
           | Every time I pull out an old laptop or phone running 5+ year
           | old software I notice that things _look_ different and
           | buttons are in different places, but functionally there 's
           | very little difference.
           | 
           | Most changes to already functional software are just
           | _changes_. They aren 't improvements or regressions; they're
           | just making things different.
        
           | oceanplexian wrote:
           | Pick up a vintage Mac and a copy of Word from the 90s. The UX
           | is superior in every way to modern word processors. It's more
           | responsive, more intuitive, faster to start up, more
           | reliable, easier to learn, and so on.
           | 
           | Decades of "experimentation" have not improved the act of
           | typing out a document in any meaningful way. Now, they have
           | improved the experience for the corporations who can profit
           | selling out their users data to the highest bidder and
           | locking them out of their own devices while charging a
           | subscription fee.
        
             | jameshart wrote:
             | The rose tinted glasses people have for old computers.
             | Here's a transcript of me using a Mac SE in 1995 to try and
             | finish writing a college essay in Microsoft Word:
             | 
             | Chugchugchugchugchugchugchug
             | 
             | Watch icon
             | 
             | Chuchchugchugchugchug
             | 
             | Disk eject sound
             | 
             | Please Insert the disk 'Docs 1'
             | 
             | Clunk
             | 
             | Watch icon
             | 
             | Chugchugchugchugchug
             | 
             | Word count: 2351
        
           | pdntspa wrote:
           | Why do you assume the changes automatically mean "much
           | better"?
           | 
           | In my experience it feels about 50/50 for better/worse,
           | whether that is features taken away due to some business
           | concern, blatant contempt for existing muscle-memory, or
           | misguided designers reworking things that did not need a
           | rework.
        
           | candiddevmike wrote:
           | Your entire counterpoint assumes the UI improves...
        
             | Godel_unicode wrote:
             | This is an excellent example of the headwinds/tailswinds
             | asymmetry; we often don't notice when a rough edge gets
             | sanded off the UX but we scream when one gets added.
        
               | 411111111111111 wrote:
               | I disagree that ui changes in general are an example of
               | this.
               | 
               | You'll be able to find examples such as that, sure. But
               | that doesn't make it generally true.
        
               | jacobr1 wrote:
               | One of the keys to rapid iteration is having a good
               | feedback-loop. So if there are UX regressions, they can
               | be rapidly identified and rolled back.
        
               | lazide wrote:
               | In my experience, that only happens when customer
               | attrition is double digits. Otherwise, everyone has their
               | head up their ass enough they'll ignore or spin the
               | feedback at 99% of organizations.
        
               | wizofaus wrote:
               | Short of having an army of testers whose job it is to
               | continually retest the same UX sequences with every
               | release, how is that possible? Rapid release to me
               | implies releasing changes faster than they could possibly
               | be manually tested. And automated tests are going to do a
               | rotten job of detecting most UX regressions.
        
           | torginus wrote:
           | You are posting on a site that is performant, light on
           | resources and has great UX - and has been mostly unchanged
           | for a decade. Its counterpart, Reddit has only regressed
           | compared to its biggest competitor, it's old self. I haven't
           | noticed _any_ significant UX changes in any software I use in
           | the past two years, and would struggle to name improvements,
           | even over a longer timeframe.
        
             | [deleted]
        
             | dntrkv wrote:
             | "Great UX"
             | 
             | If HN had such great UX, there wouldn't be multiple browser
             | extensions (and clones) that focus solely on fixing the UX.
             | I'm not saying the UX is terrible, but I wouldn't call it
             | great. I think the best term for it would be "consistent."
             | It's basically Craigslist. It's fine if you've been using
             | it for years and have adjusted your behavior to work with
             | its quirks, but it does not provide a smooth experience
             | that would work for a community you're starting today.
        
               | mod wrote:
               | What types of changes would you make?
        
               | wizofaus wrote:
               | The grey text for posts needs to go, I always think it
               | means they've been downvoted, and is surely insufficient
               | contrast to be WCAG/508 compliant.
        
               | plorkyeran wrote:
               | Light grey text does mean that the post has been
               | downvoted below zero.
        
               | mod wrote:
               | He may mean the "visited link" color in the default color
               | scheme.
        
               | wizofaus wrote:
               | My understanding is that that's only true for comments
               | not posts?
        
               | lazide wrote:
               | There is no UX that is the best for everyone, there are
               | always trade offs that hurt some or help others.
               | 
               | Just because there are other products with different
               | experiences that skin it doesn't mean _the core product
               | is bad_!
               | 
               | It actually means the opposite - the core product has
               | such value and there are so many users, it actually is
               | able to support sizable enough communities of people
               | 'outside the core' to justify niche repackagings for
               | them!
        
               | pdntspa wrote:
               | Exactly this. Too many definitions of "great UX" seem to
               | focus on pulling in as many morons as possible because we
               | refuse to force people to learn how to use something
               | 
               | Great UX should also mean fast, efficient, and dense with
               | information. Respectful of time, attention, and
               | intelligence. Not making an assumption of stupidity on
               | behalf of your users.
        
               | barnabee wrote:
               | 100%.
               | 
               | Great UX should _primarily_ mean those things, at least
               | for a certain subset of software.
        
             | crazygringo wrote:
             | Frankly, HN has terrible UX.
             | 
             | The default font size is unreadably small, it's impossible
             | to reliably tap on upvote without accidentally hitting
             | downvote on mobile screens, comments with code blocks with
             | too many characters in a line break the width of the
             | interface, huge comment threads indented 15 times aren't
             | collapsed by default, downvoted comments lighten to become
             | unreadable, I could go on.
             | 
             | But there's no decent alternative for HN's content and
             | community, so we put up with the terrible decade-old
             | interface because we don't have a choice. Although people
             | are constantly inventing their own clients for it precisely
             | _because_ it 's so terrible.
        
               | barnabee wrote:
               | String disagree.
               | 
               | The only of those complaints I even half recognise is the
               | voting buttons on mobile, but I miss them rarely enough
               | that I'd probably still rather not give up any more
               | screen real estate (horizontally or vertically) or add an
               | extra click.
               | 
               | Otherwise HN has basically the UX I wish every site had.
        
           | JohnFen wrote:
           | This an interesting point.
           | 
           | I would disagree that UI constantly improves, though. I can
           | think of lots of examples where the opposite is true. But
           | that's not due to rapid release, that's due to other issues.
           | 
           | > If you go slow, things don't change as much but they don't
           | get better much either.
           | 
           | It's hard to think of when I've seen improvements that were
           | so good as to be worth the cost, though. Things certainly
           | change, but change is not always better, and even when it is,
           | it's not always worth the cost to the user.
        
           | horsawlarway wrote:
           | I think you're missing an obvious dichotomy here.
           | 
           | Better UI is _not_ universal. And the absolute, hands down
           | unquestionably best, UI that can possibly exist for any given
           | customer is the one they 're already an expert in.
           | 
           | Can a different UI _end up_ being better for them? Sure. Will
           | it be better the day it 's released? No.
           | 
           | The user has to pay a cost every time the UI changes,
           | regardless of whether it's actually better for them or not
           | (and it will _never_ be better until much later, once they
           | 've become an expert in the new version).
           | 
           | Worse - I see more and more companies optimize UIs for the
           | "initial user experience" and onboarding new users (more
           | paying customers). The sad truth is that many times that
           | means dumbing down a much better UI for the "already expert"
           | customer - just like putting training wheels on the bike is
           | great for the person that's never biked, and utterly
           | infuriating if you already know how to ride.
           | 
           | This is why existing customer bases are so rarely fond of
           | _any_ change - regardless of whether they end up liking it in
           | 3 months time. It has a real cost, right this second, and
           | they 're being asked to pay it.
        
             | lazide wrote:
             | It's also an existential risk for many business customers.
             | 
             | If you already spent months training a new hire on how to
             | get something working with all the different random tools
             | and software they use, _changing something meaningful_
             | means breaking that persons ability to do what they need to
             | do for their job, at least for some time.
             | 
             | Even changing something inconsequential will confuse some
             | users for a bit, require making/updating new training
             | guides, etc.
             | 
             | If it's a business on the edge or under during a stressful
             | period? That can literally tip it over and make it non
             | functional. That is especially true for small businesses.
        
               | crazygringo wrote:
               | Business productivity software is _usually_ much more
               | slow-changing than consumer software, precisely for this
               | reason. They know customers ' training guides need to be
               | rewritten so changes become a big deal.
               | 
               | This is one of the reasons business productivity software
               | becomes unwieldy after a while -- it's fine to add
               | buttons and menus, you just can't change existing ones or
               | what they do. So then version 5 will finally introduce
               | the massive, desperately needed UI refactor after ten
               | years, while plenty of clients just stay on version 4.
               | (An example of this is the Blackbird CMS -- old Blackbird
               | vs new Blackbird.)
        
               | Silhouette wrote:
               | _So then version 5 will finally introduce the massive,
               | desperately needed UI refactor after ten years, while
               | plenty of clients just stay on version 4._
               | 
               | If your new product with ten years of improvements was so
               | unattractive compared with the existing version your
               | customers are currently using that they were choosing not
               | to update, what does that say about the value to the
               | customer of all those improvements?
               | 
               | In the world before SAAS we saw both extremes often. Some
               | products had incredible dev teams and people would
               | upgrade relatively often even when each new version cost
               | extra money. Other products barely produced an
               | interesting change in several new versions over several
               | years and users mostly didn't pay for those updates.
               | 
               | SAAS is a convenient business model for the developers in
               | the latter category because it locks in the ongoing
               | revenues whether any future changes add value or not.
               | They just need enough lock-in effects that once you've
               | signed up you can't easily move to one of their
               | competitors.
        
               | lazide wrote:
               | Yup, until eventually it becomes unusable due to the
               | sheer weight of cruft.
               | 
               | Then, switch to new vendor or the next major version.
               | 
               | Software Lifecycle for reals baby!
        
         | eastbound wrote:
         | - Agile - DevOps - Continuous integration - Continuous
         | deployment - Rapid release - Growth hacking, if you have 10+
         | employees, A/B testing, heatmaps.
         | 
         | I've been working alone for 12 years and all those concepts
         | were already working in the last large corp I was in.
         | 
         | What trends am I missing by being disconnected from the
         | employment market? Am I missing anything or do we still develop
         | like it's 2010? Is there no new process or evolution on the
         | style of management of software? Is Agile the final form of
         | product management?
        
           | eastbound wrote:
           | The notable trend after DevOps was Kubernetes since 2020, and
           | the notable trend for the frontend was the generalization of
           | NPM-based programming, from React to Typescript, which
           | enabled larger teams and more stable UIs, at the cost of 1GB
           | of JS in the browser.
           | 
           | But they don't seem as earth-shattering as missing the Agile
           | boat.
        
         | stcredzero wrote:
         | _> Which means I dislike Agile by consequence._
         | 
         | From the article:
         | 
         |  _If I were to generalize, I'd say most of the companies I'm
         | picking on fall into one of two categories:_
         | They believe in DevOps ideas, but they're unwilling to invest
         | in changing their ways.         They aren't convinced that some
         | of the scarier ideas (like deploying multiple times per day)
         | will help after all.
         | 
         | Isn't it amazing how one could take "DevOps" and substitute
         | "Agile" and it would fit exactly?
        
         | MetaWhirledPeas wrote:
         | I think a multi-pronged approach works best. A pipeline for
         | users who want the latest changes, and a more stable pipeline
         | for users who dislike change and mostly just want security
         | updates and bug fixes.
         | 
         | The trick would be giving users more distinct choices. For
         | example, with something like Chrome the choice is between
         | extremely frequent and just plain frequent. They don't really
         | offer any long-term version.
        
         | colechristensen wrote:
         | There is a lot of software I've used where I would have been
         | just fine if all development stopped. I don't need or want new
         | features, keeping up with security fixes, bugs, and OS updates,
         | is fine, but feature wise many things I would have been happy
         | to call the software done.
         | 
         | But that doesn't keep programmers employed, VC's making money,
         | or management justifying themselves.
         | 
         | Frustrating.
        
           | NonNefarious wrote:
           | Yep. Word for Windows 6 was pretty much the end of the road
           | for word-processor improvement. It has been nothing but
           | regressions since, to the shitshow that is Word today.
        
             | colechristensen wrote:
             | I really liked iTunes at some point around 2006 and then it
             | got progressively worse, dropped my favorite features, and
             | I no longer touch it.
             | 
             | All I needed them to do was improve performance for large
             | libraries.
        
         | mym1990 wrote:
         | I think this sounds like an issue with the change management
         | part of the cycle. It is also a bit of a chicken and egg
         | problem, where agile development has created an environment
         | where changing requirements from customer/user are almost
         | expected, and now we justify those by using agile more and
         | more. Apps also look to implement ever growing numbers of
         | features that no one really cares about, but sound like a good
         | idea in the weekly leadership call.
        
           | lazide wrote:
           | A lot of it is also out of control careerism, promo culture,
           | and short sighted product management.
           | 
           | No one is going to say No to something new and sexy,
           | especially not the customers (since we won't ask them).
        
         | cjsawyer wrote:
         | I miss being able to actually refuse an update. Or even being
         | able to disable them!
        
           | dleslie wrote:
           | I miss knowing that updates even happened. I also miss
           | important features remaining stable.
           | 
           | Which is why, for the most part, I've stopped using web-based
           | applications.
        
             | ethbr0 wrote:
             | You want a chuckle, turn off automatic app updates on an
             | Android phone and see how many prompts you get. And how
             | terrible the patch notes are ("Fixed some things")
        
               | actionablefiber wrote:
               | I wonder if developers who do this simply don't want to
               | publicly identify and take responsibility for bugs that
               | they introduced earlier. It's easier to just say "bug
               | fixes" then it is to say something like "I overlooked
               | several surprisingly common edge cases in the parser
               | module and have updated it to correctly handle several
               | character sequences that were causing exceptions to be
               | thrown." With automatic updates and vague update notes,
               | anyone who did not directly experience the bug never has
               | to know it existed in the first place and the developer
               | never has to admit it existed in the first place.
        
               | dleslie wrote:
               | In fact, my Android 11 device has no browser enabled, the
               | play store is disabled, and almost every app it came with
               | is disabled. It functions solely for email, instant
               | messaging, a TODO tracker, and taking photos.
               | 
               | Edit: not sure why this was downvoted. Did someone find
               | the idea of a narrowly-focused phone to be offensive or
               | unworthy of discussion?
        
               | OkGoDoIt wrote:
               | Nowadays even on iPhone half the apps phone home to their
               | server, determine they're not running the latest version,
               | and then refuse to let you use them until you update.
        
               | bluGill wrote:
               | I have 25 applications on my android that never get
               | updated. I don't use gmail, and I don't want it installed
               | on my phone at all. (I have the mandatory account to
               | activate the phone, but all my mail is via fastmail). I
               | update the apps I use (or sometimes apps I think I should
               | use but don't)
        
               | jjav wrote:
               | > (I have the mandatory account to activate the phone,
               | but all my mail is via fastmail).
               | 
               | Interesting, I don't have a google account associated
               | with my phone. It means the google app store doesn't
               | work, but everything else works fine AFAIK. I use FDroid
               | or just download apks directly.
        
           | techsupporter wrote:
           | Or that every ducking time an application developer pushes an
           | update, they feel the need to pop up something in front of me
           | the next time I open the application. The _worst_ time to do
           | that is when I first open the application because that 's
           | when I've made the intentional choice to use it to get
           | something done and the last thing I want is for that flow to
           | be interrupted.
           | 
           | The irritation is tripled if it's an un-skippable "let us
           | walk you through the nineteen changes in a slowly constructed
           | slideshow!"
           | 
           | This is second only to applications that don't properly
           | retain state--especially after an update--so every time I
           | open them I get a brief glimpse of what once was (and what I
           | probably wanted to go back to) before I get a 50/50 split of
           | being dumped to the application home screen or "oops,
           | something has gone wrong! lol, software, yannow! Please to be
           | logging in once more, thx."
        
             | kbelder wrote:
             | "We've improved the way you interact with our app!"
             | >click next.
             | 
             | Cursor slowly moves to the upper-right corner of the
             | screen. "We've added a 'search by topic' functionality to
             | our help menu! No more need to scroll alphabetically!"
             | >click next
             | 
             | Cursor slowly moves to the lower left corner of the screen.
             | A red circles appears, pulsing. "The toolbar now displays
             | the two-letter country code that your operating system is
             | set to. Useful for travelers!"                   >click
             | next
             | 
             | Cursor slowly moves to the upper middle of the screen. A
             | popup appears. "If you have forgotten to fill out your
             | phone number and email address in the Account Options, a
             | helpful reminder will appear each time you start the
             | program. This will help you reap a number of useful
             | benefits that sharing your information allows!"
        
             | dleslie wrote:
             | If your feature requires a tutorial _for all users_ then it
             | shouldn't be shipped.
             | 
             | I'm strict on this; I return video games with unskippable
             | tutorials. It's cause for an immediate Alt-F4 and return to
             | store.
        
               | barnabee wrote:
               | If it requires a tutorial for all users, that's called a
               | manual and it doesn't have to appear on launch or force
               | you to read it. Unskippable step by step tooltip guides
               | are the absolute worst.
               | 
               | Many features are better off hard to find and left that
               | way than shoved-in-your-face discoverable, or introduced
               | by a tutorial. Those who want to get the most out of
               | their tools will find them and will appreciate you
               | treating them like fellow intelligent beings. The rest
               | will never know what they missed and won't worry about it
               | either.
        
               | ThunderSizzle wrote:
               | How does Left 4 Dead's style fall in to this (assuming
               | the intro vid would've required on first run)
               | 
               | The tutorial is basically the intro video with some tool
               | tips periodically in game.
        
         | YetAnotherNick wrote:
         | Rapid release has nothing to do with shifting UI. You can have
         | constantly shifting UI even with weekly release.
        
           | vladvasiliu wrote:
           | I presume GP would call a weekly release "rapid", too.
           | 
           | Given the tone of the message, I expect "non rapid" to mean
           | with a much longer time frame, and, most importantly, the
           | release must be controllable by the user.
        
             | jen20 wrote:
             | > most importantly, the release must be controllable by the
             | user.
             | 
             | This is pining for a world which no longer exists, because
             | security patches must be applied to anything that matters
             | immediately.
        
               | lazide wrote:
               | There are things called 'branches' you know.
        
               | diffeomorphism wrote:
               | That is completely orthogonal and the whole selling point
               | of things like Windows LTSC or Red Hat: Stable
               | foundations with regular security updates but no
               | surprises.
        
               | Silhouette wrote:
               | It is possible to separate security updates and other
               | critical bug fixes from functional/UI changes. At least
               | it is if you're any good at software development at all.
               | The world would be a better place for users if more
               | software developers still did that.
               | 
               | Of course you do need a strategy for maintaining multiple
               | versions of something in that case instead of just
               | forcing the latest version of everything you do on users
               | whether they want it or not. With the lock-in effects of
               | the always-connected web applications of today we all
               | know which option makes more money.
               | 
               | I take some comfort in knowing that eventually most of
               | the developers that force unwanted changes on their users
               | will probably see their market share erode and lose their
               | dominant positions. It's just unfortunate that we have so
               | many quasi-monopolies in both essential and niche
               | software products that it's going to take a long time for
               | that to happen.
        
             | JohnFen wrote:
             | Yes, weekly is certainly rapid.
        
           | theptip wrote:
           | Yes, was going to post the same reply. Iterative UI, A/B
           | testing, webapp-driven (vs. installed versioned apps) are
           | completely orthogonal to the OP discussion of Continuous
           | Deployment / rapid deployment.
           | 
           | CD is a deploy strategy saying the aspiration is to do one
           | deploy per commit, rather than batching commits and deploying
           | once per N weeks or whatever. It has approximately zero to do
           | with UX and product design/iteration approaches.
        
             | wizofaus wrote:
             | One deploy per commit for a product with large team sounds
             | insane - we'd easily have 50 commits a day for some hosted
             | components. Even deploying this often to the non-prod
             | environments is problematic (it takes a fair bit longer
             | than 1/50th a day for a start!), but doing that to your
             | customer-facing production sites sounds bonkers - what
             | class of problems is there that that's the best way to
             | prevent them?
        
               | YetAnotherNick wrote:
               | > customer-facing production sites sounds bonkers
               | 
               | We came from very different lands I guess. Batching 50
               | commits with no easy way of knowing which commit caused
               | the issue seems scarier to me, in case of an issue. And
               | if you code correctly and have warmup requests, your
               | deployment could be without any downtime. Obviously there
               | will be some extra resource usage because of unready
               | resources.
        
               | wizofaus wrote:
               | But if you're doing 50+ deploys a day, how can there
               | possibly enough time to know which commit caused whatever
               | issue that gets noticed anyway? If those 50 deploys are
               | done during regular business hours that's 9 minutes per
               | deploy on average!
        
               | YetAnotherNick wrote:
               | Reply to wizofaus: 9 minutes is more than much more than
               | enough to revert, at least in our setup. Specially if
               | some automated alert triggers, then we could revert well
               | within a minute.
        
               | wizofaus wrote:
               | I can't even begin to think how to get there, at least
               | using CloudFormation.
        
               | theptip wrote:
               | Yes it's aspirational, and of course you have to do
               | batching when the rate of commits gets too high. Many
               | companies don't actually end up at the end of the
               | spectrum with full continuous delivery; most of the value
               | is captured by building what's needed _to be able to_ if
               | you chose to, and doing say daily or hourly automated
               | deploys.
               | 
               | If you're not familiar with the general arguments for why
               | CD might be desirable, a good place to start would be the
               | books Continuous Delivery
               | [https://smile.amazon.com/Continuous-Delivery-Deployment-
               | Auto...] or Accelerate
               | [https://smile.amazon.com/Accelerate-Software-Performing-
               | Tech...]. The TLDR would be that most deploy problems in
               | practice turn out to be because the process is manual,
               | infrequently-used, and/or unreliable, and by practicing
               | "if it hurts, do it more often" you are forced to iron
               | out those kinks. For example you can't deploy
               | automatically without a good acceptance/smoke test suite,
               | and without solid DB migration testing/linting/practices,
               | to name just two of many. You could just build all these
               | things without trying to do CD, but in practice it seems
               | that it's quite hard for organizations to prioritize this
               | work in isolation, and the goal of CD seems to provide a
               | fairly clear map to a much higher-quality deploy process.
               | The practice of deploying more frequently to flush out
               | issues, is also quite unintuitive as you note.
               | 
               | Here's a (probably out-of-date) article on how FB did
               | deploys in 2017:
               | https://www.infoq.com/news/2017/09/facebook-release-
               | scale/ -- batched commits and staged canary rollouts
               | automatically cut and released every few hours.
        
               | wizofaus wrote:
               | Yes I do get the general argument for CD (though I have
               | issues with how universally it can be applied based on
               | the sort of BVT I've had to do for SaaS platforms I
               | worked on in the past, where it just wasn't possible to
               | safely release new versions of key components without
               | needing several days' worth of testing, most of which was
               | waiting for verification that interactions with banks
               | etc. had worked as expected).
        
             | JohnFen wrote:
             | > It has approximately zero to do with UX and product
             | design/iteration approaches.
             | 
             | The connection is that if the commit changes how the
             | product works in a way noticeable to the user, then the
             | user must suffer the changes frequently. Since changes
             | incur a cost to the user, it's requiring the user to
             | constantly be paying those costs.
             | 
             | If the deployment happened less frequently, then the user
             | would incur the cost less frequently, and perhaps in a way
             | they could schedule when to take the change on so it is the
             | least disruptive to them.
        
           | bluGill wrote:
           | I've worked with software that has to work. Our customers put
           | it in a lab where they beat on it in simulation for 3 more,
           | months - any bugs and they stuck with the old version. (which
           | is one reason land lines were so reliable in the 1990s) We
           | did a code freeze 6 months before release in hopes of
           | catching everything.
           | 
           | Most software nobody cares if it works or even is possible
           | for untrained humans to use, and it shows.
        
             | morelisp wrote:
             | The glut of isolated B2C SaaS products/developers has
             | absolutely destroyed any communal valuation of "known
             | issues." True in-house heads still remember.
        
         | dllthomas wrote:
         | Do you feel like having different channels of varying
         | "stability" sufficiently addresses your concerns?
        
           | bombcar wrote:
           | Part of the problem is that so much of our "software" is now
           | just weaponized webpages.
           | 
           | And so there is no "staying on windows 7" for SaaS.
        
             | [deleted]
        
             | [deleted]
        
           | SpicyLemonZest wrote:
           | It can in some contexts, but I've never seen a "release
           | channels" approach for e.g. a configuration UI. I'm not sure
           | if that's because it's fundamentally incompatible with
           | business needs or it's just not something UX devs are aware
           | of.
        
             | lazide wrote:
             | It adds a huge support burden to devs and UX, as they need
             | to support an increasing number of different UX versions or
             | basically never change anything (which is bad for their
             | career).
        
           | JohnFen wrote:
           | It's better than nothing, but it isn't really sufficient.
           | Typically, the "stable" channel still releases too
           | frequently.
           | 
           | What would address my concern is if it were possible to skip
           | releases, however rare or common they are, as I wish. But
           | that raises an entirely different Bad Thing in the modern
           | software industry -- that security updates are increasingly
           | inseparable from other updates.
        
         | mint2 wrote:
         | Yes the end user experience is not good with too frequent
         | updates. Vs code asking me to restart every single week for
         | updates is just a bad user experience. I would much prefer
         | monthly or quarterly bigger updates.
        
         | damagednoob wrote:
         | Rapid release is fine but companies will try and cheat:
         | 
         | 1. Release often
         | 
         | 2. When a release goes badly (and they will) do some version of
         | '5 whys' and fix the root cause.
         | 
         | 3. Continue to release often
         | 
         | That step 2 is often missing and what usually happens is that
         | no engineering time is committed to the solution. Instead they
         | implement more process which costs nothing, doesn't solve the
         | issue and slows development.
        
           | lazide wrote:
           | Then when development gets slow, fire the laggards!
        
         | Frost1x wrote:
         | I hate both sides. Rapid development loops should be just that,
         | development. Bug fixes are fine because they tend to guarantee
         | functional consistency and no UI/Ux changes (unless it's fixing
         | one that was broken).
         | 
         | Today it's just the wild west of people basically having
         | production not far sepeeate from development in terms of
         | changes. UI/UX changes are the _worst_ in this context. I
         | opened Venmo the other day and I had to waste 5 minutes because
         | someone decided they wanted to change the UI and I had auto
         | updates applied (shame on me I suppose).
         | 
         | Do not change the interfaces with your users folks unless it's
         | truly broken.
        
       | jascii wrote:
       | I think one of the problems with the terms "Agile" and "DevOps"
       | is that they are pretty general and generic terms that they are
       | trying to cast very specific meaning too...
       | 
       | "So you're saying you're not an agile team?" Sure we are
       | appropriately agile for our needs, no need to be rigid about
       | it...
        
       | AtNightWeCode wrote:
       | You go up at 5 AM in the morning. You ride your bike to a
       | customer. You pass some security checks like, known guy, yes, no.
       | You are shown to the servers in the cellar. Cause they are safe
       | there. You get access to the admin account after 2 hours. You do
       | some backup to a local disk. Somebody screams something but you
       | can't hear what because of the noise. You say yes and hopes it
       | was about coffee even though the room is like a sauna. Now you
       | pick up that CD and insert it. You are not allowed to use a USB-
       | drive because it is mutable. Some clicks and the installation
       | begin. And now you wait.
       | 
       | I am pro-devops.
        
       | torginus wrote:
       | I'm confused about DevOps - I thought it was about dev teams
       | owning the infrastructure, testing and deployment pipelines, so
       | that there is no clear delineation between they person who writes
       | tests, the person who sets up CI/CD pipelines and the person who
       | codes the app.
       | 
       | It has nothing to do with frequency of deployments, and it does
       | not mean that no ops team is necessary for running the show, it's
       | about team autonomy and better understanding of different teams'
       | concerns because everyone is doing a bit of everything.
        
       | honkycat wrote:
       | "DevOps" is a corrupted term. It just means sysadmin at this
       | point.
        
         | mikkergp wrote:
         | While I get the underlying argument you are trying to make,
         | there are definitely still sysadmins in the world who see their
         | job as administering systems, and refuse to look at anything
         | resembling automation.
        
         | bot41 wrote:
         | I think it means devs that write their pipelines themselves.
         | Someone else has to enable the devs to do that
        
       | MetaWhirledPeas wrote:
       | > We need better definitions for these terms
       | 
       | How about this? Let's describe levels of adoption. Think of the
       | "normal form" concept for databases. Each level has a strict
       | definition, and you can objectively observe which level you are
       | using. And not every level is appropriate at all times.
       | 
       | So maybe a team could say, "we're at DevOps level 2," or, "we use
       | level 3 Scrum."
        
         | leetrout wrote:
         | Yes. I like this concept a lot. Maybe accompanied with a
         | landscape of verticals and how you progress in each one.
        
       | synu wrote:
       | I feel like devops became a marketing juggernaut then somehow
       | evolved into creating a third team, apart from dev and ops, that
       | does handoffs with both of them.
        
       | PedroBatista wrote:
       | > I think companies feel that if they aren't going to "do
       | DevOps", that they need to rebrand the term "DevOps" to mean
       | "what we happen to already do".
       | 
       | > To me, the most egregious example of this is when a company
       | simply changes the job titles of all its ops people. John use to
       | be an "IT Operations" guy, abut now he's a "DevOps Engineer".
       | 
       | This is so true it hurts..
        
       | NonNefarious wrote:
       | Key points in this article:
       | 
       | Stop using terms you don't (or can't even) define. "DevOps" is a
       | great example.
       | 
       | Admit when something sucks.
        
       | draw_down wrote:
       | I see the point, but it's funny that the author lays out the
       | reason why this goalpost-moving is happening.
       | 
       | As pointed out, you sound like an idiot if you're against Agile.
       | So it's a lot easier to get what you want if you just call it
       | Agile.
       | 
       | I get that it's not intellectually rigorous or whatever, but good
       | grief, when has that ever mattered in business. Is there any
       | class of concepts that survives unblemished after contact with
       | reality?
        
         | rufflez wrote:
         | Agile is bogus nonsense - there I said it.
        
           | latchkey wrote:
           | Cool... what's your alternative?
        
             | rufflez wrote:
             | Software used to be built before Agile...
             | 
             | Let's at least keep solution design and also get senior
             | engineers to help estimate
             | 
             | And do away with daily stand up and bs rituals.. a meeting
             | is needed when it is needed, not for the sake of "following
             | the process"
        
           | kenrose wrote:
           | Yeah, iterate quickly, learn from your customers usage,
           | repeat.
           | 
           | You're right, total rubbish. /s
        
             | stonogo wrote:
             | Your pithy dismissal leaves out hundreds of pages of
             | religious rites and project management dances which
             | invariably turn into productivity-sucking meeting overload,
             | unachievable goals, and constant product churn, effectively
             | alienating everyone from the developer to the user, leaving
             | only the managers satisfied. Like telling a college student
             | that communism doesn't work, any objection is met with
             | "well they're just not doing REAL Agile," skipping over the
             | fact that nobody on Earth seems to be "doing REAL Agile."
             | 
             | Agile could just as well serve in the headline in place of
             | DevOps.
        
             | rufflez wrote:
             | This is not how it works in real life. Contracts are made
             | to the lowest bidder, there are promises to keep or heads
             | will roll. By iteration 10, your customer is fuming that
             | the half baked idea has more exceptions than handled
             | scenarios and the finger pointing begins. Also, forget
             | about architecture, performance, security and other pesky
             | little problems. I have never seen it play out differently.
             | 
             | The biggest problem is that the person footing the bill
             | does not care about your story points or retrospective- and
             | I don't blame them for that
        
         | drewcoo wrote:
         | > Is there any class of concepts that survives unblemished
         | after contact with reality?
         | 
         | We're not talking about reality, though. We're talking about
         | things consultants market to management.
        
         | tuckerman wrote:
         | I'm not sure I agree with the author wrt Agile. I've never been
         | on a team that did anything that resembled agile but I have
         | been on teams that actively pushed back when anything that
         | looked like it sort of resembled agile got floated. I'm not
         | saying its 100% bad but I'm not sure it has a universally good
         | reputation
        
           | draw_down wrote:
        
       | g051051 wrote:
       | > The ideas behind the DevOps movements undeniably changed the
       | software development world for the better
       | 
       | I absolutely deny this.
        
       | nimbius wrote:
       | when John Willis explained it in 2016 devops had enormous
       | potential for the initiated. id guess maybe 10% of anyone who
       | says 'devops' does anything remotely close though.
       | 
       | devops in 2022 means if youre a dev, youre expected to know the
       | bowels of the OS inside and out as well and be willing to spend
       | hours debugging it alongside your normal programming work. if
       | youre in ops? it means debugging someones undocumented middleware
       | from nine years ago while the front of site burns so "rockstar"
       | coders dont jump ship having to do it instead. if youre in dev or
       | ops, it now means standups where you sit down for an hour and
       | talk about where to put your story in jira or how to enter it at
       | all, and where you discuss broad strategy too. instead of leaving
       | after your status update, youre expected to stay. its just
       | another meeting now.
       | 
       | and once boardmembers and leadership realized they could spare
       | themselves the crucifixion of any real accountability after a
       | breech, they turned devops into devSECops, meaning your infosec
       | team sits in awkward silence as you explain why double defined
       | nginx headers arent a feature and the same management team that
       | made them part of the standups also unilaterally declared ssl3
       | can never be deprecated because customers.
        
         | encryptluks2 wrote:
         | This is my issue. Companies have commoditized the average
         | developer and now treats it like a average entry level role
        
       | freshnode wrote:
       | I think there is a third category - companies that have heard of
       | DevOps, think it means that the development team and operations
       | team talk to one another, which they already do (!) and shut down
       | conversations after that.
       | 
       | I've come across this archetype in several contracting gigs and I
       | find it to be especially pernicious as it gives the incumbents
       | (who usually have more weight in the org) ammo to effectively do
       | nothing.
        
       | ciguy wrote:
       | DevOps has lost all meaning as the first paragraph in the article
       | says. Anything it says after that is just semantics. Everyone is
       | doing it on some level, but what that means to each company
       | differs widely.
        
       | itsmemattchung wrote:
       | I was one of the biggest proponents of DevOps (even started and
       | the DevOps orange county meetup) but ... after working at AWS for
       | 5+ years, my entire perspective shifted.
       | 
       | While DevOps _does_ work in _some_ organizations, sometimes what
       | you really need to do is operationalize your software development
       | team. Don 't get me wrong, system developers/engineers/operations
       | are needed but for sometimes for a small four person start up,
       | the best investment is improving the development team's
       | operational posture (e.g. run books, alarming, monitoring).
        
         | dec0dedab0de wrote:
         | Which definition of DevOps are you against? and what changed
         | your mind against it?
        
         | NonNefarious wrote:
         | That was as much a bunch of meaningless jargon as "DevOps."
        
         | bot41 wrote:
         | > the best investment is improving the development team's
         | operational posture (e.g. run books, alarming, monitoring).
         | 
         | this is devops
        
         | 0x457 wrote:
         | > for a small four person start up, the best investment is
         | improving the development team's operational posture (e.g. run
         | books, alarming, monitoring).
         | 
         | But that's literally what DevOps is? DevOps isn't a special
         | DevOps team that writes terraform.
        
         | lowbloodsugar wrote:
         | The original definition of DevOps was what you describe in your
         | second paragraph - development teams being responsible for
         | operations.
         | 
         | Now DevOps means "We've created a DevOps team! They do DevOps!"
         | Frequently its the managers of old school operation teams
         | changing their names to DevOps teams to get a promotion.
        
           | CoconutPilot wrote:
           | I always liked this early definition of DevOps:
           | 
           | "Giving developers operational responsibilities has greatly
           | enhanced the quality of the services, both from a customer
           | and a technology point of view. The traditional model is that
           | you take your software to the wall that separates development
           | and operations, and throw it over and then forget about it.
           | Not at Amazon. You build it, you run it." - Werner Vogels,
           | Amazon CTO (2006)
           | 
           | DevOps meant that Developers are also responsible for
           | Operations. But that meaning is lost to time, just like
           | hacking meaning programming.
        
           | jen20 wrote:
           | It may seem pedantic, but it's an important difference:
           | that's _not_ what the original definition was. Instead, it
           | was a cross-functional team of developers and operators being
           | responsible for a whole system.
        
             | lowbloodsugar wrote:
             | Yes, I should not have said a "development team", as it
             | makes a distinction between "people who create new
             | software" and "people who run that software in production
             | environment". Your definition is better.
        
         | calvinmorrison wrote:
         | > development team's operational posture (e.g. run books,
         | alarming, monitoring.
         | 
         | Isn't DevOps in large part forcing the developers to have
         | responsibility over the code they push and giving short and
         | frequent pushes to easily rectify problems?
         | 
         | Making developers do support work reduces support in the long
         | run because they'll get frustrated and automate solutions.
        
           | stetrain wrote:
           | Yeah, I feel like the term has been taken over to mean using
           | kubernetes and having continuous deployment pipelines for
           | your microservices.
           | 
           | My understanding of root of the term is about ownership and
           | responsibility, where the team that develops the app also
           | owns how the app deploys and runs in production, including
           | monitoring and responding to issues.
           | 
           | If the team decides that the right way to deploy their app is
           | once every two weeks using a shell script and a zip file,
           | cool. As long as it's documented and monitored so you know if
           | it's working.
        
             | calvinmorrison wrote:
             | Yeah, what's worked well for me is having ops or security
             | people draw up guidelines (for security, it may be
             | something as trivial as require ssh key auth, for ops it
             | may be: write a script that matches an existing deployment
             | procedure and we can get your app up and running very
             | quickly).
        
           | lazide wrote:
           | And/or become burnt out and develop a severe 'learned
           | helpness' complex. It depends on the environment they're in,
           | and if they have the tools and support to be able to fix
           | issues.
           | 
           | Many companies, they don't.
        
         | oneplane wrote:
         | There seems to be an increasing disconnect between the
         | marketing value of a term ("DevOps") and what we're actually
         | supposed to be doing, which might best be described as
         | blameless accountability.
         | 
         | Even in information security it basically revolves around the
         | same things before you can even do any sensible work and
         | improvements to processes.                 - Know what you have
         | - Know where it is       - Know what it is doing       - Know
         | what it is supposed to be doing
         | 
         | If you don't have any of those pieces you're just guessing and
         | hoping for the best. For software that might be "know what
         | services you are building, know where you're running them, know
         | how they are doing, know how they are supposed to be doing".
         | But if we call that "DevOps" is suddenly marketable.
        
           | aidenn0 wrote:
           | Isn't that list just "Ops" ?
        
           | lazide wrote:
           | The issue of course is those things are hard.
        
             | oneplane wrote:
             | They are only hard if they don't get the same priority and
             | attention you'd see in regulated situations. Replace
             | software or information security with "bars of gold" and
             | suddenly it's not so hard.
             | 
             | That's because marketing why you should know the
             | what/where/how/why of your gold to business decision makers
             | is apparently much more palatable than abstract concepts
             | found in IT, even if the value is the same.
             | 
             | If you get the resources (including time, money and
             | manpower) do just do it, it can be done. But you usually
             | don't get all three (and maybe not even one), and therefore
             | you're stuck with a task that cannot be executed well.
        
               | lazide wrote:
               | Did general aviation become cheaper or more available to
               | folks when it got more heavily regulated?
               | 
               | Fatality rates dropped, but it basically froze the
               | technology and capability into the late 70's/early 80's
               | and locked in just a few companies. The overall number of
               | people working in that industry is tiny now. It's almost
               | a complete dead end.
               | 
               | It's the classic trade off - cheaper, better, faster.
               | Pick any 2 if you're lucky. Most only get to pick 1.
        
               | oneplane wrote:
               | Yep, and that is generally also why IT is the way it
               | currently is. Most projects aren't all that complex (in
               | business logic terms), it's mostly CRUD interfaces and a
               | fancy viewer for the end users anyway. But industries
               | that are still on mainframes are almost completely the
               | opposite where every single element is tagged,
               | catalogued, tested, described and traceable at any point
               | in time. With a price, rigidity and lead time to match,
               | of course.
               | 
               | On one hand we don't all want to buy mainframes, on the
               | other hand, doing it as cheap and unmeasured as possible
               | makes everything flakey, demanding on individuals to keep
               | it running, insecure, and opaque to everyone involved.
               | Too bad there isn't really any middle ground, it's almost
               | like it's either "pretend to be a mainframe" or "do it as
               | cheap as possible".
               | 
               | Regarding the aviation analogy: it definitely doesn't
               | become cheaper due to regulation (it did become cheaper
               | due to subsidies), but I'd rather have a system that's
               | regulated (or forced) into safety than having it be
               | optional and left to a sociopath (i.e. a for-profit
               | publicly traded company - all the benefits of pretending
               | to be a person but none of the responsibilities or limits
               | to govern it - except the law and regulation).
        
         | plugin-baby wrote:
         | > for a small four person start up, the best investment is
         | improving the development team's operational posture (e.g. run
         | books, alarming, monitoring).
         | 
         | Oops, I thought that was DevOps.
        
           | cortesoft wrote:
           | Yeah, to me, the core of the 'DevOps' philosophy is that the
           | dev teams owns the product end to end, meaning they write it
           | and are responsible for it in production.
        
           | fennecfoxen wrote:
           | That's actual-devops. Grandparent poster has worked at firms
           | which have abandoned all its lessons and just use it to mean
           | something like "regular old ops teams, but they use
           | software". So he's using the term like all the big companies
           | do, basically.
           | 
           | This is also where you will find things like a "CI/CD" team
           | that doesn't do continuous delivery: they just build Jenkins
           | pipelines, and run them as part of the manual cluster bringup
           | every 2-4 weeks. Or "RESTful" to mean "APIs which use JSON
           | and HTTP verbs" regardless of any hypermedia. And let's not
           | even get started on "agile"...
        
             | jbverschoor wrote:
             | 'word', as they say
        
             | JamesBarney wrote:
             | Here's a 400 page book and a 12 week training course on all
             | the processes we used to implement Agile's "People over
             | process" principle.
        
               | Banana699 wrote:
               | Something something Goodhart's law something something.
        
           | __MatrixMan__ wrote:
           | Me too.
           | 
           | In my experience it went poorly because us devs are now
           | handling our own deployments and building alerting systems
           | and such, but the "devops" people won't give us the level of
           | access necessary to do that with tools designed for the job,
           | so we end up building custom ops-related tooling into the app
           | and writing backdoors that let us use it without having wake
           | the ops folk.
           | 
           | When you're accustomed to having to build everything you
           | need, and you're tasked with ops work, you end up reinventing
           | a lot of wheels in a rather weird way and compromising your
           | app in the process. Or at least that's what we did.
        
           | theptip wrote:
           | This really substantiates the OP's claim.
           | 
           | Maybe we need to come up with a new term to be more specific.
           | "DevOps for developers" vs. "DevOps for operators" or
           | something like that. (DevDevOps va OpsDevOps?)
           | 
           | I always took DevOps to be some combination of 1) applying
           | dev best practices to ops, which could still in theory be
           | done by a separate team, but more 2) unifying dev and ops in
           | one team/role, perhaps enabling developers to own it by
           | providing tools and abstractions that developers can work
           | with, and workflows that unify the processes and improve
           | collaboration by lowering cycle time.
           | 
           | It seems that the "bad" DevOps is OpsDevOps where the old ops
           | team just gets some rebranding, and maybe some new tools like
           | Terraform. I've always found the "DevDevOps" approach to be
           | quite empowering. Giving developers a tool like k8s so they
           | can abstract away the hard half of ops, and understand the
           | rest.
        
       | mnd999 wrote:
       | All these processes are a template, a starting point. Begin there
       | and optimise and refine to suit your team and your environment.
       | Dogmatically following a process that doesn't work for you is
       | silly.
        
       | RHSeeger wrote:
       | I find the article somewhat ridiculous in that it takes the tone
       | that everyone thinks the idea in question are good.
       | 
       | > They aren't convinced that some of the scarier ideas (like
       | deploying multiple times per day) will help after all
       | 
       | For some cases, rapid deployment might make sense (purely front
       | end stuff). For a lot of cases, I'm firmly of the opinion that it
       | does _not_ make sense. I've never worked on a project that
       | deployed that often, and I don't feel the need to. I have
       | depended on services that deployed that often and "oh, sorry we
       | broke that, we'll fix it later; you'll just have to live with
       | permanently broken data" gets my knickers in a twist every time.
       | 
       | > I don't buy into all the ideas behind agile, particularly some
       | of the Scrum-specific ceremonies.
       | 
       | That's because a lot of the Scrum people present it as "the"
       | Agile. And a lot of the non-Scrum people are fairly convince that
       | Scrum is awful. To which the Scrum people present "no true
       | Scotsman" arguments.
       | 
       | I'm all for the underlying "listen to your customer, listen
       | often, and change plans as often as necessary to make them happy"
       | ideas of Agile. But every "methodology and process" for it I've
       | seen completely loses that idea. It's infuriating.
        
         | 0x457 wrote:
         | I think your issue isn't rapid release cycle, but lack of
         | proper testing.
        
       | t_mann wrote:
       | If you have a separate DevOps role in your org, arguably you're
       | not doing DevOps. So probably most orgs aren't doing DevOps, at
       | least those that are talking about it.
        
       | the_jeremy wrote:
       | My team at Nasdaq (I work on other products, not the stock
       | exchange) does DevOps surprisingly well for not being a Real Tech
       | Company TM. I don't know that the business sees benefits, as it
       | meant that all of the devs on my team have had to learn
       | Terraform, Helm, AWS, and Kubernetes in addition to the actual
       | SpringBoot Java application code, so our feature velocity dropped
       | quite a bit, but it was fun to learn, and it's probably better to
       | have people who understand the full deployment and not just pass
       | blame between teams during incidents.
        
         | lmarcos wrote:
         | Oh man, that's the trap. I do like to learn new things, I
         | swear. But at work I want to focus on what I do best: feature
         | development. Sure, I do know terraform and k8s and what not
         | (because I play with them via my side projects), but at work
         | don't give me the Ops/Platform hat please! It's cheap labor
         | (your company doesn't want to hire a dedicated platform
         | engineer to handle terraform/k8s/etc. and instead assigns you
         | (the developer) that responsibility claiming "you are gonna
         | have fun learning" and paying you 25% more in the best
         | scenario)
        
           | MetaWhirledPeas wrote:
           | I don't know if you're right or not, but I do think your
           | hesitancy mirrors why a lot of organizations are unable to
           | adopt DevOps fully: many people don't _want_ to wear a bunch
           | of hats. They just want to do their one thing, and I can 't
           | really blame them.
        
             | tremon wrote:
             | But I don't think the original DevOps (TM) specified that
             | every person was required to work on every part of the
             | stack? Just that you need to have full-stack skills and
             | responsibilities within the same team?
             | 
             | That said, I probably wouldn't want to hire someone with
             | just one hat. It's ok to have a preferential hat, it's not
             | ok to refuse to wear any other.
        
               | BlargMcLarg wrote:
               | This sounds like a manager's ideal. It's a luxury to
               | demand people wear multiple hats up to the point the
               | market allows you to turn the luxury into a commonality.
               | If it was up to bosses, they'd rather have jokers ready
               | to switch from customer support, to development, to
               | sales, and back.
               | 
               | It's ok until the market decide it is no longer ok. So
               | far, it looks like all these "hats" are complicated
               | enough for the average to make the market think twice
               | about demanding that combination of breadth and depth of
               | skills.
        
               | 0x457 wrote:
               | > But I don't think the original DevOps (TM) specified
               | that every person was required to work on every part of
               | the stack? Just that you need to have full-stack skills
               | and responsibilities within the same team?
               | 
               | Then you have Ops person integrated in every dev
               | team/squad, then those Ops people have their own meetings
               | to talk about Ops things. Then there aren't enough Ops
               | people to staff every dev team (maybe they are on
               | vacation, perhaps just not enough in general), so they
               | start sharing them. Next they need to work on some
               | special Ops tasks, so they want to have their own board.
               | Oops, now you have Ops team.
               | 
               | The point of DevOps is that the team is responsible for
               | code being shipped, and after they shipped it, so
               | everyone needs to know how to work in the stack that runs
               | your app (Kubernetes, Nomad, AWS Beanstalk, bare-metal
               | with Ansible or whatever it is you use). That means devs
               | don't go to some other team to fix their CI or CD
               | pipeline.
               | 
               | I worked in a company that was doing DevOps, we had no
               | Ops team, we had a few people who only worked on infra,
               | but they were part of our team. We didn't call it
               | anything, it was just how to work, about 3/4 of the team
               | was capable of setting up infrastructure and working on
               | Jenkins.
               | 
               | I also worked in the company that "did DevOps" had a
               | "DevOps team" and devs knew absolutely nothing about what
               | happens to their code after they did `git push` (that if
               | they knew how to do `git push` without asking "DevOps
               | team").
               | 
               | Another company "for sure did DevOps right", they didn't
               | have "DevOps team" instead they had "Infra team" and
               | infra team was responsible for bare metal provisioning.
               | Devs could write CI pipelines and stuff. However, devs
               | had zero responsibility for code after they shipped it -
               | it was infra team that would be woken up by nightly Pager
               | Duty alert.
        
           | spaetzleesser wrote:
           | Same here. I recently had to take on some more roles. It's
           | fun learning but I don't perform at the level I could perform
           | when I was mainly writing C# code. Its not even close. It's
           | good to have some insights into all aspects of the work but
           | somebody who can focus on one area but can perform there with
           | multiple productive of someone who constantly jumps around.
        
       | nont wrote:
       | This is also very true in the presales engagements by top
       | consulting firms under "Digital Transformation". These presales
       | teams would come and show, on paper, how great they are with the
       | agile practices, devops, microservices, you name it.
       | 
       | However, it's a totally opposite story during the actual
       | implementations. I've witnessed that it took a top consulting
       | firm 4 months to provision a sandbox environment while being 1
       | month away from the first release. When asked about their proud
       | DevOps practices, you'd hear excuses that are in the blatant lie
       | territory. They knew they could get by by managing the
       | stakeholders instead of really doing what they had promised.
        
       | paxys wrote:
       | I worked at a large internet company in the mid-late 00s, and the
       | typical team composition was a handful of developers, a product
       | manager, a designer (if you worked on customer facing UI), a
       | project manager (for large enough projects), QAs, and a SRE.
       | 
       |  _Every_ team had dedicated engineers whose sole purpose was to
       | place purchase orders for servers, set up and manage databases,
       | deploy changes, monitor uptime and more. And this was standard
       | practice across the industry, as soon as your company outgrew a
       | shared host.
       | 
       | Even the thought of something like this would be absurd today. So
       | I'd flip the "no one admits they don't to it" around - your
       | company does DevOps already, whether you realize/admit it or not.
        
         | water-your-self wrote:
         | In the mid 2000s was the role referred to SRE yet?
         | 
         | Facebook calls it production engineer, DevOps is broad/ im
         | unsure its origin, Google calls it SRE.
        
           | jbverschoor wrote:
           | Within non-tech companies, they call everything "the
           | programmer"
        
         | dilyevsky wrote:
         | Replace "purchase orders for seevers" with "apply terraform to
         | spin up VMs" and ditch the QA person and not much has changed
        
           | [deleted]
        
         | jzawodn wrote:
         | Yahoo?
        
       | 0xbadcafebee wrote:
       | I agree the terms have become pointless. I have been thinking
       | about opening an actual school for DevOps, or at least IT Ops.
       | There is virtually no education for it, but it's a complex
       | subject matter that requires years of experience. The SRE books
       | are a peek under the covers but leave so much unsaid that it's
       | really only useful at "doing operations", not digital
       | transformation.
       | 
       | To properly build and run a high-performing organization, you
       | need to understand at least 1) the entire history (up to today)
       | of software development practice and production operations, 2)
       | all of the competing requirements of product design, development,
       | and operation, 3) architectural best practices, 4) the industry
       | standards that apply to the work, and 5) tech product-aligned
       | business management methods.
       | 
       | Basically you need a generalist who has 15+ years experience and
       | has tried and failed at everything under the sun. I think of it
       | like carpentry before there was a building code, and before the
       | introduction of apprentices and licenses. Trying to build high-
       | performance orgs without a master builder to show you is like
       | trying to build skyscrapers after having built your own home (and
       | having never seen a skyscraper). Or worse, taking an old barn
       | (non-tech enterprise) and trying to build a skyscraper out of
       | that. Without a trade school it will continue to be just a craft.
        
       | robertlagrant wrote:
       | > If I were to generalize, I'd say most of the companies I'm
       | picking on fall into one of two categories:
       | 
       | > They believe in DevOps ideas, but they're unwilling to invest
       | in changing their ways. They aren't convinced that some of the
       | scarier ideas (like deploying multiple times per day) will help
       | after all.
       | 
       | Option number 3 is the biggest one for me: thinking that having a
       | separate "DevOps" team from the Dev team that does ops via modern
       | tools _is_ DevOps. That may be a good approach for your company,
       | but it ain 't DevOps.
        
       | mikkergp wrote:
       | This seems like an example of something similar to the no true
       | scotsman fallacy. (it's inverse) These ideas are supposed to be
       | aspirational, akin to asking "is it a true democracy" or "is it
       | really capitalism". I don't entirely object to the notion of
       | trying to be more precise, but gatekeeping "devops" or "agile"
       | also seems to sort of miss the point.
        
       | ipnon wrote:
       | SRE seems the better approach. Make an error budget, measure
       | reliability, stop releasing in favor of reliability engineering
       | if the budget is missed, use the free time reduce toil. DevOps
       | seems to degrade to disorganization because the incentives are
       | not aligned between product development and software engineering.
        
         | jameshart wrote:
         | "Make an error budget, measure reliability, stop releasing in
         | favor of reliability engineering if the budget is missed"
         | 
         | These are all things that fall in the realm of 'ops'.
         | 
         | 'Dev'ops is an approach to 'doing ops'. It is one that adopts
         | certain practices that are common in 'dev'elopment - such as
         | defining things programmatically, using automated repeatable
         | processes to generate work product, and using automated testing
         | and verification.
         | 
         | So what you describe as 'SRE' doesn't seem to me to be 'not
         | devops'. Seems like a set of ideas that would benefit from a
         | devops kind of approach.
        
         | jedberg wrote:
         | If you think SRE and DevOps are opposed to each other, then you
         | might be doing one of them wrong. I headed up SRE at Netflix,
         | and our sister teams wrote the software and managed the tools
         | that enabled the developers to continuously deploy and operate
         | their own services.
         | 
         | We were in charge of reliability, but we did it through chaos
         | testing and accountability, and enhancing the deployment tools
         | to warn devs that they might be doing something high risk, but
         | still letting them do it.
        
         | tootie wrote:
         | I think you've hit at the heart of the problem, but swapping
         | the DevOps model for SRE is robbing Peter to pay Paul.
         | 
         | Last place I worked with a 100+ person tech team had more of an
         | SRE model. The SRE team were charged with reliability. They had
         | no incentive to delivery features or to improve developer
         | experience. As a result, developer tools sucked, there was s
         | rigid release schedule, every bug was met with a rollback. This
         | was probably an extreme case because the team was also not very
         | good, but either way it was a clear case of misaligned
         | incentives.
         | 
         | The entire point of DevOps was collective ownership with a
         | shared goal of customer value that is function of feature
         | richness, quality and reliability. It's the Holy Grail of
         | efficiency, it's just really hard to build that kind of
         | culture.
        
           | ipnon wrote:
           | Seems like the sweet spot is DevOps style reliability
           | ownership with SRE style indicators, objectives and
           | agreements.
        
       | drewcoo wrote:
       | The entire article avoided talking about definitions of DevOps,
       | while rambling a lot about how we have different ideas of what it
       | means and how we need better definitions.
       | 
       | I appreciate the alert that there's a problem here, but there's
       | no substance to the article.
        
       | eweise wrote:
       | I only do as much 'ops' work as required by the idiots in charge.
       | Its much more productive for me to be working on business
       | solutions than screw around with getting traces and logs into
       | datadog and figuring out all the permission issues in AWS.
       | Someone can specialize in that knowledge instead of everybody
       | needing to know everything.
        
       | mkl95 wrote:
       | Most SaaS are overcooked spaghetti floating on AWS alphabet soup.
       | A devops mentality won't fix them.
        
       | rr888 wrote:
       | For me devops was never a great idea. Ideally you want one team
       | to decide on the architecture. You do need front line ops,
       | ideally they'd take some ownership and write some scripts, but
       | the buck stops with one team for inhouse systems that is usually
       | developers.
        
       | benreesman wrote:
       | When I was a wee lad, "programmers" wrote software and sorta
       | tested it and then kicked it over the fence to "systems
       | administrators", who chain-smoked and were very, very un-woke.
       | 
       | This was a bad system.
       | 
       | To the extent that "DevOps" means anything, it's that the spheres
       | of capability and responsibility need to have tons of overlap:
       | SWE's need to be in the oncall loop and know how the filesystem
       | works, and SRE's need to be serious coders.
       | 
       | That's it: everyone codes, everyone operates. We all automate
       | where feasible.
       | 
       | Sidebar: What's with the crazy surge in BuzzFeed-style titles on
       | the front page? It's not a federal matter but it's not a slight
       | change YoY either.
        
         | kazinator wrote:
         | The test cases for Unix must have been kept on some tapes that
         | were lost, because a test case is hardly to be seen in the
         | historic sources you can find online.
        
       | spiralpolitik wrote:
       | DevOps (and Agile) both usually run aground on the same question
       | of trust. If you aren't willing to trust your delivery teams then
       | you aren't going to see the leverage that both practices bring to
       | an organization.
       | 
       | And if you aren't willing to trust your delivery teams then your
       | organization has a much bigger problem than either DevOps or
       | Agile can solve.
        
       | tibbon wrote:
       | I also go to the gym 4 times a week, go to sleep on a regular
       | schedule and eat a balanced diet!
        
       | alexjplant wrote:
       | According to "The DevOps Handbook" DevOps is...
       | 
       | > the outcome of applying the most trusted principles from the
       | domain of physical manufacturing and leadership to the IT value
       | stream. DevOps relies on bodies of knowledge from Lean, Theory of
       | Constraints, the Toyota Production System, resilience
       | engineering, learning organizations, safety culture, human
       | factors, and many others.
       | 
       | Anybody that's worked in an organization that "uses DevOps" or
       | has "DevOps Engineers" knows that this simply isn't true in the
       | vast majority of circumstances. Too often DevOps is just a new
       | name for an ops team. "True" DevOps is all about aligning
       | incentives, sharing responsibilities, and fostering collaboration
       | between people who build and maintain systems whereas "DevOps"
       | (with air quotes) is just a new spin on being a sysadmin with a
       | greater emphasis on AWS and Terraform.
        
         | baobabKoodaa wrote:
         | Both the quote and your response to it read like word soup to
         | me. I have literally no idea what the point in either of these
         | are.
         | 
         | Historically, the word "DevOps" meant that instead of having
         | separate "Dev" and "Ops" teams, the same people would do both
         | Dev and Ops work.
        
           | lazide wrote:
           | Which is what they're saying.
           | 
           | Because they do both jobs, folks understand both sides, and
           | work together to make things work better end to end for them
           | and for customers.
           | 
           | Alternatives include siloed dev teams that throw shitty
           | software that crashes in production all the time to an
           | overworked ops team that takes all the blame.
           | 
           | And overworked ops teams that have gone pager-blind and only
           | care if a VP yells at them, so ignore all but the most major
           | issues - after they already happened.
        
             | 0xbadcafebee wrote:
             | The fact that engineers are not taught those topics are why
             | DevOps has failed. We're expecting whole organizations to
             | magically improve when they don't understand how. I can
             | count on one hand the number of people I have met who
             | understand OP's comment.
        
               | lazide wrote:
               | Yup. Though like social studies, even if it was mandatory
               | education, I suspect 90% of it folks would immediately
               | forget.
               | 
               | Ops is hard, and irritating, and a pain in the ass.
               | 
               | It's also the process of _actually solving the customers
               | needs_.
               | 
               | If someone can fiddle around cutting and pasting off
               | stack overflow without having to worry about it, most
               | will.
               | 
               | I personally can't NOT be involved, but that's why my
               | resume looks the way it does.
        
         | [deleted]
        
       | rhacker wrote:
       | No one admits they don't do it because everyone does it. I have
       | worked at 6 different companies since I've heard that term and
       | all of them were able to deploy to an environment by clicking a
       | button on a CI/CD tool. Maybe I'm the outlier?
        
         | 0xbadcafebee wrote:
         | Anyone can deploy a file with one button. DevOps is what
         | happens when the app starts randomly crashing.
         | 
         | If the site keeps humming along like normal, or the app is
         | automatically rolled back, or (rarely) the devs get a page &
         | quickly diagnose in production and apply a fix in 10 minutes,
         | and deploys don't randomly fail due to unpinned versions etc,
         | that's DevOps.
         | 
         | If people start running around like chickens with heads cut off
         | and pointing fingers, that's not DevOps.
        
       ___________________________________________________________________
       (page generated 2022-08-29 23:00 UTC)