[HN Gopher] 2 Years at Twitter
___________________________________________________________________
2 Years at Twitter
Author : stanislavb
Score : 43 points
Date : 2022-11-23 20:48 UTC (2 hours ago)
(HTM) web link (eed3si9n.com)
(TXT) w3m dump (eed3si9n.com)
| black_13 wrote:
| softwaredoug wrote:
| > Looking back, I feel like this has been among the most
| productive 2 years of my career. This was facilitated precisely
| because of the work/life balance afforded by the "peak-Kumbaya"
| Twitter culture, and the psychological safety
|
| I feel like the most productive I've been is
|
| 1. Psychological safety and trust in engineering to lead
|
| 2. Coworkers that intrinsically care about their work
|
| 3. Coworkers that are fantastic people
|
| 4. Meaningful work
|
| The top down, hard ass culture of Elon etc, basically flips the
| work to being extrinsically motivated based on fear. If the goal
| is engineering excellence, it just doesn't work. A lot of people
| shut down and won't take any risks or innovate in that situation.
| Or they leave.
|
| There may be other goals of course, but there's a valuable reason
| tech companies have been selective in hiring and generous in
| trust.
| that_guy_iain wrote:
| For me, the interesting part was that there were 10+ people the
| build team and it seems like there are still people left since he
| didn't say all the people left. How large was the team? I
| personally find teams reaching 6+ members too large.
|
| and
|
| > For example, I think it was Adam who pointed out that the Mac
| laptops were unable to fully utilize the remote cache because
| action cache was platform specific. This remains to be an open
| challenge of Bazel to date.
|
| It's always macs that are suffering from problems in my
| experience.
| ladberg wrote:
| The article says about 12 people plus a few extras and that 10+
| left, so definitely a huge chunk of the team.
|
| Reading that line it's evident that the issue isn't Mac-
| specific, just that you can't reuse build artifacts across
| different platforms (which is expected for pretty much any
| build system).
| PeterStuer wrote:
| 2000 people on a build team. ...
| brian-armstrong wrote:
| I think that comment is saying there were 2000 ICs merging code
| at Twitter, at least according to publicly released figures
| ralph84 wrote:
| Another company that copied the engineering practices of Google
| without the profitability to support a huge internal tools team.
| rnk wrote:
| Google had great engineering practices and infrastructure, the
| best of my career across several FAANGS. If only they didn't
| make launching products (and killing the old one) the way for
| advancement.
| gravypod wrote:
| There are many people who are commenting that this is so much
| work for 20m lines of code, they're reinventing the wheel, they
| should have just bought something off the shelf, etc.
|
| Two things to consider:
|
| 1. It is sometimes hard to buy things that works well:
| https://danluu.com/nothing-works/
|
| 2. The productivity gains of something like Bazel and Pants are
| pretty amazing if you are working within a monorepo.
|
| It is very difficult to understand how big these two things play
| into these decisions. Having ~20 people manage build tooling,
| build caching, source control, CI (build+test), and building
| release artifacts (which it sounds like Pants did) is not a bad
| deal.
|
| Consider:
|
| 1. GitLab will charge you $20/month/user. 2. If you have 2000
| SWEs you'll pay 40k/month just to have access to the website. 3.
| You'll still need to hire a person or two to negotiate the
| purchasing of the GitLab.
|
| Now remember that GitLab might not be prefect for your use case.
| GitLab doesn't increase the performance of builds on your laptop
| by sharing a cache with all engineers. There's also a limit to
| the mount of data you can store for source + build artifacts
| (50GB). The CI runners they host for you are very expensive for
| how slow the machines are ($10/1000 minutes).
|
| Sure, you can do things like run your own GitLab runners but now
| you're back to running your own infrastructure which is an
| engineer's job. Sure, you could save money by using Gitea or
| something that if free but again hosting your own infra. As soon
| as you have someone managing that infrastructure they're bound to
| say "how else can I save time and money" and at a certain scale
| these things become reasonable or even required.
| [deleted]
| metadat wrote:
| An unfortunate title, completely devoid of any hint about the
| Very Interesting content in this post:
|
| This is an insider story chronicling the evolution of Twitter
| 1.0's approach to build infrastructure and developer efficiency
| strategies at scale on teams of hundreds to thousands of SWEs.
| msoad wrote:
| 20m lines of code is not large enough to have 6 people in build
| tooling. A lot of companies at Twitter size end up over
| engineering and resisting buying vs. building in house big shot
| projects that often times can't compete with some 3rd party
| solution they could just buy.
|
| For example my company is reinventing Next.js in house because
| the politics in the company won't allow us ditch this half-baked
| Next.js clone and just use the real thing. Our deployment
| pipeline will be much simpler if we simply bought Vercel. I'm
| sure we can negotiate prices with them that is better than hiring
| 12 engineers
| skilled wrote:
| Sooo.... Your company can afford to buy Vercel?
| woodruffw wrote:
| I think the more relevant number is the number of ICs (around
| 2000, according to TFA). 12 build engineers (I don't know where
| you got 6 from) for 2000 ICs doesn't seem large at all to me;
| it's actually less than I would have expected.
| marcinzm wrote:
| Yeah, as long as the team makes the rest of the engineers 1%
| more efficient it's a win in terms of investment.
| atsjie wrote:
| They were not building/re-inventing Bazel (its Open Source from
| Google); they were just adopting it and ensuring the migration
| for the rest of the company (which is no small feat, since it
| affects all other developers in the company).
| stefan_ wrote:
| The weirdest part here is that the author worked on a migration
| from their NIH build tool to Bazel for 2 years and never once
| mentions why all of that was even happening, and by the end of
| his employment, it was still not done.
|
| He links the pants-devel announcement and here is what it says:
|
| > Twitter has made the decision to migrate from Pants to Bazel
| as our primary build tool, as we believe that we have unique
| needs that Bazel can better address
|
| Pants was your own fucking build tool that you invented,
| controlled and maintained! How on earth can it not meet _your
| unique needs_.
| bogota wrote:
| I think you greatly underestimate how hard it is to find a
| good build engineer. Two of the smartest and hard working
| engineers did a bazel migration at one of my old companies
| and had to contribute heavily upstream to support all of out
| use cases.
|
| It's much cheaper in the long run to migrate to a well
| supported tool that you can extend
| thorncorona wrote:
| I believed he mentioned early on that it was due to the perf
| gains Bazel provided.
___________________________________________________________________
(page generated 2022-11-23 23:00 UTC)