[HN Gopher] "A Cold War Every Day" inside Apple's internal tools...
___________________________________________________________________
"A Cold War Every Day" inside Apple's internal tools group
Author : ttepasse
Score : 129 points
Date : 2020-04-07 16:21 UTC (6 hours ago)
(HTM) web link (www.buzzfeednews.com)
(TXT) w3m dump (www.buzzfeednews.com)
| larrik wrote:
| Is this the team handling iTunesConnect and other such
| "services"? I'd definitely believe it.
| danpalmer wrote:
| From what I understand, probably not.
|
| iTunesConnect has had an overhaul in recent years to give it a
| much shinier and more reliable frontend. I believe the original
| was closer to the original WebObjects code, and now they've
| extracted out the user-facing bit into something a bit more
| modern. There's a lot of legacy WebObjects stuff around the
| iTunes backend, and my guess is that it's good engineers held
| back by old tech, rather than just bad engineers.
| captainredbeard wrote:
| It's their sister org for lack of a better term.
| saagarjha wrote:
| No, services are run by Apple Media Products under Eddy Cue.
| They're not great, either, but I think they have more full-time
| engineers than IS&T does.
| rllearneratwork wrote:
| no
| Waterluvian wrote:
| I could use some blog posts about quite functional teams.
|
| Like someone on a team for a decade talking about why it works so
| well and discusses cultural and political issues and how the team
| overcame them.
| leoc wrote:
| See my old comment
| https://news.ycombinator.com/item?id=19468090#19468699 . That
| kind of thing used to have a much higher public profile about
| 30 years ago.
| ChrisMarshallNY wrote:
| I managed a high-functioning C++ image processing group for 25
| years. I kept senior-level engineers on staff for decades.
|
| We tended to write "engine code." Like pipelines and whatnot.
|
| The company had a very ( _very_ ) long tradition of engineering
| (Like, 100 years).
|
| There were many downsides to the way they worked (over-
| structured, mostly), but they always gave my team and me a
| great deal of respect.
|
| They didn't use contractors very often.
|
| To this day, design quality, code quality, product quality,
| documentation, and focus on deliverable are the major
| cornerstones of my software work, but I am shocked at how
| little that matters to modern development shops. It's been a
| really disheartening experience.
|
| I guess I was in a silo of quality-focus, all those years. They
| would treat very minor bugs like extinction-level emergencies
| (not fun).
| captainredbeard wrote:
| IS&T shouldn't be part of Apple. It's culture and engineering
| talent aren't representative of core engineering.
| chillacy wrote:
| In the old adage that "you get what you pay for", the internal
| tools at Apple aren't quite the best as a result. The Radar
| tool I recall was pretty horrible in particular.
| Austin_Conlon wrote:
| Does IS&T also build the developer-facing Feedback Assistant?
| saagarjha wrote:
| No, there's another team for that.
| taormina wrote:
| Hey, Apple still hired them. Apple is chosing to dilute its
| core engineering brand. It's not like an end user can tell if a
| "real engineer" wrote the shitty webpage that's currently
| giving them grief.
| saagarjha wrote:
| Consumers don't interact with IS&T projects.
| 5cott0 wrote:
| Body shop wars. What gets really interesting is working for a
| body shop with consultants from 2 other shops on the same team.
| coleca wrote:
| I had a similar experience w/Apple. A startup I worked for was
| brought into Cupertino for a meeting w/their internal business
| teams. They wanted us to build them an app which seemed
| relatively simple involving their internal Cafe Mac cafeterias,
| data centers, and possibly retail locations. It was something
| that a company like Apple could build in their sleep. But the
| business folks we met with said that's how it is at Apple, all
| the engineering talent goes towards the product side and almost
| nothing is left for internal IT. They told us how they struggled
| to get anything done and there were almost no resources
| available, so the business teams had to go and hire their own IT
| if they needed things done.
| [deleted]
| ethbro wrote:
| _> But the business folks we met with said that 's how it is at
| Apple_
|
| That's how it is at most companies.
|
| Split orgs (IT & business) result in only work of sufficient
| size, scope, and impact being able to cross the barrier.
|
| Multi-year logistics management rewrite? You'll get two IT
| teams.
|
| Frank in accounting needs to schedule a daily job to copy from
| one datastore to another? He doesn't have permissions to, and
| has been doing it manually for the last 5 years.
|
| IMHO, every org along those lines would benefit from an
| independent IT tiger-team whose sole job it is to find business
| problems and apply technology to solve them.
| dtech wrote:
| And god forbid IT allows tech-savy or power-users to do
| things, that's not secure.
| saagarjha wrote:
| Apple's IT is fairly hands-off.
| jfb wrote:
| At least at Apple, in the groups I was in, this was never
| an issue.
| jldugger wrote:
| IS&T is run by the CFO and it shows.
| seemslegit wrote:
| Is there any big consumer product company out there that doesn't
| deprioritize management and engineering talent for internal tools
| and systems ?
| [deleted]
| twic wrote:
| The company i work for is not big or consumer-facing, so this
| is not an answer to your question.
|
| But we do have a dynamic that i find interesting. The company
| is organised into many small business units - some very small,
| fewer than ten people. Each is accountable for its own profit
| and loss. They hire their own programmers to build the actual
| line-of-business software they need. But there are also
| internal tools and systems, shared across units or used by
| their developers, that are built by programmers hired directly
| by the top level of the company - their reporting line goes
| straight up to the CIO and CEO, rather than to a business unit
| manager. So, in a way, our company does the exact opposite of
| deprioritizing internal tools and systems! The programmers who
| develop the internal tools and systems are the elite!
| tsomctl wrote:
| Sounds like Intuit.
| summerlight wrote:
| I expect those tech giants which offer B2B solutions to have
| better situations. e.g. Google, Amazon, Facebook, Microsoft,
| etc. Many of their internal products eventually have made the
| positions in their cloud portfolios.
| dcolkitt wrote:
| It's an interesting question. And if the answer is no, I guess
| the logical conclusion is that internal tools make very little
| difference to the success of consumer product companies.
|
| That doesn't seem like an intuitive conclusion, but it's a hard
| circle to square. As someone who mostly builds internal tools
| (though in a very different context), it certainly seems like
| good systems are an Archimedean lever that acts as a force
| multiplier across the entire org. Yet, if good internal tooling
| was important, you think at least one company would have been
| able to achieve success partially on this basis.
|
| The consumer product industry is extremely competitive, so the
| counter-hypothesis of entrenched sclerotic incumbents refusing
| to pick up $20 bills off the sidewalk doesn't really seem
| likely.
| babesh wrote:
| Long feedback chains take longer to optimize so this is
| possible. Also one feedback would be to have the company
| fail. Another possibility is that it really is a tradeoff of
| where to focus.
|
| For example, we are now paying for our lack of preparation
| and sclerotic government with the economic and lives lost to
| covid-19. It just took between 3-10 years to pay the piper.
| jedberg wrote:
| > And if the answer is no, I guess the logical conclusion is
| that internal tools make very little difference to the
| success of consumer product companies.
|
| Or perhaps no one has realized the power of good internal
| tools, and so crappy tools are the status quo and no one
| needs to do better.
|
| But as soon as one company realizes the value of internal
| tools and it becomes a competitive advantage, other companies
| in that area will need to follow suit.
| seemslegit wrote:
| Internal tools are used by employees who unlike consumers
| will tolerate more usability friction, non-critical bugs and
| feature delays than consumers and will compensate for them
| with their own effort so the tools can still be critical in
| their core functionality but not polished or well-maintained.
|
| Ironically and anecdotally, WeWork tried to build a strong
| engineering org for their internal systems before it blew up.
| MereInterest wrote:
| > And if the answer is no, I guess the logical conclusion is
| that internal tools make very little difference to the
| success of consumer product companies.
|
| That is on the implicit assumption that businesses are
| reasonable, and accurately know how to manage tradeoffs. Your
| counter-hypothesis seems rather likely to me, if we look at
| the internal behavior of a company. When negotiating, the
| more dots that you need to connect before reaching "and then
| we get more money", the weaker of a negotiating position you
| have. It's the same reason why sales teams get bonuses for
| making sales, but developers don't get bonuses for enabling
| sales.
| chillacy wrote:
| To the contrary, some of the most successful software
| companies have strong cultures of internal tools (think FAANG
| companies). How much of this is cause and effect I'm not sure
| though. To some extent it's easier to make investments in
| internal tooling if you're profitable.
| Mindwipe wrote:
| Some larger video game production companies have started to do
| a lot more on their internal tooling, as it has a huge impact
| on their content production.
|
| Most of them learned the hard way though. And some learned the
| hard way and still have awful tooling despite some efforts to
| fix them (Bungie...).
| tschwimmer wrote:
| Yes, Facebook invests heavily in developer infrastructure. On
| my former team there were many staff+ engineers who were
| extremely capable. More speculatively I'd go so far as saying
| quality of talent is actually stronger than many groups working
| on public facing products.
|
| Management of the group was also strong in my opinion.
| lima wrote:
| Does Facebook maintain an internal fork of Phabricator or is
| it relatively close to mainline? It's such a great product
| but parts of it are quite clunky - I sometimes wonder if
| there's some internal secret sauce to smooth these out.
| jauer wrote:
| Phabricator was/started to be replaced a year or two ago.
| AFAIK it's been largely rewritten at this point.
| seemslegit wrote:
| Developer infrastructure does not seem to be the right area
| to compare with here just like Google's internal engineering
| products aren't, more like - how polished are the tools used
| by the ad sales people, analysts, moderators etc.
| diebeforei485 wrote:
| Apple IS&T isn't development infrastructure - while they do
| have some generic services, most groups at Apple develop and
| use their own tooling.
| saagarjha wrote:
| Or they use tools made by Developer Tools, many of which
| are used by external developers as well and ship as part of
| Xcode.
| voz_ wrote:
| + 1 to Facebook.
| jedberg wrote:
| Netflix. At least when I was there.
|
| Internal tools got just as much support as product. Engineers
| moved between working on both, and the hiring process was the
| same for both.
|
| I did an interview recently about this on the retool blog [0]
|
| [0] https://retool.com/blog/how-to-build-great-internal-tools-
| je...
| K0SM0S wrote:
| Nice interview, great mindset.
|
| Netflix is indeed becoming quite the sensation for its
| internal policies-- the general intelligence apparently
| prevailing there. Seeing tools as first-grade citizens seems
| like a very sane approach to me-- empower each other and
| watch people create marvels.
|
| Papermill1 notably is one incredibly seducing tool IMHO.
|
| 1: 2019 talk by Matthew Seal, ~40 min
| https://www.youtube.com/watch?v=3FmBJ847_y8
| AceJohnny2 wrote:
| Google?
| seemslegit wrote:
| What do we know about the quality of the tools used by say
| the chromebook sales analysts ? This isn't spanner or
| bigtable.
| danpalmer wrote:
| I've heard Google internal tools are very hit and miss. The
| engineering stuff is top-notch, but things like their
| applicant tracking system is archaic (or was a few years
| ago).
| summerlight wrote:
| Google's internal corp tools designed for majority of its
| employees are pretty nice. Tools for Chromebook sales
| analysts might not be as great though, but it also doesn't
| make sense to put tons of investments into a tool used by
| just tens of sales people. (Of course, there's a number of
| general marketing tools for thousands of sales)
| danans wrote:
| Googler here, but opinion is my own. Google's problem is that
| we have too many powerful internal engineering tools, many of
| which overlap in functionality, and so often its hard to know
| at first which one to use to solve a problem.
|
| Thankfully, there is nearly always an overlap period when one
| solution gets deprecated and the replacement is launched.
|
| The COVID19 office shutdown has really focused attention on
| our internal IT tools (most of which are also offered
| externally). So far, they've worked pretty great, in no small
| part because things like BeyondCorp [1] were designed from
| the start for distributed workforces in zero-trust security
| environments.
|
| It wasn't always like this though. Corp-Eng, as it's called
| at Google was for a long time seen as a dead end career-wise
| - disconnected from Google's marquee products. However, a
| number of developments, including the rise of the enterprise
| and cloud computing businesses, have bolstered its internal
| importance, and with that have brought excitement and talent
| to the organization. It's now an important source of product
| feature ideas that eventually make their way to customers.
|
| That said, a lot of things not core to Google's problem
| spaces do get solved with outside vendors' software, and some
| of that software is great. For example, I just used a site
| licensed version of the Chrome ScreenCastify [2] extension to
| make a video demonstrating a feature I'm developing. It
| worked perfectly.
|
| 1. https://cloud.google.com/beyondcorp
|
| 2. https://www.screencastify.com/
| danzig13 wrote:
| Sounds exactly like government contracting.
| rkho wrote:
| A while ago I interviewed for a senior fullstack role with a very
| homogenous team at Apple -- every single person on the team had
| been converted from the same consulting company (I believe it was
| Wipro).
|
| It was a bizarre interview with three of the engineers on the
| team on the same call asking me very specific programming
| language trivia and nothing about my experience nor ability to
| navigate complex political structures -- you know, the things
| that you would expect a senior fullstack engineer to have before
| advancing them to the next round.
|
| Unsurprisingly, I wasn't advanced to the next round. In light of
| this article, I can't help but wonder if it was all a ruse to
| simply demand another nepotistic FTE conversion from their same
| consultancy.
| timwaagh wrote:
| Perhaps, but perhaps the role just wasn't fit for you. A lot of
| 'senior dev' roles are much more about programming language
| trivia and much less about leadership.
| rkho wrote:
| That could be the case, I just find it difficult to believe
| that an ICT3 role at Apple of all companies would be about
| programming language trivia, given that the specific team I
| spoke with mentioned how they were very cross-collaborative
| with other LOBs.
| filoleg wrote:
| >I just find it difficult to believe that an ICT3 role at
| Apple of all companies would be about programming language
| trivia
|
| Fully agreed, but I don't think it is like that across
| Apple at all. It was like that just for that specific team
| OP was interviewing with, and it makes sense in the context
| of that whole team being Wipro converts. This is definitely
| unusual to have a team like that in the first place, so OP
| just simply got unlucky.
|
| Given a company of Apple size, no matter how pleasant and
| competent an average team is, there will always be a few
| outlier teams that are just whack and awful to work with.
|
| I can confidently say the same about my workplace. I like
| my team and org a ton (in terms of competency, how the
| responsibilities are assigned, etc.), and I believe that an
| average team in the company is very competent and high
| quality. However, I would be lying if I said that I haven't
| encountered at least a couple of teams in other orgs at the
| company that I would never want to work on due to similar
| issues.
| voz_ wrote:
| > A lot of 'senior dev' roles are much more about programming
| language trivia and much less about leadership.
|
| I am curious - where do you work?
|
| Asking so that I can potentially avoid that company, as this
| is the most inane and borderline backwards thing I've ever
| seen written about senior engineering.
| riskneutral wrote:
| Sounds like the politics of every large corporate IT department,
| but with some twists given that it's Apple. Maybe this helps
| explain why Apple is not Amazon.
| ping_pong wrote:
| One of the world's most successful and wealthy companies in
| history shouldn't be bottom fishing for the best "deal" on
| employees. I wonder how much better they could be if they decided
| to hire at the top of the market like Netflix instead of the
| bottom.
|
| That is probably why Apple software is among the worst in the
| industry, and it's a shame. I know IS&T isn't strictly software
| like iTunes but it fits the pattern I've seen at Apple. But I
| guess if it doesn't affect their sales maybe I'm wrong and it
| doesn't matter and they are right.
| saagarjha wrote:
| The quality of Apple's software engineering organization is
| head and shoulders above IS&T's; iTunes (...and Apple Music,
| News, etc.) is run under a different (services) group.
| throwaway321213 wrote:
| Why are you afraid to mention the word "Indian"?
| motohagiography wrote:
| Sounds like a bank, a government agency, or any business over a
| certain size.
|
| I tell people you only call it politics when you are losing. More
| accurately, it's a layer of literal stupidity above the competent
| to shield the money side of the company from the leverage that
| operations people would have if they had any information about
| how the money side worked.
|
| Instead of a hierarchy, rethink a company as a hub and spoke
| model with concentric rings. The main differences are the
| implication in a hierarchy that there is "gravity," keeping
| people down and that they need energy and leverage to climb "up,"
| which further implies there is a place to "fall," and that there
| is only one way "up," instead of many possible paths to the
| centre from all directions. There is no gravity, only gates and
| barriers, and even these are just information. Politics is how a
| middle manager runs interference and creates distractions to make
| sure you can't see over, around, or through them, and that the
| people behind them closer to the money can't see you. Tech is
| usually outside the main perimeter, mediated by contracting
| companies or middle managers whose job is to compartmentalize the
| value people create, and be sure it is replaceable.
|
| Viewed this way, of course this demented political farce is how
| Apple works, because it's how everything seems to work when you
| have internalized the precise and specific mental model someone
| uses to take advantage of you.
|
| Sorry if you can't unsee it now, but hopefully it will be funny
| and we can get good, competent people who value tangible skills
| into positions of power.
| seph-reed wrote:
| This is just your own personal model? It's beautiful!
|
| Is there a community where we could discuss something like
| this? "The spoke." I want to get obsessed with this.
| IST-Throwaway wrote:
| Fascinating to see an article about IS&T, definitely not
| something I ever expected to hear about on HN!
|
| I was part of a team inside Apple that developed internal-facing
| tools for the retail teams. Our team was formed because of the
| extremely high cost in dollars and time that IS&T wanted to
| charge to develop some fairly straightforward tools. My team was
| taken from the departments that used the tools, and we developed
| things that were very custom-fit to what those departments
| needed.
|
| Eventually politics overcame us, and IS&T finally managed to take
| over our team. We all became contractors in order to support the
| migration to their infrastructure, and continue development of
| the tools. A bit later we all became fired as our project was
| outsourced to contractors in India, managed by IS&T PMs who had
| no idea what our tool was used for and had never been inside a
| retail store. The tools all died shortly after that, some killed
| by IS&T, some petered out due to lack of use now that they no
| longer worked correctly or were a good fit for the users.
|
| As far as we could tell, IS&T was run as a unique company inside
| Apple, which did it's fair share of price gouging in order to
| make itself money to keep going. Its purpose never appeared to be
| helping Apple customers or Apple employees, it was simply to get
| bigger, absorbing more money and power wherever possible, with no
| apparent reigning in from the parent org.
|
| Although my experience is several years old, everything in this
| article rings true. The contracting companies they had us working
| for were taking a huge cut, the quality of the code they produced
| was dismal, (as soon as we were no longer allowed to re-write
| their code major things began breaking almost immediately) and
| people getting transferred around constantly and having no time
| to understand any one project was common. (rkho's comment about
| their hiring process seeming like it was simply a beard for a
| nepotistic contractor conversion was something we definitely saw
| a number of times.)
|
| All in all it was an extremely eye-opening experience.
| Considering how "do it the Apple way" every other department we
| interacted with was, being in the IS&T buildings was like landing
| on an alien planet.
___________________________________________________________________
(page generated 2020-04-07 23:00 UTC)