[HN Gopher] Incident report: Spotify outage on March 8
___________________________________________________________________
Incident report: Spotify outage on March 8
Author : charlieegan3
Score : 50 points
Date : 2022-03-11 21:32 UTC (1 hours ago)
(HTM) web link (engineering.atspotify.com)
(TXT) w3m dump (engineering.atspotify.com)
| ipython wrote:
| Hey, for once it wasn't DNS!
| toastal wrote:
| Just a reminder that downloaded audio files work offline, can
| have very high or lossless quality not restricted by bandwidth,
| have a concept of 'ownership' with no DRM, don't have ads or
| tries to sell your data (though voluntarily contributing to
| ListenBrainz is admirable), don't require a monthly fee, and if
| bought legally gives a bigger slice of the money pie to artists.
| Also podcasts are just RSS audio.
| myself248 wrote:
| Got a favorite source for same? I'm bewildered by too many
| options and not enough comparisons.
| jmacd wrote:
| Just here to share that if you haven't tried Apple Music for its
| lossless and Dolby Atmos sound quality then you really, really,
| should.
|
| Apple Music does not have a great UI but it is definitely worth
| the trade off.
|
| Tidal sounds great, but they make you pay extra for lossless and
| have a smaller catalogue.
| alar44 wrote:
| I'd bet my next paycheck you wouldn't be able to tell the
| difference in a blind test.
|
| http://abx.digitalfeed.net
| wwweston wrote:
| Reminder that Apple pays artists better too (average penny per
| stream over Spotify's average 3-4 _tenths_ of a penny per
| stream).
|
| And that Spotify is led by people who think artists who want
| that level of compensation are "entitled":
|
| https://www.digitalmusicnews.com/2021/06/29/spotify-executiv...
| mcfedr wrote:
| Cause they can manipulate the market and don't have to give
| 30% of their income to an unchecked monopoly
| mardifoufs wrote:
| But its not like Spotify is making billions off the artists.
| They barely break even. Their margins are very very tiny in
| the first place.
|
| Obviously Apple can afford to pay a lot lot more. Now
| obviously you can say that if they can't afford it they
| shouldn't be in business, but you can say the exact same for
| the record labels/artists. They can just pull their songs
| from Spotify.
| dijit wrote:
| I think it's unfair to say they barely break even, if they
| do then it's because they've decided to spend so much.
|
| Looking externally at the company they're listed as having
| 50,000-100,000 employees. That's a lot of people to employ
| for a audio streaming service.
|
| I also happen to know their cloud commitments (I know a
| guy) and they are large enough that it starts to make a lot
| of sense to invest into physical hardware.
|
| I'm talking so much money per year that it rivals the GDP
| of some countries.
| brayhite wrote:
| I'd prefer to use AM but they're lacking an ability like
| Spotify's to play to Google Home devices and device groups. We
| use that feature daily, and I highly doubt Apple will ever
| support broadcasting to Chromecasts.
| sorenjan wrote:
| I have no interest in paying one of the large American tech
| giants for my music. They already control too much, and music
| is a side business for them.
| Chilinot wrote:
| Apple Music is just so increadibly bad UX wise. I have a real
| hard time justifying paying for it and have considered many
| times to switch back to Spotify. Trying to find new and
| interesting music is pretty much impossible outside of the
| "here's what people listen to now" playlists. You need to know
| what to look for when searching for things.
|
| Spotify was great for finding indie music bands, and could
| really give good recommendations. Apple music is not even in
| the same league.
| CharlesW wrote:
| > _Apple Music is just so increadibly bad UX wise._
|
| Is it? When I switched to Apple Music I didn't find it
| remarkably better or worse then anything else I've used
| (including Spotify). Reviews
| (https://www.pcmag.com/reviews/apple-music) don't mention any
| UX issues either. If you know of any substantive commentary
| on this I'd appreciate a link.
| TobyTheDog123 wrote:
| It is. In my personal experience:
|
| - On Mac, Apple Music will just consistently stop playing
| audio. Force quitting & restarting fixes it.
|
| - On iOS, the app is prone to crashing more than any iOS
| app I've seen. It's not horrible (I very rarely see
| crashing period), but it's far more than 3rd party apps or
| even Apple's own set.
|
| - On iOS and on a weak connection, Music will not play even
| if it is downloaded. I suspect it has something to do with
| DRM servers, but don't know for sure.
|
| - I once changed my iCloud password on my account, and it
| sent the Apple Music app into an infinite loop of trying to
| reload content, instead of just prompting for my password
| again.
|
| - Load times are horrendous for online content, even on
| fiber connections.
|
| - I don't have many complaints about discoverability, but
| it's certainly nowhere near Spotify's recommendation
| engine. Spotify's weekly-updated new music playlist is way
| more relevant to me than Apple Music's version (New Music
| Mix).
|
| But...
|
| - I think way more people can tell the difference between
| Spotify's audio quality and Apple Music's Lossless/Atmos. I
| sure could, and it was obvious.
| flats wrote:
| Apple Music's best feature for me is its real focus on building
| up your own library. On top of that, it integrates seamlessly
| with iTunes in the Cloud. Using the same app, I can stream
| anything from my iTunes library, which includes a decent amount
| of music not available on any DSP.
| josephh wrote:
| Amazon Music Unlimited also has it included in their catalog.
| How does Apple's fare against Amazon's?
| kruxigt wrote:
| ge96 wrote:
| > lossless and Dolby Atmos sound quality
|
| General headphone recommendation? Best I have is an MDR-7506
| but I don't think it's meant for listening rather producing.
| lelandfe wrote:
| Depends on price range. DT 770 PRO's would be a nice step up
| and serve both purposes pretty well. They're over-ear like
| your MDR's. You can usually find them on sale for like
| $140-160 USD
|
| Fairly neutral response (dunno if you'd prefer something
| bassier for listening) and very comfortable
| ge96 wrote:
| Yeah I don't really have any specific demands other than it
| is meant for music... I bought these ATH-MX-20/40X from
| reviews and apparently they're monitoring headphones?... so
| yeah. Those are definitely flat.
|
| Thanks I'll put those on a list. Definitely one of those
| things have to try on in person to buy but recommendations
| help too.
|
| Edit: not too bassy actually, neutral would be nice like
| works for almost anything.
| Simon_O_Rourke wrote:
| With some of the banal EDM that's out there now, some retro
| 1980s walkman headphones would do.
| mkr-hn wrote:
| I use these for both without issue.
| sitzkrieg wrote:
| random sample but i try this every couple of years - both tidal
| and apple music do not have 10 songs in today's release radar
| list available.
|
| the tidal client is nearly as bad as spotifys too
| lelandfe wrote:
| If you can consistently tell the difference between Spotify
| Premium's 320kbps and Apple Music's lossless, I am very
| impressed. Most folks reading this won't be able to.
|
| Having used both, I'm now happily paying for Spotify to avoid
| using the train wreck that is the Apple Music app. It is a true
| shame what Apple has done to iTunes.
| ksubedi wrote:
| Just depends on how you are driving it. I can tell the
| difference on Homepods, Airpod Max, my car (wired) and few
| other places, but barely on airpod pros and most other
| bluetooth headphones. And to the crowd that has their
| keyboards handy to say "but the airpod max and homepods do
| not support lossless", you are absolutely right but the
| quality of the source does make a difference, even though
| those devices will not be playing lossless, they will still
| have better quality if the source is better.
|
| I am not completely sure if its the codec Apple uses that is
| making the difference if it is because the source is
| lossless, but bass sounds punchier and instrument separation
| is much better with Apple Music compared to Spotify's best
| quality.
|
| I was really looking forward to Spotify's lossless that was
| announced (I think it's called Spotify HQ?), but I haven't
| heard about that in a long time.
|
| Edit: If you plug in your Airpod Max with a hardwire cable,
| the difference is even more noticeable, even with iPhone's
| internal DAC.
|
| Also want to clarify that the differences and not so
| pronounced that you will be able to tell right away, but if
| you listen to Apple Music for a while and try to listen to
| the same song on Spotify, it will not sound the same. It's
| one of those cases where you don't know what you don't know
| till you know it once.
| kactus wrote:
| I switched to Apple Music from Spotify just because I don't
| like podcasts mixed in with my music, and UI that constantly
| advertises their podcasts side.
|
| The improved sound quality on Apple Music is a major benefit,
| and I was able to add all my music that isn't in their library
| (and sync across devices).
|
| I also love that upcoming albums that drip feed new tracks
| aren't categorized as EPs like Spotify does it. I can view the
| track in the upcoming album on Apple Music, add the album to my
| library, and then get notifications as they add more tracks or
| release the album.
|
| Apple Music's macOS app UI is terrible, no way around that.
| They're supposedly making some parts of it native in a future
| macOS release though.
| dotnet00 wrote:
| Over the past year I've taken to just buying my music and using
| plex/plexamp for the streaming convenience. I'd have a hard time
| going back to other streaming services after getting used to the
| convenience of having control over the entire chain. Buying music
| also serves as a nice way to spend digital credit from Amazon.
|
| It ends up costing me about as much as streaming.
| wizwit999 wrote:
| So the bigger news here is GCP had a global outage. How'd that
| happen?
|
| It mentions a pipeline deployment introduced a bug
| (https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ...)
| but why weren't they doing staged deployments to region(s) at a
| time?
| politelemon wrote:
| This is now 'par for the course' with GCP.
|
| A recent example, Firefox's Jan 2022 outage was due to an
| unannounced change deployed by GCP to their load balancers.
|
| https://hacks.mozilla.org/2022/02/retrospective-and-technica...
|
| I've read several GCP related stories over the years so my
| opinion of them has fallen quite a bit. For some reason GCP
| does not get the flak they deserve, and their role in outages
| tend to be downplayed, while AWS and Azure are scrutinized and
| raked over the coals for smaller incidents.
| charcircuit wrote:
| >unannounced change deployed by GCP to their load balancers
|
| Google doesn't need to announce every change they make. It
| was Firefox's fault for not supporting the protocol
| correctly.
| acdha wrote:
| Most outages have more than one cause. It's true that
| Firefox had a nasty bug but it's also likely that they'd
| have resolved the problem faster had a clear notification
| that GCP had upgraded their infrastructure right before the
| outage started, especially if that had an easy rollback
| option.
| toomuchtodo wrote:
| They don't get flak because most folks are in AWS or Azure.
| You notice when AWS or Azure hiccups because of this. GCP,
| not so much (besides Spotify and Snap, although I believe
| Snap's 5yr $2B GCP commit deal recently ended).
|
| https://info.flexera.com/CM-REPORT-State-of-the-Cloud
|
| https://thenewstack.io/long-running-study-finds-uptake-of-
| pr...
| londons_explore wrote:
| Traffic Directors config is global. I assume it was some kind
| of config system bug.
| dastbe wrote:
| Does anyone have more context on why "unable to retrieve their
| xDS configurations." led to " Traffic Director-managed clients
| deprogrammed as the configuration was removed" ? For example,
| Envoy will keep its configuration forever unless 1) the
| configuration has a ttl or 2) the server itself sent back empty
| results. Was it the case that TD config uses ttls, the server was
| actually sending back empty results (frightening), or is this a
| behavior specific to grpc?
| hericium wrote:
| > On March 8, Google Cloud Traffic Director experienced an
| outage. This in coordination with a bug in a client (gRPC)
| library caused the Spotify outage that affected many of our
| users: if you were logged out of a Spotify app, you were unable
| to log back in.
| superjan wrote:
| But they don't explain why I was logged out. I did not log out
| myself. Or is this "logged out" in the sense that I did not
| have their app active at the time of the disruption?
___________________________________________________________________
(page generated 2022-03-11 23:00 UTC)