[HN Gopher] Facebook Down? ___________________________________________________________________ Facebook Down? Author : gamerDude Score : 249 points Date : 2021-04-08 21:28 UTC (1 hours ago) (HTM) web link (www.facebook.com) (TXT) w3m dump (www.facebook.com) | grensley wrote: | How on earth does EVERYTHING go down like this? It just seems | unfathomable to me. Where is the single point of failure? | notretarded wrote: | Akamai | chaos_emergent wrote: | Probably an intern doing something dumb. | mateo411 wrote: | The dumb thing would be giving the intern permission to bring | down the whole site. | vesinisa wrote: | If an intern can bring down the whole presence of your | company online it's not the intern who is dumb. | abhishekbasu wrote: | some startups would like to disagree | dividedbyzero wrote: | While that's perfectly true, I'm sure a smart and | determined intern could cause an astonishing amount of | damage in many many orgs. Hopefully not at Facebook though. | sleepydog wrote: | The difference is whether or not it was intentional. If | someone can bring everything down accidentally, the | system is broken. | | Ditto if they can bring it down intentionally and cover | their tracks so no one finds out who did it. | jimmy2020 wrote: | bingo | MattGaiser wrote: | An intern shouldn't have those permissions. | jscheel wrote: | DNS (not necessarily this case, but man it can bring things | down fast) | cure wrote: | Probably DNS. It's always DNS ;) | jonny_eh wrote: | or BGP, or auth. | rogerdickey wrote: | How does it stay up as much as it does :) | korethr wrote: | "And if these people stop, the world burns. Most people don't | even know what sysadmins do, but trust me, if they all took a | lunch break at the same time they wouldn't make it to the | deli before you ran out of bullets protecting your canned | goods from roving bands of mutants."[1] | | 1. https://www.stilldrinking.org/programming-sucks | fishnchips wrote: | Blood, sweat, tears and caffeine. | kazen44 wrote: | also, some arcane shell script written by a retired | engineer somewhere in the early start of the company. | lovedswain wrote: | The global outage drinking game: | | bad DNS config push - 1 shot | | routing loop - 2 shots | | third party advertising your routes - 3 shots | | power outage at data centre it turns out everything depends on | despite decades and millions in engineering to avoid precisely | that - 4 shots | | Wolves ate through fiber - 5 shots | | And it was a full moon - 6 shots | | Single service failure, but service has not been restarted in 5 | years, and no longer restarts in any documented fashion - 7 | shots | | And service developers left the company to found a startup - 8 | shots | | Expired internal SSL certificate - 9 shots | | Daylight savings changeover-induced database corruption - 10 | shots | | Windows Update - 11 shots | ASalazarMX wrote: | > Wolves ate through fiber - 5 shots | | I hope it's wolves, that would create a new category in | https://cybersquirrel1.com/ | erik_seaberg wrote: | 5 is often broken down to "rain fade" and "backhoe fade." | | My favorite can't-restart story is "the shared password | manager is down, we need a hardware crypto key to restart it, | the key is in the safe, and the safe's combination was in the | shared password manager." | lovedswain wrote: | In a similar vein, the alternative option for 7 was a | cyclical dependency introduced since last restart | erik_seaberg wrote: | I heard a story that Google SRE had to up a sacrificial | Bigtable in any new zone before they could up enough | services to finally up a real Bigtable for the zone. | reaperducer wrote: | Coincidentally, I'm in the same predicament with Facebook, | itself. | | I can't log in to Facebook because Facebook wants approval | from the Duo Mobile 2FA app. The 2FA app won't approve the | login because it wants a code that can only be obtained by | logging in to Facebook. | | Solution: Deleted both Duo Mobile and Facebook apps. Moved | on with my life. | dylan604 wrote: | isn't that like throwin the baby out with the bath water? | i've never had problems with duo mobile, but then i don't | use FB so i've never seen how it behaves with FB. so dump | FB, but why dump duo mobile? | spondyl wrote: | I was going to link a related comment but you're actually | the author of it! | | Here it is anyway: | https://news.ycombinator.com/item?id=22316230 | neura wrote: | Please, please tell me this is a true story. It's amazing. | alfalfasprout wrote: | Don't forget cosmic rays! | partcyborg wrote: | You forgot: | | Hunter shot your fiber relay with a hunting rifle - 6 shots | michaelt wrote: | No, he only shot it once. | neura wrote: | depends on the gun... some people have interesting ideas | on hunting. | nabla9 wrote: | Apple decided to shut down Facebook. | godelmachine wrote: | DST can be such a hassle. It affects our reports and change | requests. | erik_seaberg wrote: | An adtech company once freaked out over a sudden 4% revenue | drop, until someone pointed out that yesterday was 4% | shorter than the day before. | nonameiguess wrote: | Large scale network partitioning can happen. Data center | ingress/egress can go down. If you shard to multiple data | centers, it's still possible for an entire autonomous system to | have an interconnect with a different AS break. That is rare as | hell with the actual Internet because of all the route | redundancy even between ASes, but I've seen it happen on more | specialized networks. For a few days last year, the entire | western half of the US was cutoff from the eastern half on | Raytheon's internal TS network because someone digging a hole | somewhere accidentally cut the wrong fiber line. | tyingq wrote: | I'll be curious about it coming back up. It's the sort of site | where there's an army of people continually trying to reload | the page if something isn't working. | wqsz7xn wrote: | Weird that people almost celebrating in this thread? Facebook is | a tool like any other and is only 'bad' if you allow that to | happen. | rossdavidh wrote: | That's like saying that any Turing complete language is | equivalent. Some languages encourage you to do things well, | others do not. It was totally possible to write code with a lot | of "GOTO" statements in it that was well structured, but the | actual real-world results with it were bad. It may be possible | to use FB in a constructive way, but that doesn't mean we are | better off having it. | ben509 wrote: | Social media is a square wheel. Sure, it sort of goes and if | you know no alternative, it might be better than no wheel at | all. But there's no good way to use it because it's | fundamentally flawed. | fooperdupes wrote: | Love to see it. I hate this company. | simonklitj wrote: | In Denmark right now, everything down. Was on the phone with | someone (in the US) through Messenger without issues. Hung up and | tried to send them a message and nothing. Instagram non- | functional as well. | always_left wrote: | Any bets on what happened? Fat Fingers? | tored wrote: | Snapd autoupdate. | dividedbyzero wrote: | Containment breach in the AI department, they are currently | bombing the site from space, the outages are collateral damage. | brunojppb wrote: | Down in Austria as well. | [deleted] | tamaharbor wrote: | Perhaps society can rebuild if this continues. | challengly wrote: | If Twitter follows, there might be a chance. | cjohnson318 wrote: | Nature is healing. | nja wrote: | Anyone else get a login request email from Instagram before it | went down? | | I got one, immediately logged in separately and changed my | password, logged back in, then shortly after everything went | down... | urnesst wrote: | oh no..... YOU BROKE IT!!! D: | type0 wrote: | I wish wholeheartedly that it stays down for a long-long time, | long enough to loose it's relevance. | amelius wrote: | Or long enough to have an experiment which proves the negative | externalities of the platform. | danielam wrote: | https://www.theverge.com/2021/4/8/22374499/facebook-instagra... | labster wrote: | And nothing of value was lost | peterburkimsher wrote: | mbasic.facebook.com was also down in New Zealand. Thought it was | a company firewall issue at first, but checked | https://downforeveryoneorjustme.com/ and confirmed that it's | happening to others. | jimmy2020 wrote: | Why this outage doesn't happen to twitter | [deleted] | fsflover wrote: | Dupe: https://news.ycombinator.com/item?id=26743971 | rc202402 wrote: | I'm guessing this has do something with the recent graph api | scraping? | | Facebook is kinda rude now, they probably adding api bot | detection and more phone number verification. | rgaut wrote: | good for the world :) | chis wrote: | real talk can any facebook employees remind me how to get to the | IRC chats | fishnchips wrote: | Quickly, let your FB engineering friends know. If their status | page is down, their monitoring may be down as well and they're | none the wiser. | glitchcrab wrote: | Nah, the longer it is down, the better. | hnarn wrote: | > If their status page is down, their monitoring may be down as | well | | That is not how monitoring should work. But this is also not | how status pages should work. | arcticbull wrote: | I bet FB Workplace went down too heh [1] | | [1] https://workplace.com | redcalx wrote: | How long before the riots and total societal breakdown? | yumraj wrote: | This has to be one of those rare HN instances where a _popular_ | site has gone down and people are actually happy.. | | I can't think of another such instance.. | swyx wrote: | popular? you must be new here... | yumraj wrote: | I had put _popular_ in italics. ;) | | It's not popular here on HN, but in the rest of the world it | surely is. | rossdavidh wrote: | Popular in the same sense that tobacco is popular. | transreal wrote: | The website is still down for me, I'm in San Francisco. | mullingitover wrote: | We're obviously going to take this opportunity to dunk of | facebook because it's funny, but let's not forget that there are | some valuable cultural assets that facebook hosts, with groups | such as "Foods with threatening auras" and "Animals in | predicaments." | | I long ago tamed the facebook engagement algorithm by simply | unfollowing everything and everyone except these legitimate, | useful types of entertainment venues. | ben509 wrote: | Don't forget "The Same Photo of a Gravel Pile Every Day." | Almost makes the decline of Western civilization worth it. | grishka wrote: | Is there a group that saves and reuploads the same photo | every day, compressing it into oblivion? | [deleted] | behindai wrote: | Berlin. Facebook is down. | [deleted] | ewenjo wrote: | It's back! | lostgame wrote: | Down again. | rainhacker wrote: | It's working for me now. | blhack wrote: | Looks like facebook auth is still working? | PM_MOI wrote: | I once accidentally knocked out 65K websites during a "fix | update". Back then I was mortified. Now, I just feel like that | was amateur hour... | nippur72 wrote: | down it Italy as well | behindai wrote: | Hope it's down forever! | godelmachine wrote: | Anyone knows what might have caused the outage? Someone working | in data centers might have a clue. | bayesianbot wrote: | Everything up and working again (in Thailand at least), that was | fast | tomc1985 wrote: | And of course Instagram tells me, "No Internet Connection" | | Like you couldn't connect to your server dude, don't pass the | blame on! | rvz wrote: | Oh dear. Mark Zuckerbergs website, photo gallery and his SMS | messenger are down. | | Time to use something that doesn't scan your photo book, address | book, and ultimately; everything about YOU. | chris_wot wrote: | Yup, getting oauth errors. | aerosmile wrote: | Likely not related, but you can imagine that they are quite busy | these days with two unplanned projects: | | 1. Retooling the underpinnings of their ad analytics to survive | iOS14. Big advertisers are about to lose a lot of their insights | into attribution, partially directly related to the new | limitations of iOS14, and partially due to assumptions that | Facebook had made about tracking and which are no longer accurate | and require a big rewrite. | | 2. Building an internal Shopify to allow customers to click on | ads and complete transactions without ever leaving their app. | This already exists but is practically unusable and nowhere close | to the polish of Shopify. | InitialLastName wrote: | 3) Frantically planning Zuck's next apology tour after he | leaked personal data on half a billion people. | oneplane wrote: | This is an Orville reference isn't it. (or at least, it could | be and would be extremely related) | peterburkimsher wrote: | Perhaps some security patches after the data leak? | https://news.ycombinator.com/item?id=26691868 | menocchio2021 wrote: | Fun fact! Before FB got down, I've seen a disscusion on a group | where ppl complaining about the social networks - same usual bs, | but still keep using them! | | I am curious how they will feel now! | newbie578 wrote: | I don't understand how can you allow this to happen? I get one | platform being down, but for all of your platforms to be down is | not a good sign. | | I only realized now that I only used Facebook platforms to | communicate with my friends, and guess what I cannot reach them | by Messenger, Instagram or WhatsApp. | | So I am downloading Signal... I mean come on, you are literally | loosing users because of this... | Rallen89 wrote: | Could this have anything to do with the recent data breach? | towergratis wrote: | Now stay down. | pelim wrote: | germany/berlin also outage of all fb services using http protocol | [deleted] | pelim wrote: | and its back | menocchio2021 wrote: | Funny! | | before FB got down, I was seeing a disscuion on how "bad" fb (and | other social networks are) the same usual BS where people | complaining but still keep using it! | | I am curious how they will react now?! How all those "drama | queens" will behave now when all their "reputation as moderators" | is at risk of dissapereance! | rpastuszak wrote: | Quickly, let's get shit done! | rasso wrote: | Back up on Tenerife | abujazar wrote: | Yea | yhoneycomb wrote: | Also instagram, but not whatsapp | threatripper wrote: | Whatsapp (the website) now, too. Whatsapp (the messenger) is | still transmitting messages. | kickling wrote: | Love the status page right now. | https://developers.facebook.com/status/dashboard/ | | PS. Instagram app/site, whatsapp.com, messenger.com, oculus.com | also down, anything else? | | UPDATE: Everything back up! (Even the status page) | alexcnwy wrote: | Even the status page is down | cure wrote: | Seriously, they host the status page on their own | infrastructure? That's ... not smart. | kickling wrote: | Seems like Giphy is up at least, also facebook owned. | gh-throw wrote: | Every company's actual status page (and news feed) is on | Twitter, no matter what else they say. I don't know where | Twitter hosts their real status page. Probably Facebook. | | Notably, Facebook hasn't updated theirs, though. | btown wrote: | > every company's actual status page is on Twitter | | Clearly you have not experienced the absolute joy of a | mission-critical locked-in-vendor B2B SaaS with no status | page, no active Twitter presence, and the effective status | page being a chat widget that routes to a person overseas | who says "oh yeah this is a known outage on our main | product please stay tuned" but there's no attempt to | proactively make it visible to clients that they know there | is an outage. | | EDIT: fun fact, said vendor lets us use a subdomain we own, | so we just route them through a Cloudflare Worker that | injects Sentry into their HTML, so that we can monitor | errors ourselves and raise tickets with them pretending | that we know less than we actually know, because somehow we | have a better observability culture than a SaaS vendor | that's been around for 20 years. Don't underestimate the | difference between vendor.mybrand.com and | mybrand.vendor.com on a feature matrix, it may save your | sanity. | spondyl wrote: | During the recent Microsoft Azure AD outage, I attempted | to use the chat widget. Selecting "Technical support" or | "Billing Support" both seemed to return 500 status codes | while the "Sales Support" routed somewhere else and | connected me to a person, perfectly fine. Presumably the | sales team don't use Azure for anything. | roddylindsay wrote: | Circa 2008-2009 when I was at FB I wrote a dashboard widget | for the ops team that scraped Twitter for mentions of | phrases like "Facebook down" in the past 5m. It was in use | for a while. | setpatchaddress wrote: | for posterity: <?xml version="1.0"?> | <!DOCTYPE html> <html lang="en" id="facebook"> | <head> <title>Error</title> <meta | charset="utf-8"/> <meta http-equiv="Cache-Control" | content="no-cache"/> <meta name="robots" | content="noindex,nofollow"/> <style> | html, body { color: #333; font-family: 'Lucida Grande', | 'Tahoma', 'Verdana', 'Arial', sans-serif; margin: 0; padding: | 0; text-align: center;} #header { height: 30px; | padding-bottom: 10px; padding-top: 10px; text-align: center;} | #icon { width: 30px;} .core { margin: auto; | padding: 1em 0; text-align: left; width: 904px;} h1 | { font-size: 18px;} p { font-size: 13px;} | .footer { border-top: 1px solid #ddd; color: #777; float: left; | font-size: 11px; padding: 5px 8px 6px 0; width: 904px;} | </style> </head> <body> <div | id="header"> <a href="//www.facebook.com/"> | <img id="icon" | src="//static.facebook.com/images/logos/facebook_2x.png"/> | </a> </div> <div class="core"> | <h1>Sorry, something went wrong.</h1> <p>We're | working on getting this fixed as soon as we can.</p> | <p> <a id="back" href="//www.facebook.com/">Go | Back</a> </p> <div class="footer"> | Facebook © 2021 · <a | href="//www.facebook.com/help/">Help</a></div> </div> | <script> document.getElementById("back").onclick = | function() { if (history.length > 1) { | history.back(); return false; } | }; </script> </body> </html> | | And the png does not load. | unpythonic wrote: | And here are the headers: # curl -i | https://www.facebook.com/ HTTP/1.1 500 Internal | Server Error X-Frame-Options: DENY X-XSS- | Protection: 0 X-Content-Type-Options: nosniff | Strict-Transport-Security: max-age=15552000; preload | Set-Cookie: ... Expires: Sat, 01 Jan 2000 00:00:00 | GMT Cache-Control: private, no-cache, no-store, must- | revalidate Vary: Accept-Encoding Pragma: no- | cache x-fb-rlafr: 0 Content-Type: text/html; | charset="utf-8" X-FB-Debug: ... Date: Thu, 08 | Apr 2021 21:41:49 GMT Priority: u=3,i | Transfer-Encoding: chunked Alt-Svc: h3-29=":443"; | ma=3600,h3-27=":443"; ma=3600 Connection: keep-alive | <!DOCTYPE html><html lang="en" | id="facebook"><head><title>Error</title>... | godelmachine wrote: | >> _Expires: Sat, 01 Jan 2000 00:00:00 GMT_ | | The year 2000 doesn't sound right. | ghughes wrote: | This is common, it essentially means "don't cache this". | godelmachine wrote: | Ok, I thought this was related to SSL certs and maybe | wrong dates might have caused FB to go haywire. | grensley wrote: | I like that it doesn't know that anything went wrong. | css wrote: | It loads fine for me and shows no downtime or incidents. It | even says "Facebook Platform is Healthy" | kickling wrote: | Yup, everything just came back online. | css wrote: | Now I am unable to connect at all: ~# | wget https://developers.facebook.com/status/dashboard/ | Connecting to developers.facebook.com (157.240.206.16:443) | wget: can't connect to remote host (157.240.206.16): | Operation timed out | chris_wot wrote: | Try posting | reaperducer wrote: | Not here. Status page still does not load at all. | liquidify wrote: | same, and can't visit the site at all | threatripper wrote: | Do they share infrastructure? | lavp wrote: | Given the fact that it's also down, I'd say yes | threatripper wrote: | It could be some rouge admin who actually deleted Facebook. | hk1337 wrote: | Like a moulin rouge admin? | drewzero1 wrote: | Time for them to hit the gym and call a lawyer! | viraptor wrote: | At that size of a company there's usually blast radius | restrictions and per-role permissions. I don't expect | anyone has enough rights to "delete Facebook" on their | own. | londons_explore wrote: | I guarantee you that there are 100+ people who could take | Facebook down for 24+ hours if they went rogue. | | For example the people responsible for the bootup scripts | of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf | ${TEMPDIR}/*" into crontab... They'd set the commit | message as "clear out temp monthly" and it would get | deployed across the entire fleet till in the first of | next month every disk at Facebook gets erased because | TEMPDIR isn't defined... | | I guess they have enough pending stock to deter them... | tracyhenry wrote: | How though? Every past author of that script would be | notified of such a change. It'd be insane if all of them | would pretend they didn't see it and accept that change. | londons_explore wrote: | It would be hidden amongst a big refactor, and it would | have a bunch of unit and integration tests that all work | correctly (because they do set the environment variables | correctly). | | Writing code that appears to do one thing and actually | does another is very doable - you only need to hide one | malicious line amongst thousands in a code review, while | the reviewer needs to inspect every line. | | Besides, the vast majority of code reviews are 10 minutes | or less. | austinheap wrote: | That's...now how any of this works. You can't just change | integrity-bearing things without FIM systems kicking in. | And you'd need collusion to get something mainlined that | would bypass that. | londons_explore wrote: | You don't need collusion - just a code reviewer not | paying proper attention. | ta20210405 wrote: | Facebook servers do not have cron installed. | londons_explore wrote: | Example using widely understood tooling... But there are | plenty of other one line changes with similar impact | type0 wrote: | one could hope | fastball wrote: | Yeah, that's what I was wondering. Strange that there is a | SPOF for Facebook/Insta/Whatsapp. | threatripper wrote: | I had the impression that they are still run like mostly | independent companies who only share some data on the | backend. | fastball wrote: | Yeah that's what I was thinking too, so this is probably | DNS related. | godelmachine wrote: | Might I ask what's SPOF? | fastball wrote: | Single Point of Failure. | godelmachine wrote: | Thanks | octo_t wrote: | Single Point Of Failure | godelmachine wrote: | Ta | DavidPeiffer wrote: | Anecdotally, whatsapp worked for me (message was sent and | received successfully) though Facebook gave an error. I was | already logged into both services. | eroded wrote: | whatsapp.com as in the web-based client, which is down. | aarongolliver wrote: | Finally a status page that updates the moment an issue starts | purec wrote: | Back up | sgt wrote: | Same here, shows: "Sorry, something went wrong. | | We're working on getting this fixed as soon as we can." | rmason wrote: | I'm seeing the same message | sgt wrote: | I hope they get FB back up and running within the end of the | year, otherwise I may actually start missing it. | rc202402 wrote: | I like how you're nostalgic, yet hate fb at the same time. | afandian wrote: | Nostalgia is rarely uncomplicated | jjgreen wrote: | It used to be | dylan604 wrote: | most relationships that come to a bitter end are like this. | you want to remember the good times, but the reasons you are | breaking up always come front of mind. | pope_meat wrote: | I walked away from fb a year and half ago, and I don't miss any | of it. | | Hope it stays down, it would be an overall net gain for | humanity. | WesolyKubeczek wrote: | I thought they banned me. Retried in an incognito window. | | Time to enjoy a brief social media pause, thanks to the deity who | might have been behind this! | lucasmullens wrote: | Downdetector is a bit more useful than facebook.com for this: | https://downdetector.com/status/facebook/ | mikaelaast wrote: | To be a fly on the wall at Facebook HQ right now... | vdfs wrote: | It's empty anyway | lostgame wrote: | Whoa. Wonder what on earth is going on. Can confirm here in | Canada, FB and Messenger are both down. | dboreham wrote: | There's a page saying something is wrong, which could just be a | very efficient summary feature for the site, that is working as | intended. | iandanforth wrote: | Required clip: | | https://youtu.be/wZ6ZlRnS40s?t=12 | menocchio2021 wrote: | Gold! | abujazar wrote: | Perhaps they had to patch some critical vulnerability like the | one that exposed 533M personal records and did the right thing by | pressing the red button(tm) | Chopsah2 wrote: | Someone moved fast. | seansmccullough wrote: | And broke everything? | dylan604 wrote: | Check the A/C thermostats for hidden RPi devices | booleandilemma wrote: | I can't tell if it's down or not because I block it at the | router. | purec wrote: | Facebook, Messenger, Instagram and Whatsapp sites all down | posharma wrote: | Whatsapp is also down! | jeffbee wrote: | Instagram was as well. Seems to have been global across their | properties. | godelmachine wrote: | I hope they publish a postmortem. | dividedbyzero wrote: | Do they usually? | godelmachine wrote: | They don't but they should. | | Postmortems are exciting reads and they should have a section | for postmortems in their engineering blog. | rk1987 wrote: | Down in Australia as well | m00dy wrote: | Black hawk down... hmm not this one. Blue bird down.. not again.. | ohh yea facebook is down :) | atourgates wrote: | Of course the first time in months I've used any of Facebook API | services, and they go down mid-implementation. | dekerta wrote: | Maybe you broke Facebook | rossdavidh wrote: | Sounds like positive correlation to me. | thrusong wrote: | I'm getting two different error pages with the same "Sorry, | something went wrong. We're working on getting this fixed as soon | as we can" message. | | One has the older site look, copyright 2020, the other has the | newer FB5 look, copyright 2021. | | The logo is not loading on either page. | | Part of me wonders if they discovered a big/new hole that led to | the data leak, although Instagram being down too suggests maybe | there's something happening behind the load balancers? | cdgore wrote: | I'm seeing the same thing in NYC. | thrusong wrote: | Winnipeg, Manitoba, Canada here. | chris_wot wrote: | Australia, Sydney. It's worldwide. ___________________________________________________________________ (page generated 2021-04-08 23:00 UTC)