[HN Gopher] Cloudflare was down
___________________________________________________________________
Cloudflare was down
Author : dewey
Score : 766 points
Date : 2020-07-17 21:18 UTC (1 hours ago)
(HTM) web link (www.cloudflarestatus.com)
(TXT) w3m dump (www.cloudflarestatus.com)
| MH15 wrote:
| Back online just now for me in Midwestern US.
| r1ch wrote:
| Interestingly this seemed to only affect resolver service. I use
| Cloudflare pretty extensively on all my sites, but only in DNS
| mode (no CDN / proxy). The hosts continued to resolve fine during
| the outage (following root DNS resolution chain, no recursive
| resolver involved). I imagine their CDN internally uses their
| resolver service which explains the outages, and some unrelated
| 3rd parties who don't use CF on their domain at all still created
| a hard dependency on CF by using their recursive DNS server.
| megadethz wrote:
| hn algolia search broken
| redox_ wrote:
| yes hn.algolia.com is powered by Cloudflare /o\
| andrewnicolalde wrote:
| Was wondering why my DNS wasn't working...
| [deleted]
| randomstring wrote:
| What luck, I chose today to install a new piece of network gear.
| I thought I had managed to totally FUBAR my network. DNS was
| failing, "ping 1.1" (my current goto test "Am I connected to the
| internet?" as it requires the fewest keystrokes and hits the
| Cloudflare DNS 1.0.0.1) failed and I just assumed it was _my_
| fault. Backed out my changes, and discover in fact, the internet
| was down.
| BenjiWiebe wrote:
| Ping 1.1... thanks for that!
| bartwe wrote:
| DNS is also (partially) down with my ISP (xs4all.nl) it seems
| synack wrote:
| I can't change my NS records to point to a different DNS provider
| because my registrar, Namecheap, also uses Cloudflare. Didn't
| expect that.
| Aldqueath wrote:
| this is great, i already have bad enough internet (rural area
| with 3 to 6 digits latency and average 4 digits, barely a few
| kilobytes of speed) and having both google smearing everywhere
| their recaptchas that are not really friendly toward low speed
| internet / non chrome users and cloudflare proxying half the
| internet but lately not really doing a great job at keeping a
| consistent uptime does not help much
|
| at least i am glad hn exists, it is the only thing that loads
| everywhere
| RL_Quine wrote:
| Some POPs are fine.
| bhaak wrote:
| the internet was built to withstand a nuclear war brought down by
| cloud flares
| [deleted]
| alfg wrote:
| Yep, Cloudflare, DigitalOcean and 1.1.1.1 down for me. I thought
| it was my internet and was so confused for a bit there.
| pgrote wrote:
| Did anyone else see their ATT internet go down? The DNS issues
| started and then the Pace 5268AC rebooted. I don't use cloudflare
| for dns. Does ATT's backend?
| MertsA wrote:
| On the contrary, ATT actually squats on the CloudFlare DNS IP
| address. IIRC that modem is one of the affected ones where it
| uses 1.1.1.0/24 internally. You shouldn't even be able to use
| CloudFlare DNS normally.
| drchiu wrote:
| Just got a whole bunch of alerts that my services are down. Tried
| logging into Digital Ocean (who it seems uses Cloudflare) to get
| it fixed. Could not access their dashboard to reroute things.
| mjayhn wrote:
| Thank you for the early Friday.
| sdenott wrote:
| League of Legends down too, not sure if related.
| askbill wrote:
| Felt like a BGP issue.
| html5web wrote:
| Cloudflare status site is also partially down. Some resources are
| not loading properly.
| greggyb wrote:
| DNS resolution at 1.1.1.1 seems to have gone down and come back
| up for me in the course of 10-15 minutes.
| bitclaw wrote:
| Seems like it's starting to come back.
| clairegraham wrote:
| We were down (downforeveryoneorjustme.com) completely, but back
| up now (as of a few minutes ago). Our domain wasn't even
| resolving; we use Cloudflare for frontend and DNS.
|
| We had a surge of people checking if Discord was down on our
| site, then I noticed everything went down shortly after. Discord
| is still the top check right now.
|
| I can't ever remember hitting these kind of traffic numbers
| before.
| ricardo81 wrote:
| Interesting data you get in the face of adversity, providing
| your host resolves!
| wcchandler wrote:
| I enjoy your service. Have you ever thought about expanding
| your offerings? I would love to see a recreation of "Internet
| Pulse"
| clairegraham wrote:
| Thanks! Yep, we have a lot of things on the todo. We want to
| add more user-focused / location-based outage information
| since our site is still too reliant on simple HTTP checks to
| report downtime. This is especially a problem with a Discord
| outage, for example, where the frontend website is not down,
| but there might be problems with the API, apps, or other
| components.
|
| And I'd like to be able to have our site communicate outages
| like this Cloudflare one, where more than one site might be
| affected by a larger provider. Automating that is difficult.
|
| This is still a side project, though, so I mostly work on it
| when I get the urge :)
| DangerousPie wrote:
| Is this US specific? Everything seems to work fine here in
| Europe.
| kords wrote:
| It's back now (at least for me).
| jonplackett wrote:
| Seems like it's back right?
| elviswolcott wrote:
| The status page is now showing degraded performance for the
| Cloudflare API and Recursive DNS.
| mathattack wrote:
| Interesting. They had an outage in the midst of a negotiation I
| was a part of. Are they less stable than Akamai and the others?
| floatingatoll wrote:
| > * This afternoon we saw an outage across some parts of our
| network. It was not as a result of an attack. It appears a router
| on our global backbone announced bad routes and caused some
| portions of the network to not be available. We believe we have
| addressed the root cause and are monitoring systems for stability
| now.* Jul 17, 22:09 UTC
| zadkey wrote:
| I noticed Udemy was down when I wanted to go to the next video I
| was watching.
| zadkey wrote:
| And now it's back up.
| rob-olmos wrote:
| Hopefully with this outage Cloudflare will finally provide non-
| Enterprise plans a CNAME record, allowing us to quickly bypass
| Cloudflare.
| solarkraft wrote:
| Most pages mentioned here seem functional again.
| amasad wrote:
| Looks like it's resolved -- we're coming back up at Repl.it.
| Rockjodd wrote:
| For people who reports this is down, which country are you in?
| Because all the reported sites works flawless from Norway
| (Europe) :-)
| rvnx wrote:
| Estonia, everything working fine
| Fabricio20 wrote:
| Brazil here, basically everything down.
|
| I noticed a lot of packet loss to 1.1.1.1, not an outright
| "outage", maybe they were rolling a deployment?
|
| Edit: Looks like a deployment to me (looking at the logs I
| could see cascading traces, so it took down one DC and the
| other started responding - increased latency - and then down,
| etc..), gonna be an interesting post-mortem!
| abafazi wrote:
| Australia, everything worked for me, speaking with my friends
| in the UK they are saying everything is down
| Filligree wrote:
| Ireland here, and it's all down.
| britmob wrote:
| East Coast USA here. Any cloudflare site is unreachable, and
| 1.1.1.1 is giving me massive latency and packet loss :)
| Rockjodd wrote:
| I put my bet on some peering fuckups, causing outrages since
| people are having packet loss etc.
| Rockjodd wrote:
| "Update - This afternoon we saw an outage across some parts
| of our network. It was not as a result of an attack. It
| appears a router on our global backbone announced bad routes
| and caused some portions of the network to not be available.
| We believe we have addressed the root cause and are
| monitoring systems for stability now. Jul 17, 22:09 UTC" -
| https://www.cloudflarestatus.com/
| joshstrange wrote:
| US here, everything reported that I've checked is down. My
| Cloudflare sites are down as well.
| akuji1993 wrote:
| Germany, everything down.
| looperhacks wrote:
| Interesting ... Germany here, too, but I didn't see any down
| sites
| mercer wrote:
| I'm in Holland but everything came back up just now, so maybe
| you picked the exact right moment to check?
| easytiger wrote:
| Boats. People. Stop putting them in one single boat.
| xenospn wrote:
| Yup. 1.1.1.1 stopped responding as well.
| cartoonfoxes wrote:
| Back online here.
| Jonnax wrote:
| 340ms average latency to 1.1.1.1 and 47% packet loss. Many sites
| are down. But I guess that's the problem with CDNs.
| tomxor wrote:
| Thought I was going crazy for a second.
|
| This affects so many things it's scary, and Cloudflare status
| page has still not updated. HN got there first.
| nomdep wrote:
| So THAT is why the Internet was acting weirdly!
| tikiman wrote:
| I'm surprised so many people still use them. They took my
| business down (along with half the internet) a few years ago and
| I learned that they were to large of a point of failure.
| alex_young wrote:
| Yesterday I noticed most of their lava lamps are out (which
| generate random bits). Perhaps these are a critical component.
|
| https://photos.app.goo.gl/g6eR8V2PSY3EVjCLA
| maxk42 wrote:
| I'm sure you were joking but they actually are:
| https://blog.cloudflare.com/lavarand-in-production-the-nitty...
| ATsch wrote:
| Despite them actually mixing it into their entropy pools, the
| lava lamps are still entirely for show. The noise of the
| camera sensor itself is going to contribute orders of
| magnitude more entropy than the slow movement of the lamps.
| It's not completely a fake stunt, but it's certainly
| headline-optimized.
| leijurv wrote:
| Someone needs to get these lava lamps plugged back in ASAP!
| soup10 wrote:
| Good guy cloudflare giving programmers an early weekend.
| neurostimulant wrote:
| But overtime for their own programmers :)
| wenbin wrote:
| Friendly reminder (and notes to myself):
|
| Don't use Namecheap and Cloudflare at the same time.
|
| Namecheap is using cloudflare. So if cloudflare is down, you
| can't change DNS settings on Namecheap as well!
| VectorLock wrote:
| A lot of people are saying AWS. I'm having intermittent network
| connectivity issues intra-AZ, so perhaps they lost a data center
| or route flapped one.
| LeoPanthera wrote:
| I'm having real problems with DNS, is this Cloudflare too? They
| say "All Systems Operational", so maybe not?
|
| Half the damn internet is not currently resolving.
| zubiaur wrote:
| Yes, same here. Changed DNSs to level 3, all better now.
| blisseyGo wrote:
| A lot of unusual internal traffic seems to be around Thailand
| (you might have to select "UNUSUAL"):
|
| https://www.digitalattackmap.com/#anim=1&color=0&country=ALL...
| lgats wrote:
| been pretty large numbers from thailand this month according to
| that tool
| ricardo81 wrote:
| Last I read about 7 million hosts are behind Cloudflare. Maybe
| around 3% of the web, but who knows if that counts for critical
| assets etc rather than pages served.
|
| Shameful that so much of our decentralised web is so centralised
| and breakable in one place.
| athesyn wrote:
| Ironically https://downforeveryoneorjustme.com/ is also down. HA
| ddevault wrote:
| Will we take this as a much needed lesson about putting all of
| the internet's eggs into one basket? Probably not.
| fluxsauce wrote:
| My modem also disconnected with signal problems, which was
| interesting. I'm not sure Cloudflare could have caused that?
| lgats wrote:
| maybe your modem uses 1.1.1.1 dns?
| bryan_w wrote:
| Those poor Firefox users who enabled DoH
| stri8ed wrote:
| Ironic, isitdownrightnow.com is down.
|
| All my DigitalOcean instances are down.
| chris_engel wrote:
| Seems to work again for me in germany (Frankfurt)
| heliodor wrote:
| FYI, PagerDuty is not loading!
|
| Time to go back to the drawing board, for a lot of us, to re-
| assess points of failure.
|
| Edit: many websites are failing to DNS resolve but the services
| they provide continue to function fine behind the curtain.
| twunde wrote:
| This is likely your computer's DNS resolver (if you're using
| 1.1.1.1 you're down. I'd switch to 8.8.8.8 temporarily. We've
| had pagerduty alerts coming in since the start (a whole bunch
| of DNS errors from pingdom) and when I click on the slack link,
| pagerduty works for me
| mauriciogior wrote:
| how can I have cloudflare plus something else as a DNS failover?
| We are afraid to set a long TTL and have our IP changed for some
| reason. What do you guys recommend?
| ed25519FUUU wrote:
| It's unfortunate that both the primary and secondary cloudflare
| DNS is down. I just switched my secondary to google.
|
| This allows my internet to "work" during this time, but adds
| about 1s latency to resolutions. Presumably that's the time it
| takes my internal DNS resolver to try the secondary.
| ingenium wrote:
| Considering running your own full resolver like unbound. Then
| you don't have to rely on a DNS provider like Google or
| Cloudflare. It's really nice not having the whole internet go
| down when Google or Cloudflare DNS is down.
| adrr wrote:
| What was interesting and scary is that our monitoring system
| didn't notify us. Our email was down because we use cloudflare
| for DNS and our monitoring provider's SMS gateway was down. So we
| didn't get sms messages.
| ashleyn wrote:
| It really defies the original vision of the internet to have so
| many services depend on a single company. Almost every news site
| I was reading dropped off at once. I thought for a second that I
| lost internet in my own house.
| hn_throwaway_99 wrote:
| Agreed, but I think people really underestimated the forces at
| work that would cause so much consolidation into a couple
| internet giants.
|
| The original idea was that with the barrier to entry being so
| low, anyone and everyone could set up their own websites, mail
| servers, etc.
|
| But with it being so easy to compare and contrast service (i.e.
| the market being so open), it means that the competitive forces
| naturally consolidate to a winner-take-all model. If when
| starting out Cloudflare was just 5% better than the
| competition, it could have easily taken the vast majority of
| the mindshare on the internet. Couple that with the fact that
| there are huge advantages with scale to a business like
| Cloudflare's, and it's not hard to see how so much of the
| internet has become dependent on it.
| remmargorp64 wrote:
| I consider DNS and the way how top level domains are handled to
| be one of the weakest parts of our current Internet design.
|
| We REALLY need a truly decentralized, distributed DNS system
| that is not owned by private entities.
| tenebrisalietum wrote:
| I'm down for passing around a GPG signed hosts2.txt file.
| Let's get started.
| xen2xen1 wrote:
| DNS is decentralized, it's just not when everyone goes with
| one big service.
| hpfr wrote:
| https://handshake.org is pretty interesting.
| spenczar5 wrote:
| The "decentralized internet" folks always talk a lot about
| fighting corporate control. I think they should spend more
| time talking about resiliency and blast-radius reduction.
| q3k wrote:
| DNS worked just fine throughout this. You're barking up the
| wrong tree.
| ghastmaster wrote:
| I just recently ran across this. I wonder how much
| performance would be degraded.
|
| https://ieeexplore.ieee.org/document/7530014/authors#authors
|
| > Unlike previous DNS replacement proposals, D 3 NS is
| reverse compatible with DNS and allows for incremental
| implementation within the current system.
| the8472 wrote:
| DNS is far less of a single point of failure and more
| decentralized than cloudflare. Nameservers can and are
| operated redundantly via simple, resolver-side round-robin
| scheduling and the TLD servers should have longer TTLs that
| allow plenty of caching. The rootzone even has anycast thanks
| to using UDP. Take a moment to look at DoH and laugh.
|
| You can also also register your domain on multiple TLDs.
| Meekro wrote:
| Agreed, but the real problem is DDoS and nobody seems to know
| how to globally solve it. Fighting DDoS is expensive, so you
| see consolidation. It's well and good to live in a tiny farming
| town but when raiders start attacking every week, those castle
| walls and guards start to look really appealing.
| labawi wrote:
| That's what we get for externalizing costs. It's not hard to
| track down sources, but network operators usually let it be,
| hence the incentives are probably counter-productive.
| Algent wrote:
| And the worst is if you try to raise concerns about cloudflare
| now it get brushed of as "cf already proxy half the internet,
| if it goes down our stuff will be minor concern".
| lumberingjack wrote:
| Same here. I'm working at an auto parts store looking though
| ASE parts sites and it was like well close up the store the
| catalogs are missing RN.
| cortesoft wrote:
| I don't understand why the big companies don't always have at
| least two CDN providers, so they can failover to another one if
| something like this happens.
|
| I know a lot of big companies do, but I am always surprised
| when you see ones that don't.
| LoSboccacc wrote:
| the DNS itself is not as easy to duplicate across multiple
| provider, with CF DNS down having a backup CDN wouldn't have
| helped
| cortesoft wrote:
| This isn't true... you can certainly do redundant dns with
| automatic failover between providers. Just set up NS
| records pointing to different providers.
| rickyc091 wrote:
| Same here. Rebooted the router and modem thinking it was me,
| but my phone was still on wifi then realized it was probably my
| cloudflare DNS.
| spiritplumber wrote:
| Pihole is your friend.
| cls59 wrote:
| Yeah, Pihole made it super easy to cut over to Quad-9 once
| I figured out what the problem was.
| newhotelowner wrote:
| I have a pihole. It didn't help.
| rickyc091 wrote:
| Looks like I got another weekend project.
| xen2xen1 wrote:
| Yup, reinforces the thought that you never have both DNS
| servers with the same service.
| asadlionpk wrote:
| This! I got all sorts of alerts from pingdom and my laptop
| refused to get online. Pure Panic!
| jeremyjh wrote:
| Yes its really odd that core backbone providers can go down and
| everything works like its supposed to. Even trans-pacific
| cables can be cut and things will usually work with only
| increased latency. But there is not much redundancy for many
| companies at this layer; having redundant DNS providers is I'm
| sure possible but not something we think about very often, and
| of course many of the sites that are down are depending on the
| proxy and DOS mitigation services.
|
| On my home network I use Google as a backup DNS provider so the
| whole internet didn't go dark for me, but I don't have a backup
| DNS host for my company's DNS records.
| kiobu wrote:
| I imagine most people would never expect something like this
| to happen, so having a fallback option when Cloudflare has a
| huge interruption of service like this is just unthinkable.
| macNchz wrote:
| All the major cloud infrastructure providers have had
| outages of varying severity at one point or another...it's
| something you'd want to take into account for, say, a
| system that remote controls life-critical devices, but
| likely isn't worth the engineering time and added
| complexity for a productivity or social app with a small
| userbase. Working on many of the latter over the years I've
| generally said "well if {major cloud provider} is down, the
| internet is going to be all messed up for a bit anyway, so
| we'll accept the risk of being down when they're down, and
| reassess whether that keeps making sense as we grow."
| woolcap wrote:
| Redundant DNS is possible, but challenging when you're making
| use of features like geo DNS that don't lend themselves to
| easy replication via zone transfer.
| divbzero wrote:
| Would setting up backup DNS hosts simply involve adding NS
| records to point to a different DNS provider?
| haloblue wrote:
| Looks like it's starting to come back in the SE US.
| BookmarkSaver wrote:
| Twitch.tv channels are like 50/50 right now. Some are ok, some
| aren't.
|
| Basically all Riot Games (League, Valorant, TFT) are down, dunno
| about LoR.
| jrockway wrote:
| I don't use Cloudflare, but I do notice Cloudflare services being
| down.
|
| Right now, I can't get to my own website (hosted on DigitalOcean,
| not through Cloudflare), but Oh Dear claims it's up. So I suspect
| that the problem is closer to me than it is to DigitalOcean (or
| Cloudflare).
| rgbrenner wrote:
| DO uses cloudflare for their DNS... both for digitalocean.com
| and for their DNS service.
| jrockway wrote:
| Good to know! That makes perfect sense based on what I saw
| during the outage. I had no idea.
| minxomat wrote:
| DO might use 1.1.1.1 (or Argon even) for routing between some
| of their PoPs
| jrockway wrote:
| Could be. Things are back now, but I was very surprised that
| a Cloudflare outage makes it impossible for me to get to my
| Kubernetes API server.
|
| Hidden dependency revealed.
| navinag wrote:
| yes
| semicolon_storm wrote:
| Must be regional or some other factor involved. Various sites
| others are reporting as offline load for me as does 1.1.1.1.
| exochrono wrote:
| My Pagerduty's been blowing up so I tried to go to their
| dashboard to pause the notifications for now and pagerduty.com is
| down XD
| satysin wrote:
| Guess that explains Discord vanishing from the net a few minutes
| ago.
| heliodor wrote:
| Is it me, or has this been happening way too frequently for them
| lately?
| twunde wrote:
| To be fair the last major outage they had was 1-2 years ago.
| That said, when that happened they had two outages in about a
| month.
| jeremyjh wrote:
| Honestly at their scale once a decade would be too frequent.
| Too many eggs in this particular basket.
| bithaze wrote:
| Once a decade doesn't seem realistic. At some point you get
| diminishing returns chasing as many mines as possible.
| ryanmccullagh wrote:
| I thought my issue was with Comcast, then I realized I'm using
| CF's DNS entries for my home network. I removed those 1.1.1.1
| entries and some sites are working.
|
| Dang, I'm pretty disappointed in CF. I've never experienced this
| much an effectful DNS outage.
| floren wrote:
| 1.1.1.1 is not resolving anything for me at this time.
| minxomat wrote:
| Lol, talk about timing. I'm currently working on a TLS library
| and was pulling my hair out trying to figure out why tests
| against CF sites suddenly failed. Can't even ask my cohorts on
| Discord because they are behind CF, too!
| paulmd wrote:
| "StackOverflow devs have the most difficult job in the world.
| After all, when StackOverflow is down, they can't exactly look
| for help on StackOverflow".
| cknoxrun wrote:
| No kidding! We had literally deployed a major page redesign and
| started watching our analytics drop off on it's way to zero. My
| heart is racing still. I wouldn't normally be happy for a
| cloudflare outage but in this case it's better than Google
| deciding to remove us from their index.
| akuji1993 wrote:
| That's unfortunate timing dude. Good for you, it's probably
| not your mistake :D
| ryanmccullagh wrote:
| Dns issues for sure
| aaomidi wrote:
| Oopsie Daisy half the internet goes down
| minxomat wrote:
| 2020 is all in on everything bad that can happen.
| parliament32 wrote:
| Eggs and baskets etc etc
| nobleach wrote:
| Good DNS practice (at least when I did system admin 10 years
| ago) was ALWAYS having a secondary at some other
| location/network. Why do we just put some info in Cloudflare
| and call it good these days?
| lflux wrote:
| Features, convience, pricing.
| toast0 wrote:
| It's hard to use Cloudflare as a reverse proxy without
| using them as your delegated name servers (maybe you can
| use CNAMEs on paid plans?), and fancy dynamic nameservers
| make it hard to run secondary servers with zone transfers.
| rococode wrote:
| Discord is entirely down right now, both the website and the app
| itself. Amusingly, a lot of the sites that normally track outages
| are also down, which made me think it was my internet at first.
| Downdetector, monitortheinternet, etc.
|
| Lots of other big sites that are down: Patreon, npmjs,
| DigitalOcean, Coinbase, Zendesk, Medium, GitLab (502), Fiverr,
| Upwork, Udemy
|
| Edit: 15 min later, looks like things are starting to come back
| up
| michel-slm wrote:
| Freenode's IRC servers were down which was unexpected for me. I
| was expecting old-school communication networks to not have a
| dependency on Cloudflare.
| maxk42 wrote:
| Discord attempted to route me to:
| everydayconsumers.com/displaydirect2?tp1=b49ed5eb-cc44-427d-8d3
| 0-b279c92b00bb&kw=attorney&tg1=12570&tg2=216899.marlborotech.co
| m_47.36.66.228&tg3=fbK3As-awso
|
| (Visit at your own risk.)
|
| Hack?
| [deleted]
| jpxw wrote:
| How can this be reproduced?
| iamtheyammer wrote:
| Sure you didn't misspell discord?
| maxk42 wrote:
| I've never even heard of the site before. Nor have I
| searched for "attorney" any time recently.
| biermic wrote:
| We operate that site and are using Cloudflare to prevent
| DDOS attacks. Probably some sort of hash collision...
| rocho wrote:
| Crazy stuff!
| Kye wrote:
| It looks like you mistyped it and landed on a domain with
| spammy redirects. They have all kinds of weird URLs and
| there's not always any connection to anything you did
| other than go to the wrong domain.
| maxk42 wrote:
| I didn't mis-type. I press 'd' and Firefox fills-in the
| site for me.
|
| If I type 'e' I get 'en.wikipedia.org'.
|
| I was redirected.
| jeffus wrote:
| I'd be looking at your browser extensions or malware (if you
| use the Discord app).
| e40 wrote:
| Crazily, my local name resolution started failing, because I
| have these names servers: 192.168.0.99, 1.1.1.1 and 8.8.8.8.
| The first does the local resolution, but macOS wasn't
| consulting it because 1.1.1.1 was failing?? Crazy. When I
| removed 1.1.1.1 from the list, everything started working.
| projektfu wrote:
| DNS over HTTP might bypass your local nameservers.
| e40 wrote:
| What was failing was ssh to a local host. I can't imagine
| that Brew ssh uses DNS over HTTP.
| Reason077 wrote:
| Ironically, downdetector.com is also down.
| laughinghan wrote:
| Discord confirmed Cloudflare is also the reason they're down:
| https://twitter.com/discord/status/1284237737638461453
| dafoex wrote:
| I can live without creepy instant messengers, but its shocking
| just how much everything else relies on one, central system.
| And furthermore, why is it always cloudflair?
| BrianHenryIE wrote:
| Cloudflare is free and has a nice UI. I manage ~40 domains
| from ~six domain registrars through it, the consistency is
| great. The caching is a bonus.
| LoSboccacc wrote:
| idk about the UI the redesign with the forced collapses and
| extra clicks everywhere is annoying when handling a
| multitude sub domains plus their let's encrypt text entries
| I'm there mostly for the freebies.
| RcouF1uZ4gsC wrote:
| > Amusingly, a lot of the sites that normally track outages are
| also down, which made me think it was my internet at first.
|
| That is why if you have this question, you should go to
| google.com
|
| My guess is that there are more resources invested in making
| sure google.com stays up than for any other site on the
| internet.
| qmarchi wrote:
| Depending on what part we're talking about, it varies. But
| yeah, just a few.
| emsy wrote:
| Same for the German downtime trackers.
| Cultmethod wrote:
| Thought something like this was going on. At first I thought it
| was my router and restarted everything - to no avail. Glad to
| see confirmation that it wasn't an issue on my end.
| ljm wrote:
| I wonder if they are all using Cloudflare's free DNS stuff or
| if they're paying for business accounts?
|
| My stuff is on Netlify (for the next week or so) and the rest
| is on a VPS bought from a local business who isn't reselling
| cloud resources. I'm kinda glad I moved all my stuff from
| cloudflare.
| jorgenphi wrote:
| I think it's going to be everyone. Some of my free sites are
| dead, but also huge enterprise Cloudflare users
| (Discord/Patreon/4chan) are also dead.
| thepete2 wrote:
| same here. I tried if hacker news still works and saw this
| belltaco wrote:
| https://status.discord.com/ is down. Wow.
| bufferoverflow wrote:
| Works for me.
| tremon wrote:
| I get a DNS servfail when resolving that DNS record, and
| many others: Server: 8.8.8.8
| Address: 8.8.8.8#53 ** server can't find
| status.discord.com: SERVFAIL
|
| So it's either not just cloudflare, or all those sites use
| cloudflare to host their DNS.
| anderskaseorg wrote:
| The latter. $ dig +short discord.com ns
| sima.ns.cloudflare.com. gabe.ns.cloudflare.com.
| WillPostForFood wrote:
| 8.8.8.8 is Google and is down in addition to Cloudflare.
| 8.8.4.4 was still up.
| pepemon wrote:
| 8.8.8.8 is a caching resolver and it wasn't down. Do you
| understand how caching resolvers work?
| christoph wrote:
| Same. It has a big banner saying CloudFlare is down.
| Miner49er wrote:
| It's hosted by statuspage.io, and their own status page was
| also down (metastatuspage.com). It is now back up, but their
| page shows the outage.
| proactivesvcs wrote:
| Status turtles all the way down, it seems!
| abafazi wrote:
| Works for me too... Australia
| Deathmax wrote:
| And that is why you host your status page on separate infra.
| Lyrex wrote:
| We're dealing with a deeper level problem here. Since a lot
| of the internet is relying on Cloudflare DNS at some part
| or another, even many backup solutions fail. Since so much
| of DNS is centralised in so few services, such outages hit
| the core infrastructure of the internet.
| cm2187 wrote:
| A sudden disruption on a large number of services for
| everybody at once doesn't look like a DNS problem to me,
| with all the caching in DNS. It would fail progressively
| and inconsistently.
| scarby2 wrote:
| given that TTLs are usually very short now if your DNS
| server is configured correctly then caching shouldn't
| make a bit of difference.
| cm2187 wrote:
| I looked at the example above, all of patron,
| digitalocean, coinbase and gitlab (haven't checked the
| others) have a ttl of 1h.
| Figs wrote:
| DNS absolutely was an issue. I changed DNS manually from
| Cloudflare's 1.0.0.1 (which is what DHCP was giving me)
| to 8.8.8.8 (Google) and most things I'm trying to reach
| work. There may be other failures going on as well, but
| 1.0.0.1 was completely unreachable.
| cm2187 wrote:
| I don't use cloudflare DNS but google DNS and got the
| same problems thant everyone else
|
| The problem seems to have been resolved now, you might
| have made the change when they fixed it.
| cuu508 wrote:
| > I don't use cloudflare DNS but google DNS and got the
| same problems thant everyone else
|
| Cloudflare is also the authoritative DNS server for many
| services. If Cloudflare is down, then for those services
| Google's DNS has nowhere to get the authoritative answers
| from.
| cm2187 wrote:
| Except the services mentioned in the original post have a
| ttl of 1h. Unlikely they would all go down at the same
| time.
| Figs wrote:
| No, I changed the setting back and forth while it was
| down to confirm that the issue was that I could not reach
| 1.0.0.1. All the entries I tried from my host file were
| responsive (which is how I ruled out an issue with my
| local equipment initially and confirmed that it wasn't a
| complete failure upstream -- I could still reach my
| servers). Changing 1.0.0.1 to 8.8.8.8 allowed me to reach
| websites like Google and HN, and changing back to default
| DNS settings (which reset to 1.0.0.1, confirmed in
| Wireshark) resulted in the immediate return of failures.
| 1.0.0.1 was not responsive to anything else I tried.
|
| Again, it may not have been the only issue -- and there
| are a number of possible reasons why 1.0.0.1 wasn't
| reachable -- but it certainly was an issue.
| cortesoft wrote:
| Yeah, but then your status page provider switches THEIR
| provider, and suddenly you are on the same infra again.
| bloopernova wrote:
| Doordash too. My first order. On my wife's birthday.
|
| Ah, well. This too shall pass.
| mjayhn wrote:
| I did a pickup order a few weeks ago when of all things
| Tmobile SMS went down for 3+ hours. I couldn't go in the
| restaurant (covid) and I couldn't text them the parking # I
| was sitting at in a packed parking lot. I got a flood of
| about 50 texts a few hours later. Sat there for about an hour
| waiting for a $9 sandwich. I have no idea if they didn't get
| my order until late, or if they finally realized it was me or
| what. About 45 minutes in I decided to just give up on the
| day and take a nap, woke up to a door knock.
| mackal wrote:
| I noticed discord being down, so I went to check
| downforeveryoneorjustme, also down. So I figured I'd check
| NANOG mailing list, also down :P
| clairegraham wrote:
| Yep, we were down completely. We are quite dependent on
| Cloudflare (frontend + dns).
| saagarjha wrote:
| Hacker News is an excellent status page for those cases.
| blisseyGo wrote:
| Out of curiosity, done HN use any CDN or other way of DDOS
| protection? dang?
| AdamGibbins wrote:
| It's hosted on AWS, you don't need DDoS protection, just a
| big wallet.
| gruez wrote:
| Only if you use their "infinitely scaling" services. eg.
| s3. If the attacker is hammering you with expensive
| queries and your database is on 1 ec2 server, you're
| still going to go down.
| ivalm wrote:
| The nameserver is aws but IP4 points to "m5 computer
| security"
| saagarjha wrote:
| Hacker News is self-hosted:
| https://news.ycombinator.com/item?id=22767439. Let me see
| if I can find a better link where the specs are
| discussed.
| dijit wrote:
| Thats not even slightly true.
|
| Their IP belongs to AS21581; which is registered with a
| company called 'M5 Computer hosting' out of west-coast
| USA.
|
| m5hosting.com
|
| The last hop is Santa Barbera.
|
| Definitely does not fall in the AWS ranges.
| snazz wrote:
| Don't think so. They used to use Cloudflare but stopped. To
| my knowledge, it's a single server without a database
| (using the filesystem as a database).
| ksec wrote:
| So HN is serving 5.5M page view daily (excluding API
| access ) on a single server without CDN and without a
| database?
|
| Holy crap I am thinking either there is some magic or
| everything we are doing in the modern web are wrong.
|
| Edit: The number is from Dang [1]
|
| > _These days around 5.5M page views daily and something
| like 5M unique readers a month, depending on how you try
| to count them._
|
| [1] https://news.ycombinator.com/item?id=23808787
| techntoke wrote:
| Flat-file DBs and mountable DB file systems are the
| future.
| blhack wrote:
| >Holy crap I am thinking either there is some magic or
| everything we are doing in the modern web are wrong.
|
| Spin up an apache installation and see how many requests
| you can serve per second if you're just serving static
| files off of an SSD. It's a lot.
|
| edit: I see that there are already a bunch of other
| comments to this effect. I think you're comment is really
| going to bring out the old timers, haha. From my
| perspective, the "modern web" is absolutely _insane_.
| ksec wrote:
| >I think you're comment is really going to bring out the
| old timers, haha.
|
| That is great ! :D
|
| >It's a lot.
|
| Well yes, but HN isn't really static though. Fairly
| Dynamics with Huge number of users and comments. But
| still, I think I need to rethink lots of assumption in
| terms of speed, scale and complexity.
| Izkata wrote:
| There is some caching somewhere as well, probably
| provides a bit more boost.
|
| I've been at my work laptop (not logged in) and found
| something I wanted to reply to, so I pulled out my phone
| and did so. For a good 10 seconds afterwards, I could
| refresh my phone and see my comment, but refresh the
| laptop and not see it.
| arghwhat wrote:
| Huge numbers of users don't really mean that much.
| Bandwidth is the main cost, but that's kept low by having
| a simple design.
|
| Serving the same content several times in a row requires
| very few resources - remember, reads far outnumber
| writes, so even dynamic comment pages will be served many
| times in between changes. 5.5 million page views is only
| 64 views a second, which isn't that hard to serve.
|
| As for the writes, as long as significant serialization
| is avoided, it is a non-issue.
|
| (The vast majority of websites could easily be designed
| to be as efficient.)
| cmroanirgo wrote:
| >* From my perspective, the "modern web" is absolutely
| insane. *
|
| Agreed.
|
| I was brought up as a computer systems engineer... So,
| not a scientist, but I always worked with the basic
| premise of keep it simple. I've worked on projects where
| we built all the fangled clustering and master/slave
| (sorry to the PC crowd, but that's what it was called)
| stuff but never once needed it in practice. Our stuff
| could easily handle saturated gigabit networks as the 2
| core cpu only running at 40%. We had cpu spare and could
| always add more network cards before we needed to split
| the server. It was less maintenance, for sure. It also
| had self healing so that some packets could be dropped if
| the client config allowed it, if the server decided it
| wanted to (but only ever did on the odd dodgey client
| connection)
|
| That said, I was always impressed by the map-reduce of
| for search results (yes, I know they've moved on) which
| showed how massive systems can be fast too. It seemed
| that the rest of the world wanted to become like Google,
| and the complexity grew for the std software shop, when
| it didn't need to imho.
|
| I jumped ship at that point and went embedded, which was
| a whole lot more fun for me.
|
| Sincerely, old timer
| m0xte wrote:
| Modern web is a completely broken mess.
|
| We were serving around that traffic off a single dual
| pentium 3 in 2002 quite happily off IIS/SQL Server/ASP.
| The amount of information presented has not grown either.
|
| That little box had some top tier brand main corporate
| web sites on it too and was pumping out 30-40 requests a
| second peak. There was no CDN.
| bitdeep wrote:
| Man, if this is true, this guys have steel balls.
| rodgerd wrote:
| Back in 2000 a joke project of mine got slashdotted. Ran
| outta bandwidth before anything else.
| ci5er wrote:
| With DO, these days, they don't run me out of bandwidth,
| but my instance falls over (depending on what I am doing
| - which ain't much), but with AWS, they auto-scale and I
| get a $5000 bill at the end of the month. I prefer the
| former.
| compumike wrote:
| Not sure where your 5.5M number came from, but that's
| only 64 requests per second.
|
| 90 to 99% of those are logged-out users, so fully
| cacheable.
|
| Only a handful of dynamic requests each second remain.
| ci5er wrote:
| Unlike Reddit, logged in and logged out users largely see
| the same thing. I wouldn't imagine there is much logic
| involved in serving personalized pages, when they don't
| care who you are.
| chc wrote:
| A lot of modern web technology is inefficient for the
| sake of being ergonomic. Here's what Hacker News looks
| like: https://github.com/arclanguage/anarki/blob/master/a
| pps/news/...
| mst wrote:
| From an old-school lisper perspective, the code seems
| perfectly ergonomic to me.
|
| It's ergonomic in a very lispy way but perfectly
| reasonably so from the POV of that aesthetic.
| jeffbee wrote:
| That's only ~60 QPS, assume it is peaky and hits
| something more like 1000 QPS in peak minutes, but also
| assume most of the hits are the front page which contains
| so little information it would fit literally in the
| registers of a modern x86-64 CPU.
|
| Even a heavyweight and badly written web server can hit
| 100 QPS per core, and cores are a dime a dozen these
| days, and storage devices that can hit a million ops per
| second don't cost anything anymore, either.
| saagarjha wrote:
| In-memory databases? That's amateurish. Time to make a
| service that runs out of ymm registers.
| q3k wrote:
| 1M queries per day is ~10 queries per second. It's a
| useful conversion rate to keep in mind when you see
| anyone brag about millions of requests per day.
| manquer wrote:
| It is certainly doable , PoF was running lot of page
| views famously of a single IIS server for a long time .
|
| HN is written in a lisp variant and most the stack is
| built in-house , it is not difficult to imagine
| efficiency improvements when many abstraction layers have
| been removed from your stack .
| ci5er wrote:
| I don't remember PoF being famous for that, but they got
| a lot of bang for the buck on their serving costs.
|
| What I _do_ remember, is that it was a social data
| collection experiment for a couple of social scientists,
| that never originally expected that many people would
| actually find ways to find each other and hook up using
| it.
|
| I miss their old findings reports about how weird humans
| are and what they lie about. Now, it's just plain boring
| with no insights released to the public.
| dmarlow wrote:
| For all my sibling comments, there is also context to be
| aware of. 5.5m page views daily can come is many shapes
| and sizes. Yes, modern web dev is a mess, but situation
| is very different from site to site. This should be taken
| as a nice anecdote, not as a benchmark.
| zaksoup wrote:
| I had this same reaction. Definitely feels like most of
| what we're doing with "the modern" web is probably wrong.
| idlewords wrote:
| You can serve a lot of flat files from a properly
| configured server in 2020. It's just that most people
| don't bother trying.
| loeg wrote:
| Well, it's not magic. So, the other one.
| opqpo wrote:
| I doubt this is physically possible. I think they have
| distributed edge caches for pages with short TTLs.
| layoutIfNeeded wrote:
| We've been telling this for a while now...
| [deleted]
| jyap wrote:
| You don't need a CDN if what you're serving up in this
| case is mostly all text.
|
| Just need good stable code and server side caching.
| winrid wrote:
| Makes sense based on what I've read about Arc, which HN
| is written in.
|
| I've been working on something where the DB is also part
| of the application layer. The performance you can get on
| one machine is insane, since you spend minimal time on
| marshalling structures and moving things around.
| ivalm wrote:
| Their dns record points to only one IP (209.216.230.240)
| that goes to M5 Computer Security
| compumike wrote:
| It's a hosting company in San Diego:
| https://www.m5hosting.com/
|
| I host a dedicated server there (running
| https://www.circuitlab.com/) and when I traceroute/ping
| news.ycombinator.com, it's two hops (and 0.175 ms) away
| :)
| kohtatsu wrote:
| owo
| zackees wrote:
| This was a coordinated attack that took down a bunch of
| services all at the same time:
|
| https://twitter.com/KEEMSTAR/status/1284240539437740033
| https://downdetector.com/
| brian-armstrong wrote:
| This was a BGP/routing issue and has already been
| documented. Please don't spread misinformation and
| hysteria, especially on technical issues like this
|
| https://twitter.com/eastdakota/status/1284253034596331520
| dtertman wrote:
| It was no such thing. A Cloudflare router advertised some
| bad routes.
| kingbirdy wrote:
| I wouldn't count on Keemstar as a reliable source of
| cyber-attack coverage
| [deleted]
| leon-z wrote:
| Kudos to the people at Discord. Just a few minutes after I got
| disconnected they already tweeted about the issue. Some minutes
| later and they have a message in their desktop app confirming
| it's an issue with Cloudflare. All while Cloudflare's
| statuspage says there are 'minor outages'.
| mjayhn wrote:
| Every company rushes to report an outage when they can blame
| another vendor, well that might be hyperbolic but it's sure a
| lot easier!
| [deleted]
| GeneralTspoon wrote:
| Same here!
|
| I even checked to see if an AWS region was down once I realised
| it wasn't on my side (I thought it might have been my ISP's DNS
| servers or something).
|
| The next move was to check Hacker News - thankfully it's not
| also hosted on Cloudflare, ha!
| solarkraft wrote:
| Discord works for me, but https://redbubble.com/ prints
| "Service unavailable".
| macNchz wrote:
| My iPhone actually popped up a message saying that my wifi
| didn't appear to have internet, which was strange and obviously
| false as I was actively using the internet on it and the laptop
| next to it, but now it makes sense that it must have been
| pinging something backed by cloudflare!
| neurostimulant wrote:
| Seem to be localized issue. Cloudflare is up here in my country,
| but down for many people in US.
| r0xsh wrote:
| Ah Shit, Here We Go Again
| julien wrote:
| I am probably the only one who cares but even
| https://downforeveryoneorjustme.com/ is down
| dayjah wrote:
| First place I went to also. Then HN.
| clairegraham wrote:
| Yep, we are very dependent on Cloudflare :(
| ATsch wrote:
| I personally find sites like https://outage.report or
| https://downdetector.com which tally up the number, regions and
| history of people saying it isn't working for them more
| conclusive.
| djsumdog wrote:
| Weird, it works for me (US)
| Reedx wrote:
| Now we need isdownforeveryoneorjustmedown.com
| bdibs wrote:
| It's up for me.
| Majora320 wrote:
| Seems like they managed to break half the internet for everyone.
| gregory90 wrote:
| Cloudflare DNS is down too
| jlmorton wrote:
| Monday Morning RCA: "We pushed out some routine code updates, but
| this really weird thing happened causing a resource utilization
| spike on our DNS systems. Because of this other really weird
| thing, this affected all of our global infrastructure
| simultaneously. Here's a deep engineering dive into this one
| weird thing that brought everything down."
| iJohnDoe wrote:
| Ironically StatusCake is down as well.
| sascha_sl wrote:
| DNS seems to be dead, if you have stuff in your cache and the
| site isn't low-TTL things still kinda work
| icodestuff wrote:
| Yeah this is absolutely killing us right now.
| karlmcguire wrote:
| "All systems operational"
|
| What's the point of a status page if it doesn't reflect the real
| status...
|
| It's either the status page goes down with everything else or the
| status page is wrong. Great.
|
| EDIT: Looks like it's accurate now, 20 minutes later.
| cellar_door wrote:
| This is an Atlassian Statuspage status page, so it's not hosted
| by Cloudflare.
| [deleted]
| Xenoamorphous wrote:
| IBM Cloud status is pretty much always green... although we
| have issues pretty much every week.
| mathattack wrote:
| They're still using Lotus Notes for the tracking.
| smsm42 wrote:
| this-is-fine.gif
| dewey wrote:
| Status pages are a marketing channel not a channel for
| developers most of the time. It most likely has to go through
| some layers before someone updates the status page.
| jeremyjh wrote:
| Let's start a betting pool. How many upvotes do you think OP
| will get before the status page acknowledges a problem? I say
| its going to be 600.
| HappyKasper wrote:
| And it looks like they started "investigating" at around 450!
| p0llard wrote:
| They've just started showing a message for me now, at ~450
| upvotes.
| jedberg wrote:
| You lost. ;) 476 points, status page says it's down now.
| acid__ wrote:
| As one would expect, it says "degraded performance" instead
| of "down" lol
| gpm wrote:
| Tested with tor and it's right. Some exit nodes aren't
| affected.
| acid__ wrote:
| Hm, maybe it's just the SRE in me talking, but if major
| chunks of the internet being entirely inaccessible
| doesn't count as an "outage", what does?
| saagarjha wrote:
| This post is getting something like 30 upvotes a
| minute...might want to up that a bit ;)
| Miner49er wrote:
| There are status page providers that actually monitor services
| and automatically update. Cloudfare just doesn't use them.
| parliament32 wrote:
| Despite their update, I like how they're saying only their
| recursive DNS had "degraded performance", while authoritative
| is "operational". The entire reason everything blew up was
| because their authoritative nameservers weren't responding.
| Jasper_ wrote:
| The point of the status page is so you can point to it for your
| five nines SLA and go "look? we were only down for one hour".
| As soon as the money relies on the metric, the metric will
| reflect the money.
| mjlawson wrote:
| Goodhart's Law[1] in action.
|
| [1]https://en.wikipedia.org/wiki/Goodhart%27s_law
| gautamcgoel wrote:
| I was trying to play video games but couldn't connect. Amazing
| how connected the web is now - one big hub goes down and brings
| the whole house of cards down with it.
| salmaanp wrote:
| who deployed on a friday afternoon?
| jpomykala wrote:
| Friday
| 1f60c wrote:
| That page still shows "Cloudflare System Status: All Systems
| Operational" for me, but it's _definitely_ down for me. Along
| with 1.1.1.1, which is... _bad_.
| cryptoz wrote:
| For me it says "Minor System Outage" for about 0.1s and then
| shows "All Systems Operational".
| ehsankia wrote:
| Everything works fine for me (Canada), am I missing something
| or is it over already?
| reactorofr wrote:
| Still down here.
| cartoonfoxes wrote:
| Also in Canada. Shit's fucked, yo.
| sbr464 wrote:
| Yep, the DoorDash app is affected currently, my burritos!
| Exuma wrote:
| Aside from this one issue, is switching to 1.1.1.1 a good idea in
| your guys experience? Right now I just realized I hvae the DNS
| for my ISP which is probably how they inject bullshit 404 pages
| full of ads. What is the fastest/best public DNS in your guys
| experience?
| BenjiWiebe wrote:
| For us, it's cloudflare. Our ISP is connected to KCIX, and
| cloudflare apparently has 1.1.1.1 servers in Kansas City. No
| other free DNS provider is as quick, for us. 18ms or so RTT, as
| opposed to the WISP's internal latency of ~10ms. Central
| Kansas.
| 77ko wrote:
| I've been using https://nextdns.io/ - works fast and most
| importantly blocks a bunch of adds (user configurable), so
| makes browsing on mobile much nicer.
|
| Ancedotally the interenet seems faster.
| xur17 wrote:
| I've been pretty happy with 1.1.1.1 (before now). Might be
| worth using something like 8.8.8.8 as a backup (Google).
| vulcan01 wrote:
| You could also use 9.9.9.9 as your backup, if you're avoiding
| Google (https://www.quad9.net)
| johnxie wrote:
| Can confirm for taskade.com
| JoshGlazebrook wrote:
| Having DNS issues. Had to switch to Google's DNS
| (8.8.8.8/8.8.4.4) as 1.1.1.1/etc were not resolving anything.
| en4bz wrote:
| 1.1.1.1 is dropping ICMP pings while 8.8.8.8 is not but 8.8.8.8
| is still returning DNS errors.
| jeffbee wrote:
| That only worked until all the records expired from Google's
| cache.
| fpgaminer wrote:
| Same. Even then, a bunch of sites are down. Maybe only ones
| behind Cloudflare? So far I've been trying to hit the various
| down detector sites and none of them will load. Google, Reddit,
| Hackernews are all fine.
| drrotmos wrote:
| From what little I've been able to gather, anything using
| Cloudflare's DNSes are down.
| guiambros wrote:
| Same. Had to go through the entire troubleshooting process ---
| is it my internet connection? my DNS resolver? firewall? ISP
| starting to filter DoT queries somehow?
|
| Only last in my mental list was the possibility that Cloudflare
| would be down.
|
| Hope they publish a detailed post-mortem. It's always fun to
| read (but certainly very painful for those directly involved in
| writing it).
| [deleted]
| heliodor wrote:
| WebGazer.io managed to shoot me an email about my site being
| down. This in spite of their site being down too.
| th0th wrote:
| Hi @heliodor, WebGazer founder Gokhan here :)
|
| Actually the site is running but not accessible due to the
| issue. Glad you got the heads up, after a while I had to pause
| monitoring to prevent side effects.
| jgrahamc wrote:
| This was a problem with our backbone network; wasn't caused by an
| attack. The effect was regional and not global. Naturally, we'll
| write it all up.
| EE84M3i wrote:
| Was it a problem with a provider you use?
| jgrahamc wrote:
| Looks like problem with one of our large routers in Atlanta.
| microcolonel wrote:
| What's all this about "building a better internet"? Wide-reaching
| general service outages that are invisible to your status page
| are really not great.
| erichocean wrote:
| I wonder if they'll cover why their status page is a steaming
| pile of garbage in the post-mortem?
| rozab wrote:
| We can't keep going on like this. The vulnerability of
| centralised internet infrastructure is a huge problem for
| everyone. Somebody, somewhere, really ought to sort it all out
| [deleted]
| oliverobscure wrote:
| If it impacts enough wallets, things might change. I'm not
| holding my breath though.
| parliament32 wrote:
| Why not you? Just don't use CF. The more people stay away from
| CF, the better.
| adsjhdashkj wrote:
| I feel like for a lot of sites CF & CDNs are the only way to
| survive Reddit/HN/etc - do you disagree?
|
| I definitely agree in concept with you, but then i think back
| to how frequently script kiddies took down sites ~10 years
| ago, or w/e. I feel like what has changes is the massive CDNs
| in front of so many sites.
|
| So while i do want a better solution, i'm not sure what it
| looks like. Thoughts?
| mmahemoff wrote:
| Some kind of decentralized CDN in theory.
| parliament32 wrote:
| Does it really matter? If you're small, who cares if you go
| down for half an hour? What, you'll make $0.02 this hour
| instead of $0.05? If you're big, you can afford your own
| infrastructure. Stick a few servers in a few colos around
| the world and you'll have better uptime than CF and friends
| anyway.
| atemerev wrote:
| Be the change you want to see in the world :) There are no
| somebodys somewheres.
|
| One question is how to do DDoS protection without somebody like
| Cloudflare. Some new protocol for edge caching, perhaps?
| toast0 wrote:
| DDoS has two components:
|
| a) complexity: trick your servers into doing something hard
|
| b) volumetric: overwelm your servers with a lot of traffic
|
| c) volumetric part two: overwelm your servers with a lot of
| requests, so you respond with a lot of traffic
|
| A and C are things you can work on your self --- try to limit
| the amount of work your server does in response to requests,
| and/or make resource consuming responses require resource
| consuming requests; and monitor and fix hotspots as they're
| found.
|
| B is tricky, there's two ways to solve volumentric attack;
| either have enough bandwidth to drop the packets on your end,
| or convince the other end to drop the packets (usually called
| null routing). Null routes work great, but usually drop all
| packets to a particular destination IP, which means you need
| to move your service to another IP if you want it to stay
| online; that's hard to do if your IP needs to stay fixed for
| a meaningful time (TTL for glue records at TLDs is usually at
| least a day); and IP space is limited, so if your attackers
| are quick at moving attacks, you could run out of IPs to use.
| Some attacks are going above 1 Tbps though, so that's a lot
| of bandwidth if you need to accept and drop; and of course,
| the more bandwidth people get so they can weather attacks,
| the more bandwidth that can be used to attack others if it's
| not well secured.
| tick_tock_tick wrote:
| B) is the only one that really needs a solution and traffic
| is breaking two or three levels above you.
| huderlem wrote:
| I'm not very familiar with DDoS protection strategies. Can
| you please elaborate on what is meant in (c) by "make
| resource consuming responses require resource consuming
| requests"?
| toast0 wrote:
| Make people login before doing a search is a common
| example for forums. Search is hard, unauthenticated
| search will bring low end forums down, so they make you
| create an account and login.
|
| That sort of thing.
| remmargorp64 wrote:
| Sounds like a problem for... us!
| juancampa wrote:
| If only there was some website full of computerphiles...
| Reedx wrote:
| > Somebody, somewhere, really ought to sort it all out
|
| That could be the slogan for 2020
| fivre wrote:
| 10-20 minute router misconfigurations and subsequent fixes are
| sometimes a fact of life. big network infrastructure is
| complicated, and sometimes the best laid route tables of mice
| and men do go abloop and die.
|
| Outages happen no matter what the infrastructure is. There's no
| solution, they're just something you need to recognize and
| handle, which Cloudflare seemingly did relatively quickly here.
| buro9 wrote:
| From what I can see externally this looks like DNS.
|
| I wonder if that includes the roots that Cloudflare operate.
| mongol wrote:
| How does Cloudflare compare to Akamai?
| byteofbits wrote:
| It's worth mentioning here that 1.1.1.1 is also affected by this
| outage which initially made me think my internet was gone
| completely.
|
| Changing back to an alternative (such as 8.8.8.8 from google)
| restored my access to the areas of the internet not using
| Cloudflare.
| ransom1538 wrote:
| Jesus. Does anyone know anything?
| jorgenphi wrote:
| The uptime tool I use (StatusCake) is itself down... Was
| wondering why I didn't get an alert.
| michael_j_ward wrote:
| Having issues with gitlab myself
| tomklein wrote:
| Back online for me.
| xtracto wrote:
| This is hitting my production environments as well :-(
| sillysaurusx wrote:
| Our TPU management page is also down:
| https://www.tensorfork.com/tpus
|
| Seems cloudflare took out a good chunk of the internet
| temporarily.
|
| Doesn't HN use cloudflare? Why did it survive? (I haven't looked
| for about a year, but I seem to remember HN being proxied behind
| CF at one point.)
| parliament32 wrote:
| It doesn't look like it does. HN's IP space belongs to "M5
| Computer Security", and their DNS nameservers are on "awsdns".
| Nothing there to suggest CF.
| dang wrote:
| HN went off Cloudflare a couple years ago.
|
| https://news.ycombinator.com/item?id=18188832
|
| https://news.ycombinator.com/item?id=21799045
| dom96 wrote:
| How do you deal with DDoS attacks?
| formerly_proven wrote:
| HN allegedly still runs on one machine running a single-
| threaded Lisp webserver.
| saagarjha wrote:
| https://news.ycombinator.com/item?id=22767439
| searchableguy wrote:
| Is there a status page for HN?
| saagarjha wrote:
| Yeah, it's whether news.ycombinator.com loads :P
| sillysaurusx wrote:
| EDIT: Yes: https://twitter.com/HNStatus
|
| HN is so reliable that's it's almost never needed one. I'm
| extremely curious how HN survived this; almost positive they
| used cloudflare at one point.
|
| I think the official status page is @hnstatus on Twitter, or
| something like that.
| codegeek wrote:
| HN is reliable until AWS Route53 goes down :).
| toomuchtodo wrote:
| They did use Cloudflare, but also haven't for some time.
| [deleted]
| jimz wrote:
| Parts of the site that are behind CF like the API are down.
| parliament32 wrote:
| The status page linked shows "All Systems Operational" for me.
| Tested in private browsing and on my mobile.
|
| Looks like DNS issues, their nameservers aren't reachable.
| decad wrote:
| DNS seems to be resolving for me in the UK now
| dangwu wrote:
| League of Legends, Valorant and Discord both down. I took today
| off to play games...
| [deleted]
| FireBeyond wrote:
| Another useless status site.
|
| DNS is completely broken.
|
| "All systems operational" in nice soothing green.
|
| No, not so much.
| Exuma wrote:
| Update - This afternoon we saw an outage across some parts of our
| network. It was not as a result of an attack. It appears a router
| on our global backbone announced bad routes and caused some
| potions of the network to not be available. We believe we have
| addressed the root cause and monitoring systems for stability
| now.
| whoisjuan wrote:
| Their status page says that everything is operational. So much
| for a status page when half of the internet breaks down.
| techlaw wrote:
| itch.io down
|
| isitdownrightnow.com down
| caudamus wrote:
| Cloudflare's DNS (1.1.1.1) is failing to respond to most/all
| queries, which I'm observing as the root cause of a bunch of
| connection issues (name lookup failure).
|
| Interestingly the same domains don't show up on google's
| (8.8.8.8) DNS at all.
| parliament32 wrote:
| 8.8.8.8 is a caching resolver, it still needs to talk to CF's
| nameservers for authoritative records.
| geerlingguy wrote:
| I don't think it's just Cloudflare; I just had a fun 10 minutes
| seeing servers start flipping on my Server monitoring service[1].
| This has only happened once or twice per year, and is usually due
| to weird global DNS issues.
|
| [1] https://servercheck.in/
|
| (To give an update, I'm seeing from my monitoring systems (about
| 15 points around the globe) sporadic outages for Microsoft,
| Apple, Reddit, Bing, Node.js, Twitter, Yahoo, and YouTube. And my
| own servers (not behind CF at all) are also flipping up and down.
| It started around 21:14 UTC.)
| cm2187 wrote:
| a DNS issue wouldn't cripple all of the internet at once, with
| all the caching.
| xtracto wrote:
| It was interesting that we saw our domains affected from the
| USA but from Mexico everything looked OK.
|
| The crazier thing is that I tried to login to our CloudFlare
| account, it never sent me the 2FA code... I still haven't
| been able to login (Enterprise account)
| RL_Quine wrote:
| Most sites set the absolute minimum TTL for every record, for
| no reason. There's a lot less caching than you're thinking.
| cm2187 wrote:
| No I see some services failing that have a TTL of 1h.
| qeternity wrote:
| Eh, what? There are many good reasons to have low TTL
| DNS...this exact outage being one of them. Update your
| records to go direct to your servers, and not through
| Cloudflare and bam you're back up. Doesn't work if your TTL
| is 86400
| unilynx wrote:
| Doesn't help as cloudflare wants you to host their name
| servers with them, so you can't flip any records if the
| DNS itself is in trouble, like it is now
|
| And changing DNS servers often takes many hours (or days,
| if .net is involved apparently)
| arjun27 wrote:
| more like Cloudflared
| solarkraft wrote:
| Finally we see how much we depend on this single company.
| beatrobot wrote:
| More like the Internet is down.
| saagarjha wrote:
| When you depend on a single company for much of the internet,
| such things happen :(
| logicalmonster wrote:
| Given that the US is basically in a non-shooting war with China,
| I wonder if this is something technical or part of some kind of
| attack. Something that I'd keep in mind.
| wolfgang42 wrote:
| There's enough ways for bits of the Internet to go kablooey on
| their own that "it's an attack!" is a pretty big jump to a
| conclusion. If this turns out to be something other than
| Cloudflare tripping over a weird bug, my first guess would be
| that someone fat-fingered a BGP table yet again.
| searchableguy wrote:
| Your username is funny.
| Hanabishi wrote:
| Well, sheit. This is all around the world. Press F.
| formerly_proven wrote:
| Centralising on a single host suddenly not a good idea any more?
| awinder wrote:
| NextDNS got taken out by this, id been really happy with it up
| until now. And unfortunately "dns service went down" has a wide
| enough blast radius at home now that it's a real pain.
| ricopags wrote:
| How did you verify that? I determined the issue was with
| Cloudflare's DNS by toggling on NextDNS, which worked and
| continues to.
| mindfreeze wrote:
| I was having troubles with overleaf.com
| andrewnicolalde wrote:
| 1.1.1.1 is back for me now
| rgbrenner wrote:
| Reminder for firefox users: Firefox uses DNS over HTTPS and the
| default is cloudflare. If you're having DNS issues, you need to
| disable it until cloudflare is back up.
| mxschmitt wrote:
| Site that use Discord, Linode, Patreon, npmjs, DigitalOcean,
| Coinbase, Zendesk, Medium, Gitlab (502), Fiverr, Upwork, Udemy
| and many more including 1.1.1.1 dns down. Ref:
| https://twitter.com/nixcraft/status/1284239374809395200?s=19
| icey wrote:
| Seeing a lot of people mentioning DO, but it has been up for me
| without any issues (small VPS in SF-2)
| ranrub wrote:
| Cedexis gets another lease on life
| chuckdries wrote:
| lmao it even took down my local stack
| tomklein wrote:
| NPM is down too.
| interator7 wrote:
| 2:36 PM PST - status.discord.com is back up.
| britmob wrote:
| Looks like it's back. No longer getting issues with 1.1.1.1 and
| domains are being resolved!
| ninkendo wrote:
| Not for me, `dig @1.1.1.1 google.com` is returning SERVFAIL
| still. Their anycast config may be broken in some way (ie. the
| backends for some regions are down, but still advertising
| routes)
| basch wrote:
| Resolved as of a minute ago, still having an issue now?
| unilynx wrote:
| digitalocean.com DNS (on cloudflare) is now resolving again.
| looks like several things are coming back now.
| jchw wrote:
| Something's wonky, because it's not _just_ Cloudflare. One of my
| personal sites is down that uses nothing but a VPS, and I noticed
| my Unifi AP disconnect from its controller a little bit ago.
| Fiber cut? Routing issues?
| parliament32 wrote:
| If that VPS is on DO they're down too cause of CF. Or if you
| set the resolver on your VPS to 1.1.1.1 that's also down.
| jchw wrote:
| Why are digital ocean VPSes down due to a Cloudflare outage?
| Hoping for a clarifying post mortem...
| unilynx wrote:
| digitaloceans VPSes weren't down, but there do seem to be
| routing issues as TransIP can't reach DigitalOcean AMS3
| (but it's all coming back now)
|
| Maybe the problem was somewhere on the AMS-IX
| drchiu wrote:
| My Digital Ocean load balancer went down. I think there's
| probably some internal routing? Would be interested to
| understand more.
| dpcx wrote:
| DO is still up as my machines are still up and accessible.
| Kye wrote:
| That would explain why Patreon is down. I was going to post a
| little frog I took a picture of on Lens. Went down just as I
| opened the app.
| maxioatic wrote:
| RIP someone's weekend
| emeraldd wrote:
| Looks like Digital Ocean is reporting an issue with their
| upstream provider:
|
| https://status.digitalocean.com/incidents/6wtmldty17g1
|
| As big as this is, any chance a major hub/backbone went down?
| brycewray wrote:
| Vercel also appears to be dropping out and coming back in
| intermittently over the last 30 minutes or so. Not aware they're
| using Cloudflare, although they do mention using AWS.
| davexunit wrote:
| Happy Friday, everyone!
| usr1106 wrote:
| It shows "Minor system outage" when I load the page, but it
| switches to "All systems operational" immediately. Same behaviour
| on several attempts.
| xen2xen1 wrote:
| Remind me to check and see that I have 8.8.8.8 and 1.1.1.1 on my
| networks, not just one or the other..
| kube-system wrote:
| Of course, I read this _after_ I spend an hour debugging some
| strange DNS issues.
| iamtheyammer wrote:
| Appears to be working for me now
|
| ; <<>> DiG 9.10.6 <<>> discordapp.com ;; global options: +cmd ;;
| Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:
| 8092 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0,
| ADDITIONAL: 1
|
| ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;;
| QUESTION SECTION: ;discordapp.com. IN A
|
| ;; ANSWER SECTION: discordapp.com. 140 IN A 162.159.135.233
| discordapp.com. 140 IN A 162.159.129.233 discordapp.com. 140 IN A
| 162.159.130.233 discordapp.com. 140 IN A 162.159.134.233
| discordapp.com. 140 IN A 162.159.133.233
|
| ;; Query time: 69 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN:
| Fri Jul 17 14:37:40 PDT 2020 ;; MSG SIZE rcvd: 137
| typingmonkey wrote:
| I was trying to fix my router the last 15 minutes :)
| ummonk wrote:
| Yeah I was waiting for it to fix then tried cell phone and
| realized that was down too. I assumed it was an issue regional
| / backbone routing or something. Especially cause status pages
| which I wouldn't expect to be hosted on AWS (because of the
| need for status pages to stay up when AWS goes down) seemed to
| also be down. Didn't realize it could be Cloudflare...
| arkitaip wrote:
| Same here. Only figured it out because just one of the
| computers uses Cloudflare dns and the others were fine...
| ghastmaster wrote:
| Ditto except visa versa. My machine is set to the router
| which uses cloudflare. Other machines use whatever is default
| for mac(I try not to touch those). Once I realized they were
| working and I could access internal network from outside, I
| started diagnosing DNS. Came here via 8.8.8.8.
| devy wrote:
| Looks like CF is up!
| hugoromano wrote:
| I can only see the dashboard down, all my sites with Cloudflare
| are up.
___________________________________________________________________
(page generated 2020-07-17 23:00 UTC)