[HN Gopher] East Coast Internet Outage
___________________________________________________________________
East Coast Internet Outage
Anyone else noticing a widespread internet outage on the East Coast
of the US?
Author : taf2
Score : 220 points
Date : 2021-01-26 16:35 UTC (6 hours ago)
| twistedpair wrote:
| Props to Slack, it's the only thing working well for me. GMail,
| GCP, Meet, all dragging and timing out in Boston.
| whatsmyusername wrote:
| Slack is spotty for us
| ixtli wrote:
| Verizon in NYC acting funny.
| Wistar wrote:
| On Fios in Seattle. Seems good here.
| nycben2 wrote:
| Verizon is having issues in NYC for sure.
| https://twitter.com/VerizonSupport/status/135411467509349580...
| u678u wrote:
| typical that link is useless verizon.com/outage. "learn how to
| reboot your router"
| 2dollars27cents wrote:
| I have no technical understanding of fiber networks. The
| explanation that this was caused by one fiber line in Brooklyn
| being severed seems unlikely. Curious to see what the experts
| think.
| kube-system wrote:
| I wouldn't doubt it. Single misconfigured routers have caused
| issues of similar scale.
| im3w1l wrote:
| I believe the internet is designed to handle broken links,
| but is not very robust against misconfigured (or malicious)
| routers.
| ArchOversight wrote:
| If a 100 Gbps link goes down, and all the other links that
| could carry that route are only 10 Gbps... it goes downhill
| quickly.
|
| Too much traffic is concentrated at the major carriers
| which carry a large amount of traffic over relatively
| little fiber (cause its cheaper) and when that fiber
| disappears we lose a massive amount of capacity...
| mikem170 wrote:
| Just to add to that, sometimes the redundancy doesn't work
| out as planned, maybe another weird set of circumstances,
| or some routers can't handle spikes in load as the network
| reconverges, causing other reconverges, confusing bgp
| routing peers, etc. Fun stuff!
| mlyle wrote:
| So, for clarity, it's never one fiber line being cut. It's a
| big massive bundle of fiber carrying different circuits for
| different carriers different places that were run in the same
| trench and all got cut at once.
|
| Then they get to dig it out better and someone gets to get down
| in the hole and figure out which piece of fiber needs to be
| spliced to which other one.
|
| Because of DWDM, some of those individual fibers may _each_ be
| carrying 64 channels that in turn each carry 400gbit /sec for
| different providers/services/etc.
| korethr wrote:
| And when it's a large bundle cut, it's going to take a bit
| before service is restored. I have been told that a highly
| skilled tech can splice a single strand in a matter of
| minutes. Looking it up, I find figures of an average of 30
| minutes per joint for 4-strand bundles, and slowing down from
| there. So if you have a 144-strand bundle to repair, or
| multiple 144-strand bundles, it's going to take several hours
| to effect the repair, and you're probably going to have to
| rotate through techs to avoid fatigue-induced errors slowing
| things down even more.
| fcsp wrote:
| Would it maybe be an idea to, going forward, color the
| cables individually before putting them into the ground?
| jzwinck wrote:
| Of course they are colored. Like this:
| http://www.evertopcomms.com/144-fiber-count-gyts-fiber-
| optic...
|
| It is still a lot of work, and so far not automated.
| srswtf123 wrote:
| You must be joking. Cables are Color-coded and labeled
| obsessively.
|
| You should take some time to understand the physical
| layer of infrastructure. There will always be work to do
| in meet-me-rooms!
| Scoundreller wrote:
| If they're all in use and it's point to point, might be
| easier connecting whatever to whatever. Then
| simultaneously swap line cards around as they get lit up
| or letting IP do its thing.
|
| Source: I have no idea.
| jeremyjh wrote:
| This. Even if you have redundant lines with multiple
| providers there is a good chance they are part of the same
| fiber bundle. It can be really difficult to get providers to
| admit who they are leasing from and what the physical runs
| are.
| mikem170 wrote:
| And then telco's may move leased fiber to other paths
| without telling customers, messing up the physical
| redundancy they thought they had...
| jorblumesea wrote:
| My naive explanation: fiber networks handle incredible loads,
| and a single fiber line might correspond to 10+ Tbs (terabytes
| per second). So all that data needs to go somewhere else, and
| if it gets shunted, packets get backed up, resulting in a
| cascading failure.
|
| Note: I am not a fiber technician nor do I work for a large
| ISP.
| neogodless wrote:
| While I'm also not qualified to diagnose the problem, I would
| imagine it's something like this...
|
| The major internet connections are like superhighways, with
| major central routers acting like exchanges. If a major
| superhighway had an accident blocking traffic, there would be
| personnel stationed at exchanges to help re-route the traffic,
| though that often occurs too late to prevent a bunch of
| backups. At any rate, people using Google Maps, Waze, etc.
| would see the accident and would likely be redirected to
| alternate routes that would normally be much slower - and would
| also be slowed down by the influx of higher than normal
| traffic.
|
| The fact that the internet still _worked_ albeit much slower
| than usual leads me to believe that something similar happened.
| While normally my internet traffic might go through a major hub
| in NYC and then onward to various carriers and hosts, etc. it
| had to be re-routed around that hub that is normally sub-
| optimal, and also ill-equipped to handle the spillover. (And in
| some cases, bandwidth was saturated, and failed altogether for
| some users for certain usage.)
| overkill28 wrote:
| The odd thing about this situation though is the geographic
| location of Brooklyn: on an island. Why would a significant
| amount of traffic from outside Long Island be going across
| the harbor to Brooklyn, especially given all the major
| switching facilities in Manhattan (closer to shore).
| betterunix2 wrote:
| What makes you think it is only connected to Manhattan?
| Long Island has submarine cables connected to New England,
| various points along the East Coast south of NYC, and
| several transatlantic cables.
| JimiofEden wrote:
| Philly suburbs: Fios has been inconsistent for me for the last
| hour or so, starting around 11:00
| Gene_Parmesan wrote:
| When half of my coworkers on a Zoom call (including myself) got
| kicked within ten seconds of each other, I figured something
| major had happened. Western PA. Was somewhere around 11:24 EST.
|
| Network never went down completely, bitrate just took a nosedive.
| Moogs wrote:
| Apparently, Verizon's outage page doesn't actually say if there's
| an outage. You have to log in to see if there's an outage. Too
| bad logins are failing after ~10 minutes for me.
|
| https://www.verizon.com/support/residential/service-outage
| NovemberWhiskey wrote:
| Yeah, I'm getting a 504 from their status site even after
| managing to get through the authn process.
|
| Although my packet loss to my IPv6 tunnel provider just went to
| zero, so that's a good sign here (Manhattan).
| rrrrrrrrrrrryan wrote:
| I'm nitpicking on the title, but "widespread" is one word, not
| two.
| adjkant wrote:
| Completely unable to load anything today from 11:30sh to 12:30ish
| EST in Brooklyn, all seems to be smooth now. First ever issue
| with FiOS in two years, would still fully recommend them to
| anyone when comparing to most options, especially Comcast.
| egberts wrote:
| I find it weirder that Zoom Verizon got BGP routed through
| AS42861 via 0.ae15.gw14.iad8.alter.net
|
| and that AS42861 isn't clearly defined under Roblex
| Animats wrote:
| Too much is failing from single point outages. The FCC may have
| to establish standards to prevent over-centralization.
|
| In the entire history of the Bell System, no electromechanical
| central office was ever down for more than 30 minutes for any
| reason than a natural disaster or, one occasion, a major file.
| That record has not been maintained in the Internet era.
| duxup wrote:
| >no electromechanical central office was ever down for more
| than 30 minutes for any reason than a natural disaster or, one
| occasion, a major file
|
| Are these systems even comparable? Today /= yesterday phone
| service....
| tenebrisalietum wrote:
| The Bell System's job was to operate a nationwide telephone
| system for voice, not high-speed data. Even then, you only had
| one path to the telephone CO and only one CO it could go to.
| Outages were rare because voice-grade wire is cheap and all the
| complexity is handled by the CO.
|
| Also, while my Internet may have outages from time to time, I
| can't recall the last time I couldn't use my cell phone other
| than being in a bad signal area. For voice, cellular is
| definitely the way to go as capacity is easily added by adding
| new towers; not something easy to do with landlines.
| dan-robertson wrote:
| What do you mean by saying the wire is cheap? Obviously you
| can't send much high frequency data down the wire but each
| wire could only be used for one call at a time for much of
| the company's history, so more wires were needed. What effect
| does that have on the cost? And surely the complexity is
| still in the CO with the internet, or am I missing something?
| tenebrisalietum wrote:
| > And surely the complexity is still in the CO with the
| internet
|
| Your cable modem (which has a CPU and does high-speed
| signaling) or your FIOS box is much more complicated than
| an analog phone.
|
| The carrier-grade router on the other end of your cable
| modem is really just a bigger version of routers you buy
| off the shelf or for enterprises. It's literally routing IP
| packets in the same way.
|
| Whereas you have to have a whole switching system at the
| other end of an analog voice CO. (Or an operator making
| physical connections if we're still in the 1800's). The
| best you can do for yourself in that world today is get an
| PBX system like an old NEC NEAX box or similar, but you're
| still not going to be on the same level as the CO.
| amarshall wrote:
| Natural disasters sometimes take out cellular service in
| large areas. Definitely happened for several days during
| Hurricane Sandy in NY.
| tenebrisalietum wrote:
| They can get temporary cellular devices to fill in on
| generator power pretty quickly. I know most if not all
| phone lines are underground, but something like a 2000-pair
| "wet pulp" repair (necessary in the first place due to
| negligence) takes a week. I don't know the name of these
| devices but something of that nature was deployed in
| Nashville on Dec 25 when the AT&T building was bombed and I
| think most cell service downtown was restored the next day.
| icedchai wrote:
| Most of the early internet piggy backed on the bell system:
| 56k, T1, T3 lines, frame relay, ISDN... and of course dialup.
| dylan604 wrote:
| >The FCC may have to establish standards to prevent over-
| centralization.
|
| Bwahahahahaha. Sorry, but that's like expecting the FCC to do
| something that would benefit US citizens vs US corporations. I
| think we've seen that's just not something they are interested
| in doing. Maybe a new administration will put as the head of
| the FCC someone that is "good of citizens" minded and uses a
| normal size coffee cup, but I'm not holding my breath for
| anything other than pro-corp rulings from the FCC
| Florin_Andrei wrote:
| > _Maybe a new administration will put as the head of the FCC
| someone that is "good of citizens" minded and uses a normal
| size coffee cup_
|
| Someone who is interested in anything beyond personal profit,
| status, and influence, would be nice for a change. Also
| someone who is smart would be very nice.
| Finnucane wrote:
| To be fair it's _slightly_ more likely with Ajit Pai out.
| Fej wrote:
| I am on Fios in northeast NJ, issues with some services (Netflix)
| but not others (Google except significant variance in ping times,
| Bing, AWS).
| arthurcolle wrote:
| I was just about to call FiOS to switch from Xfinity because
| these data caps are absolutely ridiculous. I am sad now.
|
| Aside: Comcast, wtf? 1.2 TB limit per month? Seriously if anyone
| works at Comcast know that I hate your company with an
| overwhelming passion.
| trimbo wrote:
| Unlimited data is $30 extra.
|
| https://www.xfinity.com/support/articles/exp-unlimited-data
| arthurcolle wrote:
| Sweet, I actually rent their modem so they said that they can
| give it to me for like $11 a month, just got it activated
| today. Thanks for the tip!
| luxuryballs wrote:
| FIOS is WAY better, they will fix their broken cable. I get (so
| far unlimited) 30gb up/down for $40/mo but it typically runs
| closer to 60 down 70 up. Fiber optic is baller. Though I do
| feel like they throttle me (web browsing becomes slow) if I am
| watching a streaming video service or mining crypto without a
| VPN on, could be an illusion but it always tends to clear up as
| soon as I activate the VPN.
| hnuser123456 wrote:
| 60-70 gigabits per second for $40/month? Guessing you mean
| megabits. Not worth being packet inspected. 1gbit on comcast
| seems to give 42 mbps upload, was 50 a few months ago.
| ~$80/mo
|
| edit: hitting 48-50 mbps again today, sweet
| ryan_j_naughton wrote:
| Comcast "gigabit" is a lie.
|
| I reliably get 900mb down and 900mb up with Verizon in DC.
| hnuser123456 wrote:
| The download isn't a lie and it's what they advertise. I
| was getting 800mbps before I updated my home router from
| a 2014 model to a 2020 one, now I hit 950-960. I wish the
| upload was symmetric, but I understand that's not always
| possible on copper, getting about 50 now. Not bad for a
| ~50-year-old apartment.
| vitus wrote:
| The problem isn't with copper, it's with the cost of
| building out amplifiers. It's a lot more cost effective
| to amplify downstream than upstream, especially when most
| use cases are heavily asymmetric.
|
| (I'd expect that only in the past year or so has upstream
| been a lot more heavily utilized, what with massively
| increased work-from-home due to the pandemic. Before
| then, the primary stressors of the network were things
| like streaming 4k60 video.
|
| And I'll be very surprised if any ISPs thought torrent
| uploads to be a use case worth optimizing for.)
| lotsofpulp wrote:
| Comcast upload is a farce. They don't bother stating it
| anywhere, I presume because they modify it as needed to
| satisfy their needs for providing TV channels or their
| "burst" download speeds.
| hnuser123456 wrote:
| It's usually 10% of your download it seems, apparently
| until you go past about 250 mbps.
| sitzkrieg wrote:
| its 3/5/10/35mbit up, but consistent. 35 is the docsis
| limit on their gigabit cable offering
| noja wrote:
| Which version of docsis is this? 2.0 from 2002?
| sitzkrieg wrote:
| i think so, but i had to upgrade to a docsis 3.0 modem,
| but told the physical infrastructure in place here cannot
| exceed 35mbit up. im not even going to pretend to know
| enough about cable to postulate how possible that is
| lotsofpulp wrote:
| They never advertise upload bandwidth anywhere, and it
| varies hugely when I'm at other people's houses that have
| cable internet. I assumed it's because they might
| provision 200mbps down / 10mbps up for a neighborhood,
| and then throttle you down as necessary.
| 2OEH8eoCRo0 wrote:
| I think the price dropped though. I was furious until I saw a
| lower bill. I also got a speed increase to 1.2gbps
|
| I still wish I had other options in my area (nh)
| SteveGerencser wrote:
| I love these posts. They remind me that there are places in the
| world where technology is moving forward. Where we live in
| Tennessee I have to run 2 internet connections to be able to
| have a service when I need it. 3gb DSL and a 5gb satellite
| connection. The sat connection has the additional bonus of
| being capped at 150Gb per month. All these new streaming
| services may as well not even exist for those of us in rural
| areas.
| rayiner wrote:
| I've got Comcast 2 gig fiber and Verizon 1 fig fiber to my
| house, plus 1 gig cable. I live 10 minutes from horse farms
| in MD. (The horse farms also have fiber.)
| lotsofpulp wrote:
| I've never seen Comcast offer fiber, except to businesses
| at astronomical costs. I can't even find it on their
| website. Do they advertise upload bandwidth for their
| fiber?
| rayiner wrote:
| https://www.xfinity.com/support/articles/requirements-to-
| run...
|
| It's actually 3g up and down. It's 10 gig Metro-E into a
| Juniper switch, with a 1g port and 10g port throttled to
| 2g.
| lotsofpulp wrote:
| That's awesome! How much is that, if you don't mind
| sharing?
|
| I wish they had it in more places.
| rayiner wrote:
| It's expensive, $300/month. I try to justify it to myself
| by noting it's on their business Metro-E network so it's
| completely reliable and has amazing support. Also an
| entire /48 of static IPv6. It was $150/month for the
| first two years, which was much more palatable.
|
| Their model is expensive because they don't wire while
| neighborhoods. For me they had to run fiber a quarter
| mile, down my street, and trench under my driveway. I
| imagine all these one-offs make the per house investment
| quite high.
| Paul-ish wrote:
| Did they charge you upfront for the installation?
| lotsofpulp wrote:
| That's not bad, I'd be willing to pay that and even more.
| I was simply unable to contact anyone at Comcast who was
| willing to entertain running fiber to a home. So I simply
| had to buy a home that had an option other than Comcast.
|
| In another instance, I needed symmetric fiber at one of
| my businesses. I contacted Comcast, the
| business/commercial departments multiple times, got
| numerous peoples' names, left voice and email messages
| left and right and got nowhere after weeks. I ended up
| contacting a different ISP, signed up with them, and then
| on the install date, Comcast showed up to install the
| fiber modem (who were, of course, reselling it to the ISP
| I contracted with). I ended up paying the other ISP
| $1,700 per month.
| cobookman wrote:
| They offer a 2Gibps fiber optic plan in certain locations
| for ~299$/month.
|
| https://todayamerican.medium.com/the-definitive-guide-on-
| how...
| lotsofpulp wrote:
| This section is an amazing indictment of the company that
| tens of millions of Americans have to deal with to get
| access to the internet.
|
| >The Important Part Because so many people had this
| question, and I finally talked to someone at Comcast who
| knew what Gigabit Pro actually is, here is how to get in
| touch with them:
| arthurcolle wrote:
| I'm pretty sure I have Comcast fiber, its 1Gbps down and
| around 750Mbps upload and there's a wire coming out of
| the wall into the router so... -\\_(tsu)_/-
| lotsofpulp wrote:
| When I bought a home, I limited my searches to ones that had
| symmetric fiber connections.
| SteveGerencser wrote:
| I priced having a 10m fiber connection brought out here. I
| was quoted just over $5k/month with a minimum 3 year
| contract. I assume they were expecting me to just pay for
| the new service to the entire area for them to resell.
|
| Wife wanted to live out here. So we traded my being able to
| work easily for amazing views and lots of land. Not
| entirely unhappy with the trade, but there are days when I
| think about just renting an office in town. (30 minutes
| away).
| xhrpost wrote:
| If you search HN you will find that negotiating plays a
| big role in the cost of leased line access.
| https://news.ycombinator.com/item?id=3892854
| lotsofpulp wrote:
| I was basically restricted to buying homes in new
| developments since 2010. Those all tended to have fiber,
| and anything before did not, and they had no plans to
| install it. I pay $65 per month for 940mbps up and down
| from Centurylink, and it's been as advertised so far.
|
| Unfortunately, the cable company usually offers some
| ludicrous 500mbps (burst) download / 5mbps upload
| connection that people are willing to accept for cheaper,
| so if the phone company comes in and pays to install
| fiber and charge accordingly for the new fiber runs,
| insufficient people would switch to the fiber to make the
| numbers work since they are okay with the shitty cable
| company connection.
| dudul wrote:
| I bought a house 1+ year ago and had "good internet"
| connection on my must-haves. I remember visiting a house
| during the search and asking the listing agent about it.
| The guy was super candid, he basically told me no, there
| was no cable internet and that was the reason he knew it
| would be sitting on the market for a year at least.
| arthurcolle wrote:
| Had to make this after seeing your post.
| https://i.imgflip.com/4vesal.jpg
| pbronez wrote:
| haha I've been dreaming about a 10GB LAN setup; getting
| that on WAN would be awesome. I'd immediately establish
| elaborate multi-site NAS systems.
| lotsofpulp wrote:
| Haha. I have a multi generational household. 4 adults, 2
| children, and we're heavy Facetime/Apple TV users. 4 to 8
| HD simultaneous video streams is normal, plus the
| security cameras. I also wanted to be able to stream from
| my server at home anywhere I am. And with 2 people
| working from home, it was an amazing decision.
| hnuser123456 wrote:
| 3 gigabits per second DSL and 5 gigabits per second to a
| satellite? You (like the other commenter...) probably mean
| megabits. And also Gb is gigabits, but your cap is probably
| in bytes, gB or GB.
| learc83 wrote:
| You should move down to Hamilton county. We have 1Gb fiber
| for around $60 from EPB (the municipal power company). Even
| out in the more rural areas.
| garettmd wrote:
| To be fair, I've had FiOS for more than a year now, and this is
| the first incident I've had with them. I'm pretty comfortable
| saying they have a much better reputation than Comcast...
| abstractbarista wrote:
| Wow, that's crazy. I use over 1TB a day. Forever thankful for
| Google Fiber.
| slivanes wrote:
| How does does one use over 1TB a day?
| arthurcolle wrote:
| Running scrapers can get you there pretty fast ;)
| frakkingcylons wrote:
| Wow 1TB per day? What is using all that bandwidth? (if you
| don't mind me asking)
| vondur wrote:
| I've had FIOS service since 2007 or so here in SoCal and I can
| count the outages over this period on one hand.
| tomjakubowski wrote:
| I was a long-term Fios customer in SoCal, and they were
| indeed very reliable up until Verizon sold the local service
| to Frontier. Glad to hear they're not a problem for everyone
| at least.
| [deleted]
| adrr wrote:
| A fiber cut can affect any ISP.
| betterunix2 wrote:
| Fios service is fantastic, and this morning's outage barely
| impacted me in Jersey City (only some disruption that seemed to
| be related to NYC area DCs more than Fios service itself). I
| have found their service to be very reliable for years now,
| with throughput that does not fluctuate (I get what I pay for
| regardless of the time of day) and lower latency than I had
| with Comcast. No regrets at all.
|
| The only real complaint I have is that Verizon still seems to
| be unable to deploy IPv6. They have promised it for years and
| never seem to get past the testing phase.
| ronnier wrote:
| What's worse is the 35 Mbps upload.
| atonse wrote:
| Honestly, at this point I've found HN to be the best "Status
| Page" for this sort of thing. I came here first when I found
| issues and fiddling around with my custom DNS lookup hierarchy
| didn't help.
|
| FiOS customer in DC Metro.
| MattSayar wrote:
| Also checking /r/sysadmin and sorting by new
| tyingq wrote:
| It would be interesting if someone could automate something to
| pick out HN outage discussions and make an HN driven
| "downdetector" page.
| xhrpost wrote:
| Except these posts seem to regularly get taken down or flagged.
| This isn't even on the front page anymore it seems.
| someHnUser wrote:
| I'm glad I'm not the only one noticing this. I could usually
| rely on HN to not be so heavily "filtered" for a lack of a
| better term but I have noticed the slide into a more
| controlled/gamed PR outlet.
| dang wrote:
| It set off the flamewar detector. We monitor that list and
| eventually cancel the penalty for threads that aren't actual
| flamewars, but it takes a while. If anyone wants to make it
| faster, emailing hn@ycombinator.com is usually a forcing
| function.
|
| Edit: Also, posts without URLs get a downweight by default.
| I've turned that off too, and merged the other threads into
| this one, since it was the first one.
| eMGm4D0zgUAVXc7 wrote:
| How does one programmatically detect a flamewar? :)
| dang wrote:
| By proxy, which is why there are false positives.
| eganist wrote:
| downdetector.com did well today. Front page showed a
| generalized web issue, and any one of the individual services'
| status pages showed it was geographically localized to the
| midatlantic and northeast US.
| dzmien wrote:
| I am experiencing connectivity issues on Fios outside of
| Philadelphia in Pennsylvania.
|
| edit: as of 13:20 (GMT-5) it appears to be resolved.
| sleepydog wrote:
| It seems to have recovered for me. I am using FIOS in Brooklyn.
| gedaxiang wrote:
| Same! Also FIOS in BK
| squeaky-clean wrote:
| Same, also fios in Brooklyn. Not getting my full speeds but I
| am getting 100Mbps, which is far better than the 60 Kbps I was
| getting.
|
| Ironically things recovered within 30 seconds me of messaging
| our systems team to ask if the development VPN was having an
| outage. That's like a magic rule of internet outages, isn't it?
| Nothing works until you try to show someone else the issue,
| then it all works.
| 813919 wrote:
| Fios outside Baltimore and yeah it's screwed currently.
| lacraig2 wrote:
| I have FiOS and my town is listed here. Internet is still
| working.
| Ansil849 wrote:
| Why is this post not showing up on the HN homepage?
| [deleted]
| lxgr wrote:
| Maybe affected users can't upvote and not affected users don't
| care enough?
| dang wrote:
| https://news.ycombinator.com/item?id=25921688
| wronglebowski wrote:
| FIOS Here in the Philadelphia area, having issues hitting AWS as
| well.
| jgrahamc wrote:
| Looks like it's Verizon having trouble.
| silveira wrote:
| Spikes everywhere https://downdetector.com/
| dx87 wrote:
| Yep, on verizon fios just outside baltimore. Can't ping 8.8.8.8,
| but can ping 1.1.1.1 fine. Also having issues connecting to
| services running in AWS.
| mikesapienza wrote:
| Yes. Pittsburgh area is spotty with FiOS right now.
| [deleted]
| sleepysysadmin wrote:
| >There is a fiber cut in Brooklyn. We have no ETR, as of yet.
|
| A fiber cut auto heals automatically via bgp. Absolutely no truth
| in this claim from verizon.
|
| Meanwhile google, zoom, aws, and microsoft went down?
|
| There isn't any shared datacenters here. There's at least 3
| distinct datacenters.
|
| So what's the truth? what actually went down?
| jasonjayr wrote:
| Traffic is passing, although slowly.
|
| I think this revealed that there is insufficient failover
| capacity.
| mikem170 wrote:
| Could be that. Also could be too much complexity, as opposed
| to capacity. Could have been timing didn't help, sometimes
| you take calculated risks or have to deal with additional
| complexity during migrations, etc.
|
| I've even seen where playing it safe while planning to back
| out of a manual fix used during a weird outage (circuit went
| down in one direction or something, just needed to be
| manually shutdown to re-route everything) brought down the
| entire network when another outage happened before that
| evening, i.e. most redundancy plans protect against single
| failures, not all combination of two failures. They wanted to
| do all planned changes after business hours, like manually
| swinging traffic back to the primary hub site. They didn't
| check the weather on the other side of the country, and the
| power at their new super-duper data center went down that
| afternoon during a thunderstorm...
|
| I wonder what percentage of fiber cuts don't cause noticeable
| outages. We'll never know, unless they let us root around in
| their ticket system.
|
| As a customer it is fair to judge them on what you see,
| though. Some telcos screw up more than others. Best to use
| two.
|
| (Typed over my phone, cuz my broadband is down!)
| ArchOversight wrote:
| If 100 Gbps of capacity suddenly went away, then BGP attempted
| to route around it, and sent that traffic down 10 Gbps routers
| instead, which overloaded, or were not ready for the deluge of
| traffic, then those BGP routes could start flapping, at which
| point those routes would get pulled, forcing it onto yet other
| circuits, which may be 1 Gbps and that's how the outage
| continues to grow.
|
| Too much capacity is centralized in certain providers, and if
| they go down there simply is no backup route that can handle
| the same traffic patterns.
| mikem170 wrote:
| Yep, and it's possible that running low on ipv4 address space
| exacerbated the situation, so many more small routes than
| there used to be, maybe not all getting summarized during an
| outage they way they used to be, causing problems on peer bgp
| routers like you said.
| qeternity wrote:
| Uh presuming that the connection is multihomed which is not the
| case for residential connections in NY.
| ASalazarMX wrote:
| It might sound conspiratorial, but when the WTC was attacked in
| 2001, the Internet slowed down a few days after. We'll probably
| never know, but I assumed it was deeper surveillance being
| implemented.
|
| Every time an outage like this happens without a following
| postmortem, I wonder if it was a surveillance oopsie.
| [deleted]
| kridsdale1 wrote:
| Why not take the simpler interpretation that millions of
| dollars of equipment was in the buildings that were
| destroyed, and it took a while for the network's self-healing
| algorithms to discover a new steady-state configuration that
| approximated the performance of what was suddenly lost?
| ASalazarMX wrote:
| Because it happened a few days after, not immediately, and
| there was nothing on the news about that. It was working
| fine immediately after, although people then mostly got
| their news from TV; a major event didn't increase bandwidth
| usage as drastically as today.
| FlyMoreRockets wrote:
| Occam's razor: when presented with competing hypotheses
| about the same prediction, one should select the solution
| with the fewest assumptions.
|
| https://en.wikipedia.org/wiki/Occam%27s_razor
|
| Why is it that conspiracy theories seem to always require a
| more complex answer than the truth?
| midasuni wrote:
| Conspiracy theories are more fun to read.
|
| One thing they always fall down is the idea that
| government bureaucracy is so quick and competent that
| they could do something like installing mass surveillance
| at a moments notice while the country is in chaos without
| it leaking due the next 20 years
| ASalazarMX wrote:
| Occam's razor is a guide to start with the simpler
| explanation. If the simpler explanation is not
| sufficient, you add more elements until it is.
|
| My point is, the Internet rerouting around damage doesn't
| explain the slowdown, mostly because it happened _days
| after_ , not immediately, meaning the damaged
| infrastructure was not critical for the global internet.
|
| The attacks unleashed a huge intelligence operation, and
| it's reasonable to expect a parallel intelligence
| operation online. If the latter caused a traffic
| bottleneck or not, we can't know because it was never
| addressed, it's speculation on my part.
| imwally wrote:
| Yes, sites are loading very slowly, if at all. Thought it was AWS
| at first but Google services are having the same issue.
| mikesapienza wrote:
| Pittsburgh area has spotty FiOS internet service also.
| TimC123456 wrote:
| Yes, lots of issues connecting to various cloud services here in
| DC.
| TimC123456 wrote:
| On Verizon FIOS here, perhaps this is another Verizon outage?
| bikenaga wrote:
| Comcast seems to be fine (at around 11:50 a.m. EST), so maybe
| it's just Verizon.
| tlunter wrote:
| Experiencing it from Verizon ISP. Any others that are also
| experiencing this not on Verizon?
| imwally wrote:
| On Fios here.
| zenexer wrote:
| Yes x2--Verizon Fios in at least two different states.
| evanc123 wrote:
| yep, i'm in DC area and having issues across multiple sites. Fios
| Fiber here
| beckingz wrote:
| Same thing here on the MD side on Fios.
| beachwood23 wrote:
| I'm in Philly, and I'm having issues with Comcast.
|
| Doesn't look like it's isolated to any one provider, or any one
| site. https://downdetector.com is showing many issues across many
| platforms.
| [deleted]
| zenexer wrote:
| Observing the same issue from various East Coast locations, all
| on Verizon Fios.
| 310260 wrote:
| No problems in Jersey City all day. Bleeping Computer article
| cites someone as saying it's a specific AS that's affected. JC
| routes through an AS in Newark, NJ and not NYC so, for me at
| least, there hasn't been an outage.
| betterunix2 wrote:
| I saw some disruptions in services that people in my household
| rely on, but I think that is because of disruptions in DCs
| located in NYC and not FiOS itself.
| crowman wrote:
| Yep. Second day of classes at George Mason University in Fairfax
| and no one can connect to Zoom or Blackboard meetings. I'm on
| FiOS as well.
| taf2 wrote:
| Getting reports that it's improving. It's nice that it never
| actually went down down... just degraded
| PStamatiou wrote:
| Same thing in NYC. on Verizon FiOS fiber
| phenkdo wrote:
| boston fios here, having intermittent issues, mostly slow
| connection.
| couchand wrote:
| For me, mtr reports quick pings (single digit ms mostly), but
| HTTP traffic is very spotty. Brooklyn FiOS.
| xhrpost wrote:
| Verizon FiOS. 50% packet loss with many "good" packets in the
| hundreds of ms when pinging Google. NYC.
| chockablock wrote:
| Connecting to a full-tunnel VPN seems to fix this for me. (FiOS
| in Brooklyn).
| boleary-gl wrote:
| Yes! Thank you for this (in hindsight, obvious) idea.
| woohoo7676 wrote:
| Thank you! This also fixed my slowdowns.
| dave_aiello wrote:
| Right now in my part of Bucks County, PA, FiOS' gigabit speed
| internet is running about normal speed for download and about 3/4
| of normal for upload.
| sweetheart wrote:
| Bucks County represent!
| neogodless wrote:
| I was getting 2 Mbps upstream on FiOS just after noon today!
| But it seems largely back to normal now. (I have 200/200, not
| Gigabit.)
| willcipriano wrote:
| Small world, I'm in the same place. I noticed long periods of
| dropped packets, it looks like existing connections are ok but
| establishing a new one is a problem. My wife working over a VPN
| doesn't notice a issue.
| cshift wrote:
| Spotify is loading weird for me (New England).
| Jonnax wrote:
| The FCC said that 3mbps upload speed is all internet users need.
| They might not be in tune with modern internet usage
|
| https://www.theverge.com/2021/1/21/22242635/fcc-ajit-pai-int...
| Brendinooo wrote:
| > "We find that the current speed benchmark of 25/3 Mbps
| remains an appropriate measure by which to assess whether a
| fixed service is providing advanced telecommunications
| capability," the report reads.
|
| Not "3mbps upload speed is all internet users need".
| lotsofpulp wrote:
| That's effectively the same thing, otherwise I don't see the
| purpose of that statement.
| cure wrote:
| To be fair, that was just Ajit Pai, who also thought net
| neutrality was not important. Now that he's gone, I hope he
| gets subjected to 3mbps upload speeds for the rest of his days.
| birdyrooster wrote:
| Well Ajit Pai only has 640K of memory so it's understandable.
| dang wrote:
| We detached this subthread from
| https://news.ycombinator.com/item?id=25920055.
| richwater wrote:
| You vastly underestimate the cost to provide broadband to rural
| communities.
| mroche wrote:
| https://twitter.com/VerizonSupport/status/135410988957298278...
|
| _There is a fiber cut in Brooklyn. We have no ETR, as of yet.
| You can use the MY Fios app for updates. *EAG_
| WarOnPrivacy wrote:
| https://www.datacenterdynamics.com/en/news/brooklyn-fiber-cu...
| heatmiser wrote:
| Does a cut cable in BK account for outages up and down the east
| coast?
| Spooky23 wrote:
| Verizon had (and apparently still has) a few vulnerable links
| in the NYC metro area that have wide ranging impacts.
| mroche wrote:
| I'm not knowledgeable enough to answer that question.
| However, this is the only Twitter comment that mentions this
| on their page, and I don't have access to a Verizon account
| to verify their outage status memo. We had a flurry of
| employees saying their connections were going haywire to the
| business and this is the only thing I've been able to gather
| outside of downdetector.
| jasonjayr wrote:
| This is speculation, of course, but: there are _several_
| large Internet Exchanges in NYC, and US /EU peering points.
| It's probable that a fiber cut could have cut several other
| links, and the failover links are now completely overwhelmed.
| jonknee wrote:
| If it's a fiber that goes into major data centers it
| certainly could.
| asmr wrote:
| Issues with FiOS in the Boston area. Traceroute seems to point to
| an issue originating in NY but that's just speculation.
| Apparently a lot of other services are down. Might be a Tier 1 or
| 2 ISP.
| vrtx0 wrote:
| Same here. Seeing heavy packet loss to most (but not all) sites
| that route through verizondigitalmedia.com.customer.alter.net
| in NYC.
| excitednumber wrote:
| Restored for me (FIOS Brooklyn, NY)
| gen220 wrote:
| My corporate laptop is affected. My roommate's corporate laptop
| is totally unaffected. We're on the same router/gateway.
|
| Any theories out there to explain this phenomenon? Fascinating.
|
| (My theory is a network partition, where my roommate's IP
| addresses happen to be in regions that are on "our side" of the
| partition, whereas my IP Addresses are on the "other",
| unreachable side).
| dx87 wrote:
| Are you using different DNS servers? My work laptop was using
| 8.8.8.8 and is getting name resolution errors, but my personal
| computer is using 1.1.1.1 and working fine.
| gen220 wrote:
| Interestingly enough, they're both deferring DNS to our
| router, and the router is configured to use DNS servers
| dynamically provided by the ISP.
| Here_For_LOLs wrote:
| its most likely your companies RDP infrastructure (or local
| environment) operates on AWS. Which is down because of this
| "cut"
| [deleted]
| Steltek wrote:
| I didn't do any digging but it seems like once you have a
| connection established, it's not too bad. Is your roommate on a
| full-tunnel VPN?
| chockablock wrote:
| Thanks for the tip; connecting to a full-tunnel VPN seems to
| fix this for me. (FiOS in Brooklyn).
| gen220 wrote:
| They weren't on a VPN, but my guess is that they got IP
| addresses from their DNS requests that were on our side of
| the partition.
|
| In any case, it looks like it's ended for us. The window for
| gathering evidence slides closed...
|
| It'll be interesting to read the post mortem on this one.
| hpkuarg wrote:
| Is your roommate on a VPN? It looks like Verizon is having
| trouble only with traffic to certain destinations (perhaps
| Google and AWS).
| gen220 wrote:
| They're not. I'm on a split VPN, for what it's worth.
| Disconnecting even from the split VPN does not help my
| laptop.
| HenryNY2020 wrote:
| Got knock out from Google Meeting a lot recently, even online
| speedtest show normal.
| excitednumber wrote:
| Restored for me (FIOS Brookly NY). Someone get that tech's venmo
| handle. Human deserves $1.
| excitednumber wrote:
| Not to say their time is only worth $1. Just to emphasize my
| thanks. Anyway. Please do not skewer me. :(
| podiki wrote:
| Issues here too, I'm on Verizon Fios. Right when starting class,
| what a mess. (As a colleague joked, Google going down (or Verizon
| I guess) is like the modern snow day)
| aaomidi wrote:
| Fios as well.
| nwsm wrote:
| We have FiOS in Boston and noticed it. Nothing terrible but for
| ~15 seconds at a time, recurring sporadically, we lost
| connection. Happened for a few hours, maybe 10am-1pm EST.
|
| Based on other comments we probably just didn't notice the big
| actual outage that happened all at once.
| willswire wrote:
| Noticing degradation in connection quality in Boston, MA. ISP is
| Verizon FIOS - operating on the 1 Gbps plan.
| jzer0cool wrote:
| When houses are created there are building codes to adhere to.
| For example, when there is a water pipe behind the walls a
| plumber (a good one) would also put a plate in front of it. In
| this way, once the walls are in place, drilling away into the
| wall, or nailing, would hit the metal plate, thus provides some
| safety mechanism. As another example, underground, in public
| areas are markings (or at least procedures) to identify such
| areas gas pipe areas before excavating.
|
| How are the fiber optics being cut? Are there no procedures for
| this or is there some challenges how the infra is setup today?
| bombcar wrote:
| Old Joke: How do you get rescued from a desert island if all
| you have is some networking gear?
|
| Simple! Bury a network cable and wait for the backhoe to show
| up.
|
| Backhoes are VERY strong and will easily go through gas pipe
| let alone anything that might protect fiber. And if the fiber
| is not where the plans say it is, even calling 811 won't save
| you.
| XorNot wrote:
| The vast majority of fiber optic cuts are someone digging
| through the cable underground.
|
| You can't realistically prevent this with additional armoring,
| because it's quite a different failure to someone drilling in
| the wrong place - the forces are a lot higher, or heavy
| machinery is involved etc. (see also: the rate at which workers
| go through water mains in footpaths).
| valbaca wrote:
| Scale: wrapping all fiber optic cables in a protective metal
| sheath would be cost prohibitive. Whereas adding a metal sheet
| to prevent a house from flooding or someone from getting
| electrocuted makes sense.
|
| One idea, borrowed from some chainsaw safety pants, would be to
| wrap the cable in a fiber web sheath that tangles in whatever
| cutting equipment
|
| https://www.youtube.com/watch?v=r5VSivQe760
| jzer0cool wrote:
| Literally took off the pants testing that! I'm unsure how
| much is worth getting paid to run a self-test. This product
| looks pretty neat. Half joking, anyone ever, or witnessed
| getting eletrocuted trying to get the internet / computer
| working again?
|
| This spun threads reminds me of nature. Spiders, silk works
| spinning web. Wonder if that's where the inspiration came
| from.
| vlan0 wrote:
| Lots of ways, unfortunately. But generally a quality network
| will be built to have physical redundancy. Yet, the ISP game at
| the top is all about cost cutting. So we'll never get what we
| paid for back in the 90s.
| betterunix2 wrote:
| Construction crews need to pay attention to what is below them
| before they start digging. A backhoe is going to defeat your
| cable every time.
| carewornalien wrote:
| Yes, large carriers can't do much when the backhoe strikes.. it
| happens.. they'll fix it.. and we'll all go on with our lives..
| t0mbstone wrote:
| I thought the whole point of the internet and protocols like
| TCP/IP is that it's supposed to be fault tolerant and traffic
| is supposed to automatically re-route. Seems kind of weird how
| a single cable being cut can cause half the country to lose
| internet...
| tenacious_tuna wrote:
| I wonder how much has to do with the cost-optimization
| pressure on a private carrier. Building robust infrastructure
| is only valuable to them insofar as it enables them to
| continue to earn money. Reliable service for the sake of
| reliable service--especially to residential regions--is a
| non-starter.
|
| Less pessimistically, it's possible that the regions that're
| affected have natural "choke points" for infrastructure,
| either moving through the mountains or along highways, etc.,
| which of course leads to a single-point-of-failure, though it
| may be unlikely.
| centimeter wrote:
| TCP doesn't know anything about route failure or rerouting.
| All it does is handle a few dropped or reordered packets here
| and there.
|
| In principle, you can have systems that very quickly route
| around damage like this, but in practice it happens so rarely
| that it's not worth it for consumer ISPs. I think it should
| exist, but the ISP space is so burdened with red tape that
| the only way anyone could compete with it was literally to
| start a rocket ship company and launch a bunch of satellites.
| pbronez wrote:
| Starting an ISP isn't all THAT hard.
|
| https://arstechnica.com/information-
| technology/2021/01/jared...
| betterunix2 wrote:
| The Internet is not as redundant as you might think, and in
| fact there is less spare capacity to go around as a result of
| the pandemic (a lot more time spent streaming and on video
| chats by a larger number of people). Internet routing is also
| non-trivial (convergence has not been reached in years), and
| when a major link goes down it can take time for routes to
| stabilize. Also keep in mind that the Internet itself was not
| down for most people who were affected by this; rather,
| connectivity between users and various services was affected.
| So people could have continued torrenting, but their Zoom
| sessions would get dropped because connectivity to some data
| center was impacted.
| zamadatix wrote:
| The ability for the internet to reroute after N path failures
| is there but someone has to pay for N paths worth of unused
| network to be ready for it. Most would rather just pay low
| cost and utilize all the bandwidth that can get them 364/365
| days a year. For others (usually businesses) there are more
| expensive options.
___________________________________________________________________
(page generated 2021-01-26 23:00 UTC)