[HN Gopher] Rogers network outage across Canada hits banks, busi...
___________________________________________________________________
Rogers network outage across Canada hits banks, businesses and
consumers
Author : cupofpython
Score : 309 points
Date : 2022-07-08 20:36 UTC (2 hours ago)
(HTM) web link (www.reuters.com)
(TXT) w3m dump (www.reuters.com)
| ilrwbwrkhv wrote:
| https://blog.cloudflare.com/cloudflares-view-of-the-rogers-c...
|
| Trust an American company to come up with a clearer picture of
| what's going on, highlighting the mediocrity of Canadian
| businesses.
| goatcode wrote:
| Having formerly lived in Canada for 30 years, I can concur.
| 542458 wrote:
| It feels like there's been a rash of companies messing up BGP
| stuff lately - Facebook, cloudflare, and now this.
| kache_ wrote:
| At what point does the anti-competitive nature and monopolistic
| qualities of Canadian telecoms become a national security threat?
|
| Maybe when it destroys the whole Canadian economy for a day
| dennis_jeeves1 wrote:
| >Maybe when it destroys the whole Canadian economy for a day
|
| Won't happen. The average Canadians are one of the most sheep
| like, docile people out there.
| drpgq wrote:
| I was hoping there would be a discussion of this here with some
| inside baseball, but then HN was down.
| sophacles wrote:
| The Cloudflare blog suggests it's BGP problems:
| https://blog.cloudflare.com/cloudflares-view-of-the-rogers-c...
| tlss wrote:
| It's always BGP......
| cperciva wrote:
| Sometimes it's DNS...
| RandomBK wrote:
| Unfortunately BGP problems make DNS problems look trivial
| :(
| openthc wrote:
| Should just merge this all into systemd-bgp-dns; kill all
| the birds with one stone.
| YarickR2 wrote:
| Well, Pottering quit RedHat to work for MS, maybe that's
| exactly the plan
| ncann wrote:
| I was so confused when I couldn't access HN - I thought to
| myself maybe HN was hosted on Rogers network or something.
| kelseydh wrote:
| Ditto, I expected to this be front-page on HN with many
| upvotes.
| bob1029 wrote:
| I used to work in a debit processing command center where we
| would monitor all of the acquirer and issuer links in real-time.
| The amount of paperwork and phone calling required just for a <60
| second MPLS VPN drop-out was pretty incredible. I can't imagine a
| day like this in that room. Would have been an absolute circus.
| brandon272 wrote:
| I see a lot of people noting the lack of communication and
| updates from Rogers.
|
| To me it would be shocking if they _were_ providing updates.
| Canada's telecommunication monopolies are not known for their
| customer service and, even worse, they seem to have internal
| corporate cultures of entitlement and arrogance that drive an "F
| U" attitude in general when it comes to external communications
| and accountability. So you won't see a post mortem. You won't see
| timely updates. You certainly will never see some kind of status
| page!
|
| I'm sure Rogers will be forced to do a post-mortem for large
| commercial clients where contractually required, but I highly
| doubt we will hear a word from them about the cause of this or
| what was/is being done to rectify it. They probably have an ETA
| but feel no obligation to share it.
|
| Rogers isn't alone. I would expect the same behaviour from Bell
| or Telus. Canada has serious issues when it comes to its big
| telco carriers and it represents major risk to the Canadian
| economy.
| 37 wrote:
| I see so many people get these two conflated, across the board,
| any time something like this happens. Some big leak or outage
| or whatever.
|
| The shocking part is not that they are giving no updates.
| Everyone is to expect that. The shocking part is that we
| (collectively) don't care that they give no updates, and let
| them get away with it.
| [deleted]
| lriel wrote:
| Agreed, although, Rogers has had notably more vaguely explained
| outages. With this, if the CRTC does not revise their decision
| to let Rogers acquire Shaw (CRTC 2022-76), then, as Canadians,
| we must expect to continue to overpay for shotty telecoms. But
| hey, there's maple sirup and affordable health care.
| whynotmaybe wrote:
| It's not possible to pay with debit card in Canada because
| Interac is down, because Rogers is down. Many atm across the
| country aren't working.
|
| It's not just Rogers' client who are impacted, the whole
| banking ecosystem is impacted.
|
| Even rumors that Costco can only accept cash, the Capital One
| Mastercard network might be down.
|
| I don't see how Rogers could keep quiet on this, unless it's
| from some malicious actor.
| acchow wrote:
| > it represents major risk to the Canadian economy.
|
| I actually think this outage will show that there's no major
| risk to the economy and every Canadian can just take every
| Friday off with no ill effects.
| Thaxll wrote:
| Sounds like a BGP problem where people needs to get onsite to fix
| the issue but since they never did the drill they can't connect
| etc ...
| tomComb wrote:
| People in other countries probably don't understand just how
| powerful - and depended upon - the big telecoms are here in
| Canada.
|
| They have no presence outside the country - they are so bloated
| and inefficient they couldn't compete - but they are omnipresent
| here.
|
| There are lots of people who depend on Rogers for home Internet,
| home phone, mobile phone, TV, and home security, not to mention
| business services that consumers also depend on, like the payment
| networks that are down.
|
| Plus their media properties, which probably come in useful when
| the government starts to think about allowing more competition or
| decreasing the taxpayer money being funneled to them.
| Johnny555 wrote:
| I think people in the USA understand bloated telecoms with
| monopoly power that means they don't need to compete.
|
| In my neighborhood, I have one effective ISP, that is the cable
| company (who offers phone+internet+cable), the other is an old-
| school phone company who offers DSL (with bonded DSL, up to
| 12mbit). I thought that I could get a slow DSL connection to
| back up my cable, but no, there's some neighborhood line
| concentrator that means I can't get DSL at any speed.
|
| So I really have only one ISP to "choose" from. (well, in
| theory I could get Starlink, but my area is still waitlisted,
| as is Verizon 5G home internet)
| edgefield wrote:
| Having lived in Canada and the US, with telecom plans with
| providers in both countries, I think Canada's telecoms take
| it to a new level. To give you a sense of how distorted the
| market is, it was cheaper for me to keep my US Verizon
| wireless plan than to use Rogers, Fido, etc. when I lived in
| Canada!
| jamal-kumar wrote:
| I do this.
|
| Unlimited (international!) roaming and everything isn't
| that expensive (Only about 100$) compared to whatever
| insane amount I'd be paying if I had some ripoff, under 5
| gigabyte or whatever pittance data capped Canadian phone
| plan for no reason. I also maintain a Central American
| phone plan that doesn't cost me very much either (2$ every
| 3 months just to keep the number outside of the country and
| receive SMS/use chat apps basically, but I don't do roaming
| when out of that region - When in the region, I spend maybe
| 20$ a month).
|
| Since the Rogers website is down (Great sales strategy!) I
| can refer anyone curious to a portion of their menu from
| some image I found; [1] 95$CAD (500MB until you hit
| overage) + 60$ for 'roam like home' (no mention of data,
| canadians still use SMS way too widely), and then probably
| more to up that data cap with no guarantee that applies to
| roaming? Yeah, no thanks! Oh yeah, the cops just install
| malware on your shit with reckless abandon, too. [2] I see
| no reason to consider Canadian nationality as anything but
| a nice passport at this point.
|
| [1] https://cdn.mobilesyrup.com/wp-
| content/uploads/2018/05/roger...
|
| [2] https://www.theguardian.com/world/2022/jul/07/canada-
| police-...
| throw7 wrote:
| I (america) also have only one internet provider to choose
| from (Spectrum TV, formerly Time Warner Cable). It's really
| annoying as just barely a mile away, my neighbor has a choice
| to use Verizon DSL (and does). -.-
|
| I dropped TV and a landline a long time ago, and use cell
| (google-fi/t-mobile).
|
| The annoying thing is all the spam mail I get from spectrum
| to bundle all the above for much lower than I'm paying now
| for an introductory rate. I don't want to play that game but
| it kills me what I could be saving for at least 6 months to a
| year.
| dylan604 wrote:
| Spectrum is so incompetent that even if you were a Spectrum
| subscriber, they'd still spam you with those offers.
| amscanne wrote:
| I have a lot of experience in both places. Americans may
| _think_ think they do, but I don't think so. As bad as
| Comcast and others are in the USA, I have found them to be
| far more efficient and competitive than Rogers or Bell in
| Canada.
|
| Same for banks in Canada. It's unbelievable what they do and
| get away with.
|
| And airlines. Rather, Airline.
|
| I believe there's a cultural element here: Canadians are more
| risk adverse and place a higher value on "established"
| companies and brands. This influences consumer behavior and
| legislation (Canadians don't seem to care about laws that
| make it hard or impossible for market entrants).
| walrus01 wrote:
| > Same for banks in Canada. It's unbelievable what they do
| and get away with.
|
| Canadian banks are so bad that they actually make the
| service and fees with a USD checking account with all the
| normal features at _Wells Fargo_ look good.
| jonahrd wrote:
| I'm a US citizen, Canadian permanent resident.
|
| I think it has to do with the fact that historically much
| of Canada's lands/provinces were literally just land owned
| by very massive British crown corporations. Massive
| corporations are baked into the history of Canada as a
| political entity.
| hackbinary wrote:
| No. "Crown Land" in Canada is owned by the provinces.
|
| There is no such thing as a "British Crown Corporation."
|
| Crown Corporations are a Canadian thing.
|
| I lived in Canada for 30 years and now live in the
| Scotland. CalMac and Scottish Water are not "Crown
| Corporations," but are nevertheless owned and controlled
| by the Scottish Ministers. This is contrast to BC
| Ferries, ICBC, and BC Hydro which are Crown Corporations.
|
| In the UK there is something called the Crown Estate,
| which is somethint again different.
| ShroudedNight wrote:
| I expect gp was referring to The Hudson's Bay Company /
| Rupert's land
| II2II wrote:
| I suspect that was a reference to the Hudson's Bay
| Company. I would have to brush up on my history, but they
| effectively controlled a huge tract of land pre-
| Confederation and continued to serve many smaller towns
| until well into the 20th century. I don't think the
| Hudson's Bay Company was considered a crown corporation,
| but they were granted a royal charter.
| hackbinary wrote:
| Yes, HBC has a Royal Charter from King Charles II which
| granted HBC exclusive economic activity in the lands that
| drained into Hudson Bay.
|
| HBC lands were surrendered to the British government in
| 1868 ahead of confederation.
| skeeter2020 wrote:
| >> No. "Crown Land" in Canada is owned by the provinces.
|
| Crown land is owned by the Canadian Crown; the monarchy
| owns all crown land officially. It's administered by a
| split across federal and provincial jurisdictions.
| ipaddr wrote:
| In Canadila about half of the crown land is federal and
| the other half provincial. In 2013 11% was private, 41%
| federal crown and 46 provincial crown
| kelseydh wrote:
| You're at a National Park. In Canada, it's crown land. In
| America, it's your park. The Canadian mentality is
| different.
| hackbinary wrote:
| National Parks are not Crown Land. National Parks are
| reserves of land owned and managed by the Federal Parks
| Department and development is not allowed on that land
| whatsoever.
|
| Crown Land is land owned and administered by the
| provinces. Crown Land can be licenced for many uses and
| sometimes it can be purchased.
|
| Also, Provincial Parks are not Crown Land.
| dredmorbius wrote:
| New Zealand often seems like a toy country with one of
| everything. There's the One Bank, the One Telecom, the One
| Airline.
|
| Not ... exactly. But effectively, yes.
|
| A combination of a tiny population (5.1 millions) and a
| long way from nowhere (roughly 3,200 km / 2,000 mi from
| Sydney).
|
| Australia and Canada already have issues from their own
| small scale (and vast land areas). NZ takes it up a notch.
| Scoundreller wrote:
| At least the Canadian banks and airlines have international
| operations.
|
| TD has more US than Canadian branches.
|
| But Rogers, Bell, Telus? Nobody has heard of them ex-Canada
| unless they're an ex-resident with bad memories.
| amscanne wrote:
| TD Bank (and Scotiabank in Latin America) operate as
| independent entities outside of Canada -- those American
| TD Bank branches are branches for a different bank (TD
| Bank USA). In fact, I believe that TD Bank (Canada)
| simply acquired a US bank rebranded it. I believe they
| have basically nothing in common operationally.
| lstamour wrote:
| Half right. Yes, TD has a lot of divisions, each
| separate: https://www.td.com/about-tdbfg/corporate-
| information/corpora...
|
| But it's also true that over the years, TD has had to
| merge systems and make changes. They've been operating in
| the US for about two decades now. So they've had time to
| merge and continue growing by merging companies in the
| US.
|
| It's true though. They aren't entirely integrated across
| borders the way other companies might be. The plans and
| cards they offer in the US are a much better value than
| those in Canada, too.
| thematrixturtle wrote:
| Oddly enough, Telus is a medium-sized but well regarded
| player in the fiercely competitive market for outsourced
| call centres in the Philippines.
| themitigating wrote:
| In Jersey City I can get xfinity (cable) Verizon 5g, or
| Fios 1gbit unmetered for $65 a month
|
| It's probably the density and income levels that make it
| worth putting out the infrastructure
| SECProto wrote:
| > And airlines. Rather, Airline.
|
| I would dispute this one, only because in addition to Air
| Canada (fleet size 312), Westjet _does_ exist (fleet size
| 162). And the other regional airlines offer (-ed,
| prepandemic) some price /service competition. Comparisons
| to the US aren't as useful with airlines as with some other
| industries due to population differences.
| rapind wrote:
| If you asked 100 Canadians if they'd prefer more
| competition to the telcos, 97 would say yes (1 works for
| Bell, 1 works for Rogers, 1 now works for the CRTC). We're
| not risk adverse. It's just that our government / regulator
| is captured.
| pid-1 wrote:
| Isn't that true in most countries?
|
| Is there any place in the world that can claim to have a
| diverse, competitive Telco market?
| ezekiel11 wrote:
| no but i think theres similar zombie giants that is
| essentially a hybrid state run enterprise but Canada has
| explicitly designated industries where you simply cannot
| execute because they are anti-competitive.
|
| and its not clear how individuals are being handed out rights
| to say run a casino in canada.
|
| if you took a look at canada's corruption, you will be
| shocked. we don't quite live in a first world even though we
| like to tell ourselves all of the government
| officials/servants are honest.
|
| after all the canadian embassy staff in Hong Kong happily
| handed out PR residency to hardcore organized criminal groups
| in exchange for various luxuries and perks. when a staff
| tried to expose it, he was quickly removed and media began to
| attack him.
|
| its not only that but you see NGO's championing for racist
| white supremacy linked groups who vandalize Chinatown in
| Vancouver and with a large chunk of locals who feel that they
| are "being overrun by a certain ethnic group" eat that up and
| the same populist individuals get elected again and again.
| then my tax goes towards those interest while none for me
| because i'm 'privileged'
|
| in the long run, I see this system breaking down, if not
| already. Canada no longer feels like a country but some
| feudalistic interest group driven, poorly run corportation.
|
| so glad i dont have to pay taxes here. the savings and
| currency difference allowed me to create jobs in another part
| of the world. there was a time where I hoped things would get
| better and I could be creating jobs locally.
|
| i just regret wasting my youth in canada and west coast.
| canada is a bubble and lot of us are moving capital/jobs out
| of it.
|
| why contribute to a country that just sees you as an ATM to
| transfer payments to others who blame everyone but themselves
| and constantly wanting hand outs?
|
| im done with canada and im warning anybody who still attach
| romantic outlooks, especially in heavily marketed cities like
| vancouver.
| walrus01 wrote:
| for last mile ISP options, there are some very fortunate US
| counties in WA state that have
|
| a) last mile dark fiber network operated by local public
| utility district which is also the electrical grid operator,
| and rents access to the fiber to 3rd party ISPs
|
| b) local cable tv operator, legacy coax operator, often
| docsis3 cablemodem
|
| c) local ILEC/POTS phone company that may or may not have
| overbuilt its last mile copper/DSL service with its own
| singlemode fiber and 1Gbps GPON service.
|
| for primarily mobile phone carriers like rogers, there's an
| effective RF planning limit of around four major LTE/3GPP
| technology based operators in any given geographical area.
|
| USA used to have 4 with sprint until the tmobile/sprint
| acquisition.
| thematrixturtle wrote:
| The model of separating the fiber network from the telcos
| that offer service on it can work of the incentives are well
| aligned. Many Asian countries, eg Singapore, do this.
|
| Or you could put the fox in charge of the hen house and end
| up in Australia's situation, where local monopolist Telstra
| owns the phone network and is supposed to play nicely with
| its competitors.
| ROTMetro wrote:
| Mountain West state, middle of nowhere town. I have the
| option of choosing between a 1Gb+ fiber provider, a 1Gb fiber
| provider, a 100Mb fiber provider, multiple wireless line of
| site providers in the 30Mb range(more targeted to the people
| outside town but have line of site to the mountain), or
| T-Mobile home internet in the 30-180Mb. Small towns are super
| friendly and responsive to the 'work remote' crowd.
| hackbinary wrote:
| Just wait until you hear about the UK.
|
| BT/Openreach has control of about 80% of the backhaul/trunk
| connectivity.
| martinald wrote:
| Completely not comparable. Openreach has significant
| marketshare but is also extremely tightly regulated. It has
| to allow other providers access to basically every element of
| its offering, now including its physical ducts and poles.
|
| Hence you have a very competitive market landscape in the UK.
| There are at least 5 major national retail players using
| various wholesale products (BT, VM, TalkTalk, Sky, Vodafone),
| plus dozens of altnets offering FTTH on totally seperate
| fibre infrastructure have started (Hyperoptic, Cityfibre,
| GNetworks, Community Fibre).
|
| In my flat in London I have access to 4 seperate FTTH
| networks (with completely different infrastructure) -
| Openreach FTTH, VM DOCSIS, Hyperoptic FTTB and Community
| Fibre FTTH. The market works here, prices are low and there
| are very few data caps.
| walrus01 wrote:
| I work in telecom and tell this to nearly everyone who asks me
| about Canada, and its ISP/telecom market and competition...
|
| "Canada is an endangered species protection wildlife reserve
| park for dinosaur telecoms"
|
| It's a travesty that the government is actually going to let
| Shaw and Rogers merge to even further concentrate power in the
| hands of a few families and reduce market competition.
| pseudoramble wrote:
| In the article, it mentions that the Canadian government
| blocked this merger. I believe it said that was in April
| though. Has that changed, or did I misread what the reporting
| said? Just want to clarify in case I missed something.
| LawnGnome wrote:
| The Canadian government have put up a mild roadblock that
| will be addressed with non-substantive changes to the
| structure of the merger, enforced by a little known,
| toothless regulatory agency who will be instructed to look
| the other way.
|
| The regulatory environment Canadian telcos operate in is
| completely captured.
| walrus01 wrote:
| I would bet $100 they'll ultimately let it go through
| anyways due to the absurdly outsized political influence
| and lobbying of the equity shareholders of Shaw. One of the
| wealthiest families in the country.
| gsatic wrote:
| Most people also don't realize telcos rake in more revenue than
| the entire tech sector. And that it is no where close to enough
| to maintain or upgrade the pipes. Cloud has been built on lot
| of assumptions that are close to breaking point.
| dylan604 wrote:
| The entire tech sector would not exist without the telcos. So
| there's that
| donalhunt wrote:
| Deploying infrastructure is not easy or cheap. But is
| clearly profitable if you can find investors to finance the
| capital spend.
|
| Google have tried to enter the FTTH market thinking there
| must be opportunities to innovate and reduce the cost to
| consumers. But even they have struggled due to the
| complexities of deploying fiber in the existing built
| environment...
| empressplay wrote:
| Goodbye Rogers, Hello Telus!
|
| Glad that Best Buy has a large stock of Telus SIMs.
|
| Never again, seriously.
| tra3 wrote:
| Staying with the inlaws that use Rogers' reseller for wired
| internet.
|
| Using Fido for mobile internet access (also by Rogers).
|
| Zero internet. It's wild to discover how reliant I am on the
| internet:
|
| - Can't get around this city (no waze, gmaps).
|
| - Can't figure out what's going on (AM radio is non stop
| commercials)
|
| - Can't figure out where we might get some internet (maybe WeWork
| somewhere? How do you find one without internet)
|
| This is just wild. Almost as critical as no water, or no power in
| the winter.
| Scoundreller wrote:
| > Can't get around this city (no waze, gmaps).
|
| offline gmaps!
|
| Thankfully Canada's horrendous roaming and overage charges has
| taught me how to use my cell phone effectively without service.
|
| (Also have the entirety of Wikipedia on my phone, thanks
| kiwix!)
| guerrilla wrote:
| > - Can't figure out what's going on (AM radio is non stop
| commercials)
|
| Don't you have public service? Isn't that what CBC Radio One
| is, with no commercials?
| EngCanMan wrote:
| Still no internet at all? I have a suspicion that you found
| some...
| markus_zhang wrote:
| Still no sadly. My phone shows 2G network. The last time I
| use 2G network was probably...decade ago.
| doubled112 wrote:
| My reseller connection (Start.ca) is down, and Telus' cell
| phone service becomes near unusable because of the extra
| load.
|
| I expect the device I send this from will be offline again in
| the next few minutes, but I'm willing to be pleasantly
| surprised.
| Scoundreller wrote:
| Have been getting all circuits busy alerts periodically
| throughout the day.
|
| Fun stuff!
| dogecoinbase wrote:
| The HN downtime was to add a post-by-postal feature
| Victerius wrote:
| Interac is down because Interac uses Rogers' network.
|
| Interac's backup network also happens to be with ... Rogers.
|
| Canadians across the country can't use debit cards, regardless of
| their financial institution. Credit and cash only. E-transfers
| are unavailable. Fun for people trying to pay their bills online.
|
| And that's just the financial side of the situation. Millions of
| people have lost Internet and phone service.
|
| Someone on Reddit wrote: "Maybe the worst part is that they have
| literally no open lines of communication. No twitter posts. All
| chat options (twitter, IG, FB, web based) are all unresponsive.
| Can't call tech support, as the call will fail. Can't even call
| billing or general inquires as it's the same thing. Shit happens,
| I get it. But to leave your customers completely in the dark?
| Brutal. Just brutal."
| acchow wrote:
| They have a system-wide outage. How can tech support help in
| any way?
| acoard wrote:
| It's hard to understate the impact of this across all of
| Canada. As someone who uses Apple Pay for almost every
| transaction, this underlines the fragility of a "cashless
| society."
|
| Also, around half my team are unable to effectively work today.
| We all work remote, and half the team's home internet is on
| Rogers, which is down. Some people are tethering, but people
| who are also on Rogers for their wireless are out of luck.
|
| I'm not a Rogers customer anymore, but my boss showed me how
| you can't even login to their account anymore. Their website is
| down too, just timing out.
|
| Funny, I left Rogers last year and just last night a
| telemarketer phoned me asking me to return to Rogers. What a
| sign that I made the right call by politely refusing.
| kgraves wrote:
| > It's hard to understate the impact of this across all of
| Canada. As someone who uses Apple Pay for almost every
| transaction, this underlines the fragility of a "cashless
| society."
|
| It seems like Bitcoin and crypto fared well during this
| outage.
| jeromegv wrote:
| If you could get on the internet to transact your crypto,
| that is.
| kgraves wrote:
| You can if you use cellular data to transact with
| Bitcoin, not everyone is using the Rogers network, but
| the banks in Canada were.
|
| So no online banking, withdrawals, payments in affected
| parts Canada.
| MAGZine wrote:
| applepay works fine. not on the interac network.
| KMnO4 wrote:
| Does Apple Pay work even with debit transactions?
| Spoom wrote:
| Imagine Bell was down too. What then?
|
| I really hope this results in the CRTC starting to allow more
| competition. I hope, but I'm not optimistic.
| anonymousab wrote:
| I think we're most likely going to see them using this as
| justification to let Rogers get their way even more. Maybe
| the CRTC will try to convince the rest of the government
| that it's a reason to give away more cash and preferential
| treatment to Rogers and Bell as well.
| paulhart wrote:
| I'd suggest giving the CRTC a call with your thoughts...
| but their phone system is down :)
|
| https://twitter.com/CRTCeng/status/1545421218534359041
| lotsofpulp wrote:
| > this underlines the fragility of a "cashless society."
|
| And a society that refuses to invest in robust
| infrastructure.
| nikanj wrote:
| Tethering with Canadian data prices is..spicy. A Zoom call
| uses about a gig per hour with HD cameras, and you typically
| get about 10 gigs of data per month. Overages charged at one
| firstborn per gig
| brianpan wrote:
| You don't need an internet connection to use Apple Pay.
| Unless you mean you didn't use credit cards before, the
| situation has been the same for the last 20-30 years when we
| stopped using carbon paper imprints.
| acchow wrote:
| The payment terminal still needs internet access to process
| your Apple Pay transaction
| mynameisvlad wrote:
| It does not for credit. Credit transactions can be
| batched offline and transmitted when things come back
| online. Debit must be online, though.
| bawolff wrote:
| Its crazy that interac does not have backup independent
| connectivity.
| paulhart wrote:
| Apparently it does - with another Rogers connection.
| acoard wrote:
| Truly, and hopefully this leads to change. But this is also a
| consequence of Canada's lack of competition with ISPs.
| bawolff wrote:
| To a certain extent yes. But its not like rogers literally
| has a monopoly. The isp i use is doing fine.
| jeromegv wrote:
| When there is so little choice, it's essentially an
| Oligopoly.
| rkagerer wrote:
| It's hard to believe these jamokes don't have redundant
| providers. It's something I roll out to even my smallest
| consulting client...
| routerl wrote:
| In Toronto, here's a small and non-exhaustive list of affected
| services:
|
| - Point of sale machines. Supermarkets are accepting only cash.
|
| - ATMs. Several banks are incapable of dispensing cash.
|
| - Public transit ticketing systems. The TTC is effectively
| running for free, today.
|
| - Public bicycle rental stations.
|
| - Public parking locations.
|
| How long before the public realizes that we are not Rogers'
| customers, but its hostages?
| dennis_jeeves1 wrote:
| >How long before the public realizes that we are not Rogers'
| customers, but its hostages?
|
| Won't happen. The average Canadians are one of the most sheep
| like docile people out there.
| KMnO4 wrote:
| Remember when we all banded together to prevent Verizon from
| introducing competition into our market because it would
| "take Canadian jobs" and "give our hard earned dollars to the
| states"?
|
| That happened because of a hostile attack ad campaign by the
| big 3.
| ehsankia wrote:
| Has any of the big cities ever considered its own municipal
| internet? I would vote for any politician that has that as a
| platform immediately.
| goatcode wrote:
| You must know that it'd run on Rogers infrastructure though.
| Victerius wrote:
| > Point of sale machines. Supermarkets are accepting only cash
|
| > ATMs. Several banks are incapable of dispensing cash
|
| Back to bartering.
| StayTrue wrote:
| At my local supermarket debit was down but credit card (Visa)
| was working.
| jonny_eh wrote:
| That would make sense, since only Interac relies on Rogers,
| and it processes debit cards, not credit cards.
| Scoundreller wrote:
| Could be one of two things:
|
| 1) the site has connectivity so only Interac is down
|
| 2) the site has no connectivity and can process credit
| transactions offline and queue them (don't tell anyone
| their expired card will work), but not for debit.
| jfim wrote:
| > don't tell anyone their expired card will work
|
| The expiration date is stored on the card, so expired
| cards wouldn't work. Revoked cards (eg. ones replaced by
| the bank) would, though.
|
| Keep in mind that the merchant sets terminal limits for
| what transactions are allowed to be processed offline, so
| one might be able to get a burger without connectivity,
| but not a new TV.
| [deleted]
| tr1ll10nb1ll wrote:
| I'm using a Roger's number in India currently.
|
| Unfortunately, I've been using that number for most of my 2FA so
| essentially, I've been unable to access a lot of services because
| of this (ones where I didn't use Authy/Google Authenticator).
|
| Lesson learnt: use a separate VoIP business phone number instead.
| bibstha wrote:
| Are you using Wifi calling to receive 2FA? What would you
| change to?
| abrichr wrote:
| Same here in the US. Which service will you port to?
| tr1ll10nb1ll wrote:
| I'm just using OpenPhone now for services tied to my startup.
|
| At least, I'd be getting verification codes without having to
| worry about the service being down.
| shirononon_ wrote:
| this is quite a significant outage and has been ongoing for over
| six hours now.
|
| would love to see a tech write up or blog post or even a Twitter
| hint but there seems to be absolutely zero transparency.
| RL_Quine wrote:
| The first withdraw was at around 7:45 UTC, a little over 13
| hours at the time of writing this comment.
| randlet wrote:
| > ongoing for over six hours now
|
| Closer to 12 hours now I think. Certainly it was out at 7am EST
| this morning when I woke up.
| timbit42 wrote:
| Problems started to appear around 4am EST and services became
| completely unavailable by 4:40am EST.
| nabaraz wrote:
| Looks very similar to Facebook's outage. A combination of BGP and
| DNS problems that started with a configuration change.
|
| https://blog.cloudflare.com/october-2021-facebook-outage/
| acoard wrote:
| What evidence are you seeing that makes you think this is
| happening in the Rogers' case? I haven't seen any diagnostics
| or analysis by anyone.
| RL_Quine wrote:
| Rogers did just abruptly withdraw all their routes, then
| reannounced some, then withdrew them again. It's not
| dissimilar at least.
| [deleted]
| Ardon wrote:
| You probably saw it up thread but since rogers isn't talking
| this is all we really have:
| https://blog.cloudflare.com/cloudflares-view-of-the-
| rogers-c...
| tvhahn wrote:
| The Canadian telecom space is dominated by 3 primary companies --
| Telus, Bell, and Rogers. They form an effective oligopoly that is
| quite detrimental to the Canadian consumer.
|
| In the ISP space, there is a bit more competition. Namely, Shaw
| provides additional coverage in some regions of the country.
| However, Rogers wants to buy Shaw. You can imagine how bad that
| will be for Canadians.
|
| I do wonder what the Rogers outage is about. Ransomeware? State
| attack? Something stupid? If anything, it shows how we should not
| have critical infrastructure centralized. Competition between
| ISPs is important.
| ju45 wrote:
| There's also Videotron and Cogeco but these are not Canada wide
| so .. yea.
| walrus01 wrote:
| In the ISP world, what will happen with Rogers buying Shaw...
|
| In the western provinces, AB and BC, Rogers runs a very
| widespread and strong LTE network. What they do not have is a
| DOCSIS3/coax and GPON cable TV plant. Nor do they have much
| terrestrial right of way for aerial plant or underground in
| conduits, which is where Telus (the historical copper POTS
| ILEC) and Shaw (the historical cable TV operator going back to
| the mid 1970s) are by far the strongest.
|
| Rogers is a facilities based last mile cable
| operator/terrestrial operator in Ontario.
|
| Shaw runs the landline cable tv networks in most of the metro
| Vancouver area. And many other small to mid sized cities in the
| west.
|
| Letting Rogers control both one of the largest/strongest mobile
| phone networks _and_ the only viable land line terrestrial
| broadband competitor to Telus by acquiring Shaw 's cable tv
| plant is an absolute outrage.
|
| To use a USA analogy, it's like if T-Mobile already owned RCN
| (Astound) in some big part of the country and then proceeded to
| buy Comcast.
|
| Or if Verizon bought Spectrum (Charter/historical TWTC cable).
| zeagle wrote:
| Hopefully this will contribute to a regulatory denial of Rogers'
| upcoming purchase of Shaw, if only to keep a competitor with its
| own separate infrastructure to minimize outages through
| consolidation.
| pigtailgirl wrote:
| -- Started to fall to bits around 4:00am EST - got woken up by my
| phone non-stop glowing as news alerts about abe flooded in - got
| out of bed to check the news on my laptop - wasn't working -
| tethered to my phone & checked BGP - AS812 was announcing bogons
| & dropping peers like it was going out of style - thought maybe
| they were doing a config update - went back to bed - was
| incredibly surprised to see it dropped all the peers & was
| sitting dead when I woke up --
| betaby wrote:
| Can you please show bogons learned from AS812? Thanks!
| RL_Quine wrote:
| https://bgp.he.net/AS812#_bogons
|
| They announced 216.176.216.0/21.
| bandwidth wrote:
| The VP of Rogers admits they have yet to identify the root cause:
|
| https://twitter.com/PnPCBC/status/1545512971878662145
| cluoma wrote:
| Not a good week for internet infrastructure in Canada. Yukon
| Territory had essentially no internet for most of this Wednesday
| after the only fibre was broken.
| walrus01 wrote:
| that, at least, was a very canadian outage caused by a beaver
| chew
|
| it's hard to build resilient rings of fiber between
| towns/cities where there is only one linear path (road) for
| right-of-way that you could economically build the fiber
| along...
| ilrwbwrkhv wrote:
| ezekiel11 wrote:
| I know many countries have problems but Canada is in a okay
| spot. Specific cities have looming crisis on their hands but
| this is largely the fault of provincial governance.
|
| Little of that I attribute to Canada as a hole, I just find BC
| in particular to be quite mediocre in terms of value compared
| to what you pay.
| seibelj wrote:
| It's a great weigh-station for immigrants who want to
| eventually move to the US. Canada is easier to immigrate to as
| a high-skilled person.
| ezekiel11 wrote:
| > Canada is easier to immigrate to as a high-skilled person.
|
| False.
|
| > It's a great weigh-station for immigrants who want to
| eventually move to the US.
|
| Many do not, they move back home.
| ju45 wrote:
| That's why I use Bell or Videotron
| betaby wrote:
| https://twitter.com/Videotron/status/1360324916176896007 last
| year
| UkrainianJew wrote:
| The utterly inefficient and overpriced Canadian telecoms exist
| only because we allow one specific behavior:
|
| 1. A new local player comes to town, builds their own
| infrastructure (e.g. connects one building to fiber) and starts
| offering competitive service.
|
| 2. Rogers/Shaw/Bell/Telus immediately offer better terms for the
| residents of that building.
|
| 3. The competitor runs out of money and leaves.
|
| 4. The telecoms revert to their usual pricing.
|
| This hurts competition, this hurts customers, this hurts long-
| term infrastructure resilience, but not a single politician ever
| comes close to even admitting the problem. I understand you
| cannot do it on the federal level where both major parties are on
| the telecoms' payroll, but it could be a very low-hanging fruit
| for someone running for a municipal position to address. But
| nope, identity politics, words, feelings and fighting climate
| change with paper straws seem to be what the electorate wants
| instead. Sad.
| UkrainianJew wrote:
| I also remember Air Canada or WestJet doing the same to some
| small air company offering cheap flights from Kelowna (?), but
| I cannot find the news piece. Does anybody here have a better
| memory than myself?
| justsid wrote:
| Are you thinking of Flair or Swoop? Never flown them out of
| Kelowna when I was living there, but I remember both being a
| big deal when they finally arrived.
| brailsafe wrote:
| They're still around
| pj_mukh wrote:
| Curious though what the government could do about this pricing
| scheme here? Seems like the solution is to setup a bureaucracy
| to watch pricing and then micro-manage the situation whenever
| its tried?
|
| The ride-hailing services pulled this in various places around
| the world. Airlines in Canada did it as well. Feels like whack-
| a-mole.
| nayuki wrote:
| These observations are very true. A friend was living in a
| Toronto condo building, and I saw them choose to switch from
| Beanfield to Bell/Rogers for the same monthly price but higher
| Internet speeds. They took the bait and now everyone pays the
| price in the long run. And yes, Bell/Rogers gouges with high
| prices for houses/apartments that don't have an independent ISP
| like Beanfield, etc.
| newsclues wrote:
| I don't understand have 911 service is impacted, doesn't
| essential services have redundancy?
|
| Society is fragile
| KMnO4 wrote:
| The actual dispatch appears to be fine. It's just the network
| isn't allowing mobile connections. Everyone has "No Service".
| MR4D wrote:
| FTA - "Rogers has not identified a cause."
|
| Seems like a rather large impact. Maybe Russia really wants its
| gas turbine back? [0]
|
| [0] - https://www.reuters.com/world/exclusive-ukraine-urges-
| canada...
| walrus01 wrote:
| This is funnier if you look at it from the English slang point of
| view when something gets thoroughly Rogered.
|
| This network has definitely rogered itself.
| NKosmatos wrote:
| "In April 2021, Rogers customers reported interruptions to
| wireless voice and data services for several hours. Rogers blamed
| that outage on a glitch tied to an Ericsson software upgrade".
| Rogers has rolled out 5G with Ericsson equipment and I hope this
| outage is not related to Ericsson activities. Network upgrades
| are usually performed late at night during low traffic, so it
| shouldn't be the case.
| jagger27 wrote:
| If this doesn't show Canadians that telecommunications are a
| public utility, I don't know what will. The CRTC's (Canadian FCC)
| own phones are down[1] today because they're a Rogers customer.
| If that doesn't scream regulatory capture...
|
| 1: https://twitter.com/CRTCeng/status/1545421218534359041
| nevereveragain wrote:
| I mean... it isn't as if the CRTC is going to set up their own
| telco just to operate their own office.
| exadrid wrote:
| Yeah, but ultimately they are pretty much owned by the big
| Telco so we are fucked
| randlet wrote:
| The lack of communication from Rogers for an event of this
| magnitude is quite something. Three tweets[0] with zero useful
| information so far today. The first one not coming until ~4 hours
| after the interruption started.
|
| [0] https://twitter.com/rogers/
| theIV wrote:
| They probably can't get online to tweet. /s
| jonny_eh wrote:
| Seriously though, someone working there had to find a friend
| with a Telus hotspot to borrow, I imagine.
| dylan604 wrote:
| Wait until you find out that the cable that Rogers uses to
| peer to the Telus network got unplugged!
| donalhunt wrote:
| Their 2FA passcodes are sent via SMS to number(s) on the
| dead network. :_(
| alphaomegacode wrote:
| The timing of that update tweet is what gives pause for
| concern. If it were the same or similar thing as to what took
| down their network nationwide in April, it seems reasonable
| that they'd just say something like "Hey guys, sorry but the
| network is down due to <insert similar cause here>."
|
| A lot of us are in IT and for a network of national importance
| in a "developed nation" to be taken offline so easily is
| worrisome.
| [deleted]
| markus_zhang wrote:
| Looks like it is still going on. I only have 2G network. I also
| realized that I cannot use Google Map for driving without
| network.
___________________________________________________________________
(page generated 2022-07-08 23:00 UTC)