[HN Gopher] The internet is held together with spit and baling wire
___________________________________________________________________
The internet is held together with spit and baling wire
Author : picture
Score : 170 points
Date : 2021-11-26 19:11 UTC (3 hours ago)
(HTM) web link (krebsonsecurity.com)
(TXT) w3m dump (krebsonsecurity.com)
| ragingrobot wrote:
| O'Brien: I'm afraid to touch anything. It's all cross-circuited
| and patched together - I can't make head nor tails of it. Bashir:
| Sounds like one of your repair jobs.
|
| Seriously though, it seems like every form of infrastructure we
| rely on is held together in such a fragile manner. I hate hate to
| think of the chaos should there be a major Internet and physical
| infra failure in close proximity, time-wise.
| hypertele-Xii wrote:
| We have a pretty good track record keeping it running though;
| The Internet has never gone down!
|
| I'd be far more concerned with agricultural logistics, though
| we've never starved to extinction, either.
|
| Perhaps we've reached a point of positive no return: We can no
| longer cease to exist!
| [deleted]
| cgh wrote:
| There's a strong probabilistic argument that says otherwise:
| https://en.wikipedia.org/wiki/Doomsday_argument
|
| Note that the argument is not dependent on any sort of cause,
| such as climate change or whatever. It is entirely
| probabilistic.
| setpatchaddress wrote:
| We absolutely can. Imagine a US president outwardly Trump-
| like but who was also a True Believer instead of a cynical,
| mentally-ill real estate huckster. They would have the
| unilateral capability to start a global nuclear war. And they
| might actively desire to do this, on the basis of religious
| delusion ("it's time for armageddon!").
|
| We are frighteningly close to this scenario in 2025. It's a
| bad idea to assume that the gop candidate will be a fascist-
| wannabe in on the joke (someone like desantis or cruz). We
| can survive run-of-the-mill fascism, although it won't be
| fun. But it could instead be someone truly existentially
| dangerous.
| throwawayboise wrote:
| Since we're imagining things, imagine a befuddled geriatric
| sock puppet president installed via fraud and controlled by
| a globalist cabal with a stated desire to depopulate the
| planet.
|
| See how crazy that sounds?
| bobthebuilders wrote:
| Not really. You'd be surprised with what 3 letter
| agencies do.
| lliamander wrote:
| Breathe
| adventured wrote:
| In this context it's a very important point that there isn't
| one Internet and there isn't one "we." Unless you strictly
| mean the human race has yet to go extinct due to starvation.
|
| A lot of starvation events have of course occurred across
| many different nations, peoples, civilizations. Europe, as
| one example, was numerous times ravaged by extreme starvation
| events that collectively killed millions of people across the
| 19th and 20th centuries. I think your agriculture concerns
| are well placed.
|
| The various Internets have gone down routinely for all sorts
| of reasons.
| zwkrt wrote:
| Unless you exist in a state of wild over abundance and are very
| conscientious, most working infrastructure is always at a state
| of almost-disrepair. If I operate a machine shop and I have 4
| partially-stripped screwdrivers, I might get a new one, but
| inevitably I will use it until it is as stripped as all the
| rest, and I don't dare throw away a tool that is currently even
| in partially working condition since I might need it later if
| another driver breaks. This is true of every single thing in my
| shop, and the end result is a system that is robust to absolute
| failure but prone to constantly needing to be patched up.
|
| As far as I can tell this is a truism across all fields:
| farming, skilled trades, build systems, transportation
| infrastructure, housing...
| AceyMan wrote:
| A corollary to this might be "a new part on the shelf is not
| truly 'better' than a 90% worn out one still in service."
|
| Having ready spares is great but I'm sure the HN crowd knows
| how much crib-death there is on new, replacement bits of all
| kinds.
|
| Until it's _actually been run-in_ for a while how do you
| really know it 's going to work when you unbox it to replace
| a truly dead piece?
|
| The upshot of this is' RAID-6' type designs are the most
| reliable in the real world since when one fails at least you
| are leaning on other parts that have been run-in and are past
| the leading edge of the bathtub curve.
| dylan604 wrote:
| >Having ready spares is great but I'm sure the HN crowd
| knows how much crib-death there is on new, replacement bits
| of all kinds. >Until it's actually been run-in for a while
| how do you really know it's going to work when you unbox it
| to replace a truly dead piece?
|
| We had the unfortunate experience of installing 4 new 16
| bay chasis with brand new drives (10+ years ago now). We
| designated one of the 16s as hot spare for each chasis,
| plus had 2 cold spares for each as well. 72 brand new
| drives in total. All from the same batch of drives from the
| manufacture. Set them all up on a Friday and configured all
| for RAID5 (pre-RAID6 availability). Plan was to let them
| build and have some burn-in time over the weekend for
| possible Monday availability. Monday provided us with
| multiple drive failures in each chasis. Drive manufacturer
| confirmed a batch batch from whichever plant, replaced all
| and delivered larger sizes for replacements. Luckily, they
| failed during burn-in rather than 1 week after deploying.
| ragingrobot wrote:
| Imagine you use that last screwdriver, and a job needs to get
| done, and your car won't start so you can't get a new one at
| a hardware store outside of walking or cycling distance. And
| now you're unable to repair a vital system.
|
| As I was writing that, I was just thinking of the recent
| Surfside collapse, what would would have happened if the
| regions data networks had gone down simultaneously (by
| chance). A major event two decades ago, cellular networks
| were overwhelmed and calls could not be made. I dare say
| we're more reliant on those networks today, as well as the
| Internet.
|
| Not that I would expect it to happen, but it was just a
| thought.
| ip26 wrote:
| Are your mills in a similar state? I'm no machinist, but I
| would think it hard to stay in business when the means of
| production are constantly down for unplanned service.
|
| Screwdrivers are more of a consumable than a capital good.
| avgcorrection wrote:
| They said "If I operate a machine shop".
|
| So presumably just another programmer making up an analogy
| with insufficient background knowledge so that we can
| nitpick the details of it.
| zemvpferreira wrote:
| Presumably. Those statements would be insulting to most
| professional machinists, carpenters and others who take
| good care of the tools/infrastructure that provide their
| job and keeps their digits in place. To not speak of an
| actual industrial facility producing high-quality or
| high-volume items.
|
| In fact, so much care is put into infrastructure that
| most people/shops have lots of tools they have designed
| and built themselves at great expense to streamline their
| operations.
| joconde wrote:
| Software too, unless every small piece of it has a dedicated
| full-time maintainer. If there's an infinite stream of tasks
| incoming, why go improve something that fits the current use
| well?
| 999900000999 wrote:
| You could also break something with your dull tools.
|
| This is why you can't just drive a car until it breaks, you
| get it checked out.
|
| Then again, cars are a private good. When it's your property
| vs our property you have more of an incentive to take care of
| it.
| WJW wrote:
| Machine shops are also private goods and their owners have
| a lot of incentive to keep them working, so I don't think
| this example is very accurate. In any case the chance of
| breaking anything with a partially stripped screwdriver is
| pretty minimal.
| 999900000999 wrote:
| If you don't know what you're doing you can strip a
| screw.
|
| Getting it out won't be fun.
| peoplefromibiza wrote:
| if you're at home, with no other tools, it could be
| painful.
|
| but in a shop unless you can't take it out using other
| means (pliers for example or a nut split remover), you
| can simply weld a bolt on the screw's head and use a
| wrench to unscrew it.
|
| or drill a hole through it after removing the head and
| use another screw to take out the moncone from the other
| side.
|
| Bad screws are more common than bad screwdrivers and even
| a brand new screwdriver could lead to the same result.
|
| as the original post said _" If it breaks all the time,
| everybody is highly experienced at patching together new
| workarounds"_
| TheMagicHorsey wrote:
| What are some other/better proposals for how to organize a world-
| wide network? This is one area where I have not seen many
| articles. But admittedly, I'm probably not looking in the right
| places. Any suggestions?
| cblconfederate wrote:
| blockchains are one
| teddyh wrote:
| You could ask Cloudflare or Google and they would probably say
| that _they_ should run it all.
| hunterb123 wrote:
| Which funny enough, are the only parts that really go down
| all at once.
|
| When the internet has troubles it's a mistake of a giant
| centralized service.
|
| Most of the time only that service is affected by their own
| mistakes, but sometimes that service hosts a lot of others,
| or is so massive that they cause DDoS attacks like when
| Facebook went down and their clients spammed DNS servers.
|
| Seems the internet is fine. Centralized services not so much.
| winternett wrote:
| And Google requiring (costly and/or time consuming) SSL certs to
| be applied on all sites to "ensure security" was also a big
| industry money making nightmare for many independent (non-income-
| driven) sites that is still playing out badly, and not providing
| much more security.
|
| Two factor authentication and account verification is really an
| elaborate corporate sham to get people's phone numbers and PII
| for free. It doesn't do anything new or good for consumers in
| terms of security over time. There, I said it.
|
| I prefer the old Internet. All these new fangled "fixes" are only
| makin it worse, more expensive, and overly complicated. :/
| oneepic wrote:
| Honest question (IT/security noob) -- why does it not provide
| that much more security? I like verifying that my traffic is
| going where I want.
| winternett wrote:
| With Encryption being applied to every site as a requirement
| is relatively new since google made it a requirement in
| Chrome.
|
| Previously it was only required for secured transactions like
| purchases and working on health care records etc... And very
| rightfully so.
|
| Now Google Chrome flags even simple (informational) sites for
| not being encrypted, and (quite possibly) rightfully so
| because of the potential for tracking/abuse, but adding
| encryption to a site is costly for independent sites (not
| hosted on social media or corporate platforms like blogs
| etc...
|
| You shouldn't be required to encrypt a baking recipe site if
| you don't want to... Ultimately laws should discourage data
| abuse, and/or encryption should be inherently provided for
| every site/app uniformly by all web host providers (natively
| and inherently, and at a far lower price than it is now,
| generally speaking).
|
| Too many people are running widely varying encryption
| measures, and implementing security in too many different
| ways to ensure that it is stable across the Internet.
| Security is best when it is uniform, fortified by rules and
| regulations, and updated ritually.
| [deleted]
| fulafel wrote:
| Deprecating unencrypted HTTP is a big systemic improvement even
| though some individual sites may not benefit much. It's a
| network effect. (What's the money grab given free let's encrypt
| certs?)
| winternett wrote:
| Lets encrypt from what I understand require time consuming
| updates every few months. My host provider also does not
| allow me to install them manually, further complicating the
| process, and conveniently they sell certs for $125 a year...
| Per site. It's been a thorn in my side because we're too big
| to easily move now.
| fulafel wrote:
| You are absolutely not meant to do the updates manually.
| winternett wrote:
| On one ISP that I host sites on, they restrict cert
| installs and don't allow SSH access. It's done in order
| to sell their cert services. I have too many sites on
| there to move easily... It's complicated. Eventually I'll
| bite the bullet and move to a new host. :/
| fulafel wrote:
| Sorry about your service provider failing at their job
| and squeezing you for $$ cert services! But I'm not
| nearly convinced this is big enough to stop encrypting
| the web.
| switch007 wrote:
| Setting up, monitoring and maintaining LE isn't free
| ertian wrote:
| Whether or not that's true, it's not a money grab.
| shadowgovt wrote:
| But monitoring and maintenance are things someone needs to
| do if they operate a site, period.
| winternett wrote:
| But if you're independently running, paying for, and
| managing multiple sites, it's a HUGE burden. It also
| kills innovation for independent devs and startups, and
| dramatically raises the cost/investment threshold for
| this kind of innovation.
|
| Pricing on cert services is also far too high when
| everyone's concern and agreement should be security as a
| basis for operations. It's not something that should be
| an upcharge or income opportunity.
|
| You buy a door lock for your home once, and it works as
| long as you don't compromise the key. If you buy a house,
| door locks are expected to come with the house in most
| circumstances.
| WJW wrote:
| Having just replaced my door lock, I can assure you that
| they too wear out and need replacing. (one of the springs
| inside broke)
|
| The pricing on Let's Encrypt is literally zero, and they
| provide (also free of charge) the `certbot` utility which
| you can run as a cronjob and which will automatically
| renew your certificates for you. The whole thing comes
| extremely well documented and with install scripts that
| take less than a minute to download, verify and run. If
| you think even that is too much of a burden I don't think
| any topic in programming is simple enough.
| shadowgovt wrote:
| And, indeed, if you build your site via a service
| provider or platform, an SSL solution is usually
| provided.
|
| Building a site from scratch in this day and age is a lot
| more analogous to building your house from scratch.
| Nobody to blame but yourself if you buy substandard locks
| and thieves get in. Only here the metaphor breaks down,
| because if you aren't encrypting your HTTP traffic and it
| is intercepted, it's your users who suffer, not the site
| owner.
|
| I, too, pine for the days of simpler internet. But that
| was a function of the user base, not the technology. It
| was always insecure... it simply hadn't been exploited
| yet. Now that it has, and is, site administrators owe it
| to users to secure their connections.
| fulafel wrote:
| Setting up certbot is easy, not a big burden for indie
| devs. Or if you want to know nothing about tls & certs,
| just get hosting that comes with tls.
| winternett wrote:
| I wasn't writing to the update process, as much as the
| original installation of a cert.
|
| On a house you own, you can change locks and keys any
| time you want to keep security up to date (for example).
|
| no house in "move in ready condition" comes without
| sufficiently keyed door locks of some kind (on day1).
| xxpor wrote:
| traefik takes care of all of this with about 5 lines of
| setup. it's so trivial i add it to every experimental
| nonsense service I setup because it's one line of nix
| config. i really don't understand the complaint.
| charcircuit wrote:
| It all happens automatically after a setup process that
| takes less than a minute.
| snoopen wrote:
| Sure, certificates can be time consuming at the moment but that
| will only get easier. Just like hosting the underlying website.
|
| The number of sites that should have had SSL but didn't was
| laughable and justification enough for browsers to require SSL.
|
| I don't know if you're being deliberately alarmist, but 2FA is
| a huge peace of mind when done correctly with one time codes.
| Those don't require phone numbers and is the properly secure
| method.
|
| Sure the old internet was a bit more fun and carefree, but it
| became far less fun when you had your online accounts
| compromises because of weak or non existent security.
| ratorx wrote:
| I'll reply to this and some of your other comments in this
| reply.
|
| In a lot of cases, SSL is not expensive or time consuming. It
| is a single line in cron. I appreciate that this is not the
| case for your hosting, but economic pressure is one of the main
| ways SSL can be more utilised. The fact that you're considering
| moving away from them, suggests that their business will suffer
| in the long term, if they don't make integrating SSL
| easier/less expensive. This is good economic pressure, and its
| likely the best pressure that can be applied right now,
| considering the glacial pace of technology laws in almost all
| countries. You seem to be generalising your situation and
| applying the blanket "it's too expensive" argument to everyone,
| even though it's mostly a non-issue for people who have better
| hosting providers or not as much legacy.
|
| Arguably, building a website with a login is a LOT easier and
| cheaper now than it was 10 years ago, because Let's Encrypt is
| such a well known option. If they wanted to do so 10 years ago,
| they would have most likely had to pay through the nose for an
| expensive certificate. You seem to also have forgotten about
| these people with your blanket statement about hosting websites
| being more expensive for everyone.
|
| Is the security provided significant in simple sites? Probably
| not. However, having SSL be a default is good overall. It gives
| less chances for operators to screw up because non-HTTPS raises
| very user-visible alarm bells. If your site is small and non-
| revenue generating, then why does the security alert even
| matter? It doesn't prevent anyone from accessing the website.
|
| Your 2FA argument is wrong. Sure, there may be multiple reasons
| for mandating it, but for regular users, 2FA is good defense in
| depth, that offers protection against password compromise.
| Again, the average consumer doesn't necessarily have strong
| passwords or unique passwords across services. 2FA is good
| protection for them.
|
| Also, if mining user data was the main reason for 2FA, big tech
| wouldn't support hardware security keys for 2FA. Mobile 2FA is
| a usability compromise because it targets a lowest common
| denominator that (almost) everyone has.
| throwawaysea wrote:
| I am not a security expert, but this article seems to just be
| saying that the Internet is held together by trust, convention,
| and an ever-evolving set of technologies. In the case of Level3
| (now Lumen), it seems they did not deprecate an insecure method
| that others already deprecated. And it seems that better
| technologies are on the horizon (RPKI) but not yet fully in use.
| To me this doesn't feel as bad as "spit and baling wire". We
| could be more secure by holding everyone to a stricter standard
| on adopting newer, more secure technologies. But is it really as
| broken as the title suggests? I don't think so.
| SahAssar wrote:
| > "LEVEL 3 is the last IRR operator which allows the use of this
| method, although they have discouraged its use since at least
| 2012," Korab told KrebsOnSecurity. "Other IRR operators have
| fully deprecated MAIL-FROM."
|
| I'd prefer if we kept deprecated and removed as two different
| terms. It sounds like level3 deprecated it, and everyone else
| removed it. To me (and most definitions I can find) deprecated
| basically means "don't start using it, if you are using it stop
| using it, we will remove it soon but have not done so yet for
| compatibility reasons"
| rufus_foreman wrote:
| Is there a shortage of spit and/or baling wire?
| a-dub wrote:
| reminds me of the old email interface run by network solutions
| when they were the sole registrar for everything that wasn't
| government.
| oblib wrote:
| I've been leaning on my app's users to do what they need to do on
| their end to implement offline/local first use of the app and
| they just do not get it. For them the only issues they've had
| were connection issue on their end with their service providers
| so they don't feel this is an issue of concern.
|
| But I read stuff like this, and in this case it's Krebs, so I
| have to expect these kinds of issues will pop up. The article
| mentions the FB outage and most everyone on my FB feed was
| freaking out over not being able to access it, and for the most
| part it's not a critical service. And when they came back online
| some of the conspiracies they were sharing about what/why it
| happened were way over the top.
|
| From my perspective it feels like everything on the internet is
| just one missed tap on a keyboard from breaking.
| HPsquared wrote:
| Held together with spit and baling wire as it is, the fact that
| it mostly works proves that the overall architecture is robust.
| azeirah wrote:
| If Alan Kay is to be believed (which I hope he is!) then the
| internet was originally inspired by multicellular lifeforms.
|
| I'd say the internet has some sort of a "biological"
| architecture. Robust in the sense that organisms are robust;
| extremely messy, sensical from a high-level view, chaotic from
| a low-level view.
| IncRnd wrote:
| > proves that the overall architecture is robust.
|
| Not really. What it shows is the stark difference between two
| ideologies. The first camp contains people who believe in
| Postel's Law, "be conservative in what you do, be liberal in
| what you accept from others". The second camp has people who
| recognize that the current world is not a cooperative network
| of researchers: "all input is untrusted".
|
| Krebs is absolutely in the second camp.
| shadowgovt wrote:
| But the two philosophies aren't really in contention. Proper
| adherence to Postel's law also includes accepting malicious
| traffic (and then doing something reasonable with it, like
| black-holing it).
|
| The "liberal in what you accept" part is mostly honest
| acceptance of the reality of the network: you cannot control
| the information sent to your service, only how you respond to
| it.
| [deleted]
| 234023048230948 wrote:
| Ah, Brian "dox the critics" Krebs. Reeaaaly solid guy.
| h2odragon wrote:
| It's Anti-Fragile. If it breaks all the time, everybody is highly
| experienced at patching together new workarounds, mechanisms for
| fail over are in place and regularly tested, and there's whole
| classes of corner case bugs that get flushed out to be stomped
| (or nurtured as cherished pets) instead of breeding in the dark
| and jumping out at you all at once.
|
| How can the "Internet routes around failure" be trusted without
| testing? Everything needs regular exercise or it atrophies.
| xwolfi wrote:
| It s almost alive, it'll wake up one day ask start asking
| questions :D
| fredophile wrote:
| "The internet routes around failure" hasn't been true for a
| long time. It refers to the original topography which has been
| replaced with a hub and spoke model. Remove a few hubs and you
| have disabled a large portion of the internet.
| peoplefromibiza wrote:
| back in the 90s all the Internet traffic in Italy was routed
| around two big hubs, hosted in two public universities (it
| was mainly one, in Rome).
|
| Internet is much more reliable now.
|
| https://en.m.wikipedia.org/wiki/GARR
|
| I studied CS in Rome at "La Sapienza" and I was merely 10
| steps away from INFN (national institute for nuclear physics)
| where GARR phisically resided.
|
| I spent more time there than in my class.
|
| That's how I got involved in this new "internet thing".
| paxys wrote:
| What original topology are you talking about? There was never
| a time when your end consumer device could be used as even a
| semi-reliable web server.
| h2odragon wrote:
| https://thecrow.uk/A-Gov.uk-site-dedicated-to-porn-
| Absolutel...
|
| From yesterday. Note the bit about being on a Pi. Stood up
| quite admirably to an HN hug.
|
| From 1998 to 2004 I served a fairly large amount of traffic
| out of my homebuilt pentium in a trailer out in the woods
| with an ISDN line. We stood up to several media mentions
| OK.
| cgriswald wrote:
| > There was never a time when your end consumer device
| could be used as even a semi-reliable web server.
|
| Since about 1999 I have never NOT been running a server of
| some kind off my home connection. I wouldn't run a business
| off of it that way, but it's reliable enough to count on it
| which has to meet any sane definition of semi-reliable. The
| two biggest problems I've had have been essentially
| unrelated to the internet. The first is when I've been
| violating TOS of the ISP or the power was unreliable in the
| place I lived.
| fiddlerwoaroof wrote:
| If you have a cable connection today, you can serve
| reliably just fine. Throughput isn't the best and there are
| other minor issues, but it's reliable enough for most
| people's purposes.
| lotsofpulp wrote:
| Coaxial cable connections in the US have such meager
| upload bandwidth that cable ISPs do not even bother
| advertising or specifying a minimum upload bandwidth.
| fiddlerwoaroof wrote:
| Yeah, but I've been able to stream music from my desktop
| to my phone while driving and run a web server with
| reasonable performance on one.
| lotsofpulp wrote:
| I consider multiple HD video streams to be reasonable
| performance. I have a family of 4 to 6 which at any point
| in time may be FaceTiming, video calling for work, video
| gaming, backing up to iCloud, streaming HD video from
| home NAS, and 5+ security cameras uploading.
| doubled112 wrote:
| It's more than adequate for personal use. I've also been
| serving myself and a few friends for years.
|
| Download being 300Mpbs but upload 20Mbps IS kind of
| irritating though.
|
| When did 100Mbps become popular for home LANs even?
| lotsofpulp wrote:
| Covid lockdowns helped with video calls, but ever since
| FaceTime came out, lack of upload became way more
| noticeable
|
| Also, smartphone proliferation made streaming from home
| NAS very convenient, as well as home security cameras and
| smart home features.
|
| I disagree that current upload capacities are adequate.
| With 1Gbps+ upload connections standard, we might
| actually see privacy forward solutions that do not
| require us to depend on cloud services.
| ScaleneTriangle wrote:
| Here in Canada all the modem/router combo units from
| large providers are gigabit for LAN. In pretty sure that
| it's been that way for at least 5 years.
|
| I'm pretty sure new PCs and laptops have had gigabit
| standard for probably about 10 years.
|
| Enthusiast and prosumer motherboards are now coming with
| 2.5 gigabit networking.
| Teever wrote:
| > There was never a time when your end consumer device
| could be used as even a semi-reliable web server.
|
| I dunno man, My used T420 laptop is serving several sites
| over a residential symmetrical fiber connection just fine.
| amatecha wrote:
| My first self-hosted web server was running on a 486 and
| ran for a few years hosting four or so sites, including a
| php-based forum!
| rvense wrote:
| Built-in UPS and KVM, too. Keep it cool and stock up on
| PSUs and that'll last you a long time.
| fabianhjr wrote:
| More P2P between research centers on the time of ARPANet;
| though it wasn't widely used or available back then.
| ip26 wrote:
| If by "remove a few hubs" you mean level a few major colo's
| then ok, but as far as I can tell there is not any single
| strand of glass or single switch or single server that can
| take everything down with it.
| toast0 wrote:
| The DNS root is pretty well distributed. There's 13
| different IPs run by several different organizations, and
| AFAIK they're all running anycast these days. And anyway,
| the root zone is tiny, changes infrequently and could
| really be AXFRed a couple times a month and you wouldn't
| miss much.
|
| The larger tlds aren't quite as diversely hosted and
| certainly aren't amenable to long term caching, but it
| should take a major f up to break those too.
|
| Some of the minor tlds, even the more popular ones do screw
| up from time to time though.
| throw0101a wrote:
| > _As of 11 /26/2021 10:53 p.m., the root server system
| consists of 1477 instances operated by the 12 independent
| root server operators._
|
| * https://root-servers.org
|
| * https://en.wikipedia.org/wiki/Root_name_server
| dylan604 wrote:
| You don't need to level the colos, you just need someone to
| make a typo in a router config that gets deployed live so
| the whole colo is unreachable. How many times have we seen
| an AWS/CloudFlare/otherLargeProvider have this happen to
| them?
| toast0 wrote:
| Back in the 90s when I got on, so much of traffic was
| exchanged at MAE-West or MAE-East, and a backhoe in Iowa
| could make nearly all the cross-US traffic go through Europe
| and Asia instead.
|
| These days, there are lively public internet exchanges up and
| down both coasts, in texas and chicago and elsewhere. A well
| placed backhoe can still make a big mess, many 'redundant
| fibers' are in the same strand, and last mile is fragile, but
| if my ISP network is up to the local internet exchange, there
| are many distinct routes to the other coast and a fiber cut
| is unlikely to route my traffic around the world.
| kingcharles wrote:
| God, this would happen all the time in the mid-90s. You
| joke about a backhoe in Iowa, but I'm pretty certain it
| _was_ a backhoe in Iowa one time. Then the whole of the UK
| 's traffic to the USA West Coast would be routed through an
| ISDN connection in Korea for three days. Your goat porn
| downloads from Usenet would drop from their high of
| 0.5Kb/sec and you'd be left with your dick in your hand. Or
| something.
| h2odragon wrote:
| Kids today will never know how we suffered in the Before
| Times
| xwolfi wrote:
| Ofc they will, people are recording the history of porn
| downlo... err the internet just as diligently as Napoleon
| conquests !
| [deleted]
| throw0101a wrote:
| > _Remove a few hubs and you have disabled a large portion of
| the internet._
|
| "A few hubs"? Just Hurricane Electric has presence in many,
| many IXPs, and they're not even the largest transit provider:
|
| * https://bgp.he.net/AS6939#_ix
|
| AT&T, the largest transit provider, has a ridiculous number
| of peers:
|
| * https://bgp.he.net/AS7018#_peers
|
| And there are several different major global providers:
|
| * https://en.wikipedia.org/wiki/Tier_1_network
|
| A lot of the Big Tech companies are also building their own
| private fibre networks so they don't even have to worry about
| sharing infrastructure with the telcos:
|
| * https://www.submarinecablemap.com
| unethical_ban wrote:
| I didn't RTFM but I could see the argument that the SPOFs
| of the Internet are AWS, Cloudflare and Google, not the
| cables and routers in an IXP.
| throw0101a wrote:
| The article is about BGP and Internet Routing Registry
| (IRR): routing.
| xwolfi wrote:
| You alread listed 3 single points of failure. I get what
| you mean: the entry point search engine, the global cache
| and the global backend/storage/compute.
|
| But look I live in Hong Kong, I dont feel that way: there
| are other backends, we could survive without the caching
| for a while and Google is forbidden for 1.4bn people who
| get on with it very well...
|
| Depends what you call the internet. Yes, facebook and
| whatsapp are gone the minute one of those 3 companies
| screwed up.
| User23 wrote:
| You're wrong. If say One Wilshire[1] or one of the very few
| other carrier hotels in its class abruptly ceased existing
| the Internet would be wrecked and rebuilding would be a
| matter of months at the very best. It doesn't matter how
| many peers a big telco has when the supermajority of
| backbone peerings are in a handful of buildings.
|
| [1] https://one-wilshire.com/
| kingcharles wrote:
| That building gives me anxiety attacks; just thinking
| about how vulnerable it is.
| mprovost wrote:
| I used to have a security badge to get in there back in
| 2003 or so. When I was shown around someone said they
| could cause a minor recession in 5 minutes with an axe.
| Also they had me grab a fibre that carried all of Japan's
| internet traffic. If you pulled that it would route
| everything the other way around the world through Europe
| and the east coast.
|
| I had a long lost picture when they were doing some
| street construction outside and marked all the buried
| cables with spray paint. Lines going everywhere...
| Iv wrote:
| Disabled? Throttled down the speed by half? Yes. But to
| disconnect whole regions, you have to do conscious sabotage
| and even Mubarak did not manage to switch off Egypt when he
| wanted to and gave orders to.
| krisrm wrote:
| I'm a bit shocked that MAIL-FROM auth was ever accepted, let
| alone until 2012. Even the other auth methods via email seem
| somewhat dangerous, though I sincerely hope these registries
| follow extremely strict policies for key management.
| throwaway984393 wrote:
| That's how validation of TLS certificates and domain
| registration still works.
| throwaway984393 wrote:
| Whenever people are predicting the end of the world because of
| some political or cultural upheaval, I think about the internet,
| or airport security. There are really simple ways that any idiot
| could totally fuck up either of them and cause catastrophic
| problems. But it doesn't happen. What that shows is that the
| potential for catastrophe has nothing to do with catastrophe
| actually happening. Even if the world _could_ fall apart around
| you at any moment, it 's probably not going to.
| oneepic wrote:
| ...and routers
| imchillyb wrote:
| Has no one here seen:
|
| https://www.reddit.com/r/cablefail/
|
| There are 1000s of these pictures.
|
| The entire world's IT infrastructure has been held together with
| spaghetti-noodle-cabling and bodged patches for over three
| decades now. I'm even guilty of it. Most IT guys are guilty of
| it.
|
| Don't even get me started talking about how bodged together
| corporate codebases are. Banks are perhaps the worst offenders.
| Old hardware running with bandaids and bubblegum. Software that
| more than 5 different teams have had their fingers in, mucking
| about, and some codebases have orders of magnitude more than
| that.
|
| Anyone that didn't know this, doesn't pay attention, or just
| started internetting.
| charcircuit wrote:
| How "pretty" your cable routing is doesn't effect performance /
| reliability.
| SahAssar wrote:
| Pretty in this context mostly means predictable, which does
| have an effect. If you are called to fix a broken connection
| and you get a rats nest it will take a lot longer and might
| impact other users.
| cgh wrote:
| I had a friend who worked in IT in a major Canadian bank some
| years back. He said they'd lost the source code (Cobol, I would
| imagine) for certain jobs and had taken to editing the binaries
| directly with a hex editor.
| SahAssar wrote:
| If you haven't read it
| https://www.stilldrinking.org/programming-sucks beautifully
| describes this and has both made me want to quit and start a
| farm and want to stay and improve things depending on the day.
|
| If you (the collective you, everyone reading this) haven't,
| read it.
| jl6 wrote:
| > Don't even get me started talking about how bodged together
| corporate codebases are. Banks are perhaps the worst offenders.
| Old hardware running with bandaids and bubblegum. Software that
| more than 5 different teams have had their fingers in, mucking
| about, and some codebases have orders of magnitude more than
| that.
|
| And yet, perhaps counterintuitively, the main downside of such
| systems is that they are slow and expensive to change, not that
| they are unreliable.
| dragonwriter wrote:
| > And yet, perhaps counterintuitively, the main downside of
| such systems is that they are slow and expensive to change,
| not that they are unreliable.
|
| They are slow and expensive to change because they are
| maintained _because_ they have been, at great expense and
| cost in both failures and remediation efforts, made tolerably
| reliable (but still extremely fragile) so long as things are
| exactly within certain expectations (which often have been
| narrowed from the intended design based on observed bugs that
| have been deemed too expensive to fix), and it is
| inordinately difficult to modify them without causing them to
| revert to a state of intolerable unreliability.
|
| They are systems that generations have been spent reshaping
| business operations around their bugs to make them
| "reliable".
___________________________________________________________________
(page generated 2021-11-26 23:00 UTC)