[HN Gopher] When Your Fiber ISP's 'Dumb' Device Screws Your TCP ...
___________________________________________________________________
When Your Fiber ISP's 'Dumb' Device Screws Your TCP Sessions
Author : neelc
Score : 34 points
Date : 2021-12-16 19:40 UTC (3 hours ago)
(HTM) web link (www.neelc.org)
(TXT) w3m dump (www.neelc.org)
| josteink wrote:
| Who'd think I'd be _happy_ to have a Huawei ONT for my FTTH
| setup?
|
| But reading this, clearly one can have much worse.
| mise_en_place wrote:
| I'm not sympathetic to the author at all. You're essentially
| using a home ISP for commercial purposes by hosting Tor relays.
| If you need resilience, then you really ought to colocate at a
| DC. 10 gbit is not that expensive these days, and you would
| provide your own switch like mikrotik.
| Taniwha wrote:
| What he does with the bandwidth he pays for is nobody's
| business
| superkuh wrote:
| An ISP provides an internet connection. When it doesn't provide
| an internet connection and only provides a web service with
| some internet features it isn't upholding it's side of the
| contract or the advertising. This is far worse than any "speeds
| up to $x!" lie.
|
| And it's not just tor relays that use a lot of TCP sessions.
| Pretty much all distributed protocols are going to hold open a
| lot of TCP connections. This is not a bad thing and it isn't a
| heavy resource usage. It's normal. What's abnormal are wireless
| telco style restrictions being applied in contexts where there
| is no justification for them.
|
| Saying everyone who does more than use a browser should
| colocate at a DC is disconnected from reality.
| psKama wrote:
| I am not sympathetic to you at all. Running a Tor relay
| shouldn't require a commercial infrastructure for anyone who
| wants to. Also, Tor is not the only service he mentions. What I
| am going to do with my internet is my business, not anyone
| else's. I shouldn't be limited in any way or form the way I
| want to use the internet as long as I stay within the limits of
| law.
| tentacleuno wrote:
| I remember something about this from a few years back. Can't
| recall the link now though.
|
| His ssh sessions were constantly timing out. It only happened
| when he left the SSH session to idle. It turns out his router was
| dropping the TCP sessions because it considered them dead. He got
| around it by implementing a "keep alive" packet, of sorts. Very
| interesting stuff. I don't really work at such a low level in the
| stack regularly, so it's quite fascinating to see the strange
| issues people encounter with these tools. Especially when ISP's
| meddle around with stable protocols.
|
| Also reminds me of how some ISP DNS servers totally ignore TTL
| values from DNS records[0].
|
| [0]: https://news.ycombinator.com/item?id=29568510
| viraptor wrote:
| This is a pretty common issue. See
| https://access.redhat.com/solutions/23874 The keep alive pings
| can be added on both the TCP and app level. If you ever cross a
| NAT, you will have some expiry on your connection. It's not
| really "meddling".
| fragmede wrote:
| Yes it is. A packet being sent isn't reaching its destination
| because your ISP is choosing not to forward it? That we've
| come to expect that broken behavior is the reality that we
| live in, but a different route would be for the firewall/NAT
| device to forge an RST to both ends, since it will no longer
| be forwarding said packets on that TCP connection.
|
| Given all the advances in technology, I don't think that's as
| bad an idea as it once was.
| Neil44 wrote:
| Yeah PuTTY has the keepalives option for exactly this reason.
| My home router doesn't seem to need them but when I'm out and
| about on 4G they help. You also have the SO_KEEPALIVE option on
| TCP connections in general.
| kevingadd wrote:
| Glad I didn't pick CenturyLink for fiber when I moved here, but
| Wave G's incredibly unreliable in its own way which makes me
| wonder if they're using the same hardware. Kinda wish I picked
| Google Fiber.
| lotsofpulp wrote:
| Where is this place that has 3 fiber ISP choices? It is hard
| enough to find residences with 1 choice of fiber ISP. I have
| yet to see a single residential location in the US that has
| more than one option for a fiber ISP.
| lipnitsk wrote:
| FWIW I'm also on CenturyLink FTTH and just a week or two ago
| noticed latency spikes and packet loss which magically went away
| after 15 minutes. Good to read this analysis for future
| reference. I really wish end users had more control over ONT
| boxes similar to how we can use own modems for cable/DSL. A
| DOCSIS-like provisioning by ISP should be possible.
|
| Off topic, but CenturyLink Fiber still uses PPPoE and 6rd instead
| of native dual stack in many markets and are unwilling to upgrade
| to more modern configurations.
|
| EDIT: I do not use Tor at all.
| zokier wrote:
| > A DOCSIS-like provisioning by ISP should be possible
|
| GSM solved provisioning 30 years ago with SIM cards, any reason
| why ONTs couldn't employ similar system?
| lipnitsk wrote:
| Good suggestion and question. Another challenge for bring-
| your-own-ONT is making a clean fiber connection without
| expensive tools, but I would imagine that's also solvable.
| mindslight wrote:
| My ONT has a standard single SC connector. The only custom
| splicework on the install was the run from the street to
| exterior junction box [0]. From there it's an off the shelf
| single mode SC-SC cable to the ONT.
|
| Knowing nothing about the GPON protocol, what does the ONT
| actually contain to authenticate to the network? Is there a
| public key and all that, or is it just keying off the
| device's serial number? Would it be possible to replace the
| ONT with a better documented model of your choosing that
| you have flashed with the appropriate identifiers?
|
| [0] I'd call it the demarc as it's next to the phone
| demarc, except the ISP provided the SC-SC cable and ONT. So
| the real demarcation point is the ONT ethernet port.
| PeterisP wrote:
| Quoting the article, the cause is identified "The Calix 716GE-I
| ONT device is working as designed by activating Denial of Service
| (DOS) attack prevention when too many connections are
| established, which includes jumbo or small packets". Sounds like
| a reasonable feature for residental devices, even if it isn't
| compatible with the niche usecase of running a Tor relay.
| bin_bash wrote:
| Presumably OP could subscribe to business fiber from
| CenturyLink and avoid having the device installed.
| lipnitsk wrote:
| Why not make it configurable by advanced users though?
| PeterisP wrote:
| Probably the expected market for advanced users who would
| need this particular feature is tiny. Like, for the Tor relay
| usecase, there are something like 6000 relays worldwide, most
| of them probably provided by various organizations (where a
| single operator runs many relays) instead of hobbyists, most
| of them outside USA, and the vast majority of them using some
| entirely different network connection not affected by this
| particular device model in any way. The described scenario
| ("10000s of concurrent TCP sessions") is literally an edge
| case for residental use; the article does follow up with
| "What about BitTorrent or cryptocurrency and Web 3.0 apps?"
| but none of those have network behavior like that.
|
| Like, perhaps this problem is also affecting other kinds of
| usage, but the original article does not attempt to claim
| that, and purely from their example it would be generous to
| assume that literally dozens of individuals would need this
| feature and, well, it's not worth to make and test features
| (even if they're just a configuration option) in this case.
| AnthonyMouse wrote:
| The problem with this logic is that ordinary users don't
| become the target of a denial of service attack either. If
| it should exist at all, the default should be _off_. And if
| then no one would turn it on, it could just as well not
| exist.
| phantomread wrote:
| I might be misunderstanding but doesn't the feature also
| help prevent home users' devices becoming part of a DDOS
| effort (high number of outbound connections)? There's
| stories here on HN about IoT devices and infected
| PCs/phones participating in DDOS on command. So I can see
| an argument that a home gateway device should try and
| help prevent participation by devices behind it.
| zokier wrote:
| > The problem with this logic is that ordinary users
| don't become the target of a denial of service attack
| either
|
| I suspect the concern is not that ordinary users would be
| targets, but that ordinary users would be sources of
| ddoses (by being part of botnet)
| jcrawfordor wrote:
| Ordinary users become a target of DDoS _way_ more often
| than you would think. These days it tends to be related
| to competitive multiplayer video games, but I 'm sure
| there's still some IRC drama and small-time Minecraft
| hosting driving it.
|
| In general it's extremely unlikely unless you are
| engaging in "high risk behavior," but at the scale of an
| ISP there are enough users doing that kind of thing
| (Twitch streaming, etc) that it becomes an appreciable
| frustration for your network operations.
| jcrawfordor wrote:
| Honestly, for various structural reasons, hobbyists are
| sort of actively discouraged from running Tor relays. It's
| less of an issue with middle relays than guard or exit but
| in practice Tor has a strong reliance on trust in relay
| operators, so small-bandwidth relays popping up onesy-twosy
| is much less desirable than institutional operators with
| significant resources.
|
| Which is all just one reason that, of the set of people
| running Tor relays on residential internet connections, I'd
| wager a solid 99% shouldn't be.
| throwaway984393 wrote:
| > But what if a large number of TCP connections is intentional?
|
| Sorry, that ship sailed long ago. Carriers have forever put
| restrictions on how their customers can use their internet
| connections, such as "no hosting servers" or even not getting a
| routable IP address. Traffic shaping is part of the deal too.
|
| I think the only means we have to change the situation (in the
| face of a lack of competition) is to lobby for municipal
| internet. Or start a company.
| rubatuga wrote:
| We started a company called Hoppy Network that does away with
| ISP bullshit like CGNAT. As long as your ISP doesn't block UDP
| packets, you're set.
|
| I talk about the rationale here:
|
| https://www.naut.ca/blog/2020/12/30/launching-a-new-service/
| superkuh wrote:
| It seems like all the real Internet Service Providers have died
| and all we're left with is web service providers with an
| incomplete internet implementation. This started with the
| wireless telcos where it was almost justified; they were late
| to the game and didn't have enough IPv4. But for established
| holders of large IP spaces this is exploitation if not outright
| fraud.
| jcvhaarst wrote:
| So ISP delivers router that breaks your internet, and they won't
| replace it with a real ONT? Then why not simply replace it
| yourself? As long as it isn't PON, but just plain AON, that
| should be relatively straight forward.
| OldTimeCoffee wrote:
| There's authentication between the ONT and OLT that you would
| have to either implement or relay. This is an edge case because
| of running Tor. The average user isn't going to run into these
| problems.
| notwedtm wrote:
| I think that's the crux here. "But I need to run an open Tor
| proxy!" is going to get some weird looks, especially on
| residential connections.
| xxpor wrote:
| An ISP is selling me fiber to transmit bits and an IP
| address to talk to the rest of the world. How many TCP
| connections I'm establishing is exactly none of their
| business unless they start receiving abuse reports (or run
| CGNAT, but that's not the issue here).
|
| Whoever thought a *stateful ONT* was a good idea should be
| shot out of a canon.
|
| Just wait until the connection timers in the ONT don't
| match your firewall. Then you'll have real fun.
| jcrawfordor wrote:
| The service that you're describing is usually called
| dedicated internet access or DIA. It is a distinct
| service from residential ISPs, and a more costly one for
| good reasons. Residential and business ISPs operate a
| shared resource on which they must impose limits to avoid
| impacts on other customers. This is as true of PON as
| other last-mile technologies.
| mise_en_place wrote:
| What a wild take. I was going to start a WISP but
| realized early on I would get potential customers like
| this. Let's be real, you're being cheap and don't want to
| shell out extra cash per month for a business line or
| colocation.
___________________________________________________________________
(page generated 2021-12-16 23:01 UTC)