[HN Gopher] Resolving an Unusual WiFi Issue
___________________________________________________________________
Resolving an Unusual WiFi Issue
Author : slimsag
Score : 174 points
Date : 2022-08-19 18:48 UTC (4 hours ago)
(HTM) web link (blog.ando.fyi)
(TXT) w3m dump (blog.ando.fyi)
| valleyer wrote:
| Nice debugging. Do I understand correctly that the registry reads
| aren't actually the cause of the problem but rather just a signal
| that a QNetworkAccessManager is active and causing a scan?
|
| If so, is there a better routine to break on in the debugger to
| see it actually initiating a scan?
| muststopmyths wrote:
| You can see a hint in the debugger screenshot. The call is not
| directly a registry read but to the iphelper API. There are
| functions in there that enumerate adapters.
|
| Knowing nothing about this scanning process, I'm just assuming
| they first enumerated wireless adapters. So you could start
| with iphelper and then explore deeper into how you tell the
| card to scan. There's probably some API for that as well.
| valleyer wrote:
| Ah, good point. Wish the backtrace were fully symbolicated
| (for the system DLLs, at least). Thanks!
| AnotherGoodName wrote:
| Anyone ever had arp storms on a home network?
|
| I've had something trigger every device into asking 'who has xyz
| ip tell [Mac]'. It made the network unusable and even rebooting
| systems it would come back since as soon as one device asked the
| question (sent the broken arp packet) all other devices decided
| they too needed to know.
|
| The solution that worked was to flip the circuit breaker for the
| whole house and reset every network device at once.
| xoa wrote:
| Depending on what network gear you're using (I'm now switching
| to Omada, previously/still on some site son UniFi, but lots of
| even 'prosumer' stuff does this) there are specific mitigations
| available. All of this falls under the heading of "managing
| broadcast traffic", which is very important even for smaller
| networks. The three major categories of traffic on a network
| are unicast, broadcast, and multicast. Unicast is the normal
| case of one device talking to a single address. Broadcast
| involves sending a packet to every single possible recipient in
| the entire broadcast domain (almost always the subnet). ARP
| fits in here. Multicast is essentially in between, more
| efficient then broadcast, can still talk to multiple devices
| that have signed up to hear it.
|
| Obviously an actual broadcast storm can take down an entire
| network, but excessive broadcast traffic on WiFi specifically
| can also suck up a huge amount of airtime for little bandwidth.
| Every single device has to go to the slowest speed and stop
| what they're doing to listen and make sure no one is left out.
| Using STP/RSTP with proper values set and LACP for aggregated
| interfaces can help prevent inadvertent network loops. Many
| switches also support some kind of port isolation and explicit
| per port storm control restricting max numbers of
| packets/second for unicast/broadcast/multicast traffic. WiFi
| APs can use proxy ARP to cut it in their domain too. The WAP
| already of course knows the MACs of every device connected to
| it by definition, so there isn't generally any reason not to
| have it answer ARP requests on their behalf then forward the
| traffic itself.
| donmcronald wrote:
| Why the switch to Omada? I'm looking at doing the same due to
| supply issues with the UniFi stuff. Any tips or opinions on
| Omada vs UniFi?
| xoa wrote:
| Opinions hooboy do I have them ;), I was just asked that
| exact question on HN a couple of weeks ago in fact. Gave it
| a shot in a response here [0] which still applies. But
| basically Ubiquiti has become a toxic dumpster fire of a
| company and their product lines (UniFi in particular) on a
| downward trajectory in terms of performance, features and
| stability for quite a while. I had a certain amount
| invested in UniFi (think the final total will end up as a
| few hundred devices) so it's been a staged switch, with a
| total change of all routing/gateway/security functions to
| OPNsense completed first. That bought a lot more runway,
| it's always been the weakest and most neglected area in the
| ecosystem while obviously also being pretty critical. Yet
| the Ubiquiti debacle has served to underline for me how
| valuable self-host is, I've been able to have a nice slow
| ramp and deal with their implosion precisely because
| UniFi/UNMS/UISP and all the hardware is fully under my
| control. So I've been hoping someone would come along and
| see the potential of the UniFi niche of the networking
| market and basically copy it without all the junk. Which
| seems to basically be Omada to a tee.
|
| I'd actually originally (and still at many sites) intended
| to hold off and wait for WiFi 7 gear, because at that point
| a bunch of clients (and myself for that matter) will be
| interested in replacing WAPs _anyway_ which is a very
| natural point to consider changing manufacturer as well.
| But a breaking point has come at a few places with a final
| feature which is PPSK, allowing the system to have many
| different passwords for an SSID that can be assigned
| different tags. Basically it allows having many of the
| benefits of WPA-Enterprise in terms of segmenting different
| clients onto different VLANs and revoking credentials and
| the like with more security and less manual work than MAB
| (MAC bypass) while still looking like a normal PSK scheme,
| which means the vast universe of brand new stuff which
| doesn 't support 802.1x and never will works with it
| happily (by the same token none of that is going to play
| directly with using a secure virtual network or other
| better systems either sadly). Lower overhead and better
| compatibility than captive portals for non or semi-
| interactive devices as well. Someone hacked together a demo
| showing this could work on UniFi WAPs like four freaking
| years ago and Ubiquiti never did anything with it in favor
| of endless bikeshedding GUI changes to add more white space
| and hide important features and information (yes I'm a
| touch bitter).
|
| So I'm not in the position of wholeheartedly recommending
| Omada yet, I don't have years under my belt there and it's
| relatively speaking fairly new. It has its own warts and
| rough edges for sure, from the software to the hardware
| physical design. But it can be self-hosted and the
| trajectory looks massively better, has already had more
| meaningful improvement in months than UniFi has had in
| years, seems to perform much better so far as well.
|
| Of course the Venn diagram of self-hosting, herding lots of
| hardware with single pane, fully networking features,
| ecosystem richness and so on is pretty minimal in the
| overlap. Take away any one or multiple of those and options
| expand a lot, Aruba InstantOn for example.
|
| And welp, this didn't end up "basically" at all did it,
| sorry about that. I am bummed by the sheer wasted potential
| with Ubiquiti. So it goes in tech over and over again
| though, we've all seen this movie many, many times.
|
| As far as tips, I would suggest if you plan to stay on the
| managing-your-own-networks route to very strongly consider
| having the router/gateway stuff be separate and fully open
| source as I ended up. Doesn't have to be OPNsense, could be
| VyOS or plain OpenBSD or whatever else you're most
| comfortable with and depending on how you want to manage
| stuff and what needs there are for others to take over. But
| it's very, very pleasant to have the full spectrum of
| quality PC hardware available, you can get far more power
| for less, and you're never stuck with a critical aspect.
| I'd still suggest generally running that on metal rather
| than virtualizing it in a (semi)production network, but
| opinions vary there.
|
| ----
|
| 0: https://news.ycombinator.com/item?id=32297556
| klysm wrote:
| I haven't had a problem where rebooting the house has fixed it
| thankfully
| mh- wrote:
| Yes, generated by AT&T's wireless STBs (cable boxes) back
| around 2014-15. It was a nightmare to figure out, as I recall.
| eikenberry wrote:
| I have had 2 external USB-power-passthrough laptop doc/hub
| things with an Ethernet port. They both cause a packet storm on
| the network if you unplug the computer and leave the ethernet
| and power plugged in. Causes all my crappy realtek NICs to
| overheat and flake out. Not exactly the same but super
| annoying.
| jfim wrote:
| I had the same issue with an USB Ethernet adapter plugged
| into a powered USB hub, disconnecting the computer would make
| the network crap out. Kind of defeats the purpose of having a
| docked setup with wired Ethernet if one can't undock it or
| needs to unplug tons of cables each time.
| dylan604 wrote:
| That takes did you turn it off and on again to an entirely
| different level. What if we need to do this city wide?
| wizofaus wrote:
| That sounds like a good plot for a movie - the day we had to
| reboot the entire internet...
| Lammy wrote:
| "South Park already did it":
| https://en.wikipedia.org/wiki/Over_Logging
| wizofaus wrote:
| Hah, yeah I did originally think something along those
| lines, but I wonder if you could actually do it non-
| comedically (e.g. somehow every node connected to the
| internet has to all be switched off at the same time and
| restarted to restore connectivity). Most likely it's
| already been tried too I guess.
| dylan604 wrote:
| usually, it's a nefarious evil doer that is threatening
| this action in the movies vs we gotta reboot the city.
|
| there's been reboot the sun plot. there's been reboot the
| earth's core plot. there's been reboot the machine
| running the city plot.
| wizofaus wrote:
| I was thinking more along the lines of a rogue self-
| replicating packet that every last instance of had to die
| before routers and switches etc. would start working
| again.
| neurostimulant wrote:
| Not sure if it's something similar but I had issue where
| attempting to setup a wifi smart plug locked up the router for
| a minute (until the smart plug gave up trying to connect to the
| router). Wired ethernet still works but the routers show 100%
| cpu usage on its management interface and the 2.4ghz wifi
| stopped working (didn't check the 5ghz one). I didn't dig in
| more because my wife was in a zoom meeting.
| networkwanderer wrote:
| I love these debugging stories but its a total nightmare to deal
| with these kinds of issues.
|
| At the moment there's this really weird network issue we're
| having where iPhones are unable to play Netflix on the Wifi.
| Every other device works fine but iphone 7, 8 (2 devices) and SE
| can't stream Netflix. I noticed there is other things they can't
| do, for example the page for the fast.com speedtest loads but the
| speedtest cant be performed. Same with the Google speedtest. The
| phones also can't access Apples update server on the wifi. Other
| network stuff does work fine, youtube works, browsing works, etc.
| The behaviour is consistent across the iphones and all these
| things work fine on multiple other devices on the same network.
|
| I can't make sense of it at all.
|
| Called internet provider and they didnt know either apparently
| other people had the same issue but nothing has changed from
| their side of things. Called Apple support and they are putting
| the blame on the network provider.
|
| Tested one of the devices on a different wifi network and works
| fine.
|
| AFAIK if you're an internet router the packets look the same no
| matter what device is being used so I think this must be some
| Apple software issue. Or maybe my router is cursed.
| mrb wrote:
| I'm fairly confident you are victim of a PMTUD black hole. It's
| easily fixed by making your router force the TCP MSS to a
| slightly lower value.
|
| All your symptoms are explained by this (some but not all
| devices are affected, some but not all services are
| unreachable).
| cryptonector wrote:
| Or better yet, use packetization PMTUD. 15 years after that
| was published by the IETF, every device should implement it
| by default.
| bsagdiyev wrote:
| Xfinity had a router that did not work with the Xbox One when
| it came out. Wired was fine but wireless just did not work. I
| believe a software update fixed it and I don't recall if it was
| the router or the Xbox that was ultimately the issue.
|
| Fielded a lot of grumpy calls that Christmas morning.
| daze42 wrote:
| Any chance it could be MTU related? Sounds like the issue is
| only popping up when attempting to use full packets.
| mbreese wrote:
| I was thinking DNS and/or IPv6.
|
| For me, it's always DNS until proven otherwise. But the
| difference of some sites loading, but others not makes me
| suspect there's a split somewhere and IPv4 vs 6 seems as
| likely as anything.
| UltimateEdge wrote:
| noncoml wrote:
| My MacBook Pro(x86) takes 10ms to ping the Asus router. Any other
| router, 1-2ms. Windows takes 1-2ms for the same ASUS router.
| chedabob wrote:
| Reminds of the issue on Mac OS where Location Services would
| cause ping spikes https://news.ycombinator.com/item?id=31356730
| hbn wrote:
| Here's a spicy meatball for you: wifi lag spikes caused by
| placing an AirPlay button in the touchbar
|
| https://mnpn.github.io/blog/airplay-network-disaster
|
| Previous discussion:
|
| https://news.ycombinator.com/item?id=31706283
| m463 wrote:
| I wish I could decisively turn off airplay on macos.
|
| It's the source of so many weird issues.
|
| For example, locked down mac, using wifi at a friend's house
| and their LG tv shows up as an airplay mirroring device. Why
| should my machine be discovering that TV without me asking?
| When I'm on a public network, I'd like to make my machine
| output-only, not promiscuous in this way.
|
| there was also an issue where a macbook would randomly lose
| its onboard sound and somehow default to using a nearby
| appletv as the output device.
| jabbany wrote:
| > I'd like to make my machine output-only, not promiscuous
| in this way.
|
| The TV advertises itself on the network so it's the one
| being promiscuous. Your machine is still being passive, it
| just shows you the devices that are have advertised
| themselves.
|
| No idea about the sound thing though, I don't use any Mac
| stuff :)
| yellow_lead wrote:
| > At the exact same time the lag spikes occur, MBPP starts
| querying the registry keys for all of the network interfaces.
|
| Can anyone tell me why this causes the network issue? Don't
| understand how querying the registry can cause this.
| Arnavion wrote:
| Querying the registry does not cause the issue. Querying the
| registry is a thing that Qt does as part of the code that
| causes the issue.
| zdw wrote:
| My most unusual wifi issue was on a system that said it had a
| great connection (SNR) and was running at high speed, but would
| just not pass traffic if it was further than about 1m from a base
| station.
|
| Turned out to have multiple antennas, and the transmit antenna
| was broken, so it could receive just fine, but not transmit over
| anything but short distances.
|
| Sometimes the physical layer is the problem, even if the logical
| layer says everything is fine.
| klysm wrote:
| I guess the logic layer only said the receive side was fine. It
| can't see what's happening within the transmit antenna I don't
| think
| Nextgrid wrote:
| I'm surprised there isn't a mechanism for the link peer to
| report the SNR to the sender from its perspective.
|
| I've had the same issue without any antenna troubles - Mac
| would constantly connect to the 5GHz network and struggle to
| send any packets out, yet the displayed signal strength was
| good. It turns out it was able to "hear" the AP just fine,
| but the AP had trouble hearing back, yet somehow there's no
| feedback mechanism for it to know.
| klysm wrote:
| A kind of SNR ping seems like it would make a lot of sense.
| I don't know enough about radios but anybody know why that
| doesn't exist?
| Fitilii wrote:
| proactivesvcs wrote:
| I've got a customer enjoying this exact fault at the moment, what
| luck.
| josephcsible wrote:
| The "new, highly-reviewed, wifi adapter" pictured in the article
| looks like Alibaba garbage. I'm surprised it didn't cause some
| worse problem.
| BruiseLee wrote:
| I have a company-issued laptop with some corporate spyware
| installed. I'm not actually required to use it for development,
| so I don't use it. But I have to switch it on from time to time
| or else I get a nice email from IT.
|
| Anyway whenever I switch it on my wifi goes to shit. Apparently
| it does some SSID scanning every 5 seconds and then keeps sending
| the scan result to the "mothership". So I switch it on once or
| twice a week for an hour or so to do its spying thing.
| mrlonglong wrote:
| Time that laptop had an "accident"
| mixmastamyk wrote:
| Leave it at work. Say you don't allow "rogue devices" on your
| network.
| m463 wrote:
| > or else I get a nice email from IT.
|
| what would it say? how ridiculous.
|
| I'll bet you can't insert USB flash drives either.
| RicoElectrico wrote:
| The "proper" solution not depending on the application layer:
|
| https://www.reddit.com/r/GlobalOffensive/comments/3ahg59/fix...
|
| Turn autoconfig back on only when you restart your PC or
| disconnect from the network (maybe someone can automate this by
| checking connectivity without scanning networks, enabling
| autoconfig, and then turning it back off)
| zionic wrote:
| Wow. This should be the kind of thing windows/your OS detects and
| warns you about.
| jtchang wrote:
| I believe this can happen on macos as well.
| icedchai wrote:
| I've seen this happen on my Mac desktop. I eventually found a
| post where someone mentioned turning off "find my mac" fixing
| it: https://developer.apple.com/forums/thread/97805?page=2
| LaputanMachine wrote:
| In macOS there are also ping spikes when you open the WiFi menu
| bar [1]. This still happens in macOS Monterey.
|
| [1]: https://developer.apple.com/forums/thread/97805
| nicoburns wrote:
| This seems more reasonable, given that:
|
| 1. I can't see how you'd enumerate wifi networks without
| degrading network performance.
|
| 2. The user has specifically initiated a wifi-related action.
|
| 3. There's not generally any need to use that menu unless the
| network is already not working.
| walrus01 wrote:
| If I had to guess it's because the system is temporarily
| pausing tdd wifi traffic while it scans the 2.4 and 5.x GHz
| bands to see what SSIDs are broadcasting.
|
| It is a bit of a trade off since if you want to see every
| possible available AP, even the shitty ones with signal
| levels at like -80, you can't be noisy on your own radio at
| the same time as you scan the band.
|
| Remember it's a half duplex medium.
|
| It does it even _more_ if you hold down option and click the
| wifi menu bar, to get detailed signal strength /info on the
| AP you're presently connected to.
| [deleted]
| lobsterboix wrote:
| This is incredible, we use QT5.12 for an embedded device
| application, and this issue has been a really weird one for us
| and this spot on resolves it! See browsing HN at work does pay
| off!
| naikrovek wrote:
| I don't think I would have ever figured this out, if it happened
| to me.
|
| the very first thing I do when I have a problem on wifi, is to
| remove wifi from the equation. wired Ethernet is so much better,
| and so far, the problem always disappears.
| brainzap wrote:
| I hate wifi issues so much, specially when you stream audio.
| wizofaus wrote:
| Definitely my biggest bugbear with relying in streaming for
| music - it's the one thing I use apple hardware for (apple tv)
| and I can't even sync music to it anymore, plus there's no
| ability to control buffering that's worked so far. So I'm stuck
| with listening to music knowing it might stutter at any
| moment...
___________________________________________________________________
(page generated 2022-08-19 23:00 UTC)