[HN Gopher] Belgian ISP under 250 Gbps DDoS for days on end ___________________________________________________________________ Belgian ISP under 250 Gbps DDoS for days on end Author : laurensr Score : 230 points Date : 2021-09-18 18:57 UTC (4 hours ago) (HTM) web link (issues.edpnet.be) (TXT) w3m dump (issues.edpnet.be) | s800 wrote: | I got hit with a ~40Gbps DDoS last week. These attacks are on the | rise. Some responses to folks above: Success working with | upstreams is quite varied. Some care, some don't, and it can be | difficult to get to folks that can help- even if their networks | are impacted as well. Some carriers immediately turn this into a | sales opp. - buy more bandwidth, buy more services. | | In our case it was based on DNS reflection from a large number of | hosts. I've contacted the top sources (ISPs hosting the largest | number of attackers) and provided IPs and timestamps. I've | received zero responses. | | Geo-based approaches yielded no helpful reduction in source | traffic. | | Also, during this event we discovered an upstream of ours had | misconfigured our realtime blackhole capability. As a result, I'm | going to add recurring testing for this capability and burn a | couple IPs to make sure upstreams are listening to our rtbh | announcements. | | Very concerned about the recent microtik CVE, as that is going to | make for some very large botnets. | | Personally this all is very disappointing because it creates an | incentive to centralize / de-distribute applications to a few | extremely large infrastructure providers who can survive an | attack of these magnitudes. | pilsetnieks wrote: | Can you give more detail on what you mean by the "recent | Mikrotik CVE?" As far as I understood, the recent botnet was | utilizing devices still unpatched since the original issue | (2018?), as well as credentials gathered then even for patched | devices that were affected but passwords weren't rotated. | alcover wrote: | I fail to understand how DNS reflection attacks are possible. | Isn't it in the interest of any ISP to block outgoing spoofed | IP packets ? So as not to be accused of letting those attacks | originate from them ? | spc476 wrote: | I'm no sure how things are now, but 15 years ago (back when I | was doing network admin type stuff), most routers did | destination routing in hardware, source routing hit the CPU | and was thus, very slow. I think that was the primary reason | not to do that back then---perhaps it's still the case now? | kortilla wrote: | The ISP originating the spoofed packets isn't apparent to the | person receiving the attack. The source is spoofed to the | victim's address so neither the DNS operator nor the victim | can see where the spoofed packets originated. | alcover wrote: | I know. But one layer below, there's still a chain linking | back to the origin ISP. The DNS operator can ask his own | ISP about it. | | (Also, to people down-voting a genuine _question_ ? wth..) | codexon wrote: | The mac addresses on the ethernet layer are rewritten at | each hop. You would only see the mac address of the | router your router is connected to, not a chain linking | back to the origin. | javajosh wrote: | Anecdotally I've noticed bot-like behavior on HN down- | voting posts almost immediately, before a human could | have read them. This is followed, typically, by a | correction over a few minutes and then regular up- (or | down-) voting resumes. I can imagine that HN is subjected | to quite a lot of novel attacks of this nature, and I | don't think it's in anyone's interest that they broadcast | the details of it. My advice: be patient and take | downvotes with a big grain of salt. They may not mean | what you think they mean. | toast0 wrote: | If you're thinking of residential ISPs which own the IPs that | their customers use, it's mostly pretty simple to prevent | spoofing (or at least, prevent spoofing outside their address | range), but once you get customers that can bring their own | IPs, it can be more cumbersome and the default is to not | care. | | There are, of course, proposals and protocols to make things | better, but not caring is easier, and strict enforcement is | hard, especially as the bandwidth goes up. | | There's also enough ISPs that don't do egress filtering that | name and shame isn't effective, and anyway, what am I going | to do if I'm connected to an ISP that doesn't filter? I have | exactly one meaningful offer of connectivity, so that's the | one I've accepted, regardless of its merits. Many ISPs have a | similar hold on customers. | alcover wrote: | Thanks. | | _> but once you get customers that can bring their own IPs | (...)_ | | You know these IPs though, since you route to your | customer. So you could drop the offending packets. (I may | be getting over my head here). | vgb2k18 wrote: | > I may be getting over my head here) | | It may be so, however my own need for clarification was | the same as yours. If ISP xyz operates a /16 ip block, | and only forwards source packets from its /16... Then, I | guess, my question: how do spoofed packers usually travel | past their first hops on their route to the target? | | [edit] after-thought: I imagine attackers might easily | spoof their source as any of their ISPs /16 or /8. Feels | like that's not entire story here though. | toast0 wrote: | The routing need not be symmetric. If I've got two /24s | and two ISPs, maybe I advertise each /24 on only one ISP, | for traffic engineering purposes (I want to steer some of | my clients to send through one ISP and some clients to | send through another), but that doesn't mean I want or | need to send the return traffic back through the same way | it came in. | | Also, the routing/BGP configuration is often separate | from the filtering config, so making sure things are | synchronized can cause problems. | | It's far less time intensive as an ISP to just not filter | once it starts getting complex. After all, you don't get | a lot of customer service calls when you let ill-advised | packets through, but you do get calls when you break | things by dropping packets your clients were authorized | to send. | codexon wrote: | Blocking outgoing spoofed packets requires time and money. | | No carrier will shut down an ISP paying them 100k/month just | because of some spoofed traffic. | CircleSpokes wrote: | DNS based amplification has been very popular for many years | now. By this point if a DNS resolver is still being used by | amplification no email or contacting the ISP will do anything | as they have received many other similar emails. | flatiron wrote: | Dns is so broken in many different ways. It's crazy it's | still the back bone of the internet when it's insecure awful | nonsense. It's just ingrained. | temptemptemp111 wrote: | What is the lowest level hardware that can help defend against | DDoS on the backbone level? Or is that not a thing? | vadfa wrote: | A 250 Gbps attack is bringing an ISP down? How is this possible | in the age of 1 Gbps connections in every home? | BenjiWiebe wrote: | 50 1Gbps connections usually peak at less than 50Gbps. | Depending on your demographic and area, it can be a lot less. | | 250 Gbps will service much more than 250 1Gbps connections. | | Also, not nearly every ISP offers 1 Gbps to all their | customers. Some ISPs don't offer it at all. | tyingq wrote: | There's also things like local Netflix/Google/etc caching | boxes where you can serve downsteam traffic without touching | your upstream as much. | KindOne wrote: | They seem to only offer 20Mbps - 500Mbps. | | https://www.edpnet.be/nl/thuis/internet.html | | https://www.edpnet.be/nl/business/internet.html | laurensr wrote: | They are mainly offering VDSL2 connections over phone lines. | They also offer Fiber over a monopolist's (called Proximus) | GPON infrastructure, but the rollout of that GPON network is | still in early stages. However, that monopolist has pledged | to accelerate that rollout in light of the increasing work- | from-home statistics following the pandemic. | | https://www.proximus.com/news/2020/20201208-proximus- | brings-... | NullPrefix wrote: | overselling | MeinBlutIstBlau wrote: | Just because everyone has 1Gbps doesn't mean if everyone tried | downloading something they'd get it. The internet is tunneled | through a system where the more people are on it, the more | likely that the bandwidth entirely gets used up. It's literally | like the phone systems back in the day where if everyone picked | up the phone, not everyone would get a dial tone. | storyinmemo wrote: | It's like a large city in a way. Most of your roads are local, | and so is most of your traffic. You try to distribute load and | makes trips as short as possible. For a city its grocery stores | in neighborhoods. | | For an ISP, they do this by peering locally with major | providers. Google, Twitter, Facebook, Netflix, etc. all have | very large networks and will openly connect at Meet Me points | in open and private internet exchange sites as will CDN | providers. | https://en.wikipedia.org/wiki/Internet_exchange_point. Some | providers such as Google and Netflix will even install local | caching servers that you can reach from your network if you | have enough traffic. As a small ISP, you try to take advantage | of all of that. | | Your option of last resort is the Tier 1 upstream provider. | That's for anything where you can't get to it via local transit | peering. This should be a smaller percentage of your traffic | and it costs more than peering arrangements. The nature of a | DDOS is going to fill that smallest and most expensive | connection that provides the filler access to the majority of | the world but doesn't handle the majority of the traffic. | | Even as a hosting provider, it doesn't have to be overselling. | If you have 250 1Gbps customers and 250 Gbps of bandwidth, | you'll still be useless in the face of 250 Gbps of junk | traffic. Pick a random packet and tell me what should decide if | it is dropped is something that has to be determined upstream | of the saturated link. | | EDPnet looks pretty well-connected too: | https://bgp.he.net/AS9031#_ix | pt_PT_guy wrote: | and no one accountable :-) | forty wrote: | Why are the people doing those attacks doing them? Ransom? Or are | they just doing that to prove and advertise/demo their capacity | to harm? | tigerlily wrote: | How can I check my home network for signs I have any devices | participating in a botnet? | mfkp wrote: | Generally large spikes in upload data usage is the best | indicator. Many consumer grade routers are starting to build in | per-device bandwidth usage, which helps narrow down the | culprit. | system2 wrote: | Invest in something like a cheap sonicwall. Without bunch of | licenses, it can still show you all the traffic information and | act as a very powerful firewall with all types of blocking. You | can search "sonicwall soho". | | If you don't want to invest, you can also use pfsense. You can | use an old computer to turn it into a perfect firewall. | antattack wrote: | I used both pfsense, opnsense and none had good facilities to | keep track where your packets are going. | ineedasername wrote: | I'm very much not a network engineer, but I'd like to understand | the magnitude of this issue because my intuition _is wrong_ : | | 250 Gbps seems like it would definitely be a lot for a server or | website but it also seem like a drop in the bucket for an ISP | providing broadband for many customers. | | _Clearly I 'm wrong_ because it is an issue here. I'd like to | understand _why_ I 'm wrong, and I hope that here, on HN, that's | taken in the spirit of curiosity intended and not negativism. | | So, what am I missing? Maybe Belgian broadband is lower capacity | than what I'm used to in a US metropolitan area? Maybe this | particular ISP served a population too small to have a... um... | "fat pipe"? I'd like to understand. | xhrpost wrote: | 250Gbps is still a lot of bandwidth for a network even today. | An enterprise ISP like Verizon or Spectrum may have less of a | capacity issue with it but for many other networks this is a | lot. | | Quick example, though not an ISP, but the entire Internet | archive operates on 60Gbps of bandwidth. So an attack of 250 | would be quite a problem. | | http://blog.archive.org/2020/05/11/thank-you-for-helping-us-... | ineedasername wrote: | _the entire Internet archive operates on 60Gbps of bandwidth_ | | Wow, yes, that definitely puts the magnitude of this into | perspective. Thank you. | Groxx wrote: | Though that's a substantial amount of traffic: isn't this | currently going through one ISP, which (I would hope) has | _many_ more customers than just archive.org? "top 160 site" | is significant, but still minuscule compared to all internet | traffic. | Hikikomori wrote: | An ISP has two types of edge connections where ddos traffic | comes into their network. Private peering where they are | directly connected to their neighbor in one or more locations, | typically neither party pays for traffic, can be 1 to 100G | interfaces and multiple of them. Second type is transit, | smaller or mid sized buys it from larger ISPs and pay for | traffic. Everything that does not go directly to a peer | mentioned before goes over this connection. Can also be very | different in size. | | You can just add upp all edge Links and know their theoretical | limit, but most of the ddos traffic likely comes from Asia and | it's unlikely that they peer directly with those ISPs. So most | of the traffic will go over transit, but the larger ISP likely | has an automated system that handles volumetric attacks. So if | there's any impact it will be on transit mostly affecting | traffic to/from outside of Europe. 250 might be a lot for this | ISP, but they might also have a lot more transit headroom, or | it light be handled by their transit provider. Who knows. | ineedasername wrote: | As an outsider to network topology that's fascinating to | understand. Thanks for explaining. | | If you don't mind going further, how does an ISP (or anti- | DDoS services) mitigate something like this without incurring | even more overhead to examine the traffic & not route it | onwards? Just a flat block against the sources, or something | more sophisticated? | Hikikomori wrote: | Haven't done much traditional networking for a few years so | my knowledge is not that current. | | They use bgp flowspec to send out filter lists using the | bgp protocol to routers, essentially a long list of ACL | rules with srcip dstip dstport. These routers can filter | any amount of traffic it gets in hardware, so as long as | your upstream (edge transit or peering links) are not full | you won't even notice the ddos. | | But you need something to analyse traffic that can | understand which traffic is ddos and what is normal. We | used devices from Arbor which are basically regular x86 | servers. These devices also received netflow so whenever a | ddos target received a lot of weird traffic (lots of DNS | for example) it was configured to redirect all traffic for | that IP to itself where it could gain a better | understanding than what you can from netflow and either | filtered it locally or sent out flowspec updates to filter | it on routers. You could also enable this manually for non- | volumetric ddos and do some filtering, but it's hard to do | if it's encrypted and if they are abusing some http call | that is just expensive to handle for the servers. | toast0 wrote: | ISP mitigation is usually simple. Figure out what the | destination of the attack traffic is and null route[1] | that, pushing those null routes upstream where possible. If | the destination IPs for the abuse are too diffuse and/or | the upstream won't do it on their routers where they | presumably have more capacity to drop than to forward to | you, you're still limited by your connection bandwidth on | the connections the abuse is flowing. Depending on the use | case, sometimes you can do pretty well by cycling your | legitimate traffic across a large block of IPs faster than | the abusers can cycle their abuse traffic, but it depends | on relationships and APIs to control null routing of your | upstreams. More sophisticated things like only block | traffic to/from port X, or throttling rather than blocking | aren't generally available (but would often be super | helpful if they were). | | Anti-DDoS services are quite a bit different, they | generally have obscene bandwidth, and BGP advertise your | attacked IPs, so they can get the traffic, then they | "scrub" the traffic and provide it back to you over a | tunnel (or something). Scrubbing can be relatively simple | stuff like rate limiting UDP and especially fragmented UDP | (which is usually stuff coming from reflection attacks) or | more complex stuff like looking at TCP options to determine | good vs bad traffic and only letting good traffic through, | or tracking TCP SYNs and only letting retries through, | plenty of other stuff with TCP state machines (or half | state machines, if they're only seeing the incoming traffic | and not the outgoing traffic). Oh, and they usually charge | you gobs of money. | | [1] if destination == IP, drop packet | bstpierre wrote: | I don't know anything about this particular isp or attack, but | if that 250 gbps is aimed at specific parts of their | infrastructure that are not designed for this much load, it's | likely to saturate individual links, servers, and/or network | gear. | | Their announcement said "all of our services", so this could be | anything from their authoritative/recursive dns to monitoring | or billing servers to specific routing nodes. | | So even if they had nice fat 100G pipes coming in, and those | pipes weren't saturated, a ddos of this size with a mix of | vectors can be debilitating on enough discrete pieces of | infrastructure that the isp is effectively "down" for many end | users. | | (Again I am only speaking generally with guesses about what | _might_ be going on at this isp. I'm also not a network | engineer but I do write code for network gear that does anti | ddos stuff.) | perlgeek wrote: | ISPs have a very decentralized network, that is, they have | routers in a lot of places. | | If you sum up the available traffic capacity, you get a number | that's far bigger than 250 Gbit/s. | | But that's not the right comparison to make. Much of that DDoS | traffic goes through multiple of the ISP's routers, and if | you're doing just a numbers comparison, you'd have to multiply | the attack rate by the (average) number of hops. | | But things are even more complicated in reality: many smaller | links will have only 10Gbit/s or 20Gbit/s or 40Gbit/s capacity | (or maybe even 1G), and saturating them not only causes | increased latency, but also routing the traffic around causes | more overall load on the network. | | Then there are management systems behind firewalls, and if you | DDoS them, it's typically the firewalls that give out first. | | Finally, customers don't get the full throughput internally | available to an ISP; if you DDoS a customer's IP, their link | will be saturated, even if the ISP itself has plenty to spare. | | (Not a network engineer myself, but working closely with some). | Groxx wrote: | Wouldn't that all be true of normal web traffic too, | presumably already capable of rates beyond 250Gbps? Though | they may not have that much free headroom on top of normal | traffic. | ericbarrett wrote: | > EDIT 16/09/2021 10:12*: During the night we had two more | attacks. We are working with the authorities, who have confirmed | they are looking into it and are doing everything in their power | to find the responsible individuals. We were contacted by an | individual who verified he was behind the attacks, asking for a | ransom. | | Roughly how many criminal groups are active in DDoS ransoms (as | opposed to data crimes, like cryptolockers and exfiltration)? How | common is this nowadays? Clearly it happens, but I've no idea the | general scale of the problem in 2021. | masklinn wrote: | > Roughly how many criminal groups are active in DDoS ransoms | | These days there are groups providing ddos as a service. They | control huge iot botnets and for a sum they'll point whatever | fraction you pay for to whatever target you want. | ericbarrett wrote: | Put more concretely, how many can front 100+ Gbps? E.g. 2-3, | or dozens, hundreds, any teenager, etc. | sva_ wrote: | Depends what kind of attack they use. A few years back, | reflection attacks were all the rage, and it was cheap as | fuck. But I can't say how many Gbps they delivered. | | They'd usually market this as "stresstest" service, and you | could pay by Paypal etc. Curiously they were all behind a | Cloudflare-anti-DDoS wall. I still wonder if Cloudflare | knew/knows about them, probably good for their business. | pstuart wrote: | I'm assuming that eBPF would be a potent tool for dealing with | this, and it looks like CloudFlare agrees: | https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig... | stingraycharles wrote: | Since this is HN, it's 2021 and DDoS'es are still a thing: why | are they still a thing? Is there some fundamental "anonymity" to | the Internet that makes it impossible to structurally prevent | DDoS attacks? Apart from CloudFlare-like approaches, are there | any R&D in the pipeline that may kill this type of attack once | and for all? | | To me it's incredibly infuriating to see the damage that still | happens with these extremely simple techniques. Will it ever end? | | Edit: to elaborate, I know that there are tons of insecure | Internet devices and whatnot. I'm more interested in standards, | and core protocol improvements that can fundamentally rid the | world of these types of attacks. | ev1 wrote: | It isn't anonymity. It's insecurity. Until you fix the human | factor, bad code, weak passwords, shitty programming practices | and minimum outsourcing to the worst vendors. | foobiekr wrote: | DDOS events typically use distributed botnets. Unless you | analyze the CNC you're probably asking for wide deployment of | acls (flow spec or otherwise); and such agreements happen | occasionally, as they operationally brutal but are slow | (otherwise they would be subject to abuse). | | You, right now, may be participating in a DDOS attack without | knowing it. Should your ISP cut you off based on information | from some Belgian ISP? | nine_k wrote: | A DDoS attack is indistinguishable from a huge legitimate | success. See Slashdot effect [1]. | | The very problem of a DDoS attack stems from that (1) each | request is a small, fully legitimate request and (2) there are | too many of them, coming from too many sources for filtering to | be efficient. It is also mixed with indistinguishable non- | attack requests by your normal customers. | | So no, even the best authentication would help little. | | [1]: https://en.wikipedia.org/wiki/Slashdot_effect | spenczar5 wrote: | DDoSes come from asymmetries in the difficulty of making a | request and providing a response. | | For example, a classic is the SYN flood. In this, the attacker | makes many requests to open TCP connections (with a SYN packet) | but does none of the rest of the handshaking. The server does | the initial bookkeeping for setting up TCP connections, before | eventually timing out and freeing the resources. The SYN | packets are very cheap and easy to form (and the attacker can | immediately free all the resources after firing them away), but | the connection state information is a little bigger. This | asymmetry builds up, making the server overloaded. | | I believe that asymmetries in resource consumption are, at some | point, unavoidable. | | You also asked: | | > are there any R&D in the pipeline that may kill this type of | attack once and for all? | | Note that DDoS attacks are a _class_ of attacks. They 're a | description for a broad genre. It's kind of like how | "bronchitis" actually just means "enflamed bronchial tubes" - | it's a symptom, not a cause. Even if we permanently defeat SYN | floods, there are a gazillion other DDoS-type attacks (UDP | flooding, messing with ARP tables - its an almost endless | list). | stingraycharles wrote: | > Note that DDoS attacks are a class of attacks. They're a | description for a broad genre. | | While they may be a class of attacks, I would describe all of | them as undesired requests. Sometimes, with some clever | protocol design, you can, in fact, mitigate a whole class of | problems. | nickthemagicman wrote: | Seems like that's another benefit of IPv6. | | Without needing NAT everyone can have their own IP address | and biz will be able to ban individual machines instead of | some giant NAT Network because one host using that IP address | is compromised. | laurensr wrote: | Doesn't it also mean addresses serving as the source of an | attack become more disposable? | bluedino wrote: | A trillion hosts.deny entries? | MrStonedOne wrote: | if you are sending packets but don't care about the reply, | you can spoof the from header. | | The other form of ddos is udp amplification. you send a | packet to a legit host and spoof the target ip as the from | ip, the reply goes to your target and bam, ddos. | kazen44 wrote: | this could be solved in ipv6 by implementing ipsec in AH | mode. | ma2rten wrote: | Yes, but these attacks are actually easy to detect if they | all come from the same IP. Sending packages from a lot of | different IPs actually is expensive. | alasdair_ wrote: | The attacks come from (up to) millions of different ip | addresses, each one a different machine and often on a | residential network. | | This is the problem with botnets - all the connections look | like real traffic at first glance. | menmob wrote: | That's.. not what a _Distributed_ Denial of Service does. | They come from thousands of IPs. | bravetraveler wrote: | That's the first 'D' in DDoS - _distributed_. | | Most robots participating in this are unwitting. It's | incredibly unlikely someone is _paying_... even with stolen | funds. | | Often a sort of malware that more or less sits dormant | until command/control sends a target to attack | | edit: To the individual machine, not a lot. The sum total | is where the denial of service comes in - everything has a | tipping point. | oneplane wrote: | It's not expensive if you have a botnet, reflective attacks | or L7 attacks. It's also not just one type of attack | (overloading the target), the only thing in common is that | it's a denial of service, and if you distribute it amongst | many sources it's a distributed denial of service. | | Distributed can mean different things, especially if you | can spoof addresses, in which case you can do fake | distribution (causing src based filtering to be useless). | If you know it's just a single bad AS that is doing it, | then you can block the AS, but if it's distributed amongst | multiple AS'es that's harder. Same with reflective attacks, | the source of the traffic might not be the source of the | attack and might be a legitimate source which you cannot | block because your internet wouldn't work anymore (i.e. | when it's DNS servers or NTP servers doing the reflection, | or a GCN address which would completely take an ISP's | section offline). | | It's not as simple as sending a large volumetric load from | A to B. It used to be that way a decade ago and it can be | effective today, but it's not the only way it's happening. | MrStonedOne wrote: | The src ip can be spoofed and randomized if you don't care | about getting the response (and to some degree, you care | more about _not_ getting it) | vladf wrote: | Why is this resource asymmetry unavoidable? It seems | reasonable that a new protocol could solve it. | | Consider augmenting TCP with a "pre-SYN" and "pre-SYNACK". | Suppose you have to send a pre-SYN which declares your | identity ip X. Then the server sends to X a "pre-SYNACK" | encrypted message containing (X, t = timestamp()) based on a | server-only key. | | Afterwards, X may send the server a SYN for real. But then | the server only allows a timeout of length, say, timestamp() | - t, as of receiving the SYN back to maintain bookkeeping. | | As benevolent cooperating clients, we may want to _ourselves_ | wait an exponentially-backed-off amount between receiving the | pre-SYNACK from the server and sending our SYN back (such | that in log-many rounds we establish our conn). | | Then any malevolent actor must then keep a resource on X | alive for at least half long as the server does (since the | server requires zero bookkeeping after firing off the pre- | SYNACK). | | Edit: this is simplified, you could just own a machine at X | which is far away from the ddos'd server and then have your | botnet spoof requests from X; by augmenting this procedure | with even more preamble rounds you could verify how long | communication takes between the server and client X and | subtract that out. | tw04 wrote: | You're too lazer focused on syn attacks, there are | countless others. What it boils down to is the sheer size | of the botnets. You stop syn? Great, I'll make it UDP with | a max payload and send it from 500k hosts. You will | eventually fall over if my botnet is big enough. | ajsnigrutin wrote: | > Why is this resource asymmetry unavoidable? It seems | reasonable that a new protocol could solve it. | | Sending a HTTP GET is easy, but the server has to process | the request and send the whole page... think a few tens of | bytes of a request, and a whole webpage in a reply. | brians wrote: | The server has to do an encryption and a decryption. A | legit client has to do the pre-SYN dance. In your protocol, | I don't spam pre-SYNs; I spam SYNs with fake tokens. Thus I | send only the same number of packets as before, but the | server has to do these decryptions! Or I can mix and match | spoofed SYNs with pre-SYNs and legitimate SYNs in whatever | way finds the best leverage. | | Your pre-SYN design is very close to the design of TLS 1.2, | by the way. That's how I knew how to attack it. TLS is | attackable in just this way: open a real connection, then | lots of fake handshake requests. No crypto required to | generate them, but the server has to do lots. | | Now that said, BCP 38 would go a LONG way towards | addressing this. There will come a day when the well- | behaved networks decide they're not accepting traffic from | non-compliant peers. | kazen44 wrote: | mind you BCP38 will do nothing against legitimate IP's | used in DDoS attacks. (which are more and more common | because of large botnets thanks to IoT devices and their | abysmal security). | catmanjan wrote: | So a bit like a proof of work concept for packets? Doesnt | sound like such a crazy idea to me | Matthias247 wrote: | > Why is this resource asymmetry unavoidable? It seems | reasonable that a new protocol could solve it. | | Protocols are trying to avoid this problem. For TCP there | exist SYN cookies to reduce the amount of work a host has | to do. For QUIC retry packets exist, as well as anti- | amplification rules. However applying those mechanisms also | has a cost. One is you still get computational cost not to | zero, since you still have to receive and reject packets. | And the other cost is now your legitimate users are | experiencing a latency penality due to potentially | requiring another round trip. | | The latter means you don't want to make those mechanisms | the default, but rather use them selectively in certain | situations. | | The former means even if you apply countermeasures you are | not fully protected. If you get flooded purely by the bare | amount of packets and the system stalls just from handling | them, all the higher level mitigations won't work very well | anymore. That's the difference between a DDoS and a DoS - | the DDoS might not even need an asymetry in cost, it just | brute-forces the system down. | jcrawfordor wrote: | This method is basically already in use for DOS mitigation | using a technique called a SYN cookie. But as the parent | said, there is a huge spectrum of different ways to produce | a DOS and so there is no single method of addressing them. | The problem tends to be much harder as you get to higher | levels, for example, requesting things from an HTTP server | - there are mitigations available there as well such as | Cloudflare's approach of capturing DoS like requests and | subjecting them to CAPTCHA, but once again no one size fits | all. | vladf wrote: | I guess the problem is "simply" the use of these | resource-asymmetric protocols. As a server-owner, if | there was a robust alternative protocol which ensured | resource symmetry, wouldn't you be naturally incentivized | to adopt it and reject all asymmetric ones? | | It still _feels_ like there 's a way of specifying a | protocol (which of course would need to be adopted) where | (1) authentication is a first-class entity (2) all non- | authenticated requests enforce resource symmetry at the | protocol level, while still allowing asymmetric work to | be done (after a resource-symmetric auth). | | Edit: augmenting the rough outline of what I originally | proposed you can essentially require an arbitrary amount | of proof-of-work, however useless, on the requester side | (e.g., literally request a nonce such that the hash of | the packet is below some value). Since only auth needs to | be symmetric the overhead doesn't seem too bad. | Bootvis wrote: | If we do some form of authentication it seems likely that | the receiving side needs to do some kind of computation. | What if I just sent garbage? Garbage is easy to create | but the receiver needs to do some work to figure this | out. | vladf wrote: | Hrm, I don't think you're engaging with the spirit of my | proposal, which _does_ require resource-symmetry to | establish auth in the protocol. | | I admit I haven't specified a full counterfactual | protocol here, but see my edit for a rough outline of how | you wouldn't just be able to generate garbage. | psd1 wrote: | You said "I feel that..." so I suppose you're an mbti 'F' | - excellent choice! But I feel you'll work on this idea | for a bit and then throw your hands up. | | The symmetry is irrelevant when the attacker has stolen | someone else's resources. Granny may notice her computer | is even slower than usual, what then? | | If we stipulate a future where attackers cannot steal | other people's resources, then work backwards, I can't | see any internet with any degree of freedom. | | It would be necessary for every system to be 100% | watertight. Mathematically-provably so. Spectre/Meltdown | demonstrates that this isn't just formal proof about | software. Eradicating the pathways to ddos is an | intractable problem. You'd need to seriously consider | preventing all network access except through certified | and regulated kiosks to which users do not have direct | physical access. Like, hand a librarian a piece of paper | with a URL on it and get a print out. | | I'm not expert, you shouldn't have confidence that I'm | correct. | spenczar5 wrote: | Yes, I think there is a lot of room for better protocols | that reduce the asymmetry. A lot of our internet is built | on protocols that assumed good behavior, since back in | the 70s and 80s and even 90s that was plausible. | | But I don't think we will ever get to a point where we | bring that asymmetry to zero. We often really do want | complex tasks to be done by a server in response to a | simple client request. | | Software people rarely put this asymmetry in the front of | their minds during design, so I think its something we | will continue to see in new systems, even if we | (magically) universally adopted better versions of the | old, too-trusting protocols. | imranhou wrote: | Not to mention, implementation of new protocols is one | thing, adoption of such protocols is another - I would | dare say the harder part of the two. If you are required | to serve a population that won't keep up, its an uphill | battle. | hansvm wrote: | > Why is this resource asymmetry unavoidable? It seems | reasonable that a new protocol could solve it. | | There's some sense in which a little asymmetry is | unavoidable -- a malicious attacker can always just send | whatever bytes they want over the wire, and a server on the | other end with any practical purpose must at a minimum | receive each of those bytes AND perform some operation | equivalent to determining if any more work needs to be | done. | | To the extent that any real work or state management needs | to be done by the server, the only way to avoid much | asymmetry is to use a protocol forcing the client calls to | be artificially more expensive before the server will agree | to respond. | psd1 wrote: | Yes and... a botnet controller simply doesn't care | anyway. She or he isn't spending their own resources. | robocat wrote: | The server would still need to track some state. | | Worse, the cost is another round trip delay. That's a | problem for anyone with a high latency connection. | vladf wrote: | Right, under the previously unspecified constraint that | you need a response in one round trip I agree there's a | natural resource asymmetry. | toast0 wrote: | The fundamental asymmetry is bandwidth. If you can | overwhelm my incoming connection, I can't do much useful | work. | | I've got to pay for capacity, but a bot net controller | doesn't need a whole lot of bandwidth to trigger a lot. If | you control 10,000 hosts with 10Mbps upload, that's | 100Gbps; and botnets are growing as are typical upload | bandwidths. | | And that's without spoofing and amplified reflection | attacks. Some of the reflection attacks have high | amplification, so if you've got access to 100Gbps of | spoofable traffic, you can direct Tbps of traffic. | | If my server has a 10G connection and you're sending me | 1Tbps, there's just no way I can work. | | Syncookies work pretty well for TCP, I had run into some | effective SYN floods against my hosts I managed in 2017, | but upgrading to FreeBSD 11 got me to handling line rate | SYNs on a 2x 1Gbps box and I didn't take the time to test | 2x10G as I wasn't really seeing that many SYN floods. I | don't recall any more effective SYN floods after that | point. We didn't tend to have dedicated attackers though; | people seemed to be testing DDoS as a service against us, | and we'd tend to see exactly 90 or 300 seconds of abuse at | random intervals. | | Our hosting provider would null route IPs that were | attacked if the volume was high enough for long enough. | Null routing is kind of the best you can hope for your | upstreams to do, but it's also a denial of service, so | there you go. | lend000 wrote: | With DDoS the problem is that botnets come from a bunch of | random devices that were compromised (for example, your un- | updated smart TV or old router could be contributing to the | DDoS without your knowledge.) | | With spam calls, ransomware, and most other forms of | cybercrime, the problem is that Russia/China and many 3rd world | countries don't extradite/cooperate or have the resources to | work with international authorities. | | I sometimes wonder how much safer the internet would be if | China and Russia were cut off. Wouldn't be so great for the | already poor state of liberty in either country, but the rest | of the world would certainly benefit. | ChuckNorris89 wrote: | _> Wouldn't be so great for the already poor state of liberty | in either country, but the rest of the world would certainly | benefit._ | | That's a pretty closed minded way of looking at things. | [deleted] | Andrew_nenakhov wrote: | Cutting off countries will do very little. Compromised | devices are spread evenly across the globe, and even if their | puppeteers sit in China or Russia, you wouldn't be able to | block them from issuing orders to botnets. | spenczar5 wrote: | It's true that they often use botnets, but amplification | attacks are more and more common, and an important component | here. | | In these, the attacker sends crafted requests to an unwitting | third party, and that third party then floods the victim with | traffic. DNS amplification is probably the best known, here, | since DNS is (usually) over UDP which has no persistent | connection for replies: | | - Send a DNS query to a DNS server, using your victim's IP | address as the "source IP" for the query. Make the query | something with a huge response, like using "ANY". | | - DNS server responds to the victim's IP address, sending | many big UDP packets with the response. After all, it can't | know any better - it must* just trust the claimed source IP! | | - Victim gets a huge flood of UDP traffic, overwhelming IP- | level infrastructure like routers and switches. | | This lets a relatively small attacker multiply ("amplify") | their impact - and they do it while traversing a third party, | making them more hidden. | | --- | | * If course, there actually are mitigations for this; it's | possible to detect this spoofing. But this is how DNS | operators have worked in the past. | NicoJuicy wrote: | That's the reason why my nextdns blocks all _.ru and_.ch | domains. | | I don't trust them + i don't speak the language. So entirely | blocking it is an easy decision. | | Nextdns on my router blocks generated domains, which could be | used against c&c servers. | forty wrote: | .ch is Swiss TLD ;) | edflsafoiewq wrote: | > the rest of the world would certainly benefit | | OTOH most pirate library sites are hosted in those countries | for the same reason. | jansimonek wrote: | "without your knowledge"... is there a way to check if my IP | was participating in the DDoS? If they are filtering the | traffic, they might be able to create a list of IPs that I | can check. Or perhaps, an IP can be associated with a contact | info so the if any of my devices would be infected and | participate in DDoS, I would get notified by the victim and | could take action. | jl6 wrote: | Imagine the real-world equivalent: someone spreads a rumor that | the Fifth Avenue Apple store is giving away free iPhones at | 6pm. Ten thousand people turn up and block the store, the | sidewalk, the roads... | | How should a business defend against that without losing | legitimate customers? | | The answer is probably along the lines of "educate people not | to believe rumors" (:: "educate people not to run insecure | software"). | smoldesu wrote: | How does secure software protect you from acknowledging | perfectly innocuous requests? | jl6 wrote: | It's the zombie-botnet-machine owners who need to be | running more secure software, not the botnet victims. | jjeaff wrote: | Most/all ddos attacks come from unwitting people whose | computers or iot devices have been taken over on the order of | tens or hundreds of thousands of devices. So there isn't much | that can be done beyond filtering or having a large enough | pipeline to absorb the traffic. | | Even if you had a central authority cutting off access from ip | addresses, do you cut off whole university campuses because | someone's "smart" coffee machine is participating in a ddos or | entire businesses because there is a computer in a closet | somewhere that is infected? | mrweasel wrote: | We talked about this problem when I worked at a telco. The | problem isn't necessarily cutting of the devices that are | part of the attack, it's dealing with the aftermatch. | | As sad as it is, most people won't understand that their | device have been effected, and the telcos won't be | responsible, financially, for paying to do the cleanup or | verification. | toxik wrote: | I feel like home router makers could intervene here and let | their owners know when they've likely been hacked. Maybe even | a regulation type deal? | logifail wrote: | > home router makers could intervene here and let their | owners know when they've likely been hacked | | How? | | A couple of years ago I had an email from our ISP telling | me that "common port X is open on your router" (forwarded | to a box exactly as I set it up) and asking if I could "fix | the problem". | | Except it wasn't a mistake or a configuration error, I | deliberately set it up that way. | foxfluff wrote: | I don't see how this would be feasible without giving them | a backdoor to snoop around in my home network. No thanks. | MertsA wrote: | That's never going to work but it very well could lead to | ISPs very strongly herding customers to use their own | managed routers. They already push for that for customer | support purposes, using this to start automatically | snooping on "malicious" traffic within a customer's network | would be a big step in the wrong direction. | | You could make a compromise here and require ISPs and | network vendors to support a common notification protocol | to identify devices sourcing malicious traffic. Most ISPs | already have systems in place to send notification messages | to customers that are sending botnet traffic. You could | mandate it for all of them and let the customer decide if | they want their router configured to automatically block a | device they were notified about or just record the MAC | address and send the ISP a URL the user can visit to view | the device sending the traffic. You could make it friendly | to the unwashed masses and still put detection on the ISPs | and not give them privileged access inside every customer's | network. | Arnt wrote: | It's not that simple. Collecting 250Gbps of uplink is not that | simple. The attack vectors are simple in principle, but | carrying it out... | | Here's next year's fine vector: You write a nice iphone or | android app and persuade fifty million people to download it. | But unknown to the owners of those mobile phones, it also has a | hidden evil feature: If the phone has the screen off, is | charging and is on a WLAN, then it can participate in a DDoS if | you direct them to. The phone's owner won't notice, but if | enough phones are charging and each can contribute 2Mbps of | upload, you could collect 250Gbps at the target. | | And tracing it back to that app will be very difficult. | jonplackett wrote: | Why would it be so hard to trace back? | [deleted] | johnklos wrote: | It wouldn't be hard. | | If you can examine traffic on a network from which | attacking traffic originates, you can see that it's coming | from a phone. | | Then you could see which app by either limiting or deleting | apps one at a time, or you'd need two sources, then just | see which apps the two sources have in common. | | When you have 250 Gbps, it'd be simple to capture a | fraction of a second's worth of traffic, then write a | script to fire off emails to network admins of every IP | involved and ask them to look in to it. Out of hundreds or | thousands of messages, you'll get a few humans who'd look | in to it and would be helpful. | | I should know, because I've done this very thing. | Arnt wrote: | If your nontechnical neighbour's ISP calls and says "you're | participating in a DDoS, will you please find out which | device behind your NAT is sending the traffic and fix | whatever the problem is", do you think your neighbour can | fix it in five minutes? | edoceo wrote: | They could remote stop the router? | tinus_hn wrote: | A responsible ISP will block the user but this means the | user is going to call the helpdesk which costs money. | | An irresponsible ISP will stall the process to see if you | give up, which is free. | MrStonedOne wrote: | because nothing says you have to include your real ip in | the src header of the ip packet if you don't care about the | reply. | lordnacho wrote: | What's the solution to this? If millions of devices are each | sending 2Mbps to some target, what can the target do? It | doesn't sound like there's a program that would be able to | identify something so diffuse. | kazen44 wrote: | remote triggered blackholing the networks from which the | ddos comes is very effective for small DDoS's. but its a | very crude methods that drops entire /24's from being able | to reach you. | | this doesn't work for massive ddos's. | Arnt wrote: | I think you understand why edpnet.be has problems now. | | What they usually do is try to write packet filter rules | and drop packets as close to the millions of origins as | possible, plus call the networks where the sending devices | are and say things like "could you please call your | customers and tell them to unfuck their {phones,dsl | routers,...}" | speakersPushAir wrote: | It is intrinsic to the design. | | The open nature of the protocol carries with it an implicit | trust. Any client may join and speak to any other client | and be heard. It is a social contract, just as we have "in | real life." | | Imagine someone knocks on your door. You can choose to not | answer, but even considering the choice costs you cognitive | bandwidth. As long as you're within earshot of your door, a | knock will cost you attention. | | We have solutions all across the spectrum, from putting up | a "no soliciting" sign, to arresting bad actors. But what | if you have a riot outside your front door (a DDoS)? | tinus_hn wrote: | On iOS the OS does not allow apps to do this, you basically | can only use the OS APIs to do networking in the background | so clever tricks are right out, and it is throttled. | sydd wrote: | Google/Apple will catch you in hours/days and all your work | goes down the drain. Also its hard to get 50M downloads, not | worth the time. | | There is already an established economy of botnets. They work | like this: | | 1. There is a hacker group (usually ex-USSR or China) that is | actively monitoring 0 day vulnerabilities. They usually | target internet connected devices which are from companies | that dont give a damn about updating them after they are | released. This is the case for most budget routers, "smart" | things (lamp, frigde, vacuum, doorbell,..). | | 2. If they find a nice one (e.g. one that affects millions of | shit-tier $40 routers), they set up port scanners and infect | as many as they can. | | 3. Rent the botnet on darknet sites, there is already an | established pricing for them, for example $1000 for 1 hour of | 1000Gbps DDOS. You just pay them in crypto and tell what to | attack. | cookiengineer wrote: | > Google/Apple will catch you in hours/days | | Tell that to the Hola and Honey extensions. They're known | to be a botnet and for this kind of abuse for years yet you | can still find them everywhere in the App/Play Store...and | _so_ many tech related youtube channels keep advertising | for them blindly, it's ridiculous. | c0nducktr wrote: | I had heard that Hola was possibly some kind of malware, | but this is the first I'm hearing that Honey (I'm | assuming we're both talking about the coupon-code | extension) is like that too. | | I just figured Honey was kind of like spyware, collecting | information about what people bought online. | deadalus wrote: | (2015) Imgur was being used to create a botnet and DDOS 8Chan | : https://news.ycombinator.com/item?id=10256942 | ma2rten wrote: | It's not that easy to get 50 million people to download your | app and if you pulled that off there are more profitable | things you could do. | nine_k wrote: | Why limit it to DDoSing? It could be just one feature, but | an important one. | leroman wrote: | Toolbars did just that and been used for many "evil" | things.. | Arnt wrote: | My point exactly. | | These things are easy to do in principle (most readers here | can write that evil payload, or learn how to do it quickly) | but there are real practical challenges to getting the code | to run on millions of strangers' computers/phones/dsl | routers, so most of the code that's actually deployed that | widely is beneficial. Happily. | mvanbaak wrote: | Unpopular opinion and will get downvoted: | | This is why the appstores do reviews etc. these kind of | things dont happen because of that | Groxx wrote: | ... except that they most definitely do still happen in | spite of that. | | But yes, they do tend to mean less obviously-just-malware | apps. | ancarda wrote: | As far as I know, a lot of DDoS attacks use UDP amplification, | which can be prevented if every ISP implements BCP 38; i.e. | drop UDP traffic at the edge of their network that has a source | that cannot have come from within their network. | | EDIT: To clarify, this won't stop layer-7 based DDoS attacks, | or anything that uses TCP (like SYN flooding). Just UDP | amplification. | woutifier wrote: | BCP 38 would stop SYN flooding if it is using source address | spoofing. It won't stop any attack not based on IP spoofing | though. | MertsA wrote: | >or anything that uses TCP (like SYN flooding) | | Plenty of SYN floods spoof IP as well. If you don't need to | get the response, and you're behind an ISP that doesn't | bother blocking IP spoofing, why would you use your actual | IP? It'll make it much harder to actually trace an attack to | the actual device doing it. It won't work on devices behind | NAT but neither will reflected UDP attacks. | hirako2000 wrote: | The issue is: bot nets. The attacks come from multiple sources | worldwide. No anonymity, rather a swarm of legitimate hosts | sending mass;requests. Research and development? Network | connections come in, one way or another some devices have to | filter them in or out. AFAIK it's an arm race. Large bot nets | on one side, and hardware optimised to filter out certain | requests on the other. | | It's a simple technique but requires compromising enough | networked resources to make the attack effective. | | Cloudflare and other providers having large capacities are well | armed to deflect DDoS that can't overwhelmed them, but even | then, they aren't safe from a larger bot net than what their | network resource can handle. | | If someone knows more than I do, maybe they could elaborate on | technologies I'm not aware of. | johnklos wrote: | Simple - egress filtering: | | https://en.wikipedia.org/wiki/Egress_filtering | | If every large network operator did this, then shutting down | abusive hosts used to generate the kind of traffic needed for | attacks like these would be possible. | | You'd make a list of all the IPs for each large provider from | which excess traffic is coming, and you'd send them to each | large provider, and they'd either block the IPs completely | (requiring customers to fix their compromised machines before | they could get back on the Internet), or at very least they | would block all traffic to the destination of the attack. | | Some silly people think this isn't possible, but if large | providers are forced to do it, then they can easily tell | smaller ones they'll be blocked if they don't also do this, and | so on, and we'd have a slightly more responsible Internet | again. | | Too bad we have shitty, huge companies that don't care about | doing the right thing :( | toast0 wrote: | > Some silly people think this isn't possible, but if large | providers are forced to do it, then they can easily tell | smaller ones they'll be blocked if they don't also do this, | and so on, and we'd have a slightly more responsible Internet | again. | | You go ahead and force a large provider to do it, and let us | know how. From what I can tell from reading NANOG mailing | list posts off and on for decades is that most of the larger | providers really won't be bothered to do this, and there's | not any effective leverage to force them. | Hikikomori wrote: | No, the solution is ingress filtering and a lot of ISPs do it | these days, see bcp38. | [deleted] | yjftsjthsd-h wrote: | > Some silly people think this isn't possible, but if large | providers are forced to do it, then they can easily tell | smaller ones they'll be blocked if they don't also do this, | and so on, and we'd have a slightly more responsible Internet | again. | | > Too bad we have shitty, huge companies that don't care | about doing the right thing :( | | There's no call to be dismissive of anyone who disagrees with | you; maybe you're "silly" for believing that companies will | threaten each other for no reason (that they care about). | Jweb_Guru wrote: | Nope, they are basically right and it's a source of endless | frustration to security researchers that these very simple | problems are still thinks that have to be dealt with in | 2021. | phkahler wrote: | >> Since this is HN, it's 2021 and DDoS'es are still a thing: | why are they still a thing? Is there some fundamental | "anonymity" to the Internet that makes it impossible to | structurally prevent DDoS attacks? | | IMHO that is exactly what the problem is. Weather its malware, | phishing scams, extortion schemes, the fundamental problem is | _knowing_ where stuff comes from. | | I propose an IPv6 subnet that routes by using a portion of the | IP address as lat/long coordinates _as a starting point_ | because routing tables are litterally a form of indirection. | yjftsjthsd-h wrote: | > an IPv6 subnet that routes by using a portion of the IP | address as lat/long coordinates | | That seems like a rather large privacy problem. | matheusmoreira wrote: | You could for example protect all networked computers with | single packet authorization. They will ignore all network | traffic unless a signed packet is sent first. To the | unauthorized it's like the computer is not even there. | kazen44 wrote: | this is basically how a vpn tunnel works in some way. | (wireguard, ipsec if properly configured etc). | | it has one, enourmous downside. it destroys reachability to | your systems from the population at large because they need | to trust some key, which is not universally distributed. | LinuxBender wrote: | There are many parts to the answer of your question, but one | part yet to be implemented in a meaningful manor is BCP38 [1]. | The RFC's around BCP38 keep changing and that may be part of | the issue. It is still too easy to spoof the source across ISP | boundaries. Then of course there is also the issue of getting | all the ISP's to work together to detect and mitigate these | attacks and that involves cost that the tier 1 providers barely | spend towards. There are so many more pieces to this problem | and I am barely touching on it. | | [1] - https://www.rfc-editor.org/info/rfc8704 | gxt wrote: | There are very little consequences. | | Until states start doing s/sea/internet/g on their old-school | maritime piracy laws[0] I don't think anything is going to | change, except attacks are gonna get bigger and bigger. | | Under current law, that's a lifetime sentence in many*(US[1] & | Canada[2] at least) countries. It should do the trick if most | states get onboard. | | > Acts of piracy threaten internet security by endangering, in | particular, the welfare of internetfarers and the security of | navigation and commerce. These criminal acts may result in the | loss of life, physical harm or hostage-taking of | internetfarers, significant disruptions to commerce and | navigation, financial losses to server owners, increased | insurance premiums and security costs, increased costs to | consumers and producers, and damage to the internet | environment. Pirate attacks can have widespread ramifications, | including preventing humanitarian assistance and increasing the | costs of future connectivity to the affected areas. | | [0] https://www.un.org/depts/los/piracy/piracy.htm [1] | https://www.law.cornell.edu/uscode/text/18/1651 [2] | https://laws-lois.justice.gc.ca/eng/acts/c-46/page-11.html#s... | mfkp wrote: | I wonder if this is related to the attack on voip.ms (ongoing for | multiple days) | | https://twitter.com/voipms ___________________________________________________________________ (page generated 2021-09-18 23:00 UTC)