[HN Gopher] DoS Attacks against my Online Game
___________________________________________________________________
DoS Attacks against my Online Game
Author : def-
Score : 80 points
Date : 2021-09-27 19:39 UTC (3 hours ago)
(HTM) web link (hookrace.net)
(TXT) w3m dump (hookrace.net)
| api wrote:
| Why would someone DoS a small free game?
| bsder wrote:
| Testing. Proof that they can DDoS something bigger. For the
| Lulz. etc.
| nkrisc wrote:
| Some people just have anti-social mental illnesses and this is
| how they manifest.
| [deleted]
| tetromino_ wrote:
| Because they are a teenage sociopath. (Quoting from the
| article: "... since the attacker was a minor ...")
| willvarfar wrote:
| From the hint about knowing the likely identity of the
| attacker, its probably an old and very specific vendetta.
| Perhaps an early player who was spurned by the community or
| developer who fell out? As the attacker has run tweaked gaming
| servers and things, it's puts it at the upper end of
| scriptkiddying.
| reggieband wrote:
| I worked on a mid-sized online game a few years ago and we
| experienced several DDoS attacks. I recall one employee tracking
| down the specific botnet that was rented for the attack against
| us and we calculated the attacker probably spent a couple of
| hundred dollars based on the rates.
|
| IIRC, we eventually used AWS Elastic Load Balancer to just soak
| up the attack, which was a pretty basic SYN flood. Then we waited
| the attacker out until he got sick of spending money. That
| temporary redirect definitely impacted performance and cost us
| some money but it pretty well mitigated the issue. We also spent
| a bit of time optimizing our servers to drop obvious nonsense
| requests as quickly as possible but in the end the ELB handled
| most of the issue for us.
| ChrisMarshallNY wrote:
| Sounds like modern-day "protection rackets."
|
| I know that, if you own a gambling site, you can look forward to
| meeting exciting slavs. I didn't realize they were taking it to
| other types of games, but I guess that makes sense. Wiseguys
| coerce Grandma's Bake Shop, just as they do Moneybags National
| Bank.
| dividuum wrote:
| Not sure if this is helpful, but I remember reading about the
| tribes network protocol years ago. IIRC it also UDP based and
| essentially had a mechanism to request a proof of work depending
| on server load before accepting any complex packets from a
| client. You could probably require some proof for their source IP
| utilizing the HTTPS request already used during server browsing.
|
| Edit: Found it.
| http://opentnl.sourceforge.net/doxydocs/history.html (the
| ,,puzzles")
| Scaevolus wrote:
| DoSing small games like this sucks.
|
| Have you looked at ddos-guard's pricing? They seem to be a common
| budget option.
| def- wrote:
| Thanks for the tip, I didn't know about them. From an initial
| look $240 for a 1 core vps is a bit too expensive for us.
| gibs0ns wrote:
| Also checkout secured.gg [0], I recently discovered them
| while researching DDoS solutions used by other popular game
| servers (for Minecraft, and GTA5). I haven't used them but
| their pricing seemed reasonable to me.
|
| [0] https://secured.gg/dedicated
| heinrich5991 wrote:
| The cheapest server seems to be about as much as we pay for
| 24 locations today.
| noobgrammer wrote:
| EDIT: Cloudflare already made a game demo on Workers...
| multiplayer Doom: https://blog.cloudflare.com/doom-multiplayer-
| workers/ that was written using WASM + WebSockets, porting over
| an open-source Doom and shimming in a UDP-over-Websockets
| networking layer. Despite all that, it's still fast enough for a
| first person shooter.
|
| Original post: Have you looked into using a serverless pub/sub
| model, like Cloudflare's Workers KV? The example they give is a
| simple IRC-like distributed chatroom
| (https://github.com/cloudflare/workers-chat-demo), but
| theoretically it may work for games too.
|
| Player state can be stored in a decentralized key-value store
| that Cloudflare manages (Cloudflare Durable Objects). They absorb
| all the DDoS and handle replication between edge nodes. You don't
| see any of that.
| https://developers.cloudflare.com/workers/learning/using-dur...
|
| Then each game client uses a worker to access that KV on a
| subscription basis, and Cloudflare will route that worker to its
| nearest edge node and retrieve the state from there (which was
| previously replicated a moment ago, internal to Cloudflare's
| infrastructure). Changes to state are replicated across the edge
| network and pushed to client workers.
|
| https://workers.cloudflare.com/
|
| I don't know if this would result in acceptable latency, but it
| could help with DDOS at least. The main benefit is that it's
| incredibly affordable, especially when you're only talking about
| thousands of players.
| cft wrote:
| In the US, Cogent offered $900 per month promotion for 10G commit
| on a 100G burstable circuit. If you rent several rack units and
| put a used Juniper hardware firewall from eBay in front, that may
| be one of the cheapest solutions to absorb 100Gbps attacks
| john37386 wrote:
| Make sure you won't use the 100Gbps in the full month because
| it might DDoS your wallet ;)
| markus_zhang wrote:
| If you received anyone asking whether you need services that
| defend against DDOS, and if you refused, those are probably the
| guys who initiated the DDOS attack.
| schemescape wrote:
| From the article, it sounds like they know at least some of the
| attackers already.
| bashy wrote:
| DoS attacks are something I've had to put up with too while
| hosting game servers since 2008. I run fshost[1] and we see
| attacks almost weekly. Even though we host mainstream games, we
| still see legitimate traffic being filtered.
|
| Do you modify any kernel options? net.ipv4.conf.all.rp_filter=1
|
| [1] https://fshost.me
| seiferteric wrote:
| Why is ddos still possible? It is possible for isp's to stop
| this. There is a proposal for isp level blocking if spoofed
| source addresses. Also there should be something like an api
| where I can tell my isp that I don't want to receive anymore
| packets from a given source and it should be propagated up the
| chain.
| remram wrote:
| That exists: DOTS: https://www.rfc-editor.org/rfc/rfc8783.html
| leath wrote:
| This is a nice read https://blog.cloudflare.com/the-root-cause-
| of-large-ddos-ip-...
|
| It is indeed possible for ISPs to stop this, but my guess is
| that it's cheaper not to :) Large ISPs could require egress
| filtering for peering with them.
| zeta0134 wrote:
| The key is in the first D: "distributed." A DDoS is designed to
| look just like legitimate traffic, but coming from many sources
| all at once. The goal of a successful attack is to both
| overwhelm the target network by sheer volume, and to make it
| difficult to stop the attacker without also blocking legitimate
| traffic. They persist in large part because they exploit the
| interconnectivity that makes the internet useful in the first
| place, without which it would cease to be.
| debian3 wrote:
| You should look into OVH. They have those gaming server and they
| handle the DDOS protection. I was getting hit by DDOS before, but
| since I moved there, nothing (except an email from OVH to let me
| know that my server is being attacked and that they are filtering
| my trafic). On the server itself you just don't feel anything.
|
| Edit: I should add that the DDOS protection is included with the
| server rental and there is no limit on the size or duration of
| the attack.
| pronoiac wrote:
| > Instead of cheap VPS servers we have tried getting dedicated
| servers at larger European hosters like OVH, Hetzner, ihor and
| NFOrce. The idea is that we have exclusive resources, so the
| chances of us impacting other customers is lower, and thus we
| won't get nullrouted so easily. Largely this works, but the
| available network bandwidth (usually 1-10 Gbit/s) as well as
| CPU usage become the limit.
| dan_wood wrote:
| I don't think OVH is viable in this case, they do mitigate the
| attack but in my personal experience they also mitigate legit
| traffic during the attack.
|
| Mind you, this is a process using a single port, with only
| around 100 active connections. You'll easily see half if not
| more lose connection during a DDoS attack.
| debian3 wrote:
| I did notice that a few years back, but now when the filter
| activate I no longer see any drop in bandwidth usage or any
| customer complaints. How long ago did you experimented with
| their filter?
| leath wrote:
| We've had several servers with OVH, including their kimisufi
| line, So You Start GAME line, their standard GAME line and
| their standard servers. While I'm sure these are great for
| common games their DDoS protection seems to get confused by our
| very non-standard protocol, ending up blocking most if not all
| traffic from non-connected players.
| willcipriano wrote:
| A shot in the dark but maybe implement a wrapper for the
| protocol in something that looks more like http? Websockets
| perhaps? Otherwise I think you will have to build your own
| countermeasures specific to your protocol.
| debian3 wrote:
| Might sound strange, but you could always contact @olesovhcom
| on Twitter. He is the CEO of OVH and he made change to their
| DDOS filter base on what we reported to him. He is always
| interested in improving is offering. But that was maybe 5
| years ago, now maybe he will put you in contact with someone
| else, but back then they were actively looking for feedback
| to improve their filter.
|
| Edit: you could always contact their support as well.
| Fighting DDOS on your own it's an expensive/difficult battle.
| But their DDOS filter is fully custom (mostly Asic and some
| Arbor as well).
| Shadonototra wrote:
| i thought valve offered protection against DDoS attacks, or it's
| not available for free titles?
| def- wrote:
| I actually received a nice email from someone at Valve about
| this following this post and we are currently evaluating if we
| can use it:
| https://partner.steamgames.com/doc/features/multiplayer/stea...
| sapphyrus wrote:
| SDR works well and the latency is decent, but this would
| probably force you to drop the non-steam release or make some
| of the servers steam-only?
| znxster wrote:
| Ah! I was going to this actually. I discovered it when Bungie
| switched Destiny2 over to use it.
| plasma wrote:
| A few suggestions:
|
| 1. Cloudflare offer TCP based DDoS protection too, see their
| Magic Transit or Spectrum product
|
| 2. This sucks, but put your servers behind WireGuard or Tailscale
| VPN so that in order to connect you need to have authenticated
| leath wrote:
| As the blogpost mentions TCP is not exactly desirable for our
| project. Moreover, if you contact Cloudflare about those
| products you'll get a monthly quote that is far beyond what an
| open source project run by donations can sustain :)
| tyingq wrote:
| _" For the individual server infos the client currently has to
| communicate with each game server by UDP, thus revealing its own
| IP address without having connected to a server. Since one of the
| known attackers is running their own DDNet server, they can use
| this method to collect legitimate player IP addresses and spoof
| them in their attacks."_
|
| Interesting. I wonder if running an overlay network would help
| there. More choices today for userspace overlay networks. Rogue
| server owners would still see an IP, but they could only attack
| it from their connected server, not the internet at large. And
| you could do some kind of ingress/egress filtering.
|
| Some sort of periodic coordinated switching from UDP port A to
| port B might help too, like a control message that tells the game
| client to switch ports. Or randomized initial port assignments
| combined with filters/firewalling or just in-band 'you're not
| supposed to send here, bye'.
| 123pie123 wrote:
| I've no idea how complex this is to code or if it will work for
| you
|
| but I'll throw the idea out to see if anyone else could improve
| on it etc..
|
| initial strawman draft idea: have a front door service that just
| verifies your gamers (eg log on server) This will need to be
| protected by a Ddos but the throughput shouldn't be large. once
| authenticated your clients IP address is then passed to some sort
| of software based firewall protecting each of the main game
| servers
| Deathmax wrote:
| The firewall would need to be able to handle all the DDoS
| traffic as well, since your current idea would still pass the
| game server's IP back to a client. This is doable if you're
| hosting on a cloud provider and let their firewalls filter the
| traffic before hitting the game server.
|
| Embark Studios recently open sourced (in alpha) a UDP proxy[1]
| designed for games that lets you implement a load balancing
| layer. This allows you to remove servers in the load balancing
| layer in the event that it comes under attack, allowing the
| game server to stay up and only having to disconnect a portion
| of players connected to the attacked loadbalancer. Having a
| proxy layer is also how Steam protects game servers using the
| Steam Datagram Relay[2].
|
| [1]: https://github.com/googleforgames/quilkin [2]:
| https://partner.steamgames.com/doc/features/multiplayer/stea...
___________________________________________________________________
(page generated 2021-09-27 23:00 UTC)