[HN Gopher] Fast-Paced Multiplayer (Part I): Client-Server Game ...
___________________________________________________________________
Fast-Paced Multiplayer (Part I): Client-Server Game Architecture
Author : mkl95
Score : 107 points
Date : 2021-10-26 17:54 UTC (5 hours ago)
(HTM) web link (www.gabrielgambetta.com)
(TXT) w3m dump (www.gabrielgambetta.com)
| Sohcahtoa82 wrote:
| I used this guide when implementing my own real-time multiplayer
| platforming game.
|
| It really understates the importance of _authority_. It blows my
| mind how many games, especially FPSes, give a lot of authority to
| the client. Lag compensation for shooting is one thing, but
| clients should not be able to shoot through walls or speed hack.
| These types of cheats are _easy_ to prevent server-side, but
| developers don 't because .... ? No idea. I've always guessed
| that it's to reduce server-side CPU load, but that's all I can
| guess.
| mathgladiator wrote:
| I believe part of the problem is that it is hard enough to
| build a game in the first place, and networking is a deep skill
| within itself.
|
| I've been doing large scale distributed systems for a decade,
| and I still feel like I'm learning especially as I retool for
| games.
|
| The way that I'm trying to make this simple is by inventing a
| new platform for "serverless game hosting" using my programming
| language (Adama : http://www.adama-lang.org/ ).
|
| My focus on board game is the ultimate in server's role on
| authority since board games are more transactional between
| players and staggered (I play this card, that card enables
| another play to react, that reaction enables another play to
| augment, etc).
|
| I'm looking into what it could mean to escape the board-game
| pit, and one thing that I'm thinking about is latency. The
| question that I'm looking at is whether or not I can have a
| programming model be differentiated such that the client
| prediction code can be generated.
|
| A trivial way to exploit my language is to bring edge computing
| into the mix such that the edge compute runs a copy of the
| entire game, and then ships deltas quickly which will be
| overwritten from the authoritative server. This will work on
| updates to existing data (i.e. moving position), but it will
| fail at creating new items unless I move unique id generate to
| clients (which, is less than happy for me).
| skipants wrote:
| Hell, I can forgive it in games -- it seems pretty easy to
| forget a detail that can accidentally give too much authority
| to a user in that space, especially when you're trying to
| crunch your frame time.
|
| What really surprises me is how how often I see this mistake in
| web development. "No, Bob, you should not increase the user's
| bank account based on that number passed in from the frontend
| React app."
| tshaddox wrote:
| I'm not sure exactly what you mean. If it's a web API request
| from a web client that says "transfer $1,000 between account
| A and account B," then what choice do you have but to "trust"
| that number? Obviously you have to check whether the request
| is authenticated and is authorized to transfer between those
| accounts, and check if account A has $1,000, but what about
| the _client origin_ of the request do you need to check?
| nend wrote:
| They're just saying to validate user input.
| skipants wrote:
| > and check if account A has $1,000
|
| It was a convoluted example for examples sake, but I'm
| pretty much referring to them missing this important check
| here.
| Tainnor wrote:
| I mean since we're talking about banking:
|
| - Generate a transaction number and associate that number
| with the respective transaction details
|
| - Send this number to the customer's mobile phone with all
| those details, or other configured device
|
| - the transfer is only authorised if the customer has
| entered the transaction number
|
| That way, the customer is very likely to have verified the
| details of the transfer.
| Thaxll wrote:
| > should not be able to shoot through walls or speed hack.
| These types of cheats are easy to prevent server-side, but
| developers don't because ....
|
| "easy" so you think veterant c++ dev that invented those
| concepts are dumb or lazy and it's a solved problem?
|
| It's not easy and not solved because those are very complicated
| issues.
| jbluepolarbear wrote:
| It is easy to prevent wall hacks and speed hacks. Prediction
| input resolution, server keeping an authoritative
| representation on the server, and limiting the synced objects
| to the current view of the player are all easy when done
| individually. Thinking about all these concepts at the same
| time can be confusing for people new to the topic, but each
| subject is relatively easy to understand and implement.
|
| I'd say that this article handles the speed hack issue
| because of authoritative server state, but doesn't go far
| into limiting what is sent to the player or not. The issue
| arises when you didn't make your game multiplayer from the
| start and adding prediction and sync visibility isn't an
| option, that's where you get player authoritative states that
| share their states with the other players. Client
| authoritative is easy to implement, impossible to prevent
| cheating.
| Sohcahtoa82 wrote:
| Baloney.
|
| Why is it hard to add server-side detection for line-of-sight
| when a player shoots at another player? The client should not
| be telling the server "I shot Player 2 and did X damage".
| Instead, the client should be saying "I shot in X direction
| at Y time" and the server should be reconstructing the world
| at Y time (As described in the Lag Compensation page of the
| article) to determine a hit. The server should be performing
| line-of-sight checks.
|
| In a game like PUBG, there's no reason a player should be
| popping headshots against players a mile away when there's a
| hill and/or several buildings between them. But the game
| allows clients to decide when they score a hit. At least, it
| did in the past.
| gameswithgo wrote:
| pubg doesn't allow shooting through things, the server
| checks if the shot was possible. instances where it feels
| like that are just due to lag. in pubg you can get hit
| after moving, with server side hit detection instead you
| miss when it looked like you hit.
| Thaxll wrote:
| You should read this:
| https://technology.riotgames.com/news/demolishing-
| wallhacks-...
|
| In anycase it does not work like it should in Valorant.
| Matheus28 wrote:
| Because of client-side interpolation and prediction, the
| client isn't on the same game state as the server, just an
| approximation of it. It's the tradeoff you have to make for
| a smooth game experience.
|
| The server can do a best effort attempt at rewinding the
| game state and take into account how the client is
| interpolating the world, but it gets very complicated as
| game complexity goes up.
|
| Someone correct me if I'm wrong, but I believe the Source
| engine does that for the attacker and the victim, but
| ignores all other entities for the ray casting (so it can
| only really prevent shooting through the map walls)
| Animats wrote:
| Yes. There are two issues here - "don't trust the
| client", and hiding lag. Not trusting the client is
| mostly about re-doing some checks the client supposedly
| already did. It's work, but it's not conceptually hard.
|
| Hiding lag requires guessing about what moving objects
| are likely to do. This works best for systems with a
| limited set of moving objects, such as players and
| bullets. As the author points out, bullets are handled as
| a very special case.
|
| I've done some work on this inside a Second Life / Open
| Simulator viewer. That's a very general system - anything
| can potentially move. You can make guns and shoot things,
| but the system has no idea of "bullet", just physics. The
| server is authoritative about everything except which way
| an avatar is facing. So turning is fast, but movement
| only happens after one round trip to the server. Round
| trip delay is typically 40-200ms. The problem is hiding
| that.
|
| The server sends the viewer updates with
| position/rotation, velocity/rotational velocity, and
| linear acceleration. That last makes projectiles describe
| an arc under gravity without server side updates.
|
| When an update comes in, the moving object has to be
| yanked back to where the server says it is. There are
| situations in which the predictor isn't very good,
| especially in the presence of jitter. And, the worst
| case, a region crossing, where there are no updates for
| 500ms-2000ms during the server handoff. The original
| predictor would sometimes put vehicles far into the air
| or into the ground for one update cycle. The predictor
| was using the last velocity, which is noisy as a vehicle
| bumps along a road following the outputs from the physics
| engine. As in real life, there's vibration. The last
| velocity, intended to be used only for 1/45 second, was
| being extrapolated to 500ms-2000ms, or sometimes as much
| as 100x the time it was intended for. So, low-pass
| filtering and a error metric was added to make vehicle
| movement look sane. The illusion can be maintained
| reasonably well up to about 200ms of lag, but beyond
| that, it shows as "rubber banding".
|
| Shooters just don't work well in a system this general.
| Archery is sort of OK.
| munk-a wrote:
| If lag exists this creates a fair - but really unfair
| feeling game. Clients will constantly be reporting shots at
| enemies that were quite accurate when the shot was taken
| but are now extremely off the mark.
|
| As the article mentions - fully server authoritativeness
| works great in lan situations and for single player content
| but can really fall on its face in multiplayer.
|
| If I, as a user, am constantly moving in random directions
| and have a lower ping than you (and thus my movements are
| reported to the server before you can actually observe them
| on your machine) then you should _never_ hit me except
| through pure blind luck - in fact accurate shots could be
| the only shots that are guaranteed _not_ to hit.
|
| Games want to feel bad at a range of reasonable pings - and
| reasonable pings usually stretch into the range of 100 or
| 150 ms - that leads to up to a half a second of lag between
| when I dodge left and you, as a shooter, would have the
| earliest chance to actually register a shot with the server
| at my new position.
|
| Additionally, the server can reconstruct what it thinks we
| saw at a given moment in time - but the server doesn't know
| what data has been lost[1] and whether the client might
| think it was right - or even whether the client might be
| misreporting dropped packets to engineer an advantageous
| game state. Also, this whole time we've been discussing
| latency as if it's a fixed number - but it is constantly
| changing and the reporting of the latency is something you
| need to trust to a client since clients can always
| artificially inflate latency figures.
|
| Due to lag correction either you're able to shoot me
| _after_ I duck behind cover, or you 're never able to shoot
| me - and both options suck for one half of the players
| involved.
|
| 1. See the two generals problem - proof of delivery is
| actually a _hard_ problem.
| thezilch wrote:
| You're not speaking to server authority but lag
| compensation.
|
| Your "behind cover" example describes best your concern
| with lag comp. It works both ways. You having lower
| latency can suffer when attempting to take cover.
| However, you have the advantage when exiting cover; you
| will see others first.
|
| It's as fair as it can be. As for feel, it definitely
| feels fairer than no lag comp.
| munk-a wrote:
| People with a higher ping can only benefit from it if
| there is lag compensation in place - that said... lag
| compensation isn't an on-off switch - it makes things
| unfair to users with that unfairness switching sides
| depending on how much compensation you offer since higher
| levels of lag compensation will result in unresponsive
| "muddy" controls where you may often be teleported back
| in time to die after locally taking an action you thought
| compensated for it.
|
| All these factors are pretty heavily interwoven in how
| the game ends up feeling to players.
| mysterydip wrote:
| Latency and performance, if you have to wait for the server
| every time you move or fire. It's a tradeoff which things are
| client and which are server.
| gameswithgo wrote:
| there are fun types of gameplay that can't happen without
| client authority. you can make an aimbot either way so it isn't
| a huge deal. like you i had a hard time accepting that pubg
| using client aide hit detection maee any sense, but after
| playing it a lot and thinking hard about it I changed my mind.
| That game is all about pixel perfect aiming. Lag compensation
| wouldn't cut it.
| flohofwoe wrote:
| One important reason is that P2P games that only require
| central services for matchmaking but not for the actual
| gameplay sessions are a lot easier and cheaper to scale to high
| player counts.
| judge2020 wrote:
| Maybe if you just have a flat world where you move in any
| direction with a max speed, but this sort of anti-cheat becomes
| incredibly difficult once you add features like terrain,
| physics, vehicles, things like gliders (in BR games), speed
| boosts, etc. It's possible, but takes a lot more effort.
|
| For example, in Fortnite, if you toggle a lagswitch (as in,
| block Fortnite UDP for some time), then reconnecting to the
| server will return you to the same position you were at when
| you disconnected, assuming you were just trying to run on the
| ground. However, if you toggle a lagswitch in a vehicle, the
| servers don't want to do expensive, complicated physics server-
| sided, so your client becomes authoritative to your position
| and the sanity checks are simply making sure you're not
| teleporting around large chunks of the map in seconds. If you
| were to hack your client and give your vehicle moon gravity,
| the server generally wouldn't care about that.
| jbluepolarbear wrote:
| I'm pretty sure the car uses dead reckoning. So you position
| in the car is static (the last place you where), but the car
| is moving at speed and will continue even if you disconnect.
| judge2020 wrote:
| Sure, but I'm talking about the control you have over the
| car when in the car. During the start of this battle pass
| season there was immense server issues, which I guessed was
| the tick rate dropping to super low values due to some bad
| code. During a few games where the servers degraded in this
| way, moving would be extremely slow on the ground as it
| kept pushing you back (since the server isn't processing
| your 'move forward' commands as often) but getting in a car
| gave back free movement since I assume it's a direct
| broadcast that doesn't hit the game loop and doesn't have
| any position-correcting sanity checks running.
| jbluepolarbear wrote:
| I'm saying a car is easier to predict because of dead
| reckoning. With less frequent updates your actions still
| feel fluid because the cars momentum can't change a whole
| lot between updates.
|
| Say you're driving a car in a straight line, there's very
| good odds that in .25 seconds you'll still be going
| straight so your predicted line will be very close to the
| real line. With players you can't guarantee that, they
| could zig zag, turn around, jump, etc much more likely to
| change and require prediction reconciliation.
| meheleventyone wrote:
| Cars and vehicles in general are actually much harder
| because player simulation is much simpler, usually not
| happening in some middleware physics engine and is much
| more bounded in terms of the accelerations and speeds
| involved.
|
| Straightforward extrapolation as with dead reckoning does
| poorly with both cases.
|
| Then once you've mastered the art of getting cars not
| looking wonky you have to deal with collisions between
| them when they're moving at speeds where typical gameplay
| latencies make it easy for discrepancies between clients
| to cause mispredictions which result in complete misses
| or large interpenetrations that physics engines have a
| bad time with.
| tshaddox wrote:
| I highly recommend this hour-long video from a Call of Duty
| developer discussing not only methods they use to transmit game
| states between client and server, but also some interesting ways
| that they measure true perceived latency (like using high-speed
| cameras and LCD monitors to compare true input-to-display
| latencies between different builds).
|
| https://www.youtube.com/watch?v=EtLHLfNpu84
| ggambetta wrote:
| Hey, I wrote that thing! Always a pleasant surprise and a weird
| feeling to open Hacker News (for the 14308560234th time today)
| and read my name like that :)
| dested wrote:
| I can't tell you how many times I find myself back on this page
| while working on multiplayer games. From the bottom of my heart
| I really can't thank you enough for this tutorial.
| adamrezich wrote:
| since you're here, it looks like the second paragraph under the
| second heading in the second article is an unfinished thought
| :) great work otherwise though!
| nightpool wrote:
| > Here's the crucial step. Since the server gets all the input
| with timestamps, it can authoritatively reconstruct the world at
| any instant in the past. In particular, it can reconstruct the
| world exactly as it looked like to any client at any point in
| time.
|
| Isn't there a really obvious exploit here? The client could lie
| about the timestamps and use that to take advantage of future
| information in making its decisions. For example, in a simplified
| fighting game where you have two attacks (top and bottom) and two
| blocks, it could always choose the perfect block to apply
| _before_ the opponent attacks--simply by seeing the incoming
| attack packet, and then immediately sending a block packet with
| an "older" timestamp. This doesn't entirely break the concept of
| an authoritative server, but it seems to diminish it
| significantly. I know lots of fighting games use this "rollback-
| type" netcode, maybe they've developed workarounds for this?
| reitzensteinm wrote:
| This absolutely is a concern with rollback networking (I have
| done a bunch of games in it).
|
| There are two ways you can cheat. Because you have the full
| simulation state and input, there is no possible precaution
| against map hacks.
|
| Some FPS games just won't send exact player positions until the
| client is about to be able to legitimately observe them. Not
| possible here.
|
| The other way is, as you say, you can build fake inputs.
|
| To do the timing attack as you describe, all of your input
| would have to be pretending to be slow. You can't send a t=10
| I'm idle packet, then a t=8 I blocked.
|
| You'd need a fantastic connection to the server, because the
| delay you're exploiting is there to protect you from your lag.
|
| Your oponent would also need a fantastic connection, because
| you aren't seeing their current inputs, you're seeing their
| past inputs.
|
| My feeling is that it would be very challenging to blend human
| like behavior with superhuman reflexes when doing automated
| responses.
|
| But there's nothing you're missing. The attack is definitely
| possible.
| hesdeadjim wrote:
| Highly recommend Glen Fiedler's articles for this topic.
|
| https://gafferongames.com/
| jbluepolarbear wrote:
| I'd also recommend this write up from JMP Waveran (really
| anything by JMP)
|
| https://mrelusive.com/publications/papers/The-DOOM-III-Netwo...
| lux wrote:
| His writing has been such a great learning resource. Can't
| recommend highly enough!
| adamrezich wrote:
| wow, that interactive example at the end is fantastic and does a
| great job of visually explaining the concepts of the articles!
| Raphaellll wrote:
| Another problem of predicting the movement of other players is
| that if they stop before you can see them, the trajectory would
| be interpolated further such that you can see them around the
| corner for a split second, knowing they are waiting there.
___________________________________________________________________
(page generated 2021-10-26 23:00 UTC)