[HN Gopher] Riot Games Tech Blog: Artificial Latency for Remote ...
___________________________________________________________________
Riot Games Tech Blog: Artificial Latency for Remote Competitors
Author : kevinU
Score : 182 points
Date : 2022-05-17 15:49 UTC (7 hours ago)
(HTM) web link (lolesports.com)
(TXT) w3m dump (lolesports.com)
| dematz wrote:
| As context this was a big international fiasco. Nobody was acting
| maliciously but fans of RNG and other teams felt wronged. RNG
| didn't want to attend due to covid restrictions on travel but
| Riot really wants Chinese viewers in international tournaments,
| hence all this online latency work. Pros on other teams were lied
| to by the false latency information which must be almost paranoia
| inducing to be told it's only 35ms even when it (correctly) feels
| higher. In the end RNG had to replay their games, which was not
| as bad as it could have been since they were vs much weaker
| teams, but still annoying.
|
| Anyways, by "scenarios where the actual ping was significantly
| lower than the target latency" do they mean like if the ping from
| Busan-server is 10 but they want it to feel like 35? I feel like
| that's the one scenario you'd care about, making the Busan
| players have 35 ping was the whole point.
|
| Edit - also the other scenario, where the actual ping is
| significantly _higher_ than the target latency...well in that
| case your latency service would be some sort of magical time
| travel box! Adding latency is literally the _only_ thing it can
| do right? It 's not possible to lower it. Maybe the it depends
| what they mean by "significantly" but idk the target latency was
| 35ms so like at most they could be 35ms lower than that.
|
| I guess they didn't catch it because the bug was in the latency
| service itself which gave the wrong value to both the in game
| display and to the old network monitoring system. Idk in
| hindsight it's easy to say if you're going to introduce this new
| ping equalizing service it might break how you test ping
| especially because it hasn't been tested in soloq. But meh a lot
| easier to see what happened in hindsight after they write it all
| up.
| ear7h wrote:
| > I feel like that's the one scenario you'd care about, making
| the Busan players have 35 ping was the whole point
|
| I think the key word is "significantly". The post states that
| the tool was originally developed for matches where both teams
| would be remotely connected, so neither team would have a
| significantly different latency (you can usually enginneer this
| by the first approach they suggested, of picking a server in
| between two locations).
| vikingerik wrote:
| You can in fact have that magical time-travel approach to get
| apparent latency lower than the real number. Keep a history of
| the game state at every tick, so you can roll back to any
| previous instant. If real latency from one client is 50 ms, and
| you're targeting 35 ms, you roll back the game state by 15 ms
| each time input comes in from that client, and re-run the game
| logic from that point forwards and send the resulting state
| update to all clients. Those clients will see a bit of
| jumpiness in the game state, which isn't ideal, but may be the
| lesser evil compared to enduring more latency.
|
| (I have no idea what LoL in particular supports or implements,
| but that's the general idea.)
| [deleted]
| __turbobrew__ wrote:
| I really dislike games which rewind state based upon latency.
| It penalizes people with good ping and can lead to very weird
| game states where a large rollback can cause teleportation of
| a laggy player as seen by non laggy players.
|
| Team Fortress 2 is particularly bad where high latency
| players can teleport around or you can shoot and hit a laggy
| player but it doesn't register by the server due to a
| rollback. Some players purposely exploit this function by
| artificially increasing their ping values to absurd levels
| (700ms or more) which makes them very hard to hit. Many
| community TF2 servers enforce maximum ping limits for this
| reason.
|
| Most notably high pings are used to exploit when playing as
| the Spy class due to the fundamental purpose of the class is
| to get behind you to backstab you. If you have 700 ping can
| abuse the fact that you you can get behind players due to the
| game being out of sync and if you can backstab a player the
| server will roll back and kill the non-laggy player even
| though you were no where near behind them from their non-
| laggy perspective.
| meinersbur wrote:
| That's just another kind of latency. Additional time has
| still passed from you (or the enemy player) doing an action,
| and you seeing the results of that action on your screen.
|
| By contrast, the server rolling back the game state is
| indistinguishable from the server always being 15ms behind
| and the clients running the simulation ahead of time (which
| they do anyway for fluency [1]). The "jumpiness" is called
| rubberbanding [2].
|
| [1] https://www.gabrielgambetta.com/client-side-prediction-
| serve...
|
| [2] https://www.urbandictionary.com/define.php?term=rubberban
| din...
| [deleted]
| jchw wrote:
| This strategy is commonly referred to as being in the class
| of "latency hiding" as it doesn't actually improve latency,
| it just allows you to reduce _apparent_ latency. In practice
| it's still useful but I think it's _mainly_ useful because it
| allows for _local_ input to be buffered significantly less
| than if you were to compensate for lag by adding buffering
| only. However, _that_ is not always useful. It is mainly
| useful in games where netplay is handled in a lock-step
| fashion, where each peer needs to keep every action coherent
| frame-wise. That makes sense for things like fighting games,
| but it doesn't make all that much for say, puzzle games, even
| though those can be quite latency sensitive too.
|
| Either way, there is nothing magic. At the end of the day,
| the soonest you can see an action is still physically
| unchanged. Especially since the prediction model that makes
| the most sense is just assuming no action.
|
| That said, I dunno enough about MOBAs to try and postulate on
| how much it matters, and I have no idea if LoL is using
| lockstep or some other approach.
| skocznymroczny wrote:
| LoL is not using lockstep. Lockstep has two big
| disadvantages for a competitive game. Tt requires both
| client to have the full game state to run the simulation,
| which makes creation of maphacks trivial. The second issue
| with lockstep is that if one player is lagging, all players
| are lagging, which is also not something you'd like to have
| in a fast paced competitive game.
| reitzensteinm wrote:
| What to do when one player is lagging is a design choice.
| You can instead keep the game going and feed in their
| delayed inputs.
| bee_rider wrote:
| I can definitely see why the maphack thing would be an
| issue generally, but in the case of a broadcast esports
| competition is it so significant? They seem to have put
| an awful lot of engineering effort into their special lag
| system for this tournament, so I assume they at least
| have some pretty decent budget. So, presumably they could
| create a special client (possibly even sending the
| competitors locked down hardware). This seems like it
| would let them do some things we'd normally consider
| impossible or 'magic' -- assume both sides of the pipes
| are going to tell you the truth about everything...
| jimmydorry wrote:
| There are pretty regular esports scandals involving
| cheating. Heck, we've seen players bring their cheats in
| via special drivers embedded in their peripherals (which
| is why almost every serious in-person tournament provides
| peripherals + PC and does not allow the player access to
| them outside of supervision).
| [deleted]
| [deleted]
| whateveracct wrote:
| A good software lesson here.
|
| They built a complex system to tweak tens of ms (at most) of ping
| to equalize. And they had a bug and it disadvantaged on team.
| Thus, they have to replay due to unfairness introduced by Riot
| itself.
|
| They could've gone Option 1 - all teams at their natural ping.
| While it wouldn't be as perfect in theory, it wouldn't have
| resulted in replaying matches. Plenty of other esports and
| fighting games play at natural ping and have real results. There
| is unfairness due to location, but only scrubs blame 5-15ms ping
| advantage for a loss in a game as strategic as LoL.
|
| People play Melee with ping differences bigger than that and it
| works fine lol. Waaay more technical game shows that there's
| tolerance.
|
| Not to mention that the server option could've had some system to
| distribute across advantageous servers over the course of a
| series.
| holmium wrote:
| > People play Melee with ping differences bigger than that and
| it works fine lol. Waaay more technical game shows that there's
| tolerance.
|
| I dunno; I'd imagine people would be mad if there was a 15ms
| ping differential in Melee. I think the online/netplay
| tournament results are already looked at different than
| offline, console results.
|
| Moreover, I'm pretty sure that you can't have a differential
| ping in a P2P game with rollback, so the competitive issues are
| different between the two scenes. I understand Riot wanting to
| equalize them. In Melee specifically, I think that the players
| have to decide on a common frame delay to play, right? If so,
| then the Melee community has come to the same solution for
| competitive netplay as Riot has.
| whateveracct wrote:
| Most Melee players use a frame delay that's equivalent to the
| lag present in a Gamecube + CRT setup. A little artificial
| input lag goes a long way!
| darkwizard42 wrote:
| I think you are completely incorrect when you say that a 5-15ms
| ping advantage doesn't cause a massive effect in a game of LoL.
| Even the greatest player of the game (Faker) mentioned the
| difference between "35"ms and the near 0 that Korean players
| are used to playing.
|
| Low ping is part of players' muscle memory in how they react to
| visual signals. Changing that slightly creates a big
| difference, especially in the early and mid game where the
| outplays are less strategic and can often be more mechanical.
|
| Even in later game teamfighting, the familiarity with latency
| between key/mouse input and actual in-game movement is
| mechanically intense (ex. auto-spacing in a close teamfight)
| DixieDev wrote:
| This shouldn't detract too much from your point, but for
| fighting games the situation is slightly different from most
| other genres:
|
| Modern fighting games, as well as older ones that have been
| modded to included this (such as Melee), have a netcode model
| called "rollback" that can negate network-induced latency on
| local inputs. The trade-off is that what you see is usually
| inaccurate until you receive inputs from the remote player's
| machine.
|
| Despite fighters often requiring fast reaction speeds, this
| downside is not a big deal granted that the ping is low. The
| first few frames of many actions are generally quite subtle,
| enough that they are quite hard to distinguish from each other;
| it's usually specific keyframes/poses that people will actually
| be reacting to.
|
| Melee is probably hurt the most though, as movement is
| extremely fast and very important, and the difference between
| wavedashing left or right is not masked by subtle animations.
| gosukiwi wrote:
| There is a huge difference between 0 ping and 35 ping though,
| particularly in pro-level LoL. It is strategic but that ping
| can be the difference between flashing a skillshot or not, and
| swinging the whole course of a teamfight.
|
| Of course 35 ping is not THAT bad and I agree, while unfair, it
| wouldnt be the end of the world, it's not even Worlds, just
| MSI, an invitational event.
| tester756 wrote:
| MSI is "just" 50% of international tournaments
|
| imo it's very serious tournament
| gosukiwi wrote:
| It is a very serious tournament, it's just not as important
| as Worlds
| [deleted]
| xboxnolifes wrote:
| > People play Melee with ping differences bigger than that and
| it works fine lol. Waaay more technical game shows that there's
| tolerance.
|
| Sure, people play with ping disparity in tons of competitive
| games, but that's because there's not really an alternative for
| online play. It's "play with a ping disparity" or "don't play
| at all".
|
| Once there is more widespread reliable alternative, which Riot
| seems to be working toward in their own game tournaments, maybe
| we'll see if this tolerance persists in competitive
| tournaments.
| gareth_untether wrote:
| Reminds me of the cable lengths for black boxes connected to the
| network in Wall Street. Each cable is the same length regardless
| of which computer is closer to the access point.
| jedberg wrote:
| I never understood why they didn't use randomized length micro
| batches to solve this.
|
| Instead of processing orders instantly, wait between 200ms and
| 500ms and then process all orders that came in that window in
| random order. Then being 5ms closer to the server wouldn't
| matter.
| blibble wrote:
| this is called a periodic auction and has been common on
| European exchanges for nearly a decade
|
| (though the matching algorithm varies)
| teen wrote:
| That's the same problem, you're just trying to get into the
| earliest window
| jedberg wrote:
| Ok but you don't know the window size for any given batch,
| so game theory-wise your best bet is to just process as
| fast as you can and then place your order.
| w-m wrote:
| I don't think I get how that solves the issue: you would have
| set a fixed cutoff time, where you switch from one
| window/batch to the next. It doesn't matter when you arrive
| within the window. But statistically, even for random window
| lengths, if you have a smaller latency, you will make the
| cutoff for the earlier window more often.
| foota wrote:
| If the window is as large as 500ms then anyone can hit that
| most of the time, provided they have a good enough clock
| and decent network.
| jedberg wrote:
| But because of the random ordering within the windows, it
| won't matter. In most cases you wouldn't hit the cutoff.
| pdpi wrote:
| 2-500 ms latency for games is atrocious. That's an order of
| magnitude higher than what this tournament was targeting.
| jedberg wrote:
| This thread is about HFT.
| pdpi wrote:
| So it is. I somehow managed to miss that at the top of
| the thread. My bad!
| axg11 wrote:
| That sounds like a complex solution. Sometimes a dumb
| solution that works good enough is better than a complex
| solution that _probably_ can't be exploited.
| jedberg wrote:
| It's complex on one end to reduce complexity on the other
| -- the trading companies wouldn't have to worry about
| millisecond optimizations if they trading batches were
| 200ms windows. So the wire lengths wouldn't matter but also
| not mattering is the processor, memory, software, etc. for
| the trading companies. Seems like a good tradeoff.
|
| And honestly the wire thing probably isn't real. Light
| moves 30cm in a nanosecond. The wire lengths could be 3
| meters different and only make a 10ns difference. Not sure
| that would matter all that much.
| JamesSwift wrote:
| I think it just moves the goal post of the problem. If I
| now have a 200ms window, then I need to optimize for that
| 200ms cutoff. I want to do as much work as possible while
| _still reaching the cutoff_ and so minimization of "non
| work" still matters. Not only that, but now you
| incentivize filling up that 200ms with 200ms of analysis.
| gmueckl wrote:
| Isn't that where the randomness of the trading interval
| comes in? How do you optimally fill out a time interval
| with a length that is unknown to you?
| JamesSwift wrote:
| Ah, I didnt interpret the comment as being a variable
| delay. I think the point still holds though that you
| would still optimize for the lower bound.
| jedberg wrote:
| You could, but then if say the delay is 439ms, then
| you'll sit idle for 239ms. It would make more sense from
| a game theory perspective to simply process for as long
| as necessary and then place the order.
| blibble wrote:
| the wire thing is definitely real for IEX
|
| it's part of the exchange's license
|
| and they have several videos of the rather large wire
|
| there's even a tom scott video on it:
| https://www.youtube.com/watch?v=d8BcCLLX4N4
| [deleted]
| axg11 wrote:
| I used to work in HFT. I promise you that companies would
| still try to exploit randomized batches. There is an
| advantage to being the very last entrant into a batch
| (most up to date information). Truly random batches are
| not trivial to implement and any statistical pattern in
| the batching could be exploited.
| jerf wrote:
| "Truly random batches are not trivial to implement and
| any statistical pattern in the batching could be
| exploited."
|
| If an HFT manages to exploit a _correctly_ implemented
| entropy-pool random number generator using AES-256 to
| extend the stream as needed, they 're welcome to pick up
| a few more bucks as far as I'm concerned.
|
| Yes, problems have existed before but I'm sure in this
| case we can assume high-assurance and careful
| programming, not some fresh grad being assigned the
| problem and thinking linear congruential generators sound
| really cool.
| teddyh wrote:
| Probably because such a system is not provably fair - you
| can't prove your system is truly random.
| jedberg wrote:
| You would use an external source of randomness, which you
| can definitely prove is _sufficiently_ random.
| teddyh wrote:
| You could do something like this:
|
| * Pre-publish, for each time batch, a public key. You
| could publish lists of these well in advance.
|
| * Let everyone submit, alongside each order, a number
| arbitrarily selected by them. It does not matter how they
| select the number, but it would be simpler if everyone
| chose distinct numbers.
|
| * When order processing is done, do it by the order of
| closeness of the submitted number to the _private_ key.
| Publish the private key after order submission is closed.
|
| * Everybody can now verify that the order of processing
| is indeed by the order of the previously secret private
| key, and everybody can verify that the private key
| corresponds to the previously published private key.
| Natfan wrote:
| Alternatively, just do what Cloudflare does (did?), and
| have some cameras pointed at lava lamps[0].
|
| [0]: https://www.youtube.com/watch?v=1cUUfMeOijg
| can16358p wrote:
| Something very similar is done on smart contracts to
| introduce randomness (at least to the practical extent)
| on blockchain. Good thought experiment nevertheless to
| provably inject randomness into a system with untrusting
| parties.
| jedberg wrote:
| That seems unnecessary. Today the traders have to trust
| the exchange to process fairly and in order -- there is
| no verifiability. The verifiability would be nice, but
| unnecessary.
| jakey_bakey wrote:
| That's neat, I didn't know about this. Very cool.
| Misdicorl wrote:
| Perhaps apocryphal/silly, but amusing nonetheless. Story goes
| that this means you want to be in the computer _furthest_ from
| the interconnect because light travels _slightly_ faster in
| straight fiber than in coiled fiber.
| colejohnson66 wrote:
| Not apocryphal; IEX has 38 miles of wire in their building.
| Tom Scott did a video a few years ago about it:
| https://www.youtube.com/watch?v=d8BcCLLX4N4
| vmception wrote:
| Does IEX have any liquidity or uptake yet?
| infinityio wrote:
| Apparently they have 2.6% market share for trading in the
| US?
| Humdeee wrote:
| I work in this space and although true (this is called modal
| dispersion), there are compensation techniques always used in
| termination equipment to 'handicap' or mitigate these
| occurrences. Not perfect, but very, very, very small deltas.
| Unsure if it's enough to act upon without knowing the length
| from output to next input.
| upwardbound wrote:
| It looks like the error they made was this: They added too much
| latency because they forgot to account for the latency they added
| in the client. You can see from their architecture diagram
| (Figure 4) that the latency measurement didn't include the client
| delay.
|
| https://images.contentstack.io/v3/assets/bltad9188aa9a70543a...
|
| The blog post states: "The existing network monitoring system
| measured the latency at the networking layer as shown by the
| green arrow."
| chabons wrote:
| I read that section as comparing what they logged before vs
| what the player experiences: the time taken for an input to be
| registered and it's effect communicated to the user. Maybe I'm
| wrong, but I don't think the intention was a precise
| description of the error, but rather the hazard of not having
| logs which reflect what the user experiences.
| upwardbound wrote:
| They stated that they use the same calculation for the logs
| as for other purposes such as latency compensation. From near
| the top of the blog post:
|
| "The reason we did not find it sooner is that the cause of
| the issue was a code bug that miscalculated latency, which
| meant that the values in our logs were also wrong."
|
| And from later in the post:
|
| "Our logs were not displaying the issue because the
| calculation was wrong. It explained why the latency was worse
| in the venue than on the internet servers"
| chabons wrote:
| Right, but my point was that I think you're reading into
| the diagram beyond it's intended purpose. I'm not sure how
| you can tell that they're logging only the server-side
| delay, and not the client-side delay, from a systems
| diagram which shows each of those as a box connected by a
| bi-directional arrow. For instance, it's clear to me that
| they don't include the latency included by the game engine
| in that metric, but not whether they include the server-
| added latency or if they're aware of it and subtract it
| off.
| mleonhard wrote:
| They used a complex and buggy solution when a simple solution
| exists: Use the `tc` traffic control [0] program to configure the
| Linux kernel to add a fixed amount of latency to traffic from the
| local players [1]. If the game server does not run on Linux, they
| could put a Linux bridge/router in front of it.
|
| [0] https://en.wikipedia.org/wiki/Tc_%28Linux%29
|
| [1] https://serverfault.com/a/841865
| connor4312 wrote:
| As I understand the blog post, the difficult (and buggy) part
| is not the addition of latency, but calculating how much
| latency to add and where to add it. I'm not sure how tc would
| help much here, and actually don't see anything to indicate
| they weren't using tc already.
| spmurrayzzz wrote:
| `tc` would give you non-deterministic microbursts that would be
| hard to quantify as to whether they impacted competitive
| integrity. This is an incredibly complex problem if they
| actually care about millisecond level precision in small duty
| cycles.
| someweirdperson wrote:
| Artificial or natural delays, I am somewhat confused that one end
| measuring/calculating ping can end up with a wrong value.
| Transmit at local time x, receive at local time y, subtract. It
| only needs a working local clock.
|
| Second surprising thing is artifical latency added on the client.
| Anything on the client I would avoid, if possible.
|
| Third, this all looks like magically connecting two ends. But it
| is packets flowing from one to the other, and the other to the
| one. Latency could be added to each independently, even on one
| end, by artificially buffering to-be-send and/or received
| packets.
|
| What are the implications for game-play with an asymetric
| latency, e.g. theoretical 0 delay to receive vs. long delay to
| transmit, or vice versa?
| chabons wrote:
| I think the symmetric delay here makes sense, since you're
| trying to simulate the latency of a (likely) symmetric
| connection between another client and the server.
|
| If you only add latency on the transmission, then 2 actions
| taken at the same time by players with different amounts of
| physical latency will be processed at different times. If we
| assume the game clocks on each client and the server are in
| sync, then this creates a competitive advantage for the player
| with lower physical latency. For instance, if we have a timing
| battle (ie: Zhonya's hourglass), where both players know that
| at time 'T' they must each take an action, and the first to do
| so comes out on top, then the player with higher physical
| latency is at a disadvantage.
| [deleted]
| bcrosby95 wrote:
| You can accomplish this by putting it all on the client, all
| on the server, or a mix of both (what they chose).
|
| For all on the server, you could add latency right after
| receiving data from the client, but before game code
| processes it, and before sending data to the client.
|
| That said, there likely were technical considerations
| involved in their decision to do a mix of both.
| chabons wrote:
| That's a good point, I hadn't thought of having independent
| send a receive delays on the server, simulating a delayed
| connection from the client's perspective without having it
| participate.
|
| I now wonder too what those technical considerations are,
| since having the client add latency requires trusting the
| client to add the delay and not cheat.
| [deleted]
| [deleted]
| frozenlettuce wrote:
| Warcraft 3's (pre-reforged) Battle.net had a 250ms delay for all
| players (when it was launched dialup connections were still
| common)
| tester756 wrote:
| I still don't understand
|
| they didn't let anyone proficient at LoL even try using it?
| colinmhayes wrote:
| The difference was .01 seconds. It's not immediately obvious
| that there is a problem.
| RheingoldRiver wrote:
| It was immediately obvious to competitors and reported on day
| 1 of competition.
| joebob42 wrote:
| I suck at lol, the amount of ping it takes for me to be able
| to tell it's different is only like 30-40 ms. Pro players
| being able to tell and care at half that seems pretty likely.
| tester756 wrote:
| >The difference was .01 seconds. It's not immediately obvious
| that there is a problem.
|
| in games you really feel ping differences
|
| if you were playing e.g for a year on 5ping and then jumped
| on 30, then you'll feel that in games like LoL and probably
| CSGO? idk about the second one.
| colinmhayes wrote:
| They said the ping was 45 instead of 35. I'm not denying
| that the pros can tell the difference, I'm saying it
| wouldn't be obvious to whatever tester they had that it was
| 10ms off.
| Karrot_Kream wrote:
| As others have said, the pros noticed. Having an
| incorrect calculation and seeing large jitter spikes can
| probably push the latency into something like 50ms
| momentarily, enough for pros to lose their "immersion".
| jannyfer wrote:
| The part I was interested about the most was glossed over.
|
| In what situation can this happen?
|
| > we realized that there was a calculation error that only
| manifested in scenarios where the actual ping was significantly
| lower than the target latency. In this situation the actual
| latency would be considerably higher than what is displayed on
| the overlay on the player's screens.
| bobogei81123 wrote:
| My guess is that they did not add in the client delay.
| upwardbound wrote:
| Exactly right.
|
| They added too much latency because they forgot to account
| for the latency they added in the client. You can see from
| their architecture diagram (Figure 4) that the latency
| measurement didn't include the client delay.
|
| https://images.contentstack.io/v3/assets/bltad9188aa9a70543a.
| ..
|
| The blog post states: "The existing network monitoring system
| measured the latency at the networking layer as shown by the
| green arrow."
| jannyfer wrote:
| That kind of error would apply all the time.
|
| But the blog post states: "a calculation error that only
| manifested in scenarios where the actual ping was
| significantly lower than the target latency".
| upwardbound wrote:
| Sadly, my guess is that they are transparently lying
| about that. Since they apply the lag half on the client
| and half on the server, their lag compensation would have
| been off by a factor of 2x (or a factor of 100% from a
| relative perspective) and so they might be just claiming
| that a 100% error isn't very large when the total lag
| difference is small. "Yes we were supposed to add 2ms and
| we added 4ms instead, but at the end we were still only
| wrong by 2ms (in the other direction) which is not a big
| deal."
| joebob42 wrote:
| I think in practice this issue doubles the added latency,
| which would only be noticable if the actual ping is
| "significantly" lower.
|
| Because realized ping is over by as much as original ping
| was under, so if original wasn't significantly under...
| mleonhard wrote:
| That's also the kind of error that one can easily catch with a
| unit test.
| wswope wrote:
| Knowing the classic Riot Games MO: I'd bet money there was a
| plus instead of a minus that made it through code review, and
| they're being intentionally vague to save face.
| NelsonMinar wrote:
| Some data: 8 years ago someone found going from 35ms to 0ms
| latency meant you were likely to win games 1-2% more of the time:
| https://www.reddit.com/r/dataisbeautiful/comments/1t23a0/lat...
|
| And this is a little different, but Riot found that playing on
| ethernet instead of wifi made you about 1% more likely to win
| games.
| https://web.archive.org/web/20160814131032/http://na.leagueo...
| Karrot_Kream wrote:
| I'm going to guess this effect is lower because it's being
| marginalized across all player skill levels (ELO). In Riot's
| post you can see that the higher tier the player is at, the
| more likely they are to use Ethernet. If you conditioned on ELO
| I suspect you'll find a much larger effect.
| Kuinox wrote:
| Seems something very hard to do in software but very easy to do
| in hardware.
| bob1029 wrote:
| I'd argue the complexity of trying to achieve this at all is
| too much to bear in practice, especially considering the type
| of customers this would be inflicted upon. Competitive gamers
| will now _also_ be wondering if the "fake lag" system is
| bugged, in addition to all of the other problems that could
| still exist.
|
| Some problems cannot be solved with clever tricks.
| infinityio wrote:
| if they pulled an IEX and just ran the Busan clients through
| a few km of fibre to induce delay they could use native pings
| and verify easily
| bostonsre wrote:
| Think other people have already done the hard part. Wouldn't be
| too hard to write something that wraps tc and ping.
| # hping -S -p 1234 somehost.com HPING somehost.com (ens3
| 1.2.3.4): S set, 40 headers + 0 data bytes len=46
| ip=1.2.3.4 ttl=64 DF id=0 sport=1234 flags=SA seq=0 win=26883
| rtt=0.9 ms len=46 ip=1.2.3.4 ttl=64 DF id=0 sport=1234
| flags=SA seq=1 win=26883 rtt=0.8 ms len=46 ip=1.2.3.4
| ttl=64 DF id=0 sport=1234 flags=SA seq=2 win=26883 rtt=0.7 ms
| # tc qdisc add dev ens3 root netem delay 25ms #
| hping -S -p 1234 somehost.com HPING somehost.com (ens3
| 1.2.3.4): S set, 40 headers + 0 data bytes len=46
| ip=1.2.3.4 ttl=64 DF id=0 sport=1234 flags=SA seq=0 win=26883
| rtt=25.9 ms len=46 ip=1.2.3.4 ttl=64 DF id=0 sport=1234
| flags=SA seq=1 win=26883 rtt=25.8 ms len=46 ip=1.2.3.4
| ttl=64 DF id=0 sport=1234 flags=SA seq=2 win=26883 rtt=25.7 ms
| merb wrote:
| well adjusting the latency is the easy part. doing it in real
| time while observing the connection and making adjustments,
| that is when it becomes hard.
|
| and they did it basically after the network stack, so if a
| paket took 40ms they added no latency if it took 20ms they
| added 15ms and so on.
|
| so EACH paket was evaluated and of course this is not an easy
| feat, because lol is serverside authoritive so the latency
| must be correct inside the server and the client for each
| packet.
| bostonsre wrote:
| Yea, I guess the question is whether or not to try to do it
| at the packet level or just keep a running average of ping
| latency and adjust it over time. They said `adjust the
| latency (ping) to 35 ms` so I am assuming they are just
| trying to do something like attempting to maintain an
| average ping latency of 35 ms.
| joshuak wrote:
| Distributed consensus is hard. Very hard. Anyone dismissing the
| problem as "should never have hopped," hasn't a clue as to how
| hard and error prone reliable distributed consensus is. 35 ms lag
| in overall responsiveness is an eternity in competitive game
| play.
|
| There are videos floating around that show drawing on a tablet
| surface with various input latencies (perhaps someone has a link,
| I can't find them at the moment). 35ms latency is very noticeable
| to anyone never mind professional competitors.
|
| Because of the tricks game designers must pull off in lieu of
| proper distributed consensus which has hard requirements bound by
| the laws of physics, it is completely likely there are lots of
| bugs in the system. I think Riot did the best anyone could
| reasonably have expected of them, and the write up is
| particularly informative and helpful.
| joebob42 wrote:
| There's no consensus involved in the bug.
| hsnewman wrote:
| Decwars used to "equalize" latency for consoles verses dialups
| (in the 1970's)...
| NelsonMinar wrote:
| Wow for real? That's amazing. I'd love to read more about that,
| it seems quite innovative!
| sdwr wrote:
| I love the intent and actions behind this post. Finding an unfair
| bug mid-tournament is serious, and it takes courage and integrity
| on the part of the technical team to
|
| -listen to the players
|
| -investigate past what your own analytics are saying
|
| -find and fix the bug under time pressure
|
| -and disclose the (potentially embarrassing!) initial mistake
|
| I think the article itself could have been worded better though.
| It didnt feel very clear. As a reader, I want to know what went
| wrong, how it was handled, and feel confident that you are on top
| of the issue. All the stuff about "why we chose our network
| topography" reads as details at best, justification/excuses at
| worst.
| bigcat12345678 wrote:
| Yep, look at what Valve is doing to dota 2...
| gverrilla wrote:
| would you care to elaborate? I didn't play dota in a few
| years
| DantesKite wrote:
| The technical team acted honorably, but the upper stewardship
| at Riot games behaved horrendously.
|
| They sacrificed the competitive integrity of their sport
| because they didn't want to exclude RNG from the tournament,
| because they wanted the audience of the Chinese market.
|
| This article goes into more detail, giving a larger context for
| why this situation shouldn't have never happened in the first
| place.
|
| https://www.invenglobal.com/articles/17207/montecristo-its-e...
| colinmhayes wrote:
| > because they wanted the audience of the Chinese market.
|
| Or maybe because the tournament would've been shit without
| them? Who wants to watch an international tournament with
| only 1 of the clearly top 2 regions represented? WHole thing
| would've been a forgone conclusion at that point.
| DantesKite wrote:
| Yes, the tournament would have been worse without them.
| What I'm saying is that does not somehow justify penalizing
| the other teams and the tournament writ large to "even" the
| playing field if you value competitive integrity.
|
| As the article states, increasing the ping doesn't make it
| more fair for the other teams because they actually
| traveled to the event. They're physically there. It only
| makes it more fair for RNG who can't attend because of
| Covid restrictions.
|
| To paraphrase, it "punishes the teams who are there for a
| team that isn't."
|
| From a business perspective of course, I agree with you.
| It's definitely better to sacrifice the level of play for
| viewership (and it does sacrifice the level of play because
| artificially increasing the ping at a LAN event makes the
| game worse).
|
| But if you're running a competitive eSport and you want a
| fair sport, it's a terrible decision.
| bee_rider wrote:
| Fairness isn't a perk applied to a team, it is a property
| of the game.
|
| Unless "ability to travel during a pandemic" is
| considered a skill for this game, it should not be
| factored into the competition. So, if teams who can't
| travel to be game are going to be included, then the host
| must find a way to provide a fair game. This means that
| remote and local players should have the same latency.
| colinmhayes wrote:
| > increasing the ping doesn't make it more fair for the
| other teams because they actually traveled to the event
|
| Can't agree. Those teams had an advantage when playing
| RNG. Taking away that advantage does actually make the
| event more fair for them, even though they are being
| disadvantaged.
| DantesKite wrote:
| Could you clarify what you mean by "those teams had an
| advantage when playing RNG"? I may have missed something.
| I'm not sure what advantage you're referring to.
| colinmhayes wrote:
| Without the ping adjustment the travelling teams would
| have a ping far lower than RNG. That is not fair, even if
| it is out of the teams control.
___________________________________________________________________
(page generated 2022-05-17 23:00 UTC)