[HN Gopher] How I cut GTA Online loading times by 70%
___________________________________________________________________
How I cut GTA Online loading times by 70%
Author : kuroguro
Score : 1078 points
Date : 2021-02-28 19:38 UTC (3 hours ago)
(HTM) web link (nee.lv)
(TXT) w3m dump (nee.lv)
| johnnyAghands wrote:
| Face palm. Thanks for doing this, maybe Rockstar will finally fix
| this bullshit.
| ufo wrote:
| The part that puzzles me the most was this comment about sscanf:
|
| > To be fair I had no idea most sscanf implementations called
| strlen so I can't blame the developer who wrote this.
|
| Is this true? Is sscanf really O(N) on the size of the string?
| Why does it need to call strlen in the first place?
| mstade wrote:
| This is some first rate debugging, and great writing to boot. I
| hope Rockstar sees this, fixes the bug and then pays this fella
| _something_. Great post, thanks for sharing!
| JoshMcguigan wrote:
| > Normally Luke would group the same functions together but since
| I don't have debugging symbols I had to eyeball nearby addresses
| to guess if it's the same place.
|
| I really enjoyed this article, but I do have a question about
| this part. Why would a single function be listed at mutliple
| addresses?
| will4274 wrote:
| Doesn't surprise me at all. It's an O(n^2) algorithm (strlen
| called in a loop) in a part of the code where N is likely much
| smaller in the test environment (in-app purchases).
|
| Overwatch is another an incredibly popular game with obvious bugs
| (the matchmaking time) front and center. And gamers are quick to
| excuse it as some sort of incredibly sophisticated matchmaking -
| just like the gamers mentioned in OP.
|
| It's easy to to say it's something about gamers / gaming / fandom
| - but I have a locked down laptop issued by my bigcorp which is
| unbelievably slow. I'd bet a dollar there's a bug in the
| enterprise management software that spins the CPU. A lot of
| software just sucks and people use it anyway.
| jrockway wrote:
| I am not sure Overwatch's matchmaking time is a bug per se. The
| time estimates are bad for sure. But the matchmaker can really
| only be sure of one state -- if you queue for a match, a match
| will be made. The rest is predicting who will show up, along
| with some time-based scale for making a suboptimal match in the
| interest of time. Players absolutely hate these suboptimal
| matches, so the time threshold ends up being pretty high. The
| rest seems to just be luck; will the right combination of 11
| other people be in the right place at the right time?
|
| I think it could be improved, but it doesn't strike me as being
| buggy.
|
| (Overwatch itself, lots of bugs. Tons of bugs. If they have any
| automated tests for game mechanics I would be pretty
| surprised.)
| will4274 wrote:
| No, it doesn't add up. I can see dozens of groups filling up
| and queueing at my groups level as I wait for matches. Worse,
| many of my matches just aren't that evenly balanced. Even if
| you believe the game is dead now, things were just as bad
| (10+ minute queues for full 6 stacks) at the peak. They don't
| do tight geographic bindings - I live on the US west coast
| and regularly get Brits and Aussies in my games.
|
| I guess what they are probably doing is batching groups of
| games and then matching for the entire batch, to ensure
| nobody gets a "bad" match. What they've missed is that well -
| 5% of matches are bad baseline because somebody is being a
| jerk or quits or has an internet disconnect or smurfs or
| whatever other reasons. They could have picked an algorithm
| that gave fast matches 99% of the time at the cost of having
| bad matches 1% of the time and nobody would have noticed,
| because their baseline bad match rate is so high.
| Optimization from too narrow a perspective.
|
| Honestly, the OW matches I get aren't any more balanced that
| the COD matches I used to get, and I got those in a minute,
| not 15.
| qyi wrote:
| It's certainly not unique to game consumers. People in general
| just blame every fault on "it's physically impossible to
| solve". One big reason why corporations get away with creating
| non stop worse and worse products.
| saagarjha wrote:
| Reading things like these is bittersweet. One one hand, I am glad
| to see that the art of figuring out "why is this thing slow" is
| still alive and well, even in the face of pushback from multiple
| fronts. On the other hand, it's clear that the bar is continually
| rising for people who are looking to do this as a result of that
| pushback. Software has always had a bottleneck of the portion of
| the code written by the person with the least knowledge or worst
| priorities, but the ability to actually work around this as an
| end user has gotten harder and harder.
|
| The first hurdle is the technical skill required: there has
| always been closed source software, but these days the software
| is so much more complex, often even obfuscated, that the level of
| knowledge necessary to diagnose an issue and fix it has gone up
| significantly. It used to be that you could hold and entire
| program in your head and fix it by patching a couple bytes, but
| these days things have many, many libraries and you may have to
| do patching at much stranger boundaries ("function level but when
| the arguments are these values and the call stack is this"). And
| that's not to say anything of increasing codesigning/DRM schemes
| that raise the bar for this kind of thing anyways.
|
| The other thing I've been seeing is that the separation between
| the perceived skills of software authors and software users has
| increased, which I think has discouraged people from trying to
| make sense of the systems they use. Software authors are
| generally large, well funded teams, and together they can make
| "formidable" programs, which leads many to forget that code is
| written by individuals who write bugs like individuals. So even
| when you put in the work to find the bug there will be people who
| refuse to believe you know what you are doing on account of "how
| could you possibly know better than $GIANT_CORPORATION".
|
| If you're looking for ways to improve this, as a user you should
| strive to understand how the things you use work and how you
| might respond to it not working-in my experience this is a
| perpetually undervalued skill. As a software author you should
| look to make your software introspectable, be that providing
| debug symbols or instructions on how users can diagnose issues.
| And from both sides, more debugging tools and stories like this
| one :)
| EdSchouten wrote:
| Both of these issues are good candidates for
| https://accidentallyquadratic.tumblr.com
| JyB wrote:
| I am absolutely shocked about this finding. The amount of money
| on microtransactions Rockstar lost because of this single issue
| must be gigantic. The amount of people that got turned off by the
| loading times over the years is massive. It's mind boggling.
| w_for_wumbo wrote:
| How many bets they'll leave it like this on the current version
| and fix it for the PS5 version, to show how dramatic of a change
| it is between consoles?
| ksec wrote:
| Even after cutting loading by 70% it still take a minute? I
| haven't played any AAA titles for a long time. But even 30s is
| way too long. Especially I used to play with HDD. Considering
| modern SSD can be 30x Faster in Seq Read and Random Read up to
| 200x.
|
| Is 1 min loading time even normal? Why did it take so long? I
| never played GTA Online so could someone explain?
| awinter-py wrote:
| woo open source
| wun0ne wrote:
| Incredible work. Hopefully R* acknowledge this and compensate you
| in some way. I won't be holding my breath.
|
| Maybe set up a donation page? I'd be more than happy to send some
| beer money your way for your time spent on this!
| raggi wrote:
| Spicy hot take: the root cause here is the awful c++ library
| ecosystem.
| Aissen wrote:
| It would make a great addition to the Accidentally Quadratic
| blog: https://accidentallyquadratic.tumblr.com/ (haven't been
| updated in ~2 years, but maybe the author is around).
| capableweb wrote:
| > the problems shouldn't take more than a day for a single dev to
| solve
|
| It bothers me that so many of us developers are annoyed by
| managers saying stupid stuff like "shouldn't be too hard" about
| things they don't understand, but then other developers walk into
| the same trap.
|
| Yes, it looks simple at the surface. It most likely isn't. The
| problem is not that it looks simple, the problem is that you
| assume it is simple, probably because you don't have the full
| context.
| asadlionpk wrote:
| The person posted a PoC that works. Surely they have context
| now?
| capableweb wrote:
| That's very true, and the PoC works for that person. It's not
| that easy in real-life development though, you can't just
| switch out the JSON parser and call it a day. Lots of testing
| has to be done and you have to go through all the previous
| changes to make sure you're not missing what the previous
| maintainers did to fix some one-in-a-million issues.
|
| I'm not saying it's impossible for this to be as easy as the
| author claims it to be. I'm just saying that it might not
| actually be that easy in reality, if you're on the inside.
| segmondy wrote:
| Let's say that JSON parser is used in so many other places
| and we are afraid of regression. Guess what? We can make a
| new one. Let's call it newImprovedJsonParser(). Use that
| only at this spot, and you have fixed the issue without
| breaking other parts. You can then replace the other's once
| you vet that they are compatible.
| jhncls wrote:
| You seem to be hinting to Chesterton's fence [0].
|
| [0]: https://fs.blog/2020/03/chestertons-fence/
| resonantjacket5 wrote:
| Accept in this case I don't really see why you'd want a
| naive o(n^2) parser when it isn't needed. Well besides
| for slow loading times.
| asadlionpk wrote:
| Thanks for sharing this, didn't have a name for this
| scenario.
| asadlionpk wrote:
| But you do understand the ROI of this relatively simple
| fix?
| toast0 wrote:
| OK, don't switch out the JSON parser. Just patch in the
| caching strlen before parsing the json, and set it back to
| normal strlen after; and fiddle with the hashmap stuff.
| stevemk14ebr wrote:
| it _is_ simple, as evident by the article if you read it in
| full
| capableweb wrote:
| I _did_ read it. Did I miss the part where Rockstar confessed
| it was a really easy issue to fix and they apologised or
| what's your point?
| stevemk14ebr wrote:
| 'Yes, it looks simple at the surface. It most likely isn't'
| my point is the issue is simple, as the article proves.
| blt wrote:
| I agree "shouldn't be too hard" should be avoided, but in this
| case the developers should fix it even if it _is_ hard.
| cma wrote:
| Back of the envelope (and if the same problem applies to
| consoles in addition to PC), the fix would save 30+ human
| waking lifetimes per year.
| bspammer wrote:
| I hope you at least agree this isn't a bug that should go
| unfixed for _7 years_.
| trollied wrote:
| Wow. I always assumed that profiling would be part of the pre-
| release test processes for AAA games...
| gbl08ma wrote:
| I had heard about this giant JSON from friends in the GTA V
| modding community. OP's idea of what it is used for is right.
| My guess is that this JSON was quite smaller when the game
| released and has been increasing in size as they add more and
| more items to sell in-game. Additionally, I speculate that most
| of the people with the knowledge to do this sort of profiling
| moved on to work on other Rockstar titles, and the "secondary
| team(s)" maintaining GTA Online throughout most of its lifespan
| either didn't notice the problem, since it's something that has
| become worse slowly over the years, or don't have enough
| bandwidth to focus on it and fix it.
|
| It's also possible they are very aware of it and are saving up
| this improvement for the next iteration of GTA Online, running
| on a newer version of their game engine :)
| Negitivefrags wrote:
| When it was released the game didn't have all the
| microtransactions so it probably took no time at all to process
| the JSON even with this issue.
|
| Then over time they slowly add data to the JSON and then this
| O(n^2) stuff starts to creep up and up, but the farther away
| from release you are, the less likely that the kind of
| engineers to who do optimisation and paying any attention.
|
| They are all off working on the next game.
| paulryanrogers wrote:
| Could it be that MMO players are just more accustomed to long
| load times? (Lack of loading patience is one of the reasons I
| don't play SWOTOR.)
| josephg wrote:
| When I used to play world of Warcraft, it never took more
| than 30 seconds or so to load. It got much faster over the
| years - when I was playing a few years ago it was more like 5
| seconds from character selection to being in the world.
|
| Nothing like the 6 minutes people are talking about for GTA.
| That's ridiculous.
| jimbob45 wrote:
| Same. I wonder if the dev didn't bother to fix it because they
| assumed profiling would identify it as a non-issue.
| bombcar wrote:
| I wonder if the list was much shorter at release and wasn't
| super slow on Developement systems.
| yread wrote:
| Could be that at release the JSON was just 200kb with 1000
| entries or something and this quadratic "algorithm" wasn't the
| slowest part of the process
| IgorPartola wrote:
| More importantly how do you release a game that takes 6 minutes
| to load? This is why mobile gaming has the advantage. In those
| 6 minutes I could have played a round of a game that's quite
| satisfying and put it down already. This just seems sloppy.
| Aerroon wrote:
| I've actually opened a game with long loading times and alt
| tabbed out, because I knew it would take a while. I booted up
| another game to play a little bit until the first game
| finishes loading. 3 hours later I was done playing and
| realized that I was supposed to play game #1.
| bluedino wrote:
| It was only one minute at first. As the game had content
| added, it got longer and longer and nobody cared.
| superjan wrote:
| It's quite possible that there were way less items in the
| store on launch, or during testing. Then it could easily be
| overlooked. Of course no excuse to not fix it later.
| IgorPartola wrote:
| OK but how come even without the store it took more than
| 20-30 seconds? This was a beefy computer 7 years ago.
| faebi wrote:
| Wow, many people argue how optimized GTA was and then this. I
| wonder how much money they lost because of this. I often stopped
| playing because it just took too long to load.
| formerly_proven wrote:
| GTA V (the single player game) is quite well optimized and
| needs a frame rate limiter on most newer systems because it
| will run at over ~180 fps, at which point the engine starts to
| barf all over itself.
|
| GTA Online is a huge, enormously buggy and slow mess that will
| generally struggle to run at 80 fps on a top-of-the-line 2020
| system (think 10900K at over 5 GHz with a 3090) and will almost
| never cross the 120 fps threshold no matter how fast your
| system is and how low the settings are.
| ecf wrote:
| > at which point the engine starts to barf all over itself.
|
| I'm really confused as to why games are determining anything
| whatsoever based on the refresh rate of the screen.
|
| Skyrim has this same problem and not being able to play over
| 60fps is the reason I haven't touched the game in years.
| gridspy wrote:
| Because if you run the simulation at a different frame-rate
| from the rendering it is a huge amount more work. Suddenly
| you have to start marshaling data through intermediate data
| structures and apply interpolation to everything.
|
| If you then try to run the simulation in parallel with the
| rendering (rather than between some frames) it is even more
| work, since inter-thread communication is hard.
|
| This stuff might seem easy for very good programmers,
| however on a game you want to hire a wide range of
| programmer skill and "game-play programmers" tend to be
| weaker on the pure programming front (their skills lie
| elsewhere)
| megameter wrote:
| It's a coding strategy intended to optimize around console
| refresh targets first and then "do whatever" for PC build.
|
| Generally, a AAA console game will target 30hz or 60hz.
| Therefore the timing loop is built to serve updates at a
| steady pace of 30 or 60, with limiting if it goes faster.
| Many game engines also interpolate animations separately
| from the rest of the gameplay, allowing them to float at
| different refresh rates. Many game engines further will
| also decouple AI routine tick rates from physics to spread
| out the load. Many game engines now also interleave updates
| and kick off the next frame before the first is complete,
| using clever buffering and multithreaded job code. All
| told, timing in games is one of those hazard zones where
| the dependencies are both numerous and invisible.
|
| When you bring this piece of intricate engineering over to
| PC and crank up the numbers, you hit edge cases. Things
| break. It's usually possible to rework the design to get
| better framerate independence, but doing do would be
| invasive - you'd be changing assumptions that are now baked
| into the content. It isn't just fixing one routine.
| eznzt wrote:
| The 3D engine is highly optimised.
| SteveNuts wrote:
| They must have hired a Solarwinds intern to write the loading
| screen.
| alborzb wrote:
| It always fascinates me how sometimes people defend things just
| because they're a fan, even if the particular aspect they're
| defending doesn't make sense!
|
| I've seen this happen with some other games which are not the
| best optimised for PCs, but the fans will still defend the
| developers, just because they like the brand
| comboy wrote:
| They did some good stuff:
| http://www.adriancourreges.com/blog/2015/11/02/gta-v-
| graphic...
|
| And just thinking about the size of the game and popularity,
| to make it work at all, requires some skill. Which makes the
| OP even more unbelievable.
| ip26 wrote:
| It's part of social validation. You inherently want other
| people to like the things you like, because it validates your
| choices. This in turn means you'll defend things you like.
|
| Even "rebels", who supposedly relish having fringe tastes,
| want other rebels to approve of their tastes.
|
| The more strongly you stake your identity as "fan of X", the
| more social disapproval of X hurts.
| c7DJTLrn wrote:
| Yep, sometimes I feel like having a drive around, and then I
| remember how long it takes to load and play something else
| instead. If you end up in a lobby with undesirables and are
| forced to switch, you've got another long wait.
| connicpu wrote:
| I'm willing to bet the developers who wrote the in game store
| are not the same developers who optimized the rendering
| pipeline
| blibble wrote:
| I tried GTA online once and indeed got fed up of the load times
| jeroenhd wrote:
| GTA, at least the core gameplay and the single player mode, is
| quite well optimised. The game ran well even on the cheaper
| side of gaming PC hardware.
|
| This... this is GTA online. It's a cash cow designed to suck
| cash out of your pocket. Ads for things you can spend your
| money on are shown while "connecting", so if this delay wasn't
| introduced intentionally, it sure isn't a high priority fix.
| The code isn't part of the optimised, streamlined, interactive
| part of the game, it's part of the menu and loader system.
|
| Most of these online games/services have so-called "whales"
| that contribute most if not all of the income the platform
| makes. If these whales are willing to spend the wads of cash
| they throw at the platform, they won't even care for another
| five minutes of ads. The amounts of cash some of these people
| spend is obscene; the millions Take Two profit from GTA every
| year are generally generated by only a tiny (usually a single
| number percentage) of the total player base.
|
| In the end, I doubt they've lost much money on this. They
| might've even made some from the extra ads.
| jsheard wrote:
| > GTA, at least the core gameplay and the single player mode,
| is quite well optimised. The game ran well even on the
| cheaper side of gaming PC hardware.
|
| It's easy to forget that GTA5 was originally a _360 /PS3_
| game, getting a game of that scope running at all on a system
| with just 256MB of RAM and VRAM was an incredible
| achievement.
|
| The A-Team developers who made that happen were probably
| moved over to Red Dead Redemption 2 though, with GTA5s long
| tail being handled by the B-Team.
| psyklic wrote:
| I wouldn't be surprised if the long wait increases profits --
| as you wait, Rockstar shows you ads for on-sale items.
| ed25519FUUU wrote:
| I love reading about this type of hacking/debugging, where they
| disassemble the binary and patch it.
|
| Can anyone recommend a youtube video where I can _watch_ (not
| necessarily learn) people doing this sort of work? I 'm in the
| mood for some vicarious hacking :-)
| notriv wrote:
| I highly recommend this forum :)
|
| https://guidedhacking.com/
| voltagex_ wrote:
| liveoverflow and stacksmashing channels on YouTube.
| dekerta wrote:
| This article is great. As a developer, I've always been curious
| about what the heck GTAO was doing that could take so long to
| load
| sim_card_map wrote:
| Unbelievable.
|
| Such silly mistakes causing such a huge delay for millions of
| gamers over 6 years.
| kevinventullo wrote:
| I gave up on playing GTA:O because everything took so long to
| load, having never spent a dime. I have to imagine there is so
| much lost revenue because of this bug; I hate to be harsh but it
| is truly an embarrassment that someone external to R* debugged
| and fixed this before they did (given they had 6 years!).
| ThomW wrote:
| Load times is absolutely the primary reason I quit playing.
| bouke wrote:
| Same here, being slow as molasses really spoiled the game for
| me. I wonder if the same software quality caused other
| unnecessary in-game loading times. I felt that time spent
| playing GTA:O was about 60% waiting in lobby/playing, and the
| remaining 40% actual play time. Same with RDR2 by the way.
| gridspy wrote:
| These problems come down to which programmers were allowed
| to remain on the team and how much they were allowed to
| change. So all similar performance problems will remain.
| josalhor wrote:
| I have a friend who works at Rockstar. I have forwarded the blog
| post to him.
| ohwaitnvm wrote:
| I haven't played GTAO in a couple years, but I vaguely remember
| using the suspend process option of windows task manager to jump
| into online much faster.
|
| Googling gave me this result, which sounds about right for what I
| remember. https://www.gfinityesports.com/grand-theft-auto/gta-
| online-h...
|
| I'm not certain it's still a valid workaround, and it's not
| nearly as sophisticated as the OP method, but at least everyone
| can do it :)
| dEnigma wrote:
| I had to use this trick to kick GTA Online back into action
| when it got stuck forever on entering buildings (mid-2020).
| Didn't know it could be used to cut the load times in general.
| pityJuke wrote:
| I remember doing this to create a private online session (i.e.
| no other users).
| powerfulclick wrote:
| This is really cool - how did you develop the background
| knowledge to solve this? I'm trying to learn more about low-level
| stuff and I would have no idea how to approach solving a problem
| like this
| spuz wrote:
| I'd recommend searching HN for threads about learning reverse
| engineering. Here are a few that I've found:
|
| Reverse Engineering Course:
| https://news.ycombinator.com/item?id=22061842
|
| Reverse Engineering For Beginners:
| https://news.ycombinator.com/item?id=21640669
|
| Introduction to reverse engineering for beginners:
| https://news.ycombinator.com/item?id=16104958
| the_only_law wrote:
| I love it but I don't have the focus or patience to ever do
| anything more than basic analysis.
|
| A lot of the reverse engineers I know seemingly have deep
| platform knowledge and can do things like cite Win32 docs
| from memory.
| mmastrac wrote:
| I was a reverse engineer for years and never was able to do
| anything like quoting docs. I'd be constantly googling or
| using reference material. The only real attribute I'd
| suggest is tenacity.
| tyingq wrote:
| _" They're parsing JSON. A whopping 10 megabytes worth of JSON
| with some 63k item entries."_
|
| Ahh. Modern software rocks.
| LukvonStrom wrote:
| why not embed node.js to do this efficiently :D
| ed25519FUUU wrote:
| Parsing 63k items in a 10 MB json string is pretty much a
| breeze on any modern system, including raspberry pi. I wouldn't
| even consider json as an anti-pattern with storing that much
| data if it's going over the wire (compressed with gzip).
|
| Down a little in the article and you'll see one of the real
| issues:
|
| > _But before it's stored? It checks the entire array, one by
| one, comparing the hash of the item to see if it's in the list
| or not. With ~63k entries that's (n^2+n) /2 = (63000^2+63000)/2
| = 1984531500 checks if my math is right. Most of them useless._
| tyingq wrote:
| The JSON patch took out more of the elapsed time. Granted, it
| was a terrible parser. But I still think JSON is a poor
| choice here. 63k x X checks for colons, balanced
| quotes/braces and so on just isn't needed.
| Time with only duplication check patch: 4m 30s Time
| with only JSON parser patch: 2m 50s
| bombcar wrote:
| At least parse it into SQLite. Once.
| tyingq wrote:
| I think just using a length encoded serialization format
| would have made this work reasonably fast.
| hobofan wrote:
| Or just any properly implemented JSON parser. That's a
| laughable small amount of JSON, which should easily be
| parsed in milliseconds.
| brianberns wrote:
| They probably add more entries over time (and maybe
| update/delete old ones), so you'd have to be careful about
| keeping the local DB in sync.
| bombcar wrote:
| So just have the client download the entire DB each time.
| Can't be that many megabytes.
| high_byte wrote:
| this is amazing. good work! unbelievable
| forrestthewoods wrote:
| Super impressive. I am generally not impressed with community
| hacks because, as every comment thread discusses, they are
| generally hacks that do not support all players and thus can't be
| shipped.
|
| But this seems pretty damn clean. And it's egregious that no one
| at R* took a week to investigate and fix "correctly".
| sailfast wrote:
| This is such an awesome article, and I love it. Thanks to the
| author for the great digging and explanation.
| dvdbloc wrote:
| I bet if the devs did more leetcode this would have never
| happened /s
| z92 wrote:
| Also note that the way he fixed it, strlen only caches the last
| call and returns quickly on an immediate second call with the
| same string.
|
| Another reason why C styled null terminated strings suck. Use a
| class or structure and store both the string pointer and its
| length.
|
| I have seen other programs where strlen was gobbling up 95% of
| execution time.
| ip26 wrote:
| Could this be worked into a compiler/stdlib from the back-end?
| Could a compiler/stdlib quietly treat all strings as a struct
| of {length,string} and redefine strlen to just fetch the length
| field? Perhaps setting a hook to transparently update "length"
| when "string" is updated is not trivial.
|
| Edit: hah, I'm decades late to the party, here we go:
|
| _Most modern libraries replace C strings with a structure
| containing a 32-bit or larger length value (far more than were
| ever considered for length-prefixed strings), and often add
| another pointer, a reference count, and even a NUL to speed up
| conversion back to a C string. Memory is far larger now, such
| that if the addition of 3 (or 16, or more) bytes to each string
| is a real problem the software will have to be dealing with so
| many small strings that some other storage method will save
| even more memory (for instance there may be so many duplicates
| that a hash table will use less memory). Examples include the
| C++ Standard Template Library std::string..._
|
| https://en.wikipedia.org/wiki/Null-terminated_string
| toast0 wrote:
| I don't think you could do it transparently, because it's
| expected to pass the tail of a character array by doing
| &s[100] or s + 100, etc. I don't think that would be easy to
| catch all of those and turn them into a string fragment
| reference.
|
| From c++ class, std::string was easy enough to use
| everywhere, and just foo.c_str() when you needed to send it
| to a C library. But that may drags in a lot of assumptions
| about memory allocation and what not. Clearly, we don't want
| to allocate when taking 6 minutes to parse 10 megs of JSON!
| :)
| ww520 wrote:
| Yep, O(n^2) has the problem that no matter how fast you upgrade
| your hardware it would still lag.
|
| Another pet peeve of mine is Civ 6's loading time for a saved
| game is atrocious. I'm sure there's a O(n^2) loop in there
| somewhere.
| thitcanh wrote:
| This is why people should use commonly-available packages instead
| of rolling their own version of whatever dumb algorithm they
| think they can write. This happens all the time. Bugs have been
| fixed by others, but everyone is too smart to use someone else's
| code.
| xyst wrote:
| seems more like a bad copy and paste situation to me. it's sort
| of what you get when you pay for the lowest bid for
| contractors.
| mixologic wrote:
| Sometimes those commonly used packages end up being whatever
| dumb algorithm the author came up with, and nobody spends the
| time to verify if the package is worth it's popularity.
| breakingcups wrote:
| It is absolutely unbelievable (and unforgivable) that a cash cow
| such as GTA V has a problem like this present for over 6 years
| and it turns out to be something so absolutely simple.
|
| I do not agree with the sibling comment saying that this problem
| only looks simple and that we are missing context.
|
| This online gamemode alone made $1 billion in 2017 alone.
|
| Tweaking two functions to go from a load time of 6 minutes to
| less than two minutes is something any developer worth their salt
| should be able to do in a codebase like this equipped with a good
| profiler.
|
| Instead, someone with no source code managed to do this to an
| obfuscated executable loaded with anti-cheat measures.
|
| The fact that this problem is _caused_ by Rockstar 's excessive
| microtransaction policy (the 10MB of JSON causing this bottleneck
| are all available microtransaction items) is the cherry on top.
|
| (And yes, I _might_ also still be salty because their parent
| company unjustly DMCA 'd re3 (https://github.com/GTAmodding/re3),
| the reverse engineered version of GTA III and Vice City. A
| twenty-year-old game. Which wasn't even playable without
| purchasing the original game.)
| kuroguro wrote:
| Small nitpick: I believe these are items/prices for the in-game
| currency, not micro-transactions.
|
| You can buy in-game currency for real world money tho:
| https://gta.fandom.com/wiki/Cash_Cards
|
| Not 100% sure, never bought anything.
| imglorp wrote:
| > This online gamemode alone made $1 billion in 2017 alone.
|
| There's the answer right there. They figure it's making $1B/yr,
| leave it alone. Maintenance? That cuts into the billion.
| Everyone moved onto the next project.
| tyingq wrote:
| Or they fix it, see that their "in game time" average drops,
| and then back it out...
| themeiguoren wrote:
| I would not at all be surprised if the long load time made
| for a sunk cost that kept people playing for longer
| sessions rather than picking it up for less than half an
| hour at a time.
| Zak wrote:
| I played through GTA V, enjoyed it, and tried out the online
| mode afterward.
|
| I've logged in exactly twice. Load times like that may be worth
| it to a hardcore gamer, but I have no patience for it. There's
| no shortage of entertainment available to someone with a PC, a
| reasonable internet connection, and a modicum of disposable
| income. Waste my time and I'll go elsewhere for my
| entertainment.
| Thaxll wrote:
| It was probably fast 10years ago when the store had couple of
| items, the dev back then never thought that it would grow to
| 60k items. Classic programming right there.
|
| As for profiling, Windows Performance Toolkit is the best
| available no?
| trulyme wrote:
| Meh. It's ok to assume low number of items and code
| accordingly. What is not ok is for the company to ignore such
| a problem for years, instead if detecting and fixing it.
| masklinn wrote:
| > The fact that this problem is caused by Rockstar's excessive
| microtransaction policy (the 10MB of JSON causing this
| bottleneck are all available microtransaction items) is the
| cherry on top.
|
| For what it's worth, 10MB of JSON is not much. Duplicating the
| example entry from the article 63000 times (replacing `key` by
| a uuid4 for unicity) yields 11.5MB JSON.
|
| Deserialising that JSON then inserting each entry in a dict
| (indexed by key) takes 450ms in Python.
|
| But as Bruce Dawson oft notes, quadratic behaviour is the sweet
| spot because it's "fast enough to go into production, and slow
| enough to fall over once it gets there". Here odds are there
| were only dozens or hundreds of items during dev so nobody
| noticed it would become slow as balls beyond a few thousand
| items.
|
| Plus load times are usually the one thing you start ignoring
| early on, just start the session, go take a coffee or a piss,
| and by the time you're back it's loaded. Especially after QA
| has notified of slow load times half a dozen times, the devs
| (with fast machines and possibly smaller development dataset)
| go "works fine", and QA just gives up.
| hanniabu wrote:
| That QA bit is too true, "it works for me!" _shrug_.
| ldng wrote:
| But is quadratic the real issue ? Isn't that a developer
| answer ?
|
| The best algorithm for small, medium or a large size are not
| the same and generally behave poorly in the other cases. And
| what is small? Medium? Large?
|
| The truth is that there is no one size fits all and
| assumptions _need_ to be reviewed periodically and adapted
| accordingly. And they never are... Ask a DBA.
| gridspy wrote:
| quadratic is a fancy way of saying "this code is super fast
| with no data, super slow once you have a decent amount"
|
| The problem is that when you double the amount of stuff in
| the JSON document, you quadruple (or more) the scanning
| penalty in both the string and the list.
|
| Why quadruple? Because you end up scanning a list which is
| twice as long. You have to scan that list twice as many
| times. 2x2 = 4. The larger list no longer fits in the fast
| (cache) memory, among other issues. The cache issue alone
| can add another 10x (or more!) penalty.
| gameswithgo wrote:
| What is complicated about it is that an online modern 3d game
| is huge, and there are 95,000 places where a dumb mistake could
| hurt performance a lot for some customers. You catch 94,999 of
| them and then "unforgiveable"
| Plutoberth wrote:
| If it was that way for a few months and then fixed... still
| pretty shoddy but sure. However, it has been that way for
| YEARS, and is one of the most common complaints among
| players. I wonder how much of the remaining loading time
| could actually be shaved off if someone with the source code
| took a crack at it.
| neatze wrote:
| I am not saying it is the case, nor understand details of
| solution in depth to comment on it, but in analogy, this reads
| to me, like an yelling at person who figure out how to solve
| rubix cube puzzle steps, because once steps are known solution
| is simple.
| burlesona wrote:
| No, other people have pointed this out, this should have been
| very easy to recognize as inefficient in the source code.
| More likely the code was written hastily and only tested
| against very small inputs, and then nobody ever actually
| tried to improve the famously long load times.
| moonchild wrote:
| > salty because their parent company unjustly DMCA'd re3
|
| Unjustly, but legally. The people you should be salty at are
| the lawmakers.
| bscphil wrote:
| Why can't I be salty at both? People have responsibility for
| their actions even if those actions are legal.
| breakingcups wrote:
| That still remains to be seen. A DMCA is not a court order.
| Anyone can file one and take a repository offline for two
| weeks.
| moonchild wrote:
| Yes but the code was clearly derived directly from a
| decompiled binary; not 'clean room' reverse engineering.
| Hence, illegal, regardless of whether a dmca takedown
| notice is filed.
| xyzzy_plugh wrote:
| That doesn't make it illegal, there are plenty of
| jurisdictions where non-clean-room reverse engineering is
| perfectly legal.
| nl wrote:
| Deriving something from a decompiled binary isn't illegal
| in itself.
| oh_sigh wrote:
| Maybe long load times are advantageous? Because it creates
| longer user sessions on average? If you devote 10 minutes to
| loading the game you will probably want to play for at least 30
| minutes.
| erik_seaberg wrote:
| Wouldn't the most impatient customers be more likely to pay
| for items, rather than earn them in game?
| formerly_proven wrote:
| Also: Rockstar being too cheap to implement anti-cheat on the
| _by far_ most successful online shooter on the planet.
|
| Also
|
| > I don't think there's any easier way out.
|
| lmfao
| buzzerbetrayed wrote:
| This may be obvious, but is GTAV the most successful online
| shooter on the planet? (Never played it)
| sickofparadox wrote:
| Its the second best selling game of all time[1], and
| because Minecraft doesn't have guns, I suppose that would
| qualify it as the most successful online shooter (even if I
| think another genre would be more applicable).
|
| [1]https://en.wikipedia.org/wiki/List_of_best-
| selling_video_gam...
| zionic wrote:
| I stopped playing GTAV online a few years back because of the
| crazy load times, not only that but you have to go through 6+
| minute load screens multiple times in many sessions.
|
| This oversight has cost them millions of dollars easy.
| wyaeld wrote:
| If it made over $1b in a year previously, and had such insane
| load times, its very plausible this bad coding has cost them
| north of another $1b.
|
| Probably ranks pretty highly up there in terms of damage to
| company financials, due to a lack of care.
| anoncake wrote:
| I find it absolutely believable that a for-profit company does
| not prioritize fixing a game that is already a cash cow anyway.
| 0xy wrote:
| Something I've noticed in highly successful companies is that
| problems never get fixed because the sound of the money printer
| from the core business is deafening.
|
| Our customer portal loads 2 versions of React, Angular,
| Knockout and jQuery on the same page? Doesn't matter, it's
| printing billions of dollars.
|
| Rockstar's money printer is so loud that they don't care about
| problems.
|
| Same thing for Valve, their money printer is so loud that they
| barely bother to make games anymore and let the Steam client
| languish for years (how did they let Discord/Twitch happen?).
| LukvonStrom wrote:
| Look at sharepoint. It's a total nightmare of a platform to
| develop on, but the people just adapted and built their
| businesses on it
| passivate wrote:
| What makes it a nightmare? Can you share a few issues
| you've run into?
| ip26 wrote:
| I suspect sharepoint is the platform version of excel &
| VBA. You & I might hate it, but it gets powerful
| capabilities into the hands of regular people.
|
| sharepoint is probably many non-engineer's very first
| exposure to actual version control with checkouts,
| checkins, version history, and merge.
| Nexxxeh wrote:
| >Same thing for Valve, their money printer is so loud that
| they barely bother to make games anymore and let the Steam
| client languish for years (how did they let Discord/Twitch
| happen?).
|
| Not sure that's a fair criticism.
|
| Alyx was widely praised. Artifact... Wasn't. I don't know
| about Dota Overlords. And that's just the last couple of
| years.
|
| They've also developed hardware like SteamLink, Steam
| Controller, some high-end VR gear...
|
| They develop a LOT. They just don't release a while lot.
|
| I agree there should be a lot more work and effort in the
| client. And they constantly fuck up handling their esports.
|
| But I don't think "barely bother to make games anymore" isn't
| one of them.
| umanwizard wrote:
| "worth their salt" is doing a lot of work here. No true
| Scotsman fallacy?
|
| I think you might be surprised by how few programmers even know
| what a profiler is, let alone how to run one.
| AnonsLadder wrote:
| Does anyone have a link to a copy of re3? Iirc, there was a
| gitrepo that kept a copy of all DMCA'd repos
| xyst wrote:
| is it really unbelievable? companies this big tend to
| prioritize hiring a shit ton of middlemen (VPs, project
| managers, developer managers, offshore managers) in order to
| avoid paying out for talent to build and constantly maintain
| the project. I guess paying a shit ton of money to 1 person to
| manage 10+ poorly paid contractors works out for them,
| accounting wise.
|
| If one really examined the accounting for GTAO, I would bet
| that most of the billions of dollars that were earned in micro
| transactions went to marketing, product research, and to middle
| management in the form of bonuses.
| spuz wrote:
| Even if you view this as a business decision rather than a
| technical one, any smart project manager would realise a 6
| minute loading time literally costs the company millions per
| year in lost revenue. (How many times have you felt like
| firing up GTA Online only to reconsider due to the agonising
| load time). I would guess this was simply a case of business
| folk failing to understand that such a technical issue could
| be so easily solved plus developers never being allowed the
| opportunity to understand and fix the issue in their spare
| time.
| somehnguy wrote:
| The insane loading times are literally the exact reason I
| haven't played in years. Every time I played I just ended
| up frustrated and got distracted doing something else while
| waiting, so I just quit playing altogether. I don't know
| how people stand the loading times.
| jonwinstanley wrote:
| Totally agree, I loved the story mode and never got into
| online due to the amount of time spent loading and
| finding interesting stuff to do.
| alien_ wrote:
| The people who observe the slow loading time already paid
| for the game, so I guess R* won't lose much revenue because
| of this nasty bug.
| bcrosby95 wrote:
| The game has microtransactions. Coincidentally, also a
| large reason why load times were so slow.
| [deleted]
| delecti wrote:
| The online mode has microtransactions. People not playing
| anymore aren't paying for microtransactions either.
| [deleted]
| [deleted]
| AuryGlenz wrote:
| I didn't buy the game to play with my friends because I
| heard of how terrible the loading situation was.
| IshKebab wrote:
| It's kind of hard to believe. GTA5's online mode is their
| cash cow, and _6 minute load times_ are common?! It 's kind
| of amazing people even play it with those load times. It's
| such a simple problem that one dev could have found and fixed
| it within a day.
| toast0 wrote:
| It's not at all hard to believe if you've been playing
| video games for a while.
|
| Everything is getting slower and slower, and nobody cares.
|
| When I played the Atari 2600, I had to wait for the TV to
| warm up, but otherwise there were no games with anything
| approaching load times (with 128 bytes of RAM in the
| console, who would know). The NES didn't have much in the
| way of load times either, but you did have to fiddle with
| the cartridge slot. SNES and Genesis didn't usually load
| (Air Buster being a noticeable exception). CD based systems
| sure did like to load, but that's somewhat understandable.
| In the mean time, more and longer boot screens. The Switch
| uses a cartridge (or system flash/SD cards), but it likes
| to load forever too.
|
| PC Gaming has had loading for longer, but it's been getting
| longer and longer.
|
| Some arcade games have lengthy loading sequences, but only
| when you turn them on while they read their entire storage
| into RAM so they can be fast for the rest of the time
| they're on (which in arcades is usually all day).
| yoz-y wrote:
| > Everything is getting slower and slower, and nobody
| cares.
|
| It really depends. The latest crop of games I've played
| (Doom Eternal, Cyberpunk) loads way faster than games
| from a few years back (aforementioned GTA-V, Shadow
| Warrior 2...).
|
| This is also on the same machine, so it's not the
| hardware that makes it faster.
| ubercow13 wrote:
| Shorter loading times were one of the main selling points
| of this console generation.
| xyzelement wrote:
| I am always amused by comments like this. You have no idea
| what development practices they follow (neither do I) but
| it's hilarious to read your tone.
|
| GTA has achieved tremendous success both as an entertaining
| game and as a business. It's enjoyed by millions of people
| and generates billions in revenue. As per this article, it
| has startup problems (which don't seem to actually really
| hurt the overall product but I agree sound annoying) but the
| bigger picture is: it's a huge success.
|
| So - Rockstar has nailed it. What exactly is your platform
| for analyzing/criticizing their processes or even having a
| shot of understanding what they are? What have you build that
| anyone uses? (not saying you haven't, but.. have you been
| involved with anything remotely close to that scale?)
|
| And if not, whence he high horse?
| brokencode wrote:
| Successful people and businesses can be wrong. You are not
| making a case for why those development practices are okay,
| but are simply appealing to authority.
|
| I and most other customers would argue that 6 minute
| loading times are atrocious, and if there is an easy fix
| like this, it makes me lose a lot of respect for the
| developer who doesn't fix it. It maybe would even make me
| avoid them in the future.
|
| A reputation is built over years, but can be lost pretty
| much instantly. Companies have to continue serving their
| customers to enjoy ongoing success.
| nikanj wrote:
| The old maxim of "Premature optimization is the root of all
| evil" has over time evolved to "If you care one iota about
| performance, you are not a good programmer".
| [deleted]
| vendiddy wrote:
| And fwiw, the full quote is:
|
| We should forget about small efficiencies, say about 97% of
| the time: premature optimization is the root of all evil.
|
| Yet we should not pass up our opportunities in that critical
| 3%.
| blowski wrote:
| Also, it's not "never optimise". It's "only optimise once
| you've identified a bottleneck". I guess in a profit-making
| business you only care about bottlenecks that are costing
| money. Perhaps this one isn't costing money.
| skzo wrote:
| ding ding ding ding
| [deleted]
| tamrix wrote:
| Well this sprint we didn't release any new features but we
| reduced the load.... Dammit hackernews!
| luckystarr wrote:
| Well, that's embarrassing. I can't even imagine the level of
| shame I would feel if I had written the offending code.
|
| But, you know, premature optimization yadda yadda.
| whatever1 wrote:
| Probably this was written under a very strict release deadline
| and it worked ok at the time (less items for
| microtransactions). The problem lies with the management that
| never picked up the issue once it became a big problem. Pretty
| sure that any developer in R* is capable of optimizing this
| parser.
| mekkkkkk wrote:
| As others have pointed out, this is a good illustration of why
| you need accurate data during development. I bet that the data
| set used during development was dirty with duplicates and way too
| small. It was faster and more convenient to just code defensive
| garbage than to be the annoying one nagging to another team about
| the state of data. So this was written, probably along with a
| TODO comment, and then forgotten about, and ultimately shipped.
| I've done this same thing. Not with any real consequences, but
| still. It's what happens when time is of the essence.
|
| How it remained undetected for so long is really weird though.
| Surely they must've had a massive amount of complaints about the
| loading times. I completely stopped playing because of them. How
| could they not have investigated the root cause, atleast once, in
| six years?
| comboy wrote:
| Holy cow, I'm a very casual gamer, I was excited about the game
| but when it came out I decided I don't want to wait that long and
| I'll wait until they sort it out. 2 years later it still sucked.
| So I abandoned it. But.. this... ?! This is unbelievable. I'm
| certain that many people left this game because of the waiting
| time. Then there are man-years wasted (in a way different than
| desired).
|
| Parsing JSON?! I thought it was some network game logic finding
| session magic. If this is true that's the biggest WTF I saw in
| the last few years and we've just finished 2020.
|
| Stunning work just having binary at hand. But how could R* not do
| this? GTAV is so full of great engineering. But if it was a CPU
| bottleneck then who works there that wouldn't just be irked to
| try to nail it? I mean it seems like a natural thing to try to
| understand what's going on inside when time is much higher than
| expected even in the case where performance is not crucial. It
| was crucial here. Almost directly translates to profits.
| Unbelievable.
| dan-robertson wrote:
| I don't think the lesson here is "be careful when parsing json"
| so much as it's "stop writing quadratic code." The json
| quadratic algorithm was subtle. I think most people's mental
| model of sscanf is that it would be linear in the number of
| bytes it scans, not that it would be linear in the length of
| the input. With smaller test data this may have been harder to
| catch. The linear search was also an example of bad quadratic
| code that works fine for small inputs.
|
| Some useful lessons might be:
|
| - try to make test more like prod.
|
| - actually measure performance and try to improve it
|
| - it's very easy to write accidentally quadratic code and the
| canonical example is this sort of triangular computation where
| you do some linear amount of work processing all the finished
| or remaining items on each item you process.
|
| As I read the article, my guess was that it was some terrible
| synchronisation bug (eg download a bit of data -> hand off to
| two sub tasks in parallel -> each tries to take out the same
| lock on something (eg some shared data or worse, a hash bucket
| but your hash function is really bad so collisions are
| frequent) -> one process takes a while doing something, the
| other doesn't take long but more data can't be downloaded until
| it's done -> the slow process consistently wins the race on
| some machines -> downloads get blocked and only 1 cpu is being
| used)
| petters wrote:
| Is there some reason sscanf can not be implemented without
| calling strlen?
| ddulaney wrote:
| It could be, and the article acknowledges that possibility.
| For example, a cursory check of the musl sscanf [0]
| suggests that it does not (though I may have missed
| something). However, whichever implementation Rockstar used
| apparently does.
|
| [0]: https://git.musl-
| libc.org/cgit/musl/tree/src/stdio/vfscanf.c
| Nitramp wrote:
| - do not implement your own JSON parser (I mean, really?).
|
| - if you do write a parser, do not use scanf (which is
| complex and subtle) for parsing, write a plain loop that
| dispatches on characters in a switch. But really, don't.
| mdoms wrote:
| Me and my twice-a-week gaming group enjoyed GTA V but abandoned
| it years ago simply because of the load times. We have 2x short
| slots (90-120 minutes) each week to play and don't want to
| waste them in loading screens.
|
| We all would have picked this game back up in a second if the
| load times were reduced. Although I must say even with the same
| results as this author, 2 minutes is still too long. But I'll
| bet that, given the source code, there are other opportunities
| to improve.
| sova wrote:
| Really great work. Here's to hoping someone at Rockstar notices.
| You didn't even need access to the original sourcecode to debug
| it! Quite impressive my friend. Thanks for the great write-up!
| 4cao wrote:
| Excellent investigation, and an elegant solution.
|
| There's a "but" though: you might end up getting banned from GTA
| Online altogether if this DLL injection is detected by some anti-
| cheat routine. The developers should really fix it on their end.
| kuroguro wrote:
| Yeah, there's some disclaimers in the PoC repo. Definetly use
| at your own risk.
| AJRF wrote:
| There's one thing that made me curious. The author said the load
| times when the game came out were awful as well.
|
| So has that JSON always been 10mb or has it grown a lot over
| time?
|
| Has the load time crept up since launch or has it gone down?
| oatz wrote:
| Love performance issues like this, good read.
| captain_price7 wrote:
| I loved so much reading it, I was thinking that if someone were
| to write _fictional_ , sherlock holmes like fantasy story where
| our sherlock would take some (maybe fictional) widely used
| software at each episode, investigate it like this, and reveal
| some (fictional) grand bug in the end- I'd totally read it.
|
| Yeah I know it sounds stupid, but I suspect real Sherlock Holmes
| was inspired by a true story like this one too, and at least some
| contemporary detectives started to enjoy reading them.
| TamDenholm wrote:
| The Github link is dead, DMCA or incorrect link?
| sbierwagen wrote:
| This link? https://github.com/tostercx/GTAO_Booster_PoC
| Avery3R wrote:
| probably just forgot to be unprivated. The blog post was
| published today
| kuroguro wrote:
| yep, my bad
| flumpcakes wrote:
| Do you have any resources or advice to building up the
| skills required to do something like this?
|
| I like the idea of fixing up some older, unsupported, games
| I have issues with but would like to upskill using some
| structured resources first (books/videos/guided projects).
|
| Great work! This is the kind of wizard-magic I wish came
| naturally to me.
| bluedino wrote:
| Learn C and Assembly, learn how to debug your own
| programs.
|
| Then try debugging other programs.
| kuroguro wrote:
| ^ yeah that. I started with C so I always had decent low
| level understanding. I've found I pick things up faster
| if I'm doing stuff, not reading about it but YMMV.
|
| I probably I picked up most of my reverse engineering
| skills practicing on crackmes and reading/watching other
| people's solutions on crackmes.de (now dead) but you can
| try your luck in the successor https://crackmes.one/
| xyst wrote:
| not surprising - the game industry is absolutely notorious for
| cutting corners. didn't know they cut corners this much though.
|
| will r* fix it? maybe, especially since some person literally did
| half of the work for them. But given R* is a large company, this
| probably wont be fixed for a long time, and GTAO is probably
| being maintained by the lowest bid contractor group.
|
| They probably have all of their full time assets working on the
| next iteration of GTA.
| CTDOCodebases wrote:
| Imagine all the time that people are wasting if they are sitting
| there waiting for it to load. I wonder how many lifespans it adds
| up to?
| samhh wrote:
| I wonder about the environmental impact of CPU cores churning
| away uselessly on so many machines so many times. Probably
| enough to power a home for a few years?
| mattbillenstein wrote:
| Christ.
| holyknight wrote:
| Thank god. I always suspected that those loading times were cause
| by some retarded implementation detail. GTA5 is not that complex
| to justify that kind of loading times. Even the hardware has
| scaled massively since their launch and it doesn't even matter.
| josephg wrote:
| It so often is. This aspect of modern computing annoys me so
| much - modern computers, networks and cdns are so fast nowadays
| that most actions should be instant. My OS should boot
| basically instantly. Applications should launch instantly.
| Websites should load almost instantly. I'll give a pass for 3D
| modelling, AAA video games and maybe optimized release builds
| of code. But everything else should happen faster than I can
| blink.
|
| But most programs are somehow still really slow! And when you
| look into why, the reason is always something like this. The
| code was either written by juniors and never optimized because
| they don't know how, or written by mids at the limit of their
| intelligence. And full of enough complex abstractions that
| nobody on the team can reason holistically about how the whole
| program works. Then things get slow at a macro level because
| fixing it feels hard.
|
| Either way it's all avoidable. The only thing that makes your
| old computer feel sluggish for everyday computing is that
| programmers got faster computers, and then got lazy, and then
| shipped you crappy software.
| c7DJTLrn wrote:
| Sad that the source
| (https://github.com/tostercx/GTAO_Booster_PoC) has been taken
| down, I wanted to give this a try myself. I imagine a lot of GTA
| Online players would.
| [deleted]
| kuroguro wrote:
| Author here, my bad, forgot to set it to public ^^;;
| c7DJTLrn wrote:
| Many thanks!
| scrollaway wrote:
| You should have a way to contact you somewhere on your site.
| That blog post is an excellent stand in for a resume :)
| polote wrote:
| I have absolute respect for people who are able to do that kind
| of things. A few years back I've tried to play with reverse
| engineering but I never managed to get the skills.
| pid_0 wrote:
| Very cool analysis. Its unbelievable that this is still unfixed.
| I don't play GTA anymore mostly because of the load times for
| every single action.
| segmondy wrote:
| This is why I come to HN, I was going to skip this because I
| thought it was about video games, but really glad to have read
| it, and loved every line of the article.
|
| So much to get from this.
|
| Even if you don't have the source, you can make a change if you
| are annoyed enough.
|
| If you don't like something, and the source code is out there,
| really go contribute.
|
| Performance matters, know how to profile and if using an external
| dependency, then figure out their implementation details.
|
| Algorithms & Data structures matter, often I see devs talking
| about how it doesn't matter much but the difference between using
| a hashmap vs array is evident.
|
| Attentive code reviews matter, chances are they gave this to a
| junior dev/intern, and it worked with a small dataset and no one
| noticed.
| madeofpalk wrote:
| > Even if you don't have the source, you can make a change if
| you are annoyed enough.
|
| Well, until you get flagged by the anti cheat and get your
| account and motherboard banned...
| zionic wrote:
| Imagine getting banned for fixing their insane load times lol
| bluedino wrote:
| I always tell a story about an application we ran, it generated
| its own interface based on whatever was in inventory. Someone
| did something really stupid and duplicated each inventory item
| for each main unit we sold...so you had recursive mess.
| Assigning 100,000 items when previously it was 100-ish
|
| Anyway, everyone just rolled their eyes and blamed the fact
| that the app was written in Java.
|
| It ended up generating an XML file during that minute long
| startup....so we just saved the file to the network and loaded
| it on startup. If inventory changed, we'd re-generate the file
| once and be done with it.
| closeparen wrote:
| I think this is a perfect example of "algorithms and data
| structures emphasis is overblown." Real world performance
| problems don't look like LeetCode Hard, they look like doing
| obviously stupid, wasteful work in tight loops.
| nikanj wrote:
| And trying to optimize them gets you stink eye at code review
| time. Someone quotes Knuth, they replace your fast 200 lines
| with slow-as-molasses 10 lines and head to the bar.
| rictic wrote:
| True that it's rare that you need to pull out obscure
| algorithms or data structures, but in many projects you'll be
| _constantly_ constructing, composing, and walking data
| structures, and it only takes one or two places that are
| accidentally quadratic to make something that should take
| milliseconds take minutes.
|
| The mindset of constantly considering the big-O category of
| the code you're writing and reviewing pays off big. And
| neglecting it costs big as well.
| ilaksh wrote:
| Except that you need to test your software and if you see
| performance problems, profile them to identify the cause.
| It's not like you have one single chance to get everything
| right.
| johnfn wrote:
| Exactly - though to add a little nuance to your post, it's
| about having a million loops in a 10M line code base and
| exactly one of them is operating maximally slowly. So
| preventing the loop from entering the code base is tough -
| finding it is key.
| segmondy wrote:
| leetcode style thinking will allow you to spot obviously
| stupid wasteful work in tight loops.
| avar wrote:
| GTA:O shows advertisements for in-game purchases on the loading
| screen. How many advertisements you see is a function of how long
| the loading screen takes.
|
| Something tells me this "problem" was discovered long ago at
| Rockstar HQ, and quietly deemed not to be a problem worth
| solving.
| softwhale wrote:
| Could be, but I can imagine people giving up on GTA: Online
| altogether because it takes too much time to load.
| anonymousab wrote:
| > Could be, but I can imagine people giving up on GTA: Online
| altogether because it takes too much time to load.
|
| Luckily for them, churn is usually a different problem solved
| by a different part of the team/org with different priorities
| and insights.
| cmeacham98 wrote:
| I was going to say surely this has extremely diminishing or
| even negative return past 30-60 seconds, but then I remembered
| lots of people are willing to sit through 10 minutes of
| commercials to watch 20-30 minutes of TV. So I guess for the
| right type of customer it works?
___________________________________________________________________
(page generated 2021-02-28 23:00 UTC)