[HN Gopher] I cut GTA Online loading times (2021)
___________________________________________________________________
I cut GTA Online loading times (2021)
Author : ddtaylor
Score : 372 points
Date : 2022-06-09 14:22 UTC (8 hours ago)
(HTM) web link (nee.lv)
(TXT) w3m dump (nee.lv)
| thewebcount wrote:
| I've noticed similar stupid problems playing Portal 2 with a
| friend. Downloading the maps takes a really long time, but I've
| got a decent internet connection and so does my friend. It
| appears that when his machine is loading, my machine has no
| network activity whatsoever, and then once he's loaded the maps,
| then my machine will start loading them and his machine has no
| network activity.
|
| The game has a number of other weird issues like this. After
| you've played a map, it asks you to rate a thumbs up or thumbs
| down, but the hit areas don't match where the button is drawn,
| for example. Sometimes the mouse doesn't cause my character to
| turn. I can quit and restart the game and it's the same. But if I
| open a different game, quit it and then restart Portal, it works
| fine. WTF?
|
| I realize this is an old game so they aren't updating it (I have
| to boot into a 32-bit OS to even play it!) but it's a little
| depressing that such egregious issues went unnoticed for so long
| that we're now probably past the point of fixing them.
| victorclf wrote:
| I wonder if this code with poor time complexity was written by
| the same people who are always complaining about leetcode being
| useless.
| secant wrote:
| I feel like these problems are all around and often present in
| games. There's one that particularly infuriates me in
| Hearthstone, a Blizzard developed TCG, where when a game ends it
| causes the processor to furiously spin up and causes all kinds of
| anomalous behaviour like webcam cutting out, mic distortion and
| other generally laggy behaviour.
|
| I'm aware that there's probably a lot going on at the game's end
| but I'm still utterly convinced there is a coding bug causing
| this amount of processor spiking but I lack the skills of
| debugging Unity, which is the game's engine, to determine what
| actually is going on. I guess that's the issue, the ability of
| conducting the research the kind in the OP article is
| extraordinarily rare even among game devs so these problems
| persist (Hearthstone is just turning 8 years old now) without
| anyone with the deep knowledge to rectify it.
| flohofwoe wrote:
| Clickbait title ;)
|
| The problem wasn't strlen(), that just showed up at the bottom of
| the call stack when profiling.
|
| IIRC the actual problem was an exceptionally dumb JSON parser
| combined with parsing a pretty big JSON file (which probably also
| grew much bigger than the original developer working on that
| feature anticipated).
|
| Edit: the title has been edited in the meantime, originally it
| was something about strlen() being responsible for the slow
| loading time.
| gbraad wrote:
| The title is still not right as the poster is not the 'I' as in
| the title.
| ConceptJunkie wrote:
| It's funny though. Several years ago a coworker did some
| profiling on one of our server software components and found
| that it was spending up to 10% of the CPU time in strlen()...
| for very similar reasons.
| sfteus wrote:
| Had something similar happen with low-level components in a
| PHP application I inherited repeatedly calling trim() and/or
| str_pad() on values that had already been processed.
| Refactoring those out led to a 10-15% performance boost in
| our test suites.
| Sesse__ wrote:
| I've seen PHP applications use 30% of their CPU in the
| "realpath cache", which at the time was a super-slow (O(n2)
| in this case) home-grown hash table for caching the result of
| realpath(), which was called all the time because someone
| decided include_once() needed to run its paths through
| realpath() in case someone included a file multiple time
| through symlinks. And I guess they couldn't trust inode
| numbers or Windows didn't have them or something...
|
| I hope it's gone. I really do hope it's gone :-)
| Joker_vD wrote:
| The JSON parser was not "exceptionally dumb": it just used
| sscanf(s, "%d", ...) for parsing numbers. But most
| implementations of C's standard library (yes, including BSD and
| Linux's) implement sscanf() by converting the source string
| into a dummy FILE* object (which involves calling strlen() to
| set the stream's size) and calling fscanf() on it. Apparently
| most people who use sscanf() are unaware of this "lovely"
| implementation quirk and I can't really blame them.
| AlexanderTheGr8 wrote:
| Is C's std impl slow because of calling strlen() or due to
| converting into a dummy FILE* object or due to calling
| fscanf?
| unwind wrote:
| All of the above, I guess.
|
| It wants to reuse code in fscanf(), but for that it needs a
| FILE * interface. But to create that, it needs to implement
| EOF-handling, and to do that it needs to provide a size, so
| it has to call strlen().
|
| I had no idea. :|
| ddtaylor wrote:
| strlen() was a major component of the slowdown. Yes, the JSON
| file was large and strlen() was indirectly caused by sscanf()
| being used. But it's not accurate to say it's a clickbait title
| when a bulk of the slowdown was caused by constantly
| recalculating the length of the C string. The article shows in
| detail how he created a monkey-patched strlen() that cached the
| length to avoid recalculating it over and over which was
| attributed to the speedup.
|
| The speedup gains were not caused by reducing the size of the
| JSON file or doing any exceptionally better JSON parsing. The
| speedups were a direct result of simply not calling strlen()
| over and over in an O(N^2) fashion.
| flohofwoe wrote:
| All true, but replacing the strlen() with a caching version
| is just fixing the symptoms (of calling strlen too often...
| but the only realistic solution without the ability to
| recompile the game - so kudos to the author to come up with
| this hack).
|
| I bet the "official" patch by Rockstar simply dropped in a
| less embarassing JSON parser library :)
| ddtaylor wrote:
| This article was written before Rockstar released a patch
| and the 70% speedup is from his strlen() caching.
| flohofwoe wrote:
| Yes I'm aware, that article basically triggered the
| official patch. Would be interesting to know if the
| Rockstar patch provided a further speedup though.
| aardvarkr wrote:
| The author provided an update at the bottom. His hack cut
| the tone from 6m to 1m 30s. The official rockstar patch
| took it down to 1m 53s. It's MUCH better but still off
| the mark by a bit.
| kuroguro wrote:
| It was 1:50 vs 1:53 - probably just noise.
| whizzter wrote:
| You don't think that it's a bigger WTF that sscanf
| implementation calls strlen when parsing a number (when it
| iirc even ignores any non-digit characters for the actual
| number parsing so no reason it couldn't terminate that part
| w/o the extra call)?
| ddtaylor wrote:
| sscanf is essentially being used as a tokenizer for the
| JSON grammar, which means something like this:
| {"foo": "bar"}
|
| Is potentially turning into 5 different tokens:
| { "foo" : "bar" }
|
| After each token the "state" of sscanf is being discarded
| and each invocation of sscanf is independent. The API
| simply isn't descriptive enough to know that a string is
| immutable and hasn't been changed and there are certainly
| many valid use cases where one might be re-using the same
| char[] buffer with different contents.
| whizzter wrote:
| How much "state" would sscanf really need? From memory
| (and with an extra peek at the man page) each directive
| is in sequence of the "format" specifier is sequential
| (it's not nearly as flexible as a Regexp) so you really
| only need:
|
| A: a processing state (int/enum). B: a current fmt str
| pointer. C: va_list struct to keep track of destinations.
| D: a source string pointer.
|
| Now if what I saw mentioned in another comment is true
| that there is some insane conversion to a FILE* stream to
| do this, then yes I can see why it happens but it's not
| really NEEDED to implement the function as specified.
| tialaramex wrote:
| Even with the cache, calculating strlen() just _once_ was
| already a waste.
|
| The JSON to be parsed was read in from somewhere, either
| a file or received over the network. When that happened
| the machine _knew_ how long it was.
|
| One of the few places where C++ is sometimes better about
| things than Rust is providing information gathered which
| is ancillary to the direct purpose, in case you wanted
| it, so that you needn't then ask for the same information
| a moment later. In Rust this information may have a
| difficult to ascertain lifespan, and so it's dangerous,
| but in C++ they figure the out-of-date nature of the
| report is your problem. Sometimes to your detriment of
| course, but often you could really have used that and
| must now go measure it yourself because the thing you
| just called _knew_ it but didn 't tell you.
|
| Here though it wouldn't matter, Rust's array types, and
| both the owned mutable String and reference &str types
| know how long they are, so you are never flailing about
| counting even once. You also would just serde_json this
| data in Rust because serde is practically part of the
| Rust Standard Library, whereas there's nothing quite so
| ubiquitous in C or C++.
| dusted wrote:
| First of all, I find it an impressive skill to be able to do this
| on a binary blob.
|
| Secondly (optimistic version), it really shows that they
| (management) didn't care, that nobody ever made a ticket to fix
| loading times, since this would be trivial do find with a normal
| profiler on a debug build.
|
| Secondly (pessimistic version), nobody at rockstar had enough
| skill and/or professional pride/sense of ownership to find this
| out themselves.
| malka wrote:
| Did rockstar ever release a patch for this ?
| MontagFTB wrote:
| A user released one first, then rockstar did subsequently:
| https://www.pcgamer.com/rockstar-thanks-gta-online-player-wh...
| legobmw99 wrote:
| Yes -
| https://www.gamesindustry.biz/articles/2021-03-16-rockstar-p...
| NickRandom wrote:
| I'm not sure if you made it to the end of TFA, but yes. And the
| author got a 10k bug bounty
| Hamuko wrote:
| They took his fix, applied it to the game in an update and paid
| him a bug bounty of $10,000.
|
| "Just got awarded $10k through their H1 in-game bounty as an
| exception"
|
| https://support.rockstargames.com/articles/360061161574/GTAV...
| yeahSo wrote:
| Anecdotally; I still wait 5-6 minutes for online to load, and
| 3-4 for single player.
| Hamuko wrote:
| Digital Foundry measured a 20.2-second boot load time for
| GTA V single player on the PS5 and 28.9-second boot load
| time on a PC. Not even the PS4 takes 3-4 minutes.
|
| https://www.youtube.com/watch?v=m5QHInJShQw
| yeahSo wrote:
| aliswe wrote:
| This is why you need a language (on a framework) with high
| introspection, with smart insights. Excuse my opinionatedism, but
| this is what .NET Core gives you. It even streams JSON
| serialization/deserialization without much memory requirements,
| iirc.
| outworlder wrote:
| Nah.
|
| We just need to stop this idea that JSON(or similar formats) is
| an acceptable data storage mechanism. It's fine when
| interacting with low bandwidth APIs across different systems,
| where the need for debugging is greater than performance.
|
| But on a shipped game? Just bake this JSON into a better,
| binary serialization format. No need to parse data that seldom
| changes over and over again.
|
| It doesn't matter if you can stream json, if you don't need to
| call strlen, what have you. Just don't parse unnecessary stuff.
| tialaramex wrote:
| This is GTA Online, so while it's a "shipped game" that JSON
| changes, Rockstar decides to offer the "Pfister Astron
| Custom" for $2.5M and they just change the JSON. I don't know
| whether it's fetched from their servers on startup, or
| whether it's on disk and just updated periodically, but
| either way choosing JSON is not as crazy as you seem to
| think.
| mypalmike wrote:
| I'd guess that json is the least of their problems. Even with
| the fix, load time is still an unreasonable 2 minutes. This
| suggests they aren't doing any of the load performance things
| game devs have figured out over the decades, like streaming
| loaders and stacked memory pools.
| hyperman1 wrote:
| He did basic troubleshooting with task manager, on an
| obfuscated executable, with a disassembler and a memory dump,
| without knowing anything about the code base, and fixed it with
| in-memory patching. So while I agree introspection can be a
| godsend, this specific case is not a tooling problem.
|
| Something this blatant is what happens if nobody takes a look.
| I can imagine they got tons of complaints, but 'it is slow' is
| nebulous enough to consider it basically unsolvable/normal.
| Maybe it never got a real bug report, maybe the report never
| made it past the service desk, maybe management thought it
| would cause a wild goose chase without resolving anything. But
| no developer looked into this, as just hitting a breakpoint in
| the 4 minute pause would insta-fix this.
| 0xFACEFEED wrote:
| > This is why you need a language (on a framework) with high
| introspection, with smart insights.
|
| That would be nice if it were true.
|
| What happens in reality is you end up paying the performance
| cost in other areas specifically due to the framework.
| Frameworks tend to cater to very broad use cases and will
| generally be less performant than custom code written for one
| specific use case. Frameworks also tend to be extremely
| interdependent meaning that to rip out one part (for perf) you
| need to rip out another huge part.
|
| That is why, to this day, if one's business NEEDS something to
| be performant (no do-overs! no second chances!) then industry
| veterans will opt for custom systems. Yes they'll have more
| edge case bugs. Yes they won't be performant in areas you don't
| care about. But you will still sell 165 million copies of your
| game because it performs well on a bunch of different platforms
| with varying specs.
|
| This particular case is more of an organizational problem. No
| one was responsible for load times, it wasn't a priority to
| improve, and users weren't complaining enough about it to
| matter.
| NickRandom wrote:
| Previous submission (as linked in tfa)
| https://news.ycombinator.com/item?id=26296339 (Feb 2021, 699
| comments, 3883 points)
| blibble wrote:
| I tried playing GTA online once and actually gave up due to the
| loading times (on a top end gaming rig)
|
| I can't believe I'm alone, I wonder how much this cost rockstar
| SketchySeaBeast wrote:
| My exact same experience. As to what it cost, in the old world
| paradigm I'd say "nothing, we already bought the game", but I
| guess these days there's probably a decent argument for a
| cohort of people who are too impatient to wait to get into the
| game and who pay extravagant sums of real-world money to get
| in-game items immediately.
| blibble wrote:
| yes, GTA online's business model is making you buy in-game
| currency and paid DLC
|
| which is kinda hard if the game takes 30 minutes to load
| hermitdev wrote:
| There's other costs: how many times per day to R* devs &
| testers have to go through the same loading? How many man-
| hours wasted? How many compute resources wasted? How many
| upgrades were justified by "I need better hardware to get my
| job done in a timely manner"?
| dieulot wrote:
| Same. Asking around why players tolerate it, the anecdotal
| conclusion I've got is that what makes these frequent and
| extremely slow loads tolerable is playing with friends on audio
| chat.
| bastardoperator wrote:
| Not much, the GTA5 online scene is alive and well. Considering
| they've sold this game over the course of three generations
| makes me believe they've made a mega flipton of money off this
| game.
| [deleted]
| blablabla987 wrote:
| I know so many people that have stopped playing due to the
| loading times. Just because a scene is active and alive,
| doesn't mean it couldn't be even more active and alive with
| such a minor fix that could be implemented within a day.
|
| I am pretty sure they lost >100.000.000$ by that. Just shows
| how shitty the management over there is that they never tried
| fixing that problem.
| bastardoperator wrote:
| By the time you get to the loading screen they've already
| profited. The fact that they continue to sell the same game
| over and over means individuals have likely purchased the
| game multiple times meaning they don't care about loading
| times, they just like the game.
|
| It's easy to say their management is shitty from the
| comfort of your computer, but the reality is Rockstar
| produces great games, and they're making a ton of money
| because there games are good. I find most multiplayer games
| have a start up cost, glad they shortened it, but I don't
| think it's stopped anyone from playing that really wants to
| play.
|
| https://www.thegamer.com/gta-5-rockstar-2-5-million-per-
| day/ https://www.tweaktown.com/news/80912/grand-theft-auto-
| made-o...
| zerocrates wrote:
| The action is, as always, on the margins. Those people
| who _kind of_ wanted to play but got frustrated at the
| load times, you want them. In some senses you care _more_
| about them, the people you can gain or lose with changes,
| than your more committed users who are a little bit
| locked in.
|
| GTA V has sold, and continues to sell, amazingly well.
| But Online, where this slowdown was exclusive to, is the
| real moneymaking engine. Any little marginal push to keep
| people out of Online, from feeling they could just jump
| in whenever, was significant.
|
| The real baffler to me is just thinking about all the
| time spent by people internal to Rockstar sitting on that
| same screen.
| blablabla987 wrote:
| My statement is mostly based on the assumption that a
| large part of their revenue is based on
| microtransactions, for which satisfied customers are key.
| Therefore if I stop playing due to the loading times,
| they effectively loose money since I certainly won't buy
| any in-game money with my real money. But you are of
| course right that I have already bought the game.
|
| Regarding the management: If it takes >5 minutes on top
| high-end gaming rigs to load a game and if your loading
| times have become a meme in the internet, I think a
| somewhat reasonable product manager should ask their
| developers to at least invest 5 minutes in figuring out
| why it takes so long (that's probably about the time
| needed to figure it out given the tools the developers
| have) and if it can be decreased in a reasonable amount
| of time. I was just amazed by how simple the problem
| actually was for an issue that has led me and many others
| to stop playing. (And I actually really wanted to play,
| but I just didn't want to look 15 minutes at loading
| screens for having 5 minutes of actual playtime.)
| mrandish wrote:
| > they've made a mega flipton of money off this game.
|
| It can be simultaneously true that they made a lot off the
| game && with lower loading times they could have made even
| more. In-game microtransaction ARPU scales with playtime. >4
| min per session spent waiting to load instead of playing ==
| lost rev.
| seanp2k2 wrote:
| Think about how many human lifetimes worth of extra minutes
| were wasted from everyone waiting for this x number of players
| X years this has been a problem. Even if you can't implement
| bubble sort on a whiteboard in your sleep, 5+ minute load times
| should have been setting off alarm bells in many engineers
| heads. I'm sure this did actually cost R* real money. Bounce
| rates for websites increase 32% for 1s vs 3s load times,
| according to Google: https://www.thinkwithgoogle.com/marketing-
| strategies/app-and... Obviously we don't have public data on it
| but I'm quite sure that some at least some people got
| frustrated with waiting and just stopped playing.
| p49k wrote:
| Your comment reminded me of this:
|
| https://www.folklore.org/StoryView.py?project=Macintosh&stor.
| ..
| jawarner wrote:
| If you think about it, those lives actually were saved.
| People either didn't play the video game, or they spent a
| moment in the loading screen in quiet contemplation.
| glenneroo wrote:
| To play devil's advocate we should balance those out with
| the impatient gamers who acted out aggressively (e.g. the
| 100th time proceeded to kick a PC, smaller family member,
| throwing a mouse across the room, etc) or consumed more
| unhealthy snacks/drinks :(
|
| I personally was often frustrated waiting several minutes,
| thankfully I don't have any violent tendencies, snacks on
| the other hand... :/
| asdff wrote:
| We are in the era where its impossible to be bored. In the
| elevator? Phone out on instagram. Loading screen? Same thing.
| Instant dopamine the minute you need a drip. The playerbase
| clearly doesn't care. To be honest, these loading screen
| times are as bad as they've always been for console games
| (especially considering it was only a generation ago that
| they ran off a disc).
| jml7c5 wrote:
| From complaints I saw online, users emphatically cared. It
| was a common complaint, and some users would report that
| the load times would deter them from playing the game.
| asdff wrote:
| Probably nothing. Loading times in rockstar games on consoles
| have always been this bad. GTA4 was just as bad too.
| baby wrote:
| You're probably wrong. I remember e-commerces trying to
| quantify how many millions of dollars they would lose when
| clicking would take too much time on their websites.
|
| EDIT: things like https://www.fastcompany.com/1825005/how-
| one-second-could-cos...
|
| > Amazon's calculated that a page load slowdown of just one
| second could cost it $1.6 billion in sales each year. Google
| has calculated that by slowing its search results by just
| four tenths of a second they could lose 8 million searches
| per day
| asdff wrote:
| That's ecommerce. This is a video game and a unique one at
| that, so there are other factors at play. I don't think
| many people go "game took forever to load, guess I'll play
| for less time then." There could even be a bullwhip effect,
| a long load time might make you want to stay playing for a
| longer period of time.
| zionic wrote:
| The same thing happened to me. GTAO earns billions/year, crazy
| that "fix the insane loading times" didn't bubble up to their
| top-priority.
| blobbers wrote:
| A heart warming story of a hacker turned game dev! Nice job.
|
| They offer you a job too?
| debdut wrote:
| Please add RSS to your blog!
| Tarragon wrote:
| I assume there is an upcoming: Show HN: Passive karma farming to
| 10k
|
| Two submissions an hour, 24 hours a day...
| lostgame wrote:
| Wow. I wanted to call you out for just being an asshole, but
| _seriously_ - is this account just a bot? And who cares about
| karma on HN?
| Tarragon wrote:
| Both things can be true.
|
| There's clearly a bot here but ddtaylor has commented on this
| thread, so there's also a person in there somewhere.
| [deleted]
| kingcharles wrote:
| I used to be a video game developer.
|
| I guess it's down to priorities and passing the buck.
|
| Our priorities were always on profiling the running game to make
| sure we got every ounce of juice out of the hardware and the game
| ran smoothly on the lowest systems. I can't think of one time we
| would have had the spare time to look at loading times. In those
| days we were being pushed into 80+ hour weeks just to make the
| game work at all, never mind finding time for "luxuries" such as
| this.
|
| And of course, on a big team, a problem like this would always be
| someone else's problem.
| blobbers wrote:
| You are saying a game had a 6 minute load screen and everybody
| would throw their hands up and say "but look at the dynamic
| lighting once it loads... it'll all be worthwhile!"
|
| If that's not group think gone bad, I don't know what is.
| barrysteve wrote:
| The customer has put up with 6 minute load times, but a low
| framerate kills sales quickly (unless you're Star Citizen).
| kingcharles wrote:
| And kills reviews. Most reviewers won't freak about the
| load time, but the framerate they will.
| outworlder wrote:
| Not to mention, unless they have a really clever system to do
| hot code reloading, EVERY SINGLE DEVELOPER AND TESTER would
| have been plagued by these load times.
|
| I wonder if the watercooler rooms at Rockstar are
| exceptionally furnished as a result.
| kingcharles wrote:
| Sometimes testers aren't even working with the full data
| set. And testers might have some seriously shit-hot
| hardware that is max spec too. When I was developing
| everyone would go buy the absolute best PCs out there at
| the start of every new game to make sure they lasted
| through dev.
| thatswrong0 wrote:
| I cannot fathom how dysfunctional Rockstar must be that this
| slipped through the cracks for so long given the fact that GTA
| Online is Rockstar's cash cow.
|
| 6 minutes is insanity and must have turned off so many people
| from playing.. this person deserved far more than $10k
| swarnie wrote:
| Because the user experience isn't important.
|
| Enough still play it and buy a few micro tractions per month
| regardless of the load times Dev time is better spent
| reskinning a hat or something.
| ssharp wrote:
| Certainly there has to be some portion of the potential
| audience who would have become microtransaction purchasers
| who didn't because they were turned off by the long load
| times.
| rvba wrote:
| You are probably being sarcastic, or cynical about current
| corporate business world where nobody cares.
|
| But user experience is important. With faster lpad timea they
| probably could get more players. More players probably means
| more money, since there are more chances to convert someone
| from a player to a paying customer.
|
| Perhaps if someone can endure a 6 minute load time they are
| more likely to buy something in the game (a fan of the
| series?), but we could say that those who like fast load
| times also use their wallets fast. But it is pure speculation
| - we dont have the data. Also obviously Rockstar doesnt have
| this data - they couldnt A/B test if load time converts to
| more sales - their game only had bad load times.
| s-xyz wrote:
| Wow, wow, wow! This was such a nice read. Definitely buying the
| guy a beer (there is a button at the bottom).
| kevinventullo wrote:
| There's an implicit story here which I find quite sad that not a
| single developer at Rockstar in the prior eight years had the
| autonomy or bandwidth to discover and fix this themselves.
| madrox wrote:
| This was discussed a bit in the original post [1] but I believe
| this comes down to the problem being sufficiently deep in the
| stack that there was an assumption that this wasn't something
| that could be improved on. I'm sure the loading code looked
| good.
|
| It took someone who didn't know what that code looked like
| being curious about what was actually happening.
|
| 1: https://news.ycombinator.com/item?id=26296339
| boredemployee wrote:
| where I work at, it is completely discouraged to do anything
| out of what was asked you to do. so I don't judge the
| workers/slaves. It's not allowed to be creative/careful in most
| places if it's not harming the bosse's wallets.
| ReaLNero wrote:
| In order for a worker to bother with this
|
| 1) It has to be alright to make miscellaneous changes like
| this, and
|
| 2) It has to contribute to getting promoted.
|
| The second point is IMO why random QoL changes like this are
| rare in large corporations.
| Thaxll wrote:
| #2: Just do your assigned task well, that's it.
| switchbak wrote:
| I've had this situation fluctuate at my workplace. When
| misc changes are supported, and those improvements are
| actually valued by management, I've seen the system improve
| dramatically.
|
| I've also seen a single-minded focus on pushing cards
| across a board, which seems to typically correlate with
| higher levels of a dogmatic SCRUM process.
|
| Pushing cards is great, but not when it prevents the
| various little things that have to happen to cultivate
| quality software. Especially not when there's a gatekeeper
| prioritizing these things into oblivion.
| aidenn0 wrote:
| I don't know that #2 is universally true. I am at a point
| in my career now where I will be able to comfortably retire
| without ever being promoted again (i.,e. CoL raises only).
| Judging by the salary ranges I see on offer for jobs for
| people with 5+ years of experience, this can't be _that_
| rare.
|
| When I see a pain point that looks soluble, I take a crack
| at it. Sometimes it works, sometimes it doesn't. Sometimes
| I get recognized for it, sometimes I don't. One, relatively
| minor (to me) thing, ended up being a big business win and
| my manager made sure I got a fat bonus for it. Other things
| have seemed more like a win for me that nobody but 2 of my
| peers ended up caring about.
|
| #1 is true to the nth power though. I've seen places where
| if you do something even one millimeter outside of your
| silo, it will come up as a negative in your performance
| review. As in, if you write a 5 line perl script that makes
| you significantly more productive, _do not_ share it as you
| will be asked time and time again why you wasted your time
| on something outside of your job description.
|
| I think something missing for the context of TFA is that
| the gaming industry tends to be much more deadline focused
| than other industries. In theory you might be free to "make
| miscellaneous changes like this" but only if things aren't
| already behind schedule, and things are _always_ behind
| schedule.
| ReaLNero wrote:
| > As in, if you write a 5 line perl script that makes you
| significantly more productive, do not share it as you
| will be asked time and time again why you wasted your
| time on something outside of your job description.
|
| This sounds like something I'd see in a Dilbert comic. Do
| you mind sharing what industry you work in? Perhaps you
| saw this in old-fashioned industries like banking or
| insurance? Have you experienced this in tech?
| aidenn0 wrote:
| This specific story was a friend working for a large
| defense contractor. I've never worked directly for a
| defense contractor, but know enough people that have that
| I should note that culture varies _greatly_ from site to
| site; different sites of very large companies might as
| well be different companies (and in many cases _were_
| different companies before being purchased).
|
| That sort of thing also varies with perceived team
| performance. If your team is perceived by upper mid-
| management (or higher) as being "good" then individuals
| on the team have more freedom. If your team is perceived
| as being "not good" then every deviation from standard
| practices is clearly the cause of the poor team
| performance.
| jenscow wrote:
| I strongly believe that R&D teams should have some time
| reserved to "do what you want" (limited to product improvement,
| obviously)
|
| Perhaps this might not be appropriate in the gaming industry...
| and who knows if Rockstar don't already have this.
|
| However, the GTA loading time is horrendous (and I had a
| spectrum sinclair when I was a kid) - surely there must have
| been a work item, ticket, bug reports, or whatever, for this.
| themerone wrote:
| I have a solid 2-3 years worth of work assigned to me in my
| issue tracker, with new issues rolling in every day.
|
| All I can do is prioritize and hope nobody gets too unhappy
| that their pet features is never going to happen.
| havblue wrote:
| You would think that the more time people spend playing the
| game, the more money the game makes. I would think online
| loading metrics would be a similar problem for whomever is
| responsible for input latency.
| azemetre wrote:
| I'm starting to slowly realize how terrible it is working in
| environments where people simply aren't curious. It's so hard
| for me as an individual contributor to decouple this way of
| thinking. Curiosity is how we find better ways of doing things
| or finding out hidden flows that can be better. Incurious
| people are a contagion, but I want to believe you can make
| people curious. It can't be innate.
|
| Every company talks about wanting to have driven curious
| employees to solve complex issues or challenges but you just
| get stomped with a boot to your face if you do anything outside
| your lane or simply ask why.
|
| It's also worse because I'm starting to realize how lucky I was
| to be raised as a curious individual from my parents who never
| went to college but at least were curious themselves about
| different things (pottery, soldering, gardening).
|
| The quote "Curiosity is insubordination in its purest form"
| suddenly has a different context when working for a company.
| rexpop wrote:
| I'd never heard that Nabakov quote, but it's terrifically
| cynical and, I hope, false but, in my experience, not
| entirely untrue.
|
| Designers' unnecessary desire to delight users staves off the
| more Kafkaesque user experiences, and we all live in the gap
| of cops' unnecessary (occasional) sympathy for (some) nominal
| outlaws (we're all nominal outlaws).[0]
|
| When you mentioned curiosity I thought of a firm where
| colleagues scoffed at me as I rummaged through conference
| room drawers, saying, "you're not going to find anything
| interesting in there." I'm curious about cupboards as well as
| log traces -- an almost ubiquitously "eyes-glaze-over" boring
| artifact of modernity that, alongside popularly mind-numbing
| "code", I am equipped to find interesting.
|
| Is it any wonder, then, in this workplace where such
| innocent, cheap, curiosity were scoffed at, I was not too
| long after threatened with the extraordinarily archaic (in
| our at-will state, at least) legal notion of
| "insubordination"?
|
| It's a shame, because insofar as "all ambiguity is resolved
| by actions of practitioners at the sharp end of the
| system,"[1] the liberated curiosity of individual
| contributors is often the only "crack in everything" that's
| "how the light gets in."[2]
|
| 0. "With two lines of a man's handwriting, an accusation
| could be made against the most innocent." (Francoise Bertaut
| de Motteville)
|
| 1. https://how.complexsystems.fail/
|
| 2. "Anthem" - Leonard Cohen
| ramesh31 wrote:
| >Every company talks about wanting to have driven curious
| employees to solve complex issues or challenges but you just
| get stomped with a boot to your face if you do anything
| outside your lane or simply ask why.
|
| That's because "curiosity" translates to "more work" at
| #BigCo. And people work at #BigCo so that they can
| comfortably do as little work as necessary to maintain their
| lifestyles. This is a hard rule for any organization. So if
| you want what you're describing, it will only be found at
| small startups where everyone has skin in the game.
| TheAceOfHearts wrote:
| Don't blame the people, blame the environment.
|
| People might start off curious but eventually they keep
| running into political roadblocks and other nonsense, which
| slowly beats the interest out of them.
|
| If those are the kinds of people that continue surviving then
| those are the kinds of people which your environment
| cultivates.
| azemetre wrote:
| That's totally fair, I honestly meant to blame the
| environment and not people. It just sucks to see the
| pattern of curiosity get stamped out of people due to
| environmental changes that aren't obvious.
| [deleted]
| miahi wrote:
| Discussion from today: "Do you know if feature X works? A
| customer is asking about it" "No, it was disabled because
| of a bug a while back" "Hmm, found the ticket, it's from 4
| years ago!" "Yes, basically it's a bug from a new version
| of a 3rd party software that made it unusable. At that
| point it was basically decided that we will not try to find
| a workaround, we just tell the customer that it's a 3rd
| party issue."
|
| The funny thing is, all the people who made that decision
| left the company/project and now we have the option to
| change it. Can't wait to see what happens next, if the new
| leadership changes that or not (my money is on "not").
|
| The other funny thing is, I can think of at least two
| workarounds that only need a day of work and a couple for
| testing it.
| [deleted]
| mrandish wrote:
| > I want to believe you can make people curious
|
| In my experience it's the opposite. Most people are born
| innately curious. It takes intense systemic conditioning to
| suppress their natural curiosity. For many of us that
| conditioning begins in the traditional education system and
| is completed in the workplace.
| jrlowe wrote:
| >It takes intense systemic conditioning to suppress their
| natural curiosity.
|
| Yep, the public school system in America excels at
| squelching curiosity and any desire to learn.
| sassy_quat wrote:
| Just to be clear, you are saying, matter of factly, that
| the whole point of the American education system, is to
| suppress free thought.
|
| You better tell Congress, the last thing they want is for
| America to fall into totalitarianism.
| NateEag wrote:
| John Taylor Gatto was an award-winning public
| schoolteacher who more-or-less agreed with this
| sentiment.
|
| https://en.wikipedia.org/wiki/John_Taylor_Gatto
| sassy_quat wrote:
| How are you incapable of understand sarcasm? In fact
| you're arguing from Appeal to Authority now. I just, I
| don't understand. The weird thing is that your argument
| is extremely convincing at first glance even though the
| entirety of your argument is in a link to wikipedia?
| wizzwizz4 wrote:
| Sharing knowledge and understanding doesn't have to be an
| argument.
| jgust wrote:
| I found this on the ground, did you drop it? </s>
| heurisko wrote:
| It's not a particularly American phenomenon.
|
| I can remember reading "Deschooling Society" by Ivan
| Illich whilst in the UK's comprehensive school system.
|
| On retrospect, I don't think I realised the importance of
| schools facilitating socialisation, rather than just
| learning.
| samstave wrote:
| My brother is the polar opposite of me when it comes to
| curiosity.
|
| I ALWAYS ask questions like "How did this come to be, How
| much did this cost, who designed this and why" etc.
|
| My brother gets annoyed that A) I ask and question
| everything, and B) Laments he doesnt know as much as me.
|
| However, this is spurred by my ADHD... and I at times wish to
| be able to have a more precise control of my focus.
| asdff wrote:
| Curiosity has to come from the top and fall all the way down.
| As others and you have noted in other comments, its the
| environment for the workers that sets up whether or not they
| will even be able to pursue curious things, and that's on the
| managers to create that environent. However, the managers
| themselves have bossess too, so that has to trickle up, to
| their bosses bosses all the way up the chain.
|
| In short, every job needs to be able to have autonomy to
| enact on the ground solutions for what their immediate team
| needs. You shouldn't be penalized for going off the
| reservation and coming back with a creative solution. The
| problem is corporate structure for the past few decades has
| been about rooting out this autonomy and anything opaque
| about job responsibilities. There is a lot that's been done
| that would need undoing to change those environments, and imo
| you are better off finding another workplace than bringing
| about the monumental change required. A mature corporation is
| like a boulder tumbling down a mountainside; you can hardly
| change its course much less stop it short of where its
| heading.
| khalladay wrote:
| I'd be careful about painting the developers at Rockstar as
| "not curious," that feels like an incredibly unfair
| conclusion to draw from this. Those developers have other
| pressures, and deadlines to hit. There are a million reasons
| why this bug wasn't solved by the team there. When a user
| complains about a bug in software you wrote, is it because
| you just "weren't curious" enough? Or is the situation more
| complicated?
|
| Lots of incredibly talented people worked on GTA Online, it's
| worth trying to think through why a team of smart developers
| might not have prioritized this, rather than assuming
| something negative about them.
| asveikau wrote:
| I agree with the sentiment. However, it is a tad surprising
| when you consider the productivity hit, even just
| internally speaking, that ~5min of cpu time in strlen() on
| every start of the app incurs.
|
| How much time did perfectly qualified engineers, who could
| have attached a profiler at any moment, spend sitting at a
| loading screen spending most of its time in unnecessary
| strlen() calls? I understand we see this through 20/20
| hindsight, serendipity can work against you, etc., but
| that's a crazy irony.
| r00fus wrote:
| A toxic culture resistant to change can obtain these
| results no matter how brilliant/skilled the people.
| asveikau wrote:
| It's true, but also, you can get this result with a
| competent team and good culture, curious people, etc. You
| could just have bad luck and nobody happened to check it.
| delusional wrote:
| You might say that assuming this was due to inherent
| incuriousity amongst the developers is quite incurious.
| 867-5309 wrote:
| it was a feature - they served ads during loading screens
| azemetre wrote:
| I wasn't trying to blame engineers that wasn't my
| intention, but someone needs to own establishing the
| environment that actively encourages or doesn't encourage
| curiosity.
|
| Talented people are found everywhere, but hopefully someone
| with authority over them doesn't slowly poison the drive
| out of them.
| Griffinsauce wrote:
| You are assuming curiosity was the problem instead of a
| lot of other possible factors like bandwidth.
| azemetre wrote:
| I wasn't blaming this problem solely existing due to
| being incurious but more of lamenting how corporate
| environments don't encourage people to be curious.
| blowski wrote:
| Why should developers necessarily be trying to cut
| loading times? Perhaps GTA is phenomenally successful
| because they have good BAs and POs who are able to
| identify and convince engineers to work on what matters
| to the bottom line.
| dingleberry420 wrote:
| You've clearly never played it nor read any of the
| community discussions about the game. The #1 complaint is
| (or was) loading times. It's why I've personally stopped
| playing years ago.
| blowski wrote:
| I play GTA, am annoyed by those loading screens, and have
| found those forums while looking for solutions.
|
| Companies are successful not because they delight every
| customer consistently, but because they do just enough to
| make the customer spend money with them.
|
| After all, I will still buy the next version.
| [deleted]
| azemetre wrote:
| I mean that's fair, but I'm talking about wanting to be
| curious and encouraging environments where curious
| individuals can be lauded instead of dismissed. Not
| everyone on a team HAS to be curious but for those that
| aren't they shouldn't be smacked upside the head with
| corporate bureaucracy.
| nwallin wrote:
| Loading times were the #1 complaint from users about the
| game. I personally wouldn't play a game where 5 minute
| loading times were the norm, 10+ minute times are common,
| and 15+ minute times are not unheard of.
|
| It's the #1 complaint. The thing that makes this
| egregious isn't so much that they didn't fix the #1
| complain, the thing that makes it egregious is that it's
| a simple enough flaw that a mediocre developer could have
| just run WPR/WPA against it and see that 90% of those 5+
| minutes was spent in strlen. Maybe it's not their area,
| maybe they don't know how to fix it, but they will
| recognize that it's a problem with a likely easy fix.
|
| So even if we completely ignore the fact that it's the #1
| complaint, it's still egregious that it's been allowed to
| go on for this long.
| s0l1dsnak3123 wrote:
| Screw this. Developers should advocate for their craft
| and for quality, and business metrics should hold an
| _adversarial_ position to balance things out. We
| definitely should not be accepting the narrative that
| developers just do what they are told and don 't push
| back on what _we_ know is good for product - we are in a
| unique position of power, we should recognise this and
| use it for the betterment of the industry as a whole. The
| alternative is we let corporate greed slide our standards
| of quality into the gutter thereby making the job at hand
| miserable.
| blowski wrote:
| Sounds like an extremely false dichotomy. There is also
| the solution that you set up your own company, or
| convince your manager your time is best spent on this.
|
| I've worked around too many startups ruled by developers.
| They're being run into the ground because those
| developers think technical NFRs as defined by them are
| the be all and end all. They're not. Those same
| developers will happily spend months fiddling with the
| system without making any actual improvements and call it
| "paying down technical debt".
|
| Give me a PM (or good engineer) that understands the
| bigger picture every time.
| s0l1dsnak3123 wrote:
| > There is also the solution that you set up your own
| company, or convince your manager your time is best spent
| on this.
|
| This is the definition of a flippant comment. Why should
| society require we become the lord of our own domain or a
| political chameleon just so that we can advocate for
| decent standards? Should people who live on flood plains
| "just move" when the tide comes in? Ridiculous.
|
| Understanding the bigger picture _includes_ understanding
| the developer 's point of view.
|
| > I've worked around too many startups ruled by
| developers. They're being run into the ground because
| those developers think technical NFRs as defined by them
| are the be all and end all. They're not. Those same
| developers will happily spend months fiddling with the
| system without making any actual improvements and call it
| "paying down technical debt".
|
| Sounds like those super-clever managers didn't understand
| what the hell was going on to me. Perhaps they should've
| held their nose and tried to learn about their product so
| they can understand and communicate that bigger picture
| they speak of instead of abdicating their adversarial
| role and leaving dev teams rudderless.
| blowski wrote:
| What makes you think they didn't understand the
| developers' point of view? Maybe they understood it, but
| didn't agree with it.
| s0l1dsnak3123 wrote:
| > Give me a PM (or good engineer) that understands the
| bigger picture every time.
|
| I was referring to this sentence: you're now telling us
| that the ideal good engineer understands the developer's
| point of view but doesn't agree with it. It sounds like
| you just want to be surrounded by yes men to me. The
| perfect recipe for a failed startup.
| pdimitar wrote:
| Even if your parent comment presented false dichotomy,
| you responding with the other extreme isn't convincing.
|
| As a senior programmer I respect business priorities and
| I don't go fixing technical debt for months.
|
| That being said, we the people suck at moderation. See,
| the problem I faced was exactly the opposite of what you
| encountered: I've been too accommodating and allowed
| myself to be micro-managed to the point of a manager
| telling me _how_ should I do my job; that 's when it
| finally clicked with me on what a toxic workplace have I
| landed back then.
|
| So neither of both extremes is good. I agree business
| priorities have to be respected; absolutely. At the same
| time we shouldn't allow the businessmen to tell us we
| can't ever fix technical debt.
| s0l1dsnak3123 wrote:
| Exactly. Managers are there to facilitate people and
| communication, business leaders are there to facilitate
| capital and strategic vision, engineers actually make the
| thing and master the detail in the process. Good
| businesses learn to listen: not just to their customers,
| but also to their domain experts.
| heretogetout wrote:
| This is why I try to hold developers accountable for
| actions their software performs or allows, like auto-
| banning accounts or not supporting billing limits.
| tshaddox wrote:
| > Developers should advocate for their craft and for
| quality, and business metrics should hold an adversarial
| position to balance things out.
|
| What makes you think this _isn 't_ the case? Business
| metrics holding an adversarial position over other
| interests obviously doesn't preclude business metrics
| _defeating_ other interests.
| pdimitar wrote:
| In my experience, business interests always override
| everything else. Often times to the detriment of the
| business, and often as soon as mere 3 to 6 months in the
| future.
| sreekotay wrote:
| Very much agree with this.
|
| That said given the quality of what they have put out,
| I'd surmise the issue here is morelikely they don't feel
| the same pain on load times?
|
| Bandwidth too good, rigs too powerful, dev envs
| overpowered, etc. to enable developer velocity can often
| also mask real world problems.
|
| Hell, even hot reload can mean you don't see load as
| often and are able to easily rationalize as "not that big
| a deal" - I wouldn't necessarily take it as a sign they
| are not deeply invested in their craft.
| foobarian wrote:
| I don't know, this seems like it was a pure CPU problem
| on good hardware. But anyway a developer who has 10 hour
| days and backlog stretching beyond the horizon will not
| have idle time to go try things out. It needs to be
| someone's explicit job to go hunt down things like this.
|
| And given it was data-size dependent, I wouldn't be
| surprised this was not an issue at the start while
| someone who's job it was to look for this kind of issue
| was looking. And the issue appeared after they got
| retasked to something else.
|
| In fact maybe this is the only kind of issue that you end
| up with after all is said and done. Just like in WW2 they
| reinforced the parts of the fuselage on returning fighter
| craft without bullet holes, because if a bullet hit
| there, the craft would not return.
| s0l1dsnak3123 wrote:
| I very much agree with you as well actually.
|
| One of the things that I've instituted at my work is our
| methodology for acquiring test phones for our product
| (WhatsApp for hospitals, basically). We go to a
| supermarket and buy the most eye-catching box of the
| budget phones available in the phone aisle.
|
| We've squashed _so_ many bugs and performance problems
| this way. It 's cheaper, it instils empathy in product
| decisions, and our customers love the user experience as
| a result.
|
| Similarly, one of my first big decisions after joining
| was to wean ourselves from our AWS addiction using
| Kubernetes. This also allowed us to design a dev env that
| fits on a modern machine comfortably, but can scale to
| entire Hospital Trusts in production. Once again:
| focusing on portability now means we can sell our product
| on-prem and hybrid, because we designed our _developer
| UX_ this way.
| terafo wrote:
| Game was out for almost EIGHT YEARS by the time this bug
| was fixed. There can be no reasonable explanation for them
| not fixing that. 5 minute load every time you try to go
| into multiplayer. The amount of revenue lost to that one
| single bug is staggering. This bug was a reason why I never
| played GTA online, for example.
| alach11 wrote:
| Sadly at Rockstar it's likely that employees are curious, but
| are under so much pressure for deadlines, and working such
| long hours, that they don't have time to follow their
| curiosity.
| asdff wrote:
| Why are they working so hard on an 8 year old game? Aren't
| they just basically producing content packs now? Doesn't
| even seem like the playerbase cares when there are
| gamebreaking bugs ongoing for months.
| switchbak wrote:
| When problems are this obvious, it's surprising that
| someone doesn't have a look, even if they're busy. Maybe
| I'm asking too much/projecting here, but I just don't
| understand how people put up with fairly obvious semi-
| breakage as much as they do.
|
| I have this at my current job, I seem to be the only one
| (not really, but seems like it sometimes) interested in
| finding out why things are slow/broken and investing the
| time to fix it. If you're "busy" all the time, but put up
| with things that slow you down, you're hamstringing
| yourself by not fixing the brokenness. When only a small
| number of people take the time to fix things (typically
| other's issues), that becomes very demotivating.
|
| That would be like being on an all-day ski tour, and
| feeling too rushed to bother getting rid of the slush on
| your skis. Yes, you feel rushed because you're moving
| slower that usual. Taking 4 minutes to kick the slush off
| will make you much faster on the overall trip. (Sharpening
| the saw analogy, etc).
|
| Sometimes "busy" is a state of mind, more than it is a
| reflection of reality. I know each situation is unique, but
| I've seen this enough times to understand that it's often a
| form of dysfunction rather than a reasonable reaction to
| the work situation. This is like when people used to say
| "we're too busy to write tests". Well, maybe you need to
| get better at communicating the trade-offs to the broader
| business, and be a bit more assertive about non-functional
| aspects that are still important to address.
| 0xFACEFEED wrote:
| > Maybe I'm asking too much/projecting here, but I just
| don't understand how people put up with fairly obvious
| semi-breakage as much as they do.
|
| > I have this at my current job, I seem to be the only
| one (not really, but seems like it sometimes) interested
| in finding out why things are slow/broken and investing
| the time to fix it.
|
| You will quickly shed yourself of this philosophy if you
| become a business owner. You can test those waters by
| releasing a non-trivial open source project that gains a
| non-trivial amount of users.
|
| There will always be at least 100 things that are broken
| in some way but not impacting the core business. You'll
| have to pick only 1 to work on. And, if you're unlucky,
| some frustrated customer may write a blog post about
| issue #65. An issue that you're fully aware of but
| haven't prioritized.
|
| Guess what you'll do as a business owner? You'll
| prioritize that issue since it's now very visible and
| maybe even throw $10,000 at the customer who took time to
| do their research; as is what happened here.
|
| > This is like when people used to say "we're too busy to
| write tests". Well, maybe you need to get better at
| communicating the trade-offs to the broader business, and
| be a bit more assertive about non-functional aspects that
| are still important to address.
|
| Or perhaps the need for tests was communicated and the
| business made the decision to not prioritize it.
|
| You may not like that decision.
|
| Mind that the decision to "not write tests" is never
| explicitly stated so simply. No one wants to look bad.
| Instead that decision is expressed through minimizing its
| level of priority compared to other things.
|
| Still don't like the decision? Well you have three
| options:
|
| 1) Ignore the stated goals of the company and do what you
| think is right instead. In which case you're acting
| against the interests of the company that's paying you.
|
| 2) Sacrifice your own free time in order to provide your
| idea of value to the company (that may actually be
| valuable! but that the company doesn't value enough to
| prioritize)
|
| 3) Do what the company asks with plausible deniability.
| Also known as kicking the can. Also known as not my
| problem.
|
| Sometimes doing #1 or #2 could greatly benefit you.
| Perhaps that thing you're doing (eg: writing tests) will
| somehow measurably improve the business in the future
| WITH PROOF so that you'll get get recognized. Or perhaps
| there's management shake-up and the new leadership
| perfectly aligns with your philosophy, etc, etc.
|
| But that's rare.
|
| More often than not your attempt at fixing the company's
| problems behind the scenes will go unnoticed and
| unrewarded. And if you do get rewarded, often times it's
| a pittance compared to the amount of time you invested.
|
| My personal philosophy is I simply do not work for
| companies that don't align with what I value. For
| example, to me, it's exhausting and unrewarding to work
| at a company that doesn't appreciate a GREAT user
| experience.
| glenneroo wrote:
| From reading a lot of comments on HN, I think you might
| be in a semi-privileged position if you have enough time
| left over (and/or are allowed by management) to find
| slow/broken code and then are allowed to even commit code
| with doesn't have a JIRA ticket (created by someone else)
| attached :)
|
| Dysfunctional... possibly true but isn't that
| (unfortunately) more or less the norm in IT? Not to say
| it shouldn't be fixed but at the end of the day you have
| 'x' time to ship 'y' features in order get paid 'z'
| dollars so that the company can continue to survive...
| combined with "management" often under-estimating with
| respect to time, requirements and (un)foreseen hurdles.
| jdwithit wrote:
| +1 to the other replies here, I think it's very poor form to
| throw the engineers under the bus for not being curious. The
| most likely explanation is they were under insane time and
| resource constraints, _especially_ in the games industry.
| They wrote something that worked, then were forced to move on
| to the other 5000 tasks in their backlog that had to get done
| before release. When you 're already being told to crunch 80+
| hour weeks for months or years at a time, you run out of gas.
|
| I guess it's possible that the engineers who were talented
| enough to create this extremely impressive game in the first
| place somehow threw up their hands and said "guess it can
| never take less than 7 minutes to load, best we can do". But
| it seems much more likely that they were well aware, but
| never allowed by poor management to go back and fix it. The
| game was released and printing money, why bother. They're on
| to the next title.
| azemetre wrote:
| It wasn't my intention to blame engineers but I agree with
| the other reply to you, someone needs to be responsible for
| creating these environments. It's not like we're dealing
| with fundamental forces of nature here, this is all self
| afflicted.
| ALittleLight wrote:
| The problem with this way of thinking is that it basically
| absolves everyone of blame. Bad stuff happens and nobody's
| at fault. You point to the managers, because the engineers
| were so busy, but the managers are going to claim they were
| busy too. Again, from the manager's perspective - the
| managers aren't even in a position to understand the
| problem or the reasons for it.
| mrandish wrote:
| Having been on both sides of this (as a developer and
| later a senior manager with product/feature
| responsibility), I'd say it can be both. A _really good_
| manager /executive has enough knowledge and experience to
| ask the right questions and parse the answers even when
| they are technical. Sadly, many managers aren't that good
| (at least yet).
|
| On the other side, a _really good_ developer has the
| knowledge and experience to frame the problem in an
| understandable business context when communicating the
| issue and options to management. Some of the best work
| experiences I 've ever had were in orgs with such
| experienced developers and managers. Working together we
| were often able to solve apparently intractable problems
| with creative options which would never have been
| conceived absent the clarity of thought and communication
| on both sides.
| 420official wrote:
| This thinking implies nothing is worth doing unless it
| has a clearly articulable direct business value whereas a
| lot of technical debt simply doesn't. When the only
| people affected by a problem are people who have already
| spent their money or people who have no autonomy it's
| unlikely the decision makers will decide to investigate
| and fix it. If you're lucky you can convince decision
| makers that there is an impact to hiring or efficiency
| but in my experience these arguments are extremely
| challenging to make in the face of other stuff that
| decision makers feel directly impact business, and for
| good reason. I'd say this is a big reason autonomy is
| crucial when it comes to cultivating a curious team
| because without autonomy you are limited to things that
| can achieve consensus in an environment where people
| inherently have different interests to attend to.
|
| I feel like the "20% time" paradigm where one day out of
| the week is left up to the developer to schedule (within
| reason) is the right way to balance the conflicts of
| interest and still leave room for people to be curious.
| delusional wrote:
| "the managers aren't even in a position to understand the
| problem or the reasons for it." Which is again the fault
| of management. It is the responsibility of management to
| facilitate proper management.
| blowski wrote:
| As a shareholder, I'd be rather happy with the management
| of GTA - it's phenomenally profitable. That sounds like
| proper management to me.
| ajuc wrote:
| In gamedev where crunch is the default and people are
| routinely working for 10, 12 or more hours a day, sometimes
| including weekends, sometimes for months on end - it's not
| very kind to assume it's curiosity that they lack.
| azemetre wrote:
| Yes but death marches are typically a sign of bad project
| management IME. As for lack of curiosity, the environment
| obviously plays a huge role in what is encouraged.
|
| I feel like history is repeating itself here (at least in
| regards to death marches):
|
| https://en.wikipedia.org/wiki/Erin_Hoffman#%22EA_Spouse%22_
| b...
| rvba wrote:
| It is not that developers arent curious. They probably are in
| their cages set by bad project managers who dont even play
| own game and dont notice it takes forever to load.
|
| The project managers could put load times as a priority but
| they genuinely dont see it as a problem or dont care at all.
| It happens all the time. There is a lot of unoptimized
| software.
|
| Perhaps the issue comes from the top: the project managers
| are supposed to only do things that increase revenue and all
| other things are not rewarded at all? Perhaps executives only
| care if they hit quartely results? (Short term gain traded
| for long term loss - the game is made for whales who are so
| invested that they will play even when it is technically bad
| - so milk them now with new content for whales and ignore
| everththing else, who cares that new players will be
| discouraged by bad load times).
|
| In my opinion if nobody plays own game / eats own dogfood,
| they dont even see its problems. Blizzard is at this stage
| now - some of their games barely work. You buy Starcraft 1
| through the launcher... and it doesnt work(I think it works
| now, but it took them 1 month to notice it and another to fix
| it). Heroes of the storm - downloaded a 137mb patch every run
| - it took them half a year to notice it and do something with
| it. It is like nobody cares about the brand at all. Game
| company where your games dont work.
|
| IMHO a problem with people - that comes from the top. They
| want only story points for things that bring money. The
| project managers either dont care, or cannot deal with other
| problems at all. I bet they dont care. This happens in many
| companies - there were few people who tried, they got burned
| for that: terminated or quit -> so now nobody cares. "The
| game is good enough to still sell some skins to the whales,
| so why bother".
|
| Look at the results: they got an external comment what to
| fix. Did they launch their own internal project to optimize
| the game? Did they check for other things that make it load
| slow? It looka that someone else did a partial fix that is
| "good enough" so everything is back to usual and nobody cares
| about load times.
| jonwinstanley wrote:
| To be fair, the person that found the issue was persistent and
| did some serious digging to figure out what was going on.
|
| I imagine that even if you were on the GTA team, you'd most
| likely think the long loading time was unavoidable due to some
| complexity you were not aware of.
|
| Great work to solve it from outside of the company.
| shultays wrote:
| Serious digging was needed because he does not have the
| source. Any devs could easily profile the game during loading
| and that sscanf or strlen would be at top
|
| Maybe they saw sscanf and assumed it is normal that it takes
| time, but more likely there were so much stuff to do and
| somehow no one checked
| baby wrote:
| Shows that dogfooding is one of the best thing you can do for
| your product. I remember reading a story about Valve when they
| were working on Dota 2, I think Gabe said that every day after
| work they would play for hours and couldn't stop.
| pinche_gazpacho wrote:
| nor the incentive
| [deleted]
| xwolfi wrote:
| I work in a company where that can happen. In fact we had just
| a case today. For the last 20 years, our ever growing C#
| monster application used across the world to make millions,
| literally, had random crashes. Comes with volume spike, usage
| spike, we never quite knew: everything would display a red
| cross, the gdi handlers would reach 10k and bam.
|
| For 19 year we just looked at it in defeat. For the last year,
| bit on and off. Starting last monday it crashed every morning
| for a user sitting next to me, a big nasty german boss (Im
| French).
|
| I sat down, read his logs, saw it failed on ImageList creation
| again for some fucking reason, looked how many of these things
| we used and what they did. 2 hours later this 20 yo mystery bug
| dozens of devs had noped out of was fixed. We fully get it now.
| Wasnt even a matter of priority, it took 2 hours of looking at
| it. A lunch break basically.
|
| I dont have a sort of morale of the story or whatnot, but well,
| Rockstar would have figured it out eventually. Just need a
| nasty fucker next to you whining abt it everyday.
| roastedpeacock wrote:
| Although a bit different rather sad seeing Take Two/Rockstar
| attempt to censor other fan projects through DMCA or lawsuits.
| holtkam2 wrote:
| Perhaps my favorite thing I've ever read on HN :) I love to see
| it crop up again
| maccard wrote:
| Seconding the other comments here - I'm a developer and I'd have
| to agree it's about priorities. Every game I've worked on has had
| load time targets but those targets were set based on what the
| loading time of the game was at the time the target was
| introduced, so the goal would be to not make them regress.
|
| The other thing to note is that with this particular problem, the
| issue is parsing a json blob on a live environment. I can count
| on one hand the number of times I connected a development client
| to a live environment _ever_, it's just not the done thing.
| Developers likely have local environments that have a much
| smaller json blob to parse (and that's likely where the load time
| benchmarks were run, if they exist).
___________________________________________________________________
(page generated 2022-06-09 23:00 UTC)