I'm not sure if "embarrassing" is the right word here.
This bug has existed for years and has lead to hundreds, if not thousands of people quitting the game citing this exact reason yet Rockstar did nothing.
I think "shameful" is really more appropriate. It's not like they simply didn't know this existed. I think they have known all along it existed and didn't give 2 shits to fix it.
Watch out not to condemn them no matter what they do. Sure, they messed up in the past, but given the situation they found themselves in when they saw this report, this response is still good.
You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.
I disagree. They ignored the issue for years and that part won’t be erased from memory. They should be given credit for finally fixing it, but nothing more. What happened before this week was and remains forever shameful.
Very likely that internally there were people who had an inkling this was going on, but could never convince product to devote time to put it in the backlog. This change is a win for that engineer.
This example will be ammunition in internal debates about how much to weigh product pressure vs engineering suggestions that improvement is technically possible; the next time an eng says "let's look at this slowness one more time, it may be fixable more easily than you think", they might be listened to.
Losing customers over such a simple, known problem, over years… is tragic for a company like Rockstar. This is like Microsoft ignoring a bug that made Windows take 10 minutes to start and just letting that happen for 5 years. Now tell me that’s not tragic.
> You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.
On the other hand, a few days of doing the absolute bare minimum of quality control doesn't undo a decade of actively hostile practices. They're still very very deeply in the red when it comes to their relations with the playerbase and for good reason.
Paying $10k to the person who documented this bug is a good start, but you need to do a whole lot of work to make up for the impressions caused by that kind of prolonged systemic negligence.
Right. I'm not saying eval(their net behavior)>0; I'm saying that if your response function is always negative, you lose the ability to influence their response.
Have you heard the one about the country with the death penalty for all crime? It just leads criminals to maximally try not to get caught, without regard to damage done [i.e. removing all witnesses to even the smallest crimes]. This isn't what you want; even within "negative" relationships both sides lose when you don't have a gradient of responses.
The crux of this is that most readers presumably found out about both the bug and the fix in relatively short proximity, and their response is negative to the combined events, not to the knowledge of just the patch.
A criminal prosecuted for murder and petty theft simultaneously can't really go around complaining that they got 25 years in gaol for stealing an apple, but if everyone had already known about the murder and dealt with that accordingly it'd be pretty harsh.
(not that fixing a bug is exactly a criminal act, but the general principle remains)
An alternative possibility: there were 100 bugs of similar consequence, all of which they cared about, and they got 99 of them. This one escaped, because stuff happens.
> I think "shameful" is really more appropriate.
When you make an inadvertent mistake in the future, karma is coming for you: people will speculate that you "didn't give 2 shits".
This bug existed for a very long time and was the #1 outcry from gamers. It is well documented all over the internet.
Every company I’ve ever worked at and every code base I’ve ever worked on would have made this a #1 priority. If it were a technical limitation of the devs I would agree it is “embarrassing” but this was not outside of their ability.
I simply don’t blame the devs, I blame management. Management should have allocated the resources to fix this truly game breaking bug but they didn’t because money was still flowing. Is it any coincidence that ads played during the 6+ minutes you would wait to get into a game?
Yea, if it was a 6 month old bug or it was a hard to repro edge case, then sure it’s embarrassing. Oops! But years long, and highly visible/reproducible, without correcting? That’s veering into shameful.
Blame should always flow up, but that doesn’t mean there aren’t other causes. For all we know there has been a team of devs putting their kids through college “working” on this for years.
More likely, there were 99 other bugs that they had clearer metrics related to monetization that they prioritized (not that this bug didn't impact monetization - I bet it did, just not in a way they were measuring). Or that their QA engineers identified and flagged as high priority or major new features with their own bugs and regressions or whatever. These things get lost in the shuffle. More likely there were 1000s of bugs in the tracker, probably 10s of thousands.
Yes, I agree with your elaboration. Also, of the 100 bugs, they didn't get 99 of them, because there are surely some left. Maybe they got 90, and there are 9 still hiding out.
The game is played by millions of people, we don't need to wonder if there were a lot of more important bugs, this has been by far the most complained about issue for many years.
> This one escaped, because stuff happens.
No, they mostly just don't listen to their players when compared with other game companies.
I understand your frustration, but anyone who's run (or been on) a dev team knows you have a million things you could/should be doing, and many, many priorities. Often supporting new launches/features is the priority.
Sure, ideally there would be someone internal advocating for a fix like this on behalf of customers. But Shameful isn't the right word. It's a miss like any other.
The egg that's on Rockstar's face has more to do with how much money they likely left on the table by not diagnosing and fixing this issue early on. (And assuming they had a team overseeing this aspect of performance with adequate tools to observe and debug it should have been trivial.)
I personally feel like the magnitude of this -- especially compared to how simple the solution is -- makes it a miss that's really much more extreme than most others.
EDIT: Though I agree that "shameful" isn't the right word to use. Rockstar wronged themselves much more so than any other party.
The cause doesn’t actually matter much, usually one doesn’t know the issue until they investigate. So often issues remain open for lack of interest, then someone finally does and it’s a missing comma. This doesn’t affect the fact that the issue has been ignored.
I’m sure GTA has unlimited bugs to look into, but a bug that turns people away before they even start should have probably been given higher priority.
> If the fix had been some super application specific super obscure thing, we would all understand.
Would we?
Even if I don't understand what it's doing, devs doing nothing about an O(n^2) algorithm that takes multiple minutes every load is clearly a big mistake.
Even if it's something that has to be that slow, cache the result.
Oh absolutely, I have a whole host of stuff on my todo list for projects I'm developing that has been there for months and will remain there for months longer.
That said, they're all things like "refactor the frobnicator code so it's more obvious what it does" and "make templating system log changes to server". Nothing on my list says "Look in to that major performance issue that literally thousands of customers complain about every single day". That stuff gets sorted immediately.
There probably is a story behind it. Like some developer looked into the bug in the beginning (when when the load time was not that bad because the json file was smaller), saw that the time was spend parsing json, and then, without thinking too much whether this makes sense, wrote in the bug ticket that, to make it faster, they would have to switch from json to something better/faster/different.
Since switching the file format is probably some work, maybe involving many differnt teams and network protocols, the bug was closed because it too much efford.
And then, whenever somebody complained, the old bug was referred which unfortunately never got a proper analysis.
Sad, but I saw this happening many times in the past in some projects. Since the old and wrong analysis will be in many powerpoint slides and heads, it is incredible hard to argue against or get even time to work on such a problem. Usually, you need someone who just takes some time form other projects without telling anyone and ninja fixes the problem to solve it.
Based on my own experiences, I'm absolutely certain many people over many years tried to get the load time fixed, and then were beaten down for their earnestness.
Just as I'm certain that this unsolicited 3rd party fix was used to settle some old scores.
I think "embarrassing" could still be the correct word. If you were to ask the development teams responsible they might feel a little embarrassed over the solution being so small (yet technical).
And if they didn't care at all given what you say then why would they care to implement a fix for it all these years later?
This is akin to that guy buying a whole-page ad to complain about his ISP recently, resulting in enough shame to trigger action. It's a bad look for Rockstar considering the income that title generates and at most it was a few weeks worth of dev time. Very interesting to watch the public reaction to this in comparison though, seems to be a lot more sympathy for Rockstar than for ISPs.
They still made like a billion dollars in shark cards so apparently all the people buying shark cards didn't give a shit about the load time issue and that's the real problem.
The game supports 8 platforms, has virtually miles of terrain, and now Rockstar is suppose to feel ashamed because they didn't fix a big performance issue that affected startup times?
The game probably has tens of millions of lines of code.
I'm glad they shipped. If they can enhance the startup time, great, bonus for them. Additional credit for thanking the third party that uncovered the perf issue.
I don't think you've been following this story. If you looked into the technical details [0], and surrounding context, you would understand why your comment isn't particularly insightful. For example, terrain has nothing to do with this bug and it isn't a complex fix that interacts with "millions of lines of code." A single run of a profiler would have caught this.
I also think it's relevant to note that GTA V has sold 140 million copies, 20 million last year alone, and Rockstar makes hundreds of millions of dollars a year on the title. For a multi billion dollar product to have a known, relatively simple, problem is shameful. I don't know what went wrong, but I don't think it was the competency of engineers or the complexity of the problem that allowed this to fester.
I think you missed previous news about this. This was an incredibly trivial issue that would show up in any use of a profiler on the slow startup screens. The fact they didn't is that they're either unable to run profilers on their code at Rockstar (which would be shameful indeed) or they never bothered to do so. Either case shows egg on face at Rockstar.
Strange that this was this downvoted. It is an issue that a single dev with a profiler should've been able to identify very easily. That they hadn't addressed it despite it being such an annoyance for so many users for so long is embarrassing.
Not to take away from the feat and initiative of doing so without access to source code.
I’m pretty shocked at all the defending of the game company here, honestly. Highly visible bug, affecting high percentage of users, relatively simple fix. I take pride in the software I wrote and would be personally very embarrassed if I caused this AND didn’t fix it for years. I guess companies don’t have feelings and can’t feel shame but I would personally feel shame as a supposed professional!
There are a lot of excuses for why this wasn’t fixed but no great reasons.
The idea that screwups are "shameful" is poison in any organization. I fight it in the companies I work for, I fight it here.
Bugs happen. Serious bugs happen. They will always happen. Shaming people for bugs undermines institutional robustness against bugs because it incentivizes people to hide mistakes. Furthermore, companies with blame-n-shame culture typically fault users when those users make inevitable "mistakes" statistically guaranteed by bad UX design.
In contrast, transparency is good, acknowledgment is good. Rockstar acknowledged fault and rewarded the reporter, which is difficult. That's a positive institutional arc I'd like to encourage, at Rockstar and in the wider industry.
If you stop at "embarrassing", I'm with you. But when you proceed to "shameful", that's quite different and we part ways.
Just to clarify my point: The bug was a screwup. We’ve all done screwups. My worst bug was an amateur hour royal screwup that resulted in DDOS. Nobody should be shamed for bugs, including serious ones that get to production.
Not fixing a serious, visible, known bug for six years is what’s shameful. There is nothing positive about neglect like that. It’s kind of admirable that they actually rewarded the guy but it shouldn’t have even got to that point.
Totally agree you should not shame for bugs. The shame is for ignoring this bug until one of your customers finally had enough and fixed it by patching the binary. If it was my company I would feel ashamed.
I haven't read all the comments on these articles, but I haven't seen anyone claim that the bug shouldn't have happened, that anyone should lose their job, that no one "good" makes those sort of mistakes, etc. But this was a bug that lasted long enough and irritated enough customers that one of them finally diagnosed and seemingly fixed it without ever looking at their source code. That's one hell of a process failure. And splitting hairs over whether we would feel professionaly "embarrassed" or "ashamed" in a similar situation is pretty silly.
Managerial problems that allow high-impact, low-fix-time, high-prevalence bugs to exist for 7 years are shameful and are in fact poison to an organization themselves.
Don't mistake the shaming for really being about the bug itself here. The problem isn't the bug, it's the organizational culture that allowed such a massive, easily fixed, widely encountered bug to persist for 7 years in a game that has made them $6 billion dollars over the last few years.
They probably just didn't have anyone working on it that had the knowledge (or incentive and/or permission) to fix it. AFAIK they've just been adding content to the online portion for the past 7~8 years, and you don't need engine programmers to do that.
"They had never gotten any programmer with the requisite expertise to take a look at the code in nearly a decade of opportunities" sounds more or less like a verbose phrasing of "They didn't care."
The fact that they didn't make this a priority to fix shows exactly how much this company actually cares about it's customers and it's reputation. It's unbelievable that despite the quite public outcry about the issue NOTHING was done until some 3rd party guy figured it out for them.
I'm quite surprised they are even going to patch the game because the level of "we dont give a shit" I thought they had about this issue certainly would have prevented them from doing anything now.
Imagine all of the quadratic poorly written code sitting on your code base at your workplace. It's probably not a handful. People need to stop pretending that their code is optimized and magically linear all the time. No one looked at this because they didn't know it about. There is no reasonable way for any person to fix things like this unless they are actually looking for it. How many people at Rockstar would be responsible for engine details like this and have the skill to find and fix it?
A game video company employees artists, sound effect engineers, and etc. They aren't full of programmers combing over the source code for quadratic footguns.
Noone is complaining that they did not look through all their code for quadratic footguns. There probably are bunch of places like that, but they don't matter and doing that would probably be just a waste of time. Premature optimization is the root of all evil and it makes all sense to dedicate all your optimization on the 1% of code that actually matters the most - but this was the 1% that obviously should have been the target of optimization and was not.
The issue is that slow loading times was a contentious issue with many, many customer complaints (IMHO it was the #1 complaint) and during many years not once did they spend even trivial effort on trying to look into that complaint, and that's shameful. As the tostercx's article shows, if they had ever assigned any engineer a couple hours to look into why loading is slow with a profiler, that issue would have been easily found; the fix as described was a bit more tricky because the author did not have access to the source code, but within a company that would have literally be done within an afternoon by a single non-exceptional person. But they never cared enough to have someone look into it.
You have customers complaining about everything all the time (and non-customers with a high profile "edgy" title like GTA), so filtering out the actually relevant topics might be hard (esp things like "bad loading time" when it might be the users computer).
The thing is that this loading time issue probably didn't exist at release as it was part of something that got expanded on over time and only got bad worse by the years.
Assuming somewhat linear growth of the data the loading time would have been at least 4x shorter only half the games lifetime ago (3 years) and 16x shorter 1.5 years after release. After that period they would probably not consider there to be issues in those places in the code so nobody really looked or realized anything was amiss there.
The problem is not that they suddenly hit this performance drop and solved it in some reasonable timeframe, or it was some rare occurrence affecting .01% of users
The problem is that it’s existed for 6 years, affected anyone and everyone, and would have been caught and resolved by basically anyone who attempted to look into the issue with fairly low effort/experience-needed. That is, the problem is that no one bothered to look into it, despite good reason to do so.
> No one looked at this because they didn't know it about.
What are you talking about? Nobody knew about it? Is that a joke?
This long loading issue was posted EVERYWHERE and was the reason so many people gave up on GTA v. It’s simply impossible that they didn’t know the issue existed.
Looking back, I realize it's wrong to assert opinions as facts with no basis at all. I definitely shouldn't have said that when I don't work at Rockstar or know anyone who does.
But I still think people are demonizing too much Rockstar and are trivializing the bug when it's far from obvious. In my opinion, there is a lot of hyperbole statements about this issue when it's more useful to think if your organization could or would catch bugs like this.
Ok, to be fair I am doing to same thing (asserting opinions without facts).
I don't work at Rockstar so it's true I have no idea how this bug was handled.
I just have a hard time believing they didn't know about the bug, and I also have a hard time believing they didn't understand that it was one of the chief complaints from their end users (a simple check of any forum or discussion group is all thats needed).
So, I an only assume short of an actual explanation, that they intentionally sidelined a fix for this bug. Why, is anyone's guess. If you had to ask me, management put all devs onto money making tasks like new items and store support, and fixing quality of life bugs even ones as egregious as this one, was a low priority.
Frankly it might've just gone unnoticed however strange that sounds. Not privy to any details but a probably scenario is probably something like this.
1: Some dev gets tasked with implementing the system, at the time there probably wasn't that many ingame purchases so that 10mb json maybe was a 500k file tops? So going by the O(N2) factor it would've buzzed by in a second (probably faster since being smaller there would've been more cache hits giving another factor or 2). Being at about a second or less they don't even consider optimizing it since it's not noticed among all other assets being loaded.
2: The content/ops teams adds more IAP over the years (it's a datafile so that devs don't have to implement everything), since they're adding big models,etc they don't notice too big changes and probably just chalk it up to bigger models,textures,etc.
3: Engine devs (those most often tasked or obsessive about optimizations) might be running a cut-down server for most testing (not noticing the huge loading time) and probably presume it's other assets if testing in a full env.
So in reality it's probably just an organizational oversight rather than "bad developers", now considering this public "shaming" they received they might actually place some people in an production optimization role for a while and optimize other parts if it's possible (the question is.. will they find it worthwhile to keep up in a longer period?).
meta comment and completely feeling based : sometimes it really seems like companies, politicians, bureaucracies, don't have a process for acknowledging mistakes and/or lack of knowledge (and/or are seemingly not allowed to).
Ah that's because at many companies if you make a mistake you are a mistake.
Which is BS tbh. Everyone makes mistakes.
I think with politicians and political appointees, it's a little different because that's very much a public trust. Nobody deserves to hold political office, they should instead be deserving of it by being ethical and having integrity.
Amen. And avoiding a culture of blame (taking inspiration from the NTSB) makes it much more straightforward to put systemic measures in place which reduce the impact of those mistakes. People still make mistakes at the same rate, but more mistakes get trapped and more negative outcomes can be avoided.
The vitriol of this discussion shows just how difficult it is to establish such a culture. People really want to see mistakes as evidence of moral inferiority.
This is very insightful. If you make a mistake = shameful, you probably aren't going to meaningfully decrease the number of mistakes made, but you will ensure that no one wants to admit they've made one.
And I guarantee anyone who's written code long enough has written plenty of footguns of their own.
I'd say that by talking about mistakes without blame, I think we actually avoid people making mistakes because folks can learn from other's mistakes, and see them as something they could just as easily do, and so internalize the lesson.
> Apologizing just gives fuel for revenge. In this case I think because the harm done was so small, it was economical to apologize.
Yeah, it's always a tradeoff.
I worked at a non-tech fortune 500 for a few years and was able to see the inside of repair/warranty/disputes. These were very high ticket items and the brand was/is very highly regarded for after sales support.
In the employee handbook for repair support staff (the people communicating with clients about the status of their repair/claim) there was a passage similar to the below:
"Apologies MUST NOT acknowledge anything other than the customer's explicit feelings without written acknowledgment of fault from $MANAGER".
I asked a member of that team to decipher WTF that meant, and they showed me an example email. They were only allowed to apologise in the below way:
"Hi bob, we're sorry you feel our product is faulty".
I always found these weasel apologies infuriating in my personal life.
Yep that’s awful. The reason for that rule is probably legal. If your company is taken to court over damages resulting from a faulty product, a written acknowledgment of blame will probably make that worse for the company.
It’s still awful though. I would hesitate to do business with a company that doesn’t take any responsibility for their own products.
Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them." or "I am trying to trick you into thinking I am apologizing even though I'm not".
That's more offensive and insulting and time-wasting than saying nothing.
More charitably, people have an instinctive tendency to bend over backwards to apologize for things they're not responsible for in the interest of cordial relations. This is good if you're casually trying to make friends, but bad if the other party might take you to court. Companies therefore very reasonably explicitly correct for this behaviour.
You're right about it being a tricky attempt at faking an apology.
But the below isn't correct:
> Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them."
I probably didn't word it perfectly in my original comment, but the company did acknowledge mistakes quite often after assessments of the claimed faulty/broken/whatever product (roughly 30% of faulty claims were approved and either repaired or reimbursed from memory). The rejected claims were often taken to court (like 15%) due to the value of products.
I was mostly commenting on the deception of only allowing pseudo-apologies.
Well, this HN hate fest of a discussion certainly bears out that saying! The internet is a terrible place. You will get dragged mercilessly no matter what you do. The incentives are all towards sweeping mistakes under the rug.
There are at least some examples of startups choosing to apologize, and try to make things better in the future.
I think the real advice is “don’t say sorry unless you actually mean it.” People see right through that, because your actions and follow through won’t match your words. But people seem to appreciate sincere apologies that come with substantive corrective action.
as-if that's the worst they could do. Suing the guy, pressing for charges with CFAA or DMCA anti-circumvention. There were far worse ways this could have gone.
It is a common misconception that the victim can "press charges" against someone for a criminal matter. This is entirely false.
Only the prosecuting attorney (someone employed by the government) gets to choose when and how someone is charged with a crime. The victim has no say whatsoever in the matter, and indeed the DA can charge someone even when the victim doesn't want them to, or decline to prosecute even when the victim requests it.
I did not say "press charges". I specifically said "press for charges" deliberately. In that context meaning to attempt to persuade someone (in this case a prosecutor/police/etc) to actually press charges.
Most of the time when people simply use "press charges" this is what they mean. I specifically used "press for charges" to try to avoid such pedantry. I guess I failed.
If shorter load times there translate to higher conversions in the same way this connection is observed by Amazon [0] this guy just made Rockstar a little fortune. Hiring the blogger outright and giving them a VP of Something Something Role would be more adequate.
