Why Your Patches Are Bigger Than They Need to Be
When we build AAA games we validate the things everyone agrees are important: Game design, narrative and game systems. The work tied to launch and post-launch waits until the end, and that order can cost you a great deal.
We asked Jean-Francois Goulet, our Studio Technical Director, to explain one of the systems hiding in that gap: game data determinism, what breaks it, and how to prove your own cook is clean.
20+ years on the systems games ship on: engine and tools programming, build and cook pipelines, patching and performance across PS5, Xbox Series, PS4, Xbox One and PC. LinkedIn
What we cover
The pattern
What’s the thing that goes wrong on AAA patches that nobody warns you about?
What waits is everything associated with launch and post-launch: DLC, patching and data chunking.
It is a strategy that may unfortunately cost you a lot, and it can come with real consequences for our players. From artificially longer intro sections to leave enough time to download install chunks, to larger patches to download, or even the failure to patch specific data without fully re-authoring it. These are just a few examples of the consequences players may have to incur due to improper planning when it comes to launch and post-launch.
One of the things that many developers are not aware of is the importance of ensuring your game data is deterministic. Not taking care of it will not generally result in your inability to launch or patch your game. But the impact of not caring about it may have serious implications for players once your game is live, and coming back from that near launch, or even during post-launch, can prove very difficult. Let us take a closer look at what game data determinism is, why it matters and some of the tools at our disposal to deal with it.
The definition
What is game data determinism?
Game data determinism can be defined as follows: for a fixed set of inputs (raw data, build parameters such as the target platform, the compression algorithm, and so on) you always get the same exact output. In other words, in a much more simplified explanation, you can guarantee that 2 plus 2 will always be equal to 4. Sounds easy in theory. Unfortunately in practice, it is a bit more complicated.
The causes
Why is my game data non-deterministic?
There are plenty of reasons why game data sometimes becomes non-deterministic. Let’s take a look at some of them.
Identifiers generated during the cook
Sometimes we add our own tech stack on top of the engine, and some data has to be generated during the process of building the game data. Unique IDs are a common case. Programmers use GUIDs, a 128 bit structure designed so that at the moment of generating one, nobody else in the world can generate the same value. That is exactly what makes a GUID useful for identity, and exactly what makes it a problem inside a cook. Generated during the build, it outputs a different result every run even though the processed asset and the build parameters have not changed.
Load order dependencies
Code invoked during the cook can behave differently because it has load order dependencies. Take Asset A, which depends on Asset B. The code processing Asset A may initialise an internal structure and save it to disk conditionally on whether Asset B is present in memory at that moment. If B is already loaded, everything is fine. If by some unfortunate coincidence it is not, the result for Asset A is different, and a non-deterministic pattern enters the data.
Why would B sometimes be in memory and sometimes not? In the Unreal Editor the cooker arbitrarily runs the garbage collector during cooking to keep memory usage low. If Asset A only keeps a soft reference to Asset B, and nothing else keeps B alive through natural references, that pass may collect it. The GC runs on heuristics that are not deterministic, so the moment it runs differs from one cook to another.
These are just a few simple examples, and as you can imagine there are plenty more. Every game is unique in the way its data is put together, and may be dealing with its own tech stack, which adds another layer of complexity. It is therefore important to take a look at cook determinism long enough before thinking about launch and post-launch.
The cost
Why does game data determinism matter?
Game data determinism, as we have seen above, is the ability to guarantee constant output results from the cooking process given the very same inputs. In other words, it is the ability as a developer to create one or more builds at a specific changelist and guarantee that the result of the cooking process will be the exact same.
This assumption is important because patch tools work by comparing the results of 2 different builds and keeping only the differences between them. That is what constitutes the patch, and it is how patch sizes stay minimal.
Smaller patches save players download time and, more importantly, local storage, a rare and pricier than ever commodity. There is nothing more infuriating to a player who wants to spend time in your game than a large patch to download first. Suddenly the 60 minutes they had for your game melts to 50, to 40, or less. That can be enough for them to reconsider which game they play that day.
But there is more to it than just download size.
Install size and disk usage matter just as much, and both are costly to players. Bigger games mean bigger drives, and that is a direct cost to the player at a time when SSDs are very expensive.
Behaviour can vary based on the results of the build process. That is a scenario no developer wants to face, and it produces complicated and costly bugs to chase.
Non-deterministic data plays badly with internal caches, invalidating results from one run to another and reducing how much the cache helps during development.
The method
How do you get a live game’s data to behave the same way every time it’s cooked?
If you happen to work with Unreal Engine, you are in luck. Epic provides data determinism check tools with detailed reports that help figure out where and why non-deterministic data is introduced.
- 01Cook the game entirely from a clean state. That produces your reference output.
- 02Run a second cook using the exact same arguments, plus -diffonly.
- 03That run leaves the previously cooked packages on disk instead of deleting them, compares the cooked data in memory against them byte to byte, and reports any difference it finds.
- 04Reach a state with 0 determinism issues as early as possible, then keep running the validation frequently in CI and fix each new pattern the moment it appears.
Acting immediately is the part that matters. This way it is much easier to detect which change was responsible for introducing the non-deterministic pattern, and to act accordingly.
In conclusion
Game data determinism matters for developers and for players alike. The tools and the reports already exist for your project to use. We strongly recommend automating this validation earlier rather than later, in order to help identify culprit changes that may introduce non-deterministic patterns. With all of that, your game project should be in the best situation to efficiently produce optimal patches and make your players happy.
How Devoted Studios works on this
Cook pipelines, patching and build health are part of what our game engineering services teams take on inside client projects. Our engineers work as part of the client’s team, on the client’s pipeline, to the client’s standards.
Engineering work we have shipped
Arc Raiders
Co-development, UI engineering, gameplay features, performance
Risk of Rain 2
Ported to 5 platforms at once, certification passed on first submission
FNAF: Secret of the Mimic
Co-development, porting, post-launch DLC, UI and gameplay engineering
Invincible VS
Co-development partner with Skybound
Palia
Engineering co-development
If your cook is producing results you cannot explain, or launch is close and nobody has looked at patch size yet, that conversation is worth having early rather than late.