In many respects, last week seemed like a disaster. Perhaps Murphy's Law had recently been amended to include special consideration for indie game development. Or maybe I am just being a bit dramatic.
Last week started as a continuation from the week before. By that I mean the asset pipeline build was still running. The pipeline took nearly 7 very long days to complete. I had expected a relatively long build process since I had created quite a large landscape... but 7 days of continuous processing? I had plans for 7 more similarly sized landscapes, and any 3rd grader can figure out the math on that. But of course that was only the beginning.
The build log was beyond insane. Over 1.7 million lines in the log file. The log file told a tragic tale, one of many sorrows... but failed to properly connect cause and effect. Upon reviewing the error riddled log, I found some gaping holes: some of the processes crashed and failed to write to the log. After a week's time, I had more errors than there are leaves on the tree in my front yard, and nothing to determine the reasons for the bulk of them.
In order to figure out what was going on, I decided I needed to untangle the build pipeline. The only way I was going to find the causes of the errors was to tear down the pipeline into individual projects and run each one separately and watch the console output. This resulted in 26 individual projects with 7 phases each. I ran the first 3 phases for each project, one at a time. I studied the logs and console outputs, and fixed whatever issues I could find, and repeated each process until I was satisfied.
The first 24 projects went relatively fast, taking only a few days to get through. The 25th project, however, was dealing with exporting and converting my new landscape. This process had me extremely frustrated, pulling my hair out, and occasionally screaming like a crazy woman at my computer. Maybe not "like" a crazy woman... more like 'definitely insane, institutionalize me now'. Perhaps even saying "occasionally" is sugar coating things...
First there were a few missing files and some incorrect settings in some config files that caused early processes to crash, thereby causing a chain reaction throughout the rest of the scripts. Then there was a required folder that failed to be created during the process initialization, which resulted in a mass of file write errors ( which of course led to another mass of failures in reading those non-existent files. ) Getting my hopes up, I watched as one of the main output processes - which had never run successfully before - started up. I watched the little line of dots grow longer... and longer... and then boom. I looked at my task manager. The process had eaten 8GB of ram, and then died.
Apparently I hit an architectural ceiling. I started to think about the amount of time spent on this thus far, including the time spent building the landscape. I felt that I was floating above a fork in the road, one leading to pure madness and the other leading to despair. Friday night, I drifted off toward despair.
Saturday was spent with family, and ended with yummy mexican food. Sunday I spent being as lazy as possible, wearing as little as possible, with the AC as cold as possible and I slept as much as I could. Ah the glorious sleep, complete with a buffet of dreams!
Monday I woke up refreshed. I built a new smaller landscape, one that by my rough calculations should get through the super RAM hungry process. Unfortunately that did not work out. The build process never fully compiled all of the parts of the original landscape ( for some unknown reasons, probably size related also ) and had therefore crashed way later than it should have. This time more pieces of the smaller landscape got compiled correctly, and the output process hit the ceiling and crashed quite a bit earlier.
I wonder what exactly the game engine's developers consider "large" when they say the engine "Supports large open terrains." Actually, I have yet to find a definitive answer to that. I was working with the limits imposed by the world editor, but apparently those limits are beyond the actual capabilities of the pipeline.
Next, I rebuilt the landscape yet again, this time around 1/4 the size of the original. Surely, this system can handle this, right? I very much dislike the idea of iterating in a trial and error fashion to find the largest size the system can cope with, but I do not seem to have much of a choice. This time though, the insane RAM crunch came and went without a glitch. Finally!
Oh but wait, theres a few more processes that never ran successfully before...? Will there be more hair pulling? More screaming? Will I end up mentally beaten, crawling into bed whimpering? Or will I emerge victorious, floating on a fluffy cloud of accomplishment?
I guess we'll all find out, next time...
the Undead Dev