Coder Diary #4 -- All Quiet on the West Front?
And the East Front? And in the Empire of the Rising Sun?
No way. No summer letup. We continue working on this as hard as ever.
When I first got access to the Campaign Series code in early March, there were two separate codebases
[*]the inherited Campaign Series codebase
[*]the inherited Modern Wars codebase (a fork of the Campaign Series codebase)
Since early March, I have been working with a copy of the latter inherited Modern Wars codebase
[*]cleaning up, removing legacy cruft
[*]reorganizing
[*]tweaking and fixing to achieve first builds
[*]thereafter, adding new features (windowing the game, hot keys, variable animation speeds, etc.)
So now, we have a third codebase
[*]my Modern Wars codebase (a fork of the inherited Modern Wars codebase)
That's three codebases in total. It gets to be rather hard to manage, not to mention confusing.
It is absolutely vital that I merge the codebases! And the sooner the better. Why so?
[*]It's less work. One edit vs. two (or more).
[*]For better Quality Control. Fix once, in one location, versus twice, in two locations (two or more codebases, the Campaign Series one, and the Modern Wars one).
[*]For greater project flexibility, and more frequent builds.
Where there are multiple codebases
[*]Bugs get fixed in some of the games, but maybe not in others.
[*]Game features maybe get out of sync.
[*]Updates and patches are sequential, not simultaneous.
No, just by throwing the appropriate IDE switches, we want the capability to rebuild the entire Campaign Series of games (including Modern Wars and all future titles) in an hour or two -- the time required for the Microsoft Visual Studio IDE to compile and link the half dozen or games in the current lineup (EF, WF, RS, and the future ME, VW, and others). With all new features and bug fixes etc. in perfect sync.
Diff'ing the inherited Campaign Series and Modern Wars codebases, there are
telemann:/games/matrix/cs/devel # egrep -c "^diff" 20130304_CS_CSM.diff
122
telemann:/games/matrix/cs/devel # wc -l 20130304_CS_CSM.diff
8378 20130304_CS_CSM.diff
122 files, with 8378 lines (actually more like half that) that differ.
And let's not forget: My working Modern Wars codebase differs in some measure from the inherited Modern Wars codebase.
How much greater the pain if I attempt to reconcile all the diffs, merge the several codebases, six months to a year from now (after Modern Wars is released).
Putting it another way:
[*]Merging now, difficult.
[*]Merging six months to a year from now, effectively impossible.
So, by merging now, we avoid the future impossibility. Less pain.
And more gain. By merging now, we have greater flexibility in our release schedules. (Hint, hint. [;)])
Not to mention that future titles in the series after Middle East and Vietnam War -- Korean War, NATO Wars, WWI, etc. -- they will come quicker with a merged code base than not.
Okay, okay, I think you get it: Merge, and merge now.
Merging is a delicate and arduous process. I have done a global diff of the two inherited CS & MW codebases, with output like so:
telemann:/games/matrix/cs/devel # head -n 40 20130304_CS_CSM.diff
Binary files 20130304_CS_MCS_diff/Campaign Series/audiere/bindings/python/pyAudiere.txt and 20130304_CS_MCS_diff/Campaign Series Modern/audiere/bindings/python/pyAudiere.txt differ
Binary files 20130304_CS_MCS_diff/Campaign Series/audiere/doc/dependencies.txt and 20130304_CS_MCS_diff/Campaign Series Modern/audiere/doc/dependencies.txt differ
Only in 20130304_CS_MCS_diff/Campaign Series Modern: Btl_gen.bmp
diff -r 20130304_CS_MCS_diff/Campaign Series/Campaigns Engine/appl.cpp 20130304_CS_MCS_diff/Campaign Series Modern/Campaigns Engine/appl.cpp MERGED
378,379c378,379
< case XRussiaNation : cell = RussiaIconCell; break;
< case XGermanyNation : cell = GermanyIconCell; break;
---
> case XChinaNation : cell = RussiaIconCell; break;
> case XWarsawPactNation : cell = GermanyIconCell; break;
1434a1435,1446
> // V i e w W r e c k C l e a r i n g
> //
> // View the results of Wreck Clearing.
>
> void ApplWindow::ViewWreckClearing (HexCoor hex)
> {
> HasChanged ();
> _map->DelBurning (hex);
> _map->DrawArea (hex);
> }
>
>
1582a1595,1607
> // C l e a r E f f e c t s
> //
> // Do clear effects. April 08
>
> void ApplWindow::ClearEffects (HexCoor hex)
> {
> ClearSound ();
> if (_map->Isometric ())
> _map->AnimateBomb (hex);
> _map->AddBurning (hex);
>
> }
>
1599,1600d1623
< if (_map->Isometric ())
< _map->AnimateMine (hex, dir);
[...]
In my high-tech basement "man cave", I am surrounded by four systems -- Windows, Linux, etc. -- and six monitors, all of them internetworked. I use my swivel chair to turn from one system and display to the next as the need arises. (Crazy, no?)
I am doing this code merge in Linux. Why? For speed and ease of use. On my dual-monitor Linux systems, I typically have a half dozen to a dozen terminal windows open simultaneously side by side by side:
[*]A terminal showing the diffs (the .diff file).
[*]A terminal showing the full text of the first file version.
[*]A terminal showing the full text of the second file version.
[*]A terminal where I am editing (using Emacs) my version of the file.
[*]Other terminals where I can do various Linux commands to aid in the diff'ing and editing process.
I can't imagine doing this work in a single IDE on Windows, without the multiple terminal window and tool support available in Linux. Simply impossible. So I do it in Linux.
When I am done merging the codebases in Linux, I will re-import the unified all-game codebase back into the Windows IDE, and be on my merry coding way again.
What's the merging process like? File by file, I look through the diffs, compare and analyze the diffs against the one version versus the other version and edit in the needed changes in my version. Using copy-and-paste and other techniques as necessary.
What forms do the actual edits take?
They can be easy, like this diff
378,379c378,379
< case XRussiaNation : cell = RussiaIconCell; break;
< case XGermanyNation : cell = GermanyIconCell; break;
---
> case XChinaNation : cell = RussiaIconCell; break;
> case XWarsawPactNation : cell = GermanyIconCell; break;
leading to this merged code
#if defined(EAST_FRONT) || defined(WEST_FRONT) || defined(RISING_SUN)
case XRussiaNation : cell = RussiaIconCell; break;
case XGermanyNation : cell = GermanyIconCell; break;
#elif defined(MIDDLE_EAST) || defined(VIETNAM_WAR) || defined(KOREAN_WAR) || defined(NATO_WAR)
case XChinaNation : cell = RussiaIconCell; break;
case XWarsawPactNation : cell = GermanyIconCell; break;
#endif
Then there are the difficult cases, leading to a merged code tangle such as this one
// MERGE:
#if defined(MIDDLE_EAST) || defined(VIETNAM_WAR) || defined(KOREAN_WAR) || defined(NATO_WAR) || defined(WEST_FRONT) || defined(RISING_SUN)
int region;
sscanf (buffer, "%*d %*d %*d %*d %*d %*d %*d %*d %*d %*d %*d "
"%*d %*d %*d %*d %*d %*d %*d "
"%*d %*d %*d %*d %*d %*d %*d %*d %*d "
"%*d %*d %*d %*d %*d %*d %*d %*d %*d %*d %*d"
#if defined(VIETNAM_WAR) || defined(KOREAN_WAR) || defined(RISING_SUN)
" %*d %*d %*d %*d %*d %*d %*d"
#endif
" %d", ®ion);
return (area == region) ? 2 : 1;
#elif defined(EAST_FRONT)
return 2;
#endif
Yikes!!
I could show you even more convoluted examples besides.
In this harrowing process of wrapping '#if ... #endif' preprocessor directives around one snippet of code from another, it is quite easy to
[*]Miss a file.
[*]Miss a set of diffs.
[*]Miss a line.
[*]Misinterpret the diffs.
[*]Make an editing mistake.
and thereby apply the inappropriate edits to my codebase, or maybe miss some edits entirely.
So I can't be rushed; I have to take my time. Rushing through this process might introduce bugs it could take me hours and even days to figure out later (perhaps at a much later date, when the code merge is only hazily remembered).
By now, I am more than half way through the code merge process. I anticipate finishing up by sometime next week. Then I re-import back into the MS Visual Studio IDE, and I get to see if the whole thing blows up! [:D]
Tedious, boring, slow-going work -- but no less important than the sexy stuff -- adding new features, redesigning the UI, etc.
Another glimpse behind the scenes:
A couple of weeks ago, it was reported that in one of the Vietnam War scenarios, a certain mortar unit was unable to fire. Naturally fingers pointed in my direction. (Understandable. No problem.)
As it happens, the problem was not in the game code; it was in the game data. Specifically, (North Vietnamese) units referenced in the .scn file were missing from the counterpart .org file. An easy mistake to make. Scenario designers make mistakes, I (certainly!) make mistakes, we all make mistakes. This stuff is so mindnumbingly technical, and our brains & eyes are so humanly frail, how can we not make mistakes?
The game data can be as obscure, complex, and confusing as the CS C++ code. It takes two to tango. It takes both game engine and data to make a game. We can have the most perfect, bug-free game code. But if the game data files are buggy, we have bugs. And vice versa.
And once again, I turn to Linux (and its Windows cousin, Cygwin).
Here is a "rather" complex Linux command line to discover (North Vietnamese) units referenced in a .scn file missing from the counterpart .org file:
$ S="Binh_An_1968"; egrep "[[:space:]]21" $S.scn | awk '$4>20000 {print $4}; $1>20000 {print $1}' | sort | uniq | sort -n > junk1; egrep "P21" $S.org | awk '{print $1}' | sed '1,$ s/^P//' | sort | uniq | sort -n > junk2; cat junk1 junk2 junk2 | sort | uniq -u
[nil]
WTF?! [&:]
[:D]
Let's break that down.
[*]S="Binh_An_1968" sets a variable S, for the scenario name
[*]; separates commands, one from the next
[*]egrep "[[:space:]]21" $S.scn | awk '$4>20000 {print $4}; $1>20000 {print $1}' | sort | uniq | sort -n > junk1 gets the unit list from the .scn file -- but without the instance counts, just the unit list -- and captures that list to a file, junk1
[*]egrep "P21" $S.org | awk '{print $1}' | sed '1,$ s/^P//' | sort | uniq | sort -n > junk2 gets the unit list from the .org file (where sed '1,$ s/^P//' strips the preceding P character), and captures that list to a second file, junk2
[*]cat junk1 junk2 junk2 outputs the junk1 file once and the junk2 file twice in succession
[*]sort | uniq -u sorts the units, then shows only the units that uniquely appear once in the grand sorted list; if a unit only appears once, it can only only be in the junk1 file, not in the junk2 file (because all units in the latter file are shown twice) -- follow?
Because that command above yields no output, we can confidently say that, in the Binh_An_1968 scenario, every (North Vietnamese) unit in the .scn file is present in the counterpart .org file.
Let's see what that problematic Hiep_Hoa_1963 scenario (the one with the mortar fire failure bug) shows:
$ S="Hiep_Hoa_1963"; egrep "[[:space:]]21" $S.scn | awk '$4>20000 {print $4}; $1>20000 {print $1}' | sort | uniq | sort -n > junk1; egrep "P21" $S.org | awk '{print $1}' | sed '1,$ s/^P//' | sort | uniq | sort -n > junk2; cat junk1 junk2 junk2 | sort | uniq -u
21049
21107
21201
21402
Those four listed unit codes -- they are present in the .scn file but absent from the counterpart .org file. The 21107 unit code corresponds to the no-fire mortar unit. Adding it (and the other units) to the scenario .org file fixed the bug. With the .scn & .org files back in perfect sync, the mortar unit was able to fire again.
Got that? [:D]
Can we automate checking
[*]all units (North Vietnamese, US, ARVN, ..., Israeli, Egyptian, ..., German, ... Russian, ... Japanese, ...)
across
[*]all scenarios
across
[*]all games (EF, WF, RS, ME, VN, ...)
?
But of course. Why, yes we can! [:)]
I have begun developing a set of Cygwin-based tools for
[*]Checking scenario files for internal consistency (so as to prevent, for example, the no-fire unit bug described above).
[*]Checking all game data files for all manner of errors.
[*]Reporting game data in easy-to-read fashion (e.g., csoobrpt.pl).
[*]Converting hexadecimal C++ unit attribute codes to their decimal .oob file equivalents, and vice versa (csflags.pl).
[*]Other purposes and uses.
I call my toolkit "CSlint":
With the CSlint toolkit, there are all manner of things we can QA check (and all manner of things we can make easier).What is CSlint? Akin to the well-known C programming language utility lint. ("Lint: A Unix C language processor which carries out more thorough checks on the code than is usual with C compilers. Lint is named after the bits of fluff it supposedly picks from programs." See here.) Similarly, CSlint is a toolkit of programs designed to discover and report bugs, problems, glitches, anomalies in the Campaign Series game data files.
And because our computers are so fast, we can run these checks in just a few minutes, across all games. As often as we wish! Create a scenario, edit a unit file, run the QA tools on the spot. Real-time QA, as we develop. Not post-release. [:-]
Bless their hearts, other members of the Development Team have taken the leap of faith with me, installed Cygwin on their systems, and begun learning some of this Linux/Cygwin wizardry -- magic that will help to discover and weed out mistakes and to make for a much better, less buggy game.
It is entirely within the realm of possibility that we will release these tools, and tutor their use (also Linux/Cygwin techniques), to the general player/modder community. So as you design your own scenarios, you too can benefit from some of the design, management, and Quality Assurance tools that we the Development Team are using.
But that's for the future. For now, despite the apparent quiet, rest assured: all things are not quiet on the CS development front. Far, far from it!
Until the next time ...
