Matrix Games Forums

Forums  Register  Login  Photo Gallery  Member List  Search  Calendars  FAQ 

My Profile  Inbox  Address Book  My Subscription  My Forums  Log Out

Coder Diary #28 -- Is It Soup Yet?

 
View related threads: (in this forum | in all forums)

Logged in as: Guest
Users viewing this topic: none
  Printable Version
All Forums >> [Current Games From Matrix.] >> [World War II] >> John Tiller's Campaign Series >> Coder Diary #28 -- Is It Soup Yet? Page: [1]
Login
Message << Older Topic   Newer Topic >>
Coder Diary #28 -- Is It Soup Yet? - 5/25/2015 3:49:54 PM   
berto


Posts: 19699
Joined: 3/13/2002
From: metro Chicago, Illinois, USA
Status: offline

Coder Diary #28 -- Is It Soup Yet?, Part 1



Long time, no write. Time for a new Coder Diary! [a very long one]

quote:

Everyone therefore who hears these words of mine, and does them, I will liken him to a wise man, who built his house on a rock. The rain came down, the floods came, and the winds blew, and beat on that house; and it didn't fall, for it was founded on the rock. Everyone who hears these words of mine, and doesn't do them will be like a foolish man, who built his house on the sand. The rain came down, the floods came, and the winds blew, and beat on that house; and it fell—and great was its fall.

— Matthew 7:24–27, World English Bible

Like the wise man, we want to re-build the Campaign Series on strong foundations. So much of the legacy code and data was built on shifting sand!

The game's -- note I wrote "game's" (singular), not "games'" (plural), referring to the original game in the series, East Front -- original code was designed and organized around a single conflict, the East Front. When the Campaign Series was expanded to include West Front, then Rising Sun, code and data were extended in a rather ad hoc fashion. A while later, there was Divided Ground, and still more code and data jerry-rigging. Take-Two acquired Talonsoft, which then died, then ... eventually Matrix acquired rights to the Campaign Series, and work was begun on the Modern War Series, including Middle East and Vietnam.

All of this Series expansion and passing of code and data from one company to another, from one dev team to the next. And every step of the way, new games and new data and new features and new graphics and ... all built atop the original foundations which were never designed to support so many different games and so many cool new features and special situations. Nobody ever stopped to assess: Where is this all heading?

By the time I inherited the code as Campaign Series Lead Coder, what a mess it all was! For the sake of our sanity, for the sake of bug-free code and Quality Assurance (QA), we have had to rethink everything, redesign and redo not a little.

So much of my time over the past two years and more has been devoted to tearing down and building back up the original games (EF, WF & RS) and the newer games (ME, VN & ...) on sturdier and more rational and more extensible foundations. Without toppling the older games in the process. Quite a challenge!


File/Folder Reorganization

Here's something that really needed to be fixed: the games' file/folder organization. Basically, there really wasn't any. Everything was simply dumped into a single file folder. So many files, so hard to wade through it all!

For example, in East Front 2.02, in the East Front folder, there are



Robert@roberto /cygdrive/f/Games/Matrix Games/John Tiller's Campaign Series 2.02 SUPPORT/East Front
$ ls -1 | wc -l
7449



nearly 7,500 files! All lumped together in just one folder!

Here are the first few files in a very long list of Rising Sun's 3,853 files:



Note how save game BTL files are mixed among the game's Application files, and SCN files, and DAT, ORG, OBX, OOB, BMP, and ... What a mess!

In Middle East, all future games, and eventually soon also the original JTCS games, we have reorganized the files into categorical folders thusly:



Here are the save files, in one tidy package (folder):



As developers, and eventually you too as modders and players, we find it so much easier to find our way around the many, many game files. File folders are there for a purpose -- to create sense and order. Let's make use of them!

The File/Folder Reorganization Project -- a very big step forward. But it came at a price. The legacy code expected to find everything in the same place, the one top-level game folder. Moving files into subfolders broke just about everything. It was a long and tedious process hunting many dozens of file references in the code and revising them just right. LCGs, for instance, were broken for several months, because in dozens of places the code was looking for this or that file and, not finding it, would typically crash the game. I devoted an arduous two weeks to identifying and fixing the broken file references to fix LCGs alone.

A long and hard process. But was it worth it? You bet!

[to be continued]

_____________________________

Post #: 1
RE: Coder Diary #28 -- Is It Soup Yet? - 5/25/2015 3:51:20 PM   
berto


Posts: 19699
Joined: 3/13/2002
From: metro Chicago, Illinois, USA
Status: offline

Coder Diary #28 -- Is It Soup Yet?, Part 2



Terrain Graphics BMP Reorganization

Orginally, there was just East Front. With its own set of graphics BMPs. Then there came West Front, sharing some files in common with East Front, but WF too having its own set of graphics BMPs unique to that game. Then along came Rising Sun ... and eventually also Middle East, Vietnam, ...

Here too, so many games, so many files, so much confusion!

How were the differences in game BMPs handled? One way was by way of file prefixes, for example:



Robert@roberto /cygdrive/f/Games/Matrix Games/John Tiller's Campaign Series 2.02 SUPPORT/Rising Sun
$ ls -1 *[Tt]errain*
Des-Terrain1d.bmp
Des-Terrain2d.bmp
Des-Terrain3d.bmp
Des-Terrain4d.bmp
Des-Terrain5d.bmp
Med-SnowTerrain3d.bmp
Med-SnowTerrain4d.bmp
Med-SnowTerrain5d.bmp
Med-Terrain1d.bmp
Med-Terrain2d.bmp
Med-Terrain3d.bmp
Med-Terrain4d.bmp
Med-Terrain5d.bmp
Nwe-SnowTerrain3d.BMP
Nwe-SnowTerrain4d.BMP
nwe-SnowTerrain5d.BMP
Nwe-terrain1d.bmp
Nwe-terrain2d.bmp
Nwe-Terrain3d.BMP
Nwe-Terrain4d.BMP
Nwe-Terrain5d.BMP
SnowTerrain3d.bmp
SnowTerrain4d.bmp
SnowTerrain5d.bmp
Tem-SnowTerrain3d.bmp
Tem-SnowTerrain4d.bmp
Tem-SnowTerrain5d.bmp
Tem-Terrain1d.bmp
Tem-Terrain2d.bmp
Tem-Terrain3d.bmp
Tem-Terrain4d.bmp
Tem-Terrain5d.bmp
Terrain1d.bmp
Terrain2d.bmp
Terrain3d.bmp
Terrain4d.bmp
Terrain5d.bmp
Tro-Terrain1d.bmp
Tro-Terrain2d.bmp
Tro-Terrain3d.bmp
Tro-Terrain4d.bmp
Tro-Terrain5d.bmp



Yikes! So many files! Just for terrains! Here's the thing: So many of those files were much the same, sometimes identical, often differing by just a graphics "cell" or several.

Multiply those files across so many different games, and again we have quite a tangled mess.



Robert@roberto /cygdrive/f/Games/Matrix Games/John Tiller's Campaign Series 2.02 SUPPORT
$ find . -print | egrep -i "terrain.*bmp" | wc -l
99



In the JTCS games, as many as 99 terrain BMP files!

Here is the terrain2d.bmp file from EF 2.02:



Some terrain unique to East Front, most of it no different from terrains in the other games.

A better way:

  • Rather than have many different Des- this and Tro -that variants, have a single common terrain file for all games in the Series -- JTCS, but also any new games (ME etc.).
  • In the code, reference not many different files, rather different cell offsets in the one common terrain file, using the game identifier as a handle.
  • Separate out vegetation into its own separate file.
  • Create empty spaces, for future expansion.

    Here for example is the reorganized terrain2d.bmp file:



    You might note the:

  • Empty rows (due mainly to removal of the vegetation cells).
  • At the bottom, Middle East buildings, but also Vietnam buildings, and blank rows for eventual incorporation of the JTCS buildings.

    Here is the new vegetation2d.bmp file:



    Vegetation specific to Middle East, but to Vietnam (and eventually also the other games) too.

    Did all of this terrain BMP file and cell reorganization break stuff? To be sure! More graphics glitches and crashes. More days and days of fixing things and getting it all right. But in the end, and especially looking forward, entirely worth the effort!

    [to be continued]

    _____________________________


    (in reply to berto)
  • Post #: 2
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/25/2015 3:52:58 PM   
    berto


    Posts: 19699
    Joined: 3/13/2002
    From: metro Chicago, Illinois, USA
    Status: offline

    Coder Diary #28 -- Is It Soup Yet?, Part 3



    Unit Graphics BMP Reorganization

    That's terrains. What about unit icons?



    An even worse mess. Here, for example, the icons2d.bmp file from East Front 2.02:



    Do you see?

  • There is no rhyme or reason to the order and sequencing.
  • Graphics and icons of one type are freely mixed in among others.
  • NATO icons are scattered all over the place.
  • No free rows, no room to expand.

    Worse still (can it get any worse?), icons for Russia and Germany were special handled differently from other nations from later games in the Series. Trust me, you don't want to see the kludgey legacy code to handle this tangle!

    Here is the redone icons2d.bmp -- one new icons2d.bmp file common to all games, past, present & future:



    Much better organized, much cleaner, plenty of free spaces with room to expand.

    Where did the unit graphics go? Into expanded, nation-specific icons2d##.bmp files. Here for example is the Israeli icons2d22.bmp from Middle East:



    So much better organized, so much easier to work with. [Right, Warhorse?]

    Here too, as the icons BMP files evolved and things were rearranged and separated out into different files (other files also not mentioned here), stuff broke, games crashed. More days and weeks of debugging, fixing, testing. Time consuming, but again oh so totally worth it.

    Aside from Quality Assurance, an overriding aim here is to reduce the number of files, across all games, to an absolute minimum. Maximum common file sharing. Manage all the differences in the code, not in the proliferation of data files. Trust me, it's much easier that way.

    [to be continued]

    < Message edited by berto -- 5/25/2015 4:53:27 PM >


    _____________________________


    (in reply to berto)
  • Post #: 3
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/25/2015 3:56:10 PM   
    berto


    Posts: 19699
    Joined: 3/13/2002
    From: metro Chicago, Illinois, USA
    Status: offline

    Coder Diary #28 -- Is It Soup Yet?, Part 4



    Data File Reorganization

    I have mentioned the UID Slots Expansion Project in a previous Coder Diary #24.

    Weeks of effort and delay there also. But again, so totally needed.

    New features, new data fields, new bugs!

    For example, in the legacy data file design, the "magic number" -1 was often used as a guidepost to signify the beginning or end of this or that data file section. Okay, but what happens when you add some new feature requiring some new data field that just so happens to permit -1 as a valid value number? The -1 value is mistaken as a section file marker, and: bugs, crashes!

    We can no longer rely on -1 as data file subsection markers. Instead, much like the legacy



    ----- PEM Header -----

    ...

    ----- PEM Trailer -----



    section markers in the BTE files, we have a new system of data file section markers:



    // data file markers
    const char DataHeader[] = "[-------------------------------- data header --------------------------------]"; // at very beginning, first section
    const char DataSection[] = "[-------------------------------- data section -------------------------------]"; // generic section marker
    const char DataFooter[] = "[-------------------------------- data footer --------------------------------]"; // last section
    const char DataTrailer[] = "[-------------------------------- data trailer -------------------------------]"; // at very end
    const char UnitData[] = "[--------------------------------- unit data ---------------------------------]"; // functionally equivalent to DataSection



    Used, for example, in the .scn files like so:



    [-------------------------------- data header --------------------------------]
    20
    Bootcamp One
    0 0 0 1-8[d]
    0 0 0 0 1-8[5]
    0 0 0 0 80 80 100 100
    0
    -25 25 0 50 8
    50 50 0 0
    0 0 0 0 0
    0 0 0 0 0
    |JJO><HK:khH<K
    |JJO><HK:khJMB
    3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
    3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
    0 0
    0 0 0 0
    0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
    [--------------------------------- unit data ---------------------------------]
    0 3 20 P92040 434 2 2 2 100 9 0 0 0 3 3 3 3 0 0 -1 -1
    0 4 19 P92040 432 2 2 2 100 9 0 0 0 3 3 3 3 0 0 -1 -1
    0 4 19 P92040 433 2 16 2 100 9 0 0 0 3 3 3 3 0 0 -1 -1
    0 5 19 P94002 638 2 2 2 100 9 0 0 0 3 3 3 3 0 0 -1 -1
    0 5 19 P92040 431 2 1 2 100 9 0 0 0 3 3 3 3 0 0 -1 -1
    0 10 8 P288808 595 4 4 4 100 6 0 16384 0 3 3 3 3 0 0 -1 -1
    0 10 9 P282018 599 4 4 1 100 6 0 16384 0 3 3 3 3 0 0 -1 -1
    0 10 12 P282037 355 4 4 3 100 5 0 16384 0 3 3 3 3 0 0 -1 -1
    0 10 14 P288808 593 4 4 4 100 6 0 16384 0 3 3 3 3 0 0 -1 -1
    5 11 10 25 28
    5 12 7 5 28
    5 12 12 5 28
    0 13 10 P288808 594 32 32 4 100 6 0 16384 0 3 3 3 3 0 0 -1 -1
    0 13 11 P282037 354 16 16 3 100 5 0 16384 0 3 3 3 3 0 0 -1 -1
    [-------------------------------- data footer --------------------------------]
    0 -1
    1
    Jason Petho
    15 August, 1963
    [Habilayn, Radfan, Yemen]: [SIDE A][FIC][CSL]: Welcome to Bootcamp
    One. This scenario is intended to be played as the United Kingdom.
    It is recommended to open the Bootcamp One document found in
    the manuals folder. It will take you through a step by step process
    of how to play the game and things to think about when you are
    playing a turn. Good luck! [ALL][1.00][NONE]



    The sharp-eyed modders among you might spy some unfamiliar, new data fields. We will explain them eventually if and when needed.

    File/folder reorganization, graphics file reorganization, UID slots expansion, data file redesign, new data fields, field resequencing, etc., etc...

    ... and new bugs, and new crashes ... requiring hours, and days, and weeks of debugging, fixing, and testing.

    Did we revise and edit the data files by hand? No! Much use of Perl scripts to revamp the many, many hundreds of game data files.

    And much effective use of an expanded (more effort! more delay! am I repeating myself? ) CSlint to vet and QA the whole mass of them.


    Vietnam

    Many weeks of delays and tedious foundational reworking.

    Not to mention: After JTCS 2.02's release last summer, we turned our attentions in full to the Middle East game. Leaving Vietnam to languish.



    Not good, because if left mostly idle the Dev Team's Vietnam expert, and worse, Vietnam was getting so out of sync with Middle East that the last time we had a working Vietnam game was last September.

    So we devoted another couple of weeks to, setting aside Middle East work, bringing Vietnam back up to Middle East's level. As you know, we have a common codebase, such that, with a single set of common code files, and by means of the appropriate IDE commands and switches, we can crank out all of the Campaign Series games, old and new.

    That's in theory. In practice, we did have a common codebase among all games, until -- for technical reasons I won't go into here and now -- ME (and later VN) development broke apart from JTCS development. For the time being, from last summer until now, we have been unable to rebuild the older JTCS games using the new, "modern" ME/VN codebase. After ME ships in September, we plan to rectify that. Beginning in September, we plan to take several weeks to bring the older JTCS games back into alignment. Never to go out of realignment ever again, not if we can help it.

    We have not been focusing just on Middle East. We have been thinking strategically. There are other games in the Campaign Series future, updated old ones (like the proposed JTCS "mega patch" planned for late 2016), and after Middle East also Vietnam (first half of 2016), and beyond.


    Excuses, Excuses...

    We know that the Campaign Series player community is frustrated by all the delay. We want our Middle East, and we want it now! If it's any comfort, we do too!

    But now you know the many reasons for the delay. (Oh, there are other reasons, most especially new features -- topics for future Coder Diaries.) Rome Wasn't Built in a Day. And neither will the Campaign Series be rebuilt in a day, or a week, or a month, or even just a single year either. We beg your patienece. It takes time. Lots of time. But not too much longer.

    Until the next time...

    < Message edited by berto -- 5/25/2015 5:37:59 PM >


    _____________________________


    (in reply to berto)
    Post #: 4
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/25/2015 7:26:01 PM   
    wings7


    Posts: 4608
    Joined: 8/11/2003
    From: Phoenix, Arizona
    Status: offline
    The soup is cooking, when it is finished it will taste great! Thank's for all your hard work!

    Patrick

    _____________________________

    Please come and join and befriend me at the great Steam portal! There are quite a few Matrix/Slitherine players on Steam! My member page: http://steamcommunity.com/profiles/76561197988402427

    (in reply to berto)
    Post #: 5
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/25/2015 11:52:10 PM   
    scottintacoma

     

    Posts: 192
    Joined: 1/25/2008
    Status: offline
    Jason, Berto, and the whole team.

    I appreciate your hard work and dedication. Cant wait for ME to come out.


    (in reply to wings7)
    Post #: 6
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/26/2015 4:42:13 AM   
    76mm


    Posts: 4150
    Joined: 5/2/2004
    From: Washington, DC
    Status: offline
    Respect. Wish more games could be revived like this, definitely buying this one to support you guys (and maybe I'll even like a scenario or two )

    < Message edited by 76mm -- 5/27/2015 6:07:29 AM >

    (in reply to scottintacoma)
    Post #: 7
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/26/2015 9:57:43 PM   
    Warhorse


    Posts: 5698
    Joined: 5/12/2000
    From: Birdsboro, PA, USA
    Status: offline
    Guys, there is sooo much going on with this game, we are certain it will bring many hours of enjoyment!!

    _____________________________

    Mike Amos

    Meine Ehre heißt Treue
    www.cslegion.com

    (in reply to 76mm)
    Post #: 8
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/26/2015 11:58:25 PM   
    Plodder


    Posts: 998
    Joined: 7/28/2003
    From: New Zealand
    Status: offline
    I have a question for berto, How difficult would it be to change the graphics from bmp-based to png-based? Pngs give much better transparency effects and don't they also use less memory as well?

    edit: Forgot to add: awesome work on keeping this baby alive and modern.

    < Message edited by The Plodder -- 5/27/2015 1:00:35 AM >


    _____________________________

    Gen. Montgomery: "Your men don't salute much."
    Gen. Freyberg: "Well, if you wave at them they'll usually wave back."

    (in reply to Warhorse)
    Post #: 9
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/27/2015 12:04:55 AM   
    Jason Petho


    Posts: 14080
    Joined: 6/22/2004
    From: Terrace, BC, Canada
    Status: offline
    It's too big a project to tackle at the moment with our pending schedules.

    It is on the internal Wish List for the future though.

    Jason Petho

    _____________________________


    (in reply to Plodder)
    Post #: 10
    RE: Coder Diary #28 -- Is It Soup Yet? - 5/27/2015 6:40:55 AM   
    Plodder


    Posts: 998
    Joined: 7/28/2003
    From: New Zealand
    Status: offline

    quote:

    ORIGINAL: Jason Petho

    It's too big a project to tackle at the moment with our pending schedules.

    It is on the internal Wish List for the future though.

    Jason Petho


    Cool. Glad to see it's on List.

    _____________________________

    Gen. Montgomery: "Your men don't salute much."
    Gen. Freyberg: "Well, if you wave at them they'll usually wave back."

    (in reply to Jason Petho)
    Post #: 11
    Page:   [1]
    All Forums >> [Current Games From Matrix.] >> [World War II] >> John Tiller's Campaign Series >> Coder Diary #28 -- Is It Soup Yet? Page: [1]
    Jump to:





    New Messages No New Messages
    Hot Topic w/ New Messages Hot Topic w/o New Messages
    Locked w/ New Messages Locked w/o New Messages
     Post New Thread
     Reply to Message
     Post New Poll
     Submit Vote
     Delete My Own Post
     Delete My Own Thread
     Rate Posts


    Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI

    0.132