Submarine operations - the first year

[tiemanjw]

So after getting through the first year, I wanted to look at how successful submarine operations were (mine and the computers), and how to get improved performance in the next year.
Some background - this is my first game, and I'm still learning, so I made plenty of errors in the early game. One is that I did not change my sub commanders in the first year. I am playing the Dec 8 quiet China campaign. I wrote a program to go through the combat reports and pull out all the sub related ones. It creates a spreadsheet of several statistics that may (or may not) be relevant to submarine successes. I then copied the spreadsheet into excel, separated the boats into classes (Fleet, S, O, Dutch, Brit, I, RO, midget). From this I did some additional processing to try to identify trends. I also copied the skipper data from the editor for all the Fleet boats to see if any correlations existed to some skipper attributes. Then I plotted attacks on a 2d graph to figure out if location had anything to do with success or failure.

First to define terms:
Total engagements - number of times the submarine shows up in a combat report (excluding in port airstrikes)
Successful attacks - number of times at least one torpedo detonated against the hull. Note that gun only surface attacks would NOT be scored as a successful attack
Spotted - the number of times the sub was spotted by an escort prior to an attack
Surface attacks - number of surface attacks made by the sub
Submerged attacks - number of submerged attacks made by the sub. Note that although there is not direct code linkage between the 4, spotted + surface + submerged tends to equal total engagements.
torps launched - number of torpedoes launched by the sub. Note that the combat report only announces the number of torpedoes launched in submerged attacks
torps hit - number of torpedo hits scored. Only incremented when torps launched is > 0 (so in effect, these are only torpedo hits during submerged attacks)
torps / target - number of torpedoes launched divided by submerged attacks
likely sunk - a metric that I hoped would mean something. if 2 torpedoes were launched at a target, then 1 hit would increment this. If more than 2 were launched, then 2 hits would be needed to increment this. However, watching combat more closely now, I see several times were a sub attacks with 2 torpedoes, then later in the turn attacks the same boat with 2 more.
warships attacked - number of cruisers and bigger boats attacked
escorts attacked - number of DDs, DEs, PBs, etc attacked
merchants attacked - number of merchants / cargo / aux ships attacked
success rate - successful attacks / total engagements
successful submerged % - successful submerged attacks / submerged attacks
hit % - torps hit / torps launched
likely sunk % - likely sunk / total engagements
spotted % - spotted / total engagements
grade - 1=boat has higher hit% then average and has over 10 engagements, -1=boat has lower hit% then average and has over 10 engagements, 0=boat has 10 or fewer engagements
skill - the skippers "skill" from the editor
naval skill - the skippers "naval skill" from the editor
aggr - the skippers aggression from the editor
deployments - a subjective rough area the boat was deployed, from looking at combat reports.



Some observations:
Fleet boat torpedoes SUCK. You knew that already. But they really do suck. Fleet boats only have a success rate of 16.9% and a hit% of 6.8% The next worst are the various iterations of Dutch boats at about 30% success. The S boats are at 40%, and the Brits are shooting 60% (though in limited samples). That said, it is still important to use them as they have great range, and even with their awful success rate, they made almost double the number of successful attacks then all other allied subs combined.

British boats appear to be devastating. Though that comes with the asterisk that there are only 2 of them, and the sample size is limited. Still, I plan to start using them more aggressively.

The Dutch boats get spotted a lot. 46% of the time for KX series and 33.8% for O. I'm not sure what to make of this. I operated them in the Dutch East Indies in the early war, and now in the Solomon’s areas. Perhaps it was something Japan was doing there (air cover, escorts ?). The KX merchants to escorts attacked ratio is the same as the Fleet boats.


I ran some more detailed analyses of the Fleet boats. Since I hadn't changed any of their skippers (why bother when their torpedoes suck so bad - I put new skippers on the S Boats), I was able to go to the editor and copy the skill levels of the skippers and map that to their boats. From here, I ran a correlation between "skill", "naval skill", "aggression", "naval skill" + "aggression", and "skill" + "aggression".
Most of the counter columns have a negative correlation... however I think this is an artifact. The more skilled / aggressive skippers come in later then the starting skippers. Because of this they haven't had the time that the starting skippers have had, so I think it is meaningless. The rate statistics though, look interesting:
The biggest correlation is a negative one between naval skill and aggression for torpedoes / target. It appears that the better commanders shoot fewer torpedoes. This is good, as they will be able to stay on station longer. (By the way, the correlation between torps/target and hit% is a modest .168)
More important is the correlation between success based metrics is positive for the 3 skills measured, but combining aggression with either is a higher correlation. The best combination is skill and aggression (I don't know how / if "skill" even shows up when picking commanders). Of the individual skills, aggression has the highest correlation. So when picking new commanders, don't look at just aggression, but rather the sum of aggression and skill (or aggression and naval skill), with a slight bias towards higher aggressions. Actually, aggression may be more important, as the standard deviation of aggression is high (17.18), whereas skills are more clustered around the mean (8.75 and 6.97 for skill and nav skill).


I scored each boat on hit % (success rate is largely influenced by spotted %, which in turn is largely influenced by patrol location). Most of the time, high skill + aggr ratings meant a high hit % as mentioned above. But some boats are outliers. So I examined a few of them.
Sturgeon, Swordfish, and Seadragion have very high scoring skippers, but below average to awful scores. Apart from beginning with "s" (nearly all my boats that have a name starting with "s" have poor scores - what sub analyses would be complete without luck involved?). These boats seem to be patrolling of the coast of Japan a lot, which may be lowering their scores (with Japanese air patrols in the area).
"P" seems to be the letter you want your boat to start with. Most "P" boats score well, even Pickerel. Despite a lower than average skipper, it has an astounding 13% gut rate and a success rate of 25% - including popping a CVL twice.
Of course Sailfish (with a truly awful skipper), is the exception to the "P" is good, "S" is bad trend. With a 27% success rate, I decided to dig into his history. I found he made all his attacks near Kwajalein. So I decided to map the locations of successes and where boats have been spotted, and this may be the most important (why I didn't start with this, I don't know):


It appears that air patrols are a major factor in the success rate of submarines in the game (no real surprise). The attached graphs mark the location of all successful sub attacks, and all sightings from game start through May and June through December 1942:
For some reason, Robo Yamamato does not effectively patrol the central Pacific. Turk and Kwajalein both have many successful attacks and very few sightings. Guam is even more pronounced with many successful attacks and NO sightings. Also, around Rabaul has gone from somewhat successful to a killing ground now that I've taken control of the Solomon’s and neutralized Rabaul. I don't know why the Dutch East Indies has such a poor success rate, though it could be due to the constricted waters and interlocking air patrols. The region between Formosa and Japan appears to be well patrolled, but there is a gap in his coverage somewhat north of the Philippines. Also if you can stand off from Japan about 5 hexes, the success rate goes from hit or miss to very high.

Likewise, on the Japanese side, the west coast went from early war success to absolutely nothing now that my ASW patrols are up and the pilot skill level is respectable. Controversy, he is hitting me hard around India and by Perth where I don't yet have effective pilots.

[tiemanjw]

I screwed up the attachment to the previous post. Here are all the maps and spreadsheets.

[dr.hal]

These data are amazing. I know it is only one game but it still is telling. I hope you continue the analysis as your game progresses. I believe your outcomes are about what I've experienced, although my Dutch boats have done much better than you indicate in your game. One thing that is not clear, I am assuming that your boats were all under human control, is that right? Also for the US non "S" class boats, I'm not sure what the various colors indicate, as I didn't find a key, did I miss it? Thanks Hal

[Lokasenna]

Thanks for this. Good to see data backing up anecdote.

I'm of the impression that the I-boats can be absolutely devastating, at least early war, if they manage to find shipping. I would love to see the data for a human using I-boats.

[tiemanjw]

The red text is a low scoring boat (hit%), green is high. Black is 10 or fewer engagements. Highlighted lines are the outliers (above average skill + aggression with low hit %, or below average skill + aggression with a high score). I just did this so I could quickly see who was performing, and who was not, and who I needed to spend more time investigating.

At the start of the game I checked auto sub ops thinking it would free me of some hassle. It only took a turn or 2 to realize that was an error, and I have taken control of my boats. I still occasionally find the odd boat under computer control. I also disagree that I have had poor performance from the Dutch boats. I just seems their failures are due to getting spotted. When they are not spotted, they have very good success rates. And I'm not sure what to make of that high spotted rate. For the first 6 months or so they opperated in DEI waters. Since then some fleet boats have moved into that area, and as the map shows, they too are spotted at a high rate in this region. But of course tje Japanese didn't have the same level of airpower then. I'm at a loss to explain it.

If intrested, I can post an executable and any set or subset of combat reports can be analyzed. You would need the labview10 runtime engine (I'm a old hardware guy miscast as a programmer)... A free download from national instruments. I don't know if it works on non-windows 7 machines though.

[inqistor]

It seems you lack the most important - experience of ship crews.

I know, that they increase with every success, but this is supposed to be more important, than Captain statistics.

[tiemanjw]

ORIGINAL: inqistor

It seems you lack the most important - experience of ship crews.

I know, that they increase with every success, but this is supposed to be more important, than Captain statistics.

So I looked into it a bit more. Sadly, the editor doesn't have original skill values for the ships crews. It just lists "0", which I guess means some kind of random number, or programmed elsewhere. Further, as you say, it changes throughout the game... and I don't know how useful it would be to say that success rate correlates to success rate.

That said, in digging into this, I went to look up what the leadership values was for individual captains. My understanding is that leadership value is a value for how effective crew experience gain is. For this reason, I look for a high leadership value for my training squadrons. Leadership, though, is not listed in the editor... rather the first skill type field in the editor is simply "skill". So I checked for half a dozen leaders, and found that this value matches what is in the game leadership.

So this makes perfect sense. In my original post, I found that this mysterious "skill" value was the greatest indicator of success, but didn't know how to use that in the game. But since "skill" translates to "leadership", it means that when selecting a new leader, you should use a high leadership value first, then aggression. I think naval skill is useless as the correlation of success rate and hit % to "skill" + "aggression" is identical to 3 decimal places of "skill" + "aggression" + "nav skill".

[Lokasenna]

ORIGINAL: tiemanj

ORIGINAL: inqistor

It seems you lack the most important - experience of ship crews.

I know, that they increase with every success, but this is supposed to be more important, than Captain statistics.

So I looked into it a bit more. Sadly, the editor doesn't have original skill values for the ships crews. It just lists "0", which I guess means some kind of random number, or programmed elsewhere. Further, as you say, it changes throughout the game... and I don't know how useful it would be to say that success rate correlates to success rate.

That said, in digging into this, I went to look up what the leadership values was for individual captains. My understanding is that leadership value is a value for how effective crew experience gain is. For this reason, I look for a high leadership value for my training squadrons. Leadership, though, is not listed in the editor... rather the first skill type field in the editor is simply "skill". So I checked for half a dozen leaders, and found that this value matches what is in the game leadership.

So this makes perfect sense. In my original post, I found that this mysterious "skill" value was the greatest indicator of success, but didn't know how to use that in the game. But since "skill" translates to "leadership", it means that when selecting a new leader, you should use a high leadership value first, then aggression. I think naval skill is useless as the correlation of success rate and hit % to "skill" + "aggression" is identical to 3 decimal places of "skill" + "aggression" + "nav skill".

I really wish this was something that Tracker tracked, a la the Ship History buttons and such.

[gregplatt52]

Very interesting. Thanks for posting

[bigred]

ORIGINAL: dr.hal

These data are amazing. I know it is only one game but it still is telling. I hope you continue the analysis as your game progresses. I believe your outcomes are about what I've experienced, although my Dutch boats have done much better than you indicate in your game. One thing that is not clear, I am assuming that your boats were all under human control, is that right? Also for the US non "S" class boats, I'm not sure what the various colors indicate, as I didn't find a key, did I miss it? Thanks Hal

good job...

[John Lansford]

The water around the DEI is shallow; that's probably why the Dutch subs are being spotted so often. I try and keep my subs out of shallow water since they are much more vulnerable there than in deeper depths, but some times that's just not an option.

[obvert]

Very interesting results. Fun to get all of this data together. Have you used the map on tracker to check ship movements? If you have each day loaded it locates the ships and draws out their paths. Could be a nice visual component to see their patrols and even use the map to plot your points and data (by hand of course as you've done on the graphs. I do like how the data begin to show evidence of the game map as you plot the info).

Here is an incomplete but still interesting example for the I-25.

[tiemanjw]

So, the next 3 months of sub operations in my game. Full results are attached at the bottom of the page. Not surprisingly, the last 3 months have seen a significant increase the success of the fleet boats to almost 36%. Further, several of the "failures" are of the type "elects not to launch torpedoes at this target". I'll have to modify the code to pull those out. I don't want to fail skippers because they aren't launching at small, worthless boats.

I know most of the increase has been the drop in dud rate from 80% to (if I read the manual correctly) 60%. But it is notable that, despite efforts to get them more involved, the Dutch, British, and S class boats have nearly disappeared from the equation. The numbers and range of the fleet boats make them effective weapons even with crappy torpedoes.

January was an excellent month... I think I only had one boat spotted by an escort. But this ended in February, as the escorts were back, and seemingly more effective. My guess is that he had an upgrade come available that added better ASW gear to his boats. That said, he is still only damaging my ships.

As for where. I moved my boats back a bit from the home islands (3 or 4 hexes or so instead of right up on them). Also near Batan Island between Luzon and Formosa is a killing grounds. Recently I've moved a few boats into the Sea of Japan and the East China Sea with success. With my successes in the Marshal islands, I'm thinking of putting a forward refueling / rearming point there, which will allow more extended patrols into the South China Sea. Also there are successes in the middle of the Philippine Sea and off to the West / North West of the Marshals. I'm hoping upcoming operations against NW New Guinea and into Moluccas (and possibly Peleliu) will close the southern exit from the Dutch East Indies, and I can concentrate my subs in the South China Sea.

As for his subs - they are now nearly a non-factor. He has 3 operating areas now. Near Columbia and Sydney, I can't seem to remove a pesky sub that has made multiple successful attacks. He also has one off Vancouver Island, but he hasn't had any success. There is also one between Green Island and Kavieng, but he hasn't done anything (occasionally spotted by air patrols). Air power with effective (ASW ratings around 70 or so) and long range planes seems to be quite effective and suppressing submarine attacks - though kills are still hard to come by. Aggressive strikes at forward bases (I got a dozen of 'em with some sustained airstrikes on Turk).

[John Lansford]

It's late 1943 in my current CG and the IJN's subs have very nearly vanished from the ocean. I've seen a few in the Indian Ocean but that's about it. One of the worst chokepoints for the Japanese is south of the PI; I have all my remaining Dutch and S-boats operating out of Darwin and they have turned that area into a tanker graveyard. It appears that the routing of tankers from Java and Borneo is south of Davao and up the east side of the PI, rather than to the west of the PI.