Search arc statistical test

[rader]

I've always been diligent about trying to set search arcs. They seem like a neat feature, and appeal to my micromanagement side. However, after reading the thread about search arcs, I decided the run a test.

Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s.

The Nells are looking for the 4 US TFs, all 14 hexes from Rabaul (see picture). They are the default ones from Coral Sea scenario. The TFs are relatively small; 2 have a CV and a few CAs/DDs each; 2 have an AO and DD (or maybe ~two AO each + DD(s)). Anyway, somewhat hard targets to spot.

Comparison of 3 cases, 7 samples each:
1. 7 samples of Nells with no search arcs set (0-360).
2. 7 samples of Nells set to search "optimal" search arc, covering exactly the 40 degrees the US TFs are aligned along at 14 hexes. This is about as good a search as you could do if you knew exactly where the enemy was (apart from sending more planes!).
3. 7 samples of Nells set to *opposite* search arcs (looking north in exactly the wrong direction, around 340-020 deg).

Results (number of TFs spoted per sample):
No search arcs: (1/1/1/2/1/2/0) (mean = 1.14 TFs spotted)
"Optimal" search arc: (0/2/1/1/1/2/0) (mean = 1 TF spotted)
Wrong (opposite) search arc: (0/0/1/0/0/0/0) (mean = 0.14 TF spotted)

I ran two 2 sample t-tests in SYSTAT to compare cases (1. & 2.), and (1. & 3.)

No search arcs (1) vs. Optimal search arc (2). p = 0.73. (no significance)
Optimal search arc vs. wrong search arc. p < 0.006 (quite significant difference).

Discussion:
I don't claim this covers all cases, and there are confounding effects like weather which I didn't examine. Maybe in some cases you do get better results by setting the right search arcs (longer/shorter search distance/altitude/larger TFs, etc.). And although the mean is slightly lower, I certainly can't claim that setting the right search arc is better or worse than setting none at all. But I do think this demonstrates that setting the *wrong* search arc is worse than setting the right one OR setting none at all. So in this type of situation you probably aren't much worse off setting no search arc than setting the right one. Which, of course, you can't perfectly predict.

Conclusion: On average (assuming the enemy sometimes comes from directions you don't set as arcs), you will find him more often *under similar conditions to this test*, if you don't set arcs. I think I'm going to take off all my search arcs.

[stuman]

Hmm.

[Lrfss]

Starting to think the same thing myself!

[witpqs]

...which would mean it's broken. Either too good results without arcs, or not good enough results with arcs.

[Cribtop]

Given the huge player time investment, setting search arcs should provide a meaningful benefit.

[witpqs]

ORIGINAL: Cribtop

Given the huge player time investment, setting search arcs should provide a meaningful benefit.

Search arcs are meant to be more realistic. Setting no arcs, IIRC, means "choose the search paths vectors randomly". As the player knows where the likely threat is, search arcs should yield significantly more real contacts that using randomly selected vectors. From the anecdotes of those who have tried both (I've only used arcs in AE), and from the test that rader performed, it looks like there is a program problem.

[crsutton]

I abandoned search arcs about 90 game turns ago. I find the results just as good with no search arc set and you can cover more territory. Your tests seem to back this up. I would think some more testing is called for.

[Sardaukar]

Run the tests with enemy TFs bit closer to base. I seem to recall, that you rarely spot anything more than 12 hexes away or so. This may significantly change the results.&nbsp;

[Misconduct]

ORIGINAL: crsutton

I abandoned search arcs about 90 game turns ago. I find the results just as good with no search arc set and you can cover more territory. Your tests seem to back this up. I would think some more testing is called for.

Agreed, I have yet to set a "search" arc in 4 games and have yet had a problem with spotting. Biggest problem is setting the arc and forgetting about it tends to annoy me having to go back and switch it, so I just leave everything on 80% search, 20% rest.

[rader]

ORIGINAL: Sardaukar

Run the tests with enemy TFs bit closer to base. I seem to recall, that you rarely spot anything more than 12 hexes away or so. This may significantly change the results. 

Good idea, will do that. How far would you suggest? 10 hexes?

[Sardaukar]

ORIGINAL: rader

ORIGINAL: Sardaukar

Run the tests with enemy TFs bit closer to base. I seem to recall, that you rarely spot anything more than 12 hexes away or so. This may significantly change the results. 

Good idea, will do that. How far would you suggest? 10 hexes?

8-10 should be fine.

[jomni]

This is the reason why I don't manually set the arcs. An unncesary nuisance just like pilot training.

[Mistmatz]

I like your test setup, but IMHO the sample size is by far too small too be statistically significant.

[castor troy]

shouldn´t NO search archs be far worse than with your search archs of only 1/9 of the area covered with archs. Yet, even when the Nells have to cover 9 times more area they have spotted 14% more in your test. [&:]

[Apollo11]

Hi all,

ORIGINAL: rader

I've always been diligent about trying to set search arcs. They seem like a neat feature, and appeal to my micromanagement side. However, after reading the thread about search arcs, I decided the run a test.

Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s.

The Nells are looking for the 4 US TFs, all 14 hexes from Rabaul (see picture). They are the default ones from Coral Sea scenario. The TFs are relatively small; 2 have a CV and a few CAs/DDs each; 2 have an AO and DD (or maybe ~two AO each + DD(s)). Anyway, somewhat hard targets to spot.

Comparison of 3 cases, 7 samples each:
1. 7 samples of Nells with no search arcs set (0-360).
2. 7 samples of Nells set to search "optimal" search arc, covering exactly the 40 degrees the US TFs are aligned along at 14 hexes. This is about as good a search as you could do if you knew exactly where the enemy was (apart from sending more planes!).
3. 7 samples of Nells set to *opposite* search arcs (looking north in exactly the wrong direction, around 340-020 deg).

Results (number of TFs spoted per sample):
No search arcs: (1/1/1/2/1/2/0) (mean = 1.14 TFs spotted)
"Optimal" search arc: (0/2/1/1/1/2/0) (mean = 1 TF spotted)
Wrong (opposite) search arc: (0/0/1/0/0/0/0) (mean = 0.14 TF spotted)

I ran two 2 sample t-tests in SYSTAT to compare cases (1. & 2.), and (1. & 3.)

No search arcs (1) vs. Optimal search arc (2). p = 0.73. (no significance)
Optimal search arc vs. wrong search arc. p < 0.006 (quite significant difference).

Discussion:
I don't claim this covers all cases, and there are confounding effects like weather which I didn't examine. Maybe in some cases you do get better results by setting the right search arcs (longer/shorter search distance/altitude/larger TFs, etc.). And although the mean is slightly lower, I certainly can't claim that setting the right search arc is better or worse than setting none at all. But I do think this demonstrates that setting the *wrong* search arc is worse than setting the right one OR setting none at all. So in this type of situation you probably aren't much worse off setting no search arc than setting the right one. Which, of course, you can't perfectly predict.

Conclusion: On average (assuming the enemy sometimes comes from directions you don't set as arcs), you will find him more often *under similar conditions to this test*, if you don't set arcs. I think I'm going to take off all my search arcs.

Very nice test - thanks! [:)]


BTW, I did some extensive research in UV and WitP regarding search but didn't have time to do it in WitP-AE...


Anyway here are my old threads (please click on links below):

10/16/2005 Leo's ASW TEST (and scenario) - problems spotted!

11/21/2005 Leo's air ASW Search TEST WitP v1.795...


In those I discovered that in "Search" every airplane is actually flying each HEX it covers and that in each HEX the crew has a chance of discovery (depending on various other variables like altitude, weather, skill etc.)...

Here is how I think it works:

#1
Every pilot in every squadron assigned for air search ("Naval Search" or "ASW") that passes all pre-flight checks (enough pilots, enough aircraft, leader check, percentage of aircraft assigned to mission, weather etc.) individually flies the "spiral" (i.e. concentric circles with increased range) up to the range user selected (halved in case of ASW).

#2
The "spiral" (i.e. concentric circles with increased range) covers every single HEX (in range) and then checks are made in every such HEX for discovery and countermeasures (enemy CAP, AA etc.).

What worries me now is that even with opposite search (i.e. searching nortn in your test) there were some discoveries. If the routine works similarly as it used to work in WitP then only the northern HEXes and not at all in the south... hmmm...


BTW, did you stand down all other assets that can do the search?



Leo "Apollo11"

[castor troy]

ORIGINAL: Apollo11

Hi all,

ORIGINAL: rader

I've always been diligent about trying to set search arcs. They seem like a neat feature, and appeal to my micromanagement side. However, after reading the thread about search arcs, I decided the run a test.

Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s.

The Nells are looking for the 4 US TFs, all 14 hexes from Rabaul (see picture). They are the default ones from Coral Sea scenario. The TFs are relatively small; 2 have a CV and a few CAs/DDs each; 2 have an AO and DD (or maybe ~two AO each + DD(s)). Anyway, somewhat hard targets to spot.

Comparison of 3 cases, 7 samples each:
1. 7 samples of Nells with no search arcs set (0-360).
2. 7 samples of Nells set to search "optimal" search arc, covering exactly the 40 degrees the US TFs are aligned along at 14 hexes. This is about as good a search as you could do if you knew exactly where the enemy was (apart from sending more planes!).
3. 7 samples of Nells set to *opposite* search arcs (looking north in exactly the wrong direction, around 340-020 deg).

Results (number of TFs spoted per sample):
No search arcs: (1/1/1/2/1/2/0) (mean = 1.14 TFs spotted)
"Optimal" search arc: (0/2/1/1/1/2/0) (mean = 1 TF spotted)
Wrong (opposite) search arc: (0/0/1/0/0/0/0) (mean = 0.14 TF spotted)

I ran two 2 sample t-tests in SYSTAT to compare cases (1. & 2.), and (1. & 3.)

No search arcs (1) vs. Optimal search arc (2). p = 0.73. (no significance)
Optimal search arc vs. wrong search arc. p < 0.006 (quite significant difference).

Discussion:
I don't claim this covers all cases, and there are confounding effects like weather which I didn't examine. Maybe in some cases you do get better results by setting the right search arcs (longer/shorter search distance/altitude/larger TFs, etc.). And although the mean is slightly lower, I certainly can't claim that setting the right search arc is better or worse than setting none at all. But I do think this demonstrates that setting the *wrong* search arc is worse than setting the right one OR setting none at all. So in this type of situation you probably aren't much worse off setting no search arc than setting the right one. Which, of course, you can't perfectly predict.

Conclusion: On average (assuming the enemy sometimes comes from directions you don't set as arcs), you will find him more often *under similar conditions to this test*, if you don't set arcs. I think I'm going to take off all my search arcs.

Very nice test - thanks! [:)]


BTW, I did some extensive research in UV and WitP regarding search but didn't have time to do it in WitP-AE...


Anyway here are my old threads (please click on links below):

10/16/2005 Leo's ASW TEST (and scenario) - problems spotted!

11/21/2005 Leo's air ASW Search TEST WitP v1.795...


In those I discovered that in "Search" every airplane is actually flying each HEX it covers and that in each HEX the crew has a chance of discovery (depending on various other variables like altitude, weather, skill etc.)...

Here is how I think it works:

#1
Every pilot in every squadron assigned for air search ("Naval Search" or "ASW") that passes all pre-flight checks (enough pilots, enough aircraft, leader check, percentage of aircraft assigned to mission, weather etc.) individually flies the "spiral" (i.e. concentric circles with increased range) up to the range user selected (halved in case of ASW).

#2
The "spiral" (i.e. concentric circles with increased range) covers every single HEX (in range) and then checks are made in every such HEX for discovery and countermeasures (enemy CAP, AA etc.).

What worries me now is that even with opposite search (i.e. searching nortn in your test) there were some discoveries. If the routine works similarly as it used to work in WitP then only the northern HEXes and not at all in the south... hmmm...


BTW, did you stand down all other assets that can do the search?



Leo "Apollo11"

he did:

Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s.

[Apollo11]

Hi all,

ORIGINAL: castor troy

he did:

Conditions: Coral Sea scenario, all Japanese a/c stood down except 27xNell in Rabaul set to search 30% (8 a/c total searching), 6000 ft, 14 hexes. Experience/search skill around mid-50s.

Christian, yep... I read that in original first post... but I was wondering about all other assets as well... [;)]


Leo "Apollo11"

[d0mbo]

Inquiring minds follow this thread and hope to hear a DEV's opinion on this.
&nbsp;
If I don't have to bother with search arcs (or at least: it doesn't give an distinct advantage in doing so),&nbsp;that would be good to know!
&nbsp;
&nbsp;

[jomni]

I would rather hear them say that it will give only a slight advantage (as the way it is working now) in order not to make it a requirement for all players.
If people claim that it is broken and the devs should make it work... this will penalize those people who don't bother setting search arcs like me.
Same sentiments with pilot training.

[Sardaukar]

ORIGINAL: Apollo11


What worries me now is that even with opposite search (i.e. searching nortn in your test) there were some discoveries. If the routine works similarly as it used to work in WitP then only the northern HEXes and not at all in the south... hmmm...


BTW, did you stand down all other assets that can do the search?



Leo "Apollo11"

Devs (I think it was michaelm) eplained that search arcs are supposed to increase chance of detection within arc (and most likely decrease it outside it). Setting arc does not prevent spotting ourside arc, but it should concentrate search more inside arc.