From: Near Portland, OR
Pre-war Japanese doctrine considered surface ships the more potent threat. They figured the USN aircraft to be poor quality and the crews just as poor. The TBD was a generation behind the Kate, and Japanese doctrine saw the torpedo bomber as the primary naval weapon with the dive bomber being the secondary weapon that split enemy forces and crippled ships for the torpedo bombers to finish off.
The Japanese badly underestimated the SBD. The SBDs dropped their bombs at a higher altitude than the Vals. The Japanese thought that was because American pilots were weak, they didn't realize it was because the US had a superior weapon.
The US had a technology nobody else did, the trailing edge split flap. Japanese and German dive bombers used fence type dive brakes. Fence style dive brakes did work, but the plane bucked and bounced on the way down. To ensure a hit, pilots had to hold the dive to near suicidal altitudes. With the trailing edge split flap, the dive was very stable which allowed the pilot to make gentle course corrections within the dive. SBDs could release at a higher altitude and still have a good chance of a hit because the pilot had a better assurance where the bomb was going after release.
The SBD could also carry a larger load. The max load was 1600 lbs for an SBD, but that was almost never used. The SBD could and did carry 1000 lb bombs, the Val was limited to 250 Kg (about 550 lb). And early war USN strike package was usually about 2/3 500 lb bombs and 1/3 1000 lb bombs, but the bigger bombs were there.
The lack of a potent torpedo bomber at Midway did ensure that the USN didn't actually sink any carriers. But the SBDs did such massive damage that the IJN scuttled all 4 carriers. The description of the Kaga in Shattered Sword is striking. Even if the ship had been capable of getting to a friendly base, she might have been scuttled anyway. The fires gutted all the upper decks of the ship.
In any war, especially a long one, everybody goes in with assumptions that are proven wrong in combat. The Japanese thought their carrier aviation was vastly superior to the USN and in some ways it was, but they underestimated US strengths.
Everybody underestimated how much flak they would need. Both sides upgraded flak suites on everything throughout the war. The US was more aggressive about it, but they also had the industrial infrastructure to afford it.
The US had the mutual defense doctrine down early war (though effectiveness lacked), but the Japanese thought the big threat was coming from surface ships coming in after the air attack, so they built their CV formations with that in mind. The Japanese war doctrine was built on the subs thinning the enemy fleet, then the CVs taking out some more, and finally the big surface ships would come in to finish the job. The US was thinking along the same lines, with CVs and BBs, but the loss of so many BBs at PH made them rethink. By Midway the US knew they were much weaker in surface assets, but they could attain near parity in air power between the 3 Yorktowns and Midway. Even still, TF 1 was waiting out in the middle of the Pacific between San Francisco and Midway with all the serviceable BBs in the Pacific fleet. A bit of the US OOB nobody knows about.
Early war USN air defense doctrine was a bit better than Japan, but early carrier losses demonstrate it had holes too. It's telling the only US carrier losses after 1942 were due to 3 factors:
1) A lucky shot by a lone attacker slipping through the CAP (the Princeton)
2) A massive screw up (Battle Off Samar)
3) Kamikazes (which never sank a carrier larger than a CVE, though they damages a lot of fleet carriers)
After 1942, the US started building a lot of advantages from lessons learned, better AA, better AA direction, better CAP, better fighters as well as better pilot training, and better CAP direction. At the same time Japanese pilot quality was in steep decline and newer aircraft designs were often inferior to US designs. The US had the luxury of building fighters around big engines like the P&W 2800 which had just about worked out all the bugs by Pearl Harbor. The Japanese had to start from scratch with bigger engines and it wasn't until late war they had anything they could use and Japanese large engines remained finicky to the end.
I'll stop rambling now...
WitP AE - Test team lead, programmer