So I decided to test this on some Pacific islands. Same set up, except I am using a red unit with longer-ranged radar. The red unit seems to detect all units in open, in front of terrain, and hidden by the island...at 17 nm regardless of position.
I have included two files; the Pacific file is set up before I hit start and the Pacific 2 file is after it ran and initially detected the Osa behind the island at 17 nm. On the surface, it appears that terrain masking isn't working at all, but it is 99% of the time. The issue is at the radar detection range, any unit is immediately detected, whether in open ocean, in front of terrain, or behind terrain. But if you really watch what happens, after that initial detection, it loses the terrain masked unit, never to reacquire it. You can tell by looking at the timer on the detected unit. It times up to 3-4 hrs and then contact is lost. Unless you turn the red unit around and head back away. As soon as it hits that 17 mi initial detection range, it gets that one detection and then the detection times starts as the red unit speeds away.
So something happens at the initial detection range that causes an initial detection, no matter what. But then the game goes on and seems to use terrain masking properly. This is on units behind terrain. I am trying to get at the terrain clutter issue, and it might be the same thing.
My uneducated guess.