Yeah, but as it is now your very first colony - apart from your home planet - triggers waves of pirate attacks including ground forces up to 250k troops.
So best thing is, going in laager mode on your home planet, research heavily (esp. economics, bunkers, ground defense, weapons of your choice) and then build some really powerful big fleets simultaniously going for pirate space ports. And only then start colonizing. It's quite telling that, as it is now, other empires also are not capable of colonizing the way it has been before. A few 100 years into play and all they have is 2 to 3 colonies - if any at all.
I don't think this is *bad*. But sometimes I'd like to play non-tower defense modes. Please let me do so.
The other empires in my game are colonizing, now about 60 odd years in and most of them are at 6-10 planets, 2 are on 15+. Just a thought but did you pay for protection at game start? I never bother no matter how many problems they cause, I would rather just ride the storm. Reason I ask is I have noticed that quite a few pirate factions that are in other empires sphere of influence do seem to have there numbers increasing by the month, can only presume they are paying protection and the pirates are using the cash to increase there strength, I have wiped out all the factions anywhere near my systems and any factions near my sphere of influence are now down to 1-5 ships due to my constant attack and defend stance.
Yup, usually pay for protection to the first two pirate factions until I have about twice their firepower, bunkers, (anti-)boarding tech. Then I do attack.
Never pay for tech, colonies and other stuff pirates offer to me.
BTW, first pirate protection offers seems to be triggered by having built first space port or ships. No matter how near they are set. One of the first things with my first space port I do is building 20 frigates. But pirates have better ships at the very beginning.