Yeah, it's one of the limitations of the DW diplomacy system - unless one of their bases is located inside your territory border, you can't request anything be done about it - and the AI requests completely exorbitant costs to hand over even the facilities inside your borders, so it's always far better to ignore them or destroy them via sabotage.
Unless you're willing to go to war over it, the only things you can do are hamper the colony by declaring trade sanctions and blockading it (doesn't really help anyone IMO because assuming you want to eventually take it over your just hurting its growth), offering a pirate attack contract against it (but again, if anything this'll probably make the enemy empire assign more ships close to your border), use your spies to periodically sabotage construction at the planet and any spaceports they build, or... just live with it there - at least it'll facilitate trade between the both of you if you're friendly, and makes an easy target should you go to war if you're not.
Beyond that, all I can say is I hear it used to me much worse. Before one of the earlier expansions, borders weren't in the game at all, and anyone could build anywhere. =P