It is not easy to keep a galactic wide empire.
I would guess it is about colony happiness. Likely suspects are:
- Tax. If you do this manually, be aware that too high tax combined with other negatives may make them defect.
- Being oppressive. Some governments are probably better at keeping colonies than other.
- Being a bad boy. You reputation drops a lot if you attack without declaring war, or use a lot of bombardment. If you annihilate colonies it is probably even worse. Too low reputation will make people angry, and it can sound like this was a problem for you. Kill pirates to keep your reputation up.
- War exhaustion. It builds while at war, and is a negative happiness modifier. Your government type has influence on this.
- Alien populations can have negative modifiers.
There are also positive modifiers. A spaceport with recreation and medical modules will improve happiness.
In general, look at colony details from the colony page. The population page will give a list of all positive and negative happiness modifiers, including their sizes. It also give a quick view of the sad faces of colonies...
In a securan game of mine, I have a dhayut colony. Securans area peaceful humanoids. Dhayut are violent insects. Not a perfect match. Their unhappiness at being in the Securan empire (-4) pales among their complaint about taxes (-28). And they are a happy bunch, the empire is at peace. The AI will adjust taxes to try to keep colonies outside the red, sad faces.