Yes, I always thought that the "Shadow" thing was, from a developer's perspective, the hardest feature to drop, I mean, in the manual there is paragraph after paragraph explaining it, just to become meaningless once the game starts.
The thing is that the Shadow is fragmented to a incredible degree.
He had planned the Dissolution war to take power via puppets, but totally underestimated the severity of hte conflict.
Seperate fragments of his mind might be following different plans on the way to restore the whole. So he can make one story without conflicting with another.
Another alternative that can complement a late game crisis is to have "fallen regimes" (idea borrowed from Stellaris). The idea is that there are some (few) regimes that survived from the galactic republic era, but they just became totally isolationist, so they are technologically advanced, but they won't expand, they won't build armies, they won't bother you as long as you don't declare war.
However, once you reach a certain level of tech or victory score, they will then consider you a risk, and "activate", meaning they will start expanding and eventually declare war on you
Well, Stellaris stole a lot of things from Distant Worlds 1. Including the Fallen Empires (there called the "Ancient Guardians").
So it is only fair from another Matrix game to steal soemthing back.
The existence of robot Infantry and GR era factories that were never damaged would explain how they could match a empire that literally controls the whole planet.