From: Miami, Florida, U.S.A.
I have actually had ships (Mostly explorers) try to run up to the edge before. I usually have them on "Explore x Sector" Missions if that helps narrow down the problem.
Same here - explorers go to 0,0 when told to explore specific sector.
So here's how it works behind the scense. A ship has a mission. A mission have many fields including (from memory) target, habitatTarget, builtObjectTarget, creatureTarget, and maybe more. Depending on what type of mission it is one of those fields will hold the X,Y values to go to. I had to take into account a lot of things when I added gravity wells and epicially when I taught the AI how to find the shortest path out of a gravity well. It seems that searching a sector is one scenario I didn't cover as I never ordered a scout to do that. Searching a system works though. So it might be best just to tell the scout to search a system. You can use a reminder note (!note) to notify you when it is done. Personally I always have my scouts on auto explore. Also, don't forget to add labs to your scouts so they can be researching for you as science ships. In most of the playthrough videos of people using my mod nobody ever does that for some reason.