It happened to me too yesterday, it drifted all over. So i did factory reset on both Solo and controller, it worked! Had to update and all, but when tested, i noticed less drift.
Today i got my GPS mod installed M8N and it drift even less, i think. i was able to fly manual all over between roof...