You obviously know him, so have reason to believe him or not, but from an outsiders perspective, all the signs are there, especially if even your friends and family suspect he is having an affair.
Of course he is going to say no he isn't, they always do. And when they move someone in within weeks of a split, they will still claim nothing happened until after the separation and it was totally innocent/platonic until that point.
It is a very depressing cliche...
If there hasn't been much active talk from him about splitting in the past, and now the only thing on his mind is getting you out of the house, there is someone waiting to take your place.
Read up about 'spousal abandonment' and see if any of it fits. Men follow an alarmingly predictable pattern when they do it, including the lengths they'll go to to repaint history and also to go out of their way to be nasty and cruel after the split
Wishing you all the best