If you are not happy I would talk to the nanny. But also why not invite them over for a Christmas drink one evening - means you can interrogate him and find out what he does / done.
I would also ask for a dbs if he wants to be in your house and around you children.