You sound lovely for trying to help! He does sound terribly lonely (as well as awful) and I'm sure he's miserable about how his life has turned out. And it may be that rather than being a straightforward racist, he was trying to convey that he can't ask for help or understand the advice he's given by healthcare professionals, which is also isolating and frustrating.
You are under no obligation, though, to make up for what his family are not doing. You are obviously worried about him but you can set boundaries for how much interaction you need to have with him. Can you have him round for a cup of tea when your children are not there, so you don't need to worry about what he says? Then make it clear that you're going out in an hour, non-negotiably, so you are setting the terms of how long he has in your home. If you are really nice you could have made some meals for him and say 'I'll put these in your fridge/freezer on my way' so he has to go to let you into his house! Then you can leave him there. At the moment he is taking advantage of your kindness and politeness, possibly out of desperation, but it's much too hard on you. If he has more support from other agencies, he may be less needy.