Yep, we've lived very close to the railway (runs diagonally to our house, five houses away) for over a decade and yes you do hear it of course, but we're so used to it now that it really doesn't matter. When we first came to view the house, as soon as I saw how close it was to the railway I thought there was no point even looking at it (I'd lived for years with a very noisy upstairs neighbour), but my husband persuaded me to give it a chance and I'm so glad I did. I got used to it pretty quickly, and hardly notice it now, except in the garden when you have to stop talking when a train passes by.
It probably helps a bit that we sleep at the front of the house and the noise is greater at the back, but our kids sleep at the back and have slept through all train noise since they were born. Freight trains can be particularly noisy, and we do give overnight guests earplugs (the guest room is also at the back of the house), but those who visit often (in-laws) have also got used to the noise. And as a previous poster said, the proximity to the railway meant the price was a bit lower than it would otherwise have been - totally worth it, in my opinion. Oh, and we have never had Japanese Knotweed, and nor (to my knowledge) have our neighbours.