Hello!
I don't think you are ever going to be 100% sure before you make your decision (taking into account what you and your husband like in life, your birth experience, wanting to give the other child a sibling etc..) BUT! Whatever decision you make, you will adapt. It's when you don't make a decision that it feels hard. (if you have been back and forth for a long time)
We didn't plan out first one and I was actually going to have an abortion (the timing was awful along with a million other things) but we changed our minds and we are so happy now we made that decision. I wouldn't change it for the world!!!
I can really relate to what you are saying as I'm considering trying for a second one when my daughter is 4. (she will be 3 in Feb) I'm really not worried about the age difference between them. I'll only do myself a favour if I wait. I found the beginning so hard and it took me ages before I recovered from the birth. She wasn't a great sleeper either (still isn't) so taking all this into account, for me, it would be easier if my first daughter is older. I want to give my daughter all the attention now when she is 2 and a half as I think she needs it. My partner on the other hand-don't want another one so I have a problem there..
I'm 35, my other half is 41 so we are aware that we don't have all the time in the world. And like you, I like other things in life too especially my 'me time' and this will get sacrifised for quite some time with a baby (and a toddler!)
I grew up as an only child and it was fine as I didn't know any different. Of course now I know it's going to be hard when my parents get older and they also live abroad so I might have to move back at some point.
My other half is a twin and they never got on growing up, they still don't. It is fair to say that they actually dislike each other..
Whatever decision you make, it is going to be fine. You will find a way, either way.