A shared holiday home will always be a tricky compromise unless you're lucky enough to be completely on the same page as your co-owners for all the decision-making that comes with owning a property. You can and should explore in advance what might be potential pitfalls and draw up an agreement, but this will never cover every eventuality that may arise, as it's a purchase for the long-term and tricky to extricate yourself from.
I speak from experience, as my father and some friends were practically given a decrepit cottage when in their 20s, which they did up together and holidayed in over the years - losing two of the friends to arguments along the way. It was fun for us as kids but has now passed to be my generation's responsibility, so we're trying to manage it between 5 families. If it belonged to just me, I would invest proper capital to do it up, put in central heating and better amenities and then let it out at decent rates when we weren't using it. But not all the others can or want to afford to renovate, and are happy with it being 'rustic', so it limps along, shored up with DIY efforts and not quite comfortable or nice enough for us to want to spend much of our holidays there or for it to attract many paying visitors. It's become a bit of a millstone, frankly, as there are ongoing costs such as council tax which are payable whether it's in use or not and regular emergency repairs.
Although you're only talking about two families, I would be very wary in your situation. Firstly because both will be tying up some capital in it, so there will be some expectation of it being an investment (which I don't have in my case) so there will be strong opinions on what will be the best return. And secondly because of the difference in what you will be able to contribute financially subsequently - which you are already noticing in the discussions. If you can't pay half of the co-owners' proposed modernisation plans, they may subsequently resent this and feel frustrated that they can't get the house to the standard they want. If they pay more than half, then you will always feel it's more theirs than yours and that you ought to let them have first refusal on bookings etc, which won't feel satisfactory to you. There's a big risk anyway that one family will end up getting much more use of the house and the other will feel guilty if they then want their money out.
Shared intention, clear expectations and good communication might get round these problems - my advice would be to use the initial discussions that you're having to assess the extent that you have these and to trust your instinct on whether it will work or not. Good luck!
