I think you are entitled to do for you, regardless of what anybody else has to say. If that is what will ultimately make you happiest, follow that road.
I'm confused I thought you said you where going to try doing both. I can't tell you what is best for you since their are lots of things to consider like where you would live and all that kind of stuff. But from what you wrote it sounds like you could have tried to do both and I hope you try to do both.
If you really really want to do it and are feeling conflicted, I think you should follow your heart.
You said you might be part-timing. Is that a feasible option?