How much is too much

I was having an interesting debate with my roommate a couple of days back. I was telling him, if I was allowed to have my way during school times, probably I'd have ended up doing a M.Sc / PhD in Physics. Probably I'd retire early ( around age of 36 ) from my current line of profession and get into atleast teaching Physics for school kids.

He asked me, 'So, why age of 36? '.

'Coz. that's when I'd have earned enough and would not be needing to continue on well paid job to support myself'.

He said, 'Well, You'd end up thinking the same way even at 36. What you'd be earning then would not be 'too much' for you. If you thought you'd be contented with the current life, you'd have quit now and taken up teaching'.

That was a good point to think about. So, what is holding me from pursuing my so called 'dream job' which is teaching Physics. The fact that it's a 'dream job' just in one aspect, i.e Physics. On other aspects regarding pay, challenges, etc. it tends to be not such a nice dream. But, would this viewpoint change at all even at 36. Something that I need to keep an eye on.