I loved Math in undergrad. When I got to grad school it was no longer fun. I didn't enjoy going to my classes, the material was boring, my professors spoke poor English, my classmates were not native English speakers either so study groups weren't a thing. Maybe I would enjoy it more now at 32 than I did at 22, but it wasn't for me.

I took 3 classes my first semester: logic, abstract algebra, and analysis. I got an A-, B+, and a C+ and found out the C+ was not a passing grade. I also started falling too far behind to keep up with the required exams I had to pass at the end of first year, didn't want to continue teaching classes, and math PhD is not guaranteed after research is done. In other subject areas if you do the research and get an unexpected result, you still have a thesis. In math you need to come up with something new. I wasn't ready to commit to a 5 year program that I wasn't that enthusiastic about and that may take longer if I hit roadblocks.