Well more power to you for wanting to become an Engineer. Hate to say it but both Math and Physics came easy to me, in fact if anything I struggled a little with Math. But as you seem to be hell-bent on the path, you appear to have worked hard to get up to speed with Math, and I am sure given a little time and lotsa effort you will get there with Physics. Both are beautifully logical once the basics are understood so do not be put off.
I did my degree while working a regular 9-5 as a Design Draftsman and then off to class in the darkness, sometimes for four or even five nights in a week. I was (some say still am) a driven individual but I find I enjoy the pressure and I really wanted to get up to that next level. So hang in there, you sound like you have the right approach to do well.
As for sanity, I am not qualified to speak!