A pure math major with no physics classes beyond those required for an undergraduate degree is going to be woefully unprepared for graduate school in physics. There is, in fact, relatively little overlap between pure math and theoretical physics; vector calculus, linear algebra, and group theory are about it. (I double majored in pure math and in physics as an undergraduate, and there was very little useful overlap between the two, although some more subtle connections do become apparent at the advanced graduate level.) I would be very surprised if graduate program in physics would admit somebody with no physics beyond two introductory semesters of mechanics and electromagnetism.
Many (although not all) graduate programs in physics will let their less-prepared graduate students take (or retake) senior-level classes in classical mechanics, quantum mechanics, statistical mechanics, and electrodynamics. However, if you have little to no familiarity with the intermediate level physics topics, like waves, special relativity, thermodynamics, statistics (even people doing research in purely theoretical physics generally need to have to have a very firm understanding of statistical methods), etc., you are simply not going to be prepared for physics graduate school. An undergraduate major in applied mathematics is much more likely to cover material relevant to graduate study in physics.
Many of the strongest physics students double major in math, since a physics major already typically requires a certain number of advanced math classes (selected from things like vector analysis, linear algebra, group theory, probability theory, etc.). And there are also people who major only in math, but take enough intermediate- and advanced-level physics classes as well to make them plausible candidates for graduate school in physics. Some physics departments (such as mine) would seriously consider math majors with this level of preparation, but others will not even look at applicants without a full program of quantum mechanics, statistical mechanics, etc. classes. So if you are hoping to go to graduate school in physics, you will probably need to take a significant number of intermediate physics courses, at a minimum. (Sometimes, these courses may not actually need to be taught by a physics department though. There are often applied math classes covering wave behavior, for example, that could be suitable substitutes.)
