Well, there is an entire broad field known as mathematical physics.
Your primary limitation will probably be time. There are a few issues of "language barriers", curiously enough, as mathematicians and physicists don't always use the same terminology for the same concepts, but that is a minor issue to get around if it is important to you.
You already have to deal with the requirements of the physics department, leaving you less time to also deal with courses in mathematics. Furthermore, the two departments don't always have synchronized schedules: you may find that their seminars conflict, or even that you have courses to attend during the other department's seminar times. This is especially likely if you have some sort of teaching/lab responsibilities. In my graduating university the physics department did their lectures mostly on Tuesday/Thursday, and labs were mostly on MWF. However, the math department lectured mostly on MWF, with discussion sections set for Tuesday/Thursday. Since the professors of both departments tended to prefer seminars and colloquia to occur on lecture days (since that's when the professors were most likely to be there, and graduate students are less likely to have conflicts), this meant that one department's graduate students were doing teaching duties during the other's seminars. If you get an early enough time slot for these duties you may be available, but you are still going to have to measure if you need to spend your time doing other work more--grading, your own homework, preparing for qualifying exams, research, etc. all take time.
However, this is not to say that it's impossible. I've seen physics and EE graduate students in graduate mathematics courses before (and vice versa, by virtue of my being an example). And not simply in applied math courses. I even know of one physics undergrad+master's student who, for the heck of it, took an undergraduate topology course, and was found so exceptionally gifted by the professor that he was given a full TA-ship in the mathematics department so he could spend more time doing mathematics for his last two years or so. Ultimately said student went on to do his doctoral work in particle physics at a top university, has written at least one book in the (vague) area of "abstract mathematics for physicists", and many other things befitting his intellect and character. As a curious aside, some other mathematics students and I had to actually convince him to at least leave with a Bachelor's degree. We couldn't quite get him to go through the trouble of a Master's degree, but I can assure you he could have easily earned a doctorate in mathematics if he had so desired.
There are several physicists who have joint appointments in the mathematics department, and vice versa--though in my experience it tends to be rare when they have a significant presence in the other department, even if they are known and well-respected there--and legions of graduate students who are unsure of whether they should be considered a mathematician or a physicist. At least one of which has made the hilariously inconvenient mistake of saying exactly this to US immigration officials.
So in the actual practice of your research, you can easily cross barriers or reside in grey zones between physics and mathematics, with researchers in either department taking note of your results, and working with you. Your ability to be actively involved with both departments, either as a student or faculty, however, will be heavily limited by time constraints.
