Granted this was decades ago, but I had a friend who graduated from St. John's in Annapolis with a degree in mathematics, and though he did not, he certainly could have gone on to a PhD program in mathematics. Looking at the program (https://www.sjc.edu/academic-programs/undergraduate/subjects/mathematics), the emphasis is certainly more classical (geometry, set theory, number theory, mathematical physics) than most other programs, but ultimately if you're pursuing a PhD in mathematics, the undergraduate degree is more about learning how to address and solve problems...how to work your way from "huh?" to rigorous proof. Very few students enter graduate school without what some would consider "holes" in their undergraduate mathematics resumé. My undergraduate program did not include classes in number theory or topology; other students in my graduate classes had different deficits.
The lack of "modern mathematical research" opportunities can be remedied in multiple ways. At St. John's you are less than a mile from the US Naval Academy, which has a large, active mathematics department. Understanding the security situation is different there from most institutions, you may be able to attend their research seminars, and get to know people there...a faculty advisor at St. John's may be able to facilitate. University of Maryland and UMBC are both within a 45 minute drive, as well. You may be able to hook into these institutions for activities like the Putnam Exam and the Mathematical Contest in Modelling, if St. John's does not host.
I'm not saying it will be easy, but as St. John's is the environment in which your interest in mathematics seems to be blossoming, I'd be loath to recommend you leave just for a potential future that time may steer you away from.
UPDATE
I'd like to address several criticisms of my thoughts that have popped up in the comments. The first is that the St. John's curriculum does not prepare a student for modern mathematics. The world may have changed in the decades since I went to graduate school, but I don't remember jumping into cutting edge research during my first year classes; rather it was about strengthening skills and deepening knowledge in core disciplines. One can learn to study mathematical texts and write proofs from 400 year old mathematics as well as from 100 year old mathematics. I agree that this is a harder way to go about it, and the student will enter with knowledge deficits. But that doesn't mean impossible, and there will be skills that students learn in the St. John's curriculum that will put them in a better position than many others to close their knowledge gap.
The second criticism is that the student will need to leverage outside resources. Well, yes, but isn't that true for most other mathematics programs? At how many schools is a student who simply takes the classes required for a mathematics degree prepared for graduate school without some level of external preparation? Out-of-class research, enrichment activities like the Putnam exam, and attending seminars are always encouraged for students looking to attend graduate school. Again, this will be harder at St. John's, but with at least three active research departments within an hour's drive, it is not impossible.
Several commenters, and other answers, suggest that the OP should just change schools. And if the OP were attending a smaller state institution or liberal arts school I would likely agree with that advice. But St. John's is such a different environment from other schools that such a transition is not risk-free. My friend who attended St. John's told me several stories of some of his eccentric fellow students, and admitted that he himself probably would not have been successful at a more traditional institution. My answer is attempting to meet the OP where the OP is, and offer strategies that the OP can use to increase engagement with mathematics, without second guessing a personal choice to attend an unusual institution. I freely acknowledge that this will be a significantly harder path. But it is a path that can absolutely lead to a fulfilling career in mathematics, because I know someone for whom it worked.