"Good enough for journal X" is not a mathematical claim. Operationally speaking, it is precisely a matter of collective opinion. Math journals group themselves into rough, approximate equivalence classes (which is also entirely a matter of collective opinion), and you should take this negative referee result as one data point that your paper is not good enough for a journal of that class. If your prior belief was that the paper was roughly equal or better in quality to other papers published in journals of the same class, you should try at least once more at a different journal of about the same level.
A good referee report reveals something about why the referee thinks the quality of the paper is not sufficient, either in absolute or relative terms. This may give you something to improve: either actual mathematical improvements or improvements in the exposition to make referees believe your work is more valuable.
In summary: it doesn't necessarily mean much on its own. Revise if applicable and resubmit. Good luck.