There exists a submission flow that editors try to manage. Since you talk about "a reputed mathematics journal", I exclude from the answer journals with a poor reputation, which often accept most of their submissions.
For a reputed journal, the publication of high quality content is important, and they rely on editors and reviewers, who generally perform the reviewing task in their free time. One frequent belief, among editors and reviewers, is the following: if an author submits a paper in some language and to some journal, he should follow the journal's guidelines, often including correctness in the expression, typographic rules. This is to ensure that the author has made some efforts to read the guidelines, and to "fit in the landscape" of already published papers. The look somehow precedes the content. Indeed, unless you have a sharp eye, you see the shape before the proof.
This assumes the author has chosen "this journal" among other journals. If the editor, or reviewers, feel that this prior shaping work has not been performed, they feel entitled to reject, or to revise and resubmit. The latter case might indicate that they have found some potential in the paper for that journal.
There might be exceptions: if one detects a ground breaking result, or a paper that would very nicely fit a journal, or a special issue, one might bypass the shaping, and prefer ask for a major revision, to "keep the paper around".
Briefly, to understand such a decision, one should put oneself in the editor's or reviewer's shoes.
