As a constructivist brother who places as much credence in Platonic Forms as he does in the Irish tuatha da dannan or the Norwegian troll, let me dispute the premise that LLMs do math or have much in the way of semantic awareness.
Human mathematicians are enriched with two cognitive processes that LLMs have no analogue for since the transformer model deals only with syntactic awareness when it statistically evaluates sequences of tokens in a corpus. First, human beings work with chunks of information that go beyond arbitrary short-sequenced token collections. In fact, human grammars are processed by tokens and accompanying morphology at several levels: the morpheme, the lexeme, the phrase, and the sentence. The result of this is that an LLM is blind to all of the concomitant forms of semantics for each type of compositional element. Second, human beings ground those grammars in three major ways: operational, denotational, and axiomatic semantics. LLMs are capable of none of these types of semantics, and obviously without the systems for grounding the elements, no mathematics is actually "being done".
That is not to say that LLMs don't appear to be reasoning. LLMs and various prompt engineering strategies such as chain-of-thought can even craft language that can be run through an automated proof checker. However, the mechanisms by which an LLM understands text is limited to what the NLP community calls text processing, which might best be understood as character and string manipulation grammars and don't go beyond. More sophisticated models of reasoning such as phrase structure grammars, case grammars, and semantic networks have an expanded capacity for semantic action that LLMs do not possess. And those are old techniques which are relatively unsophisticated unlike type-theoretic semantics which finally begin to approach the vision Richard Montague set out in his seminal work marrying natural language to mathematical logic.
With that crash course in formal semantics relevant to grading classes of semantics related to mathematics, what can be shown is that an LLM is missing a fundamental requirement to "do math". LLMs have no teleological aspirations, and therefore are not capable of playing the game of math because they do not understand the rules of math. From the SEP's article Formalism in the Philosophy of Math (SEP):
One common understanding of formalism in the philosophy of mathematics takes it as holding that mathematics is not a body of propositions representing an abstract sector of reality but is much more akin to a game, bringing with it no more commitment to an ontology of objects or properties than ludo or chess.
LLMs simply don't have the operational, denotational, and axiomatic semantics in their engines to receive, manipulate, and succeed in the axiomatic reasoning necessary to do math. Nor do they have anything that remotely resembles the basic natural language ontology (SEP) to have even the simplest of type theories to accommodate the hyperonymy, synonymy, or identity required to relate mathematical primitives. LLMs at best manifest a statistical awareness of the syntax of what a mathematician is likely to say, and therefore possesses mere faint traces of semantic awareness.
As any use-based theory of acquisition of grammar presumes, semantics lies in the usage of the propositions of math, and LLMs simply are not aware of propositional content in any way. Without awareness of the propositional semantics, then the notion they can meaninful engage in the manipulation of propositional content is not possible. They do not have the ability to engage in speech acts nor do they possess any game-theoretic abilities. Therefore, there are many reasons to see Platonic reasoning as metaphysically flawed (such as it's rejection of simplest empirical presumptions of burden of proof about existence granting its ontological extravagance), but LLMs are not evidence against it.
