Short answer
It's messy.
Background
The classical psychedelics (mescaline, DMT etc.) are derived from plants (peyote, ayahuasca) and have a long history of use by shamans and originally often mainly served religious purposes, before being exploited by the New World peoples for recreational (and occasionally scientific) purposes. LSD is also considered to be classic yet it is a synthetic derivative, but readily made from lysergic acid derivatives extracted from the mold Claviceps purpurea. These classic psychedelics are indeed all, primarily, 5HT2A agonists.
Having sad that, 'psychedelic' is a classical term meaning 'mind opening', and was first coined back in the 1950's. It's a broad, ill-defined term often associated with illicit drugs, hippies and alternative rock. The term serotinergic drug is more specific, while the term '5HT2A agonist' is yet even more specific. The former also includes MDMA, which is imo mind-opening, yet (generally) not hallucinogenic. The latter includes those drugs that produce profound hallucinations, i.e., the classic psychedelics.
This is the reason I tend to move away from 'psychedelic' as often soft drugs like THC are also termed 'psychedelic'. Hallucinogenics are a more specific class of drugs and is the one I think you are referring to, i.e., those compounds producing hallucinations, i.e., sensations that are not there (a hallucination being 'a perception in the absence of external stimulus that has qualities of real perceptions').
Now what about non-serotinergic hallucinogens? Well, there are particularly potent hallucinogens that are primarily not serotinergic, but glutaminergic, particularly the dissociative anesthetics. These include PCP and ketamine that are hallucinogenic as they
can produce visual and auditory distortions and a sense of floating and dissociation (feeling detached from reality)
However, here again it comes down to a matter of definition; a 'true' hallucination is a percept in the absence of a stimulus; a distortion is more akin to an illusion, namely
an instance of a wrong or misinterpreted perception of a sensory experience
In addition, most and perhaps all hallucinogens are 'dirty' drugs, in that they affect multiple neurotransmitter systems and therefore have broad effects. MDMA for instance is psychedelic because of its 5HT effects but definitely has stimulant, dopaminergic properties too (having the methamphetamine backbone this isn't surprising of course).
In all, not only is the scientific terminology messy, also the science behind it is messy, as 5HT2A agonists may still have effects on other neurotransmitter systems too. This can explain why every 5HT2A drug has slightly, or even grossly different actions than structurally related ones.