I am learning geometric algebra, and it is incredible how much it helps me understand other branches of mathematics. I wish I had been exposed to it earlier.
Additionally I feel the same way about enumerative combinatorics.
What are some less popular mathematical subjects that you think should be more popular?
Category theory and algebraic geometry.
I spent a lot of time in undergrad studying things that were kinda nifty, but way too classical to be of any use/interest beyond "fun math". When I got to grad school, category theory was assumed and made some of my courses much harder than they should've been.
In the words of Ravi Vakil, "algebraic geometry should be learned slowly over a number of years". I currently NEED algebraic geometry, so I don't have this number of years. I wish I would've started that a long time ago. Additionally, both of these topics would've helped me learn the things I was thinking about anyways, in particular commutative algebra.
Though I'm sure it's not unpopular, I don't think many people learn it early: Group Theory. It's a real nice area with a lot of cool math and some neat applications (like cryptography).
Lattices and order theory. While these concepts are so ubiquitous, they seem to be banned from mathematics courses. Also, if you know something about order theory, many concepts from category theory turn out to be quite familiar. (E.g. view a poset as a category, then product resp. coproduct become infimum resp. supremum, the slice and coslice category are up and down set, etc.)
I wish I'd learned logic much, much earlier. Obviously young students couldn't handle much depth, but at least a basic introduction to a few concepts would be nice. Just understanding the concept of axioms and deductive rules would put all of math into some perspective. When I finally understood that math was constructed with formal definitions and proofs (or, for example, that there was more than one useful way to axiomatize), I felt I'd been kept in the dark my whole life, doing something (math) that I had absolutely no understanding of.
I wish I'd understood the importance of inequalities earlier. I wish I'd carefully gone through the classic book Inequalities by Hardy, Littlewood, and Poyla early on. Another good book is The Cauchy-Schwarz Masterclass.
You can study inequalities as a subject in their own right, often without using advanced math. But they're critical techniques for advanced math.
Graph Theory is a fantastic field. First, the fact that abstract concepts can be readily visualized makes it engaging for new students. And second, I believe it provides solid foundations into mathematical thinking like proofs, and engages the student to explore other related fields.
I don't know what level of mathematics you are referring to, but here's my opinion after recently finishing my university's undergraduate curriculum.
Firstly, I would like to second Jan Gorzny's reply of Group Theory.
Second, I wish that I had learned linear algebra earlier. The topic usually has two semesters: matrix algebra and then an early proof-based introduction to vector spaces and linear transformations. The real work in this topic can't begin until after both of these classes are completed.
I also wish I had been exposed to topology earlier than I was. Of course there are two "standard" approaches here, and I suppose the approach that introduces general topology before advanced calculus would have better suited my tastes.
Here is a good book that may give some more insight into the heavily debated area where your question lies: Thomas Garrity, All the Mathematics You Missed But Need to Know for Graduate School
Theory of computation, information theory and logic/foundation of mathematics are very interesting topics. I wish I knew them earlier. They are not unpopular(almost every university have a bunch of ToC people in CS depatment...) , but many math major I know have never touched them.
They show you the limits of mathematics, computation and communication.
Logic shows there are things can't be proved from a set of axioms even if it's true--Godel's incompleteness theorem. There are other interesting theorems in foundation of mathematics. Like the independence of continuum hypothesis to ZFC.
Theory of computation showed me things that's not computable. Problems that takes exponential time, exponential space, no matter what kind of algorithm you come up with.
Information theory proves the minimum amount of information required to reconstruct some other information. It pops up in unexpected places. There is a proof of there are infinite number of primes by information theory (Sorry I can't find it, I can only tell you it exists. I might find it later).
Statistics is the topic in which I am still poor and it is still useful to me which I learned so late and that's why I am poor in Statistics.
I wish I'd learned about special functions earlier. The subject is a treasure trove of results that were commonly known a century ago but now few people know.
I don't really think that graph theory is a "less popular mathematical subject," but I certainly wish I had been exposed to it earlier.
I did mathematics as an undergrad, and I thought that differential equations were boring and pointless. Type of diff eq -> existence and uniqueness proofs for solutions -> rinse and repeat. Yawn.
But now I find my lack of knowledge of differential equations is hampering my learning some interesting parts of physics that I'd like to know more about...
My answer is: fundamental concepts and methods of both first order logic and set theory. I really wish I learned them much earlier, since all mathematics is based on them.
Information theory.
Incredibly deep field - it will have you perceive the world in a completely new way.
Not unpopular, but I wish I had studied the theory of Rings and Fields, and basic Topology earlier, because in my opinion both these branches appear in many interesting subjects studied relatively early during one's undergraduate studies. For example, the whole concept of a minimal polynomial of a linear transformation $T$ is more intuitive (at least for me) when viewed as the generator of the ideal of polynomials such that $P(t)=0$. Metric Spaces (studied in the introductory Topology course at my university) appear as early as in calculus, and are generally a basis to many definitions there. Also, many algebraic structures can be viewed as topologies, which can sometimes give a new insight or assist in proofs (a favourite of mine is a topological proof of the infinity of the primes, which can be found in Proofs from the Book).
Riemann's Explicite Prime Counting Formula: $$ \pi_{0}(x) = \operatorname{R}(x) - \sum_{\rho}\operatorname{R}(x^{\rho}) - \frac1{\ln x} + \frac1\pi \arctan \frac\pi{\ln x} $$ and all the theory of Dirichlet functions behind it...
