There is no right answer for this question. And the answer I am about to give is solely based on my personal experience as a Mathematics student.
Self studying Mathematics is not an easy task. I was lucky enough to have great professors at university. Most of them had a vast teaching experience. After teaching something for so long, you start to notice a (not regular) pattern. You start to realise that there are some topics that an undergraduate student can easily learn but there other topics that are not. In this way, you start to develop your own pedagogical approaches to the subject as you are able to predict what the difficulties are going to be. And this is most certainly important because now you can help and guide your students. When you study Mathematics by yourself, you lack this “compass”. Although a professor is irreplaceable, you can adopt some strategies. Here are some tips.
Read carefully the textbook. You can’t learn Mathematics, if you don’t read Mathematics. You should always try to make an effort to understand every sentence that you read. Don’t take things for granted. I’m not going to lie, this is not easy (at least at first). But it certainly gets better with perseverance.
Do the exercises. You can’t learn Mathematics, if you don’t do Mathematics. The problems at the end of each chapter/section will play a big role here. They will be the main way you have to make sure you are doing a good job and you are understanding things.
Just a side note for these two points. It is easy to lose track of your way, while following 1 and 2. Imagine you are self studying linear algebra for the first time and you come across the statement of the Fundamental Theorem of Algebra.
Yes, I said that one should never take things for granted. But it highly depends on what those things are. In this case, my advice would be to take this theorem for granted (for now). Later, you will be able to prove it (if you continue your studies). Without any backgorund on Topology or Complex Analysis, I don't believe that would be able to come up with a proof of it.
[Any introductory textbook on linear algebra will probably say something like “we will be using the Fundamental Theorem of Algebra that states (blah blah blah). Its proof is outside the scope of this book and can be seen in [References].”].
Also, as said, there are problems that are not meant to be fully solved, in the sense that what is more important is the process of "chewing" on it than the resolution itself. Sometimes the questions that arise from a problem and the chain of thoughts that occur to you in attempting to solve them are more important. For example, some problems require a use of a lot of different notation. One can easily get lost in the labyrinth of letters. But if you have understood the problem and come up with a solution to it, you are doing great. Of course you should always try to turn the idea into a rigorous solution. But don’t get sad if you can’t. This will happen a lot.
In this two points, a professor is a valuable help. Because the professor can make sure that you
- don’t waste too much time. This may seem paradoxical. On the one hand, there are several ideas that take quite a few to grasp. On the other hand, spending too much time until you fully grasp it may be harmful. Imagine you are self studying set theory. You are most likely to start with naive set theory. But if you are a curious person, you may be tempted to jump right away into axiomatic set theory. Well, this is a mistake. Yes, you should be curious and persistent — take all the time you need to understand things. Once you have a good understanding, move on. There will be a lot of things that will only make sense once you move forward to a new topic. Can you imagine a first year undergraduate student learning category theory without a solid mathematical background?Just stick to what the textbook is all about, and do the exercises.
About doing the exercises. I had some colleagues that used to solve (not necessarily accurately) every single exercise from the problem set given by the professor. There would be (for example) ten exercises concerning matrix arithmetic, and they would do all of them. Is this really necessary? Well, if you have doubts, it may be a good idea. But if you have already understand it, then do one or two out of the ten, and move to next topic. The time that you will save by doing this can be spent in studying something more difficult that requires more time. In short, study smarter, not harder.
[Fun fact: I didn't solve every single exercise. But my grades were way better than those who did it. And I am pretty sure that's because they wasted too much time doing routine exercises instead of taking that time to study the more advanced stuff].
But I haven’t said anything about how you can check your knowledge. Well, there are a few things that come to me.
Search for course material. There are a lot of universities with good online resources. For example, the MIT OpenCourseWare. There you can find lecture notes (and even lecture videos), problem sets, and midterm and final exams (and some of its solutions). In this way, you can check your performance.
Use StackExchange. We are here to help. There a lot of good mathematicians here. Each with its own vision and understanding. And they all can contribute to your formation.
Find a study companion. As Accelerator said in the comments, having someone to talk to can help you a lot. Have you ever heard of Feynman technique? If you understand something, you must be able to teach it to someone. If you have someone to study with, you can try to explain them your ideas. This is a good way to check how good is your understanding of something. And I’m sure that are a lot of people who are willing to do this (me among them).
There is one last thing. And a really important one. It is easy to get sad or frustrated while self studying Mathematics. Forget the idea of the solitary mathematician, isolated from the world, living fully on the realm of ideas and abstraction. We are social beings. We need to communicate. We need a way to somehow express our insecurities and fears.
When I was an undergraduate student a lot of my professors helped me with my self doubts and insecurities. They gave me the confidence that I needed to learn Mathematics. Their professors also gave them the confidence they needed to become the mathematicians that they are today. And so on. As cliche as it may sounds, we need to be there for each other because Mathematics is not a single player game.
I really encourage other users to share their experience. We can all learn from each other.
— Edit —
On doing exercises.
- Routine exercises. I must emphasise that routine exercises are also important. They are most likely the first ones that you should do when you learn something new. They will work as a first check point of your understanding. Solve a few of them. If you did them correctly, you are good to go. Else you should review your definitions.
- Exercises asking for examples. These are of major importance. A good mathematician should always know an example and a counter-example of something. You can actually do this while you read your textbook. For example, imagine that you are learning Real Analysis and you have just learned Rolle’s theorem. It is always a good idea to look at each hypothesis of the theorem and to ask if they are really necessary. This will strengthen our intuition and improve your reasoning (and it will also give you a hint on how to prove the theorem).
- More theoretical exercises. Don’t fear them. Try to do them. And don’t get sad if you have struggles when trying to solve them. That is completely normal and is part of the learning process. These kind of exercises require you to be really comfortable with the material. But not being able to solve them does not mean that you are doing bad. Not being able to solve them only means that there are some ideas that need more time to settle down. As you study, your mathematical maturity grows. So, try to solve them. If you succeed, reward yourself. Else, be persistent. Try to fully focus on the exercise. But watch out the time you spend on it. For example, after 1 hour, if nothing has come up to you, just let it go and move to the next thing. This can be difficult (at least for me). Try it again later (one day or week after).
A curious thing regarding the last bullet: when I am working on a hard problem and can’t figure out the solution, I usually try to forget it and do something else (even if it is not related to Mathematics). Amazingly, this is when the solution usually appears. Or at least it allows me to go back later but with a new perspective.
When solving exercises, you can use this website. There are a lot of smart people here that can provide you with hints to the more difficult exercises. Always prefer the hint instead of the solution. Use the solution only to check your work.