Addressing your question 1
I can think of several arguments supporting both the separate paper idea and the appendix option:
Arguments in favor of an appendix
Since your adviser proposed it, on the assumption that they are experienced, have a clear view of the situation, and have your best interests at heart, that suggests that going along with their suggestion is the best course of action.
Even if they don't fully have your best interests at heart, going against their suggestion might antagonize them, reduce the chance of the fruitful collaboration you have with them continuing, and/or lead to other not so good effects. All other things being equal, it is best not to get into conflicts with one's adviser.
The appendix will likely end up being published in a better journal than your contribution written up as a stand-alone paper.
The appendix will make you an official collaborator (though not a coauthor) of two big named experts in the field, creating an association that may be impressive to other people from the field.
Arguments in favor of a stand-alone paper
In my entire math career I've only seen one — maybe two — papers that had an appendix authored by a person who was not listed as a coauthor of the paper. In other words, it is a highly nonstandard type of authorship credit. That means that if you choose the appendix option, it will complicate things for you whenever you are communicating your achievements to other people, requiring various asterisks, footnotes, and complicated explanations (with attendant risks of misunderstanding) in your publication list and when you discuss your publication record. In other words, it goes against the KISS principle, and I would probably avoid this type of arrangement myself if at all possible for that reason alone.
Because the arrangement is so non-standard, I can't help but wonder what your adviser is thinking exactly in proposing this idea, and whether they in fact have your best interests at heart, or are proposing what is best for them. Another possibility is that they are simply a kind of person who is well-intentioned but isn't very thoughtful about things having to do with real life and who sometimes has ideas that don't make a lot of practical sense. Thus, although I can't say for sure that their proposal doesn't actually make sense, some caution seems warranted.
To counter the point mentioned above that "the appendix will likely end up being published in a better journal than your contribution written up as a stand-alone paper", it is not at all clear that being an author of an appendix (who is not listed as a coauthor of the paper it accompanies) in a very impressive journal will be perceived as a greater achievement than being a sole author of a paper published in a less impressive journal. Depending on the specific journals, the opposite could be true.
Being a sole author of a stand-alone work sends a clear signal about your ability to do independent research. So it's good to have at least some single-authored papers on your CV. (However, not everybody seems to care about this, and I have seen plenty of successful mathematicians who have very few solely authored papers or even none at all. And I should mention that many of my own papers are solely authored, so that's possibly one reason why I think that such a thing is impressive :-)).
Bottom line. There is no obviously correct answer here, only trade-offs between different things you can be trying to optimize for.
Addressing your other questions
- [...] it will look more reasonable (from their perspective as well as the referee's) to put my work with them to obtain a full result rather than two separate papers [...] So if having a separate paper is better than being an appendix author, am I hurting myself by solving a problem quickly?
I partially disagree with your premise that "it will look more reasonable [etc]". Perhaps from your adviser's point of view that results in the cleanest way to publish what they consider to be an important research contribution. Certainly it's nicer when the research literature is all published in a nicely organized way with conceptually related results appearing together instead of being fragmented across multiple papers; so possibly your adviser is (reasonably) guided by that type of consideration. However, it's also true that everyone in the business understands that contributions made by different people usually appear in separate papers, and that as messy and annoying as that may be, that's just how academic publishing works. So I'm pretty sure that no reasonable referee will criticize you for publishing your contribution as a stand-alone paper, or think that doing things this way is "less reasonable".
That being said, there are indeed situations where you may be "hurting yourself by solving a problem quickly", but for a different reason. The reason is that the importance of an open problem usually grows over time. If author A publishes a conjecture or a partial result and then author B proves the conjecture or improves the partial result a short time later, most people would conclude that the improvement wasn't that difficult. But if the problem stands open for 10 years, or 20 years, or 50 years etc, then people are much more impressed by the later improvement, since they assume that since it took 50 years to come up with, that must mean that many people tried and failed to achieve that improvement, and that therefore there must be some very ingenious idea that was required to achieve it.
Of course, it's not practical to delay publication of your ideas by 50 years. Even if you delay by one or two years (which still has some hypothetical benefit of making your contribution seem at least a little bit more impressive), you run the very real risk of being scooped by someone else who doesn't mind appearing to solve a problem immediately after it was presented.
So, all things considered, while there is a little bit of a perverse incentive in this type of situation where you might get a bigger reward for taking more time to do something, I would not recommend ever intentionally solving a problem slowly when you can solve it right away. Still, your question about this is actually very reasonable and not silly at all.
- Since I don't know the techniques in topology/knots very well from two other sections that are used to prove the main theorem (or rather I did not contribute in those parts), claiming to be a coauthor might not be reasonable. However, I wonder in what situation similar to mine one should qualify to claim coauthorship?
It's hard to know for sure without knowing the details, but from your description of what happened, I'd say you likely deserve coauthorship merely based on the fact that "the result you obtained allows them to fully generalize the main theorem in their preprint". Like I said, I think the idea of an appendix without coauthorship is so non-standard in math publishing that I would never suggest it to anyone except in some very unusual set of circumstances (and likely never to a student), and I wonder what your adviser is thinking in suggesting it. Coauthorship is the standard approach for the scenario you are discussing, although a separate publication bearing only your name, or your name and that of your adviser, is also a valid way to proceed.
As for the fact that you don't know some of the techniques the other collaborators are using, that is completely irrelevant; for example, I am a coauthor on many papers that have sections where I didn't know the techniques my collaborators were using. Indeed, some might say that this is precisely the point of collaborating with other people! You get to create something that combines the knowledge and expertise of several people and achieves something none of the individual contributors could have created on their own. If everyone knew the same techniques or had the same ideas there would be no need for a group effort.
