I doubt that this will really be an answer, and might even add to your stress. I would rather sit down with you for a couple of hours and chat than try to capture all of this "advice from an old codger" in written form.
First, an observation. If you work in a "hot" area of research, mathematics or other, you will be at a disadvantage with respect to collaborative groups. Such groups throw around a lot of ideas, some good, some bad, and find a way to filter out the better ones. It is hard for an individual to gain priority in time in such an environment.
I never had that particular problem, since my mathematical work was extremely arcane (but fun). There were, at the time, only half a dozen people worldwide who had any real interest in it. Math was heading in different directions at the time. So, no, I didn't experience just what you are. But there was a different, perhaps related issue.
My main personal issue was that I was so introverted that it was difficult for me to even speak up for myself when necessary. I would probably have completed my doctorate in four years (rather than seven) if I'd only spoken up to the department head about my need for a better advisor at the time.
But, through an accident of fate I was able to learn that, even though remaining introverted, I could act as if I were not. I could overcome my reluctance, even, to look people straight in the eye as I spoke to them. Painful, but possible. It just requires commitment and practice.
The specific training I got was related to something called Self Defeating Behavior and learned that my natural tendencies didn't need to dominate my actions.
So, the stress increaser, is a suggestion that you find a way to do some collaborations. You don't need to collaborate with everyone or anyone, of course, and you get to choose. But a friendly, small, working group can increase everyone's productivity. Set the rules at the beginning to keep it friendly: equal authorship in all joint work, for example.
Make the collaboration something you do, independent of what you think you fundamentally are.
It will be difficult, at first, if you have any tendencies similar to mine, but, over time, you will find it necessary, even rewarding.
I once had a chance to verify this with a colleague who knew me in both phases of my professional life. He agreed (a bit incorrectly) that I'd changed my personality from complete introversion to a fair amount of extroversion. In fact, I remain introverted, but have learned to act in public when appropriate. It made me a better educator as well. I know others in my field who have made similar transitions. Some of them are very well known in the world of CS.