I partially disagree with the answer given by astronat, since I think it does not reflect how common it is in mathematics to rediscover existing results (and to publish them without knowing that they already exist). So here is a somewhat different perspective:
Terminology.
What the OP calls mistakingly publishing existing research is, in my experience, more commonly and with a less negative connotation, referred to as rediscovery of existing research.
Distinction from plagiarism.
This point has already been explained in astronat's answer: independently rediscovering a result (and publishing it without knowing about its former existence) is something completely different from plagiarism.
Consequences.
What will happen if you publish a result which is already in the literature but you have not been aware of?
Nothing.
The math departments of this world would be empty halls if every mathematics professor who has unconsciously published a theorem or idea that already existed in the literature was banned from the university. (See the point "Prevalence" below.)
Of course, if this happens very often to you, this might have a negative impact on your reputation; but it happens to almost everybody now and then, so there is not much reason to worry in case that it does happen to you. (Unless in very, very obvious situations - if, for instance, I claimed to be a functional analyst and then published a "novel" result which turns out to simply be the Hahn-Banach theorem, chances are that I would have a much harder time to be taken seriously from now own. But this really refers to extreme cases only.)
Prevalence.
This is one of the points where I disagree with the assessment in astronat's answer. Experience shows that results are rediscovered and re-published (without the authors knowing or citing the earlier occurrance(s) of the result) all the time.
I could write a lengthy list of examples where this happened in papers I read or to people I know or to myself. But there are much better sources for this claim than my personal experience.
Just to get an impression, see for instance these two MathOverflow posts (post 1, post 2) or this blog post by Terry Tao.
Some reasons why this happens.
In earlier days, mathematical results could not be distributed as quickly as they are now (in particular, due to the internet).
Even today, access to many results is somewhat restricted, for instance if they are published in somewhat obscure journals (or, say, proceedings of small conferences) which are hardly accessible, or if they are published in languages which are not widely spoken.
While we have the internet and search engines available nowadays, the amount of published mathematical literature has also become much larger than only a few decades ago, which makes it literally impossible to know everything that could be relevant to your work.
Moreover, search engines are only useful if you know the right key words to search (which is often not the case due to varying terminological conventions).
There a many different ways to phrase or describe a mathematical result (and every more so for a mathematical idea). So even if our idea / argument / theorem is essentially present in a paper and even if we have a close look at this paper, it can sometimes be very hard to recognize it there. Since there are numerous papers which we ought to take a look at before we publish a result, there is simply no way to read all related papers in depth, and we will probably spend more time reading those papers which are obviously very closely related to our work. So if a similar idea occurs in a related, but not closely related paper, and if it is worded quite differently from how we would phrase it, there is a considerable chance that we just miss it.
Many ideas are so fundamental that they occur in many different fields of mathematics (and in many different forms). Our knowledge of a particular mathematical field will typically depend monotonically on how close the field is to our own field. So if a similar idea or result occurs in another area, it is quite likely that we are unaware of it. (The fact that different mathematical areas often use quite different terminology and notation for the same thing doesn't make it easier.)
In my experience, people tend to publish a lot of research work, while it is less common to write expository work (to some extent, this is certainly due to incentives imposed by typical career options in academia). For instance, I can think of various mathematical topics where, in my opinion, a comphrehensive survey article or an up-to-date monograph is missing. This is part of the problem since well-written surveys and books are an excellent aid in identifying relevant earlier results.
Some points to be aware of.
While results or arguments or ideas are often re-discovered, it happens more rarely that two papers have very similar content. Quite often, while some results in a paper might already be in the literature (but unknown to the author), there will be other results or insights in the paper that are still new. This is closely related to the point about novelty that user2768 made in their answer.
Even if some results in two papers are very similar, it will sometimes (though not always) happen that their proofs are quite different.
What to do?
That's a difficult question, and the answer that any particular mathematician gives to this might depend very much on their personality. Here are a few suggestions (biased by my own point of view, of course):
Obviously, we should carefully study the literature before we publish something. Citation databases (in mathematics these are, in particular MathSciNet und zbMATH) can be very helpful for this.
The more work we put into putting our results into context, the more we help potential readers of our paper to also become familiar with earlier research and results on the topic. For instance, there is a considerable difference between merely citing papers and explaining, if only briefly, the relevance of these papers. For instance, compare the following to ways to cite relevant literature:
-- "Recently, there has bee a lot of work on this topic [1, 3, 9, 10, 12, 15, 21, 23, 24, 25, 31, 36]"
-- "Interest in the topic abc originally stems from the study of xyz spaces, because their geometric properties can be classified by abc [9, 10, 36]. Later, abc has been extended to a more abstract setting in [1, 3, 21, 23] which made it possible to also classify the geometry of xyz' spaces [24, 25]. In the present paper, we build on the so-called fgh approach to abc recently developped in [12, 15]; combining it with the, at first glance unrelated, ijk technique from operator theory published in [4], we show that all major results in abc theory are actually independent of the foo assumption. This allows us to classify the geometric properties of very large classes of xyz* spaces. In addition, we obtain recent results from [31] as special cases of our general theory."
When writing a referee's report about a paper, we should take much care to point out missing references. In particular, I strongly support the idea that, as a referee, we should point to relevant articles by different authors, and not only to our own (obviously super-important but - heaven knows why - completely underappreciated) work which might not have been cited in the paper under review.
If we have made a preprint publicly available, and - before publication - we get aware you relevant literature that we have missed, we should, of course, make a serious effort to include it in the paper before publication.
If it is too late and you get aware of relevant references after publication, don't rack your brains over it. As explained above, this happens very often. Just make sure to properly include your newly gained knowledge when you write another article on a related topic (of course, this does not only mean to cite the references you now became aware of; it also means to take these references as a starting point for yet another literature search).
It is, in my experience, very uncommon to take any formal actions (such as notifying the journal, or even submitting an erratum) if you become aware of prior relevant work after publication. (But please note that I don't make any assessment as to whether I approve or disapprove of this practice; I'm just pointing out what is common practice.)
