Surely you want to publish it into a scientific journal which
guarantees high visibility to your manuscript, ... Surely you want a journal with the audience who are interested in the paper subject.
These aspects are not independent of the impact factor: For example, a higher visibility leads to more citations, leads to a higher impact factor. You just can not take these out of the equation!
But in the end, is a specific paper P on scientific novelty N published on a journal J1 having impact factor 50 really more important/relevant/better than the same paper P about the same scientific novelty N published on a journal J2 having impact factor 5?
The impact factor is only one (very popular, yet error-prone) metric to measure the reputation of a journal. It is rather a hint that correlates very loosely with the more soft metric that actually counts in the end and that is how the journal is actually perceived in the research community.
Therefore, when deciding between different journals, choosing the one with the higher impact factor is not always the best choice. Every field of research has its own prestigious journals that might or might not have a high impact factor. Especially if the group of researchers is rather small, the impact factor tends to be small, just because there are less overall citations. However, if the most influential researchers publish regularly in this journal, they will also read your paper more likely if you publish there instead of a very generic journal that covers a broad range of topics.
Of course, the reputation of a journal does not make a paper any more important/relevant/better, but it is the other way around: The better your work is, the more likely it is that you can publish it in a journal with a high reputation. There are so many papers published every day that no researcher can afford to read all papers. So even if your paper might be exceptional, the chance that your paper will be read is lower if you publish it in a journal with low reputation.
Will it affect your career?
Because of the reasons mentioned before, it is not the case that the impact factors are somehow combined to generate an overall score that finally counts who gets the tenure, but of course, the committee knows the journals in your field and will tend to choose applicants that published in more prestigious journals just because they know that it is harder to publish there.