One additional factor is the US's dominant election method: first past the post. What this means is that it is simply the candidate with the greatest number of votes who wins each seat, with no option to specify preferences.
It is very difficult for third parties to make any kind of progress in first past the post systems, as few people will be willing to vote for a third party when that means they don't get to express their choice about the more dominant parties. So the third parties rarely get many votes, and they don't gain mindshare. This is called Duverger's law. By contrast, with an election method that allows you to specify preferences there are no downsides to giving your first vote to any party at all. So even when they don't win, third parties do get substantial numbers of first preferences, and they can therefore build in prominence until they manage to win seats.
The US already has a party which many would see as being at least partially aligned with libertarian ideology. With no option to specify preferences, voting for the actual Libertarian party would split the vote making it even less likely that someone who represents their position would be elected.