The Dead Men and the Nazgûl were essentially both "undead" characters. While they were "undead" for various reasons, with the Nazgûl having been semi-resurrected and the Dead Men not properly dead because of a curse, I cannot find much other difference between them.
What are the other main differences between these characters?
Annoyingly, this is a question that seems to have a different answer in the books than in the films. I'll do my best to give both interpretations.
There's not actually any indication that the Nazgûl are dead; rather they appear to be very much alive, but their physical forms have been subsumed by the Unseen world (emphasis mine)1:
'A mortal, Frodo, who keeps one of the Great Rings, does not die, but he does not grow or obtain more life, he merely continues, until at last every minute is a weariness. And if he often uses the Ring to make himself invisible, he fades: he becomes in the end invisible permanently, and walks in the twilight under the eye of the dark power that rules the Rings. Yes, sooner or later - later, if he is strong or well-meaning to begin with, but neither strength nor good purpose will last - sooner or later the dark power will devour him.
Fellowship of the Ring Book I Chapter 2: "The Shadow of the Past"
And an early draft of Fellowship describes the "wraithification" effect, though the canonicity of Tolkien's early drafts is a matter of personal preference:
Yes, if the Ring overcomes you, you yourself become permanently invisible - and it is a horrible cold feeling. Everything becomes very faint like grey ghost pictures against the black background in which you live; but you can smell more clearly than you can hear or see. You have no power however like a Ring of making other things invisible: you are a ringwraith; and your clothes are visible, unless the Lord lends you a ring. But you are under the command of the Lord of the Rings.
History of Middle-earth VI The Return of the Shadow Chapter III: "Of Gollum and the Ring"
We can see this most clearly by the fact they they're able to interact with the physical world; they ride horses (and Winged...things), wield swords, and wear crowns:
The Black Rider flung back his hood, and behold! he had a kingly crown; and yet upon no head visible was it set.
Return of the King Book V Chapter 4: "The Siege of Gondor"
If they were actually dead, unhoused souls, this would not be possible.
Which brings us to the Dead Men. It's notable that there's no indication that the Dead Men actually have any real power in the physical world:
[S]uddenly the Shadow Host that had hung back at the last came up like a grey tide, sweeping all away before it. Faint cries I heard, and dim horns blowing, and a murmur as of countless far voices: it was like the echo of some forgotten battle in the Dark Years long ago. Pale swords were drawn; but I know not whether their blades would still bite, for the Dead needed no longer any weapon but fear. None would withstand them.
Return of the King Book V Chapter 9: "The Last Debate"
Although we can't be absolutely certain what would happen if a Dead Man and a Nazgûl threw down, I'd be inclined to suggest it would be a stalemate: the Nazgûl wouldn't be able to defeat the Dead Man, because you can't kill a soul, and the Dead Man wouldn't be able to defeat the Nazgûl, because they can't interact with the physical world.
Unstoppable force, meet immovable object.
Annoyingly, Peter Jackson threw most of that stuff to the wind, and the line between the two is much blurrier.
It's actually not entirely clear whether Jackson's Nazgûl are alive or dead; when Frodo is suffering from the Morgul-wound, Aragorn implies that they're not:
Pippin: Is he going to die?
Strider: No. He is passing into the shadow world, he will soon become a wraith like them.
Fellowship of the Ring (2001)
But in An Unexpected Journey, dialog suggests that the Witch-king, at least, was killed and then brought back by some force:
Galadriel: When Angmar fell, men of the North took his body and all that he possessed and sealed in the High-Fells of Rhudaur. Deep within the rock they buried them, in a tomb so dark it would never come to light.
An Unexpected Journey (2012)
We do know they they are (or can be) tangible, since they still ride horses and wield swords and stuff.
Frustratingly, the Dead Men also appear at least quasi-tangible; Aragorn is able to block the sword of the King of the Dead with Andúril:
And then grabs his throat:
And, of course, the Dead Men actually kill a whole ton of Orcs at the Battle of the Pelennor later on.
But they're also sometimes intangible; Legolas shoots an arrow at the King of the Dead, which goes right through him with no harm done.
There's really no explanation given for any of this, besides "story," so it's hard to say with any kind of certainty what would happen if a film-Dead Man and a film-Nazgûl got into a disagreement. It seems unlikely that the weapons of the Nazgûl would be able to harm the Dead Men, since that seems to be an ability specific to Isildur's heir. But it's also not clear that the ghostly weapons of the Dead Men would have any effect on the Nazgûl; no other weapons do, aside from the torch that Aragorn stabs one of them in the face with.
I want to say that the outcome would be the same as in the books, but I say that with much more trepidation. There are so many inconsistencies here, it's hard to make a definitive statement.
1 Nod to ulmo for suggesting this quote here
Difference between Nazgûl and Dead Men of Dunharrow:
The Nazgûl were once nine great Kings of Men, all of whom were given Rings of Power by Sauron. The Nine took them and used them to achieve great power, wealth and prestige in life. But as time passed, the Rings continued to exert their corrupting influence. The Rings eventually left the kings spectral, invisible to all but those who could see into the wraith world, and slaves to the will of Sauron. Their lives and power became bound to Sauron's via the One Ring; as Sauron grew or diminished in power, so to did the Nazgûl.
The Army of the Dead, also known as the Dead Men of Dunharrow or Oathbreakers, were the ghosts of deceased Men of the White Mountains, cursed to remain in Middle-earth by Isildur after they abandoned their oath to aid him in the War of the Last Alliance. They haunted the caverns beneath the Dwimorberg, and the valley of Harrowdale that lay in its shadow, though they were said to appear in the valley only in times of trouble or death.
And regarding the showdown between them I'd say the Nazgûl could (my opinion only of course) defeat the Dead Men of Dunharrow since they are individually far more stronger than any Dead Men and would have used some sort of sorcery instead of just engaging directly.
