This wasn't a literal battle, it was a contest, a battle of wills - for Beren and Finrod to keep their orc disguises, and for Sauron to strip them of it and reveal their true selves. The prose text surrounding the poem goes "Thus befell the contest of Sauron and Felagund which is renowned. For Felagund strove with Sauron in songs of power, and the power of the King was very great; but Sauron had the mastery [...] Then Sauron stripped from them their disguise, and they stood before him naked and afraid. But though their kinds were revealed, Sauron could not discover their names or their purposes.
The poem comes from the Lay of Leithian, and epic poem which Tolkien worked on from 1925 but never finished. It offers some more detail about the encounter, such as:
"Then his flaming eyes he on them bent,
and darkness black fell round them all.
Only they saw as through a pall
of eddying smoke those eyes profound
in which their senses choked and drowned."
As this is an epic, romantic poem rather than the usual narrative prose we're accustomed to, I think we can allow for a bit of poetic license here. The tale of Beren and Luthien features other instances of "magical" singing and spellcasting, not to mention the transformations of Beren and Luthien into animal forms.
[Edit] There are some notes about the encounter, from History of Middle-Earth (at this point, Sauron was a wizard called Thu):
There may seem to be a difference between the outline and the Lay, in
that the former says that 'after a contest of riddling questions and
answers they are revealed as spies', whereas in the latter Felagund is
overcome by song of greater power. In fact, the riddling contest is
present, but seems not to have been fully developed. In the original
draft my father scribbled the following note before he wrote the
passage lines 2100 ff.: Riddling questions. Where have you been, who
have you slain? Thirty men. Who reigns in Nargothrond? Who is captain
of Orcs? Who wrought the world? Who is king &c. They show Elfin
[?bias] and too little knowledge of Angband, too much of Elfland. Thu
and Felagund ..... enchantments against one another and Thu's slowly
win, till they stand revealed as Elves.