This would be Piers Anthony's Apprentice Adept series. The first three books are Split Infinity (1980), Blue Adept (1981), and Juxtaposition (1982). I've read those as a trilogy in one go, so I'm a bit hazy on what happens in which book, but I think that while most of the plot points you remember are present in the first book, they feature more prominently in the second.
I remember reading it in the late 80s or early 90s.
The first book, Split Inifinity, is from 1980, so that's entirely possible.
It was about a man who was a slave on a planet where the absurdly wealthy slave owners would have a planet-wide series of games; everything from chess to football.
The planet is called Proton. The protagonist is the serf Stile, competing in these games. Games are chosen from a grid, with one contestant naming one aspect and their opponent the other.
Their turn came, and they went to the grid. Stile got the letters, and was afraid the boy would go for CHANCE—and was correct. It came up 3C, Machine-Assisted CHANCE. Any CHANCE was bad; Stile had tried to mitigate it, but ultimately it remained potential disaster.
If he could steer it into one of the more complex mechanical variants, a pinball machine—for a person like him, with experience and a fine touch, one of those became a game of skill.
—Blue Adept
A slave would compete in a random event then move forward in the bracket. The citizens would bet on the slaves. The slave who won the last bracket would earn their freedom and enough money to fit in with the masters on the planet.
Not just serfs compete. Citizens and aliens can compete too. A winning serf is indeed granted citizenship.
Stile was spared the awkwardness of answering by the Game Computer's introductory announcement. "Attention all entrants. The Tourney roster is now complete: four hundred Citizens, six hundred serfs, and twenty-four aliens.
Pairing for individual matches is random each Round.
The Tourney is double-elimination; only entrants with two losses are barred from further competition. Serfs among the final sixty-four survivors will receive one year extension of tenure. Those proceeding beyond that level will receive commensurately greater rewards. The Tourney winner will be granted Proton Citizenship."
—Blue Adept
Every citizen has a share of the mining of protonite, a valuable mineral which is used as rocket fuel.
"Your minimum share of the Protonite harvest can not be impinged upon—but only your luck and competence and determination can establish your place in the Citizen hierarchy. This is a new game."
—Juxtaposition
Only the masters were allowed to wear clothing so that slaves could be easily identified.
Serfs go naked while citizens usually do not, although they're allowed to.
Stile knew they were Citizens, though they were naked, because of their demeanor and the deference the clustered serfs were paying. Clothing distinguished the Citizen, but was not the basis of Citizenship; a Citizen could go naked [...]
—Split Inifinity
A large part of the plot was a parallel world called Phaze, where magic worked by singing. One could only pass from one world to the other if their counterpart in the other world was dead. There was some unbalance between the worlds, which was resolved in the end. Stile opted to remain in the magic world, where magic only was half as powerful now, which he noticed when he sang a lamp on and it only shone half as bright as he had expected.