"No organized force, ably led, is ever outnumbered by an uncoordinated mob."
Simply put, the Emperor wasn't interested in killing them all quickly. He admitted as much to Luke.
He let many of the live (tying his Fleet's hands for political reasons) and kept their side of the fight relatively low-key.
The rebels couldn't leave - if they had fled, the massive morale hit would have destroyed the Alliance. Even if it had survived, the Second Death Star would have been completed, and their already-impossible fight would have become harder than ever. The Rebels were committed to the fight. They had to win, destroying the Second Death Star, or die trying.
That said, they weren't suicidal. Neither side was fighting all-out in the fleet action. The Empire was holding back while their Emperor tried to seduce a young man to the Dark Side, using his friends as hostages. If they'd killed all the Rebels as quickly as they could have, it would have lessened Palpatine's time to subvert the last Jedi. The Rebels must have recognized the fact that the Imperial Fleet wasn't giving it their all, and chosen not to poke the sleeping giant too hard.
The expanded universe describes the way the Emperor used the Force to subtly direct his fleet, making them much more combat effective. When he died, there was immediate and massive confusion in the Imperial Fleet. Couple this with the spectacular destructions of the flagship Executor and the 'Invincible' Death Star, and it's easy to see how their coordination, morale, and willingness to fight would be shattered.
The Rebels, on the other hand, had already done three impossible things that day, saw a chance at complete victory, and took it.
To answer your question succintly: The Imperials started out with their hands tied as their leader played a different game, and when it went badly for him they suffered massive morale hits, leadership losses, confusion, and panic. The Rebels had a simple numerical disadvantage. When the two largest threat sources were removed and their enemy was panicked and routing, they swept the battlefield clean.