Building off of @Mick O'Hea's answer, I think it's probably:
A game of Euchre
We are a family of members twenty-four. / There used to be more, / But we threw them out the door.
Standard Euchre is played with 24 cards: the Ace, King, Queen, Jack, ten, and nine of each of the four suits of a deck of playing cards. Any other cards are left in the box or, in the case of Euchre-specific decks, aren't even printed/sold at all.
Except four of them would rather stay / And I think that's okay / As long as they don't play.
A Euchre game goes to ten points, and it is tradition to use the unplayed fives of the deck to keep score. One team gets the red fives, and one the black; starting face down, points are displayed by flipping and covering one card with the other so that the number of visible shapes matches the number of points.
Every day into five tables we split. / In the kiddie table, some sit. / But sometimes a kid doesn't quite fit.
As @Mick O'Hea guessed, the five tables are when the cards are dealt into four hands of five cards each, and a central pile of four cards. The central pile is smaller than the rest, hence it is the kiddie table. [EDIT: @Chowzen points out that the central pile is called the 'kitty', which is a way better connection to have drawn, both as a sound-alike and in reference to a young one.] The kid that doesn't quite fit is the top card, which is flipped face-up at the start for the beginning of the bidding process.
Then with an adult, we make a swap. / And then a noise: pop! / On the floor someone drops.
If a bid is made in the first go-around, then there is one last process before the trick-taking starts: the dealer of that round swaps one card from their (adult) hand with that face-up central card. Now the trick-taking portion of the game begins, with each player laying down cards one at a time.
Then one by one, everyone proceeds to fall. / My right-hand man can crush them all / Even though he's only 3.5 inches tall.
@Mick O'Hea has noted the trick-taking and the 3.5 inch reference to playing cards. The "right-hand man" is a specific reference to the right Bauer/Bower trump card, which is the Jack of the trump suit and is the most powerful trump card.
After all the commotion, we regroup. / Alas, again to the same level we stoop. / The events keep repeating in a loop.
When all five tricks have been played, count them up, score the points, and then the next dealer collects the cards to be shuffled and redealt.
Remember the guys who can't participate? / They determine the night's fate / When they say "that's enough, mate."
Back to the fives that are used to score the game, which ends when a team has ten points, with both their fives face-up and uncovered.
If you like this answer, please like @Mick O'Hea's as well, since they made the major identification. This answer is just an attempt to get the finishing touch and collect it all.
