The simple response is to show that the question does not apply to people who believe in the righteousness of God's judgments against unrepentant sinners.
God speaks of sin against himself. Humans speak of crimes against humanity.
This leads into exposing the notion that human ideas of 'crime' equate with sin in God's view. The need is for sinners to understand - from God's point of view - why humans are guilty of infinite sin. Until that is faced up to, the question itself prevents understanding of answers given.
For example, note how the Bible speaks of two criminals being crucified alongside Jesus (Luke 23:32). They had committed crimes against the Roman Empire which issued the tortuous crucifixion death-penalty. That judgment was based on the Empire knowing that if such criminals were not put to death, they would continue to commit more crimes against the Empire, with a view to overturning it. Compare Jesus. Pilate publicly declared that he found nothing in Jesus deserving of death (Luke 23:14-15). None of Jesus' detractors could prove he had sinned (John 8:46) so they had to come up with false charges to get him crucified, claiming he had committed a crime against the Roman Empire (which was only interested in crimes, not sin - the human perspective.)
But given who they had put to death (the Son of God), they had all sinned against God, and so does everyone who rejects Christ as the crucified and resurrected Son of God who died to save sinners from the wrath of God. This is on a par with sinning against the Holy Spirit, for which there is no forgiveness, Jesus warned (Mark 3:39). All three are sinned against, Father, Son, and Holy Spirit, when their three-fold, divine, witness is denied by sinful humans. Yet they would try to dismiss that as mere 'finite crimes'?
Now, the "infinite punishment" bit of the sentence. Today, people think criminals should not be punished, but 'rehabilitated'. This lies at back of what is wrong with the question posed. They could go along with a temporary punishment that led to the rehabilitation of the person, but people who believe in today's "Social Justice" agenda are utterly offended at the way God will deal with those who could never be "rehabilitated" in a trillion years. And he knows who those ones are, yet still holds out forgiveness and restoration based on believing what his Son did to bear the punishment they deserve in their stead. God does not tinker around with "rehabilitation". He restores utterly. He makes new creatures in Christ; they are given the mind of Christ; they are given a new heart that beats for the righteousness of God and the establishment of his Kingdom, with Christ as their King.
Ah, but the cry back then was, "We will not have this man rule over us!" and it remains the same today, effectively coming from all who will not call 'sin' sin. Jesus came to take away the sin of the world (John 1:29) but those who love sin prefer to walk in their darkness. They do not want the light of Christ to show up their deeds of darkness. Therefore their portion will be "the blackness of darkness for ever" (Jude vs.12). Note that this speaks of men.
A biblical principle is that we reap what we sow (Galatians 6:7). Never will that truth be more keenly realized than after the final judgment has been declared against those who refused to believe the testimony (witness) of Father, Son and Holy Spirit, in effect, calling God an unjust liar - as did the serpent in the garden of Eden. Note where the posted question originates from.