Two points:
First, the words of the Elves are to the King of Numenor. They are not addressing all mortals, but carrying a message to Numenor and specifically the king. When they say "and there you would but wither and grow weary the sooner" they may well have been speaking to Ar Pharazon himself and, perhaps, to others who have fallen so far from the grace of their distant ancestors that the light and the very air of Aman would hurt him.
Recall, how Gollum is hurt by light and by Elvish things like Lembas and rope, and how Orcs fear sunlight and trolls are turned to stone by it. The more powerful Witch King, who could endure light, is driven away at Weathertop by the name of Elbereth:
...all blades perish that pierce that dreadful King. More deadly to him was the name of Elbereth.'
Shelob, daughter of Ungoliant, could tolerate even the name of Elbereth:
Aiya Eärendil Elenion Ancalima! he cried, and knew not what he had spoken...
But other potencies there are in Middle-earth, powers of night, and they are old and strong. And She that walked in the darkness had heard the Elves cry that cry far back in the deeps of time, and she had not heeded it, and it did not daunt her now.
But Frodo carried the phial of Galadriel whose light was the light of Earendil's star, itself the light of a Silmaril and, ultimately, the light of the Two Trees. That light could daunt even her:
Then holding the star aloft and the bright sword advanced, Frodo, hobbit of the Shire, walked steadily down to meet the eyes.
They wavered. Doubt came into them as the light approached. One by one they dimmed, and slowly they drew back. No brightness so deadly had ever afflicted them before. From sun and moon and star they had been safe underground, but now a star had descended into the very earth. Still it approached, and the eyes began to quail. One by one they all went dark; they turned away, and a great bulk, beyond the light's reach, heaved its huge shadow in between. They were gone.
The fallen are hurt by the near presence -- even the name! -- of signs of the transcendent. The King of Numenor was not yet, perhaps, completely fallen, but it is easy to see how he might have been too far gone to be able to tolerate the undying lands. It does not follow from this that mortals less lost to evil would be so quickly or so badly affected.
Second, carrying the Ring to great peril to Rivendell changed Frodo and made him something other -- more? -- than a simple Hobbit. After he is cured of the Morgul wound by Elrond,
Gandalf moved his chair to the bedside, and took a good look at Frodo. The colour had come back to his face, and his eyes were clear, and fully awake and aware. He was smiling, and there seemed to be little wrong with him. But to the wizard's eye there was a faint change just a hint as it were of transparency, about him, and especially about the left hand that lay outside upon the coverlet.
'Still that must be expected,' said Gandalf to himself. 'He is not half through yet, and to what he will come in the end not even Elrond can foretell. Not to evil, I think. He may become like a glass filled with a clear light for eyes to see that can.'
This transparency was the result of Frodo's carrying the Ring and of the trials he underwent in the process. (It clearly was not the invisibility of a ringwraith -- Gandalf was not concerned when he saw it.) It is a visible manifestation of a spiritual change in him, and that change seems like it might be exactly what is needed to adapt a mortal to the Undying Lands so that that mortal can, for a time, endure -- and even by healed by -- the "light too strong and steadfast." And, likewise, Sam and Bilbo.
Added: As for Gimli, we don't know that he ever went to the Undying Lands. All we know is that the Red Book of Westmarch included a sceptical late note:
We have heard tell that Legolas took Gimli Glóin's son with him because of their great friendship, greater than any that has been between Elf and Dwarf. If this is true, then it is strange indeed: that a Dwarf should be willing to leave Middle-earth for any love, or that the Eldar should receive him, or that the Lords of the West should permit it. But it is said that Gimli went also out of desire to see again the beauty of Galadriel; and it may be that she, being mighty among the Eldar, obtained this grace for him. More cannot be said of this matter.
We just don't know.