"Window", a short story by Bob Leman, first published in The Magazine of Fantasy & Science Fiction, May 1980, available at the Internet Archive. This famous story has been the subject of several questions on this site.
Scientists or mathematicians find a way to peer into an alternate dimension.
Sort of. The interdimensional portal was the accidental result of a mad parapsychologist's experiments on telekinesis:
Krantz was occupied with the lighting of a cigar. He blew a cloud of foul smoke, and through it he said, "We're missing one prefab building, one POBEC computer, some medical machinery, and one, uh, researcher named Culvergast."
"Explain missing," Gibson said.
"Gone. Disappeared. A building and everything in it. Just not there any more. But we do have something in exchange."
[. . . .]
Krantz resumed his lecturing voice again. "As to the nature of it, nothing. We have a window, which we believe to open into the past. We can see into it, so we know that light passes through; but it passes in only one direction, as evidenced by the fact that the people over there are wholly unaware of us. Nothing else goes through. You saw what happened to the rocks. We've shoved poles through the interface there—there's no resistance at all—but anything that goes through is gone. God knows where. Whatever you put through stays there. Your pole is cut off clean. Fascinating.
In the alternate dimension the inhabitants appear to be living a peaceful bucolic lifestyle.
A fine little family, as Reeves had said. After watching them for half an hour, Gibson was ready to concede that they were indeed most engaging, as perfect in their way as their house. They were just what it took to complete the picture, to make an authentic Victorian genre painting. Mama and Papa were good-looking and still in love, the children were healthy and merry and content with their world. Or so it seemed to him as he watched them in the darkening evening, imagining the comfortable, affectionate conversation of the parents as they sat on the porch swing, almost hearing the squeals of the children and the barking of the dog as they raced about the lawn.
Finally one of the scientist/mathematicians finds a way to cross over into the alternate dimension only to be ripped to shreds and by the seemingly peaceful inhabitants. His science peers watch in horror from the other side of the portal/window.
Gibson did not hear him. He was staring with shock and disbelief at the child in the window, trying to comprehend what he saw and did not believe he was seeing. Her behavior was wrong, it was very, very wrong. A man had materialized on her lawn, suddenly, out of thin air, on a sunny morning, and she had evinced no surprise or amazement or fear. Instead she had smiled—instantly, spontaneously, a smile that broadened and broadened until it seemed to split the lower half of her face, a smile that showed too many teeth, a smile fixed and incongruous and terrible below her bright blue eyes. Gibson felt his stomach knot; he realized that he was dreadfully afraid.
[. . . .]
In a moment they all came running out, mother, father, little boy, and granny, all naked, all undergoing that hideous transformation of the mouth. Without pause or diminution of speed they scuttled to the body, crouched around it, and frenziedly tore off its clothes. Then, squatting on the lawn in the morning sunshine, the fine little family began horribly to feed.
