In the novel the rope is noted to have been secured tightly
He took up the rope and made it fast over the stump nearest to the brink.
LOTR: The Two Towers
and it does indeed fall down when tugged, just as occurs in the film.
"Noodles! My beautiful rope! There it is tied to a stump, and we're at the bottom. Just as nice a little stair for that slinking Gollum as we could leave. Better put up a signpost to say which way we've gone! I thought it seemed a bit too easy." "If you can think of any way we could have both used the rope and yet brought it down with us, then you can pass on to me ninnyhammer, or any other name your Gaffer gave you," said Frodo. "Climb up and untie it and let yourself down, if you want to!" Sam scratched his head. "No, I can't think how, begging your pardon," he said. "But I don't like leaving it, and that's a fact." He stroked the rope's end and shook it gently. "It goes hard parting with anything I brought out of the Elf-country. Made by Galadriel herself, too, maybe. Galadriel," he murmured nodding his head mournfully. He looked up and gave one last pull to the rope as if in farewell. To the complete surprise of both the hobbits it came loose. Sam fell over, and the long grey coils slithered silently down on top of him. Frodo laughed. "Who tied the rope? " he said. "A good thing it held as long as it did! To think that I trusted all my weight to your knot!" Sam did not laugh. "I may not be much good at climbing, Mr. Frodo," he said in injured tones, "but I do know something about rope and about knots. It's in the family, as you might say. Why, my grand-dad, and my uncle Andy after him, him that was the Gaffer's eldest brother he had a rope-walk over by Tighfield many a year. And I put as fast a hitch over the stump as any one could have done, in the Shire or out of it." "Then the rope must have broken - frayed on the rock-edge, I expect," said Frodo. "I bet it didn't! " said Sam in an even more injured voice. He stooped and examined the ends. "Nor it hasn't neither. Not a strand!" "Then I'm afraid it must have been the knot," said Frodo. Sam shook his head and did not answer. He was passing the rope through his fingers thoughtfully. "Have it your own way, Mr. Frodo," he said at last, "but I think the rope came off itself - when I called." He coiled it up and stowed it lovingly in his pack. "It certainly came," said Frodo, "and that's the chief thing".
Noting that the rope burns Gollum's skin when he touches it and Sam's proclivity with knots, it seems most likely that the rope is imbued with Elven magic (it glows, for example) rather than that Gollum untied it or that Sam tied a duff knot.
In the film script, the order of events is even clearer. The rope "unties itself".
SAM tugs on the rope - hard.
ANGLE ON: The ELVISH ROPE unties itself and plummets.
ANGLE ON: The ROPE lands at SAM’S FEET...SAM stands speechless for a
moment, and then he looks at FRODO who shrugs.
FRODO: Real Elvish rope.