Yes, you do have something to worry about. Even with a lot of weight on a jack, the jack could move, either back and forth, or side to side, and the whole shebang could pivot down until the car rests on the ground.
Hydraulic jacks are indeed easier to use, but even they can fail. Would you like to be lying underneath when one of the hydraulic seals lets go and the jack collapses?
It's a poor idea to put any part of your body under a car resting on any kind of jack. If you're going under a car, the car should be held by jackstands or sturdy supports that can't collapse, can't pivot, and can't move.