For a large bipropellant rocket engine to fire safely and stably, the fuel and oxidizer have to mix very thoroughly at high flow rate and pressure before they ignite. Otherwise, there will be sputtering and popping, which is very bad at that scale.
Also, in many engines, a "film" of unburned propellant flowing along the walls of the nozzle is critical for cooling. You need that film to be even or you'll get burn-through. When exhausting into atmosphere at low power levels, the exhaust flow tends to separate one side of the nozzle and adhere to the other, causing off-center thrust and undesirable spot heating, instead of moving freely down the middle. To avoid this flow separation you either need a very short nozzle (reducing specific impulse) or an aerospike, expansion-deflection, or other compensating nozzle; those generally increase weight and haven't been thoroughly researched and developed.
Therefore, the design of the fuel injection system is critical, and it's very sensitive to flow rates. The common injector designs for large liquid engines are referred to as "showerhead" injectors and like common showerheads, if the propellant valves are partially shut, you'll get an unsteady dribble out of it instead of a fine, predictable spray.
So early engines in particular were designed for stability in a narrow regime around full throttle.
Pintle injectors like that used in the SpaceX Merlin are apparently less sensitive than showerhead injectors, and the state of the art in fluid dynamic simulation is way better today than it was in the 60s, so the Merlin is more throttlable than some, and can throttle over a wide range. The October 2015 revision of the Falcon 9 user's guide claims 70%-100% throttling for the first-stage Merlins, but 39%-100% for the second stage Merlin Vacuum; this suggests that nozzle flow issues (vastly reduced out of atmosphere) are the limiting factor rather than chamber stability.
Henry Spencer's usenet posts, as always, have some good information:
These indicate that 70%-100% throttling is relatively easy to
accomplish, with the challenge being to get much below that.
However, this NASA study on deep throttling (2010?) is more optimistic, suggesting that 4:1 ranges (i.e. 25%-100%) are straightforward to achieve with small modifications to fixed-thrust engines. It mentions in particular:
- The RL-10-derived CECE with better than 10:1 range (i.e. <10%-100%)
- The SSME, designed to throttle down to 65%, but stable combustion demonstrated as low as 17% of rated power (this is somewhat misleading, because the pumps, and therefore the engine as a whole, have various problems at low power).
A bunch of modern launchers use engines with substantial throttling capability, used to reduce aerodynamic stress around max-Q and/or excessive g-force at the end of the first-stage run when fuel tanks are near-empty: