I've seen methane described as the best of both worlds---the worlds being hydrogen's energy density + pain-in-the-ass factor and RP-1's lower energy density + ... ease of use?
And this has me wondering: if methane is so great, what kept SpaceX from using it in Falcon 9's Merlin engines? And likewise, what kept Blue Origin from using it in New Shepard's BE-3 engines (which sip liquid hydrogen)?
SpaceX chose to use RP-1 because the development path for a good RP1 engine was a full decade shorter! Plus many other factors favoring KeroLox for launch from Earth surface, for example the very high energy density of RP-1 allows smaller tankage, which is a great advantage when flying through the atmosphere. The Excellent thrust density of RP-1 also allows simpler engines to deliver good thrust to mass ratio, which is the most important criteria for launch engines.
What is this "RP-1's lower energy density" you mention? It beats the pants off both Methane and Hydrogen for energy density. It actually beats any other fuel, short of a few really strange exotics like Fluorine-based rockets.
SpaceX chose Methane for the Starship's Raptor engines because it is a lot easier to work with than Hydrogen, provides easier tankage, and makes for a much cheaper engine and plumbing. It is not as good in performance for vacuum operations, but is a lot better than RP-1. Crucial for SpaceX, there is a relatively easy pathway to making both Methane and Oxygen using materials on Mars, which is their ultimate goal. The same is (enormously) not true for Hydrogen. Longterm storage of Hydrogen requires much fancier, larger and extensive infrastructure than for Methane.
Methane is not the ideal fuel for operation from earth surface, but it is ... ok. More efficient than RP-1, reasonably good thrust density, and much superior performance once out of the atmosphere. It is, as you noted, a good compromise, and "the best of both worlds"
Hydrogen provides one thing, and only that one thing, better than any other fuel. It provides more thrust per mass, thus more efficiency, than anything else. This is a real advantage out in the vacuum of space, where you need to get the most speed out of a given mass of ship possible.
It require ludicrously large fuel tanks, because the fuel's natural density is so low.
It requires somewhat fancy tankage, because the storage temperature is so incredibly cold.
It also requires pretty fancy tankage, plumbing and handling, because it leaks through cracks that other gases would never notice. It reacts to contaminants in quite amazing ways. And most metals, when cooled to liquid hydrogen's temperatures, become brittle due to both temperature and because the hydrogen seeps into most metals.
And forget about using any organic/plastic plumbing. It just falls apart at those temperatures. This makes hydrogen engines expensive and difficult to design, as one is more severely limited in materials, lubricants, etc than with other fuels.
BO chose Hydrogen because it is fashionable and "green". It is, frankly, a ludicrous fuel choice for their New Shepard suborbital hopper. That hopper flies in exactly the regime where KeroLox > MethaLox > Hydrolox.
The first methalox engine I've been able to find, the RS-18, was developed two years after the first flight of Falcon 1 and the Merlin engine. In contrast, RP-1 had a half-century of development history behind it, and hundreds of engine designs that used it.
SpaceX is, first and foremost, a business. Going with a proven technology for your first product is a far more sensible decision than going with a cutting-edge one, particularly if it lets you get to market a decade earlier.
Because in 2002 there were no existing methane engines they could use or copy (at least none suitable for a booster).
SpaceX didn't have the time or money to develop a whole new engine using a whole new fuel from the ground up. Instead, they based the Merlin 1-A off of NASA's existing Fastrac design, which was a kerolox gas generator.
Initial raptor development started in 2012 - it took seven years and a lot of money to get from design to the first flight test on the Hopper, and they're still refining and tweaking the design.
In addition to the factors already mentioned: The Falcon operates in a very different economic model than anything that has come before.
Look at the electronics industry for a comparison, specifically the term "bleeding edge". You pay through the nose for the best performance. SpaceX took this lesson to heart in the design of the Falcon--they deliberately chose to stay far away from the limits of rocketry. It's not about getting a pound to orbit with the least fuel, it's about getting a pound to orbit for the least cost. Fuel is a tiny percentage of the cost of an orbital rocket, they don't care about it other than the complexities of hauling it.
A simple illustration of this is the Falcon Heavy vs the Falcon 9 expending the core. Nearly 3x the fuel, nearly 3x the hardware thundering into space, yet it's cheaper.
