A big part of modern engineering is making sure that nothing is really a waste product - partly for simple reasons of economics (someone who can make use of more of the products will likely be more profitable) and in a much bigger part, environmental legislation. When crude oil first started being used in the US, it was used to produce kerosene for lighting (seen as a "cleaner" alternative to coal gas); what happened to the waste products (making up the vast majority of the input)? Mostly the highly toxic waste was just dumped into rivers or burnt off. Gasoline was a waste byproduct once. Heck, the very idea of using a chemical resource to produce energy reeks of (irresponsible) "waste disposal" nowadays - we can't use it productively? Burn it to release energy!
Your list of possible uses of all those products is already quite large, and is not exhaustive by any means. Remember, when you can't just dump or burn the waste (as was long possible, and is still being done when you can get away with it), it accumulates - that's expensive. And given the huge volumes of crude oil processed, even a small percentage amount of waste would accumulate very quickly indeed. And we're talking about stuff that's very toxic, flammable and hazardous, and generally quite hard to store safely.
What exactly each refinery does is very specific, depending on the demands they can hope to satisfy. As weird as it sounds, even today, many oil plants and refineries just burn off excess methane, for example - it's a very valuable byproduct... but only if you can reasonably get it to consumers. There's also a lot of variability with regards to the input feed - crude oil is ridiculously far from being a homogenous substance. Every oil field has a very different combination of hydrocarbons (and other stuff, like various sulfuric compounds) that heavily affects the outputs.
Of course, while petroleum distillation is a very visible and known part of the refinery, refineries don't just separate what's already there. There's a lot of processing steps that change the kinds of outputs you get; there's certain flexibility to allow for changing product demands. And while we talk about things like "gasoline", that's not really a homogeneous chemical product either - it's generally a mix of hydrocarbons (and additives) that satisfies conditions that fit within the requirements specified for particular ICEs. "Gasoline" from two different refineries (even from the same company) can be a very different material indeed. This is in contrast to competing products like ethanol, which are actually just that - ethanol, water, some impurities. This of course requires engines that burn gasoline to be a lot more lenient towards fuel used than engines that only need to burn ethanol or methane (and indeed, gasoline engines usually didn't need much beyond tweaking the ignition to use ethanol or methane instead of gasoline).
So, primarily, you want to sell as much product as possible, and if it makes sense, use the less wanted products for running the refinery. Indeed, a lot of the heating and mechanical elements use fuels or electricity generated on-site. It's a lot easier to power your own refinery with excess methane than building and maintaining a gas pipeline to the nearest gas power station.
If you can't sell the products you get from separation and distillation directly, you have two main options - change it to something you can sell, or dump it. Dumping is severely limited by environmental legislation (and to a lesser extent, practicality); for example, you'll be dumping a whole lot of waste water, but you need to do a lot of processing to make it reasonably safe to be released into the environment.
Worldwide, the main use of petroleum products is in chemical engineering - producing things like fertilizers, plastics etc.; in the US, the majority is used for transportation. According to US EIA, staggering 43% of all petroleum consumed is used just for the production of gasoline, some 20% for diesel fuel and 8% for jet fuel - and that's just transportation. So yes, in the US, energy production is a massive part of what petroleum is used for; it's a great waste of a valuable chemical resource if you ask me :D
It also drives the need to use the waste from the main product somehow - hence all the push to use bitumen for asphalt production (and the associated "we need more roads!", both helping sell the waste and further encourage fuel consumption), or petroleum coke as an industrial fuel source (fairly poor coke, mind - you wouldn't use it for steel production for example). The price of such products is generally very low (petroleum coke tends to be about 1/3rd to 1/4th the price of coal coke, just to get rid of it). The waste sulphur has also historically been a big problem in the US due to the massive volume of petroleum used for fuel purposes. Sulphur is the start of a lot of other chemical processes... but if you can't use it, it's yet another bulky, hazardous chemical. Again, it's in the interests of the oil companies to boost demand, for example by encouraging the use of artificial fertilizers (and indeed, the US oil industry managed to get people quite addicted to "cheap" fertilizers that make it even harder to de-oilify the economy).
There's also the very heavy fuel oils. Noone would ever use them as fuel if they had a choice - they're even more toxic, hard to work with and extremely dirty. But the oil companies still found ways to work around the environmental legislation to sell that product, as horrible as it is - used mainly by seafaring ships that burn it where it's not covered by the environmental legislation. Though it seems there's finally some coordinated push to stop that nowadays, with stricter world-wide regulation coming.
So yeah, what with the waste? If you can, sell it. If you can't, use it. If you can't, convince people how useful and important it is. If you can't, turn it into something more useful. If you can't, dump it. If you can't, dump it even further away. The oil industry is very good at pushing tons of horrible consequences on others :D And because the main product is still fuel in the US, the other products can be sold pretty cheaply and still remain profitable (indeed, paying people for taking the "waste" off their hands would often be cheaper than handling the waste "properly" or turning it into other, more in-demand products).
The big challenge is that by far the main reason we use petroleum at all is how cheap it is; it doesn't take long for additional processing to eliminate that advantage. In fact, there was a time when bioethanol was cheaper than gasoline in the US after the government eliminated alcohol taxes - which completely coincidentally resulted in the Prohibition (Rockefeller funding and owning several of the "grass roots" organizations pushing for alcohol prohibition is completely coincidental, of course :P ).
Engineering alone only explains a small part of the mess that is oil production and refinement - but as it advances, it has the potential to make more valuable products, rather than just finding a spin on selling the less valuable products. For example, finding buyers for bitumen/asphalt used to be very important in the 50s-70s in the US - modern refineries tend to crack it instead, to get shorter chains. But even today, burning off the unwanted products is still fairly significant, and the amount of e.g. methane just leaking into the atmosphere is scary. You can change pretty much everything into more useful products, but that makes the processes more expensive. The oil and gas industry is very protective of their massive profit margins.