As others have said, once you have been refused entry, the airline has little say in the matter, you will not show them your passport at the check-in desk in the regular way but be escorted directly to the gate by the police/border guards. Airlines have broad legal obligations to carry you back to your country of departure (in national and international law) but the local police usually makes the decisions.
That's not to say airlines always go along, an airline or an individual crew can obviously refuse to carry out a removal. After that, it's a matter of negotiation between the airline and the authorities, they might have to face some penalties, etc. but it's still their airplane. Usually, it's because the person being removed becomes agitated and the police escort reacts with aggressive tactics that passengers and crew find disturbing but it does occasionally happen.
If you have been removed in that way but cannot be admitted in the country you originally left, reasonable countries will not ask the airline to carry you again to your original destination. In some countries, that's explicitly specified in the law, e.g. in France. But it does happen, I have heard of a few cases of people who did two round-trips before one of the countries involved budged (sorry, no reference at hand but I am not thinking about celebrity cases, just some unlucky unknown travellers).
The alternative is to move the person to the regular deportation/removal system, which is a bit messy but deals with many other thorny cases including people who are being expelled after a court order, have been caught staying illegally in the country, won't say where they are from or lie about it, etc. The way this works is that you will be detained for some time while the authorities look for a country that is ready to admit you. That could be your country of citizenship but also some other third country, depending on the specifics of the case.
In any case, they should in principle get clearance from that country's consulate before putting you on a plane, especially if you don't have a valid passport, which is often the case (some people try to ditch their passport after landing in an effort to make removal more difficult).In this scenario (i.e. outside of a straightforward “bounce”), it's not necessarily the airline that brought the person to the country that will take care of the transport. The country deporting you might also pay for your tickets (and those of the escort, as applicable), possibly on another airline.
If there is no solution (e.g. because your country isn't safe or isn't cooperating), you could be detained for a long time, in some countries indefinitely, or possibly simply let go with a temporary visa and an order to leave the country by yourself (this happens regularly in France for example).