A while back a friend of mine (South American) traveled from the USA to the Schengen area without a return ticket. Back then, he was applying for Schengen residence (didn't have one yet and didn't present any evidence of it when boarding).
In an European Union FAQ it says:
"you might be asked to... show... for example tickets for further journeys and return tickets"
And this thread says:
The airlines are responsible for your return transport if you are refused entry into a country, so the airline has interest in avoiding you getting refused entry at the destination.
He was questioned about not having a return ticket when entering the Schengen area and after showing some documents to prove he had a residency application ongoing, he was allowed in. Because he is from South America he was eligible to a 90-day tourist stay.
Usually the airlines are stricter than immigration agents. So I wonder if he had been traveling directly out of his home country to the Schengen area, would he have been denied boarding? The airline is not even flying from Europe to South America, so what would happen in case he was denied boarding upon arrival in Europe? Would he be deported to the USA?