There is a downside to increasingly time-consuming and intrusive checks. You need to pay the border guards who perform them and/or contend with time loss at the border waiting. If you follow public discourse, you might sometimes get the feeling that the time and comfort of non-citizens is a negligible quantity but that's very short-sighted. It's not good for tourism, not good for business, annoying airport operators and airlines, and apart from the receiving country's own benefit calculation it's simply wasteful, humanely and economically. All this for what? Catching a few thousand future overstayers a year?
Another point to consider is how effective these checks really are. They certainly do create a lot of misery, from the simple discomfort of long lines and hostile questioning to the distress of detention and “removal at port” (i.e. the police forcing someone onto a plane, which is technically different from a “deportation”, decided by a judge, and other types of forced removals). When you hear about specific cases, it's easy to explain them away with admonitions like “you should have done this” or “you should have done that” but how many of these people would have posed a genuine problem if they had been let through? How many people do get through who ideally shouldn't have? And how much discomfort can you inflict on people who have no intention of breaking the law just because they are non-citizens?
An ID and database check, together with a simple filter like providing a straight answer to a basic question should catch most “low-hanging fruit”. I have no way to tell exactly how fast but it stands to reason that the return on additional checks then decreases very fast, as does their accuracy. This added value ought to be measured against the costs I mentioned earlier and there are others, more effective uses of public money. Schengen countries are for example focusing on generalising visa biometrics and database checks, which are still not 100% systematic as far as I know.
In general, the UK does not seem much better than its peers at preventing illegal immigration. Perhaps it would be even worse off without these intrusive checks (which would be an explanation in itself) but that's not obvious. Without evidence of that, the checks are just theatre, inflicting discomfort for the purpose of demonstrating strength, not an effective way to tell “good” visitors from “bad”.
Incidentally, for better or for worse, the Schengen area isn't that open either. Most people in the world still require a visa, a rather intrusive process fraught with difficulties (even if it is also easier and cheaper than the UK in this respect) and your experience at the border depends a lot on your appearance (including race, wealth, and how confident you look). If you do not require a visa and look like a tourist or better yet, a businessman, it might feel very easy but that's because you already went through a number of implicit filters and there is little point in bothering you further.
Anybody else, including women and children travelling alone, backpackers from other developed countries and citizens from developing countries occasionally face additional scrutiny. You can easily find countless stories of people turned away even though they had a visa and even a few scare stories of citizens detained over concerns regarding their passport or some such. And according to Eurostat, in a regular year, countries like France, Poland, and Hungary, register a similar number of entry refusals as the UK, not to mention Spain, which reports many more. From that perspective, the policy in the Schengen area isn't particularly “careless”.
In fact most somewhat open countries are similar to the Schengen area in this respect, with only a handful of exceptions. The question therefore becomes why does the UK in particular go above and beyond? One factor is surely the decades of controversies around immigration and the fact that some politicians basically built their careers on being tough on this topic. It doesn't matter that immigration is often a scapegoat used to hide other policy failures or that the fact earlier restrictions did not make the problem go away should logically give one pause, you always hear calls to be even more restrictive. Another factor, already mentioned, is that the UK is already very attractive for people willing to stay illegally as it is and understandably concerned about the problem being even bigger without aggressive enforcement to deter newcomers.
Finally, one very specific factor is that Britain is an island, with very few entry ports so that focusing enforcement on the border feels like a reasonable proposition, in a way that isn't true in countries with long and complex land borders. This insular mentality is also on display in other subtler ways. Consider for example, the link provided by GayotFow in a comment: A conservative MP basically concedes that the commonly repeated line about people flocking to the UK to abuse purportedly generous benefits is a lie and that EU citizens basically come to work on a par with British citizens, in a way that's broadly beneficial to the economy and the country.
But that, in itself, is unacceptable to this MP, his party, and a large part of the British public. Being allowed to reside under restrictive conditions (namely having work) becomes an “entitlement” and these people must urgently be downgraded to the status of workers from “Bangladesh, Australia, America, Canada or India”, who have to prove they are exceptionally skilled, pay hundreds if not thousands of pounds in visa fees, and generally be made to feel that coming to the UK is a privilege. From this perspective, intrusive checks (and Brexit) are an end in themselves, quite apart from any cost/benefit consideration.
