Long-term supported distros' kernel policies are all wrong

Comment A new hire at Rocky Linux creator CIQ is rocking the LTS-Linux-distro boat – by shining a spotlight on the elephant in the room (or one of the herd).

A recent blog post from Rocky Linux developer CIQ, subtitled Cracks in the Ice, examines "Why a 'frozen' distribution Linux kernel isn't the safest choice for security." The post itself is an executive summary of a study the company conducted comparing the numbers of bugs and bug fixes in RHEL 8's kernel, which it has published in a white paper titled Vendor Kernels, Bugs and Stability.

FOSS software, which provides the building blocks of projects like Linux distributions, is built around community development. The problem is that this is hard to monetize. Some companies have found ways to do this, but they aren't sharing enough of their work, leaving vital parts of the ecosystem starved. There are clearly visible and perfectly feasible ways around this. The snag is that implementing them would mean persuading billion-dollar companies to play nicely together.

The troublemaker behind the blog post is SAMBA co-founder (and regular Register commenter) Jeremy Allison, who recently landed a new job at Rocky Linux developer CIQ, after being laid off from Google last year, in a decision The Reg questioned at the time. (It's not Allison's first such rodeo, as we reported way back in 2001 – when times are hard, even FOSS rock-star developers aren't safe.)

This is not a shocking revelation in and of itself. There is a core problem here, and as is often the case, it has at least two sides to it. The Reg FOSS desk has previously reported on both of them.

One side is that Red Hat has made FOSS software development pay, and pay very well indeed, by producing extremely slowly-changing distros, and then supporting them for a decade or more, which enables large, slow-moving organizations to run the same versions of a distro for years on end. Red Hat is the most visible player, but it isn't the only company doing this: SUSE does it too, and the Debian project does something similar.

The specific aspect of this that CIQ analyzes is that these very long-term enterprise distros pick one specific kernel version, and then keep it on life support for a decade by backporting bug-fixes – and entire new subsystems – from later kernels into it. We reported from last year's DevConf event on a Red Hat kernel developer's account of What it takes to keep an enterprise 'Frankenkernel' alive.

That alone isn't the problem. The problem is that each vendor picks its own kernel version to do this with, and they don't coordinate with the team which develops the kernel. The kernel team has its own set of long-term support releases, which the enterprise vendors essentially ignore. As a result, last year the kernel team cut back its LTS kernel versions harshly, as we reported from the Open Source Summit in September. The overworked and understaffed kernel team is cutting the support lifetime for its LTS kernels from six years to just two. At present, there are six long term releases, one of which goes end-of-life at the end of this year, and another next year.

We feel that we must point out that CIQ does have a reason for highlighting this issue. Its Rocky Linux distro is a third-party rebuild of RHEL, and since Red Hat stopped sharing all its source code with the world in June last year, that means that CIQ can't readily obtain RHEL's kernel source code any more. Although later in June CIQ announced it had a workaround for this, which in early July it described in more detail, questions remain over how long this can work.

So it is perhaps not a big shock that CIQ is now endorsing LTS kernel supremo Greg Kroah-Hartman's advice, which we quoted in our kernel-team story:

In other words, the kernel team feels that the only really secure option is its LTS options – whereas, as we quoted in the Frankenkernel story, Red Hat's kernel maintainers feel that its kernels are better-tested and safer. When we read this from CIQ, our first thought was Mandy Rice-Davies applies.

If the enterprise Linux industry can somehow be forced to grasp the nettle, there is an obvious answer to this. We pointed it out when we looked at the OpenELA extension of support for 4.14 announced in March.

All that is needed is for the enterprise distro vendors to choose the upstream long term support kernel releases for their long term support distro releases. Then, this problem would effectively go away, replaced by the smaller issue of sharing their bug-fixes upstream.

If Red Hat, SUSE, Debian and the other big distros confined themselves to LTS kernels and shared the burden of maintaining them, it would lower their costs. It wouldn't hurt the bottom line. The customers paying for double-digit years of updates don't care what specific version is supported, just so long as there is a version that keeps getting updates.

• Red Hat middleware takes a back seat in strategic shuffle
• Linux kernel 4.14 gets a life extension, thanks to OpenELA
• Rocky Linux details the loopholes that will help its RHEL rebuild live on
• Red Hat strikes a crushing blow against RHEL downstreams

There's even an obvious exemption here to aid monetization: exclude subsystem and driver backports from the patches that are shared upstream. If you want to run an old distro but keep getting new functionality and new device support, then you pay for an enterprise distro. If you don't want to pay, then run a newer kernel. It's as easy as that.

We're sure that some commentators will be able to come up with elaborate justifications why this is completely impossible, to which we pre-emptively respond: cui bono? Keeping their kernels in-house is part of how these companies make big money from a free OS. If you want to know why things can't change even if it would help everyone, the answer is, as William Goldman put it, "Follow the money." ®