Is the Linux / Debian software and package-management ecosystem secure?

Question

My basic question is this:

When we are using the Linux / Debian apt-get or yum package system, we're placing a lot of faith in that the packages we're downloading and installing are secure. For that matter, even downloading Linux images from Ubuntu or Debian or Red Hat or whatever, we are assuming these servers are secure and the sources are trustworthy.

Given that there's a broad variety of distros, package managers, mirrors, and sites serving us this content, it seems worth questioning the security of this whole ecosystem.

How secure is this open-source software ecosystem from nation-state security department hackers?

As we know, our own "guv'mint" has already been responsible for compromising the servers and security standards of other open-source projects such as the SSL encryption standard, and implicated in hacking private corporate systems which we rely on (Google/Gmail), which gives me reason to consider how many possible attack vectors there could be in an ecosystem as complex as the open-source / unix / linux community is.

To give a concrete example:

If I use Ubuntu's apt-get package manager in the US (or another country), I am blindly downloading packages from servers hosted in that nation and assuming that the software I'm installing is safe. What is there to secure that the servers haven't been compromised, or had malicious code embedded in trusted software packages? It seems like a likely target for nations with Information security departments which have strong ambitions towards controlling cyberspace.

Answer 1

It isn't. The threat model attempts to be resistant to external attack, but if all it takes is a malicious line in a build script on a package used on most systems (e.g. libc, x11, etc.) then all they need to do is compromise one build machine to gain near-universal control.

Attempting to protect against this is hard, and the only way to do it is to build on lots of machines and diff the outputs (which isn't always feasible due to the entropy involved in minute package version changes across build systems). This isn't cost or time effective for open source, so it isn't done.

Realistically, if your adversary is a nation state and they actually give a crap about getting at your stuff, you're likely not going to win that fight unless you're also a nation state with a capable security budget and a big helping of luck.

As James Mickens said in his brilliant paper "This World of Ours":

If your adversary is not-Mossad, then you’ll probably be fine if you pick a good password and don’t respond to emails from [email protected]. If your adversary is the Mossad, YOU’RE GONNA DIE AND THERE’S NOTHING THAT YOU CAN DO ABOUT IT.

The Mossad is not intimidated by the fact that you employ https://. If the Mossad wants your data, they’re going to use a drone to replace your cellphone with a piece of uranium that’s shaped like a cellphone, and when you die of tumors filled with tumors, they’re going to hold a press conference and say “It wasn't us” as they wear t-shirts that say “IT WAS DEFINITELY US,” and then they’re going to buy all of your stuff at your estate sale so that they can directly look at the photos of your vacation instead of reading your insipid emails about them.

TL;DR - Including nation states in your threat model is essentially equivalent to including wizards in your threat model: nobody really understands what they're capable of, and nobody really has any reasonable capability of defending against them.

Answer 2

Debian and Ubuntu packages are signed with GPG keys, making it pretty hard to replace a software package in the repository with another one.

For the installation medium itself (the ISO) there are some hashes and GPG signatures available, but these are not automatically checked. This makes it possible to swap the installation ISO with another one, as happened with Linux Mint.

Answer 3

If you haven't already, please read the fantastic answer by Polynomial before reading this. IMO he's totally on the money!

I particularly couldn't help but laugh (because I'd rather not cry ATM) at this:

Including nation states in your threat model is essentially equivalent to including wizards in your threat model[!]

Having said that, Debian are making significant efforts to reduce the attack vectors.

Arguably the most important of these is the Reproducible Builds project. It's still doesn't have full coverage, but significant progress is being made. Whilst still not infallible, it certainly raises the bar.

There are also other efforts, e.g. Declarative Packaging, which should also improve the security of package installs (especially packages installed from third party repos).

Answer 4

i have written a similar idea in a question yesterday. and from the "related" block/widget of my question i have found this question.

i answer only about apt (debian) system. text of my question answers this question, about apt system. i cite it and copy parts of it with modification.

apt InRelease files update like twice every day, in ubuntu repositories and their mirrors. similarly in ubuntu and debian based distros, and other apt repositories. these files are small text files that contain hashes of other [fresh] text files, which contain hashes of [fresh] package files. and these files are signed. so, these files can prove that package has really come from the repository owner unmodified. this system is used automatically by the apt scripts on updating in order to prevent malware being installed in packages by third parties like network providers or mirrors ( they are "man in the middle" ) (and i think also can prevent from simple bit errors appearing).

in case some user has found malware in package, they may need to check where from does it come and proof that in court.

ie this, the system described in that my question, can be named "delayed punishment" security system.

a malicious file may be sent only to a several/single user, in the process of update, (if repository owners think that the user is not going to catch them). third party archives are required also to prevent this. to prevent this, users should check, after apt update, and before apt upgrade, whether they has got the proper inrelease files, they are saved in /var/lib/apt/lists/, same as saved in one or several third party/independent/witness sites, (which must have [recently] downloaded the same file from other ips and with other cookies etc).

default ubuntu and debian installations are not secure:

1. they compel user to use their own (debian's and ubuntu's) servers for security updates. they are possible to change only via manual editing, but not via their software sources window. for me, by this, they look quite like the wolf in "Little Red Riding Hood". when 3rd party mirror is used, they cannot give malicious file to them, because mirrors are organisations and they may archive all that files for investigation. also, if mirror is used, repository owner does not know when user has connected to it. while user connects to distro's own server, they can identify user by his ip, in case they somehow know his ip, for example, they may know it when he visits their sites.

2. as i know, nowadays, all downloaded packages are automatically deleted by default. if saved, they can be used to investigate, what is in them. if there is malware in them, it cannot modify them, because that would require to keep their hashes not modified, but package installer runs as root and, i think, it can just delete this package... or all packages (.apt files) after installation. it is going to delete all packages, because if it deletes only one, it is indicating to the malicious package. but even if all packages are deleted by malware, if admin notices this, he can conclude that malware has come. when they are automatically deleted by itself, admin may not even know that his system is hacked and continue to use it.

3. the old InRelease files are rewritten automatically after every apt update. seems, this is not configurable. this files could be saved for investigation piurposes. if they are saved, they still can be deleted by malware similar to above item #2.

but more advanced user can easily secure his ubuntu/debian system: there are 2 ways:

1. he can use a mirror for all repositories including security updates. only disadvantage is that mirror can hold updates not updated for some limited time, maybe that time can be configured in apt config. as i know, if the time is passed, apt update reports a error message. update time can be checked manually with commands like less /var/lib/apt/lists/mirror.yandex.ru_ubuntu_dists_focal-security_InRelease.

2. he can continue to use distro's own server for security updates, as configured by default, but after apt update he should configure his browser to use a proxy server and download InRelease file via it and compare it with diff with apropriate file in /var/lib/apt/lists/.

these 2 ways are against when repo owners send a malware only to a single user.

what if they send malware not only single users, but publicly distribute them, to all mirrors? they may do that, if they hope that old package files and inrelease files and other files are not saved by mirrors and users. to fight that, user need to save all that files to separate computer, so that malware cannot manage it, before installing them, or have a trusted third party archive. saving all old packages requires many space.

debian already saves them on their own server, https://snapshot.debian.org/ , and this can be used, together with saving the small InRelease files in independent server, to check something... (i need to think about it before i can write). so... if you have signed inrelease files in simple server, if they are not too old, you can prove that they are indeed was signed by debian, even if same inrelease files are deleted from snapshot.debian.org. if you have signed inrelease file in blockchain, saved when it was fresh, you can prove it even after long time. but since you do not have package files saved, assuming they also are deleted on snapshot.debian.org, you cannot investigate them and check how exactly they worked. you can try to search for that files from users of debian.

---

about downloading isos, similarly, fighting 2 problems:

1. user should check hash of iso directly via https and via proxy server via https, or check that hash from a mirror via https. this is to fight that he gets specially crafted malicious iso only himself, while most users get normal iso.

2. and he should save signature of that iso in simple server, but better to a blockchain, or use what is saved by other users. this is to prove that iso was made by the distro, in case all users and mirrors (and distro itself) lost not only iso but even the hash of it, so they cannot witness in future. isos are typically saved many years in mirrors, so, blockchain is not needed. blockchain can be useful in case of using a very infamous distro. i think, probably, unlike inrelease file, iso signature is useless without iso file itself. inrelease file contains both signature and signed content within itself. so, you need to keep iso file, if you are afraid you cannot find it later somewhere in internet. if you lose iso file and only have hashes and signatures, and only you have these hashes and signatures, probably you cannot even prove, that this signature belongs to the distro without the iso file. if you have lost iso file, but other users have same hashes as you, they can witness that iso with this hash existed, but you cannot investigate, how did it work.