Lenovo Thinkpad T450s: "Error: The non-volatile system UEFI variable storage is nearly full."

Question

A customer of mine is using a Lenovo Thinkpad T450s since something like a year. The machine is running Debian jessie, with a kernel out of jessie-backports, right now 4.5+73~bpo8+1. The latest UEFI version is flashed / installed. The operating system is installed in "UEFI mode", with an extra EFI partition etc. Up until ~ four weeks ago, this setup was rock solid and stable: Lenovo Thinkpad and Debian, what could possibly go wrong?

Since four weeks, at each boot, the machine shows the error I've put into the subject. Here is an image of this:

Pressing Esc will continue the boot process, which worked "well"...for two more weeks. The message then changed from

Press Esc to continue or F1 to enter setup.

to

Clean up YES or NO

(sadly, I've got no image of this).

My client hit "YES", which erased the storage, as far as I see all of this currently, which rendered the machine unbootable. Afterwards, I've restored the "debian" boot entry, and the machine was happy again, booting up fine, etc. This lasted for some more days; since ~ one week, the message is popping up again at each boot.

I've tried to contact the Lenovo support four times on four different days, and gave up after spending ~ 30 minutes in the phone queue each time.

I've used all my $(your-favorite-search-engine-here)-skills these last days, and found next to nothing: Where this comes from, how to debug and, most importantly, how to fix this. As things stand currently, I guess, the machine will be unbootable again soon.

Any pointers highly appreciated!

Answer 1

The Linux kernel since version 3.8 abstracts UEFI variable storage as efivarfs.

Mounting efivarfs

If mount | grep '^efivarfs' doesn't return anything, you can mount efivarfs using this command:

mount -t efivarfs efivarfs /sys/firmware/efi/efivars

Now, you can browse /sys/firmware/efi/efivars to see if any variables stand out.

Sorting UEFI variables by size

efivarfs doesn't have a concept of disk usage, but it does report each variable by size. This command sorts the variables by size, ascending:

ls -lh /sys/firmware/efi/efivars | sort -k5 -h

Next steps

This is as far as I can take you using the information that you've provided. Next up, you need to figure out what is taking up so much space in the UEFI variable NVRAM.

The Arch Linux wiki suggests deleting /sys/firmware/efi/efivars/dump-* files/variables if they exist, though it doesn't mention what creates those variables.

As discussed in chat, one approach would be to take a snapshot of the UEFI variables, flush them as proposed by the Lenovo firmware, reinstall Debian's EFI boot, take a snapshot again, wait for the UEFI variables to fill up again, and take one more snapshot. Then, you'll be able to compare the snapshots to see what changed and hopefully identify what is causing the problematic variable or variables to take so much space.

If all else fails, you could go back to legacy booting.

Answer 2

I've hit the same error on a Thinkpad T430 with kernel 5.13. In my case the reason was an outdated, incompatible version of acpi_call module (which is called by TLP through tpacpi-bat script), causing kernel oops, as explained here. Although the bug in acpi_call has already been fixed, most distributions still have an outdated version.

Here is a proposed fix, in case someone stumbles upon this thread while googling for a solution.