NVIDIA graphics driver can't be loaded

Question

After upgrading Debian 11 to 12 and kernel version 6.5.0-1, I found my NVIDIA driver somehow broken. Using (sudo) nvidia-settings, I got following output.

caeleste@spectre:~$ nvidia-settings 

ERROR: NVIDIA driver is not loaded


(nvidia-settings:35073): GLib-GObject-CRITICAL **: 10:59:12.413: g_object_unref: assertion 'G_IS_OBJECT (object)' failed

** (nvidia-settings:35073): CRITICAL **: 10:59:12.414: ctk_powermode_new: assertion '(ctrl_target != NULL) && (ctrl_target->h != NULL)' failed

Not understanding exactly where this comes from, I tried a clean reinstallation by purging nvidia* and installing linux-headers-amd64 linux-image-amd64 nvidia-detect nvidia-driver nvidia-cuda-dev. I got exactly the same error.

Some system info:

$> neofetch

OS: Debian GNU/Linux 12 (bookworm) x86_64 
Kernel: 6.5.0-1-amd64 
Packages: 2995 (dpkg)
Shell: bash 5.2.15
Resolution: 3840x2160
DE: Plasma 5.27.5
WM: KWin
Terminal: caelestis-custom
CPU: Intel i7-10750H (12) @ 5.000GHz && Intel UHD Graphics
GPU: NVIDIA GeForce GTX 1650 Ti Mobile
Memory: 3541MiB / 15601MiB

$> nvidia-detect

Detected NVIDIA GPUs:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117M [GeForce GTX 1650 Ti Mobile] [10de:1f95] (rev a1)

#> dkms status

nvidia-current/525.125.06: added

#> apt-cache policy nvidia-driver

nvidia-driver:
  Installed: 525.125.06-1~deb12u1
  Candidate: 525.125.06-1~deb12u1
  Version table:
     525.125.06-2 100
        100 http://deb.debian.org/debian unstable/non-free amd64 Packages
 *** 525.125.06-1~deb12u1 500
        500 http://ftp.uni-stuttgart.de/debian bookworm/non-free amd64 Packages
        100 /var/lib/dpkg/status

#> systemctl status nvidia-persistenced.service

× nvidia-persistenced.service - NVIDIA Persistence Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-persistenced.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-09-18 10:56:26 CEST; 26min ago
    Process: 21064 ExecStart=/usr/bin/nvidia-persistenced --user nvpd (code=exited, status=1/FAILURE)
    Process: 21066 ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced (code=exited, status=0/SUCCESS)
        CPU: 2ms

Sep 18 10:56:26 spectre systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
Sep 18 10:56:26 spectre nvidia-persistenced[21065]: Started (21065)
Sep 18 10:56:26 spectre nvidia-persistenced[21065]: Failed to open libnvidia-cfg.so.1: libnvidia-cfg.so.1: cannot open shared object file: No such file or directory
Sep 18 10:56:26 spectre nvidia-persistenced[21064]: nvidia-persistenced failed to initialize. Check syslog for more details.
Sep 18 10:56:26 spectre nvidia-persistenced[21065]: Shutdown (21065)
Sep 18 10:56:26 spectre systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
Sep 18 10:56:26 spectre systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
Sep 18 10:56:26 spectre systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.

Secure-boot is turned off.

Answer 1

This is always, because the nvidia driver you try to load is not made for the kernel you just updated to.

Check your system-packages source repos, I guess /etc/apt/sources or run a sudo grep -R nvidia /etc/apt/. to investigate. In ARCH-type of systems check or update your pacman caches.

For example, I had cuda installed on my ubuntu system, for which I included an ubuntu2204 source repo from nvidia in my system. This failed exactly as you reported. After removing and re-installing the correct drivers from the standard repos, everything works.

Of course, after updating your system repos you will always have to also update your system, i.e. apt-get update && apt-get upgrade but also in the nvidia case remove the incompatible drivers and re-install the correct ones.