Grub2/Debian10 change root partition on boot

Question

I have a headless NAS system that ran out of disk space on it's system partition. It's running Debian 10, usually managed via OpenMediaVault 5 with a grub2 bootloader on an EFI system. Apart from the HDD data raid it had an SSD with partitions for / and /boot. I've installed an additional SSD with partitions for /, /boot/EFI and /var. I've carefully stopped all service and copied over the data via rsync. I've modified the /etc/fstab of the new partition to reflect the new mounting scheme with the new UUIDs (read from the output of blkid). I did not modify the BIOS or the bootloader. I intended to still use the /boot partition of the original SSD for booting up.

By now, grub has already detected the new system in the /boot/grub/grub.cfg file. So I changed the entry for the default entry in /etc/default/grub, ran grub-mkconfig and rebooted. But it still boots with the first entry into my old system. It's hard to tell on a headless system, but at least according to /proc/cmdline, it looks like the first entry with the old UUID as value for root=. But I know of no way to tell if this is the first attempt or maybe a fallback after a fail.

Also I noticed that /boot/grub/grub.cfg does not change after I run grub-mkconfig. And while I see the UUIDs of the root partitions in the /boot/grub/grub.cfg file, I don't see neither the old nor the new ones anywhere in the /etc/grub.d structure.

I've clearly not fully understood how grub2 works and how it is supposed to be configured. Can somebody give me some guidance on the problem or point me to a documentation that could help me?

Update: Thanks to User @user1686 's answer I re-created my grub.cfg by redirecting the output of grub-mkconfig into a file with the -o option. This took over the GRUB_DEFAULT setting. For some reason, while mentioning the right UUID several times in the entry, in the actual linux = boot command it still showed the UUID of the old system. I manually corrected this and finally booted into the new system. /Update

Some relevant files:

output of blkid (shortened)

/dev/sdb: UUID="c71b542e-8149-827c-e596-32c80ade181d" TYPE="linux_raid_member"
/dev/sda: UUID="c71b542e-8149-827c-e596-32c80ade181d" TYPE="linux_raid_member"
/dev/sdc1: UUID="EAFD-673A" TYPE="vfat" PARTLABEL="boot"
/dev/sdc2: UUID="533dfffd-ebd1-4be6-8e7b-55489f1ffece" TYPE="ext4" PARTLABEL="system" 
/dev/sdd1: LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="97CC-45A4" TYPE="vfat" PARTLABEL="boot" 
/dev/sdd2: LABEL="system" UUID="bd04242b-b64d-46e7-97ba-116d3c987c67" TYPE="ext4" PARTLABEL="system"
/dev/sdd3: LABEL="var" UUID="7142881b-3165-4b17-bf20-445e22b0cb03" TYPE="ext4" PARTLABEL="var" 
/dev/md0: UUID="4a432c95-b1ad-4ff6-b10b-66a343f2bfad" TYPE="crypto_LUKS"

/etc/default/grub (old root partition)

GRUB_DEFAULT=4
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_SUBMENU=y

/boot/grub/grub.cfg

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
set root='hd2,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
else
  search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=C
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=5
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=5
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
        set gfxpayload="${1}"
}
set linux_gfx_mode=
export linux_gfx_mode
menuentry 'Debian GNU/Linux, with Linux 4.19.0-14-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-14-amd64-advanced-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-14-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-14-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-14-amd64
}
menuentry 'Debian GNU/Linux, with Linux 4.19.0-14-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-14-amd64-recovery-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-14-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-14-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-14-amd64
}
menuentry 'Debian GNU/Linux, with Linux 4.19.0-10-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-10-amd64-advanced-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-10-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-10-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-10-amd64
}
menuentry 'Debian GNU/Linux, with Linux 4.19.0-10-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-10-amd64-recovery-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-10-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-10-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-10-amd64
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
        menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.9-amd64 (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.9-amd64--bd04242b-b64d-46e7-97ba-116d3c987c67' {
                insmod part_gpt
                insmod ext2
                set root='hd3,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
                else
                  search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
                fi
                linux /boot/vmlinuz-4.19.0-0.bpo.9-amd64 root=bd04242b-b64d-46e7-97ba-116d3c987c67 ro quiet
                initrd /boot/initrd.img-4.19.0-0.bpo.9-amd64
        }
        menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.9-amd64 (recovery mode) (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.9-amd64-root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single-bd04242b-b64d-46e7-97ba-116d3c987c67' {
                insmod part_gpt
                insmod ext2
                set root='hd3,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
                else
                  search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
                fi
                linux /boot/vmlinuz-4.19.0-0.bpo.9-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
                initrd /boot/initrd.img-4.19.0-0.bpo.9-amd64
        }
        menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.8-amd64 (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.8-amd64--bd04242b-b64d-46e7-97ba-116d3c987c67' {
                insmod part_gpt
                insmod ext2
                set root='hd3,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
                else
                  search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
                fi
                linux /boot/vmlinuz-4.19.0-0.bpo.8-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro quiet
                initrd /boot/initrd.img-4.19.0-0.bpo.8-amd64
        }
        menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.8-amd64 (recovery mode) (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.8-amd64-root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single-bd04242b-b64d-46e7-97ba-116d3c987c67' {
                insmod part_gpt
                insmod ext2
                set root='hd3,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
                else
                  search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
                fi
                linux /boot/vmlinuz-4.19.0-0.bpo.8-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
                initrd /boot/initrd.img-4.19.0-0.bpo.8-amd64
        }

### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
menuentry 'System setup' $menuentry_id_option 'uefi-firmware' {
        fwsetup
}
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

/etc/fstab (new root partition)

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / on new SDD on /dev/sdd2
UUID=bd04242b-b64d-46e7-97ba-116d3c987c67 /               ext4    noatime,nodiratime,errors=remount-ro 0       1
# /boot/efi on /dev/sdd1
UUID=97CC-45A4  /boot/efi       vfat    umask=0077      0       1
# /var on /dev/sdd3
UUID=7142881b-3165-4b17-bf20-445e22b0cb03 /var            ext4    defaults   0    2
tmpfs           /tmp            tmpfs   defaults        0       0
# >>> [openmediavault]
/dev/disk/by-label/data /srv/dev-disk-by-label-data ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
# <<< [openmediavault]

Answer 1

grub-mkconfig is mainly built to generate GRUB configs for the system it is currently running from, so in order to determine the root UUID it simply looks at the / filesystem.

So if you need to switch the root filesystem, you'll need to override this by manually specifying a new root= parameter in GRUB_CMDLINE_LINUX.

(I would just manually edit grub.cfg by hand to have the UUIDs I want, get it to boot, and once it's actually booting it's finally time to go back to grub-mkconfig.)

Also I noticed that /boot/grub/grub.cfg does not change after I run grub-mkconfig.

That's expected. You actually have to tell the program to write its output to a file, otherwise it won't. For example, this is what Debian's update-grub does:

grub-mkconfig -o /boot/grub/grub.cfg

And while I see the UUIDs of the root partitions in the /boot/grub/grub.cfg file, I don't see neither the old nor the new ones anywhere in the /etc/grub.d structure.

The /etc/grub.d directory doesn't contain the configuration – it contains scripts which try to automatically determine the configuration on the fly. Their output is what goes into grub.cfg.

For example, /etc/grub.d/10_linux is the script responsible for guessing your current root partition UUID and outputting all of the primary menuitems.