Understanding the “memmap” Linux parameter

Question

I'm trying to understand the “memmap” parameter from the Linux kernel (see The kernel’s command-line parameters), in particular, the following options:

memmap=nn[KMG]$ss[KMG]
    [KNL,ACPI] Mark specific memory as reserved.
    Region of memory to be reserved is from ss to ss+nn.
    Example: Exclude memory from 0x18690000-0x1869ffff
        memmap=64K$0x18690000
        or
        memmap=0x10000$0x18690000
    Some bootloaders may need an escape character before '$',
    like Grub2, otherwise '$' and the following number
    will be eaten.

memmap=nn[KMG]!ss[KMG]
    [KNL,X86] Mark specific memory as protected.
    Region of memory to be used, from ss to ss+nn.
    The memory region may be marked as e820 type 12 (0xc)
    and is NVDIMM or ADR memory.

As per the documentation, I fail to grasp the differences of those options or the ultimate goals. While question Linux kernel difference between protected and reserved memory? (memmap parameter) states the differences, I still do not understand the output of top and dmesg generate with different memmap options as detailed below.

I'm using Ubuntu 24.04 through QEMU to experiment with the options and see the effects. The QEMU is running an x86_64 with 2 GB of memory and 1 hotplug memory stick (the one used to play with the parameters). This is the command line to launch the guest:

./qemu-system-x86_64 -M pc -m 2G,slots=4,maxmem=16G -object memory-backend-file,size=1G,id=mem0,mem-path=./test_ram -device pc-dimm,memdev=mem0,id=dimm0 -drive file=ubuntu_24.04.raw,format=raw -net nic -net user -nographic -serial mon:stdio

No memmap set:

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.8.0-31-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro console=ttyS0 nokaslr             

Top reports:

GiB Mem :      2.9 total,      2.5 free,      0.4 used,      0.2 buff/cache     
GiB Swap:      1.6 total,      1.6 free,      0.0 used.      2.5 avail Mem

Memmap set to 1G$2G

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.8.0-31-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro console=ttyS0 nokaslr memmap=1G$2G

Top reports:

GiB Mem :      2.9 total,      2.5 free,      0.4 used,      0.2 buff/cache     
GiB Swap:      1.6 total,      1.6 free,      0.0 used.      2.5 avail Mem 

Memmap set to 1G!2G:

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.8.0-31-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro console=ttyS0 nokaslr memmap=1G\\!2G

Top reports:

GiB Mem :      1.9 total,      1.6 free,      0.4 used,      0.2 buff/cache     
GiB Swap:      1.6 total,      1.6 free,      0.0 used.      1.5 avail Mem 

dmesg for all three options:

[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffdffff] usable
[    0.000000] BIOS-e820: [mem 0x000000007ffe0000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved

Seeing this, multiple questions:

1. Why is the physical RAM map the same for all of them?
2. The usable RAM is from 0x100000 to 0x7ffdffff, which is roughly 2 GB.  Where does this number come from?  Why not from 0x100000 to 0x80100000 (exactly 2 GB)?
3. Why for option "Memmap set to 1G$2G" does Linux top report almost 3 GB of memory?  We reserved 1 GB.
4. For options "Memmap set to 1G$2G" and "Memmap set to 1G!2G", where exactly in the physical RAM map is the 1 G reserved?
5. Why would I want to use the "$" option over the "!" option for memmap?

Answer 1

Why is the physical RAM map the same for all of them?

The address map is provided by the BIOS, as stated in dmesg. So there's no reason to expect it to change based on OS kernel parameter. Your memmap parameter SHOULD be the one that follows the BIOS-provided map, not the other way around.

Since you use QEMU, its worth to check whether there's a mountable pmem when you allocate protected memory. Since IIRC they have their own persistent memory emulation layer (both virtio pmem and nvdimm device).

The usable RAM is from 0x100000 to 0x7ffdffff, which is roughly 2 GB. Where does this number come from? Why not from 0x100000 to 0x80100000 (exactly 2 GB)?

Same reason as above (OS did not control addressing on physical level).

Why for option "Memmap set to 1G$2G" does Linux top report almost 3 GB of memory? We reserved 1 GB.

memmap=1G$2G vs memmap=1G\\!2G

Escaping seems to be inconsistent.

For options "Memmap set to 1G$2G" and "Memmap set to 1G!2G", where exactly in the physical RAM map is the 1 G reserved?

One thing explicitly pointed out in the documentation is that memmap deal with memory region and offset, not size. While they accept KMG multiplier, it rarely make sense and cause confusion like this. But to answer the question: When you supply 1G[$!]2G, it will start from region address 2G, ended in offset 1G from start address.

Why would I want to use the "$" option over the "!" option for memmap?

It's not an "either X or Y" choice since their use case are not overlapping.

• You "reserve" memory (using $) to prevent crash and corruption from reasons beyond OS's control (e.g faulty RAM or wonky BIOS behavior).
• You "protect" memory (using !) to prevent it from being used as volatile data store. After allocation, it will be available as mountable device under /dev/pmem.