Cryptsetup with dm-integrity - weird benchmark results

Question

I'm benchmarking various cryptsetup volumes and I'm getting unexpected results on Debian.

I'm using numbers from this talk as a rough reference. One of the slides shows benchmark results for various configurations:

My setup is not identical and I'm running all tests in VMs, so I don't expect results to be exactly identical, but I think they should roughly reflect what's on the slide. In particular I expect to see performance drop of about 35 for authenticated integrity modes (AES-XTS,HMAC-SHA256) compared to non-authenticated counterparts (AES-XTS) and then another 35% for journaled integrity vs. non-journaled integrity.

But here are my results, similar for Ubuntu Server 20.04 and Debian 10.4:

LUKS2 container:
    Capacity    1056964608 B
    Read        26.5MB/s
    Write       8855kB/s

LUKS2 with hmac-sha256, no journal:
    Capacity    1040322560 B
    Read        19.0MB/s
    Write       6352kB/s

LUKS2 with hmac-sha256, journaled:
    Capacity    1040322560 B
    Read        18.9MB/s
    Write       6311kB/s

About 30% performance drop after enabling integrity, that's expected. But then the difference between journaled and non-journaled integrity is marginal. I mean, that's much better than original benchmark so I should be happy, but how do I know that the journal is actually working and if it is, how do I opt out?

Here are my cryptsetup format commands:

cryptsetup luksFormat --type luks2 /dev/sdb --sector-size 4096
cryptsetup luksFormat --type luks2 /dev/sdb --sector-size 4096 --integrity hmac-sha256
cryptsetup luksFormat --type luks2 /dev/sdb --sector-size 4096 --integrity hmac-sha256 --integrity-no-journal

Benchmark command:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/dev/mapper/sdb --bs=4k --iodepth=64 --readwrite=randrw --rwmixread=75

VMs are configured on VirtualBox 6.1 with settings default for Debian or Ubuntu respectively. Disks are 1 GB VDIs, fixed size and pre-filled with zeros, host buffering disabled. Underlying SSD is using 4k sectors, hence --sector-size 4096.

Interestingly, both the basic --integrity variant and the --integrity-no-journal one create intermediate sdb_dif mapped device with journal and both sdb devices have identical size:

$ sudo integritysetup status /dev/mapper/sdb_dif
/dev/mapper/sdb_dif is active and is in use.
  type:    INTEGRITY
  tag size: 32
  integrity: (none)
  device:  /dev/sdb
  sector size:  4096 bytes
  interleave sectors: 32768
  size:    2031880 sectors
  mode:    read/write
  failures: 0
  journal size: 8380416 bytes
  journal watermark: 50%
  journal commit time: 10000 ms

$ sudo blockdev --getsize64 /dev/mapper/sdb
1040322560

Answer 1

Summary of answer:

cryptsetup format ignores the --integrity-no-journal flag.

Instead, your options are:

• At each open, always provide --integrity-no-journal.
• At your first open (i.e. when formatting the inner device with a filesystem, or to add the inner device to an MD RAID), provide --persistent --integrity-no-journal to persist the --integrity-no-journal setting. Then future open will not need the flag. This option only works with cryptsetup, not if you are using direct integritysetup.
• While the device is already opened, issue a refresh --persistent --integrity-no-journal. This option only works with cryptsetup, not if you are using direct integritysetup.

---

Old text:

Did you provide the --integrity-no-journal flag to integritysetup open? It looks like dm-integrity does not save the (non-)existence of a journal in the superblock when you format.

I formatted a USB 2.0 flash disk partition with integritysetup format /dev/sdb1 --no-wipe.

Then I opened it with integritysetup open /dev/sdb1 int-sdb1, and did sync; echo 1 > /proc/sys/vm/drop_caches; dd count=16384 bs=4096 if=/dev/zero of=/dev/mapper/int-sdb1. This consistently gave me results of between 2.1Mb/s to 2.4Mb/s.

I closed it and then reopened with integritysetup open /dev/sdb1 int-sdb1 --integrity-no-journal, and issued the same dd command. This time it gave me from between 4.0Mb/s to 7.0Mb/s, a marked improvement. The massive variance might be due to the flash translation layer; it's a lousy throwaway cheap disk.

I repeated this again with integritysetup format /dev/sdb1 --no-wipe --integrity-no-journal. Again, what matters is wheter you gave --integrity-no-journal to the open command, not to the format command.

So it might be a non-clarity of integritysetup. If you gave --integrity-no=journal to the format command.