How to diff three binary files to show only what changed between 1 and 2 and then changed back between 2 and 3?

Question

I have three binary files (memory dumps). Call them file1, file2, file3. I'm trying to debug some software, and I'm toggling a switch.

• file1 = switch off

• file2 = switch on

• file3 = switch off

I need to know which bytes changed between file1 and file2, which then also changed back to the same values (as file1) in file3.

There are a lot of unrelated changes between file1, file2 so that diff alone is not enough for me to determine what is changing when I toggle this switch, I'm trying to identify the unique bytes of entropy that change from file1,2,3,…

I know there are tools such as xxd, diff, vimdiff, colordiff. I'm just not sure how best to use them for this problem.

Answer 1

To know bytes that changed between file1 and file2 and their respective values, use cmp -l:

cmp -l file1 file2 > changes12

Similarly for file3 and file2. The trick is to always investigate in the same direction (here: switch off to on), that's why I put file2 at the end:

cmp -l file3 file2 > changes32

Now you can find identical changes:

comm -12 changes12 changes32

and the output will be like (example):

     1629 152 112

which means the byte 1629 (decimal, numbering starts with 1) changed from 152 (octal) to 112 (octal).

---

Notes:

• cmp -l in my Ubuntu "columnizes" its output. This means it may print lines with leading spaces, the width of the first column depends on the input size. I guess some implementations may not do this. There are few concerns:
  • If the original files are of different size (probably not your case), one cmp may generate wider first column than the other. In the context of later comm this is unacceptable. You can "decolumnize" the output by piping to awk '{print $1" "$2" "$3}'.
  • If the output is not "columnized" (or has been "decolumnized"), comm may complain the files are not sorted. You need sort (not sort -n) before you save to changesAB. This may generate somewhat unexpected order (e.g. differing byte 23 will appear after differing byte 100453) which may be fixed by piping the output of comm to sort -n.

• changes* files may be huge. They are intermediate and temporary, so process substitution may be a good approach. This is non-POSIX though:

  # Korn shell syntax example
  comm -12 <(cmp -l file1 file2) <(cmp -l file3 file2)
  

• The output from comm can be used with yet another and another output of cmp to filter out unrelated changes better:

  comm -12 changes12 changes32 > result1
  cmp -l file4 file5 | comm -12 - result1 > result2
  cmp -l file6 file5 | comm -12 - result2 > result3
  

  But remember:

  • You always need to cmp in the same direction (e.g. switch off to on).
  • Beware of this: Using the same file as stdin and stdout results in empty file.
    … | comm -12 - result1 > result1 is wrong.

---

Documentation:

• cmp
• comm

Answer 2

I'm a Windows user and I've used Beyond Compare for many years for file compares, including comparing 3 files (2 at a time) to each other.

It appears that Beyond Compare also has a Linux distro, so you might want to check it out.

https://www.scootersoftware.com/download.php?zz=kb_linux_install

I know the Windows version has options to show just differences, as well as a lot of other features that might help you out. I can't remember if they have a 3 way compare or not.

I'm not dev or sales for them, I just like the software a lot.

Answer 3

Based on the response from Kamil, I used this to get what I needed:

cmp -l file1 file2 | awk '{print $1" "$2" "$3}' | sort > changes_12

cmp -l file3 file2 | awk '{print $1" "$2" "$3}' | sort > changes_32

comm -12 changes_12 changes_32 > common_changes