The go to open source product on Linux (and everything else) is [Radare][1] which provides [`radiff2`][2] explicitly for this purpose. 

> *for every different byte*

That's insane though. Because as asked, if you insert one byte at the first byte in the file, you'd find every subsequent byte was different and so the diff would repeat the whole file, for an actual difference of one byte.

Slightly more practical is `radiff -O`. The `-O` is for ""Do code diffing with all bytes instead of just the fixed opcode bytes""

    0x000000a4 0c01 => 3802 0x000000a4
    0x000000a8 1401 => 3802 0x000000a8
    0x000000ac 06 => 05 0x000000ac
    0x000000b4 02 => 01 0x000000b4
    0x000000b8 4c05 => 0020 0x000000b8
    0x000000bc 4c95 => 00a0 0x000000bc
    0x000000c0 4c95 => 00a0 0x000000c0

Like IDA Pro, Radare is a tool primary for binary analysis, and you can also show delta diffing with `-d`, or display the disassembled bytes instead of hex with `-D`.

See also:

* [Radare's `radiff2` for binary diffing](https://r2wiki.readthedocs.io/en/latest/tools/radiff2/)

  [1]: https://github.com/radare/radare2
  [2]: https://r2wiki.readthedocs.io/en/latest/tools/radiff2/