Diff outputs "binary files <file1> <file2> differ" but not the usual detail output

Question

I would like to compare two files - "orienv" and "currenv", using the command diff.

The way I had created the two files was as follow:

1. Createing the "currenv" file

   $cat /proc/1/environ >> currenv
   $cat /pcoc/279/environ >> currenv
   $cat /proc/295/environ >> currenv
   //295 is the pid of the current console
   

2. Creating the orienv file

   $printenv > orienv
   

Then I called diff as follow

diff -u orienv currenv

and got the following output

Binary files orienv and currenv differ

I was expecting a normal diff output with the flag -u (e.g. output in which it shows the differences in hunks, indicating which file has which information that the other does not.

What went wrong?

Answer 1

The value of an environment variable can contain line breaks. In /proc/PID/environ, variables are separated by null bytes, which cannot appear in the value or in the name of an environment variable. (This is, not coincidentally, how the environment is represented in memory.) When diff sees null bytes, it decides that the file is binary (which is true by definition: text files don't contain null bytes) and gives up on displaying differences, because for most binary formats what diff would print out is not useful.

You can tell diff to go ahead and treat the files as text with diff --text, but if you do this between two /proc/PID/environ files, this is likely to give you a display consisting of just one or two big changed lines, because in practice the environment doesn't contain many line breaks. Diff only works with lines that are separated with a newline character. And if you do this between the content of /proc/PID/environ and the output of printenv, this will say that everything has changed, because printenv uses newlines as the separator.

To get useful output, translate the null bytes into line breaks. This way each environment variable will start at the beginning of a line.

diff -u orienv --label=currenv <(tr '\0' '\n' <currenv)

To get useful and unambiguous output, also translate line breaks into null bytes, so that each environment variable is on its own single line. Then, if there's a null byte in the diff output, it indicates that the original file contains a newline at this position.

diff -u --text --label=currenv <(tr '\0\n' '\n\0' <currenv1) --label=currenv <(tr '\0\n' '\n\0' <currenv2)

Answer 2

/proc/*/environ are not text files. use strings:

strings /proc/{1,279,295}/environ >> currenv
env > orienv
diff -u orienv currenv

Answer 3

The issue is because of /proc/<pid>/environ contains null byte. You can view the unprintable byte represented as text by cat -v, e.g. you can see ^@ represents null byte in below output :

$ cat -v /proc/20148/environ 
CLUTTER_IM_MODULE=xim^@COLORFGBG=15;0^@COLORTERM=truecolor^@ ...cont.

Despite of this, there are three common issues need to consider when print environment variables:

1. The ANSI color escape codes value will print as "color" instead of literal on terminal, e.g. export p_red=$(tput setaf 1). This causes your diff output undesired color and also missing the color code value. diff --color=always will also not working as expected because of that interrupted colors.

2. Function definition normally contains multiple lines. And also it's possible for variable value contains multiple lines. e.g.:

   $ cat /etc/environment love=" 1st line 2nd line"

3. Diff without sort first will get confusing result.

You can solve all of this issues with:

$ sudo cat /proc/1/environ | tr '\n\0' ' \n' | cat -v > currenv
$ sudo cat /proc/279/environ | tr '\n\0' ' \n' | cat -v >> currenv
$ sudo cat /proc/295/environ | tr '\n\0' ' \n' | cat -v >> currenv
$ sort -u currenv -o currenv
$ printenv -0 | tr '\n\0' ' \n' | cat -v | sort -u > orienv
$ diff --unified="$(cat currenv orienv | wc -l)" orienv currenv --color=always

The output would be like that:

...
 LOGNAME=xiaobai
 love= 1st line 2nd line
..
 PKG_CONFIG_PATH=:/usr/lib/pkgconfig:/usr/local/lib/pkgconfig
 p_lblue=^[[38;5;50m
 p_lgreen=^[[38;5;118m
 p_lred=^[[38;5;196m
 p_orig=^[(B^[[m
 p_red=^[[31m
 PROFILEHOME=
...
 QT_IM_MODULE=ibus
+recovery=
+rootmnt=/root
 S_COLORS=auto

Explanations:

1. tr '\n\0' ' \n' means short-form of tr '\n' ' ' | tr '\0' '\n'. We replace newline(i.e. multiple lines value) \n to empty space ' ' first, then replace null byte \0 to newline '\n'. We no need worry ambiguous of multiple lines can either value or null byte, because we are replacing multiple lines value with space first.
2. printenv has -0 option to end each output line with NUL, not newline. And the output of printenv -0 would be same behavior as /proc/<pid>/environ. And so we can do the same thing tr '\n\0' ' \n'.
3. sort -u to sort, and also remove duplicate lines to avoid >> currenv keep append the same variables and hard to see the difference.
4. We set unified value with value of cat currenv orienv | wc -l to see all possible unified values. diff tool actually calls strtoumax() to convert that value, so you will get garbage value if you pass negative value.
5. cat -v to convert color codes to printable text. We no need worry about cat -v need separate null byte and color code because we already done tr '\n\0' ' \n' first to replace all null bytes.
6. And finally, diff --color=always to print diff with color block (green/red/white). We no need worry color code interrupt the diff color because cat -v already converted color code.