Skip to main content
Super User

Return to Question

deleted 2 characters in body; edited tags; edited title
Source Link
slhck
slhck
  • 230.2k
  • 71
  • 621
  • 603

How to diff large files on linux?Linux

I'm getting a "diff: memory exhausted"diff: memory exhausted error when trying to diff two 27 GB files that are largely similar on a linuxLinux box with CentOS 5 and 4 GBsGB of RAM.

  This is a known problem, it seems: http://zotline.com/shownote.zot/NoteNum/4375.html.

I would expect there to be an alternative for such an essential utility, but I can't find one. I imagine the solution would have to use temporary files rather than memory to store the information it needs.

I tried to use rdiff and xdelta, but they are better for showing the changes between two files, like a patch, and are not that useful for inspecting the differences between two files.

Tried VBinDiff, but it is a visual tool which is better for comparing binary files. I need something that can pipe the differences to STDOUT like regular diff.

There are a lot of other utilities such as vimdiff that only work with smaller files.

I've also read about Solaris bdiff but I could not find a port to linux.

  • I tried to use rdiff and xdelta, but they are better for showing the changes between two files, like a patch, and are not that useful for inspecting the differences between two files.
  • Tried VBinDiff, but it is a visual tool which is better for comparing binary files. I need something that can pipe the differences to STDOUT like regular diff.
  • There are a lot of other utilities such as vimdiff that only work with smaller files.
  • I've also read about Solaris bdiff but I could not find a port for Linux.

Any ideas besides splitting the file into smaller pieces? I have 40 of these files so trying to avoid the work of breaking them up.

Thanks

How to diff large files on linux?

I'm getting a "diff: memory exhausted" error when trying to diff two 27 GB files that are largely similar on a linux box with CentOS 5 and 4 GBs of RAM.

  This is a known problem, it seems: http://zotline.com/shownote.zot/NoteNum/4375.html

I would expect there to be an alternative for such an essential utility, but I can't find one. I imagine the solution would have to use temporary files rather than memory to store the information it needs.

I tried to use rdiff and xdelta, but they are better for showing the changes between two files, like a patch, and are not that useful for inspecting the differences between two files.

Tried VBinDiff, but it is a visual tool which is better for comparing binary files. I need something that can pipe the differences to STDOUT like regular diff.

There are a lot of other utilities such as vimdiff that only work with smaller files.

I've also read about Solaris bdiff but I could not find a port to linux.

Any ideas besides splitting the file into smaller pieces? I have 40 of these files so trying to avoid the work of breaking them up.

Thanks

How to diff large files on Linux

I'm getting a diff: memory exhausted error when trying to diff two 27 GB files that are largely similar on a Linux box with CentOS 5 and 4 GB of RAM. This is a known problem, it seems.

I would expect there to be an alternative for such an essential utility, but I can't find one. I imagine the solution would have to use temporary files rather than memory to store the information it needs.

  • I tried to use rdiff and xdelta, but they are better for showing the changes between two files, like a patch, and are not that useful for inspecting the differences between two files.
  • Tried VBinDiff, but it is a visual tool which is better for comparing binary files. I need something that can pipe the differences to STDOUT like regular diff.
  • There are a lot of other utilities such as vimdiff that only work with smaller files.
  • I've also read about Solaris bdiff but I could not find a port for Linux.

Any ideas besides splitting the file into smaller pieces? I have 40 of these files so trying to avoid the work of breaking them up.

edited tags
Link
Journeyman Geek ♦
Journeyman Geek ♦
  • 130.3k
  • 52
  • 272
  • 443
Source Link
Tom B
Tom B
  • 441
  • 1
  • 4
  • 3

How to diff large files on linux?

I'm getting a "diff: memory exhausted" error when trying to diff two 27 GB files that are largely similar on a linux box with CentOS 5 and 4 GBs of RAM.

This is a known problem, it seems: http://zotline.com/shownote.zot/NoteNum/4375.html

I would expect there to be an alternative for such an essential utility, but I can't find one. I imagine the solution would have to use temporary files rather than memory to store the information it needs.

I tried to use rdiff and xdelta, but they are better for showing the changes between two files, like a patch, and are not that useful for inspecting the differences between two files.

Tried VBinDiff, but it is a visual tool which is better for comparing binary files. I need something that can pipe the differences to STDOUT like regular diff.

There are a lot of other utilities such as vimdiff that only work with smaller files.

I've also read about Solaris bdiff but I could not find a port to linux.

Any ideas besides splitting the file into smaller pieces? I have 40 of these files so trying to avoid the work of breaking them up.

Thanks