Questions tagged [deduplication]
Deduplication is the finding and removal of duplicate files, or to keep both files but share the same allocation unit on the storage medium
154
questions
0
votes
0
answers
46
views
Use fdupes to compare two directories
Is it possible to use fdupes to compare the contents of two directories (dir1 and dir2), identify files in dir2 that are present in dir1, and delete those duplicates files from dir2?
Alternatively, ...
- 153
0
votes
1
answer
67
views
In Linux with limited features (NAS box), how can I find duplicate files, then remove duplicates and substitute hardlinks to one file?
I have a NAS box running some version of Linux that I use for backing up anything and everything.
It is essentially an absolute certainty that some of the files are identical duplicates.
That being ...
- 123
0
votes
2
answers
358
views
Is there a way to select the first instance of a duplicated file in dupeguru?
dupeguru has helped me find duplicated files, but it only allows me to delete the second instance of duplicated files (as shown in the screenshot). I wonder if there's a way to delete the first ...
- 113
2
votes
1
answer
120
views
Can I control which file jdupes preserves when using hardlinks?
The jdupes deduplication program allows you to create hardlinks, so that the files are not deleted, but linked together so that the disk space is used only once for multiple copies.
Is there a way to ...
- 1,158
1
vote
1
answer
215
views
How to remove duplicated text on every line using Notepad++?
I want to remove duplicated words in each line using Notepad++.
Example:
Flooring Services, Carpet, Flooring Services, Tile, Flooring Services
In the above, Flooring Services is repeated 3x. I only ...
- 19
0
votes
1
answer
692
views
Removing duplicate rows in a CSV file as compared to another CSV file
I have a CSV file named CSV1 with over 5000 rows and 3 columns - last name, first name and email address. I have another file named CSV2 with about 2700 rows with the same 3 columns. I have to remove ...
0
votes
0
answers
108
views
ZFS deduplication table not paged out to RAM
I have a pool on which there is 3.41 TB of data, deduplication is enabled on the volumes, judging by the output of the
zpool status -D <pool_name>
...
dedup: DDT entries 73665285, size 696B on ...
- 1
0
votes
0
answers
143
views
Remove but 'remember' (to later restore) duplicate frames from video/ image sequence?
I'm upscaling animated content (via ChaiNNer) in which a number of frames (often 25-50%) are duplicates, wasting significant gpu time; I've been attempting to remove duplicates pre upscale but have ...
0
votes
1
answer
397
views
btrfs fi du: several sets of shared extents
I have a lot of files in various directories on a btrfs filesystem. They are all identical, and some may have been created using a "old-style" full copy, some are clones (cp --reflink). I ...
- 1,118
1
vote
0
answers
446
views
How to overwrite all files only if SAME size and exact duplicate while renaming if different size?
I'm using Teracopy, but Windows also does not have this function. Instead it wants me to replace the file only if DIFFERENT, or skip it if the name is the same. I want to do the opposite.
I need to ...
- 3,524
1
vote
2
answers
914
views
Can I find duplicate files with the same path in different locations?
I have two folders/directories: C:\MyData and C:\MyDataBackup and the person that owns those two folders/directories and does not remember if they have edited the files in the original or in the ...
- 167
11
votes
1
answer
643
views
In bash, how to find all copies of a given file in particular directories?
Let's say we have a file /a_long_path_1/foo.doc of size, say, 12345 bytes, and we would like to find all copies of this file in directories /a_long_path_2 and /a_long_path_3 including all their ...
user1757545
0
votes
0
answers
253
views
How can I look for duplicate files from different base folders in bash?
I know fdupes, which can find duplicate files from various directories I provide it:
fdupes -r <directoryA> <directoryB> <...>
But this seems to “throw all files into the same ...
- 1,537
1
vote
1
answer
1k
views
Remove duplicate files within specific directories, keeping them in another with Bash
I've seen a few questions on here regarding removing duplicate files in various circumstances but I can't find anything along the lines of what I'm trying to accomplish.
The closest I came was seeing ...
- 11
0
votes
2
answers
544
views
File Deduplication and Filtering
I had to run a TestDisk file recovery software on one of my drives. The result was a little over 16,000 files who's names are basically an alpha numeric string. Many of these files contain duplicate ...
- 103