Questions tagged [deduplication]
The deduplication tag has no usage guidance.
99
questions
2
votes
1
answer
102
views
How do Borg's "prune" work with deduplication?
Here's how I currently (mis)understand how it works : everytime I run
borg create /path/to/my/repo::[archive name] /home/myData
Borg creates a new archive. Thanks to the deduplication mechanism, this ...
- 267
5
votes
1
answer
414
views
Windows Data Deduplication: Determine which file/s use a given chunkfile?
Using Windows Server's data deduplication feature:
Given a specific chunk file in the chunk store, is it possible to determine which file/s include that chunk?
The use case here is that unreadable ...
- 181
0
votes
1
answer
314
views
rsync cache/deduplication for similar files
I am using rsync to transfer/update 100s of TB from one fileserver to another, millions of files.
Many of those files are quite similar (but not identical).
I was wondering if there's a way to benefit ...
- 325
1
vote
1
answer
545
views
Dovecot process synchronization for duplicate email detection
We have a dovecot server version 2.3.16 with sieve duplicate extension enabled to detect and discard duplicates. We have duplicate routing in Postfix virtual_aliases file which causes duplicate copies ...
- 111
1
vote
1
answer
996
views
Migrating file server with Windows Deduplication by switching disk
I'm migrating a bunch of large file servers to new Windows VMs as a way of updating the OS, planning on switching the virtual disks over and exporting/importing the lanman share registry and retaining ...
- 21
0
votes
1
answer
186
views
Recover files from deduplicated volume in Windows Storage Server 2016
We have a Windows Storage Server 2016 having some volumes with Storage Deduplication enabled.
Its OS crashed so we had to re-install the OS (couldn't restore from backup).
There are some files which ...
- 103
1
vote
1
answer
582
views
zfs zpool dedup stat seems very wrong - how to interpret these values?
I run a fileserver receiving backups of user containers. Two containers are poorly run docker systems with hundreds of near-identical directories not using overlayfs nor zfs clones. (I cannot touch ...
- 275
1
vote
1
answer
183
views
rsync Delete only Matching files
How can I use rsync (or another program) to delete identical files between directories? To be identical, location and metadata should match (permissions, ownership, timestamp)
For example, I backup ...
- 166
1
vote
1
answer
218
views
Windows hardlinks - why it consumes same space as original file
Im using finddupe tool to de-duplicate files and save some space on HDD.
But checking before and after operation I cant see any free space released - see below.
Can somebody advise what is wrong here ?...
- 133
-1
votes
2
answers
1k
views
Find file duplicates and convert them into links [WINDOWS] [closed]
My users tend to save tons of duplicate files what consumes more and more space and generate HW and archiving cost.
Im thinking to create some scheduled job, to:
find duplicate files (check file MD5 ...
- 133
1
vote
1
answer
1k
views
Workaround for Windows Search/Indexing on a deduplicated volume
According to Microsoft, Microsoft Windows Server 2019 still does not support Windows Search on Data Deduplication enabled volumes (source):
Windows Search doesn't support Data Deduplication. Data ...
- 687
1
vote
1
answer
403
views
Dedup.sys 0x7E after inplace upgrade Server 2016 => 19
I've recently performed an in-place upgrade from Server 2016 => 2019 (non-domain).
2 of the drives connected has been configured with data deduplication (mostly VHDs for Hyper-V). Ever since the ...
3
votes
1
answer
3k
views
Cheap/fast deduplication with hardlinks?
I've got shared hosting with a few thousand Wordpress installs and I've wanted for ages to have a nice way of removing all the duplicate files in a sensible and secure way.
I'm looking for better disk ...
- 287
3
votes
1
answer
1k
views
ZFS interpret output of zdb -S tank
I wanted to know if it would pay it out for me to activate the zfs deduplication so I ran the command
zdb -S tank but know I need some help to interpret the output.
Simulated DDT histogram:
bucket ...
- 135
1
vote
0
answers
624
views
btrfs compress existing data with snapshots
I'm runninga a btrfs raid1 on 5 8 TB disks, where primarily urbackup stores backups to. Urbackup uses snapshots extensively. compsize produces the following output:
Processed 106161732 files, 57335496 ...
- 111