-2

I have set up a NAS Server running Ubuntu 23.10 minimal and Jellyfin Media Server for my Family in December 2023. We put our whole family files on it (films, photo, music etc), last time I checked there was 5.6TB of data.

Then, three weeks ago, out of the blue, my brother tells me there that he doesn't see anything left on /8TB, the mountpoint of our 8TB drive. Jokingly, I tell him that it must be some kind of mount error, so I unplug and replug the server outlet. To my dismay, it didnt solve the problem. Our ~6 Terabytes of data really aren't there anymore.

Fortunately, 99.9% of the files were backed up elsewhere, but as a somewhat experienced Ubuntu user, it baffles me that I cant find out what happened to my files.

Please help me. As you will see, I have tried to understand what happened, but I don't get anywhere, so any idea is welcome.

As for the configuration, The SSD Drive is a Samsung QVO 8TB, whose physical volume (PV) is /dev/sda, whose logic volume (LV) is /dev/mapper/nas-8TB which contains an ext4 filesystem which was mounted on /8TB directory

I had configured my /etc/fstab file with the following directive so that the disk would be mounted upon booting :

UUID=d2890436-a37f-4bd7-91d6-e2d7ae5a747a /8TB ext4 defaults 0 0

The UUID being that of /dev/mapper/NAS-8TB

Here are my attemps at recovering my files :

  1. I checked if my /8TB directory what correctly mounted with mount | grep /8TB output : /dev/mapper/NAS-8TB on /8TB type ext4 (rw,relatime) -> the /8TB folder is correclty mounted

  2. I unmounted and remounted the filesystem with umount /8TB and mount /dev/mapper/NAS-8TB /8TB -> still no files (actually, I have to be precise, there are SOME files left, I have a /SERIES directory with about twenty-five .nfo files in there in there, they have the same name as the .mkv files that were here before, but instead they are now .nfo, example Eden.2019.S01E01.FRENCH.HDTV.XviD-EXTREME.mkv -> Eden.2019.S01E01.FRENCH.HDTV.XviD-EXTREME.nfo these new .nfo files weigh around 6ko (yes...), I checked them with VSCode, its just some metadata. The previous MKV files were around ~700mo I think

  3. I checked the disk usage with df -h /dev/mapper/NAS-8TB output : /dev/mapper/NAS-8TB 7.3T 2.7M 6.9T 1% /8TB -> it correlates with what I see on my terminal, I don't have anything left

  4. Tried du -sh /8TB output : 708K /8TB -> it correlates with what I see on my terminal, I don't have anything left

  5. Tried to check for errors with dmesg | grep /dev/mapper/NAS-8TB and sudo journalctl -xe | grep /dev/mapper/NAS-8TB output : nothing

  6. I Unmounted and Inspected the Mount Point with sudo umount /8TB and ls -l /8TB output : total 0

  7. I remounted the filesystem and checked again with sudo mount /dev/mapper/NAS-8TB /8TB" and "ls -l /8TB output : total 4 -> I have around 8 folders containing 25 .nfo files in total so 4 doesn't correlate

  8. I mounted read only and inspected again with sudo mount -o ro /dev/mapper/NAS-8TB /8TB and ls -l /8TB output : total 4 -> still the same

  9. I checked for filesystem errors with sudo umount /dev/mapper/NAS-8TB and sudo fsck /dev/mapper/NAS-8TB" ouput : fsck from util-linux 2.39.1 e2fsck 1.47.0 (5-Feb-2023) /dev/mapper/NAS-8TB: clean, 144/244191232 files, 15689031/1953506304 blocks -> filesystem looks clean

  10. I checked disk health with sudo smartctl -a /dev/sda output :

    === START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 QVO 8TB
Serial Number:    S5SSNF0W410252Y
LU WWN Device Id: 5 002538 f43420290
Firmware Version: SVQ02B6Q
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jun 26 17:05:49 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
(lot of lines cropped)

    -> The output from smartctl indicates that my Samsung SSD 870 QVO 8TB drive is in good health, with no reallocated sectors, uncorrectable errors, or other signs of imminent failure

  11. Tried to check the journal for 8TB with sudo journalctl | grep "sudo journalctl | grep '8TB' output : 433k log lines -> the only thing I can deduct from these is that, by analyzing the jellyfin logs, jellyfin starts removing all my files from its library from the 29th may 12:33 -> I don't see any unusual activity before

  12. Tried to check for unusual login activity with sudo journalctl _COMM=sshd | grep "session opened" -> I don't see any unusual activity, my brother never logs via SSH, as he is not used to command line. I have to indicate something here, while yes it is possible that it was my brother, the only other user, who deleted all the files, it is highly unlikely as he is the one who spent DAYS formatting all the files (eg deleting all the release info from movie titles, grouping series into folders etc) and that is the main thing we lost, and he was really bummed to see all his work gone

  13. I tried to check the LVM metadata Backups with sudo vgcfgrestore --list NAS

    output :

    File:       /etc/lvm/archive/NAS_00000-1406878796.vg/NAS_00000-1406878796.vg
VG name:        NAS
Description:    Created *before* executing 'vgscan'
Backup Time:    Wed Jun 12 00:35:55 2024


File:       /etc/lvm/backup/NAS/NAS
VG name:        NAS
Description:    Created *after* executing 'vgscan'
Backup Time:    Wed Jun 12 00:35:55 2024

    -> both backups date is 12th june, so two weeks after my files are gone

  14. I have a lot of lines like this one : jellyfin[870]: [00:27:23] [INF] Removing item, Type: Series, Name: The Night Manager, Path: /8TB/SERIES/The night manager S01, Id: 37448ee9-3204-ba94-4daa-12a495486740 but after checking, it's just jellyfin removing items from its own library because the original files aren't there anymore. It could have been jellyfin who deleted all my files, IF I had made the mistake of configurating the cache folder as my /8TB folder, which was NOT the case

  15. I checked both my brother and mine SSH history thoroughly, nothing comes close to a delete command.

  16. I checked our FTP history thoroughly, once again no "delete" command

  17. I tried to check if the server was hacked, but it doesn't show ANY signs of a hack (and I've been hacked before so security was pretty tight) :

    • our router public IP isn't blacklisted
    • no users were created
    • and more importantly, after one week of Monitoring, no extra services/disk usage/cpu usage was detected

I don't know what to do anymore.

Improve this question
New contributor
granville is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
9
  • 3
    What is the question?
    – Joep van Steen
    Commented Jun 29 at 22:12
  • how is the three week time interval relevant to the problem?
    – jsotola
    Commented Jun 30 at 1:12
  • 3
    " Jokingly, I tell him that it must be some kind of mount error, so I unplug and replug the server outlet. " NOOOOOOOO. That in itself could potentially cause data loss, and 'hide' transient issues.
    – Journeyman Geek ♦
    Commented Jun 30 at 1:51
  • @jsotola the three weeks mean that I've spent lot of time on this, for example checking if the server has been hacked is not an easy task in itself and took time
    – granville
    Commented Jun 30 at 23:50
  • @JoepvanSteen I've edited the question for clarity
    – granville
    Commented Jun 30 at 23:51

0

Reset to default

Browse other questions tagged .