I have been running simulations in supercomputer often writing large output files in h5 format (around 750MB). For backing up data after simulations to my local computer, I used rsync command.
At the beginning, the command parses list of files and starts the copy process. On the way, the large files (in h5 format) disappear (I guess they are deleted in the supercomputer itself) for unknown reasons. The copy log is below;
sathish@HP-EliteBook:~/Documents/Research/journal/results/oblate2.5$ rsync -av --exclude '*.xdr' [email protected]:/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang* .
receiving incremental file list
Re0.1ang0/
Re0.1ang0/angle.sh
Re0.1ang0/chk_uid-info
Re0.1ang0/init.cfg
Re0.1ang0/input-default
Re0.1ang0/input-default.md
Re0.1ang0/myjob.cmd
Re0.1ang0/slurm-2430487.out
Re0.1ang0/Production/
Re0.1ang0/Production/init.cfg
Re0.1ang0/Production/input-default
Re0.1ang0/Production/input-default.md
Re0.1ang0/Production/md-cfg-desc_out_t00000000-0882612060.txt
Re0.1ang0/Production/md-cfg_out_p********-0882612060.asc
Re0.1ang0/Production/md-summary_out_t00030000-0882612060.txt
Re0.1ang0/Production/od_out_t00030000-0882612060.h5
Re0.1ang0/Production/velx_out_t00030000-0882612060.h5
Re0.1ang0/Production/vely_out_t00030000-0882612060.h5
WARNING: Re0.1ang0/Production/vely_out_t00030000-0882612060.h5 failed verification -- update discarded (will try again).
rsync: read errors mapping "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang0/Production/vely_out_t00030000-0882612060.h5": Stale file handle (116)
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang0/Production/velz_out_t00030000-0882612060.h5"
Re0.1ang10/
Re0.1ang10/angle.sh
Re0.1ang10/chk_uid-info
Re0.1ang10/init.cfg
Re0.1ang10/input-default
Re0.1ang10/input-default.md
Re0.1ang10/myjob.cmd
Re0.1ang10/slurm-2430488.out
Re0.1ang10/Production/
Re0.1ang10/Production/init.cfg
Re0.1ang10/Production/input-default
Re0.1ang10/Production/input-default.md
Re0.1ang10/Production/md-cfg-desc_out_t00000000-0882624620.txt
Re0.1ang10/Production/md-cfg_out_p00000000-0882624620.asc
Re0.1ang10/Production/md-summary_out_t00030000-0882624620.txt
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang10/Production/od_out_t00030000-0882624620.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang10/Production/velx_out_t00030000-0882624620.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang10/Production/vely_out_t00030000-0882624620.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang10/Production/velz_out_t00030000-0882624620.h5"
Re0.1ang30/
Re0.1ang30/angle.sh
Re0.1ang30/chk_uid-info
Re0.1ang30/init.cfg
Re0.1ang30/input-default
Re0.1ang30/input-default.md
Re0.1ang30/myjob.cmd
Re0.1ang30/slurm-2430489.out
Re0.1ang30/Production/
Re0.1ang30/Production/init.cfg
Re0.1ang30/Production/input-default
Re0.1ang30/Production/input-default.md
Re0.1ang30/Production/md-cfg-desc_out_t00000000-0882622039.txt
Re0.1ang30/Production/md-cfg_out_p********-0882622039.asc
Re0.1ang30/Production/md-summary_out_t00030000-0882622039.txt
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang30/Production/od_out_t00030000-0882622039.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang30/Production/velx_out_t00030000-0882622039.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang30/Production/vely_out_t00030000-0882622039.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang30/Production/velz_out_t00030000-0882622039.h5"
Re0.1ang45/
Re0.1ang45/angle.sh
Re0.1ang45/chk_uid-info
Re0.1ang45/init.cfg
Re0.1ang45/input-default
Re0.1ang45/input-default.md
Re0.1ang45/myjob.cmd
Re0.1ang45/slurm-2430490.out
Re0.1ang45/Production/
Re0.1ang45/Production/init.cfg
Re0.1ang45/Production/input-default
Re0.1ang45/Production/input-default.md
Re0.1ang45/Production/md-cfg-desc_out_t00000000-0882629804.txt
Re0.1ang45/Production/md-cfg_out_p00000000-0882629804.asc
Re0.1ang45/Production/md-summary_out_t00030000-0882629804.txt
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang45/Production/od_out_t00030000-0882629804.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang45/Production/velx_out_t00030000-0882629804.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang45/Production/vely_out_t00030000-0882629804.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang45/Production/velz_out_t00030000-0882629804.h5"
Re0.1ang60/
Re0.1ang60/angle.sh
Re0.1ang60/chk_uid-info
Re0.1ang60/init.cfg
Re0.1ang60/input-default
Re0.1ang60/input-default.md
Re0.1ang60/myjob.cmd
Re0.1ang60/slurm-2430491.out
Re0.1ang60/Production/
Re0.1ang60/Production/init.cfg
Re0.1ang60/Production/input-default
Re0.1ang60/Production/input-default.md
Re0.1ang60/Production/md-cfg-desc_out_t00000000-0882627357.txt
Re0.1ang60/Production/md-cfg_out_p********-0882627357.asc
Re0.1ang60/Production/md-summary_out_t00030000-0882627357.txt
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang60/Production/od_out_t00030000-0882627357.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang60/Production/velx_out_t00030000-0882627357.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang60/Production/vely_out_t00030000-0882627357.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang60/Production/velz_out_t00030000-0882627357.h5"
Re0.1ang80/
Re0.1ang80/angle.sh
Re0.1ang80/chk_uid-info
Re0.1ang80/init.cfg
Re0.1ang80/input-default
Re0.1ang80/input-default.md
Re0.1ang80/myjob.cmd
Re0.1ang80/slurm-2430492.out
Re0.1ang80/Production/
Re0.1ang80/Production/md-cfg-desc_out_t00000000-0882622795.txt
Re0.1ang80/Production/md-cfg_out_p00000000-0882622795.asc
Re0.1ang90/
Re0.1ang90/angle.sh
Re0.1ang90/chk_uid-info
Re0.1ang90/init.cfg
Re0.1ang90/input-default
Re0.1ang90/input-default.md
Re0.1ang90/myjob.cmd
Re0.1ang90/slurm-2430493.out
Re0.1ang90/Production/
Re0.1ang90/Production/init.cfg
Re0.1ang90/Production/input-default
Re0.1ang90/Production/input-default.md
Re0.1ang90/Production/md-cfg-desc_out_t00000000-0882623909.txt
Re0.1ang90/Production/md-cfg_out_p********-0882623909.asc
Re0.1ang90/Production/md-summary_out_t00030000-0882623909.txt
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang90/Production/od_out_t00030000-0882623909.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang90/Production/velx_out_t00030000-0882623909.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang90/Production/vely_out_t00030000-0882623909.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang90/Production/velz_out_t00030000-0882623909.h5"
file has vanished: "/home/ncteh317/Desktop/lb3d/drag_test/test/bouzidi/Oblate_2.5/Re0.1ang0/Production/vely_out_t00030000-0882612060.h5"
Basically the lines with file has vanished: are the large files that I wanted to back up. I guess they are indexed at the beginning of the rsync command. But during the copy process somehow the files are deleted. I searched similar topics in the forums. But, the issues mentioned are file being deleted during the copy process.
The same copy process I tried with FileZilla, GUI based file transfer software, after new simulations and the issue still remains - the large files disappear during the copy process in the supercomputer itself.
In my case, the simulations are completed and there is no operation/modification done on the files during the copy process. This being the case, can someone throw some light on what is going wrong?
