How to Copy Large Directory to External Hard Drive Linux

Question

I've formatted a 3TB external hard drive as ext4 using fdisk (actually gnu fdisk), mounted it at /media/external and am trying to copy my entire home directory to it.

My first attempt I thought was successful, but after a while when the process stopped, I couldn't cd into the copied home directory. cd seemed to fail:

sudo: cd: command not found

I then thought there might be some limit to cp and then tried again using tar.

cd /from-stuff/
tar cf – . | (cd /to-stuff; tar xvf -)

I measured my home directory size to be ~100G. When in my home directory, I ran:

du -ch | grep total

I've mounted it as:

sudo mount -t ext4 /dev/sdd1 /media/external

Note that I could not mount /dev/sdd which shows up on sudo fdisk -l while /dev/sdd1 does not.

When formatting I did the most default setup, 1 partition, default everything else. I confirmed that formatting correctly by not having any 'invalid partition table errors.' I'm new to this so please bear with. I may have accidentally chosen GUID? but do not know how to check that or re-format away from that to whatever it may need to be.

Is there anything glaring that I did incorrectly? Can someone maybe offer an fdisk sequence to really make sure everything is 1 basic partition formatted as ext4?

Here's my documentation on what I did to format. Please clarify/augment if you can. There may have been a failed gparted attempt before this:

sudo fdisk /dev/sdb # this will pull up an interactive menu
n       # create a new partition
p       # create a primary partition
[enter] # use default value of 1 for the partition number
[enter] # use default value of 256 for first sector
[enter] # use default value for last sector
w       # write these settings and leave the interactive menu

# make the file system
sudo mkfs.ext4 /dev/sdb1

Answer 1

• about the sudo: cd: command not found

  That one is normal. cd is a shell builtin. This doesn't mean the destination directory doesn't exist. Try "su - root" if you need to be root to go to that dir, and then cd to the destination.

• about the fact /dev/sdd can't be mounted

  /dev/sdd is an entire sd disk (including its master partitino table, everything, even the unpartitionned space). It is not a partition. /dev/sdd1 is its first partition. And beware: you show you want to mount /dev/sdd1 , but you show a fdisk on /dev/sdb1 (another disk! /dev/sdb is different from /dev/sdd)

• if that new drive is to make backups : you should, instead of using tar | ( cd dest ; tar), which copies everything each time (and won't delete destination files/dirs if they get deleted on the source), you shuold look into the command rsync. Especially locally (but also when copying to a remote location, as rsync will only copy the changed parts of each changed files, not the entirety of the file). But beware its usage: rsync can be tricky at first. Don't use the "--delete" option before reading a lot about it, it could wipe out the source or destination files if they have no match on the other side... see the different ways to specify source/destinatino directories, sometimes you'll add the trailing "/", sometimes not, depending on how you want rsync to act.

Answer 2

I'm guessing your problem is that fdisk doesn't do large disks. It has been superseded by parted.

It's similar to fdisk, just updated for newer technology. Assuming the disk is /dev/sdb (check with sudo parted -l):

1. sudo parted /dev/sdb
2. mklabel (type gpt)
3. mkpart (default anwswers make one max size partition)
4. q to exit parted
5. sudo mkfs.ext4 /dev/sdb1 to make the filesystem.

After that it should just be a matter of sudo mount /dev/sdb1 /media/external. It would be a good idea to make sure you own the directory first with sudo chown tarabyte.tarabyte /media/external

I usually use rsync for large copies, even local. It has better output and resumes easily if there are interruptions.

rsync -avh --progress /from-stuff /to-stuff

See man rsync for more info.