Copy a structure (including the subdirectories) of a directory to another location and move a percentage of files

Question

I have a folder with all data inside with this structure:

Data
-> Group 1
-> Group 2
...
-> Group n

In each subdirectory have a number of files. Now I want to create another directory with the same structure and move some of these files to that new directory(about 20% of total files)

New Data
-> Group 1
-> Group 2
...
-> Group n

I think of using something to read the structure of the folder and file list, then pipe it to another command to create and move the files, but I haven't know the syntax yet. Thank you

Answer 1

You can achieve this using the terminal. I have included instructions for both bash and fish shell. If you don't know what kind of shell you have, it's probably bash.

1. First, navigate to the folder containing the files you want to move (the Data/ folder from the question).

   • Bash & Fish: cd /path/to/folder

2. Define where you want to store the moved files. This can either be a relative path, or an absolute path.

   • Bash: export NEW_DIR="../New Data"
   • Fish: set NEW_DIR "../New Data"

3. Define the fraction of files to move, if you want 20%, set it to 5 (which will be 1/5 = 0.2 = 20%):

   • Bash: export FRACTION=5
   • Fish: set FRACTION 5

4. Run the following one-liner. See below for a more readable version:

   • Bash:

     find . -type f | xargs -I _ dirname _ | sort | uniq -c | while read n d; do echo "=== $d ($n files) ==="; if [ $(($n / $FRACTION)) -gt 0 ]; then find "$d" -type f | sort -R | head -n $(($n / $FRACTION)) | while read file; do echo "$file  ->  $NEW_DIR/$d"; mkdir -p "$NEW_DIR/$d"; mv "$file" "$NEW_DIR/$d"; done; fi; echo; done
     

   • Fish:

     find . -type f | xargs -I _ dirname _ | sort | uniq -c | while read n d; echo "=== $d ($n files) ==="; if math "$n/$FRACTION > 0" > /dev/null; find "$d" -type f | sort -R | head -n (math "$n" / $FRACTION) | while read file; echo "$file  ->  $NEW_DIR/$d"; mkdir -p "$NEW_DIR/$d"; mv "$file" "$NEW_DIR/$d"; end; end; echo; end
     

5. The script prints every file it moves, so it's easy to see what it moved.

Readable bash script:

find . -type f | xargs -I _ dirname _ | sort | uniq -c | while read n d; do
  echo "=== $d ($n files) ===";
  if [ $(($n / $FRACTION)) -gt 0 ]; then
    find "$d" -type f | sort -R | head -n $(($n / $FRACTION)) | while read file; do
      echo "$file  ->  $NEW_DIR/$d";
      mkdir -p "$NEW_DIR/$d";
      mv "$file" "$NEW_DIR/$d";
    done;
  fi;
  echo;
done

Readable fish script:

find . -type f | xargs -I _ dirname _ | sort | uniq -c | while read n d;
  echo "=== $d ($n files) ===";
  if math "$n/$FRACTION > 0" > /dev/null;
    find "$d" -type f | sort -R | head -n (math "$n" / $FRACTION) | while read file;
      echo "$file  ->  $NEW_DIR/$d";
      mkdir -p "$NEW_DIR/$d";
      mv "$file" "$NEW_DIR/$d";
    end;
  end;
  echo;
end