Use scp to copy files of specific extension from directory

manpreet Best Answer 2 years ago

I'm writing a bash script that needs to fetch all *_out.csv from a directory, on a remote server. All these files are several directories deep inside of another directory. So for instance, say the directory is called ox_20190404/. I can find all my files by going:

find ox_20190404/assessment/LWR/validation -type f -name "*_out.csv"

This question answers part of my question, but since I don't want to copy the directory in it's entirety I need to figure out how to implement the above code. Suppose I start with this:

$ dir="/projects/ox/git"
$ server="myusername@server"
$ scp $server:$dir/$(ssh $server 'ls -t $dir | head -1') .

How would I grab the f="https://forum.tuteehub.com/tag/file">files I need from there?

The last part of my question wonders if there is a way to then take all the copied f="https://forum.tuteehub.com/tag/file">files and place them in the same f="https://forum.tuteehub.com/tag/file">file path and directory they were in on the remote server.

manpreet 2 years ago

I've adjusted some of your variable names a bit.

Surely there are better ways to do this than something dangerous like parsing the output of ls, but see whether this works for you:

$ pth="/projects/ox/git"
$ server="myusername@server"
$ dir="$(ssh $server "ls -t \"$pth\" | head -1")"
$ mkdir -p "$pth/$dir"
$ scp -p $server:"$pth/$dir"/'*_out.csv' "$pth/$dir"/

Once dir has been set to the newest remote directory, mkdir -p is used to ensure that the same directory name exists locally. Then scp the files into a local directory with the same path and name as the remote directory. I was looking for an rsync solution, but couldn't think of one.