Rsync commands

rsync.md

To sync files from remote to local

rsync -avz --stats --dry-run --relative -e "ssh -i path/to/somekey" "[email protected]:/remote/path/./data/*/datatype/" ~/local/path/data/datatype/

Replace the local and remote path as per your requirements. This is for getting data from the datatype folder for all folders within the data folder. Put path to your ssh key in place of path/to/somekey and replace [email protected] with your connection string for your remote machine.

This does the following-

1. --stats shows the progress and other statistics
2. --dry-run performs a check run but does not tranfer any data. Remove it to actually intiate the data transfer
3. --relative creates a relative path of the remote folder on the local path so we don't have very long path. The relative path is calculated from the remote path based on the ./ identifier as shown in the above path.
4. -e "ssh -i path/to/somekey" instructs rsync to authenticate using the ssh key provided. You can also add the key under your host and user in the ~/.ssh/config file.
5. The path can be very flexible, for example to get only a certain type of file (csv in this case) you can do /remote/path/./data/*/datatype/*.csv or for a particular file names /remote/path/./data/*/datatype/data-2019-05-*.csv will get all filenames with a prefix of data-2019-05- of type csv.
6. -v enables verbose logging for better debugging of any potential issues.