Rsync over SSH - (40MB/s over 1GB NICs)

Documentation.md

The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

This creates an archive that does the following:

rsync (Everyone seems to like -z, but it is much slower for me)

• a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
• H: preserves hard-links
• A: preserves ACLs
• X: preserves extended attributes
• x: don't cross file-system boundaries
• v: increase verbosity
• --numeric-ds: don't map uid/gid values by user/group name
• --delete: delete extraneous files from dest dirs (differential clean-up during sync)
• --progress: show progress during transfer

ssh

• T: turn off pseudo-tty to decrease cpu load on destination.
• c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.
• o Compression=no: Turn off SSH compression.
• x: turn off X forwarding if it is on by default.

Original

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>

Flip

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" [source_dir] [dest_host:/dest_dir]

SSHrsync.sh

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>

To decide which cipher is the best, I recommend using this script to benchmark for yourself: https://gist.github.com/joeharr4/c7599c52f9fad9e53f62e9c8ae690e6b

dunno, dunno

git, tons of settings tried. 100gb+ repos. Millions of files, etc of content. Just for bench purposes (not working dirs). But results pretty sad on internal network on Vultr.

strange - i'm only getting 27-30MB/s with rsync -aHAxv --numeric-ids --progress -e "ssh -T -c [email protected] -o Compression=no -x"
transferring a 1.6TB file between two Epyc servers with plenty of resources and 10GB networking. i get 500MB/s transferring video files from my desktop to my storage server. not sure what's up here.

update: it increased to 52 MB/s, which is... definitely not as fast as I would like, but it's fine.

Not recommended, I almost trashed my entire movies collection by doing this, good thing I canceled it.

🤡

I'm not sure if people are still interested in this, but if you don't care about encryption then tar + netcat is by far the quickest way to transfer directories:

destination:
nc -l -p 7777 | tar -xpf -

source:
tar -cf - sourceDir/ | nc [dest ip] 7777

throw in 'pv' to see xfer speed.

Oh rad! I’ll try this today! Appreciate that.

…

On Sun, Sep 11, 2022 at 8:49 AM ip-rw ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ I'm not sure if people are still interested in this, but if you don't care about encryption then tar + netcat is by far the quickest way to transfer directories: destination: nc -l -p 7777 | tar -xpf - source: tar -cf - sourceDir/ | nc [dest ip] 7777 throw in 'pv' to see xfer speed. — Reply to this email directly, view it on GitHub <https://gist.github.com/4393116#gistcomment-4297214>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7LWR3KKCZOMWXXKDNWFZTV5XPOLANCNFSM4HMITOZA> . You are receiving this because you commented.Message ID: ***@***.***>

Great discussion.

I found this to be the best option. "ssh -T -c [email protected] -o Compression=no -x" .Probably aes256 was faster than arcfour due to hardware optimizations or something. Also might play with/without rsync -z based on the quantity/size of the files to transfer. No compression was faster for an already compressed single big tar.gz file

https://gbe0.com/posts/linux/server/benchmark-ssh-ciphers/

That’s a great report , thanks!

…

On Tue, Sep 27, 2022 at 12:37 AM Akhil Jalagam ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ https://gbe0.com/posts/linux/server/benchmark-ssh-ciphers/ — Reply to this email directly, view it on GitHub <https://gist.github.com/4393116#gistcomment-4316545>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7LWR5ZYQVG4JQ4NPS57PDWAKBYXANCNFSM4HMITOZA> . You are receiving this because you commented.Message ID: ***@***.***>

To not let this stand as is, some facts: compression is off by default in ssh (and always has been in openssh), tty allocation is off when used in rsync and x forwarding does not affect bulk bandwidth in any way. Any difference in speed measured is not due to these options, but more likely because of a bad test setup, such as first making tests with cold disk cache and the with hot cache. The only change that can affect speed is the cipher (and not turning compression explicitly on in rsync or ssh).