Difference between revisions of "Rsync"

From wikieduonline
Jump to navigation Jump to search
(Rsync is a single thread C application that includes a progress indicator.)
 
(47 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[wikipedia:rsync|rsync]]<ref>http://man7.org/linux/man-pages/man1/rsync.1.html</ref> command is used to copy and synchronization files and folders. Rsync is a single thread C application that includes a progress indicator.
+
{{lowercase}}
 +
<code>[[wikipedia:rsync|rsync]]</code> ([[1996]]) command is used to copy and synchronization files and folders. Rsync is a [[single thread]] C application that includes a [[progress]] indicator.
 +
 
 +
* Man: http://man7.org/linux/man-pages/man1/rsync.1.html
  
 
== Configuration files ==
 
== Configuration files ==
Line 5: Line 8:
  
 
== Options ==
 
== Options ==
:<code>-W, --whole-file</code>
+
:<code>-W, --whole-file</code> copy files whole (without [[delta-xfer]] algorithm)
:<code>-z, --compress</code> compresses the file data as it is sent to the destination machine using [[zlib]] compression library
+
:<code>-z, --[[compress]]</code> compresses the file data as it is sent to the destination machine using [[zlib]] compression library
 +
:<code>--temp-dir=DIR, -T</code> create temporary files in directory DIR, by default temp file is created as a [[hidden file]] in the target directory and named <code>.<FILE_NAME>.<RANDOM_STRING></code>
 +
:<code>--partial</code>
 +
:<code>[[--delete]]</code> delete files in the target not present in source
 +
      --delete                delete extraneous files from dest dirs
 +
      --delete-before          receiver deletes before xfer, not during
 +
      --delete-during          receiver deletes during the transfer
 +
      --delete-delay          find deletions during, delete after
 +
      --delete-after          receiver deletes after transfer, not during
 +
      --delete-excluded        also delete excluded files from dest dirs
 +
      --delete-missing-args    delete missing source args from destination
  
 
== Basic commands ==
 
== Basic commands ==
 +
* <code>rsync -Pr ORIGIN_FOLDER_OR_FILE DESTINATION_FOLDER</code>
 +
: <code>-P, --partial [[--progress]] (--partial</code> keep partially transferred files)
 +
: <code>-r --recursive</code>
  
 
* <code>rsync -P ORIGIN DESTINATION</code>  
 
* <code>rsync -P ORIGIN DESTINATION</code>  
: <code>-P, --partial --progress</code>. --partial keep partially transferred files
+
: <code>-P, --partial --progress (--partial</code> keep partially transferred files)
 
* Copy files from remote server to local machine: <code>rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage</code><ref>https://stackoverflow.com/questions/9090817/copying-files-using-rsync-from-remote-server-to-local-machine</ref> (explainshell<ref>https://explainshell.com/explain?cmd=rsync+-chavzP+--stats+user%40remote.host%3A%2Fpath%2Fto%2Fcopy+%2Fpath%2Fto%2Flocal%2Fstorage</ref>)
 
* Copy files from remote server to local machine: <code>rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage</code><ref>https://stackoverflow.com/questions/9090817/copying-files-using-rsync-from-remote-server-to-local-machine</ref> (explainshell<ref>https://explainshell.com/explain?cmd=rsync+-chavzP+--stats+user%40remote.host%3A%2Fpath%2Fto%2Fcopy+%2Fpath%2Fto%2Flocal%2Fstorage</ref>)
 
: <code>-c, --checksum</code> option can slow transfers specially for large files   
 
: <code>-c, --checksum</code> option can slow transfers specially for large files   
Line 21: Line 37:
 
: <code>--stats</code>, print statistics
 
: <code>--stats</code>, print statistics
  
:-a, --archive. Equivalent to <code>-rlptgoD</code>
+
:<code>-a, --archive</code>. Equivalent to <code>-rlptgoD</code>
::-r --recursive            recurse into directoriees
+
::<code>-r --recursive</code>             recurse into directoriees
::-l --links                copy symlinks as symlinks
+
::<code>-l --links</code>                 copy symlinks as symlinks
::-p --perms                preserve permissions
+
::<code>-p --perms</code>                 preserve permissions
::-t --times                preserve modification times
+
::<code>-t --times</code>                 preserve modification times
::-g --group                preserve group
+
::<code>-g --group</code>                 preserve group
::-o --owner                preserve owner (super-user only)
+
::<code>-o --owner</code>                 preserve owner (super-user only)
::-D same as --devices --specials
+
::<code>-D</code> same as <code>--devices --specials</code>
  
 +
<pre>rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage
 +
receiving incremental file list
 +
.../...
 +
</pre>
  
 
+
Remove <code>-c</code> option to avoid calculating checksums which can slow transmissions.
<pre>rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage
+
<pre>rsync -havzP --stats [email protected]:/path/to/copy /path/to/local/storage
 
receiving incremental file list
 
receiving incremental file list
 
.../...
 
.../...
 
</pre>
 
</pre>
  
 +
.../...
 +
sending incremental file list
  
 +
 +
* <code>rsync -Pav --ignore-existing src dst</code>
 +
:Learn difference between: <code>--ignore-existing</code> and <code>--update</code>
 
* <code>rsync -av --ignore-existing src dst</code>
 
* <code>rsync -av --ignore-existing src dst</code>
  
 
== Compression ==
 
== Compression ==
 
rsync support [[compression]], but you can also compress your data with other external compression tools before using rsync. Such as [[bzip]], [[gzip]] or <code>[[xz]]</code>  
 
rsync support [[compression]], but you can also compress your data with other external compression tools before using rsync. Such as [[bzip]], [[gzip]] or <code>[[xz]]</code>  
 +
 +
:<code>-z, --[[compress]]</code> compresses the file data as it is sent to the destination machine using [[zlib]] compression library
  
  
Compressing using <code>[[xz]]</code><ref>https://linux.die.net/man/1/xz</ref>
+
Compressing file before rsync using <code>[[xz]]</code><ref>https://linux.die.net/man/1/xz</ref>
 
* <code>[[xz]] -v -T0 file_to_compress</code>  
 
* <code>[[xz]] -v -T0 file_to_compress</code>  
::-v provides progress bar
+
::<code>-v</code> provides progress bar
::-T0 use all available Threads, multi-threaded compression supported since 2014, version 5.2.0. <ref>https://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD</ref>
+
::<code>-T0</code> use all available Threads, multi-threaded compression supported since 2014, version 5.2.0. <ref>https://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD</ref>
  
 
== Activities ==
 
== Activities ==
 
# Create a remote copy of your files for backup proposes using <code>rsync</code>
 
# Create a remote copy of your files for backup proposes using <code>rsync</code>
 
# Read StackOverflow questions about rsync: https://stackoverflow.com/questions/tagged/rsync?tab=Votes
 
# Read StackOverflow questions about rsync: https://stackoverflow.com/questions/tagged/rsync?tab=Votes
# Understand the differences between [[linux/Basic commands/cp|cp]], [[scp]] and [[linux/Basic commands/rsync|rsync]]: https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync
+
# Understand the differences between <code>[[cp]]</code>, <code>[[scp]]</code> and <code>[[rsync]]</code>: https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync
 
# Learn Effect of Trailing Slash / in rsync: https://www.alibabacloud.com/blog/speeding-up-network-file-transfers-with-rsync_594337
 
# Learn Effect of Trailing Slash / in rsync: https://www.alibabacloud.com/blog/speeding-up-network-file-transfers-with-rsync_594337
 
# Learn use of <code>--inplace</code> (implies also <code>--partial</code>) for transferring large files and its important implications: http://man7.org/linux/man-pages/man1/rsync.1.html, also evaluate <code>-c, --checksum</code> impact on transfering large files
 
# Learn use of <code>--inplace</code> (implies also <code>--partial</code>) for transferring large files and its important implications: http://man7.org/linux/man-pages/man1/rsync.1.html, also evaluate <code>-c, --checksum</code> impact on transfering large files
 +
# Learn about <code>--sparse</code> option and conflicts with  <code>--inplace</code>
 +
Performance
 +
# Understand how single thread <code>rsync</code> design can affect your transfer speeds.<ref>https://stackoverflow.com/questions/24058544/speed-up-rsync-with-simultaneous-concurrent-file-transfers</ref>
 
# Learn about <code>-W, --whole-file</code> copy files whole (without delta-xfer algorithm), implications and usage cases such as not making a diff copy.
 
# Learn about <code>-W, --whole-file</code> copy files whole (without delta-xfer algorithm), implications and usage cases such as not making a diff copy.
# Learn about <code>--sparse</code> option and conflicts with  <code>--inplace</code>
+
 
 +
== Related terms ==
 +
* <code>[[aws s3 rsync]]</code>
 +
* <code>[[gzip]] --[[rsyncable]]</code>
 +
* <code>[[duplicity]]</code>
 +
* [[fork]]
 +
* <code>rsyncy</code> status/[[progress]] bar for [[rsync]]
 +
* [[Andrew Tridgell]]
 +
* [[AWS S3 replication]]
  
 
== See also ==
 
== See also ==
 +
* {{rsync}}
 
* {{cp}}
 
* {{cp}}
 
* <code>[[dd]]</code>, <code>[[mv]]</code>
 
* <code>[[dd]]</code>, <code>[[mv]]</code>

Latest revision as of 14:56, 31 August 2023

rsync (1996) command is used to copy and synchronization files and folders. Rsync is a single thread C application that includes a progress indicator.

Configuration files[edit]

  • /etc/rsyncd.conf which includes don't compress options for some file extensions

Options[edit]

-W, --whole-file copy files whole (without delta-xfer algorithm)
-z, --compress compresses the file data as it is sent to the destination machine using zlib compression library
--temp-dir=DIR, -T create temporary files in directory DIR, by default temp file is created as a hidden file in the target directory and named .<FILE_NAME>.<RANDOM_STRING>
--partial
--delete delete files in the target not present in source
      --delete                 delete extraneous files from dest dirs
      --delete-before          receiver deletes before xfer, not during
      --delete-during          receiver deletes during the transfer
      --delete-delay           find deletions during, delete after
      --delete-after           receiver deletes after transfer, not during
      --delete-excluded        also delete excluded files from dest dirs
      --delete-missing-args    delete missing source args from destination

Basic commands[edit]

  • rsync -Pr ORIGIN_FOLDER_OR_FILE DESTINATION_FOLDER
-P, --partial --progress (--partial keep partially transferred files)
-r --recursive
  • rsync -P ORIGIN DESTINATION
-P, --partial --progress (--partial keep partially transferred files)
  • Copy files from remote server to local machine: rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage[1] (explainshell[2])
-c, --checksum option can slow transfers specially for large files
-h, --human-readable
-a, --archive. Equivalent to -rlptgoD
-v, --verbose
-z, --compress
-P, --partial --progress. --partial keep partially transferred files
--stats, print statistics
-a, --archive. Equivalent to -rlptgoD
-r --recursive recurse into directoriees
-l --links copy symlinks as symlinks
-p --perms preserve permissions
-t --times preserve modification times
-g --group preserve group
-o --owner preserve owner (super-user only)
-D same as --devices --specials
rsync -chavzP --stats [email protected]:/path/to/copy /path/to/local/storage
receiving incremental file list
.../...

Remove -c option to avoid calculating checksums which can slow transmissions.

rsync -havzP --stats [email protected]:/path/to/copy /path/to/local/storage
receiving incremental file list
.../...
.../...
sending incremental file list


  • rsync -Pav --ignore-existing src dst
Learn difference between: --ignore-existing and --update
  • rsync -av --ignore-existing src dst

Compression[edit]

rsync support compression, but you can also compress your data with other external compression tools before using rsync. Such as bzip, gzip or xz

-z, --compress compresses the file data as it is sent to the destination machine using zlib compression library


Compressing file before rsync using xz[3]

  • xz -v -T0 file_to_compress
-v provides progress bar
-T0 use all available Threads, multi-threaded compression supported since 2014, version 5.2.0. [4]

Activities[edit]

  1. Create a remote copy of your files for backup proposes using rsync
  2. Read StackOverflow questions about rsync: https://stackoverflow.com/questions/tagged/rsync?tab=Votes
  3. Understand the differences between cp, scp and rsync: https://stackoverflow.com/questions/20244585/how-does-scp-differ-from-rsync
  4. Learn Effect of Trailing Slash / in rsync: https://www.alibabacloud.com/blog/speeding-up-network-file-transfers-with-rsync_594337
  5. Learn use of --inplace (implies also --partial) for transferring large files and its important implications: http://man7.org/linux/man-pages/man1/rsync.1.html, also evaluate -c, --checksum impact on transfering large files
  6. Learn about --sparse option and conflicts with --inplace

Performance

  1. Understand how single thread rsync design can affect your transfer speeds.[5]
  2. Learn about -W, --whole-file copy files whole (without delta-xfer algorithm), implications and usage cases such as not making a diff copy.

Related terms[edit]

See also[edit]

Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy.

Source: https://en.wikiversity.org/wiki/Linux/Basic_commands/rsync

Advertising: