Rclone Quick Reference

By Dag, on January 25th, 2017

Some quick reference notes i made while checking out Rclone for the first time on GNU/Linux. This goes a little beyond
Rclone, and expands to UnionFS as well. Rclone is an excellent program
for cloud storage management - with copy, sync, mount and encryption all in the same package. UnionFS enhances this even more with seamless integration between local and cloud storage, by the help
of just a little extra bash scripting for automagic. I'm also including a few notes on how to use it on Windows.

Download and unzip to anywhere.
Add its path to system PATH environment variable for ease of use.

Configuring Rclone

rclone config
^ This may lead to the need to visit localhost:n/auth
(Amazon) which can be a problem from CLI on a headless
setup. I tried using lynx and w3m but none of them worked.
So I SSH tunneled from my desktop at home to the server,
which worked. This gave a token, that I could paste into
the configuration process. Done deal.

Encryption
Rclone uses NaCl secret-key encryption. AES-256 for the file names, and
XSalsa20-Poly1305 for content. This is reportedly very fast and a
very solid choice of encryption after doing some research, so I'm going for this
instead of EncFS (which has fingerprint/watermark issues anyway). I love
that they have integrated good encryption into Rclone, less programs to manage.

Rclone Crypt works like this:
- You already have an existing remote:path
- You create an encrypted remote against remote:path.

The encrypted remote will list files from remote:path as unencrypted.
While if you look directly in remote:path, the files are actually
encrypted. As long as it was copied through the encrypted remote.

Example Encryption Usage:

$ rclone copy somedata encryptedremote:
^ Will copy somedata to remote:path and be encrypted.
'sync' instead makes source and dest identical, modifying destination only.
'move' for moving files from source to dest.
$ rclone copy somedata remote:path
^ Will copy somedata to remote:path NOT encrypted.
$ rclone ls remote:path
^ Will list files in encrypted form.
$ rclone ls encryptedremote:
^ Will list files in decrypted form.
^ NOTE: There are several related list commands
* `ls` to list size and path of objects only
* `lsl` to list modification time, size and path of objects only
* `lsd` to list directories only
* `lsf` to list objects and directories in easy to parse format
* `lsjson` to list objects and directories in JSON format
`ls`,`lsl`,`lsd` are designed to be human readable.
`lsf` is designed to be human and machine readable.
`lsjson` is designed to be machine readable.
Interesting parameters:
--bwlimit BwTimetable Bandwidth limit in kBytes/s, use suffix b|k|M|G.
--drive-chunk-size SizeSuffix Upload chunk size. Must a power of 2 >= 256k. (default 8M)
--buffer-size int In memory buffer size when reading files. (default 16M)
--exclude stringArray Exclude files matching pattern (add /** for recursive dirs)
-v, --verbose count Print lots more stuff (repeat for more)
--log-file string Log everything to this file
-n, --dry-run Do a trial run with no permanent changes

Rclone Mount is an EXPERIMENTAL feature at the time of this writing
(2017-01-25) so I'll simply use it for reading, and upload data through
copy/sync. I hope they stick with it though, because copy/sync/crypt/mount
together in one single package is simply... Awesome.

UnionFS
Using UnionFS means is that we can decide that unionfolder will ReadWrite
to localfiles but ReadOnly from encryptedfolder, but reads from both as a
single (not showing duplicates). So you get a united/union folder that
transparently uses two other folders. Perfect for my need. This way I can
have automated downloads from Sonarr that's immediately visible for Plex,
with an hourly script that moves/syncs to cloud storage and cleans up local
file copies to save local space. Make sure to leave .unionfs folders alone,
these have metadata about files you delete from unionfolder, since it can't
delete directly from the ReadOnly location.

After all this it's off to installing Plex and Sonarr and just point them to the union folder. In theory. I'll probably tweak a thing or two
but having this reference note is a good backup for future setups in general for anything needing Rclone.

# rclone cryptcheck remote:path cryptedremote:path --one-way
^ If you supply the --one-way flag, it will only check that files in
source match the files in destination, not the other way around.
Meaning extra files in destination that are not in the source will
not trigger an error.

NOTE: If you create log files, you'll need a way to rotate them yourself.