Title: Port of the week: rclone
Author: Solène
Date: 28 October 2020
Tags: portoftheweek
Description:
New **Port of the Week** after 3 years! I never thought it was so long
since last blog post about slrn.
This post is about the awesome **rclone** program, written in Go and
available on most popular platforms (including OpenBSD!). I will
explain how to configure it from the interactive command, from file
and what you can do with rclone.
**rclone** can be see as a rsync on steroids which supports lot of
Cloud backend and also support creating an encrypted data repository
over any backend (local file, ftp, sftp, webdav, Dropbox, AWS S3,
etc...).
It's **not** a automatic synchronization tool or a **backup**
software. It can copy files from A to B, synchronize two places
(can be harmful if you don't pay attention).
Let's see how to use it with an ssh server on which we will
create an encrypted repository to store important data.
[Official documentation](https://rclone.org/)
## Installation
Most of the time, run your package manager to install `rclone`.
It's a single binary.
## Interactive configuration
*You can skip this LONG section if you want to learn what rclone
can do and how to configure it in a 10 lines files.*
There is a parameter to have a question / answer interface to
configure your repository, using `rclone config`.
I'll make a full walkthrough to enable an encrypted repository
because I struggled to understand the logic behind rclone when I
started using it.
Let's start. I'll create an encrypted destination on my local NAS
which doesn't have full disk encryption, so anyone who access the
system won't be able to read my data. First, this will require to
set up an sftp repository and then an encrypted repository using the
previous one as a backend.
Let's create a new config named `home_nas`.
$ rclone config
2020/10/27 21:30:48 NOTICE: Config file
"/home/solene/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> home_nas
We want the storage type 29, "SSH/SFTP" (I removed all 50+ others
storages for readability).
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
29 / SSH/SFTP Connection
\ "sftp"
[...]
Storage> 29
My host is 192.168.1.200
** See help for sftp backend at: https://rclone.org/sftp/ **
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Connect to example.com
\ "example.com"
host> 192.168.1.200
I will connect with the username `solene`.
SSH username, leave blank for current username, solene
Enter a string value. Press Enter for the default ("").
user> solene
Standard port 22, which is the default
SSH port, leave blank to use default (22)
Enter a string value. Press Enter for the default ("").
port>
I answer **n** because I want rclone to use ssh agent, this could
be the ssh password to the remote user, but I highly discourage
everyone from using password authentication on SSH!
SSH password, leave blank to use ssh-agent.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> n
Leave this except if you want to provide a private key.
Raw PEM-encoded private key, If specified, will override key_file
parameter.
Enter a string value. Press Enter for the default ("").
key_pem>
Leave this except if you want to provide a PEM-encoded private key.
Path to PEM-encoded private key file, leave blank or set
key-use-agent to use ssh-agent.
variables such as `${RCLONE_CONFIG_DIR}`.
key_file>
Leave this except if you need to use a password to unlock your
private key. I use ssh agent so I don't need it.
The passphrase to decrypt the PEM-encoded private key file.
Encrypted keys
in the new OpenSSH format can't be used.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> n
If your user agent manage multiples keys, you should enter the
correct value here, I only have one key so I leave it empty.
When set forces the usage of the ssh-agent.
key-file is read and only the associated key is
requested from the ssh-agent. This allows to avoid `Too many
authentication failures for *username*` errors
when the ssh-agent contains many keys.
Enter a boolean value (true or false). Press Enter for the default
("false").
key_use_agent>
This is a question about crypto, accept the default except if you
have to connect to old servers.
Enable the use of insecure ciphers and key exchange methods.
exchange methods:
- aes192-cbc
- aes256-cbc
- 3des-cbc
- diffie-hellman-group-exchange-sha256
- diffie-hellman-group-exchange-sha1
recovered by an attacker.
Enter a boolean value (true or false). Press Enter for the default
("false").
Choose a number from below, or type in your own value
1 / Use default Cipher list.
\ "false"
2 / Enables the use of the aes128-cbc cipher and
diffie-hellman-group-exchange-sha256,
diffie-hellman-group-exchange-sha1 key exchange.
\ "true"
use_insecure_cipher>
We want to keep hashcheck feature so just skip the answer to keep
the default value.
Disable the execution of SSH commands to determine if remote file
hashing is available.
Leave blank or set to false to enable hashing (recommended), set to
true to disable hashing.
Enter a boolean value (true or false). Press Enter for the default
("false").
disable_hashcheck>
We are at the end of the configuration, we are proposed to change
more parameters but we don't need to.
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Now we can see the output of the configuration file of rclone in
regards to my `home_nas` destination. I agree with the configuration
to continue.
Remote config
--------------------
[home_nas]
type = sftp
host = 192.168.1.200
user = solene
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Here is a summary of the configuration, we have only one remote
here.
Current remotes:
==== ====
home_nas sftp
In the menu, I will choose to add another remote. Let's name it
`home_nas_encrypted`
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> home_nas_encrypted
We will choose the special storage `crypt` which work on an existing
backend.
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
10 / Encrypt/Decrypt a remote
\ "crypt"
Storage> 10
To this question, we will define we want the data stored to
`home_nas_encrypted` being saved in `home_nas` remote in the
`encrypted_repo` directory.
** See help for crypt backend at: https://rclone.org/crypt/ **
Normally should contain a ':' and a path, eg
"myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> home_nas:encrypted_repo
Depending on the level of obfuscation your choice may vary. The
simple filename obfuscation is fine for me.
How to encrypt the filenames.
Enter a string value. Press Enter for the default ("standard").
Choose a number from below, or type in your own value
1 / Encrypt the filenames see the docs for the details.
\ "standard"
2 / Very simple filename obfuscation.
\ "obfuscate"
3 / Don't encrypt the file names. Adds a ".bin" extension only.
\ "off"
filename_encryption> 2
As for the directory names obfuscation, I recommend to enable it,
otherwise that leave the whole directory tree readable!
Option to either encrypt directory names or leave them intact.
nothing.
Enter a boolean value (true or false). Press Enter for the default
("true").
Choose a number from below, or type in your own value
1 / Encrypt directory names.
\ "true"
2 / Don't encrypt directory names, leave them intact.
\ "false"
directory_name_encryption> 1
Type the password that will be used to encrypt the data.
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:
You can add a salt to the passphrase, I choose not too.
Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n>
No need to change advanced parameters.
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Here is a summary of the configuration of this remote backend.
I'm fine with it.
Remote config
--------------------
[home_nas_encrypted]
type = crypt
remote = home_nas:encrypted_repo
directory_name_encryption = true
password = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
We see we have now two remote backends, one with the crypt type.
Current remotes:
==== ====
home_nas sftp
home_nas_encrypted crypt
Quit rclone, the configuration is done.
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
## Configuration file
The previous configuration process only produced this short
configuration file, so you may copy/paste from it and adapt to add
more backends if you want, instead of doing the tedious `config`
process.
Here is my file `~/.config/rclone/rclone.conf` on my desktop.
[home_nas]
type = sftp
host = 192.168.1.200
user = solene
type = crypt
remote = home_nas:encrypted_repo
directory_name_encryption = true
password = GDS9B1B1LrBa3ltQrSbLf1Vq5C6VbaA1AJVlSZ8
## First usage
Now we defined our configuration, we need to create the remote
directory that will be used as a backend, this is important to avoid
errors when using rclone, this is a simple step required only once.
$ rclone mkdir home_nas_encrypted:
On the remote server, I can see a `/home/solene/encryted_repo`
directory. It's now ready to use!
## A few commands
**rclone** has a LOT of commands available, I will present a few
of them.
### Copying files to/from backend
Let's say I want to copy files to the encrypted repository. There
is a `copy` command.
$ rclone copy /home/solene/log/templates
home_nas_encrypted:blog_template
There are no output by default when the program runs fine. You can
use `-v` flag to have some verbose output (I prefer it).
### List files on a remote backend
Now, we want to see if the files were copied correctly, we will use
the `ls` command.
$ rclone ls home_nas_encrypted:
299 blog_template/article.tpl
700 blog_template/gopher_head.tpl
2505 blog_template/layout.tpl
295 blog_template/more.tpl
236 blog_template/navigation.tpl
57 blog_template/one-tag.tpl
34 blog_template/page.tpl
189 blog_template/rss-item.tpl
326 blog_template/rss.tpl
We can also use `ncdu` to mimic the **ncdu** program displaying a
curses interfaces to visualize disk usage in a nice browsing tree.
$ rclone ncdu home_nas_encrypted
-- home_nas_encrypted: ------------------
6.379k [##########] /blog_template
### The sync command
Files and directories can also be copied with the `sync` command,
but this must be used with care because it makes sure the destination
matches exactly the origin when you use it. It's the equivalent of
`rsync -a --delete origin/ destination/`, so any extra files will
be removed! Note that you can use `--dry-run` to see what will
happen.
### Filters
When you copy files using the various available method, instead of
using a path, you can provide a filter file or a list of paths to
transfers. This can be very efficient when you want to recover
specifics data.
The documentation about
[filtering is available here](https://rclone.org/filtering/)
### Parameters
**rclone** supports a lot of parameters, like to limit upload
bandwidth, copy multiples files at once, enable an interactive mode
in case of file deletion/overwriting.
### Mount
On Linux, FreeBSD and MacOS, **rclone** can use a FUSE filesystem
to mount the remote repository on the filesystem, making its uses
totally transparent.
This is extremely useful, avoiding the tediousness of the get/put
paradigm of **rclone**.
**This can even be used to make an encrypted repository on the local
filesystem! :)**
### Create a webdav/sftp/ftp server
**rclone** has the capability of act as a server and expose a
configured remote backend on various network protocol like webdav,
sftp, ftp, s3 (minio) !
The
[serv document is available
here](https://rclone.org/commands/rclone_serve/)
Example running a simple webdav server with hardcoded login/password:
$ rclone serv webdav --user solene --password ANicePassword
home_nas_encrypted: