Title: Configuration deployment made easy with drist
Author: Solène
Date: 29 November 2018
Tags: unix drist automation
Description:
Hello, in this article I will present you my deployement tool **drist**
(if you
speak Russian, I am already aware of what you think). It reached a
feature
complete status today and now I can write about it.
As a system administrator, I started using *salt* a few years ago. And
honestly, I can not cope with it anymore. It is slow, it can get very
complicated for some tasks like correctly ordering commands and a
configuration file can become a nightmare when you start using
condition in it.
### History
I also tried alternatives like *ansible*, *puppet*, *Rex* etc... One
day, when
lurking in the ports tree, I found **sysutils/radmind** which got a lot
interest from me even if it is really poorly documented. It is a
project from
1995 if I remember correctly, but I liked the base idea. *Radmind*
works with
files, you create a known working set of files for your system, and you
can
propagate that whole set to other machines, or see differences between
the
reference and the current system. Sets could be negative, meaning that
the
listed files should not be present on the system, but it was also
possible to
add extra sets for specific hosts. The whole thing is really really
cumbersome,
this requires a lot of work, I found little documentation etc... so I
did not
used it but, that lead me to write my own deployment tool using ideas
from
*radmind* (working with files) and from *Rex* (using a script for doing
changes).
### Concept
**drist** aims at being simple to understand and pluggable with
standard tools.
There is no special syntax to learn, no daemon to run, no agent, and it
relies
on base tools like awk, sed, ssh and rsync.
**drist** is cross platform as it has a few requirements but it is not
well
suited for deploying on too much differents operating systems.
When executed, **drist** will execute six steps in a specific order,
you can
use only steps you need.
Shamelessly copied from the man page, explanations after:
1. If folder **files** exists, its content is copied to server
rsync(1).
2. If folder **files-HOSTNAME** exists, its content is copied to server
using rsync(1).
3. If folder **absent** exists, filenames in it are deleted on server.
4. If folder **absent-HOSTNAME** exists, filenames in it are deleted on
server.
5. If file **script** exists, it is copied to server and executed
there.
6. If file **script-HOSTNAME** exists, it is copied to server and
executed there.
In the previous list, all the existences checks are done from the
current
working directory where drist is started. The text **HOSTNAME** is
replaced by
the output of `uname -n` of the remote server, and files are copied
starting from
the root directory.
drist does not do anything more. In a more litteral manner, it copies
files to
the remote server, using a local filesystem tree (folder **files**). It
will
delete on the remote server all files present in the local filesystem
tree
(folder **absent**), and it will run on the remote server a script
named
**script**.
Each of theses can be customized per-host by adding a "-HOSTNAME"
suffix to the
folder or file name, because experience taught me that some hosts does
require
specific configuration.
If a folder or a file does not exist, **drist** will skip it. So it is
possible
to only copy files, or only execute a script, or delete files and
execute a
script after.
### Drist usage
The usage is pretty simple. **drist** has 3 flags which are optionals.
- -n flag will show what happens (simuation mode)
- -s flag tells drist to use sudo on the remote host
- -e flag with a parameter will tell drist to use a specific path for
the sudo
program
The remote server address (ssh format like user@host) is mandatory.
$ drist my_user@my_remote_host
drist will look at files and folders in the current directory when
executed,
this allow to organize as you want using your filesystem and a revision
control
system.
### Simple examples
Here are two examples to illustrate its usage. The examples are easy,
for
learning purpose.
#### Deploying ssh keys
I want to easily copy my users ssh keys to a remote server.
$ mkdir drist_deploy_ssh_keys
$ cd drist_deploy_ssh_keys
$ mkdir -p files/home/my_user1/.ssh
$ mkdir -p files/home/my_user2/.ssh
$ cp -fr /path/to/key1/id_rsa files/home/my_user1/.ssh/
$ cp -fr /path/to/key2/id_rsa files/home/my_user2/.ssh/
$ drist user@remote-host
Copying files from folder "files":
/home/my_user1/.ssh/id_rsa
/home/my_user2/.ssh/id_rsa
#### Deploying authorized_keys file
We can easily create the authorized_key file by using cat.
$ mkdir drist_deploy_ssh_authorized
$ cd drist_deploy_ssh_authorized
$ mkdir -p files/home/user/.ssh/
$ cat /path/to/user/keys/*.pub >
files/home/user/.ssh/authorized_keys
$ drist user@remote-host
Copying files from folder "files":
/home/user/.ssh/authorized_keys
This can be automated using a makefile running the cat command and then
running
drist.
all:
cat /path/to/keys/*.pub >
files/home/user.ssh/authorized_keys
drist user@remote-host
#### Installing nginx on FreeBSD
This module (aka a folder which contain material for drist) will
install nginx
on FreeBSD and start it.
$ mkdir deploy_nginx
$ cd deploy_nginx
$ cat >script <<EOF
#!/bin/sh
test -f /usr/local/bin/nginx
if [ $? -ne 0 ]; then
pkg install -y nginx
fi
sysrc nginx_enable=yes
service nginx restart
EOF
$ drist user@remote-host
Executing file "script":
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):
nginx: 1.14.1,2
421 KiB to be downloaded.
[1/1] Fetching nginx-1.14.1,2.txz: 100% 421 KiB 430.7kB/s
00:01
Checking integrity... done (0 conflicting)
[1/1] Installing nginx-1.14.1,2...
===> Creating groups.
Using existing group 'www'.
===> Creating users
Using existing user 'www'.
[1/1] Extracting nginx-1.14.1,2: 100%
Message from nginx-1.14.1,2:
===================================================================
Recent version of the NGINX introduces dynamic modules
support. In
FreeBSD ports tree this feature was enabled by default with
the DSO
knob. Several vendor's and third-party modules have been
converted
to dynamic modules. Unset the DSO knob builds an NGINX
without
dynamic modules support.
directive in the main context, specifying the path to the
shared
object file for the module, enclosed in quotation marks.
When you
reload the configuration or restart NGINX, the module is
loaded in.
It is possible to specify a path relative to the source
directory,
or a full path, please see
https://www.nginx.com/blog/dynamic-modules-nginx-1-9-11/
and
http://nginx.org/en/docs/ngx_core_module.html#load_module
for
details.
===================================================================
nginx_enable: -> yes
Performing sanity check on nginx configuration:
nginx: the configuration file
/usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf
test is successful
nginx not running? (check /var/run/nginx.pid).
Performing sanity check on nginx configuration:
nginx: the configuration file
/usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf
test is successful
Starting nginx.
### More complex example
Now I will show more complexes examples, with host specific steps. I
will not
display the output because the previous output were sufficient enough
to give a
rough idea of what drist does.
#### Removing someone ssh access
We will reuse an existing module here, a user should not be able to
login
anymore on its account on the servers using the ssh key.
$ cd ssh
$ mkdir -p absent/home/user/.ssh/
$ touch absent/home/user/.ssh/authorized_keys
$ drist user@server
#### Installing php on FreeBSD
The following module will install php and remove the opcache.ini file,
and will
install php72-pdo_pgsql if it is run on server
*production.domain.private*.
$ mkdir deploy_php && cd deploy_php
$ mkdir -p files/usr/local/etc
$ cp /some/correct/config.ini files/usr/local/etc/php.ini
$ cat > script <<EOF
#!/bin/sh
test -f /usr/local/etc/php-fpm.conf || pkg install -f
php-extensions
sysrc php_fpm_enable=yes
service php-fpm restart
test -f /usr/local/etc/php/opcache.ini || rm
/usr/local/etc/php/opcache.ini
EOF
$ cat > script-production.domain.private <<EOF
#!/bin/sh
test -f /usr/local/etc/php/pdo_pgsql.ini || pkg install -f
php72-pdo_pgsql
service php-fpm restart
EOF
#### The monitoring machine
This one is unique and I would like to avoid applying its configuration
against
another server (that happened to me once with salt and it was really
really
bad). So I will just do all the job using the hostname specific cases.
$ mkdir my_unique_machine && cd my_unique_machine
$ mkdir -p
files-unique-machine.private/usr/local/etc/{smokeping,munin}
$ cp /good/config
files-unique-machine.private/usr/local/etc/smokeping/config
$ cp /correct/conf
files-unique-machine.private/usr/local/etc/munin/munin.conf
$ cat > script-unique-machine.private <<EOF
#!/bin/sh
pkg install -y smokeping munin-master munin-node
munin-configure --shell --suggest | sh
sysrc munin_node_enable=yes
sysrc smokeping_enable=yes
service munin-node restart
service smokeping restart
EOF
$ drist user@incorrect-host
$ drist user@unique-machine.private
Copying files from folder "files-unique-machine.private":
/usr/local/etc/smokeping/config
/usr/local/etc/munin/munin.conf
Executing file "script-unique-machine.private":
[...]
Nothing happened on the wrong system.
#### Be creative
Everything can be automated easily. I have some makefile in a lot of my
drist
modules, because I just need to type "make" to run it correctly.
Sometimes it
requires concatenating files before being run, sometimes I do not want
to make
mistake or having to remember on which module apply on which server (if
it's
specific), so the makefile does the job for me.
One of my drist module will look at all my SSL certificates from
another
module, and make a reed-alert configuration file using awk and
deploying it on
the monitoring server. All I do is typing "make" and enjoy my free
time.
### How to get it and install it
- Drist can be downloaded [at this
address](ftp://ftp.bitreich.org/releases/drist/drist-v1.02.tgz).
- Sources can be cloned using `git clone git://bitreich.org/drist`
In the sources folder, type "make install" as root, that will copy
drist binary
to /usr/bin/drist and its man page to /usr/share/man/man1/drist.1
For copying files, drist requires rsync on both local and remote hosts.
For running the script file, a sh compatible shell is required (csh is
not working).