Improving TLS Security

Added auth SHA256 so MACs on the individual packets are done with SHA256 instead of SHA1.

Added tls-version-min 1.2 to drop SSL3 + TLS v1.0 support. This breaks older clients (2.3.2+), but those updated versions have been out for a while.

Restricted the tls-ciphers allowed to a subset of Mozilla’s modern cipher list + DHE for older clients. ECDSA support is included for when ECDSA keys can be used. I’m uncertain of the usefulness of the ECDHE ciphers, as both my client and server support it, but the RSA cipher that’s 3rd in the list is still used. Continuing to investigate this.

The last 2 changes are gated by the openvpn_use_modern_tls variable, which defaults to true.

New keys are 2048 bit by default, downgraded from 4096 bit. This is based on Mozilla’s SSL guidance, combined with the expectation of being able to use ECDSA keys in a later revision of this playbook.

As part of the move to 2048 bit keys, the 4096 bit DH parameters are no longer distributed. It was originally distributed since generating it took ~75 minutes, but the new 2048 bit parameters take considerably less time.

Adding Cert Validations

EKU

Previously only the client was verifying that the server cert had the correct usage, now the verification is bi-directional.OpenVPN, more about EKU checks: 1 & 2

Certificate content

Added the ability to verify the common name that is part of each certificate. This required changing the common names that each certificate is generated with, which means that the ability to wipe out the existing keys was added as well.

The server verifies client names by looking at the common name prefix using verify-x509-name ... name-prefix, while the client checks the exact name provided by the server.

Again, both these changes are gated by a variable (openvpn_verify_cn). Because this requires rather large client changes, it is off by default.

Wiping out & reinstalling

Added the ability to wipe out & reinstall OpenVPN. Currently it leaves firewall rules behind, but other than that everything is removed.

Use ansible-playbook -v openvpn.yml --extra-vars="openvpn_uninstall=true" --tags uninstall to just run the uninstall portion.

Connect over IPv6

Previously, you had to explicitly use udp6 or tcp6 to use IPv6. OpenVPN isn’t dual stacked if you use plain udp/tcp, which results in being unable to connect to the OpenVPN server if it has an AAAA record, on your device has a functional IPv6 connection, since the client will choose which stack to use if you just use plain udp/tcp.

I recently discovered the slurp module within Ansible when I was attempting to find new modules in Ansible 2.0. It is particularly interesting for me since I’ve been doing a bunch of stuff involving the contents of files on remote nodes for my OpenVPN playbook. So I decided to try using it in one of my latest playbooks and see how much better it is than doing command: cat <file>.

The suggested method was to install python using the raw command, and then run the setup module to make the facts available.

But I was going to reboot the node right after the install in any case, so I didn’t feel like running the full setup module, so this was a perfect place to try the slurp module.

Using it is simple – there’s only one parameter: src, the file you want to get the contents of.

Similarly, using the results is also simple, with one exception: The content of the file is base64 encoded, so it must be decoded before use. Thankfully, Ansible/Jinja2 provides the b64decode filter to easily get the contents into a usable form.

Functionally, it’s pretty much identical to using the old command: cat <file> , register, and when: xyz in cmd.stdout style to get & use the contents of files. All of those elements are still there, just renamed at most – register is still being used unmodified.

The fact that I’m using a dedicated module for it though makes my playbook look a lot more Ansible-ish, which is something I like. (And the fact I don’t need to have a changed_when entry is a strong plus for code cleanliness.)

The main thing I’m excluding is the build artifacts – because I’m building RPMs, the SRPMS are rather large (nginx-pagespeed SRPMs weigh in at 110+MB), so I exclude all files ending in .rpm.

Next, I’m excluding most of the stuff in the plugin folder. My reasoning behind this is that the plugins themselves are downloadable. However, Jenkins disables plugins/pins plugins to the currently installed version by creating empty files of the form <plugin>.jpi.disabled/<plugin>.jpi.pinned. I want these settings to carry over between versions. Unfortunately trying + **/plugins/*.jpi.pinned showed that everything else got removed from the backup. I’m assuming this is due to the use of an inclusive rule, so the default include got changed to default exclude.

In any case, I end up explicitly excluding things I don’t care about, which is probably good if something that I might need ever ends up in the plugins folder.

I also exclude workspace because everything can be recreated by building from specific git commits if need be. The job information is logged in jobs/, so I can easily find past commits even though the workspace itself no longer exists.

Finally, I also exclude the jenkins war folder. I believe that this is an unpacked version of the .war file that gets installed to /usr/lib/jenkins. It seems to get created when I start jenkins itself.

With just these 6 excludes, I’ve dropped the backup archive size down to <5 MB, which is a big win.

Normally I’d just take a live backup while Jenkins is running, but in this case where I’m moving servers, I completely shutdown Jenkins first, before taking a final backup with duply jenkins full.

Restoring

For the restore, I first installed Jenkins using a Jenkins playbook from Ansible Galaxy. It’s fairly barebones, but it works well – and I don’t need to spend time developing my own playbook. I also installed duply, and I manually installed an updated version of duplicity for CentOS 7 from koji to get the latest fixes.

Once I got duply set back up, I restored all the files to a new folder with duply jenkins restore /root/jenkins. I restored it to a separate folder because duply appears to remove the entire destination folder if it exists, and I wanted to merge the two folders.

After the restore was complete, I ran rsync -rtl --remove-source-files /tmp/jenkins/ /var/lib/jenkins to merge the restored data into the newly installed Jenkins instance.

At this point, everything should have worked, except I was unable to login. After spending some time fruitlessly searching Google, I ran chown -R jenkins:jenkins /var/lib/jenkins, as the rsync didn’t preserve the file owner when it created the new files. Luckily enough, that fixed the problem, and I could now login.

I then spent a few hours working all this into an Ansible playbook so future moves are much easier.

The motivation for this came from trying implement running a command that depended on whether or not a previous command succeeded.

In this case, I was trying to make the creation of duply profiles idempotent. Duply will exit with an error if you attempt to create a profile that already exists, and I didn’t want that to interrupt my playbook.

My first thought was “stat the folder & check if it exists”, which got my this: