Just back from my vacation I learned two cool new tools on the first day. Of course both are in EPEL and Fedora.

The Silver Search: ag $STRING is an alternative to grep -ir $STING . But a lot of the hell faster, nicer output and multithreaded. https://geoff.greer.fm/ag/

myrepos: From the developer of etckeeper, to pull, commit and doing other stuff with multiple repos at the same time. Simply got to each git repo run mr register and than to a mr pull to pull all git repos at the same time. https://myrepos.branchable.com/

As I’ve written in my previous post, I moved to a new Hetzner Box. This is how I setup my virtual Host and how I moved my server, including all virtual machines, to the new data center with a downtime of a second or less. This setup is only possible if hetzner allows you to move you network via datacenter.

TL;DR

With the newer kvm/qemu you can use virsh migrate --live --copy-storage-all

Don’t waste IP addresses

Because fixed IP addresses are expensive wasting two of the eight IPs to have a broadcast network is not an option.

Virtual Machine setup

We create virtual Interfaces vif0 to vif7 and configure the routing to which we connect the virtual machines on eth0. We let libvirt call a little script on startup of the virtual machines. To do this we add the configurations to the libvirt/quemu xml config of the VMs with virsh edit virt0:

Setup inside the virtual Machine

Now we configure the network and routing for the virtual machines in /etc/sysconfig/network-scripts/ It should be noted, that the IP address is arbitrary. We need it becouse we cannot add a default route directly to a device, in this case eth0.

Moving

Now we want to move the VM to our new server.

The old server, let’s call it Hetzner-Old, is a CentOS 7 with a bridged net/29 network and several virtual machines running on it (e.g. this Blog). The VMs are running on LVMs which themself are running on crypto devices. Each VM has its own LVM group.

The new server, Hetzner-New is also a CentOS 7 machine. The LVM devices have the same name and are of the same size or bigger. They have to have the same names. This is also a good opportunity to make an LVM device bigger without the need to shut down the according VM.

Now we add the repo for SIG Virtualisation on both machines.

yum install centos-release-qemu-ev

and update qemu/kvm etc

Next we open a tunnel between the two machines with

ssh -w 2:2 tun2-IP

This way we have encrytion and don’t have to worry encryting it with TLS on qemu.

On Hetzner old we have to route through the tunnel tun2

ip r a net/29 via tun2-IP

Next we do

echo 0 | tee /proc/sys/net/ip4/conf/*/rp_filter

That way, when one of the network interfaces of a VM (vif+) gets removed, when the VM is moved, the according route is removed. Otherwise it would try to go through the default GW of Hetzner-New, wich is not allowed by Hetzner yet. Now, when the VM starts on Hetzner-New it has no interface to this IP-Address and it’s routed through the tunnel, through the GW of Hetzner-Old.

On Hetzner-New we add a new routing table with:

echo 201 vm-out >> /etc/iproute2/rt_tables

The number 201 is only internally used, therefore arbitrary.

Next we set a rule for every package where fwmark 1 is set use table vm-out:

ip rul a fwmark 1 table vm-out

Now we set a new default GATEWAY

ip r a default via IP-Hetzner-Old table vm-out

and set a firewall rule (vif+ are the interfaces for the VMs and iptables uses + instead of the commonly used *

Finally we wait for Hetzner to swich the net/29 network over to Hetzner-New and throw away the previously set iptables rule. Since the new machine is in a different data center, Hetzner has to allow this moving. Because for this to work the net/29 network has to be in Hetzners core routers. With this command we wait for the first tcp package to arrive at Hetzner-New and flush our iptables rule.

tcpdump -i br0 net net/29 -s 1 && iptables -F PREROUTING -t mangle

Posted inFedora|TaggedFedora-Planet|Comments Off on Moving my Server with less than a second downtime

I recently rented a new Hetzner Box to replace my old. I moved from EX40 to EX41 and saving even some money every month :-).

Every thing went smooth, but the sensors did not work. I had to use nct6775, but the module from Centos7 said “No Device”. ElRepo to the rescue. They have a nct6775 kmod available, but this module is not compatible with Centos7.4 :-(.

First I had to created a fixed package. I changed only the release-number and the kversion to 693.2.2 instead of 327. The fixed version is available at: https://www.kuehnel.org/nct6775/. I also open a ticket with elrepo: https://elrepo.org/bugs/view.php?id=792. I don’t know how they will fix this. ElRepo uses kABI-tracking kmods and things like that should not happen. We will see.

With this fixed package installed I created the following configuration file (/etc/sensors.d/hetzner) with the help of sensors -u:

I’m setting up two FreeNAS Server for Backup and Archiving and I really like FreeNAS 11. Thank good I didn’t have time to update it to FreeNAS Coral. 🙂

But I’m using check_mk for monitoring and I would like to use it to monitor FreeNAS as well. There is a check_mk agent for FreeBSD so the only problem is to run it.

I created this script to run it as a Init/Shutdown Script (both pre-init and post-init) . It will create everything you need, only define the BASEDIR at the beginning and put the check_mk_agent for FreeBSD in this directory. Make sure this script (check_mk_setup) and check_mk_agent are executable.

You also need to make sure inetd is running. I enable tftpd for that. Maybe some other service are possible as well. But I only tested it with tftpd.

after more then 10 years of using VMWare Workstation (Starting with VMWare Workstation 5). I’m in the process of moving to KVM/libvirt, but I want to use qcow2 with trim support.

I’m using Fedora 25 with virt-manager to create my virtual machines. A lot of pages describe that very well like Chris Irwins. But I found another problem.

To support trim you need to make sure you have at least 2.1, but I want the latest version 2.7. This it the default, so normally you don’t need to change this with virsh edit DOMAIN:

<type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>

One parameter that you need to change manually (and that can not be done with virt-manager) tells qemu that the discard/trim should be forwarded to the underlying image.It looks like this:

<driver name='qemu' type='qcow2' discard='unmap'/>

All of this can be found on the Internet. A problem that I faced with RHEL7 and others is that virt-manager creates the disc-controller as a antique LSI/NCR controller and that RHEL7 does not support this. To fix this you have to this model to the scsi controller:

<controller type='scsi' index='0' model='virtio-scsi'>

With this RHEL7, Windows 10 and FreeBSD 11 Machines can be configured to run with trimming there images them self.

To migrate you Windows 10 from “libvirt” auf “libvirt-scsi” is quite easy. Add a new libvirt-scsi Disc to an existing Installation. Install the driver for libvirt-scsi, reboot. Now you system supports this driver. After the this reboot, shutdown the machines again and change the disc type of all discs to libvirt-scsi like above and reboot again. Your system should start up with it’s libvirt-scsi enabled disc.

To run the trim command to cleanup unused space, run this in an Admin PowerShell:

sorry for the longtime being absent from this blog. But marriage and a child takes time and the blog was the first to go. But I will restart writing blog posts today with a project I started a couple of days ago.

Selinux and the Citrix Reciever

I have to use the Citrix Reciever to access the Citrix farm in our company. This is the only way to access the company network remotely. But I don’t like to run it as unconfined_t on my Fedora 25. So I sat down and created a selinux modules to limit the access of this close source software on my system. In the process I found that it tries to read the mozilla profile and other stuff that I didn’t like and I therefore disabled this. The code is available on GitHub. Simply install the citrix reciever with the rpm from the Citrix website.

This is only a fast and dirty solution. If you want to clean in up, I look forward to it. If I have time I will clean in up, but maybe someone is faster then me (aka. has more sparetime) 😉

After a lot of hard work I finished my bachelor thesis end of August and gave my colloquium end of September. Because of all the other stuff going on in my life I just have now time to upload my thesis.

I’m interested in log file analysis for a long time, but in the last years a lot has happened in this area. Here the abstract:

This thesis gives an overview on the Open Source and Free Software tools available for a centralized and structured log file analysis. This includes the tools to convert unstructured logs into structured log and different possibilities to transport this log to a central analyzing and storage station. The different storage and analyzing tools will be introduced, as well as the different web front ends to be used by the system administrator. At the end different tool chains will be introduced, that are well tested in this field.

Because of the time delay, some infos are already dated, but only two things, as fas as I know:

1. Graylog2 has now a new version available in Beta.

2. logstash has release a new version 1.2.1

I will keep this updated over time here on the blog, so have a look from time to time. If you find any problem, please inform me here at the blog or via mail: “logfiles jens.kuehnel.org” (add the @ at the right place).