contact

Author: vasil

Today, I meet with a challenge. I need to set up good and working Wifi network over the building. I need to use sixteen access points (AP). My previous configurations was simple deployment of this access points with laborious configuration of each AP. And there were many channels, and things, that I must configure.

So I create centralized Access Point management setup for office environment that is scalable to many Access Point. This can be done by setting up Controlled Access Point system Manager (CAPsMAN) on your router and connecting Controlled Access Points (CAPs) to it. I have two bands: 2,4GHz and 5GHz. Everything with one SSID. I use this howto:

https://wiki.mikrotik.com/wiki/Manual:Simple_CAPsMAN_setup

As CAPSMAN I used powerfull hardware: MikroTik CCR1009-7G-1C-1S+. As CAPs I will use HAP AC – dual band wifi AP.

I assume, that you have some skills with Mikrotiks and configuration. So I will use only terminal commands in this post with explanation.

So, lets begin.

Assume, that we have default VLAN 600, with no DHCP and now internet connection. Its dummy vlan, lead to nowhere. Than we have more 3 VLANs. One vlan is management (3), one si for guests (4) and one for employees (5). Routing between this vlans provides linux router beyond our scope here. We used:

192.168.1.0/24 – management vlan ID 3

192.168.2.0/24 – management vlan ID 4

192.168.3.0/24 – management vlan ID 5

Create a Bond with four links, to high bandwidth, with default vlan 600:

Now, we can configure our first CAP. This happened only once. Any WIFI setting will be configured via CAPSMAN itself. So I set CAPs up for using, accessing and sending data only via management vlan (vlan id = 3). Every traffic will be forwarded to the CAPSMAN.

Now, we download our CA public certificate from our CAPSMAN, import it. Then we create a local certificate, and send it as template to Scep server running on CAPSMAN. Then we must manually approve this template, and it will be signed by our previously created CA certificate on CAPSMAN. And this signed certificate will by user for encrypted communication between CAPs and CAPSMAN. This step must by manually set for each CAP separately.

And finally, we set this on CAPSMAN for provision radio setting to CAP1, or next CAP2…We can limit these for MAC address of CAP1. This my setting allow to connect any CAP with certificate, that has been previously granted.

I create a basic installation of Centos 8 stream from iso: CentOS-Stream-8-x86_64-20191219-boot.iso

During installation I choose minimal applications and standard utilities. Please, enable, network time and set lvm for virtio disk. I set password for root and create a new user, which have root privileges.

After instalation, I create and LVM encrypted partition, to store encrypted data of nextcloud on it. I will not use nextcloud data encryption. Command below creates encrypted disk. We must enter a passphrase twice

Now, I write 4GB zeros to this device to see, if everything is OK. It is possible, to full-up tho whole device, but it can take a long time. But the true reason is, that this will allocate block data with zeros. This ensures that outside world will see this as random data i.e. it protect against disclosure of usage patterns.

Installing nextcloud and prerequisites

And now, we can start with preparing our Centos for nextcloud

At first, update system. Via dnf (DNF is the next upcoming major version of YUM, a package manager for RPM-based Linux distributions. It roughly maintains CLI compatibility with YUM and defines a strict API for extensions and plugins.)

dnf update -y

Next, we install and create empty database for our nextcloud. Then we start it and enable for autostart after boot. If you wish, you can skip installations of MariaDB and you can use built-in SQLite. Then you can continue with installing apache web server.

And finally, we can access our nextcloud and set up administrators password via our web: http://you-ip/

If you see default httpd welcome page, disable all lines in: /etc/httpd/conf.d/welcome.confNow you must complete the installation via web interface. Set Administrator’s password and locate to MariaDB with used credentials:

In settings of nextcloud, go to section Administration > Overview. You can see some problems. If so, try to fix it. I had three problems. No apcu memory cache configured. So add at nextcloud config.php:

'memcache.local' => '\OC\Memcache\APCu',

Then I must edit som php variables, to set properly opcache: edit and adjust:

vim /etc/php.d/10-opcache.ini

Then I must edit httpd setting, because .htaccess wont working. So change apache config:

Next, I find out, that my nextcloud instance cannot connect to internet and checks for update. I think, that this is on selinux (enforcing mode). So run check and find out, what is happening:

sealert -a /var/log/audit/audit.log

And the result:

SELinux is preventing /usr/sbin/php-fpm from name_connect access on the tcp_socket port 80
Additional Information:
Source Context system_u:system_r:httpd_t:s0
Source Path /usr/sbin/php-fpm
Port 80
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
---------
If you believe that php-fpm should be allowed name_connect access on the port 80 tcp_socket by default.
If you want to allow httpd to can network connect
Then you must tell SELinux about this by enabling the 'httpd_can_network_connect' boolean.

So I allow httpd to can network connect via:

setsebool -P httpd_can_network_connect 1

And that is complete. If you wont secure http (https), try to find out another post on this page.

And now, gracefully restart your web server and point your browser to you domain: www.example.com (I edit my /etc/hosts to point this domain at my internal IP).

apachectl graceful

If you test page is working, lets begin with more thinks. We must install additional packages (software) for wordpress. Its mysql server and php. As mysql server, I use mariadb. Then create an initial configuration for mysql and create database for wordpress. I set no password for mysql.

-A INPUT: The -A flag appends a rule to the end of a chain. This is the portion of the command that tells iptables that we wish to add a new rule, that we want that rule added to the end of the chain, and that the chain we want to operate on is the INPUT chain.

And now, we can allow specific port or service, which we want to allow:

Now we must add some rule for loopback. because we block it now. If we add it right now with above command, we add it at the end of chain (after drop all). So all traffic will be blocked. We must add it at the begining of this chain:

iptables -I INPUT 1 -i lo -j ACCEPT

-I INPUT 1: The -I flag tells iptables to insert a rule. This is different than the -A flag which appends a rule to the end. The -I flag takes a chain and the rule position where you want to insert the new rule.

-i lo: This component of the rule matches if the interface that the packet is using is the “lo” interface. The “lo” interface is another name for the loopback device. This means that any packet using that interface to communicate (packets generated on our server, for our server) should be accepted.

Now we have five rules to ACCEPT packets, which we want. The we have the sixth rule for DROP all another packets.

The policy DROP everything can be done by two ways. We have the first way (Default policy of chain is ACCEPT everything. Our five rules catch certain packets and at the end we have the sixth rule to DROP all packet which catch all other remain packets). In case of breaking firewall, or accidentally flush our rules, we still can connect to our server (by default chain policy ACCEPT).

The second way is set default chain policy to DROP, and set our five rules first. So if packets are catch by one of this rules, is ACCEPTed. Then it is DROPPEd by default. There is a possibility, that if we flush our firewall rules, we never reach our server from network because the default chain policy is DROP. So first, we need the rules like above mentioned except the DROP rule. And then, at the end, change the default chain policy by command:

So we can see, that we DROP all packet, we want and ACCEPT packets we want. It can be done by this two ways. So pick one, which you want. I prefer the second way, because I have another access to server (via console-keyboard connected directly to server). So if something go wrong, I am still be able to connect it.

So if you choose the first way, you must add others rules before the DROP rule, because it will be matched by this rule. Like the loopback rule, you must insert it somewhere before the DROP rules. See the lines:

Introduction

In addition to being a “basic” web server, and providing static and dynamic content to end-users, Apache httpd (as well as most other web servers) can also act as a reverse proxy server, also-known-as a “gateway” server.

In such scenarios, httpd itself does not generate or host the data, but rather the content is obtained by one or several backend servers, which normally have no direct connection to the external network. As httpd receives a request from a client, the request itself is proxied to one of these backend servers, which then handles the request, generates the content and then sends this content back to httpd, which then generates the actual HTTP response back to the client.

There are numerous reasons for such an implementation, but generally the typical rationales are due to security, high-availability, load-balancing and centralized authentication/authorization.

It is critical in these implementations that the layout, design and architecture of the backend infrastructure (those servers which actually handle the requests) are insulated and protected from the outside; as far as the client is concerned, the reverse proxy server is the sole source of all content.

In this tutorial, we will set up Apache as a basic reverse proxy using the mod_proxy extension to redirect incoming connections to one or several backend servers running on the same network. This Apache Proxy Server also creates and manages security (ssl engine, https). Conection to the backend servers from this Proxy Server is not encrypted (only http). Next, we will use https (ssl certificates from Let’s Encrypt for ours conections from outside world, but not to backend.

Installation

For a minimum HTTP server instalation install apache itself:

yum install httpd -y

Make sure, that the “/etc/hosts” file contain references for the loopback address and the hostname

ProxyPreserveHost makes Apache pass the original Host header to the backend server. This is useful, as it makes the backend server aware of the address used to access the application.

ProxyPass is the main proxy configuration directive. In this case, it specifies that everything under the root URL (/) should be mapped to the backend server at the given address. For example, if Apache gets a request for /example, it will connect to http://media.gonscak.sk/example and return the response to the original client.

ProxyPassReverse should have the same configuration as ProxyPass. It tells Apache to modify the response headers from backend server. This makes sure that if the backend server returns a location redirect header, the client’s browser will be redirected to the proxy address and not the backend server address, which would not work as intended.

Now, we can test out configuration with the first command below. It runs a configuration file syntax test and report OK or error. And with second command we gracefully restarts Apache httpd daemon. If the daemon is not running, it is not started. Currently open connections are not aborted:

apachectl configtest
apachectl graceful

And now, if everything is OK, we can open out web page now (http://192.168.3.3). We now not see the default page of apache, but the content of backend server media.gonscak.sk. We are not connected directly to the media.gonscak.sk, but only to the “edge” server with Apache.

Enabling SSL support, set certificates from LetsEcnrypt

First, we must install package mod_ssl for Apache to support SSL:

yum install mod_ssl.x86_64

Now, we must open port 443 for Apache in firewall:

firewall-cmd --add-service=https --permanent
firewall-cmd --reload

Now, we create o text file, where we set up some directives for vhost. And then we can simple change som SSL directives for all vhosts in Apache. I use some Mozilla recommendations via https://mozilla.github.io/server-side-tls/ssl-config-generator:

I Installed proxmox environment on Intel 240GB SSD. Installation take the whole disk for lvm. So I need to reduce the used space and create a new partition for drbd.
This is my disk. You can see, that the disk is full allocated with 171G free.

We can see, the size of PV is 223,44G and we have free 171,44G. So, we must shrink this physical volume about 171,44G. So compute the space for size of this physical volume: 223,44 – 171,44 = 52G. So, our PV must have at least 52G. Next, we resize this pv:

At this point, we must work on the lowest layer of disk, so we must delete this partition and create a new one. The new partition must start on the same sector as previous and the last sector must be after last segment of physical volume. I use gdisk, because my disk have GPT partition table:

RAID 10, also called as RAID 1+0 is a stripe of mirrors. It require four disks at least. It stripes data across mirrored pairs. So, as long as one disk in each mirrored pair is functional, data can be retrieved. If two disks in the same mirrored pair fail, all data will be lost, because there is no parity.
Raid 10 provides redundancy and performance despite of 50% capacity of disks.Note on why to use different manufacturers disks: Disks will fail, this is not a matter of a “if” but a “when”. Disks of the same manufacturer and the same model have similar properties, and so, higher chances of failing together under the same conditions and time of use. The suggestion so is to use disks from different manufacturers, different models and, in special, that do not belong to the same batch (consider buying from different stores if you are buying disks of the same manufacturer and model). This is not uncommon that a second disk fail happen during a resotre after a disk replacement when disks of the same batch are used. You certainly don’t want this to happen to you.
So we have four disk fo this: /dev/sdc, /dev/sdd, /dev/sde, /dev/sdf. At first, we check, if there is any previous md superblock. So we examine this disks:

Now, we must create a partitions with the same size. Disks from different manufacturers (or even different models of the “same” capacity from the same manufacturer) don’t necessarily have the exact same disk size. And in future, we can replace failed disk with another disk (maybe a bigger), but we must create partition with the same size.
So, list the disks size:

How to create a disk or usb image, and compress it on the fly? And how to restore it?
I have own operating system on USB key. To create a full-backup and then possible restore to another device, I use linux command dd (dd – convert and copy a file).
Now, we must determine, on which patch we have s source disk. I my case, it is

sudo fdisk -l /dev/sdb
Disk /dev/sdb: 29,5 GiB

First, I install additional software for monitoring and best compressing on more cores

sudo apt-get install pigz pv

Then, I create a full copy of the usb key. Without compression it takes 30GB, with compression, it take only 3GB. With command “pv” we can watch progress. Pigz compress the source image with multiple threads and cores. With parameter -c it writes all processed output to stdout. So with operand “>” we write this pigz output to a file:

If we had som bad blocks on source disk, and we want to clone it anyway, we can use another conv options. Like:

conv=sync,noerror

This means:

noerror – This makes use dd continue even after a read error is encountered;

sync – This option has sense especially when used together with noerror.

In such a case the noerror option will make dd continue running even if it a sector cannot be read successfully, and the sync option will make so that the amount of data failed to be read its replaced by NULs, so that the length of the data is preserved even if the actual data is lost (since it’s not possible to read it).

Then, I remove the source usb key and insert new one. It also has a path /dev/sdb. Now, I restore it with this command:

pigz -cdk Documents/corsair-work.dd.gz |pv| sudo dd of=/dev/sdb bs=4M

Parameter -c also write output to stdout and program dd writes it to disk. Parameter -k menas, that keep original file after decompress. And parameter -d means decompress.
Now, we can boot system with new usb key. And this image is identical as the source.
I hope, that this help someone. Have a nice day.

Today I met with backup problem. I nee to find and set up solution for backup and possible restore of files in windows or linux. I heard about bacula, but after som searching and reading, I choose a new fork of bacula – bareos.

Installing Bareos itself

So I install it on new, clean vm centos 7. At first define a hostname:

cp /etc/bareos/bareos-dir.d/storage/File.conf /etc/bareos/bareos-dir.d/storage/backups.conf
chown bareos:bareos /etc/bareos/bareos-dir.d/storage/backups.conf
vim /etc/bareos/bareos-dir.d/storage/backups.conf
- change Name and Device. Name must be the same as above:
Name = Backups
Device = Backups

At first, please update your centos. Every command I use, is used as root 😉

yum -y update

Installing database server MariaDB

Next, we install and create empty database for our nextcloud. Then we start it and enable for autostart after boot.
If you wish, you can skip installations of MariaDB and you can use built-in SQLite. Then you can continue with installing apache web server.

Now we can access our server via https://out.server.sk
If we want signed certificate from letsencrypt, we can do it with next commands. Certboot will ask some questions, so answer them.

yum -y install python-certbot-apache
certbot --apache -d example.com

If we are good, we can see:

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at
/etc/letsencrypt/live/example.com/fullchain.pem.
...

And we can test our page with this:

https://www.ssllabs.com/ssltest/analyze.html?d=example.com&latest

Install PHP 7

As creators of nextcloud recommends at minimal PHP 5.4, I use php 7.
PHP 5.4 has been end-of-life since September 2015 and is no longer supported by the PHP team. RHEL 7 still ships with PHP 5.4, and Red Hat supports it. Nextcloud also supports PHP 5.4, so upgrading is not required. However, it is highly recommended to upgrade to PHP 5.5+ for best security and performance.
Now we must add some additional repositories:

If you see error “-bash: semanage: command not found”, install packages:

yum provides /usr/sbin/semanage

yum install policycoreutils-python-2.5-33.el7.x86_64

And finally, we can access our nextcloud and set up administrators password via our web: https://you-ip/nextcloud
Now you must complete the installation via web interface. Set Administrator’s password and locate to MariaDB with used credentials:

This short post show you, how to resize small vhd/vdi file to one bigger file. And this bigger file can be dynamic or fixed size on hard drive. I working on SSD disk, so it is very fast 🙂 I use comnad line in windows (start > run > cmd). And enter into virtualbox directory:

And now, we can clone it into fixed size. Fixed size of this disk is better for performance on classic disk. Dynamic is better on SSD disks, because there is never-ending resize of this file and virtualbox must allocate new space if the virtual machine grows in lifetime. So dynamic file allocate its space at the beginning. It ok for me, because I don’t care about the space of this file on beginning.

/mnt/nfs/sdb.log is the log file, where is written every bad block and actual position of ddrescue. We can brake this rescue at any time and continue it later with the same command. When ddrescue finish, we can repeat this check only on bad blocks with more retries

22.3.2017 – it was stared. post will continue after it finished 😀 maybe it take 3 days to finish, maybe more 🙂 This operation takes a long time to finish…

Today, I met with an interesting problem. I tried to create a primary-primary (dual primary) DRBD cluster on proxmox.
The first we must have fully configured proxmox Two-node cluster. Like this:
https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster
We must have a good configuration of /etc/hosts to resolve names into IP:

One server was created on hardware raid PCI-E LSI 9240-4i (/dev/sdb) and second server was build on software raid via mdadm (/dev/md1) on debian jessie with installation with proxmox packages. So the backend for drbd devices was on one side – hardware raid and software raid on the other side. We must create a two disks with the same size (in sectors):

So we can see, that I have a bonded network from two network cards and the resulting speed is almost 2Gbps.
Now, we can continue with installing and setting up the drbd resource.

apt-get install drbd-utils drbdmanage

All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf. Normally, this configuration file is just a skeleton with the following contents:include “/etc/drbd.d/global_common.conf”;include “/etc/drbd.d/*.res”;
The simplest configuration is:

Start the initial full synchronization. This step must be performed on only one node, only on initial resource configuration, and only on the node you selected as the synchronization source. To perform this step, issue this command:

We can configure this drbd block device as physical volume for lvm. This lvm is on top of this drbd. So, we can continue as it is a physical disk. Do it only on one server. The change will reflect on second server, due to primary-primary disk of drbd:

pvcreate /dev/drbd0
Physical volume "/dev/drbd0" successfully created

As we can see, we must adapt /etc/lvm/lvm.conf to our needs, because it scans all block devices and we can found duplicate entries:

So, we must edit filter option in this configuration. Look at our resouce configuration r0.res. We must exlude our backend devices (/dev/sdb1 on one server and /dev/md1p1 on second server), or we can reject all devices and allow only specific. I prefer reject all and allow only what we want. So edit the filter variable.

And finally we add the LVM group to the proxmox. It can be done via web interface. So, go to proxmox web interface to Datacenter, click on storage and add (LVM).
Then create your ID (this is the name of your storage. It can not be changed later. Maybe: drbd0-vg), next you will see the previously created volume group drbd0-vg. So select it and enable the sharing by click the ‘shared’ box.
Now, we can create virtual machine on this LVM and when we can migrate it without downtime from one server to another because of drbd. There is one shared storage. So when the migration starts, machine is started on another server and through ssh tunnel is migrate content of ram. And after few seconds, it is started.
Sometimes, after some circumstances with network disconnect and connect, there is split-brain detected. So if this happened, don’t panic. When this happened, both servers are marked as “standalone” and drbd storage started to diverge. From this time there happened different writes to both sides. We must one of this servers mark as victim, because one of these servers has the “right” data and the other has “wrong” data. So the only way is backup the running virtuals on the “victim” and then we must destroy/discard this data on drbd storage and synchronize it from other server, which has “right” data. So if this is happening, this is in logs:

I waited until finish this operation. Then I halted this server, remove the exact drive and insert a new one. After power-on, we create a new partition table on /dev/sda exactly as old one, or as active disks now. The we re-add it as spare to the raid: