At the Circus we have been having random reboots and are struggling to figure out the culprit as no one can seem to find any errors in the logs. In order to help narrow the scope I wrote this script to help us hunt down the culprit.

The script pings a number of infrastructure services such as gateways, ESXi hosts and VMs. I run it every minute from one of the linux servers.

After setting up TACACS+ and FreeRADIUS I decided to go ahead and add more services to my main test lab server. I am using CentOS in the lab, and decided to add a syslog server and an FTP server to the mix.

Rsyslogd
This is a very simple process as we use Rsyslogd as our production syslog server. First we need to uncomment some lines in the file /etc/rsyslog.conf. The most important lines are the ones at the bottom of the code listing, they tell Rsyslogd to listen on UDP port 514.

Configure FTP
First add a user where the configuration files from the routers will be stored. Just to keep things simple and consistent I added the user cisco with the password CCIE. Obviously this is a lab only environment, I would never do this production.

useradd cisco
passwd cisco

For this portion of the post I am just using one of the many howto’s on the internet. Once again, I have been burned by not documenting my steps for a process so I will document them below.

Install proftpd.

yum -y install proftpd

Make a backup of the configuration file.

cp /etc/proftpd.conf /etc/proftpd.conf.0

Make sure that users are chroot’ed to their home directories.

# Cause every FTP user except adm to be chrooted into their home directory
DefaultRoot ~ !adm

I have a number of servers in the lab, but my main server is a jumpbox that straddles the lab and our network named TLTS1. I wanted to be able to really test authentication in the lab so I decided to set up TACACS+ and FreeRADIUS. Here are the steps I followed.

Add a user that will be authenticated in the /etc/raddb/users file. First we will do some simple authentication, then we will setup a router. The “me” user is to test from localhost to make sure everything is working, the “cisco” user is for testing from a router in the lab.

I have put quite a bit of work into my lab routers and the configurations, enough that I have begun to worry about about how much work it would be to replace my current set up.

I have been backing up my primary server for the lab, but tonight I finally decided to back up the configurations on the routers. Here are the commands I ran on the ten CSR 1000V routers that make up the backbone of my lab.

I then created a tar of all of the backups and put on my separate USB drive that I only plugin for backups. Yes I use Time Machine, but I also keep a backup of the most important files on a USB drive that is not plugged into my MAC. Paranoid maybe, but I sleep well at night.

I just fought installing VIRL on bare metal for nearly four days. I worked my way through multiple how-to’s and installed it three different times. Eventually I gave up doing it by the book and tried my own thing.

First, some back story. Linux changed the way network cards are named, and it appears that each vendor has implemented their own fix. As of version 1.2.84 VIRL is based on Lightweight Ubuntu (LUBUNTU) 14.04. Ubuntu has changed the network cards from being named eth0 through eth5 to em1 through whatever. I say whatever because it depends upon the cards you have installed.

For instance on my Dell R710 the cards were named em1, em2, em3, em4, p1p1 and p1p2. These stand for embedded network interface one through four, PCI slot 1 interface 1 and PCI slot 1 interface 2. Under the old naming scheme they would have been named eth0 through eth5.

The problem arises in that Openstack underneath VIRL is configured for eth0 through eth5. I worked my way through a number of different how-to’s, which are listed below, but I always seemed to end up with a problem.

Finally I just changed my interface back from the new naming scheme to the old ethX naming scheme.

I edited the file /etc/default/grub and added the following lines.
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="biosdevname=0"
GRUB_CMDLINE_LINUX="biosdevname=0

Because I had been fighting with the install, I also rehosted the install to make sure my interfaces lined back up.
sudo vinstall rehost

Then I rebooted my VIRL host and everything worked. I should have followed my gut and saved myself four days of lost study time.

I haven’t been posting much about my studies for the CCIE this time around. My studies have been all consuming as I have been trying to get in 30 hours a week on top of my normal work. I may start blogging my notes, but have not made the decision yet. It would be nice to share my notes with others as I have worked extremely hard on them, but many of my notes are quotes from sources whom I do not remember.

After taking Narbik’s class I wanted to give my thoughts on the class. First, the class will be international with a wide range of abilities. Here are is a list of guys in my class.
George from Sydney Australia
Martin from Holland
Dustin from South Dakota
Me from Alabama
Denis from Canada
Dennis from California
Mike from California
Martin from California
Joe (not his real name) from the NSA

We were broken into basically three groups. There were three of the class that were a few months away taking the lab, there were three of us that were 6 to 9 months away from the lab and three that needed to decide if they were ready to dedicate the time needed.

The class starts with the traditional, tell the class about yourself. Narbik also invites each student to detail what you need to work on. Do not be shy. In fact I would make a list before you get to class of your strengths and weaknesses because this cumulative list sets the structure of the class.

The class is also designed for each student to get out of it as much as they put into it. If decide to slack on the labs, it will only hurt yourself. The first few days are some theory, but the focus is on configuration labs. As you lab into the night or morning, Narbik gets an idea of where each student is in their studies.

I also believe each student should make a concerted effort to finish most of the Micornics Training workbooks before you get there. As I have done different labs, I have taken notes to jog my memory or clarify my understanding of a technology. Many times I recognized the technology and topology we were discussing from labs I had done and was updating my personal notes from Narbik’s notes on the board.

If a CCNA attended Narbik’s class, from the lectures they would not think the CCIE is that difficult. Narbik focuses on portions of the technologies that he has seen students not understand correctly. I had multiple “ah hah” moments where I had a misunderstanding that I corrected in my notes. But the lectures are short and to the point, doesn’t go over the basics, the class is more about fine tuning your understanding.

The first “big” lab is Thursday night, you can work as late into the next morning as you would like. Most of our class left by 3:00am Friday morning. Saturday is the big assessment lab. One student decided to wait and save the big assessment lab for when he was better prepared. I thought that was a wise decision, and in retrospect it might have been good for me to as well, but it also helped me highlight gaps in my preparation.

The second week of classes is where I got the most benefit. While there are still lectures, the labs shift from technology labs to troubleshooting labs. While I could not get all of the tickets, the labs were designed to highlight technological misunderstandings. I enjoyed these more than the configuration labs as they helped drive home portions of the technologies.

I recommend Narbik’s class. My personal opinion is that students should be close to taking the lab and probably have passed the written. I am neither of these, but the experience has changed some of my training plans, and that is invaluable.

This stack is part of a larger project that I created nearly ten years ago. I am on the fourth rewrite of this for some internal email. We are in the process of migrating from Cyrus and Squirrelmail to Dovecot and RoundCube. These are my notes from that build process.

At this point I installed Thunderbird and tested sending and receiving email.

Enable SSL for Apache.
First go through and remove all of the Listen directives in the Apache configuration files. If you don't it will come back to bite you.

grep -ir Listen /etc/httpd
/etc/httpd/conf/httpd.conf:# Listen: Allows you to bind Apache to specific IP addresses and/or
/etc/httpd/conf/httpd.conf:# Change this to Listen on specific IP addresses as shown below to
/etc/httpd/conf/httpd.conf:#Listen 12.34.56.78:80
/etc/httpd/conf/httpd.conf:#Listen 80
/etc/httpd/conf.d/securemail.conf:Listen 192.168.1.1:443
/etc/httpd/conf.d/ssl.conf:# When we also provide SSL we have to listen to the
/etc/httpd/conf.d/ssl.conf:#Listen 12.34.56.78:443 https

The only thing I changed in the main httpd.conf is I commented out the Listen directives for port 80 and changed the DocumentRoot, however, we just want to make sure that we have Apache up and running with SSL. We will get to that in the RoundCube later. From /etc/httpd/httpd.conf.