This is likely more for myself than anyone else, because I’ve had to create so many KEY and CSR files recently for all sorts of third party devices and appliances. Assuming you have access to a Linux server with OpenSSL you can easily and quickly generate the private key and certificate request with very little hassle.

We need to generate the following pieces:

Generate a private key for this specific use

Using the private key generate Certificate Signing Request (CSR)

Have the CSR signed by a private or public Certificate Authority which will provide the certificate

Upload the private key and signed certificate to your device or system.

Let’s start by creating a directory just for this specific certificate, makes it easier to track all the files we’ll have when we’re complete. In this example I’m going to request a certificate for a Cisco ASA to be used with the Cisco AnyConnect VPN client, vpn.acme.com.

mkdir ~/vpn.acme.com/
cd ~/vpn.acme.com/

Let’s generate a private key, using a key size of 4096 which should future proof us sufficiently.

openssl genrsa -out vpn.acme.com.key 4096

Now let’s generate a SHA 256 certificate request using the private key we generated above.

openssl req -new -sha256 -key vpn.acme.com.key -out vpn.acme.com.csr

We now need to take the certificate request and have that signed by a Certificate Authority. The resulting certificate (filename: vpn.acme.com.crt) will need to be installed along with the private key onto the appliance or device that we’re generating the certificate for.

Since we’re working with a Cisco ASA we need to combine the private key, certificate and any intermediate certificate authorities into a single PKCS12 file so we can upload that file into our Cisco ASA. Again we’ll use OpenSSL for this task and it’s pretty easy. (You’ll be prompted to set a password on the file, make sure you don’t forget it because you’ll need it to upload the file into the Cisco ASA).

Here’s another short story detailing how a simple little change ended up being a bigger headache than I had planned.

It was a simple task, replace the SSL certificate for the discussion forums since it was soon expiring. Renewing the certificate from RapidSSL was a relatively easy task. I uploaded the new intermediate root and certificate files to the server, bundled them together into a single file and modified the Nginx configuration and proceeded to restart the Nginx process. Oddly enough after I restarted the Nginx process I got an “ERR_CONNECTION_REFUSED” from Chrome. A quick test via cURL provided the same result, “connection refused“. I backed out the configuration change and restarted Nginx only to still have the same problem. I thought, “now that is very odd indeed”. I had backed out the configuration change yet I was still having an issue. I quickly realized that the problem was impacting all the websites I managed on that specific server and it appeared that any HTTP or HTTPS connections were getting refused, I confirmed this from a packet trace by observing a TCP reset packet being sent by the server upon receipt of a SYN packet from the client. I checked to see that Nginx was listening on TCP/80 and TCP/443 and it was listening on both ports [Example: lsof -i / netstat -an]. I got a hint when I checked the IPv6 address using cURL and got a response. Nginx was answering IPv6 requests but essentially ignoring IPv4 requests. Something else must have changed outside of the simple certificate configuration change that I had already rolled back.

A quick look at yum revealed that Nginx was updated back on September 10th, that was a significant find.

This was the first time I had restarted Nginx since the update back in September, and that was the key to unlocking the mystery. I tried backing out the update

yum history undo 42

but that left me without Nginx installed at all. I suspected something changed in Nginx with the update, I know that the server was responding to IPv6 requests but not IPv4 requests so I started looking at the configuration files for the virtual hosts and quickly focused on my use of a single listen directive for both IPv4 and IPv6.

listen [::]:80;

I looked back at the server logs and determined that Nginx was upgraded from 1.0.15-5 to 1.10.1 back in September. It turns out that as of 1.3.4, the ipv6only directive is enabled by default which disables IPv4. While doing some research I also stumbled across an article from Michael Hughes titled ‘Nginx ipv6only setting gotcha‘ which described the same issue I was experiencing.

I adjusted the configuration of my virtual hosts by using the following;

listen 80;
listen [::]:80;

I had planned to spend about 30 minutes replacing the SSL certificate, after almost 2 hours of downtime I finally managed to get the websites up and running again. This is par for the norm working in Information Technology, you usually need to be a part-time detective to figure out what broke before you can fix anything. I eventually got back around to replacing the SSL certificate and that worked without issue.

The past two weeks have been an interesting blur thanks to GeoTrust revoking the Aruba certificate securelogin.arubanetworks.com which is used in the captive portal for all Aruba APs, including Aruba ClearPass. The problem started when I received a notification from Aruba on September 9th with the subject line of “Aruba Support Advisory ARUBA-SA-20160908-01 :: ArubaOS Default Certification Revocation”. Unfortunately I didn’t get around to really reading the notification until September 12th. And wouldn’t you know I got my first call about a problem with guest wireless the next day on September 13th.

In summary, a third party named SEC Consult posted a large number of private keys to GitHub including the private key for securelogin.arubanetworks.com which they had extracted from an Alcatel-Lucent OmniAccess Wireless Access Point. It just happens that Aruba is the OEM for the Alcatel-Lucent OmniAccess product line. With the private key public, it’s fairly easy for anyone to perform a man-in-the-middle attack and eavesdrop on supposedly secure traffic. Shortly afterwards GeoTrust revoked the certificate which will now cause browsers to throw an error when using the certificate for guest registration or captive portal.

This will only happen for browsers which have recently updated their CRL list. If we check the CRL we’ll find the certificate with serial number 01DA52.

We had to replace the certificate, but how do you replace the certificate on almost 600+ Aruba Instant AP Virtual Controllers? That’s where we looked at utilizing Airwave Management Platform (AMP), and the process appeared to be pretty straight forward although like almost everything with technology we quickly ran into problems. We started by getting a new public certificate and combined the certificate, intermediate certificate and root certificate all into a single file. We uploaded that file to AMP and then went through the multiple groups within Airwave to enable the new certificate. And we quickly thought, that was too easy and it was. While a number of groups worked and the VCs in those groups were getting the new certificate, a large number of groups were not working and those VCs were not getting the new certificate. We turned to Auba TAC should guided us through the following steps 1) upgrade of AMP from 8.0.9.2 to 8.0.10, 2) upgrade of the IAPs from 6.3.1.2-4.0.0.4 to 6.4.2.6-4.1.3.1, 3) second upgrade of AMP from 8.0.10 to 8.2.2. We finally discovered that the following was missing from the template; %captive_portal_cert_checksum%. Without this line in the configuration template AMP will not push down the certificate to the IAPs.

It’s been a crazy for weeks for me… vacation, consulting engagements, traveling to Reno, NV to stand up a new network – rack, stack, install, configure, test and turnover. So I thought after returning to Philadelphia this past week that things would slow down a little, boy was I wrong. I had a number of challenges and what follows is just one of them involving wireless – I also have another one involving the Lenovo Thinkpad T460 and the Intel AC 8260 Wireless adapter having issues with 802.11n over a Cisco 1262N AP but that’s another story.

On Wednesday morning I had two Cisco 5508 Wireless LAN Controllers both crash with a “Bonjour_Process_Task” taking too much cpu: 100% error message. It turns out that this is a known issue (CSCux78464 WLC crashes in Process Bonjour_Process_Task) that is resolved in 8.0.135.1, an engineering release which you need to contact Cisco TAC to obtain. If that wasn’t enough excitement for the morning I quickly noticed that of 120 APs that we usually have connected to the WLC we only had about 70 APs connected. A quick examination of the debug logs (debug capwap errors enable) showed that multiple APs were failed to join the controller with messages like “Discarding non-ClientHello Handshake or DTLS encrypted packet” and “DTLS session is not established”. A quick call to Cisco TAC revealed that there are built-in certificates into the APs that can expire over time and that’s what had essentially happened. The certificates had expired since the APs had last joined the WLC and now that the certificates were expired they were not able to join the controller. Thankfully there’s a command in the CLI to ignore the certificate expiration;

config ap cert-expiry-ignore mic enable

With that command configured on the WLC the APs starting joining the controller and all was well again.

The field notice from Cisco providing all the details can be found here.

In early January I enabled HTTPS/SSL on the discussion forums with the primary goal of securing user credentials that were being used to log into the forums. Almost twelve months later I’ve found some free time to migrate my blog to HTTPS/SSL as well. The goal is essentially the same, however, this change is more to protect my credentials as I’m really the only user that logs into WordPress. That said there’s more to secure than just the user credentials. Security is becoming a big and bigger topic as ISPs have been taking more and more liberties with customer traffic. While some ISPs have been data mining and profiling their customer traffic other ISP are actively inserting third parties cookies and headers into the datastream. You only need to search Google for a few minutes to find some blatant examples. It’s one of the reasons I’ve started using private VPN services, using HTTPS/SSL over an IPSec VPN I should say.

In addition, Google has been using HTTP/HTTPS signals in their ranking for quite a few months now. Not sure if that will have any impact on my little blog but I’m happy to try and push that percentage of sites using HTTPS/SSL just a little higher.

I purchased a wildcard SSL certificate from RapidSSL that covers *.michaelfmcnamara.com. I have multiple servers and virtual hosts so it only made sense to purchase a wildcard certificate instead of purchasing multiple individual certificates.

The installation was pretty simple, I did need to bundle all the certificates including the root GeoTrust Global CA, the intermediate RapidSSL SHA265 CA – G3 and then my certificate into the certificate file so the browser was presented the proper SSL chaining. I changed the WordPress Address and Site Address URLs from within WordPress and then I setup a redirect from Nginx;

Any Issues?

Yes, there will be a few issues… Internet Explorer 8 for Windows XP doesn’t support SNI (Server Name Indicator) so that browser won’t be able to connect now that I’ve enabled two SSL enabled sites on the same IP address using two different certificates in Nginx. If you are still using Internet Explorer 8 on Windows XP you should really consider migrating off Windows XP.

Are you going to enable HTTPS/SSL on your blog or website?

Cheers!

Update: December 30, 2015 – I had issues uploading images via WordPress after turning on the HTTP redirect. I was getting the error “An error occurred in the upload. Please try again later.” when I tried to upload an image via HTTPS/SSL. I had to go into wp-config.php and add the following, “define(‘FORCE_SSL_ADMIN’, true);” which appears to have resolved the problem.