Open letter to Cisco Systems: When are you going to start building more network troubleshooting and monitoring commands into your operating systems?

Case and point, why isn’t there any sort of port scanner or host discovery command available for Cisco IOS? The nmap program is open source (GPL), written in C/C++. Wouldn’t it be fairly straightforward to make something like that available for Cisco routers and/or firewalls?

It would be highly useful to be able to perform more in-depth testing, troubleshooting, and security monitoring from the Cisco CLI. As it stands now, your best bet is to just perform basic configuration on your routers, and then log into the nearest Linux system to actually take some measurements and see how everything is working.

This isn’t to say that Cisco routers should become full-blown monitoring stations. But how about a slightly more full-featured set of command-line tools? The available monitoring commands on Cisco routers haven’t changed in years, maybe decades. And yet, much network management work is still being done from the command-line.

What do you think? Should Cisco beef up its command-line network monitoring tools? Let us know in the comments.

The following guide shows how in 10 easy steps you can connect to and perform a base configuration of an Amazon Linux AMI running in the EC2 cloud. The configuration includes customization and important security settings.

Amazon’s native Linux AMIs are optimized for their cloud environment and are eligible for the free tier, which allows you to run your own Linux server free for a year.

The following tutorial assumes that you have registered for an AWS account, and have just launched a new EC2 instance based on one of the Amazon Linux AMIs. To start using your new Linux server, you’ll need to connect via SSH and manage it via the command-line interface (shell). Read on to learn how…

Step 1. Connect to the Instance

First things first, you need to connect to your shiny new Linux server. To do so, you will need access to a system with some sort of SSH client installed on it. If you have a Linux or Mac system, you can use SSH from the command-line using the standard “ssh” command. If you’re using Windows, you can use the Putty program instead.

To connect, you need 2 things:

The public hostname assigned to the instance by Amazon. While logged into your AWS account, click the EC2 tab, and go to “Instances”. Select the new instance and note the “Public DNS” as listed in the instance “Description”. The hostname should look something like: ec2-50-37-26-225.compute-1.amazonaws.com

The private key. Typically, each Amazon AWS account has an associated keypair that’s assigned to each instance launched from that account. When you initially create the keypair, you’ll be prompted to save the private key as a “.pem” file. You need to store this file on the system where you’re using SSH to connect. If you forgot to download it, you can access it via the AWS menu by clicking on your account name at top and then selecting “Security Credentials”.

Once you have these things, you’re ready to connect via SSH. The default username is “ec2-user”. If you’re using SSH from the command-line, change into the directory where the “.pem” file is stored, and connect as follows: ssh -i mykeypair.pem ec2-user@ec2-50-37-26-225.compute-1.amazonaws.com

If you’re using the Windows “Putty” program to connect, go to “Connection, Data” and input “ec2-user” as the username. Then go to “SSH, Auth” and browse to select the location of your “.pem” private key file.

Step 2. Add a New User Account

You’re now connected to your new Linux server. The first thing to do is add a new user account to use instead of the default user account. In this case, we’ll add a user account called “myadmin”:

First, open a root shell: sudo -s

Create the new user account: useradd myadmin

Assign password to new user: passwd myadmin

Step 3. Configure SSH Server

Next, configure the SSH server to allow access via standard username/password logins (instead of the key). To edit the SSH config file and most other Linux config files, you’ll need to use a text editor program. The recommended one is vi, which gets invoked from the command-line as: vi FILE.

Follow these steps to configure the SSH server:

Open file for editing: vi /etc/ssh/sshd_config

Scroll down until you see the line that reads PasswordAuthentication no … change the “no” to “yes”

Save your changes by entering :wq

Once you are done updating the SSH config file, you must restart the server to apply your changes. To do so, you can make the command /etc/init.d/sshd restart while logged in as root (sudo -s, see above).

Step 4. Further Lock Down SSH Server

This is optional, but usually a good idea. You can lock down SSH server security by adding an AllowUsers line to the SSH config file. You can add it anywhere. The directive is followed by a space-separated listed of users who are allowed to connect, as in the following example: AllowUsers myadmin myadmin1 myadmin2

With the above line in place, only those specific users will be allowed to log in via SSH. Remember that you have to restart the SSH server (/etc/init.d/sshd restart) after making any changes to the config.

Step 5. Test New SSH Connection

Once you’ve added your new user account(s) and re-configured SSH, make absolutely sure that you can log into it again before closing your existing session!!!

In other words, keep your active SSH session open, and try initiating an additional SSH connection using the new user account you have configured. If you are able to log in, you’re good to go. Otherwise, check your SSH config file again and make sure you’ve followed the above steps.

Step 6. Set New Root Password

Next set a new root password. For security reasons, set something different than for your main user account. To change the root password, simply make the passwd command while logged in as root. You’ll be prompted to change it.

Once you’ve set the new root password, make sure that you can open a root shell by making the su command and entering it.

Step 7. Get Rid of Default User

Now that you have custom user account with SSH access and a working root password, you should get rid of the default “ec2-user” account for security reasons. To do so, use the following steps:

Lock the account by making this command as root: passwd -l ec2-user

Remove the account from the “/etc/sudoers” file. To do so, make the visudo command and then find the line for the “ec2-user” account and either delete it or comment it out, then save (:wq) your changes

Step 8. Set Hostname

You can set a new hostname, which will be displayed in the command prompt so you can make sure that you’re in the right place. To change the hostname, make sure you’re logged in as root and make the command: hostname NAME

The name you set with the above command will not be saved if the system gets rebooted. To make it permanent, edit the “/etc/sysconfig/network” file and set the name in the line: HOSTNAME=NAME

Step 9. Set Timezone

All Amazon instances are in GMT time by default. To make sure that logs and such reflect the correct times, you should set the correct local timezone. To do so, you first point the correct timezone data file at your “/etc/localtime” file, and then edit the “/etc/sysconfig/clock” file to ensure that the correct timezone gets applied the next time the system reboots.

In the following example, we set the local timezone to US Eastern Time via the America/New_York timezone data:

Make the change permanent by adding the following line to the "/etc/sysconfig/clock" file: ZONE="America/New_York"

Step 10. Set Login Message

The last thing that you should consider doing is setting up a custom login message to display whenever a user logs in. The main file used to set this is “/etc/motd”, but on some of the new Amazon AMI versions you may need to set it in one of the files in the “/etc/update-motd.d/” directory instead.

…At this point, you’re all set! The next thing to do is start adding new software to the instance. This will be discussed in a future post.

The Problem: There’s a bug in the httpd.spec file of certain releases of the Apache httpd server source RPM. This definitely affects recent versions of Fedora, and perhaps other Redhat-based Linux distros.

You may encounter this if you find yourself in the unfortunate position of having to re-build the httpd RPM from source. For example, perhaps you are trying to get Virtualmin to work and need to rebuild Apache with the “suexec_docroot” option set to the /home directory (NOTE: to do so, edit the httpd.spec file as follows: %define contentdir /home).

Anyway, the bug in this case is related to a safety check performed to ensure that you have the correct value set for the Apache module magic number (MMN) in the httpd.spec file. When you try to build the new httpd RPMs with rpmbuild -bb httpd.spec, you’ll get an error message containing some text like this:

Just make sure that you do indeed have the correct value set for the MMN at the top of the file, in this line:

%define mmn 20051115

Note that the above value is for Apache version 2.2.19. You can find the complete table of valid MMN values in the BUILD/httpd-VERSION/include/ap_mmn.h file from your source RPM, where VERSION is the httpd version number you’re building.

The Problem: You’re setting up inter-VLAN routing on your Cisco ASA firewall (5510, et al) using sub-interfaces. There should be no restrictions on what traffic can flow where between the internal VLANs, so you’ve set the same security level on all of the sub-interfaces and have added the configuration command(s) to allow “same security” traffic to move freely.

Should work like a champ, right? Not so fast! Things never work quite as planned when we’re talking about Cisco’s powerful but often frustrating line of firewalls.

When you use a separate internal router for inter-VLAN routing, this type of setup is sometimes referred to as a “router on a stick”. But as pointed out in an earlier blog post more and more small and mid-sized companies don’t want to spend the money on an extra router (or have an extra point of failure), so they’re just using their ASAs to do the inter-VLAN routing.

The main difference here is that the ASA is also the gateway out to the Internet, which means that it performs NAT. Another difference is that Cisco still hasn’t made the inter-VLAN routing implementation very user-friendly on ASAs. But it does work if you know how to set it up.

The Solution: From a technical standpoint, the main issue here is that the ASA also does NAT whenever inside traffic connects out to anywhere else. You might think that it would know to keep the same addresses when traffic flows from one VLAN-assigned same-security interface to another, but it doesn’t. You have to add static commands for every possible subint-to-subint (vlan-to-vlan) interconnection and specifically tell it to use the same addresses before and after NAT. See below for an actual configuration.

Configuration: Here’s a typical configuration for setting this up on an ASA 5510, along with some comments.

Let’s say that you are routing for 3 different VLANs via sub-interfaces on the physical E0/0 interface. The base configuration on the ASA looks something like this:

Note the two same-security-traffic statements that allow traffic to flow between and within interfaces and sub-interfaces that have the same security level. This prevents you from having to add access lists, but that’s about it.

Inter-VLAN routing won’t work with the above configuration as is. You might think you could just add some additional globals for the sub-ints, but that also won’t do the job with this setup. What you need are a series of static entries, basically telling it not to NAT traffic between the various sub-ints (actually, it does NAT them, but keeps the same IP address before and after). You’ll need a separate static for each subint-to-subint (vlan-to-vlan) connection. For each one, the first sub-int specified is the one whose IP network you need to enter twice in the static statement.

For example, to get the above configuration to work, you would add the following:

Once you get those statics in there, inter-VLAN routing will work just fine. But it’s a bit ugly, isn’t it?!? For N number of VLANs, you need (N-1) x N of these statements, so for just 5 total VLANs you need 20 of them.

Discussion: Cisco needs to re-think the way that ASAs work in terms of VLANs and routing. It would be great if they realized how their devices are actually being used out in the field, and put some thought into making things more user-friendly. Who are the actual end users of Cisco gear? The people using the network to check their friends’ Facebook statuses, or the people who actually have to install this gear and administer it on a day-to-day basis? The correct answer is both, but you sometimes get the sense that Cisco only cares about the former.

At the end of the day, this type of extra configuration is not really a huge deal, but it’s the type of thing that can throw you for a loop and make your work-day seem longer than it should. The good news is that, once you get it working, it works well. And it’s nice to eliminate that separate router and do things on the ASA instead.

The Problem: You’ve set up a trunk link between a Cisco ASA firewall (5505, 5510, et al) and a Cisco switch (2960, 3560, et al). However, you’re not able to establish a connection between them at either Layer 2 or 3.

If you look at the interface counters on the firewall, you may notice that you’re piling up a bunch of “L2 Decode Drops” errors. Other than that, there isn’t much evidence about what could be wrong.

The most likely cause is the native VLAN on the switch side of the trunk. Chances are, you have a sub-interface on the ASA that is supposed to route for that native VLAN. Well, when the switch sends out a frame from the native VLAN, it does NOT tag it. But if the ASA has a sub-interface for that VLAN, it expects tagged frames only for it, so communications for that particular VLAN will fail between the switch and the ASA.

Bottom line: if you have a sub-interface for a VLAN on the ASA, then it CANNOT also be used as the native VLAN on the switch’s trunk port. Any VLAN set up as such will not work. (To verify the native VLAN on your switch, use the command: show int trunk. It should be set to VLAN1 by default.)

To solve this, change the sub-interface/VLAN configuration on the ASA to avoid the switch port’s native VLAN, or just change the native VLAN on the switch to something else (example command from interface sub-config mode: switchport trunk native vlan 99).

Discussion: Cisco’s implementation of the “native VLAN” on 802.1q trunk ports is clunky. It is especially clunky because it’s not consistent across different devices made by Cisco.

More and more small and medium-sized networks are using their ASA firewalls to do inter-VLAN routing, but ASAs do not afford very much control over trunking or VLAN-related parameters. At some point, hopefully Cisco will provide a more full-featured VLAN/trunk implementation for ASAs.

As it stands now, VLAN configuration on an ASA consists of simply adding the sub-interfaces for the VLANs, which in turn automatically enables 802.1q trunking on the physical port, and that’s that. If you want to have a so-called “native VLAN” on the ASA, you must assign the corresponding IP address to the physical interface, and cannot add a separate sub-interface for that VLAN. Since all Cisco switches have an explicit “native vlan” command, it would make sense to have a corresponding command that could be assigned to one of the ASA sub-interfaces. (Or maybe just get rid of native VLANs altogether, but that discussion will have to wait until another day!)