Another cool plugin for Firebird (or Mozilla for that matter) is Live HTTP Headers. You can use this plugin to watch your browser's HTTP requests and the server's HTTP responses. To install this plugin, change the permissions on your /usr/X11R6/lib/firefox/lib/mozilla-1.6/components and /usr/X11R6/lib/firefox/lib/mozilla-1.6/chrome directories to be writable by the user installing the plugin. Alternatively, run Firefox as root and then install the plugin.

I'd read Tracking for Multiple Machines in the FreeBSD Handbook, which gives hints on building the FreeBSD userland, or "world," and kernel on one system and installing them on another system. You might do this because the target system is slow and your build machine is fast, or because you prefer to let production machines serve users rather than use CPU cycles rebuilding the world and kernel.

Inspired by this post, I decided to try building the world and kernel on my package builder, "neely," and installing them on target systems. I chose my sensor platform "bourque," as a test system.

First I made sure that neely's world was up to date. I'm tracking the security release of FreeBSD 5.2 (which takes care of 5.2.1). My /usr/local/etc/release-supfile looks like this, with all of the commented lines removed:

Now that neely's world and kernel were up to date, I decided to rebuild the kernel for other machines.

I created a separate kernel configuration file for bourque in neely's /usr/src/sys/i386/conf directory. I named that file "bourque" and edited it to suit the bourque system's kernel needs. Next I created the following script to build a kernel for bourque:

I've started using this method on all of my FreeBSD machines with nonstandard kernels. Systems with GENERIC kernels can still track Colin Percival's freebsd-update.

Update: After speaking with Colin, I have another option that is a hybrid of building your own kernel and using Colin's updates of the userland.

Build the kernel you need for the remote system as shown above with the rebuild_kernel_bourque.sh script. This creates a kernel for the remote machine. On the remote machine, run freebsd-update. If you have a GENERIC kernel on the remote machine, freebsd-update will update the kernel as well as the userland. Now install your kernel using this modified script:

However, I think you use some throughput terms in odd ways. I believe a few changes could make it easier for the reader to appreciate your analysis.

For example, here you say "133 MB/s," presumably to mean 133 MegaBytes per second, for the bandwidth of a 32 bit 33 MHz PCI bus. That is correct. Farther down the same page, you mention "a 100 MBit interface," which I guess you mean 100 Megabits per second. Would it not be better to standardize on 133 MBps and 100 Mbps, respectively?

Just after that you say "2 GBit/s or 266 MBit/s". This is where you are mixing these terms and creating confusion. From the context the first term should be 2 Gigabits per second (2 Gbps) and the second should be 2 MegaBytes per second (266 MBps). I checked the Intel CSAdocument to confirm these values.

On the next page you write "The 100 MBit/s, 100BaseT-Ethernet standard offers a theoretical maximum data transfer rate of 12.5 MBytes/s (MB/s). In real world applications, the actual rate is usually 8 MB/s." I recommend this be changed to "The 100 Mbps, 100BaseT-Ethernet standard offers a theoretical maximum data transfer rate of 12.5 MBps. In real world applications, the actual rate is usually 8 MBps." I think other mentions of "MB/s" should similarly be replaced by "MBps" for clarity.

I bought 64 MB extra main memory and 16 MB extra flash memory. When I opened up the router, the insides looked like this diagram:

I had a single 64 MB DRAM DIMM in the "Primary memory" slot with one free. I had no memory in the "System-code SIMM (Flash memory)" slot, since the 2651XM must ship with 16 MB on the motherboard. Once I snapped in the extra memory, my router recognized it without any trouble, as will be seen later.

I also decided to upgrade from a 12.2 release to 12.3. My old IOS was 12.2(11)T10, which corresponded to the c2600-ik9s-mz.122-11.T10.bin image. The 'show flash' command showed this after the memory upgrades:

First I searched for a suitable IOS from the Cisco IOS Feature Navigator and Upgrade Planner tools. I located a version of IOS which offered NetFlow and SSH v2 in the 12.3 train, 12.3(4)T4 (image c2600-a3jk9s-mz.123-4.T4.bin). I downloaded it to a TFTP server on the same network as the router, into the TFTP server's /tftpboot directory.

I did not make a copy of the existing router flash image as I already had it elsewhere for safekeeping.

If any of the above Memory Requirements are"UNKNOWN", you may be using an unsupportedconfiguration or there is a software problem andsystem operation may be compromised.Rounded IOMEM up to: 5Mb.Using 3 percent iomem. [5Mb/128Mb]...edited...program load complete, entry point: 0x80008000, size: 0x172c840Self decompressing the image : ###########...edited...############ [OK]

Friday, May 21, 2004

I've been following an interesting thread on snort-users about collecting alert data on high speed networks. Users are debating how much traffic Snort can handle. One way to at least start answering this questions is to enable the performance monitor. Vjay Larosa's post was helpful, as it pointed me towards perfmon-graph. This Perl script works with Snort performance monitor output and RRDtool output to produce graphs of Snort performance statistics.

Using perfmon-graph requires two steps. First, enable Snort to output the statistics you need to a text file. Add the following line to your snort.conf file:

This tells Snort to output statistics every 60 seconds to a file called /nsm/snort/perfmon.txt The perfmon-graph README warns us not to set the number following 'pktcnt' too high. For example, if 500 packets are not collected in 60 seconds, then we will not get a statistics output.

After restarting Snort with this new preprocessor, your perfmon.txt file will begin collecting entries like these:

Next decide where to run perfmon-graph and RRDtool. I chose to deploy both on a system other than the sensor running Snort, called janney. On FreeBSD use the /usr/ports/net/rrdtool port to install RRDtool. When running perfmon-graph I got an error about a missing tie/File.pm file, which I learned ships with /usr/ports/devel/p5-Tie-File.

To set up perfmon-graph itself, download and extract it to a convenient directory. I chose to follow the perfmon-graph author's suggestion to use SSH to periodically copy the perfmon.txt file to the system running perfmon-graph. I made the following entry in my crontab to do this:

This setup relies on user worker being able to SSH from janney to my sensor, bourque, using public key authentication. I'm using perfmon-graph.pl on FreeBSD 4.9 STABLE, which had an old version of Perl installed (5.005_03). I installed a newer version using the latest package (5.6.1), and executed 'use.perl port' to make it the default. (I can revert to the base with use.perl system.)

Once this is working, a set of files will appear in /usr/local/www/data/perfmon-graph/bourque. You may recognize this as a likely place where Apache could serve up Web pages, and that is the case here. A visit presents graphs like that above, which shows Snort's perception of bandwidth monitoring in Mbps.

The accuracy and value of these statistics are debatable, but they are at least a start.

In addition to setting up performance monitoring, I learned of a great source of information on high speed monitoring issues through this post. It led me to SCAMPI, "A Scaleable Monitoring Platform for the Internet." Their publications page lists many interesting papers with advice and research regarding monitoring high bandwidth networks. The SCAMPI project has the goal of monitoring traffic at 10 to 100 Gbps.

Finally, Brian Caswell posted details on a massive rule update designed to improve Snort performance. He directs interested readers to the updated section of the Snort manual.

Now I needed OpenSSH. I used the packages at Darren Tucker's OpenSSH Page. Unfortunately, as I was downloading the package I needed, I ran out of disk space. This was a result of doing a default install of AIX 5.1 and not understanding how their filesystems work. After deleting the interrupted package, my df output looked like this:

I learned enough about the AIX filesystem via this thread to run 'smit chfs' and make the changes I needed to extend the /usr filesystem. Within smit I selected "Change / Show Characteristics of a Journaled File System" and then selected /usr. Smit reported /usr looked like this:

SIZE of file system (in 512-byte blocks) 1048576

I changed that value to be ten times bigger and commited the change. When done, df showed this output:

Monday, May 17, 2004

This weekend at BSDCan Michael Richardson mentioned a security-oriented IETF working group I'd never heard of before. It's called Incident Handling and its purpose is "to define a data format for exchanging security incident information used by a CSIRT." Also:

"The working group has created four documents. A data model named the Incident Object Description Exchange Format (IODEF), and an associated implementation in an XML DTD, is the format defined for exchanging incident data. The IODEF conforms to a set of requirements for a Format for INcident Report Exchange (FINE). Additionally, guidelines for implementors are provided."

The report also mentioned availability of OpenOffice.org packages for FreeBSD and updates on the TrustedBSD Security-Enhanced BSD (SEBSD) project. I missed Robert Watson's talk at BSDCan.org due to attending Michael Richardson's libpcap 1.0 discussion. There are several papers on TrustedBSD and SEBSD available, however.

Sunday, May 16, 2004

Yesterday I mentioned the report of the theft of Cisco's IOS. While I have no evidence to support this theory, I always assumed that various nefarious parties already had access to some or all of Cisco's previous IOS versions. While access to source code is not necessary to discover vulnerabilities, the allure of obtaining such a prize (for intellectual and competitive intelligence pursuits) made theft a likely scenario. The February report of the theft of Microsoft's source surely did not represent the first time unsavory parties had access to that intellectual property, either.

What does this event mean for Cisco? I found several excellent News.com articles which gave me food for thought. First, last month News.com reported that Cisco will release a new version of IOS on a new product this summer. This new device, the Huge Faster Router (HFR), is designed to compete with Juniper's core routing product, the T640 (Juniper gear pictured below). The new version of IOS will initially only run on the HFR, but it is expected to eventually migrate down the Cisco product line to the edge networking devices.
Unfortunately for Cisco, the first version of the HFR seems hobbled from the start. It's 23" wide, 4" too wide for the standard telecom rack. According to News.com, IPv6 support is incomplete, degrading purchase propositions from the US DoD. Juniper already won the contract for the Global Information Grid Bandwidth Expansion (GIG-BE). MPLS may also not be fully supported.

On the positive side, News.com reports the new IOS will be more modular. Upgrading IOS may not require taking the device out of service. Certain rumors indicate that IOS might be replaced by the real-time operating system QNX. Just last week QNX Software Systems issued a press release describing how their "QNX Neutrino realtime operating system (RTOS) will be shipping as part of the Cisco uMG9850 QAM Module, a new quadrature amplitude modulation product designed to let cable operators use Gigabit Ethernet to deliver video-on-demand."
Meanwhile, another open source router project is paving the way for alternatives for routing at the network edge. XORP, the eXtensible Open Router Platform, plans to release a live CD once the product is released (soon). XORP is developed on Linux and FreeBSD, and News.com reported on the project last month.
Last from the world of Cisco, two new books from Cisco Press look to be excellent reads for network security architects. Sean Convery, who has a new paper on IPv4 and IPv6 threats, just published Network Security Architectures. Mauricio Arregoces and Maurizio Portolani also just published Data Center Fundamentals. When I read both (almost 2000 pages -- give me a few months) I'll review them at Amazon.com.
Update: Cisco announced their new Carrier Routing System on 24 May 04. According to this article, the new 92 Tbps router runs on top of QNX:

"IOS XR helps Cisco catch up in areas such as hot upgrades of software and separation of control, data, and management planes. The software is based on a kernel licensed from QNX Software Systems, but tailored for the job. 'We have made some pretty substantial modifications to [the QNX code] that are Cisco proprietary,' Volpi says."

Also:

"The CRS-1 truly is huge and fast, with a capacity of 640 Gbit/s in a 7-foot rack. It scales to 72 shelves rather than the 18 reported by sources, for an unreal 46 Tbit/s maximum capacity, or 1,152 OC768 ports. (Cisco reports this as 92 Tbit/s, using its usual convention of counting ingress and egress capacity separately.)"

Last month I described the security/portaudit tool, which checks for vulnerable ports and prevents their installation. Sometimes it's reasonable to install a port that has a vulnerability, if the risk is acceptable. For example, the databases/mysql-client port currently reports a security problem when I try to install it:

Saturday, May 15, 2004

"As it became known to SecurityLab, the source code of operating system
CISCO IOS 12.3, 12.3t, which is used in the majority of Cisco network
devices has been stolen on May 13, 2004. The total volume of the stolen
information represents about 800MB in an archive file."

The Russian site shows "ipv6_discovery_test.c -- Neighbor Discovery unit tests" and "ipv6_tcp.c -- IP version 6 support functions for TCP" as some sample code.

Day two of the first ever BSDCan is over. This concludes the conference, which we believe was a great success. Dan Langille reported over 175 attendees and is making plans for a second conference next year.

I started the day with Michael Richardson discussing libpcap 1.0. Michael described how the current libpcap file format, major version 2 minor version 4, will eventually become major version 3. The current format presents a header (pcap_file_header) for every trace file as well as per-packet headers (currently pcap_pkthdr), making it difficult to concatenate two separate trace files. The proposed new version eliminates this header and uses more per-packet headers to facilitate mixing packets from various sources into a single trace file. Compare pcap.h and pcap1.h to get a sense of what he means.

Besides the new header Michael discussed, he also presented a two year old alternative format with its own issues. I also learned to pay attention to the savefile.c file, which describes linktypes.

Michael described his work with netdissect, a library of protocol printers for Tcpdump. Michael mentions it in posts to tcpdump-workers here and here. This is part of an effort to modularize Tcpdump. It will eventually provide options for sending Tcpdump output to places other than the screen. In the future, Tcpdump could be separated into a privilege separated version (different from the existing OpenBSD implementation) where one program uses the kernel and BPF to get traffic, which is passed to a lower-privilege dissector program.
Next I attended FreeBSD Core Team member Wes Peters' talk. He discussed how his company St. Bernard Software builds dependable appliances using FreeBSD. I thought his talk was interesting, but his decision to post the text of his presentation on the screen was not very clever. He said that 14 people have been killed as a result of PowerPoint-like presentations, a claim alluded to elsewhere and discussed in a recent issue of Software Development magazine. That doesn't mean it makes sense to post pages of text in paragraph form, and then stare at them looking for the point one needs to make while standing in front of a crowd. The key to using PowerPoint or slides in general is to present key points, and not get bogged down in descending levels of detail where critical issues are buried from view.

One of the practical points I took from Wes' talk was advice to avoid flash disks in favor of hard drives. If flash must be used, Wes had praise for SanDisk and Lexar Media. He also claimed Samba 2.2.x is poor when transferring large (> 1 GB) files and 3.x isn't much better.

Wes said his appliances keep three images on disk, one as primary, one as backup, and one as a read-only failsafe/panic boot partition. His solution makes extensive uses of vnodes and stores configuration files in a PostgreSQL database. The appliance operates in degraded mode when something fails, offering enough information to perform troubleshooting.

His appliance does IP configuration by listening for a packet sent by an administrative console application using a source IP owned by St. Bernard. Connecting to the device's serial port launches a configuration wizard, not a shell. Only through working with St. Bernard tech support could one access a console of any kind.

Their product makes three sorts of updates: (1) subscriptions update application data changes; (2) patches update application bugs and security flaws; and (3) releases upgrade the entire system image, including the kernel. To retrieve updates, the appliances poll servers owned by St. Bernard on a daily basis.
Ryan McBride from the OpenBSD team gave the next talk I attended. He discussed Pf, the OpenBSD packet filter. He gave a nifty demo of firewall failover with Pfsync and CARP. A cool aspect of CARP (Common Address Redundancy Protocol) is its ability to work on devices other than firewalls. During the talk, Theo said a university using CARP to offer Samba via four servers. Ryan and Theo mentioned spamd, a sort of La Brea Tarpit to catch spammers.

Speaking of Theo, he gave the last talk I attended. It was an updated version of the exploit mitigation presented he gave at CanSecWest last month. His comment about Microsoft's adoption of a ProPolice-like system in Windows is flawed. Instead of setting the canary used to protect the stack at run-time, Microsoft computed the canary at compile-time. This means every copy of the same application has the same canary, making life easier for intruders. In Theo's words: "They completely missed the point!" Theo also commented that it's a bad idea to have a single 'nobody' user for multiple jailed processes. It's much better to give each jailed process its own unprivileged user, like _tcpdump, _apache, and so forth. That way, an intruder can't use ptrace between jails running under the same user ID and string together the means to escape.
Overall, I thought the conference was excellent. I intend to return next year. I paid for the affair myself and I feel I got my money's worth. I got to meet some really interesting people, including BSD Hacks author Dru Lavigne. Too bad Michael Lucas was absent -- I'm waiting for his book on NetBSD and Cisco Routers for the Desperate.

Over the course of the weekend, several people spoke to me about Sguil and monitoring in general. A few had questions about how to conduct monitoring when asymmetric routing is used. Asymmetric routing typically involves traffic being sent over one interface and route and returned over a different interface and route. There are two ways this seems to be used. First, at the client side, one might have a downlink served by a satellite feed with a phone line used for an uplink. Not only are such routes asymmetric, the latency and bandwidth is asymmetric too. This causes all sorts of problems for TCP, which generally assumes similar link performance for inbound and outbound traffic. A 1997 paper and slides explain the problems with such setups. Another paper can be found here.

Second, at the server side, one might have a server connected to two or more links for redundancy and performance issues. While one of the links may be a primary and hence have better performance, links of equal capability are often used.

For monitoring issues, administrators are more concerned with the second scenario at the server side. Some vendors, like Top Layer, sell products to bring the traffic together for monitoring purposes. Some also advocate per-flow rather than per-packet balancing on routing gear, if possible. I think there must be an open source solution to this, perhaps involving bridging promiscious interfaces and creating a virtual interface to monitor.

Wednesday I reported the publication of an exploit for the FTP service used by the Sasser worm. Now there's a new worm called Dabber exploiting the same vulnerability in Sasser's FTP service. Read each link for LURHQ's analysis of each worm.

If you've been seeing increased scans to ports 9898 and 5554 TCP, you'll know why after reading the advisories. Port 5554 TCP is the Sasser FTP server. Port 9898 is the Dabber back door.

Thursday, May 13, 2004

I found this Neowin.net article to be a good summary of future Microsoft OS release plans. The key points I noted are:

Microsoft is adopting a "4 year release schedule for Windows Server then that would place Windows Server Longhorn in 2007 (4 years from Server 2003)"

"[W]e'll be seeing Windows Client Longhorn in 2006"

"Microsoft will release an update every 2 years after an initial major release which refreshes the Operating System with add-ons, security fixes and new features. The next planned update is for Windows Server 2003 which is due to be released in 2005 and currently code-named R2."

A visit to Amazon.com reveals a page for my book. Amazon.com reports the publication date as 14 July 2004. This is a little earlier than I expected, but everything remains on schedule. Perhaps my publisher built in a little time for problems, and thankfully we haven't had any major difficulties yet. You may notice the cover is similar to Secure Architectures with OpenBSD and the second edition of Know Your Enemy. All three books are part of Addison-Wesley's new lineup of security books.

This installed the version of sudo in the Debian testing distribution. You can see the appropriate package here.

Testing, also known as 'sarge,' is a middleground between stable (aka 'woody') and unstable (aka 'sid'). There are packages for each of those as well. The stable package offers sudo 1.6.6-1.1, while the unstable package, 1.6.7p5-1, matches that installed by testing.

We've heard of intruders exploiting systems already infected by worms, but this is another way to take advantage of poorly deployed systems. A Romanian coder released sasserftpd.c recently. This code attacks the FTP server used by Sasser to propogate. The rogue Sasser FTP server listens on port 5554 TCP on versions a through d and port 1023 TCP on version e. The Romanian exploit attacks this FTP server.

I want to note a couple of helpful hints I stumbled across. First, I learned something new about the xterm program. I run FreeBSD on many systems and start X manually with 'startx'. One system has Windowmaker for a window manager. When I launch an xterm, the new instance doesn't read .profile. This means the prompt stays with the default, rather than changing to suit my needs. For example, my .profile has this entry to change the prompt:

PS1='`hostname -s`:$PWD$ '

This creates a prompt like this:

drury:/var/log$

Unfortunately, prior to today I manually sourced the .profile to change the prompt, using '. .profile' in the user's home directory.

While perusing this Unix for Advanced Users guide, I came across this article: Is my .login or .profile being used?. It explained that I needed to start xterm with the '-ls' option to specify it running as a login shell. In that case it will read the user's .profile. Here is the menu command I use to start xterm:

I also want to make note of a file that I use to set a resolution of 100x100 when X starts. My .xserverrc file looks like this:

exec /usr/X11R6/bin/X -dpi 100 -nolisten tcp

I can confirm this with xdpyinfo:

resolution: 100x100 dots per inch

A final usability issue involves batteries and FreeBSD laptops. This post to freebsd-mobile is part of a thread discussing differences between suspending and hibernating a laptop. I'm able to have my laptop suspend, thanks to the BIOS I believe. Read the posts for more information if interested.

Tuesday, May 11, 2004

I was looking to upgrade a few packages installed from Sunfreeware.com when I stumbled upon Blastwave.org. Blastwave.org is a "community software" (CSW) site which emulates the Debian apt-get system for installing Solaris packages.

Once you install the pkg-get package, you can install Solaris software as easily as this:

pkg-get install mutt

Pkg-get installs the dependencies and the desired package. The executable's home is /opt/csw/bin, unlike /usr/local/bin for packages installed from Sunfreeware.com.

Here is a comparison of how mutt, from Blastwave.org, and OpenSSH, from Sunfreeware, appear to pkginfo:

I'm interested in experimenting with IPv6 at some point. Since most of the operating systems I use in my lab have IPv6 stacks, I plan to run a native IPv6 VLAN internally. I'm also interested in connectivity to other IPv6-enabled sites.

This OpenBSD Journal article offers a few options for people wanting to use IPv6 across the IPv4 Internet. I plan to try one of these solutions and post my results here in the future.

Monday, May 10, 2004

Normally a change from a 2.0.5 to 2.0.6 release wouldn't be big news. That's not the case with Argus, however. 2.0.6 has been about a year in the making. Argus is the world's longest living open source session data collection program. It runs on most any UNIX distribution and appears in my book. Give it a try!

Monday, May 03, 2004

"Network Security Assessment (NSA) is the latest in a long line of vulnerability assessment / penetration testing books, stretching back to Maximum Security in 1997 and Hacking Exposed shortly thereafter. NSA is also the second major security title from O'Reilly this year, soon to be followed by Network Security Hacks. NSA is a good book with some new material to offer, but don't expect to find deep security insight in this or similar assessment books.

NSA begins with the almost obligatory reference to the king of assessment books, Hacking Exposed (HE), saying 'I leave listings of obscure techniques to behemoth 800-page "hacking" books.' I don't think some of the techniques covered in HE but not NSA are "obscure." Noticably lacking in NSA is coverage of dial-up techniques, wireless insecurities, Novell vulnerabilities, and attacking clients rather than servers. Should NSA receive a second edition, I expect to see the book expand closer to the 'behemoth' it seems to deride."

Saturday, May 01, 2004

"Ethereal Packet Sniffing is the first book in Jay Beale's new Open Source Security Series with Syngress. It's a great book to lead the way. Ethereal is full of helpful tips and clear discussions that benefit newbies and wizards alike.

I've been using Ethereal for around five years, and this book still taught me a few new tricks. The key to the new material is Ethereal's development, from 0.2 in July 1998 to 0.10.3 this year. (The book covers 0.10.0 which is far from being outdated.) The many improvements lend themselves to the sort of explanations found in Ethereal. For example, my favorite material involved filters. Although chs. 4 and 5 had minor overlap regarding this feature, I learned new ways to manipulate Ethereal's packet search and display capabilities."