Lars Strand's blog

29 Dec 2011

A couple of weeks ago, I held a Linux/Unix elementary course. One of the toughest concepts in that course are the concept of pipes and redirect.

I usually begin explaining pipe as "the output of one command becomes input to the next", and show by an example:

$ zcat pureftpd.log.gz | cut -f1 -d' ' | sort | uniq | wc -l

1259073

This command reads a ~550MB large compressed pureftpd logfile (from ftp.uio.no), and finds the number of unique visitors. Several commands are linked together by pipe, so the output of one command is input to the next.

However, I received and interesting question: "Which command use the longest time?"

There is no easy way to tell, we can just take an educated guess. However, we can use a handy little unix utility called "Pipe Viewer" to monitor and measure the data going through a pipe. Install from apt:

$ sudo apt-get install pv

Next, we craft our command above using pv. Since pv behave like cat with respect to input/output, we measure the throughput between each command:

$ zcat pureftpd.log.gz | pv -cN zcat | cut -f1 -d' ' | \

> pv -cN cut | sort | pv -cN sort | uniq | pv -cN uniq | \

> wc -l

As we see from the command, the command that had the slowest throughput was "uniq". Both cut and sort had an impressive 6-7MB/s throughput.

"The development of security architecture in fixed and mobile telephone systems"

One of the toughest tasks was to interpret the title and limit the scope of the lecture. I discussed with my supervisors and co-researchers and received several tips and relevant references. Then started two intense weeks with study and preparation.

I was satisfied with the disposition and result, and felt comfortable presenting the lecture.

3 Jun 2011

The end is nigh! ...or at least in sight! After four years, my PhD is nearing completion. The plan is to to deliver and defend my thesis sometimes this autumn. In these years, I've used several (free) tools that others might find useful doing the same, or similar, kind of work:

* Zotero - a Firefox-plugin that help me organize papers, citations, web-pages and other resources. All papers/notes are saved in "the cloud" for easy access across multiple terminals (...or Firefoxes). A great tool that enables you to tag, add notes, cross-link papers, add comments to/highlight text in PDFs, etc.

The Zotero Firefox plugin.

* Freemind - a mind-mapping tool. It was the best (open source) mind-mapping tool four years ago - and I believe it still is. Its written in java, so some slowness is included. It has great export functionality as shown in this HTML exported mind-map with the topic "FLOSS development".

* I write all my papers in Emacs with the AUCTeX LaTex mode and RefTex to create the TOC speedbar (see screenshot below). For notes I use xpad, and all revision control is done in Subversion (papers, documents, figures, presentations, code).

The desktop manager is Fluxbox with a bunch of key-bindings, bfpager, and gkrellm with bfm-plugin for system monitoring (I like to watch my system resources). Its the same setup I've used for years:

27 Feb 2011

It has been almost seven years since I last played with IPv6 (link, link). There has been lots of talk of IPv6 lately, and the company I work for has done some real-world IPv6 testing and deployment (check out Tore's IPv6 page: http://fud.no/ipv6/).

Since my local ISP has not deployed IPv6, and probably will not for a long time - I went looking for a "IPv6 tunnel broker". A tunnel broker enables you to tunnel IPv6 traffic over IPv4 to a IPv6 gateway (called "PoP").

So, which tunnel broker to choose from? Wikipedia give me a lot of choices. Several colleagues tipped me of SixXS -- their service is stable and professional, you get a /48 network and they have a PoP here in Oslo they said. Easy choice.

Be patient - this can take some time since it require manual (human) verification. It took me 8 hours from I applied to my account was accepted.

2) Once you get your username and password, log into the web-interface, and proceed to request a tunnel. Choose "Dynamic NAT-traversing" as type of tunnel. Choose your neares POP, and submit. This process is automated, and it took approx 30 minutes before my request was accepted.

3) When 2) is complete - you have ONE routable IPv6 address. Since we need more than that, we proceed to request a subnet from SixXS webpage. 30 minutes later, my subnet was allocated.

So I've received from SixXS the following information (not actual IPv6 addresses):A) IPv6 gateway at SixXS: 2001:FFFF:FFFF:FFF::1/64B) My local IPv6 gateway: 2001:FFFF:FFFF:FFF::2/64C) My allocated IPv6 subnet, which is routed to B): 2001:FFFF:EEEE::/48

Good, we have IPv6 connectivity, but only from one host. We want IPv6 on our whole home network.

6) Since we're given a /48 network, it enables us to have 65536 /64 networks. Which should suffice for most needs. Our network architecture would look like this when complete:

We configure our IPv6 gateway (vallhall-r6) to route IPv6 traffic for our local network. First we need to assign a IPv6 address on the actual interface facing our network. This will be our IPv6 gw address for our network:

9) We don't want to configure all our host manually, so we need some kind of auto-configure. While IPv6 have DHCPv6, (like IPv4's DHCP), a more elegant solution is to use "stateful address autoconfiguration".

The host configure themselves by acquiring a prefix from a local IPv6 router, and combined with the local MAC address, creates a IPv6 address (router advertisements + MAC address of interface = IPv6 address).

We use radvd for sending these "router advertisements". Install and configure:

10) Excellent! Radvd will now send IPv6 prefix periodically (or when requested by a new client) to your network. All IPv6 capable host should now automatically configure themselves with a IPv6 address.

11) A final warning and advice: ADD A FIREWALL on your IPv6 gateway! Since IPv6 don't use NAT, every hosts that use IPv6 is directly accessable from the Internet. This is a good thing, but it also expose all your IPv6 enabled hosts and their services.

29 Dec 2010

This post will explain how to automatically convert MS Word files (with images) to Mediawiki pages. Any filetype OpenOffice supports can be converted.

Short explanation: We use OpenOffice to convert the Word files to wiki-syntax, but some voodoo is needed to fetch and upload any images included in the Word-file (the "voodoo" is depicted yellow in the flowchart below):

More detailed explanation: The perl script word2mediawiki.pl take a Word file as input. After some rudimentary checks, it calls the python script DocumentConverter.py which calls OpenOffice to do the actual conversion. This is done twice; we convert to both .wiki and .xml files. Since the .wiki file DO NOT contain any images (it only adds empty [[Image:]] wiki-tags where the images are supposed to be), we convert to .xml that DO include images. Here the images are base64 encoded, so we parse the .xml file, fetch all base64-images, decode and save as ordinary images files. We re-write the .wiki file to update all empty [[Image:]] wiki-tags with the correct image file just decoded. Finally we upload the original Word file (for reference), all images and create a wiki-page based on the .wiki files using the pywikipediabot. Se example below.Prerequisite and install:

A) Linux - but may work on other platforms as well (not tested)
B) Install Perl and Python
C) Install the Python-UNO bridge. This enable Python to talk to the OpenOffice API (and do the conversion)

# apt-get install python-uno

D) Install OpenOffice. We run OpenOffice "headless", so X is not required.

F) Create a word2mediawiki directory. Download the word2mediawiki.pl script, and the PyODConverter script. Note! I've modified the PyODConverter script to support .wiki and .xml. You can download the modified version below:

Note 1: You might get a warning when the pywikipediabot tries to upload the images/.doc file or create the wiki-page. This can happen if the same image/.doc file already have been uploaded. If the a wiki-page with the same name already exists, the bot issue a warning and abort.

Note 2: When OpenOffice convert the .doc file, it might spew out a bunch of warning and/or error messages. These can be ignored. OpenOffice complains a lot.

Note 3: If the the script exits with a complain about "Can't connect to soffice on port" - just re-run the script. OpenOffice can be a little slow to start. (It will fork into the background the first time).

Note 4: The conversion is nowhere near perfect, and you might want to look over the wiki-page to ensure correct formatting.

Note 5: The filename of the Word file is used as name of the Wikipage. Example: "Testfile.doc" result in "mediawiki/index.php/Testfile"

27 Nov 2010

The winter has arrived early in Norway this year. Cold winds from Sibir have brought freezing temperatures all over the country. Its now approx -10C here in Oslo, and the temperatures are expected drop further next week. This is good news for my balcony server!

I use Munin to monitor the temperature sensors in the server. One of the CPU cores is showing a nice 6C. One of the disks records 5C. Hopefully the fans will start spinning again once the temperatures starts rising again in the spring...

Since I'm a little worried what will happen if the temperatures drops below 0C inside the disks, I started the folding client to generate some heat (it uses 100% CPU on all cores). The temperature immediately jumped ~5-10 degrees:

8 Nov 2010

I'm attending the IETF79 meeting here in Beijing. So far, it has been great. Meeting the people I've only read about, and participating in discussions. In particular, I'm looking forward to the kitten WG meeting (GSS-API authentication) and anything related to SIP, in particular sipcore.

But IETF's NOC have taken over the hotel network (both wired and wireless) and are currently bypassing the firewall. In cooperation with Tsinghua University, two 1Gbps links connect us to the CERNET (with backup to CSTNet).

A couple of test network has also been deployed. Including a IPv6-only network and a IPv6 network using NAT64.

These numbers are more than adequate for most of my multimedia needs. A 720p HD film encoded in MPEG2 needs around 20Mbits/s (~2.4MB/s). But since most films are encoded using MPEG4 (or similar) - a proper encoded 1080p movie will only require around 2-3MB/s.

23 Aug 2010

My balcony server finally died on my the other day. It has been running 24/7 for four years in all kinds of weather. I wasn't very surprised - in fact I've been waiting for it to happen. The motherboard had died. I've replaced the motherboard, and its back up. But for how long before a disk or something else fails?

I have backup of (mostly) everything here and there, but I would like to have everything on a separate NAS box. One of the most exciting NAS boxes on the market right now is something called Bubba|Two.

Bubba Two is produced by the Swedish company Excito. Its basically a small Linux server with a big disk. You can use the slick web-interface, or you can ssh into the NAS and treat it like an ordinary Linux-server. It is a LAMP server with SSH running Debian Etch. Samba, proftpd and Mediatomb (upnp) provide the box with file-server capabilities. It even have Squeezecenter installed if you have a Squeezebox (which I happen to have).

Its a ARM processor clocked at 333MHz with 256MB RAM and a 2TB disk. It uses ridiculous low amount of power (max 12W). There is no fan, so the only noise is from the HDD itself - which is barely audible.

Since the default apt-repositories are no longer working (Etch is too old), I change sources.list to:

deb http://archive.debian.org/debian/ etch main

I can now proceed to install NFS-server, Munin-node and Bind. A couple of minutes later, and its all running smoothly. Too easy.

6 Jun 2010

Some time ago, I needed an updated resume (and no, I could not "just send a Linkedin link"). I started editing it in OpenOffice, and was (again) struck by how terrible it is to edit, format and align a nice layout. I wanted to use something else - something like Latex.

I've been trying out some Latex resume templates, but none have been good enough (they often have terribly layout). I stumbled across the resume to Martin Michlmayr, and immediately spotted that it was created using Latex. It was nice, clean and looked professional - just what I was looking for. One email later, and he sent me the template he used. He has used and modified res, originally developed by someone else (Michael DeCorte in 1988 according to the header).

19 Apr 2010

My soon-to-be-wife and I are brewing delicious fortified wine. We do everything ourselves; we pick various berries, prepare and tap on bottles. This weekend we held a wine-tasting party, and I held a presentation of the whole wine-making-process.

6 Jan 2010

In my last post I used MRTG to monitor the network equipment. MRTG works great with SNMP, but it only present a graph per network port of the switch/router. So, unless you are the network guy, these graphs do not make much sense.

It would be nice to plug in the data from MRTG into a Network Weathermap of some sort. After searching around and trying different weathermaps, the choice fell on "PHP Network Weathermap". It is actively developed, has good documentation and works great for small/medium-sized networks (the map is hand crafted).

The weathermap can use sources from RRDtool which is the backend used by software like Munin, Cacti and MRTG (if enabled) or from the "original" MRTG (comments on the html-pages generated by MRTG). I'll be using the latter datasource - but I'll be sure to try this weathermap with Munin 1.4 another time.

The configuration for each "map" you create is a text-file. This text file can be created using a (simple) editor or manually hand-crafted. Once you know the (simple) syntax and have an overview of the network, a map is easy to create.

The weathermap comes with a (rudimentary) map editor, but I found it much easier to edit the configuration file myself while I consult the reference manual.
The config file for each map consist of three main parts:

Global section

Node section and

Link section (between the nodes)

3.1 Create a global section

In the global section we define the size of the map, title, and so forth. I also define some additional fonts and template for NODE and LINKS.

## PHP Weathermap config## Map: Company Name Core Network##

# The size of the map and titleWIDTH 1100HEIGHT 740HTMLSTYLE overlib

TITLE Company Name - Core Network WeathermapTITLEPOS 10 20TITLEFONT 14

# The output of the mapHTMLOUTPUTFILE company-core-network.htmlIMAGEOUTPUTFILE company-core-network.png

26 Nov 2009

If you work with any kind of networks, the chances are you've heard of, or even used, MRTG. There are not active development of MRTG today, but bugfix patches are still being added now and then.

So why are we still using it? MRTG just works. It stable and robust. And it does what it is supposed to do and nothing else. This makes MRTG still king of monitoring network equipment over SNMP. Well, that is, until Munin 1.4 is released in a couple of days. Munin 1.4 have (better) SNMP support - they are aiming at MRTG.

So until Munin 1.4 is released and stabilizes, I'll use MRTG in production. I've added some minor wrappers around MRTG so that adding and removing nodes gets dead easy. This way, other people can add/remove equipments without knowing too much about MRTG.

Every Linux distribution out there have MRTG in their repository. If yours don't, change distribution or compile MRTG yourself.

27 Aug 2009

Slackware 13 was released yesterday. It's the oldest currently maintained Linux distribution out there, and with good reason. It is clean, simple and without the "bells and whistles" that clobbers up so many other distributions.

Slackware was for myself (and for many of my friends and colleagues) my first encounter with Linux. I used Slackware for many years, and still do on occasions. Its a great distribution to really learn Linux and not learn a Linux distribution. I will say that if you really know Linux (and Slackware), you will know most Linux distributions as well.

You would think Slackware was abandoned for other more popular distributions nowadays, but there is still a large active Slackware community and user base out there. It is in fact, one of the most downloaded Linux distributions (in both MB and number of hits) for Norway's largest ftp-site:

16 Jun 2009

To quickly encrypt a file with a password of your choice you can use OpenSSL. OpenSSL supports a whole range of ciphers, including government approved encryption algorithms. The encryption algorithm AES is the only accepted open confidentiality algorithm here in Norway (read more here). AES is the new algorithm replacing DES. You can read all about AES and DES elsewhere.

Since symmetric block ciphers process one block of data at the time (AES uses a block length of 128 bits), it is important that we use CBC mode. CBC prevents repeating plaintext to create the same (repeating) ciphertext. Use option -p to have OpenSSL print out the salt, key and IV used:

To decrypt the file: Note! If you type in the wrong password, you'll get garbled output since there is no way to check if the password is correct.$ openssl enc -d -aes-256-cbc -in filename.odp.enc -out filename.odpenter aes-256-cbc decryption password:

And the decrypted file is found as filename.odp

For example: You can encrypt a file with a password of your choice. Send the file to the receiver, and then communicate to him over another secure communication channel what the password is (and that you've used "aes-256-cbc").

19 May 2009

I really would like to have an encrypted swap, tmp and home partition on my laptop. In case it gets stolen or if I should forget it somewhere, I can be sure that no-one would be able to read my private files. In this mini-howto I set my home partition using LVM, but using a regular partition should work just fine. This howto should also work, with minor modification, if you use another distribution than Ubuntu.

Note! Both the "server" and "alternate" Ubuntu ISO-images provide the option to encrypt your home directory (but in a different way using eCryptfs. Swap and /tmp are not encrypted). It might be an easier solution if you find this page too hard to follow. The difference? They are two different implementations. eCryptfs is file level encryption, LUKS is block device (/dev/sda3). Think of it like SSL vs. IPSec. Both have their advantages and drawbacks. Read more here and here

By using Linux Unified Key Setup (LUKS) setting up encrypted partition in Linux is done in no time.

Why /dev/urandom and not /dev/random? The latter blocks until it got enough entropy to continue, urandom don't. So if you use random instead urandom you might have to wait during boot until enough entropy is collected. (It does help to type your keyboard and move the mouse.) Use /dev/random if you're really paranoid.

Next, change your swap entry in /etc/fstab to this:

# cat /etc/fstab ... /dev/mapper/cryptoswap swap swap sw 0 0

For every time we boot, swap will be encrypted with a different encryption key.

To protect /tmp, we have two choices. 1) we can encrypt it like we did with swap or 2) we can create a ramdisk. The content of a ramdisk don't survive a reboot and /tmp rarely is used for any big files, its is also a good option. But, paranoid as we are, we choose option 1)

Now, since /tmp is encrypted with a new key every time, the filsystem must be created every time as well. The option "tmp" fixes that for us and calls mkfs before mount. Since it is created with filesystem ext2, we add in fstab:

Now you have to take a choice. A) You can enable the partition at boot time, but then the boot sequence is interrupted asking you for the LUKS password. B) If you want the partition automatically mounted when you login, skip to the next section.

Instead of manually typing in password, you can have the key stored externally - for instance on a usb-stick. Read more about that here.

When you now reboot, the boot process is interrupted asking you for the LUKS password. If you type it correctly, the home partition is mounted. When you now log in, you will have an encrypted home partition ready waiting for you.

Part IV: Automatically mount when logging in.

A more elegant solution would be to automatically mount the home partition the same time you log in. This require that you use the same password for login as for the encrypted partition. (Actually that is not entirely true. You may have the password stored on file somewhere. But in this howto, we assume you have the same password for both.)

Step 1: Remove home partition from /etc/fstab

If there is an entry to your (encrypted) home partition in /etc/fstab, remove it