Friday, April 27, 2012

It's been awhile again since my last post, but as always there were many issues to deal with along the way. I would have to start with the technical changes (surprise, surprise). The diskless OpenBSD IPv6 router that I was blogging about in my previous post is obsolete now, replaced by a diskless FreeBSD version because as we saw when certain circumstances were present the OpenBSD would just crash (kernel coredump upon NFS failure for example) which is not tolerable so I rebooted it under FreeBSD (since I have a few OSes ready for netboot) and all the problems are gone. You can see some basic stats about our IPv6 connectivity and usage at http://IPv6.onlineDirect.bg. We also added new project to the mirrored ones called Sagemath and you can access it at http://Sagemath.Igor.onlineDirect.bg as usual. Continuing with the tradition of providing new services for our clients I decided to run our own Counter-Strike servers since most of the public ones are extremely crappy, I've added the minimum add-ons so the game can be enjoyable and also optimized the server a lot so the latency should be minimal. You can play CS Classic at cs.onlinedirect.bg:27015 and CS Italy2 Respawn at cs.onlinedirect.bg:27016.
Another new development would be the new advanced DPI engine for my router which extremely accurately is able to differentiate between numerous protocols and if configured apply QoS based on some portion (or portions for that matter, it can be composed of few different parts) of the packet. We are using it to throttle "bad" traffic in favor of the "normal" user traffic so the service we provide would be optimal all the time. Currently I've implemented it as hybrid system doing the analysis in user-space and the matching/QoS in my router in kernel-space. I had to modify heavily the tree structures that we use because of the space requirements (there can be millions of concurrent matches per single machine) so they are as compact as possible. My tests show that currently for a little over 1 Mpps (that's our international traffic) about 4 - 6 % of the CPU is used for analysis (of course if a stream is analyzed it is cached etc) with superb accuracy and the improvement of our service is very noticeable. I've also added new user-space tools so one can track how the DPI engine works and see some statistics about it in real-time. Oh and the router now supports new load-balancing algorithm which can use up to 10 x 1/10 gbps network adapters in a single trunk.
In the spirit of improvements I also decided to finally fix a few little issues with my BGPv4 implementation and in the process I also developed new peer balancing algorithm (which is very simplified version of the one I was writing paper about "Using combinatorics to load-balance Internet traffic") and results have jaw dropping effect. The Internet traffic got balanced evenly between all of our peers with difference margin of at worst 15 % (which means extremely well). I've added various triggers to the BGPv4 software which go off upon changes in the state of the peer and the balancing operation is fully automated. Another huge improvement in the BGPv4 software is the better use of tree structures to minimize CPU-time and now when a peer "dies" or is connected you can't see the BGP on the "top" process list for more than a second (even then taking 5 % of CPU time). I've added locking for the asynchronous events since I found a bug there which in very rare cases caused it to crash. All in all after all the changes I haven't had to log on to the machine that it's running on for any reason other than monitoring it.
Oh wait! There's even more! :-) There's some news about the IOlib I mentioned in some of my previous posts. I've been actively developing it to add cross-platform compatibility and the results are very satisfying as the current state is:
OSes tested: *BSD (kqueue / poll), Linux (epoll / poll), Solaris (/dev/poll, poll)
IPv4 TCP/UDP sockets
IPv6 TCP/UDP sockets
Unix domain sockets
FILE I/O
SSL sockets (using OpenSSL)
I've easily managed to make a daemon that sustained over 200 000 active connections transferring over 500 MB/s with very little effort (most of which went into optimizing kernel tunables). When I add all the protocols and functionality that I wish for such a library I will release it as open source (with no license - simply free).
On a personal note I got myself an SSD disk (OCZ Vertex 3) and I'm in heaven right now! This is by far the best upgrade one can make. As my CPU stayed idle because of my old slow HDD, now I can really feel the speed (I upgraded my PC to Sandy Bridge Core i5-2500 with 1600 MHz DDR3 memory). Boot time is little over 10 seconds without optimizing the operating system start-up (only removed few things from the bootloader), everything just pops up immediately. Since my work is connected with a lot of kernel recompilation the Linux kernel with default config from Debian compiled for about 10 minutes, bonnie++ showed 700 MB/s read speed and nearly 500 MB/s write speed when tested. The productivity went through the roof with this upgrade and working on the PC is a lot more pleasant :-)
But wait! There's more, I moved all my sources in a git repository that I setup on my storage. I'm pleasantly surprised by how well it works and how easy it is to develop software by using it. I've configured different SSH keys for various access types and also added active backup. In this spirit (it is a Bulgarian saying) I decided to switch to a more centralized identification, authorization and accounting system using OpenLDAP, Kerberos and the ldap pam module. I also played around with the Single Sign On idea using Kerberos, I even locked myself out of the machines one time so it's been fun :-) but I prefer more paranoid approaches to security.
Some good news about the ISP is that we won the project we had applied for and now let there be PON service :-) In my next post I'll divulge more about this one, there would be some really big and nice new toys to play with. I'm hoping for 10G NetFPGA dev boards!
It was about time I generate myself a PGP keypair :-)
My key id is: E01113B1
Keyserver: pgp.mit.edu
I also updated my CV and started using Dropbox (very convenient by the way) so you can find my CV in PDF format at http://dl.dropbox.com/u/75462401/CV.pdf

That's it for now! (since there're always a lot of ideas before writing and while writing they all disappear)

P.S. One interesting project that I'm working on is an ARC cache implementation for a VM subsystem :-)

P^2.S. The title is a quote from the cult movie "Spawn"

P^3.S. You can access my FreeBSD's loader.conf & sysctl.conf these are used on one of the most loaded servers in terms of jails, services & traffic. The machine is performing great (it has network and file i/o load). You can see info about the machine at http://Igor.onlineDirect.bg.

Monday, February 13, 2012

Hey guys, another chill morning after another blizzard and so another post comes :-)
It's been quite cold the last few weeks, some temperature records broken (and I was trying to figure out why a temperature record broken would be a good thing..) and half of the country is in despair.
First things first in January we went to the RIPE training in Skopje, Macedonia. The weather was in its infancy but still was able to snow so much as not to be able to see where the road ends and the field begins, the snow plows were not released to clean yet, so on one sharp turn we couldn't make it and went in the snow on the side of the road. It wasn't very bad, all of us were fine and the car seemed OK, the only problem was that it was floating freely as the tires didn't touch the ground :-) After waiting for an hour and a half some man with a truck pulled us out and we continued to Skopje. I have to say this was on the Bulgarian side of the road, on the Macedonian side the snow was cleaned very well and the roads were fine.
After we got to the hotel which had to be 4-star (I would give it at most 2 stars), it was like pulled from the 60s and not in the good way. The rooms were cold, there was very loud wind. Anyway the next day we started the training session at 9 am. The session itself was divided in various different topics, there was information about PI space, IPv4 exhaustion, the end game (after there are no IPv4 networks), IPv6 deployment technologies, LIR's involvement in RIPE NCC rules and discussion about some RIPE NCC rules, various interesting projects at RIPE NCC like Atlas.ripe.net (I'll discuss this one later), Labs.ripe.net, Stat.ripe.net and much more.
The session continued from 9 am to 17 pm with a small lunch break. The next day was all about IPv6 :-)
Afterwards we were on our way back to Sofia. On the next day of work I decided to ask for IPv6 allocation from RIPE NCC and we got our 2a00:87c0::/32 network. And all hell broke loose!
I wanted to start using it right away, so I called one of our international providers and arranged for another BGP session just for IPv6. So far so good, then I decided to route it with OpenBSD since we no longer have any OpenBSDs at the datacenter (after Igor many machines were taken out). Of course it had to be more interesting than that - I wanted to run the OpenBSD on a diskless machine. I have to be honest - on the Internet there are manuals and tutorials to run OpenBSD diskless either 1. from another OpenBSD or 2. to install OpenBSD diskless (which proved to be very easy).
But I had to run OpenBSD diskless from FreeBSD. The process is basically the same (as with running it from another OpenBSD) just the details of the implementation differ. I'll try to give very brief overview of what has to be done in order for OpenBSD to run diskless:
1. Get OpenBSD sources and recompile the kernel with:
"config bsd root on nfs swap on nfs"
2. You must have a configured RARP daemon because that's how it gets an IP address at boot.
3. You must have DHCP daemon for the pxeboot process mainly
4. You must have BOOTPARAM daemon because it gets its NFS parameters through it.
5. As 4 suggests you need NFS exports :-)
6. You MUST populate OpenBSD's /dev in the right way!
(This one could really make you sweat if you're not doing it from OpenBSD!)
Easiest way - install ksh and execute the MAKEDEV script (with small rearrangements of some command arguments) you'll be ok. Also a way to generate password for OpenBSD from the bsd.rd (the ramdisk install):
run httpd on the dhcpd, put mount_nfs there and then use OpenBSD's "ftp" to download it and mount the nfs, chroot to it and then use "encrypt -b 6 <password>" and useradd's -p argument to supply the encrypted password on the command line. You can boot bsd.rd directly (just with DHCP running for the pxeboot).
And after all this we have IPv6 dedicated server, the BGP session is running since. There are about 8000 routes right now :-) I also ran DNS64 implementation just for testing purposes. There are already some clients on IPv6 and we are routing it amongst client VLANs. It's a fresh look on the world I have to say, you read all about the protocols beforehand but the actual implementation and practical running still feels new and very different. I love having such huge address space, I could do any addressing scheme for the clients with VLANs/MAC addresses integrated in the IPv6 address or any combination of those. Oh and we have IPv6-only DNS server running as well.
A little sidenote: We also volunteered for RIPE NCC's Atlas probe and they approved us. We are waiting for the probe to arrive and we'll be able to assign it IPv4 and IPv6 addresses. I think to make it public one.
I also ordered Arduino Uno Rev.3 board with some sensors, resistors and a breadboard. I've been looking at various MCUs all weekend and wondered if I could make my own board with ATmel's Xmega MCU (it has AES/DES support), although I found software AES implementation for ATmega328. I also got permission to implement a project about RFID for our data center. I've ordered RFID reader and a couple of RFID tags which will be assigned to the people who have access. I'll connect it to an Arduino board with ethernet shield and will make a check-in system. I also decided to add few sensors for the data center (movement, temp. and humidity). I was thinking if we were to put more RFID devices at various points in the data center we could do triangulation as well, but I guess the same effect can be obtained by using appropriate sensors with single Arduino board. By the way I think to try and implement IPv6 on the ATmega328 and use it over the ethernet shield (although it's quite limited).
Looks like this is it for now! On the mathematics front I have some news but I'll wait some more before sharing :-)

P.S. For your sake don't be wondering where the title is from and go watch the movie straight away!

Sunday, January 15, 2012

Hey long time no see :-)
The song for this post will be: Rammstein - Fuhre Mich
I have forgotten about this little place under the net. Also I had a lot to do in the past months (6 I think since my last post). So I'm gonna start from that point on and see where that takes me.
We've moved back to Sofia, Bulgaria from UK/DE for now at least. My girlfriend is looking into the possibility for masters degree in the UK, so nothing is final yet :)
What bothers me the most is probably that I haven't released/fixed the software I was talking about way back just some minor additions/fixes. Although with the router I've made huge progress and I had the chance to play with 10G SFP+ interfaces at last, making them line-rate with millions of packets per second was a delight. Of course such generated traffic is far from the truth, it is still nice to see some new options and problems. Also I had the chance to migrate the master router (edge) of the ISP to the new xeons based on nehalem architecture and I can say without even optimizations for this particular architecture the results are staggering - about 10x better performance for the same load with the router (it is basically standing on 98 % idle while 800 MB/s / 1 Mpps are going through it and it shapes also!). Imagine now a better software version of that and with optimizations for that particular architecture, it will be a beast of different nature. Oh I had the pleasure of trying netmap for the purpose of getting the router out of kernel- and in user- space but I find the project still immature. I've reported few bugs and also asked about when some new features will be available (or should I write them myself), but the idea of me supporting another set of patches/software which is concurrently being developed - no thank you :) So I'll wait it out and probably use it in some future release.
By the way the uptime of the shapers is almost 2 years now :-)
There were also some changes about Igor, I've removed the FreeBSD FTP mirror (left only the CVSup mirror which is up-to-date and syncs every 3 hours) because it was taking over 900 GB and no one uses it, also there was no feedback from the mirrors mailing list of FreeBSD because the official Bulgarian mirror isn't working for a long time. I've added few new mirrors: official LibreOffice mirror(which is quite active by the way), PC-BSD official mirror and Calculate Linux official mirror. Also I'm still looking for projects to mirror so expect to see some new ones soon as well. Oh lets not forget about the official Bulgarian Freenode IRC server which is running from Igor - Hitchcock.Freenode.net.You can find me there under the nickname Raz- in #FreeBSD, #Math, #Crypto and #Linux-BG.
There should probably be a post about my phone adventures alone but I'll try to sum them up in a paragraph here. I decided to switch to something new after being 3 years with my iPhone (which is still working by the way) so I got myself an Android phone, actually the best hardware-wise Android phone I was able to find - Samsung Galaxy S II and then all hell broke loose. It was the beginning of my headaches, first my coverage went down considerably (and with bugs, the phone wouldn't recover network coverage, also it wasn't showing the wi-fi signal strength properly loosing it from time to time) after that the chaos that Android is having adware, viruses and so on in the Market. I had a GSM with anti-virus software just in case... Also the fact that it is linux based is tickling all the wrong places. The input lag which is considerably higher than iPhone/Windows Mobile/BlackBerry 9900. I really don't like the idea of the need for dual-core CPU and 1G of RAM so my phone to run slightly normal. That is against everything I believe in software-wise, as is Java and the likes. I tried the original firmware, then rooted the device and tried optimized firmwares - in the end my problems still persisted. I saw demos of Android ICS and the input lag is still there it is just more eye-candy and the menus are better arranged compared to the total chaos. So I decided to solve my problem in a more cardinal way, I got rid of the bloody disaster and got myself a nice BlackBerry 9900 (not QNX, but still very nice look and feel). A really well-built device which works flawlessly and has very nice features that help me with my work a lot but most of all doesn't irritate me. I've spared some details here so I'll leave the story here with a happy ending after all.
Some other news on the university front - it is possible to start doing a Ph.D. this year in BAS (Bulgarian Academy of Sciences). I hope it will be in abstract algebra or coding theory, we'll see there are still many variables :) I got an innovation certificate for the router and related software as well.
In the end of January I'll be attending RIPE's meeting in Skopje, Macedonia related to IPv4/IPv6 resource allocation and object manipulation in the RIPE database. After we had an audit from RIPE which went well given that we exchanged some 20 mails or something like that. I really have to start making graphs for the usage of each address pool (handling a /17 divided in so many pools ain't pleasant) if I'm to request new allocations from RIPE.
Also we are preparing a new massive project involving many new technologies where innovation is still to come and good ideas to be implemented, but I'll give more details on this some later time when all is ready :-)
My birthday went very well, friends came from different cities and it was very pleasant evening. I would like to thank everyone who could come and hope to see them next time too :)
On new year's eve we were in a friend's new apartment where we could see the fireworks from all of Sofia (very nice view) and it was amazing. All went very well, there was just one irritating person (some self-proclaimed zen- master) who said that mathematics limits you in your thinking (trying to re-create the phrase in English), I decided that talking to such a person is a waste of time so he was simply ignored by me all night :-)

P.P.S. Happy FreeBSD 9.0-RELEASE! Very nice new features and also it is dedicated to the memory of Dennis M. Ritchie:

The FreeBSD Project dedicates the FreeBSD 9.0-RELEASE to the memory of Dennis M. Ritchie, one of the founding fathers of the UNIX[tm] operating system. It is on the foundation laid by the work of visionaries like Dennis that software like the FreeBSD operating system came to be. The fact that his work of so many years ago continues to influence new design decisions to this very day speaks for the brilliant engineer that he was.