We have been running ganeti cluster in our institution for more than a year now.
We did two cycles of machine upgrades during that time, and so far we where very
pleased with ability of this cloud platform. However, last week we had a problem
with our instances -- two of them got owned and started generating DoS service attack
to external resources. From our side it seemed at first like our upstream link is
over saturated, and we needed to way to figure out why it is.

At first, it seemed like this would be easy to do. Using dstat,i
I found that we are generating over 3 Gb/s traffic every few seconds to
outside world. We have 1 Gb/s upstream link, but our bonded interfaces
on ganeti nodes can handle 3 Gb/s of traffic, so for a start we where
saturating our own link.

But which instance did that? I had to run dstat on every node in our
cluster until I found two nodes which had instances which where overloading our link.
Using iftop I was able to get hostname and IP address of instances which
I wanted to shut down. However, this is where problems started.
We didn't have DNS entries for them, and although I had IP and mac address of
instances I didn't had easy way to figure our which instance has that mac.

Than I figured out that I can get mac from kvm itself, using ps.
Once I found instances it was easy to stop then and examine what happened with them.

But, this got me thinking. Every time I have a troubleshooting problem with ganeti,
I basically use more or less same command-line tools to figure out what is going on.
But I didn't have a tool which would display me some basic stats about instance,
but including mac addresses and network traffic (which in our configuration are tap
devices added to bridges).
So I wrote, gnt-info
which presents nice overview of your instances in ganeti cluster which you can
grep to drill-down into particular instance or host.

In my case, all four light sockets turn on or off with any button press
on remote which was a shame. When I opened remote and socket, I also had
bad surprise. My version didn't have any SPI eeprom, but just two chips,
ST F081 FB 445 in remote and ST ED08 AFB422 in light bulb (in picture hidden
below receiver board).

But, I already had acquired two sets so I wanted to see what I can do with
them. Since I couldn't read eeprom to figure out code, I decided to use
rtl-sdr to sniff radio signals and try to command them using cheap 315 MHz
Arduino module.

I used gqrx to sniff radio signals and I was not pleased. Remote drifted all over
the place mostly around 316 MHz and it was some trial and error to capture
signals which are generated when buttons are pressed. However, I have
verified that it's sending same signal multiple times no matter which
keys I press (which would explain why four pins on remote are soldered
together).

After a while I had two traces (since I have two sets of light sockets) and
could decode binary data which is sent from following picture:

How I knew that one set is transmitting
1000100110110000000000010 and another one 1011001001011111000000010.
From looking into timing in audacity, it seemed that each bit is encoded in short-long or
long-short sequence where short one is about third of long one, and one bit
is about 1200 ms. I cheated here a little and stuck scope into scope into
transmit trace on remote to verify length of pulses just to be sure.

So, I compiled it, uploaded to Arduino and... nothing happens.
Back to the drawing board, I guess.

When I was looking into gqrx I could see that signal is sent as long as
I'm holding button up to 10 seconds. From experience before I know that
this cheap receivers need some tome to tune into frequency so next logical
step was to send same signal multiple times. And guess what, when I sent same
singal twice with 2000 ms delay between them everything started to work.

Again somewhat. Light socket in far corner of hall seemed to have problems
receiving signal which would put two light socket in hall in opposite state:
one would be on and another would be off. This was fun, and could be fixed
with simple antenna on Arduino module (since currently I don't have any) but
I will conclude that your IoT device should send different codes for on
and off state so something like this won't happen to you.

Then I got carried away and added commands to change all parameters to
experiment how sensitive receiver is. You can find full code at
http://git.rot13.org/?p=Arduino;a=blob;f=light_sockets/light_sockets.ino
With this experiments I found out that you don't have to be precise with timings
(so my oscilloscope step was really not needed). Receiver works with 500 ms low
and 1100 ms high (for total of 1600 ms per bit) on high end, down to
200 ms for low and 800 ms for high (for total of 1000 ms per bit).

I suspect that chips are some kind of 26 bit remote encoders/decoders but I
can't find any trace of datasheet on Internet. This is a shame, because I
suspect that it's possible to program light sockets to respond to any code
and in theory address each of them individually (which was my goal in
beginning). However poor construction quality, and same code for on and off
state (combined with poor reception) makes me wonder if this project is
worth additional time.

I must confess that I'm pack rat. When I see old printer, something inside my head tries to figure out what I can do with all parts inside it instead of passing it to land-fill. However, I'm sysadmin and software guy, so JTAGs and programming is more up my wally than hardware. However, I decided to figure out how to drive one of servos using Arduino and this is my journey through this experience.

So I started with printer disassembly and got one stepper motor and some gears on it. It is Mitsumi M42SP-6TE. It has four wires and I couldn't find any data sheet about it. So what do I do now?

First some educated guesses.I assumed that it's 12V servo. This was somewhat influenced by examining similar Mitsumi MP42SP-6NK motor which have rating of 12V or 24V. Using unimer and taking ohm measurement between wires I confirmed that it has 10 Ω between coils which means it's bipolar, having two different coils which had both to be driven at the same time.

To connect it to Arduino, I acquired some time ago clone of Adafruit motor shield. When you buy cheap clones you expect some problems, and mine was fact that screw terminals on board weren't cut flash with board, so I had to use flat cutters and shorten them to prevent motor power from shorting with ICSP header on Arduino and USB connector on Uno. I also used red electrical tape and put it on USB connector just to be safe(r).

I also needed to add power jumper (white on picture) to provide power from Arduino (which in turn is powered by 12V 1A adapter). However, in this configuration L293D H-bridge becomes very hot to touch, so for testing I modified StepperTest example to provide me with serial control and powered Arduino from USB port (from which it draws 0.42 A and stepper still works with 5V supply which makes my 12 V assumption somewhat questionable). This enabled me to deduce that this stepper is also 7.5° which takes 48 steps to do full turn (small red dot on stepper gear helped to verify this). I also verified that top gear has 13:1 ratio to stepper motor making gear mechanism useful for smaller movements and better tork.

I hope this blog post will motive you to take old printers, scanners, faxes and similar devices apart and take useful parts out if it. Re-using boards for driving steppers is also very interesting, but this particular printer didn't come with power supply (and it has strange connector) and driver chip on it doesn't have any publicly available info, so this will have to wait some other printer which will decide to give up it's parts for my next project...

Last week I had pleasure of attending FSec 2014, annual security conference. Just like last year, I had hardware presentation, this time about reverse engineering NComputing CPLD dongle. You can find it on http://bit.ly/fsec2014-jtag or embedded below.

I had great time at conference, but I'm somewhat wondering did audience got something from my lecture. It was very interesting for me to figure out JTAG pinout on this board, and connect it to various JTAG programmers (all with their's good and bad sides) and I noticed that there are not any introductory text on the web how to approach this problem for the first time. So, I decided to present this topic in hope that this will motivate other people to take a hack at some board which would otherwise end up on e-waste of even worse, land-fill. And, who can resist call of free hardware which you can re-purpose? :-)

I have been using Hantek DSO-2090 USB oscilloscope for more than half a year now. While scope purist will say that usb oscilloscopes are not good enough for serious use for my use it's quite sufficient. However, this weekend, I was reverse engineering CPLD with R2R digital to analog converter, and I needed to figure out which steps are produced by turning pins on CPLD on or off. Sure, I can use multi-meter to do this, but if I already have oscilloscope it's much more powerful tool for task like this.

Every year, our three day annual DORS/CLUC 2014 conference is happening. This year, the dates shifted a few weeks later, which resulted in less students showing up because of exams, so it was a somewhat different experience than years before. For few years now we are not at the University of Zagreb, FER location so it also changed conference a bit. Having said that, even after move from FER, we still had a bus of students from my own faculty FOI in Varaždin, and they where missing this year.

It was still full conference in new (2nd floor, not ideal for breaks in fresh air which is a must to stay for 11 hours each day, mind you) location at Croatian Chamber of Economy
new and nice conference hall with wifi which was stable but didn't allow UDP traffic. Both mosh and n2n didn't work for me.

It was also in very different format. I would love to know did it worked for people or not. Instead of charging for workshops, they where included in conference price, and as every year, it you where interested in topic, nobody will turn you away from workshop because of space :-) This also meant that workshops are three hours slots at the end of the day after 7 hours of lectures. When conference started, we where afraid how will we accommodate all that people at workshops, but sense prevailed and about 20 or so people stayed for workshop each day.

Parallella and Epiphany 16 core mesh CPU

I had 5-minute lightning talk about Parallella, and hopefully managed to explain, that there is now interesting dual-core ARM, with interesting DSP-like capabilities backed by OpenCL and FPGA. This is unique combination of processing power, and it would be interesting to see which part of this machine can run OpenVPN encryption best for example, because it has 1Gbit/s ethernet interface.

ZFS workshop, updated to 0.6.3

ZFS on Linux had a 0.6.3 release just in time, and I presented two and half hour long workshop about ZFS for which 10-20 people stayed, after 7 ours of presentations. I somewhat field to show enough in command-line, I'm afraid, because I was typing too little. I did managed to show what will you get if you re-purpose several year old hardware for ZFS storage. Something along lines of 2004 year hardware with 8 SCSI disks.

I managed to create raid-10 like setup, but with all benefits of ZFS, fill it up and scrub it during workshop.

I think it might be good idea to pxeboot this machine on demand (for long-term archival storage) and copy snapshots to it on weekly basis for example. Think of it as tape alternative (quite small, 300G) but with rather fast random IO. Idea was to use this setup for ganeti-backup target, but dump format of ext file-system forced us to use zfs volumes to restore backup on other RAIDZ1 4*1.5T SATA pool, and it was very slow.
In current state, it can receive zfs snapshots at 30-40 MB/s and it's using single core for ssh, which is bottleneck. More benchmarks have to be done on this machine to see weather it's worth electricity it's using...

Ganeti - our own cloud

Another interesting part of infrastructure work last year for me was with Luka Blašković. We migrated all servers from faculty and library to two Ganeti groups. We are running cluster of reasonable size (10+ nodes, 70+ instances). Everything we did is done from legacy hardware which is now much better utilized. Some machines where never backuped and firmware upgraded so it was first time for them to have this kind of maintenance in last 10 years. Now we can move VM instances to another machine, and we are much more confident that services will stay running via live migration for scheduled maintenance or restart in case of hardware failure.

For workshop, we decided to chew a bit more than we can swallow. We spun up KVM images on our ganeti cluster and went through installation of workshop ganeti on them and joining them to new cluster. This went fairly well, but then we started configuring xen to spawn new instances (ganeti kvm with ganeti xen on top of it) we had some problems with memory limits which we managed to fix before end of workshop.
In our defense, we really believe that workshop was more interesting this way, probably because people didn't want to leave (few brave ones which where with us all the way to the end, that is). When you try to deploy something as complex as Ganeti you will run into some problems, so seeing troubleshooting methods used is usually as helpful as solution itself.

All in all, it was interesting and very involved three days. Hope to see you all again next year.

After 18 months Parallella kickstarter project delivered and I got the device in my hands. To be honest, I was prepared to write off $100 for it, but decided to support the project because I believe that we should have alternative architectures developed and Epiphany had such a goal.

As you can see on the picture, I got parallella board and heatsink for FPGA in nice box together with the pack slip. Heatsink is recent addition because the FPGA get very hot. However, it's not enough because you will also need some air flow over it to ensure stable operation. And 5V 2A power supply. So, I decided to do some research before the first power-on because burning board on the first try is not a good option.

Here is where Parallella forums came in very handy. It's full of very supportive community, and to learn how to use your board it's better place than the official documentation (and more up-to-date). On it you can learn that there are jumpers on the board to provide 5V for fan, and various other hints about the platform including ability to power the board over USB connector which proved helpful since I could use a 2A Nexus power supply.

Official image for Parallella is based on Ubuntu (which I don't like much, it even doesn't move devtmpfs by default), so I opted to install the unsupported Debian installation and try to lower power usage by disabling HDMI support since I'm not using it. And thanks to helpful parallella community and the forum post about with alternative parallella bitstreams and device tree I was successful in that task lowering power draw to ~0.75 A in idle mode and ~0.86 A while testing with aobench from parallella-examples. CPU load alone (two arm cores) seem to consume ~0.81 A. For comparison, HDMI bitstream consumes ~1.03 A in idle and ~1.19 A under load. All values are maximal ones which I measured using USB charger doctor, so they might not be the most precise.

To cool the device, I have salvaged small fan from an old disk drawer and attached it to the board using zip ties.

Power is supplied from a USB port on PC (for now), but the next logical step is to connect it to the jumpers on board and print the case for it on 3D printer. This involves mocking up with 3D software to the design case, so it might take some time. However, so far I'm very happy with my new toy.

Few days ago, I noticed odd problem with koha-common package. It depends on mysql-client which on squeeze tries to install version 5.1 which conflicts with my installation which uses Percona MySQL build. How can we fix this?

As it turns out, it rather easy. I will just create fake package which will provide mysql-client and in turn depend on percona-server-client using something like:

Finally I decided to upgrade my wireless network to 802.11n, and to do so I picked up cheap TP-Link TL-WR740N and decided to install OpenVPN, n2n and munin node on it. This is where the problems started because simple opkg install openvpn filled up whole file-system. Instead of declaring fail on this front, I decided to ask a friend how to make this work...

Reason for this upgrade was change in my router provided by ADSL provider. I didn't have any administration privileges on it, and it was only 802.11g device, so my previous configuration with igel which provided pppoe wasn't possible any more (since I can't turn ADSL router into bridge mode). So I decided to scrap igel and move openvpn and n2n to TP-Link instead (which will also help with head dissipation on my closet which hosts all those devices).

Since router has just 4MiB of flash storage, installing large packages is not solution for this platform. However, all is not lost, and there is alternative way to make this work. Trick is in way how OpenWRT uses flash storage. Image which you download from internet contains squashfs (which is compressed) that enable really efficient usage of storage on router itself. All additional packages are installed into overlay file-system, which doesn't support compression so you will fill root file-system really quick. However, there is solution. OpenWrt project provides Image Builder which enables you to select packages which are included in base installation, and thus ends up in squash file-system nicely reducing need for flash storage. Even better, you can also exclude packages which you are not going to use. However, to make this really useful you also have to provide files directory which contains modifications needed to make your specific router configuration work (like IP addresses, OpenVPN keys, n2n keys and similar modification).

First, I downloaded OpenWrt Barrier Breaker (Bleeding Edge Snapshots) and created files directory in which I will create files which are specific for my setup. For a first build (to make sure that it works I just copied /etc/config/network into it and rebuild image with

I didn't need dnsmasq (because ADSL modem will provide DHCP service for my network) and along the same lines, I also excluded ppp and nat but added openssl, n2n and muninlite (which is munin node written in C).
After rebuild, I copied created image to router and started upgrade with

Than I hold my breath and after re-flashing router it rebooted and connected to my network. So far, so good. Now I had all required packages installed, so I started configuring packages to my specific need. In the end, I had following configuration files which I copied back to my files folder

As you know by now, few months ago I built 433 MHz control of power sockets using rc-switch. Since then, I somehow lost remote control for it so I decided to have more permanent solution than bunch of wires. This time around I also used USBee AX Pro clone to check voltage levels which meant that I first had to make it work on Linux.

You can see final result on picture included in this post. It was mixed experience which included few surprises which might be useful to other people doing hacking with Raspberry Pi, so I decided to write this post. As I'm somewhat cheap, I didn't want to buy Adafruit Pi Cobbler but instead went with Raspberry Pi GPIO adapter board module which has additional benefit of routing 5V and 3.3V to power rails on breadboard. I already had 26 pin cable, but if you don't have one, make sure you get that also. Breakout board also have markings for wiringPi (as opposed to one of other variants of GPIO pins on pi). When breakout board arrived I noticed something interesting: it requires power pins which are aligned with holes on main board. Examine picture below to see difference.

As you can see on top 400 point breadboard, power pins are in the middle pins of main area which means that you won't be able to plug in breakout board into power rails. I was fortunate enough that another 800 point breadboard had power pins aligned with main area, but none of smaller 400 point breadboards I have are correctly aligned. Picture on seller's site does show 400 point breadboard, but adapter board shows V2.0 while mine is V2.2, but judging from picture 400 point breadboards with aligned pins do exist.

This time around, I also received USB Oscilloscope and logic analyzer which includes two analog ports which is clone of USBee AX Pro so I decided to see signals which receiver generated using it and picture is included below.

As you can see, I was forced to use Windows XP in virtual machine to make it work. Although there is free software support for logic analyzer part of it in sigrok developers aren't really excited with USBee AX Pro clones so support for analog channels is missing. There is code in ax branch, but I haven't had a chance to try it yet. For few months I thought that it doesn't work at all, mostly because I was trying to make it work in kvm and VirtualBox. Finally, a friend suggested to try vmware, and it worked without a problem. Since there is open source implementation of USBee SX protocol, I think it's quite possible to extend it to support analog ports on AX Pro, but I haven't had time to do this yet. Have in mind that buying USBee Test Pod clones is not something CWAV condones, and newer versions of software from their's web site will re-flash the device (original is read only) which will make it unusable. DX has a long thread with dead links to Cypress site how to fix this, but easiest solution I found so far was to go to site of another clone called ESLA201A
and download fix_esla201.rar which includes everything you need to re-flash USB ID back to USBee AX Pro in single rar archive.

When you are trying to configure touch screen on Linux machine, internet offers examples Xorg.conf configuration but without explanation were numbers in it came from. If you have different touch screen you might be out of luck or guess what to do. In this post, I will try to explain how to examine your device using evtest and try out settings using xinput without restarting X server or installing any drivers other than built-in evdev.

We have a couple of 3M MicroTouch M150 touch screens which are VGA monitors (1024*768 resolution) with USB touchscreen interface which is reported as:

A bit of googling later, I found out that there are two different drivers for microtouch devices, but both of them support serial devices only. Not giving up that easily I decided to see what xinput reports about it (without any additional drivers installed!):

This seems like a good news, but when I tried to use it, it seemed that cursor would move only in middle of screen (with X axis swapped) so I wasn't very happy about it. Examining properties of device in more detail revealed that it has property to swap axes and calibrate them, but what to write into those values?

First task was was to flip x axes to make it move left-right instead of right-left. This can be acomplised using following command:

dpavlin@t42:~$ xinput set-prop 9 267 1 0

Parameters are device id, property id, X axis swap and Y axis swap. If you don't know how many parameters property takes, just put one, try it out and if it returns errors, keep adding parameters until it suceeds.

Next, I needed to calibrate screen to track my finger moving over surface.
This is where evtest comes into play. It's low level utility which enables you to see input events before they are passwd to Xorg server. You will have to run it as root as follows:

After a few touches I had coordinates which where something like this:

14046,2722

2380,2986

7994,7819

13743,13616

2624,13545

Strangly it seems that origin is top-right corner, but we shouldn't care much about it beacuse we can specify them using following command (after rounding them a bit):

dpavlin@t42:~$ xinput set-prop 9 268 2380 14000 2800 13500

Trying it out on screen proved that it now works as expected. Let's call this success and remember that current Xorg knows a lot of tricks itself (recognising USB touch devices is one of them).

As a side note you don't really need to use evtest to get device position. Using xinput list id syntax displays you more-or-less same information, including last point which you touched on device as seen below:

Last week I had workshop on CARNet User Conference on topic of ZFS on Linux. In it, I tried to describe my experience with it, from zfs-fuse days up until today. There are quite a few useful bits in this presentation, covering everything from history and basic ZFS concepts up to smart hints and some downsides. I hope it was useful to people who attended it and that you, dear reader, will find something useful in presentation which is embedded below.

One of simplest home automation projects is to control power sockets. Back in the old days, this involved relays and parallel port, but nowadays you can buy ready made power sockets with remote control which work on 433 MHz IMS band. So next logical step is to replace remote control with computer controlled 433 MHz module. In this post, I will explain how to do it using Arduino or Raspberry Pi.

Power sockets with 433 MHz remote control

For this project I decided to try out two different types of power sockets available at our local hardware shop.

As you can see they are quite different, but any of then will work fine. On the other hand, big bulky square ones have much more useful indicator on then which lights up then they are enabled, while smaller round ones have light which just denotes that they are plugged into electricity but doesn't show status of relay inside of them.

433 MHz modules

If you want to communicate with these devices you will need some kind of radio modems. For this project, I decided to use cheap 433 MHz radio modules which come in pair: one of them is sender while another is receiver. On picture you can see modules with attached antenna.

I can't stress enough need for antenna. Although modules will work without it, they will have a really poor range of just few centimeters. Antenna should be 172 mm (1/4 of wavelength), and according to my friend who knows something about this things, it's equally bad to have longer or shorter antenna than that. I added 3 mm more for solder joint and it did improve reception and sending range dramatically. You can use any wire for antenna, in my case, I used wire used for wrapping power cords which was long enough to make two antennas.

Arduino

As a first step, I decided to hook them to Arduino Uno to see some waveforms and test rc-switch library. Sure enough it worked on first try. I added project to my ~/Arduino/libraries/RCSwitch directory and used ReceiveDemo_Advanced example to capture signals.
Another example, SendDemo enabled me to send signals back to power sockets and turn them on or off. Success on the first try.

With a little bit of tweaking I created RF433_Sockets.ino which allows receiving signals, turning sockets on or off or sending raw binary data to test protocols.

Raspberry Pi

Having verified that signal receiver is generating 3.3V signals (up to 3.6V is OK and mine is 3.46V - electronics isn't exact it seems :-), I decided to hook it up to Raspberry Pi so I will have Internet connected control over my power sockets.

Following link on rc-switch site to port for Raspberry Pi didn't end up all that well. This version of code doesn't support receiving of data, and sending code generates segfaults. However, after a little bit of searching I found blog post about Adding 433 to your Raspberry Pi from NinjaBlocks which inclues link to github repository with code which works fine.

Both versions of code use WiringPi library which is nice way to port Arduino code to Raspberry Pi. When sending signals to 5V devices 3.3V Raspberry Pi GPIO will be usually fine, but when receiving singals, make sure that they are 3.3V or you will fry your Raspberry Pi.

Less than $10 and few hours of work...

Having said all that, it's really easy to create your own home automation system. So I don't see a reason not to do so...

This event has special place in my calender since I'm alumni and it's always nice to meet again friends and professors and visit again place where I started my Unix administration back in 1995. Thanks to everybody for nice three days in beautiful Varaždin.

A few weeks ago my WRT54GL decided to die. This was quite fortunate because I wanted to upgrade my always on machine to something more powerful and install Debian on it. I had ADSL modem with wifi, so I didn't need wifi in my new box. Fortunately, I got a Igel 3/4 thin client with VIA Eden chipset, 256Mb of RAM and 128 Mb of CF storage so I decided to move by main gateway functions to it.

I had following requirements:

pppoe to connect to by ISP

dnsmasq to provide DHCP for my LAN

iptables will provide NAT

openvpn and n2n to provide VPN

motion so I can see what my dog is doing when I'm not home

As a first installation I started with Thinkpad T22 (since Igel was still on the way). All went quite well (after figuring out that I can't boot T22 from USB) but it took 2GB of disk space. That won't fit on 128MB CF card, so I acquired 2GB CF card. Still, even with it, the storage will be tight, so I decided to use btrfs with compression. And this is where the real story starts...

As a first step, I plugged 2GB CF card into USB adapter on my desktop (Igel can't boot from USB), created btrfs filesystem and mounted it using compress=zlib,ssd. After coping files over size of installation dropped from 1.9GB to just 840MB. This card will be adequate choice for my gateway. I was also toying with idea of moving my apt-cache-ng to this machine, so I wanted more space, and decided to also plug additional 8GB USB keychain for swap and cache storage. But, since I have two devices wouldn't it be better to create btrfs raid1 on it so I can survive failure of CF or USB (ignoring cache and swap)?

Next I plugged CF and USB into Igel and booted it. First boot was slow, but what can you expect from 600 MHz VIA Eden? Here is hdparm speed of CF (sda) and USB (sdb):

This is where the problems started. I was running Debian wheezy with 3.2 kernel and rebalance filters are introduced in kernel 3.3. So I upgraded to sid with 3.10 kernel to see weather it will fix at least some btrfs issues. As we will see it really didn't.
I got reports about corrupt blocks and after a while I decided to unplug CF and USB, move then over to desktop and create mirror filesystem there.

This worked (and didn't report any errors after scrubing) so I was somewhat confident that my data is safe. This time around, I decided to use noatime,compress=lzo,ssd_spread mount options to gain some speed. Filesystem did grow to 1.2GB (50% increase since I was using lzo instead of zlib for compression) but it was still acceptable. Next, I returned storage to Igel and tried to boot it. Another surprise: it couldn't mount root file system. When I got initrd shell, I could indeed see that I don't have /dev/disk/by-uuid/ nodes for root mostly because it wasn't mounted. However, manual mount from /dev/sda1 didn't work either. However, manual mount from /dev/sdb1 did work. After searching a web for reason it seems that Debian's initrd doesn't issue btrfs device scan on boot so multi-device btrfs filesystems don't get correctly recognized. Why does it work from second disk in mirror is still a mystery to me. I enabled GRUB_DISABLE_LINUX_UUID=true in /etc/default/grub and moved on...

However, my problems weren't over just yet. After running update-grub on booted filesystem Igel was again unbootable. It was reporting strange errors in part because grub-probe got both devices:

root@igel:~# grub-probe --target=device /
/dev/sda1
/dev/sdb1

This was easy to fix. I modified /usr/sbin/grub-mkconfig to report just /dev/sdb1 so it will be bootable:

GRUB_DEVICE="`${grub_probe} --target=device / | tail -1`"

The story would end here if I didn't try again to run btrfs balance start /. This time around, it didn't report any corrupt blocks, but it did oops kernel with out of memory backtrace. Lesson learned, 256MB RAM is too small for 2GB btrfs filesystem balance. Live and learn. Problem now was that balance restarted when I booted system. So as solution, I unplugged USB which dropped me into initrd shell where I could issue btrfs balance pause / followed by btrfs balance cancel / (cancel alone wouldn't do the trick).

So, what can I conclude about current state of btrfs? It does work (with some hand-holding) but it also has rough edges. But, if you really have flash based storage and need compression it is a valid choice.

Upload Files

Click "Browse" to find the file you want to upload. When you click "Upload file" your file will be uploaded and added to the list of attachments for this page.

Maximum file size: 50MB

Add a link to the attachment at the top of the page? Images will appear in the page.