/dev/bloghttp://possiblelossofprecision.net
...why bore others needlesslyTue, 15 May 2018 13:25:25 +0000en-UShourly1https://wordpress.org/?v=4.9.8Enabling ‘reserved host RAM’ on VMWare Wokstation Playerhttp://possiblelossofprecision.net/?p=2476
http://possiblelossofprecision.net/?p=2476#respondTue, 15 May 2018 13:25:25 +0000http://possiblelossofprecision.net/?p=2476Read more »]]>VMware’s Workstation Player checks how much swap space is available before starting up any virtual machine. If the host’s available swap space isn’t at lest 50% of the VM’s memory it spits out a warning:

VMWare Workstation Player showing error message due to too little swap being available

Unfortunately the GUI does not offer an option to change this behavior and disable memory overcommitment. However this can be done by adding prefvmx.minVmMemPct = “100” to /etc/vmware/config:

[...]
prefvmx.minVmMemPct = "100"

Note that this option has to be set globally in /etc/vmware/config and does not work in a virtual machine’s *.vmx file or on a per-user basis in ~/.vmware/preferences.

]]>http://possiblelossofprecision.net/?feed=rss2&p=24760Mounting a VHD disk image filehttp://possiblelossofprecision.net/?p=2460
http://possiblelossofprecision.net/?p=2460#respondWed, 02 May 2018 05:30:24 +0000http://possiblelossofprecision.net/?p=2460On Linux qemu-nbd can be used to access disk images in different formats as if they were block devices.

]]>http://possiblelossofprecision.net/?feed=rss2&p=24520Building a custom Fedora image to run on Google Compute Enginehttp://possiblelossofprecision.net/?p=2438
http://possiblelossofprecision.net/?p=2438#respondSat, 31 Mar 2018 22:15:41 +0000http://possiblelossofprecision.net/?p=2438Read more »]]>Thankfully the Fedora Project provides cloud images that can be directly used or easily modified to run on almost any cloud environment. Though sometimes, it can be handy to build your own image.

0. Prerequisites

Fedora images in general and the cloud images especially are built using ImageFactory which itself uses Oz. The latest versions can always be found on koji. To recreate an image locally, there are a couple of tools needed:

2. Modifying Oz default config

ImageFactory will create a temporary virtual machine that’s used to install Fedora into a raw disk file. Depending on the package selection you might need more that 1GB of RAM which is the Oz default on Fedora. To modify the memory limit, change the memory parameter in the Oz configuration file

4. Creating a Google Compute Engine image

There is a GCE plugin for ImageFactory that allows you to easily package the created raw disk image:

imagefactory target_image --id <UUID> gce

Unfortunately, that plugin is not packaged for Fedora yet, so this needs to be done manually. Looking at the plugin source code, we’ll have to reproduce what the builder_did_create_target_image function does:

]]>http://possiblelossofprecision.net/?feed=rss2&p=24380Running Fedora 27 on Google Compute Enginehttp://possiblelossofprecision.net/?p=2433
http://possiblelossofprecision.net/?p=2433#commentsMon, 26 Mar 2018 20:03:09 +0000http://possiblelossofprecision.net/?p=2433Read more »]]>Usually Linux distributions with a long life cycle like RHEL (or its free derivative CentOS), Debian or SLES are the way to go for virtual machines in a cloud environment. But sometimes you need to be a little bit closer to upstream. Maybe because your applications relies on newer version of some packages that are not (easily) available on distributions with long term support or maybe because you need a feature that has just not yet made it to RHEL, Debian or SLES.
In those cases, Fedora is an interesting choice, since it’s probably the Linux distribution that’s closest to upstream and provides the most features that could be considered ‘bleeding edge’. Unfortunately there’s currently no publicly available Fedora image on the Google Cloud Platform. But not to worry, it’s quite easy to run Fedora 27 on GCE.

The Fedora Project provides a compressed raw disk image that can be used to spawn VMs on different platforms, e.g. GCE. To use it with the Google Compute Engine, the image has to be renamed and repackaged though:

Of course, you might want to choose a different machine-type or zone here.

Once the VM is booted (and assuming a metadata key value pair for the project provides a public ssh key) one can connect to the instance via:

gcloud compute ssh fedora@fedora27

]]>http://possiblelossofprecision.net/?feed=rss2&p=24332Allow blacklisted VMWare Workstation Player graphics driverhttp://possiblelossofprecision.net/?p=2464
http://possiblelossofprecision.net/?p=2464#respondTue, 27 Feb 2018 14:08:14 +0000http://possiblelossofprecision.net/?p=2464Read more »]]>For some reason, VMWare decided to blacklist some graphics drivers for their VMware Workstation Player. That includes the Mesa DRI drivers for most Intel IGPs, which results in unbearably slow graphic performance and potentially error messages such as “Hardware graphics acceleration is not available” or “No 3D support is available from the host” when starting a virtual machine

VMWare Workstation Player showing error message due to blacklisted driver

To enable hardware 3D acceleration for blacklisted drivers, the option mks.gl.allowBlacklistedDrivers needs to be enabled:

...
mks.gl.allowBlacklistedDrivers = TRUE

This can either be done globally in /etc/vmware/config, on a per-user basis in ~/.vmware/preferences or for each individual VM in the corresponding .vmx file.

]]>http://possiblelossofprecision.net/?feed=rss2&p=24640Arduino development with Eclipsehttp://possiblelossofprecision.net/?p=2342
http://possiblelossofprecision.net/?p=2342#respondTue, 18 Apr 2017 18:41:40 +0000http://possiblelossofprecision.net/?p=2342Read more »]]>The Arduino IDE is great for beginners: It makes it really easy to write simple programs without having to care about compiler options, include paths, language standards or how to actually flash firmware onto the microcontroller. It even comes with a built-in serial monitor which can be a great tool for debugging.
Inevitably, as people dive deeper into the world of microcontrollers and the Arduino platform specifically, they usually want more control of the toolchain and use features like referencing source code across different projects and include external libraries. While that’s technically all possible with the Arduino IDE, some of the things might be a bit clunky to set up. IDEs like Eclipse are much more suited for these use cases.

0. Prerequisites

While there are dedicated alternative Arduino IDEs, some of which are actually based on Eclipse such as Sloeber, using the AVR Eclipse plugin is probably the most versatile option. Most distributions (including Fedora) offer a dedicated package in their repositories. Of course, you’ll also need the Eclipse C/C++ Development Tools:

dnf install eclipse-avr eclipse-cdt arduino-core

1. Creating a new C++ Project

Now that we’ve installed the required packages, let’s start up Eclipse and create a new project (File → New → C++ Project):

After clicking on Next you can create additional configurations or change the default ones. For the purpose of a sample program we’ll only need a single Release configuration:

Apart from the include statement in the first line, this should look very familiar.

3. Including Arduino specific headers and sources

Now how do we include the Arduino specific code into our project? There’s actually multiple ways to do that. Most resources online (e.g. Francesco Rigoni at Dr. Strangecode) recommend creating some sort of ‘core library’, i.e. compiling the Arduino core project source code once in a separate Eclipse project and then reference that shared library from your other Arduino projects.
The advantage of this approach is that you need to compile the sources only once. However, this ‘core library’ depends on the actual target hardware: it’s compiled for a specific microcontroller running at a specific clock speed. If you change the clock speed or want to deploy your firmware to another Arduino variant with a different microcontroller, you either need another ‘core library’ project or you need to change the parameters of your existing ‘core library’ project (e.g. with an additonal build configuration), potentially breaking things for other projects that use it as a shared library. All in all: A bit too messy and impractical…

An alternative to that approach is linking the Arduino core source tree as another source directory in your eclipse project. If you do change the microcontroller or the clock speed, the Arduino sources are just recompiled along with your own source code. The disadvantage is that compilation takes more time since you’re recompiling the whole Arduino core (or at least the parts that your source code is referencing) along with your firmware. But since that takes just about one second on a modern PC, it could be deemed insignificant.

To create a link to an external source location go to Project → Properties → C/C++ General → Paths and Symbols → Source Location → Link folder and create a new link to the Arduino core directory:

The only thing left to do is adding two additional directories to the include directories list so the compiler actually knows where to find the corresponding header files. One is the newly created source location link (which could be a workspace path)

]]>http://possiblelossofprecision.net/?feed=rss2&p=23420Updating BIOS firmware via iPXEhttp://possiblelossofprecision.net/?p=2312
http://possiblelossofprecision.net/?p=2312#respondFri, 03 Mar 2017 21:17:15 +0000http://possiblelossofprecision.net/?p=2312Read more »]]>These days mainboards usually come with some sort of wizbang tool that allow the user to update the BIOS from a USB drive or straight via network. Except, of course, that one single mainboard that absolutely needs a new BIOS version on a late Friday afternoon. And obviously the manufacturer only provides a flash tool for DOS and the mainboard is not supported by flashrom yet.
In those cases booting FreeDOS can be really handy. Booting FreeDOS via PXE is not that hard and it can also be booted via iPXE quite easily. If you do boot it via PXE, the easiest way to access the mainboard manufacturer’s flash tool and the new BIOS firmware from within FreeDOS is probably to include it in the PXE image file (see here). With iPXE however there’s a much more elegant way…

Minimal iPXE script to boot FreeDOS

While a comprehensive tutorial on how to set up a dhcp, tftp and iPXE server would go beyond the scope of this post, here’s a short outline of the necessary steps involved.
First, we need to configure the dhcp server so that the iPXE firmware is handed out to legacy PXE clients, and an iPXE boot configuration file is handed out to actual iPXE clients. With the ISC dhcp server this can be done by handing out different configurations based on the DHCP user class (see also http://ipxe.org/howto/dhcpd):

The memdisk kernel is part of the syslinux-tftpboot package. Creating a symlink to /tftpboot/memdisk in the same directory in which the freedos.ipxe boot configuration file lives should be sufficient.

iPXE booting FreeDOS

Creating a payload image

Instead of manipulating the FreeDOS floppy image to include the mainboard manufacturer’s flash tool and the BIOS update file, we’re going to create a separate payload image and attach it to FreeDOS as an additional hard disk.

First, we’ll create an empty 32MB file (which could be bigger if one needed more space) and partition it with the first partition starting at LBA Address 63

truncate payload.img --size 32MB
sfdisk payload.img << EOF
63,,b
EOF

Next, set up the resulting payload.img file as loop device and create a filesystem:

losetup /dev/loop0 payload.img
mkfs.vfat /dev/loop0p1

If there’s no separate device for the first partition (e.g. /dev/loop0p1 you might want to pass a max_part option to the loop kernel module module so that loop devices are partitionable:

modprobe -r loop
modprobe loop max_part=16

Now you can mount the partition and copy the necessary files (flash tool and BIOS firmware)

Don’t forget to copy the payload.img to the host handing out the freedos.ipxe iPXE boot configuration file.

FreeDOS bootin with additional hard disk attached via sanhook

After booting up FreeDOS you can find the contents of the payload image file under C: (or D: if you booted the FreeDOS cdrom images instead of the floppy image).

]]>http://possiblelossofprecision.net/?feed=rss2&p=23120Migrating a virtual machine from KVM to ESXihttp://possiblelossofprecision.net/?p=2293
http://possiblelossofprecision.net/?p=2293#respondSun, 09 Oct 2016 06:52:17 +0000http://possiblelossofprecision.net/?p=2293Read more »]]>Migrating a virtual machine from one host to another is usually no big deal if both hosts run the same VMM. But what if one wants to move a VM from a host running a different hypervisor than the target host? In the case of moving a VM from KVM to ESXi that’s just not (easily) possible. However, one can convert the virtual hard drive and recreate the virtual machine on the target host, which should be good enough for most use cases.

Exporting the virtual hard drive

To export the virtual hard drive on the KVM host, there’s a handy little command line tool called qemu-img, which is usually part of the distribution running KVM. On Fedora, RHEL and CentOS it can be installed with:

For a list of supported formats have a look at the qemu-img manpage. We can now copy the resulting vmdk file VM.tmp.vmdk to the target ESXi host, e.g. using scp.

Importing the vmdk file

Even though we’ve just created a vmdk image file, it cannot be used as a virtual hard drive for a VM on ESXi. If you try to create a virtual machine with this particular vmdk file as hard drive, you’ll most likely get a error message in /var/log/hostd.log similar to this:

[...]
Power On message: Failed to open disk scsi0:0: Unsupported or invalid disk type 7. Ensure that the disk has been imported.
[...]

Fortunately, on ESXi there’s another handy command line tool for handling vmdk image files, called vmkfstools. However before we import the vmdk image file it is advisable to create a directory for the new virtual machine on the datastore first. This allows you to put the imported image into the correct directory (i.e. the directory where ESXi keeps all the files concerting one particular VM) straight away.

Of course, there are other disk formats available, too. You can throw away the original VM.tmp.vmdk image file after the import was completed successfully

rm /vmfs/volumes/datastore0/VM.tmp.vmdk

You can now go ahead an create a new virtual machine on the target host using the newly imported vmdk file /vmfs/volumes/datastore0/New-VM/New-VM.vmdk as a virtual hard drive. But depending on your VM, you might want to change the type of the hard drive from ide to scsi first:

Making adjustments to the virtual machine

Booting the newly created virtual machine might not work straight away, e.g. on Fedora, RHEL and CentOS the UUID of the hard drive is included in the initial RAM file system. The easiest way to fix this is to boot into rescue mode (there’s usually a rescue mode entry in the grub boot menu) and run

dracut --regenerate-all --force

to recreate the initramfs. You might also have to edit your network configuration since your network device usually get’s a different name.

]]>http://possiblelossofprecision.net/?feed=rss2&p=22930OpenSSH cipher performancehttp://possiblelossofprecision.net/?p=2255
http://possiblelossofprecision.net/?p=2255#respondWed, 01 Jun 2016 16:28:26 +0000http://possiblelossofprecision.net/?p=2255Read more »]]>The achievable speed of copying a file with OpenSSH (e.g. with scp) will depend on quite a few different factors such as CPU speed, CPU architecture, Network throughput, OpenSSH implementation, OS, hard drive speed, etc. But how much of a difference does choosing a different cipher algorithm make? And what’s the fastest OpenSSH cipher algorithm?

Turns out, there’s no simple answer to this question, since most of the factors that influence the transfer speed can be ruled out, but the results will at least depend on the hardware platform and OpenSSH version. There’s quite a few different benchmarks out there, e.g. for the Bifferbord, E5 xeon CPUs or different consumer grade CPUs and ARM processors. But since the results are so heavily platform dependent, it’s a good idea to run your own benchmark on the particular platform you are interested in. So here’s another data point for an Intel Xeon E5-2640 and OpenSSH 6.9p1 (OpenSSL 1.0.1k).

The test setup is quite similar to the one described at blog.famzah.net. The bash script used to produce the data is:

The test file consists of 5GiB random data. Both the source and target file system are RAM backed to remove the influence of HDD read and write speeds. In addition to that, the test file is written to localhost to ensure that network speed, load and NIC drivers do not influence the test results.

SCP file transfer speed

The results clearly show, that the Xeon’s AES instruction set is used. Most modern x86 CPUs do come with this extension these days.

While this data clearly suggests, that AES encryption is the faster cipher OpenSSH cipher (if there is hardware support for it as in this case), copying large amounts of data with scp is not a particularly interesting use case. Sending big streams of data through a pipe into ssh, as you do when you send and receive ZFS snapshots over ssh, is a very common application. For benchmarking reasons, sending actual ZFS snapshots is not ideal, since ZFS takes some extra time to check the receiving file system (and its snapshots) before starting the sending process. So here’s an altered script that should tell us, what the fastest cipher for that particular use case is:

The only difference can be found in the highlighted line: Instead of using scp the file is now piped directly into ssh and discarded on the receiving side. Again, the 5GiB test file lives on a RAM backed file system and the transfer is done to localhost.