Monday, May 18. 2015

That's what you get for using a version x.0, we should have known better .

And - of course - the release notes follow the common practice of not even mentioning any of the issues we encountered so far.

Ah, well, let's go through the first one:

In Debian 7 (Wheezy) there was netatalk 2.2.2 (packages link). Now during upgrades that package may or may not get removed. There is no netatalk in Debian 8 (Jessie) anymore. Duh.
There is in sid (aka Debian unstable) (packages link) so we may see a backport some time. Or not. In any case this is still 2.2.5 at the time of writing and as Adrian Knoth put it in the three year old bug asking for a upgrade to Netatalk 3:

Let's not ship another release without netatalk3, it's embarrassing.

Yes. It is. Removing a working version and not even mentioning it in the release notes is even worse though. So no cookies there.

Luckily the absolutely awesome Debian and greater FLOSS community have sorted 90% of the problem out for us already:

A quick Google search turns up an excellent article on netatalk's wiki that details installing Netatalk 3.1.7 on Debian 8 Jessie. There are two shortcomings to this: First it doesn't compile to .debs but installs besides apt and friends. And second it compiles with Spotlight search, courtesy of Gnome tracker, which doesn't really work well on servers yet. Hence we're lucky that Adrian Knoth's debified install has not yet added the tracker dependencies. It does compile for systemd use (the default for Debian 8 Jessie). If you want to continue using SysVInit, you need to modify debian/rules.

Thursday, March 5. 2015

The core team organizing DebConf, the annual Debian developer conference, reached out to me two weeks ago to help support this year's effort a bit.

I'm very happy to do so as Debian is a cornerstone of everything I do in the Open Source/Free Software space.

To get me started I got access to a lot of mailing lists and irc channels. And even more git repositories. So many that the DebConf team even has an instruction page on how the repositories all fit together.

It's unfortunately quite common to split a bigger project into many git repositories to ease access rights management and reduce the noise and data transfer volume for the average user. The downside is, everybody ends up with a dozen or more individual repositories to keep pulling. And then there's git annex for yet another level of indirection.

Joey Hess, a former Debian developer, has even written an extensive tool, myrepos, to meta-manage the different repositories and it can do quite some magic across different SCMSs1. In my case this is a bit of an overkill though.

And using myrepos may get you confused at some point whether to now run mr or git directly for each batch of repos you have inherited over some time of working on multiple projects.2 Thus I prefer the simple route:

Check out each repository into a common top-level directory (~/debconf/ in this case) and then put the following two lines into an executable script git_pull_all into that top level directory:

This will allow you to pull all git repos with one command and keep the normal syntax for everything else you do with each repo.

The --mindepth and --maxdepth will instruct find to just go and run your git pull only inside each direct child of the top level directory. So recursion depth = 1. That is the single trick there is to this.

Updates:

If you like to have some color and a bit of a spaced layout for improved readability, try:

When you have pull.rebase=true set in your .gitconfig, you can run ./git_pull_all --no-rebase to avoid rebases in case you work somewhere and want to have the merge commits.

P.S.: The DebConf15 Heidelberg registration just opened, please check the DebConf15 homepage for news, venue information and please register if you want to come around.

Source Code Management Systems, like git, mercurial (hg) or subversion (svn). Or God forbid ... cvs. I don't like the (D)VCS (Distributed) Version Control Systems moniker. Because that's not really all these systems do. Not even the most important piece of what they do these days. ↩

With myrepos you can still work with each individual repository via git. Just so nobody will write in "but...". ↩

This will copy "file" from "/path/to/directory/" into exactly the same directory on the remote cluster partner.
Which system to target is detected from the hostname of the local system and hard-coded into the script.

Scpover can also take multiple filepaths on one command line so you can beam over a few related config files from multiple locations in one go.
Not an atomic change but at least all within a reasonably short timespan.

Friday, June 6. 2014

Memory can mean RSS (Resident Set Size) which is the memory of a process held in RAM (so not swapped out).
That does include shared memory allocations. So if you add two RSS numbers, you're probably wrong already.
Still this is usually the number we look for in most practical investigations.

Then there is VSZ (Virtual Set siZe) also called SIZE. The VSZ includes code, data and stack segments a process has allocated.
And again that will count some shared address space. So usually bash will have a VSZ that's lower than its RSS.

man ps will also tell you:

The SIZE and RSS fields don't count some parts of a process including the page tables, kernel stack, struct
thread_info, and struct task_struct. This is usually at least 20 KiB of memory that is always resident.

In most (if not all) practical scenarios that difference won't matter. If it were, you'd be using valgrind to look into the memory usage of your application in minute detail. Wouldn't you?

If you want to have an as-detailed-as-possible look into the memory allocations of a process pmap <pid> will give you the information.
The summary at the end is a gross over-estimation of the total memory a process has allocated as it counts all mapped memory (and may still be wrong due to de-duplication and other factors). But that number may well serve as an upper bound if you need something like that.

For running processes

ps-eo'pid user rss:8 size:8 cmd'--sort'rss'

will give you a nice sorted list of processes and their RSS and VSZ (SIZE) in kiB (old school kB...).

For short running commands GNU time (not the bash build-in time command) has a nice capability that's not widely known yet:

/usr/bin/time-f"RSS: %MkiB"<command>

will tell you the maximum RSS size the <command> has had during its lifetime. That's better than top or watch ps and trying to spot the process.

Monday, February 17. 2014

Every once in a while iTunes on Windows decides to start but does not show the main application window when trying to sync an iPhone (or any other iDevice).
May be it pops up after half an hour, may be not.

This behavior is caused by iTunes waiting for its Bonjour zeroconf network service.
Unfortunately though iTunes is updated very frequently that bug is persistent for years now.
The work-around is easy though:

Open up CMD as Administrator and type

net stop "bonjour service"

and iTunes should pop up its window a few moments later.
It will complain about Bonjour not running but that is not needed for anything but network self-discovery. Which you usually don't need anyways.

If you do, you can start Bonjour again with

net start "bonjour service"

anytime also while iTunes is running. Once it decides to show its main window, it's fine to go for the session.

Friday, September 20. 2013

Automatically transferring (syncing) files between multiple computers is easy these days. Dropbox, owncloud or bitpocket to name a few. You can imagine I use the latter (if you want a recommendation)1.

In any case you want to encrypt what you send to be stored in "the cloud" even if it is just for a short time. There are many options how to encrypt the "in flight" data. Symmetric ciphers are probably the safest and most widely researched cryptography these days and easier to use than asymmetric key pairs in this context as well.

Encryption is notoriously hard to implement correctly and worthless when the implementation is flawed. So I looked at gpg, a well known reference implementation, and was amazed that it can neither use a proper keyfile for symmetric encryption (you can just supply a passphrase via --passphrase-file) nor does it handle multiple files on the command line consistently.
You can use --multifile (wondering...why does a command need that at all?) with --decrypt and --encrypt (asymmetric public/private key pair encryption) but not with --symmetric (symmetric shared key encryption). Duh!

With a bit of scripting around the gpg shortcomings, you end up with crypt_gpg that can nicely encrypt or decrypt multiple files (symmetric cipher) in one go.

Dropbox is closed source so it cannot be assessed for its security. Owncloud needs a thorough code review before I would dare to run it on my systems. ↩

Sunday, October 7. 2012

Since version 2.0 the behaviour of grub regarding passwords has changed quite substantially.
It can be nicely used to secure the boot process so that a X display manager (gdm, kdm, lightdm, ...) or login prompt cannot be circumvented by editing the Linux kernel boot command line parameters. The documentation is concise but many old how-tos may lead you down the wrong GNU grub "legacy" (the pre-2.0 versions) path.

So this assumes you have a grub installed and working.
I.e. if you press Shift during boot, you get a grub menu and can edit menu entries via the e key.

First you need to setup grub users and corresponding passwords:

Run grub-mkpasswd-pbkdf2 to encrypt every password you want to use for grub users (which are technically unrelated to Linux system users at this time).
You'll get a string like 'grub.pbkdf2.sha512.10000...'. It will replace the plain text passwords.

In '/etc/grub/40_custom' add lines like:

# These users can change the config at boot time and run any menuentry:
set superusers="root user1"
password_pbkdf2 root grub.pbkdf2.sha512.10000.aaa...
password_pbkdf2 user1 grub.pbkdf2.sha512.10000.bbb...
# This user can only run specifically designated menuentries (not a superuser):
password_pbkdf2 user2 grub.pbkdf2.sha512.10000.ccc...

Now once you did this grub v. 2.0+ will ask for a supervisor password every time you want to boot any menu item.
This is a changed behavior from v. 1.9x which defaulted to allow all entries if no user restriction was specified.
So you need to add '--unrestricted' to all 'menuentries' that any user shall be able to boot.
You can edit '/boot/grub/grub.cfg' and add --unrestricted to (the default) menuentries.
Or you can edit the 'linux_entry ()' function in '/etc/grub/10_linux' so that the 'echo "menuentry ..."' lines include --unrestricted by default:

Make a backup of this file as it will be overwritten by grub updates.
This way all Linux kernels detected by the script will be available to all users without identifying to grub via username / password.

Now issue update-grub to re-generate 'grub.cfg' with the amended menuentries.

If everything worked well, your system can now be booted unrestricted but the grub configuration can only be changed from the grub superusers after identifying with their username and password at the grub prompt.

Bonus point:

If you want to create menuentries that user2 (and any superuser) from the above example user list can run, add blocks like these to the end of '40_custom':

Update

16.12.2015:
Hector Marco and Ismael Ripoll have found a nearly unbelievable exploit in Grub2 that allows you to tap backspace 28 times to get a rescue shell and that way bypass a password prompt. Time to update!
Read the excellent analysis of the bug and the exploit vector in Hector Marco's blog post.

Saturday, February 4. 2012

I found a very nice script to create Notes on the iPhone from the command line by hossman over at Perlmonks.

For some weird reason Perlmonks does not allow me to reply with amendments even after I created an account. I can "preview" a reply at Perlmonks but after "create" I get "Permission Denied". Duh. vroom, if you want screenshots, contact me on IRC .

As I wrote everything up for the Perlmonks reply anyways, I'll post it here instead.

Against hossman's version 32 from 2011-02-22 I changed the following:

removed .pl from filename and documentation

added --list to list existing notes

added --hosteurope for Hosteurope mail account preferences and with it a sample how to add username and password into the script for unattended use

made the "Notes" folder the default (so -f Notes becomes obsolete)

added some UTF-8 conversions to make Umlauts work better (this is a mess in perl, see Jeremy Zawodny's writeup and Ivan Kurmanov's blog entry for some further solutions). Please try combinations of utf8::encode and ::decode, binmode utf8 for STDIN and/or STDOUT and the other hints from these linked blog entries in your local setup to get Umlauts and other non-7bit ASCII characters working. Be patient. There's more than one way to do it .

Sunday, March 27. 2011

Apple has crippled the iPhone to not allow normal music files as ringtones. Business decision. Technically any sub 40 second MP4 audio file will do once you rename it to *.m4r and drag-and-drop it to the ringtones folder of your phone in iTunes. Longer ones will work, too. But you'd need a jailbroken iPhone for that as iTunes will refuse to transfer the ringtone file if it's too long. Not much of an issue imho, who keeps ringing your phone for 40 seconds or more?

There's a gazillion websites available telling you how to convert a single .mp3-file to a ringtone with or without iTunes help and there are hundreds of tools doing that for you if you can't find out how to do it with just iTunes itself. Still the ones I tried failed for me as I wanted to convert my 20 or so standard ringtones from the good old Motorola K3 to iPhone ringtones all in one go. Without having to edit each one by hand. They are already nice ringtones and have served me well for years, just too long for the iPhone and in .mp3 format.

So below is the free shell script to create multiple ringtones in one go on any Linux system. You need to install cutmp3, mplayer and faac for it, so apt-get install cutmp3 mplayer faac on Debian or Ubuntu. cutmp3 is currently not in the portage tree for Gentoo, but you can download an ebuild from Polynomial-C's overlay (mirror). Or you just download the cutmp3 binary from Jochen Puchalla's homepage. There's no error checking in the script, so know your way around the shell before running it.

Update

23.12.14 Apparently the faac package in Debian and Ubuntu has had the MP4 writing capability removed in v1.28-5 and later due to a minor license incompatibility. See the Debian Changelogs. Duh.

faac (1.28-5) unstable; urgency=low
[ Andres Mejia ]
* Disable mp4v2 support.
This only disables mp4v2 for the faac utility program. The faac
utility is GPL-2 but the mp4v2 library is MPL-1.1. The two licenses
are incompatible with each other.

So ... unfortunately you have to built faac from source yourself or pin the v1.28-4 version which is identical except for the castration anyways.

Saturday, January 9. 2010

These days many systems are multi-homed in the sense that they have more than one IP address bound at the same time.
I.e. for different network cards, virtual IPs for shared servers or just using WiFi and a wired network connection at the same time on a laptop.

Murphy of course makes sure that your system will choose to worst IP (i.e. that on slow WiFi or the one reserved for admin access) when an application does not specifically supports binding to a selected IP address.
And Mozilla Firefox for example doesn't.

The kernel chooses an outgoing IP from those in the routing table with the same metric:

You can obviously play around with the metric and make the kernel router prefer the desired interface above others. This will affect all applications though.
Some people use the firewall to nat all packages to port 80 onto the network interface desired for web browsing. Gee, beware the http://somewebsite.tld:8080 links...

Thankfully Daniel Ryde has solved the problem via a LD_PRELOAD shim. With his code you can run

If you don't have gcc available (and trust me) you can download pre-compiled 32bit and 64bit (glibc-2) bind.so libraries here (4.5kB).

I guess because Daniel Ryde hid his code so well on his webpage, Robert J. McKay wrote another LD_PRELOAD shim, called Bindhack (4.5kB, local mirror). This will - as is - only compile on 32bit machines. But YMMV.

Run the above command (*) with your desired (and locally bound) IP address in bash and visit MyIP.dk or DNStools.ch or any of the other services that show your external IP to see whether you've succeeded.

Notes:

Daniel Ryde did not specify the -D_GNU_SOURCE in the comments section of bind.c. Modern glibc/gcc need that as he used RTLD_NEXT which is Unix98 and not POSIX. I amended the local copy of bind.c and sent him an email so he can update his.

Both are IPv4 only, no IPv6 support.

Updates:

19.03.15 madmakz wrote in to clarify that all of the bind LD_PRELOAD shims only work with TCP connections. So not with UDP.
I'm not aware of a shim that manipulates UDP sockets.

14.01.14 Christian Pellegrin wrote a superb article on how to achieve per-application routing with the help of Linux network namespaces.

16.06.13 showip.be seems to be gone, so I replaced it with dnstools.ch in the text above. There are plenty of others as well.

22.06.12 Lennart Poettering has a IPv4 only version of a shim and a rather good readme available at his site.

29.11.10 Catalin M. Boie wrote another LD_PRELOAD shim, force_bind. I have not tested this one. It's capable of handling IPv6 binds.

11.01.09 Daniel Ryde has replied to my email and updated his local copy now as well.

Monday, December 21. 2009

Using a mobile phone's Bluetooth Dial-up network (DUN) to connect to the Internet (UMTS/GPRS) while on the road is quite convenient for me. Sadly so this is not supported out-of-the-box in Ubuntu Karmic 9.10 (Netbook Remix) as it uses Network-Manager to handle - well - network connections. And that is not quite there on Bluetooth managed devices yet.

While the default solution (rfcomm and Gnome-PPP) still works, it's ugly to set up. Sadly so, zillions of Ubuntu-Forum threads and blog entries still detail this solution - or the issues encountered with it along the way.

The much better solutions is using Blueman, an improved Gnome-Bluetooth primarily developed by Valmantas Palikša. It brings the right UDEV magic along to teach Network-Manager about the Bluetooth devices it handles.

Blueman Screenshot on Ubuntu Karmic 9.10 Netbook Edition

Just follow the steps on their downloads page to set up the Blueman PPA (Personal Package Archive) to get things working.

Sunday, December 6. 2009

For some unknown reason the (K)Ubuntu developers did not update the Java plugin for firefox after jaunty (yet?).

The version that Karmic (9.10) pulls out of the multiverse repository is still jaunty's (9.04).

So when you try:

apt-getinstall sun-java6-plugin

you'll get something like

Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
sun-java6-plugin: Depends: sun-java6-bin (= 6-15-1) but 6-16-0ubuntu1.9.04 is to be installed
E: Broken packages

Duh.

Actually if you have the Java Runtime Environment (JRE, package name sun-java6-jre) installed all files needed are already present.
Just not put in the right place on the filesystem.

Thursday, July 30. 2009

I needed to expand a lot of jars (Java zips) and other zips of various names into directories of the same name for each file.
With 6,239 files of which some are jars, some other zips and many xml and other filetypes all not properly identified by a file extension, this gets a bit too much to do manually.

So:
Finding candidates for these is easy with find . -type f.
The file is most probably a zip archive if the first two characters are "PK", good old Phil Katz' signature. A friendly head -c 2 checks that.
All combined with some rudimentary error checking:

#!/bin/bash

# There is little data security here, so know what you're doing.

# All risks in using this code are yours. It moves and deletes files quite stupidly.

This will expand all zips under the current directory.
Leave the zip2dir out for a dry run (xargs will just print to the tty then). Look at the -exec switch when digging around a bit more into what find can do for you.

Sunday, May 10. 2009

There is a add-on technology to X or VNC called NX by an Italian company called NoMachine.
It's quite useful as it speeds up working on remote desktops via slow network connections (i.e. DSL pipes) substantially.

The libraries that implement NX are released under GPLv2 by that company.
A server wrapping up the libraries' functionality is available as closed source from NoMachine or as a free product (GPLv2 again) by Fabian Franz, called FreeNX.

FreeNX itself is amazing as it is written in BASH (with a few helper functions in C). It's also able to mend some of the shortcomings of the NX architecture. E.g. stock NX requires a technical user called "nx" to able to ssh into the NX server with a public/private keypair.
FreeNX can work around that for more secure set-ups.

One issue I bumped into quite regularly with Linux clients and Linux hosts from different distributions/localisations is that the keymaps are not compatible. This usually results in the ALTGr key not usable, so German keyboard users can't enter a pipe ("|"), tilde ("~") or a backslash ("\") character. Also the up and down keys are usually resulting in weird characters being pasted to the shell. Now all of that makes using a shell/terminal prompt quite interesting.