Tag - television

"Give me a place to stand and a lever long enough and I will move the world."

I was grumbling under my breath at a configuration management system today, and reminded of this wonderful statement by Archimedes.

Configuration management is the discipline of building systems which manage other systems -- cfengine is a well-known open source example. I needed to reboot a few hosts on a regular schedule -- easily handled in 5 minutes with "vi /etc/crontab" on each, or an ssh loop to append to the crontab on each affected system. I was struck by how many levels of indirection I needed to traverse to get this done with configuration management. This in turn prompted some thought about why jumping through the various hoops was worthwhile.

There are many excellent reasons to use configuration management:

Time savings -- over repeating the same actions over and over; this increases with the number of hosts involved.

Reproducibility -- because CMS is naturally tied into version control, it is easy to either examine or recreate the state of affairs at an arbitrary time in the past.

Modeling -- a CMS ends encompasses a representation of all the systems it manages. This efficient representation of those systems is quite useful for examining and comparing them. It's especially useful with a large or dynamic population of administrators, as it provides a single place to learn about the whole constellation of systems, and enforces some consistency among the various ways admins can manage systems.

In the simplest case, to make a machine reboot once, I could pull the plug and put it back (assuming I was near, or could get to, the machine). In a non-CMS scenario, I would do it with ssh and the shutdown -r. In this case, it was considerably more involved:

Launch PuTTY.

Log into a system with a checkout of the CMS configuration files.

Find the appropriate file (non-trivial if the managed constellation is complicated).

Fetch the latest version of the file (with multiple users, it's unlikely my checkout is current).

Edit the file corresponding to /etc/crontab or /var/spool/cron/root (I used kate, as I don't enjoy either vi or emacs, and BBEdit wasn't available); kate popped back an X11 session tunneled through ssh.

Create a pair of local machine sets in the file (cfengine calls these 'aliases'), each including half the covered systems (the systems reboot at staggered times, so they're not all down at once).

Create the pair of crontab lines, one for each machine set, embedding the pair of different reboot times and the shutdown -r command.

Check the modified crontab file back into the version control system; enter a message for the change log.

In a distributed CMS, staging hosts pick up the changes from version control, either on a schedule or when manually kicked for emergency/rush changes.

The affected hosts pick up the change from the CMS, and implement the specified change.

The reason Archimedes' quote is apropos is that configuration management provides excellent leverage -- I can edit one file in one place, and easily affect several systems (potentially hundreds or thousands). Each hoop I have to jump through provides an additional fulcrum. I can sit at my desk and use PuTTY to log into dozens of systems, across the world -- without even knowing where they are. Each change I make to the version control system is automatically picked up by every host participating in the system, and available to every admin with a checkout. I don't have to log into 8 machines (even uninteractively) to make them reboot -- I can orchestrate it all from my local workstation.

Unfortunately, mistakes are leveraged too; there is often no good way to test changes to production systems during business hours. If the changes are restricted to non-production hours, when the admin might not be around to monitor them (and shouldn't have to -- it's an automated system, after all!), the window could be closed by the time the admin sees whether the change was successful. Missing a change window can easily defer a change 24 hours.

Please call me if you have an account on reppep.com and haven't received your password already, or find anything not working right.

I switched from Apple's jabberd to Openfire, which doesn't use the UNIX system accounts, so let me know if you want a chat account (compatible with iChat & GTalk).

[Done] I forgot SquirrelMail address books -- should be able to bring those over too.

Firewall problem fixed. SMTP MX issue fixed.

Virus filtering problem fixed.

Webmail certificate fixed.

Quota problem fixed.

Virtual domains for email fixed.

As of 5pm, I don't know anything that doesn't work (aside from SquirrelMail address books) [fixed Thursday].

Thanks for your patience!

As of 10:30 on the 20th, things seem to be working. Something's screwy with amavisd-new's quarantine, but mail is going through. I reinstalled Openfire, and chat seems okay under the correct hostname/certificate name now (will try signing it as ca.reppep.com later).

Good timing -- the optical drive on the old server died tonight.

I have distributed all the new temporary passwords, so any users having trouble logging in should let me know.

Markdown.cgi is still broken, but I'm the only person who uses it here, so I'll get to it.

On Thursday the 21st, I found a problem with amavisd-new -- it had quarantined 32,000 messages in a single directory, and was stuck (apparently ext3 doesn't support more than 32,000 files in a directory). I cleared it out and finally managed to disable quarantine, which wasn't as easy as it should have been, and the backlog of messages have been delivered as of 9:15pm.

At 11pm, I fixed an issue preventing SMTP AUTH from working properly, which was interfering with sending email to non-reppep addresses.

After getting burned too many times, I dropped my .Mac subscription. I never trusted my Apple keychains to iDisk anyway, but this means I have different subsets of passwords on different machines, and no good way to keep them in sync. I thought of a solution for manual sync last week: One keychain per Mac. Say I have 3 systems: work, home, and other. Each system has 3 Apple keychains: work.keychain, home.keychain, and other.keychain, with each host using its own as the default. Then I can rsyncwork.keychain to home.keychain & other.keychain, etc. This is awkward with rsync because it's inherently unidirectional, but keychains are small so it's quite feasible to script.

In Tiger, I know the keychain is actually stored in memory once it's unlocked, so it's good to lock (unload) all keychains with "security lock-keychain -a" before updating the files -- this goes in the same script. I also set mine to lock after 2 hours of inactivity, or (on those systems where I run SSHKeychain) when sleeping or activating the (locking) screen saver.

I reversed over 10 years of history today, moving my Eudora Folder off my iPod. I've been carrying my email around with me for a long long time.

I will now reply upon IMAP to keep my mail in sync (as many people do -- this is much of the purpose of IMAP). Two main issues kept me carrying around my email after I switched Eudora from POP (which it does wonderfully) to IMAP (which it does less well):

Message Status. I know there are messages I have read and marked as such in Eudora, but where the server has the messages marked as unread. I suspect in some cases this is because Eudora lost connectivity to the server and was unable to update the read/unread status immediately, but I've been shielded from this by carrying my mail (and the ToC files where Eudora keeps read/unread status) with me on disk for years.

I use open Eudora messages as a To Do list, and each copy of Eudora will keep its own independent list of open windows. I don't know if I'll use saved searches or how I'll keep track of messages that require attention yet.

I have (and needed!) several reasons to make the switch:

I no longer have to carry around an iPod all the time. To and from work isn't too bad, since I was often listening to it, and the iPod is much less obtrusive on a belt clip than my VST 10gb 2.5" FireWire drive was, but it's still something else to carry/remember/worry about losing.

I can now once again use my iPod. Previously it was really only available when travelling, because the rest of the time the iPod was plugged into a Mac in FireWire Disk Mode. To take and use the iPod required first quitting Eudora and unmounting the iPod, and later plugging it in and letting Eudora relaunch and open all its windows.

I am getting an iPhone (rather than the iPod touch I ordered last week), and don't want to give up 1.5gb of its 8gb for email.

I cannot leave my iPhone tethered to a computer in disk mode when I walk away from my desk.

I can now easily run Eudora on my work laptop (the iPod was always plugged into my work desktop). I actually started doing this a while ago, and my head did not explode due to IMAP sync discrepancies.

Moving my laptop around our apartment will be more convenient -- I won't have to carry the iPod around (plugged in) on top of the keyboard as I walk up and down stairs.

Hooray! I now have a functional script for copying my iCal, Address Book, and Safari bookmarks. I suspect it's copying more than necessary, so I may refine it, but it's doing what I need now. I am amazed at how many places Apple sticks calendar data!

A big thank you to the rsync and OpenSSH developers. Once again, the script is largely comments, but that's good -- it is fundamentally simple.

#!/bin/sh
# rsync-dotmac.sh
# v1.0, 2007/07/11 by Chris Pepper
# Check for a newer version at:
# <http://www.extrapepperoni.com/category/computers/synchronization/>
# <http://www.reppep.com/~pepper/code/>
for h in mac1 mac2
do
ssh $h rm -Rf ~/Library/Caches/com.apple.iCal ~/Library/Caches/iCal \
~/Library/Caches/Metadata/iCal \
~/Library/Preferences/com.apple.iCal.alarmsCache.plist \
~/Library/Preferences/com.apple.iCal.AlarmScheduler.plist \
~/Library/Preferences/com.apple.iCal.plist \
~/Library/Preferences/com.apple.iCal.sources.plist \
~/Library/Preferences/com.apple.iCalSync.AlarmScheduler.plist
rsync -va --delete ~/Library/Application\ Support/AddressBook/ \
$h:"Library/Application\ Support/AddressBook/"
rsync -va --delete ~/Library/Application\ Support/iCal/ \
$h:"Library/Application\ Support/iCal/"
rsync -va --delete ~/Library/Calendars/ $h:Library/Calendars/
rsync -va --delete ~/Library/Safari/Bookmarks.plist \
$h:Library/Safari/Bookmarks.plist
done
# To use, enter your read-only hostnames on the 'for' line above.
# They must, of course, be accessible from the system you run this script on.
# The script pushes data (Address Book, iCal, and Safari bookmarks) from
# a master to one or more read-only systems. It is intended as a free
# replacement for a part of Apple's .Mac service that caused me some
# frustration.
# I wish Apple provided a way to tell those applications not to allow
# (futile) changes on read-only systems, although with .Mac services
# that should not be necessary.
# I don't sync ~/Library/Keychains/, because my keychain items vary
# between systems, so unidirectional sync would be suboptimal.
# You may get warnings when going between OS versions; I have had no
# trouble just ignoring them (so far).
# If you get double birthdays, try disabling and re-enabling the
# Birthdays calendar in iCal's Preferences.

Now that I've given up on .Mac sync, I see it's very simple to push the data out. The biggest problem is that rsync doesn't approve of Apple's spaces in names, so I have to get a bit silly with double quoting. Not a big deal.

This raises a big question, though: How does .Mac sync notify apps that their data has just been changed out from under them, and they should reload? Obviously all these apps have some mechanism to reread their data, because Apple uses it, but I have no idea what it is. For Address Book & iCal, it would be reasonable to close them as part of the script (if I could find a way that didn't involve sending a Quit AppleEvent, which has an annoying tendency to launch non-running apps solely so it can tell them to Quit). But for Safari, it would be quite bad to lose all the tabs I left open.

Note that this script doesn't actually update anything. Once you have your real hostnames entered, remove -n from the script, and perhaps the v option as well, since you won't be seeing the output once it's in your crontab.

Hmm, I wonder if Library/Application\ Support/AddressBook/.database.lockN is relevant...

I give up on iSync. Last week it tanked again, and I cleared out all the Mac bits I could find and restarted from just my PowerBook (as well as I could), but it kept bloating and not finishing. On the third attempt, Conduit Manager started crashing during iSync.

Eventually I wiped my Treo, which is not an acceptable fix for Apple's data corruption. Now I'm trying to get everything back in sync.

I thought it was going to work, after a half-hour iSync run, but CM crashed again. What a load.

I think I just have to accept that Apple hasn't solved this problem (despite their claims), and switch to rsync updating my calendar, Address Book, and bookmarks from my PowerBook with rsync. So much for the (fantasy) of being able to update my calendar on home & work Macs.

I had to wipe the Palm, but after Reverting iCal to a current backup, Conduit Manager miraculously stopped crashing (after 4 crashes, so it was a surprise).

Apple intends to renew my .Mac membership in 47 days, so that gives me a timeframe to wean myself from it. Murphy says I will do so, and then the iPhone or Leopard will have a great .Mac feature that sucks me back in...

Last week my PowerBook ran out of free space. After cleaning up and rebooting a couple times, I realized it was because ~/Library/Application\ Support/SyncServices/ was bloating again. I cleared it out, re-enabled iSync (deleting this folder disabled the iSync Palm HotSync conduit), and ran out of space. I deleted again & left iSync off, because I didn't have time to fight with it.

Tonight, I again cleared my .Mac sync data (unregistered all clients) and iSync history, rebooted, did a .Mac sync, and re-enabled Palm synching. Perhaps half an hour later, the Treo 650 just finished synching "for the first time" and SyncServices is 1.4gb. That was a "Fast sync"! No good.

I have a lot of trouble with iSync (to my Treo) and .Mac sync. A few weeks ago, it was generating bogus complaints and bloating ~/Library/Application Support/SyncServices (up to 9gb in a single HotSync, which ran the PowerBook's hard disk out of space). I eventually cleared that one.

I had to abort a HotSync a couple days ago. Yesterday, Palm syncs (which I do twice per day or more, to get fresh plucker content) stopped updating my calendar. I cleared iSync's state, cleared .Mac's state, completely erased my Treo (scary, when the master data on the PowerBook might have been [and in the end actually was] corrupt), yada yada yada. I got different (related) errors, but nothing actually did the trick, until I backed up my data from iCal and reloaded it (luckily, this apparently blew away the corruption). Even this was fraught. I tested by loading it on another system -- trying to make sure the export was good, as I didn't trust the source. But when I attempted to open the iCal backup on the other system, iCal just spun and spun but never seemed to get anywhere. Eventually I realized it was calculating the disk size on the iCal backup. iCal backups are actually packages (folders), and it was taking much longer than it should have to total up the size of the folder, during which iCal was unresponsive (I actually force quit it a few times before I figured out what was going on).

Lesson for Apple is (aside from please fix synching in Leopard!!!): when complaining about a database synchronization error, identify the database you're complaining about! The iSync conduit was spewing errors about updating a specific record number. Of course, I have no reference for what that record number means. Had it mentioned that this was a problem in the iCal database, not the Palm DB or the "Truth database" (on iDisk), I could have saved hours and skipped erasing the Treo.

Another bug, just confirmed and reported. In the .Mac sync reconciliation dialog, I can't activate the "default" (blue throbbing) button with Return/Enter; it gets passed through to the window behind the dialog, although I can change the dialog's pop-up menu from the keyboard.

As it turned out, i started this posting prematurely last month. iSync was still corrupted. I had to wipe my Palm a couple more times, delete all my .Mac sync clients a couple more times, and even enable synching from iCal's & Address Book's preferences because System Preferences:.Mac:Sync wasn't usable -- it would just spin forever, never letting me actually click on the check-boxes to enable synching or specific types of data.

A week later, that part of System Preferences works again, and I have no idea why.

I also don't know how much longer it will continue to work. I was very intrigued by SyncTogether, until I discovered it just uses the same Sync Services infrastructure that keeps breaking so badly on me. I basically pay $100/year for .Mac sync, since I don't use their promo software and I run a more reliable mail server myself (on Apple hardware and software, no less!), so I'd love to find a cheaper alternative, but it's not clear if SyncTogether would really be cheaper (depends on upgrade schedule and pricing, and how many machines, and would require me to poke additional holes in my firewall). Unless and until Apple's syching gets stable, it doesn't make sense to invest more in it than I already am. Additionally, I don't know how soon Mark/Space will have Leopard support.

I do not have a comprehensive backup strategy. I keep reading well-reasoned arguments for at least 3 complete (bootable) backups, scratching my head, and saying, "Yeah, if I had a couple thousand unused, I could do that." Yes, I know drives are cheap, but I have 3 important systems to back up, and 4 more where perhaps a dozen files matter combined. Between the important systems, I'd need about 600gb usable, and Joe thinks you need 3 bootable backups, and I'm a strong believer in keeping the backups permanently away from the originals (not on a rotating cycle, but never-the-twain-shall-meet time -- what else is broadband for?).

Unfortunately, whenever I kick off an rsync job to update my remote backups (one set, not 3), Amy complains every day because Internet access is so slow. I think I have throttled it back enough to be unintrusive, but I just don't remember, and (especially at an unobtrusive fraction of our 384kbps DSL uplink), it just takes so dang long.

But I'm not too worried, because 95%+ of my important data is in Eudora on my iPod (yes, I know iPods die -- I've killed more than my fair share, along with Zip disks, Orb disks, and even a 2.5" FireWire drive), and I use Synchronize Plus X to back up my iPod to my primary home and work machines automatically, every time I plug it in. This is a great feature of Synchronize, which mitigates my annoyance with their licensing.

Every morning when I get to work, I dock the iPod; Synchronize sees it, launches Eudora, backs up changed files to my work desktop's hard disk, and launches an AppleScript to bring up a couple applications. When I'm done, I type eject in a Terminal window, my iPod unmounts, and login.keychain is locked (flushing ssh keys in the process). On my home PowerBook, the process is the same.

So I know my backup system works well, because I test restores whenever there's a problem with the iPod (perhaps twice a year). But I still feel nervous erasing it before sending it away. I actually did this today -- a couple months after its 2-year anniversary, my 60gb iPod photo's battery won't last an hour. Apple told me they'd fix it under extended AppleCare, but changed their minds and cancelled the repair -- without telling me. I paid the $70 and sent it in today; I'm running off our old 10gb iPod in the interim.

For the next step, I'm moving my website into Subversion. I already keep Julia's school site & Bjorn's site & documentation in Subversion, and last week I got sick of making manual backups of a piece I'm writing for TidBITS, so I reorganized and checked in my writing (not that there's much of it). Tonight, I threw out 4gb of my 7gb ~/public_html in preparation for checking in (most of) the rest of it.

Backing up a Subversion repository is easy, and will bring me to about 99% of important data backed up (not counting MP3s -- are already replicated to 2 Macs & an iPod -- or iPhoto libraries, which require different handling, but should back up quite well via rsync; but I have to get Julia's few [important] pages into Subversion too).

.Mac sync doesn't work without saving your password in the Apple Keychain -- BROKEN! I don't want to save my password on a laptop that's likely to get stolen.

If you delete your .Mac password from the keychain, Sync Now from the iSync menu fails with an error, but without an opportunity to enter the password:

In System Preferences:.Mac:Sync, clicking Sync Now generates the same error message with a different icon:

After entering a password in System Preferences:.Mac:Sign In, the system pops up a keychain password prompt. If a password is provided, the .Mac password is immediately saved to the default keychain. This is the only keychain access dialog I'm aware of which doesn't offer a checkbox to save the password, and instead forces password saving on -- BROKEN!:

For a long time, I ignored the Apple Keychain (see earlier posts for more Keychain travails). I didn't want to keep my passwords anywhere accessible without my intervention. What finally made me give up was the fact that Safari prompted me for my Keychain passphrase on every single flipping page with a text entry field. Eventually I gave up, and started unlocking the keychain. It's very useful, but Apple (Safari) effectively forces many users to use the Keychain by making it so intrusive and unpleasant. Why isn't there a "don't bother me" option in that dialog?

Then when I started using SSHKeychain, the Apple Keychain became much more important to me, because it contained the passphrase for ssh private keys. I am an aggressive locker. When I leave the room, I lock the screen. I do this at home (and irritate Amy), and I do this when I leave the cube farm at work. As a result, I unlock the Mac frequently, with a longer-than-average password. It's a minor nuisance to type one password more than hourly, but if I had to unlock the screensaver, at least one Apple keychain, and one or more ssh private keys, I wouldn't be able to get any actual work done before I bought an Uzi.

With SSHKeychain, I discovered that Apple doesn't support locking Apple keychain(s) when the screensaver locks. Now I know that a major reason for this is that things break when the keychain is locked. In particular, .Mac sync throws all kinds of hissy when it doesn't have access to your .Mac password through a keychain (I've counted 5 different prompts for my Keychain password so different parts of .Mac sync can connect). That's obnoxious, and bad security.

Today's brokenness is related. If you don't have your .Mac password in an unlocked keychain, it's impossible to get a .Mac iChat certificate. Instead you get a bogus error pointing to the "Forgot password" page. I didn't forget my password, you robotic clown! I just won't give it to you for safekeeping (this is on a multi-user server I rarely use, and where I don't want or need saved passwords). I tried entering the password directly into iChat (faster than getting it into the Keychain at that point), but again iChat's Encryption Assisstant failed with a misleading error. As soon as I cached my .Mac password, the Encryption Assistant worked. Two bugs (not accepting manual password entry, and not using a password stored in iChat preferences) + a misleading error message + forcing the user to inferior security (cached password) in order to get a security feature (encrypted chat)!

I find the Apple Keychain very useful, but need to protect my keys -- apparently more than Apple anticipated. On all my machines, I used to have my login keychain set to close after a couple hours of "inactivity". This was very intrusive, as it also locked SSHKeychain, even if I was actively using ssh keys (since Keychain didn't know SSHKeychain was actively being used). I recently discovered the security command. Now I have a cron job which flushes my login keychain when I'm done working for the day/night on all my machines, and it's enabled me to crank up login's inactivity timer. Twice a day, cron runs "security lock-keychain ~/Library/Keychains/login.keychain".

Locking the Keychain has revealed several problems (which I have reported to Apple). First, .Mac syncing assumes the Keychain is always unlocked. This makes a certain amount of sense, but means .Mac syncing doesn't really work unless you leave your .Mac password always available in a keychain. To address this, I use a separate keychain (lowsec) for the relatively low-security .Mac password -- lowsec never locks itself (until I reboot). I'm not happy about this, but it's the best I could do without giving up the one really useful .Mac service.

This leads to the second problem (a straight bug): extra password prompts. If I use my account password for my login keychain, it should be automatically unlocked whenever I unlock the screensaver (just as it's unlocked when I log in after a reboot). But it isn't. I still get prompted to unlock my login keychain after unlocking the screensaver. What's worse, I often get a couple prompts for my login keychain, and on an untrusted laptop I typically get a couple more for lowsec (for a stalled .Mac sync, and to set my iChat status). On the laptop, SSHKeychain is set to lock my keychains when it sleeps, and SSHKeychain is not selective -- it locks all keychains (also reported to Bart).

The untrusted laptop found another serious problem. I don't trust FileVault, so I created a 2gb AES-encrypted disk image with Disk Utility. I moved ~/.ssh and ~/Library/Keychains (yes, I know about ~/Library/Caches/) onto the disk image, and created symlinks from the original locations. The bug I found is that, if the Keychain can't be used (perhaps, say, because ~/Library/Keychains is a dangling symlink, and it's configured to use one or more keychains there), Disk Utility is unable to present the password dialog for an encrypted disk image. This meant because I'd "broken" the Keychain by dismounting the disk image containing my keys, I was unable to remount the disk image to "fix" the keychain, because the broken keychain was preventing disk mounting! I escaped the vicious circle by moving aside the symlink and creating a new ~/Libraries/Keychain directory, which un-broke the Keychain enough that I could get the disk image back online. Now I leave ~/Library/Keychains/lowsec in the normal place, so the Keychain is functional, and manually specified the locations of my (normally unavailable and mostly empty) keychains on the disk image, so they're available when it's mounted.

iSync and .Mac Sync are great. I keep my Treo synchronized with three different Macs, and they all have the same data. Unfortunately, while Apple was enhancing the capabilities of .Mac Sync during the Tiger betas, they broke a bunch of things (including compatibility with Panther). Things have gotten better since then, but I still see problems on about a monthly basis. Symptoms are data not making it over and ".Mac login failed." errors from the .Mac System Preferences pane. Apple has a "fix", but it's pretty draconian, and Sync Services tends to get corrupted again. Here's the procedure:

Disable .Mac syncing (if applicable)

Close all programs. Log out and log back in.

Make sure no programs are running after logging back in. If the user is [the original email I received trails off here. --cp]

Create an archive of ~/Library/Application Support/SyncServices. Delete the directory after archiving it

Re-enable .Mac syncing (if applicable)

Sync with .Mac: go to the advanced tab, click Reset Sync Data and choose to Reset Sync Data on .Mac or on this computer depending on which one has the most recent data

If you have devices, now is the time to manually synchronize them (manually add any new data from the device that is not on the computer), then do a sync with the device choosing Erase data on device then sync

I add step 0: Back up your data (iCal & Address Book have handy "Back Up" items, and Safari has "Export Bookmarks"). When logging back in, hold down the Shift key to avoid automatically launching Login Items.