Tag - Brooklyn

I got caught up in Goldman's layoffs Wednesday. I hope to have a new job before the severance runs out, but if you are (or know someone who is) looking for a Linux/Solaris admin in the NYC area, please let me know.

I actually wish this had happened a day earlier. It was a big comedown from the Obama victory the night before; in the reverse order, winning back the country would have been a nice counter to the layoff news.

After DreamHost's breach 8 months ago, I was aggravated at their poor handling of the situation, but willing to give them the benefit of the doubt, and still happy with their low prices and flexible services.

I have moved Extra Pepperoni back onto my own hardware. I started blogging on Apple's Blojsom install, but gave up on Tiger Server for Blojsom (and Mailman) because the services kept silently shutting down, leaving me to notice they were disabled days or weeks later (no fault of Blojsom or Mailman -- Apple didn't do a good job porting SpamAssassin either). Bringing up a WordPress blog and mailing lists at DreamHost was easy and cheap, but that's no good if they are unsafe.

I'll look at moving a couple very light-duty Mailman lists off DH next, but the lists are so lightly used I'm not too concerned. There just isn't any confidential information on the mailing lists, aside from their tiny subscriber lists.

Ah, well. I now know much more about WordPress and MySQL than I cared too, but the setup wasn't too bad. I hadn't realized how many customizations and tweaks I made to WordPress until it came time to recreate them on my own system:

Apple put a lot of effort into making network sharing (Mac and Windows networking using the AFP & SMB/CIFS protocols) easier in Leopard. One of the things they did was introduce credential caching at the system level, so once you mount another Mac via AppleShare (for instance), you could then connect to it with Screen Sharing too, without authenticating. This is neat, but a bit problematic. I have had cases where:

I had to kill NetAuthAgent (the background process that appears to hold username/password pairs on your behalf) to make mounting work

I had to rearrange windows around onscreen, because a (stalled) progress window was hiding a username/password window, and never going to get anywhere without some help; other times I have dismissed the progress dialog without realizing it was waiting for a concealed window.

I have had to Force Quit and relaunch the Finder before it could (re-)mount some or all network volumes.

I have had to reboot the Leopard server before I could (re-)mount its volumes.

I have had Leopard systems fail to share out volumes, and had to re-share them manually. Part of this appears to be a different issue, where Leopard systems don't even mount additional drives until a user logs in (obviously unmounted volumes cannot be mounted over the network). That's not right!

Tonight's problem was a bit different -- I was connecting to a Windows server running Samba, and not getting the right permissions. When I looked in the server's /var/log/samba/smbd.log (because I cannot find any way to see the account used for a network mount in in the Finder), I discovered that the share was mounted as the wrong user. I had never gotten the username/password dialog for this mount, as I had (the wrong) user credentials cached in NetAuthAgent.

The Tiger behavior is to default to the client username (the account mounting the share from the server). Leopard instead uses whichever user it has a cached credential for. I have now changed my scripts to always specify the username when mounting shares, e.g.,open smb://pepper@inspectore/inspector.

Please call me if you have an account on reppep.com and haven't received your password already, or find anything not working right.

I switched from Apple's jabberd to Openfire, which doesn't use the UNIX system accounts, so let me know if you want a chat account (compatible with iChat & GTalk).

[Done] I forgot SquirrelMail address books -- should be able to bring those over too.

Firewall problem fixed. SMTP MX issue fixed.

Virus filtering problem fixed.

Webmail certificate fixed.

Quota problem fixed.

Virtual domains for email fixed.

As of 5pm, I don't know anything that doesn't work (aside from SquirrelMail address books) [fixed Thursday].

Thanks for your patience!

As of 10:30 on the 20th, things seem to be working. Something's screwy with amavisd-new's quarantine, but mail is going through. I reinstalled Openfire, and chat seems okay under the correct hostname/certificate name now (will try signing it as ca.reppep.com later).

Good timing -- the optical drive on the old server died tonight.

I have distributed all the new temporary passwords, so any users having trouble logging in should let me know.

Markdown.cgi is still broken, but I'm the only person who uses it here, so I'll get to it.

On Thursday the 21st, I found a problem with amavisd-new -- it had quarantined 32,000 messages in a single directory, and was stuck (apparently ext3 doesn't support more than 32,000 files in a directory). I cleared it out and finally managed to disable quarantine, which wasn't as easy as it should have been, and the backlog of messages have been delivered as of 9:15pm.

At 11pm, I fixed an issue preventing SMTP AUTH from working properly, which was interfering with sending email to non-reppep addresses.

For several years, I've been saying Apple made a bad choice when they picked Cyrus IMAPd as the POP/IMAP server for Mac OS X Server. It's a huge and complicated system, encompassing IMAP, POP, SSL, Sieve filtering, LMTP delivery, USENET news, clustering/proxy (Murder), pluggable authentication (SASL), etc. I cannot think of a single company outside Cupertino where it would make sense to run an enterprise mail system on Mac OS X Server, but Apple continues to add these inexplicable high-end features to its mail server, most recently XSan-based email clustering in Leopard Server.

The statement that convinced me (shortly after I had migrated to Cyrus IMAPd on Mac OS X Server 10.4 "Tiger") that I would never choose to run Cyrus for my personal use, was the following -- which I came across again today:

This system should be expected to have the same order-of-magnitude installation complexity as a netnews system. Maintenance should have similar complexity, except administrators will have to deal with creation and deletion of users and will have the option of managing quotas and access control lists.

USENET news is infamously demanding and bandwidth intensive. It would be wonderful if Apple had taken Cyrus IMAPd, repackaged it (without too many changes!), and put a powerful and simple interface on top. The did this quite successfully with Apache httpd (although Server Admin breaks down on complicated configurations and has obscure bugs). Lots of people use Mac OS X Server to run websites and think it's easy & simple. Considering the typical reactions of those same people to the httpd.conf files "under the hood", this is a noteworthy triumph. Similarly, Time Machine provides a reasonable approximation of scheduled snapshots on a high-end NAS for do-it-yourself file recovery, with a simple interface that insulates users from the nitty-gritty of copy-on-write and hard links.

Cyrus did not get as much attention, though. Basically, Apple makes it pretty easy to create email accounts, provides a Repair button for the overall Cyrus database, and provides a Reconstruct button for individual accounts. That's about it. Unfortunately, Apple doesn't really document maintenance beyond "press the button and it will fix your problem". I've had several serious database problems which Apple's Repair button did not help with. Those were bad times.

Similarly, I have had problems where users could not log in, but Workgroup Manager claimed their accounts were usable. I eventually discovered that resetting passwords with passwd works sometimes, and re-setting passwords in Workgroup Manager works consistently, but when I asked Apple about it, the eventual response was basically, "Yes, that's bad; you should restore your accounts from your recent Open Directory export." Not a good answer.

It doesn't help that Apple's SpamAssassin and ClamAV installations are broken, as these result in more spam and slower deliveries.

So why am I planning to migrate to Cyrus IMAPd on CentOS 5.1? Well, I'd really like to just copy my 5gb mail directory to the new system and have my clients not notice the difference. Eudora doesn't handle (IMAP) change well -- renaming a single IMAP directory can force it to download all messages again, and various other things can cause Eudora to lose date stamps on sent mail, or message state information (when it gets disassociated from the actual message on the IMAP server). If I can make Cyrus work, I'll be very happy, and if I can't I'll try Dovecot (Red Hat's default) or Courier (which I hear is also good).

Also, I know it can work, and I have a rough model to work from on my Tiger Server, but if I wasn't using Cyrus already I would stay away from it, as I wish Apple had done.

For over a year now, I've been following the development of Mac OS X Server 10.5 Leopard and testing betas, and anticipating upgrading reppep.com from Tiger Server on a dual 1.25GHz Power Mac G4 to Leopard Server on a dual 2GHz Power Mac G5. Over the weekend I had a change of plans, though.

Although I support Mac OS X Server at Rockefeller, I don't recommend it for most requirements, as Linux compares favorably for transparency (some of the MOSXS internals are unique and poorly documented), server software compatibility (although Macs are quite good here too), and price/features at the low end. A Core Duo Mac mini has plenty of juice to saturate our 768kbps/3mbps DSL circuit, but adding a couple drives more than doubles its price, and Apple's software RAID is quite broken; Linux software RAID is apparently quite good; I might eventually switch to hardware RAID. An Xserve is a great piece of hardware, but it's a bit exotic and I can get a fast generic PC cheaper; I don't want all the high-end features for a box that sits in our apartment.

Additionally, I've read perhaps 600 pages of docs on Leopard Server, and had at another 400-1500 yet to go. This is an investment I was finding hard to justify. The migration process is quite complicated, and Apple doesn't support migrating accounts from a Tiger system to a Leopard system -- I don't want to do an upgrade. I could clone the G4 to the G5 and upgrade it there, but I prefer to handle upgrades as scratch installations with manual migration of applications, so I know exactly what's been done. A lot of this is masked by upgrade procedures.

As part of this, I've decided to invest a bit more time in learning RHEL5 -- we have a couple systems at Rockefeller, but not much in production yet, and now seems like a good time to dig in some more.

Fortunately, all the services I've been using on reppep.com are available on Linux (and FreeBSD), so aside from another incredibly inconvenient password change cycle (for which it is arguably time anyway), the switch should be largely transparent to reppep.com users, although I still have plenty of research to do.

1999: I left the National Audubon Society, and bought the Power Mac 7300 with accelerator card I'd been using there. I set it up with LinuxPPC and Apache, and started offering free web hosting to friends & family. LinuxPPC was eventually discontinued.

I upgraded from LinuxPPC to Yellow Dog Linux, which was better than LinuxPPC, but had serious flaws.

2001: I was working on a couple remote FreeBSD machines (as admin of the Info-Mac server, and a user on the Apache Software Foundation userhost), and decided to learn more; I bought a cheap Celeron PC and installed FreeBSD 4.3 (IIRC); I upgraded through about v5.1 and a Pentium 4 (giving the Celeron box to the Info-Mac Archive, where it became the Info-Mac server for a while). I learned a lot about FreeBSD and UNIX in general, but eventually realized I was investing more time learning FreeBSD than I could justify. The best thing about FreeBSD is not a technical feature, but rather that the user community is so rich with knowledge. Reading the FreeBSD-STABLE list was amazing, as there was so much depth, freely shared with the community. While running on FreeBSD, I added mail services to the web services I had been offering. Note: Disruptions to personal email service are much worse than problems with personal web service.

2005: It became clear that I needed anti-spam, so I began researching SpamAssassin. While I was figuring out how to build the SMTP sandwich, with a public untrusted Postfix listener on port 25 & 587, and a filter, and then a listener on a high port like 10025 to accept and deliver mail to actual users, I installed a beta of Mac OS X Server 10.4 "Tiger", which had the whole thing implemented, plus ClamAV as a bonus. I started testing heavily before the release, and switched to MOSXS 10.4 shortly after it was finalized. It's been very good, but as time has passed, I've had more and more problems. In particular, Apple chose to use Cyrus as an IMAP/POP server, and Cyrus is complicated, but Apple ignores the complexity; this can make troubleshooting impossible. The SpamAssassin installation is slightly broken; it's a bit too old to offer the newer SpamAssassin self-upgrade mechanism. Server Admin is great, but has a bunch of bugs around SSL certificates, some of which destroy the certificates. Blojsom was nice, but Apple's installation was very unstable; I eventually moved my blog to WordPress hosted externally.

2008: I intend to switch to CentOS 5.1, which is basically a (legal) no-charge clone of Red Hat Enterprise Linux 5.1. This should make future upgrades a bit more straightforward, as I won't have to deal with Apple's Open Directory (OpenLDAP); it will also give me a bit more experience with RHEL5, which is a better investment for my time than Leopard Server.

Time Machine has a hidden feature, to "Exclude All System Files". In Leopard Server's Standard mode, Time Machine is a service, and in Server Preferences you can control whether clients back up their system files, or skip them. This is logical -- for personal backups you want everything, but if you have enough users to justify a file server, you might well not want to back up the same Leopard system files for each user.

Today's handy-dandy discovery was that Mac OS X Leopard "user" has this feature too, but there's no visible knob to turn it on. Interestingly, I cannot find such a control in Server Admin either, which could be my oversight or could simply be a bug (I've reported it, anyway).

Instead, if on the client you add /System to Time Machine's list of directories not to back up (I also skip /Developer, /sw, and my music files), Leopard pops up a handy dialog, asking if you really want to "Exclude All System Files". I chose yes, although I'd like to know exactly what (directories) are excluded by this option.

10.5.0 Server: I have 3 interfaces. The onboard GE is broken, so I have GE cards in PCI 2 & PCI 3. When I delete Ethernet in Server Assistant (at installation time), it crashes and bounces me back to the Welcome screen at the beginning of the process. I can, however, just not leave it at DHCP (physically disconnected) and get through; I think I could also turn off TCP/IP for that port, but haven't verified in 10.5.0 GM.

I have been reading about Mac OS X 10.5 Leopard Server and non-Server lately, and I was surprised to realize how many different authentication & authorization systems are running each with its own timer.

Access to the "console" (keyboard/video/mouse): Ends when you log out or the locking screen saver kicks in.

Authentication for administrative actions in Carbon/Cocoa programs (such as modifying system directories in the Finder): 5 minutes (I believe).

Alas, they cannot; this means there's no way to manage a Leopard and a Tiger system from the same box without screen sharing -- something I and every other Tiger admin would like to do for the upgrade process! More generally, people who have upgraded to Leopard must use screen sharing to manage our Tiger Servers. Let's hope Apple fixes this quickly -- rather than after most of us have slogged through our own upgrades.

Setting up new Leopard (user) and Leopard Server systems, I was shocked to discover that, in System Preferences:Accounts:Guest, "Allow guests to connect to shared folders" is enabled by default. This is lousy security. Fortunately file sharing is off by default, but I turned file sharing on a day or two before I noticed guest access was enabled. I don't want guest access enabled, and it shouldn't be on without a conscious decision.

A bizarre and perverse journey is completed. At 12:21am 2007/10/19, I reported my 1,500th documentable bug to Apple. I have actually reported a bunch more over the years, which have since been lost to the sands of history. I remember reporting bugs against eWorld and Newton beta software! But I can currently identify 1,500 bug reports against Apple's products.

A few of these, of course, are bogus -- there have been times I just made a mistake, and thought it was an Apple problem. Some of my mistakes indicate that Apple's user interface needed clarification or improvement; others are simply my foolish mistakes.

Many of my reports are documentation issues. Right now, I'm looking at Apple's thousands of pages of brand-new documentation on Mac OS X 10.5 "Leopard" Server and sighing (repeatedly) -- I don't have time to read the half on topics that interest me -- but as an admin, the documentation has to be correct. Rockefeller has an Apple Enterprise Support contract, but they are limited, expensive, and problematic to use. Most Mac admins have to make do with peer support, and Apple supports this because Apple only has to support (some of) the fora -- not pay support staff. This means Mac admins need to be able to help ourselves through researching and the documentation. Ambiguous or simply incorrect documentation is bad. Fortunately Apple aspires to perfection (though they don't always aspire very hard -- the early Mac OS X manual pages were badly neglected).

Other reports are feature requests, handled slightly differently but through the same bug reporting system. For example, I want to use my iPhone as a secure password store, an offline web browser, and with a Bluetooth GPS. Feature requests are how I tell Apple my priorities for product development -- sometimes they even pay attention! ;)

And lots of bug reports are bugs. This is a bittersweet time, as I recognize my reports behind a bunch of fixes in Leopard, but I also know I'm about to lose a lot of traction. Until very recently, Apple has been focused on perfecting Leopard -- meaning things have been fluid and could be improved, and there was lots of pressure to fix bugs. Now that they have finalized 10.5.0 and are preparing to sell it, the developers are hoping they didn't miss any hideous bugs and recovering. In a little while they'll go back to the grindstone to start fixing and building 10.5.1, but 10.5 will never be as flexible again. It's going to be a while before I can start pestering Apple about what to do for Mac OS 10.6, and various bugs or design flaws will be too large to build into a point release, meaning they are already baked into the 10.5.x series, not eligible for fixing during Leopard's lifetime.

After a discussion with Rich Mogull, where we both agreed that AFP is a threat (note that Apple fixed 4 different AFP threats in Security Update 2006-004), I decided to require ssh tunneling for AFP connections to www.reppep.com. Apple provides a neat feature for automatically tunneling AFP through ssh, but unfortunately it's broken in half a dozen ways...

My initial report:

It is impossible to connect to an AFP server without access to port 548 -- this should work if ssh is available, and AFP-over-SSH is enabled. Instead, with 548 blocked by a firewall, the AFP connection times out -- even using an alias created when connected via AFP-over-SSH.

Connect To Server should accept afps://host as a scheme that specifies AFP-over-SSH. Instead it gets converted to afp://afps/host, which is wrong and nonfunctional.

It's impossible to require ssh for AFP from the server.

It's impossible to support AFP on the server without leaving port 548 open, even though with ssh tunnelling 548 shouldn't be needed.

Note: These are not exploits, but they are real problems with the security of Mac OS X (Server