Pages

Tuesday, November 24, 2009

I've been chasing a simple attachment issue in Lotus Notes 8.5.1 with IBM support for a while now but we're not really getting "engagement" yet. I thought I'd document it and see if we're the only ones with this problem.

The Problem

What we've discovered is that if we attach a file using a Lotus Notes 8.5.1 client, everything behaves as per normal in the Notes client but if we use a web browser to look at the NotesDocument, the file name disappears.

It's best illustrated.

On the left, you can see how the attachment renders in the Notes client using a standard discussion database. On the right, you can see how it renders in the web browser. There is no filename. It's not white-on-white, it's simply not there.

Our Parameters

It only happens when we use Lotus Notes 8.5.1 clients, but it's not a problem with 8.5

I've only tested the problem on Microsoft Windows XP Service Pack 3

It happens on lots of computers, regardless of brand, configuration and memory.

Our Domino server is currently still 7.0.2 (this might be the problem?)

It happens with customised databases and with the standard R7 and R851 templates.

IBM Support

As I've said before, I haven't used IBM support (other than online discussion databases) in the last ten years because the online community was so much better than support. I opened two PMRs recently mainly to test support.

I've been sending a few emails back and forth but I'm running out of ways to describe the problem. Right now, we're just six days shy of the one-month anniversary of my original submission (I'm glad I marked it urgent) - hence my comment about not really getting engagement.

Wednesday, November 04, 2009

It has been a busy week for disaster recovery so far - we had an outage on Monday and we had another one today.

Monday's Email Outage

On Monday, we lost our e-mail services. No, it had nothing to do with the Lotus Domino server. That was fine.

At first, it looked as if we'd forgotten to renew our domain name. This was quickly followed with a back and forward check of various DNS services out there on the Web, our domain registrar and our Internet service providers. All seemed ok with our domain name but there was definitely something weird happening.

Eventually we discovered that our Internet service provider had their DNS running off a domain which they had forgotten to renew. Since they were providing our primary DNS, all of our inbound mail was getting confused when it went to resolve our domain.

After being told by us (yes, they were unaware of the situation despite the fact that it had occurred during the night and it wasn't noticed immediately in the morning), they quickly got to work renewing their domain. Of course, given the sorts of problems associated with domain name propagation, our problems persisted in one form or another for several hours.

Years ago, such a problem would've been effectively "over" within a matter of hours but unfortunately, more and more companies are outsourcing their services overseas, and it takes more than fixing the local domain services to resolve the problem.

Today's Building Outage

Our Wednesday problem occurred while I was at lunch. I hurried back and arrived at a darkened building. Luckily, since we moved offices, we aren't as high up as we used to be and I only had to run up six stories worth of stairs (although immediately after lunch, it felt like more than that).

It turned out that the entire local grid had lost power. Our domino server and our main file server were happily running off UPS but unfortunately the UPS handling our communications gear was not up to scratch. It didn't matter because there was no way that the UPS would be able to power our systems for more than 30 minutes. Even if this was possible, the temperature inside the computer room was rapidly climbing now that the air-conditioning was off.

We had no choice but to start shutting down the servers. It took us a little while to make that decision because we knew we had a little time and we were hoping for the power to come back on. Of course, as soon as we got halfway through the shutdown procedure, the power came back on. This was after a 45 minute outage in the centre of Sydney's CBD.

Once again, the problem was "environmental" and out of our control. We could have switched to our offsite systems but it is a hard call to make because although our systems are clustered, we have a few special requirements which make a partially manual cut-over desirable. When the cut-over is not automatic, it becomes very difficult to make a decision as to when to flip the switch.

Out of Control Problems

The thing that these stories brings to mind is the fact that I keep reading anti-cloud computing "horror" stories from various vendors. In particular, they talk about Google's Gmail outages. I don't personally understand how people can think that cloud computing is any more or less unsafe than normal computing. As I said before, the problems had nothing to do with the Domino server. In fact, I can't remember a time when we've had an outage due to the Domino server.

I can remember plenty of times when we had hardware failures, ISP failures, power, air-conditioning, gas leaks and DNS failures. We've had problems with Anti-Virus and Anti-Spam services running on the domino server - and when we moved them off the server, they still caused us the occasional problem at the gateway. We've even had problems because of Windows itself and device driver updates. It's never domino though. The server product is entirely stable.

In some respects, our Domino mail is in exactly the same position as Gmail. It's not the product that is at fault, it's the underlying infrastructure - and it's out of our hands.

Today, we had a discussion about Active Directory. It's something we need to install, mainly to tick the business continuity box, and not something I'm particularly keen on. It's a messy rip and replace job with very little business gain. As is often the case for Microsoft products, there's no upgrade path. That's right, we're going from Old Microsoft product to New Microsoft product and the only way to do it is to throw our baby out with the bathwater and start again. I'm not happy, but it's no worse than I expected.

Messy Introductions

Our contact arrived during a painful ISP outage (more on those amazing adventures in a different post - soon). We explained about the outage and he started telling us about how he'd been head hunted by them and was considering his options.

Observation No 1: If you are a business person trying to sell a service. It's probably not wise to let potential clients know that you're thinking about leaving the company.

Product Knowledge

Discussions moved on to our systems and I mentioned that I wasn't particularly keen on Active Directory and that overall, it wasn't critically important to us except to tick some boxes. "We're primarily a domino shop", I said, expecting him to understand. I went on to suggest that although we currently have file shares, we'd probably want to replace them with something like Quickr eventually.

Our BP then launched into a giant spiel about how it was dangerous to deviate from the Microsoft Windows platform and how "maybe you know these systems really well but when the time comes to replace you, there'll only be people with Microsoft certifications about".

I stopped him and explained that our Domino servers were running on Windows but that it was great to have a choice and not be tied to a single platform. The BP went a little loony and started talking about this dangerous Quickr thing "whatever it was", and how cloud computing would fall apart as soon as your ISP did something stupid.

I had to stop him again and ask... "Are you guys really IBM business partners?". He briefly answered "Yes" and then went back to flogging the MS view of the world.

Observation 2: If you're going to use the word Business Partner to describe yourself, then you must at least have a passing familiarity with your BP's products.

Observation 3: Nobody likes being told that their roadmap is "incorrect" - particularly not by people who aren't privy to it. That sort of behaviour doesn't sell services.

I have a bunch more observations but they essentially say the same things. As a customer, I demand product loyalty from my service providers. Sure, it's ok to not be happy about moves that a parent company makes (I whine about IBM too - as I'm sure people have noticed).