Every day I experience life in the world of healthcare IT, supporting 3000 doctors, 18000 faculty, and 3 million patients. In this blog I record my experiences with infrastructure, applications, policies, management, and governance as well as muse on such topics such as reducing our carbon footprint, standardizing data in healthcare, and living life to its fullest.

Monday, September 22, 2008

When Technology Gets Too Complex

At home, my family relies on me for tech support. When a home requires an MIT trained CIO to keep it running, you know that technology has gotten too complex for the consumer. Here's the story.

I have Verizon FIOS at home. Despite a rocky start up it's been a fantastic service, offering me consistent 20 megabit/second download speeds.

Verizon has partnered with ActionTec to provide a wireless router called the MI424-WR to home users.

As one of the first FIOS subscribers, I received Hardware Revision A of this device.

Verizon automatically pushes firmware upgrades to their FIOS customers and on January 3, 2008, my ActionTec router received the 4.0.16.1.56.0.10.7 upgrade.

Since that point my home wireless connections have become unstable, DHCP leases cannot be renewed, and my family has made a ritual of rebooting the router several times a day.

Our household echoed with commentary that was like a scene from the Waltons. Goodnight, John Boy. Mary Ellen, can you reboot the router?

I searched the web and found hundreds of similar stories. Each one contained a "fix" that sounded more like voodoo than technology.

Hardcode the wireless channel to 11 and it will solve the problem.

Begin DHCP IP assignment at 192.168.1.150 and it will solve the problem.

Limit wireless to 802.11g only and it will solve the problem.

I reset the router to factory defaults and tried each one of these solutions. Since each required a router reboot, it appeared to solve the problem temporarily but in the end the router was still unstable.

My comprehensive review of all the blogs, forums, and user sob stories all over the web suggested one final solution. At this point, ActionTec has redesigned the MI424-WR four times and the new circuit board, hardware Revision D, contains completely different wireless networking chips

Interesting that ActionTec has redesigned the device to address some technical issues, but Verizon does not have a customer notification or router replacement program. It also seems strange that one set of firmware revisions could work well on 4 different hardware revisions.

I called Verizon and spoke to a very competent and helpful person. I explained the history of my problem and the various attempts I'd made to solve it. I requested an exchange of my Model A router for a Model D router. She checked with management and agreed to the exchange. She did note that Verizon provides a wireless router but does not support wireless - it's a use at your own risk service.

Two days later the Model D router arrived and it was clearly a major technology change. The form factor of the device was half the size of the Model A.

I connected the Model D router and could not connect to the internet - wired or wirelessly.

As a CIO, I predicted that the Verizon uses MAC address filtering to prevent rogue connections to FIOS. Thus, Verizon would have to activate the MAC address of my new router in order for it to work.

There was no documentation in the router package or user manual that suggested this step, it was just my CIO intuition.

I called the automated Verizon FIOS service number and it identified me as a customer who recently received a new router. The system then offered me the option of activating my new router, which I did. Internet connectivity was restored in 90 seconds.

Dumb question - if this is required and expected of all consumers, then why not put a big label on the router - please call to activate?

I then began setting up wireless connections. The manual that accompanied the router had a MAC address and WEP key printed on it for my router, so I used that information in my laptop setup. No luck.

I then logged into the router via a wired connection and checked the MAC address and WEP key assigned to the device - they were completely different than those provided in the printed documentation.

A week has passed without a router reboot. After 9 months of struggle, the problem is resolved.

The bottomline of all of this is that Verizon did a great job getting me the router I needed, but it took a CIO with 25 years of technology troubleshooting experience to diagnose and repair the problem. Clearly, the combination of Verizon's policy to not provide full support for wireless, the bleeding edge technology provided by ActionTec, and the lack of change management caused by numerous hardware and firmware upgrades is too much complexity for the average consumer to deal with.

Between the HP printer woes of early September and this Verizon/Actiontec issue, my family is convinced that every modern household needs its own CIO. Now you know why Geek Squad and similar home technology consulting services are likely to be a growing business.

What you say is so true. I have 30 years of IT experience. I find that I'm able to solve ~85% of the problems I see right away, 5% take some research, 5% take a lot of research, and 5% I never solve. This is after 30 years of experience.

I often wonder how non-technical people are able to solve those bizarre problems that non-technical people seem to attract. If I were a teenager I'd learn about how to keep a computer working. If I was any good I'd never have to work in a fast-food restaurent because there would easily be enough work to keep me going forever.

I have had Verizon FIOS since it came to Malden and havent had any issue except when I tried to switch to Linux and it wouldnt recognize the router. I might have to contact Verizon about getting my router replaced.

As a self professed IT geek - I am truly amazed at the complexity that has been "deployed" to the public. I can't help but think about fielding the next call from my somewhat technically capable mom when she calls and says: "Frank, the Internet is down, can you fix it?" We really need to make this stuff easier to use.

I am out here in California and guess what, Verizon does have a contract with the Geek Squad. I found that out a few months ago from a customer service rep while involved in some idle chat while working on my router. She said they could not do it all the time, but in cases where the consumer was just not getting it, they could authorize an appointment from the Geek Squad. The level of geek though that you get to come out depends on the store though.

I used to do some retail training at many of their stores with Intel about processor platforms and just like anything else, there are some very sharp ones and then there are the others.

So in short, we have reached the plateau of endangering what the average consumer (and some geeks like myself too)can bear:)

Dump the router and run your own. I use a FreeBSD based distro called pfSense running off a USB stick. Works like charm.

Then I use my own WAP, have better access control, etc.

Only issue: you either have to drop the WAN lease before you power off their router or wait ~2 hours for the old lease to drop before your new firewall will pickup a WAN lease. The call you made automated this process on their end.