Tuesday, April 28, 2009

As the swine flu (dare I say) hysteria was building over the weekend, my boss thought it was time to test our nimbleness and mettle. This story is growing -- that's born out in the news tonight that a close-by school district is shutting down for two weeks, right in the middle of TAKS standardized testing -- and won't be going away anytime soon.

So the charge for my team yesterday morning was to build a news aggregation site by the end of the day. Alas, we didn't quite make that deadline, but it was an, um, eventful day. But we did have a site going some 30 hours later, though DNS propagation is still keeping us from seeing it within our own building.

It went down like this:

Domain searches started some time Sunday or early yesterday, from what I can gather. All the good swine flu domains are long gone ... swinefluinfo or swinflunews, etc. Many are for sale, of course. (I hate domain stalkers.) We ended up with swineflucare.net, certainly not among the better choices, but it's what we had to work with come Monday morning.

It was about 9ish when my team was brought into this effort. The charge was laid out: Build an aggregation web site under this new domain. It needed basic blog entries, links to other sites, resource links to CDC, etc., simple metrics, maybe a twitter feed and/or news feed. We'd start with Google AdSense, more for SEO than revenue, but we figured we could graduate to Yahoo! APT if we felt need.

The team got together and discussed CMS platforms. We wanted something more than hand-coding. We've done a lot of development work in Django, so that came up, but we've also been doing a quite a bit of work in WordPress, and we were all pretty sure that was where we were headed ... I just wanted to talk it out.

Wordpress was the easy choice, and for ease and quickness we probably should've started at wordpress.com and to get going. But we knew SEO-wise we wanted our own domain. Where do we put it then? We host three external WordPress environments now: An MU environment for our community papers and a more traditional site for our Spanish language daily (ahorasi.com) -- both hosted by an external vendor -- and our Collective Vision photo blog, on our own load-balanced servers. That load balancing thing was a trick, so we opted to piggy-back on ahorasi.com.

That decision has caused much grief for us over the last two days, but I'll get to that.

So Andy Nguyen, who championed bringing WordPress into our publishing environment, started going to work on this new site. Within a couple of hours, he had most everything working ... blog rolls, a juitter feed for the #swineflu Twitter hash tag, blog entries direct to outside stories ... just damn nice, rock star work. He was working with our host, Westhost, to get all going, told me we have to upgrade, used my Amex and we are off to the races. We could get this thing launched by late afternoon, I'm thinking.

What I didn't understand is this upgrade wasn't a simple server reboot that would take a couple of minutes ... Westhost had to move our existing ahorasi.com site to a new server, with a new IP address. (For the un-tenchnical, that's like you moving and changing your address with the U.S. Post Office. On the Internet, it takes 24-48 hours for everyone to learn the new address.)

To make matters worse, when Westhost upgraded us, and when we changed our DNS to point to the new server IP address, the new server was not actually up and functional yet. Remember, we had an EXISTING SITE on that old server. So suddenly, DNS started propagating to this new server and it didn't exist! Not only did the site go blank, but we couldn't log into it, we couldn't ssh into the new server, nothing. I thought we were in for a short downtime, but now the Publisher and editors of the site are saying they can't get to it, and I'm learning it could be 24 hours before we get back. This Swine Flu story is huge for them ... this is a Spanish language website where the main following is from Mexico, the epicenter of the outbreak.

I was sick to my stomach. The worst day of my TSG career. Ahorasi.com isn't the most visited site we work on, by far, but I hate the way this happened, on what was a really, really big news day for them. They were trying to do the right thing ... good journalism specific to their market, and the rug was pulled out from under them.

It was about 6 p.m. when the new server came up and ahorasi.com was back online and publishing. Of course, work on the swineflucare.net site had stopped when the server went down, so Andy still had to finish up today and we ended up launching by about 2 p.m. The only initial feature we didn't have that we targeted was the Google AdSense ads, and that's just because we didn't have an existing account and Google has to approve it. They say that could take a week.

Maybe we could've done some setup faster, but I think creating on a new domain in a short timeframe will always be tricky. Anyone have any suggestions on the quickest way? I'm sure this isn't the last time we'll do this. I'd like to make that by the EOB deadline next time.

Wednesday, April 01, 2009

My paper started a new Statesman News Update podcast (and the blog that goes with it) this week.

Our Internet editor Robert Quigley sent a shoutout on Twitter to announce it. One of the suggestions he got back was the need for an on-page audio player, which we didn't really have in our web toolkit. The reader pointed us to KUT for an example.

Well, some quick research showed they use a free WordPress plugin by 1 Pixel Out for their audio player, but we don't use WordPress on our home page. 1 Pixel Out points out a tutorial by Journalism professor Mindy McAdams that dissects that plugin, which I used to whip up a player in about half an hour.