“When I joined the company, Google was not crawling the Times,” said Marshall. “I’m pushing to get more traffic to the site.”

In a series of telephone and email interviews with Marshall, we learned what steps he took to change things at the Times to make the company’s web sites more search friendly. Most of the changes he made will be of interest to anyone responsible for in-house search engine optimization efforts.

The biggest problem for the Times with search engines was one common to most newspapers: The Times required users to register to read an article. Registration forms effectively block search engines from indexing a web site, as the crawlers can’t type a user name or password to access articles.

“Yahoo had indexed our registration page 20 million times,” quipped Marshall, but that turned out to be a serious issue. Yahoo’s crawler recognized the importance of the Times’ web site, but was unable to do anything but hammer away on the registration page that was ultimately displayed whenever the search engine crawler attempted to access a Times article by following a link.

To put it kindly, I would say it was extreme non-genius on the part of the New York Times that their archives weren’t already crawlable, but, hey, that’s part of why the newspaper industry is in deep doodoo.

Follow me on Twitter

The Missing Runner

Click photo to buy it on Amazon or read free sample

In January 2014 Jeffrey Boucher — a completely normal guy — went out for a run and never returned. The police still don't know what happened to him. This short e-book (magazine article length) presents my theory.