Dan Crow is Google's product manager of crawl services - overseeing of the Googlebots that go out and find what's on the web. We asked him about the problem with indexing Web 2.0; whether Google is working with Adobe; the dark web - and the future for robots.txt.

BJ: There are lots of developments in the way content is delivered on the web over the past few years. What are the biggest problems with crawling at the moment?

DC: We do a really efficient job with HTML, but with Flash we could improve. If we hit a web page with a Flash movie on it, we just extract the text out of it and index that text. But a Flash movie is much richer than that. So one of the projects I'm working on is to try and improve our Flash processing - that's an example of an area where we could be better. But it's not unique to us.

BJ: There's an irony in the fact that you're unable to do any complex crawling of the videos on YouTube, despite the fact that it is owned by Google. Do you have to try and analyse the content of Flash videos, or find a way for it to include a lot of metadata?

DC: At the moment our advice is that webmasters need to give us a lot of help - a Flash movie is basically a set of virtual pages. If you used HTML for the links, then we'd be able to see the overall structure. The content will still be in Flash, but we can at least get some of it. Ultimately we'd like to be smart enough to look inside the Flash movie. We're not quite there today.

BJ: Are you working with Adobe to build better crawling data into applications like Flash?

DC: I can't talk about that.

BJ: Because Flex and Apollo would offer a new chance to find a baseline.

DC: Absolutely. That's not an unsensible observation. We clearly need to figure this out... it's not been the highest on our priority list, but in the last six months it has become more so.

BJ: Do you think there are formats which could emerge that are completely uncrawlable?

DC: I don't know of any emerging formats that fundamentally change what we're doing. Ajax is probably the closest thing to that. Of course, it's always possible, and we try to keep our ears to the ground. But so far our model of crawl, index, search has fundamentally sound. My guess is that it will stay that way for a while.

BJ: What about the dark web - sites that are unlinked, and therefore invisible to you? How good is Google at understanding what else is out there?

DC: Well, crawl certainly has limits - the physical limit to how fast we can crawl, both on our machines and on the machines we are indexing. What that means is that we can build a seed list of sites that's bigger and bigger, and crawl them at different frequencies.

But we don't quite know how much of the web our seed list covers. We've got a good estimate, and but we can't be sure of exactly what percentage of information online we are covering.

BJ: We've heard a lot recently about copyright, and the recent Belgian ruling showed that companies are concerned that indexing their content is unfair. Do you think companies concerned about your indexing simply use robots.txt to block spiders?

DC: We could certainly do a better job of educating people about how they can use robots.txt. It's a technology everyone knows about, but it's not always immediately obvious how to use it. There are lots of ways you can employ it. It's surprising how sophisticated a policy you can create with robots.

And it's a robust system. The nice thing about it is that it's a standard that everybody uses - Google, Yahoo, MSN and others.

BJ: What about using robots to tell Google to destroy information in its index? The Belgian case revolved around cached news stories that had disappeared behind a registration wall.

DC: We've heard that request before - "after date X no longer show this in the index". It seems like a reasonable request and I can see that there's something that could be done in the future.

It's not formally a web standard. I think we might eventually revive that effort.

Ultimately it is in all the search engines' best interests, and the public interest, to have a good mechanism. I think we're doing a pretty good job so far.