Microsoft has a plan to improve Bing’s poor indexing

Microsoft has publicly admitted that Bing is slow at indexing and has outlined …

Recently a user on the Bing community forums asked why his new site was taking so long to be indexed by Redmond's search engine, to which a Microsoft employee replied that the slowness was an unfortunately expected behavior.

User prathaban1 wanted to know why www.kidandparent.in, which he submitted to the Webmaster Console using a sitemap.xml almost six weeks ago (and Bing calculates has about 14 backlinks) was not getting any love from Microsoft. He said he managed to get the homepage indexed with the help of a Brett Yount, Program Manager of the Bing Webmaster Center, but that was nothing compared to what Google and Yahoo had done: they indexed almost 400 and 200 pages, respectively.

In just four minutes (if only Bing indexed that quickly) Yount had posted his reply. The response wasn't good news. "It is well known in the industry that MSNbot is fairly slow," he wrote. "I suggest reading our FAQs stickied at the top of the indexing forum to get some ideas of what to do." The instructions Yount is referring to are outlined in a forum topic titled Sites not in the index:

[I]f your site is not in the index, please do the following:

verify in our tools that your site is not blocked

run a site: query to verify there are no pages in the index

Copy the URL of the site query and post on this thread.

I will work with you to at least get your home page indexed. Deeper indexing will require good content and backlinks as described in the FAQ.

Scanning through the thread, it appears that Yount is spending a lot of time telling users what they are doing wrong that results in Bing indexing their site incompletely or not at all. In other cases though, he is contacting the Bing indexing team so that they can figure out what is going on wrong on their end. At the time of writing, the thread had 858 replies with the first one posted on November 11, 2009.

MSN search, Live Search, and now Bing have all suffered from a very small index that is updated very slowly. One of the biggest problems Microsoft has with Bing is a basic one. While Bing has a lot of great features, it continues to struggle with the same issue previous incarnations of the service have. The actual index size, and the corresponding relevant results that come with those searches for the less popular queries, pales in comparison to what Google and Yahoo offer.

We asked Microsoft how it was planning improve Bing's indexing problem. "We're always working to improve the crawler," a Microsoft spokesperson told Ars. "With our latest crawler release still in beta, we doubled our crawling capacity worldwide. We increased our sitemap URL size to 50K and we made it easier for webmasters to control the crawler's aggressiveness."

The trouble is that the competition isn't waiting for Bing to catch up: their indexes are growing at rapid rates as well. Naturally, as the Internet continues to expand, so does the number of sites they index. "But we know we have to continue to build a system which better reflects the changing state of the Web," the Microsoft spokesperson continued. "We introduced the ability to crawl and surface fresh results in minutes last year which enables us to more aggressively index fast-changing domains (like news sites) and crawling itself isn't the answer to the new Web landscape."

That's where Microsoft's new strategy with Bing (compared to MSN search and Live search) comes in: it has become well aware that it does not have the resources available to take Google head-on. As such, the company is prettying up how Bing displays search results, focusing on social networking, and improving areas it thinks users are most interested in when they search, such as health and auto.

"For things like Visual Search and Twitter Search, we work with data providers to get high-quality, structured data that don't have to go through the computationally complex and never-100%-accurate process of mining pages for meaning," the Microsoft spokesperson told Ars. "As we work toward augmenting our decision engine with more tools to help customers actually get things done on the Web, the need for a new notion of crawling and indexing arises. While we're continually focused on how to improve freshness, speed, politeness and intelligence of what has to be crawled or crawled again, we have significant investments in semantic, structured, and real-time indexing that are required to do more than just return URLs for keywords."

To sum up, Microsoft wants to continue to improve in the areas that webmasters expect from a standard search crawler, while at the same time broadening how Bing collects and presents information from across the Web's constantly changing data.

Microsoft's strategy may be spot on for slowly gaining marketshare, but it won't last if webmasters who want to care about Bing are getting frustrated. They aren't interested in waiting around till Microsoft improves the MSNBot to a point where it notices their websites automatically, and the solution Microsoft currently offers is flawed. Microsoft hopes that the new Search Engine Optimization Toolkit it posted this week that works with Google, Yahoo, and Bing will help.

23 Reader Comments

Anecdote time! I'm responsible for basic SEO for our website launches (about one per week on average). Using the search engine submission tools, Yahoo and Google are almost always indexed within 1-2 weeks. Bing often takes 6+ weeks to index.

Back when they were relevant, there was the 'official' Yellow Pages, and then there were the competitors who only offered a subset of entries. Guess which one I reached for when hunting down obscure parts or compiling a list to comparison-shop?

This totally sounds like "We've given up on trying to catch up to Google, or even keep up with them. Instead, we're going for crawling fewer, bigger sites, and pretty much ignore all the little fish. That's OK because they don't pay us anything anyway, unlike the bigger sites. So, basically, you either have to be very popular or pay us money to show up in our results."

This is what happens when you get into a sector because you think you should be in it, rather than being in an area where you are passionate about what you do... Bing will always be the lack-lustre, slightly shit also-ran. If they had left Yahoo in charge of searching at least they would've been interested in doing it well...

Originally posted by Joel_B:This is what happens when you get into a sector because you think you should be in it, rather than being in an area where you are passionate about what you do... Bing will always be the lack-lustre, slightly shit also-ran. If they had left Yahoo in charge of searching at least they would've been interested in doing it well...

Well, MS has been in the search business for some time now, although failing abysmally hehehe. But more to the point, maybe that's the reason they actually tried to buy Yahoo a while back?

In the grand scheme of things, I can't really see the role of Bing in MS's business plan. In any cas, I think despite its shortcomings Bing performs reasonably well in terms of relevant results, and the image search is far better than Google's. So let there be Bing, if only for the sake of competition!

Having to post URL's in a forum thread to let them index it? LOL, what a misery.

And of course they're telling that they're working towards better indexing. There have already been two revolutions from Redmond, two "Google killers". Windows Live Search and now Bing. And that's where we are now with this amazing indexing. Suggested to post URL's to a forum thread, with a straight face.

Werent Ars and a load of other reviewers saying Bing was as good as Google when it launched?

Has Bing gotten worse? or has Microsoft stopped sending out freebies?

I've noticed this trend at Ars and other places, too, but not just with Bing. Anytime a new product comes out, particularly if it's an underdog product, the initial reviews are absolutely glowing. Then 6 months later you get a bunch of articles about how much the thing sucks.

Another example -- every smartphone to come out since the iPhone was released. For every single one of them, we see an article telling us how this is the great iPhone killer and Apple had better be real worried, etc etc. Then a couple of months pass, and another iPhone killer comes out, and I ask myself "but wait, wasn't iPhone killed 2 months ago?" and then I read the article and discover that the previous iPhone killer was actually a piece of cr@p, but this new iPhone killer is the sh!t.

Because $foo+killer is a hyperbolic myth designed to drive clicks and attract fanboi arguments which drive more clicks. No leading product in any market is killed in 2 months, I repeat, no leading product in any market is killed in two months.

Even the iPhone wasn't a previous product "killer." WinMo is far from dead. Palm is closer to dead, but still not dead.

Don't buy into $foo+killer hype. It's just shorthand for "We think some people will buy this. Maybe a lot of people." But if you write it like that, you don't get on Techmeme, you don't get a million Diggs, and you don't get the big advertising bucks from the $foo and $foo+killer products.

Originally posted by The Real Blastdoor:Another example -- every smartphone to come out since the iPhone was released. For every single one of them, we see an article telling us how this is the great iPhone killer and Apple had better be real worried, etc etc.

I haven't read every phone review here at Ars, but I don't recall anyone explicitly saying 'iPhone-killer' or words to that effect. Overly-optimistic reviews? Yes, in some cases. But the overall assessments of how the various phones will fare in the marketplace has been rather guarded.

I don't have a problem with the writers at Ars showing excitement over a new product, provided it's warranted. There's certainly no shortage of articles panning new releases here (and the worst of the worst probably don't even get reviewed, unless they are from an important industry player.) Occasionally an author will be called a shill in the comments but no one's perfect. And despite our own dearly-held prejudices, competition is a good thing, even if it's from a company we don't particularly like.

The whole point of Google advantage over its competitors was getting humans out of content management and into algorithm management. And Bing expects to improve by going backwards on that? Well, good luck to them. They'll need it.

10 minutes with bing and I could tell you their index was shit. What do I lookup? Me.

I know what comes up when I lookup my name, and it's easy to spot long expired results. Bing presented many expired results within the first page of results. Strong indicator their index crawl and updates are significantly out of sync with reality.

When discussing it with others the response I got was, yeah, ok, maybe try aren't completely up to date; but look at how cool their site looks!

Great, anoter 10 years of crap before people realize the Microsoft is using the same technique they always have.

Just give me the damn results, Microsoft! Stop moseying around with billions in the bank and start making that money work for Bing. Until then, I don't think I'll use your search engine for this reason. And please stop pretending that "prettifying" the meager search results will bring in users...