Well done Bing!

webmaster error fixed, index updated in 3 days

Senior Member

joined:Feb 16, 2007
posts:846
votes: 0

I made a redirection mistake on one of my sites that caused http://example.com/robots.txt to serve a default robots file with the 'Disallow /' directive rather than 301 redirecting to http://www.example.com/robots.txt as intended.

The redirect worked fine for any other file except robots.txt but the result was that most pages were ranking for http://example.com/ rather than http://www.example.com/ - None of those pages would have a description, just the URL.

The redirect was fixed last Saturday, and today I'm already starting to see some pages rank properly. Impressive!

Senior Member from US

Senior Member from US

joined:Apr 14, 2008
posts:2910
votes: 62

Yeah, I think they're doing a great job too.

I read in the Google forum someone thought they were not treating the noindex tag correctly, and don't know if they were for a time or not, but it's been working fine for me... They aren't quite as quick as G yet, but it looks like they're getting there.

Senior Member

joined:Jan 22, 2005
posts:1176
votes: 15

Good for you. I have two sites where Bing has me number one in the category-rightfully so, G agrees--with eight sublinks on the primary listing. We completely re-did those pages with new links and google handled it within three weeks. Bing? Still waiting, three months-plus...

Still, percent of traffic from Bing to the home page--where 87 percent want to go anyway continues to rise. Overall traffic is up, so it's tough to say if they are getting any share from Google--not that my sites would be any kind of real test.

Preferred Member

joined:Dec 23, 2004
posts:574
votes: 2

Sure would be nice if they followed NOODP robots tag instead of using that outdated ODP title and description tag of mine. It's actually embarrassing every time I see it. The editor used the url instead of a description in the title.

I have widgetsubject.com as the title seen in the SERPS...that's it! Description shows I sell certain products. Shut down sales two years ago. Now it's a "general widget guide"

Senior Member

Senior Member from IN

joined:Nov 27, 2003
posts:790
votes: 0

Bing Spidering and Updating SERPs Much Faster.....

and also noticed URLs in search result which are specifically disallowed in robots.txt - although just URLs not the content snippet or title etc. but this too is alarming for us as we do not want anyone to see url path too. So in this case it is displaying a URL domain.com/directory/page.htm where directory and all content under it is disallowed for all spiders in robots.txt

Is this for me only or do we have others here with similar experinece.

Senior Member

joined:Dec 27, 2004
posts:1951
votes: 61

I have several Bot Trap pages listed in BING, Clearly disallowed in Robots.txt, each page has <meta name="robots" content="noindex, noarchive">. Bing like zis for the past 3 years or zo. PITA when normal users click on them.

Junior Member

joined:Jan 25, 2005
posts:90
votes: 2

Bing has a very tiny index when compared to Google and Yahoo. Just do your own test on a few websites including yours and I bet you wouldn't find 1 website that have more pages indexed in Bing than Google & Yahoo.

Bing is slowest of all in indexing and crawling.

btw, do not trust the numbers Bing display (found xx results), they are mostly fake, go to the last page and you will see actual number of pages indexed.

Senior Member from US

When you disallow a page in the robots.txt it will be indexed as URL only because the bot can't access the page to see your noindex,noarchive tag. This is Standard Procedure for search engines, not only Bing...

It's not Bing's fault you disallowed a page in the robots.txt (which they obviously obeyed) and didn't see your noindex tag. They handled it exactly as they should. They did not spider the page, which is what a disallow in the robots.txt tells them to do. Disallow does not mean 'noindex'. They're two different things and one cannot be used in conjunction with, or as a replacement for, the other...

Disallow Mean: Do not open the URL. Do not access the URL. The URL is 'off-limits'. The more links there are pointing to the Disallowed URL the more likely it is to be shown as a URL only result, because the bot cannot access the page to see if it's a content rich page visitors expect to see in the results or garbage, so they have to rely completely on links to the disallowed page to make any type of determination.

NoIndex Means: Make no reference to the URL in the results people see.

Senior Member

joined:Dec 27, 2004
posts:1951
votes: 61

which they obviously obeyed

NOpe, the bot still hits that page at least 2 times a month, they get a 403 along with <meta name="robots" content="noindex, noarchive">. It's a 1px image & link that is linked from several pages on the site. They disobey Robots.txt. So does the SLURP/3.0, but about twice a week. The URL is NOT shown in Y! SERP. I've never seen this from GoogleBot are ASK Spiders.

In my opinion if the URL is disallowed in robots.txt, the robots should simply stay away from it all together.

Preferred Member

joined:Jan 19, 2004
posts:374
votes: 0

Google.de is horribly slow at indexing new pages. Bing listed and ranked pages within a matter of days. Still waiting for Google to list pages uploaded 6 days ago! Not even picking up the unique text on the pages.

The issue might be to do with the fact that while working on the new pages we had used no noindex,nofollow robot directives. Also, did not link from site to new pages until we were ready to go live. The tags were removed and sitemap was submitted 6 days ago to WMT.

Preferred Member

Preferred Member

joined:Jan 11, 2005
posts:513
votes: 0

I'm finding new pages on my established site appear in Bing much faster than Google. Google is taking forever, and Googlebot is on that site every day, throughout the day. Weeks go by and still...I wait.

Bing and Yahoo are much fresher for the things I search for. And since I started using Bing, I've discovered some really wonderful sites I never found in Google, when I used it exclusively.

People who use Google search without exception are missing out as far as I'm concerned.