Facebook Robots

The new forum software differentiates between human guests and robots (Members / Current Visitors / Robots) and one interesting thing that I have noticed more than once is a robot that is identified as Facebook. Is Facebook planning something new or does its current operations make use of a spider?

That new software is very interesting and I see it here on this forum first time in my life. When I open the home page that always surprise me. Really funny.

I think that Facebook uses robots for web scraping, and when some robot presents itself as Facebook, it came to collect some data. Maybe it uses robots and for some other purpose, but I think that data collection is primarly goal.

This means that Facebook is a very active on the Internet. It is using robots to follow the developments of other similar social networks and forums. Facebook is always planning something new, and it wants to see what competitors are doing.

I had had some job offer and after that I little investigated about robots and their purpose on the Internet. I had found new useful things, so after I wanted to start using robots on Facebook, but finally I never did that.

Oh, so they still continue doing this. In fact, I had read about the crawler or scraper (as Fb said) in around 2010. When I had my site up and we integrated the Fb Plugin and set up the page (now defunct) we had seen this on our servers too. We had contacted Facebook but there was no response from their end (as it happens with the Biggies - they dont care about their users )

So, we did a search and found that Fb sends out "scrapers" to the sites that get themselves a page or even put up a Facebook Plugin on their websites. This is done to authenticate the site and also, keep track of what and how the activities are going on on the connected site. According to the Fb Documentation of today -

Our system retrieves this information only after a user provides us with a link.

You can find about this here - facebook.com/externalhit_uatext.php (it is just a small paragraph with the above as its Gist).

So, yeah, maybe your inclusion of the Login Via Fb or the Recommend on FB has got your site listed with Fb.

ps: Cannot post a link as I do not have 25+ posts (have 20 at the moment). Hence, I removed the http from the link above.

Probably a stupid question but I am going to ask anyway... what impact is this going to have on websites? Think it might help peoples websites in SE's or will it have no effect at all?

Click to expand...

As far as my understanding of the things go -
1. FB is not a search engine(as of now).
2. Google or other search engines (to my knowledge) do not take into consideration who crawls the site in question
3. FB only crawls to collect info from your site - maybe about the legitimacy (maybe - about the content, shared via Facebook, is actually there on your site or not) or anything else (but definitely not to judge your site about SEO aspects - again, my best guess)

I do know there are too many Guesses here, still, I dont think FB crawling your website (or any other) has anything to do with the results from Google - specially on the SEO aspects.

Over a year ago I spent a whole day trying to figure out Why my blog's homepage URL, when I share it to Facebook, doesn't show the picture I want it to show, even though I have implemented all the og: meta tags and things properly.

Seriously, the above link was a godsend. What happened was, they had already crawled and "indexed" my site before I implemented the meta tags and put the picture I wanted as the header, and so Facebook kept showing the same old random picture it had picked up as my blog's homepage photo - because they had that in their cache - and it was actually an RSS feed button I had on there, doh...

So after running my blog's homepage through the debug tool, they re-crawled my site, saw the new meta tags, and then started doing what I wanted them to do.

I have no idea if they have any other reason for sending robots to a site, but I'm sure that this is one of them.

Awesome little tool. Thank you for sharing. I just used it and it says that my meta tags are in the body not the head.... however, when I went and checked the meta's were in the head not the body. So now I am wondering if there is a problem in my html. I will have to figure out how to correct it.

I vaguely remember reading somewhere that there was this idea that Facebook might just be considering becoming a search engine, too. So those Facbook bots may just be the beginning. Myself, I have no problems with search bots from Facebook crawling all over my site. The more the merrier, I say.

Awesome little tool. Thank you for sharing. I just used it and it says that my meta tags are in the body not the head.... however, when I went and checked the meta's were in the head not the body. So now I am wondering if there is a problem in my html. I will have to figure out how to correct it.

Click to expand...

One thing though, their debug tool is oftentimes buggy as well.

I mean, I've read of people who had issues reported by the tool even though - as they claimed - they were confident that everything on their website was in order, as it should be.

Generally, I've had more issues with Facebook than with any other social network out there. Of course, that does make sense, since they are the bigger social network out there by far, but still...

Just, take any errors they give you with a grain of salt. And if it doesn't affect you any (thumbnail pictures, descriptions, etc), you can just let it go.

Read up on it, the debug tool is an interesting thing, and it has helped me at times... It still does, if I decide to make any change to a page and want to share it on Facebook for example, and I want it to show the new photo I posted as the thumbnail, the debug tool will make facebook re-crawl my site and show the proper thumbnail photo...

Funny thing, recently I started a blogger blog which was previously owned by someone else and then deleted. It was non-existent for some time. Then I registered that domain, posted an article, and when I shared that article on FB I got some thumbnail pictures I've never seen before, which was confusing...

Then I realized that Facebook has simply cached the photos from the previous blog and remembered those still. Then I debugged the site and things came to normal

Useful Searches

Forums

About Us

Web Directory Forum is a project of the Web Directory Reviews Org site and WDR Directory of Directories. While the primary function of this forum is to support and discuss the web directory industry, topics relating to web design and site management, hosting, search engine optimization, and other webmaster issues are also appropriate.