Secret Facebook Search Engine

Facebook is definitely getting pretty big these days. A Harvard University science student goes from no fame to seemly instant success. I guess if
you have a really cool project, paying that Ivy League tuition may actually be worth the investment.

Well I've heard of a few conspiracies that Facebook may be sharing some of their data with HS. I pretty much shrugged that off as with 80% of the
rest of the nonsense I see floating about the Internet. That was until I ran into something interesting in a server log of a website I put together
less than two days ago. I saw the following entry under the User Agent.

Hardly anyone knows about the new website - its not even finished yet! When I pull up the link provided here, this is the explanation I get.

Why does Facebook appear in my server logs?

Facebook allows its users to send links to interesting web content to other Facebook users. Part of how this works on the Facebook system involves the
temporary display of certain images or details related to the web content, such as the title of the webpage or the embed tag of a video. Our system
retrieves this information only after a user provides us with a link. You may have found this page because a Facebook user sent a link from your
website to other Facebook users. If you have any questions or concerns about any links or content sent by one of our users, please contact us at
legal@facebook.com.

There is no way someone has linked this site to their Facebook account already. It's an impossibility. So I decided to log onto my Facebook
account, which I hardly ever use. I typed into their search bar some of the key words of the website I'm working on. Zero results showed up.

Honestly, I am Pro-Cloud. I think it's great to have an on-line backup of data that I personally give access too. But I can see Facebook is
retrieving information for a system that isn't publicly available. That's just my thought. I wanted to post something here for your opinions.

What information do you think it is retrieving? Its nothing more than a bot that collects data from a website when a user provides a link to that
website in order to direct a user to the originating source. It does not appear to be a crawler in the sense that it just randomly traverses the web
indexing data like say Google does.

Thanks! I appreciate your input. I'm not linking to the site or explaining the setup in detail for the sake of my anonymousness. It's a stock
Wordpress installation with no cron jobs installed in a directory of an existing website. The site has only been up since this past Sunday. I half
expected to see a Google bot as a first visitor.

I don't have a facebook account , but I have a question that some of you may be able to answer .

My son and daughter-in-law came to the house one night and she pulled her cell-phone from her purse and said "That's odd" . We asked her what she
meant , and she showed us a detailed , running list , of everywhere she had been for the last hour or so . What I mean by this , is the list showed
her location , from minute to minute , giving locations all the way to my house .

For instance , "time : x:xx , member at 1234 West XYZ street", and it showed a running list , meaning that she was being tracked as she drove , and
the location updated every minute . And , it showed to be somehow connected to facebook .

Has anyone else seen this ? Is it normal for facebook members , or what the heck is going on ?

That sounds like foursquare, or something similar. I have no idea how foursquare works, so I don't know if it is an automatic update of locations, or
if you must approve before the posts show. I do know several of my friends use foursquare, and I think it is ridiculous. I don't care if someone "Just
checked in at Red Robin/Toys'R'Us/Target/amillionotherplaces".

Originally posted by okbmd
I don't have a facebook account , but I have a question that some of you may be able to answer .

My son and daughter-in-law came to the house one night and she pulled her cell-phone from her purse and said "That's odd" . We asked her what she
meant , and she showed us a detailed , running list , of everywhere she had been for the last hour or so . What I mean by this , is the list showed
her location , from minute to minute , giving locations all the way to my house .

For instance , "time : x:xx , member at 1234 West XYZ street", and it showed a running list , meaning that she was being tracked as she drove , and
the location updated every minute . And , it showed to be somehow connected to facebook .

Has anyone else seen this ? Is it normal for facebook members , or what the heck is going on ?

Facebook mobile has a thing where it you can tell your facebook friends where you are. Its called 'places'. Its supposed to ask you if you want it
turned on or off. I used it once but Im sure you have to manually say where you are, at home, at the bar, at the shops etc. I never use it. I dont
want people to know when Im not at home

LOL! That's hilarious. I think it's pretty ironic that most people are so enthralled with the new technology that they don't mind exposing their
intimacy. I think you can come up with a similar situation growing up.

Instead of Facebook Mobile or Foursquare, why not take a trip down to the local Sheriff's station and ask for an ankle bracelet?

I would say most people rather their shackles be self imposed, versus imposed by authorities (sheriffs).

Originally posted by
CodeRed3D
LOL! That's hilarious. I think it's pretty ironic that most people are so enthralled with the new technology, that they don't mind exposing their
intimacy. I think you can come up with a similar situation while growing up.

Instead of Facebook Mobile or Foursquare, why not take a trip down to the local Sheriff's station and ask for an ankle bracelet?

Add the user agent to robots.txt on the server, that should take care of the problem. Same goes for any other bots you dont want hitting your server.
Of course it doesnt answer the question of what promted the bot to crawl your site especially if they were not provided with a link. It would also be
interesting to know if it does this only with links posted on someones wall, ect or whether it does this with links exchanged during a chat session.
The first would be reasonable, the latter would be a little suspicious imo.

Originally posted by dasysgod
Interesting find! I am going to look into this some more for sure.

Yeah, I'm rather surprised that more people haven't responded to this. The ramifications are pretty huge. Because of the amount of very personal
information collected by Facebook, any bot technology being developed by them would be extremely invasive. Also, to collect information from your
customers that doesn't constitute as part of the public service is a violation of privacy. Facebook doesn't need to be caught using bots.

What issues with privacy are there? The bot would only have access to the exact same information any passing member of the public would have access
to. That being the case, it's down to site admins or those who are making the information available, to ensure that access to that information is
restricted only to those they wish to have access to it.

Its like leaving your lounge curtains open then complaining that passers-by keep looking in.

What issues with privacy are there? The bot would only have access to the exact same information any passing member of the public would have access
to.

That's a great question! I'd be happy to take that topic on. It's not always about the amount of information available. The danger is how you
prepare it. A good example of this would be the sheer amount of cables released by Wikileaks. When people know what to look for, that's when the
information is dangerous. Facebook has an amazing insight into a persons likes, habits, activities, interests, and psychological make-up. Tailor a
search engine using that information, along with the personal notes, contact information, etc, a lot of info can be instantly asserted through the
Facebook account and most of their contacts.

But this isn't about a search engine in the strictest sence of the word. It's about a bot that gathers metadata. It's not quite the same thing. The
bot simply gathers publicly available data from user generated content where that content is off-site. If a user posted a url to an image on your site
then it's the bots job to gather any information from your site about that image. Where are the privacy implications in that? If an administrator
objects bots crawling their site for such data then there are ways to prevent them from doing so. It's not a secret either. The page you linked to
earlier is there to inform admins what the bot is and why it was visiting their site.

But this isn't about a search engine in the strictest sence of the word. It's about a bot that gathers metadata. It's not quite the same thing. The
bot simply gathers publicly available data from user generated content where that content is off-site. If a user posted a url to an image on your site
then it's the bots job to gather any information from your site about that image. Where are the privacy implications in that? If an administrator
objects bots crawling their site for such data then there are ways to prevent them from doing so. It's not a secret either. The page you linked to
earlier is there to inform admins what the bot is and why it was visiting their site.

I'm sorry. I honestly can't explain this any clearer. If a website has enough personal information to do a profile on a person, create a bot to
collect information connected to that profile, that information is worth millions to private intelligence firms. I just don't feel that Facebook
should be using bots, especially when it pertains to the gathering of information that isn't intended for public use or to better their public
service. That's where I feel people's privacy is being invaded.

By seeing the lack of popularity of this thread, most people feel the same way you do, quackers. So no offense. This is just my personal opinion on
how I feel about the subject. But honestly, it is the lack of concern that will continue to make Facebook a force to be reckoned with.

so what is being said here is that facebook has a way to access private information and disperse is accordingly to a specific target audience. the
possibilities of the internet are frightening to my privacy.

I'm sorry. I honestly can't explain this any clearer. If a website has enough personal information to do a profile on a person, create a bot to
collect information connected to that profile, that information is worth millions to private intelligence firms.

That's not what the bot does though. Facebook users gladly submit their own information to facebook, it's all there on their database. They don't
need a bot to gleen information when the users themselves submit that information. The bot in question gathers extra information, not personal
information about users, but about external content users submit.

I just don't feel that Facebook should be using bots, especially when it pertains to the gathering of information that isn't intended for public use
or to better their public service. That's where I feel people's privacy is being invaded.

As I stated, the bot gathers information that is already publicly available. Or are you suggesting that this bot hacks sites to gain information
stored in private directories?

By seeing the lack of popularity of this thread, most people feel the same way you do, quackers. So no offense. This is just my personal
opinion on how I feel about the subject. But honestly, it is the lack of concern that will continue to make Facebook a force to be reckoned
with.

Hey we're all entitled to our opinion. I think the lack of interest is mostly down to your own misunderstanding of what this bot does. In otherwords
you've got the wrong end of the stick. Facebook isnt the only place on the web that has bots that operate in this fashion. They are everywhere, and
they are an efficient method of information gathering. Don't get me wrong, I'm no fan of Facebook. But the responsability for privacy lies with
those supplying the information to Facebook, the end user, not some bit of code whose purpose is to gather related data for information supplied that
in reality has little to do with individuals privacy.

That's not what the bot does though. Facebook users gladly submit their own information to facebook, it's all there on their database. They don't need
a bot to gleen information when the users themselves submit that information. The bot in question gathers extra information, not personal information
about users, but about external content users submit.

I or no one else submitted information about the site that the bot was crawling. The site had only been on the net for three days. I tried their
search engine and found no information regarding my site. So the information the bot was gleaning was for a private Facebook database to use for
advertising or whatever. I do understand what the bot is doing, Facebook is crawling sites for profiling (associating data with existing accounts),
not because a user submitted a Facebook Link. And how would a link be posted on someone's Facebook page when nobody else knew about the site?

Maybe the title of this post should have been "Facebook Has a Crawling Bot". I realize I hadn't filled out the robot.txt, but I had planned on making
the site public eventually. I was just surprised that Facebook would be using this type of tech.

This content community relies on user-generated content from our member contributors. The opinions of our members are not those of site ownership who maintains strict editorial agnosticism and simply provides a collaborative venue for free expression.