Posted
by
CmdrTaco
on Wednesday April 13, 2011 @11:32AM
from the need-a-bandaid dept.

srwellman writes "The practice of Web 'scraping' is growing as many firms offer to collect personal, and potentially incriminating, data about users from their social networking profiles and discussions. Many companies even collect online conversations and personal details from social networks, job sites and forums where people might discuss their lives and even potentially sensitive data, such as health issues. These scrapers operate in a legal grey area leaving many users exposed." We ban scrapers like this regularly here simply for not adhering to the rules spelled out in robots.txt.

You mean like Google already does for its advertisers? In fact, one of the related links in the article is a story about Google titled Google Agonizes on Privacy as Ad World Vaults Ahead [wsj.com], discussing their plans for utilizing their vast archive of valuable user data. The battle for online privacy was lost long ago.

This is a new form of privacy of which the news has not come to Harvard.

I'm pretty sure information posted for the entire planet to read is not private.

Out on the street, a huckster can size you up in about ten seconds, with 90% accuracy. Online, in text, you're not wearing that tribal-armband tattoo, so it might take a few minutes to figure out you're a joiner with delusions of individuality.

Time to revise my motto: The Internet is not secure, and open forums are not private.

> I'm pretty sure information posted for the entire planet to read is not private

Well, that's what I think too, but amazingly, about 98% of humanity doesn't seem to agree. It seems to me that they're insane if they expect something posted to the whole world to be private, but there are SO many who think that way, I'm not sure what to make of it.

The majority of humanity probably think posting something to facebook or whatever is similar to writing "Got totally plastered on holiday" on the back of a postcard and posting it to their local (something that people do)

Sure, it's public but after a few years it will have vanished without trace.

Only because one side of the battle never bothered to fight. Nobody was forced to go to social networking websites and post their life story, anyone could encrypt their email and IM conversations, and ad blocking software is widely available. Large amounts of the information that these companies are aggregating could have been made far more difficult to obtain if the majority of computer users could have been bothered.

Sadly, the Internet has become more of an adversarial game than a way to unite people.

Only because one side of the battle never bothered to fight. Nobody was forced to go to social networking websites and post their life story, anyone could encrypt their email and IM conversations, and ad blocking software is widely available. Large amounts of the information that these companies are aggregating could have been made far more difficult to obtain if the majority of computer users could have been bothered.
Sadly, the Internet has become more of an adversarial game than a way to unite people.

Its ridiculous to expect users to anticipate and thwart privacy invasions. These companies could be shut down overnight (or at least rendered illegal) with common-sense legislation. The problem is not users, it is their bought-and-paid-for "representative" government(s) which sell out their constituents to be deceived and abused by sleazy industries.

Its ridiculous to expect users to anticipate and thwart privacy invasions. These companies could be shut down overnight (or at least rendered illegal) with common-sense legislation. The problem is not users, it is their bought-and-paid-for "representative" government(s) which sell out their constituents to be deceived and abused by sleazy industries.

It's "ridiculous"? Someone held a gun to your head and told you to post your oh-so-pitiful life story on line? They made you post that picture of you drinking with some friends at a stripper bar, or the story about that time you were snorting coke off a hooker's ass? You think some all-powerful government should come and save your irresponsible neck from someone else trying to make a buck off your drunken stupidity, and do so by censoring your writings from them? And you think that doesn't sound ridicul

"Its ridiculous to expect users to anticipate and thwart privacy invasions. These companies could be shut down overnight (or at least rendered illegal) with common-sense legislation. The problem is not users, it is their bought-and-paid-for "representative" government(s) which sell out their constituents to be deceived and abused by sleazy industries."

Not really. I mean yes, in part. Some of what OP was talking about is completely free (as in freely available to anybody) public information. But OP doesn't like scrapers because (1) if used irresponsibly they can hit servers too hard for comfort, and (2) while the information might be freely available, it takes "normal" people a lot of time to go online and sort through all that information, while a scraper can grab it and sift it in a very short time indeed.

There's that and there's the fact that the US (one of the largest consumers of data) has no data privacy laws and has been pressuring places that do (such as the EU) to violate their own laws. The laws don't solve the problem in and of themselves, what they do is make the public more* aware that the problem even exists. (*You can have more than nothing.)

The older ITAR laws and RSA patents didn't help - it effectively criminalized any effort to produce a product, since you'd need to sell the product in the U

Open source has an uphill battle educating the masses as more uneducated people join it with zero expectacions of passing some required level of readiness prior to being let loose online.

Merge a good version of a "secure" OS, like Debian, say, Ubuntu with a paranoid version out there where your proposed security is ON by default --no need to know where to get Adblock for grandma's firefox. Test and tweak to ensure the security doesn't cripple the top 50 websites, (youtube, facebook, myspace, hotmail, google

I'm not on FB, Twitter, MyCloud or whatever else, so there's no data out there about me. If there's nothing to harvest then they can't harvest it - I'd rather be classified as 'boring' or 'not with it' (whatever the fuck 'It' is), than have stuff out there that might come back to bite me in the ass in 10 or 20 years time.

I suppose if you have nothing to hide and have avoided getting too controversial in your online discussions, or too outrageous in your social network photos and statuses, you're probably safe from major problems.

Yep. That's why my pic on chatroulette is an exact average size penis.

Bollocks. Utter nonsense. The people who have "lost" this "fight" are only the ones who were never "fighting" in the first place!

They weren't using different information (or even names and locations) on different sites. They weren't using different IP addresses and MAC addresses. They weren't... doing ANYTHING. Because they didn't even know they had to. That's a pretty weird definition of a "fight".

Pardon me, but (as is probably the case with most internet users in the US today) getting repeatedly sod

fundamentally that's what I do.There is a real me on FB. Then there is me here (and this ID is shared across multiple sites) which would not be too hard to link to the real me.For stuff I really don't want tied to me in re. job interviews, non-gov't background checks etc. I use other identities. For something that I would be afraid of coming out in a relatively thorough discovery && || government background check I simply don't post it on line. At all.-nB

Check out my name. I have several email addresses under that name with different providers, and under different names as well. I have for years. And none of those email accounts are attached to my "real" name or personal information, in any way. And most of them were established from different IP addresses. Also: other people use that name. That is one of the reasons I chose it.

I fully believe (because history clearly demonstrates as much) that the ability to communicate privately and anonymously is esse

Wow, that's pretty inappropriate for an interviewer to require you to open your personal family or friends circle to him. What if my family is discussing my alcoholic father, my pregnant niece, my HIV+ friend, and my habit of killing interviewers and burying them in my backyard?

There are many applicants for each job, so employers can be picky. If they have a set of candidates who are all qualified and of similar levels of experience, they'll pick the one who is most 'normal' in their personal life, and thus least likely to somehow embarass the company or to just not get on with other employees.

Well done - you can track my previous postings on/. Do you want a prize? I'm now accepted as one of the 6.5 million people in the UK who have their DNA on record because this country stores DNA samples from everyone convicted (and many who are not convicted). Assuming of course that I'm not just posting things to try and make a point and gain Karma points - just like all the people on here who post about "My wife had this happen to her..." - we know that they haven't got a wife or they wouldn't be on here

I was half way to a contextual analysis based on some of your more creative phrases but I ran out of time to rule out false positives. At a minimum I think you post on at least five sites and cross referencing those is almost enough. The last trick requires one of the web admins (for easy sake start with slashdot) to use the new geolocation trick based on public nets to narrow it down. The point is that it's a When-Not-If world out there so plan your future expecting to be tracked

Even though you never post a thing, someone else may post something about you. You may already be tagged in multiple photos on Facebook. You may have loan applications visible on the web. Your information is not entirely under your control - with pervasive digital storage, constant security challenges, and an increasing cultural trend to blurring the line between public and private, there is a growing chance that your information will leak out into the public.

Slashdot is filled to the brim with people who take the time to create an alias and then list their homepage on their profile, which of course, is displayed in a link on the same line as their alias in the post they just made.

I click on those homepages whenever I read something really stupid or ridiculous or inflammatory or completely polar opposite my perspective. Which is to say, I click on them A LOT. I am amazed at how many of these "homepages" are links to commerce sites, or sites advertising some ki

You aren't supposed to buy from them. The link isn't there for your benefit. It's an SEO trick, part of the strategy in trying to raise the page rank for that site.

If you run a blog, you'll find you'll get a commenters that say stuff like "hi, your site is a good understand! one for my book marks." It's flatteringly nice, and obviously English isn't their native tongue, so you thank them for their kind words. And with luck, you may not follow the link in their user name, which you might then discover lin

However, there are patterns of browsing that are clearly not human. Humans do not make 100 requests in a 10 second timespan, nor do humans traverse every post made by every user.

Yes, it is imperfect and you might ban an occasional human, but this is essentially the situation we have with spam filtering. It is a bit sad that the Internet is becoming so adversarial, but that is what we face.

A smart discrete scraper will scrape breadth-first, ie: scrape 100 websites alternating the next page from each site in turn, instead of the next page on a single site until that site is finished. Some scraping on active sites like Slashdot or just Google's spidering is never done; It just continues on as new content is created. It would be easy for a scraper to act just like a human on Slashdot, just keep clicking 'refresh' every once in a while. An astro-turf post from GNA would really throw the admins of

...unless they have a fistful of mod points to spend...heck, sometimes I'm just very interested in a story and want to see what everyone has to say about it. True, that doesn't happen often, and I certainly don't read 10 posts a second, but it does happen...

robots.txt isn't meant to have any enforcement capability; by its nature it's just an advisory mechanism telling bots who and what they will and will not accept. If a bot chooses to ignore it (as pretty much all of the types of bots described in this article do), it's up to the site admins to enforce it via IP bans etc.

A good place to begin would be to examine the robots.txt of large sites to see what they're blocking. Sometimes they leave helpful comments in the text files as well. The most interesting I've come across so far is Wikipedia's robots.txt file [wikipedia.org] which has comments for every disallow or series of disallows.

The most interesting I've come across so far is Wikipedia's robots.txt file [wikipedia.org] which has comments for every disallow or series of disallows.

Well.. it bothers the hell out of me that I can't Google VfD/Afd/Page for deletion Articles on Wikipedia, because a few people were annoyed there were VfD articles about their nonnotable vanity page on WP.
Wtf are the Wiki people thinking?
Sometimes interesting points arise in a discussion, and it would be useful to be able to search those discussions i

Sometimes, bots can be detected by their patterns or behavior. If a bot doesn't want to comply with robots.txt and ends up sucking a site's bandwidth, the site may ban it automatically if it's configured to do so. Not sure if Wiki does this, though

Listing Firefox/MSIE in robots.txt also wouldn't do anything because those are browsers, not web crawlers, so they don't have to even acknowledge the robots.txt standard. Though, that's not to say that it wouldn't be fun, let alone downright tempting, to disallow

Listing Firefox/MSIE in robots.txt also wouldn't do anything because those are browsers, not web crawlers, so they don't have to even acknowledge the robots.txt standard.

Shouldn't effect users.... but I was thinking some of the 'evil bots' might be using an API/framework for making bots, where they supplied the fake UA field to, and that framework might be so gracious as to _force_ the bot application developer to comply (?)

I was also wondering if FF/MSIE might have some auto-crawler features that

Shouldn't effect users.... but I was thinking some of the 'evil bots' might be using an API/framework for making bots, where they supplied the fake UA field to, and that framework might be so gracious as to _force_ the bot application developer to comply (?)

Yeah, there are some frameworks and free-to-use bots all around, but because of the diversity of bots and their uses as well as the functions of various servers, it'd be hard to control their behavior so simply. That's part of the reason why robots.txt

A good place to begin would be to examine the robots.txt of large sites to see what they're blocking. Sometimes they leave helpful comments in the text files as well. The most interesting I've come across so far is Wikipedia's robots.txt file [wikipedia.org] which has comments for every disallow or series of disallows.

After reading this the first thing I thought was, "Now we need a meta-robots.txt file to stop robots from scraping the robots.txt file."

There are a few specialist blacklists popping up. Here is one [stopforumspam.com] specifically for listing spam robots that attack the most popular forum softwares (phpBB, SMF, etc). What I would really like to see is one that lists all the latest "scrapers to detect when people say negative things about your company/product and C&D them" services. I'd sign onto that in a minute - a no-brainer security measure for yourself, your blog and your forum users.

I've always wondered -- how would this work for future politicians from our generation?

All your comments, history etc are probably available in a multitude of places, and anyone with enough motivation can go around digging and find some pretty serious material. Combined with the fact that most people know (or care) little to nothing about privacy, you will have an entire generation of users with a good chunk of their private lives and opinions shared out on the Internet for everyone to see.

There's already pretty damning video clips of many US politicians that are widely available. It doesn't seem to have any real impact on their ability to get (re) elected here. Watching the Daily Show for a week, you will come up with numerous examples.

Unless of course you're referring to the effects these sorts of things might have on the political proceedings in smoke filled rooms.

How slowly? Could you download all Slashdot comments in a profitable amount of time? You would also have to use a download pattern that is not obviously automated (e.g. sequentially requesting each link on a page).

In short, it is not the easiest thing to do. It is like trying to pass the Turing test (which software is getting pretty good at doing, as it so happens).

Run a separate scraper from different IP addresses for each "category" on Slashdot. Each scraper will read all of the articles in that category and refresh the comments from time to time (random intervals) just like a human would. That would be pretty hard to detect.

Depends. Am I allowed to use a botnet? From a previous story, I know that you can buy machines on botnets for about five cents each. For a dollar, I could have 20 machines, all grabbing one Slashdot story per minute (probably slow enough not to be seen as a spider). That's about a million Slashdot stories every four days. Maybe make it a million a week to make sure. Spread it over a big botnet and you can get the entire archive in an hour or so, without it looking like anything other than a few hundre

Because the public sector has very little time to handle FOIA requests and they sometimes cost more money to complete than I'm willing to pay (usually because they don't do much of their own data work in-house and have to call on a contractor to do it for me), I use their websites to glean the data I want.

So while there are any number of pitfalls to screen scraping (not understanding the meaning of the data and trends, being fed incomplete or purposefully incorrect data, or even being banned outright) screen scraping can be great for learning about and reporting on the public sector when they are physically or financially incapable or simply unwilling to do it themselves.

The company was SEM/SEO then they moved to social optimization and scraping. It was a black art, like the SEO stuff, and totally dependent on the provider (in this case facebook and twitter) to not change anything. It's the same basic the problem with SEO and Google; if facebook's (or Google's) API coughs the social media scrapers (or SEM/SEO people) get pneumonia. If Facebook wants to stop it, they can do so fairly easily.

Unfortunately for privacy, a huge part of FB's business model (like Google) is selling that data to the scrapers and the scrapers' clients.

Face it, the type of people who go into marketing have very little to offer this world. Their whole reason for existence is to hopefully sell something to somebody who might not otherwise buy it. The only redeeming aspect of marketing is that it is a non-violent sinkhole in which to drop money, vs say a war in some God forsaken desert.

Have you ever met a marketing/advertising person who actually liked people?

Collecting data about others is somewhat an essential freedom. But my view and the modern view differ as most people do not feel the same way. But if we take the usual view any company collecting data about a specific person could be charged with stalking. We usually think of a pervert stalking a child or pretty girl. But stalking is stalking regardless of whether it is a corporation or a pervert. The motive for the stalking is irrelevant. Considering the current mood huge civil suits might take p

Add a line in your acceptable use / EULA section stating that you expect the user of the account to be human and that any attempt to scrape the data off of the server is fined at $100,000 per message, plus $10,000 to each message author.

Add a line in your acceptable use / EULA section stating that you expect the user of the account to be human and that any attempt to scrape the data off of the server is fined at $100,000 per message, plus $10,000 to each message author.

And also, you reserve the right to sue the Tooth Fairy for lost unicorns.

There is no "legal gray area" in scraping. By publishing data on a public webserver, you give consent to clients for viewing it. And what does "the user of the account to be human" mean, anyway? Presumably, humans will eventually view the data downloaded by the scraper. Challenge of the day: give me a legally watertight definition of "web browser" that includes user agents like Lynx (which downloads data from a remote server and presen

Sure- Automated process that stores the results in a database or is otherwise used in a system where the results are aggregated and retrievable for 4th party consumption with a method to tie back to a person.

That wasn't difficult at all. Just because I write something for consumption to the members of a particular web site (assuming that it's NOT out in the public like Slashdot's or any other comment system), I would not expect it to be slurped up and sold by 3rd parties. On a member's only web site, such

The report is back sir, and the results are disturbing. Almost everybody likes sex, and a lot of them are weird. The ones that don't like sex have very strange hobbies. The ones that don't abuse illegal drugs are abusing legal drugs, and almost nobody weighs what they say or looks like their online picture. What should we do?

Our SiteTruth [sitetruth.com] system does some "scraping". We're looking for the name and address of the company behind the web site, so we can check the business out. We also look for ad links and a few other things, like BBBonline seals, which we check. We use a user agent name of SiteTruth.com site rating system. We don't look very deeply into a site; if after examining the most likely 20 pages, we haven't found out who runs the site, we figure they're not going to tell us. The site is down-rated accordingly.

Hmm. Sitetruth seems to be a little flawed. Not the least because it considers itself to be a little questionable, and secondly because it doesn't consider the possibility that a subdomain might have more authoritative information than the main domain (for example, "store.company.com" might have an EV certificate, giving you a high assurance of identity and location, while the main site at "www.company.com" has no high assurance sources). I also notice the complete lack of contact information. Ironic, f

for example, "store.company.com" might have an EV certificate, giving you a high assurance of identity and location, while the main site at "www.company.com" has no high assurance sources

It's rare to see that. Know of a significant example? One might expect it for "store.yahoo.com", but that site won't even accept a HTTPS connection. Neither will "disney.go.com".
Citibank has separate certs for "www.citibank.com" and "online.citibank.com".

On this topic, here is some bad practices in HR that needs to end:1. Hiring based on stereotypes is NOT a good idea. [com.com]2. The purpose of HR should not be to minimize legal liability.3. The illusion that celebrities are perfect needs to end.4. Filtering people based on health problems to minimize health insurance costs is not a good idea.5. Not hiring people based on debt creates a paradox for those who have to pay it off.And as a side note, companies with seriously broken HR often have other problems too.

Would that be legal? Could I setup a company that collected DNA samples without their owners permission(say, by tying the hair clippings from a salon to the CC that paid for the cut)? Could I sell that info to the government?

If no one's done it, someone should, if for no other reason than to scare the shit out of people and hopefully wake them up.

Umm..... yes, someone obviously could do it, but you'd probably have some difficulty linking up the clippings you found to specific individuals. (I mean, would you propose the hair stylists themselves start indexing their customers' hair clippings? They'd be the ones who know their clients' names, addresses and phone numbers since everyone's in their computer system already. If they started acting as the data collectors for this type of operation, it would cause a big loss of business when people started

I think they are 2 distinct issues that do not combine the way you suggest.

1. If you violate a websites TOS the website can come after you.

2. The info they gain spidering a website is pretty much free for them to use to discriminate against you.

Anything I post on slashdot/FB/any online forum I treat like it is viewable by every future and past employer, insurer, lender, ex girlfriend etc. Anything online will exist forever and if it's not already permanently linked to you, it will be before you die. If th

So if I write a book, can I include TOS that makes it illegal for anyone to use the information within the book?
If I write a book about how much my boss sucks, and how I slack off at work, can I include TOS so that nobody is allowed to relay that information to him?
Even if I only sell my book to members of a book club, I wouldn't think this changes anything.

If you intentionally post information about yourself on a widely viewable forum, I would expect oth

Well, the problem with (1) is that a TOS is an agreement with no signature, no confirmation of acceptance (implicit is unlikely to hold up in court) and no proof that the TOS was even visible by the user (since what is visible to the user is a function of the browser and cannot be established at the server-side).