Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

Slashdot regular contributor Bennett Haselton writes
"In my last article, I proposed an algorithm that Facebook could use to handle
abuse complaints, which would make it difficult for co-ordinated mobs to get unpopular content removed by filing complaints all at once. I offered a total of $100 for the best reader suggestions on how to improve the idea, or why they thought it wouldn't work. Read their suggestions and decide what value I got for my infotainment dollar."

tester data

In my last
article,
I proposed an algorithm that Facebook could use to handle
abuse complaints, which would scale to a large number of users while also making it difficult for
co-ordinated mobs to get unpopular content removed by filing complaints all at once.
I offered a total of $100 to readers sending in the best suggestions for improvements, or alternative
algorithms, or fatal flaws in the whole idea that would require starting from scratch.
As the suggestions were coming in, Facebook obligingly kept the issue in the news by
removing
a photo of two men kissing from a user's profile, sending a form letter to the user
that they had violated Facebook's prohibition on "nudity, or any kind of graphic or sexually suggestive content".
(It would be a cheap shot to say that a photo of a man and a woman kissing probably would not have been removed;
in truth, probably just about anything will get removed from Facebook
automatically if enough users file complaints against
it, which is the problem for unpopular but legal content.)

How would these complaints have been handled under my proposed algorithm? The gist of my idea was that any users
could sign up to be voluntary reviewers of "abuse complaints" filed against public content on Facebook. Once Facebook
had built up a roster of tens of thousands of reviewers,
new abuse complaints would be handled as follows. When a complaint (or some threshold of complaints)
is filed against a piece of content, a random group of, say, 100
users could be selected from the entire population of eligible reviewers, and Facebook would send them a request
to "vote" on whether that content violated the Terms of Service. If the number of "Yes" votes exceeded some
threshold, the content would be removed (or at least, put in a high-priority queue for a Facebook employee to determine
if the content really did warrant removal). The main benefit of this algorithm is that would be much harder for
co-ordinated mobs to "game the system", because in order to swing the vote, they would have to comprise a significant
fraction of the 100 randomly selected reviewers, and to achieve that, the mob members would have to comprise a significant
fraction of the entire reviewer population. This would be prohibitively difficult if hundreds of thousands
of users signed up as content reviewers.

All of the emails I received -- not just "almost" all of them, but really all of them -- contained some insightful
suggestions worth mentioning, although there was some duplication between the ideas. If you didn't see the
last
article, you might consider it worth while to stop reading before proceeding further, and mull over the description
of the algorithm above to see how you would improve it. Then read the suggestions that came in to see how well
your ideas matched up with the submissions I received.

The upshot is that nobody found what I believed to be completely fatal flaws, although one reader brought something
to my attention that might cause trouble for the algorithm after a few more years. Beyond that, reader suggestions
could be divided essentially into two categories. The first category of suggestions related to ensuring that the basic
premise would actually work -- that the votes cast by a random sample would be representative of general user opinion,
and could not be gamed by a coordinated mob or a very resourceful cabal trying to game the system. The second category
of suggestions started by assuming that the voting system would work, and suggested other features that could be added
to the algorithm -- or, in one case, an entire alternative algorithm to replace it.

To begin with the attacks and counter-attacks against the basic voting algorithm.
Walter Freeman and Haydn Huntley independently suggested monitoring for users who vote in a small minority in
a significant portion of vote-offs, and reducing their influence in future votes (by either not inviting them to vote
on future juries, or sending them the future invites but then ignoring their votes anyway). The assumption
is that if a user is frequently among the 10% who vote "Yes [this is abuse]" when the other 90% of
respondents are voting "No [this is not abuse]", or vice versa,
then that user is voting randomly, or their point of view is so skewed
that their votes could safely be ignored even if they are sincere.
I like the idea of eliminating deadweight voters, but this might also incentivize voters to vote the way they think
the crowd would vote, instead of voting their true opinions -- for example, if they were called to vote on an
anti-Obama page that showed Obama wearing a Hitler mustache. Some people's knee-jerk reaction would be to call the
page "racist" or "hate speech" or "a threat of violence", even though comparing Obama to Hitler is not,
strictly speaking, any of those things. If I were voting my honest opinion, I would count that page as "not abuse".
But if I knew that I were voting along with dozens of other people, and my future voting rights might be revoked
if I didn't vote with the majority, I might be tempted to vote "abuse".

Similarly, Walter Freeman and reader "mjrosenbaum" both suggested setting deliberate traps for deadweight users,
by creating artificial cases where the answer was pre-determined to be obviously yes or obviously no,
calling for votes, and
revoking privileges for users who gave the wrong answer. This would eliminate the problem of borderline cases
like the one above, where smart users think, "I suspect the majority will give the wrong answer, so I'm just going
to go with the crowd, to keep my voting rights." On the other hand, it's more labor for Facebook to create
the cases, and any public content authored by them -- especially content that is deliberately crafted to
be "questionable" -- would probably have to run a gauntlet of being reviewed by
lawyers and PR mavens before being released. My suggestion would be to use these artificial scenarios periodically
to make sure that the system is working (i.e. that juries are giving the right answers), but it would be
too inefficient to use it to try and weed out problem voters.

In fact, these and several other suggestions fell into a category of ideas that could possibly improve the efficiency
of the algorithm by reducing voter shenanigans (where "efficient" means that fewer users have to be invited to each vote-off in order to get
statistically valid results), but might not be worth the effort. As long as most
of the votes cast by users are sane and sincere, all you have to do is invite enough voters to a vote-off, and
the majority will still get the correct answer most of the time, even if you have problem voters in the system.
That's the simplest possible algorithm. The more complicated an algorithm you come up with, the more likely that
Facebook (or any other site you recommended this to) would just throw up their hands and say, "Sounds too hard",
and leave the idea dead in the water. That's why I like the algorithm as lean and tight as possible.

So it's not quite like designing an algorithm for your own use, where you could feel free to introduce more complications
as long as you're responsible for keeping track of them. In recommending an algorithm for widespread adoption, the
basic form of the algorithm should be as simple as possible. In the case of the voting algorithm
some interesting wrinkles may come up if you don't eliminate problem voters, but this is
not fatal to the idea as long as it's still true that, given a large enough random sample of voters, the majority will tend to
vote the correct answer.

For example, James Renken pointed out that as voters dropped out due to boredom,
the remaining users casting votes would tend to be either (1) weirdos who just wanted to view questionable material; and
(2) prudes bent on removing as much material from Facebook as possible. But that's OK, as long as those two
groups vote sanely enough (or as long as there are enough sane users outside those two groups) that material which
does violate the TOS, tends to get more "Yes [this is abuse]" votes than material that doesn't. Then all you have to do
is make the jury size large enough to make a statistically significant distinction between those two cases.

Similarly, Joshua Megerman suggested surveying users for their religious, political, and other beliefs when they sign
up as volunteer reviewers (they could of course decline the survey). This makes it possible, insofar as people answer
truthfully, to make sure that a jury is composed of a group with diverse belief sets. (On the other hand, users could
game the system by reporting beliefs that are the opposite of what they truly feel. For example, if you're a leftist,
register as a right-winger. Then when an abuse case comes before you, if it's a piece of content more offensive to leftists,
then the real leftists on the jury will tend to vote against it -- but as a registered right-winger, you'll be able to
cast a vote against it as well, and you'll be displacing a real right-wing voter who probably wouldn't have voted
that way, so your vote will be worth more!) Again, it's fine if Facebook wants to do this, but even without collecting this
data and simply selecting jurors at random, it should still be true that genuinely abusive pages get more "Yes" votes
in a jury vote, than non-abusive pages.

Lastly in the "keep the jurors honest" category, Paul Ellsworth suggested allowing jurors to anonymously review each
other -- when a given juror is chosen
for the "hot seat" (perhaps randomly, perhaps as a result of a history of skewed voting), other jurors are randomly
selected from the voting pool, to review that juror's voting record and decide whether that juror has been voting
honestly and judiciously, or not. When I first read this idea, I instinctively thought that because a contaminated
jury pool would be reviewing itself, it would not be able to reduce the percentage of problem voters, but a little
more thought revealed that this isn't true. Suppose initially your jury pool consists of 80% "honest voters" and 20%
"dishonest voters", that honest voters who review the voting record of another voter will always vote correctly whether
that person is "honest" or "dishonest", and that dishonest voters will always vote incorrectly. It's still the case
that when a voter's record is reviewed by a panel of, say, 20 other voters, virtually 100% of the time the majority
will get the right answer. If you strip voting rights from a voter whenever a jury of other voters determines them
to be a "dishonest voter", then over time, the percentage of honest voters in the system will creep from 80% to 100%.
So again, this might work, and again, it might just be adding unnecessary complexity if the basic algorithm could
work without it.

Note that none of these precautions would address the case of a "sleeper" voter -- a voter who joins the system
with the sole intention of voting incorrectly on particular types of cases (perhaps planning on voting "yes" to shut
down pages made by a particular organization, or pages advocating a particular view on a single issue), while still
planning to vote correctly on everything else. By voting honestly in all other cases, they prevent themselves
from being flagged by the system for casting too many minority votes, or from being blacklisted by other jurors for
having a questionable overall voting record. The only real way I can see to address this problem is to hope that
such users are outnumbered by the honest users in the system, and that juries are large enough that the chances of
"rogue voters" gaining a majority on any one jury are nearly zero.

Which brings us to the one potentially fatal weakness in the system that I'm aware of: reader George Lawton
referred me to a program run by the U.S. government to create armies of fake accounts to infiltrate social
media, named, apparently without irony,
Earnest Voice:

The project aims to enable military personnel to control multiple 'sock puppets' located at a range
of geographically diverse IP addresses, with the aim of spreading pro-US propaganda.

An entity with the resources of the U.S. military could potentially create enough remote-controlled voters to overwhelm
the system. I'm not sure if there is a way to deal with a system if the majority of voters are compromised.
Presumably by making all decisions appealable to a core group of trusted Facebook employees at the top (although this
then creates a bottleneck and limits scalability, especially if filing an appeal is free and all the parties who lose
abuse cases are constantly filing appeals to the next level up).

Now. On to the second category of suggestions: Assuming the majority of voters are honest, what other features
would be desirable to build into the system?

Walter Freeman, on the subject of filing appeals, suggested putting appealed pages in a special queue where they
could be publicly viewed and users could comment on the ongoing appeals process, in addition to reading arguments
posted by either side; this also negates the censorship itself due to the
to the Streisand effect. I agree, but it's not obvious
why this is a desirable feature. This does create perverse incentives, since some users could get
extra traffic for their content by creating a page that makes whatever argument you're trying to promote,
spiking it with some TOS-violating content, waiting for the page to get shut down, appealing the decision, and
enjoying all the extra Streisand attention that it gets while on public display during the "appeal".

Meanwhile, James Renken pointed out that the system would work best
for content that was originally public anyway, like a controversial Facebook page or event. If someone filed
a complaint regarding a private message that they received, and they wanted a "jury vote" about whether the
content of the message constituted abuse, then either the sender or the recipient would have to waive their
right to privacy regarding the message before it could be shared with jurors. If the message really was abusive,
then in some cases the recipient might waive their privacy rights -- reasoning that they didn't mind sharing
the nasty message that someone sent them, in order to get the sender's account penalized. The problem arises if
the message also contains sensitive personal facts about the recipient, which they wouldn't want to share with
anonymous jurors. The system could allow them to black out any personal information before submitting the
message for review, but that creates a recursive problem of abuse within the abuse system -- how do you know
that someone didn't alter the content (and thus the offensiveness) of the message through their selective
blacking-out? So it's not obvious whether this idea could be applied to non-public content at all.

Reader George Lawton suggested allowing content reviewers to vote on the funniest or weirdest content they
had to review, to be posted in a public "Hall of Infamy". I love the thought of this, but I think Facebook's
lawyers would be uncomfortable glamorizing anything questionable even if it were ultimately voted to be non-abusive
(and certainly if it was voted to be abusive). Besides, this also has the perverse-incentives problem --
tie your message to something that you know will not only get an abuse complaint, but will hopefully end up
in the Hall of Weird. (Even without the abuse jury system, there are already plenty of incentives for people
to make a
political point and hope that it will go viral.)

David Piepgrass suggested that new content reviewers should be allowed to specify certain types of content
that they don't want to be asked to review -- nudity, graphic violence, etc. This sounds like a good idea.
He adds that users probably shouldn't be able to opt-in only to review certain categories of content
(or jurors might sign up only to review nudity, and then who would be left to review the death threats?).

Finally, in the other corner: Jerome Shaver suggested bypassing the jury voting system altogether
and working on a heuristic algorithm to determine when abuse reports were being submitted
by organized mobs of users, based
on the patterns shown by mutual friendships between the users filing the abuse reports. The difficulties
in designing such an algorithm, are too complicated to summarize quickly, and could fill an entire separate article.
(Convince yourself
that it's not an easy problem to solve. You can't just ignore abuse complaints from clusters of users that have
many mutual friendships, because it can happen that real tight-knit communities of users might file abuse complaints
against a piece of content, where the complaints are actually genuine.) But again, there is the problem that
if a proposed solution is too complicated or too nebulous, Facebook has the excuse that they are "weighing several options",
that they're "already working on something similar internally", etc. The jury vote system has the advantage that it
can be described in just a few sentences, and the general public always knows whether it has been implemented or not -- which
means that as long as abuses of the complaint system continue, people can ask, "Why doesn't Facebook try this?"

You'll notice this is just a laundry list of the ideas I received, without any definitive conclusions about which ones
are good or bad, but that's all I was going for. The original algorithm, I could argue with the force of mathematical
proof that, under certain reasonable assumptions, it would work. There's no such proof or disproof for any of the
suggested modifications, so I don't feel as strongly about any of them. But at the top of the article I suggested
for readers to stop reading and see how many of these ideas they could come up with on their own. How did you do?

The final honor roll of readers who were each the first, or only, person to submit an original idea:
Walter Freeman (bonus points for getting in several good ones), James Renken,
Joshua Megerman, Paul Ellsworth, George Lawton, Jerome Shaver, and David Piepgrass.
Most of them volunteered to donate their winnings to charity, and agreed to let me donate their share
to Vittana, which arranges microloans to college students in developing
countries. One preferred a charity of their choosing, and only one actually kept the money. To be clear, for future
contests, it's awesome if you want to donate the money to charity, but it's not dickish to keep it. That was the
original deal after all.

So, all very clever and interesting suggestions, some of which might inspire readers to keep coming up with their
own further variations. I said which ideas I probably would have incorporated and which ones I wouldn't, and I'm sure
many of you would tell me that I'm wrong on some of those points. Although from here on out you're doing it for free.

The point? At least an intellectual exercise., and possibly a scheme that Facebook (who have stepped on their dicks any number of times over this very issue) might at least consider, along with anybody else who hosts user content and feels a need to monitor its appropriateness.

It's basically the/. mod system, except mods only vote on posts/messages that are flagged by a user as violating the ToS and the choices are 'OK' or 'Not OK'.

As for who watches the watchers, it's basically the/. meta-mod system.

The "algorithm" is safety in numbers. You get a large enough pool of mods, and hopefully the "normal" people sufficiently outnumber the extreme folks who mod everything as 'Not OK' or who volunteer to mod just to get a peek at naught

How is voting the way you feel the majority of people will vote a good thing? While I don't want to out and out say that Democracy doesn't work, there certainly are instances where the majority of people are wrong. The whole crowd-sourcing system doesn't really account for this. Still, it's probably still better than existing systems.

How is voting the way you feel the majority of people will vote a good thing?

He's not saying people should strive to vote with the majority. He's worried about folks voting with the expected majority to hide their bias.

Someone who decides to vote against everything is easy to find. Once you get a large enough set of 99-to-1 votes where the same person is the 1, you know you can ignore that person's vote. Whether it's someone who things everything is inappropriate, doesn't understand the ToS, or whatever, the author's feeling is once you've identified whether something is inapprop

To address the selection problem for the pool of reviewers and avoid having the self selection result in a bias, when an entry needs judging put the question to a random group of users. Have it pop up when they log in and require a vote of 'OK' or 'Not OK' before they finish logging in to their account.

How about facebook simply employ the appropriate number of individuals necessary to responsibly run their business? It may cost them 900M out of every billion to do it. Does it mean they shouldn't? No, it means they overestimated how lucrative running a social networking site is by cutting important corners. This sort of problem is what necessitates legislation and government oversight.

This sort of problem is what necessitates legislation and government oversight.

No it doesn't, this is bullshit. Here's a real wild idea: run your shit how you want to run it. Why the fuck would facebook need government regulation to tell them how to handle removing stuff from their site? Once something gets popular everybody should be able to use it and it must be about free speech? That's a flimsy argument and stupid too. Facebook runs the risk of removing more than they and their users would want, but that should not leave them open to any kind of law suit or regulation any more tha

How about they do what they are already doing and keep more of the money for themselves. if people really won't put up with the errors and start leaving for greener pastures then they can consider spending more money on it.

I can't figure out why these articles don't even make a casual reference to the fact that they are being posted on Slashdot, where we have a system that already fulfills all of the requirements for this "project".

Nor do I recall it. I think the idea is ready to tested and might bestow benefits at numerous applicable sites.

I think it has more limitations with re. to lifting up of comments of value (though it also appears beneficial here), such as for slashdot comments, in comparison to stopping wrongful censure, where I really can't see a down side.

I don't see any practical way of gaming it. There are the threats of politically correct mindedness and herd-think, and perhaps some of the comments could be used

No, I didn't read the first article. Yes, I super-speed-skimmed through this one... Nevertheless, from what I understand, isn't what you are proposing currently being implemented on/. as metamoderation? Watching the watchers and so on? Furthermore, trying to find ways to categorise the "values" of reviewers etc. sounds similar to weighted voting, a system in use on sites like IMDb.

My personal opinion on the whole matter though, is that all the sewage of the world should be redirected into the Facebook da

It's not a bad idea, provided that people are actually willing to work for you for free. Usually, they aren't. It's been tried before, for spam filtering, but the reviewers were overwhelmed. (You'd get random messages containing spam which you have to rate. Right.) This approach sometimes works when the number of items to rate is much smaller than the number of raters, and when the user has to read the thing they're rating anyway, as with Slashdot.

suggested allowing jurors to anonymously review each other -- when a given juror is chosen for the "hot seat" (perhaps randomly, perhaps as a result of a history of skewed voting), other jurors are randomly selected from the voting pool, to review that juror's voting record and decide whether that juror has been voting honestly and judiciously, or not.

Have you metamoderated recently? In fact, the whole thing seems like what Slashdot's been doing for years and years. Randomly select moderators based on a number of factors, including previous history and past performance, and so on.

A business will only host material that is profitable. If it is unprofitable to host the material, the business will not host it. Unless they are protected by a more powerful government, a business will always remove material when directed to remove that material by a government. Google and Facebook are in this to make money for their shareholders.

In the sixties, a black man couldn't kiss a white woman on TV because many affiliates wouldn't carry the show. You still don't see it very much on TV. The TV

While setting up juries, etc, may be useful, it would be quite easy to use a Google-ish algorithm to discount abusive mob complaints in the first place. If you find that certain people have a high rate of complaining, their complaints probably should have a low weight. And you could use the age and activity level of the account to ignore some accounts (new accounts, and rarely used accounts do not represent Facebook users well). So if someone arranges a mob to get groups dealing with an issue removed, the m

I still think there is a far simpler solution to this problem. Simply chose 1 in 10 people that _can_ vote. That is, 90% of Facebook members simply wouldn't be allowed to vote on any specific topic, on average they would have to try 10 different ones (or more) before they were allowed to vote.

That would make organized vote mobs very difficult to arrange, which is what we're trying to accomplish. Yet this version would not require any special effort on the part of the users, or the system. No setup either.

The author kindly forgot about the legal issues of this algorithm. You can't just spread people's photos to your users just because you want their opinion. These photos contain private data and cannot simply be sent to random people on the internet, not even with faces blurred away.
Great idea; legally not possible until Google becomes the president of the USA.

Make sure that the vote options are randomly displayed using a graphic image to determine which is yes and which is no, so you could stop someone from writing some simple script to watch for the facebook voting page and click yes or no by default. Sort of like forcing two fields to capture email with neither of them called email on signup pages.