Posted
by
timothy
on Tuesday January 01, 2008 @10:01PM
from the wisdom-of-crowds dept.

cagnol writes "The Washington Post reports that Jimmy Wales, the founder of online encyclopedia Wikipedia, has announced the launch of a new open-source search engine, Wikia Search, on January 7th, 2008. The project will allow the community to help rank search results, in a model close to Wikipedia. However the company is a for-profit organization. This new search is supposed to challenge Google and Yahoo."

No, it is no response to Knol. I have been working on this for a year. The press has talked about it endlessly.:-)

It'd be sort of cool if we could create a search engine in a week or two to respond to Knol, but actually it takes a bit longer.:)

I see Larry and Sergei socially from time to time. I spoke about the search project at Google Zeigeist a few months ago. Going to a google party next month. The media loves a "fight" but really, that's just a nice story arc the press makes up. (Notice: google is not in the search business, google is in the advertising matching business. This search engine doesn't hurt that business at all, indeed it probably makes it marginally less likely we will see the emergence of a proprietary competitor to topple them.)

It is actually possible for people to just enjoy doing cool stuff without being bastards about it. People forget this sometimes, maybe due to the reputation of a certain dominant software provider.:)

It looks like you've entered some sort of partnership with Grub http://www.grub.org/ [grub.org].If so, kudos... Grub's been languishing in not-ready-for-primetime land for far too long, and the ability to crawl your own site to keep results current is a bonus, too.

Hey Jimmy: quit goofing around on Slashdot and get to straightening out the Wikipedia "administration" system. Check out your current fundraising campaign - that little green guy is moving REALLY slowly, and things like faked credentials for editors and the "notability purge" aren't helping.

There have only been two fundamental revenue models of content for 25 years now - EndUser and Advertiser. The ISP's went through the throes of the switch from PerHour to FlatRate in the 1990's, and the RIAA is struggling with it now.I don't know anyone who would "pay to search" casual queries. There are some professional databases which do operate on this principle for high powered content.

From the RIAA threads we learn people don't want to pay as endusers for their content. The post above asks about the ad

From the RIAA threads we learn people don't want to pay as endusers for their content.

Great post, except this part doesn't make any sense. I pay as an end user for content all the time, and not just for high-end data: Magazine subscriptions, membership in various societies (and their publications), newspapers, my ISP, government funding (I pay through taxes), direct donations to non-profits, contributions to wikipedia and other open content systems directly. While some of them are for high-end data, a lot of it is not.

Is content going to ever be totally free? It will be if people understand the inherent rewards of an open society. Information's negligible cost of duplication is the revolutionary model is the thing that is shattering the old models (c.f. http://homes.eff.org/~barlow/EconomyOfIdeas.html [eff.org]). Wikipedia is already doing that. As much as I'm a critic of Jimmy Wales, citizendium, etc. (with their NPOV lunacy), the system he's helped build is saving people's lives and improving quality of life in ways the old world just doesn't understand yet.

Personally, I'm hopeful that as long as we still have the Right to Read (c.f. http://www.gnu.org/philosophy/right-to-read.html [gnu.org]), we're on the path to freedom and salvation. A corporation who makes up a new "model" to take advantage of content producers isn't going to take hold anymore. There's just not a point anymore. The price of content is already quite low for common knowledge. Even if the arbiters of knowledge try to keep it from common knowledge, we can paraphrase it. The greatest risk to real productive use of our knowledge still remains Patents. Information may finally be free, but the freedom to tinker is not.

"the system he's helped build is saving people's lives and improving quality of life in ways the old world just doesn't understand yet."

The old world was based on personal greed and individualism, even ideology cannot trump efficiency and human community, it's nice to know that inside there is a helper/altruistic tendency to human beings.

I was a little fuzzy with my set theory. "A certain segment of people" doesn't want to pay as an enduser. Those are the people who originated the new battlefront. If you pay the classical way, great for you, great for them, everyone goes home to dinner.

Content is becoming "free as in beer" to the enduser for the next phase, financed by ads.someone can figure out how I can earn back the check I just dropped off to my landlord today.

There is a difference between healthy competition and trying to drive your competition into the ground in order to squeeze every last penny from the market.

Competition should not be confused with the anticompetitive mafia like behavior that we all too often see from some other big business. For example, if Google acted like Microsoft (or the old AT&T and IBM) they would use their market position and simply require websites to exclude other search engines from indexing their web pages or else be exclud

So basically...they're asking for people to abuse the ranking system. To patrol something like this would require a company with resources like Google, and most likely the reason Google doesn't have such functionality. Just my two cents.

The poster may be joking, but it's also true. If the search engine becomes popular you better believe SEO people will be the first to get their sites on there and eventually the site will have to change it's stance on being an open search engine. Who wants a search engine that doesn't find what you're looking for.

an open search algorithm can more easily be checked for any flaws and by extension, can be fixed- closed search can only be reverse engineered for good or bad. with closed source you may be able to find a problem with the software/algorithm but there is nothing you can really do about it, it's completely at the whims of whoever created it and that's the problem.

Having an open algorithm is good, as non-disclosure isn't security, but the issue is allowing people to rank searches and such. Having that public is asking for people to abuse the system, and as noted before, a lof of malicious parties could seemingly legitimately rank their sites (porn sites, etc) higher, leading to ranking battles by bots. Of course, the issue of vandalism occurs with Wikipedia, however when people are looking to make money off of it they'll likely be more persistent.

Having an open algorithm is good, as non-disclosure isn't security, but the issue is allowing people to rank searches and such. Having that public is asking for people to abuse the system, and as noted before, a lot of malicious parties could seemingly legitimately rank their sites (porn sites, etc) higher, leading to ranking battles by bots. Of course, the issue of vandalism occurs with Wikipedia, however when people are looking to make money off of it they'll likely be more persistent.

With the criteria publicly accessible, yes, porn sites may be able to try to game it, but all the somewhat large producers will also be able to game it as well. Any engine with a flaw that allows it to be "gamed" is not a robust engine at all. If anybody spends a few weeks on the problem they can investigate any system and reverse-engineer it. Google is no different in this regard. A lot of what you don't realize is that a lot of google's processes are actually very manual. What do you think they are

Having an open algorithm isn't enough. If the community is contributing to a database, and relying on that database, then the database has to be open, distributed, and resilient, along with the algorithms, protocols, etc. Otherwise, it won't belong to the community that created it, but to some company that can act as gatekeeper to our own data, or even coerce us into using it.

it's completely at the whims of whoever created it and that's the problem.

Funny, I prefer it to be under control of someone that's in the business of making good search results rathar than a bunch of wankers/trolls/bots trying to lure me to their site even though there's a hundred others that would be more relevant to my search.

Point well made - while spam attacks may be pretty obvious, they could be spread out over time to make them less obvious.

Additionally, I can see this search engine being very much affected by public mood. For example, say there was a royal death and a certain right-wing 'upmarket' tabloid newspaper [dailyexpress.co.uk] decided to claim that it was a conspiracy by the Government to kill the royal off. This is linked to from said newspaper's web site, and this people improve its ranking. Therefore it floats to the top of the results pile, thus giving it more exposure and setting off a vicious cycle.

Just a hypothetical situation, but certainly possible. Such a model would also make it possible to carry out smear attacks and to ruin the rankings of competing companies, parties, organisations, whatever - a practice that IMHO should be left to search engine admins.

Such a model would
also make it possible to carry out smear attacks and to ruin the rankings
of competing companies, parties, organisations, whatever - a practice that
IMHO should be left to search engine admins.

Oh yeah. Let's give the highly underpaid, highly overworked admins yet more
unrelated tasks to carry out! Can't you people do your own company smear attacks? Why do you want to bother the admins with that? Besides, smearing competitors is technically new content, and content creation i

I remember reading a slashdot story one or two years back explaining
that there are people at google whose job is just that. Of course, once
they go down that path, it becomes impossible to analyze, scientifically,
the behaviour of such a search engine.

There are a lot more models than that. The backlink model is one way, then there's the hubs/authorities model which goes both ways, and there are many obvious variations from there.

A "community" model is unlikely to scale. The early search engines like Yahoo were community models, and a good example is still dmoz. These models started getting beaten around 96 or so by fully automated systems, because such models don't have depth. You get a tip of the iceberg effect, because there is only so much editorial

The question of abuse is obviously one that we are taking very seriously in thinking about design issues. My belief is that the key to solving this thorny question is hinted at by the success of wikis and the wiki model: the key is to put tools in the hands of the community that allow for broad oversight and control by the community in a process of open dialogue and discussion. This is very different from approaches that allow only for atomistic participation by a "community" which is never allowed to really become a community due to excessive reliance on algorithmic voting systems and similar.

One of the first lines of defense in the early days will be use of a community (wiki) generated whitelist [wikia.com] of sites to crawl. We will want to work outward from there, but basically the first thing is for us to assess "look, what are the most important must-have sites on the net" and crawl them.
One thing that the mainstream media never seems to report very well, mostly because I think they don't get why it is important, is that we are doing everything here under free licenses. The software GPL, the data we generate under free licenses, etc. The aim here is not just to create a good search engine, but to create it and *give it all away* in a way that I think has a chance to restructure the entire search industry. Well, maybe not, maybe so, but what the hell, it'll be fun to see.:-)

Unfortunately for you your track record disagrees with your promises. You and your website have a history of abuse and bias that rivals that of any on the Internet. Your management incompetence of Wikipedia is so bad that you have dedicated websites documenting it. From secret mailing lists to the junior high style politics that rule your sham open organization, you are incompetent.

The thought that Jimmy Wales, cofounder of Wikipedia could have an open site without abuse is laughable. You operate under the sham of an open community, yet exclude those outside a very narrow political agenda. Your a fraud, using open source principals as a smokescreen that presents your personal world-view set as fact to the world. I don't buy what your selling, and I'm calling your bluff. The sad thing is that this will probably make you a fair amount of money if more people don't start to see through you.

But then the wonderful thing about leading revisionist history is you can substitute your own revisions for reality....

As trollish as parent is perhaps, he is unfortunately speaking a trollish truth.

Speaking explicitly as a reader of slashdot, with all the group-think biases a site like this introduces, wikipedia is floundering in a mire of their own arrogance, and the dissatisfaction with this needs to be heard.

"You operate under the sham of an open community, yet exclude those outside a very narrow political agenda. Your a fraud, using open source principals as a smokescreen that presents your personal world-view set as fact to the world."Actually, no. Wikipedia can be criticized on a lot of grounds, some of them even valid:-); but that it presents my personal-world view or that we exclude people outside a narrow political agenda is just... not grounded in fact.

Yes, clearly, just like Google Search has an abused algorithm, Wikia Search will surely see abuse somewhere in its system too. I don't think there's any question of that, and neither that Wales challenge your opinion in saying abuse will be non-existant, only that the issue will be tried to be dealt with, for example like it is on Wikipedia, and like it is on Google. The important part here isn't whether it'll be free from abuse (it will see abuse), but how efficiently it will be able to deal with the abuse

The thought that Jimmy Wales, cofounder of Wikipedia could have an open site without abuse is laughable. You operate under the sham of an open community, yet exclude those outside a very narrow political agenda. Your a fraud, using open source principals as a smokescreen that presents your personal world-view set as fact to the world....

Abuse potential is the first entry on they whiteboard when it comes to designing a new internet site these days. It's a pity, but that's the way it is.I've been running a (small, nothing compared to what you're doing) community powered search engine for a while now (little less than one year), it's been a neat little project and I've learned a lot.

I think the combined power of having your name and wikipedia as a launchpad and quite probably the capital to see this through may give you a chance worth taking.

So basically...they're asking for people to abuse the ranking system. To patrol something like this would require a company with resources like Google, and most likely the reason Google doesn't have such functionality. Just my two cents.

And when you think about it, Google's pagerank algorithm already returns search results based on what the community thinks. This new venture is simply a means to take other peoples' sweat equity and turn it into profit for good old Jimmy while giving the people who did all the work little more than warm fuzzies inside, if that.

Pagerank is based on others linking to websites. So in Google's case, the community == the entire internet instead of some limited subset who get to decide what's relevant and what isn't.

is that anything like addwords are democratic (as long as you have the money to pay for them)

You mean AdWords? No, I wasn't referring to that -- see my comment above. Also if this new venture is to be for-profit, you think there won't be paid ads of some kind? Perhaps money exchanging hands for certain benefits as long as you have the money to pay for it?

One of the factors when you do a search on Google is PageRank.It is community driven yet difficult to manipulate.

If I link to your site from my site then its a vote for your site.The vote quality depends on the number of votes for my site.Paradox? Yes it is but you can calculate it to a certain extent and it gives the perceived quality of a site by the community.

These comments reek of jealousy. The peoples sweat is volunteered by them, and in return Jimmy gives them free hosting and exposure while making a tidy profit. He's a hero of the public domain movement to me, he has enabled an unparalleled wealth of free content (perhaps Sourceforge is a parallel), anyone who finds a way to make open collaboration profitable and therefore competitive in this capitalist world should be heralded a champion of the open source movement. Profit is not intrinsically good or evil,

He's a hero of the public domain movement to me, he has enabled an unparalleled wealth of free content (perhaps Sourceforge is a parallel), anyone who finds a way to make open collaboration profitable and therefore competitive in this capitalist world should be heralded a champion of the open source movement.

And Wikipedia is profitable? I thought it was sustained by the donations of corporations (bandwidth, money, etc.), individuals (such as the latest fundraiser), and the efforts of its volunteer army. Where's the capitalistic profit? If he manages to make this one profitable then start the hero worship at your leisure, but so far I don't see much in the way of success.

Out of the equation. Open source is fine, but it will end up making itself useless as it will be profitable to get the code and figure out how to artificially UP your rankings, or sell that info, or pretend you know how, and just take people's money with the promise that you can help them. We don't need more companies doing that, god knows.

The idea is to challenge the established players by offering a search service that is more transparent to end users, meaning they can see how search results are arrived at. Wales has described Yahoo and Google as opaque services that don't explain how results are arrived at.

Personally, I don't care how search engines rank the websites they return as long as what is returned is proper, relevant and useful.

I predict significantly more vandalism and self promotion with this project than with Wikipedia. That said, I still think it's a good idea. But the article had a very low content:words ratio, so I don't really have a good idea as to how it will be implemented.

Since this project would seem to depend on the participation and good-will of users in order to work, my guess is that a nonprofit version will follow shortly afterwards, paralleling the open-source model. I also predict that without the benefit of a massive Microsoft-esque head start, the for-profit version will be put of business in short order.

Someone loading the dice on what I get back from a search. At least with the current crop, I can more or less figure what they're doing. With a dynamic, anything goes approach, I seriously doubt I'll be using it much.

Search for Kobar Towers and you get 0 relevant articles. Search for Khobar Towers and you get 62 articles. Yeah, the first is a misspelling, but it's 1 letter off, nothing difficult for a spell checker to check against a dictionary of existing articles. What use is a search engine if it is so strict that I have to enter the terms exactly to get an article when I could just do that in the URL?

As long as I need to use google to search Wikipedia, I don't see Wikipedia creating a google killer.

I completely agree. I am continually amazed at how good google's input-correction is - if I do a search for 'pale gire', it knows to correct it to 'pale fire [wikipedia.org]', yet if I do a search for 'canadian gire', it's clever enough to work out that I mean 'canadian tire [wikipedia.org]'. I'm also continually amazed that people running other search services haven't yet realised just how vital this feature is - it's probably one of my favourite things about Google. Less so for monosyllables, but it's useful for words like "monosyllables". I'm particulary surprised that prominent online dictionaries don't have similar funcionality, seeing as I would imagine a large portion of their usage is to find the correct spelling of words.

It's not hard to make a list of all the search phrases
people enter into the search bar, and if you have millions of people searching,
you'll get a lot of redundancy on even fairly esoteric typos. Since many
people also correct their own typos, if you combine and remember
all consecutive search
phrases by the originating IP address, you'll get a frequency bump for when
a common typo is immediately fixed.

With the volume of search
phrases going through their servers, there's no need for

Google's mentioned a variety of techniques publicly, although there's sure to be some secret sauce as well. The most obvious check would be a dictionary-based spellchecker. They can also look for letter transpositions, misstruck keys, word-form matching, etc.They also do a variety of statistical analysis on a ridiculously large data set. For example, if a particular phrase appears over and over again, and all of the words in the query match the phrase save one, it may be more likely that the non-matching

Exactly. I have *always* used google to find wikipedia articles. You can't beat google with a 'site:' prefix. As a matter of fact, I have the following firefox bookmark stashed under the "wik" keyword"

This is simply his response to Google starting what amounts to competition for wikpedia. I'm sure google is having quite the laugh from it - one wonders how much of the donations for wikpedia are being used towards this thing.

If you think wikpedia gets vandalized, wait until there's money involved. Wikpedia for all it's trappings, doesn't directly influence spam. But a search engine... IF, and this is a big IF, this thing becomes mainstream, having the code public will make it very easy for the bot herders to control it. The idea is simply flawed. Google is currently dealing with bot herders attempting to manipulate it's page ranks - while the idea of it being open source sounds great (well, ok it doesn't to me - I don't have the love affair with open source that most slashdotters do - I've never bought into the security myth that there's GOOD coders out there with so much free time on their hands that they are walking OTHER peoples code. I don't like doing that when I'm PAID to do it. Not too mention there just aren't that many good coders out there....but I digress) it's simply going to work right into the hands of the malware crowd - especially now that it's more organized crime than it is vandalism.

having the code public will make it very easy for the bot herders to control it

This assumes that it's impossible to devise a web search algorithm that can automatically and reliably avoid poisoning. I don't think there's any particular reason to believe that this is true. Humans are quite capable of identifying with a fairly high amount of accuracy what sites are just linkfarms, or useful sites which are the target of a de-ranking campaign, so it should be possible to have a computer automatically do it.

I'm not saying they will succeed, just that it isn't necessarily flawed at a f

Just because we don't currently have an algorithm that can defeat spammers, it doesn't mean it's impossible to ever create it. The fact that "even Google" are constantly tweaking their search methods suggests that they don't believe it's as good as it can possibly get, and theirs is just one possible way of indexing the internet. The answer may well in a completely different approach.

My hunch is that the "end game" will be a massively distributed network where individual users index some subset of the int

Again, it would be hard for this to be a response to Knol, since I announced it and have been working on it for a year.:-)

And, if you read the linked article, you would know that *zero* donations from Wikipedia have anything at all to do with this: Wikia is a completely separate organization.

Also don't make the classic mistake of thinking that "open source" automatically means "volunteer coders". It generally does not, and the classic FUD from the proprietary world fails to describe reality for precisely this reason.

And finally, one of the most important concepts here is that of a broad deep whitelist, which is something that I think can be done realiably and well with appropriate tools in the hands of the end users. The entire problem of bot-driven spam comes from a lack of reliable quantities of human oversight in the process. All you have to do to massively spam google is fool a computer. (Well, even then, google does a pretty damned good job of preventing massive spam though of course there are always some problems.) Pretty hard to get that nonsense by a properly organized community effort.

(But of course, the design of a community which can move things forward quickly without a lot of useless work is nontrivial.)

What I think is a joke is the advertising model. Wikipedia wouldn't be Wikipedia is there were sponsored ads. One of the first things I saw on the Wikia Search link was an ad for Netflix and a promise of the site's four Open Principles (which includes Transparency).

Can I ask a question I think is important... whose pockets does the advertising revenue go into? Because my desire to support the site is incredibly contingent on the answer to that question. And my inquisitiveness about the answer is what

Ugh, this comment needs to be modded down ASAP -- we have already had info on this in the comments that the Google competition aspect is untrue, heck, even I remember many past Slashdot stories on this since before Google Knols were announced.

But building a search engine is a little ambitious, even for the co-founder of Wikipedia. Maybe he should start off small, like searching one website. I even have the perfect one to start off with: its search feature is so bad that if your search is off by one letter, you have a good chance of not finding what you're looking for. Maybe you've heard of it [wikipedia.org].

Wikia's search engine, based on the user demographic they have now, is going to have great coverage of furry fan fiction. [wikia.com]

There's already a good manually-updated search engine. It's called Open Directory [dmoz.org]. It's quite useful as a data source for answering the question "what is this web site about"? It tends to run months behind changes to the web, since it's manually updated. While not many people query DMOZ manually, it's used by Yahoo, Google, etc. to get some basic information about a web site.

As an example of how great Wikia search is going to be, Wales suggested searching for "Tampa hotels". [techcrunch.com] The major search engines return too many bottom-feeder reseller and directory sites for searches like that. As I point out occasionally, we've already solved that problem over at SiteTruth [sitetruth.com], which looks for business legitimacy. Type in "Tampa hotels" there and watch it push the marginal sites to the bottom of the search results. We have that one handled.

Wikipedia works because people are willing to do substantial work for free for a non-profit organization. That doesn't work for a commercial business. You can get people to write about themselves (Myspace, Facebook, etc.) but beyond that, "crowdsourcing" doesn't go very far.

that if this succeeds, and can race past MSN in terms of popularity, it will show to the world, that MS's gripe about Google truly was worthless. Of course, MS will use that to tell our congressmen that the glass is half empty, not half full.

Wikipedia receives most of its traffic from its articles appearing in Google's search results, Wikipedia being relevant content, and Google being the top search engine.

How is Wikipedia to draw traffic to their search engine? Obviously not via Google, as search engines are content free on their own. Integrating it with Wikipedia? But again, Wikipedia is the end target, not a start point, so how could this work.

I don't think Wikipedia has the strategy or money for this to reach critical mass and show its potential, but it'll be interesting as an experiment.

How is Wikipedia to draw traffic to their search engine? Obviously not via Google, as search engines are content free on their own. Integrating it with Wikipedia? But again, Wikipedia is the end target, not a start point, so how could this work.

and it's pretty useful. I don't know why everyone is complaining in the comments so far. This user-participates-in-ranking magic is not exactly news, and anyone who has studied or worked in information retrieval knows this. With a large enough number of benign participants, it should work.And since people are bringing up Google as competition: Google Search has an estimated retrieval accuracy* of around 10%. Not very hard to beat, except that the Internet is a rather large document set. Have you ever browse

Wikipedia is falling apart because no one agrees on anything. Any additions are being deleted because they don't contain PC(pol.correct) language.

Wikipedia works for me, despite the occasional flaws.

Why would the search website work?

With the amount of adverts wikia keeps shoving on their free hosted Wikis, I already feel assaulted - I'm not sure if Wikia can pull it off. Another company like the Wikimedia foundation on the other hand, in my opinion could pull it off.

Doesn't Google do this already? AIUI the PageRank system looks at how a web site is linked to and connected to others. If a site is good an relevant, it will be related to other sites through links, blog posts etc, i.e. the net community in general. Perhaps that's even better than a group of self-appointed editors, and of course it's not the only factor Google uses.Because of Google's anti-spam techniques this method is very hard to game. Better still, the defences are automatic. On Wikipedia, a lot of vand

If they let me remove results from my personal ranking profile.... maybe with an option to show me how many total hits and a way to retrieve those alternate results if I need them.... then I'll be using it.What I want is to get rid of things outside my daily context. I work in web development so when I do a search for "CSS 3 column hack" I don't want to get results related to some sport team abbreviated CSS where they have a coach whose a hack but likes to use a 3 column lineup.... stretching a but but you

Funny to read this today, after I spent a couple of hours yesterday searching Google for something that doesn't exist -- a Plucker [plkr.org] type app for the iPod Classic. What struck me was just how badly Google performed. Any search containing the word "iPod" seems to return pages upon pages of blog entries about the (long since released) iPhone. What one tends to find with a Google search are a lot of loud, content-light blog entries, popping with ads, with short dashed-off articles broken across several pages.

Completely different.:) For one thing, we are doing everything completely freely licensed. Mahalo is proprietary.

For another thing, Mahalo is "human edited" search results for the top queries, which is not a bad idea of course, but it is not intended to be a full search engine. Mahalo have indicated an interest in replacing their google search backup with our open source alternative, if we get to be good enough, which is obviously a far from foregone conclusion.