Posted
by
ScuttleMonkey
on Friday March 20, 2009 @04:30PM
from the keep-your-crowd-source-off-of-mine dept.

Wired is reporting that ecologists think the internet could act as an early ecological warning system based on data mining human interactions. While much of this work has been based on systems like Google Flu Trends, the system will remain largely theoretical for the near future. "The six billion people on Earth are changing the biosphere so quickly that traditional ecological methods can't keep up. Humans, though, are acute observers of their environments and bodies, so scientists are combing through the text and numbers on the Internet in hopes of extracting otherwise unavailable or expensive information. It's more crowd mining than crowd sourcing."

I foresee a situation where as more of the world becomes ever present on the Internet, ecologists will report a fast growing decline in the ecosystem. This will result from data mining efforts showing increasing change in the environment simply because there are more people reporting.

You mean, kind of like the Department of Homeland Security, the CIA, the NSA, and of course, our wonderful color-coded "Homeland Security Advisory System" [dhs.gov]? The same people who consider citizens who carry the Constitution or know it well and who support libertarian-minded candidates such as Ron Paul to be domestic terrorists, and yet don't deem illegal aliens, er, "illegal immigrants" to be any kind of threat whatsoever?

True and this is really, really old news. I recall around the time of a quake (I think the world series quake) in the SF area, a time so far back in the past that SCO were Good Guys, other more remote folk followed the aftermath of postings on ca.earthquakes and we got the first real data on what happened by determining machines that were down from nearby posters.

It turned out that that particular quake was practically underneath SCO in Santa Cruz and everyone was relieved to see a "We're not dead yet!" me

If you create a futures market for prediction of future events you give people incentive to share the info they have, and a way to benefit from putting this information on the market, to pay them for sharing knowledge. The Pentagon tried to create such a market for terrorist attacks a number of years ago and the political establishment caught wind of it and murdered it in its crib, calling it a cynical way to make money from terrorist attacks. The truth is, it's a damn effective way to get people to share rare but critical info and no other mechanism is as effective.

This reminds me of an old quote that is apropos in today's political and news-climate: "We should never choose what we want to be true over what we know to be true." Politics shouldn't kill things that produce results.

So, if you want to predict ecological disasters, nothing is stopping us from doing things in a similar way. Some lonely janitor in a nuclear facility with inside info about how poorly the place is being run could bet a thousand dollars in the ecological futures market that said reactor was going to experience an environmental problem within the next year and that would quickly become a sign, as others in the know bet the same way, that the place needs attention and needs it now.

Not exactly. The problem is how to provide incentive for those with rare but useful information to spread that info in society. Doesn't matter what topic we're worried about, that is a problem all of society has. The answer to that problem is a futures market on events.

As for manipulating the markets... lol. Let's say our janitor does bet that reactor 5 is currently leaking. If that's true there's no way to manipulate it. All that can be done is to take his bet. If someone takes his bet, buys his future, th

After all, there was no investigation of the tremendous imbalance of put options on companies hurt by the disaster prior to 9/11.

Actually, it was investigated by the 9/11 commission, you just don't like the conclusion they came to. (He who does not see the conspiracy must be part of the conspiracy, right?):
Snopes: Put Paid [snopes.com]

(You know, I like a good conspiracy theory myself... but you Truthies really need to give it a rest [ww4report.com].)

"The market offers no way to subvert that process."
Other than the janitor placing the bet, then loosening a key bolt the night before he takes a vacation over seas.

Yup. A market in disaster prediction would offer an economic incentive to causing disasters combined with a way of laundering the income from the crime -- and the libertarians in the audience can't see a way that that might cause problems. But isn't human behavior ruled by economic incentives?

I'm not an expert on these types of markets, however, just know that they do work extremely well for what they're designed to do: place value on rare and useful information that would help society if it were known. The market offers no way to subvert that process.

Sure it does. As the other replier notes, moral hazard is one of the bigger problems with markets. Let's give an extreme example. Suppose someone had placed a bet on your life for $10, double or nothing that you would die by the end of the month. Nothing of consequence would probably occur except you'd probably think it was in poor taste. Nobody sane will do you in for ten bucks. Now supposed someone placed a billion dollar bet that would pay out double or nothing to them if you died by the end of the month

You can't have a free market and regulation at the same time. It doesn't even make sense - laws and regulation are the opposite of free. You might as well say "We'll have free speech right after we stop people from communicating."

The real markets only collapsed because the government messed with them. If the government hadn't made it profitable to make bad loans, the banks wouldn't have done it.

Dur, the government "made it profitable" by deregulating the mortgage industry in the 1990s. There (likely) would not have been a housing bubble if the government had maintained its pre-Clinton housing regulations.

No, the deregulation in the 90s was just so they could make it profitable later. They made it profitable later on, by having Fannie Mae and Freddie Mac "secure" bad mortgages, in the name of helping poor people buy houses. Previously, the banks wouldn't give loans to people unlikely to pay the

No, the deregulation in the 90s was just so they could make it profitable later. They made it profitable later on, by having Fannie Mae and Freddie Mac "secure" bad mortgages, in the name of helping poor people buy houses.

The real markets only collapsed because the government messed with them.

That's only part of the story. I'm a libertarian, but I've lived too long in the 3rd world to accept the idea that private enterprise is going to build certain critical infrastructure (like roads, sewers, etc.)

If the government hadn't made it profitable to make bad loans, the banks wouldn't have done it.

The bad loans are only the tip of the iceberg. The real monster is all the outstanding derivatives. I don't think anyone knows how big it really is, estimates vary by the hundreds of trillions of dollars how much it is, and the most pessimistic figures I've seen dwarf global GDP.

Let's try to come up with regulations that actually work before we expand their use.

Hmmm, like the regulations that Wall Street got around by inventing new financial instruments that hadn't been regulated yet?

It seems that the answer someone got in an open questions period in an upper division Accounting class[1] I took - she asked for an explanation of derivatives and the answer was "I can't explain it here", is still true. But the reason is that no one probably understands the full extent of what has happened.

Markets are very, very, very easy to manipulate.

As are regulations for insiders with the right connections. Why are USians so

Markets are very, very, very easy to manipulate. That has been proven beyond any doubt. Let's try to come up with regulations that actually work before we expand their use.

Compared to bribeable, incompetent regulators? I don't buy it. Besides market manipulation is a non-problem. It grows more interest in the securities, creates more opportunities for knowledgeable traders to make money, and adds liquidity to markets. I'm all for it. Honestly.

In my view, there is a profound ignorance about what is and isn't a problem in markets. Market manipulation, insider trading, abrupt changes when new information comes out. These are means by which traders in the market become informed.

That profit motive already exists. The nuclear facility often is owned by a public corporation. Hence, if the janitor short sells the relevant stock, they can make the killing without requiring a terrorism market.

"Web crawlers can collect information on the drivers of ecosystem change, rather than the resultant ecological responses," they write.

Which assumes they have a good model of what's driving the changes in the first place. Of course, I presume data miners are well aware of these problems. Are there any experts on it here who can briefly describe how effective (or not) the current techniques are at sifting gold from the silt?

Are there any experts on it here who can briefly describe how effective (or not) the current techniques are at sifting gold from the silt?

I would imagine so. In the old days, we used to call this a "bullshit filter". So long as you have a good representative sample, statistical methods Just Plain Work. I described some of this in a previous message in this thread.

The biggest danger in any of this sort of work is wanting the results to come out the way you wish them to. If I took a direct poll of my friends and coworkers I talk to, the approval rating of the man bowling 129 in the White House basement would be just around 0, but that is ob

I could be a little off base - it's happened before, I'm sure it'll happen again - but this sounds like a load of shit.
We already know there's a problem, what we need to do is fix it.
I don't think that looking into articles on the internet is even close to a solution. It just sounds like a group of people that are good with data trying to cash in on some grants or corporate funding. We don't need more spin doctors.
They need to get off of their asses and do something useful, like take part in some re

people are TERRIBLE witnesses to events. ask 2 people what they thought happened and you will get 2 different stories. just look at how crap the mainsteam media is at reporting on science subjects, and your telling me you want to use this as a data feed to determine the state of the planet? the red light for "everything is fucked" will be flashing constantly no matter what because no one ever writes stories about stuff that's fine.

Arguably, signal cuts through noise. If millions of people are wrong about it but in random directions, the handful of people that are completely right about it, and the elite cadre who are just slightly more right than wrong about it, might turn into a usable few bits of info on the subject.

millions of people voted for bush, twice. just look at the millions of blogs and the quality of their posts. i argue that speed and efficency of information is useless of the information is no good. hell look at/. and how this supposed "wizdom of the group" thing fails hard.

i argue that speed and efficency of information is useless of the information is no good. hell look at/. and how this supposed "wizdom of the group" thing fails hard.

See my earlier postings in this thread how I have used data similar to this in real life.

There is absolute gold to be found on slashdot. It requires a lot of work, patience and reading at -1. Suggest you learn how to use that big key on the left side of your keyboard labeled "shift" to make an uppercase letter at the start of each sentence. It's a lot easier than data mining on/., though equally as rewarding.

Arguably, signal cuts through noise. If millions of people are wrong about it but in random directions, the handful of people that are completely right about it, and the elite cadre who are just slightly more right than wrong about it, might turn into a usable few bits of info on the subject.

Precisely correct. Excellent summary.

Also note that assigning a value to "elite cadre" is critical. Ex: Al Gore is NOT an expert on the ecology of the earth, he is a politician. He is expert on begging people for campaign donations (and perhaps knowing when to drink tea so he is outside peeing when critical conversations are taking place in meetings he is supposed to be attending).

people are TERRIBLE witnesses to events. ask 2 people what they thought happened and you will get 2 different stories. just look at how crap the mainsteam media is at reporting on science subjects, and your telling me you want to use this as a data feed to determine the state of the planet? the red light for "everything is fucked" will be flashing constantly no matter what because no one ever writes stories about stuff that's fine.

That's an excellent point, but weather stations (and other sensors) and cellphone cameras are not people and are pretty good witnesses.

So by the time we burn more electricity to filter out the bullshit and gross errors we'll be well seated to look at a data set that say we've done too little too late.
Sounds just like everything else on the net if you ask me.

Most climate change theories put the temperature shift at fractions of a degree every decade. But this is too fast to keep up with the information?
Most areas with extensive internet are already under 24X7 satellite surveillance that can measure temperature and precipitation. What are they expecting to get out of this?
If I'm sick, I search for "flu remedy". I will likely be sick tomorrow. It is possible I've spread germs and others will get sick. A new trend of searches for "flu remedy" is a decent indicator that people are concerned about the flu, and that is a decent indicator that they know sick people.
If I'm cold when I go out, I search for "winter coat". It may or may not be cold tomorrow, but probably will be since cold days group together. It is likely other people are also cold. But this isn't a predictive indicator. I'm cold because the temp already has dropped. The weatherman knows this, and I may have done my search because he said it was going to be cold tomorrow. The satellite knows this. Vast records are kept about how cold it was in Podunk, Kansas. You could predict that coat searches will go up because it was cold, but once people are searching for coats it's as a result of data you've already got from much more scientific measurements.
Even if you could find a method that was predictive, what indication is there that populations, even if good at predicting short term weather (I'm not convinced), are competent in any way at predicting long term climate changes? Are they really arguing that the people, many of whom have a disturbing habit of living on flood plains, barrier islands and below sea level, are going to produce accurate data about future environmental shifts?
It seems far more likely that this is a chance for someone to do research using a cool new toy. Our climate models current are pretty universal in their failure to predict the future, and that's with tons of solid data. This seems like a buzzword filled hole designed to sound urgent and cutting edge. An unproven model using an unproven data collection method can be used to justify any conclusion that will keep this project funded as long as it sounds cool enough to the people footing the bill.

Most climate change theories put the temperature shift at fractions of a degree every decade.

The climate change models are only based on fairly recent data (and the most accurate data is only a few years old).

The scare mongers (like Algore and company) are insisting that climate change is caused by people and the longest-term records show that the earth has undergone many different long-term climate changes like ice ages and subsequent global warming to get things back to "normal".

I would give the recent data putting temperatures as cooling in the last couple of years as much weight as I would put

We already are overpopulated, we already are consuming much more than the biosphere is able to produce with sustainability and we know it.

Maybe, no, and no. There are all sorts of hysterical claims that the "carrying capacity" of Earth is lower than it currently is. To be blunt, we have plenty of food excess now, we have little evidence of this lack of sustainability.

It looks like, in the US, the per-acre yeild for wheat production has increased, over the past century, at about the same rate of population. At the same time, fewer people are involved in agriculture, and the price of food is declining. The amount of land used for farming seems to be decreasing. From the looks of it, the production of food follows much the same model as the production of other common widely distributed goods. Over time, methods grow more efficient, the price drops, fewer people are nee

Here's an excellent example of human beings not being able to sense long term trends because the pain here and now looms so much larger than the good over time. While the short term trend is that fuel, and therefor food, prices are increasing, the long term trend is decline. According to the USDA In 1930's people spent about a fourth of their total income on food. In 2004 it's less than a tenth. Also, that's the consumption of food in a time when American's have grown fatter. More food is being bought