Gamers' designs turned into real molecules, and work better than computers'.

Citizen science, the movement to draft non-specialists into areas of scientific research, doesn't require the volunteers to put on lab coats. In at least one case, scientists turned a prickly biochemical problem into a game and found that the gamers could typically beat the best computer algorithms out there.

But all that work was done on cases where we already knew the answers, which was how we were able to measure the gamers' success. Now some researchers have taken this approach one step further and created a hybrid project that mixes volunteers with lab-coated workers. 37,000 enthusiasts were given the chance to take on algorithms in designing new RNA molecules. And once the gamers had a chance to vote on the best designs, the winning designs were sent to a lab, synthesized, and tested. After a few rounds of this, players were not only handily beating the computers but providing rules that went into designing the next-generation algorithm.

A consortium of researchers at Carnegie Mellon, Stanford, and Seoul National University put together what they called a Massive Open Laboratory. Operated through a Web portal called "eterna," it provides a few tutorials that allow people to bring themselves up to speed on the base pairing rules that govern the structure of RNA molecules. These structures can fold up a linear RNA molecule into a catalytic form or allow it to bind other molecules and proteins. These structures are essential to basic cell functions, such as turning genes into mature messenger RNAs and then converting these messengers into proteins.

Once the tutorial is done, volunteers can start taking part in challenges like the one shown on top. By setting the composition of an RNA's bases, they can attempt to get it to fold into a structure provided by the researchers. You can then perform further tweaks to make the structure more robust. Through this process, the players get to learn the basic rules of what makes an RNA structure energetically stable.

At least, energetically stable based on calculations. But the Massive Open Lab also had a lab, and after a round of challenges ended, those in lab coats actually synthesized and tested the top molecules. The results were given as feedback to the players, and a new round of challenges began.

In the first round, the current state-of-the-art software did a better job of designing RNA structures than the players did. That didn't last, though. "As the community gained experience with empirical RNA design cycles," the authors write, "performance improved, and community submissions converged to successful designs." By the third challenge, the best player-contributed designs were outperforming the best algorithms; by the sixth round, the median player design was better than the best that the computers could come up with.

The system also allowed the users to create simple rules to help them design effective structures (things like "put a G-C pair at the base of a stem"). The authors checked these and found that many of the design principles hadn't ever been identified in the scientific literature. So they designed an algorithm that could incorporate the user-designed rules. That algorithm also beat the previous state-of-the-art and often approached the best of the human designs.

From here, the researchers behind the Massive Open Lab plan on taking the work in two directions. They intend to try to figure out why, at the biochemical level, the players' rules result in a more stable RNA structure. And, at the same time, they want to set their players onto designing more complicated structures—ones that could bind a specific molecule or perform a catalytic reaction.

People are just naturally much better at "guessing" randomly than computers are.... I guess?

I'd guess the trick is the computer can only do what its told, so its partly on that and any blind-spots the programmers had. It can't improvise, guess, or go "eureka!", nor think outside its programmed box.

edit: in short, when the new ideas were added in, the computer did better because of it. Everybody is learning and improving along with the program!

I'd argue the exact opposite, ToolGuy3. The human gamers learned the basic rules, and are clearly using their intelligence to set ground rules, and designing molecules that work much better than a mechanical process.

The computer just follows the "laws of nature" as given to it by the programmers. The humans have the same laws, but can apply intelligence.

The real question here is this: given both the computer and the gamers molecules--is there any evidence just from examining the molecules that one is designed, and the other is generated by the computer?

I'd argue the exact opposite, ToolGuy3. The human gamers learned the basic rules, and are clearly using their intelligence to set ground rules, and designing molecules that work much better than a mechanical process.

The computer just follows the "laws of nature" as given to it by the programmers. The humans have the same laws, but can apply intelligence.

The real question here is this: given both the computer and the gamers molecules--is there any evidence just from examining the molecules that one is designed, and the other is generated by the computer?

Well, it was just a joke, but even so, greater variety than pre-programmed algorithms is still more or less random chance and the intermediate goal was "energetically stable", which is exactly what you would expect with random evolution. The end result ("animal" or whatever) doesn't seem to be known at all.

Edit: Clearly, I'm not as funny as I thought I was. My only real point was that you need lots more "random" (either actually or apparently) choices to have the best chance of finding energetically stable combinations. A few billion years would probably work and we could probably do it faster.

I agree that the interesting question is if the molecules look designed or not. I'm guessing that these will be considerably more "logical" than our own rather haphazard genome.

So if I'm understanding this correctly, a bunch of amateurs basically guessing i.e. random chance, works better than the experts i.e. Intelligent Design.

Hmmm......

I don't think that's it. I mean, the gamers weren't doing stuff by random, as they were introduced to the laws that govern RNA in the form of tutorials, so they didn't start from scratch, and repeatedly iterated on their own designs in order to perform better. The article stated that, after six iterations, the gamers' algorithms were better than the state-of-the-art system's, which would mean there's nothing random, instead the know-how was improved over each time.This is more like what Genetic Algorithms would behave, but instead of entering randomized individuals, the individuals at the initial generation were based on the gamer's grasp of the rules, intuition and a little bit of luck.And hey, the lab-coat guys might be experts, but everyone can learn to do better!

It's reassuring that there are still some games where humans do better than computers. I guess I shouldn't underestimate the power of communal knowledge in solving problems [games], enabled by online communication.

Keep in mind that this is still community-generated, from what I can tell. It's not like individual people were coming up with the best structures, right?

Though it strikes me as odd that no one previously thought of putting a G-C pair at the base of a stem. Really? I'm getting my Ph.D. in biochemistry and it seems like that's a basic thing that anyone working with DNA or RNA should have thought of.

EDIT: For those who don't know, G-C pairs are stronger because there are 3 bonds between them rather than the 2 for A-T, so G-C pairs are often used as "clamps" when designing an oligonucleotide primer for polymerase chain reactions that amplify a DNA sequence, which is more efficient when the polymerase enzyme can bind to a strongly-bound end.

Empirical science at its best. Pattern matching and thinking outside the box is what humans excel at, so it's not surprising that they were able to beat the existing algo's. Still, good to see science input from the masses.

People are just naturally much better at "guessing" randomly than computers are.... I guess?

Alas(?) no, computers produce better "random" numbers than people by many orders of magnitude. What people do is analysis, on both a conscious and sub-conscious level. We're ridiculously good at noticing patterns and do well extrapolating from them. So you throw thousands of people at a problem like this and what you'd expect to see is what actually happened: At first people were largely guessing and didn't know all the rules, so they did poorly. The longer they had to work the more they noticed patterns that worked and the more custom rules they built, leading to steadily improving results.

So if I'm understanding this correctly, a bunch of amateurs basically guessing i.e. random chance, works better than the experts i.e. Intelligent Design.

Hmmm......

See above: This wasn't random. If it was guessing at random then there'd be no real "state of the art" algorithm, you'd just perform a bunch of random changes to some RNA, synth it, and test it to see.

Though it strikes me as odd that no one previously thought of putting a G-C pair at the base of a stem. Really? I'm getting my Ph.D. in biochemistry and it seems like that's a basic thing that anyone working with DNA or RNA should have thought of.

No, the gamers came up with a mix of known and unknown rules. That was one of the known ones; i used it as an example because it was illustrative for people who don't know biochemistry, and easy to interpret for those who do.

An example of the unknown stuff would be what to put in the loop next to the G-C pairs in the stem.

Though it strikes me as odd that no one previously thought of putting a G-C pair at the base of a stem. Really? I'm getting my Ph.D. in biochemistry and it seems like that's a basic thing that anyone working with DNA or RNA should have thought of.

No, the gamers came up with a mix of known and unknown rules. That was one of the known ones; i used it as an example because it was illustrative for people who don't know biochemistry, and easy to interpret for those who do.

An example of the unknown stuff would be what to put in the loop next to the G-C pairs in the stem.

People are just naturally much better at "guessing" randomly than computers are.... I guess?

Alas(?) no, computers produce better "random" numbers than people by many orders of magnitude. What people do is analysis, on both a conscious and sub-conscious level. We're ridiculously good at noticing patterns and do well extrapolating from them. So you throw thousands of people at a problem like this and what you'd expect to see is what actually happened: At first people were largely guessing and didn't know all the rules, so they did poorly. The longer they had to work the more they noticed patterns that worked and the more custom rules they built, leading to steadily improving results.

So if I'm understanding this correctly, a bunch of amateurs basically guessing i.e. random chance, works better than the experts i.e. Intelligent Design.

Hmmm......

See above: This wasn't random. If it was guessing at random then there'd be no real "state of the art" algorithm, you'd just perform a bunch of random changes to some RNA, synth it, and test it to see.

Of course, one of our best ways to construct AI is to use evolutionary algorithms. The design is the intelligence.

What exactly is happening inside the heads of those humans, to beat the computers? Some sort of very plastic pattern recognition is feeding the trial and error, and that seems crucial. After all, the bit we are consciously aware of feels like trial and error where computers are much faster than us. Somehow we overlay with patterns which are chosen below conscious guidance and those prune our choices.

Not impressed with the design in the article. That sucker's going to be way too wobbly and definitely needs more struts.

On a serious note, this is very cool but maybe not too surprising that the gamers can beat the algorithms. This is basically a puzzle in optimising conditions according to an arbitrary set of rules to obtain the desired outcome - aka min-maxing. And if there's one thing that gamers excel at its min-maxing. See any games forum at all where the game involves a modest number of variables - it doesn't take long at all to see the 'best' builds popping up.

The AI in any game always pales in comparison to a good player. Usually the AI can only win by cheating. One of the other things humans have over an algorithm is that they can predict what will happen in certain scenarios and this becomes a part of their design process. Sort of like bite sized rapid prototyping that is internalized to the person's brain. The more experience a person gets, the more accurate and fast a person's prediction becomes. This is more or less a big part of how anyone gets better at any activity.

An algorithm on the other has a fixed set of rules it operates on. A programmer can possibly build in limited predictive cases, but they are restricted by what the programmer knows and can implement. Additionally an algorithm is pure trial and error. It cannot learn from its failures and it cannot rewrite its own rules.

Stuff like this sometimes makes me wonder if I'm wasting my comp's time on BOINC projects. They farm off work for the comp to crunch on using algorithms. But, if the algorithms aren't very good, then it's just a waste of time.

If we can make more games out of science research, I'm sure there'd be tons of asians folks that would show just how amazing they are at beating the pants off of gamification.

The AI in any game always pales in comparison to a good player. Usually the AI can only win by cheating. One of the other things humans have over an algorithm is that they can predict what will happen in certain scenarios and this becomes a part of their design process. Sort of like bite sized rapid prototyping that is internalized to the person's brain. The more experience a person gets, the more accurate and fast a person's prediction becomes. This is more or less a big part of how anyone gets better at any activity.

An algorithm on the other has a fixed set of rules it operates on. A programmer can possibly build in limited predictive cases, but they are restricted by what the programmer knows and can implement. Additionally an algorithm is pure trial and error. It cannot learn from its failures and it cannot rewrite its own rules.

I'd argue the exact opposite, ToolGuy3. The human gamers learned the basic rules, and are clearly using their intelligence to set ground rules, and designing molecules that work much better than a mechanical process.

The computer just follows the "laws of nature" as given to it by the programmers. The humans have the same laws, but can apply intelligence.

The real question here is this: given both the computer and the gamers molecules--is there any evidence just from examining the molecules that one is designed, and the other is generated by the computer?

Well, it was just a joke, but even so, greater variety than pre-programmed algorithms is still more or less random chance and the intermediate goal was "energetically stable", which is exactly what you would expect with random evolution. The end result ("animal" or whatever) doesn't seem to be known at all.

I think you may be confusing evolution and mutation. Evolution is not random. Mutation is random, and most of the time a random mutation is useless or even damaging. The small subset of the random mutations that are actually useful are weeded out by the selection part, that is usually not random at all.

Mutations that are beneficial give the organism a better chance to survive and reproduce, so they are kept. Mutations that are lowering the survival chance the organism or making reproduction hard do not survive, and are lost. So selection is not random, making evolution also not random.

It's reassuring that there are still some games where humans do better than computers. I guess I shouldn't underestimate the power of communal knowledge in solving problems [games], enabled by online communication.

It's no different than cellular automama.

Humans have built turing complete computers out of game of life, but a computer can't tell you what the next state is without advancing the state.