Become A Peer Reviewer For Citizen Scientist

One of the keys to our scientific community is the concept of Peer Review. When important discoveries are made, the work is reviewed by others accomplished in the same field to test the findings. This can verify the work, but it can also open up new questions and lead to new discoveries.

We’re adding Peer Review to the Hackaday Prize. It’s a new way to apply your skills for the benefit of all. The current challenge is Citizen Scientist; calling for projects that help make scientific research more widely available. A set of independent eyes giving constructive feedback to these entries can be a huge end run to success. After all, you don’t know what you don’t know. Having help recognizing stumbling points, or just receiving a second opinion that you’re on the right track makes a big difference when treading in unknown territory.

Becoming a Peer Reviewer is simple. Pick a project you are interested in, review it thoroughly while making notes in a respectful, positive, and constructive way. When you’re ready, submit your Peer Review using this form. We will privately share your review with the project creator.

Hackaday.io is the most vibrant hardware collaboration platform in the world. Peer Review is yet another interesting way to get more brilliant minds in our community involved in building something that matters.

10 thoughts on “Become A Peer Reviewer For Citizen Scientist”

Wow. Mixed feelings about this. On the one hand, it might encourage reviewers to offer opinions that they might otherwise not share publicly. On the other hand, it might encourage reviewers to offer opinions that they might otherwise not share publicly :-)

Having fled academia several times in my life and having seen the politics of the peer review process there, I’ll approach this with caution. In that environment, publication can be entirely suppressed by reviewers; it sounds like you envision a feedback-only system, which seems more suited to this community. However, I’d hate to see some brilliant idea get discouraged early on by “experts” who don’t recognize the potential. Brilliant ideas often sound crazy when you first hear them (as do truly crazy ones), especially if you’re mired in the dogma of the field. We’ve all read some of the negative comments that get posted here anonymously – having this delivered through the HaD editors might lend a sense of legitimacy to this nonsense. I hope some filter is planned.

That being said, this could be a positive thing, if it encourages legitimate feedback and reviewees understand that it’s a self-appointed expert offering the review.

A lot of times a hacker is trying something that simply won’t work, and this is an opportunity for people to give their expertise and save the person some work.

If I can take an example, one project from last year wanted to bust open a CRT and use the stem as an ion-implantation device. A simple back-of-the-envelope calculation can show that this won’t work – the “mean free path” in the vacuum is the distance an ion would travel before hitting a leftover molecule.

It’s a simple calculation that showed that his proposed vacuum system wasn’t good enough, and that his system wouldn’t work in his proposed configuration.

His idea was interesting and valid, but he needed someone with outside experience (vacuum engineering) to show him what it would take to make it work.

As the saying goes, four hours in the lab will save you an hour in the library…

Then there’s the “that’s simple, just do *this*” answers where the reviewer hasn’t really thought through or tried out anything, they’re just using common sense to make helpful suggestions.

To take another example from my own project (lasercut-optics-bench), a HAD editor suggested using ball-plunger screws instead of regular screws. These have rounded tips which would seem (at first glance) to be better suited to rotation. It turns out that ball-plunger screws are inappropriate for a variety of reasons the author hadn’t thought of.

I hadn’t thought of them either.

The value of his well-meaning suggestions is that now I have a response to the sorts of objections people could make. I could make a stronger kickstarter presentation by accounting for these objections ahead of time in the description.

Having a list of answers to peoples’ suggestions is enormously valuable to a project. It forces you to validate your design decisions and keeps you honest.

Also, it might highlight something you’ve missed.

I’m all for the peer review concept. We’ve got people with expertise in all sorts of things, but no one is an expert in everything.

I disagree, if insensitive comments don’t push you harder, you weren’t in it for the right reasons in the first place. That being said constructive criticism is worth a million praises and I welcome it. People sometimes get offended by projects when they see them regardless if they are titled in a way to do just that or if they are meant to test a hypothesis.

Problematically commentators don’t see the hundreds of failures it takes to get anywhere, and the legitimate science that happens behind the scenes of “here is the one successful thing I built.” I simply don’t want to take the time to record every last dead end and failure or variable that produced no result and the simple fact is that peer reviewed papers never do this either.

To that end it seems rather silly to ask a single person with minimal funding to collate as much data and present it in a way that whole teams of researchers at a university with thousands in funding, SEMicroscopy, AFMicroscopy, GCMS, raman spectroscopy, ICP OES, Cyclic Voltammograms, Logging software, clean rooms, vacuum/gas chambers, ad infinitum, or for that matter 99.99% purity chemicals purchased from sigma aldritch.

The beauty of the HAD prize should be that things can be done simply without all the glorified pomp and circumstance. Regardless, of whether or not every last material is analyzed and scrutinized. If the prize seeks to do anything else then I believe it betrays it’s every-man origins. This is what commentators need to keep in mind, that people are doing the best with what they can.

Peer Review should not be negative feedback, it should be constructive feedback. I’m certain there are all kinds of people with experience and skills that would benefit a project. This is their chance to contribute help without taking the full plunge and joining the team for the the entire project. Over the long term I hope it will connect the right people so that as projects progress there is a wider network to ask advice when needed.

It’s a very cool new addition and I can’t wait to see what comes of it!