Scientific Irreproducibility

When I entered graduate school, I was excited to actually start working on a project that I felt was not only meaningful but also aligned with my long-term career interests. However, as I’ve gotten deeper into research -- performing experiments and familiarizing myself with the literature, my excitement has been dampened by the finding that I fail to reproduce many published experiments. Maybe I’m a terrible scientist, but I rarely believe the entirety of a scientific research paper published in a top tier journal; the story and results reported are always too perfect to be true. And I’m not the only one noticing this trend. For instance, to give you an idea of how bad it can get, back in 2012 Nature published a comment addressing reproducibility in cancer research.

“Over the past decade, before pursuing a particular line of research, scientists ( including C.G.B.) in the haematology and oncology department at the biotechnology firm Amgen in Thousand Oaks, California, tried to confirm published findings related to that work. Fifty-three papers were deemed 'landmark' studies (see 'Reproducibility of research findings'). It was acknowledged from the outset that some of the data might not hold up, because papers were deliberately selected that described something completely new, such as fresh approaches to targeting cancers or alternative clinical uses for existing therapeutics. Nevertheless, scientific findings were confirmed in only 6 (11%) cases. Even knowing the limitations of preclinical research, this was a shocking result.” (1)

A similar project at Bayer found that when they attempted to replicate 67 studies (mainly) in the oncology field , only 25% were replicable (2).

Results of Bayer's attempts to replicate published studies (2).

Of course, novel drug discovery in a disease of such variability like cancer may not be the fairest comparison, but nevertheless, there have been tons of editorials in the past years on citing possible causes of irreproducibility – from raising statistical standards (3), to bias, fraud, study design (4, 5), and failure to publish reproducibility attempts. The most comprehensive analysis of scientific studies found that under most study designs and settings, it is more likely for a research claim to be false than true (6). Other studies (7) also point to the concept of “p-hacking” which essentially involves testing your data in as many ways as possible in order to yield a significant result – a technique that unfortunately passes by because the majority of scientists don’t have a full understanding of performing statistical testing properly, which statisticians have cited for years (8, 9). But all hope is not lost. Due to the sheer amount of publications that are coming out, we can use strategies to perform meta-analyses that can give a more reliable, clear picture as to what to make of findings. As Head et al. found, these types of studies seem to be immune from p-hacking. I think that “Watson-esque” data mining of publications will be a valuable, important, and necessary part of science as we move into the future.

Prominent statisticians' criticisms over the past few decades (source unknown).

My own personal opinion is that the replication problem is likely a combination of all previously mentioned factors, driven in part by the lack of true hypothesis-driven science in many studies bound with the constantly decreasing supply of federally funded money. As you can see in the chart below, the NIH budget has been steadily decreasing over 20% since 2003 when accounting for inflation, while there has been an 11% increase in funding applications. Likewise, many graduate students and post-doctoral students who end up performing research in academia is shrinking accordingly (10).

This combination of decreased funding and increased competition results in labs that are struggling for money. The effect of which is increased pressure on graduate students and post doctoral researchers – the ones who are performing the research – to be incentivizedto publish, as their lab and their careers largely depend on published results. This creates an environment where the ones carrying out their research are looking for results that fit their own story of expected results in an attempt to publish anything regardless of what the actual data show. The classic “publish or perish” mentality. I personally believe that much of the data that I see in papers are the result of multiple experimental attempts with cherry picked results to fit a story and a complete disregard of results that show variability, inconclusiveness, or data to support completely different alternative explanations. Harvard professor Bjorn Olsen cites a similar mentality as he describes his experience with spending a year trying to replicate an experiment in a top journal,

“At wits’ end, he contacted the scientist who had led the research to ask what he was doing wrong.

As Olsen recalled, the scientist seemed surprised to hear from him and said that the technique was very difficult and that he would not recommend that people do it. It became clear to Olsen that it was something that the original research team had tried and failed to do many times, and then had gone and published a paper based on the few times it worked.” (12).

This environment breeds flawed science. But all of this has likely been said before and isn’t too new. So what can we do to fix this aside from sitting here bitching about it?

The Public’s Perception of Science is Damaged

If the majority of scientific findings are wrong, then it comes as no surprise as to why the public’s perception of science is damaged. The problem is that while there is an increasing amount of science funding from philanthropic efforts (aka a super rich guy or his family member got sick), the majority of research expenditures in academia come from federal money (14, 15). And where does that money come from? The public’s tax dollars, which makes the public an important stakeholder in research. Therefore, if the general public starts to find out more about the problems of false findings in science, then there could be even more severe consequences to the NIH’s already decreasing budget. Additionally, there is an intimate link between publically funded academic research and the eventual drugs and treatments that we have available to us. A large portion of research funding comes from corporations (via R&D programs) and corporations often benefit from the fact that preliminary evidence and new discoveries are most often made in academia, allowing them to focus on developing drugs based on these findings. If these corporations can’t replicate or trust the findings in academia, then they are wasting literally billions of dollars developing drugs based on findings that aren’t true. This, in effect, could lead to increasing amounts of money going toward private R&D, which in turn leads to increased costs of the drugs for you and I.

Where money in research comes from (15).

It’s somewhat paradoxical that the public hears about science in the media not from those who are at the top of their field or from those performing the research, but rather from “science journalists” who rarely have the credentials to report on the findings (there are of course some legitimate science reporting outlets). If you read or listen to media reports on new discoveries in science at all, you’ve encountered the sensational headlines and skewed interpretations of the actual result (13). One may argue that these are attempts to generate funding from outside entities and philanthropic organizations that are looking for particularly successful labs to fund. But the truth is that federal money still dominates the majority of research expenditures in academia (14, 15) and unfortunately, there is no section to cite CNN’s coverage of our recent discoveries on our federal grant applications.

It may be hard to pick up on the errors or misconceptions reported in media or forum comments if you’re not an expert in a specific topic, but the problems are there. This often leads me into wondering if I can find so many mistakes in reporting of topics within my field, what does that say about the information I am taking in when I read about other fields that I’m not as familiar with?

Reddit is the world’s largest internet forum (and I’ll have more on this later), clocking in at #23 in the world’s most popular websites and #10 in the United States (alexa.com). I have moderator status on Reddit’s /r/science and /r/askscience subreddits which enables me to see all the deleted comments in a thread and thus witness some of the problems in bringing science to the public first hand. Without a doubt, in every single thread that gets some amount of traffic, there will be several comments along the lines of “so Reddit, tell me what’s wrong with this headline,” “how many times do we have to cure cancer/AIDS/etc before it actually matters,” “who cares about results in a mouse or a cell?” The layperson’s perception of science is sad, really, and it is largely to do with the complete disconnect of actual scientists from the public. Here’s a chart to back up my own anecdotal evidence (16). While the public is blinded to the issues that were previously discussed in the first section, I do believe that the two problems are interrelated. So what can we do to fix this aside from sitting here bitching about it? I swear, I’m getting to it.

Science Needs to Utilize the Internet

In 2014, two huge break through stem cell papers, known collectively as Stimulus-Triggered Acquisition of Pluripotency (STAP), were published in Nature (17). The papers essentially concluded that through mechanical aggravation or submersion in a weak acid, adult somatic cells can be reverted to a pluripotent state. Not only this…they were totipotent too! Of course, this would have been a landmark finding, as Yamanaka’s discovery of iPSC reprogramming had already earned him a Nobel Prize no later than 7 years after his initial report. Luckily for the science world, this study was perfect in putting the power of fast, free communication of the internet to the test.

Not shortly after the study was published, several labs (including mine), quickly went to replicate the findings. The methodology in the paper said that in as little as 7 days, you would be able to see Oct4 (a pluripotency marker) expression – a blip of time in the grand scheme of research. Now normally what would happen is that all of these labs would be working independently on replication of the data. They wouldn’t be in communication with other labs at all as there would be no way of knowing who is trying to perform the experiments. One would assume that other labs are doing it, but as soon as you fail to replicate it, you would immediately assume that you did something wrong. So you try again. And again, and again, until you eventually give up. And then you may eventually hear later at a conference or in a publication the findings from other labs – the same results: a failure to replicate. And now we are left with the realization that not only you, but hundreds of others likely wasted incredible amounts of time, money, and effort into attempts at replication. Luckily in the case of STAP, the experiment was on such a short time frame and required such simple reagents, that the impact wasn’t felt. But this is rarely the case, and right now there are thousands of hours and dollars being wasted in attempts to replicate data that likely cannot even be replicated for the reasons cited in part 1. Even worse, the failed attempts won’t be reported.

On a brighter note, the STAP issue was covered by a scientist who actually figured out how to use the internet. In Dr. Paul Knoepfler’s blog at ipscell.com, he quickly wrote about the STAP paper and his rightfully raised eyebrows at the findings. But more importantly, he offered an invitation to crowd source the STAP stem cell replication attempts (18). Wow!!! Someone has figured out that we have this amazing thing called the internet where data packets can be sent and viewed for free. Just imagine the implications!! On a more serious note, people began posting their results, and it quickly became theorized that the data was complete crap, just days after the initial publication (19). Some labs went the extra mile to prove the original reports were wrong, and a similar, more in-depth crowd-sourced version for sharing results appeared on Research Gate led by Dr. Ken Lee (20). Nature eventually retracted the papers a few months later but not before one of the paper’s senior authors killed himself under the pressure (17). This situation makes for the perfect example of what we can do when we as scientists communicate properly. However, in order to do so, there needs to exist a viable model system in place.

The Current Models

But Elliot, haven’t you heard of PubPeer? Yes, I have, and it’s a system that is structurally set up for failure. For those that don’t know, PubPeer is a website where people can anonymously comment on a published paper. It’s usually utilized to call people out on their data that may be troublesome or down-right fabricated. You can check out some of the most recent interesting discussions about Jacob Hanna, a prominent stem cell biologist, here (21). Also worth mentioning here is Research Gate’s “Open Review,” which is a commendable attempt to encourage scientists to critique research findings in a transparent fashion, as is the central point of what I will discuss later (22).

PubPeer is on the right track by utilizing the internet to create discussion on research findings, but it’s bound for failure. The main reason: lack of incentives. If you’re a scientist, the likelihood that you’ve heard of or read anything on PubPeer is probably way less than the percentage of papers published that are never cited (a whopping 50%, by the way, 4). And why should you know about it? The userbase right now is so small that it is rare that you will encounter a paper in your tiny research niche that actually has comments. Even if you do, the comments are completely anonymous, which means there is again a huge disconnect in proper communication and dialogue. (Read PubPeer’s take on anonymity here, 23). This means that the comments could be motivated for reasons anywhere from purely altruistic purposes (e.g. those that are trying to identify poor science for the good of the community) to vengeance (e.g. I dislike this scientist, so I’m going to try to discredit him/her). When you’re anonymous on the internet, you can write whatever you want and be unaccounted for, and anyone who goes on to read the comments of Yahoo, YouTube, or other news outlet sources knows the downsides of that. With that said, anonymity on Pubpeer may be short lived, as a judge has ruled against protecting the identity of an anonymous Pubpeer commenter. As one of the Pubpeer moderators commented, ““If this decision creates a precedent, future online comment will be limited to trivial matters, while all discussion of serious matters will be discouraged by fear of legal entanglement.” (24, 70). Anonymity is simply not desirable in the realm of serious scientific matters.

But as I mentioned previously, the lack of incentive structure likely explains the site’s poor adoption. Why should I put in the time and effort to research and write a comment? After all, it often takes a high degree of analysis to find errors in published papers, which can even be carried across publications over many years. What am I really getting out of it? Additionally, another failure is the lack of the public’s involvement. It would be really great if the concerns and questions of both fellow scientists and the average Joe could be addressed under one umbrella – a discussion that doesn’t have to include just criticisms (more on these thoughts later). So let’s examine a model that incorporates the public.

For this, I again turn to Reddit, not only for its massive user base but also because it truly is a great source for reading and discussing interesting articles when you’re not wasting time looking at cat pictures. Reddit’s /r/science subreddit is the most applicable to this discussion (26). /r/science is a default subreddit, which means it can be seen and accessed directly by anyone who is just visiting the site’s main homepage. With over 8 million subscribers and millions of page views per month, it has a lot of posts and thus needs a lot of moderation to keep it functional and informative. In its rules on comments and submissions, it’s requested that only links to recent, peer-reviewed scientific articles are discussed, with off top-topic, medical advice, and personal anecdotes being strictly prohibited from discussion (27). In this sense, the forum aims to keep discussion on topic and relevant to the article at hand. With that being said, the majority of threads still end up cluttered with jokes or irrelevant discussions that are heavily moderated and deleted in an attempt to keep the discussion on topic. The reasons this happens, in my view, are that there simply aren’t enough educated scientists that are making posts to educate and stimulate discussion in combination with the lack of structure to fend off comments that don’t adhere to the rules (i.e. there is no punishment or discouragement) which we can otherwise refer to as “trolling” on forums or perhaps more akin to spam for e-mail. (more on this later).

What the /r/science subreddit does do well is give “flair” to those who are verified as having a specialty in a certain field, such that their posts will indicate “Graduate Student, Biophysics” or “PhD, Genetics, Human Disease,” etc, perhaps giving more credence to their posts which tends to generally work very well. By getting verified as a student or degree holder (you have to send the moderators proof such as a diploma, student ID, etc) you lose anonymity only to the moderators, which means you are still “pseudonymous” to those reading your posts. Additionally, the moderator team brings in actual scientists from various fields as part of a science “Ask Me Anything” series where researchers can answer questions about their field directly. With these two features, the /r/science forum does a pretty good job at providing readers of the general public with useful information about articles or fields of research that they may be interested in and forming a useful communication bridge between scientists and the general public. But in order for this method to be even more effective, there has to be a way to bring more scientists into the discussion to bridge the gap between science and the public.

A New Model Proposal

As any economist will tell you, proper incentive structure is a necessity for any successful venture. Therefore, in order to bring more actual scientists into the age of communication via the internet, there has to be proper incentives in place to do so. And the best incentive of all? You guessed it – money (sex over the internet isn’t quite there yet..).

There are many intelligent and educated people that browse Reddit, but of that proportion there are few that consistently make well researched, informative posts. The reason for this I think is twofold: because it takes time and effort, which many of us just don’t have enough of, and that the return on our investment (i.e. time spent browsing) just simply isn’t worth it. Why should any scientist be going there to browse in the first place? Personally, I try to make posts in threads that I have expertise on ( neuroscience and stem cell biology) in order to “set the record straight,” answer questions, clear up confusion, or simply educate those in a topic I feel strongly about. With that being said, there really is no true return for me for doing so aside from perhaps doing a bit of research to remain sharp on a topic that I may have forgotten and the personal satisfaction that is inherently rewarded upon contributing something for good intent. Oh, and karma.

For those unfamiliar, Reddit uses an upvote/downvote system where users can upvote posts that they find informative or funny, and downvote those which are irrelevant or misinformative, thus creating a system where the best posts are voted to the top and seen by the most users. The upvote/downvote ratio that a user’s posts receive are kept track through the accumulation of karma, which essentially amount to “useless internet points” that can be seen attached to your username and post history. Really, they are entirely useless (28). In fact, this feature is really one of the only things that has catapulted Reddit’s relatively simple interface to its soaring popularity versus other web forums. Karma acts as Reddit’s incentive system, and it’s truly scary how well it has worked. Essentially, just knowing that other people share, support, or appreciate your opinion is enough for people to come back and provide amazing content. It’s the basis of the site’s community which wouldn’t function without it.

Enter Cryptocurrency

One of the inspirations for writing this particular post was my excitement when Reddit announced that they had hired a cryptocurrency engineer with the intention of utilizing Bitcoin’s blockchain and bitcoin itself as an asset. (29, 30). To me, this meant that Reddit was looking to take charge in integration of Bitcoin’s ability for microtransactions online. Essentially, instead of useless karma contributing to voting structure, karma could actually now be pegged to a digital asset like Bitcoin thus creating a monetary incentive structure within the existing system. So how could this work?

Let’s say I am commenting on an article posted to /r/science. Since I have particular expertise, I am able to lay out the findings of the paper in an easily understood manner in addition to putting the paper in context and offering my own critique. In a normal situation, I would be rewarded for my work by hopefully being voted to the top of the thread or perhaps receiving a gift of reddit gold, which is an independent “rewards” system that enables extra website features and helps to generate revenue outside of advertisements to fund reddit’s servers but in reality cannot be redeemed for actual money. By stimulating discussion and continuing to answer other users’ questions, I earn more upvotes (karma) and perhaps build a reputation as a reliable user on the website.

As previously discussed, this model is currently limited due to the lack of incentives that are strong enough to maintain the integrity of a subreddit that requires higher-level posts, but clearly does work for subreddits where posts do not need to be as thorough. So we can now imagine a system where each user has a Bitcoin wallet, as may have been in the works before Reddit’s cryptocurrency project was disbanded following the resignation of their pro-Bitcoin CEO, Yishan, in combination with their planned crypto-equity system that likely had huge legal ramifications (32). Users would then be able to fund their wallets, which would be connected to their accounts on the website, or perhaps even separately through a browser extension (33) or exterior account like Coinbase. The wallet can then be used to send bitcoin to other users based on upvotes – essentially a tip for providing good content or information. Let’s now imagine a scenario where one upvote that I give transfers a very small amount of bitcoin to the other up-voted user, perhaps 1 penny worth of bitcoin (.00004 BTC), which we can call a “micro-tip.”

What makes this possible is that because of Bitcoin’s underlying structure, it enables the possibility of microtransactions online which have always been otherwise impossible due to overhead fees issued by credit card companies. The current system makes sending anything under $1.00 over the internet essentially impossible due to base fees ranging from $0.15-$0.20 in addition to a 2-3% processing fee (34). These surcharges employed by VISA, Mastercard, etc, are done to cover the massive overhead costs in managing credit card fraud, risk management, etc. Importantly, this eliminates business models that incorporate microtransactions because the fees are over 20% of the entire purchase!

If you understand how Bitcoin works, you know that typical fees broadcast over the network are set based on size (data, not monetary value), and will fluctuate based on current market rate, but have typically been between $.02 and $.04 (essentially 0.0001 BTC). This means a fee of 3 cents can be appended to a transaction of $1.00 or $10,000,000 as the network is apathetic to the amount of value being sent. Importantly, you may notice that a 3 cent fee for a $1.00 transaction is in line with the current fee rate of credit card companies, so we are again stuck in an unwanted position. If we wanted to send $0.01 for an upvote on Reddit, we would have to pay a 3 cent fee. Clearly, this wouldn’t work.

Luckily for us, Bitcoin is versatile and can perform what’s known as off-chain transactions. Instead of the transaction being broadcast over the normal Bitcoin blockchain, it essentially occurs directly between the two users (35). In order for the users to “cash out,” they may need a certain amount of bitcoin such that the fee is significantly lower than the amount being sent such that it makes sense to perform the transaction. They can then send this amount of bitcoin by broadcasting the transaction over the Bitcoin blockchain into a wallet that they control themselves, or perhaps to make a payment or purchase on another website. On-chain fees can be paid by the forum itself, supplemented by paywall fees (discussed below) or alternatively by the user if they so choose.

An important note on off-chain transactions is that because value is being transferred internally, the users must trust the third party (in this case, the forum operators) to honestly maintain the balances of accounts within the ledger. In the blog post cited, I can move Bitcoin between my Coinbase wallet addresses in a free, off-chain manner because I trust them to maintain the balances within my wallet addresses. This is essentially how banks and other centralized services (PayPal) also work. In our case, we resort to off-chain transactions because we are operating with microtransactions which cannot be performed in a monetarily-logical way with the normal fees required by the Bitcoin network. Instead, they are sent and transferred instantly and without fees between user’s wallets, however we sacrifice the security of the decentralized Bitcoin blockchain for a trusted third party, which some will argue means we aren’t actually using Bitcoin when transacting in this manner. Importantly, however, users can move their bitcoin out of their account-associated wallets by sending on-chain transactions in a decentralized manner.

This type of off-chain strategy is also utilized by the most popular Bitcoin tipping service, Changetip (71). Changetip links social media accounts like Reddit, Twitter, Github, YouTube, Google+, etc, to their service, which allows users to directly tip other users specific bitcoin amounts when specifying the service (for instance mentioning @changetip on Twitter signals the service). Changetip is an excellent service which runs as an additional layer on top of a pre-existing system. In our system, users have Bitcoin wallets associated with their accounts from the get-go. Despite this, the concept is the same.

An example of a Changetip transaction over Twitter.

It should also be noted that there are several players in the space working to solve this microtransaction problem and capitalize on the market potential for this business model. One of these companies, called Blockcypher, aims to allow microtransactions (between 0.5 cents to $9) to occur on the Bitcoin blockchain by developing a system to calculate fees adaptively such that they can guarantee the transaction will be picked up by the miners (36, 37). If you’re unfamiliar with Bitcoin transaction fees, essentially the fees exist as incentive for miners to include the transaction within a block. While you can send transactions without a fee attached, you also run the risk of your transaction sitting unconfirmed, essentially in “transaction purgatory,” which is more likely to happen with low value amounts (38). So, solving this problem is important. However it also raises an issue of “blockchain bloat” where block sizes (in terms of data) are restricted by the current protocol. Therefore, users worry that by essentially “spamming” the blockchain with high amounts of microtransactions will cause blocks to “fill up,” causing a traffic jam of longer confirmation times for transactions as they await to be included in a block. This is a separate issue that is outside the scope of this post, but there are plans in place to increase block size limits as the network grows (39, 40). The newly proposed “Lightning Network” may also offer a solution by allowing for instantaneous transactions and resolution of scalability issues (41), although I defer to experts who have a greater understanding of this proposal. These issues are relevant however as we determine whether to build a forum based on off-chain or on-chain transactions. For more on off-chain transactions, refer to these sources (42, 43).

With this structure in place, we are now left with a scenario with a monetary incentive structure that rewards users for providing valuable content to the ecosystem. Instead of my post receiving 500 karma points, I actually made $5.00 (assuming 1 upvote = 1 cent). I can then go transfer that money over the Bitcoin blockchain for a small fee and use it to spend on the internet or cash out via a USD to Bitcoin exchange.

Implications, Structure, and Solutions to Potential Problems

]1.Bitcoin is global. Because Bitcoin is a global payment system, users in Russia, Brazil, South Africa, or anywhere on Earth can easily (it’s getting easier, guys) buy bitcoin in order to fund their user wallets and use/earn bitcoin just as someone in America would (we assume here that there are effective on/off ramps for local currency to BTC established in these locations). No one is monetarily excluded from participating in the system, which is important for a global science conversation (linguistic limitations still exist).

2.Microtransactions enable incentive structure. The ability for off-blockchain microtransactions enables transfers of value so low such that users are not discouraged from participating. In this case, the users are paying to create an ecosystem that contains content and information that is more valuable than the one before, but the payment to do so is so low that it would not be inherently inhibitory to participation(see early e-mail spam strategies discussed below). Additionally, they are also monetarily incentivized to earn back their payments by contributing additional valuable content, thus creating a positive feedback loop.

Aside from just discussing articles akin to Reddit’s /r/science forum, scientists will also have separate subforums where they will have the ability to discuss and critique research papers similar to Pubpeer’s system, post replication data publicly, and discuss techniques similarl to ResearchGate’s forum. Having a problem replicating an experiment? Post your results in the “replication” subforum. Discuss what you did and how your results differed. What could have went wrong? Receive feedback from others – perhaps you’re not alone in your attempts. Having trouble getting your PCR to work? Receive feedback from experts in that technique. Learn new tips and tricks for your experiments. Improve your experimental design and statistical power. Have a critique of a particular study? Post it in the “critiques” subforum. You get the idea.

It’s true that these discussion forums for these types of problems exist (for the record, there is a huge list of websites that try to accomplish this task, 44). The problem is that they’re not incentivized properly, plain and simple. Cryptocurrency enables this type of business model where it previously could not exist. If I want to learn more about what MOI my lentiviral experiment should use, I’ll type “help lentiviral MOI” into Google. I get back some protocols, some publications, and a discussion on ResearchGate. ResearchGate is great for advice (does anyone know why practically all the users are outside the U.S.?), but under an incentivized system with a Reddit-style structure could be vastly improved upon. If you disagree with this, then you likely disagree with the entire basis of this write-up. Additionally, a Reddit-style structure enables both the public and scientists to function under the same umbrella. In this scenario, we aren’t excluding the public from participating in scientific discussions, we encourage it. This is not to say that there could be restricted access to certain sub-forums where input from only scientists is desired.

3.Downvotes. How do we handle downvotes? Because we cannot allow users to subtract money from other users’ accounts, downvotes would likely not play a role in this format. Therefore, the way Reddit’s current voting system works would have to be re-worked for our forum. Alternatively, downvotes can serve as flags to signal to moderators if a particular user is trolling or spamming (discussed further below).

4.Paywalls. How do we discourage users who are viewing content for free? Since one of the main goals of a system such as this would be to encourage the discussion between scientists and the general public, it is likely possible that many members of the general public would have nothing of value to actually contribute to the discussion, as it would be foreign to them. But we still want these users to be able to read the discussion in order to learn. Therefore, these users would be getting a “free pass” to the high quality content of the forum that others are using their own money to produce. In order to “even the playing field,” one could implement a fee in order to view the actual discussion page.

How would this work? For instance, if I want to view the discussion surrounding a newly published popular paper, I would be prompted with a paywall in order to access that page. Scientists are all too familiar with paywalls, as those who attempt to access primary journal articles outside of a paid University connection are prompted with absurd article paywalls that frequently cost over $25 (42). But now that Bitcoin exists, paywalls do not need to cost this much. How about instead of a $25 paywall to access a webpage or article, we now charge $0.25, $0.10, or $0.05. Now, in order to view the discussion, the user is charged a small fee in order to access the content. This value can algorithmically fluctuate based on the discussion’s popularity such that a newly posted discussion would be free to access and as content accumulates, the fee to access that content adjusts accordingly. A maximum value or decreasing sliding value scale over time could be issued for pages accessed past a certain time-point, such that users viewing a post from months ago may still need to pay to access the content, but a value less than when the topic was “hot.” In case you’re curious, these Bitcoin paywall systems already exist (43).

A problem arises when we consider the annoyance to browsing if we’re being prompted with a paywall every time we click on a new thread. Some possible adjustments could be made. Users could just select an “autopay” option such that they avoid prompts when browsing. Let’s say a user is interested in reading the answers to a thread concerning a question on a lab technique. They autopay to enter the thread only to find some answers that are lacking. This leaves the user frustrated, feeling like they just wasted money (even if it is a low amount) for no reason. To curtail this problem, we can additionally implement a thread rating system, where the people within that thread can rate on certain features of its utility which can be viewed from users outside of the thread (see example). While Reddit’s upvoting algorithm for submission also serves this purpose in addition to being a time-dependent mechanism (threads are recycled based on time, 44), we add an additional layer of options to improve user experience. Something that allows a quick preview of the thread comments could also work as well.

A possible added layer of ratings for threads

Additionally, we could implement weekly or monthly subscriptions for access to all the threads in a certain subsection of the forum for the week or month. Many news outlets have tried to implement paywalls or subscriptions for internet users with little success, mainly because the paywalls don’t allow for microtransactions (although they could if they used Bitcoin) and quite simply because the information within them is easily copied and accessible elsewhere on a website like Reddit. Because our system is built on microtransactions and Bitcoin, we allow for subscriptions that can be more realistically priced. Maybe I would choose to subscribe to the “neuroscience,” “stem cell,” and “experimental help” sub-forums, paying a monthly fee of $2 each, for instance.

5.Fee revenue. Where or to whom does that fee money go? In an ideal scenario, we may imagine that some of this money generated from viewing content would go to subsidize partnerships with publishing companies where an article being discussed is published. This brings up the possibility for complete disruption of the business model of science publishing, which is in desperate need for change. The top 3 publishers, Elsevier, Springer, and Wiley make up roughly 42% of all articles published, with price increases of over 600% in a 20 year period, catapulting science publishing into a multi-billion dollar for-profit operation (45). Elsevier alone operates at 36% profit margins. While most of these profits come from University subscriptions (46), the fact that these publishers charge upwards of $25 to view a single article is laughable and all too familiar to other corporate giants that dominate our everyday services in America. It surely isn’t helping the general public’s access to science.

Instead of obscene pay walls, our fee money generated during the discussion would subsidize access to the primary research article for everyone. I could not find numbers on percentage of revenue generated from online paywalls, but I can’t seriously imagine that people are paying those prices. Subsidizing partnerships with publishing companies would hopefully make up a portion of this revenue, provide public access to primary research articles, and allow for a bridge in the discussion of science between the public and actual scientists.

This model may be possible because the days of these paywalls and overpriced subscriptions are likely limited due to the trend toward open access. Despite this, it still remains true that publishing is definitely not free (47, 48). You see those gray boxes in the chart below? The ones that show the additional cost of publishing if the reviewers were actually paid? Ah yes, perhaps subsidizing the peer review process may actually help to alleviate some of the problems initially discussed. Maybe hiring a statistician to review the statistics used in papers? An incentive to thoroughly review the work in front of you could go a long way.

A graph predicting the exponential increase in Open Access articles (25).

In fact, this is being done now (49). The problem is that the company in question, Research Square, guarantees a review within 3 weeks using a scorecard and pending the payment of $750 to Nature Publishing Group. In this model, the incentive is shifted to get the review out quickly, rather than provide more thorough review! I see this model resulting only in profits for the company at the expense of more adequate reviews.

So we may now imagine a scenario where the cumulative effect of small microtransactions subsidized by viewership may be able to offset the paywall problem while open-access publishing continues to gain ground. Eventually, these payments can be transitioned into subsidizing the peer-review and editing processes, which would hopefully improve the overall quality and assessment of papers that get published. In this scenario, a free-market system evolves to fund the journals that publish the best work, although the established giants in the field (Cell, Nature, Science) would have a clear upper-hand. Perhaps this scenario would not work out as easily as I am talking about now, since the leaders in high quality content are already well established. Or perhaps this would help to break down the broken system of impact scores, as Nobel laureate Randy Schekman suggests (50). Interestingly, this forum would also open up the idea of scientists being able to self-publish their ideas to a broader audience, aside from just starting a blog like this.

It’s also possible that the fee revenue can be used to actually offset the cost of publishing from the lab that produced its work. After all, the lab has to pay hundreds or thousands of dollars just to publish. Now, instead of subsidizing the journal directly, we can actually subsidize the lab that produced the work. Did your lab have a cool paper in a lesser known journal that was worth discussing? If the readers of the paper are intrigued by the work, or the study is picked up by the media, they will generate viewership and discussion, thus subsidizing the costs to actually publish. A nice kickback for conducting an interesting or important study. How’s that for “donate to fund cancer research?” (hint: most of your money isn’t actually going to fund cancer research, which is an entirely different issue that I won’t go into, 51). Alternatively, and perhaps the most realistic option, these fees may be used to pay forum moderators (discussed below). Could this work? I don’t know…it’s important to keep in mind these are simply ideas.

6.How to handle trolls, spam, shills. Couldn’t a corporate entity still pay off shills to make comments on posts in favor of their own business or agenda? Indeed, the use of money in politics and media is well known for pushing agendas and the idea that many of the comments that you read online are actually made by people that are paid to do so has been floating around for quite some time. While I don’t believe this problem can be completely avoided, there are ways to curtail its effectiveness.

In the early days of e-mail, spam mail was a huge problem. The reason for this was simple: e-mails are free to send. Before the advent of Google’s powerful spam filters, many industry leaders such as Bill Gates proposed imposing a small payment in order to send e-mail. While the average user wouldn’t mind paying a penny to send an e-mail, a spam mailer would certainly have trouble dishing out the money to spam millions of addresses at a time. But as discussed in this article (52), this solution isn’t cut and dry. Interestingly, one of the payment ideas was instead of money, usage of computing time to solve a problem could be used before sending e-mails. Sound familiar? This was possible at the time via Adam Back’s Hashcash system, which is the basis for Bitcoin’s proof-of-work scheme (see video, 53).

While spam filters eventually eroded away the necessity for an e-mail payment system, the same principle applies in our situation. By imposing a fee to enter and access a forum thread’s content, a corporate entity may be dissuaded from paying these fees in addition to the person making the comments. In this way, we monetarily discourage spamming, trolling, or shilling.

Perhaps the best way to actually reduce this is through effective moderation. In all successful forums, a reliable and effective moderation team is key. Moderators of a science forum would be real scientists that would moderate a sub-forum of their expertise. For instance, a neuroscientist would moderate a sub-forum that deals in neuroscience. Because of a moderator’s overall familiarity with the subject matter, it would be easy for him/her to spot replies in a thread that cite pseudoscience – an article like this (54), for instance, which may otherwise be up-voted in a less strictly moderated environment.

In order to get effective moderation, the team would need to be incentivized. As previously mentioned, the fees used to access the thread could be used as payment to the moderators for their work. Moderators can additionally comment themselves and be “tipped” by other users to earn additional payment and contribute to the thread. Because I have moderator status on Reddit, I know first hand how overrun the comments are with unnecessary jokes, anecdotes, pseudoscience, and incorrect information. Moderators on Reddit do a pretty good job keeping things clean (largely through the use of AutoModerator bots that recognize and delete posts with key words) for just doing it out of the kindness of their hearts. If moderator teams are monetarily incentivized, the results could be dramatically improved. I imagine a moderator position on this forum to take the role of an editor at a journal. The position would be subsidized by fees as an incentive and moderators could serve democratic “terms” with limits such that we don’t have a situation where moderators can sway the tone of the subreddit without consequence. If this is successful, it could even be a real job for those that care about science education and writing – a growing job market due to the stagnating availability of positions in academia.

Lastly, a huge factor in trolling and shilling on internet forums is the fact that the commenter is anonymous. As I have previously discussed, anonymity is not desirable in a serious forum setting. Certainly anyone who claims to be a scientist will have to be verified accordingly and post under their real name/affiliation – similar to Reddit’s flair system, but only with a user name reflective of your actual identity. But should all commenters be anonymous? This is a question that is hard to answer. On one hand, you sacrifice the privacy of your users by collecting and verifying their identifying information. Additionally, a “power user” that frequently makes good posts may actually make a decent amount of money and be more vulnerable to hacking attempts to steal identifying information or money in their linked wallets. In this day and age, privacy is virtually lost, but people are starting to care more. On the other hand, you make an environment that further discourages trolling and shilling. Perhaps a compromise is what many websites and social media applications use – connecting accounts through “verified” or linked accounts such as Facebook or LinkedIn. In this way, it is harder to generate hundreds of new accounts while also easy enough and less invasive than something like the Know Your Customer regulations that banks and businesses use (55).

I believe that the combination of these three factors will work synergistically to oppose trolling and shilling in the proposed forum environment.

7. Submissions. The model used for submissions on the Let’s Talk Bitcoin network (discussed below) is well structured and could be applied to our previous Bitcoin model as well. Essentially, unless you are an approved submitter with proper permissions, you will have to pay a fee to submit an article or press release. This helps reduce spam submissions and likely would increase quality submissions. Depending on how your submission does based on upvotes/likes by other users, the user that created the submission would earn back their fees and actually make a profit for their submission. We can structure the earnings based on the ratios or amounts of upvotes/likes that the submission earns.

This incentivizes users to add-on to their submissions such that they have a higher likelihood of stimulating discussion and being up-voted by other users to the front page. Let’s say I want to submit a press release from Harvard’s Stem Cell Institute on a new study. Instead of just posting a link to the press release, I attach a DOI number (56) linking the primary research article (hopefully accessible to all – discussed in 5) as well as a summary of the actual article, in my own words. This encourages users to submit articles that are in their field, such that they can give a summary and interpretation of the data presented within – kick-starting a discussion for users entering the thread. Here, we again rely on an incentive to exist for users to earn back their submission fees and profit from their submissions, which in turn filters out lower quality content for higher quality submissions. Another example would be asking for help with a technique. Rather than just asking, “I can’t get my Western to work – help!!” the user making the submission would explain what they have done so far, what protocols they are following, and what resources they have already used for troubleshooting.

8.Content Copying. Wouldn’t a paying user just be able to access the thread, take a screenshot, and post the image online for others to view for free? One of the downsides of a forum on the internet is that digital information is easy to copy. This has been a problem since the dawn of the internet, as the attempts to stop the downloading and sharing of digital music content ironically led to the developmental progression of Napster, Limewire, and eventual Bittorrent. Sharing of content on the internet is inevitable, but there are a few reasons why this issue may not be such a big deal, and it all relates back to the main point: incentives.

Because cryptocurrency enables microtransactions, we are now able to adjust payments to miniscule amounts, which not only enables the monetary incentive structure that I have described, but also disincentivizes stealing of content. The reason for this is simple: if it only costs $0.25 to view the content you want, is it really worth stealing? While I don’t discount the screenshot and posting scenario described, the ability to pay such small amounts with a single click of the mouse in order to access the content is so easy to do that the vast majority of users will do it. In an age where 7 second Vine videos can thrive due to the shortening attention span of humans on the internet, surely a single click and a $0.10 fee will be preferable to minutes searching Google for the copied content.

Another underrated, yet important aspect is that of donations and altruism. One of the best examples of this is podcasts, which have seen tremendous growth in listeners over the past few years (57).

I could not find any specific numbers, but as a podcast listener myself, many are subsidized not only through advertisements but also through small donations by the listeners. The point I attempt to make here is that people will pay for content they want and enjoy, especially when the cost is low. Services such as Kickstarter, Indiegogo, GoFundMe, and the Bitcoin-based Lighthouse I think count as further evidence for this. The existence and success of companies like Spotify also support this notion. I can easily torrent all the music that I want, but instead I pay for convenience, ease of use, and the belief that content creators should be subsidized.

Along the same lines, and perhaps most relevant to this topic is the aforementioned Reddit Gold (58). Reddit Gold, at its core, is an overpriced token that enables users access to some extra website features while simultaneously serving as a method to pay for Reddit’s servers. When you receive gold, they even tell you how much server time your gold has paid for. Therefore, while the extra features of Reddit Gold are relatively weak, the real reason people buy Reddit Gold is likely to support the website that they enjoy. In this sense, altruism is a real contributor to the website’s health.

Reddit gold facilitates the costs for server time

9.A Model for All. All of the above details are also applicable to Reddit’s /r/askscience forum as well and in reality could be extended to cover any forum where serious discussions are encouraged.

10. What else can we disrupt with this model? In science we communicate our ideas through publication, which I have mentioned is ripe for disruption using our model forum in tandem with the incoming age of open access publication. Additionally, the other major way we communicate is by attending conferences, where scientists in the same field travel to share data, ideas, and awkward social interactions. When we take a step back, the entire idea of a conference in the year 2015 is akin to having to pay your rent via check. Why are we doing this?

This past year at the Society for Neuroscience Conference, I (my lab/travel award/graduate division) paid $105 for an abstract submission, $65 for student membership, and $170 for registration for the conference. This excludes costs like airfare and hotel. Now, consider that this event is attended by a global cohort of scientists 30,000 strong and you’re talking money in the millions to organize the event. Let’s consider what we do at conferences (speaking as a student). I walk around the poster sessions, talk to presenters that are perhaps working on similar projects, interpret new data, attend talks, and socialize or catch up with other friends in the area. A few days of hardcore science. The experience is actually quite worthwhile as it allows for the exchange of ideas, feedback from other scientists, etc. But do I really need to travel across the country to do this?

What if we could hold an entire conference online via our forum model? First, the overhead costs of putting on the event drop dramatically. Poster sessions would be held online with the poster presenter present in a separate forum room at a specific time, just as they would be present during their session at a specific time. Users can tune in by streaming a webinar of the presenter showing and talking through his/her poster on their computer. The presenter can answer questions as they come in a live-feed format. This may even work in a similar fashion to Reddit’s AMA series. Short lectures and talks would be hosted in the same way. Users on the forum can easily jump from talk to talk or poster to poster within sessions.

Fees would be collected and implemented on a “per-session” basis. For instance, if I wanted to attend the “iPSC Models of Neurodegenerative Disease” poster session, I would pay a fee in Bitcoin to do so. I don’t pay to view sessions that I’m not interested in.

Pros: 1. This model significantly reduces cost. Consider the costs are in the millions to organize and fund the event, our costs now are reduced to server space for streaming/bandwidth and event organization. I don’t know enough about this to give exact numbers, but considering it cost YouTube roughly $300,000 to run the servers needed to stream “Gangnam Style” 500 million times, I’d say this model is cost-effective (69).

2. The public gets in on it, from anywhere in the world. If my parents wanted to enter the Society for Neuroscience conference, they had to pay $50 per day to do so. No thanks. With our model, we allow the public to tune in to talks and poster sessions to learn more about what scientists actually do. Why are we hiding leading scientific research and the most informative and interesting talks from the public? By allowing the public access (again, not free but significantly reduced cost) we further bridge the gap of science communication between the public and actual scientists.

Neutral?: 1. In order for me to present my poster, I just have to sit in a live session with a powerpoint up. If I’m delivering a talk am I just standing in a room by myself, clicking next on my screen while talking to a wall? This certainly puts presenters in unfamiliar and perhaps awkward territory. Of course, it can be done – but the question remains if it is desirable.

2. It’s easy to steal data presented. I put this in the neutral section because although there will often be signs at conferences saying “no taking pictures,” you’d have to be pretty naïve to believe that’s not happening. We all have phones. If you don’t want your data being a risk to having your ideas and projects scooped, then don’t present that data, plain and simple.

Cons: 1. You lose the face to face interactions. You lose socializing. You lose the ability to conduct other events such as award presentations. All of these are definitely important to the way we as humans interact and exchange information. The overall “feel” of a conference style event taking place online would certainly be different. Not all would sacrifice the experience to attend an online conference – but I think I would.

2. Vendors would have to go elsewhere to market their products and offer information and demos.

It Doesn’t Have to Be Bitcoin

Note: Most of the concepts discussed in this section are derived from the discussion from this Let’s Talk Bitcoin podcast episode and blog posts by Adam Levine (59, 60).

The invention of Bitcoin laid out a technology that can be copied and modified in many different ways. What follows has been the invention of hundreds of different cryptocurrencies. No one knows the end-point of Bitcoin and whether it will function as a global currency with the complete elimination of fiat, alongside fiat, as just a store of value, as a commodity, in tandem with several other cryptocurrencies, or fail completely. Because of the ease of making new cryptocurrencies, we can actually make a coin or token of value tailored uniquely to our forum model.

Why would we want to do this? The concept of creating a token of value for your own independent network isn’t a new idea. Virtually every large payment network, large retailer, or corporation has some form of rewards system, to redeemable cash kickbacks for using your credit card, to airline miles, to hotel points, these systems exist to incentivize users to remain in that specific company’s network. In the same way, issuance of a unique cryptocurrency for a forum can empower and incentivize users to remain and interact exclusively on that forum’s network.

In the previously proposed model, funding your account’s wallet would be done via the purchasing and transfer of Bitcoin. It may be appropriate to give new, verified users a small amount of Bitcoin when their accounts are created in order to kickstart the community. However, by doing this we run into the problem where new users could create fake accounts solely for the purpose of receiving this free Bitcoin. Alternatively, and perhaps a better way, would be to just let the forum evolve on its own by allowing the fees for accessing pages and upvoting comments to be extremely miniscule at the start (fractions of a penny). This way, new users are not burdened by the cost to use the forum. A single dollar here could go a long way. As users and content increase over time, the fees would increase accordingly such that the incentive to contribute is tangible (i.e. no one is going to waste their time writing and contributing if the reward for doing so adds up to pennies).

In this alternative model, we create a forum-specific coin (token) -- let’s call it “Science Coin.” Science Coin is not mined like Bitcoin, but rather uses the Bitcoin network as the backbone for transferring value. Science Coin can operate under the security of Bitcoin’s blockchain via the Counterparty protocol (61). Counterparty allows for the creation of unique assets (in this case Science Coin) as well as a multi-wallet that contains a Bitcoin address which can also receive other alternative coins, such as our newly minted Science Coin. The creation of Science Coin is done via Counterparty such that the amount of coins created (remember, no mining) can be set and locked to a fixed amount and distributed to users in any desired fashion.

The best way to understand how this works is to analyze the best example, the official token of the Let’s Talk Bitcoin network, LTBCoin (62). 510,000,000 LTBCoins were created on Counterparty with the intent on distributing the coins among users over the course of 5 years. The distribution is set according to a fixed, weekly schedule to users who create content, make posts on the forums, platform development, etc. Essentially, users earn coins by contributing to the network, which can be done in various ways nicknamed “Proof of Participation,” “Proof of Quality,” and “Proof of Value.” (PoP, PoQ, PoV). (63). With this system in place, the PoP, PoQ, and PoV points are computed per audience member, per week, which in turn gives them a specific set amount of LTBCoin according to their point totals. In this sense, LTBCoin is “mined” through the proof of users contributing in some fashion to the network via forum or article views, comments, or creation. LTBCoin therefore serves as the official currency of the LTB network, the same way that rewards points do for hotels. Importantly, LTBCoin costs virtually nothing to actually make and distribute.

A key distinction between this model and the one previously proposed is that it doesn’t utilize a Reddit-style interface, such that the concept of upvotes and downvotes does not exist in the same fashion. Rather, users can accumulate points from commenting on articles, posting on the forums, and just viewing pages, but they can also lose points if a comment is heavily downvoted. Additionally, a user’s total points get weighted toward the influence they have on likes such that a user with more points liking a submission by a user with less points earns the submitter more points (64). Because Let’s Talk Bitcoin aims to be a network around all forms of content (videos, podcasts, articles, etc) rather than just a forum for discussion, its methods for accumulation of LTBCoin are skewed toward content creation in the network, with PoV and PoQ weighing heavily. In our previous model, the incentive is mainly around creating useful comments and accumulation of “points” – in our case, Bitcoin, via upvotes or micro-tips.

Perhaps the most striking difference between this model and our Bitcoin model is that instead of starting with a coin that has value like Bitcoin, we instead need to create value within the system. As mentioned earlier, the whole point of creating a separate, network-specific rewards system, token, or whatever you want to call it is to incentivize users to remain within that network. Therefore, LTBCoins are completely worthless without an incentive to use them.

Value of LTBCoin is therefore created by making it cheaper to use on the LTB network than other payment alternatives such as USD or BTC. For instance, you could offer adspace on the LTB network for a 40% discount when paying in LTBCoin. The moment that an advertisement is purchased in LTBCoin, value is created, as LTBCoin has successfully functioned as a medium of exchange of goods and services. The same thing happened when 10,000 Bitcoins were exchanged for a pizza back in 2010 (65). By giving our token of value an incentive to use, we offer a method to bootstrap a community that is otherwise voluntary in participation. This same strategy could be used for Science Coin. Perhaps using Science Coin to access a thread’s content can be 10% cheaper than a different alternative such as Bitcoin. Again, any type of website fee, advertising, etc, would be offered at a discounted price when using Science Coin.

How do we actually determine value? Here, the versatility behind cryptocurrencies comes into play again. Counterparty is not just an asset creator, it also functions as a full blown cryptocurrency exchange. This means people can trade Science Coin for Bitcoin via Counterparty and Bitcoin to USD on another exchange, which offers a “cash out” option. Alternatively, Science Coin could be adopted for crypto-crypto trading on a platform like Shapeshift.io. By allowing open trade of Science Coin, its value is determined by what the market determines its value to be. This opens up a few interesting points.

Because LTBCoin can be easily exchanged for other cryptocurrencies, which can in turn be exchanged for other fiat currencies, LTBCoin’s value essentially becomes pegged to Bitcoin, exposing it to trading manipulation, speculation, and subsequent volatility. We can take a look at how this may work by looking at the brief history of LTBCoin (66).

The market cap of LTBCoin is only ~$30,000 at the time of writing, which means that if someone wanted to, they could buy up all the LTBCoin for sale on an exchange relatively cheaply (keeping in mind that not all LTBCoin are for sale). In this sense, we would give Science Coin value by making it valuable to use within our created network. However, by allowing it to be exchanged for other currencies that have value outside of our network, it is at the mercy of regular outside market forces. Not many are going to want to use Science Coin if their Science Coin is worth half of what it was yesterday.

On the other hand, LTBCoin is deflationary in nature since it has a fixed amount and known distribution timeline. Therefore, similarly to Bitcoin, its value should increase as its utility increases without being diluted out by inflation. The deflationary aspect of LTBCoin may also make it a popular target for speculators, who may seek out purchasing LTBCoin if the network shows promise for consistent growth. We may then consider this as an extra incentive for early users of the network. Their early work in generating content acts as an investment that will accumulate value due to the foundation of utility they originally created. It will be interesting to see how this experiment plays out.

One important point to make here is that the creation of a separate token like LTBCoin does not mean it is decentralized. Rather, it is a centralized system on the backbone of a decentralized system. We still have to trust the creator to distribute the coins as scheduled. This is similar to our previously discussed model that moves Bitcoin by utilizing off-chain transactions, which also requires trust. I don’t think it is necessary to create a truly decentralized forum for our use-case, which would necessitate the creation and maintenance of its own blockchain. For us, a trusted entity increases efficiency dramatically.

Can We Build It?

Disclaimer: I am not a software developer and have no experience in programming.

Given that Reddit is open source (67), it is rather trivial to make a clone of Reddit (68). Additionally, Reddit's former cryptocurrency engineer, Ryan Charles, has claimed that he was able to implement a peer-to-peer payments and wallet system in Reddit within a single day working with the team at Chain (32, 35:30-40:00). Given the amount of APIs that are being developed and the open source nature in the Bitcoin space, I don’t imagine building this forum would actually be difficult. Rather, the most difficult part would be in building a user base. So who's in?