Post navigation

Last week I was invited to give a talk at the Big Data Innovation Summit in Boston. My colleague, Patrick Philips, and I gave a talk about how LinkedIn uses crowdsourcing to improve its machine learning algorithms and data products. Since this post is not about this talk, I will just mention that we got great feedbacks and you can watch it here.

Parallel to the data innovation summit, there was another conference held in Boston, the Sports Analytics Innovation Summit. Since Patrick and I are big sports fans, our work involves doing a lot of analysis and we had a free pass because both conferences were organized by the same group, we decided to drop by.

We weren’t wrong. It was incredibly interesting to see the difference between our world – data products oriented consumer internet companies and the world of sports analytics. While state of the art for the internet companies is analyzing terabytes of data using distributed big data frameworks and applying machine learning algorithms such as deep learning neural networks, the state of the art for sports analytics to put it mildly … is different.

One of the most informative sessions at the conferences was given by the data analysts of the National Football League. In this session, the people in charge of analytics for the NFL explained that since data gathering and cleaning is a task that needs to be performed by all NFL teams, the NFL has built a new platform for teams to consume this data.

Platform is a pretty big word. One might imagine this platform as something similar to Google analytics where a team coach could log in and watch fancy graphs and charts about his team performance. It’s not exactly like that. It’s more like an Excel spreadsheet that holds very similar data to the one you might encounter at Yahoo! Sports or ESPN. Actually, it’s not more like an Excel spreadsheet, but it is exactly an Excel spreadsheet that is emailed to the teams in the league every week.

The spreadsheet contains a lot of tabs and a lot of canned reports about which team played, what was the performance of the players, which players were on the field for any given play and links to the videos of the plays. Pretty straightforward stuff. But there was something also a little bit different there. Something that immediately caught my eye, because it was a data product, and one that in the idea, very similar to many of the products we develop at LinkedIn. The product was named “Similar Running Backs”, which sounds very similar to the “Similar Profiles”, “Similar Companies” and “Similar Schools” LinkedIn data products.

The way the NFL analysts explained the idea behind Similar Running Backs is that every year the teams need to renegotiate contracts with their players. To make the negotiations (which sometimes go up to more than ten million dollars a year contracts) efficient, it is very helpful to the teams to understand how similar players are compensated. So the league created this tool for the teams as part of their new platform and the first version of this tool was comparing running backs.

Here is how it works – you select a player from a drop down list and the select two numbers which represent the similarity range of the players you are looking for. The smaller the range, the less players will fit the criteria and vice versa. For example: the values 95% and 105% will return the players whose regular season stats are between 95% and 105% of the corresponding statistic of the selected player.

Now let’s look at real data and see how the algorithm works. Note: for this analysis I only looked at players who in the 2012 season played at least 10 games and who had at least 10 rushing attempts. Let’s see what data do we have. The NFL used the following stats to assess running back similarity: number of games played, rushing attempts made, total rushing yards, rushing yards per game, rushing yards per attempt and rushing touchdowns.

Issue #1

Since the job of the running back is to carry the ball through the defensive line, which is basically a set of about seven 300 lbs. guys, running backs tend to get injured a lot. This causes them to miss a lot of games and makes hard to compare players who played all games in a season to players who played part of the games.

Solution #1

Normalize the data by the amount of games played. That is, instead of counting total rushing attempts and total rushing touchdowns, use rushing attempts per game and rushing touchdowns per game.

Now that we have the stats right, let’s try to find all the players with +/-5% range to every statistic.

Issue #2

Not all stats are alike. For example, in 2012 the top rushing yards player was Adrian Peterson with 131 yards per game while the lowest player, Jorvorskie Lane, rushed only 0.8 yards per game, about 160X times less. This means that the gaps between the rushing yards per game of players can be very significant. In comparison, in the rushing yards per attempt category, the top player, Cedric Peerman, rushed for 7.2 yards per attempt, while the lowest player rush for 1 yard per attempt which is only 7.2X lower. Since the differences between players on those two metrics are very different, it doesn’t make sense to compare 5% difference on these two stats as they symbolize the same “similarity”. Being within 5% of rushing yards per game is very similar, while for rushing per attempt it’s not.

Solution #2

Normalize the data to have the same units of distance. What we want to do here is to transform of our measurements to have the same range. One way to do so is to use the standard deviation metric. Standard deviation is a measurement for how wide is our range. Think of a bell curve, the wider the bell curve, the higher the standard deviation. We want the bell curves for all of our stats to look similar to each other. To accomplish this, we can normalize our data by the standard deviation. (This post is too short to explain why this concept works. Feel free to read more in this article).

Now that we have the right stats and they are all comparable, we can start looking at what players are similar to each other. Remember, since we have only four stats to work with, the most similar players will have all of their stats within 95%-105% of each other. Less similar players will have only 3, 2, so on and so forth.

It appears there are no two players in the league who are very similar on all four stats, but there are some who share three. Here is a visualization of these similarities:

We can see from this graph that there are three pairs of players that are very similar to each other in their stats and a cluster of six players who are also similar to one another.

Issue #3

While Darius Reynaud is similar to D.J. Ware who is similar to Le’Ron McClain who is similar to Jason Snelling. The first and the last are not very similar to each other. While both their output wasn’t high, Jason Snelling rushed twice more per game and per attempt than Darius Reynaud.

Issue #4

This similarity metric is too coarse. It’s all or nothing, either the players are within 5% from each other in most stats or they don’t. Even if we reduce the number of stats players have to be similar at to two as can be seen in this graph.

We still get only pairs of players who are similar to each other, but it is hard to see how they compare to other players.

Issue #5

This similarity product has too many levers. We need to provide it with the range of what it means that two players are similar to each other and we also need to provide how many stats should be similar.

Solution to #3, #4 and #5

We can provide a visualization that:

Displays all the players

Uses a continuous similarity metric where closer means more similar instead of the binary similar or not similar we used before

Doesn’t need any levers

In order to achieve that, we will just cluster all players into groups and then display all the players on a chart where similar players will be close to each other and dissimilar people are far.

Now we can see all the players in a single graph separated into five groups. The red group (number 1) are the superstars, guys like Adrian Peterson, Marshawn Lynch and Arian Foster. These are the guys with the most rushing attempts, the highest yardage per game and the guys who by far scored the most touchdowns. The group closest to it, in magenta (number 5), are the second tier guys. These guys are very productive running backs, just not as productive as the guys in the first group. But while these two groups are interesting, pretty much every football fan could break these players into these three buckets. What is more interesting is who are the other three groups.

The second magenta group (number 3) is our least productive players. These players rushed only for 4 yards on average in a game with each attempt advancing them slightly more than 2 yards. The blue group (number 4) is made of players that while rushing 5 times the yards per game and twice the yardage per attempt, managed to score about the same numbers of touchdowns, 0.07 a game. The green group is made of players who are very similar to the blue group, only twice more effective in scoring touchdowns.

While this analysis does not provide a myriad of insights that are not already known to subject matter experts it does provide a nice and robust framework to understand player similarities with a single look. Also, while it’s very easy to compare players according to only their rushing abilities, things become more complicated when we add more dimensions to look at like fumbles and catches. Which running backs finished last season most similar to each other in terms of all this stats combine? This is a much harder question to answer for which I will let you guess in the comments, but the answer could be easily displayed once again as point on a flat surface.

Most interviews at tech companies are very similar. You are greeted, perhaps given a brief intro about the interviewer and the interview quickly becomes technical. The interviewers want to learn as much as they can about the person they are talking to in the allotted time to make the best hiring decision they can. The way tech companies train their employees to interview, is to ask the same question, have specific answers, and in the case of multiple possible answers, know which ones are better.

This process tries to minimize the variance and make it easy to compare candidates to each other. It creates a very efficient process for selecting candidates, but it doesn’t leave much room for the candidates to connect with their interviewers and letting them know how awesome they are as more than just a code generating apparatus.

Even though on most interviews candidates are not asked this exact question, they are still given the opportunity to show how great they are. This question is asked at every single interview, but everyone gets it wrong. The question is …

“Do you have any questions?”

Most people answer this question by asking meaningless questions or even worse, not asking anything, signaling that they are not very interested in the opportunity you are offering.

To understand why everyone gets this questions wrong, we need to step back and to figure out what is the goal of the interview. Candidates get it wrong all the time, similarly to how people misinterpret the purpose of a resume. The same way that resume is not a brief history of our professional career but just a document with the sole purpose of getting the candidate through the screening process to an interview, an interview is not an opportunity to gather more information about the perks and the working environment of the company you are interviewing for, but just a step to another interview with the final goal of getting an offer.

Perks, reporting, projects, etc. are all very important in the decision of which company to work for, but in this competitive environment where companies battle for the best and the brightest, once a candidate passes the interview stage, not only companies are willing to provide this information, they will invite candidates for a full day of hanging around in the office, meeting all their future teammates and learning about their projects and having a nice lunch or even a fancy dinner to answer any questions they might have.

Although most people do not like to admit it, personal impression of the candidate will carry a significant weight in interview decisions. And since most of the interview experiences as an interviewer is rejecting candidates, leaving an interview that went very well is a nice change.

Interview is a lot like a first date, it is a brief encounter that has a great effect on whether there will be a second one and perhaps a long lasting relationship. The way to make this meeting successful is doing some basic research to find out their interests and why do they love what they do. Bring up these topics at the end of the interview and make your interviewer leave the meeting with a feeling that she just met a person that could be her friend.

One of the examples for the best interviews I ever had is when a candidate have seen a conference I attended and realized I’ve asked a question at the end of one of the sessions. He realized that I care enough about the subject to be the first to ask a question and followed up with a discussion about the subject stating his own opinion. Another example is when a candidate have watched one of my talks online and recognized one of the tradeoff points I’ve made and started a discussion around the reasoning behind this tradeoff. The depth of this conversations have signaled to me that the candidates are genuinely passionate about the subjects I care about as well or at the very least care enough about the interview process to go through hours of online videos.

The goal of every interview should be to get to the next interview, and the best way to go about it besides answering the questions right is asking the right questions. So next time you are asked whether do you have any questions, be different.

A lot of articles have been written about data science being the greatest thing since sliced bread (including this one by me). Data products are the driving force behind new multi-billion dollar companies and a lot of the things we do today on a day to day basis have machine learning algorithms behind them. But unfortunately, even though data science is a concept invented in the 21st century, in practice the state of data science is more similar to software engineering in mid 20th century.

The pioneers of data science did a great job of making it very accessible and fairly easy to pick up, but since it’s beginning circa 2005, not much effort has been made to bring it up to par with modern software engineering practices. Machine learning code is still code, and as any software that reaches production environments it should follow standard software engineering practices, like modularity, maintainability and quality (among many others).

The talk, Scalable and Flexible Machine Learning, which I gave with Christopher Severs in multiple venues reflected our frustration around this issue and proposed a solution that we feel brings us closer to where data science should be.

The first thing to understand is that data science is mostly manipulation of data. Usually the data can be of large scale and complex, but these manipulations are commonly found in non data science code bases as well. Even more, since in the work of a data scientist you don’t know which functionality achieves the desired result, for example whether the right metric is mean or median, the argument for modular code becomes much stronger.

Our second proposition was that the current tooling that data scientist use is inadequate for the type of systems that end up in production. Real world data processing in many cases is at least as complicated as writing regular software tasks such as fetching data from a database, passing messages between mobile devices or throttling the bit rate of a streamed video, only in the former case the tools being used are on one spectrum the SQL-like Hive and the very basic scripting language Pig. Although these language can be extended using user defined functions, those functions are very ad-hoc in nature and very difficult to reuse. On the other spectrum, there is the vanilla Java MapReduce which consists of an awful amount of boilerplate code that has very little to do with the actual desired functionality.

We tried to propose a more modern alternative that will combine the best of both worlds, concise and high level as Pig and Hive and a fully powered programming language like Java. Our technology of choice was Scalding which is a high level abstraction framework over MapReduce written in Scala. It has all of the functionality of Pig and Hive achieved sometimes with fewer lines of code. The fact that it is written in Scala, which is a modern language over the Java Virtual Machine is a great feature since all of the Java libraries can be reused. But Scala is not just a modern version of Java, it is a language designed for the functional paradigm.

I won’t go in this post on the what is a functional programming language and all the features that make it better for data processing, but if I would have to provide just a single proof to support my case, is that MapReduce is a very functional concept. Both map and reduce are higher-order functions, a cornerstone of functional programming. It is not a coincidence that Google chose this simple functional paradigm to be the base of their entire data processing framework.

Our last point was addressing the tendency of many data scientists to overcomplicate their algorithms. These data scientists begin their search for the solution for their problem by reading all the literature on the topic of their problem. One of the problems with this approach, is that it tends to over complicate a solution for the sake of a minor gain. The best example of this phenomena is the Netflix Challenge. Netflix offered one million dollars to a team that will beat its inhouse algorithm by at least 10%. The winning algorithm was so complicated that Netflix decided to pay the prize without actually implementing it. In many cases, the first algorithm can be an off-the-shelf algorithm that can be found in almost any machine learning package, for example: PageRank for ranking or Collaborative Filtering for recommendations.

To sum it all up, if you are writing your data processing code in complete disregard of all the engineering principles of the last few decades and developing your algorithms according to the state of the art publications without trying the simple ones first – You’re Doing It Wrong!

My first position as a data scientist, was in the israeli intelligence, in a unit which is the equivalent of the American National Security Agency (NSA). I do not know anything about PRISM, NSA’s surveillance program, but my experience both in government positions and afterwards working for two big data companies helps me understand what are the drivers of the people who work on it.

The first thing to understand about this problem, is that it is not a big data problem. It is a huge data problem. Words cannot describe the amounts of data we are talking about. Storing all the communication information from Google, Facebook and the wireless carriers would require multiple data centers each the size of several football fields. Such amounts of data cannot be processed by people, instead, intelligence organizations rely on sophisticated algorithms to do most of the work to find this needle in a haystack they are looking for.

There are a lot of different challenges intelligence agencies are trying to solve. For this post I will focus on a single challenge – which individuals present a threat to national security. This problem is referred to in the machine learning world as a classification problem. Classification is the problem of identifying to which category an observation belongs, for example: given the level of force applied to a car’s door handle determine whether the alarm should go off, given an image determine whether it contains a human face, or given a person’s communication records determine whether she poses a threat.

The way to build a good classification model is to have a good training set. A training set is a set of records which you already know the answer for. If we use the face recognition challenge, the training will have some images that have faces and some that do not. It is very easy to come up with an extensive training set for the facial recognition challenge. Not so much for terrorists. There are just not so many cases of real threats to national security to learn from, and those that who exist, differ considerably from each other. A very small training set leads to a very inaccurate classification model.

In the case of national security, this is simply not good enough. Every threat that the model fails to detect might result in a very bad outcome. After 9/11 no one wants to be the person who let a terrorism event happen on his watch. This concern drives data scientist to constantly try to improve their models. Unfortunately, it is not always possible to improve models using the same training data. So when data scientists exhaust their modeling capabilities, they turn to get more data. But in the case of identifying people who don’t want to be identified, the most useful data is not one you would find in public records.

At a certain point, the fear of being responsible for people’s lives numbs the sense of right and wrong. People start to see the model and forget about its consequences. When every percentage point of improvement is another terrorist that will be detected before committing something terrible, privacy takes a backseat.

But privacy is not necessarily less important than safety. In some sense, privacy is safety and “National Security” is not a magic term that allows the government to trample over basic rights. In a democratic society, the government is required to do better than paternalize its people. Drastic times call for drastic measures, but these are not drastic times; so drastic measures call for drastic explanations.

disclaimer: This post does not represent the official opinion of anyone but myself. I have no knowledge whatsoever on PRISM or equivalent programs. This post is just my educated guess on what is driving the people who work on it.

People are the most important resource a tech company has. It is the employees who come up with brilliant new products, develop them efficiently and find customers to use them. The best employees can be far more valuable than the average employee, and there is no better example than Steve Jobs – the man who transformed Apple into the largest company in the world.

It is very hard to assess the impact an employee has on an organization. We cannot perform an A/B test to compare the outcomes of a company with or without the same employee. But in the cases of some CEOs, sometimes, it is very hard to argue with their success. In 1997, when Jobs returned to Apple, the company he co-founded and was fired from 12 years earlier, it was worth a bit more than $3B. Just before his death, Apple’s worth reached an all time high, over $650B. In his 15 year tenure as the CEO, Apple’s market cap has increased by more than half a trillion dollars – making Apple the most valuable company in the world.

Steve Jobs didn’t start his career as the icon he is today. He graduated from high school with a 2.65 GPA and unlike Bill Gates and Mark Zuckerberg, who dropped out of Harvard to start their companies, Jobs dropped out of the much less prestigious Reed College. A candidate with his credentials would find it hard to get an interview at a fast food chain, let alone Apple, the company he co-founded and led.

Great companies require great employees across the entire organization – not only at the CEO position. This is why hiring is the top priority on many tech companies’ lists. Valve, the company that developed the Half-Life series, even writes in its handbook for new employees

“Hiring well is the most important thing in the universe. Nothing else comes close. It’s more important than breathing. So when you’re working on hiring—participating in an interview loop or innovating in the general area of recruiting—everything else you could be doing is stupid and should be ignored!”.

So why do most hiring processes look more like an assembly line?

Truth is, hiring is a very time consuming task. Successful companies will hire only about 1% of all the candidates who apply. Since interviewing 99% of the candidates results in a waste of time for the employees who interview them, companies are doing everything to optimize this process. These optimizations include filtering out all resumes that don’t have the right keywords, conducting automated coding screens that don’t require people and front-loading the most difficult interview to reduce the average amount of interviews per candidate. The other problem with hiring, is the asymmetry of the mistakes. Hiring the wrong person is much more noticeable than not hiring the right one. This puts the hiring priorities as follows:

Go through as many candidates as possible for a given unit of time

Avoid hiring the wrong people

Hire the right people

This prioritization causes people involved in a hiring process to look for any reason to reject a candidate instead of looking for reasons to accept a candidate. This is very unfortunate. While a great employee can increase the value of a company by orders of magnitude, the risk of very bad employees can be mitigated by termination.

Much digital ink has been spilled on describing how a 45 minute interview should be used to assess certain skills. But this shouldn’t be the main discussion. The best employees are not necessarily the best at any particular skill. The most valuable employees usually stand out for their passion, enthusiasm and creativity – traits that are very hard to pick up in an interview.

In this information age, gathering information about candidates is easier than ever. My second degree network has more than 200,000 people. For everyone of these people, I can find at least one person I know that can tell me something about them. If you include other colleagues at your company who you are not connected to, but can give an honest opinion about a candidate, that network grows even larger. One of the best people I’ve worked with, was a DBA in a different part of the company, but his name kept coming up as this great person to work with. When I met him I found out that one of his hobbies is data science and while putting crazy hours on his understaffed team, he still managed to find some time to hack cool data products. If he would just pass his resume back then, it wouldn’t even get an interview, and I would be without a great employee.

At my current company, LinkedIn, we have a tradition to devote one day each month to a hackday. People work one day on a project and then have 2 minutes to present it to a panel of judges. Each month about 30 projects submitted, and some are just amazing for a day worth of work. If a candidate came to an interview and showed me one of those projects, I would probably hire her on the spot. At the very least, I would assign much more weight to this showcase of talent than any interview question I can think of. The average about hiring process at LinkedIn takes, in total, roughly the same time that people spend on their hackday projects. Allow people an opportunity to amaze you by hacking a project at the comfort of their own home and showing the results afterwards. With some creativity this method can be applied not only to engineers, but also to designers, marketers, salespeople and many other professions.

If the candidate is new to the area so he doesn’t have a network to vouch for her and the circumstances do not allow for her to showcase her skills. Another approach might be – “Try Before You Buy”. This is very similar to an internship, the employee works for a predetermined period and at the end, either gets a full time offer or not. This method is the best in terms of the amount of information gathered on the candidate, but can be tricky to perform. First, there is the problem of not many employees will agree to this conditions, but since the alternative is usually a flat no, the employer doesn’t have a lot to lose. The added benefit here, that the people who will agree are probably the most passionate about your company. Second, even the best employees might find it hard to prove themselves in a short period of time. This requires a very careful planning and to be honest and open with the candidate about what is expected from her.

Great employees are like great startups – a handful of them create most of the value. Steve Jobs, Larry Page or Mark Zuckerberg did not have the best credentials for entrepreneurs, but it didn’t stop them from building some of the most exciting companies in the world. Same goes for employees. The best people I have worked with did not have the shiniest resumes, but their value to their companies was a great deal higher than their average colleague. Avoiding hiring the wrong people does not result in hiring the right people, but in passing over the next Steve Jobs.

Up until 2010, Dwyane Wade was the star of the Miami Heat. No, actually, Dwyane Wade was the Miami Heat. In 2006 he led his team to the NBA championship and won the NBA Finals Most Valuable Player award. And he didn’t stop there. In the following years he only got better, scoring more than 30 points per game in 2009, winning the NBA scoring title and becoming the NBA All-Star Game MVP in 2010. Wade was the hottest player in the league, and was paid accordingly, almost 16 million dollars per season.

But then, something terrible happened – the Miami Heat decided to recruit talent from another place and signed LeBron James. Suddenly everything has changed. Wade was no longer the star of his team. The number of his shots dropped, his assists were down by 30% and his salary was down by almost two million dollars from the year before. Basically, Dwyane Wade got screwed, but this is not the full story.

With James’s help, the Heat won more games, reached the NBA finals on their first year together and won the NBA championship a year later. They have become the most exciting team in the league, attracting many more viewers to their games and getting a lot more money through a new television deal. The entire team has enjoyed its new place in the spotlight. Dwyane Wade was no exception. His renewed stardom has landed him a 6 year contract worth over 100 million dollars, and one of the largest in history shoe endorsement campaigns with the Chinese athletic company, Li Ning.

The history of sports is full of organizations which transformed themselves and became successful after acquiring brilliant new talent. This is not limited to just sports teams. American companies have been attracting the best and the brightest employees from all over the world for decades, but the recent surge in unemployment has caused people to argue against this policy. While this arguments sounds reasonable, it is missing the big picture.

Over the last decades, hundreds of American companies were founded, led or contributed to by immigrants from all over the world. They were not born in the US, but still have made a significant contribution to the US economy. According to Forbes, 40% of the Fortune 500 companies were founded by either immigrants or their children and 75% of the companies who received venture capital funding had at least one foreign born core member. Google, PayPal and eBay all had immigrants as co-founders and currently employ large number of foreigners. Besides creating thousands of jobs for the people they currently employ, they have also created ecosystems that give jobs to millions of others. Great talent creates great opportunities that everyone can benefit from.

Without James, Wade would still be the star of his team. He would have scored the most points, have the highest exposure, and the highest salary. But the Heat would not be the most popular team in the league and its players would go mostly unnoticed. Success is achieved by increasing the size of the pie, not by preventing pieces from others.

I encountered bitcoin for the first time when I was researching anonymous web commerce. The exchange rate at the time, almost $10 for a single bitcoin, seemed as an awfully high premium for anonymity. I was dead wrong. When the total value of bitcoins in circulation rose above billion dollars, a media frenzy started and the increasing interest drove prices through the roof, reaching $266 at the peak.

For those unfamiliar with bitcoin, it is a new type of currency with the following differences from regular currencies:

Anonymous – customers of online contraband drugs or pornography – few of bitcoin’s main use cases – prefer anonymity. Unlike credit cards, bitcoins are not attached to a name of a person but rather to a large set of different computer generated ids.

Decentralized – like any commodity, the value of currency is driven by supply and demand. When a government increases the supply of a currency (for example, by printing more notes), it depreciates its value. People who do not trust their governments to handle fiscal responsibilities properly are always looking for alternatives to depreciating currencies. Since bitcoin is not regulated by a central bank, and its algorithm ensures that there will never be more than 21 million coins in circulation, bitcoins should hold their value better than centralized currencies.

Virtual – when the Cypriot government decided to tax the domestic bank accounts of all its citizens by almost 10%, people all over the world started looking for safe alternatives. Bitcoin’s virtual nature and the fact that there is no central repository that holds them, makes them nearly impossible to confiscate or tax by governments. This is similar to why peer-to-peer file sharing networks like BitTorrent are hard to shut down.

While bitcoin was created as an alternative for common currency, it is hardly used in online commerce. According to the bitcoin official website, less than 0.3% of bitcoins are used in trade. When the value of a currency constantly increases, the amount of bitcoins an item costs constantly goes down, stagnating trade in the process. But why the value of bitcoins increases?

People love get-rich-quick schemes. There is nothing better than to be the person who succeeded to do that and the center of everyone’s conversation. Since both the financial and the emotional rewards are high, the demand for such opportunities is high as well. According to auction theory, excess demand for an item of limited supply, leads to the overvaluation of those items. This phenomenon is known as the “Winner’s Curse”.

The winner’s curse states that in common value auctions (auctions where the value of the item is identical to all bidders) with incomplete information (auctions where people do not know the real value of the item), the winner of the auction will tend to overpay. Think about an auction of a jar of coins, the winner of this auction will be the one that thought that the jar contains more money than anyone else in the auction. The probability that the winner is the only one who is right and everyone else underestimated the value of the item is very low, which leads to overpayment. The higher the demand for an item, the higher the error. Bitcoin is limited in supply by design. The fact that most of the bitcoins are hoarded and not available to buy, lowers supply even further, leading to exorbitant prices.

The value of bitcoin can double or halve within a week so it cannot be used as a currency, and since it has no intrinsic value, it is no good as an investment. Historical trend driven valuations such as: the 17th century tulips, early millennia dot-com companies and recent real estate, have all eventually crashed. The catalysts for bitcoin could range anywhere between the lose of public interest in the coin to the rise of a competitor which will reduce the scarcity of virtual coins, the main driver behind bitcoin’s price.

Bitcoin is a fascinating achievement in both cryptography and new economic thinking that solves a series of real problems. Its only fault – it was cursed with success.