Posts categorized "Film"

Last time I mentioned Moviepass in May, the stock of its owner (NASDAQ: HMNY) was over $2. Today, it is nearing 20 cents! Lost 9/10 of its value in two months.

This latest plunge is related to a "hail mary" attempt to raise cash to fund its giveaway business model. It currently claims 3 million users and is losing $40 million ... every month!

Other news: the movie chains launched their own rival service. That's a smart move - kill off the value destroyer while it is still weak.

In the meantime, what does Moviepass say they are doing to keep alive?

One idea is "surge pricing": they are to impose surcharges for popular movies at popular times. And they have floated various other ideas to "upsell", generating more revenues from existing customers. It doesn't seem to realize that many of its customers are not the average movie-goer but the nickel-and-diming customers who smell a good deal when they see one, and will walk away if the deal isn't there.

The fundamental problem of the Moviepass business model has never changed - it loses money on all new customers, because it is paying full price for the tickets. Instead of fixing the problem on the new customers, it is trying to get as many customers in the "store" and then squeeze more money out of them while they are inside.

According to this Bloomberg article, the CEO explains that the strategy is "to sign up more subscribers who don’t go to the movies quite so often, making them more profitable." So he is betting on innumerate customers - people who don't understand that if they are paying $9.95 a month, they have to see at least one movie a month to make the subscription worth it.

Now, imagine you subscribe because you smell a good deal. The first movie you see is paid for by the monthly subscription. The next movie you see has to be paid for by another subscriber who has an idle month ("subscribers who don't go to the movies quite so often"). The third movie you see has to be paid for by another subscriber's idle month, etc. etc. If you see 10 movies a month, then Moviepass has to find 9 idle months to break even on your one subscription. And that is one month. The next month, if you see 10 movies, it has to find another 9 idle months to pay for them. (I'm using 10 movies not the maximum of 30 movies a month since I don't think there are 30 movies a month that someone can pay me to watch.)

Older readers might not have heard of the company MoviePass but millennials likely have, as they account for half of the 2 million subscribers (link).

Moviepass is basically Groupon, which we debunked on this blog years ago. Scratch that. Moviepass is worse than Groupon. At the start, Groupon had at least some cheerleaders but Moviepass sounds dead on arrival.

Recall: Groupon is the digital coupon startup that offered an incredible deal to consumers - a typical coupon for a restaurant will let you dine there at half price or more. So for a $100-meal, the restaurant gets paid $50, but Groupon also claims it is bringing diners to the restaurant and takes a revenue share, meaning the restaurant takes in $25 before paying rent, food costs, people costs, etc.

Groupon definitely has a great deal for consumers but the restaurants got the short end of the stick. The deal only makes sense if the coupon users would otherwise not have dined at the restaurant. If the coupon users are regular diners, then the restaurant loses $75 for every such table! You'd need three new diners to cover the loss of one regular diner (that only gets the restaurant to break even on the offer cost). In Numbersense, I use this example to illustrate counterfactual thinking.

Worse still, the regular diner has a higher chance of wanting that Groupon, just because s/he already likes the product.

***

What is Moviepass? It is a monthly subscription plan for movie-going. It made the news in 2017 when the company dropped the monthly subscription price to $9.95, and grew subscribers 100-fold to 2 million. For $9.95 a month - roughly the average price of one movie ticket, subscribers are allowed to see one movie a day. Yes, this is insane.

It is even more insane. Moviepass pays the movie theaters full price for the tickets so the only way it makes any profit is if the subscriber does not watch even one movie a month. They would need subscribers who pay them monthly but don't watch movies.

Exactly the opposite should happen. The more movies one watches, the more lucrative is this subscription plan, and the more likely one will sign up. If movies is not a big part of your life, you're just not going to buy a subscription in Moviepass. If a subscriber watches one movie a day (the maximum allowed), then Moviepass loses $290 each month for that customer, month after month. I also know a subscriber who has a habit of just booking tickets and not using them. Just for giggles, he says.

***

There is speculation that maybe the cinemas eventually will strike a deal with Moviepass to sell them tickets at a discount. If cinemas were to do that, they become like the restaurant owners who bought into the Groupon scheme. The cinemas might think this brings them new customers. The reality is that it's their most loyal customers who will sign up most enthusiastically.

For each loyal customer who sign up, the cinemas would lose revenues on the discounting. If they gave Moviepass a 50% discount, then instead of getting $10 a ticket from these loyal movie-goers, they get $5 a ticket. To pay for that discount, Moviepass has to deliver one additional ticket from a newbie who would otherwise not have gone to the movie. But the loyal customer may watch say one movie a week but the newbie might watch one movie a month. So the cinemas might require four newbies to pay for the discounting of one loyal customer!

Moviepass is no doubt a great deal for movie lovers. But how long can this last?

In the on-going series of posts about the IMDB dataset, from Kaggle, I have so far looked at several of the scraped variables, including the number of faces on movie posters (1, 2), plot keywords (3), and movie rating by title year (4).

In this post, I tackle the variables resulting from a data merge between IMDB and Facebook. These columns have names like "Director Facebook Likes", "Actor 1 Facebook Likes", etc. I didn't investigate exactly how they did this merging. Presumably, they matched the names of the actors and directors to their official fan pages on Facebook, and took the Like counts. (I suppose identifying the right pages is not a trivial task.)

There is clearly a "theory" behind computing these variables: star power. So we should expect that movies directed by famous directors, or starring famous actors or actresses would be correlated with bigger box office receipts. The number of Facebook likes is a proxy measure for the concept of "star power" or "fame" or "popularity".

Proxies are necessary when dealing with quantities that are hard or impossible to measure directly. But be careful to choose proxies that describe the underlying quantities properly.

Facebook likes as a proxy for star power has a host of problems. For example:

Not all actors and directors use social media, or favor social media. If they do, Facebook may not be their chosen platform. In the extreme case, some countries like China block Facebook so obviously, a Chinese actor or actress would not be investing in a Facebook presence.

The Facebook like count is a snapshot. What is captured is count on the day on which the data was compiled. What you want is the Facebook like count in the days or weeks prior to the release of the respective movie!

Not only the protagonists but also the fans have preference for social media platforms. Actors and directors with older followers will have different Facebook statistics than those followed by younger generations.

All famous directors or actors have a breakthrough movie. They were nobodies before this movie, and became stars after this movie. The Facebook like count is a summary of the entire career of each person.

A quick glimpse of the data should give the analyst a pause.

Christopher Nolan with 22,000 likes dwarfs anybody else in this snippet but James Cameron with zero? Bryan Singer, zero? Sam Mendes, zero? (This could be a data merge error, or it could be a structural problem.)

Actors are not much more telling either, as this list of the top actors shows:

Darcy Donavan is 13 times more "famous" than Robin Williams (RIP) according to this metric. Darcy is primarily a TV actress so that's yet another issue with using Facebook likes to predict movie receipts.

Let's get back to the basic premise. We hypothesize that the Facebook like count of the director and/or top-billed actors and actresses is predictive of the movie's box office. But as you can see from Robin Williams's various entries, the Facebook like count for a given actor is invariant so if this factor is deemed useful to the model, it will contribute equally to each of Robin's movies throughout his career. When this model is used to predict revenues for early-career movies, it is using information that is out of bounds - the model in effect learned that Robin Williams would become a superstar in the future.

***

The lesson here is that proxies have lives of their own. There are a whole host of factors that drive the value of the proxy metrics. Understanding those issues and how they muddle the picture of your primary metric is essential to constructing a meaningful model.

In the last few posts about the IMDB dataset (1, 2, 3), I looked at two variables, the count of people in the movies' posters, and the plot keywords.

The people count variable is a modeled variable, which means that the output of one predictive algorithm is used to build another predictive model. We investigated the accuracy of the face recognition algorithm, and also learned that even for humans, counting faces is not as simple as you think - besides, we don't know who saw which poster.

The plot keywords variable looks rich at the surface but the person who wrote the scraper didn't realize that each movie is associated with hundreds of keywords, and only five or so are shown on the IMDB movie front page. This causes the variable to become completely useless.

In this post, I investigate the title_year variable, presumed to be the year in which each movie arrived at theaters. (Even this definition lacks clarity but I shall not tackle this aspect here.)

A key question related to any time-series data is stability over time. With IMDB, we can investigate the trend of IMDB score over time. IMDB score is the average (max:10) rating by the visitors to the IMDB website.

The average film scored between 6.5 and 7.5 at all times, with a clear downward trend. One could naively conclude that the quality of movies, as reflected by IMDB rating, has declined over time.

But what separates the good from the not-so-good analysts is the quest for other explanations. Which is the most plausible explanation?

The trendline is only about the average film. What the chart above fails to show is information about the distribution around the average. The following chart contains some added details:

The first thing one notices is the extreme skew in density of samples since the 1990s. There are few movies in the database from earlier times. This is confirmed by the histogram at the bottom of the chart, in which I annotated three key dates.

The website IMDB launched in 1990. It should be obvious that most users will be reviewing current or recent movies, and not surprising that movies before 1990s get reviews at a much lower clip. Secondly, IMDB got a boost in 1998 when it was purchased by the behemoth Amazon. Finally, the last year in the dataset is a steep drop - most likely to do with the lag between watching and reviewing movies.

The other feature of the scatter plot is the shape of the lower envelope (minimum). Clearly, older movies are rated only if they are good movies while IMDB users are much more likely to review recent movies that they did not enjoy. Not surprising if you think about the "causal structure" behind the data: the typical user likely reviews movies that he or she recently watched, the user is more likely to have watched a recently opened movie than an oldie, and if the user happens to watch an oldie, he or she is likely to pull out a well-reviewed title.

We have uncovered a bias in the IMDB average rating data - bad oldies are not in the database. (A smaller but still visible bias is the lull in the most recent titles due to lag.)

***

For most problems, it is safest to restrict your analysis to a period of time when the IMDB website is "mature". The analyst needs to figure out how this bias affects his or her analysis, if at all. For those keeping track, strike #3 is for those analysts who make statements about trends in this data without recognizing and dealing with the bias.

I should explain the title of the post. Each year, many students claim that they have a dataset with no missing values and no obvious typos, and therefore, they have nothing to do for the data pre-processing assignment. In this post, I did not discover any errors in the data - I uncovered bias in the data by deeply understanding how the data was collected.

Today I continue to explore the movie dataset, found on Kaggle. To catch up with previous work, see the blog posts 1 and 2.

One of the students came up with an interesting problem. Among the genre of action movies, are there particular plot elements that are correlated with box office? This problem is solvable because the dataset contains a variable called "plot keywords" lifted from IMDB.

Plot keywords are given in a single column in a pipe-delimited form. For example, one entry says "caribbean|curse|governor|pirate|undead." Sounds like one of those Johnny Depp movies.

Clearly, the first order of business is to unbind the keywords. We need to turn these individual words into "dummy variables" (i.e. 1 for the presence of "caribbean", 0 otherwise). Any analytical software will have such a capability. With Python, you might use the split function. With JMP, there is an excellent built-in function that converts such delimited strings into dummy variables with one click.

So now the one plot keyword column is turned into over 1,000 columns, each column relating to a specific keyword.

In theory, we are in business and can analyze this data using regression, trees, or whatever method one prefers.

***

If it is that simple, I wouldn't be blogging about this. As we like to say, you must look at your data. I ran some quick distribution analyses and was staring at these in confusion:

If the data were to be believed, words like "abduction" and "action hero" showed up in only one or two movies, out of a total of over 400 action films in this dataset.

Here is the entire histogram of how many times each keyword showed up in the data:

Huge alarm bells should be going off in the analyst's head right around now. There were only eleven movies about vampires? Only eleven martial arts movies? Only twelve movies involving superheroes?

I asked the student to figure out how this variable was collected. Go find the data dictionary to see if the keywords have some special criteria, such as statistically improbable words. Go to the IMDB page and look at what the scraping code might be doing (wrongly).

***

So what did we find? (See if you can spot the problem!) This is the screen shot of the section of the IMDB page for that Pirates of the Caribbean movie from which the plot keywords were lifted:

I have a bad habit of clicking on things when I am prompted to click. So I click on the "See All" link and a whole new world opened in front of my eyes!

So - the plot keyword column in the Kaggle dataset is completely useless. It contains five or six words in a list of hundreds of keywords. This also explains, at least partially, why the keywords appear so unique.

It pains me to think how many people have analyzed this dataset, and used these keywords to build models.

***

Back to the class project. Now this student was in serious trouble! Remember the problem she was hoping to solve is to find out which plot elements are predictive of box office. Turns out those keywords are extremely curtailed. So we had to immediately switch gears, and concoct a different problem that does not rely on these keywords.

(It was days before the due date. If there were more time, we can write a different scraping script to take the full list of keywords from the keyword pages and merge those to the existing dataset. Maybe someone who reads this post will be inspired to do it.)

If you are keeping count, that is strike #2. There are more posts coming.

In the previous post, I diagnosed one data issue with the IMDB dataset found on Kaggle. On average, the third-party face-recognition software undercounted the number of people on movie posters by 50%.

It turns out that counting the number of people on movie posters is a subjective activity. Reasonable people can disagree about the number of heads on some of those posters.

For example, here is a larger view of the Better Luck Tomorrow poster I showed yesterday:

By my count, there are six people on this poster. But notice the row of photos below the red title: someone could argue that there are more than six people on this poster. (Regardless, the algorithm is still completely wrong in this case, as it counted only one head.)

So one of the "rules" that I followed when counting heads is only count those people to whom the designer of the poster is drawing attention. Using this rule, I ignore the row of photos below the red title. Also by this rule, if a poster contains a main character, and its shadow, I only count the person once. If the poster contains a number of people in the background, such as generic soldiers in the battlefield, I do not count them.

Another rule I used is to count the back or side of a person even if I could not see his or her face provided that this person is a main character of the movie. For example, the following Rocky Balboa poster has one person on it.

(cf. The algorithm counted zero heads.)

***

According to the distribution of number of heads predicted by the algorithm, I learned that some posters may have dozens of people on them. So I pulled out these outliers and looked at them.

This poster of The Master (2012) is said to contain 31 people.

On a closer look, this is a tesselation of a triangle of faces. Should that count as three people or lots of people? As the color fades off on the sides of the poster, should we count those barely visible faces?

Counting is harder than it seems.

***

The discussion above leads to an important issue in building models. The analyst must have some working theory about how X is related to Y. If it is believed that the number of faces on the movie poster affects movie-goers' enthusiam, then that guides us to count certain people but not others.

***

If one were to keep pushing on the rationale of using this face count data, one inevitably arrives at a dead end. Here are the top results from a Google Image Search on "The Master 2012 poster":

Well, every movie is supported by a variety of posters. The bigger the movie, the bigger the marketing budget, the more numerous are the posters. There are two key observations from the above:

The blue tesselation is one of the key designs used for this movie. Within this design framework, some posters contain only three heads, some maybe a dozen heads, and some (like the one shown on IMDB) many dozens of heads.

Further, there are at least three other design concepts, completely different from the IMDB poster, and showing different number of people!

Going back to the theory that movie-goers respond to the poster design (in particular, the number of people in the poster), the analyst now realizes that he or she has a huge hole in the dataset. Which of these posters did the movie-goer see? Did IMDB know which poster was seen the most number of times?

Thus, not only are the counts subjective and imprecise, it is not even clear we are analyzing the right posters.

***

Once I led the students down this path, almost everyone decided to drop this variable from the dataset.

On my sister blog last week, I wrote about how to screw up a column chart. The chart designer apparently wanted to explore whether Rotten Tomato Scores are correlated with box office success, and whether the running time of a movie is correlated with box office success. In either case, the set of movies is a small one, those directed by Chris Nolan. Here is a better view of the data:

There were a few questions on Twitter, which I will address in this post. Someone complained about the horizontal axis thinking that year data is continuous and the axis should not be discrete. That would be true if the data is truly continuous. The way I interpret the year data here is ordinal: Chris Nolan only makes a movie once every 1 to 3 years; his career is also developing during this period of time; so I think of the horizontal axis as ordinal, that is, his first film, his second film, etc.

Another Twitter user tried a scatter plot. It's very satisfying to see the following chart. It shows a strong positive correlation between the running time of a movie and the box office receipts.

In fact, a hockey-stick line fits the data even better. This implies a multiplicative relationship between running time and box office receipts. We can fit this by first taking the logarithm of box office receipts. So, the following chart showing how good this fit is:

If you run a regression, the R-squared is 90 percent, and the effect of running time is extremely significant (p < 0.0002). So we have proved that Chris Nolan should make longer movies because the longer the running time of his movies, the bigger the box office.

You might have noticed that both running time and box office numbers have gone up over time. (That is to say, running time and box office numbers are highly correlated.) Do you think that is because moviegoers are motivated to see longer films, or because movies are just getting longer?

And these items:

1) Chris Nolan's career experienced a hockey-stick growth during this time.

2) Movies have become longer and longer in general.

3) Rotten Tomatoes also experienced a hockey-stick growth in users during this time period. In January 2000, the site had 250K visitors, at which point the founders said they "started in earnest as a company". (link) Today, according to Wikipedia, they had almost 20 million monthly visitors. In other words, several of the early movies by Chris Nolan came out before RT came into its own.

There was a lively, fun discussion after my talk yesterday night in New York. For those who couldn't attend, let me review some of the conversation. Here you go:

Q: Tell us more about the chapter in Numbersense titled "Are They New Jobs When No One Can Apply?" Related to economic data, can you talk about the idea that we still need to import foreign workers because there aren't enough skilled labor available domestically?

A: That chapter is really about the use of statistical adjustments. Some people say they are skeptical of the official unemployment numbers, the seasonally adjusted numbers, because it includes jobs that are made up--there isn't a job posting that you can apply to. These people think that raw data is always better than adjusted data, because statisticians are doing naughty things to the data. In reality, it's the opposite. Raw data is bad and adjusted data is better.

In terms of the skilled labor issue, this is related to the argument about structural unemployment, the claim that the unemployment problem cannot be solved because people need retraining.The proposed solution is to send more people to college. This is usually supported by data showing that the unemployment rate of college graduates is much lower than that of non college graduates. However, there is a cohort problem here. If we drill down to recent college graduates, there are many who aren't finding jobs. So if we make more college graduates, there will be even more competition for those jobs, and it will suppress the income of those jobs.

Unemployment rate and economic indicators are complex things. The chapter [Chapters 6-7 in Numbersense] goes into quite a bit of detail to address the question of why they seem out of touch with reality.

***

Q: How is Big Data going to impact the movie industry? Can you predict what people will watch?

A: To some extent. But remember the example of the Target model predicting pregnancy I just described [Chapter 5 in Numbersense]. If you use a statistical criterion, like the predictive lift, you can congratulate yourself on having built a great model. But then if you apply a more common-sense metric, like the hit rate, you notice the really high proportion of errors even when the lift is high.

In social-science problems like this, I always advocate the combination of quantitative and qualitative data. It is not sufficient to just use frequencies. They don't tell you why someone watched something. [Then I went on a rant about why correlation is still not causation in the Big Data era.]

***

Q: What can you say about how the Democratic and Republican
parties used Big Data in the recent presidential elections? Can you talk
about the likely voter models?

A: Somewhere in the book (Prologue), I discussed the likely voter
models. I used this to illustrate the point I was making earlier about
understanding what part of the argument is data-driven and which part is
theory-driven. In the book, I compared the work of Nate Silver and the
guy who created the UnSkewedPolls.com website. They both used the same
data set of the poll results but came out with really different
projections.

This points to the important idea that any data analysis involves
theory--you can't avoid it. There is this myth out there that says when
you have loads of data, the data itself is objective and stops all
debate. That's so far from the truth. Anyone who has worked with data
knows that you have to make assumptions. In the case of likely voters,
the Republican guy made assumptions about how their party members would
be motivated to come out to vote. This is a necessary assumption because
if you just use the data, you will always predict that the future is
the past, and you will never be able to predict a surge in interest. In
this case, the theory part of the analysis turned out to be
spectacularly wrong.

***

Q: Big Data is impacting the education sector in many ways, such as Value Added Models being applied to evaluate teachers. What do you think of these models?

Education is a good example of Big Data. It fits the five criteria that I just laid out for what is a Big Data study. The value-added models for instance are silly, partly because the data is co-opted--test scores are originally intended to measure student performance but then they're being used to evaluate teachers. Also, the whole pay-for-performance concept doesn't work when you can't measure performance well; it backfires and causes rampant cheating. The first chapter of Numbersense was originally going to be about how teachers and principals all over the country were turned into cheaters; but then I ended up writing about a different kind of fraud--how schools game the school rankings.

***

Q: On one of your slides, you mentioned eHarmony trying to port its algorithm from matchmaking to hooking up employees and employers. That sounds like a promising thing. What do you think of it?

A: Just like all predictive models, you have to be careful in understanding how accurate the predictions are. The media do a poor job of reporting the accuracy. The eHarmony model can be evaluated in the same way that I evaluated the Target model for predicting pregnancy. You have to think about both false positives and false negatives, and the fact that those two trade off each other. In the media, you often read about one of those two metrics, and they hide the other one.

A: Not really. However, they did use statistics to come to that conclusion. To their credit, the people in the Gates Foundation ran some rigorous analyses comparing the small schools they funded with larger schools. And they learned that the small schools were not better than the larger schools, and in some cases, even worse. So they started to spend the money on other things like curriculum development.

***

Q: Do you have an opinion on the recent Malcolm Gladwell piece about doping in sports?

A: I do have reactions to the Gladwell piece. Those of you who came
to this talk three years ago for my other book remember that anti-doping
testing was one of the main examples I spoke about. In fact, other
people have requested that I write about Gladwell's assertion that
doping should be made legal. I will be putting up my response hopefully
in the next few days. Look for it on my blog.

***

I'm sure there were more questions. So, please accept my apologies to any attendee whose question I could not recall. Thanks for attending.

When you hear about Big Data, you almost always hear about
the supply side: Behold the data in un-pronounceable units of bytes! Admire the
new science inspired by all the data! Missing from this narrative is the
consumption side. A direct consequence of Big Data will be the explosion of dataanalyses—there
will be more people producing more data analyses more quickly. This will be a
world of confusing and contradictory findings.

In my new book, Numbersense, I argue that the ability to analyze and interpret these data
analyses will give one a competitive edge in this world of Big Data.

Numbersense is the noise you hear in your head when you see
bad data or bad analysis. After years of managing teams of data analysts, I’ve
learned that what distinguishes the best from the merely good is not math
degrees or computer skills; it is numbersense.

Numbersense is an intangible quality that you can’t teach in
a classroom. The best way to pick it up is by learning from people who have it. For this blog post, I selected two great analyses of data
analyses that have impressed me recently. These are highly instructive examples.

***

Eating red meat makes us die sooner! Zoë Harcombe didn’t think so.

In March, 2013, nutritional epidemiologists from Harvard University
circulated new research linking red meat consumption with increased risk of
death. All major mass media outlets ran the story, with headlines such as “Risks:
More Red Meat, More Mortality.” (link) This high-class
treatment is typical, given Harvard’s brand, the reputation of the research
team, and the pending publication in a peer-reviewed journal. Readers are told
that the finding came from large studies with hundreds of thousands of
subjects, and that the researchers “controlled for” other potential causes of
death.

Zoë
Harcombe, an author of books on obesity, was one of the readers who did not buy
the story. She heard that noise in her head when she reviewed the Harvard
study. In a blog post, titled “Red meat & Mortality & the Usual Bad
Science,” (link)
Harcombe outlined how she determined the research was junk science.

She
knows this type of research methodology rarely if ever delivers conclusive
evidence of causation. Then, she found support from a data table included in
the research paper. The table shows that the cohort of people who report eating
more red meat also report higher levels of unhealthy behaviors, including more
smoking, more drinking, and less exercise. Thus, the increased risk of death
observed in the study could have been explained by factors other than red meat
consumption.

For a full dissection of Harcome’s amazing post, please click here.
Chapter 2 of Numbersense looks at the
quality of data analyses of the obesity crisis.

In February, 2013, Netflix, ever the
media darling, premiered House of Cards,
a re-make of the successful British television show, their second foray into
producing original content for its tens of millions of subscribers. Netflix
executives regaled the press with stories of how Big Data analysis took the
risk out of their $100 million decision.

Andrew Leonard, the technology
reporter for Salon.com, gobbled up the Netflix story, even interpreting it as a
“symptom of a society-wide shift.” (link) Like
other news analysts, Leonard was convinced by the “pure geek wizardry” used to analyze
mountains of data collected from Netflix customers. The machine, we’re told,
decided that David Fincher should be the director and Kevin Spacey, the star.
From here, it is a short trip to the lala land of viewers as puppets with
machines as the overlord.

This analysis aroused the skeptic
in Felix Salmon, the finance blogger for Reuters. In his blog post, “Why the Quants
Won’t Take Over Hollywood,” (link) Salmon
raised other factors that affect the box office, including billions spent on
marketing and publicity, the quality of the writing, the sociopolitical
climate, the complex relationship between originals and remakes, and the poor
track record of predictive modeling in Hollywood. On this last point, Salmon exhibits
a keen sense of the limitations of science, speaking of
“impossible-to-formulate cocktail of creativity, inspiration, teamwork, and luck.”

Chapters 4 and 5 of Numbersense explains how you should
judge predictive models used by marketers.

***

When their respective blog posts surfaced, Harcombe and
Salmon were lone voices vetting carefully the claims based on other people’s data
analyses. Their well-honed numbersense allows them to stand firm in the face of
mountains of data, worship of high science, formidable-sounding technical
jargon, and academic reputations. The problems with the original research are
far from obvious. The point is not to debunk these studies—no data analysis is
ever infallible—but to figure out for yourself what is credible, and what is
junk.

In a study of 2000 American adults, 12 percent confessed to watching
ahead on TV shows they were supposed to save to watch with their
partners. Ten percent admitted to being the victim of Netflix adultery,
which means either 2 percent are blissfully unaware of their partners’
indiscretions, or the cheaters are hitting multiple victims.

This last sentence is "story time". There is nothing in the study to prove or disprove this story. A third explanation -- which is more likely than the other two -- is that the 2 percent difference is pure noise. The margin of error of this study is about 1.3 percent plus or minus around each of those percentages.

The conclusion also contains a number of suspicious elements. First, it's unclear how exactly 2000 people responded. Second, not all adults have partners so either the 2000 people were pre-screened and not just any adult, or the percentages are biased by unattached people who could not have cheated.

Thirdly, Maureen made an assumption that both partners of each couple responded, which is highly unlikely. There are really eight types of couples: both cheated, which leads to four types depending on whether each partner confessed to the other; the first partner cheated or the second partner cheated, each of which leads to two types depending on whether there was a confession. Each couple may have returned one or two surveys. Given such complexity, the two percentages from the survey cannot be simply interpreted.