Audiences and creators know that on one level or another, there’s an inherent gender bias in the movie business — whether it’s the disproportionately low number of films with female leads, the process of pigeonholing actresses into predefined roles (action chick, romantic interest, middle-aged mother, etc.), or the lack of serious character development for women on screen compared to their male counterparts. What’s challenging is quantifying this dysfunction, putting numbers to a trend that is — at least anecdotally — a pretty clear reality.

One of the most enduring tools to measure Hollywood’s gender bias is a test originally promoted by cartoonist Alison Bechdel in a 1985 strip from her “Dykes To Watch Out For” series. Bechdel said that if a movie can satisfy three criteria — there are at least two named women in the picture, they have a conversation with each other at some point, and that conversation isn’t about a male character — then it passes “The Rule,” whereby female characters are allocated a bare minimum of depth. You can see a copy of that strip here.

Using Bechdel test data, we analyzed 1,615 films released from 1990 to 2013 to examine the relationship between the prominence of women in a film and that film’s budget and gross profits. We found that the median budget of movies that passed the test — those that featured a conversation between two women about something other than a man — was substantially lower than the median budget of all films in the sample. What’s more, we found that the data doesn’t appear to support the persistent Hollywood belief that films featuring women do worse at the box office. Instead, we found evidence that films that feature meaningful interactions between women may in fact have a better return on investment, overall, than films that don’t.

The test

You’d be hard pressed to think of a single film that doesn’t have a scene where two men have a conversation that isn’t about a woman. Plots need to advance, after all. But it’s remarkable how many iconic films disastrously fail the Bechdel test.

Let’s look at several recent films to explain how the test works. “The Hobbit: An Unexpected Journey” contains fewer than two named women and thus fails the test on the first criterion. And while “The Avengers” has at least two women in the film — Pepper Potts, Black Widow and Agent Hill come to mind — they don’t speak to each other, so it fails on the second criterion. And while the wives of Doug and Stu are both named and do indeed have a conversation in “The Hangover Part 3,” it’s about Alan, Zach Galifianakis’ character, so it fails on the third criterion. The animated film, “Frozen,”1 passes the test since two central female characters, Anna and Elsa, discuss the isolationist policies of Arendelle, plans to build a snowman, and the time Elsa locked their civilization in an eternal winter.

In a larger sample of 1,794 movies released from 1970 to 2013, we found that only half had at least one scene in which women talked to each other about something other than a man.

It’s not hard to imagine that movies passing the Bechdel test are probably doing a better job reflecting society as it currently exists, where half the planet is comprised of women who have fulfilling conversations about things other than men. But the test isn’t perfect. Take “American Hustle.” The film passes the test, but only barely, thanks to a single scene where a con artist’s wife, played by Jennifer Lawrence, discusses nail polish with a politician’s wife, played by Elisabeth Röhm. On the other hand, “Gravity” — which is dominated almost entirely by Sandra Bullock, in a highly praised performance — fails the test, as Bullock never speaks to another woman in the film. Since it’s difficult to argue that “American Hustle” is more progressive than “Gravity” when it comes to the portrayal of women, we should be clear about the limits of the test. The Bechdel test isn’t measuring whether a film is a model of gender equality. It doesn’t certify that a movie is “good” when it comes to integrating women. And passing it doesn’t mean that female characters are well written, play crucial roles in the plot or display meaningful depth of character. But it’s the best test on gender equity in film we have — and, perhaps more important for our purposes, the only test we have data on.

The data

Our analysis relies on two data sets: BechdelTest.com and The-Numbers.com. The site BechdelTest.com is operated by committed moviegoers who analyze films and ascertain if they pass the Bechdel test. The site has detailed, coded information for about 5,000 films.

To find financial information on these films, we went to The-Numbers.com, a leading site for box office and budget data. It inventories financial information for roughly 4,500 films.

The intersection of The-Numbers and BechdelTest was a set of 1,615 films released between 1990 and 2013. When considering the financial information, we adjusted all numbers for inflation, using 2013 dollars. While hardly a complete record of contemporary films, this gave us a sample that has both rigorous evaluations of female character involvement as well as the most accurate financial data available online.

Our findings

One difference between what we found in our broader sample of 1,794 films released between 1970 and 2013 and the analysis at BechdelTest.com of about 5,000 films is that BechdelTest.com shows 56 percent of films passing the test, compared to our 53 percent. The site’s slightly higher percentage could be a result of inherent biases in what contributors watch and evaluate. As with any film database that relies on user submissions to compile its film list, it’s in many ways a product of its user base, which in this case is feminist-leaning. People inclined to watch films with a realistic depiction of women will likely add those movies to the set, skewing the value upwards. Since our set is a subset of the BechdelTest.com’s entire set, this potential bias is important to keep in mind.

Looking more closely at our smaller sample and how films performed on the Bechdel test over time, we found — not unexpectedly — that a greater share of films are passing today compared to 1970, but that the level has plateaued in the past two decades. There’s a surprisingly large contingent of films — about 10 percent — that have either no or only one named female character in their casts. And the number of films decisively passing the test remains below 50 percent. This chart shows the breakdown of our sample of films (“dubious” means that some BechdelTest.com contributors were skeptical about whether the films in question passed the test2):

In order to get a consistent look at budget data, we’re going to focus on films released from 1990 to 2013, since the data has significantly more depth since then.

We ran a statistical test analyzing the inflation-adjusted median budgets of films, and found that films passing the Bechdel Test had a median budget that was 16 percent lower than the median budget of all films in the set.3 Smaller budgets constrain filmmakers in production, in landing talented performers and in promoting and advertising their films. Funding can make a big difference when it comes to how a movie performs at the box office, and how many people see it.4

The funding distinctions were even more remarkable when comparing the set of films that passed and the set of films that failed. The median budget of a film that failed the test was $48.4 million. The median budget of a film that passed was $31.7 million, or 35 percent less.

Looking at the above chart, it’s remarkable to see that films that had at least two women in them got higher budgets than films that didn’t, but only when those women never spoke to one another. Think about the kind of movie where there are multiple women but they don’t talk to each other. These women are often playing the wives of leading men, damsels in distress or side characters used to help explain the action.

These median budget numbers suggest that Hollywood puts more money behind male-only films than films in which women talk to each other.

Some recent films that completely failed the test were mainly war movies (“Act of Valor”), one-dimensional children’s movies (“Frankenweenie” and “Jack The Giant Slayer”) 5 or films with minimalist casts (“Gravity” and “All Is Lost”). You’d think that any movie with an ensemble cast stands a stronger chance of satisfying the first criterion of having at least two named women characters.

In trying to explain the difference in funding levels between movies that pass the Bechdel test and your typical movie, Hollywood insiders referred to a culture in which men control the creative process and the purse strings, and a pervasive belief that audiences — both in the U.S. and internationally — just don’t like films with strong female characters. Yet we found no evidence in the data to support the idea that films with women perform any worse at the box office than films without them, and what’s more, films with women appeared to outperform expectations.

What gives?

To find out why films involving women in meaningful roles were getting less funding than movies that sidelined them, I reached out to several Hollywood producers, journalists and entrepreneurs. They pointed to the scant numbers of women in writing, directing, producing and financing roles in Hollywood; the fact that foreign pre-sales are crucial in paying for the vast majority of films and the belief, among investors, that movies featuring women do not “travel” well internationally; and the persistent assumption that American audiences are more likely to prefer male-anchored films, especially in the lucrative action genre.

In the top 100 grossing films of 2012, women accounted for 4.1 percent of directors, 12.2 percent of writers and 20 percent of producers, according to a 2013 study by Stacy Smith, an associate professor at the Annenberg School for Communication and Journalism at the University of Southern California. Of 4,475 speaking roles in those films, 28.4 percent were women. Smith says when more women were involved in the production of a film, it was more likely to have female cast members. In short, when one gender dominates the creative process for a picture, that comes out on the screen.

In recent years, as the DVD market has bottomed out and the U.S. market for theaters has become saturated, Hollywood has pivoted its strategy to emphasize international distribution. And since the vast majority of films are now financed through the independent market — not by major studios — an increasingly important source of funding is foreign pre-sales, said Cathy Schulman, Oscar-winning producer of “Crash” (2004) and president of Women In Film, an advocacy organization. What this means is that in advance of filming, producers will sell the rights to distribute their films to international buyers. Producers say they stand a better chance of selling these rights at a high price if the movies are attached to internationally popular male actors.

“Movies that are female-driven do not travel,” said Krista Smith, West Coast editor of Vanity Fair, describing the broader sentiment in Hollywood. There are almost no women who have sales value in multiple international territories, maybe with the exception of Sandra Bullock, she said.

Finally, there’s a wide-ranging perception in Hollywood that audiences — in the U.S. and abroad — simply don’t care for women in leading roles and that movies for and about men are more likely to have better cross-market appeal than movies about women. The theory is that “women will go to a ‘guy’s movie’ more easily than guys will go to a ‘woman’s movie,’” said Michael Shamberg, who produced “Pulp Fiction” (1994), “Django Unchained”(2012) and “Garden State” (2004).6

This assumption is up for debate; we found that films that pass the Bechdel test tend to do better dollar for dollar than those that don’t — even internationally.

Better return on investment

We did a statistical analysis of films to test two claims: first, that films that pass the Bechdel test — featuring women in stronger roles — see a lower return on investment, and second, that they see lower gross profits. We found no evidence to support either claim.

On the first test, we ran a regression to find out if passing the Bechdel test corresponded to lower return on investment. Controlling for the movie’s budget, which has a negative and significant relationship to a film’s return on investment,7 passing the Bechdel test had no effect on the film’s return on investment. In other words, adding women to a film’s cast didn’t hurt its investors’ returns, contrary to what Hollywood investors seem to believe.

The total median gross return on investment for a film that passed the Bechdel test was $2.68 for each dollar spent. The total median gross return on investment for films that failed was only $2.45 for each dollar spent. And while this might be a side effect of films with lower budgets tending to have higher returns on investment than films with higher budgets, it’s still a strong indicator that films with women in somewhat prominent roles are performing well.

Since Hollywood believes that international markets don’t want to see women in film, we also broke down the median return on investment for films based on domestic (U.S. and Canada) and international box office numbers. We found that Bechdel-passing films still have comparable returns on investment when the movies “travel.”

On the second test, we ran a regression to find out if passing the Bechdel test corresponded to having lower gross profits — domestic and international. Also controlling for the movie’s budget, which has a positive and significant relationship to a film’s gross profits,8 once again passing the Bechdel test did not have any effect on a film’s gross profits.

What next?

Is there any hope of change in Hollywood? Despite the fact that our numbers challenge these preconceived ideas, they’re pretty entrenched. “In Hollywood, there are still a lot of dinosaurs behind the desks,” said Suzanne Todd, who produced “Alice in Wonderland” (2010), the “Austin Powers” films, “Memento” (2000) and “Across the Universe” (2007).

More movies are passing the Bechdel test than before, but recall that the level has flatlined at about half over the last 20 years, and women don’t make up any more than 20 percent of producers, directors and writers across the board.

Todd is bearish on Hollywood’s advancement. She says it’s still a struggle to get films with female protagonists made. It was only in 2007 that Warner Brothers chairman Jeff Robinov decreed that his studio would no longer make films with female leads. And while franchises anchored by young women protagonists such as “The Hunger Games” and “Twilight” are greeted with industry-wide praise, Todd says she believes their success is often credited to the screenwriters, directors and production teams instead of to the leading women.

But there are anecdotal signs that there’s a shift in thinking when it comes to movies featuring women and female relationships. Recently, Hollywood has been able to boast about the success of female-dominated films in the marketplace. Shamberg said he believes younger viewers tend to have a “more equal view of men and women,” citing the success of “Twilight” and “The Hunger Games” and pointing to the production of “Divergent.”

In addition to these young adult franchises, Disney’s female-centric “Frozen” was also cited as another leading indicator that Hollywood is moving beyond the idea that movies anchored by women don’t perform well. “Girls have gone to see it every weekend,” said Krista Smith, propelling it to gross more than $1 billion and become the most successful film in Disney Animation Studios’ history. (FiveThirtyEight’s parent company, ESPN, is owned by Disney.)

Hollywood is the business of making money. Since our data demonstrates that films containing meaningful interactions between women do better at the box office than movies that don’t, it may be only a matter of time before the data of dollars and cents overcomes the rumors and prejudices defining the budgeting process of films for, by and about women.

CORRECTION (Aug. 18, 9:58 a.m.): A previous version of this article misstated the title of a film. It is “Divergent,” not “Divergence.”

Footnotes

Frozen made headlines recently as the highest-grossing animated film of all time. Box office revenues, however, are not adjusted for inflation. ^

In our analysis, we include films labeled “dubious” as passing the Bechdel test. Their inclusion did not have a significant effect on our findings. ^

We rely on median, as opposed to average, budgets to minimize the effect of outliers, or in this case, huge blockbusters whose budgets are orders of magnitude larger than the typical movie’s budget. ^

Clearly genre plays a role in the budgeting process, where typically male-driven action and thriller films tend to be financed at a much higher rate. Some industry insiders argue that drama and romantic comedies, which tend to feature women, are cheaper to produce. ^

The fact that many children’s movies aren’t passing the Bechdel test is, in itself, a huge issue. Here’s some reading on this. ^

Additionally, Shamberg said, “Statistically, teenagers go to more movies than adults.” A 2014 MPAA study found that viewers ages 12 to 17 and 18 to 24 went to the cinema more often than any other age group. Vanity Fair’s Smith said a film that’s popular among teens can become a financial hit because teens are more likely to see a film multiple times in theaters. The perception that teens want action movies could be driving the differences in budgets. ^

The bigger the budget, the more a film makes in gross profits; however, each new dollar that you spend making a movie also lowers its return on investment. ^

The bigger the budget, the more a film tends to spend on advertising; advertising has a strong correlation with box office revenues. This can lead to a self-fulfilling prophecy based on the preconceived attitudes of the distributors, according to Franklin Leonard, the founder of the Black List, a script aggregation and rating website for Hollywood producers. “Distributors generally increase their marketing budgets relative to how well they think the movie can do, so inevitably, films with larger marketing budgets tend to do well.” ^

Interactives

Sign up for our newsletters to keep up with our favorite articles, charts and regressions.
We have three on offer: a curated digest of the best of FiveThirtyEight from the past week; The Week In Data, our weekly look at the best data journalism from around the web; and Significant Digits, our roundup of numbers in the news.
Enter your email below, and we’ll be in touch.