Watching the Watchers: Gauging Audience Response

NPAC session description, "Stop Taking Attendance and Start Measuring the Intrinsic Impact of Your Programs": What really happens when the lights go down and the curtain rises? Most
arts groups do a great job of tracking attendance and revenues, but
these are poor indicators of impact. Aside from the buzz in the lobby,
is it possible to define - and even measure - how audiences are
transformed? If you had this information, what would you do with it?
Results of a groundbreaking new study commissioned by the Major
University Presenters consortium in the U.S. suggests that intrinsic
impacts can, in fact, be assessed using a simple questionnaire. Alan
Brown, who directed the study, will discuss the results of the
research, which involved pre- and post-performance surveys at 19
performances by a wide range of music, theater and dance artists...

While I can't speak to the specifics of the study in question, I generally think that surveys measuring audience response are a bad idea. I care very deeply about what my audience thinks or feels, but I don't feel that surveys are the best way to assess this, and so don't use them. If the theater wants them, I consent, but I don't read them. This is not because I am a snob who is disinterested in what my audience thinks - on the contrary, I care very much - but because I think our contemporary culture has a weird fetish for quantifying everything, and something so delicate and ineffable as the relationship between artist and viewer can't even really be expressed verbally, let alone numerically. I am, in many cases, a believer inthe wisdom of crowds and a fan of most open-source projects, but theater isn't computer programming or the collective hive-mind of Wikipedia. I find it much more instructive, actually, to watch an audience watch my work (as was easy to do at the Denver Center's in-the-round Space Theater in 2007), a technique recommended by the filmmaker Francois Truffaut, among others. Collectively, an audience is very intelligent, but not necessarily in a way that individual members can articulate - often I can better tell whether or not a play is working by observing body language. When are people laughing, crying, shifting, on the edge of their seats, dozing off, walking out?

And who are they specifically, and when do I want or not want them to be doing each of these? This tells me much more than most of the feedback I get at "talk-backs," which is usually more about giving the audience a greater sense of involvement (a perfectly laudable goal in itself) than about soliciting "notes." Indeed, I am often very eager to interact with my audience, and make myself very easy to contact via email, Blogger, MySpace, and Facebook. But this is not because I intend to use feedback to make changes to my work, but because increasingly, people see their relationships with artists as interactive (I have often corresponded with my favorite novelists and rock musicians myself), and because I deliberately set out to write plays that foster arguments and conversations. I am very happy when these discussions take place, but I see them as parallel to the artistic experience, not part of it.

With all due respect to Mr. Yoshitomi and Mr. Brown, with whose methods I am not at all familiar, I feel that "Measuring the Intrinsic Impact" of any work of art is an idea with potentially disastrous results for the creative process generally. To be fair, I am not in a position to judge their study - only the underlying impulse, which I have encountered in many resource-strapped arts institutions who seek to use private-sector methods to improve their lots. This approach disregards the many flaws in corporate culture. For some time, the film industry has used audience surveys in preliminary screenings in an attempt to predict audience response, and to take the uncertainty out of what is a famously volatile marketplace. While I can't cite any precise data on the failure or success of these methods, and of course can't speak to their similarity to the study in question, I have heard anecdote after anecdote about its failure as both an aesthetic and a profit-making strategy. Frequently, test audiences will have an initial negative response to a fine movie, and producers will demand changes that water it down and make it even less appealing to audiences. Alternately, many recent screenwriters and filmmakers (The Wedding Crashers' Steve Faber and John Fisher and Hostel's Eli Roth, to cite two examples) have stated publicly in interviews that they received negative audience evaluations, fought to preserve their respective artistic visions, and went on to have great success at the box office in spite of the initial test audience reaction.

Of course producers and presenters would like to be able to predict and manage audience and critical response to what they put on out, as arts presenting is a notoriously stressful and erratic business, but I don't think this can be done without severely compromising the integrity of the art. Risk is, in most cases, the entire point. Of course, arts presenters could probably predict, with some degree of accuracy, the acts or exhibits that would be most popular, but this would, in all likelihood, lead to a steep decline in "difficult" but ultimately rewarding works of art, and the rise of gimmick-driven art, and ultimately of arts institutions as weak imitators of the multiplex, the mall, the computer, and the television set - a competition which they would, most likely, lose. Why would I go to the trouble of going to a theater or museum if I won't be offered an experience that is fundamentally different from what I can get at home, on TV or on the internet, often for significantly less money? Even assuming that the questions being asked related to being "moved" or "affected" by the work (as opposed to the simple thumbs-ups or thumbs-downs of Hollywood test surveys), this is still a flattening and oversimplification of something that is, when it works, impossible to articulate in any coherent way.

The best art polarizes as much as it unites. Most art that seeks to please everyone is doomed to failure, mediocrity or, at best, a sort of temporary popularity. This is not to say that genre art can never be good - I'm a fan of The Wire, Philip K. Dick, sketch comedy, comic books and pop music as much as I am of, say, Lawrence Shainberg's novel Crust, the poetry of John Ashbery, opera, or performance art, and often the two categories are not mutually exclusive (note the references to Flann O'Brien's The Third Policeman on the TV show Lost, an incident that caused the postmodern novel to sell more in the last year or so than it did in the entire 20th Century). What I object to is the attempt to domesticate and commodify a process that tends to sour at its very contact with such concepts.

It is also worth mentioning that even the most educated among us are rarely able to articulate why we like or don't like a work or art. Highly specialized languages have developed around criticism and dramaturgy, not because these are pursuits exclusively practiced by elites (though they sometimes are), but because it's so difficult to put these thoughts and feelings into words. I would also point out that the objectives of a work of art are often counter to what most of us have come to expect in a consumer society. That is, our society is built around the imperative to enjoy, and the merits of a thing, any thing, are often judged on how well it fulfills that imperative.

There is nothing intrinsically wrong with enjoyment per se - I like having a good time as much as anyone. However, in order for a work of art to be successful, it needs to pull as well as push - often, the goal is to anger, disturb, or even to deliberately bore or tax the audience, viewer, or listener in pursuit of some larger goal. The obvious response to this is the one we have heard throughout the history of modern art - "it's pretentious bullshit," "my kid could paint that," etc. And yes, that is often the case. However, even work that enlightens or entertains often needs to mystify, or to defer pleasure, in order to be successful. Of course what we say we want is the thrill, the laugh, the cheer, the beautiful sound or object, but most often those moments need to be surrounded by something else, or the experience is meaningless - art as a series of positive stimuli that zaps our animal brains in a pleasing way, but offers little else.

As an audience member (or viewer, or listener, or whatever) I like to work at it, and I want to make my audiences work too. I was often told that my play 1001, which opened in Denver to critical and commercial success and positive audience response, should never have made it past the barrage of arts administrators who should have tried to get me to tame it, simplify it, dumb it down and make it more like a typical American play - that is, a mildly comic love story with some digestible, moderately liberal political themes. Indeed, some people did try to get me to do this (to his great credit, the Denver Center's artistic director Kent Thompson was not one of them), but I simply refused, and I have never once regretted my decision.

The monologist Mike Daisey has a recent show titled How Theater Failed America. One of the factors he blames is the increased institutionalization and bureaucratization of not-for-profit theater. He's got a point. Imitating Hollywood has done little to make theater more relevant or central to the culture at large. Perhaps theater's centrality, like that of literature, is a product of a bygone era, when people still believed in the concept of the commons, and it can never be regained. I choose to believe that this is not the case, but even if it was, theater would do better to adjust to its new place in the scheme of things and try to make great work for the people who love it (and use smart marketing to make it easier for others to discover it). Imitating Hollywood even more than we do now (without, I should note, paying the talent anywhere near Hollywood levels) would surely be disastrous. I can't speak for other playwrights, but the ability to control my own work is one of the very few things that keeps me writing plays. Having to tailor my work in order to score more highly on an abstracted, numerical expression of audience response would probably drive me out of the theater for good.

To hear more from Jason Grote, visit his blog. To
learn more about NPAC sessions such as "Stop Taking Attendance and
Start Measuring the Intrinsic Impact of Your Programs", visit the website.

Tags:

9 Comments

Jason -- You are writing a blog, not publishing an essay. If you feel that critical comments are a breach of etiquette, then for God's sake turn off the comments box and we'd all be happier. If not, then expect that others will say things about what you write, and that those things may not be 100% fulsome praise. So no, I am not your editor, nor am I paying you, but I am a blog reader, and part of blogging is participating in the comments. So I won't apologize for my critique.

I'm glad you read the summary, and not surprised you didn't find it helpful, given your stated prejudice against such studies. Yes, the magic of theatre is fantastic, and to some extent what happens in an audience is inexplicable. But to object to any attempt to delve into the subject out of "respect" for the magic strikes me as similar to those who objected to Galileo's research because it failed to "respect" God. A study can ultimately understand only a part of what happens when an audience meets a work of art, not every aspect. There is still a place for magic, even if we understand more fully what is happening.

The irony is that, when I wrote about this study, I did so the post after I wrote about the success of "1001" in Denver, and I used the findings as support for the viability of producing new plays instead of simply re-doing classics. The audience response to "Macbeth" seemed to suggest that familiarity might lead to lack of engagement. But all you want to do is play the Wizard of Oz complaining because Toto is pulling away the curtain and destroying the illusion. Too bad.

Actually, I lied: I do have one more thing to say. While it might be generally laudable of the WolfBrown folks to attempt to measure the intrinsic value of the art (rather than expressing it monetarily), I still feel as if there is something really insidious about quantifying something so magical and ineffable. Again, this is specific to my position, not just as a playwright, but as the specific playwright that I am, but there you have it, folks.

Scott, are you serious about that analogy? I never trashed the study in question. I never said their study was about post-show discussions. It would be more accurate to ask, what if you had been invited to comment on a website about a festival that included 1001, and you admitted that you haven't read or seen the show, and weren't commenting on it, but were generally critical of avant-garde or nonlinear theater, and delineated your reasons why, based on other experiences you've had. I wouldn't agree, but it's a perfectly valid point of view.

If you do this let me know and I will link it from my blog. I think it's a hilarious idea, seriously. I might even do it myself.

Back to the topic: I did read the abstract at your urging, and frankly I'm even less inclined to find the study valuable now. Would it be preferable if I gave a point-by-point refutation of the thing in a way that would be probably be more insulting and damaging to its authors? Then, no doubt, the argument against me would be something along the lines of the notion that I'm unqualified to comment on survey methodology because I'm just a playwright.

I was asked to comment on a general theme from my point of view as an artist. Who (besides you) ever said anything otherwise? I happen to have a bias against surveys as a way of determining audience response because my extensive experience with such things, as I've laid out extensively in the original post. I have been utterly frank about my biases. If you want to key in on marginal details in my original piece and twist them around to claim that I am somehow conducting a hatchet job on these guys, I can't stop you, but I think it's pretty ridiculous.

You are free to imagine yourself as my editor, but unless you we are corresponding from two parallel universes, that's just not ever going to be the case. No one asked me to analyze the study, I don't want to (in fact, I would like the time that I spent looking at it back), and I bent over backwards to make it clear that I wasn't. The only connection between my post and the study is a thematic one. You can jump up and down and swear at me all you want, but do you really expect me to write something because you said I should? Are you going to pay me?

I'm not really interested in getting into one of those endless, angels-on-the-head-of-a-pin internet arguments anyway, so the last word is yours if you want it. I'm not taking any more bait.

Let's put the shoe on the other foot, Jason. I write a blog post about one of your plays that I haven't seen, haven't read, but I have read the info in the Samuel French of Dramatists Play Service catalog. And from reading that blurb, I dismiss the artistic value of your play. Now, how long would it take for every blogger in the theatrosphere to converge and tear me limb from limb? I seem to remember George Hunka being dismantled for doing something similar after having walked out of a play at intermission, admitted he did so, and writing a scathing review of the play nonetheless. Imagine if he had written the same thing after having only read a press release!

The thing that artists often forget is that there are people behind academic studies, just like there are artists behind plays. This particular study probably took several years to do, and represents an important part of the researchers' work. It deserves to be taken seriously.

The direction this discussion has gone on this site makes us look bad. The study isn't about post-show discussions, for crying out loud. Read the damn report and think about it so that you can have an informed opinion. Simply admitting that you don't really know what you're talking about isn't enough. Make the effort -- the study isn't that long, and the summary is even shorter.

David, I've heard about your program and I can't make it this month but I think that it sounds like a terrific idea. It reminds me of the term used by WFMU station manager Ken Freedman to describe WFMU's semi-curated free music archive: Web 1.5. That is, it's participatory like Web 2.0, but lightly curated and moderated to make sure it doesn't get inundated with garbage.

So Scott, you're crushed because I didn't misrepresent myself? The fine people at NPAC didn't ask me to comment on this study or presentation, and the quotes you put in italics were intended to make that absolutely clear. Had they, I would have declined. I'm not a marketer or a producer. I was asked to write a blog entry somehow related to the general topic of feedback and audience response, something I do have experience with as a playwright. I don't know what the "gotcha" stuff is all about, but I stand by everything I say there. If you'd like to blog about the particular study, go for it, but I never claimed to be doing that. I will say, though, that nothing you mention in your summary of the study really changes my mind, and theaters using this type of data to make programming and development decisions (not just marketing ones) is just as likely as anything.

Why do I worry that what "transform" means here is less spiritual than "transformed into repeat happy customers!"

This is a really salient point about talkbacks, by the way. (And of course "surveys" only bring to mind "test screenings" and "Fatal Attraction"-like surgical rewrites.) As someone who has moderated quite a few talkbacks in my time, I want to go out on a limb say that they more often than not, suck. First, I'm also struck by how often audience members take it upon themselves to indeed "give notes" to the playwright or artists--unsolicited--as if they're playwriting mentors themselves, or, worse, Broadway backers. (Imagine patrons telling Joanne Akalaitis that her staging of Heiner Muller's Quartet better get more accessible or else people may not like it! Well, that happened, I was there.)

Also, the outright rudeness of some of the "questions" I've hear never ceased to amaze me. To whit: "Were those Southern accents intended to be inconsistent?" "Did you mean the characters to be unbelievable?"

Obviously, it's just a few "bad apples," if I may use the phrase, who take over and, if they're lucky, get a groupthink going in the crowd that skews the perceived "response" one way or the other. For instance, at one new play I moderated at the first questioner could still not get over the first line (two hours ago) said the word "bullshit", which then became the only topic of conversation for the next twenty minutes. I'm sure the playwright was helped by that.

The tragedy of talkbacks (and really I'm not overstating!) is that the insightful people either leave or stay around to watch the freakshow, while the rest of us are held hostage by the crankiest, most throwback savants who insist that nothing good has happened in the theatre since Death of a Salesman.

Yes, we need to take down that fourth wall and let artists and audiences engage more. But somehow the more formal and structured the setting (meant to protect the artists) ends up privileging the looniest and least fruitful comments--often by sheer luck of the draw. Instead, it often is much better to just keep the bar open after the show and let people congregate on their own (un-moderated and un-mediated) and invite some artists to mingle.

Hi Jason: Thanks for this post. As a critic and supporter of experimental work (drat the terminology) I am very interested in the three-way dialogue between artist, critic and spectator. I’m cross-posting here and at Playgoer. Hope that’s okay.

I've done a few postshows (and some preshows) at Montclair State University's Peak Performance series and it's true, they can seem really pointless or worse, a soapbox for cranky audience members. Still trying to work out a new model so that they can be fun and informative.

Which leads me to... self promotional part ahead... a new preshow project I'm starting with Helen Shaw and playwright-essayist Jeff Jones. It's called the Program, it's a preshow event, and we're doing the first, casual one at PS 122 this Saturday night before Jay Scheib's Untitled Mars.

More details: THE PROGRAM is David Cote, Helen Shaw and Jeffrey Jones— two reviewers and a playwright-cultural critic—who want the widest possible audience to feel welcome at the widest range of dramatically ambitious work. Armed with pre-show discussions and supplementary dramaturgical materials, THE PROGRAM roams from theater to theater, providing context to audiences at selected experimental productions. In the fine arts, museum-goers feel welcome at even the most abstract, difficult shows: docents, catalogues and wall text reach out to new viewers. But in the theater, we get tossed in front of the avant garde with little preparation. So, in the interest of deepening the conversation between audiences and those pieces pushing formal boundaries, THE PROGRAM offers a casual opportunity for enrichment and investigation and conversation—and maybe a glass of wine. For UNTITLED MARS, we’ll be talking to director Jay Scheib about his fascination with technology, his influences (many of them cinematic) and how his unusually collaborative process turns into the highly choreographed works that are his specialty. Join us!

This is a great post. I'm curious what the "groundbreaking" results of the study are - I've always thought of audience questionnaire as a way to survey audience demographics. One thing I wonder about, though, is how to judge *why* a play is or isn't working. Of course, the ephemeral "It just is" is lovely and at the heart of art-making, but, for example, we can assume a work is too off-putting or challenging or not funny enough, but what if we're wrong? If we decide to rework something to be more effective to our audience (which isn't always the decision made), is intuition the only tool? I don't mean to discount intuition, or watching an audience, or eavesdropping in the lobby, all methods I believe in. And I'm not endorsing the study, necessarily. (Especially since I don't know what it says or what the methods were.) I just wonder.

I must admit to being puzzled by Jason Grote's blog post on the "blog of the National Performing Arts Convention" entitled "Watching the Watchers: Gauging Audience Response." Perhaps it is the difficulty of writing something about an event that hasn't happened yet, but the more I read of his post the more I found myself despairing. Why? The first sentence of Jason's post reads, "While I can't speak to the specifics of the study in question, I generally think that surveys measuring audience response are a bad idea." A little later, he writes, "With all due respect to Mr. Yoshitomi and Mr. Brown, with whose methods I am not at all familiar, I feel that "Measuring the Intrinsic Impact" of any work of art is an idea with potentially disastrous results for the creative process generally." Further down, he draws a parallel to film focus groups (because, I guess, the description of the study reminds him of them), saying "While I can't cite any precise data on the failure or success of these methods, and of course can't speak to their similarity to the study in question, I have heard anecdote after anecdote about its failure as both an aesthetic and a profit-making strategy." (italics in above quotations all mine)

First of all, while this presentation hasn't occurred yet, the study is available as a
pdf download. There's even a summary, if reading the whole study takes too much time. In addition,
Andrew Taylor wrote about the study on the ArtsJournal blog The Artful Manager on January 9th and
January 16th, I wrote about the study on January 18th, and Butts in Seats commented on this study on
January 24th.

I know that Jason is a busy man and he can't spend time doing tons of research, but at the same time if you are going to dismiss a presentation as "potentially dangerous" without having read more than a conference blurb is a little irresponsible, especially when a quick Google search would have uncovered all these useful connections.

The fact is that the WolfBrown study has nothing to do with the creation of art, so Jason need not worry, but is actually about what factors lead to a higher level of engagement and satisfaction with the art that is being put in front of an audience. For instance, as I wrote in January, the study discussed the ""Context Index" -- "the amount of information and personal experience with the art and artist" that an audience member has. The Context Index was a "significant predictor for Captivation ["the degree to which an individual was engrossed and absorbed in the performance"], Intellectual Stimulation ["mental engagement, including both personal and social dimensions"], Emotional Resonance ["intensity of emotional response and degree of empathy with the performers"] and Spiritual Value ["transcendent, inspiring, or empowering experience"]. The level of satisfaction with a performance was correlated, among other things, to the level of Captivation."

In actuality, the WolfBrown study might help theatres figure out how best to market and prepare the potential audience for, say, Jason's next marvelous play by examining what sort of pre-show activities and information for a similar type of arts activity leads to an audience most likely to be engaged by Jason's work. It might help theatres to figure out just why they exist and what experience they offer.

Anyway, the point is that this was a significant study that actually has some interesting things to say about how audiences perceive the arts, so instead of defending the arts on the basis of how much people spend in restaurants when they go to a play, or by saying that kids who participate in the arts do better in their classes, we might talk about the impact the arts have on people. To reject it after consideration is one thing -- Andrew loved it, I loved it, Butts in Seats seemed uncomfortable; to reject it having only read a conference blurb, especially when the report is available on line, is another.

I am considering attending NPAC, and if I do I suspect I will also attend this presentation. I also intend to keep up with the NPAC blog, and I look forward to what will undoubtedly be more well-considered and thoughtful posts.

Please note: the entries posted by the attendees above represent their personal impressions, not the viewpoints of the organizations they work for.

About this blog From April 1 through June 9, 2008, weekly entries will be posted here by some of the performing arts community's top bloggers. This 10-week intensive blog will serve as a unique forum for digital debate and brainstorming, and both the entries and comments will be archived for use at the live NPAC sessions in June. New entries will be posted every Monday morning. Please note: the views expressed in this blog represent those of the
independent contributors and participants, not the National Performing
Arts Convention.

NPAC - the National Performing Arts Convention - will take place in Denver, Colorado on June 10-14, 2008. "Taking Action Together," NPAC will lay the foundation for future cross-disciplinary collaborations, cooperative programs and effective advocacy. Formed by 30 distinct performing arts service organizations demonstrating a new maturity and uniting as one a sector, NPAC is dedicated to enriching national life and strengthening performing arts communities across the country. Click here to register, and we'll see you in Denver!