Online Continuing Education Programs for the Market Research Profession

Engagement in the Moment

This is the eighth in a series of articles on the topic of Participant Engagement and Survey Experience Satisfaction.

Perhaps the most common form of market research is the study of customer satisfaction. A cornerstone of continuous improvement, customer feedback on the goods and services provided by businesses is a part of every researcher’s portfolio of deliverables.

Unfortunately, like all forms of survey work, response rates to satisfaction study requests have plummeted over the past 20 years. When Internet based surveys were novel (circa late 1990s) response rates (among online panelists) were nearly triple that of RDD CATI phone. But as Web has become ubiquitous, the response rates for either data collection medium have become practically identical—hovering around 5%.

In keeping with this series’ central thesis—that one of the core problems with our survey work is that the tasks are boring, difficult and/or irrelevant to many potential participants, researchers are exploring many new ways to reinvigorate response and cooperation rates. One such relatively new technique is “in-the-moment” intercept.

A serious threat to data validity when asking people about their experiences and levels of satisfaction is the passage of time. As the time between the direct experience and the feedback request increases, people’s memories tend to moderate. In one study that we conducted with users of an online brokerage system, we found that people’s negative experiences became more and more under-reported as time passed. From the customer’s database we had the exact number of times that people had called customer support, the number of times that they had to follow-up before resolution of a problem and the total amount of time they’d spent attempting correction. The longer the time between the incident(s) and the follow-up feedback invitation, the estimation of the “level of hassle” diminished.

Another interesting sidebar finding was that if a live person called for the feedback, the variance between actual and estimated hassle also diminished significantly. This interviewer effect was demonstrated when the same questions were asked in an online setting and the responses to time and effort questions were far more accurate.

Through experimentation with mobile technologies for the delivery of satisfaction questions, both passage of time and interviewer effects have been shown to be ameliorated. Although almost all survey work via the Internet now has the requirement of having a “responsive design” (i.e. that the same survey must play accurately on Workstation, Laptop, Tablet or Smart Phone), the opportunity to leverage the in-the-moment-intercept has been very effective.

For example, we have worked with a number of conference producers (with many different industry focuses.) A great deal of time and effort is expended to judge conference attendee satisfaction and reaction to events, content, speakers, venue, networking, etc. Unfortunately, getting attendees of large events to fill out paper-based evaluation forms is difficult with an average of less than 30% providing helpful feedback. Follow-up survey requests to “fill-in-the-blanks” of attendee experience have low response rates and suffer from time delay effects.

However, when a mobile survey is delivered in real time to attendees’ Smart Phones, quick feedback (sometimes within minutes) is much more likely. When attendees are offered the event schedule through an app, this can also become the platform for requesting feedback. Requests delivered by text reminder have yielded response rates of up to 80% for specific session attendance.

Figure 1: Example of Instructions for Session Evaluation

An excellent example of best practices in modern in-the-moment satisfaction tracking are several initiatives that various companies have undertaken over the years.

One company maintains a number of Executive Briefing Centers, where they entertain and provide forward looking consultative services for teams of key client participants. As a part of this experience, this company used mobile computing equipment to capture session-to-session satisfaction ratings from the attendees and generated real-time reports for senior account managers to pivot to topics of interest and address concerns.

Another firm has sponsored many high-visibility international events. As a side-activity to the main event, senior managers have spoken to invited attendees to share vision and insights about the evolving markets. By using mobile technology to gather instantaneous satisfaction feedback from the guests, specific topics can be addressed and aggregated opinions can be discussed.

As technology improves and customers become well-grounded in the use of mobile device platforms, many other forms of in-the-moment research opportunities are emerging:

In store intercept for shoppers regarding satisfaction with retail experience

Geo-fencing for identifying people in locations related to experiences (e.g. exposure to advertising or specific use environments)