Survey completions on mobile devices have been increasing rapidly. This important shift is something market researchers should definitely consider when designing and conducting self-administered online surveys. View Summary

Survey completions on mobile devices have been increasing rapidly. This important shift is something market researchers should definitely consider when designing and conducting self-administered online surveys. This article briefly summarises existing research and empirical results from mobile surveys. Based on the specific findings discussed, market researchers should be better aware of what to expect when fielding surveys completed by mobile respondents, whether this is intended or not. Bringing the findings together and discussing more broadly, for online surveys, market researchers should consider consciously and deliberately accommodating both mobile and PC respondents. Thus far, the research on mobile surveys indicates that consumers want the choice and ability to take surveys when they want, where they want and on the device of their choosing. It is to be hoped that market researchers are listening and become willing to accommodate survey respondents in terms of device and, by extension, time and location.

This paper argues that online surveys are a bad way of measuring digital habits - including time and money spent - because survey respondents over-index on these things against the general population. View Summary

This paper argues that online surveys are a bad way of measuring digital habits - including time and money spent - because survey respondents over-index on these things against the general population.

Research has previously suggested the 'digital advertising gap' - the idea that people spent more time online than watching TV - but advertising spend had not matched this switch.

However, this understanding was based on online survey data and studies that used metering found that people actually spent less time with digital devices than thought.

Further analysis found that online survey respondents are two to three times heavier users of the internet than average, so their answers did not reflect the broader population.

A similar issue also arises for questions around online purchasing habits, as panel members spend more money online than the average population.

3

A gamification effect in longitudinal web surveys among children and adolescents

The paper measures a gamification effect in longitudinal web surveys among children and adolescents 7–15 years old. View Summary

The paper measures a gamification effect in longitudinal web surveys among children and adolescents 7–15 years old. Two waves of the study were conducted using a volunteer online access panel in Russia among 737 children. Respondents were randomly assigned to one of the three conditions in the first wave without changing the treatment in the second wave: (1) a text-only survey, (2) a visual survey and (3) a gamified survey. Though in the first wave of the study respondents found it more enjoyable and easier to complete the gamified survey, no differences in participation rates were found between the conditions in the second wave. Contrary to expectations, a higher breakoff was found in the gamified condition. Moreover, it produced lower test-retest reliability correlations than the text-only and visual conditions in all survey questions. The promising gamification effect found in the first wave of the study faded in the second wave. It seems that implementing gamified elements in longitudinal web surveys might differ from the implementation of gamified elements in cross-sectional surveys.

4

Can a non-probabilistic online panel achieve question quality similar to that of the European Social Survey?

Recently, Revilla and Saris (2012) showed, using data from the Netherlands, that the quality of responses (product of reliability and validity) in a probability-based online panel (LISS) can be similar to those from face-to-face surveys (European Social Survey round 4). View Summary

Recently, Revilla and Saris (2012) showed, using data from the Netherlands, that the quality of responses (product of reliability and validity) in a probability-based online panel (LISS) can be similar to those from face-to-face surveys (European Social Survey round 4). However, most online panels select their members in a non probability-based way. They usually also send many more surveys per month to their panellists. Both together can generate professional respondents whose quality of answers may be different. Therefore, it makes sense to make a similar comparison for a non-probability-based online panel (Netquest). Although differences are found, the similarities prevail. Overall, we cannot say that one of the surveys has higher estimates of quality, when defined as the product of reliability and validity, than the other.

5

Coca-Cola's agile research adventure: Implementing an agile research program in Asia

This paper outlines how Coca-Cola used online research to create a rapid-response system for testing advertising creative in Southeast Asia. View Summary

This paper outlines how Coca-Cola used online research to create a rapid-response system for testing advertising creative in Southeast Asia.

Coca-Cola wanted to add speed and flexibility to its advertising development and testing tools, to enable its marketing teams to iterate rapidly in the early stages of new communication development.

It formed collaborative partnerships with several consumer insights companies to produce an online research system from which it could choose an audience, upload creative and define survey parameters, with results returned in hours or days.

The project adopted a test-and-learn and 'fail-fast' approach, using parallel traditional research methods to compare and validate online findings.

This paper explores the use of three survey methods in Brazil - online, telephone and face-to-face - to understand why each method is chosen, and proposes a hybrid methodology that can be used in public opinion research. View Summary

This paper explores the use of three survey methods in Brazil - online, telephone and face-to-face - to understand why each method is chosen, and proposes a hybrid methodology that can be used in public opinion research.

Face-to-face surveys continue to dominate, followed by online and telephone; and as landline penetration decreases and internet penetration increases this is expected to change.

Face-to-face interviews are expensive, but will continue because internet penetration stands at around 50%, so traditional methods are needed for nationally representative samples; and researchers are attached to traditional methods.

This is already done in some areas of market research and public opinion, but not yet in voting intention studies.

There are important differences to be considered when combining the datasets though, as well as demographic differences, online panellists tend to be more critical - most likely because they feel more anonymous.

This article discusses the use of non-probability web surveys to measure sexual behaviours and attitudes in the British general population, as introduced in a session on changing social research at th... View Summary

This article discusses the use of non-probability web surveys to measure sexual behaviours and attitudes in the British general population, as introduced in a session on changing social research at the SRA Annual Conference held in London in December 2014.

Surveys mainly use questions in which it is allowed to answer only through a closed series of alternatives. View Summary

Surveys mainly use questions in which it is allowed to answer only through a closed series of alternatives. The choice of labels for these closed alternatives is an important decision. Depending on this choice, different results can be found. This paper focuses on the impact of using low versus high frequencies or durations scales. The novelty is that it studies panellists of an online panel oriented towards marketing surveys. Also, it uses data from countries little studied before: Spain, Mexico and Colombia. Using a split-ballot experimental design, it shows that significant differences in answers are obtained depending on the scale used. In order to determine which scale gives results closer to the reality, the correlation with an external variable is used; the higher this correlation, the better the scale. In practice, this information can and should be used to select the best scale for a survey.

9

Gamification in market research: Increasing enjoyment, participant engagement and richness of data, but what of data validity?

Research undertaken into the role of gamification in online surveys has already clearly demonstrated that applying some gamification principles can significantly increase the richness of spontaneous data and participant engagement, as well as the time that participants take to complete a survey. View Summary

Research undertaken into the role of gamification in online surveys has already clearly demonstrated that applying some gamification principles can significantly increase the richness of spontaneous data and participant engagement, as well as the time that participants take to complete a survey. It is obviously appreciated that consumer engagement is critical for ensuring completion rates, reducing boredom within survey and also for panel membership moving forward, but the primary consideration and focus when designing any research survey has to be on accessing reality for the consumer and hence data validity. This paper shares the results of a research-on-research study that was conducted to understand the role of gamification, not only in terms of participant engagement and richness of data but also data validity.

10

The Great Market Research Debate: Are mobile insights better than online?

Includes video content

Recommended by Warc editors

Trends

Best Practice

Ben Leet and Edward Appleton, Research on Warc, January 2015

This article explores the argument that mobile research gives more accurate insights than online research, with a three-stage study reaching the conclusion that it does. View Summary

This article explores the argument that mobile research gives more accurate insights than online research, with a three-stage study reaching the conclusion that it does.

Smartphones are increasingly important to consumers, whilst research agencies and clients are uncertain of how mobile might improve insights and fear change.

This study ran in the UK and US with separate mobile and desktop survey respondents.

Geo-fencing was used to make sure participants in the mobile group answered survey questions in-store, whilst desktop participants relied on memory later.

This reliance on memory proved important as online respondents guessed the answers to some questions, whilst mobile respondents provided more accurate answers and richer detail.

By being closer to the moment, mobile provides more accurate answers and open ended questions generate more detailed response, but at present it costs more to execute.

11

Comparing approaches to elicit brand attributes both face-to-face and online

Brand attributes play an important role in tracking customer-based brand equity. Therefore researchers need an effective approach for eliciting attributes. View Summary

Brand attributes play an important role in tracking customer-based brand equity. Therefore researchers need an effective approach for eliciting attributes. This paper has two aims: to determine which of four different techniques elicit(s) better results; and to test if online data collection is a viable alternative to face-to-face collection. The techniques compared are: Zaltman Metaphor Elicitation Technique (ZMET), Free Elicitation (FE), Repertory Grid (RG) and Projective Elicitation (PE). These approaches are compared on the number and variety of attributes generated, as well as respondent evaluation. FE is the best-performing technique in a face-to-face context, generating the most attributes, evaluated positively by respondents and providing a typical distribution of attribute types. We also provide evidence that online is a viable data collection method for attribute elicitation studies, except ZMET due to respondent drop-out. Online we recommend a combination of FE and PE to obtain a range and variety of responses.

12

The Impact of Survey Routers On Sampling and Surveys: Unraveling the Mysteries of Survey-Router Design and Deployment

This study conducted by Ipsos, Cogenti Applied Strategies and DM2 explores potential consequences of using survey routers - software for the real-time assignment of surveys to a continuous flow of online respondents. View Summary

This study conducted by Ipsos, Cogenti Applied Strategies and DM2 explores potential consequences of using survey routers - software for the real-time assignment of surveys to a continuous flow of online respondents. The use of these routers is becoming the norm in an era of opposing market forces: increasing demand for online respondents and decreasing respondent participation.

13

Viewpoint: Helping the public see the value of social research using social media

This article looks at how the internet has changed the way we communicate and its effect on market research. View Summary

This article looks at how the internet has changed the way we communicate and its effect on market research. Both participants and research professionals are unclear on what information can be collected, who or what governs it and whether researchers' code of conduct should be bound by more than just the law. Due to users' concerns about their data, scepticism and suspicion about social research is tied up with their wider concerns about the online world and researchers must be more transparent about research objectives.

Scientific and commercial researchers around the world are relying increasingly on the responder insights obtained from online panels. View Summary

Scientific and commercial researchers around the world are relying increasingly on the responder insights obtained from online panels. However, from the researcher viewpoint, what distinguishes a good panel from a bad one is not clear and new metrics need to be developed to consider panel health. Given that online panels are composed of individuals, it is proposed that new individual-based metrics, taken across an individual’s career, should also be developed. This paper proposes that each respondent’s overall average relative response time (OARRT) across all surveys they have completed is one such measure, and examines how it is affected by a range of control factors. Three cohorts of panellists – that is, ‘ongoing’, ‘unsubscribers’ and ‘three-strikers’ – are examined. We found that the number of surveys completed and the respondent’s age affected OARRT across the three cohorts, with participation in other panels affecting OARRT for unsubscribers. Gender, education and employment status did not impact OARRT for any group. The results suggest there may be a learning effect.

15

ESOMAR Digital Dimensions 2014: New directions in digital research

Includes video content

Recommended by Warc editors

Trends

Best Practice

Manfred Mareck, Event Reports, June 2014

This report summarises the key papers and presentations from ESOMAR's Digital Dimensions 2014 conference, exploring the themes of online research and big data, social media research and mobile research. View Summary

This report summarises the key papers and presentations from ESOMAR's Digital Dimensions 2014 conference, exploring the themes of online research and big data, social media research and mobile research.

Digital is providing new opportunities for market researchers, including web-tracking, mobile devices and analysis of user-generated content.

Insightful applications of Big Data include social media tracking, geo-location and search data.

'Silo approaches' to research need to be broken down to maximise business opportunities for research agencies and end clients.

Social media have provided new opportunities to consumers to engage in social interaction on the internet. View Summary

Social media have provided new opportunities to consumers to engage in social interaction on the internet. Consumers use social media, such as online communities, to generate content and to network with other users. The study of social media can also identify the advantages to be gained by business. A multidisciplinary model, building on the technology acceptance model and relevant literature on trust and social media, has been devised. The model has been validated by SEM-PLS, demonstrating the role of social media in the development of e-commerce into social commerce. The data emerging from a survey show how social media facilitate the social interaction of consumers, leading to increased trust and intention to buy. The results also show that trust has a significant direct effect on intention to buy. The perceived usefulness (PU) of a site is also identified as a contributory factor. At the end of the paper, the author discusses the results, along with implications, limitations and recommended future research directions.

17

Research Quality: Fast and Furious ... or Much Ado About Nothing? Sub-Optimal Respondent Behavior and Data Quality

The "Foundations of Quality 2" (FoQ 2) study is a broad-based initiative seeking to understand how sampling, weighting, and respondent behavior affect the quality of survey results for web-based respondents. View Summary

The "Foundations of Quality 2" (FoQ 2) study is a broad-based initiative seeking to understand how sampling, weighting, and respondent behavior affect the quality of survey results for web-based respondents. This paper focuses on how sub-optimal respondent behavior is related to response accuracy, summarizing tha analyses of three commonly used indicators of sub-optimal response within the FoQ 2 study: speeding; grid non-differentiation; and failure of compliance traps. This portion of the study first will clarify how each indicator was measured; what the overall results for each indicator were; and what demographic factors were associated with each sub-optimal behavior. The paper will also examine the overall effects of sub-optimal behaviors on substantive responses and, the researcher will provide some indication of the differences in the sub-optimal behaviors obtained across the 17 different sample providers.

18

Fusing data to reveal customer insights at Everything Everywhere

Includes video content

Recommended by Warc editors

Trends

Best Practice

Will Goodhand and Tim Pritchard, Warc Exclusive, Next Generation Research, January 2014

This presentation discusses how market research is used at Everything Everywhere, the UK mobile phone network. View Summary

This presentation discusses how market research is used at Everything Everywhere, the UK mobile phone network. The importance of storytelling is discussed, with the introduction of the 4Ps to structure research - 'problem', 'proof', 'promise' and 'proposal'. Two stages of modelling are explained, starting with linking survey results to actual behaviour, and then linking touchpoint experiences with perceptions and behaviour.

19

Research Quality: The Interaction of Sampling and Weighting in Producing a Representative Sample Online: An Excerpt from the ARF's "Foundations of Quality 2" Initiative

This paper discusses the effects of sampling and weighting, and whether tight sampling control removes the need for weighting. View Summary

This paper discusses the effects of sampling and weighting, and whether tight sampling control removes the need for weighting. Weighting, the practice of giving some responses more influence over a final result, is a controversial tool. Weighting is often used in online surveys in order to better reflect the population. The need for weighting is discussed when tight sampling specifications have been implemented - whereby the people surveyed have been selected to accurately reflect population demographics. This study finds that there is no firm conclusion as to the need for weighting after tight sampling, but that complex sampling specifications may result in different practices from providers. Weighting reduces the risk of unreliable sampling.

This article describes pilot research in China by AkzoNobel, the paints and chemicals multinational and owner of the Dulux brand, which sought to create a global bank of cultural insights to underpin existing consumer knowledge. View Summary

This article describes pilot research in China by AkzoNobel, the paints and chemicals multinational and owner of the Dulux brand, which sought to create a global bank of cultural insights to underpin existing consumer knowledge.

Desk research, interviews with key people, and culturally connected trend-spotters allowed the development of insights on Chinese society, revealing the importance of stability and limits to individual progress.

Understanding of this trend was developed through online discussions, revealing that younger generations are beginning, in a small way, to move away from this.

This article describes how Passenger Focus, a government agency to protect the interests of rail passengers, established a national tram passenger survey. View Summary

This article describes how Passenger Focus, a government agency to protect the interests of rail passengers, established a national tram passenger survey.

The organisation already conducted surveys of bus and train passengers, and wanted to use the establishment of a tram passenger survey to test an online approach.

Four survey methods were tested, finding that a combination paper and online approach produced the largest response, though some responses differed by completion method.

22

Premier Inn: Pillow talk

Includes video content

Recommended by Warc editors

Trends

Best Practice

MRS Awards, Finalist, December 2013

This article explains how Premier Inn, the UK hotel chain, integrated findings from two pre-existing surveys - one for employees, and one for guests - in order to provide enhanced business insights. View Summary

This article explains how Premier Inn, the UK hotel chain, integrated findings from two pre-existing surveys - one for employees, and one for guests - in order to provide enhanced business insights.

Changes made as a result of data gathered in these surveys include changing advertising to emphasise friendly staff and comfortable beds, and adapting pricing structures for wireless internet access.

Analysing the two surveys together allowed the company to understand that high levels of employee engagement results in greater customer satisfaction, and in turn leads to greater revenue.

23

Creating a sustainable future for MROCs: Preventing the exhaustion of the most promising development of our industry

This paper discusses recruitment methods for Market research Online Communities (MROCs), arguing that by taking a new approach such communities can generate more insights for less investment. View Summary

This paper discusses recruitment methods for Market research Online Communities (MROCs), arguing that by taking a new approach such communities can generate more insights for less investment. Researchers are increasingly concerned with the cost-efficiency of MROCs and are seeking to fit in as many MROCs projects as possible. This scenario has led to response wearing out and members dropping out faster than new members can be recruited. A new promising methodology becomes exhausted before it even gets the chance to shine. It is argued that successful communities have the correct balance of 'creators' and 'contributors', with an example of a successful MROC described.

This paper discusses survey response styles, considering the personal characteristics - such as gender, age and nationality - which create response style and the difference between response style in online, telephone and other surveys. View Summary

This paper discusses survey response styles, considering the personal characteristics - such as gender, age and nationality - which create response style and the difference between response style in online, telephone and other surveys. Response style is a person's tendency to systematically respond to questionnaire items regardless of content, e.g. by giving extreme or mid-point responses on a scale. The impact of dropping or retaining the neutral point on scales is examined and the reliability of different measurement scales compared. Amongst the findings, the research showed that men are more likely to use the negative side of the scale, while women are most likely to use the extreme positive side. Guidelines for designing global online and mobile surveys which take response style into account are developed.

Online consumer reviews have become an increasingly important source of information for both consumers (i.e. View Summary

Online consumer reviews have become an increasingly important source of information for both consumers (i.e. about whether to buy) and marketers (i.e. about product strengths and weaknesses). However, online consumer reviews are unstructured and unsystematic in nature, making interpretation of these reviews an enormous challenge. The current paper sheds light on a particular methodology that can be used to investigate what consumers say about companies, brands or products. Consumer reviews of the four best-selling games available on Apple’s App Store were compiled. Leximancer, a content analysis package, was used to compare comments from users who provided games with a five-star rating versus a one-star rating. Results from the Leximancer analysis reveal the most common themes and concepts that consumers use to describe their experience with these games. Specifically, five-star reviewers describe games as fun, awesome, amazing and addictive; one-star reviewers describe games as boring, easy and stupid. Additionally, negative reviews include themes regarding the presence of ads, technological difficulties and value. Future research should explore how consumers and marketers use this information.