Online Resources

03/21/2018

Here's a second video in a series by Pew Research. This 5 minute clip describes some of the issues in writing good questions for an opinion poll. It basically summarizes the first part of Chapter 6, and provides some new, concrete examples.

01/10/2018

When the general public critiques research, I often hear them say that the samples are "too small." It's true that sample sizes (N) in psychology research should be large. One of the outcomes of the so-called "replication crisis" is that large samples are more and more important in psychology. But why?

A common misconception--held by both students and the general public--is that large samples are important because they ensure external validity. This misconception is incorrect. External validity (that is, the ability to generalize from a sample to a population of interest) is about how a sample has been recruited, not how many people are in it (see Chapter 7, 14). For example, say you recruited a sample of 1000 fans attending the national championship college game. You'd have a pretty large sample, but you couldn't generalize from that sample to college students in the U.S. (for example). In fact, unless the 1000 fans were selected at random from the 70,000 fans at the game, you couldn't even generalize from this sample to "people attending the national championship football game."

If not external validity, why are large samples important? It's about accuracy of our statistical estimates. When estimating values in the population such as means or differences between means, large samples are less likely to be influenced by chance variability. For example, imagine you're estimating the mean height of kindergarteners in your local school. Now imagine that you select 5 kindergarteners at random, one of whom, by chance, turns out to be extremely tall for her age. That tall kindergartener is going to "pull" the mean estimate upwards when combined with only 4 other kids. But what if you select 25 kindergarteners instead? Now the tall kindergartener is going to be balanced out by 24 other scores, and her height will have less influence on the mean estimate.

Below is a pair of animations that illustrate this principle. They come from the data science blog R Explorations. The animation used the program R to run a simulation study over and over and over. First, they created a very large population of scores whose mean was known to be 10.0 and whose standard deviation was known to be 1.0. Then they asked the computer to draw a random sample of size 10, compute the mean of the 10 scores, and plot them. You can watch the samples appear in real time on the animation below. Here, xbar is the sample's mean and s is the sample's standard deviation. The red line represents the mean for each sample as it is drawn:

Questions

a) First, watch the top animation, where N = 10. What do you notice about the movement of the vertical red line representing the mean in the top animation? What is it doing, and what does that represent?

b) Now watch the bottom animation, where N = 1000. What do you notice about the movement of the vertical red line representing the mean in this second animation? What is it doing, and what does that represent?

c) What do you notice about the s values of the two animations? Which animation has a steadier estimate of s?

d) Answer this one only if you've had a statistics course: Which of the two animations will have a smaller standard error? How is the standard error represented in the two animations?

e) Given the behavior of the two animations, explain why a large sample is important for research.

f) Which validity does sample size best address, if not external validity?

g) Let's tie this concept back to the "replication crisis" (or, as some are now calling it, "credibility revolution"*). When a finding in psychology has not replicated in a direct replication study, one reason might be that the original study used a small sample. Another reason might be that the replication study used a small sample. Why might the sample size of a study be linked to its replicability? Explain in your own words.

12/20/2017

Most research methods instructors hope their course will teach students to be better consumers of information. They want to not only help students read empirical journals; they also want to help students become critical thinkers about anything they encounter in the "real world" of the Internet.

Maybe you'd like to start next semester with a few outside readings on spotting fake news. If so, here are some resources you . might use. I got these from Morton Ann Gernsbacher's wonderful online, open-access methods course, which focuses on identifying and critically reading online news. Check it out for other wonderful resources.

08/15/2017

Pew Research is my favorite polling resource, partly because they ask such interesting questions, and partly because they are so transparent about sharing their methodology. (For examples, see their Methods page or click on the full Report Materials for a study they did on gun ownership in America.) They make their sampling techniques and question wording easily available.

Now Pew has shared a video that explains how a sample of 1,000 can be used to draw inferences about the population. Instructors: Save this one for when you teach Chapter 7!

08/09/2017

Science progresses one study at a time. As scientists conduct research and make the results public, we enable others to build upon, replicate, and critique our work, improving the field and building a body of knowledge. Even the studies in the textbook are not certain "truths," but rather steps on a scientific path, selected at one moment in time.

As the third edition of the textbook explains (Chapter 14), psychologists are investing new energy in improving our field. More than ever, psychologists are conducting replication studies, improving data analysis techniques, and making science open and transparent.

In same spirit, we've added a new section of the blog called Replication Updates. When I learn that a study that is featured in the textbook has not been replicated or has had its conclusions questioned, I will devote a post to the issue so that instructors can stay informed. To read them, click the appropriate filter on the blog menu.

I hope that as teachers, we will find ways to include students in thinking about issues of replication, scientific openness, and progress. As we do, let's keep in mind that some (but not all) of the critiques appear in non-peer reviewed outlets, and discuss this aspect with them as well.

Going forward, I hope readers will let me know when they hear updates on studies featured in the book!

The year 2016 provided multiple references to implicit and explicit racial biases, especially in politics. So you might be wondering, What does it mean to hold "implicit biases?" Why are people biased against some ethnic groups, and what can we do about it?

It turns out there is a strong research tradition concerned with measuring and correcting implicit bias. There's a series of short videos grouped under the title, What, me biased? Each presents a real-world situation relevant to racial bias and discusses a research study.

a) In the opening minute, TV host Heather McGhee poses a theory about how to reduce racism to the caller. What is the theory? How did the researchers use data to test the theory?

b) What was the independent (manipulated) variable in the study? What was the dependent variable?

c) How do you know the study was an experiment? Was it an independent groups or within groups design?

d) Sketch a graph of the result, labelling your axes mindfully.

e) Work through the theory-data cycle: Did the data support Ms. McGhee's theory, or not?

More Resources for Instructors:

There are seven videos in this series, providing other opportunities to practice research methods concepts. For example, students can practice an individualized version of the theory-data cycle (Chapter 1), where Dr. Dolly Chugh discusses the idea of an "audit" in the video, Check Our Bias to Wreck Our Bias.

07/10/2015

There's a fun interactive datagraphic on gallup.com's website. It's called "State of the States." You can select a polling variable, such as "overall well-being," "support for Obama," or "religiosity," and it will show you how each U.S. state scores on that variable.

Feel free to take a minute to play with the interactive right now. (I'll wait.)

I've pasted a screen shot from the "well-being" results below. Take a look at it, and consider the questions that follow.

a) In the figure above, the variable I selected was "Well being." The thermometer below indicates that darker states are higher in well-being than lighter states. Using that rule, which states are the highest in well-being? Which are the lowest?

b) You might notice that South Dakota is higher in well-being than North Dakota--their shades of green are noticeably different. In fact, you might even imagine a news story in which a reporter suggests that South Dakotans are "happier." But I want you to consider the effect size of the difference. About how much happier are South Dakotans, according to the scale?

Now consider the next screen map (below). This one shows religiosity, indicating the percentage of state residents who consider themselves "Very religious":

c) As before, the thermometer below indicates that darker states are higher in saying they are "very religious" compared to lighter states. Using that rule, what states are the highest in religiosity? Which are the lowest?

d) Take a look at the scale for this variable--what do you notice about the range for Religiosity compared to the range for well-being?

e) On the map, the states of Utah and Idaho are about the same shades of green as South and North Dakota were on the well-being variable. Indeed, the shades of green for Utah and Idaho are noticeably different. In fact, you might now imagine a news story in which a reporter suggests that Utahans are "more religious." Once again, I want you to consider the effect size of the difference. How much more religious are Utahans, according to the scale?

e) What do you think? How is Gallup using these shades of green in this interactive data map? Is their use misleading? If so, what might be better?

06/18/2015

Yet another case of fabricated data, this time from political science, is being analyzed in the news lately. Last December, Science published an article showing that certain types of canvassing techniques could dramatically change people's attitudes toward marriage equality. However, the data for the study were apparently fabricated by one of its key authors. Here are two good pieces that cover the timeline of the article, its discovery, and its retraction:

02/19/2015

These articles primarily come from Psychological Science or Personality and Social Psychological Science. Our students have used these articles in their final projects for Research Methods in the past. In this final project, students summarize and analyze the validities for two empirical studies. (I can provide the assignment upon request: morling@udel.edu.)

Most of these articles report more than one study, and at least one of the studies is simple enough for a student with basic skills in research methods to understand. The most complex experimental design would be a 3-way factorial design. The most complex correlational design uses multiple regression and (sometimes) mediation.

If you’re a research methods instructor or student and would like us to consider your guest post for everydayresearchmethods.com, please contact Dr. Morling. If, as an instructor, you write your own critical thinking questions to accompany the entry, we will credit you as a guest blogger.