Best Practices for Analyzing and Interpreting List Experiment Results

Why use a list experiment?

Navigating the landscape of private opinion on sensitive topics can be challenging, with individuals often reluctant to openly express their true beliefs. This is where list experiments emerge as a potent tool for researchers and policymakers alike. A list experiment is a method designed to indirectly measure private opinions, safeguarding respondent anonymity and encouraging candid responses. This approach protects privacy and yields insights that may otherwise remain obscured. In this post, we delve into the intricacies of analyzing and interpreting list experiments, offering a step-by-step guide to unlock the wealth of information they hold and empowering decision-makers to navigate complex societal issues with greater clarity and understanding.

Analyzing list experiment data

Difference-in-means estimation

Historically, list experiments have been analyzed using a simple difference-in-means estimator. Specifically, the mean number of items reported by the control group subtracted from the mean number of items reported by the treatment group is equal to the proportion of the population that privately supports the sensitive item. 

There is a bias-variance tradeoff to using the list experiment. The primary advantage of using the list experiment is lower bias in revealing private opinion – historically, the primary disadvantage is higher variance. By design, we increase the noise, making it harder to find a signal. However, multivariate regression models allow us to recoup some of the variance costs of the standard difference-in-means estimator by more efficiently using the data collected from list experiments.

Multivariate regression techniques

Recent advancements in statistics and multivariate modeling have led to more sophisticated analyses (Blair & Imai, 2012; Imai, 2014). Multivariate regression techniques are now freely available through open-source platforms that allow for the following advantages over the standard difference-in-means estimator:

1. Investigation of individual characteristics differences to understand who is most likely to endorse sensitive statements
2. Analysis of counterfactual probabilities – i.e., which sub-groups are more likely to answer the question differently if asked directly (using traditional public opinion polling methods) or indirectly (using the list experiment)
3. Test of the design effect – does inclusion of a sensitive item change the way people respond to the control items?
4. Adjustments of results for the presence of ceiling or floor effects

Using statistical software, we can calculate the joint distribution of responses to the sensitive item asked both directly and indirectly. Assuming there are no liars and no design effect, we can use the maximum and minimum responses in the treatment group to calculate the population proportion for each respondent type. The table below depicts all respondent types.

Note: The first number in the parentheses refers to the number of control items a participant agrees with and the second number refers to whether or not they agree with the sensitive item.

Case study: Private Opinion in America

In Gradient's journey to uncover how to accurately measure private opinion in America, we often partner with think tank Populace. One of Populace's core beliefs is that Americans don’t feel like they are living in a democracy of opinion—that Americans don’t feel comfortable voicing their true thoughts because they are counter to society’s. This lack of public honesty creates widespread preference falsification to the point that systems and institutions are designed with a purpose with which many Americans actually disagree. 

Social desirability bias is a problem in all self-report research, but it is especially common in research on sensitive topics. Respondents feel pressure to conceal the truth due to a variety of factors, including social pressures (e.g., reporting private opinions expected to be dissonant with the collective attitude), fear of reprisal, or lack of privacy. It is often difficult or impossible to determine the size and the direction of misreporting bias.

In the Private Opinion in America project, Gradient deployed a list experiment to estimate how Americans privately feel on a variety of social issues and compare that against public opinion on the same issues.

The results of the project revealed the pressure to misrepresent our private views—to offer answers on politically and socially sensitive questions that are out of sync with our true beliefs—is pervasive in society today. Across all demographics every subgroup had multiple issues with at least a double-digit gap between public and private opinion.

For example, when it comes to abortion, men are not as supportive as their public opinions would suggest. In public, a majority of men (60%) agree the decision to have an abortion should be left to a woman and her doctor; however, in private that number is only 45%. In addition, most men (52%) publicly say that “abortion should be legal in most cases,” but in private that is not the majority view (48%).

Conclusion

Using a list experiments offers a transformative approach to uncovering private opinions on sensitive topics, illuminating the stark disparities between public and private stances. Gradient's sophisticated approach to analyzing and interpreting list experiment data underscores the significance of safeguarding respondent anonymity and employing rigorous statistical techniques to extract nuanced—and actionable—insights.

Through this methodology, decision-makers at think tank Populace and other organizations can gain a clearer understanding of societal dynamics, enabling them to navigate complex issues with heightened sensitivity and informed decision-making. Results from list experiments can reveal the pervasive pressure on the public to misrepresent private views, thereby underscoring the imperative of embracing innovative methodologies to capture the true pulse of private opinion.

Does your organization need to capture the private opinion of your audience? Gradient can help! Let us know your research needs.