1. What is an Experiment?

In common parlance, people often say “experiment” when they mean “study.” All experiments are studies, but not all studies (in fact the vast majority of studies) are not experiments. In social science research methods, labeling a study as an experiment means something very precise.

Morton and Williams (2010) write that an experiment is conducted when “a researcher intervenes in the data generating process by purposefully manipulating elements of that process” (p. 42). Researchers want as much control as possible over their manipulations to minimize the interference of confounding factors. The most common way of controlling confounding factors is to compare experimental results to a baseline where all observable conditions are identical except for the presence of the manipulation. In within-subjects designs, subjects serve as their own baseline: all facets of the study are held constant and the same subjects experience all manipulations. However, between-subjects designs are more versatile and more common, and the selection of an appropriate baseline comparison, a control group, is essential.

Consequently, when most political scientists use the term “experiment,” they refer to a study where subjects (or whatever entity we are studying) are randomly assigned to different treatments, where the treatments are causal interventions. Druckman et al. (2011) write “In contrast to modes of research that address descriptive or interpretive questions, researchers design experiments to address causal questions. A causal question invites a comparison between two states of the world: one in which some sort of intervention is administered (a treated state, i.e. exposing a subject to a stimulus) and another in which it is not (an untreated state)” (pg. 16).

2. When are Experiments Most Appropriate?

When is an experiment the optimal design for a research question? First, if you are interested in studying causation, an experiment is the cleanest way to show a causal relationship between two variables. The biggest problem plaguing many observation studies is assessing whether one variable (the independent variable) actually causes another variable (the dependent variable). For example, does television news actually influence public opinion? It could be the case that people likely to watch television news differ in important ways from those people who do not, so in an observational study, we cannot pinpoint the role of television news in actually changing opinion (Morton and Williams 2010, p. 13). This kind of question is well-suited to an experimental design.

Second, if you are interested in human behavior, experiments often provide the most straightforward way to understand the mechanisms that explain why people do what they do. Demonstrating a relationship between two variables in observational data does not tell you how that relationship actually operates, while experimental techniques can isolate the process of influence.

Finally, some research puzzles are more conducive to experiments than others. We would never want to randomly assign individuals to experience civil war, nor would we want to randomly assign some people to be disenfranchised, for example. However, a special kind of study, natural experiments (described more below) can help researchers leverage the concept of random assignment even if they are not the force actually assigning the treatment. A creative experimentalist can design a treatment to ethically test a wide variety of political and social phenomenon. In recent years, there have been papers published in political science journals utilizing a wide variety of experimental stimuli to test questions like a “get out the vote” experiment on 61-million Facebook users.

Causal Inference: Key Concepts

3. Fundamental Problem of Causal Inference

If causal questions involve comparisons between two states of the world, the fundamental problem of causal inference declares that we cannot simultaneously observe those two states of the world, one in which an individual is treated and one in which the individual is not. We cannot directly overcome this reality but we can work around it by using random assignment. Random assignment means that “each entity being studied has an equal chance to be in a particular treatment condition” (Druckman et al. 2011, p. 16-17). While we expect there to be random sampling variation (we don’t expect the two treatment groups to be exactly identical on all characteristics), this variation is due to chance alone, meaning it is uncorrelated with the treatment subjects receive. Except for the intervention itself, we can assume that randomly assigned groups are probabilistically equivalent. In other words, we can assume that the control group acts like the treatment group would have if the treatment group had not received the treatment, allowing us to assess the average treatment effect.

4. The Meaning of “Control”

Morton and Williams (2010) define control in experiments as a researcher fixing or holding constant elements of the data generating process in an experiment to better measure the effects of the manipulations (treatments) (p. 44). This control is done to minimize the effect of observable and unobservable confounding factors in an experiment. Researchers have the most control in a laboratory environment, where they are able to measure more factors to make them observable and hold constant more of the factors influencing how subjects receive the treatments.

5. Random Assignment vs. Random Selection

Although these two terms sound similar, they are distinct concepts. Random sampling refers to the process by which subjects are selected from a population such that every potential participant has an equal chance of being selected for the study. An experiment does not require a random sample, although its external validity can be increased if a random sample is used. Random assignment refers to the procedure for determining how subjects in an experimental study are assigned to treatment groups.

Within and Between Subject Designs: In a between-subjects design, each subject is randomly assigned to a single treatment, and the researcher compares the treatment groups to each other or to the control group. In a within subjects design, individuals serve as a control to themselves, and a subject is observed before and after receiving the treatment. While within-subjects designs are appropriate in many instances, the concern is that there can be many confounding factors with the delivery of the treatment itself, making it harder to establish a causal connection between the treatment and the outcome variable.

6. Internal vs. External Validity

Experiments offer very strong internal validity—the ability to assess the degree to which the independent variable (the treatment you are manipulating) actually causes your dependent variable. The best experimental research designs include appropriate baseline or control groups, so that researchers can make the most precise causal inferences possible. Conversely, experiments are weaker in their external validity, or the degree to which the causal relationship demonstrated in the experiment could be replicated in other contexts, with different people, or with different operationalizations of the treatment or outcome variables.

7. Types of Experiments

Survey Experiments: The term survey experiment refers to a broad category of stimuli where an individual decision-making experiment is embedded within a survey (Morton and Williams 2010, p. 79). These can take many forms. In the most basic form of survey experiments, researchers can vary question wording or question ordering to assess whether these factors affect respondents’ answers. More complicated survey experiments involve vignettes, where subjects read about hypothetical scenarios in which researchers manipulate various features of the situation. Increasingly common are list experiments, a form of a permissive design (Druckman et al. 2011, p. 107) that removes pressure favoring one response over another, allowing respondents to more honestly convey their behaviors or attitudes. In a list experiment, subjects report the number of statements with which they agree, not the statements themselves. Subjects are randomly assigned to one of two groups, a baseline condition with a list of statements of length n, and a treatment condition with a list of identical statements but where an additional, potentially objectionable statement is inserted (length n+1). By comparing the mean number of items agreed with between the two groups, a researcher can estimate the proportion of people who agreed with the objectionable statement.

Lab Experiments: The basic requirements for a lab experiment are that subjects are recruited to a common physical location where the study takes place, and a researcher directs the behavior of the subjects. Lab experiments in political science have their root in the experimental techniques developed in other disciplines. Experimental and social psychology have had an enormous influence on the field of political psychology, and political psychology experiments typically adhere to the norms established in these disciplines. Subjects are paid a flat fee (or given class credit if student convenience samples are used) and researchers assume that subjects will behave sincerely. Researchers often go to great length to make their studies seem naturalistic and may use deception. Conversely, in experiments modeled after techniques used in behavioral economics, subjects are paid based on the choices they make in the experiment under the assumption that the payment motivates subjects to behave in a more natural fashion. These experiments tend to be abstracted away from reality and are used to test theories and formal models. Deception is rarely used. Finally, “lab in the field” experiments occur when researchers bring the controlled environment of the lab into a field environment, in an effort to conduct experiments on a wider variety of populations and to increase the external validity of the results of the experiment.

Field Experiments: Field experiments occur when a researcher’s intervention takes place in subjects’ natural environment. One of the most common types of field experiments in political science is “get out the vote” experiments, where registered voters are randomly assigned to receive an encouragement to vote in some form (mailing, door hanger, phone call, etc.) and the turnout rate of the treatment group is compared to a group that did not receive the encouragement. However, there are many other clever and interesting manipulations that take place in the natural environment.

“Natural” Experiments: The term natural experiment is applied to studies where researchers take advantage of random assignments that occur naturally in the world. One of the most common examples of this is a lottery. Researchers often label their study as a natural experiment even if the events that happen to some people but not others are not determined in a truly random way, arguing that the process that led to the assignment of the “treatment” was administered as if it were random. While these are not true natural experiments, they can be leveraged in meaningful ways to help researchers understand a wider variety of phenomena.