1. Finding Subjects

All research designs pose unique challenges. One of the biggest in experimental designs is the need to recruit subjects to take your study. In an ideal world, every experiment would be conducted on a random sample of the population of interest to achieve both strong internal and external validity. But one of the main benefits of experimentation is that you can achieve strong internal validity on a convenience sample. Therefore, experiments disproportionately rely on samples that are not randomly selected and researchers make no claim that their sample is representative.

Although reliance on non-probability convenience samples is widely accepted, it is still a considerable amount of effort to identify and recruit subjects to participate in your study. To assist you in this process, the SSRMC implements the Omnibus Project every semester. The project coordinates and streamlines the development of a student subject pool (a convenience sample) for faculty and student research. Project coordinators collect data for a set of common variables, such as demographic information and political covariates, which are provided to all researchers submitting proposals. Individual researchers submit their survey questions and customize their own portion of the survey instrument. The subject pool is comprised of students enrolled in departmental classes; some instructors will require participation or offer extra credit to students to participate in the study.

Another cost-effective option is to use Amazon’s Mechanical Turk (MTurk) service, a website that allows researchers to publish tasks (HITs or Human Intelligence Tasks) and provide payment to subjects who choose to participate. Those who request a task can limit the availability of the task to respondents who meet certain qualifications, such as age or location. Studies using samples from Mechanical Turk have been published in the top journals in political science and have been found to replicate important experimental findings in psychology. While MTurk is still a convenience sample, it is more representative of adult populations than undergraduate samples or samples populated from those who respond to web advertisements.

If you have used the above samples to collect pilot results that are promising tests of your hypotheses, with the support of a faculty member, you can put together a proposal for the Time Share Experiments in the Social Sciences. This project, sponsored by the National Science Foundation, allows researchers to submit experimental proposals for consideration to be fielded on a representative sample of American adults on an Internet survey platform.

2. Delivering the Treatments

First and foremost, studies involving human subjects always require approval from the Institutional Review Board (IRB). Applications to the IRB must include documentation that you have successfully completed the ethics training mandated by the federal government. Depending on the nature of your experiment, your study may be exempt from full review of the board, but that is a decision made by the chair of the IRB, not by you as the researcher. You should allow at least three weeks for your study to be approved before you can collect data.

Political scientists tend to conduct three main types of experiments: survey experiments, lab experiments (including lab-in-the-field experiments), and field experiments.

If you want to do a survey experiment, the simplest option is Qualtrics, a software program for which the College maintains a subscription for both computer-based and mobile platforms (for experiments you want to do remotely). Qualtrics can be programmed in very sophisticated ways to randomly assign subjects to different treatments.

One of the drawbacks of survey experiments that subjects take in the comfort of their own environment is a lack of control on the part of the researcher in controlling that environment. For example, if you are conducting your survey on an Internet sample, subjects are able to browse the Internet or walk away from the computer while they are taking your study. This is most problematic if there is reason to think that some facet of your treatment might make subjects more likely to get distracted or visit other webpages, for example if your treatment is very long or if you ask post-test questions about political knowledge, where subjects might feel inclined to seek out the correct answers online. Programs like Qualtrics have some built in functionalities to be able to detect this.

However, in some instances, you may want to conduct a survey experiment in the laboratory, either to exert more control over the experiment or because you want to deliver a treatment that is not well suited to the online or phone format. In that instance you may want to use the research lab facilities of the SSRMC. If you are doing a lab experiment and you are interested in delivering media (images, audio, or videos) to subjects in a laboratory environment, one of the most popular software programs to do so is SuperLab. The SSRMC currently has licenses for this program on one computer. An example honors thesis that relies on SuperLab for stimulus delivery can be found here.

If you are interested in doing an experiment in the field, academics frequently partner with outside organizations—such as campaigns or advocacy organizations—because it is often difficult to get access to the large subject pools necessary to conduct these studies. William & Mary undergraduates have successfully done this, and this honors thesis is a great example.