What is a field experiment of religion?

In general terms, a field experiment of religion is an experiment conducted in the field that measures variation in religion in one of its components.

Experiments come in different flavors. They can be quasi-experiments or true experiments. True experiments, in turn, can be conducted in the laboratory or in a naturalistic setting. True experiments in a naturalistic setting can use laboratory-style or naturalistic treatments.

If an experiment uses a laboratory-style treatment in a naturalistic setting, it is termed a lab-in-the-field experiment. This uses a treatment created for a laboratory experiment but assigns it to participants in a natural setting. For example, Duhaime (2015) used an economic decision-making game similar to the dictator game as the experimental treatment. He gave it to shopkeepers to play in the markets of Marrakesh, Morocco. He tested if they played it differently as a function of when the call to prayer could be heard. He found that they demonstrated a more prosocial manner when the call to prayer was heard.

Field experiments that use a naturalistic treatment, in contrast, randomly assign the presentation of a phenomenon that naturally occurs in the world. For example, Ghumnan and Jackson (2010) randomly assigned college-aged female students to apply for jobs at retail stores and restaurants at local malls. They randomly assigned whether the students would wear a hijab. They found that the women were treated more negatively when the were wearing a hijab as compared to when they were not.

While quasi-experiments and lab-in-the-experiments play an important role in experimental research, they are not the focus of the Field Experiment of Religion Research Network (FERRN). Instead, FERRN promotes the use of field experiments with randomly assigned, naturalistic treatments.

Field experiments have four basic components (Gerber and Green 2012). The first component is the experimental intervention. It’s what the researcher randomly assigns. It necessarily has different levels, either by quantity or quality, thus creating a treatment and control group (or multiple treatment groups). The second component is the sample. This is the group of people who receive the experimental intervention. The third component is the setting. This is where the experimental intervention is delivered. This is the key difference between a lab experiment and a field experiment. Field experiments happen outside the laboratory in the real world. This supports the generalizability of their findings. The fourth component is the outcome. This is expected consequence of the experimental intervention. It can be measured in different ways, including survey questions, direct observation, or by having sample members perform some a measurement task.

Field experiments of religion incorporate variation of religion into one or more of these four components. Treatment groups can have different levels or qualities of religion. The sample members can vary in their religiousness. The setting can entail different aspects of religion. The outcome can measure religion.

The most straightforward approach to creating field experiments of religion is to incorporate religion into the experimental intervention. Easily overlooked, however, is that they can also be done with the other components as well. A field experiment of religion does not need to have religion in the experimental intervention. It can also be incorporated into the sample, setting, or outcome.

Suppose, for example, that researchers wanted to design a field experiment to study the relationship between an education program and religion. To put religion into the experiment, they could randomly assign participation in an educational program that incorporates religion and one that doesn’t. To put religion into the sample, they could draw a sample of people of different religious characteristics, and then test if the effect of an educational program varied across these characteristics—even if the program studied had nothing to do with religion. To put religion into the setting, the educational program could be offered at different locations, some of which are religious settings. Note: if the setting is randomly assigned, it becomes another experimental intervention. Finally, they could incorporate religion into the outcome by asking religion questions in the follow-up survey given after the educational program.

This approach to research design creates a wide range of possibilities for field experiments of religion.

References

Duhaime, Erik P. 2015. “Is the Call to Prayer a Call to Cooperate? A Field Experiment of the Impact of Religious Salience on Prosocial Behavior.” Judgement and Decision Making 10(6): 593-96.

Gerber, Alan S., and Donald P. Green. Field Experiments: Design, Analysis, and Interpretation. WW Norton, 2012.

Sonia Ghumman and Ann Marie Ryan. 2013. “Not Welcome Here: Discrimination towards Women Who Wear the Muslim Headscarf.” Human Relations 66(5): 671-98.

Audit Studies

This is perhaps the most straightforward approach to conducting a field experiment which accounts for its popularity.

The sample members in audit studies are people who control some resource or opportunity. The experimental intervention is a request for or inquiry about this resource. The researcher measures how sample members respond to the request or inquiry depending on how it was presented.

Sample members for audit studies include resource-controllers such as employers, landlords, car dealers and bankers. These people serve as gatekeepers for resources such as jobs, an apartment, a car, or a loan.

Researchers present their request or inquiry using normative means. Job application are sent to employers. Inquiry letters are sent to landlords. Questions are asked of car dealers. Loan applications are submitted to bankers.

Variation in the request or inquiry can be done in various ways. The name of the requester can be changed; the appearance can be changed; the wording can be changed.

An audit study of religion randomly assigns some aspect of religion to the request or inquiry.

Names can be used to signify religion only if a religion has a distinctive, widely recognized set of names. For the most part, this is limited to using names to study discrimination against Muslims. Arabic-sounding names are assumed to associate with the Islamic religion. Likewise, religious garb can be randomly assigned—again if it’s distinctively and widely recognized. This creates a wider range of studies, for many religions have distinctive garb—especially of their leaders. Catholic clergy, Buddhist monks, Orthodox Jews, Islamic women wearing hijabs are all examples of clothing associated with a religion. The same principle holds for religious-themed jewelry or other forms of religious symbols.

The content of the request can also indicate religion. Wright et al. (2013) sent resumes to employers. They embedded a religion reference into body of the applicants’ resumes. The resumes listed leadership experience in college, and they included having leadership positions in religious student organizations. Taking this approach, they were able to test the impact of 7 different types of religious standing: atheism, Catholicism, evangelical Christianity, Judaism, Islam, paganism, and a fictitious religious group. An eighth group was the control group, which made no mention of religion. Using this approach, they found that simply having the word “Muslim” embedded in a job application reduced callback rates by one-third as compared to the control group.

Religious Organizations

A second general strategy for conducting religion field experiments is to randomly assign the activities of a religious organization. These activities are interventions. In some way the religious organization is trying to change the people who are participating in the activities—whether they be in the organization or outside of it. If the organization provides the treatment, then the researcher only needs to randomly assign it and measure outcomes to create a field experiment of religion.

A straightforward way of experimenting with organization activities is to randomly assign participation in them to some people and not to others. This was the approach of Bryan, Choi, and Karlan (2018). They conducted a randomized evaluation of an evangelical Protestant Christian values and theology training program in the Philippines. They found that that it resulted in significant increases in religiosity and income, had no effect on food security or life satisfaction, and it decreased perceived relative economic status.

Opportunities for this type of experiment exists whenever a religious organization has a waiting list for its offerings. A waiting list means that more people want to participate in the activity than the organization can accommodate. If the organization is willing to randomly assign who gets to participate in the activity, those who receive it are the treatment group and those who don’t are the control group.

A variation of this approach is to assign all participants to the treatment group but randomly assign the timing of their participation. This allows for the comparison of the early recipients with the later recipients. This approach was adopted by Rink (2017) for research in South Sudan. A Christian missionary group allowed Rink to randomize the order of their evangelistic efforts by village. Rink found that the missionary activities lowered group-level social capital while increasing individual-level pro-social behavior. This approach of randomly assigning the timing of participation might be acceptable to organizations that want to study the impact of their activities but don’t want to exclude any possible participants.

Existing Field Experiments

A third approach to field experiments is to add them as an outcome into existing, non-religious field experiment. Field experiments test the impact of some cause on some effect in the social world. Scholars have theorized extensively about the relationship between phenomenon in the social world and religion. That means that just about any treatment studied in a field experiment has a conceptual connection with religion. If religion can be added as an outcome to the study, it becomes a field experiment of religion.

As a hypothetical example, imagine a field experiment designed to test the effect of a health intervention on children’s educational outcomes. As conceived of by its researchers, it has no religion component to it. Then, a religion researcher contacts them and places several questions relevant to children’s religiosity to the outcome survey. The field experiment now tests the impact of health on religion, a research question long studied with observational data.

A variation of this approach is termed a “downstream” experiment. It was developed by Green and Gerber (2002). It takes an existing field experiment and uses the outcome of that experiment as the treatment for a next experiment. If a treatment in the first study is randomly assigned, then there is randomly generated variation in the outcome that it produces. This random variation can be reinterpreted as a randomly assigned treatment to which outcomes can be added.

To illustrate how this might work, consider the previous example of a study that randomly assigned a health training program and measured its effects on education. After it’s completed, if heath has a significant effect on education, the sample becomes a group of people for whom random variation in education has been introduced. A later study could follow-up on this sample and measure aspects of their religiosity. This would produce a field experiment of the effects of education on religion, another topic well-studied with observational data and in need of field experimentation.

References

Bryan, Gharad T., James J. Choi, and Dean Karlan. 2018. “Randomizing Religion: The Impact of Protestant Evangelism on Economic Outcomes.” No. w24278. National Bureau of Economic Research, 2018.

Green, Donald P., and Alan S. Gerber. 2002. “The Downstream Benefits of Experimentation.” Political Analysis 10(4): 394-402.

Rink, Anselm F. Essays on Inequality and Social Cohesion. PhD diss., Columbia University, 2017.

Wright, Bradley R.E., Michael Wallace, John Bailey, and Allen Hyde. “Religious Affiliation and Hiring Discrimination in New England: A Field Experiment.” 2013. Research in Social Stratification and Mobility 34: 111-26.