Validity in Quasi-Experimental Research
This module discusses internal and external validity in quasi-experimental research studies and addresses threats to validity.
- Define internal and external validity.
- Describe threats to validity in quasi-experimental research.
- Discuss ways to address threats to validity and provide examples.
There are two types of validity concerns: internal validity and external validity. Below is a link to a video, Validity, that provides an excellent introduction to the topic of validity concerns in research. The video describes internal and external validity, discusses threats to validity, and offers examples. Click here to watch the video.Validity concerns are very important in quasi-experimental research because the research designs lack the same level of control mechanisms as true experimental designs, thereby raising questions regarding the validity of the research findings.
Internal validity refers to the validity of the findings within the research study. It is primarily concerned with controlling the extraneous variables and outside influences that may impact the outcome. This is important in experimental studies and quasi-experimental studies that are attempting to demonstrate causation to ensure that the experimental treatment (X) is, in fact, responsible for a change in the dependent variable (Y). Internal validity is critical if the study is going to be able to determine a causal relationship. Therefore, the researcher must plan to control or eliminate the influence of other variables in order to be confident when making conclusions about the relationship between X and Y. For example, if a researcher wanted to determine if there was a causal relationship between increasing physical activity and lowering cholesterol levels, he or she would need to consider other factors that impact cholesterol levels and attempt to eliminate those influences in the test group. This is considerably more difficult in quasi-experimental studies than in experimental studies, primarily because participants are not randomly selected and therefore, it is more difficult to control the extraneous variables that may influence the findings.
External validity refers to the extent to which the results of study can be generalized or applied to other members of the larger population being studied. External validity is concerned with real life applications that have relevance beyond the confines of the experiment. In true experimental studies, the random selection of participants and random assignment of the study participants into groups ensures that the members of the study are truly representative of the larger population. Random selection is really the key ensuring that results are generalizable. In the physical activity/cholesterol example, the researcher would want to be sure that race, gender, age, BMI, and other factors that may differ among test subjects did not influence the results. Again, random selection of participants should control for these influences. Quasi-experimental studies present challenges because participants are NOT randomly selected. Construction of a reasonably similar control group is key and the matching methods described in the previous module may be used help minimize this issue. The best evidence for external validity is whether or not the research findings can be reproduced with different populations, settings, or times. Researchers should keep replication in mind when designing the study to ensure external validity.
Factors that have the potential to influence the findings or the generalizability of the findings are called threats to validity. Following are lists of threats to internal and external validity.
Threats to Internal Validity
- History – This refers to unplanned events that may occur during a study that impact the results unintentionally. Test subjects often have different experiences as the study progresses that may have an influence. For example, if doing a pretest and a post-test assessment at the beginning and end of a semester for two different classrooms to compare test results, one group may have a different classroom atmosphere or dynamic that influences the post-test results.
- Maturation – Natural changes, biological or psychological, within the participants over the time of the study may impact the results. Test subjects may become bored, tired, hungry, and so forth during the time of the study. This is more of an issue with long-term studies.
- Testing – Experiments that pretest the subjects may influence the performance of subjects on subsequent tests simply due to the fact that participants have already seen or completed the test before. People tend to perform better at any activity the more they are exposed to it.
- Instrumentation – Changes in testing instrumentation during a study may affect what is being measured and how it is measured. Similarly, if human observations are involved, the observations or perceptions of the of the observers may change over time, rather than the actual performance of the test subjects.
- Statistical Regression – Statistical regression, or regression to the mean, can be a concern in studies with extreme scores, either particularly high or low. Scores are typically not as extreme in subsequent testing in most situations, making meaningful pretest and post-test comparisons more difficult.
- Selection – If the subjects placed into the groups are selected in a non-random manner or are functionally inequivalent at the beginning of the study, the results of the study will be biased when making comparisons between the groups at the end.
- Experimental Mortality – Test subjects drop out of studies for a variety of reasons. The loss of participants from comparison groups may impact the study if the withdrawal or mortality rate is higher in one group or if it is particularly high in both groups. For example, a larger number of participants may drop out of a group due to illness while several less motivated participants drop out of the other group. The groups no longer have a similar make up of individuals.
- Selection Interaction – The selection method may interact with one or more of the other threats and impact results. For example, groups with larger numbers of elderly participants may be impacted more by maturation during the study.
Threats to External Validity
- Interaction Effects of Testing – The pretest may make the participants more aware of or sensitive to the treatment that will be applied and therefore, may influence the response to the treatment.
- Selection Bias – This occurs when subjects are selected in a manner that does not ensure that they are representative of the overall population. The random selection of subjects is a critical factor in determining external validity.
- Reactive Effects of Experimental Testing – The fact that treatments in a controlled, laboratory setting may differ from those in a less controlled, real world environment. The performance of the subjects may actually be more due to the setting than the independent variable.
- Multiple Treatment Interference – When subjects receive more than one treatment, the effects of previous treatments may influence responses. Early treatments may have a cumulative effect on how subjects perform or respond.
No experimental research project is perfect or free from potential threats to validity, and it is even more important in quasi-experimental studies because the lack of random selection of test subjects creates uncertainty. Researchers must take the necessary steps to ensure that the threats are controlled as best as possible. Careful selection of the appropriate research design and attention to control methods is critical. Following are examples of threats to validity in the most common quasi-experimental designs (discussed in previous modules) and how those threats may be addressed.
- Pre-test/Post-test No Control Group Design – A researcher is studying the impact of a nutrition education program on a group of overweight teenagers and plans to weigh subjects prior to the program and then again after 6 months. If the group loses weight, can the researcher be sure it is due to the education program? Internal validity threats may include a maturation effect, a history effect and a testing effect. The best way to address these threats may be to add a control group to the design to compare results. This would provide additional control to ensure that something like a change in the school lunch program is not also contributing to the results. Perhaps growth spurts of teenagers at that age is contributing to the weight loss. A control group would allow for comparison with other teenagers to rule out the impact of these types of variables. A simple Pre-Test/Post-Test design with no control group typically creates the largest validity concerns.
- Non-equivalent Control Group Design – A researcher is looking at the incidence of depression in divorced individuals ages 40-50 years old. The incidence of depression in this group will need to be compared to the general population in that age category. The biggest threat to validity in this design is in the selection of the test subjects and creation of the control group. Matching methods will need to be used to create a comparison group that has similar demographics to ensure that the depression is not due to income levels, health conditions or other factors. If a valid control group can be created, the Non-equivalent Control Group Design is typically considered the strongest of the quasi-experimental design. While the groups may not be equivalent on potential variables, this design does control the other validity threats better than other quasi-experimental designs.
- Time-Series Designs – A study is being done to examine the impact of a new campus crime prevention program on new student enrollment. The researcher would collect several years worth of data regarding the crime rate and new student enrollment prior to the implementation of the program and several years of data following the implementation of the new program. The major threats to validity in this study are the history effect and perhaps maturation. The researcher would have to examine other campus events or activities that may be impacting new student enrollment during this time, but the Time-Series Design does control for other types of validity threats quite well.
The following YouTube video, Quasi-Experiments, provides a more detailed discussion regarding threats to validity in quasi-experimental research designs, with an emphasis on selecting the appropriate research design to maximize validity and reduce threats.
- Bernard, H. R., & Bernard, H. R. (2012). Social research methods: Qualitative and quantitative approaches. Sage.
- Brown, L. (2010). Quasi-experimental research. Doing Early Childhood Research: International perspectives on theory and practice, 345.
- Campbell, D. T., & Stanley, J. C. (2015). Experimental and quasi-experimental designs for research. Ravenio Books.
- Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.
- Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into practice, 39(3), 124-130.
- Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research (Vol. 19). Sage.
- Maxwell, J. (1992). Understanding and validity in qualitative research. Harvard educational review, 62(3), 279-301.
- Punch, K. F. (2013). Introduction to social research: Quantitative and qualitative approaches. Sage.
- William R.. Shadish, Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage learning.
Probability and Statistics > Sampling > Stratified Random Sample
Watch the video or read the steps below:
Stratified Random Sampling: Definition
Stratified random sampling is used when your population is divided into strata (characteristics like male and female or education level), and you want to include the stratum when taking your sample. The stratum may be already defined (like census data) or you might make the stratum yourself to fit the purposes of your research. Stratified random sampling is very similar to random sampling. However, these samples are more difficult to create as you must have detailed information about what categories your population falls into.
The stratum in this map are defined by EPA region. Image: USGS
How to Perform Stratified Random Sampling
To perform stratified random sampling, take a random sample from within each category or stratum. Let’s say you have a population divided into the following strata:
- Category 1: Low socioeconomic status — 39 percent
- Category 2: Middle class — 38 percent
- Category 3: Upper income — 23 percent
To get the stratified random sample, you would randomly sample the categories so that your eventual sample size has 39 percent of participants taken from category 1, 38 percent from category 2 and 23 percent from category 3. What you end up with is a mini representation of your population. According to University of California at Davis, the following steps should be taken to obtain the stratified sample:
- Name the target population.
- Name the categories (stratum) in the population.
- Figure out what sample size you need.
- List all of the cases within each stratum.
- Make a decision rule to select cases (for example, you might select the items using the largest set of random numbers).
- Assign a random number to each case.
- Sort each case by random number.
- Follow your decision rule (#5 above) to choose your participants.
Stratified random sampling for larger data sets is usually performed using statistical software. For example, click here for the procedures in SAS.
How to Get a Stratified Random Sample: Example
Stratified random sampling is useful when you can subdivide areas. Image: Oregon State
“Stratified” means “in layers,” so in order to get a stratified random sample you first need to make the layers. What layers you have depends on characteristics of your population. For example, if you are surveying U.S. residents about their plans for retirement, you might want your layers to represent different age groups. The sample size for each strata (layer) is proportional to the size of the layer:
Sample size of the strata = size of entire sample / population size * layer size.
How to Get a Stratified Random Sample: Steps
Sample question: You work for a small company of 1,000 people and want to find out how they are saving for retirement. Use stratified random sampling to obtain your sample.
Step 1: Decide how you want to stratify (divide up) your population. For example, people in their twenties might have different saving strategies than people in their fifties.
Step 2: Make a table representing your strata. The following table shows age groups and how many people in the population are in that strata:
|Age||Total Number of People in Strata|
Step 3: Decide on your sample size. If you don’t know how to find a sample size, see: Sample size (how to find one). For this example, we’ll assume your sample size is 50.
Step 4: Use the stratified sample formula (Sample size of the strata = size of entire sample / population size * layer size) to calculate the proportion of people from each group:
|Age||Number of People in Strata||Number of People in Sample|
|20-29||160||50/1000 * 160 = 8|
|30-39||220||50/1000 * 220 = 11|
|40-49||240||50/1000 * 240 = 12|
|50-59||200||50/1000 * 200 = 10|
|60+||180||50/1000 * 180 = 9|
Note that all of the individual results from the stratum add up to your sample size of 50: 8 + 11 + 12 + 10 + 9 = 50
Step 5: Perform random sampling (i.e. simple random sampling) in each stratum to select your survey participants.
That’s how to get a stratified random sample!
Tip: Each element in your population should only fit into one stratum. In other words, one person cannot be in more than one group.------------------------------------------------------------------------------
If you prefer an online interactive environment to learn R and statistics, this free R Tutorial by Datacamp is a great way to get started. If you're are somewhat comfortable with R and are interested in going deeper into Statistics, try this Statistics with R track.Comments are now closed for this post. Need help or want to post a correction? Please post a comment on our Facebook page and I'll do my best to help!