RES 6023 Module 2 Validity Threats Analysis Example

What Could Make the Answer Wrong? Threats to Internal and External Validity in a Stepped-Wedge Evaluation of Text-Message Immunization Reminders

Student Name

American College of Education

RES6023: Quantitative Research Designs

Module 2 Assignment

Instructor Name

October 19, 2026

Introduction

Module 1 recommended a stepped-wedge design for testing whether texting parents before each due date lifts on-time series completion at our county health department. In that design, the order in which our four clinics begin sending reminders is decided at random, and every clinic eventually takes part. A design is only as good as its defense against the things that could make its answer wrong. This paper examines threats to internal validity, whether an observed difference can be credited to the reminders, and to external validity, whether the result would hold elsewhere, and then two further kinds of validity that are easy to overlook.

Why a Stepped Wedge Changes the Threats

In a parallel randomized trial, both groups move through the same calendar time, so events in the outside world affect them equally. A stepped wedge is different. Hemming et al. (2015) explained that because clinics switch from control to intervention at different times, the intervention periods fall later in calendar time than most control periods, and any change over time can be confused with the effect of the intervention. Time is therefore a confounder that the analysis must model explicitly. Their guidance shapes much of what follows: several of the classic threats reappear in a new form because of the staggered start.

History

History refers to events outside the study that affect the outcome. A measles outbreak in a neighboring state, a change in school-entry requirements or a national news story about vaccine safety could all shift completion rates during the study. In a stepped wedge, a history event that strikes in the second year will fall mostly on clinics already sending reminders, making the reminders look better or worse than they are. The response is twofold: the analysis will include calendar period as a fixed effect, as Hemming et al. (2015) recommended, and I will keep a log of local and national events with possible effects on vaccination, so that unexpected shifts can be examined after the fact.

Secular Trends and Maturation

Completion rates may be rising or falling on their own, a secular trend, independent of the reminders. Our registry data show year-to-year movement in completion, and a steady trend would be confounded with the stepped rollout in the same way as a history event. Maturation, in the classic sense of participants changing over time, applies here as children age toward their 24-month measurement point, but every child is measured at the same age, which removes that threat for the outcome itself. The period effects in the model address secular trends, and three years of registry data from before the study will show the trend's shape in advance.

Selection

Selection threatens validity when groups differ before the intervention in ways that affect the outcome. Randomizing the order of clinic start protects against deliberate choice, such as starting with the clinic whose staff are most enthusiastic, but with only four clinics randomization cannot guarantee balance. If the clinic serving the most rural families happens to start last, the late control periods will be weighted toward families who face the greatest barriers. The response is to adjust for clinic as a random or fixed effect and for child-level characteristics such as insurance type, language and distance from the clinic, and to report baseline completion by clinic so readers can judge balance.

Instrumentation

Instrumentation threats arise when the measurement itself changes. If Nevada's registry changes how it records doses given at pharmacies, or a large private practice begins reporting more completely partway through the study, completion would appear to rise for reasons unrelated to reminders. Using the registry rather than clinic records already helps, and I will ask the state registry office to notify the department of any reporting changes during the study period.

Regression to the Mean and Testing

Harris et al. (2006) warned that interventions are often started after an unusually bad period, so later improvement may simply be a return toward the average rather than an effect. The department decided on reminders after a year in which completion dipped, and a naive before-and-after comparison would be vulnerable to exactly this. Randomizing clinic order and modeling several years of baseline data protect against it, since the comparison is between clinics in the same periods, not between one bad year and the next. Testing, in which the act of measurement changes behavior, is a minor threat here because the registry records doses without contacting families.

Attrition

Families who move out of state, or switch to providers who do not report to the registry, will disappear from the data. If reminders lead some families to seek vaccines at pharmacies that report less reliably, attrition could differ between conditions. The analysis will report the number of children lost in each period and compare their characteristics with those retained, and a sensitivity analysis will treat lost children first as incomplete and then as complete to show how much the result depends on them.

Contamination

Contamination occurs when the control condition receives part of the intervention. Families sometimes visit more than one of our clinics, and a family enrolled at a clinic already sending reminders could carry the habit or the messages to a sibling's appointment at a control clinic. Assigning each family to the clinic of its first visit, and counting it there throughout, will keep the comparison clear, and the number of families using more than one clinic will be reported.

External Validity

External validity concerns whether the results would hold for other people, places and times. Our county's mix of rural and small-city families, a large Spanish-speaking population and mostly Medicaid-insured children differs from the urban population that Stockwell et al. (2012) studied in New York, where text reminders produced a modest gain in influenza vaccination. That difference cuts both ways. A positive result here would extend the evidence to rural and mixed settings, but it might not transfer to counties with better broadband coverage or fewer barriers. Reporting the setting in detail, and examining whether the effect differs by language and distance, will let other health departments judge relevance for themselves.

Statistical Conclusion and Construct Validity

Two further kinds of validity deserve attention. Statistical conclusion validity concerns whether the analysis can detect a true effect and avoid false ones. With only four clinics, the effective sample size is limited by clustering, since children at the same clinic tend to have similar outcomes, so the power analysis in Module 4 must account for the intraclass correlation. Construct validity concerns whether the study measures what it claims. The intervention is text reminders, but if some families' phones are disconnected or their prepaid plans run out, they never receive the messages. The study will record delivery receipts to separate the effect of being assigned reminders from the effect of receiving them.

Conclusion

The stepped-wedge design protects against deliberate selection, but its staggered timing makes history and secular trends the leading threats, which the analysis will address by modeling calendar period. Small clinic numbers, attrition from the registry, contamination between clinics and undelivered messages each have a specific response. Module 3 turns to the instruments and data sources that will measure the study's variables. None of these threats is fatal, but each must be named in the proposal.

References

Harris, A. D., McGregor, J. C., Perencevich, E. N., Furuno, J. P., Zhu, J., Peterson, D. E., & Finkelstein, J. (2006). The use and interpretation of quasi-experimental studies in medical informatics. Journal of the American Medical Informatics Association, 13(1), 16-23. https://doi.org/10.1197/jamia.M1749

Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015). The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ, 350, Article h391. https://doi.org/10.1136/bmj.h391

Stockwell, M. S., Kharbanda, E. O., Martinez, R. A., Vargas, C. Y., Vawdrey, D. K., & Camargo, S. (2012). Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: A randomized controlled trial. JAMA, 307(16), 1702-1708. https://doi.org/10.1001/jama.2012.502