RES 6023 Module 3 Instrument Selection Paper Example

Measuring Completion, Hesitancy and Message Receipt: Selecting Data Sources and Instruments With Reliability and Validity Evidence for a Text-Reminder Immunization Study

Student Name

American College of Education

RES6023: Quantitative Research Designs

Module 3 Assignment

Instructor Name

October 26, 2026

Introduction

The stepped-wedge study planned in Modules 1 and 2 needs to measure three things well: whether each child completes the recommended vaccine series by 24 months, how hesitant each parent is about vaccines and whether each family actually receives the text reminders. The first is the outcome, the second is a key covariate and possible moderator and the third separates assignment to reminders from exposure to them. This paper selects a data source or instrument for each, examines the evidence that it measures consistently and accurately and plans how to fill the gaps that evidence leaves.

Reliability and Validity in Brief

A measure is reliable when repeated use under similar conditions yields similar scores. For a multi-item scale, internal consistency is usually reported with Cronbach's alpha. Tavakol and Dennick (2011) cautioned that alpha depends on the number of items and on the scale measuring a single construct, so a high value does not by itself show that a scale is good, and values above about .90 may signal redundant items. Validity concerns whether the evidence backs the meaning we plan to give the scores. It is not a single property but a body of evidence, including content, response processes, internal structure, relationships with other variables and consequences, and every instrument here will be judged by what evidence exists for our use.

Outcome: The State Immunization Registry

Series completion will come from Nevada's statewide immunization information system rather than from clinic charts. The registry's main advantage is coverage: doses given at private practices, pharmacies and other counties' health departments are recorded in one place, so a child vaccinated outside our clinics is not wrongly counted as incomplete. Its reliability depends on consistent reporting by providers, and its validity depends on completeness and accuracy. Two weaknesses matter. Some providers report late or incompletely, and children who move out of state remain in the registry with records that stop, looking incomplete when they may be fully vaccinated elsewhere.

Three steps address these weaknesses. Before the study, I will compare registry records with clinic charts for a random sample of 100 children at our own clinics to estimate how often the registry misses doses we gave. During the study, children with no registry activity for 12 months and a returned-mail flag will be classified as lost rather than incomplete. And the state registry office will be asked to report any change in provider reporting during the study, the instrumentation threat identified in Module 2.

Covariate: Parent Vaccine Hesitancy

Parents who are hesitant about vaccines may respond to reminders differently, and hesitancy may differ across clinics, so measuring it helps both with balance and with understanding who benefits. The Parent Attitudes About Childhood Vaccines survey, known as the PACV, was built for this purpose. Its developers drew its items from the literature and from focus groups with parents and pediatricians, had six immunization experts rank them, dropped the lowest-ranked third and pretested the result with 25 parents, giving the survey evidence of content and face validity (Opel et al., 2011). The survey produces a score from 0 to 100, with higher scores indicating greater hesitancy.

The strongest evidence for my purpose concerns relationships with other variables. In a follow-up validation study of 437 parents in Seattle, PACV scores predicted their children's later immunization status: children of parents scoring 70 to 100 were underimmunized for 46.8% more days from birth to 19 months than children of parents scoring under 50. Scores at baseline and eight weeks later were highly concordant, ρ = .84, supporting test-retest reliability (Opel et al., 2013). That predictive evidence is exactly what this study needs, since the reason to measure hesitancy is its expected link to completion.

Gaps in the PACV Evidence

The PACV's evidence comes from English-speaking parents in an urban health system in the Pacific Northwest, and about half of our families prefer Spanish. Validity evidence does not travel automatically to a new language or population. I will therefore have the survey translated into Spanish by one bilingual translator, translated back into English by a second who has not seen the original and reconciled by a panel that includes two Spanish-speaking parents from our clinics. Cognitive interviews with eight Spanish-speaking parents will show whether parents read each item the way its authors meant, which is response process evidence. In the study itself, internal consistency will be reported separately for each language version, and I will check whether scores predict completion similarly in both groups.

Exposure: Delivery Receipts

Assignment to reminders is recorded when a clinic starts the program, but exposure requires that messages reach a working phone. The text-messaging platform records delivery status for every message, which will be the exposure measure. This is a log rather than a scale, but it still needs validity checks. A delivered status shows only that a carrier accepted the message, not that a parent read it. A short item on the parent survey, asking whether the parent remembers receiving a reminder about the child's shots in the past month, will provide a second, self-reported measure, and agreement between the two will be reported.

Respondent Burden and Timing

Measurement quality also depends on whether parents will answer at all. The PACV has 15 items and takes most parents only a few minutes, which keeps the full parent survey, with the reminder-recall item and a few demographic questions, under 10 minutes. Parents will be offered the survey at the child's first clinic visit during the study, on a clinic tablet before the visit or through a link sent by text, in English or Spanish. Measuring hesitancy at enrollment, before most reminders arrive, matters for validity: if reminders change attitudes, a later measurement would mix the covariate with the outcome. Parents who decline the survey will still be included in the main analysis through registry data, and their children's characteristics will be compared with those of responders.

Why Not Build a New Instrument?

Writing a new hesitancy survey tailored to rural Nevada families was tempting, but it would require its own development and validation study, adding a year or more before the main study could begin and leaving committee members with no outside evidence to judge it by. An existing instrument with published predictive evidence, adapted carefully and tested in our population, offers a stronger foundation. The same reasoning favors the registry over a new chart abstraction form, which would cover only our clinics and introduce a new source of abstractor error.

Other Variables

Child-level covariates, including date of birth, insurance type and preferred language, come from the clinic's registration system, which has been used for billing and reporting for years and is checked by staff at every visit. Distance from home to clinic will be calculated from addresses using mapping software, an objective measure with known limits for families whose mailing address differs from where they live.

Conclusion

The study will measure series completion through the state registry, checked against clinic charts; parent hesitancy through the PACV, chosen for its evidence of predicting immunization status and adapted to Spanish with formal translation and cognitive testing; and exposure through delivery receipts paired with a self-report item. Each choice rests on evidence, and each gap in that evidence has a plan. Module 4 uses these measures to size the sample.

References

Opel, D. J., Mangione-Smith, R., Taylor, J. A., Korfiatis, C., Wiese, C., Catz, S., & Martin, D. P. (2011). Development of a survey to identify vaccine-hesitant parents: The parent attitudes about childhood vaccines survey. Human Vaccines, 7(4), 419-425. https://doi.org/10.4161/hv.7.4.14120

Opel, D. J., Taylor, J. A., Zhou, C., Catz, S., Myaing, M., & Mangione-Smith, R. (2013). The relationship between parent attitudes about childhood vaccines survey scores and future child immunization status: A validation study. JAMA Pediatrics, 167(11), 1065-1071. https://doi.org/10.1001/jamapediatrics.2013.2483

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. International Journal of Medical Education, 2, 53-55. https://doi.org/10.5116/ijme.4dfb.8dfd