LEAD 6523 Module 3 Evaluation Design Example

Did the Visits Prevent the Calls? An Evaluation Design for a Rural Community Paramedicine Program Using RE-AIM, a Matched Comparison and an Interrupted Time Series

Student Name

American College of Education

LEAD6523: Planning, Evaluation, and Accountability

Module 3 Assignment

Instructor Name

October 26, 2026

Introduction

The logic model in Module 2 ended by flagging three links as both vital and unproven: whether enough frequent callers enroll and stay, whether visits reduce calls and readmissions and whether fewer calls by enrolled patients free ambulance capacity for emergencies. This paper designs an evaluation to test them. It uses a public health planning framework to organize questions, a comparison design to estimate effects, a time series to look at county-wide capacity and a set of indicators with their data sources, and it is candid about what the design cannot rule out.

Organizing Questions With RE-AIM

Glasgow et al. (1999) offered RE-AIM as a way to weigh an intervention's real-world value along five lines: reach, the proportion of eligible people who join and how closely they mirror those who do not; effectiveness, meaning what it changes for them; adoption, meaning how many sites and staff actually take it on; implementation, how faithfully it is delivered; and maintenance, whether effects and delivery last. Their argument was that an intervention's impact depends on all five, not on effectiveness alone. Applied here, RE-AIM turns the three links from Module 2 into five sets of questions.

Evaluation Questions

Reach: What share of the 72 identified frequent callers enroll, and how do enrolled patients differ from those who decline? Effectiveness: Do enrolled patients make fewer 911 calls and emergency department visits, and have fewer 30-day heart failure readmissions, than comparable patients who are not enrolled? Adoption: Do all four stations' crews refer patients from the field, and does the hospital refer at discharge? Implementation: Are post-discharge visits made within 72 hours and scheduled visits completed as planned? Maintenance: After the grant's second year, do effects and visit volume hold, and has a payment source been secured?

Design for Effectiveness

Randomly assigning frequent callers to receive or not receive visits would give the strongest evidence, but county leaders and crews consider it unacceptable to withhold the program from people who ask for it, and the number of eligible patients is small. The design therefore compares enrolled patients with a matched group of frequent callers who are eligible but not yet enrolled, either because capacity is full in the first months or because they decline, matched on age, primary diagnosis and call frequency in the year before. Each patient's calls, emergency department visits and readmissions are compared for the twelve months before and after enrollment, or a matched start date for comparison patients. The comparison will show whether enrolled patients improve more than similar patients who were not visited, which is the claim the program rests on. If more patients are eligible than the program can serve in its first year, a waiting list ordered by date of referral rather than by staff choice will create the comparison group fairly and transparently.

Design for Ambulance Capacity

The question of whether fewer calls free capacity requires county-level data. The evaluation will treat monthly county call volume, average response time to the most urgent calls and the number of hours with no ambulance available as an interrupted time series, with 36 months of data before the program and 24 months after. A change in the level or trend after the program starts, beyond what the prior pattern predicts, would suggest an effect, though other changes in the county could produce the same pattern; quasi-experimental designs of this kind can support causal claims only when such alternative explanations are examined one by one (Shadish et al., 2002).

Indicators and Data Sources

Indicators come from four sources. The dispatch system provides calls per patient, call times and ambulance availability. The hospital provides emergency department visits and 30-day readmissions for consenting patients, linked by a data-sharing agreement. The program's own visit records provide enrollment, visit timing, medication reviews and referrals. And a short questionnaire at enrollment and six months, adapted from a validated heart failure self-care measure, provides short-term self-management outcomes. Each indicator has a defined numerator and denominator, for example readmissions within 30 days per 100 heart failure discharges among enrolled patients.

The Cost Question

The county commission and the foundation will ask whether the program is worth its cost, so the evaluation includes a simple cost analysis. Program costs, staff time, vehicle, training and supplies, will be tracked monthly. Avoided costs will be estimated from the difference between enrolled and comparison patients in ambulance transports, emergency department visits and readmissions, valued at the average costs reported by the ambulance service and the hospital. The analysis will report both the cost per enrolled patient and the estimated cost avoided per patient, with a range rather than a single figure, because the effect estimates will be uncertain. A finding that the program roughly pays for itself in avoided transports would matter a great deal to whether a payer funds it after the grant.

Threats to Validity

Three threats deserve attention. Regression to the mean is the largest: frequent callers are selected because they called often in a bad year, and many would call less the next year anyway; the matched comparison group addresses this, since it is selected the same way. Selection bias is the second: patients who decline the program may differ in ways matching cannot capture, such as motivation. History is the third: a new clinic hire or a hospital readmission initiative during the evaluation period could change outcomes regardless of the program. The evaluation will record such events and interpret results with them in mind.

Timeline and Roles

Baseline data for all 72 frequent callers will be pulled before the first enrollment. The program's data coordinator, a quarter-time role funded by the grant, will compile dispatch and visit data monthly; the hospital's quality office will supply readmission data quarterly under the data-sharing agreement; and a faculty member from the regional university's public health program has agreed to review the analysis plan and the final results as an outside evaluator, which adds credibility with the commission. Interim results will be reported at twelve months and final results at twenty-four.

Standards for the Evaluation

The public health evaluation framework calls for evaluations to meet standards of utility, feasibility, propriety and accuracy (Centers for Disease Control and Prevention, 1999). This design aims at utility by answering the questions the county commission and foundation have asked, at feasibility by relying mostly on data already collected, at propriety through patient consent and a data-sharing agreement that protects privacy and at accuracy through matching and a time series rather than simple before-and-after comparisons.

Conclusion

The design tests the program's central claims with the strongest methods the setting allows: RE-AIM to keep reach, adoption, implementation and maintenance in view, a matched comparison to estimate effects on patients and an interrupted time series to look at county-wide capacity. It will not prove that visits caused fewer calls, but it can show whether enrolled patients improved more than similar patients, which is the evidence the county needs to decide whether to keep the program. Because the commission will decide on continued funding at the end of the second year, the design is built to produce a clear answer by then, even if that answer is that the effect is smaller than hoped.

References

Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR-11), 1-40. https://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. https://doi.org/10.2105/AJPH.89.9.1322

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.