RES5303 Module 3 research design paper example

No Control Group Available: Matching a Design to the Question of Whether Text-Message Preoperative Instructions Reduced Same-Day Surgery Cancellations

Student Name

American College of Education

RES5303: Research Methods and Applied Statistics in Healthcare

Module 3 Assignment

Instructor Name

September 20, 2027

The Question and the Constraint

A 250-bed hospital, a composite invented for this assignment, performs about 700 scheduled surgeries a month. Before March of last year, patients received preoperative instructions on paper at their preadmission visit, covering fasting times, which medications to hold and what to bring. Same-day cancellations, surgeries canceled on the day of the procedure, averaged 6.1 percent of scheduled cases, and a review attributed about a third of them to patient preparation problems such as eating too close to surgery or taking an anticoagulant that should have been held. On March 1, the hospital replaced paper instructions with a sequence of text messages sent seven days, two days and the evening before surgery, for every surgical patient at once.

The question is whether the text-message sequence reduced same-day cancellations. The constraint is that the change applied to everyone on the same date. No surgical patient continued to receive paper instructions only, and no comparable hospital in the system made a similar change at a different time. The design has to answer a causal question without the thing most designs rely on, a group that did not receive the intervention.

Design One: A Randomized Trial

For a cause-and-effect question, the design of first choice is a randomized controlled trial, in which patients would be randomly assigned to text messages or paper instructions and cancellation rates compared. Randomization would balance known and unknown differences between the groups, so any difference in cancellations could be attributed to the instructions. Shadish et al. (2002) describe randomized experiments as the design that most directly supports causal inference, because random assignment rules out selection as an explanation for differences in outcome.

For this question the trial is no longer possible. The hospital has already changed its process for every patient, and reverting half of them to paper instructions to test the change after the fact would be difficult to justify to patients, surgeons and the ethics committee. A trial could have been done before the change, and the lesson for future decisions is that the best time to evaluate a change is before it is made universal.

Design Two: A Simple Before-and-After Comparison

The easiest design is to compare the cancellation rate in the twelve months before March 1 with the rate in the twelve months after. Suppose the rate fell from 6.1 percent to 4.8 percent. That comparison would be simple to compute and easy to present, but it could not separate the effect of the text messages from anything else that changed over the same period. Cancellation rates might already have been falling because of a new anesthesia screening clinic that opened the previous autumn, or they might vary by season, rising in winter with respiratory infections.

A single before-and-after comparison treats all of the months before the change as one number and all of the months after as another, which hides whatever trend was already under way. It is the weakest of the designs considered here, and it is the design most often used in practice.

Design Three: A Comparison Group After All

A controlled before-and-after design would find a group that did not receive the change and compare the change in its cancellation rate over the same period. Two candidates were considered. Patients having urgent add-on surgery do not receive the text sequence because they are scheduled too late, but they differ from scheduled patients in almost every way that affects cancellation, so they are a poor comparison. A sister hospital in the same system still uses paper instructions, but its surgical mix and cancellation rate differ, and it opened a new ambulatory surgery wing during the same year.

Neither comparison group is close enough to the hospital's scheduled surgical population to make a fair control. Using one anyway would introduce a new source of bias while appearing to solve the old one. The honest conclusion is that no adequate comparison group exists, which is the situation the module asks about.

Design Four: An Interrupted Time Series

An interrupted time series uses the population's own history as its comparison. Instead of two averages, it uses many measurements at regular intervals before and after the change and asks whether the level or the trend of the outcome changed at the moment of the intervention more than the preexisting pattern would predict. Lopez Bernal et al. (2017) describe interrupted time series as one of the strongest quasi-experimental designs for evaluating interventions introduced at a known point in time to a whole population, particularly when randomization is not possible.

For this question, the series would use monthly same-day cancellation rates for 24 months before March 1 and at least 12 months after, 36 data points in all. The analysis would use segmented regression, which estimates the baseline level and trend, the immediate change in level at the intervention and the change in trend afterward (Wagner et al., 2002). Seasonality would be addressed by including terms for calendar month, and the opening of the anesthesia screening clinic, which occurred at a known earlier date, would be modeled as a separate change so its effect is not credited to the text messages. A time series does not create a control group; it makes the past behave like one.

Threats That Remain

The interrupted time series is the strongest available design here, but it is not immune to error. Its main remaining threat is history: any other event that occurred at about the same time as March 1 and affected cancellations would be confounded with the text messages. The research team would need to check hospital records for changes in surgical scheduling, staffing or policy around that date. A second threat is a change in how cancellations were recorded; if the reasons for cancellation were coded differently after March 1, the outcome would change for reasons unrelated to patients. A third is that monthly rates for about 700 cases carry random variation, so small changes may not be detectable.

One refinement would strengthen the design further. The outcome most directly linked to the intervention is not all cancellations but those due to patient preparation. Analyzing preparation-related cancellations as a separate series, alongside cancellations for other reasons such as equipment or surgeon illness, would provide an internal check: if the text messages worked, preparation-related cancellations should fall while other causes remain unchanged.

Conclusion

The question of whether text-message instructions reduced same-day surgery cancellations cannot be answered with a randomized trial, because the change is already universal, and it is answered poorly by a simple before-and-after comparison, because that design cannot separate the change from existing trends. No adequate comparison group exists. An interrupted time series with segmented regression, seasonal adjustment, a term for the earlier screening clinic and a separate series for preparation-related cancellations is the strongest design available. Matching the design to the question means starting from what cannot be done and choosing the best of what remains.

References

Lopez Bernal, J., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348-355. https://doi.org/10.1093/ije/dyw098

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Wagner, A. K., Soumerai, S. B., Zhang, F., & Ross-Degnan, D. (2002). Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics, 27(4), 299-309. https://doi.org/10.1046/j.1365-2710.2002.00430.x