RES 4353 Module 3 Critical Appraisal of a Study Example

A Photograph of a Filled Syringe as the Outcome: Appraising the Validity of a 2024 Randomized Trial of Health Literacy-Informed Discharge Counseling

Student Name

American College of Education

RES4353: Evidence-based Health Education and Literacy

Module 3 Assignment

Instructor Name

October 19, 2026

Why This Study

The literature search in the previous module found eight studies relevant to a community health center pharmacy's proposal to change how liquid medicines are dispensed to families. Of those, the trial by Carroll et al. (2024) deserves the closest reading. It is the most recent randomized trial, it measured dosing at home rather than in a clinic, and it tested a bundle of the same elements the pharmacy is considering: a pictogram-based instruction sheet, teach-back, in which caregivers state back what they were taught, and show-back, in which they demonstrate drawing the dose into an oral syringe. If the pharmacy relies on one study to justify the change, this is likely to be it, so its validity matters.

The appraisal is organized by the five domains of RoB 2, the Cochrane group's updated bias tool for randomized trials: the randomization process, deviations from the intended interventions, missing outcome data, measurement of the outcome and selection of the reported result (Sterne et al., 2019). It then considers external validity, whether the results would hold in a different setting.

The Study in Brief

The trial took place at a single U.S. tertiary children's hospital from June 2021 to August 2022. English- and Spanish-speaking caregivers of children aged 6 years or younger who were discharged with a new, scheduled liquid medicine were randomized one to one, 99 to each group, to the counseling bundle or standard discharge counseling. Two to three days after discharge, caregivers used a mobile application to send a photograph of the syringe filled with their child's next dose. Two trained staff members who did not know group assignment estimated the measured volume. The mean dosing error was 1.0% in the intervention group and 3.3% in the standard group, an absolute difference of 2.3 percentage points (95% CI, 1.0 to 3.6). Of those with an outcome, 30.4% of the intervention group and 54.2% of the standard group measured an incorrect dose.

Randomization

Allocation used permuted blocks of four generated through the study's electronic data system, which is a sound method when the next assignment is concealed from recruiters. Blocks of a fixed small size carry one known risk: staff who know the block size can sometimes predict the last assignment in a block, and the article does not say whether recruiters were aware of it. More telling is a baseline imbalance. On the health literacy screening measure, 57.6% of intervention caregivers scored in the adequate range compared with 42.4% of the standard group. Randomization can produce such differences by chance, especially in a trial of this size, but here the imbalance favors the intervention on the very characteristic the intervention targets. The authors anticipated this possibility and planned a weighted analysis, discussed below. This domain is best judged as some concern.

Deviations From the Intended Interventions

Caregivers and the people delivering counseling could not be blinded, since one group received a visibly different conversation. That is unavoidable in education trials and matters mainly if knowing the group changed behavior in ways other than the intervention itself. A more practical concern is who delivered the bundle: the study's principal investigator, a pediatric hospitalist, or a trained research assistant, with each instruction sheet customized by hand. Standard counseling was given by the usual discharge staff. The comparison is therefore between a carefully delivered research intervention and routine practice, which tests what the bundle can achieve under good conditions rather than what busy staff will achieve with it. This does not bias the comparison as run, but it limits what the result means for everyday use.

Missing Outcome Data

This is the domain of greatest concern. The primary outcome was available for 151 of the 198 caregivers, 76.3%, well below the 90% the investigators planned for when they calculated their sample size. The analysis is described as following intention-to-treat principles, but it included only caregivers who completed the photograph upload, which makes it a complete-case analysis. The authors report that baseline characteristics were similar between those who completed follow-up and those who did not and that loss was similar between groups, which is reassuring. Still, caregivers who struggled most with the medicine may have been the least likely to photograph it, and that could differ by group in ways baseline measures do not capture. When nearly a quarter of the outcomes are missing, the question is not only how many are gone but who they were. Judgment: some concern.

Measuring the Outcome and Reporting the Result

Measurement was a strength. The outcome was observed rather than self-reported, the two assessors were blinded to group, and they agreed on the exact volume in 93% of photographs, with a third reviewer resolving differences. Two limits remain. Caregivers knew they would photograph the dose, which may have made them more careful than on an ordinary night, in both groups. And every caregiver received the smallest appropriate syringe, which the authors note placed an upper bound on how large an error could be.

On reporting, the trial was registered in advance, and the prespecified weighted analysis that adjusted for baseline imbalances was reported alongside the main result. That analysis matters: after weighting, the difference in mean dosing error fell from 2.3 to 1.5 percentage points, with a confidence interval of 0.1 to 2.9. The effect remained statistically significant, but it was smaller and less certain, consistent with part of the unadjusted result reflecting the health literacy imbalance. Judgment for measurement: low risk; for reporting: low risk.

Setting the Result Beside Earlier Evidence

The size of the effect looks modest next to the earliest trial in this literature. Yin et al. (2008) reported that, for daily-dose medicines, errors beyond the 20% threshold fell from nearly half of caregivers under usual counseling to about one in twenty with pictogram-based counseling. The 2024 trial's headline difference of a few percentage points in mean error is not directly comparable, because it averages all doses, including the many measured correctly, rather than counting large errors. Its secondary result, 30.4% against 54.2% measuring an incorrect dose, is closer in form and points the same way. The two trials together suggest that the benefit is real across settings more than a decade apart, while the smaller adjusted estimate in the newer trial is a caution against expecting the dramatic early result in routine practice.

Overall Judgment and External Validity

Taken together, the trial has some concerns about bias, chiefly from missing outcome data and a baseline imbalance, but its core finding holds in the adjusted analysis and agrees with the experiments and quality improvement work summarized in the previous module. Internal validity is reasonably good. External validity is more limited for the pharmacy's purposes. The trial took place at one academic children's hospital, most caregivers were White, follow-up lasted only two to three days, and the bundle was delivered by research staff at discharge, not by pharmacists at a community counter. The study therefore supports adopting the three elements, but it cannot tell the pharmacy how large the benefit will be in its own setting, which is a reason to measure dosing accuracy locally after the change.

References

Carroll, A. R., Johnson, J. A., Stassun, J. C., Greevy, R. A., Mixon, A. S., & Williams, D. J. (2024). Health literacy-informed communication to reduce discharge medication errors in hospitalized children: A randomized clinical trial. JAMA Network Open, 7(1), Article e2350969. https://doi.org/10.1001/jamanetworkopen.2023.50969

Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., Emberson, J. R., Hernán, M. A., Hopewell, S., Hróbjartsson, A., Junqueira, D. R., Jüni, P., Kirkham, J. J., Lasserson, T., Li, T., ... Higgins, J. P. T. (2019). RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ, 366, Article l4898. https://doi.org/10.1136/bmj.l4898

Yin, H. S., Dreyer, B. P., van Schaick, L., Foltin, G. L., Dinglas, C., & Mendelsohn, A. L. (2008). Randomized controlled trial of a pictogram-based intervention to reduce liquid medication dosing errors and improve adherence among caregivers of young children. Archives of Pediatrics & Adolescent Medicine, 162(9), 814-822. https://doi.org/10.1001/archpedi.162.9.814