RES 6512 Module 4 Methodology Overview Example

Two Semesters, One Rubric, Blinded Raters: A Methodology Overview for a Quasi-Experimental Study of Debriefing and Clinical Judgment

Student Name

American College of Education

RES6512: Research Concept Paper

Module 4 Assignment

Instructor Name

November 2, 2026

Introduction

The framework set out in Module 3 predicts that structured reflection after one scenario will improve a student's clinical judgment in the next. This overview describes how the study will test that prediction. It covers the design and the reasons for it, the participants and the number needed, the instrument and how raters will be prepared, the procedures for both debriefing conditions, the planned analysis and the ethical protections required when a researcher studies students in the program where they work. Each section names the threat to validity it is meant to reduce, since the design's main weakness, the lack of random assignment, has to be answered by care everywhere else.

Design and Rationale

A quantitative quasi-experiment with nonequivalent comparison groups is planned, and a pretest used as a covariate (Shadish et al., 2002). Students in one semester will receive usual debriefing, and students in the following semester will receive Debriefing for Meaningful Learning. Random assignment of individual students was considered and rejected. Students in the same course share simulation days and talk to one another, so a structured method used with half a cohort would likely spread to the other half, and faculty cannot easily switch methods from one group to the next within a day. Assigning whole semesters avoids that contamination at the cost of comparability. The early-semester rubric score, the use of the same scenarios and raters in both semesters and the documentation of usual debriefing are the main defenses against selection and history threats.

Population and Sample

The population is second-year associate degree nursing students at the college's two campuses. Each semester roughly 72 students enroll in the medical-surgical course that carries the program's simulation hours. All enrolled students who consent will be included; students repeating the course will be excluded, because they have already experienced the scenarios.

An a priori power analysis in G*Power (Faul et al., 2007) for analysis of covariance with two groups and one covariate, a medium effect size of f = 0.25, an alpha of .05 and power of .80 indicated a total sample of 128. Two semesters should provide about 144 eligible students. Allowing for roughly 10% who decline or withdraw, the expected sample of about 130 meets the requirement with little margin, so a third semester has been approved by the program director as a contingency if consent rates fall short.

Instrument and Raters

Clinical judgment will be measured with the Lasater Clinical Judgment Rubric, whose 11 indicators are each rated at one of four levels, giving total scores from 11 to 44 (Lasater, 2007). Evidence from three studies supports its reliability and validity for simulation, while also showing that agreement depends on how well raters are trained (Adamson et al., 2012). Two experienced nursing faculty from outside the program will rate every recording. They will complete a training session using practice videos and will not begin scoring until their agreement on a calibration set reaches an intraclass correlation of at least .75, the threshold commonly described as good reliability (Koo & Li, 2016). Raters will not know which semester a recording comes from, since uniforms, room setup and scenario scripts will be identical, and file names will be coded.

Procedures

In week 3 of each semester, every student will complete a baseline scenario, recorded and later scored to provide the covariate. Across weeks 4 through 13, students will complete the course's six regular scenarios, each followed by debriefing in the condition assigned to their semester. In week 14 they will complete the outcome scenario, a deteriorating postoperative patient, which will be recorded and scored.

Fidelity will be checked in both conditions. Before the intervention semester, the faculty who debrief will complete Debriefing for Meaningful Learning training, and a sample of their debriefings will be observed against a checklist of the method's phases (Dreifuerst, 2015). In the comparison semester, a short form completed after each debriefing will record its length and the topics covered, so that usual practice can be described rather than assumed.

Data Analysis

Analysis will begin with screening for missing data and outliers and with descriptive statistics for both groups. The first research question will be tested with analysis of covariance, using the early-semester total score as the covariate. Before interpreting results, I will check the assumptions of the test: a linear relationship between covariate and outcome in each group, homogeneity of regression slopes through a group-by-covariate interaction term, normally distributed residuals and equal error variances by Levene's test. If slopes are not homogeneous, the analysis will report the interaction and use a moderated regression instead. The second research question will repeat the analysis for each of the four rubric dimensions, judged against a Bonferroni-adjusted alpha. Every result will be reported with adjusted means, a 95% confidence interval for the difference and partial eta squared, so that the size of any effect is visible and not only its significance.

Ethical Considerations

The study will be reviewed by the American College of Education institutional review board and by the college's own research committee. Because I coordinate the simulation center, students could feel pressure to take part. To reduce that pressure, a faculty member with no grading role will present the study to students and hold the signed consent forms, which I will not see until final grades are posted. Participation will not affect grades, and students who decline will complete every scenario and debriefing as usual; only their recordings will be excluded from the analysis. Recordings will be stored on an encrypted college server, coded by number and destroyed three years after the study ends. Both debriefing methods are accepted educational practice, so no student is denied a recognized standard of instruction.

Threats Not Fully Controlled

Some threats remain. Students in the second semester will be admitted a term later, and a change in admission criteria, in the faculty mix or in a prerequisite course could make the groups differ in ways the baseline score does not capture; I will compare the groups on prerequisite grades and age and report any differences. The faculty who debrief may grow more skilled across the year regardless of method, which would favor the intervention semester. Students in the second semester might hear about the first semester's scenarios from friends, although the outcome scenario will not be used in regular teaching. Finally, raters cannot be blinded to the debriefing itself, but they never see it, because only the scenarios are recorded for scoring. These threats will be named in the limitations of the final study rather than explained away.

Feasibility

The study fits within existing course activities: the scenarios, recording equipment and simulation hours already exist. The new costs are faculty training in the debriefing method, which the college's professional development fund has agreed to cover, and rater time, estimated at about 60 hours across both semesters. The main risk to completion is the sample size, and the contingency semester addresses it.

Conclusion

The methodology tests the framework's prediction with a design suited to a working nursing program. Whole semesters are compared to prevent contamination, a baseline score and identical scenarios reduce the effects of nonequivalent groups, trained and blinded raters protect the measure, fidelity checks describe both conditions and the analysis plan includes assumption checks and effect sizes. Module 5 assembles the problem, purpose, framework and this overview into the complete concept paper.

References

Adamson, K. A., Gubrud, P., Sideras, S., & Lasater, K. (2012). Assessing the reliability, validity, and use of the Lasater Clinical Judgment Rubric: Three approaches. Journal of Nursing Education, 51(2), 66-73. https://doi.org/10.3928/01484834-20111130-03

Dreifuerst, K. T. (2015). Getting started with debriefing for meaningful learning. Clinical Simulation in Nursing, 11(5), 268-275. https://doi.org/10.1016/j.ecns.2015.01.005

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). GPower 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39*(2), 175-191. https://doi.org/10.3758/BF03193146

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012

Lasater, K. (2007). Clinical judgment development: Using simulation to create an assessment rubric. Journal of Nursing Education, 46(11), 496-503. https://doi.org/10.3928/01484834-20071101-04

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.