NUR5194 Module 5 project evaluation report example

Thirty-Eight Percent, Not Eleven: Evaluating One Patient, Four Questions Against Its Objectives, With a Comparison Group and an Honest Account of What the Data Cannot Prove

Student Name

American College of Education

NUR5194: Capstone Practicum for Role of the Nurse Educator

Module 5 Assignment

Instructor Name

December 5, 2028

Objectives and Methods

Module 3 committed the project to four targets for the intervention groups, to be judged at week twelve. More than a third of instructors' questions were to reach the analyze level, against a baseline of 11%. Students were to hold the floor for more than half the conference, against a baseline of 38%. Gains on the interpreting item of the clinical evaluation tool were to exceed those in the comparison groups. And seven in ten students were to find post-conference usually or always useful. The evaluation repeated the needs assessment's methods in weeks eleven and twelve: the author observed two post-conferences in each of the eight groups, coded every question by cognitive level and estimated student talk time, with the mentor independently coding four conferences; all students were invited to complete the same survey; and the mentor provided de-identified midterm and final evaluation scores.

Results

Question level. In the eight observed intervention conferences, instructors asked 187 questions, of which 71, or 38%, were at the analyze level or above, meeting the target of 35%. In the eight comparison conferences, 12% of 203 questions were at that level, essentially unchanged from the baseline of 11%. The mentor's independent coding agreed with the author's on 86% of questions.

Student talk time. Students in the intervention groups spoke for an estimated 58% of conference time, meeting the target of 55%, compared with 40% in the comparison groups.

Interpreting patient data. On the four-point clinical evaluation rating for interpreting patient data, the intervention groups' mean rose from 2.4 at midterm to 3.0 at the final evaluation, an increase of 0.6. The comparison groups' mean rose from 2.4 to 2.7, an increase of 0.3. The intervention groups' improvement was twice as large, meeting the objective as written. Both sets of students improved, as students do over a semester; the students who reasoned aloud about their patients twice a week improved about twice as much.

Student ratings. Of 31 intervention students who responded, 23, or 74%, rated post-conference as usually or always useful, meeting the target of 70%, compared with 12 of 29, or 41%, in the comparison groups and 17% at baseline across all groups.

Fidelity and Dose

Results can only be interpreted alongside how fully the intervention was delivered. Module 4 reported that adherence to the five-element checklist averaged 83% across observed conferences, rising from 75% in the first two weeks to 90% in the last two. The evaluation observations in weeks eleven and twelve therefore captured the structure at its most faithful, which may flatter the results slightly compared with a full semester's average. The dose students received was also uneven. Each student presented once or twice as the one patient, while every student took part in answering the four questions twice a week for six weeks, about 12 conferences in all. Students who missed clinical days, three in the intervention groups missed two or more, received less. When the author looked separately at the 29 intervention students who attended at least ten of the twelve conferences, their mean change on the interpreting item was 0.7, slightly larger than the group as a whole. The difference is small and the numbers are too few for firm conclusions, but the direction, more exposure with somewhat more improvement, is what would be expected if the structure contributed to the change. It also suggests that a full semester of the structure, rather than six weeks, might produce a larger effect, which the course can test when it adopts the structure from the start of the next term.

What Students and Instructors Said

The survey's open comments and a short debrief with the four intervention instructors added texture to the numbers. Students most often wrote that they liked hearing how classmates thought about the same findings and that the what-if question prepared them for things that might happen on their next shift. Several wrote that the conferences were harder than before and that they were more tired afterward, which the author takes as a sign of the intended cognitive effort. Two students wrote that they missed hearing about every patient. The instructors said the four questions made post-conference easier to lead, not harder, because they no longer had to invent questions at the end of a long day, and that waiting after a question was the skill they were still building. All four said they would continue using the structure.

Interpreting the Results by Level

Kirkpatrick and Kirkpatrick (2016) describe four levels at which training and education can be evaluated: reaction, learning, behavior and results. The project's measures touch three of them. At the level of reaction, students' and instructors' ratings were favorable. At the level of behavior, measured in the teachers rather than the students, instructors changed what they did: the proportion of higher-level questions more than tripled, and students talked more. At the level of results, which for this project means students' clinical performance, the change in the interpreting item favored the intervention groups. The project did not directly measure learning, such as students' performance on a clinical reasoning test, which would have strengthened the chain between the change in post-conference and the change in clinical ratings.

The pattern is consistent with the guiding theory. Tanner (2006) describes interpreting as the phase in which nurses make sense of what they have noticed, and the four questions gave students repeated practice at exactly that phase, in a group where their reasoning could be heard and corrected. The findings are also consistent with the evidence that structured reflection improves clinical reasoning after simulation (Dreifuerst, 2012), extended here, cautiously, to post-conference after real clinical days.

What the Data Cannot Prove

The design has limits that must be stated plainly. Groups were not randomly assigned; the intervention instructors volunteered and may be more engaged teachers, which alone could explain part of the difference. The author coded the observed conferences knowing which groups were which, so coding may have been biased toward the expected result, although the mentor's independent coding of a subset agreed closely. The clinical evaluation scores were given by the same instructors who led the intervention, and instructors who had invested in the change may have rated their students more generously. The groups are small, with eight students each, and no statistical test would give a reliable answer with these numbers. For these reasons, the results show that the structure is feasible, that it changed instructors' questioning and students' participation as intended and that the change in clinical ratings is promising, not that the structure caused better clinical judgment. A stronger test would assign groups at random next semester and use a clinical judgment rubric scored by raters who do not know the groups. The final module reports how the course has decided to proceed.

References

Dreifuerst, K. T. (2012). Using debriefing for meaningful learning to foster development of clinical reasoning in simulation. Journal of Nursing Education, 51(6), 326-333. https://doi.org/10.3928/01484834-20120409-02

Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. ATD Press.

Tanner, C. A. (2006). Thinking like a nurse: A research-based model of clinical judgment in nursing. Journal of Nursing Education, 45(6), 204-211. https://doi.org/10.3928/01484834-20060601-04