BUS 6533 Module 5 Intervention Evaluation Plan Example

Did the Path Widen? Planning the Evaluation of Five Access Programs in a Midwestern Grocery Chain

Student Name

American College of Education

BUS6533: Advancing Access and Engagement in Organizations

Module 5 Assignment

Instructor Name

September 3, 2029

Introduction

The previous paper designed five interventions for the chain at the center of this course, Prairie Fresh Markets, to open leadership roles to more of its hourly staff and lift engagement: a redesigned leadership course offered on every shift and in Spanish, open postings with accountable review, bilingual mentoring, a steadier scheduling pilot and regular manager contact for overnight crews. Programs of this kind are often launched and never tested. This paper plans how the chain will find out whether those five changes actually opened doors and lifted engagement, with designs strong enough to support decisions about continuing, changing or ending each one.

A Logic Model

The evaluation begins with a logic model that states how each intervention is expected to work. Inputs such as paid course time and scheduling software support activities such as course sessions, panel reviews and mentoring meetings. These produce outputs, including enrollments by group and promotions through open postings. Short-term outcomes are increased access to development and supervisor support, as measured in the third paper. Medium-term outcomes are narrower gaps in promotion and engagement between groups. Long-term outcomes are lower turnover and stronger store performance. Making these links explicit lets the evaluation test each step rather than only the end result.

Evaluation Questions

Five questions guide the plan. Did the interventions reach part-time, overnight and Spanish-speaking associates as intended? Were they delivered as designed? Did access to development and supervisor support rise for those groups? Did gaps in promotion and engagement narrow? Did the scheduling pilot affect turnover, absenteeism and store performance? The first two are process questions and the last three are outcome questions; both are needed, because an intervention that fails may have been poorly delivered rather than poorly designed.

Design: A Staggered Rollout

Randomly assigning associates to receive or not receive access to promotion would be unethical and impractical. The plan instead uses the rollout schedule. Grant and Wall (2009) argued that quasi-experimental designs, used carefully, can provide strong evidence in organizations where randomization is not possible, especially when designs include comparison groups and multiple time points. The open posting process and the redesigned course will start in two regions in the first half of the year and in the remaining two regions six months later. Comparing changes in the early regions with changes in the later regions over the same period separates the interventions' effects from chain-wide trends.

Design: The Scheduling Pilot

The scheduling pilot allows a stronger design. Eight pilot stores will be matched with eight comparison stores of similar size, location type and baseline turnover, and the evaluation will use a difference-in-differences approach, comparing changes before and after the pilot in both sets of stores. Kesavan et al. (2022) used a randomized design at Gap stores, which the chain cannot fully replicate, but matching and pre-period data reduce the risk that differences reflect the stores chosen. Shadish et al. (2002) noted that the credibility of such designs depends on whether the comparison group would have followed the same trend without the intervention, which pre-period data allow the evaluators to check.

Outcome Measures

Outcome measures come from existing systems and the engagement survey. Promotions to department lead and assistant manager will be tracked by work arrangement, shift and language each quarter, along with course enrollments and completions and applications through open postings. The nine-item Utrecht scale and the two resource scales will be repeated annually using the same administration methods as the baseline survey. Turnover, absenteeism and schedule changes will come from HR and scheduling systems, and sales per labor hour from store reports for the scheduling pilot.

Process Measures

Process measures show whether each intervention was delivered as designed. For the course: sessions held by time slot and language, and attendance by group. For open postings: the share of openings posted, the share reviewed by the panel and the use of structured interviews. For mentoring: matches made and meetings held. For scheduling: the share of schedules posted two weeks ahead. For overnight contact: the share of months in which store managers worked an overnight shift. Low fidelity would explain weak results and guide improvements before conclusions are drawn.

Threats to Validity

Several threats are anticipated. Other changes, such as a new wage policy or a tight labor market, could affect outcomes across regions; the staggered design helps control for these. Regions may differ in ways that affect results; pre-period data will be compared. Survey responses may shift because associates know changes are under way; using the same validated scale and administration reduces but does not remove this risk. And small numbers of promotions in some groups will make quarterly estimates unstable, so conclusions will rest on annual totals.

Ethical Safeguards

The evaluation will protect associates. Survey results will continue to be reported only for groups of at least 20. Promotion data will be analyzed by group, not by individual, and access to individual records will be limited to HR analysts. Associates in later rollout regions will receive the interventions within six months, so no group is denied them for long. Results, including disappointing ones, will be shared with associates in both languages, consistent with the commitment made when the baseline survey was conducted.

Reporting and Decision Rules

Every quarter the steering group will receive the results, and once a year the executive team will. Decision rules are set in advance. An intervention will continue if outcomes improve and fidelity is adequate. It will be revised if fidelity is low, since the problem may be delivery. It will be ended or redesigned if fidelity is adequate but outcomes do not improve after eighteen months. For the scheduling pilot, the chain will expand to all stores if turnover falls and sales per labor hour do not decline relative to comparison stores.

Timeline

The evaluation runs for two years. Baseline data, already collected in the third paper, are supplemented in the first month with promotion and turnover records for the prior two years. The first regions begin the course and open postings in month two, the scheduling pilot starts in month three and the later regions follow in month eight. The engagement survey is repeated in months twelve and twenty-four. The first full outcome report goes to the executive team at month thirteen and the final report at month twenty-five.

Costs of Evaluation

The evaluation is designed to be inexpensive. Most data come from existing systems, the annual survey replaces the vendor pulse survey and the staggered rollout requires only scheduling. Added costs are about $45,000 a year for analyst time and survey administration, roughly 11 percent of the interventions' cost. Kalev et al. (2006) found that responsibility structures, which include regular review of data, were associated with progress, which means the review cycle itself may help produce change, not only measure it.

Conclusion

The evaluation plan links each intervention to outcomes through a logic model, asks process and outcome questions, uses a staggered rollout and a matched difference-in-differences design to separate effects from other changes and draws measures from existing records and the validated engagement survey. Threats to validity are anticipated, associates are protected and decisions follow rules set in advance. With this plan, the chain can learn whether the path to leadership has widened and whether wider access has raised engagement.

References

Grant, A. M., & Wall, T. D. (2009). The neglected science and art of quasi-experimentation: Why-to, when-to, and how-to advice for organizational researchers. Organizational Research Methods, 12(4), 653-686. https://doi.org/10.1177/1094428108320737

Kalev, A., Dobbin, F., & Kelly, E. (2006). Best practices or best guesses? Assessing the efficacy of corporate affirmative action and diversity policies. American Sociological Review, 71(4), 589-617. https://doi.org/10.1177/000312240607100404

Kesavan, S., Lambert, S. J., Williams, J. C., & Pendem, P. K. (2022). Doing well by doing good: Improving retail store performance with responsible scheduling practices at the Gap, Inc. Management Science, 68(11), 7818-7836. https://doi.org/10.1287/mnsc.2021.4291

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.