HRM 5463 Module 3 Performance Evaluation Methods and Rating Bias Analysis Example

Whose Opinion Counts? Comparing Evaluation Methods and Guarding Against Rating Bias at a Wisconsin Resort

Student Name

American College of Education

HRM5463: Employee Relations and Performance Management

Module 3 Assignment

Instructor Name

November 29, 2027

Introduction

Last module's performance plan for our resort includes semiannual written reviews that determine pay steps for housekeeping and banquet staff. If those reviews are inaccurate or biased, the plan will repeat the unfairness that Module 1 found employees already resent. This paper compares five evaluation methods, examines the rating biases most likely in this workplace, reviews research on rater effects and training, and designs an evaluation process with safeguards. It also proposes how the resort will check whether ratings are fair across groups of employees.

How Much Ratings Reflect the Rater

The starting point is a sobering finding. Scullen et al. (2000), analyzing ratings of managers by bosses, peers and subordinates, found that idiosyncratic tendencies of individual raters accounted for more of the variation in ratings than the actual performance of the people being rated. In other words, a rating often says as much about who gave it as about who received it. For a resort where each housekeeper is rated by one supervisor, and where Module 1 found that supervisors make assignment decisions by personal impression, that finding argues for methods that rely less on a single rater's judgment and more on observable evidence and multiple viewpoints.

Five Methods Compared

Graphic rating scales, which ask supervisors to rate traits such as dependability from one to five, are simple but vague, leaving room for each supervisor's own interpretation. Behaviorally anchored rating scales describe specific behaviors at each level, for example how a housekeeper responds when a guest asks for extra towels, which narrows interpretation but takes time to build. Objective measures, such as inspection pass rates and room credits from Module 2, are hard to dispute but miss behaviors like teamwork. Forced ranking compares employees against one another and would undermine the shared team goals set in Module 2. Multisource feedback adds peers, inspectors or guests, broadening the view at the cost of more effort.

What Research Says About Multisource Feedback

Multisource feedback is popular but not a cure. Combining studies that tracked people after they received ratings from several sources, Smither et al. (2005) found that performance improvements were generally small, and were more likely when the feedback showed a need to change, when recipients reacted positively and believed change was needed, and when they set goals in response. For frontline hospitality jobs, a full multisource survey would be burdensome. A limited version, combining the supervisor's rating with an independent inspector's scores and guest comments, captures the main benefit, multiple viewpoints, without the cost.

Biases Most Likely Here

Several rating biases are likely at the resort. Leniency, rating everyone high to avoid conflict, is common where supervisors work alongside the people they rate. Halo occurs when one strong impression, such as speed, colors ratings on unrelated dimensions like teamwork. Recency gives too much weight to the last few weeks before a review. Similarity bias favors employees who resemble the rater. And there is a specific risk here: supervisors who speak only English may rate employees with limited English lower on communication or guest interaction, even when their guest scores are strong, confusing language fluency with competence. Any of these could distort pay decisions. Central tendency, rating nearly everyone as average to avoid hard conversations, is also likely among supervisors who have never been trained to rate.

The Chosen Design

The semiannual review will combine three parts. Half the rating comes from objective measures already collected under Module 2: inspection pass rates, room credits or setup accuracy, and attendance. A quarter comes from a behaviorally anchored scale covering four dimensions, teamwork, guest interaction, safety and response to feedback, with anchors written in English and Spanish and based on examples gathered from housekeepers and supervisors. The final quarter comes from a second rater: for housekeeping, the independent inspector who scores rooms; for banquets, the event captain. Guest comments that name an employee are attached to the review as evidence, but are not scored, because guests comment unevenly.

Training Raters

Rater training matters, but its type matters too. Woehr and Huffcutt (1994), in a quantitative review of rater training studies, found that frame-of-reference training, in which raters practice until they agree on what each dimension means and how each level of performance appears, was the most effective approach for improving rating accuracy. Supervisors and inspectors will spend three hours together scoring identical short videos and written examples, compare their ratings with the agreed anchors and discuss differences. A refresher will be held before each review cycle, and new supervisors will complete the session before rating anyone.

Calibration, Audits and Appeals

Three further safeguards follow. First, before ratings are final, supervisors and inspectors in each department will meet for a calibration session to compare ratings and discuss outliers, reducing individual idiosyncrasy. Second, HR will audit ratings each cycle by department, by sex and by primary language, comparing scores on the behavioral scale with objective measures; if one group consistently receives lower behavioral ratings than its objective scores would predict, the ratings will be reviewed. Third, any employee may appeal a rating to HR within ten days, with the review conducted by a manager from another department and an interpreter available.

A Worked Example

To test the design, I applied it to one anonymized housekeeper from last year's records. This employee's inspection pass rate averaged 96 percent, the room credit standard was met on 91 percent of shifts and there was one unexcused absence, which together earn a strong objective score. The supervisor's old review had given a 2 out of 5 on communication, noting that the housekeeper rarely speaks in meetings. Under the new design, the behavioral anchor for guest interaction asks whether guest requests are handled promptly and courteously, and guest comments named this housekeeper positively three times. The inspector's rating was also high. The overall score would rise from below average to well above it, showing how the old single-rater review penalized limited English rather than performance. Running a few past cases through a new design before launch is a cheap way to find problems.

Costs and Limits

The design requires more effort than a single supervisor's form: about four hours of rater training per person per year, a calibration meeting per department each cycle and HR time for audits, an estimated 120 staff hours a year in total. It also has limits. Behavioral anchors must be revised as jobs change, objective measures can be gamed if workers rush to meet credits, and audits by language group require collecting primary language information carefully and with consent. These costs are justified by the stakes: pay decisions that employees see as fair, and a reduction in the favoritism complaints that drove turnover.

Conclusion

Research shows that ratings often reflect raters as much as performance, which is why the resort's semiannual reviews will rely half on objective measures, use behaviorally anchored scales in two languages, add a second rater and pass through calibration, bias audits and an appeal process. Frame-of-reference training will give raters a shared standard. The next module will address what happens when performance falls short and discipline is needed.

References

Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85(6), 956-970. https://doi.org/10.1037/0021-9010.85.6.956

Smither, J. W., London, M., & Reilly, R. R. (2005). Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Personnel Psychology, 58(1), 33-66. https://doi.org/10.1111/j.1744-6570.2005.514_1.x

Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189-205. https://doi.org/10.1111/j.2044-8325.1994.tb00562.x