Rater Bias in Performance Reviews: What Actually Works

In short

Rater bias is a systematic distortion in performance ratings that comes from the rater, their standard or the situation rather than from performance. Familiar forms are leniency, severity, central tendency and halo. Often the largest is the idiosyncratic rater effect: in two large samples of developmental 360° ratings it explained 62% and 53% of the variance. A shared standard, frame-of-reference training, several raters and statistical correction can help, with a person making the final call.

Key takeaways

Rater bias comes from raters’ goals, incentives and relationships as well as perception, so teaching the list of biases is not enough.
In two large samples of developmental 360° ratings, each rater’s own way of using the scale explained more variance than actual performance did.
Ratings for pay or promotion are on average about one-third of a standard deviation more lenient than those for development or research.
Behavioural anchors create a shared standard but do not remove bias alone; the evidence mainly supports frame-of-reference training.
Several project leads per person usually make ratings more reliable, most of all when they judge independently; averaging does not cancel a bias they all share.
Statistics can detect and remove leniency and severity when raters overlap on the same people, but not a bias every rater shares; a person should decide.

What is rater bias?

Raters are not measuring instruments. They judge with a partial view of the work, their own standard and their own interests. Ratings track actual performance only weakly; Murphy offers this as one explanation and argues that raters need tools and incentives to rate accurately (Murphy, 2008). In professional-services firms this matters because promotion and bonus depend on these judgements.

Definition

Rater bias A systematic difference between a performance rating and the performance it describes, caused by the rater, their standard, their goals or the situation rather than by the person rated. It distorts comparisons most when people are rated by different managers.

Perception and memory. Raters see part of the work. Not just average performance but its trend and variability shape the rating (Reb and Greguras, 2010).
Goals and incentives. Ratings used for pay or promotion are on average about one-third of a standard deviation more lenient than those for development or research (Jawahar and Williams, 1997).
Relationships and effort. Costly information and close relationships with the people rated increase central tendency and leniency (Bol, 2011).

Types of rater bias: leniency, severity, central tendency, halo, recency, similar-to-me

Six biases cover most of practice. For each, ask how well it is supported: not everything in textbooks was measured at work.

An invented scenario, not a client case. Six well-known rater biases, one consulting example each, and how well each is supported (our classification, based on the sources listed below).
Bias	Example in a consultancy	What the evidence says
Leniency bias	A project lead rates almost everyone “exceeds expectations”, including work that had to be redone twice.	Well documented, especially when ratings drive pay or promotion (Jawahar and Williams, 1997).
Severity bias	A project lead never gives the top rating, not even for the best analysis on the team.	Measurable as a trait of individual raters (Myford and Wolfe, 2003).
Central tendency error	Nearly everyone gets the middle rating; you can no longer tell who stands out and who struggles.	Rises with costly information and close relationships; held back later improvement in one study (Bol, 2011).
Halo effect	One polished client presentation lifts the ratings for team leadership and analysis too.	Real, but not always an error, because criteria can genuinely go together (Murphy, Jako and Anhalt, 1993).
Recency bias	A strong finish in December outweighs a weak spring.	Not directly shown at work in the studies cited here; the trend and variability of performance do shape ratings (Reb and Greguras, 2010).
Similar-to-me bias	People who work closely with the lead, or resemble them, get better ratings.	Only a related finding from one call centre: ratings were higher in smaller teams, which the authors use as a proxy for closer social ties (Breuer, Nieken and Sliwka, 2013).

One finding cuts against the usual list: leniency is not always harmful. In one firm’s data, where central tendency held back later improvement, leniency was positively associated with future performance; that is a correlation, not proof of cause (Bol, 2011). The goal is not “zero leniency” but ratings that compare across raters.

The idiosyncratic rater effect: often the largest source of variance

Most lists miss what is often the largest effect: each rater uses the scale in their own way. Two large samples of 360° ratings covered 2,350 and 2,142 managers, seven ratings each. There, this idiosyncratic rater effect explained 62% and 53% of the variance; actual performance only 21% and 25% (Scullen, Mount and Goff, 2000). In those data, a rating said more about who gave it than about who received it.

The effect attaches more to the individual rater than to their role: in this sample, method variance was more strongly associated with the specific rater than with their level as boss, peer or subordinate (Mount et al., 1998). The mean inter-rater reliability of supervisors’ overall performance ratings was 0.52, and peer ratings were less reliable still (Viswesvaran, Ones and Schmidt, 1996).

Why several project leads per person help, and where that stops

In a consultancy, several project leads often rate one person over a year. If each lead errs in their own way, the quirks partly cancel out and the overall rating becomes more reliable.

More raters do not automatically mean less bias. In a laboratory experiment, groups of raters were more accurate than single raters, which the authors attribute to the group combining its different information rationally (Ockenfels, Sliwka and Werner, 2025). A lab cannot show whether real panels do the same. Four limits apply:

Different work. One lead sees the analysis, another the client work; part of their difference is real.
Shared bias. Averaging cannot fix a bias that all raters share, such as general leniency when bonuses depend on ratings.
Dependent judgements. Leads who know each other’s views before rating no longer err independently.
Close ties. In one call centre, ratings were biased upwards in smaller teams, which the authors read as a sign of closer social ties (Breuer, Nieken and Sliwka, 2013).

What works and what doesn’t: BARS, frame-of-reference training, calibration, statistics

Each remedy addresses a different cause, so order matters: a shared standard first, then training on it, then calibration and statistics. Start with statistics and you correct numbers whose meaning nobody agreed.

Five remedies for rater bias: what each does and where it stops (our classification, based on the sources cited).
Remedy	What it does	Where it stops
Behaviourally anchored rating scales (BARS)	Describe observable behaviour per rating step and criterion: a shared standard and a basis for feedback.	Little or no advantage over other formats on leniency, halo, agreement or accuracy; the compared studies had method problems (Kingstrom and Bass, 1981).
Frame-of-reference training	Raters practise on examples what each rating step means. Accuracy improves (Roch et al., 2012) and their picture of performance moves closer to the experts’ (Gorman and Rentsch, 2009).	Lasting field effects are not shown in the studies cited here, nor is lower leniency when pay is attached.
Several raters per person	Averages out individual quirks if judgements are independent.	No help against shared bias; real differences in perspective remain (Hoffman et al., 2010).
Calibration meeting	Aligns standards; in one study, calibration committees curbed leniency (Demeré, Sedatole and Woods, 2019).	The same study found more central tendency; see how to run a calibration meeting.
Statistical correction	Detects and removes each rater’s leniency or severity (Raymond and Viswesvaran, 1993).	Needs raters who overlap; cannot find a bias that all raters share.

Detecting and correcting leniency and severity statistically

When several people rate the same employees, you can estimate how far each rater sits above or below the others, and remove that shift. In a reanalysis of a medical oral exam with 40 examiners and about 115 candidates, such an adjustment would have changed pass or fail for about 7–9% of candidates, and reliability rose (Raymond and Viswesvaran, 1993). Many-facet Rasch models also flag, per rater, central tendency, randomness and halo (Myford and Wolfe, 2003).

Overlap. Raters must rate people whom others rate too. A lead who rates only their own team cannot be separated statistically from that team’s performance.
Enough ratings. Few ratings mean uncertain estimates. Show rater effects with a confidence interval and discuss only the clear ones.
Check the middle separately. Central tendency narrows the spread rather than shifting the level, so it needs its own check.
A proposal, not an automatic fix. The calibration meeting reviews each correction. Some differences are real, for example when one lead runs only the hardest projects.

The evidence also comes from exams; transferring it to business appraisals is our inference, not a study result. And correction makes ratings more comparable, not objective.

Rules to check in the EU and Germany

No decision based solely on automated processing (EU). Automated evaluation of performance at work is profiling under Article 4(4) GDPR. Article 22 gives people the right not to be subject to a decision with significant effects based solely on it. Recital 71 says the controller should use “appropriate mathematical or statistical procedures for the profiling”. In practice, the correction is a proposal and a person decides.
The works council (Germany). Setting up general appraisal principles needs its consent under § 94(2) BetrVG; the Federal Labour Court extends this to the design of the appraisal procedure (1 ABR 48/13). Standards and anchors therefore belong in that conversation. That statistical correction rules count too is our reading, not something the court has decided; raise them as well and have this checked. Software objectively suitable for monitoring performance can be subject to co-determination under § 87(1) no. 6 BetrVG (1 ABR 7/15). A tool that flags lenient raters also evaluates them.

A checklist for professional-services firms

Set the standard first. Describe observable behaviour for each criterion and rating step, with examples that separate a 3 from a 4. An invented scenario, not a client case. For “problem structuring”, 3 might read “breaks the problem down unaided but needs help prioritising” and 4 “breaks it down and prioritises unaided, and flags missing data early”.
Train on the standard, not the list of biases. Before the cycle, have project leads rate invented cases and discuss where they differ.
Be open about the purpose. Pay-linked ratings invite leniency, so keep developmental feedback separate from the appraisal that decides pay.
Rate independently. Each lead rates alone, before seeing other views, and records the basis: project, duration, role.
Rate close to the work. Rate at the end of each project, not only at year end.
Check the data before the meeting. Compare each lead’s mean and spread with colleagues who rated the same people.
Calibrate the top and the bottom. Review both ends of the distribution in the calibration meeting, because calibration can increase central tendency.
Decide in a way you can explain. A person decides on each correction and records why. For pay, see linking ratings to bonus.

What ValueScore does about rater bias, and where statistics stop

LEADBeyond built ValueScore for performance appraisals in consultancies and professional-services firms. Ratings are based on concrete behaviour, with anchors for each rating step and criterion and a nudge against score inflation. Leniency and severity are detected statistically and corrected for; rater effects are shown with a confidence interval. A cross-project view shows one consultant as several project leads rated them. We recommend treating every correction as a proposal that a person decides on in the calibration meeting.

Frequently asked questions

What are the types of rater bias?

Common types are leniency, severity, central tendency, the halo effect, recency bias and similar-to-me bias. In the 360° studies cited here, the largest was one without a catchy name: the idiosyncratic rater effect, meaning each rater’s personal way of using the scale.

What is leniency bias, and how is it different from the halo effect?

Leniency bias means a rater rates everyone too generously, which shifts the level of all their ratings. The halo effect works within one person: one strong impression colours their ratings on other criteria, which is not always an error.

What is central tendency error?

A rater puts almost everyone in the middle of the scale, so real differences disappear. It increases when information is costly and relationships are close, and in one study it held back later performance improvement (Bol, 2011). A standard that spells out the rating steps around the middle can help.

What is the idiosyncratic rater effect?

The part of a rating that reflects how a particular rater uses the scale rather than how the person performed. In two large samples of developmental 360° ratings it explained 62% and 53% of the variance, against 21% and 25% for actual performance (Scullen, Mount and Goff, 2000). The shares do not transfer directly to pay-linked appraisals.

How do you reduce bias in performance reviews?

Agree a shared standard with behavioural anchors and train raters on it. Have several project leads rate independently and soon after each project. Before the calibration meeting, check statistically which raters are unusually lenient or severe. A person decides on every correction. Bias can be reduced, not removed.

What is frame-of-reference training?

It teaches raters a shared picture of performance: they rate example cases, compare their ratings with the intended standard and discuss the gaps. A meta-analysis found it improves rating accuracy (Roch et al., 2012). Lasting effects in the field, and lower leniency when pay is attached, have not been shown in the studies cited here.

Can rater bias be corrected statistically?

Partly. With overlapping raters, each rater’s leniency or severity can be estimated and removed, and central tendency detected. No statistic finds a bias every rater shares, and some rater differences are genuine. Keep corrections as proposals; Article 22 GDPR gives people the right not to be subject to significant decisions based solely on automated processing. This is general information, not legal advice.

Sources

Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85(6), 956–970., American Psychological Association (2000). Peer-reviewed study of two large samples of 360° ratings (2,350 and 2,142 managers, seven ratings each). Cited for the shares of variance given in the abstract: idiosyncratic rater effects 62% and 53%, actual performance 21% and 25%. The data are developmental ratings; they do not establish shares for pay-linked appraisals or for consulting firms.
Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51(3), 557–576., Wiley (1998). Cited for the abstract’s finding that method variance attaches more to the individual rater than to the rater’s level (boss, peer, subordinate). No shares taken outside this 360° sample.
Hoffman, B., Lance, C. E., Bynum, B., & Gentry, W. A. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63(1), 119–151., Wiley (2010). Cited as a caveat: modelled properly, rater source factors explain far more variance than previously believed, so not every difference between raters is error. Not cited for the claim that rater differences are harmless.
Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81(5), 557–574., American Psychological Association (1996). Meta-analysis; cited for the mean inter-rater reliability of 0.52 for supervisors’ overall performance ratings and the lower reliability of peer ratings. Not a value for any one firm, and its interpretation is debated. The Spearman–Brown calculation in the example is our illustration, not a result of the study.
Murphy, K. R. (2008). Explaining the weak relationship between job performance and ratings of job performance. Industrial and Organizational Psychology, 1(2), 148–160., Cambridge University Press (2008). Focal article; cited for the weak link between performance and ratings and for the point that raters need tools and incentives to rate accurately. No effect sizes for any remedy taken.
Jawahar, I. M., & Williams, C. R. (1997). Where all the children are above average: The performance appraisal purpose effect. Personnel Psychology, 50(4), 905–925., Wiley (1997). Meta-analysis of 22 studies (N = 57,775): ratings for administrative purposes such as pay and promotion were on average about one-third of a standard deviation more lenient than ratings for research or development. No fixed inflation percentage for any one firm.
Kingstrom, P. O., & Bass, A. R. (1981). A critical analysis of studies comparing behaviorally anchored rating scales (BARS) and other rating formats. Personnel Psychology, 34(2), 263–289., Wiley (1981). Cited for the finding that BARS differed little, if at all, from other formats on leniency, halo, inter-rater agreement and accuracy. Not cited for “anchors are useless”; their value as a shared standard is argued in the article from design logic.
Roch, S. G., Woehr, D. J., Mishra, V., & Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85(2), 370–395., Wiley (British Psychological Society) (2012). Meta-analysis; cited for the finding that frame-of-reference training improves rating accuracy. No effect sizes taken; not evidence of lower leniency when pay is attached.
Gorman, C. A., & Rentsch, J. R. (2009). Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of Applied Psychology, 94(5), 1336–1344., American Psychological Association (2009). Cited for the finding that after frame-of-reference training raters’ mental models of performance were closer to the expert model, and that this closeness went with more accurate ratings. Not evidence of lasting effects in the field.
Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78(2), 218–225., American Psychological Association (1993). Cited for the argument that halo is not simply an error and for the authors’ call for a moratorium on halo indices. Not cited for “halo does not exist”.
Raymond, M. R., & Viswesvaran, C. (1993). Least squares models to correct for rater effects in performance assessment. Journal of Educational Measurement, 30(3), 253–268., Wiley (National Council on Measurement in Education) (1993). Cited for correcting leniency and severity when raters overlap, and for the medical oral exam example (40 examiners, about 115 candidates, four examiners each) in which adjustment would have changed pass or fail for about 7–9% of candidates in a reanalysis. Applying this to business appraisals is our inference.
Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422., JAM Press (2003). Methods paper without a DOI (abstract via PubMed); cited for detecting, per rater, leniency or severity, central tendency, randomness and halo. Part II: 2004 (Journal of Applied Measurement, 5(2), 189–227). Not evidence that a typical firm’s data have enough overlap for it.
Breuer, K., Nieken, P., & Sliwka, D. (2013). Social ties and subjective performance evaluations: An empirical investigation. Review of Managerial Science, 7(2), 141–157., Springer (2013). Call-centre field data from one company; cited for the finding that ratings were biased upwards in smaller teams, with team size used by the authors as a proxy for closer social ties. Not evidence of demographic similar-to-me bias.
Bol, J. C. (2011). The determinants and performance effects of managers’ performance evaluation biases. The Accounting Review, 86(5), 1549–1575., American Accounting Association (2011). Cited for the findings that information-gathering costs and close relationships increase centrality and leniency, that centrality hurt later performance improvement, and that leniency was positively associated with future performance in this one firm’s data (a correlation, not evidence of cause). Not cited for “leniency is always harmful”.
Ockenfels, A., Sliwka, D., & Werner, P. (2025). Multirater performance evaluations and incentives. Journal of Labor Economics, 43(4), 985–1004., University of Chicago Press (2025). Formal model and laboratory experiment; no field data. Cited for the finding that group evaluations were more accurate than single-rater evaluations in the lab, which the authors attribute to the group pooling its information. Not cited for “more raters remove bias automatically”, nor for the performance effect under pay-linked evaluation, which for three-rater evaluations was only marginally significant, in the final rounds of the lab experiment.
Reb, J., & Greguras, G. J. (2010). Understanding performance ratings: Dynamic performance, attributions, and rating purpose. Journal of Applied Psychology, 95(1), 213–220., American Psychological Association (2010). Cited for the finding that the trend and variability of performance, not only its average, shape ratings. Not direct evidence of a recency effect at work.
Demeré, B. W., Sedatole, K. L., & Woods, A. (2019). The role of calibration committees in subjective performance evaluation systems. Management Science, 65(4), 1562–1585., INFORMS (2019). Proprietary data from one large multinational. Cited for the abstract’s finding that calibration committees mitigated leniency but exacerbated centrality. Sample figures quoted in press releases are not used.
§ 94 BetrVG – Personalfragebogen, Beurteilungsgrundsätze (Works Constitution Act, personnel questionnaires and appraisal principles), gesetze-im-internet.de (German Federal Ministry of Justice) (2026). Official German text as of the research date, 14 October 2026. Cited for para. 2 (works council consent to general appraisal principles); the English rendering is ours. Not legal advice.
Bundesarbeitsgericht (Federal Labour Court), decision of 17 March 2015 – 1 ABR 48/13, Bundesarbeitsgericht (2015). Cited for the headnote: co-determination under section 94(2) BetrVG extends to the design of the appraisal procedure, including appraisal conversations. That statistical correction rules fall under it is our reading, not a holding; not legal advice.
§ 87 BetrVG – Mitbestimmungsrechte (Works Constitution Act, co-determination rights), gesetze-im-internet.de (German Federal Ministry of Justice) (2026). Official German text as of the research date, 14 October 2026. Cited for para. 1 no. 6 (technical devices for monitoring behaviour or performance); the English rendering is ours. Not legal advice.
Bundesarbeitsgericht (Federal Labour Court), decision of 13 December 2016 – 1 ABR 7/15, Bundesarbeitsgericht (2016). Cited for para. 22: technical devices are “intended” for monitoring if they are objectively suitable for it, regardless of the employer’s intention. The case concerned an employer’s Facebook page; applying it to appraisal software is our reading.
Regulation (EU) 2016/679 (General Data Protection Regulation) – Article 4(4), Article 22 and Recital 71, EUR-Lex, Publications Office of the European Union (2016). Official text. Cited for the definition of profiling (performance at work), the right not to be subject to a decision based solely on automated processing with significant effects, and Recital 71’s “appropriate mathematical or statistical procedures”. Not legal advice; UK GDPR is not covered.