Judge agreement calculator

What it does

Paste the ratings — one row per item, one column per rater — and it measures how far the raters agree, whether they are people, LLM judges, or a judge checked against people. With two raters it gives percent agreement and Cohen's kappa, weighted kappa for ordered scores, Krippendorff's alpha, and the confusion matrix. With three or more it adds Fleiss' kappa and every pair's agreement, and shows the confusion matrix of the pair you choose.

Try it: a human and a judge, pass or fail, three raters scoring 1 to 5, with gaps, or Krippendorff's own example, which should give α = 0.849 as interval data — and 0.743, 0.815 and 0.797 as nominal, ordinal and ratio.

What it reads

CSV or TSV with a header naming the raters, and optionally an item column first (item, id, unit and the like). Or JSON: a list of objects, [{"item": "q1", "human": 3, "judge": 2}], or an object of columns, {"human": [3, 2], "judge": [2, 2]}. A cell in quotes may hold commas, line breaks and "" for a quote, as spreadsheets write them. An empty first header cell, as pandas writes its index, marks the item column too.

In CSV an empty cell, NA or null is a missing rating; in JSON, null, an empty string or a key left out. Nothing else is: none, - or ? is a label like any other, and is compared as one. A number is compared as a number, so 3 and 3.0 are the same rating.

Choose what the ratings are. Nominal ratings are labels: the order of pass and fail means nothing. Ordinal ratings are ordered scores. Interval ratings have meaningful distances between them, and ratio ratings also a true zero. Ordinal, interval and ratio need numbers, and ratio numbers of zero or more.

The measures

• Percent agreement: the share of items both raters rated the same. Easy to read, and flattered by chance: two raters who say "pass" to 90% of items at random agree on 82% of them.
• Cohen's kappa (Cohen, 1960): κ = (pₒ − pₑ) ÷ (1 − pₑ), where pₒ is the share of items the two agree on and pₑ the share they would agree on by chance, from each rater's own distribution. 1 is perfect agreement, 0 is chance, below 0 worse than chance.
• Weighted kappa (Cohen, 1968), for ordered scores: near misses earn partial credit. The categories are the scores either rater used, in order, taken as equally spaced — as R's irr package and scikit-learn take them. Linear weights 1 − |i − j| ÷ (k − 1) (Cicchetti & Allison, 1971); quadratic weights 1 − (i − j)² ÷ (k − 1)² (Fleiss & Cohen, 1973), which forgive small misses more.
• Fleiss' kappa (Fleiss, 1971), for three or more raters: κ = (P̄ − P̄ₑ) ÷ (1 − P̄ₑ), with the ratings taken as labels.
• Krippendorff's alpha (Krippendorff, 2011): α = 1 − Dₒ ÷ Dₑ, the disagreement observed over the disagreement expected by chance, with a distance for each level — any difference for nominal, ranks for ordinal, squared differences for interval, and squared relative differences for ratio. It takes any number of raters and missing ratings.

Missing ratings

Each measure uses the items it can, and the page says how many:

• percent agreement and both kappas: the items both raters of the pair rated;
• Fleiss' kappa: the items every rater rated;
• Krippendorff's alpha: every item with at least two ratings, however many raters that is.

A measure is shown as undefined, never as 0, when it has nothing to divide by — two raters who used one and the same category throughout, say.

Reading the numbers

For kappa, the page gives the bands of Landis & Koch (1977): below 0 poor, 0–.20 slight, .21–.40 fair, .41–.60 moderate, .61–.80 substantial, .81–1 almost perfect. They called their divisions arbitrary, and they are conventions, not tests. For alpha, Krippendorff (2004) advises relying on data with α ≥ .800, and drawing only tentative conclusions from α between .667 and .800.

What it does not do

It measures agreement in ratings you already have; it does not run a judge or rate anything. Salyro does not run evaluations, and nothing you paste is sent to it or anywhere else.