Eval rubric builder

What it does

Write a rubric as a list of criteria. Give each one a description of what it measures and two or more levels: a score, a label, what earns that score and, if it helps, an example. The tool exports it four ways:

• A JSON Schema for the judge's answer: one object per criterion, with the reasoning before the score, and the score restricted to that criterion's levels. Every property is required and none other is allowed — the form OpenAI's strict structured outputs require — and it uses only type, properties, required, additionalProperties, enum and description. A mode that accepts only a subset of JSON Schema may still refuse additionalProperties or an integer enum; check it against the mode you use.
• A judge prompt: a fixed template filled in with the rubric, with {{input}} and {{response}} where each item to grade goes. It is not written by a model; the same rubric always gives the same text.
• A labelling sheet, CSV: one row per item, with a score and a notes column per criterion, for people to fill in.
• The rubric as CSV, one row per level, to keep beside the sheet.

Try it: a RAG answer-quality rubric, a customer-support reply rubric, or a single pass/fail check.

What makes a rubric usable

The tool flags what would make grading ambiguous: a criterion without a name, two criteria with the same name, fewer than two levels, two levels with the same score, or a score that is not a whole number from −100 to 100. It warns when a level or a criterion has no description, since a judge then has only a number and a word to go on.

Each criterion's key in the JSON output is its name in snake_case, with _2, _3… added when another criterion already has it; the schema, the prompt and the sheet all use the same keys, so a judge's answers and people's labels line up column for column.

Checking the judge

A judge's scores are worth what their agreement with people is worth. Grade the same items both ways — the labelling sheet is for that — and measure it with the judge agreement calculator.

What it does not do

It does not run a judge: it builds what one needs. Salyro does not run evaluations, and nothing you write here is sent to it or anywhere else. CSV cells that a spreadsheet would run as a formula are prefixed so they open as text.