Retrieval metrics calculator

What it does

Give it two things — which documents are relevant to each query, and what your system returned for each, best first — and it scores the ranking at k: recall, precision, reciprocal rank and nDCG, for every query and as a mean.

Try it: Wikipedia's graded nDCG example, which should give nDCG@6 = 0.785, three queries as CSV, or TREC qrels and a TREC run.

What it reads

Relevance judgements ("qrels"):

• JSON {"q1": {"doc-a": 2, "doc-b": 0}}, a grade for each judged document, or {"q1": ["doc-a"]} when every listed document is relevant.
• Rows of query, doc, grade, or query, doc for grade 1, in CSV, TSV or spaces.
• TREC qrels: query 0 doc grade.

Ranked results:

• JSON {"q1": ["doc-b", "doc-a"]}, best first.
• Rows of query, doc in rank order, or query, doc, rank.
• A TREC run: query Q0 doc rank score tag, ordered as trec_eval orders it: by score, highest first, and equal scores by document id, descending.

Every row needs the same shape. When the third column of query, doc, rank rows holds any number that is not whole, the whole file's third column is read as a score, highest first, and the page says so.

A header row is skipped when its first cell is query, query-id, qid or topic. A document listed twice for one query counts at its first rank only.

The formulas

A document is relevant when its grade is above 0. A document the judgements do not mention counts as grade 0.

• Recall@k = relevant documents in the top k ÷ all relevant documents for the query.
• Precision@k = relevant documents in the top k ÷ k. The denominator is k even when fewer than k results came back, as trec_eval counts it.
• MRR@k = the mean, over queries, of 1 ÷ the rank of the first relevant document, or 0 when none is in the top k.
• nDCG@k = DCG@k ÷ IDCG@k, where DCG@k = Σᵢ₌₁ᵏ gain(gradeᵢ) ÷ log₂(i + 1), and IDCG@k is the DCG of the best possible order of every judged document, retrieved or not.

The gain is the grade itself by default, as trec_eval's nDCG takes it. Choose 2^grade − 1 for the form of Burges et al. (2005), which weighs a highly relevant document more; it takes grades up to 1,000. With grades of only 0 and 1 the two agree.

Which queries count

The means are over the queries in the judgements that have at least one relevant document. A query judged but missing from the results scores 0 on every metric, and the page says how many — as trec_eval -c counts; without -c, trec_eval leaves such a query out. A query in the results that nobody judged is not scored, since there is nothing to score it against.

What it does not do

It scores rankings you already have; it does not run a search or make embeddings. Nothing you paste is sent to Salyro or anywhere else.