Skip to main content
Evaluations run your retriever against a curated set of queries with known-relevant documents, then compute standard information retrieval metrics at multiple cutoff points. Use them to quantify retriever quality, compare configurations, and catch regressions before they reach production.

​
Quickstart

1

Create a ground truth dataset

Each query pairs an input with the document IDs that should be returned.
2

Run the evaluation

Execute your retriever against every query in the dataset. The evaluation runs asynchronously and returns a task_id for progress tracking.
3

Get results

Poll the evaluation until status is completed.
Example response

​
Metrics

Every metric is computed at each K value you specify. The defaults cover most use cases:
F2 vs F1: Use F2 when missing a relevant document is worse than showing an irrelevant one — the common case in search, recommendations, and discovery. F1 treats both errors equally.

​
Reading Your Scores

​
Graded Relevance

When you provide relevance_scores in your dataset, NDCG uses graded relevance instead of binary. This distinguishes “exactly right” from “somewhat relevant”:
Without relevance_scores, all metrics use binary relevance (relevant or not).

​
Comparing Retrievers

Run the same dataset against different retriever configurations to find the best pipeline:
Compare the metrics_by_k side by side:
Run the same dataset after every pipeline change — adding stages, swapping models, adjusting fusion weights — to quantify the impact before deploying.

​
Ground Truth Datasets

​
Dataset Requirements

  • At least 1 query (aim for 50+ for statistically meaningful results)
  • Each query must have at least 1 relevant document
  • query_input must use the keys your retriever’s input_schema declares
  • relevance_scores, if provided, must cover all relevant_documents
A dataset needs dataset_name and queries, and each query needs query_id, query_input and relevant_documents. relevance_scores is optional and takes 0 to 5 per document, which is what NDCG reads. A run keys on the dataset name, not its id, and returns a task_id immediately while the evaluation proceeds asynchronously.

​
Grouped retrievers: relevance_field

A retriever with a group_by or reduce stage returns one document per group, picked from several that share a key. No list of document ids can say “this frame is relevant”. Listing every tile of a frame lowers recall, whichever tile the retriever returns. Set relevance_field on the dataset to grade by that field’s value:
  • relevant_documents lists values of the field in place of document ids, and relevance_scores keys on the same values.
  • Results that share a value count once, at the rank of the first.
  • Dotted paths reach nested fields, as in metadata.frame_key.
  • A query skips when none of its results has the field. Its error in query_results names the field.

​
When every query skips

Input validation is strict. A key in query_input that the retriever does not declare is rejected rather than ignored, and the message names what it does accept:
That query skips. If every key is wrong, every query skips.
An all-skipped run finishes as failed, not as zeros. EvaluationStatus is pending, in_progress, completed or failed, and a run that evaluated nothing takes the failed branch rather than reporting 0.0 across the board.Read query_results on the evaluation record to see which branch each query took and the actual error for each skip. total_queries, evaluated_queries and skipped_queries give you the same story as counts. Diagnosing a skipped run needs neither worker logs nor a re-run.
The runner injects top_k only when the retriever’s input_schema declares it, and a value in your own dataset wins over the injected one.If you are holding an older all-skip result, re-run it before treating it as a dataset problem. Earlier runners injected top_k unconditionally, which failed validation against any retriever that does not declare a top_k input.

​
Managing Datasets

​
Building Good Datasets

Include query diversity

Cover head queries (popular), torso (moderate), and tail (rare/specific). Don’t just test easy cases.

Use graded relevance

Binary relevant/not-relevant misses nuance. Score documents 0–5 so NDCG can distinguish good rankings from great ones.

Match real traffic

Sample queries from production logs. Synthetic queries test what you think users ask, not what they actually ask.

Version your datasets

Keep datasets stable across evaluations so you can track metric trends over time. Create new versions for schema changes.

​
Configuration Reference

​
string
required
Name of the ground truth dataset to evaluate against.
​
string
Set on the dataset at creation. Grades by this field of each result in place of its document id, for a retriever that groups documents. See Grouped retrievers.
​
integer[]
default:"[1, 5, 10, 20]"
Cutoff positions for @K metrics. Include the K values that match your UI — if you show 10 results per page, include 10.
​
string[]
Metrics to compute. Available: precision, recall, f1, f2, map, ndcg, mrr.
​
integer[]
Also grade each query at a multiple of R, the number of entries in its relevant_documents. With [1, 2], a query with 21 relevant documents is graded at k=21 and k=42, and a query with 4 at k=4 and k=8. Scores appear as f2_at_2r in overall_metrics and in each query’s metrics, and under "2r" in metrics_by_k. k_values still apply. A query whose k exceeds the retriever pagination limit is skipped, and query_results gives the reason.

​
Per-query results

GET /v1/retrievers/{retriever_id}/evaluations/{evaluation_id} returns one entry in query_results for each attempted query. An evaluated query carries its own scores under the same keys as overall_metrics, so one run can score each class of a dataset separately:
If the dataset sets relevance_field, these scores grade the values of that field, and relevant_count counts the values listed. The list route omits metrics to keep pages small, so read a single evaluation for them.