> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Measure and compare retriever quality with ground truth datasets and standard IR metrics

Evaluations run your retriever against a curated set of queries with known-relevant documents, then compute standard information retrieval metrics at multiple cutoff points. Use them to quantify retriever quality, compare configurations, and catch regressions before they reach production.

## Quickstart

**1. Create a ground truth dataset**

Each query pairs an input with the document IDs that should be returned.

```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/evaluations/datasets" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_name": "product-search-golden",
    "queries": [
      {
        "query_id": "q1",
        "query_input": {"query": "wireless earbuds"},
        "relevant_documents": ["doc_a1", "doc_a2", "doc_a3"],
        "relevance_scores": {"doc_a1": 5, "doc_a2": 3, "doc_a3": 2}
      },
      {
        "query_id": "q2",
        "query_input": {"query": "noise canceling headphones"},
        "relevant_documents": ["doc_b1", "doc_b4", "doc_b7"]
      }
    ]
  }'
```

```python Python theme={null}
dataset = client.retrievers.evaluations.create_dataset(
    dataset_name="product-search-golden",
    queries=[
        {
            "query_id": "q1",
            "query_input": {"query": "wireless earbuds"},
            "relevant_documents": ["doc_a1", "doc_a2", "doc_a3"],
            "relevance_scores": {"doc_a1": 5, "doc_a2": 3, "doc_a3": 2},
        },
        {
            "query_id": "q2",
            "query_input": {"query": "noise canceling headphones"},
            "relevant_documents": ["doc_b1", "doc_b4", "doc_b7"],
        },
    ],
)
```

```javascript JavaScript theme={null}
const dataset = await client.retrievers.evaluations.createDataset({
  datasetName: "product-search-golden",
  queries: [
    {
      queryId: "q1",
      queryInput: { query: "wireless earbuds" },
      relevantDocuments: ["doc_a1", "doc_a2", "doc_a3"],
      relevanceScores: { doc_a1: 5, doc_a2: 3, doc_a3: 2 },
    },
    {
      queryId: "q2",
      queryInput: { query: "noise canceling headphones" },
      relevantDocuments: ["doc_b1", "doc_b4", "doc_b7"],
    },
  ],
});
```

**2. Run the evaluation**

Execute your retriever against every query in the dataset. The evaluation runs asynchronously and returns a `task_id` for progress tracking.

```bash cURL theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/evaluations" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_name": "product-search-golden",
    "evaluation_config": {
      "k_values": [1, 5, 10, 20],
      "metrics": ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"]
    }
  }'
```

```python Python theme={null}
evaluation = client.retrievers.evaluations.run(
    retriever_id="ret_abc123",
    dataset_name="product-search-golden",
    evaluation_config={
        "k_values": [1, 5, 10, 20],
        "metrics": ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"],
    },
)
# evaluation["task_id"], evaluation["evaluation_id"]
```

```javascript JavaScript theme={null}
const evaluation = await client.retrievers.evaluations.run({
  retrieverId: "ret_abc123",
  datasetName: "product-search-golden",
  evaluationConfig: {
    kValues: [1, 5, 10, 20],
    metrics: ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"],
  },
});
// evaluation.taskId, evaluation.evaluationId
```

**3. Get results**

Poll the evaluation until `status` is `completed`.

```bash theme={null}
curl "$MP_API_URL/v1/retrievers/{retriever_id}/evaluations/{evaluation_id}" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE"
```

```json Example response theme={null}
{
  "evaluation_id": "eval_abc123",
  "status": "completed",
  "query_count": 50,
  "overall_metrics": {
    "precision_at_5": 0.85,
    "recall_at_5": 0.72,
    "f1_at_5": 0.78,
    "f2_at_5": 0.74,
    "ndcg_at_5": 0.81,
    "map": 0.71,
    "mrr": 0.93
  },
  "metrics_by_k": {
    "1":  {"precision": 0.90, "recall": 0.18, "f1": 0.30, "f2": 0.21, "ndcg": 0.90},
    "5":  {"precision": 0.85, "recall": 0.72, "f1": 0.78, "f2": 0.74, "ndcg": 0.81},
    "10": {"precision": 0.75, "recall": 0.88, "f1": 0.81, "f2": 0.85, "ndcg": 0.89},
    "20": {"precision": 0.62, "recall": 0.95, "f1": 0.75, "f2": 0.86, "ndcg": 0.91}
  }
}
```

## Metrics

Every metric is computed at each K value you specify. The defaults cover most use cases:

| Metric | What It Measures | Formula |
| - | - | - |
| **Precision\@K** | Accuracy of top results | relevant in top K ÷ K |
| **Recall\@K** | Coverage of relevant documents | relevant in top K ÷ total relevant |
| **F1\@K** | Balanced precision/recall | harmonic mean of P and R |
| **F2\@K** | Recall-weighted balance | weighted harmonic mean (β=2), penalizes missed docs 4× more than false positives |
| **MAP** | Ranking quality across all queries | average of precision at each relevant doc's position |
| **MRR** | How quickly users find a relevant result | 1 ÷ rank of first relevant document |
| **NDCG\@K** | Ranking quality with graded relevance | normalized discounted cumulative gain |

> [!NOTE]
>
> **F2 vs F1:** Use F2 when missing a relevant document is worse than showing an irrelevant one — the common case in search, recommendations, and discovery. F1 treats both errors equally.

### Reading Your Scores

| Score | What It Tells You |
| - | - |
| **NDCG\@10 = 0.89** | Your top-10 ranking captures 89% of the ideal ordering. Relevant docs appear near the top. |
| **Precision\@5 = 0.85** | 4–5 of every 5 results are relevant. Users see high-quality results. |
| **Recall\@20 = 0.95** | You surface 95% of all relevant documents within the top 20. Strong coverage. |
| **F2\@10 = 0.85** | Recall-weighted balance is strong — few relevant documents are being missed. |
| **MRR = 0.93** | The first relevant result typically appears at position 1 or 2. |
| **MAP = 0.71** | Overall ranking quality is solid but there's room to improve ordering. |

### Graded Relevance

When you provide `relevance_scores` in your dataset, NDCG uses graded relevance instead of binary. This distinguishes "exactly right" from "somewhat relevant":

```json theme={null}
{
  "query_id": "q1",
  "query_input": {"query": "wireless earbuds"},
  "relevant_documents": ["doc_a1", "doc_a2", "doc_a3"],
  "relevance_scores": {
    "doc_a1": 5,
    "doc_a2": 3,
    "doc_a3": 1
  }
}
```

| Score | Meaning |
| - | - |
| 5 | Perfect match |
| 3–4 | Highly relevant |
| 1–2 | Marginally relevant |
| 0 | Not relevant |

Without `relevance_scores`, all metrics use binary relevance (relevant or not).

## Comparing Retrievers

Run the same dataset against different retriever configurations to find the best pipeline:

```bash theme={null}
# Evaluate baseline (vector search only)
curl -X POST "$MP_API_URL/v1/retrievers/ret_baseline/evaluations" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -d '{"dataset_name": "product-search-golden"}'

# Evaluate candidate (vector search + reranker)
curl -X POST "$MP_API_URL/v1/retrievers/ret_reranked/evaluations" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -d '{"dataset_name": "product-search-golden"}'
```

Compare the `metrics_by_k` side by side:

| Metric | Baseline | + Reranker | Delta |
| - | - | - | - |
| NDCG\@10 | 0.78 | 0.89 | +14% |
| Precision\@5 | 0.72 | 0.85 | +18% |
| F2\@10 | 0.76 | 0.85 | +12% |
| MRR | 0.81 | 0.93 | +15% |

> [!TIP]
>
> Run the same dataset after every pipeline change — adding stages, swapping models, adjusting fusion weights — to quantify the impact before deploying.

## Ground Truth Datasets

### Dataset Requirements

* At least 1 query (aim for 50+ for statistically meaningful results)
* Each query must have at least 1 relevant document
* `query_input` must use the keys your retriever's `input_schema` declares
* `relevance_scores`, if provided, must cover all `relevant_documents`

A dataset needs `dataset_name` and `queries`, and each query needs `query_id`,
`query_input` and `relevant_documents`. `relevance_scores` is optional and takes
0 to 5 per document, which is what NDCG reads.

A run keys on the dataset **name**, not its id, and returns a `task_id`
immediately while the evaluation proceeds asynchronously.

### Grouped retrievers: `relevance_field`

A retriever with a `group_by` or reduce stage returns one document per group, picked from several that share a key. No list of document ids can say "this frame is relevant". Listing every tile of a frame lowers recall, whichever tile the retriever returns.

Set `relevance_field` on the dataset to grade by that field's value:

```bash theme={null}
curl -X POST "$MP_API_URL/v1/retrievers/evaluations/datasets" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_name": "frames-golden",
    "relevance_field": "metadata.frame_key",
    "queries": [
      {
        "query_id": "q1",
        "query_input": {"query": "timber on a truck"},
        "relevant_documents": ["frame_0012", "frame_0340"]
      }
    ]
  }'
```

* `relevant_documents` lists values of the field in place of document ids, and `relevance_scores` keys on the same values.
* Results that share a value count once, at the rank of the first.
* Dotted paths reach nested fields, as in `metadata.frame_key`.
* A query skips when none of its results has the field. Its `error` in `query_results` names the field.

### When every query skips

Input validation is strict. A key in `query_input` that the retriever does not
declare is rejected rather than ignored, and the message names what it does
accept:

```
Unknown input field(s) 'top_k'. This retriever accepts: ['query', ...]
```

That query skips. If every key is wrong, every query skips.

> [!NOTE]
>
> **An all-skipped run finishes as `failed`, not as zeros.** `EvaluationStatus`
> is `pending`, `in_progress`, `completed` or `failed`, and a run that evaluated
> nothing takes the failed branch rather than reporting 0.0 across the board.
>
> Read `query_results` on the evaluation record to see which branch each query
> took and the actual error for each skip. `total_queries`, `evaluated_queries`
> and `skipped_queries` give you the same story as counts. Diagnosing a skipped
> run needs neither worker logs nor a re-run.

> [!WARNING]
>
> The runner injects `top_k` only when the retriever's `input_schema` declares it, and a
> value in your own dataset wins over the injected one.
>
> If you are holding an older all-skip result, re-run it before treating it as a dataset
> problem. Earlier runners injected `top_k` unconditionally, which failed validation
> against any retriever that does not declare a `top_k` input.

### Managing Datasets

```bash theme={null}
# List all datasets
curl "$MP_API_URL/v1/retrievers/evaluations/datasets" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE"

# Get a specific dataset
curl "$MP_API_URL/v1/retrievers/evaluations/datasets/product-search-golden" \
  -H "Authorization: Bearer $MP_API_KEY" \
  -H "X-Namespace: $MP_NAMESPACE"
```

### Building Good Datasets

> [card] Include query diversity
>
> Cover head queries (popular), torso (moderate), and tail (rare/specific). Don't just test easy cases.

> [card] Use graded relevance
>
> Binary relevant/not-relevant misses nuance. Score documents 0–5 so NDCG can distinguish good rankings from great ones.

> [card] Match real traffic
>
> Sample queries from production logs. Synthetic queries test what you think users ask, not what they actually ask.

> [card] Version your datasets
>
> Keep datasets stable across evaluations so you can track metric trends over time. Create new versions for schema changes.

## Configuration Reference

**dataset\_name** `string` *required*

Name of the ground truth dataset to evaluate against.

**relevance\_field** `string`

Set on the dataset at creation. Grades by this field of each result in place of its document id, for a retriever that groups documents. See [Grouped retrievers](#grouped-retrievers-relevance_field).

**evaluation\_config.k\_values** `integer[]` (default: \[1, 5, 10, 20])

Cutoff positions for @K metrics. Include the K values that match your UI — if you show 10 results per page, include `10`.

**evaluation\_config.metrics** `string[]` (default: \["precision", "recall", "f1", "f2", "map", "ndcg", "mrr"])

Metrics to compute. Available: `precision`, `recall`, `f1`, `f2`, `map`, `ndcg`, `mrr`.

**evaluation\_config.k\_multiples\_of\_relevant** `integer[]`

Also grade each query at a multiple of R, the number of entries in its `relevant_documents`. With `[1, 2]`, a query with 21 relevant documents is graded at k=21 and k=42, and a query with 4 at k=4 and k=8. Scores appear as `f2_at_2r` in `overall_metrics` and in each query's `metrics`, and under `"2r"` in `metrics_by_k`. `k_values` still apply. A query whose k exceeds the retriever pagination limit is skipped, and `query_results` gives the reason.

## Per-query results

`GET /v1/retrievers/{retriever_id}/evaluations/{evaluation_id}` returns one entry in `query_results` for each attempted query. An evaluated query carries its own scores under the same keys as `overall_metrics`, so one run can score each class of a dataset separately:

```json theme={null}
{
  "query_id": "dry_brush_grass",
  "status": "evaluated",
  "retrieved_count": 42,
  "relevant_count": 21,
  "metrics": {"f2_at_2r": 0.71, "recall_at_10": 0.38}
}
```

If the dataset sets `relevance_field`, these scores grade the values of that field, and `relevant_count` counts the values listed. The list route omits `metrics` to keep pages small, so read a single evaluation for them.

## Related

* [Retriever Benchmarks](https://docs.mixpeek.com/docs/api-reference/retriever-benchmarks/create-benchmark.md) — replay live sessions against candidate retrievers
* [Improve Relevance](https://docs.mixpeek.com/docs/platform/improve-relevance.md) — interaction signals, fusion strategies, and the feedback loop
* [API Reference: Run Evaluation](https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/run-evaluation.md) — full endpoint specification
* [API Reference: Create Dataset](https://docs.mixpeek.com/docs/api-reference/retriever-evaluations/create-evaluation-dataset.md) — dataset creation endpoint


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.