> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG Prepare

> Prepare documents for LLM context windows with token management and source citations — evidence/provenance per result so every claim traces back to a source clip

![RAG Prepare stage showing document formatting for LLM context](https://mintcdn.com/mixpeek/TwtTrae3Fi3EFJ72/assets/retrievers/rag-prepare.svg?fit=max\&auto=format\&n=TwtTrae3Fi3EFJ72\&q=85\&s=d27d7a01f367d2643a5d9bf2737a654a)

The RAG Prepare stage formats search results for LLM consumption by managing token budgets, formatting documents, and adding citations. This is a **preparation stage that does NOT call an LLM** - it prepares content for downstream LLM stages or external LLM calls.

> [!NOTE]
>
> **Stage Category**: APPLY
>
> **Transformation**:
>
> * `single_context` mode: N documents → 1 combined context document
> * `formatted_list` mode: N documents → N formatted documents

## When to Use

| Use Case | Description |
| - | - |
| **Before LLM generation** | Prepare context for summarization or Q\&A |
| **Token budget management** | Fit multiple docs into context window |
| **Citation tracking** | Enable source attribution in responses |
| **Consistent formatting** | Standardize document format for LLM input |

## When NOT to Use

| Scenario | Recommended Alternative |
| - | - |
| Want LLM to generate summary | `summarize` stage (calls LLM) |
| Don't need token management | Pass documents directly |
| Simple pass-through | Skip this stage |

## Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `max_tokens` | integer | `8000` | Maximum tokens for combined output |
| `tokenizer` | string | `cl100k_base` | Tokenizer to use (GPT-4 compatible) |
| `truncation_strategy` | string | `priority_truncate` | How to handle token overflow |
| `output_mode` | string | `single_context` | Output format |
| `document_template` | string | `[{{CONTEXT.INDEX}}] {{DOC.content}}\n\n` | Template for each document |
| `content_field` | string | `content` | Field to extract content from |
| `separator` | string | `\n` | Separator between documents |
| `citation` | object | `{style: "numbered"}` | Citation configuration |

### Truncation Strategies

| Strategy | Behavior |
| - | - |
| `priority_truncate` | Include docs in score order, truncate last to fit |
| `proportional` | Give each doc proportional token budget |
| `drop_last` | Include complete docs until limit, drop remaining |

### Output Modes

| Mode | Output | Use Case |
| - | - | - |
| `single_context` | 1 document with combined `context` string | Direct LLM input |
| `formatted_list` | N documents with `formatted_content` field | Custom processing |

## Configuration Examples

```json Basic RAG Context theme={null}
{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 8000,
      "output_mode": "single_context"
    }
  }
}
```

```json With Numbered Citations theme={null}
{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 4000,
      "document_template": "[{{CONTEXT.INDEX}}] {{DOC.metadata.title}}\n{{DOC.content}}\n\n",
      "truncation_strategy": "priority_truncate",
      "citation": {
        "style": "numbered",
        "include_title": true
      }
    }
  }
}
```

```json Large Context Window (GPT-4 Turbo) theme={null}
{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 32000,
      "tokenizer": "cl100k_base",
      "truncation_strategy": "proportional"
    }
  }
}
```

```json Formatted List Mode theme={null}
{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "output_mode": "formatted_list",
      "document_template": "Source: {{DOC.metadata.source}}\n{{DOC.content}}"
    }
  }
}
```

```json Custom Template with URL theme={null}
{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 8000,
      "document_template": "Document {{CONTEXT.INDEX}}:\nTitle: {{DOC.metadata.title}}\nURL: {{DOC.metadata.url}}\n\n{{DOC.content}}\n\n---\n",
      "citation": {
        "style": "bracketed",
        "include_url": true
      }
    }
  }
}
```

## Template Placeholders

| Placeholder | Description |
| - | - |
| `{{CONTEXT.INDEX}}` | 1-based position in result set (1, 2, 3...) |
| `{{CONTEXT.CITATION}}` | Citation marker based on citation.style |
| `{{DOC.*}}` | Any document field (e.g., `{{DOC.content}}`, `{{DOC.metadata.title}}`) |

## Citation Styles

| Style | Output | Example |
| - | - | - |
| `numbered` | `[1]`, `[2]`, `[3]` | Default, clean |
| `bracketed` | `[doc_id]` | Document ID references |
| `footnote` | Superscript numbers | Academic style |
| `none` | No citations | When not needed |

## Output Schema

### single\_context Mode

```json theme={null}
{
  "rag_context": "[1] First document content...\n\n[2] Second document content...",
  "citations": [
    {"index": 1, "title": "Document Title", "document_id": "doc_123"},
    {"index": 2, "title": "Another Title", "document_id": "doc_456"}
  ]
}
```

### formatted\_list Mode

Each document gets:

```json theme={null}
{
  "document_id": "doc_123",
  "formatted_content": "[1] Title\nContent here...",
  "original_content": "Content here...",
  "metadata": {...}
}
```

## Citing Timestamped Source Clips (Video / Audio)

The default `citation` object is `{index, title, document_id}` — enough to name the
source, but it does **not** inline the timestamp or a link back to the source clip.
For investigative / RAG use cases where every claim must resolve to a *timestamped,
clickable source* (e.g. "watch this at 03:12"), surface the anchor yourself — the
data is already on every document, so this needs **no new stage or endpoint**:

* Video/audio segment documents carry `start_time_s` / `end_time_s`
  (see [Universal Extractor → Output](https://docs.mixpeek.com/docs/processing/extractors/universal.md)).
* Every document carries `_internal.lineage.root_object_id` pointing back to the
  original source object (see [Lineage Traversal](https://docs.mixpeek.com/docs/retrieval/lineage-traversal.md)).

Reference those fields directly in `document_template` so the timestamp travels into
the LLM context (and into any answer the LLM cites):

```json Timestamped clip citations theme={null}
{
  "stage_name": "rag_prepare",
  "stage_type": "apply",
  "config": {
    "stage_id": "rag_prepare",
    "parameters": {
      "max_tokens": 8000,
      "output_mode": "single_context",
      "document_template": "[{{CONTEXT.INDEX}}] {{DOC.metadata.title}} @ {{DOC.start_time_s}}s–{{DOC.end_time_s}}s (source: {{DOC._internal.lineage.root_object_id}})\n{{DOC.text}}\n\n",
      "citation": { "style": "numbered", "include_title": true }
    }
  }
}
```

To render a **clickable** clip link, hydrate the `document_id` returned in the
`citations[]` array (or the `root_object_id` above) into a playable URL with
`GET /v1/documents/{id}?expand=root_object` — see
[Lineage Traversal](https://docs.mixpeek.com/docs/retrieval/lineage-traversal.md). Do this in the app layer after
`execute`; the timestamp and source object are preserved end-to-end in the document,
they are simply not inlined into the structured `citations[]` array by default.

> [!NOTE]
>
> Citations attach at the **document** level. `rag_prepare` (and `summarize`'s
> `source_document_ids`) cite the set of source documents behind the context, not
> individual sentences — per-claim/per-sentence attribution is up to the downstream
> LLM prompt. Ask the LLM to emit the `[n]` marker inline with each claim so the
> numbered citation resolves back to the timestamped clip above.

## Performance

| Metric | Value |
| - | - |
| **Latency** | \< 10ms |
| **Token counting** | Uses tiktoken (accurate) |
| **No LLM calls** | Pure formatting |

> [!TIP]
>
> This stage does NOT call an LLM. It only formats content for LLM consumption. Use the `summarize` stage if you want LLM-generated summaries.

## Common Pipeline Patterns

### Search + Prepare + External LLM

```json theme={null}
[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 50 }
        ],
        "final_top_k": 50
      }
    }
  },
  {
    "stage_name": "rerank",
    "stage_type": "sort",
    "config": {
      "stage_id": "rerank",
      "parameters": {
        "inference_name": "BAAI__bge_reranker_v2_m3",
        "top_k": 10
      }
    }
  },
  {
    "stage_name": "rag_prepare",
    "stage_type": "apply",
    "config": {
      "stage_id": "rag_prepare",
      "parameters": {
        "max_tokens": 8000,
        "output_mode": "single_context",
        "citation": {"style": "numbered"}
      }
    }
  }
]
```

The output `rag_context` can then be passed to an external LLM call.

### vs Summarize Stage

| Feature | rag\_prepare | summarize |
| - | - | - |
| Calls LLM | No | Yes |
| Output | Formatted context | Generated summary |
| Latency | \< 10ms | 500-2000ms |
| Cost | Free | LLM API costs |
| Use case | Prepare for external LLM | End-to-end RAG |

## Related

* [Summarize](https://docs.mixpeek.com/docs/retrieval/stages/summarize.md) - LLM-powered summarization
* [JSON Transform](https://docs.mixpeek.com/docs/retrieval/stages/json-transform.md) - Custom document formatting
* [LLM Enrich](https://docs.mixpeek.com/docs/retrieval/stages/llm-enrich.md) - Extract structured data with LLM


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.