> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cluster

> Group documents by embedding similarity into semantic clusters

![Cluster stage showing semantic grouping of search results](https://mintcdn.com/mixpeek/TwtTrae3Fi3EFJ72/assets/retrievers/cluster.svg?fit=max\&auto=format\&n=TwtTrae3Fi3EFJ72\&q=85\&s=2e8b5f4a714a1beefd9733f7fbb0acbd)

The Cluster stage groups documents based on embedding similarity, creating semantic clusters of related content. This helps organize search results into meaningful groups and discover themes within your results.

> [!NOTE]
>
> **Stage Category**: GROUP (Groups documents)
>
> **Transformation**: N documents → K clusters with documents

## When to Use

| Use Case | Description |
| - | - |
| **Theme discovery** | Find topics within search results |
| **Result organization** | Group similar items together |
| **Deduplication** | Find near-duplicate content |
| **Exploration** | Understand result diversity |

## When NOT to Use

| Scenario | Recommended Alternative |
| - | - |
| Grouping by field value | `group_by` |
| Removing duplicates | `deduplicate` |
| Pre-defined categories | `taxonomy_enrich` |
| Single representative | `sample` per group |

## Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `algorithm` | string | `hdbscan` | Clustering algorithm (see below) |
| `n_clusters` | integer | `null` | Number of clusters, 2 to 100. Used by `kmeans`, `agglomerative`, `spectral` and `gaussian_mixture`, and ignored by `hdbscan` and `dbscan`. When omitted for an algorithm that needs it, the stage picks `min(8, N/10)` with a floor of 2, where N is the number of documents |
| `min_cluster_size` | integer | `5` | Minimum documents per cluster, 2 to 100. Used by `hdbscan` and `dbscan`. Lowered automatically for small result sets |
| `feature_uri` | string | *auto* | Embedding to cluster; auto-inherited from a prior `feature_search`, otherwise the first vector field found on the documents |
| `output_mode` | string | `clusters` | `clusters`, `labeled`, or `representatives` |
| `include_centroids` | boolean | `true` | Include each cluster's centroid vector in `clusters` output. The centroid is returned only when the query also requests vectors |
| `max_members_per_cluster` | integer | `50` | Maximum members listed per cluster in `clusters` output, 1 to 500. Members are ordered closest to the centroid first |

## Clustering Algorithms

| Algorithm | Description | Best For |
| - | - | - |
| `hdbscan` | Density-based clustering, chooses the cluster count and marks noise (default) | Unknown cluster count |
| `kmeans` | K-means clustering | Fixed number of clusters |
| `dbscan` | Density-based with fixed epsilon | Noisy data with clear density separation |
| `agglomerative` | Hierarchical clustering | Nested clusters |
| `spectral` | Graph-based clustering | Non-convex cluster shapes |
| `gaussian_mixture` | Probabilistic clustering | Overlapping clusters |

Documents that `hdbscan` or `dbscan` treat as noise get the cluster ID `-1`. The `clusters` and `representatives` output modes leave them out, and `labeled` keeps them with `cluster_id: -1`.

## Configuration Examples

```json Basic Clustering theme={null}
{
  "stage_name": "cluster",
  "stage_type": "reduce",
  "config": {
    "stage_id": "cluster",
    "parameters": {
      "algorithm": "kmeans",
      "n_clusters": 5
    }
  }
}
```

```json KMeans With Known K theme={null}
{
  "stage_name": "cluster",
  "stage_type": "reduce",
  "config": {
    "stage_id": "cluster",
    "parameters": {
      "algorithm": "kmeans",
      "n_clusters": 8
    }
  }
}
```

```json Density-Based Clustering theme={null}
{
  "stage_name": "cluster",
  "stage_type": "reduce",
  "config": {
    "stage_id": "cluster",
    "parameters": {
      "algorithm": "hdbscan",
      "min_cluster_size": 3
    }
  }
}
```

```json Custom Embedding Field theme={null}
{
  "stage_name": "cluster",
  "stage_type": "reduce",
  "config": {
    "stage_id": "cluster",
    "parameters": {
      "algorithm": "kmeans",
      "n_clusters": 6,
      "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1"
    }
  }
}
```

```json Fine-Grained Clustering theme={null}
{
  "stage_name": "cluster",
  "stage_type": "reduce",
  "config": {
    "stage_id": "cluster",
    "parameters": {
      "n_clusters": 15,
      "min_cluster_size": 2,
      "algorithm": "kmeans"
    }
  }
}
```

```json Representative Documents theme={null}
{
  "stage_name": "cluster",
  "stage_type": "reduce",
  "config": {
    "stage_id": "cluster",
    "parameters": {
      "algorithm": "kmeans",
      "n_clusters": 8,
      "output_mode": "representatives"
    }
  }
}
```

## How Clustering Works

1. **Extract Embeddings**: Get embedding vectors from each document
2. **Apply Algorithm**: Run the clustering algorithm (e.g., k-means)
3. **Assign Documents**: Each document gets a cluster ID
4. **Compute Centroids**: Calculate cluster centers
5. **Format Output**: Return cluster documents, labeled documents, or one representative per cluster, depending on `output_mode`

The stage needs at least 3 documents with an embedding. With fewer, it returns no results and reports the reason under `skipped` in the stage metadata.

## Output Schema

The shape depends on `output_mode`.

### `clusters` (default)

The stage returns one document per cluster. Each has a generated `document_id`, the `collection_id` of the source documents, and a `score` equal to the member count. The members and the centroid are carried in `payload`.

```json theme={null}
{
  "document_id": "doc_a1b2c3",
  "collection_id": "col_articles",
  "score": 12.0,
  "cluster_id": 0,
  "member_count": 12,
  "representative_id": "doc_123",
  "payload": {
    "members": [
      {
        "document_id": "doc_123",
        "content": "Introduction to neural networks...",
        "cluster_distance": 0.15
      }
    ],
    "centroid": null
  }
}
```

`representative_id` is the member closest to the centroid. `centroid` holds the vector only when `include_centroids` is `true` and the query requests vectors. Otherwise it is `null`. At most `max_members_per_cluster` members are listed, and `member_count` counts all of them.

### `labeled`

The stage returns the original documents with two fields added.

```json theme={null}
{
  "document_id": "doc_123",
  "content": "Introduction to neural networks...",
  "cluster_id": 0,
  "cluster_distance": 0.15
}
```

`cluster_distance` is the distance to the cluster centroid, so lower is closer. It is absent when the algorithm returns no centroid, and for noise documents (`cluster_id: -1`).

### `representatives`

The stage returns one document per cluster, the one closest to the centroid.

```json theme={null}
{
  "document_id": "doc_123",
  "content": "Introduction to neural networks...",
  "cluster_id": 0,
  "cluster_distance": 0.15,
  "is_representative": true
}
```

The stage metadata reports `algorithm`, `n_clusters`, `n_documents_in`, `n_documents_out`, `n_valid_embeddings` and `n_invalid_documents`.

## Performance

| Metric | Value |
| - | - |
| **Latency** | 50-200ms |
| **Memory** | O(N × embedding\_dim) |
| **Cost** | Free |
| **Scalability** | Built for result sets of 10 to 500 documents |

> [!WARNING]
>
> Clustering large document sets (over 1,000) can be slow. Consider pre-filtering or sampling before clustering.

## Common Pipeline Patterns

### Search + Cluster

```json theme={null}
[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
        ],
        "final_top_k": 100
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "kmeans",
        "n_clusters": 5
      }
    }
  }
]
```

### Cluster + Sample Representatives

```json theme={null}
[
  {
    "stage_name": "hybrid_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 }
        ],
        "final_top_k": 200
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "kmeans",
        "n_clusters": 10,
        "output_mode": "labeled"
      }
    }
  },
  {
    "stage_name": "sample",
    "stage_type": "reduce",
    "config": {
      "stage_id": "sample",
      "parameters": {
        "strategy": "stratified",
        "stratify_by": "cluster_id",
        "min_per_stratum": 2
      }
    }
  }
]
```

### Theme Discovery Pipeline

```json theme={null}
[
  {
    "stage_name": "structured_filter",
    "stage_type": "filter",
    "config": {
      "stage_id": "attribute_filter",
      "parameters": {
        "field": "metadata.date",
        "operator": "gte",
        "value": "2024-01-01"
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "hdbscan",
        "min_cluster_size": 5
      }
    }
  },
  {
    "stage_name": "summarize",
    "stage_type": "reduce",
    "config": {
      "stage_id": "summarize",
      "parameters": {
        "provider": "google",
        "model_name": "gemini-2.5-flash-lite",
        "prompt": "Summarize the main themes found in these clusters:\n\n{{DOCUMENTS}}"
      }
    }
  }
]
```

### Diverse Results Pipeline

```json theme={null}
[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
        ],
        "final_top_k": 100
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "kmeans",
        "n_clusters": 5,
        "output_mode": "representatives"
      }
    }
  }
]
```

## Choosing n\_clusters

| Result Size | Recommended Clusters |
| - | - |
| \< 50 docs | 3-5 clusters |
| 50-200 docs | 5-10 clusters |
| 200-500 docs | 8-15 clusters |
| 500+ docs | 10-20 clusters |

> [!TIP]
>
> Start with fewer clusters and increase if clusters are too broad. Use `hdbscan` if you don't know the optimal number.

## Error Handling

| Condition | Behavior |
| - | - |
| Document has no embedding | Left out of clustering |
| Fewer than 3 documents with embeddings | No results, with the reason under `skipped` in the metadata |
| No embedding can be identified | Request fails. Set `feature_uri` or run `feature_search` first |
| Clustering service fails | Request fails with a clustering error |

## Related

* [Group By](https://docs.mixpeek.com/docs/retrieval/stages/group-by.md) - Group by field values
* [Sample](https://docs.mixpeek.com/docs/retrieval/stages/sample.md) - Select representatives
* [MMR](https://docs.mixpeek.com/docs/retrieval/stages/mmr.md) - Diversity in ranking
* [Deduplicate](https://docs.mixpeek.com/docs/retrieval/stages/deduplicate.md) - Remove duplicates


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.