Cluster

Group documents by embedding similarity into semantic clusters

The Cluster stage groups documents based on embedding similarity, creating semantic clusters of related content. This helps organize search results into meaningful groups and discover themes within your results.

Stage Category: GROUP (Groups documents)

Transformation: N documents → K clusters with documents

When to Use

Use Case	Description
Theme discovery	Find topics within search results
Result organization	Group similar items together
Deduplication	Find near-duplicate content
Exploration	Understand result diversity

When NOT to Use

Scenario	Recommended Alternative
Grouping by field value	group_by
Removing duplicates	deduplicate
Pre-defined categories	taxonomy_enrich
Single representative	sample per group

Parameters

Parameter	Type	Default	Description
algorithm	string	hdbscan	Clustering algorithm (see below)
n_clusters	integer	null	Number of clusters, 2 to 100. Used by kmeans, agglomerative, spectral and gaussian_mixture, and ignored by hdbscan and dbscan. When omitted for an algorithm that needs it, the stage picks min(8, N/10) with a floor of 2, where N is the number of documents
min_cluster_size	integer	5	Minimum documents per cluster, 2 to 100. Used by hdbscan and dbscan. Lowered automatically for small result sets
feature_uri	string	auto	Embedding to cluster; auto-inherited from a prior feature_search, otherwise the first vector field found on the documents
output_mode	string	clusters	clusters, labeled, or representatives
include_centroids	boolean	true	Include each cluster’s centroid vector in clusters output. The centroid is returned only when the query also requests vectors
max_members_per_cluster	integer	50	Maximum members listed per cluster in clusters output, 1 to 500. Members are ordered closest to the centroid first

Clustering Algorithms

Algorithm	Description	Best For
hdbscan	Density-based clustering, chooses the cluster count and marks noise (default)	Unknown cluster count
kmeans	K-means clustering	Fixed number of clusters
dbscan	Density-based with fixed epsilon	Noisy data with clear density separation
agglomerative	Hierarchical clustering	Nested clusters
spectral	Graph-based clustering	Non-convex cluster shapes
gaussian_mixture	Probabilistic clustering	Overlapping clusters

Documents that hdbscan or dbscan treat as noise get the cluster ID -1. The clusters and representatives output modes leave them out, and labeled keeps them with cluster_id: -1.

Configuration Examples

Basic Clustering

{
  "stage_name": "cluster",
  "stage_type": "reduce",
  "config": {
    "stage_id": "cluster",
    "parameters": {
      "algorithm": "kmeans",
      "n_clusters": 5
    }
  }
}

How Clustering Works

1. Extract Embeddings: Get embedding vectors from each document
2. Apply Algorithm: Run the clustering algorithm (e.g., k-means)
3. Assign Documents: Each document gets a cluster ID
4. Compute Centroids: Calculate cluster centers
5. Format Output: Return cluster documents, labeled documents, or one representative per cluster, depending on output_mode

The stage needs at least 3 documents with an embedding. With fewer, it returns no results and reports the reason under skipped in the stage metadata.

Output Schema

The shape depends on output_mode.

clusters (default)

The stage returns one document per cluster. Each has a generated document_id, the collection_id of the source documents, and a score equal to the member count. The members and the centroid are carried in payload.

{
  "document_id": "doc_a1b2c3",
  "collection_id": "col_articles",
  "score": 12.0,
  "cluster_id": 0,
  "member_count": 12,
  "representative_id": "doc_123",
  "payload": {
    "members": [
      {
        "document_id": "doc_123",
        "content": "Introduction to neural networks...",
        "cluster_distance": 0.15
      }
    ],
    "centroid": null
  }
}

representative_id is the member closest to the centroid. centroid holds the vector only when include_centroids is true and the query requests vectors. Otherwise it is null. At most max_members_per_cluster members are listed, and member_count counts all of them.

labeled

The stage returns the original documents with two fields added.

{
  "document_id": "doc_123",
  "content": "Introduction to neural networks...",
  "cluster_id": 0,
  "cluster_distance": 0.15
}

cluster_distance is the distance to the cluster centroid, so lower is closer. It is absent when the algorithm returns no centroid, and for noise documents (cluster_id: -1).

representatives

The stage returns one document per cluster, the one closest to the centroid.

{
  "document_id": "doc_123",
  "content": "Introduction to neural networks...",
  "cluster_id": 0,
  "cluster_distance": 0.15,
  "is_representative": true
}

The stage metadata reports algorithm, n_clusters, n_documents_in, n_documents_out, n_valid_embeddings and n_invalid_documents.

Performance

Metric	Value
Latency	50-200ms
Memory	O(N × embedding_dim)
Cost	Free
Scalability	Built for result sets of 10 to 500 documents

Clustering large document sets (over 1,000) can be slow. Consider pre-filtering or sampling before clustering.

Common Pipeline Patterns

Search + Cluster

[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
        ],
        "final_top_k": 100
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "kmeans",
        "n_clusters": 5
      }
    }
  }
]

Cluster + Sample Representatives

[
  {
    "stage_name": "hybrid_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 200 }
        ],
        "final_top_k": 200
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "kmeans",
        "n_clusters": 10,
        "output_mode": "labeled"
      }
    }
  },
  {
    "stage_name": "sample",
    "stage_type": "reduce",
    "config": {
      "stage_id": "sample",
      "parameters": {
        "strategy": "stratified",
        "stratify_by": "cluster_id",
        "min_per_stratum": 2
      }
    }
  }
]

Theme Discovery Pipeline

[
  {
    "stage_name": "structured_filter",
    "stage_type": "filter",
    "config": {
      "stage_id": "attribute_filter",
      "parameters": {
        "field": "metadata.date",
        "operator": "gte",
        "value": "2024-01-01"
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "hdbscan",
        "min_cluster_size": 5
      }
    }
  },
  {
    "stage_name": "summarize",
    "stage_type": "reduce",
    "config": {
      "stage_id": "summarize",
      "parameters": {
        "provider": "google",
        "model_name": "gemini-2.5-flash-lite",
        "prompt": "Summarize the main themes found in these clusters:\n\n{{DOCUMENTS}}"
      }
    }
  }
]

Diverse Results Pipeline

[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 }
        ],
        "final_top_k": 100
      }
    }
  },
  {
    "stage_name": "cluster",
    "stage_type": "reduce",
    "config": {
      "stage_id": "cluster",
      "parameters": {
        "algorithm": "kmeans",
        "n_clusters": 5,
        "output_mode": "representatives"
      }
    }
  }
]

Choosing n_clusters

Result Size	Recommended Clusters
< 50 docs	3-5 clusters
50-200 docs	5-10 clusters
200-500 docs	8-15 clusters
500+ docs	10-20 clusters

Start with fewer clusters and increase if clusters are too broad. Use hdbscan if you don’t know the optimal number.

Error Handling

Condition	Behavior
Document has no embedding	Left out of clustering
Fewer than 3 documents with embeddings	No results, with the reason under skipped in the metadata
No embedding can be identified	Request fails. Set feature_uri or run feature_search first
Clustering service fails	Request fails with a clustering error

Related

• Group By - Group by field values
• Sample - Select representatives
• MMR - Diversity in ranking
• Deduplicate - Remove duplicates