> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Universal Extractor

> All-in-one multimodal extractor — image, video, audio, and documents — producing 3072-d Gemini embeddings plus text descriptions, OCR, and transcription

> [!WARNING]
>
> Built-in extractor names are a **deprecated alias** — collections are now created by picking [features](https://docs.mixpeek.com/docs/processing/features.md). This pipeline is selected with `features: ["video_search"]`. Existing `feature_extractor` configs keep working; see the [migration guide](https://docs.mixpeek.com/docs/processing/extractor-migration.md).

> [card] [View on GitHub](https://github.com/mixpeek/mixpeek-extractors/blob/main/extractors/universal_extractor/README.md)
>
> Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.

The universal extractor is an all-in-one feature extractor that handles **image, video, audio, and documents** through Google's Gemini APIs. It produces a single 3072-dimensional embedding (Gemini Embedding 2) per object alongside rich text extraction: AI-generated descriptions, OCR for images and documents, and transcription for audio and video. It runs on the fast path, as a work-runtime task instead of a batch RayJob, so small batches skip cluster startup.

> [!NOTE]
>
> View extractor details at [api.mixpeek.com/v1/collections/features/extractors/universal\_extractor\_v1](https://api.mixpeek.com/v1/collections/features/extractors/universal_extractor_v1) or fetch programmatically with `GET /v1/collections/features/extractors/{feature_extractor_id}`.

## Pipeline Steps

1. **Resolve input** — apply `input_mappings` to get the file URL/path from the source object (`content` field).
2. **Detect modality** — classify the object as image, video, audio, or document.
3. **Segment (if needed)** — video and audio are processed in up to `max_video_segments` 30s segments; documents up to `max_document_pages` pages.
4. **Gemini embedding** — generate a 3072-d Gemini Embedding 2 vector (`output_dimensionality` configurable 256–3072).
5. **Text extraction** (if `extract_text`) — OCR for images/documents, transcription for audio/video.
6. **Description** (if `generate_description`) — Gemini vision/understanding produces a natural-language description.
7. **Output** — one document per object (or per segment/page for chunked content).

## When to Use

| Use Case | Description |
| - | - |
| **Mixed-modality corpora** | A single bucket containing images, video, audio, and PDFs you want searchable with one extractor |
| **Fast onboarding** | The fast path skips RayJob startup, so small batches return quickly |
| **Cross-modal search** | One shared 3072-d embedding space across all four modalities |
| **Rich metadata** | Need descriptions, OCR text, and transcription alongside the vector |

## When NOT to Use

| Scenario | Recommended Alternative |
| - | - |
| High-volume single-modality at lowest cost | Modality-specific extractor (`text_extractor`, `image_extractor`) |
| Audio fingerprinting / sound-mark matching | `audio_fingerprint_extractor` |
| Spatial/layout document analysis | `document_graph_extractor` |
| Self-hosted, no external API calls | `text_extractor` / `image_extractor` |

## Input Schema

| Field | Type | Required | Description |
| - | - | - | - |
| `content` | string | **Yes** | URL or path to the file to process. Populated from `input_mappings`. |

```json theme={null}
{
  "content": "s3://my-bucket/assets/report.pdf"
}
```

Supported input types: **IMAGE, VIDEO, AUDIO, PDF, TEXT, STRING**.

## Output Schema

| Field | Type | Description |
| - | - | - |
| `universal_extractor_v1_embedding` | float\[3072] | Gemini Embedding 2 vector for the content |
| `modality` | string | Detected modality: `image`, `video`, `audio`, or `document` |
| `text` | string \| null | Extracted text (OCR, transcription, or document text) |
| `description` | string \| null | AI-generated description of the content |
| `segment_index` | integer \| null | Segment index (chunked video/audio/documents) |
| `segment_total` | integer \| null | Total segments for this source object |
| `page_number` | integer \| null | Page number (documents only) |
| `start_time_s` / `end_time_s` | float \| null | Segment start/end time in seconds (video/audio) |
| `start_time` / `end_time` | float \| null | The same segment times under the name the other video extractors use (video/audio) |
| `duration_s` | float \| null | Total file duration in seconds (video/audio) |

```json theme={null}
{
  "universal_extractor_v1_embedding": [0.012, -0.034, 0.008, ...],
  "modality": "document",
  "text": "Quarterly revenue grew 12% year over year...",
  "description": "A financial report page with a revenue bar chart",
  "page_number": 1,
  "segment_total": 12
}
```

## Output by Modality

The extractor writes one document per unit: an image is one document, video and audio split into 30-second segments, and a document splits into pages. Every document carries `universal_extractor_v1_embedding`, `modality`, `text`, `description`, `segment_index`, and `segment_total`. The table lists the fields each modality adds and what fills `text` and `description`.

| Modality | Documents per object | Extra fields | `text` holds | `description` holds |
| - | - | - | - | - |
| `image` | 1 | none | The description and the OCR text, joined by a newline | Gemini description of the image |
| `video` | One per 30-second segment, up to `max_video_segments` | `start_time_s`, `end_time_s`, `start_time`, `end_time`, `duration_s`, `covered_s` | The segment description and the speech transcript, joined by a newline | Gemini description of the segment |
| `audio` | One per 30-second segment, up to `max_video_segments` | `start_time_s`, `end_time_s`, `start_time`, `end_time`, `duration_s`, `covered_s` | The transcript of the segment | Always null |
| `document` | One per page, up to `max_document_pages` | `page_number` | The PDF text layer, or Gemini text extraction for a page without one | A one-sentence summary on page 1, null on every other page |

`covered_s` is the number of seconds the indexed segments span, capped by `duration_s`. It shows when `max_video_segments` cut a long file short.

Two parameters change the fields. `generate_description: false` sets `description` to null. `extract_text: false` leaves OCR text, the transcript, and page text out of `text`.

If no segment of a video or audio file produces a document, the extractor writes one document for the whole file. That document has a null `text` and no `start_time_s` or `end_time_s`.

## What the Extractor Does Not Emit

The output has no field for a hook segment, a music or tone descriptor, keyframes, scene boundaries, or faces. Video segments are fixed 30-second windows that start at 0. The first segment (`segment_index` 0) spans the first 30 seconds of the file and cannot be narrowed to the first 3. The video `description` is free text, so no music or tone value is searchable as its own field.

To get these fields, use one of two paths:

* **Face data:** the [Face Identity Extractor](https://docs.mixpeek.com/docs/processing/extractors/face-identity.md) detects and embeds faces.
* **Hook, music, or tone fields:** build a [custom extractor](https://docs.mixpeek.com/docs/processing/custom-extractors.md) that writes them. Upload and deploy run on dedicated infrastructure. On the shared API, submit the extractor through [Extractor Submissions](https://docs.mixpeek.com/docs/processing/extractor-marketplace.md).

## Parameters

| Parameter | Type | Default | Range | Description |
| - | - | - | - | - |
| `output_dimensionality` | integer | `3072` | 256–3072 | Output embedding dimensions (Gemini Embedding 2 supports 256–3072) |
| `task_type` | string | `"RETRIEVAL_DOCUMENT"` | — | Embedding intent for Gemini Embedding 2. Common values: `RETRIEVAL_DOCUMENT`, `RETRIEVAL_QUERY`, `SEMANTIC_SIMILARITY` |
| `generate_description` | boolean | `true` | — | Generate a text description via Gemini vision/understanding |
| `extract_text` | boolean | `true` | — | Extract text (OCR for images/docs, transcription for audio/video) |
| `max_video_segments` | integer | `10` | 1–50 | Maximum number of 30s segments to process for video and audio files |
| `max_document_pages` | integer | `50` | 1–200 | Maximum number of pages to process for document files |
| `max_file_download_mb` | integer | `500` | 1–1024 | Maximum file download size (MB) for fast-path processing |
| `max_concurrency` | integer | `4` | 1–32 | Maximum per-task object concurrency for fast-path processing |

> [!WARNING]
>
> Dimensions are locked at namespace creation. Switching `output_dimensionality` on an existing namespace requires a migration since the vector index dimensionality is fixed.

## Configuration Examples

```json All-in-One Defaults theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "universal_extractor",
    "version": "v1",
    "input_mappings": {
      "content": "file_url"
    },
    "parameters": {}
  }
}
```

```json Embeddings Only (No Text/Description) theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "universal_extractor",
    "version": "v1",
    "input_mappings": {
      "content": "file_url"
    },
    "parameters": {
      "generate_description": false,
      "extract_text": false
    }
  }
}
```

```json Compact Embeddings for Clustering theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "universal_extractor",
    "version": "v1",
    "input_mappings": {
      "content": "file_url"
    },
    "parameters": {
      "output_dimensionality": 768,
      "task_type": "CLUSTERING"
    }
  }
}
```

```json Long-Form Video & Documents theme={null}
{
  "feature_extractor": {
    "feature_extractor_name": "universal_extractor",
    "version": "v1",
    "input_mappings": {
      "content": "file_url"
    },
    "parameters": {
      "max_video_segments": 30,
      "max_document_pages": 200
    }
  }
}
```

## Performance & Costs

| Metric | Value |
| - | - |
| **Compute** | Fast path (a work-runtime task, no RayJob startup) |
| **Cost** | See [Billing & Pricing](https://docs.mixpeek.com/docs/platform/billing.md) — rates come from `GET /v1/billing/pricing`. One charge covers all Gemini API calls (embedding, description, OCR/transcription) |
| **External API** | Google Gemini (embedding + vision/understanding) |
| **Max download** | 500 MB per object (configurable to 1024 MB) |

## Vector Index

| Property | Value |
| - | - |
| **Index name** | `universal_extractor_v1_embedding` |
| **Dimensions** | 3072 (configurable 256–3072) |
| **Type** | Dense |
| **Distance metric** | Cosine |
| **Inference model** | `google/gemini-embedding-2` |

## Limitations

* **External dependency**: Requires Google Gemini API availability; subject to its rate limits.
* **Per-object cost**: Higher per-object cost than self-hosted single-modality extractors.
* **Segment/page caps**: Video beyond `max_video_segments` and documents beyond `max_document_pages` are truncated.
* **Download ceiling**: Files larger than `max_file_download_mb` are skipped on the fast path.

## Related

* [Feature Extractors Overview](https://docs.mixpeek.com/docs/processing/feature-extractors.md)
* [Multimodal Extractor](https://docs.mixpeek.com/docs/processing/extractors/multimodal.md)
* [Gemini Multifile Extractor](https://docs.mixpeek.com/docs/processing/extractors/gemini-multifile.md)
* [Audio Fingerprint Extractor](https://docs.mixpeek.com/docs/processing/extractors/audio-fingerprint.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.