Universal Extractor

All-in-one multimodal extractor — image, video, audio, and documents — producing 3072-d Gemini embeddings plus text descriptions, OCR, and transcription

Built-in extractor names are a deprecated alias — collections are now created by picking features. This pipeline is selected with features: ["video_search"]. Existing feature_extractor configs keep working; see the migration guide.

View on GitHub

Runnable reference for this extractor — inputs, parameters, output fields, embedding models, and copy-paste examples. Auto-generated from the live registry.

The universal extractor is an all-in-one feature extractor that handles image, video, audio, and documents through Google’s Gemini APIs. It produces a single 3072-dimensional embedding (Gemini Embedding 2) per object alongside rich text extraction — AI-generated descriptions, OCR for images and documents, and transcription for audio and video. It runs on Celery (not Ray) for zero cluster-startup latency, making it a fast path for mixed-modality corpora.

View extractor details at api.mixpeek.com/v1/collections/features/extractors/universal_extractor_v1 or fetch programmatically with GET /v1/collections/features/extractors/{feature_extractor_id}.

Pipeline Steps

1. Resolve input — apply input_mappings to get the file URL/path from the source object (content field).
2. Detect modality — classify the object as image, video, audio, or document.
3. Segment (if needed) — video and audio are processed in up to max_video_segments 30s segments; documents up to max_document_pages pages.
4. Gemini embedding — generate a 3072-d Gemini Embedding 2 vector (output_dimensionality configurable 256–3072).
5. Text extraction (if extract_text) — OCR for images/documents, transcription for audio/video.
6. Description (if generate_description) — Gemini vision/understanding produces a natural-language description.
7. Output — one document per object (or per segment/page for chunked content).

When to Use

Use Case	Description
Mixed-modality corpora	A single bucket containing images, video, audio, and PDFs you want searchable with one extractor
Fast onboarding	Celery fast-path avoids Ray cluster startup, so small batches return quickly
Cross-modal search	One shared 3072-d embedding space across all four modalities
Rich metadata	Need descriptions, OCR text, and transcription alongside the vector

When NOT to Use

Scenario	Recommended Alternative
High-volume single-modality at lowest cost	Modality-specific extractor (text_extractor, image_extractor)
Audio fingerprinting / sound-mark matching	audio_fingerprint_extractor
Spatial/layout document analysis	document_graph_extractor
Self-hosted, no external API calls	text_extractor / image_extractor

Input Schema

Field	Type	Required	Description
content	string	Yes	URL or path to the file to process. Populated from input_mappings.

{
  "content": "s3://my-bucket/assets/report.pdf"
}

Supported input types: IMAGE, VIDEO, AUDIO, PDF, TEXT, STRING.

Output Schema

Field	Type	Description
universal_extractor_v1_embedding	float[3072]	Gemini Embedding 2 vector for the content
modality	string	Detected modality: image, video, audio, or document
text	string | null	Extracted text (OCR, transcription, or document text)
description	string | null	AI-generated description of the content
segment_index	integer | null	Segment index (chunked video/audio/documents)
segment_total	integer | null	Total segments for this source object
page_number	integer | null	Page number (documents only)
start_time_s / end_time_s	float | null	Segment start/end time in seconds (video/audio)
duration_s	float | null	Total file duration in seconds (video/audio)

{
  "universal_extractor_v1_embedding": [0.012, -0.034, 0.008, ...],
  "modality": "document",
  "text": "Quarterly revenue grew 12% year over year...",
  "description": "A financial report page with a revenue bar chart",
  "page_number": 1,
  "segment_total": 12
}

Output by Modality

The extractor writes one document per unit: an image is one document, video and audio split into 30-second segments, and a document splits into pages. Every document carries universal_extractor_v1_embedding, modality, text, description, segment_index, and segment_total. The table lists the fields each modality adds and what fills text and description.

Modality	Documents per object	Extra fields	text holds	description holds
image	1	none	The description and the OCR text, joined by a newline	Gemini description of the image
video	One per 30-second segment, up to max_video_segments	start_time_s, end_time_s, duration_s, covered_s	The segment description and the speech transcript, joined by a newline	Gemini description of the segment
audio	One per 30-second segment, up to max_video_segments	start_time_s, end_time_s, duration_s, covered_s	The transcript of the segment	Always null
document	One per page, up to max_document_pages	page_number	The PDF text layer, or Gemini text extraction for a page without one	A one-sentence summary on page 1, null on every other page

covered_s is the number of seconds the indexed segments span, capped by duration_s. It shows when max_video_segments cut a long file short.

Two parameters change the fields. generate_description: false sets description to null. extract_text: false leaves OCR text, the transcript, and page text out of text.

If no segment of a video or audio file produces a document, the extractor writes one document for the whole file. That document has a null text and no start_time_s or end_time_s.

What the Extractor Does Not Emit

The output has no field for a hook segment, a music or tone descriptor, keyframes, scene boundaries, or faces. Video segments are fixed 30-second windows that start at 0. The first segment (segment_index 0) spans the first 30 seconds of the file and cannot be narrowed to the first 3. The video description is free text, so no music or tone value is searchable as its own field.

To get these fields, use one of two paths:

• Face data: the Face Identity Extractor detects and embeds faces.
• Hook, music, or tone fields: build a custom extractor that writes them. Upload and deploy run on dedicated infrastructure. On the shared API, submit the extractor through Extractor Submissions.

Parameters

Parameter	Type	Default	Range	Description
output_dimensionality	integer	3072	256–3072	Output embedding dimensions (Gemini Embedding 2 supports 256–3072)
task_type	string	"RETRIEVAL_DOCUMENT"	—	Embedding intent for Gemini Embedding 2. Common values: RETRIEVAL_DOCUMENT, RETRIEVAL_QUERY, SEMANTIC_SIMILARITY
generate_description	boolean	true	—	Generate a text description via Gemini vision/understanding
extract_text	boolean	true	—	Extract text (OCR for images/docs, transcription for audio/video)
max_video_segments	integer	10	1–50	Maximum number of 30s segments to process for video and audio files
max_document_pages	integer	50	1–200	Maximum number of pages to process for document files
max_file_download_mb	integer	500	1–1024	Maximum file download size (MB) for Celery fast-path processing
max_concurrency	integer	4	1–32	Maximum per-task object concurrency for Celery fast-path processing

Dimensions are locked at namespace creation. Switching output_dimensionality on an existing namespace requires a migration since the vector index dimensionality is fixed.

Configuration Examples

All-in-One Defaults

{
  "feature_extractor": {
    "feature_extractor_name": "universal_extractor",
    "version": "v1",
    "input_mappings": {
      "content": "file_url"
    },
    "parameters": {}
  }
}

Performance & Costs

Metric	Value
Compute	Celery fast-path (no Ray cluster startup)
Cost	See Billing & Pricing — rates come from GET /v1/billing/pricing. One charge covers all Gemini API calls (embedding, description, OCR/transcription)
External API	Google Gemini (embedding + vision/understanding)
Max download	500 MB per object (configurable to 1024 MB)

Vector Index

Property	Value
Index name	universal_extractor_v1_embedding
Dimensions	3072 (configurable 256–3072)
Type	Dense
Distance metric	Cosine
Inference model	google/gemini-embedding-2

Limitations

• External dependency: Requires Google Gemini API availability; subject to its rate limits.
• Per-object cost: Higher per-object cost than self-hosted single-modality extractors.
• Segment/page caps: Video beyond max_video_segments and documents beyond max_document_pages are truncated.
• Download ceiling: Files larger than max_file_download_mb are skipped on the Celery fast-path.

Related

• Feature Extractors Overview
• Multimodal Extractor
• Gemini Multifile Extractor
• Audio Fingerprint Extractor