Deduplication & Re-processing

Content-hash dedup decides what work is skipped, replaced, or forced when objects are processed into collections — and lineage decides what re-runs when a source changes

Every object ingested into Mixpeek is fingerprinted with a SHA-256 content hash, and every document produced from it carries that hash in its system envelope alongside a full lineage record. Together they answer the two questions every pipeline eventually asks:

1. Have I already processed this? — the content hash.
2. What downstream work is stale now that this changed? — the lineage chain.

​

Content hashing

The hash is computed from the source content at ingestion and stored:

• on the object (in the bucket), and
• on every document derived from it (content_hash in the document’s system envelope).

Two objects with identical bytes produce the same hash — regardless of filename, upload time, or path. A changed file produces a new hash, which is what marks its derived documents as stale.

“Latest version” of an asset is therefore a content-hash lookup, not a version counter you maintain yourself.

​

Dedup strategy

When you submit a batch, dedup_strategy controls how objects that were already processed in a prior batch are handled. Dedup is scoped to the (bucket, collection) pair — an object is a duplicate if the target collection already has documents produced from that same source object.

Strategy	Behavior	Cost profile
skip (default)	An object is reprocessed only if its content or the collection’s extractor configuration changed since it was last extracted; everything else is skipped	Free for unchanged content — no extraction, no inference
replace	Every object is reprocessed. Once its new documents are written, its earlier documents in the collection are deleted. If the batch fails, or fails on that object, its earlier documents are kept	Full extraction cost; use for a clean rebuild after the source content or the extractor configuration changed
force	Every object is reprocessed and its documents are overwritten in place. Documents a re-run no longer produces are not removed	Full extraction cost; prefer replace, which also removes those

{
  "bucket_id": "bkt_abc123",
  "collection_id": "col_xyz789",
  "dedup_strategy": "skip"
}

skip reprocesses an object when its content or the extractor configuration changed since it was last extracted, and skips everything else. It does not delete documents a re-run no longer produces, for example the extra chunks left behind when a new chunking setting yields fewer of them. For a clean rebuild that removes those, use replace.

​

The re-processing cascade

Derived assets stay in sync with their source through the combination of both mechanisms:

1. A source object’s content changes → its hash changes.
2. Re-submitting with replace (or re-syncing the source) reprocesses it — and every derived document downstream in its lineage (transcode → embed → tag) is rebuilt from the new source.
3. Sources whose hash didn’t change are skipped — unchanged content is never re-paid for.

This is what keeps N derived artifacts consistent with one upstream asset without a separate file system or job-tracking database: the document is the record, the hash is the version, and lineage is the dependency graph.

​

Related

• Lineage traversal — querying and walking derivation chains
• Processing pipeline — how objects become documents
• Batch ingestion at scale — batch mechanics, tiers, and monitoring