> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Syncs

> Automatically ingest files from external storage into Mixpeek buckets on a schedule

Syncs pull files from your existing storage providers into Mixpeek buckets on a schedule. No migration required — connect your cloud storage, configure what to sync, and Mixpeek keeps your bucket up to date.

## How Syncs Work

```
┌──────────────┐     poll      ┌──────────────┐    create     ┌──────────────┐
│   External   │◄──────────────│    Sync      │───objects────►│    Bucket     │
│   Storage    │───file list──►│   Worker     │               │              │
│  (S3, GCS,   │               │              │───submit─────►│  Collection  │
│   Mux, etc.) │               │              │   batches     │   Pipeline   │
└──────────────┘               └──────────────┘               └──────────────┘
                                     │
                                     ▼
                              Resume cursor,
                              metrics, DLQ
```

1. **Poll** — At each interval, Mixpeek lists files from your storage provider using the configured `source_path` and filters.
2. **Filter** — Files are matched against glob patterns, size limits, MIME types, and provider-specific metadata filters.
3. **Create objects** — Matching files are registered in the target bucket. Duplicates are skipped by default via source tracking.
4. **Submit batches** — Objects are grouped into batches and submitted to the bucket's collection pipeline for processing.
5. **Checkpoint** — A resume cursor is saved so the next poll picks up where the last one left off.

## Sync Modes

| Mode | Behavior | Use Case |
| - | - | - |
| `continuous` | Polls repeatedly at `polling_interval_seconds` | Ongoing ingestion — new files are picked up automatically |
| `initial_only` | Runs once, then stops | One-time backfill or migration |

## Create a Sync

**1. Connect your storage**

Create a [storage connection](https://docs.mixpeek.com/docs/integrations/object-storage/overview.md) with credentials for your provider.

This step needs an API key with the **`admin`** permission, because connections hold credentials and sit behind organization administration. The key presets in Studio stop at read + write + delete; tick **Admin** in the key's Edit dialog to grant it. Everything below (buckets, syncs, triggers) works with a `write` key.

**2. Configure the sync**

```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "X-Namespace: $NAMESPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "connection_id": "conn_abc123",
    "source_path": "/videos/2025/",
    "sync_mode": "continuous",
    "polling_interval_seconds": 300,
    "skip_duplicates": true,
    "file_filters": {
      "include_patterns": ["*.mp4", "*.mov"],
      "max_size_bytes": 5368709120
    }
  }'
```

**3. Trigger the first sync**

```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/trigger" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "X-Namespace: $NAMESPACE_ID"
```

After the initial run, continuous syncs poll automatically at the configured interval.

## Configuration Reference

### Core Settings

| Field | Type | Default | Description |
| - | - | - | - |
| `connection_id` | string | required | Storage connection to pull from |
| `source_path` | string | required | Path in external storage (format varies by provider) |
| `sync_mode` | string | `continuous` | `continuous` or `initial_only` |
| `polling_interval_seconds` | int | `300` | Poll frequency (30–900 seconds) |
| `batch_size` | int | `50` | Files per batch (1–100) |
| `skip_duplicates` | bool | `true` | Skip files already in the bucket |
| `max_objects_per_run` | int | `100000` | Hard cap on objects processed in one sync run. Minimum `1` |
| `description` | string | — | Free-text description stored with the sync configuration |
| `reconcile` | object | — | Reconcile on source change: `on_delete` (cascade-delete objects when the source asset is removed, default `true`), `on_update` (propagate metadata changes + re-extract, default `true`; on Iconik, `true` repeats the per-asset calls for every asset on every walk, see [Iconik](https://docs.mixpeek.com/docs/integrations/object-storage/iconik.md#modification-detection)), `on_filter_drift` (drop objects that no longer match filters, default `true`) |

> [!NOTE]
>
> `max_objects_per_run` and `description` are accepted at create as well as on update. Both
> were previously settable only by `PATCH`ing a sync after creating it.

### File Filters

Narrow which files get synced. All filters combine with AND logic.

`include_patterns` and `exclude_patterns` are glob patterns, matched the same way for every provider. A pattern without `/` matches the file name in any folder, so `*.mp4` and `qa-*.mp4` match files at any depth. A pattern with `/` matches the path from the sync's source root, so `videos/*.mp4` matches only files in the `videos` folder.

When the filters exclude every file a run discovers, the run's warning says so, and the sync job's `metadata.files_excluded_by_filter` lists how many files each filter excluded with a few example paths.

```json theme={null}
{
  "file_filters": {
    "include_patterns": ["*.mp4", "*.mov", "*.webm"],
    "exclude_patterns": ["*/drafts/*", "*_temp.*"],
    "min_size_bytes": 1024,
    "max_size_bytes": 5368709120,
    "modified_after": "2025-01-01T00:00:00Z",
    "mime_types": ["video/mp4", "video/quicktime"]
  }
}
```

### Metadata Filters

Filter on provider-specific metadata fields. Useful for syncing only assets that match certain tags, statuses, or custom fields in your storage system.

```json theme={null}
{
  "file_filters": {
    "metadata_filters": [
      { "field": "status", "operator": "equals", "value": "approved" },
      { "field": "tags", "operator": "contains", "value": "hero" }
    ]
  }
}
```

Supported operators: `equals`, `not_equals`, `contains`, `not_contains`, `gt`, `lt`, `gte`, `lte`, `exists`.

### Schema Mapping

Map provider metadata to bucket schema fields during sync, so structured data arrives alongside your files.

```json theme={null}
{
  "schema_mapping": {
    "mappings": {
      "product_name": { "target_type": "field", "source": { "type": "metadata", "key": "title" } },
      "category": { "target_type": "field", "source": { "type": "tag", "key": "category" } }
    }
  }
}
```

## Lifecycle Management

### Pause and Resume

Temporarily stop a sync without losing progress:

```bash Pause theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/pause" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "X-Namespace: $NAMESPACE_ID"
```

```bash Resume theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/resume" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "X-Namespace: $NAMESPACE_ID"
```

### Manual Trigger

Force a sync to run immediately, outside the polling schedule:

```bash theme={null}
curl -X POST "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID/trigger" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "X-Namespace: $NAMESPACE_ID"
```

### Monitoring

Check sync status and metrics:

```bash theme={null}
curl "https://api.mixpeek.com/v1/buckets/$BUCKET_ID/syncs/$SYNC_ID" \
  -H "Authorization: Bearer $MIXPEEK_API_KEY" \
  -H "X-Namespace: $NAMESPACE_ID"
```

Response includes:

| Metric | Description |
| - | - |
| `total_files_discovered` | Cumulative files found in source |
| `total_files_synced` | Successfully synced files |
| `total_files_failed` | Files that failed after retries (sent to DLQ) |
| `total_bytes_synced` | Total data transferred |
| `last_sync_at` | When the last sync completed |
| `next_sync_at` | When the next poll is scheduled |
| `consecutive_failures` | Sequential failure count (auto-suspends after threshold) |

### Job timing

Each run is a sync job. List them at `GET /v1/buckets/{bucket_id}/syncs/{sync_config_id}/jobs`.

| Field | Meaning |
| - | - |
| `started_at` | When the current attempt started executing. `null` while the job waits to run. |
| `completed_at` | When the job finished. |
| `metadata.started_at_source` | Set to `running_transition` when `started_at` is a measured execution start. |
| `metadata.prior_attempts` | Earlier attempts of a job that was re-run, each with its own start and elapsed time. |
| `resumed_via` | `acks_late_redelivery` when the job was re-run after its worker died mid-run. |

A job has no pending status. It is created as `RUNNING` before it executes, so to find jobs that have not started yet, filter on `started_at` being `null` rather than on `status`.

`completed_at` minus `started_at` is the execution time of the current attempt. `started_at` minus the job's creation time covers dispatch, admission and setup, so it measures more than queue time.

> [!WARNING]
>
> Check `metadata.started_at_source` before you compute a duration. Jobs without it were recorded before September 14, 2026, and their `started_at` holds the time the record was created, so their execution time is unknown. Averaging durations across both kinds of job mixes measured times with creation times, and the result looks plausible.

`metadata.prior_attempts` only records attempts that carried `started_at_source`, so an older job without it may still have been retried. On those jobs `resumed_via` is the only retry signal, and its absence does not show that a job never retried.

## Robustness

Syncs are designed for unattended, long-running operation:

* **Distributed locking** prevents concurrent runs of the same sync
* **Resume cursors** checkpoint progress so interrupted syncs pick up where they left off
* **Dead letter queue** retries failed files up to 3 times before marking them as failed
* **Auto-suspend** pauses syncs after consecutive failures to prevent runaway errors
* **Idempotent ingestion** uses source tracking to never duplicate objects on retries
* **Reconciliation** (the `reconcile` object) cascades source deletes (`on_delete`), propagates metadata updates (`on_update`), and drops objects that no longer match your filters (`on_filter_drift`) — all default `true`

> [!WARNING]
>
> If a sync gets stuck (e.g., a worker crashed mid-run), use the [force unlock endpoint](https://docs.mixpeek.com/docs/api-reference/bucket-syncs/force-unlock-sync-configuration.md) to release the distributed lock.

[Sync API reference →](https://docs.mixpeek.com/docs/api-reference/bucket-syncs/create-sync-configuration.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.