ELT/CDC: Pipeline definition import and export

Overview

Every pipeline can be exported from the dashboard as a YAML definition document and reimported to overwrite its configuration. The exported document uses the same schema as the REST API, so the same file works for both flows. Use this to:

Snapshot a pipeline before making large changes.
Copy a pipeline configuration between environments.
Review configuration changes in version control.

Export a pipeline

Open the pipeline’s detail page and go to the Settings tab.

In the Pipeline definition section, click Export.

A YAML file is downloaded with the pipeline’s current configuration. Sensitive fields (credentials, internal state, region routing) are omitted.

Import a pipeline definition

Importing overwrites the pipeline’s configuration to match the uploaded document.

Prerequisites

The pipeline must be stopped. Active or transitioning pipelines (status RUNNING, STARTING, or PAUSING) cannot be imported into. The Import button is disabled with a tooltip explaining why.

Stop the pipeline if it is currently running.

On the Settings tab, click Import in the Pipeline definition section.

Upload a YAML or JSON file with Choose file, or paste the document into the editor.

Click Import. The dashboard validates the document and applies the change. Validation errors are listed inline.

Pipeline definition document

The definition is a stable, customer-facing projection of the internal pipeline. It omits credentials, state backend configuration, region orchestration, and other internal fields. Both YAML and JSON map to the same schema, and unknown keys are rejected. The dashboard import/export and the REST API accept the same document.

Minimal example

definition_version: 1
name: orders-to-snowflake
source:
  connection:
    id: 42
destination:
  connection:
    id: 17
sync:
  frequency: 5
  auto_sync_new_tables: false
tables:
  - name: public.orders
  - name: public.customers
    columns:
      exclude:
        - ssn

Top-level fields

Field	Required	Description
`definition_version`	Yes	Contract version. Currently `1`.
`name`	Yes	Pipeline name.
`source.connection.id`	Yes	Existing source connection ID.
`destination.connection.id`	Yes	Existing destination connection ID.
`sync`	Yes	Sync schedule. See below.
`tables`	Yes	List of tables to replicate.
`transformations`	No	Field-level transforms applied to records.
`hooks`	No	Existing notification hooks to attach to pipeline events.

Sync schedule

Provide either frequency (which sets both source and destination) or both source_frequency and destination_frequency. Values are in minutes.

sync:
  source_frequency: 5
  destination_frequency: 15
  auto_sync_new_tables: true

Table selection

Each table can optionally select columns with include or exclude (not both):

tables:
  - name: public.orders
    append_only: false
    columns:
      include:
        - id
        - created_at
        - total_cents

Transformations

Supported type values: exclude_fields, rename_fields, toggle_case, find_and_replace, replace_in_field_name, history_mode.

transformations:
  - type: rename_fields
    table: public.orders
    config:
      mapping:
        order_id: id

Hooks

Attach existing notification hooks (created in Settings > Hooks) to pipeline events:

hooks:
  - id: 9
    events:
      - pipeline_failed
      - pipeline_stopped

Getting Started

Pipelines

Sources

Destinations

Monitoring

Security

Transforms

Data Observability

ELT/CDC: Pipeline definition import and export

Overview

Export a pipeline

Import a pipeline definition

Prerequisites

Pipeline definition document

Minimal example

Top-level fields

Sync schedule

Table selection

Transformations

Hooks

See also

​Overview

​Export a pipeline

​Import a pipeline definition

​Prerequisites

​Pipeline definition document

​Minimal example

​Top-level fields

​Sync schedule

​Table selection

​Transformations

​Hooks

​See also

Overview

Export a pipeline

Import a pipeline definition

Prerequisites

Pipeline definition document

Minimal example

Top-level fields

Sync schedule

Table selection

Transformations

Hooks

See also