When to use these tools
- You want an AI agent to build a new source-to-destination package in a single tool call from a natural-language prompt.
- You want the agent to clone an existing pipeline as a template, then swap in different connections, tables, or paths.
- You want the agent to add, edit, or remove components on an existing pipeline without opening the package designer.
- You want the agent to rename a package or define its public variables in place.
- You want the agent to fix a broken edge or component without opening the package designer.
- You want the agent to detect the columns of a CSV or other delimited file on an SFTP, S3, or GCS connection before wiring it into a flow.
- You want the agent to archive packages that failed an authoring attempt so they don’t clutter the active list.
Tool reference
build_pipeline (mutation)
Builds a complete source → destination pipeline in one call from a high-level intent. The server discovers the source schema, picks the matching component types for each connection, wiressource → select → destination with edges, maps fields 1:1, and persists the package. The agent only passes intent (which connections, which table or file path, which fields, any computed columns), so the build turn stays fast even on wide schemas.
Prefer build_pipeline over create_package for a straightforward “load X into Y” build. Use create_package or add_package_components when the shape isn’t covered here: filters, joins, multiple sources, a file or cloud-storage destination, or edits to an existing package.
string
required
Package name shown in the dashboard.
object
required
Source intent. See below.
object
required
Destination intent. Must be a database or warehouse connection (for example Snowflake, Redshift, BigQuery, MySQL, Postgres). File and cloud-storage destinations are not yet supported here, use
create_package for those."all" or array
default:"\"all\""
"all" (default) carries every discovered source column. An array of source column names keeps only that subset.array
Derived columns to append. Each entry:
{ "name": "<output column>", "expression": "<Pig expression>" }. A computed name equal to a source column overrides that column.string
default:"dataflow"
dataflow (default) or workflow.integer
Must belong to the calling account. Omit to create the package as Unassigned.
string
Free-form, up to 4096 characters.
source accepts:
connection_id(integer, required).path(string) for file or cloud-storage sources (SFTP, S3, GCS, SharePoint).table_name(string) for database or warehouse sources.schema_name(string), optional, database sources only.delimiter(string), file field delimiter, default,.header_row(boolean), whether the file has a header row, defaulttrue.source_path_field_alias(string), optional, file and cloud-storage sources only. When set, adds an output column with this exact name carrying the source file path each row was read from. Use this when the agent wants the filepath as a column for lineage. Never hardcode the literal path as a field value.
destination accepts:
connection_id(integer, required).table_name(string, required).schema_name(string), optional.operation_type(string), write mode (for exampleappend,truncate_and_insert,merge). Defaults toappend. Validated against the destination’s allowedwrite_modesfromdescribe_component_type.create_table(boolean), auto-create the table, defaulttrue.merge_keys(array of string), required for merge and upsert modes (merge,merge_update_and_insert,insert_or_update). Each name flags the matching destination column withis_merge_key: true. The platform never infers the key, so a merge withoutmerge_keyswill not upsert correctly. Append, overwrite, and truncate modes ignore this field.
package_id and also runs validate_package automatically so the caller knows whether the build is clean on the first response.
Example: SFTP to Snowflake with a merge key and filepath column
Error envelopes
create_package (mutation)
Mints a new package from a fulldata_flow_json spec. Components and edges are persisted in a single transaction, partial packages are never written.
string
required
Package name shown in the dashboard.
string
required
dataflow or workflow.array
required
Array of single-key wrapper hashes (see below).
array
required
Array of edge hashes referencing component
ids.integer
Must belong to the calling account. Omit to create the package as Unassigned.
string
Free-form, up to 4096 characters.
string
default:"2.0.0"
Defaults to
2.0.0 (the modern package designer).variables is not accepted. Packages that need package-level variables must be created through the REST API.
Component shape
Each component is a single-key wrapper hash. The key is the wrapper type (for exampledatabase_source_component), and the inner hash carries the configuration. Common fields:
id, stable identifier referenced by edges. Convention iscomponent-<hex>. Omit to let the server generate one.name, human-readable label shown on the canvas.xy,[x, y]position pair.alias, data-flow alias. Sources expose this; destinations consume it viainput_alias.connection, connection metadata block. Uses the generic keyconnection, not type-specific keys. Shape:{ "id": <integer>, "name": "<display name>", "type": "<connection-type-slug>" }. Withoutnameandtype, the dashboard can’t render the connection chip. Whenupdate_package_componentschanges a*_connection_id, the server clears this block and re-embeds it from the new connection so the dashboard chip stays in sync with the connection the job will run.<type>_connection_id, string form of the connection id (for example"cloud_storage_connection_id": "47376").specificComponentType, vendor sub-type (for examplesftp_source_component,mysql_source_component). Required when the wrapper covers multiple vendors so the dashboard renders the correct icon and form.schema, nested object{ "fields": [{ name, alias, data_type }, ...] }. Each field needs analias.
Edge shape
Edges reference components by theirid field:
id and label are optional, the server generates them when absent.
Best practice: clone from an existing package
The safest way to compose a validdata_flow_json is to read a similar existing package first:
- Call
list_packagesand find a pipeline with the same source and destination connection types. - Call
get_package(<that_id>, include_full_graph: true)and inspect itsdata_flow_json. - Use the result as a template. Swap in your own connection ids and table or file paths. Keep the structural keys.
api_key query params, Authorization headers, basic-auth passwords, and similar) come back as [REDACTED] in the full graph. Structure and non-secret fields are preserved, so the result is still a valid template, refer real credentials through a connection or a secret variable when you re-author the package. See Inline secret redaction in get_package.
describe_component_type output is sufficient to build a component on its own. Every component returns an example (real fixture or generated skeleton), property_values publishes the allowed values for enum-constrained properties, and example_omitted names the required properties the example could not render. Reading a similar package with get_package is an optional cross-check, useful mainly for the nested shapes named in example_omitted.
Returns
On success:{ package_id, package_version, name, flow_type, flow_version, workspace_id, created_at, warnings }. The warnings array lists shape fixups the server applied (for example rewriting a vendor-named wrapper to the canonical wrapper plus specificComponentType).
On failure: { error: "..." }, workspace not in account, name too long, save failure, and so on.
clone_package (mutation)
Mutation. Copies an existing package into a NEW package to use as a template, without editing the original. The clone copies the source’sdata_flow_json, variables, description, and flow type, is attributed to the calling user, and gets a new package_id. For workflow packages, sub-package references inside data_flow_json are preserved as-is, not recursively deep-cloned.
integer
required
Package to copy.
string
required
Name for the new package (up to 128 characters).
integer
Workspace to place the clone in. Must belong to the calling account. Defaults to the source package’s workspace.
string
Description for the new package, up to 1024 characters. Defaults to the source package’s description.
boolean
default:"false"
Return what would be created without writing.
{ new_package_id, source_package_id, name, flow_type, workspace_id, owner_id, created_at } on success.
add_package_components (mutation)
Appends one or more NEW components (and, optionally, new edges) to an existing package’s flow. This is the “extend a package the user already has open” operation, for example “add a Select component” or “insert a Filter between the source and destination.” It fills the gap betweencreate_package (builds a whole package) and update_package_components (only edits components that already exist).
integer
required
Package to extend.
array
required
One or more single-key wrapper hashes, the same shape
create_package uses. See Component shape.array
New edges wiring the added components in. Omit to add components unwired, then call
update_package_edges.create_package. Two placement details are specific to adding:
namemust be unique within the package, it is howupdate_package_componentslater targets the component.xyis optional when the component is wired to an existing one: the server places a newly-added component just below its parent and nudges the downstream chain down to make room. Provide an explicitxyonly for a standalone component with no incoming edge (for example a new source).
discover_schema, discover_file_schema, or preview_data returned them, same spelling and case. Never add an f_ prefix or rename a field; the platform makes Pig aliases valid and reserved-word-safe automatically.
The call is all-or-nothing and row-locked: if any new component’s name or id collides with one already in the flow, nothing is written. Each successful call bumps the package version via PaperTrail and is reversible from the version history UI.
Returns { package_id, components_added, edges_added, new_version, updated_at } on success, plus a warnings array listing any wrapper-shape fixups the server applied.
update_package_components (mutation)
Mutation. Edits specific components that ALREADY EXIST in a package’sdata_flow_json. Each entry is { component_name, updates }, where component_name matches the inner name field on a component and updates is a hash of fields to merge onto it. Untouched top-level fields are preserved, and nested hashes are merged recursively (deep merge), so a partial config update does not drop sibling keys. To ADD a new component use add_package_components, to REMOVE one use remove_package_components, and to re-wire edges use update_package_edges.
integer
required
Package to update.
array
required
At least one entry. Each entry is
{ component_name, updates }: component_name matches a component’s inner name, and updates is a hash of fields to deep-merge onto that component. The name and id fields are protected and cannot be overwritten, since edges reference them. Do not place credentials in updates; connection secrets live in connection management, and putting them in data_flow_json leaks them through the version history.boolean
default:"false"
Report which components would change without writing.
component_name is not found in the flow, the entire update is rejected and nothing is persisted. When an updates hash changes a *_connection_id, the server clears the stale connection display block and re-embeds it from the new connection so the dashboard chip stays in sync. The package row is locked (SELECT FOR UPDATE) for the save so concurrent agent calls serialize correctly. Each successful call bumps the package version via PaperTrail and is reversible from the version history UI.
Returns { package_id, components_updated, new_version, updated_at } on success.
list_component_types (read)
Enumerates every component registered in the platform catalog. Returns one entry per concrete subclass with:name, internal name (for examplemysql_source_component). Pass this todescribe_component_type.category, one ofsource,transformation,destination, orworkflow. Workflow tasks (for examplerun_package_component) categorize asworkflow, nottransformation.class_name, Ruby class name, informational.description, header docstring from the component’s source file.wrapper_key, the outer key to use indata_flow_json. May benullfor transformations or unmapped classes.specific_component_type, value for the innerspecificComponentTypefield. May benullwhen the vendor has its own top-level wrapper.
category argument filters the result. Accepts source, transformation, destination, or workflow.
describe_component_type output is sufficient to build a component. Reading a similar package with get_package is an optional cross-check, not a prerequisite.
describe_component_type (read)
Per-component introspection by internal name. Returns:properties, writable attribute names extracted fromattr_accessordeclarations.required, attribute names with unconditional presence validators.conditionally_required, attributes required only when a gating flag or mode is set (for exampleerrors_output_pathonly whentrack_errorsis true). Do not set these unless the gating condition applies.category,source,transformation,destination, orworkflow.description, header docstring.property_values, allowed values for properties this component constrains to a fixed set (for examplemethod,response_type,file_format,pagination_scheme). If a property is listed here you must use one of its values.operation_typeis never here; it appears underwrite_modeswhen the destination has curated modes.property_value_conditions, the subset ofproperty_valuesthat only takes effect when another property is set a certain way. Keyed as{"<property>": {"property": "<gate>", "must_be": true|false}}. For example,pagination_schemerequiresuse_pagination: true. Sending a scheme on its own validates green and then fetches only the first page.write_modes, theoperation_typevalues a destination supports, with labels forpresent_options.nullfor non-destinations and destinations without selectable modes. Whennull, do not guess anoperation_type. Cloud-storage destinations derive it fromdestination_exists_action+delete_destination_path.example, a ready component shape. Always present.example_source,"fixture"when the example comes from a real component,"generated"when it is a minimal shape built fromrequired+property_values. In a generated example, values written<like this>are placeholders to fill in. Enum properties render as a choice (for example<GET|POST|PATCH|PUT>), never a defaulted first value.example_omitted, required properties the example leaves out. Two kinds: plain scalars only the caller knows (table_name,object_name,path,url, and so on), and nested objects (schema,predicates,relations,aggregated_fields,destination_type) whose inner shape the example cannot render. For a nested one, look up its shape in the Component shape section oncreate_packageand in this component’sdescription.wrapper_key,specific_component_type,valid_specific_types,wrapper_example, copy-paste-ready wrapper recipe.
type (the name from list_component_types). Returns { error: "unknown component type: <type>" } for unknown names.
A property in neither required nor conditionally_required, and not called out in description, is optional.
Connections are the exception: a connection property absent from those lists does not mean the component works without one. Some components declare no validator and simply fail later. Treat a connection as needed unless properties also offers an inline-auth alternative (headers, or username + password). Only then may you ask which auth method to use (none, a token or key in headers, Basic, or an existing connection).
update_package_edges (mutation)
Replaces a package’s edges array wholesale. Pass the complete desired edges array, existing edges are overwritten.integer
required
Package to update.
array
required
Full edges array. Each edge needs at minimum
source and target matching a component id in the current package.source and target against the package’s current components. If any edge references an unknown component, the entire call is rejected, no partial writes. Each write bumps the package version via PaperTrail and is reversible through the existing UI history.
Use this when validate_package surfaces an edge-related error after create_package and you need to fix wiring in place. For component-internal edits (changing a connection, adjusting a schema), use update_package_components.
remove_package_components (mutation)
Removes one or more components from an existing package’s flow by componentname. Any edge whose source or target was a removed component is dropped at the same time, so the package never ends up with dangling edges.
integer
required
Package to update.
array
required
Inner
name values from the package’s components. The same key update_package_components targets.add_package_components. To edit one in place, use update_package_components. To archive the entire package, use delete_package.
Returns { package_id, components_removed, edges_removed, new_version, updated_at } on success.
rename_package (mutation)
Renames a package and, optionally, updates its description. Metadata-only.data_flow_json (components and edges) is left untouched.
integer
required
Package to rename.
string
required
New package name (3 to 128 characters).
string
Free-form, up to 1024 characters.
{ package_id, previous_name, name, updated_at } on success.
manage_package_variables (mutation)
Defines, updates, or removes a package’s public variables. Public variables are the named defaults a package exposes; per-run overrides are still passed throughrun_package’s variables argument.
integer
required
Package to update.
object
Hash of
{ variable_name => default_value } to add or update. Values are stored as given; for Pig string literals, follow the embedded single-quote rule used by run_package. At least one of set or remove is required.array
Variable names to delete. At least one of
set or remove is required.set would store it as a plaintext public variable.
Returns { package_id, variables } (the full resulting public-variable map) on success.
File-schema detection is covered by discover_file_schema on the Reading & Inspecting page.
delete_package (mutation)
Archives a package, same semantics as the dashboard’s Archive action. TheJob row is preserved with status archived, removed from the active package list, and remains queryable via list_packages(status: 'archived').
integer
required
Package to archive.
toggle_schedule(enabled: false).
Use this to clean up after an unsalvageable create_package attempt so failed iterations don’t accumulate as dead rows.
Returns { package_id, status: 'archived', archived_at } on success, or { error: "..." } for package-not-found or rejected transitions.
Version history
Every write through the package-authoring tools is versioned via PaperTrail. These two tools let an agent inspect that history and roll back a change.list_package_versions (read)
Read-only. Lists a package’s saved version history (PaperTrail snapshots), oldest first, with the current state last. Summary only, not the fulldata_flow_json of each version. Up to 100 entries.
integer
required
The package (job) id.
array
Each entry:
version (the index to pass to revert_package), package_version, description, author, created_at, component_count, and current. To undo the most recent change, use the highest-numbered current: false entry (the one immediately before the current entry).revert_package (mutation)
Mutation. Forward-only rollback to a prior version returned bylist_package_versions. It restores the chosen version’s data_flow_json as a NEW version, so the revert is itself undoable, and leaves the package’s schedules and run history untouched.
integer
required
Package to revert.
integer
required
The index of a
list_package_versions entry whose current is false. That list is oldest-first with the current state last, so to undo the most recent change pick the highest-numbered current: false entry (the one immediately before the current entry). Never pass the current entry.boolean
default:"false"
Report what would be reverted without writing.
{ package_id, reverted_to_version, restored_from_package_version, new_package_version, updated_at } on success.
Recommended agent flow
Example: create a minimal SFTP-to-S3 package
package_id. Pass it to validate_package next, fix any errors with update_package_components or update_package_edges, then run with run_package.
Error envelopes
Mutating tools return a plain{ "error": "..." } object instead of a JSON-RPC error when the failure is expected and recoverable. Common cases:
Unrecoverable failures (auth, malformed JSON-RPC) return standard JSON-RPC error responses, see the MCP Server overview for HTTP-level errors.
Audit trail
Every write performed through these tools is captured in your account’s audit history via PaperTrail. Package creates, edge updates, and archive transitions all record the calling user as the author. Component edits made byupdate_package_components are versioned and reversible through the existing package version history UI.