ETL: SFTP Destination

The SFTP destination writes pipeline output to any Secure File Transfer Protocol (SFTP) server. It supports CSV, TSV, JSON, Parquet, and Fixed width output formats, with optional compression and PGP encryption. Once your connection is set up, see File Storage Options for the full set of write options (output formats, compression, PGP, and more) shared across every file storage provider.

Connection Setup

See SFTP Source for steps to create and configure an SFTP connection in Integrate.io ETL, including firewall rules, authentication options, and SSH tunnel setup.

Destination Configuration

Connection

Select your SFTP connection from the dropdown, or create a new one.

Target Directory

The path on the SFTP server where output files will be written. Paths must begin with /. Integrate.io prepends the slash automatically if omitted. One or more files are created in this directory. By default, the job fails if the directory already exists. Use package variables to generate unique directory names per run (e.g. /output/${_JOB_ID}).

SFTP has no bucket or container field. The path is relative to the SFTP server root.

Destination Format

Format	Description
Delimited values	CSV, TSV, or any delimiter-separated format. Output files are UTF-8 encoded.
Line delimited JSON	One JSON object per line. UTF-8 encoded.
Parquet	Columnar format compatible with Athena, Impala, Presto, and other tools. `datetime` and complex types are not supported. Compression is not available for Parquet.
Fixed width	Records written as fixed-width lines with no separators. Requires configuring the Fixed Width Layout.

Delimited Values Options

Field	Description
Delimiter	Character separating fields. Select from the dropdown or type a single character. Default is tab.
String qualifier	If selected (`'` or `"`), fields containing the delimiter are enclosed in quotes.
Escape character	Available when a string qualifier is set. Character used to escape qualifier characters in field data.
Line ending	Unix/Linux (LF) or Windows (CRLF).
Write field names in header	Adds a header row with field names to each output file.

Header names with spaces: use triple underscores (___) in the alias or column name in a Select component to produce a space in the output header. To use a different separator character, add a package variable _FS_REPLACE_WITH_SPACE_IDENTIFIER with the character you want to replace.

Output Compression

Available for Delimited values, JSON, and Fixed width formats. Not available for Parquet.

Option	File suffix added
None	(none)
Gzip	`.gz` appended to output directory name
Bzip2	`.bz2` appended to output directory name

PGP Encryption

To encrypt output files with a trading partner's OpenPGP public key, select an active entry from the PGP Key dropdown. The entry must hold a public key. Files are encrypted before being written to the target path.

Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.

See PGP Keys for how to add and manage keys.

Destination Action

Controls what happens when the target directory already exists.

Action	Description
Fail if target directory exists	Default. The job fails if the destination directory already exists. Use variables to generate unique directory names per run (e.g. `/output/${_JOB_ID}`).
Delete target directory if exists	Deletes the existing directory before writing.
Replace files in directory	Writes to the existing directory. Files with the same names are overwritten. To add files without overwriting, use a unique file prefix (e.g. include `${_JOB_ID}`). Warning: the target directory is deleted on job failure.
Use intermediate storage, copy to existing directory	Writes to intermediate storage first, then copies to the existing target directory. The target directory must already exist. Supports character encoding selection.
Use intermediate storage, copy to new directory	Writes to intermediate storage first, then copies to a new target directory. Fails if the directory already exists.

Merge Output to Single File

Check Merge output to single file to write all output to a single file rather than multiple part files. Note that file size limits on some platforms may cause the job to fail for large outputs.

Character Encoding

Available only when Destination action is set to Use intermediate storage, copy to existing directory.

Option	Description
UTF-8	Default
UTF-16
Shift-JIS
CP932

Target File Names

Default pattern: output files are named part-[mr]-[0-9]{5} (e.g. part-m-00000). To use a custom pattern, select Custom pattern:

Field	Description
File name prefix	Leave empty to keep the default prefix (`part-[mr]-`), or enter a custom prefix. Supports package variables for dynamic names (e.g. a timestamp).
File name suffix	Integrate.io suggests a suffix based on the selected format and compression type.

Fixed Width Layout

When Destination format is set to Fixed width, a Fixed width layout step appears. The layout is a table with one row per incoming field, in schema order. Every incoming field is written with no separators between columns. To write only some fields or in a different order, add a Select transformation before this destination.

Column	Description
Field	Incoming field name. Read-only.
Width	Number of characters this field occupies. Whole number, 1-1000.
Align	Left or Right. Controls which side padding is added to.
Pad	Single character used to fill unused width. Defaults to a space.
If too long	Fail the job (stops on first overflow; always applies to numeric fields) or Truncate (text fields only).
Decimals	Implied decimal places for numeric fields. `0` writes the value as-is. With `2`, the value `12.34` is written as `1234`. Values are rounded HALF_UP. Whole number, 0-1000.
Sign	Where to place the sign for numeric fields: None, Leading (start of field), or Trailing (end of field).

The total record width in characters is shown below the table and updates as you edit.

Preview with Data

Click Preview with data to run the component through the previewer and see the actual records the job would write. Rows whose sample value does not fit the configured width are marked with an advisory message. Blocking errors (invalid width, pad, or decimals) appear in the errors box and prevent saving.

Persisted Layout Format

The layout is stored as fixed_width_destination_type on the component. Example:

{
  "fields": [
    {
      "name": "id",
      "width": 8,
      "align": "right",
      "pad": "0",
      "on_overflow": "fail",
      "implied_decimals": 0,
      "sign": "none"
    },
    {
      "name": "amount",
      "width": 12,
      "align": "right",
      "pad": " ",
      "on_overflow": "fail",
      "implied_decimals": 2,
      "sign": "leading"
    }
  ],
  "record_delimiter": "\n",
  "char_encoding": "UTF-8"
}

record_delimiter and char_encoding round-trip through the API but are not editable in the UI.

Skip Empty Output

Available for Delimited values, JSON, and Fixed width formats. When checked, no output files are written if the job produces zero records. Useful for downstream processes that should not receive empty files.

FAQ

Which destination actions are available for SFTP? All five destination actions are available: fail if exists, delete and recreate, replace files, copy to existing directory (with character encoding), and copy to new directory. Can I write to a different SFTP server than the one I read from? Yes. The source and destination components each have their own connection. You can use different servers, credentials, or paths.

Getting Started

Packages & Workflows

Sources

Transformations

Expressions & Functions

Destinations

Scheduling, Jobs & Clusters

Alerts & Monitoring

Security & Access

Account Management

Connection Setup

Destination Configuration

Connection

Target Directory

Destination Format

Delimited Values Options

Output Compression

PGP Encryption

Destination Action

Merge Output to Single File

Character Encoding

Target File Names

Fixed Width Layout

Preview with Data

Persisted Layout Format

Skip Empty Output

FAQ

SFTP Source

File Storage Options

Integrate.io IP Address List

SSH Tunnel Access

Using Variables in Packages

SFTPtoGo

​Connection Setup

​Destination Configuration

​Connection

​Target Directory

​Destination Format

​Delimited Values Options

​Output Compression

​PGP Encryption

​Destination Action

​Merge Output to Single File

​Character Encoding

​Target File Names

​Fixed Width Layout

​Preview with Data

​Persisted Layout Format

​Skip Empty Output

​FAQ

​Related

SFTP Source

File Storage Options

Integrate.io IP Address List

SSH Tunnel Access

Using Variables in Packages

SFTPtoGo

Connection Setup

Destination Configuration

Connection

Target Directory

Destination Format

Delimited Values Options

Output Compression

PGP Encryption

Destination Action

Merge Output to Single File

Character Encoding

Target File Names

Fixed Width Layout

Preview with Data

Persisted Layout Format

Skip Empty Output

FAQ

Related