Connection Setup
See Amazon S3 Source for steps to create an IAM user, attach the required policy, and set up the connection in Integrate.io ETL. For a destination pipeline, the IAM user needss3:ListBucket, s3:PutObject, s3:GetObject, and s3:DeleteObject on the target bucket.
Destination Configuration
Connection
Select your Amazon S3 connection from the dropdown, or create a new one.Bucket
Enter the name of the S3 bucket where output files will be written.Target Directory
The path within the bucket where output files are created. One or more files will be written to this directory. By default, the job fails if the directory already exists. Use package variables to generate unique directory names per run (e.g./output/${_JOB_ID}).
Destination Format
Delimited Values Options
Header names with spaces: use triple underscores (
___) in the alias or column name in a Select component to produce a space in the output header. To use a different separator character, add a package variable _FS_REPLACE_WITH_SPACE_IDENTIFIER with the character you want to replace.
Output Compression
Available for Delimited values, JSON, and Fixed width formats. Not available for Parquet.PGP Encryption
To encrypt output files with a trading partner's OpenPGP public key, select an active entry from the PGP Key dropdown. The entry must hold a public key. Files are encrypted before being written to the target path.Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.
Destination Action
Controls what happens when the target directory already exists.Merge Output to Single File
Check Merge output to single file to write all output to a single file rather than multiple part files. Note that file size limits on some platforms may cause the job to fail for large outputs.Character Encoding
Available only when Destination action is set to Use intermediate storage, copy to existing directory.Target File Names
Default pattern: output files are namedpart-[mr]-[0-9]{5} (e.g. part-m-00000).
To use a custom pattern, select Custom pattern:
Fixed Width Layout
When Destination format is set to Fixed width, a Fixed width layout step appears. The layout is a table with one row per incoming field, in schema order. Every incoming field is written with no separators between columns. To write only some fields or in a different order, add a Select transformation before this destination.
The total record width in characters is shown below the table and updates as you edit.
Preview with Data
Click Preview with data to run the component through the previewer and see the actual records the job would write. Rows whose sample value does not fit the configured width are marked with an advisory message. Blocking errors (invalid width, pad, or decimals) appear in the errors box and prevent saving.Persisted Layout Format
The layout is stored asfixed_width_destination_type on the component. Example:
record_delimiter and char_encoding round-trip through the API but are not editable in the UI.
Skip Empty Output
Available for Delimited values, JSON, and Fixed width formats. When checked, no output files are written if the job produces zero records. Useful for downstream processes that should not receive empty files.FAQ
What permissions does the IAM user need for a destination pipeline? The IAM user needss3:ListBucket on the bucket and s3:PutObject, s3:GetObject, and s3:DeleteObject on its objects. All three object actions are required because Integrate.io finalizes output files with a copy-and-delete operation.
Can I write to a different bucket than the one I read from?
Yes. The source and destination components each have their own connection and bucket settings. You can use different buckets, different IAM users, or even different AWS accounts.
Does S3 support all destination action options?
Yes. All five destination action options are available for Amazon S3.