Connection Setup
Create an IAM User
1
Log into the AWS Management Console and open IAM (Identity and Access Management).
2
In the left navigation, click Users, then click Add User.
3
Enter a user name (for example, 
integrate-io-etl) and select Programmatic access. Click Next: Permissions.4
Click Create User. On the confirmation page, click Download .csv and save the credentials file. You will need the Access Key ID and Secret Access Key when creating the connection in Integrate.io.
If any credential key contains a slash, or you cannot locate the downloaded credentials file, return to the IAM user, click Make inactive under Security credentials, then click Create access key to generate new credentials.
Never use an underscore in an S3 bucket name, and never end a bucket name with a dash. Both will prevent Integrate.io from accessing the bucket correctly.
Attach an S3 Policy to the IAM User
1
In the IAM user list, click the user you created. Scroll down to Add permissions.
2
For read-only access, attach the AmazonS3ReadOnlyAccess managed policy.
3
For read-write access (required for destination pipelines), click Create policy, choose JSON, and paste the following policy, replacing Click Review policy, then Create policy.
bucket-name with your bucket name:All three object actions are required for destination pipelines. Integrate.io writes each output file to a temporary object and then finalizes it, which on S3 is a copy-and-delete operation. Granting only
s3:PutObject causes writes to fail during finalization.Create the Connection in Integrate.io ETL
1
Click the Connections icon (lightning bolt) in the top-left menu.
2
Click New connection.
3
Select Amazon S3.
4
Name the connection and enter the Access Key ID and Secret Access Key from the credentials file.
5
If your S3 buckets are in a region that requires AWS Signature Version 4, enter the region. See the AWS region list for applicable regions.
6
Click Test connection to verify the credentials, then click Create amazon s3 connection.
Source Configuration
Connection
Select your Amazon S3 connection from the dropdown, or create a new one using the steps above.Bucket
Enter the name of the S3 bucket that contains your source files.Source Path
The path to the input file, folder, or pattern within the bucket.
You can use wildcard characters for pattern globbing. File and directory names beginning with an underscore (
_) or a dot (.) are ignored.
Source File Format
Record delimiter: what breaks the data into records:- New line (
\n,\n\r,\r): each line in the file is one record. - End of file: each file is treated as a single record.
Avro file format is not supported.
.zip, a Case-insensitive path checkbox appears. Enable it to match the zip file path without regard to letter case.
To read ASC X12 EDI files such as 834 benefit enrollment files, use the dedicated EDI (X12) source instead.
Delimited Values Parameters
Applies when record delimiter is New line and record type is Delimited values.1
Field delimiter: select from the dropdown (comma, tab) or type a single character or escape sequence:
2
String qualifier: if fields are enclosed in quotes, select
' or ". To allow line breaks inside quoted fields, select " (newline inside) or ' (newline inside). Use the newline-inside option with caution, as unbalanced quotes can affect job performance.3
Escape character: available when a string qualifier is set. Specifies the character used to escape qualifier characters within field data.
4
First row contains column names: check to skip the header row in each file.
5
Detect header by name: locates the header row dynamically by matching your schema column names. Use this when the header is not always on row 1 (for example, files with a leading title or blank rows before the header). The full schema header must still appear in column order in each file.
Excel Parameters
Applies when record type is Excel.Password-Protected Workbooks
If a workbook is encrypted with a password (set via Protect Workbook in Excel), check This file is password protected and enter the password in the File Password field. Integrate.io decrypts the workbook during schema preview and job runs. Passwords containing special characters such as$ are supported.
The checkbox appears when the source path ends in .xlsx or .xlsm. Legacy .xls files are not supported for password decryption. Unchecking the box clears any saved password.
Password-protected workbooks do not use streaming mode. When a password is set, enabling Ingest large Excel file reads the workbook in standard mode.When streaming a single sheet, date cells come through as Excel serial numbers (e.g.
45306). Declare those columns as int or decimal in your schema, or transform them downstream.JSON Parameters
Applies when record type is JSON object. Base record JSONPath Expression: defines the base record and extracts nested objects or arrays.
Read more about processing JSON data.
Source Action
All four source actions are available for Amazon S3. When using Move processed files, the Append string to filename option is supported.
Incremental Load (Process Only New Files)
Move Processed Files
PGP Decryption
To decrypt source files encrypted with OpenPGP, select an active entry from the PGP Key dropdown. The entry must hold a private key. Integrate.io decrypts each matching file before parsing it.Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.
Source Schema
After defining the source location and format, select the fields to include.
For each field, define:
- Alias: how to refer to the field in downstream components. Illegal characters are flagged before you close the dialog.
- Data type: the type to cast the field to.
- Key (JSON only): the key name in the source JSON.
file_path field from the Meta Data section to capture the source file path as a field in your data.
FAQ
What IAM permissions does Integrate.io need for a source-only connection? Attach AmazonS3ReadOnlyAccess to the IAM user, or create a custom policy grantings3:ListBucket on the bucket and s3:GetObject on its objects.
What permissions are needed for a destination pipeline?
The IAM user needs s3:ListBucket, s3:PutObject, s3:GetObject, and s3:DeleteObject. All three object actions are required because Integrate.io finalizes output files with a copy-and-delete operation.
Why does Integrate.io require all three object actions for writes?
Integrate.io writes each output file to a temporary object first, then finalizes it. On S3, finalization is a copy (GetObject) followed by deletion of the temporary object (DeleteObject). Without all three, the finalization step fails.
Can I connect to a bucket in a region that requires Signature Version 4?
Yes. Enter the bucket’s AWS region in the connection form. See the AWS region list for regions that require Sig v4.