Allowing Integrate.io ETL access to data on Amazon S3

To enable Integrate.io ETL access to your S3 data, you need to perform the following steps:

  1. In AWS's IAM (Identity and Access Management) module, create a user account for working with Integrate.io ETL.
  2. In IAM, save the user account's security credentials for later use when defining a connection in Integrate.io ETL.
  3. In IAM, attach a user policy (provided by Integrate.io ETL) to the newly-created user account.
  4. In Integrate.io ETL, create a connection using the IAM security credentials you saved previously.

Each of these steps are detailed below.

Note: To enable Integrate.io ETL to operate with S3:

  • never use an underscore in an S3 bucket name.
  • never end an S3 bucket name with a dash.

To create a user account in IAM

  1. Log into the AWS Management Console.
  2. Click Identity & Access Management (also known as IAM).
    thumbnail image
  3. On the left navigation bar, click Users.
  4. Click Add User.
    thumbnail image
  5. Enter a user name, such as Integrate.io ETL, and make sure the check box Programmatic access is selected. Click Next: Permissions to proceed.
    thumbnail image
  6. Click Create User. A user account with security credentials is created.
    thumbnail image

To save user security credentials

Click Download .csv and save the credentials.csv file for later when you will need to configure Integrate.io ETL to work with S3.


thumbnail image

Click Close to close the window.

Creating new security credentials for a user

Note: Perform this step if in the procedure to save user security credentials above, any of the user account security credential keys contain a slash, or if you cannot find the credentials.csv file you saved.

  1. In the list of users, click the user you created for working with Integrate.io ETL, then scroll down to Security credentials , and then click Make inactive.
    thumbnail image

  2. Click Create access key

      thumbnail image

Continue from Step 1 in to download and save user security credentials above.

To attach an Integrate.io ETL user policy to the IAM user account

  1. In the list of users, click the user you created for working with Integrate.io ETL, then scroll down to Add permissions.
  2. Click Attach existing policies directly.
  3. In the search box, search 's3 ...'.
  4. For read-only buckets, check AmazonS3ReadOnlyAccess

    thumbnail image
  5. For read-write buckets, Click create policy. Choose JSON, then Copy the following policy and replace "your-bucket" with your bucket name:

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::bucket-name"]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": ["arn:aws:s3:::bucket-name/*"]
        }
    ]
}

thumbnail image

Then click Review policy. On the next page, click Create policy.

For more information, refer to AWS IAM documentation on the web

To create an Amazon S3 connection in Integrate.io ETL: 

  1. Click the Connections icon (lightning bolt) on the top left menu.
  2. To create a connection, click New connection. thumbnail image
  3. Choose Amazon S3.
  4. Type a name for the connection, then from the credentials.csv file you previously saved in To save user security credentials above, copy the Access key id and Secret access key into the respective fields. thumbnail image
  5. If the S3 buckets reside in a region that requires AWS Signature v4, fill in the region for the connection. See region list here.
  6. Click Test connection. If the credentials are correct, a message that the connection test was successful appears.
  7. Click Create amazon s3 connection.
  8. The connection is created and appears in the list of connections.
  9. Now you can create a package and test it on your actual data stored in S3.