Connection Setup
Get Your Azure Storage Account Credentials
1
Navigate to the Azure Portal and select Storage Accounts from the portal menu.
2
Select the storage account you want to connect.
3
Click All Settings, then click Access Keys.
4
Note the Storage account name and copy one of the Access keys. You will need both when creating the connection in Integrate.io.
Create the Connection in Integrate.io ETL
1
Click the Connections icon (lightning bolt) in the top-left menu.
2
Click New connection.
3
Select Azure Blob Storage.
4
Name the connection and fill in the fields:
5
Click Test Connection to verify the credentials, then click Create Azure Blob Storage Connection.
Source Configuration
Connection
Select your Azure Blob Storage connection from the dropdown, or create a new one using the steps above.Container
Enter the name of the Azure Blob container that holds your source files.Source Path
The path to the input file, folder, or pattern within the container.
You can use wildcard characters for pattern globbing. File and directory names beginning with an underscore (
_) or a dot (.) are ignored.
Source File Format
Record delimiter: what breaks the data into records:- New line (
\n,\n\r,\r): each line in the file is one record. - End of file: each file is treated as a single record.
Avro file format is not supported.
.zip, a Case-insensitive path checkbox appears. Enable it to match the zip file path without regard to letter case.
To read ASC X12 EDI files such as 834 benefit enrollment files, use the dedicated EDI (X12) source instead.
Delimited Values Parameters
Applies when record delimiter is New line and record type is Delimited values.1
Field delimiter: select from the dropdown (comma, tab) or type a single character or escape sequence:
2
String qualifier: if fields are enclosed in quotes, select
' or ". To allow line breaks inside quoted fields, select " (newline inside) or ' (newline inside). Use the newline-inside option with caution, as unbalanced quotes can affect job performance.3
Escape character: available when a string qualifier is set. Specifies the character used to escape qualifier characters within field data.
4
First row contains column names: check to skip the header row in each file.
5
Detect header by name: locates the header row dynamically by matching your schema column names. Use this when the header is not always on row 1 (for example, files with a leading title or blank rows before the header). The full schema header must still appear in column order in each file.
Excel Parameters
Applies when record type is Excel.Password-Protected Workbooks
If a workbook is encrypted with a password (set via Protect Workbook in Excel), check This file is password protected and enter the password in the File Password field. Integrate.io decrypts the workbook during schema preview and job runs. Passwords containing special characters such as$ are supported.
The checkbox appears when the source path ends in .xlsx or .xlsm. Legacy .xls files are not supported for password decryption. Unchecking the box clears any saved password.
Password-protected workbooks do not use streaming mode. When a password is set, enabling Ingest large Excel file reads the workbook in standard mode.When streaming a single sheet, date cells come through as Excel serial numbers (e.g.
45306). Declare those columns as int or decimal in your schema, or transform them downstream.JSON Parameters
Applies when record type is JSON object. Base record JSONPath Expression: defines the base record and extracts nested objects or arrays.
Read more about processing JSON data.
Source Action
Three of the four source actions are available for Azure Blob Storage. Copy, merge and process all files is not supported.
Incremental Load (Process Only New Files)
Move Processed Files
PGP Decryption
To decrypt source files encrypted with OpenPGP, select an active entry from the PGP Key dropdown. The entry must hold a private key. Integrate.io decrypts each matching file before parsing it.Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.
Source Schema
After defining the source location and format, select the fields to include.
For each field, define:
- Alias: how to refer to the field in downstream components. Illegal characters are flagged before you close the dialog.
- Data type: the type to cast the field to.
- Key (JSON only): the key name in the source JSON.
file_path field from the Meta Data section to capture the source file path as a field in your data.