ETL: Azure Blob Storage Source

The Azure Blob Storage source reads files from an Azure container and feeds them into your pipeline. It supports CSV, TSV, JSON, Excel, Parquet, Raw, and compressed formats. Once your connection is set up, see File Storage Options for the full set of read options (file formats, schema, and more) shared across every file storage provider.

Connection Setup

Choose an Authentication Method

The connection supports two authentication methods:

Access key: an account access key from the Azure Portal. Use this when you manage the storage account.
SAS token: a shared access signature (SAS). Use this when you only have a SAS token, for example when a vendor grants you access to its storage account or to a single container.

Get Your Azure Storage Account Credentials

Navigate to the Azure Portal and select Storage Accounts from the portal menu.

Select the storage account you want to connect.

Click All Settings, then click Access Keys.

Note the Storage account name and copy one of the Access keys. You will need both when creating the connection in Integrate.io.

Get a SAS Token

If someone else manages the storage account, ask them for a SAS token with the permissions below. To generate an account SAS yourself:

In the Azure Portal, open your storage account and select Security + networking > Shared access signature.

Under Allowed services, select Blob.

Under Allowed resource types, select Service, Container, and Object. Service lets Test Connection check the token against the storage account.

Select the permissions your components need:

Use	Required permissions
Sources and data preview	Read, List
Destinations and File Mover	Read, List, Write, Add, Create
Sources that move or delete processed files, File Mover moves, and destinations that replace or delete files	Also Delete

Set an expiry date. Jobs that use the connection fail once the token expires.

Keep Allowed protocols set to HTTPS only. Integrate.io always connects over HTTPS when you use a SAS token.

Click Generate SAS and connection string. Copy the SAS token, the Blob service SAS URL, or the Connection string. You can paste any of these into the connection.

You can also generate an account SAS with the Azure CLI. This example creates a Blob-only token that is valid for 7 days:

az storage account generate-sas \
  --account-name mystorageaccount \
  --services b \
  --resource-types sco \
  --permissions rwdlac \
  --https-only \
  --expiry "$(date -u -d '+7 days' '+%Y-%m-%dT%H:%MZ')" \
  --account-key "<your-access-key>"

A container SAS grants access to one container. To create one, open the storage account, select Containers, click the … menu next to the container, and select Generate SAS. Components that use this connection can only read from and write to that container.

A SAS token is a credential. Do not share it in tickets, chat, or documentation. Integrate.io stores the token encrypted, the same way it stores an access key, and redacts it from job logs.

Create the Connection in Integrate.io ETL

Click the Connections icon (lightning bolt) in the top-left menu.

Click New connection.

New connection button in the Connections menu

Select Azure Blob Storage.

Selecting Azure Blob Storage from the connection type list

Name the connection and fill in the fields:

Field	Description
Storage account name	The storage account name from the Azure Portal. If you paste a SAS URL or connection string, Integrate.io fills this in from the pasted value.
Authentication method	Access key (the default) or SAS token. Existing connections use Access key.
Access key	Shown for Access key authentication. One of the access keys from the Access Keys screen.
SAS token	Shown for SAS token authentication. Paste the SAS token (with or without the leading `?`), the blob SAS URL, or the SAS connection string.
Connect using SSL	Shown for Access key authentication only. SAS token connections always use HTTPS.

The connection has no container field. You set the container in each source and destination component.If you change the authentication method on an existing connection, enter the new access key or SAS token. Integrate.io does not reuse the stored secret from the other method.

Click Test Connection to verify the credentials, then click Create Azure Blob Storage Connection.For SAS tokens, Test Connection checks an account SAS against the storage account. It checks a container SAS against its container when you paste the blob SAS URL. A container SAS token pasted on its own is only checked for format and expiry. Its access is checked the first time you preview a component that uses it.

Source Configuration

Connection

Select your Azure Blob Storage connection from the dropdown, or create a new one using the steps above.

Container

Enter the name of the Azure Blob container that holds your source files.

Source Path

The path to the input file, folder, or pattern within the container.

Example	Description
`sales/2015/01/`	All files in a folder
`sales/2015/01/log.csv`	A single file
`sales/2015/{01,02}/`	A pattern matching multiple folders

You can use wildcard characters for pattern globbing. File and directory names beginning with an underscore (_) or a dot (.) are ignored.

Source File Format

Record delimiter: what breaks the data into records:

New line (\n, \n\r, \r): each line in the file is one record.
End of file: each file is treated as a single record.

Record type: the format of each record:

Type	Description
Delimited values	Fields separated by a delimiter (comma, tab, or custom). See Delimited values parameters.
Excel	An `.xlsx` workbook. See Excel parameters.
JSON object	Each record is a JSON object enclosed in `{}`. See JSON parameters.
Raw	The entire record is read as a single string or binary field.
Parquet	Reads `.parquet` files. Columns with `DATETIME`/`INT96` data type are not supported.

Avro file format is not supported.

Source data can be compressed (zip, gzip, or bzip2) or uncompressed. Zip files can be password-protected; enter the password in the Zip Password field. Source data must be UTF-8 encoded. For other encodings, see processing a different encoding. Case-insensitive path (zip files only): when the source path contains .zip, a Case-insensitive path checkbox appears. Enable it to match the zip file path without regard to letter case.

To read ASC X12 EDI files such as 834 benefit enrollment files, use the dedicated EDI (X12) source instead.

Delimited Values Parameters

Applies when record delimiter is New line and record type is Delimited values.

Field delimiter: select from the dropdown (comma, tab) or type a single character or escape sequence:

Escape	Character
`\b`	Backspace
`\f`	Form feed
`\n`	Newline
`\r`	Carriage return
`\t`	Tab
`\'`	Single quote
`\"`	Double quote
`\\`	Backslash

String qualifier: if fields are enclosed in quotes, select ' or ". To allow line breaks inside quoted fields, select " (newline inside) or ' (newline inside). Use the newline-inside option with caution, as unbalanced quotes can affect job performance.

Escape character: available when a string qualifier is set. Specifies the character used to escape qualifier characters within field data.

First row contains column names: check to skip the header row in each file.

Detect header by name: locates the header row dynamically by matching your schema column names. Use this when the header is not always on row 1 (for example, files with a leading title or blank rows before the header). The full schema header must still appear in column order in each file.

Excel Parameters

Applies when record type is Excel.

Field	Description
Select sheet by	Name: target a sheet by its tab name. Position: target by position in the workbook (1 = first visible sheet, hidden sheets not counted). Leave both unset to combine all sheets (all sheets must share the same schema).
Sheet Name	Shown in Name mode. Enter the exact tab name.
Sheet Number	Shown in Position mode. Use position mode when the sheet name changes between files but its position stays the same.
Ingest large Excel file	Enables streaming mode for large files. Target a single sheet by name or position, or leave both unset to combine all sheets.
First row contains column names	Skip the header row.
Match columns by name	Maps workbook columns to your schema by header name instead of position. Use when column order varies between files.
Detect header by name	Locates the header row dynamically. Combined with Match columns by name, both the row and column order can vary. Glob patterns (e.g. `2025_*.xlsx`) are supported when matching by name.

Password-Protected Workbooks

If a workbook is encrypted with a password (set via Protect Workbook in Excel), check This file is password protected and enter the password in the File Password field. Integrate.io decrypts the workbook during schema preview and job runs. Passwords containing special characters such as $ are supported. The checkbox appears when the source path ends in .xlsx or .xlsm. Legacy .xls files are not supported for password decryption. Unchecking the box clears any saved password.

Password-protected workbooks do not use streaming mode. When a password is set, enabling Ingest large Excel file reads the workbook in standard mode.When streaming a single sheet, date cells come through as Excel serial numbers (e.g. 45306). Declare those columns as int or decimal in your schema, or transform them downstream.

JSON Parameters

Applies when record type is JSON object. Base record JSONPath Expression: defines the base record and extracts nested objects or arrays.

Preset	Behavior
Object	Uses the keys of the root JSON object as input fields.
Array	Uses the keys of JSON objects within an array as input fields.
Custom expression	Example: `$.data[*]/` extracts fields from objects inside an array named `data`.

Source Action

Three of the four source actions are available for Azure Blob Storage. Copy, merge and process all files is not supported.

Determines what happens to source files during and after processing.

Action	Description
Process all files directly from source	Default. Files are read and processed in place. Available for all providers.
Process only new files (Incremental load)	Reads only files that have not been processed before, tracked via a manifest file. Not supported for Excel. See incremental load.
Copy, merge and process all files	Reads all files, merges them into larger files, then processes. Use for sources with many small files. Fails if the input path points to a single file.
Move processed files to another folder	Moves source files to a backup directory after processing.

Incremental Load (Process Only New Files)

Field	Description
Manifest connection	The connection used to store the manifest file. Defaults to the source connection.
Manifest path	Path to the manifest file, e.g. `bucket/path/to/manifest.gz`. Created on first run. Each run compares the current file listing to the manifest and processes only new files. Previous manifests are kept as backups.

Move Processed Files

Field	Description
Destination path	The folder to move processed files to.
Append string to filename	Optionally append a custom string (e.g. `_processed`) to each moved file's name.

PGP Decryption

To decrypt source files encrypted with OpenPGP, select an active entry from the PGP Key dropdown. The entry must hold a private key. Integrate.io decrypts each matching file before parsing it.

Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.

See PGP Keys for how to add and manage keys.

Source Schema

After defining the source location and format, select the fields to include.

Format	Schema behavior
Delimited values	Fields are read by position. Define all fields in order.
JSON	Define only the fields you want to use. Specify the JSON key for each field.
Raw	Single field containing the entire record.

For each field, define:

Alias: how to refer to the field in downstream components. Illegal characters are flagged before you close the dialog.
Data type: the type to cast the field to.
Key (JSON only): the key name in the source JSON.

You can also add a file_path field from the Meta Data section to capture the source file path as a field in your data.

FAQ

Where do I find my Azure storage account name and access key? In the Azure Portal, go to Storage Accounts, select your account, then click All Settings > Access Keys. The account name and keys are both shown on that screen. Can I use a SAS token instead of an access key? Yes. Set Authentication method to SAS token and paste the token, the blob SAS URL, or the connection string. See Get a SAS Token for the required permissions. What happens when my SAS token expires? Jobs that use the connection fail. Integrate.io refuses an expired token when you save the connection, and the error names the expiry time in UTC. Generate a new token and update the connection before the current one expires. Do I need to list my containers in the connection? No. Each source and destination component sets its own container, the same as with an access key. With a container SAS, components can only use the container the token was issued for. Which source actions are available for Azure Blob Storage? Process all files directly, Process only new files (Incremental load), and Move processed files to another folder. Copy, merge and process all files is not available.

Getting Started

Packages & Workflows

Sources

Transformations

Expressions & Functions

Destinations

Scheduling, Jobs & Clusters

Alerts & Monitoring

Log Drains

Security & Access

Account Management

ETL: Azure Blob Storage Source

Connection Setup

Choose an Authentication Method

Get Your Azure Storage Account Credentials

Get a SAS Token

Create the Connection in Integrate.io ETL

Source Configuration

Connection

Container

Source Path

Source File Format

Delimited Values Parameters

Excel Parameters

Password-Protected Workbooks

JSON Parameters

Source Action

Incremental Load (Process Only New Files)

Move Processed Files

PGP Decryption

Source Schema

FAQ

Azure Blob Storage Destination

File Storage Options

Pattern Matching in Source Paths

Reading File Storage Data Incrementally

PGP Keys

​Connection Setup

​Choose an Authentication Method

​Get Your Azure Storage Account Credentials

​Get a SAS Token

​Create the Connection in Integrate.io ETL

​Source Configuration

​Connection

​Container

​Source Path

​Source File Format

​Delimited Values Parameters

​Excel Parameters

​Password-Protected Workbooks

​JSON Parameters

​Source Action

​Incremental Load (Process Only New Files)

​Move Processed Files

​PGP Decryption

​Source Schema

​FAQ

​Related

Azure Blob Storage Destination

File Storage Options

Pattern Matching in Source Paths

Reading File Storage Data Incrementally

PGP Keys

Connection Setup

Choose an Authentication Method

Get Your Azure Storage Account Credentials

Get a SAS Token

Create the Connection in Integrate.io ETL

Source Configuration

Connection

Container

Source Path

Source File Format

Delimited Values Parameters

Excel Parameters

Password-Protected Workbooks

JSON Parameters

Source Action

Incremental Load (Process Only New Files)

Move Processed Files

PGP Decryption

Source Schema

FAQ

Related