> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# ETL: File Storage Options
> Common source and destination configuration options for file and object storage components in Integrate.io ETL.
This page is a reference for the configuration options that are shared across all file and object storage source and destination components: Amazon S3, Google Cloud Storage, Azure Blob Storage, SFTP, FTPS, and HDFS.
Provider-specific details (connection setup, which bucket or container label appears, which source actions are available for that provider) are documented on each provider's own page. Google Drive source is also documented separately because its options differ significantly from the other providers.
***
## Source Options
### Source Location
**Bucket / Container** (S3, GCS, Azure Blob only)
The name of the cloud storage bucket or container that holds the source files. The label varies by provider:
| Provider | Label |
| :------------------- | :-------- |
| Amazon S3 | Bucket |
| Google Cloud Storage | Bucket |
| Azure Blob Storage | Container |
| SFTP, FTPS, HDFS | Not shown |
**Source path**
The path to the input folder, file, or pattern within the bucket or container (for object stores) or the server (for SFTP, FTPS, HDFS).
| Example | Description |
| :---------------------- | :------------------------------------------------------------------- |
| `sales/2015/01/` | All files in a folder (not supported for GCS; use a pattern instead) |
| `sales/2015/01/log.csv` | A single file |
| `sales/2015/{01,02}/` | A glob pattern matching multiple folders |
You can use wildcard characters for [pattern globbing](/docs/etl/using-pattern-matching-in-source-component-paths). File and directory names that begin with an underscore (`_`) or a dot (`.`) are ignored.
For SFTP and FTPS, the path must begin with `/`. Integrate.io prepends it automatically if omitted.
**Case-insensitive path** (zip files only)
When the source path contains `.zip`, a **Case-insensitive path** checkbox appears. Enable it to match the zip file path without regard to letter case.
***
### Source File Format
**Record delimiter**: what breaks the data into records:
* **New line** (`\n`, `\n\r`, `\r`): each line in the file is one record.
* **End of file**: each file is treated as a single record.
**Record type**: the format of each record:
| Type | Description |
| :--------------- | :------------------------------------------------------------------------------------------------------------------------ |
| Delimited values | Fields separated by a delimiter (comma, tab, or custom). See [Delimited values parameters](#delimited-values-parameters). |
| Excel | An `.xlsx` workbook. See [Excel parameters](#excel-parameters). |
| JSON object | Each record is a JSON object enclosed in `{}`. See [JSON parameters](#json-parameters). |
| Raw | The entire record is read as a single string or binary field. |
| Parquet | Reads `.parquet` files. Columns with `DATETIME`/`INT96` data type are not supported. |
Avro file format is not supported.
Source data can be compressed (zip, gzip, or bzip2) or uncompressed. Zip files can be password-protected; enter the password in the **Zip Password** field. Source data must be UTF-8 encoded. For other encodings, see [processing a different encoding](/docs/etl/how-do-i-process-a-different-encoding).
To read ASC X12 EDI files such as 834 benefit enrollment files, use the dedicated [EDI (X12) source](/docs/etl/using-components-edi-x12-source) instead.
***
### Delimited Values Parameters
Applies when record delimiter is **New line** and record type is **Delimited values**.
**Field delimiter**: select from the dropdown (comma, tab) or type a single character or escape sequence:
| Escape | Character |
| :----- | :-------------- |
| `\b` | Backspace |
| `\f` | Form feed |
| `\n` | Newline |
| `\r` | Carriage return |
| `\t` | Tab |
| `\'` | Single quote |
| `\"` | Double quote |
| `\\` | Backslash |
**String qualifier**: if fields are enclosed in quotes, select `'` or `"`. To allow line breaks inside quoted fields, select **" (newline inside)** or **' (newline inside)**. Use the newline-inside option with caution, as unbalanced quotes can affect job performance.
**Escape character**: available when a string qualifier is set. Specifies the character used to escape qualifier characters within field data.
**First row contains column names**: check to skip the header row in each file.
**Detect header by name**: locates the header row dynamically by matching your schema column names. Use this when the header is not always on row 1 (for example, files with a leading title or blank rows before the header). The full schema header must still appear in column order in each file.
***
### Excel Parameters
Applies when record type is **Excel**.
| Field | Description |
| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Select sheet by** | **Name**: target a sheet by its tab name. **Position**: target by position in the workbook (1 = first visible sheet, hidden sheets not counted). Leave both unset to combine all sheets (all sheets must share the same schema). |
| **Sheet Name** | Shown in Name mode. Enter the exact tab name. |
| **Sheet Number** | Shown in Position mode. Use position mode when the sheet name changes between files but its position stays the same. |
| **Ingest large Excel file** | Enables streaming mode for large files. Target a single sheet by name or position, or leave both unset to combine all sheets. |
| **First row contains column names** | Skip the header row. |
| **Match columns by name** | Maps workbook columns to your schema by header name instead of position. Use when column order varies between files. |
| **Detect header by name** | Locates the header row dynamically. Combined with **Match columns by name**, both the row and column order can vary. Glob patterns (e.g. `2025_*.xlsx`) are supported when matching by name. |
#### Password-Protected Workbooks
If a workbook is encrypted with a password (set via **Protect Workbook** in Excel), check **This file is password protected** and enter the password in the **File Password** field. Integrate.io decrypts the workbook during schema preview and job runs. Passwords containing special characters such as `$` are supported.
The checkbox appears when the source path ends in `.xlsx` or `.xlsm`. Legacy `.xls` files are not supported for password decryption. Unchecking the box clears any saved password.
Password-protected workbooks do not use streaming mode. When a password is set, enabling **Ingest large Excel file** reads the workbook in standard mode.
When streaming a single sheet, date cells come through as Excel serial numbers (e.g. `45306`). Declare those columns as `int` or `decimal` in your schema, or transform them downstream.
***
### JSON Parameters
Applies when record type is **JSON object**.
**Base record JSONPath Expression**: defines the base record and extracts nested objects or arrays.
| Preset | Behavior |
| :---------------- | :------------------------------------------------------------------------------- |
| Object | Uses the keys of the root JSON object as input fields. |
| Array | Uses the keys of JSON objects within an array as input fields. |
| Custom expression | Example: `$.data[*]/` extracts fields from objects inside an array named `data`. |
Read more about [processing JSON data](/docs/etl/how-do-i-process-json-data).
***
### Source Action
Determines what happens to source files during and after processing.
| Action | Description |
| :-------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Process all files directly from source** | Default. Files are read and processed in place. |
| **Process only new files (Incremental load)** | Reads only files that have not been processed before, tracked via a manifest file. Not supported for Excel. See [incremental load](/docs/etl/how-do-i-read-data-from-file-storage-incrementally). |
| **Copy, merge and process all files** | Reads all files, merges them into larger files, then processes. Use for sources with many small files. Fails if the input path points to a single file. Not available for all providers. Check your provider's page. |
| **Move processed files to another folder** | Moves source files to a backup directory after processing. |
Available source actions vary by provider. Not all actions are available for every storage type. Refer to your provider's page for details.
#### Incremental Load (Process Only New Files)
| Field | Description |
| :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Manifest connection** | The connection used to store the manifest file. Defaults to the source connection. |
| **Manifest path** | Path to the manifest file, e.g. `bucket/path/to/manifest.gz`. Created on first run. Each run compares the current file listing to the manifest and processes only new files. Previous manifests are kept as backups. |
#### Move Processed Files
| Field | Description |
| :---------------------------- | :------------------------------------------------------------------------------- |
| **Destination path** | The folder to move processed files to. |
| **Append string to filename** | Optionally append a custom string (e.g. `_processed`) to each moved file's name. |
***
### PGP Decryption
To decrypt source files encrypted with OpenPGP, select an active entry from the **PGP Key** dropdown. The entry must hold a private key. Integrate.io decrypts each matching file before parsing it.
Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.
See [PGP Keys](/docs/etl/pgp-keys-account-level) for how to add and manage keys.
***
### Source Schema
After defining the source location and format, select the fields to include.
| Format | Schema behavior |
| :--------------- | :--------------------------------------------------------------------------- |
| Delimited values | Fields are read by position. Define all fields in order. |
| JSON | Define only the fields you want to use. Specify the JSON key for each field. |
| Raw | Single field containing the entire record. |
For each field, define:
* **Alias**: how to refer to the field in downstream components. Illegal characters are flagged before you close the dialog.
* **Data type**: the type to cast the field to.
* **Key** (JSON only): the key name in the source JSON.
You can also add a `file_path` field from the **Meta Data** section to capture the source file path as a field in your data.
***
## Destination Options
### Target Location
**Bucket / Container** (S3, GCS, Azure Blob only)
The name of the target bucket or container. The label varies by provider:
| Provider | Label |
| :------------------- | :-------- |
| Amazon S3 | Bucket |
| Google Cloud Storage | Bucket |
| Azure Blob Storage | Container |
| SFTP, FTPS, HDFS | Not shown |
**Target directory**: the directory within the bucket or container where output files are written. One or more files are created in this directory. By default, the job fails if the directory already exists (see [Destination Action](#destination-action)).
***
### Destination Format
| Format | Description |
| :---------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Delimited values** | CSV, TSV, or any delimiter-separated format. Output files are UTF-8 encoded. |
| **Line delimited JSON** | One JSON object per line. UTF-8 encoded. |
| **Parquet** | Columnar format compatible with Athena, Impala, Presto, and other tools. `datetime` and complex types are not supported. Compression is not available for Parquet. |
| **Fixed width** | Records written as fixed-width lines with no separators. Requires configuring the [Fixed Width Layout](#fixed-width-layout). |
***
### Delimited Values Options
| Field | Description |
| :------------------------------ | :----------------------------------------------------------------------------------------------------- |
| **Delimiter** | Character separating fields. Select from the dropdown or type a single character. Default is tab. |
| **String qualifier** | If selected (`'` or `"`), fields containing the delimiter are enclosed in quotes. |
| **Escape character** | Available when a string qualifier is set. Character used to escape qualifier characters in field data. |
| **Line ending** | **Unix/Linux (LF)** or **Windows (CRLF)**. |
| **Write field names in header** | Adds a header row with field names to each output file. |
**Header names with spaces**: use triple underscores (`___`) in the alias or column name in a Select component to produce a space in the output header. To use a different separator character, add a package variable `_FS_REPLACE_WITH_SPACE_IDENTIFIER` with the character you want to replace.
***
### Output Compression
Available for Delimited values, JSON, and Fixed width formats. Not available for Parquet.
| Option | File suffix added |
| :----- | :--------------------------------------- |
| None | (none) |
| Gzip | `.gz` appended to output directory name |
| Bzip2 | `.bz2` appended to output directory name |
***
### PGP Encryption
To encrypt output files with a trading partner's OpenPGP public key, select an active entry from the **PGP Key** dropdown. The entry must hold a public key. Files are encrypted before being written to the target path.
Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.
See [PGP Keys](/docs/etl/pgp-keys-account-level) for how to add and manage keys.
***
### Destination Action
Controls what happens when the target directory already exists.
| Action | Description |
| :------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Fail if target directory exists** | Default. The job fails if the destination directory already exists. Use variables to generate unique directory names per run (e.g. `/output/${_JOB_ID}`). |
| **Delete target directory if exists** | Deletes the existing directory before writing. |
| **Replace files in directory** | Writes to the existing directory. Files with the same names are overwritten. To add files without overwriting, use a unique file prefix (e.g. include `${_JOB_ID}`). **Warning:** the target directory is deleted on job failure. |
| **Use intermediate storage, copy to existing directory** | Writes to intermediate storage first, then copies to the existing target directory. The target directory must already exist. Supports [character encoding](#character-encoding) selection. |
| **Use intermediate storage, copy to new directory** | Writes to intermediate storage first, then copies to a new target directory. Fails if the directory already exists. |
***
### Merge Output to Single File
Check **Merge output to single file** to write all output to a single file rather than multiple part files. Note that file size limits on some platforms may cause the job to fail for large outputs.
***
### Character Encoding
Available only when **Destination action** is set to **Use intermediate storage, copy to existing directory**.
| Option | Description |
| :-------- | :---------- |
| UTF-8 | Default |
| UTF-16 | |
| Shift-JIS | |
| CP932 | |
***
### Target File Names
**Default pattern**: output files are named `part-[mr]-[0-9]{5}` (e.g. `part-m-00000`).
To use a custom pattern, select **Custom pattern**:
| Field | Description |
| :------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **File name prefix** | Leave empty to keep the default prefix (`part-[mr]-`), or enter a custom prefix. Supports [package variables](/docs/etl/using-and-setting-variables-in-your-packages) for dynamic names (e.g. a timestamp). |
| **File name suffix** | Integrate.io suggests a suffix based on the selected format and compression type. |
***
### Fixed Width Layout
When **Destination format** is set to **Fixed width**, a **Fixed width layout** step appears. The layout is a table with one row per incoming field, in schema order. Every incoming field is written with no separators between columns.
To write only some fields or in a different order, add a [Select transformation](/docs/etl/using-components-select-transformation) before this destination.
| Column | Description |
| :-------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Field** | Incoming field name. Read-only. |
| **Width** | Number of characters this field occupies. Whole number, 1-1000. |
| **Align** | **Left** or **Right**. Controls which side padding is added to. |
| **Pad** | Single character used to fill unused width. Defaults to a space. |
| **If too long** | **Fail the job** (stops on first overflow; always applies to numeric fields) or **Truncate** (text fields only). |
| **Decimals** | Implied decimal places for numeric fields. `0` writes the value as-is. With `2`, the value `12.34` is written as `1234`. Values are rounded HALF\_UP. Whole number, 0-1000. |
| **Sign** | Where to place the sign for numeric fields: **None**, **Leading** (start of field), or **Trailing** (end of field). |
The total record width in characters is shown below the table and updates as you edit.
#### Preview with Data
Click **Preview with data** to run the component through the previewer and see the actual records the job would write. Rows whose sample value does not fit the configured width are marked with an advisory message. Blocking errors (invalid width, pad, or decimals) appear in the errors box and prevent saving.
#### Persisted Layout Format
The layout is stored as `fixed_width_destination_type` on the component. Example:
```json theme={null}
{
"fields": [
{
"name": "id",
"width": 8,
"align": "right",
"pad": "0",
"on_overflow": "fail",
"implied_decimals": 0,
"sign": "none"
},
{
"name": "amount",
"width": 12,
"align": "right",
"pad": " ",
"on_overflow": "fail",
"implied_decimals": 2,
"sign": "leading"
}
],
"record_delimiter": "\n",
"char_encoding": "UTF-8"
}
```
`record_delimiter` and `char_encoding` round-trip through the API but are not editable in the UI.
***
### Skip Empty Output
Available for Delimited values, JSON, and Fixed width formats. When checked, no output files are written if the job produces zero records. Useful for downstream processes that should not receive empty files.
***
## FAQ
**Which source actions are available for my storage provider?**
It depends on the provider. Amazon S3, GCS, and SFTP/FTPS support all four actions. Azure Blob Storage supports three. HDFS supports only "process all files directly." Check your provider's page for the exact list.
**Can I read compressed files?**
Yes. Source files can be zip, gzip, or bzip2 compressed. Password-protected zip files are also supported; enter the password in the **Zip Password** field.
**Can I read files in a different character encoding?**
Source data must be UTF-8 encoded. For other encodings, see [processing a different encoding](/docs/etl/how-do-i-process-a-different-encoding).
**How does incremental load work?**
Integrate.io tracks which files have been processed in a manifest file stored at a path you specify. Each run compares the current file listing to the manifest and reads only new files. See [reading file storage data incrementally](/docs/etl/how-do-i-read-data-from-file-storage-incrementally).
**Can I write to a single output file instead of multiple part files?**
Yes. Enable **Merge output to single file** in the destination component. Be aware that very large outputs may exceed file size limits on some platforms.
**How do I include spaces in output header names?**
Use triple underscores (`___`) in the alias in a Select component. They become spaces in the output header. To use a different replacement character, set the package variable `_FS_REPLACE_WITH_SPACE_IDENTIFIER`.
***
## Related