Skip to main content
The HDFS source reads files from a Hadoop Distributed File System cluster and feeds them into your pipeline. It supports CSV, TSV, JSON, Excel, Parquet, Raw, and compressed formats. Once your connection is set up, see File Storage Options for the full set of read options (file formats, schema, and more) shared across every file storage provider.

​
Connection Setup

Your HDFS cluster must be accessible from Integrate.io’s platform. If the cluster is behind a firewall, contact Integrate.io support to arrange access.
1
Click the Connections icon (lightning bolt) in the top-left menu.
2
Click New connection.
New connection button in the Connections menu
3
Select Hadoop Distributed File System (HDFS).
Selecting Hadoop Distributed File System from the connection type list
4
Name the connection and fill in the connection details:
5
Click Test connection to verify the credentials.
6
Click Create HDFS connection.
HDFS connection form with hostname and port fields
The connection appears in the list of connections.
HDFS connection created and listed in file storage connections

​
Source Configuration

​
Connection

Select your HDFS connection from the dropdown, or create a new one using the steps above.

​
Source Path

The path to the input file, folder, or pattern on the HDFS cluster. You can use wildcard characters for pattern globbing. File and directory names beginning with an underscore (_) or a dot (.) are ignored.
HDFS has no bucket or container field. The path is relative to the HDFS root.

​
Source File Format

Record delimiter: what breaks the data into records:
  • New line (\n, \n\r, \r): each line in the file is one record.
  • End of file: each file is treated as a single record.
Record type: the format of each record:
Avro file format is not supported.
Source data can be compressed (zip, gzip, or bzip2) or uncompressed. Zip files can be password-protected; enter the password in the Zip Password field. Source data must be UTF-8 encoded. For other encodings, see processing a different encoding. Case-insensitive path (zip files only): when the source path contains .zip, a Case-insensitive path checkbox appears. Enable it to match the zip file path without regard to letter case.
To read ASC X12 EDI files such as 834 benefit enrollment files, use the dedicated EDI (X12) source instead.

​
Delimited Values Parameters

Applies when record delimiter is New line and record type is Delimited values.
1
Field delimiter: select from the dropdown (comma, tab) or type a single character or escape sequence:
2
String qualifier: if fields are enclosed in quotes, select ' or ". To allow line breaks inside quoted fields, select " (newline inside) or ' (newline inside). Use the newline-inside option with caution, as unbalanced quotes can affect job performance.
3
Escape character: available when a string qualifier is set. Specifies the character used to escape qualifier characters within field data.
4
First row contains column names: check to skip the header row in each file.
5
Detect header by name: locates the header row dynamically by matching your schema column names. Use this when the header is not always on row 1 (for example, files with a leading title or blank rows before the header). The full schema header must still appear in column order in each file.

​
Excel Parameters

Applies when record type is Excel.

​
Password-Protected Workbooks

If a workbook is encrypted with a password (set via Protect Workbook in Excel), check This file is password protected and enter the password in the File Password field. Integrate.io decrypts the workbook during schema preview and job runs. Passwords containing special characters such as $ are supported. The checkbox appears when the source path ends in .xlsx or .xlsm. Legacy .xls files are not supported for password decryption. Unchecking the box clears any saved password.
Password-protected workbooks do not use streaming mode. When a password is set, enabling Ingest large Excel file reads the workbook in standard mode.When streaming a single sheet, date cells come through as Excel serial numbers (e.g. 45306). Declare those columns as int or decimal in your schema, or transform them downstream.

​
JSON Parameters

Applies when record type is JSON object. Base record JSONPath Expression: defines the base record and extracts nested objects or arrays. Read more about processing JSON data.

​
Source Action

HDFS reads all files directly from the source path. Incremental load (process only new files), Copy/merge, and Move processed files are not available for HDFS.

​
PGP Decryption

To decrypt source files encrypted with OpenPGP, select an active entry from the PGP Key dropdown. The entry must hold a private key. Integrate.io decrypts each matching file before parsing it.
Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set.
See PGP Keys for how to add and manage keys.

​
Source Schema

After defining the source location and format, select the fields to include. For each field, define:
  • Alias: how to refer to the field in downstream components. Illegal characters are flagged before you close the dialog.
  • Data type: the type to cast the field to.
  • Key (JSON only): the key name in the source JSON.
You can also add a file_path field from the Meta Data section to capture the source file path as a field in your data.

​
FAQ

Does HDFS support Kerberos authentication? No. Kerberos authentication is not currently supported. Use a standard username in the connection form. What is the HttpFS gateway and why is it required? Integrate.io accesses HDFS through the HttpFS REST gateway rather than the native HDFS protocol. The HttpFS hostname must be reachable from Integrate.io’s platform. If your cluster is behind a firewall, contact Integrate.io support. Can I use incremental load with HDFS? No. Only Process all files directly from source is available for HDFS.

HDFS Destination

File Storage Options

Pattern Matching in Source Paths

PGP Keys

Last modified on September 7, 2026