> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL: HDFS Source

> How to connect to HDFS and configure the Hadoop Distributed File System source component to read files in your Integrate.io ETL pipeline.

The HDFS source reads files from a Hadoop Distributed File System cluster and feeds them into your pipeline. It supports CSV, TSV, JSON, Excel, Parquet, Raw, and compressed formats.

Once your connection is set up, see [File Storage Options](/docs/etl/file-storage-options) for the full set of read options (file formats, schema, and more) shared across every file storage provider.

## Connection Setup

Your HDFS cluster must be accessible from Integrate.io's platform. If the cluster is behind a firewall, contact [Integrate.io support](https://www.integrate.io/contact/) to arrange access.

<Steps>
  <Step>
    Click the **Connections** icon (lightning bolt) in the top-left menu.
  </Step>

  <Step>
    Click **New connection**.

    <Frame>
      <img src="https://mintcdn.com/integrateio/V1Ns1A4dfxUcZVN3/images/connectivity-and-security/image-204.webp?fit=max&auto=format&n=V1Ns1A4dfxUcZVN3&q=85&s=141925a33fe000563dfbc263d87cc31a" alt="New connection button in the Connections menu" width="1200" height="828" data-path="images/connectivity-and-security/image-204.webp" />
    </Frame>
  </Step>

  <Step>
    Select **Hadoop Distributed File System (HDFS)**.

    <Frame>
      <img src="https://mintcdn.com/integrateio/V1Ns1A4dfxUcZVN3/images/connectivity-and-security/image-205.webp?fit=max&auto=format&n=V1Ns1A4dfxUcZVN3&q=85&s=0608fd990a570d7c9a9eb6185053561c" alt="Selecting Hadoop Distributed File System from the connection type list" width="1200" height="828" data-path="images/connectivity-and-security/image-205.webp" />
    </Frame>
  </Step>

  <Step>
    Name the connection and fill in the connection details:

    | Field                 | Description                                                                                      |
    | :-------------------- | :----------------------------------------------------------------------------------------------- |
    | **User Name**         | The username to connect with. Kerberos authentication is not currently supported.                |
    | **NameNode Hostname** | The hostname of the NameNode, or the logical name in a high-availability configuration.          |
    | **NameNode Port**     | The TCP port of the NameNode. Leave empty for high-availability configurations.                  |
    | **HttpFS Hostname**   | The hostname of the Hadoop HttpFS gateway node. Must be accessible from Integrate.io's platform. |
    | **HttpFS Port**       | The TCP port of the HttpFS gateway. Default is 14000.                                            |
  </Step>

  <Step>
    Click **Test connection** to verify the credentials.
  </Step>

  <Step>
    Click **Create HDFS connection**.

    <Frame>
      <img src="https://mintcdn.com/integrateio/V1Ns1A4dfxUcZVN3/images/connectivity-and-security/image-206.webp?fit=max&auto=format&n=V1Ns1A4dfxUcZVN3&q=85&s=c64d7737da56fc00ab3c6c6149536393" alt="HDFS connection form with hostname and port fields" width="1200" height="828" data-path="images/connectivity-and-security/image-206.webp" />
    </Frame>

    The connection appears in the list of connections.

    <Frame>
      <img src="https://mintcdn.com/integrateio/V1Ns1A4dfxUcZVN3/images/connectivity-and-security/image-207.webp?fit=max&auto=format&n=V1Ns1A4dfxUcZVN3&q=85&s=5e281a8071353451ad516f545e7ed65b" alt="HDFS connection created and listed in file storage connections" width="1200" height="828" data-path="images/connectivity-and-security/image-207.webp" />
    </Frame>
  </Step>
</Steps>

***

## Source Configuration

### Connection

Select your HDFS connection from the dropdown, or create a new one using the steps above.

### Source Path

The path to the input file, folder, or pattern on the HDFS cluster.

| Example                       | Description                         |
| :---------------------------- | :---------------------------------- |
| `/data/sales/2015/01/log.csv` | A single file                       |
| `/data/sales/2015/01/`        | All files in a folder               |
| `/data/sales/2015/{01,02}/`   | A pattern matching multiple folders |

You can use wildcard characters for [pattern globbing](/docs/etl/using-pattern-matching-in-source-component-paths). File and directory names beginning with an underscore (`_`) or a dot (`.`) are ignored.

<Note>
  HDFS has no bucket or container field. The path is relative to the HDFS root.
</Note>

<Snippet file="snippets/file-storage-source-format.mdx" />

### Source Action

HDFS reads all files directly from the source path. Incremental load (process only new files), Copy/merge, and Move processed files are not available for HDFS.

<Snippet file="snippets/file-storage-source-tail.mdx" />

***

## FAQ

**Does HDFS support Kerberos authentication?**

No. Kerberos authentication is not currently supported. Use a standard username in the connection form.

**What is the HttpFS gateway and why is it required?**

Integrate.io accesses HDFS through the HttpFS REST gateway rather than the native HDFS protocol. The HttpFS hostname must be reachable from Integrate.io's platform. If your cluster is behind a firewall, contact [Integrate.io support](https://www.integrate.io/contact/).

**Can I use incremental load with HDFS?**

No. Only **Process all files directly from source** is available for HDFS.

***

## Related

<CardGroup cols={2}>
  <Card title="HDFS Destination" icon="arrow-right" href="/docs/etl/using-components-hdfs-destination" horizontal />

  <Card title="File Storage Options" icon="arrow-right" href="/docs/etl/file-storage-options" horizontal />

  <Card title="Pattern Matching in Source Paths" icon="arrow-right" href="/docs/etl/using-pattern-matching-in-source-component-paths" horizontal />

  <Card title="PGP Keys" icon="arrow-right" href="/docs/etl/pgp-keys-account-level" horizontal />
</CardGroup>