> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# ETL: Databricks Destination
> Configure the Databricks destination to load ETL pipeline output into Delta tables using S3 staging and the COPY INTO SQL command.
## Overview
The Databricks destination writes pipeline output into Delta tables in a Databricks catalog and schema. Integrate.io ETL stages the data as CSV files on Amazon S3, then a footer script runs `COPY INTO` on your Databricks SQL warehouse to load the staged files into the target table.
Use this destination to land ETL job output in Databricks Unity Catalog for analytics, BI, or downstream Databricks workloads.
## Connection setup
To create a Databricks connection you need:
* A running Databricks SQL warehouse. Copy its **Server hostname** and **HTTP path** from the warehouse's **Connection details** tab.
* A personal access token (PAT) for a user or service principal with privileges on the target catalogs and schemas.
For production, use a [service principal](https://docs.databricks.com/aws/en/admin/users-groups/service-principals) token so the connection keeps working if the user who created it leaves. A single connection can reach every catalog the token is entitled to — the target catalog is set on each destination component, not on the connection.
### Required privileges
The token's user or service principal needs, at minimum:
* `USE CATALOG` on each target catalog.
* `USE SCHEMA`, `CREATE TABLE`, `MODIFY`, and `SELECT` on each target schema.
* `CAN USE` on the SQL warehouse.
### To create a Databricks connection in Integrate.io ETL
Click the Connections icon (lightning bolt) on the top left menu.
Click **New connection**.
Select **Databricks**.
Enter a name for the connection.
Enter the **Server hostname** and **HTTP path** from your Databricks SQL warehouse. Integrate.io ETL always connects on port 443 over TLS.
Paste the **personal access token** for the user or service principal Integrate.io ETL will use. Authentication uses PAT (`AuthMech=3`) with `token` as the JDBC user, so no separate username is required.
Click **Test connection**. A success message confirms the credentials are valid.
Click **Create connection**.
## Destination properties
* **Connection** - select an existing Databricks connection or create a new one.
* **Target catalog** - the Unity Catalog catalog to write into. Leave blank to use the SQL warehouse's default catalog.
* **Target schema** - the target table's schema within the selected catalog.
* **Target table** - the name of the target table. If the table does not exist, it is created automatically as a Delta table.
* **Automatically create table if it doesn't exist** - when unchecked, the job fails if the target table is missing.
* **Automatically add missing columns** - when checked, columns present in the pipeline output but not in the target table are added with `ALTER TABLE ... ADD COLUMN`. Key columns cannot be added automatically.
* **Intermediate compression** - compression applied to the staged CSV files on S3 before `COPY INTO` reads them. Choose `none` or `gzip`. Defaults to `none`.
### Operation type
* **Append (Insert only)** - default. Rows are appended to the target table.
* **Overwrite (Truncate and insert)** - truncate the target table before loading.
* **Overwrite (Delete all rows on table and insert)** - delete every row in the target with `DELETE FROM` before loading. Use this when the token cannot truncate the table.
* **Merge with existing data using delete and insert** - stage into a temporary table, delete rows in the target that match on the merge keys, then insert all staged rows. Requires at least one field marked as a key in the schema mapping.
* **Merge with existing data using update and insert** - stage into a temporary table, then run a `MERGE` that updates existing rows on the merge keys and inserts new ones. Requires at least one key column.
### Pre and post action SQL
* **Pre-action SQL** - SQL to run before data is loaded into the target table. For merge operations, runs before the staging table is created.
* **Post-action SQL** - SQL to run after the load completes. For merge operations, runs after the staging table is merged.
## Schema mapping
Map dataflow fields to target table columns. Mark one or more fields as keys when using a merge operation.
When Integrate.io ETL creates the table automatically, dataflow types map to Databricks types as follows:
| **Integrate.io ETL** | **Databricks** |
| :------------------- | :------------- |
| String | STRING |
| Integer | INT |
| Long | BIGINT |
| Float | FLOAT |
| Double | DOUBLE |
| DateTime | TIMESTAMP |
| Boolean | BOOLEAN |
| ByteArray | BINARY |
Auto-created tables use `CREATE TABLE IF NOT EXISTS ... USING DELTA`. Identifiers are quoted with backticks, and columns are nullable unless marked as keys.
## Related