Skip to main content

​
Overview

The Databricks destination writes pipeline output into Delta tables in a Databricks catalog and schema. Integrate.io ETL stages the data as CSV files on Amazon S3, then a footer script runs COPY INTO on your Databricks SQL warehouse to load the staged files into the target table. Use this destination to land ETL job output in Databricks Unity Catalog for analytics, BI, or downstream Databricks workloads.

​
Connection setup

To create a Databricks connection you need:
  • A running Databricks SQL warehouse. Copy its Server hostname and HTTP path from the warehouse’s Connection details tab.
  • A personal access token (PAT) for a user or service principal with privileges on the target catalogs and schemas.
For production, use a service principal token so the connection keeps working if the user who created it leaves. A single connection can reach every catalog the token is entitled to — the target catalog is set on each destination component, not on the connection.

​
Required privileges

The token’s user or service principal needs, at minimum:
  • USE CATALOG on each target catalog.
  • USE SCHEMA, CREATE TABLE, MODIFY, and SELECT on each target schema.
  • CAN USE on the SQL warehouse.

​
To create a Databricks connection in Integrate.io ETL

1
Click the Connections icon (lightning bolt) on the top left menu.
2
Click New connection.
3
Select Databricks.
4
Enter a name for the connection.
5
Enter the Server hostname and HTTP path from your Databricks SQL warehouse. Integrate.io ETL always connects on port 443 over TLS.
6
Paste the personal access token for the user or service principal Integrate.io ETL will use. Authentication uses PAT (AuthMech=3) with token as the JDBC user, so no separate username is required.
7
Click Test connection. A success message confirms the credentials are valid.
8
Click Create connection.

​
Destination properties

  • Connection - select an existing Databricks connection or create a new one.
  • Target catalog - the Unity Catalog catalog to write into. Leave blank to use the SQL warehouse’s default catalog.
  • Target schema - the target table’s schema within the selected catalog.
  • Target table - the name of the target table. If the table does not exist, it is created automatically as a Delta table.
  • Automatically create table if it doesn’t exist - when unchecked, the job fails if the target table is missing.
  • Automatically add missing columns - when checked, columns present in the pipeline output but not in the target table are added with ALTER TABLE ... ADD COLUMN. Key columns cannot be added automatically.
  • Intermediate compression - compression applied to the staged CSV files on S3 before COPY INTO reads them. Choose none or gzip. Defaults to none.

​
Operation type

  • Append (Insert only) - default. Rows are appended to the target table.
  • Overwrite (Truncate and insert) - truncate the target table before loading.
  • Overwrite (Delete all rows on table and insert) - delete every row in the target with DELETE FROM before loading. Use this when the token cannot truncate the table.
  • Merge with existing data using delete and insert - stage into a temporary table, delete rows in the target that match on the merge keys, then insert all staged rows. Requires at least one field marked as a key in the schema mapping.
  • Merge with existing data using update and insert - stage into a temporary table, then run a MERGE that updates existing rows on the merge keys and inserts new ones. Requires at least one key column.

​
Pre and post action SQL

  • Pre-action SQL - SQL to run before data is loaded into the target table. For merge operations, runs before the staging table is created.
  • Post-action SQL - SQL to run after the load completes. For merge operations, runs after the staging table is merged.

​
Schema mapping

Map dataflow fields to target table columns. Mark one or more fields as keys when using a merge operation. When Integrate.io ETL creates the table automatically, dataflow types map to Databricks types as follows:
Integrate.io ETLDatabricks
StringSTRING
IntegerINT
LongBIGINT
FloatFLOAT
DoubleDOUBLE
DateTimeTIMESTAMP
BooleanBOOLEAN
ByteArrayBINARY
Auto-created tables use CREATE TABLE IF NOT EXISTS ... USING DELTA. Identifiers are quoted with backticks, and columns are nullable unless marked as keys.

Snowflake Destination

Amazon Redshift Destination

Running Jobs

IP Allowlist

Last modified on July 8, 2026