> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks destination for ELT & CDC

> Configure Databricks as a destination in Integrate.io ELT & CDC. Set up your Databricks SQL warehouse to receive replicated data from your sources.

## Overview

Integrate.io ELT & CDC replicates your data to Databricks as Delta tables in Unity Catalog. Data is loaded in periodic batches and merged into the target tables based on the primary key, so each table always reflects the latest state of your source.

ELT & CDC connects to Databricks through a SQL warehouse and loads data into the catalog and schema you choose. You provide the warehouse connection details, a catalog and schema, and a personal access token.

## Requirements

* A running Databricks SQL warehouse. You need its **Server Hostname** and **HTTP Path**.
* A Unity Catalog catalog (or `hive_metastore`) and a target schema.
* A personal access token belonging to a user or service principal that holds the Unity Catalog privileges listed below.

For production, use a [service principal](https://docs.databricks.com/aws/en/admin/users-groups/service-principals) token rather than a personal user token so the connection keeps working if the user leaves.

## Setup

### Connection details

In your Databricks workspace, open **SQL Warehouses** and select the warehouse you want to use. On the **Connection details** tab, copy the **Server Hostname** and **HTTP Path** into the Integrate.io destination form.

### Personal access token

Integrate.io authenticates with a personal access token (PAT).

1. In Databricks, click your username (top right) → **Settings** → **Developer**. Next to **Access tokens**, click **Manage** → **Generate new token**.
2. Enter a **Name** and a **Lifetime (days)**. The token stops working when it expires, so pick a long lifetime and rotate before then, or use a service principal token.
3. If a **Scope** selector appears, choose **BI Tools**. If you pick **Other APIs**, select the `sql` scope. No other API scopes are needed.
4. Click **Generate** and copy the token immediately. Databricks shows it only once.

See [Databricks personal access token authentication](https://docs.databricks.com/aws/en/dev-tools/auth/pat) for details.

### Grant Unity Catalog privileges

The token's user or service principal needs privileges on the target catalog and schema to create and load tables. If it is already a workspace or metastore admin, or owns the catalog, you can skip this step. No storage credential or external location is required.

Run the following queries as a metastore admin or the catalog owner. Replace `<principal>` with the token's identity (a user email, service principal application ID, or account group name), and `<catalog>` and `<schema>` with your target catalog and schema.

```sql theme={null}
-- Catalog access (always required):
GRANT USE CATALOG ON CATALOG <catalog> TO `<principal>`;
GRANT CREATE SCHEMA ON CATALOG <catalog> TO `<principal>`;

-- If the schema already exists, also grant on it
-- (skip if Integrate.io creates it; the creator owns the schema):
GRANT USE SCHEMA, CREATE TABLE, MODIFY, SELECT ON SCHEMA <catalog>.<schema> TO `<principal>`;
```

### Destination credentials

Enter the Server Hostname, HTTP Path, personal access token, catalog, and schema in the Integrate.io destination form.

### Databricks IP access list

If your workspace enforces [IP access lists](https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list), add the ELT & CDC [IP addresses](/cdc/ip-list) to the workspace allowlist.

### Required privileges

| Privilege     | Scope   |
| :------------ | :------ |
| USE CATALOG   | Catalog |
| CREATE SCHEMA | Catalog |
| USE SCHEMA    | Schema  |
| CREATE TABLE  | Schema  |
| MODIFY        | Schema  |
| SELECT        | Schema  |

## VARIANT columns

By default, JSON, object, and array columns are stored as `STRING` for broad runtime compatibility. If your SQL warehouse runs Databricks Runtime 15.3 or later, you can opt into the `VARIANT` type for these columns. Contact support to enable this option on your destination.

## Schema evolution

New columns added to source tables are added to the destination Delta table with `ALTER TABLE ADD COLUMNS`. Column type changes are not applied automatically.

## Related

<CardGroup cols={2}>
  <Card title="IP Allowlist" icon="arrow-right" href="/cdc/ip-list" horizontal />

  <Card title="Initial Sync Process" icon="arrow-right" href="/cdc/initial-sync-process" horizontal />
</CardGroup>