> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL: Webhook Integrations

> Trigger an Integrate.io ETL package from an incoming webhook, read the payload into variables, and send outbound HTTP requests from a package.

> [!NOTE]
>
> Webhook triggers are not enabled by default. Contact [support@integrate.io](mailto:support@integrate.io) or your Customer Success Manager to have the feature turned on for your account before you follow this guide.

## Overview

A webhook trigger gives a package a public URL. When an external application POSTs a JSON body to that URL, Integrate.io ETL reads the body into package variables and runs the package as a job on the cluster named in the URL.

The payload arrives as variables, not as records. A package still needs a source component to produce at least one record before the rest of the dataflow can run, so a webhook package uses a source component that produces a single record and takes its values from the variables in a Select component downstream.

## Trigger URL

```
https://gateway.xplenty.com/run-job/{account}/{package_id}/{cluster_id}
```

| Segment | Value | Unique per |
| :- | :- | :- |
| `{account}` | Account identifier | Account |
| `{package_id}` | Numeric package ID | Package |
| `{cluster_id}` | Cluster the job runs on | Cluster |

## Prerequisites

* Webhook triggers enabled on your account by the Integrate.io team.
* A running cluster. See [Creating a Cluster](https://www.integrate.io/docs/etl/creating-a-cluster.md).
* An external application that can send outbound webhooks to a URL you supply.
* A connection created for the destination you plan to write to.

## Build order

You cannot define the package variables until an event has arrived, because the variable names come from the payload. Build the package in two passes, in this order.

| # | Step | Where |
| :- | :- | :- |
| 1 | Prepare the cluster | Clusters |
| 2 | Add the source component | Package designer |
| 3 | Generate the URL and register it | Package settings |
| 4 | Send a few sample events | External application |
| 5 | Read the variable names | Job details, Variables tab |
| 6 | Define the package variables | Package designer, Variables |
| 7 | Map the variables to fields | Select component |
| 8 | Configure the destination | Destination component |
| 9 | Re-enable and verify | Job details |

## Building the webhook package

**1. Prepare the cluster**

Use a running cluster rather than an on-demand one. An on-demand cluster starts up for each job, and that startup time is added to every event.

A webhook runs only on the cluster named in its URL. Adding a second cluster does not spread the load of a single webhook. To spread load, create separate packages on separate clusters and point different event types at each one.

**2. Add the source component**

Add one source component that produces a single record. Do not map any fields yet.

| Source component | Recommended | Notes |
| :- | :- | :- |
| [Dummy Source](https://www.integrate.io/docs/etl/using-components-dummy-source.md) | Yes | Produces one record, with no external call and no credentials. |
| [File Storage Source](https://www.integrate.io/docs/etl/using-components-file-storage-source.md) reading a one-record file | Only if already in place | Adds a read from storage on every event. Replace with a Dummy Source when convenient. |
| [REST API Source](https://www.integrate.io/docs/etl/using-components-rest-api-source.md) calling an external endpoint | No | Throughput is limited by the external service's rate limit, and the package fails whenever that service is down. |

**3. Generate the URL and register it**

1. Generate the trigger URL from the package settings. Do not assemble or edit the URL by hand.
2. Register the URL in the external application, subscribed to a single event type.

**4. Send a few sample events**

1. Enable the webhook in the external application.
2. Send two or three real events.
3. Disable the webhook.
4. Complete steps 5 to 8 using the jobs those events created.
5. Re-enable the webhook.

Events that arrive while the package is incomplete are not replayed, so keep the webhook disabled while you build.

**5. Read the variable names**

Open the job created by one of your sample events and go to the **Variables** tab in **Job details**. The list is read-only and is the definitive source for the variable names.

Do not copy names from the external application's sample payload. Integrate.io ETL derives the names from the payload structure, and they will not always match that application's documentation.

### Sample payload

```json expandable theme={null}
{
  "id": "66a26696c7314f7815db6a1f",
  "event_id": "66a26696c7314f7815db6a1e",
  "event_schema": "v1",
  "event_resource": "AppointmentGroup",
  "event_type": "AppointmentGroup.Created",
  "event_timestamp": "2026-08-04T13:30:00.1234567Z",
  "data": {
    "appointment_group_id": "b3c7d8e9-2f1a-4b3c-8d9e-0f1a2b3c4d5e",
    "invoice_id": "a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d",
    "center_id": "99f8e7d6-c5b4-a3f2-1e0d-9c8b7a6f5e4d",
    "guest_id": "5e4d3c2b-1a0f-9e8d-7c6b-5a4f3e2d1c0b",
    "status": "Created",
    "notes": "Prefers window seating if available.",
    "appointments": [
      {
        "id": "4d5e6f7a-8b9c-0d1e-2f3a-4b5c6d7e8f9a",
        "appointment_segment_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
        "service": { "id": "f5e4d3c2...", "name": "Deep Tissue Massage 60m",
                     "display_name": "Signature Deep Tissue Massage" },
        "therapist": { "id": "2b3c4d5e...", "name": "Jane Doe" },
        "start_time": "2026-08-10T14:00:00",
        "start_time_utc": "2026-08-10T19:00:00Z",
        "end_time": "2026-08-10T15:00:00",
        "end_time_utc": "2026-08-10T20:00:00Z",
        "duration_minutes": 60,
        "is_rebooking": false,
        "source": "External-Webstore"
      }
    ]
  }
}
```

### Resulting variables

| Variable | Value |
| :- | :- |
| `id` | `"66a26696c7314f7815db6a1f"` |
| `event_id` | `"66a26696c7314f7815db6a1e"` |
| `event_schema` | `"v1"` |
| `event_resource` | `"AppointmentGroup"` |
| `event_type` | `"AppointmentGroup.Created"` |
| `event_timestamp` | `"2026-08-04T13:30:00.1234567Z"` |
| `data_appointment_group_id` | `"b3c7d8e9-2f1a-4b3c-8d9e-0f1a2b3c4d5e"` |
| `data_invoice_id` | `"a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d"` |
| `data_center_id` | `"99f8e7d6-c5b4-a3f2-1e0d-9c8b7a6f5e4d"` |
| `data_guest_id` | `"5e4d3c2b-1a0f-9e8d-7c6b-5a4f3e2d1c0b"` |
| `data_status` | `"Created"` |
| `data_notes` | `"Prefers window seating if available."` |
| `data_appointments_json` | `"[{\"id\":\"4d5e6f7a-8b9c-...\"}]"` |
| `data_appointments_0_id` | `"4d5e6f7a-8b9c-0d1e-2f3a-4b5c6d7e8f9a"` |

The list is truncated here. A production payload commonly produces 100 or more variables.

### How the payload maps to variable names

Nested keys are joined with underscores. Arrays produce two kinds of variable: one holding the whole array as a JSON string, and one per element position.

| Key in the payload | Variable | Use it |
| :- | :- | :- |
| `event_type` | `event_type` | Yes |
| `data.guest_id` | `data_guest_id` | Yes |
| `data.appointments` | `data_appointments_json` | Yes, this holds the whole array |
| `data.appointments[0].id` | `data_appointments_0_id` | No, the position is fixed into the name |
| `data.appointments[0].service.name` | No variable is created | Extract it with an expression instead |

**6. Define the package variables**

In the package designer, open **Variables** and select the **Package** tab. Add one variable for each field you plan to use, copying each name exactly from step 5. Set each value to two single quotes (`''`), which acts as an empty default until the payload supplies the real value.

| Tab | What it holds |
| :- | :- |
| **Package** | Variables you define for this package. Use this tab for the payload variables. |
| **Secrets** | Sensitive string values, excluded from logs and exports. |
| **Global**, **Global Secrets** | Account-level values shared across packages. |
| **System** | Values set by Integrate.io ETL when the job runs. |

For the payload above, define these:

```
event_id                    ''
event_type                  ''
event_timestamp             ''
data_appointment_group_id   ''
data_invoice_id             ''
data_guest_id               ''
data_status                 ''
data_appointments_json      ''
```

See [Using and Setting Variables](https://www.integrate.io/docs/etl/using-and-setting-variables-in-your-packages.md) for how variables are evaluated and scoped.

**7. Map the variables to fields**

Add a [Select component](https://www.integrate.io/docs/etl/using-components-select-transformation.md) after the source. Reference each variable with `$` and enclose it in single quotes, because the value is a string.

| Expression | Alias |
| :- | :- |
| `'$event_id'` | `event_id` |
| `'$event_type'` | `event_type` |
| `'$data_appointment_group_id'` | `appointment_group_id` |
| `'$data_invoice_id'` | `invoice_id` |
| `'$data_guest_id'` | `guest_id` |
| `'$data_status'` | `status` |

Convert data types and clean values in the same component.

| Expression | Alias |
| :- | :- |
| `ParseDate(SUBSTRING('$event_timestamp',0,10),'yyyy-MM-dd')` | `event_date` |
| `CASE WHEN IsEmpty('$data_notes') THEN NULL ELSE '$data_notes' END` | `notes` |

**8. Configure the destination**

| Destination | Use it for |
| :- | :- |
| [Database Destination](https://www.integrate.io/docs/etl/using-components-database-destination.md) or a warehouse destination | Keeping a history of events in production. |
| [File Storage Destination](https://www.integrate.io/docs/etl/using-components-file-storage-destination.md) | Handing events to another system. Also the easiest destination to use while building. |
| A CRM or application destination | Writing directly into an operational tool. Set the batch size to around 10. |
| [Email Report](https://www.integrate.io/docs/etl/using-components-email-report.md) | Reviewing output while building. Sends the records as a CSV attachment or inline in the body. |

**9. Re-enable and verify**

1. Send an event and confirm the job succeeds.
2. Confirm every column has a value. An empty column usually means the variable name does not match, so go back to step 5.
3. Send an event with an empty array and one with a missing optional field.
4. Send two or three events in quick succession and check that the jobs complete.

## Processing nested arrays

An array in the payload does not have the same number of elements in every event, so the position-indexed variables (`data_appointments_0_id` and similar) break as soon as the length changes. Use the variable that holds the whole array and expand it in two Select components.

| Variable | Contains | Use it |
| :- | :- | :- |
| `data_appointments_json` | The whole array, as a JSON string | Yes |
| `data_appointments_0_id` | The first element only | No |

### First Select component: flatten the array

[JsonStringToBag](https://www.integrate.io/docs/etl/jsonstringtobag.md) converts the JSON string into a bag, and [Flatten](https://www.integrate.io/docs/etl/flatten.md) turns the bag into one record per element. The fields from the parent object pass through unchanged, so each record keeps the keys that identify the event.

| Expression | Alias |
| :- | :- |
| `'$data_appointment_group_id'` | `appointment_group_id` |
| `'$data_invoice_id'` | `invoice_id` |
| `Flatten(JsonStringToBag('$data_appointments_json'))` | `appointment` |

### Second Select component: extract the fields

Each record now holds one array element in the `appointment` field. Extract the values you need from it.

| Expression | Alias |
| :- | :- |
| `appointment_group_id` | `appointment_group_id` |
| `invoice_id` | `invoice_id` |
| `JsonStringToMap(appointment)#'id'` | `appointment_id` |
| `JsonStringToMap(appointment)#'start_time_utc'` | `start_time_utc` |
| `JsonStringToMap(appointment)#'duration_minutes'` | `duration_minutes` |
| `JsonStringToMap(appointment)#'is_rebooking'` | `is_rebooking` |
| `JsonExtractScalar(appointment, '$.service.name')` | `service_name` |
| `JsonExtractScalar(appointment, '$.service.display_name')` | `service_display_name` |
| `JsonExtractScalar(appointment, '$.therapist.name')` | `therapist_name` |

Two components are required. Extraction only works on a record that already holds a single element, so flattening and extraction cannot share one Select component.

### Choosing the extraction function

| Function | Reads | Example fields |
| :- | :- | :- |
| [JsonStringToMap](https://www.integrate.io/docs/etl/jsonstringtomap.md) with `#'key'` | A value at the top level of the element | `id`, `start_time_utc`, `duration_minutes` |
| [JsonExtractScalar](https://www.integrate.io/docs/etl/jsonextractscalar.md) with a JSONPath | A value inside a nested object, at any depth | `service.name`, `therapist.name` |

For more on maps, bags, and JSONPath, see [Processing JSON Data](https://www.integrate.io/docs/etl/how-do-i-process-json-data.md).

### Splitting the dataflow

The fields from the parent object produce one record per event, while the array produces one record per element. Those cannot be written to the same destination, so use a separate dataflow and a separate destination for each.

```
Dummy Source  -->  Select (parent fields)                                 -->  Database destination
Dummy Source  -->  Select (flatten array)  -->  Select (extract fields)   -->  Database destination
```

* Give each dataflow its own source component.
* The parent dataflow needs one Select component. The array dataflow needs two.
* Carry the same key fields through every dataflow, so the records can be matched later.
* Join the results in the database, not inside the package.
* Prefix component names with the dataflow they belong to.

This example needs two dataflows. Add one more for each additional array in the payload.

## Sending outbound requests

A package can POST to an HTTP endpoint, once per record. The most common use is starting another Integrate.io ETL package through its trigger URL, but the same approach works for any API that accepts a JSON body.

There are three parts: read the data you want to send, build it into a JSON body, then send the request and read the response.

### 1. Read the data you want to send

Read the records you want to send from wherever they live, using a [Database Source](https://www.integrate.io/docs/etl/using-components-database-source.md), a [File Storage Source](https://www.integrate.io/docs/etl/using-components-file-storage-source.md), or the output of an earlier transformation. One request is sent per record, so make sure each record represents exactly one call. Use a [Filter](https://www.integrate.io/docs/etl/using-components-filter-transformation.md) or [Aggregate](https://www.integrate.io/docs/etl/using-components-aggregate-transformation.md) component upstream to control how many requests the job makes.

If the package sends a single fixed payload rather than one per record, use a [Dummy Source](https://www.integrate.io/docs/etl/using-components-dummy-source.md) and build the values from expressions and variables.

### 2. Build the payload

Build the JSON body in a Select component. [TOMAP](https://www.integrate.io/docs/etl/tomap.md) pairs each key with its value, and [ToJson](https://www.integrate.io/docs/etl/tojson.md) converts the result into a JSON string.

| Expression | Alias |
| :- | :- |
| `ToJson(TOMAP('record_id', record_id, 'status', status, 'sent_at', CurrentTime()))` | `json_body` |

Keys are literal strings in single quotes. Values are field names or expressions. Add a key and a value to add another field to the payload.

### 3. Send the request

[Curl](https://www.integrate.io/docs/etl/curl.md) takes the URL, the method, the headers as a JSON string, and the request body. Give the output field a name of its own, such as `api_response`, so it does not clash with the fields you extract from it in the next step.

| Expression | Alias |
| :- | :- |
| `Curl(url, method, headers, json_body)` | `api_response` |

```sql theme={null}
Curl(
  CONCAT('https://gateway.xplenty.com/run-job/',
         '$account', '/', '$package_id', '/', '$cluster_id'),
  'POST',
  '{"Content-Type":"application/json",
    "Accept":"application/vnd.xplenty+json; version=2"}',
  json_body
)
```

The `Accept` header with `version=2` is required when calling the Integrate.io gateway. For any other endpoint, send the headers that API expects.

For calls that need authentication, use [CCurl](https://www.integrate.io/docs/etl/ccurl.md) and pass a connection ID as the last argument. You can find the ID on the connection's page. Do not put a token directly into an expression.

### 4. Read the response

`Curl` returns a map with three keys: `status`, `body`, and `headers`. Extract the ones you need in a second Select component.

| Expression | Alias |
| :- | :- |
| `api_response#'status'` | `api_status_code` |
| `api_response#'body'` | `api_body` |
| `UUID()` | `request_id` |

`request_id` is optional and gives each call a unique value you can match against later.

The request does not wait for the receiving system to finish its work. A 2xx status means the endpoint accepted the request, not that the work behind it succeeded. Write `api_status_code` and `api_body` to a destination if you need a record of what happened.

## Cluster sizing

| Event rate | Recommended setup |
| :- | :- |
| A few per hour | One small cluster shared by all your webhook packages. |
| Tens per hour | A small cluster used only for webhook packages. |
| Hundreds per hour, or in bursts | A dedicated cluster, a Dummy Source, and a narrower event subscription. Consider a scheduled package reading the same data in batches instead. |

## Naming and tracking

* Name the package after the exact event it handles.
* Note which package version is known to work. Webhook packages are edited often.
* Keep a record of which event type points at which package, cluster, and target tables. The link exists only as a URL inside the external application, so nothing in Integrate.io ETL shows it.

## Troubleshooting

### URL and registration

| Symptom | Cause and fix |
| :- | :- |
| Two webhooks appear to share a URL | The cluster ID is the same across your account, so two URLs differ only in the package segment. Generate the URL from each package rather than editing a copy. |
| The gateway returns 404 | One of the three URL segments is wrong. Generate the URL again rather than correcting it by hand. |
| Several event types point at one URL | The variable list changes between events and failures cannot be traced to an event type. Use one package per event type. |
| Nobody knows which package handles an event | The link exists only in the external application's settings. Record it somewhere outside both systems. |

### Variables

| Symptom | Cause and fix |
| :- | :- |
| The package fails validation | A variable is referenced downstream but not defined. Define it with an empty-string default. |
| A destination column is empty and no error appears | The variable name is misspelled. Compare it against the Variables tab in Job details. |
| The same value appears under several variable names | Not all of them are populated. A nested variant can be empty while the top-level one holds the value. Check the value column before choosing a name. |
| A name that looks correct does not resolve | Names are case-sensitive and preserve camelCase. Copy them rather than retyping. |
| Names taken from the external application's docs do not exist | Integrate.io ETL derives names from the payload structure. Use the Variables tab in Job details. |
| `id` and `event_id` are mixed up | They differ by one character. Store whichever one the external application documents as the unique event key. |

### Arrays

| Symptom | Cause and fix |
| :- | :- |
| Array fields are empty or wrong for some events | Position-indexed variables were used, and the array length changed. Use the variable holding the whole array. |
| The extraction expressions return nothing | Flattening and extraction were combined in one Select component. Use two. |
| `JsonStringToMap` returns nothing for a nested value | It only reads the top level of the element. Use `JsonExtractScalar` with a JSONPath. |
| Array elements were defined as individual variables | Define only the variable holding the whole array. |
| Records are lost when the dataflows are joined | Joining inside the package depends on a key being present and unique in every array. Join in the database instead. |

### Destination

| Symptom | Cause and fix |
| :- | :- |
| The target table keeps gaining columns | **Automatically add missing columns** is enabled and the external application changed its payload. Turn it off once the payload shape is settled. |
| Duplicate records for one event | The external application retried delivery. Remove duplicates downstream using the event key. |

### Runtime

| Symptom | Cause and fix |
| :- | :- |
| Events sent during a build are missing | They are not replayed. Disable the webhook in the external application while you build. |
| Jobs queue up behind each other | The source component makes an external call, or the cluster is shared with scheduled jobs. |
| A component preview succeeds but the job fails | A preview and a job run do not follow identical paths. Verify against a real event. |
| The job fails only when a long text field has a value | Line breaks in the value interfere with parsing. Exclude the field or clean it before the destination. |
| An outbound request appears to do nothing | The request does not wait for the receiving system. The endpoint accepted it and the work behind it failed separately. Check `api_status_code` and `api_body`. |

## FAQ

**How do I enable webhook triggers on my account?**

Contact [support@integrate.io](mailto:support@integrate.io) or your Customer Success Manager. Webhook triggers are not self-serve, and the trigger URL cannot be generated until the feature is turned on.

**Can one webhook run on more than one cluster?**

No. A webhook runs only on the cluster named in its URL. To spread load, create separate packages on separate clusters and point different event types at each one.

**Why is a destination column empty when the job succeeded?**

The variable name in the package does not match the name in the payload. Open the **Variables** tab in **Job details**, copy the name exactly, and check the capitalization.

**Should I use `data_appointments_0_id` or `data_appointments_json`?**

Use `data_appointments_json`. Position-indexed variables stop matching as soon as the array length changes between events.

**What is the difference between a webhook trigger and a service hook?**

A webhook trigger is incoming: an external application POSTs to Integrate.io ETL to start a job. A [service hook](https://www.integrate.io/docs/etl/service-hooks.md) is outgoing: Integrate.io ETL POSTs to a URL you own when something happens on the platform, such as a job failing or a cluster idling.

**Are events replayed if the package is not working?**

No. Events that arrive while the package is broken are lost. Disable the webhook in the external application while you build, and re-enable it once you have verified the package.

## Related

> **[Dummy Source](https://www.integrate.io/docs/etl/using-components-dummy-source.md)**

> **[Using and Setting Variables](https://www.integrate.io/docs/etl/using-and-setting-variables-in-your-packages.md)**

> **[Processing JSON Data](https://www.integrate.io/docs/etl/how-do-i-process-json-data.md)**

> **[Select Transformation](https://www.integrate.io/docs/etl/using-components-select-transformation.md)**

> **[Database Destination](https://www.integrate.io/docs/etl/using-components-database-destination.md)**

> **[Service Hooks](https://www.integrate.io/docs/etl/service-hooks.md)**

> **[Creating a Cluster](https://www.integrate.io/docs/etl/creating-a-cluster.md)**

> **[Curl](https://www.integrate.io/docs/etl/curl.md)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.