ETL: Webhook Integrations

Webhook triggers are not enabled by default. Contact support@integrate.io or your Customer Success Manager to have the feature turned on for your account before you follow this guide.

Overview

A webhook trigger gives a package a public URL. When an external application POSTs a JSON body to that URL, Integrate.io ETL reads the body into package variables and runs the package as a job on the cluster named in the URL. The payload arrives as variables, not as records. A package still needs a source component to produce at least one record before the rest of the dataflow can run, so a webhook package uses a source component that produces a single record and takes its values from the variables in a Select component downstream.

Trigger URL

https://gateway.xplenty.com/run-job/{account}/{package_id}/{cluster_id}

Segment	Value	Unique per
`{account}`	Account identifier	Account
`{package_id}`	Numeric package ID	Package
`{cluster_id}`	Cluster the job runs on	Cluster

Prerequisites

Webhook triggers enabled on your account by the Integrate.io team.
A running cluster. See Creating a Cluster.
An external application that can send outbound webhooks to a URL you supply.
A connection created for the destination you plan to write to.

Build order

You cannot define the package variables until an event has arrived, because the variable names come from the payload. Build the package in two passes, in this order.

#	Step	Where
1	Prepare the cluster	Clusters
2	Add the source component	Package designer
3	Generate the URL and register it	Package settings
4	Send a few sample events	External application
5	Read the variable names	Job details, Variables tab
6	Define the package variables	Package designer, Variables
7	Map the variables to fields	Select component
8	Configure the destination	Destination component
9	Re-enable and verify	Job details

Building the webhook package

Prepare the cluster

Use a running cluster rather than an on-demand one. An on-demand cluster starts up for each job, and that startup time is added to every event.A webhook runs only on the cluster named in its URL. Adding a second cluster does not spread the load of a single webhook. To spread load, create separate packages on separate clusters and point different event types at each one.

Add the source component

Add one source component that produces a single record. Do not map any fields yet.

Source component	Recommended	Notes
Dummy Source	Yes	Produces one record, with no external call and no credentials.
File Storage Source reading a one-record file	Only if already in place	Adds a read from storage on every event. Replace with a Dummy Source when convenient.
REST API Source calling an external endpoint	No	Throughput is limited by the external service’s rate limit, and the package fails whenever that service is down.

Generate the URL and register it

Generate the trigger URL from the package settings. Do not assemble or edit the URL by hand.
Register the URL in the external application, subscribed to a single event type.

Send a few sample events

Enable the webhook in the external application.
Send two or three real events.
Disable the webhook.
Complete steps 5 to 8 using the jobs those events created.
Re-enable the webhook.

Events that arrive while the package is incomplete are not replayed, so keep the webhook disabled while you build.

Read the variable names

Open the job created by one of your sample events and go to the Variables tab in Job details. The list is read-only and is the definitive source for the variable names.Do not copy names from the external application’s sample payload. Integrate.io ETL derives the names from the payload structure, and they will not always match that application’s documentation.

Sample payload

{
  "id": "66a26696c7314f7815db6a1f",
  "event_id": "66a26696c7314f7815db6a1e",
  "event_schema": "v1",
  "event_resource": "AppointmentGroup",
  "event_type": "AppointmentGroup.Created",
  "event_timestamp": "2026-08-04T13:30:00.1234567Z",
  "data": {
    "appointment_group_id": "b3c7d8e9-2f1a-4b3c-8d9e-0f1a2b3c4d5e",
    "invoice_id": "a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d",
    "center_id": "99f8e7d6-c5b4-a3f2-1e0d-9c8b7a6f5e4d",
    "guest_id": "5e4d3c2b-1a0f-9e8d-7c6b-5a4f3e2d1c0b",
    "status": "Created",
    "notes": "Prefers window seating if available.",
    "appointments": [
      {
        "id": "4d5e6f7a-8b9c-0d1e-2f3a-4b5c6d7e8f9a",
        "appointment_segment_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
        "service": { "id": "f5e4d3c2...", "name": "Deep Tissue Massage 60m",
                     "display_name": "Signature Deep Tissue Massage" },
        "therapist": { "id": "2b3c4d5e...", "name": "Jane Doe" },
        "start_time": "2026-08-10T14:00:00",
        "start_time_utc": "2026-08-10T19:00:00Z",
        "end_time": "2026-08-10T15:00:00",
        "end_time_utc": "2026-08-10T20:00:00Z",
        "duration_minutes": 60,
        "is_rebooking": false,
        "source": "External-Webstore"
      }
    ]
  }
}

Resulting variables

Variable	Value
`id`	`"66a26696c7314f7815db6a1f"`
`event_id`	`"66a26696c7314f7815db6a1e"`
`event_schema`	`"v1"`
`event_resource`	`"AppointmentGroup"`
`event_type`	`"AppointmentGroup.Created"`
`event_timestamp`	`"2026-08-04T13:30:00.1234567Z"`
`data_appointment_group_id`	`"b3c7d8e9-2f1a-4b3c-8d9e-0f1a2b3c4d5e"`
`data_invoice_id`	`"a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d"`
`data_center_id`	`"99f8e7d6-c5b4-a3f2-1e0d-9c8b7a6f5e4d"`
`data_guest_id`	`"5e4d3c2b-1a0f-9e8d-7c6b-5a4f3e2d1c0b"`
`data_status`	`"Created"`
`data_notes`	`"Prefers window seating if available."`
`data_appointments_json`	`"[{\"id\":\"4d5e6f7a-8b9c-...\"}]"`
`data_appointments_0_id`	`"4d5e6f7a-8b9c-0d1e-2f3a-4b5c6d7e8f9a"`

The list is truncated here. A production payload commonly produces 100 or more variables.

How the payload maps to variable names

Nested keys are joined with underscores. Arrays produce two kinds of variable: one holding the whole array as a JSON string, and one per element position.

Key in the payload	Variable	Use it
`event_type`	`event_type`	Yes
`data.guest_id`	`data_guest_id`	Yes
`data.appointments`	`data_appointments_json`	Yes, this holds the whole array
`data.appointments[0].id`	`data_appointments_0_id`	No, the position is fixed into the name
`data.appointments[0].service.name`	No variable is created	Extract it with an expression instead

Define the package variables

In the package designer, open Variables and select the Package tab. Add one variable for each field you plan to use, copying each name exactly from step 5. Set each value to two single quotes (''), which acts as an empty default until the payload supplies the real value.

Tab	What it holds
Package	Variables you define for this package. Use this tab for the payload variables.
Secrets	Sensitive string values, excluded from logs and exports.
Global, Global Secrets	Account-level values shared across packages.
System	Values set by Integrate.io ETL when the job runs.

For the payload above, define these:

event_id                    ''
event_type                  ''
event_timestamp             ''
data_appointment_group_id   ''
data_invoice_id             ''
data_guest_id               ''
data_status                 ''
data_appointments_json      ''

See Using and Setting Variables for how variables are evaluated and scoped.

Map the variables to fields

Add a Select component after the source. Reference each variable with $ and enclose it in single quotes, because the value is a string.

Expression	Alias
`'$event_id'`	`event_id`
`'$event_type'`	`event_type`
`'$data_appointment_group_id'`	`appointment_group_id`
`'$data_invoice_id'`	`invoice_id`
`'$data_guest_id'`	`guest_id`
`'$data_status'`	`status`

Convert data types and clean values in the same component.

Expression	Alias
`ParseDate(SUBSTRING('$event_timestamp',0,10),'yyyy-MM-dd')`	`event_date`
`CASE WHEN IsEmpty('$data_notes') THEN NULL ELSE '$data_notes' END`	`notes`

Configure the destination

Destination	Use it for
Database Destination or a warehouse destination	Keeping a history of events in production.
File Storage Destination	Handing events to another system. Also the easiest destination to use while building.
A CRM or application destination	Writing directly into an operational tool. Set the batch size to around 10.
Email Report	Reviewing output while building. Sends the records as a CSV attachment or inline in the body.

Re-enable and verify

Send an event and confirm the job succeeds.
Confirm every column has a value. An empty column usually means the variable name does not match, so go back to step 5.
Send an event with an empty array and one with a missing optional field.
Send two or three events in quick succession and check that the jobs complete.

Processing nested arrays

An array in the payload does not have the same number of elements in every event, so the position-indexed variables (data_appointments_0_id and similar) break as soon as the length changes. Use the variable that holds the whole array and expand it in two Select components.

Variable	Contains	Use it
`data_appointments_json`	The whole array, as a JSON string	Yes
`data_appointments_0_id`	The first element only	No

First Select component: flatten the array

JsonStringToBag converts the JSON string into a bag, and Flatten turns the bag into one record per element. The fields from the parent object pass through unchanged, so each record keeps the keys that identify the event.

Expression	Alias
`'$data_appointment_group_id'`	`appointment_group_id`
`'$data_invoice_id'`	`invoice_id`
`Flatten(JsonStringToBag('$data_appointments_json'))`	`appointment`

Second Select component: extract the fields

Each record now holds one array element in the appointment field. Extract the values you need from it.

Expression	Alias
`appointment_group_id`	`appointment_group_id`
`invoice_id`	`invoice_id`
`JsonStringToMap(appointment)#'id'`	`appointment_id`
`JsonStringToMap(appointment)#'start_time_utc'`	`start_time_utc`
`JsonStringToMap(appointment)#'duration_minutes'`	`duration_minutes`
`JsonStringToMap(appointment)#'is_rebooking'`	`is_rebooking`
`JsonExtractScalar(appointment, '$.service.name')`	`service_name`
`JsonExtractScalar(appointment, '$.service.display_name')`	`service_display_name`
`JsonExtractScalar(appointment, '$.therapist.name')`	`therapist_name`

Two components are required. Extraction only works on a record that already holds a single element, so flattening and extraction cannot share one Select component.

Choosing the extraction function

Function	Reads	Example fields
JsonStringToMap with `#'key'`	A value at the top level of the element	`id`, `start_time_utc`, `duration_minutes`
JsonExtractScalar with a JSONPath	A value inside a nested object, at any depth	`service.name`, `therapist.name`

For more on maps, bags, and JSONPath, see Processing JSON Data.

Splitting the dataflow

The fields from the parent object produce one record per event, while the array produces one record per element. Those cannot be written to the same destination, so use a separate dataflow and a separate destination for each.

Dummy Source  -->  Select (parent fields)                                 -->  Database destination
Dummy Source  -->  Select (flatten array)  -->  Select (extract fields)   -->  Database destination

Give each dataflow its own source component.
The parent dataflow needs one Select component. The array dataflow needs two.
Carry the same key fields through every dataflow, so the records can be matched later.
Join the results in the database, not inside the package.
Prefix component names with the dataflow they belong to.

This example needs two dataflows. Add one more for each additional array in the payload.

Sending outbound requests

A package can POST to an HTTP endpoint, once per record. The most common use is starting another Integrate.io ETL package through its trigger URL, but the same approach works for any API that accepts a JSON body. There are three parts: read the data you want to send, build it into a JSON body, then send the request and read the response.

1. Read the data you want to send

Read the records you want to send from wherever they live, using a Database Source, a File Storage Source, or the output of an earlier transformation. One request is sent per record, so make sure each record represents exactly one call. Use a Filter or Aggregate component upstream to control how many requests the job makes. If the package sends a single fixed payload rather than one per record, use a Dummy Source and build the values from expressions and variables.

2. Build the payload

Build the JSON body in a Select component. TOMAP pairs each key with its value, and ToJson converts the result into a JSON string.

Expression	Alias
`ToJson(TOMAP('record_id', record_id, 'status', status, 'sent_at', CurrentTime()))`	`json_body`

Keys are literal strings in single quotes. Values are field names or expressions. Add a key and a value to add another field to the payload.

3. Send the request

Curl takes the URL, the method, the headers as a JSON string, and the request body. Give the output field a name of its own, such as api_response, so it does not clash with the fields you extract from it in the next step.

Expression	Alias
`Curl(url, method, headers, json_body)`	`api_response`

Curl(
  CONCAT('https://gateway.xplenty.com/run-job/',
         '$account', '/', '$package_id', '/', '$cluster_id'),
  'POST',
  '{"Content-Type":"application/json",
    "Accept":"application/vnd.xplenty+json; version=2"}',
  json_body
)

The Accept header with version=2 is required when calling the Integrate.io gateway. For any other endpoint, send the headers that API expects. For calls that need authentication, use CCurl and pass a connection ID as the last argument. You can find the ID on the connection’s page. Do not put a token directly into an expression.

4. Read the response

Curl returns a map with three keys: status, body, and headers. Extract the ones you need in a second Select component.

Expression	Alias
`api_response#'status'`	`api_status_code`
`api_response#'body'`	`api_body`
`UUID()`	`request_id`

request_id is optional and gives each call a unique value you can match against later. The request does not wait for the receiving system to finish its work. A 2xx status means the endpoint accepted the request, not that the work behind it succeeded. Write api_status_code and api_body to a destination if you need a record of what happened.

Cluster sizing

Event rate	Recommended setup
A few per hour	One small cluster shared by all your webhook packages.
Tens per hour	A small cluster used only for webhook packages.
Hundreds per hour, or in bursts	A dedicated cluster, a Dummy Source, and a narrower event subscription. Consider a scheduled package reading the same data in batches instead.

Naming and tracking

Name the package after the exact event it handles.
Note which package version is known to work. Webhook packages are edited often.
Keep a record of which event type points at which package, cluster, and target tables. The link exists only as a URL inside the external application, so nothing in Integrate.io ETL shows it.

Troubleshooting

URL and registration

Symptom	Cause and fix
Two webhooks appear to share a URL	The cluster ID is the same across your account, so two URLs differ only in the package segment. Generate the URL from each package rather than editing a copy.
The gateway returns 404	One of the three URL segments is wrong. Generate the URL again rather than correcting it by hand.
Several event types point at one URL	The variable list changes between events and failures cannot be traced to an event type. Use one package per event type.
Nobody knows which package handles an event	The link exists only in the external application’s settings. Record it somewhere outside both systems.

Variables

Symptom	Cause and fix
The package fails validation	A variable is referenced downstream but not defined. Define it with an empty-string default.
A destination column is empty and no error appears	The variable name is misspelled. Compare it against the Variables tab in Job details.
The same value appears under several variable names	Not all of them are populated. A nested variant can be empty while the top-level one holds the value. Check the value column before choosing a name.
A name that looks correct does not resolve	Names are case-sensitive and preserve camelCase. Copy them rather than retyping.
Names taken from the external application’s docs do not exist	Integrate.io ETL derives names from the payload structure. Use the Variables tab in Job details.
`id` and `event_id` are mixed up	They differ by one character. Store whichever one the external application documents as the unique event key.

Arrays

Symptom	Cause and fix
Array fields are empty or wrong for some events	Position-indexed variables were used, and the array length changed. Use the variable holding the whole array.
The extraction expressions return nothing	Flattening and extraction were combined in one Select component. Use two.
`JsonStringToMap` returns nothing for a nested value	It only reads the top level of the element. Use `JsonExtractScalar` with a JSONPath.
Array elements were defined as individual variables	Define only the variable holding the whole array.
Records are lost when the dataflows are joined	Joining inside the package depends on a key being present and unique in every array. Join in the database instead.

Destination

Symptom	Cause and fix
The target table keeps gaining columns	Automatically add missing columns is enabled and the external application changed its payload. Turn it off once the payload shape is settled.
Duplicate records for one event	The external application retried delivery. Remove duplicates downstream using the event key.

Runtime

Symptom	Cause and fix
Events sent during a build are missing	They are not replayed. Disable the webhook in the external application while you build.
Jobs queue up behind each other	The source component makes an external call, or the cluster is shared with scheduled jobs.
A component preview succeeds but the job fails	A preview and a job run do not follow identical paths. Verify against a real event.
The job fails only when a long text field has a value	Line breaks in the value interfere with parsing. Exclude the field or clean it before the destination.
An outbound request appears to do nothing	The request does not wait for the receiving system. The endpoint accepted it and the work behind it failed separately. Check `api_status_code` and `api_body`.

FAQ

How do I enable webhook triggers on my account? Contact support@integrate.io or your Customer Success Manager. Webhook triggers are not self-serve, and the trigger URL cannot be generated until the feature is turned on. Can one webhook run on more than one cluster? No. A webhook runs only on the cluster named in its URL. To spread load, create separate packages on separate clusters and point different event types at each one. Why is a destination column empty when the job succeeded? The variable name in the package does not match the name in the payload. Open the Variables tab in Job details, copy the name exactly, and check the capitalization. Should I use data_appointments_0_id or data_appointments_json? Use data_appointments_json. Position-indexed variables stop matching as soon as the array length changes between events. What is the difference between a webhook trigger and a service hook? A webhook trigger is incoming: an external application POSTs to Integrate.io ETL to start a job. A service hook is outgoing: Integrate.io ETL POSTs to a URL you own when something happens on the platform, such as a job failing or a cluster idling. Are events replayed if the package is not working? No. Events that arrive while the package is broken are lost. Disable the webhook in the external application while you build, and re-enable it once you have verified the package.

​Overview

​Trigger URL

​Prerequisites

​Build order

​Building the webhook package

​Sample payload

​Resulting variables

​How the payload maps to variable names

​Processing nested arrays

​First Select component: flatten the array

​Second Select component: extract the fields

​Choosing the extraction function

​Splitting the dataflow

​Sending outbound requests

​1. Read the data you want to send

​2. Build the payload

​3. Send the request

​4. Read the response

​Cluster sizing

​Naming and tracking

​Troubleshooting

​URL and registration

​Variables

​Arrays

​Destination

​Runtime

​FAQ

​Related

Dummy Source

Using and Setting Variables

Processing JSON Data

Select Transformation

Database Destination

Service Hooks

Creating a Cluster

Curl

Overview

Trigger URL

Prerequisites

Build order

Building the webhook package

Sample payload

Resulting variables

How the payload maps to variable names

Processing nested arrays

First Select component: flatten the array

Second Select component: extract the fields

Choosing the extraction function

Splitting the dataflow

Sending outbound requests

1. Read the data you want to send

2. Build the payload

3. Send the request

4. Read the response

Cluster sizing

Naming and tracking

Troubleshooting

URL and registration

Variables

Arrays

Destination

Runtime

FAQ

Related