ETL: Python Transformation

The Python Transformation component allows you to run custom Python code as part of your Integrate.io ETL pipeline. It transforms records without changing the schema of the input data.

Overview

Accepts an array of JSON records as input
Requires a defined transform function
Input and output schemas must match exactly
Useful for string manipulation, cleaning, enrichment, or any custom logic
You can remove records from the array (filtering), but returned records must follow the original schema
You may return an empty array, but not a different schema

Example Transformation Code

import json

def transform(event, context):
    """
    Transforms input records by converting all string values to uppercase.
    The output schema must match the input schema exactly.
    """
    try:
        data = event
        results = []

        for record in data:
            transformed_record = {}
            for key, value in record.items():
                if isinstance(value, str):
                    transformed_record[key] = value.upper()
                else:
                    transformed_record[key] = value
            results.append(transformed_record)

        return results

    except Exception as e:
        raise e

Example Input and Output

Input:

[
  { "id": 1, "name": "Belgian Waffles", "price": 5.95 },
  { "id": 2, "name": "Pancakes", "price": 4.95 }
]

Output:

[
  { "id": 1, "name": "BELGIAN WAFFLES", "price": 5.95 },
  { "id": 2, "name": "PANCAKES", "price": 4.95 }
]

Configuration

Batch Size: Define the number of records per batch. The total payload size should not exceed 6 MB.

Testing

Test your transformation in the Package Designer: provide a sample payload in table format, click Run Code, and view the transformed output. The test table mirrors the schema of the connected input component.

Variables

Use package, secret, and global variables with Python import format:

from package_variables import variable_name

All variables (including secrets and globals) are evaluated before passing to the function.

Best Practices

Always ensure output matches input schema (field names and types).
Valid to return fewer records or an empty array for filtering.
Catch and log exceptions to help debug errors.
Break complex logic into helper functions within the same script.

Getting Started

Packages & Workflows

Sources

Transformations

Expressions & Functions

Destinations

Scheduling, Jobs & Clusters

Alerts & Monitoring

Security & Access

Account Management

ETL: Python Transformation

Overview

Example Transformation Code

Example Input and Output

Configuration

Testing

Variables

Best Practices

​Overview

​Example Transformation Code

​Example Input and Output

​Configuration

​Testing

​Variables

​Best Practices

Overview

Example Transformation Code

Example Input and Output

Configuration

Testing

Variables

Best Practices