RELEASE NOTES

What’s New On Integrate.io

Jan 2024

January 2024

ETL & Reverse ETL

Releases & Updates

  • Rails version upgrade

  • File Mover functionality

    • Task component in Workflow

    • Full documentation available here

  • Queued Job status

    • Users can now see if the job is actually running or if it’s waiting for cluster resources. Some notes below:

      • Queued status - Job is waiting for cluster resources and hasn’t started running yet

      • Running status - Job is actually running

      • Runtime is now the actual runtime of the job though not retroactive (jobs before the deployment still show queued time + run time)

      • There are now three timestamps for a Completed job. Created, Started, and Completed

  • Job Transition from Queued to Stopped State

    • Addressed an issue where stopping jobs directly from a queued state was not a valid transition. The fix ensures that the job state transition logic behaves as expected

  • Note component is movable using both white and yellow areas, also it updates its height and width after text change

  • Note components are ignored in auto-aligning functionality

Connectors

  • Salesforce 

    • Bulk API 2.0 - line endings updates

  • File Storage

    • XML file loader on source component

      • “End of File” needs to be selected as record delimiter

      • “Base Record XML Key” must be specified

    • No longer need to flatten and parse XML data as BAG, we can now call XPath to parse each attribute of the record

    • Full documentation available here

  • Webgains

    • New pagination support added

ELT & CDC

Releases & Updates

  • Process pipeline events in background to avoid unnecessary response wait time

  • Show detailed error message on test connection

Connectors

  • Snowflake

    • Temporarily remove validation on table due to casing issues

    • Fix issue when adding multiple columns

    • Add validation to check if table exists before adding columns

  • Redshift

    • Truncate the object string values for specific source connectors

  • MySQL

    • Use unnamed timezone instead of named in MySQL connection validation

  • Salesforce

    • Add automatic backoff on long-running fetches for BULK API

    • Update default timeout for BULK job completion

    • Fixes on continuous sync state

    • Fixes on edit source / add re-authenticate button on the dashboard

    • Show excluded columns in multiple rows

  • SQL Server

    • Add proper error handling during the validation

  • Airtable

    • Upgrade from API key authentication to OAuth2 (due to API key deprecation by the end of January 2024)

Dec 2023

December 2023

ETL & Reverse ETL

Releases & Updates

  • New User Management module for more granular access levels (documentation here)

  • Validation for archiving packages

    • Added validation to prevent archival of a package if associated with schedule(s)

  • Added pending jobs count when calculating number of jobs on a cluster for scheduler load balancing strategy to avoid unbalanced jobs distribution

Connectors

  • Google Ads

    • Ads v14 updates

    • Ads v15 release

  • Salesforce 

    • Bulk API 2.0 - output error updates

    • Bulk API 2.0 - null values updates

ELT & CDC

Connectors

  • SQL Server

    • Backpressure mechanism for low memory consumption

    • Support string chunking

    • Fixes on continuous sync

    • Fixes for admin console queries

    • Support custom primary key tables

  • Snowflake connector

    • Add validation for future ownership grants on schema

    • Improve error logging when resync is needed

  • Google Search Console

    • YAML definition fixes/allow URI encoding on dependencies

  • MySQL

    • Allow syncing of custom PK tables with chunking

    • Fix initial sync row count

  • PostgreSQL

    • Fix initial sync row count

Nov 2023

November 2023

ETL & Reverse ETL

Releases & Updates

  • Workspaces Dashboard

    • Optimized how we compute the packages’ metadata on the Workspaces list dashboard to reduce load time for accounts with a large number of Workspaces

  • Salesforce Bulk API 2.0 - Error output CSV

  • Schedule Modal

    • Fixed an issue where modals are sometimes being closed if there are multiple users navigating on the schedule modals

  • Enabled Workspaces feature by default for all customers

Connectors

  • Salesforce Bulk API 2.0

    • Error output CSV update

  • Google Ads

    • Updated to v14

  • GA4

    • Variable name updates

ELT & CDC

Releases & Updates

  • Support for PrivateLink added

  • Set Parallel Sync as default for pipelines using Snowflake as a destination

  • Add a check condition for CDC rows schema selection page to remove the error in the logs

  • Update the default nullable setting for the columns in converters

  • Add toggle to reset binlog state when resyncing tables through the dashboard

  • Modify notifications to show if the state has been reset when performing a resync action

Connectors

  • SQLServer

    • Fix for tables without a non-integer primary key

    • Fixes for the connector to support parallel sync on large databases

  • Postgres 

    • Fix for tables without a non-integer primary key

  • Salesforce V2 fixes

    • Refactor Streaming API sync

  • Mandrill Connector

    • Set URLs table as unsupported since the endpoint was deprecated

    • Fix issues for the subaccounts table

  • Salesforce Connectors

    • Fix the backpressure mechanism to avoid duplicate records

  • Secure Tunnel (VPN Tunnel)

    • Add advanced options to set custom SSH user & custom SSH port

Oct 2023

October 2023

ETL & Reverse ETL

Releases & Updates

  • PostgreSQL database upgrade

  • Angular update to dashboard

  • Update to coreDNS and Pod disruption budgets for improved cluster reliability

  • Added 1 hour idle time to sandbox clusters

  • Component Previewer EU intermediate storage upgrade

  • Update to host change plugin application

Connectors

  • REST API Connector

    • Added PUT and PATCH Http method support

  • BigQuery

    • Functionality for deleting BigQuery staging tables

  • Salesforce Bulk API 2.0

    • Relationships fields issue updates

ELT & CDC

Releases & Updates

  • New dashboard UI release

  • Responsive Web Design - application is now usable on mobile phones

Connectors

  • Snowflake

    • Update the AWS role storage locations to support list of values

    • Change AWS S3 role creation flow for Snowflake

    • Delete events updates

    • Updates on queries for column names

  • GA4 Export

    • Report option updates

  • Postgres

    • Improve connection error messages

  • Google Ads

    • Upgrade version to v14

    • Automatic multi-account sync

  • Azure Postgres

    • Setup guide - Single Server & Flexible Server

  • SQL Server

    • Support for timestamp and row version data types

    • Datetimeoffset data type support

  • Salesforce connector

    • Sync data from REST/Bulk/Streaming API through same connector

    • Deprecate previous Salesforce connectors (Salesforce CDC, Salesforce REST/Bulk)

Sep 2023

ETL & Reverse ETL

Releases & Updates

  • Mailer alerts for consumption-based pricing model customers

  • SSL cert migration to Heroku Automated Certificate Management

  • PostgreSQL upgrade testing, maintenance window on 15th of October (Sunday) from 18:30 to 19:30 Tokyo time

Connectors

  • Salesforce Bulk API 2.0 for destination 

    • To support more streamlined and optimized workflow. (Updated KB here). In this version, we:

      • Disable “Batch size” option as it’s now automatically handled by Salesforce.

      • Outputting failed records is not supported yet.

ELT & CDC

Releases & Updates

  • Monthly Mailer

    • Updated case issue where job wasn’t triggering as expected

  • EU region production support for high-volume traffic

  • Turn off GTID by default on new pipelines

  • Change threshold value to the percentage value on the billing notification email

Connectors

  • Snowflake

    • Changed some of the error messages from ‘info’ to ‘error’ level

    • Added the trimming of the values to the max length value allowed by Snowflake during the creation of the columns

    • Updated the max length on SQL Server being interpreted incorrectly

    • Snowflake error logging improvements

  • All Connectors

    • User can now reauthenticate sources in the edit view

  • GA4 Export connector

    • Allows customer to pull raw GA4 data from BigQuery after setting up GA4 BigQuery Export

  • Netsuite Connector

    • Added error logging

    • Increase timeout on requests

    • Remove the InventoryDetail table

  • Google Ads Connector

    • Fix data mismatch issue

  • MySQL

    • JSON datatype is replicated to Redshift as VARCHAR and will be truncated if it exceeds VARCHAR max length

  • SQL Server

    • Update pooling query formatting

Aug 2023

ETL & Reverse ETL

Releases & Updates

  • ETL and IIO Dashboard redesign release

  • Software patch update for Jetty

  • Error log summary explanation feature now enabled for all accounts

  • Temporary staging table updates to avoid race condition

  • Connections/Packages/schedules etc lists are now full-width on the screen filling all the space

Connectors

  • Salesforce

    • Package validation on source

    • Destination - removed case sensitivity on Auto-Fill button. Will now auto-fill fields as long as the incoming fields alias is the same and not case sensitive.

    • Bulk API 2.0 for source - to support more streamlined and optimized workflow. In this version, we:

      • Introduce “Max records” - specify number of records per pages to prevent timeout

      • Disable “PK Chunking” option

  • MongoDB

    • Merge operation _id updates

  • Snowflake

    • Destination - load empty data as null option

  • HubSpot

    • Destination’s new system variable: _HUBSPOT_API_REQUEST_MAX_RETRIES

  • Paginations

    • Beauhurst

    • CitySpark

  • Google Analytics (GA4)

    • Added documentation on the Connection creation modal

ELT & CDC

Releases & Updates

  • Pipeline Engine Update

  • Released version auto update

  • Billing Mailer - new opt-in feature, which allows customers to get notified when they surpass certain percentages of their monthly volume limit.

  • Bulk selection for tables

    • You can type comma separate table names in the input instead of clicking checkboxes for each item to select given tables.

Connectors

  • MySQL 

    • Removed RELOAD from required permissions

    • Upgraded library for large data volume timeouts

    • Add backoff retry support on retryable errors

    • Added new logging metrics for MySQL and S3 write duration

Jul 2023

ETL & Reverse ETL

Releases & Updates

  • Improved Job Error Debugging With Error Log Summaries

    • Receive AI-generated summary of job error log to pinpoint reason for failure

  • Job load balancing option on scheduler

    • Ability to reuse cluster with the least number of jobs running

  • Google Service Account Json Key validation fixes

    • Allowing universe_domain in JSON key

  • Auto-map workspace when importing package JSON

    • Automatically assigning the package to a workspace based on the name

  • DB Source Component

    • Updated where the values on Table mode isn’t being removed when switching to Query mode which causes jobs to fail

  • Create Major Version

    • Updated where it removes the last few changes when creating a major version

  • Create Connection Forms

    • Added a modal to refer to our documentation for setup instructions

  • Heroku Addon V3

  • Workspace auto-mapping on package import JSON

  • Mitigations from pentest in re-test

  • SOC 2 tests and evidence collection

  • Password limit on Rest API component connection (up to 128 characters in length)

Connectors

  • Microsoft Excel 

  • Facebook Marketing API update to v17

  • Netsuite JDBC upgrades to 8.10.136.0

  • Paginations

    • Survio

ELT & CDC

Releases & Updates

  • Validation server certificate update

  • Mitigations from pentest in re-test

  • SOC 2 tests and evidence collection

  • Github connector validation fixes

  • Parsing the latest position when a GTID set contains multiple positions.

  • Adds tinyint and smallint as datatypes for number like primary keys.

  • Version fetching for MariaDB > 11

Connectors

  • GA360

  • Snowflake connector

    • Fixed permission validation bug when database, warehouse or schema name are the same

  • Shopify connector

    • Auto retry on SSLv3 connection error

  • MySQL connector 

    • GTID support fixes and configurations net_read_timeout , net_write_timeout and wait_timeout are no longer mandatory for test connection

    • Show only warning if non-mandatory configurations are not set to recommended values for test connection

Integrate.io Dashboard

  • Updated password policy

    • Enforced new password policy on password update / signup (at least 1 lowercase, 1 uppercase,  1 number, and min. 12 characters)

Jun 2023

ETL & Reverse ETL

Releases & Updates

  • SSO

    • Updated cases for handling camel-cased emails

  • Updated Workspaces view with pagination support

  • AWS marketplace signup and billing updates

  • RSS feed updates

  • API key retrieval without password update

  • Package variables saving updates

Connectors

  • HubSpot Destination

    • Custom objects and properties supports

  • Google Spanner Data Boost

  • Netsuite 

    • SOAP validation updates

  • Salesforce

    • Multi-reference column support

  • Paginations

    • LeadFeeder

    • TST Vandale

ELT & CDC

Releases & Updates

  • Add PIPELINE_ERRORED event tracking

  • New metric gauge which calculates the binlog lag based on the latest record received

  • Deployed SSH Tunnel feature

  • Updated session expiration to 12 hours

  • Add new pipeline events to logging feature

  • OAuth token leakage fix

  • Add the collection name to the error object to identify the collection to remove from sync temporarily

  • Release of hooks feature for notifications

  • Add a validation run before the force-start pipeline action on staff view

  • Make SSH tunnel as default and move it to first option

  • Add security response headers

Connectors

  • MariaDB

    • 10.6 support

    • Validation support for BINLOG MONITOR and REPLICA MONITOR grants for 10.5.8 and above

    • Convert array and bbject to string

  • Salesforce 

    • String PK chunking support

  • GA4

May 2023

ETL & Reverse ETL

Releases & Updates

  • Global variables on account level

    • We now support global variables creation at account level. It can be modified from Account Settings -> Global Variables

  • Package version on jobs list 

    • We now show the particular version of the package that ran on the jobs list (and clicking on it views the particular version of the package read-only)

  • Global variables validation function 

    • We are now validating the value or expression of global variables when saving. Please note that creating a global variable using values from other global variables as reference is currently not supported.

  • Released schema-importer OOM fixes

  • EKS upgrades

  • API

    • Security updates for bug bounty reports

Connectors

  • BigQuery 

    • Complex datatype support - load MAP datatype from source to BQ (as BQ’s JSON datatype)

  • GA4 

    • Error logging with Property ID

    • Introduced backoff retry mechanism with maximum attempts system variable (_GA4_API_REQUEST_MAX_ATTEMPTS default: 3). This is to address issue when customer exceed their quota limits.

    • Rate limit workaround

  • Google Ads

    • Stability improvement. Customers might need to increase _ADWORDS_API_REQUEST_READ_TIMEOUT in order to adhere with these changes (especially customers with high number of properties/records).

  • Salesforce

    • Error output - empty file fixes

    • Column with multiple references - visible through source component’s schema and destination’s column mapping.

  • Bing Ads 

    • Authentication updates

  • Facebook

    • Add fields (breakdown properties)

ELT & CDC

Releases & Updates

  • Intermittent connection errors will now be auto retried without failing (ECONNRESET, Connection timed out, Timeout acquiring a connection)

  • Fixed issue on invite when email has capital letters

  • Create new pipeline event RATE_LIMIT_ERROR for connectors

  • Migrations, models and constants for SSH tunnel

Connectors

  • Salesforce CDC

    • Fetch source schema in batches, set query batch size to 200, update column fetching, implement renew credentials on backend

  • Salesforce REST & BULK 

    • Add KnowledgeArticle to tables not supported by BULK API

    • Show username when first creating the connector, renew and save credentials on the backend

  • Snowflake 

    • Updated setup on dashboard - separated small scripts to ease customer source setup

    • Test connection now shows the encountered error (previously only generic message)

  • Google Sheets

    • OAuth request only needed scopes by each connector (instead of requesting all previous granted scopes)

  • Github 

    • Rate-limit issues fix

Apr 2023

ETL & Reverse ETL

Releases & Updates

  • Dynamic connection on Execute SQL Task component on Workflow. This feature is behind our existing feature flag “Dynamic connection”

  • Import JSON API - updated variables when creating new packages with dataflow JSON

  • Sandbox clusters idle time update

  • Documentation search improvements

Connectors

  • Google Cloud Storage 

    • V2 connection

  • Facebook Ads 

    • Update to v16

  • GA4 

    • Property name parsing updates

  • Google Ads 

    • Missing fields fixes on Product Partition Report: campaign.shopping_setting.merchant_id, ad_group_criterion.listing_group.case_value.product_item_id.value

  • File Storage destination

    • Dynamic destination directory creation

  • File Storage Source 

    • Disabled re-ordering of fields which caused data integrity issues

  • Salesforce Source

    • Disabled re-ordering of fields on Query mode which could cause data integrity issues

  • Paginations

    • Inkdesk

ELT & CDC

Releases & Updates

  • Increase timeout when fetching dynamic schema connector catalog

  • Sentry tracing updates

  • Added ToggleCase and ReplaceInField transforms in Staff API

  • Salesforce - 

  • Secure Tunnel Creation - prevent the page from reloading when downloading tunnel template

  • Confirmation modal for Archive and Deactivate

  • Create subscription proration behavior setting support

Connectors

  • HTTP Connector

    • Change JobId from random number to UUID

  • Salesforce Marketing Cloud 

    • Incremental sync updates

    • Allow choosing between REST and Bulk

  • Salesforce

    • Move tables LoginAsEvent, ApiEvents and ListViewEvent to forced Full Sync list. 

    • Move table AccountUserTerritory2View to unsupported table list.

  • SQL Server 

    • Custom certificate updates

  • Github 

    • Action parallel build run optimization

  • Snowflake 

    • Role permission updates

  • Github Connector

    • Implement new tables: release_asset, branches, commit_files, commit_parents, commit_pull_request, commit_users_emails, deployments, deployment_statuses, issue_assignees, issue_labels, repository_teams, repository_topics, repositories, workflows, workflow_runs, workflow_pull_requests, collaborator_details

    • Add option to handle nested bookmarks (both on incremental sync and resync)

  • Redshift 

    • Fallback to string on Geometry types

Mar 2023

ETL & Reverse ETL

Releases & Updates

  • New last successful job timestamp added:

    • _WORKFLOW_PACKAGE_LAST_SUCCESSFUL_JOB_SUBMISSION_TIMESTAMP

Connectors

  • Salesforce

    • Hard Delete operation - allowing records to bypass the recycle bin and immediately become available for deletion

    • Error output file updates

  • HubSpot

    • Support for Reverse ETL destination added

    • Support for Rest API source. Supports using HubSpot connection for authentication in Rest API source component.

  • Bing Ads

    • SDK upgrade to 13.0.15.1

    • Added new fields

  • GA4

    • Added new dimensions

    • Support empty rows

  • Google Ads

    • Update to AdsV13

    • New fields on Campaign report

  • Snowflake

    • New regions added:

      • North America

        • Snowflake GCP - US East (N. Virginia)

        • Snowflake Azure - South Central US (Texas)

      • Europe

        • Snowflake AWS - EU West (Paris)

        • Snowflake Azure - UK South (London)

      • Asia Pacific

        • Snowflake AWS - Asia Pacific (Osaka)

        • Snowflake Azure - Central India (Pune)

      • Middle East

        • Snowflake Azure - UAE North (Dubai)

      • South America

        • Snowflake AWS - South America (Sao Paulo)

  • Paginations

    • Klaviyo (a.klaviyo.com) (March)

ELT & CDC

Releases & Updates

  • Released pipeline event optimization

  • Google sign-in updates

  • BigQuery support on Data Observability

  • Secure tunnel updates

    • Existing tunnels shows all tunnels (inactive ones included) but active ones are on top of the list

    • Test tunnel connection button

    • Link to documentation on secure tunnels

    • Download bash script from tunnel creation

  • Select date range for pipeline usage dynamically in the pipeline details page based on the pipeline creation date

  • Update observability graph

Connectors

  • Salesforce

    • Added type casting support for avro-based destinations

    • Support more retry error messages

  • Github

    • Added rate-limit error message handling

  • BigQuery destination

    • Parse years to forcefully have 4 digits.

  • Google Ads 

    • Update API version to v13

  • MariaDB

    • Fix GTID issue

  • NetSuite

    • Update collections on yaml definition

  • Redshift

    • Additional types

  • httpTap 

    • nodeType support

  • Salesforce Marketing optimizations

    • Support catalog S3 fetch and update (For dynamic connectors, reuse of catalog files)

Feb 2023

ETL & Reverse ETL

Releases & Updates

  • Processing Engine

    • Fix NullPointerException during staging table deletion

Connectors

  • GA4

    • Update for accounts fetching

  • Paginations

    • Google Drive

ELT & CDC

Releases & Updates

  • Source Tokens Autofill when creating from “Sources” tab.

  • Logging feature (Phase 1)

    • Show pipeline events on pipeline details page (up to 1000 events)

    • Highlight errored events

    • Hide full errors on errored pipeline stopped events

Connectors

  • Salesforce

    • Change Salesforce types to align with other snowflake converters

  • Adobe Analytics

    • Add source attributes

  • SQLServer

    • Create mappings for data types.

  • Salesforce CDC

    • Redshift schema and avro converter update

    • Redshift add support for scale and precision

    • Salesforce update for column type changed error

    • Add support for scale and precision for floating value types

  • Google Ads

    • Updated catalog replication method

  • Facebook Ads

    • Update catalog.json for Full Sync tables

    • Update bufferTime to 7 days

    • Upgrade to v14.0 (singer-io repository)

    • Add sync_frequency recommendations

  • MySQL 

    • UTF16 support

  • ​​Google Search

    • Add bufferTime / custom PK feature support

  • Youtube Analytics

    • Add bufferTime / custom PK feature support

    • Updated yaml to fetch video details

  • TikTok Ads

    • Fix data mismatch issue

    • Add bufferTime / custom PK feature support

  • Shopify

    • Upgrade to v2022-07

    • Add rate-limit handling

    • Fix referenced schemas

  • LinkedIn Ads

    • Add sync_frequency recommendations

Jan 2023

ETL & Reverse ETL

Releases & Updates

  • Frontend framework migration

  • Heroku add-on SSO updates

  • SAML SSO update for an issue during the callback phase of authentication

  • System upgrade (Ruby version)

Connectors

  • Tiktok Ads

    • Launch as Reverse ETL destination

  • BigQuery

    • Staging table deletion on job failure

  • Salesforce 

    • Added destination parallelism (both Bulk and SOAP API). 

    • Adding a new field: Thread count to apply parallelism during loading data to the destination and reduce the job runtime. 

      • This is very helpful for customers running a job with high number of records in small batch on production cluster with high number of nodes.

    • Fix discrepancy on error file output

  • LinkedIn creation/reconnect update

  • Paginations

    • Younium

ELT & CDC

Releases & Updates

  • Auth2AuthProvider - Automatic renew credentials function

  • Data Observability

    • Updated alert sync frequency update

    • Updated alert validation issues

    • Added date range filter

    • Updated date format from MON-YY to MON-YYYY (Pipeline and Observability)

  • Clamp precision and scale to 38 and 37 in Redshift

Connectors

  • BigQuery

    • Add support for null insert on array columns

  • Shopify 

    • Metafields collection data mismatch

  • Sendgrid 

    • Updated YAML definitions 

The Evolved Stack
for Tomorrow's Leaders

The no-code pipeline platform for
your entire data journey

Ensure Data Quality