RELEASE NOTES

What’s New On Integrate.io

Aug 2023

ETL & Reverse ETL

Releases & Updates

  • ETL and IIO Dashboard redesign release

  • Software patch update for Jetty

  • Error log summary explanation feature now enabled for all accounts

  • Temporary staging table updates to avoid race condition

  • Connections/Packages/schedules etc lists are now full-width on the screen filling all the space

Connectors

  • Salesforce

    • Package validation on source

    • Destination - removed case sensitivity on Auto-Fill button. Will now auto-fill fields as long as the incoming fields alias is the same and not case sensitive.

    • Bulk API 2.0 for source - to support more streamlined and optimized workflow. In this version, we:

      • Introduce “Max records” - specify number of records per pages to prevent timeout

      • Disable “PK Chunking” option

  • MongoDB

    • Merge operation _id updates

  • Snowflake

    • Destination - load empty data as null option

  • HubSpot

    • Destination’s new system variable: _HUBSPOT_API_REQUEST_MAX_RETRIES

  • Paginations

    • Beauhurst

    • CitySpark

  • Google Analytics (GA4)

    • Added documentation on the Connection creation modal

ELT & CDC

Releases & Updates

  • Pipeline Engine Update

  • Released version auto update

  • Billing Mailer - new opt-in feature, which allows customers to get notified when they surpass certain percentages of their monthly volume limit.

  • Bulk selection for tables

    • You can type comma separate table names in the input instead of clicking checkboxes for each item to select given tables.

Connectors

  • MySQL 

    • Removed RELOAD from required permissions

    • Upgraded library for large data volume timeouts

    • Add backoff retry support on retryable errors

    • Added new logging metrics for MySQL and S3 write duration

Jul 2023

ETL & Reverse ETL

Releases & Updates

  • Improved Job Error Debugging With Error Log Summaries

    • Receive AI-generated summary of job error log to pinpoint reason for failure

  • Job load balancing option on scheduler

    • Ability to reuse cluster with the least number of jobs running

  • Google Service Account Json Key validation fixes

    • Allowing universe_domain in JSON key

  • Auto-map workspace when importing package JSON

    • Automatically assigning the package to a workspace based on the name

  • DB Source Component

    • Updated where the values on Table mode isn’t being removed when switching to Query mode which causes jobs to fail

  • Create Major Version

    • Updated where it removes the last few changes when creating a major version

  • Create Connection Forms

    • Added a modal to refer to our documentation for setup instructions

  • Heroku Addon V3

  • Workspace auto-mapping on package import JSON

  • Mitigations from pentest in re-test

  • SOC 2 tests and evidence collection

  • Password limit on Rest API component connection (up to 128 characters in length)

Connectors

  • Microsoft Excel 

  • Facebook Marketing API update to v17

  • Netsuite JDBC upgrades to 8.10.136.0

  • Paginations

    • Survio

ELT & CDC

Releases & Updates

  • Validation server certificate update

  • Mitigations from pentest in re-test

  • SOC 2 tests and evidence collection

  • Github connector validation fixes

  • Parsing the latest position when a GTID set contains multiple positions.

  • Adds tinyint and smallint as datatypes for number like primary keys.

  • Version fetching for MariaDB > 11

Connectors

  • GA360

  • Snowflake connector

    • Fixed permission validation bug when database, warehouse or schema name are the same

  • Shopify connector

    • Auto retry on SSLv3 connection error

  • MySQL connector 

    • GTID support fixes and configurations net_read_timeout , net_write_timeout and wait_timeout are no longer mandatory for test connection

    • Show only warning if non-mandatory configurations are not set to recommended values for test connection

Integrate.io Dashboard

  • Updated password policy

    • Enforced new password policy on password update / signup (at least 1 lowercase, 1 uppercase,  1 number, and min. 12 characters)

Jun 2023

ETL & Reverse ETL

Releases & Updates

  • SSO

    • Updated cases for handling camel-cased emails

  • Updated Workspaces view with pagination support

  • AWS marketplace signup and billing updates

  • RSS feed updates

  • API key retrieval without password update

  • Package variables saving updates

Connectors

  • HubSpot Destination

    • Custom objects and properties supports

  • Google Spanner Data Boost

  • Netsuite 

    • SOAP validation updates

  • Salesforce

    • Multi-reference column support

  • Paginations

    • LeadFeeder

    • TST Vandale

ELT & CDC

Releases & Updates

  • Add PIPELINE_ERRORED event tracking

  • New metric gauge which calculates the binlog lag based on the latest record received

  • Deployed SSH Tunnel feature

  • Updated session expiration to 12 hours

  • Add new pipeline events to logging feature

  • OAuth token leakage fix

  • Add the collection name to the error object to identify the collection to remove from sync temporarily

  • Release of hooks feature for notifications

  • Add a validation run before the force-start pipeline action on staff view

  • Make SSH tunnel as default and move it to first option

  • Add security response headers

Connectors

  • MariaDB

    • 10.6 support

    • Validation support for BINLOG MONITOR and REPLICA MONITOR grants for 10.5.8 and above

    • Convert array and bbject to string

  • Salesforce 

    • String PK chunking support

  • GA4

May 2023

ETL & Reverse ETL

Releases & Updates

  • Global variables on account level

    • We now support global variables creation at account level. It can be modified from Account Settings -> Global Variables

  • Package version on jobs list 

    • We now show the particular version of the package that ran on the jobs list (and clicking on it views the particular version of the package read-only)

  • Global variables validation function 

    • We are now validating the value or expression of global variables when saving. Please note that creating a global variable using values from other global variables as reference is currently not supported.

  • Released schema-importer OOM fixes

  • EKS upgrades

  • API

    • Security updates for bug bounty reports

Connectors

  • BigQuery 

    • Complex datatype support - load MAP datatype from source to BQ (as BQ’s JSON datatype)

  • GA4 

    • Error logging with Property ID

    • Introduced backoff retry mechanism with maximum attempts system variable (_GA4_API_REQUEST_MAX_ATTEMPTS default: 3). This is to address issue when customer exceed their quota limits.

    • Rate limit workaround

  • Google Ads

    • Stability improvement. Customers might need to increase _ADWORDS_API_REQUEST_READ_TIMEOUT in order to adhere with these changes (especially customers with high number of properties/records).

  • Salesforce

    • Error output - empty file fixes

    • Column with multiple references - visible through source component’s schema and destination’s column mapping.

  • Bing Ads 

    • Authentication updates

  • Facebook

    • Add fields (breakdown properties)

ELT & CDC

Releases & Updates

  • Intermittent connection errors will now be auto retried without failing (ECONNRESET, Connection timed out, Timeout acquiring a connection)

  • Fixed issue on invite when email has capital letters

  • Create new pipeline event RATE_LIMIT_ERROR for connectors

  • Migrations, models and constants for SSH tunnel

Connectors

  • Salesforce CDC

    • Fetch source schema in batches, set query batch size to 200, update column fetching, implement renew credentials on backend

  • Salesforce REST & BULK 

    • Add KnowledgeArticle to tables not supported by BULK API

    • Show username when first creating the connector, renew and save credentials on the backend

  • Snowflake 

    • Updated setup on dashboard - separated small scripts to ease customer source setup

    • Test connection now shows the encountered error (previously only generic message)

  • Google Sheets

    • OAuth request only needed scopes by each connector (instead of requesting all previous granted scopes)

  • Github 

    • Rate-limit issues fix

Apr 2023

ETL & Reverse ETL

Releases & Updates

  • Dynamic connection on Execute SQL Task component on Workflow. This feature is behind our existing feature flag “Dynamic connection”

  • Import JSON API - updated variables when creating new packages with dataflow JSON

  • Sandbox clusters idle time update

  • Documentation search improvements

Connectors

  • Google Cloud Storage 

    • V2 connection

  • Facebook Ads 

    • Update to v16

  • GA4 

    • Property name parsing updates

  • Google Ads 

    • Missing fields fixes on Product Partition Report: campaign.shopping_setting.merchant_id, ad_group_criterion.listing_group.case_value.product_item_id.value

  • File Storage destination

    • Dynamic destination directory creation

  • File Storage Source 

    • Disabled re-ordering of fields which caused data integrity issues

  • Salesforce Source

    • Disabled re-ordering of fields on Query mode which could cause data integrity issues

  • Paginations

    • Inkdesk

ELT & CDC

Releases & Updates

  • Increase timeout when fetching dynamic schema connector catalog

  • Sentry tracing updates

  • Added ToggleCase and ReplaceInField transforms in Staff API

  • Salesforce - 

  • Secure Tunnel Creation - prevent the page from reloading when downloading tunnel template

  • Confirmation modal for Archive and Deactivate

  • Create subscription proration behavior setting support

Connectors

  • HTTP Connector

    • Change JobId from random number to UUID

  • Salesforce Marketing Cloud 

    • Incremental sync updates

    • Allow choosing between REST and Bulk

  • Salesforce

    • Move tables LoginAsEvent, ApiEvents and ListViewEvent to forced Full Sync list. 

    • Move table AccountUserTerritory2View to unsupported table list.

  • SQL Server 

    • Custom certificate updates

  • Github 

    • Action parallel build run optimization

  • Snowflake 

    • Role permission updates

  • Github Connector

    • Implement new tables: release_asset, branches, commit_files, commit_parents, commit_pull_request, commit_users_emails, deployments, deployment_statuses, issue_assignees, issue_labels, repository_teams, repository_topics, repositories, workflows, workflow_runs, workflow_pull_requests, collaborator_details

    • Add option to handle nested bookmarks (both on incremental sync and resync)

  • Redshift 

    • Fallback to string on Geometry types

Mar 2023

ETL & Reverse ETL

Releases & Updates

  • New last successful job timestamp added:

    • _WORKFLOW_PACKAGE_LAST_SUCCESSFUL_JOB_SUBMISSION_TIMESTAMP

Connectors

  • Salesforce

    • Hard Delete operation - allowing records to bypass the recycle bin and immediately become available for deletion

    • Error output file updates

  • HubSpot

    • Support for Reverse ETL destination added

    • Support for Rest API source. Supports using HubSpot connection for authentication in Rest API source component.

  • Bing Ads

    • SDK upgrade to 13.0.15.1

    • Added new fields

  • GA4

    • Added new dimensions

    • Support empty rows

  • Google Ads

    • Update to AdsV13

    • New fields on Campaign report

  • Snowflake

    • New regions added:

      • North America

        • Snowflake GCP - US East (N. Virginia)

        • Snowflake Azure - South Central US (Texas)

      • Europe

        • Snowflake AWS - EU West (Paris)

        • Snowflake Azure - UK South (London)

      • Asia Pacific

        • Snowflake AWS - Asia Pacific (Osaka)

        • Snowflake Azure - Central India (Pune)

      • Middle East

        • Snowflake Azure - UAE North (Dubai)

      • South America

        • Snowflake AWS - South America (Sao Paulo)

  • Paginations

    • Klaviyo (a.klaviyo.com) (March)

ELT & CDC

Releases & Updates

  • Released pipeline event optimization

  • Google sign-in updates

  • BigQuery support on Data Observability

  • Secure tunnel updates

    • Existing tunnels shows all tunnels (inactive ones included) but active ones are on top of the list

    • Test tunnel connection button

    • Link to documentation on secure tunnels

    • Download bash script from tunnel creation

  • Select date range for pipeline usage dynamically in the pipeline details page based on the pipeline creation date

  • Update observability graph

Connectors

  • Salesforce

    • Added type casting support for avro-based destinations

    • Support more retry error messages

  • Github

    • Added rate-limit error message handling

  • BigQuery destination

    • Parse years to forcefully have 4 digits.

  • Google Ads 

    • Update API version to v13

  • MariaDB

    • Fix GTID issue

  • NetSuite

    • Update collections on yaml definition

  • Redshift

    • Additional types

  • httpTap 

    • nodeType support

  • Salesforce Marketing optimizations

    • Support catalog S3 fetch and update (For dynamic connectors, reuse of catalog files)

Feb 2023

ETL & Reverse ETL

Releases & Updates

  • Processing Engine

    • Fix NullPointerException during staging table deletion

Connectors

  • GA4

    • Update for accounts fetching

  • Paginations

    • Google Drive

ELT & CDC

Releases & Updates

  • Source Tokens Autofill when creating from “Sources” tab.

  • Logging feature (Phase 1)

    • Show pipeline events on pipeline details page (up to 1000 events)

    • Highlight errored events

    • Hide full errors on errored pipeline stopped events

Connectors

  • Salesforce

    • Change Salesforce types to align with other snowflake converters

  • Adobe Analytics

    • Add source attributes

  • SQLServer

    • Create mappings for data types.

  • Salesforce CDC

    • Redshift schema and avro converter update

    • Redshift add support for scale and precision

    • Salesforce update for column type changed error

    • Add support for scale and precision for floating value types

  • Google Ads

    • Updated catalog replication method

  • Facebook Ads

    • Update catalog.json for Full Sync tables

    • Update bufferTime to 7 days

    • Upgrade to v14.0 (singer-io repository)

    • Add sync_frequency recommendations

  • MySQL 

    • UTF16 support

  • ​​Google Search

    • Add bufferTime / custom PK feature support

  • Youtube Analytics

    • Add bufferTime / custom PK feature support

    • Updated yaml to fetch video details

  • TikTok Ads

    • Fix data mismatch issue

    • Add bufferTime / custom PK feature support

  • Shopify

    • Upgrade to v2022-07

    • Add rate-limit handling

    • Fix referenced schemas

  • LinkedIn Ads

    • Add sync_frequency recommendations

Jan 2023

ETL & Reverse ETL

Releases & Updates

  • Frontend framework migration

  • Heroku add-on SSO updates

  • SAML SSO update for an issue during the callback phase of authentication

  • System upgrade (Ruby version)

Connectors

  • Tiktok Ads

    • Launch as Reverse ETL destination

  • BigQuery

    • Staging table deletion on job failure

  • Salesforce 

    • Added destination parallelism (both Bulk and SOAP API). 

    • Adding a new field: Thread count to apply parallelism during loading data to the destination and reduce the job runtime. 

      • This is very helpful for customers running a job with high number of records in small batch on production cluster with high number of nodes.

    • Fix discrepancy on error file output

  • LinkedIn creation/reconnect update

  • Paginations

    • Younium

ELT & CDC

Releases & Updates

  • Auth2AuthProvider - Automatic renew credentials function

  • Data Observability

    • Updated alert sync frequency update

    • Updated alert validation issues

    • Added date range filter

    • Updated date format from MON-YY to MON-YYYY (Pipeline and Observability)

  • Clamp precision and scale to 38 and 37 in Redshift

Connectors

  • BigQuery

    • Add support for null insert on array columns

  • Shopify 

    • Metafields collection data mismatch

  • Sendgrid 

    • Updated YAML definitions 

The Evolved Stack
for Tomorrow's Leaders

The no-code pipeline platform for
your entire data journey

Ensure Data Quality

hello@integrate.io
+1-888-884-6405

Ensure Data Quality Ensure Data Quality Ensure Data Quality Ensure Data Quality Ensure Data Quality

©2023 Integrate.io

Solutions
Support
Company
Language

©2023 Integrate.io