ELT & CDC Features and Requirements

Features

• Full/Historical Sync

• Incremental Sync (CDC/Binlog Replication and some SaaS connectors)

• Append Only

• Excluding of Columns

• Renaming of Columns

• Setting of Custom Primary Keys

• Record Transformations (Can be requested from support@integrate.io)

  • RegEx Find and Replace/Masking

  • Replace in Field

  • Toggle Case

• Use of own S3 bucket (Can be requested from support)

CDC/Binlog Replication Sources

MySQL

Version

• >= 5.7

Variants

• AWS RDS

• GCP Cloud SQL

• Azure

• MariaDB

• Self Hosted

Constraints

• Table events with CASCADING DELETES and CASCADING UPDATES (Not supported by MySQL Binlog replication).

• Columns with spaces or starts with digits (Please contact support@integrate.io to override this manually)

• Configuration binlog_row_value_options=PARTIAL_JSON is not supported as it makes the Binlog replication records not full or complete.

PostgreSQL

Versions

• >= 10

Variants

• AWS PostgreSQL

• Azure Single Node PostgreSQL

• Self Hosted

Constraints

• Heroku PostgreSQL (Heroku doesn’t support logical replication)

• GCP PostgreSQL (GCP doesn’t support logical replication)

• PostgreSQL (<10)

• Columns with spaces or starts with digits (Please contact support@integrate.io to override this manually)

• Table with generated column

SQL Server

Note:

When adding new column to a CDC-Enabled table, disabling/enabling CDC is required to capture the change. Customer can also opt to create another capture instance for a table (to a maximum of two capture instances per table). Reference here.

Constraints

•  Computed columns that are included in a capture instance always have a value of NULL.

SaaS Connectors

Features per connector listed here 

Destinations

Redshift

Supported

Snowflake

Supported

Note:

• The maximum allowed VARCHAR/TEXT length of single-byte characters is 16,777,216 and  4,194,304 for 4 byte characters (per snowflake's documentation).

• The maximum length of a VARIANT is 16 MB.

S3

Supported

• Authorization

  • IAM Role and not AWS AccessKey / Secret

• File format

  • Avro

BigQuery

Supported

Note:

• BigQuery doesn’t support deduplication. Instead, the following columns are added to the destination table for future deduplication on the customer’s side.
  
  __offset - Offset column. Keeps track of the offset in incremental order to determine which existing row is the latest.
  __op - Operation column. Tells what kind of operation is performed for a particular row (0 - Created, 1 - Inserted, 2 - Updated, 3 - Deleted)