Features
-
Full/Historical Sync
-
Incremental Sync (CDC/Binlog Replication and some SAAS connectors)
-
Append Only
-
Excluding of Columns
-
Renaming of Columns
-
Setting of Custom Primary Keys
-
Record Transformations (Can be requested from support@integrate.io)
-
RegEx Find and Replace/Masking
-
Replace in Field
-
Toggle Case
-
-
Use of own S3 bucket (Can be requested from support)
CDC/Binlog Replication Sources
MySQL
Version
- >= 5.7
Variants
-
AWS RDS
-
GCP Cloud SQL
-
Azure
-
MariaDB
-
Self Hosted
Limitations
-
Table events with
CASCADING DELETES
andCASCADING UPDATES
(Not supported by MySQL Binlog replication). -
Columns with spaces or starts with digits (Please contact support@integrate.io to override this manually)
-
Configuration
binlog_row_value_options=PARTIAL_JSON
is not supported as it makes the Binlog replication records not full or complete.
PostgreSQL
Versions
- >= 10
Variants
-
AWS PostgreSQL
-
Azure Single Node PostgreSQL
-
Self Hosted
Limitations
-
Heroku PostgreSQL (Heroku doesn’t support
logical
replication) -
GCP PostgreSQL (GCP doesn’t support
logical
replication) -
PostgreSQL (<10)
-
Columns with spaces or starts with digits (Please contact support@integrate.io to override this manually)
-
Table with generated column
SQL Server
Note:
When adding new column to a CDC-Enabled table, disabling/enabling CDC is required to capture the change. Customer can also opt to create another capture instance for a table (to a maximum of two capture instances per table). Reference here.
Limitations
- Computed columns that are included in a capture instance always have a value of NULL.
SAAS Connectors
Features per connector listed here
Destinations
Redshift
Supported
Snowflake
Supported
Note:
- The maximum allowed VARCHAR/TEXT length of single-byte characters is 16,777,216 and 4,194,304 for 4 byte characters (per snowflake's documentation).
-
The maximum length of a VARIANT is 16 MB.
S3
Supported
-
Authorization
-
IAM Role and not AWS AccessKey / Secret
-
-
File format
-
Avro
-
BigQuery
Supported
Note:
- BigQuery doesn’t support deduplication. Instead, the following columns are added to the destination table for future deduplication on the customer’s side.
__offset - Offset column. Keeps track of the offset in incremental order to determine which existing row is the latest.
__op - Operation column. Tells what kind of operation is performed for a particular row (0 - Created, 1 - Inserted, 2 - Updated, 3 - Deleted)