Features
- Full/Historical Sync
- Incremental Sync (CDC/Binlog Replication and some SaaS connectors)
- Append Only
- Excluding of Columns
- Renaming of Columns
- Setting of Custom Primary Keys
- Record Transformations (Can be requested from support@integrate.io)
- RegEx Find and Replace/Masking
- Replace in Field
- Toggle Case
- RegEx Find and Replace/Masking
- Use of own S3 bucket (Can be requested from support)
CDC/Binlog Replication Sources
MySQL
Version- >= 5.7
- AWS RDS
- GCP Cloud SQL
- Azure
- MariaDB
- Self Hosted
- Table events with
CASCADING DELETESandCASCADING UPDATES(Not supported by MySQL Binlog replication). - Configuration
binlog_row_value_options=PARTIAL_JSONis not supported as it makes the Binlog replication records not full or complete.
PostgreSQL
Versions- >= 10
- AWS PostgreSQL
- Azure Single Node PostgreSQL
- Self Hosted
- GCP Cloud Postgres
- Heroku PostgreSQL (Heroku doesn’t support
logicalreplication) - PostgreSQL (<10)
- Table with generated column
TRUNCATE CASCADEandTRUNCATE
SQL Server
Note:When adding new column to a CDC-Enabled table, disabling/enabling CDC is required to capture the change. Customer can also opt to create another capture instance for a table (to a maximum of two capture instances per table). Reference here.
SQL Server System Constraints
- Computed columns that are included in a capture instance always have a value of NULL.
SaaS Connectors
Features per connector listed hereDestinations
Redshift
SupportedSnowflake
SupportedNote:
- The maximum allowed VARCHAR/TEXT length of single-byte characters is 16,777,216 and 4,194,304 for 4 byte characters (per snowflake’s documentation).
- The maximum length of a VARIANT is 16 MB.
S3
Supported- Authorization
- IAM Role and not AWS AccessKey / Secret
- File format
- Avro
BigQuery
SupportedNote:
-
BigQuery doesn’t support deduplication. Instead, the following columns are added to the destination table for future deduplication on the customer’s side.
__offset - Offset column. Keeps track of the offset in incremental order to determine which existing row is the latest.
__op - Operation column. Tells what kind of operation is performed for a particular row (0 - Created, 1 - Inserted, 2 - Updated, 3 - Deleted)