Our ELT & CDC offerings support various schema changes in connectors with CDC replication. Below are the details of how each type of schema change is handled:
Supported Schema Changes
Column Addition
- Automatic Addition: Newly added columns will automatically be added to the replication target.
- MySQL: The column is created once there is a data record change on the table.†
- Postgres: The column is created once there is a data record change on the table.†
- SQL Server: New columns are not automatically added. Restarting the pipeline will reflect the new column in the destination.
- No Backfilling: Newly added columns will not have data from previously created rows. If backfilling is needed, re-syncing the table is recommended to include previous data.
Column Deletion
- Handling in Destination: Deleted columns from the source will not be removed from the destination; new records will have null values for the deleted column.
- Supported Connectors: All database connectors support column deletion.
- Optional Removal: To remove the column from the destination, a table re-sync is required.
Column Renaming
- Handling Renaming: Treated as a combination of column deletion and addition.
- No Backfilling: Similar to column addition, there is no backfilling, and a re-sync is needed to reflect previous rows' data in the new column.
- MySQL: Supports column renaming once there is a data record change on the table.†
- Postgres: Supports column renaming once there is a data record change on the table.†
- SQL Server: Does not support column renaming due to the lack of support for adding new columns.
Table Addition
- Automatic Addition: Newly inserted tables will be automatically added to the ongoing sync.
- Optional Setting: This feature can be toggled from the pipeline settings dashboard and is disabled by default.
- Supported Connectors: Only MySQL supports this feature. Postgres and SQL Server are not yet supported.
Table Deletion
- Handling in Destination: Deleted tables will remain in the destination.
- Supported Connectors: All database connectors support table deletion.
Unsupported Schema Changes
Certain schema changes are not supported directly by our ELT & CDC offerings and will require manual intervention:
Column Data Type Change
- Impact: This change will cause the pipeline to stop.
- Action Required: A re-sync for the affected tables is necessary. Please contact our support team to request a re-sync.
Table Renaming
- Impact: Renamed tables are not automatically recognized.
- Action Required: The renamed table must be selected manually from the dashboard as a new table.
Primary Key Column Change
- Impact: Changes in the primary key column can cause issues with replication and may compromise data integrity at the destination.
- Action Required: We strongly recommend a re-sync of the table with the changed primary key. Please contact our support team to request a re-sync.
† Replication is not instant, and changes will only be applied to the destination once we receive a new record from the table