Currently available on our MySQL connector. If you would like this functionality for another of our connectors, please contact support@integrate.io.
Overview
The "Auto Sync New Tables" feature is designed to automatically detect and add new tables created under the source's schema to the sync process. This ensures that any new tables added to your database schema are immediately and automatically included in your data pipeline without any manual intervention.
Key Features
- Automatic Detection: New tables created in the source schema are automatically detected.
- Default Column Selection: All columns of the new tables are selected by default for sync.
- Primary Key Requirement: Tables without a primary key will be skipped from the sync process.
How It Works
- Monitoring for New Tables: The connector continuously monitors the source schema for the creation of new tables.
- Eligibility Check: When a new table is detected, the connector checks if the table has a primary key. If it does not, the table is skipped.
- Automatic Addition: Eligible new tables (those with a primary key) are automatically added to the synchronization process.
- Pipeline Restart: To ensure the new table is included in the sync, the pipeline is automatically restarted. This ensures data integrity and consistency across the entire schema.
Configuration
The feature can be configured in the final step of pipeline creation, or in an existing pipeline on the Settings tab of the pipeline dashboard as shown below.
Requirements
[MySQL] RELOAD Privileges: The MySQL sources must have the RELOAD privileges on the schema being monitored. This is necessary to ensure the connector can detect the difference between initial vs continuous records.
Considerations
- Primary Key Requirement: Only tables with a primary key will be added to the sync. Tables without a primary key are skipped.
- Pipeline Restart: The pipeline will restart each time a new table is added. This may temporarily pause data synchronization. Typical restart times for the pipelines range from 2 to 5 minutes.
- Pipeline-Level Feature: This is a pipeline-level feature, not a source-level feature. Each pipeline should have this feature enabled individually, even if they are syncing from the same source.