Overview
By default, when a table fails to load into the destination, the pipeline stops and restarts. Tables that were loading without problems have to wait until the failing table recovers. The Keep loading healthy tables feature isolates load failures to the affected table. When a table fails to load, the pipeline skips that table for the current sync and the remaining tables keep loading. The pipeline retries the skipped table automatically on the next sync. No manual reset is required. This feature is available for pipelines with a Snowflake destination. It is off by default.When to use it
Enable this feature when a single problem table should not delay data delivery for the rest of the pipeline. Common cases include:- A table’s staging object was dropped or a grant was revoked in Snowflake, so only that table fails to load.
- A pipeline syncs many tables and you want the healthy ones to keep loading while you fix the failing one.
How it works
1
A table fails to load
During a sync, one table’s load into Snowflake fails. The failure is recorded for that table only.
2
Healthy tables keep loading
The remaining tables in the pipeline continue loading normally. The pipeline stays running.
3
The failed table is retried
On the next sync, the pipeline retries the failed table from the point where it stopped. No data is skipped. Once the load succeeds, the table is fully caught up.
Configuration
1
Open your pipeline and go to the Settings tab.
2
Under Sync Options, enable the toggle Destination: Keep loading healthy tables even if others fail.
3
Save the pipeline settings.
Monitoring failed loads
To get notified when a table fails to load, attach a hook to the pipeline and select the Destination Load Failed event. The notification lists the destination tables that failed to load. This event requires the Keep loading healthy tables toggle to be enabled. See Hooks for monitoring for setup steps. You can also spot a persistently failing table by its last sync time, which stops advancing until a retry succeeds.Source vs. destination toggles
The Sync Options section contains two similar toggles that behave differently:Considerations
- Snowflake destinations only. Other destinations are not supported at this time.
- Off by default. With the toggle off, a load failure stops and restarts the pipeline. This is the previous behavior.
- Retries are unbounded. A table that keeps failing is retried on every sync until it succeeds. Use the Destination Load Failed hook event to catch tables that fail repeatedly.