You alert or open a ticket when an expected client file doesn't arrive by building a file-arrival check that runs on a schedule, compares what was expected against what actually landed, and fires a notification or ticket the moment a gap appears. This guide is for data analysts and data engineers who manage recurring file-based ingestion from multiple clients or vendors. By the end, you'll be able to set up monitoring that catches a missing file within minutes instead of finding out when a downstream report looks wrong.
A missing file is different from a failed job. The pipeline doesn't error out; it simply has nothing to process, so it finishes "successfully" with zero rows loaded. That silent failure is what makes missing-file detection a distinct problem from standard job monitoring, and why it needs its own check.
The Problem
Most ETL monitoring watches for job failures, not job absences. If client data usually arrives by 6 a.m. and a vendor's SFTP drop fails silently overnight, a schedule-only pipeline runs on time, finds an empty folder, and completes without error. Nobody hears about it until a report looks off or a client asks where their dashboard update went. Teams running dozens of client feeds often only learn about a missing file when someone downstream notices stale data, sometimes a full day later.
What You'll Need
- A defined arrival schedule per source (time window, frequency, expected file naming pattern)
- Write access to the pipeline or orchestration layer that ingests the file
- A notification channel: email, Slack, or a ticketing system (Jira, Zendesk, PagerDuty)
- Integrate.io or a similar ETL platform with scheduling and alerting built in, if you're not maintaining custom scripts
How to Alert When an Expected Client File Doesn't Arrive: Step-by-Step
Step 1: Define the Expected Arrival Window Per Source
Every client or vendor feed needs its own expectation, not a single global rule. A daily servicer file might be expected by 6 a.m.; a weekly export might only need a Monday check.
What to do:
- List each source with its expected delivery frequency (hourly, daily, weekly)
- Set a cutoff time or window, not just a day (e.g., "by 7:00 a.m. ET")
- Include a grace period (15-30 minutes) to absorb normal delivery jitter
- Store this as configuration, not hardcoded logic, so it's easy to update per client
Output of this step: A per-source arrival schedule (source name, expected frequency, cutoff time, grace period) that the monitoring check will reference.
Step 2: Standardize the File Naming and Location Pattern
A missing-file check can't work if it doesn't know exactly what filename or path to look for. Inconsistent naming is the most common reason these checks produce false positives.
What to do:
- Confirm the expected file naming convention with each source (e.g.,
clientname_YYYYMMDD.csv)
- Confirm the landing location: SFTP folder, S3 bucket, or email inbox
- Document any variation (some vendors add a suffix on holidays or reruns)
- Flag sources with inconsistent naming for a follow-up conversation with the vendor
Output of this step: A confirmed filename pattern and landing path for every monitored source.
Where Integrate.io helps: Integrate.io's file connectors support wildcard and pattern matching on SFTP, S3, and FTP sources, so a check can match on clientname_*.csv instead of requiring an exact filename, which absorbs minor vendor-side naming drift without triggering a false alert.
Step 3: Build the Arrival Check
This is the core logic: a scheduled job that runs slightly after the expected cutoff and checks whether a matching file exists.
What to do:
- Schedule the check to run 15-30 minutes after the defined cutoff from Step 1
- Query the source location for a file matching the expected pattern within the expected window
- Return a boolean result: file found or file missing
- Log the check result regardless of outcome, so you have a history to audit later
Output of this step: A scheduled job that produces a pass/fail result for each source, every cycle.
Where Integrate.io helps: Integrate.io lets you build this check as a standard pipeline step using its scheduling engine, which supports intervals from every 5 minutes up to custom cron-style schedules, without writing or maintaining a separate cron job outside the platform.
Step 4: Route the Failure to a Notification Channel
A failed check is only useful if someone sees it fast. Email is the lowest common denominator; Slack and PagerDuty get attention faster for anything time-sensitive.
What to do:
- Decide the notification tier per source: low-priority sources to email, high-priority to Slack or PagerDuty
- Include the source name, expected file pattern, and cutoff time in the message body
- Include a direct link to the source folder or bucket so the on-call person doesn't have to hunt
- Avoid routing every missing-file alert to a single shared inbox; ownership gets diffused fast
Output of this step: A notification that reaches the right person or channel within minutes of a missed cutoff, with enough context to act without additional lookup.
Step 5: Auto-Create a Ticket for Recurring or High-Priority Gaps
For sources tied to SLAs or client-facing deliverables, a chat notification can get missed. A ticket creates an auditable record and forces follow-up.
What to do:
- Connect the alert to a ticketing system (Jira, Zendesk, ServiceNow) via webhook or native integration
- Auto-populate the ticket with source name, expected time, and last successful load timestamp
- Set a default priority based on the source's SLA tier
- Assign tickets to a rotating on-call owner rather than a static individual, so response time doesn't depend on one person's availability
Output of this step: A ticket exists in the team's tracking system within minutes of a missed file, with enough detail for whoever picks it up to start troubleshooting immediately.
Step 6: Review and Tune False Positive Rates Monthly
Arrival checks that fire too often get ignored. A check that fires too rarely misses real gaps. Both failure modes need periodic review.
What to do:
- Pull a monthly report of all triggered alerts and their resolution (real gap vs. false positive)
- Adjust grace periods for sources with frequent, harmless late-but-fine arrivals
- Tighten checks for sources where a real gap was missed or caught too late
- Retire checks for sources that have been decommissioned or consolidated
Output of this step: An arrival-monitoring system with a documented false-positive rate, reviewed and adjusted at a known cadence rather than left to drift.
Common Mistakes to Avoid
- Treating "job ran successfully" as proof data arrived; a job can complete with zero rows processed and still report success, so the check has to be independent of the ingestion job itself.
- Using one global cutoff time for every source instead of per-source windows; a mismatch here is the single biggest cause of both missed alerts and false positives.
- Routing all alerts to one inbox that nobody monitors closely on weekends. Assign per-source or tiered ownership instead.
- Skipping the grace period, which causes normal delivery jitter to generate noisy false alerts. Build in 15-30 minutes minimum.
- Not logging check results when the file does arrive. Without a full history, you can't distinguish a one-off fluke from a pattern.
- Building custom cron scripts for this from scratch when a platform like Integrate.io already supports scheduled file checks with pattern matching and built-in alert routing, which removes the maintenance burden of a separate monitoring script.
Conclusion
Alerting or opening a ticket when an expected client file doesn't arrive comes down to defining a per-source arrival window, checking for the file against that window on a schedule, and routing any gap to a channel someone actually watches. Once that loop is running, missing data gets caught in minutes instead of surfacing days later as a client complaint about a stale dashboard. Teams running dozens of client feeds through a platform like Integrate.io get this monitoring as part of the scheduling layer itself, rather than as a separate script to maintain. With reliable file-arrival monitoring in place, the next problem worth solving is reducing the manual effort spent investigating each gap once it's flagged.