To onboard a new health plan's eligibility file without building a new pipeline, profile the plan's real files against its spec, map them to one standard member schema, add validation and arrival alerts, and save the finished pipeline as a template for the next plan. This guide is for data and engineering leads at companies that health plans pay to serve their members, such as benefits vendors, care partners and Medicare Advantage or Medicaid service providers, along with the operations teams who run the files day to day. By the end, you will have a repeatable process that turns each new plan's file into clean member records without starting from scratch.
Every new plan client brings its own eligibility file. When the second, fifth and tenth plans arrive, the work should be a configuration change on a pipeline you already trust, not a new build.
The Problem: One Pipeline per Plan Does Not Scale
Health plans rarely send eligibility files the same way. One sends a monthly 834 enrollment file, another sends a weekly CSV, and a third emails a spreadsheet. Even plans that agree to your file spec drift from it: dates arrive as MMDDYY from one plan and with dashes from another, columns get renamed, and some deliveries are blank or are the wrong file entirely.
The first pipeline usually gets built quickly and hardcoded to one plan's layout. By the tenth plan, the team maintains ten one-off pipelines, and the ops team cleans exceptions by hand in spreadsheets. Late files go unnoticed until a member can't be served or a downstream report looks wrong.
What You'll Need Before Starting
- The plan's file specification or 834 companion guide, plus the expected delivery schedule
- Two or three de-identified sample files from the plan (scrub member data before testing)
- A dedicated SFTP folder for the plan's deliveries
- Your standard member schema: the fields every plan's data must end up in
- A destination, such as a Postgres or Snowflake table or an internal API
- An alert channel your team watches, such as email, Slack or PagerDuty
How to Onboard a Health Plan's Eligibility File: Step-by-Step
Step 1: Profile the Plan's Real Files Against Its Spec
The spec describes what the plan intends to send; the sample files show what it actually sends. Comparing the two first prevents most failures after go-live.
What to do: - Open each sample file and record its format (834, CSV, fixed-width or Excel), delimiter, header row and encoding - List every column and compare it with the spec: missing, renamed, extra or reordered fields - Check date, phone and ID formats across all rows, not just the first few - Note whether files arrive encrypted with PGP and how they are named, such as planname_eligibility_YYYYMMDD.csv
Output of this step: A profile of the plan's real file layout and every difference from its spec.
Where Integrate.io helps: For 834 files, the EDI (X12) source flattens the member and coverage loops into rows with named columns such as member_id and eligibility_begin, and decrypts PGP files in memory so plaintext is never written to storage. For background on the format, see Understanding the 834 file layout.
Step 2: Map the Plan's Fields to Your Standard Member Schema
Each plan keeps its own field names, but your systems should only ever see one schema. Mapping is where a plan's layout becomes your layout.
What to do: - Map each source column to a field in your standard member schema (member ID, name, date of birth, coverage start and end dates, plan identifier) - Add a source_plan column so every record carries its origin - Decide how to handle fields the plan doesn't send: leave them null, apply a default or reject the file - Record the mapping in a shared document so ops and engineering use the same reference
Output of this step: A field-by-field mapping from the plan's layout to your member schema.
Step 3: Normalize Formats Across Every Field
Plans format the same values differently. Normalizing them here means every downstream system can trust the data without plan-specific logic.
What to do: - Convert all dates to one format, such as YYYY-MM-DD; 834 dates arrive as CCYYMMDD strings and need converting too - Standardize names (case, trimmed spaces) and phone numbers (digits only) - Split combined columns (a full name in one field) or merge separate ones, as your schema requires - Trim leading zeros or pad IDs so member IDs match your existing records
Output of this step: Plan data in your standard formats, ready to validate.
Step 4: Add Validation Rules That Stop Bad Data at the Door
Validation decides what loads, what waits and what gets rejected. Catching problems here is cheaper than cleaning them out of your database later.
What to do: - Reject the whole file if it is empty, has the wrong column count or fails a header check, since that usually means the plan sent the wrong file - Check required fields on each row (member ID, date of birth, coverage dates) and route failing rows to a review table - Hold rows with duplicate member IDs for review and load the clean rows, so one bad record doesn't block a whole delivery - Compare row counts with the previous delivery and flag large drops, which often signal a partial file
Output of this step: Clean rows ready to load, plus a review queue of rejected files and held rows.
Step 5: Set Up Delivery Handling for Each Plan
How files arrive determines whether they get processed twice, skipped or lost. A consistent setup per plan removes most of that risk.
What to do: - Give each plan its own SFTP folder so files never mix - Match files by pattern rather than exact name, so small naming changes don't break the pipeline - Process each file exactly once and record which files have already run - Move processed files to an archive folder after a successful run, and keep failed files in place for review
Output of this step: A delivery setup where each plan's files are picked up once, processed and archived.
Step 6: Set Alerts for Failed Runs and Late or Missing Files
A failed run and a missing file are different problems. Most monitoring catches the first; the second stays silent until someone notices missing members.
What to do: - Alert on every failed run, with the file name, plan and error summary in the message - Record each plan's expected delivery schedule, such as every Monday or the first business day of the month - Run a scheduled check that compares expected deliveries with files actually received, with a grace period - Route alerts by severity: email for routine issues, Slack or PagerDuty for files that affect member services
Output of this step: Alerts that tell the team about failed runs and late or missing files before members are affected.
Where Integrate.io helps: File connectors can check an SFTP or S3 location for expected files and return their last-modified time, so a scheduled pipeline can flag a missing delivery and send the alert. This guide walks through it: How to alert or open a ticket when an expected client file does not arrive.
Step 7: Test With Scrubbed Files and Run in Parallel
Testing against real layouts, with member data removed, catches issues the spec never mentioned. A short parallel run proves the pipeline before anyone relies on it.
What to do: - Run the pipeline on the de-identified sample files and check every output field against expected values - Feed it a blank file, a wrong file and a file with duplicate member IDs to confirm the validation rules fire - Run it alongside the current process for one or two delivery cycles and compare record counts - Confirm that alerts reach the right channel by triggering a test failure
Output of this step: A tested pipeline with matching output from the parallel run, ready for production.
Step 8: Save the Pipeline as a Template for the Next Plan
The goal is that the next plan takes hours, not weeks. Turning this pipeline into a template is what makes that possible.
What to do: - Replace plan-specific values (folder path, file pattern, schedule, field mapping) with variables - Document which parts change per plan and which stay fixed, usually the validation rules and the destination - Export the pipeline definition to version control so engineers can review changes - For the next plan, copy the template, update the variables, and repeat steps 1 and 7
Output of this step: A reusable template that onboards the next plan by changing variables instead of rebuilding.
Where Integrate.io helps: Pipelines can switch connections and file paths through variables per schedule, so one template runs for every plan client. Pipeline definitions export as JSON through the API, which lets developers keep them in GitHub while ops teams work in the visual editor.
Common Mistakes to Avoid
- Building from the spec alone (step 1): plans drift from their specs. Profile real sample files before mapping anything.
- Loading plan field names directly (step 2): downstream systems end up with plan-specific logic. Map everything to one member schema first.
- Rejecting a whole file for a few bad rows (step 4): one duplicate member ID shouldn't block thousands of valid records. Hold failing rows and load the rest.
- Monitoring only for failed runs (step 6): a file that never arrives doesn't fail; it just stays silent. Add a scheduled check for expected deliveries.
- Testing with live member data (step 7): eligibility files contain PHI. Scrub sample files before testing outside production.
- Copying pipelines instead of templating them (step 8): ten copies means ten places to fix the same bug. Use variables so one template serves every plan.
Conclusion
Onboarding a new health plan's eligibility file without building a new pipeline comes down to eight repeatable steps: profile the real files, map to one schema, normalize, validate, handle deliveries, alert, test and template. Once the first plan runs through that process, each new plan reuses the same validation, alerts and destination, and only the variables change. Integrate.io shortens the slowest parts, parsing 834 files, checking for missing deliveries and running one template across every plan, so engineers stop rebuilding pipelines and ops teams stop cleaning files by hand. With onboarding reduced to configuration, teams can sign new plans faster and spend their time on the members those files represent. To see how this works for companies serving health plan members, read about eligibility file ingestion for point solutions.
Frequently Asked Questions
How do you handle eligibility files from multiple health plans in different formats?
Map every plan's layout to one standard member schema and normalize formats before loading. Keep plan-specific details, such as file paths, naming patterns and field mappings, in variables on a shared template. Each new plan then becomes a configuration change rather than a new pipeline.
How do you validate an eligibility file before it loads?
Check the whole file first: reject it if it is empty, has the wrong number of columns or fails a header check. Then check each row for required fields and duplicate member IDs. Hold failing rows for review and load the clean ones.
How do you get alerted when a health plan's eligibility file is late or missing?
Record each plan's expected delivery schedule, then run a scheduled check that compares expected files with files received, allowing a grace period. Send an alert for each missing file, and deduplicate alerts so the team gets one notice per missed delivery.
What is the difference between an 834 file and a CSV eligibility file?
An 834 is the ANSI X12 standard for benefit enrollment and maintenance. It uses nested segments for members and coverages and needs parsing before it becomes rows. A CSV eligibility file is a flat table in a layout the plan defines, so it is easier to read but varies more from plan to plan.