Detecting and surviving breaking changes in vendor APIs

On a Friday night the patient sync for one customer starts to return empty pages. Nothing in your code changed. The vendor shipped a release. This post covers how to see a vendor change in the first minutes, how to tell a vendor change from your own bug, and how to ship a connector fix while customers keep working.

What vendors promise

Certified API developers have an obligation to limit disruption. Under 45 CFR 170.404(a)(4)(iii), a certified API developer must make reasonable efforts to maintain compatibility. Except in exigent circumstances, it must give notice and a reasonable opportunity to update applications before it changes its certified API or its terms. That obligation covers the certified API only. Proprietary endpoints and web portals sit outside it.

The notice arrives as release notes. athenahealth publishes a change log with production releases every two weeks on Fridays and Preview releases on Saturdays. Entries carry the line "This is a breaking change" when they apply. One entry for the 26.11 release reduces the maximum LIMIT parameter on the patient case endpoints from 10,000 to 1,000. A client that requests 5,000 fails after that release. The page also sets a date by which breaking-change testing must be complete.

Larger changes arrive as end-of-support notices. Oracle Health announced the end of FHIR DSTU2 on June 3, 2024, closed new DSTU2 registrations on September 3, 2025 and set the deprecation date at December 1, 2025. Subscribe to every release notes feed and forum your connectors depend on, and assign a person to read each one.

What FHIR says about unknown content

FHIR separates changes into classes. The R4 versioning page defines breaking changes as changes that leave previously conformant applications non-conformant, and substantive changes as new functionality that does not break existing applications. New optional elements may be added at any location, and codes are never removed from fixed value sets. The page tells clients to ignore unexpected elements.

The extensibility rules add two instructions. Applications should ignore extensions they do not recognize. Applications must not process a resource that carries a modifierExtension they do not understand. A schema check that rejects every unknown field breaks on each harmless addition. A schema check that ignores every unknown field misses a modifier. Write the check to match the specification: unknown elements pass, unknown modifier extensions fail.

Profiles change too. US Core STU7 added Meta.lastUpdated as Must Support on several profiles and rewrote the Average Blood Pressure profile. A vendor that moves from one US Core version to another changes what your connector receives.

Schema checks on every response

Validate each response at the connector boundary before it enters your system. The check has four parts: the fields you require are present, the fields you read have the expected type, the enumerated values are in the set you handle, and the record count matches the paging headers. Write the schema from what your product consumes, not from the full vendor specification. A field you never read cannot break you, so do not check it.

{
  "resourceType": "Patient",
  "id": "a1b2",
  "name": [{ "family": "Doe", "given": ["Jane"] }],
  "birthDate": "1980-04-02",
  "identifier": [{ "system": "urn:oid:2.16.840.1.113883.4.1", "value": "..." }]
}

Emit one metric per connector per check: schema_fail_total{connector="athenaone", check="birthDate.type"}. A spike in one check across many customers is a vendor change. A spike for one customer is configuration at that site. Oracle Health's FAQ notes that data can be filtered per domain and code mappings differ by site, so the single-customer case is common.

Canary accounts

A canary is a scheduled job that runs the full connector path against an account you control. For certified APIs, use the vendor sandbox. athenahealth's Preview environment receives each release the day after production, so a sandbox canary sees the change after customers do. Pair it with a production canary on a customer that has consented, with a known test patient the practice created for that purpose.

For portals, the canary logs in, reads one known page and logs out. Portals have no sandbox and no release notes, so the canary is your only early signal. Run it at the start of each business day and before each scheduled batch.

Success-rate alerts per connector

Track the ratio of successful requests to total requests per connector and per operation over a short window. Alert when the ratio falls below the trailing baseline for that connector. A global threshold fails because connectors have different normal rates. A portal connector and a certified FHIR connector need different lines.

Separate rate limiting from failure. Oracle Health returns HTTP 429 when you exceed a customer's rate limit and recommends a retry after about one second with the delay doubled on each attempt, up to about three retries. A 429 that clears on retry is load. A 400 that repeats on every request after a release is a change.

Rolling a connector fix without downtime

Version each connector as its own deployable unit. When the vendor changes, you ship athenaone@2026.11.2 while athenaone@2026.11.1 still runs. Route traffic by customer: new version for one consenting customer, then a small group, then all. Each step checks the schema-failure metric and the success rate for the new version against the old.

Keep both response shapes legal during the transition. If a vendor renames a field, read the old name and the new name for one release and emit a metric when each is seen. Remove the old name when its count reaches zero.

When a change breaks reads but not writes, degrade rather than stop. Mark the affected fields unavailable in your output and let the rest of the record through. A record with a missing field and a clear flag is better than no record and better than a wrong value.

TheFhirStorm's open-source connectors for eClinicalWorks in Python and TypeScript show the boundary where this validation belongs. Put the schema check there, once, and every consumer above it inherits the protection.