Post-market surveillance analytics for MedTech, built on the real-world behaviour of the installed base rather than on the subset of events somebody chose to report.
Post-market surveillance in this industry is built almost entirely on complaints, and a complaint is not an observation of the product. It is an observation of somebody deciding to pick up the phone. The numerator counts the fraction of events that a user noticed, attributed to the device, judged worth reporting and had time to report. The denominator, which is how many devices were actually in use and for how long, is usually an estimate from shipment records. Every trend computed from those two numbers is a trend in reporting behaviour at least as much as in product behaviour, and a quiet quarter can mean the product improved or that a distributor changed.
Consider what has to happen for an event to become a complaint record. Something goes wrong. A user notices, attributes it to the device rather than to themselves, the procedure or bad luck, judges it worth mentioning, knows how to report it and has time to. It then survives triage and gets classified in a way that keeps it findable. Each step has a pass rate well below one and the product rarely sits at the centre of the decision. A confident clinician normalises what a nervous one reports, and a hospital that has just had a bad experience with another vendor reports everything for six months.
The denominator is weaker still. Units shipped is not units in use, and it does not know what was never installed, what sits in a cupboard, what was retired early, or what is used daily rather than twice a year. The rate driving surveillance decisions is therefore a biased numerator over an estimated denominator, and the organization reasons about safety from it anyway because it has been the only thing available. The consequence is not that the data is useless. It is that the direction of a trend is frequently uninterpretable, and people learn to discount the figures, which is worse than not having them.
The numerator is reporting behaviour
Complaint volume moves with attention, relationships and staffing as readily as with product performance.
The denominator is an estimate
Shipments are not usage, so exposure-adjusted rates are approximations presented as measurements.
Signals arrive after accumulation
Trending needs volume, volume needs time, and the pattern becomes visible after the exposure that produced it.
Free text defeats trending
The same failure is described five ways, so counts by category undercount anything inconsistently classified.
Surveillance data is spread across systems
Complaints, service, returns, registry, literature and field data each sit somewhere different.
Periodic reporting shapes the analysis
Reports built to demonstrate that nothing is wrong are not instruments designed to find things.
Date the evidence, not the discovery. For the last issue your organization investigated seriously, find the earliest point at which the data you already held would have supported the conclusion. Not when somebody noticed, and not when the complaint was filed, but when the evidence first existed somewhere in your systems. The gap between that date and the day the investigation opened is your detection lag, and it is the single most useful number in post-market surveillance. It is also almost never calculated, because calculating it requires admitting the answer could have been known sooner.
Most surveillance analytics projects begin with the complaint data because it is structured, available and owned by the function paying for the work. That produces better charts of the same biased measure. The sequence below starts with exposure, because a rate with a real denominator changes what every subsequent analysis means.
Step 1
Establish what you are actually measuring
Step 2
Build the exposure model
Step 3
Resolve product and device identity across sources
Step 4
Structure what is currently free text
Step 5
Integrate the surveillance sources
Step 6
Define signals with your risk function
Step 7
Instrument detection against those definitions
Step 8
Measure your own sensitivity
Step 9
Define what happens when a signal fires
Step 10
Monitor continuously and report from the same data
Earlier detection is an obligation, and the answer is a process, not a slower system. A surveillance capability that finds things months earlier means the organization is in possession of the information months earlier, with everything that follows from that. This is the unspoken reason these programmes stall in the middle, and it deserves saying plainly rather than being left to operate quietly. The resolution is not to see less. It is to decide in advance, with your regulatory and legal functions, what happens when something is seen: how it is triaged, who assesses it, how quickly, what record is made of a determination that no action is required, and who is accountable for that decision. An organization with that process is safer and more defensible than one whose surveillance is slow enough to be comfortable.
We build the data and analytics capability. We do not make reportability determinations, set risk acceptability or author surveillance strategy, all of which belong to your regulatory, clinical and risk functions working to your procedures.
Surveillance data foundation
Exposure modelling, so rates are expressed against devices in service and usage rather than units shipped
Device and product identity resolution across complaints, service, telemetry, registration and distribution
Structuring of free-text narrative into consistent failure modes, applied retrospectively as well as going forward
Integration of complaints, service and repair, returns, field actions, registry data and published literature
Device telemetry integration where the product is connected, as a surveillance input rather than an engineering one
Lineage from any surveillance figure back to the records it was computed from
Detection and analysis
Trend and rate monitoring against defined denominators, stratified by version, lot, site, configuration and cohort
Signal detection implemented to thresholds your risk function sets, with the definitions held as governed configuration
Disproportionality and comparative analysis across products, populations and periods, with the caveats stated
Time-to-event and survival analysis where failure timing rather than failure count is the question
Predictive modelling where the data supports it, bounded by what the evidence can carry
Sensitivity measurement of the surveillance system itself, against historical issues
Workflow, reporting and evidence
Signal triage workflow, with escalation, assessment and the recorded determination including no-action outcomes
Periodic surveillance reporting generated from live data rather than assembled separately
Vigilance and adverse event reporting data preparation, to the determinations your regulatory function makes
Dashboards for surveillance operations, including Power BI where that is your environment
Reproducibility of any analysis, so a conclusion can be regenerated exactly as it was reached
Validation of the analytics path, scoped with your quality function and proportionate to use
What a surveillance programme can know is set by what the product can see. The third column is the denominator available in each case, which determines how far the analytics can be pushed before it is estimating rather than measuring.
| Product type | What reaches you | The denominator available |
|---|---|---|
| Unconnected device | Complaints, service records, returns | Shipments, adjusted by assumption. The weakest position and the most common |
| Connected device | Complaints plus continuous telemetry | Devices in service and usage intensity, measured rather than estimated |
| SaMD | Complaints, support tickets, application logs | Active users and sessions, usually precise but rarely treated as surveillance data |
| Digital health platform | Support contacts, engagement data, outcomes | Enrolled and active population, with the engagement drop-off as its own signal |
| Implant or long-life device | Complaints, explants, registry data, revisions | Implanted population over time, where registries exist and cover the market |
Surveillance analytics is a join problem before it is an analysis problem. Each source below observes a different slice of the same population through a different instrument, and until they can be related to one device, one patient cohort and one time period, every figure computed from them is approximate in a way nobody can quantify.
| Source | What it observes | Its blind spot |
|---|---|---|
| Complaints and vigilance | Events somebody chose to report, with clinical context | Everything unreported, which is most of it |
| Service and repair | What actually failed, physically and specifically | Failures resolved without a service call, and usage context |
| Returns and explants | Confirmed failure with the artifact available | Timing, since the return happens long after the event |
| Device telemetry | Behaviour, errors, usage, version, environment | Clinical consequence, and anything the device cannot sense |
| Registry and literature | Outcomes at population scale, independently observed | Lag, coverage, and attribution to a specific manufacturer |
| Distribution and install base | Where units went and what configuration they carry | Whether they are in use, which is the thing you need |
A rate you cannot stratify is a rate that will mislead you. Aggregate rates are stable, reassuring and precisely the wrong instrument for finding a problem that affects one manufacturing lot, one firmware version, one hospital or one patient subgroup. A localised issue at serious severity can move the overall figure by an amount indistinguishable from noise, and the organization will see a flat line for the entire period in which the evidence was accumulating. Stratification should be the default view rather than the drill-down, and the dimensions that matter should be designed into the data model from the start, because adding one retrospectively usually means the history cannot be split at all.
Nowhere is the boundary between engineering and determination more important than here. We do not decide whether an event is reportable, what threshold constitutes a signal, whether a risk remains acceptable, or what your surveillance plan should contain. Those belong to your regulatory, clinical and risk management functions. We build the data foundation, the analytics and the evidence trail that make their decisions faster, better informed and defensible, and we build it so that the basis of any determination can be reproduced.
Frameworks the controls may need to support
- FDA reporting obligations applicable to the device, as your regulatory function determines them
- EU MDR post-market surveillance expectations, including the plan, periodic reporting and clinical follow-up where applicable
- ISO 14971, with surveillance feeding the risk file rather than running alongside it
- ISO 13485, where surveillance outputs connect to CAPA, change control and management review
Surveillance as a risk input
- Real-world rates compared against the assumptions the original risk analysis was built on
- Occurrence estimates updated from observed data rather than from the engineering judgement made before launch
- New hazards and use errors fed back into the risk file with the evidence attached
- Residual risk re-evaluated where the field tells a different story from the design, which is the point of surveillance
Data integrity
- Lineage from any surveillance figure back to the records and the analysis version that produced it
- Reproducibility, so a determination can be regenerated with the data as it stood at the time
- Audit trails over signal definitions and thresholds, since changing one changes what the system can find
- Controlled correction, where restating a historical rate creates a record of what changed and why
Decisions and evidence
- A recorded determination for every signal assessed, including the ones concluded as requiring no action
- The assessment basis preserved, since a no-action decision is the one most likely to be examined later
- Validation of the analytics path, scoped with your quality function and proportionate to use
- Validation of the classification and detection logic, because a wrong rule produces confidently wrong surveillance
A system that misses a real signal fails in the way everybody fears. A system that generates signals that are not real fails more insidiously, because the organization responds by raising thresholds until the noise stops, and the sensitivity it gave up is never measured. Both failures are quiet. The work below exists to make them visible.
| Failure | How it presents |
|---|---|
| The denominator is wrong | Every rate is wrong by the same factor, consistently, so nothing looks anomalous |
| A feed stops | Fewer events looks like improvement, which is the most dangerous direction for a gap |
| One incident is counted several times | The same event arrives through complaints, service and telemetry and inflates the trend |
| Alert volume drowns the real finding | A noisy threshold trains reviewers to clear the queue rather than read it |
| Classification drifts | A category changes meaning over time and a trend appears or disappears with no product cause |
| An aggregate conceals a cluster | A serious localised issue moves the overall rate within normal variation |
| Reporting behaviour changes | A distributor, process or relationship change moves the numerator and reads as product change |
| Thresholds were tuned to capacity | Alert volume matches what the team can handle, which is not a risk-based threshold |
| An analysis cannot be reproduced | The conclusion stands and the basis for it cannot be regenerated as it was |
| Fleet composition shifts | The mix of versions or configurations changes and the rate moves for no product reason |
Surveillance programmes get reported as sources integrated and dashboards delivered. The outcomes that matter are whether the organization finds out sooner than it used to, whether its rates mean anything, and whether it can defend how it reached a conclusion.
| Category | What we measure | Why it matters |
|---|---|---|
| Detection lag | Days between evidence first existing and the investigation opening | The headline measure of a surveillance programme, and almost never calculated |
| Denominator quality | Proportion of the installed base with measured rather than estimated exposure | Determines whether any rate on the page means anything |
| System sensitivity | Smallest effect the detection logic reliably catches, and how fast | Converts no signals identified from a hope into a quantified statement |
| Stratification coverage | Proportion of rates splittable by version, lot, site and cohort | Aggregates conceal the localised problems surveillance exists to find |
| Narrative structured | Share of complaint text consistently classified, including history | Decides whether failure mode counts describe failures or classification habit |
| Determination completeness | Signals assessed with a recorded basis, including no-action outcomes | The evidence that gets examined years later |
| Periodic reporting effort | Hours assembling figures versus hours interpreting them | Separates the work that should disappear from the work that matters |
We will establish what your surveillance can currently observe, build the exposure model that gives your rates a real denominator, calculate your detection lag against issues you have already found, and design detection to the thresholds your risk function sets, with the determination trail that makes a conclusion defensible.
CaliberFocus is a MedTech engineering partner delivering post-market surveillance analytics built on the real-world behaviour of the installed base. From exposure models that give your rates a real denominator and device identity resolved across complaints, service and telemetry, to structured narrative, stratified signal detection and a reproducible determination trail, we build the instrument while every determination stays with your regulatory, clinical and risk functions.
- Exposure-adjusted rates with a measured denominator, not shipments
- Detection lag and surveillance sensitivity measured, not assumed
- Rates stratified by version, lot, site and cohort by default
- Every determination recorded and reproducible, including no-action outcomes
Security & Compliance
