Validation and compliance solutions for MedTech, built so the effort goes where the risk is and the evidence is generated rather than assembled.
The direction of regulatory thinking on computer systems has been clear for some years: assurance effort should be proportionate to risk, testing should concentrate where failure matters, and documentation exists to support confidence rather than to demonstrate diligence. Most organizations know this and most still validate every function to the same depth. The reason is not that anybody misread the guidance. It is that uniform exhaustive testing requires no judgement and therefore no defence, while a risk-based approach requires somebody to write down that a function was tested less and to put their name next to it. Nobody is rewarded for that decision and one person will be asked about it in an audit.
Which makes this a governance problem wearing the costume of a technical one, and it is solvable in exactly that order.
Look at where validation hours actually land. Login behaviour, mandatory field enforcement, audit trail capture, user permissions, screen navigation: all extensively scripted, all easy to write protocols for, all low-risk and supplied largely intact by the vendor. Now look at what receives light coverage. The derivation rule that calculates a reported figure. The integration behaviour when the upstream system returns nothing rather than an error. The workflow path taken when an approver is unavailable and the record is reassigned. The data migration where counts reconciled and individual relationships did not. Each of those is where a real failure originates and each is hard to script, which is precisely why the protocol is thin there.
This is not negligence, it is the predictable output of measuring validation by volume of evidence. A protocol with four hundred test steps looks more rigorous than one with sixty, regardless of what the steps cover, and nobody auditing the plan will question depth in the easy areas. So the effort distribution ends up roughly inverse to the risk distribution, the organization spends heavily, and the failures that occur after go-live are concentrated in exactly the areas the validation was thinnest.
Rigour is measured by volume of evidence
Four hundred shallow test steps read as more diligent than sixty targeted ones.
Documentation is assembled rather than generated
Protocols, results and traceability are compiled by people after the fact, which is where the cost sits.
The vendor validated the platform, you configured the risk
Most failures live in configuration, integration and rules, which standard protocols cover least.
Change after go-live is prohibitively expensive
So the system is improved once, at implementation, and then frozen for a decade.
Traceability is maintained by hand
Requirements to tests to evidence, kept current manually, and out of date within one release.
Periodic review is a formality
A signature confirming nothing changed, on a system where a great deal has changed.
Price a small change before you plan the validation. Take a trivial modification to one of your validated systems, a changed field label or an added approver, and total what it costs end to end: assessment, protocol update, test execution, documentation, review, approval. Then ask how many improvements your team proposed in the last two years and how many were made. That ratio is the real output of your validation approach, and it is usually more informative than any audit finding. An organization where a small change costs weeks has a compliant system it has stopped being able to improve, and nobody writes that down anywhere.
Two things have to be true for risk-based validation to actually happen. The reasoning has to be documented and owned at a level where it can be defended, so no individual engineer is carrying the decision alone. And the evidence has to be produced by the system rather than by people, because that is what makes a lighter approach cheaper rather than merely thinner.
Step 1
Establish intended use, function by function
Step 2
Assess risk per function, not per system
Step 3
Agree the assurance approach with your quality function
Step 4
Classify by risk and assign effort accordingly
Step 5
Target the high-risk functions properly
Step 6
Build the test automation alongside the system
Step 7
Generate the evidence from the pipeline
Step 8
Automate traceability maintenance
Step 9
Design the change path before go-live
Step 10
Operate the lifecycle deliberately
Lighter validation is only cheaper if the evidence is automatic. A risk-based approach that still produces its evidence through people executing scripts and compiling documents saves some effort at implementation and none afterwards, which means change stays expensive and the system still freezes. The combination that works is risk-based scoping with automated execution and generated evidence: the high-risk functions get deeper coverage than they do today, the low-risk ones get appropriate coverage, and the whole suite runs on every change at close to zero marginal cost. That is what makes the second, third and tenth change affordable, which is the outcome the organization is actually buying.
We deliver validation alongside the systems we build and for systems we did not build. The distinction that matters is that the test automation and the evidence generation are engineering work, so they belong with the engineering rather than in a separate phase run by a separate team against a finished product.
Assurance strategy and risk
Intended use definition function by function, which is the input every subsequent judgement depends on
Risk assessment per function, covering consequence of failure, who is affected and whether it would be detected
Assurance approach documented and structured for approval at the level that can carry it
Scripted, unscripted and exploratory testing allocated by what each is actually good at
Vendor and supplier assessment, so platform assurance is leveraged rather than duplicated
Validation planning that distinguishes the vendor platform from your configuration, which is where your risk is
Testing and evidence
Automated test suites built alongside the system, covering workflow, integration, calculation and exception paths
Targeted depth on derivation rules and calculations, validated against worked examples your function agrees
Integration testing under failure conditions, not only under the conditions where everything responds
Data migration verification at record and relationship level, not only at reconciled totals
Evidence generated as a build output, versioned with the release and current by construction
Requirements traceability derived from the pipeline rather than maintained as a document
Lifecycle and change
Change impact assessment supported by the traceability, so the affected tests are identified rather than guessed
Regression suites that run on every change, which is what keeps later improvement affordable
Periodic review with evidence attached, rather than a signature confirming nothing changed
Configuration drift detection, since an undeclared change to a validated system is an uncontrolled change
Vendor upgrade assessment, because the platform will change on a schedule you do not control
Compliance reporting on validation state across the estate, including in Power BI where that is your environment
These are genuinely different activities that share a word. The third column is where the risk concentrates, which is where the effort should go rather than spreading it evenly across the system.
| Context | What is being validated | Where the risk concentrates |
|---|---|---|
| Regulated enterprise system | A purchased platform plus your configuration | Your configuration, integrations and migrated data. Not the vendor platform |
| Product software in device scope | Software that is part of the device | Behaviour under fault, boundary conditions and anything safety-related |
| SaMD released continuously | A product changing weekly against a process built for annual change | The change path itself, since the process rather than the testing is the constraint |
| Connected device cloud | Services the deployed fleet depends on | Version compatibility across a fleet that cannot all be updated |
| Analytics and reporting systems | Calculations feeding regulated decisions | Derivation rules, which standard protocols barely touch |
There are only two architectures here. In the first, people execute tests and compile documents, and every change repeats that cost. In the second, the delivery pipeline runs the tests and emits the evidence, and the marginal cost of the next change approaches zero. Everything else on this page depends on which one you have.
| Artifact | Generated by the pipeline as | What it replaces |
|---|---|---|
| Test execution records | Output of the automated run, timestamped and attributable | Signed screenshots and manually completed scripts |
| Requirements traceability | Derived from test annotations and build metadata | A matrix maintained by hand and stale within one release |
| Test coverage evidence | Measured against the risk classification per function | An assertion in the validation summary report |
| Change impact | Affected tests identified from the traceability graph | Judgement about what might be affected, then testing everything |
| Configuration state | Infrastructure and config declared in code and diffed | An environment nobody can describe precisely |
| Release evidence package | Assembled at build time, versioned with the release | A document compiled in the week before go-live |
If validation is a phase, it will always be done by hand. The structural obstacle to everything on this page is organizational rather than technical. When validation is scheduled as a stage that begins once the engineering is complete, run by a team that was not involved in building the system, generating evidence from the build is no longer possible and the only remaining option is people executing scripts against a finished product. Nothing improves while that sequence holds, however good the intentions. The change that makes the rest achievable is putting the test automation and the evidence generation inside the engineering work, with the quality function setting the approach and reviewing what comes out rather than executing it.
Worth stating plainly, because it is the crux of this page: the frameworks below generally expect assurance proportionate to risk and intended use. They do not require uniform exhaustive testing, and the organizations doing it are responding to perceived audit risk rather than to a written requirement. That said, what applies to your systems, to what extent, and what evidence is sufficient are determinations for your quality and regulatory functions. We build to the approach they set, we make the reasoning defensible, and we do not make the determination.
Frameworks the approach may need to support
- FDA QMSR and ISO 13485 as the quality system frameworks the validation activity sits inside
- FDA computer software assurance guidance, where the organization adopts a risk-based assurance position
- GAMP 5 as the lifecycle framework, including its categorisation of software and supplier reliance
- IEC 62304 and ISO 14971 where the software is product rather than infrastructure, and 21 CFR Part 11 and EU MDR where the records fall within scope
Risk and intended use
- Intended use established per function, since it is the input every proportionality judgement depends on
- Risk assessed on consequence of failure, population affected and whether failure would be detected
- The assurance method approved once by quality leadership, with each system an instance of it
- Supplier assessment documented where platform assurance is being relied upon
Evidence and data integrity
- Evidence generated by the execution that produced it, attributable and versioned with the release
- Attributable, legible, contemporaneous, original and accurate applied to the evidence chain, not only to the record
- Traceability from requirement through test to evidence, derived rather than maintained by hand
- Controlled correction, where re-running or restating evidence creates a record of what changed and why
Lifecycle control
- Change categories and their assurance paths agreed in advance rather than argued per change
- Configuration and environment under version control, with drift detected automatically
- Periodic review driven by evidence of what actually changed, rather than by a date and a signature
- Validation treated as a qualified decision rather than a system status, since all tests passing is not acceptance
- Vendor upgrade assessment, since the platform changes on a schedule the manufacturer does not control
A system is validated at a point in time against a configuration that then begins to change. Everything below concerns the gap that opens between the validated state and the running state, which is where audit findings come from and which almost no organization measures.
| Failure | How it presents |
|---|---|
| Configuration drifts from the validated baseline | Small adjustments accumulate and nobody can say what the system is now |
| A vendor upgrade changes validated behaviour | The platform moves on the vendor schedule and the assessment was cursory |
| Traceability goes stale | The matrix describes a release two versions ago and is still being signed |
| A rule is corrected without revalidation | The calculation changes and the evidence still refers to the previous logic |
| Test data does not represent real use | Everything passes against clean synthetic records and production is not clean |
| An integration change alters meaning, not format | The field transfers successfully and is interpreted differently at the other end |
| Evidence cannot be tied to a configuration | Results exist and nobody can say which version of the system produced them |
| Validation records are updated in place | History is overwritten and an earlier approval can no longer be reconstructed |
| An integration counterparty changes | The other side alters a field and your validated behaviour quietly breaks |
| Periodic review confirms nothing changed | A great deal changed and the review had no evidence to look at |
| Access accumulates beyond role | Permissions follow people rather than responsibilities and segregation lapses |
| Change impact is judged rather than derived | Affected tests are guessed, so either too much or too little is re-run |
Validation programmes get reported as systems validated and protocols executed. The outcomes that matter are whether the effort went where the risk is, whether the organization can still improve its systems, and whether the running system is demonstrably the validated one.
| Category | What we measure | Why it matters |
|---|---|---|
| Cost of a small change | Hours end to end for a minor modification to a validated system | Determines whether the system will ever be improved again |
| Effort against risk | Distribution of validation effort across risk-classified functions | Shows whether the approach is risk-based in practice or only on paper |
| Evidence generated automatically | Share of validation evidence produced by the pipeline rather than by people | The mechanism that makes lighter validation genuinely cheaper |
| Traceability currency | Whether the matrix reflects the release currently running | A stale matrix is worse than none, because it is signed |
| Undeclared drift | Configuration changes since baseline that were not assessed, by age | The most honest available measure of validated state |
| Vendor upgrade backlog | Platform releases received and not yet assessed | An accumulating liability nobody reports and an auditor will find |
| Improvements proposed and made | Ratio over the last two years, per system | The quiet cost of the validation approach, visible nowhere else |
We will measure what a change actually costs you, assess where validation effort currently sits against where the risk is, structure a risk-based approach your quality leadership can approve as a method rather than as a judgement call, and build the automation that generates the evidence so the second change is cheap.
CaliberFocus is a MedTech engineering partner delivering risk-based validation and compliance solutions where the effort goes where the risk is and the evidence is generated rather than assembled. From function-by-function intended use and risk assessment to test automation built alongside the system, pipeline-generated evidence, derived traceability and drift detection, we keep your systems validated and still affordable to change, working to the approach your quality function sets.
- Validation effort concentrated where failure actually matters
- Evidence generated by the pipeline, not assembled before a release
- Traceability derived automatically and current with every release
- Validated state held over time with drift and vendor upgrades tracked
Security & Compliance
