Clinical and research data integration for life sciences, built so the standardised form and the source form both survive.
Standards exist to make data comparable, and comparability is achieved by discarding the things that make datasets different. A submission standard imposes a structure that a study did not collect in. An observational model harmonises across sources by reducing each to a common denominator. Coding a verbatim term to a dictionary replaces what somebody actually wrote with the nearest controlled concept. Converting units, deriving a visit window, collapsing a free-text qualifier: each of these is a reasonable trade of detail for consistency, and each one quietly removes something. That is fine until a later question needs the part that was removed, which is when an organization discovers that the mapping was performed, the result was accepted, and nobody recorded what did not come across.
Clinical and research data integration is unusual in having an abundance of standards rather than a shortage, which creates its own confusion. A submission data structure is designed so a regulator can review studies consistently; it is an output format and it was never intended as a working model for analysis across your portfolio. An analysis dataset structure serves statistical reporting for a study. An observational common data model exists to make heterogeneous real-world sources comparable, which it achieves by mapping everything onto a deliberately general structure. An exchange standard moves information between clinical systems. Each is excellent at its own job and none of them is a general-purpose model of your clinical and research data.
The recurring error is selecting one standard as the internal model and bending everything else to fit it. Research data gets forced into a submission structure with no place for it, rich study detail is flattened into an observational model built for breadth, and information that matters operationally has nowhere to live. The workable position is less tidy and more durable: hold your own model, map to each standard at the boundary where it is required, and treat every mapping as a documented, reversible transformation rather than the canonical form.
Mapping discards detail and nobody records what
The transformation is performed, the output accepted, and the loss is never written down.
A subject has a different identity in every system
And joining those identities is a governed act, not only a technical one.
Terminology is versioned and history was coded differently
A trend across several years is partly a trend across dictionary releases.
Real-world data was generated for care and billing
Not for your research question, and the absence of a code is not the absence of a condition.
Reconciliation is treated as a quality check
When in clinical work it is the evidence that the integration can be relied upon.
Each study has its own conventions
So an integration built for one study needs rebuilding for the next unless the variability was designed for.
Two fields with the same name mean different things
And the join succeeds, which is what makes it dangerous.
One principle runs through everything below. The standardised dataset is an output, not the record. The source form is retained, the transformation is specified both ways where possible, and what the mapping discards is written down as deliberately as what it produces.
Step 1
Establish what each standard is being produced for
Step 2
Model your own clinical and research entities
Step 3
Settle the subject identity model and its governance
Step 4
Profile the real sources before specifying anything
Step 5
Specify mappings with the loss documented
Step 6
Version terminology explicitly
Step 7
Build the pipelines with reconciliation attached
Step 8
Enforce blinding and consent in the integration
Step 9
Validate the transformations against worked examples
Step 10
Operate with exception management and change control
Three groups. The first is well-understood integration work, the second is where clinical and research data differs from everything else, and the third is what makes the result usable by people who are accountable for what they publish.
Connect
EDC integration across study builds, with the per-study variability designed for rather than rebuilt around
CTMS, eTMF and operational system integration, where the study and site entities have to agree with the clinical data
LIMS and ELN integration, carrying method, instrument and condition rather than results alone
EHR and real-world data integration with source provenance and coding context retained
Laboratory and central vendor data, including the formats that arrive by file transfer and have no interface
Pipeline engineering in batch and incremental patterns, with reconciliation produced on every load
Harmonise
Subject, study, site, visit and sample identity resolution, with the governance of each join made explicit
Mapping to CDISC structures for submission and analysis, with the discarded detail documented
Observational model conversion where cross-source research requires it, and a clear statement of what it costs
Terminology coding and version management across medical, laboratory and clinical dictionaries
Unit, range and convention harmonisation, including the central and local laboratory differences
Master data for products, sites, investigators and suppliers across studies and systems
Prove
Source-to-target reconciliation at count, record and value level, produced by the pipeline on every run
Lineage from any analysis value back through transformations to the originating record
Exception management with classification and ownership, so a discrepancy has an addressee
Study metadata governance, so a dataset can be interpreted by somebody who was not on the study
Blinding and consent enforcement within the integration, per user and per study at query time
Validation of transformation logic against worked examples agreed with data management and statistics
The third column is the join each integration depends on, and it is where the work is. An integration scoped from the systems involved underestimates every one of these.
| Integration | What it brings together | What it joins on |
|---|---|---|
| Operational study view | EDC, CTMS, eTMF and vendor data | Study, site and visit, where each system defines them slightly differently |
| Clinical and laboratory | EDC and central or local laboratory results | Subject and sample, with collection and accession identifiers in between |
| Translational research | Clinical outcomes and biomarker or omics data | Subject and specimen through preparation, aliquoting and re-analysis |
| Submission preparation | Study data mapped to submission structures | The study model and the standard, with the mapping as the artifact |
| Safety and clinical | Case data and study data | Subject and event, across systems with different coding and timing |
| Real-world evidence | EHR, claims and registry sources | Patient across sources, where a stable identity may not legally exist |
| Cross-study research | Several completed studies | Harmonised endpoints and populations defined differently per protocol |
The architecture that works holds an internal model and produces each standard as a view of it. The architecture that causes trouble picks one standard as the internal model. The table below is what each standard is for, which should settle the question of whether any of them is your model.
| Standard | What it is for | What it is not |
|---|---|---|
| CDISC SDTM | Presenting collected study data for regulatory review | A working model for cross-study analysis or research |
| CDISC ADaM | Analysis-ready datasets supporting a study’s statistical reporting | A general store of everything a study collected |
| CDISC ODM | Exchanging study metadata and clinical data between systems | An analysis structure |
| OMOP CDM | Making heterogeneous real-world sources comparable for observational research | A high-fidelity record of any one source |
| HL7 FHIR | Exchanging health information between clinical systems | A research data model, and not designed for trial data |
| HL7 v2 | Established messaging between healthcare systems | A storage or analysis format |
| MedDRA, SNOMED, LOINC | Controlled vocabularies for events, concepts and tests | Stable over time, which is why versioning matters |
Reading the third column together makes the architectural point. Every one of these was designed for a purpose that is not holding your clinical and research data in its most complete form, and that is not a criticism of any of them. An internal model that is yours, with each standard generated from it when required, lets the organization satisfy a submission, support observational research and exchange with a clinical system without any of those requirements degrading the others.
Clinical and research data arrives carrying obligations attached to its origin: consent that permits specific uses, blinding that restricts who may see what and when, privacy requirements that follow the data wherever it goes, and sponsor or jurisdictional constraints. An integration brings all of those into one place simultaneously and the strictest applicable requirement governs the result. That is manageable when it is understood at design time and close to impossible to retrofit, which is why these determinations belong at the start. They are made by your clinical, quality, regulatory, privacy and legal functions, and we build to them.
Frameworks where they apply
- ICH GCP across clinical activity, with sponsor and investigator obligations reflected in the integration
- GxP and GAMP 5 where the integration supports regulated activity, with assurance proportionate to use
- 21 CFR Part 11 and EU Annex 11 where the records and signatures held fall within scope
- CDISC where submission data is produced, and FAIR as principles the internal model should satisfy
Subject protection
- Consent scope recorded with the data, so a secondary use can be checked rather than assumed
- Blinding enforced within the integration, per user and per study, through unblinding and after it
- Re-identification risk assessed where datasets are linked, since linkage can defeat de-identification
- HIPAA and GDPR obligations as your counsel determines them, including for any processor involved
Data integrity
- ALCOA+ applied to integrated data as to source data, since combining does not reduce the obligation
- Lineage from any analysis value through every transformation to the originating record
- Audit history preserved across the integration rather than left in the source system
- Controlled correction, where a restated value preserves what it replaced and the reason
Validation
- Transformation logic validated against worked examples agreed with data management and statistics
- Reconciliation evidence produced by the pipeline and retained with the dataset it describes
- Terminology version changes treated as changes requiring assessment, because they alter results
- Re-validation on protocol amendment, source change or standard revision, all of which will occur
A clinical integration sits downstream of protocols that get amended, study builds that get modified mid-study, laboratories that change methods, vendors that change file formats, terminologies that release new versions and standards that get revised. None of those changes is announced to the integration team, and each one alters what the integrated data means.
| Failure | How it presents |
|---|---|
| A protocol amendment changes a definition | Pre and post-amendment data are pooled and the endpoint means two things |
| A study build is modified mid-study | A form or variable changes and the mapping silently stops covering it |
| A terminology version is released | Coding shifts, a trend moves, and no code changed on your side |
| A laboratory changes method or range | Values remain plausible and are no longer comparable with earlier ones |
| A vendor changes a file layout | Parsing succeeds against the wrong columns, or a field silently empties |
| Reconciliation passes on counts only | Totals agree, individual records do not, and nobody looked that far |
| A derived variable is corrected | Downstream outputs already issued now disagree and nothing flags it |
| Linked identity drifts | New subject or specimen identifiers arrive unmatched and the join quietly narrows |
| Study status changes and entitlements do not | A role or blinding state moves on and the integrated data keeps the old permissions |
| An integration has no named owner | Exceptions accumulate unresolved because responsibility was never assigned |
Integration programmes are reported in systems connected and datasets delivered. The outcome that matters is whether a statistician, a data manager or a researcher uses the integrated data without verifying it against the source system first, because that behaviour is the honest measure of whether the work succeeded.
| Category | What we measure | Why it matters |
|---|---|---|
| Value-level reconciliation rate | Records matching at value level, with discrepancies classified | The evidence the dataset rests on, and counts are not it |
| Source verification behaviour | How often users check the integrated data against the source | The honest measure of trust, and it is observable |
| Identity match rate | Subjects, specimens and sites resolving across systems, by study | Every analysis built on a join inherits this number |
| Mapping loss documented | Transformations with the discarded detail recorded at specification | Determines what questions the dataset can answer later |
| Time to integrate a new study | Elapsed effort for study N against study N minus one | Shows whether the design is compounding or rebuilding |
| Terminology version coverage | Coded values carrying an explicit version, and verbatim retained | Without it a multi-year trend cannot be interpreted |
| Exception ageing | Unresolved discrepancies by age and owner | An exception queue nobody works is a reconciliation nobody trusts |
We will establish what each standard is being produced for, model your clinical and research entities independently of them, document what every mapping discards, build reconciliation that runs on values rather than counts, and enforce blinding and consent where the data actually moves.
CaliberFocus integrates clinical and research data across EDC, CTMS, eTMF, LIMS, ELN and real-world sources with the source form preserved alongside the standard. We model your clinical and research entities independently of any one standard, document what every mapping discards, build reconciliation that runs on values rather than counts, and enforce blinding and consent where the data actually moves.
- Source form retained permanently alongside the standardised one
- Terminology versions and verbatim terms kept with every coded value
- Value-level reconciliation produced on every load
- Study-specific mapping as configuration, so each new study costs less
Security & Compliance
