Data platform engineering for life sciences, built so the conditions that make scientific data interpretable survive the journey into the platform.
Every measurement in this industry was generated to answer something specific. An assay result exists because somebody was testing a particular hypothesis under conditions chosen for that purpose. A clinical observation was collected to a protocol with inclusion criteria that shaped who is in it. A batch record documents one manufacturing run under the process in force that week. Each is entirely valid for its original purpose and reusing it for a different question requires knowing those conditions, because without them you cannot tell whether two results are comparable, whether a population is relevant, or whether an apparent difference is biology or circumstance. That context is exactly what gets dropped when data is extracted, flattened and landed somewhere central.
Most life sciences data platform programmes succeed at the part that is measurable and stall at the part that matters. Connectors are built, pipelines run, volumes grow, and a lakehouse accumulates a faithful copy of what the source systems hold. Then somebody asks a question that spans two sources and discovers that the same compound has three identifiers, that the two assays were run on different instruments under conditions nobody recorded, that one dataset uses a unit convention the other does not, and that the field everyone assumed meant the same thing in both systems does not.
None of that is a pipeline failure. The pipelines did precisely what they were built to do, which was to move data. What was not built is the layer that makes the data mean something across sources: entity resolution, unit and terminology normalisation, the experimental and clinical context that explains how each value came to exist, and the lineage that lets somebody establish where a number came from. That layer is unglamorous, it does not demonstrate well, it is where essentially all of the difficulty lives, and it is routinely scoped as a later phase that never arrives.
The conditions that explain a result are not in the pipeline
Instrument, operator, protocol version and deviations stayed behind in the source or in a notebook.
The same entity has different identifiers everywhere
Compound, subject, sample, site and product resolve differently in each system.
Units and terminology are inconsistent
And the inconsistency is silent, because both values are numbers and both look reasonable.
Permissions were designed around systems, not data
A platform that unifies access across sources can breach boundaries the sources enforced.
Two teams compute the same figure differently
Same source data, different transformations, and the board sees two numbers for one thing.
Point-to-point integration costs more every year
Each interface needs maintaining, and one source change breaks several things nobody mapped.
Migration drops provenance
Historical data arrives in the new platform without the lineage that made it defensible.
Nobody owns a pipeline after it works
It runs unattended until it quietly does not, and the absence looks like a flat trend.
Ask one question that crosses two systems and watch what happens.
Pick something your scientists or operations people would genuinely want to know that requires joining two sources: results across two assay platforms, outcomes against manufacturing batch, safety signals against study population. Then try to answer it. The exercise rarely fails at the pipeline. It fails at identifier mismatch, at missing context, at a unit convention nobody documented, or at a permission nobody can resolve. Whatever it fails at is the actual scope of your platform programme, and it will not appear in a proposal that was written from a source system inventory.
Platform programmes scoped from a source system inventory build connectors. Programmes scoped from the questions people cannot currently answer build the semantic layer that makes the connectors worth having. The second approach is harder to put in a proposal and it is the one that produces a platform anybody uses.
Step 1
Collect the questions that cross systems
Step 2
Trace each question to the sources and the joins it requires
Step 3
Establish the entity model and resolution strategy
Step 4
Define what context must travel with each data type
Step 5
Settle the permission model before the architecture
Step 6
Design the layers deliberately
Step 7
Build pipelines with observability from the first one
Step 8
Migrate historical data with its provenance
Step 9
Validate proportionate to what the data supports
Step 10
Operate with ownership, cost control and change management
The semantic layer is the product. The pipelines are plumbing.
Connectors and pipelines are well-understood engineering with mature tooling, and any competent team can build them. What determines whether a platform is used is the layer that makes data from different sources mean the same thing: resolved entities, normalised units and terminology, preserved context, and lineage back to origin. That layer is specific to your systems, your science and your organization, which is exactly why no product supplies it and why it is where the effort should concentrate. A programme that budgets for connectors and treats semantics as a later phase has budgeted for the easy half.
Three groups. The first is where most proposals concentrate, the second is where the value is created, and the third is what stops the whole thing degrading eighteen months after go-live.
Platform and movement
Platform strategy and architecture across lakehouse, warehouse and hybrid patterns, chosen against the workloads you actually have
Integration across LIMS, ELN, CTMS, EDC, eTMF, RIM, QMS, ERP, MES and instrument systems
Pipeline engineering in batch, streaming and event patterns, sized for the data rather than for the diagram
Migration from legacy platforms with provenance and audit history preserved, not only current values
Cloud architecture and modernisation, including the question of what genuinely needs to move
DataOps practice: version control, environments, testing and deployment applied to data work as to software
Meaning and modelling
Entity resolution and master data across compounds, subjects, samples, studies, products, sites and suppliers
Data modelling for scientific and clinical domains, where the relationships carry as much meaning as the values
Unit, terminology and controlled vocabulary normalisation, including the mappings nobody has documented
Experimental and clinical context captured as part of the record rather than as a methods note
Metadata management and cataloguing, so a dataset can be understood by somebody who did not create it
Lineage from any figure back through every transformation to the source record that produced it
Trust and operation
Data quality rules defined against the questions the data has to answer, not against the database generally
Data observability covering freshness, volume, schema and distribution, instrumented as pipelines are built
Reconciliation against source systems on a schedule, which catches what internal validation cannot
Access control enforced at query time, per user, respecting blinding, consent and separation obligations
Cost management, since a platform that is cheap to build can be expensive to run and nobody owns the bill
Validation of the platform scoped with your quality function, proportionate to what the data supports
The third column is the specific join each domain needs and cannot currently make. Scoping a platform from these is more productive than scoping it from a list of source systems, because the join is where the engineering actually is.
| Domain | What the platform holds | The join that is hard |
|---|---|---|
| Discovery research | Assay results, compounds, screening output, reports | Compound identity across series, salts, batches and internal naming |
| Clinical development | Study data, operational data, documents, outcomes | Subject identity across systems, with blinding enforced throughout |
| CRO operations | Many sponsors, many studies, many conventions | Client separation that holds at query time, per study and per user |
| Pharmacovigilance | Cases, narratives, products, literature | Product identity across markets, formulations and naming conventions |
| Quality and manufacturing | Batches, deviations, specifications, process data | Batch to deviation to specification version in force at the time |
| Laboratory operations | Instruments, samples, methods, results | Sample identity through preparation, aliquoting and re-testing |
| Commercial and supply | Demand, distribution, inventory, market data | Product and customer identity across commercial and operational systems |
The storage, compute and pipeline layers are mature, well supported and broadly interchangeable between credible platforms. The semantic layer is the one nobody can sell you, because it encodes your entities, your science and your obligations, and it is where a platform programme succeeds or fails.
| Layer | What it does | How much is a product choice |
|---|---|---|
| Storage and compute | Hold the data, run the processing | Almost entirely. Fabric, Databricks, Snowflake and the cloud platforms all work |
| Ingestion and pipelines | Move and transform, batch and streaming | Largely. Tooling is mature and the patterns are well established |
| Semantics and entities | Make data from different sources mean the same thing | None. This is yours, and it is the whole job |
| Access and governance | Enforce who may see what, and prove it | Partly. Products supply mechanisms, your obligations supply the model |
That distribution explains a common disappointment. A platform selection exercise runs for months, concludes carefully, and the organization is no closer to answering its cross-system question, because the decision it agonised over was in the layer where the options were all adequate. The decision that mattered, which is how entities resolve and what context travels with each record, was never framed as a decision at all.
Nobody owns a pipeline after it works.
A pipeline is built, tested, deployed and celebrated, and then it runs unattended for two years until a source system changes a field and it quietly starts delivering less than it should. Downstream, fewer records looks like a flat trend rather than a failure, and nothing alerts anybody because the job completed successfully. Every pipeline needs a named owner, freshness and volume monitoring with alerting on absence rather than on error, and a reconciliation against source on a schedule. That is a small amount of engineering per pipeline and it is the difference between a platform people trust and one they check against the source system before using.
This is the asymmetry that makes platform work in this industry different. A source system carries the obligations attaching to its own data. A platform that consolidates many sources inherits all of them simultaneously, and the strictest requirement applying to any dataset tends to govern the design. That is manageable when it is understood at the start and extremely difficult to retrofit, which is why the classification of what will be held belongs in the architecture phase rather than in a later governance workstream.
Frameworks where they apply
- GxP and GAMP 5 where the platform supports regulated activity, with assurance proportionate to what the data does
- 21 CFR Part 11 and EU Annex 11 where records and signatures held in the platform fall within scope
- ICH GCP for clinical data, with blinding and sponsor obligations enforced by the platform rather than around it
- CDISC where submission data is prepared, and FAIR as principles the internal model should satisfy
Data integrity
- ALCOA+ applied to data in the platform as it would be in the source, since consolidation does not dilute the obligation
- Lineage from any value back through transformation to the originating record
- Audit history carried through migration, not left behind in the system being retired
- Controlled correction, where a restated value preserves what it replaced and why
Privacy and access
- Personal and health data identified at ingestion, since obligations follow the data rather than the system
- HIPAA and GDPR as your counsel determines them, including for processing and for any third-party component
- Consent scope recorded with the data, so a secondary use can be checked rather than assumed
- Entitlement enforced per user at query time, covering blinding, separation and jurisdiction
Validation
- Scoped to what the platform carries, since regulated and exploratory data are different assurance problems
- Pipeline logic validated against worked examples, because a correct platform running a wrong transformation is consistent and wrong
- Migration verified at record and relationship level rather than at reconciled totals
- Re-validation triggered by schema and source changes, which happen whether or not anybody plans for them
Classify what the platform will hold before you design how it holds it.
The design consequences of holding regulated clinical data, personal health data, blinded study data and commercially confidential partner data are different from each other and different again from holding exploratory research data. A platform designed for the easiest of those and later asked to accommodate the hardest will be rebuilt rather than extended. The classification exercise takes days, requires your quality, regulatory and legal functions in one room, and it is the cheapest architectural decision in the programme because it is the one that constrains everything else.
A data platform rarely stops working in a way anybody notices. It gets slightly less complete, slightly less current and slightly less trusted over a period of months, and the point at which people start checking figures against the source system before using them is the point at which it has failed, long after the failure began.
| Failure | How it presents |
|---|---|
| A source adds or renames a field | The pipeline succeeds, the new data is silently absent, and totals look plausible |
| A pipeline delivers fewer records | The job completed, the volume dropped, and downstream it reads as a flat trend |
| Reference data changes externally | A code list or annotation moves and historical results change with no code change |
| Entity resolution degrades | New identifiers arrive unmatched and the join quietly covers less of the population |
| A permission change is not reflected | Someone retains access the source system revoked, or loses access they need |
| Cost grows faster than usage | A query pattern or retention policy changes and finance raises it before engineering does |
| Nobody owns a dataset | Questions about it have no addressee and corrections are nobody’s responsibility |
| Lineage breaks at a hand-built step | One manual transformation in the middle and the chain of provenance ends there |
Platform programmes are reported in sources connected and terabytes ingested. The outcomes that matter are whether people can answer questions they could not answer before, whether they trust the answer enough to act on it, and whether the platform is still current in two years.
| Category | What we measure | Why it matters |
|---|---|---|
| Cross-system questions answerable | Questions requiring a join that can now be answered, and how quickly | The only outcome that distinguishes a platform from a copy |
| Entity match rate | Proportion of records resolving to a known entity, by domain, trended | Every figure built on a join inherits this number |
| Context completeness | Share of records carrying the conditions needed to interpret them | Determines whether the data supports a second question at all |
| Trust behaviour | How often users verify a platform figure against the source system | The honest measure of whether the platform is believed |
| Pipeline freshness and completeness | Data arriving on time and in expected volume, by source | Degradation is silent and this is where it shows first |
| Lineage coverage | Proportion of published figures traceable to source records | What converts a number into something somebody can defend |
| Cost per workload | Running cost attributed to the teams and use cases generating it | Unattributed cost grows until it becomes a programme risk |
We will start from the questions that cross your systems, establish the entity model and the context that has to travel with each record, settle the permission model before the architecture, migrate with provenance intact, and leave you with a platform measured by what people can now answer.
CaliberFocus builds life sciences data platforms where the context that makes scientific data reusable survives the journey into them. We start from the questions that cross your systems, resolve entities and preserve experimental and clinical context, settle the permission model before the architecture, migrate with provenance intact, and measure the platform by what people can now answer.
- Semantic layer of resolved entities, units, terminology and context
- Entitlement enforced per user at query time for blinding, consent and separation
- Migration that carries lineage and audit history, not only current values
- Pipelines instrumented to alert on absent data, not only failed jobs
Security & Compliance
