That is the actual problem, and it is not a modelling problem. A forecast is a set of beliefs about the future expressed as a number: beliefs about uptake, about coverage, about a competitor, about how long a trend holds. When the number misses and the beliefs were never written down, the organization learns nothing and builds the next forecast the same way. Forecasts improve when assumptions are documented, owned, versioned and tested against what actually happened.
We build forecasting and statistical modelling for pharmaceutical, biotechnology, research and CRO organizations across the range the work actually covers: demand and launch forecasting, time series and regression work, survival and multivariate analysis, scenario and simulation modelling, and the validation and monitoring that keeps a model usable after it is deployed. The statistical methods are well established and mostly not where programmes fail. They fail on assumptions nobody recorded, on models fitted across structural breaks, on point estimates presented without uncertainty, and on confusion about which of the three different things called validation a given model actually needs.
Most organizations respond to a bad forecast by improving the method. A better algorithm, more variables, a machine learning approach, an external consultant with a more impressive technique. Sometimes that helps. Far more often the forecast missed because somebody assumed a formulary position would hold, or that a competitor would launch in the second half, or that uptake would follow an analogue product whose market differed in a way that mattered, and none of those beliefs was written down anywhere that the post-mortem could find.
The second recurring problem is structural. Life Sciences time series are interrupted constantly: a patent expires, a competitor enters, a guideline changes, a label expands, a payer changes coverage, supply is interrupted. Each is a break, and a model fitted across one has learned the behaviour of a market that no longer exists. Launch forecasting is the extreme case, with no history at all and analogues chosen partly on availability. None of this argues for forecasting less. It argues for being explicit about what the model assumes and where its evidence runs out.
The assumptions were never written down
So when the forecast misses, the organization cannot identify what it got wrong.
The model was fitted across a structural break
And has learned the behaviour of a market that ended two years ago.
A point estimate was published without an interval
Which turns a forecast into a commitment and removes the information the planner needed.
Historical data is fragmented and inconsistently defined
So the series being modelled is partly an artefact of how it was assembled.
Two statistical cultures are using the same word differently
Pre-specified clinical analysis and iterative commercial fitting have opposite governance.
Validation means three different things
Statistical validity, system validation and backtesting are separate obligations and get conflated.
Predictive accuracy gets read as causation
The model predicts the outcome from variables nobody established would change it.
The model was deployed and then left alone
Drift is silent, and a model degrading slowly outperforms nothing right up until it does not.
Take last year’s forecast and ask what it believed. Find a forecast your organization issued twelve to eighteen months ago, retrieve the version that was actually issued rather than the one that has since been revised, and try to list the assumptions it rested on: uptake curve, coverage position, competitor timing, pricing, channel behaviour, capacity. Then compare each assumption with what happened. In most organizations this exercise cannot be completed, because the assumptions were held in a conversation and the forecast was edited in place. That inability is the finding, and it is the single highest-return thing to fix before any method changes.
Two steps in this sequence are the ones usually skipped, and they are the first two. Classifying the work by what its output will be used to claim determines the governance for everything that follows. Writing the assumptions down before fitting anything determines whether the forecast can be learned from. The modelling itself is the part the team is already good at.
Step 1
Classify the work by intended use
Step 2
Write the assumptions down first
Step 3
Assess and prepare the data honestly
Step 4
Explore before specifying
Step 5
Specify the model and test its assumptions
Step 6
Validate against held-out reality
Step 7
Quantify uncertainty and name its sources
Step 8
Simulate where the question is about risk
Step 9
Deploy with versioning and reproducibility
Step 10
Monitor for drift and re-examine assumptions
The model is not the deliverable. The decomposition is. Ask what the forecasting function hands over. If it is a number, or a number with a scenario range built by moving the number up and down, the organization cannot learn from it. What it should hand over is the forecast, the assumption register behind it, the sensitivity of the output to each assumption, the interval with its sources of uncertainty named, and the record of how the previous forecast performed against each of its assumptions. That package takes marginally longer to produce and is the difference between a forecasting function that gets better every cycle and one that re-argues the same number every year.
Organized by question rather than by technique, because the choice of method follows from the question and the available data rather than from preference. Most engagements use a subset, and the simpler method with documented assumptions usually beats the sophisticated one without them.
What is happening, and is it real
Statistical analysis, hypothesis testing and estimation with assumptions tested and reported
Regression analysis, including model specification and diagnostic checking
Multivariate analysis where predictors are correlated, with stability assessed
Survival and time-to-event analysis, with the proportional hazards assumption examined
Clinical and research statistics supporting study and research questions
What happens next
Time-series forecasting with structural breaks identified and handled explicitly
Demand forecasting separating consumption from shipment and channel inventory movement
Launch forecasting built on stated analogue logic and documented uptake assumptions
Predictive modelling with the limits of the training data recorded
Hierarchical and reconciled forecasts where product, geography and channel views must agree
How wrong could we be
Uncertainty quantification with parameter, model and scenario sources separated
Confidence and prediction intervals produced and interpreted correctly
Scenario modelling built around decisions rather than around arbitrary high and low cases
Monte Carlo simulation with inherited input assumptions stated on the output
Risk modelling and sensitivity analysis ranking assumptions by their effect on the answer
Is it still working
Forecast accuracy measurement against issued forecasts, by product and by horizon
Backtesting on data the model never saw, over the horizon that matters
Model validation scoped to intended use, with the right meaning of validation applied
Drift detection, recalibration triggers and documented change control
Decision support that states what the model is evidence of and where it should not be used
The technique often transfers between these and the governance does not. A regression supporting a research hypothesis and a regression supporting a supply decision can be the same model and carry completely different documentation, validation and review obligations. Classify first.
| Area | The question | What the statistics has to get right |
|---|---|---|
| Pharmaceutical R&D | Which candidates and designs justify continued investment | Multiplicity, effect size realism, and the difference between exploratory and confirmatory |
| Biotechnology research | What the experimental data supports | Small samples, batch effects, and replication treated as a requirement rather than an aspiration |
| Drug discovery | Which signals in high-dimensional data are worth pursuing | False discovery control, because testing many hypotheses guarantees interesting noise |
| Clinical trials | Whether the pre-specified analysis supports the claim | Conformance to the statistical analysis plan, with any deviation documented as such |
| Clinical development | How to size, power and sequence studies | Assumptions about variance and effect that are stated and sourced, not inherited |
| CRO operations | Whether delivery and resourcing are on plan across a portfolio | Operational forecasting by study and sponsor with contractual separation maintained |
| Epidemiology | What the population-level pattern is | Confounding, selection, and the limits of the data source behind the estimate |
| Real-world evidence | What observational data can and cannot establish | Design before analysis, and a stated position on unmeasured confounding |
| Pharmacovigilance | Whether a safety signal is present | That the data measures reporting behaviour, which is not the same as incidence |
| Product launch forecasting | What uptake to plan for with no history | Analogue selection logic stated and defensible, and uptake assumptions owned by name |
| Commercial demand planning | What demand to plan production and inventory against | Consumption separated from shipment, and channel inventory movement identified |
| Market access | How coverage changes affect expected volume | Coverage treated as a modelled driver rather than a constant |
| Manufacturing | Where process variation threatens output or quality | Control limits derived from the process rather than from specification convenience |
| Supply chain and inventory | What to hold, where, against what service level | Forecast error distribution by horizon, because safety stock is priced off the error |
Statistical work has an engineering requirement that most analytics work does not: somebody must be able to regenerate a specific result, from the same data, with the same code, in the same environment, possibly years later, possibly after the author has left. That single requirement determines most of the architecture.
| Capability | Why it is not optional |
|---|---|
| Data snapshotting | A result is tied to the data as it stood, and source systems change underneath you |
| Environment pinning | A library version change can alter a numerical result without any code change |
| Seed and randomness control | Simulation and resampling results must be regenerable exactly |
| Code version control | The analysis that produced a filed or decided-upon number must be identifiable |
| Parameter and assumption capture | The inputs are part of the result and belong with it, not in a separate deck |
| Model registry and versioning | Which version was in use when a given forecast was issued has to be answerable |
| Actual outcome capture | Performance cannot be measured unless actuals are wired back and stored against the forecast |
| Execution logging | Who ran what, when, against which data, for the runs that mattered |
Pick a number somebody acted on and try to regenerate it. Choose a forecast or an analysis result that drove a real decision six months ago and attempt to reproduce it exactly: same data as it stood, same code version, same parameters, same environment. The common outcome is that the number cannot be reproduced, usually because the data has moved, the script was edited, or the parameters lived in somebody’s session. If a result cannot be regenerated, it cannot be audited, it cannot be compared with the next version, and the organization cannot establish whether its models are improving. This is an engineering fix, it is well understood, and it is worth doing before any methodological ambition.
This is the most consequential confusion in statistical work in regulated organizations, and it produces both over-engineering and exposure. Three separate obligations share one word, and which of them applies depends on what the output will be used for. Your quality, regulatory and biostatistics functions make that determination. Our role is to build so that whichever applies is satisfiable.
Frameworks where they apply
- GxP and GAMP 5 where a statistical or forecasting system supports a regulated activity
- FDA 21 CFR Part 11 and EU Annex 11 for electronic records and signatures where applicable
- ICH Good Clinical Practice principles for clinical research activity
- CDISC structures and conventions where submission data is involved
Evidence
- ALCOA plus as the standard the analysis record is held to where it applies
- Statistical analysis plans pre-specified and version controlled
- Audit trails over data, code, parameters and model versions
- Deviations documented as deviations rather than absorbed into the method
Privacy
- HIPAA where protected health information is processed
- GDPR and equivalent regimes for personal data in modelling datasets
- De-identification and minimization applied to the modelling dataset, not only to the source
- Re-identification risk assessed before linking sources for a model
Model governance
- Named owner, documented assumptions, intended use and stated limits per model
- Reproducibility as an enforced property: data snapshot, code version, environment, seed
- Explainability appropriate to the decision the model informs
- Change control and controlled promotion for models that feed regulated or material decisions
Models do not fail by stopping. They keep returning plausible values while the relationship they learned weakens, the input distribution shifts, a competitor changes the market or a data source alters its definitions. Unmonitored, a model is trusted for exactly as long as nobody checks, and the check usually happens after a decision went badly.
| Failure | How it presents |
|---|---|
| The backtest leaked future information | Historical accuracy looks excellent and live accuracy does not match it |
| The model was retuned on the test set | Every candidate improved, and the reported performance is optimistic by an unknown margin |
| A structural break occurred after training | Errors grow in one direction while individual predictions stay plausible |
| Input distributions drifted | The model is extrapolating beyond its training range without saying so |
| Intervals are too narrow | Actuals fall outside the stated interval far more often than the interval implies |
| An assumption silently expired | A coverage or capacity belief changed and the forecast still carries the old one |
| A library or environment changed | Numbers moved and no code was modified |
| The forecast was edited in place | Accuracy cannot be measured because the issued version no longer exists |
| Aggregate accuracy hides a bad subgroup | The overall figure is acceptable while one product, site or region is consistently wrong |
| A recalibration introduced a new error | The new version was promoted without being compared against the one it replaced |
Forecasting functions are usually measured on accuracy, which is partly outside their control. The better test is whether the organization can decompose a miss into the assumption that was wrong, and whether that changed anything about the next cycle.
| Outcome | How to tell |
|---|---|
| Forecast accuracy improved | Error by product and horizon is measured against issued forecasts and trending down |
| Uncertainty is quantified | Intervals are published, their coverage is checked, and planning uses the spread |
| Misses are explainable | A missed forecast is attributed to a named assumption rather than to the model |
| Scientific decisions better supported | Analyses state their assumptions, their diagnostics and the limits of their evidence |
| Resource planning optimized | Safety stock and capacity are priced off a measured error distribution, not a buffer |
| Operational risk reduced | Scenario and simulation work gives decision-relevant probabilities, with inputs documented |
| Decisions are evidence-based in practice | Every material model has an owner, a stated intended use and documented limits |
We work with pharmaceutical, biotechnology, research and CRO organizations on forecasting and statistical modelling: assumptions documented and owned, structural breaks handled explicitly, uncertainty quantified with its sources named, results reproducible, and validation matched to what the output will actually be used to claim.
CaliberFocus builds forecasting and statistical models for pharmaceutical, biotechnology, research and CRO organizations that can be learned from and defended. We write the assumptions down before fitting anything, handle structural breaks explicitly, quantify uncertainty with its sources named, make every result reproducible, and match validation to what the output will actually be used to claim.
- Assumption registers with a named owner for every belief
- Honest backtesting against simple baselines and held-out data
- Intervals published and their coverage checked against actuals
- Reproducible results with data, code, environment and seed captured
Security & Compliance
