Project Risk Lab

Each reference class is the empirical distribution of the outcome ratio (actual divided by estimate) for many completed projects of that type, reported at five percentiles P5, P25, P50, P75, P95 from large-scale studies. Because ratios are positive and right-skewed, the tool works in log space. It fits a monotone cubic (Fritsch-Carlson) spline through the five log-percentiles, so the curve passes exactly through the data and never crosses itself. Multiplying your point estimate by the ratio at a chosen percentile gives the reference-class forecast at that confidence level:

The parametric cousin, used by the Noise Audit and the other selection tools, is the log-normal case \( \ln(\text{actual}/\text{estimate})\sim\mathcal{N}(\mu_0,\sigma_0^{2}) \), for which \( q_P = e^{\mu_0+\sigma_0\Phi^{-1}(P)} \) with \( \sigma_0=(\ln q_{90}-\ln q_{10})/(2\times 1.2816) \).

Your drawing is resampled into a curve with one height per position on the ratio axis, so wherever your pointer passes, the curve follows. Treating the heights as relative likelihoods gives a cumulative version of your sketch, and the median and P90 are read off where it crosses 50% and 90%. The data curve is the same monotone spline through the class's five reported percentiles that the Reference Class Forecasting tool uses, converted to a density on the same axis. On reveal, both curves are rescaled to the same peak height, so compare positions and shapes rather than areas. Heavy-tailed classes place part of their probability past the right edge of the drawing area, and the verdict reports how much.

The Gauntlet scores each round with the Winkler interval score at the 90% level, the same proper scoring rule the Calibration Quiz uses, normalised by the true ratio so rounds are comparable. Lower is better. The five classes and the five outcomes are seeded from the UTC date, so everyone who plays on the same day faces the same puzzle. This tool stores nothing and sends nothing itself.

Cost and schedule strips sit on a logarithmic scale, since overrun ratios are right-skewed and a log scale keeps a doubling the same width everywhere. For cost and schedule, everything right of the 1.0 line is an overrun, and the long right tails are where projects blow up. Benefit ratios read the other way. A ratio below 1.0 is a shortfall, so the bad tail is on the left, and the benefit view uses a linear scale because one class (IT) reaches a negative P5. The Reference Class Forecasting tool mirrors its benefit axis for the same reason, so a higher percentile always reads as a worse outcome there.

Sorting by median ranks classes by the typical outcome, sorting by tail ranks them by the bad end of the range (P95 for cost and schedule, P5 for benefits), and sorting by sample size puts the best-evidenced classes first. Worst is at the top in every sort except sample size.

You're deciding whether to proceed with a project. The decision hinges on an uncertain variable, such as expected revenue, NPV, or cost savings. If the true value falls below your threshold, proceeding would be a mistake. EVPI quantifies the maximum you should pay to resolve that uncertainty before committing.

The tool treats your 90% confidence interval \([l_{90}, u_{90}]\) as a normal distribution with \(\sigma = (u_{90} - l_{90}) / (2 \times 1.645)\), then values information as the gap between deciding with perfect information and deciding now:

\[ \mathrm{EVPI} = \mathbb{E}_{x}\!\left[\max_a u(a, x)\right] - \max_a \mathbb{E}_{x}\!\left[u(a, x)\right] \]

For a normal variable and a proceed-if-above-threshold \(T\) decision, this integral has a closed form, the expected opportunity loss from acting under uncertainty:

where \(L\) is the unit-normal loss function. It is largest when the estimate sits right at the threshold (the decision is a coin toss) and shrinks to zero as \(\mu\) moves far from \(T\). You should never pay more than the EVPI to measure the variable. Running it with a smaller \(\sigma\) gives the value of a partial measurement (the EVI): if a study cuts the standard deviation to \(\sigma'\), \( \mathrm{EVI} = \mathrm{EVPI}(\sigma) - \mathrm{EVPI}(\sigma') \), and a lower \(\sigma'\) can only help (Blackwell), so its net value is \( \mathrm{EVI} - \text{cost} \), which the comparison panel below ranks.

EVPI tells you the most you should pay to remove all uncertainty. In practice the choice is rarely perfect information versus none. It is usually which study to buy, a cheap survey that cuts uncertainty a little, or an expensive one that cuts it a lot. The panel below values each at the same decision and threshold.

Each study is modelled as reducing your standard deviation by a given percentage (the same way the EVI above models halving uncertainty). Its EVI is how much that reduction lowers your expected cost of a wrong call. Its net value is EVI minus the study's price. The better buy is the one with the higher net value, and the cheaper study can win even when it is less informative.

This is the practitioner's face of Blackwell's theorem. A more informative signal is worth at least as much in every decision problem. For measurements of the form "estimate = truth + noise," less noise is more informative in Lehmann's precise sense. The full garbling apparatus is not needed to make the call that matters here, which signal is worth more, and whether the cheaper one is nearly as good.

Sources: de Oliveira, H. (2018). Blackwell's informativeness theorem using diagrams. Games and Economic Behavior 109, 126–131. · Lehmann, E. L. (1988). Comparing Location Experiments. Annals of Statistics 16(2), 521–533.

The model’s key mechanism is the budget constraint acting as a selection filter. Organisations cannot fund every project, so they rank proposals by estimated benefit-cost ratio (BCR) and fund the top fraction (the budget share α). Among funded projects, costs systematically exceed estimates because the project was selected partly because noise made its costs look lower than they were. This is the optimizer’s curse applied to project portfolios.

Two parameters govern the effect: σ (how noisy the estimates are) and α (how selective the budget is). Tighter budgets (lower α) force selection from the extreme tail of the estimated BCR distribution, amplifying the overrun. When α = 1 (everything funded), there is no selection effect and only the small Jensen convexity term remains.

For equal cost and benefit noise, the expected realised-to-estimated cost ratio among funded projects has a closed form:

\[ \mathbb{E}\!\left[\frac{C}{\hat{C}} \,\middle|\, \text{selected}\right] = \frac{1}{\alpha}\,e^{\sigma^2/2}\,\Phi\!\left(\frac{\sigma}{\sqrt{2}} - \Phi^{-1}(1-\alpha)\right) \]

where \(\Phi\) is the standard normal CDF. At \(\alpha = 1\) this collapses to the pure Jensen term \(e^{\sigma^2/2}\). Tighter budgets multiply it by the selection factor on the right.

With unequal cost and benefit noise the same result reads (this tool uses the equal-noise case above; the Selection-Adjusted RCF tool computes this general form):

The ratio exceeds one because selection truncates the joint error distribution, giving funded projects a negative average cost error:

The benefit-shortfall mode is the mirror image: \( \mathbb{E}[B/\hat B\mid\text{sel}] = (1/\alpha)\,e^{\sigma_b^{2}/2}\,\Phi(-\sigma_b^{2}/\sigma_\eta - z^{*}) \). For selective budgets this sits below one. Near \(\alpha = 1\) the Jensen term dominates and the ratio can edge above one, though the selection factor itself stays below one for every \(\alpha < 1\). Closed forms from Leed (2026), Noise, Selection, and the Illusion of Bias.

When organisations rank projects by estimated benefit-cost ratio (BCR) and fund the top fraction α, projects with accidentally underestimated costs are more likely to be selected. The observed overrun in the reference class therefore reflects both true estimation bias and this statistical selection effect (the optimizer’s curse). A uniform uplift calibrated to the full observed overrun is ranking-neutral and budgets the funded portfolio correctly, but it overstates the true bias by the selection share. It raises value-for-money and BCR-above-one approval bars too far, and it distorts comparisons across categories whose selection shares differ. The bias-only component is the part to apply to an individual estimate. The selection part belongs in a portfolio-level reserve.

This tool decomposes the observed overrun into a selection component (teal) and a residual bias component (gold). The observed reference-class overrun factors into a Jensen term, a selection term, and true bias:

Writing \(S\) for the selection term alone (the middle factor above; the Jensen term is kept out of it because convexity is a property of noise, not selection, and recurs for any new project), the tool reports the selection share as the levels ratio \( (S-1)/(O-1) \) and removes that share from the observed mean overrun, leaving the rest (true bias plus the Jensen term) as the per-project uplift. The benefit-shortfall mode uses the mirror selection factor \( S^{b}=[1-\Phi(z^{*}+\sigma_b^{2}/\sigma_\eta)]/[1-\Phi(z^{*})] \le 1 \).

All reference classes are conditional. When you pick "IT cost overruns" as your reference class, you've already conditioned on project type. The term Conditional Reference Class Forecasting adds a further condition. It asks how far along the project is and what actual performance looks like so far. At 0% progress, you have only the unconditional distribution. At 50% progress with an observed cost ratio of 1.15, you combine that evidence with the reference class to produce a tighter, more informed forecast.

Under the hood this is a conjugate update in log space. With the reference-class log-median \(\mu_0\), its log-spread \(\sigma_0\), an observed log-ratio \(y = \ln(\text{actual} / \text{estimate})\), and fraction complete \(p\), the posterior median and spread are:

\[ \mu_{\text{post}} = (1-p)\,\mu_0 + p\,y, \qquad \sigma_{\text{post}} = \sigma_0\sqrt{1-p} \]

At \(p = 0\) the posterior is the reference class. As \(p \to 1\) it collapses onto what you have observed. The forecast shifts toward the evidence with weight equal to progress, and its spread contracts as the project completes. A sequence of such Bayesian updates is itself a martingale, a property you can check in your own revision history with the Martingale Test.

The parameters α (alpha) and β (beta) shape the distribution. Think of them as pseudo-counts. α = prior "hits" + 1 and β = prior "misses" + 1. Start with α=1, β=1 for a flat (uninformative) prior. Larger values indicate stronger prior belief, meaning a sharper peak and less spread. When you observe new data (hits and misses), the posterior is Beta(α + hits, β + misses).

With \(s\) hits in \(n\) trials, the posterior and its mean are:

\[ p \mid \text{data} \;\sim\; \mathrm{Beta}(\alpha + s,\ \beta + n - s), \qquad \mathbb{E}[p] = \frac{\alpha + s}{\alpha + \beta + n} \]

Two forces compete. Cost decline rewards waiting (technology gets cheaper), while the discount rate and finite benefit window penalize delay (future benefits are worth less, and the planning horizon shrinks). The model discounts all cash flows to year 0, so you can compare adopt-now vs. adopt-later on equal footing. The optimal year balances the savings from cheaper technology against the benefits forgone by waiting.

Risk matrices are ubiquitous in project management, but research shows they suffer from at least three fatal flaws: (1) verbal probability labels mean vastly different things to different people, (2) colour-coded cells hide enormous differences in expected loss, and (3) arbitrary category boundaries can reverse the ranking of risks. This interactive challenge lets you experience all three.

Five trades (Civil, Mechanical, E&I, Fit-out, Commissioning) must each process 35 rooms in sequence. Each trade rolls a die each week to determine capacity, but can only process rooms the previous trade has already completed. The deterministic plan, using the average roll of 3.5 rooms/week, expects completion in 14 weeks. You will play three rounds with dice of identical averages but different variability to see how uncertainty amplifies through the system.

This demonstrates a general principle. When tasks must happen in sequence and each step has variable output, the total project duration is longer than the sum of averages on average, and usually in any single run. Planning with averages sets you up to be late.

Each candidate has a true cost and a true benefit fixed when the deck is dealt. What you see is an estimate, the true cost divided by a noise draw from the project's reference class in Hubbard, Budzier & Leed (2025), with each class rescaled so its median ratio is 1.0. The rescaling removes shared bias, so the typical estimate lands on the truth, half come in high and half low. Because the rescaled errors stay right-skewed, even an unselected portfolio would overrun by a couple of percent. Selection multiplies that. Benefits are shown at their true value to keep the mechanism clean. Approving on the benefit-cost ratio still favours the flattering cost errors, which is the whole game.

Round two replays the identical projects and the identical outcomes. The only change is round one's average uplift, every estimate multiplied by the average outcome ratio of your round-one approvals, before you commit the same budget again. Comparing the two rounds shows what the uplift buys, fewer approvals and a committed total that predicts the final bill.

Sources: Hubbard, D. W., Budzier, A., & Leed, A. B. (2025). How to Measure Anything in Project Management, Wiley, Appendix 2, Table A2.1. · Leed, A. B. (2026). Noise, Selection, and the Illusion of Bias, working paper.

This data comes from a survey conducted by Wade Fagen-Ulmschneider (Teaching Professor, Computer Science, University of Illinois at Urbana-Champaign). Each of the 123 internet survey respondents was asked to assign a numeric probability (0–100%) to 17 common probability words. The box-and-whisker chart below shows the distribution of responses for each word. The box spans the interquartile range (Q1–Q3, the middle 50% of responses), the vertical line marks the median, and the whiskers extend to the minimum and maximum values observed. The study builds on a long tradition of research into verbal probability, notably Sherman Kent’s 1964 “Words of Estimative Probability” for the CIA.

Enter the forecast made at each review or milestone (total cost, percent over baseline, or forecast completion in weeks). The tool measures each revision, tests whether the revisions point in one direction more often than coin flips would explain, and tells you whether the path looks like new information or the slow correction of a biased anchor. It needs no realized outcome.

Let F_t be the forecast made at review t = 0, 1, 2, … (expected final cost, or P50 completion). If F_t is the conditional expectation of the true final value given everything known at t, that is, the forecaster updates like a Bayesian, then the sequence {F_t} is a martingale:

\[ \mathbb{E}\!\left[\,F_{t+1} \mid \mathcal{F}_t\,\right] = F_t \]

Two consequences follow. The tool checks the first directly. The second is the theory behind why drift matters.

1. Revisions are mean-zero and directionally unpredictable. The increments \(\Delta_t = F_t - F_{t-1}\) have conditional mean zero. Persistent one-signed drift means the forecaster was not using all available information at the start. The anchor was biased. Upward drift in cost is optimism in the initial estimate.

2. Movement variance equals uncertainty reduction. For a rational updater (Augenblick & Rabin 2021), the total variance of belief movement equals the total reduction in uncertainty. Both excess movement (over-reaction) and insufficient movement (under-reaction, anchoring) show up in the revision path.

The drift test here is an exact two-sided sign test. With m non-zero revisions of which k are upward, it reports the probability that coin-flip revisions would be at least this lopsided (binomial, p = 0.5). The momentum readout is the share of consecutive revisions that share a sign, a light check for anchoring-and-adjustment.

Read it with care. In practice a forecast is a true martingale component plus incoming news, and a single project's path is short. So drift is a signal to investigate rather than proof of bias. The test becomes powerful across a portfolio. Drift that repeats across many projects is the clearest indictment of systematic optimism.

This audits the path of updates. Use the Conditional RCF tool to generate the next update, and the Calibration Quiz to work on the anchor itself.

Sources: Augenblick, N. & Rabin, M. (2021). Belief Movement, Uncertainty Reduction, and Rational Updating. Quarterly Journal of Economics 136(2), 933–985. · Cross-reference Hubbard, Budzier & Leed, How to Measure Anything in Project Management (Wiley, 2025), and the working paper Noise, Selection, and the Illusion of Bias.

Set the nominal size of your reference class and how much estimators copy a shared benchmark rather than estimate independently. The tool computes the effective sample size and shows how much your reported uncertainty understates the truth.

If each of N estimates were an independent signal of the true value, the variance of their average would fall like σ²/N. If a fraction of estimators instead copy, so the estimates share a common component with intra-class correlation ρ, that variance is inflated by the design effect:

Dividing N by the design effect gives the effective sample size, the number of independent observations, which is bounded no matter how large N grows:

As ρ → 0, N_eff → N (independent); as ρ grows it collapses toward 1/ρ. Reported uncertainty, which assumes independence, is too small by a factor \( \widehat{\mathrm{SE}}_{\text{naive}}/\mathrm{SE}_{\text{true}} = \sqrt{N / N_{\mathrm{eff}}} = \sqrt{1 + (N-1)\rho} \).

This tool models the spread, how copying inflates apparent confidence. Copying can also bias the level, a related phenomenon it does not simulate. In a sequential cascade (Bikhchandani, Hirshleifer & Welch 1992; Banerjee 1992), once a few early estimates point the same way, later estimators rationally ignore their own signal and follow. With bounded private beliefs (Smith & Sørensen 2000), the herd can lock onto the wrong value with positive probability, and information stops accumulating once the cascade forms. An optimistic benchmark need not be corrected by the crowd.

RCF, the Noise Audit, and Selection-Adjusted RCF all assume independent reference observations. If the database is a herd, their uplift bands are over-confident. The raw count overstates the independent evidence.

Caveat. ρ is not directly observable. This is a sensitivity and awareness device. If estimators copy even moderately, here is how over-confident your bands become. Estimating ρ from project-filing data would be a worthwhile research product. This contamination (behavioural correlation among estimators) is a different mechanism from the mechanical selection on noise in Selection-Adjusted RCF. The two are complementary stories for why a reference class can mislead.

Sources: Smith, L. & Sørensen, P. N. (2000). Pathological Outcomes of Observational Learning. Econometrica 68(2), 371–398. · Bikhchandani, S., Hirshleifer, D. & Welch, I. (1992). A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades. Journal of Political Economy 100(5), 992–1026. · Banerjee, A. V. (1992). A Simple Model of Herd Behavior. Quarterly Journal of Economics 107(3), 797–817. · Cross-reference Leed, Noise, Selection, and the Illusion of Bias.

Each point is a project. The horizontal axis is a characteristic x, the vertical axis its true outcome y. A project is funded when a selection index (which depends on x, an exclusion variable w, and noise correlated with the outcome) clears zero. You only see y for the funded (solid) points. The rest are dropped (hollow). The grey dashed line is the truth, the coral line is a naive regression on funded projects only, and the teal line is the Heckman-corrected fit. Everything is estimated live. A probit for the selection stage, then ordinary least squares with the inverse Mills ratio added.

Heckman’s two-step estimator recovers the true slope from a selected sample. Step one fits a probit of “was it funded” on the selection variables and forms the inverse Mills ratio for every unit:

Step two regresses the outcome on the original predictors plus λ, over the funded rows only:

The coefficient on λ estimates ρσ_ε, so its sign is the sign of the selection correlation, and testing H₀: β_λ = 0 is a test for selection bias. If it is not different from zero, plain least squares on the funded sample was already fine. The naive slope that leaves λ out is biased by

which is exactly the tilt in the coral line.

The whole thing rests on an exclusion restriction, a variable that shifts who gets selected but has no direct effect on the outcome. Here that is w. Set γ₂ = 0 and λ(γ₀+γ₁x) becomes a function of x alone, so corr(x, λ) → 1 over the funded range, the two regressors collide, and the corrected estimate swings from sample to sample. That fragility is why the exclusion restriction matters. Try it. Set the exclusion strength to zero and press “New sample” a few times.

Source: Heckman, J. J. (1979). Sample Selection Bias as a Specification Error. Econometrica 47(1), 153–161.

In the noise-and-selection paper the same machinery recovers the true cost bias β_c from funded projects. A project is funded when its estimated benefit-cost ratio ranks in the top α share, which reduces to the composite noise η = ε_b − ε_c clearing a threshold. Because the cost error and η are negatively correlated (Cov = −σ_c²), funded projects had systematically underestimated costs, so their overruns look like optimism even when no bias exists. The average cost error among funded projects is

Regress overruns on funded-project data and leave λ out, and the intercept is pulled up by exactly that amount:

Put λ back in as a regressor and it nets out the selection-induced mean, leaving the true β_c. That second term is exactly the overrun that Comparative Statics plots, whose leading-order form is exp(σ_c²λ(z*) / σ_η). There the inverse Mills ratio predicts the overrun, here it removes it. And where Comparative Statics shows how ρ reshapes the overrun curve, here ρ is what biases the naive regression and what the correction backs out.

Source: Leed, A. B. (2026). Noise, Selection, and the Illusion of Bias, Remark on the Heckman connection and eq. for E[ε^c | selected].

A cost-benefit analysis compares a project against the world without it and adopts it when its net present value is positive, the present value of benefits minus the present value of costs.

Nearby nature is one of the project's impacts. Valuing it at the hedonic estimate and adding it to the account gives

where \(H\) is the affected household count, \(\Delta b\) is the signed change in local bioscore, and \(v\) is the capitalised value per household per unit. \(V_{\text{bio}}\) is treated as the present value of the local amenity change applied to the affected housing stock, so it enters the account already discounted, and a degrading project with \(\Delta b < 0\) makes it a negative benefit.

A negative impact can be netted out of benefits or added to costs. The net present value is the same either way, so the decision on NPV does not turn on the choice, while the benefit-cost ratio does, which is why Boardman and co-authors call the ratio manipulable and advise deciding on net present value. This tool nets the biodiversity value into benefits and reports

with net present value as the criterion that decides the case. The breakeven readout is threshold sensitivity analysis, the value of an uncertain input at which net present value crosses zero. Holding the rest fixed, the biodiversity impact flips the decision once

with a matching household count for a given \(\Delta b\). The default figure of about DKK 56,000 is one full bioscore unit at roughly +2.8% of a DKK 2.0m home. The paper's own headline is smaller, about 1.6% per within-neighbourhood standard deviation of 0.56 units, roughly DKK 32,000, and a full unit is about 1.8 of those steps, so the per-unit figure is an extrapolation. It is a marginal slope estimated at today's allocation, so stretching it across a large \(\Delta b\) or a large household count extrapolates beyond where it was measured.

Method from Boardman, A. E., Greenberg, D. H., Vining, A. R., & Weimer, D. L. (2018). Cost-Benefit Analysis: Concepts and Practice, Cambridge University Press, on present value and net present value (Ch. 9), the benefit-cost ratio and the netting of negative impacts (Ch. 2, pp. 34–35), and threshold sensitivity analysis (Ch. 11). Measurement framing from Hubbard, D. W., Budzier, A., & Leed, A. B. (2025). How to Measure Anything in Project Management, Wiley. Hedonic figure from Leed, A. B. (in progress), The Hedonic Price of Ecological Quality: Separating Biodiversity from Greenness in Danish Housing Markets, working paper, Aarhus University.

Across the roughly 964,000 home sales, satellite greenness and each ecological-quality index barely track each other. The chart bins those sales by greenness and by the selected index, with the fitted line and the correlation.

NDVI, the normalised difference vegetation index, is built from how strongly a surface reflects near-infrared against red light. Live green leaves reflect near-infrared strongly, so NDVI rises with the amount and vigour of vegetation. It says nothing about which species are present or how rich the habitat is, and a fertilised monoculture and a diverse meadow can carry the same leaf area and read the same.

The correlations here are Pearson and Spearman between each sale's 1 km-mean NDVI, from Sentinel-2 imagery in summer 2021, and its 1 km-mean Biodiversitetskort score, over the roughly 964,000 sales in the study. The composite score correlates with greenness at a Pearson r of about 0.10 and a Spearman of about 0.04, so ranking places where people buy homes by greenness barely ranks them by ecological quality. The divergence map above shows the same thing at parish level. The species part of the score, the threat-weighted records of red-listed wildlife, is a little more related to greenness than the landscape part, and both are weak. This is the evidence behind the line on the biodiversity page that a mown lawn and a species-rich meadow look the same from a satellite.

NDVI from Sentinel-2 imagery, summer 2021. Ecological-quality scores from the Biodiversitetskort (DCE, Aarhus University). Correlations computed at the transaction level from Leed, A. B. (in progress), The Hedonic Price of Ecological Quality: Separating Biodiversity from Greenness in Danish Housing Markets, working paper, Aarhus University.

The estimate is a hedonic regression of log sale price on the mean Biodiversitetskort bioscore within 1 km, with parish-by-property-type and year-quarter fixed effects and 56 further controls (size, age, energy rating, distance to coast, station and protected nature, noise, and satellite greenness). Comparing homes of the same type within the same parish and quarter, a one-unit higher bioscore is associated with about +2.8% in price (95% CI roughly +2.0% to +3.7%; SE 0.42 pp; N = 849,288; within-R² = 0.245). The paper's headline is stated per within-neighbourhood standard deviation, 0.56 index units, worth about 1.6% of the price, roughly DKK 32,000 at the median home price of DKK 2.0 million. A full unit is about 1.8 of those steps, so the per-unit reading of about DKK 57,000 is an extrapolation the paper does not use as its appraisal figure.

The simple correlation is about three and a half times larger (about +9.9% per unit). The fixed effects remove roughly seventy percent of it, which reflects spatial sorting, as wealthier buyers cluster near nature. The association is strongest within a few hundred metres (it peaks around 500 m) and is statistically indistinguishable from zero by 5 km. It is significant for apartments (about +3.0%, p<0.01) and detached villas (about +2.2%, p<0.01), and weaker for terraced houses (about +1.3%, significant only at the 10% level). It is stable over the 2015–2025 sample.

The premium is not merely proximity to the coast. The species part of the score alone gives essentially the same premium, but that near-match is close to built in, since the species part carries most of the composite. The sharper test enters the two parts together. There the species records carry the premium at 500 m while the landscape part, which contains the coast, is not distinguishable from zero, and dropping every home within a kilometre of the coast makes the premium larger, not smaller. That points to biodiversity itself being priced, in line with survey evidence that people value species richness.

The hedonic price is the marginal value of the visible and audible landscape that a high bioscore indexes, at the current allocation. It is not the total value of biodiversity, and it does not capture non-use or ecosystem-service value.

Based on Leed, A. B. (in progress), The Hedonic Price of Ecological Quality: Separating Biodiversity from Greenness in Danish Housing Markets, working paper, Aarhus University. Ecological-quality scores from the Biodiversitetskort (DCE, Aarhus University). Preliminary findings. Associations, not causal effects.

A raw regression of price on the local bioscore mixes two things. One is the value of the nature itself. The other is that people who can pay more choose to live where the nature is better, which is spatial sorting, and it inflates the raw gradient.

Adding parish-by-property-type fixed effects compares homes of the same type within the same parish, so the parts of price that are shared across a neighbourhood, including the sorting, are held constant. What is left is the within-neighbourhood relationship between a home's local bioscore and its price. In the paper this cuts the gradient from about +9.9% per bioscore unit to about +2.8%, so roughly seventy percent of the raw number is sorting rather than nature.

The species-only version of the score, which leaves the coast and the other landscape proxies out, gives about the same within-neighbourhood figure, which points to biodiversity itself being priced rather than proximity to the sea.

Illustrative model, generated in the browser. Endpoint figures from Leed, A. B. (in progress), The Hedonic Price of Ecological Quality: Separating Biodiversity from Greenness in Danish Housing Markets, working paper, Aarhus University. Associations, not causal effects.

The chart sweeps a reference class from the tightest handful of near-twins out to the broadest comparable class and splits the error of the class average into two parts. Sampling error falls as the class grows, and heterogeneity bias rises as the class takes in less similar projects. Their sum is smallest at a width in between. A second view, the blend, stops choosing a width and instead combines a small specific class with a large general one, weighting each by how much you can trust it. You can switch the chart to it from the Go Deeper section below.

No reference class is handed to you as the right one. Venn (1876) and Reichenbach (1949) proposed the narrowest class for which reliable statistics can be compiled, and Alan Hájek (2007) points out that this never says exactly where to stop. Narrow all the way to a single project and it is unique, so the outside view disappears. Broaden without limit and the class stops resembling your project. Flyvbjerg (2013) gives the working rule. Statistical similarity is enough for statistical prediction, so you widen only as far as the projects still behave alike.

Split the error of a class average into two parts. Drawing n comparable projects estimates the class mean with sampling variance that falls as the class grows, while broadening the class admits less similar projects and adds a heterogeneity bias that grows with width:

The two pull in opposite directions, so the sum is smallest at a width in between. The most similar class is rarely that width. It has almost no bias but too few projects to pin the average down. Here u(n) runs from 0 at the tightest class of n_min projects to 1 at the broadest comparable class of N projects (in this tool, n_min is 8 and N is about 500), and the tool marks the size that minimises the sum.

Rather than pick a width, you can blend. Combine the specific class mean with the broad-pool mean, weighting each by how much it can be trusted. This is the credibility, or partial-pooling, estimate:

The tool takes this blend in log space, since overrun ratios are right-skewed, and reports the result back in levels.

A large or consistent specific class earns most of the weight (Z near 1). A tiny or noisy one leans on the general pool (Z near 0). Stein showed that shrinking three or more estimates toward a common value beats using each raw estimate, measured by total squared error, which is why blending is a discipline and not a fudge. It is the same shape as the paper's discussion of shrinking a class estimate before ranking it.

Show the blend view in the chart above

Two limits. The linear bias model is a simplifying assumption rather than a law. Heterogeneity may grow faster or in steps, and the balance point depends on how different the classes are, which can only be estimated. The lowest-error size is also capped by how many comparable projects exist. If even the broadest sensible class is smaller than the optimum, you are data-limited, and the sensible move is to use every comparable project you can find. Shrinkage lowers total error across many classes, but it is not free for any one of them. It pulls the specific estimate toward the pool, so if your class is unusual the blend is biased toward the crowd and can be worse than the raw specific number. And a large class makes the class average precise while leaving any single project unpredictable. Even with thousands of past projects, one new project still varies across the full range of outcomes, so its own interval stays wide no matter how much data you have. This tool handles sampling error and heterogeneity in the class average. It does not fix selection in the realised outcomes (see Selection-Adjusted RCF) or copying between estimators (see Reference Class Independence).

Sources: Reichenbach, H. (1949). The Theory of Probability. Univ. of California Press. · Hájek, A. (2007). The reference class problem is your problem too. Synthese 156(3), 563–585. · Flyvbjerg, B. (2013). Quality control and due diligence in project management. International Journal of Project Management 31(5), 760–774. · Bühlmann, H. (1967). Experience rating and credibility. ASTIN Bulletin 4(3), 199–207. · Efron, B. & Morris, C. (1977). Stein's paradox in statistics. Scientific American 236(5), 119–127. · Gelman, A. & Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models, Cambridge Univ. Press. · Cross-reference Leed, Noise, Selection, and the Illusion of Bias. · Reference-class means from Hubbard, Budzier & Leed (2025), How to Measure Anything in Project Management, Wiley.

The naive rate is absurd at the extremes. Five events in five trials reads as certainty, zero in seven reads as impossibility, and neither is a rate you would plan around. Put a uniform prior on the unknown rate and, after s events in n trials, the posterior is Beta(s+1, n−s+1) and the chance the next trial is an event is its mean:

This adds one imagined event and one imagined non-event, so it can never return exactly 0 or 1, and it slides back to the raw rate s/n as n grows. It is a special case of a family. Put a Beta(α, β) prior and the estimate becomes a weighted average of the data and the prior mean,

The uniform prior is add-one, or Laplace. The Jeffreys prior Beta(½, ½) is add-a-half. A stronger prior pulls harder toward its own mean. In language modelling the same device is called Laplace smoothing.

When the event has never happened in n independent trials, a different question has a clean answer. Set the chance of seeing zero events to 5 percent and solve for the rate:

The rough version is the rule of three, p up to 3/n. The exact bound 1−0.05^1/n is better when n is small. The 3 comes from a one-sided 95 percent bound. A two-sided 95 percent interval uses about 3.7 instead. This is an upper bound, not a claim that the rate is low. With seven clean trials the bound is about 0.35, so seven successes are still consistent with a failure rate near one in three.

The caveats. Laplace's own illustration was the chance the sun rises tomorrow given that it has risen every day so far, and it is the standard warning. A bare count ignores everything you know about the mechanism. The rule of succession is a Bayesian point estimate under a chosen prior, and the rule of three is a frequentist upper bound. They answer different questions and should not be merged into one number. Both assume the trials are independent draws, which for projects is an assumption, not a fact. If they share a team, a client, or a technology, treat the effective sample as smaller than n (see Reference Class Independence). This tool is the Beta Distribution tool at its flat prior. Open that tool to explore other priors.

Sources: Laplace, P. S. (1774). Mémoire sur la probabilité des causes par les évènemens. Mémoires de mathématique et de physique, Tome VI, 621–656; sunrise illustration in Essai philosophique sur les probabilités (1814). · Jeffreys, H. (1961). Theory of Probability, 3rd ed., Oxford Univ. Press. · Hanley, J. A. & Lippman-Hand, A. (1983). If nothing goes wrong, is everything all right? JAMA 249(13), 1743–1745. · Jovanovic, B. D. & Levy, P. S. (1997). A look at the rule of three. The American Statistician 51(2), 137–139. · Brown, L. D., Cai, T. T. & DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science 16(2), 101–133.

Pick a model, then move the parameter on the horizontal axis. The teal curve is the outcome as that parameter varies, the dot is your current operating point, and the straight line through it is the tangent, whose slope is the partial derivative (estimated by a central finite difference). A chip reads the sign: green up for a positive response, red down for a negative one. The table lists every partial at once, so a sign that runs against intuition stands out.

Comparative statics asks a single question. You have an outcome that solves an optimisation, and you nudge one input. Which way does the outcome move, and how far? There are two ways to answer it.

The classical way is calculus. Differentiate the first-order condition in the parameter, which is the implicit function theorem:

The denominator is positive at a maximum (the second-order condition c'' − b_xx > 0), so the sign of the response is the sign of the cross-partial: \( \operatorname{sign}(dx^{*}/d\theta) = \operatorname{sign}(b_{x\theta}) \). This is what the textbook mode draws as a tangent line. The cost of this route is its assumptions: a real-valued choice, differentiability, and enough curvature.

The second way needs none of that. If the return to a higher x never falls as θ rises, a property called increasing differences,

which for a smooth objective is \( f_{x\theta}\ge 0 \), then the optimiser is nondecreasing: \( \arg\max_x f(x,\theta) \) rises in θ even when the choice is discrete, the objective has kinks, or there are several optima. That is Topkis’ theorem, sharpened by Milgrom and Shannon to the minimal single-crossing condition. Monotone comparative statics keeps the conclusion that matters, the direction of the response, and throws away the machinery you do not need to get it.

Sources: Topkis, D. (1978). Minimizing a submodular function on a lattice. Operations Research 26(2), 305–321. · Milgrom, P. & Shannon, C. (1994). Monotone comparative statics. Econometrica 62(1), 157–180.

Paper mode applies the same lens to a number an executive reports, the average cost overrun among funded projects. The outcome is E[R^c | selected]. The two parameters on offer are the ones with the surprising responses, the noise in benefit estimates σ_b and the budget share α, with the cost noise σ_c as an adjustable baseline.

Take the log of the leading-order form and every sign falls out of a one-line derivative:

The overrun rises in cost noise, as intuition expects. It falls in benefit noise, the result worth pausing on. Adding benefit noise dilutes cost noise’s share of the composite signal σ_η, so the cost overrun shrinks back toward the pure Jensen term:

This is a true cross-effect, a negative partial where intuition expects a positive one. Sharpening your benefit estimates while holding cost estimates fixed makes the measured cost overrun look worse. The overrun also falls in the budget share α (a more generous budget reaches less far into the tail), because λ is increasing and z* decreases with α: \( \partial\ln\mathbb{E}[R^{c}\mid\text{sel}]/\partial\alpha = (\sigma_c^{2}/\sigma_\eta)\,\lambda'(z^{*})\,dz^{*}/d\alpha < 0 \), with \( \lambda'(z)=\lambda(z)\,(\lambda(z)-z) > 0 \) and \( dz^{*}/d\alpha = -1/\phi(z^{*}) \).

The caveat belongs here too. The measured overrun is not welfare. Better benefit precision still improves the ranking and cuts misallocation loss even as the overrun statistic worsens. The closed form is also an upper bound once true project quality varies, which is the paper’s central calibration point.

Source: Leed, A. B. (2026). Noise, Selection, and the Illusion of Bias: How Unbiased Estimates Can Produce Systematic Cost Overruns, working paper.