Methodology Notes | Causal Sandbox

01 · DATA-GENERATING PROCESS

The causal world

The full sandbox generates 2,400 people. Baseline covariates C₁ and C₂ are observed; U is an unobserved common cause. Treatment A can affect outcome Y directly and through mediator M. A and Y both cause K, making K a collider measured after the outcome.

Dashed arrows originate from U because this common cause is hidden from the analyst.

Baseline summary and interaction

S_{i} = 0.8 C_{1 i} + 0.6 C_{2 i}, I_{i} = C_{1 i} C_{2 i}

Treatment assignment

P (A_{i} = 1) = expit [- 0.8 + α_{C A} (S_{i} + τ_{A} I_{i}) + α_{U A} U_{i}]

Mediator, outcome, and collider

M_{i} = α_{A M} A_{i} + ε_{M i}

Y_{i} = α_{A Y} A_{i} + α_{C Y} (S_{i} + τ_{Y} I_{i}) + α_{U Y} U_{i} + α_{M Y} M_{i} + ε_{Y i}

K_{i} = 0.9 A_{i} + 0.9 Y_{i} + ε_{K i}

C₁ and C₂ are independent uniform variables on [−√3, √3], giving each mean 0 and variance 1. U and the three error terms are independent standard normal variables.

How the four model worlds differ

The arrows and variables stay fixed. Only the interaction terms in treatment assignment and the outcome equation change.

World	τ_A	τ_Y
Additive relationships	0	0
Outcome interaction	0	1.5
Treatment interaction	0.7	0
Interactions in both	0.7	1.5

02 · ESTIMAND

The target is the population total effect

The truth marker compares everyone under treatment with those same people under no treatment. It includes both the direct and mediated paths from A to Y.

Known average treatment effect

ATE = \underset{direct path}{α_{A Y}} + \underset{mediated path}{α_{A M} α_{M Y}}

Adjusting for M blocks part of this total effect. Adjusting for K conditions on a collider. The sandbox allows both mistakes so their consequences remain visible.

03 · FULL-SANDBOX ESTIMATORS

How each estimate is calculated

Let X be the variables selected for adjustment, Ŷ_a(X) the fitted outcome under treatment a, and ê(X) the fitted probability of treatment. The app fits ordinary least squares for outcome and logistic regression for treatment.

Unadjusted association

Subtract the untreated outcome mean from the treated outcome mean. OLS with A as its only predictor gives the same number, so the interface combines “raw” and “naive” estimates.

{\hat{ψ}}_{raw} = \frac{\sum A_{i} Y_{i}}{\sum A_{i}} - \frac{\sum (1 - A_{i}) Y_{i}}{\sum (1 - A_{i})}

Outcome regression

Predict each person’s outcome twice, once under A = 1 and once under A = 0, then average the within-person contrasts. These are fitted counterfactual predictions, not two observed outcomes.

{\hat{ψ}}_{OR} = \frac{1}{n} \sum [{\hat{Y}}_{1} (X_{i}) - {\hat{Y}}_{0} (X_{i})]

Inverse probability weighting

Weight each observed outcome by the inverse probability of receiving that person’s treatment. The app uses separate Hájek normalization within the treated and untreated groups.

{\hat{ψ}}_{IPW} = \frac{\sum A_{i} Y_{i} / {\hat{e}}_{i}}{\sum A_{i} / {\hat{e}}_{i}} - \frac{\sum (1 - A_{i}) Y_{i} / (1 - {\hat{e}}_{i})}{\sum (1 - A_{i}) / (1 - {\hat{e}}_{i})}

Augmented IPW

Begin with each outcome-regression contrast, then add an inverse-weighted residual correction. Under the causal assumptions, AIPW can remain consistent when either the outcome model or treatment model is correct.

{\hat{ψ}}_{AIPW} = \frac{1}{n} \sum {{\hat{Y}}_{1} (X_{i}) - {\hat{Y}}_{0} (X_{i}) + \frac{A_{i}}{{\hat{e}}_{i}} [Y_{i} - {\hat{Y}}_{1} (X_{i})] - \frac{1 - A_{i}}{1 - {\hat{e}}_{i}} [Y_{i} - {\hat{Y}}_{0} (X_{i})]}

Model features, clipping, and effective sample size

Selecting C adds both C₁ and C₂. In an interaction world, the corresponding “include interaction” control adds C₁C₂ to the chosen nuisance model. Selecting M or K adds that post-treatment variable directly; U cannot be selected because it is hidden.

Fitted propensities are clipped to [0.02, 0.98] before IPW, AIPW, or TMLE uses them. Clipping prevents infinite numerical weights but can introduce bias; it does not create missing treatment support.

Weight diagnostic shown by the app

ESS = \frac{{(\sum w_{i})}^{2}}{\sum w_{i}^{2}}

04 · TARGETING

Where TMLE fits

The TMLE lesson starts from outcome predictions and fitted propensities, then performs a one-dimensional squared-error update for the continuous outcome. The update is chosen so the empirical weighted residual score is zero before the revised prediction contrast is averaged. Here rᵢ = Yᵢ − Ŷ_Aᵢ(Xᵢ) is the observed residual from the initial outcome model.

Clever covariate and targeting coefficient

H_{i} = \frac{A_{i}}{{\hat{e}}_{i}} - \frac{1 - A_{i}}{1 - {\hat{e}}_{i}}, \hat{ε} = \frac{\sum H_{i} r_{i}}{\sum H_{i}^{2}}

Updated predictions and estimate

{\hat{Y}}_{1 i}^{*} = {\hat{Y}}_{1 i} + \frac{\hat{ε}}{{\hat{e}}_{i}}, {\hat{Y}}_{0 i}^{*} = {\hat{Y}}_{0 i} - \frac{\hat{ε}}{1 - {\hat{e}}_{i}}, {\hat{ψ}}_{TMLE} = \frac{1}{n} \sum ({\hat{Y}}_{1 i}^{*} - {\hat{Y}}_{0 i}^{*})

This is a linear fluctuation for an unbounded continuous outcome, not the logistic update used for bounded outcomes. Targeting does not repair hidden confounding or absent overlap.

Open the interactive TMLE lesson →

05 · INTERPRETATION

When an estimate has a causal interpretation

Consistency and no interference

Each person’s observed outcome follows the treatment they received, and one person’s treatment does not alter another person’s outcome. The simulator builds both conditions in.

Conditional exchangeability

All common causes of treatment and outcome must be measured and handled. When U causes both A and Y, adjustment with the observed variables cannot identify the total effect.

Positivity

Every covariate pattern used for the target population needs a positive chance of both treatments. Extreme probabilities produce unstable weights; zero support cannot be repaired by clipping.

Appropriate model specification

Outcome regression relies on its outcome model, and IPW on its treatment model. AIPW and this TMLE implementation require one of those two nuisance models to be correct, plus the causal assumptions above.

06 · BOUNDARIES

What the sandbox does not establish

Results are finite-sample point estimates. The app provides no confidence intervals, standard errors, or hypothesis tests.
A fixed draw makes changes easy to compare. One seeded sample cannot establish a general ranking between estimators.
The red error tint and “close” threshold are visual teaching aids, not inferential claims.
Known truth is available only because the data-generating process is simulated. Real observational studies do not reveal their causal effect for comparison.
The estimators are dependency-free educational implementations, not replacements for validated statistical software or a study design tailored to a real scientific question.

How the sandbox creates data and estimates effects