Scaling Experiments and Heterogeneous Treatment Effects

Summary of the Previous Session

Throughout this course, we have explored a comprehensive range of experimental designs, building a robust foundation for conducting rigorous scientific research. We began with randomized controlled trials (RCTs), the gold standard for establishing causality, and progressed to more complex frameworks such as Just-in-Time Adaptive Interventions (JITAIs), Micro-Randomized Trials (MRTs), and Sequential Multiple Assignment Randomized Trials (SMARTs). Our primary focus has been on methodological rigor—ensuring reliable, valid results through careful design, randomization, and control of confounding variables to support causal inference.

Having covered the spectrum of experimental design methodologies, we are now well-equipped to shift our focus. In today’s session, we will explore practical challenges in implementing experiments, such as logistical constraints and real-world applicability. This session bridges the gap between theoretical design and actionable, innovative research practice.

Scalability in Experiments

According to Al-Ubaydli et al. (2019), many experiments fail to scale because of what they call the Voltage Effect: the idea that the impact (or “voltage”) of an intervention diminishes as it moves from a controlled trial to broader implementation.

Scaling is hard not just because of logistical complexity, but because the effect size observed in pilot studies is often inflated, unstable, or conditional on specific circumstances that don’t generalize. The authors highlight three critical threats to scalability:

1. Statistical Inference – Are the results even real?

   • False positives due to small sample sizes or noisy data

   • Publication bias and selective reporting inflate effect sizes

   • Winner’s curse: the largest effects may come from random noise

2. Representativeness of the Population – Does this work for the general public?

   • Pilot subjects may be more motivated, educated, or incentivized

   • Results don’t generalize to broader, more heterogeneous populations

3. Representativeness of the Situation – Does the setting match real-world conditions?

   • Implementation fidelity often drops at scale

   • Admin quality, delivery agents, or context vary widely

   • Network effects and diseconomies of scale can distort results

The Head Start home visiting program was an early childhood intervention targeting at-risk families. In its initial trials, the program showed strong effects on child development and parenting outcomes.

However, when scaled up, the results diminished significantly. Studies found that:

• Home visits to “at-risk” families were often interrupted by distractions

• Less time was spent on child-focused activities during visits

• Program delivery was inconsistent across facilitators

This is a textbook case of the Voltage Effect: the situation changed (threat #3), the fidelity declined, and the population scaled up in ways the original study didn’t account for.

How can we design experiments that actually work when taken beyond the lab or pilot setting?

Scaling is not just about increasing size — it’s about maintaining impact across broader populations, messier environments, and lower control. To design experiments that truly work at scale, we must anticipate threats and build solutions directly into the design phase.

Five Strategies for Designing Scalable Experiments

1. Anticipate threats to scale in the design

   • Use List and Suskind’s three-bin framework to evaluate:

     • Statistical inference risk

     • Population representativeness

     • Situational fidelity

2. Use large, diverse, real-world samples

   • Avoid “WEIRD” subject pools

   • Design for generalizability from the outset

3. Leverage digital platforms for experimentation

   • Conduct A/B tests, and adaptive trials online

   • Use real-time dashboards for iterative learning and deployment

4. Measure and monitor implementation fidelity

   • Track delivery agents, dosage, and adherence at scale

   • Prevent program drift through standardization or tech-enabled delivery

5. Combine traditional methods with predictive modeling

   • Use machine learning to estimate heterogeneous treatment effects

   • Improve counterfactual estimates and policy targeting

Case Example: Amazon Cart Recommendations

Amazon tested personalized cart recommendations despite leadership opposition. Rather than relying on expert opinion, they ran a large-scale online experiment.

• The test occurred on live traffic — high external validity

• Digital platform ensured fidelity and consistent delivery

• The effect was robust and profitable — led to full-scale rollout

This example demonstrates how modern experimentation infrastructure can reduce scalability threats and allow evidence-based product decisions.

Application of Machine Learning to Estimate Heterogeneous Treatment effects

Traditional RCTs focus on estimating the average treatment effect (ATE): $$\text{ATE} = {E}[Y(1) - Y(0)]$$ However, in many real-world settings, the treatment effect may vary across individuals or subgroups. The goal is to estimate: $$\tau(x) = {E}[Y(1) - Y(0) \mid X = x]$$ which is known as the Conditional Average Treatment Effect (CATE). Machine learning (ML) can help estimate this function, enabling personalized policy decisions and improving external validity.

Step-by-Step Framework

1. Run an RCT. Randomly assign units to treatment and control groups to ensure internal validity.

2. Collect covariates and outcomes. For each unit $i$, observe: $$(X_i, D_i, Y_i)$$ where $X_i$ are pre-treatment covariates, $D_i \in \{0,1\}$ is treatment assignment, and $Y_i$ is the outcome.

3. Use ML to estimate treatment effects conditional on $X_i$.

Two Approaches to Estimating CATE

1. T-Learner (Two-Model Approach)

The T-learner separately estimates potential outcomes under treatment and control.

1. Split the Data:

   • Divide your dataset into treatment ($W = 1$) and control ($W = 0$) groups.

   • Each group has outcomes $Y$ and covariates $X$.

2. Fit Two Models:

   • Train a model $\mu_1(X)$ on the treatment group to predict $Y$ given $X$.

   • Train a separate model $\mu_0(X)$ on the control group to predict $Y$ given $X$.

   • Any supervised learning model can be used (e.g., linear regression, random forests, neural networks), depending on the data complexity.

3. Estimate CATE:

   • For any individual with covariates $X = x$, compute: $$\text{CATE}(x) = \mu_1(x) - \mu_0(x)$$

   • This gives the estimated treatment effect conditional on $X$.

Advantages:

• Simple and modular: any regression model can be used for $\hat{\mu}_1$ and $\hat{\mu}_0$

• Flexible for high-dimensional covariates

Limitations:

• If the treated and control groups differ greatly in $X$, each model is trained on different support → extrapolation becomes risky

• No built-in mechanism for prioritizing treatment effect heterogeneity during training

2. Causal Forests (Wager & Athey, 2018)

Causal Forests, part of the Generalized Random Forest framework, directly estimate CATE by adapting random forests to focus on treatment effect heterogeneity. They partition the covariate space to find regions where treatment effects differ, optimizing for causal inference rather than just prediction.

Before discussing Causal Forests, it is useful to understand standard Random Forests. A Random Forest is an ensemble learning method that constructs multiple decision trees, each trained on a random subset of the data and features. It predicts outcomes by averaging the predictions of all trees, excelling at capturing complex, non-linear relationships in data. Random Forests are widely used for prediction tasks but do not inherently focus on estimating causal effects or treatment heterogeneity.

Causal Forests build on the Random Forest framework by modifying the splitting criterion to prioritize heterogeneity in treatment effects rather than predictive accuracy. This adaptation links the two methods: while Random Forests optimize for outcome prediction, Causal Forests optimize for estimating the conditional average treatment effect, making them a specialized tool for causal inference.

Steps to Implement

1. Prepare Data:

   • Ensure you have $X$ (covariates), $W$ (treatment indicator, binary), and $Y$ (outcome).

   • Data should be from a randomized experiment or observational study with unconfoundedness ($\{Y(1), Y(0)\} \perp W \mid X$), meaning that the data used to estimate causal effects must satisfy conditions ensuring that the treatment assignment is not systematically biased in a way that distorts the relationship between the treatment and outcomes.

2. Fit Causal Forest:

   • Use a Causal Forest algorithm (e.g., from the grf package in R or econml in Python).

   • The algorithm builds trees that split the covariate space to maximize heterogeneity in treatment effects, estimating: $$\text{CATE}(x) = E[Y(1) - Y(0) \mid X = x]$$ Imagine studying a tutoring program’s effect on test scores ((Y), with covariates like prior grades and study hours (X). A Causal Forest might split students by prior grades, finding that tutoring boosts scores more for low-performing students than high-performing ones, estimating different CATEs for each group

   • It accounts for propensity scores ($P(W = 1 \mid X)$) to handle confounding.

     Causal Forests estimate propensity scores (e.g., using a separate model or internally) to understand how $X$ influences treatment assignment. When estimating CATE, they use these scores to reweight or adjust comparisons between treated and control groups, ensuring that individuals with similar $X$ are compared fairly. This adjustment mimics randomization, reducing bias from confounding under the unconfoundedness assumption ($\left\{ Y(1), Y(0) \right\} \perp W \mid X$), meaning all confounders are captured in $X$.

3. Predict CATE:

   • For any $X = x$, the fitted Causal Forest outputs the estimated $\text{CATE}(x)$.

4. Evaluate:

   • Check model calibration (e.g., using out-of-bag predictions).

   • Analyze variable importance to understand which covariates drive heterogeneity.

   • Validate on a holdout set if feasible.

Advantages:

• Built for estimating heterogeneity directly

• Uses honest estimation: splits are trained on one half of the data, effect sizes are estimated on the other

• Provides valid confidence intervals for $\hat{\tau}(x)$ under regularity assumptions

Limitations:

• Less interpretable than simple models

• Requires careful tuning and validation

Use Cases

• Policy targeting: Deliver interventions only to subgroups with high $\hat{\tau}(x)$

• Resource allocation: Prioritize high-impact populations

• Generalization: Simulate effects for new populations by applying $\hat{\tau}(x)$ to new $X$ values

Key References

• Susan Athey and Stefan Wager. “Estimating Treatment Effects with Causal Forests,” JASA, 2018.

• Kunzel, Sekhon, Bickel, and Yu. “Meta-learners for Estimating Heterogeneous Treatment Effects,” PNAS, 2019.

Required readings