Overview
Homepage Redesign Experiment — 14 days, 247,892 visitors
Live
Total Visitors
247,892
+12.3%
Total Conversions
10,784
+17.5%
Relative Lift
+17.5%
CI: [12.1%, 23.2%]
Statistical Sig.
p < 0.001
Power: 98.2%
Daily Conversion Rate
Cumulative Lift Over Time
Conversion Funnel
Performance by Channel
Device Breakdown
A
Control Group
50% traffic — Original design
Visitors
123,946
Conversions
4,958
Conversion Rate
4.00%
B
Winner
Variant Group
50% traffic — New design
Visitors
123,946
Conversions
5,826
Conversion Rate
4.70%
+17.5% lift
Bayesian Posterior Distribution
Confidence Interval Over Time
Statistical Summary
# Two-Sample Z-Test for Proportions
Control (A): n=123,946 conversions=4,958 rate=4.00%
Variant (B): n=123,946 conversions=5,826 rate=4.70%
Absolute Lift: +0.70 pp
Relative Lift: +17.50%
95% CI (Abs): [0.48pp, 0.92pp]
Z-Statistic: 8.42
P-value: < 0.0001 ***
Power (1-beta): 98.2%
Min. Sample Size: 24,800 per group (achieved: 123,946)
# Conclusion: Reject H0. Variant B significantly outperforms Control.
Logistic Regression
Incrementality validation: Does the treatment have a genuine causal effect on conversion, controlling for covariates?
# Model: conversion ~ treatment + age_group + device + channel + hour_of_day + is_returning
AIC: 142,387 BIC: 142,501 Pseudo R2: 0.0847 N: 247,892
Coefficient Plot (Log-Odds)
Odds Ratios with 95% CI
ROC Curve
Regression Table
| Variable | Coeff | Std Err | z | P>|z| | OR |
|---|---|---|---|---|---|
| treatment (B) | 0.182 | 0.021 | 8.67 | <0.001 | 1.200 |
| age: 25-34 | 0.342 | 0.028 | 12.21 | <0.001 | 1.408 |
| age: 35-44 | 0.287 | 0.031 | 9.26 | <0.001 | 1.332 |
| device: mobile | -0.156 | 0.024 | -6.50 | <0.001 | 0.856 |
| channel: organic | 0.421 | 0.026 | 16.19 | <0.001 | 1.524 |
| channel: paid | 0.198 | 0.029 | 6.83 | <0.001 | 1.219 |
| is_returning | 0.534 | 0.022 | 24.27 | <0.001 | 1.706 |
| hour: peak (12-14) | 0.112 | 0.033 | 3.39 | 0.001 | 1.119 |
Causal Forest (GRF)
Heterogeneous treatment effect estimation using Generalized Random Forest
# grf::causal_forest(X, Y, W, num.trees=2000, honesty=TRUE)
ATE (Average Treatment Effect): 0.0070 (SE: 0.0008) # +0.70pp conversion lift
CATE Range: [-0.002, 0.031] # strong heterogeneity detected
Variable Importance (top 3): is_returning (0.34), age (0.28), channel (0.19)
CATE Distribution
Variable Importance
Uplift Curve (AUUC)
Subgroup Treatment Effects
Treatment Effect by Age
Treatment Effect by Channel
Conversion by Device
Hourly Conversion Pattern
Targeting Recommendations
High Uplift
Returning users, Age 25-34, Organic
CATE: +3.1pp | 18% of traffic | Expected +892 conversions
Medium Uplift
New users, Age 35-44, Paid Search
CATE: +1.2pp | 24% of traffic | Expected +712 conversions
Low / Negative
Mobile, Age 45+, Social
CATE: -0.2pp | 12% of traffic | Consider excluding