Paired comparisons & uncertainty

A mean describes observed measurements. An uncertainty interval describes how an estimate varies under a sampling procedure and its assumptions; it does not remove dataset bias. Independent observations are important. Thousands of correlated variants of one prompt are not thousands of independent tasks.

For n independent numeric observations x, the sample variance is the sum of squared deviations from their mean divided by n-1. The standard error of the mean is sqrt(sample_variance/n). n=1 has no sample variance estimate, so our helper rejects it rather than reporting unwarranted zero uncertainty.

Bootstrap the paired differences so each resampled unit remains the same example in both conditions.

import numpy as np

x = np.array([1.0, 2.0, 3.0])
print(x.mean())
print(round(x.std(ddof=1) / np.sqrt(x.size), 6))

When baseline and intervention evaluate the same examples, compare paired differences. Resampling baseline and treatment independently throws away the pairing. Resample example indices once, use those indices for both conditions, and compute the mean difference for each bootstrap draw.

import numpy as np

base = np.array([0.1, 0.8, 0.4])
treatment = base + 0.2
diff = treatment - base
rng = np.random.default_rng(4)
sampled = rng.integers(0, len(diff), size=(5, len(diff)))
print(np.round(diff[sampled].mean(axis=1), 3))

The percentile bootstrap takes lower/upper quantiles of these means. It is a practical approximation, not a guarantee of coverage for every distribution. It cannot estimate behavior outside observed data. Predeclare the interval, sample size and unit of resampling; if examples form groups, resample groups.

A standard error needs an estimated sample variance; one observation cannot establish that variance.

The build uses iid rows, a 95% percentile interval, a fixed local RNG sequence and NumPy's default linear quantile interpolation. Its interval measures the difference treatment minus baseline. Check constant differences, a zero effect, repeated seeds, malformed lengths and n<2 for the standard-error helper.