Tools › Statistics › Hypothesis Tests

Hypothesis Tests

2 tools in the Statistics area.

Hypothesis Tests

The common tests of means, proportions, counts and group differences, each reported the way a statistics section of a report wants it: the hypotheses, the statistic with its formula and the substituted numbers, the degrees of freedom, the critical value, the p-value, the decision at α, the confidence interval, the effect size and the power at the sample effect. One-sample, two-sample (pooled and Welch) and paired t tests from raw data or summary statistics; one- and two-proportion z tests with the exact binomial p-value; χ² goodness of fit and test of independence with the expected counts and Cramér's V; one-way ANOVA with the ANOVA table, η² and Tukey HSD pairwise comparisons. A planning section gives the sample size for a target effect size at a target power (exact noncentral t for the t tests), with the sampling distribution and the power curve drawn. The default case is Student's 1908 sleep data as a paired t test.

3 worked examples reproduced, listed on the tool page.

A/B Test

Analysis of an online controlled experiment with a control and up to four treatments, for conversion rates (visitors and conversions per variant) or a continuous metric (n, mean and standard deviation, or pasted raw values). Frequentist part: the rate of every variant with its Wilson interval, the pooled two-proportion z test against the control, the difference with Newcombe's hybrid-score interval, the relative uplift with a log-scale large-sample interval, Fisher's exact p-value, the χ² test across all variants, Welch's t test with Cohen's d for continuous metrics, and Bonferroni or Holm adjustment of the p-values. Bayesian part: Be⁠ta posteriors, the probability that each treatment beats the control by Cook's closed-form sum and by 200,000 seeded Monte Carlo draws, the expected loss of choosing each variant and the 95 % highest-density interval of the uplift, with the posteriors drawn. Planning: the sample size per variant for a minimum detectable effect at the chosen power and significance (R's power.prop.test formula and Kohavi's 16 σ²/δ² rule), the duration from the daily traffic, the effect detectable with the current sample and the observed power. A closing note states the peeking problem and lists the Pocock and O'Brien–Fleming boundaries for the entered number of looks. The default case is an illustrative 10,000-visitor-per-variant test with 500 against 560 conversions (5.0 % against 5.6 %).

1 worked example reproduced, listed on the tool page.

Other Statistics sections

Descriptive and Outliers · Regression and Modelling · Distributions · Forecasting · Surveys and Sampling · Uncertainty and Monte Carlo · Probability