Statistical Inference

Units

1. 1

   Statistics and sampling distributions

   4 lessons

   1. Populations, samples and statisticsDistinguish a population parameter from a sample statistic and find the sampling distribution of a two-draw mean.
   2. Sampling distribution of the meanFind the center and standard error of a sample mean, and decide whether a normal model for it is exact or approximate.
   3. Chi-square, t and F distributionsConstruct χ², t and F statistics from independent normal samples and identify their reference distributions and degrees of freedom.
   4. Order statisticsFind cutoff probabilities for the minimum, maximum, k-th ordered value, and odd-sample median of independent, identically distributed draws.
2. 2

   Point estimation

   3 lessons

   1. Estimators and estimatesTell an estimator (a statistic, a random variable) from an estimate (its value on one sample), and use X̄, S² and the sample proportion p̂ to estimate μ, σ² and p.
   2. Bias, variance and MSECompute the bias and variance of an estimator, combine them as MSE = Var + bias², show that S² is unbiased for σ² because it divides by n − 1, and see that an unbiased estimator need not have the smallest MSE.
   3. Consistency, efficiency and sufficiencyShow an estimator is consistent, compare two estimators by relative efficiency, compute the Cramér-Rao lower bound from the Fisher information, and find a sufficient statistic by the factorization theorem.
3. 3

   Method of moments

   3 lessons

   1. The method of momentsBuild moment estimators by equating sample moments to population moments and solving for the parameters, for the exponential rate and for the normal mean and variance, and judge one by simulation.
   2. Moment estimators for two parametersSolve two moment equations for two parameters, for the gamma distribution and the uniform distribution, and recognise when a moment estimate is impossible.
   3. Comparing estimators by simulationCompare the moment estimator 2X̄ and the sample maximum for uniform(0, θ) by bias, variance and MSE, from simulated sampling distributions side by side and from exact formulas.
4. 4

   Maximum likelihood

   4 lessons

   1. The likelihood functionWrite the likelihood L(θ) as the joint density or pmf of the data read as a function of θ, plot it for Bernoulli data, compare parameter values by likelihood ratios, and turn the product into a sum with the log-likelihood.
   2. Finding maximum likelihood estimatorsFind maximum likelihood estimators by setting the score to zero and checking the second derivative, for Bernoulli, Poisson, exponential and normal models, and find the MLE at the boundary for uniform(0, θ).
   3. Properties of the MLEUse the invariance of the MLE, state its consistency and asymptotic normality with variance 1/(nI(θ)), and compute a standard error from the observed information.
   4. Numerical maximum likelihoodWhen no closed form exists, maximizing the log-likelihood numerically with Newton-Raphson iterations, illustrated for the gamma shape parameter.
5. 5

   Confidence intervals

   3 lessons

   1. Confidence intervals and pivotsA 95% confidence interval as a procedure covering theta in 95% of samples, built from a pivotal quantity, shown with many simulated intervals.
   2. Intervals for a meanThe z interval with known sigma and the t interval X-bar +/- t* S/sqrt(n) for unknown sigma, and choosing sample size from a target margin of error.
   3. Intervals for proportions and variancesWald and Wilson intervals for a proportion, the chi-square interval for a normal variance, and large-sample intervals from asymptotic normality of the MLE.
6. 6

   Hypothesis testing

   4 lessons

   1. The logic of hypothesis testsNull and alternative hypotheses, test statistic, rejection region, type I and type II errors in a 2 by 2 table, and the significance level alpha.
   2. P-valuesThe p-value as the probability under H0 of a result at least as extreme, one-sided versus two-sided tests, common misreadings, and duality with confidence intervals.
   3. Tests for means and proportionsOne-sample z tests for a mean and for a proportion, worked numerically with a normal curve shading the rejection region and the p-value.
   4. Multiple testingInflation of false positives across many tests, the Bonferroni correction, and the Benjamini-Hochberg false discovery rate procedure on a worked list of p-values.
7. 7

   Power and optimal tests

   3 lessons

   1. Power and sample sizeThe power function, the probability of rejecting H0 as a function of theta, plotted for a one-sided z test, and choosing n to reach a target power.
   2. The Neyman-Pearson lemmaThe most powerful test of a simple null against a simple alternative rejects when the likelihood ratio is large, proved and applied to a normal mean.
   3. Uniformly most powerful testsExtending Neyman-Pearson to one-sided composite alternatives via monotone likelihood ratio (Karlin-Rubin theorem), and why two-sided UMP tests usually do not exist.
8. 8

   Likelihood ratio tests

   3 lessons

   1. Generalized likelihood ratio testsLambda as the maximized likelihood under H0 over the unrestricted maximum, rejecting for small Lambda, derived for a normal mean with unknown variance to recover the t test.
   2. Wilks' theoremUnder H0, -2 log Lambda is approximately chi-square with degrees of freedom equal to the difference in free parameters, applied to comparing two Poisson rates.
   3. Wald and score testsWald, score and likelihood ratio tests as three asymptotically equivalent tests, compared geometrically on a plotted log-likelihood curve.
9. 9

   t, chi-square and F tests

   3 lessons

   1. One and two sample t testsOne-sample, paired and two-sample t tests (pooled and Welch), their assumptions, and a worked comparison of two group means.
   2. Chi-square testsPearson's chi-square goodness-of-fit test and test of independence for contingency tables, the statistic sum of (O - E)^2/E, expected counts and degrees of freedom.
   3. F test and one-way ANOVAThe F test for equal variances, and one-way ANOVA splitting the total sum of squares into between and within groups, with F = MSB/MSW in an ANOVA table.
10. 10

    Nonparametric methods and the bootstrap

    4 lessons

    1. Rank-based testsThe sign test, Wilcoxon signed-rank test and Mann-Whitney (Wilcoxon rank-sum) test as distribution-free alternatives to t tests, with a worked ranking table.
    2. Permutation testsExact and Monte Carlo permutation tests for a difference in means, the null permutation distribution as a histogram, and the exchangeability assumption.
    3. Empirical CDF and Kolmogorov-SmirnovThe empirical CDF, the Glivenko-Cantelli theorem stated, and the Kolmogorov-Smirnov statistic as the largest gap between the empirical and hypothesized CDFs.
    4. The bootstrapEfron's nonparametric bootstrap: resampling with replacement to estimate standard errors, and percentile bootstrap confidence intervals, illustrated for a sample median.
11. 11

    Bayesian inference

    4 lessons

    1. Priors, likelihoods and posteriorsPosterior proportional to likelihood times prior, parameters treated as random, and a discrete grid prior for a coin's bias updated after observed flips.
    2. Conjugate priorsBeta-binomial and normal-normal conjugate updating, the posterior mean as a weighted average of prior mean and data, with prior and posterior densities plotted.
    3. Bayesian estimates and credible intervalsPosterior mean, median and MAP estimates, equal-tailed and highest posterior density credible intervals, and the contrast with frequentist confidence intervals.
    4. Bayesian testing and computationBayes factors for comparing hypotheses, posterior predictive distributions, and a brief look at Markov chain Monte Carlo for posteriors without closed form.