Regression Analysis

Units

1. 1

   Simple linear regression

   4 lessons

   1. The simple linear regression modelWrite and interpret a simple linear regression model, find its conditional mean at a given study time, and calculate a score’s signed departure from that mean.
   2. Least squares estimatesCalculate a least-squares slope and intercept from a small data set and explain why they minimize squared vertical residuals.
   3. Fit and R-squaredCalculate and interpret R², relate it to correlation, and calculate residual standard error for a simple least-squares line with an intercept.
   4. Interpreting the fitted lineInterpret coefficients and fitted scores in context; recognize extrapolation, regression toward the mean, and the limits of causal claims.
2. 2

   Inference in simple regression

   4 lessons

   1. Sampling properties of the estimatorsShow that b₁ and b₀ are unbiased under the regression model, derive Var(b₁) = σ²/Sxx, and see how the spread of x controls the precision of the slope.
   2. Tests and intervals for the slopeTest H₀: β₁ = 0 with t = b₁/SE(b₁) on n − 2 degrees of freedom, build a confidence interval for β₁, and read both from a regression output table.
   3. Confidence and prediction intervalsBuild a confidence interval for the mean response at x₀ and a prediction interval for one new observation, and explain why both widen away from x̄.
   4. ANOVA for regressionSplit the total variation SST into SSR + SSE, lay them out in an ANOVA table, test the regression with F = MSR/MSE on 1 and n − 2 degrees of freedom, and see that F = t² for one predictor.
3. 3

   Residual diagnostics

   3 lessons

   1. Residual plotsPlot residuals against x or fitted values and read them for the two conditions they check best: a straight-line mean and constant error variance.
   2. Checking normality and independenceRead a normal Q-Q plot of residuals for normality, plot residuals in time order to spot autocorrelation, and compute and judge the Durbin-Watson statistic.
   3. Outliers, leverage and influenceSeparate three properties of one unusual point: an outlier (large studentized residual), a high-leverage point (large hᵢᵢ) and an influential point (large Cook's distance), and compute each.
4. 4

   Multiple regression in matrix form

   4 lessons

   1. The model in matrix formWrite a regression with several predictors as y = Xβ + ε, build the design matrix X with its column of ones, and state the assumptions E(ε) = 0 and Cov(ε) = σ²I.
   2. Normal equations and the OLS solutionDerive the normal equations XᵀXb = Xᵀy by minimizing ‖y − Xb‖², solve them as b = (XᵀX)⁻¹Xᵀy, and read the solution as the projection of y onto the column space of X.
   3. The hat matrix and residualsDefine the hat matrix H = X(XᵀX)⁻¹Xᵀ, show it is symmetric and idempotent, write the residuals as e = (I − H)y, and estimate σ² without bias by s² = SSE/(n − p).
   4. Inference for coefficientsEstimate Cov(b) = σ²(XᵀX)⁻¹ by s²(XᵀX)⁻¹, test single coefficients with t on n − p degrees of freedom, build their confidence intervals, run the overall F test, and compare R² with adjusted R².
5. 5

   Interpreting multiple regression

   4 lessons

   1. Coefficients as partial effectsInterpret βⱼ as the change in E(Y) per unit of xⱼ with the other predictors held fixed, and explain why it differs from the simple-regression slope when predictors are correlated.
   2. Omitted variable biasDerive the bias of a simple slope when a correlated predictor is left out, E(b₁) = β₁ + β₂δ, and reason about its sign in the wage, education and ability example.
   3. Added-variable plotsCompute a multiple regression coefficient as the slope of residuals on residuals (the Frisch-Waugh-Lovell theorem) and read an added-variable plot for its strength, outliers and shape.
   4. Comparing nested modelsTest whether a group of q coefficients is zero with the general linear F statistic F = ((SSE_R − SSE_F)/q)/(SSE_F/(n − p)), and read extra sums of squares.
6. 6

   Categorical predictors and interactions

   3 lessons

   1. Indicator variablesCode a categorical predictor with k − 1 indicator variables, interpret each coefficient against the baseline category, and explain the dummy variable trap.
   2. Interaction termsAdd a product term x·D so that one predictor's slope depends on another, interpret the fitted non-parallel lines, and test the interaction.
   3. Analysis of covarianceCompare group means while adjusting for a continuous covariate: test equal slopes first, then equal intercepts with nested F tests, and report adjusted means.
7. 7

   Model selection

   3 lessons

   1. Criteria for comparing modelsCompute and compare adjusted R², Mallows' Cp, AIC and BIC for candidate models, and explain each as a trade between fit and complexity.
   2. Best subsets and stepwise selectionSearch candidate models by best subsets, forward selection and backward elimination, and explain why significance after selection is overstated.
   3. Cross-validationEstimate out-of-sample prediction error by holdout, k-fold and leave-one-out cross-validation, using the PRESS shortcut eᵢ/(1 − hᵢᵢ), and choose a model by it.
8. 8

   Multicollinearity

   3 lessons

   1. What multicollinearity doesExplain how highly correlated predictors inflate coefficient variances, make individual t tests insignificant while the overall F test is significant, and let small data changes flip signs.
   2. Variance inflation factorsCompute VIFⱼ = 1/(1 − Rⱼ²) by regressing each predictor on the others, apply the common rules of thumb, and use the eigenvalues and condition number of the predictors' correlation matrix to locate a near-dependence.
   3. Remedies for multicollinearityChoose a remedy for collinearity to fit the model's purpose: drop or combine predictors, centre before forming polynomial and interaction terms, collect data that breaks the dependence, or shrink the coefficients with ridge regression.
9. 9

   Transformations and nonlinearity

   3 lessons

   1. Log transformationsFit and interpret log-level, level-log and log-log models: a slope as a percent change in y, a change in y per percent change in x, or an elasticity.
   2. Variance-stabilizing transformationsDetect error variance that grows with the mean, choose a Box-Cox power λ by maximizing the profile likelihood, and alternatively fit weighted least squares with weights 1/Var(εᵢ).
   3. Polynomial and spline termsFit quadratic and cubic terms, follow the hierarchy principle and centring, and fit a piecewise linear spline with a knot as a flexible alternative.
10. 10

    Logistic regression

    4 lessons

    1. The logistic modelExplain why a straight line fails for a 0-1 response, define the logit link log(p/(1 − p)) = β₀ + β₁x, and read the fitted S-shaped probability curve.
    2. Coefficients as odds ratiosInterpret e^βⱼ as the odds ratio for a one-unit increase in xⱼ with the other predictors fixed, compute predicted probabilities at chosen values, and give an interval for an odds ratio.
    3. Fitting and inferenceFit a logistic model by maximum likelihood through iteratively reweighted least squares, test coefficients with Wald and likelihood ratio (deviance) tests, and compare models with AIC.
    4. Assessing classificationTurn predicted probabilities into classifications at a threshold, compute the confusion matrix, sensitivity and specificity, draw the ROC curve with its AUC, and check calibration with the Hosmer-Lemeshow idea.
11. 11

    Regularization

    3 lessons

    1. Ridge regressionFit ridge regression by minimizing SSE + λ∑βⱼ², use the closed form (XᵀX + λI)⁻¹Xᵀy on standardized predictors, and explain shrinkage as trading bias for variance.
    2. The lassoFit the lasso, minimizing ½SSE + λ∑|βⱼ|, read its coefficient path with exact zeros, explain sparsity by the diamond against circle picture, and compare it with ridge.
    3. Choosing the penaltyChoose the lasso penalty λ by k-fold cross-validation, apply the one-standard-error rule to prefer a simpler model, and explain when the elastic net, mixing lasso and ridge penalties, is the better tool.