Introduction to Data Science

Units

1. 1

   Data tables and tidy data

   4 lessons

   1. Tables, rows and variablesIdentify what rows and columns represent, classify variables, and select rows and columns to answer a question.
   2. Tidy dataReshape a wide library-visits sheet into tidy visit rows, separate branch details, and explain what one row means in each table.
   3. Grouping and aggregatingGroup a tidy sales table, calculate counts, sums and means, arrange sums in a pivot table, and reproduce the summaries with pandas.
   4. Joining tablesJoin two small tables by customer ID using inner, left, and full outer joins; check for missing and duplicated keys.
2. 2

   Exploratory analysis and visualization

   3 lessons

   1. Visualizing one variableDraw a bar chart for a categorical variable, a histogram on the density scale for a numerical one, and a box plot, and choose bins that show the shape honestly.
   2. Visualizing relationshipsChoose and read a scatter plot, a line plot or a group comparison for two variables, and say what each shows that the others hide.
   3. Principles of good graphicsChoose a chart that fits the comparison, keep its axes honest, and cut clutter so the data carry the message.
3. 3

   Summary statistics

   3 lessons

   1. Measures of centerCompute the mean, median and mode of a numerical column, explain the mean as the balance point of the data, and choose between mean and median when a column is skewed or has outliers.
   2. Measures of spreadCompute the range, quartiles, interquartile range, percentiles, variance and standard deviation of a column, read the five-number summary off a box plot, and choose the IQR or the SD to report.
   3. Standard units and Chebyshev's boundConvert values to standard units (z-scores) and back, use Chebyshev's bound to guarantee how much of any column lies within k SDs of its mean, and contrast it with the normal curve's 68-95-99.7 rule.
4. 4

   Data cleaning

   3 lessons

   1. Missing valuesDetect and count missing values in a table, including coded ones, name the type of missingness (MCAR, MAR, MNAR) from evidence, and judge the bias that dropping rows or mean imputation introduces.
   2. Fixing types and formatsConvert numbers stored as text with mixed units into one numeric column, merge category labels spelled several ways, and parse dates with the right day and month order, checking each fix with a before and after summary.
   3. Outliers and validity checksFind impossible values with range rules and extreme values with plots, decide for each whether to correct, set to missing or keep it, and record every decision in a reproducible cleaning script.
5. 5

   Sampling and bias

   3 lessons

   1. Probability samplesName the population, sample, parameter and statistic of a study, draw a simple random sample reproducibly in code with a seed, and describe stratified and cluster sampling and when each helps.
   2. Sources of biasRecognize selection bias, convenience samples, nonresponse bias and response bias in a study, explain why a bigger sample does not cure them, and analyse the 1936 Literary Digest poll.
   3. Observational studies and experimentsTell an observational study from a randomized controlled experiment, explain how a confounder can create or hide an association, and explain why random assignment supports a causal conclusion, with John Snow's cholera study as an example.
6. 6

   Simulation and the bootstrap

   3 lessons

   1. Simulating chanceUsing random number generation to simulate coin flips and dice, empirical distributions approaching probability distributions, and the law of averages.
   2. Sampling distributions by simulationSimulating many samples to build the empirical distribution of a statistic such as the mean, and seeing its spread shrink as sample size grows.
   3. Bootstrap resamplingResampling the original sample with replacement to estimate variability, and building a 95% bootstrap percentile confidence interval for a median.
7. 7

   Hypothesis testing by permutation

   3 lessons

   1. Assessing a model by simulationA null hypothesis as a chance model, choosing a test statistic, simulating it under the null and comparing the observed value, using a jury selection example.
   2. A/B testing with permutationsComparing two groups by shuffling group labels to build the null distribution of the difference in means, and computing an empirical p-value.
   3. P-values and errorsInterpreting empirical p-values, conventional cutoffs such as 0.05, type I errors, p-hacking and the multiple comparisons problem across many tests.
8. 8

   Correlation and prediction

   3 lessons

   1. The correlation coefficientr as the average product of two variables in standard units, its range and properties, scatter plots for several values of r, and nonlinear patterns with r near zero.
   2. The regression lineThe prediction line in standard units with slope r, converted to original units with slope r SD_y/SD_x, and regression to the mean.
   3. Prediction errorResiduals, root mean squared error, the SD of residuals equal to sqrt(1 - r^2) SD_y, and residual plots that reveal poor fits.
9. 9

   Linear regression

   3 lessons

   1. Least squares by optimizationFinding the line that minimizes mean squared error by numerical optimization, matching the formula line, with the RMSE plotted against candidate slopes.
   2. Regression with several featuresMultiple linear regression with several features and one-hot encoded categories, interpreting coefficients, using a scikit-learn style fit and predict.
   3. Inference for the slopeBootstrapping the regression slope for a confidence interval, testing whether the true slope is zero, and bootstrap prediction intervals.
10. 10

    Classification

    3 lessons

    1. The k-nearest neighbors classifierClassifying a new point by majority vote of its k nearest training points under Euclidean distance, drawn on a two-feature scatter plot.
    2. Distance and feature scalingEuclidean distance in several dimensions, why features must be standardized before computing distances, and choosing odd k to avoid ties.
    3. Decision boundaries and accuracyDecision boundaries of k-nearest neighbors for small and large k, accuracy and the confusion matrix, and the effect of class imbalance.
11. 11

    Train/test split and overfitting

    3 lessons

    1. Training and test setsRandomly splitting data into training and test sets, why accuracy on training data is optimistic, and estimating performance on unseen data.
    2. Overfitting and model complexityTraining versus test error as complexity grows (k in k-nearest neighbors, polynomial degree), the U-shaped test error curve, and the bias-variance tradeoff.
    3. Cross-validation for model choicek-fold cross-validation to choose a hyperparameter such as k, keeping a final test set untouched, and data leakage pitfalls.
12. 12

    Ethics and privacy

    3 lessons

    1. Privacy and re-identificationPersonally identifiable information, re-identification from quasi-identifiers (Sweeney's zip code, birth date and sex finding), limits of anonymization, and the idea of differential privacy.
    2. Fairness and bias in algorithmsHow biased training data produces unfair predictions, the COMPAS recidivism case, and competing fairness criteria such as equal error rates versus calibration.
    3. Responsible data practiceInformed consent, data ownership, reproducibility and transparency, and professional codes such as the ACM Code of Ethics, applied to a case scenario.