Lesson 4–5: Probability & Statistics for Data Science

Introduction

Most probability and statistics curricula teach concepts in isolation. You finish a notebook on Bayes' theorem, close it, and open another on variance. Each exercise works—but nothing resembles what teams ship: a service with contracts, composable logic, and a path from raw data to a decision.

week_4_5_aiml_integrated_project closes that gap. It is a self-contained capstone that unifies probability, Bayesian inference, distributions, descriptive analytics, correlation, statistical inference, and dataset readiness scoring into a single stats_engine: an installable Python package composed of topic modules, exposed through FastAPI and an optional React dashboard. The goal is not more theory in a vacuum—it is system-first learning: understand each concept in context, then see how engineers unify them behind stable endpoints and one product surface.

For engineers, the payoff is traceability. Coin simulation is not just a loop—it is a function with seed control, pytest coverage, and an HTTP contract. Dataset readiness is not just a pandas script—it is an async job with polling, structured quality flags, and a 0–100 score returned as JSON.

Core Components

The stack is layered. Each topic below is a distinct building block; together they form one coherent statistics system.

Probability and simulation

Models uncertainty through classical probability: simple and joint probabilities, coin and die simulations, and frequency-based spam classification.

In the domain package, ProbabilityBasics and ProbabilityDistribution handle simulations; SpamClassifier trains on labeled phrases and classifies new text. The stats_engine wrapper exposes simulate_coin, simulate_die, train_spam_classifier, and classify_spam with deterministic seeds for reproducible API responses.

Bayesian inference

Extends uncertainty modeling with conditional probability, Naive Bayes spam filtering, and medical diagnosis scenarios.

Key engine functions: conditional_probability, classify_spam_bayes, diagnose_medical. The product endpoint POST /api/v1/stats/classify/spam returns is_spam, spam_probability, confidence, label, and top_indicators—an uncertainty layer in production shape.

Probability distributions and sampling

Samples from Bernoulli, binomial, normal, and exponential distributions; includes a Central Limit Theorem preview for teaching convergence behavior.

distributions.sample_distribution returns sample arrays and summary statistics; histogram plots are rendered server-side via Matplotlib and returned as structured PlotFigure objects for the frontend.

Descriptive statistics — central tendency

Computes mean, median, mode, min, max, and range for numeric series. Includes a built-in scenario metrics endpoint for quick smoke testing.

descriptive_stats.profile_central raises EMPTY_DATASET on empty input. scenario_metrics returns a fixed three-value demo dataset for lab exploration.

Spread, outliers, and data quality

Measures variance, standard deviation, coefficient of variation, IQR, and outlier detection via z-score and IQR methods.

profile_spread raises ZERO_VARIANCE when all values are identical. These signals feed directly into the dataset analyzer's quality-issue detector.

Correlation analysis

Computes covariance and Pearson correlation matrices, identifies highly correlated feature pairs above a configurable threshold, and flags multicollinearity risks before model training.

correlation.correlation_analysis accepts optional raw values or uses a built-in demo matrix when none are provided.

Statistical inference

Covers Central Limit Theorem metrics, accuracy confidence intervals, and A/B test sample-size estimation.

Functions: clt_metrics, accuracy_ci, ab_sample_size. These support experiment design and model-evaluation workflows without leaving the same API envelope.

Dataset analyzer and readiness scoring