Lesson 7 : Machine Learning Core Concepts

Introduction

Most machine-learning curricula teach one idea at a time: what supervised learning means, how to split data, which metric to trust. That works for concepts. It breaks down when you need to ship.

Software engineers rarely deploy isolated exercises. They deploy pipelines—define the problem, prepare data, train, evaluate, persist artifacts, serve predictions. Without a system view, tutorials feel disconnected and APIs multiply without purpose.

week_7_aiml_integrated_project closes that gap. It is a self-contained capstone that unifies seven core ML topics into one repository:

• An installable Python package (week7_python) holding domain logic for AI/ML framing, workflow stages, paradigms, task types, diagnostics, splitting, and metrics

• A composition engine (core_ml_engine) that wires those topics into product workflows

• A FastAPI backend exposing two API surfaces: a lab path for per-topic exploration and a product path for the composed lifecycle

• An optional React frontend that demonstrates the composed path—not seven separate dashboards

The goal is system-first learning: each concept stays teachable on its own, yet all of them participate in one sentiment-classification lifecycle you can run with a single HTTP call.

For engineers, the payoff is traceability. Stratified splitting is not just a notebook cell—it is a facade with pytest coverage and a POST /api/v1/core-ml/split/stratified contract. Evaluation metrics are not just formulas on a slide—they are computed from scratch, compared against scikit-learn, and returned inside a pipeline summary with a readiness score.

Core Components

The stack is layered. Each topic below is a distinct building block; together they form one coherent ML lifecycle system.

AI, ML, and deep learning orientation

Covers the AI ⊃ ML ⊃ DL hierarchy, supervised/unsupervised/reinforcement learning types, traditional programming versus learned rules, and a lightweight supervised regression demo.

In the domain package, AIMLIntroduction exposes hierarchy maps, learning-type overviews, and JSON-safe supervised demos. The core_ml_engine.concepts facade provides hierarchy(), learning_types(), orient_summary(), and orient_lite()—the last skips matplotlib output for fast product demos.

ML workflow pipeline

Implements a seven-stage sentiment-classification pipeline: problem definition, data collection, preparation, training, evaluation, deployment, and monitoring.

MLWorkflowPipeline in the workflow domain module owns the full lifecycle on 40 bundled product reviews. It logs stage events, trains a TF-IDF + logistic regression classifier, evaluates with sklearn metrics, persists sentiment_model.pkl and vectorizer.pkl under backend/data/models/, and serves batch predictions. The core_ml_engine.workflow facade adds run_stage() and run_product_pipeline()—the product path composes stratified splits and from-scratch metrics into one orchestrated run.

Learning paradigms

Demonstrates supervised classification, unsupervised clustering with optional plot artifacts, and side-by-side paradigm comparison tables.

The paradigms domain module runs sklearn demos on synthetic data. core_ml_engine.paradigms exposes run_supervised(), run_unsupervised(), comparison_table(), and recommend_paradigm(has_labels, task)—a decision-support helper used at the start of the demo pipeline.

Regression versus classification

Generates synthetic house-price data and compares linear regression (continuous target) against logistic classification (price bucket) on the same feature set.

HousePriceSystem powers data generation and model comparison. core_ml_engine.task_types.compare_house_models() returns JSON-safe metrics and optional plot paths under backend/data/plots/.

Overfitting diagnostics

Runs a polynomial-complexity sweep with train/validation curves to illustrate bias–variance tradeoffs.

The diagnostics domain module produces degree-by-degree error curves. core_ml_engine.diagnostics.complexity_scan(plots_dir, max_degree) is the product entry point—default max_degree is 10 via the API schema.

Data splitting