Lesson 25-26: Natural Language Processing (NLP) 

NLP Intelligence Platform: From Fragmented NLP Scripts to a Unified Production Stack

Introduction

Most NLP tutorials stop at isolated notebooks: one file for tokenization, another for sentiment, a third for intent, none wired to HTTP, persistence, or observability. week_25_26_aiml_integrated_project is a self-contained capstone that consolidates classical text processing, transformer tokenization, embeddings, sequence modeling, sentiment analysis, intent routing, and response selection into one runnable system:

• An installable Python package (week2526_python) with a core engine, learning lab wrappers, and a product NLP pipeline

• A single FastAPI backend exposing a lab path for component exploration and a product path for analyze, predict, and experiment training

• A React dashboard with live telemetry, a support-assistant workflow, a pipeline studio, and an operations console

• Postgres-backed experiment metadata, PyTorch artifacts on disk, and production hardening (timeouts, structured logging, graceful ML degradation)

The engineer payoff is operational clarity: one /analyze call runs preprocess → intent → sentiment → response; training jobs become Run rows you poll from the dashboard; missing models fall back to lexicon and keyword rules instead of crashing the API.

Core Components

Each block below is a distinct capability inside this repository—not an external dependency.

Classical text pipeline

week2526_python.core.text_pipeline provides NLTK-backed normalization, tokenization, stemming, lemmatization, POS tagging, and TF-IDF feature extraction. nltk_bootstrap.py lazily downloads punkt and stopwords only when classical processing runs, keeping cold starts fast.

Subword tokenization

core.tokenization.distilbert and core.tokenization.subword wrap Hugging Face tokenizers (DistilBERT, BERT, GPT-2). Lab endpoints expose side-by-side token comparisons without loading full models on every health check.

Word embeddings

core.embeddings.glove loads GloVe vectors or falls back to a deterministic synthetic vocabulary for offline demos. glove_bootstrap.py caches vector files under GLOVE_CACHE_DIR on startup when WEEK2526_BOOTSTRAP_MISSING=true.

Character sequence model

core.sequence_models.char_lstm implements a character-level LSTM for text generation. Checkpoints live under artifacts/sequence/char_lstm.pt after a lab train call.

Sentiment stack

The sentiment module spans vocabulary building (vocabulary.py), Bi-LSTM architecture (model.py), training (training.py), lexicon scoring (lexicon.py), and inference (inference.py). Artifacts: artifacts/sentiment/best_model.pt, vocab.json, and metrics.json.

Intent classification

core.intent.classifier trains a DistilBERT head over six support intents. core.intent.lightweight provides keyword-based routing when no checkpoint exists or when WEEK2526_LIGHTWEIGHT_MODE=true. Checkpoint path: artifacts/intent/intent_classifier.pt.

Response selection

core.response_selection.selector maps intent + sentiment score to template replies, optionally reranking with embedding similarity when ML artifacts are present.

Product NLP engine