LLM Serving & Inference

Serving large language models means managing GPUs, batching, caching and latency. These lessons cover inference servers, prompt caching, quantisation, streaming and cost control.

16 lessons across 2 courses

LLM Serving & Inference lessons by course

MLOps Engineering – From Training to Automated Production

View course →
  1. Day 57: Inference vs. Training: Why Inference is the “Hidden 90%”
  2. Day 60: No-Wait Batching: Case Study of Snowflake’s Optimized Inference Engine
  3. Day 62: Asynchronous Inference: Decoupling Prediction from Consumption
  4. Day 65: Precision Engineering: Quantization (FP16, INT8) and its Impact on Scale
  5. Day 66: Paged Attention and KV Caching: Scaling LLM Inference to Millions of Tokens
  6. Day 68: Memory Leak Prevention in Long-Running Python Inference Services

Practical AI System Architecture: Building Intelligent Systems with LLMs, RAG, and Agent Frameworks

View course →
  1. Day 9 : Cost-Aware Design: Managing LLM API Expenses.
  2. Day 27 Scaling Agentic Workloads: Inference Optimization and Distributed Systems Design

Related topics

AI Agents (27) MLOps (9) Embeddings & Vector Databases (6) RAG: Retrieval-Augmented Generation (4) All topics →