Lesson 12: Advanced Analytics

Advanced Log Analytics: From Free Text to Root Cause

Summary

• One Spring Boot analytics service turns each log line into templates, severity labels, clusters, forecasts, correlations, root-cause candidates, and ranked fixes

• Algorithms include Drain template mining, TF-IDF, rule-based classification, online centroid clustering, Holt-Winters forecasting, PMI and Pearson correlation, and random walk with restart

• Runs alone on H2 with an inline pipeline, or against Kafka, PostgreSQL, Redis, and Prometheus in distributed mode

• A dashboard and Prometheus metrics show all seven capabilities producing non-zero results under synthetic multi-source traffic

Core Concepts

Log language processing. Free-text lines are noisy. Normalization, entity masking, Drain templates, and TF-IDF turn each message into a stable shape plus distinctive terms. Downstream stages then operate on structure instead of unique strings.

Classification and clustering. A term-frequency classifier assigns severity and category with confidence. An online clustering engine assigns feature vectors to centroids, seeds novel clusters when distance is high, and periodically merges near duplicates.

Predictive rates. Holt-Winters models seasonal event rates per series. A breach detector alerts when the lower prediction band clears a threshold, reducing false alarms from single-point spikes.

Correlation and causation. PMI captures co-occurrence inside sliding windows; Welford accumulators stream Pearson correlation on error rates. Lag-consistent edges feed a directed causal graph. Random walk with restart ranks upstream causes of a symptom service.