Building SentinelStream: A Real-Time SPC & Anomaly Detection Pipeline with PySpark, Kafka, and ClickHouse

As industrial IoT networks and high-frequency financial platforms grow, processing data reactively is no longer enough. Waiting for an overnight batch job to discover a failing asset or a systemic drift means lost revenue, broken SLA agreements, and operational blindness.

To solve this, I built SentinelStream—an end-to-end, open-source stream processing pipeline designed to ingest high-frequency telemetry, calculate dynamic rolling baselines on the fly, and execute multivariate anomaly detection in real time.

Here is a deep dive into the architecture, the technical bottlenecks I encountered, and how I solved them using Apache Kafka, PySpark Structured Streaming, ClickHouse, and MinIO.

🗺️ The Architecture: Separation of Concerns

A resilient streaming architecture must separate long-term analytical storage from instant, automated data mutation and quarantine mechanics. SentinelStream splits these demands across four distinct layers:

[Data Generator (Python)] 
       │ 
       │ (IoT Sensor Payload) 
       ▼
[Apache Kafka] (Message Broker / Ingestion)
       │ 
       ▼
[PySpark Streaming] (Real-Time Compute Engine) 
       ├── Task 1: Moving Window Metrics (Avg, StdDev) -> SPC 3-Sigma Limits
       ├── Task 2: Real-time Z-Score Evaluation (Instant Outlier Detection)
       └── Task 3: Isolation Forest Micro-batching (Drift & Multi-variable Anomalies)
               │
               ├──► [Anomalies / Corrupt Payloads] ──► [Dead Letter Queue] ──► [MinIO/S3 Logs]
               └──► [Clean / Validated Data] ──► [ClickHouse OLAP Store] ──► [Grafana]
    

1. Ingestion & Buffering: Apache Kafka absorbs raw telemetry from simulated IoT edge devices. Partitioning records by device_id guarantees horizontal scalability.

2. Real-Time Stream Processing: PySpark acts as our core compute matrix, processing streams using three layered, progressive analytical tests.

3. The Clean Path (OLAP Analytics): Clean, validated data is streamed down to ClickHouse, an exceptionally fast columnar database designed for time-series aggregation.

4. The Quarantine Path (Dead Letter Queue): Flagged data drops immediately out of the hot production path into a Kafka Dead Letter Queue (DLQ) topic, where a dedicated archiver pushes alerts down to a MinIO S3 bucket for auditing.

🛠️ Deep Dive: The Three Analytical Streaming Tasks

Task 1: Overcoming Unbounded State with Sliding SPC Windows

Statistical Process Control (SPC) uses traditional 3-Sigma control bounds (UCL = μ + 3σ and LCL = μ - 3σ) to flag anomalous events. In a streaming environment, calculating a rigid global mean is useless—the engine must track a moving historical baseline.

Using PySpark, I configured a 5-minute sliding window evaluated every minute. To prevent infinite memory consumption from storing endless state arrays, I implemented a strict 10-minute event-time watermark:

# Prevent infinite state memory accumulation
parsed_stream = raw_stream.withWatermark("timestamp", "10 minutes")
# Group by windows to calculate moving baselines
windowed_stats = parsed_stream.groupBy

Task 2: Dynamic Point-In-Time Z-Score Checks

While Task 1 tracks historical baselines, Task 2 executes stateless calculations to capture sudden, extreme telemetry spikes. By performing an immediate stream-to-stream temporal join against the baseline matrix, the incoming values can instantly be scored against their exact temporal cohorts:

$$\text{Z-Score} = \frac{\vert\text{Metric Value} - \text{Moving Avg}\vert{}}{\text{Moving StdDev}}$$

If an incoming value displays an impossible statistical leap (Z > 3.5), it is flagged immediately to protect down-stream analytics from structural contamination.

Task 3: Solving the PySpark MLlib Bottleneck via Vectorized Apache Arrow

Here is where traditional data engineering patterns struggle. Standard PySpark MLlib lacks a native IsolationForest model to detect multivariate anomaly drift (e.g., when temperature and vibration are both technically within normal bounds, but their relationship is uncharacteristic).

To embed Scikit-Learn's highly optimized IsolationForest without ruining streaming throughput, I leveraged Apache Arrow via PySpark mapInPandas.

This lets Spark serialize micro-batches using zero-copy memory transfers, dropping straight down to low-overhead Python memory for vectorized execution:

# Implementation detail placeholder (Actual code omitted but structure shown)
import pyspark.sql.functions as F
from pyspark.ml.feature import VectorAssembler

# Example leveraging Arrow via mapInPandas:
result_df = parsed_stream.mapInPandas(
    lambda row: run_isolation_forest_on_batch(row), 
    vector_cols # Assumes vector assembly happens here

def run_isolation_forest_microbatch(iterator: Iterator[pd.DataFrame]) -> Iterator[pd.DataFrame]:
    for pdf in iterator:
        if pdf.empty: yield pdf; continue
        
        # Scikit-learn feature slice
        features = pdf[['metric_value', 'z_score']].fillna(0.0)
        model = IsolationForest(n_estimators=100, contamination=0.05, random_state=42)
        
        # -1 = anomaly, 1 = normal
        predictions = model.fit_predict(features)
        pdf['is_iforest_anomaly'] = [1 if pred == -1 else 0 for pred in predictions]
        yield pdf


)

📈 Engineering for Scale: The Storage Footprint

ClickHouse Columnar Database Layout

For the clean telemetry path, storing millions of raw text keys breaks performance profiles. I deployed ClickHouse with a ReplacingMergeTree engine ordered by (metric_name, device_id, timestamp) while wrapping repetitive keys inside LowCardinality(String) types. This architecture heavily compresses redundant metadata, reducing disk I/O to an absolute minimum during dashboard sweeps.

Hive-Style Cold Storage Storage For DLQ Logs

When anomalies strike, the pipeline exports payloads to MinIO using a strict Hive-style directory architecture:

s3://sentinel-dlq-archive/year=2026/month=10/day=05/sensor-device-01/

This decision guarantees that long-term audits or ad-hoc debugging queries can run highly scoped external table engine tools (like Snowflake or AWS Athena) without executing expensive global bucket scans.

💡 Key Key Takeaways

Building SentinelStream highlighted three fundamental tenets of production data architecture:

• Never allow infinite states: Streaming state management requires rigid execution of watermarks to keep system memory stable over time.
• Optimize IPC Boundaries: Python data science libraries can coexist natively inside Java-based PySpark grids if you deploy Apache Arrow (mapInPandas) to remove serialization boundaries.
• Quarantine early: Dropping corrupted or heavily anomalous payloads into a DLQ immediately prevents hot storage systems from scaling up costs unnecessarily.

🛠️ Reproduce It Yourself

The complete project infrastructure—including a comprehensive multi-container docker-compose.yml, automated infrastructure provisioning scripts, and a real-time IoT anomaly generator—is open-source and available on GitHub.

👉 Check out the code here: https://github.com/Logic-HQ/sentinel-stream-spc

Let's connect! If you have built similar real-time validation layers or are actively solving streaming bottlenecks, let's talk in the comments below.