Digital Transformation | June 2026
The entire history of data analytics described in the preceding eleven parts was built on batch processing. Data accumulated, was processed overnight, and was available for analysis the next morning. The latency minutes, hours, sometimes days was accepted as a structural constraint. Decision-makers worked from yesterday’s numbers because there was no architectural path to today’s.
By 2015, that constraint was no longer acceptable for an expanding set of use cases. Fraud detection systems that took fifteen minutes to flag a suspicious transaction were useless the fraudster was gone. Recommendation engines that updated product suggestions based on a user’s behaviour from last night failed to capitalise on what that user was doing right now. Operational dashboards that showed the state of a manufacturing line from two hours ago could not support real-time process control. Batch processing had defined the architecture of analytics for fifty years. The streaming era was about to redefine it.
Apache Kafka: The Universal Event Backbone
Apache Kafka was developed at LinkedIn to solve a specific problem: the need to reliably distribute high-velocity event streams page views, clicks, user actions, system events between a large number of producers and consumers with low latency and no data loss. The solution, described in the paper “Kafka: a Distributed Messaging System for Log Aggregation” (Kreps, Narkhede & Rao, 2011), was architecturally elegant: a distributed commit log in which producers appended events to named topics, and consumers read events at their own pace, maintaining their position in the log as a simple integer offset.
The key insight in Jay Kreps’s influential 2013 essay “The Log: What every software engineer should know about real-time data’s unifying abstraction” was that the append-only log is the fundamental data structure underlying databases, version control systems, and distributed consensus protocols. A log is ordered, immutable, and durable properties that make it the ideal substrate for reliable event streaming. Kafka made the log a first-class infrastructure component, available as a managed service for any application that needed to produce or consume events.
Kafka’s architecture organises events into topics, partitioned across a cluster of broker nodes. Each partition is an ordered, immutable sequence of records, replicated across multiple brokers for fault tolerance. Producers append to the tail of a partition; consumers read from any position, maintaining their offset independently. Multiple consumer groups can read the same topic simultaneously a critical property that allows a single Kafka topic to serve as the event source for multiple downstream systems simultaneously: a real-time fraud detector, a batch analytics pipeline, a recommendation engine, and a data lake ingestion job can all consume the same stream independently at their own pace.
Apache Flink: Stateful Stream Processing
Kafka provides durable, ordered event delivery. What Kafka does not provide is computation: aggregating events, joining streams, detecting patterns, or maintaining state across a sequence of related events. That computation is the domain of stream processing engines, of which Apache Flink became the most capable and widely adopted.
Apache Flink, developed at TU Berlin and open-sourced in 2014, distinguishes itself from earlier stream processing systems through its handling of time. There are two time concepts in streaming: event time (when an event actually occurred, encoded in the event itself) and processing time (when the event is processed by the system). Systems that use processing time produce incorrect results when events arrive out of order or late which they frequently do in real-world distributed systems where network delays, mobile devices going offline and reconnecting, and log buffering introduce latency and reordering.
Flink operates on event time with a mechanism called watermarks periodic assertions about the completeness of the event stream up to a given timestamp. A watermark at time T means “all events with timestamps earlier than T have been received; it is now safe to emit aggregations for windows closing before T.” This allows Flink to handle late-arriving events correctly: a window aggregate waits for the relevant watermark before computing, rather than emitting as soon as processing time reaches the window boundary.
Flink’s stateful processing the ability to maintain arbitrary state (running totals, recent event counts, session state) across a stream, with automatic checkpointing and recovery made it the platform of choice for complex event processing: fraud detection (maintaining a running count of declined transactions per card over a sliding window), recommendation systems (tracking a user’s recent interactions), and operational analytics (monitoring production line metrics with real-time alerting).
Lambda and Kappa Architectures
As organisations began combining batch and streaming processing, two architectural patterns emerged to manage the complexity.
The Lambda architecture, proposed by Nathan Marz around 2011, maintained two separate processing paths: a batch layer that processed all historical data to produce accurate, complete views (slowly, with full correctness), and a speed layer that processed the most recent data in real time (quickly, with approximate or incremental results). A serving layer merged the outputs of both layers to answer queries. Lambda’s strength was resilience: the batch layer could reprocess and correct the speed layer’s approximations. Its weakness was complexity: maintaining two code paths for the same logic in two different frameworks was expensive to develop and operate.
The Kappa architecture, proposed by Jay Kreps in 2014, simplified this by eliminating the separate batch layer. Everything was stream processing historical data was replayed through the same streaming pipeline as real-time data. The approach was elegant in theory and practical in systems where reprocessing historical data through Kafka was feasible. It became increasingly viable as stream processing frameworks (Flink, Spark Structured Streaming) gained sufficient maturity and performance to serve as the single processing paradigm for both historical and real-time workloads.
OLAP on Streams: Druid, Pinot, and Materialize
Kafka and Flink solve the processing problem; they do not solve the analytical query problem. Interactive BI dashboards need sub-second query response incompatible with the seconds-to-minutes latency of stream processing followed by bulk loading to a data warehouse. A specialised category of real-time OLAP databases emerged to bridge this gap.
Apache Druid (2011, open-sourced by Metamarkets) pre-aggregates ingested events into columnar segments that can be queried with sub-second latency, even over billions of records. Apache Pinot (developed at LinkedIn, open-sourced 2018) provided similar capabilities with a different architecture optimised for freshness data ingested from Kafka was queryable within seconds of arrival. Materialize (2019) took a different approach: a streaming SQL database that maintained materialised views as continuous queries over Kafka topics, updating them incrementally as new events arrived.
References
- Kreps, J., Narkhede, N. & Rao, J. (2011). Kafka: a Distributed Messaging System for Log Aggregation. NetDB Workshop at SOSP 2011.
- Kreps, J. (2013). The Log: What every software engineer should know about real-time data’s unifying abstraction. LinkedIn Engineering Blog.
- Carbone, P. et al. (2015). Apache Flink: Stream and Batch Processing in a Single Engine. IEEE Data Engineering Bulletin, 38(4), 28–38.
- Marz, N. & Warren, J. (2015). Big Data: Principles and best practices of scalable realtime data systems. Manning Publications.
- Kreps, J. (2014). Questioning the Lambda Architecture. O’Reilly Radar.
- Yang, F. et al. (2014). Druid: A Real-time Analytical Data Store. Proceedings of SIGMOD 2014.
- Confluent (2020). ksqlDB: The Streaming Database. Confluent Blog.
