Data Streaming: Kafka Streams & Flink

Overview

Stream processing transforms an unbounded sequence of events into derived facts in real time. The three killer applications in financial services are fraud detection (sliding-window aggregation over card authorisations), real-time payment screening (joining payment events against sanctions lists), and market data enrichment (deriving VWAP, position, P&L from tick streams).

Two engines dominate: Kafka Streams (a library that runs inside your application) and Apache Flink (a separate runtime cluster). IBM packages Kafka as Event Streams; the streaming patterns are the same.

Streams vs Flink

Pipeline topology

A stream processing pipeline is a directed acyclic graph: source topics, processing operators, and sink topics. The runtime maps operators to tasks, tasks to threads, and partitions the work by key.

Windowing

Windows turn an unbounded stream into bounded computations. Four kinds, each with a specific use case:

• Tumbling — non-overlapping fixed-size windows. Use for hourly counts, daily totals.
• Hopping (sliding) — overlapping fixed-size windows that advance by an interval. Use for “how many transactions in the last 5 minutes, updated every 30 seconds.”
• Session — windows defined by activity gaps. Use for grouping a customer’s actions into a session.
• Global — one window for the whole stream. Use only with custom triggers.

State stores

Stateful operators (joins, aggregations, deduplication) keep state in a local RocksDB instance. Kafka Streams replicates this to a changelog topic — on failure, a new instance reads the changelog and rebuilds. Flink uses incremental checkpoints to S3 or HDFS.

State size is the variable nobody plans for and everybody fights with. A 10 GB customer master KTable times 10 partitions is 100 GB of disk on each instance. Plan capacity at the state level, not at the throughput level.

Building a pipeline

1. Define the topology

   Source topics, transformations, sinks. Draw the DAG before writing code — it’s the design document.

2. Co-partition the inputs

   Stream-table joins require the stream and table to be partitioned by the same key with the same partition count. Repartition explicitly when this isn’t already true.

3. Configure state

   Local disk size for RocksDB, changelog retention (default compact), and rocksdb-tuning if state is large (block cache, write buffer).

4. Set processing guarantee

   at_least_once for most flows; exactly_once_v2 when downstream cannot tolerate duplicates and side effects are Kafka-only.

5. Wire metrics & alerts

   Lag per task, state store size, restore time on rebalance, processing rate. Without these you’ll discover problems via business complaints.

Example: fraud-detection topology in Kafka Streams

public Topology build() {
  StreamsBuilder b = new StreamsBuilder();

  // Source: card authorisations, keyed by cardId
  KStream<String, CardAuth> auths =
      b.stream("card-auth", Consumed.with(Serdes.String(), authSerde));

  // KTable: customer master, compacted by customerId
  KTable<String, Customer> customers =
      b.table("customer-master", Consumed.with(Serdes.String(), customerSerde));

  // Stream-table join: enrich auth with customer profile
  KStream<String, EnrichedAuth> enriched = auths
      .selectKey((k, v) -> v.getCustomerId())
      .leftJoin(customers, EnrichedAuth::new);

  // 5-minute sliding window count per customer
  KTable<Windowed<String>, Long> recentCount = enriched
      .groupByKey()
      .windowedBy(SlidingWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5)))
      .count(Materialized.as("recent-count-store"));

  // Branch: high-velocity customers go to fraud-alerts
  enriched
      .leftJoin(recentCount.toStream().selectKey((k, v) -> k.key()).toTable(),
                (e, c) -> e.withRecentCount(c == null ? 0 : c))
      .split()
      .branch((k, v) -> v.getRecentCount() > 10,
              Branched.withConsumer(s -> s.to("fraud-alerts")))
      .defaultBranch(Branched.withConsumer(s -> s.to("enriched-auth")));

  return b.build();
}

Stream joins

Three join semantics, each with different state requirements:

• Stream ↔ KTable: stateless on the stream side, stateful on the table side. The most common pattern — enrichment.
• KTable ↔ KTable: stateful on both sides. Use for derived materialised views.
• Stream ↔ Stream (windowed): stateful on both sides for the window duration. Use for matching events that should arrive close in time (auth + clearing).

Exactly-once

Kafka Streams supports exactly_once_v2 end-to-end: consume from input topics, update state, produce to output topics, all atomically. Latency cost is around 30–50 ms per commit, throughput cost is around 5%. For most stream pipelines that’s a fair trade.

application.id=fraud-detection
bootstrap.servers=kafka:9092
processing.guarantee=exactly_once_v2
commit.interval.ms=100
num.stream.threads=8

# State store tuning
state.dir=/var/lib/kafka-streams
rocksdb.config.setter=com.acmebank.streams.RocksTuner

# Restore tuning — speeds up rebalance recovery
num.standby.replicas=1
acceptable.recovery.lag=10000

HA & rebalancing

Stream applications scale horizontally up to the partition count of the input topics. Each instance owns a subset of partitions; on failure or scale-up Kafka rebalances assignments. The rebalance is the operational soft underbelly — everything is fine until you scale.

Common pitfalls

When not to use

Streaming is the right tool when latency budgets are sub-second and the work is per-event. It is the wrong tool when:

• The query is interactive ad-hoc analytics. Use a query engine (Presto, Snowflake, BigQuery) over a data lake. Stream processing is for known, fixed pipelines.
• The state requirement exceeds available local disk. A 5 TB lookup table doesn’t belong in RocksDB on each instance. Use an external store (DynamoDB, Cassandra) and accept the latency.
• Backpressure is intolerable. Streaming engines pause sources when downstream slows; for a hard real-time SLA where late processing is worse than dropping, use a different tool.
• The pipeline is a one-off. A nightly Spark job is simpler than a Streams app for a once-a-day rollup. Don’t over-engineer.