Your team has a working Kafka platform. A vendor or an engineer proposes adding ksqlDB or Flink SQL to give the data team a real-time analytics layer. The pitch is straightforward: SQL on streams is familiar, it is productive, it shortens the path from raw events to business insight. That part is true. What the pitch leaves out is that streaming SQL is not a query engine—it is a persistent infrastructure commitment. Before you approve the initiative, you need to understand what you are deciding to own.

The difference between batch SQL and streaming SQL is not just latency. It is that streaming SQL queries never stop. They run indefinitely, accumulate state, and require operational attention even when nobody is changing them. A batch job runs and finishes. A streaming query is a service.

The decision hiding inside the technology choice

There are two very different things you can do with SQL on streams. The first is operational enrichment: every transaction that enters the payment system is enriched with customer data and screened against a sanctions list before a response is sent. This is latency-sensitive, operationally critical, and requires sub-200-millisecond outcomes. The second is analytical aggregation: build a real-time fraud scoring layer or a running daily spend total for SAMA compliance. This is also latency-sensitive but tolerates a few seconds and can sometimes replay safely after an outage.

Both use cases have value. They are not the same decision. Operational enrichment requires you to staff the streaming layer with the same rigour as your payment processing systems—on-call, runbooks, SLAs, a tested rollback procedure for query changes. Analytical aggregation is closer to a data pipeline and can be operated with slightly different procedures. Most teams start with the analytics pitch and find themselves running operational infrastructure they are not ready to support. The VP role here is to ask the question before the architecture is built: what is the on-call model for this new system?

What the regulator cares about

SAMA does not prescribe technology, but two aspects of a streaming SQL deployment surface in every integration audit. The first is schema lineage: can you demonstrate that the data used to make a sanctions screening decision at 14:32:07 on a given date matched the official sanctions list as it stood at that time? Temporal joins in Flink SQL can answer this question directly. A naive implementation that looks up the current sanctions table cannot. The difference is architectural, not cosmetic, and you need to make it explicit in your design documentation before the audit, not during it.

The second is replay and reconciliation. If a streaming query produces incorrect output due to a code defect, can you replay the affected window and regenerate the correct output? Batch systems handle this naturally; streaming systems require deliberate design—checkpoint retention, topic retention policy, and a tested replay procedure. SAMA’s data governance framework expects you to have this, and the answer “we would need to rebuild from source systems” is accepted once, not twice.

The team capability question you must ask first

SQL on streams is more accessible than writing Kafka Streams topologies in Java. It is not as accessible as writing a SQL query against a data warehouse. The difference is that streaming SQL requires its author to understand event time versus processing time, watermarks, window types, join semantics, and the difference between a stream and a table. These are not hard concepts—but they are concepts that most data analysts who are comfortable with SQL have never needed. A team that deploys streaming SQL without this foundation will produce queries that work correctly in testing and fail in production when events arrive late, when a topic is replayed, or when a window advances unexpectedly.

The investment in capability must precede the investment in infrastructure. Two or three engineers who deeply understand stream processing semantics can run a production streaming platform. A team of ten who each understand it partially will operate it badly. Before approving the initiative, confirm that you have the right two or three people, and that they have enough protected time to build what needs to be built correctly.

The trade-off the team will not surface to you

There is a specific operational trade-off in streaming SQL that engineers rarely escalate because it looks like a technical detail: changing a persistent streaming query is expensive. Adding a column, modifying a join condition, or adjusting a window size may require resetting the consumer offset and replaying from the beginning of the topic retention window. On a high-volume production topic, this replay can take hours. It means that a query change that would be a five-minute deployment in a batch system is a four-hour maintenance window in a streaming system. If your team is planning to iterate rapidly on the analytics logic—as analytics teams typically do—this cost compounds quickly.

The mitigation is to design for immutability: versioned output topics, new query names for each logic change, fan-out to downstream consumers from the new version once it has caught up. This works, but it requires discipline and generates operational sprawl if left unmanaged. Your team needs a governance model for streaming query lifecycle, not just a technical deployment pattern.

What to ask before approving the initiative

Four questions worth asking explicitly before a streaming SQL initiative is approved:

What is the on-call model? Name the person who holds the pager when a streaming query falls behind on a Friday night. If the answer is unclear, the platform is not ready to run in production.

What is the query change procedure? How long does a query change take in the worst case? Is there a tested rollback? Who approves it?

Can we demonstrate sanctions lineage for a historical transaction? If the answer requires manual reconstruction from multiple systems, the architecture needs temporal joins before go-live, not after.

What stays in batch? The streaming layer should not try to replace everything. Daily regulatory files, SOX close reports, and ad-hoc analytical queries belong in batch. Protect the batch layer from scope creep and protect the streaming layer from use cases it was not designed for.

The teams that get streaming SQL right are not the ones with the most sophisticated technology. They are the ones where the leadership made deliberate decisions about scope, staffed it with the right people, and built the operational model before the first query went live. The technology is available. The discipline is the differentiator.

For the technical depth behind these trade-offs—ksqlDB vs Flink SQL, stateful aggregations, temporal joins, and windowing semantics—see the companion Lab article: Data Streaming: ksqlDB & Flink SQL for Stream Analytics.