Kafka vs MQTT: Device Protocol or Streaming Log, and Why Plants Use Both

Kafka vs MQTT is one of the most common questions in plant data architecture, and it is usually the wrong contest. MQTT is a lightweight publish and subscribe protocol built for many small devices on unreliable networks: a client publishes a state change to a broker, and the broker pushes it to subscribers. Apache Kafka is a distributed, durable log: producers append events to partitioned topics, and consumers pull them at their own pace and can replay history. One moves current state off the plant floor; the other keeps and serves event history to the enterprise.

So the practical answer for most manufacturers is both, each where it fits: MQTT at the edge and in the plant, Kafka (or a cloud equivalent) in the data centre or cloud, with a deliberate bridge between them. This page compares the two on the same dimensions and shows that pattern. It is part of our event-driven architecture guide.

Kafka vs MQTT at a glance

Dimension	MQTT	Apache Kafka
What it is	A messaging protocol, implemented by many brokers	A distributed log platform with its own wire protocol
Standard and owner	OASIS standard; 3.1.1 is also ISO/IEC 20922:2016 (OASIS)	Apache Software Foundation project, Apache 2.0 licence
Delivery model	Broker pushes to connected subscribers	Consumers pull from the log and track their own offset
Persistence	Until delivered; retained last value per topic; queues for persistent sessions	Every event kept for a retention period, by time or size
Replay	No standard replay	Any consumer can rewind to an earlier offset
Topics	Deep hierarchy, + and # wildcards, created on publish	Flat named topics, split into partitions, usually created deliberately
Ordering	Per topic and client session	Per partition
Delivery guarantees	QoS 0, 1 and 2 between client and broker	At least once by default; exactly once within Kafka with idempotence and transactions (Kafka)
Client footprint	Small enough for gateways, controllers and embedded devices	Heavier clients, normally on servers
Device features	Last will message, keep-alive, session state	None of these; built for applications
Natural home	Plant floor, edge, unified namespace	Data centre or cloud, data platforms, stream processing

Read the table as two job descriptions. MQTT answers “what is the state of this machine now, and tell me when it changes”. Kafka answers “give me every event of this kind for the last thirty days, in order, and let me process it again tomorrow”.

How MQTT fits the plant floor

MQTT was designed in 1999 for telemetry over constrained links, and its features follow from that: a small binary header, a persistent TCP connection with keep-alive, a last will message that announces when a device drops off, and retained messages that give a new subscriber the current value immediately. The Sparkplug specification adds a topic namespace, a payload format and birth and death certificates for OT data on top. We cover MQTT itself in depth in our guide to MQTT in manufacturing, so this page does not repeat it.

The plant consequence: an edge gateway can report state by exception for thousands of tags with little bandwidth and little load on the controllers, and SCADA, MES and dashboards see the change in real time.

How Kafka fits the data centre and cloud

Kafka stores each topic as a set of partitions, each an append-only log replicated across brokers. Producers write; consumers in a consumer group divide the partitions between them and commit their position. Because events stay on disk for the retention period, any number of independent consumers can read the same history, and a new one can start from the beginning. Kafka 4.0, released in March 2025, runs only in KRaft mode without ZooKeeper (Kafka upgrade notes) and introduced Queues for Kafka (KIP-932) in early access, which adds point-to-point consumption alongside the log.

The plant consequence: quality, reliability and data science teams can analyse months of production events, rebuild a model from history, or add a new consumer without asking the plant to resend anything.

Why neither replaces the other

Kafka on the plant floor is the wrong shape. Each device would need a Kafka client, connections to several brokers, and topics planned in advance. There is no last will to tell SCADA a gateway died, and no retained current value for a dashboard that has just opened. Thousands of fine-grained tag topics also strain a platform designed around fewer, larger partitioned topics.

MQTT as the data platform is the wrong shape too. A broker deletes a message once it is delivered. If the analytics job was down on Tuesday, Tuesday’s events are gone unless a persistent session held them, and nobody can replay last quarter. Some MQTT brokers add persistence features, but they are vendor extensions, not part of the standard.

The common pattern: MQTT at the edge, Kafka in the core

Ways to build the bridge. A Kafka Connect MQTT source connector subscribes to the plant broker and writes to Kafka. Several MQTT brokers ship their own Kafka integration. A proxy can accept MQTT and write to Kafka directly. An enterprise broker that speaks both protocols can sit between them; Solace’s Kafka integration is one example. Open-source UNS stacks such as the United Manufacturing Hub embed a Kafka-compatible buffer (Redpanda) inside the edge stack (UMH docs).

The mapping decision matters more than the tool. There are three common choices:

Mapping	Example	Good for	Watch out for
One Kafka topic per MQTT topic	site1.packaging.line3.filler.state	Small estates, easy debugging	Topic explosion; thousands of partitions
One Kafka topic per event type, MQTT path as key	Topic machine-state, key site1/packaging/line3/filler	Most plants: order per asset, few topics	Consumers filter by key, not by topic wildcard
One Kafka topic per site	Topic site1-events, key by asset	Data residency, per-site retention	Mixed event types need a schema with a type field

In our IT/OT integration experience, the second option ages best: it keeps order for each asset, keeps the topic count manageable and lets the data team reason about event types rather than tag paths.

An illustrative scenario

A manufacturer with six plants runs an MQTT broker per site behind Ignition and a dashboard layer. The corporate data team wants machine states and quality results in its lakehouse, and reliability engineering wants to replay a year of stop events. Each plant bridges two event types, state and quality, into Kafka, keyed by asset. When the WAN to plant four fails for three hours, the plant keeps running on its local broker, and the bridge catches up when the link returns. Neither the controllers nor SCADA know Kafka exists. That separation is the point.

When Kafka does belong at a site

There are exceptions to “Kafka in the core”. A large site with its own analytics (a refinery, a semiconductor fab, a site with an on-premises data platform) may run a Kafka cluster locally, because the data volume is high and the consumers are on site. An edge buffer is the other case: some UNS stacks place a Kafka-compatible log at the plant so that events survive a long WAN outage and can be replayed into the cloud later. Redpanda, a Kafka-API-compatible broker written in C++ as a single binary, is often chosen for that role because of its smaller operational footprint.

Even then, devices and gateways usually keep speaking MQTT or OPC UA to the site, and the log sits one layer up. The test is who connects to it. If the answer is controllers and gateways, MQTT is the better fit; if it is applications and data pipelines, the log is.

Limits and open questions

• Two platforms, two skill sets. MQTT brokers are often run by controls teams; Kafka by IT or data engineering. The bridge needs one owner.
• Exactly once does not survive the bridge. Guarantees are per hop. Use unique event ids and idempotent consumers.
• QoS 2 is not universal. Some brokers and cloud services accept QoS 2 but deliver at QoS 1. Check conformance notes, including those of enterprise brokers that speak MQTT.
• Sparkplug payloads are Protobuf. Bridging Sparkplug into Kafka means either decoding it at the bridge or teaching every consumer to decode it.
• Kafka is changing. Queues for Kafka and diskless designs are moving its boundaries, and IBM now owns Confluent, the largest commercial Kafka vendor (closed March 2026). Recheck vendor roadmaps before a long commitment.

How to evaluate a Kafka and MQTT design

• Which consumers need current state only, and which need history or replay?
• Where does each plant buffer events while the WAN or Kafka is unavailable?
• How do MQTT topics map to Kafka topics and keys, and is that written down?
• Which events cross the DMZ, and who approves a new one?
• How are payload schemas versioned on both sides?
• Who owns the bridge, monitors its lag and is paged when it stops?
• What retention does the data team actually need, and what does it cost?

Deeper

There are no child pages under this guide yet. These go further:

• Solace vs Kafka: event messaging against event streaming, from one vendor’s side
• Solace Kafka integration: how one enterprise broker bridges to Kafka
• UNS Broker: choosing the MQTT side of this pattern
• What Is an Event Mesh: the alternative to bridging plant by plant
• Plant data: how this fits our wider view of historians, MES and data platforms

Back to the event-driven architecture hub.