Message queues vs Kafka: queues vs logs

Engineers often ask whether they should use RabbitMQ / SQS or Apache Kafka. While both decouple producers from consumers, their architectural models are completely different:

Traditional Message Queue (RabbitMQ, SQS): A To-Do List. Once a worker processes a message and sends an ACK, the message is permanently deleted.
Distributed Event Log (Kafka, Pulsar): An Append-Only Ledger. Messages are written sequentially to disk, kept for days or weeks, and multiple independent consumers track their own reading offsets.

Message Queue (RabbitMQ):  [ Msg 1 ] [ Msg 2 ] ──> Consumer ──> (Deleted upon ACK)
Distributed Log (Kafka):   [0][1][2][3][4][5]  ──> Consumer A (Offset: 3)
                                               ──> Consumer B (Offset: 5, Analytics)

How Message Queues work (Smart Broker, Dumb Consumer)

In systems like RabbitMQ:

The broker is responsible for state tracking, routing keys, fan-out exchanges, dead-letter queues, and acknowledging message delivery.
When a consumer pulls a message, the broker marks it invisible.
If the consumer succeeds, the message is removed. If it crashes, the broker re-queues it for another worker.
Ideal for: Background tasks, sending transactional emails, video encoding jobs, asynchronous work distribution.

How Kafka works (Dumb Broker, Smart Consumer)

Kafka treats topics as ordered partitions stored on disk:

Producers append events to the end of a partition: { offset: 42, payload: { order_id: 101 } }.
Consumers are grouped into Consumer Groups. Each consumer maintains its own pointer (offset) in the stream.
Messages are not deleted when read. They remain available for other consumer groups (e.g. Analytics, Billing, Notification service) to read at their own pace.
If a bug is fixed in production, a consumer group can rewind its offset to 3 days ago and replay history from scratch.
Ideal for: Event sourcing, real-time analytics, microservice event streaming, IoT telemetry.

Key Architectural Differences

Feature	Message Queues (RabbitMQ / SQS)	Event Streams (Kafka / Redpanda)
Data model	Transient FIFO message buffer	Append-only partitioned commit log
Message lifecycle	Deleted after successful consumption	Retained on disk by time or size (e.g. 7 days)
Replayability	❌ No replay (once ACKed, it’s gone)	✅ Yes (can rewind offsets to any point in time)
Ordering	Per-queue FIFO (can mix during concurrency)	Strictly ordered within each partition
Routing complexity	High (Topics, headers, fanouts, dead-letters)	Simple (Partitioning by message key)
Throughput	~10k–50k msgs/sec	~1,000,000+ msgs/sec