A couple of years ago I wrote about how our Data Warehouse hit the performance wall and the nightmare it was having ETL (Extract, Transform, and Load) processes take all night. No matter how much we optimize the hardware, the root problem is still there: the "batch" model means data always arrives hours late. And today, for example, the marketing department doesn't want to know what happened yesterday, they want to know what users are clicking right now.

Looking into how to solve this latency issue, I stumbled upon a tool born in the bowels of LinkedIn that has become the de facto standard for data "streaming": Apache Kafka.

It's not a simple message queue

At first, my mind assimilated it as a traditional queue system like RabbitMQ or ActiveMQ. "A publisher sends a message, the queue holds it, and a consumer reads it and deletes it". False.

Kafka is not a queue, it's a Distributed Commit Log. Messages (events) are written to disk sequentially at the end of a file. When a consumer reads a message, it is not deleted. It stays there. This allows the billing system and the fraud system to read exactly the same data stream simultaneously, each at their own pace.

At the code level, writing an event in Python is ridiculously easy using the kafka-python library:

from kafka import KafkaProducer
import json

# Conectamos a nuestro clúster de Kafka
productor = KafkaProducer(
    bootstrap_servers=['kafka-nodo1:9092', 'kafka-nodo2:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# Simulamos un evento capturado en nuestra web
evento_clic = {
    "id_usuario": 84752,
    "accion": "añadir_carrito",
    "id_producto": "ZAP-405",
    "timestamp": "2016-05-14T10:15:30Z"
}

# Mandamos el evento al "topic" (tema) llamado 'clicks_web'
productor.send('clicks_web', value=evento_clic)

# Forzamos el envío asíncrono
productor.flush()
print("Evento inyectado en el torrente circulatorio.")

The magic of Kafka's performance (capable of processing millions of messages per second) lies in how it treats the hard drive. Instead of doing random seeks, it does brutal sequential writes and uses "Zero-Copy" techniques at the OS level to move data directly from disk to the network card without going through the application.

Reflection: The inverted database

Kafka proposes a shift in architectural paradigm. Until now, the Relational Database was the "System of Truth", and we pulled data from there.

With Kafka, the architecture is inverted. The immutable "Event Log" flowing through Kafka becomes the absolute source of truth. Data no longer rests passively in tables waiting to be queried; data is in constant motion.