7.6 Event-Driven Messaging with Apache Kafka

← Back to Chapter 7: Microservices & Spring Cloud

Synchronous HTTP calls couple services temporally (both must be online simultaneously). Event-Driven Architecture (EDA) with Apache Kafka decouples services chronologically, providing massive throughput, persistence, and backpressure buffering.

---

1. Kafka Architectural Concepts

┌─────────────────────────────────────────────────────────────┐
│                       KAFKA BROKER                          │
│                                                             │
│ Topic: "order-events" (e.g. 3 Partitions)                   │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Partition 0: [msg 0][msg 1][msg 2][msg 3]...            │ │
│ ├─────────────────────────────────────────────────────────┤ │
│ │ Partition 1: [msg 0][msg 1][msg 2]...                   │ │
│ ├─────────────────────────────────────────────────────────┤ │
│ │ Partition 2: [msg 0][msg 1][msg 2][msg 3][msg 4]...     │ │
│ └─────────────────────────────────────────────────────────┘ │
└──────────────▲───────────────────────────┬──────────────────┘
               │ Produce                   │ Consume
┌──────────────┴──────────┐     ┌──────────▼──────────────────┐
│     Producer Service    │     │  Consumer Group: "billing"  │
│    (KafkaTemplate)      │     │     (@KafkaListener)        │
└─────────────────────────┘     └─────────────────────────────┘

• Topic: An append-only distributed log partitioned across nodes.
• Partition: Physical unit of parallelism and ordering guarantee. Messages with the same Partition Key are strictly ordered.
• Consumer Group: Set of consumer instances collaborating to consume a topic. Each partition is assigned to exactly one consumer within a group.

---

2. Producing Events with KafkaTemplate

@Service
public class OrderEventProducer {

    private static final String TOPIC = "order-events";
    private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;

    public OrderEventProducer(KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishOrderCreated(OrderCreatedEvent event) {
        // Using orderId as partition key guarantees in-order delivery for that order!
        CompletableFuture<SendResult<String, OrderCreatedEvent>> future = 
            kafkaTemplate.send(TOPIC, event.orderId().toString(), event);

        future.whenComplete((result, ex) -> {
            if (ex == null) {
                System.out.println("Message sent successfully to offset: " + 
                    result.getRecordMetadata().offset());
            } else {
                System.err.println("Failed to publish event: " + ex.getMessage());
            }
        });
    }
}

---

3. Consuming Events with @KafkaListener

@Service
public class EmailNotificationConsumer {

    @KafkaListener(
        topics = "order-events",
        groupId = "notification-service-group",
        concurrency = "3" // 3 parallel consumer threads matching 3 partitions!
    )
    public void handleOrderEvent(@Payload OrderCreatedEvent event, 
                                 @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
                                 @Header(KafkaHeaders.OFFSET) long offset) {
        System.out.printf("Received order event [%s] from partition %d at offset %d%n",
                event.orderId(), partition, offset);
        // Process email notification
    }
}

---

4. Idempotency & Dead Letter Queue (DLQ)

• Idempotent Consumers: Because network retries can deliver messages more than once ("at least once delivery"), consumers must track processed message IDs in a DB table or Redis cache to avoid processing duplicates.
• Dead Letter Queue (DLQ): When a message repeatedly fails processing (poison pill), route it to a order-events-dlq topic for offline analysis rather than blocking the consumer partition permanently.