Home › Apache Kafka › Apache Kafka producers and consumers Windsurf rules
Apache Kafka producers and consumers Windsurf rules
Windsurf rules for Kafka client code: keys and partitioning, idempotent producers, manual commits after processing, retries and DLQs, and schema evolution.
WindsurfApache Kafka
How to use
Save this as .windsurf/rules/kafka.md in your repository root, commit it, and your agent will pick it up automatically. Edit the commands to match your project.
Note: Review and test before use. Provided as is, without warranty. See Terms.
.windsurf/rules/kafka.mdDownload ---
trigger: model_decision
description: Apply when writing or changing Kafka producer, consumer or stream-processing code
---
# Kafka clients
- Topics, partitions and configs are created by infrastructure code (Terraform/Strimzi/scripts), not by app code at startup. Topic names come from config.
- Keys decide ordering: choose a message key (e.g. `orderId`) so all events for one entity go to the same partition. Never rely on ordering across partitions.
- Producers: `enable.idempotence=true`, `acks=all`, bounded retries with delivery timeout, compression (`lz4`/`zstd`). Check every send result or delivery callback; don't fire and forget.
- Use transactions only when you need exactly-once across read-process-write; otherwise design idempotent consumers.
- Consumers: disable auto-commit, commit offsets only after the message is fully processed (or stored). Processing must be idempotent because redelivery happens.
- Handle poison messages: catch per-record errors, retry with backoff a few times, then publish to a dead-letter topic with error metadata. Never let one bad record block a partition forever.
- Keep poll loops fast: don't do long blocking work between polls beyond `max.poll.interval.ms`; hand off to a worker pool with care for commit ordering.
- Messages have a schema (Avro/Protobuf/JSON Schema with a registry). Evolve compatibly: add optional fields with defaults, never rename or reuse field numbers.
- Include an event ID, type, version and timestamp in each message; consumers ignore unknown fields.
- Graceful shutdown: stop polling, finish in-flight records, commit, then close the consumer and flush the producer.
## Tests
Integration tests with Testcontainers Kafka (or the repo's embedded broker) covering redelivery, DLQ routing and schema compatibility.