The key to a message at Kafka isn’t just an identifier to group data. In a partitioned production, you can determine what records share an ordered log, how jobs are distributed and what boundaries appear as the number of consumers grows. To design that key without understanding the order guarantee is a way to introduce matching bugs that later appear to be broken bugs.
That article focused on Apache Kafka 4.4.0 and the Java client’s partitioner as expressly indicated. I did not conduct load tests here, and the final lab is a reproducible proposal. The original example associated a key to a fixed consumer and assumed that a device would have its own partition. Both Simplifications are incorrect: many devices share partitions, and a rebalance can change the sewer.
The partition is the order domain, but not the business entity
A topic stores partitions with growing offsets. Kafka guarantees the reading order within a partition, not a total order among all partitions of a topic. Two events that fall into different partitions have no comparable global order just because their timprints are similar.
For the same key, a stable partitioning strategy usually leads to the same partition as long as they do not change the number of partitions, the partitioner and its configuration. That creates a basis for processing an entity’s events in order as long as producers also preserve their sending order. It does not automatically solve competing producers, late historic events and causation between services.
A consumer group can assign each partition to a single active group member at a given time. If we have 12 partitions and 24 consumers in the same group, there cannot be 24 active consumers reading those same partitions in parallel. If there are 100,000 entities, this does not mean that there are 100,000 partitions: the entities are distributed among the existing partitions.
Hashing: The client’s details matter
For key messages, the integrated Java production partitioner, in his classic key path, uses Murmur2 and assignment based on number of partitions. A conceptual representation is:
partition = positive(murmur2(serialized_key)) % partition_count
That expression sums up the corresponding behavior: it does not implement the algorithm, does not amount to a cryptographic hash and should not be copied as an executable code or assumed for all SDK. The serialisation of the key forms part of the entrance to the hash; userId=42 as string UTF-8 does not necessarily produce the same result as an integer serialised in binary. Some non-Java libraries and custom partitists have different conventions. Your implementations and test cases have to be checked before mixing producers.
The choice for logs with null key depends on the production and can involve batch-oriented partitioning mechanisms: it’s wrong to say that she always uses round-robin. In any event, there’s no stable affinity by entity based on a key that doesn’t exist.
Documentation: Kafka producer configs 4.4, Kafka operations.
Define the invariants before choosing orderId
Consider a platform that records OrderCreated, PaymentAuthorized, OrderPacked and OrderShipped events. If logic requires to rebuild an order applying its events in order, orderId is a natural candidate for the key.
{
"event_id": "evt-0007",
"aggregate_type": "order",
"aggregate_id": "order-1092",
"aggregate_version": 4,
"event_type": "OrderPacked",
"occurred_at": "2026-11-17T10:00:00Z"
}
The JSON is illustrative synthetic data. If the producer publishes using aggregate_id as key, all events of that aggregate can share partition and preserve the order of appand, given the above conditions. aggregate_version serves as a logical defense: the consumer can detect a version leap or a dual event even if transport has been altered by a retry or a reply.
The key is decided by the required serialization domain:
| Candidate | What an order. | Risk or limit |
|---|---|---|
orderId | Events of an order | An exceptionally active order creates a hot key |
accountId | Retirement of an account | A popular account can concentrate traffic |
userId | User events | Unequal users generate ske w |
deviceId | Telemetry of a device | A fleet with highly active devices isn’t balanced |
country | Events by country | Low cardinality and unequal demographic distribution |
| No key | No order by entity | Requires design without code affinity |
A high cardinality key does not guarantee a uniform distribution of bytes, CPU or processing costs. A single userId with million events per minute can saturate a partition even if there are millions of inactive users.
Skew: The problem is the rate and the cost, not just count keys
It secures three phenomena:
- Skew of keys: non-uniform distribution of registries between entities.
- Skew partitions: distribution of bytes or unequal events after hash and number of partitions.
- Skew processing: A partition consumes more CPU / I / O per event even if it gets less messages.
To diagnose, register by partition * bytes in / out *, message rate, batch size, production latency and lag by partition and group. A high lag can be caused by a slow sink or an unbalanced consumer, and not necessarily by an incorrect key.
Ejemplo hipotético, no medición:
partición 0 → 15 % de eventos, 40 % del coste de procesamiento
partición 1 → 10 % de eventos, 8 % del coste
partición 2 → 75 % de eventos, 52 % del coste
A perfect distribution of records does not amount to a perfect work balance. The end to end latency percentile is another axis: a hot key can block later events from the same partition even if the total tail looks manageable.
Why don’t we just add partitions?
Suppose an initial topic has N = 12 partitions. If increased to 24, operation hash(key) % N can assign the same key to a different partition. Kafka does not automatically move the history from old to new partition. During a transition period, events from the same entity can be put into two different partitions: two consumers could process both sequences without a global barrier.
The problem isn’t just that a key changes place. The problem is that the invariance “all events of an order are in a single sequence” ceases to be true throughout the life of the topic if interpreted without conditions.
For a system strictly dependent on order by entity, a change of partitions requires a ** migration plan *. Conceptual options include production of a new topic with explicit partitioning, building a controlled traffic cut, expecting a verifiable drainage or verting the assignment to manage transitions. The right choice depends on the possibility of pausing producers, the volume, the replay and the SLA available.
No “double partitions in production without impact” recipe valid for all applications. Source: Kafka operations.
Jumping keys: Paralelism for guarantees
A * hot key * can motivate to distribute events into multiple synthetic keys, for example accountId#bucket. The potential benefit is to distribute load and costs are to break the single order on account.
# Representación de diseño, no receta recomendada por defecto
account-4#0
account-4#1
account-4#2
The Hash now distribute subflows that were once one. If an operation requires applying movements of an account in sequence, the * jumping * isn’t secure without an additional protocol of sequencing, conciliation or partition by really independent units. It can be used when operations are commutative or aggregated, or when the composite keys represent subdomains with an independent order.
Before we distribute a hot key, we have to evaluate other options: better handler costs, get heavy jobs out of the sequential path, group operations, increase partition capacity or redesign the business aggregates. ** More paralleling isn’t free if you sacrifice correction. ♪ ♪
Writing order: production, restiries and idempotency
The tuning by key isn’t enough if the production introduces reordering. In modern versions of the Java production, the idempotency supports max.in.flight.requests.per.connection up to 5 with acks=all and activated reattempts. That range preserves the order of an idempowerful production. Instead, if idempotency is disabled and multiple requests are allowed in flight with retries, a missing and retrying shipment can be put after another later shipment that came first.
# Ejemplo deliberadamente explícito para estudiar ordering
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
That’s not a global mechanism among multiple producers. If two processes simultaneously update the same aggregate, the broker orders its arrival but doesn’t know which business operation should first occur. In these cases, we require preconditions and versions of added, optimistic concurrence, a sequencing authority or an equivalent model.
Apache Kafka: producer configuration.
Consumers: partition, offsets and rebalances
The consumer receives and confirms offsets according to his processing strategy. An offset commission does not confirm that an outside service has correctly applied the change. If the process fails after writing at the target base but before confirming the offset, Kafka can deliver the same event again. If you confirm before persisting, a crash can cause the application to lose its effect.
That’s why a robust handler usually combines a unique event identifier, an idempowerful operation at destination, a local transaction as appropriate and a clear Reby / DLQ policy. Unordered events, duplicates and * poison messages * require different treatment.
Also, it’s also been refused that “same key = same consumer forever.” Rebalances change assignments and the relevant guarantee is that a partition belongs, under normal conditions, to an active member by consumer group and that proper processing preserves the necessary order.
Reducible Laboratory: propose, not performed
The experiment should generate synthetic data with fixed seed and publish a known set of events. No muchput, CPU and p99 measured in this article.
- Fixed Kafka 4.4, client version, serializers, acks, idempotency, partitions, hardware and replication factor.
- Compare uniform keys with Zipfian distribution and a * heavy hitter *; count assignments before throwing consumers.
- Measuring logs / second, bytes by partition, max lag, p95 / p99 by event and memory of group.
- Repeat with 12, 48 and 192 partitions, without confusing consumer scalability with that of a producer.
- Simulate a laboratory rebalance and check order maintenance by addition and duplication.
- To discuss a change in the number of partitions with drained consumers: to register rematted keys and their historic offsets.
- To document what results invalidate a change of the hashing algorithm or serializer.
The report should include complete distribution of partitions, not just an average. A result in which the system quickly publishes but a single partition accumulates all the lag reveals a real limitation of design.
Conclusion: A key represents an order contract
The partitions are not an abstract technique for “climbing Kafka.” They are a decision about what needs to be put together, what can be run in parallel and what happens when topology changes.
The best practice is to first propose an aggregate’s invariance, to test distribution with representative data and to document migration before increasing partitions. To compare the broker that runs that design, see Redpanda vs Kafka: arquitectura, compatibilidad y coste operativo.
Main sources
- Apache Kafka 4.4: documentación y configuración
- Kafka: operaciones y particiones
- ProducerConfig 4.4 (API Java)
- Kafka: semánticas de entrega