mirror of
https://github.com/apache/kafka.git
synced 2026-10-28 08:24:58 +08:00
Apoorv Mittal
d193373f24
A share fetch hands its record budget to partitions in the order they are read, so the partition read first takes the largest share. Two things decided that order earlier and together they could starve a partition, hence changed the behaviour. Earlier: - `SharePartitionManager` rotated the fetch list by `sessionEpoch % size`. The client bumps the epoch on both ShareFetch and ShareAcknowledge, so in a poll-and-commit loop fetch epochs step by two and the rotation is constant. One unpaired request flips the parity, and with two partitions the rotation then undoes the session reorder on every fetch, so the same partition is read first until the other starves. - `ShareSessionContext` moved served partitions to the end of the session from inside the response iterator, in the iteration order of the response map. That map had passed through `HashMap` copies, so when every partition returned records the session ended up in hash order and the same partition led every fetch. PR: This change removes the epoch rotation from the fetch path and makes the session reorder deterministic from the fetch order. The fetch order is preserved through `ShareFetchUtils`, `ShareFetch` and `KafkaApis` with `LinkedHashMap`s. `ShareSessionContext` reorders the session in one pass before pruning the response: partitions without records keep their position at the front, and partitions with records are appended in fetch order rotated by one, so the partition read first is read last next time. `PartitionRotateStrategy` is reduced to that rotation and the unused `sessionEpoch` parameter of `fetchMessages` is dropped. `ShareConsumerFairnessTest` adds cases for the epoch parity change, for the first position rotating when all partitions return records with explicit and implicit acknowledgement, and for an idle partition being read first once it has records. `SharePartitionManagerTest` verifies the session order across subsequent fetches. Reviewers: Abhinav Dixit <adixit@confluent.io>, Andrew Schofield <aschofield@confluent.io>