-
Notifications
You must be signed in to change notification settings - Fork 14.9k
KAFKA-20019: Fetch responses can exceed maxBytes when mixing local and tiered storage partitions #21211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
...d tiered storage partitions When a fetch request spans multiple partitions with a mix of local and remote (tiered storage) data, the broker can return more than `fetch.max.bytes` in a single response. For example, with `fetch.max.bytes=1MB`: - If partition 1 returns 1 MB from local storage - And partition 2 returns 1 MB from tiered storage - The total response can be 2 MB, exceeding the configured limit This can cause consumer-side issues if consumers aren't configured to accept payloads this large.
apoorvmittal10
commented
Dec 24, 2025
Though I haven't read the implementation yet but fetch.max.bytes is a soft limit which can anywways be exceeded when a single log batch is of higher size than fetch.max.bytes. However, it would be good to fix in other cases where batches are smaller. I think the latter is intended in the current PR, correct?
Thanks for your comment! Yes, correct - this fixes the aggregate size accounting issue when combining multiple smaller fetches across local and tiered storage partitions, not the single-large-batch case.
Though I haven't read the implementation yet but
fetch.max.bytesis a soft limit which can anywways be exceeded when a single log batch is of higher size than fetch.max.bytes. However, it would be good to fix in other cases where batches are smaller. I think the latter is intended in the current PR, correct?
nandini12396
commented
Dec 26, 2025
kamalcph
commented
Dec 26, 2025
Thanks for the patch! I couldn't understand the issue and patch. Could you write a unit / integration test to show the issue?
When a fetch request spans multiple partitions with a mix of local and remote (tiered storage) data, the broker can return more than
fetch.max.bytesin a single response. For example, withfetch.max.bytes=1MB:This can cause consumer-side issues if consumers aren't configured to accept payloads this large.