Skip to content

Navigation Menu

Sign in
Sign up

fix: support all non-negative int64 deletion vector positions - #917

Draft
wirybeaver wants to merge 2 commits into
apache:main from
wirybeaver:fix/dv-full-positive-int64
Draft

fix: support all non-negative int64 deletion vector positions #917
wirybeaver wants to merge 2 commits into
apache:main from
wirybeaver:fix/dv-full-positive-int64

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026
edited
Loading

Copy link
Copy Markdown

Summary

  • replace dense high-32-bit bitmap storage with a sparse ordered map
  • support every non-negative int64_t position through INT64_MAX without allocating intervening buckets
  • serialize only populated buckets in ascending key order and normalize empty decoded buckets
  • document that the half-open range API cannot include INT64_MAX, which must be added separately
  • avoid overflow when a coalesced position-delete range ends at INT64_MAX
  • reject deletion-vector payloads that exceed the signed 32-bit framing limit before allocating the complete serialized payload
  • preflight every merged position index before creating the Puffin output, so an oversized first or later vector cannot leave an empty or partial file

Tracking issue: #916

Depends on #913. This branch includes the cross-language fixture commit from that PR and should remain draft until the dependency is resolved.

Test plan

  • build/src/iceberg/test/util_test --gtest_filter='RoaringPositionBitmapTest.*:RoaringPositionBitmapInteropTest.*:PositionDeleteIndexTest.*:PositionDeleteRangeConsumerTest.*:DVWriterPreflightTest.*' (65 passed)
  • build/src/iceberg/test/puffin_test (31 passed, including 5 cross-language DV fixtures)
  • pre-commit run --all-files (passed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /