Skip to content

Navigation Menu

Sign in
Sign up

feat: add position_deletes metadata table - #920

Draft
wirybeaver wants to merge 3 commits into
apache:main from
wirybeaver:feature/position-deletes-metadata-table
Draft

feat: add position_deletes metadata table #920
wirybeaver wants to merge 3 commits into
apache:main from
wirybeaver:feature/position-deletes-metadata-table

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026
edited
Loading

Copy link
Copy Markdown

Summary

  • add the general position_deletes metadata-table kind and materialized scan API
  • expand live Parquet position-delete files and v3 Puffin deletion vectors into logical rows
  • preserve Parquet deleted-row payloads, project physical subset row schemas into the current table row schema with omitted fields remaining null even when the current schema defines defaults, and reject null row values when the physical row column exists
  • resolve partition output fields against the current schema, omit dropped historical-only sources, promote compatible historical partition values such as int to long, and reject active spec conflicts
  • assign partition metadata fields fresh IDs that do not conflict with metadata fields or nested table-row fields
  • support partitioned and unpartitioned tables, upgraded mixed tables, multi-blob Puffin files, and projected/reordered top-level output schemas
  • expose v3 content_offset and content_size_in_bytes metadata

Refs #916.

Depends on #913 and #914. This branch includes those prerequisite commits.

Test plan

  • cmake --build build-clang --target position_deletes_table_test -j2
  • position_deletes_table_test — 16 passed, including required/non-null and subset Parquet rows, nested field-ID collisions, int to long partition promotion, dropped historical partition sources, incompatible active partition transforms, projected empty scans, and the documented nested-projection rejection
  • cmake --build build-clang --target metadata_table_test data_test puffin_test util_test -j2
  • metadata_table_test — 4 passed
  • data_test --gtest_filter='DVWriterTest.*:DeleteLoaderTest.*' — 30 passed
  • puffin_test --gtest_filter='DVUtilTest.*:PuffinDVInteropTest.*' — 8 passed
  • util_test --gtest_filter='PositionDeleteIndexTest.*' — 23 passed
  • uvx pre-commit run --all-files — passed

Known limitation

Nested projection such as row.id is not supported by the shared projection layer. ProjectionContext / ProjectBatch currently requires complete top-level fields and rejects pruning row to a differently typed struct. Top-level projections, including projected empty scans, are supported. Adding nested projection requires a shared projection-infrastructure change and is intentionally outside this narrow metadata-table PR.

wirybeaver force-pushed the feature/position-deletes-metadata-table branch 2 times, most recently from 3fe420f to 74edbb6 Compare September 5, 2026 23:03
wirybeaver force-pushed the feature/position-deletes-metadata-table branch from 74edbb6 to 7965dcd Compare September 5, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /