Skip to content

Navigation Menu

Sign in
Sign up

[python] Support window joins in temporal alignment - #9760

Draft
XiaoHongbo-Hope wants to merge 12 commits into
apache:master from
XiaoHongbo-Hope:codex/temporal-window-aggregate
Draft

[python] Support window joins in temporal alignment #9760
XiaoHongbo-Hope wants to merge 12 commits into
apache:master from
XiaoHongbo-Hope:codex/temporal-window-aggregate

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 12, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Join higher-frequency samples, such as IMU, to temporal anchor rows and aggregate each matched window.

Changes

  • add top-level and chainable join_window
  • support group-local preceding / following ranges with configurable endpoint closure
  • support named outputs and multiple aggregations per source column
  • support mean, min, max, first, last, and count; the latter three also accept non-numeric values
  • prune right payload reads to aggregation columns
  • reuse snapshot pinning, authorization masking, batched row-ID fetches, and the bounded row-group cache

API choice

rolling().agg() aggregates windows within one input. This operation uses each left timestamp to select rows from a right input, then aggregates the matches, so it follows established window-join terminology. The verb-first join_window also matches the public join_asof API.

Named (source, operation) pairs allow one source column to produce multiple outputs; same-name operation strings remain shorthand.

Tests

147 temporal and multimodal table tests passed on Python 3.11 / PyArrow 19, including regressions for preserving masked field types after pruning colliding projection aliases. The regressions cover mean after STRING-to-DOUBLE masking, first/last after INT-to-STRING masking, unmatched windows, and rejection of mean/min/max on masked strings. Both integer-boundary regression tests also passed on Python 3.6 / NumPy 1.19.5 / PyArrow 6.0.1. An additional 2,400 boundary cases passed on each of NumPy 1.19.5 and 2.2.6. Python 3.6 syntax and flake8 checks passed.

The masked-alias regressions cannot complete on PyArrow 6 because the existing masking reader requires RecordBatch.set_column, which is unavailable in that version.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title (削除) [python] Add window aggregation to temporal alignment (削除ここまで) (追記) [python] Add temporal window join (追記ここまで) Sep 12, 2026
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title (削除) [python] Add temporal window join (削除ここまで) (追記) [python] Support window joins in temporal alignment (追記ここまで) Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /