-
Notifications
You must be signed in to change notification settings - Fork 91
DataFusion: paimon_incremental_query table function #506
Open
Description
Describe the feature
Expose batch incremental reads in DataFusion SQL via a table function, roughly:
SELECT * FROM paimon_incremental_query( 'table_name', start_snapshot_exclusive, end_snapshot_inclusive [, 'auto' | 'delta' | 'changelog' | 'diff'] );
Also support the $audit_log style suffix so row kinds show up as a first column when needed.
Motivation
The Rust core incremental API is useful, but most people will want to poke at this from SQL. DataFusion should only parse args and adapt the execution plan — snapshot planning stays in paimon core.
Depends on
Core incremental Delta / Changelog / Diff support landing first (or the PR can sit on top of that stack).
Out of scope
- Reimplementing snapshot planning inside the DataFusion crate
- Lookup/compact changelog generation
Activity
Metadata
Metadata
Assignees
Labels
No labels