Skip to content

Navigation Menu

Sign in
Sign up

feat: support DataFusion paimon_incremental_query - #511

Open
Pandas886 wants to merge 2 commits into
apache:main from
Pandas886:feat/datafusion-incremental-query
Open

feat: support DataFusion paimon_incremental_query #511
Pandas886 wants to merge 2 commits into
apache:main from
Pandas886:feat/datafusion-incremental-query

Conversation

@Pandas886

@Pandas886 Pandas886 commented Jul 11, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #506

This PR adds the DataFusion SQL surface for the batch incremental read APIs now available on main.

SELECT * FROM paimon_incremental_query('t', 0, 2);
SELECT * FROM paimon_incremental_query('t', 0, 2, 'diff');
SELECT rowkind, id, value FROM paimon_incremental_query('t$audit_log', 0, 2, 'changelog');

Brief change log

  • Register TVF paimon_incremental_query in SQLContext
  • 3-arg form defaults to auto; optional 4th arg: auto / delta / changelog / diff
  • Delegate physical planning to core IncrementalScan / AuditLogTable
  • Support $audit_log suffix for rowkind output
  • Support projection and residual filters
  • Resolve qualified table names against the named catalog; unqualified names follow the current catalog
  • Update docs/src/sql.md with usage and examples

Tests

cargo test -p paimon-datafusion --test incremental_query
cargo fmt --all -- --check
cargo clippy -p paimon-datafusion --all-targets -- -D warnings

API and Format

  • New SQL table function only; storage format unchanged
  • Relies on the core incremental APIs already merged into main

Documentation

docs/src/sql.md documents registration, arguments, modes, and examples.

Copy link
Copy Markdown
Contributor

The Diff problem of inheriting '# 510'; The non audit mode will first push the SQL projection down to the core, directly triggering the issue of "only comparing projection columns/missing PK". In addition, the catalog section of 'catalog. db. table' is discarded after parsing, and the TVF global registration will be overwritten by the last registered catalog. Multiple catalog scenarios may result in reading the wrong catalog.

Pandas886 force-pushed the feat/datafusion-incremental-query branch from 9e235b1 to c833e97 Compare July 30, 2026 11:30

Pandas886 commented Jul 30, 2026
edited
Loading

Copy link
Copy Markdown
Contributor Author

Rebased on the latest main and fixed catalog/database resolution. Added coverage for multiple catalogs and USE database. Could you take another look?

Pandas886 force-pushed the feat/datafusion-incremental-query branch from c833e97 to eb99a81 Compare July 30, 2026 11:40
Comment thread docs/src/sql.md
```rust
use paimon_datafusion::register_incremental_query;

register_incremental_query(&ctx, catalog.clone(), "default");

@JingsongLi JingsongLi Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

register by default? so user don't need to register this again.

Pandas886 reacted with thumbs up emoji
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JingsongLi JingsongLi JingsongLi left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

DataFusion: paimon_incremental_query table function

AltStyle によって変換されたページ (->オリジナル) /