-
Notifications
You must be signed in to change notification settings - Fork 91
feat(datafusion): support REST format table partitions (umbrella for child PRs) - #591
feat(datafusion): support REST format table partitions (umbrella for child PRs) #591sundapeng wants to merge 7 commits into
Conversation
JingsongLi
commented
Jul 24, 2026
-
High: DROP PARTITION is inconsistent with Java semantics
- Rust only supports single, complete partitions:
sql_context.rs:1064rejects multiple specs, andsql_context.rs:1734usesrequire_complete=true. - Java supports dropping multiple partitions at once and also supports arbitrary partial specifications such as
dt=...andhh=..., which are expanded to include all matching leaf partitions. - Impact: Commands executable in Java, such as
DROP PARTITION (dt=‘20260715’)and batch DROP operations, fail outright in Rust. Recommendation: Follow Java’s approach—pre-check complete specs usinglist-by-names; expand partial specs after a single catalog traversal.
- Rust only supports single, complete partitions:
-
High: Catalog-managed scans lack partition pruning via the REST endpoint
- Rust’s
format_table_scan.rs:216unconditionally useslist_partitionsto retrieve all partitions in the table before executing the predicate locally. - Java
FormatTableScan.java:230extracts the leading equality prefix and the full predicate;CatalogFormatTablePartitionManager.java:68pushes the pattern/predicate and page size (1000) down to REST. - Even when querying a single partition in a table with many partitions, all partition metadata is downloaded and parsed, which may cause significant latency and memory issues.
- Rust’s
-
Note: Boolean partition values are incompatible with Java
- Repair preserves and registers the original values in the catalog, but Rust
format_partition.rs:216usesstr::parse::<bool>(), which only accepts lowercasetrue/false. - Java ignores case and accepts
t/y/yes/1andf/n/no/0. - Therefore, if a catalog entry
active=TRUEis registered via Rust MSCK, subsequentscanorSHOWoperations will report "invalid catalog partition metadata"; Java can read it normally. It is recommended to adopt the Java-compatible rules and perform additional cross-platform testing.
- Repair preserves and registers the original values in the catalog, but Rust
78f19a5 to
2d40c53
Compare
sundapeng
commented
Jul 31, 2026
@JingsongLi Thanks for the review. All three are fixed, and the branch is rebased onto current main (it was conflicting with #600 and #627). Details below, including one place where I could not go all the way and would like your call.
1. DROP PARTITION semantics
DROP PARTITION now takes several specifications per statement, and a specification that fixes only some of the partition keys expands to every registered partition it matches. The keys need not be a leading prefix, so DROP PARTITION (hh = '10') on a (dt, hh) table works, matching PaimonFormatTable.dropFormatTablePartitions where partial specs are matched by arbitrary key subset. One catalog listing serves the whole statement, however many specifications it carries.
Error semantics follow Java as well: a complete specification names one partition, so a missing one raises unless IF EXISTS; a partial one describes a set that is allowed to come out empty, so it is a no-op. A specification that raises leaves the whole statement unapplied.
One thing I could not do. Hive and Spark write several specs as DROP PARTITION (a), (b), and sqlparser 0.62 cannot parse that: parse_alter_table comma-separates ALTER operations, and a bare second PARTITION is read as RENAME PARTITION. So multiple specs are written as repeated clauses:
ALTER TABLE t DROP PARTITION (dt = '20260722'), DROP PARTITION (dt = '20260723');
The alternatives are a change in sqlparser, or a hand-rolled pre-parse of the statement like the one SHOW PARTITIONS already uses here. I did not add the pre-parse because it duplicates the ALTER TABLE grammar for one syntax variant, but I will add it if you want the Hive spelling accepted.
I also did not add list-by-names. Java resolves complete specs through it and asserts in tests that the complete-spec path never does a full traversal, but paimon-rust has no listPartitionsByNames anywhere yet (no Catalog method, no RESTApi method, no resource path, no request type). Adding the whole stack here would grow a PR that is already large. The current code needs the registered spec anyway to resolve the directory, so it costs one listing per statement rather than one per specification. Happy to do it as a follow-up, or here if you prefer.
2. Partition pruning through the REST endpoint
A catalog-managed scan now extracts the leading equality prefix from its filter, builds the partition-name prefix pattern with the same contract as PartitionPathUtils.buildPartitionNamePrefixPattern (escaped key=value joined by /, % the only wildcard, complete prefix means the exact name, shorter prefix gets /%, no pattern when a value is blank or escaping produced a literal %), and sends it as partitionNamePattern with maxResults=1000. The local per-partition match stays, so a catalog that ignores the pattern still produces the same result set.
One Rust-specific detail worth flagging. A straight port of the Java extractor would have pushed nothing in the case that matters most. PartitionFilter::from_predicate collapses a filter that pins every partition key into a PartitionSet and drops the predicate, so WHERE dt = 'a' AND hh = '10' never reaches the scan as a Predicate. The extractor therefore also handles PartitionSet, taking the longest common leading prefix of its rows. A single-partition set gives the exact name, and dt = 'a' AND hh IN ('10', '11') gives dt=a/%.
Predicate pushdown to listPartitionsByFilter is not included: #500 added REST predicate JSON parsing but not serialization, so there is nothing to encode with yet.
3. Boolean partition values
parse_format_partition_value now accepts t/true/y/yes/1 and f/false/n/no/0 case-insensitively, mirroring BinaryStringUtils.toBoolean. There was a unit test asserting that yes is rejected, which is exactly the bug, so it is replaced with a table covering every spelling in both directions plus the rejections.
While confirming this I found the same class of problem is wider than booleans: parse_format_partition_value covers BOOLEAN, the integer types, CHAR/VARCHAR, DATE and TIME, while TypeUtils.castFromString also covers DECIMAL, FLOAT, DOUBLE, TIMESTAMP, TIMESTAMP_LTZ and BINARY. A catalog-managed table partitioned by any of those cannot be scanned. That is not a regression from this PR, and closing it properly means aligning both directions of the conversion, which are currently split across two modules that have already drifted. I would rather do it in its own PR than widen this one. Tell me if you want it here instead.
Also in this push
docs/src/sql.md documents all four statements, and the PR description now follows the template.
@JingsongLi
JingsongLi
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Compared this change with the Apache Paimon Java implementation. I found two correctness issues and four Java-parity/performance gaps; details are inline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] Preserve raw partition identity during partial DROP
Partial specifications are matched after converting catalog values to typed Datum values. A table partitioned by (year STRING, month INT) can legitimately contain raw registrations such as {year=2025, month=01} and {year=2026, month=1} after MSCK, because repair preserves directory spelling. DROP PARTITION (month=1) has no verbatim match because the request is partial, so both values normalize to Int(1) and both directories are unregistered and deleted. Java compares partial specs using the raw catalog values, deleting only month=1. Please match partial specs against raw strings and add a mixed-spelling regression test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] Avoid lossy name-pattern pushdown for typed partitions
This formats typed equality literals canonically before building the raw partitionNamePattern. MSCK preserves valid raw spellings such as active=TRUE or month=01, but predicates active = true / month = 1 produce exact patterns active=true / month=1. A catalog that honors the pattern removes those registrations before the local typed check runs, so the query silently loses rows. Please restrict name-pattern pushdown to types with a unique raw spelling (at least CHAR/VARCHAR), or use a typed-filter/list-all fallback, and add raw TRUE/01 scan tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Preserve ordinary-table DROP semantics
RESTCatalog::drop_partitions always calls the metadata-only REST endpoint. Java only does that for catalog-managed Format Tables; for ordinary Paimon tables it loads the table and commits truncatePartitions. A public Rust Catalog caller can therefore get an error or success without the expected partition-data change. Please branch on has_catalog_managed_partitions(): keep REST unregistering for managed Format Tables and use a table commit for ordinary tables, or explicitly reject non-managed calls until implemented.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Push the complete partition predicate to REST
Only a leading-equality name pattern is sent. Filters without such a prefix (for example, hour = 10 for keys (dt, hour), ranges, or starts-with) download the whole registry in 1,000-row pages before local filtering. Java uses paged /partitions/list-by-filter with the serialized predicate and still rechecks locally. Please add the compatible request/path/API with a safe fallback.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Bound partition-directory listing concurrency
This awaits one recursive object-store listing per registered partition inside a plain loop. An unfiltered 10k-partition table therefore pays roughly 10k list latencies serially. Java uses configurable bounded concurrency through format-table.scan.list-parallelism (default 64). Please use bounded async concurrency while preserving deterministic final sorting and fail-fast behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Use list-by-names for complete DROP specs
Even a complete exact DROP PARTITION calls list_partitions and materializes every registration. Java resolves complete specs through batched /partitions/list-by-names and reserves a full traversal for partial specs. Please implement the corresponding REST/Catalog method and use it for complete requests; this avoids O(all partitions) network and memory for the common exact-drop case.
2d40c53 to
b65cb57
Compare
@JingsongLi Thanks for the second pass. Rebased onto current main; all six comments are addressed:
- [P1] Raw identity in DROP: specs are compared with the values the catalog holds, as Java
resolveFormatTablePartitionsForDropdoes, soDROP PARTITION (month = 1)no longer takesmonth=01. - [P1] Lossy pattern pushdown: the name pattern is built only from leading CHAR/VARCHAR equalities. Java still pushes typed values here; happy to port the restriction if you agree.
- [P2] Ordinary-table DROP:
RESTCatalog::drop_partitionsnow refuses tables without catalog-managed partitions. - [P2] Full predicate pushdown: added
Predicate::to_rest_jsonandlist-by-filter. On 501 the scan retries the paged pattern listing (still the catalog, never the filesystem), where Java throws. - [P2] Listing concurrency: bounded by
format-table.scan.list-parallelism(default 64, clamped to [1, 1000]). - [P2] list-by-names: used when every DROP spec is complete, batched at 1000. Java master now reads the registry once instead; the result is the same.
Also in this push:
fix(datafusion): keep format table partition columns out of decoder filters: since perf(datafusion): push filters into Paimon Parquet readers #574 , a filter such asWHERE activeon a partition column returned no rows. Can split it out if you prefer.fix(datafusion): leave partitions at a custom location where they are: such partitions fail closed until location support lands.feat(datafusion): support ANALYZE TABLE on catalog-managed format tables, following the Java command, with docs.
Details and the end-to-end validation are in the PR description.
sundapeng
commented
Sep 11, 2026
@JingsongLi This PR has grown too large to review comfortably, so I am splitting it. The first three PRs each apply to main on their own:
- a fix that keeps format table partition columns out of Parquet decoder filters;
- a format table listing fix: skip committer staging directories and list partitions concurrently;
- the REST partition catalog APIs: list-by-names, list-by-filter, statistics on create, and
Predicate::to_rest_json.
The catalog-managed scan, the partition commands and ANALYZE follow once those land, and this PR will be narrowed to the partition commands. I will link the new PRs here.
b65cb57 to
c745fb7
Compare
Split out of apache#591. Adds the REST catalog partition APIs that the catalog-managed Format Table work builds on, without changing anything reachable from SQL: - RESTApi can create partitions with optional statistics, drop them, look them up by name, and list them by predicate or by name pattern. Partition paging follows a token past an empty page, stops on an empty token and fails on a repeated one. - Predicate::to_rest_json writes the REST catalog predicate JSON, the inverse of from_rest_json, matching the Java wire strings. - Catalog gains create_partitions, create_partitions_with_statistics and list_partitions_by_names. RESTCatalog batches idempotent creates at 1000 with per-batch statistics, validates the statistics, and looks partitions up by name in batches, using the plain listing on 501. - Partition decodes absent statistics as unknown, and PartitionStatistics::last_file_creation_time becomes i64, as in Java.
...at table Split out of apache#591. A Format Table loaded from a REST catalog with metastore.partitioned-table=true has its partitions managed by the catalog: the registrations, not the directory tree, decide what a scan reads. - RESTEnv keeps the partition settings the catalog returned when the table was loaded (table path, file format, path layout), and loading an external or engine-implemented table with managed partitions fails. Dynamic options cannot switch the partition source, the path layout or the implementation of such a table. - The format table scan lists the registered partitions instead of directories. It sends the catalog a partition-name pattern built only from leading CHAR/VARCHAR equalities, whose values have one spelling, and the partition predicate through list-by-filter, keeping the conjuncts of an AND that have a wire form. A catalog answering 501 for the filter is asked by pattern; the directory tree is never used. Registered values accept the Java boolean spellings, and a partition registered at a custom location fails the scan instead of reading the table directory in its place. - RESTCatalog no longer falls back to a file system listing for a table with catalog-managed partitions when the partition endpoints answer 501. - Partition values and path names go through shared helpers, which escape path names the way Java and PartitionComputer do: a non-ASCII value in a leading-equality or value-only path is no longer percent-encoded.
c745fb7 to
32ef2ff
Compare
...at table Split out of apache#591. A Format Table loaded from a REST catalog with metastore.partitioned-table=true has its partitions managed by the catalog: the registrations, not the directory tree, decide what a scan reads. - RESTEnv keeps the partition settings the catalog returned when the table was loaded (table path, file format, path layout), and loading an external or engine-implemented table with managed partitions fails. Dynamic options cannot switch the partition source, the path layout or the implementation of such a table. - The format table scan lists the registered partitions instead of directories. It sends the catalog a partition-name pattern built only from leading CHAR/VARCHAR equalities, whose values have one spelling, and the partition predicate through list-by-filter, keeping the conjuncts of an AND that have a wire form. A catalog answering 501 for the filter is asked by pattern; the directory tree is never used. Registered values accept the Java boolean spellings, and a partition registered at a custom location fails the scan instead of reading the table directory in its place. - RESTCatalog no longer falls back to a file system listing for a table with catalog-managed partitions when the partition endpoints answer 501. - Partition values and path names go through shared helpers, which escape path names the way Java and PartitionComputer do: a non-ASCII value in a leading-equality or value-only path is no longer percent-encoded.
32ef2ff to
e998847
Compare
Split out of apache#591. A Format Table whose partitions the REST catalog manages had no way to report what those partitions hold. ANALYZE TABLE t [PARTITION (...)] COMPUTE STATISTICS [NOSCAN] measures the registered partitions from storage and reports the result through create_partitions_with_statistics with replaceStatistics, as Java PaimonAnalyzeFormatTablePartitionsCommand does: - NOSCAN stops at the listing: file count, byte size and the latest file modification time. A full ANALYZE also reads Parquet and ORC footers for row counts; a footer that cannot be read leaves the partition's row count unknown rather than short, and an empty partition holds exactly zero rows. - PARTITION (...) selects a leading run of partition values. A prefix with no registered partition, a partition at a custom location, FOR COLUMNS and CACHE METADATA are refused, as is a table whose partitions the catalog does not manage. - format-table.statistics.parallelism (default 8) bounds the listings and footer reads in flight. The collector lists each partition through the same helper the scan now uses, so a measurement counts exactly the files a scan of that partition reads and leaves committer staging trees out. Its streams own their items so that the future of every SQLContext statement stays Send.
e998847 to
ee58bd7
Compare
Split out of apache#591. A Format Table whose partitions the REST catalog manages had no way to report what those partitions hold. ANALYZE TABLE t [PARTITION (...)] COMPUTE STATISTICS [NOSCAN] measures the registered partitions from storage and reports the result through create_partitions_with_statistics with replaceStatistics, as Java PaimonAnalyzeFormatTablePartitionsCommand does: - NOSCAN stops at the listing: file count, byte size and the latest file modification time. A full ANALYZE also reads Parquet and ORC footers for row counts; a footer that cannot be read leaves the partition's row count unknown rather than short, and an empty partition holds exactly zero rows. - PARTITION (...) selects a leading run of partition values. A prefix with no registered partition, a partition at a custom location, FOR COLUMNS and CACHE METADATA are refused, as is a table whose partitions the catalog does not manage. - format-table.statistics.parallelism (default 8) bounds the listings and footer reads in flight. The collector lists each partition through the same helper the scan now uses, so a measurement counts exactly the files a scan of that partition reads and leaves committer staging trees out. Its streams own their items so that the future of every SQLContext statement stays Send.
ee58bd7 to
5090601
Compare
Split out of apache#591. A Format Table whose partitions the REST catalog manages had no way to report what those partitions hold. ANALYZE TABLE t [PARTITION (...)] COMPUTE STATISTICS [NOSCAN] measures the registered partitions from storage and reports the result through create_partitions_with_statistics with replaceStatistics, as Java PaimonAnalyzeFormatTablePartitionsCommand does: - NOSCAN stops at the listing: file count, byte size and the latest file modification time. A full ANALYZE also reads Parquet and ORC footers for row counts; a footer that cannot be read leaves the partition's row count unknown rather than short, and an empty partition holds exactly zero rows. - PARTITION (...) selects a leading run of partition values. A prefix with no registered partition, a partition at a custom location, FOR COLUMNS and CACHE METADATA are refused, as is a table whose partitions the catalog does not manage. - format-table.statistics.parallelism (default 8) bounds the listings and footer reads in flight. The collector lists each partition through the same helper the scan now uses, so a measurement counts exactly the files a scan of that partition reads and leaves committer staging trees out. Its streams own their items so that the future of every SQLContext statement stays Send.
5090601 to
260b462
Compare
Split out of apache#591. A Format Table whose partitions the REST catalog manages had no way to report what those partitions hold. ANALYZE TABLE t [PARTITION (...)] COMPUTE STATISTICS [NOSCAN] measures the registered partitions from storage and reports the result through create_partitions_with_statistics with replaceStatistics, as Java PaimonAnalyzeFormatTablePartitionsCommand does: - NOSCAN stops at the listing: file count, byte size and the latest file modification time. A full ANALYZE also reads Parquet and ORC footers for row counts; a footer that cannot be read leaves the partition's row count unknown rather than short, and an empty partition holds exactly zero rows. - PARTITION (...) selects a leading run of partition values. A prefix with no registered partition, a partition at a custom location, FOR COLUMNS and CACHE METADATA are refused, as is a table whose partitions the catalog does not manage. - format-table.statistics.parallelism (default 8) bounds the listings and footer reads in flight. The collector lists each partition through the same helper the scan now uses, so a measurement counts exactly the files a scan of that partition reads and leaves committer staging trees out. Its streams own their items so that the future of every SQLContext statement stays Send.
sundapeng
commented
Sep 11, 2026
Moving this back to draft: it stays as the full view of the feature, and the remaining changes land through smaller child PRs (partition DDL, MSCK REPAIR, and #815 for ANALYZE). I will link them here.
...d format tables A Format Table whose partitions a REST catalog manages is read from its registrations, but DataFusion could not change them. This adds the statements that do: - SHOW PARTITIONS [PARTITION (...)] lists the registrations in escaped key=value form, read with the column types and optionally filtered by any subset of partition values. - ALTER TABLE ... ADD [IF NOT EXISTS] PARTITION (...) registers complete specs, then creates their directories. - ALTER TABLE ... DROP [IF EXISTS] PARTITION (...) takes several specifications, expands a partial one to every registered partition it matches and compares values as the catalog holds them, so `month = 1` does not select `month=01`, as Java resolveFormatTablePartitionsForDrop does. Complete specifications are looked up by name; a partial one reads the registry once. Registrations go first and directories after; a partition at a custom location is only unregistered. Partition literals are read with the column type through the parser that reads registrations and directories, the way Java casts partition strings, so a BOOLEAN value accepts t/true/y/yes/1 and their false counterparts. Catalog gains drop_partitions. RESTCatalog refuses it for a table without catalog-managed partitions, since the endpoint only removes metadata and would leave the data of any other table in place while reporting success.
...bles Describe catalog-managed Format Table partitions in the SQL guide: the metastore.partitioned-table prerequisite, SHOW / ADD / DROP PARTITION, how partition values are read, what each statement does to registrations and directories, and how a partition at a custom location is treated.
Split out of apache#591. A Format Table whose partitions the REST catalog manages had no way to report what those partitions hold. ANALYZE TABLE t [PARTITION (...)] COMPUTE STATISTICS [NOSCAN] measures the registered partitions from storage and reports the result through create_partitions_with_statistics with replaceStatistics, as Java PaimonAnalyzeFormatTablePartitionsCommand does: - NOSCAN stops at the listing: file count, byte size and the latest file modification time. A full ANALYZE also reads Parquet and ORC footers for row counts; a footer that cannot be read leaves the partition's row count unknown rather than short, and an empty partition holds exactly zero rows. - PARTITION (...) selects a leading run of partition values. A prefix with no registered partition, a partition at a custom location, FOR COLUMNS and CACHE METADATA are refused, as is a table whose partitions the catalog does not manage. - format-table.statistics.parallelism (default 8) bounds the listings and footer reads in flight. The collector lists each partition through the same helper the scan now uses, so a measurement counts exactly the files a scan of that partition reads and leaves committer staging trees out. Its streams own their items so that the future of every SQLContext statement stays Send.
260b462 to
c2a25f2
Compare
sundapeng
commented
Sep 11, 2026
... PARTITION A blank string for a string partition column is formatted as the default partition name, so DROP PARTITION (dt = '') unregistered the NULL partition and deleted its directory, and ADD PARTITION (dt = '') registered it. Refuse an empty or whitespace-only string for a string partition column in ADD and DROP PARTITION, as Java PaimonFormatTable.requireNameablePartitionValues does. SHOW PARTITIONS still accepts it as a filter.
...bles
A catalog-managed Format Table reads only the partitions its catalog
registers, so a partition directory written without a registration stays
invisible, and a registration whose directory is gone points at nothing.
MSCK REPAIR TABLE [{ADD|DROP|SYNC} PARTITIONS] reconciles the two:
- ADD, the default, registers every discovered partition the catalog does
not hold; DROP unregisters every registration without a directory; SYNC
does both. Nothing on storage is created or deleted.
- Directory discovery and the catalog listing both finish before any
change, and DROP and SYNC fail on a table directory that cannot be
listed rather than reading it as empty.
- Partitions are matched by escaped name and registered with the values
as their directories spell them, so month=01 stays month=01.
- A partition registered at a custom location is never unregistered,
since its data does not live under the table directory.
FormatTablePartitionPaths gains discover, which skips hidden directories
and segments outside the configured layout, and fails on a matching
segment that is not canonically escaped. unescape_path_name moves from
the scan to spec::partition_utils, beside escape_path_name, so discovery
and the scan share it.
Describe the ADD, DROP and SYNC modes, that repair only changes registrations, that discovery and the catalog listing finish before any change, and that a partition at a custom location is never unregistered.
Split out of apache#591. A Format Table whose partitions the REST catalog manages had no way to report what those partitions hold. ANALYZE TABLE t [PARTITION (...)] COMPUTE STATISTICS [NOSCAN] measures the registered partitions from storage and reports the result through create_partitions_with_statistics with replaceStatistics, as Java PaimonAnalyzeFormatTablePartitionsCommand does: - NOSCAN stops at the listing: file count, byte size and the latest file modification time. A full ANALYZE also reads Parquet and ORC footers for row counts; a footer that cannot be read leaves the partition's row count unknown rather than short, and an empty partition holds exactly zero rows. - PARTITION (...) selects a leading run of partition values. A prefix with no registered partition, a partition at a custom location, FOR COLUMNS and CACHE METADATA are refused, as is a table whose partitions the catalog does not manage. - format-table.statistics.parallelism (default 8) bounds the listings and footer reads in flight. The collector lists each partition through the same helper the scan now uses, so a measurement counts exactly the files a scan of that partition reads and leaves committer staging trees out. Its streams own their items so that the future of every SQLContext statement stays Send.
Describe what NOSCAN and a full ANALYZE measure, how PARTITION selects partitions, what is refused, and format-table.statistics.parallelism.
c2a25f2 to
4f83b41
Compare
Uh oh!
There was an error while loading. Please reload this page.
Purpose
A Format Table loaded from REST Catalog can have its partitions managed by the catalog instead of discovered from the directory layout. This draft PR shows the complete Rust support for such tables in one place. It is not meant to be merged: every change reaches
mainthrough the child PRs below, and this branch follows the top of the child stack so the whole feature can be read and tried together.Child PRs
7cb6f8a04965772bb2abb6824487SHOW PARTITIONS,ALTER TABLE ... ADD / DROP PARTITION,Catalog::drop_partitionsmainMSCK REPAIR TABLE, partition directory discoveryANALYZE TABLE ... COMPUTE STATISTICS [NOSCAN]Review happens in the child PRs; comments here are welcome but will be carried over to the child that owns the code.
Brief change log
Merged on
main:list-by-names,list-by-filter, and partition statistics reported with a create (partitionStatistics/replaceStatistics); a partition listed without statistics decodes them as unknown (-1) (feat(rest): add partition registration, lookup and statistics APIs #812 ).list-by-filterwith a partition-name pattern built from leading CHAR/VARCHAR equalities, falls back to the paged listing when the catalog answers 501, and fails on a partition registered at a custom location (feat(table): read the registered partitions of a catalog-managed format table #814 ).format-table.scan.list-parallelism, and entries starting with.or_below a partition directory, such as_temporaryand__magic_*, are skipped (fix(table): skip staging files and list format table partitions concurrently #813 ).In review:
SHOW PARTITIONS table [PARTITION (...)],ALTER TABLE table ADD [IF NOT EXISTS] PARTITION (...)andALTER TABLE table DROP [IF EXISTS] PARTITION (...). DROP follows JavaresolveFormatTablePartitionsForDrop: several specifications, partial specifications expanded to every registered match, values compared as the catalog holds them, registrations removed before directories, and a partition at a custom location only unregistered. Partition literals are read with the column type as JavaTypeUtils.castFromStringreads partition strings.Catalog::drop_partitions, whichRESTCatalogrefuses for tables without catalog-managed partitions.MSCK REPAIR TABLE table [{ADD|DROP|SYNC} PARTITIONS]reconciles registrations with the directories that exist. Discovery and the catalog listing both finish before any change, DROP and SYNC fail if the table directory cannot be listed, and a partition at a custom location is never unregistered.ANALYZE TABLE table [PARTITION (...)] COMPUTE STATISTICS [NOSCAN]measures registered partitions from storage and replaces the statistics the catalog holds.NOSCANreports file count, size and last file creation time; a full run also reads Parquet and ORC footers for row counts.format-table.statistics.parallelism(default 8) bounds the work.Tests
Each child PR lists its tests. Commands run on the top of the stack (#815's head, the same tree as this branch):
fmt and clippy pass.
paimon: 2967 tests pass.paimon-rest-server: 10 tests pass.paimon-datafusion: 780 pass, including the 15 tests inrest_format_partition_sql.rs, and 39 fail. The failures are the 39 tests that also fail by name onmainin this environment, because they read fixture tables provisioned bymake docker-up(two also need the lumina native library).Before the split, the branch was also validated end to end against a Bennett REST catalog server with a local file warehouse (10,000 registered partitions through
SHOW PARTITIONS, ADD, MSCK ADD/SYNC, filtered reads, ANALYZE NOSCAN and full, and DROP). That run has not been repeated on the restructured stack.Additions without a direct Java counterpart
Listed in each child PR: #816 (SQL literal conversion, the
SHOW PARTITIONSparser, theRESTCatalog::drop_partitionsrefusal, theSendcheck onSQLContext::sql), #817 (repair safeguards) and #815 (footer row counts, in-process measurement).API and Format
Changes still in review:
Catalog::drop_partitions, with a default implementation that returnsUnsupported(feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables #816 ).paimon::table::FormatTablePartitionPaths(withdiscoverfrom feat(datafusion): add MSCK REPAIR TABLE for catalog-managed format tables #817 ),format_partition_valueandparse_format_partition_valuebecome public (feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables #816 ).paimon::table::FormatTablePartitionStatsCollectorand optionformat-table.statistics.parallelism(feat(datafusion): support ANALYZE TABLE on catalog-managed format tables #815 ).The API changes that merged with #812 are described there. The storage format is unchanged.
Documentation
docs/src/sql.mdgains a Format Table Partitions section covering themetastore.partitioned-tableprerequisite,SHOW/ADD/DROP PARTITION,MSCK REPAIR TABLE,ANALYZE TABLE, how partition values are read, and how a partition at a custom location is treated. Each child PR adds its own part.Scope and limitations
metastore.partitioned-table=trueand a non-engineimplementation.DROP PARTITIONspecifications are written as repeated clauses,DROP PARTITION (...), DROP PARTITION (...); sqlparser 0.62 cannot parse the Hive formDROP PARTITION (...), (...).INSERT INTOfails before anything is written or registered.