feat(datafusion): add SHOW, ADD and DROP PARTITION for catalog-managed format tables - #816

Purpose

Child of #591, which stays open as a draft that shows the full feature; its remaining changes reach main through child PRs.

A Format Table loaded from a REST catalog with metastore.partitioned-table=true has its partitions registered in the catalog, and since #814 a scan reads exactly those registrations. DataFusion could not list or change them. This PR adds SHOW PARTITIONS and ALTER TABLE ... ADD / DROP PARTITION for such tables.

Depends on: nothing. It is based on main, where #811, #813, #812 and #814 have merged. MSCK REPAIR TABLE (<MSCK_PR>) and ANALYZE TABLE (#815) build on this PR.

Brief change log

• SHOW PARTITIONS table [PARTITION (...)] lists the registered partitions in escaped key=value/... form, sorted. Values are read with the column type: a DATE registered as epoch days prints as dt=2026年07月22日, month=01 and month=1 both print as month=1, and the default partition prints as null. The optional PARTITION (...) clause filters by any subset of partition values, compared the same way.
• ALTER TABLE table ADD [IF NOT EXISTS] PARTITION (...) [PARTITION (...)] registers complete specifications with the catalog, then creates their directories. LOCATION is refused.
• ALTER TABLE table DROP [IF EXISTS] PARTITION (...) follows Java resolveFormatTablePartitionsForDrop:
  • several specifications per statement; a specification that fixes only some keys, leading or not, expands to every registered partition it matches, and may match none;
  • a complete specification that matches nothing fails the whole statement unless IF EXISTS is given;
  • values are compared as the catalog holds them, so month = 1 does not select a partition registered as month=01;
  • a statement made only of complete specifications looks them up through list-by-names; one with a partial specification reads the registry once;
  • registrations are removed first and directories deleted afterwards; a partition at a custom location (options.path) is only unregistered and its data is left in place;
  • it cannot be combined with other ALTER TABLE operations.
• Partition literals are read the way Java reads partition strings. A SQL literal becomes the string Spark would hand to Paimon (quoted strings and DATE '...' as written, booleans as true/false, numbers in canonical form, NULL as the default partition), and parse_format_partition_value reads it with the column type, as TypeUtils.castFromString does. So month = '01' and month = 01 both name month=1, a BOOLEAN accepts t/true/y/yes/1 and f/false/n/no/0, and a DATE accepts yyyy-MM-dd. An invalid value fails with its column named.
• ADD and DROP PARTITION refuse an empty or whitespace-only string for a string partition column, which would otherwise address the default partition, as Java PaimonFormatTable.requireNameablePartitionValues does.
• Catalog::drop_partitions, with a default that returns Unsupported. RESTCatalog sends it in batches of 1000 and refuses tables without catalog-managed partitions.
• FormatTablePartitionPaths, format_partition_value and parse_format_partition_value become public in paimon::table.

Tests

• crates/integrations/datafusion/tests/rest_format_partition_sql.rs:
  • test_partition_commands_update_rest_metadata_and_directories: ADD of several partitions, SHOW with and without a filter, DROP, and their directories;
  • test_partition_literals: DATE, INT, BOOLEAN and VARCHAR literals, including Java boolean spellings, unquoted, zero-padded and negative numbers and NULL, with the names and directories they produce, and invalid literals rejected with the column named;
  • test_drop_partition_specifications: partial and non-leading specifications, several specifications, a missing complete specification with and without IF EXISTS, an empty partial match, a failing specification leaving the statement unapplied, and the refusal to combine with other operations, each checking the remaining registrations and directories;
  • test_drop_partition_matches_values_as_the_catalog_holds_them: SHOW reads month=01 and month=1 with the column type, while DROP compares the values as registered;
  • test_drop_partition_looks_up_complete_specifications_by_name;
  • test_partition_ddl_refuses_blank_string_values: ADD and DROP with an empty or whitespace-only string are refused, and the NULL partition and its directory stay;
  • test_drop_partition_leaves_a_custom_location_in_place;
  • a compile-time check that the future of SQLContext::sql stays Send.
• crates/paimon/tests/rest_catalog_test.rs: test_rest_catalog_skips_empty_and_batches_drop_partitions, test_rest_catalog_refuses_to_drop_partitions_it_does_not_manage and test_rest_catalog_drop_partitions_maps_missing_table.
• sql_context unit test test_drop_if_exists_partition_does_not_ignore_missing_table.

Commands run on the top commit:

cargo fmt --all -- --check
cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
cargo test --locked -p paimon --all-targets --features fulltext,vortex
cargo test --locked --no-fail-fast -p paimon-datafusion --all-targets
cargo test --locked -p paimon-rest-server --all-targets

fmt and clippy pass. paimon: 2960 tests pass. paimon-rest-server: 10 tests pass. paimon-datafusion: 772 pass, including the 7 tests in rest_format_partition_sql.rs, and 39 fail. The failures are the 39 tests that also fail by name on main in this environment, because they read fixture tables provisioned by make docker-up (two also need the lumina native library).

Additions without a direct Java counterpart

• SQLContext turns SQL partition literals into partition strings itself (partition_literal_to_string): in Java, Spark resolves the partition spec before Paimon sees it, and DataFusion has no such step for these statements. The strings are then read by parse_format_partition_value, the counterpart of TypeUtils.castFromString.
• SQLContext parses SHOW PARTITIONS itself, since sqlparser 0.62 has no such statement.
• RESTCatalog::drop_partitions refuses tables without catalog-managed partitions, where Java commits truncatePartitions: turning string specs into typed values for every Paimon partition type is not available yet, and the endpoint alone would report success without touching data.
• The compile-time sql_future_is_send test keeps the future of SQLContext::sql Send, which Rust callers need for tokio::spawn and #[async_trait].

API and Format

• New Catalog::drop_partitions, with a default implementation that returns Unsupported.
• paimon::table::FormatTablePartitionPaths, format_partition_value and parse_format_partition_value become public.

The storage format is unchanged.

Documentation

docs/src/sql.md gains a Format Table Partitions section: the metastore.partitioned-table prerequisite, SHOW / ADD / DROP PARTITION, how partition values are read, the ordering guarantees of ADD and DROP, and how a partition at a custom location is treated. The SQL support scope lists the new statements.

Scope and limitations

• Applies only to partitioned internal REST Format Tables with metastore.partitioned-table=true and a non-engine implementation.
• Several DROP PARTITION specifications are written as repeated clauses, DROP PARTITION (...), DROP PARTITION (...); sqlparser 0.62 cannot parse the Hive form DROP PARTITION (...), (...).
• Reading a partition from its custom location is not implemented yet; such partitions fail closed.
• A TIME partition value must be given as an integer. Partition columns typed DECIMAL, TIMESTAMP, FLOAT, DOUBLE or BINARY remain unsupported on this path.