feat(datafusion): add MSCK REPAIR TABLE for catalog-managed format tables - #817

Purpose

Child of #591, which stays open as a draft that shows the full feature; its remaining changes reach main through child PRs.

A Format Table with catalog-managed partitions reads only the partitions its catalog registers. A partition directory written by a tool that does not register partitions stays invisible to a scan, and a registration whose directory was removed points at nothing. This PR adds MSCK REPAIR TABLE for such tables, following Java FormatTablePartitionRepair, which backs the same statement in Spark.

Depends on #816. Until it merges, this branch includes its three commits; please review only the last two: feat(datafusion): add MSCK REPAIR TABLE for catalog-managed format tables and docs(datafusion): document MSCK REPAIR TABLE for format tables. ANALYZE TABLE (#815) builds on this PR.

Brief change log

• MSCK REPAIR TABLE table [{ADD|DROP|SYNC} PARTITIONS] on a Format Table with catalog-managed partitions:
  • ADD, which plain MSCK REPAIR TABLE also means, registers every partition directory the catalog does not hold; DROP unregisters every registration whose directory is gone; SYNC does both. New registrations are created, ignoring any that already exist, before any is dropped.
  • Only registrations change; no directory or data file is created or deleted.
  • Directory discovery and the catalog listing both finish before any change, so a listing failure changes nothing.
  • Partitions are matched by their escaped key=value/... name and registered with the values their directories spell, so a directory month=01 is registered as month=01, as in Java.
  • A partition registered at a custom location (options.path) is never unregistered, since its data does not live under the table directory.
  • MSCK without REPAIR and tables without catalog-managed partitions are refused.
• FormatTablePartitionPaths::discover finds the complete partition specs under the table directory, the counterpart of Java PartitionPathUtils.searchPartSpecAndPaths as repair calls it. Hidden directories are skipped except the value-only default partition directory, segments outside the configured layout and paths shallower than the partition depth are ignored, a missing root holds no partitions, and results are sorted and deduplicated by partition name.
• unescape_path_name moves from the Format Table scan to spec::partition_utils, beside escape_path_name, so the scan and discovery share it.

Tests

• crates/integrations/datafusion/tests/rest_format_partition_sql.rs:
  • test_msck_repair_reconciles_registrations_with_directories: ADD registers a directory the catalog does not know, and SYNC unregisters a partition whose directory was removed while the other directory stays;
  • test_msck_repair_keeps_a_partition_at_a_custom_location: SYNC keeps a partition registered at a custom location even though no directory for it exists under the table.
• crates/paimon/tests/format_partition_test.rs: discover_format_partitions_returns_complete_sorted_specs, discover_format_partitions_treats_missing_root_as_empty, discover_value_only_partitions_rejects_parent_traversal_value and discover_value_only_partitions_keeps_hidden_default_directory.
• Unit tests spec::partition_utils::tests::test_unescape_path_name and table::format_partition::tests::test_storage_not_found_matches_only_not_found.

Commands run on the top commit:

cargo fmt --all -- --check
cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
cargo test --locked -p paimon --all-targets --features fulltext,vortex
cargo test --locked --no-fail-fast -p paimon-datafusion --all-targets
cargo test --locked -p paimon-rest-server --all-targets

fmt and clippy pass. paimon: 2966 tests pass. paimon-rest-server: 10 tests pass. paimon-datafusion: 774 pass, including the 9 tests in rest_format_partition_sql.rs, and 39 fail. The failures are the 39 tests that also fail by name on main in this environment, because they read fixture tables provisioned by make docker-up (two also need the lumina native library).

Additions without a direct Java counterpart

• DROP and SYNC list the table directory before anything else and fail if it cannot be listed. Discovery, like Java's, reads a missing root as a table without partitions; Java's repair has no such check, so a missing table directory would make DROP or SYNC unregister every partition.
• Discovery fails on a matching directory name whose escaping is malformed or not canonical, because the spec registered for it would not resolve back to the same directory. Java's unescapePathName keeps a malformed escape as written.

API and Format

• New public method paimon::table::FormatTablePartitionPaths::discover.

The storage format is unchanged.

Documentation

docs/src/sql.md gains an MSCK REPAIR TABLE subsection under Format Table Partitions and lists MSCK REPAIR TABLE in the SQL support scope.

Scope and limitations

• Applies only to partitioned internal REST Format Tables with metastore.partitioned-table=true and a non-engine implementation.
• Repair always covers the whole table, and there is no dry run.
• Java's format-table.repair.collect-statistics, which also measures the partitions a repair finds, is not supported; ANALYZE TABLE (feat(datafusion): support ANALYZE TABLE on catalog-managed format tables #815 ) measures them afterwards.