Support FileIndex generation for append writes #818

Search before asking

• I searched in the issues and found nothing similar.

Description

Rust can read Bitmap and Bloom Filter FileIndexes, but normal append writes do not generate them: DataFileWriter leaves extra_files empty and embedded_index unset.

This is a write-side follow-up to #34 and #780. Reuse the existing index writers and format to:

• Parse index column declarations, per-column options, and file-index.in-manifest-threshold, with explicit errors for invalid or unsupported configurations.
• Build indexes per physical data file, preserving row order and NULL positions across batches and resetting state on file rolling.
• Store the complete serialized index in embedded_index when its size is at most the threshold; otherwise write a .index sidecar and register it in extra_files. Match Java's default of 500 B.
• Handle both foreground and background close paths. Return metadata only after data and index completion succeeds, and clean up newly created sidecars on failure.

Tests should cover unconfigured writes, Bitmap/Bloom, multiple columns, NULLs, empty batches, rolling, threshold boundaries, and failure/abort cleanup. After commit, verify identical query results with indexes enabled and disabled, actual FileIndex pruning, and residual filtering for Bloom false positives.

Scope is limited to ordinary append writes and supported top-level columns. KV writes, COW/DataEvolution rewrites, dedicated Blob/Vector paths, nested indexes, additional index types, and public API expansion are excluded. Java-generated table fixtures remain a separate read-compatibility follow-up.

Willingness to contribute

• I'm willing to submit a PR!