perf(vindex): enable 8-way multipart uploads for index files - #798

Purpose

Reduce the vector-index serialization/upload bottleneck by allowing up to eight concurrent multipart uploads, while keeping each chunk at 8 MiB.

This is a standalone upload-only change. The existing single remote scan, training, and add flow remain unchanged.

Brief change log

• Extract an internal async_writer_with_concurrency() helper. Existing async_writer() callers retain concurrency 1 and 8 MiB chunks.
• Use concurrency 8 for vector-index output directly, without an MVP environment switch.
• Add round-trip, out-of-order multipart completion, and index-build failure tests through the existing FileIOProvider test entry point. Reuse the existing build failure handling.

Tests

Coverage added or updated; runtime acceptance is still pending:

• cargo fmt --all -- --check.
• git diff --no-ext-diff --check.
• Memory/fs round trips at concurrency 1 and 8: write nine full 8 MiB chunks plus a 12,345-byte tail, then verify length and every byte after shutdown.
• Exercise OpenDAL's multipart scheduler with deterministic out-of-order part completion and verify the final content.
• Run existing FileIO, cache-invalidation, and vector-index build tests. The cache-invalidation test now exercises concurrency 8.
• Inject both a part-write failure and a final close failure: require a build error, unchanged committed index metadata, unchanged existing index bytes and query results, and cleanup of newly completed shard files.
• Run the ignored streaming_upload_oss test against real OSS multipart storage and verify scratch-object cleanup. It requires PAIMON_CATALOG_OPTIONS, PAIMON_UPLOAD_TEST_DATABASE, and PAIMON_UPLOAD_TEST_TABLE.
• Complete three interleaved runs per variant on vdb2: default upload versus 8-way upload, with the same source baseline, PR90 core, 10M data, index parameters, and queries. Record source/binary checksums, total build and serialization/upload time, peak memory, index size, and Recall; retain all runs and check an absolute Recall difference of at most 0.005.

The vdb2 verification attempt stopped during dependency download, before compilation, because the Cargo registry connection timed out. Runtime correctness and performance acceptance have not been established; no measured speedup is claimed here.

API and Format

No public API or storage-format changes. Upload concurrency changes only for vector-index output.

Known existing limitation: the AsyncWrite adapter does not expose multipart abort. The failure test explicitly records one unfinished upload and zero abort calls, separately from completed-object cleanup. This patch does not add multipart-abort support; real OSS failure cleanup still needs verification.

Documentation

No new user-facing configuration. No documentation changes are included.