perf(vindex): optimize IVF index build reads - #800

[P2] Preserve representative training samples across the shard

These fixed gaps permanently exclude contiguous regions; the seed only shifts the remainder allocation by approximately one row. For a 100,000-row shard with the default sample ratio of 1 and 65,536 retained vectors, the last sampled row is 99,468, so a newly appended distribution in the final 500 rows is never trained.

I reproduced this using the actual planner and vindex 0.4 trainer: 90,000 vectors [0], then 9,500 [1], then 500 [100], with IVF-SQ, L2, and nlist=nprobe=1. Querying [100] returns 10/10 results from the final cluster with the previous full-stream reservoir sampling, but 0/10 with these ranges. The SQ bounds are trained only on the older distributions, so the new vectors are clamped to the old upper bound even though every vector is subsequently added and every list is searched. Please retain the existing reservoir path until representative sampling is preserved, and add a recall regression test for data clustered by append order.

Verified at 28d6670: both original tail-distribution cases (100,000 rows with a final 500-row cluster, and 1,000,000 rows with a final 10,000-row cluster) now return 10/10 correct-cluster results. The specific fixed-gap regression described above is addressed. A separate small-sample case still fails and is documented on the current planner line here: #800 (comment)

[P2] Check actual page savings before enabling the extra source pass

An offset index being present does not mean these ranges avoid reading pages. With the default 100,000-row shard, 65,536 training rows, 128-dimensional vectors, and default Zstd/page settings, the 64 sample ranges leave gaps too small to skip pages or survive the reader's existing 1 MiB range coalescing. A tracking FileRead measured exactly 47,788,791 data bytes for both the sample and the full read, excluding metadata; the offset-index check still returned true. This fixture fits the default writer-buffer and file-size limits.

The subsequent full scan therefore doubles source data reads relative to the previous single source scan, including remote reads when the files are on OSS. This measures bytes, not overall wall time versus the saved local spill. Please estimate selected page ranges after coalescing and fall back, or adapt the sample plan, when the sample would read essentially the whole projection.

Rechecked at 28d6670: this remains reproducible with the current 512 short, jittered ranges. For the same default 100,000-row shard with 128-dimensional vectors, the sample and full scan still each fetch exactly 47,788,791 data bytes, excluding metadata. The latest concurrency and selected-page budget changes do not prevent the extra effectively full source pass. The sparse-path gate at writer.rs:173 still only checks offset-index availability; please include actual page savings after range coalescing, or fall back when the sample reads essentially the entire projection.

[P2] Keep small training samples distributed across the shard

When training_rows <= 128, this calculation produces just one contiguous range. A realistic incremental shard with 1,000 new rows and train.sample-ratio=0.1 therefore trains on 100 adjacent rows, whereas the previous implementation sampled every tenth row across the whole shard.

Using the actual planner and vindex 0.4 trainer, I reproduced this with 450 vectors [0], 450 [1], and 100 [100], IVF-SQ, L2, and nlist=nprobe=1. For snapshot 1, bucket 0, and an empty partition, the new range is [385,484], which excludes the entire final distribution. Querying [100] returns 10/10 results from the final cluster with the baseline, but 0/10 with this plan: SQ clamps those vectors to the older upper bound and returns rows 450–459. Please retain multiple strata for small samples, or fall back to the original sampling path, and cover this incremental-shard case in a recall regression test.

[P2] Include retained coalesced buffers in sparse read admission

scan_ranges sums the selected pages before ArrowFileReader merges byte ranges separated by at most 1 MiB. Its returned Bytes slices retain the larger coalesced allocations, but the new concurrent row-group admission uses only the smaller selected-page estimate. This leaves live source buffers out of the memory estimate used to increase parallelism.

I verified this with four valid Parquet row groups of 16,384 rows ×ばつ 128 floats, default Zstd/page settings, and one selected row every 4,096 rows. With a 20,971,520-byte budget, all four groups were admitted and simultaneously retained 25,390,984 bytes of owned read buffers. Allocation ownership and release were tracked with Bytes::from_owner and Drop; these counts exclude decoded Arrow arrays, and all tracked buffers were released on completion. Please account for the coalesced allocations when calculating admission costs, or retain conservative full-column admission when they cannot be estimated.