[core] Prune manifest blocks with row-id sidecar indexes - #9743

Here is a refined version of the block-oriented layout, keeping the file-level partition dictionary and making each block's two payloads independently extensible.

Header
 magic
 formatVersion
 manifest identity (name hash, file length, entry count)
 avroHeaderLength : int
 original Avro header : bytes
Partition Dictionary
 partitionCount : int
 entries[] // position is the partition ID
 partitionByteLength : int
 partitionBytes : bytes
blockCount : int
BlockIndexRecord[] // original manifest order
 offset : long // byte offset in the manifest
 length : long // complete Avro block length
 recordCount : long // number of manifest entries
 partitionEncoding : byte
 partitionPayloadLength : int
 partitionPayload : bytes
 rowIdEncoding : byte
 rowIdPayloadLength : int
 rowIdPayload : bytes
Checksum of all preceding bytes

The dictionary stores each complete partition tuple once, using the existing manifest partition serialization. This preserves tuple values and nulls; the scan's existing partitionType supplies their interpretation. Blocks reference dictionary IDs. The block ID is implicit in its position, and firstRecord is derived from preceding entry counts.

The encoding bytes identify how to decode the corresponding payload, with separate ID namespaces for partition and row-id payloads. They replace the availability flags:

The container's integers and the encoding-1 payload integers use fixed-width big-endian representation; partition bytes retain their existing serialization. Encoding bytes are interpreted as unsigned IDs. Each payload length counts only its payload bytes, excluding the encoding and length fields.

Other nonzero encoding IDs are reserved for future representations. If a reader does not recognize one, it skips exactly that payload length and treats that dimension as unavailable, while still being able to use the other dimension. Lengths must be bounded and validated. The outer formatVersion governs the container and dictionary framing; unsupported container versions or malformed metadata/payloads fall back to the normal manifest read.

For example, rowIdEncoding=1 with rangeCount=2 and ranges [100,109], [300,309] has a 36-byte payload: 4 + 2 * 16.

There are several important correctness and budget rules:

• Encoding 0 means "cannot prune using this information," never "no matches." An available payload must cover all relevant entries in the block, including ADD, DELETE and all column groups.
• Row-id coverage may be a conservative superset. If exact interval unions exceed the budget, merge intervals; the coarsest representation is rangeCount=1, [min,max], still using encoding 1. Continue processing the entire block to extend the bounds and detect unknown row IDs. If complete coverage cannot be established, use encoding 0.
• Partition information can independently become unavailable when its budget is exceeded. Consequently, the global dictionary is not necessarily a complete list of partitions touched by the manifest. A dictionary miss must not eliminate blocks with unavailable partition coverage.
• The physical block directory must always cover the entire manifest. Budget exhaustion may omit optional index payloads, but must never omit block descriptors. Validate byte coverage and entry counts, and verify the whole-file checksum before making pruning decisions.

For conjunctive partition and row-id filters, select each block using:

keepBlock =
 (partition coverage unavailable || partition predicate matches)
 &&
 (row-id coverage unavailable || query intersects indexed ranges)

Only an empty candidate block set permits skipping the manifest. Selected blocks still pass through the existing entry filtering and ADD/DELETE merge.

This keeps one sidecar and one record per block. It still assumes a bounded whole-sidecar read: payload lengths allow skipping decoding and unknown encodings, but do not by themselves save storage I/O. Index size, block selectivity and planning latency should determine whether selective physical reads are worthwhile later.