[WIP][POC][BenchmarkingOnly] PFOR interleaved and fastlanes numbers - #51296

Implements the PFOR (Patched Frame of Reference) integer compression
algorithm as a standalone utility library in arrow/util/pfor/. Includes:
- Cost model for optimal bit width selection (histogram-based)
- Vector-level encode/decode with FOR + bit-packing + exceptions
- Page-level wrapper with header, offset array, and multi-vector layout
- Comprehensive unit tests covering edge cases and round-trips

Adds PFOR = 11 to the Encoding enum and wires it into the parquet
read/write pipeline:
- PforEncoder<DType> in encoder.cc (buffers values, calls PforWrapper::Encode)
- PforDecoder<DType> in decoder.cc (decodes all values on first access)
- PFOR case in column_reader.cc InitializeDataDecoder
- Encoding string mapping in types.cc
Supports INT32 and INT64 column types.

Benchmarks encode/decode throughput for int32/int64 across 10 data
distributions inspired by Snowflake's NumericComprBenchmark: constant,
sequential, small range, high-base-small-range (timestamps), with
outliers (exception path), random, TPC-DS date/store/item/quantity keys.
Each distribution runs at 1K/10K/100K/1M elements. Reports bytes/s,
items/s, and compression ratio.

...alize

...ith SafeCopy

...ainder

...Size

Load() now returns Result<PforVectorInfo> after the Status/Result
refactoring. Use ASSERT_OK_AND_ASSIGN to properly unwrap the result
in tests.

Make LoadHeader fallible: move the header-size check from Decode into
LoadHeader, return Result<PforHeader>, and update Decode to use
ARROW_ASSIGN_OR_RAISE. Mirrors the corresponding ALP review fix on
gh540-alp-pseudoDecimal-encoding.

Replace std::memcpy / raw byte writes in PforWrapper::StoreHeader,
LoadHeader, and the offset-array read/write paths with
util::SafeLoadAs and util::SafeStore. Mirrors the corresponding ALP
review fix on gh540-alp-pseudoDecimal-encoding.

Reject invalid packing_mode, value_byte_width mismatch, log_vector_size
out of [kMin, kMax] range, and negative num_elements when loading the
PFOR page header. Removes the redundant packing_mode and
value_byte_width checks from Decode now that they live in LoadHeader.
Mirrors the corresponding ALP review fix on
gh540-alp-pseudoDecimal-encoding.

...sites
Replace size_t with int64_t for max_size/comp_size to match the
PforWrapper API signature, and qualify pfor::PforWrapper as
::arrow::util::pfor::PforWrapper to avoid ADL ambiguity.

Aligns with Arrow buffer conventions (Buffer::data() returns uint8_t*).
Removes the reinterpret_cast<char*> at the parquet encoder/decoder
call sites and switches std::vector<char> compressed buffers to
std::vector<uint8_t> in the unit test and benchmark.
Also fixes a pre-existing size_t / int64_t* mismatch in
pfor_benchmark.cc that surfaced once the buffer pointer type was
tightened. Mirrors the corresponding ALP review fix on
gh540-alp-pseudoDecimal-encoding.

...th validation
Per Google C++ style, replace the PforVectorInfo struct with a class
that has private trailing-underscore members and getter/setter
accessors. Replace std::memcpy calls in Store/Load and the exception
patch loop in DecodeVector with util::SafeLoadAs / util::SafeStore.
Add bit_width range validation inside Load() so callers don't have to
repeat the check.
Updates all access sites in pfor.cc and pfor_test.cc to go through
the new accessors. Caches num_exceptions() in a local in DecodeVector
so the #pragma GCC unroll can still see a constant loop bound.
Mirrors the corresponding ALP review fix on
gh540-alp-pseudoDecimal-encoding.

Per Google C++ style, both types become classes with private
trailing-underscore members and const getters, mutable getters, and
setters. Updates all access sites in pfor.cc (EncodeVector,
LoadView, SerializedVectorSize, SerializeVector) and pfor_test.cc
to go through the new accessors. Mirrors the corresponding ALP
review fix on gh540-alp-pseudoDecimal-encoding.

..., use ctor in EncodeVector
- Move the num_exceptions < 0 check from DecodeVector into
 PforVectorInfo::Load alongside the bit_width range check, so all
 loaded-data invariants are enforced at the same layer.
- Use PforVectorInfo's parameterized constructor in EncodeVector
 instead of three separate setter calls on a default-constructed
 instance.

...DOs (PFOR)

Commit 00b6318 introduced ARROW_DCHECK(bit_util::IsPowerOf2(vector_size))
in PforWrapper<T>::Encode, but vector_size is int32_t and bit_util has
overloads only for int64_t and uint64_t -- the call is ambiguous and the
file no longer compiles.
Cast to int64_t to disambiguate. CeilDiv calls in the same file already
promote to int64_t implicitly via its int64_t-only signature.

Portable C++ port of FastLanes (Afroozeh & Boncz, VLDB '23) for int32_t
columnar data. No SIMD intrinsics in the kernels — the inner lane loop
is structured (contiguous loads from packed[w*kLanes + lane], contiguous
stores to transposed[r*kLanes + lane]) so the compiler auto-vectorizes
to 4-wide NEON / 8-wide AVX2 / 16-wide AVX512 without source changes.
Layout: lane-interleaved 1024-bit format per the paper. 1024 values
pack as w u32 rows of 32 u32 lanes. FL_ORDER (8x16 -> 16x8 sub-block
transpose + 3-bit-reversal sub-block reorder) is applied OUTSIDE the
kernel: FastLanesForCodec::Encode gathers input[fromTransposed32(t)]
before packing; Decode produces output in transposed order (no scatter,
output[t] == input[fromTransposed32(t)] + min within each 1024-block).
FastLanesForCodec adds Frame-of-Reference on top:
 - 2048-value chunks (2 FastLanes blocks per chunk)
 - Per-chunk 5-byte header: [min(4B int32 LE)] [bit_width(1B)]
 - Subtract min before packing; add back on decode
 - bit_width=0 path stores no payload (constant chunk)
Files:
 cpp/src/arrow/util/fastlanes/fastlanes_kernels.h
 - PackBlock<W>(in, out) / UnpackBlock<W>(packed, out)
 - W=32 fast path: std::memcpy
 - fromTransposed32 helper
 cpp/src/arrow/util/fastlanes/fastlanes_for.{h,cc}
 - FastLanesForCodec::{Encode,Decode}
 cpp/src/arrow/util/fastlanes/fastlanes_for_test.cc
 - 5 round-trip tests (narrow range, single value, full int32 range,
 multiple chunks, boundary values) — all passing
CMakeLists.txt wires the test as arrow-fastlanes-for-test.

Wires the new FastLanesForCodec into the existing pfor_comparison_benchmark
harness alongside PFOR, DeltaBitPack, ZSTD, LZ4, RleBitPack, and Bss
codecs. New BM_FastLanesEncode / BM_FastLanesDecode functions follow the
same Gen32 + ::Apply(CustomArgs) shape; REGISTER_DATASET macro picks them
up for every ClickBench dataset.
Notes on the comparison:
- FastLanes decoder produces output in TRANSPOSED order
 (output[chunk*2048 + block*1024 + t] == input[chunk*2048 + block*1024 +
 fromTransposed32(t)] + min). PFOR/DeltaBitPack produce flat output.
 The benchmark measures decoder throughput head-to-head; consumers of
 FastLanes output must be permutation-aware (which is the FastLanes
 paper's intended architecture).
- num_values is rounded down to a multiple of 2048 (FastLanes chunk
 size) inside BM_FastLanesEncode / BM_FastLanesDecode for compatibility
 with the existing 102400-value test sizes.
Also guards add_executable(parquet-pfor-comparison-benchmark) with
if(ARROW_BUILD_BENCHMARKS) so non-benchmark configurations don't fail
the cmake configure step.
Bench numbers on aarch64 (102400 int32, 3-run median):
 EventDate decode: FastLanes 20us vs PFOR 36us vs Delta 122us
 EventTime decode: FastLanes 24us vs PFOR 56us vs Delta 140us
 GoodEvent decode: FastLanes 20us vs PFOR 34us vs Delta 119us
Compression ratios match or slightly beat PFOR on every dataset tested.

FastLanesForCodec::DecodeFlat unpacks into a transposed scratch buffer
per chunk and then scatters via fromTransposed32 to produce output in
original input order — output[i] == input[i] for the encoded input.
This is the FL_ORDER inverse of the gather step in Encode.
Adds:
 - DecodeFlat method + round-trip test (DecodeFlatIsIdentity) covering
 4 chunks of random data. All 6 round-trip tests still pass.
 - BM_FastLanesDecodeFlat in pfor_comparison_benchmark, registered in
 the per-dataset macro for apples-to-apples vs PFOR / DeltaBitPack
 (both of which produce flat output).
Bench (102400 int32, 3-run median, aarch64):
 Dataset FL Decode FL DecodeFlat PFOR Decode Delta Decode
 EventDate 20 us 108 us 38 us 123 us
 EventTime 23 us 113 us 57 us 140 us
 GoodEvent 20 us 107 us 35 us 119 us
The transposed-kernel decode beats every other codec by 1.5-7x. The
flat-output decode pays an ~85 us scatter cost per 100K values that
makes it slower than PFOR but still faster than DeltaBitPack. The gap
is exactly the FL_ORDER scatter — the reason FastLanes' intended
architecture keeps data in transposed order through the query.

The 8x16 -> 16x8 within-sub-block transpose is mutual-inverse with the
16x8 -> 8x16 transpose, NOT self-inverse. The previous docstring on
fromTransposed32 said "Self-inverse: fromTransposed32 is also
toTransposed32" — that was wrong. fromTransposed32(fromTransposed32(t))
does not equal t in general; e.g. fromTransposed32(1) = 16,
fromTransposed32(16) = 2.
Add the actual toTransposed32 (forward-direction mapping) and fix the
docstring. Callers that need to invert a gather computed with
fromTransposed32 (i.e. read out[i] = transposed["the t whose
fromTransposed32(t) = i"]) must use toTransposed32(i).

Adds an additive packing-mode option to PFOR. Existing vectors round-trip
unchanged (default PackingMode::BitPack); new vectors can opt in to the
FastLanes lane-interleaved bit-packing layout via the per-vector flag.
On-disk format change (backwards-compatible):
 - The 1-byte bit_width field of PforVectorInfo now packs two values:
 bits 0..5 = the actual bit width (range 0..32 fits in 6 bits)
 bit 7 = packing-mode flag (0 = BitPack, 1 = FastLanes)
 bit 6 = reserved
 - Legacy encoders only wrote the bit width, leaving high bits clear,
 so they decode as PackingMode::BitPack via the new Load.
 - PFOR header (page-level) is unchanged.
API:
 - New enum class arrow::util::pfor::PackingMode { BitPack, FastLanes }.
 - PforVectorInfo gains a packing_mode field and getter/setter.
 - PforCompression<T>::EncodeVector takes an optional PackingMode (default
 BitPack). FastLanes mode is only honored when num_elements equals the
 FastLanes block size (1024) and T is 32-bit; otherwise it falls back
 to BitPack per-vector (so tails and 64-bit values continue to work).
 - PforCompression<T>::DecodeVector reads the per-vector flag and
 dispatches between arrow::internal::unpack and the FastLanes kernel.
 - PforWrapper<T>::Encode takes an optional PackingMode threaded down to
 EncodeVector.
Decode-side perf (fused gather + FOR-add + SafeCopy):
 The FL_ORDER inverse needs toTransposed32(i) — note: NOT
 fromTransposed32(i), the two are mutual inverses, not self-inverse.
 The scalar gather over can't be SIMD-vectorized, so
 PFOR+FastLanes decode is ~2-3x slower than PFOR+BitPack end-to-end
 despite the kernel itself being competitive. The win is only available
 when the downstream consumer can work with data in FastLanes transposed
 order (i.e. relax the flat-output contract).
Tests: 5 new tests in PforPackingModeTest cover round-trip identity for
both modes, the partial-tail fallback to BitPack, mixed-mode round-trip
through PforWrapper, and the bit_width=0 (constant vector) path. All 30
PFOR tests pass.
Benchmark: BM_PforFastLanesEncode / BM_PforFastLanesDecode added to
pfor_comparison_benchmark.cc, registered per dataset alongside the
existing 8 codec variants.

For FastLanes-encoded vectors the decoder previously always paid a
1024-element scalar FL_ORDER gather to produce flat output. That gather
is what made pfor+fastlanes 2-3x slower than pfor+bitpack overall, even
though the FastLanes unpack kernel itself is competitive.
The FastLanes paper's intended decode path is to NOT do that scatter at
all: keep the data in FastLanes stream order and let downstream
operators be permutation-aware (apply fromTransposed32 lazily, when
they need original index). This commit exposes that path.
API:
 - New enum class arrow::util::pfor::OutputOrder { Flat, Transposed }.
 - PforCompression<T>::DecodeVector and PforWrapper<T>::Decode take an
 optional OutputOrder (default Flat, backwards-compatible).
 - OutputOrder::Transposed only affects FastLanes-encoded vectors.
 BitPack vectors have no permutation to skip, so they always produce
 flat output regardless of the argument (mixed pages with a BitPack
 tail end up flat in the tail, transposed in the full blocks).
Decoder paths in DecodeVector when packing_mode == FastLanes:
 - Flat (existing): unpack -> scratch transposed[] -> fused
 values[i] = SafeCopy(transposed[toTransposed32(i)] + FOR)
 The toTransposed32 gather is scalar, breaks auto-vec.
 - Transposed (new): unpack -> scratch transposed[] -> sequential
 values[t] = SafeCopy(transposed[t] + FOR)
 Pure sequential read/write, auto-vectorizes cleanly. Exceptions are
 patched at toTransposed32(pos) so the stored-flat positions land in
 the right transposed slots.
Tests: 4 new tests in PforOutputOrderTest cover (a) transposed output
satisfies the FL_ORDER relation, (b) manual inversion of the
permutation reconstructs the input, (c) BitPack vectors ignore the
Transposed request, (d) wrapper-level transposed decode across many
vectors. All 34 PFOR tests pass.
Benchmark: BM_PforFastLanesDecodeTransposed added, registered per
dataset. On 18 ClickBench-style datasets (102400 int32 each):
 pfor+bitpack 33-57 us
 pfor+fastlanes (flat) 98-108 us (0.34-0.53x — slower)
 pfor+fastlanes (transp) 20-23 us (1.6-2.5x faster than bitpack)
The transposed path beats every other codec measured in the comparison
benchmark on every dataset.

The comparison benchmark's generators are all either unordered or perfectly
regular, so none of them separates an encoding that differences neighbouring
values from one that packs them, and none of them puts a frame of reference
anywhere but the minimum. That left PFOR's delta mode and
DELTA_BINARY_PACKED untested against each other on the columns where they
actually make different choices.
Register the ten shapes from pfor_benchmark.cc here too, at both widths:
timestamps regular and bursty, a sawtooth, bounded-rate series, monotonic
ids with and without gaps, a low sentinel below the cluster, and two
clusters no single window covers.
These are templates rather than a pair of per-width functions, which keeps
the int32 and int64 arms on provably the same distribution. They sit in
their own namespace because two of them build on base distributions whose
names this file already uses for columns drawn from different seeds; the
figures are therefore comparable with pfor_benchmark.cc.
Benchmark count goes from 476 to 756 (20 datasets x 14 codec arms).

The delta mode was costed by writing every difference out and running a
frame search over them, which is most of what encoding a vector costs, and
it was charged on every vector including the ones that went on to decline
it. Estimate first from a strided sample of the differences and drop the
mode there when the estimate cannot reach the incumbent, so a vector that
will not use it pays a fraction of a pass instead of two full ones.
The estimate samples widths rather than a span. A gate on the span of the
differences was tried first and had to go: a sawtooth is a tight cluster of
small positive differences with a handful of large negative ones, so its
span is as wide as its raw span while its cost is a fraction of it.
Zigzagging is what lets a histogram stand in for a search that has not run,
since differences in [-k, k] zigzag into the same [0, 2k] a frame at -k
would produce.
On 20 distributions at both widths, encode throughput on the vectors that
decline the mode is 1.55-1.93x what it was, and the vectors that accept it
pay 1.4-5.2%. Two earlier shapes were measured and dropped: accumulating
the histogram over every difference inside the differencing walk costs
6-27% on accepting vectors, and moving it to a pass of its own changes that
by under 3%, so the cost is the histogram work and not a lost
vectorization.
A test pins the estimate against the ungated chooser -- same mode, width,
frame and cost -- over every distribution the benchmark covers, so an
estimate that turns pessimistic shows up as a lost delta rather than as a
silent ratio regression.

PFOR is a Preview feature in the Parquet format, so a writer must not emit
it unless the user has asked for it. Selecting Encoding::PFOR without
calling enable_pfor_encoding() now makes WriterProperties::Builder::build()
throw, naming whichever column asked for it; the Builder's constructor is
the only way to reach a WriterProperties, so the check cannot be bypassed.
Decoding is unaffected -- a file that exists is always readable.
The delta mode gets its own property, on by default. It is part of PFOR
rather than a separate encoding: every reader has to handle it, because each
vector says in its own header which mode it used, so a page written with the
mode disabled reads back through the same decoder. The property exists
because differencing costs encode time on vectors it then declines, so a
writer that knows its data is not sequential can skip the search.
The option travels as a PforEncodeOptions struct rather than a bare bool, so
a later mode does not change these signatures again. MakeEncoder and
MakeTypedEncoder take the WriterProperties, which is how the per-column
delta setting reaches the encoder; a caller with no properties to hand gets
the defaults.
Tests: 14 property tests covering the opt-in, the per-column overrides and
the interaction between the global and per-column maps; and an end-to-end
test that writes an arithmetic run twice and requires the column chunk to be
smaller with the mode left on. A negative control -- the encoder factory
ignoring the property -- makes that end-to-end test fail with both sizes
equal, so it does reach the encoder rather than stopping at build().

Two cases the corruption tests did not reach, both specific to the flag in
bit 7 of the bit-width byte.
Setting the flag on a vector that was written without one makes the decoder
consume sizeof(T) extra bytes of metadata. On a page sized exactly, as the
encoder writes it, that pushes the vector past the end of the buffer and the
decode has to fail rather than read on.
Truncating a page inside a delta vector's start value leaves enough bytes for
the info block and not enough for the vector. The bound that catches this is
the one place a length check reads is_delta(), and it sits before the start
value is loaded -- so without it the load runs sizeof(T) bytes past the end
and only the later payload bound reports the problem. Removing it as a
negative control leaves the test failing on the error message, which is all a
non-sanitizer build can see of an overread; the assertion is on the message
for that reason.
Store also asserts, in debug builds, that the width it is about to pack fits
the seven bits it has. A wider one would lose its high bits to the mask and
set the delta flag on the way out, which is the failure the seven-bit mask was
introduced to prevent -- Load rejects it on the way in, and this reports it at
the encoder instead.

The sum is one dependent add per value and uses no lanes, which is why a
delta vector does not pick up the speed a narrower type gives the unpack.
A scan across lanes plus a carry broadcast would shorten the chain, but
only for a vector with no exceptions: patching sits between the frame add
and the sum, so a vector holding even one exception keeps this loop.

DecodeVector left UnpackOptions::max_read_bytes at its -1 default, which
tells the bit-unpacker it may not read a byte past the vector's own
packed payload. The vector kernels load a fixed-size window per step,
wider than a step consumes at most bit widths, so under that bound the
last step is refused and the tail of the vector falls to the scalar
epilog -- 32 of 1024 values at bit width 3.
The span DecodeVector is handed runs from this vector to the end of the
page, so the true bound is already in hand; pass it. Whole-page int32
decode gains a median 1.13x over a 29-column corpus and 1.9x on the
columns that pack to 3 bits, and nothing at widths 1, 2, 4, 8, 16 and
31, whose kernels strand no values to begin with. The gain is flat from
a 78 KiB data page to a 4 MB destination, which is what a per-vector
cost looks like.
The bound is the distance from the packed payload to the end of the
span, so it is measured from the metadata size this vector actually
wrote: a delta vector's header carries a start value as well, and the
read pointer is already past it. Taking the fixed metadata size instead
would offer the kernels sizeof(T) bytes past the end of the caller's
buffer -- and, because those kernels only widen their window when it
fits, would do so on exactly the vectors whose last step currently gets
refused.
(cherry picked from commit d3490a55710e1c83fc0e2f8378e9c101e3e593f0)
(cherry picked from commit 3a3b55b)

The layout is the container FastLanes describes (Afroozeh & Boncz, VLDB
2023): a block of 1024 values is cut into 32 lanes of 32 rows, and row r
of every lane is packed at the same bit offset. Because the shift and the
straddle depend on the row and never on the lane, all 32 lanes do
identical work at every step, and the lane loop autovectorizes without a
single intrinsic.
The payload is the same size as the sequential layout for a full block --
1024 * w bits either way -- so this trades no space. At w = 32 it is the
same bytes as well, and the kernel is a memcpy there.
UnpackBlock takes a kHasBias template parameter so a frame-of-reference
add can be folded into the kernel's own store. The add itself is nearly
free; what it saves is the second traversal of the output, which is the
part that costs.
Only the container is here. FastLanes also defines a value permutation
(FL_ORDER, 04261537), which is not implemented: its customer is a codec
that reads a lane sequentially, and a positional decoder would have to
gather the permutation away again.

The page header already carries a validated packing_mode byte, so the
choice of layout needs no new field and no spare bit: mode 0 is the
sequential bit-packing this encoder has always written, and mode 1 is the
lane-interleaved layout. A reader that predates mode 1 rejects the page
instead of misreading it, which matters here because the two layouts
produce byte-identical payload sizes -- a reader that guessed wrong would
return wrong values rather than run short.
The interleaved layout applies only to a vector that is exactly 1024
values wide and 4 bytes per element. Both sides derive that from the
element count rather than from a per-vector flag, so the short tail
vector at the end of a page is sequential without having to say so.
LoadHeader also rejects a page that declares mode 1 alongside a vector
size or element width that cannot support it.
Encode takes the mode as a request, not an instruction: an int64 column
or a non-default vector size silently records mode 0, so one writer-side
setting can cover a file whose columns are not all eligible.

Every bit width round-trips, framed and unframed, and the framed cases go
through the unpack that folds the frame into its own store. The widths are
built from a delta mask rather than a range so the cost model cannot land
on a narrower width and leave the loop testing one kernel thirty-one
times; the test reads back the width the encoder settled on and checks it.
One test exists only to keep the others honest. Because the two layouts
write the same number of bytes, a mode request that quietly fell back to
sequential on both the encode and the decode side would pass every round
trip above. So one test asserts the payloads really are different bytes at
widths 1 through 31 -- and identical at 32, where the two layouts coincide
and the kernel is a memcpy.

The layout is not in the Parquet specification, so a file written with it
can only be read by an implementation that knows it. It is therefore off
unless a writer asks:
WriterProperties::Builder::enable_pfor_interleaved_bit_packing. A reader
that does not know the layout rejects the page rather than misreading it,
because the mode is recorded in a header byte the reader already
validates.
The flag is a request. It reaches only 4-byte columns, and within those
only the full vectors of a page, so a file whose columns are not all
eligible can still be written with one setting: everything the layout
cannot cover is written the way PFOR has always written it.
MakeEncoder grows a defaulted parameter rather than taking the properties,
which would put the whole of the writer's configuration behind a factory
that needs one bit of it.

Because both layouts write the same number of bytes, a flag that never
reached the encoder would still round-trip a table. So the test writes the
same table with the flag and without it and compares the two files: same
length, different bytes is what shows the flag arrived.
The int64 case asserts the opposite -- the two files are identical, byte
for byte -- because the layout cannot apply there and the request is
dropped rather than refused.

The benchmark only ever encoded with the default packing mode, so nothing
in the tree measured the layout this branch adds. Thread the mode through
the two benchmark bodies and register a paired int32 arm for each of the
ten distributions.
The encode arms report CompRatio%, which comes out identical to the
sequential arm's for every distribution: the two layouts write the same
number of bits, so the layout is a decode-speed choice and not a size
trade-off. There is no int64 arm because only 32-bit values reach the
interleaved kernels.

Every existing size is an exact multiple of the 1024-value vector, so the
tail path that falls back to sequential packing was never timed. The
Parquet writer caps a data page at max_rows_per_page = 20,000 rows, which
for int32 always binds before the 1 MB size target, so 20,000 values is
the destination a reader actually decodes into. Add it.
It also has a partial last vector (19 x 1024 + 544), which is what makes
it worth measuring separately from 10240 and 102400.

DecodeVector left UnpackOptions::max_read_bytes at its -1 default, which
tells the bit-unpacker it may not read a byte past the vector's own
packed payload. The vector kernels load a fixed-size window per step,
wider than a step consumes at most bit widths, so under that bound the
last step is refused and the tail of the vector falls to the scalar
epilog -- 32 of 1024 values at bit width 3.
The span DecodeVector is handed runs from this vector to the end of the
page, so the true bound is already in hand; pass it. Whole-page int32
decode gains a median 1.13x over a 29-column corpus and 1.9x on the
columns that pack to 3 bits, and nothing at widths 1, 2, 4, 8, 16 and
31, whose kernels strand no values to begin with. The gain is flat from
a 78 KiB data page to a 4 MB destination, which is what a per-vector
cost looks like.

The property told a writer the layout "decodes faster", which the
measurements do not support without a qualifier: on whole-page int32
decode it is about 1.08x at -O3, a tie at the -O2 an Arrow Release build
compiles with, and slower than the sequential layout once the
destination reaches a few MB. Its kernels are plain loops that depend on
the autovectorizer, which is where the level sensitivity comes from.
The byte-count half of the claim stands -- the two layouts pack the same
number of bytes, which InterleavedCostsNoSpace pins.

lane_delta.h gives lane l the container rows l, 32+l, ..., so its in-lane
predecessor is 32 positions back in file order and its differences are wider
than DELTA_BINARY_PACKED's. The paper assigns lane l the contiguous run
[32l, 32l+32) instead, which makes the in-lane predecessor the immediately
preceding value and the stored differences exactly the format's, at the price
of 32 entry points per block and a transpose back to file order.
transposed_delta.h implements that, with the bases stored either raw or
delta-encoded across lanes, and the benchmark adds four decode arms so the
base stream and the transpose can be priced separately from the container.

The paper's lane assignment finishes a block in transposed order, and a
Parquet decoder's contract is positional, so it owes a 32x32 permutation.
Run as a separate pass that permutation reads 1.14x even in NEON, which
made the ordering look like a trade against the container.
It is not one. A lane's prefix sum does not depend on any other lane, so
four lanes' chains run in one register and four rows of results can be
transposed and stored while they are still in registers. Fused that way
the permutation costs 0.995x against an arm that skips it and memcpys the
block out -- the same speed -- because it also spares the block the second
4 KB traversal the memcpy arm pays.
The base stream is now the whole remaining gap, so make its bit reader
branchless: one unaligned 64-bit load per base covers any width up to 32
at any bit offset, in place of a data-dependent shift loop. The size
bound gains 8 bytes of slack for the last load's over-read.
Conforming decode over 33 columns, -O2: 11.13 GB/s with a separate
transpose, 11.81 fused, 12.28 fused and branchless, against 12.17 for the
non-conforming arm and 13.41 for the +32 stride.

The layout had only ever been called as a kernel, from a benchmark that
handed it a payload and a value count. That skips the page header, the
decoder's dispatch and the level handling, so dividing a kernel rate by a
full decoder's rate overstates what a reader would gain.
Give it page framing and register it as an encoding, so the same encoder
and decoder interfaces every other encoding is reached through also reach
this one. The framing adds a little-endian value count ahead of the
payload: the payload's own header does not carry one, decode needs it to
locate every section, and a page's level count cannot stand in because it
counts nulls too.
INT32 only, because the packing kernel works on 32-bit lanes. The payload
still stores its packed words and per-block minimums in host byte order,
which a format specification would have to pin down before a file written
this way could be read anywhere else.

The existing arm measures the kernel. Add one that goes through the
registered decoder, so it pays for page framing, for SetData and for
reading the value count out of the page, and add the matching encode arm.
Both are then comparable with the DELTA_BINARY_PACKED encoder and decoder
in the same binary, which is the comparison a margin should be quoted
over: decoder against decoder rather than kernel against decoder.

The encoding is registered for INT32, so the fuzzer reaches it with pages
it has mutated. Four kinds of damage now raise instead of reading past the
buffer: a page shorter than the value count it declares, a page too short
to hold a count at all, a declared count larger than the page header's
level count allows, and a block claiming a bit width no 32-bit lane can
hold.

The 32 entry points of a block are bit-packed, and reading them with one
unaligned 64-bit load per value reads up to 7 bytes past the stream. That
was covered by slack in the kernel's own size bound, but a page is sized to
its contents: when the last block's differences are all equal its payload is
empty, the entry points end the page, and the load runs off the buffer.
Staging the stream -- at most 128 bytes, once per 1024 values -- keeps the
loop free of both a bounds test and short loads, and costs 3% of decode.
Splitting the loop instead, so that only the last values take a short load,
costs 8%: the split needs a division to find where to stop, and the fast
half loses its compile-time trip count.
A sanitizer harness reports an 8-byte read one byte past a 478-byte page
against the old code and nothing against this one, and a sweep of all 33
widths against a reference unpack agrees value for value.

The kernel encodes a whole array and is told its length; a page has to say
how long it is, and a decoder has to be able to tell a truncated or mutated
page from a valid one before it reads a byte of payload.
The framing is a 4-byte value count followed by the kernel's output, and
validation walks the blocks: each block's payload width and entry-point
width are bounded, and every block's contribution to the size is added up
and checked against the page. Decode reads in place when the payload is
4-byte aligned, which it is for a page written by this encoder, and copies
only when a caller hands it something else.

Both lane assignments break the format's single 1024-long dependency chain
into 32 independent ones, and both therefore decode with one vector add per
row. They differ in what a lane's predecessor is, and that decides the size:
striding 32 positions apart stores differences the format never would, which
costs 2.1-3.4x the bytes on the three correlated columns of the benchmark
corpus. Holding a contiguous run of 32 values per lane stores exactly the
format's differences, and lands within 1.5% of DELTA_BINARY_PACKED's size
across the corpus. One encoding cannot mean both, so the encoding writes the
second and the stride stays a kernel-level comparison.
Measured against the DELTA_BINARY_PACKED decoder in the same binary, over 33
columns at -O2: decode 2.9x faster on 33 of 33, encode 2.6x on 33 of 33. The
page framing costs under 2% of decode and no bytes.
The corruption test moves with the layout: block widths now open the payload
rather than sitting 128 bytes in, and the entry-point width lives inside a
block, so there are two positions to corrupt instead of one. Two tests cover
what the arrangement adds -- a page whose last block carries no payload, so
the entry points end it, and entry points far enough apart to need the full
width.

Compares the portable sequential bit-unpacker (new, arm 1), the in-tree
interleaved kernel (arm 2), and Arrow's dispatched unpacker (arm 3,
called through its real header against the built library) on
instructions/value and IPC per bit width, via self-process
perf_event_open counters with correctness gates on every arm.
Arm 1 and arm 3 are fixed, prebuilt inputs; only arm 2 recompiles
between the driver's -O2 and -O3 builds. Not wired into CMake:
perf_event_open is Linux-only and needs counter access the build
system can't guarantee everywhere. Build commands are documented in
driver.cc's header comment.

InterleavedBitPackingLayout and InterleavedRequestIgnoredForInt64 built
WriterProperties directly and never called enable_pfor_encoding(), so
both threw the preview-feature guard instead of exercising the layout.