[Python][JSON] Leading null list elements produce invalid list offsets during type inference #51305

Reproduced offline with PyArrow 25.0.0, Python 3.11.5, on Linux.
The reproducer uses only PyArrow and Python's standard library; Hugging Face
Datasets is not required. The downstream issue is huggingface/datasets#5531.

Minimal reproducer

import io
import json
import pyarrow as pa
import pyarrow.json as paj
data = b'{"a":[null,1]}\n' # 15 bytes, one valid JSON object
assert json.loads(data) == {"a": [None, 1]}
print(pa.__version__)
table = paj.read_json(
 io.BytesIO(data),
 read_options=paj.ReadOptions(block_size=len(data)),
)
array = table["a"].chunk(0)
print(array.offsets.to_pylist(), len(array.values))
table.validate(full=True)

Observed output (the read itself succeeds):

25.0.0
[0, 2] 1
pyarrow.lib.ArrowInvalid: Column 0: In chunk 0: Invalid: Length spanned by list offsets (2) larger than values array (length 1)

Expected: validate(full=True) succeeds, offsets are [0, 2], the child array
contains [None, 1], and the table contains one row {"a": [None, 1]}.
The list offsets count the leading null, but the child array is one element short.
Two leading nulls ([null,null,1]) produce offsets spanning three elements with
a child array of length one.

Block-dependent example

import io
import pyarrow as pa
import pyarrow.json as paj
data = b'{"a":[1]}\n{"a":[null,2]}\n' # 25 bytes
for block_size in (13, len(data)):
 table = paj.read_json(io.BytesIO(data), read_options=paj.ReadOptions(block_size=block_size))
 try:
 table.validate(full=True)
 print(block_size, "valid")
 except pa.ArrowInvalid as error:
 print(block_size, error)

Observed:

13 Column 0: In chunk 1: Invalid: Length spanned by list offsets (2) larger than values array (length 1)
25 valid

The preceding typed list prevents corruption when both records are in one block.
When they are parsed in separate blocks, the later record can corrupt its block.
The one-record reproducer above needs no internal block boundary or long string.

Reduction of the original attachment

The local sanity_oscar_en.jsonl from datasets#5531 contains 74,639,409 bytes,
10,000 rows, each accepted by json.loads. Direct PyArrow reads measured:

Column 2 is meta; struct child 3 is line_identifications, a
list<struct<label: string, prob: double>>. Of the 10,000 lists, 6,706 contain a
null and 945 begin with one. The first failing 1 MiB chunk starts at zero-based
source row 336, whose list begins with four nulls, matching the four missing child
values in that chunk.

Row bisection (trying 1 MiB, maximum row length, and subset length block sizes as
alignment changed) reduced the attachment to original row 2, saved as
min.jsonl: 3,410 bytes, one row. A nonempty row set cannot be smaller in row
count. Its meta.line_identifications list has 22 elements, including 10 nulls;
positions 0–3 are null, and position 4 is the first typed struct
{"label":"en","prob":0.8105751276}. Null positions are
[0,1,2,3,5,8,13,15,17,19].

Both block_size=2048 (record spans a boundary) and block_size=4096 (one block)
return an invalid table. At 4096, full validation reports:

Column 2: In chunk 0: Invalid: Struct child array #3 invalid: Invalid: Length spanned by list offsets (22) larger than values array (length 18)

Removing text and all unrelated fields still fails. Removing just the four
leading nulls makes this row validate; the later nulls can remain. The record's
long text and metadata are therefore unnecessary for this reproducer.

Additional single-block controls run on 25.0.0:

• [1,null], [null], and []: valid.
• A null list followed by a typed list in another row: valid.
• [null] followed by [1] in another row: valid.
• [null,"s"] and [null,[1]]: invalid, one child value missing.
• A list inside a struct containing [null,{"label":"en","prob":1.0}]:
  invalid, one child value missing.

These results implicate null-to-typed list-value inference within a parser block.
They do not establish a particular C++ implementation defect; no Arrow C++ build
or internal debugger investigation was performed.

Downstream effect and workaround limits

read_json returns normally with invalid arrays, so consumers may fail later.
Before the datasets fix, the original-row regression failed at full validation
with the 22-versus-18 error above. Tests with typed and leading-null records in
separate blocks also failed before the fix.

A whole-file single-block retry validates for the complete attachment, but cannot
repair a file beginning with the minimal leading-null list. Such inputs need a
different parser, for example pandas' JSON decoder followed by an object-typed
DataFrame and Arrow conversion. The datasets fallback uses the decoder with
precise_float=True; object dtype avoids automatic date conversion and rounding
of nullable integers through floats. Neither invalid tables nor their nested arrays should
be converted to Python objects before successful validation.

Only PyArrow 25.0.0 was tested in this investigation. Other Arrow versions,
the gated dataset mentioned later in the issue, and the exact C++ cause remain
unverified. This report has not been submitted upstream (offline task).