test: cover slice over expression-produced non-null element arrays (#... - #5839

Correctness

Prior behavior and regression coverage

The planner fix for #5743 is already present through #5766. This PR adds coverage and changes no production behavior. The earlier local-table test covers non-null element fields arriving at a scan boundary. These new queries produce their arrays after reading primitive columns from Parquet, covering the path that scan normalization cannot protect.

The column-dependent split, sequence and concat compositions cannot be folded away. Each uses checkSparkAnswerAndOperator, which compares against Spark and checks the Comet operator plan without exclusions. Source tracing confirms that the outer slice uses the native kernel. By default, split and array-valued concat use Spark codegen through the JVM dispatcher inside that pipeline. The integral sequence(1, n) case meets the native argument restrictions. A fallback Spark projection would remain visible to the operator check.

The maintained Spark 3.5 and 4.0 implementations preserve the input array's element nullability in slice. Positive starts are one-based, negative starts count from the end, zero starts and negative lengths error, and empty or out-of-range slices return empty arrays. Null arguments return a null array. The new cases exercise valid positive bounds and the non-null element field. The empty string row produces [""], not an empty array. Existing Rust and SQL tests cover empty arrays, null arguments, negative and out-of-range starts, zero lengths and invalid bounds.

The Rust addition checks the complete inner field. In the locked Arrow version, datatype equality includes the field name, nullability and metadata, so the assertion detects losing any of them. Existing builders retain their previous nullable item field. I found no actionable P1/P2 issue in either test addition or the relevant base interaction.

Validation and limits

Reviewed head 2c24d6375c13ae27306ebdbff7eacb5572976bf8 against base 8e489ea514c09f0a6cc7c04bc6d8735b6b097456. Local validation consisted of source, schema-contract and diff checks. I did not execute Rust or JVM tests. The reported debug/release Spark 4.1.3 runs and Rust test results are author evidence, not independently reproduced results. At 05:33 UTC on September 11, CI and three other workflows required approval and had no jobs. Only the labeling check had succeeded. No synthetic-merge test result is credited. Maintained Spark 3.4 and 4.1 sources were unavailable.

Performance

There is no production-path change or new performance claim requiring a benchmark. The JVM test creates one three-row Parquet fixture and runs three Spark/Comet comparisons. The nested withParquetTable(DataFrame, ...) call only registers a temporary view, so it does not write the fixture again. The Rust check uses one short list and adds negligible setup relative to the existing suite.

Design

The JVM cases exercise the planner-to-kernel contract, while the Rust test pins the kernel's field-preservation responsibility. That division fits a regression already fixed in the planner. Retaining the existing scan-boundary test separately makes the two origins of non-null element fields clear. No configuration or fallback behavior is changed.

Abstraction & complexity

The only helper extraction accepts a custom field while retaining the existing builder as a wrapper with its original defaults. It avoids duplicating offsets, values and validity construction for the new case. The Scala addition reuses the existing fixture and comparison helpers, with no new test framework or production abstraction.