As for the ASF Generative Tooling Guidance:

Anthropic's Commercial Terms still state:

Anthropic agrees that Customer (a) retains all rights to its Inputs, and (b) owns its Outputs.

So, I can confirm that:

• The terms and conditions of the generative AI tool do not place any restrictions on use of the output that would be inconsistent with the Open Source Definition.
• The output is not copyrightable subject matter (and would not be even if produced by a human).
• No third party materials are included in the output.

@kylebarron @supermacro @pmaciolek @GeorgeLeePatterson Could you review this?
(You're in related issue/PR: #70 #299)

Could you compare with #299 and explain what's different?

I started this PR by repeating the original #299 verbatim, and then building on it for full parity of every feature available for the List type, while considering feedback to the original PR.

The scope of #299 - a LargeList type stub, not even quite an MVP: you could manually build a LargeList column and read individual values out of one already in memory. Anything beyond that - saving it to an Arrow file, loading it in IPC binary or JSON form, comparing two LargeList schemas, modifying values, or constructing one through the normal Builder API - was either silently broken or missing entirely.

The scope of this PR - LargeList as a fully-featured column type, on par with List. You can build one, read and write values, compare schemas, iterate, slice, serialize it to the Arrow IPC binary or JSON form, and round-trip it back - all with safe handling of 64-bit offsets and clear errors (rather than silent corruption) if a value would exceed JavaScript's safe integer range. Existing List, LargeUtf8, and LargeBinary types also get a latent slicing-on-write bug fixed as a side effect.

Added on top of the original PR

• IPC write path — VectorAssembler.visitLargeList + generalized assembleListVector with bigIntToNumber coercion
• IPC read path — VectorLoader.visitLargeList
• JSON form — JSONVectorAssembler.visitLargeList (OFFSET via bigNumsToStrings)
• Type comparison — TypeComparator.visitLargeList (widened compareList to List | LargeList)
• Mutation — SetVisitor.visitLargeList (merged into setList via bigIntToNumber)
• Iterator interface declaration — completes IteratorVisitor typing
• LargeListBuilder — new src/builder/largelist.ts with BigInt() accumulation and bigIntToNumber for safe child-cursor narrowing
• Builder plumbing — widened VariableWidthBuilder bound; GetBuilderCtor.visitLargeList; LargeList / LargeListBuilder mapped into TypeToDataType, TypeToBuilder, DataTypeToBuilder
• Public API — exported LargeListBuilder from Arrow.ts / Arrow.dom.ts
• Latent bug fix — rebaseValueOffsets now bigint-safe (also fixes silent breakage on LargeUtf8 / LargeBinary sliced writes)
• "Get" path consolidation — getList / getLargeList merged with bigIntToNumber boundary coercion
• Test generator robustness — shared generateListLike; createVariableWidthOffsets64 truncates min / max on entry so fractional stride doesn't RangeError in BigInt()
• Test coverage — LargeListBuilder entry in builder-tests.ts; visitLargeList in BasicVisitor / FeatureVisitor and both describe matrices in visitor-tests.ts

It looks like a lot of this PR's code overlaps with #325, a PR I created back in November. Happy to see it land finally, additional work was needed.

Could you add a reference to #325 in the PR description / commit, and call out the parts that genuinely differ?

I noticed test coverage is missing. The tests missing are:

1. slicing
2. IPC/JSON
3. overflow tests

Also, the LargeList class has no public-facing JSDocs on overflow semantics, this would be helpful to provide.

and call out the parts that genuinely differ

Sure I'll add a reference, but what is the purpose of inspecting the #325 to compile the list of differences to it? In general, its scope was beyond LargeList hence why I did not considered as a base for the new PR. Do you imply it may have some functionality not accounted for in this PR, and is a nice-to-have?

Happy to see it land finally

Not yet, fingers crossed :)

@GeorgeLeePatterson , thanks for the pointer!
Unless you mean something else, slicing is covered by validateVector, run from test/unit/generated-data-tests.ts.
Addressed other points.

Also, I caught I missed a case for typeFromJSON in src/ipc/metadata/json.ts, and one in src/visitor/typector.ts. The only relevant thing in #325 that was missing here was the change in typector.ts. Otherwise this PR looks like a superset of #325 in terms of LargeList implementation; this PR does not include extra LargeListView tests or other *ListView additions.

@trxcllnt I appreciate you raising this as I take the spec compliance point seriously, but I want to lay out why I think the full 64-bit indexing is a much larger conversation that goes way beyond the scope of this PR, and is not a prerequisite for landing it.

TLDR: The Number precision is not the bottleneck for what can be represented using the Large* types, the current project architecture is.

After analyzing the problem it seems to me you underestimate the amount of combined design, discussion and implementation effort needed to implement full 64-bit indexing.
The constraint isn't the offset arithmetic - BigInt64Array already stores offsets at full 64-bit precision in this PR, and the helpers around bigint are indeed straightforward.

The constraint is on the child elements buffer - the contiguous buffer holding the actual list elements that the offsets point into. All rows of a list-like column in one Arrow batch share a single such buffer, and that buffer is a JavaScript ArrayBuffer, which every major engine caps at roughly 2^32 bytes. Since offset values are positions into this buffer, the maximum offset value is bounded by the buffer's element count (~2^32 / sizeof(element)) - many orders of magnitude below 2^53, let alone 2^63. So the int64 offset width is headroom the JS runtime physically can't fill in a single contiguous child buffer.

To fully support LargeList rows beyond that cap, the child can't be a single Data's elements buffer - it has to be chunked across multiple Data objects behind a Vector-style rope. That's a change to a load-bearing invariant (Data.children[0] is a single contiguous Data) that the entire visitor framework relies on, and it would also need to flow into IPC read/write semantics, since the wire writer today assumes one contiguous child per batch.

The same reasoning applies to LargeUtf8 and LargeBinary: their backing values buffer is a Uint8Array, which has the same per-buffer ceiling. So the 53-bit narrowing isn't a quirk introduced by this PR - it's the existing project-wide policy for every Large* type, and matching it is what I meant by "feature parity with List".

At the same time the same buffer-size constraint also bounds plain List. The spec allows List up to 2^31 child elements, but JS's per-ArrayBuffer cap means a List<Float64> tops out around 2^29 elements - a quarter of the spec ceiling. So the property "JS implementation matches the spec's offset range" isn't currently true for List either, and bringing it up to that bar means the same chunked-children redesign, applied across List/LargeList/LargeUtf8/LargeBinary together. That's the design discussion I'd want to open as a separate issue rather than fold it in a wiring PR.

Unfortunately, designing and shepherding through a chunked-children rework - across all four types, across the visitor framework, the Builder layer, and the IPC reader/writer - is beyond what I can take on, neither as a standalone PR, and especially not as a part of this PR. I'm comfortable leaving this PR open and rebasing it once the community settles on a design for compliant >2^32-byte child storage that applies uniformly to the Large* family.

Otherwise, I maintain that the current PR stands on its own:

• it's fully featured;
• it introduces no new constraints or regressions and follows the existing implementation and API surface conventions;
• it is very useful to the community to close the feature gap despite not being fully compliant on the maximum element count - today the library throws when it encounters a LargeList.

P.S. conforming producers should normalize the offsets in a batch, so in practice as long as the batch spans no more elements than can fit in the child buffer the current 2^53 arithmetic is unused headroom.

v8 supports ArrayBuffers larger than 4GiB:

# Allocate a 64GiB ArrayBuffer
$ { node -p 'var a = new ArrayBuffer(2**36); setTimeout(() => {}, 2000); a.byteLength / 1024' & }; pid=$!; sleep 1; ps -p $pid -o vsz,rss,cmd; wait $pid
[1] 1431480
 VSZ RSS CMD
68109056 41444 node -p var a = new ArrayBuffer(2**36); setTimeout(() => {}, 2000); a.byteLength / 1024
67108864
[1]+ Done node -p 'var a = new ArrayBuffer(2**36); setTimeout(() => {}, 2000); a.byteLength / 1024'

And it's not unusual for other language implementations to create RecordBatches larger than 4GiB (e.g. by using table.combine_chunks() from Python).

That said, it's unlikely anyone needs to address 8192 TiB of memory. I agree with your point that 53-bit ints are fine for indices, thanks for walking me through that 😅.

JS's per-ArrayBuffer cap means a List tops out around 2^29 elements - a quarter of the spec ceiling

Could you explain this more? List's offsets are Int32Array, which means its child should be able to support up to 2^31 individual elements. How did you arrive at 2^29 for the max child vector length?

edit:

I think I understand the confusion. You're saying if ArrayBuffer size was limited to 4GiB, then you could only represent 2^(32-4) 32-bit integers. Even if that were true, that doesn't mean the List child length is constrained, it would mean the number of lists that the ListVector represented would be constrained.

The child could have 2GiB of uint8_t values, and the ListVector offsets buffer could contain (2^28)+1) offsets, but the offset values themselves are still in the range 0..2^31. This is true in all other Arrow implementations as well.

Fair point — my List<Float64> analogy conflated two different things, and your clarification helps. Compliance is about correctly interpreting offsets the wire format presents, not about the implementation realizing enough child storage to exercise the full offset range. Under that definition, List is conforming today (int32 offset values all fit in Number), and LargeList/LargeUtf8/LargeBinary aren't, because bigIntToNumber throws on offset values > 2^53.

One thing I wasn't sure about in your reply: when you said "that doesn't mean the List child length is constrained, it would mean the number of lists that the ListVector represented would be constrained" I couldn't tell whether you were pointing at a mechanism in the codebase that sidesteps the single-ArrayBuffer limit on child storage — I didn't find one, so I focused on the wire-format parsing compliance angle, which is what a new commit I pushed addresses.

The commit: VectorLoader now rebases offsets to 0 on load for LargeList, LargeUtf8, and LargeBinary. After rebasing, in-memory offsets are always bounded by the child buffer's element count (which fits in Number for anything the runtime can allocate), so downstream narrowing succeeds for any spec-conforming wire input — including sliced views with absolute, non-rebased offsets. Inputs whose offsets imply a child buffer larger than JS's ArrayBuffer cap now fail honestly at child-buffer allocation in readData, rather than later at offset narrowing — a cleaner failure mode that's a property of the JS runtime, not of the implementation.

Please let me know if this commit addresses your primary concern.

The remaining ceiling — JS's per-ArrayBuffer cap of ~2^32 bytes on a single contiguous child buffer — is allocation capacity, not interpretation, and it applies uniformly across List/LargeList/LargeUtf8/LargeBinary. Lifting it would mean a chunked-children redesign (child as Vector rather than single Data<U>), which is a substantial design change. As I previously expressed, I think that's a separate, more ambitious effort and deserves its own issue.

I couldn't tell whether you were pointing at a mechanism in the codebase that sidesteps the single-ArrayBuffer limit on child storage
...
The remaining ceiling — JS's per-ArrayBuffer cap of ~2^32 bytes on a single contiguous child buffer — is allocation capacity, not interpretation, and it applies uniformly across List/LargeList/LargeUtf8/LargeBinary. Lifting it would mean a chunked-children redesign (child as Vector rather than single Data), which is a substantial design change. As I previously expressed, I think that's a separate, more ambitious effort and deserves its own issue.

As demonstrated by the example I pasted above, there is no uniform 4GiB cap on ArrayBuffer size in JS. The only limit to List child size is that the value offsets type is Int32, thus the child can only contain 2^31 elements. This means the ListVector could represent a column of entirely single-element lists, but there could only be 2^31 of them due to the offset type being Int32, not any limit in ArrayBuffer allocation size.

I focused on the wire-format parsing compliance angle, which is what a new commit I pushed addresses

I don't think this commit is right. The reader should be zero-copy, and even if it was the correct thing to do, rebasing offsets when reading is not zero-copy.

You're right, I've pushed the updated commit. This version keeps the reader zero-copy for everything under 2^53 and only falls back to a copy for when the fallback rebase is necessary:

• Final offset ≤ 2^53 (common): offsets returned as a zero-copy BigInt64Array view over the wire bytes — no allocation, no copy. Narrowing to number stays lazy (at element access) and is always lossless.
• Final offset > 2^53, span ≤ 2^53 (slice with absolute offsets): rebased to 0 in bigint precision into a fresh buffer — the only path that copies — so the values stay narrowable.
• Final offset > 2^53, span > 2^53: unreachable in practice — needs a child buffer > 2^53 elements present in the message, far past any engine's allocation limit.
• bigIntToNumber 2^53 guard: after the rebase, never tripped by well-formed wire data; only reachable via hand-built in-memory Data (makeData, not rebased by design) — a loud failure for corrupt input.
• Offset < 2^53 but past the child buffer: clamps silently in subarray/slice (pre-existing - built in methods behavior, unchanged).

@trxcllnt I'd appreciate it if you helped me carry this PR through review!

Could you check CI failures?

I rseolved the issue, I needed to avoid running the new test under UMD build. The test is still executed in other builds

Thank you for helping to land this PR!