Skip to content

Navigation Menu

Sign in
Sign up

GH-36010: [GLib][Ruby][Parquet] Add buffered reader properties - #51277

Open
emecii wants to merge 2 commits into
apache:main from
emecii:gh-36010-buffered-reader
Open

GH-36010: [GLib][Ruby][Parquet] Add buffered reader properties #51277
emecii wants to merge 2 commits into
apache:main from
emecii:gh-36010-buffered-reader

Conversation

@emecii

@emecii emecii commented Sep 10, 2026
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

GH-36010 requests Ruby bindings for parquet::ReaderProperties, following the buffered-read discussion in GH-36001. GLib currently opens readers with defaults, so Ruby cannot select buffered streams or their buffer size.

What changes are included in this PR?

  • Add GParquetReaderProperties with buffered-stream enable/disable/state and buffer-size get/set.
  • Add source/path constructors accepting nullable reader properties. Existing constructor declarations and implementations remain unchanged. GI exposes the new overload as Parquet::ArrowFileReader.new(source_or_path, properties) without a Ruby adapter.
  • Copy native properties into reader construction and retain the source GObject, whose disposal would otherwise close the shared native stream.
  • When buffering is enabled, disable Arrow's whole-column read-ahead, which otherwise bypasses buffered streams. Default constructors and default properties retain their existing read-ahead behavior.

Are these changes tested?

Fresh Debug C++/Parquet/GLib/introspection and native Ruby builds on macOS arm64, Ruby 4.0.6:

  • Before the change: 9 focused GLib errors and 2 Ruby errors for the missing type/constructors.
  • Focused GLib reader/properties tests: 14 tests, 20 assertions, passing. These cover defaults, toggles, size, nullable properties, source/path reads, property mutation/destruction, source release and constructor errors.
  • Full red-parquet: 18 tests, 24 assertions, passing; full red-arrow: 2,667 tests, 2,675 assertions, passing with two ORC omissions.
  • Full GLib in the local core+Parquet configuration: 2,247 tests, 2,384 assertions, passing with 173 optional/platform omissions. The local runner excludes incompatible installed Arrow 25 optional typelibs.
  • A compiled C client exercises both old and both new constructor signatures. An instrumented native source used through the new Ruby overload verifies 10 real row-group/table reads, including unchanged defaults and property snapshots after mutation/destruction. For 100,000 uncompressed int64 values, default reading makes one 801,421-byte data read; 4 KiB buffering makes 50 reads (largest 16,413 bytes), and 32 KiB buffering makes 27 (largest 32,826 bytes). All read the same bytes and return the same table. This is an I/O behavior check, not an RSS measurement or a hard read-size cap.
  • Clang-format 18.1.8, repository RuboCop 1.71.0, Ruby syntax, diff checks and the repository RAT audit pass.

Are there any user-facing changes?

Ruby/GLib callers can opt into buffered Parquet reading and configure the buffer size. The API documentation explains the read-ahead interaction and that decoded data and individual page reads can exceed that size.

AI assistance

OpenAI Codex generated the implementation, regression tests and local validation harnesses, and ran the checks described above.

Expose buffered stream controls and additive properties-aware reader constructors. Copy native properties and retain the input stream; disable whole-column read-ahead when buffered streams are selected.
Generated-by: OpenAI Codex
Free the reader after the .open block so Windows can remove the
memory-mapped temporary file without waiting for garbage collection.
Generated-by: OpenAI Codex

emecii commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the MinGW temporary-file cleanup failure in b39e1a5. The new path test passed its assertions but retained the memory-mapped reader after the .open block, causing Windows to reject Tempfile deletion. The test now explicitly releases the reader in ensure.

Validated with a native mapping check that fails before the change and passes afterward, the full local Red Parquet suite (18 tests, 24 assertions), and the GLib suite (2,247 tests, zero failures/errors; 173 unavailable-feature omissions). Ruby syntax, RuboCop, and RAT checks pass. The hosted Windows rerun is pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kou kou Awaiting requested review from kou kou is a code owner

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /