Skip to content

Navigation Menu

Sign in
Sign up

Flight DoGet multiple batch DictionaryArray deltas #47718

Discussion options

I have a FlightServer written in c++ using subclasses of flight::RecordBatchStream and arrow::RecordBatchReader. For efficiency, I'm trying to convert some StringArray fields to DictionaryArrays. It all seems to work if all dictionary values are known with the first RecordBatch streamed back, but if new values occur in subsequent RecordBatches the client doesn't see them. As far as I can tell the client only ever sees the dictionary associated with the first RecordBatch streamed back. I checked both c++ and python clients.

It seems like the IPC protocol itself supports this, but flight doesn't seem to support it automatically, and I don't see any obvious way to enable it.

Does anybody know if this is even possible, and if yes how I can accomplish it? I already tried using the same Dictionary32Builder between batches, but it didn't help.

You must be logged in to vote

Which Arrow version are you using? Dictionary replacement is available: #45056

Replies: 1 comment 3 replies

Comment options

Which Arrow version are you using? Dictionary replacement is available: #45056

You must be logged in to vote
3 replies
Comment options

It looks like we're still on 21.0.0 and that was added to 22.0.0. I'll have to discuss upgrading with the team. Thank you for pointing that out.

Comment options

raulcd Oct 7, 2025
Collaborator

As a minor note the Arrow 22.0.0 release is currently on-going. The feature freeze has been done and we are still solving some issues. The PRs pointed there will be available once we release.

Comment options

lidavidm Oct 7, 2025
Collaborator

Ah my bad, I had thought 22 was out for some reason 😅 please bear with us!

Answer selected by jskowronski-factset
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet

AltStyle によって変換されたページ (->オリジナル) /