Skip to content

Navigation Menu

Sign in
Sign up

How to get columns and change their names in a pyrrow compute expression. #48927

davlee1972 started this conversation in General
Discussion options

Here is a sample pyarrow compute expression used for filtering data:

>>> import pyarrow.compute as pc
>>>
>>> exp = (pc.field("total") > 100) & (pc.field("items") < 10)
>>> exp
<pyarrow.compute.Expression ((total > 100) and (items < 10))>

My odd use case is that I need to modify the expression to:

>>> import pyarrow.compute as pc
>>>
>>> exp = (pc.field("TOTAL") > 100) & (pc.field("ITEMS") < 10)
>>> exp
<pyarrow.compute.Expression ((TOTAL > 100) and (ITEMS < 10))>

Some of my files have lower case column names and some files use upper case column names. I know which files are upper case vs lower case so I need to take a pyarrow compute expression using lower case column names including nested columns in nested conditions and rewrite the expression with all upper case column names.

Is there any way to create something like an abstract syntax tree to get all possible column names and reconstruct the expression with all upper case column names?

You must be logged in to vote

Replies: 1 comment

Comment options

you probably want to use something like acero (substrait) to do what you want. I can try and figure out how to do it, I don't know it off the top of my head.

docs here (I think): https://arrow.apache.org/docs/python/api/acero.html

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants

AltStyle によって変換されたページ (->オリジナル) /