This looks great, thank you! So just so I'm clear on the user-facing impact here: this should make it possible for PX to accept Vaex and Modin dataframes (which don't have .to_pandas()) and will use this new (better? faster?) pathway for other non-Pandas systems that happen to also have a .to_pandas() like... CuDF and Polars? So hard to keep track of all the flavours 🍨

This looks great, thank you! So just so I'm clear on the user-facing impact here: this should make it possible for PX to accept Vaex and Modin dataframes (which don't have .to_pandas()) and will use this new (better? faster?) pathway for other non-Pandas systems that happen to also have a .to_pandas() like... CuDF and Polars? So hard to keep track of all the flavours 🍨

You're right. The new path will be used even if the library has to_pandas method. I believe this is the preferred path, as this protocol aims to become a widely accepted standard, which should be accompanied by good support. From a performance point of view, it seems most likely that this way could be more performant (for example, in the protocol itself there is the following mention: Must be zero-copy wherever possible), at least as good as a simple to_pandas call.

@nicolaskruchten CI is green, may I know what do you think about merging it?

I'll endorse this PR but I'll defer to @alexcjohnson and @LiamConnors for timing on when to merge it, how to document it etc :)

Hi @alexcjohnson! May I have your opinion on this pull request?

I haven't looked enough into what you do with the DataFrame objects, but I just wanted to note that if df is a polars DataFrame, then:

• pd.api.interchange.from_dataframe(df) will probably make some copy (currently, the result doesn't use pyarrow)
• df.to_pandas() should typically be zero-copy

So it may be more user-friendly to first try to_pandas, and then fallback to the interchange protocl? (cc @ritchie46 in case I got this wrong)

df.to_pandas() should typically be zero-copy

df.to_pandas(use_pyarrow_extension_array=True) is zero copy yes.

Right, and that's not the default (I thought it was - sorry, I should've checked)

OK, then using the interchange protocol wouldn't be a regression for polars

Maybe a separate PR could be made so that if it is a polars dataframe, then to_pandas with use_pyarrow_extension_array=True should be passed

But that's a separate discussion, sorry everyone for the noise

+1 for proceeding with this!

This looks great, thanks @anmyachev, and thanks @nicolaskruchten for reviewing. I'd love to see a test or two that actually directly use polars and vaex dataframes in px. Compliance with a standard protocol is a great way to achieve this, and in principle if these other packages don't support this it's their problem to solve not ours, but (a) fundamentally what our users care about is that they can use their preferred dataframe manager, and (b) having tests will allow us over time to deepen this support to improve performance.

That and a changelog entry 😎

This looks great, thanks @anmyachev, and thanks @nicolaskruchten for reviewing. I'd love to see a test or two that actually directly use polars and vaex dataframes in px. Compliance with a standard protocol is a great way to achieve this, and in principle if these other packages don't support this it's their problem to solve not ours, but (a) fundamentally what our users care about is that they can use their preferred dataframe manager, and (b) having tests will allow us over time to deepen this support to improve performance.

That and a changelog entry 😎

@alexcjohnson thanks for the answer! I made the changes you mentioned. Ready for review.

Could there be a polars test for this as well? It would be great for us to know that plotly always stays working on both sides. :)

Could there be a polars test for this as well? It would be great for us to know that plotly always stays working on both sides. :)

@ritchie46 Sure, I can add. Does polars have a function like from_pandas in vaex? (for example, vaex.from_pandas(iris_pandas) call in the test)

Yes, there is pl.from_pandas. 👍

@alexcjohnson friendly ping :)

I tried this out by creating a new venv and installing pandas and polars. When I run an app I get the following traceback asking for pyarrow>=11.0.0

Traceback (most recent call last):
 File "app.py", line 19, in <module>
 fig = px.scatter(df, x="date", y="float")
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_chart_types.py", line 66, in scatter
 return make_figure(args=locals(), constructor=go.Scatter)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 2012, in make_figure
 args = build_dataframe(args, constructor)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 1317, in build_dataframe
 df_pandas = pandas.api.interchange.from_dataframe(df_not_pandas)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/pandas/core/interchange/from_dataframe.py", line 54, in from_dataframe
 return _from_dataframe(df.__dataframe__(allow_copy=allow_copy))
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/polars/dataframe/frame.py", line 1212, in __dataframe__
 raise ImportError(
ImportError: pyarrow>=11.0.0 is required for converting a Polars dataframe to a dataframe interchange object.

Hmm we definitely don't want a hard dependency on pyarrow ... can we check to see if it's present before using this branch if it's routinely-required?

pyarrow isn't strictly required for interchanging to pandas, I think it's just that polars implements the interchange protocol by going via pyarrow's implementation of it

why not swap the branches, and try to_pandas first?

@LiamConnors thanks for the feedback.

Hmm we definitely don't want a hard dependency on pyarrow ... can we check to see if it's present before using this branch if it's routinely-required?

I think right now I can rewrite the logic as follows (so that when the version of plotly is updated, the code does not stop working for users):

try:
 pandas.api.interchange.from_dataframe(df_not_pandas)
except (ImportError, NotImplemetedError):
 df_not_pandas.to_pandas()

why not swap the branches, and try to_pandas first?

@MarcoGorelli I guess we should try the more standardized and basic way first than specialized. In the case when the dataframe library has both options implemented, then the execution of the code will not reach a potentially better way.

@nicolaskruchten CI build failed for another reason. Could you manually restart it if possible?

@LiamConnors thanks for testing this out - would you try again, both without pyarrow and then with it installed? AFAICT the logic is solid at this point, so if you're happy with how it works I'll be happy to approve and merge. Thanks for the iterations @anmyachev 🙇

I guess this was already there, because I now get this:

Traceback (most recent call last):
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 1318, in build_dataframe
 df_pandas = pandas.api.interchange.from_dataframe(df_not_pandas)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/pandas/core/interchange/from_dataframe.py", line 54, in from_dataframe
 return _from_dataframe(df.__dataframe__(allow_copy=allow_copy))
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/polars/dataframe/frame.py", line 1212, in __dataframe__
 raise ImportError(
ImportError: pyarrow>=11.0.0 is required for converting a Polars dataframe to a dataframe interchange object.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "app.py", line 19, in <module>
 fig = px.scatter(df, x="date", y="float")
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_chart_types.py", line 66, in scatter
 return make_figure(args=locals(), constructor=go.Scatter)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 2022, in make_figure
 args = build_dataframe(args, constructor)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 1327, in build_dataframe
 df_pandas = df_not_pandas.to_pandas()
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/polars/dataframe/frame.py", line 2076, in to_pandas
 record_batches = self._df.to_pandas()
ModuleNotFoundError: No module named 'pyarrow'

I guess this was already there, because I now get this:

Traceback (most recent call last):
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 1318, in build_dataframe
 df_pandas = pandas.api.interchange.from_dataframe(df_not_pandas)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/pandas/core/interchange/from_dataframe.py", line 54, in from_dataframe
 return _from_dataframe(df.__dataframe__(allow_copy=allow_copy))
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/polars/dataframe/frame.py", line 1212, in __dataframe__
 raise ImportError(
ImportError: pyarrow>=11.0.0 is required for converting a Polars dataframe to a dataframe interchange object.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "app.py", line 19, in <module>
 fig = px.scatter(df, x="date", y="float")
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_chart_types.py", line 66, in scatter
 return make_figure(args=locals(), constructor=go.Scatter)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 2022, in make_figure
 args = build_dataframe(args, constructor)
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/plotly/express/_core.py", line 1327, in build_dataframe
 df_pandas = df_not_pandas.to_pandas()
 File "/Users/liamconnors/Desktop/pltest/venv/lib/python3.8/site-packages/polars/dataframe/frame.py", line 2076, in to_pandas
 record_batches = self._df.to_pandas()
ModuleNotFoundError: No module named 'pyarrow'

Do I understand correctly that this should not stop the merge of this pull request?

Thanks for the review!