Looks like AI

@jbrockmendel

I did use AI to help draft this. I tried setting up a pandas development environment (both via pip and the Docker image) to create a reproducible test case, but running pytest from the CLI kept failing with a message about pandas._libs.

There seems to be a significant issue with the pyarrow implementation. Specifically, pyarrow does not enforce dtypes during load - it applies them afterward. As a result, integer-to-string conversions lose leading zeros. I wanted to at least contribute a working test that highlights this problem.

@jbrockmendel test is passing now. I have it marked as xfail for pyarrow only, but you can clearly see the issue. Once the issue is remedied we can remove the try/except block.

We discourage AI-generated PRs since they take more time and effort to review than they do to write. I'll take a look since you took the time to get the CI passing, but in the future please avoid it.

big picture adding a test isn't wrong, but not a good use of time. if you'd like to actually fix the issue, i think there are some good comments in the original thread

big picture adding a test isn't wrong, but not a good use of time. if you'd like to actually fix the issue, i think there are some good comments in the original thread

I agree, but the fix doesn't appear to be very straightforward. Happy to work on it if there is some guidance on where to find the relevant pieces. It requires proper mapping of pandas dtypes to pyarrow types, and also handling other logic that pandas supports but pyarrow doesn't (e.g., col index-based dtypes, global dtypes, etc.).

@jbrockmendel I dug into the pandas and PyArrow APIs and landed on a more general fix for the issue. I wasn't sure how to run pytest locally against the dev branch - my attempts with the Docker container didn't work - so I relied on pandas' test suite.

This patch improves dtype handling during the pyarrow write path by converting supported column-specific dtypes into pyarrow types and passing them via convert_options["column_types"]. However, a few limitations remain:

Known Issues / Remaining Work

1. Global dtype support is not implemented
   The current logic handles only column-specific dtype dictionaries. Global dtypes (e.g., dtype=str) are ignored, which could lead to inconsistent behavior across engines, especially with things like leading-zero string preservation. Supporting this would require column name/index context, which doesn't seem to be readily available here. I couldn't find a safe and clean way to retrieve it without broader architectural changes.

EDIT: After review, I think this feature would be best handled with a change to the PyArrow API, which we could adapt here quite easily. I've posted that issue here: apache/arrow#47502

2. Unsupported dtypes are silently skipped
   If a dtype (e.g., "category") is not mappable to a PyArrow type, we currently drop it from column_types. This fallback behavior avoids breaking the pipeline, but it may lead to silent mismatches when PyArrow falls back to its default inference. We may want to revisit this to either emit warnings or fail explicitly for better visibility.

3. Possible redundancy in _finalize_dtype()
   Now that dtypes are mapped earlier during the pyarrow conversion, we may no longer need the final call to self._finalize_dtype(). However, it might still be necessary for preserving native pandas types (e.g., CategoricalDtype) in some cases that do not have native pyarrow support.