Skip to content

fix!: Fix type equality between arro3 and pyarrow#483

Merged
kylebarron merged 5 commits intomainfrom
revert-462-kyle/use-simpler-pyo3-eq
Feb 23, 2026
Merged

fix!: Fix type equality between arro3 and pyarrow#483
kylebarron merged 5 commits intomainfrom
revert-462-kyle/use-simpler-pyo3-eq

Conversation

@kylebarron
Copy link
Owner

@kylebarron kylebarron commented Feb 23, 2026

#462 looked like it should've been a simple refactor for nicer code, but it actually created #482, because the derived equality checking from pyo3 doesn't use the Arrow PyCapsule extraction. So it would only check for equality with other pyo3 objects, not any Arrow-like object.

This PR reverts #462 and adds tests that we can assert equality for all types (except for the lazy readers) with PyArrow.

The other part of equality that was biting me was #451, since I incorrectly was taking in references, not owned types.

Unfortunately, part of this change is that we now export DataType with nullable: true instead of nullable: false, so we probably need to put this in a new minor release, not a bugfix release.

Change list

  • Revert refactor(pyo3-arrow): Use simpler derived eq for python __eq__ duner #462
  • Add tests for equality of all types. Python's __eq__ calls the left side first. So ideally anyone who wants to compare arro3 types to other types should put the arro3 types first. This works reflexively with pyarrow for most types, because pyarrow returns NotImplemented, which then tells Python to check the other direction for equality. This doesn't work for RecordBatch because apparently PyArrow has some custom __eq__ there.
  • We also now export a bare DataType through __arrow_c_schema__ with nullable: true, which was required to match equality semantics with pyarrow.

Reverts #462, Closes #482

@kylebarron kylebarron changed the title Revert "refactor(pyo3-arrow): Use simpler derived eq for python __eq__ duner" fix!: Fix type equality between arro3 and pyarrow Feb 23, 2026
@kylebarron kylebarron merged commit 1136130 into main Feb 23, 2026
10 checks passed
@kylebarron kylebarron deleted the revert-462-kyle/use-simpler-pyo3-eq branch February 23, 2026 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The 0.7.0 release breaks DataType equivalence with pyarrow

1 participant