Skip to content

fix: convert BigInt to strings in Arrow IPC row.toJSON()#270

Open
jamesbroadhead wants to merge 1 commit intodatabricks:mainfrom
jamesbroadhead:fix/bigint-arrow-serialization
Open

fix: convert BigInt to strings in Arrow IPC row.toJSON()#270
jamesbroadhead wants to merge 1 commit intodatabricks:mainfrom
jamesbroadhead:fix/bigint-arrow-serialization

Conversation

@jamesbroadhead
Copy link
Copy Markdown
Contributor

Summary

  • row.toJSON() from apache-arrow preserves BigInt for Int64 columns. When this data is later serialized with JSON.stringify() for SSE streaming, it throws TypeError: Do not know how to serialize a BigInt.
  • Convert BigInt values to strings in _transformArrowAttachment during row transformation, before the data reaches JSON serialization.
  • This ensures Arrow IPC deserialization works correctly for serverless warehouses that return inline results as Arrow IPC in result.attachment.

Test plan

  • Verify that queries returning Int64 columns via Arrow IPC attachment no longer throw TypeError: Do not know how to serialize a BigInt during SSE streaming
  • Verify that non-BigInt columns are unaffected by the change
  • Verify that the string representation of BigInt values is correct and usable downstream

This pull request was AI-assisted by Isaac.

Co-authored-by: Isaac
Signed-off-by: James Broadhead <jamesbroadhead@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant