Parquet scan silently drops one of two columns with the same name

[sovsparrow]

Describe the bug

When a Parquet file contains two columns with the same name and type,
DataFusion reads it successfully but only returns the first column. The
second column is missing from the result.

I reproduced this with DataFusion 54.0.0 and PyArrow 25.0.0.

To Reproduce

This example joins two Arrow tables that both have a column named value.
PyArrow keeps both columns in the joined table and writes them to Parquet.

from pathlib import Path
from tempfile import TemporaryDirectory

import pyarrow as pa
import pyarrow.parquet as pq
from datafusion import SessionContext


left = pa.table({
    "id": [1, 2, 3],
    "value": [10, 20, 30],
})
right = pa.table({
    "id": [1, 2, 3],
    "value": [100, 200, 300],
})

joined = left.join(right, keys="id")

with TemporaryDirectory() as directory:
    path = Path(directory) / "joined.parquet"
    pq.write_table(joined, path)

    context = SessionContext()
    result = pa.Table.from_batches(
        context.read_parquet(str(path)).collect()
    )

print("Written:")
print(joined.column_names)
print([column.to_pylist() for column in joined.columns])

print("DataFusion:")
print(result.column_names)
print([column.to_pylist() for column in result.columns])

Output:

Written:
['id', 'value', 'value']
[[1, 2, 3], [10, 20, 30], [100, 200, 300]]

DataFusion:
['id', 'value']
[[1, 2, 3], [10, 20, 30]]

Expected behavior

DataFusion should either preserve all three physical columns, using a
deterministic way to distinguish the duplicate names, or reject the file
with a clear duplicate-column error.

It shouldn't read successfully after dropping one of the columns.

Additional context

DuckDB reads the same file and keeps all three columns, renaming the second
value column to value_1.

PyArrow's low-level ParquetFile.read() also preserves all three physical
columns. Its higher-level pq.read_table() and dataset reader reject the
ambiguous name, and Polars raises a duplicate-column error.

DataFusion is the only reader in this comparison that returns success while
omitting one of the physical columns.

PyArrow's Table.join allows colliding names when left_suffix and
right_suffix are not supplied, and pq.write_table writes the result.

DataFusion's current Parquet schema inference passes the fetched schemas
through Schema::try_merge.
That appears to merge the two same-name, same-type fields before the file is
scanned.

A related report for duplicate CSV headers is present:
#12852. That issue asks
for duplicate columns to be renamed or rejected clearly. This is the Parquet version.

Apache Arrow also tracks duplicate-column handling in
apache/arrow#24407.

Environment:

• DataFusion 54.0.0
• PyArrow 25.0.0
• Python 3.12.13
• macOS arm64

I found this while comparing Parquet readers with
Parquity 0.2.0.