Skip to content

NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)#11391

Open
maltesander wants to merge 1 commit into
apache:mainfrom
maltesander:NIFI-16069
Open

NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)#11391
maltesander wants to merge 1 commit into
apache:mainfrom
maltesander:NIFI-16069

Conversation

@maltesander

Copy link
Copy Markdown

Summary

NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)

PutIcebergRecord fails to write FlowFiles whose schema contains complex/nested types like Iceberg list, map, or struct columns. RecordConverter only translates top-level scalar values (java.sql timestamp/date/time -> java.time) and passes complex values through unchanged.
As a result, values reach Iceberg's Parquet writer in NiFi's native representation, which is incompatible with what Iceberg expects:

  • Nested records arrive as org.apache.nifi.serialization.record.MapRecord but Iceberg requires org.apache.iceberg.StructLike.
  • Array fields arrive as Object[] but Iceberg's writer requires a java.util.Collection.
  • Maps and elements/values nested inside these types are likewise not converted (e.g. a date inside an array or map value).

Because conversion is gated on scalar field types only, records consisting solely of complex fields skip conversion entirely.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
  • Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using ./mvnw clean install -P contrib-check
    • JDK 21
    • JDK 25

Licensing

  • New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

…ing complex types (arrays, maps, nested records)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant