NIFI-16067 - PutIcebergRecord maps fields by ordinal position instead of column name#11387
NIFI-16067 - PutIcebergRecord maps fields by ordinal position instead of column name#11387maltesander wants to merge 4 commits into
Conversation
exceptionfactory
left a comment
exceptionfactory
left a comment
There was a problem hiding this comment.
Thanks for flagging and addressing this issue @maltesander. The general approach looks good, I will take a closer look at the changes and follow up soon.
exceptionfactory
left a comment
exceptionfactory
left a comment
There was a problem hiding this comment.
On further review, the functional change to DelegatedRecord looks like good approach. I noted some room to improve the test class, as the various field names and values could be declared statically and reused. If you can make some adjustments there, this should be close to completion.
| final Types.StructType structType = Types.StructType.of( | ||
| Types.NestedField.required(1, "id", Types.IntegerType.get()), | ||
| Types.NestedField.optional(2, "amount", Types.DecimalType.of(10, 2)), | ||
| Types.NestedField.optional(3, "label", Types.StringType.get()) | ||
| ); |
There was a problem hiding this comment.
There are some existing static field names and types in the test class that can be used. Additional field names and types should be declared statically and reused.
There was a problem hiding this comment.
Thank you for the feedback @exceptionfactory . I moved the hardcoded constants to static variables and tried reusing existing ones: fedc6d4
2 participants
Summary
NIFI-16067 - PutIcebergRecord maps fields by ordinal position instead of column name
PutIcebergRecord (added in NIFI-15062) doesn't match columns by name. It writes each record's values into the Iceberg table columns by position, and the position it uses is the one from the incoming record schema, not the table schema. So the mapping is wrong whenever the input field order doesn't line up with the table's column order.
If the record has fewer fields than the table, the missing columns aren't set to NULL either - the values just shift left into whatever table column happens to sit at that position. Sometimes that lands on an incompatible type and you get a ClassCastException, which is the good outcome because at least it fails loudly. When the types happen to line up, the row is written with values in the wrong columns and nothing complains.
This gets worse with schema evolution: as soon as you add a column, the producer and the table no longer agree on the column set, and existing flows start silently writing data into the wrong place.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000VerifiedstatusPull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation