GH-3690: Update third-party NOTICE and LICENSE

[Fokko]

Rationale for this change

Updates the LICENSE and NOTICE with the 3rd party packages that are bundled in the binaries. luben/zstd-jni with the BSD-2 License is important.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[wgtmac]

Why do we need to copy the license of a bundled dependency?

[Fokko]

This is an interpretation of the legalese around what needs to be included in the LICENSE/NOTICE. Please note that a lot of projects do this differently.

My interpretation of it. For the ASF it is required for all the 3rd party source that is bundled with the source itself (so, borrowing a class from Avro etc) should be noted in the LICENSE. Since the ASF mostly cares around the source distribution, it is a bit of a gray area on how to handle the binary distribution where we include 3rd party binary because of bundling (in the bundle or runtime jar) or shading (mostly FastUtil). Some licenses explicitly call out that the license should be included when when binary distributed (eg, the BSD-2 Clause: Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

So, including these makes sure that we adhere to all the requirements. The big downside is that the LICENSE and NOTICE grow pretty big, which should be avoided according to the {LICENSE,NOTICE} assembling how-to. Since Parquet only has two packages (parquet-hadoop-bundle and parquet-cli-runtime) that ship a reasonable amound of 3rd party packages, I don't think it is an issue of having a somewhat bigger LICENSE and NOTICE.

[kevinjqliu]

I don't think it is an issue of having a somewhat bigger LICENSE and NOTICE.

Big is relative 😆 I've seen wayyy longer LICENSE and NOTICE files

BTW these are the specific 3rd party dependencies bundled:
#3690 (comment)

[kevinjqliu]

taking a look now, everything looks good. im double checking CLI since it has the largest set of 3rd party dependencies (22!)

[kevinjqliu]

Thanks for working on this Fokko! LICENSE and NOTICE changes are extremely hard, but most important to get right 😄

I double checked all 7 jars (as mentioned in #3690), everything looks great except for some possible minor issues with parquet-cli

I highlighted the potential issues in the respective files' comments.

I was pairing with codex so I also tried to fix them: https://github.com/apache/parquet-java/compare/master...kevinjqliu:parquet-java:kevinjqliu/codex-pr-3694-notice?expand=1

• LICENSE 986dae0
• NOTICE bb22ede

Please take a look and let me know what you think

[kevinjqliu]

I don't think it is an issue of having a somewhat bigger LICENSE and NOTICE.

Big is relative 😆 I've seen wayyy longer LICENSE and NOTICE files

BTW these are the specific 3rd party dependencies bundled:
#3690 (comment)

[kevinjqliu]

nit: Maven’s default generated NOTICE already uses Apache Parquet <module’s name>. Since these NOTICE file doesnt contain any other entries, we dont necessary need to add them.
This applies to both

• parquet-column/src/main/resources/META-INF/NOTICE
• parquet-hadoop/src/main/resources/META-INF/NOTICE

Furthermore the opposite problem occurs in parquet-protobuf/src/main/resources/META-INF/NOTICE. We should not use "Apache Parquet Java" but use the module name instead.

[kevinjqliu]

Paired with codex, these 13 (out of 22) dependencies should propagate NOTICE.
we captured all except 2 (highlighted)

Dependency	Required NOTICE content
org.apache.thrift:libthrift:0.23.0	Apache Thrift copyright notice
org.apache.avro:avro:1.11.5	Apache Avro copyright notice
com.fasterxml.jackson.core:jackson-annotations:2.22	Shared Jackson notice
com.fasterxml.jackson.core:jackson-core:2.22.1	Shared Jackson notice, including FastDoubleParser and Schubfach
com.fasterxml.jackson.core:jackson-databind:2.22.1	Shared Jackson notice
com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1	Covered by the shared Jackson notice; no separate text
commons-codec:commons-codec:1.11	Commons Codec copyright and Beider–Morse attribution; exclude the test-only DoubleMetaphoneTest notice
org.apache.commons:commons-compress:1.26.2	Commons Compress copyright notice
org.apache.commons:commons-lang3:3.20.0	Commons Lang copyright notice
commons-pool:commons-pool:1.6	Commons Pool copyright notice
org.apache.commons:commons-text:1.15.0	Commons Text copyright notice
io.airlift:aircompressor:2.0.3	Dain Sundstrom and Google Snappy copyrights
org.xerial.snappy:snappy-java:1.1.10.8	Google Snappy, Apache PureJavaCrc32C, static libstdc++/GCC Runtime Library Exception, and contributor notices

[kevinjqliu]

verified that all 22 dependencies are present (according to #3690 (comment))

com.beust:jcommander:1.82
com.fasterxml.jackson.core:jackson-annotations:2.22
com.fasterxml.jackson.core:jackson-core:2.22.1
com.fasterxml.jackson.core:jackson-databind:2.22.1
com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1
com.github.luben:zstd-jni:1.5.7-11
commons-codec:commons-codec:1.11
commons-pool:commons-pool:1.6
io.airlift:aircompressor:2.0.3
it.unimi.dsi:fastutil:8.5.18
javax.annotation:javax.annotation-api:1.3.2
net.openhft:zero-allocation-hashing:0.27ea1
net.sf.opencsv:opencsv:2.3
org.apache.avro:avro:1.11.5
org.apache.commons:commons-compress:1.26.2
org.apache.commons:commons-lang3:3.20.0
org.apache.commons:commons-text:1.15.0
org.apache.thrift:libthrift:0.23.0
org.locationtech.jts:jts-core:1.20.0
org.slf4j:slf4j-api:1.7.33
org.tukaani:xz:1.12
org.xerial.snappy:snappy-java:1.1.10.8

the 7 highlighted ones needs attention

Dependency	Required LICENSE change	Reason
javax.annotation:javax.annotation-api:1.3.2	Select and include CDDL-1.0; provide the source location and identify it as ASF Category B, binary-only.	It is dual-licensed under CDDL-1.0 or GPLv2 with Classpath Exception. CDDL-1.0 is the preferable option for the shaded binary.
org.locationtech.jts:jts-core:1.20.0	Select and include EDL-1.0.	It is dual-licensed under EPL-2.0 or EDL-1.0. EDL-1.0 is the ASF Category-A option.
org.xerial.snappy:snappy-java:1.1.10.8	Include Google Snappy BSD-3, Bitshuffle MIT, and Intel PureJavaCrc32C BSD terms and copyrights.	The Apache-2.0 dependency bundles native and copied code under these additional licenses.
io.airlift:aircompressor:2.0.3	Include Google Snappy BSD-3 terms and copyright.	Its upstream notice.md identifies bundled Google Snappy code under BSD-3.
commons-codec:commons-codec:1.11	Include the Intel PureJavaCrc32C BSD terms and copyright.	Its PureJavaCrc32C implementation contains Intel-owned BSD-licensed portions. One shared Intel section can also cover snappy-java.
org.tukaani:xz:1.12	Replace the public-domain statement with the 0BSD terms and author attribution.	Version 1.12 source files explicitly use SPDX-License-Identifier: 0BSD.
com.github.luben:zstd-jni:1.5.7-11	Include the wrapper BSD-2, native Zstandard BSD-3, libdivsufsort-lite MIT, and legacy Zstandard BSD-2 coverage.	The JAR physically embeds the JNI wrapper and native code governed by these different licenses. Shared MIT and BSD-2 terms can avoid duplication.