diff --git a/CHANGELOG.md b/CHANGELOG.md index b3e14ae6..a9a9dc58 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file. ### Added +- Added permissions required by Topology Provider ([#738]). - Add conversion webhook ([#753]). - Support objectOverrides using `.spec.objectOverrides`. See [objectOverrides concepts page](https://docs.stackable.tech/home/nightly/concepts/overrides/#object-overrides) for details ([#741]). @@ -19,6 +20,7 @@ All notable changes to this project will be documented in this file. - Previously, some shell output of init-containers was not logged properly and therefore not aggregated, which is fixed now ([#746]). +[#738]: https://github.com/stackabletech/hdfs-operator/pull/738 [#741]: https://github.com/stackabletech/hdfs-operator/pull/741 [#746]: https://github.com/stackabletech/hdfs-operator/pull/746 [#747]: https://github.com/stackabletech/hdfs-operator/pull/747 diff --git a/deploy/helm/hdfs-operator/templates/roles.yaml b/deploy/helm/hdfs-operator/templates/roles.yaml index 7ea3c6ef..442e9948 100644 --- a/deploy/helm/hdfs-operator/templates/roles.yaml +++ b/deploy/helm/hdfs-operator/templates/roles.yaml @@ -212,6 +212,8 @@ rules: verbs: - get - list + # needed for pod informer + - watch - apiGroups: - listeners.stackable.tech resources: @@ -219,3 +221,10 @@ rules: verbs: - get - list + # needed to query the crd version (v1alpha1 etc.) before fetching listeners + - apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions + verbs: + - get diff --git a/docs/modules/hdfs/pages/usage-guide/operations/rack-awareness.adoc b/docs/modules/hdfs/pages/usage-guide/operations/rack-awareness.adoc index 057032d6..621e1e2e 100644 --- a/docs/modules/hdfs/pages/usage-guide/operations/rack-awareness.adoc +++ b/docs/modules/hdfs/pages/usage-guide/operations/rack-awareness.adoc @@ -1,6 +1,6 @@ = HDFS Rack Awareness :rack-awareness-docs: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/RackAwareness.html -:hdfs-topology-provider: https://github.com/stackabletech/hdfs-topology-provider +:hdfs-topology-provider: https://github.com/stackabletech/hdfs-utils/blob/main/src/main/java/tech/stackable/hadoop/StackableTopologyProvider.java {rack-awareness-docs}[Rack awareness] is a feature in Apache Hadoop that allows users to define a cluster's node topology. Hadoop uses that topology to distribute block replicas in a way that maximizes fault tolerance. @@ -35,3 +35,7 @@ This creates an internal topology label by combining the values of the `topology In order to enable gathering this information the Hadoop images contain the {hdfs-topology-provider}[hdfs-topology-provider] on the classpath, which can be configured to read labels from Kubernetes objects. The operator deploys ClusterRoles and ServicesAccounts with the relevant RBAC rules to allow the Hadoop Pod to access the necessary Kubernetes objects. +Topologies and other metadata such as Node- and Pod-IPs and endpoints are held in separate caches so that they can be refeshed independently of one another. +The {hdfs-topology-provider}[hdfs-topology-provider] is namespace-scoped and pods in the active namespace are watched so that changes can be propagated to the internal cache to minimise cache misses. + +NOTE: Rack awareness may not work as expected on clusters such as `kind` or `k3s` that configure IP-masquerading differently to production-ready distributions.