refactor: Introduce build aggregator

[maltesander]

Description

Part of stackabletech/issues#864.

• Plain KubernetesResources; fallible build(cluster, sa_name) reusing existing builders
• Reconcile rewritten to a build-then-apply loop (StatefulSets last)
• Built-SA name

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[adwk67]

Looks fine, but we should look at the labelling mechanism.

[adwk67]

This service construction looks different to what we do in e.g. HBase and Druid, where we use full v2 labelling mechanisms. Did we forget to do this for HDFS? HDFs stills constructs labels using plain strings, which the "new" mechanism doesn't allow (and which is why the resulting service builders are infallible, whereas HDFS's is fallible). Or was there a specific reason for this?

[adwk67]

How does this fit into the dereference/validate/build/apply paradigm?
let discovery_cm = build_discovery_config_map
That requires a client and can only be done once the listeners are deployed. I guess no way around that?

[siegfriedweber]

How does this fit into the dereference/validate/build/apply paradigm? let discovery_cm = build_discovery_config_map That requires a client and can only be done once the listeners are deployed. I guess no way around that?

The build_discovery_config_map should return Option<ConfigMap>. If the Listener status is not set, None is returned. The operator should watch the Listener. When its status is set, reconcile is called and the discovery ConfigMap is eventually created.