Evaluate configuring local and CI OTEL to send logs to GCP #35
Description
Consider supporting sending logs from local & CI clusters to GCP Logging and AWS CloudWatch. This would enable having logs available for inspection long after the relevant pods or even clusters were removed (e.g. to debug failed tests in CI).
It seems that for GCP the following would be needed (similar mechanisms for AWS can be devised):
- Service account key would need to be available
- In CI this can be generated once and then provided from a GitHub repository secret
- Locally, each developer would need to generate a service account key, and provide it to the OTEL Collector Helm chart
- Either way, a
Secretobject should be created from it (see next step)
- The OTEL Helm chart would need the service account key to be mounted to the collector. This can be done by utilizing the Helm chart's
extraVolumes,extraVolumeMountsandextraEnvskeys to mount such a key from theSecretobject mentioned in the first step - The OTEL collector would also be configured with an exporter that uses the mounted service account key to send logs to GCP Logging.
Here's how to mount a local file to the OTEL collector through its Helm chart's values:
...
presets:
logsCollection:
enabled: true # <--- Enable this
config:
...
processors:
...
attributes/clusterid:
actions:
- key: cluster_id
value: local-joe # <--- Each developer as well as CI should use some unique value
action: insert
transform:
metric_statements:
- context: datapoint
statements:
- set(attributes["exported_service_name"], attributes["service_name"])
- delete_key(attributes, "service_name")
- set(attributes["exported_service_version"], attributes["service_name"])
- delete_key(attributes, "service_name")
- set(attributes["exported_service_namespace"], attributes["service_namespace"])
- delete_key(attributes, "service_namespace")
- set(attributes["exported_service_instance_id"], attributes["service_instance_id"])
- delete_key(attributes, "service_instance_id")
- set(attributes["exported_instrumentation_source"], attributes["instrumentation_source"])
- delete_key(attributes, "instrumentation_source")
- set(attributes["exported_instrumentation_version"], attributes["instrumentation_version"])
- delete_key(attributes, "instrumentation_version")
...
exporters:
...
googlecloud:
log:
default_log_name: "devbot.kfirs.com/dev/arikkfir"
...
service:
pipelines:
logs:
processors: [ ..., attributes/clusterid, ... ]
exporters: [ googlecloud ]
...
extraVolumes:
- name: otel-collector-gcp
secret:
secretName: otel-collector-gcp # <--- Creates a volume from this secret
optional: false
extraVolumeMounts:
- name: otel-collector-gcp
mountPath: /conf/otel-collector-gcp # <--- Mounts it into the container
extraEnvs:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /conf/otel-collector-gcp/service-account-key.json # <--- Tells GCP SDK to use the mounted key from the secret