[charzl]

What does this PR do?

Makes metrics.exporter: prometheus report all master processes on one /metrics endpoint.

The exporter now uses prometheus_client in multiprocess mode instead of the OpenTelemetry SDK. Every process (master, MWorkers, minion job processes) writes its values to a shared directory, and the process that serves /metrics adds them up on each scrape. Observable gauges stay callbacks and are evaluated at scrape time in the serving process. OTLP and console exporters are not touched.

What issues does this PR fix or reference?

Fixes #70365
Refs #70248 (the OTLP side is #70360)

Previous Behavior

Only the process that bound the port was visible. Processes forked from it could not bind the same port, so what the MWorkers counted was never exported.

New Behavior

One series per metric and label set, with the total over all master processes. No per-process label, nothing to sum. Counts of a worker that exited stay in the total until the master stops, so counters do not go down.

Behaviour to be aware of:

• service_name / resource_attributes are still exposed on target_info, but the otel_scope_* labels are no longer added to each series. Metric names are unchanged.
• The shared directory is a fresh temp directory per start (or metrics.prometheus.multiproc_dir, whose *.db files are removed at start). If PROMETHEUS_MULTIPROC_DIR is already set in the environment, Salt logs a warning and uses its own directory.
• After a failed bind, processes forked later do not try again. A problem updating a metric is logged once at warning level.
• get_meter() returns None with this exporter.

Known limitations:

• The files of exited processes stay in the directory until the master stops (64 KiB each). They cannot simply be deleted while running, because counters would drop. prometheus_client has no compaction for counters and histograms.
• Salt sets PROMETHEUS_MULTIPROC_DIR in its own processes, so a Python program started by Salt (for example with cmd.run) that uses prometheus_client inherits it.
• Not verified on Windows (the new tests are skipped there), on a real master with a real Prometheus server, or for the effect on fire_event latency.

Merge requirements satisfied?

• Docs
• Changelog
• Tests written/updated

Tests: tests/pytests/unit/utils/test_metrics_prometheus.py (names, labels, threads writing while scraping, stale files, bad config, environment variable, bind failure) and tests/pytests/functional/utils/test_metrics_prometheus_multiprocess.py (three forked workers plus the parent, real HTTP scrape). They passed on Ubuntu 24.04/26.04 and Rocky 9 in CI on my fork, where the only failures are Test Package / downgrade (test_libyaml.py) and ssh/test_saltext.py, which also fail on unmodified 3008.x.

Run on a real master

I also ran this branch on a real master with a real Prometheus server: the tests/monitoring Docker Compose setup, in my fork, with the stress load for 10 minutes, 5 MWorkers and metrics.exporter: prometheus. The workflow changes needed for that are in the last two commits: a metrics_exporter input for the nightly stress workflow, a scrape target for the master's own metrics, three checks in analyze_stats.py that only run with metrics_exporter: prometheus, and the ownership of the prometheus data directory (as on master). With the default inputs the stress test behaves as before.

• The Prometheus scrape target was up. salt_master_requests_handled_total, which is only incremented inside the MWorkers, was 7032 with one series per cmd and no per-process label.
• resets() of that counter over the run was 0, so the values did not go down between scrapes.
• The shared directory held 6 counter files and 5 histogram files: the main process and the 5 MWorkers.
• The stress analyzer's memory check failed (RSS slope 91 KB/s, its limit is 10 KB/s). It fails the same way on this branch with metrics disabled (119 KB/s, and its FD check as well), so it looks like the start-up of a 10 minute run and not something this change adds. That is one run each, so the two numbers only say the same order of magnitude.

Runs: prometheus, metrics disabled.

How the stress test checks it

Run "Nightly Stress Test" by hand (workflow_dispatch) with enable_metrics: true and metrics_exporter: prometheus. The scheduled nightly run keeps enable_metrics: false, so it does not exercise this exporter.

The workflow adds a metrics: block with exporter: prometheus and host: 0.0.0.0 to master.conf, restarts the master, runs the stress load, and the existing Prometheus container scrapes salt-master:9464 (new salt-metrics job). analyze_stats.py then checks, only in this mode:

• the salt-metrics target is up;
• salt_master_requests_handled_total is above 0 (it is only incremented in the MWorkers, so this shows their values reach the one endpoint);
• resets() of that counter over the run is 0.

The workflow also saves the master's /metrics output and the listing of the shared directory as artifacts. The existing memory, FD and process-count checks run as before.

Limits: the load is a loop of commands, so the checks are about presence and trend, not exact totals (the functional test covers those). The slope checks look at the last 30 minutes, so runs shorter than that are dominated by start-up.

Commits signed with GPG?

No

[dwoz]

@charzl Pre-commit is failing

[twangboy]

Does this still work when multiprocessing is False?

[charzl]

Does this still work when multiprocessing is False?

Yes. With multiprocessing: False, minion jobs run as threads in the same process. Added one more test to validate it.