Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion chart/skywalking/templates/NOTES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Learn more, please visit https://skywalking.apache.org/
{{- if not .Values.ui.enabled }}
The web UI is disabled in this release (ui.enabled=false). No UI Deployment,
Service, ConfigMap, Ingress, or PVC was created. OAP is still reachable on
its GraphQL port (default 12800) for any external UI you point at it.
its GraphQL port ({{ .Values.oap.ports.query | default .Values.oap.ports.rest }}) for any external UI you point at it.
{{- else }}
Get the UI URL by running these commands:
{{- if .Values.ui.ingress.enabled }}
Expand Down
12 changes: 12 additions & 0 deletions chart/skywalking/templates/oap-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,18 @@ spec:
value: default
- name: SW_QUERY_ZIPKIN_REST_PORT
value: "{{ index .Values.oap.ports "zipkin-query" }}"
{{- end }}
{{- if index .Values.oap.ports "receiver-grpc" }}
- name: SW_RECEIVER_GRPC_PORT
value: "{{ index .Values.oap.ports "receiver-grpc" }}"
{{- end }}
{{- if index .Values.oap.ports "receiver-rest" }}
- name: SW_RECEIVER_SHARING_REST_PORT
value: "{{ index .Values.oap.ports "receiver-rest" }}"
{{- end }}
{{- if .Values.oap.ports.query }}
- name: SW_QUERY_SERVER_REST_PORT
value: "{{ .Values.oap.ports.query }}"
{{- end }}
- name: JAVA_OPTS
value: "-Dmode=no-init {{ .Values.oap.javaOpts }}"
Expand Down
2 changes: 1 addition & 1 deletion chart/skywalking/templates/satellite-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ spec:
- name: SATELLITE_GRPC_CLIENT_KUBERNETES_SELECTOR_LABEL
value: "{{ template "skywalking.oap.labels" . }}"
- name: SATELLITE_GRPC_CLIENT_KUBERNETES_EXTRA_PORT
value: "{{ .Values.oap.ports.grpc }}"
value: "{{ index .Values.oap.ports "receiver-grpc" | default .Values.oap.ports.grpc }}"
{{- range $key, $value := .Values.satellite.env }}
- name: {{ $key }}
value: {{ $value | quote }}
Expand Down
2 changes: 1 addition & 1 deletion chart/skywalking/templates/ui-configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ default is left out so upstream owns it.
*/ -}}
{{- $oap := include "skywalking.oap.fullname" . }}
{{- $oapDefaults := dict
"queryUrl" (printf "${HORIZON_OAP_QUERY_URL:http://%s:%v}" $oap .Values.oap.ports.rest)
"queryUrl" (printf "${HORIZON_OAP_QUERY_URL:http://%s:%v}" $oap (.Values.oap.ports.query | default .Values.oap.ports.rest))
}}
{{- /* The admin host arrived in OAP 11. On a 10.x release 17128 is the
AI-pipeline URI-recognition server, so advertising an adminUrl there would point
Expand Down
2 changes: 1 addition & 1 deletion chart/skywalking/templates/ui-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ spec:
- name: HORIZON_SERVER_PORT
value: {{ .Values.ui.service.internalPort | quote }}
- name: HORIZON_OAP_QUERY_URL
value: "http://{{ $oap }}:{{ .Values.oap.ports.rest }}"
value: "http://{{ $oap }}:{{ .Values.oap.ports.query | default .Values.oap.ports.rest }}"
{{- if .Values.oap.ports.admin }}
- name: HORIZON_OAP_ADMIN_URL
value: "http://{{ $oap }}:{{ .Values.oap.ports.admin }}"
Expand Down
13 changes: 13 additions & 0 deletions chart/skywalking/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,19 @@ oap:
# null on any 10.x release, where 17128 is the AI-pipeline URI-recognition
# server instead, so pointing Horizon at it reaches the wrong service.
admin: 17128
# Separated ports. Each key, when set, starts that OAP server and moves its
# traffic off grpc / rest. With all three set, no agent or query traffic is
# left there: grpc carries the cluster traffic between OAP nodes, and rest
# the probes.
# receiver-grpc: gRPC receivers (agents, Envoy, OTLP gRPC); Satellite
# forwards here when it is set.
# receiver-rest: HTTP receivers (browser SDK, HTTP agents, OTLP HTTP).
# query: GraphQL, the AI agent conversation API and /status/config/ttl;
# Horizon's queryUrl follows it. Needs OAP 11.2.0 or later, older OAP
# ignores it and the UI would query a closed port.
# receiver-grpc: 11811
# receiver-rest: 12811
# query: 12810
# zipkin-receiver: 9411
# zipkin-query: 9412
# promql: 9090
Expand Down
1 change: 1 addition & 0 deletions docs/changes/changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
working tree with identical values, checking that the old labels and selectors are kept and the
init Job is never broken by an immutable pod-template change. `e2e-config-override.yaml` now checks
that a fresh install carries only the recommended labels and selects by them.
- **Separated receiver and query ports.** Three opt-in keys under `oap.ports` move OAP traffic off `grpc` / `rest`, so each port can be exposed, firewalled and given TLS on its own: `receiver-grpc` (sets `SW_RECEIVER_GRPC_PORT`; Satellite forwards to it), `receiver-rest` (sets `SW_RECEIVER_SHARING_REST_PORT`) and `query` (sets `SW_QUERY_SERVER_REST_PORT`, needs OAP 11.2.0 or later; Horizon's `queryUrl` follows it). With none set, nothing changes. See [OAP Endpoints](../expose/oap-endpoints.md#separate-the-receiver-and-query-ports).

### Bugs

Expand Down
40 changes: 37 additions & 3 deletions docs/expose/oap-endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,10 @@ helm template skywalking chart/skywalking \
| Key | Default | Serves | In `oap.ports` by default | Needs OAP env too |
|---|---|---|---|---|
| `grpc` | `11800` | Native agent protocol (traces, JVM/meters, profiling, logs, events). Every other gRPC receiver — OTLP, Envoy ALS, Rover eBPF, Satellite forwarding — shares it, because `SW_RECEIVER_GRPC_PORT` defaults to `0` (fall back to the core gRPC server) | yes | no |
| `rest` | `12800` | GraphQL query API at `/graphql` (Horizon's `oap.queryUrl`, `swctl`). HTTP receivers share it for the same reason (`SW_RECEIVER_SHARING_REST_PORT:0`). All three pod probes `tcpSocket` this port | yes | no |
| `rest` | `12800` | GraphQL query API at `/graphql` (Horizon's `oap.queryUrl`, `swctl`), unless `query` is set. HTTP receivers share it for the same reason (`SW_RECEIVER_SHARING_REST_PORT:0`). All three pod probes `tcpSocket` this port | yes | no |
| `receiver-grpc` | `11811` in the commented example | The gRPC receivers moved off `grpc` — see [Separate the receiver and query ports](#separate-the-receiver-and-query-ports). Satellite forwards here when it is set | no | no — the chart sets `SW_RECEIVER_GRPC_PORT` |
| `receiver-rest` | `12811` in the commented example | The HTTP receivers moved off `rest` | no | no — the chart sets `SW_RECEIVER_SHARING_REST_PORT` |
| `query` | `12810` in the commented example | The GraphQL query API, the AI agent conversation API and `/status/config/ttl`, moved off `rest`. Horizon's `oap.queryUrl` follows it. Needs OAP 11.2.0 or later | no | no — the chart sets `SW_QUERY_SERVER_REST_PORT` |
| `admin` | `17128` | admin-server: `/status/*`, `/debugging/*`, inspect, dsl-debugging, runtime-rule, and the `ui-management` template store Horizon uses when `ui.config.templates.mode` is `live`. Wired into Horizon as `oap.adminUrl` | yes | no |
| `zipkin-receiver` | `9411` | Zipkin span ingestion, `POST /api/v2/spans` | no | no — the chart sets `SW_RECEIVER_ZIPKIN=default` and `SW_RECEIVER_ZIPKIN_REST_PORT` for you |
| `zipkin-query` | `9412` | Zipkin query API under `/zipkin` (Zipkin-Lens compatible). Also makes the chart write Horizon's `oap.zipkinUrl` | no | no — the chart sets `SW_QUERY_ZIPKIN=default` and `SW_QUERY_ZIPKIN_REST_PORT` |
Expand Down Expand Up @@ -107,7 +110,38 @@ oap:
periodSeconds: 10
```

The default probes hardcode `12800`; leaving them behind makes the pod restart-loop. Horizon needs no change either way — the UI ConfigMap renders `oap.queryUrl` / `oap.adminUrl` from `oap.ports.rest` / `oap.ports.admin`. `zipkin-receiver` and `zipkin-query` have no such problem — the chart derives their env from the value.
The default probes hardcode `12800`; leaving them behind makes the pod restart-loop. Horizon needs no change either way — the UI ConfigMap renders `oap.queryUrl` / `oap.adminUrl` from `oap.ports.rest` (or `oap.ports.query` when set) / `oap.ports.admin`. `zipkin-receiver` and `zipkin-query` have no such problem — the chart derives their env from the value.

## Separate the receiver and query ports

By default `grpc` and `rest` carry both the agent and the query traffic: agents report on the same
ports that the UI queries, and `grpc` also carries the cluster traffic between OAP nodes.
Three keys split them, so each port can be published, firewalled and given TLS on its own:

```yaml
oap:
ports:
grpc: 11800
rest: 12800
admin: 17128
receiver-grpc: 11811 # agents, Envoy, OTLP gRPC, Satellite
receiver-rest: 12811 # browser SDK, HTTP agents, OTLP HTTP
query: 12810 # GraphQL, AI agent conversations, /status/config/ttl
```

For each key that is set, the chart publishes the port and sets the OAP variable that starts the
server (`SW_RECEIVER_GRPC_PORT`, `SW_RECEIVER_SHARING_REST_PORT`, `SW_QUERY_SERVER_REST_PORT`). It
also points its own clients at the new ports: Horizon's `oap.queryUrl` uses `query`, and Satellite
forwards to `receiver-grpc`. With all three set, no agent or query traffic is left on `grpc` and
`rest`: `grpc` carries the cluster traffic between OAP nodes, and `rest` the probes, which keep
checking `12800`. The keys are independent; set only the ones you need.

`receiver-grpc` and `receiver-rest` work on every OAP release this chart supports. `query` needs OAP
11.2.0 or later: an older OAP ignores the variable and keeps GraphQL on `12800`, while Horizon would
query the closed `query` port.

Repoint the clients outside the chart yourself: agents to `receiver-grpc`, browser and other HTTP
reporters to `receiver-rest`, and `swctl --base-url` or any other GraphQL client to `query`.

## What to point at what

Expand Down Expand Up @@ -154,7 +188,7 @@ oap:

`oap.service.type` accepts any Service type (`ClusterIP`, `NodePort`, `LoadBalancer`) and `oap.service.annotations` is passed straight through to the Service metadata. The agent protocol on `11800` is gRPC over HTTP/2, so an L7 HTTP ingress will not do — use an L4 / NLB Service or a gRPC-aware gateway.

A public OAP Service publishes **every** key in `oap.ports`, and `admin: 17128` is one of them by default — it serves `/status/*`, `/debugging/*`, inspect and the runtime-rule write APIs. Front the public address with something that forwards only `11800` / `12800` rather than dropping `admin` from `oap.ports`: the UI ConfigMap still renders `oap.adminUrl` as `:17128`, so removing the port breaks Horizon. See [TLS](tls.md) before putting any of these on the internet.
A public OAP Service publishes **every** key in `oap.ports`, and `admin: 17128` is one of them by default — it serves `/status/*`, `/debugging/*`, inspect and the runtime-rule write APIs. Front the public address with something that forwards only `11800` / `12800` (or only the receiver ports, when [separated](#separate-the-receiver-and-query-ports)) rather than dropping `admin` from `oap.ports`: the UI ConfigMap still renders `oap.adminUrl` as `:17128`, so removing the port breaks Horizon. See [TLS](tls.md) before putting any of these on the internet.

## Related

Expand Down
5 changes: 4 additions & 1 deletion docs/reference/skywalking-chart-values.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,11 @@ The following table lists the configurable parameters of the Skywalking chart an
| `oap.image.tag` | OAP container image tag. Required on every install -- no default | `null` |
| `oap.image.pullPolicy` | OAP container image pull policy | `IfNotPresent` |
| `oap.ports.grpc` | OAP grpc port for tracing or metric | `11800` |
| `oap.ports.rest` | OAP http port for the GraphQL query protocol (used by the UI and by `swctl`). The chart always builds the UI's `HORIZON_OAP_QUERY_URL` from it | `12800` |
| `oap.ports.rest` | OAP http port. It serves the GraphQL query protocol (used by the UI and by `swctl`), and the chart builds the UI's `HORIZON_OAP_QUERY_URL` from it, unless `oap.ports.query` is set | `12800` |
| `oap.ports.admin` | OAP admin REST port (admin-server, status, inspect, ui-management, dsl-debugging, runtime-rule). Introduced in OAP 11, which enables all of them by default and serves `/status/*` and `/debugging/*` here exclusively. Set to `null` on any OAP 10.x release, where 17128 is the AI-pipeline URI-recognition server instead. The chart sets the UI's `HORIZON_OAP_ADMIN_URL` only while this port is set | `17128` |
| `oap.ports.receiver-grpc` | OAP gRPC port of the receivers (agents, Envoy, OTLP gRPC). When set, the chart sets `SW_RECEIVER_GRPC_PORT` and Satellite forwards to it, and no agent traffic is left on `grpc`. See [OAP Endpoints](../expose/oap-endpoints.md#separate-the-receiver-and-query-ports) | not set (commented out in `values.yaml`) |
| `oap.ports.receiver-rest` | OAP HTTP port of the receivers (browser SDK, HTTP agents, OTLP HTTP). When set, the chart sets `SW_RECEIVER_SHARING_REST_PORT` | not set (commented out in `values.yaml`) |
| `oap.ports.query` | OAP HTTP port of the GraphQL query API, the AI agent conversation API and `/status/config/ttl`. When set, the chart sets `SW_QUERY_SERVER_REST_PORT` and builds the UI's `HORIZON_OAP_QUERY_URL` from it. Needs OAP 11.2.0 or later | not set (commented out in `values.yaml`) |
| `oap.ports.zipkin-receiver` | OAP http port for Zipkin receiver(not exposed by default) | not set (commented out in `values.yaml`) |
| `oap.ports.zipkin-query` | OAP http port for querying Zipkin traces and UI(not exposed by default). When set, the chart also gives the UI `HORIZON_OAP_ZIPKIN_URL`, so the Zipkin trace pages resolve | not set (commented out in `values.yaml`) |
| `oap.ports.promql` / `logql` / `traceql` / `metrics` | Further OAP listeners, each commented out in `values.yaml`. Any key added under `oap.ports` becomes both a container port and a Service port, named after the key | not set (commented out in `values.yaml`) |
Expand Down
2 changes: 1 addition & 1 deletion docs/ui/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ The field list itself is owned upstream:
| Environment variable | Value | Set when |
|---|---|---|
| `HORIZON_SERVER_PORT` | `ui.service.internalPort` (`8081`) | always |
| `HORIZON_OAP_QUERY_URL` | in-cluster OAP service + `oap.ports.rest` (`12800`) | always |
| `HORIZON_OAP_QUERY_URL` | in-cluster OAP service + `oap.ports.query` when set, else `oap.ports.rest` (`12800`) | always |
| `HORIZON_OAP_ADMIN_URL` | in-cluster OAP service + `oap.ports.admin` (`17128`) | only when `oap.ports.admin` is set |
| `HORIZON_OAP_ZIPKIN_URL` | in-cluster OAP service + `oap.ports.zipkin-query` + `/zipkin` | only when `oap.ports.zipkin-query` is set |
| `HORIZON_PUBLIC_URL` | first entry of `ui.ingress.hosts`, `https` when a `tls` block covers that host, else `http` | only when `ui.ingress.enabled` **and** `ui.ingress.hosts` are set |
Expand Down
2 changes: 1 addition & 1 deletion docs/ui/horizon-ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Booster UI needed only the GraphQL query port. Horizon's BFF needs two, and the

| field | OAP port | chart value | what it carries |
|---|---|---|---|
| `oap.queryUrl` | `12800` | `oap.ports.rest` | the GraphQL query API |
| `oap.queryUrl` | `12800` by default | `oap.ports.query` when set, else `oap.ports.rest` | the GraphQL query API |
Comment thread
wu-sheng marked this conversation as resolved.
Resolved
| `oap.adminUrl` | `17128` | `oap.ports.admin` | `/status/*`, `/debugging/*`, inspect, DSL debugging, runtime rules, dashboard templates |
| `oap.zipkinUrl` | commented out by default | `oap.ports.zipkin-query` | written as `http://<oap>:<port>/zipkin`, only when that port is set |

Expand Down
Loading