Cherry-picks for 10.2.1

doc/admin-guide/files/records.yaml.en.rst

@@ -2007,6 +2007,94 @@ Origin Server Connect Attempts
    the connection. Useful when the origin supports keep-alive, removing the time needed to set up a
    new connection from the next request at the expense of added (inactive) connections.
 
+.. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_enabled INT 0
+   :reloadable:
+   :overridable:
+
+   Enable per upstream server connection metrics. These metrics are dynamically named, one set per
+   upstream server group, so the number of them scales with the number of distinct upstream servers
+   seen. See :ref:`per-server-connection-metrics`.
+
+   ===== ======================================================================================
+   Value Effect
+   ===== ======================================================================================
+   ``0`` No per server connection metrics.
+   ``1`` Per server connection metrics are collected for each upstream server group.
+   ===== ======================================================================================
+
+   What is published from them is controlled separately by
+   :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate`, which by default publishes the
+   per group metrics themselves.
+
+   Because this is overridable, metrics can be enabled for the upstreams of interest and left off
+   for the rest, for example with :ref:`admin-plugins-conf-remap` on a specific mapping.
+
+   The value is applied when a connection group is created. Where two mappings that disagree about
+   this setting resolve to the same group -- that is, the same key under
+   :ts:cv:`proxy.config.http.per_server.connection.match` -- the transaction that creates the group
+   determines its metrics, and later transactions do not change them. A group is discarded once its
+   connection count reaches zero, so *raising* the level of publication is picked up the next time
+   that upstream is reopened: enabling metrics, or enabling the aggregates, takes effect as upstreams
+   reconnect. Lowering it does not. Metrics are never retired once published, so disabling this
+   setting, or switching
+   :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` to ``2``, leaves the names that
+   are already published in place, frozen at their last sampled value, until |TS| is restarted. This
+   affects only which metrics exist; enforcement of
+   :ts:cv:`proxy.config.http.per_server.connection.max` uses the group's own connection count and is
+   unaffected.
+
+.. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_aggregate INT 0
+   :reloadable:
+   :overridable:
+
+   Control what is published from the per server connection metrics enabled by
+   :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Has no effect when that setting
+   is ``0``.
+
+   A per hostname aggregate sums a counter across every group belonging to that hostname that has
+   aggregation enabled, and exists only for
+   :ts:cv:`match type <proxy.config.http.per_server.connection.match>` ``both``, since that is the
+   only match type whose group key carries the hostname. See :ref:`per-server-connection-metrics`.
+
+   ===== ======================================================================================
+   Value Effect
+   ===== ======================================================================================
+   ``0`` No aggregates. The per group metrics are published under their own names.
+   ``1`` Publish the per hostname aggregates and the per group metrics.
+   ``2`` Publish only the per hostname aggregates. The per group metrics from which they are
+         computed are collected but not published, which keeps the number of published metrics
+         proportional to hostnames rather than to groups.
+   ===== ======================================================================================
+
+   With value ``2``, a group that has no aggregate to belong to -- any match type other than
+   ``both`` -- has its per group metrics published anyway, since otherwise nothing at all would be
+   reported for it.
+
+   Values ``0`` and ``1`` can produce a very large number of metrics when the match type includes the
+   address or port, since there is then one set per address and port rather than one per hostname.
+
+   Like :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`, this is applied when a
+   connection group is created, with the same consequence for mappings that disagree and resolve to
+   the same group. A group joins its hostname's aggregate only if the mapping that first opened that
+   upstream had aggregation enabled, so mappings that disagree for one hostname produce an aggregate
+   that covers only part of it.
+
+   The reload is one-directional for the same reason given under
+   :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Raising the value takes effect
+   as upstreams reconnect, but moving to ``2`` does not hide per group metrics that are already
+   published, and moving from ``1`` to ``0`` does not stop the hostname aggregates from publishing.
+   Reducing the number of published metrics therefore requires a restart, which matters most for
+   ``2``, the value chosen specifically to bound that number.
+
+.. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_prefix STRING NULL
+   :reloadable:
+
+   An optional prefix inserted into the per server connection metric names, between the fixed
+   ``proxy.process.http.per_server.<counter>.`` portion of the name and the upstream server group
+   or hostname. Useful to distinguish metrics from separate
+   :ts:cv:`match <proxy.config.http.per_server.connection.match>` configurations sharing the same
+   upstream. See :ref:`per-server-connection-metrics`.
+
 .. ts:cv:: CONFIG proxy.config.http.connect_attempts_rr_retries INT 3
    :reloadable:
    :overridable:

doc/admin-guide/monitoring/statistics/core/http-connection.en.rst

@@ -201,6 +201,77 @@ HTTP Connection
    Current number of TCP connections for tunnels where the far end is the server,
    except for those counted by ``proxy.process.tunnel.current_server_connections_tls``
 
+.. _per-server-connection-metrics:
+
+Per Server Connection Metrics
+-----------------------------
+
+Unlike the metrics above these do not have fixed names. They are created dynamically, one set per
+upstream server group as defined by :ts:cv:`proxy.config.http.per_server.connection.match`, and,
+when the match type is ``both``, one aggregate set per hostname. Whether they are collected at all is
+controlled by :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`, and which of them are
+published by :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate`. An optional
+:ts:cv:`proxy.config.http.per_server.connection.metric_prefix` can be inserted into the names.
+
+Per group names are ``proxy.process.http.per_server.<counter>.<group>``, where ``<group>`` depends on
+the match type: an IP address, an ``address:port`` pair, a hostname, or, for ``both``,
+``<hostname>.<address:port>``. Per hostname names are
+``proxy.process.http.per_server.<counter>.<hostname>``. Aggregates exist only for match type
+``both``, because that is the only match type whose group key carries the hostname. An ``ip`` or
+``port`` group is keyed on the address alone and is shared by every hostname that resolves to it, so
+there is no single hostname to aggregate it under. For match type ``host`` the group name is already
+the bare hostname, so an aggregate would carry the same name as the single group it summarises.
+
+For a group, ``<counter>`` is one of:
+
+current_connection
+   Gauge. The number of connections currently open to the group.
+
+total_connection
+   Counter. The total number of connections ever opened to the group. Never decreases.
+
+blocked_connection
+   Counter. The total number of connection attempts to the group blocked by
+   :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases.
+
+For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
+hostname which have aggregation enabled, plus:
+
+current_connection_max
+   Gauge. The largest ``current_connection`` value among the groups of that hostname at the moment
+   of sampling, so the maximum rather than the sum of the groups' current counts. This is useful
+   because :ts:cv:`proxy.config.http.per_server.connection.max` is enforced per group rather than
+   per hostname, so the busiest group is what determines whether connections are about to be
+   blocked. Like ``current_connection`` it rises and falls with traffic and is not a high-water
+   mark. There is no per group ``current_connection_max``; it exists only as a hostname aggregate.
+
+Because :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` is overridable, a group
+joins its hostname's aggregate only if the mapping that first opened that upstream had aggregation
+enabled. Mappings that disagree for one hostname therefore produce an aggregate over part of it: the
+sums cover a subset of the groups and ``current_connection_max`` takes its maximum over that same
+subset, with nothing in the metric to indicate it. Keeping the setting uniform across the mappings
+for a hostname avoids this.
+
+Because :ts:cv:`proxy.config.http.per_server.connection.match` is also overridable, one hostname can
+use match type ``host`` on one mapping and ``both`` on another. The ``host`` group and the hostname
+aggregate are then published under the same name and merged into a single metric that carries both,
+so a hostname should use one match type throughout.
+
+Every published per server metric is recomputed periodically, currently every 5 seconds, rather than
+on every connection event, so a reader sees a value up to that interval old. This is true of the
+hostname aggregates and of the published per group metrics alike: those are
+mirrored from the internal ones by the same periodic mechanism, not written as connections open and
+close. It applies to ``current_connection_max`` too, which reports the maximum across groups as of
+the last sample rather than a running peak. To obtain the peak over a longer window, compute a
+maximum over time from this gauge in the monitoring system.
+
+At :ts:cv:`metric_aggregate <proxy.config.http.per_server.connection.metric_aggregate>` value ``2``
+the per group metrics still exist internally, since the aggregates are computed from them, but are not
+published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which
+reads them directly and so is not subject to the sampling delay above. That visibility is intended
+for debugging and is not a stable interface: the existence, granularity and naming of the per group
+metrics may change independently of the published aggregates.
+
 HTTP/2
 ------
 

doc/admin-guide/plugins/lua.en.rst

@@ -4746,6 +4746,8 @@ Http config constants
     TS_LUA_CONFIG_HTTP_SERVER_MIN_KEEP_ALIVE_CONNS
     TS_LUA_CONFIG_HTTP_PER_SERVER_CONNECTION_MAX
     TS_LUA_CONFIG_HTTP_PER_SERVER_CONNECTION_MATCH
+    TS_LUA_CONFIG_HTTP_PER_SERVER_CONNECTION_METRIC_ENABLED
+    TS_LUA_CONFIG_HTTP_PER_SERVER_CONNECTION_METRIC_AGGREGATE
     TS_LUA_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES
     TS_LUA_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER
     TS_LUA_CONFIG_HTTP_CONNECT_DOWN_POLICY

doc/admin-guide/plugins/prefetch.en.rst

@@ -34,6 +34,15 @@ On every **incoming** URL request, the plugin can decide to pre-fetch the
 **next object** or more objects based on the common URL path pattern and a
 pre-defined pre-fetch policy.
 
+.. note::
+
+   The plugin only prefetches on requests that go through a cache lookup.
+   On transactions where ATS skips the cache lookup -- non-cacheable
+   methods, or caching turned off (:ts:cv:`proxy.config.http.cache.http`
+   set to ``0``) -- the plugin does not prefetch. Issuing prefetches into
+   a cache that will not be consulted would do real work for no cache
+   benefit.
+
 Currently, most HLS video urls follow a predictable pattern, with most URLs
 containing a segment number. Since the segments are ~10s of content, the normal
 usage pattern is to fetch the incremental segment every few seconds. The CDN

doc/admin-guide/plugins/rate_limit.en.rst

@@ -92,8 +92,11 @@ are available:
 
 .. option:: --maxage
 
-   An optional ``max-age`` for how long a transaction can sit in the delay queue.
-   The value (default 0) is the age in seconds.
+   An optional maximum age for how long a transaction can sit in the delay queue.
+   The value (default 0) is the age in **milliseconds**.
+
+   Note that the equivalent YAML setting, ``max_age`` under a ``queue`` node, is in
+   seconds.
 
 .. option:: --prefix
 
@@ -149,7 +152,7 @@ and nodes are documented below.
             rate: 200
             queue:
                size: 1000
-               max-age: 30
+               max_age: 30
             metrics:
                tag: example.com
                prefix: ddos
@@ -163,11 +166,11 @@ and nodes are documented below.
             buckets: 10
             size: 15
             percentage: 90
-            max-age: 300
+            max_age: 300
             perma-block:
                limit: 100
                threshold: 1
-               max-age: 1800
+               max_age: 1800
       lists:
          - name: internal
             cidr:
@@ -212,9 +215,9 @@ For the top level `selector` node, the following options are available:
    how many queued transactions we will allow. When this threshold is reached,
    all additional connections are immediately errored out in the TLS handshake.
 
-   The queue option can include a `size` and a `max-age` option. The size is
-   default to ``UINT_MAX``, which is essentially unlimited. The max-age is
-   default to ``0``, which means no age limit.
+   The queue option can include a `size` and a `max_age` option. The size defaults
+   to ``UINT_MAX``, which is essentially unlimited. The max_age is in seconds and
+   defaults to ``0``, which means no age limit.
 
    No queue is enabled without this configuration directive, but it can also be
    disabled explicitly if the size is set to ``0``.
@@ -268,7 +271,7 @@ and the following options:
    This is the minimum percentage of the ``limit`` that the pressure must be at, before
    we start blocking IPs. The default is ``0.9`` which means ``90%`` of the limit.
 
-.. option:: max-age
+.. option:: max_age
 
    This is used for aging out entries out of the LRU, the default is ``0`` which means
    no aging happens. Even with no aging, entries will eventually fall out of buckets
@@ -289,7 +292,7 @@ blocked for a long time. The configuration for this bucket is:
    This option specifies from which bucket an IP is allowed to move from into the
    perma block bucket. A good value here is likely ``0`` or ``1``, which is very conservative.
 
-.. option:: max-age
+.. option:: max_age
 
    Like above, but only applies to the long term (`perma-block`) bucket. Default is
    ``0``, which means no aging to this bucket is applied.

doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst

@@ -155,6 +155,8 @@ TSOverridableConfigKey Value                                               Confi
 :enumerator:`TS_CONFIG_HTTP_PER_PARENT_CONNECT_ATTEMPTS`                 :ts:cv:`proxy.config.http.parent_proxy.per_parent_connect_attempts`
 :enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MATCH`                 :ts:cv:`proxy.config.http.per_server.connection.match`
 :enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MAX`                   :ts:cv:`proxy.config.http.per_server.connection.max`
+:enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_METRIC_AGGREGATE`      :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate`
+:enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_METRIC_ENABLED`        :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`
 :enumerator:`TS_CONFIG_HTTP_POST_CHECK_CONTENT_LENGTH_ENABLED`           :ts:cv:`proxy.config.http.post.check.content_length.enabled`
 :enumerator:`TS_CONFIG_HTTP_REDIRECT_USE_ORIG_CACHE_KEY`                 :ts:cv:`proxy.config.http.redirect_use_orig_cache_key`
 :enumerator:`TS_CONFIG_HTTP_REQUEST_BUFFER_ENABLED`                      :ts:cv:`proxy.config.http.request_buffer_enabled`

doc/developer-guide/api/types/TSOverridableConfigKey.en.rst

@@ -150,6 +150,8 @@ Enumeration Members
 .. enumerator:: TS_CONFIG_HTTP_ALLOW_HALF_OPEN
 .. enumerator:: TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MAX
 .. enumerator:: TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MATCH
+.. enumerator:: TS_CONFIG_HTTP_PER_SERVER_CONNECTION_METRIC_ENABLED
+.. enumerator:: TS_CONFIG_HTTP_PER_SERVER_CONNECTION_METRIC_AGGREGATE
 .. enumerator:: TS_CONFIG_SSL_CLIENT_VERIFY_SERVER_POLICY
 .. enumerator:: TS_CONFIG_SSL_CLIENT_VERIFY_SERVER_PROPERTIES
 .. enumerator:: TS_CONFIG_SSL_CLIENT_SNI_POLICY