DBMON-6747 ClickHouse Async Inserts Monitoring Buffer State Collection by jenny-chung · Pull Request #24517 · DataDog/integrations-core

jenny-chung · 2026-07-10T21:45:06Z

What does this PR do?

Add buffer state collection for async inserts in ClickHouse. It collects snapshots from system.asynchronous_inserts, including pending bytes, pending entry count, active buffer count, and each buffer's scheduled flush deadline. This collection logic shares the existing query samples job instead of running as its own job to avoid the overhead of a new connection.

Motivation

Support for diagnosing ClickHouse asynchronous inserts in DBM. We can't tell if a buffer is backing up, how big it's getting, or when its pending data is scheduled to flush, so this gives us a way to actually diagnose these issues instead of manually querying the system.* tables.

Agent payload (in mitm proxy):

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

datadog-datadog-prod-us1 · 2026-07-10T21:47:00Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 78.54%
• Overall Coverage: 93.38% (+5.01%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 069b175 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 918a5c439d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-10T21:49:31Z

        arbitrary_types_allowed=True,
        frozen=True,
    )
+    asynchronous_insert_buffer_snapshot: Optional[AsynchronousInsertBufferSnapshot] = None


Update the defaults test for the new field

Adding this InstanceConfig annotation without adding asynchronous_insert_buffer_snapshot to clickhouse/tests/test_config_defaults.py makes test_all_config_defaults fail: that test compares InstanceConfig.__annotations__ against EXPECTED_DEFAULTS and explicitly fails for any missing field. The ClickHouse unit suite will therefore reject this change even though the runtime default is provided in dict_defaults.py.

Useful? React with 👍 / 👎.

Fixed in commit d8febb3

chatgpt-codex-connector · 2026-07-10T21:49:31Z

+            self.statement_samples = ClickhouseStatementSamples(
+                self, self._config.query_samples, self._config.asynchronous_insert_buffer_snapshot
+            )


Decouple buffer snapshots from query samples

This wires the buffer snapshot only through ClickhouseStatementSamples, which is constructed under the query_samples.enabled guard immediately above and scheduled at query_samples.collection_interval. If an operator enables asynchronous_insert_buffer_snapshot while disabling query samples, no buffer events are emitted; if they raise the query-samples interval above the buffer interval, snapshots are silently under-collected. Since the new option is documented as its own DBM feature requiring only dbm: true, it needs to be scheduled when the buffer feature is enabled rather than depending on query samples.

Useful? React with 👍 / 👎.

Addressed in commits 23635e1 and c25b6e4

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e3f1f53ed3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T22:25:42Z

+        if buffer_config.enabled:
+            enabled_intervals.append(buffer_config.collection_interval)
+        collection_interval = (
+            math.gcd(*(int(i) for i in enabled_intervals)) if enabled_intervals else samples_collection_interval


Preserve fractional collection intervals

When an existing user sets query_samples.collection_interval: 0.5 or enables the new buffer snapshot with collection_interval: 0.5, this truncates the positive float to 0, so the subsequent rate_limit=1 / collection_interval raises during check initialization. The config schema accepts number values and validation only rejects intervals <= 0, so positive fractional intervals that previously worked now crash instead of being scheduled at the requested rate.

Useful? React with 👍 / 👎.

Fixed division by zero error to preserve fractional collection intervals - c7e90e7

sangeetashivaji · 2026-07-20T11:36:19Z

+        ## Enable collection of asynchronous insert buffer snapshots from system.asynchronous_inserts.
+        ## Requires `dbm: true`. Sends per-buffer records as DBM events (dbm_type="async_inserts_buffer")
+        ## including the query text, showing bytes pending, pending entry count, and oldest
+        ## entry age per buffer.


This says oldest entry age per buffer but looks like we emit flush_deadline_us. Lets confirm the correct behavior with the CH folks and update this description and the field correctly to match what we emit

Waiting on confirmation with CH team, but description has been updated in the meantime 5d4c294

sangeetashivaji · 2026-07-20T11:40:33Z

+                raw=True,
+            )
+            return []
+        except Exception as e:


On ClickHouse versions that predate system.asynchronous_inserts, this query raises "Unknown table" every cycle, which this branch logs as a full ERROR stack trace and counts as async_inserts_buffer.error in every collection_interval. Let's have a version check or detect the "unknown table" case and downgrade it to a single warning + skip.

Added case to detect unknown table and downgrades it to single warning + skip in df0ee69

we'd want to skip the collection entirely when we see UNKNOWN_TABLE_ERROR_CODE or if version check is possible to avoid the check when the table is not present that would be even better too

sangeetashivaji · 2026-07-20T11:45:26Z

            elapsed_ms,
        )

+    def _collect_buffer_snapshot(self):


https://github.com/DataDog/integrations-core/blob/master/AGENTS.md#python-code-style asks that any new code should provide type hints. let's add these to all new methods

Added type hints - d97bdac

sangeetashivaji · 2026-07-20T11:48:29Z


        try:
-            self._collect_samples()
+            if self._config.enabled and time.time() - self._last_samples_time >= self._collection_interval:


let's use one time.time() call here

Changed to single call in 3bc61f5

sangeetashivaji · 2026-07-20T11:52:32Z

+    assert check.statement_samples._expected_collection_interval == 5
+
+
+def test_query_samples_independent_of_buffer():


this is nice, but there's no mirror test for the buffer side - i.e. that _collect_buffer_snapshot only fires once per _buffer_collection_interval via _last_buffer_snapshot_time

Added mirror test for buffer side - 09c32cc

…async_inserts

…r greater than 0 validation

…eturned data

…use-async-inserts # Conflicts: # clickhouse/datadog_checks/clickhouse/statement_samples.py

dd-octo-sts · 2026-07-26T19:43:26Z

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and code coverage settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`dep`	Verify dependency pins are consistent and Agent-compatible	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`qa-label`	Validate the pull request declares whether it needs QA for the next Agent release	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

jenny-chung added 11 commits June 24, 2026 18:16

Add initial scaffolding for async insert buffer snapshot collection

26a60d5

Add async insert buffer snapshot to config spec

72ad2af

Add async insert buffer snapshot config defaults and validation

9d7c7ec

Register async insert buffer gauge metrics in metadata

62da945

Add async insert buffer snapshot collection job and metrics

9ccd4f7

Wire async insert buffer snapshot job

6f5d90a

Remove async insert buffer gauge metrics from metadata

8b0eb55

Update spec.yaml to match design

506f19f

Wire async insert buffer snapshot into query samples

3703bf3

Set up config for async insert buffer snapshot

daae7d3

Add async insert buffer snapshot collection to query samples job

918a5c4

jenny-chung requested review from a team as code owners July 10, 2026 21:45

temporal-github-worker-1 Bot added agent/review-requested ecosystems/review-requested product/review-requested labels Jul 10, 2026

dd-octo-sts Bot added documentation integration/clickhouse team/agent-integrations labels Jul 10, 2026

Merge branch 'master' into jenny-chung/clickhouse-async-inserts

c292e8b

dd-octo-sts Bot added the team/database-monitoring-agent label Jul 10, 2026

jenny-chung changed the title ~~DBMON-6747~~ DBMON-6747 ClickHouse Async Inserts Monitoring Buffer State Collection Jul 10, 2026

chatgpt-codex-connector Bot reviewed Jul 10, 2026

View reviewed changes

jenny-chung marked this pull request as draft July 10, 2026 21:50

jenny-chung added the qa/required QA is required for this PR and will generate a QA card label Jul 10, 2026

jenny-chung added 3 commits July 10, 2026 18:04

Add changelog entry for async insert buffer snapshot

032329e

Add expected defaults for async insert buffer snapshot config

d8febb3

Fix buffer snapshot and query samples coupling

23635e1

jenny-chung force-pushed the jenny-chung/clickhouse-async-inserts branch from 75590a5 to 23635e1 Compare July 13, 2026 18:22

jenny-chung added 2 commits July 13, 2026 14:23

Use GCD to schedule buffer snapshot and query samples collection

c25b6e4

Add unit tests for async insert buffer snapshot collection

e3f1f53

jenny-chung marked this pull request as ready for review July 13, 2026 22:21

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

jenny-chung added 4 commits July 16, 2026 10:28

Add to comment to explain why we collapse collection into query samples

78b1d5c

Rename buffer field oldest_entry_us to flush_deadline_us

647d204

Use a single time.time() call in buffer snapshot collection

5536bff

Use dedicated error metric for buffer snapshot collection

b6ac4ba

sangeetashivaji reviewed Jul 20, 2026

View reviewed changes

jenny-chung added 13 commits July 21, 2026 12:08

Use single time.time call in sample collection

3bc61f5

Add type hints to new methods in statement_samples

d97bdac

Add mirror test for buffer snapshot collection interval

09c32cc

Fix zero division error from fractional collection intervals

c7e90e7

Detect unknown table error and log as single warning + skip

df0ee69

Fix incorrect description for buffer snapshot config

5d4c294

Rename asynchronous_insert_buffer_snapshot config to collect_pending_…

c8de67d

…async_inserts

Rename payload_row_limit to max_samples_per_collection and add intege…

2dda410

…r greater than 0 validation

Add dd.clickhouse.async_inserts_buffer metrics to track the size of r…

b26c4e0

…eturned data

Remove bytes_submitted metric from async_inserts_buffer

ce4c582

Merge remote-tracking branch 'origin/master' into jenny-chung/clickho…

428990a

…use-async-inserts # Conflicts: # clickhouse/datadog_checks/clickhouse/statement_samples.py

Send buffer snapshot on the dbm-metrics track

bcb7ded

Add server clock to buffer snapshot query

069b175

		assert check.statement_samples._expected_collection_interval == 5


		def test_query_samples_independent_of_buffer():

Uh oh!

Conversation

jenny-chung commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 10, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

jenny-chung Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jenny-chung Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Jul 26, 2026

Validation Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jenny-chung commented Jul 10, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 10, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

jenny-chung Jul 13, 2026 •

edited

Loading

jenny-chung Jul 24, 2026 •

edited

Loading