storage: estimate MySQL snapshot row counts for large tables by peterdukelarsen · Pull Request #37907 · MaterializeInc/materialize

peterdukelarsen · 2026-07-28T15:36:46Z

Motivation

Part of SS-360.

Count(*) can be very slow and we can speed it up with using table size statistics.

Description

If size estimate is > 1M just use the size estimate. Otherwise still use count(*).

This broadly matches how Postgres handles grabbing/reporting the full row-count and is inspired by that. Row count is used for Reporting progress in the frontend and if we roll out parallel snapshotting for partitioning data. Accuracy is not critical for correctness.

Mirror the Postgres source's approach to the snapshot size gauge: ask the optimizer for a row estimate first and only run an exact COUNT(*) when the estimate is at or below a threshold. Large tables report the information_schema.tables estimate directly, skipping an O(rows) index walk during snapshot setup. A NULL or zero estimate falls through to the exact count. The threshold is the new mysql_source_snapshot_exact_count_max_rows dyncfg, defaulting to 1M rows to match the Postgres source's cutoff. Setting it to 0 forces the estimated path everywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

peterdukelarsen · 2026-07-28T19:56:32Z

Ran a local test to confirm what happens if no stats have been calculated. Looks like we'll get a row reporting 0 rows.

plarsen@Peters-MacBook-Pro ~ % docker run --name mysql-stats-test -d -p 3306:3306 \
  -e MYSQL_ROOT_PASSWORD='p@ssw0rd' \
  mysql:8.0 \
  --innodb-stats-auto-recalc=OFF

... login...

mysql> create table bar (a int);
Query OK, 0 rows affected (0.076 sec)

mysql> mysql> select table_name, table_rows from information_schema.tables where table_name='bar';
+------------+------------+
| TABLE_NAME | TABLE_ROWS |
+------------+------------+
| bar        |          0 |
+------------+------------+
1 row in set (0.040 sec)

mysql> insert into bar values (1);
Query OK, 1 row affected (0.007 sec)

mysql> insert into bar values (2);
Query OK, 1 row affected (0.006 sec)

mysql> select table_name, table_rows from information_schema.tables where table_name='bar';
+------------+------------+
| TABLE_NAME | TABLE_ROWS |
+------------+------------+
| bar        |          0 |
+------------+------------+
1 row in set (0.005 sec)

ublubu

At a glance, the logic is reasonable.

But I'm having trouble understanding the comments.

ublubu · 2026-07-28T20:55:42Z

+        # 0 forces the estimated-size path for every table, the default forces
+        # the exact COUNT(*) path for workload-sized tables.


Is this comment saying that parallel_workload's tables all have <1M records?

The code here doesn't control either workload size or the default flag value.

They are going to be way smaller than 1 million records. My experience with parallel workload was that everything has to be extremely small, otherwise Materialize deadlocks quickly. There are a bunch of old issues open for that, but we can retry/revisit.

ublubu · 2026-07-28T20:58:11Z

+/// The largest optimizer-estimated table size for which the MySQL snapshot size
+/// gauge is computed with an exact `COUNT(*)`. Larger tables report the
+/// `information_schema` estimate directly, skipping the O(rows) count.


Suggested change

/// The largest optimizer-estimated table size for which the MySQL snapshot size

/// gauge is computed with an exact `COUNT(*)`. Larger tables report the

/// `information_schema` estimate directly, skipping the O(rows) count.

/// If the optimizer estimates the table has fewer rows than this, compute the exact row count with `COUNT(*)`.

/// Otherwise, report the `information_schema` estimate directly.

ublubu · 2026-07-28T21:05:33Z

+/// is O(total). Worker count is small, so the OFFSET scans dominate. `total` is the row
+/// count supplied by the caller. It can be an optimizer estimate for large tables, so
+/// the partitions are approximate. An overestimate walks off the end of the index and
+/// stops with fewer boundaries, an underestimate leaves a larger final partition, both
+/// still correctly partition the table. Returns None if the primary key column type is
+/// not supported or the table is too small to split.


I'm struggling with this comment.

In particular, I'm not sure about:

what the OFFSET scans are dominating

"this" in "so this is O(total)"

"walks off the end of the index and stops with fewer boundaries" (Also, that's a very long sentence.)

Nit: total is the row count ~~supplied by the caller~~

ublubu · 2026-07-28T21:07:36Z

+/// For every table, read the row count (exact only for small tables) and, for a
+/// supported single-column primary key, compute the PK-range split boundaries,
+/// concurrently over at most `worker_count` connections. `None` bounds means
+/// single-worker fallback for that table. The counts are reused for both the sampling
+/// stride and the snapshot size gauge.


This mostly reads clearly, but I can't tell what "snapshot size gauge" is. "Sampling stride" feels better, but I'm still not confident what it means either.

ublubu · 2026-07-28T21:08:45Z

-        .query_first(format!("SELECT COUNT(*) FROM {}", table))
+    // The optimizer's row estimate for the table. InnoDB keeps it roughly current
+    // (within the churn since the last stats recalculation), but it can be NULL or
+    // stale-at-zero, in which case we fall through to the exact count.


What is "stale-at-zero"?

peterdukelarsen marked this pull request as ready for review July 28, 2026 16:10

peterdukelarsen requested review from a team as code owners July 28, 2026 16:10

ublubu reviewed Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

storage: estimate MySQL snapshot row counts for large tables - #37907

storage: estimate MySQL snapshot row counts for large tables#37907
peterdukelarsen wants to merge 1 commit into
MaterializeInc:mainfrom
peterdukelarsen:pl/mysql-snapshot-estimate-count

peterdukelarsen commented Jul 28, 2026 •

edited

Loading

Uh oh!

peterdukelarsen commented Jul 28, 2026

Uh oh!

ublubu left a comment

Uh oh!

ublubu Jul 28, 2026

Uh oh!

def- Jul 28, 2026

Uh oh!

ublubu Jul 28, 2026

Uh oh!

ublubu Jul 28, 2026

Uh oh!

ublubu Jul 28, 2026

Uh oh!

ublubu Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		# 0 forces the estimated-size path for every table, the default forces
		# the exact COUNT(*) path for workload-sized tables.

Uh oh!

Conversation

peterdukelarsen commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Uh oh!

peterdukelarsen commented Jul 28, 2026

Uh oh!

ublubu left a comment

Choose a reason for hiding this comment

Uh oh!

ublubu Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

def- Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

ublubu Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

ublubu Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

ublubu Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

ublubu Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

peterdukelarsen commented Jul 28, 2026 •

edited

Loading