VortexMetrics Memory Growth: Children Accumulate Indefinitely in Long-Lived Sessions Resulting in OOM

[sgrebnov]

Issue Description

VortexMetrics accumulates child metrics indefinitely when VortexSession is reused across multiple queries, causing memory leaks in production workloads. The child_with_tags() method stores strong references to children that are never cleared, leading to memory growth in high-throughput scenarios.

Expected Behavior

No memory growth when VortexSession is re-used across multiple queries

Actual Behavior

Memory growth when VortexSession is re-used across multiple queries

Current behavior:

With temporary disabled metrics (memory is higher due to larger data)

pub fn child_with_tags(&self, additional_tags: impl Into<DefaultTags>) -> Self {
        let child = Self::new_with_tags(self.inner.default_tags.merge(&additional_tags.into()));
        // TEMPORARY: Disabled child tracking to prevent memory leak during queries
        // self.inner.children.write().push(child.clone());
        child

Reproduction Steps

Ownership chain:

VortexSession (Arc<SessionVars>)
  └─> DashMap<TypeId, Box<dyn SessionVar>>
      └─> Box<VortexMetrics> (root)
          └─> Arc<Inner>
              └─> children: RwLock<Vec<VortexMetrics>>  ← NEVER CLEARED

pub fn child_with_tags(&self, tags: impl IntoIterator<Item = (String, String)>) -> Self {
    let child = Self { inner: Arc::new(...) };
    self.inner.children.write().push(child.clone());  // ← Strong reference, never removed
    child
}

Each query creates many children (one per partition × files), but they're never cleared:

1. VortexSession is designed to be long-lived and reused
2. Children stay alive as long as root VortexMetrics exists
3. Root exists as long as any VortexSession clone exists

Questions / Notes

1. VortexSession to be per-query (new session each time) or long-lived?
2. Root cause has been identified using heaptrack, I can share a heaptrack heap profile if it helps.

OS Version Information

Ubuntu 24.04

I acknowledge that:

• I have searched the Vortex repository (both open and closed Discussions and Issues) and confirm this is not a duplicate of an existing issue or discussion.
• I have checked the "Preview" tab on all text fields to ensure that everything looks right, and have wrapped all configuration and code in code blocks with a group of three backticks (```) on separate lines.

[AdamGS]

This might be the nudge I needed to look deeper into metrics, this is a really good report!

[AdamGS]

@sgrebnov can you try #6187 out? I wonder if it (together with some other previous changes) makes a significant difference

[AdamGS]

From my testing, this issue has been solved, with the final piece being the work attached to #6078