[Question] Why is HugeGraph's memory usage steadily increasing?

[cichengzi]

Problem Type (问题类型)

None

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• Server Version: 1.5.0 (Apache Release Version)
• Backend: RocksDB

Your Question (问题描述)

Why does memory usage increase as HugeGraph is used in RocksDB mode? During this process, we created new graphs, inserted vertices and edges, and executed various queries. When memory usage approaches its limit, the HugeGraph service becomes unavailable or even crashes; how can this be prevented?

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

[dosubot]

@cichengzi — this is essentially the same issue you raised in #3059, and the guidance there still applies. Here's a consolidated summary:

Why memory keeps growing:

1. Off-heap RocksDB allocations (memtables, block cache) are invisible to JVM GC. The default rocksdb.total_memory_size is 48GB, which alone can exceed your machine's RAM. Set it to a realistic value (e.g., 8–10GB on a 32GB machine). [1]

2. glibc allocator fragmentation — RocksDB's allocation pattern with the default ptmalloc allocator causes freed memory to never be returned to the OS. Replacing it with jemalloc is often the single most impactful fix [1]:

   export LD_PRELOAD=/path/to/libjemalloc.so

3. Transaction listener leak — repeatedly creating graphs accumulates orphaned event listeners in CachedGraphTransaction that are never cleaned up, causing persistent heap growth [1].

4. Accumulated task results — completed tasks leave ~task and ~taskresult vertices in the backend that get reloaded into memory and are never automatically cleaned up. Periodically delete old task results, and reduce task.result_size_limit [1].

5. Vertex/edge caches — HugeGraph's caches evict by capacity/TTL only, not by memory pressure, so they stay full even when heap is stressed. Reduce vertex.cache_capacity, edge.cache_capacity, and set a shorter cache_expire [1].

Key RocksDB settings to tune [1]:

Setting	Default	Recommendation
rocksdb.total_memory_size	48GB	8–10GB on 32GB machines
rocksdb.write_buffer_size	128MB	Reduce to 64MB
rocksdb.max_write_buffer_number	6	Reduce to 3

Diagnosis tip: Compare JVM heap (via jstat -gc) against total process RSS (/proc/<pid>/status). If heap is fine but RSS is very high, the problem is RocksDB native memory / allocator fragmentation. If heap itself is high, look at task results and caches first [1].

For large Gremlin queries specifically, avoid returning huge result sets through the task result mechanism — it's not designed as a bulk download API. Write results incrementally to external storage instead [1].

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[cichengzi]

Is there any way I can manually clear the memory without restarting the service?

[dosubot]

@cichengzi There's no single "clear all memory" button in HugeGraph, but here are several things you can do without restarting:

1. Trigger RocksDB compaction — This releases off-heap native memory pinned by obsolete SSTables after deletions or graph drops:

PUT http://{IP}:{PORT}/graphspaces/{graphspace}/graphs/{graph}/compact

[1]

2. Delete accumulated task results — Completed task results (~task/~taskresult vertices) are never automatically cleaned up and accumulate in the backend, consuming both heap and storage. Periodically delete completed/failed tasks via:

DELETE http://{IP}:{PORT}/graphspaces/{graphspace}/graphs/{graph}/tasks/{task_id}

This is especially important for tasks with large result sets. [2]

3. Force JVM GC via jcmd — You can trigger a garbage collection on the running process from the OS:

jcmd <PID> GC.run

Note: this only reclaims JVM heap garbage — it has no effect on RocksDB's off-heap native memory.

4. Tune RocksDB memory settings — Reducing the configured limits takes effect as current memtables flush, without a restart [2]:

Setting	Default	Action
rocksdb.total_memory_size	48GB	Reduce to match your actual available off-heap budget
rocksdb.max_write_buffer_number	6	Reduce to 3-4 to limit memtable memory
rocksdb.write_buffer_size	128MB	Reduce if memory constrained

Important context: Most memory growth in RocksDB mode is off-heap native memory (memtables, block cache), which is completely invisible to JVM GC [2]. There is no dedicated cache-flush API endpoint exposed — the internal CacheManager.clearCache() method exists [3] but is only triggered as a side effect of operations like truncation, not through a standalone REST call.

If memory fragmentation is a recurring issue, switching to jemalloc via LD_PRELOAD is often the single most impactful fix for the "memory never gets reclaimed" symptom [2].

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[cichengzi]

We currently have around 100 graphs, each with a scale of approximately 500,000 nodes and 1 million edges. When we start the HugeGraph service using the RocksDB backend, it consumes 55 GB of memory; this causes subsequent Gremlin queries to lag severely, rendering the system virtually unusable. What should we do?

[dosubot]

@cichengzi With 100 graphs on RocksDB, the memory problem is a multiplier effect: each graph opens its own RocksDB instance with multiple column families, and each column family gets its own write buffers and block cache. The code creates a new LRUCache(cacheCapacity) per column family in initTableConfig, and write buffer sizes are set per-CF via setWriteBufferSize and setMaxWriteBufferNumber [1]. Across 100 graphs with many CFs each, this adds up to enormous memory consumption even at modest per-CF defaults.

The most impactful tuning changes:

1. Reduce block cache memory — since a separate LRUCache is created for every column family [2], with 100 graphs you could have over 1000 independent caches. Set rocksdb.block_cache_capacity to a small value (e.g., 1–2MB per CF) or even 0 to disable it, since the total across all CFs will still be large.

2. Reduce write buffers per column family:

   rocksdb.write_buffer_size=16m
   rocksdb.max_write_buffer_number=2

   Also set rocksdb.db_write_buffer_size (mapped to setDbWriteBufferSize) to cap total memtable memory per RocksDB instance — this is a global limit across all CFs in a single DB and is critical for your scenario.

3. Use jemalloc to prevent glibc memory allocator fragmentation, which can inflate RSS well beyond what RocksDB actually needs [3]:

   export LD_PRELOAD=/path/to/libjemalloc.so

4. Reduce JVM heap — with RocksDB dominating off-heap memory, lower -Xmx (e.g., from 16GB to 8GB) to leave more physical RAM for native allocations.

5. Lower vertex/edge cache capacities in hugegraph.properties — these are heap-side caches managed by the CacheManager and they also multiply across 100 graphs [3].

The core constraint is that 100 simultaneously open RocksDB instances each with many column families has a very high fixed memory cost at startup. If many of these graphs are infrequently accessed, a lazy-load/unload pattern for inactive graphs would be the most effective long-term solution — though that would require code changes.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[imbajin]

We currently have around 100 graphs, each with a scale of approximately 500,000 nodes and 1 million edges. When we start the HugeGraph service using the RocksDB backend, it consumes 55 GB of memory; this causes subsequent Gremlin queries to lag severely, rendering the system virtually unusable. What should we do?

@cichengzi some suggestions:

1. use the latest 1.8.0 (master) branch (will be released in 2 weeks?)
2. enable the memory management feature (in-heap first)
3. use jemalloc instead of ptmalloc
4. use ToplingDB (RocksDB Plus) to get perf improvement (see refers)
5. use 1.8.0 distributed mode

Refer:

• https://hugegraph.apache.org/blog/2025/10/09/toplingdb-quick-start/
• https://github.com/hugegraph/hugegraph/tree/toplingdb (will be merged into master soon, welcome to test it)

[cichengzi]

rocksdb.write_buffer_size=16m
rocksdb.max_write_buffer_number=2

Are these two methods effective?