Skip to content

Compaction: eliminating disk scan and updating PQ retrain #691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dian-lun-lin wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Compaction: eliminating disk scan and updating PQ retrain #691

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions ...or-base/src/main/java/io/github/jbellis/jvector/graph/disk/OnDiskGraphIndexCompactor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1354,11 +1354,18 @@ private List<CommonHeader.LayerInfo> computeLayerInfoFromSources() {
int count = 0;
for (int s = 0; s < sources.size(); s++) {
if (level > sources.get(s).getMaxLevel()) continue;
NodesIterator it = sources.get(s).getNodes(level);
FixedBitSet alive = liveNodes.get(s);
while (it.hasNext()) {
int node = it.next();
if (alive.get(node)) count++;
if (level == 0) {
// Every live node is present at level 0 (HNSW base layer invariant),
// so count directly from the in-memory bitset instead of scanning node
// records on disk (which touches gigabytes of source data on a cold cache).
count += liveNodes.get(s).cardinality();
} else {
NodesIterator it = sources.get(s).getNodes(level);
FixedBitSet alive = liveNodes.get(s);
while (it.hasNext()) {
int node = it.next();
if (alive.get(node)) count++;
}
}
}
layerInfo.add(new CommonHeader.LayerInfo(count, maxDegrees.get(level)));
Expand Down
33 changes: 20 additions & 13 deletions jvector-base/src/main/java/io/github/jbellis/jvector/graph/disk/PQRetrainer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import io.github.jbellis.jvector.quantization.ProductQuantization;
import io.github.jbellis.jvector.util.DocIdSetIterator;
import io.github.jbellis.jvector.util.FixedBitSet;
import io.github.jbellis.jvector.util.PhysicalCoreExecutor;
import io.github.jbellis.jvector.vector.VectorizationProvider;
import io.github.jbellis.jvector.vector.VectorSimilarityFunction;
import io.github.jbellis.jvector.vector.types.VectorFloat;
Expand All @@ -32,6 +33,7 @@
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.ThreadLocalRandom;

/**
Expand Down Expand Up @@ -96,22 +98,27 @@ public ProductQuantization retrain(VectorSimilarityFunction similarityFunction,

log.info("Collected {} training samples", samples.size());

// Extract vectors sequentially in sorted (source, node) order so disk reads are
// purely sequential and the OS read-ahead can cover them efficiently. We do this
// here rather than letting ProductQuantization.compute() drive the reads via its
// parallel stream, which would scatter page faults across a potentially very large
// file and cause I/O that scales with dataset size rather than sample count.
List<VectorFloat<?>> trainingVectors = extractVectorsSequential(samples);
var ravv = new ListRandomAccessVectorValues(trainingVectors, dimension);

boolean center = similarityFunction == VectorSimilarityFunction.EUCLIDEAN;
long t0 = System.nanoTime();

@MarkWolters MarkWolters Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

initialization of t0 should occur prior to call to extractVectorsSequential as this is the time reported as being measured on line 104. As written this will always be ~0

log.info("Extracted {} vectors in {}ms; starting PQ refinement",
trainingVectors.size(), (System.nanoTime() - t0) / 1_000_000L);

var ravv = new ListRandomAccessVectorValues(trainingVectors, dimension);

return ProductQuantization.compute(
ravv,
basePQ.getSubspaceCount(),
basePQ.getClusterCount(),
center
);
// Warm-start from the existing codebook via Lloyd's-only refinement rather than
// re-running k-means++ from scratch. k-means++ initialization visits every point
// once per centroid (256 passes for k=256), which dominates training time.
// Since the source codebooks are already trained on data from the same underlying
// distribution, this warm-start converges in far fewer passes with no recall loss.
long t1 = System.nanoTime();
ProductQuantization result = basePQ.refine(ravv,
ProductQuantization.K_MEANS_ITERATIONS,
-1.0f, // UNWEIGHTED / isotropic
PhysicalCoreExecutor.pool(),
ForkJoinPool.commonPool());
log.info("PQ refinement complete in {}ms", (System.nanoTime() - t1) / 1_000_000L);
return result;
}

/**
Expand Down
2 changes: 1 addition & 1 deletion jvector-base/src/main/java/io/github/jbellis/jvector/quantization/ProductQuantization.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ public class ProductQuantization implements VectorCompressor<ByteSequence<?>>, A

private static final VectorTypeSupport vectorTypeSupport = VectorizationProvider.getInstance().getVectorTypeSupport();
static final int DEFAULT_CLUSTERS = 256; // number of clusters per subspace = one byte's worth
static final int K_MEANS_ITERATIONS = 6;
public static final int K_MEANS_ITERATIONS = 6;
public static final int MAX_PQ_TRAINING_SET_SIZE = 128000;

final VectorFloat<?>[] codebooks; // array of codebooks, where each codebook is a VectorFloat consisting of k contiguous subvectors each of length M
Expand Down
Loading