FindClusters in n dimensions, runnable reference pages, and the distance functions

[msollami]

Summary

Five commits, rebased onto current main. Three separable pieces of work:

1. FindClusters beyond one dimension — vectors, colours and strings — plus seven distance functions, none of which existed.
2. Reference pages gain Algorithm, Performance, See also and Options sections, all derived from content the repo already had and was silently discarding.
3. The notebook can open a reference page as a runnable notebook and look up the symbol under the cursor.

Changes

Kernel

• FindClusters accepts equal-length numeric vectors, colours (RGBColor, GrayLevel, Hue, CMYKColor) and strings. One dimension is unchanged by construction: the structure is now a spanning tree with exact edge weights, and on a line the MST is the sorted adjacency chain, so the general code reduces to the original algorithm there. The 132 existing acceptance rows were the gate and stayed green; 40 were added.
• Exactness survives in n dimensions by ranking on squared Euclidean distance — rational for rational input, monotone in the true distance, so no root is taken.
• A colour needs no colour-specific code: RGBColor[r, g, b] is a compound expression carrying numeric arguments, the same shape as a 3-vector. Point heads are nonetheless an explicit list, because Rational and Complex are also compound — reading any compound head as a point would cluster 1/2 by its numerator and denominator.
• New: EuclideanDistance, SquaredEuclideanDistance, ManhattanDistance, CosineDistance, EditDistance, HammingDistance. Names["*Distance*"] previously returned only GraphDistance, while DistanceFunction accepted these names as inert symbols — which is why the docs could truthfully claim all the choices behaved alike.

Two performance fixes, both found by measuring

• Explicit cluster count was quadratic. FindClusters[data, 10] at n=100,000 took 2.24 s against 0.087 s for Automatic on identical data, growing 4× per doubling. The union-find had no path compression, so few cuts left one long chain and every find walked it. Automatic hid it because many cuts leave many short components. 2.24 s → 0.083 s, linear again.

• Exact Prim allocated an Expr per distance even when the input was already machine-precision, where exact arithmetic preserves nothing. Machine-precision points now take a double Prim; Rational/bigint/MPFR keep the exact builder.

  2-D points	before	after
  500	95 ms	0.62 ms
  1000	398 ms	2.0 ms
  2000	1.49 s	6.8 ms

  219×, lifting the machine ceiling from 2,000 to 20,000 points (0.81 s there). Two builders for one definition is a standing risk, so a test clusters the same points both ways — as integers (machine) and divided by a constant (exact) — and requires matching partitions, including over 400 random points.

Documentation generator

Two silent bugs in the example pipeline:

• Graphics results were discarded. A Plot result arrives as {"type":"plot"}, not "expr", so the verifier recorded nothing and judged every graphics example unevaluated. Plot's page was left with the one example that trivially reproduces itself, Plot[Sin[x], {x, a, b}] — a signature, not a worked example.
• Multi-line examples were truncated at the line break, then fed to the binary as a syntax error and dropped.

Together: 1792 → 1900 verified examples.

The Features miner stopped at the first wrapped bullet. The spec files hard-wrap near 80 columns, so this truncated 168 of 305 sections — 3,351 lines, often mid-sentence. TrigFactor kept 2 lines of 42.

New sections, each derived from something that already existed: Algorithm (197 pages, from the implementation's own header comment), Performance (65, measured by the new tools/docs_perf.py plus benchmark-suite comparison), See also (467, from the spec's own groupings), Options & behaviour (60, spec prose that never reached a page). Examples are grouped with counts; graphics examples save their figure (18 MB → 1.6 MB after rounding and a 150 KB per-figure cap).

Notebook

• Cmd/Ctrl+click or Cmd+I on a symbol opens its reference page as a notebook: prose cells, foldable section cells, and every example a runnable code cell seeded with its recorded output. Graphics examples arrive drawn.
• ?pat* returns the Names[...] list instead of printing, so Length[?Find*] is 7.
• Fixes: light mode was broken app-wide (:root held the light palette while App.svelte assumed dark); reference pages were centre-aligned; two-finger pan died over a card being edited; closing a full-screen notebook left a dangling focusedId.

Testing

Full suite, both trees built fresh and run from tests/build:

	suites	failing
base	412	9
this branch	412	9

No new failures. The 9 are pre-existing (compiledfunction, eigen, iter, lapack_builtin, mateigen_direct, ndarray_linalg, simplify, singularvaluedecomposition, zero_test; two are segfaults).

Re-verified after the rebase onto current main: list_tests green, make check-c99 passes, exactness and the 219× both hold. Distance values cross-checked against wolframscript, including two conventions that are not derivable: CosineDistance of a zero vector is 0 rather than Indeterminate, and HammingDistance on unequal lengths stays unevaluated.

Known limitations, stated rather than discovered

• Only Automatic, "Agglomerate" and "SpanningTree" accept the new element kinds; the other seven methods read a sorted 1-D projection and decline.
• Caps: 20,000 machine points, 2,000 exact points or strings. Both paths are quadratic. One dimension has no cap and does 10^6 in ~2.3 s.
• Abs and Sign decline on rationals with bigint components (Abs[-1/10^20] does not evaluate). Pre-existing, not introduced here, but it blocked every exact high-precision distance; the distances route around it and the underlying gap deserves its own fix.
• 289 pages still have no verified examples: their spec sections contain none to mine.
• MeanShift costs 1.8 s at n=4,000, an order of magnitude above the rest of the density family — measured, not addressed.

[msollami]

Update: reference pages are now readable documents (0e324631, 351e19a0)

Two commits from a review-and-polish pass over the reference pages in the notebook.

Generator

• Prose worked examples become runnable. 108 pairs across 19 spec files read `Expr[...]` → `result` where the result is prose (√π/2, Γ[s]), not Mathilda syntax. Only the input is taken; the binary supplies the output. 1900 → 1976 verified examples, 25 pages gaining a Worked examples group.

  This immediately exposed a discrepancy the prose was hiding:

  Integrate[Exp[-a x] Sin[b x]/x, {x,0,Infinity}, Assumptions->a>0]
    spec says   ArcTan[b/a]
    build says  1/2 (Pi b)/Sqrt[b^2] - ArcTan[a/b]
  

  Same value for b > 0, but not simplified — a real gap, visible only because the example became executable.

• Per-example notes. An example carrying a (* ... *) comment renders it as a sentence above its own cell. Only 61 of 3814 examples have one; the rest are grouped as before, because inventing explanations would put made-up prose on a documentation page. Expand's overlay is the exemplar — its six "applications" were four variations of one binomial expansion, now six distinct cases each with its reason.

• Empty headings dropped. The spec marks example blocks with ### Examples; those blocks are stripped here (the examples are mined and re-verified into the Examples section), which left the heading behind. Integrate carried four empty ones interleaved with its real topics.

• H4 demoted to H3 (a page has two levels below its title); See also folded into References; Applications rendered through the same path as every other group.

Notebook

• Section folding, broken in two ways that only reference pages exposed, being the first notebooks with sibling subsections: the parent scan tested every subsection it passed rather than the nearest, so collapsing one group emptied the ones below it; and a subsection heading stopped at its sibling and never learned its section was collapsed, leaving a heading that could not be opened.

• Blank gaps: insertion strips were rendered outside the visibility guard (93px behind a collapsed Examples (26)), and punctuation-only segments became cells that rendered as nothing but occupied a row.

• Scroll chaining is gesture-aware. A flick that reached a card's bottom handed its momentum to the canvas and the view shot away. Events within 160 ms are one gesture; a gesture that scrolled a card cannot pan. A pause ends it, so scrolling again does pan.

• Cmd+I read getSelection().anchorNode and required a text node, but CodeMirror's caret anchor is usually the line element — it now uses the caret's screen rect and the same lookup Cmd+click uses.

• Reference pages behave as documents: non-editable headings that fold on click, no cell-type control, derived from content so it survives reload. Pages open beside the symbol and in front. Collapsible TOC after the definition; one button to collapse/expand all sections.

Testing

list_tests green, make check-c99 passes, svelte-check at its 2 pre-existing errors, 759 pages regenerated against this commit.