Spike: Orama - #3372
Conversation
bed6fe0 to
b4a37d7
Compare
bbe6232 to
452365d
Compare
|
Pull request environment is available at https://stoctodocspr3372.z22.web.core.windows.net. You can view the ephemeral environment status in Octopus Deploy. This environment will be automatically deprovisioned when the pull request is closed, or after 7 days of inactivity. |
Correction and staging resultsThe 2.4 MB gzipped figure in the description is not achievable on this hosting. Azure Blob static website hosting does no on-the-fly compression — requesting Nothing on that host is compressed, Pagefind's assets included. So the real wire cost for Orama is 11.4 MB, not 2.4 MB. Getting the compressed figure would need the It works, and on a fast connection it feels fine
That is the honest picture at ~47 Mbps: the download is 2 seconds and you would not notice. The gap is entirely about what happens below that.
Download time only; arithmetic from the measured wire sizes, not measured directly. RecommendationClose this in favour of #3371. Payload was always the deciding axis and this makes it worse, not better — every visitor who searches pays 11.4 MB uncompressed, against 1.86 MB today. Typo tolerance, the one thing that could have justified the cost, returns nothing for Worth keeping from this spike if Pagefind is ever revisited: reusing |
e4870cd to
344c86c
Compare
Builds an Orama index from the markdown llm-md-emitter already writes into dist/docs, and puts it behind the SearchEngine seam. Body text is indexed for the first time, so a phrase that appears in an article but not its title or headings is now findable. Reusing the emitted markdown means there is no second extraction pipeline: the emitter runs the same eligibility predicate search uses, so redirect stubs and navSearch:false pages are already gone. 1,253 pages indexed. The index is restored in a Web Worker. Restore is CPU-bound and would otherwise freeze the page for as long as it takes, with the overlay open and taking keystrokes. Uses Orama core save/load rather than @orama/plugin-data-persistence, which reaches for Node's filesystem and buffers and fails to bundle for a browser worker. Removes the search.json endpoint and the client scoring it fed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The worker created its database with only a schema, so query terms were tokenized without the English stemmer the index was built with. Searching `variables` matched 5 pages instead of 321, and typo tolerance matched nothing at all. Search kept returning results throughout, so nothing failed loudly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes to what the index carries, none of which change what it can find. Nothing in the engine sorts, but Orama builds and serializes a sort store for every sortable field unless it is turned off. That was 2.69MB. Each document was stored whole, so 2.2MB of page text shipped to every visitor purely so the excerpt could be cut from it client-side. The inverted index is a separate structure and keeps every term, so trimming the stored copy to 200 characters costs no recall. The excerpt window is 180. Orama defaults stopWords to an empty list, so `how`, `do`, `the` and 177 others were live search terms. The list ships separately as @orama/stopwords. Measured against the deployed index this is worth 80% to 87% on natural-language queries. The index goes 11.4MB to 5.3MB raw, and 1.32MB to 0.66MB brotli. That is below the predicted 6.53MB because the stop-word list shrinks the index as well, which had not been measured before. sort and the tokenizer are set in both the integration and the worker, for the same reason: the worker restores into a database it creates itself, so anything the index was built with has to be declared on both sides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The engine asked the worker for 200 hits and counted sections over them, so every count saturated: a search for `variables` showed All (200) when the real total is 321. It also filtered those 200 client-side, so a section whose matches fell outside them came back short. The worker now asks for facets, which report the whole match set, and applies the selected section as a `where` clause so the filtered list is complete. Counts come from an unfiltered search on purpose. A `where` clause narrows the facet values to the section being filtered on, and the strip has to keep showing what the other tabs hold. Verified against the built index: All (321), Docs (290), API (19), CLI (12), and selecting CLI returns all 12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
excerptFrom preferred the description outright, so any page with a subtitle showed its subtitle even when the match that earned it its rank was in the body. It now picks whichever field the query actually hit, falling back as before. BODY_LIMIT stays at 4,000, now as a measured decision rather than an assumption. Scored against the bake-off query set with a local build: 2,000 4.28MB Success@5 78% intent 87% 4,000 5.33MB Success@5 78% intent 87% 20,000 6.63MB Success@5 76% intent 80% Raising it is worse and bigger — the extra text dilutes the terms that identify a page, which is the opposite of what the audit expected. Lowering it scores the same and saves 1.05MB, but narrows what is findable on long pages to buy bytes the payload budget does not need: brotli is 0.66MB against a 0.81MB gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
26% of top-5 rows rendered with a blank excerpt — 127 of 482 across the query set. Two things compounded, and the first was mine: `?? hit.description` treats an empty string as a value worth keeping, so it won the fallback and blocked `hit.body`. 1,200 of the 1,254 documents have an empty description and none have an empty body, so that path was taken almost every time it was reached. It was reached far more often than it looks, because the index matches stemmed terms and the excerpt matches literally. "Guided failures" legitimately ranks a page whose text only ever says "guides" — both stem to "guid" — and the literal regex then finds nothing, falling straight into the trap above. `||` instead, with body first. When nothing matches literally the window opens at the start of the page, which is what a reader wants from a stemmed match; it just arrives unhighlighted. Trimming the stored body to 200 characters did not cause this. At 4,000 a literal match usually turned up somewhere by luck, so it hid the bug. Measured against the same 102 queries: blank rows 127/482 to 0/482, and queries with a blank in the top five 50/102 to 0/102. The highlightable rate is unchanged at 72%, which is the point — this replaces blanks with real text rather than inventing highlights for stemmed matches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`warm()` only ran when the overlay opened, so the whole index was fetched and parsed while the reader waited. It now starts on the first pointer or focus reaching a search field, which costs nothing for the majority who never go near it. `SearchEngine` carries the policy rather than the overlay assuming one, because the right answer differs by engine: an engine whose runtime is small enough can warm on page load, and this one cannot — 5.3MB spent on every visitor, searcher or not. Ctrl/Cmd+K goes straight to open, which still warms, so the policy only decides how much earlier it can start. Measured against a local build on one server, so caching is held constant: 778ms to first result without the hover, 646ms with. The saving is smaller than the load cost because the load is not what dominates — roughly 500ms of that 646ms is the query itself, which is worth looking at separately now that facet counts are computed over the whole match set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Main published 106 generated pages under docs/api and deliberately kept them out of search and the sitemap until the section has a landing page. That check lived in search.json.ts, which this branch deletes, so it had to move with the index rather than be lost with it. It does not come along on its own. The Orama index is built from what llm-md-emitter writes, and the emitter's eligibility check knows nothing about the section — those pages carry a layout and no `navSitemap: false`, so they pass. The emitter now writes 1,360 files where it wrote 1,254, and all 106 of the difference would have been searchable. Indexed count is back to 1,254, with nothing under /docs/api/ in the index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
91035c7 to
eece801
Compare
BM25 has no notion of a site's shape, so `variables` returned projects/variables/system-variables and `tentacle` returned tentacle/linux: a child page repeats the term more often in less text. On the search terms readers actually type this was the largest single source of missed traffic. Two changes. The title boost goes from 4 to 8, worth 26 points of Success@5 on its own, since most searches are one or two words naming a page. Then hits are reordered so a page the query names outright comes first, and everything else is scored with a penalty per path segment. Measured against three query sets — real search terms, top-visited pages, and a curated set — rather than only the one it was tuned on. On real search terms Success@5 goes 54% to 80% and rank one 17% to 51%. Deep pages still rank first when they are what was asked for. A tiebreak on depth was tried first and did nothing: BM25 scores almost never tie, so depth has to scale the score. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test read /docs/search.json, which this branch deletes along with the rest of the legacy engine, so it was parsing a 404 page as JSON. It now drives the search overlay instead. The two spikes ship indexes of different shapes — one JSON document, one directory of compressed chunks — and neither is readable the way the old one was, but what has to hold is the same either way: a reader searching a word the API reference is full of must not be sent into it. `accounts` names pages both inside the API reference and outside it, so the result list is never empty. A query that matched nothing would pass without proving anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Closing unmerged. We are going with Pagefind (#3371). Relevance did not decide it. On the real search log the two engines are level — 81% against 80% in the top five, and identical at rank one. On the terms that carry the traffic Pagefind is ahead, 98% against 90%. Payload and cold start decided it. Orama ships 0.66 MB on the wire and 5.33 MB parsed against Pagefind's 0.08 MB and 0.11 MB, and every visitor pays that on every page load whether they open search or not. First search is 3950 ms against 1821 ms, and Orama needs about five seconds of reading time before it is ready. Behind both sits the 8 MB compression cliff: Front Door only compresses between 1 KB and 8 MB, so a corpus that grows past it would silently lose compression and take the cold start back to about 5.7 s. What we are giving up is real. Typo tolerance drops from 67% to 42%, and top-five on a partial query from 72% to 57%. An as-you-type overlay spends most of its life on partial queries, so that is the price of the decision rather than a rounding error. Worth revisiting if search analytics later show typo and partial queries are a bigger share of traffic than the log suggests. The branch stays for reference. The updated description above has the full numbers and the configuration notes — the tokenizer mismatch in particular is the kind of thing that fails silently and would cost someone a day. |
|
Archived, so this survives the branch being deleted.
To re-run the comparison later, start at Worth re-running when there is real search analytics. Pagefind was chosen knowing it is worse at typos (42% against 67%) and partial queries (57% against 72%), and nobody had data on how much traffic those account for. |
2 participants
Spike 2 of 2 for the search engine bake-off. Branched off #3369, so that PR's commit is in this diff. Pagefind is #3371.
Closing unmerged. Pagefind won the bake-off — the comparison is at the bottom of this description. The branch stays for reference.
What it does
Replaces the hand-rolled
search.jsonengine with Orama, behind theSearchEngineseam from #3369. Body text is searchable for the first time.The corpus is the markdown
llm-md-emitteralready writes intodist/docs/, so there is no second extraction pipeline to keep in step — the emitter runs the same eligibility predicate search uses, so redirect stubs andnavSearch: falsepages are gone before this sees them.The whole index is downloaded and restored into memory once, in a worker, and every query after that costs nothing. Restore is CPU-bound and would freeze the page while the overlay is open, which is why it is off the main thread.
Deletes 645 lines of custom search code: a Porter stemmer, a synonym map, the string helpers and the legacy engine.
Results
Measured through the search overlay against expected URLs written before each run. Two query sets, both from analytics and both weighted by traffic.
Orama's real strength is the shape of the queries it handles, which is where an as-you-type overlay lives:
Once the index is in memory Orama is the fastest thing measured. Getting it there is the problem: it needs about five seconds of reading time before the first search is ready.
Configuration that matters
Four settings that are easy to get wrong and were, at first:
loadrestores the data but the tokenizer comes fromcreate, so building with the English stemmer and restoring without it compares unstemmed query terms against stemmed index terms. Searchingvariablesmatched 5 pages instead of 321 and no typo matched at all, while search kept returning results throughout.stopWordsto an empty list, sohow,doand 178 others were live search terms. The list ships separately as@orama/stopwords.sort: { enabled: false }— nothing here sorts, and the sort store was 2.69 MB of the index.save(). The inverted index keeps every term; this is only the copy the result excerpt is cut from.Together the last two took the index from 11.43 MB to 5.33 MB with rankings unchanged. Ranking was then tuned separately:
boost: { title: 8, ... }, and the same landing-page reorder Pagefind needed, because a page ranking below its own children was a defect in every engine tested.Why Pagefind was chosen
Relevance did not decide it. On the real search log the two are level — 81% against 80% in the top five, and identical at rank one. On the terms that carry the traffic Pagefind is clearly ahead, 98% against 90%.
Three things decided it:
What was given up is real and worth recording: typo tolerance drops from 67% to 42%, and top-five on a partial query from 72% to 57%. An as-you-type overlay spends most of its life on partial queries, so this is the cost of the decision rather than a rounding error.
Notes
.mdxpages are missing from the index. All of them import.astrocomponents, which the markdown emitter cannot resolve. Not fixed here.@orama/plugin-data-persistencecannot be bundled for a browser worker — it reaches for Node's filesystem and buffers, and the worker dies on import withClass extends value undefined. Coresave/loaddo the same job and are what this uses.facetsrather than being counted from returned hits, which capped them at the worker's result limit.Plausible.astrostill listens for asearchedevent that nothing fires.🤖 Generated with Claude Code