Improve site search: replace the search index with Pagefind - #3371
Merged
Conversation
Closed
enf0rc3
force-pushed
the
willlaugesen/docs-search-pagefind
branch
from
August 17, 2026 21:23
5ac3f67 to
6cbd53f
Compare
enf0rc3
force-pushed
the
willlaugesen/docs-search-pagefind
branch
3 times, most recently
from
August 17, 2026 22:38
0d065ca to
1c011db
Compare
|
Pull request environment is available at https://stoctodocspr3371.z22.web.core.windows.net. You can view the ephemeral environment status in Octopus Deploy. This environment will be automatically deprovisioned when the pull request is closed, or after 7 days of inactivity. |
Contributor
Author
Verified on the ephemeral environment
Every Pagefind file type serves. Search is correct on real hosting. Result links stay on the host. Zero absolute hrefs; clicking a result landed on One thing to know about the hostingAzure Blob static website does no compression — Bake-off result
Both feel the same on a fast connection. The difference is what happens below that, and payload was the defect this work set out to fix. Recommend merging this and closing #3372. |
enf0rc3
force-pushed
the
willlaugesen/docs-search-pagefind
branch
5 times, most recently
from
August 18, 2026 20:59
d9d2279 to
49885ce
Compare
enf0rc3
force-pushed
the
willlaugesen/docs-search-pagefind
branch
2 times, most recently
from
August 20, 2026 03:06
febe16b to
f03a873
Compare
enf0rc3
commented
Aug 20, 2026
|
|
||
| // Gates the header's search field only. The overlay below is always rendered: | ||
| // /docs/search has its own field and opens that same overlay. | ||
| const showSearch = !isSearchPage; |
Contributor
Author
There was a problem hiding this comment.
This can be removed if / when we remove the search page (no longer needed imo)
enf0rc3
commented
Aug 20, 2026
| return item; | ||
| } | ||
|
|
||
| function drawSection( |
Contributor
Author
There was a problem hiding this comment.
Essentially to allow scrolling to pull more results
enf0rc3
changed the title
Contributor
|
Haven't done a review yet but just started to play on the staging site, and I apparently found 1 API result yet no results found, what might be going on here? 
|
borland
approved these changes
Aug 20, 2026
Contributor
There was a problem hiding this comment.
In terms of code, it all looks like claude and it all seems fine from the skim-review I gave it.
In terms of experience, I had a play on the staging site, and the search results are noticeably better. Nice stuff.
Performance-wise, the fragmented indexing definitely resulted in less data being transferred, however it seems like the index is hyper-fragmented into heaps of tiny files.
I did a test using the MS Edge dev tools to set the network speed to "Slow 4G", and pagefind was noticeably slower:
I hard-refreshed the front docs page, clicked on the search box, and typed "kubernetes"
-
The old search on octopus.com loads the 1.8 MB search.json (compressed to 415kb) in 2.9 seconds. This was enough that I noticed the delay; I finished typing the word and still had to wait for the results to appear, but it wasn't too bad.
-
The new search in the staging site only loads 95kb of indexes, and they're already compressed, so that's a huge win. However it took over 5 seconds for results to appear, feeling noticeably slower. It loaded 32 fragments, each of which was about 1-2 kilobytes, and it seemed like only about 6 would download at any one time; this concurrency-limiting behaviour seemed to be the cause of the slowness.
Claude tells me that the pf_fragment files aren't the search results, but rather the preview fragment data for each page that the search finds, so we can't chunk or merge them, but we can reduce the number we download by tweaking the number of visible results rows we fetch. The UI only shows Five results on screen. We want to over-fetch a little bit to allow for scrolling, but maybe 10 would be a better number than 32.
In saying that, maybe it was just down to the staging site lacking Azure Front Door, or maybe it was an artifact of the browser network throttling tooling. I'm going to take an educated guess that it's probably fine and say let's merge it, but we might need to think about tuning how many results we fetch or something.
Contributor
Author
|
Thanks for the review: Breaking down the timings, I put a majority of the delay with the server "waiting for server response" so it will improve on production. We can definitely fine-tune this more. Prod
Staging
I will do a quick look to see if there are any quick wins or changes we can't make to improve this. |
Indexes the built HTML at the end of the build and puts Pagefind behind the SearchEngine seam. Body text is indexed for the first time, so a phrase that appears in an article but not its title or headings is now findable. The index goes to dist/docs/pagefind, before pruneDist would delete it, and is built from dist/docs so stored URLs match the /docs/ prefix the client sets as its basePath. data-pagefind-body on the article is what scopes indexing and, as a side effect, excludes all 1,409 redirect stubs: they render through a minimal layout that has no article at all. 1,250 pages are indexed, matching the eligible count the markdown emitter reports. Removes the search.json endpoint and the client scoring it fed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three configuration corrections from the Pagefind audit. Its indexing and ranking were already idiomatic, so none of this touches either. preload() now runs on each keystroke ahead of the debounced search, which is what Pagefind documents as the way to fetch the chunks a query needs while the reader is still typing. Warming moves to page load for Pagefind only. That is 118KB of runtime and WASM against a cold first result measured at 2239ms. Which side of that trade is right differs by engine, so `SearchEngine` carries an `eager` flag and the engine states its own answer rather than the overlay assuming one — Orama pays for its whole index on every navigation and must stay lazy. excerptLength drops from its 30-word default to 20, to fit the single line the result row gives it. Sub-results are deliberately not used. They render heading-scoped matches with their own anchors, which is real, but the corpus fights it: the CLI tree has 511 headings across only 49 distinct titles, 213 of them "Learn more" and 176 "Examples", and the API tree 291 across 100. Only documentation pages are distinctive, at 4160 titles across 5302 headings. Suppressing the boilerplate needs a threshold nobody has justified yet, and the audit is clear that sub-results move neither Success@1 nor Success@5 — so this stays out of a comparison it cannot decide, and can be added later against whichever engine wins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six capabilities that were available and unused. Indexing. A depth-derived data-pagefind-weight on the page header, so a bare section name ranks the section above the pages inside it — the largest single source of missed traffic on the search terms readers actually type. Image alt text becomes searchable via data-pagefind-index-attrs, which has no inheritance and so needs a hast plugin to reach every markdown image. A date sort key is indexed, and a frontmatter title is registered as fallback metadata for any page whose heading comes back empty. Querying. highlightParam so result links carry the query for the destination page to highlight, and sub_results rendered as their own options beneath their page, so a match inside a long page can be arrowed onto and lands on that heading. The weight is the one worth measuring: applied as a client-side reorder it took Pagefind from 42% to 62% weighted Success@5 on real search terms, and from 73% to 95% on top-visited pages. At index time it has no top-30 reach limit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Built and measured locally, which corrected two things. data-pagefind-weight on its own did nothing. Scaling it with depth scored 42% weighted Success@5 on real search terms against the deployed 42%, because plus or minus a point around Pagefind's default h1 weight of 7 is far too weak a lever. It now boosts shallow pages to 10 and leaves deep pages at the default, and the reordering that does the work lives in the engine: the same name-then-depth rule the Orama worker uses, with the same constant. Three matched headings across thirty results made two thirds of the list headings, which buried the pages a reader is choosing between. Only the leading three results carry them now, two each — 15% of rows rather than 67%. Real search terms go 42% to 62% Success@5 and 19% to 43% at rank one. Top-visited pages go 73% to 95% and 37% to 82%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three gaps, all measured on a local build.
Pagefind strips punctuation from index and query alike, so `<head>` matched nothing
and `Octopus.Action.Package` was indistinguishable from prose using those words.
includeCharacters keeps `.#{}<>+$_`, chosen from what appears in this corpus and in
the search logs. Indexing is additive — both `head` and `<head>` are stored — so
ordinary searches are untouched and the index grew 1%, 8.8MB to 8.9MB. `<head>`
went from 0 results to 55, and `octopus.server.exe` now leads with the
octopus.server.exe reference page.
metaWeights sets title to 8 and trail to 0.5. The breadcrumb is derived from the
URL, so every page under /docs/projects/ carries "Projects" and matched a search
for it as well as the Projects page did. It belongs in the result row, not the
ranking.
Both ignore blocks move to `all`. Bare ignore still lets a title or metadata be
read out of the block, and neither holds anything worth reading.
Real search terms: 62% to 81% Success@5, 43% to 52% at rank one.
Top-visited pages: 95% to 98%, and 82% to 89%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test read /docs/search.json, which this branch deletes along with the rest of the legacy engine, so it was parsing a 404 page as JSON. It now drives the search overlay instead. The two spikes ship indexes of different shapes — one JSON document, one directory of compressed chunks — and neither is readable the way the old one was, but what has to hold is the same either way: a reader searching a word the API reference is full of must not be sent into it. `accounts` names pages both inside the API reference and outside it, so the result list is never empty. A query that matched nothing would pass without proving anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two changes, both measured on a local build. Searches now need three characters. Almost the whole of Pagefind's keystroke lag lives in the two keystrokes nobody could act on anyway: one character costs 975ms and two cost 837ms, against 342ms at three and under 200ms beyond. On a desktop this takes p95 from 927ms to 347ms, and the median from 346ms to 200ms. A search whose best match scores under 8 now returns nothing. Pagefind has no notion of a query it cannot answer — asked for `sssieddqxsx`, the most-typed term in the log, it offered three security articles. Calibrated against every logged term: the weakest genuine query scores 9.5, the mash 6.0. The floor suppresses 17 of the 33 terms with no right answer and costs none of the 57 that have one. Relevance is unchanged either way: 81% Success@5 on real terms and 98% on top-visited pages, the same as before. Nonsense returning nothing went from 0% to 76%. The floor was first set at 6, which did nothing, because it was calibrated while earlier sweep parameters were still active — `options()` merges rather than replaces, and every score came back depressed. calibrate-score-floor.mjs starts from a clean page for that reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The overlay carried the combobox roles without the one part that speaks. A screen reader was told a listbox existed and which row was active, and never how many results arrived or that none had. The live region sits outside the body, which is hidden until there is a query. Inside it, the region would leave and re-enter the accessibility tree on every open, and a region that has only just appeared does not reliably announce the first thing written to it. It is clipped rather than hidden for the same reason: `display: none` would silence it. It speaks from `render`, which runs once per settled query, so it keeps pace with the results rather than with the keyboard. Below the three character minimum it says nothing, matching the panel's own waiting state. Pagefind's own Component UI does this and ours did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
They pointed at the bake-off and at the other candidate engine, and both are gone from the tree. The rationale each comment carried is kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- preload ignored MIN_QUERY_LENGTH, so the first keystrokes still fetched the chunks the guard exists to avoid. - A failed fragment fetch rejected the whole search and left the previous query's rows under the new text. - The score floor was applied to the filtered results while the tab counts came from the unfiltered ones, so a tab could advertise results and then show none. It is now asked of the unfiltered scores. - highlightParam was set and then stripped off every link, so it is dropped. - The minimum query length goes to two characters. - A missing fragment directory threw instead of reaching the page-count warning that exists to report it. - Future-dated pages are indexed, and the comment claiming otherwise is fixed. - The live region counted pages while the listbox held pages and sections. - Removed an unreachable filter, logged the swallowed index-load error, and corrected the deleted call site named in underConstruction.ts. pagefind moves to a real dependency: astro.config.mjs imports it statically, so a production-only install could not load the config at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removed the running commentary on approach, the measurements that only justified a decision at the time, and a reference to a calibration script that is not in the repo. What is left explains Pagefind behaviour and coupling that the code does not show on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous pass cut too far in four places. Restored: what each character in `includeCharacters` is for, the evidence behind the score floor and the `options()` merge that skews a recalibration run, and which of the two section constants is which. The comment on the API reference test claimed the search term was what stopped an empty list passing; the visibility assertion is. `hashOf` is gone. `URL.hash` already returned the same string, so the anchor is read off the URL that is being parsed anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- One failed fragment fetch took the whole result set with it. allSettled drops the row it belongs to and keeps the rest. - Asking the score floor of the unfiltered corpus fixed a false negative and opened a false positive: a tab could hold nothing but accidents scoring far below the query and still render them. The floor now answers both questions — nothing anywhere returns empty, nothing under this tab returns no rows while keeping the counts that say where the answers are. - The two searches run together, so the second no longer sits in front of every fragment fetch. - An index with no pages in it fails the build instead of warning. Every query returning nothing must not ship green. - The live region counts pages the way the tab strip does, and names the matched sections separately. - components.mdx still told readers production used legacyEngine over search.json. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plausible.astro has listened for a `searched` event since 2023, but the page
that fired it was `public/docs/js/search.js`, deleted when the site's JS moved
into src/. There has been no search analytics since.
The overlay now dispatches the same event, with the same shape: a CustomEvent on
document carrying `{ search }`, which Plausible.astro turns into a `Search` goal.
Reported once per distinct settled query, after the results render, so the count
matches answered queries rather than keystrokes. The showcase overlay is
excluded because its results are a fixture.
Asserted in the suite rather than described, since this contract broke once
without anything failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page-count warning fired on a hardcoded number that could not detect the failure its own message named, and the empty-index throw duplicated what the search test already covers before deploy. Errors from Pagefind now fail the build rather than logging and continuing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`rootSelector` and `excludeSelectors` on `createIndex` decide what is indexed for the whole site, so a layout no longer has to opt in. Api.astro carried no Pagefind attributes and its pages were being dropped; the under construction exclusion that hid them is now an explicit check. The per-page attributes that remain live in one helper. Two additions on top. A second search over the shallow pages puts the page a query names first, which recovers section landing pages ranked below the fetch window. The result list keeps the stubs it was discarding and extends as it is scrolled, the way Pagefind's own UI pages. Measured with tools/search-bakeoff against the deployed branch: top-pages w-S@1 89 to 90 per cent and w-S@5 98 to 99, real-searches level, latency unchanged within run to run spread. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects the overlay shipped with. A word on nearly every page scores near zero, because BM25's IDF term collapses when a term is everywhere: `octopus` is on 1177 of 1251 pages and scores 0.87 against a floor of 8. Suppressing the rows is right and calling it "no results" was wrong, so a query matching most of the corpus now asks the reader to narrow it. A keyboard mash keeps the plain message; Pagefind scores partial matches, so the mashes reach 74 per cent of the corpus and the real words start at 79. Three CLI pages carry the slug `create-release`, so the rule that promotes a page the query names handed `create release` to one of them. A reference page now has to be asked for before it claims that, and is discounted 1.3x otherwise. Asking means the query names the api or cli, or opens with `octopus `, which is how every CLI page is titled. Measured on the traffic-weighted sets: real-searches w-S@1 52 to 57 per cent, w-S@5 81 to 84, MRR 0.644 to 0.689; top-pages holds at 90 per cent. The only top-pages term it loses is `create release` itself, whose expected page is recorded as the CLI one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`namedPage` only runs when nothing on the first page names the query, so the page it promotes always ranks past PAGE_SIZE — and its own stub was still sitting in the list for `more()` to draw a second time. Searching "deployment targets" and scrolling once listed /docs/infrastructure/deployment-targets/ twice. The row ids are handed out by position so they stayed unique, which is why the paging test missed it; the new test asserts on urls and uses a query that promotes a page. `total` counted the promoted page twice, since it was already among the stubs, which overstated the announced result count. `page` was assigned with no per-search guard. The three searches run together, so two overlapping queries can settle out of order and the caller's generation check can only discard the stale rows, leaving `more()` appending the previous query's. Separately, `close()` now runs in a `finally`: the three throw sites above it left the Pagefind binary running, so a failed index reported the error and then hung. Found by code review. Two comments corrected in passing: the page count logged by the integration says nothing about a layout missing its article, and dropping `PostFiltering.showInSearch` also dropped its `draft` and `listable` exclusions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Searching kubernetes and picking the API tab read "API (1)" on the strip and "No results for kubernetes" in the panel. The one API page scored 6.59 against a floor of 8, so the rows were suppressed while the count kept advertising them: counts come from Pagefind in one pass and take no notice of the floor. The floor now runs once, against the whole corpus, which is the question it answers. A tab with nothing in it was never the floor's job — its count already sends the reader back to All. The new test walks every tab the strip offers for one query and requires rows behind each, so any tab advertising a number it cannot show fails. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@borland Given this PR is already so big, i will do a follow up PR for further tweaking to the search algo, I have implemented a fix for the issue @rosslovas raised above, so that will come with this merge. |
enf0rc3
force-pushed
the
willlaugesen/docs-search-pagefind
branch
from
August 20, 2026 22:31
1be84c4 to
75eeb4d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The current search has always been "lackluster" and often would give you completely irrelevent search results.
We want to improve this by using a library that specialises in searching on static sites and gives us the ability to tweak it.
This is a type of task that can have a lot of iteration, so i think getting something out that is better than the current should be the goal.
Results:
Site now uses Pagefind. It indexes the HTML the build emits, and does some smarts and ranking to better rank the content we want, we did tweak it but found the defaults and headings were suffice.
AI Generated Details of the pr
<head>,.nupkgand#{Octopus.Environment.Name}are findable as written.searchedeventPlausible.astrohas been waiting for now fires, so what readers type is finally measurable.How this was measured
Relevance was tested by replaying the site's own search log — the terms readers actually typed, weighted by how many typed each one — plus a second set derived from page analytics covering 12,558 visitors. The harness types into the real overlay in a browser, so what it measures is what a reader gets.
Against production, the top five answers 84% of real searches, where the current engine answers 81%. Latency is unchanged.
The harness is
tools/search-bakeoff, kept so the comparison can be re-run.