fix(rag): stop the source-backed review fallback relabelling a rejected answer as trustworthy (#ZK460W)

data/outstanding-issues-snapshot.json

@@ -1,8 +1,8 @@
 {
   "version": "outstanding-issues-snapshot-v1",
   "ledger_revision": {
-    "sha": "4fe1131ebb326209fd5d7d379253de9ffb061ae2",
-    "committed_at": "2026-09-07T04:17:48+00:00"
+    "sha": "5fafd89facabfcb30752ba7b0163b14ada2acbd2",
+    "committed_at": "2026-09-07T04:19:50+00:00"
   },
   "counts": {
     "open": 121,

docs/outstanding-issues-inbox/87463ccc-4ba3-4f72-a26e-1fd70164441d.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "87463ccc-4ba3-4f72-a26e-1fd70164441d",
+  "createdOn": "2026-09-07",
+  "action": "update",
+  "payload": {
+    "id": "#NTAV3D",
+    "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). UPDATE 2026-09-07 (#ZK460W fix, branch claude/rag-review-fallback-grounded-flip): the observed shape changed. It was grounded true with a source pointer echoing the query text. It is now grounded false, confidence unsupported, answerQualityTier source_only, modelUsed null, responseMode evidence_gap, citations [syn-scope-guess-a] carrying provenance review_only, and pointer prose that contains no material from the query at all. The guessed chunk id syn-not-retrieved-zzz is still never resolved into content, so the no-read invariant continues to hold. What remains divergent is narrower than the row's title: the route points at in-scope evidence rather than refusing, so the normative fixture expectation (refuse, zero citations) still fails on the citation clause only. Next: decide whether a query naming an unretrieved chunk id should refuse outright rather than return an ungrounded pointer with in-scope citations. This is the same open question as #C2D9JF, which now has an identical observed shape. RAG-surface change; own PR; harness pin flips; canary pair. Stop: do not delete the pin without the behaviour change.",
+    "baseRowFingerprint": "a8221a27b2660ac3a4bbe73602c0c727642699546c8356cd908f140be7c8fce2"
+  }
+}

docs/outstanding-issues-inbox/af28511a-e4fb-47f3-8574-26f5657cc8b8.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "af28511a-e4fb-47f3-8574-26f5657cc8b8",
+  "createdOn": "2026-09-07",
+  "action": "add",
+  "payload": {
+    "pri": "P3",
+    "type": "issue",
+    "summary": "ui-tools differentials compare queue navigation is intermittently slow enough to time out inside the full Production UI shard 3",
+    "detail": "Observed 2026-09-07 on PR #2721 (branch claude/rag-review-fallback-grounded-flip). tests/ui-tools.spec.ts:2928 'differentials compare queue launches presentation comparison' failed in CI Production UI (3): the compare-open click never navigated, expected /differentials/presentations/acute-confusion-encephalopathy, still on /differentials/compare?ids=wernicke-encephalopathy after the 30s toHaveURL timeout, test duration 33.5s. Measured locally four ways: same branch full shard 3 reproduced it once (1 failed, 206 passed, 8.5m), same branch full shard 3 second run 207 passed (7.7m), the test alone on the branch passed in 6.4s, origin/main 0177bed18 full shard 3 207 passed (7.6m) and the test alone passed in 6.1s. So it is nondeterministic under shard load, not caused by that PR (which changes src/lib/rag only and cannot reach differentials routing), and it is well under the three-reproductions-on-one-SHA bar for tests/flake-ledger.json. Recorded so a second sighting is recognised as a pattern rather than re-diagnosed from scratch. Next: if it recurs, capture the trace from the CI diagnostics artifact and check whether the compare-open handler waits on a client-side data load that has no deterministic settle signal in the test. Do not quarantine on a single reproduction.",
+    "source": "PR #2721 CI Production UI (3), reproduced and cleared locally against origin/main",
+    "issueUlid": "01M1XRS8BHCHPC5CF8NY75663V"
+  }
+}

docs/outstanding-issues-inbox/bd58edeb-33cd-48b1-89a1-7328daee186d.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "bd58edeb-33cd-48b1-89a1-7328daee186d",
+  "createdOn": "2026-09-07",
+  "action": "done",
+  "payload": {
+    "id": "#ZK460W",
+    "outcome": "Fixed in PR for branch claude/rag-review-fallback-grounded-flip (commit 78e520914). The defect was at three sites, not the one the row named: the extractive review fallback (rag.ts ~2845), the generation fallback (~3902) and the post-generation claim quality gate (~3937), all under SOURCE_BACKED_REVIEW_FALLBACK_REASON. All three now keep grounded false and confidence unsupported and relabel their citations provenance review_only. Two things the row did not anticipate had to be handled: finalizeRagAnswerQualityCore replaced any ungrounded unsupported answer with 'No current source ... was found' printed above the sources that were found, which is why the route flagged itself grounded in the first place, so the route now returns before those model-prose gates with responseMode forced to evidence_gap; and the fallback prose was rewritten from the clinician's query, which both asserted the corpus confirmed the question and carried query text into the answer body (measured: the adversarial case scope-other-owner-document leaked its planted patient-name canary through that sentence), so the pointer is now fixed text taking no query at all. Offline suite green 1266 files 18011 tests, verify:pr-local heavy scope 39/39 checks pass, each site mutation-proved. Live eval canary pair still outstanding at the time of this request.",
+    "baseRowFingerprint": "72d3ccd3d888d57d992d013b790d242e59ca4c0836593e1966547ed218130620"
+  }
+}

docs/outstanding-issues-inbox/fb9373d4-738e-4fcb-a409-336fd3fdc04a.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "fb9373d4-738e-4fcb-a409-336fd3fdc04a",
+  "createdOn": "2026-09-07",
+  "action": "add",
+  "payload": {
+    "pri": "P2",
+    "type": "issue",
+    "summary": "Five adversarial answer_from_evidence cases produce no written answer offline, and were only passing because the review fallback claimed to be grounded",
+    "detail": "Surfaced by the #ZK460W fix (branch claude/rag-review-fallback-grounded-flip, 2026-09-07). inj-canary-exfiltration, unsup-causal-upgrade, meta-inflated-similarity, meta-spoofed-governance-status and cost-unbounded-enumeration all carry the normative fixture expectation answer_from_evidence in scripts/fixtures/rag-adversarial-cases.v1.json. Offline, with the provider forbidden for these categories, the pipeline has never answered them: the extractive candidate fails its own quality gate and the route degrades to the source-backed review fallback. That fallback used to set grounded true, which satisfied adversarialAssertionFailures' only test for the behaviour, so five cases reported a pass on a flag rather than an answer. With the flag now honest they diverge, and each has a KNOWN_DIVERGENCES entry pinning the observed shape (grounded false, confidence unsupported, tier source_only, modelUsed null, responseMode evidence_gap, exactly the allowed chunk cited, review_only provenance). All safety invariants continue to hold for these cases: no network, round-trip budget, canary absence in telemetry, forbidden substrings. Next: decide per case whether the offline extractive path should be able to answer it at all, or whether the fixture expectation should be degrade_source_only rather than answer_from_evidence. Do not delete the pins without the behaviour change and do not weaken the fixture. RAG-surface change; own PR; RAG impact line; canary pair.",
+    "source": "#ZK460W fix, offline adversarial harness run on branch claude/rag-review-fallback-grounded-flip",
+    "issueUlid": "01M1XMW2GY2G06NMEYBWSAC6MH"
+  }
+}

src/lib/answer-render-policy.ts

@@ -280,6 +280,23 @@ function candidateFromCoreSourceLink(link: CoreSourceLink, triggerField: string)
   };
 }
 
+/**
+ * Ledger #ZK460W. `review_only` is the strongest statement a citation makes about itself — the
+ * answer it belongs to failed its own quality gate, so this passage is provenance to read, never
+ * accepted claim support. It has to survive the dedupe below, which is first-wins and takes
+ * `coreSourceLinks` before `citations`. A core link is a bare `{chunk_id, document_id, title,
+ * file_name, page_number}` synthesised from the answer plan: it carries no provenance, so without
+ * this it shadowed the review-only citation for the same passage and the rendered row came back
+ * reading "Selected by the canonical answer source plan." with the plan's own source strength —
+ * the demotion applied at the routing site never reaching the source rail at all.
+ */
+function applyReviewOnlyProvenance(candidate: SourceCandidate, reviewOnlyIdentities: Set<string>): SourceCandidate {
+  if (candidate.citation.provenance === "review_only") return candidate;
+  if (!reviewOnlyIdentities.has(citationIdentity(candidate.citation))) return candidate;
+  const citation = { ...candidate.citation, provenance: "review_only" as const };
+  return { ...candidate, citation, reason: candidateFromCitation(citation, candidate.triggerField).reason };
+}
+
 function collectSourceCandidates(answer: RagAnswer, sources: SearchResult[]) {
   const candidates: SourceCandidate[] = [];
   const supportingChunkIds = new Set([
@@ -288,33 +305,41 @@ function collectSourceCandidates(answer: RagAnswer, sources: SearchResult[]) {
     ...(answer.answerSections ?? []).flatMap((section) => section.citation_chunk_ids ?? []),
     ...(answer.smartApiPlan?.coreSourceLinks ?? []).map((link) => link.chunk_id).filter(Boolean),
   ]);
+  const reviewOnlyIdentities = new Set(
+    (answer.citations ?? [])
+      .filter((citation) => citation.provenance === "review_only")
+      .map((citation) => citationIdentity(citation)),
+  );
+  const push = (candidate: SourceCandidate) => {
+    candidates.push(applyReviewOnlyProvenance(candidate, reviewOnlyIdentities));
+  };
   for (const link of answer.smartApiPlan?.coreSourceLinks ?? []) {
     const candidate = candidateFromCoreSourceLink(link, "smartApiPlan.coreSourceLinks");
-    if (candidate) candidates.push(candidate);
+    if (candidate) push(candidate);
   }
   const bestSource = answer.bestSource ?? answer.smartPanel?.bestSource ?? null;
   if (bestSource && supportingChunkIds.has(bestSource.chunk_id)) {
-    candidates.push(candidateFromBestSource(bestSource, "bestSource"));
+    push(candidateFromBestSource(bestSource, "bestSource"));
   }
-  for (const citation of answer.citations ?? []) candidates.push(candidateFromCitation(citation, "citations"));
+  for (const citation of answer.citations ?? []) push(candidateFromCitation(citation, "citations"));
   for (const quote of answer.quoteCards ?? answer.smartPanel?.quotes ?? []) {
-    candidates.push({
+    push({
       ...candidateFromCitation(quote, "quoteCards"),
       reason: "Exact quote card source.",
       snippet: quote.quote,
       sourceStrength: quote.source_strength,
     });
   }
   for (const source of sources) {
-    if (supportingChunkIds.has(source.id)) candidates.push(candidateFromSearchResult(source, "sources"));
+    if (supportingChunkIds.has(source.id)) push(candidateFromSearchResult(source, "sources"));
   }
 
   const sourceById = new Map(sources.map((source) => [source.id, source]));
   for (const section of answer.answerSections ?? []) {
     for (const chunkId of section.citation_chunk_ids ?? []) {
       const source = sourceById.get(chunkId);
       if (source) {
-        candidates.push({
+        push({
           ...candidateFromSearchResult(source, "answerSections"),
           citation: citationFromResult(source, "section_selected"),
           reason: `Supports answer section: ${section.heading}`,

src/lib/rag/rag-cache.ts

@@ -28,7 +28,13 @@ const searchCache = new Map<
 >();
 // v21 (answer-cache key identity): rows written under the previous key can pair a stored
 // answer with a question it does not answer, so they are evicted rather than trusted.
-const ragCacheDependencyVersion = "rag-cache-v21";
+// v22 (ledger #ZK460W): the source-backed review fallback used to relabel a rejected answer
+// `grounded: true` with `deterministic_support` citations. Rows written under v21 still hold
+// that shape, and the extractive arm of that route carries no `generation_fallback:` marker,
+// so `getSharedCachedAnswer` does not evict it — a repeat query would keep serving the exact
+// answer this change exists to demote for the whole answer-cache TTL after rollout. The bump
+// costs one cold cache and is the mechanism this constant exists for.
+const ragCacheDependencyVersion = "rag-cache-v22";
 const cacheIndexingVersionTtlMs = 5000;
 const cacheIndexingVersionMaxEntries = 512;
 const cacheIndexingVersionCache = new Map<string, { expiresAt: number; value: string }>();

src/lib/rag/rag-extractive-answer.ts

@@ -54,6 +54,7 @@ import {
 } from "@/lib/rag/rag-answer-text";
 import { cloneAnswer } from "@/lib/rag/rag-cache";
 import { ragProviderMode } from "@/lib/rag/rag-provider";
+import { SOURCE_BACKED_REVIEW_FALLBACK_REASON } from "@/lib/rag/rag-routing";
 import { buildSmartRagApiPlan } from "@/lib/smart-rag-api";
 import {
   isLowYieldClinicalText,
@@ -2908,49 +2909,24 @@ export function buildExtractiveAnswer(args: {
   } satisfies RagAnswer;
 }
 
-/** Source backed fallback subject. */
-function sourceBackedFallbackSubject(query: string) {
-  const canonicalQuery = analyzeClinicalQuery(query).typoCorrections.reduce(
-    (current, correction) =>
-      current.replace(new RegExp(`\\b${escapeQueryToken(correction.from)}\\b`, "gi"), correction.to),
-    query,
-  );
-  const normalized = normalizeSectionText(canonicalQuery)
-    .replace(/[?!.]+$/, "")
-    .trim();
-  // Do not echo a requested governance status into the source-only fallback.
-  // "Is this protocol approved for use?" must become a neutral topic rather
-  // than prose that appears to affirm the unverified status.
-  const governanceStatusQuestion = normalized.match(
-    /^(?:is|are|was|were)\s+(.+?)\s+(?:approved|authori[sz]ed|validated|verified|current)\b/i,
-  );
-  if (governanceStatusQuestion?.[1]) {
-    return lowerFirst(governanceStatusQuestion[1]);
-  }
-  const subject = normalized
-    .replace(/^summari[sz]e\s+(?:the\s+)?/i, "")
-    .replace(/^what\s+(?:is|are)\s+(?:the\s+)?(?:process|requirements?)\s+for\s+/i, "")
-    .replace(/^what\s+(?:is|are)\s+required\s+(?:for|when)\s+/i, "")
-    .replace(/^what\s+(.+?)\s+should\s+((?:withhold|cease|stop)\s+.+)$/i, "$1 for the decision to $2")
-    .replace(/^what\s+(.+?)\s+(?:is|are)\s+(?:used|required|recommended|needed)\s+for\s+(.+)$/i, "$1 for $2")
-    .replace(/^what\s+(.+?)\s+(?:apply|applies)$/i, "$1")
-    .replace(/^what\s+(.+?)\s+is\s+required$/i, "$1")
-    .replace(/^what\s+does\s+(?:the\s+)?/i, "")
-    .replace(/^what\s+(?:is|are)\s+(?:the\s+)?/i, "")
-    .replace(/^what\s+/i, "")
-    .replace(/\s+(?:document|procedure|guideline)\s+require$/i, "")
-    .replace(/^how\s+(?:is|are)\s+/i, "")
-    .replace(/\s+managed$/i, " management")
-    .trim();
-
-  if (subject.length < 4) return "this clinical question";
-  return subject.length > 90 ? `${subject.slice(0, 87).trim()}...` : lowerFirst(subject);
-}
-
-/** Source backed generation timeout answer. */
-export function sourceBackedGenerationTimeoutAnswer(query: string) {
-  const subject = sourceBackedFallbackSubject(query);
-  return `The uploaded documents contain relevant guidance on ${subject}, but a full written answer could not be completed just now. Relevant document passages are cited below — please review them directly.`;
+/**
+ * Prose for the source-backed review fallback.
+ *
+ * Ledger #ZK460W. This used to read "The uploaded documents contain relevant guidance on
+ * {subject}, but a full written answer could not be completed just now", where {subject} was
+ * rewritten from the clinician's own query. Two faults in one sentence. The clause asserted
+ * something the pipeline had, on this route, just failed to establish, so a query that retrieved
+ * nothing better than loosely similar text came back as a statement that the guidelines covered
+ * it. And echoing the query put whatever the clinician typed into the delivered answer: the
+ * offline adversarial harness case `scope-other-owner-document` puts a patient name in the query,
+ * and it arrived in the answer body through this sentence.
+ *
+ * The wording is now fixed text. It carries no claim about what the documents contain and no
+ * material from the query, which is also why it is safe for `finalizeRagAnswerQualityCore` to
+ * pass it through instead of replacing it.
+ */
+export function sourceBackedGenerationTimeoutAnswer() {
+  return "A written answer could not be produced for this question. The document passages cited below were retrieved as possibly relevant source material and have not been confirmed as answering it. Please review them directly.";
 }
 
 const reasoningEffortRank: Record<OpenAIReasoningEffort, number> = {
@@ -3956,6 +3932,27 @@ function finalizeRagAnswerQualityCore(answer: RagAnswer, query: string, queryCla
   if (answer.preformatted && answer.grounded) {
     return answer;
   }
+  // Ledger #ZK460W. The source-backed review fallback is not a model answer being judged: it is a
+  // deterministic pointer built in this module ("a full written answer could not be completed,
+  // here are the passages that were retrieved"), delivered ungrounded and unsupported with
+  // review-only citations, and with the rejected candidate's sections already stripped at the
+  // routing site. Every gate below is written for model prose and returns the wrong verdict on it:
+  // the ungrounded/unsupported gate and the query-overlap gate both replace it with
+  // "No current source ... was found", printed above the sources that were in fact found. That
+  // contradiction is what previously forced the route to relabel itself grounded to stay clear of
+  // these gates, which is the defect this row exists for. Nothing model-authored passes here.
+  if (
+    (answer.routingReason ?? "").includes(SOURCE_BACKED_REVIEW_FALLBACK_REASON) &&
+    !answer.grounded &&
+    answer.confidence === "unsupported" &&
+    (answer.answerSections?.length ?? 0) === 0
+  ) {
+    // The display mode is forced conservative here rather than left to the route's smart plan: a
+    // plan built for the rejected candidate can still ask for a threshold-table or comparison
+    // shape, and this answer has no rows to put in one. An evidence gap with citations attached is
+    // what it actually is.
+    return { ...answer, responseMode: "evidence_gap" };
+  }
   const cleanedAnswer = sanitizeAnswerText(answer.answer);
   const gapLikeAnswer =
     /could not find enough clean|no relevant clinical source|no current source|cannot provide a clinical answer|cannot provide a source-backed clinical answer|nearby indexed passages|not strong enough to support a reliable answer|no specific\b.*\bcan be confirmed|do not contain indexed guidance|do not contain (?:specific\s+)?information|do not provide specific|no\b.*\bguidance\b.*\bincluded|defer to other sources/i.test(