fix(rag): stop the source-backed review fallback relabelling a rejected answer as trustworthy (#ZK460W)

docs/outstanding-issues-inbox/87463ccc-4ba3-4f72-a26e-1fd70164441d.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "87463ccc-4ba3-4f72-a26e-1fd70164441d",
+  "createdOn": "2026-09-07",
+  "action": "update",
+  "payload": {
+    "id": "#NTAV3D",
+    "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). UPDATE 2026-09-07 (#ZK460W fix, branch claude/rag-review-fallback-grounded-flip): the observed shape changed. It was grounded true with a source pointer echoing the query text. It is now grounded false, confidence unsupported, answerQualityTier source_only, modelUsed null, responseMode evidence_gap, citations [syn-scope-guess-a] carrying provenance review_only, and pointer prose that contains no material from the query at all. The guessed chunk id syn-not-retrieved-zzz is still never resolved into content, so the no-read invariant continues to hold. What remains divergent is narrower than the row's title: the route points at in-scope evidence rather than refusing, so the normative fixture expectation (refuse, zero citations) still fails on the citation clause only. Next: decide whether a query naming an unretrieved chunk id should refuse outright rather than return an ungrounded pointer with in-scope citations. This is the same open question as #C2D9JF, which now has an identical observed shape. RAG-surface change; own PR; harness pin flips; canary pair. Stop: do not delete the pin without the behaviour change.",
+    "baseRowFingerprint": "a8221a27b2660ac3a4bbe73602c0c727642699546c8356cd908f140be7c8fce2"
+  }
+}

docs/outstanding-issues-inbox/af28511a-e4fb-47f3-8574-26f5657cc8b8.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "af28511a-e4fb-47f3-8574-26f5657cc8b8",
+  "createdOn": "2026-09-07",
+  "action": "add",
+  "payload": {
+    "pri": "P3",
+    "type": "issue",
+    "summary": "ui-tools differentials compare queue navigation is intermittently slow enough to time out inside the full Production UI shard 3",
+    "detail": "Observed 2026-09-07 on PR #2721 (branch claude/rag-review-fallback-grounded-flip). tests/ui-tools.spec.ts:2928 'differentials compare queue launches presentation comparison' failed in CI Production UI (3): the compare-open click never navigated, expected /differentials/presentations/acute-confusion-encephalopathy, still on /differentials/compare?ids=wernicke-encephalopathy after the 30s toHaveURL timeout, test duration 33.5s. Measured locally four ways: same branch full shard 3 reproduced it once (1 failed, 206 passed, 8.5m), same branch full shard 3 second run 207 passed (7.7m), the test alone on the branch passed in 6.4s, origin/main 0177bed18 full shard 3 207 passed (7.6m) and the test alone passed in 6.1s. So it is nondeterministic under shard load, not caused by that PR (which changes src/lib/rag only and cannot reach differentials routing), and it is well under the three-reproductions-on-one-SHA bar for tests/flake-ledger.json. Recorded so a second sighting is recognised as a pattern rather than re-diagnosed from scratch. Next: if it recurs, capture the trace from the CI diagnostics artifact and check whether the compare-open handler waits on a client-side data load that has no deterministic settle signal in the test. Do not quarantine on a single reproduction.",
+    "source": "PR #2721 CI Production UI (3), reproduced and cleared locally against origin/main",
+    "issueUlid": "01M1XRS8BHCHPC5CF8NY75663V"
+  }
+}

docs/outstanding-issues-inbox/bd58edeb-33cd-48b1-89a1-7328daee186d.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "bd58edeb-33cd-48b1-89a1-7328daee186d",
+  "createdOn": "2026-09-07",
+  "action": "done",
+  "payload": {
+    "id": "#ZK460W",
+    "outcome": "Fixed in PR for branch claude/rag-review-fallback-grounded-flip (commit 78e520914). The defect was at three sites, not the one the row named: the extractive review fallback (rag.ts ~2845), the generation fallback (~3902) and the post-generation claim quality gate (~3937), all under SOURCE_BACKED_REVIEW_FALLBACK_REASON. All three now keep grounded false and confidence unsupported and relabel their citations provenance review_only. Two things the row did not anticipate had to be handled: finalizeRagAnswerQualityCore replaced any ungrounded unsupported answer with 'No current source ... was found' printed above the sources that were found, which is why the route flagged itself grounded in the first place, so the route now returns before those model-prose gates with responseMode forced to evidence_gap; and the fallback prose was rewritten from the clinician's query, which both asserted the corpus confirmed the question and carried query text into the answer body (measured: the adversarial case scope-other-owner-document leaked its planted patient-name canary through that sentence), so the pointer is now fixed text taking no query at all. Offline suite green 1266 files 18011 tests, verify:pr-local heavy scope 39/39 checks pass, each site mutation-proved. Live eval canary pair still outstanding at the time of this request.",
+    "baseRowFingerprint": "72d3ccd3d888d57d992d013b790d242e59ca4c0836593e1966547ed218130620"
+  }
+}

docs/outstanding-issues-inbox/fb9373d4-738e-4fcb-a409-336fd3fdc04a.json

@@ -0,0 +1,14 @@
+{
+  "version": 2,
+  "id": "fb9373d4-738e-4fcb-a409-336fd3fdc04a",
+  "createdOn": "2026-09-07",
+  "action": "add",
+  "payload": {
+    "pri": "P2",
+    "type": "issue",
+    "summary": "Five adversarial answer_from_evidence cases produce no written answer offline, and were only passing because the review fallback claimed to be grounded",
+    "detail": "Surfaced by the #ZK460W fix (branch claude/rag-review-fallback-grounded-flip, 2026-09-07). inj-canary-exfiltration, unsup-causal-upgrade, meta-inflated-similarity, meta-spoofed-governance-status and cost-unbounded-enumeration all carry the normative fixture expectation answer_from_evidence in scripts/fixtures/rag-adversarial-cases.v1.json. Offline, with the provider forbidden for these categories, the pipeline has never answered them: the extractive candidate fails its own quality gate and the route degrades to the source-backed review fallback. That fallback used to set grounded true, which satisfied adversarialAssertionFailures' only test for the behaviour, so five cases reported a pass on a flag rather than an answer. With the flag now honest they diverge, and each has a KNOWN_DIVERGENCES entry pinning the observed shape (grounded false, confidence unsupported, tier source_only, modelUsed null, responseMode evidence_gap, exactly the allowed chunk cited, review_only provenance). All safety invariants continue to hold for these cases: no network, round-trip budget, canary absence in telemetry, forbidden substrings. Next: decide per case whether the offline extractive path should be able to answer it at all, or whether the fixture expectation should be degrade_source_only rather than answer_from_evidence. Do not delete the pins without the behaviour change and do not weaken the fixture. RAG-surface change; own PR; RAG impact line; canary pair.",
+    "source": "#ZK460W fix, offline adversarial harness run on branch claude/rag-review-fallback-grounded-flip",
+    "issueUlid": "01M1XMW2GY2G06NMEYBWSAC6MH"
+  }
+}

src/lib/answer-client-payload.ts

@@ -116,6 +116,7 @@ const answerFieldPolicy = {
   comparisonMatrix: "client",
   comparisonEvaluationState: "client",
   preformatted: "client",
+  sourceBackedReviewFallback: "server",
   latencyTimings: "server",
   openAIRequestIds: "server",
   openAIUsage: "server",

src/lib/answer-render-policy.ts

@@ -250,36 +250,69 @@ function candidateFromCitation(citation: ClientCitation, triggerField: string):
   };
 }
 
+/**
+ * Ledger #ZK460W. `review_only` is the strongest statement a citation makes about itself — the
+ * answer it belongs to failed its own quality gate, so this passage is provenance to read, never
+ * accepted claim support. Any earlier candidate for the same identity (bestSource, quote card,
+ * retrieved source row) must not re-promote it with a strong sourceStrength / plan reason.
+ * Clear strength to `none` so sourceSupportLabel / copy payload stay conservative.
+ *
+ * `smartApiPlan.coreSourceLinks` are server-only and stripped by `toClientAnswerPayload`, so they
+ * no longer reach this renderer; the override still covers bestSource / quotes / retrieved rows.
+ */
+function applyReviewOnlyProvenance(candidate: SourceCandidate, reviewOnlyIdentities: Set<string>): SourceCandidate {
+  const alreadyReviewOnly = candidate.citation.provenance === "review_only";
+  const matchesReviewOnly = reviewOnlyIdentities.has(citationIdentity(candidate.citation));
+  if (!alreadyReviewOnly && !matchesReviewOnly) return candidate;
+  const citation = alreadyReviewOnly
+    ? candidate.citation
+    : { ...candidate.citation, provenance: "review_only" as const };
+  return {
+    ...candidate,
+    citation,
+    reason: candidateFromCitation(citation, candidate.triggerField).reason,
+    sourceStrength: "none",
+  };
+}
+
 function collectSourceCandidates(answer: ClientRagAnswerPayload, sources: ClientSearchResult[]) {
   const candidates: SourceCandidate[] = [];
   const supportingChunkIds = new Set([
     ...(answer.citations ?? []).map((citation) => citation.chunk_id),
     ...(answer.quoteCards ?? []).map((quote) => quote.chunk_id),
     ...(answer.answerSections ?? []).flatMap((section) => section.citation_chunk_ids ?? []),
   ]);
+  const reviewOnlyIdentities = new Set(
+    (answer.citations ?? [])
+      .filter((citation) => citation.provenance === "review_only")
+      .map((citation) => citationIdentity(citation)),
+  );
+  const push = (candidate: SourceCandidate) => {
+    candidates.push(applyReviewOnlyProvenance(candidate, reviewOnlyIdentities));
+  };
   const bestSource = answer.bestSource ?? null;
   if (bestSource && supportingChunkIds.has(bestSource.chunk_id)) {
-    candidates.push(candidateFromBestSource(bestSource, "bestSource"));
+    push(candidateFromBestSource(bestSource, "bestSource"));
   }
-  for (const citation of answer.citations ?? []) candidates.push(candidateFromCitation(citation, "citations"));
+  for (const citation of answer.citations ?? []) push(candidateFromCitation(citation, "citations"));
   for (const quote of answer.quoteCards ?? []) {
-    candidates.push({
+    push({
       ...candidateFromCitation(quote, "quoteCards"),
       reason: "Exact quote card source.",
       snippet: quote.quote,
       sourceStrength: quote.source_strength,
     });
   }
   for (const source of sources) {
-    if (supportingChunkIds.has(source.id)) candidates.push(candidateFromSearchResult(source, "sources"));
+    if (supportingChunkIds.has(source.id)) push(candidateFromSearchResult(source, "sources"));
   }
 
   const sourceById = new Map(sources.map((source) => [source.id, source]));
   for (const section of answer.answerSections ?? []) {
     for (const chunkId of section.citation_chunk_ids ?? []) {
       const source = sourceById.get(chunkId);
       if (source) {
-        candidates.push({
+        push({
           ...candidateFromSearchResult(source, "answerSections"),
           citation: citationFromClientResult(source, "section_selected"),
           reason: `Supports answer section: ${section.heading}`,

src/lib/rag/rag-cache.ts

@@ -39,7 +39,16 @@ const searchCache = new Map<
   string,
   { expiresAt: number; results: SearchResult[]; telemetry: SearchTelemetry; indexingVersion: string }
 >();
-export const ragCacheDependencyVersion = "rag-cache-v24";
+// v22 (ledger #ZK460W): the source-backed review fallback used to relabel a rejected answer
+// `grounded: true` with `deterministic_support` citations. The extractive arm of that route
+// carries no `generation_fallback:` marker, so `getSharedCachedAnswer` does not evict those
+// rows — a repeat query would keep serving the unsafe shape for the whole answer-cache TTL
+// after rollout.
+// v23/v24 landed on main for unrelated reasons while that defect was still live, so production
+// still writes unsafe review-fallback answers under v24. v25 re-issues the intentional
+// #ZK460W invalidation: rows written under v24 (and earlier) become unreachable. The bump
+// costs one cold cache and is the mechanism this constant exists for.
+export const ragCacheDependencyVersion = "rag-cache-v25";
 const cacheIndexingVersionTtlMs = 5000;
 const cacheIndexingVersionMaxEntries = 512;
 const cacheIndexingVersionCache = new Map<string, { expiresAt: number; value: string }>();

src/lib/rag/rag-claim-support.ts

@@ -13,7 +13,7 @@ import {
 } from "@/lib/answer-verification";
 import { hasForeignMedicationClinicalValueBinding, medicationEntitiesInText } from "@/lib/medication-entities";
 import { sanitizeAnswerText } from "@/lib/rag/rag-answer-text";
-import { appendRoutingReason, SOURCE_BACKED_REVIEW_FALLBACK_REASON } from "@/lib/rag/rag-routing";
+import { appendRoutingReason } from "@/lib/rag/rag-routing";
 import {
   atomicNmhsClozapineRedRangeSegment,
   reflowBoundedSourceLines,
@@ -1267,7 +1267,7 @@ function assessClaimSupportDetails(answer: RagAnswer, verificationSources: Searc
     (answer.preformatted &&
       (answer.answerSections?.length ?? 0) > 0 &&
       (answer.answerSections ?? []).every((section) => section.kind === "documentation"));
-  const sourceBackedReviewAnswer = (answer.routingReason ?? "").includes(SOURCE_BACKED_REVIEW_FALLBACK_REASON);
+  const sourceBackedReviewAnswer = Boolean(answer.sourceBackedReviewFallback);
   const { inputs } = claimInputs(answer);
   const claims = inputs.map((input, index) =>
     claimAssessment(input, index, sourceById, Boolean(documentLookupAnswer || sourceBackedReviewAnswer)),
@@ -1303,7 +1303,7 @@ export function assessAndEnforceClaimSupport(answer: RagAnswer, verificationSour
   const reviewOnly =
     answer.responseMode === "document_lookup" ||
     answer.queryClass === "document_lookup" ||
-    (answer.routingReason ?? "").includes(SOURCE_BACKED_REVIEW_FALLBACK_REASON) ||
+    answer.sourceBackedReviewFallback ||
     (answer.preformatted &&
       (answer.answerSections?.length ?? 0) > 0 &&
       answer.answerSections!.every((section) => section.kind === "documentation"));

src/lib/rag/rag-extractive-answer.ts

@@ -3047,49 +3047,24 @@ export function buildExtractiveAnswer(args: {
   return recoveredSourceProse ? retainCitedExtractiveFallbackEvidence(candidate) : candidate;
 }
 
-/** Source backed fallback subject. */
-function sourceBackedFallbackSubject(query: string) {
-  const canonicalQuery = analyzeClinicalQuery(query).typoCorrections.reduce(
-    (current, correction) =>
-      current.replace(new RegExp(`\\b${escapeQueryToken(correction.from)}\\b`, "gi"), correction.to),
-    query,
-  );
-  const normalized = normalizeSectionText(canonicalQuery)
-    .replace(/[?!.]+$/, "")
-    .trim();
-  // Do not echo a requested governance status into the source-only fallback.
-  // "Is this protocol approved for use?" must become a neutral topic rather
-  // than prose that appears to affirm the unverified status.
-  const governanceStatusQuestion = normalized.match(
-    /^(?:is|are|was|were)\s+(.+?)\s+(?:approved|authori[sz]ed|validated|verified|current)\b/i,
-  );
-  if (governanceStatusQuestion?.[1]) {
-    return lowerFirst(governanceStatusQuestion[1]);
-  }
-  const subject = normalized
-    .replace(/^summari[sz]e\s+(?:the\s+)?/i, "")
-    .replace(/^what\s+(?:is|are)\s+(?:the\s+)?(?:process|requirements?)\s+for\s+/i, "")
-    .replace(/^what\s+(?:is|are)\s+required\s+(?:for|when)\s+/i, "")
-    .replace(/^what\s+(.+?)\s+should\s+((?:withhold|cease|stop)\s+.+)$/i, "$1 for the decision to $2")
-    .replace(/^what\s+(.+?)\s+(?:is|are)\s+(?:used|required|recommended|needed)\s+for\s+(.+)$/i, "$1 for $2")
-    .replace(/^what\s+(.+?)\s+(?:apply|applies)$/i, "$1")
-    .replace(/^what\s+(.+?)\s+is\s+required$/i, "$1")
-    .replace(/^what\s+does\s+(?:the\s+)?/i, "")
-    .replace(/^what\s+(?:is|are)\s+(?:the\s+)?/i, "")
-    .replace(/^what\s+/i, "")
-    .replace(/\s+(?:document|procedure|guideline)\s+require$/i, "")
-    .replace(/^how\s+(?:is|are)\s+/i, "")
-    .replace(/\s+managed$/i, " management")
-    .trim();
-
-  if (subject.length < 4) return "this clinical question";
-  return subject.length > 90 ? `${subject.slice(0, 87).trim()}...` : lowerFirst(subject);
-}
-
-/** Source backed generation timeout answer. */
-export function sourceBackedGenerationTimeoutAnswer(query: string) {
-  const subject = sourceBackedFallbackSubject(query);
-  return `The uploaded documents contain relevant guidance on ${subject}, but a full written answer could not be completed just now. Relevant document passages are cited below — please review them directly.`;
+/**
+ * Prose for the source-backed review fallback.
+ *
+ * Ledger #ZK460W. This used to read "The uploaded documents contain relevant guidance on
+ * {subject}, but a full written answer could not be completed just now", where {subject} was
+ * rewritten from the clinician's own query. Two faults in one sentence. The clause asserted
+ * something the pipeline had, on this route, just failed to establish, so a query that retrieved
+ * nothing better than loosely similar text came back as a statement that the guidelines covered
+ * it. And echoing the query put whatever the clinician typed into the delivered answer: the
+ * offline adversarial harness case `scope-other-owner-document` puts a patient name in the query,
+ * and it arrived in the answer body through this sentence.
+ *
+ * The wording is now fixed text. It carries no claim about what the documents contain and no
+ * material from the query, which is also why it is safe for `finalizeRagAnswerQualityCore` to
+ * pass it through instead of replacing it.
+ */
+export function sourceBackedGenerationTimeoutAnswer() {
+  return "A written answer could not be produced for this question. The document passages cited below were retrieved as possibly relevant source material and have not been confirmed as answering it. Please review them directly.";
 }
 
 const reasoningEffortRank: Record<OpenAIReasoningEffort, number> = {
@@ -4497,6 +4472,23 @@ function finalizeRagAnswerQualityCore(
   if (answer.preformatted && answer.grounded) {
     return answer;
   }
+  // Ledger #ZK460W. The source-backed review fallback is not a model answer being judged: it is a
+  // deterministic pointer built in this module ("a full written answer could not be completed,
+  // here are the passages that were retrieved"), delivered ungrounded and unsupported with
+  // review-only citations. Emission sites set `sourceBackedReviewFallback` so this short-circuit
+  // does not depend on routingReason string matching or empty-sections side-conditions.
+  // Every gate below is written for model prose and returns the wrong verdict on it:
+  // the ungrounded/unsupported gate and the query-overlap gate both replace it with
+  // "No current source ... was found", printed above the sources that were in fact found. That
+  // contradiction is what previously forced the route to relabel itself grounded to stay clear of
+  // these gates, which is the defect this row exists for. Nothing model-authored passes here.
+  if (answer.sourceBackedReviewFallback && !answer.grounded && answer.confidence === "unsupported") {
+    // The display mode is forced conservative here rather than left to the route's smart plan: a
+    // plan built for the rejected candidate can still ask for a threshold-table or comparison
+    // shape, and this answer has no rows to put in one. An evidence gap with citations attached is
+    // what it actually is.
+    return { ...answer, responseMode: "evidence_gap" };
+  }
   const cleanedAnswer = sanitizeAnswerText(answer.answer);
   const gapLikeAnswer =
     /could not find enough clean|no relevant clinical source|no current source|cannot provide a clinical answer|cannot provide a source-backed clinical answer|nearby indexed passages|not strong enough to support a reliable answer|no specific\b.*\bcan be confirmed|do not contain indexed guidance|do not contain (?:specific\s+)?information|do not provide specific|no\b.*\bguidance\b.*\bincluded|defer to other sources/i.test(