Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
3bdd799
Register nine Australian clinical publishers in the source authority …
claude Sep 7, 2026
a025a1e
Merge origin/main into claude/register-australian-publishers
claude Sep 7, 2026
87e43c3
Merge origin/main into claude/register-australian-publishers
claude Sep 7, 2026
ac21a2f
Protect the source authority register behind the RAG impact gate, and…
claude Sep 7, 2026
9d84ed5
Consolidate publisher registration and source authority policy guard
BigSimmo Sep 10, 2026
4f98a1e
fix(rag): refresh snapshot and cover publisher context ordering
BigSimmo Sep 11, 2026
3a5d0a9
Merge origin/main into claude/register-australian-publishers
BigSimmo Sep 12, 2026
31b7c95
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 12, 2026
cd692f4
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 12, 2026
83057cf
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 12, 2026
53607d8
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 12, 2026
e4113f4
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 12, 2026
4599462
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 12, 2026
7c3b34f
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 12, 2026
1fb90b7
Merge origin/main into claude/register-australian-publishers
BigSimmo Sep 13, 2026
1db5cc9
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 13, 2026
e6a1c89
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 13, 2026
0f9fa3b
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 13, 2026
6a599df
Merge branch 'main' into claude/register-australian-publishers
BigSimmo Sep 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"version": 2,
"id": "2054f3ed-afce-4be9-a7e2-88cdeb8b147c",
"createdOn": "2026-09-07",
"action": "add",
"payload": {
"pri": "P2",
"type": "issue",
"summary": "Weekly eval canary has failed on the same three answer-quality cases since at least 2026-08-23, and nothing tracks it",
"detail": "Golden retrieval passes throughout: run 34099357211 (2026-09-07, repository_dispatch) recorded cases=36, document_recall@5=1, content_recall@5=1, mrr@10=0.8921, zero failing retrieval cases. The run is red only because the separate live answer-quality subset fails its blocking threshold: 'RAG source_backed_review_fallback unaccounted' on clozapine-monitoring (high_confidence_extractive_retrieval; final_quality_gate:guidance_wrapper_fragment; extractive_quality_gate:provider_source_gap), clozapine-monitoring-paraphrase (identical signature), and nocc-requirements (strong_routine_retrieval; generation_fallback:generation_quality_failed; extractive_quality_gate:guidance_wrapper_fragment). The five most recent canary runs are all red: 34099357211, 34050934409 (scheduled), 33582708071, 33327352850 (scheduled), 32656925301 (scheduled, 2026-08-23). CONSEQUENCE: the canary is the pair-comparison instrument for every protected RAG change, and a permanently red run trains readers to ignore it. A future retrieval regression would land inside an already-red signal. NEXT: this is the same failure family as #J8SJQ9 (source-backed review stub where an authoritative source exists) but names different cases, so check that row for overlap before starting. First diagnostic step is whether generation was attempted at all for these three, exactly as #J8SJQ9 sets out. STOP: do not add acceptSourceOnly to these eval cases and do not relax the threshold to get the canary green. The retrieval half of the run remains a valid baseline in the meantime.",
"source": "Canary run logs 34099357211 and 34050934409, read during PR #2711 (source authority register), 2026-09-07",
"issueUlid": "01M1XSYJ9DSXPQ0A9B1TZ8J6NY"
}
}
7 changes: 7 additions & 0 deletions docs/rag-behaviour/safeguards.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ Code and ground truth whose edits change (or re-measure) retrieval/ranking behav
`src/lib/released-search-order.ts`, `src/lib/ranking-config.ts`, `src/lib/evidence.ts`,
`src/lib/result-sort.ts`, `src/lib/answer-ranking.ts`, `src/lib/evidence-relevance.ts`,
`src/lib/semantic-rerank.ts`, `src/lib/eval-document-matching.ts`
- `src/lib/source-authority-registry.ts`, `src/lib/australian-source-priority.ts` — one step
further back, but still ordering: `classifySourceAuthority` assigns a result's tier,
`australianSourceTier` reads it, and `selectAustralianClinicalContext` orders and trims the
model's context by it. Registering or re-scoping a publisher moves that ordering without
touching a score. Added 2026-09-07 (PR #2711 declared its RAG impact voluntarily because the
gate did not yet ask). Note this is authority _classification_, not a governance ranking
penalty or boost — the latter stays refuted, see `refuted-approaches.md` § Refutation 3.
- `scripts/eval-retrieval.ts`, `scripts/lib/clinical-aliases.ts`,
`scripts/lib/ranking-tuning.ts`, `scripts/lib/ranking-snapshot-builder.ts`,
`scripts/build-ranking-snapshot.ts`, `scripts/tune-search-weights.ts`
Expand Down
8 changes: 5 additions & 3 deletions docs/source-acquisition-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,9 +135,11 @@ prints the outstanding queue, most local first.
- **An unregistered publisher cannot leave D band.** The catalogue derives jurisdiction solely
from `src/lib/source-authority-registry.ts`, so a source from a publisher that is not
registered has no jurisdiction and is forced to D. Therapeutic Guidelines, the Australian
Medicines Handbook, Australian Prescriber, RACP, Cochrane, AIHW, HealthyWA and the Mental
Health Commission WA are all currently unregistered. The gate detects this and names the
publisher rather than failing obscurely.
Medicines Handbook, Australian Prescriber, RACP, Cochrane, AIHW, Healthdirect Australia,
HealthyWA and the Mental Health Commission WA were registered on 2026-09-07. Beyond Blue,
the NSW Agency for Clinical Innovation and Western Sydney Local Health District are examples
that remain unregistered. The gate detects this and names the publisher rather than failing
obscurely.

Registering a publisher changes `classifySourceAuthority`, which retrieval selection uses to
tier results. That makes it a retrieval behaviour change under the RAG ranking protection
Expand Down
9 changes: 9 additions & 0 deletions scripts/pr-policy.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,13 @@ const migrationSubjectPattern =
const ragRankingPatterns = [
/^src\/lib\/rag\//,
/^src\/lib\/(?:clinical-search|retrieval-selection|released-search-order|ranking-config|evidence|result-sort|answer-ranking|evidence-relevance|semantic-rerank|eval-document-matching)\.ts$/,
// Source authority feeds ordering too, one step further back: classifySourceAuthority tiers a
// result, australianSourceTier reads that tier, and selectAustralianClinicalContext orders and
// trims the model's context by it. Registering a publisher is therefore a retrieval behaviour
// change even though neither file computes a score. Added 2026-09-07 after PR #2711 registered
// nine Australian publishers and had to declare its RAG impact voluntarily, because this gate
// did not ask.
/^src\/lib\/(?:source-authority-registry|australian-source-priority)\.ts$/,
/^scripts\/(?:eval-retrieval|build-ranking-snapshot|tune-search-weights)\.ts$/,
/^scripts\/lib\/(?:clinical-aliases|ranking-tuning|ranking-snapshot-builder)\.ts$/,
/^scripts\/fixtures\/(?:rag-retrieval-golden|rag-ranking-candidate-snapshot\.v1)\.json$/,
Expand Down Expand Up @@ -690,6 +697,8 @@ function selfTest() {
// The golden fixture and contract tests are protected surfaces too.
assert.equal(classifyPullRequestFiles(["scripts/fixtures/rag-retrieval-golden.json"]).ragRanking, true);
assert.equal(classifyPullRequestFiles(["tests/ranking-tuning.test.ts"]).ragRanking, true);
assert.equal(classifyPullRequestFiles(["src/lib/source-authority-registry.ts"]).ragRanking, true);
assert.equal(classifyPullRequestFiles(["src/lib/australian-source-priority.ts"]).ragRanking, true);
// Answer synthesis is clinical-risk but NOT rag-ranking (retrieval ordering is the
// protected axis here; generation keeps the governance gate only).
assert.equal(classifyPullRequestFiles(["src/lib/answer-synthesis.ts"]).ragRanking, false);
Expand Down
76 changes: 75 additions & 1 deletion src/lib/source-authority-registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,15 @@ export const sourceAuthorityRegistry = [
designation: "official",
officialBasis: "wa_hospital",
}),
authority({
key: "healthywa",
codes: ["HEALTHYWA"],
publisher: "HealthyWA",
publisherAliases: ["Healthy WA", "HealthyWA - Western Australian Government health information"],
jurisdictions: waJurisdictions,
scope: "wa",
tier: "wa_validated",
}),
authority({
key: "king-edward-memorial-hospital",
codes: ["KEMH", "KEMHS"],
Expand All @@ -205,6 +214,19 @@ export const sourceAuthorityRegistry = [
designation: "official",
officialBasis: "wa_hospital",
}),
authority({
key: "mental-health-commission-wa",
codes: ["MHCWA", "MHC WA"],
publisher: "Mental Health Commission WA",
publisherAliases: [
"Mental Health Commission Western Australia",
"Western Australian Mental Health Commission",
"Government of Western Australia Mental Health Commission",
],
jurisdictions: waJurisdictions,
scope: "wa",
tier: "wa_validated",
}),
authority({
key: "north-metropolitan-health-service",
codes: ["NMHS"],
Expand Down Expand Up @@ -275,9 +297,17 @@ export const sourceAuthorityRegistry = [
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "aihw",
codes: ["AIHW"],
publisher: "Australian Institute of Health and Welfare",
jurisdictions: nationalJurisdictions,
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "australian-prescriber",
codes: ["AUSPRES"],
codes: ["AUSPRES", "AUSTPRESC"],
publisher: "Australian Prescriber",
jurisdictions: nationalJurisdictions,
scope: "australian_national",
Expand All @@ -293,6 +323,24 @@ export const sourceAuthorityRegistry = [
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "australian-medicines-handbook",
codes: ["AMH"],
publisher: "Australian Medicines Handbook",
publisherAliases: ["Australian Medicines Handbook Pty Ltd"],
jurisdictions: nationalJurisdictions,
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "healthdirect-australia",
Comment thread
BigSimmo marked this conversation as resolved.
codes: ["HEALTHDIRECT"],
publisher: "Healthdirect Australia",
publisherAliases: ["healthdirect"],
jurisdictions: nationalJurisdictions,
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "nhmrc",
codes: ["NHMRC"],
Expand Down Expand Up @@ -327,6 +375,14 @@ export const sourceAuthorityRegistry = [
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "racp",
codes: ["RACP"],
publisher: "Royal Australasian College of Physicians",
jurisdictions: nationalJurisdictions,
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "ranzcp",
codes: ["RANZCP"],
Expand All @@ -343,6 +399,15 @@ export const sourceAuthorityRegistry = [
scope: "australian_national",
tier: "australian_national",
}),
authority({
key: "therapeutic-guidelines",
codes: ["TG", "ETG"],
publisher: "Therapeutic Guidelines",
publisherAliases: ["Therapeutic Guidelines Limited", "Therapeutic Guidelines Ltd", "eTG complete"],
jurisdictions: nationalJurisdictions,
scope: "australian_national",
tier: "australian_national",
}),
...[
["act-health", "ACTHEALTH", "ACT Health", "Australia/ACT"],
["nsw-health", "NSWHEALTH", "NSW Health", "Australia/NSW"],
Expand Down Expand Up @@ -370,6 +435,15 @@ export const sourceAuthorityRegistry = [
scope: "international",
tier: "supplementary",
}),
authority({
key: "cochrane",
codes: ["COCHRANE"],
publisher: "Cochrane",
publisherAliases: ["The Cochrane Collaboration", "Cochrane Library", "Cochrane Database of Systematic Reviews"],
jurisdictions: ["International", "Global"],
scope: "international",
tier: "supplementary",
}),
authority({
key: "nice",
codes: ["NICE"],
Expand Down
44 changes: 44 additions & 0 deletions tests/rag-context-budget.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,50 @@ describe("RAG model context budgeting", () => {
expect(selected.map((result) => result.id)).toEqual(["chunk-1", "chunk-2", "chunk-3", "chunk-4"]);
});

it("orders newly registered WA and national publishers ahead of equally relevant supplementary context", () => {
Comment thread
BigSimmo marked this conversation as resolved.
const selected = selectModelContextResults({
routeMode: "strong",
queryClass: "medication_dose_risk",
crossDocument: false,
results: [
withRelevance(bmjSupplementarySource(1), "direct"),
withRelevance(
governedSource(2, {
documentId: "tg-doc",
publisherCode: "TG",
publisher: "Therapeutic Guidelines",
jurisdiction: "Australia/National",
validation: "unverified",
}),
"direct",
),
withRelevance(
governedSource(3, {
documentId: "healthywa-doc",
publisherCode: "HEALTHYWA",
publisher: "HealthyWA",
jurisdiction: "Australia/WA",
}),
"direct",
),
withRelevance(
governedSource(4, {
documentId: "cochrane-doc",
publisherCode: "COCHRANE",
publisher: "Cochrane",
jurisdiction: "International",
validation: "unverified",
}),
"direct",
),
],
});

// Registration changes only same-relevance context ordering: HealthyWA is WA-validated,
// Therapeutic Guidelines is Australian national, while Cochrane remains supplementary.
expect(selected.map((result) => result.id)).toEqual(["chunk-3", "chunk-2", "chunk-1", "chunk-4"]);
});

it("keeps supplementary evidence when authoritative Australian coverage is not sufficient", () => {
const selected = selectModelContextResults({
routeMode: "strong",
Expand Down
11 changes: 10 additions & 1 deletion tests/source-acquisition-ledger.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import {
sourceAcquisitionRecords,
type SourceAcquisitionRecord,
} from "@/lib/sources/acquisition-ledger";
import { sourceAuthorityForPublisher } from "@/lib/source-authority-registry";
import { canonicalizeSourceReferences } from "@/lib/sources/catalogue-core";
import {
classifySourceAuthority,
Expand Down Expand Up @@ -47,6 +48,10 @@ const baseRecord: SourceAcquisitionRecord = {
notes: null,
};

// A real Australian clinical publisher that is deliberately absent from the source authority
// register, so the "unrecognised publisher" path has a genuine example to exercise.
const unregisteredPublisher = "Beyond Blue";

function record(overrides: Partial<SourceAcquisitionRecord> = {}): SourceAcquisitionRecord {
return { ...baseRecord, ...overrides };
}
Expand Down Expand Up @@ -174,7 +179,11 @@ describe("source acquisition rungs", () => {
});

it("names the register as the blocker when a publisher is not recognised, and warns that fixing it moves retrieval", () => {
const [issue] = issuesFor({ publisher: "Therapeutic Guidelines Limited", publisherCode: null, canonicalUrl: null });
// The example publisher has to be one the register genuinely does not carry, so assert that
// here rather than trusting the constant: Therapeutic Guidelines used to sit in this slot and
// the premise silently went false the day it was registered.
expect(sourceAuthorityForPublisher(unregisteredPublisher)).toBeNull();
const [issue] = issuesFor({ publisher: unregisteredPublisher, publisherCode: null, canonicalUrl: null });
expect(issue).toContain("is not in the source authority register");
expect(issue).toContain("can never leave D band");
expect(issue).toContain("changes retrieval selection");
Expand Down
Loading
Loading