ci: codeql-guard to keep CodeQL out of evalops/*

[haasonsaas]

Summary

• Adds .github/workflows/codeql-guard.yml. Two jobs:
  • guard-self: rejects PRs/pushes here that add github/codeql-action references.
  • guard-org: daily sweep of every evalops/* repo via search/code; opens an issue in this repo if any workflow file references github/codeql-action.
• Documents the CodeQL-disabled policy in SECURITY.md and points to the EvalOps Blacksmith recommended code security configuration (id=245233) which is now:
  • the org default for new repos,
  • attached to all 77 existing repos,
  • configured with code_scanning_default_setup: disabled.

This closes the on-ramps so re-enabling CodeQL requires deleting the workflow + amending SECURITY.md rather than slipping in via a one-off codeql.yml.

Test plan

• CI runs guard-self on this PR and passes (no codeql-action references in .github/workflows).
• After merge, run the workflow once via workflow_dispatch to confirm guard-org reports clean.
• Add a throwaway codeql-action reference on a scratch branch in this repo and confirm guard-self fails the PR.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Introduces a scheduled org-wide GitHub search plus automated issue creation, which could generate noise or fail unexpectedly if API/permissions/quotas change; otherwise changes are limited to CI/policy and documentation.

Overview
Adds a new codeql-guard GitHub Actions workflow that (1) blocks PRs/pushes to this repo if .github/workflows or .github/workflow-templates introduce github/codeql-action, and (2) on a daily schedule/manual run searches the evalops org for CodeQL workflow references and opens an issue in evalops/.github if any are found.

Updates SECURITY.md to explicitly state the org policy of not using GitHub CodeQL, reference the Blacksmith security configuration used instead, and point to the new guard workflow as the enforcement mechanism.

^{Reviewed by Cursor Bugbot for commit 11a107a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: Guard grep always matches its own file
  • The self-guard grep now excludes codeql-guard.yml, so the workflow no longer trips on its own policy text.
• ✅ Fixed: Org sweep silently fails with insufficient token scope
  • The org sweep now requires a dedicated EVALOPS_ORG_READ_TOKEN and fails loudly if it is missing instead of silently reporting a clean sweep.
• ✅ Fixed: Daily cron creates duplicate issues without deduplication
  • The workflow now checks for an existing open tracking issue with the same title before creating a new one.

Preview (fae51734c0)

diff --git a/.github/workflows/codeql-guard.yml b/.github/workflows/codeql-guard.yml
new file mode 100644
--- /dev/null
+++ b/.github/workflows/codeql-guard.yml
@@ -1,0 +1,101 @@
+name: codeql-guard
+
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/**"
+      - ".github/workflow-templates/**"
+  push:
+    branches: [main]
+    paths:
+      - ".github/workflows/**"
+      - ".github/workflow-templates/**"
+  schedule:
+    # Daily org-wide drift sweep.
+    - cron: "17 9 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  issues: write
+
+jobs:
+  guard-self:
+    name: Forbid CodeQL in evalops/.github
+    if: ${{ github.event_name != 'schedule' }}
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
+
+      - name: Forbid github/codeql-action references
+        shell: bash
+        run: |
+          set -euo pipefail
+          shopt -s globstar nullglob
+          targets=(.github/workflows .github/workflow-templates)
+          existing=()
+          for d in "${targets[@]}"; do
+            [ -d "$d" ] && existing+=("$d")
+          done
+          if [ "${#existing[@]}" -eq 0 ]; then
+            echo "no workflow directories present"
+            exit 0
+          fi
+          if grep -RIn --include='*.yml' --include='*.yaml' --exclude='codeql-guard.yml' \
+               'github/codeql-action' "${existing[@]}" 2>/dev/null; then
+            echo "::error::EvalOps policy forbids github/codeql-action. See SECURITY.md (Code Scanning)."
+            exit 1
+          fi
+          echo "ok: no codeql-action references in evalops/.github"
+
+  guard-org:
+    name: Sweep evalops/* for CodeQL workflow drift
+    if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    timeout-minutes: 15
+    env:
+      GH_TOKEN: ${{ github.token }}
+      ORG_CODE_SEARCH_TOKEN: ${{ secrets.EVALOPS_ORG_READ_TOKEN }}
+    steps:
+      - name: Search org for github/codeql-action references
+        shell: bash
+        run: |
+          set -euo pipefail
+          if [ -z "${ORG_CODE_SEARCH_TOKEN}" ]; then
+            echo "::error::Set secrets.EVALOPS_ORG_READ_TOKEN to a token with org-wide code search access."
+            exit 1
+          fi
+          response="$(
+            GH_TOKEN="${ORG_CODE_SEARCH_TOKEN}" gh api -X GET search/code \
+              -f q='org:evalops "github/codeql-action" path:.github/workflows' \
+              --jq '.items[] | "\(.repository.full_name)\t\(.path)"' \
+          )"
+          if [ -z "${response}" ]; then
+            echo "ok: no CodeQL workflow files found in any evalops repo"
+            exit 0
+          fi
+          mapfile -t hits <<< "${response}"
+          {
+            echo "## codeql-guard tripped"
+            echo
+            echo "EvalOps does not run GitHub CodeQL (see \`SECURITY.md\` and the Blacksmith"
+            echo "code security configuration). The following workflow files reference"
+            echo "\`github/codeql-action\` and need to be removed or the policy amended:"
+            echo
+            for h in "${hits[@]}"; do
+              repo="${h%%$'\t'*}"
+              path="${h##*$'\t'}"
+              echo "- \`${repo}\` — \`${path}\`"
+            done
+          } > /tmp/body.md
+          title="codeql-guard: CodeQL workflow drift detected"
+          if issue_number="$(gh issue list --repo evalops/.github --state open --search "\"${title}\" in:title" --limit 1 --json number --jq '.[0].number // empty')" && [ -n "${issue_number}" ]; then
+            echo "open tracking issue already exists: #${issue_number}"
+          else
+            gh issue create \
+              --repo evalops/.github \
+              --title "${title}" \
+              --body-file /tmp/body.md
+          fi
+          exit 1

diff --git a/SECURITY.md b/SECURITY.md
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -22,3 +22,24 @@
 ## Scope
 
 This policy applies to all repositories in the [evalops](https://github.com/evalops) GitHub organization.
+
+## Code Scanning
+
+EvalOps does not enable GitHub CodeQL. Every repository is attached to the
+**EvalOps Blacksmith recommended** code security configuration
+(`id=245233`), which sets `code_scanning_default_setup: disabled` and is the
+default for new repositories.
+
+Equivalent static analysis lives elsewhere:
+
+- `semgrep`-based custom rules in service repos (see `.semgrep/` directories
+  and the `semgrep-custom` workflows).
+- Service-specific gates such as `architecture-review`, `contract-skew-check`,
+  and `migration-check` in `evalops/platform`.
+- The [`codeql-guard`](.github/workflows/codeql-guard.yml) workflow in this
+  repo enforces the policy: it rejects PRs that introduce
+  `github/codeql-action` here, and it sweeps every `evalops/*` repo daily,
+  opening an issue if a CodeQL workflow file appears anywhere in the org.
+
+To request a policy change, open a PR against this file and the guard
+workflow.

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 11a107a. Configure here.}