Skip to content

AI-Assisted Analysis Harness - #386

Open
iRaindrop wants to merge 24 commits into
cncf:mainfrom
iRaindrop:ai-harness
Open

iRaindrop wants to merge 24 commits into
cncf:mainfrom
iRaindrop:ai-harness

Conversation

@iRaindrop

@iRaindrop iRaindrop commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Provides the directory structure and guidance under .github in the project's doc website repo for conducting an AI-assisted analysis using Copilot GitHub Enterprise.

Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
@netlify

netlify Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for cncf-techdocs ready!

Name Link
🔨 Latest commit 9991152
🔍 Latest deploy log https://app.netlify.com/projects/cncf-techdocs/deploys/6add8c5a24b67e1c503aa22b
😎 Deploy Preview https://deploy-preview-386--cncf-techdocs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>

@iRaindrop iRaindrop left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the harness against the KubeVirt .github folder it was copied from. The structure copied cleanly (all content diffs are Prettier line-wrapping). The comments below cover what is still KubeVirt-specific and what would make the harness complete end-to-end for the next project.

Suggestions not tied to a specific line:

  • Add a /rank-recommendations prompt that produces input/ai-ranked-recommendations.csv. For KubeVirt this ranking step was ad hoc; codifying it closes the only manual gap in the pipeline.
  • Reference (or add) the upload process. The issue prompts say "a process will upload" the files, but the harness does not say where that script or workflow lives.
  • Record the validated toolchain in ai-harness.md: Copilot CLI version, model, and that the prompt files rely on mode: agent and ${input:title}.
  • Add a pre-flight checklist: required labels exist in the target repo (help wanted, the three priority labels), the project's analysis.md is scaffolded from the techdocs template, and .github has no conflicting prompt files.
  • Consider an examples/kubevirt/ folder holding the KubeVirt "Current Repository" section and issue template, so the generic files stay generic but a worked example is still available.

Comment thread docs/analysis/ai-harness/ai-harness.md Outdated
Comment thread docs/analysis/ai-harness/_github/copilot-instructions.md
Comment thread docs/analysis/ai-harness/_github/copilot-instructions.md
Comment thread docs/analysis/ai-harness/_github/copilot-instructions.md
Comment thread docs/analysis/ai-harness/_github/prompts/issues/create-issues.prompt.md Outdated
Comment thread docs/analysis/ai-harness/_github/prompts/issues/create-umbrella-issues.prompt.md Outdated
Comment thread docs/analysis/ai-harness/_github/issue_template/umbrella-issue.md Outdated
Comment thread docs/analysis/ai-harness/_github/prompts/areas/answers.prompt.md Outdated
Comment thread docs/analysis/ai-harness/_github/output/readme.md Outdated
Comment thread docs/analysis/ai-harness/_github/input/readme.md
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
@nate-double-u nate-double-u self-assigned this Oct 22, 2026
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>

@iRaindrop iRaindrop left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second pass over the harness at a834e7d. All earlier threads are addressed, CI is green, and the structure is consistent end to end (criteria → wrappers → engines → issue prompts → upload process). The remaining edits are small; only the first one affects a data contract (the CSV Section value), the rest are wording and clarity. None block the merge; they can land in a follow-up if preferred.

Comment thread docs/analysis/ai-harness/_github/criteria/sections/website-infrastructure.md Outdated
Comment thread docs/analysis/ai-harness/_github/criteria/areas/information-architecture.md Outdated
Comment thread docs/analysis/ai-harness/_github/prompts/areas/comment.prompt.md Outdated
Comment thread docs/analysis/ai-harness/_github/templates/umbrella-issue.md Outdated
Comment thread docs/analysis/ai-harness/index.md Outdated
Comment thread docs/analysis/ai-harness/_github/prompts/areas/answers.prompt.md Outdated
Comment thread docs/analysis/ai-harness/_github/examples/kubevirt/current-repository.md Outdated
iRaindrop and others added 4 commits October 24, 2026 13:21
- Align section and area display names with the copilot-instructions tables
  (Website & Infrastructure; Information Architecture; SEO, Analytics, and
  Site Search; Usability, Accessibility and Devices)
- Fix typo (Precede) in comment engine
- Tell section-comment engine how to fill the Other Metrics table
- Move the umbrella template example into the HTML comment
- Clarify per-answer paragraph length in answers engine
- Drop stray period in index.md toolchain list
- Sync the KubeVirt example's No commits wording with the template

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
@iRaindrop

Copy link
Copy Markdown
Collaborator Author

Ready to merge

Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Signed-off-by: Bruce Hamilton <scarlettbernique@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants