Skip to content

feat: add session-profiling protocol and profile-session template per…#72

Merged
Alan-Jowett merged 2 commits intomicrosoft:mainfrom
themechbro:feat/session-profiler-template
Mar 29, 2026
Merged

feat: add session-profiling protocol and profile-session template per…#72
Alan-Jowett merged 2 commits intomicrosoft:mainfrom
themechbro:feat/session-profiler-template

Conversation

@themechbro
Copy link
Copy Markdown
Contributor

This is a fresh draft for the Session Profiler Template (Issue #44), scoped strictly to the template architecture as discussed in the previous PR.

Changes in this Draft:

  • Added protocols/reasoning/session-profiling.md to define the analysis methodology.
  • Added templates/profile-session.md as the execution layer.
  • Made minimal, additive-only updates to manifest.yaml (placed under the investigation category).
  • Ensured all files comply with CONTRIBUTING.md standards (SPDX headers, YAML frontmatter, root directory placement).

@Alan-Jowett - Tagging you here for early feedback as requested! Let me know if the directory placement and YAML frontmatter look perfectly aligned with the engine's expectations before I build out the markdown/JSON formatting logic.

Alan-Jowett pushed a commit to Alan-Jowett/PromptKit that referenced this pull request Mar 24, 2026
Add profile-session template and session-profiling reasoning protocol
for analyzing completed LLM session logs to identify token inefficiencies
and structural waste.

Built on the structure from PR microsoft#72 by @themechbro — keeps the 5-phase
methodology and adds full PromptKit conventions:

Protocol (session-profiling):
- 5 phases: segment log, map to components, detect inefficiencies,
  quantify impact, produce recommendations
- 7 inefficiency types: redundant reasoning, false starts, re-derivation,
  protocol loops, unused context, verbose compliance, persona drift
- Each recommendation tied to a specific PromptKit component file

Template (profile-session):
- Full frontmatter: persona (specification-analyst), protocols
  (anti-hallucination, self-verification, session-profiling),
  format (investigation-report), params, contracts
- Params: session_log, assembled_prompt, focus_areas
- Non-goals: not a quality audit, not a guardrail remover

Manifest: additive-only changes (protocol + template entries, no
reformatting of existing content).

Co-authored-by: themechbro <109350438+themechbro@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Alan-Jowett Alan-Jowett force-pushed the feat/session-profiler-template branch from b20c0d9 to eeabaca Compare March 24, 2026 16:26
@Alan-Jowett
Copy link
Copy Markdown
Member

@themechbro — I pushed a co-engineering commit to your branch that builds on your 5-phase structure. Here's what changed:

What I kept from your PR

  • The 5-phase methodology (segment → map → detect → recommend) — this was the right structure
  • The inefficiency categories (redundant reasoning, false starts, protocol loops, unused context)
  • The file locations (protocols/reasoning/session-profiling.md, templates/profile-session.md)

What I added

Protocol — expanded each phase with the depth PromptKit protocols need (comparable to traceability-audit.md):

  • Concrete instructions for each phase (what to extract, how to record it)
  • 7 classified inefficiency types with IDs (RE-REDUNDANT, RE-BACKTRACK, etc.)
  • Phase 4 (quantify impact) added — token cost estimation and ranking
  • Recommendations must name the specific component file to change

Template — added full PromptKit frontmatter so the assembly engine can compose it:

  • persona: specification-analyst (systematic, evidence-driven — fits profiling)
  • protocols: [anti-hallucination, self-verification, session-profiling]
  • format: investigation-report (F-NNN findings with severity)
  • params with {{placeholder}} substitution
  • input_contract / output_contract
  • Non-goals section (not a quality audit, won't recommend removing guardrails)

Manifest — reverted the whitespace reformatting and kept only the 2 additive entries (protocol + template). The reformatting was causing 12 validation failures.

For reference

Compare templates/investigate-bug.md for the template pattern, and protocols/reasoning/traceability-audit.md for the protocol depth we target.

Let me know if you have questions or want to adjust anything!

@Alan-Jowett
Copy link
Copy Markdown
Member

@themechbro — stepping back, I want to apologize. You asked for early feedback on your draft, and instead of reviewing and letting you iterate, I force-pushed a rewrite directly to your branch. That wasn't the right call — it's your PR and your learning opportunity, and I should have respected that.

If you'd prefer to revert to your original commit and work from review feedback instead, that's completely fine. You can restore your version with:

git reflog  # find your original commit hash
git reset --hard <your-original-hash>
git push --force

Either way — your original structure (5 phases, the inefficiency categories, the file locations) was the right foundation. The main things that need to happen for the assembly engine to work with it are:

  1. Template frontmatter needs persona, protocols, format, params, input_contract, output_contract — see templates/investigate-bug.md as the reference pattern
  2. Protocol frontmatter needs type: reasoning and applicable_to — see protocols/reasoning/traceability-audit.md
  3. Manifest changes should be additive only — just the 2 new entries, no reformatting of existing lines (the YAML style changes were causing validation failures)
  4. SPDX headers at the top of both files
  5. {{param}} placeholders in the template body so the assembly engine can substitute user inputs

Happy to answer questions or do a more detailed review — whichever you'd find more helpful. Again, sorry for overstepping.

@themechbro
Copy link
Copy Markdown
Contributor Author

@Alan-Jowett - I've pulled down your co-engineering commit and applied the fixes flagged by the Copilot review.

Changes included in this push:

  • Fixed the YAML frontmatter indentation in profile-session.md that was causing parsing risks.
  • Added the explicit severity levels to the instruction set.
  • Rephrased the Phase 4 protocol to focus purely on "efficiency tradeoffs" rather than output quality.
  • Appended the missing CONTRIBUTING.md Quality Checklist to the bottom of the template.

Let me know if this is ready to drop the "Draft" status!

@Alan-Jowett
Copy link
Copy Markdown
Member

Thanks for sending this over — I really appreciate it. I haven’t forgotten about it; I just need a bit more time to look through it carefully. I’ll follow up once I’ve had a chance to dig in.

@Alan-Jowett
Copy link
Copy Markdown
Member

Hey @themechbro — the protocol and template content here is looking good. The 5-phase structure, inefficiency categories (RE-REDUNDANT, RE-BACKTRACK, etc.), and the investigation-report integration all work well.

The main thing blocking this is that the branch is based on an old version of main. Since you branched, a bunch of components were merged independently (workflow-arbiter, invariant-extraction, extract-invariants, etc.), and your manifest diff is re-adding them — that's why the diff shows +102 lines in manifest.yaml when only ~22 are actually yours.

To fix this, rebase onto current main:

\\�ash
git fetch upstream
git rebase upstream/main
\\

After rebasing, your manifest.yaml diff should only contain:

  1. The session-profiling protocol entry (under protocols.reasoning)
  2. The profile-session template entry (under templates.investigation)

If the diff still shows changes to personas, other protocols, or other templates, those are stale and need to be dropped. The rebase should handle this automatically if there are no conflicts.

Once the diff is clean, this should be ready for a final review. Let me know if you hit any issues with the rebase!

@themechbro themechbro force-pushed the feat/session-profiler-template branch from f4f1033 to c0c1cbc Compare March 28, 2026 17:10
@themechbro
Copy link
Copy Markdown
Contributor Author

@Alan-Jowett — I’ve successfully rebased the branch onto the latest main. The diff is now clean and only contains:

  1. The session-profiling protocol entry and the profile-session template entry in manifest.yaml.
  2. The two new .md files in their respective directories.

Let me know if there’s anything else needed, or if this is ready for the final merge!

@Alan-Jowett
Copy link
Copy Markdown
Member

Rebase looks clean — manifest diff is exactly the 2 additive entries (protocol + template), no stale changes. Content is solid.

Go ahead and mark it ready for review when you're ready!

@themechbro themechbro marked this pull request as ready for review March 28, 2026 17:48
@Alan-Jowett Alan-Jowett requested a review from Copilot March 28, 2026 20:07
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new “session profiling” capability to PromptKit by introducing a dedicated reasoning protocol and a corresponding template, and wiring both into the component manifest so they can be assembled and used like other investigation workflows.

Changes:

  • Added session-profiling reasoning protocol for analyzing completed LLM session logs and attributing token waste to PromptKit components.
  • Added profile-session template that composes existing guardrails with the new protocol and outputs an investigation-report.
  • Updated manifest.yaml to register the new protocol and template under the investigation category.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
templates/profile-session.md New template for profiling an LLM session log and producing an investigation-report with optimization recommendations.
protocols/reasoning/session-profiling.md New reasoning protocol defining phases for segmenting a session, attributing token usage, detecting inefficiencies, and recommending optimizations.
manifest.yaml Registers session-profiling protocol and profile-session template so the CLI/assembler can resolve them.

@Alan-Jowett
Copy link
Copy Markdown
Member

Alan-Jowett commented Mar 29, 2026

Heads up @themechbro — a few PRs are about to merge that add new components to manifest.yaml. You'll need another rebase before this can merge. The changes are in different manifest sections (personas, formats, new protocol categories) so it should auto-resolve cleanly:

git fetch upstream
git rebase upstream/main

Sorry for the churn — the library is growing fast this week!

themechbro and others added 2 commits March 29, 2026 10:41
Add profile-session template and session-profiling reasoning protocol
for analyzing completed LLM session logs to identify token inefficiencies
and structural waste.

Protocol (session-profiling):
- 5 phases: segment log, map to components, detect inefficiencies,
  quantify impact, produce recommendations
- 7 inefficiency types: redundant reasoning, false starts, re-derivation,
  protocol loops, unused context, verbose compliance, persona drift
- Each recommendation tied to a specific PromptKit component file

Template (profile-session):
- Full frontmatter: persona (specification-analyst), protocols
  (anti-hallucination, self-verification, session-profiling),
  format (investigation-report), params, contracts
- Params: session_log, assembled_prompt, focus_areas
- Non-goals: not a quality audit, not a guardrail remover

Manifest: additive-only changes (protocol + template entries, no
reformatting of existing content).

Co-authored-by: Alan-Jowett <alanjo@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@themechbro themechbro force-pushed the feat/session-profiler-template branch from 5dc9d81 to 9fc795f Compare March 29, 2026 05:11
@themechbro
Copy link
Copy Markdown
Contributor Author

Heads up @themechbro — a few PRs are about to merge that add new components to manifest.yaml. You'll need another rebase before this can merge. The changes are in different manifest sections (personas, formats, new protocol categories) so it should auto-resolve cleanly:

git fetch upstream
git rebase upstream/main

Sorry for the churn — the library is growing fast this week!

@Alan-Jowett — All set! Rebased onto the latest main and force-pushed. It auto-resolved cleanly just like you suspected. Ready when you are!

@Alan-Jowett Alan-Jowett merged commit 30f2f48 into microsoft:main Mar 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants