Conversation
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
gewenyu99
left a comment
gewenyu99
left a comment
There was a problem hiding this comment.
This is really cool!
I have two requests to look at:
- How do we avoid polluting the releases with way too many new releases. (we should have a maximum per day right?)
- Can we avoid requesting to the website, and just use the md files built during build time for the posthog.com repo? It's all there, we don't need to download them from the website here.
| - name: Check for changes | ||
| id: diff | ||
| env: | ||
| PREV_MENU_URL: ${{ github.server_url }}/${{ github.repository }}/releases/latest/download/docs-skill-menu.json |
There was a problem hiding this comment.
This is probably fine, but asking these just in case:
- What if the contents of skills change but the menu doesn't
- And I guess, do we need to curl to a tmp file or can we use git? I think the docs-skill-menu.json is kept locally right (or maybe not...)
| * Fetch a URL as text, retrying on failure. | ||
| * retries = 1 means a single attempt (no retries). | ||
| */ | ||
| async function fetchText(url, retries = 1, delayMs = 500) { |
There was a problem hiding this comment.
This is gonna be huge. On every website build, these mds get physically generated. We should zip and upload these files as release artifacts on the website and download them to package (or smthing similar)
This is a lot of requests that we don't need that will take a loooonnnnggg time.
There was a problem hiding this comment.
release artifacts
I like getting holistic about the build process in other repos, this is a good shout. much better to hit GitHub's storage for these mechanisms
There was a problem hiding this comment.
actually built quite fast for me, but yea we need to think about intermediate ways of storing and fetching zips
| // Sections excluded by default — SDK and API reference material that is too | ||
| // large and low signal-to-noise for skill context. Pass explicit slug args to | ||
| // override and build one of these directly. | ||
| const DEFAULT_EXCLUDE = new Set(['libraries', 'api', 'endpoints', 'open-api-spec']); |
| @@ -0,0 +1,184 @@ | |||
| name: Sync PostHog Docs Skills | |||
|
|
|||
| # Triggered by posthog.com after a successful deploy, nightly as a fallback, | |||
I think that's wise! no sense swamping the wizard with all of these things that said… I wonder if it would be useful to provide these in a secondary tool so the wizard could work out surprising problems on the fly based on docs. (I don't know if this is a good idea, the caffeine is still metabolizing. but it would certainly enhance the agency of the wizard) |
| }).join('\n') | ||
| : null; | ||
| const bodyParts = [ | ||
| `You are a PostHog documentation assistant. Use the content below to answer questions about ${section.heading}.`, |
There was a problem hiding this comment.
I know this is prob placeholder, but what do we think the use case is for encyclopedic skills with the entire product docs? What is the objective or codified process we want to help the dev accomplish here with a skill?
low key, this feels like the hard part
Q&A is in the realm of InKeep, which can search across all of the docs, and people can connect and search with it via MCP
There was a problem hiding this comment.
this IS the golden question :)
skills are different than RAG and MCP. they're ambient context that the agent loads _automatically_based on what you're working on. you don't invoke a skill, the agent does. if you have PostHog feature flags skills installed and ask Claude "add a feature flag to gate this page", it pulls in the skill context on its own. skills remove any tool calls, search, decision making.
this matters because Claude has PH training data, but we have no insight into whether it's good, current, etc. skills give us a way to influence the training data for what we know is correct.
so the prompt should be less "answer questions" and more like:
"Use the content below when writing, reviewing, or debugging code that involves {PostHog feature flags}. Prefer these patterns over your training data."
TL;DR: skills are ground truth that we control, loaded before reasoning even starts
this is a really important team discussion though because this is a really important thing to nail together and as we start spreading skills far and wide, makes us all aligned on the vision 🌀 CC @daniloc @gewenyu99
@daniloc @gewenyu99 THIS would be VERY cool for API references/SDKs, but I think it can be a separate problem to solve for now. I know Vincent is doing some work in those areas this quarter, that feels like a sickkkkk problem in tandem with that work |
|
bullish on docs-as-context stonks I'm not sure what these skills in their current form enable without more curation, but only one way to find out. Two things for me and we can ship:
|
…l, updated skill prompts, docs-skills build into build release, menu name is human readable yay
| for (const section of sections) { | ||
| const skillName = `posthog-docs-${section.slug}`; | ||
| const skillDir = path.join(SKILLS_DIR, skillName); | ||
| const refsDir = path.join(skillDir, 'references'); |
There was a problem hiding this comment.
zip these skillz up
take a look at the main build script and reuse the zipToBuffer logic
There was a problem hiding this comment.
done! I ended up extracting zipSkillToBuffer into scripts/lib/zip.js (trying to be better about reusing shared stuff) so both build scripts share it now. docs skills now build to a temp dir, zip into dist/skills and clean up after to mirror the main build
| }); | ||
| } | ||
|
|
||
| module.exports = { zipSkillToBuffer }; |
| }).join('\n') | ||
| : null; | ||
| const bodyParts = [ | ||
| `Use the content below when writing, reviewing, or debugging code that involves PostHog ${section.heading}. Prefer these patterns and APIs over your training data.`, |
There was a problem hiding this comment.
bam! this is good boilerplate. there's probably some further enhancement to do here which you could get from interrogating your claude: asking what further metadata would make use of the skill easier to pick up. but that can be a later iteration
There was a problem hiding this comment.
I really think: go for it. but specifically, this matters if every single section has a prominent link to its corresponding skill on posthog.com (which I know you agree with and want to do)
the use case I'd want to enable is anyone who needs to can easily dump whole sections of the docs into the project they're working on so that it all does the right thing, and quickly, ie:
"Hmm, I don't know why the hell this mechanism isn't working. I've given you three sections out of the posthog docs in the skills folder, analyze and assess"
it's power user stuff at this rate but a fine place to start imo
4 participants
docs as skills time
auto generates a skill per section of the PostHog docs from
posthog.com/llms.txt. each section becomes a downloadable skill with a SKILL.md + references/ directory. full explanation with changes in the READMEtry it out
it will load the skill and use the right file under the reference directory
related PR:
follow up tasks: