Add hybrid-duplication-audit skill - #129
Conversation
Audits asset duplication across the AssetBundle / ContentDirectory boundary in a hybrid Addressables 4.x build. Runs one `analyze` combining the Addressables build layout report with the content directory build output, then matches source-asset paths across both - catching cases a CRC-based diff misses, such as a shader built with a different variant set on each side (which is also why Addressables' own DuplicatedAssetCount can read 0 even when real duplication exists).
Linux/macOS equivalent of Compare-HybridDuplication.ps1 - same defaults, same queries, same output. Verified against this session's build data: reproduces the PowerShell version's result exactly.
SkowronskiAndrew
left a comment
SkowronskiAndrew
left a comment
There was a problem hiding this comment.
I remembered i hadn't looked at this yet. I am giving some quick feedback but ran out of time to read the skill fully. This does seem very helpful because it is tricky to figure out how to use UnityDataTools for a hybrid build, and raises some interesting questions about what comparison means for Shaders and when and how we should analyze the actual content as well.
| name: hybrid-duplication-audit | ||
| description: Audits asset duplication across the AssetBundle / ContentDirectory boundary in a hybrid Addressables 4.x build, where some groups build to .bundle files and others build through BuildPipeline.BuildContentDirectory. Use when asked to compare duplication in a hybrid project, find what's duplicated between bundles and the content directory, explain why Addressables' own DuplicatedAssetCount reads 0 or looks too low despite shared shaders, textures, or meshes across bundle and content-directory groups, or size the "hybrid tax" of assets baked twice because an AssetBundle cannot reference content-directory content. Runs one UnityDataTool analyze combining the Addressables build layout report with the content directory build output, then matches source-asset paths across both to find what's genuinely duplicated — including cases a plain CRC/hash diff misses, such as the same shader built with a different variant set on each side. | ||
| compatibility: Requires UnityDataTool (github.com/Unity-Technologies/UnityDataTools) built from a version with ContentLayout.json support, sqlite3 on PATH, and a completed hybrid Addressables 4.x build (at least one AssetBundle-producing group and one ContentDirectoryGroupSchema group). | ||
| --- |
There was a problem hiding this comment.
I think this description could be slimmed down, it feels like it gets far into details, and all that would get loaded in the context even if doing completely different work with UnityDataTools?
(Eventually I think we should have a "analyze" skill, or "compare-builds" skill that acts as a guide for other specific skills to use depending on the type of build but thats not needed immediately for this PR)
| Write-Output " Build history: $BuildHistory" | ||
| Write-Output "" | ||
|
|
||
| & $resolvedToolPath analyze $BuildLayout $ContentDirectory --build-history $BuildHistory -o $Database |
There was a problem hiding this comment.
One detail, although I'm not certain how to "fix it" with the existing analyze API. It seems you do full analysis of the all the content directory serialized files (because it is an argument) but you don't ever use that information in the queries (because the content_layout file is what we need). Meanwhile we don't analyze the asset bundles.
I think this is happening because in order to find the correct build history entry we want the BuildManifestHash.txt file. But the act of providing that info is resulting in a full scan.
We may face similar situation for Addressables when we move the layout file into build history. E.g. we want to match the output to the correct build history entries. But for this calculation we don't strictly need to do the big analysis of the contents.
On the other hand maybe doing the full analysis will be worth it - i think that differing CRCs is important information because two shaders that have different variants are not really a duplicate of the same built content (but its important information that they are the same shader)
There was a problem hiding this comment.
I was thinking perhaps we should have a separate script for finding the buildhistory folder from the build output folder. Maybe even part of a "buildhistory" command in UnityDataTools instead of being a script inside the skill.
So the helper would take care of the detail of finding the correct folder with the ContentLayout.json file based on the manifest hash, and then when calling analyze we pass in the specific ContentLayout.json file as an argument. That avoids doing a full analysis of all the ContentDirectory content just for getting a single hash value.
(We can also think how that we should look into finding the Addressables layout file in a general way that works with both the current Library structure and the upcoming BuildHistory integration, but that could be deferred)
On the other hand, its possible that always analyzing the build content is better, e.g. to get CRC values and supporting certain comparison operations, so my line of thinking about the optimization may be going the wrong direction. I just know that a full analysis with all references can take 30 minutes and produce a database that is hundred of MB large so thats not good for a temporary database if we never look at the resulting tables.
| forms doesn't necessarily produce byte-identical objects (a shader's variants, for example, are | ||
| stripped independently on each side), so CRC matching can miss it entirely. See | ||
| [Skills/hybrid-duplication-audit](../Skills/hybrid-duplication-audit/SKILL.md) for a query that | ||
| matches by source asset path instead, which catches this case. |
There was a problem hiding this comment.
I think this paragraph needs a bit more context to explain what scenario it is talking about, it starts talking, e.g. that it is clear that it is about hybrid Addressables builds. I'm not actually sure how assetbundle specific this view is currently, the docs so far talk only about AssetBundles.
I also think that different variants between shaders is an important detail, that is not a duplicate in the sense of it being completely identical
2 participants
Summary
Skills/hybrid-duplication-audit, a Claude Code skill that audits asset duplication across the AssetBundle / ContentDirectory boundary in a hybrid Addressables 4.x build (some groups build to.bundlefiles, others build throughBuildPipeline.BuildContentDirectory).analyzeinvocation combining the Addressables build layout report with the content directory build output, then matches source-asset paths acrossaddressables_build_explicit_assets/addressables_build_data_from_other_assetsandcontent_layout_source_assetsto find what's genuinely duplicated.DuplicatedAssetCountcan read 0 even when real cross-boundary duplication exists.README.md,AGENTS.md, andDocumentation/analyzer.mdfor discoverability, following this repo's existing manual-index convention.Test plan
UnityDataToolfrom this branch and ranSkills/hybrid-duplication-audit/scripts/Compare-HybridDuplication.ps1against a real hybrid Addressables project, confirming it reproduces a hand-verified duplication total (three shared shaders, 214,824 bytes) exactly.-KeepDatabasepreserves the generated database and its tables are as documented.