You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Replace voluptuous with a faster schema validator #8
When profiling taskgraph in Gecko, we discovered that voluptuous is responsible for a significant performance hit to Taskgraph. So much so that we disbale schema validation entirely when taskgraph.fast is set. We also avoid using voluptuous to set default values and instead set defaults via a transform.
There are many more performant validation libraries out there. In another project, I once successfully replaced voluptuous with validx (but we should investigate all options).
I have tried migrating over the task_decision_schema to validx and with careful ordering of the OneOf attributes to prevent error creation and fallback and general attention to validx best practices, validating the schema drops from ~4s to ~1s over all calls within a large decision task when run on my computer.
This was an experiment to see whether this would make any impact and it does show promise, but only if we can add native extensions (validx uses Cython) to the decision task, and migrate over more schemas.
@abhishekmadan30 has been working on this in #844. There are some complications around getting it landed, due to it being written in rust, but those are being worked through.
Activity
Alphare commentedon May 28, 2024
I have tried migrating over the
task_decision_schematovalidxand with careful ordering of theOneOfattributes to prevent error creation and fallback and general attention tovalidxbest practices, validating the schema drops from ~4s to ~1s over all calls within a large decision task when run on my computer.This was an experiment to see whether this would make any impact and it does show promise, but only if we can add native extensions (
validxuses Cython) to the decision task, and migrate over more schemas.See https://hg.mozilla.org/try/rev/fcdcde5e36caba6bd74bf7c5ee41114874fec397
bhearsum commentedon Jul 4, 2025
pydantic seems to be the standard for this sort of thing these days. According to https://validx.readthedocs.io/en/latest/benchmarks.html, validx is ~5x/faster than pydantic 1.x. According to this issues, pydantic 2.x is massively faster than 1.x when it comes to validation.
We should do a small scale test of validx vs. pydantic to help inform which we should choose.
ahal commentedon Jul 4, 2025
Yeah, it looks like validx isn't well maintained anymore, so I'd lean towards pydantic between the two of them (even if 2.x is slightly slower).
I was also meaning to look into https://github.com/jcrist/msgspec, which I saw someone recommend as a faster, lighter weight alternative to pydantic.
bhearsum commentedon Jul 4, 2025
https://www.reddit.com/r/Python/comments/w68nv3/a_quick_benchmark_comparing_pydantic_v2_with/ suggests that msgspec is significantly faster than even pydantic 2.x. https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b743 has similar results (albeit that's from the msgspec author; I don't know if there's any bias there.)
bhearsum commentedon Nov 26, 2025
@abhishekmadan30 has been working on this in #844. There are some complications around getting it landed, due to it being written in rust, but those are being worked through.
marco-c commentedon Sep 7, 2026
@ahal @bhearsum can we close this?
jcristau commentedon Sep 8, 2026
Yes we can!