Skip to content

WIP minor refactoring of autogen code - #6573

Closed
Sankalp-Mittal wants to merge 9 commits into
sankalp-mittal/pydabs-jobs-tests-normalizefrom
sankalp-mittal/pydabs-autogen-refactor
Closed

WIP minor refactoring of autogen code#6573
Sankalp-Mittal wants to merge 9 commits into
sankalp-mittal/pydabs-jobs-tests-normalizefrom
sankalp-mittal/pydabs-autogen-refactor

Conversation

@Sankalp-Mittal

Copy link
Copy Markdown
Collaborator

Summary

Minor follow-up refactoring on the PyDABs autogen code, addressing non-blocking review comments left on earlier PRs (#6421, #6528). Stacked on top of #6553.

Details

  • generated_test_cases.py
    • Use _STAGE_RANK.get(prop.stage, 0) instead of a bare dict lookup, so an unknown/absent launch stage is treated as GA rather than raising KeyError and killing codegen (matches the Go launchStageRank semantics).
    • Route _synth_ref's module resolution through _module_of to drop the duplicated get_package(...) + assert logic.
  • _resource_test_case.py: add __test__ = False to TestCase so pytest stops trying to collect it as a test class.
  • generated_dataclass.py: collapse the four near-duplicate GeneratedField(...) branches in generate_field into one base value plus per-branch dataclasses.replace(...) deltas (keeps pyright's type checking intact).

All changes are behavior-preserving: regenerating the PyDABs output produces a byte-identical tree. Codegen unit tests (38) and test_resources.py (223) pass.

This pull request and its description were written by Isaac.

Sankalp-Mittal and others added 9 commits September 8, 2026 17:14
Map the SDK time.Time type to Python str (an RFC3339 timestamp string,
matching how the Go side models it in libs/dyn/convert/sdk_native_types.go)
and mark it terminal for reachability, so codegen no longer fails on it.
This lets resources.Secret (whose expire_time field is a *time.Time) drop
off the denylist and generate.

Co-authored-by: Isaac <no-reply@databricks.com>
Adds the secrets-support fixture required by test_python_support_coverage.
It loads a Secret from YAML and Python, runs a secret_mutator over both, and
exercises the expire_time field (RFC3339 string) end to end. value is a
default-less variable reference supplied at validate time, as the secret
value validator requires; direct-engine only.

Co-authored-by: Isaac <no-reply@databricks.com>
_synth_scalar only knew string/int/float/bool, so a required time.Time field
would fail codegen with "Unknown primitive: time.Time". Add a case producing an
RFC3339 placeholder string (time.Time generates as str). No effect on current
output: the only time.Time field, secrets.expire_time, is optional and dropped
by the field policy before synthesis.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
- Use _STAGE_RANK.get(prop.stage, 0) so an unknown/absent launch stage is
  treated as GA instead of raising KeyError and killing codegen.
- Route _synth_ref's module resolution through _module_of to avoid
  duplicating the get_package + assert logic.
- Add __test__ = False to TestCase so pytest stops trying to collect it.

Co-authored-by: Isaac <no-reply@databricks.com>
The four GeneratedField branches differed in only a couple of kwargs; build
one base field and override the per-branch deltas via replace(), keeping
pyright's type checking intact.

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 7bc1873

Run: 34244551746

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux 1 275 34 10:05
💚​ aws windows 1 277 32 7:26
🔄​ azure linux 1 1 273 34 14:14
💚​ azure windows 1 276 32 8:03
🔄​ gcp linux 1 1 274 34 12:44
Test Name aws linux aws windows azure linux azure windows gcp linux
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalFileSync ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestSyncNestedFolderDoesntFailOnNonEmptyDirectory ✅​p ✅​p ✅​p ✅​p 🔄​f
Top 11 slowest tests (at least 2 minutes):
duration env testname
4:28 gcp linux TestFilerWorkspaceFilesExtensionsReadDir
4:00 azure windows TestAccept
3:19 aws windows TestAccept
3:09 azure windows TestFilerWorkspaceFilesExtensionsReadDir
2:44 azure windows TestFilerWorkspaceFilesExtensionsStat
2:39 aws windows TestExportDir
2:22 azure linux TestSyncIncrementalFileSync
2:22 azure linux TestFilerReadDir/workspace_files
2:14 gcp linux TestFilerRecursiveDelete/workspace_files
2:12 azure linux TestExportDir
2:09 azure linux TestFilerWorkspaceFilesExtensionsReadDir

Base automatically changed from sankalp-mittal/cli-secrets-support to sankalp-mittal/pydabs-jobs-tests-normalize September 8, 2026 16:18
@Sankalp-Mittal
Sankalp-Mittal deleted the branch sankalp-mittal/pydabs-jobs-tests-normalize September 8, 2026 16:18
@Sankalp-Mittal

Copy link
Copy Markdown
Collaborator Author

Superseded by #6575 (this PR was auto-closed when the stack it was based on merged to main; GitHub won't reopen a closed PR whose base branch was deleted). Continuing there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants