Repository navigation
Conversation
The client submits a document packed by cwltool --pack: a $graph of processes in which a step's run names another entry or holds an inline body. A process is identified by hash_json of the process as submitted, with each run replaced by its child's identity, so a workflow's identity covers its children's (a Merkle tree). Nothing is normalised: another cwltool version, an edited doc or a renamed file gives another identity, which only costs a duplicate row, and DiracX never rewrites what the user tested. check_self_contained refuses a document that needs anything from outside itself: a $ key other than the top-level $graph and $namespaces, a run that is not an entry of the $graph or an inline body, and a File or Directory given by path or location rather than inline contents. subgraph returns the document one entry runs. Closes DIRACGrid#1106 Part of DIRACGrid#1080
CWL is written as YAML files, but services only receive JSON. pack runs cwltool's pack in-process, as cwltool --pack does, so the services compute the same identities as for any other client using cwltool. It reads nothing but local files, and runs check_self_contained on the result. DiracX's dirac: hints and fields say how the work is scheduled, not what runs, so they are taken out of the document and returned by the id of the object that held them, with the ids and names pack scoped inside them restored. The file names cwltool makes up for stdout and stderr shorthands depend on the directory, so they are replaced by fixed ones. The MCSimulation files of DIRACGrid#1080 are a fixture: packed, their transformations get the identities of the packed fixture in diracx-core, from any directory. cwltool is a diracx-cli dependency only. Closes DIRACGrid#1112 Part of DIRACGrid#1080
ryuwd
changed the title
Documentation build overview
27 files changed ·
|
|
| Dependency | Before | After | Change | Environments |
|---|
Implicit dependencies
| Dependency | Before | After | Change | Environments |
|---|---|---|---|---|
| argcomplete | 3.7.2 | Added | all | |
| cachecontrol | 0.14.4 | Added | all | |
| coloredlogs | 15.0.1 | Added | all | |
| cwl_upgrader | 1.2.15 | Added | all | |
| cwl_utils | 0.47 | Added | all | |
| cwltool | 3.3.20261106131234 | Added | all | |
| filelock | 4.2.4 | Added | all | |
| humanfriendly | 10.0 | Added | all | |
| lxml | 6.1.3 | Added | all | |
| mistune | 3.4.0 | Added | all | |
| msgpack | 1.2.3 | Added | {container-client, diracx-cli, gubbins-cli, gubbins-container-client} on all platforms | |
| mypy_extensions | 1.2.0 | Added | {default-gubbins, gubbins-cli, gubbins-container-client} on all platforms | |
| mypy_extensions | 1.1.0 | Added | {container-client, diracx-cli} on all platforms | |
| prov | 1.5.1 | Added | all | |
| pydot | 4.0.1 | Added | all | |
| rdflib | 7.6.0 | Added | all | |
| rich_argparse | 1.8.0 | Added | all | |
| ruamel_yaml | 0.19.1 | Added | all | |
| schema_salad | 8.10.20261106141337 | Added | all | |
| spython | 0.3.14 | Added | all |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1112.
pack(path)runs cwltool's pack in-process, ascwltool --packdoes, reading nothing but local files, so the services compute the same identities as for any client using cwltool. It takes DiracX'sdirac:hints and fields out of the document and returns them beside it, keyed by the id of the object that held them, as written in the YAML. The file names cwltool makes up forstdout/stderrshorthands depend on the directory, so they are replaced by fixed ones, and the result must passcheck_self_contained. Packed, the MCSimulation files of #1080 give the transformations the identities of the fixture in #1114, from any directory. cwltool is a diracx-cli dependency only.Blocked by #1114, whose commit this branch includes until it merges. The
diracsubmission command will use it.