test(cli): Test coverage for CLI commands#311
Conversation
blainekasten
force-pushed
the
add-cli-tests
branch
2 times, most recently
from
4a32ca0 to
99db80a
Compare
blainekasten
changed the title
blainekasten
changed the title
| text=True, | ||
| ) | ||
|
|
||
| command_result = run_command(command) |
There was a problem hiding this comment.
This only checks whether stdout is parseable by jq, not whether the CLI command itself succeeded. Since jq exits 0 on empty input, a command that fails before producing output can still make this test pass; we should assert command_result.returncode == 0 before running jq.
There was a problem hiding this comment.
So this was kind of intentional. I didn't specifically care if the command failed or succeeded as much as I wanted to validate that users can script against this command with jq
| ) | ||
| if "y" not in confirm_response.lower(): | ||
| click.echo(json.dumps({"status": "Cancel not submitted"}, indent=4)) | ||
| click.echo(json_lib.dumps({"status": "Cancel not submitted"}, indent=4)) |
There was a problem hiding this comment.
When the user types "n" at the interactive prompt (non-JSON mode), this still outputs raw JSON. Should be a plain message here since we're in the interactive (non-json) path: click.echo("Cancel not submitted")
…ith improved environment configuration
blainekasten
force-pushed
the
add-cli-tests
branch
from
642415d to
6666787
Compare
| ): | ||
| click.secho("Invalid hardware selected.", fg="red", err=True) | ||
| click.echo("\nAvailable hardware options:") | ||
| click.secho("Invalid hardware selected.", fg="red", err=True, file=sys.stderr) |
There was a problem hiding this comment.
both err=True and file=sys.stderr are passed to click.secho. Click's err=True already redirects to stderr. Passing both is redundant and may behave unexpectedly depending on Click version.
can just be err=True
2 participants
Summary
Adds integration-style CLI tests that run the real
togetherbinary against the test base URL to verify json mode is pipable to jq, and have tests to protect core business logic on every command.Tests
tests/cli/test_json_mode_pipeable_to_jq.pytests/cli/test_endpoints.pytests/cli/test_models.pytests/cli/test_files.pytest_beta_clusters.pytest_beta_jig.pytest_evals.pytest_fine_tuning.pytest_main.pyCLI / product
together files checktogether files deletetogether files listtogether files retrievetogether files uploadtogether fine-tuning canceltogether fine-tuning deletetogether fine-tuning downloadtogether fine-tuning listtogether fine-tuning list-checkpointstogether fine-tuning list-eventstogether fine-tuning retrievetogether beta clusters createtogether beta jig list--non-interactivemode toclusters create. This will be a future foundation option but right now is needed to make tests workOther
.stats.ymlupdate from Stainless.uv.lockbump tied to the test/tooling stack.Test plan
./scripts/test tests/cli -vvFixes DX-360