Run your suites in a pipeline with the CLI

Run a suite or a whole project from your own CI pipeline, and see the result in {{brand}}.

In short

The {{brand}} CLI, kamocore, runs a suite or a whole project from your own machine or CI pipeline. Requests leave from that machine, so private APIs work and nothing counts against your request allowance. The result shows in {{brand}} as "Reported by the CLI".

Before you start

• Who: you need Run on the suite or project and an access token with the scopes ApiTester:read and ApiTester:write.
• Plan: requests leave your machine, so a CLI run does not count against your request allowance. Needs .NET 10 on the machine.
• First: name the hosts the run may send to with --allow-hosts, and give the pipeline the secrets as environment variables.

Steps

1. Open a suite and choose "Run in CI" in its … menu - or, on a project's requests, open the chevron beside "Run all" and choose "Run in CI…". Choose the "Environment". For a project, open "Write the pipeline yourself" to see the command and the pipeline steps.
2. Install the CLI with the install line the panel shows. It needs .NET 10.
3. Create an access token in "Settings", under "Access tokens", with the scopes ApiTester:read and ApiTester:write.
4. Keep the token in a secret variable of your pipeline and hand it to the CLI as the environment variable KAMOCORE_TOKEN.
5. Run a suite: kamocore run --suite <suiteId> --environment <name> --allow-hosts <host> --junit results.xml - or a project with --project <projectId>.
6. Publish results.xml as your pipeline's test results. It is a JUnit file.

Result: the run shows in the suite's run history, and turns the suite and its requests green or red.

Say where the run may send

Every run names the hosts it may send to with --allow-hosts, and it sends nowhere else - no request and no secret. Without --allow-hosts the run does not start.

• Separate several hosts with commas: api.staging.example.com,localhost:5000.
• A host with a port allows that port only. *.internal allows every name under it, and 10.0.0.0/8 allows a range.
• "Run in CI" fills in the host of the environment's base URL. Add any other host your steps call.
• Link-local and cloud metadata addresses are never sent to.
• --allow-any-host lets the suite decide instead. Use it only for suites you trust.

Secrets are yours to give

Secrets never leave {{brand}}: the CLI gets the steps without one secret in them. You give each secret to your pipeline, and the CLI reads it from an environment variable. The variable's name is the secret's name in upper case, with every character that is not a letter or a digit written as an underscore.

Before it sends anything, the CLI lists each secret it reads and each one that is missing, by name. It never shows a value.

In Azure Pipelines

1. In "Run in CI", under "Pipeline format", choose "Azure template" or "Azure script".
2. Copy the snippet into your pipeline's YAML file. The template form needs templates/kamocore-run.yml in your repository.
3. Add a secret pipeline variable named kamocoreToken that holds the access token, and one for each secret the snippet lists.
4. Run the pipeline. Its Tests tab shows every step of the suite.

For any other CI, choose "Shell": it sets the same environment variables and runs the same command.

Check every pull request

1. On a project's requests, open the chevron beside "Run all", choose "Run in CI…" and choose the "Environment".
2. Under "Check every pull request", choose the "Repository" your organization connected and its "Repository path". With none connected, choose where "Pipelines run on": GitHub Actions, Azure Pipelines, GitLab CI or Bitbucket Pipelines.
3. Type "Your password" and press "Make the token and fill the file".
4. Under "1. Add this file", press "Copy the file" and save it in your repository under the name shown.
5. Under "2. Add one secret", add the secret it names, with the value shown, where it says. The value is shown only now. Add any other secret it lists the same way.
6. Open a pull request: the checks run on it and fail it when a check fails.

The token has the two scopes a run needs and an end date the panel shows; {{brand}} emails you a week before it ends. Your password is asked again because the token works without you signed in.

Good to know

• Name the environment as the project calls it, or by its id. Without --environment the run uses the project's default environment. kamocore run --help lists every option, such as --fail-fast and --timeout.
• The run is kept with the pipeline's name and the commit when your pipeline says them.
• It is history, not evidence: {{brand}} did not send the requests itself, so the run is never used as evidence for a verification.
• Secrets and personal data are masked again when the result arrives. Add --report summary to post no body and no header, or --report none to post nothing.
• The CLI reads the token only from the environment, never from an option.
• To start the affected suites after a deploy, see /help/api-testing/tests-after-a-deploy.

If something goes wrong

• The run does not start: add --allow-hosts with the hosts your steps call.
• A secret is listed as missing: set its environment variable in the pipeline.
• A message that begins "This project has no environment named" or "More than one environment of this project is named" means the name does not fit one environment. Use the name as the project spells it, or its id.
• A message that begins "This project has more than" means the project is too large for one run. Run one test group at a time.
• "The suite or project changed while it ran, so this result no longer fits it. Run it again."

Related

• /help/api-testing/tests-after-a-deploy
• /help/api-testing/cli-agent
• /help/settings/access-tokens