Run your tests from CI/CD

The Studio's CI/CD screen writes the pipeline file for GitHub Actions or GitLab CI. This guide covers what that file runs, the three places the job can run, secrets, the JUnit XML and evidence it leaves, and the errors a CI machine meets.

The pipeline file

On the CI/CD screen, pick GitHub Actions or GitLab CI, the browsers, the checks and where the job runs, then generate. The Studio validates the file locally first, and can commit and push it from the same screen. From a terminal, nexus init --ci github or nexus init --ci gitlab writes the same file.

With more than one browser, GitHub runs one job per browser, with fail-fast: false, and each one adds --project=<browser>. The checks you pick add --visual, --a11y, --chaos=, --loop= or --no-retry.

Exit codes and JUnit XML

nexus ci runs the tests you name (TC1,TC2) or --all, and exits 0 when they pass, 1 on a failure and 2 when nothing ran or the command was wrong. It writes JUnit XML to evidence/ci/junit.xml; --junit=<path> writes it somewhere else.

Where the job runs: three options

Every option needs a machine with an activation: the engine does not run suites on a machine that has none, and each activated machine counts toward your plan (plans and machines). A cloud-hosted runner, runs-on: ubuntu-latest, starts a new virtual machine for every job, so it never keeps one; use a persistent runner of your own.

Option A: a self-hosted runner on the Studio's machine

1. On the Windows 11 PC or mini PC where Nexus Studio is installed and activated, register a runner: on GitHub, in the repository's Settings, Actions, Runners, New self-hosted runner; on GitLab, in Settings, CI/CD, Runners.
2. On the CI/CD screen, choose My machine. The file then reads runs-on: [self-hosted] on GitHub or tags: [self-hosted] on GitLab.
3. Run the runner as the Windows user that activated the Studio. The engine reads the activation from that user's profile, in .sentinel-studio\license.json, so a runner service under another account finds none.

The job uses this machine's activation: one activation per machine, already counted in your plan, and no key in the pipeline. Use a self-hosted runner with a private repository only, as the Studio warns: on a public one, anyone who opens a pull request can run code on your machine.

Option B: a runner on a VPS, in a container

A VPS with Ubuntu 24.04 LTS x86_64, Docker Engine and Compose v2, reached with SSH key authentication. It counts as one machine of your plan.

1. On the CI/CD screen, choose My machine + Docker. Besides the pipeline file, the Studio writes docker-compose.sentinel-runner.yml, Dockerfile.runner and the runner's two entrypoint scripts into the workspace. Dockerfile.runner starts from mcr.microsoft.com/playwright:v1.62.1-noble.
2. The screen shows the commands to paste in an SSH session on the VPS, not in the Studio's own terminal. For GitHub it fetches a registration token, valid for 1 hour, and sets REPO_URL, RUNNER_TOKEN and RUNNER_NAME; for GitLab, GITLAB_URL and GITLAB_RUNNER_TOKEN. Then:

docker compose -f docker-compose.sentinel-runner.yml up -d --build sentinel-runner
docker compose -f docker-compose.sentinel-runner.yml up -d sentinel-runner-gitlab

The first line is for GitHub, the second for GitLab. Inside a Linux container the license identifies the machine by /etc/machine-id: mount the host's read-only, /etc/machine-id:/etc/machine-id:ro, and keep the .sentinel-studio folder of the runner's home on a volume, so the activation survives a rebuild. Activate once, by hand, with nexus license activate and your key, never from the pipeline file.

Today the runner image is built on the VPS from Dockerfile.runner. The repository has no container registry yet: a published image on GitHub Container Registry, the one the Studio's VPS runtime runs, ships with the next release.

Option C: the CI job starts a run on a host over SSH

The job stays small, on any runner, and the suites run on a host you own, with the workspace and the engine at a path there, as the Studio's SSH runtime expects. The CI/CD screen does not generate this job; these are the options the Studio's own SSH runtime uses:

ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=8 <user>@<host> "cd <path> && node bin/nexus.js ci --all"
scp -r -o BatchMode=yes -o StrictHostKeyChecking=yes <user>@<host>:<path>/evidence ./evidence

• Key authentication only. BatchMode=yes never asks for a password. Write the private key from a CI secret to a file readable only by the job, and delete it when the job ends.
• StrictHostKeyChecking=yes. The host's key must already be in the runner's known_hosts: keep the host's public key line in a CI variable and write it there before the first ssh. Never set it to no.
• The exit code comes back. ssh exits with the remote command's code, so the step fails when the suite fails, and scp brings the evidence, JUnit XML included, back for the artifact step.
• The host checks its own license and counts as one machine of your plan.

Secrets

• Store tokens, webhook URLs and SSH private keys as CI secrets: on GitHub in Settings, Secrets and variables, Actions, or with gh secret set <KEY>; on GitLab as masked variables in Settings, CI/CD, Variables. Never in the repository, the pipeline file or a log.
• The generated GitHub workflow reads only the integration keys, from secrets.<KEY>, and only on the Nexus steps: TELEGRAM_*, SLACK_ENABLED, SLACK_WEBHOOK_URL, JIRA_* and the notification switches. The CI/CD screen lists a gh secret set command for each; gh asks for the value, and nothing is printed.
• The license key does not belong in CI. Activate a persistent runner once, by hand: the key is sent once and is not kept. If your process requires the key in the CI system anyway, store it only as a masked secret; nothing in the generated pipeline reads it.
• Never echo a secret in a step. The engine scrubs secrets from the text it files as evidence, such as JUnit failure messages and logs; it does not scrub the job's own console output, and the CI service masks only values stored as secrets.

JUnit XML and evidence as artifacts

• GitHub uploads evidence/ as nexus-evidence, one nexus-evidence-<browser> per browser with a matrix, whether the run passed or failed. The JUnit XML is inside, at evidence/ci/junit.xml.
• GitLab keeps evidence/ and reports evidence/ci/junit.xml as a JUnit report, so failed tests show in the merge request.
• From the Studio, the CI/CD screen can start a GitHub workflow run and download its evidence into evidence/ci-remote/run-<id> in the workspace.
• Where the copy lives. An artifact is a copy of the evidence, screenshots and logs included, on your CI service's storage; on a hosted service that is someone else's hardware. Set the artifact retention your policy allows, or keep the evidence on your runner and upload only evidence/ci/junit.xml.

Troubleshooting

SEAT_LIMIT on a cloud-hosted or ephemeral runner

Each new virtual machine is a new machine to the license, so an ephemeral runner would need an activation for every job. Once every machine of your plan is active, activation is refused with SEAT_LIMIT: “All machines on your plan are in use. Deactivate one in your account.” Use a persistent self-hosted runner (option A or B), and deactivate machines you no longer use under License & machines in your account (more on SEAT_LIMIT).

The runner was offline for days

A machine's license must validate at least every 7 days. After that it gets 4 more days with a notice, then turns read-only: suites do not run until a check succeeds, and the evidence stays readable. On a runner back online, nexus license renew checks again (working offline).

“no /etc/machine-id” in a container

The container has no machine ID for the license to read. Mount the host's: -v /etc/machine-id:/etc/machine-id:ro, or the same line under volumes: in the compose file.

The job exits with code 2

Nothing ran: no test matched the IDs, or the command was wrong. Check the IDs passed to nexus ci, in tc_ids on GitHub or TC_IDS on GitLab.

Host key verification failed

StrictHostKeyChecking=yes refused a host that is not in known_hosts, or whose key changed. Add the host's key from a source you trust, after checking its fingerprint on the host itself. Do not turn the check off.