Run checks
A check is one thing that can pass or fail on a machine: a test command, a Docker smoke, a signing verification. A run is one or more checks started together. This guide covers running checks in the desktop app, reading what they say, starting runs on a machine from the web, and letting them run by themselves.
It assumes the app is installed and this machine is paired: see Install the desktop app and Pair a runner. With nobody signed in, the app runs checks too, and keeps their results on the machine: see Start without an account. For vitals in a terminal, see The vitals command line.
Without a runner: browser checks
To check that a page loads and says what it should, you don't need a machine of your own. On a project's Health tab, Browser checks → Add browser check has ShipVitals' cloud open the page in a real browser on a schedule, with a screenshot of each run that loads. It runs no code of yours: your own Playwright or other tests run on a paired machine (this guide) or in CI (see Send test results and your own metrics). See Browser checks.
This Mac
This Mac sits at the foot of the sidebar with one word: Ready, Running, Paused or Needs setup. Needs setup means a linked check waits for review, or a tool a check needs is missing (Doctor says which); the menu bar panel says "This Mac needs setup" too. Press it, or press g then m. On Windows it's This PC.
Its header says the machine's role and how many checks are linked ("Laptop · 12 checks"). Its tabs are Doctor, Links, Privacy, Relays, Background and Storage. Links, Privacy and Relays belong to one workspace; when the machine runs checks for more than one, a Workspace switcher appears above them.
Anything that needs you shows at the top with its next step, such as "Can't find ~/code/120." with Choose folder.
Doctor
Doctor says what this machine can run: "This Mac can run 11 of 12 checks." Each line names a tool the linked checks need and where it was found ("Node 24.22 · found through your shell"), or what's missing:
- "Android SDK · not found. 1 check needs it." with the command to install it, Copy and Show me how.
- "Docker · not running. 2 checks need it." with Start Docker.
- An optional scanner such as Trivy, with Why we pin this: ShipVitals offers one pinned version and never installs a scanner by itself.
After fixing something, press Check again.
Background
Background holds the machine's role (Laptop or Always on), Checks at once, Pause, Open at login, Notifications and Global shortcut. A laptop waits for power for runs it didn't start by hand.
To pause: press Pause and choose Pause for 1 hour, Pause until tomorrow or Pause until I resume. "A check that's running finishes." Resume starts it again. The menu bar panel has the same Pause.
Storage
Storage shows how much the runs kept here use ("1.2 GB of 5 GB used"). Logs, reports and screenshots stay on this machine; when they reach the limit the oldest runs go first, and every check's latest result is kept.
- Choose a limit under Keep up to, or press Clean up now.
- Read Remove older runs? and press Remove older runs. The page says how many went. Results already in the cloud stay.
Approve checks
Nothing runs until a person at the machine has read and approved its exact command. An approval covers that command, on that branch, on this machine: the code on the branch runs with it.
- Open This Mac → Links. Each linked folder shows its project, "Approved branch: main" and "5 of 6 checks approved".
- A project with checks to read says "1 check waits for someone here to review it." Press Review.
- Read each command, where it was found and what it needs. Untick what you don't want; Edit changes a command.
- Press Approve 1 check, then Approve in the system dialog, which lists each command again.
Link a folder links more projects (it opens Where's your code?).
When a check changes
If a check's command changes (in the repo's Makefile or .shipvitals/checks.yml, say), it stops on this machine. Run all's sheet counts it among the checks that won't run, and on the board it's Couldn't run: "The check changed. It won't run until someone reviews it", which is no proof for that run. Links shows it waiting. Press Review: the system dialog shows "Changed since you approved it:" with the old and new lines. Approve it again, or leave it.
On the web, the same check shows "Waiting for approval on MacBook Pro". It can only be approved at the machine.
Forget approvals, at the foot of Links, stops every check for the workspace on this machine until someone reviews it again. The links and their results stay.
Run all
- Press
⌘R(or Run → Run All, Run all on Runs, or Run all in the menu bar panel). This opens a sheet; nothing starts yet. - The sheet says what would run: "3 projects · 12 checks · about 38 min" ("first time, no estimate" before there's history), then each project with its commit, checks and time. It warns before you start:
- "2 files changed, won't count for the gate"
- "on feature/x, not the approved branch"
- "1 check can't run here: Android needs the Android SDK" with Fix, which opens Doctor
- "On battery (64%). Plug in, or runs may be slow."
- Tick Only what changed to skip checks whose files didn't change since their last pass, Choose checks to pick them, or Stop on first failure.
- Press Start (or
Enter).
The run board opens: a row per check, grouped by project. Its header reads "Running 6 of 12 · about 14 min left · 4 passed · 1 failed". Failed rows rise to the top, and a finished project's passed rows fold into "All 5 passed" (Show passed opens them). A running row shows its step ("Running tests · 120 of 214") and a clock; a waiting one says why ("Waiting for the simulator"). Select a row, or press j and k, to see its log beside the list.
Closing the window doesn't stop the run. The menu bar panel shows its progress.
To stop it, press Stop (⌘.). Checks still running come back Cancelled.
The verdicts
When the run ends, the board becomes Verdicts, one line per project:
- "smart-home isn't shippable. Android: 3 of 88 failed."
- "agentgarm is unproven. Files were changed, so its results don't count."
- "120 is shippable at a7d311."
Under them: the counts, the time and All results sent (or Results stay on this machine).
- When something failed, press Re-run failed (
⇧⌘R) to run just those checks again, on the machine that ran them. - Otherwise press Done.
A project's first results wait until you've seen what's sent: Before web's first upload. See What leaves your machine.
Read a result
Every check ends in one of these, always with its word and a reason:
- Word
- Passed
- Means
- It ran and its results were read: "Passed · 214 tests"
- Counts for the gate
- Yes
- Word
- Ran
- Means
- It exited 0 with nothing to count
- Counts for the gate
- Only if the check says so
- Word
- Failed
- Means
- The product did something wrong: "3 of 214"
- Counts for the gate
- Blocks
- Word
- Flaky
- Means
- It failed, then passed on retry on the same commit
- Counts for the gate
- Passes, and is noted
- Word
- Couldn't run
- Means
- The machine stopped it, not the product: "Docker isn't running"
- Counts for the gate
- No proof, never a failure
- Word
- Skipped
- Means
- Deliberately not run: "needs Xcode", "nothing it covers changed"
- Counts for the gate
- No proof
- Word
- Cancelled
- Means
- A person stopped it
- Counts for the gate
- No proof
- Word
- Interrupted
- Means
- The machine stopped mid-run: "The machine went to sleep"
- Counts for the gate
- No proof
- Word
- Out of date
- Means
- True, but about an older commit
- Counts for the gate
- No proof
| Word | Means | Counts for the gate |
|---|---|---|
| Passed | It ran and its results were read: "Passed · 214 tests" | Yes |
| Ran | It exited 0 with nothing to count | Only if the check says so |
| Failed | The product did something wrong: "3 of 214" | Blocks |
| Flaky | It failed, then passed on retry on the same commit | Passes, and is noted |
| Couldn't run | The machine stopped it, not the product: "Docker isn't running" | No proof, never a failure |
| Skipped | Deliberately not run: "needs Xcode", "nothing it covers changed" | No proof |
| Cancelled | A person stopped it | No proof |
| Interrupted | The machine stopped mid-run: "The machine went to sleep" | No proof |
| Out of date | True, but about an older commit | No proof |
Press a check on the board to open its result.
A failed check leads with what broke: the check, "3 of 214 failed", the commit, the time and the machine, then the first error with its file and line. Below:
- Open in editor (
e) opens that file and line in the editor chosen beside it. - Re-run runs this check again.
- Copy for your agent (
⇧⌘C) copies the check, its command, the first errors and the commit as a prompt to paste into your coding agent. "Copied. Paste it into your agent." - Save to the repo writes the project's checks to
.shipvitals/checks.yml, uncommitted, so another machine can share them. Edit opens the check's review. - "Passed yesterday on b81d0e. 3 commits since, by 1 person." with Compare, then the other failures grouped by their first error, the evidence and the log.
Couldn't run leads with the fix instead: "Couldn't run: Docker isn't running." and "Nothing was tested, so this says nothing about agentgarm." Press Start Docker (or Open System Settings), then Check again, then Re-run.
On the web, a result shows the first error that was sent: "On the web only the first error that was sent is shown. The full log stays on the machine that ran it."
Start a run from the web
Runs appears in the sidebar once the workspace has a paired machine (g then r). From a browser or a phone you can start runs on a machine you paired, or on one that lets editors start runs (see Pair a runner). Only checks already approved on that machine run.
From Runs:
- Press Run all.
- Under Run on, choose the machine: "Studio Mac mini · ready", "MacBook Pro · asleep, runs when it wakes". The sheet says when it will start: "Studio Mac mini checks in about every 30 seconds, so it usually starts within 30 seconds."
- Press Start. The run opens as Queued, then Running, then its verdicts.
From a project's Health tab, an empty Tests or Signing cell offers Run on Studio Mac mini (or Run on… and a list of machines). It says "Queued. Studio Mac mini checks in about every 30 seconds." When you can't start runs there, it says which machine can ("Studio Mac mini can run this.") or offers Set up a runner. The release gate page has the same buttons: see The release gate.
Runs lists the current run under Now and the rest under Earlier, each with its machine and how it started (By hand, From the web, On a schedule, On a commit). A run you start from the web on a machine you paired counts as By hand; From the web is someone else's run on it.
Run on a schedule or on each commit
Both are off until someone turns them on. They run approved checks only and never change your working tree.
A nightly run
- Open the project, then Health, and find Runs (it appears once the workspace has a paired machine).
- Under Schedules, tick Every night at 02:00, or at next wake, and set the Time of day. Missed runs collapse into one.
- Run again before proof runs out re-runs checks before the release gate's proof expires, "So a quiet project stays Shippable."
Viewers read these; the project's editors change them.
Commit runs
Commit runs are turned on at the machine, because they let anyone who can push run code on it.
- Open This Mac → Links and find the project.
- Tick Commit runs.
- Read Run checks on every push to main? "Anyone who can push to main will have code run on this Mac, as you. Turn this on only for repositories whose contributors you trust." Press Turn on commit runs, or Not now.
A check runs only when the folder is on that commit; otherwise it says "This folder is on feature/x, 3 commits away". Tick Clean checkout to test every commit in a separate copy in ShipVitals' own folder, leaving your working tree alone.
A clean checkout runs nothing the repository names, Git LFS's filter included, so its Git LFS files are only pointers there. A check that covers any of them (the paths it names, else its working folder) doesn't run against them: it's Couldn't run, "Its Git LFS files are only pointers in the clean checkout. Turn Clean checkout off to run it in your folder". Untick Clean checkout for that link, or give the check paths that leave the Git LFS files out.
The project's Schedules then shows On every push to main, "Turned on at MacBook Pro." Editors can untick it there to switch it off for the project.
Offline
Checks keep running without a connection. The line above every page says "Offline. Showing figures from 09:14. Changes are off until you're back online." and, when results are waiting, "3 results waiting to send". They send, in order, when the connection returns. The menu bar panel says "Offline. Runs still work, and results send when you're back."
A part of a page that wasn't kept says Not available offline.