Making a game testable from outside
A game has no function to call and no return value to assert on. A small read-only global changes that, and reporting unjudgeable rules as skipped rather than passed keeps the green build honest.
Automated tests for a game are awkward in a way that automated tests for most software are not. There is no function to call and no return value to assert on. The thing you want to check is whether a moving picture is behaving, and the only interface is a canvas element that tells you nothing.
The approach used here is to make each game expose a tiny, deliberately boring object that a test harness can read from the outside. It is about twelve lines per game, and it is the reason every other quality gate on this site is possible.
The contract
Every game defines one global:
window.__GAME__ = {
api: 1,
session: 'cozy',
get state() { return state; }, // 'playing' | 'paused' | 'over'
get score() { return given; }, // any monotonic progress number
get tick() { return tick; },
get field() { return { x: OX, w: W, h: H, screenW: SW }; },
start() { state = 'playing'; },
pause() { state = 'paused'; },
resume() { state = 'playing'; }
};
That is the whole interface. Note what is absent: no event emitters, no test hooks scattered through the game logic, no debug mode, no way to mutate internal state. The contract is read-only apart from three lifecycle calls, and everything it exposes is something the game already knew.
From those few values a surprising amount is checkable. Does anything advance when input
arrives? Sample score and see whether it ever changes. Does the round end when left
alone? Watch state for twenty seconds. Is the playable region the right shape in a
wide window? Read field, which is the subject of
its own note. Does the game restart cleanly? Call
start() and check that state is playing and score is back at zero.
Why api: 1 is there from the start
The version number does nothing today. It exists so that the checker can say “this game speaks a contract version I do not understand” and stop, instead of reading fields that have quietly changed meaning and reporting confident nonsense. Adding a version to a format after the format has shipped requires guessing what the unversioned ones meant; adding it at the start costs one line.
Declaring a variant instead of loosening a rule
The session field is the most useful thing in the contract, and it took a while
to arrive at.
The default assumption behind the play gate is that a game is a run you survive: it should end if you stop playing, and a round that lasts under three seconds means the game is unplayable. Both rules caught real problems. Both are wrong for some perfectly good games.
A game that ends after a single action — one shot, one throw — always scores badly on mean survival, because the automated tester fires immediately and never deliberates. A cozy game with no fail state must not end when idle; ending is the bug. Two different genres, two different rules, and in both cases the temptation is to weaken the general rule with a condition.
Weakening it is the wrong move, because every future game then falls into the exemption without anyone deciding that it should. Instead the game states which kind of thing it is, and the checker branches on the declaration:
const DECLARED_SESSIONS = ['single-shot', 'cozy'];
const session = await page.evaluate((allowed) => {
const s = window.__GAME__ && window.__GAME__.session;
return allowed.includes(s) ? s : 'run';
}, DECLARED_SESSIONS);
An unrecognised value falls back to the strict path. The exception lives in the game's own source where a reader will see it, the strict rule stays strict for everything else, and nothing opts out by accident. (Wiring that declaration through the collector was itself forgotten for two weeks, with the gate passing its unit tests the whole time — that story is here.)
The games that came first
A dozen games on this site predate the contract. Retrofitting all of them would have been a day's work with a real chance of breaking games that currently work, so they run in a legacy mode where the harness judges only what it can observe from outside: frame rate, load time, console errors, whether the canvas is drawing, and whether the screen changes in response to input.
The critical design decision is that unjudgeable rules are reported as skipped, not as passed:
skipped.push(`${at}: termination, idle-end, restart and play band
not checked — no window.__GAME__ contract`);
The report distinguishes three states — failed, passed, and not checkable — and the summary prints the skips. This sounds pedantic until you have a suite where missing instrumentation silently counts as success, at which point the green build becomes a measure of how much is not being tested.
Measuring frames without trusting the game
One thing deliberately sits outside the contract. Frame rate is measured by injecting a probe
that wraps requestAnimationFrame and counts calls, rather than asking the game to
report its own frame rate.
The reason is straightforward: a game reporting its own performance reports the performance of its logic loop, not of the thing the player sees. A game can be stuttering visibly while its internal counter is perfectly happy. Measure the browser's actual paint cycle and there is nothing for the game to be wrong about.
This also produces the one case where the harness has to be careful about honesty. A game that reloads itself on game over resets the probe's counter, and the difference between consecutive samples goes negative — a five-second window once reported −134.9 frames per second. The gate now treats any non-positive frame rate as a broken measurement rather than a failing game, because those are different claims and only one of them is about the game.
What this buys
Roughly twelve lines per game, written once, make it possible to check twenty games on every build for: load under two seconds, average frame rate above 50, no five-second window under 30, heap growth under 2.5× across a minute of play, touch handlers present, no console errors, no failed network requests, input producing progress, correct termination behaviour for the declared genre, and a playing field with the right proportions on a desktop monitor.
Without the contract, every one of those would be a manual check performed by a person who is, by the twentieth game, no longer really looking.