A gate that graded the opposite

· by · 1111 words

Five new unit tests passed while the branch they covered was unreachable from any real game. A test that builds its own input cannot verify that anything produces that input.

Every game on this site has to pass an automated play test before it can be published. One of the rules in that test reads, in effect: if nobody touches the controls for twenty seconds, the round must end. A game that runs forever while the player does nothing is a game whose fail state is broken, and that rule has caught real bugs.

Then I set out to build a cozy game — no score, no timer, nothing to lose — and the rule became exactly backwards. Taking the session away because someone put their phone down is not a defect being caught. In a game like that, it is the defect.

Three rules, all inverted

It was not one rule. The play gate's contract mode ended with three assertions, and a good cozy game fails all three:

And in the scoring rubric used to judge whether a game is good enough to publish, two of the five axes were measuring the opposite of the target as well. One awarded twenty points for fitting a “one to three minute time-killer” shape; the new game wants to be stayed in, and being short is a fault. Another graded the difficulty curve, and this game does not have a difficulty curve at all — there is no difficulty.

This is a specific and common trap. Quality gates encode a definition of good, and that definition is invisible until you try to build something the authors did not have in mind. The checks did not fail loudly or say “this looks like a different genre”. They simply reported the new game as broken, in the confident tone of a passing test suite.

Widening the rule versus declaring the variant

The tempting fix is to relax the assertion — only require termination if the game has a score, say, or skip the restart check when there is no restart. That is the wrong shape, because every future game then falls into the exemption without anybody deciding that it should. A year later an arcade game with a genuinely broken fail state passes silently and nobody can reconstruct why the gate stopped catching it.

The gate already had the right pattern in it for a different reason. Games that end after a single action — one shot, one throw — cannot be measured by survival time, because the automated tester fires immediately and always scores badly. Rather than weakening the survival rule, those games declare themselves: the game exposes session: 'single-shot' and the gate takes a different branch for it. The rule stays strict; the exception is explicit, visible in the game's own source, and impossible to fall into by accident.

So the cozy game declares session: 'cozy', and the gate inverts precisely the rule that needs inverting:

if (r.session === 'cozy') {
  if (r.idle?.ended) {
    errors.push(`${at}: ended while idle after ${r.idle?.afterMs}ms — a cozy
      game must not take the session away when the player stops`);
  }
  skipped.push(`${at}: restart not checked — a cozy game has no run to restart`);
  return { errors, skipped };
}

Note what did not change. The progress check stayed exactly where it was. A cozy game still has to demonstrate that something advances when you play it — it just reports threads given to the seal in the field where an arcade game reports points. “Nothing is happening” is a defect in any genre.

The rubric got the same treatment: the two arcade axes were replaced rather than removed, by hand feel (is the tapping worth doing when it yields nothing?) and dwell (is there a reason to remain?). Five axes, a hundred points, same pass mark. The scoring did not get easier; it started measuring the right thing.

The part that is actually worth reading

All of that shipped on 1 September, with four new unit tests, and the suite went from 215 passing to 220. It was, on the evidence available, done.

Two weeks later I pointed the real gate at the real game for the first time, and it failed — with never ends when idle and score did not reset on restart. The two errors the entire change existed to prevent.

The branch was there. The tests covering it passed. They passed because each test constructs a report object by hand, sets session: 'cozy' on it, and hands it to the judging function. What no test covered was how that field gets into the report in the first place. The collector reads the value out of the running game like this:

const session = await page.evaluate(() => {
  const s = window.__GAME__ && window.__GAME__.session;
  return s === 'single-shot' ? 'single-shot' : 'run';
});

Everything that is not single-shot is flattened to run. The game declared itself cozy, correctly, and one line in the collector threw the declaration away before the judge ever saw it. The branch was unreachable from the real world, and the unit tests could not tell, because they never went through that line — they started downstream of it.

The fix is four lines and a whitelist, so that an unrecognised value still falls back to the strict path:

const DECLARED_SESSIONS = ['single-shot', 'cozy'];
const session = await page.evaluate((allowed) => {
  const s = window.__GAME__ && window.__GAME__.session;
  return allowed.includes(s) ? s : 'run';
}, DECLARED_SESSIONS);

The lesson, stated precisely

It would be easy to draw the lazy conclusion here — unit tests are not enough, write integration tests — and that is true but too general to act on. The sharper version is this: a test that builds its own input cannot verify that anything ever produces that input. Every hand-built fixture is a claim about the shape of real data, and a claim is not a check.

The signal was available and I ignored it. The change had a stated purpose — “a game that declares itself cozy is judged by cozy rules” — and the only way to see that purpose achieved is to have a game that declares itself cozy and watch it get judged. That took one command. It was not run for two weeks, because five extra green tests felt like proof.

There is now a real cozy game on the site, it declares itself, and the gate agrees: restart not checked — a cozy game has no run to restart. You can play the game that caused all of this at Feltling.