How to check that a gate can fail

An automated gate that fires for no reason is annoying and cheap. Somebody looks at it, argues for five minutes, tunes it and moves on. The expensive failure mode is the other one, the one almost nobody audits: the gate that reports green over a defect that is alive, served, and in plain sight of anyone who opens the page.

And it is expensive because it is invisible by construction. A gate that shouts gets fixed; a gate that stays quiet gets cited. Every time somebody sees it green, they stop looking at the thing the gate claimed to cover, and the hole grows exactly where nobody will walk again.

There is a method for finding out whether a gate is worth anything, and it is the one almost nobody applies: run it against the defect while the defect is still alive, and count how many it finds. Green over already-clean code is evidence of nothing: there is nothing to find, so a broken detector and a perfect one return exactly the same result.

What follows are five cases where the method caught a gate measuring a proxy: something correlated with the answer, easier to compute, and that at the moment of truth was not the question. All five failed in the same direction, and out of the five came four rules that are no longer up for discussion.

The probe that asked whether the heading was in view, not whether it was placed correctly

A probe walked the 6 section links on a page and checked, for each one, that the target ended up inside the viewport after the jump. Green on all six. The boolean was true.

The day we measured the number instead of the boolean, this showed up: the target landed 272 pixels from the top edge of the window, with a fixed bar of 80 pixels above it. Subtract and you will see the problem. Between the bottom edge of the bar and the heading you had just jumped to there was, empty, more than twice the height of the whole bar. You clicked a link in the index, the browser jumped, and the heading appeared floating in the middle of the screen with a band of black above it that meant nothing.

"It is in the viewport" was true. And it was not the question.

The cause was three scroll offsets adding up without any of them knowing about the others: 96 pixels declared on the target element, 96 more on the document root, and 80 inside the smooth-scroll library. 96 plus 96 plus 80 is exactly 272. The arithmetic closed perfectly and nobody had ever done it, because the gate said green.

The hardest part to digest was not the number. It was that there was a comment in the CSS stating explicitly that there was no double offset there, "because the library ignores scroll-padding". The library did not ignore it: it subtracts it first and then adds its own offset. We do not know whether that comment was ever true. We know nobody checked it again, and that the comment ended up being the reason nobody checked.

A comment that states something verifiable and that nobody verifies is a switched-off gate shaped like documentation. It has all the authority of a check and none of its properties.

The fix has two halves. The first: a single source of offset, derived from the real height of the bar, instead of three independent values overwriting each other.

:root {
  --alto-barra: 80px;
  --compensacion-ancla: var(--alto-barra);
}

The second: the probe now measures the number of pixels, not a boolean, and it measures it both ways, with JavaScript and without it. If the two figures do not match, the page looks different depending on whether the JavaScript loaded, and that is a defect in its own right even if each figure on its own looks reasonable.

The comment stripper that swallowed half a line of markup

Several of our text gates need to strip comments from the code before searching. The reason is specific to how we work: here, comments document defects by name. When a claim is pulled from a page, a comment stays behind explaining what was pulled and why. So a gate that searches for that claim to make sure it has not come back finds, without fail, the comment saying it left.

Hence the stripper. And the stripper had a one-line bug.

It treated // as the start of a comment, always. In JSX the visible text is not inside quotes, it sits loose in the markup, so a paragraph containing a web address carries two bare slashes in the middle of the sentence.

// The naive stripper: everything after // is a comment.
function quitarComentarios(src: string): string {
  return src.replace(/\/\/.*$/gm, '')
}

// Input:  <p>Write to us at https://domain.example/contact before Thursday</p>
// Output: <p>Write to us at
// The gate looks for "before Thursday", does not find it, and reports PASS.

Everything after those two slashes, on that line, stopped existing for the gate. It did not flag it as suspicious: it deleted it. The defect stayed published and the report said it was clean.

What matters is not the bug, which is trivial. It is the direction of the bug. A transformation of the input can be wrong in two ways: deleting too much, and then the gate stops seeing things that are there, or deleting too little, and then the gate complains about things that really were comments. Both are errors. They do not cost the same.

When a transformation can be wrong in both directions, pick the direction that produces false positives.

A false positive comes out red, someone looks at it, grumbles and dismisses it in a minute. A false negative comes out green and nobody ever looks at it. That asymmetry has to be in the design of the tool explicitly, not left to how the regular expression happened to come out.

The fix was not a cleverer regular expression either. You can write a guard for the URL scheme case and you will cover that particular hole, not the class:

function quitarComentarios(src: string): string {
  // Two slashes preceded by ':' are a URL scheme, not a comment.
  return src.replace(/(^|[^:])\/\/.*$/gm, '$1')
}

What actually changed the situation was giving the stripper its own bank of cases: 13 inputs with their expected output, run before the gate looks at a single file in the project. If any of them fails, the gate refuses to analyse anything and comes out red. A gate that does not validate itself cannot validate anything.

The gate written against the specimen, not against the species

A piece of structured data that should not have been published was pulled from a page. So it would not come back by accident, a gate was written. The gate looked for the exact shape of what had just been killed: the same string, the same type, the same page.

Months later somebody looked at the page next door. 24 instances of the same data type, alive, served, present the whole time. The gate was still green. It never saw them because it never looked for them: it had been written against a specimen, not against the species.

// Against the specimen that motivated the gate:
const contraElEjemplar = /"@type"\s*:\s*"AggregateRating"/

// Against the class of the defect:
const contraLaClase = /"@type"\s*:\s*"(AggregateRating|Rating|Review|Offer)"/

The missing question fits on one line, and you have to write it down, not think it: what other shapes does this defect have? And then list them. If your list comes out with a single entry, you did not finish thinking; you finished remembering the case that brought you here.

The most treacherous variant of this error is the numeric one. When a gate chases a figure, it is very easy to write it as a digit search and walk away feeling the matter is covered. It is not. You also have to look for the form that says the same thing without a figure, which is exactly the one no digit search ever finds.

It happened to us with a page that claimed total sector coverage without writing a single figure. That sentence is stronger than any number we could have put there, and it survived every sweep for the sole reason of not carrying a digit inside.

And there is a detail that closes the case: when we wrote this article, the gate fired again. The original sentence, quoted here as an example, was caught exactly as it should have been caught back then. It was right: a search engine reading this page finds the claim, not the context that disproves it. So it is described, not reproduced.

The negative control that could only return zero

A negative control checked that a certain file contained no identifier with a given prefix. It ran, returned zero, and zero was the correct result. The gate was installed and happy.

The problem is that in that file there was no identifier written literally, of any kind. They were all built at runtime from a piece of data. The control would have returned zero just the same if the emitted identifier had been anything else, including exactly the one it claimed to forbid. It was not checking the absence of the defect: it was checking the absence of literals, which is a different thing and which was true by construction.

A gate that can only come out green is not a gate, it is one more line in the report.

That is where the rule came from that today is not up for discussion: a new gate is run against the defect it claims to detect, still alive, and you count how many it finds. Green over already-clean code is evidence of nothing, because there is no way to tell "there are no defects" from "I do not know how to look".

In practice it is cheap. You pull the previous version from version control, the one that still had the problem, point the gate at it and count. If you know how many there are and it finds them all, the gate works. If it finds fewer, the gate is wrong, and you also already know in which direction it is wrong.

The class that was called gradient and painted a flat colour

An accessibility rule flagged a CSS class named .text-gradient, and the reasoning behind the flag was impeccable: a gradient applied over letters does not have a contrast ratio, it has a range, and the dark end of that range can fall below the minimum. All correct.

The class was not a gradient. An earlier refactor had removed the gradient and left a flat colour. What it did not remove was the name. Measured against the real background it is painted on, that colour gives 7.04:1, which passes AA and AAA too.

The rule did not reason wrongly. It inherited the lie in the name: it read the selector, inferred the technique and applied a correct argument to the wrong object.

The name of a class is a claim. The body of the CSS rule is the evidence.

When the name no longer describes what it does, the cheap fix is to rename the class, not to reopen the debate about the effect. Renaming costs a minute. Reopening the debate costs an afternoon, and the next time somebody reads the selector it will cost another one.

The corollary of the corollary does not stop at CSS: measure the token, do not assume it. A value declared in a design system is checkable in seconds, and guessing its contrast from what it is called is the same mistake wearing different clothes.

The four rules that stayed written down

Out of the five cases come four rules, and today they are not negotiable.

Every gate is written against the class of the defect, not against the specimen that motivated it. The specimen is what made you open the editor. It is not what you are hunting.

Every gate arrives with two controls. A positive one, shaped like the real defect, to prove it can find it. A negative one, with a legitimate shape that resembles it, to prove it does not catch that. A gate without a positive control is not a gate: it is decoration with an exit code.

No gate may fail towards green. If it is missing a piece of data, if it could not load the page, if the previous transformation blew up, it comes out red. Absence of evidence is never reported as evidence of absence.

And if the gate transforms the input before searching, if it strips comments, normalises whitespace or removes markup, that transformation is tested separately, with its own bank of cases, before the gate has an opinion about anything.

None of the four follows from theory: all four come from asking what would have had to happen for a green to be red, and then checking it with the defect in front of you. If you want to see how this translates into day-to-day work, we describe it in how we work.

And that is the question that orders everything else. Faced with a green, the first question is not whether it is right: it is under what concrete condition this would have come out red. If there is no answer, there is no check: there is an exit code.