the test that proved the testbed

This is the note about the day we caught ourselves testing the wrong party. The claim under test was that two peers behind different kinds of address translation could find each other and hold a direct session. The test passed, repeatedly, on both address families, and we recorded it as verified.

It was not verified. The testbed's gateways had static port-forwarding rules configured, and those rules — not the pathfinding code — were what made the incoming packets arrive. The traversal mechanism was never exercised, because the path had already been opened by hand before it started. Our evidence said the system worked; the evidence was actually about the harness.

The tell was mundane: the forwarding rules had been added while debugging something else and never removed. Once we asked what would happen without them, the test was not merely weaker, it was meaningless. We invalidated the result rather than reinterpreting it, rewrote the change record to say what had and had not been shown, and adopted a standing rule that the translation layer must be configured by the platform itself, from a clean state, with the forwarding tables checked empty before every run.

The cost was a milestone already recorded as passed, withdrawn in public inside our own records. The value is the rule it produced: a test that can pass for the wrong reason is worse than no test, because it converts an unknown into a false certainty that nobody revisits.