
ID	0021
Class	TEST
Severity	2
Status	ready
Models	—
Found	2026-09-25, filed from fhsim_fishery FISH-0027 (its remedy 3) on the owner's decision
Decision needed	— (Karl-Johan Reite decided 2026-09-25 to file this as `ready`)

Evidence

The regression harness decides pass or fail per output column from one number, the RMS of the difference between the run and the reference (src/testtools/GeneralMethods.cpp:66-86):

double squareSum = 0.0;
for (size_t i = 0; i < v1.size(); ++i) {
    const double diff = v1[i] - v2[i];
    squareSum += diff * diff;
}
 
return std::sqrt(squareSum / static_cast<double>(v1.size()));

Both comparison loops in TestResult use only that number: against a TestSpec reference (src/testtools/TestResult.cpp:220-238, RMS at :229) and against another TestResult (:268-285, RMS at :277). Neither the loops nor GetRootMeanSquareError look at the samples themselves; in src/testtools/ only JacobianChecker.cpp uses std::isfinite.

So:

a subnormal sample reproduced in both files contributes a difference of zero and passes;
a subnormal sample in one file and a zero in the other contributes a squared difference that underflows to zero, and passes;
a NaN in either file makes the RMS NaN, and both rms < 0.0 and rms > tolerance are false for NaN (TestResult.cpp:230, :234, :278, :282), so the column passes.

Effect

A port that publishes uninitialised memory is invisible to every regression case. The downstream library fhsim_fishery hit this twice (its FISH-0027 is the architecture item; the instances are its FISH-0021 and FISH-0024): Trawl/PIDController's Out was recorded in tests/in/PIDController/PIDController_ref.csv as the denormal 5.928788e-323 at t=0, and Seine/Operation's BuoyPos read 9.148902e-95, 3.479239e-278 when its case ran after other cases in the same process. Both passed. Re-recording a baseline can freeze such a value, and nothing flags it.

Possible fix

In the column comparison, report an issue naming the object and port when any sample in the run or in the reference is not finite, or is subnormal and non-zero (std::fpclassify(x) == FP_SUBNORMAL), independently of the RMS: a helper next to GetRootMeanSquareError in src/testtools/GeneralMethods.cpp, called from both loops in src/testtools/TestResult.cpp (:229, :277). The message should say whether the run or the reference holds the sample, and at which row, so a bad baseline is told apart from a bad run. No runtime cost outside the harness.

Test that would prove it

A TestResult unit test comparing two equal outputs whose column holds 5.928788e-323 (the historical fishery value), and another whose column holds a NaN, must report an issue for each; the same columns holding 0.0 must not. Today all three pass.

Risk

Downstream suites that pass today may fail: any recorded baseline holding a denormal or NaN becomes a failure the moment the check exists. That is the point, but the change should be announced to downstream libraries so they can fix the instance and re-record. It does not catch garbage that happens to be a plausible normal number, so it detects the denormal and non-finite cases only; FHSIM-0022 is the general enforcement.