FhSim  3.1.0
Marine systems simulation
Loading...
Searching...
No Matches
Testing report

State of the checked-out commit d29101c, dated 2026-09-24.

1. How the library is tested

Tests are GTest, run through fhsim::test::TestRunner (tests/RunTests.cpp), built as the CMake test target fhsim_fishery / binary test_fhsim_fishery (tests/CMakeLists.txt:1-53). ctest -R fhsim_fishery runs it from a build tree; the binary itself can also be run directly from ${PLAYPEN_DIR}/bin with "${TARGET}" "<repo>" 0 4 n . --gtest_filter=-*_NEEDS_GUI (tests/CMakeLists.txt:33-38). No test in this library is DISABLED_ or _NEEDS_GUI (grep -rn "DISABLED_\|_NEEDS_GUI" tests/*.cpp is empty), so every test that exists runs in CI.

Inputs live under tests/in/<ID>/ as XML scenarios; most single-object tests call the shared RunAndCompareRegressionXmlTiming helper (declared in tests/TestUtils.h), which runs the scenario and compares its logged outputs against a pinned *_ref.csv/.ref baseline recorded under tests/out/. A baseline proves "unchanged", not "correct" — several test files carry a doc comment linking specific pinned values to a tracked issue (e.g. TrawlDoor_Test.cpp:3-14 ties AngleOfAttack to the fitted working point behind FISH-0022; SeineOperation_Test.cpp:3-29 documents the FISH-0011/0012/0024 edges its baseline straddles).

Auto-mode Jacobian checking

TestSpec.jacobianMode defaults to Auto (fhsim/include/fhsim/testtools/TestSpec.h). When any SimObject in a scenario reports HasJacobians()==true, RunTest runs CheckJacobians after the simulation and fails the test on a mismatch (fhsim/src/testtools/TestRunner.cpp: 328-332). Every regression scenario in this library is therefore also an implicit Jacobian test unless it opts out with jacobianMode=No — that is a real, free check, but it has two structural blind spots:

  • It only probes near the initial state. The check compares the analytic Jacobian against a central finite difference at the initial condition plus 3 states perturbed by 1% (fhsim/src/testtools/JacobianChecker.cpp, relTol 1e-4, absTol 1e-10). JacobianCheckConfig:: timePoints changes only T, not the state, so "later time points" does not mean "away from the initial condition" (JacobianChecker.cpp:200-222). A term that only switches on away from the IC (bottom contact, a saturation branch, a slack-to-taut transition) is invisible to it.
  • A partial Jacobian in one object disables the check for the whole scenario. jacobianMode =No switches Auto mode off for every object in that scenario, not just the partial one. This library uses that switch three times: TrawlDoorBase_Test.cpp:10 (own partial Jacobian never verified, see FISH-0047 (resolved)), and AuvUndulateDiveControl_Test.cpp:269, AuvUndulateMissionControl_Test.cpp:504, AuvCtd_Test.cpp:103, AuvSilCam_Test.cpp:126, AuvEchosounder_Test.cpp:62 (all because they load Auv, whose attitude-dependent gravity/buoyancy terms are documented omissions, not because those five classes have Jacobians of their own — none of them declares HasJacobians()).

A registered SimObject's own analytic Jacobian can instead be verified by a dedicated test that sets jacobianMode=Yes and asserts run.jacobian.ok directly — CenterWeight_Test.cpp:14-33 is this library's example, with relTol relaxed to 1e-2 to accommodate the terms its header documents as omitted. Auv_Test.cpp goes further and scopes the assertion to specific Jacobian blocks (Auv_AnalyticJacobianBlocksMatchFiniteDifferences, :772) and to a non-zero base state offset (Auv_JacobianBlocksAreCorrectAtANonZeroStateOffset, :924, the regression test for the fix half of FISH-0032, since resolved). No model library, including this one, uses PortJacobianBlockChecker.

2. Coverage table

One row per SimObject registered in src/fhsim_fishery.cpp (ADD_SIMOBJECT_REF, 19 objects, none commented out). Rating: A behaviour asserted and the Jacobian verified where one exists; B partly; C smoke/regression only or indirect; D untested.

SimObject Input name Tests What is asserted Jacobian implementation Jacobian verified Rating Issues
TrawlDoorBase Trawl/TrawlDoorBase TrawlDoorBase_Test.cpp:3 (1 scenario) Regression only Dense, deliberately partial: pos/localVel coupling + 2 damping diagonals; hydro, gravity, quaternion kinematics, bottom contact omitted (TrawlDoorBase.h:127-136) No — jacobianMode=No (TrawlDoorBase_Test.cpp:10) B FISH-0032 (resolved), FISH-0047 (resolved)
ToTrawlDoor (TrawlDoor) Trawl/TrawlDoor TrawlDoor_Test.cpp:15 (1 scenario) Regression; doc comment ties pinned AngleOfAttack to a fitted working point (FISH-0022, resolved) None — HasJacobians() explicitly returns false (ToTrawlDoor.h:146) N/A (FD fallback) B FISH-0018, FISH-0048 and FISH-0051 (resolved)
TrawlDoorAddedFoil Trawl/TrawlDoorAddedFoil TrawlDoorAddedFoil_Test.cpp (8 fixtures under tests/in/TrawlDoorAddedFoil/) + FoilPolar_Test.cpp (the polar interpolation) Regression plus direct assertions: the foil lift changes the trajectory, a horizontal foil and an unsorted polar are rejected, an angle beyond the table holds the last row, and FoilEnergyConsumed grows monotonically and matches the hinge moment at a constant rate None (inherits ToTrawlDoor::HasJacobians()==false) N/A B FISH-0004, FISH-0005, FISH-0006 and FISH-0046 (resolved)
ToVessel Trawl/Vessel None under tests/ (only examples/input/SimpleTrawlVessel.xml and data/trawl/** demos) — Inherited TrawlVessel::OdeJacobian (dense, complete, TrawlVessel.cpp:126-152) No — never run by any test D FISH-0032 (resolved), FISH-0045 (resolved)
CenterWeight Trawl/CenterWeight CenterWeight_Test.cpp:9 (regression) + :14 (dedicated Jacobian test), 2 XML fixtures (CenterWeight, CenterWeight_Loaded) Regression + ASSERT_TRUE(run.jacobian.ok) Dense OdeJacobian + InputPortJacobian; deliberately excludes seafloor forces, input-port forces, water-velocity gradient (CenterWeight.h:110-121) Yes — dedicated test, TestJacobianMode::Yes, relTol relaxed to 1e-2 (CenterWeight_Test.cpp:14-33) A FISH-0023 (resolved), FISH-0030 (resolved), FISH-0032 (resolved)
PidController Trawl/PIDController PIDController_Test.cpp:3,17 (1 scenario, 2 cases) + heavy indirect use as the loop controller in Auv_ClosedLoop*/Auv_Undulate*/Auv_Mission* scenarios Regression plus the closed-form PI law Out(t) = 1 + 0.1 t (PIDController_Test.cpp:17, pins the gain sign); indirect settling-time/sign/step-response checks under load in AuvClosedLoop_Test.cpp None declared N/A (FD fallback) B FISH-0048 (resolved)
Actuator Trawl/Actuator Actuator_Test.cpp:3 (1 scenario) + used to drive Crowfoot_Test.cpp Regression; Actuator_LengthMovesAtTheCommandedSpeed asserts Length ramps at the commanded 0.1 m/s to 1 % Dense OdeJacobian, trivial: both state derivatives are input-driven, so identically zero (Actuator.h:88-89) Implicit — Auto mode on the default regression scenario (Actuator_Test.cpp:3-6) A FISH-0048 (resolved)
Crowfoot Trawl/Crowfoot Crowfoot_Test.cpp:11 (1 scenario, driven by an Actuator paying out a line) Regression; doc comment pins ExternalVel against the backward difference of ExternalPos (FISH-0003, resolved), asserted against the central difference by Crowfoot_ExternalVelIsTheRateOfExternalPos; Crowfoot_FourLinesIsFatal checks that NumLines = 4 is a load error (FISH-0014, resolved) None declared N/A (FD fallback) B FISH-0048 (resolved)
ValuesDisplay Trawl/ValuesDisplay ExamplesLoad_Test.cpp loads examples/input/ValuesDisplay.xml and Example.xml; ValuesDisplay_Test.cpp has a _NEEDS_GUI case, left out of ctest Setup and the first step only; in a visualisation build, setup succeeds with In1/In2 connected None declared N/A C FISH-0046 (resolved), FISH-0062 (resolved)
VesselDeploy Seine/Vessel Only indirectly, as the vessel in SeineOperation_Test.cpp:30 Regression (whole-scenario baseline); doc comment documents the phase-schedule edges and open/resolved defects it straddles Inherited TrawlVessel::OdeJacobian (dense, complete, TrawlVessel.cpp:126-152) Yes — implicit, Auto mode, no jacobianMode override in SeineOperation_Test.cpp B FISH-0002 and FISH-0012 (resolved), FISH-0017 (resolved)
SeineOperation Seine/Operation SeineOperation_Test.cpp (1 scenario, 3 tests) Regression baseline plus closed-form assertions: buoy Pos/Vel from the same source across every Connected edge, and the haul ramp at -WinchHaulSpeed down to MinWarpLength None declared (phase-schedule logic) N/A B FISH-0002 and FISH-0012 (resolved), FISH-0017 (resolved)
SeineConnector (Buoy) Seine/Buoy Inside SeineOperation_Test.cpp:56,62 Indirect regression, plus SeineOperation_BuoyPosVelSameSource asserting that Pos and Vel come from the same source on both sides of every Connected edge (FISH-0002, resolved) None declared N/A B FISH-0017 (resolved)
CableBranchedAttractor Seine/Cable SeineCable_Test.cpp (1 scenario, 60 s, placeholder geometry examples/input/SeineCableGeometry.xml) Regression only None — HasJacobians() explicitly returns false, with a documented rationale: needs fhsim_marine_elements internals and sfh::math::Bound breaks the analytic form No — exempted by instance name (scopeOverrides["C"]) C FISH-0017 (resolved), 0058 — InternalCableWithAttractor::AddElementForce is non-const and never overrides the const base
Auv Auv/Vehicle 39 dedicated tests (Auv_Test.cpp 29, AuvClosedLoop_Test.cpp 6, AuvFoil_Test.cpp 4), plus 6 integration tests in AuvSurvey_Test.cpp Closed-form physics (buoyancy, terminal speed, added mass, strip-theory wrench, turn radius, battery discharge, saturation), not just smoke Dense OdeJacobian (kinematic blocks + full 6x6 velocity block) + input/output port Jacobians for ThrustPower/ExternalForce/RudderYaw/RudderDive inputs and Pos/Battery outputs only (Auv.h:496-553) Yes, scoped: Auv_AnalyticJacobianBlocksMatchFiniteDifferences (Auv_Test.cpp:772), Auv_JacobianBlocksAreCorrectAtANonZeroStateOffset (:924), Auv_RudderWrenchJacobianMatchesFiniteDifferences (:1150). Most behaviour scenarios set jacobianMode=No because of Auv's own documented gravity/buoyancy omissions A FISH-0041 (resolved), FISH-0033, FISH-0034 and FISH-0037 (resolved)
AuvUndulateDiveControl Auv/UndulateDiveControl AuvUndulateDiveControl_Test.cpp (11 tests: 7 unit + 4 Auv_Undulate* integration) Behaviour incl. edge cases: hysteresis/no chatter at turning points, depth/seafloor clamping, thin-water fallback with a single warning None declared N/A (FD fallback); scenario jacobianMode=No (:269) because it loads Auv A FISH-0037 (resolved)
AuvUndulateMissionControl Auv/UndulateMissionControl AuvUndulateMissionControl_Test.cpp (17 tests) + Auv_Mission* integration Behaviour incl. edge cases: waypoint looping, coincident waypoints, heading-seam continuity, battery-gate latching, empty/mismatched waypoint list is an error None declared N/A; scenario jacobianMode=No (:504) because it loads Auv A FISH-0037 (resolved)
SensorCtd Auv/CTD AuvCtd_Test.cpp (12 tests) + Auv_Ctd*/AuvSurvey integration Behaviour incl. analytic first-order lag response, quantisation, bias+noise+seed, sample-and-hold, mount-offset rotation None declared N/A; scenario jacobianMode=No (:103) because it loads Auv A —
SensorSilCam Auv/SilCam AuvSilCam_Test.cpp (15 tests) + Auv_SilCam*/AuvSurvey integration Behaviour incl. Poisson vs Gaussian counting statistics, false-positive bias, empty-window NaN avoidance, missing-field fatal error None declared N/A; scenario jacobianMode=No (:126) because it loads Auv A —
SensorEchosounder Auv/Echosounder AuvEchosounder_Test.cpp (10 tests) + Auv_Echo*/AuvSurvey integration Behaviour incl. slant-range projection, bin indexing, TVG noise floor, target-strength slope None declared N/A; scenario jacobianMode=No (:62) because it loads Auv A FISH-0040 and FISH-0041 (resolved)

Summary: 19 SimObjects — 8 A, 6 B, 1 C, 4 D. 6 report HasJacobians()==true (TrawlDoorBase, ToVessel, CenterWeight, Actuator, VesselDeploy, Auv); 4 of those 6 are verified against finite differences somewhere in CI (CenterWeight, Actuator, VesselDeploy, Auv), 2 are not (TrawlDoorBase, ToVessel — see FISH-0047 (resolved) and FISH-0045 (resolved)).

3. Jacobian implementation notes

  • TrawlDoorBase (src/trawl/TrawlDoorBase.h:127-136, .cpp implementation): hand-derived, deliberately partial — only the rotation-matrix pos/localVel coupling and the local-velocity and angular-rate linear-damping diagonals. Hydrodynamic forces, gravity orientation, quaternion kinematics and bottom contact are omitted by design; subclasses that add contact (ToTrawlDoor) must and do override HasJacobians() to false. Never checked against finite differences in any test (FISH-0047 (resolved)).
  • TrawlVessel (src/trawl/TrawlVessel.cpp:126-152, shared by ToVessel and VesselDeploy): hand-derived, complete for the 3-state kinematic model (pos, pos, heading); the only nonzero partials are the heading-column entries. Verified for VesselDeploy via implicit Auto-mode checking in SeineOperation_Test.cpp; never checked for ToVessel because no test loads it (FISH-0045 (resolved)).
  • CenterWeight (src/trawl/CenterWeight.h:110-121): hand-derived OdeJacobian + InputPortJacobian, deliberately excludes seafloor contact forces, input-port forces and the spatial gradient of the water velocity, treating them as frozen inputs. The best-documented example in this library of testing a partial Jacobian honestly: CenterWeight_Test.cpp:14-33 runs a dedicated TestJacobianMode::Yes check with relTol relaxed to 1e-2 specifically because of those omissions, rather than disabling the check outright.
  • Actuator (src/trawl/Actuator.h:88-93): hand-derived, trivial — both states are input-driven, so every state partial is identically zero. The saturation clamp on length also contributes zero derivative by the same rule. Implicitly checked (Auto mode passes on the zero matrix in the one existing regression scenario).
  • Auv (src/auv/Auv.h:490-553): hand-derived, complete for the 6-state kinematic + velocity block, plus analytic input-port Jacobians for ThrustPower, ExternalForce, RudderYaw and RudderDive, and output-port Jacobians for Pos and Battery only (every other output port — Vel, Euler, Speed, Depth, Thrust, etc. — is left unwritten, which the assembler reads as a zero block; the header records this as the standing library convention shared with CenterWeight, not a defect of this class). The rudder-port columns are not reached by the assembled-system check because no fixture drives them from another SimObject's output port; Auv_RudderWrenchJacobianMatchesFiniteDifferences (Auv_Test.cpp:1150) checks the underlying wrench derivative directly instead. This class is also the fixed half of FISH-0032, since resolved (global-vs-local state-offset indexing): it caches its own base offset and every callback indexes through it, with a dedicated non-zero-offset regression test (Auv_JacobianBlocksAreCorrectAtANonZeroStateOffset, Auv_Test.cpp:924).
  • TrawlDoorAddedFoil, ToTrawlDoor, CableBranchedAttractor: all three explicitly return HasJacobians()==false (inherited or overridden) with a documented reason — bottom contact on top of a partial base Jacobian, or a limiter (sfh::math::Bound) that breaks the analytic form — so the engine falls back to finite differences for them. Not a gap, a documented choice.
  • PidController, Crowfoot, SeineOperation, SeineConnector, ValuesDisplay, and the five Auv-family helper/sensor classes (AuvUndulateDiveControl, AuvUndulateMissionControl, SensorCtd, SensorSilCam, SensorEchosounder): none declares an analytic Jacobian; the engine differentiates all of them by finite differences.

4. Prioritised improvement list

  1. Give ToVessel (Trawl/Vessel) a test. It is the only registered SimObject in this library with zero coverage of any kind, including an inherited analytic Jacobian that has never been finite-differenced. examples/input/SimpleTrawlVessel.xml is a ready-made scenario to adapt. See FISH-0045 (resolved).
  2. Add a fixture for TrawlDoorAddedFoil and ValuesDisplay. Both are registered, in-practice-used SimObjects (data/trawl/demos/trawldoorcontrol/FoilControl.xml, examples/input/ValuesDisplay.xml) with no test under tests/. See FISH-0046 (resolved).
  3. Give TrawlDoorBase's partial Jacobian a scoped test. Its one test disables the finite-difference check entirely (jacobianMode=No) rather than checking the three terms it claims, unlike CenterWeight's equivalent case. See FISH-0047 (resolved).
  4. **Seine/Cable (CableBranchedAttractor) has a regression test only**, on a placeholder geometry that is a format illustration, not a validated seine (FISH-0017, resolved).
  5. Port closed-form assertions into the single-scenario regression tests for PIDController, Actuator, Crowfoot and TrawlDoor — today they assert only "unchanged from the pinned baseline", not "correct", unlike the Auv family's physics-literate tests. See FISH-0048 (resolved).
  6. **Seine/Cable has a shipped input**, examples/input/SeineCable.xml (FISH-0017, resolved). The SeineConnector source mismatch (FISH-0002) is resolved and now asserted in SeineOperation_Test.cpp.
  7. Adopt PortJacobianBlockChecker for CenterWeight's InputPortJacobian and for TrawlVessel's Jacobian at a non-zero state offset, mirroring what Auv_JacobianBlocksAreCorrectAtANonZeroStateOffset already does for Auv — the local/global index-offset bug class (FISH-0032, since resolved) is currently regression-tested only for Auv.