HEC-RAS failure taxonomy

HEC-RAS 6.6 Linux — Failure Taxonomy & Runtime Behavior

Experimental campaign 2026-10-12 on the Linux compute lab (~/workspace/hec-ras-research/hecras-v66-linux/, binaries via git LFS from github.com/neeraip/hecras-v66-linux). Muncie 2D example, Workflow A staging. Each experiment ran in its own /tmp/ras-exp-<name>/ from pristine inputs.

The single most important finding: exit code 0 means nothing. A 426%-volume-error run, a truncated HDF, and a missing plan file ALL exited 0. The only non-zero exits observed were Fortran file-not-found (29) and SIGKILL (137).

1. Per-experiment results

BASE (pristine, harness validation)

• GeomPreprocess: exit 0, 0.40s wall, 55.6 MB peak RSS
• Unsteady: exit 0, 6.96s wall, 106 MB peak RSS
• Volume error 0.049%; Results/Unsteady written; Finished Unsteady Flow Simulation

P1 — .b01 text interval 1MIN→10MIN (HDF still 1MIN)

• The .b01 TEXT wins over the HDF. SIMTIME stepped 0.1667 hr (10 min); wall 5.27s; volume error rose 15× to 0.77%; exit 0, results written.
• The .b01 "Job Control Information" section is LIVE input, not echo/metadata.

A1 — HDF Computation Time Step Base → 12HOUR (.b01 still 1MIN)

• HDF attribute ignored. SIMTIME stepped 0.0167 hr; wall 5.23s; volume error 0.049%.
• The HDF plan attribute is dead metadata (or fallback). Text is the source of truth for the computation interval.

A1b — .b01 text interval → 12HOUR (TRUE Courant violation; 2 steps for 24 h)

• Exit 0. Wall 0.27s. Volume error 426.6%. The maximum cell wsel error was 6.08 (at Perimeter 1, cell 3142). Results WRITTEN (garbage).
• Log shows FONT= RED before the volume error, FONT= BLUE BOLD on the wsel warning — then FONT= GREEN BOLD / Finished Unsteady Flow Simulation anyway.
• The engine does not refuse, warn-fatally, or non-zero-exit on Courant violation. It completes and writes garbage with a smile.

A2a / A2b — Manning's n → 0.5 / 0.001 (plan HDF Cells Center Manning's n)

• Exit 0, normal wall time, volume error 0.049%. Water-surface results bit-identical to baseline (max abs diff 0.0 across 25×3403).
• Editing Cells Center Manning's n in EITHER the .g01.hdf or the plan HDF has zero effect — silently ignored, no warning. (The plan HDF is self-contained: RasUnsteady reads Geometry from the plan HDF, not from Muncie.g01.hdf — verified: plan-HDF copy had pristine n while edited g01.hdf had 0.5.)
• Where the 2D solver actually gets roughness on Linux is UNKNOWN (open question).

A3 — .b01 deleted

• RasGeomPreprocess: exit 0 (doesn't need it). RasUnsteady: exit 29, 0.03s.
• Signature: forrtl: No such file or directory forrtl: severe (29): file not found, unit 39, file /tmp/ras-exp-A3/Muncie.b01 Image PC Routine Line Source RasUnsteady 000000000146E1B8 Unknown Unknown Unknown RasUnsteady 00000000007CAD58 openfiles_ 261 Openfile.for
• Intel Fortran runtime traceback. Clean, parseable, fails fast. No results.

A4 — plan HDF truncated to half

• Both exits 0. 0.02s / 0.03s. No volume lines, no results.
• Signature: HDF5-DIAG: Error detected in HDF5 (1.10.8), H5F__super_read(): truncated file, HDF_ERROR trying to open HDF output file: Muncie.p01.tmp.hdf, *** The file must be closed if it is being used by another application ***

A5 — plan HDF deleted

• Both exits 0. Same HDF_ERROR trying to open HDF output file signature as A4.
• Silent failure. The ONLY signal is log text.

A6 — zero-flow event (US BC hydrograph ordinates → 0 in plan HDF)

• Exit 0, 6.52s wall, volume error 0.077% (slightly above baseline 0.049%, fine).
• Dry runs complete cleanly. No special-casing needed by the orchestrator.

A7 — SIGKILL at 2 s (mid-run, was at SIMTIME 9.2/24 h)

• Exit 137. No Results group (results are written only on clean completion); plan HDF remains readable.
• Stray files left behind: Muncie.bco01 (988 B computation log with project/plan names, sim window, timestamp), Muncie.ic.o01 (12.8 KB initial conditions), systemInfoEncoded.txt (0 B).
• .bco01 is ALSO written on normal runs — it's the human-readable run record.

B8a / B8b — baseline variance (pristine ×2 more)

• Unsteady wall: 5.41s / 5.73s (vs 6.96s BASE). ±15% run-to-run variance on a shared VM. Peak RSS rock-stable: 106–109 MB (unsteady), 55–56 MB (geompreprocess).

B9 — .b01 text interval → 10SEC (6× steps)

• Wall 30.63s vs ~6s baseline → ~5× wall for 6× steps (roughly linear after fixed init/I-O overhead). Volume error improved to 0.029% (smaller dt = tighter).
• Peak RSS unchanged (109 MB) — memory does NOT scale with step count.

2. Consolidated failure taxonomy

3. Log parser spec

What to key on (in priority order):

1. Pass/fail: Overall Volume Accounting Error as percentage: — the ONLY trustworthy quality signal. Suggested gates: warn > 1%, fail > 5% (baseline 0.05%; 10MIN dt gave 0.77%; 12HOUR dt gave 427%).
2. Completion: Finished Unsteady Flow Simulation / Finished Processing Geometry — necessary but NOT sufficient (A1b printed it with 427% error).
3. Color markers: FONT= RED = bad news nearby (volume error); FONT= BLUE = warning (maximum cell wsel error); FONT= GREEN BOLD = done. All other FONT= lines are noise — strip them.
4. Phase tracking: LABEL= <phase> — Reading Data, Initializing, Reading 2D Area(s), initializing 2D Area(s), Initial Backwater, Unsteady Flow Computations. Use for progress reporting.
5. Step tracking: SIMTIME= <hours> + ABSDATE=/ABSTIME= per output step; ITER2D= <n> (nonzero = solver iterating that step; long nonzero runs = struggling). PROGRESS= <0..1> resets per phase — not monotonic overall, don't use as a global bar.
6. Hard errors: forrtl: severe (NN): (Fortran runtime; 29 = file not found); HDF5-DIAG: + HDF_ERROR (HDF layer); a following Image | PC | Routine | Line | Source block = Fortran traceback.
7. Run identity: Performing Unsteady Flow Simulation HEC-RAS 6.6 September 2024 (version string); 2D Unsteady Diffusion Wave Equation Set (equation set); 2D number of Solver Cores: N (actual cores used).
8. Noise to ignore: sh: 1: wmic: not found (Linux core detection fails; falls back to 2 cores — benign but note the default), Writing Results to DSS, Please review "Computational Log File" output…, *** The file must be closed… (this one is benign boilerplate appended to HDF errors, not a real lock).

4. Middleware implications (concrete rules)

1. Never gate on exit code alone. Gate on: log contains Finished… AND volume error % below threshold AND Results/Unsteady group exists in the output HDF. All three. (A4/A5 exit 0 with no results; A1b exits 0 with garbage.)
2. Volume-error gates: warn > 1%, fail > 5%. Calibrate against CHAMP production models once available; these bounds come from one demo.
3. Pre-run Courant validation is mandatory. The engine will happily run a 12-hour dt on 50-ft cells, exit 0, and write garbage. The middleware must check dt against nominal cell size before launch — this is a deterministic QC gate, not a triage judgment.
4. Timeout heuristic: wall ≈ linear in step count (~6s per 1440 steps here → ~4 ms/step + ~1 s fixed). Estimate timeout = 10 × (sim_hours/dt_hours) × sec_per_step measured from a pilot run; the ±15% run variance means anything tighter than 5× risks false kills. Memory is flat (~110 MB here) — no memory-based timeout needed at this scale; re-measure per model size.
5. Stage validation before launch: assert every file the .prj/plan needs exists (.b01, .x01, HDFs). The forrtl (29) is catchable before spending a run.
6. Text-vs-HDF precedence: the .b01 text's Job Control (Computation Interval) is LIVE and beats the HDF's Computation Time Step Base (dead). If the middleware writes text inputs, it must set the interval in the text. If it writes HDFs, it must ALSO keep the text in sync — or stop shipping text the engine will read. (Untested: .p01 text vs .b01 text precedence — both carry the interval.)
7. The plan HDF is the artifact of record. RasUnsteady reads Geometry from the plan HDF, not from Muncie.g01.hdf. Fingerprint and archive the plan HDF for provenance; the .g##.hdf is only preprocessor input.
8. Retry hygiene after kill/timeout: delete .bco01, .ic.o01, systemInfoEncoded.txt and re-stage the plan HDF (strip Results) before retry. The HDF itself survives SIGKILL readable — no repair needed, just cleanup.
9. Set solver cores explicitly. Linux core autodetect fails (wmic: not found) → silent default of 2. If the worker has more, pass it; don't rely on detection.
10. Roughness changes need input-level validation. Editing Cells Center Manning's n is silently ignored (bit-identical outputs, exit 0) — the engine gives no signal. The middleware must diff roughness inputs pre-run and confirm effect post-run (e.g., compare WSE summaries), not trust the run to complain.
11. Dry runs are safe — no orchestrator special-casing for zero-flow events.
12. Keep the .bco01. It's a small human-readable run record (project, plan, sim window, timestamp) — cheap provenance, archive it per run.

5. Open questions (need 7.0.1 Windows engine or RAS Mapper)

1. Where does RasUnsteady actually read 2D Manning's n from? — investigated 2026-10-12, see §6. Short version: NONE of the candidate inputs moved the needle on this Linux build in the Muncie regime; a roughness liveness probe gate is now the recommended middleware rule instead of assuming a live knob.
2. .p01 text vs .b01 text precedence for Computation Interval (both carry it).
3. Does RasSteady share the exit-0-on-failure behavior? (Not tested.)
4. 7.0.1 engine differences in any of the above (only 6.6 tested).
5. Whether the Windows engine's wmic core detection changes default-core behavior (and whether explicit core setting is honored the same way).
6. What ITER2D sustained-nonzero patterns look like for a genuinely struggling (but converging) model — needed to set a "solver struggling" warning threshold.
7. Whether 2D roughness becomes a live knob in a friction-dominated regime (steeper/channelized 2D model) — the Muncie regime is level-pool-like (mean|v| ≈ 0.02), intrinsically n-insensitive. Needs a different test model.

6. Roughness-input investigation (follow-up campaign, 2026-10-12)

Motivation: open question §5.1 — the A2a/A2b finding that editing per-cell Manning's n is silently ignored. Goal: find the LIVE 2D roughness input for the middleware's calibration use case. Result: no live 2D roughness input found on this Linux build in the Muncie regime. Campaign scripts: /tmp/ras-campaign.py, /tmp/ras-campaign2.py, /tmp/ras-campaign3.py, /tmp/ras-campaign4.py (transient); experiment dirs /tmp/ras-exp-C*.

6.1 Input-path mapping (established first)

• The authoritative land-cover → n lookup table is the Variables dataset (Name, ManningsN, Percent Impervious) in the standalone Examples/Muncie/Land Classification/LandCover.hdf — used by RAS Mapper at model-build time. Its values (0.030–0.150) exactly match the 9 unique per-cell n values baked into the plan HDF.
• The plan HDF's own Geometry/Land Cover (Manning's n)/Calibration Table is all-NaN placeholders (both .g01.hdf and .p01.hdf).
• strace -f -e openat on RasUnsteady proves it opens ONLY Muncie.p01.tmp.hdf, Muncie.b01, Muncie.x01 (+ its own outputs). It never touches Muncie.g01.hdf or LandCover.hdf. The plan HDF is the sole HDF input.
• RasGeomPreprocess does NOT recompute Cells Center Manning's n — an edit to 0.5 survived the preprocessor unchanged. Per-cell n is baked at RAS Mapper time, not at preprocess time.
• The 2D area is driven by its OWN boundary conditions in the plan HDF's Event Conditions/Unsteady/Boundary Conditions: a US flow hydrograph (2D: Perimeter 1 BCLine: US, 28.3 → 0.5 cfs over 24 h) plus a precipitation trickle (≤ 0.07) and a DS normal-depth BC (friction slope 0.001). The 1D river (Fake River, XS n = 0.03 in .x01) is NOT simulated (2D Only = True in plan parameters) — changing its n 10× (C7) moved 2D results 0.0000 ft.
• Regime: 947/3403 cells wet, mean wet depth 0.39 ft, mean|face velocity| ≈ 0.02 — essentially a level pool draining through a normal-depth outlet. Intrinsically insensitive to friction (see §6.3).

6.2 Perturbation results (all vs unsteady-only baselines)

| ID | Perturbation | max|dWSE| vs baseline | Verdict | |---|---|---|---| | C2 | Perimeter 1 attr Manning's n 0.06 → 0.35 | 0.0000 ft | dead | | C3 | same, through full pipeline (geompre+unsteady) | 0.0000 ft | dead; attr survives geompre | | C4 | per-cell n → 0.5, unsteady only (clean A2a replication) | 0.0000 ft | dead | | C6 | Composite LC 0 → 1 alone | 0.0000 ft | dead | | C5/C10 | per-cell n → 0.5 AND Composite LC = 1 | 0.0070 / 0.0007 ft | not reproduced — noise (see §6.3) | | C11 | plan-HDF Calibration Table base n → 0.5 (all 16 classes) | 0.0000 ft | dead | | C7 | 1D XS n 0.03 → 0.30 in .x01 text | 0.0000 ft | dead (1D not simulated) | | C8/C9 | 50× US inflow (1415 cfs peak, 1287 wet cells, 14.65 ft max depth): per-cell n pristine vs 0.5 | 0.0183 ft | dead within noise (see §6.3) |

6.3 New finding: run-to-run result nondeterminism

The original campaign measured ±15% WALL-TIME variance only. Re-running bit-identical inputs and diffing WSE fields shows the RESULTS also vary: ~0.001–0.007 ft at baseline regime (C5 vs C10), ~0.004 ft at 50× flow (C8 vs C12 replicate). Likely source: threaded solver (PARDISO/MKL, 2 cores) reduction ordering. Middleware consequence: regression tests on engine outputs must use tolerances (≈ 0.01 ft for this model scale), never bit-equality.

6.4 Middleware rules (added)

1. Roughness liveness probe (deterministic gate). Before ANY calibration campaign, perturb the candidate roughness input ±, run, and verify the objective moves beyond the nondeterminism envelope (~0.01 ft here). If it doesn't move, the knob is dead — halt the campaign, do not "calibrate". Never assume a roughness input is live on the basis of where it is stored.
2. Regression-test with tolerances, not bit-equality. Engine WSE outputs vary run-to-run at the mm level (threaded solver). Gate on |Δ| < tolerance.
3. Know the model's regime before trusting sensitivity. A level-pool-like 2D regime (low velocity, normal-depth outlet) is intrinsically friction-insensitive — a dead liveness probe may indict the regime, not the input. Re-probe on a friction-dominated model before concluding the Linux build can't steer roughness at all (§5.7).

Appendix: experiment index

Harness: ~/workspace/hec-ras-research/run_exp.sh + sup.py (timing/peak-RSS) + per-experiment modifiers in ~/workspace/hec-ras-research/mods/. Reusable for the next campaign (RasSteady, 7.0.1, production-model calibration).

Liveness probe (rule 13, implemented 2026-10-12): liveness_probe.py --mod mods/<name>.py stages the model fresh, runs baseline ×2 + mod ×1, and compares the 2D Water Surface field. Verdict LIVE if the mod shift exceeds max(3×envelope, 0.01 ft), DEAD otherwise (exit 0/2/1 = live/dead/error), with max|Volume Error| per run. Verified: A2a (dead n knob) → DEAD; POS (US flow ×10) → LIVE (shift 8.09 ft vs 0.0038 ft envelope). Mods: POS.py is the positive control; any calibration campaign must pass this gate first. Lab Python: .venv (h5py/numpy); run_exp.sh prefers it on PATH.