Add test sharding, proactive clean, and retry logic for self-hosted CI

[sbryngelson]

Summary

Hardens self-hosted CI with test sharding, retry logic, and script deduplication.

Test sharding & retry

• Add --shard i/n flag to ./mfc.sh test — splits tests via modular arithmetic for even distribution
• Frontier GPU matrix now runs 2 shards per interface (acc/omp), halving wall-clock time
• Zero-test guard on both --only and --shard — empty results raise an error instead of silent green CI
• GitHub runner tests retry up to 5 sporadic failures using tests/failed_uuids.txt
• Abort path cleans failed_uuids.txt to prevent stale retries

--only filter improvements

• UUIDs use OR logic (match any), labels use AND logic (match all)
• --only matching zero tests now raises an error instead of silently passing

CI script consolidation

• Merge submit-bench.sh into submit.sh for all 3 clusters (frontier, frontier_amd, phoenix) — submit.sh auto-detects bench vs test mode from the submitted script's basename
• Unify frontier/ and frontier_amd/ scripts via directory-name detection — build.sh, bench.sh, submit.sh, and test.sh are now byte-identical across both directories
• Net deletion of 3 files and ~120 lines of duplicated shell code

Other

• Frontier test jobs use --qos=normal on batch partition (1h59m, CFD154 account)
• --requeue on Phoenix SLURM jobs for preemption recovery
• Build retry wrapper (3 attempts with clean between)
• Pin nick-fields/retry to commit SHA for security on self-hosted runners
• Lint-gate must pass before self-hosted tests run
• Skip benchmark workflow for bot review events

Depends on: #1170

Test plan

• Frontier GPU tests run in 2 shards per interface and complete within 2h
• Phoenix tests pass with --requeue and preemption recovery
• Lint-gate blocks self-hosted tests on lint failure
• GitHub runner retry logic fires on ≤5 test failures
• Benchmark jobs submit correctly via merged submit.sh (bench mode auto-detected)
• frontier/ and frontier_amd/ scripts are identical and detect cluster correctly
• --shard with zero resulting tests raises an error (not silent pass)

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (4)

.github/workflows/phoenix/bench.sh (1)

37-37: Quote $(nproc) to prevent word splitting.

The static analysis tool correctly identifies that $(nproc) should be quoted to prevent potential word splitting issues.

Proposed fix

-    if ./mfc.sh build -j $(nproc) $build_opts; then
+    if ./mfc.sh build -j "$(nproc)" $build_opts; then

Also apply to line 53:

-./mfc.sh bench $bench_opts -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
+./mfc.sh bench $bench_opts -j "$(nproc)" -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks

.github/workflows/frontier_amd/build.sh (1)

46-46: Quote the command substitution to prevent word splitting.

Per shellcheck SC2046, the unquoted $(...) can cause unexpected word splitting. While the current output is a single flag, quoting improves robustness.

Suggested fix

-        if ./mfc.sh test -v -a --dry-run $([ "$cluster_name" = "frontier" ] && echo "--rdma-mpi") -j 8 $build_opts; then
+        if ./mfc.sh test -v -a --dry-run "$([ "$cluster_name" = "frontier" ] && echo "--rdma-mpi")" -j 8 $build_opts; then

Note: Quoting the empty string when the condition is false will pass an empty argument. If mfc.sh cannot handle empty arguments gracefully, use a conditional approach instead:

rdma_opt=""
[ "$cluster_name" = "frontier" ] && rdma_opt="--rdma-mpi"
if ./mfc.sh test -v -a --dry-run $rdma_opt -j 8 $build_opts; then

.github/workflows/frontier_amd/submit.sh (1)

21-22: Quote $1 to handle filenames with spaces or special characters.

While unlikely in this CI context, quoting the variable is a shell best practice.

Suggested fix

 if [ ! -z "$1" ]; then
-    sbatch_script_contents=`cat $1`
+    sbatch_script_contents="$(cat "$1")"

.github/workflows/phoenix/submit.sh (1)

12-13: Quote $1 for robustness.

Same recommendation as the frontier submit scripts.

Suggested fix

 if [ ! -z "$1" ]; then
-    sbatch_script_contents=`cat $1`
+    sbatch_script_contents="$(cat "$1")"

---

ℹ️ Review info

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1412eb2 and 73fd804.

📒 Files selected for processing (24)

• .github/scripts/submit_and_monitor_bench.sh
• .github/workflows/bench.yml
• .github/workflows/frontier/bench.sh
• .github/workflows/frontier/build.sh
• .github/workflows/frontier/submit-bench.sh
• .github/workflows/frontier/submit.sh
• .github/workflows/frontier/test.sh
• .github/workflows/frontier_amd/bench.sh
• .github/workflows/frontier_amd/build.sh
• .github/workflows/frontier_amd/submit-bench.sh
• .github/workflows/frontier_amd/submit.sh
• .github/workflows/frontier_amd/test.sh
• .github/workflows/phoenix/bench.sh
• .github/workflows/phoenix/submit-bench.sh
• .github/workflows/phoenix/submit.sh
• .github/workflows/test.yml
• benchmarks/5eq_rk3_weno3_hllc/case.py
• benchmarks/hypo_hll/case.py
• benchmarks/ibm/case.py
• benchmarks/igr/case.py
• benchmarks/viscous_weno5_sgb_acoustic/case.py
• toolchain/mfc/bench.py
• toolchain/mfc/cli/commands.py
• toolchain/mfc/test/test.py

💤 Files with no reviewable changes (3)

• .github/workflows/frontier/submit-bench.sh
• .github/workflows/frontier_amd/submit-bench.sh
• .github/workflows/phoenix/submit-bench.sh

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Validate shard format before deriving job_slug.

An invalid shard value currently gets queued and only fails later in test argument parsing. Early validation here avoids wasting scheduler time.

🔧 Proposed fix

 shard_suffix=""
 if [ -n "$4" ]; then
-    shard_suffix="-$(echo "$4" | sed 's|/|-of-|')"
+    if [[ ! "$4" =~ ^[1-9][0-9]*/[1-9][0-9]*$ ]]; then
+        echo "ERROR: Invalid shard '$4'. Expected i/n (e.g., 1/2)."
+        exit 1
+    fi
+    shard_i="${4%/*}"
+    shard_n="${4#*/}"
+    if [ "$shard_i" -gt "$shard_n" ]; then
+        echo "ERROR: Invalid shard '$4'. Expected i<=n."
+        exit 1
+    fi
+    shard_suffix="-${shard_i}-of-${shard_n}"
 fi

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	shard_suffix=""
	if [ -n "$4" ]; then
	shard_suffix="-$(echo "$4" | sed 's|/|-of-|')"
	fi
	job_slug="`basename "$1" | sed 's/\.sh$//' | sed 's/[^a-zA-Z0-9]/-/g'`-$2-$3${shard_suffix}"
	shard_suffix=""
	if [ -n "$4" ]; then
	if [[ ! "$4" =~ ^[1-9][0-9]*/[1-9][0-9]*$ ]]; then
	echo "ERROR: Invalid shard '$4'. Expected i/n (e.g., 1/2)."
	exit 1
	fi
	shard_i="${4%/*}"
	shard_n="${4#*/}"
	if [ "$shard_i" -gt "$shard_n" ]; then
	echo "ERROR: Invalid shard '$4'. Expected i<=n."
	exit 1
	fi
	shard_suffix="-${shard_i}-of-${shard_n}"
	fi
	job_slug="`basename "$1" | sed 's/\.sh$//' | sed 's/[^a-zA-Z0-9]/-/g'`-$2-$3${shard_suffix}"

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Normalize --only UUID terms before matching.

Line 53 accepts uppercase UUID input, but Lines 60 and 63 perform case-sensitive membership checks. That can miss valid UUID filters like AB12CD34.

🔧 Proposed fix

-    uuids  = [t for t in ARG("only") if is_uuid(t)]
+    uuids  = [t.lower() for t in ARG("only") if is_uuid(t)]
@@
-        check.add(case.get_uuid())
+        check.add(case.get_uuid().lower())

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.04%. Comparing base (1412eb2) to head (73fd804).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1171      +/-   ##
==========================================
- Coverage   44.05%   44.04%   -0.02%     
==========================================
  Files          70       70              
  Lines       20496    20499       +3     
  Branches     1991     1993       +2     
==========================================
- Hits         9029     9028       -1     
- Misses      10328    10330       +2     
- Partials     1139     1141       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.