Skip to content

[uss_qualifier] Add global timeout and run-to-completion requirement#1399

Open
BenjaminPelletier wants to merge 1 commit intointeruss:mainfrom
BenjaminPelletier:global-timeout
Open

[uss_qualifier] Add global timeout and run-to-completion requirement#1399
BenjaminPelletier wants to merge 1 commit intointeruss:mainfrom
BenjaminPelletier:global-timeout

Conversation

@BenjaminPelletier
Copy link
Member

uss_qualifier provides a great deal of useful information in its test report for any given test run. But, that test report information isn't accessible until the test completes. If a test takes far longer than a reasonable amount of time, the information captured so far in the test report would be very useful even if the test never ran to completion. This PR addresses that problem by adding an stop_after flag to the execution configuration -- when set to a timedelta, all remaining test actions (scenarios, suites, action generators) will be skipped after elapsed time exceeds this timedelta. So, execution will not stop immediately at stop_after, but it should stop fairly soon after this duration (whenever the currently-running test scenario completes, plus a small amount of time to finalize). This allows a user, for instance, to set stop_after to 2 hours and then force-stop the process after 2.5 hours to almost always provide access to the information that caused the test run to take more than 2 hours.

A concern stop_after introduces is that test behavior becomes somewhat less predictable since process completion doesn't mean the test run actually ran to completion normally. The fact that this had happened could be less obvious than ideal in some cases, so this PR also adds a new special interuss.automated_testing.execution.RunToCompletion requirement. If a user includes this requirement in a tested requirements artifact's requirement set, it will be populated with success upon running to completion normally, or failures providing information about the premature stop if the test run stops early. RunToCompletion failures due to Critical-severity check failure fit nicely into tested requirements' existing layout. RunToCompletion success and RunToCompletion failures due to timeout-based action skips bend the scenario-based breakdown a bit with an N/A scenario for success and (test suite)/(action generator)-prefixed scenario names when those components are skipped. But, RunToCompletion failures due to a test run timeout should be rare, and the content of tested requirements should be of less interest in those cases, so I think this bending is acceptable.

The special requirement is added to f3548_self_contained, but would be difficult to continually trigger in our CI as we have view expected-failed test scenarios. I manually verified that the tested requirements artifact looks as expected when a test run ended early due to a Critical-severity check failure, and when a test run ended early due to stop_after.

@BenjaminPelletier BenjaminPelletier marked this pull request as ready for review March 21, 2026 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant