Nightly tests don't run for release test jobs

[berquist]

5d44262 (#1290) introduced functionality for reducing the testing time in PRs with the Autotester by selectively setting tests to only run during nightly testing.

sst-core/src/sst/core/testingframework/sst_unittest_support.py

Lines 82 to 87 in 20f9319

	def testing_check_is_nightly() -> bool:
	"""If Nightly is in the BUILD_TAG it's very likely a Nightly build."""
	build_tag = os.getenv("BUILD_TAG")
	if build_tag is not None and "Nightly" in build_tag:
	return True
	return False

An example of its usage is https://github.com/sstsimulator/sst-elements/blob/3e4b9308866aeab46c3a79d0b3d48bc042fa3b44/src/sst/elements/prospero/tests/testsuite_default_prospero.py#L56:

    @unittest.skipIf(not testing_check_is_nightly(), "test_prospero_text_using_PIN_traces only runs on Nightly builds.")
    def test_prospero_text_using_PIN_traces(self):
        ...

The problem with this is that we want release tests (tests are scheduled to run weekly on the versions tagged for release) to also run all of these tests. BUILD_TAG is one of the environment variables Jenkins automatically creates for every job and looks like jenkins-SST__15.1.0_beta_NewFW_Ubuntu-22.04_OMPI-4.1.4_PY3.10_Mainline_MR-2-75 for a release job.

None of the core tests are tagged with this, only elements, but because this involves the test framework and will require changes in this repo, the issue is created here.

This functionality will be reworked from an environment variable to a command-line argument so that it is clearer on how to control which tests are chosen or excluded¹. Rather than the binary choice of "skip unless nightly", the logic should be changed to control which subset of tests are run. We think there are three current subsets which are not necessarily disjoint:

• tests that should run for PRs
• tests that should run nightly and for releases
• tests that should run weekly

We should investigate if scenarios can be used. It looks like specifying a scenario means that test is skipped, but we are looking for something closer to pytest markers.

In the meantime, I will see if it is possible to override BUILD_TAG in the Jenkins shell script section to trigger release jobs to run the nightly tests.

Footnotes

1. BUILD_TAG may still be used inside of bamboo in order to determine what the testing command-line is. ↩

[berquist]

Thoughts:

• Repurpose --scenarios and invert to work like pytest markers, labeling tests with their purpose
• AT PR ones are currently triggered by default, which might be equivalent to unmarked

[berquist]

• "Nightly" implies also wanting PR tests
• "Weekly" implies not wanting nightly

[berquist]

3 disjoint sets, be specific about what you want your test to run as; if you don't specify anything, then it runs in a PR.