This page discusses some of the design issues and decisions of the Software Assurance Reference Dataset (SARD). The SARD user interface and its manual are on-line. It evolves quickly. We appreciate and acknowledge those who contributed test cases to the SARD.
Test cases in the SARD have a different "status", as evidenced by the C (candidate), A (accepted) or D (deprecated) tag assigned to each test case when viewed searching the SARD using its web interface. The purpose of giving SARD test cases different a status type is to provide the test case user with an indicator of the case's quality, both in terms of test case documentation and test case construction. Each status tag is described in more detail below.
When a test case is first added to the SARD, it is assigned a status of "candidate". This means that the test case has not been reviewed by the SARD librarian to determine if it is complete in its documentation, correct in its construction and acceptable in quality. Until a test case is examined in this way, it will keep its status of "candidate".
An "accepted" test case is one that we believe meets the necessary documentation, correctness and quality requirements that permit an SARD user to test a tool against a particular source code weakness. In order to meet those requirements, the test must be thorough in its documentation, correct in representing a particular weakness in the source code, and of high enough quality that it is simple to understand and free of extraneous weaknesses that confuse a user of the intent of the test case. If you download a test case that has a status of "accepted", you can expect the following:
Who can change test cases or test suites? When? Why?
To have long term value, the content of a test case is "write once". That is, once source code is added to the SARD, it keeps the same test case ID and never changes. This permanence allows research work to refer to, say SARD test case 1552, knowing that that exact code can always be retrieved. Later work could reliably get exactly what was used before.
What if there is a mistake in the code, for instance, there is a second, unintended weakness? It would be marked with a status of "deprecated", and a new "correct" version submitted to the SARD. Deprecated test cases should not be used for new work. They remain in the SARD with their original identifiers as a reference to redo previous work.
Test suites are similarly "write once". Once they are designated, they should not change. A test suite might be superceded by an improved test suite, which refers to test cases conforming to the latest language standard or has better coverage.