Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028 - FAQs

Please refer to the accompanying document for technical descriptions and explanations.

The following FAQs provide additional information.

1. Are software developers required to follow this? What is the relationship between this work under Section 4(r) and future work under Section 4(e)?

No. These are guidelines and remain voluntary. At a later date, under Section 4(e) NIST will be developing guidance for “employing automated tools, or comparable processes, that check for known and potential vulnerabilities and remediate them, which shall operate regularly, or at a minimum prior to product, version, or update release; [and] providing, when requested by a purchaser, artifacts of the execution of the tools and processes described in subsection (e)(iii) and (iv) of this section, and making publicly available summary information on completion of these actions, to include a summary description of the risks assessed and mitigated….” Under Section 4(j). OMB “shall take appropriate steps to require that agencies comply with such guidelines with respect to software procured after the date of this order.” NIST anticipates that the material developed for Section 4(e) will reference this set of voluntary guidelines.

2. The EO directs NIST to publish guidelines recommending minimum standards for vendors’ testing of their software source code. Why do the standards include testing of executables and other tasks beyond source code testing? 

The EO directs NIST to address software source code, including identifying recommended types of manual or automated testing (such as code review tools, static and dynamic analysis, software composition tools, and penetration testing). The inclusion of a broad range of testing types, specifically including dynamic testing, shows that the intent is to broadly address vendor or developer testing. NIST took a holistic approach and included precursors to effective testing, threat modeling, and outputs of testing, fixing critical bugs.  

3. Why do you say “vendor or developer testing” when the EO says vendor testing?   

The vendor and the developer are often the same organization, but they can be different.  The verification tasks described in the document can be used by both vendors and developers. Verification should be done as early in the software development life cycle (SDLC) as possible, which will be by the developer. A vendor who is not also the developer should also perform verification but will not have the opportunity to be involved early in the process. See the SSDF, especially PW Practice 3, Verify Third-Party Software Complies with Security Requirements (PW.3) and PW Practice 4, Reuse Existing, Well-Secured Software When Feasible Instead of Duplicating Functionality (PW.4).

4. Why do you say “verification” instead of “testing”?   

“Testing” is often used to refer to dynamic analysis only. The EO is explicit that the testing should include a broad range of activities. NIST uses “verification” to ensure that the goal of the EO is met. It also is the more technically accurate term for the myriad types of software testing referred to in the EO, as defined in ISO/IEC/IEEE 15288:2015 Systems and software engineering — System life cycle processes and ISO/IEC/IEEE 12207:2017 Systems and software engineering — Software life cycle processes. See FAQ #2. 

5. What about penetration testing?   

Information about penetration testing is provided in Section 3.5 of the accompanying document. It includes supplemental material for web application scanning.    

Previous Sections:

• Introduction
• Background & Approach
• Recommended Minimum Standard for Vendor or Developer Verification of Code