SAMATE Publications

Papers and Reports | Workshops | Other Publications

We also have a bibliography of other relevant papers and publications.

Papers and Reports

• "Vulnerability Test Suite Generator (VTSG) Version 3", October 2023, NIST Interagency Report (IR) 8493. Paul E. Black, William Mentzer, Elizabeth Fong, and Bertrand Stivalet, DOI 10.6028/NIST.IR.8493

• "SATE VI Report: Bug Injection and Collection," June 2023, NIST Special Publication (SP) 500-341. Aurelien Delaitre, Paul E. Black, Damien Cupif, Guillaume Haben, Alex-Kevin Loembe, Vadim Okun, Yann Prono, DOI 10.6028/NIST.SP.500-341

• Static Analysis Tool Exposition (SATE) VI: Mobile Track Report, March 2023, NIST Internal Report (IR) 8462, Michael Ogata. DOI 10.6028/NIST.IR.8462.

• I. Bojanova, C. E. Galhardo and S. Moshtari, "Data Type Bugs Taxonomy: Integer Overflow, Juggling, and Pointer Arithmetics in Spotlight," 2022 IEEE 29th Annual Software Technology Conference (STC), 2022, pp. 192-205, doi: 10.1109/STC55697.2022.00035.

• I. Bojanova, C. E. Galhardo and S. Moshtari, "Input/Output Check Bugs Taxonomy: Injection Errors in Spotlight," 2021 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), 2021, pp. 111-120 , doi: 10.1109/ISSREW53611.2021.00052.

• A. Gueye, C. E. Galhardo, I. Bojanova and P. Mell, "A Decade of Reoccurring Software Weaknesses," in IEEE Security & Privacy, vol. 19, no. 6, pp. 74-82, Nov.-Dec. 2021, doi: 10.1109/MSEC.2021.3082757.

• "Guidelines on Minimum Standards for Developer Verification of Software," October 2021, NIST Internal Report (IR) 8397, Paul E. Black, Vadim Okun, and Barbara Guttman, DOI 10.6028/NIST.IR.8397

• I. Bojanova and C. Eduardo Galhardo, "Classifying Memory Bugs Using Bugs Framework Approach," 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC, 2021, pp. 1157-1164, doi: 10.1109/COMPSAC51774.2021.00159.

• Algorithms and Data Structures for New Models of Computation, Jan/Feb 2021, IT Professional, 23(1):9-15, Paul E. Black, David Flater, and Irena Bojanova. DOI 10.1109/MITP.2020.3042858.

• C. E. Galhardo, P. Mell, I. Bojanova and A. Gueye, “Measurements of the Most Significant Software Security Weaknesses,” Annual Computer Security Applications Conference (ACSAC), pp. 154–164, Dec. 2020, doi: 10.1145/3427228.3427257.

• SATE VI Ockham Sound Analysis Criteria, April 2020, NIST Internal Report (IR) 8304, Paul E. Black and Kanwardeep Singh Walia. DOI 10.6028/NIST.IR.8304. 
  The data and programs to reproduce these results are available at DOI 10.18434/M32187 or https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-VI-2020/ockhamCriteriaSATEVIdata2020.tar.xz (4.5 Megabytes download; 128 Megabytes uncompressed). A README file is available at https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-VI-2020/README
• I. Bojanova, Y. Yesha, P. E. Black and Y. Wu, "Information Exposure (IEX): A New Class in the Bugs Framework (BF)," 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), 2019, pp. 559-564, doi: 10.1109/COMPSAC.2019.00086.
• Opaque Wrappers and Patching: Negative Results, November 2019, Computer, 52(12):89-93, Paul E. Black and Monika Singh. DOI 10.1109/MC.2019.2936071. PMCID PMC7066996.
• Formal Methods for Statistical Software, October 2019, National Institute of Standards and Technology (NIST) Interagency Report (IR) 8274, Paul E. Black. DOI 10.6028/NIST.IR.8274.
• SATE V Report: Ten Years of Static Analysis Tool Expositions, October 2018, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-326, Aurelien Delaitre, Bertrand Stivalet,Paul E. Black, Vadim Okun, Athos Ribeiro, and Terry S. Cohen. DOI 10.6028/NIST.SP.500-326.
• Randomness Classes in Bugs Framework (BF): True-Random Number Bugs (TRN) and Pseudo-Random Number Bugs (PRN), July 2018, 2018 IEEE 42nd Annual Computers, Software & Applications Conference (COMPSAC), Tokyo, Japan, 2018, pp. 738-745, Irena Bojanova, Yaacov Yesha, and Paul E. Black. DOI 10.1109/COMPSAC.2018.00110.
• Juliet 1.3 Test Suite: Changes From 1.2, June 2018, National Institute of Standards and Technology (NIST) Technical Note (TN) 1995, Paul E. Black. DOI 10.6028/NIST.TN.1995.
• A Software Assurance Reference Dataset: Thousands of Programs With Known Bugs, April 2018, Journal of Research of NIST, Volume 123:123005, Paul E. Black. DOI 10.6028/jres.123.005.
• SARD: Thousands of Reference Programs for Software Assurance, October 2017, Journal of Cyber Security and Information Systems - Tools & Testing Techniques for Assured Software - DoD Software Assurance Community of Practice: Volume 2, 5(3):6-13, Paul E. Black.
• Improving Software Assurance through Static Analysis Tool Expositions, October 2017, Journal of Cyber Security and Information Systems - Tools & Testing Techniques for Assured Software - DoD Software Assurance Community of Practice: Volume 2, 5(3):14-22, Terry S. Cohen, Damien Cupif, Aurelien Delaitre, Charles D. De Oliveira, Elizabeth Fong, and Vadim Okun.
• Impact of Code Complexity on Software Analysis, February 2017, NIST Internal Report (IR) 8165 Update 1, Charles D. DeOliveira, Elizabeth Fong, and Paul E. Black. DOI 10.6028/NIST.IR.8165-upd1.
• Defeating Buffer Overflow: A Trivial but Dangerous Bug, November/December 2016, IT Professional, 18(6):58-61, Paul E. Black and Irena Bojanova. DOI 10.1109/MITP.2016.117.
• Report of the Workshop on Software Measures and Metrics to Reduce Security Vulnerabilities (SwMM-RSV), October 2016, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-320, Paul E. Black and Elizabeth Fong. DOI 10.6028/NIST.SP.500-320.
• The Bugs Framework (BF): A Structured Approach to Express Bugs, August 2016, 2016 IEEE International Conference on Software Quality, Reliability, and Security (QRS 2016), Vienna, Austria, Irena Bojanova, Paul E. Black, Yaacov Yesha, and Yan Wu. DOI 10.1109/QRS.2016.29.
• Large Scale Generation of Complex and Faulty PHP Test Cases, April 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), Chicago, IL, Bertrand Stivalet and Elizabeth Fong. DOI: 10.1109/ICST.2016.43. Best paper award for the tool demo track.
• SATE V Ockham Sound Analysis Criteria, March 2016, NIST Internal Report (IR) 8113, Paul E. Black and Athos Ribeiro. Noted 23 June 2017. DOI 10.6028/NIST.IR.8113. 
  The data and programs to reproduce these results are available at DOI 10.18434/T4WC7V or https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-V-2016/ockhamCriteriaSATEVdata2017.tar.xz (9.3 Megabytes download; 390 Megabytes uncompressed). A README file is available at https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-V-2016/README
• Evaluating Bug Finders - Test and Measurement of Static Code Analyzers, May 2015, 2015 IEEE/ACM 1st International Workshop on Complex Faults and Failures in Large Software Systems (COUFLESS), Florence, Italy, Aurelien Delaitre, Bertrand Stivalet, Elizabeth Fong, and Vadim Okun.
• Fuzz Testing for Software Assurance, 1 March 2015, Crosstalk, The Journal of Defense Software Engineering, 28:35–37, Vadim Okun and Elizabeth N. Fong.
• A Basic CWE-121 Buffer Overflow Effectiveness Test Suite, April 2013, Proc. Sixth Latin-America Symposium on Dependable Computing (LADC 2013), Paul E. Black, Hsiao-Ming (Michael) Koo, and Thomas Irish.
• Report on the Static Analysis Tool Exposition (SATE) IV, January 2013, NIST Special Publication (SP) 500-297, Vadim Okun, Aurelien Delaitre, and Paul E. Black. DOI 10.6028/NIST.SP.500-297.
• Report on the Metrics and Standards for Software Testing (MaSST) Workshop 2012, December 2012, NIST Internal Report (IR) 7920, Paul E. Black and Elizabeth Fong. DOI 10.6028/NIST.IR.7920.
• Juliet 1.1 C/C++ and Java Test Suite, October 2012, Computer, 45(10):88-90, Tim Boland and Paul E. Black. DOI 10.1109/MC.2012.345.
• Static Analyzers: Seat Belts for Your Code, May-June 2012, Security & Privacy, 10(3):48-52, Paul E. Black, DOI 10.1109/MSP.2012.2.
• Software Vulnerabilities Precluded by SPARK, November 2011, ACM Int'l Conf. on Ada and Related Technologies: Engineering Safe, Secure, and Reliable Software (SIGAda 2011), Paul E. Black (NIST), Chris E. Dupilka (U.S. DoD), F. David Jones, and Joyce Tokar (Pyrrhus Software).
• Report on the Third Static Analysis Tool Exposition (SATE 2010), October 2011, NIST Special Publication (SP) 500-283, Vadim Okun, Aurelien Delaitre, and Paul E. Black. DOI 10.6028/NIST.SP.500-283.
• Counting Bugs is Harder Than You Think, September 2011, 11th IEEE Int'l Working Conference on Source Code Analysis and Manipulation (SCAM 2011), Williamsburg, VA, Paul E. Black.
• Source Code Security Analysis Tool Test Plan Version 1.1, July 2011, NIST Special Publication 500-270 v1.1, Michael Koo, Romain Gaucher,  Charline Cleraux, and Jenise Reyes Rodriguez. DOI 10.6028/NIST.SP.500-270v1.1.
• Source Code Security Analysis Tool Functional Specification Version 1.1, NIST Special Publication 500-268 v1.1, February 2011, Paul E. Black, Michael Kass, Michael Koo, and Elizabeth Fong. DOI 10.6028/NIST.SP.500-268v1.1.
• Toward a Preliminary Framework for Assessing the Trustworthiness of Software, November 2010, National Institute of Standards and Technology (NIST) Internal Report (IR) 7755, Tim Boland, Charline Cleraux, and Elizabeth Fong. DOI 10.6028/NIST.IR.7755.
• The Second Static Analysis Tool Exposition (SATE) 2009, June 2010, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-287, Vadim Okun, Aurelien Delaitre, and Paul E. Black. DOI 10.6028/NIST.SP.500-287.
• Static Analysis Tool Exposition (SATE) 2008, June 2009, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-279, Vadim Okun, Romain Gaucher, and Paul E. Black, editors. DOI 10.6028/NIST.SP.500-279.
• Static Analyzers in Software Engineering, CrossTalk, The Journal of Defense Software Engineering, 22(3):16-17, March/April 2009, Paul E. Black.
• Software Assurance Tools: Web Application Security Scanner Functional Specification Version 1.0, January 2008, NIST Special Publication (SP) 500-269, Paul E. Black, Elizabeth Fong, Vadim Okun, and Romain Gaucher. DOI 10.6028/NIST.SP.500-269.
• Building a Test Suite for Web Application Scanners, January 2008, 41st Hawaii Int'l Conf. on System Sciences (HICSS), Elizabeth Fong, Romain Gaucher, Vadim Okun, Paul E. Black, and Eric Dalci.
• Software Assurance with SAMATE Reference Dataset, Tool Standards, and Studies, October 2007, 26th Digital Avionics Systems Conference (DASC), Paul E. Black.
• Effect of Static Analysis Tools on Software Security: Preliminary Investigation, October 2007, Third Workshop on Quality of Protection (QoP), Vadim Okun, William F. Guthrie, Romain Gaucher, and Paul E. Black.
• SAMATE and Evaluating Static Analysis Tools, June 2007, Int'l Conf. on Reliable Software Technologies - Ada-Europe, Paul E. Black.
• Source Code Security Analysis Tool Functional Specification Version 1.0, NIST Special Publication 500-268, May 2007, Paul E. Black, Michael Kass, and Michael Koo. Superceded by Version 1.1.
• Web Application Scanners: Definitions and Functions, January 2007, 40th Hawaii Int'l Conf. on System Sciences (HICSS), Elizabeth Fong and Vadim Okun.
• SAMATE's Contribution to Information Assurance, Fall 2006, IAnewsletter, 9(2):4-7, Paul E. Black.
• Software Assurance During Maintenance, September 2006, Int'l Conf. on Software Maintenance (ICSM), Paul E. Black.
• Software Assurance Metrics And Tool Evaluation, June 2005, Int'l Conf. on Software Engineering Research and Practice (SERP), Paul E. Black.

Workshops 

• Static Analysis Tool Exposition (SATE) VI Workshop, MITRE, McLean, Virginia, September 2019.
• Static Analysis Tool Exposition (SATE) V Workshop, NIST, Gaithersburg, Maryland, March 2014.
• Static Analysis Tool Exposition (SATE) IV Workshop, co-located with the Spring 2012 Software Assurance Forum, MITRE, McLean, Virginia, March 2012.
• Static Analysis Tool Exposition (SATE) 2010 Workshop, co-located with the 13^th semi-annual Software Assurance Forum, NIST, Gaithersburg, Maryland, October 2010.
• Static Analysis Tool Exposition (SATE) 2009 Workshop, co-located with the 11^th semi-annual Software Assurance Forum, Arlington, Virginia, November 2009.
• Static Analysis Workshop (SAW), including Static Analysis Tool Exposition (SATE) 2008 reports, co-located with PLDI, Tucson, Arizona, June 2008.
• Static Analysis Summit II (SASII) in conjunction with SIGAda, Fairfax, Virginia, Nov 2007.
• Static Analysis Summit (SAS), Gaithersburg, Maryland, Jun 2006.
• Workshop on Software Security Assurance Tools, Techniques, and Metrics (SSATTM), Long Beach, California, Nov 2005.
• Workshop on Defining the State of the Art in Software Security Tools, Gaithersburg, Maryland, Aug 2005.

Other Presentations 

Many of these are available from us.

• Toward a "Periodic Table" of Bugs, or, How Can I Really Tell What’s Wrong With My Code?, 18 November 2015, OWASP Northern Virginia Chapter, Paul E. Black.
• SARD: A Software Assurance Reference Dataset, 10 September 2015, 2015 Cybersecurity Innovation Forum, Washington D.C., Paul E. Black.
• Towards a "Periodic Table" of Bugs, 7 May 2015, 15th High Confidence Software and Systems Conference (HCSS), Annapolis, Maryland, Paul E. Black, Irena Bojanova, Yaacov Yesha, and Yan Wu.
• A More Orthogonal Encyclopedia of Software Weaknesses than CWE, 15 April 2015, Software Security Assurance Exploratory Group, Washington, D.C., Paul E. Black, Irena Bojanova, Yaacov Yesha, and Yan Wu.
• Toward Precise and Accurate Descriptions of Weaknesses, May 2014, 14th High Confidence Software and Systems Conference (HCSS), Annapolis, Maryland, Paul E. Black.
• SATE V background, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Vadim Okun.
• Synthetic Test Cases (Juliet) Analysis Results, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Aurelien Delaitre.
• SATE V Ockham Sound Analysis Criteria, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Paul E. Black.
• CVE-Selected Analysis Results, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Bertrand Stivalet.
• Counting Bugs is Harder Than You Think, 26 October 2012, University of Pretoria, Paul E. Black.
• Choosing the Right Software Assurance Tools, 18 September 2012, Software Assurance Forum Fall 2012, MITRE, Virginia, Paul E. Black.
• Road to Confidence in IT Systems: SAMATE's SATE and SARD projects, 26 May 2012, Information Security and Privacy Advisory Board (ISPAB) Workshop (NIST), Paul E. Black.
• Toward CWE Compatibility Effectiveness, 31 October 2011, 7th Annual IT Security Automation Conference, Paul E. Black.
• Static Analysis & Static Analysis Tools: Their Role in Software Development, 28 October 2011, Information-technology Promotion Agency (IPA) Software Engineering Center, Japan, Paul E. Black. 
• Software Vulnerabilities Precluded by SPARK, 6 May 2011, 11th annual High Confidence Software and Systems Conference, Paul E. Black.
• View on Software Conformance Testing, 26 Aug 2010, Software Certification Consortium, Paul E. Black.
• Static Analysis Tool Exposition (SATE) and Reality, 13 May 2010, NSA CAS Workshop at HCSS, Paul E. Black.
• The Role of Static Analysis in Software Development, 16 April 2010, ACCU 2010, Paul E. Black.
• Product Labeling, 11 March 2010, 12th Semi-Annual Software Assurance Forum, Paul E. Black.
• Evaluating Static Analysis Tools, 8 July 2009, CNW at MIT/Lincoln Labs, Paul E. Black.
• Static Analysis Tool Exposition (SATE), 17 June 2009, DHS SwA Forum, Vadim Okun.
• Problems Counting Weaknesses from Static Analysis Tool Exposition (SATE), 22 May 2009, CAS SwA Forum at HCSS, Paul E. Black.
• Code Transparency and Diagnostic Capabilities, 21 April 2009, SSTC, Paul E. Black.
• Can Tools Help Software Assurance?, 29 August 2008, briefing to INFOSEC Research Council, Paul E. Black.
• Briefing on Static Analysis Tool Exposition (SATE) 2008, 25 June 2008, Center for Assured Software (CAS) Software Assurance Workshop, Paul E. Black.
• Observations on Static Analysis to Detect Weaknesses, 12 June 2008, SAW, Paul E. Black.
• SATE 2008 background, 12 June 2008, SAW, Vadim Okun.
• TT&PE Working Group Outbrief, 07 May 2008, DHS Forum Plenary Session, Michael Kass.
• Software Bugtraps: Software That Makes Software Better, 7 May 2008, DHS Software Assurance Forum, Paul E. Black.
• Code Transparency Panel: What's in YOUR Code?, 7 May 2008, DHS Software Assurance Forum, Paul E. Black (facilitator).
• Coordinating Session for May DHS Forum, 31 March 2008, DHS Working Group Chair Strategy Meeting, Michael Kass.
• Software Assurance Case NIST Role, 13 March 2008, OMG Software Assurance AB SIG meeting, Elizabeth Fong.
• Panel Discussion on SwA Tool Testing, 11 March 2008, OMG Government Information Days, Michael Kass.
• SAMATE Project Update; Understanding Web App Scanners, 31 January 2008, DHS Software Assurance Working Group, Paul E. Black and Romain Gaucher.
• Testing Web Application Scanner Tools, 30 October 2007, Verify Conference, Elizabeth Fong and Romain Gaucher.
• Source Code Security: WHY?, 9 August 2007, NIST SURF Review, Nathaniel Vaughn.
• Designing test cases for security analyzers, 9 August 2007, NIST SURF Review, Jonathan Diamond.
• C/C++/Java Source Code Obfuscator: A Filename Scrambler to Minimize Collisions, 1 August 2007, SAMATE Group Meeting, Cyril Lan.
• SAMATE Update: Web App & Source Code Analysis Tools, July 2007, DHS Software Assurance Working Group, Paul E. Black.
• Upcoming SAMATE Projects, May 2007, DHS Software Assurance Forum, Paul E. Black.
• SAMATE, May 2007, NIST, Paul E. Black.
• A Standard Reference Dataset (SRD) for Software Security, 5 March 2007, NIST, Paul E. Black.
• Software Assurance Metrics And Tool Evaluation, 22 January 2007, DHS Software Assurance Forum, Paul E. Black.
• SAMATE Source Code Security Analysis Specification, 22 January 2007, DHS Software Assurance Forum, Mike Kass.
• SAMATE Source Code Analysis Tool Test Plan, 22 January 2007, DHS Software Assurance Forum, Mike Koo.
• SAMATE Web Application Scanner Tool Testing, 22 January 2007, DHS Software Assurance Forum, Elizabeth Fong.
• Effect of Source Code Analysis Tools on Software Security: Preliminary Assessment, 22 January 2007, DHS Software Assurance Forum, Vadim Okun.
• Software Assurance Metrics And Tool Evaluation, or, Does the Emperor Really Have New Clothes?, October 2006, Tactical Information Assurance, Paul E. Black.
• Software Assurance Metrics and Tool Evaluation to Enhance Software Security, 8 August 2006, NIST SURF Review, Jeff Meister.
• Security Flaws & Testing, 14 April 2006, Virginia State University, Paul E. Black.
• Languages, 14 April 2006, Virginia State University, Paul E. Black.
• SAMATE and Web Application Vulnerability Assessment Tools, March 16, 2006, DHS Forum, Elizabeth Fong.
• Secure Software Tool Evaluation, March 2006, Lawrence Livermore National Laboratory, Paul E. Black.
• The SAMATE Project and How it Helps Enhance Software Trustworthiness, February 2006, OMG Technical Meeting, Vadim Okun.
• The Software Assurance Metrics and Tool Evaluation (SAMATE) Project, October 2005, OWASP AppSec DC, Paul E. Black.
• Software Assurance Metrics And Tool Evaluation, July 2005, DHS Software Assurance Forum, Paul E. Black.
• Testing, SAMATE, and Metrics, April 2005, Workshop on Assessment of IT Forensic Tools, Paul E. Black.