SAMATE Publications

Papers and Reports | Workshops | Other Publications

We also have a bibliography of other relevant papers and publications.

Papers and Reports

• Comprehensively Labeled Weakness and Vulnerability Datasets via Unambiguous Formal Bugs Framework (BF) Specifications, in IT Professional, vol. 26, no. 1, pp. 60-68, Jan.-Feb. 2024, Irena Bojanova, DOI: 10.1109/MITP.2024.3358970.
• "Vulnerability Test Suite Generator (VTSG) Version 3", October 2023, NIST Interagency Report (IR) 8493. Paul E. Black, William Mentzer, Elizabeth Fong, and Bertrand Stivalet, DOI 10.6028/NIST.IR.8493
• Labeling Software Security Vulnerabilities," in IT Professional, vol. 25, no. 5, pp. 64-70, Sep.-Oct. 2023, Irena Bojanova and John J. Guerrerio, DOI: 10.1109/MITP.2023.3314368 .
• Critical Software Security Weaknesses, in IT Professional, vol. 25, no. 04, pp. 11-16, 2023, Assane Gueye, Carlos E. Galhardo, and Irena Bojanova, DOI: 10.1109/MITP.2023.3297387.
• "SATE VI Report: Bug Injection and Collection," June 2023, NIST Special Publication (SP) 500-341. Aurelien Delaitre, Paul E. Black, Damien Cupif, Guillaume Haben, Alex-Kevin Loembe, Vadim Okun, Yann Prono, DOI: 10.6028/NIST.SP.500-341
• Heartbleed Revisited: Is it just a Buffer Over-Read?, in IT Professional, vol. 25, no. 2, pp. 83-89, Mar.-Apr. 2023, Irena Bojanova and Carlos E. Galhardo, DOI: 10.1109/MITP.2023.3259119.
• Static Analysis Tool Exposition (SATE) VI: Mobile Track Report, March 2023, NIST Internal Report (IR) 8462, Michael Ogata. DOI: 10.6028/NIST.IR.8462.
• Bug, Fault, Error, or Weakness: Demystifying Software Security Vulnerabilities, IT Professional, vol. 25, no. 1, pp. 7-12, Jan.-Feb. 2023, Irena Bojanova and Carlos E. Galhardo, DOI: 10.1109/MITP.2023.3238631
• Data Type Bugs Taxonomy: Integer Overflow, Juggling, and Pointer Arithmetics in Spotlight, 2022 IEEE 29th Annual Software Technology Conference (STC 2022), All Virtual, 2022, pp. 192-205, Irena Bojanova, Carlos E. Galhardo, Sara Moshtari, DOI: 10.1109/STC55697.2022.00035.
• Input/Output Check Bugs Taxonomy: Injection Errors in Spotlight, 2021 IEEE 31st  International Symposium on Software Reliability Engineering Workshops (ISSREW),  Wuhan, CN, 2021, pp. 111-120 , Irena Bojanova, Carlos E. Galhardo, Sara Moshtari, DIO: 10.1109/ISSREW53611.2021.00052.
• A Decade of Reoccurring Software Weaknesses, in IEEE Security & Privacy, vol. 19, no. 6, pp. 74-82, Nov.-Dec. 2021, Assane Gueye, Carlos E. Galhardo, Irena Bojanova, Peter Mell, DOI: 10.1109/MSEC.2021.3082757.
• "Guidelines on Minimum Standards for Developer Verification of Software," October 2021, NIST Internal Report (IR) 8397, Paul E. Black, Vadim Okun, and Barbara Guttman, DOI 10.6028/NIST.IR.8397
• Classifying Memory Bugs Using Bugs Framework Approach, 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC) ,  All Virtual,  2021, pp. 1157-1164, Irena Bojanova, Carlos E. Galhardo, DOI: 10.1109/COMPSAC51774.2021.00159.
• Algorithms and Data Structures for New Models of Computation, Jan/Feb 2021, IT Professional, 23(1):9-15, Paul E. Black, David Flater, and Irena Bojanova. DOI 10.1109/MITP.2020.3042858.
• C. E. Galhardo, P. Mell, I. Bojanova and A. Gueye, “Measurements of the Most Significant Software Security Weaknesses,” Annual Computer Security Applications Conference (ACSAC), pp. 154–164, Dec. 2020, Carlos E. Galhardo, Peter Mell, Irena Bojanova, Assane Gueye, DOI: 10.1145/3427228.3427257.
• SATE VI Ockham Sound Analysis Criteria, April 2020, NIST Internal Report (IR) 8304, Paul E. Black and Kanwardeep Singh Walia. DOI 10.6028/NIST.IR.8304. 
  The data and programs to reproduce these results are available at DOI 10.18434/M32187 or https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-VI-2020/ockhamCriteriaSATEVIdata2020.tar.xz (4.5 Megabytes download; 128 Megabytes uncompressed). A README file is available at https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-VI-2020/README
• Information Exposure (IEX): A New Class in the Bugs Framework (BF), 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Milwaukee, WI, US, 2019, pp. 559-564, Irena Bojanova,  Yaacov Yesha, Paul E. Black, Yan Wu, DOI: 10.1109/COMPSAC.2019.00086 .
• Opaque Wrappers and Patching: Negative Results, November 2019, Computer, 52(12):89-93, Paul E. Black and Monika Singh. DOI 10.1109/MC.2019.2936071. PMCID PMC7066996.
• Formal Methods for Statistical Software, October 2019, National Institute of Standards and Technology (NIST) Interagency Report (IR) 8274, Paul E. Black. DOI 10.6028/NIST.IR.8274.
• SATE V Report: Ten Years of Static Analysis Tool Expositions, October 2018, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-326, Aurelien Delaitre, Bertrand Stivalet,Paul E. Black, Vadim Okun, Athos Ribeiro, and Terry S. Cohen. DOI 10.6028/NIST.SP.500-326.
• Randomness Classes in Bugs Framework (BF): True-Random Number Bugs (TRN) and Pseudo-Random Number Bugs (PRN), July 2018, 2018 IEEE 42nd Annual Computers, Software & Applications Conference (COMPSAC), Tokyo, Japan, 2018, pp. 738-745, Irena Bojanova, Yaacov Yesha, and Paul E. Black. DOI 10.1109/COMPSAC.2018.00110.
• Juliet 1.3 Test Suite: Changes From 1.2, June 2018, National Institute of Standards and Technology (NIST) Technical Note (TN) 1995, Paul E. Black. DOI 10.6028/NIST.TN.1995.
• A Software Assurance Reference Dataset: Thousands of Programs With Known Bugs, April 2018, Journal of Research of NIST, Volume 123:123005, Paul E. Black. DOI 10.6028/jres.123.005.
• Cryptography classes in bugs framework (BF): Encryption bugs (ENC), verification bugs (VRF), and key management bugs (KMN), 2017 IEEE 28th Annual Software Technology Conference (STC), 2017, Gaithersburg, MD, US, pp. 1-8, Irena Bojanova, Yaacov Yesha, and Paul E. Black, DOI: 10.1109/STC.2017.8234453
• SARD: Thousands of Reference Programs for Software Assurance, October 2017, Journal of Cyber Security and Information Systems - Tools & Testing Techniques for Assured Software - DoD Software Assurance Community of Practice: Volume 2, 5(3):6-13, Paul E. Black.
• Improving Software Assurance through Static Analysis Tool Expositions, October 2017, Journal of Cyber Security and Information Systems - Tools & Testing Techniques for Assured Software - DoD Software Assurance Community of Practice: Volume 2, 5(3):14-22, Terry S. Cohen, Damien Cupif, Aurelien Delaitre, Charles D. De Oliveira, Elizabeth Fong, and Vadim Okun.
• Impact of Code Complexity on Software Analysis, February 2017, NIST Internal Report (IR) 8165 Update 1, Charles D. DeOliveira, Elizabeth Fong, and Paul E. Black. DOI 10.6028/NIST.IR.8165-upd1.
• Defeating Buffer Overflow: A Trivial but Dangerous Bug, November/December 2016, IT Professional, 18(6):58-61, Paul E. Black and Irena Bojanova. DOI 10.1109/MITP.2016.117.
• Report of the Workshop on Software Measures and Metrics to Reduce Security Vulnerabilities (SwMM-RSV), October 2016, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-320, Paul E. Black and Elizabeth Fong. DOI 10.6028/NIST.SP.500-320.
• The Bugs Framework (BF): A Structured Approach to Express Bugs, August 2016, 2016 IEEE International Conference on Software Quality, Reliability, and Security (QRS 2016), Vienna, Austria, Irena Bojanova, Paul E. Black, Yaacov Yesha, and Yan Wu. DOI 10.1109/QRS.2016.29.
• Large Scale Generation of Complex and Faulty PHP Test Cases, April 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), Chicago, IL, Bertrand Stivalet and Elizabeth Fong. DOI: 10.1109/ICST.2016.43. Best paper award for the tool demo track.
• SATE V Ockham Sound Analysis Criteria, March 2016, NIST Internal Report (IR) 8113, Paul E. Black and Athos Ribeiro. Noted 23 June 2017. DOI 10.6028/NIST.IR.8113. 
  The data and programs to reproduce these results are available at DOI 10.18434/T4WC7V or https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-V-2016/ockhamCriteriaSATEVdata2017.tar.xz (9.3 Megabytes download; 390 Megabytes uncompressed). A README file is available at https://nist-sate-ockham-sound-analysis-criteria-evaluation-material.s3.amazonaws.com/ockham-sate-V-2016/README
• Evaluating Bug Finders - Test and Measurement of Static Code Analyzers, May 2015, 2015 IEEE/ACM 1st International Workshop on Complex Faults and Failures in Large Software Systems (COUFLESS), Florence, Italy, Aurelien Delaitre, Bertrand Stivalet, Elizabeth Fong, and Vadim Okun.
• Fuzz Testing for Software Assurance, 1 March 2015, Crosstalk, The Journal of Defense Software Engineering, 28:35–37, Vadim Okun and Elizabeth N. Fong.
• A Basic CWE-121 Buffer Overflow Effectiveness Test Suite, April 2013, Proc. Sixth Latin-America Symposium on Dependable Computing (LADC 2013), Paul E. Black, Hsiao-Ming (Michael) Koo, and Thomas Irish.
• Report on the Static Analysis Tool Exposition (SATE) IV, January 2013, NIST Special Publication (SP) 500-297, Vadim Okun, Aurelien Delaitre, and Paul E. Black. DOI 10.6028/NIST.SP.500-297.
• Report on the Metrics and Standards for Software Testing (MaSST) Workshop 2012, December 2012, NIST Internal Report (IR) 7920, Paul E. Black and Elizabeth Fong. DOI 10.6028/NIST.IR.7920.
• Juliet 1.1 C/C++ and Java Test Suite, October 2012, Computer, 45(10):88-90, Tim Boland and Paul E. Black. DOI 10.1109/MC.2012.345.
• Static Analyzers: Seat Belts for Your Code, May-June 2012, Security & Privacy, 10(3):48-52, Paul E. Black, DOI 10.1109/MSP.2012.2.
• Software Vulnerabilities Precluded by SPARK, November 2011, ACM Int'l Conf. on Ada and Related Technologies: Engineering Safe, Secure, and Reliable Software (SIGAda 2011), Paul E. Black (NIST), Chris E. Dupilka (U.S. DoD), F. David Jones, and Joyce Tokar (Pyrrhus Software).
• Report on the Third Static Analysis Tool Exposition (SATE 2010), October 2011, NIST Special Publication (SP) 500-283, Vadim Okun, Aurelien Delaitre, and Paul E. Black. DOI 10.6028/NIST.SP.500-283.
• Counting Bugs is Harder Than You Think, September 2011, 11th IEEE Int'l Working Conference on Source Code Analysis and Manipulation (SCAM 2011), Williamsburg, VA, Paul E. Black.
• Source Code Security Analysis Tool Test Plan Version 1.1, July 2011, NIST Special Publication 500-270 v1.1, Michael Koo, Romain Gaucher,  Charline Cleraux, and Jenise Reyes Rodriguez. DOI 10.6028/NIST.SP.500-270v1.1.
• Source Code Security Analysis Tool Functional Specification Version 1.1, NIST Special Publication 500-268 v1.1, February 2011, Paul E. Black, Michael Kass, Michael Koo, and Elizabeth Fong. DOI 10.6028/NIST.SP.500-268v1.1.
• Toward a Preliminary Framework for Assessing the Trustworthiness of Software, November 2010, National Institute of Standards and Technology (NIST) Internal Report (IR) 7755, Tim Boland, Charline Cleraux, and Elizabeth Fong. DOI 10.6028/NIST.IR.7755.
• The Second Static Analysis Tool Exposition (SATE) 2009, June 2010, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-287, Vadim Okun, Aurelien Delaitre, and Paul E. Black. DOI 10.6028/NIST.SP.500-287.
• Static Analysis Tool Exposition (SATE) 2008, June 2009, National Institute of Standards and Technology (NIST) Special Publication (SP) 500-279, Vadim Okun, Romain Gaucher, and Paul E. Black, editors. DOI 10.6028/NIST.SP.500-279.
• Static Analyzers in Software Engineering, CrossTalk, The Journal of Defense Software Engineering, 22(3):16-17, March/April 2009, Paul E. Black.
• Software Assurance Tools: Web Application Security Scanner Functional Specification Version 1.0, January 2008, NIST Special Publication (SP) 500-269, Paul E. Black, Elizabeth Fong, Vadim Okun, and Romain Gaucher. DOI 10.6028/NIST.SP.500-269.
• Building a Test Suite for Web Application Scanners, January 2008, 41st Hawaii Int'l Conf. on System Sciences (HICSS), Elizabeth Fong, Romain Gaucher, Vadim Okun, Paul E. Black, and Eric Dalci.
• Software Assurance with SAMATE Reference Dataset, Tool Standards, and Studies, October 2007, 26th Digital Avionics Systems Conference (DASC), Paul E. Black.
• Effect of Static Analysis Tools on Software Security: Preliminary Investigation, October 2007, Third Workshop on Quality of Protection (QoP), Vadim Okun, William F. Guthrie, Romain Gaucher, and Paul E. Black.
• SAMATE and Evaluating Static Analysis Tools, June 2007, Int'l Conf. on Reliable Software Technologies - Ada-Europe, Paul E. Black.
• Source Code Security Analysis Tool Functional Specification Version 1.0, NIST Special Publication 500-268, May 2007, Paul E. Black, Michael Kass, and Michael Koo. Superceded by Version 1.1.
• Web Application Scanners: Definitions and Functions, January 2007, 40th Hawaii Int'l Conf. on System Sciences (HICSS), Elizabeth Fong and Vadim Okun.
• SAMATE's Contribution to Information Assurance, Fall 2006, IAnewsletter, 9(2):4-7, Paul E. Black.
• Software Assurance During Maintenance, September 2006, Int'l Conf. on Software Maintenance (ICSM), Paul E. Black.
• Software Assurance Metrics And Tool Evaluation, June 2005, Int'l Conf. on Software Engineering Research and Practice (SERP), Paul E. Black.

Workshops 

• Static Analysis Tool Exposition (SATE) VI Workshop, MITRE, McLean, Virginia, September 2019.
• Static Analysis Tool Exposition (SATE) V Workshop, NIST, Gaithersburg, Maryland, March 2014.
• Static Analysis Tool Exposition (SATE) IV Workshop, co-located with the Spring 2012 Software Assurance Forum, MITRE, McLean, Virginia, March 2012.
• Static Analysis Tool Exposition (SATE) 2010 Workshop, co-located with the 13^th semi-annual Software Assurance Forum, NIST, Gaithersburg, Maryland, October 2010.
• Static Analysis Tool Exposition (SATE) 2009 Workshop, co-located with the 11^th semi-annual Software Assurance Forum, Arlington, Virginia, November 2009.
• Static Analysis Workshop (SAW), including Static Analysis Tool Exposition (SATE) 2008 reports, co-located with PLDI, Tucson, Arizona, June 2008.
• Static Analysis Summit II (SASII) in conjunction with SIGAda, Fairfax, Virginia, Nov 2007.
• Static Analysis Summit (SAS), Gaithersburg, Maryland, Jun 2006.
• Workshop on Software Security Assurance Tools, Techniques, and Metrics (SSATTM), Long Beach, California, Nov 2005.
• Workshop on Defining the State of the Art in Software Security Tools, Gaithersburg, Maryland, Aug 2005.

Other Presentations 

Many of these are available from us.

• Bugs Framework: Formalizing Cybersecurity Weaknesses and Vulnerabilities, 4 April 2024, Office of the National Cyber Director (ONCD), Irena Bojanova.
• Bugs Framework: Formalizing Cybersecurity Weaknesses and Vulnerabilities, 28 March, 2024, National Defense Industrial Association (NDIA) Trust & Assurance Committee (T&AC), Irena Bojanova.
• Bugs Framework (BF): Formalizing Software Security Bugs, Weaknesses, and Vulnerabilities, 8 November 2023, NIST ITL Science Day 2023, Irena Bojanova.
• BF: Bug, Fault, Error, Weakness, or Vulnerability, 8 November 2023, NIST ITL Science Day 2023, Irena Bojanova.
• Labeling Software Security Vulnerabilities, 8 November 2023, NIST ITL Science Day 2023, Irena Bojanova.
• Bugs Framework (BF): Overview, 8 November 2023, NIST - INMETRO Discussion with Brazilian Government officials, Irena Bojanova.
• Bugs Framework (BF): BF Formal Language, October 25 2023, NIST ITL CSD Security Research Review (SRR), Irena Bojanova.
• Bugs Framework (BF): BF for AI and ML (Ontology of Software Bugs and Weaknesses; and Reference Dataset of Formally Described Software Security Vulnerabilities), 17 February 2023, Johns Hopkins University Applied Physics Laboratory (JHU APL), Irena Bojanova.
• BF for CHIPS: A Formal Language for Describing and Backtracking Chips Triggered Software Vulnerabilities, 8 February 2023, NIST ITL SSD SAMATE meeting, Irena Bojanova.
• Explainable Vulnerabilities Descriptions with NIST BF, 1 December 2022, Ericson Program Analysis Workshop, Irena Bojanova.
• BF Keynote-Explainable Vulnerabilities Descriptions with NIST BF, 31 October 2022, IEEE International Symposium on Software Reliability Engineering, Software Hardware Interaction Faults & International Workshop on Software Faults (ISSRE, SHIFT & IWSF 2022), Irena Bojanova.
• BF Lecture: Understanding Software Security Vulnerabilities Descriptions with NIST BF, IEEE Reliability Society (RS) Certificate Program 2022, Jul. 15, 2022, Irena Bojanova.
• Bugs Framework (BF), BIECO EU Research Project, Nov. 16, 2021, Irena Bojanova.
• Bugs Framework (BF),NIST ITL CSD Security Research Review (SRR), Nov. 8, 2021, Irena Bojanova.
• Input/Output Check Bugs and Injection, NIST ITL Science Day 2021, 28 October 2021, Irena Bojanova, Carlos C. Galhardo.
• The NIST Bugs Framework (BF) - Input/Output Check Bugs Taxonomy: Injection Errors in Spotlight, 25 October 2021, IEEE International Symposium on Software Reliability Engineering’ (ISSRE 2021), Irena Bojanova.
• Bugs Framework (BF), 23 July 2021, Johns Hopkins University Applied Physics Laboratory (JHU APL), Irena Bojanova.
• Bugs Framework(BF), 20, July 2021, Cybersecurity and Infrastructure Security Agency (CISA), Irena Bojanova.
• Bugs Framework (BF): Data Type Bugs Taxonomy: Integer Overflow, Juggling, and Pointer Arithmetics in Spotlight, 3 October 2022, IEEE Software Technology Conference (STC 2022), Irena Bojanova.
• Classifying Memory Bugs Using Bugs Framework Approach, July 12 2021, 2021 IEEE 45rd Annual Computer Software and Applications Conference (COMPSAC), Irena Bojanova.
• Bugs Framework (BF), 20 June 2021, NIST ITL SSD Assessment Panel, Irena Bojanova.
• Bugs Framework (BF), 4 May 2021, Invited Talk, St. John’s University, MS CYB Spring 2021 Spring Research Seminary, Irena Bojanova.
• Memory Bugs Classes in Bugs Framework, 29 October 2020 NIST ITL Science Day 2020, Irena Bojanova, Carlos C. Galhardo.
• Bugs Framework (BF): Memory Corruption/Disclosure Classes, 16 September 2020, NIST ITL CSD Security Research Review (SRR), Irena Bojanova.
• Memory Bugs Classes in NIST Bugs Framework (BF), - and Handouts -, 15 September 2020, High Confidence Software and Systems Conference (HCSS), Irena Bojanova, Carlos C. Galhardo.
• Bugs Framework (BF), 24 July 2020, NIST ITL SSD Division Chief meeting with Vint Cerf, VP and Chief Internet Evangelist, Google, Irena Bojanova.
• Bugs Framework (BF), 18 March 2020, Rochester Institute of Technology (RIT), Irena Bojanova.
• Information Exposure (IEX) Class in the Bugs Framework (BF), 6 November 2019, NIST ITL Science Day 2019, Irena Bojanova.
• Bugs Framework (BF) – Your Best Friend?, 19 September 2019, SATE VI Workshop, Irena Bojanova.
• Bugs Framework (BF), 3 September 2019, NIST ITL SSD Software Systems Review (SSR), Irena Bojanova.
• Bugs Framework (BF): Introduction, Bugs Framework (BF): Information Exposure (IEX), Random Number Generation (RND), Cryptographic Store or Transfer (CST), 22 August 2019, Networking and Information Technology Research and Development (NITRD) Program, National Coordination Office (NCO), CSIA, Irena Bojanova.
• Information Exposure (IEX): A New Class in the Bugs Framework (BF), 15 July 2019, 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Irena Bojanova.
• Bugs Framework (BF): Introduction, Bugs Framework (BF): Information Exposure (IEX), Random Number Generation (RND), Cryptographic Store or Transfer (CST), 11 July 2019, Networking and Information Technology Research and Development (NITRD) Program, National Coordination Office (NCO), SPSQ, Irena Bojanova.
• Information Exposure (IEX) Class in the Bugs Framework (BF), 29 April 2019, High Confidence Software and Systems Conference (HCSS) 2019, Irena Bojanova.
• Bugs Framework (BF), 1 February 2018, Networking and Information Technology Research and Development (NITRD) Program, National Coordination Office (NCO), SPSQ, Irena Bojanova, Paul E. Black.
• Cryptography Classes in Bugs Framework (BF): Encryption Bugs (ENC), Verification Bugs (VRF), and Key Management Bugs (KMN).pdf, 2 November 2017, NIST ITL Science Day 2017, Irena Bojanova.
• Cryptography Classes in Bugs Framework (BF), 25 September 2017, IEEE Software Technology Conference (STC), Irena Bojanova.
• The Bugs Framework (BF) Hands-On, - and Exercises -, 25 July 2017, IEEE Software Quality, Reliability, and Security Conference (QRS) 2017, Irena Bojanova.
• The new Cryptographic Store/Transfer (CST) Class from Bugs Framework (BF), 8 May 2017, High Confidence Software and Systems Conference (HCSS) 2017, Irena Bojanova.
• Bugs Framework (BF) Tutorial, - and Handouts -, 4 April 2017, Symposium on the Science of Security (HotSoS), Irena Bojanova.
• Bugs Framework (BF): Software developer’s and tester’s Best Friend, 2 November 2016, NIST ITL CSD Security Research Review (SRR), Irena Bojanova.
• Bugs Framework (BF), 13 October 2016, NIST ITL Science Day 2016, Irena Bojanova, Paul E. Black.
• Bugs Framework (BF): A Structured Integrated Framework to Express Bugs, 10 May 10 2016, High Confidence Software and Systems Conference (HCSS), Irena Bojanova, Paul E. Black.
• Toward a "Periodic Table" of Bugs, or, How Can I Really Tell What’s Wrong With My Code?, 18 November 2015, OWASP Northern Virginia Chapter, Paul E. Black.
• SARD: A Software Assurance Reference Dataset, 10 September 2015, 2015 Cybersecurity Innovation Forum, Washington D.C., Paul E. Black.
• Towards a "Periodic Table" of Bugs, 7 May 2015, 15th High Confidence Software and Systems Conference (HCSS), Annapolis, Maryland, Paul E. Black, Irena Bojanova, Yaacov Yesha, and Yan Wu.
• A More Orthogonal Encyclopedia of Software Weaknesses than CWE, 15 April 2015, Software Security Assurance Exploratory Group, Washington, D.C., Paul E. Black, Irena Bojanova, Yaacov Yesha, and Yan Wu.
• Towards a Periodic Table of Bugs, 8 April 2015, NIST ITL SSD SAMATE meeting, Irena Bojanova.
• Formalizing Software Bugs, 9 December 2014, NIST ITL SSD Division Chief meeting with the Information-technology Promotion Agency (IPA), Japan delegation, NIST 222/A318, Irena Bojanova.
• Toward Precise and Accurate Descriptions of Weaknesses, May 2014, 14th High Confidence Software and Systems Conference (HCSS), Annapolis, Maryland, Paul E. Black.
• SATE V background, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Vadim Okun.
• Synthetic Test Cases (Juliet) Analysis Results, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Aurelien Delaitre.
• SATE V Ockham Sound Analysis Criteria, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Paul E. Black.
• CVE-Selected Analysis Results, 14 March 2014, Static Analysis Tool Exposition Workshop (SATE V), NIST, Gaithersburg, Maryland, Bertrand Stivalet.
• Counting Bugs is Harder Than You Think, 26 October 2012, University of Pretoria, Paul E. Black.
• Choosing the Right Software Assurance Tools, 18 September 2012, Software Assurance Forum Fall 2012, MITRE, Virginia, Paul E. Black.
• Road to Confidence in IT Systems: SAMATE's SATE and SARD projects, 26 May 2012, Information Security and Privacy Advisory Board (ISPAB) Workshop (NIST), Paul E. Black.
• Toward CWE Compatibility Effectiveness, 31 October 2011, 7th Annual IT Security Automation Conference, Paul E. Black.
• Static Analysis & Static Analysis Tools: Their Role in Software Development, 28 October 2011, Information-technology Promotion Agency (IPA) Software Engineering Center, Japan, Paul E. Black. 
• Software Vulnerabilities Precluded by SPARK, 6 May 2011, 11th annual High Confidence Software and Systems Conference, Paul E. Black.
• View on Software Conformance Testing, 26 Aug 2010, Software Certification Consortium, Paul E. Black.
• Static Analysis Tool Exposition (SATE) and Reality, 13 May 2010, NSA CAS Workshop at HCSS, Paul E. Black.
• The Role of Static Analysis in Software Development, 16 April 2010, ACCU 2010, Paul E. Black.
• Product Labeling, 11 March 2010, 12th Semi-Annual Software Assurance Forum, Paul E. Black.
• Evaluating Static Analysis Tools, 8 July 2009, CNW at MIT/Lincoln Labs, Paul E. Black.
• Static Analysis Tool Exposition (SATE), 17 June 2009, DHS SwA Forum, Vadim Okun.
• Problems Counting Weaknesses from Static Analysis Tool Exposition (SATE), 22 May 2009, CAS SwA Forum at HCSS, Paul E. Black.
• Code Transparency and Diagnostic Capabilities, 21 April 2009, SSTC, Paul E. Black.
• Can Tools Help Software Assurance?, 29 August 2008, briefing to INFOSEC Research Council, Paul E. Black.
• Briefing on Static Analysis Tool Exposition (SATE) 2008, 25 June 2008, Center for Assured Software (CAS) Software Assurance Workshop, Paul E. Black.
• Observations on Static Analysis to Detect Weaknesses, 12 June 2008, SAW, Paul E. Black.
• SATE 2008 background, 12 June 2008, SAW, Vadim Okun.
• TT&PE Working Group Outbrief, 07 May 2008, DHS Forum Plenary Session, Michael Kass.
• Software Bugtraps: Software That Makes Software Better, 7 May 2008, DHS Software Assurance Forum, Paul E. Black.
• Code Transparency Panel: What's in YOUR Code?, 7 May 2008, DHS Software Assurance Forum, Paul E. Black (facilitator).
• Coordinating Session for May DHS Forum, 31 March 2008, DHS Working Group Chair Strategy Meeting, Michael Kass.
• Software Assurance Case NIST Role, 13 March 2008, OMG Software Assurance AB SIG meeting, Elizabeth Fong.
• Panel Discussion on SwA Tool Testing, 11 March 2008, OMG Government Information Days, Michael Kass.
• SAMATE Project Update; Understanding Web App Scanners, 31 January 2008, DHS Software Assurance Working Group, Paul E. Black and Romain Gaucher.
• Testing Web Application Scanner Tools, 30 October 2007, Verify Conference, Elizabeth Fong and Romain Gaucher.
• Source Code Security: WHY?, 9 August 2007, NIST SURF Review, Nathaniel Vaughn.
• Designing test cases for security analyzers, 9 August 2007, NIST SURF Review, Jonathan Diamond.
• C/C++/Java Source Code Obfuscator: A Filename Scrambler to Minimize Collisions, 1 August 2007, SAMATE Group Meeting, Cyril Lan.
• SAMATE Update: Web App & Source Code Analysis Tools, July 2007, DHS Software Assurance Working Group, Paul E. Black.
• Upcoming SAMATE Projects, May 2007, DHS Software Assurance Forum, Paul E. Black.
• SAMATE, May 2007, NIST, Paul E. Black.
• A Standard Reference Dataset (SRD) for Software Security, 5 March 2007, NIST, Paul E. Black.
• Software Assurance Metrics And Tool Evaluation, 22 January 2007, DHS Software Assurance Forum, Paul E. Black.
• SAMATE Source Code Security Analysis Specification, 22 January 2007, DHS Software Assurance Forum, Mike Kass.
• SAMATE Source Code Analysis Tool Test Plan, 22 January 2007, DHS Software Assurance Forum, Mike Koo.
• SAMATE Web Application Scanner Tool Testing, 22 January 2007, DHS Software Assurance Forum, Elizabeth Fong.
• Effect of Source Code Analysis Tools on Software Security: Preliminary Assessment, 22 January 2007, DHS Software Assurance Forum, Vadim Okun.
• Software Assurance Metrics And Tool Evaluation, or, Does the Emperor Really Have New Clothes?, October 2006, Tactical Information Assurance, Paul E. Black.
• Software Assurance Metrics and Tool Evaluation to Enhance Software Security, 8 August 2006, NIST SURF Review, Jeff Meister.
• Security Flaws & Testing, 14 April 2006, Virginia State University, Paul E. Black.
• Languages, 14 April 2006, Virginia State University, Paul E. Black.
• SAMATE and Web Application Vulnerability Assessment Tools, March 16, 2006, DHS Forum, Elizabeth Fong.
• Secure Software Tool Evaluation, March 2006, Lawrence Livermore National Laboratory, Paul E. Black.
• The SAMATE Project and How it Helps Enhance Software Trustworthiness, February 2006, OMG Technical Meeting, Vadim Okun.
• The Software Assurance Metrics and Tool Evaluation (SAMATE) Project, October 2005, OWASP AppSec DC, Paul E. Black.
• Software Assurance Metrics And Tool Evaluation, July 2005, DHS Software Assurance Forum, Paul E. Black.
• Testing, SAMATE, and Metrics, April 2005, Workshop on Assessment of IT Forensic Tools, Paul E. Black.