The amount of software in today's information world is far too large to check manually. Automated tools are a must. These tools can help design and build the right software in the first place, but they can also help if the system being designed includes contract software. The NIST Software Assurance Metrics and Tool Evaluation (SAMATE) project seeks to help develop standard evaluation measures and methods for software assurance. This article outlines how SAMATE is developing a taxonomy of tools and techniques, helping develop a taxonomy of weaknesses, developing test matter and procedures for classes of tools, and an on-line, publicly available reference dataset of thousands of samples of flawed software. SAMATE is also embarking on studies and experiments to quantify software assurance.