The mission of the Software Performance Project is to strengthen the scientific foundations of software performance measurement (ITL Objective A1, Develop the foundations of measurement science for IT).Its goals are:
Generally, any cost vs. benefits decision on how to construct, compile, deploy, or run software requires the ability to measure the performance effects well enough.The problem is that the practices, tools, and methods commonly used to measure those effects often yield results that are plausible but wrong.The underlying issues include small samples, faults and limitations in measuring instruments and statistical software, statistical assumptions being violated by the complex behaviors of modern hardware and operating systems, and analytical models of uncertainty that do not describe the actual variability of results.
A motivating example is measuring the performance cost of defensive code.Defensive code can mitigate risks that faulty software poses to economic and national security, but it is often omitted because of its perceived performance cost.When measured carefully, these costs may prove to be negligible, as they have in most cases covered in experiments done at NIST thus far.
NIST overcame the obstacles to its own experiments through a combination of
Our publications and software contributions enable others to apply the methods and tools that we used to their own software performance measurement needs, enabling more reliable measurements and fewer wrong decisions.