SP 500-268 - 02/05/2007-01

[SAMATE Home | IntrO TO SAMATE | SARD | SATE | Bugs Framework | Publications | Tool Survey | Resources]

Thanks! Here's another comment. If you point to CWE definitions, then what do you do about tools that find only a few cases in a category? I know of no existing code scanner that can find all common ways in which stack overflows happen. In fact, I recently played with a commercial scanner and it found only 1 out of 5 more or less common overflows in tests that I crafted. It found nothing wrong in 4 out of 5 tests, and I believe that it's one of the best code scanners available. I'm certain that I would find a similar situation in most of the other CWEs you listed. Will there be standard tests defined for tools to pass? It seems likely (and true according to Rice's theorem) that someone could always come up with a more obfuscated example (note that I don't consider the tests I used to be obfuscated) that the tools couldn't find. So, when does a tool meet the criteria? How well is good enough? It seems to me that you have no choice but to limit the general case to some test cases, and possibly grade the tools on how many cases they can discover (hoping that vendors don't include code specific for the tests you use).

Regards, Pascal