Building Trust in Forensic Science: Footwear Analysis

Summary

NIST statisticians are improving forensic science by questioning how evidence is interpreted and creating tools to support examiners.

Description

Deep Learning for Footwear Matching in Forensics

Starting in 2015, SED staff members began reviewing published literature on the recommended practice for forensic examiners to provide likelihood ratios (LRs) as a weight of evidence to summarize their findings in reports or testimony.  While several stated motivations for providing likelihood ratios were admirable (e.g., requiring examiners to consider at least two propositions when assessing the evidence, stopping the examiner from characterizing the probability any given proposition is true, and increasing the transparency of the examiner’s reasoning), several prominent LR proponents overstated the theoretical support for providing an LR and failed to properly address the influence of subjective modeling choices required to obtain an LR value.  Based on extensive review and study of the recommended practice, in 2017 SED staff published the paper “Likelihood Ratio as Weight of Evidence: A Closer Look” in the Journal of Research of NIST identifying potential concerns with examiners providing LR values and suggesting that examiners should either carefully address how modeling choices may affect their offered LR value or focus on providing factual presentations of the data they would use to form an LR, rather than the LR itself.  As a result of their work, these staff members were subpoenaed to testify a Federal evidence admissibility hearing for the case United States v. Gissantaner, in which the LR evidence was ultimately ruled inadmissible.  Though four oppositional responses to the paper have been published since 2017, none have addressed the fundamental concerns outlined in the paper. A follow-up article is in preparation after recent articles in the forensic science literature began citing the previously published responses as having refuted the concerns originally presented. Since 2015, SED members have led the technical aspects of a research project aiming to provide meaningful algorithmic support to examiners evaluating forensic footwear impression evidence. While several algorithms exist to help examiners determine the shoe make and model exhibited in crime scene impressions, other phases of evidence evaluation, such as quantifying correspondence of wear regions or randomly acquired characteristics (RACs), have not been algorithmically supported.  The NIST team has been developing an end-to-end comparison workflow to support examiners in all phases of evidence evaluation (design, size, wear, and RACs).  Major tasks have included assessing crime scene impression clarity, aligning test impressions from shoes of interest with the crime scene impression, evaluating pattern similarity, and finding relevant reference (i.e., ground truth known) comparisons to provide context for algorithm outputs.   

Initially funded through the Information Technology Laboratory’s Building the Future program, this project has been funded externally since 2018 through two separate National Institute of Justice (NIJ) grants.  The first NIJ grant led to two peer-reviewed publications with SED authors, one that described the workflow of the NIST system and another that examined how well an initial outsole pattern (i.e., whole shoe) comparison metric performed in separating truly mated pairs from other shoes of the same make, model, side, and size.  The aim of the current grant is to refine the NIST Footwear Impression Comparison System and ultimately to evaluate its performance on a pool of footwear impression comparisons that were used by the FBI to conduct a black box study of U.S. examiners.