Principle Investigators: Steve Talpas, William Drew
New Jersey Office of Homeland Security and Preparedness
Computer vision capabilities have rapidly been advancing and are expected to become an important component to incident and disaster response. However, majority of video analytics are not meeting public safety’s needs due to the lack of appropriate training data and requirements. In response, the State of New Jersey Office of Homeland Security and Preparedness (NJOHSP), in collaboration with the Massachusetts Institute of Technology (MIT) Lincoln Laboratory and Civil and Environmental Engineering departments, and Wayne State University has built an imagery and video dataset of representative and operational public safety scenarios.
Community outreach strongly informed the decision to have this dataset focus on foot pursuits and approaching an incident with body worn cameras and an airborne-based dataset to support wide area search and rescue operations. The dataset includes images from all fifty state of the United States. It includes operational images and videos from the Civil Air Patrol (CAP), the Defense Visual Information Distribution Service (DVIDS), Massachusetts Task Force One(MA-TF1), Unmanned Robotics Systems Analysis (URSA), and the United States Geological Survey (USGS). Representative content was largely complied from Creative Commons video hosted on YouTube. A small quantity of non-Creative Commons content was obtained with the permission of the content’s owners. Along with their collaborators, NJOHSP and MIT generated over thirty hours of video representative of some public safety scenarios. Working with the Lincoln Laboratory Supercomputing Center, the dataset was organized for efficient storage and compute.
Notably, included in the dataset are annotated operational images of various disasters collected by the Civil Air Patrol. These images were previously released into the public domain but just as raw images without labels. Since the Civil Air Patrol is a nationwide asset, we’ve labeled a wide range of relevant public safety features across different environments. Most excitingly, the corpus of available images that will continue to grow independent of our dataset effort.
Based on feedback from the public safety community, this dataset is positioned to support training analytics and building capabilities that improve incident or threat descriptions; enhance two-way situational awareness; and minimize implementing new workloads or policies.
To support development, two reference architectures were developed: one for testing at the FirstNet Innovation and Test Lab and AutoVAPS for evaluating IoT-enabled services hosted on a vehicle. AutoVAPS in particular used the HydraOne, an indoor experimental research and education platform for connected and autonomous driving research. These efforts can help build the next generation of first responder vehicles and drones.
More information on the effort can be found on ResearchGate and much of the software used to develop the dataset are hosted on GitHub. AutoVAPS is affliated with the Wayne State CAR Lab. Lastly this effort produced more than five peer-review publications.