Agreement Behavior of Isolated Annotators for Maintenance Work-Order Data Mining
Emily Hastings, Thurston B. Sexton, Michael P. Brundage, Melinda Hodkiewicz
Maintenance work orders (MWOs) are an integral part of the maintenance workflow. These documents allow technicians to capture vital aspects of a maintenance job: observed symptoms, potential causes, solutions implemented, etc. These MWOs have often been disregarded during analysis because of the unstructured nature of the text they contain. However, many research efforts have recently emerged that clean these MWOs for analysis. One such effort uses a tagging method with an open source toolkit, named Nestor, which relies on experts classifying and annotating the words used in the MWOs. For example, an expert might classify the words "replace," "replaced," and "repalce" as "Solutions" and give the alias "replace" to all of them. This method greatly reduces the volume of words used in the MWOs and links words, including misspellings, that have the same or similar meanings. However, one issue with the current iteration of this tool, along with practical usage of data-annotation tools on the shop-floor more generally, is the usage of only one expert annotator at a time. How do we know that the classifications of a single annotator are correct, or if it is, for example, feasible to divide the tagging task among multiple experts? This paper examines the agreement behavior of multiple isolated experts classifying and annotating MWO data, and provides implications for implementing this tagging technique for use in authentic contexts. The results described here will help improve MWO classification leading to more accurate analysis of MWOS for decision-making support.
2019 Annual Conference of the Prognostics and Health Management Society