Agreement Behavior of Isolated Annotators for Maintenance Work-Order Data Mining

Published

September 27, 2019

Author(s)

Emily Hastings, Thurston Sexton, Michael Brundage, Melinda Hodkiewicz

Abstract

Maintenance work orders (MWOs) are an integral part of the maintenance workflow. These documents allow technicians to capture vital aspects of a maintenance job: observed symptoms, potential causes, solutions implemented, etc. These MWOs have often been disregarded during analysis because of the unstructured nature of the text they contain. However, many research efforts have recently emerged that clean these MWOs for analysis. One such effort uses a tagging method with an open source toolkit, named Nestor, which relies on experts classifying and annotating the words used in the MWOs. For example, an expert might classify the words "replace," "replaced," and "repalce" as "Solutions" and give the alias "replace" to all of them. This method greatly reduces the volume of words used in the MWOs and links words, including misspellings, that have the same or similar meanings. However, one issue with the current iteration of this tool, along with practical usage of data-annotation tools on the shop-floor more generally, is the usage of only one expert annotator at a time. How do we know that the classifications of a single annotator are correct, or if it is, for example, feasible to divide the tagging task among multiple experts? This paper examines the agreement behavior of multiple isolated experts classifying and annotating MWO data, and provides implications for implementing this tagging technique for use in authentic contexts. The results described here will help improve MWO classification leading to more accurate analysis of MWOS for decision-making support.

Proceedings Title

2019 Annual Conference of the Prognostics and Health Management Society

Conference Dates

September 21-27, 2019

Conference Location

Scottsdale, AZ, US

Pub Type

Conferences

Download Paper

Local Download

Keywords

maintenance, data cleaning, maintenance work orders

Citation

Hastings, E. , Sexton, T. , Brundage, M. and Hodkiewicz, M. (2019), Agreement Behavior of Isolated Annotators for Maintenance Work-Order Data Mining, 2019 Annual Conference of the Prognostics and Health Management Society, Scottsdale, AZ, US, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=928113 (Accessed April 23, 2024)