Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Agreement Behavior of Isolated Annotators for Maintenance Work-Order Data Mining



Emily Hastings, Thurston Sexton, Michael Brundage, Melinda Hodkiewicz


Maintenance work orders (MWOs) are an integral part of the maintenance workflow. These documents allow technicians to capture vital aspects of a maintenance job: observed symptoms, potential causes, solutions implemented, etc. These MWOs have often been disregarded during analysis because of the unstructured nature of the text they contain. However, many research efforts have recently emerged that clean these MWOs for analysis. One such effort uses a tagging method with an open source toolkit, named Nestor, which relies on experts classifying and annotating the words used in the MWOs. For example, an expert might classify the words "replace," "replaced," and "repalce" as "Solutions" and give the alias "replace" to all of them. This method greatly reduces the volume of words used in the MWOs and links words, including misspellings, that have the same or similar meanings. However, one issue with the current iteration of this tool, along with practical usage of data-annotation tools on the shop-floor more generally, is the usage of only one expert annotator at a time. How do we know that the classifications of a single annotator are correct, or if it is, for example, feasible to divide the tagging task among multiple experts? This paper examines the agreement behavior of multiple isolated experts classifying and annotating MWO data, and provides implications for implementing this tagging technique for use in authentic contexts. The results described here will help improve MWO classification leading to more accurate analysis of MWOS for decision-making support.
Proceedings Title
2019 Annual Conference of the Prognostics and Health Management Society
Conference Dates
September 21-27, 2019
Conference Location
Scottsdale, AZ, US


maintenance, data cleaning, maintenance work orders


Hastings, E. , Sexton, T. , Brundage, M. and Hodkiewicz, M. (2019), Agreement Behavior of Isolated Annotators for Maintenance Work-Order Data Mining, 2019 Annual Conference of the Prognostics and Health Management Society, Scottsdale, AZ, US, [online], (Accessed June 13, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created September 26, 2019, Updated October 12, 2021