NOTICE: Due to a lapse in annual appropriations, most of this website is not being updated. Learn more.
Form submissions will still be accepted but will not receive responses at this time. Sections of this site for programs using non-appropriated funds (such as NVLAP) or those that are excepted from the shutdown (such as CHIPS and NVD) will continue to be updated.
An official website of the United States government
Here’s how you know
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis
Published
Author(s)
Zongxia Li, Andrew Mao, Jordan Boyd-Graber, Daniel Stephens, Emily Walpole, Alden A. Dima, Juan Fung
Abstract
Topic models are a popular tool for understanding text collections, but their evaluation has been a point of contention. Automated evaluation metrics such as coherence are often used, however, their validity has been questioned for neural topic models (NTMs) and can overlook a model's benefits in real-world applications. To this end, we conduct the first evaluation of neural, supervised and classical topic models in an interactive task-based setting. We combine topic models with a classifier and test their ability to help humans conduct content analysis and document annotation. From simulated, real user and expert pilot studies, the Contextual Neural Topic Model does the best on cluster evaluation metrics and human evaluations; however, LDA is competitive with two other NTMs under our simulated experiment and user study results, contrary to what coherence scores suggest. We show that current automated metrics do not provide a complete picture of topic modeling capabilities, but the right choice of NTMs can be better than classical models on practical tasks.
Proceedings Title
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Conference Dates
March 17-22, 2024
Conference Location
St. Julian's, MT
Conference Title
The 18th Conference of the European Chapter of the Association for Computational Linguistics
Li, Z.
, Mao, A.
, Boyd-Graber, J.
, Stephens, D.
, Walpole, E.
, Dima, A.
and Fung, J.
(2024),
Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian's, MT, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=956761
(Accessed October 8, 2025)