Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Development of a Bias Compensating Q-Learning Controller for a Multi-Zone HVAC Facility

Published

Author(s)

Syed Ali Asad Rizvi, Amanda Pertzborn

Abstract

We present the development of a bias compensating reinforcement learning (RL) algorithm that optimizes thermal comfort (by minimizing tracking error) and control utilization (by penalizing setpoint deviations) in a multi-zone heating, ventilation, and air-conditioning (HVAC) lab facility subject to unmeasurable disturbances and unknown dynamics. It is shown that the presence of unmeasurable disturbance results in an inconsistent learning equation in traditional RL controllers leading to parameter estimation bias (even with integral action support), and in the extreme case, the divergence of the learning algorithm. We demonstrate this issue by applying the popular Q-learning algorithm to linear quadratic regulation (LQR) of a multi-zone HVAC environment and showing that, even with integral support, the algorithm exhibits bias issue during the learning phase when the HVAC disturbance is unmeasurable due to unknown heat gains, occupancy variations, light sources, and outside weather changes. To address this difficulty, we present a bias compensating learning equation that learns a lumped bias term as a result of disturbances (and possibly other sources) in conjunction with the optimal control parameters. Experimental results show that the proposed scheme not only recovers the bias-free optimal control parameters but it does so without explicitly learning the dynamic model or estimating the disturbances, demonstrating the effectiveness of the algorithm in addressing the above challenges.
Citation
IEEE/CAA Journal of Automatica Sinica
Volume
10
Issue
8

Keywords

Reinforcement learning, Q-learning, optimal tracking, HVAC control

Citation

Rizvi, S. and Pertzborn, A. (2023), Development of a Bias Compensating Q-Learning Controller for a Multi-Zone HVAC Facility, IEEE/CAA Journal of Automatica Sinica, [online], https://doi.org/10.1109/JAS.2023.123624, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934529 (Accessed April 23, 2024)
Created August 1, 2023, Updated March 6, 2024