TRUC is a new evaluation conference series that will build the infrastructure required for evaluating aspects of trustworthiness in AI systems. TRUC is distinct from, but related to, the NIST AI Risk Management Framework effort. The purpose of the framework is to provide guidance on managing the risks associated with the use of artificial intelligence. The goal of TRUC is to identify risk/harms of systems developed to perform selected tasks before such systems are deployed and to define (and eventually create) data sets and evaluation infrastructure that will allow system builders to detect the extent to which their system exhibits those harms. Experience suggests that such infrastructure will support the development of better---more trustworthy---systems for those tasks.
This initial workshop in the series will be a planning workshop consisting of three separate tracks, each track focused on a different task: an automated employment decision track, a track focused on automatic language translation within the context of international border crossings, and a track focused on the management of consumer health (mis)information. Within each track attendees will participate in a roundtable discussion to identify the most salient potential harm(s) an automated system for the task is likely to engender, and to define an evaluation protocol, including necessary data resources, that measures the extent to which a given system exhibits harmful behavior. Workshops in subsequent years will implement the protocols providing community evaluations for the target tasks. (TRUC is being patterned after the Text REtrieval Conference (TREC) series that have successfully supported research on information retrieval through community evaluations.)
The meeting will run from 9:00am to 5:00pm EDT with a lunch break from 12:00-1:45pm each day.