Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Search Publications by:

Displaying 51 - 75 of 85

2012 Proceedings of the Performance Metrics for Intelligent Systems (PerMI'12) Workshop

November 21, 2012

Author(s)

Rajmohan (. Madhavan, Elena R. Messina, Brian Weiss

The 2012 Performance Metrics for Intelligent Systems workshop is the eleventh in a series dedicated to defining measures and methodologies of evaluating performance of intelligent systems. Started in 2000, the PerMIS series focuses on applications of

Multi-Relationship Evaluation Design: Modeling an Automatic Test Plan Generator

April 11, 2012

Author(s)

Brian A. Weiss, Linda C. Schmidt

Advanced and intelligent systems within the manufacturing, military, homeland security, and automotive fields are constantly emerging and progressing. Testing these technologies is crucial to (1) inform the technology developers of targeted areas for

Multi-Relationship Evaluation Design: Formalizing Evaluation-Design Input and Output Blueprint Elements for Testing Developing Intelligent Systems

January 20, 2012

Author(s)

Brian A. Weiss, Linda C. Schmidt

Intelligent technologies within the military, law enforcement, and homeland security fields are continuously evolving. Testing these technologies is crucial to (1) inform the technology developers of specific aspects for enhancement, (2) request end-user

A Retrospective Analysis of Lessons Learned in Evaluating Advanced Military Technologies

December 30, 2011

Author(s)

Craig I. Schlenoff, Brian A. Weiss, Michelle P. Steves

For the past six years, personnel from the National Institute of Standards and Technology (NIST) have served as the Independent Evaluation Team (IET) for two major DARPA programs. DARPA ASSIST (Advanced Soldier Sensor Information System and Technology) is

Multi-Relationship Evaluation Design: Formalizing Evaluation Design Input and Output Blueprint Elements for Testing Developing Intelligent Systems

October 5, 2011

Author(s)

Brian A. Weiss, Linda C. Schmidt

Multi-Relationship Evaluation Design: Formalizing Test Plan Input and Output Elements for Evaluating Developing Intelligent Systems

August 31, 2011

Author(s)

Brian A. Weiss, Linda C. Schmidt

Advanced and intelligent systems within the manufacturing, military, homeland security, and automotive fields are constantly under development or improvement. Testing the performance of these technologies is critical to (1) update the system designers of

The Multi-Relationship Evaluation Design Framework: Producing Evaluation Blueprints to Test Emerging, Advanced, and Intelligent Technologies

June 30, 2011

Author(s)

Brian A. Weiss, Linda C. Schmidt

This article introduces the Multi-Relationship Evaluation Design (MRED) framework whose objective is to take uncertain input data and automatically output comprehensive evaluation blueprints complete with targeted evaluation elements. MRED is unique in

A Detailed Discussion of Lessons Learned in Evaluating Emerging and Advanced Military Technologies

June 3, 2011

Author(s)

Craig I. Schlenoff, Brian A. Weiss, Michelle P. Steves

Performance Assessments of Two-Way, Free-Form, Speech-to-Speech Translation Systems for Tactical Use

June 3, 2011

Author(s)

Brian A. Weiss, Craig I. Schlenoff

A critical challenge for military personnel when operating in foreign countries is effective communication with the local population. To address this issue, the Defense Advanced Research Projects Agency (DARPA) created the Spoken Language Communication and

Development of Domain-Specific Scenarios for Training and Evaluation of Two-Way, Free Form, Spoken Language Translation Devices

June 1, 2011

Author(s)

Brian A. Weiss, Marnie Menzel

To create effective and accurate two-way, free form, spoken language translation devices, the technologies must have appropriate training data. The goal of the Defense Advanced Research Projects Agency Spoken Language Communication and Translation System

Providing NIST Supervisors with a Continuous Learning Program

April 14, 2011

Author(s)

David W. Griffith, Susan F. Heller-Zeisler, Joy A. Herman, Andrew Jackson, Janine Kerns, Donna J. Sirk, William E. Wallace, Brian Weiss

2010 Proceedings of the Performance Metrics for Intelligent Systems (PerMIS'10) Workshop

February 8, 2011

Author(s)

Rajmohan (. Madhavan, Elena R. Messina, Brian A. Weiss

The 2010 Performance Metrics for Intelligent Systems workshop is the tenth in a series dedicated to defining measures and methodologies of evaluating performance of intelligent systems. Started in 2000, the PerMIS series focuses on applications of

Co-X Panel Discussion

February 4, 2011

Author(s)

Elena R. Messina, Brian A. Weiss, Rajmohan Madhavan

Robots as co-protectors, co-inhabitants, co-workers, co-integrants, or more generically, "Co-X" will improve effectiveness, efficiency, safety, security, and improve quality of life in the coming years. In-keeping with the theme of PerMIS'10, which

The Multi-Relationship Evaluation Design Framework: Creating Evaluation Blueprints to Assess Advanced and Intelligent Technologies

October 29, 2010

Author(s)

Brian A. Weiss, Linda C. Schmidt

Technological evolutions are constantly occurring across advanced and intelligent systems within a range of fields including those within the military, law enforcement, automobile, and manufacturing industries. Testing the performance of these technologies

Performance Assessments of Two-Way, Free-Form, Speech-to-Speech Translation Systems for Tactical Use

October 20, 2010

Author(s)

Brian A. Weiss, Craig I. Schlenoff

Lessons Learned in Evaluating DARPA Advanced Military Technologies

October 15, 2010

Author(s)

Craig I. Schlenoff, Brian A. Weiss, Michelle P. Steves

The Multi-Relationship Evaluation Design (MRED) Framework: Producing Evaluation Blueprints to Test Emerging, Advanced, and Intelligent Systems

September 30, 2010

Author(s)

Brian A. Weiss, Linda C. Schmidt

This paper introduces the Multi-Relationship Evaluation Design (MRED) framework whose objective is to take uncertain input data and automatically output comprehensive evaluation blueprints complete with targeted evaluation elements. MRED is unique in that

APPROACHES AND BEST PRACTICES: Data Collection of Audio Dialogues to Support the Training of Speech-to-Speech Translation Systems

June 25, 2010

Author(s)

Brian A. Weiss, Craig I. Schlenoff, Ann M. Virts

The purpose of this document is to describe the best practices that personnel from the National Institute of Standards and Technology (NIST) have developed and implemented to efficiently and effectively capture two-way, free-form speech-to-speech audio

THE MULTI-RELATIONSHIP EVALUATION DESIGN FRAMEWORK: DESIGNING TESTING PLANS TO COMPREHENSIVELY ASSESS ADVANCED AND INTELLIGENT TECHNOLOGIES

May 5, 2010

Author(s)

Brian A. Weiss, Linda C. Schmidt, Harry A. Scott, Craig I. Schlenoff

As new technologies are developed and mature, it becomes extremely important to provide both formative and summative assessments on their performance. Performance assessment events range in form from a few simple tests of key elements of the technology to

Development of Domain-Specific Scenarios for Training and Evaluation of Two-Way, Free Form, Spoken Language Translation Devices

October 1, 2009

Author(s)

Brian A. Weiss, Marnie Menzel

To create effective and accurate two-way, free form, spoken language translation devices, they must be supplied with appropriate training data. The goal of the DARPA TRANSTAC (Spoken Language Communication and Translation System for Tactical Use) program

Evaluating Speech Translation Systems: Applying SCORE to TRANSTAC Technologies

September 29, 2009

Author(s)

Craig I. Schlenoff, Brian A. Weiss, Michelle P. Steves, Gregory A. Sanders, Frederick M. Proctor, Ann M. Virts

The Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program is a Defense Advanced Research Projects Agency (DARPA) advanced technology research and development program. The goal of the TRANSTAC program is to demonstrate

The Impact of Scenario Development on the Performance of Speech Translation Systems Prescribed by the SCORE Framework

September 25, 2009

Author(s)

Brian A. Weiss, Craig I. Schlenoff

The Defense Advanced Research Projects Agency's (DARPA) Spoken Language Communication and Translation for Tactical Use (TRANSTAC) program is a focused advanced technology research and development program. The intent of the TRANSTAC program is to

Evolution of the SCORE Framework to Enhance Field-Based Performance Evaluations of Emerging Technologies

August 21, 2008

Author(s)

Brian A. Weiss, Craig I. Schlenoff

Designing and implementing a performance evaluation of an emerging technology to present a broad picture of how this technology would perform in its typical operating environment is a very challenging goal. Personnel from the National Institute of

Performance Evaluation of Speech Translation Systems

May 28, 2008

Author(s)

Brian A. Weiss, Craig I. Schlenoff, Gregory A. Sanders, Michelle P. Steves, Sherri Condon, Jon Phillips, Dan Parvaz

One of the most challenging tasks for uniformed service personnel serving in foreign countries is effective verbal communication with the local population. To remedy this problem, several companies and academic institutions have been funded to develop

Fused quad audio/visual and tracking data collection to enhance mobile robot and operator performance analyses

March 20, 2008

Author(s)

Brian A. Weiss, Brian Antonishek, Richard J. Norcross

Personnel at the National Institute of Standards and Technology (NIST), tasked with developing performance measures and standards for urban search and rescue and explosive ordnance disposal robots, have been designing advanced ground truth data collection