Rules of the ARIAC Prize Competition
The official version of the Rules will be posted here when they become available.
In the meantime, here are the rules that have been used in previous years to give an idea of what they will look like.
Subject of the Competition:
In June 2017, the National Institute of Standard and Technology (NIST) put on the first Agile Robotics for Industrial Applications Competition (ARIAC). The goal of the competition was to test the agility of industrial robot systems, with the goal of enabling industrial robots on the shop floors to be more productive, more autonomous, and to require less time from shop floor workers. For the second year of the competition, we are introducing a cash prize to motivate and expand participation.
In this context, agility is defined broadly to address:
The competition Participants are required to develop a robot control system (software) to control a robot in a simulated environment. Gazebo (gazebosim.com), which is an open source robotics simulation environment, will be used as the testing platform and the Robot Operating Systems (ROS), which is an open source set of software libraries and tools, will be used to define the interfaces to the simulation system.
Software interfaces will be made publicly available that will allow the Participants to control the robot. There will be a qualifier in the April 2019 timeframe with the final competition occurring in mid-May 2019. The competition will be performed virtually in the cloud, so physical attendance is not required to compete. NIST is proposing a workshop to a major robotics conference to allow the winning Participants to present their approaches; details, when available, will be posted to the official challenge website.
Dates of the Competition:
Up to three winners will be selected. The Prize Purse for the ARIAC Prize Competition is a total of $17,500. In addition, each winner may be able to select any one representative from their team to present their results at a robotics conference in 2019 in a workshop dedicated to the ARIAC competition. NIST may be able to provide travel funds subject to availability. The number of winners will be fewer than three if there are fewer than three Participants that qualify for the competition. The Prize Purse may increase, but will not decrease. Any increases in the Prize Purse will be posted on the Event Website. NIST reserves the right to announce additional winners of non-cash prizes.
Rules for being eligible to participate in the Competition:
Participation in the ARIAC Prize Competition is open to ALL; however, not all participants are eligible to win cash prizes as explained in the next section.
Each Competition Participant (individual, team, or legal entity) is required to register on the ARIAC website. There shall be one Official Representative for each Competition Participant. The Official Representative must provide a name, username (which may serve as a team or affiliation name), email address, and affirm that he/she has read and consents to be governed by the Competition Rules. Multiple registrations per Participant are NOT allowed. At NIST's discretion, any violation of this rule will be grounds for disqualification from the Competition. Multiple individuals and/or legal entities may collaborate as a team to submit a single entry
(Participant’s code submission to a qualifier or final), in which case the designated Official Representative will be responsible for meeting all entry and evaluation requirements. Participation is subject to all U.S. federal, state and local laws and regulations. Participants must not be suspended, debarred, or otherwise excluded from doing business with the Federal Government. Individuals entering on behalf of or representing a company, institution or other legal entity are responsible for confirming that their entry does not violate any policies of that company, institution or legal entity. Any other individuals or legal entities involved with the design, production, execution, distribution or evaluation of ARIAC are not eligible to participate.
Once registered, Participants will have access to the interfaces needed to participate in the competition through the ARIAC website. NIST will create an account to allow the Participants to upload their submissions to the qualifiers. Modifications and improvements to a Participant's control system during the qualifying period are to be expected, but the Participant should have one final control system ready for the final competition.
Rules for being eligible to win a Cash Prize:
To be eligible for a cash prize:
In addition, interested Participants who do not meet the eligibility requirements to win a prize (i.e., individuals who are neither a US citizen nor a permanent resident of the United States or non-US-based entities) are encouraged to participate in the Competition. They are invited to register on the ARIAC website and download the training material. The performance obtained by these Participants will be displayed on the ARIAC website in the same manner as the performance obtained by Participants who are eligible to win cash prizes.
How to Participate in the Competition:
An interested Participant (individual, team, or legal entity) must initiate the process of participating in the Competition by registering at the ARIAC website. The party is then given access to view the documentation and download the tutorials (which provides instructions on how to run the simulations) from the ARIAC website. In the event NIST determines modifications to the documentation and tutorials are needed, all website registrants will be notified by email.
Once registered, the Participants are eligible to participate in the qualifiers. There will be two qualifiers, both occurring in the March/April 2018 timeframe (specific deadlines will be published at the official challenge website and all Participants notified by email). Participants only have to complete one of the two qualifiers to be eligible to compete in the final competition. A minimum score will be determined, based on the metrics described below, which Participants have to meet or exceed in the qualifiers to be eligible for the final competition.
During ARIAC Qualifiers/Finals, each team will submit their system so that it can be run in the automated evaluation setup. This is to ensure that teams are not using prohibited interfaces to interact with the simulation. They will have two work weeks from the date that each qualifier is released on the website (available to be downloaded) to submit their software control system that addresses the challenges in the qualifier. Registered participants will be emailed when the qualifier is released. The following files should be uploaded to an account (nfiles.nist.gov) that NIST will set up specifically for each Participant.
To allow the ARIAC Competition controllers to automatically run each team's system, teams are required to submit the following files:
The build_team_system.bash script will be run only once to setup the team's system, and the run_team_system.bash script will be run one time for each of the scenarios to be tested against.
The following terminology is frequently used in this document
ARIAC requires Participants to complete a series of tests centered in an industrial scenario that is based around order fulfillment. The robot system will work within the environment specified in the Work Environment section.
Challenges will be introduced in each scenario. Scenarios may include, but are not limited to:
The competition will consist of 15 trials: Each trial will receive a score based on completion and efficiency metrics outlined in the Scoring section.
The simulation environment is a representation of an industrial order fulfillment work cell with a robot arm, product bins, and boxes.
There are multiple product bins that may be used for order fulfillment. Products in these bins will not be replaced once used.
There is a robot arm mounted on a linear actuator. The linear actuator measures 4 m.
An automated guided vehicle (AGV) and/or drone are located at either end of the linear actuator. Orders are fulfilled at the AGV/drone. A Participant will programmatically signal the AGV/drone when the fulfilled orders are ready to be taken away. The signaled AGV/drone will depart with the shipment box.
Boxes measuring 0.5 x 0.7 m. are used for order fulfillment.
The robot arm's position is controlled through the linear actuator on which it is mounted.
The end of the arm is equipped with a vacuum gripper. The vacuum gripper is controlled in a binary manner and reports whether or not it is successfully gripping an object.
A Participant can choose which sensors to use around the environment. Each sensor has a cost that factors into the final score.
A sample of available sensors may include:
An order is an instruction containing products for the robot system to place in the box.
Each order will specify the products to be gathered.
Above the belt at a key location is a quality control sensor that detects faulty products. If faulty products are detected while Participants are filling boxes, those products should be removed from the box and replaced with another product of the same type. Faulty products are considered unwanted products: they will not count for any points when the order is fulfilled, and they will cost Participants the all-products bonus if left in the boxes (see Scoring, below).
Each trial will consist of the following steps:
The NIST Director will appoint a panel of three qualified judges. The Judges will be robot agility and/or simulation experts from inside and/or outside of NIST. The Judges will determine winners according to the Judging Criteria described herein. The Judges may not have personal or financial interests in, or be an employee, officer, director, or agent of, any entity that is a registered Participant in the Competition and may not have a familial or financial relationship with an individual who is a registered Participant. In the event of such a conflict, a Judge must recuse himself or herself and a new Judge may be appointed.
Basis on which Finalists and Winners will be selected
A portion of the scores will be automatically calculated, shared with the Participants, and publicly posted for each Trial at the conclusion of the Finals (after all Trials are run) as a combination of cost and performance metrics, described below in the Trial Score Calculation section. A Participant's final automated score is the sum of scores for each of the 15 competition Trials.
The following values are calculated for a Participant’s system setup. As Participants are to use the same system setup for all Trials, the values will remain unchanged between Trials.
The choices made by the competitors (choices of sensors) will have a specific cost associated with them. The final cost for individual sensors will be made available to the Participants at least two weeks before the Finals. For planning purposes, Participants should use:
The sum of these cost choices made by each team will be divided by a baseline cost of $1700 and made into a cost factor for the scoring. The team costs are designated as TC, and the baseline cost is designated as BC. The cost factor, is then calculated as:
Performance metrics that cover both completion and efficiency are calculated for each Trial separately. Completion captures the quality of the orders fulfilled (that the boxes contain the correct products in the correct position/orientations); efficiency captures the responsiveness in fulfilling orders (that the orders were filled quickly).
Each trial, since it has the option to have more than one order, will have a certain set of scores for each order. Each order consists of a list of products needed and a prescribed position and orientation for each product within the box. The completion score, CS, is a combined score for each order, starting with 1 point for each product that is placed in the box. For each product that is within 3 cm of the correct position, and within 0.1 radians of the correct orientation will result in an additional point. Next, since the main focus of the competition is to have fulfilled orders, any order with all of the products in the confines of the box and in the correct position/orientation receives an additional point for each product. So, for an order s_j with i products in it would have a maximum score of:
The Efficiency of the approaches chosen by the teams are accounted for in the scoring by using the times that each team takes to complete each trial. The times are counted in seconds from the start of the trial until the first order is completed and is designated as T_01. In the trials with multiple orders, the time, T_j is counted in seconds from the time that the jth order is sent to the team. For each trial, an average of the times for each team (ATi) is compared to the individual team time to calculate the Efficiency Factor:
If a team’s system times out for a trial, the efficiency factor for the trial is set to 0 and the trial time is not used to calculate the average times for that trial.
For the multi-order trials where one of the orders is a high priority order, an additional multiplier, h, is applied with the efficiency factor to give a higher proportion of the score to that order, which is initially set to a value of 3. The final h value will be made available to the Participants at least two weeks before the finals.
Trial Score Calculation
All of the scoring factors described above are combined as shown below to calculate the score for each trial:
Final Score Calculation
The final overall score is the equally weighted sum of all trial scores. As mentioned in the Competition Scenarios section, trials are expected to focus on the Baseline Order Fulfillment, Dropped Part, and In-Process Order Change scenarios, among others.
Winners: The Winners of cash prizes will be determined by three Judges appointed by the NIST Director, using the Judging Criteria outlined herein.
NIST will announce the Finalists via the Event Website as well as those Finalists that have been awarded a cash award (each, an "Award"). The anticipated number and amount of the Awards that will be awarded for this Competition is set forth in these rules; however, NIST is not obligated to make all or any Awards, and reserves the right to award fewer than the anticipated number of Awards in the event an insufficient number of eligible Participants meet any one or more of the Judging Criteria for this Competition, based on NIST's sole discretion. Using the Judging Criteria described herein, three qualified Judges (appointed by the NIST Director) will determine the winners of the ARIAC Prize Competition.
The winner verification process for being eligible to receive an Award includes providing the full legal name, tax identification number or social security number, routing number and banking account to which the prize money can be deposited directly. Return of any notification as
"undeliverable" will result in disqualification. The verification form must be returned within seven (7) calendar days of receipt. After verification of eligibility, Awards will be distributed in the form of electronic funds transfer addressed to the Official Representatives specified in the winning Entries. That Official Representative will have sole responsibility for further distribution of any Award among Participants in a group Entry or within a company or institution that has submitted an Entry through that representative. The list of Entries receiving Awards for the Competition, including the names of all members of a team, will be made public according to the timeline outlined on the Event Website.
Winners are responsible for all taxes and reporting related to any Award received as part of the Competition.
All costs incurred in the preparation of Competition Entries are to be borne by Participants.
Presentation of Winning Approaches at a To Be Determined Workshop in a Robotics Conference:
Up to three Winners of the ARIAC Competition may be invited by NIST to attend a workshop dedicated to the ARIAC Competition at a future TBD robotics conference. The Winners will be required to make a 25-minute presentation on their approach in the ARIAC Competition followed by audience question and answer.
NIST may arrange and pay for the travel for one team-selected representative from each of the winning Participants to cover eligible travel costs such as airfare, conference registration fee, and lodging.
Point of Contact for the Competition:
Questions about the ARIAC Prize Competition can be directed to Craig Schlenoff, craig.schlenoff [at] nist.gov.
-- TERMS AND CONDITIONS --
Intellectual Property Rights
Other than as set forth herein, NIST does not make any claim to ownership of your Entry or any of your intellectual property or third party intellectual property that it may contain. By participating in the ARIAC Prize Competition, you are not granting any rights in any patents or pending patent applications related to your Entry; provided that by submitting an entry (i.e., participating in the Competition), you are granting NIST certain limited rights as set forth herein.
However, following the competition, Participants in the ARIAC competition will be invited, but not required, to upload their code to a NIST central repository which will be made available on an unrestricted, irrevocable, royalty-free basis to the community as a whole, with the goal of furthering the state of the art in robot agility. The decision of whether or not to upload their code will not impact the scoring.
By submitting an Entry, you grant to NIST the right to review and score your Entry as described in the section “Basis on which finalists and winners will be selected,” to describe your Entry in connection with any materials created in connection with the Competition including making public your name, scores, and the date/time that score was obtained at the Event Website, and to have the Judges, Competition administrators, and the designees of any of them, review your Entry.
By submitting an Entry in the ARIAC Prize Competition, you grant a non-exclusive, irrevocable, paid up right and license to NIST to use your name, likeness, biographical information, image, any other personal data submitted with your Entry and the contents in your Entry, in connection with the ARIAC Prize Competition for any purpose, including promotion and advertisement of the competition and future challenges.
You agree that nothing in these Rules grants you a right or license to use any names or logos of NIST or the Department of Commerce, or any other intellectual property or proprietary rights of NIST or the Department of Commerce or their employees or contractors. You grant to NIST the right to include your name and your company or institution name and logo (if your Entry is from a company or institution) as a Participant on the Event Website and in materials from NIST announcing Winners, Finalists, or Participants in the Competition. Other than these uses or as otherwise set forth herein, you are not granting NIST any rights to your trademarks.
Entries containing any matter which, in the sole discretion of NIST, is indecent, defamatory, in obvious bad taste, which demonstrates a lack of respect for public morals or conduct, which promotes discrimination in any form, which shows unlawful acts being performed, which is slanderous or libelous, or which adversely affects the reputation of NIST, will not be accepted. If NIST, in its sole discretion, finds any Entry to be unacceptable, then such Entry shall be deemed disqualified and will not be evaluated or considered for award. NIST shall have the right to remove any content from the Event Website in its sole discretion at any time and for any reason, including, but not limited to, any online comment or posting related to the Competition.
To advance the goals of NIST's robot agility research and other relevant technical programs, following the competition, NIST may negotiate a license for the use of intellectual property developed by a registered Participant in the ARIAC Prize Competition.
By making a submission to the ARIAC Prize Competition, you agree that no part of your submission includes any confidential or proprietary information, ideas or products, including but not limited to information, ideas or products within the scope of the Trade Secrets Act, 18 USC §1905. Because NIST will not receive or hold any submitted materials "in confidence," it is agreed that, with respect to your Entry, no confidential or fiduciary relationship or obligation of secrecy is established between NIST and you, your Entry team, the company or institution you represent when submitting an Entry, or any other person or entity associated with any part of your Entry.
By participating in the ARIAC Prize Competition, you represent and warrant that all information you submit is true and complete to the best of your knowledge, that you have the right and authority to submit the data on your own behalf or on behalf of the persons and entities that you specify, and that your data uploaded to the ARIAC Website:
(a) is your own original work, or is submitted by permission with full and proper credit given within your Entry;
(b) does not contain confidential information or trade secrets (yours or anyone else's);
(c) does not knowingly violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights in technical data, rights of privacy, publicity or other intellectual property or other rights of any person or entity;
(d) does not contain malicious code, such as viruses, malware, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information;
(e) does not and will not violate any applicable law, statute, ordinance, rule or regulation, including, without limitation, United States export laws and regulations, including, but not limited to, the International Traffic in Arms Regulations and the Department of Commerce Export Regulations; and
(f) does not trigger any reporting or royalty or other obligation to any third party.
Limitation of Liability
By participating in the ARIAC Prize Competition, you agree to assume any and all risks and to release, indemnify and hold harmless NIST from and against any injuries, losses, damages, claims, actions and any liability of any kind (including attorneys' fees) resulting from or arising out of your participation in, association with or submission to the ARIAC Prize Competition (including any claims alleging that your Entry infringes, misappropriates or violates any third party's intellectual property rights). In addition, you agree to waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from your participation in the ARIAC Prize Competition, whether the injury, death, damage, or loss arises through negligence or otherwise.
NIST is not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete or late Entries. Any compromise to the fair and proper conduct of the ARIAC Prize Competition may result in the disqualification of an Entry or Participant, termination of the ARIAC Prize Competition, or other remedial action, at the sole discretion of NIST. NIST reserves the right in its sole discretion to extend or modify the dates of the ARIAC Prize Competition, to modify the test and training data provided at the Event Website, and to change the terms set forth herein governing any phases taking place after the effective date of any such change. By entering, you agree to the terms set forth herein and to all decisions of NIST and/or all of their respective agents, which are final and binding in all respects.
NIST is not responsible for: (1) Any incorrect or inaccurate information, whether caused by a Participant, printing errors, or by any of the equipment or programming associated with or used in the ARIAC Prize Competition; (2) unauthorized human intervention in any part of the Entry Process for the ARIAC Prize Competition; (3) technical or human error that may occur in the administration of the ARIAC Prize Competition or the processing of Entries; or (4) any injury or damage to persons or property that may be caused, directly or indirectly, in whole or in part, from a Participant's participation in the ARIAC Prize Competition or receipt or use or misuse of an Award. If for any reason an Entry is confirmed to have been deleted erroneously, lost, or otherwise destroyed or corrupted, the Participant's sole remedy is to submit another Entry in the ARIAC Prize Competition.
Termination and Disqualification
NIST reserves the authority to cancel, suspend, and/or modify the ARIAC Prize Competition, or any part of it, if any fraud, technical failures, or any other factor beyond NIST's reasonable control impairs the integrity or proper functioning of the ARIAC Prize Competition, as determined by NIST in its sole discretion.
NIST reserves the right to disqualify any Participant it believes to be tampering with the Entry process or the operation of the ARIAC Prize Competition or to be acting in violation of any applicable rule or condition.
Any attempt by any person to undermine the legitimate operation of the ARIAC Prize Competition may be a violation of criminal and civil law, and, should such an attempt be made, NIST reserves the authority to seek damages from any such person to the fullest extent permitted by law.
Verification of Potential Winner(s)
All potential winners are subject to verification by NIST, whose decisions are final and binding in all matters related to the ARIAC Prize Competition.
Potential winner(s) must continue to comply with all terms and conditions of the ARIAC Prize Competition Rules described herein, and winning is contingent upon fulfilling all requirements. In the event that a potential winner, or an announced winner, is found to be ineligible or is disqualified for any reason, NIST may make an award, instead, to another Participant.
Privacy and Disclosure under FOIA
Except as provided herein, information submitted throughout the ARIAC Prize Competition will be used only to communicate with Participants regarding Entries and/or the ARIAC Prize Competition. Participant Entries and submissions to this competition may be subject to disclosure under the Freedom of Information Act ("FOIA").
15 U.S.C. 3719.