PerfLoc Competition Rules

PerfLoc Navigation

• PerfLoc: Performance Evaluation of Smartphone Indoor Localization Apps
• User Guide
• FAQs
• Data
• Archived Announcements
  • Winners of the PerfLoc Prize Competition
  • Ranking of PerfLoc Participating Teams
  • Announcement for the PerfLoc Webinar of May 18, 2017
  • PerfLoc Webinar Slides & Video
  • PerfLoc Competition Rules
• Back to Location-Based Services portfolio page

Subject of the competition

The National Institute of Standard and Technology (NIST) is announcing the PerfLoc Prize Competition. This competition is about developing the best possible indoor localization and tracking app for Android smartphones. The competition participants are required to use the extensive Android smartphone data repository developed by NIST to develop their apps and have their apps' performance evaluated through the PerfLoc web portal. There shall be a live demonstration of the finalist apps at the NIST campus in Gaithersburg, Maryland, USA, and a subsequent demonstration of the winning apps at the 2018 Microsoft Indoor Localization Competition to be held at the International Conference on Information Processing in Sensor Networks (IPSN) in Porto, Portugal.

Dates of the competition

• Competition Testing Period Begins: March 22, 2017
• Competition Testing Period Ends: 5 PM, EST (UTC-05:00), January 17, 2018
• Announcement of Finalists for Cash Prize: January 19, 2018
• Finalist App Demo Days at NIST: February 15-16, 2018
• Announcement of Cash Prize Winners: February 16, 2018
• Showcase Winning Apps at 2018 Microsoft Indoor Localization Competition to be held at the International Conference on Information Processing in Sensor Networks (IPSN) in Porto, Portugal, April 11-13, 2018

Prize amounts

• 1^st place - $20,000
• 2^nd place - $10,000
• 3^rd place - $5,000

The Prize Purse for the PerfLoc Prize Competition is a total of $35,000. In addition, up to five finalists, the designated official representative in the case of a team or a legal entity, will be provided invitational travel funds to travel to the NIST campus in Gaithersburg, MD for live tests of their apps. Up to three finalists will be selected as winners at the App Demo Days at NIST. The number of winners will be less than three if there are less than three finalists invited to the PerfLoc App Demo Days due to the level of participation in the PerfLoc Prize Competition or if fewer than three finalist apps generate location estimates within five seconds of a request to generate one all the time. NIST is looking for indoor localization and tracking Android apps that work in real-time. Up to three winners will be provided invitational travel funds by NIST to showcase their apps at the 2018 Microsoft Indoor Localization Competition to be held at 2018 IPSN in Porto, Portugal. The Prize Purse may increase, but will not decrease. Any increases in the Prize Purse will be posted on the Event Website. NIST reserves the right to announce additional winners of non-cash prizes.

Rules for being eligible to participate in the Competition

Participation in the PerfLoc Prize Competition is open to ALL; however, not all participants are eligible to win cash prizes as explained in the next section.

Each Competition Participant (individual, team, or legal entity) is required to register on the PerfLoc website and create ONE password-protected user account. There shall be one Official Representative for each Competition Participant. To create a user account, the Official Representative must provide a name, username, password, email address, and geographical location, and affirm that he/she has read and consents to be governed by the Competition Rules. Multiple accounts per Participant are NOT allowed. At NIST's discretion, any violation of this rule will be grounds for disqualification from the Competition. Multiple individuals and/or legal entities may collaborate as a team to submit a single entry, in which case the designated Official Representative will be responsible for meeting all entry and evaluation requirements. Participation is subject to all U.S. federal, state and local laws and regulations. Individuals entering on behalf of or representing a company, institution or other legal entity are responsible for confirming that their entry does not violate any policies of that company, institution or legal entity.

During the competition testing period, Participants are allowed a total of 150 uses of the PerfLoc Performance Evaluation Portal and a maximum of three uses per day. At NIST's discretion, any violation of this requirement will be grounds for disqualification from the Competition. Modifications and improvements to a Participant's App during the competition testing period are to be expected, but the Participant should have one final App ready at the end of the competition testing period.

Rules for being eligible to win a cash prize

To be eligible for a cash prize:

• A Participant (whether an individual, team, or legal entity) must have registered to participate and complied with all of the requirements under section 3719 of title 15, United States Code as contained herein.
• At the time of Entry, the Official Representative (individual or team lead, in the case of a group project) must be age 18 or older and a U.S. citizen or permanent resident of the United States or its territories.
• In the case of a private entity, the business shall be incorporated in and maintain a primary place of business in the United States or its territories.
• Participants may not be a Federal entity or Federal employee acting within the scope of their employment. NIST employees and NIST Research Associates as well as direct recipients of NIST funding awards for NIST projects in the development of an Android app in the area of the challenge are not eligible to participate. Non-NIST Federal employees acting in their personal capacities should consult with their respective agency ethics officials to determine whether their participation in this Competition is permissible.
• Any other individuals or legal entities involved with the design, production, execution, distribution or evaluation of the PerfLoc Prize Competition are also not eligible to participate.
• A Participant shall not be deemed ineligible because the Participant consulted with Federal employees or used Federal facilities in preparing its submission to the PerfLoc Prize Competition if the Federal employees and facilities are made available to all Participants on an equitable basis.

In addition, interested Participants who do not meet the eligibility requirements to win a prize (i.e., individuals who are neither a US citizen nor a permanent resident of the United States or non-US-based entities) are encouraged to participate in the Competition. They are invited to register on the PerfLoc website, download the data sets, and use the PerfLoc Performance Evaluation Portal according to the PerfLoc Prize Competition Rules. The performance obtained by these Participants will be displayed on the PerfLoc website in the same manner as the performance obtained by Participants who are eligible to win cash prizes.

How to participate in the Competition

An interested party (individual, team, or legal entity) must initiate the process of participating in the Competition by registering at the PerfLoc website and creating a user account. The party is then allowed to download a number of smartphone data sets collected by NIST from the PerfLoc website. In the event NIST determines modifications to the data sets are needed, all website registrants will be notified by email.

The party becomes a Participant once he/she has built a smartphone indoor localization and tracking Android App (i.e., working software application that operates on a Wi-Fi and GPS enabled Android smartphone with possibly cellular service) AND has had the App evaluated at least once through the PerfLoc Performance Evaluation Portal. An entry consists of the file(s) uploaded to the PerfLoc Performance Evaluation Portal and, in the case of the Finalists, the App demonstrated at NIST (hereinafter referred to as an "Entry").

Types of PerfLoc Data

The PerfLoc website contains two types of data sets: training data and test data, collected by NIST researchers in four buildings using four different Android smartphones. Details for these two types of data sets are provided below:

1. Training data: These are similar to solved problems in a textbook. There are a total of four training datasets, one for each of the buildings in which NIST collected data. For each training data set, NIST is providing the ground-truth locations of the test subject that collected the data at a number of time instances during the course of data collection. Participants can compare the location estimates generated by their Apps at the abovementioned time instances with the ground-truth locations to decide how well their Apps are doing. Based on such comparisons, the Participants can modify their Apps and hopefully develop better versions. Participants should not upload location estimates for training data sets, because the ground truth locations for designated time instances in those data sets have already been provided to them.
2. Test data: There are multiple test data sets, corresponding to various Test & Evaluation Scenarios, for each of the four buildings in which NIST collected data. There are also a number of time instances associated with each test data set that are available for download from the PerfLoc website. Participants are asked to upload location estimates generated by their Apps for ALL the designated time instances for each test data set. The Participants must provide such location estimates for ALL test data sets.

App Requirements

Participants should create an original indoor localization and tracking Android-based App. Note that Finalists invited to the PerfLoc App Demo Days will be required to run their Apps on a NIST-provided Google Pixel XL smartphone running on Android 7.1.2 Nougat OS. The Apps must be able to compute location estimates in real-time based on the data collected by the smartphone in the course of the tests to be administered at that event. Each time an App is requested to generate a location estimate, it should produce the estimate within five seconds.

PerfLoc Performance Evaluation Portal Scoring

Each App is evaluated over the web instantly when the Participant uploads on the PerfLoc Performance Evaluation Portal the location estimates generated by the App at the designated time instances. The Participant has to upload such location estimates for all test data sets, but not for training data sets. Each upload of location estimates for test data sets constitutes one use of the PerfLoc Performance Evaluation Portal.

The PerfLoc Performance Evaluation Portal will instantly return five values of a single performance metric, namely spherical error 95% (SE95), whose definition is included in the Appendix to this document. One SE95 value is returned for each of the four buildings and an overall SE95 value is returned for all time instances and all test data sets. This is another way for the Participant to know how well his/her App is performing and what steps to take, if any, to make it better.

The following formula will be used for assigning up to 80 points based on the overall SE95 performance (in meters):

This function is plotted below and some of its sample values are given in the table to the right.

All Participant data uploads must be completed by 5:00 PM EDT (UTC-05:00) on January 17, 2018. Uploads to the PerfLoc Performance Evaluation Portal before or after the competition testing period will not be evaluated or considered for awards. Entries sent to NIST in any manner other than through the Event Website will not be evaluated or considered for award.

The best performance of an App (i.e., the best score of the data generated by the App and uploaded to the portal) will be listed in the publicly available rank-ordered table of performances, along with the Participant's username, name, geographic location and the date and time of the data upload. If a Participant modifies his/her App, but the modification results in a lower score than the highest score obtained by the Participant previously, this will count as one use of the PerfLoc Performance Evaluation Portal, but the lower score will not replace the highest one obtained by the Participant previously.

At the end of the competition testing period, in addition to SE95, many other performance metrics specified in the international standard ISO/IEC 18305 will be computed for the best version of the App developed by each Participant and will be published on the PerfLoc Website.

Judging

The NIST Director will appoint a highly qualified judge. The judge will be a NIST employee with expertise in public safety and advanced communications. The Judge will determine winners according to the Judging Criteria described below. The Judge may not have personal or financial interests in, or be an employee, officer, director, or agent of, any entity that is a registered Participant in the Competition and may not have a familial or financial relationship with an individual who is a registered Participant. In the event of such a conflict, the Judge must recuse himself or herself and a new Judge may be appointed.

Basis on which finalists and winners will be selected

Finalists: The top 3-5 Participants, as determined by their standing on the PerfLoc Performance Evaluation Portal at the end of the competition testing period, who are also eligible to receive a cash prize and have complied with the Rules of the Competition, will be selected as the Finalists. A minimum of three Finalists will be selected assuming that there are at least three Participants who meet the eligibility and compliance requirements stated above. NIST may select up to two more Participants satisfying the abovementioned requirements as Finalists if the overall SE95 performance of their Apps are within 20% of the overall SE95 performance of the top Finalist. This is an opportunity for the fourth and fifth Finalists to break into the top three slots and declared as winners of cash prizes based on the points awarded at the PerfLoc App Demo Days at NIST. Therefore, there is an incentive for the Participants to develop not just an algorithm that yields the best overall SE95 performance, but also an App that runs in real-time and, each time queried, generates a location estimate as fast as possible (e.g.,in less than five seconds).

Upon receipt of an email from NIST informing you of your selection as a Finalist, you must respond within 48 hours to confirm your eligibility and willingness to attend the PerfLoc App Demo Days at NIST in person. If any of the selected Finalists is unable to attend the PerfLoc App Demo Days, NIST will make an attempt to select other Participants as Finalists, time permitting and if warranted. In the unlikely event of tie scores on the Leaderboard, Participants will be rank ordered by the time stamp of their best overall SE95 performance, with the earliest time ranking highest. These Participants (individuals, teams, or legal entities) constitute the Finalists and they (the Official Representative in the case of a team or legal entity) will be invited to attend the PerfLoc App Demo Days on the NIST campus in Gaithersburg, MD. NIST will arrange and pay for the Official Representatives for the Finalists to travel to NIST to take part in the PerfLoc App Demo Days.

At the PerfLoc App Demo Days Event, the finalists will install their Apps on a NIST-provided Google Pixel XL smartphone running on Android 7.1.2 Nougat OS and undergo testing in a NIST building. Each Finalist will also make a 10-minute presentation followed by audience question and answer. NIST reserves the right to run security scans on the installed App to ensure there is no risk to NIST IT systems during the Perfloc App Demo Days Event. At the event's conclusion, NIST will remove the Participants' app from the NIST smartphone.

Winners: The Winners of cash prizes will be determined by a Judge appointed by the NIST Director, using the Judging Criteria outlined here.

Judging Criteria:

1. Overall Performance on the PerfLoc Performance Evaluation Portal (up to 80 points)
   1. Using the final ranking at the close of the competition testing period, a score of up to 80 points will be calculated using the Scoring Methodology described above.
2. Performance during App Demo Days at NIST (20 points)
   1. Localization accuracy and latency of the App

If a Finalist at the PerfLoc App Demo Days does not have a functioning App, he/she will be disqualified from winning a prize. Specifically, an App is required to generate a location estimate, each time requested, in real-time (e.g., less than five seconds).

NIST will announce the Finalists via the Event Website as well as those Finalists that have been awarded a cash award (each, an "Award"). The anticipated number and amount of the Awards that will be awarded for this Competition is set forth in these rules; however, NIST is not obligated to make all or any Awards, and reserves the right to award fewer than the anticipated number of Awards in the event an insufficient number of eligible Apps meet any one or more of the Judging Criteria for this Competition, based on NIST's sole discretion. Using the Judging Criteria, a highly-qualified Judge (NIST employee appointed by the NIST Director) will determine the winners of the Perfloc Prize Competition.

The winner verification process for being eligible to receive an Award includes providing the full legal name, tax identification number or social security number, routing number and banking account to which the prize money can be deposited directly. Return of any notification as "undeliverable" will result in disqualification. After verification of eligibility, Awards will be distributed in the form of electronic funds transfer addressed to the Official Representatives specified in the winning Entries. That Official Representative will have sole responsibility for further distribution of any Award among Participants in a group Entry or within a company or institution that has submitted an Entry through that representative. The list of Entries receiving Awards for the Competition, including the names of all members of a team, will be made public according to the timeline outlined on the Event Website.

Winners are responsible for all taxes and reporting related to any Award received as part of the Competition.

All costs incurred in the preparation of Competition Entries are to be borne by Participants.

Demonstration of winning Apps at 2018 Microsoft Indoor Localization Competition

Up to three Winners selected at the PerfLoc App Demo Days at NIST may be invited by NIST to attend the 2018 Microsoft Indoor Localization Competition to be held at 2018 IPSN in Porto, Portugal. The Winners will be required to go through another round of demos and evaluations at the Microsoft Indoor Localization Competition, even though these evaluations will not affect their status as Winners. The purpose of these demos and evaluations is for the conference participants, who do not have the opportunity to attend PerfLoc App Demo Days at NIST, to see how well the Winning Apps perform. The Winners will also be required to make a 10-minute presentation on their Apps followed by audience question and answer.

NIST will arrange and pay for the travel of the Official Representatives of the invited Winners to the 2018 Microsoft Indoor Localization Competition to cover eligible travel costs such as airfare, conference registration fee, and lodging.

The test and presentation event at the 2018 Microsoft Indoor Localization Competition is also an opportunity for Participants who are not eligible to win cash prizes to showcase their Apps. NIST may invite up to three such Apps to be showcased (demo and presentation) at the 2018 Microsoft Indoor Localization Competition in place of one or more Winning Entries (cash prize recipients), if their overall SE95 performance is better than one or more of the Winning Entries.

Point of Contact for the Competition

Questions about the PerfLoc Prize Competition should be sent to PerfLoc [at] nist.gov (PerfLoc[at]nist[dot]gov) . 

Terms and Conditions

Intellectual property rights

Other than as set forth herein, NIST does not make any claim to ownership of your Entry or any of your intellectual property or third party intellectual property that it may contain. By participating in the PerfLoc Prize Competition, you are not granting any rights in any patents or pending patent applications related to your Entry; provided that by submitting an entry (i.e., participating in the Competition and uploading data to the PerfLoc Performance Evaluation Portal), you are granting NIST certain limited rights as set forth herein.

By submitting an Entry, you grant to NIST the right to review and score your Entry as described in the section "PerfLoc Performance Evaluation Portal Scoring", to describe your Entry in connection with any materials created in connection with the Competition including making public your username, name, highest score, and the date/time that score was obtained at the Event Website, and to have the Judge, Competition administrators, and the designees of any of them, review your Entry.

By submitting an Entry in the PerfLoc Prize Competition, you grant a non-exclusive, irrevocable, paid up right and license to NIST to use your name, likeness, biographical information, image, any other personal data submitted with your Entry and the contents in your Entry, in connection with the Perfloc Prize Competition for any purpose, including promotion and advertisement of the competition and future challenges.

You agree that nothing in these Rules grants you a right or license to use any names or logos of NIST or the Department of Commerce, or any other intellectual property or proprietary rights of NIST or the Department of Commerce or their employees or contractors. You grant to NIST the right to include your name and your company or institution name and logo (if your Entry is from a company or institution) as a Participant on the Event Website and in materials from NIST announcing Winners, Finalists, or Participants in the Competition. Other than these uses or as otherwise set forth herein, you are not granting NIST any rights to your trademarks.

Entries containing any matter which, in the sole discretion of NIST, is indecent, defamatory, in obvious bad taste, which demonstrates a lack of respect for public morals or conduct, which promotes discrimination in any form, which shows unlawful acts being performed, which is slanderous or libelous, or which adversely affects the reputation of NIST, will not be accepted. If NIST, in its sole discretion, finds any Entry to be unacceptable, then such Entry shall be deemed disqualified and will not be evaluated or considered for award. NIST shall have the right to remove any content from the Event Website in its sole discretion at any time and for any reason, including, but not limited to, any online comment or posting related to the Competition.

To advance the goals of NIST's public safety communications research and other relevant technical programs, NIST may negotiate a license for the use of intellectual property developed by a registered participant in the PerfLoc Prize Competition.

Confidential Information

By making a submission to the PerfLoc Prize Competition, you agree that no part of your submission includes any confidential or proprietary information, ideas or products, including but not limited to information, ideas or products within the scope of the Trade Secrets Act, 18 USC §1905. Because NIST will not receive or hold any submitted materials "in confidence," it is agreed that, with respect to your Entry, no confidential or fiduciary relationship or obligation of secrecy is established between NIST and you, your Entry team, the company or institution you represent when submitting an Entry, or any other person or entity associated with any part of your Entry.

Warranties

By participating in the PerfLoc Prize Competition, you represent and warrant that all information you submit is true and complete to the best of your knowledge, that you have the right and authority to submit the data generated by your App on your own behalf or on behalf of the persons and entities that you specify, and that your data uploaded to the PerfLoc Performance Evaluation Portal (both the information submitted in the upload and the underlying technologies or concepts in the App):

(a)   is your own original work, or is submitted by permission with full and proper credit given within your Entry;

(b)   does not contain confidential information or trade secrets (yours or anyone else's);

(c)   does not knowingly violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights in technical data, rights of privacy, publicity or other intellectual property or other rights of any person or entity;

(d)   does not contain malicious code, such as viruses, malware, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information;

(e)   does not and will not violate any applicable law, statute, ordinance, rule or regulation, including, without limitation, United States export laws and regulations, including, but not limited to, the International Traffic in Arms Regulations and the Department of Commerce Export Regulations; and

(f)   does not trigger any reporting or royalty or other obligation to any third party.

Limitation of Liability

By participating in the PerfLoc Prize Competition, you agree to assume any and all risks and to release, indemnify and hold harmless NIST from and against any injuries, losses, damages, claims, actions and any liability of any kind (including attorneys' fees) resulting from or arising out of your participation in, association with or submission to the PerfLoc Prize Competition (including any claims alleging that your Entry infringes, misappropriates or violates any third party's intellectual property rights). In addition, you agree to waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from your participation in the PerfLoc Prize Competition, whether the injury, death, damage, or loss arises through negligence or otherwise.

NIST is not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete or late Entries. Any compromise to the fair and proper conduct of the PerfLoc Prize Competition may result in the disqualification of an Entry or Participant, termination of the PerfLoc Prize Competition, or other remedial action, at the sole discretion of NIST. NIST reserves the right in its sole discretion to extend or modify the dates of the PerfLoc Prize Competition, to modify the test and training data provided at the event website, and to change the terms set forth herein governing any phases taking place after the effective date of any such change. By entering, you agree to the terms set forth herein and to all decisions of NIST and/or all of their respective agents, which are final and binding in all respects.

NIST is not responsible for: (1) Any incorrect or inaccurate information, whether caused by a Participant, printing errors, or by any of the equipment or programming associated with or used in the PerfLoc Prize Competition; (2) unauthorized human intervention in any part of the Entry Process for the PerfLoc Prize Competition; (3) technical or human error that may occur in the administration of the PerfLoc Prize Competition or the processing of Entries; or (4) any injury or damage to persons or property that may be caused, directly or indirectly, in whole or in part, from a Participant's participation in the PerfLoc Prize Competition or receipt or use or misuse of an Award. If for any reason an Entry is confirmed to have been deleted erroneously, lost, or otherwise destroyed or corrupted, the Participant's sole remedy is to submit another Entry in the PerfLoc Prize Competition.

Termination and Disqualification

NIST reserves the authority to cancel, suspend, and/or modify the PerfLoc Prize Competition, or any part of it, if any fraud, technical failures, or any other factor beyond NIST's reasonable control impairs the integrity or proper functioning of the PerfLoc Prize Competition, as determined by NIST in its sole discretion.

NIST reserves the right to disqualify any Participant or Participant team it believes to be tampering with the Entry process or the operation of the PerfLoc Prize Competition or to be acting in violation of any applicable rule or condition.

Any attempt by any person to undermine the legitimate operation of the PerfLoc Prize Competition may be a violation of criminal and civil law, and, should such an attempt be made, NIST reserves the authority to seek damages from any such person to the fullest extent permitted by law.

Verification of Potential Winner(s)

All potential winners are subject to verification by NIST, whose decisions are final and binding in all matters related to the PerfLoc Prize Competition.

Potential winner(s) must continue to comply with all terms and conditions of the PerfLoc Prize Competition Rules described herein, and winning is contingent upon fulfilling all requirements. In the event that a potential winner, or an announced winner, is found to be ineligible or is disqualified for any reason, NIST may make an award, instead, to another Participant.

Privacy and Disclosure under FOIA

Except as provided herein, information submitted throughout the PerfLoc Prize Competition will be used only to communicate with Participants regarding Entries and/or the PerfLoc Prize Competition. Participant Entries and submissions to this competition may be subject to disclosure under the Freedom of Information Act ("FOIA").

Authority

15 U.S.C. 3719.

Appendix: SE95 Definition

Introduction

The performance analysis boils down to comparing the location estimates generated by an App under test at a finite number N of test points with the corresponding ground-truth coordinates of those points. For i=1,2,...,N, the following terminology is introduced for "test point" i :

It is assumed that the z-axis corresponds to the vertical direction. 

Spherical Error 95% (SE95)

SE95 is defined as the radius of the smallest sphere centered at the origin of the 3D Cartesian coordinate system that encloses 95% of ε_i's. Specifically:

Once again, it is a good idea to plot the empirical CDF of the magnitude of 3D error. Once this empirical CDF has been plotted, then SE95 is simply the inverse value of this function at 0.95.