SHREC 2012 - Sketch-Based 3D Shape Retrieval

Call For Participation:

SHREC 2012 - Sketch-Based 3D Shape Retrieval

Objective

The objective of this track is to evaluate the performance of different sketch-based 3D model retrieval algorithms using both hand-drawn and standard line drawings sketch queries on a watertight 3D model dataset.

Introduction

Sketch-based 3D model retrieval is to retrieve 3D models using a 2D sketch as input. This scheme is intuitive and convenient for users to search for relevant 3D models and also important for several applications including sketch-based modeling and sketch-based shape recognition. However, most existing 3D model retrieval algorithms target the Query-by-Model framework, that is, using existing 3D models as queries. Much less research work has been done regarding the Query-by-Sketch framework. In addition, until now there is no comprehensive evaluation or comparison for available sketch-based retrieval algorithms. Considering of this, we organize this track to foster this challenging research area by providing a common sketch-based retrieval benchmark and soliciting retrieval results from current state-of-the-art retrieval methods for comparison. We will also provide corresponding evaluation code for computing a set of performance metrics similar to those used in the Query-by-Model retrieval technique.

Task description

The test dataset will be made available on the 7th of February and the results will be due one week after that. Every participant will perform the queries and send us their retrieval results. We will then do the performance assessment. Participants and organizers will write a joint contest report to detail the results. Results of the track will be presented during the 3DOR workshop 2012 in Cagliari, Italy.

Dataset

• 3D target dataset
  
  Our 3D benchmark dataset is built based on the Watertight Model Benchmark (WMB) dataset [1] which has 400 watertight models, divided into 20 classes, with 20 models each. The 3D target dataset contains two versions: Basic and Extended. 
   

  (1) Basic version

  It comprises 13 selected classes from the WMB dataset with each 20 models (in summary, 260 models). In the basic version, all 13 classes are considered relevant for the retrieval challenge. Fig. 1(c) shows one typical example for each class of the basic benchmark.

  (2) Extended version

  It adds to the basic version all remaining 7 classes of the WMB dataset (each 20 models). These additional classes, however, are not considered relevant for the retrieval challenge but added to increase the retrieval difficulty of the basic version. Fig. 1(d) illustrates typical examples for these remaining 7 irrelevant classes. The Extended version is utilized to test the robustness performance of a sketch-based retrieval algorithm.
• 2D query set
  
  The 2D query set comprises two subsets, falling into two different types.

  (1) Hand-drawn sketches

  We utilize the hand-drawn sketch data compiled by TU Darmstadt and Fraunhofer IGD [2]. It contains 250 hand-drawn sketches, divided into 13 classes. One typical example for each class is shown in Fig. 1(a).

  (2) Standard line drawings

  We also select relevant sketches from the Snograss and Vanderwart's standard line drawings dataset [3]. Some examples are shown in Fig. 1(b).

  In this track, the two subsets will be tested separately. However, users can also form a query set by combining these two to form a query set which contains diverse types of sketches.

Evaluation Methodology

To have a comprehensive evaluation of the retrieval algorithm, we employ seven commonly adopted performance metrics in 3D model retrieval technique. They are Precision-Recall curve (PR), Nearest Neighbor (NN), First Tier (FT), Second Tier (ST), E-Measures (E), Discounted Cumulated Gain (DCG) and Average Precision (AP). We also have developed the code to compute them.

Procedure

The following list is a step-by-step description of the activities:

• The participants must register by sending a message to %20shrec [at] nist.gov (SHREC[at]nist[dot]gov). Early registration is encouraged, so that we get an impression of the number of participants at an early stage.
• The database will be made available via this website. Test dataset.
• Participants will submit the dissimilarity matrix (also named as distance matrix) for the test database. Up to 5 matrices per group may be submitted, resulting from different runs. Each run may be a different algorithm, or a different parameter setting. More information on the dissimilarity matrix file format. More information on the dissimilarity matrix file format.
• The evaluations will be done automatically.
• The organization will release the evaluation scores of all the runs.
• The participants write a one page description of their method with two figures and send their comments on the evaluation results.
• The track results are combined into a joint paper, published in the proceedings of the Eurographics Workshop on 3D Object Retrieval.
• The description of the tracks and their results are presented at the Eurographics Workshop on 3D Object Retrieval (May 13, 2012).

Schedule

Organizers

Bo Li, Afzal Godil - National Institute of Standards and Technology 
Tobias Schreck - University of Konstanz

Acknowledgement

We would like to thank Sang Min Yoon (Yonsei University, Korea), Maximilian Scherer (TU Darmstadt, Germany), Tobias Schreck (University of Konstanz) and Arjan Kuijper (Fraunhofer IGD) who collected the TU Darmstadt and Fraunhofer IGD sketch data. 
We would also like to thank Daniela Giorgi who built the Watertight Shape Benchmark for SHREC 2007. 
We would like to thank Snograss and Vanderwart who built the standard line drawings dataset.

References

[1] R. C. Veltkamp and F. B. ter Haar. SHREC 2007 3D Retrieval Contest. Technical Report UU-CS-2007-015, Department of Information and Computing Sciences, Utrecht University, 2007. 
[2] S. M. Yoon, M. Scherer, T. Schreck, and A. Kuijper. Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours. In ACM Multimedia, pages 193-200, 2010. 
[3] J. G. Snodgrass and M. Vanderwart. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Human Learning and Memory, 6(2):174-215, 1980.

Please cite the papers:

[1] B. Li, T. Schreck, A. Godil, M. Alexa, T. Boubekeur, B. Bustos, J. Chen, M. Eitz, T. Furuya, K. Hildebrand, S. Huang, H. Johan, A. Kuijper, R. Ohbuchi, R. Richter, J. M. Saavedra, M. Scherer, T. Yanagimachi, G. J. Yoon, S. M. Yoon, In: M. Spagnuolo, M. Bronstein, A. Bronstein, and A. Ferreira (eds.), SHREC'12 Track: Sketch-Based 3D Shape Retrieval, Eurographics Workshop on 3D Object Retrieval 2012 (3DOR 2012), 2012. 
[2] B. Li, Y. Lu, A. Godil, T. Schreck, B. Bustos, A. Ferreira, T. Furuya, M.J. Fonseca, H. Johan, T. Matsuda, R. Ohbuchi, P.B. Pascoal, J.M. Saavedra, A comparison of methods for sketch-based 3D shape retrieval, Computer Vision and Image Understanding (2013), doi: http://dx.doi.org/10.1016/j.cviu.2013.11.008.