Ohbuchi Laboratory Graduate School of Engineering, University of Yamanashi, Yamanashi, Japan.

SHREC 2008 Generic models track

News

(2008/03/14) Deadline extension, Ranklist file naming convension. Deadline to turn in the results are extended by a day to March 15, 2008 11:59 pm UTC. There are modifications to the Ranklist file naming convention.

(2008/03/12) The second query set has been released. This set of queries is new for the SHREC 2008 generic models track.

(2008/03/10) The database and one of the two sets of queries have been released. (Well, they are the same as the SHREC 2006 database and the SHREC 2006 query set.)

Track Overview

The first of the SHREC series of 3D model retrieval contests, SHREC 2006 organized by Prof. Veltkamp, has made an impact in the way researchers compare performances of their 3D model retrieval methods. While many researchers used the SHREC 2006 as their benchmark, there has been no "officeial" contest since 2006 that used the same SHREC 2006 format but with up-to-date algorithms and methods.

This track essentially tries to repeat the SHREC 2006 so that we can compare state-of-the-art methods using a benchmark stable over time.

The database to be queried is identical to SHREC 2006, that is a union of the train and the test set of the Princeton Shape Benchmark. It is a collection of 1,814 polygon soup models. Please refer to the SHREC 2006 home page for the overview of the past contest. There will be more than one query sets, one of which will be identical to SHREC 2006. The database, the tools for quantitative performance evaluation, etc. are largely borrwoed from the SHREC 2006, courtesy of Prof. Veltkamp and his team.

A change from SHREC 2006 to this track is the acknowledgement of learning based algorithms for 3D model retrieval. SHREC 2008 generic models track has two entry categories (see below) depending on if supervised learning is used or not. We want to encourage various form of learning algorithms in 3D model retrieval, as we believe learning algorithms are as essential as feature themselves for effective retrieval. At the same time, however, we do not want to discourage methods without learining. So we decided to create two entry categories; unsupervised mehod and supervised methods.

Information on SHREC 2008 and its tracks are here.

Two entry categories

This track accepts methods that employ machine learning. Benchmark results will be clearly marked to indicate the category the method belongs. The two categories are;

1. Unsupervised methods: This category include methods that do not use any machine learning as well as methods that employ UNSUPERVISED learning. The training set can be anything if the algorithm does not use labels (classification) of the models in the set. For example, the algorithm may use the SHREC 2006 dataset (i.e., PSB test + train set), or the National Taiwan University 3D model database so far as the algorithm ignores the class labels.
2. Supervised methods: This category include methods that employ off-line supervised learning of multiple categories. However, on-line supervised learning, e.g., by using relevance feedback, is not allowed. (That is, all the learning must be done before the retrieval starts, and that no training (e.g., information on classes) are allowed during the retrieval session.) The evaluation will be done using SHREC 2006 categories. The participants are asked to submit the results for the following two cases;
   1. SS: Train the algorithm by using the SHREC2006 ground truth classes (30 classes). Evaluate the algorithm by using the same SHREC database and the same SHREC 2006 groud truth classes.
   2. PS: Train the algorithm by using the PSB train set (90 classes, 907 models) classes. (Do not use PSB test classes.) Evaluate the algorithm by using SHREC database and groud truth classification.

It is debatable as to whether machine learning based methods should be included in the track After some discussion, we decided to include machine learning-based approaches as we believe some form of learning is essential to an effective multimedia data retrieval system.

If you look at such field as content based 2D image retrieval or movie topic detection, in most of the recent papers, learning algorithms takes center stage. Our recent experience tells us that learning is essential to effective 3D model retrieval. Methods that employ learning has appeared in the field of 3D model retrieval. Examples of learning based 3D model retrieval are, for example, "distinctive regions" algorithm by Phillip Shilane, et al., "semantic oriented" retrieval algorithm using relevance feed back by Leifman, et al, "purity" based multi-feature combiner by Benjamin Bustos, et al., "characteristic views" algorithm by Tarik Filali, and our semi-supervised learning algorith. These are interesting approaches that should not be excluded from the comparison.

Machine learning algorithms can be unsupervised or supervised. It is difficult to disallow unsupervised algorithms, for many of the methods already use them in the form of Principal Component Analysis (PCA) to filter out, or reduce dimension of, features.

Supervised learning comes in two flavors; off-line and on-line. An off-line algorithm learns multiple categories (classes) from a set of labeled (classified) training 3D models prior to retrieval. We allow this form of supervised learning in this track.

For this benchmark, we decided NOT to allow on-line supervised learning such as those using relevance feedback. This is mainly because we do not know how to do this properly. For example, relevance feedback has variations, e.g., positive only feedback, positive-and-negative feedback, graded feedback. We can't nail down the protocol for benchmark.

A benchmark of off-line supervised method requires specification of (1) database to train, (2) classes of the database to train, (3) database to evaluate (4) classes of the database to evaluate. For example, it is easier for a learning algorithm if the both classes and database entries for the training set is equal to those of the test (evaluation) set. If they are different, the learning algorithm must have a good generalization ability of classes and/or database to perform well. We decided to use two training set to evaluate the method's generalization capability.

Instruction for competitors

The following is the schedule.

• Registration:
  • Register during March 1 ~ March 7, 2008
  • Register by sending Email to the organizer (After March 1, 2008.) The registration should include,
    • Entry name: Name of the team/method.
    • Contact information: Name, affiliation, contact address and email of the participant.
    • Entry category: Either "Unsupervised method" or "Supervised method"
• Download the database:
  • Database will be abailable by March 10.
    • Database for the SHREC 2008 ( identical to SHREC 2006 database) (zip)
    • The database to be used for this year's track will contain the same models as the SHREC 2006 database, that is, the union of the PSB test set and train set, having 1,814 models. The SHREC 2006 (and 2008) database models are randomly rotated.
    • We decided NOT to fiddle with the database. It is exactly the same as the SHREC 2006 database. (I just downloaded it from the SHREC 2006 site, and placed it here.)
  • Software tools
    • Performance evaluation tools developed by Prof. Veltkamp et al for the SHREC 2006 are still available.Use these tools to evaluate your method(s).
• Download the query sets:
  • Query sets is made available during March 12 ~ March 14.
  • The query set will be a union of two subsets;
    • Query set 1 of the SHREC 2008 (The query set 1 is identical to the SHREC 2006 query set)
      • Models are numbered from 1 to 30.
      • A classification file, the one we used for SHREC 2006 can be found here.
        • Use this classification file
      • A queries file, the one we used for SHREC 2006 can be found here.
      • For a supervised algorithm, use this classification file to train an algorithm for the "SS" category.
      • You may also use the query set1, the classification file, and the performance evaluator to test your algorithm, tune parameters, etc.
    • Query set 2 of the SHREC 2008 (The set 2 is the new set of queries created for the SHREC 2008.)
      • Models are numbered from 31 to 60.
      • Classification file for the query set 2 will be released after the contest.
• Perform retrieval experiments:
  • Perform retrieval experiments on your own ("Honor system").
• Return results and a draft paper:by March 15, 2008 11:59 pm UTC. (instead of March 14, 2008)
  • Return the results of retrieval experiment.
    • For each query, rank all the models in the database in the order of decreasing relevance. (That is, rank them in the order of increasing dissimilarity.)
    • An algorithm is allowed to submit results of two runs of the algorithm using two parameter sets.
    • Clearly name the ranked list file with the following;
      • Entry name. (shorten to less than 8 letters.)
      • The distinction between unsupervised ("UL") and supervised ("SL") methods.
      • For the SL method, add "SS" or "PS" depending on the training set. "SS" is added if the training is done using SHREC 2006 classes, and "PS" is added if the training is done using PSB train classes.
      • The query set used (Q1 or Q2, respectively for the SHREC 2006 set or SHREC 2008 set)The runs (R1 or R2).
      • The Ranklist file name format
        • <Entry name>_<UL|SL>_<""|SS|PS>_<Q1|Q2>_<R1|R2>
        • Example: "FooBar" team using supervised learning. Learning SHREC and retrieving SHREC Query set 1. Ranklist for run 1. The ranklist file name would be;
          • FooBar_SL_SS_Q1_R1
        • Note that not all of the combinations are valid.
  • Return also the draft of the 2-page summary paper describing the retrieval method. The paper shoud describe the retrieval method employed, their performance, and how the method and its results compares to the others in this track. The last part, comparison, can be left emply as it can only be filled after all the results are in. Please use the template here for preparation.
• Submit final version of the paper: by March 24, 2008.
  • The deadline for the final version of the summary paper submission is March 24 (Monday) for the paper to be included in the SMI 2008 proceedings.
  • This year, the 2 pages summary will be included in the SMI 2008 proceedings published by the IEEE Computer Society Press and indexed and archived in the IEEE Computer Society Digital Library (CSDL). The summary paper must follow the SMI paper submission guidelines found at the SMI 2008 paper submission page. (This imposed the tight deadline requirements on the SHREC 2008.)

Evaluation Results

Query set 1

Runfile(s) per participant Q1

Per query_Q1

All runfiles Q1

Query set 2

Runfile(s) per participant Q2

Notice: Queries in this set of results are numbered wrong (you must add 30 to the number displayed) in this set of results. Their associated images are wrong also.
Due to the way we used the SHREC 2006 evaluator for the query set 2, the model index and the image corresponding to the inded are wrong. For example, the Query 31 (axe) in Set 2 appers as Query 1 with the associated wrong image link. However, the evaluation is done correctly as the class information "belongs to axe, blade" indicates. For the correct query IDs and query snapshot images, please see "Query 1 and Query 2 combined"case.
(We numbered the models in the Set 2 from 31~60. The evaluator, however, disregards the filename and number the queries by the order of their apperance in the list. We will rectify the problem in the future.)

Per query Q2

I manually tweaked the edited the query ID numbers for this so you can view the IDs and images correctly.

All runfiles Q2

The evaluation results here are correct, despite the misnumbering of thequeries in the "per participant Q2"above.

Query set 1 and set 2 combined

Queries in both set 1 and set 2 are numbered correctly, and their postate stamp images appears correctly in the following.

Runfile(s) per participant_Q1Q2

Per query Q1Q2

All runfiles Q1Q2

Contact Information

Ryutarou OHBUCHI （Japanese -> 大渕 竜太郎）

Computer Science Department
University of Yamanashi
4-3-11 Takeda, Kofu-shi,
Yamanashi-ken, 400-8511
Japan
Phone: +81-55-220-8570
ohbuchiAT yamanashi DOT ac DOT jp