home · shared task

Shared task: comparing semantic representations

STEP 2008 will also feature a "shared task" to compare semantic representations as output by state-of-the-art NLP systems. Participating systems will be given a number of (small) texts, before the workshop. The output of these systems will be judged on a number of aspects by a panel of experts in the field, during the workshop. Aim of the shared task is to discuss the feasibility of a gold standard for deep semantic representations. Aim of the panel is to identify a set of problematic and relevant issues for semantic evaluation. The panel will reward the system with the most complete and accurate semantic representation with a special prize. Important dates for the Step Shared Task are:

Intention of participation: June  1, 2008
Shared Task paper submission: June  6, 2008
Notification of acceptance: June 23, 2008
Release of test data: June 25, 2008
System's results due: July  4, 2008
Final version paper due: July 25, 2008
Workshop: Sept 22-24, 2008

To participate in the shared task, submit a paper containing (1) a system description, (2) a description of the semantic formalism used by the system, and (3) an authentic small text and the way it is analysed by the system. This text should be in English (but see below) and not exceed five sentences and 120 tokens. The test data for the shared task will be composed out of all the texts submitted by the participants. Please email Johan Bos (bos@di.uniroma1.it) by June 1 if you intend to participate in the shared task.

Shared task submissions should follow the workshop format for regular papers and submission guidelines, and will be published in the STEP 2008 proceedings. Please choose the category "shared task" when submitting a paper using the EasyChair system. The final paper must include a discussion of the system's performance on the shared task data. Please contact Johan Bos (bos@di.uniroma1.it) for further questions on the shared task, or if you would like to participate with a language other than English.

  2008 · Laboratorio LC · EmailWebmaster