|
Shared task: comparing semantic representations
STEP 2008 will also feature a "shared task" to compare semantic
representations as output by state-of-the-art NLP systems.
Participating systems will be given a number of (small) texts, before
the workshop. The output of these systems will be judged on a number
of aspects by a panel of experts in the field, during the workshop.
Aim of the shared task is to discuss the feasibility of a gold
standard for deep semantic representations. Aim of the panel is to
identify a set of problematic and relevant issues for semantic
evaluation. The panel will reward the system with the most complete
and accurate semantic representation with a special prize. Important
dates for the Step Shared Task are:
| Intention of participation: | June 1, 2008 |
| Shared Task paper submission: | June 6, 2008 |
| Notification of acceptance: | June 23, 2008 |
| Release of test data: | June 25, 2008 |
| System's results due: | July 4, 2008 |
| Final version paper due: | July 25, 2008 |
| Workshop: | Sept 22-24, 2008 |
To participate in the shared task, submit a paper containing (1) a
system description, (2) a description of the semantic formalism used
by the system, and (3) an authentic small text and the way it is
analysed by the system. This text should be in English (but see below)
and not exceed five sentences and 120 tokens. The test data for the
shared task will be composed out of all the texts submitted by the
participants. Please email Johan Bos (bos@di.uniroma1.it) by June 1 if
you intend to participate in the shared task.
Shared task submissions should follow the workshop format for regular
papers and submission guidelines, and will be
published in the STEP 2008 proceedings. Please choose the category
"shared task" when submitting a paper using the EasyChair system. The
final paper must include a discussion of the system's performance on
the shared task data. Please contact Johan Bos (bos@di.uniroma1.it) for further
questions on the shared task, or if you would like to participate with
a language other than English.
|
Webmaster