Automatic text summarization has made tremendous progress in recent years. However, the rating of a summary is still an open research topic. Especially when it comes to measuring the abstractiveness, existing evaluation metrics like ROUGE, BLEU or METEOR show severe shortcomings.
We aim to explore new ideas and solutions regarding an automatic quality assessment of German text summarizations and invite you to participate in the 2nd German Text Summarization Challenge. For the challenge, we provide a text corpus together with several summaries per text. Your goal is to assign a quality measure in the range from 0 (bad) to 1 (excellent) to each summary. We ask you to consider aspects such as correctness in content and grammar as well as facets like compactness and abstractiveness.
You can submit (and resubmit) your solution to our evaluation board (see below for registration). Your solution will be evaluated automatically, and your rank will be published on the leaderboard. At the submission deadline, the best four participants will be invited to present their solution at the 5th Swiss Text Analytics Conference (SwissText 2020), which will be held June 23-25 in Zurich, Switzerland. All participants are invited to submit a paper that outlines their solution/metric.
With this challenge we hope to create a basis for interesting discussions and take the NLP community for German text understanding a step further. We look forward to many interesting solutions and a lively discussion.
You’ll find the competition here.
- Dominik Frefel (FHNW, Switzerland)
- Fabian Märki (FHNW, Switzerland)
- Manfred Vogel (FHNW, Switzerland)
- 3 February 2020: Start of challenge (registration and submission page opens, release of data)
- 4 May 2020: Metric and Paper submission deadline and speaker invitation
- 15 May 2020: Paper reviews due
- 29 May 2020: Camera-ready paper deadline
- 23 – 25 June 2020: SwissText (presentations + discussions + prize awarding)