Evaluation of Cross-Lingual Video Retrieval for How-to Questions

Data:

• questions.csv
• video_results.csv

The dataset and annotation scheme is described in the paper:

Pavel Braslavski, Suzan Verberne, Ruslan Talipov. Show me how to tie a tie: evaluation of cross-lingual video retrieval. CLEF'2016. (preprint, SpringerLink, BibTeX)

Each original Russian question in questions.csv is annotated with

1. machine translation used for querying YouTube API;
2. category;
3. whether or not video would be a good answer medium for this question (0 — no, 1 — possibly, 2 — yes);
4. whether or not results in English would be useful (0 — no, 1 — possibly, 2 — yes).

Video search results contain queryID from questions.csv file, language (ru — original Russian query, en — machine translated query), YouTube video IDs, and three relevance judgements from MTurk workers on 4-point scale (0 — not relevant, 3 — excellent answer; -2 — video unavailable).

Please refer to the above mentioned publication when using the data.

Please send questions and suggestions to pbras аТ yandex dоТ ru