QuAIL - Question Answering for Artificial Intelligence

What is QuAIL?

A new kind of question-answering dataset that combines commonsense, text-based, and unanswerable questions, balanced for different genres and reasoning types. Reasoning type annotation for 9 types of reasoning: temporal, causality, factoid, coreference, character properties, their belief states, subsequent entity states, event durations, and unanswerable. Genres: CC license fiction, Voice of America news, blogs, user stories from Quora 800 texts, 18 questions for each (~14K questions).

Getting started

Questions?

Send an email to Matt Downey (matt.downey18@gmail.com) or Anna Rogers (anna.gld@gmail.com).

Leaderboard (Last updated: 31 Mar 20 17:31 EDT)

Rank All Temp. Caus. Fact. Char. Ent. Belief Sub. Dur. Unans. Team Submitted Model
1 59.4 59.6 68.3 49.2 52.1 53.3 57.1 55.8 62.9 75.8 matt.downey18 30 Mar 20 21:36 EDT BERT Baseline UML Text Machine Lab
2 51.2 50.0 54.2 40.8 44.2 52.9 44.6 46.7 57.9 69.2 matt.downey18 21 Feb 20 13:54 EST XLNet baseline UML Text Machine Lab
3 29.7 30.4 30.8 30.8 32.1 29.2 26.7 24.2 17.9 45.4 matt.downey18 11 Feb 20 14:47 EST PMI baseline UML Text Machine Lab

Notes

After submission, please allow a few days for your results to show up in the leaderboard.