QuAIL - Question Answering for Artificial Intelligence

What is QuAIL?

A new kind of question-answering dataset that combines commonsense, text-based, and unanswerable questions, balanced for different genres and reasoning types. Reasoning type annotation for 9 types of reasoning: temporal, causality, factoid, coreference, character properties, their belief states, subsequent entity states, event durations, and unanswerable. Genres: CC license fiction, Voice of America news, blogs, user stories from Quora 800 texts, 18 questions for each (~14K questions).

Getting started

Questions?

Contact Matt Downey regarding the leaderboard, and Anna Rogers regarding QuAIL data.

Leaderboard (Last updated: 30 Jun 21 10:34 EDT)

Rank All Temp. Caus. Fact. Char. Ent. Belief Sub. Dur. Unans. Team Submitted Model
1 53.4 53.3 61.2 62.1 42.9 55.4 58.8 53.3 62.9 30.8 matt.downey18 31 Oct 20 09:02 EDT TML BERT Baseline
2 31.1 34.2 29.2 35.4 33.3 26.2 25.8 25.0 25.8 45.0 matt.downey18 30 Oct 20 20:16 EDT TML PMI Baseline

Notes

The leaderboard is live!

After submission, please allow a few hours for your results to show up in the leaderboard.