e-CARE

a New Dataset for Exploring Explainable Causal Reasoning

News

  • 02/15/2022 e-CARE is now public available!

What is e-CARE?

Understanding causality has vital importance for various Natural Language Processing (NLP) applications. Beyond the labeled instances, conceptual explanations of the causality can provide deep understanding of the causal fact to facilitate the causal reasoning process. We present a human-annotated explainable CAusal REasoning dataset (e-CARE)


e-CARE contains over 20K causal reasoning questions, together with natural language formed explanations of the causal questions.


Getting Started

We've built a few resources to help you get started with the dataset.

Download a copy of the dataset in jsonl format:

Read the following Readme to get familiar with the data structure.

Submission tutorial

The test set of e-CARE is a blind set, you should following the instruction to get the performace on test set. And the submitted models will be added to the leaderboard with the premission of the author.

Evaluation Scripts

We provide two evaluation scripts for obtaining evaluation metrics on dev and test sets. The format of prediction files can refer to github.

  • Causal reasoning
  • Conceptual Explanation Generation
  • License

  • License: e-CARE is under the MIT License
  • Have Questions?

    Ask us questions at our ldu@ir.hit.edu.cn or at kxiong@ir.hit.edu.cn.

    Acknowledgements

    We thank the SQuAD team and theReCoRD team for using their code and templates for generating this website..

    CausalĀ Reasoning

    Rank Model Accuracy (%)
    Human Performance

    HIT-SCIR

    (Du et al., 2022)

    92

    1

    Mar 15, 2022
    BERT-base-cased

    HIT [modification of the Google AI implementation]

    (Devlin et al., 2019)
    75.38

    2

    Mar 15, 2022
    ALBERT

    HIT [modification of the Google Research implementation]

    (Lan et al., 2019)
    74.60

    3

    Mar 15, 2022
    XLNet-base-cased

    HIT [modification of the Google Brain implementation]

    (Yang et al., 2019)
    74.58

    4

    Mar 15, 2022
    BART-base

    HIT [modification of the Facebook AI implementation]

    (Lewis et al., 2020)
    71.65

    5

    Mar 15, 2022
    RoBERTa-base

    HIT [modification of the Facebook AI implementation]

    (Liu et al., 2019)
    70.73

    6

    Mar 15, 2022
    GPT-2

    HIT [modification of the OpenAI implementation]

    (Radford et al., 2019)
    69.51

    7

    Mar 15, 2022
    GPT

    HIT [modification of the OpenAI implementation]

    (Radford et al., 2018)
    68.15

    Conceptual Explanation Generation

    Rank Model    BLEU       Rouge-l   
    Human Performance

    HIT-SCIR

    (Du et al., 2022)

    35.51 33.46

    1

    Mar 15, 2022
    GPT-2

    HIT [modification of the OpenAI implementation]

    (Radford et al., 2019)
    32.05 31.47

    2

    Mar 15, 2022
    RNN

    HIT [modification of the LSTM]

    (Hochreiter and Schmidhuber, 1997)
    18.09 20.85