Data Science Chair

    XAI Evaluation Datasets

    Boolean XAI evaluation datasets

    In our paper "Evaluation of post-hoc XAI approaches through synthetic tabular data", we introduce an evaluation setting using synthetic data, in order to investigate which explainable aritificial intelligence (XAI) approaches correctly explain the decision making of deep neural networks that solve basic Boolean functions. Finding that providing explanations on datasets proves no trivial task for the investigated XAI approaches, we publish the generated synthetic data as benchmark datasets.

    The datasets contain 12 Boolean features with every possible permutation being included exactly once in the dataset, resulting in 2*12=4096 data samples in each dataset. The labels 'y' are generated as described in the paper, with the first columns being used for label calculation (i.e. for the XOR dataset the label is calculated by y = f1 XOR f2 ). The 'explanation' column contains the relevant feature columns for each data sample, according to the definition given in the paper.

    You can download the datasets using this link.


    • Evaluation of post-hoc XA...
      Evaluation of post-hoc XAI approaches through synthetic tabular data. Tritscher, Julian; Ring, Markus; Schlör, Daniel; Hettinger, Lena; Hotho, Andreas. In International Symposium on Methodologies for Intelligent Systems. Springer, 2020.