RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic

We present a comprehensive study of chart visual question-answering(QA) task,to address the challenges faced in comprehending and extracting data from chartvisualizations within documents. Despite efforts to tackle this problem usingsynthetic charts, solutions are limited by the shortage of annotated real-worlddata. To fill this gap, we introduce a benchmark and dataset for chart visualQA on real-world charts, offering a systematic analysis of the task and a noveltaxonomy for template-based chart question creation. Our contribution includesthe introduction of a new answer type, 'list', with both ranked and unrankedvariations. Our study is conducted on a real-world chart dataset fromscientific literature, showcasing higher visual complexity compared to otherworks. Our focus is on template-based QA and how it can serve as a standard forevaluating the first-order logic capabilities of models. The results of ourexperiments, conducted on a real-world out-of-distribution dataset, provide arobust evaluation of large-scale pre-trained models and advance the field ofchart visual QA and formal logic verification for neural networks in general.