Charts are very popular for analyzing data. When exploring charts, peopleoften ask a variety of complex reasoning questions that involve several logicaland arithmetic operations. They also commonly refer to visual features of achart in their questions. However, most existing datasets do not focus on suchcomplex reasoning questions as their questions are template-based and answerscome from a fixed-vocabulary. In this work, we present a large-scale benchmarkcovering 9.6K human-written questions as well as 23.1K questions generated fromhuman-written chart summaries. To address the unique challenges in ourbenchmark involving visual and logical reasoning over charts, we present twotransformer-based models that combine visual features and the data table of thechart in a unified way to answer questions. While our models achieve thestate-of-the-art results on the previous datasets as well as on our benchmark,the evaluation also reveals several challenges in answering complex reasoningquestions.