Recognizing human actions in videos requires spatial and temporalunderstanding. Most existing action recognition models lack a balancedspatio-temporal understanding of videos. In this work, we propose a noveltwo-stream architecture, called Cross-Attention in Space and Time (CAST), thatachieves a balanced spatio-temporal understanding of videos using only RGBinput. Our proposed bottleneck cross-attention mechanism enables the spatialand temporal expert models to exchange information and make synergisticpredictions, leading to improved performance. We validate the proposed methodwith extensive experiments on public benchmarks with different characteristics:EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400. Our methodconsistently shows favorable performance across these datasets, while theperformance of existing methods fluctuates depending on the datasetcharacteristics.