This paper presents a simple approach leveraging temporal learning on informative frames for action recognition. We propose a training-free simple adaptive frame selection scenario employing just the similarity technique in a temporal window. The proposed frame selection method provides an appropriate strategy to capture informative frames and provide meaningful features. Moreover, we use transfer learning for spatial feature extraction and employ LSTM and GRU for temporal modeling. Our method is evaluated on two popular datasets, UCF11 and KTH, and it demonstrates acceptable results.