Successful active speaker detection requires a three-stage pipeline: (i)audio-visual encoding for all speakers in the clip, (ii) inter-speaker relationmodeling between a reference speaker and the background speakers within eachframe, and (iii) temporal modeling for the reference speaker. Each stage ofthis pipeline plays an important role for the final performance of the createdarchitecture. Based on a series of controlled experiments, this work presentsseveral practical guidelines for audio-visual active speaker detection.Correspondingly, we present a new architecture called ASDNet, which achieves anew state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 93.5%outperforming the second best with a large margin of 4.7%. Our code andpretrained models are publicly available.