A major concern with co-administration of different drugs is the high risk of interference between their mechanisms of action, known as adverse drug–drug interactions (DDIs), which can cause serious injuries to the organism. Although several computational methods have been proposed for identifying potential adverse DDIs, there is still room for improvement. Existing methods are not explicitly based on the knowledge that DDIs are fundamentally caused by chemical substructure interactions instead of whole drugs’ chemical structures. Furthermore, most of existing methods rely on manually engineered molecular representation, which is limited by the domain expert’s knowledge.We propose substructure–substructure interaction–drug–drug interaction (SSI–DDI), a deep learning framework, which operates directly on the raw molecular graph representations of drugs for richer feature extraction; and, most importantly, breaks the DDI prediction task between two drugs down to identifying pairwise interactions between their respective substructures. SSI–DDI is evaluated on real-world data and improves DDI prediction performance compared to state-of-the-art methods. Source code is freely available at https://github.com/kanz76/SSI-DDI.