Acquired amusia provides a unique opportunity to investigate the fundamental neural architectures of musical processing due to the transition from a functioning to defective music processing system. Yet, the white matter (WM) deficits in amusia remain systematically unexplored. To evaluate which WM structures form the neural basis for acquired amusia and its recovery, we studied 42 stroke patients longitudinally at acute, 3-month, and 6-month post-stroke stages using DTI [tract-based spatial statistics (TBSS) and deterministic tractography (DT)] and the Scale and Rhythm subtests of the Montreal Battery of Evaluation of Amusia (MBEA). Non-recovered amusia was associated with structural damage and subsequent degeneration in multiple WM tracts including the right inferior fronto-occipital fasciculus (IFOF), arcuate fasciculus (AF), inferior longitudinal fasciculus(ILF), uncinate fasciculus (UF), and frontal aslant tract (FAT), as well as in the corpus callosum (CC) and its posterior part (tapetum). In a linear regression analysis, the volume of the right IFOF was the main predictor of MBEA performance across time. Overall, our results provide a comprehensive picture of the large-scale deficits in intra- and interhemispheric structural connectivity underlying amusia, and conversely highlight which pathways are crucial for normal music perception.