Although there is growing interest in the reasons for avoiding leisure-time physical activity (PA), the impact of past negative experiences on leisure-time PA has received scant attention in the research literature. This research explores the triggers of negative PA experiences, their impact on leisure-time PA., and the role of experiential avoidance in leisure-time PA. The analysis of rich interview data on adults’ past PA experiences and their current emotions and behaviors related to leisure-time PA showed that past negative PA experiences can hinder, prevent—or occasionally even increase—leisure-time PA. Furthermore, the study provides a classification of distinct triggers for negative PA experiences and adds on the research on experiential avoidance by extending it to the context of leisure-time PA.