The effects of sport participation (SP) are typically assessed over relatively short time frames, with limited information regarding long-term impacts. The aims of this study were to evaluate the relationship of SP during childhood or adolescence with physical activity (PA) and health-related quality of life (HRQoL) in adulthood. We performed an observational cross-sectional study. A population survey was conducted in two rural Finnish towns during 2005–7. Apparently healthy cardiovascular risk subjects aged 45–70 years were identified, and information regarding current PA and HRQoL (36-item Short Form Health Survey) and childhood or adolescent SP were gathered using questionnaires. Current PA was measured in metabolic equivalent hours per week (MET-h/week). Participants (n = 2503; mean age 58 years) were divided into three groups based on their SP levels during childhood or adolescence: none (n = 338), hobby (n = 1713), and competitive (n = 452). The mean level of current PA was 8.1 (SD 6.8) MET-h/week in none group, 9.2 (7.1) in hobby group, and 10.4 (7.8) in competitive group (P < .001). Participants in the competitive group reported significantly better HRQoL compared to other groups (P < .001). Childhood or adolescent SP modified the relationship between adulthood PA and HRQoL. Individuals with high levels of SP during childhood or adolescence were more likely to remain active in adulthood. SP in early life modified the association between adulthood PA and HRQoL and was also associated with better HRQoL in adulthood especially in mental health-related domains. These findings highlight the importance of promoting PA from an early age.