We investigated the longitudinal associations between sports participation patterns in youth and physical activity (PA) in adulthood. PA was self-reported triannually between ages 9–18 (n = 2550, 52% females) and measured by accelerometers in mid-adulthood (n = 1002, 61% females, aged 48 ± 4 years). Three latent classes of youth sports participation emerged for both genders: 1) “organized sports” (persistent high PA with regular sports club activities), 2) “unorganized sports” (persistent high PA without sports club activities and 3) “low activity” (low PA with decreasing sports involvement). These groups comprised 29%, 34% and 37% of males, and 23%, 27% and 50% of females, respectively. Youth “organized sports” was associated with higher adult PA in both males (+1166 steps/day, p = 0.012) and females (+15 min/day moderate-to-vigorous PA [MVPA], +1064 steps/day, +1066 leisure-time steps/day; p ≤ 0.005) compared to “low activity”. In males, youth “organized sports” was associated with higher adult PA (+1103 steps/day, −26 min/day sedentary time and +133 counts/minute higher total PA, p ≤ 0.039) compared to “unorganized sports”. In females, “unorganized sports” in youth was related to higher adult PA (+10 min/day MVPA, p = 0.034) when compared to “low activity”. Sustained participation in youth organized sports, and for females, also in unorganized sports, is positively linked with adult PA.