The number of people with dementia will increase as populations age. Insulin resistance (IR) is associated with cognitive decline and dementia. We studied in this longitudinal, population-based cohort study, if change in late-life cognitive function during 5-year follow-up would be predicted by midlife IR (measured 15 years before the first late-life visit) or by late-life brain beta-amyloid (Aβ) accumulation or longitudinal change in late-life Aβ accumulation. Participants (n = 43) were recruited from the population-based Finnish Health 2000 study according to their homeostatic model assessment of insulin resistance (HOMA-IR) values measured in midlife (mean age at midlife 55.1 years), and their APOE ε4 genotype. [¹¹C]PIB-PET imaging (n = 42) and neurocognitive testing (n = 43) were conducted at late-life baseline in 2014 −2016 and after 5-year follow-up in 2019 −2021 (mean age at follow-up 74.8 years). During 5-year follow-up, the decline in late-life (from 2014 −2016 to 2019 −2021) episodic memory was greater in midlife insulin resistance (IR+) group than in the group with normal midlife insulin sensitivity (IR−): IR+ group −0.92 (0.22) vs. IR− group −0.43 (0.26), p = 0.03, (adjusted mean change in episodic memory z-score (standard error)). Late-life brain Aβ burden predicted a greater decline in episodic memory (p = 0.02). The association between midlife IR and episodic memory decline was mediated through late-life Aβ burden. Change in Aβ accumulation during the follow-up was not associated with change in cognition at late-life. In conclusion, midlife IR and late-life brain Aβ burden increased the risk of episodic memory decline in late-life. Our results suggest a link between IR and Alzheimer´s disease.