Light and moderate alcohol use has been reported to be associated with both impaired and enhanced cognition. The purpose of this study was to explore whether there was a linear relationship between visual memory and alcohol consumption in males and females in a large middle-aged birth cohort population in cross-sectional and longitudinal settings. Data were collected from 5585 participants completing 31-year (1997–1998) and 46-year (2012–2014) follow-ups including Paired Associate Learning (PAL) test at 46-years follow-up. The participants were originally from 12,231 study population of the Northern Finland Birth Cohort 1966 (NFBC1966). The PAL test was conducted to assess visual memory. Reported alcohol use was measured as total daily use of alcohol, beer, wine, and spirits converted into grams and as frequency and amount of use of beer, wine, and spirits. The total daily alcohol use was not associated with reduced visual memory. The frequency of use of beer and wine in males was associated with better visual memory in cross-sectional and longitudinal settings. Using six or more servings of spirits was associated with worse visual memory in males in cross-sectional and longitudinal settings. The study suggested a lack of a linear association between drinking and visual memory in the middle-aged population.