Age of acquisition (AoA) may have long-lasting effects on second language learning, such as vowel perception, especially in immersion contexts. However, it's important to note that an earlier start of second language learning may not be equally important for all learners. Identifying the key individual characteristics that moderate the AoA effect is essential to elucidate the nature of the AoA effect. This study aimed to investigate the significance of the AoA effect in a non-immersion second language learning context and examine whether individual characteristics, such as second language proficiency, general auditory processing, and attention switching abilities may moderate this effect. Fifty-eight native Chinese-speaking university students, with over ten years of non-immersion English learning, participated in tasks assessing English vowel perception, non-linguistic sound perception, and attention switching ability. The results showed that the AoA effect remained significant among non-immersion Chinese English learners. Moreover, stronger individual attention switching and higher English listening proficiency, but not auditory processing ability, buffered the AoA effect. The moderating effect of attention switching remained significant even after controlling for the moderating effect of English listening proficiency. Earlier start of second language learning may be particularly important for learners with a slower attention switch (indicated by a longer P3 latency) to learn speech perception, while not so much for those with a quicker attention switch (indicated by a shorter P3 latency). The findings were the first empirical evidence highlighting that the AoA effect on second language learning in non-immersive contexts may differ by learners' cognitive ability. The present study expanded the knowledge of the complexity of the AoA effect to more specific individual characteristics.