Mathematics anxiety hinders students' mathematical achievement already in primary school, but research on its effects beyond whole number knowledge is limited. The main aim of the current study is to examine how state and trait mathematics anxiety relate to performance across five tasks that are relevant for the development of mathematics in primary school, including a measure of adaptive expertise with school mathematics. These include mathematical tasks with non-symbolic quantities, whole numbers, and rational numbers. The participants were 406 primary school students attending the 5th grade (N = 188) and 6th grade (N = 218). Our results showed that state anxiety varies across task type. Furthermore, students' self-evaluated state and trait mathematics anxiety had varying negative relations with performance depending on the task type. In particular, we found that mathematics anxiety may limit students' adaptive expertise with rational numbers, even after controlling for other relevant mathematical skills. Overall, our results indicate that existing accounts on the role mathematics anxiety plays in school mathematics should expand to consider differences across task type and measures of anxiety.