The rapid white matter (WM) maturation of first years of life is followed by slower yet long-lasting development, accompanied by learning of more elaborate skills. By the age of 5 years, behavioural and cognitive differences between females and males, and functions associated with brain lateralization such as language skills are appearing. Diffusion tensor imaging (DTI) can be used to quantify fractional anisotropy (FA) within the WM and increasing values correspond to advancing brain development. To investigate the normal features of WM development during early childhood, we gathered a DTI data set of 166 healthy infants (mean 3.8 wk, range 2-5 wk; 89 males; born on gestational week 36 or later) and 144 healthy children (mean 5.4 years, range 5.1-5.8 years; 76 males). The sex differences, lateralization patterns and age-dependent changes were examined using tract-based spatial statistics (TBSS). In 5-year-olds, females showed higher FA in wide-spread regions in the posterior and the temporal WM and more so in the right hemisphere, while sex differences were not detected in infants. Gestational age showed stronger association with FA values compared to age after birth in infants. Additionally, child age at scan associated positively with FA around the age of 5 years in the body of corpus callosum, the connections of which are important especially for sensory and motor functions. Lastly, asymmetry of WM microstructure was detected already in infants, yet significant changes in lateralization pattern seem to occur during early childhood, and in 5-year-olds the pattern already resembles adult-like WM asymmetry.