Information on normal brain structure and development facilitates the recognition of abnormal developmental trajectories and thus needs to be studied in more detail. We imaged 68 healthy infants aged 2–5 weeks with high-resolution structural MRI (magnetic resonance imaging) and investigated hemispheric asymmetry as well as the associations of various total and lobar brain volumes with infant age and sex. We found similar hemispheric asymmetry in both sexes, seen as larger volumes of the right temporal lobe, and of the left parietal and occipital lobes. The degree of asymmetry did not vary with age. Regardless of controlling for gestational age, gray and white matter had different age-related growth patterns. This is a reflection of gray matter growth being greater in the first years, while white matter growth extends into early adulthood. Sex-dependent differences were seen in gray matter as larger regional absolute volumes in males and as larger regional relative volumes in females. Our results are in line with previous studies and expand our understanding of infant brain development.