Despite the ubiquitous function of macrophages across the body, the diversity, origin, and function of adrenal gland macrophages remain largely unknown. We define the heterogeneity of adrenal gland immune cells using single-cell RNA sequencing and use genetic models to explore the developmental mechanisms yielding macrophage diversity. We define populations of monocyte-derived and embryonically seeded adrenal gland macrophages and identify a female-specific subset with low major histocompatibility complex (MHC) class II expression. In adulthood, monocyte recruitment dominates adrenal gland macrophage maintenance in female mice. Adrenal gland macrophage sub-tissular distribution follows a sex-dimorphic pattern, with MHC class II^low macrophages located at the cortico-medullary junction. Macrophage sex dimorphism depends on the presence of the cortical X-zone. Adrenal gland macrophage depletion results in altered tissue homeostasis, modulated lipid metabolism, and decreased local aldosterone production during stress exposure. Overall, these data reveal the heterogeneity of adrenal gland macrophages and point toward sex-restricted distribution and functions of these cells.