Alexithymia is a personality construct characterized by difficulties in identifying and verbalizing feelings, a restricted imagination, and an externally oriented thinking style. As alexithymia shows marked overlap with depression, its independent nature as a personality construct is still being debated. The etiology of alexithymia is unknown, although childhood emotional neglect and attachment formation are thought to play important roles. In the FinnBrain Birth Cohort Study, experiences of early-life adversities (EA) and childhood maltreatment (CM) were studied in a sample of 2,604 men and women. The overlap and differences between depression and alexithymia were investigated by comparing their associations with EA types and adult attachment style. Alexithymia was specifically associated with childhood emotional neglect (odds ratio (OR) 3.8, p < .001), whereas depression was related to several types of EA. In depression co-occurring with alexithymia, there was a higher prevalence of emotional neglect (81.3% vs. 54.4%, p < .001), attachment anxiety (t = 2.38, p = .018), and attachment avoidance (t = 4.03, p < .001). Early-life adversities were markedly different in the alexithymia group compared to those suffering from depression, or healthy controls. Depression with concurrent alexithymia may represent a distinct subtype, specifically associated with childhood experiences of emotional neglect, and increased attachment insecurity compared to non-alexithymic depression.