The brain regions responsible for hallucinations remain unclear. We studied 89 brain lesions causing hallucinations using a recently validated technique termed lesion network mapping. We found that hallucinations occurred following lesions to a variety of different brain regions, but these lesion locations fell within a single functionally connected brain network. This network was defined by connectivity to the cerebellar vermis, inferior cerebellum (bilateral lobule X), and the right superior temporal sulcus. Within this single hallucination network, additional connections with the lesion location dictated the sensory modality of the hallucination: lesions causing visual hallucinations were connected to the lateral geniculate nucleus in the thalamus while lesions causing auditory hallucinations were connected to the dentate nucleus in the cerebellum. Our results suggest that lesions causing hallucinations localize to a single common brain network, but additional connections within this network dictate the sensory modality, lending insight into the causal neuroanatomical substrate of hallucinations.