Semen quality has declined especially among Western men. Experimental and epidemiological studies have shown potential links between exposure to environmental toxicants and poor male fertility. Some environmental exposures in utero can disrupt fetal testicular function and result in cryptorchidism, low semen quality, low serum testosterone levels, and low fertility. Environmental exposure in childhood and adulthood can also adversely affect germ cells, Sertoli cells, Leydig cells, or the hypothalamic-pituitary-testicular axis, resulting in impaired male fertility. In this review, we report the latest results from human studies that investigated the role of endocrine disrupting chemicals, heavy metals, tobacco smoking, alcohol drinking, and use of marijuana in low semen quality and impaired male fertility. Current evidence suggests the relationship between these environmental factors and low male fertility; however, some factors showed conflicting results which need further investigation.