Male reproductive functions are strictly regulated in order to maintain sperm production and fertility. All processes are controlled by precise regulation of gene expression, which creates specific gene expression programs for different developmental stages and cell types, and forms the functional basis for the reproductive system. Small non-coding RNAs (sncRNAs) are involved in gene regulation by targeting mRNAs for translational repression and degradation through complementary base pairing to recognize their targets. This review article summarizes the current knowledge on the function of different classes of sncRNAs, in particular microRNAs (miRNAs) and PIWI-interacting RNAs (piRNAs), during male germ cell differentiation, with the focus on sncRNAs expressed in the germline. Although transcriptionally inactive, mature spermatozoa contain a complex population of sncRNAs, and we also discuss the recently identified role of sperm sncRNAs in the intergenerational transmission of epigenetic information on father's environmental and lifestyle exposures to offspring. Finally, we summarize the current information on the utility of sncRNAs as potential biomarkers of infertility that may aid in the diagnosis and prediction of outcomes of medically assisted reproduction.