We report on a study of dynamic nuclear polarization and electron and nuclear spin relaxation of atomic hydrogen and deuterium in solid molecular matrices of , and HD mixtures. The electron and nuclear spin relaxation times ( and ) were measured within the temperature range 0.15-2.5 K in a magnetic field of 4.6 T, conditions which ensure a high polarization of electron spins. We found that is nearly temperature independent in this temperature range, while decreased by two orders of magnitude upon raising temperature. Such strong temperature dependence is typical for the nuclear Orbach mechanism of relaxation via the electron spins. We found that the nuclear spins of H atoms in solid and can be efficiently polarized by the Overhauser effect. Pumping the forbidden transitions of H atoms also leads to DNP, with the efficiency strongly dependent on the concentration of D atoms. This behavior indicates the cross effect mechanism of the DNP and nuclear relaxation, which turns out to be well resolved in the conditions of our experiments. Efficient DNP of H atoms was also observed when pumping the middle D line located in the center of the ESR spectrum. This phenomenon can be explained in terms of clusters or pairs of H atoms with a strong exchange interaction. These clusters have partially allowed transitions in the center of the ESR spectrum, and DNP may be created via the resolved cross effect.