Amplitude modulation exists in many environmental noise types. A penalty has been suggested for legal noise assessment to such sounds, but the scientific evidence is limited. The purpose of this research was to determine the annoyance penalty of amplitude modulated (AM) sound as a function of the modulation frequency fm and depth Dm. A psychoacoustic laboratory experiment was conducted with 40 participants to explore how subjective loudness and annoyance of AM sound depends on fm (from 0.25 to 16 Hz), Dm (from 1 to 14 dB), and overall spectrum (two alternatives). The sounds consisted of both AM sounds and reference sounds without amplitude modulation. The AM sounds were played at 35 dB LAeq, which is typical for environmental noise both indoors and in residential yards. The annoyance penalty increased with increasing fm and Dm. The penalties varied from 4 to 12 dB, when Dm ranged from 4 to 14 dB and fm ranged from 1 to 16 Hz. For the lowest fm= 0.25 Hz, and Dm = 1 dB, no penalty could be suggested. The results suggest a potential need for a penalty for low-level AM sounds for certain ranges of fm and Dm, applied for the periods with AM sound.