Impulsive sound has been found to annoy people more than steady-state sound or many other types of sound presented at the same sound level. This study examined the physiological, performance, and subjective effects of impulsive sound on working humans. Exposure to impulsive sound (65 dB LAeq) was compared with quiet sound (35 dB LAeq) and steady-state sound (65 dB LAeq). This parallel-group study, where each participant was exposed to one sound condition, had altogether 59 participants. Physiological stress was measured with stress hormone concentrations in plasma (cortisol and noradrenaline), heart rate variability (HRV), and blood pressure. Psychological stress was measured with subjective noise annoyance, workload, and fatigue. Performance was measured in tasks requiring constant concentration (visual and auditory serial recall and N-back). Compared to quiet sound, impulsive sound caused more annoyance, workload, and lack of energy, raised cortisol concentrations, reduced systolic blood pressure, and decreased accuracy in the 3-back task. Compared with steady-state sound, impulsive sound was experienced as more annoying and causing a higher workload and more lack of energy. Impulsive sound caused physiological and psychological stress and decreased performance compared to quiet sound. Part of this load was due to the increased sound level, which was evident as a physiological stress reaction. Still, there was also an extra stress effect related to the impulsiveness of the sound, reflected as a psychological experience. Special care should be paid to impulsive sound, especially in environments where people are performing mental work.