This study evaluated the performance of several publicly available signal quality indices (SQI) in assessing the quality of synthetic electrocardiogram (ECG) signals with varying categories and levels of noise. We used an existing framework to generate realistic ECG signals with controlled increases in heart rate, power line interference, white noise, and motion artifacts. ECG signals were generated at the threshold of acceptable and unacceptable outputs from each SQI across four categories of noise. The 16 signals were then evaluated by a cardiologist based on four specific criteria and these responses were compared against the SQI outputs. Results showed that the four SQI’s were inconsistent with each other; they also frequently disagreed with the cardiologist assessment. When assessing whether the ECG could be used to ’estimate a plausible heart rate’, the cardiologist assessment agreed with the SQI outputs in between 9/16 and 15/16 cases. When asked whether the ECG was ’clinically useful’, the cardiologist assessment only agreed with SQI’s in between 4/16 and 10/16 cases. The findings from this study underscore the importance of users critically analysing the outputs of SQI’s as their suitability may be limited to only basic heart rate extraction from ECG signals, rather than more comprehensive clinical applications.