Respiratory infections could possibly be detected using wearables, as resting heart rate increases during infections. Other physiological processes, such as stress and exercise, can cause similar alterations which might result in a large number of false positives. However, if these alterations happen close to the symptom onset, they might be classified as true positives. Hence, when evaluating performance of infection detectors, it is essential to consider how to define true and false positives. The aim of this study was to examine how changing the definition of true and false positives and the length of the time period, detection window, where positive detections are considered as true positives, can affect sensitivity and specificity of the infection detector. Three open datasets with infection annotations were used. Infections were detected using two previously published detectors. Changes in detector performance were evaluated using three detection windows, where length and location of the window changed, and two methods to define true positives and false negatives. Length and location of the detection window affected the results considerably. In one dataset, sensitivity decreased from 78.3 % to 41.0 % by shortening the window from long pre-symptomatic period to few days around symptom onset. Sensitivities were significantly impacted by the definition of true positives and false negatives. In conclusion, reasonable detection window and evaluation metrics are crucial for useful detection. Future studies should focus on real-world applicability of detectors.