In order to achieve autonomous warehouse inspection, a reliable rack monitoring and instant detection of rack is necessary. Damage detection is an essential task for pallet rack maintenance and it requires large amount of manual efforts. To address this problem, we employ deep learning to automatically detect damages with their per-pixel segmentation mask in this paper. We also investigated the impact of using related and unrelated data on the detection performance. For this purpose, we compared the performance of a detector when it is trained on: (1) the COCO dataset, (2) the ImageNet dataset and (3) a related task such as car damage dataset. Moreover, we evaluated the performance of the proposed detector based on different backbones. Experiments show that the detector with ResNet101 as feature extractor can achieve 93.45% accuracy in our real dataset. The code and dataset can be viewed at: https://gitlab.utu.fi/drone-warehouse/beam_defect_detection.git.