In this paper, we propose an effective object detection framework based on proposal fusion of multiple sensors such as infrared camera, RGB cameras, radar and LiDAR. Our framework first applies the Selective Search (SS) method on RGB image data to extract possible candidate proposals which likely contain the objects of interest. Then it uses the information from other sensors in order to reduce the number of generated proposals by SS and find more dense proposals. Finally, the class of objects within the final proposals are identified by Convolutional Neural Network (CNN). Experimental results on real dataset demonstrate that our framework can precisely detect meaningful object regions using a smaller number of proposals than other object proposals methods. Further, our framework can achieve reliable object detection and classification results in maritime environments.