Robust real-time object detection and tracking are challenging problems in autonomous transportation systems due to operation of algorithms in inherently uncertain and dynamic environments and rapid movement of objects. Therefore, tracking and detection algorithms must cooperate with each other to achieve smooth tracking of detected objects that later can be used by the navigation system. In this paper, we first present an efficient multi-sensor fusion approach based on the probabilistic data association method in order to achieve accurate object detection and tracking results. The proposed approach fuses the detection results obtained independently from four main sensors: radar, LiDAR, RGB camera and infrared camera. It generates object region proposals based on the fused detection result. Then, a Convolutional Neural Network (CNN) approach is used to identify the object categories within these regions. The CNN is trained on a real dataset from different ferry driving scenarios. The experimental results of tracking and classification on real datasets show that the proposed approach provides reliable object detection and classification results in maritime environments.