Detecting driver drowsiness is a very important task for enhancing the safety of road driving and reducing numerous accidents. In this paper, we proposed a fusion drowsiness detection framework based on video images without any additional wearable devices. The framework first applies Haar feature-based cascade classifiers on the input image to extract the region proposal of the driver's face, eyes and mouth. These interest proposals are then fed into three separate Convolutional Neural Network (CNN) to extract features and predict a class for each proposal based on defined classes. To improve classification performance, we applied transfer learning by using the pre-trained CNNs on images that belong to each region proposal. Finally, the framework can identify driver drowsiness through the defined rules applying to the predicted classes of each region. The rules specify the final class based on the class of mouth and eye to increase the robustness of the framework. The obtained results on the real RLDD dataset [1] show that the proposed framework can identify driver drowsiness with high accuracy and speed.