Underwater robots, such as ROVs (Remotely Operated underwater Vehicles), are widely employed in both inspection and maintenance of offshore structures, in order to avoid the life risks associated with these operations at great depths. However, in view of the high levels of uncertainty that are inherent to the underwater environment, conventional control approaches usually can not provide the required performance to allow precise maneuvers with underwater robots. In this paper, an intelligent controller is proposed for the accurate depth control of a ROV. An artificial neural network is embedded in the control law to compensate for external disturbances and unmodeled dynamics. In addition, a reinforcement learning scheme, namely the Upper Confidence Bound algorithm, is used to tune some of the network parameters. The boundedness and convergence properties of the closed-loop signals are analytically proven. Numerical results confirm the improved performance of the proposed control approach.