Computer vision and real-time object detection and classification play a crucial role in modern surveillance systems that enhance public security and traffic management. YOLO-based object detection techniques for urban environments are thoroughly examined in this paper, focusing on the areas around the Helsinki Port. By utilizing live stream data, the research examines and depicts the challenges of object detection and classification in real-life scenarios. Unbalanced data distributions, variable camera angles and different weather and lighting conditions posed several challenges, prompting innovative solutions. This research work not only discussed these challenges, but also provides detailed insights into the data collection and model training methodology. To guarantee the dataset’s accuracy and diversity, several sophisticated methods were analyzed, such as binary filter masks and Computer Vision Annotation Tool (CVAT) annotations. The datasets were used for training various YOLO (you only look once) models, to compare efficiency, accuracy, and speed.