The increasing dependence on the Internet of Things (IoT) across various sectors necessitates enhanced security mechanisms, particularly for authentication and authorization (AA) processes. Owing to their dynamic nature and resource constraints, IoT networks are vulnerable to various security breaches, underscoring the need for robust cybersecurity solutions. This study aims to optimize IoT security by evaluating the performance of Artificial Neural Network (ANN) algorithms in this era of Artificial Intelligence (AI) for AA, utilizing physical layer (PHY-layer) attributes, such as device temperature, antenna orientation, Received Signal Strength Indicator (RSSI), and Link Quality Indicator (LQI). Ten different ANN algorithms in Machine Learning (ML), including both classical and recent optimization techniques, were evaluated for their ability to improve convergence rates and minimize errors. These methods were assessed using performance metrics such as convergence epochs, Mean Squared Error (MSE), and correlation coefficients (R-values). The results indicate that the Bayesian Regularization (BR) and Levenberg–Marquardt (LM) algorithms outperformed the others, with the lowest MSE and highest R-values, demonstrating superior performance in IoT AA tasks. This study offers significant insights into the selection of ANN algorithms for robust IoT security and contributes to the development of more reliable and adaptive security solutions for IoT networks. Future work will focus on integrating these optimized algorithms into federated learning systems to enhance the scalability and adaptability of IoT network security mechanisms in evolving network environments.