Small cells are a promising technique to improve the capacity and throughput of future wireless networks. However, user association and power allocation in heterogeneous networks is complicated by the dense deployment of small cells, resulting in non-convex and combinatorial problems. Conventionally, machine learning techniques are applied to the joint optimization problem, which has different action spaces. Gauging the continuous spaces to discrete spaces results in the loss of granularity due to discretization (e.g. potential power values in power allocation). Due to its hybrid action space, it is sub-optimal to solve joint user association (discrete spaces) and power allocation (continuous spaces) problems by applying traditional machine learning approaches. This work proposes a Multi-Agent Parameterized Deep Reinforcement Learning (MA-PDRL) approach to address the joint user association and power allocation problem efficiently. According to simulation results, the proposed multi-agent PDRL performs better in energy efficiency and QoS satisfaction than WMMSE, game theory, Q-learning, and DRL techniques.