The impact that artificial intelligence (AI) has made across several industries in today’s society is clearly seen in applications ranging from medical diagnosis to customer service chatbots, to financial trading. It is also evident that AI has a huge role to play in emerging and future applications and will be increasingly used in mission-critical and time-sensitive applications such as remote surgeries, cybersecurity and self-driving cars. To satisfy the latency, security and privacy requirements that such applications require, AI which has gained its merit by utilising resource-heavy cloud infrastructure, needs to perform well in resource-constrained environments at the network edge. To address this need, this paper characterises the cloud-to-thing continuum and provides an architecture for enabling AI in fully edge-based scenarios. In addition, the paper provides strategies to tackle the communication inefficiencies that arise from the distributed nature of fully edge-based scenarios. Performance improvements exhibited by these strategies in state-of-the art research is presented, as well as directions where further advancements can be made. The material is presented in a simple manner to catalyse the understanding and hence the participation of multidisciplinary researchers in addressing this challenge.