Coalescing black hole (BH) binaries forming in the dense core of globular clusters (GCs) are expected to be one of the brightest sources of gravitational wave (GW) radiation for the next generation of ground-based laser interferometers. Favorable conditions for a merger are initiated by the Kozai resonance in which the gravitational interaction with a third distant object, typically another BH, induces quasi-periodic variations of the inner BH binary eccentricity. In this article we perform high precision three-body simulations of the long-term evolution of hierarchical BH triples and investigate the conditions that lead to the merging of the BH binary and the way it might become an observable source of GW radiation. We find that the secular orbit average treatment, which was adopted in previous works, does not reliably describe the dynamics of these systems if the binary is orbited by the outer BH on a highly inclined orbit at a moderate distance. We show that 50% of coalescing BH binaries driven by the Kozai mechanism in GCs will have eccentricities larger than 0.1, with 10% of them being extremely eccentric, (1-e) less than or similar to 10(-4), when they first chirp in the frequency band of ground-based laser interferometers. This implies that a large fraction of such GW sources could be missed if conventional quasi-circular templates are used for analysis of GW detector data. The efficient detection of all coalescing BH binaries in GCs will therefore require template banks of eccentric inspiral waveforms for matched-filtering and dedicated search strategies.