We show that the non-compact generalised analytic Volterra operators $T_g$, where $g in BMOA$, have the following structural rigidity property on the Hardy spaces $H^p$ for $1 le p < infty$ and $p
neq 2$: if $T_g$ is bounded below on an infinite-dimensional subspace $M subset H^p$, then $M$ contains a subspace linearly isomorphic to $ell^p$. This implies in particular that any Volterra operator $T_g: H^p to H^p$ is $ell^2$-singular for $p neq 2$.