This paper is concerned with the numerical solution of Volterra integro-differential equations with noncompact operators. The focus is on the problems with weakly singular solutions. To handle the initial weak singularity of the solution, a fractional collocation method is applied. A rigorous $hp$-version error analysis of the numerical method under a weighted $H^1$-norm is carried out. The result shows that the method can achieve high order convergence for such equations. Numerical experiments are also presented to confirm the effectiveness of the proposed method.