In the present paper, we show that on a compact Riemannian manifold $(M,g)$ of dimension $d\leqslant 4$ whose metric has negative curvature, the renormalized partition function $Z_g(\lambda)$ of a massive Gaussian Free Field determines the length spectrum of $(M,g)$ and imposes some strong geometric constraints on the Riemannian structure of $(M,g)$. In any finite dimensional family of Riemannian metrics of negative sectional curvature bounded from below and above and whose isometry group is trivial, there is only a \textbf{finite number of isometry classes} of metrics with given partition function $Z_g(\lambda)$. When $d<4$, the same result holds true if the random variable $\int_M:\phi^2:dv$ has given probability distribution and without the lower bound on the sectional curvatures.