On a compact manifold $M$, we consider the affine space $A$ of non self-adjoint perturbations of some invertible elliptic operator acting on sections of some Hermitian bundle, by some differential operator of lower order. We construct and classify all complex analytic functions on the Fr\'echet space $A$ vanishing exactly over non invertible elements, having minimal order and which are obtained by local renormalizations, a concept coming from quantum field theory, called renormalized determinants. The additive group of local polynomial functionals of finite degrees acts freely and transitively on the space of renormalized determinants. We provide different representations of the renormalized determinants in terms of spectral zeta determinants, Gaussian Free Fields, infinite product and renormalized Feynman amplitudes in perturbation theory in position space \`a la Epstein-Glaser. Specializing to the case of Dirac operators coupled to vector potentials and reformulating our results in terms of determinant line bundles, we prove our renormalized determinants define some complex analytic trivializations of some holomorphic line bundle over $A$ relating our results to a conjectural picture from some unpublished notes by Quillen [52] from April 1989.