We provide a model independent construction of a net of C*-algebras satisfying the Haag-Kastler axioms over any spacetime manifold. Such a net, called the net of causal loops, is constructed by selecting a suitable base K encoding causal and symmetry properties of the spacetime. Considering K as a partially ordered set (poset) with respect to the inclusion order relation, we define groups of closed paths (loops) formed by the elements of K. These groups come equipped with a causal disjointness relation and an action of the symmetry group of the spacetime. In this way the local algebras of the net are the group C*-algebras of the groups of loops, quotiented by the causal disjointness relation. We also provide a geometric interpretation of a class of representations of this net in terms of causal and covariant connections of the poset K. In the case of the Minkowski spacetime, we prove the existence of Poincar\'e covariant representations satisfying the spectrum condition. This is obtained by virtue of a remarkable feature of our construction: any Hermitian scalar quantum field defines causal and covariant connections of K. Similar results hold for the chiral spacetime $S^1$ with conformal symmetry.