We make a proposal for gauge-invariant observables in perturbative quantum gravity in cosmological spacetimes, building on the recent work of Brunetti et al. [JHEP 08 (2016) 032]. These observables are relational, and are obtained by evaluating the field operator in a field-dependent coordinate system. We show that it is possible to define this coordinate system such that the non-localities inherent in any higher-order observable in quantum gravity are causal, i.e., the value of the gauge-invariant observable at a point $x$ only depends on the metric and inflation perturbations in the past light cone of $x$. We then construct propagators for the metric and inflaton perturbations in a gauge adapted to that coordinate system, which simplifies the calculation of loop corrections, and give explicit expressions for relevant cases: matter- and radiation-dominated eras and slow-roll inflation.