An algebraic characterization of vacuum states in Minkowski space is given which relies on recently proposed conditions of geometric modular action and modular stability for algebras of observables associated with wedge-shaped regions. In contrast to previous work, continuity properties of these algebras are not assumed but derived from their inclusion structure. Moreover, a unique continuous unitary representation of spacetime translations is constructed from these data. Thus the dynamics of relativistic quantum systems in Minkowski space is encoded in the observables and state and requires no prior assumption about any action of the spacetime symmetry group upon these quantities.