The Tomita-Takesaki modular groups and conjugations for the observable algebras of space-like wedges and the vacuum state are computed for translationally covariant, but possibly not Lorentz covariant, generalized free quantum fields in arbitrary space-time dimension d. It is shown that for $d\geq 4$ the condition of geometric modular action (CGMA) of Buchholz, Dreyer, Florig and Summers, Lorentz covariance and wedge duality are all equivalent in these models. The same holds for $d=3$ if there is a mass gap. For massless fields in $d=3$, and for $d=2$ and arbitrary mass, CGMA does not imply Lorentz covariance of the field itself, but only of the maximal local net generated by the field.