We study spontaneous symmetry breaking for field algebras on Minkowski space in the presence of a condition of geometric modular action (CGMA) proposed earlier as a selection criterion for vacuum states on general space-times. We show that any internal symmetry group must commute with the representation of the Poincare group (whose existence is assured by the CGMA) and each translation-invariant vector is also Poincare invariant. The subspace of these vectors can be centrally decomposed into pure invariant states and the CGMA holds in the resulting sectors. As positivity of the energy is not assumed, similar results may be expected to hold for other space--times.