Wigner's irreducible positive energy representations of the Poincare group are often used to give additional justifications for the Lagrangian quantization formalism of standard QFT. Here we study another more recent aspect. We explain in this paper modular concepts by which we are able to construct the local operator algebras for all standard positive energy representations directly i.e. without going through field coordinatizations. In this way the artificial emphasis on Lagrangian field coordinates is avoided from the very beginning. These new concepts allow to treat also those cases of ``exceptional'' Wigner representations associated with anyons and the famous Wigner ``spin tower''which have remained inaccessible to Lagrangian quantization. Together with the d=1+1 factorizing models (whose modular construction has been studied previously), they form an interesting family of theories with a rich vacuum-polarization structure (but no on shell real particle creation) to which the modular methods can be applied for their explicit construction. We explain and illustrate the algebraic strategy of this construction. We also comment on possibilities of formulating the Wigner theory in a setting of a noncommutative spacetime substrate. This is potentially interesting in connection with recent unitarity- and Lorentz invariance- preserving results of the special nonlocality caused by this kind of noncommutativity.