We study the set of infinite volume ground states of Kitaev's quantum double model on $\mathbb{Z}^2$ for an arbitrary finite abelian group $G$. It is known that these models have a unique frustration-free ground state. Here we drop the requirement of frustration freeness, and classify the full set of ground states. We show that the ground state space decomposes into $|G|^2$ different charged sectors, corresponding to the different types of abelian anyons (also known as superselection sectors). In particular, all pure ground states are equivalent to ground states that can be interpreted as describing a single excitation. Our proof proceeds by showing that each ground state can be obtained as the weak$^*$ limit of finite volume ground states of the quantum double model with suitable boundary terms. The boundary terms allow for states which represent a pair of excitations, with one excitation in the bulk and one pinned to the boundary, to be included in the ground state space.