When speaking of symmetries of a polygon, there's an implicit assumption that a symmetry is a Euclidean motion. When speaking of manifolds, it's crucial to be explicit about the structure being preserved.
A sphere, for example, may be viewed as a Riemannian manifold with a round metric, in which case a "symmetry" is an isometry, and the group of isometries is isomorphic to $O(3)$, the group of real orthogonal $3 \times 3$ matrices under matrix multiplication. If we fix an orientation in addition, the symmetry group is isomorphic to $SO(3)$.
A sphere may also be viewed as the Riemann sphere, equipped with a conformal structure and an orientation (equivalently, since we're on a real surface, a holomorphic structure), in which case a "symmetry" is a Möbius transformation and the group of symmetries is isomorphic to $PGL(2, \mathbf{C})$, the complex projective general linear group of fractional linear transformations.
Instead, one might equip the sphere with:
- A smooth structure and the area form of the round metric, in which case a "symmetry" is an area-preserving diffeomorphism;
- A smooth structure, in which case a "symmetry" is a diffeomorphism;
- A topological structure, in which case a "symmetry" is a homeomorphism;
- The structure of an uncountable set, in which case a "symmetry" is a bijection.
In these examples (which by no means exhaust all possibilities), there are infinitely many symmetries, so counting symmetries is not a distinguishing invariant. These symmetry groups are, however, mutually non-isomorphic.
In the other direction, we might equip the sphere with a Riemannian metric having the same symmetry as a polyhedron (think of the solution set of $x^{4} + y^{4} + z^{4} = 1$ with the metric induced from Euclidean three-space, a "rounded cube"), which would have finite symmetry group. The extreme here is a generic Riemannian metric, which has precisely one symmetry, the identity map.