According to Wikipedia,
A spectrochemical series is a list of ligands ordered on ligand strength and a list of metal ions based on oxidation number, group and its identity.
$$\ce{I- < Br- < SCN- < Cl- < S^2- < F- < OH- < C2O4^2- < H2O < NCS- < edta^4- < NH3 < en < CN- < CO}$$
Ligands arranged on the left end of this spectrochemical series are generally regarded as weaker ligands and cannot cause forcible pairing of electrons within 3d level and thus form outer orbital octahedral complexes. On the other hand ligands lying at the right end are stronger ligands and form inner orbital octahedral complexes after forcible pairing of electrons within 3d level.
According to above passage $\ce{CO}$ is a stronger ligand than $\ce{NH3}$. So, carbonyl complexes are more stable than ammine complexes.
Now, there are many other factors which determine the stability of complex:
- Nature of central metal atom
- Nature of ligand
- Crystal field effects
Nature of ligand has further classification:
- Basic character of ligands
- Charge on ligands
- Chelate effects
Classification source: see here and here
Basic character of ligands states that the more basic a ligand is, the greater the ease of ligation with the central metal atom is and the more stable the complex is. Now, $\ce{NH3}$ is more or less a basic compound and $\ce{CO}$ is a neutral molecule.
So, according to above assumption, $\ce{NH3}$ is a stronger ligand than $\ce{CO}$. So, ammine complexes are more stable than complex complexes.
So, which one is the stronger ligand: $\ce{CO}$ or $\ce{NH3}$? or which complexes are stable: ammine complexes or carbonyl complexes?