Rutherford's gold foil experiment taught us that the protons and neutrons fill only a tiny portion of the atom, which we call the nucleus. The size of the atom is therefore only defined by the electron shells (orbitals), which in turn depend on the nuclear charge.
Your intuition about more electrons leading to a larger atom is correct, but more protons actually lead to a smaller atom. Which effect dominates depends on whether or not an additional electron finds a place in an existing electron shell.
To gain some intuition as to how this works, it may be helpful to do a Gedankenexperiment wherein we build atoms by adding protons and electrons one at a time, neglecting the neutrons which are not important here.
Imagine you start with one proton and one electron; this is the $\ce{H}$ atom. Let's say its radius is 1 (the Bohr radius).
Add one proton to the nucleus to obtain the $\ce{He+}$ ion. Because the central charge of the nucleus has increased by a factor of two, but the number of electrons hasn't changed, the radius of this ion is halved compared to $\ce{H}$, i.e. 0.500.
Now add an electron to this system to obtain neutral $\ce{He}$. This second electron joins the first one in the 1s shell, expanding the atom only a bit to a radius of 0.567. This is just a sliver larger than $\ce{He+}$ and considerably smaller than $\ce{H}$.
If you were to move on to $\ce{Li+}$, the additional proton would contract the radius to about 2/3 of $\ce{He}$, but when you add a third electron to obtain $\ce{Li}$, it spawns a new 2s shell at a radius of 2.556, making the $\ce{Li}$ atom much larger than $\ce{H}.$
You can in principle continue this procedure, whereby each additional proton contracts the radius, and each additional electron expands it slightly. Whenever an electron shell is filled, adding another electron will cause a step-change in the atomic radius as can be seen in the jagged graph in one of the answers above.
Helium turns out to be the smallest atom on that graph, which is one of the reasons it was discovered in the sun before it was discovered on earth.