Let's consider a simple pinhole camera, taking a picture of a uniformly illuminated wall.
Because this is a uniformly illuminated wall, each of these rays represents the same amount of light. There are nine rays hitting the film.
Now if we move the film back, we get a narrower field of view. This corresponds to the longer focal length of your 105mm lens.
The aperture is still the same size, and there is still the same amount of light (9 rays) passing through it, but because the field of view is smaller, less of it hits the film. Now there are only 5 rays hitting the film.
As you enlarge this hole, more light passes though, but you will need to add a lens to focus it if you want a clear picture. However, the basic principle still applies: a longer lens or a narrower field of view means you are capturing less of the light passing through the aperture on the film.
The f-number compensates for this effect, giving us a number which if equal, gathers equal light at the film regardless of focal length. The f-number is calculated by the focal length divided by the aperture's effective diameter1:
Somewhat confusingly, "f" is the focal length, and "N" is the f-number. But it's less confusing if you recall that f-numbers are usually written with a "f/" prefix, as in "f/4". What this means is "the aperture's effective diameter is the focal length (f) divided by 4: f/4".
1: effective because the physical size of the aperture may be modified by the optics around it, and exactly where in the optical path the aperture is.