I am a high school student and I am a little confused in understanding the spring potential energy formula. When we calculate the spring potential energy of a spring attached to a wall at one end and we say that if we stretch the spring end say up to length $x$ we do work against the restoring conservative force hence we increase the potential energy of spring by $\frac{1}{2} k x^2$ where $k$ is the spring constant and $x$ is the displacement.
The problem is why do we only see the displacement of the end point of the spring? According to this formula this is the potential energy of only the end point because we applied force only at the end point, right?
I am comparing this with say a mass raised up to the height $h$ from the earth surface considering the reference at surface of earth. We see that the negative work done by the gravity is $-mgh$ which increases the potential energy of the mass. Why don't we see the displacement of each point and add them up to get the total energy of stretched or compressed spring?
If we do this my answer comes out to be $\frac{d}{L} \frac{1}{2} k x^2$ where $d$ is the point for which we have to see the displacement against its restoring conservative force with reference to the fixed end and $L$ is the which is equal i.e $kx$ for all the points as spring is massless so at any point the net force at any point should be 0 otherwise there would be infinite acceleration, see that at the end point i.e length $L$ the work done against conservative restoring force is $\frac{1}{2} k x^2$. So what's the problem with my understanding, why do we only see the displacement of end points or how much the total spring is stretched from its relaxed position?