About the general concept of 'motivation of the Lagrangian formalism'.
We have that textbook authors have converged on presenting Hamilton's stationary action as the motivation of the Lagrangian formalism, so in this answer, I will discuss Hamilton's stationary action.
That said: the approach to classical mechanics introduced by Joseph Louis Lagrange predates Hamilton's stationary action. If Hamilton's stationary action would have never been introduced the development of Lagrangian mechanics would not have been impacted; Lagrangian mechanics has the same capabilities either way. What Lagrange introduced was the systematic use of generalized coordinates, in combination with expressing the physics taking place in terms of energy. In that sense Lagrangian mechanics is independent of Hamilton's stationary action.
The following exposition consists of demonstrations for the purpose of gaining intuition.
To make this a general answer I will start with a short exposition of calculus of variations.
Differential calculus and calculus of variations both have a unit of operation. In differential calculus, the unit of operation is a pair of points, the line through those two points is tangent to the curve you are looking to solve for. In calculus of variations the unit of operation is a triplet of points.
Diagram 1 is an animated GIF that represents that unit of operation.
(The animated GIF is composited from screenshots of an interactive diagram.)
Diagram 1 - Unit of operation
In diagram 1 the dashed line is a parabola. This parabola represents height as a function of time, of an object that has been thrown upwards. A uniform force is acting in the downward direction. To obtain the simplest possible parabola the value of the (uniform) acceleration is set at 2 $m/s^2$
The outer points ($t_1$ and $t_3$) are treated as fixed; variation is executed by varying the position coordinate of the middle point.
In the diagram two adjacent time intervals are evaluated; $t_{1,2}$ and $t_{2,3}$
The sweet spot is the point where the change in velocity is such that the difference in kinetic energy matches the difference in potential energy.
(In this diagram the case of a uniform force is presented, which means that in this diagram the potential increases linear with height, hence in this diagram $\Delta E_p$ is a constant. In the general case, the force changes as a function of the position coordinate. Then the potential is not linear, and the value of $\Delta E_p$ changes as a function of the position coordinate.)
Diagram 2 - valid at any scale
Diagram 2 illustrates that the logic of this unit of operation is valid at any scale, down to infinitesimally short time intervals.
Diagram 3 - Concatenation
Diagram 3 illustrates the concatenation of the unit of operation over the entire domain of the function you are looking to solve for.
The analogy with the unit of operation of differential calculus: you concatenate the unit of operation over the domain of the function you are looking to solve for, and then you take the limit of infinitesimally small intervals.
An example of a source where this idea of the concatenation of the unit of operation is discussed is the book 'Calculus of Variations' by Gelfand and Fomin. Gelfand and Fomin demonstrate that the Euler-Lagrange equation can be derived using the functional derivative concept.
In your question, you describe that Lancaster and Blundell present expressions for 'average kinetic energy' and 'average potential energy', but diagrams 1, 2, and 3 illustrate that the logic can be stated such that it applies at infinitesimal scale. From down at infinitesimal scale the logic propagates out to the evaluation of the entire trajectory.
The next stage is to state explicitly where we obtain the criterion that must be satisfied.
To that end, we derive the Work-Energy theorem.
The starting point is $F=ma$. Both sides of $F-ma$ are integrated with respect to position, from starting point $s_0$ to endpoint $s$
$$ \int_{s_0}^s F \ ds = \int_{s_0}^s ma \ ds \tag{1.1} $$
The right-hand side $\int_{s_0}^s ma \ ds$ can be processed further. The reason for that: acceleration (as a function of time) and position (as a function of time) are not independent; acceleration is the second derivative of position. The work-energy theorem capitalizes on that relation.
In the course of the derivation the following relations will be used:
$$ v = \frac{ds}{dt} \quad \Leftrightarrow \quad ds = v \ dt \tag{1.2} $$
$$ a = \frac{dv}{dt} \quad \Leftrightarrow \quad dv = a \ dt \tag{1.3} $$
For the time being, I will not write the factor
$m$, it is a multiplicative factor that is just carried from step to step. In the final expression, I will include the factor
$m$ again.
The integral for acceleration from a starting point
$s_0$ to a final point
$s$
$$ \int_{s_0}^s a \ ds \tag{1.4} $$
Use (1.2) to change the differential from $ds$ to $dt$. Since the differential is changed the limits change accordingly.
$$ \int_{t_0}^t a \ v \ dt \tag{1.5} $$
Change the order:
$$ \int_{t_0}^t v \ a \ dt \tag{1.6} $$
Change of differential according to (1.3), with corresponding change of limits.
$$ \int_{v_0}^v v \ dv \tag{1.7} $$
So we have:
$$ \int_{s_0}^s a \ ds = \tfrac{1}{2}v^2 - \tfrac{1}{2}v_0^2 \tag{1.8} $$
Here (1.8) is worked out for the entities position, velocity, and acceleration. Note that this result generalizes. What counts is the pattern of of first and second derivatives.
Using the result (1.8) for (1.1) gives the Work-Energy theorem:
$$ \int_{s_0}^s F \ ds = \tfrac{1}{2}mv^2 - \tfrac{1}{2}mv_0^2 \tag{1.9} $$
We have that potential energy is defined as the negative of the integral of force over distance.
$$ E_p = -\int_{s_0}^s F \ ds \tag{1.10} $$
Hence:
$$ \Delta E_k = -\Delta E_p \tag{1.11} $$
This gives us our criterion explicitly. Over any interval, the amount of change of kinetic energy must match the amount of change of potential energy.
Below, between the horizontal lines, is a group of three statements. I present these three statements as a unit to emphasize the tight interconnection; while the statements are different mathematically, the physics content of these three is the same.
$$ \begin{array}{rcl}
F & = & ma \\
\int_{s_0}^s F \ ds & = & \tfrac{1}{2}mv^2 - \tfrac{1}{2}mv_0^2 \\
-\Delta E_p & = & \Delta E_k
\end{array} $$
The validity of (1.11) extends down to infinitesimal change:
$$ d(E_k) = d(-E_p) \tag{1.12} $$
Diagram 4 - Evaluation of area
Diagram 4 demonstrates an area evaluation property that we get for free. We get it for free in the sense that it doesn't require any additional assumption; it follows logically.
The two time intervals $t_{1,2}$ and $t_{2,3}$ are displayed in three of the four subpanels: upper-left, upper-right, and lower-left. The time intervals are set up to be of equal duration.
In the upper-right sub-panel: the height of each bar represents the value of the energy corresponding to that time interval. The number displayed in red is the summed area of the two red bars; the number displayed in green is the summed area of the two green bars.
Reminder: the energies that are displayed in this diagram are hypothetical energies. The variation sweep is hypothetical: at each position of the movable point, the diagram shows what the energies would be if the object would move along that particular trial trajectory.
Stating the two different kinds of change explicitly:
- Rate of change of actual energy as an object is moving along the true trajectory.
- Rate of change of hypothetical energy (as a function of variation sweep)
In motion along the true trajectory one form of energy is transformed into the other; the energies are counter-changing; $\Delta E_k = -\Delta E_p$
As illustrated in diagram 4: in sweeping out variation the (hypothetical) energies are co-changing.
Rate of change
In the case of Hamilton's stationary action, the variation sweep is variation of position coordinate. This means that the derivative of Hamilton's action with respect to variation boils down to taking the derivative with respect to position.
For emphasis: taking the derivative (of Hamilton's action) with respect to variation results in an expression for derivative with respect to position.
$$ \frac{d(E_k)}{ds} = \frac{d(E_p)}{ds} \tag{1.13} $$
The true energy satisfies (1.13) Therefore when the variation sweep is at the sweet spot the trial trajectory coincides with the true trajectory
Subtraction
In the hypothetical variation sweep the energies are co-changing, so to identify the point where they change at the same rate one energy is subtracted from the other. (By convention it is the potential energy that gets the minus sign.)
Given that we are subtracting the potential energy from the kinetic energy: in the lower-left subpanel the green area is displayed as the area below the coordinate's zero line. We can treat the green area as a signed area; area below the zero point of the coordinate system is counted as negative.
The lower-right sub-panel represents in blue the result of the subtraction.
The motion of the blue dot over the diagram represents how the value 'area($E_k-E_p$)' is responding to variation sweep.
The two time intervals $t_{1,2}$ and $t_{2,3}$ are set to be equal in duration. Because of that the time factor drops out of the evaluation. Hence the outcome of the evaluation is determined by the value of the energy only.
At the point where the variation hits the sweet spot the value of 'area($E_k-E_p$)' is stationary. That means: at the point where the variation hits the sweet spot the rate of change of summed red area matches the rate of change of summed green area.
Summarizing:
There is a point in the variation sweep where the rate of change of (hypothetical) kinetic energy matches the rate of change of (hypothetical) potential energy. If the time intervals are set to be of equal duration then it follows mathematically that the rate of change of summed red area matches the rate of change of summed green area.
Integral
The logic that is demonstrated in diagram 4 is valid at any scale, down to infinitesimal.
When concatenating units of operation along the length of the overall time the bars end up exactly adjacent; no overlap.
This explains why the integral of the quantity ($E_k - E_p$) has the property that it is stationary when the variation hits the sweet spot. The value of ($E_k - E_p$) already has that property at the infinitesimal scale, and from there it propagates to the value of the integral of ($E_k - E_p$).
The crucial point: the above reasoning does not require making any additional physical assumption. Hence: in any situation where the Work-Energy theorem holds good it follows that Hamilton's stationary action will hold good.
Narrowing down to what is necessary
The Euler-Lagrange equation is very compact. The Euler-Lagrange equation is an equation that has been narrowed down to what is necessary. You don't need more; the Euler-Lagrange equation is sufficient.
In other words: in retrospect, we can see that all elements that were removed during the process of deriving the Euler-Lagrange equation were unnecessary elements.
The standing convention is to state the functional as an integral, but at the end the integral is no longer there; the Euler-Lagrange equation is a differential equation.
The unit of operation of diagram 1 explains why the Euler-Lagrange equation is a differential equation: the unit of operation is valid at infinitesimal scale. The logic operates at the infinitesimal scale, and from there, it propagates out to the trajectory as a whole.
The integral that is present at the start is redundant and is accordingly removed in the process of deriving the Euler-Lagrange equation
The relation between variational calculus and differential calculus
To make the comparison I take a simple case that is a natural case for the application of variational calculus: the catenary.
As we know, mathematically the curve of a hanging chain is described by the hyperbolic cosine; the hyperbolic cosine gives the height as a function of the horizontal coordinate.
We have with the catenary that the force that is acting is perpendicular to the horizontal. And if the length of the catenary is changed then the change in the curve is change perpendicular to the horizontal.
Variational calculus pivots the direction of the differentiation. Variational calculus is a form of differential calculus where instead of differentiating with respect to $x$ (the horizontal coordinate), the differentiation is with respect to $y$, in such a way that you do end up with a function that gives the height of the catenary as a function of the horizontal coordinate.
The meaning of the 'stationary' criterion
I have noticed that authors of physics textbooks tend to have a cavalier attitude towards the question: "What is the meaning of the 'stationary' criterion?" The authors will acknowledge that the name 'stationary action' is better, but will then express that to actually use that name would be pedantic.
The purpose of the following discussion is to show that understanding the meaning of the 'stationary' criterion is key.
Diagram 5 - The blue curve has a point where it is stationary
In Diagram 5 the red curve and the green curve are both ascending functions.
Red: $f(x) = \frac{1}{2} x^2$
Green: $ g(x) = ln(x) $
Problem: at what x-coordinate do the functions $f(x)$ and $g(x)$ have the same slope?
Phrased differently: at what x-coordinate is the rate of change of $f(x)$ equal to the rate of change of $g(x)$?
The direct way to answer that question is to start by taking the derivative of each function, and then solve for the point where the two derivatives have the same value:
$$ \frac{f(x)}{dx} = \frac{g(x)}{dx} \tag{3.1} $$
$$ \frac{\tfrac{1}{2}x^2}{dx} = \frac{ln(x)}{dx} \tag{3.2} $$
$$ x = \frac{1}{x} \tag{3.3} $$
There is also a more convoluted way to address the problem:
I will refer to this more convoluted way as constructing a 'mathematical action':
Create a third function by subtracting the second function from the first function.
$$ h(x) = f(x) - g(x) \tag{3.4} $$
$$ h(x) = \tfrac{1}{2} x^2 - ln(x) \tag{3.5} $$
In diagram 5 the blue curve represents $h(x)$, the 'mathematical action'.
In diagram 4 the motion of the blue dot in response to variation sweep traces a curve, and that curve has a stationary point. That stationary point is the point where the rate of change of kinetic energy matches the rate of change of potential energy.
The interactive diagrams (that were the source of the frames of the animated GIF diagrams) are available on my own website. A link to my website is available on my stackexchange profile page.
