Let's assume $P_1=(x_1, y_1)$ and $P_2=(x_2, y_2)$ and $P_3=(x_3, y_3)$.
How to find the closest distance between $P_3$ and the line segment between $P_1$ and $P_2$?
I tried using the formula $\frac{area(P_1, P_2, P_3)}{distance(P_1, P_2)}$: $$\operatorname{distance}(P_1, P_2, P_3) = \frac{|(x_2-x_1)(y_1-y_3)-(x_1-x_3)(y_2-y_1)|}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}$$
Sadly it seems like it only works for infinite lines, not for line segments like in my case.
there is three area to take in count:
For $P1$ the distance is $|P1-B|$
For $P3$ the distance is $|P3-A|$
to summarise, it juste the distance between two point when the point P is not between the two perpendicular line passing trough A and B.
the hardest part is P2 :
since it between the two point, the distance to the segment is the distance to the point $C$ on the segment $AB$. We can know where the point C is located with a parameter $h$ (the doted line) because of how a segment is parameterised.
$$(A,B) = (hx_B + (1-h)x_A,hy_B + (1-h)y_A)$$
if C is on B then $h=1$ and $h=0$ if C is on A, in the exemple h might be equal to 0,75 or 0,8.
To find h we need to do :
$$h = \frac{<P_2-A, B-A>}{|B-A|^2}$$
this is a projection of the length of the vector $\overrightarrow{P_2A}$ on to $\overrightarrow{BA}$ we use a dot product for that, then devide it to the length of $\overrightarrow{BA}$ then normalise it ench the power of 2.
then to find the distance $|P2-C|$ we do :
$$|P_2 - (A +h*(B-A))|$$
to finish we can compute the 3 expression at the same time wich lead us to :
$$h = min(1,max(0,\frac{<P_2-A, B-A>}{|B-A|^2}))$$ $$d = |P - A - h*(B-A)|$$
The line segment can be parameterized as $$(x,y) = (tx_2+(1-t)x_3,ty_2+(1-t)y_3) $$ for $0\le t\le 1$. The distance (squared) from $P_1$ to any point $(x,y)$ on this line is then $$(tx_2 +(1-t)x_3 -x_1)^2 +(ty_2 +(1-t)y_3 -y_1 )^2 $$ This is a positive quadratic in $t$. To minimize this, first check the $t$ value ($t_0$) for the minimum, by differentiating and setting it equal to zero.
$\bullet $ If $t_0 \in [0,1]$, choose $t=t_0$.
$\bullet$ If $t_0 \gt 1$, choose $t=1$.
$\bullet$ If $t_0\lt 1$, choose $t=0$.
Let $d_{ij}$ and $d_{i(jk)}$ denote the distance between points $i,j$ and the distance between the point $i$ and the line $(jk)$ respectively.
Then the distance you are looking for is $$ \big[d_{3(12)}\big]_{d_{12}^2\ge\left|d_{31}^2-d_{32}^2\right|}+\big[\min(d_{31},d_{32})\big]_{d_{12}^2<\left|d_{31}^2-d_{32}^2\right|}. $$
