Start by determining potential anchor points of both boxes (left, right, top, or bottom edge). You can do a simple check for that, e.g.:
The left edge is a potential anchor point iff the other box' center x-coordinate is smaller than the current box' smaller x-coordinate.
The bottom edge is a potential anchor point iff the other box' center y-coordinate is greater than the current box' greater y-coordinate.
And so on...
After this, you have a list of candidates for both boxes. The actual anchor points can be calculated as the edge's center (( (edgeLeft + edgeRight) / 2, (edgeTop + edgeBottom) / 2)
). Now, you need to check, which combination results in the shortest line. Iterate every candidate:
for each candidate1 in candidates of box 1
for each candidate2 in candidates of box2
...
Now check if the combination results in an intersection. E.g. if candidate1
is the left edge and candidate2
has a greater x-coordinate, there is an intersection. Skip the combinations that result in an intersection.
From the other combinations, calculate the length of the line (actually the squared length: (candidate1.x-candidate2.x)^2 + (candidate1.y-candidate2.y)^2
). Remember the shortest one and you've got your optimal connection line.
Since you have only 4 anchor points per box, resulting in 16 possible combinations, you might as well skip the candidate search and test all edges for intersection.
You can also fuse the candidate search and intersection test (resulting in a candidate combination search). This lets you pick viable combinations, where it may be easier to test for intersections.