The largest equilateral triangle circumscribing a given triangle

Question

Seven years ago, one of my many contributions to the March 2010 edition of Erich Friedman's Math Magic was a packing of eight circles of unit diameter and one equilateral triangle of unit side length into as small a circle as I could manage.

To minimise the bounding circle's radius $r$, I had to determine the largest equilateral triangle externally tangent to the three circles adjacent to it, then numerically adjust $r$ until said triangle had side lengths $1$. This is not a trivial problem, and back then I did trial and error in GeoGebra. ($2r=3.4133707107\dots$)

It is easy to see that the circles can be shrunk and the equilateral triangle enlarged about its centre while preserving tangencies, so the problem is equivalent to finding the largest equilateral triangle $\Delta^*$ where each edge is incident to one of $A,B,C$ where $A,B,C$ are the circle centres. The following construction doesn't count since one edge does not meet $\triangle ABC$:

Now, having reinvigorated my interest in triangles, I believe I have found a construction for $\Delta^*$ for arbitrary $A,B,C$, and am asking here for a more traditional proof of its optimality. My construction proceeds as follows:

1. Construct the first isogonic centre/Fermat point of the triangle, $X_{13}$ in the Encyclopedia of Triangle Centers (ETC). That is, construct outward equilateral triangles $A'BC, AB'C, ABC'$ on the sides of $\triangle ABC$, then $X_{13}$ is the concurrence of $AA',BB',CC'$.
2. $\Delta^*$ is (conjecturally) the antipedal triangle of $X_{13}$.$^\dagger$ That is, $\Delta^*$ is the triangle formed by perpendiculars at $A,B,C$ of $AX_{13},BX_{13},CX_{13}$ respectively; it is guaranteed to be equilateral because the lines meeting at $X_{13}$ are evenly spaced around it.

The centroid of the triangle thus constructed is $X_{5463}$ in the ETC, the reflection of $X_{13}$ in the centroid of $\triangle ABC$, and its area is $\frac{a^2+b^2+c^2}{2\sqrt3}+2\operatorname{area}(\triangle ABC)$ where $a,b,c$ are the side lengths. (Alternatively, the area is $2\left(1+\frac{\cot\omega}{\sqrt3}\right)\operatorname{area}(\triangle ABC)$ where $\omega$ is the Brocard angle.)

How can I prove that my construction actually produces $\Delta^*$, the largest circumscribing equilateral triangle?

---

$^\dagger$Without loss of generality, assume $AB$ is the largest side, and let $P,Q$ be the midpoints of $AC',BC'$ respectively. If $\overline{CP}$ or $\overline{CQ}$ is less than $\frac12\overline{AB}$, my construction will not produce an equilateral triangle actually incident to all of $A,B,C$. In that case, $\Delta^*$ has a side collinear with the triangle's shortest side $s$ and one vertex coincident with the original triangle's smaller angle incident to $s$. If $A,B,C$ are circle centres in a packing problem, this means that a larger equilateral triangle can be drawn by introducing a point contact.

Answer 1

I'll use $\triangle PQR$ instead of $\triangle ABC$, to avoid some notational confusion.

---

First, a little prep work.

Given $\triangle PQR$, we erect on (directed) segment $\overline{PQ}$ an equilateral triangle $\triangle PQR'$ with a clockwise orientation. Similarly, we erect clockwise equilaterals $\triangle QRP'$ and $\triangle RPQ'$. As it happens, the lines $\overleftrightarrow{PP'}$, $\overleftrightarrow{QQ'}$, $\overleftrightarrow{RR'}$ meet at a common point, and do so symmetrically. When $\triangle PQR$ itself has a counter-clockwise orientation, the three equilaterals are external to it, and the three lines meet at the first isogonic center (Kimberling center $X_{13}$); when $\triangle PQR$ has a clockwise orientation, the equilaterals overlap its interior, and the common point is the second isogonic center ($X_{14}$).

In any case, we find that any triangle can be positioned such that each of its vertices lies on one of three concurrent, symmetrically-arranged lines. Taking the common point to be the origin, and one of the lines to be the $x$-axis, we can coordinatize $\triangle PQR$ thusly (abusing notation so that $\operatorname{cis}\theta := (\cos\theta, \sin\theta)$) as $$P := p \operatorname{cis} 0 \qquad Q := q \operatorname{cis}\tfrac23\pi\qquad R := r \operatorname{cis}(-\tfrac23\pi) \tag{1}$$ where we may assume $q$ and $r$ are non-negative (and not simultaneously zero). One can show that the $x$-intercept of $\overline{QR}$ is $-qr/(q+r)$; consequently, $\triangle PQR$'s orientation depends upon $p$'s relation to that value, and we can write $$\text{The origin is}\;\triangle PQR\text{'s}\; \left\{\begin{array}{c}\text{first} \\ \text{second} \\ \text{(either)} \end{array}\right\}\; \text{isogonic center if} \;\; p q + q r +r p \left\{\begin{array}{c} > \\ < \\ = \end{array}\right\} 0 \tag{2}$$

---

Now to the topic at hand.

The sides of an equilateral triangle circumscribing $\triangle PQR$ are lines through $P$, $Q$, $R$ with symmetrically-arranged normal vectors, say, $$u := \operatorname{cis}\theta \qquad v := \operatorname{cis}\left(\theta+\tfrac23\pi\right) \qquad w := \operatorname{cis}\left(\theta-\tfrac23\pi\right) \tag{3}$$ Let $T_\theta$ be the resulting equilateral triangle. I'll forego giving its vertices. The important thing is to compare the sizes of these triangles across all $\theta$, which we can do by computing the areas:

$$|T_\theta| = \frac{1}{\sqrt{3}}\;\left(\,p + q + r\,\right)^2\,\cos^2\theta \tag{4}$$

Clearly, $|T_\theta|$ is maximized with $\theta = 0$ (or $\pi$), giving a maximal area

$$|T_0| = \frac1{\sqrt{3}} (p+q+r)^2 \tag{$\star$}$$

when $u$, $v$, $w$ are parallel to $\overrightarrow{OA}$, $\overrightarrow{OB}$, $\overrightarrow{OC}$. When $O$ is $\triangle PQR$'s first isogonic center (see $(2)$), then $T_0$ matches OP's construction.

Now, while the $T_\theta$ triangles form an infinite family of equilaterals with side-lines containing the vertices of $\triangle PQR$, theirs is not the only such family. We get another by changing the signs in $(3)$; that is, by exchanging the roles of $v$ and $w$. The corresponding triangles, $T_\theta'$, have areas given by

$$|T^\prime_\theta| = \frac{1}{\sqrt{3}}\,\left(\,p \cos\theta + q \cos\left(\theta+\tfrac23\pi\right) + r \cos\left(\theta-\tfrac23\pi\right) \,\right)^2 \tag{5}$$

A quick derivative tells us that the critical values of $(4)$ occur for $$\cot\theta = \frac{\sqrt{3}\,(q-r)}{2p-q-r} \quad\text{or}\quad \tan\theta = \frac{\sqrt{3}\,(r-q)}{2p-q-r} \tag{6}$$ The former corresponds to an area of zero; the latter is maximizing, and we have

$$\phi := \tan^{-1}\frac{\sqrt{3}\,(r-q)}{2p-q-r} \quad\to\quad |T^\prime_\phi| = \frac1{\sqrt{3}} \left(p^2+q^2+r^2-p q-q r-r p\right) \tag{$\star\star$}$$

Comparing $(\star)$ to $(\star\star)$, we have $$|T_0| - |T_\phi^\prime| = \sqrt{3} \left(p q + q r + r p\right) \tag{7}$$

which hearkens back to $(2)$. So, $T_0$ is maximal when $O$ is the first isogonic center of $\triangle PQR$; otherwise, $T^\prime_\phi$ is.

Importantly, the reader can verify that, at the key angle $\theta = \phi$ from $(\star\star)$, the lines through $P$, $Q$, $R$ with the direction vectors $u$, $v$, $w$ (the last two with their signs exchanged) meet at a point; specifically, they meet at $\triangle PQR$'s "other" isogonic center. This tells us that $T^\prime_\phi$ is actually the equilateral triangle obtained from OP's construction relative to that "other" center. Since, by $(7)$ and $(2)$, equilateral $T^\prime_\phi$ maximizes all triangles precisely when the "other" center is the first isogonic center, we have shown that

OP's construction relative to the first isogonic center is always maximal.

$\square$

---

Here are a couple of animations, showing differently-oriented $\triangle PQR$ (counter-clockwise vs clockwise); the origin (unmarked black dot) is either the first or second isogonic center, respectively. Triangles $T_\theta$ are green, while $T^\prime_\theta$ are light blue.

These images highlight that, while the various equilateral triangles have side-lines passing through the vertices of $\triangle PQR$, not all have $\triangle PQR$ in their interiors; hence, they are not all "circumscribing" in the traditional sense.