The square vertices are $A(0,0), B(S, 0), C(S,S), D(0, S)$ where $S$ is known. The first circle has a radius given in terms of the the side length $S$. Let's call this radius $r_1$, then we are given

$ r_1 = \alpha S $

For some known $\alpha$. The reason I introduced $\alpha$ is to keep track of the relative ratio between $r_1$ and $S$.

So $r_1$ is known. If $r_2$ is the radius of the upper left circle, then its center is $ (r_2, S - r_2) $. From tangency with the lower circle, we know that

$ (r_1 - r_2)^2 + (S - r_2 - r_1)^2 = (r_1 + r_2)^2 $

So that

$ - 2 r_1 r_2 + S^2 + r_2^2 + r_1^2 - 2 S r_1 - 2 S r_2 = 0 $

$ r_2^2 - 2 r_2 \left( S + r_1 \right) + \left( S - r_1 \right)^2 = 0 $

The discriminant is

$ 4 \left( S + r_1 \right)^2 - 4 \left( S - r_1 \right)^2 = 16 S r_1 $

Hence, the solution of this quadratic equation is

$ r_2 = \dfrac{1}{2} \left( 2 (S + r_1) \pm 4 \sqrt{ S r_1 } \right) $

which simplifies to,

$ r_2 = S + r_1 - 2 \sqrt{ S r_1 } = (\sqrt{S} - \sqrt{r_1})^2 $

Next, we we want to identify the ellipse that is tangent to the top, right, and bottom sides as well as the two circles on the left. Let the center of this ellipse be $G = (G_x, G_y)$, then necessarily $G_y = \dfrac{S}{2}$. The equation of this ellipse is

$ (r - G)^T Q (r - G) = 1 $

The gradient vector of the ellipse is given by

$ \nabla = 2 Q (r - G) $

At the right tangency point $P_1$, we have

$ Q (P_1 - G) = k \mathbf{i} $

where $k \gt 0 $ is some positive constant, and $\mathbf{i} = [1, 0]^T $.

Solving for $P_1 - G$ , we obtain

$ P_1 - G = k Q^{-1} \mathbf{i} $

Substituting this into the ellipse equation above, gives

$ k = \dfrac{1}{\sqrt{ \mathbf{i}^T Q^{-1} \mathbf{i} } } $

Pre-multiplying the above expression for $P_1 - G$ by $\mathbf{i}^T $ gives

$ \mathbf{i}^T {P_1} - \mathbf{i}^T G = k \ \mathbf{i}^T Q^{-1} \mathbf{i} = \sqrt{ \mathbf{i}^T Q^{-1} \mathbf{i} } $

Since $\mathbf{i}^T {P_1} = S $ and $\mathbf{i}^T G = G_x $ , then we now have

$ ( S - G_x )^2 = Q^{-1}_{11} \tag{1} $

Now considering the top the tangency point $P_2$, we have

$ Q (P_2 - G) = k' \ \mathbf{j} $

where $k' \gt 0 $. Similar to the above analysis, we end up with

$ \mathbf{j}^T P_2 - \mathbf{j}^T G = \sqrt{ \mathbf{j}^T Q^{-1} \mathbf{j} } $

But $ \mathbf{j}^T P_2 = S $ and $ \mathbf{j}^T G = G_y = \dfrac{S}{2} $; therefore,

$ \left( \dfrac{S}{2} \right)^2 = Q^{-1}_{22} \tag{2}$

Turning now to tagency with the two circles. Introduce the tangency point $T_1$ with the lower circle and $T_2$ with the top circle. Parametrically, we have

$T_1 = (r_1, r_1) + r_1 (\cos t_1, \sin t_1 ) $

$T_2 = (r_2, S - r_2) + r_2 (\cos t_2, \sin t_2) $

The first requirement for both $T_1$ and $T_2$ is that they lie on the ellipse, therefore, we immediately have two additional equations, namely,

$ (T_1 - G)^T Q (T_1 - G) = 1 \tag{3} $

$ (T_2 - G)^T Q (T_2 - G) = 1 \tag{4} $

Secondly, the gradients of the circle and the ellipse at $T_1, T_2$ have to be parallel, and this gives the final two equations

$ (T_1 - (r_1, r_1) ) \times Q (T_1 - G) = \mathbf{0} \tag{5} $

$ (T_2 - (S - r_2, r_2) \times Q (T_2 - G) = \mathbf{0} \tag{6} $

These are the $(6)$ equations that govern the six unknowns: $t_1, t_2, a, b, \theta, G_x $. Note that

$ Q = R D R^T $

where $D = \begin{bmatrix} \dfrac{1}{a^2} && 0 \\ 0 && \dfrac{1}{b^2} \end{bmatrix} $ and $ R = \begin{bmatrix} \cos \theta && - \sin \theta \\ \sin \theta && \cos \theta \end{bmatrix} $

These $6$ equations can be solved numerically using the Newton-Raphson multivariate method. The Newton-Raphson recurrence converged to a solution within few iterations. The number of iterations necessary for convergence depends on the initial guess of the unknowns. The figure attached to the question was obtained for $r_1 = 0.29 S $.

The square vertices are $A(0,0), B(S, 0), C(S,S), D(0, S)$ where $S$ is known. The first circle has a radius given in terms of the the side length $S$. Let's call this radius $r_1$, then we are given

$ r_1 = \alpha S $

For some known $\alpha$. The reason I introduced $\alpha$ is to keep track of the relative ratio between $r_1$ and $S$.

So $r_1$ is known. If $r_2$ is the radius of the upper left circle, then its center is $ (r_2, S - r_2) $. From tangency with the lower circle, we know that

$ (r_1 - r_2)^2 + (S - r_2 - r_1)^2 = (r_1 + r_2)^2 $

So that

$ - 2 r_1 r_2 + S^2 + r_2^2 + r_1^2 - 2 S r_1 - 2 S r_2 = 0 $

$ r_2^2 - 2 r_2 \left( S + r_1 \right) + \left( S - r_1 \right)^2 = 0 $

The discriminant is

$ 4 \left( S + r_1 \right)^2 - 4 \left( S - r_1 \right)^2 = 16 S r_1 $

Hence, the solution of this quadratic equation is

$ r_2 = \dfrac{1}{2} \left( 2 (S + r_1) \pm 4 \sqrt{ S r_1 } \right) $

which simplifies to,

$ r_2 = S + r_1 - 2 \sqrt{ S r_1 } = (\sqrt{S} - \sqrt{r_1})^2 $

Next, we we want to identify the ellipse that is tangent to the top, right, and bottom sides as well as the two circles on the left. Let the center of this ellipse be $G = (G_x, G_y)$, then necessarily $G_y = \dfrac{S}{2}$. The equation of this ellipse is

$ (r - G)^T Q (r - G) = 1 $

The gradient vector of the ellipse is given by

$ \nabla = 2 Q (r - G) $

At the right tangency point $P_1$, we have

$ Q (P_1 - G) = k \mathbf{i} $

where $k \gt 0 $ is some positive constant, and $\mathbf{i} = [1, 0]^T $.

Solving for $P_1 - G$ , we obtain

$ P_1 - G = k Q^{-1} \mathbf{i} $

Substituting this into the ellipse equation above, gives

$ k = \dfrac{1}{\sqrt{ \mathbf{i}^T Q^{-1} \mathbf{i} } } $

Pre-multiplying the above expression for $P_1 - G$ by $\mathbf{i}^T $ gives

$ \mathbf{i}^T {P_1} - \mathbf{i}^T G = k \ \mathbf{i}^T Q^{-1} \mathbf{i} = \sqrt{ \mathbf{i}^T Q^{-1} \mathbf{i} } $

Since $\mathbf{i}^T {P_1} = S $ and $\mathbf{i}^T G = G_x $ , then we now have

$ ( S - G_x )^2 = Q^{-1}_{11} \tag{1} $

Now considering the top the tangency point $P_2$, we have

$ Q (P_2 - G) = k' \ \mathbf{j} $

where $k' \gt 0 $. Similar to the above analysis, we end up with

$ \mathbf{j}^T P_2 - \mathbf{j}^T G = \sqrt{ \mathbf{j}^T Q^{-1} \mathbf{j} } $

But $ \mathbf{j}^T P_2 = S $ and $ \mathbf{j}^T G = G_y = \dfrac{S}{2} $; therefore,

$ \left( \dfrac{S}{2} \right)^2 = Q^{-1}_{22} \tag{2}$

Turning now to tangency with the two circles. Introduce the tangency point $T_1$ with the lower circle and $T_2$ with the top circle. Parametrically, we have

$T_1 = (r_1, r_1) + r_1 (\cos t_1, \sin t_1 ) $

$T_2 = (r_2, S - r_2) + r_2 (\cos t_2, \sin t_2) $

The first requirement for both $T_1$ and $T_2$ is that they lie on the ellipse, therefore, we immediately have two additional equations, namely,

$ (T_1 - G)^T Q (T_1 - G) = 1 \tag{3} $

$ (T_2 - G)^T Q (T_2 - G) = 1 \tag{4} $

Secondly, the gradients of the circle and the ellipse at $T_1, T_2$ have to be parallel, and this gives the final two equations

$ (T_1 - (r_1, r_1) ) \times Q (T_1 - G) = \mathbf{0} \tag{5} $

$ (T_2 - (S - r_2, r_2) \times Q (T_2 - G) = \mathbf{0} \tag{6} $

These are the $(6)$ equations that govern the six unknowns: $t_1, t_2, a, b, \theta, G_x $. Note that

$ Q = R D R^T $

where $D = \begin{bmatrix} \dfrac{1}{a^2} && 0 \\ 0 && \dfrac{1}{b^2} \end{bmatrix} $ and $ R = \begin{bmatrix} \cos \theta && - \sin \theta \\ \sin \theta && \cos \theta \end{bmatrix} $

These $6$ equations can be solved numerically using the Newton-Raphson multivariate method. The Newton-Raphson recurrence converged to a solution within few iterations. The number of iterations necessary for convergence depends on the initial guess of the unknowns. The figure attached to the question was obtained for $r_1 = 0.29 S $.