Inscribing one regular polygon in another

Question

Say that one polygon $P$ is inscribed in another one $Q$, if $P$ is contained entirely in (the interior and boundary of) $Q$ and every vertex of $P$ lies on an edge of $Q$. It's clear that a regular $m$-gon can be inscribed in a regular $n$-gon if $m\mid n$ or $m=2n$. Well-known results (e.g. the Square Peg problem) imply that it is always possible when $m$ is 3 or 4, and it's not hard to use the Intermediate Value Theorem to see that it can happen whenever $m\mid 2n$. It's also not hard to see that a regular pentagon cannot be inscribed in an equilateral triangle, square, or regular hexagon.

Is it ever possible to inscribe a regular $m$-gon in a regular $n$-gon when $m>4$ and $n$ are relatively prime? If not, are there even any cases where it can happen that are not on the above list? Note that this question is not original; it was asked for example as P220 of The Playground in Math Horizons, vol. 16, no. 1, 2008 Sep. But no answer was ever published in Math Horizons and I have not been able to find any other reference.

Answer 1

Here is a proof for large enough coprime $m$ and $n$.

I use some basic properties of cyclotomic polynomials, be free to ask for details if needed. In particular, I use that the sum of roots of $\Phi_k$ equals $\mu(k)$ (Möbius function).

The main result which I use may be phrased as follows: if a sum of $O(1)$ roots of unity (not necessarily distinct) equals 0, this sum is a disjoint collection of zero subsums in each of which your roots of unity are some vertices of $O(1)$-gon.

This certainly must be known, but I do not know the reference and reproduce below a (short) proof, in slightly different form.

Let $N$ be a positive integer, Denote $\xi=e^{2\pi i/N}$. For an element $x\in \mathbb{Q}[\xi]$, i.e., $x=f(\xi)$ with $f(t)\in \mathbb{Q}[t]$, we denote the normalized trace $$T(x)=\frac1{\varphi(N)}\sum_{(k,N)=1}f(\xi^k),$$ where the summation is taken over all $\varphi(N)$ residues $k$ modulo $N$ which are coprime to $N$. Note that although the polynomial $f(t)$ for given $x$ is not uniquely defined, it is defined uniquely modulo the cyclotomic polynomial $\Phi_N(t)$, which is a minimal polynomial of the algebraic number $\xi$. Thus, $f(\xi^k)$ are well defined for all $k$ coprime to $N$ (the numbers $\xi^k$ are roots of $\Phi_N$, in other words, algebraic conjugates of $\xi$), and $T(x)$ is well-defined. Further we also use not polynomials, but Laurent polynomials $f(\xi)$, which does not abuse generality, since we may always replace negative powers of $\xi$ by non-negative powers of $\xi$ using $\xi^{-1}=\xi^{N-1}$.

If $f(t)=t^r$, then the numbers $f(\xi^k)$, where $k$ is coprime to $N$, are the roots of $\Phi_{N/(r,N)}$, each counted $\varphi(N)/\varphi(N/(r,N))$ times. Therefore $T(\xi^r)=\frac{\mu(N/(r,N))}{\varphi(N/(r,N))}$, for $r=N$ this gives $T(1)=1$. The idea is that the trace $T(\xi^r)$ is small when $N/(r,N)$ is large.

Consider the expression of the type $h(\xi)=\sum_{j\in A} c_j\xi^j$, where $A$ is a finite set of integers, $c_j\ne 0$ for $j\in A$ are integers.

Lemma. $h(\xi)=0$ if and only if $T(\xi^{-j}h(\xi))=0$ for all $j\in A$.

Proof. "Only if" part is trivial. For proving "if" part, by linearity of $T$ we see that these relations yield $T((\sum_j c_j \xi^{-j})h(\xi))=0$, but for Laurent polynomial $g(t):=(\sum_j c_j t^{-j})h(t)$ all values on the unit circle are non-negative real numbers, and $T(g(\xi))=0$ yields $|h(\xi)|^2=g(\xi)=0$.

Assume now that $h(\xi)=0$ in the above notations for $h(\xi)=\sum_{j\in A} c_j\xi^j$ and $\sum |c_j|=O(1)$ while $N$ may be arbitrarily large. Then we may suppose (by passing a subsequence) that $A=\{j_1,\ldots,j_R\}$ for fixed $R$, $c_{j_1},\ldots,c_{j_R}$ are also fixed. Also, we may suppose that, for every $1\leqslant a,b\leqslant R$, the normalized trace $T(\xi^{j_a-j_b})$ is either fixed or goes to 0 when $m,n$ become large.

Join two elements $j_1,j_2$ of $A$ by an edge if $T(\xi^{j_a-j_b})$ takes a fixed non-zero value. Let, say, $\{j_1,\ldots,j_r\}$ (where $r\leqslant R$) form a connected component. In this component, the ratio $\xi^{j_a-j_b}$ of every two numbers $\xi^{j_1},\ldots,\xi^{j_r}$ is a root of unity of bounded degree. I claim that $h_1(\xi)=\sum_{a=1}^r c_{j_a}\xi^{j_a}$ is zero for large $n,m$. For proving this, note that for $a\leqslant r$ the numbers $T(\xi^{-j_a}h_1(\xi))$ and $0=T(\xi^{-j_a}h(\xi))$ differ by $o(1)$. Therefore the former is 0 for large $N$, as it may take only finitely many different values. It remains to apply Lemma.

Now back to your $m$-gon and $n$-gon. Denote $N=mn$, $\xi=e^{2\pi i/N}$, $\omega=e^{2\pi i/m}=\xi^n$, $\theta=e^{2\pi i/n}=\xi^m$. Every side of the $n$-gon (which is assumed to be formed by powers of $\theta$) may contain at most 2 vertices of the $m$-gon (which is assumed to be formed by the numbers of the form $A\omega^j+B$ for some complex constants $A\ne 0$ and $B$). Let $A\omega^j+B$ belongs to the line between $\theta^{n_j}$ and $\theta^{n_j+1}$, $0\leqslant n_j<n$.

We may find a set $I$ of five distinct indices $j$ between 0 and 8 for which and all $n_j$, $j\in I$, are distinct. The condition that $A\omega^j+B$ belongs to the line between $\theta^{n_j}$ and $\theta^{n_j+1}$ implies that $p_j:=A\omega^j\theta^{-n_j}+B\theta^{-n_j}$ belongs to a fixed line between 1 and $\theta$ which has equation $\theta \bar{z}+z-(1+\theta)=0$ (it is not important which exact equation, though). Substituting $p_j$ to this equation, we get a linear combination of five numbers $\omega^j\theta^{-n_j}, \theta^{-n_j}, \omega^{-j}\theta^{n_j}, \theta^{n_j},1$ with fixed (and not all 0) coefficients. By fixed, I mean not dependent on $j$. Therefore, the determinant of the $5\times 5$ matrix formed by the above vectors of length 5 equals 0. This determinant $D$ is a linear combination of at most 120 roots of unity of degree $N$ with coefficients $\pm 1$. Note that the number of the form $\omega^J\theta^M$ with $0<|J|\leqslant 8$ and $M$ is arbitrary is not a root of unity of bounded degree if $m\to \infty$. Therefore, the above argument yields that if we formally expand the determinant and recollect the terms in the form $D=\sum_{J=-8}^8 \omega^J f_J(\theta)$, then each specific $f_j(\theta)$ must vanish (for large $m,n$). But, if $I=\{j_1<j_2<j_3<j_4<j_5\}$, then for $J=j_5-j_1$ the guy $f_J(\theta)$ is $\theta^{j_1-j_5}$ times a $3\times 3$ determinant with rows $(\theta^{n_j},\theta^{-n_j},1)$ for $j=2,3,4$. Its value is non-zero, since if you multiply the row $(\theta^{n_j},\theta^{-n_j},1)$ by $\theta^{n_j}$ for all $j=2,3,4$, you get a Vandermonde determinant for three distinct numbers $\theta^{n_2},\theta^{n_3},\theta^{n_4}$.