Example of similar matrices $A$ and $B$ such that products $AB$ and $BA$ are not similar

Question

I'm looking for the simplest possible example of square matrices $A$ and $B$ such that

• $A$ is similar to $B$,
• $AB$ is not similar to $BA$.

Such an example should exist, but I would like to find the "smallest" one. If either $A$ or $B$ is invertible, then $AB$ will be similar to $BA$, so one needs to look at singular matrices.

Answer 1

$A = \begin{bmatrix}1 & 0\\ 1 & 0\end{bmatrix},\ B = \begin{bmatrix}0 & 0\\1 & 1\end{bmatrix}$

Let $P = \begin{bmatrix}1 & -1\\-3 & 1\end{bmatrix}$. Then
$PAP^{-1} = \begin{bmatrix}1 & -1\\-3 & 1\end{bmatrix}\begin{bmatrix}1 & 0\\ 1 & 0\end{bmatrix}\begin{bmatrix}-1/2 & -1/2\\-3/2 & -1/2\end{bmatrix} = B$.

$AB = \begin{bmatrix}0 & 0\\0 & 0\end{bmatrix},\ BA = \begin{bmatrix}0 & 0\\2 & 0\end{bmatrix}$

$AB$ and $BA$ are not similar as $\text{rank}(AB) = 0$ but $\text{rank}(BA) = 1$.