Summary: using 3 angles to represent a magnetic dipole's orientation is redundant because the rotation around the $z$-axis of the dipole does not change the magnetic field, there are only 2 DOFs for rotation for a magnetic dipole. After taking the linear approximation of the magnetic field at a certain orientation, it relates the change of orientation to the change of magnetic field through the Jacobian. The Jacobian always loses at least 1 rank, which is expected. But why the null space of the Jacobian is not a rotation around the $z$-axis? What I am trying to do is to use a least square method to solve for the magnet's orientation, which involves the Jacobian. Then I found this phenomenon that I couldn't understand.
I would like to be as rigorous as possible so I would write the math from the beginning. It makes the description long.
Detailed mathematics: The magnetic field of a dipole can be represented as: $$B(r,\mu)=\frac{\mu_0}{4\pi}\frac{|\mu|}{|r|^3} (3\hat{r}\hat{r}^T-I)\hat{\mu} $$
where $r$ is the vector from the magnet to the position where the field is measured, and $\mu$ is the magnetic dipole moment. The orientation of the magnetic dipole is embedded in $\hat{\mu}$, which can be represented as an intrinsic rotation by 3 Euler angles, e.g. Euler angles X-Y-X.
For simplicity, $r$ is considered known and fixed. Assuming the default dipole moment direction is attached to its body's z-axis, the magnetic field expression becomes a function of 3 Euler angles: $$B(\alpha,\beta,\gamma)=\frac{\mu_0}{4\pi}\frac{|\mu|}{|r|^3} (3\hat{r}\hat{r}^T-I)R_x(\alpha)R_y(\beta)R_x(\gamma)\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}\\=\frac{\mu_0}{4\pi}\frac{|\mu|}{|r|^3} (3\hat{r}\hat{r}^T-I)\begin{bmatrix} s_{\beta} c_{\gamma} \\ -c_{\alpha}s_{\gamma}-s_{\alpha}c_{\beta}c_{\gamma} \\ c_{\alpha}c_{\beta}c_{\gamma}-s_{\alpha}s_{\gamma} \end{bmatrix} \\ =('')\begin{bmatrix} s_{\beta} c_{\gamma} \\ -c_{\alpha}s_{\gamma}-s_{\alpha}c_{\beta}c_{\gamma} \\ c_{\alpha}c_{\beta}c_{\gamma}-s_{\alpha}s_{\gamma} \end{bmatrix}$$
The first half is abbreviated as $('')$ because it's full rank and doesn't affect the null space of $J_o$. Linearize the magnetic field at a certain orientation $(\overline{\alpha},\overline{\beta},\overline{\gamma})$: $$ B(\alpha,\beta,\gamma) \approx B(\overline{\alpha},\overline{\beta},\overline{\gamma})+J_o(\overline{\alpha},\overline{\beta},\overline{\gamma}) (\begin{bmatrix} \alpha\\ \beta \\ \gamma \end{bmatrix}-\begin{bmatrix} \overline{\alpha} \\ \overline{\beta} \\ \overline{\gamma} \end{bmatrix})$$
where $J_o$ is the jacobian of $B$ with respect to $(\alpha,\beta,\gamma)$:
$$J_o=('')\begin{bmatrix} 0 & c_{\beta}c_{\gamma} & -s_{\beta}s_{\gamma}\\ s_{\alpha}s_{\gamma}-c_{\alpha}c_{\beta}c_{\gamma} & s_{\alpha}s_{\beta}c_{\gamma} & s_{\alpha}c_{\beta}s_{\gamma}-c_{\alpha}c_{\gamma} \\ c_{\alpha}s_{\gamma}-s_{\alpha}c_{\beta}c_{\gamma} & -c_{\alpha}s_{\beta}c_{\gamma} & -s_{\alpha}c_{\gamma}-c_{\alpha}c_{\beta}c_{\gamma} \end{bmatrix}$$
The linear approximation also relates, at a certain configuration, how the change in orientation is mapped to the change in magnetic field: $$ B(\alpha,\beta,\gamma) - B(\overline{\alpha},\overline{\beta},\overline{\gamma}) \approx J_o(\overline{\alpha},\overline{\beta},\overline{\gamma}) (\begin{bmatrix} \alpha\\ \beta\\ \gamma \end{bmatrix}-\begin{bmatrix} \overline{\alpha} \\ \overline{\beta} \\ \overline{\gamma} \end{bmatrix})$$ $$\delta_B(\overline{\alpha},\overline{\beta},\overline{\gamma}) = J_o(\overline{\alpha},\overline{\beta},\overline{\gamma}) \delta_{angle}(\overline{\alpha},\overline{\beta},\overline{\gamma})$$
Using 3 angles to represent the orientation of the magnet is redundant because the rotation around the z-axis cannot be observed. It results in the rank deficiency of $J_o$. It also means there is a null space for $J_o$.
Presumably, the null space will represent a rotation around the z-axis. However, take $(\overline{\alpha},\overline{\beta},\overline{\gamma}) = (0,0,0)$ for example, in this case, the dipole is pointing upward.
$$J_o(0,0,0) = ('')\begin{bmatrix} 0 & 1 & 0\\ -1 & 0 & 1\\ 0 & 0 & 0 \end{bmatrix}$$
$$null(J_o) = \begin{bmatrix} 1\\ 0\\ -1 \end{bmatrix}$$
The null space is not representing a rotation around the z-axis. In fact, the rotation around the z-axis at $(\overline{\alpha},\overline{\beta},\overline{\gamma}) = (0,0,0)$ is $\begin{bmatrix} \alpha\\ \beta \\ \gamma \end{bmatrix}-\begin{bmatrix} \overline{\alpha} \\ \overline{\beta} \\ \overline{\gamma} \end{bmatrix} = \begin{bmatrix} \frac{\pi}{2}\\ \delta_{\beta}\\ -\frac{\pi}{2} \end{bmatrix}$, which is not a sub-space in $\mathbb{R}^3$, it's a line shifted from the origin.
Why this would happen?
EDIT
I found Prof. Barfoot's paper on linearizing the rotation: Pose estimation using linearized rotations and quaternion algebra. What concerns my question would be Section 3.1. One difference is that the paper uses extrinsic rotation. But it just changes the order of the rotation matrix and does not affect the discussion.
In the paper, it shows that linearizing the rotation matrix using Taylor expansion (26) is not correct, because the linearized "rotation matrix" is not a rotation matrix anymore (27). What should be used is derived as (31):
$$ C(\theta) = C(\overline{\theta}+\delta_{\theta}) = C(\delta_{\phi})C(\overline{\theta}) = C(S(\overline{\theta})\delta_{\theta})C(\overline{\theta}) $$
where $\theta = (\alpha, \beta, \gamma)$ in my question, and $C(\theta)$ is the rotation matrix obtained by rotating the frame around the Euler angles $\theta$ extrinsically. $S(\theta)$ is defined in (21): $$S(\theta) = S(\alpha, \beta, \gamma) = \begin{bmatrix} C_{x}(\gamma)C_{y}(\beta)\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} & C_{x}(\gamma)\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} & \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} \end{bmatrix} $$
After plugging in $\overline{\theta}=\begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$, $\delta_{\theta} = \begin{bmatrix} \frac{\pi}{2} \\ \delta_{\beta} \\ -\frac{\pi}{2} \end{bmatrix}$, which is a rotation around the z-axis:
$$C(\begin{bmatrix} \frac{\pi}{2} \\ \delta_{\beta} \\ -\frac{\pi}{2} \end{bmatrix}) = C(\begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \\ 0&0&0 \end{bmatrix}\begin{bmatrix} \frac{\pi}{2} \\ \delta_{\beta} \\ -\frac{\pi}{2} \end{bmatrix})I = C(\begin{bmatrix} 0 \\ \delta_{\beta} \\ 0 \end{bmatrix})$$
The equality does not hold anymore. And the linearized rotation matrix is not a rotation around the z-axis. It does not solve the problem.
