Let $G$ be a reductive group (we can work on an algebraically closed field if needed) and let $L$ be a parabolic subgroup, i.e. the centralizer of a certain torus $T \subseteq G$.
Associated with this situation is the normalizer subgroup $N=N_G(L)$ of the Levi subgroup, which acts on $L$ by conjugation. Fix a finite subset $X \in L$. We will suppose that $L$ is minimal for $X$ in the sense that there exists no proper Levi $L' \subsetneq L$ containing $S$.
Now, suppose that we take $g \in G$ satisfying that $gxg^{-1} \in L$ for all $x \in X$.
Question: Does there exist an element $n \in N$ such that $nxn^{-1} = gxg^{-1}$ for all $x \in X$?
Example: Take $G = \text{GL}_4$ and consider the torus consisting of matrices of the form
\begin{pmatrix}\lambda_1 & 0 & 0 & 0 \\ 0 & \lambda_2 & 0 & 0 \\ 0 & 0 & \lambda_3 & 0 \\ 0 & 0 & 0 & \lambda_3 \end{pmatrix}
whose associated Levi subgroup is made of matrices of the form
$$ \begin{pmatrix}\star & 0 & 0 & 0 \\ 0 & \star & 0 & 0 \\ 0 & 0 & \star & \star \\ 0 & 0 & \star & \star \end{pmatrix}. $$
In particular, we can take $$ x = \begin{pmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & a_1 & a_2 \\ 0 & 0 & a_3 & a_4 \end{pmatrix} \in L, \qquad g = \begin{pmatrix} g_1 & g_2 & 0 & 0 \\ g_3 & g_4 & 0 & 0 \\ 0 & 0 & h_1 & h_2 \\ 0 & 0 & h_3 & h_4 \end{pmatrix} \in G. $$ We have that $gxg^{-1} \in L$ but in general $g \not\in N$ (it does not normalize elements with different eigenvalues in the first two columns). However, we can take $$ n = \begin{pmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & h_1 & h_2 \\ 0 & 0 & h_3 & h_4 \end{pmatrix} $$ that belongs to $N$ (actually, it's an element of $L$) and satisfies that $nxn^{-1} = gxg^{-1}$.
Remarks:
I think it is not hard to show that this fact is true in $\text{GL}_n$ (and probably in $\text{SL}_n$ and $\text{PGL}_n$ too). Sketch: $N = N_G(L)$ is a semidirect product of $L$ and a certain subgroup $W_L$ of the Weyl group $W = S_n$. Furthermore, without loss of generality, we can suppose that $L$ is a standard Levi subgroup, so $L$ is a product of general linear groups of lower order (the blocks of the matrix). Then, $gxg^{-1}$ must have the same block structure as $x$. Even though $g$ might not be an element of $L$ if some block of $x$ lies in the center, the same effect can be obtained by applying an 'intra-block conjugation' (an element of $L$) followed by an 'inter-block conjugation' (an element of $W_L$).
A naive attempt to prove it would be to consider $Y = \{g \in G \mid gXg^{-1} \subseteq L\}$ and analyze the action of $N$ on $Y$ by left multiplication. Then, the question reduces to show that on each $N$-orbit there is an element fixing $X$. However, I don't know how to prove this fact either.
As @LSpice pointed out in a comment, the hypothesis that $L$ is minimal is needed to guarantee that you cannot mix different blocks.
Extra: Is this true for general subgroups? More precisely, given $H < G$ and $x \in H$, is it true that $G \cdot x \cap H = N_G(H) \cdot x$? Here, $K \cdot x$ denotes the $K$-orbit of $x$ by conjugation, and $N_G(H)$ is the normalizer of $H$. I don't think this is true, but I couldn't find a simple counterexample.
Edit: New hypotheses added to consider only minimal Levi subgroups.