How to get correct axis for scaling from transformed space?

Question

I have scaled and non-scaled representation of model so that I don't destroy verticis information if model will be scaled to plane, for example. Now I trying implement scaling on model elements (face, edge), I set up axis of scale tool on dislplay (scaled) model, but I stucked with produce inverse transformed axis for correct scaling in non-scaled model version. What I mean:

Here not rotated not scaled cube.

Then I rotate and scale by global X axis (1, 0, 0), for rotated model this will be X Y axis. Faded axis is original axis, more bright is axis that scale tool produce and with which user interact.

This is how scale tool axis looks after just straight full inverse transform, under "straight" I meen just muliplying on inverted matrixs in right order. That actualy expected result because they orthognal in scaled space, but not correct result for perform correct scaling.

I have been trying to find an answer for several weeks now, trying many combinations of transformation order, after I realize that I am just starting to randomly place the matrix I was looking at options maybe I could somehow construct or line up vector of axis but not succesefull.

Here I want emphasize that inverted axis not necessary should be line up with origin axis (1, 0, 0) (0, 1, 0) (0, 0, 1), they can point in any direction, case that I take above just for simplicity. I think that my knowlend in linear algebra just not enough and it's actually possible tusk even if it's not one line matrix multiplication in some tricky order. Or maybe not:).

Edit: For clarification because maybe unlear what I trying to do

In post, as I said that I seting up axis in global space, in wich model full transformed and in wich user interact with scale tool, because scaling direction must match how user see axis I try transform this axis to how they will look in model space (non-scaled not-rotated). For this I try apply inverse transform and as I said was trying this in different variations but not succes. I attached a screenshots for showing how looks my data (initial, constructed, and inverted axis).

After DMGregory answer I checked what his said and I find out that:

1. this still not produce orthogonal axis

2. I cant align axis after transform because this solution can steel produce incorect result for X and Y axis also in some case (attached additional screenshots in comment)

Edit 2:

I use row-major matrix so multiplication will be from left to right

I do this for building axis in global space for model face. For case with multiple selected face I just return identity matrix for now.

m3x3
SetAxisForTool(work_model *Model, element_id_buffer *Selected, u32 ElementTarget)
{
     m3x3 Result = {};
     ...
     switch (ElementTarget)
     {
         ...
         case ModelTargetElement_Face:
         {
             model_face *Face = Model->Data.Faces + ElementID;
             face_vertex Vertex = GetFaceVertex(Model, Face);

             v3 OriginZ = V3(0, 0, 1);
             Result.Z = GetPlaneAvgNormal(Vertex);

             f32 ZDotResult = Abs(Dot(Result.Z, OriginZ));
             if (ZDotResult == 1.0f)
                 OriginZ = V3(-1, 0, 0);

                Result.Y = Normalize(Cross(OriginZ, Result.Z));
                Result.X = Cross(Result.Y, Result.Z);
         }
     }

    return Result;
}

Then after I get scale factor on axis that user see I try to process this transform like this. (For finding scale factor I just find difference between points on ray and gets length of this vector).

Model->Axis is orientation of model in global space, Tool->Axis is axis that I get from code above. IsGlobalSpace tell that I map scaling along original axis ((1,0,0)(0,1,0)(0,0,1)) to model space, so it's no need in invering scale this works in all case in expected way. Without scaling another branch also work in expected way.

void ApplyScale(work_model *Model, scale_tools *Tool, element_id_buffer 
 *UniqIndeces,  model_target_element TargetElement, b32 IsGlobalSpace)

{
    ...

    u32 AxisID = GetIntAxisID(Tool->InteractAxis);
    v3 ScaleV = {};
    ScaleV.E[AxisID] = 1.0f * ScaleFactor;
    ScaleV += V3(1.0f);

    switch (TargetElement)
    {
       case ModelTargetElement_Model:
       {
            m4x4 MAxis = ToM4x4(Model->Axis);
            m4x4 InvRot = Transpose(MAxis);

            m4x4 ResultScale;
            if (IsGlobalSpace)
            {
                ResultScale = ScaleMat(ScaleV);
                ResultScale = MAxis * ResultScale * InvRot;
            }
            else
            {
                ResultScale = ScaleMat(ScaleV);
            }

            Model->ScaleMat = Model->ScaleMat * ResultScale;
            m4x4 Transform = Model->ScaleMat * MAxis;

            for (u32 Index = 0;
                 Index < Model->Data.VertexCount;
                 ++Index)
            {
                v3 VCache = VerticesCache[Index];
                DisplayVertices[Index] = VCache * Transform;
            }
       } break;

       case ModelTargetElement_Face:
       case ModelTargetElement_Edge:
       {
            m4x4 MAxis = ToM4x4(Model->Axis);
            m4x4 InvRot = Transpose(MAxis);
            m4x4 ModelTrans = Model->ScaleMat * MAxis;

            m3x3 S = ToM3x3(Model->ScaleMat);
            m4x4 InvScale = ToM4x4(Inverse(S));

            v3 ScaleOrigin = Tool->P - Model->Offset;
            ScaleOrigin = ScaleOrigin * InvRot * InvScale;

            m4x4 ResultScale;
            if (IsGlobalSpace)
            {
                ResultScale = ScaleMat(ScaleV);
                ResultScale = MAxis * ResultScale * InvRot;
            }
            else
            {
                m4x4 ScaleAxis = ToM4x4(Tool->Axis) * InvRot * InvScale;
                ScaleAxis.Row0.xyz = Normalize(ScaleAxis.Row0.xyz);
                ScaleAxis.Row1.xyz = Normalize(ScaleAxis.Row1.xyz);
                ScaleAxis.Row2.xyz = Normalize(ScaleAxis.Row2.xyz);
                m4x4 InvScaleAxis = Transpose(ScaleAxis);

                ResultScale = InvScaleAxis * ScaleMat(ScaleV) * ScaleAxis;
            }

           //m4x4 Transform = TranslateMat(-ScaleOrigin) * ResultScale * TranslateMat(ScaleOrigin);

            for (u32 Index = 0;
                Index < UniqIndeces->Count;
                ++Index)
            {
                u32 VertexIndex = UniqIndeces->Elements[Index];
                v3 VCache = VerticesCache[VertexIndex];
            
                VCache -= ScaleOrigin;
                VCache = VCache * ResultScale;
                VCache += ScaleOrigin;

                VerticesCache[VertexIndex] = VCache;
                DisplayVertices[VertexIndex] = VCache * ModelTrans;
            }
        } break;
    }

    Model->AABB = ComputeMeshAABB(Model->Data.Vertices, Model->Data.VertexCount);
}

Answer 1

Here I'm going to assume your goal is to keep the red and green vectors in the plane of the face in both the original and transformed coordinate spaces, and keep the blue vector perpendicular to the face in both spaces. If that's not accurate, please edit the question to be more precise about what vectors you want here.

If you have a matrix \$M\$ that takes vertex positions from the original space to the transformed space...

$$\vec v_{\text{transformed}} = M \vec v_{\text{original}}$$

and its inverse matrix takes those transformed vertices back to the original space...

$$\vec v_{\text{original}} = M^{-1}\vec v_{\text{transformed}}$$

Then you can also use this same matrix to transform tangent-space vectors along the surface of your models, or offsets between points on your model. Just remember to set the w component of these vectors to 0, so they're not affected by the translation component of the matrix, unlike position vectors which should have w = 1.

To transform vectors perpendicular to the faces of your model (ie. face normals or vertex normals), you need to use the Inverse Transpose of that matrix:

$$\hat n_{\text{transformed}} = (M^{-1})^T \hat n_{\text{original}}$$

$$\hat n_{\text{original}} = M^T \hat n_{\text{transformed}}$$

So in this example you'd use \$M^{-1}\$ and \$M\$ to transform your red and green vectors back and forth between coordinate spaces, and \$M^T\$ and \$(M^{-1})^T\$ to transform your blue vector back and forth.

(Note that the length of the vectors can be affected by this transformation, so if you need them to be unit vectors, be sure to re-normalize them after transforming)