How to get the transform amount from a screen vector and a direction vector?

Question

I am working on the move tool in a 3D modeling software. I need to know how much to move when the user drags an axis of the gizmo (for example along the x axis).

I have the 2D vector describing how much the mouse moved on the screen MouseDelta, and the vector corresponding to the selected axis AxisVector.

I would like to know the best way to compute the amount of extrusion I should apply.

Currently I try to project the AxisVector in screen space, project MouseDelta on it and get the ratio between the size of the AxisVectorOnScreen and the computed projection. Then I have to clamp the result and multiply it by a factor. Here is a pseudo code:

Vector3 AxisVectorOnScreen = GetViewProjMatrix().TransformVector(AxisVector);
Vector2 AxisVectorOnScreen2D(AxisVectorOnScreen .X * 0.5f * Screen.Width(), AxisVectorOnScreen .Y * 0.5f * Screen.Height());
Vector2 MouseDelta = EndMousePosition - BeginMousePosition;
float Size = AxisVectorOnScreen2D.Length();
float ProjectionLength = MouseDelta | AxisVectorOnScreen2D.GetSafeNormal();
float Ratio = Size / ProjectionLength;
float Amount = 10.f * FMath::Clamp(Ratio, -2.5f, 2.5f);

I think I could also project my mouse vector on a plane perpendicular to the camera and do the same maths on this plane. What is the best solution?

Note: I have the same problem to get the amount of extrusion I should apply with my extrude tool (along the normal of a face).

Answer 1

I would not recommend to correlate the extrusion direction with its screen projection because of the asymptotic behaviour in case the axis points toward the screen or the resulting extrusion crosses the focal point of the camera.

One consistent way of doing it would be computing the extrusion height independently of the axis direction:

1. Find a virtual construction plane of your operation which is parallel to the screen. For example, in the real world, this plane would be a plane of your computer monitor screen reachable by hand as you sit at your computer.

Usually, you have some kind of central point which serves as a center of screen camera rotation as you navigate your viewport. You can draw your construction plane through this point.

Another way is drawing your construction plane through the intersection of the mouse cursor and your object as you start your extrusion operation.

Either way, the construction plane must be parallel to the screen and in front of your camera view for this method to work.

If you are using an orthogonal projection (camera without a Field of View) then you can draw the construction plane through any point.

2. Compute the length of the projection of mouse motion on the construction plane your figured out in point (1). This is your desired extrusion height.

3. Provide a means of moving the construction plane back and forth along the view axis, if the current viewport has FOV. You don't need to display and the move the construction plane if your viewport uses orthogonal projection.

Using this method your extrusion operation height would be visually consistent with what user expects with mouse motion, and the user would not experience any asymptotic and singularity issues as with the axis projection method.

Answer 2

This is a good solution.

I just had a problem transforming AxisVector in screen coordinates, it turns out I must project two points from this vector on the screen, as explained in this answer.