Casting ray from 2D to 3D

Question

As an exercise in trying to learn 3D math, I've been experimenting with hit-testing by casting a ray from 2D screen space into 3D world space. My first attempt was as follows:

Matrix m = Matrix.Invert(
    this.scene.Camera.ViewMatrix *
    this.scene.Camera.ProjectionMatrix *
    homogeneousToViewportTransform());
Vector3 rayOrigin = Vector3.TransformCoordinate(new Vector3(p.X, p.Y, 0), m);
Vector3 rayVector = Vector3.TransformCoordinate(new Vector3(0, 0, -1), m);

This gets the ray origin correct, but not the ray vector. My reasoning was that the vector [0, 0, -1] is facing into the screen when in screen space, so transforming that into world space would give the correct vector. But that doesn't seem to be the case. Looking around, I've found that the correct way to calculate this is by doing:

Vector3 rayVector = new Vector3(
    m.M31 - (m.M34 * rayOrigin.X),
    m.M32 - (m.M34 * rayOrigin.Y),
    m.M33 - (m.M34 * rayOrigin.Z));
rayVector.Normalize();

Can anyone explain what's wrong with my original reasoning, and what the actual algorithm for calculating the ray vector is doing?

Answer 1

EDIT

Please note that the following answer does not take into account several things, and is more of an attempt to visualize differences in picking between parallel and perspective projection, and probably not a great one at that. Please read all the comments for more details.

-----------------------

I believe that your original reasoning would be correct if you were using parallel projection. But since (I believe) you're using perspective projection, it doesn't work.

Imagine you have two planes that are perpendicular to the camera (both facing the camera). The green plane is closest and the red plane behind that. From the camera view it looks like this:

Now when I cast a ray out to the black dot, to do so correctly in perspective projection, I have to cast from where my camera is (which is the sharp point on the left) out to the dot. That's what the extra math does when creating the vector for the ray.

If I were to do the same in parallel projection, I can pretend that my camera is not a pin-point, but a huge plane. Here is where I can use the vector[0,0,-1]:

Here's the same view as the top image, but in parallel projection.

Anyway I'm pretty sure that's your issue. It's not a mathematical answer, but I hope it helps you understand the difference in what you were thinking and how it's supposed to work. There is of course the Math SE that you could get more details on the math.

Answer 2

Assuming you have a good understanding of how a point is transformed, the difference is that a vector is always defined as the vector to a point relative to some point of reference (e.g., the origin). When you transform the point to which a vector points, the point of reference is not transformed with it, so the vector changes and you get incorrect results.

To transform the vector v by matrix M as you would expect, transform M(v), and the point of reference/origin M((0,0,0,1)) (i.e., the - vector). Then compute M(v) - M((0,0,0,1)) to get the correctly transformed vector. Obviously, transforming by M((0,0,0,1)) zeroes out most of the components of a matrix, so you land up with just the translation components remaining and the matrix homogenous component, which with some minor math transforms into the code you have as a fix.

Answer 3

Perhaps this would be of use? It's (probably the best) article on mouse picking.

Answer 4

Long ago I developed an multiprocessor raytracing renderer for an old supecomputer in my college.

I addressed this problem by defining an observation point and the screen plane in world coordinate. The screen was defined as the coordinate for the upper-left pixel in world coordinate and the U, V direction pointing toward the left and lower pixel respectively.

In this way the pixel (x,y) is centered on P + x·U + y·V.

If the view point is W then you can cast a ray W + (P + x·U + y·V)·t that is in world coordinate.

I think that in your case the trick is to put the screen in world coordinate i.e. find the coordinate for the top-left, top-right and bottom-left pixels. With those vectors and the world coordinate view point you can cast a ray through any pixel and rely in a world coordinate parametric formulation.