1
$\begingroup$

I'm working in a photogrammetry camera and laser application where I need to measure a displacement in a specific plane, this is a quick diagram of the setup: enter image description here

In order to get proper measurements I'm trying to do a projection from the 3d points to the 2d camera plane proceeding like this:

  • First I'm doing the Camera calibration using opencv and a chessboard, I'm taking a few chessboard shots in different angles and applying the function initUndistortRectifyMap, I´ll have the distortion coefficients, intrinsic and extrinsic parameters.
  • With the intrinsic parameters and the coefficients I'll apply lens correction.
  • The extrinsic parameters of the previous step are for each chessboard position and I assume that they won't be valid for my particular application, so I'm placing a chessboard in the above marked area as "Measurement Plane" and running again the initUndistortRectifyMap function which this time it's giving me, I think, correct extrinsic parameters of the ROI.
  • Once I've got the "correct" intrinsic and extrinsic parameters I´m applying the camera matrix to calculate the 2d points (u,v) projection:

enter image description here

My questions are:

Is it the correct way to proceed for this application?

The camera system only reads X,Y pixel points each time, but to perform a 2d projection seems like I'll need the Z, how could I have this last coordinate of the 3d point?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ To clarify: You have a camera at a fixed location, with a plane surface upon which a laser is shining and you want to locate the laser spot in plane (real world) coordinates? $\endgroup$
    – Cedron Dawg
    Commented Jan 21, 2018 at 15:59
  • $\begingroup$ Thanks for the reply, yes, that is the setup, but what I'm trying to do is a perspective transformation moving the laser points plane of the measurement region to the camera plane... $\endgroup$
    – joe
    Commented Jan 21, 2018 at 16:08
  • $\begingroup$ Don't you want the opposite? Given the laser location on the image (pixel coordinates), find the location (real world coordinates) on the plane? $\endgroup$
    – Cedron Dawg
    Commented Jan 21, 2018 at 16:14
  • $\begingroup$ Yes, sorry, I meant find the world coordinates of the laser plane. $\endgroup$
    – joe
    Commented Jan 21, 2018 at 16:25

2 Answers 2

Reset to default
1
$\begingroup$

You can bypass all the OpenCV stuff, and create a custom mapping from pixel location to real world location of the form:

$$ x = A_x m^2 + B_x mn + C_x n^2 + D_x m + E_x n + F_x $$ $$ y = A_y m^2 + B_y mn + C_y n^2 + D_y m + E_y n + F_y $$

Where $(x,y)$ is the location on the plane using a grid on the plane, and $(m,n)$ is the pixel location in the image.

To find the coefficients simply requires a set of calibration points and some linear algebra. You will need at least six points, but more points spread out over you entire viewing field of operation would be better.

Establish a grid on your plane. A point on the plane will be $(x_i,y_i)$ which corresponds to a point $(m_i,n_i)$ on the image. N is the number of calibration points.

Construct a matrix equation like this:

$$ \left[ \begin{array}{ccccccc} m_1^2 & m_1 n_1 & n_1^2 & m_1 & n_1 & 1 \\ m_2^2 & m_2 n_2 & n_2^2 & m_2 & n_2 & 1 \\ .&.&.&.&.&.\\ .&.&.&.&.&.\\ m_i^2 & m_i n_i & n_i^2 & m_i & n_i & 1 \\ .&.&.&.&.&.\\ .&.&.&.&.&.\\ m_N^2 & m_N n_i & n_N^2 & m_N & n_N & 1 \\ \end{array} \right] \cdot \left[ \begin{array}{cc} A_x & A_y \\ B_x & B_y \\ C_x & C_y \\ D_x & D_y \\ E_x & E_y \\ F_x & F_y \\ \end{array} \right] = \left[ \begin{array}{cc} x_1 & y_1 \\ x_2 & y_2 \\ .&.\\ x_i & y_i \\ .&.\\ x_N & y_N \\ \end{array} \right] $$

Which is in the form:

$$ M \cdot U = R $$

Multiply both sides by $ M^T $.

$$ M^TM U = M^T R $$

The product, $M^TM$, is a 6x6 matrix. Find its inverse and multiply both sides by it.

$$ U = (M^TM)^{-1}M^T R $$

The coefficients you are seeking are the elements of U.

The same technique can be used to create a mapping in the reverse direction as well.

Hope this helps.

Ced

Improve this answer
$\endgroup$
4
  • $\begingroup$ Could you provide an intuition on why this works or some source for the theory behind this solution? $\endgroup$
    – Mike
    Commented Feb 1, 2020 at 9:09
  • $\begingroup$ Do we use the points to fit a plane? Could you give a little more background on how this helps us find the coordinates in 3D? $\endgroup$
    – r4bb1t
    Commented Feb 24, 2020 at 23:38
  • $\begingroup$ @PyWalker2797 If you look carefully, this problem is about a 2D to 2D transform, to which I gave a generalized best fit solution. Sort of similar to a billiards ball application on a pool table. Localization in 3D is a different matter and generally requires some extra information, such as a different perspective or a priori knowledge of object sizes. You can contact me at cedron at exede dot net if you have any specific questions, or post a new question of your own. $\endgroup$
    – Cedron Dawg
    Commented Feb 25, 2020 at 1:41
  • 1
    $\begingroup$ @Mike This is standard Linear Algebra. Search on "least squares fit on overdetermined systems". $\endgroup$
    – Cedron Dawg
    Commented Feb 25, 2020 at 1:44
1
$\begingroup$

For your application, here are the steps I would follow:

  1. Calibrate the camera and obtain the intrinsics - You have obviously done that and corrected for the distortion already. initUndistortRectifyMap only lets you to precompute the projection. Otherwise, you might as well use projectPoints each time. This is a detail and I would not care about that initially. But it's a good practice anyways.)

  2. The main idea is to find the laser plane (with extrinsic calibration to the camera). To find the extrinsics:

    a. Place a checkerboard in front of the camera and leave enough space on the board for the laser line to be projected.

    b. Image the laser line projections together with the checkerboard.

    c. Now we can compute two planes with respect to the camera: The plane joining the laser line with the camera center, and the plane that checkerboard lies in (we now the checkerboard).

    d. Intersect the two planes to find the line in 3D space.

    e. Collect all such lines from multiple images and fit a plane to all of them. This is the equation of the laser plane.

  3. To make measurements in 3D, simply shoot lines through the observed laser points in 2D image and intersect the rays with the laser plane. Each intersection will give a single point in 3D space.

This way we also do not explicitly need $(R,t)$. Our extrinsics are rather the equation of the laser plane.

In case we use a point-laser, the equation also remains the same, as instead of lines, we can collect sets of points and still do the plane fit.

By the way, here is a toolbox and a recent further read.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer, just to clarify, I guess what I'm interested is in the planes intersection then if the extrinsic parameter are not necessary we don't want to detect any corners so could the checkerboard just a white plane board? $\endgroup$
    – joe
    Commented Jan 21, 2018 at 16:38
  • $\begingroup$ The pose of the camera with respect to the checkerboard is needed of course, but this is not an explicit $(R,t)$ between camera and the laser. That's what I meant. From the pose of checkerboard, $(R,t)$, it's possible to calculate the plane of it: $p_{checker}=[a,b,c,d]$. $\endgroup$
    – Tolga Birdal
    Commented Jan 21, 2018 at 16:47
  • $\begingroup$ Could you elaborate a little more on step (c) and step (d) ? How do we compute the plane joining the laser line with the camera center? $\endgroup$
    – r4bb1t
    Commented Feb 24, 2020 at 21:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.