04. Calibration

• incomplete detection due to limitaion of existing algorithm (only partial line detected…)

• occluded detected image line due to limited FOV (rest of line outside of current image…)

• consider: we have several 3D lines lying on the same projection plane

• each 3D line has unique plucker coordinates

• => thus have different scale when not considered not considering the projection…

• process to determine

• extrinsic parameters (R,T) of camera

• intrinsic parameters (K plus lens distortion)

• for known intrinsic parameters

• -> figure out extrinsic parameters

• => in later chapter in detail…

• as for lateron tasks, they change

• -> but intrinsic stay fixed

• => only store intrinsic for lateron…

• measuring 3D position of n>= 6 3D control points on a 3D calibartion target

• and the 2D coordinates of their projections in the image

• => based on only a single image

• plus prior knowledge about i.e. size of each square…

=>

• use standard methods to solve this linear system of equations…

• multiply both intrinsic and extrinsic matrices

• rewrite the individual elements to m11, m12, …, m34

=> rewrite as column vectors; replace right matrix with P

• Set up homogenous LGS

• -> here, under the division part (with mT3) due to it being lambda according to the left hand side…

=> (remiinder P is world coordinates we know…)

• for n points -> stack all these equations into a big matrix

For each point, add two rows to the Q matrix:

(here split to have the image bigger…)

• no -> we have homogenous LGS QM = 0

• => actual scale can be revocered lateron based on the last element of K (which is part of decomposition of M) where we know that the laset element must be 1…

• to have unique, already scaled solution -> rank 12 (unique solution)

  • 12 due to M being 12 dimensional

• In practice: 11 suffice as we can scale lateron…

• each 3D - 2D correspondense provides 2 independent equation

• -> 12 equations needed

• -> 6 point correspondances

• Least sqare solition

• -> minimizing the sum of the squared residuals ||QM||^2

• subject to constraing ||M||^2 = 1 (why this is is explained later)

• => can be solved using SVD

• Constraint introduced to avoid obvious non-zero solution…!

• Q is known

• M is unknown

• we want to minimize the residuals

  • min ||QM||^2 _ 2

• as we want to find M

  • arg min _M ||QM||^2_2

• subject to ||M||_2 = 1

• use SVD composition on Q

• Q = UEV

• optimal solution M* is the column of V corresponding to the smallest singular value (which are denoted in E)

• M = K(R|T)

• M now known, K(R|T) still unknown

• use QR factorization (as it inherently satisfies orthogonaly which is required in our rotation matrix, due to it being in SO(3))

• -> QR (or rather RQ) decomposes M into orthogonal R, T and upper triangluar matrix (i.e.K)

=> only works for square matrices…

advantage:

• require only single picture

• corners can be detected with very high accuracy < 0.1 pixels currently

setup:

• use many more than 6 points (ideally more than 20) and non coplanar

• set of points are called coplanar

• if there exists a plane in 3D space on which all of these points lie

• => can fit a plane in 3D space through all points…

• zhangs method

• on 3D coplanar points

Tsais:

• single image

• non-coplanar points (more complex image such as cube)

Zhangs:

• multi-view images

• coplanar points (i.e. chess board)

• same as in tsai: neglect radial distortion

• but due to all points being coplanar: Zw can be set to 0 (as only X and Y axis are important…)

  • => set Z axis paralell to the plane of the 3D poitns

-> when writing more compact, can simply leave out part with 0 and reduce size of matrix…

• where similar to tsais (m), h represents the combined intrinsic (kalibartion) and extrinsic (rotation and translation) matrices

• Here, H is called Homography

• And keep in mind -> Zhang requires several images thus P corresponds to one of them

-> Then applied to the i-th image

• where lds Q is known (as we know the different points in the camera frame) and H is unknown

• Again, search for minimal solution

• -> Q (2nx9) should have rank 8 to have unique (up to scale) non-trovoal solutoin

  • detailed explanation later when discussing Homographies…

• each point correspondence agian provides 2 independent equations

• => thus minimum of 4 non-collinear points required

  • colinear: lying on single line

• solution for n>=4 points solved similar to tsais using SVD (as we again search for minimal solution…)

• QR not possible

• => as R part is not orthogonal!

=> compared to tsai: one row missing…

approach:

• first focus on recovering intrinsic matrix

• and then after fixing it, revocer the extrinsic matrix with regular LGS…

• intrinsic parameters stay the same for different camera perspectives (for same camera)

• only extrinsic change

• => find intrinsic by using multiple images from differnet views…

• each view j has different homography H^j

  • also different R^j (rotation) and T^j (translation) parameterts

• But K stays the same for all views… (intrinsic…)

  • also denoted as M

=> in zhang, we were able to revocer individual homographies for each image but not yet its decomposition

=> use regular DLT method

1. determine intrinsic matrix M from a set of known homographies

express columns of rotation by unknown intrinsic parameters

• then enforce constraints of the rotaiton matrix

Constraint 1:

• again due to orthogonality of elements -> must be 0 when dot product with translation…

Constraint 2:

• rotation matrix is orghogonal -> langth of each element is 1…

-> As parts in red squares are the same, replace with variable B

• when we can estimate B -> we can recover M from this equation above

As b is symmetric, we are only interested in 6 of its elements

• h is known (obtained after DLT in previous step)

• -> rewrite above equations to linear system

We have two constriants for our homography

-> each homography contributes two linear equations

• stack 2N equations from N views to yield linear system Ab=0

• solve for b using SVD

• => typically we need more than 3 views (each view provides two constraints)

• we now know intrinsic and homography

• thus, from previous equaitons, compute each column

• and enforce constraints on scale

(each column, not only r1…)

-> after this, Ri = (r1, r2, r3) might not be strict orthogonal (but should be)

=> last step: project result from matrix space onto the SO(3) manifold to enforce orghogonality (not into more detail, only keep in mind that after previous step additional is required to enforce strict orthogonality….)

• radial distortion

  • light rays bend more near edges of lens than they do at its optical center

• tangential distortion

  • if lens misaligned (not perfectly pparalell to image sensor) tangential distortion occurs

• given object points and image points (detected chessboard corners)

• -> conduct zhangs method

1. compute initial intrinsic parameters

   1. while initially setting distortion coefficients all to 0

2. estimate initial extrinsic parameters as if the intrinsic have been already known

3. run gradient descent to minimize reprojection error to jointly optimize/estimate intrinsic, extrinsic and distortion parameters

• euclidean distance in pixels between

• observed image point and corresponding 3D point reprojected onto the camera frame

observed: how it really is

reprojected: how it should be using current parameters

• where left hand side of equation in sum is observed and right hand side reprojected

=> gradient descent

• might end in local minimum…

-> use tsais / zhangs method to find good initial values

-> then apply gradient descent (here in right image…)