05.01 Correspondence Estimation

single camera:

• use single camera to obtain multiple images from different view points

• i.e. VO/SLAM/SFM

multiple camera frames:

• multiple camera frames

• multiple camera

in practice:

• usually single camera multiple camera frames (one camera taking images from different locations)

Projection plane:

• defined by camera center and 3D line

image plane:

• plane where the 3D points are mapped to

• y = kx + b

• -> ax +by +c = 0

• (ab,c,) -> homogenous coordinates of 2D line

• (1,2,3) equivalent to (2,4,6)

• two points (x0,y0) , (x1,y1) determine 2D line

• to solve -> choose arbitrary value of c

• -> (a,b,c) directly obtainable by cross product between (x0,y0,1) and (x1,y1,1)

have a template image

• problem: estimate the transformation W (warping) between the template T and a current image I

=> all (inlier) 2D-2D point correspondences should satsfy the same warp model

• if we have the coordinates of the point correspondances

  • -> easy to calculate warp

• if we have the warping parameters

  • -> easy to calculate the coordinates of the point correspondance

• => in practice we have none at first

• => usually first find point correspondances…

• direct methods

• indirect methods

idea:

• detect and match features (point or lines)

pros:

• can cpope with large frame-to-frame motions and strong illumination changes

cons:

• slow due to costly feature extraction, matching and outlier removal (e.g. RANSAC)

1. detect and match features that are invariant to scale, rotation, view point changes (e.g. SIFT)

2. geometric verification (RANSAC)

3. refine estimation by minimizing sum of squared reprojeciton errors between the observed feature fi in the current image and teh warped correspondance feature W(xi,p) from the template

=> feature distance

pros:

• all info in the image can be exploited (higher accuracy, higher robustness to motion blur and weak texture (i.e. weak gradients))

• increasing the camera frame rate reduces computational cost per frame (no RANSAC needed)

cons:

• very sensitive to initial value limited frame to frame motion

• use brightness of every pixel in the image and compare it to the current image

• -> find transformation parameters p that minimize the distance of brightness

• brigthness constancy

• temporal consistancy

• spatial coherence

• indensity of pixels to track does not chnage much over consecutive frames

• does not cope with strong illumination changes

• small frame to frame motion (1-2 pixels)

• does not cope with large frame to frame motion

  • however -> can be addresses using coarse to fine multi scale implementaitons (introduces later)

• all pixels in the template (initial image) undego the same transformation (i.e. they roughly lie on the same 3D surface)

requires:

• no errors in template image boundaries:

  • -> only object to track apperas in template image

  • -> else, foreground and background would move differently and thus not be transformed the same way…

• no occlusion:

  • the entire template is visible in the input image (template: initial; input: where we want to find our template)

• kanade-lucas-tmoasi tracker

• -> tracker for small motion

  
  

• how should we select features?

  • -> tomasi-kanade sub algorithm

    • method for choosing best feature (image patch) for tracking

    • => find reliable pixels to track…

• how sholud we track them from frame to frame?

  • lucas-kanade sub algorithm

    • method for aligning (tracking) an image patch

• consider we have patch centered at (x,y) (template) and shifted patch (x+u,y+v) in input image; patch has size omeag

• -> find motion vector (u,v) that minimizes sum of squared differences (SSD) w.r.t. intensity

• !!based on intensity invariance assumption!!

single univariate:

• f(x)

single multivariate:

• f(x1, x2, …, xn)

multiple multivariate:

• F(x1, x2, …, xn)

• =

single univariate:

• derivative f’(x) = df / dx

single multivariate:

• gradient ∇f(x1, x2, …, xn)

• =

• (df / dx1, df / dx2, df / dx3,…, df / dxn)

Multiple Multivariate:

• jacobian matrix dF / dx

• optimality -> find minimum

• 1D:

  • derivative of the function must be 0

• 2D:

  • partial derivatives for w1, w2 must be 0

    • (y = aw1 + bw1)…

  • => gradient must be 0 vector