C11 - Photometric Error and Direct SLAM

• create only sparse map of the world

• does not sample across all available image data

  • -> discard info around edges & weak intensities

• compared to indirect (two-step -> feature track and then movement) these are one-step methods

• create a potentially more dense feature map

• more accurate

• less prone to error propagation (as only has one step…)

• The foundation of direct methods

• -> i.e. brightness consistency…

• we have sparse 2D-2D correspondances (monocular…)

• reduces image to set of sparse keypoints matched with feature descriptors

• -> reconstructed also sparse

• simultaneously

• least squares error

• based on depth -> can back project any pixel in 3D space and then project into next image

• RGB-D cameras

• binocular (stereo) camera -> pixel depth based on disparity

• monocular -> have to treat depth as unknown -> optimize it along with camera pose

• we have to images p1 and p2

• they are the perspective projections of 3D point P

• we do not know P, nor do we know the rotation and translation between camera 1 and 2

• => we want to express p2 as a formula of p1

1. we can express the poitn P as a “stretched” version of the direction p1 (normalized)

• we can then express p2 as the projection of P onto camera frame 2

• in general, K is assmued to be known, R,t and Zdepth is assumed to be unknown

• brightness constancy

• -> estimate the depth, R, and t

• such that the photometric error between p1 and p2 is minimized…

• => which we obtain simultaneously

• no feature extraction, no matching, no RANSAC needed

• -> directly minimize photometric error

• we want to find the optimal rotation R, translation T, and points {P_i} (i.e. depth information)

• so that we minimize

• the error between intensity at p1 in the left image (i.e. image k-1 -> as we assume SLAM -> continuous number fo frames…)

• and the intensity from p1 back projected point P, projected onto image k with R,T

• here, we also introduce a “robust kernel” instead of the squared distance (comes in later lecture)

• -> minimize this over the sum of all point we have in the right image

pros:

• all image pixels can be used

  • higher accuracy

  • higher rboustness to motion blur and weak textures (i.e. weak gradients)

cons:

• very sensitive to initial value

• limited frame to frame motion

• some features in environment (like clouds) are far off

• -> leading distance to be infinity…

• => can cause prblems with numerical stability

• use inverse depth paraetrization

• -> replace depth with the inverse of it

• -> if norm is very large (i.e. distancce point p to camera center c0) -> roh goes to 0

• -> if distance is very small ->

• improves numerical stability

• no -> typically not necesarry as some pixels might be redundant

• -> choose from one of three strategies

• dense direct method

  • use all pixels

• semi-dense direct method

  • track partial pixels with significant gradients

• sparse direct mehthod

  • track sparse key points

• not really achiveable in real time

• not all pixels contribute to solution (i.e. pixel with non obvious gradient…)

• track partial pixels with significant gradients

• -> if gradient 0 -> jacobian is 0 (i.e. no contribution to problem)

• -> only use pixels with high gradients (-> discard areas where gardietn non-obvious)

• use tracked pixels to reconstruct semi dense structure

• yields less but more reliable pixels

• kind of combine indirect method with direct

• -> use i.e. harris do erxtract key points

• but instead of using e.g. SIFT -> we only want the position but no descriptor

  • -> thus, speed up compared to indirect method as we have no feature matching to establish corerspondances….

• fastest method

• but can only calculate sparse reconstruction

• direct SLAM not suitable for large baselines for two reasons:

1. initial pose may be unreliable -> leads to local minimum

2. photometric conssitency assumption not satisfied

-> in practice small preferred

• indirect:

  • PTAM

  • ORB-SLAM

  • SVO

• direct:

  • SVO (is direct and indirect)

  • LSD-SLAM

  • DSO

• reduce various effects that affect our brightnes constancy assumption

• different exposure times

• vignetting

• response funciton

• response function maps irradienace (energy per minute falling onto sensor) into brightness

• -> is not linear

• => thus, to actually correctly calculate the difference in “brightness” -> we should use irrediance (as brightness is a non-linear function of irrediance and thus somewhat biased…)

• => thus we should calibrate the response funciton

• longer exposure -> brighter image

• in practice: pair of image differnet exposure times (i.e. cell phone automatically adjusts it…)

• => given we consider consistency of irradiacen rather than brightness

• -> we have to calibrat the exposure time to have the same value

• vignetting -> reduction of images brightness towards the periphery compared to image center

• mainly caused by manufacturing flaws

• => should remove this effect before apply photometric loss…