03. Image Formation

• semantic segmentation

  • find what the objects are

• instance segmentation

  • differentiate the different objects from each other

• panoptic segmentation

  • instance segmantation that also considers the background

• model selection based on type of data

  • -> for data fitting

  • -> e.g. if only rotation and translation needed

    • -> SO(3) is good

  • -> if scaling is needed

    • -> requires SE(3)

• parameter estimation by

  • minimizing the distances

  • between correspondences

• -> done by computing sum of distances

• => find optimal rotation, translation (parameters) to minimize sum of correspondence distances (objective function)

• maps log-exposure value (scene radiance)

• to intensity levels in teh input images

• => basically transforms light signal to electruc signal…

• => is not linear, thus needs calibration

• has converging lens

• all rays parallel to optical axis converge at focal point

• flips the image

• image coordinates: x

• 3D coordinates: X

• far away objects appear smaller

• intersection of parallel lines in 2D

• vanishing directions

• 
  

• size is inversively proportional with distance

• paralell lines intersect at “vanishing point” in the image

• can fall both inside or outside of the image

• connection between two horizontal vanishing points is the horizon

  • => i.e. wie in gemoetrie in HfT -> fernpunkte wenn tatsächlich paralell

  • -> oder in echt paralell aber durch perspektive nivht mehr…

-> blue line is horizon / vanishing line

• vanishing direction is defined by connecting vanishing point and camera center

• -> is paralell to a 3D dominant direction (which yield the vanishing point…)

-> more on that in later lectures…

• angular portion of 3D scene seen by camera

• => FOV are dimensions of the scene of the target object

• -> the furtehr we are away, the smaller the FOV…

• inversely proportional to focal length

-> large POV -> use short focal length

-> small POV -> use large focal lenght

homogenous:

(a,b,c)

cartesian:

(a/c, b/c)

• C: optical center

  • -> center of lens, i.e. center of projection

• Xc, Yc, Zc: axes of camera frame

• Zc: optical axis (principal frame)

• O: principal point

  • -> i.e. intersection of optical axis and image plane

perspective:

• size varies inversely with distance -> looks realistic

• parallel lines do not (in general) remain paralell

paralell:

• good for exact measurements

• paralell lines remain paralell

• less realistic looking

• based on similar triangles

!! we use front image plane!!!

• ku and kv are conversoin factors (between mm and pixels)

• u,v are pixel coordinates

• here, we can expres the focal length

• in mm -> ku*f; kv*f

• in pixels -> au, av (replacing ku*f…)

• use formulas we derived so far (camera plane -> image plane -> pixel)

• where focal length au av (in pixels…)

• and principal points u0 v0

• principal point (center of camera) does not completely align with center of image frame

• => due to manufacturing process…

• => nowadays very good manufacturing -> skew factor assumed to be 0…

• => au = av (i.e. suqare pixels…)

• world frame -> camera frame (take extrinsic parameters into consideratoin)

• camera frame -> pixel

• camera frame

• image coordinates

• pixel coordinates

• world frame

intrinsic:

• camera frame -> image frame

extrinsic:

• world frame -> camera frame

  • -> corresponds to rotation and translation…

• [R|T] -> rotation and translation

• => called extrinsic parameters

-> combine intrinsic and extrinsic parameters

• two-step computatoin method

• one-step computation method

• independently project the two endpoints of the line (homogenous)

• line is then cross product of these homogenous endpoints

• -> line also in homogenous representation… (as it is 3D vector…)

• 4D vector (A,B,C,D)

• where n=[A,B,C] is the unit vector

• D is the distance from the origin (projection from origin onto normal)

1. take two arbitrary points on plane to calculate the vector between them

   1. dot product with normal must be 0 (as it is orthogonal…)

2. Use this to compute D

• point P (X,Y,Z,1)

• -> PTn = 0

• => lies on plane

• camera center and line on image plane span projection plane…

• P: projection matrix (i.e. combined intrinsic and extrinsic matrix -> but here transposed!)

• I: line on image plane (result of 1-step or 2-step method…)

• D: intersectoin

• L: Plucker matrix of line

  • => derived from the plucker coordinates

• pi: plane

• intersectoin of camera center with image plane… (center (if no intrinsic deviation) of image plane…)

• virtual image plane

• focal length equal to 1

• origin of pixle coordinates at principal point

• in certain cases, want to do thing in 3D

• -> treat/extend 2D coordinates of image plane in 3D (w.r.t. camera center etc.)

• => normalized images improve ease of computation

• inverse K and multiply wiht original coordinates

• it is a geometric constrait that

• -> image frame points are parelell to camera frame points

• => lie on same line from camera center

=> only differ in scale

• describes a geometric constraint of lines

• -> normal on line on image frame and camera frame only differ by scale

• => normals are paralell

• 3D line projection (image to camera frame) is orthogonal to the normal of projection plane

• direction defined by 3D point lying on 3D line and the origin is orthogonal to the normal of projeciton plane

  • line from camera center to point on line …

• given: (camera frame) 3D point o, 3D lines ex, ey

• given: projection of theme onto image frame of two cameras (p, dx, dy)

• -> use constraints to calculate rotation and translation matrix between them

1. calculate normal nx, ny between p and dx, dy

2. ex, ey is scaled paralell to the cross products of both norms with translaiton

• using this, we can calculate

• sperical has larger fov than planar projection

-> sperical: iamge plane is sphere…

• use omnicamera (360° camera…)

• normalize 3D point to map onto sphere

• convert to spherical coordinates

  • using angles…

• “up / down” using polar angle (0-pi)

• “left/right” using asimuth (0-2pi)

• r needed if we do not have unit sphere (to express distance from center if not 1…)

• s defined size of final image

• use cross-shaped expansion

• using cube-based representaiotn

• map sphere to different surfaces of the cube…

• create circles arond the sphere

• when mapping two lines onto the sphere

• the normals to the projected planes from these lines

• lie on the same circle if the 3D lines are paralell

• spatial distortion due to equi-rectangular representation!!!

• use icosahedral representation

• => map to icosahedron and define triangles

• map to triangle shaped sphere

• expand to triangle image