02_Perception

• computer acquiring ability to interpret data related to environment

• => sensors as input…

• Machine Perception that focuses on the use of Camera as inputs…

• cheapest sensor of car

• -> includes most important information about environment

• images can be processes by classical CV algorithms and deep learning algos

• occlusion (not whole object is seen)

• viewpoint variation

• illumination

• background clutter

• defomrmation (e.g. schlangenmensch -> has not “natural / usual” shape)

• inter-class similarities

• intra-class variation

• have lots and lots of labelled images…

• chose hardware (camera, lens…)

• aquire image/video stream

• integrate image in SW pipeline

• preprocess image for further evaluatoin

• detect/extract features in image

• localize object

• classify object

• lighting

  • machine vision not as capable as human

  • -> carefully iluminate scene one wants to capture

  • => problem autonomous driving: light changes during day, weather,…

• distance and area

  • camera allows to chose Field of View (FOV)

  • -> to make good fotos, have to know FOV and work distnace…

• resolution

  • sensor that converts light from lens into electrical signal

  • -> array of signals (pixels..)

• frame grabber and software

  • -> frame grabber (normally) sends digitized image over bus system to computer

    
    

pros:

• easy to use

• good documentation

• GPU booost

cons:

• closed environment

• performance

• free for everyone (not like matlab only for students)

• language C++, python

pros:

• everything you need

• good documentation

• powerful

• GPU boost

cons:

• installation (especially GPU)

• method to perform operaitons on image in order to

• enhance image

• extraction of useful information

Camera takes picture

1. camera calibration

2. digital transformation

3. color spaces

4. filtering

5. contrast enhancment

6. affine transformation

7. resampling / compression

8. save to image

• camera moves 3d world points in 2d

• -> most cameras produce distortion

• => camera calibration :

  • find oarameters of camera and lens that affect image processing

• extrinsic camera parameters

  • position of camera center, camera heading

• intrinsic camera parametes

  • focal length

  • image sensor formant

• lens distortion parameters

• radial:

  • curve in lens

  • -> edges of images distorted

  • -> objects / lines appearr more or less curved than they actual are

• tangential:

  • camera not perfect alligned to image plane

  • -> image look tilted that some objects appera farther or closer than they actually are

• find and draw corners

• -> find distortion coefficients and correction

• correct

• matrix of values based on color (channels x dim_x in pixels x dim_y in pixels)

• => represents rectangular grid of evenly spaced pixels

• filter for red, green and blue

• -> at each pixel, measure amount of light falling into sensor…

• for different problems, as have different attributes

• RGB

• HSV

• CMYK

• CIE

• 8 bit for each channel

• -> 0-255

• -> 255^3 possible colors

• cube in 3d plane with length 255,255,255

• -> points in cube represent color

• Hue (0° red, 120° green, 240° blue) -> in between the transition from these colors…

• saturation (0% gray, 100% saturated color)

• value (0% dark, 100% light

• zylinder

• go around it for different colors

• go up for lighter, down for darker

• go to center for no gray (white when fuly light) and to border for intensive color

• apply matrix convolution to emphysize information

• by transforming the image

• => convolution with kernel (filter/mask/conv matrix)

mean

gaussian blur

sharpen

• range of difference in color and brightness

• -> low : image values concentrated in narrow range

• enhancment -> change image value distribution to cover wider range

• histogramm

• -> cumulative distribution functoin should optimally be linear …

• they preserve points, straight lines and planes…

• => paralell lines remain paralell after tranformation

• => does not necesarrily preserve angles or distances…

• -> but preserver ratios of distances between points lying on a straight line

• change resolution (number of pixels)

• -> downsampling -> decrease resolution (e.g. mean/max/min filtering)

• -> upsampling .-> increase resolution (e.g. interpolation)

• each cell in new raster must be computed by

  • sampling or interpolation over some neighborhood cells in corresponding position in original raster object

• nearest neighbor

• bilinear

• bicubic

• piece of information relevant for solving computational task related to certain application

• includes methods for computing abstractions of image informatino

• -> and making llocal decisions at every image point

• => resulting features subsets of image domain like isolaed points, continuous curves, connected regions

=> e.g. edges, corners, lines,…

• after detecting features

• -> local image patch around feature can be extracted

• => e.g. isolation of shapes of digitized image or video stream

• object detection

• robot navigation

• motion tracking

• …

• Identify sudden changes

  (discontinuities) in an image

  • intuitively:

  • most semantic and shape info can be encoded in edges

  • more compact than pixels

• artists line drawing of an object

• -> but artist has object-level knowledge …

• surface normal discontinuity (surface ends)

• depth discontinuity (e.g. backside of ball not visible although continuous surface…)

• surface color discontinuity

• illumination discontinuity (e.g. shadow)

• -> rapid change in intensity function of image… (grayscale…)

• => first derivative -> extrema are edges

• canny edge detection

1. gaussian filter to smooth image

   • remove noise

   • larger kernel size decreases sensitivity to noise but increase localization error

2. compute intensity graidents for each pixel

3. non-maximum suppresion -> thin multiple-pixle wide ridges down to single pixel width -> in narrow intensity drops -> chose only maximum / minimum…

4. apply lower and upper threshold for edges (hysteris)

   • intensity above are used to start edge curve

   • if neighboring are above lower thershold, edge is continued

   • if intensity below lower thershold, discarded as noise

• edge detection makes possible to considearbly reduce amount of data in an image

• -> but image still described by pixels…

• => if lines, ellipses, … could be defined by characteristic equations => amount of data reduced even more

• hough transformation

• all lines in 2d representable by y=ax+b

• => each line represented by (a,b)

• => each line (a,b) corresponds to single point pi in hough space

• => hough space has the two features a,b

• are part of infinite number of lines -> each point creates infinite points in hough space

• => these points form line hough space determined by all (a,b) comninations going thoruhg the point in the image space

• the intersection of all these points on the line L in the hough space

• => correesponds to the single line in real space…

• shifting a window in any direction should result in large changes…

• => flat region -> same intensity -> no changes when moving

• => edge -> no change along edge directoin movement

• => corner -> significant changes in all directions

• have to images available

• have baseline (distance between cameras)

• have focal length f

• calculate difference of same object w.r.t. position on x-axis

• Z = B*f/d

• distance = Baseline * focal length / disparity (x2-x1)

• => map depth information to each pixel…

• rule based

  • ->manual feature extraction and classification logic

• machine learning

  • -> manual feature extraction

  • classifier learned automatically

• deep learning

  • -> relevant features and classifier learned jointly

• video as sequence of frames over time

• -> image data is function of space (x,y) and time t

• => optical flow is pattern of apparent motion of ibjects surface and edges

• => in visual scene caused by relative motion between observer and scene

• simultaneous localization and mappinf

• => visual odometry -> process of determining position and orientation of robot by analyzing associated camera images

• on both axis bell curve (normal distribution…)

• “Thin” the lines to create clear edges and not blurry ones…

• suppress noise

• have input image

• change color and/or contrast (e.g. HSV is prominent, better edge detection…)

• compute histogram of oriented gradients (HOGs)

• Normalize contrast over overlapping spatial blocks

• collect HOGs over detection window

• apply machine learning

• -> person / non-person classification

• no one of many possible methods…

• e.g. SIFT

• SURF

• manually choose kernel to compute gradients

• calculate orientation for each pixel (one of 8 angles \|/…)

=> use these features in ML algo…

1. extract image shape

2. transform image to greyscale

3. blue image

4. perform canny edge detection

5. mask special region in image

6. detect hough line