Chapter 24: Perception

1. Introduction
   • sensors: change state of agent when environment changes.
   • Want to determine what the world's like, given sensory data. Problems include:
     • Sensors may not see whole environment
     • Environment may be ambiguous
     • Don't necessarily need all the info in the environment.

2. Image Formation
   • perspective projection (e.g. a pinhole camera):
     • x = -fX/Z
     • y = -fY/Z
     • where: (x,y) are points on image plane,
     • (X,Y,Z) are points in real space.
   • Scaled Orthograthic projection: when assume depth of object is much less than its distance, therefore:
     • x=sX
     • y=sY
   • Surface reflectance a combination of diffuse and specular characteristics.
     • diffuse: intensity the same in all directions. Intensity given by Lambertian:
       E = rho E0 cos(theta), where
       • rho = "albedo" (between 0 for black and 1 for while)
       • E0 = intensity of light source
       • theta = angle of incoming light relative to normal
     • Specular: light bounces off at one angle, like a mirror

3. Image-processing operations for early vision
   • Convolution:
     • Used to do smoothing and differentiating
     • Convolving f and g gives new function:
     • h(x) = (f * g)(x) =
       integral[ f(u)g(x-u)du, -infinity, +infinity ]
     • h(x) = (f * g) =
       SUM[ f(u)g(x-u), -infinity, +infinity ]
     • For Gaussian smoothing, convolve with
       G(x)=exp[-x^2/2sigma^2]/sqrt(2 pi sigma^2)
   • Edge detection
     1. Convolve the image I(x,y) with f_v(x,y) and f_h(x,y) to obtain R_v(x,y) and R_h(x,y), respectively. Define R(x,y) = R_h^2(x,y) + R_v^2(x,y).
     2. Mark those peaks in R(x,y) that are above some prespecified threshold T_n.
4. Extracting 3-D Information Using Vision
   • segmentation: grouping pixels together by semantic meaning.
   • pose: position & orientation of object relative to observer.
   • Motion: measure optical flow
     Ways to detect correspondences between similar images:
     • sum of squared distances (SSD):

       SSD(Dx,Dy) = SUM (I(x,y,t) - I(x+Dx,y+Dy,t+Dt))^2
                   (x,y)
       

     • cross correlation:

       Correlation(Dx,Dy) = SUM (I(x,y,t) I(x+Dx,y+Dy,t+Dt))^2
                           (x,y)
       

   • Binocular stereopsis: recover depth by using 2 simulaneous images.
     • For parallel optical axes:
       • horizontal disparity: H=b/Z
       • vertical disparity: V=0
       • where:
         • b=baseline (dist between cameras)
         • Z=dist from camera
     • When optical axes intersect:
       • dtheta/dZ = -b/Z^2
       • where:
         • dtheta corresponds to pixel arc
         • dZ corresponds to incremental dist
     • Constraints to find corresponding points:
       • epipolar lines: corresponding pts in different images must lie on
       • points in one image must match to at most 1 point in other image.
       • nearby points generally have similar depth
   • Texture: spatially similar or repeating patterns
     • texels: texture elements
     • texture gradient: rate of change of texture
       • Due to distance from camera
   • Shading
     • Lambertian surface image brightness is I(x,y) = k n(x,y) . s where k is a scaling constant, n is the unit surface normal and s is the unit vector in the direction of the light source.
     • This only constrains n to lie on a cone around s.
   • Contour
     • line labelling: determining relative constraints of lines
     • "+" for convex edges and "-" for concave
     • "<-" and "->" for occluding convex edge
     • "<-<-" and "->->" for limbs
     • legal junctions:
       • 6? L (2 visible edges),
       • 3? Y (3 edges, all angles < 180)
       • 3? arrow (3 edges, one of which > 180)
       • 4 T (for occlusion)
     • Line labelling is NP complete
5. Using Vision for Manipulation and Navigation
   • obstacle avoidance
   • distance to objects
6. Object Representation and Recognition
   • Given a set of known objects, which are in an image
     • Also want position and orientation
   • Two popular methods of idealizing 3-D objects:
     • Polyhedral approximation
     • Generalized cylinders
     • Both may require large amount of specification to get desired accuracy
   • Alignment
7. Speech Recognition
   • phones: basic sounds
   • Problems include:
     • homophones: 2 words that sound identically
     • segmentation: separating words