[CS5670] Lecture 1: Images and image filtering

1. What is an image?

A grid (matrix) of intensity values

(common to use one byte per value: 0 = black, 255 = white)

 

Can think of a (grayscale) image as a function $f$ from $R^2$ to $R$

• $f(x,y)$ gives the intensity at position (x,y)
• A digital image is a discrete (sampled, quantized) version of this function

 

Image transformations

• As with any function, we can apply perators to an image
• Today we'll talk about a special kind of operator, convolution (linear filtering)

 

2. Filters

• Filtering
  • Form a new image whose pixel values are a combination of the original pixel values
• Why?
  • To get useful information from images
    • E.g., extract edges or contours (to understand shape)
  • To enhance the image
    • E.g., to remove noise
    • E.g., to sharpen and 'enhance image' a la CSI
  • A key operator in Convolutional Neural Networks

 

Canonical Image Processing problems

 

• Image Restoration
  • denoising
  • deblurring

• Image Compression
  • JPEG, HEIF, MPEG, $\dots$

• Computiong Field Properties
  • optical flow
  • disparity

• Locating Structural Features
  • corners
  • edges

 

Question: Noise reduction

• Given a camera and a still scene, how can you reduce noise?

 

Image filtering

• Modify the pixels in an image based on some function of a local neighborhood of each pixel

 

 

Linear filtering

• One simple version of filtering: linear filtering (cross-correlation, convolution)
  • Replace each pixel by a lnear combination (a weighted sum) of its neighbors
• The prescription for the linear combination is called the 'kernel' (or 'mask', 'filter')

 

 

Cross-corrleation

 

Let $F$ be the image, $H$ be the kernel (of size $2k+1$ x $2k+1$), and G be the output image

 

$$G[i, j] = \Sigma^k_{u=-k}\Sigma^k_{v=-k} H[u, v] F[i+u, j+v]$$

 

This is called a **cross-correlation** operation:

 

$$G = H \bigotimes F$$

 

Can think of as a 'dot product' between local neighborhood and kernel for each pixel

 

Convolution

 

Same as cross-correlation, except that the kernel is 'flipped' (horizontally and vertically)

 

$$G[i, j] = \Sigma^k_{u=-k}\Sigma^k_{v=-k} H[u, v] F[i-u, j-v]$$

 

 

This is called a convolution operation:

 

$$G = H \star F$$

 

Convolution is commutative and asscociative

 

 

Mean filtering

 

 

Mean filtering/Moving average

 

 

 

 

Linear filters: examples

 

identical image

Shifted left by 1 pixel

Blur (with a mean filter)

sharpening filter (accentuates edges)

 

 

 

 

Sharpening

 

 

Smoothing with box filter revisited

 

 

Gaussian kernel

 

$$G_{\sigma} = \frac{1}{2\pi \sigma^2} e ^ {\frac{x^2+y^2}{2\sigma^2}}$$

 

 

Gaussian filters

 

 

 

Mean vs. Gaussian filtering

 

 

 

Gaussian filter

 

• Removes 'high-frequency' components from the image (low-pass filter)
• Convolution with self is another Gaussian

 

 

• Convolving twice with Gaussian kernel of width $\sigma$

     = convolving once with kernel of width $\sigma \sqrt{2}$

 

 

Shapening revisited

• What does blurring take away?

 

 

Sharpen filter

 

$$F+\alpha (F-F \star H) = $$

 

 

• $F$: image
• $F \star H$: blurred image

 

unit impulse (identity kernel with single 1 in center, zeros elsewhere)

 

 

'Optical' convolution

 

Camera shake

 

Bokeh : Blur in out-of-focus regions of an image

 

 

Filters: Thresholding

 

 

$$g(m,n) = \begin{cases} 255 , & \mbox{f(m,n)} > A \\ 0, & \mbox {otherwise} \end {cases}$$

 

 

Linear filters

Can thresholding be implemented with a linear filter?

 

 

 

 

 

 

 

Reference: Lecture 1: Images and image filtering