Open videorectification.m in the Editor
Run in the Command Window

# Image Rectification

Rectification is the process of transforming stereo images, such that the corresponding points have the same row coordinates in the two images. It is a useful procedure in stereo vision, as the 2-D stereo correspondence problem is reduced to a 1-D problem when rectified image pairs are used. Image rectification is a prerequisite step for the Stereo Vision Demo.

This demo uses the Computer Vision System Toolbox™ to compute the rectification of two uncalibrated images where the camera intrinsics are unknown.

## Contents

- Step 1. Read Stereo Image Pair
- Step 2. Collect Salient Points from Each Image
- Step 3. Select Correspondences Between Points
- Step 4. Remove Outliers Using Random Sample Consensus (RANSAC) Algorithm
- Step 5. Rectify Images
- References

## Step 1. Read Stereo Image Pair

Here we read in two color images of the same scene taken from different positions. We convert them to grayscale since colors are not necessary for the matching process.

Below we show both images side by side, and we produce a color composite to demonstrate the pixel-wise differences between them. There is an obvious offset between the images in both dimensions. The goal of rectification is to transform the images, aligning them such that corresponding points in each image will appear on the same row.

    I1 = im2double(rgb2gray(imread('yellowstone_left.png')));
    I2 = im2double(rgb2gray(imread('yellowstone_right.png')));

    cvexShowImagePair(I1, I2, 'Left image', 'Right image');
    cvexShowMatches(I1, I2, [], [], 'Left image', 'Right image');

## Step 2. Collect Salient Points from Each Image

The rectification process requires a set of point correspondences between the two images. To generate these correspondences, we collect points of interest from both images, then choose potential matches between them.

In this step we produce these candidate points for each image. To have the best chance that these points will have correspondences in the other image, we want points around salient image features such as corners. We use the `CornerDetector` System object with its default detection method of Harris corners. The detected points in both images are shown below. Observe how many of them cover the same image features, as desired.

    hcornerdet = vision.CornerDetector('MaximumCornerCount', 500, ...
        'CornerThreshold', 0.001, 'NeighborhoodSize', [21 21]);

    cornerPoints1 = step(hcornerdet, I1);

    cornerPoints2 = step(hcornerdet, I2);

    cvexShowImagePair(I1, I2, 'Corners in left image', ...
        'Corners in right image', 'SingleColor', cornerPoints1, cornerPoints2);

## Step 3. Select Correspondences Between Points

Next we determine correspondences between the left and right image points. We use the `extractFeatures` and `matchFeatures` functions for this purpose. For each point, we extract a 9-by-9 block from the image centered around it. We use the normalized cross-correlation (normxcorr) between these image regions as the matching cost between points. Each point in either image is matched with the points in the other image with the lowest cost. There is no uniqueness constraint, so points from either images can participate in multiple correspondences.

    [features1, validPoints1] = extractFeatures(I1, cornerPoints1, ...
        'BlockSize', 9);
    [features2, validPoints2] = extractFeatures(I2, cornerPoints2, ...
        'BlockSize', 9);

    [indexPairs, match_metric] = matchFeatures(features1, features2, ...
        'Metric', 'normxcorr', 'MatchThreshold', 5);

    % Retrieve locations of matched points for each image
    matchedPoints1 = validPoints1(indexPairs(:, 1), :);
    matchedPoints2 = validPoints2(indexPairs(:, 2), :);

The image below is another color composite. Points from the left image are shown as red, and points from the right image are shown as green. Yellow lines are drawn between the points to show the correspondences selected by this alignment procedure. Most of these are correct, but there are also some outliers.

    cvexShowMatches(I1, I2, matchedPoints1, matchedPoints2, ...
      'Matched points in I1', 'Matched points in I2');

## Step 4. Remove Outliers Using Random Sample Consensus (RANSAC) Algorithm

Some of the point correspondences obtained in the previous step are incorrect. We cannot proceed to rectification with erroneous point correspondences as above, so we must first discard bad matches (i.e., outliers). A standard way to do this is with the Random Sample Consensus (RANSAC) algorithm [2,3,4]. We use the `estimateFundamentalMatrix` function with the Method set to 'RANSAC'.

    [fMatrix, inliers] = estimateFundamentalMatrix(...
        matchedPoints1, matchedPoints2, 'Method', 'RANSAC', ...
        'NumTrials', 1000, 'DistanceThreshold', 0.1, 'Confidence', 99.99);

    inlierPoints1 = matchedPoints1(inliers, :);
    inlierPoints2 = matchedPoints2(inliers, :);

    cvexShowMatches(I1, I2, inlierPoints1, inlierPoints2, ...
        'Inlier points in I1', 'Inlier points in I2');

## Step 5. Rectify Images

The rectification procedure transforms the images such that the corresponding points will appear on the same rows in both images. We use the `estimateUncalibratedRectification` function to compute the rectification transformation matrix for each of the images. We then use the `cvexShowMatches` and `cvexShowStereoImages` functions to apply the rectification transformations and to display the images.

    if isEpipoleInImage(fMatrix, size(I1)) ...
        || isEpipoleInImage(fMatrix', size(I2))
        error(['For the rectification to succeed, the epipoles must be ',...
            'outside the images.']);
    end

    [t1, t2] = estimateUncalibratedRectification(fMatrix, ...
        inlierPoints1, inlierPoints2, size(I2));

Rectify the images using projective transformations, t1 and t2, and show a color composite of the rectified images demonstrating point correspondences.

    cvexShowMatches(I1, I2, inlierPoints1, inlierPoints2, ...
        'Inlier points in I1', 'Inlier points in I2', 'RC', t1, t2);

Show the rectified images and the epipolar lines corresponding to the features points used during the rectification process. Note that the corresponding lines are in the same rows demonstrating that the rectification was successful.

    cvexShowStereoImages('Rectified image 1', 'Rectified image 2',...
        I1, I2, inlierPoints1, inlierPoints2, fMatrix, t1, t2);

## References

[1] Trucco, E; Verri, A. "Introductory Techniques for 3-D Computer Vision." Prentice Hall, 1998.

[2] Hartley, R; Zisserman, A. "Multiple View Geometry in Computer Vision." Cambridge University Press, 2003.

[3] Hartley, R. "In Defense of the Eight-Point Algorithm." IEEE® Transactions on Pattern Analysis and Machine Intelligence, v.19 n.6, June 1997.

[4] Fischler, MA; Bolles, RC. "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography." Comm. Of the ACM 24, June 1981.