Open videomosaicking.m in the Editor
Run in the Command Window

# Video Mosaicking

Video mosaicking is the process of stitching video frames together to form a comprehensive view of the scene. The resulting mosaic image is a compact representation of the video data, which is often used in video compression and surveillance applications.

## Contents

- Introduction
- Initialization
- Stream Processing Loop

## Introduction

This demo illustrates how to use the CornerDetector System object, the GeometricTransformEstimator System object, the AlphaBlender System object, and the GeometricTransformer System object to create a mosaic image from a video sequence. First, the demo identifies the corners in the first (reference) and second video frame. Then, it calculates the projective transformation matrix that best describes the transformation between corner positions in these frames. Finally, the demo overlays the second image onto the first image. The demo repeats this process to create a mosaic image of the video scene.

## Initialization

Use these next sections of code to initialize the required System objects.

    featWinLen   = 9;            % Length of feature window
    maxNumPoints = int32(75);    % Maximum number of points
    sizePano     = [400 680];
    origPano     = [5 60];
    classToUse   = 'single';

Create a VideoFileReader System object to read video from a file.

    hsrc = vision.VideoFileReader('vipmosaicking.avi', 'ImageColorSpace', ...
        'RGB', 'PlayCount', 1);

Create a ColorSpaceConverter System object to convert the RGB image to intensity format.

    hcsc = vision.ColorSpaceConverter('Conversion', 'RGB to intensity');

Create a CornerDetector System object to identify the corner locations of current and the previous video frames.

     hcornerdet = vision.CornerDetector( ...
        'Method', 'Local intensity comparison (Rosen & Drummond)', ...
        'IntensityThreshold', 0.1, 'MaximumCornerCount', maxNumPoints, ...
        'CornerThreshold', 0.001, 'NeighborhoodSize', [21 21]);

Create an GeometricTransformEstimator System object to implement the RANSAC (RANdom SAmple Consensus) algorithm. This System object verifies that the corners have been properly identified. It also calculates the transformation matrix that maps the corner positions in the current frame to the corner positions in previous frame.

    hestgeotform = vision.GeometricTransformEstimator;

Create a GeometricTransformer System object.

    hgeotrans = vision.GeometricTransformer( ...
        'OutputImagePositionSource', 'Property', 'ROIInputPort', true, ...
        'OutputImagePosition', [-origPano sizePano]);

Create a AlphaBlender System object to overlay the consecutive video frames to produce the panorama.

    halphablender = vision.AlphaBlender( ...
        'Operation', 'Binary mask', 'MaskSource', 'Input port');

Create a MarkerInserter System object to draw the corner locations in each video frame.

    hdrawmarkers = vision.MarkerInserter('Shape', 'Circle', ...
        'BorderColor', 'Custom', 'CustomBorderColor', [1 0 0]);

Create two VideoPlayer System objects, one to display the corners of each frame and other to draw the panorama.

    hVideo1 = vision.VideoPlayer('Name', 'Corners');
    hVideo1.Position(1) = hVideo1.Position(1) - 350;

    hVideo2 = vision.VideoPlayer('Name', 'Mosaic');
    hVideo2.Position(1) = hVideo1.Position(1) + 400;
    hVideo2.Position([3 4]) = [750 500];

## Stream Processing Loop

Create a processing loop to create panorama from the input video. This loop uses the System objects you instantiated above.

    points   = zeros([0 2], classToUse);
    features = zeros([0 featWinLen^2], classToUse);

    while ~isDone(hsrc)
        % Save the points and features computed from the previous image
        pointsPrev   = points;
        featuresPrev = features;

        % To speed up mosaicking, select and process every 5th image
        for i = 1:5
            rgb = step(hsrc);
            if isDone(hsrc)
                break;
            end
        end
        I = step(hcsc, rgb);
        roi = int32([2 2 size(I, 2)-2 size(I, 1)-2]);

        % Detect corners in the image
        cornerPoints = step(hcornerdet, I);
        cornerPoints = cast(cornerPoints, classToUse);

        % Extract the features for the corners
        [features, points] = extractFeatures(I, ...
            cornerPoints, 'BlockSize', featWinLen);

        % Match features computed from the current and the previous images
        indexPairs = matchFeatures(features, featuresPrev);

        % Check if there are enough corresponding points in the current and the
        % previous images
        isMatching = false;
        if size(indexPairs, 1) > 2
            matchedPoints     = points(indexPairs(:, 1), :);
            matchedPointsPrev = pointsPrev(indexPairs(:, 2), :);

            % Find corresponding points in the current and the previous images,
            % and compute a geometric transformation from the corresponding
            % points
            [tform, inlier] = step(hestgeotform, matchedPoints, matchedPointsPrev);

            if sum(inlier) >= 4
                % If there are at least 4 corresponding points, we declare the
                % current and the previous images matching
                isMatching = true;
            end
        end

        if isMatching
            % If the current image matches with the previous one, compute the
            % transformation for mapping the current image onto the mosaic
            % image
            xtform = xtform * [tform, [0 0 1]'];
        else
            % If the current image does not match the previous one, reset the
            % transformation and the mosaic image
            xtform = eye(3, classToUse);
            mosaic = zeros([sizePano,3], classToUse);
        end

        % Display the current image and the corner points
        cornerImage = step(hdrawmarkers, rgb, cornerPoints);
        step(hVideo1, cornerImage);

        % Warp the current image onto the mosaic image
        transformedImage = step(hgeotrans, rgb, xtform, roi);
        mosaic = step(halphablender, mosaic, transformedImage, ...
            transformedImage(:,:,1)>0);
        step(hVideo2, mosaic);
    end

    release(hsrc);

The Corners window displays the input video along with the detected corners and Mosaic window displays the panorama created from the input video.