photometric registration

The next step in the super resolution algorithm is photometric registration. Since not all images are necesarily taken under the same lighting conditions it may be necesary to adjust for lighting changes. These lighting changes could be due to
1.)automatic camera adjustments
2.)illumination change.
It is assumed that there is a linear transformation between each of the red, green, blue, color channels in the corresponding images. This can then be written as follows:

It is then necesary to estimate the 6 photometric parameters. The first step in this process is doing geometric registration, and warping the images so that they are aligned. Since many model-outliers can occur due to shadows, saturation, or specularities, we must use a robust algorithm to do the line fitting. The one that Capel discusses is MSAC(a.k.a. MLESAC) which is a variation of RANSAC. The difference is that RANSAC adjusts the model parameters in order to maximize the number of inliers and MSAC adjusts the model parameters in order to maximize the likelyhood of getting the given data with the proposed model parameters.
I am currently working on understanding an implementation of MLESAC by Torr. Torr has this and several other image processing programs in matlab available at:



Hopefully this code will fit my needs and I will soon have the photometric registration portion of the super resolution algorithm working.

image registration finalized

At this point I have the image registration algorithm complete, and it seems to be running well. Here are the images for some of the steps.

Here is the first image and the interest points that were detected.


Here is the second image points and the interest points that were detected.


Here are the putative matches found by maximum correlation around interest points. These matches are then linked by a blue line and this line is plotted on the first image.


After running RANSAC we get the inliers, or most consistent interest points under a single homography. These points are then plotted on the first image and linked with a line.


After several rounds of optimization of the homography and finding new interest points matches based on the optimized homography we get a more accurate homograpy. This is the second image with its interest points plotted in green and the matching interest points from the first image under the homography plotted over the second image in red.



Finaly we have the first image under the found homography with it's original interest points plotted in green and it's transformed interest points plotted in red.

image registration progress

I found most of the code necessary for the image registration process. For steps 1.) features, 2.) Putative correspondences, 3.) RANSAC robust estimation, I was able to use code from Peter Kovesi, which can be found at:

http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/index.html

for the 4.) Optimal estimation step I am using a nonlinear maximum likelihood homography estimation program from code made available by Andrew Zisserman at:

http://www.robots.ox.ac.uk/~vgg/hzbook/code/


With a little bit of tweaking this code fits correctly. I also am using code that I got from Carolina Galleguillos, who got it from Serge Belongie, which lets me directly calculate homographies using user input and test the homographies that I get from the registration process.

Here are the initial results that I am getting. This is the image that we start with.




The next image is one that is computed using the code to calculate and apply homographies based on user input.



Using the image registration process to find a homography for the starting images and the second calculated image, then applying that transformation to the starting image this is what we get.



We can see that this is not exact because the lines for the section chosen are not completely square like in the second image. Finally after applying the optimazation step we end up with this image.




It is apparent from this example that the optimization step is actually making the homography less accurate. I am not sure why this is yet, but hopefully while implementint the last step 5.) Guided matching, and then iterating steps 4.) and 5.) as per the algorithm I will either discover a bug or the process will resolve the issues of inaccuracy.

My next steps are to finish the last portion of the registration algorithm, run it on a couple of more examples, and start the next steps in the super resolution algorithm. These steps will be photometric registration, and if that goes quickly, starting on the mosaicing step.

image registration algorithm

Here is the image registration algorithm that I will implement; it is taken from David Capel's thesis on super resolution (pg 29):


Objective: computer the 2D homography between two images
Algorithm:

1. Features: compute interest point feature in each image to sub pixel accuracy (eg Harris corners).

2. Putative correspondences: Compute a set of interest point matches based on proximity and similarity of their intensity neighborhood.

3. RANSAC robust estimation: repeat for N samples
a. select a random sample of 4 correspondences and compute the homography H.
b. Calculate a geometric image distance error for each putative correspondence.
c. Compute the number of inliers consistent with H by the number of correspondences for which the distance error is less than a threshold.

Choose the H with the largest number of inliers.

4. Optimal estimation: re-estimate H from all correspondences classified as inliers, by maximizing the likelihood function using a suitable numerical optimizer, such as the Levenburg-Marquardt algorithm.

5. Guided matching: further interest point correspondences are now determined using the estimated H to define a search region about the transferred point position.