bayesian super-resolution

Currently i am working on implementing a super-resolution algorithm which is based on bayesian image priors. This can be found in Capel's thesis on super-resolution. This method uses a generative model. The equations and ideas are almost identical to those in my previous post "failed start". Fortunately in capel's work he goes into grusome detail on many of the things which could not be understood in the other paper.

The generative model relies on the parameters that are found by the image registration algorithms, as well as some Bayesian image priors. Using these things and optimization it is then possible to solve for the super-resolution image.

mosaicing

I now have a working mosaicing algorithm. Here are the pictures.






These are the starting sequence of images. I got them by cropping a set of images I received thanks to kristin branson, a computer science graduate student at ucsd.







These are the four original images after we have found the homography that maps each image into the coordinate system defined by the fourth image.




Finaly, this is the composite mosaic. Currently this is done using a simple method of picking pixels for the final image based on poximity to the centers of the four transformed images. As can be seen this algorithm results in a blank spot in the the mosaic where clearly there are some overlapping images. There are several smarter ways to implement this, but since my focus is on super-resolution that will be lower on my priority list.

photometric registration

The next step in the super resolution algorithm is photometric registration. Since not all images are necesarily taken under the same lighting conditions it may be necesary to adjust for lighting changes. These lighting changes could be due to
1.)automatic camera adjustments
2.)illumination change.
It is assumed that there is a linear transformation between each of the red, green, blue, color channels in the corresponding images. This can then be written as follows:

It is then necesary to estimate the 6 photometric parameters. The first step in this process is doing geometric registration, and warping the images so that they are aligned. Since many model-outliers can occur due to shadows, saturation, or specularities, we must use a robust algorithm to do the line fitting. The one that Capel discusses is MSAC(a.k.a. MLESAC) which is a variation of RANSAC. The difference is that RANSAC adjusts the model parameters in order to maximize the number of inliers and MSAC adjusts the model parameters in order to maximize the likelyhood of getting the given data with the proposed model parameters.
I am currently working on understanding an implementation of MLESAC by Torr. Torr has this and several other image processing programs in matlab available at:



Hopefully this code will fit my needs and I will soon have the photometric registration portion of the super resolution algorithm working.

image registration finalized

At this point I have the image registration algorithm complete, and it seems to be running well. Here are the images for some of the steps.

Here is the first image and the interest points that were detected.


Here is the second image points and the interest points that were detected.


Here are the putative matches found by maximum correlation around interest points. These matches are then linked by a blue line and this line is plotted on the first image.


After running RANSAC we get the inliers, or most consistent interest points under a single homography. These points are then plotted on the first image and linked with a line.


After several rounds of optimization of the homography and finding new interest points matches based on the optimized homography we get a more accurate homograpy. This is the second image with its interest points plotted in green and the matching interest points from the first image under the homography plotted over the second image in red.



Finaly we have the first image under the found homography with it's original interest points plotted in green and it's transformed interest points plotted in red.

image registration progress

I found most of the code necessary for the image registration process. For steps 1.) features, 2.) Putative correspondences, 3.) RANSAC robust estimation, I was able to use code from Peter Kovesi, which can be found at:

http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/index.html

for the 4.) Optimal estimation step I am using a nonlinear maximum likelihood homography estimation program from code made available by Andrew Zisserman at:

http://www.robots.ox.ac.uk/~vgg/hzbook/code/


With a little bit of tweaking this code fits correctly. I also am using code that I got from Carolina Galleguillos, who got it from Serge Belongie, which lets me directly calculate homographies using user input and test the homographies that I get from the registration process.

Here are the initial results that I am getting. This is the image that we start with.




The next image is one that is computed using the code to calculate and apply homographies based on user input.



Using the image registration process to find a homography for the starting images and the second calculated image, then applying that transformation to the starting image this is what we get.



We can see that this is not exact because the lines for the section chosen are not completely square like in the second image. Finally after applying the optimazation step we end up with this image.




It is apparent from this example that the optimization step is actually making the homography less accurate. I am not sure why this is yet, but hopefully while implementint the last step 5.) Guided matching, and then iterating steps 4.) and 5.) as per the algorithm I will either discover a bug or the process will resolve the issues of inaccuracy.

My next steps are to finish the last portion of the registration algorithm, run it on a couple of more examples, and start the next steps in the super resolution algorithm. These steps will be photometric registration, and if that goes quickly, starting on the mosaicing step.

image registration algorithm

Here is the image registration algorithm that I will implement; it is taken from David Capel's thesis on super resolution (pg 29):


Objective: computer the 2D homography between two images
Algorithm:

1. Features: compute interest point feature in each image to sub pixel accuracy (eg Harris corners).

2. Putative correspondences: Compute a set of interest point matches based on proximity and similarity of their intensity neighborhood.

3. RANSAC robust estimation: repeat for N samples
a. select a random sample of 4 correspondences and compute the homography H.
b. Calculate a geometric image distance error for each putative correspondence.
c. Compute the number of inliers consistent with H by the number of correspondences for which the distance error is less than a threshold.

Choose the H with the largest number of inliers.

4. Optimal estimation: re-estimate H from all correspondences classified as inliers, by maximizing the likelihood function using a suitable numerical optimizer, such as the Levenburg-Marquardt algorithm.

5. Guided matching: further interest point correspondences are now determined using the estimated H to define a search region about the transferred point position.

a new angle

I’ve been looking at “Image Mosaicing and Super-resolution”, which is the thesis for David Peter Capel, on recommendation of Serge.

After consideration, I believe following this thesis will be more beneficial and a better use of my time than trying to use my previous papers. The reason that I believe this is because the thesis brings together all of the elements that I need to implement into one unified framework. Rather than trying to use bits and pieces that I’ve found in different papers, I can hopefully use a single paper, albeit a long one. In this way I’ll have everything I need to start off with in one place, and later on if I want to make improvements based on other papers I will be able to start from a solid foundation.

Based on this thesis, the three major steps I will need to implement are as follows:
1.) Image registration : aligning overlapping images accurately.
2.) Image mosaicing : putting multiple overlapping images into a single image.
3.) Super-resolution : using extra information gained from overlapping images to extract a higher resolution image which will contain more fine detail.

The next steps that I will take towards accomplishing these three items will be to implement the method of image registration from the thesis and apply it to synthetic low resolution image sets that I will create from high resolution images.

failed start

As my first step in starting this project I picked one paper:

super-resolution from multiple images having arbitrary mutual motion - zomet, peleg
-http://www.cs.huji.ac.il/~peleg/papers/supres-book.pdf

At first look this paper was promising as a basis for an algorithm. It had simple concise mathematical equations that made a lot of sense for the super-resolution step, and a straight forward explanation of the ideas behind their super-resolution algorithm. They use a model based algorithm which is basically as follows:
Low resolution images are the result of projecting a high resolution image onto a plane, then sampling.
The goal is to use these low resolution images to find a high resolution image to fit this model.

The actual formulation of the super resolution algorithm was as a large sparse optimization problem. First a form of steepest decent was talked about, then a more effective conjugant gradient method was discussed. Here is the basic formulation:



-xL is the nth low resolution image reordered as a vector.
-xH is the high resolution image reordered as a vector.
-Wn is the geometric warp matrix.
-Hn is the blurring matrix.
-D is the decimation matrix.

Then the solution could be found by minimizing

where A = DHnWn



Unfortunately much of the details that seemed clear in concept where hidden in this paper. Things such as the Wn geometric warp matrix where side stepped by saying that they were beyond the scope of the paper, which was really focused on the conjugant gradient portion of the super resolution algorithm.

I have begun following up on two of the papers that where referenced in the geometric transformation section:
“True multi-image alignment and its application to mosaicing and lens distortion correction” by H.S. Sawhney and R. Kumar
“Multi-Frame Estimation of Planar Motion” by L Zelnik-Manor and M. Irani

In the beggining...

Here is the first post for my blog. I'll start by linking the papers that I'm looking at and the page where i'll be getting the video files for my first data set.

video sequence:
-http://www.cs.huji.ac.il/~zomet/superResolution.html

papers:
Super-Resolution Enhancement of Text Image Sequences - Capel & zisserman
-http://www.robots.ox.ac.uk/~vgg/publications/papers/capel00.pdf
Automated mosaicin with super-resolution zoom - capel and zisserman
-http://citeseer.ist.psu.edu/cache/papers/cs/2287/http:zSzzSzimogen.robots.ox.ac.uk:20000zSz~vggzSzvggpaperszSzCapel98.pdf/capel98automated.pdf
Robust super resolution - zomet, rav-acha, peleg
-http://www.cs.huji.ac.il/~peleg/papers/cvpr01-RobustSR.pdf
super-resolution from multiple images having arbitrary mutual motion - zomet, peleg
-http://www.cs.huji.ac.il/~peleg/papers/supres-book.pdf
mosaicing from a translating camera, with efficient application of super-resolution - zomet (thesis)
-http://www.cs.huji.ac.il/~zomet/thesis.pdf