Tech Primer

Super Resolution

Super Resolution is a technique that uses multiple frames of the same object to create a higher resolution image.  It works only if the frames are shifted by fractions of a pixel from each other.  The super-resolution algorithm is able to produce a larger image that contains the information in the smaller original frames.

These techniques in some way enhance the resolution of an imaging system. There are different views as to what can be considered to be a SR technique. Some consider only techniques that break the diffraction-limit of systems, while others also consider techniques that merely break the limit of the digital imaging sensor as SR.

There are both single-frame and multiple-frame variants of SR, where multiple-frame are the most useful. Algorithms can also be divided by their domain: frequency or space domain. By fusing together several low-resolution (LR) one enhanced-resolution image is formed.

Sneaking into the algorithm

For a variety of applications, we want to be able to increase the resolution of images. The ideal algorithm should be fast, and should add sharpness and detail, both at edges and in regions of texture, without adding artefacts.

In one such algorithm, many pairs of high resolution, and low-resolution versions of the same image data are examined. Then each image is divided into patches, both high resolution and low-resolution. The patches are described as vectors in a continuous space, and then the probability densities are modelled as Gaussian mixtures.

Each patch of the low and high-resolution images is a node in a Markov network. Given some new image, we seek to infer the corresponding high-resolution image components. During inference, the prior and conditional distributions of the high-resolution data are evaluated, given the low-resolution observation. The high-resolution components are a sampling of those high-resolution components, which correspond to the observed low-resolution components at that node. This is taken as a “line-up of suspects”. Each node has its own set of suspects.

Each scene in a node’s lineup has in common the fact that it renders to the low-resolution observation at that node. Next is to evaluate the likelihoods by a set of belief propagation equations. The computation converges in just three iterations. The iterations themselves take about five seconds each. However, the set-up time prior to beginning the computation takes about one hour.

The route to HD

In a demonstration at Intel’s Santa Clara campus, the company ran an SR algorithm on a dual-core Intel machine to add resolution to a Webcam. The algorithm was able to add four times the resolution to captured video, with no noticeable delay. The algorithm is written to scale to the number of cores that it runs on: the more cores available, the more the processing is divided. Different regions of the video are assigned to different cores. Using this method, it would take about 150 cores to convert DVD video into HD DVD in real time.

As people start buying HDTV sets, they are finding that a lot of content hasn’t caught up with the resolution that these sets can provide. This could be a driver for the adoption of this technology. That said, to most people’s disappointment, at this point, Intel doesn’t have any plans to make the algorithm commercially available.

— Varun Aggarwal

For further information, visit:www.merl.com/projects/fastsuperres/