Enlargement process that maintains the resolution of the original photo
Neural Network Upscaling Technology
Desiring the enlarged photo to be as high-resolution image quality as the original one when cropping the photo or making large format prints is probably something all photographers can relate to. These desires can now come true with Canon’s upscaling tool, which Canon developed with its proprietary deep learning image processing technology with particular attention to ensuring that scenes would be reproduced true-to-life down to the details.
November 13, 2023
Upscaling: Image processing to convert to a higher resolution
Upscaled photos might also seem slightly blurrier to many photographers who will crop their image for stronger composition, and also desire to enlarge the remaining pixels for print enlargement. Since the resolution of a photo is determined by the number of pixels in the camera, it's natural for a photo to look grainy if you enlarge the photo by simply enlarging each pixel.
To solve this problem, an image processing method called "upscaling" was developed to convert an image to a resolution higher than the original resolution. The key to upscaling is how to maintain apparent resolution. This means that simply increasing the resolution in terms of the number of pixels will not preserve the impression (i.e., the apparent resolution) that the details are well described.
Conventional upscaling often decreases the apparent resolution.
The most common method of upscaling to date has been bicubic interpolation - a sharpening and enlarging technique which estimates the necessary unknown data through the brightness and color information of the known data within an image. Bicubic interpolation increases the quantity of pixels by predicting the needed pixels to enlarge the image by sampling the color and brightness information not only from the neighboring pixels, but also the surrounding pixels to enlarge an image in a manner which makes it less likely that the subject's sharpness will be lost.
The challenge of traditional upscaling using bicubic interpolation is that as the image resolution is increased the boundary areas such as subject outlines, where white and black edges are visible, exhibit thicker outlines compared to the original image. Correcting for this effect by balancing the relationship between the thickness of the outlines and the apparent resolution conversely decreases the apparent resolution compared to the original image.
Achieving an upscaling process that preserves image quality while keeping outcomes “true to life”
Canon, in its position as a camera manufacturer that pursues image quality, had been exploring technology of improving image quality post-shooting. For example, it had produced and made available the Neural Network Image Processing Tool, which uses Canon’s proprietary deep learning image processing technology to correct noise, false colors, lens aberrations, and other issues inherent to photography, thereby enabling the reproduction of true-to-life images. Canon proceeded to develop an upscaling solution, certain that it would be able to find one by leveraging on its knowledge and experience with deep learning-based image processing and thorough expertise on cameras and lenses.
The result of these efforts is the released Neural Network Upscaling Tool, the information of pixels in the boundary part is also faithfully estimated with deep learning technology without over-correction. The software makes it possible to double the number of vertical and horizontal pixels and quadruple the total number of pixels without altering color, brightness, or noise. These upscaled images are as life-like as the original image while minimizing blur at the subject boundaries and additionally maintaining the natural sense of sharpness that is visible in the original image size.
The effect of upscaling while maintaining high apparent resolution is especially noticeable in fine animal fur, buildings or text with defined edges, or expansive wide-angle landscape photos. The tool can also be used with images captured with Canon cameras already on the market and cameras of other manufacturers. It makes the photos you have taken clearer and better defined—you would enjoy your memories of old fun times all the better for the clearer images.
You can enlarge a photo of a cliff while maintaining apparent resolution of the rock outline and rock surface.
Deep learning—enabled by Canon’s thorough expertise on cameras and lenses
In deep learning image processing, the key to improving the accuracy of the results, is training with enough quantity of paired "Student" and "Teacher" images. The "Student" images are images that would be desired to have correction, and the "Teacher" images are images that are the goal of the corrections.
Canon was able to fully leverage on the strengths it had built up in its decades of developing cameras and lenses, generating a massive training dataset consisting of a high-resolution "Teacher" image corresponding to the enlarged image and a lower-resolution "Student" image corresponding to an actual image captured by users in the real world.
In preparing a large quantity of high-quality "Training" data, Canon draws on two main advantages. The first is the vast image database accumulated over decades of camera and lens development. A large amount of "Training" data can be generated from high-resolution RAW data captured using a camera covering an array of subjects, and this data holds more information than other formats such as JPEG.
Another major advantage is Canon’s photography process simulation technology, which was established through the company’s experience and expertise from years of developing cameras and lenses. The photography process simulation has been supporting speedy product development. The technology provides a precise reproduction of the entire shooting process from the time the light captured by the lens enters and is turned into an image by the camera, and simulates the image quality.
The technology was able to reproduce differences depending on the camera, interchangeable lens model, shooting settings, etc., and it also showed great power in generating training dataset. Together, these two Canon advantages enabled generation of a vast amount of "Training" data for the development of deep learning image processing.
Upscaling producing true-to-life imagery down to the details
It is technically possible to increase the number of pixels further to achieve even high-resolution by deep learning technology. However, an image with an "excessive" apparent resolution enhancement may appear to have a high apparent resolution, but may contain structures that are not present in the subject. The image would no longer be considered a “true to life” depiction of the scene.
Canon developed an upscaling process that reduced deterioration in veracity while enhancing apparent resolution by combining its image processing technology cultivated through detailed understanding of the process through which images are produced by cameras with deep learning techniques. In other words, Canon was able to achieve a technology that could maintain the impression of original image even in enlarged print.
Canon will continue to advance its cameras and lenses, while also focusing on improving the post-shooting image quality, in pursuit of a shooting experience that makes customers feel happy.
You can enlarge a photo of a small bird while maintaining apparent resolution of its feathers.
Related article
Please see this document for detailed technical explanation.
PDF download
（PDF／8.84MB）
Related article
Please experience upscaling image processing.
Neural network Upscaling Tool