Canon has established proprietary deep learning image processing technology that enables corrections to issues inherent to photography

June 22, 2023

#Pioneering Technology#AI#Camera

Every moment is a slice of time. It won’t occur the same way again, but you can record it with your camera. Whether it was a stunning view you saw for the first time, or a special moment to cherish forever, your camera can capture such scenes and preserve them in photographs.

Until recently, however, photography had unavoidable image quality issues. They include, for example, noise that makes a photograph look grainy, moire that causes mottled patterns in the resulting photograph, and image blurring caused by the basic mechanism of the lens. These issues are a result of optical elements causing phenomena, not visible to the naked eye, to appear in photographs. For example, the peripheral areas of a wide-angle lens have lower optical performance than the center part. Therefore, images captured through that part of the lens are more likely to be blurred. This problem could not be prevented even by skilled photographers.

In recent years, AI technology has progressed tremendously, and deep learning¹ technology is now being used in many applications. As an industry-leading camera and lens manufacturer with unrivaled expertise in the field, Canon tackled the previously "unsolvable" problems of cameras caused by the principle of photography, by developing its own deep-learning image processing technology.

To achieve "true-to-life" photography, Canon used deep learning image technology to tackle three domains that had previously been considered inevitable: noise reduction, color interpolation, and aberration and diffraction correction (the correction of lens blur).

The accuracy of the results of deep-learning image-processing technology depends largely on how much training data, made from pairs comprising a teacher image and a student image, can be prepared.

Through decades of camera and lens development, Canon had accumulated an immense image database of every sort of subject possible. Furthermore, these images are in RAW format, which holds more information than other image file formats like JPEG. Canon was able to prepare a large amount of ideal training data using this image database, while also leveraging its in-depth knowledge as a camera manufacturers about how camera settings affect image quality.

Many noise reduction methods have been developed to date. When they were used, however, not only was the noise erased, but also the details of the subject, and as a result, image quality was often degraded. Thus, there were many requests for improvement from professional photographers. Unable to find a definitive solution, Canon decided to pursue noise-reduction image processing that could produce clear, high-quality images with deep learning, using the large amount of low-degradation images and hard-to-process images it possessed.

But deep learning can't solve every problem. In some shooting scenarios it performs "false corrections," the effects of which are in some cases worse than traditional image processing. To address this formidable challenge, Canon modified the architecture of the neural network, training process, and training data to reflect its own deep knowledge of the noise produced by cameras.

These efforts resulted in the establishment of the Neural network Noise Reduction function, which enables clear, high-quality results. The function suppresses the noise in high-ISO shooting, where noise as well as light signals are amplified as the sensitivity is increased, and it also improves the expression of smooth skin textures (skin tones), which is often damaged by noise.

Digital cameras have an inherent problem caused by a basic mechanism of digital photography, called moire, which are mottled patterns that appear in captured images when photographing subjects with repeated patterns, such as stripes and checkered patterns. One cause is the regular configuration of the pixels on a digital camera’s image sensor. The other cause is color interpolation.

One pixel in the image sensor generates pixel data by color interpolation that generates a set of the three primary colors of light: red (R), green (G), and blue (B), by detecting only one of them, and inferring the other two colors by referring to the information of surrounding pixels. Canon could not avoid the "false color, or color moire," that does not exist in the actual appearance of the subject and the "jaggies" that cause diagonal lines to look jagged, as both were caused by the basic mechanism of color interpolation. Similar to the issues in noise reduction, many methods of reducing such causes of image quality deterioration have been developed. Unfortunately, resolution perception and color reproduction inadvertently suffered at the same time in this process.

Using its rich image database, Canon established a deep learning image processing technology for color interpolation, which it named “Neural network Demosaic.”

The training dataset for that purpose was carefully constructed, taking into account the visual properties of human vision, which is sensitive to differences in brightness and less sensitive to changes in color.

Aiming to suppress false interpolation, the training data set was centered on the kinds of subjects that are difficult for correct interpolation in the color interpolation process. As a result, accurate interpolation is now possible even for such subjects as striped shirts that are prone to false colors, diagonal lines that are prone to jaggies, and pets that are prone to moire and false colors, thereby improving resolution perception and color reproduction.

A lens, the device through which a camera captures light, has theoretically unavoidable optical problems such as aberrations² and diffraction³ blurring.

High-performance lenses use combinations of concave and convex lens elements, aspherical lens elements, and special materials to minimize aberrations. However, eliminating them completely is theoretically impossible.

Canon’s Neural network Lens Optimizer, which is a deep learning image processing technology that corrects aberrations and diffraction, corrects the deterioration caused by such issues, significantly improving visual resolution.

What kind of aberrations and diffraction blur are there, and how do they happen?

For every lens that Canon produces, it has a complete grasp of the aberrations and diffraction blur of the lens during development and design.

By fully leveraging its simulation technology, Canon therefore could create a large number of high-precision pairs of student images with aberration and diffraction blurring and teacher images without those undesirable effects, based on such product data as design values for each lens model, and used them as training data for deep learning. By using these pairs, Canon succeeded in correcting the various types of blurring that occurred depending on the position of the image. This is especially useful when a user wants to take clear image of landscapes, celestial bodies, and other subjects, even in the periphery of the shooting area. Furthermore, unlike conventional methods, the technology corrects aberration and diffraction blurring without amplifying noise.

Blurring around blown-out highlights is noticeable, but such blurring is hard to remedy because the information needed for correction processing is lost. However, Canon’s aberration and diffraction correction also works for such blurring. But it was not easy to achieve. When a system is simply trained with training data consisting of images with and without blurring around blown-out highlights, the system may perform erroneous correction. Because little work had been done thus far to address blur correction around blown-out highlights, Canon needed to discover the problems associated with correction and clarify the principles of its occurrence. The company worked on improving training data through such means as using computer graphics and trial-and-error through various methods of architecture and post-processing of neural network. Through these efforts, Canon successfully achieved correction processing that also accurately corrects blurring around blown-out highlights.

The image correction methods achieved by deep learning image processing techniques in the three areas of noise reduction, color interpolation, and aberration and diffraction correction can perform better corrections when applied in combination, rather than when performed alone, thus greatly improving the finer details of image quality and dimensionality. For example, if the image data contains noise, aberration and diffraction correction is not effective in the finer details of the image. But when properly combined with noise reduction method, the performance of aberration and diffraction correction is maximized.

Accomplishing this required the development of algorithms to optimize the combination and order of the technologies. It is no exaggeration to say that Canon was uniquely positioned to achieve this image correction technology because its related departments ,such as camera development, lens design, image quality evaluation and product implementation are able to work closely together using actual equipment and products.

In the past, there were challenges that made it difficult to capture a photo exactly as you imagined it.

For example, when shooting night scenes, if you open up the lens aperture so that more light can be captured, the aberration reduces resolution, and if you narrow the aperture in response, the amount of light that can be captured decreases, forcing you to increase the ISO sensitivity, which in turn increases the amount of noise. Furthermore, when taking a picture of a landscape that is in focus all the way from the foreground to the background, if you narrow the aperture to increase the depth of focus, the diffraction blur will impair sharpness. But if you try to avoid this diffraction blur, you will not be able to expand the area of focus.

Canon's deep learning image processing technology enables image correction processing that was previously impossible, as well as significant reduction of aberrations, noise and moire.

This has greatly expanded the range of shooting scenarios and types of photographic expression available to users. Now, there are more situations where the user can shoot in exactly the setting they want without lowering the ISO sensitivity for fear of image quality degradation. It's easier than ever to recreate the "scene as it is" in fine detail.

Canon will continue to evolve its image processing technology to provide users with a shooting experience that will bring them joy.