Imagine new car models still under development or buildings yet to be constructed, “appearing” in real space, as though they were really physically present, right in front of you. This is possible with Canon’s Mixed Reality (MR) technology, which integrates the real world and virtual world to deliver a realistic experience when viewing objects from any angle. Combining both optical and imaging technologies, Canon’s MR system offers solutions for a wide range of fields.
The rapid evolution of imaging technology has led to the emergence of new visual experiences. By simply wearing the head-mounted display (HMD) which covers the user’s entire field of view, they will be able to soar through the sky, make a virtual trip around the world or enjoy any number of new experiences, all while physically remaining where they are.
Canon is currently engaged in the development of “Mixed Reality” technology which integrates virtual and real-world environments. This is a step beyond virtual reality (VR), which is created by using only computer-generated images. Merging the real and virtual worlds makes it possible to display life-size, 3-D data of objects as though they were actually there.
Canon’s latest HMD, “MREAL DISPLAY MD-20” is lightweight, delivers high image quality and offers a stress-free experience for users.
Virtual 3-D CG images can be displayed at life-size from any viewpoint, as if were physically there.
One of the growing number of use cases for MR is at manufacturing sites. In recent years, 3-D CAD (computer-aided design) has been used increasingly in the design process. One of the development techniques that has garnered attention is “front-loading development,” which uses 3-D models instead of physical prototypes to identify potential problems at an earlier stage. MR is a key element of this technique, which helps speed up the manufacturing process and contribute to lowered costs. Using MR, it is possible to integrate a 3-D model generated from CAD data into a real-world environment and display it there in life size, thereby enabling realistic verification and checks in a way similar to the use of a physical mockup.
“MREAL,” Canon’s system based on its MR technology, consists of the HMD, “Canon MREAL Platform,” base software and various “MREAL display applications” including the “Canon MREAL Visualizer,” which displays 3-D data. It adopts a video see-through system, which creates an MR image by merging in real time images of the real world captured by the camera mounted to the HMD with the virtual 3-D CG image and projects the MR image onto the display panel of the HMD. Compared to the optical see-through system that merges images using a half mirror, video see-through offers many more advantages including accuracy and realism.
Basic structure of MREAL
A system for viewing both the real world seen through the goggles and the CG image projected on the half-mirror at the same time. The CG image is translucent and does not look real. (Conceptual image)
Displays the image after combining the real-world video image captured by the MREAL’s sensor with the CG image. The CG image does not look translucent, thereby appearing more realistic.
Consider another example: Imagine a hand positioned as though it were holding an object. If a virtual object were added to the image and to appear as if someone were holding it, the fingers positioned behind the object would be erased via the MREAL system’s image processing. This makes possible a more faithful depiction of the sense of distance and color, and is thus meets the requirements of designs and manufacturing sites for which precision is an important consideration.
Image processing based on the sense of distance in the real world, as well as the position, size and shape of the CG helps to create a visual effect as if the hand is really touching the CG object.
In order to smoothly combine real-world and virtual images and produce images that are as realistic as what is seen by the naked eye, it is necessary to ensure that the viewpoint of the user and the optical axis of the camera are in alignment. To strike a balance between optical axis alignment and an HMD that is both compact and lightweight, Canon uses a “free-form prism” that helps to realize a compact design by folding the optical axis that extends to the screen, which displays the MR image, instead of using a prism that forms a straight line. In addition, the free-form prism allows for effective corrections, resulting in high image quality, and enables optimal positioning of such components as the display panel and camera lens. Manufacturing of the free-form prism requires an extremely high level of precision, and this is made possible thanks to the optical and processing technologies that Canon has accumulated over its long history.
Meanwhile, the HMD’s camera uses a CMOS sensor employing a global-shutter produced by Canon. This makes it capable of capturing moving scenery without distortion.
The HMD’s optical system
Yet another important consideration of MR is the positional alignment of the virtual world that is to be combined with the real world. For instance, when displaying a 3-D CG model of a car, the MR view will be adversely affected if the positions are out of alignment or if the behavior is unstable. Previously, the system required special markers and optical sensors. Now, however, Canon has developed “spatial characteristic-based positioning technology,” which uses stationary objects such as floor patterns and desks as markers that enable positional alignment using only the HMD under specific conditions. This achievement increases the potential applications of the MR system, as it significantly reduces the work involved in setup and enables immediate use at outdoor locations where placement of sensors can be difficult.
Conceptual image when using MR based on marker and sensor settings. (3-D CG vehicle)
Conceptual image when using MR based on spatial characteristic-based positioning technology. (The car on the left is real, while the car on the right is 3-D CG.)
Positioning based on spatial characteristics extracts the features from the image captured by the HMD to determine the positions of both the user and the 3-D CG object.
Because MR technology is capable of smoothly integrating real-world and virtual world, it is now being utilized for a wide variety of purposes, including assessment of product design, improving manufacturing processes at plants or simulating the design of architectural structures. With this system, it is possible for users to see life-size representations of virtual objects from any angle. This new realism created by MR not only helps to reduce the need for reworking during development and design processes, but also leads to new insights in manufacturing.