Imagine new car models still under development or buildings yet to be constructed, “appearing” in real space, as though they were really physically present, right in front of you. This is possible with Canon’s Mixed Reality (MR) technology, which integrates the real world and virtual world to deliver a realistic experience when viewing objects from any angle. Combining both optical and imaging technologies, Canon’s MR system offers solutions for a wide range of fields.

2022/12/26Featured Technology

#Pioneering Technology#Camera

The evolution of imaging technology is providing new kinds of visual experiences. Using a head-mounted display (HMD) that covers the viewer’s field of vision, people can enjoy all kinds of experiences wherever they are, such as a virtual trip around the world or immersion in a virtual world (like the metaverse) as an avatar.

Canon is currently engaged in the development of “Mixed Reality” technology which integrates virtual and real-world environments. This is a step beyond virtual reality (VR), which is created by using only computer-generated images. Merging the real and virtual worlds makes it possible to display life-size, 3D data of objects as though they were actually there.

One of the growing number of use cases for MR is at manufacturing sites. In recent years, 3D CAD (computer-aided design) has been used increasingly in the design process. One of the development techniques that has garnered attention is “front-loading development,” which uses 3D models instead of physical prototypes to identify potential problems at an earlier stage. MR is a key element of this technique, which helps speed up the manufacturing process and contribute to lowered costs. Using MR, it is possible to integrate a 3D model generated from CAD data into a real-world environment and display it there in life size, thereby enabling realistic inspections and checks in a way similar to the use of a physical mockup.

The applications of MR are not limited to just manufacturing. It can be used at construction sites to showcase a completed building to interested parties, at car dealerships to provide customers with new buying experiences by showing them colors and options that may not be available on the lot, and at sporting events to allow audiences to experience the excitement up-close.

“MREAL,” Canon’s system based on its MR technology, consists of the HMD, “Canon MREAL Platform,” base software and 3D display applications” including the “Canon MREAL Visualizer,” which displays 3D data. It adopts a video see-through system, which creates an MR image by merging in real time images of the real world captured by the camera mounted to the HMD with the virtual 3D CG image and projects the MR image onto the display panel of the HMD. Compared to the optical see-through system that merges images using a half mirror, video see-through offers many more advantages including accuracy and realism.

Consider another example: Imagine a hand positioned as though it were holding an object. If a virtual object were added to the image and to appear as if someone were holding it, the fingers positioned behind the object would be erased via the MREAL system’s image processing. The distance and positional relationship between your hand and the CG can be more accurately depicted. This helps MREAL meet the demands of design and manufacturing sites, where accuracy is critical.

Image processing based on the sense of distance in the real world, as well as the position, size and shape of the CG helps to create a visual effect as if the hand is really touching the CG object.

Canon’s unique display panel and the optical technologies cultivated throughout its long history have been incorporated into its HMDs, realizing smaller and lighter displays while maintaining high image quality. The MREAL X1 headset fits in the palm of your hand and weighs approximately 158g, reducing the burden on users. In order to smoothly integrate reality and CG, the camera and display panel are aligned with the wearer’s pupil. In addition, a thinner headset allows for closer positioning between the camera and the wearer’s pupil.

In order to provide a system that is comfortable and easy to use, the headset is designed to fit well and be easily taken on or off. This is made possible by capitalizing on our knowledge of ergonomics and the expertise has gained through development of the MREAL series thus far.

Yet another important consideration of MR is the positional alignment of the virtual world that is to be combined with the real world. For instance, when displaying a 3D CG model of a car, the MR view will be adversely affected if the positions are out of alignment or if the behavior is unstable. Previously, the system required special markers and optical sensors. Now, however, Canon has developed “Spatial Feature Positioning Technology” which uses stationary objects such as floor patterns and desks that enable positional alignment using only the HMD under specific conditions. This achievement increases the potential applications of the MR system, as it significantly reduces the work involved in setup and enables immediate use at outdoor locations where placement of sensors can be difficult.

Because MR technology is capable of smoothly integrating real-world and virtual world, it is now being utilized for a wide variety of purposes, including assessment of product design, improving manufacturing processes at plants or simulating the design of architectural structures. With this system, it is possible for users to see life-size representations of virtual objects from any angle. This new realism created by MR not only helps to reduce the need for reworking during development and design processes, but also leads to new insights in manufacturing.