Mixed reality (MR) systems
Imagine new car models still under development or buildings yet to be constructed, “appearing” in real space, as though they were really physically present, right in front of you. This is possible with Canon’s Mixed Reality (MR) technology, which integrates the real world and virtual world to deliver a realistic experience when viewing objects from any angle. Combining both optical and imaging technologies, Canon’s MR system offers solutions for a wide range of fields.
The evolution of imaging technology is providing new kinds of visual experiences. Using a head-mounted display (HMD) that covers the viewer’s field of vision, people can enjoy all kinds of experiences wherever they are, such as a virtual trip around the world or immersion in a virtual world (like the metaverse) as an avatar.
Canon is currently engaged in the development of “Mixed Reality” technology which integrates virtual and real-world environments. This is a step beyond virtual reality (VR), which is created by using only computer-generated images. Merging the real and virtual worlds makes it possible to display life-size, 3D data of objects as though they were actually there.
Canon’s latest HMD is the MREAL X1. In addition to being compact and lightweight and offering high image quality, the X1 boasts the largest display in the MREAL series.
The MREAL X1 displays virtual 3D CG models as if they were really there, in life-size and from the wearer’s chosen viewpoint.
One of the growing number of use cases for MR is at manufacturing sites. In recent years, 3D CAD (computer-aided design) has been used increasingly in the design process. One of the development techniques that has garnered attention is “front-loading development,” which uses 3D models instead of physical prototypes to identify potential problems at an earlier stage. MR is a key element of this technique, which helps speed up the manufacturing process and contribute to lowered costs. Using MR, it is possible to integrate a 3D model generated from CAD data into a real-world environment and display it there in life size, thereby enabling realistic inspections and checks in a way similar to the use of a physical mockup.
The applications of MR are not limited to just manufacturing. It can be used at construction sites to showcase a completed building to interested parties, at car dealerships to provide customers with new buying experiences by showing them colors and options that may not be available on the lot, and at sporting events to allow audiences to experience the excitement up-close.
“MREAL,” Canon’s system based on its MR technology, consists of the HMD, “Canon MREAL Platform,” base software and 3D display applications” including the “Canon MREAL Visualizer,” which displays 3D data. It adopts a video see-through system, which creates an MR image by merging in real time images of the real world captured by the camera mounted to the HMD with the virtual 3D CG image and projects the MR image onto the display panel of the HMD. Compared to the optical see-through system that merges images using a half mirror, video see-through offers many more advantages including accuracy and realism.
Basic structure of MREAL
A system for viewing both the real world seen through the HMD and the CG image projected on the half-mirror at the same time. The CG image is translucent and does not look real. (Conceptual image)
Video See-through (MR)
Displays the image after combining the real-world video image captured by the MREAL’s sensor with the CG image. The CG image does not look translucent, thereby appearing more realistic.
Consider another example: Imagine a hand positioned as though it were holding an object. If a virtual object were added to the image and to appear as if someone were holding it, the fingers positioned behind the object would be erased via the MREAL system’s image processing. The distance and positional relationship between your hand and the CG can be more accurately depicted. This helps MREAL meet the demands of design and manufacturing sites, where accuracy is critical.
Image processing based on the sense of distance in the real world, as well as the position, size and shape of the CG helps to create a visual effect as if the hand is really touching the CG object.
Canon’s unique display panel and the optical technologies cultivated throughout its long history have been incorporated into its HMDs, realizing smaller and lighter displays while maintaining high image quality. The MREAL X1 headset fits in the palm of your hand and weighs approximately 158g, reducing the burden on users. In order to smoothly integrate reality and CG, the camera and display panel are aligned with the wearer’s pupil. In addition, a thinner headset allows for closer positioning between the camera and the wearer’s pupil.
High image quality and compact size
Optical axis alignment reduces the discordance of size and distance
A thinner headset better expresses an object’s actual size
In order to provide a system that is comfortable and easy to use, the headset is designed to fit well and be easily taken on or off. This is made possible by capitalizing on our knowledge of ergonomics and the expertise has gained through development of the MREAL series thus far.
Variable fitting has been improved by adding adjustments for the size of the wearer’s head, headset height, interpupillary distance (IPD). Angle adjustment (to establish field of view) and a flip-up mechanism allow the wearer to safely navigate real space and take notes while wearing the headset.
Optimal weight distribution and a redesigned padding that comes in contact with the front and back of the head provide a comfortable fit even when the wearer tilts their head.
Yet another important consideration of MR is the positional alignment of the virtual world that is to be combined with the real world. For instance, when displaying a 3D CG model of a car, the MR view will be adversely affected if the positions are out of alignment or if the behavior is unstable. Previously, the system required special markers and optical sensors. Now, however, Canon has developed “Spatial Feature Positioning Technology” which uses stationary objects such as floor patterns and desks that enable positional alignment using only the HMD under specific conditions. This achievement increases the potential applications of the MR system, as it significantly reduces the work involved in setup and enables immediate use at outdoor locations where placement of sensors can be difficult.
Conceptual image when using MR based on marker and sensor settings. (3D CG vehicle)
MREAL DISPLAY MD-20
Conceptual image when using MR based on Spatial Feature Positioning Technology. (The car on the left is real, while the car on the right is 3D CG.)
Spatial Feature Positioning Technology extracts the features from the image captured by the HMD to determine the positions of both the user and the 3D CG object.
Spatial feature positioning technology and a system configuration suitable for mobile workstations allow the system to be used outdoors or on work sites.
Because MR technology is capable of smoothly integrating real-world and virtual world, it is now being utilized for a wide variety of purposes, including assessment of product design, improving manufacturing processes at plants or simulating the design of architectural structures. With this system, it is possible for users to see life-size representations of virtual objects from any angle. This new realism created by MR not only helps to reduce the need for reworking during development and design processes, but also leads to new insights in manufacturing.