When did Canon start developing this MR System?

Toshiyuki Yasuda
Development of MR technology began in 1997 with the joint establishment of the Mixed Reality Systems Laboratory with the Japan Key Technology Center of the Ministry of International Trade and Industry (now the Ministry of Economy, Trade and Industry).

Was it your goal from the start to apply MR to manufacturing?

Ryoji Kondo
We decided to apply this technology in the industrial field, including manufacturing from the beginning. By around 2007, car manufacturers were using system prototypes and providing us with feedback.

Yasuda
It was in 2007 that, technologically speaking, we created a prototype of the current MREAL, with an HMD employing a free-form prism, alignment technology using markers, and external sensors.

Basic HMDs come in a variety of types. In terms of display method, they can be classified into three types: non-see-through, optical see-through and video see-through. Non-see-through HMDs don't allow wearers to see any of their surroundings while optical see-through type HMDs let users view directly through them by using half-silvered mirrors or other means. As for video see-through HMDs, although users can't directly see what's around them, built-in cameras and displays provide a view of the surroundings. Also, optical systems typically employed in video see-through HMDs may be simple in design, requiring users to peer through a lens at displays located in front of the eyes, or they may project images directly onto a user's retinas or make use of prisms. Canon, capitalizing on the optical technologies the company has accumulated over many years, developed a video see-through type HMD that employs a free-form prism. Despite its compact, lightweight body, Canon's HMD aligns the optical axes of the built-in cameras with the user's line of sight to seamlessly display both real and computer-generated images.

Why did you decide to use a video see-through type HMD employing a free-form prism for MREAL?

Kenji Hatori
With optical see-through type HMDs, the delay caused by CG rendering invariably results in a time lag between the real-world and computer-generated images. Such time lags, however, can be avoided by outputting images that have been previously synthesized by a PC, which results in the seamless merging of the real and virtual CG worlds. That's why video see-through HMDs are ideally suited for mixed reality.

Kondo
One of the reasons we used a free-form prism to project the images was to keep the HMD small. Another was to ensure a natural viewing experience. In order to achieve images that appear no different from those seen by the naked eye, MREAL captures real-world images by the HMD's built-in video cameras. If these video cameras aren't positioned correctly, the effect would be drastically different from what the user would usually see, which would feel quite strange. So, we went with a compact free-form prism, created some space, and then optimized the positioning of the video cameras to achieve a comfortable viewing experience. Although this approach isn't necessary for viewing just CG images, as is the case with VR (virtual reality), we currently believe this is the best way to provide users with a highly realistic MR experience.

Free-form prisms aren't used very much in other optical devices, are they?

How did you design a prism with such a complicated shape?

Kondo
In recent years, computer simulations have come to play an important role in designing optical instruments. When designing our free-form prism, we used computer simulations to calculate the optimal angles and distances between surfaces and to determine the optimal surface shape in order to achieve the necessary performance. Nevertheless, designing a special optical component like a free-form prism poses a distinct set of difficulties not usually faced when designing an ordinary lens, and the process can be tricky.

Why is that?

This means that if the position at which images appear in proper focus is fixed at a pinpoint location, then attaining proper alignment when putting on the HMD will be problematic. Images must remain in focus throughout a certain range so they can promptly be seen by users as soon as they put on the HMD. It wouldn't do for the image quality to suffer every time the HMD shifted around a little. We put a lot of effort into striking a balance between ease of use and optical performance.

Our prism requires an exceptionally high level of accuracy not exceeding one hundredth of a millimeter (0.01 mm). But Canon has consistently worked with optical technology since the company's founding and has accumulated the necessary know-how. I believe this was the key to our success.

When I gave MREAL a try, I was able to see images as soon as I put on the HMD and hardly had to adjust it at all.

The MR System is more than just an HMD. Capturing real-world images and seamlessly merging them with CG images also requires precision alignment technology. To this end, Canon developed proprietary MREAL Markers incorporating hexagonal patterns. Furthermore, with the addition of various external sensors, such as gyro sensors and infrared sensors, the MR System can be applied to a variety of environments.

In order to seamlessly merge the real world with the virtual world of CG images, the MR System must accurately align both sets of images. To do this, I've heard that MREAL uses markers as well as Gyro sensors and other kinds of sensors.

Hatori
In the early stages of development, we used only sensors, but sensors are expensive. In order to provide our MR System to customers at the most reasonable price, we developed an alternative technology: the MREAL Marker. Because the built-in video cameras in the HMD capture real-world images, we came up with the idea of affixing markers on objects which, when viewed, enable the detection of each object's orientation and location.

MREAL Markers can be used not only as spatial coordinates by affixing them to walls or the floor, but also in the verification process of virtual tools by attaching them to mock-ups of the tools, enabling CG images to be overlaid on them. MREAL Markers make possible the identification of 2,048 ID patterns, enabling users to improve alignment accuracy by combining several markers.

The MREAL Markers consist of hexagonal patterns. Why did you choose hexagons?

Hatori
Although the system can recognize any shape, including circles, triangles and squares, in order to make the proportions of the markers as small as possible, we chose hexagons, which offer the greatest area efficiency. In actual use, the markers don't always face the cameras directly. Over the course of development, we found hexagons to be the most easily recognizable shape even when viewed at an angle.

When looking at the markers through the HMD, the CG images are displayed right in that location. What gave you the most trouble during development?

Hatori
I'd have to say speeding up processing. The system has to display natural images even when the user walks around wearing the HMD. It was a challenge for us to develop an algorithm (calculation procedure) to perform the high-speed processing of the entire series of necessary steps involved, namely the image-capture, recognition and position calculation of the markers, as well as the display of the CG images.
But it wasn't simply enough to speed up processing. Markers aren't always captured under ideal conditions. For example, they may be viewed at an angle, or viewed in a dark environment. As such, some kind of correction needs to be added depending on the situation, but that slows down the entire process, which made getting the proper balance quite difficult.

Yuji Takayama
I was responsible for the electrical design. With MREAL, images captured by the HMD's cameras are sent to a PC, where they are synthesized with CG images and sent back to the displays in the HMD. This means that the HMD sends and receives data over a total of four channels: one input signal per eye for the real-world images, and one output signal per eye for the displays. That's a lot of data. In addition, the HMD and the PC are connected by a cable that's about 10 meters long. With a cable that long, you're going to have signal degradation, so it was a real challenge to design an HMD that could accurately send and receive signals without any delays.

Using sensors along with markers was a unique approach.

Hatori
MREAL Markers were made to be recognizable even in less than ideal conditions but—although an extreme scenario—they cannot be seen in complete darkness. I believe one of the real advantages of MREAL is that markers can be combined with infrared sensors and gyro sensors in accordance with the environment in which the system is used.

When I tried out MREAL just now, I put my hand into a computer-generated ring and it appeared just as though it was inside the ring. How did you get the system to recognize the relational positioning of objects?

That's why we designed MREAL to recognize positional relationships. By allowing users to pre-register certain colors, the system can control whether or not CG images are displayed. In the future, we'd like to achieve the accurate recognition of positional relationships without having to register colors in advance.

The automobile, building and construction industries are already making use of MREAL to support their design and production efforts. In the future, project team members may one day work together while sharing full-scale images of some enormous object.

In what fields is MREAL being used?

Because the system seamlessly merges the real and virtual computer-generated worlds, I find myself wanting to reach out and touch the CG images.

Hatori
When I'm absorbed in software development, I sometimes feel like touching the CG images myself.

Yasuda
You could call that an occupational hazard (laughs).

What kind of changes do you think the spread of MREAL and similar MR systems could bring about in the ways we work and live?

Hatori
In conventional design work, each engineer usually works alone on whatever he or she has been assigned. But, if MR systems were to come down in price and become more lightweight, we could very well see all engineers wearing HMDs, doing their work while discussing this aspect and that aspect of the project among themselves.

Takayama
Although videophones already exist, communication can be further enhanced by seeing images that incorporate CG images. For example, you could remotely offer instruction to people in another location while taking part in a meeting.

Yasuda
If MR can make you feel as though you've taken a trip, who knows, maybe we'll see more people giving up travel (laughs).

Takayama
For people lacking freedom of movement, MR could help expand their world. I believe there are still many untapped needs for MR systems, which could change human behavior. By making the HMD easier to put on and wear, we could enhance not only business communication, but also the way we communicate with various people, which would make the world more enjoyable.

Now more compact and easier to transport, the MR System has begun to play an active role in manufacturing.
If HMDs can be made even smaller than they are now and the real and virtual worlds can be completely merged, MR systems could dramatically change not only the world of manufacturing, but also the way we live.
The enthusiasm of these Canon engineers is opening the door to the new and exciting world of the unknown.