Based on the video analysis technology Canon has cultivated as a camera maker, People Counter is a software that can count up to approximately 1,500 people appearing in video footage^*1captured with network cameras. We decided to commercialize this software after it caused quite a stir at an international exhibition.

I was surprised by the demonstration of People Counter I saw; it was able to count several hundred people in real time. This software can also recognize the direction of movement of up to a total of several thousand people as it counts them, am I right? Can you give me some details about development of People Counter?

Naotsugu Sagawa
Sometimes you will see someone sitting in a chair on the sidewalk or in a train station who is counting the flow of people using a handheld click-counter. We started development of the underlying technology for this software that employs the face-detection technology Canon has developed for its digital cameras to automate this counting task and enhance work efficiency.

By that time, had the technology for counting people by recognizing their posture and silhouette already been established?

Sagawa
The technology had already been studied, but it hadn't yet reached a level suitable for practical use.

When it comes to face-detection technology, the average positions of the human eyes and mouth are generally fixed, so face detection based on image density information is possible. However, the human body comes in all different shapes and sizes, so detection using the same method won't work as well. The human body can look different depending on such factors as orientation, body size and clothing, so through repeated trial and error we determined which characteristics the software should use in order to effectively detect people.

What mechanism is used for detecting the human body?

Yuji Kaneda
To detect faces and the human body, we use a machine learning algorithm, or calculation method, which combines various characteristics to solve problems. The advent of this method led to a breakthrough in the face-detection function employed by digital cameras.

Sagawa
In order for it to detect people in an image, we have to enable the software to learn the image characteristics that differentiate people and other subjects in that image. For example, characteristics unique to the human body, such as the profile of the head and shoulders, are learned as such characteristics. To detect the human body with any degree of accuracy, it must learn anywhere from several hundred to several thousand image characteristics. A set of these image characteristics represents a human model, which is used for the detection of people.

Differentiation of human bodies using several different recognition models

In the human body detection process, the software calculates a probability score for whether or not such characteristics as a head and shoulders are present relative to the image in question. By summing up the scores, the software determines whether the detected subject is human or not. However, in crowds or other situations where people are standing in front of each other, it becomes harder to determine what is or isn't a person. And so we prepared human models that can be used when part of the body is obscured which, when compared with a reference probability score, enables the software to achieve highly accurate human body detection.

Parallel processing of human body detection and tracing

Tetsuhiro Funagi
In cases where the image resolution is high or a very large number of people are present in the video image, the processing time for human body detection can be prolonged. In such cases, it may be impossible to process all video images from the network camera in real time. To solve this problem, the software attempts to perform human body detection and tracing in parallel, thus enabling real-time processing. For tracing, the software predicts the motion of a detected figure and performs detection in the target area only. So, the software enables real-time processing by reducing the amount of information required to be processed. What's more, the software gathers information on the motion of the figures, enabling detection of the people's direction of movement. It can also count the number of people crossing a line in a specified period.

Sagawa
Development of the underlying technologies for this project had finished by around 2013. The focus then shifted to the product development phase. However, at the time, there were no similar products anywhere else in the world, and with such issues as product specifications and price range still undecided, we were unsure about how to commercialize it. In an effort to commercialize the product, we presented this software at various exhibitions to get some feedback from prospective customers. We heard comments like, "This technology is great!" But we didn't receive any feedback that would help lead to commercialization.

"INTERPOL World" in Singapore in 2015

The turning point was our participation in "INTERPOL World," an international security exhibition involving with worldwide investigation authorities and military personnel held in Singapore in 2015. This exhibition attracted many who were highly conscious of various problems and were seeking prompt, specific solutions. Several interested parties requested a trial of our technology.

What kinds of customers participated in the exhibition?

Sagawa
Attendees included those who need to count people in locations where people gather in large numbers, such as stadiums and parades, in order to deploy security guards to optimum positions.

At that time, we had had little experience carrying out proof-of-concept tests in Japan, so we were uncertain as to whether the software really would work as intended. However, we decided to conduct some tests to meet the strong demand we were seeing from prospective customers.

What feedback did you get from these tests?

Sagawa
At the beginning, we had planned to count about one hundred people in real time. However, the prospective customers weren't necessarily looking for real-time counting if instead they could count several thousand people at once.

Funagi
False detection of an object other than a human body is called a "detection error," and a failure to detect a human body is called a "detection failure." The prospective customer requested that detection failures be minimized, but was willing to accept detection errors. At the potential customer's request, we considered this approach for the first time.

Sagawa
To tell the truth, several of our competitors had participated in proof-of-concept tests in the past, but their test results did not reach an acceptable standard. In a test that took place over the course of around half a year, we made countless improvements in response to feedback from prospective customers, and in the end we cleared the standard criteria for a successful test to achieve commercialization. Following that result, customers expressed their desire to utilize the product and are currently using it in the field.

In a meeting following that, the team members involved celebrated this success with tears in their eyes having finally succeeded after six months of pressure. It was impressive that our team was able to create a solution product by working together, and it felt much like be on the winning team playing sports back in our school days.

We continued our trial-and-error approach as we developed People Counter, a revolutionary product that can instantaneously count several thousand people. How many tears were shed in this development effort?

Did you experience any difficulties in developing a technology that can accurately detect people with a variety of physical characteristics and wearing various types of clothing?

Sagawa
This was Canon's first such project, so we adopted a trial-and-error approach in our development and verification methods.

Funagi
Detection results can vary considerably depending on the shooting conditions. Detection can fail if the target is backlit, for example, and detection failure can occur with certain camera angles or with a black floor in the background.

Fumiko Tomatsu
We learned a lot while shooting in various situations; for example, when several people overlap in an image, those in the background are obscured, so we reconsidered the height of the network camera setup to ensure they are detected, or when the people are wearing clothing of similar colors, it was difficult to accurately detect them.

Sagawa
Of course, the software must be able to detect people other than just Asian people. So, I purchased a blond wig and wore it to better evaluate the detection result. (Laughs)

This was clearly a painstaking effort. (Laughs)

Sagawa
For the purpose of experimentation, we asked several hundred employees, as well as several hundred foreign nationals hired as extras, to walk around the company grounds and reserved space in a convenience store.

For machine learning, a large number of images of people taken from various angles are required. For this purpose, we developed dedicated camera equipment, consisting of several dozen cameras to capture images from multiple angles at once. In this way were able to efficiently capture a large quantity of images at once.

Tomatsu
However, even with the cooperation of employees, there were limits to how many times we could gather a large number of people in one place. So, we had to come up with a creative evaluation method.

Computer-generated image used for the evaluation

First, we had difficulty preparing an image depicting 7,000 people in order to evaluate the software. So, we created an image of 7,000 people by compiling an image using several different photos. This came as a surprise to those around us who weren't familiar with the particulars of the project. (Laughs)

This gave us a chance to prove that People Counter could correctly detect an image that had been generated on a computer. It's actually quite difficult to order hired extras to perform exactly the same action several times in a row. However, with a computer-generated image, camera and lighting positions can be easily changed in the same situation.

Sagawa
With tests conducted outside Japan in particular, cameras could not always be installed in the field. In such a case, we determined the dimensions of the area we were testing and reproduced the environment with computer graphics software. We cannot say that the simulation is completely accurate, but we were able to obtain an approximation of actual data. This formed the foundation on which we fine-tuned the software.

Mizuki Tajima
In order to obtain analysis results most suitable for the actual environments where customers would utilize the software, we prepared documents based on the verification and fine-tuning of our R&D division for our sales companies and field engineers to use that included such know-how as the height and angle that network cameras should be installed.

As a result of these efforts, we were able to launch People Counter in October 2016.

People Counter has already been adopted by some commercial facilities. As a result of expanding the product line and utilizing the software in various other systems, business operations have undergone considerable changes.

What is the product line for People Counter?

Hisashi Kawabayashi
In cooperation with Milestone Systems and AXIS, which has become a member of the Canon Group, we have expanded our product line by offering such video analysis software as People Counter Library^*2, People Counter for Milestone XProtect^*3, People Counter for ACAP^*4and other products.

In this project, I was responsible for development of People Counter for ACAP. For ACAP, which runs on a network camera, improving the speed was a challenge because fewer resources are available compared to PC software. To cope with this problem, we set out to increase the speed by improving both the people-counting algorithm and specifications.

AXIS and Canon engineers worked together to fine-tune the algorithm for AXIS network cameras and achieve a higher speed.

As for specifications, we added two functions that enable customers to increase the speed in certain situations. Specifically, one function uses Canon's moving-object detection technology to count people only in the portion of the video image where movement is detected; the other function sets a minimum required area for counting people and human body size. Working with the Design Division, we also created a GUI^*5that makes these functions easy to configure. Commercializing a product through trial and error together with people working in different sections—and even different companies—was a challenging but highly rewarding experience.

How do the customers who use People Counter employ this system?

Funagi
People Counter is used merely to count the number of people present in a certain location in a few scenarios. Most customers create new value by integrating the count data with other systems and data.

Tajima
To give a simple example, when the number of people waiting in line at a checkout counter exceeds a preset number, staff are notified by a signal lamp or other message to assign more counter staff to minimize the congestion. As a more advanced application, People Counter is integrated with an elevator system to count the people waiting in the elevator halls on each floor. The system can then allocate elevators, giving priority to floors with larger numbers of waiting passengers.

Conceptual image of integration with a POS system

Sagawa
In another example, People Counter is being used in combination with a POS system^*6. In a commercial facility, a network camera linked to People Counter has been installed at the entrance to count customers. By combining visitor numbers by day or by time period with POS data, you can visualize a correlation between when customers come to the store and how much they spend. Retailers can then utilize this data to plan sales promotion events targeting certain days of the week and time periods and verify their effectiveness at a later date^*7.

Tajima
In addition to counting people captured on video, People Counter also provides a function for counting people who cross a preset line in a video image.

For example, large-scale facilities such as shopping malls can monitor the number of people entering and exiting each doorway. This enables them to visualize the number of people inside the facility at a given time period and identify entrances with high traffic. Based on this information, People Counter can be used to plan shifts for security guards and other staff. It also helps them to determine the most effective locations and time periods for sales campaigns.

Is there also software that can estimate the ages and gender of those captured by a network camera?

Kaneda
We released Profile Analyzer in June 2017. This software is used to estimate human attributes such as age and gender. I was responsible for development of this underlying technology.

How does the software determine these attributes?

Kaneda
This software detects faces using the algorithm employed by People Counter. After face detection, the face characteristic detection technology identifies the position of the eyes, nose, and other parts of the face. Age and gender can then be estimated based on statistical information of feature values extracted from these positions. When developing Profile Analyzer, we were able to make use of the huge volume of images that Canon has thus far collected through testing.

Conceptual image of the Profile Analyzer human attribute estimation and results screen

What fields will video analysis technologies such as People Counter likely be applied to in the future?

Sagawa
We are now focused on surveillance and marketing applications. We also believe these technologies can be expanded to include medicine and nursing care in the future. For example, assume a case where an aged person in a senior care facility has fallen. This system can detect such an incident and issue a notification.

Furthermore, analysis targets are not limited to people. In agriculture, for example, this system can be used to detect colors, shapes and numbers of fruit growing on a plant in order to determine the appropriate amount of fertilizer and the timing of the harvest.

People Counter came into being after years of development. The realization of this product was the result of teamwork among the developers, who achieved results through repeated trial and error without being trapped in conventional ideas.

I'm sure the development of a revolutionary product like People Counter was far from straightforward. What do you feel when you reflect on this project?

Sagawa
The performance of People Counter can be affected by the location and angle at which a camera is installed. Therefore, to ensure adequate performance in a typical environment in which customers will actually use this system, we conducted several interviews with customers in order to better understand possible environments and usage conditions. While working to commercialize this product, we also conducted verification experiments for customers outside Japan. We were communicating with members of overseas sales offices on a daily basis, and we found it difficult to prepare English reference materials in such short order.

Through the experience of launching a new kind of product, we learned a great deal about developing products from the customer's viewpoint. Also, we were able to strengthen cooperation with our overseas sales offices, which we feel will be very useful when doing business in the future.

Funagi
People Counter is a software product that becomes even more useful to our customers when combined with other systems. Neither I nor anyone at Canon had ever experienced a software development project quite like this. However, after participating in the development of People Counter, I'm glad to know what it's like to approach things from a different angle and devise new solutions.

Tomatsu
Actually, I had been responsible for a biological study in my previous workplace, which is a field completely different from this project. This being my first software project, I was feeling my way, wondering how to prepare samples and how to evaluate them. Thanks to the help of the other members, I managed to create an evaluation system using graphic software. I expect more varied applications will be developed in the future, and I'm looking forward to devising new systems to evaluate such applications.

Kawabayashi
In developing applications such as People Counter that are designed to operate on network cameras, cooperation between Canon, overseas sales offices and Axis is essential. We all had different ideas and approaches at the beginning, but we were able to widen our perspectives and adopt each other's strong points. Going forward, I hope we can combine the best of the Canon Group's diverse values and strengths to develop truly great products.

Kaneda
Yes, collaborating with employees of Group companies outside to develop a new algorithm was an enriching experience. I belong to a division different from the other members, but I feel that the various divisions communicated quite well. I also came to realize that outstanding products are the result of collaboration that extends beyond the boundaries of an organization.

Tajima
Since I began participating in the People Counter project, there have been difficult times, but I have lived every day with a sense of accomplishment. All of us were feeling our way forward because People Counter is a product without precedent. After struggling to achieve success, I was very glad to see this product, which resulted from collaboration among members, launch on the market and be accepted by customers. I feel that it was very much worth the hard work.

The world of manufacturing has reached a significant turning point. Products that realize higher performance can be manufactured in larger quantities at lower costs. It's no exaggeration to say that the old era is coming to an end.

The way forward now is to create new value through unique products and services that were never before available; the key to creating this value is software.

The People Counter product introduced in this issue is a specialized software for counting people, but the video analysis technology it employs has enormous and wide-ranging potential. Used in combination with other solutions, this software may one day be used to detect and count animals and other nonhuman subjects, or even other types of measurements.

For example, "how many of a subject are present, where are they and in which direction are they moving?" Real situations may be monitored via software that can instantaneously determine the optimum distribution of resources and issue instructions.

In this way, business and society is expected to advance at an even greater pace.

For example, many people from around the world will be visiting Japan to attend large-scale, world-famous sporting events in 2020. At these events, this system will contribute to effective site management, for example, by determining the state of congestion and helping to guide spectators and assign the optimum numbers of staff and security guards.

Recently, in Japan there has been growing awareness of such issues as its aging society, low birth rate and manpower shortage. However, effective use of technology will allow us to create a brighter future. People Counter software can help us realize our potential.