The revolutionary technology that instantaneously counts up to several thousand people The Story behind People Counter Video Analysis Software
It is now possible to count people instantaneously from video images captured by network cameras at such locations as stadiums, commercial facilities and transport facilities. This has the potential to eliminate long lines and other issues associated with overcrowding. How did the developers meet the challenge of creating this revolutionary video analysis technology?
- Product Planning
- Product Planning
- Software Development
- Software Development
- Software Development
- Software Development
Introducing a technology that the world has been waiting for: people-counting software
Based on the video analysis technology Canon has cultivated as a camera maker, People Counter is a software that can count up to approximately 1,500 people appearing in video footage*1captured with network cameras. We decided to commercialize this software after it caused quite a stir at an international exhibition.
- The number of people that can be counted when using a Full HD (2 megapixel) network camera. A maximum of 7,000 people can be counted from 4K still images.
In charge of product planning
Mr. Sagawa joined the company as a software engineer in 2003 and was responsible for image processing and image recognition technologies. He is currently responsible for product planning. As an engineer, he adheres to the aphorism, "Excellent evaluation environments create excellent technology," and when it comes to product planning, he believes "The best products are those that fill the world with wonder and bring smiles to people's faces."
I was surprised by the demonstration of People Counter I saw; it was able to count several hundred people in real time. This software can also recognize the direction of movement of up to a total of several thousand people as it counts them, am I right? Can you give me some details about development of People Counter?
Sometimes you will see someone sitting in a chair on the sidewalk or in a train station who is counting the flow of people using a handheld click-counter. We started development of the underlying technology for this software that employs the face-detection technology Canon has developed for its digital cameras to automate this counting task and enhance work efficiency.
By that time, had the technology for counting people by recognizing their posture and silhouette already been established?
The technology had already been studied, but it hadn't yet reached a level suitable for practical use.
When it comes to face-detection technology, the average positions of the human eyes and mouth are generally fixed, so face detection based on image density information is possible. However, the human body comes in all different shapes and sizes, so detection using the same method won't work as well. The human body can look different depending on such factors as orientation, body size and clothing, so through repeated trial and error we determined which characteristics the software should use in order to effectively detect people.
In charge of software development
Since joining the company in 2002, Mr. Kaneda has been responsible for developing image recognition software for to detect faces and the human body. He keeps an eye on various technologies and keeps an open mind to how he can effectively adopt them.
What mechanism is used for detecting the human body?
To detect faces and the human body, we use a machine learning algorithm, or calculation method, which combines various characteristics to solve problems. The advent of this method led to a breakthrough in the face-detection function employed by digital cameras.
In order for it to detect people in an image, we have to enable the software to learn the image characteristics that differentiate people and other subjects in that image. For example, characteristics unique to the human body, such as the profile of the head and shoulders, are learned as such characteristics. To detect the human body with any degree of accuracy, it must learn anywhere from several hundred to several thousand image characteristics. A set of these image characteristics represents a human model, which is used for the detection of people.
In the human body detection process, the software calculates a probability score for whether or not such characteristics as a head and shoulders are present relative to the image in question. By summing up the scores, the software determines whether the detected subject is human or not. However, in crowds or other situations where people are standing in front of each other, it becomes harder to determine what is or isn't a person. And so we prepared human models that can be used when part of the body is obscured which, when compared with a reference probability score, enables the software to achieve highly accurate human body detection.
In charge of software development
After joining the company in 2002, Mr. Funagi was responsible for developing network camera video recording applications. He is now in charge of product development for People Counter. He adheres to the motto, "Whether it's writing, music or rhythm, everything beautiful, graceful and excellent is simple."
In cases where the image resolution is high or a very large number of people are present in the video image, the processing time for human body detection can be prolonged. In such cases, it may be impossible to process all video images from the network camera in real time. To solve this problem, the software attempts to perform human body detection and tracing in parallel, thus enabling real-time processing. For tracing, the software predicts the motion of a detected figure and performs detection in the target area only. So, the software enables real-time processing by reducing the amount of information required to be processed. What's more, the software gathers information on the motion of the figures, enabling detection of the people's direction of movement. It can also count the number of people crossing a line in a specified period.
Development of the underlying technologies for this project had finished by around 2013. The focus then shifted to the product development phase. However, at the time, there were no similar products anywhere else in the world, and with such issues as product specifications and price range still undecided, we were unsure about how to commercialize it. In an effort to commercialize the product, we presented this software at various exhibitions to get some feedback from prospective customers. We heard comments like, "This technology is great!" But we didn't receive any feedback that would help lead to commercialization.
The turning point was our participation in "INTERPOL World," an international security exhibition involving with worldwide investigation authorities and military personnel held in Singapore in 2015. This exhibition attracted many who were highly conscious of various problems and were seeking prompt, specific solutions. Several interested parties requested a trial of our technology.
What kinds of customers participated in the exhibition?
Attendees included those who need to count people in locations where people gather in large numbers, such as stadiums and parades, in order to deploy security guards to optimum positions.
At that time, we had had little experience carrying out proof-of-concept tests in Japan, so we were uncertain as to whether the software really would work as intended. However, we decided to conduct some tests to meet the strong demand we were seeing from prospective customers.
What feedback did you get from these tests?
At the beginning, we had planned to count about one hundred people in real time. However, the prospective customers weren't necessarily looking for real-time counting if instead they could count several thousand people at once.
False detection of an object other than a human body is called a "detection error," and a failure to detect a human body is called a "detection failure." The prospective customer requested that detection failures be minimized, but was willing to accept detection errors. At the potential customer's request, we considered this approach for the first time.
To tell the truth, several of our competitors had participated in proof-of-concept tests in the past, but their test results did not reach an acceptable standard. In a test that took place over the course of around half a year, we made countless improvements in response to feedback from prospective customers, and in the end we cleared the standard criteria for a successful test to achieve commercialization. Following that result, customers expressed their desire to utilize the product and are currently using it in the field.
In a meeting following that, the team members involved celebrated this success with tears in their eyes having finally succeeded after six months of pressure. It was impressive that our team was able to create a solution product by working together, and it felt much like be on the winning team playing sports back in our school days.
Studying a Computer-Generated Image of Several Thousand People
Product development through repeated trial and error
We continued our trial-and-error approach as we developed People Counter, a revolutionary product that can instantaneously count several thousand people. How many tears were shed in this development effort?
Did you experience any difficulties in developing a technology that can accurately detect people with a variety of physical characteristics and wearing various types of clothing?
This was Canon's first such project, so we adopted a trial-and-error approach in our development and verification methods.
Detection results can vary considerably depending on the shooting conditions. Detection can fail if the target is backlit, for example, and detection failure can occur with certain camera angles or with a black floor in the background.
In charge of software development
Ms. Tomatsu joined the company in 2005. After working in the R&D division, she has been responsible for the performance improvement study of our People Counter product and involved in preparing computer-generated graphics since 2015. Her motto is "Keep an open mind and avoid being caught up in conventional ways of thinking."
We learned a lot while shooting in various situations; for example, when several people overlap in an image, those in the background are obscured, so we reconsidered the height of the network camera setup to ensure they are detected, or when the people are wearing clothing of similar colors, it was difficult to accurately detect them.
Of course, the software must be able to detect people other than just Asian people. So, I purchased a blond wig and wore it to better evaluate the detection result. (Laughs)
This was clearly a painstaking effort. (Laughs)
For the purpose of experimentation, we asked several hundred employees, as well as several hundred foreign nationals hired as extras, to walk around the company grounds and reserved space in a convenience store.
For machine learning, a large number of images of people taken from various angles are required. For this purpose, we developed dedicated camera equipment, consisting of several dozen cameras to capture images from multiple angles at once. In this way were able to efficiently capture a large quantity of images at once.
However, even with the cooperation of employees, there were limits to how many times we could gather a large number of people in one place. So, we had to come up with a creative evaluation method.
First, we had difficulty preparing an image depicting 7,000 people in order to evaluate the software. So, we created an image of 7,000 people by compiling an image using several different photos. This came as a surprise to those around us who weren't familiar with the particulars of the project. (Laughs)
This gave us a chance to prove that People Counter could correctly detect an image that had been generated on a computer. It's actually quite difficult to order hired extras to perform exactly the same action several times in a row. However, with a computer-generated image, camera and lighting positions can be easily changed in the same situation.
With tests conducted outside Japan in particular, cameras could not always be installed in the field. In such a case, we determined the dimensions of the area we were testing and reproduced the environment with computer graphics software. We cannot say that the simulation is completely accurate, but we were able to obtain an approximation of actual data. This formed the foundation on which we fine-tuned the software.
In charge of product planning
Ms. Tajima joined the company in 2013 and gained three years of experience in procurement, production management and accounting as a trainee at a cartridge component factory. She is currently responsible for product planning for video analysis solutions. She is committed to confirming things with her own eyes and describing things in her own words.
In order to obtain analysis results most suitable for the actual environments where customers would utilize the software, we prepared documents based on the verification and fine-tuning of our R&D division for our sales companies and field engineers to use that included such know-how as the height and angle that network cameras should be installed.
As a result of these efforts, we were able to launch People Counter in October 2016.
Offering a Wide Range of Innovative Business Applications, from Crime Prevention to Marketing
People Counter has already been adopted by some commercial facilities. As a result of expanding the product line and utilizing the software in various other systems, business operations have undergone considerable changes.
In charge of software development
Mr. Kawabayashi joined the company in 2015. After developing network camera applications, he was put in charge of developing People Counter for ACAP. His motto is "Maintain interest in a wide variety of subjects and an appetite for knowledge." He also believes in "giving top priority to rapid commercialization."
What is the product line for People Counter?
In cooperation with Milestone Systems and AXIS, which has become a member of the Canon Group, we have expanded our product line by offering such video analysis software as People Counter Library*2, People Counter for Milestone XProtect*3, People Counter for ACAP*4and other products.
In this project, I was responsible for development of People Counter for ACAP. For ACAP, which runs on a network camera, improving the speed was a challenge because fewer resources are available compared to PC software. To cope with this problem, we set out to increase the speed by improving both the people-counting algorithm and specifications.
AXIS and Canon engineers worked together to fine-tune the algorithm for AXIS network cameras and achieve a higher speed.
As for specifications, we added two functions that enable customers to increase the speed in certain situations. Specifically, one function uses Canon's moving-object detection technology to count people only in the portion of the video image where movement is detected; the other function sets a minimum required area for counting people and human body size. Working with the Design Division, we also created a GUI*5that makes these functions easy to configure. Commercializing a product through trial and error together with people working in different sections—and even different companies—was a challenging but highly rewarding experience.
- Targeting system integrators, this product provides an API for integration with external system functions. It launches People Counter from an external system and provides analysis results via a library in real time.
- People Counter is available as a plug-in system for use with XProtect, a video control software product from Milestone Systems.
- Axis Camera Application Platform; an application that enables the counting of people through an AXIS network camera.
- A Graphical User Interface is a method of using images and figures to present information. It enables users to execute most basic operations by interacting with specific positions on a screen.
How do the customers who use People Counter employ this system?
People Counter is used merely to count the number of people present in a certain location in a few scenarios. Most customers create new value by integrating the count data with other systems and data.
To give a simple example, when the number of people waiting in line at a checkout counter exceeds a preset number, staff are notified by a signal lamp or other message to assign more counter staff to minimize the congestion. As a more advanced application, People Counter is integrated with an elevator system to count the people waiting in the elevator halls on each floor. The system can then allocate elevators, giving priority to floors with larger numbers of waiting passengers.
In another example, People Counter is being used in combination with a POS system*6. In a commercial facility, a network camera linked to People Counter has been installed at the entrance to count customers. By combining visitor numbers by day or by time period with POS data, you can visualize a correlation between when customers come to the store and how much they spend. Retailers can then utilize this data to plan sales promotion events targeting certain days of the week and time periods and verify their effectiveness at a later date*7.
In addition to counting people captured on video, People Counter also provides a function for counting people who cross a preset line in a video image.
For example, large-scale facilities such as shopping malls can monitor the number of people entering and exiting each doorway. This enables them to visualize the number of people inside the facility at a given time period and identify entrances with high traffic. Based on this information, People Counter can be used to plan shifts for security guards and other staff. It also helps them to determine the most effective locations and time periods for sales campaigns.
- A Point of Sale system records product sales by store. The collected results can be used as reference for inventory management and marketing data.
- When customers are filmed during a verification experiment, the purpose and period of data collection is posted near the camera location so that customers are clearly informed. Canon is committed to protecting consumer privacy; for example, only analytical data (such as number of people, age and sex) is stored; no images are stored.
Is there also software that can estimate the ages and gender of those captured by a network camera?
We released Profile Analyzer in June 2017. This software is used to estimate human attributes such as age and gender. I was responsible for development of this underlying technology.
How does the software determine these attributes?
This software detects faces using the algorithm employed by People Counter. After face detection, the face characteristic detection technology identifies the position of the eyes, nose, and other parts of the face. Age and gender can then be estimated based on statistical information of feature values extracted from these positions. When developing Profile Analyzer, we were able to make use of the huge volume of images that Canon has thus far collected through testing.
What fields will video analysis technologies such as People Counter likely be applied to in the future?
We are now focused on surveillance and marketing applications. We also believe these technologies can be expanded to include medicine and nursing care in the future. For example, assume a case where an aged person in a senior care facility has fallen. This system can detect such an incident and issue a notification.
Furthermore, analysis targets are not limited to people. In agriculture, for example, this system can be used to detect colors, shapes and numbers of fruit growing on a plant in order to determine the appropriate amount of fertilizer and the timing of the harvest.
The key to creating a revolutionary product is to engage in teamwork that transcends the boundaries of an organization.
People Counter came into being after years of development. The realization of this product was the result of teamwork among the developers, who achieved results through repeated trial and error without being trapped in conventional ideas.
I'm sure the development of a revolutionary product like People Counter was far from straightforward. What do you feel when you reflect on this project?
The performance of People Counter can be affected by the location and angle at which a camera is installed. Therefore, to ensure adequate performance in a typical environment in which customers will actually use this system, we conducted several interviews with customers in order to better understand possible environments and usage conditions. While working to commercialize this product, we also conducted verification experiments for customers outside Japan. We were communicating with members of overseas sales offices on a daily basis, and we found it difficult to prepare English reference materials in such short order.
Through the experience of launching a new kind of product, we learned a great deal about developing products from the customer's viewpoint. Also, we were able to strengthen cooperation with our overseas sales offices, which we feel will be very useful when doing business in the future.
People Counter is a software product that becomes even more useful to our customers when combined with other systems. Neither I nor anyone at Canon had ever experienced a software development project quite like this. However, after participating in the development of People Counter, I'm glad to know what it's like to approach things from a different angle and devise new solutions.
Actually, I had been responsible for a biological study in my previous workplace, which is a field completely different from this project. This being my first software project, I was feeling my way, wondering how to prepare samples and how to evaluate them. Thanks to the help of the other members, I managed to create an evaluation system using graphic software. I expect more varied applications will be developed in the future, and I'm looking forward to devising new systems to evaluate such applications.
In developing applications such as People Counter that are designed to operate on network cameras, cooperation between Canon, overseas sales offices and Axis is essential. We all had different ideas and approaches at the beginning, but we were able to widen our perspectives and adopt each other's strong points. Going forward, I hope we can combine the best of the Canon Group's diverse values and strengths to develop truly great products.
Yes, collaborating with employees of Group companies outside to develop a new algorithm was an enriching experience. I belong to a division different from the other members, but I feel that the various divisions communicated quite well. I also came to realize that outstanding products are the result of collaboration that extends beyond the boundaries of an organization.
Since I began participating in the People Counter project, there have been difficult times, but I have lived every day with a sense of accomplishment. All of us were feeling our way forward because People Counter is a product without precedent. After struggling to achieve success, I was very glad to see this product, which resulted from collaboration among members, launch on the market and be accepted by customers. I feel that it was very much worth the hard work.
The world of manufacturing has reached a significant turning point. Products that realize higher performance can be manufactured in larger quantities at lower costs. It's no exaggeration to say that the old era is coming to an end.
The way forward now is to create new value through unique products and services that were never before available; the key to creating this value is software.
The People Counter product introduced in this issue is a specialized software for counting people, but the video analysis technology it employs has enormous and wide-ranging potential. Used in combination with other solutions, this software may one day be used to detect and count animals and other nonhuman subjects, or even other types of measurements.
For example, "how many of a subject are present, where are they and in which direction are they moving?" Real situations may be monitored via software that can instantaneously determine the optimum distribution of resources and issue instructions.
In this way, business and society is expected to advance at an even greater pace.
For example, many people from around the world will be visiting Japan to attend large-scale, world-famous sporting events in 2020. At these events, this system will contribute to effective site management, for example, by determining the state of congestion and helping to guide spectators and assign the optimum numbers of staff and security guards.
Recently, in Japan there has been growing awareness of such issues as its aging society, low birth rate and manpower shortage. However, effective use of technology will allow us to create a brighter future. People Counter software can help us realize our potential.
Interview & Composition
Born in 1970. After working as a magazine editor, he became a freelance writer/editor and has been active as a researcher, interviewer and writer in the fields of IT, science and the environment.
Publications include The Day Apple and Google Become Gods (co-author), New Guide to Superconductivity, 72 Hours of Google (co-author), Affirmation (co-author), and others.