Technologies That Support RICOH360 Described by Engineers

vol.1 Enthusiasm in the Development of Technology That Will “Go beyond Limits”

　This column series will introduce interviews spotlighting “technology” and “people” supporting RICOH360 over four articles to learn about them.

　The theme of the first issue is image processing technology using AI. We interviewed two engineers about the enthusiasm behind the scenes that we would not have known about, including examples of the use of image processing technology in RICOH360 and the insatiable passion for research and development for super-resolution.

Application Is a Step for Practical Use

　For RICOH360, various functions have been developed utilizing AI technology. For example, in virtual tours for real estate, still images and videos can be automatically clipped from 360-degree images and CG furniture can be placed for simulations (Figure 1).

　Many AI technologies have already been released all over the world. However, Mr. Odamaki, in charge of development, says that it is difficult to provide services by using them as they are.

Makoto Odamaki

Manager of the Technology Development Office
-----

Joined Ricoh in 1997. Visiting scholar in Colombia University from 2014 to 2016.

In his career, he has been leading the development of computer vision and AI-related technologies for 360-degree images.

His hobby is visiting BOOK-OFF shops (used book stores). Lately he has been obsessed with world history.

‍

“Although there are various functions, they are provided by applying basic technologies. There are so many AI-related technologies in the world. However, it is a little tricky to actually use them in products or services. The first issue is the ‘barrier of licenses for commercial use’ and the second issue is the ‘barrier for development to a practical level.’

　Needless to say, we cannot use something while infringing upon licenses so we must be careful about this from the very first stage.

　There is an example regarding practical level. When analyzing images of a living room, AI is used to detect doors and closets. In general analysis, however, it is difficult to distinguish between a closed door that we ‘can go through’ and a closet door that we ‘cannot go through’. Even humans may not be able to identify these just by looking at images. Therefore, a cycle where we create new definitions and make improvements toward practical uses is necessary.”

‍

　“AI Image Enhancement” whose first version was announced in July 2020 was developed through such trial and error. This technology, where “super-resolution” for up-converting low-resolution images is turned into practical use, was developed by Ricoh’s unique idea. What we took note of was the 360-degree camera “RICOH THETA” and Ricoh high-performance compact digital camera “GR.”

　We thought of using GR with high resolution and less noise and THETA with lower resolution and more noise than GR as a pair and having AI learn images acquired by them. It sounds good but actually there were a number of challenges.

“In addition to differences in lenses, design for white balance, color reproduction, and other features are different. We had to be creative in making AI learn ‘images with completely different conditions’ by comparing them.

　As a result, we completed a sophisticated algorithm that can reduce noise and correct any chromatic aberrations in addition to the improvements to the resolution. Some of this result has been announced in the OmniCV² workshop in ‘CVPR2020¹’ which is a top conference in the computer vision field.”

1: The Conference on Computer Vision and Pattern Recognition. It is an authoritative conference that has the most authority in the field of computer vision and pattern recognition. It gathers several thousands of people from communities for AI, machine learning, and computer vision all over the world.

2: Omnidirectional Computer Vision in Research and Industry

Pursuing beyond Machine Learning

　Research and development for further quality improvements continued even after the release in 2020. This time, the goal was more comprehensive image quality improvements instead of improvements only in resolution, noise reduction, and chromatic aberration correction.

　However, image quality has very troublesome characteristics. The question really is, “What defines good image quality?” — Although evaluation axes based on numerical values can apply, some aspects are still left to the perception of each individual. We spoke with Mr. Suito, who conducted the actual tuning to improve the image quality.

Hiroshi Suito

------

Joined Ricoh in 2003. As a student, he engaged in research on artificial intelligence.

In his career, he has worked on firmware development for digital cameras, and more recently has been working on image processing of 360-degree images and implementation of AI Virtual Staging.

His hobbies include watching baseball games and jogging.

‍

“‘Mathematical evaluation’ and ‘perceptual evaluation’ are considered to be difficult to reconcile. Many times it looks good on the numbers but not so good visually, or vice versa. However, it is also difficult to verbalize or formulate the sensory acceptance levels, so in tuning, the voice of customers who are actually using the product becomes important.”

‍

　Even so, it can be difficult to gather feedback from customers, or the statistics could potentially be biased. However, Mr. Suito had previously worked on digital cameras for consumers and had already accommodated various requests related to image quality.

‍

“We get a lot of feedback from customers with particular preferences, such as, ‘Though it was updated, I liked the tints better in the last version,’ as well as, ‘The colors appear strange in this environment.’ There are cases where updates to a new model with a technical aim are not well accepted.

　It was very challenging to meet each request one by one, but through repeated communication, I developed a rule of thumb such as, ‘This is acceptable’ or ‘This is probably unacceptable.’ The experience of being so close to the customers’ voice helps me evaluate image quality today.”

Figure 2: Left: Original image Right: AI-corrected image

‍

　Thus, the AI Image Enhancement updated in September 2022 achieved an overall improvement in image quality beyond super-resolution by reviewing the network structure and combining multiple AI image processing such as scene recognition (Figure 2).

　While utilizing the latest AI technology, supplement the delicate senses with human hands. By spinning both of these wheels, learning data with good quality has been identified, and an enhancement model that “achieves better results with less amount of data than before as well as being twice as fast” has been completed.

‍

　Also, with regard to image quality evaluation, there seems to be an evaluation axis and specialty cultivated in each company. Just like other manufacturers, Ricoh has a dedicated team for “Image Quality.” The know-how accumulated over many years has been inherited as the DNA, and this baton will be passed on to the next generation.

“When we conducted an in-house image quality evaluation test, Mr. Suito and I were almost unanimous in our answers. It may be that the preference and quality of images have already been inherited in the organization,” says Mr. Odamaki.

‍

　Further quality improvement over two years. The year 2022 was a big leap forward, but they are already discussing future updates.

“As a team, we hope to pass on the skills we have developed so far, while also strengthening our cooperation with other teams,” they said. Toward the launch of RICOH360 SaaS plus a Box, activities to foster engineer interaction and development will be carried out across hardware, software, and the cloud.

‍