The often-quoted line when it comes to photography is “It is the photographer, not the camera, that is the instrument”. While it is absolutely true, with changing technology and the advancements in Smartphone cameras, we can also add safely the importance of the algorithms in producing great imaging.
Oppp has always been the harbinger of camera advancements and innovation in mobile imaging. Right from Oppo Find 7 launched back in 2014, which then was the world’s first phone that can take 50MP image. 2019 was a landmark year for Oppo, when they launched Oppo Reno 10X with the critically acclaimed and lauded 10X hybrid zoom with periscope telephoto lens. It was the first-ever smartphones to sport a periscope camera and offered full bandwidth of 16-160mm in a camera phone. Find X2 Pro in 2020 was equipped with Ultra Vision Camera System sporting a 10X Hybrid Zoom, All Pixel Omni-Directional PDAF and 12-bit True Capture.
The credit for the breakthroughs goes to Oppo’s 5 imaging R&D team spread across Yokohama in Japan, Silicon Valley United States, Shenzhen, Chengdu and Shanghai in China. Together they have built an extensive Patent Portfolio in Imaging Technology with 7500+ patents filed and 2,350+ granted across 25+ countries.
In 2021, Oppo is bringing out an all-new addition to the imaging world with their Full Dimension Fusion (FDF) Portrait Video System, first introduced in OPPO Reno5 Pro 5G. As the name would suggest, the new algorithm enhancements bring improved video quality and portrait effects in videos.
Full Dimension Fusion (FDF) Portrait Video System Explained
The charm of camera phone is that unlike professional camera equipments like DSLR that requires a fair amount of learning of controls and modes, you can simply pull out your Smartphone and take easily a better photo. This is possible with the algorithms that work in background to ensure the end result is stellar. The hardware advancement in mobile imaging has to be complimented by software enhancements and algorithms for enriching output. With the Full Dimension Fusion (FDF) Portrait video system, Oppo is bringing on board an augmented portrait video experience for users on the new OPPO Reno 5 Pro 5G based on their extensive expertise in portrait scene detection and portrait feature design.
Let’s delve deep to see how it is achieved. The OPPO FDF Portrait Video System is itself comprised of two engines –the Quality Enhancement Engine and the Portrait Perception Engine.
The Quality Enhancement Engine:
The Quality Enhancement Engine is a suite of algorithms that ensure users can capture quality video images in any situation. Whether in low light, bright light, or fast motion environments, the engine optimizes settings to deliver images that stay true to life. Oppo achived this with AI highlight Video and Ultra Steady Video 3.0
AI Highlight Video:
A good bright lit environment can produce decently good videos but things gets grainier (pun intended) with not so ideal lighting conditions and especially low light where most mobile cameras struggle with videos.
Oppo has crafted AI ultra night video combined with live HDR to enhance the video quality in low light conditions. Oppo claims this to be World’s first feature to use AI algorithms to detect light conditions and automatically apply the required algorithm based on different scenarios on videos.
Oppo’s new quality enhancement engine in a world’s first manages to apply Ultra Night Video and Live HDR effect simultaneously. End result is what you have so far seen in ultra night mode only on photos coming to life on videos with live HDR joining the party for more optimized lit videos taken in not so optimal lighting conditions. The key lies in the adaptive dynamic response for enhancement in videos.
The fusion algorithm sets in action to adaptively enhance and boost the brightness, color and clarity of the low light scenarios captured as well as adaptively reducing the noise levels.
With the Adaptive brightness Auto Exposure (AE) Adaptive Gain is utilized to ensure the signal to noise ratio is in its optimal level.
The Adaptive gain controls the exposure time to bump up the brightness of the image while maintaining a smooth video from frame to frame.
The problem is not solved just by enhancing the brightness levels. There still would be over-exposed and under-exposed areas which need to be corrected, enters Adaptive Dynamic range.
Adaptive Dynamic Range applies dynamic range adjustments to each frame in a video with Dynamic Range Conversion and Fimic Tone Mapping technology to restore lost colors, brightness, and darkness to their natural levels. It also uses Multi-frame HDR Fusion to deliver a greater dynamic range across the entire video.
While it is tedious work when it comes to videos which are basically multiple frames in seconds, the Quality enhancement engine achieves this by mapping the brightness and color in each frame accordingly.
With Live HDR setting in, it is possible to apply multi-frame HDR Fusion thereby enabling every area in the video to get the most appropriate exposure level.
Details in bright areas are kept while details in dark areas are made visible creating high dynamic videos that are much closer to what eyes see.
Separating the subject and its surroundings has always been a software challenge, but Oppo tries to apply a localized contrast adjustment algorithm to distinguish the subject and its surroundings and locally adjust the brightness, color and contrast of the face.
Noise grains have been always an issue with low light videos on mobile, Oppo tries to fix it with Inter-frame Fusion for Temporal Noise Reduction analyzing the small changes from frame to frame and restore pixels to their original position.
The adaptive noise reduction realigns multiple frames within 5 milliseconds by intelligently merging pixels based on localized movement and macroscopic movement.
Intra-frame Fusion for Spatial Noise Reduction denoises based on three levels of frequency detail in the image. It removes noise in low frequency areas while preserving the rich textures in high frequency areas. All this happens almost real time taking up only 20ms processing time.
Oppo claims its tests shows night time videos appear 74.4% brighter and 33.7% clearer than those recorded without the AI Highlight feature enabled. All this background algorithm processing also takes as little as 20ms.
Reno 5 Pro also brings ultra steady 3.0 video to the front camera.
The Portrait Perception Engine:
People love portrait photography, Oppo brings portrait video feature with Reno 5 Pro. A collection of algorithms deliver higher accuracy in portrait recognition and processing.
The Portrait Perception engine handles and manages strict criteria for the features within a portrait video. These criteria ensure that the portrait subjects and the background are clearly and naturally defined, while also providing the basis from which the Reno5 Pro 5G’ various portrait video effects can be optimally applied.
While portrait photos are still a challenge for many companies, getting it right for videos is a bigger challenge with continuously changing frames. It needs major works on dynamic scene detection, AI modelling and real-time portrait processing. In real case scenarios, it is more complicated. The Portrait Perception engine has to account for the portrait subject when they appear side-on the camera or if they are partially obscured. In real-time, the algorithm processing has to pull out specific features of each person via mapping and recognize each face. Then the AI highlight video can set in to enhance the brightness, color or contrast adjustment precisely. Moving subjects offer more challenges for processing and add to it complex background, to achieve portrait videography is really pulling a rabbit out of a magic hat but possible with the AI algorithm expertise Oppo has mastered all through the years and it’s understanding of motion tracking, face optimization and instance segmentation.
Scene detection is a bigger challenge. It not only has to account for the changing dynamic scenes but also take into account and analyze the subjects and the objects handled by subjects. In the photo below, the AI scene analyser has to take into consideration the bottle held by the subject while analyzing the portrait.
The algorithm also digs into 10 Millions+ Level Miniaturization Portrait Database for AI modelling. The Portrait perception engine also maps and engages in Face Optimization, Motion Tracking and Instance Segmentation for effective video results. It’s a complicated and tricky algorithm challenge Oppo has achieved for video portrait by its software AI algorithm which is a commendable feat.
With Reno 5 Pro Oppo is set to bring in another milestone to its illustrious camera achievements. It is to be noted in this year’s CVPR Awards –OPPO took first place in the “Perceptual Extreme Super-Resolution challenge” and “Visual localization for handheld devices challenge”, as well as a third-place for “Activity detection in extended videos” challenge. What Oppo brings into videography would be defining moment for Mobile imaging and opening up new possibilities and enriching user experience.
