Samsung introduces ISOCELL Zoom Anyplace and E2E AI Remosaic Solutions for the 200MP ISOCELL HP2, a step more to explain why more pixels is better: zoom up to 4x with no drop off in resolution… in video!

In the war for smartphone sensors with more pixels, Samsung is ahead of the competition, with the distinct types of 200MP sensors it offers; the HP1, HP2 and HP3. While some suggest 200MP is just something for the company’s marketing to impress the public, as Sony claims that sensors around the 50MP mark should be the standard, there are other reasons why Samsung has invested in 108MP and 200MP.

The recent Samsung Galaxy S23 series has 50MP sensors on the S23 and S23+ versions, but offers a 200MP sensor on the main camera of the S23 Ultra. It’s not there just for show – although it impresses – but because Samsung uses it to offer versatility in terms of capture. Smartphones usually create final images with 12.5MP, which are considered enough for most purposes. The ISOCELL HP2 sensor used in the  S23 Ultra, with 200MP, allows users to create images with 200MP, which are, according to Samsung “16 times bigger with 16 times more information than the 12.5MP image”. The company adds that “this means that the image maintains its level of detail even after cropping.”

In fact, when zooming in on a photo for cropping, something many smartphone users do, apparently unaware of the physical limitations of the pixels, the photo typically appears blurry… due to few image pixels. By having a 200MP sensor, the Galaxy S23 Ultra makes it possible to obtain, even if a 200MP image is cropped in half, both vertically and horizontally, a resulting 50MP image that maintains detail.

A new in-sensor zoom system

The Samsung Galaxy S23 Ultra goes a step further, as  the ISOCELL HP2 sensor provides 200MP, 50MP and 12MP output options that allow the user to change resolution depending on their needs. In its default mode, the ISOCELL HP2 produces a 12MP image by merging 16 pixels into one big pixel. When you need a photo that has ultra-high quality, you can choose 200MP mode and the ISOCELL HP2 produces the image with rich details.

First announced in 2013, the ISOCELL technology reduces color crosstalk between pixels by placing a physical barrier, allowing small-sized pixels to achieve higher color fidelity. Based on this technology, Samsung introduced the industry’s first 1.0μm-pixel image sensor in 2015 and a 0.9-pixel sensor in 2017. In June 2018, Samsung introduced an upgraded pixel isolation technology, the ISOCELL Plus. In 2020, Samsung introduced a 1.2μm-pixel image sensor, the first to offer both Dual Pixel and Tetracell technologies. In 2021, Samsung introduced the industry’s first 0.64-μm-pixel image sensor and continued to innovate with diverse technologies.

Although the growing number of pixels on sensors gets attention from the consumer market, and is good for advertising, there are other reasons why smartphone companies increment the number of pixels on their sensors, and Samsung continues to reveal why its 200MP ISOCELL HP2 exists. The Samsung Galaxy S23 Ultra already offered a 2x digital zoom option, which is common in smartphones with sensors offering  many megapixels, but last August, through a firmware update, the company introduced a new in-sensor zoom system.

Samsung reveals Zoom Anyplace technology

The feature uses in-sensor crop, allowed by the 200MP sensor, to deliver better image quality. The 2x zoom can also be used for videos shot in Super Steady mode. Interestingly, Samsung had already offered a 2X zoom for portrait mode in the S23 Ultra, a sign that the company was exploring the technology to use in-sensor zoom.

Smartphones companies continue to not be able to create optical zooms with an extended range, although Oppo promised a 10x zoom (16 to 160mm) in 2019 that never saw the light of day. We’re talking about a real zoom, not digital, and not of a prime lens, like the 230mm from the Samsung S22 Ultra. Samsung has decided to use prime lenses on its top models – the S23 Ultra has 13, 24, 70 and 230mm prime lenses – and then offer 2x digital zoom for the main camera – which has the 24mm lens – which was “upgraded” with the new 2x in-zoom sensor option in Photo and Video mode.

At the start of October, during its inaugural Samsung System LSI Tech Day 2023 event, at Samsung Semiconductor U.S. headquarters, the company showcased groundbreaking innovations, including the next-generation flagship mobile processor, the Exynos 2400 with Xclipse 940 GPU based on the latest AMD RDNA 3 architecture.

Also revealed for the first time was Samsung’s Zoom Anyplace technology based on its 200-million pixel image sensor. According to the company, “this sensor innovation will enable an entirely new camera zoom experience, allowing mobile users to take up to 4x close-ups of moving objects without any image degradation. AI-based tracking technology automatically follows and captures objects, all while recordings are made in full screen simultaneously, ensuring that no moment or detail is left uncaptured.”

Now, the same day Xiaomi introduces its new Xiaomi 14 and Xiaomi 14 Pro, with the new HyperOS and Leica cameras… and the surprise of the 1-inch sensor of the Xiaomi 13 Pro being ditched for a smaller sensor (!) Samsung officially introduces its ISOCELL Zoom Anyplace technology for the 200MP. The new feature, the company claims, “gives users more freedom and creativity while filming with their smartphones.”

ISOCELL Zoom Anyplace  for video

With the new solution for 200MP image sensors, users can select the desired subject and the camera will automatically track and film it. Until now, users have had to focus on a moving subject when shooting a video themselves. But with ISOCELL Zoom Anyplace and Qualcomm AI engine, the camera quickly finds a moving subject and tracks and follows it, ensuring that it is not missed during filming. Even fast-moving subjects are prevented from going off-screen. Additionally, videos can be shot with less screen shake.

Samsung adds that “this feature applies to zoomed-in areas with high resolution, as well, thanks to real-time remosaic capabilities. ISOCELL Zoom Anyplace takes charge of tasks involved in tracking moving subjects while shooting video, and users can simply enjoy the scene in front of them with their own eyes.”

Previously, if users took a full-screen video and zoomed in on a subject later, the zoomed-in section suffered from poor video quality. Now, ISOCELL Zoom Anyplace allows users to simultaneously capture the full field of view and zoomed-in areas on video. Not only that, but both the full-screen and zoomed-in views can be filmed at the same time while maintaining high resolution.

Samsung believes that “this ability to take full-screen and close-up shots at the same time unlocks creative potential by providing the versatility to record one scene in many ways. As a result, creators can film from more angles, and they can do so more freely and dynamically. Even if the chosen subject is on the edge of the screen, users can zoom in on it without modifying the angle of their full-scope filming. In other words, users can zoom just about however they like. If they want to start a shot zoomed in on a subject and then zoom out, they can. If they want to do the inverse, they can do that, as well. Functions allow for a broader range of artistic choices to be made while filming, which enhances the overall user experience.”

E2E AI Remosaic for Image Capture

Samsung says that “thanks to in-sensor zoom by Tetra2pixel, users don’t need to be selective because of zoom limitations, either. Previously, with digital zoom, if the center region of interest (ROI) area was zoomed in 4x at a 12.5MP sum mode, it theoretically resulted in 0.78MP (12.5MP/16) image resolution. With in-sensor zoom, however, the selected area to be zoomed in on is selected from the entire scope and remosaiced, which allows the sensor to maintain the original 12.5MP resolution and enable video to be shot much more vividly. Cameras used to switch from wide to tele when zooming in at 3x or more, which caused the image quality and the angle to change. However, the 200MP wide camera can seamlessly zoom in at 2x or even 4x with no drop off in resolution. Because of this consistent quality, users can now enjoy a true-to-life filming experience.”

All this magic happens, too, because of another technology announced by Samsung: E2E AI Remosaic for Image Capture. In addition to Zoom Anyplace as a video solution, Samsung is introducing End-to-End (E2E) AI Remosaic for image capture. Currently, the steps the 200MP image sensor follows for processing images after shooting are: raw data output occurs before a remosaic, which then leads to image signal processing (ISP) before finishing with JPEG image output. In this process, all of the steps occur in sequence.

E2E AI Remosaic improves the process, Samsung says, “by enabling the remosaic and ISP to take place simultaneously. In other words, E2E AI Remosaic takes a process that occurs in sequence and makes it occur in parallel, thereby reducing remosaic latency by up to half. The end result is an overall faster image processing time, which reduces the shooting time of 200MP images and improves image quality. Users will be able to take and check photos more quickly, and image quality will deteriorate on account of data loss from latency less often. Because of this, photos taken will have richer details and colors.”

Although we will not see the results of these advancements in the recent series of smartphones, it’s clear that the Samsung Galaxy S24 Ultra and other S24 models, to be launched early in 2024, will feature ISOCELL Zoom Anyplace for video and E2E AI Remosaic for image capture. The images used by Samsung to demonstrate the concept are, the company notes, “not taken with ISOCELL Zoom Anyplace or E2E AI Remosaic, and are provided for illustrative purposes only for the expected capabilities of a technology still in development” but it’s only logic to expect some of the technology to appear on upcoming models, as, according to Samsung, “the company is continuing to work on solutions that will improve the user experience even further. “