One of the most impressive camera tricks that Google introduced with the Pixel 3 last week is Super Res Zoom. The Google AI team behind the feature detailed the technical aspects today, and how Google wants to challenge the idea that digital zoom is “the ‘ugly duckling’ of mobile device cameras.”
In a nutshell, Super Res Zoom works by merging many frames to form a higher resolution image rather than upscaling the crop of a single image. This results in improved details that are “roughly competitive with the 2x optical zoom lenses on many other smartphones.”
Super Res Zoom means that if you pinch-zoom before pressing the shutter, you’ll get a lot more details in your picture than if you crop afterwards.
The goal on the Pixel 3 is to capture more detail when zooming into a subject. Machine learning-backed image upscalers, like Google’s RAISR, can magnify straight edges and certain texture, but “cannot recover natural high-resolution details.”
To get around this lack of detail, including missing colors, Google again turned to capturing multiple frames, like with HDR+. Used in other fields like astronomy, “drizzle” has largely been confined to controlled settings “where the sensor and the subject were aligned and the movement between them was either deliberately controlled or tightly constrained.”
It has been known for more than a decade, including in astronomy where the basic concept is known as “drizzle”, that capturing and combining multiple images taken from slightly different positions can yield resolution equivalent to optical zoom, at least at low magnifications like 2x or 3x and in good lighting conditions. In this process, called muti-frame super-resolution, the general idea is to align and merge low-resolution bursts directly onto a grid of the desired (higher) resolution.
However, Google is leveraging that lack of stability, and even using optical image stabilization (OIS) in reverse to move the camera between shots if the Pixel 3 is resting still on a tripod.
To take advantage of hand tremor, we first need to align the pictures in a burst together. We choose a single image in the burst as the “base” or reference frame, and align every other frame relative to it.
Google had to overcome a number of other features, with the blog post providing an in-depth explanation. The team also provides a list of tips on how to get the most out of Super Res Zoom:
- Pinch and zoom, or use the + button to increase zoom by discrete steps.
- Double-tap the preview to quickly toggle between zoomed in and zoomed out.
- Super Res works well at all zoom factors, though for performance reasons, it activates only above 1.2x. That’s about half way between no zoom and the first “click” in the zoom UI.
- There are fundamental limits to the optical resolution of a wide-angle camera. So to get the most out of (any) zoom, keep the magnification factor modest.
- Avoid fast moving objects. Super Res zoom will capture them correctly, but you will not likely get increased resolution.
Check out 9to5Google on YouTube for more news:
FTC: We use income earning auto affiliate links. More.