A gigapixel is a billion pixels, or 1000 megapixels — a 45,000 x 25,000 pixel image. A high-end DSLR is typically 12 – 24 megapixels, and next-generation cameras will get as high as 50 megapixels, but that’s still 20x smaller than a single gigapixel. That’s a lot of pixels, but Photospector can view and edit these images in real-time (after a few minutes of pre-processing):
Large images turn up in a few interesting places: mapping, medical imaging, visual effects and printing but most often from stitching together smaller images, like when creating a panorama picture.
Viewing these images on mobile devices starts with getting the image onto the device. Be careful! The standard iTunes desktop to tablet transfer process will not transfer the original file! All images are reduced to a maximum resolution of about 2048 before transfer. We recommend using the Photo Transfer App, which has successfully transferred some very large images in our testing. Panorama stitching apps are another option.
What do other apps do?
Handling large images is hard. The majority of photo apps, including PhotoForge2, PicShop, Pixlr and Filterstorm Pro, crash or freeze when they try to load the gigapixel image. Some well-tested apps, like Photoshop Express, inform you that the image is too large and refuse to load it. Other apps use a low resolution version of the image, generally 100 – 250 times smaller that allows you to see the image and even perform editing, but the vast bulk of the data is lost.
The iOS Photos app, is able to load the image, but it uses a low-resolution stand-in, which appears to limit the displayed resolution to about 2048 along one axis (the size of the iPad3 retina screen), instead of the original 45,000 pixel resolution, or about 1/250th the data size of the original. Photo Toaster ($1.99) and Aviary have identical results. Here’s a screenshot of the Photos app displaying the gigapixel image on an iPad3:
iPhoto ($4.99), Apple’s advanced image editor, does a much better job than the Photos app, allowing you to zoom in about twice as far, probably supporting a maximum resolution of 4096, double that of the Photos app. It also uses superior filtering which results in legible characters for this 12 point font:
Photogene ($2.99) uses a reduced resolution similar to iPhoto, but it has a fast mode for moving around that uses a still lower resolution image which improves when you stop moving. This allows you to move around smoothly and have more detail when you stop to view the current region. Here’s what you see when you stop and display the best quality:
Big Photo ($1.99) specializes in handling large images. It is able to quickly load the gigapixel test image, initially displaying the same reduced resolution image as the default Photos.app. Unlike the Photos.app, Big Photo lets you zoom in, and, once you pass a threshold, it pauses to load a higher resolution version of the current region. This approach allows Big Photo to display full resolution images for professional photo resolutions from 20 – 50 megapixels. Here’s the gigapixel (1000 megapixels) image displayed in Big Photo:
Photospector’s approach is to pre-process a tiled and filtered version of the image. The other apps have short preprocessing times of 20 – 60 seconds, with no option for additional processing. Photspector takes about 9 minutes to load the gigapixel image, during which it filters and caches a tiled version of the full-resolution file. While computing the tiled cache, you can continue working on other images in Photospector and a progress bar is displayed to let you know the status of the background processing. Once computed, you can open the image instantly so it can be compared against other gigapixel-sized images just like any other picture. The cache is stored on disk and its size can be controlled using a setting that allows up to ~4 gigapixel caches to be stored at the same time.
Here’s how Photospector displays the same block of 12 point text shown in the examples above:
Loading and displaying gigapixel images on mobile devices is a challenging task even for the most sophisticated image editing programs. Photospector always works at full resolution and supports UNLIMITED resolutions, trading preprocessing time for interactive updates. Instead of charging for large image support superior to that found in other apps, Photospector includes it for FREE!
How does it work? (WARNING: Technobabble Ahead)
The memory (RAM) on mobile devices is extremely limited. While desktop workstations may have 16 GB of memory, iPad 3 & 4 both have 1 GB, and much of that is used by other apps and the operating system. In practice, apps can only allocate 300-500 MB of memory before they are summarily killed by the OS. Since a gigapixel image is about 4GB in memory, when stored without compression, so you can only store a small portion of the image in memory at any given time.
Even then, you still need to special algorithms to draw the image, because standard image drawing functions cannot handle partial images. The solution to both the memory and drawing problem is to store and draw the image in tiles. We also pre-filter (shrink) the tiles and use these smaller versions whenever possible without losing fidelity. This multilevel tiled file is called a MIP-map (MIP stands for “Multum Im Parvo”, because graphics geeks love using latin phrases nobody understands).
Photospector has two big pieces of tech that are used to handle big images with minimal memory: OpenImageIO and Virtuoso.
The Apple iOS image libraries have no way to load partial images, requiring instead that you load the entire image in a single block, which we learned above, will crash your app. That’s why the vast bulk of mobile image applications crash immediately when you try to open a large image. OIIO is an Open Source project started by Larry Gritz and is supported by his work at Sony Picture Imageworks. OIIO is widely used in feature film production software to work with image files, and Photospector is the first iOS app using the library. OIIO streams in image files, loading only tiles or scanlines as needed. We recently enhanced OIIO to build mipmaps using a minimal memory footprint, streaming tiles in and caching them efficiently using the same texture caching technology used by Sony Picture Imagework’s feature-film renderer.
You can display images on iOS using UIKit or OpenGL, and both have a maximum image size of 4096 x 4096. But, even if you could draw larger images, they would look terrible if they were not properly filtered, a step we perform when constructing the smaller levels in the mipmap during preprocessing. Photospector reads tiles from the filtered mipmap file on-demand, passing them along to Virtuoso, our proprietary virtual texture mapping engine. Virtuoso uses game tech similar to the Megatexture algorithm developed by the folks at id Software, the makers of Doom & Quake.
Virtuoso maintains a single 2048×2048 texture tile pool which is just large enough to accurately display images on an iPad retina screen. Tiles are added and removed to the pool as you move around and zoom in and out of the image by calling OIIO to retrieve the tile from the cache file. Updates happen at over 30 frames per second! Together Virtuoso and OIIO’s caching systems allow us to seamlessly explore ginormous images in real-time with full fidelity.
Read the Game Tech for Photos blog post for more about Virtuoso and the other game engine technology developed by The 11ers for Photospector.