Multispectral Imaging Moves into the Mainstream

Simultaneous imaging in multiple wavelength bands across the spectrum enables us to “see” details far beyond the capabilities of the human eye. Combined with telescopes, multispectral imaging (MSI) exposes the secrets of the universe; combined with high-resolution microscopes, it identifies hidden diseases. As recently as 15 years ago, MSI was largely limited to very expensive, bulky and custom-built systems for laboratory and government applications. Systems in Earth orbit required public funding because of the high cost associated with building and launching them. Recent advances in shared apertures, sensors, filters and design have helped MSI evolve to more accessible, affordable, compact systems for commercial use.

Multispectral imaging combines two to five spectral imaging bands of relatively large bandwidth into a single optical system. A multispectral system usually provides a combination of visible (0.4 to 0.7 µm), near infrared (NIR; 0.7 to 1 µm), short-wave infrared (SWIR; 1 to 1.7 µm), mid-wave infrared (MWIR; 3.5 to 5 µm) or long-wave infrared (LWIR; 8 to 12 µm) bands into a single system.

Many electro-optic/infrared multi­spectral systems for intelligence, surveillance and reconnaissance (ISR) applications combine at least one reflective-band (EO) sensor with one thermal-band (IR) sensor. Visible, NIR and SWIR are usually considered EO bands and form images using reflected light from a target. MWIR and LWIR are thermal IR bands, and they directly image the blackbody radiation from a target.

Multi- vs. shared aperture

One way to categorize multispectral systems is by aperture type, according to Craig Olson, senior optical engineer at L-3 Communications (Santa Rosa, Calif., U.S.A.). Multi-aperture systems have one window for each sensor, and sensors generally do not share common optics; they have relatively uncomplicated sensor designs but difficult packaging constraints. Shared-aperture systems, in contrast, combine as much of the optical path as possible among the various focal planes to minimize the size and weight of the total system. Shared-aperture systems can support much larger optical apertures for the same overall system size.

According to Olson, spectral systems incorporating the 2- to 3.5-µm band are less common because both the atmosphere and common optical glasses absorb heavily in that region. Systems for scientific or industrial process control applications might need these wavelengths, but combining that specific band into a multispectral system with another band poses significant engineering challenges. Applications between 2 and 3.5 µm are usually quite specialized. For example, the detection of explosives and chemicals falls into this category.

Standard definition imagers at 640 × 512 pixels are the most common configurations for non-visible bands. Focal planes with high-definition format (1,280 × 720 or larger) are recent developments in sensors other than visible charge-coupled devices (CCD) or complementary metal oxide semiconductors (CMOS). Such imagers can exploit large apertures since the large pixel size no longer limits spatial resolution. “Maintaining sensitivity in small pixels is an ongoing challenge for focal plane manufacturers, as is increasing the dynamic range or well depth,” says Olson.

Multispectral shared-aperture systems provide a lot of benefit when they exploit disparate physical phenomena between reflected and thermally emitted target signatures. Moreover, such systems also maximize spatial resolution. With a larger aperture, the sensor optics can support a much longer focal length with true resolution (i.e., not empty magnification) at the low f-numbers required for extremely sensitive detection and identification tasks common in ISR applications. Mission capability and operational envelopes can be extended by performing imaging tasks later into the evening, for example, or from a longer range.

Image fusion

MSI systems have an advantage in image fusion, says Olson, a capability that has become accessible to many systems in recent years. They are compatible by design with image fusion and other applications that require close alignment of spatial information from two or more spectral bands. While not providing as dense a dataset or “image cube” as a high-end hyperspectral system, a multispectral system combines just two panchromatic data sets with different insights into target characteristics to augment the information contained in a snapshot or video sequence.

For example, when observing a parked automobile, someone using thermal infrared can easily determine how long the car has been parked, if the engine is on, and whether the air conditioner is running. However, since most common glass is opaque to the thermal infrared, such imagers cannot detect if anyone is still inside the car. A visible or SWIR imager will determine the presence of people inside the car but will yield no information about the thermal history of the car. A multispectral tactical sensor providing fused imagery is a great asset in that case, when both high spatial resolution and detection of multiple target phenomena are necessary.

Airborne/ground tactical sensors

An emerging group of commercial MSI sensors combines SWIR and MWIR. Both spectral bands typically require low f-numbers, have similarly matched pixel sizes available commercially, and can exploit well-known and mature glass in their optical systems; consequently, these two bands are well-suited for large shared-aperture systems.

Both SWIR and MWIR are common bands in which the penetration of atmospheric haze, smoke and/or clouds are important. Airborne surveillance, tactical law enforcement, strategic ISR and aircraft monitoring applications can all benefit from these bands when combined into multispectral systems.

The development of specialized SWIR lenses has advanced a lot in the past four years, according to Olson. Although astronomical systems have used NIR and SWIR imaging almost since the discovery of the infrared band by William Herschel in 1800, only with the availability of sensors has the SWIR band been considered cost-effective for industrial inspection systems and ISR applications. SWIR is now viable for both single and multispectral systems thanks to the introduction of new optical glasses, improvements in the techniques for designing lenses using these materials and the increased availability of SWIR imaging sensors.

In addition, newly available dual-band infrared detectors can image simultaneously in both MWIR and LWIR using the same focal plane and camera engine. With properly designed optical systems, multispectral imagers using these focal planes can dramatically reduce size, weight and power for systems that require both infrared bands. “This technology is quite recent and is an enabling technology for true MSI systems,” says Olson.

Earth-observing satellites

One of the most fascinating applications of MSI is the remote sensing of Earth from orbit. In the past 15 years, MSI instruments installed on satellites such as NASA’s Terra and DigitalGlobe’s Worldview-2 have become powerful tools for observing Earth, particularly for agricultural research and mapping of man-made and natural disasters.

In spite of a name that derives from its Disaster Monitoring Constellation of five satellites, DMC International Imaging, Ltd. (DMCii; Guildford, Surrey, England) does more than disaster monitoring. DMCii coordinates multiple satellites to provide rapid-response imaging for disasters, precision agriculture, land-cover mapping, forest monitoring and other applications. DMCii currently owns and operates two satellites: the UK-DMC with a 32-m ground sample distance (GSD), and the UK-DMC-2 with a 22-m GSD. It also coordinates several other satellites in the constellation owned and operated by different nations. Each can image a 650-km swath of the Earth simultaneously in red, green and NIR spectral bands—ideal for vegetation observations. The three spectral bands are similar to those of the long-established Landsat satellite constellation, which serves as a standard for cross-calibrating satellites from different nations.

“The imagers are cross calibrated in orbit to within 1 percent of Landsat,” says Paul Stephens, director of sales and marketing at DMCii, “enabling a high image quality that is crucial to our end-users, such as the U.S. Department of Agriculture, which relied on DMC 22-m imagery in 2011 for crop statistics in the U.S.A.”

The choice to design new satellites with the same key spectral bands as Landsat allows for cross compatibility and data continuity. The DMC constellation replicates the core data supply of the Landsat program, while dramatically improving Landsat’s 16-day revisit capability. “Our image swath is 650 km, rather than Landsat’s 185 km, thanks to a sensor development,” says Stephens. “With only four spacecraft, we achieve the daily revisit required to view rapidly changing phenomena.” This is something that the Landsat community had high on their wish list, according to Stephens, but the cost of a single Landsat precludes having multiple units in orbit. Using commercial off-the-shelf technology has changed the economics of space, he says, enabling spacecraft to be built at a much lower price. “Owning and operating spacecraft profitably, without a government subsidy, points the way to sustainable Earth observation.”

Sensor sensibility

The improved technology behind Earth-orbiting MSI satellites is the development in the past two years of advanced sensors, says David Cochrane, director of technology marketing at Teledyne DALSA (Waterloo, Ontario, Canada). “The latest advance in these sensors is twofold,” says Cochrane. “All the color lines are combined on one chip, and advanced dichroic filters provide high transmission of light in the required color band only. It’s a great combination for our customers’ satellites.”

Modern CCD and CMOS fabrication techniques combined with advanced dichroic filters have resulted in sensors that are more cost-effective while maintaining the high performance needed in remote-sensing applications. By bonding advanced dichroic filters onto the cover glass directly in the imaging path, a single device can be tailored to image numerous visible and IR bandwidths in a cost-effective and reliable package. The advanced technology filter approach enables up to 12,000 linear pixel arrays, while individual elements are based on high-resolution time delay and integration (TDI) technology to maximize sensitivity and throughput.

For multispectral sensors, a unique Teledyne DALSA process combines the multispectral filter with a multisegmented linear CCD in a single package. The multispectral filters, developed in cooperation with experts in optical and metallic thin-film coatings and dichroic filters, are multilayered thin-film dielectric interference filters that optimize transmission and bandwidth selectivity. The five-band filter obtains an average in-band transmission of greater than 90 percent and out-of-band transmission is typically much less than 1 percent. High-transmission filters and TDI devices are useful in remote-sensing applications where satellites are typically in high orbit and light signals can be weak.

Another feature of these filters is their precise alignment to the individual sensing bands of the CCD using alignment marks on both the filter array and the multisegment CCD. In this design, four bands were designed for R, G and B signals; one NIR band; and a fifth broad panchromatic passband encompassing all four narrow bands. These bands corresponded to five high-resolution TDI segments on the single CCD chip. The exact resolution is proprietary. However, typical resolutions in this application range from 4,000 to 12,000 pixels with pixel sizes from 5 to 33 µm. For aerospace remote-sensing applications, the CCD and package have also been designed for high reliability and radiation tolerance.

Prototype SWIR/LWIR binoculars

In 2010, the Advanced Development Group at FLIR (Wilsonville, Ore., U.S.A.) created the first pair of infrared binoculars using two uncooled LWIR microbolometer sensors (8 to 14 µm). “Holding two thermal weapon sights side by side, we realized that depth perception was possible,” says Noel Jolivet, project manager. (That’s him in the images below.) “No one else had done this because of the expense, size, weight and power associated with running two cameras simultaneously. Others had built so-called “bioculars:” one camera with the video going to both eyes, but that doesn’t provide depth perception. With true binoculars (as opposed to bioculars), the human vision system comes into play, turning horizontal and vertical disparity information from two sources into perception of depth.”

The team proceeded to make a set of SWIR binoculars (using InGaAS arrays of 1 to 1.7 µm), and finally, a pair of binoculars with an LWIR sensor for one eye and an SWIR sensor for the other. “The beauty of this multispectral system,” says Jolivet, “is that the human vision system does the fusing, as long as you match the FOVs [fields of vision] of the optics, can register the images to some degree, and can adjust the brightness of the display to each eye.”

Whereas IR binoculars alone make it difficult to recognize a familiar person, the SWIR/LWIR prototype enables recognition of facial features that overlaps with a broad-brush IR image. “While frankly the effect is a bit creepy, SWIR enables identification via facial features through glass, while LWIR provides thermal info.” Road signage is another good target for this technology, says Jolivet, “since now you can read the sign, not just see that it is there. With a portable pair of LWIR/SWIR binoculars—the only ones in the world—we’ll be seeing new things in all kinds of applications.”

Biomedical applications

The way that light interacts with biological tissue varies considerably with wavelength, making spectral imaging a powerful tool for biomedical and chemical applications, says Randel Mercer, vice president of business development at Ocean Thin Films (Golden, Colo., U.S.A.). For example, imaging in the NIR enables depth measurement in tissue and blood chromophores such as oxy-hemoglobin, deoxy-hemoglobin and bilirubin, when compared to the visible image. Because spectral imaging is noninvasive, it’s also useful in the assessment of burns and skin inflammation.

Unfortunately, such MSI systems are not readily available commercially. Researchers typically build their own custom machines using expensive off-the-shelf components—a cumbersome endeavor that costs from tens of thousands to millions of dollars. Ocean Thin Films recently developed a new MSI technology, the SpectroCam, which features a proprietary rotating filter wheel (RFW) with custom dichroic filters in either segmented or monolithic wheels that can be cost-effectively produced in mass volume. The segmented RFW can be custom configured with up to eight interchangeable optical filters.

While filter wheels and CCD cameras aren’t novel, what is new is the high-speed integration of filter assemblies that operate with a rotation rate of 1,800 RPM, a rate that far exceeds the requirements necessary for most imaging applications. Combined with long-life motors and a scientific-grade CCD array, this filter design enables a portable, configurable, high-speed MSI camera.

“To put together a system you can use to customize, once you’ve done analysis in spectral space, and you know the four to eight wavelengths you need, you can specify a low-cost custom camera for $25,000, whereas use of the filter with other types of multispectral imaging technologies like acousto-optic or liquid tunable filters might cost around $50,000,” says Jason Eichenholz, chief technology officer at sister company Ocean Optics, manufacturer of low-cost commercial spectroscopy systems. SpectroCam can be used for 2-D spectroscopy research in a variety of fields, including food safety and water-quality measurement, product screening, machine vision, medical imaging, surveillance and authentication. “The goal is to make spectral imaging compact and affordable,” says Eichenholz.

Valerie C. Coffey is a freelance science and technology writer and editor based in Boxborough, Mass., U.S.A.

References and Resources

>> R.A. Schowengerdt. Remote sensing: Models and methods for image processing, Academic Press, 3rd ed., (2007).
>> J. Eichenholz et al. “Sequential Filter Wheel Multispectral Imaging Systems,” Proc. OSA/AIO (2010).
>> J. Eichenholz et al. “Real Time Megapixel Multispectral Bioimaging,” Proc. SPIE BIOS Vol. 7568 (2010).
>> J. Miller et al. “Hyperspectral and Multispectral Sensors for Remote Sensing,” DALSA Teledyne white paper (2012).