Hyperspectral Imaging for Safety and Security

Hyperspectral imaging (HSI) is a technique combining imaging and spectroscopy to survey a scene and extract detailed information. Also called imaging spectroscopy, HSI is a powerful, data-processing-intensive method that creates a “data cube” containing information about the properties of a target at hundreds to thousands of narrow wavelength bands within the system’s field of view. Scientists can cross-check the obtained spectra with known, unique spectral fingerprints to detect, characterize and identify chemical compositions, materials, liquids, activities or events, which can then be referenced to a precise location in a scene.

Originally designed for top-secret military satellites and surveillance in the 1980s, hyperspectral technology has been limited by its high cost and the difficulty of processing the huge amounts of data it produces. But recent computing advances and improvements in data acquisition and analysis have extended HSI’s benefits to countless applications, from astronomy to weather forecasting. And many of these new applications—such as food inspection, pathogen detection, airport security, law enforcement and oil and gas pipeline inspection—are all about security and safety.

In addition to computing and data-processing advances, volume fabrication techniques are pushing down the size and cost of sensors and detectors—and, thus, the overall cost of the HSI systems that rely on them. A February 2015 report from the analysis firm MicroMarket Monitor (Dallas, Tex., USA) predicts that the nascent global market for hyperspectral imaging will grow at a compound annual rate of 12 percent from 2014 to 2019. Driving that expansion, according to the report, will be a shift toward inexpensive, compact systems that can process large datasets at high speed.

This has made HSI systems more affordable and has prompted a surge in new commercial applications, including agricultural monitoring, precise medical diagnostics, environmental surveillance (like oil spill monitoring and pollutant dispersal), and other research projects in geology, marine science and forestry. “HSI has been one of the fastest-growing technologies in electro-optics in the 21st century,” says Peter Yuen, a researcher at Cranfield University (Swindon, U.K.).

Food safety

HSI has grown in industrial applications like food safety in recent years. The advent of cost-effective fabrication has enabled compact, lightweight and less-expensive HSI systems in commercial machine-vision applications like food sorting and pharmaceutical quality control. Such applications require the ability to “see” targets in a scene and ignore undesired foreground and background information. It’s a task that humans are good at; the goal of machine vision is to match human cognitive ability and conserve the need for human labor, thereby saving cost in the long term.

HSI machine-vision technology can do more than humans, faster, and it can see far beyond the visible region, where humans cannot. One HSI system manufacturer with recent success in this area is Resonon (Bozeman, Mont., USA). Resonon specializes in hyperspectral cameras combined with partnering robotic machine-vision systems that can identify and sort objects—such as edible nuts coming down a conveyor belt—faster than humans.

The Pika II hyperspectral camera is a pushbroom line scanner that covers 240 wavelengths, from the UV at 400 nm to the IR at 900 nm, with a spectral resolution of 2.1 nm. With 640 spatial channels and a maximum frame rate of 145 frames per second (fps), the camera and real-time statistical pattern recognition software tell the robotic pick-and-sort system which hulls and detritus to remove from a conveyor belt full of nuts based on their unique spectral signatures.

In the case of almonds, the system can distinguish “good” brown almond nuts from brown almond hulls, which a standard three-color camera system cannot. The system rejects foreign materials, chipped nuts, shells and anything else that doesn’t belong in the food stream in real time, faster than manual inspectors or traditional machine-vision systems. The Resonon camera, partnered with automated sorting solutions from Bratney Companies (Des Moines, Iowa, USA) and FANUC Robotics (Rochester Mills, Mich., USA), can identify and sort up to 220 culls per minute.

“Hyperspectral imaging produces enormous datasets,” says Adam Stern, senior scientist at Resonon. “Each picture has a million pixels, and each pixel has 240 12-bit data points. Modern computers are finally able to handle the processing and keep it economically feasible.”

Indium gallium selenide (InGaS) sensors extending into the NIR-SWIR range, from 900 to 1,700 nm and beyond, would provide a great deal more information, according to Stern, but the cost of such systems often ranges to US$50,000 or more. The next step in commercial machine vision will be to make that spectral range affordable.

Detecting dangerous microbes

Building on past success using HSI systems to detect fecal matter and E. coli bacteria in the food supply, the U.S. Department of Agriculture is conducting research in the use of hyperspectral microscope imaging (HMI) to track foodborne disease outbreaks. Agricultural engineer Bosoon Park and colleagues at the USDA’s Agricultural Research Service in Athens, Ga., USA, experimented with imaging five different Salmonella serotypes at different incubation times using an HMI system based on an acousto-optical tunable filter (AOTF) from Gooch & Housego (Ilminster, U.K.).

The AOTF HSI-400 enables flexible, fast tuning of selected passbands of wavelengths at programmable bandwidths, which is not possible with a dispersive spectrograph system. The AOTF therefore eliminates the need to scan the system past the target or vice versa. An AOTF system passes an acoustic wave simultaneously with the incident light through a piezoelectric telllurium dioxide (TeO₂) crystal, which behaves like a grating to diffract the light into the sensor.

The experiment allowed Park and his team to conclude that HMI can detect Salmonella and other bacteria on the cellular level earlier than previously thought—after only 8 hours of incubation rather than 24.

“We can implement HMI on the cellular level as a rapid and early tool for presumptive detection of several bacterial pathogens,” said Park. “Future work will focus on fewer wavelengths, to reduce the required data storage and processing time, which is necessary to implement this technology in the food industry.”

Stress detection

In another application with interesting implications, an international team of researchers led by Peter Yuen at Cranfield University has applied HSI to the detection of emotional stress signals to identify potential criminals. Stress causes a variation in blood oxygenation in exposed areas of the body, such as in the face and hands. This blood oxygenation level is invisible to the eye—but it can’t hide from HSI.

The non-contact technique involves the use of a very-near IR (VNIR) spectrograph from Headwall Photonics (Fitchburg, Mass., USA) along with a camera operating at a wavelength range from 400 to 1,000 nm and a thermal mid-wave IR (MWIR) camera (a FLIR SC7600) to measure tissue oxygen saturation (StO2) as a physiological indication of stress.

As the subjects were subjected to situations (such as public speaking and mental arithmetic) used in standard psychological studies to elicit acute stress, the HSI system placed two to three meters away took 10 seconds to record one data cube of 300 wavelengths. All 85 participants exhibited increased StO2 levels, particularly in the forehead region. The detection of stress using HSI was correlated with thermal imaging data, as skin temperature is another variable used as a stress indicator.

Yuen believes that his team’s HSI technique can effectively distinguish would-be terrorist suspects or extremely mentally stressed individuals in airports from people with ordinary levels of stress. It could also complement other indicators of stress, such as temperature (for which the stress signature can vary depending on ambient temperature) and perspiration (which may not indicate psychological stress in some situations, such as that of an individual sweating after running to catch a plane).

“We are only just beginning to research these ways of applying hyperspectral imaging for surveillance and law enforcement,” said Yuen. “It will take more research to improve its discrimination rate, and eventually, we would have to figure out how to use it without violating privacy laws.”

Hyperspectral in motion

The development of HSI at video rates over the past decade presents altogether new problems in data storage and data processing rates. How can so much information be useful if an HSI video produces a massive data cube for every frame? Researchers are working with industry partners to discover new ways to use and analyze HSI video for numerous applications.

Researchers at Bodkin Design & Engineering (Newton, Mass., USA), the University of Wisconsin-Madison (USA), Norton Engineering Optics (Palo Alto, Calif., USA) and Space Computer Corporation (Los Angeles, Calif., USA) have collaborated on an HSI system that works in both snapshot and staring mode, along with a unique optical processor that provides video-rate hyperspectral data cubes. Pinholes create parallel optical paths in the system to fashion a full (x, y, λ) data cube for each video frame.

The design re-images individual scene elements in parallel through a prism onto a focal plane array. The elements are read via a frame grabber and stored on a PC, where “cube formation software” converts the recorded spectra into a 3-D data cube in a format easily processed by standard image software. The software processes one data cube every two seconds, which can lead to a backlog at video rates of 15 fps, but it can continue processing as additional video is acquired.

The experiment tested numerous hyperspectral systems from visible through LWIR. Two systems from Bodkin were tested in Boston. In one result, a boat on Boston Harbor was tracked by its spectral signature, which could be used to positively identify the same boat if acquired again later. In another result, the team was able to determine the spectrum of any point in the hyperspectral video—which enabled highlighting of, for example, magnolia blossoms from a block away. A similar technique with a thermal MWIR HSI could also be useful for identifying explosive ingredients like potassium nitrate and ammonium nitrate residue on surfaces.

The design results in a low-cost, small-form-factor device with no motion artifacts; data rates are reduced enough for reasonable processing time as the video is acquired. The disadvantage is that high spatial resolution comes at the cost of low spectral resolution and vice versa.

Microsize me

Hyperspectral sensor designer IMEC (Leuven, Belgium) announced in January a set of new snapshot HSI sensor designs that enable video data rates of 340 fps. In addition to enabling fast machine vision, the sensors are built for extremely compact but high-volume applications like precision agriculture via unmanned aerial vehicle (UAV), security and medical imaging.

The new CMOS image sensors are mass produced to lower their cost, and are available in 4×4 and 5×5 arrays. One new design features mosaic filters with one filter per pixel. The 4×4 sensor features 16 spectral bands between 470 and 630 nm of the visible range. The 5×5 provides 25 bands between 600 and 1,000 nm of the visible-IR range. Another HSI snapshot sensor for line-scanning applications places a Fabry-Pérot wedge filter over a CMOS imager to transmit 100 narrow bands at 5-nm resolution from 470 to 900 nm.

“When partnered with the right camera for the application, these new mosaic sensors are ideal for industrial applications with or without linescan capability,” said Andy Lambrechts, program manager at IMEC. “The compact sensors can even be scaled to handheld devices, and the spectral range now reaches down to 470 nm.” HSI system designers may find new applications for such sensors, and be able to develop custom, proprietary solutions that optimize (depending on their requirements) for speed, compactness, spatial versus spectral resolution and band selection.

Another company in the news for ultra-compact HSI systems is BaySpec (San Jose, Calif., USA). In October 2014, the company announced the “world’s smallest UAV-optimized hyperspectral imager,” designed for use in airborne precision agriculture, remote sensing, law enforcement, and security and defense, among other areas. The OCI-UAV imager can move while on the UAV to scan at the desired rate and acquire continuous visible-to-NIR hyperspectral data. The camera head measures 8×6×6 cm and weighs 0.4 lb (180 g). The computer can capture frame rates of 120 fps.

The trend toward ultracompact HSI is timely, as the U.S. Federal Aviation Administration (FAA) ruled in April that commercial drone operators can apply for exemptions from FAA regulations to operate UAVs in safety applications, such as inspections of power lines, pipelines, and oil and gas flare stacks. Other applications that can apply for the waiver include precision agriculture, filmmaking and real estate photography. Shortly after the announcement, Amazon received approval to fly drones in testing of new unmanned delivery methods. Hyperspectral imaging seems poised for takeoff.

---

Valerie C. Coffey is a freelance science and technology writer and editor roving the United States in a 45-foot motor coach with a mobile hotspot.

References and Resources

• J. Gilchrist. “Hyperspectral imaging spectroscopy: A look at real-life applications,” in Photonics Handbook (Lauren Publishing, 2007).

• W. Hungate et al. “History and description of hyperspectral imaging,” in Hyperspectral Remote Sensing: Principles and Applications, (Taylor and Francis, 2008).

• D. Bannon and C. Van Veen. “Hyperspectral imaging gets stamp of approval for food processing,” Photonics Spectra, June 2012.

• A. Bodkin et al. “Video-rate chemical identification and visualization with snapshot hyperspectral imaging,” Proc. SPIE 8374, Next-Generation Spectroscopic Technologies V, 83740C (2012).

• J. Happich. “The future of video surveillance: HD, hyperspectral and stereoscopic,” EE Times Europe, 5 February 2014.

• T. Chen et al. “Detection of psychological stress using a hyperspectral imaging technique,” IEEE Trans. Affec. Comp. 5, 391 (2014).

• M. Eady et al. “Rapid and early detection of Salmonella serotypes with hyperspectral microscopy and multivariate data analysis,” J. Food Prot. 78, 668 (2015).

• MicroMarket Monitor, “Hyperspectral Imaging Market by Application – Global Forecast to 2019,” February 2015.