Computational Imaging XXI
Monday 16 January 2023
KEYNOTE: Neutron Imaging Beyond Traditional Radiography (M1)
Session Chairs: Alexander Long, Los Alamos National Laboratory (United States) and Sven Vogel, Los Alamos National Laboratory (United States)
8:45 – 10:20 AM
Market Street
8:45
Conference Welcome
8:50COIMG-129
KEYNOTE: Advanced neutron imaging, Markus Strobl, Paul Scherrer Institut (PSI) (Switzerland) [view abstract]
Prof. Dr. Markus Strobl is the leader of the Applied Materials group at the Paul Scherrer Institute (PSI) of Switzerland. The Applied Materials Group (AMG) is a group within the Laboratory for Neutron Scattering and Imaging LNS, in the division research with Neutrons and Muons NUM of PSI. AMG is operating 2 dedicated neutron imaging facilities and the neutron strain scanner (diffractometer) POLDI for users from scientific institutions and industry. AMG also provides complementary x-ray imaging (in-situ bi-modal) and has dedicated beamtimes for imaging studies at the test beamline BOA providing an intense cold polarized neutron beam. Strobl has over 230 publications in the field.
The last decade in neutron imaging saw extensive activity in method development. This was only partially due to novel pulsed neutron sources and corresponding new imaging instruments having been established. Also at the state-of-the-art instruments at continuous sources wavelength resolution, polarization and grating interferometry, to name a few, changed the field of neutron imaging sustainably. Both, these new methodical developments but also increased accuracy and demand for quantification, implied a need for more detailed and concise description of the utilized neutron interaction, in particular with respect to neutron scattering. Thus, finally the scattering aspect in neutron imaging gained significantly in weight, which on the one hand puts additional demands on neutron imaging scientists and on the other hand increases the integration of neutron imaging within the suite of neutron science techniques and their applications at large scale neutron sources. Consequently, the basic principles required to be embraced with the access to novel methods, interactions and information shall be out- and underlined. A number of interactions providing valuable new information in novel techniques are known for causing artifacts in conventional applications. These are more important with respect to the better resolution achieved nowadays and the corresponding higher demands for quantification. For the session: Neutron Imaging Beyond Traditional Radiography.
9:20COIMG-130
Material decomposition in neutron time-of-flight radiography, Thilo Balke1, Alexander M. Long2, Sven C. Vogel2, Brendt Wohlberg2, and Charles A. Bouman1; 1Purdue University and 2Los Alamos National Laboratory (United States) [view abstract]
As a pulsed neutron beam travels through a sample, neutrons arrive at the imaging detector over a range of time according to their velocity, i.e. kinetic energy. The radiographic measurements at the detector are thus inherently energy resolved. Since the nuclei in the sample have characteristic absorption spectra that are isotope specific, the hyperspectral measurements contain information about the density distribution of the isotopes in the sample. We present a methodology to estimate the isotopic distribution in the sample. However, this task is very challenging due to the extremely large amount of noise even with long exposures, and the large number of unknown parameters such as the neutron beam flux and background events, and the finite pulse width. All these have to be properly accounted for in order to achieve quantitatively accurate density estimates. Our approach deals with the extreme noise by modeling the system in its native, non-linear domain and an ability of native modeling as Poisson noise. The estimation of large amount nuisance parameters is achieved through a novel reduction in dimensionality. We demonstrate our approach with experimental data from FP-5 at LANSCE as well as Monte Carlo simulations. Our reconstruction densities in 2D (radiography) and 3D (tomography) are quantitatively verified through multiple means such as known ground truth samples, comparison to state-of-the art nuclear code, and mass spectrometry. For the session: Neutron Imaging Beyond Traditional Radiography.
9:40COIMG-131
Artificial intelligence-driven hyperspectral neutron computed tomography (HSnCT) systems, Shimin Tang1, Diyu Yang2, Mohammad S. Chowdhury2, Singanallur Venkatakrishnan1, Hassina Z. Bilheux1, Charles A. Bouman2, Gregery T. Buzzard2, Jean-Christophe Bilheux1, George J. Nelson3, Maria Cekanova4, and Ray Gregory1; 1Oak Ridge National Laboratory, 2Purdue University, 3University of Alabama in Huntsville, and 4Integrity Laboratories (United States) [view abstract]
Hyperspectral neutron computed tomography (HSnCT) is a technique that can provide complementary information to X-ray computed tomography (CT). Recently wavelength-dependent imaging beamlines that can measure Bragg edge and resonance spectra have been built at spallation neutron sources. The traditional approach to reconstruction involves acquiring projection data by rotating the object over a predefined set of angles about a single axis. Once all the data has been collected (typically over the span of a few days) the volume is reconstructed using an analytical reconstruction algorithm such as the filtered back projection (FBP). However, this method is not efficient for pulsed sources because : 1) each projection is time-consuming, i.e. usually requiring several hours of acquisition time to reach a reasonable signal-to-noise ratio (SNR); 2) the projection directions are not adapted to the sample features, and 3) the reconstruction is always done at the end of the experiment which might yield low SNR and artifact-ridden reconstructions at which point several days’ worth of measurement time has been spent. In this talk, we will present algorithms for an autonomous HSnCT system that can acquire projection directions in a sample in an adaptive manner and assess the quality of scans in real-time during the measurement. Our method involves acquiring data and reconstructing using a model-based iterative reconstruction approach. Furthermore, after a few projections have been measured, the system utilizes an edge-alignment scanning angle selection algorithm to dynamically decide the next projection direction. Simultaneously, a 3D convolutional neural network (CNN) estimates the reconstruction quality after each scan in order to determine if the scan can be stopped. Our proposed method can lead to dramatic decreases in acquisition time for HSnCT scans while acquiring data from the most informative orientations that are adaptive to the object being measured. For the session: Neutron Imaging Beyond Traditional Radiography.
10:00COIMG-132
Enabling turnkey multiscale imaging/tomography of advanced materials with powerful and intuitive software, Adrian Brügger, Columbia University (United States) [view abstract]
We present the work of the speaker’s Columbia team along with collaborators at the Oak Ridge National Laboratory (ORNL) and Los Alamos National Laboratory (LANL) on novel methods to capture neutron imaging data at multiple length scales using a variety of complementary and simultaneous techniques. Traditional attenuation-based neutron imaging and tomography is generally limited by real-space resolution of sensors, on the order of 50 microns. The full information volume of neutrons is only now being realized through the more widespread use of complementary techniques such as Bragg edge imaging (BEI), neutron grating interferometry (nGI), and hyperspectral imaging. The development of a novel beamline, CUPI ^2 D, the Complex, Unique and Powerful Imaging Instrument for Dynamics at ORNL, presents a unique opportunity to fine-tune these new methods and make them accessible to a broad user base. This development must focus on delivering not only experimental capabilities to users with turnkey workflows in experiment design/execution (sample preparation, doping, alignment, etc.), but most importantly provide data reduction and refinement (computed tomography, porosity measurement, strain mapping, etc.) tools that can be used by engineers and scientists outside the neutron scattering field. The success of new instruments invariably depends on the accessibility of the instrument’s underlying software to a diverse user base. For the session: Neutron Imaging Beyond Traditional Radiography.
10:20 – 10:50 AM Coffee Break
Neutron Imaging Beyond Traditional Radiography (M2)
Session Chairs:
Alexander Long, Los Alamos National Laboratory (United States) and Sven Vogel, Los Alamos National Laboratory (United States)
10:50 AM – 12:30 PM
Market Street
10:50COIMG-133
Characterization of irradiated nuclear transmutation fuel with neutron resonance imaging at LANSCE, Sven C. Vogel1, Thilo Balke1, Charles A. Bouman2, Luca Capriotti3, Jason M. Harp4, Alexander M. Long1, and Brendt Wohlberg1; 1Los Alamos National Laboratory, 2Purdue University, 3Idaho National Laboratory, and 4Oak Ridge National Laboratory (United States) [view abstract]
Short pulse neutron sources uniquely enable mapping the densities of isotopes in irradiated nuclear fuels, providing crucial information for the development of advanced fuel forms. We present results on a sample from an irradiated U-10wt%-1wt%Pd metallic fuel in which the densities of uranium isotopes were mapped in moles per cm2 with neutron absorption resonances at the Los Alamos Neutron Science Center (LANSCE). Nuclear burn-up models exist predicting such isotope densities and imaging methods quantifying these with good confidence are of paramount importance to develop this characterization tool and apply it to accelerate carbon-neutral nuclear fuel development. For the session: Neutron Imaging Beyond Traditional Radiography.
11:10COIMG-134
Advanced neutron imaging techniques at FRM II, Adrian Losko1, Richi Kumar2, Alexander M. Long3, Tobias Neuwirth1, Simon Sebold1, Lucas Sommer1, Anton Tremsin4, Sven C. Vogel3, Alexander Wolfertz1, and Michael Schulz1; 1Technical University Munich (Germany), 2Helmholtz-Zentrum Hereon GmbH (Germany), 3Los Alamos National Laboratory (United States), and 4University of California, Berkeley (United States) [view abstract]
The neutron imaging group at FRM II tries to push the limits of several advanced neutron imaging techniques such as grating interferometry, multi-modal imaging or imaging with polarized neutrons. Due to the limited neutron count rate in such applications, frequently a compromise needs to be made between data quality and measurement time. Moreover, quantitative evaluation is often complicated by background and side effects. In this presentation we will discuss challenges and opportunities for improvements of data quality. For the session: Neutron Imaging Beyond Traditional Radiography.
11:30COIMG-135
Assessment of imaging properties of scintillators at FP60R for neutron imaging applications, Showera H. Haque, Stuart Miller, Stuart Baker, Katherine Walters, and Jesus Castaneda, Nevada National Security Site (United States) [view abstract]
Time-gated neutron radiography provides a non-invasive way to image through dense objects, making it a favorable diagnostic for subcritical experiments and other dynamic experiments. The Nevada National Security Site (NNSS) is investigating using pulsed neutron sources such as those created by a dense plasma focus (DPF) device to produce radiographs of dynamic experiments using deuterium-tritium neutrons. The fast neutron beamline at the WNR facility at LANSCE is used to evaluate the imaging properties of various scintillators to be used on future dynamic experiments. Using the accelerator’s pulsed beam structure, we program our gated imaging camera to trigger repetitively at fixed neutron time-of-flight delays. By integrating many thousands of these 10 ns wide energy bands, we reproduce a dose equivalent to what was produced at experiment. We present results of radiographs of polyethylene rolled edges and tungsten step wedges taken at the WNR Flight Path 60R using plastic slab, crystalline, and deposited scintillators. For the session: Neutron Imaging Beyond Traditional Radiography.
11:50COIMG-136
In-situ thermal neutron imaging of roots in soil at LSU Pennington Lab, Les Butler1, Kyungmin Ham1, J. Theodore Cremer2, Randall Urdahl2, Eugene Guan2, Craig Brown2, Allan Chen2, Charles Gary2, Michael Vincent3, and Charles Hartman3; 1Louisiana State University, 2Adelphi Technology, Inc., and 3Refined Imaging LLC (United States) [view abstract]
Installed at the LSU Pennington Lab, is the radiographic/tomographic imaging system, which is based on a thermal neutron scatter image of the plant/soil system. Revealing details at 1-micron, this thermal neutron scatter, plant root imaging instrument, combines interferometric grating-optics, developed by Refined Imaging of Baton Rouge, and combined with Adelphi Technology’s portable thermal neutron source. For the session: Neutron Imaging Beyond Traditional Radiography.
12:10COIMG-137
LSU and MIT reactor collaboration - Simultaneous X-ray/neutron imaging for energy, materials, and ecosystem applications, Les Butler1, Gerald Schneider1, Markus Bleuel2, Navid Jafari1, Shengmin Guo1, Joyoni Dey1, Boris Khaykovich3, and J. Theodore Cremer4; 1Louisiana State University, 2National Institute of Science and Technology (NIST), 3Massachusetts Institute of Technology, and 4Adelphi Technology, Inc (United States) [view abstract]
Access to combined X-ray/neutron imaging in the US currently exists only at the NIST Center for Neutron Research BT2 beamline. The proposed LSU-MIT project will double the US capacity and will have a much more powerful X-ray imaging system. The MIT neutron tomography system was recently upgraded by NIST researchers, close to specifications of worldwide national facility neutron imaging beamlines. MIT beamline specifications: 1. Thermal neutron beam size of 50 mm x 75 mm at the sample, small beam angular divergence (L/D = 500); 2. Image acquisition time of 100 seconds with 25 micron pixels. LSU will build the X-ray imaging system, both hardware and software, test and evaluate, then move the system to the MIT neutron imaging beamline. The X-ray system will be two-fold: Mode 1 will operate as a Talbot-Lau interferometer based around a Sigray microarray X-ray source. Mode 2 will use a high-energy X-ray source. An X-ray beam path of nearly 3 meters is available. The 5-D data (XYZ, X-ray/neutron, time) is appropriate for the new Louisiana AI/ML supercomputer system. Efficient analysis of the 5-D requires collaborative virtual reality and uses the VR experience developed by LSU chemistry and psychology faculty. For the session: Neutron Imaging Beyond Traditional Radiography.
12:30 – 2:00 PM Lunch
Monday 16 January PLENARY: Neural Operators for Solving PDEs
Session Chair: Robin Jenkin, NVIDIA Corporation (United States)
2:00 PM – 3:00 PM
Cyril Magnin I/II/III
Deep learning surrogate models have shown promise in modeling complex physical phenomena such as fluid flows, molecular dynamics, and material properties. However, standard neural networks assume finite-dimensional inputs and outputs, and hence, cannot withstand a change in resolution or discretization between training and testing. We introduce Fourier neural operators that can learn operators, which are mappings between infinite dimensional spaces. They are independent of the resolution or grid of training data and allow for zero-shot generalization to higher resolution evaluations. When applied to weather forecasting, neural operators capture fine-scale phenomena and have similar skill as gold-standard numerical weather models for predictions up to a week or longer, while being 4-5 orders of magnitude faster.
Anima Anandkumar, Bren professor, California Institute of Technology, and senior director of AI Research, NVIDIA Corporation (United States)
Anima Anandkumar is a Bren Professor at Caltech and Senior Director of AI Research at NVIDIA. She is passionate about designing principled AI algorithms and applying them to interdisciplinary domains. She has received several honors such as the IEEE fellowship, Alfred. P. Sloan Fellowship, NSF Career Award, and Faculty Fellowships from Microsoft, Google, Facebook, and Adobe. She is part of the World Economic Forum's Expert Network. Anandkumar received her BTech from Indian Institute of Technology Madras, her PhD from Cornell University, and did her postdoctoral research at MIT and assistant professorship at University of California Irvine.
3:00 – 3:30 PM Coffee Break
KEYNOTE: Computational Imaging using Fourier Ptychography and Phase Retrieval (M3)
Session Chairs: Tony Allen, Purdue University (United States) and Andre Van Rynbach, U.S. Air Force (United States)
3:30 – 4:50 PM
Market Street
3:30COIMG-138
KEYNOTE: Computational phase imaging, Laura Waller, University of California, Berkeley (United States) [view abstract]
Laura Waller leads the Computational Imaging Lab, which develops new methods for optical imaging, with optics and computational algorithms designed jointly. She holds the Ted Van Duzer Endowed Professorship and is a Senior Fellow at the Berkeley Institute of Data Science (BIDS), with affiliations in Bioengineering and Applied Sciences & Technology. Laura was a Postdoctoral Researcher and Lecturer of Physics at Princeton University from 2010-2012 and received BS, MEng and PhD degrees from MIT in 2004, 2005 and 2010, respectively. She is a Moore Foundation Data-Driven Investigator, Bakar fellow, Distinguished Graduate Student Mentoring awardee, NSF CAREER awardee, Chan-Zuckerberg Biohub Investigator, SPIE Early Career Achievement Awardee and Packard Fellow.
Scattering severely limits the visual acuity of an imaging system. This talk discusses how diversity in illumination wavelength can be utilized to circumvent the problem of phase randomization in scattered light fields. Amongst other applications, the introduced method allows for holographic measurements of hidden objects around corners and through scattering media, or for interferometric measurements of macroscopic objects with rough surfaces. This is possible as the technique interrogates the scene at two closely spaced optical wavelengths and computationally assembles a complex “synthetic field” at a “synthetic wave,” which is used for further processing. As the synthetic wavelength is the beat wavelength of the two optical wavelengths, it can be picked orders of magnitudes larger, and the computationally assembled synthetic field becomes immune to the deleterious effect of speckle. During the talk, different flavors of the technique will be introduced, using the examples of our latest experimental results. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
4:10COIMG-139
I can see clearly now: Sub-diffraction limit synthetic aperture lidar, Tony G. Allen1,2, David J. Rabb2, Gregery T. Buzzard1, and Charles A. Bouman1; 1Purdue University and 2Air Force Research Laboratory (United States) [view abstract]
In synthetic aperture lidar (SAL), simultaneous aperture synthesis and pulse compression of a high bandwidth wave enables high-resolution three-dimensional (3D) imaging capabilities. Unfortunately, like other coherent imaging methods, SAL imagery is subject to noise and speckle. Many methods that aim to reduce these effects operate on the average of images reconstructed from sub-apertures, and hence come at a cost of resolution. Here, we present a model-based image reconstruction algorithm that uses multi-agent consensus equilibrium (MACE) as a framework for creating a joint reconstruction of the 3D image from data collected over the entire synthetic aperture. We show that this approach is able to reconstruct images with suppressed noise and speckle while preserving resolution beyond the diffraction limit. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
4:30COIMG-140
The role of phase retrieval for imaging and beam forming through turbulence, Timothy J. Schulz1 and David J. Brady2; 1Michigan Technological University and 2The University of Arizona (United States) [view abstract]
In this presentation, we'll discuss the upper bounds on performance that phase retrieval can provide for imaging and beam-forming through a turbulent medium. We'll begin by examining performance limits when the medium is perfectly known—as in the presence of an oracle—which provide upper bounds on the performance of any system. Then we'll discuss the ways phase-retrieval can characterize the medium, and show how those methods can advance performance toward the ultimate bounds. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
EI 2023 Highlights Session
Session Chair: Robin Jenkin, NVIDIA Corporation (United States)
3:30 – 5:00 PM
Cyril Magnin II
Join us for a session that celebrates the breadth of what EI has to offer with short papers selected from EI conferences.
NOTE: The EI-wide "EI 2023 Highlights" session is concurrent with Monday afternoon COIMG, COLOR, IMAGE, and IQSP conference sessions.
IQSP-309
Evaluation of image quality metrics designed for DRI tasks with automotive cameras, Valentine Klein, Yiqi LI, Claudio Greco, Laurent Chanas, and Frédéric Guichard, DXOMARK (France) [view abstract]
Driving assistance is increasingly used in new car models. Most driving assistance systems are based on automotive cameras and computer vision. Computer Vision, regardless of the underlying algorithms and technology, requires the images to have good image quality, defined according to the task. This notion of good image quality is still to be defined in the case of computer vision as it has very different criteria than human vision: humans have a better contrast detection ability than image chains. The aim of this article is to compare three different metrics designed for detection of objects with computer vision: the Contrast Detection Probability (CDP) [1, 2, 3, 4], the Contrast Signal to Noise Ratio (CSNR) [5] and the Frequency of Correct Resolution (FCR) [6]. For this purpose, the computer vision task of reading the characters on a license plate will be used as a benchmark. The objective is to check the correlation between the objective metric and the ability of a neural network to perform this task. Thus, a protocol to test these metrics and compare them to the output of the neural network has been designed and the pros and cons of each of these three metrics have been noted.
SD&A-224
Human performance using stereo 3D in a helmet mounted display and association with individual stereo acuity, Bonnie Posselt, RAF Centre of Aviation Medicine (United Kingdom) [view abstract]
Binocular Helmet Mounted Displays (HMDs) are a critical part of the aircraft system, allowing information to be presented to the aviator with stereoscopic 3D (S3D) depth, potentially enhancing situational awareness and improving performance. The utility of S3D in an HMD may be linked to an individual’s ability to perceive changes in binocular disparity (stereo acuity). Though minimum stereo acuity standards exist for most military aviators, current test methods may be unable to characterise this relationship. This presentation will investigate the effect of S3D on performance when used in a warning alert displayed in an HMD. Furthermore, any effect on performance, ocular symptoms, and cognitive workload shall be evaluated in regard to individual stereo acuity measured with a variety of paper-based and digital stereo tests.
IMAGE-281
Smartphone-enabled point-of-care blood hemoglobin testing with color accuracy-assisted spectral learning, Sang Mok Park1, Yuhyun Ji1, Semin Kwon1, Andrew R. O’Brien2, Ying Wang2, and Young L. Kim1; 1Purdue University and 2Indiana University School of Medicine (United States) [view abstract]
We develop an mHealth technology for noninvasively measuring blood Hgb levels in patients with sickle cell anemia, using the photos of peripheral tissue acquired by the built-in camera of a smartphone. As an easily accessible sensing site, the inner eyelid (i.e., palpebral conjunctiva) is used because of the relatively uniform microvasculature and the absence of skin pigments. Color correction (color reproduction) and spectral learning (spectral super-resolution spectroscopy) algorithms are integrated for accurate and precise mHealth blood Hgb testing. First, color correction using a color reference chart with multiple color patches extracts absolute color information of the inner eyelid, compensating for smartphone models, ambient light conditions, and data formats during photo acquisition. Second, spectral learning virtually transforms the smartphone camera into a hyperspectral imaging system, mathematically reconstructing high-resolution spectra from color-corrected eyelid images. Third, color correction and spectral learning algorithms are combined with a spectroscopic model for blood Hgb quantification among sickle cell patients. Importantly, single-shot photo acquisition of the inner eyelid using the color reference chart allows straightforward, real-time, and instantaneous reading of blood Hgb levels. Overall, our mHealth blood Hgb tests could potentially be scalable, robust, and sustainable in resource-limited and homecare settings.
AVM-118
Designing scenes to quantify the performance of automotive perception systems, Zhenyi Liu1, Devesh Shah2, Alireza Rahimpour2, Joyce Farrell1, and Brian Wandell1; 1Stanford University and 2Ford Motor Company (United States) [view abstract]
We implemented an end-to-end simulation for perception systems, based on cameras, that are used in automotive applications. The open-source software creates complex driving scenes and simulates cameras that acquire images of these scenes. The camera images are then used by a neural network in the perception system to identify the locations of scene objects, providing the results as input to the decision system. In this paper, we design collections of test scenes that can be used to quantify the perception system’s performance under a range of (a) environmental conditions (object distance, occlusion ratio, lighting levels), and (b) camera parameters (pixel size, lens type, color filter array). We are designing scene collections to analyze performance for detecting vehicles, traffic signs and vulnerable road users in a range of environmental conditions and for a range of camera parameters. With experience, such scene collections may serve a role similar to that of standardized test targets that are used to quantify camera image quality (e.g., acuity, color).
VDA-403
Visualizing and monitoring the process of injection molding, Christian A. Steinparz1, Thomas Mitterlehner2, Bernhard Praher2, Klaus Straka1,2, Holger Stitz1,3, and Marc Streit1,3; 1Johannes Kepler University, 2Moldsonics GmbH, and 3datavisyn GmbH (Austria) [view abstract]
In injection molding machines the molds are rarely equipped with sensor systems. The availability of non-invasive ultrasound-based in-mold sensors provides better means for guiding operators of injection molding machines throughout the production process. However, existing visualizations are mostly limited to plots of temperature and pressure over time. In this work, we present the result of a design study created in collaboration with domain experts. The resulting prototypical application uses real-world data taken from live ultrasound sensor measurements for injection molding cavities captured over multiple cycles during the injection process. Our contribution includes a definition of tasks for setting up and monitoring the machines during the process, and the corresponding web-based visual analysis tool addressing these tasks. The interface consists of a multi-view display with various levels of data aggregation that is updated live for newly streamed data of ongoing injection cycles.
COIMG-155
Commissioning the James Webb Space Telescope, Joseph M. Howard, NASA Goddard Space Flight Center (United States) [view abstract]
Astronomy is arguably in a golden age, where current and future NASA space telescopes are expected to contribute to this rapid growth in understanding of our universe. The most recent addition to our space-based telescopes dedicated to astronomy and astrophysics is the James Webb Space Telescope (JWST), which launched on 25 December 2021. This talk will discuss the first six months in space for JWST, which were spent commissioning the observatory with many deployments, alignments, and system and instrumentation checks. These engineering activities help verify the proper working of the telescope prior to commencing full science operations. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
HVEI-223
Critical flicker frequency (CFF) at high luminance levels, Alexandre Chapiro1, Nathan Matsuda1, Maliha Ashraf2, and Rafal Mantiuk3; 1Meta (United States), 2University of Liverpool (United Kingdom), and 3University of Cambridge (United Kingdom) [view abstract]
The critical flicker fusion (CFF) is the frequency of changes at which a temporally periodic light will begin to appear completely steady to an observer. This value is affected by several visual factors, such as the luminance of the stimulus or its location on the retina. With new high dynamic range (HDR) displays, operating at higher luminance levels, and virtual reality (VR) displays, presenting at wide fields-of-view, the effective CFF may change significantly from values expected for traditional presentation. In this work we use a prototype HDR VR display capable of luminances up to 20,000 cd/m^2 to gather a novel set of CFF measurements for never before examined levels of luminance, eccentricity, and size. Our data is useful to study the temporal behavior of the visual system at high luminance levels, as well as setting useful thresholds for display engineering.
HPCI-228
Physics guided machine learning for image-based material decomposition of tissues from simulated breast models with calcifications, Muralikrishnan Gopalakrishnan Meena1, Amir K. Ziabari1, Singanallur Venkatakrishnan1, Isaac R. Lyngaas1, Matthew R. Norman1, Balint Joo1, Thomas L. Beck1, Charles A. Bouman2, Anuj Kapadia1, and Xiao Wang1; 1Oak Ridge National Laboratory and 2Purdue University (United States) [view abstract]
Material decomposition of Computed Tomography (CT) scans using projection-based approaches, while highly accurate, poses a challenge for medical imaging researchers and clinicians due to limited or no access to projection data. We introduce a deep learning image-based material decomposition method guided by physics and requiring no access to projection data. The method is demonstrated to decompose tissues from simulated dual-energy X-ray CT scans of virtual human phantoms containing four materials - adipose, fibroglandular, calcification, and air. The method uses a hybrid unsupervised and supervised learning technique to tackle the material decomposition problem. We take advantage of the unique X-ray absorption rate of calcium compared to body tissues to perform a preliminary segmentation of calcification from the images using unsupervised learning. We then perform supervised material decomposition using a deep learned UNET model which is trained using GPUs in the high-performant systems at the Oak Ridge Leadership Computing Facility. The method is demonstrated on simulated breast models to decompose calcification, adipose, fibroglandular, and air.
3DIA-104
Layered view synthesis for general images, Loïc Dehan, Wiebe Van Ranst, and Patrick Vandewalle, Katholieke University Leuven (Belgium) [view abstract]
We describe a novel method for monocular view synthesis. The goal of our work is to create a visually pleasing set of horizontally spaced views based on a single image. This can be applied in view synthesis for virtual reality and glasses-free 3D displays. Previous methods produce realistic results on images that show a clear distinction between a foreground object and the background. We aim to create novel views in more general, crowded scenes in which there is no clear distinction. Our main contributions are a computationally efficient method for realistic occlusion inpainting and blending, especially in complex scenes. Our method can be effectively applied to any image, which is shown both qualitatively and quantitatively on a large dataset of stereo images. Our method performs natural disocclusion inpainting and maintains the shape and edge quality of foreground objects.
ISS-329
A self-powered asynchronous image sensor with independent in-pixel harvesting and sensing operations, Ruben Gomez-Merchan, Juan Antonio Leñero-Bardallo, and Ángel Rodríguez-Vázquez, University of Seville (Spain) [view abstract]
A new self-powered asynchronous sensor with a novel pixel architecture is presented. Pixels are autonomous and can harvest or sense energy independently. During the image acquisition, pixels toggle to a harvesting operation mode once they have sensed their local illumination level. With the proposed pixel architecture, most illuminated pixels provide an early contribution to power the sensor, while low illuminated ones spend more time sensing their local illumination. Thus, the equivalent frame rate is higher than the offered by conventional self-powered sensors that harvest and sense illumination in independient phases. The proposed sensor uses a Time-to-First-Spike readout that allows trading between image quality and data and bandwidth consumption. The sensor has HDR operation with a dynamic range of 80 dB. Pixel power consumption is only 70 pW. In the article, we describe the sensor’s and pixel’s architectures in detail. Experimental results are provided and discussed. Sensor specifications are benchmarked against the art.
COLOR-184
Color blindness and modern board games, Alessandro Rizzi1 and Matteo Sassi2; 1Università degli Studi di Milano and 2consultant (Italy) [view abstract]
Board game industry is experiencing a strong renewed interest. In the last few years, about 4000 new board games have been designed and distributed each year. Board game players gender balance is reaching the equality, but nowadays the male component is a slight majority. This means that (at least) around 10% of board game players are color blind. How does the board game industry deal with this ? Recently, a raising of awareness in the board game design has started but so far there is a big gap compared with (e.g.) the computer game industry. This paper presents some data about the actual situation, discussing exemplary cases of successful board games.
5:00 – 6:15 PM EI 2023 All-Conference Welcome Reception (in the Cyril Magnin Foyer)
Tuesday 17 January 2023
Neutron Imaging Beyond Traditional Radiography (T1)
Session Chairs:
Alexander Long, Los Alamos National Laboratory (United States) and Sven Vogel, Los Alamos National Laboratory (United States)
8:50 – 10:10 AM
Market Street
8:50COIMG-141
Neutron Bragg-edge/dip imaging with least squares method and machine learning, Hirotaka Sato, Hokkaido University (Japan) [view abstract]
Neutron Bragg-edge and Bragg-dip imaging methods, which are energy-resolved neutron transmission imaging methods, are attractive imaging technologies that can non-destructively/invasively visualize bulk information of crystalline microstructure in polycrystalline materials, oligocrystals and single crystals. The visualization area is several-centimeters squared to approximately 10 cm squared. The spatial resolution is sub-millimeters to several-millimeters. Neutrons can easily penetrate a material of several-centimeters thickness. The crystallographic analysis with such spatial recognition ability can be carried out by this method. This method is recently developing owing to neutron time-of-flight (TOF) imaging detectors at particle accelerator-driven pulsed white neutron sources, and optical devices for neutron energy selection at nuclear reactor-based steady-state white neutron sources. For quantitative analysis of crystallographic information, it is required to extract the information from energy-dependent neutron transmission data, neutron transmission spectrum, measured at each pixel of a neutron imaging detector. Therefore, various least squares data analysis methods have been developed for extracting the information of crystalline phase fractions, crystal orientations, grain/crystallite sizes and macro/micro-strains. On the other hand, completely performing such data analyses at all pixels of the imaging detector is a hard work. Thus, several machine learning-assisted data analysis methods have been also developed. As a result, various materials engineering, industrial applications and cultural heritage research using neutron Bragg-edge/dip imaging have been successfully carried out. In this presentation, recent progresses of neutron Bragg-edge/dip imaging with least squares method and machine learning and its applications are reported. For the session: Neutron Imaging Beyond Traditional Radiography.
9:10COIMG-143
Event mode data collection for neutron imaging applications, Adrian Losko1, Jason Gochanour2, Alex Gustschin1, Yiyong Han1, Alexander M. Long2, Manuel Morgano3, Michael Schulz1, Anton Tremsin4, Sven C. Vogel2, and Alexander Wolfertz1; 1Technical University Munich (Germany), 2Los Alamos National Laboratory (United States), 3European Spallation Source (Sweden), and 4University of California, Berkeley (United States) [view abstract]
With the transformative development of event-based detectors, new perspectives for detection systems for various types of radiation were opened up. A recently developed event-driven imaging system based on Timepix3 sensor technology is capable of observing and time-stamping the optical signal induced by particle interactions in scintillator materials with nanosecond temporal and micrometer spatial resolution, providing a pathway to fuse the benefits of integrating camera type with counting type detectors. In this approach, the reconstruction of the interaction position of a neutron with the scintillator with sub-pixel accuracy can provide a precise determination in location, as well as in time-of-arrival of the individual neutrons. Utilizing such a principle, it was shown that spatial and temporal resolution can be improved beyond the classical limits of “regular” neutron imaging. Additionally, a significant increase of signal-to-noise ratio was achieved using the unique potential of event-mode detection to discriminate gamma background from neutron signal based on the spatiotemporal signature of single neutron events produced in the scintillator. Utilizing such sparse data collection, many advantages compared to regular imaging systems emerge and require redesign of existing image processing routines to benefit from this new approach. Here, we present the recent progress in this development with applications to white beam and time-of-flight neutron imaging. For the session: Neutron Imaging Beyond Traditional Radiography.
9:30COIMG-144
Recent developments on diffraction-based and polarized neutron imaging modalities, Søren Schmidt1, Patrick Tung2, Stavros Samothrakitis3, Camilla B. Larsen3, Markus Strobl3, Luise T. Kuhn4, Ryoji Kiyanagi5, and Takenao Shinohara5; 1European Spallation Source ERIC (Sweden), 2University of New South Wales (Australia), 3Paul Scherrer Institute (Switzerland), 4Technical University of Denmark (Denmark), and 5Japan Proton Accelerator Research Complex (J-PARC) Center (Japan) [view abstract]
Various modalities on diffraction-based neutron imaging for spatially resolved crystalline microstructures and polarized neutron imaging for measuring 3D magnetic fields have emerged recently. The talk will briefly outline a few methodologies along with examples. For the session: Neutron Imaging Beyond Traditional Radiography.
10:00 AM – 7:30 PM Industry Exhibition - Tuesday (in the Cyril Magnin Foyer)
10:20 – 10:50 AM Coffee Break
Neutron Imaging Beyond Traditional Radiography (T2)
Session Chairs:
Alexander Long, Los Alamos National Laboratory (United States) and Sven Vogel, Los Alamos National Laboratory (United States)
10:50 – 11:50 AM
Market Street
10:50COIMG-145
Strain tomography using neutrons, Christopher M. Wensrich1, Alexander W. Gregg1, Johannes N. Hendriks1, Anton Tremsin2, Adrian Wills1, Takenao Shinohara3, Oliver Kirstein1, Vladimir Luzin1, and Erich H. Kisi1; 1University of Newcastle (Australia), 2University of California, Berkeley (United States), and 3Japan Proton Accelerator Research Complex (J-PARC) Center (Japan) [view abstract]
Bragg-edge strain imaging presents a natural rich-tomography problem focused on the reconstruction of strain distributions within polycrystalline solids. This problem has been the focus of some attention over the past decade or more following several successful demonstrations in axisymmetric systems. The extension of this work to general two and three-dimensional systems was hampered by the fact that the relevant inverse-problem was inherently ill-posed. This presentation provides an overview of a new approach to this problem based on Gaussian Process regression that involves the application of known constraints such as equilibrium and boundary conditions to the tomographic reconstruction of strain fields. Experimental demonstrations of the technique in two and three dimensions are provided. The latter involved the reconstruction of a 3D strain field within a small steel-aluminum ‘cube-and-plug’ from a set of 70 Bragg-edge strain images measured using the RADEN instrument at J-PARC in Japan in conjunction with an MCP/timepix detector. Comparisons are made to traditional point-wise strain scans from the KOWARI instrument at the Australian Centre for Neutron Scattering and a finite element model. Opportunities and challenges for the wider application of the technique will also be discussed. For the session: Neutron Imaging Beyond Traditional Radiography.
11:10COIMG-146
Data processing for non-destructive studies of material properties through energy resolved neutron imaging, Anton Tremsin1, Winfried Kockelmann2, Daniel Pooley2, Saurabh Kabra2, Takenao Shinohara3, Kenichi Oikawa3, Hassina Z. Bilheux4, Jean-Christophe Bilheux4, Adrian Losko5, Sven C. Vogel6, Alexander M. Long6, John Rakovan7, Christopher M. Wensrich8, Florencia Malamud9, Markus Strobl9, and Javier Santisteban10; 1University of California, Berkeley (United States), 2STFC-Rutherford Appleton Laboratory (United Kingdom), 3Japan Proton Accelerator Research Complex (J-PARC) Center (Japan), 4Oak Ridge National Laboratory (United States), 5Forschungs-Neutronenquelle Heinz Maier-Leibnitz (Germany), 6Los Alamos National Laboratory (United States), 7Miami University (United States), 8University of Newcastle (Australia), 9Paul Scherrer Institut (PSI) (Switzerland), and 10Comisión Nacional de Energía Atómica CNEA/CONICET (Argentina) [view abstract]
Recent development of various modes of energy resolved neutron imaging dramatically enhanced the capabilities of many non-destructive testing studies far beyond conventional radiography. The possibility to measure the neutron transmission spectrum in each pixel of imaging dataset enables many new experimental capabilities, among which are spatially-resolved characterization of microstructure and elemental composition within crystalline materials, in-situ diagnostics of crystal growth, temperature mapping of enclosed materials and many others. The reconstruction of these properties requires sometimes quite substantial data processing and analysis in order to reconstruct the needed maps and distributions with sub 100 µm resolution over several cm area. These experiments are enabled by the development of two crucial components: bright pulsed neutron sources and fast neutron counting detection technology. In this paper we will present results of several experimental studies where extensive computation was involved in reconstruction of required maps of material parameters. Reconstruction of residual strain, temperature distribution, elemental composition with spatial resolution as low as 55 µm in some cases will be presented for various engineering materials and assemblies, single crystal scintillators, unique geological samples and others. For the session: Neutron Imaging Beyond Traditional Radiography.
11:30COIMG-148
Neutron imaging at LANSCE: Characterizing materials for the next generation of nuclear reactor designs, Alexander M. Long, Sven C. Vogel, James Torres, D. Travis Carver, S. Scott Parker, Marisa Monreal, J. Matthew Jackson, Holly Trellue, Aditya Shivprasad, Caitlin Taylor, and Erik Luther, Los Alamos National Laboratory (United States) [view abstract]
Neutrons are an ideal probe for characterizing nuclear fuels and moderator materials for next generation nuclear reactors as their interactions with matter create complex attenuations that result in a unique combination of isotopic specific contrast mechanisms and penetrabilities, thus making neutrons well suited for investigating both high-z materials (actinides in nuclear fuels) and low-z materials (metal hydrides). Furthermore, the high material penetrability with neutron imaging allows for in-situ measurements at extreme conditions (high temperatures or activity) where bulky sample environments are required. Presented work will include the ongoing efforts at the Los Alamos Neutron Science Center (LANSCE) to develop advanced neutron imaging capabilities on Flight Path 5 (FP5) specifically for characterizing materials for advanced reactor designs. These efforts range from thermophysical property measurements of chloride-based molten salts, to hydrogen characterization in metal hydrides moderator materials, to post-irradiation examination with energy resolved neutron imaging of actinides in fresh and irradiated fuels. For the session: Neutron Imaging Beyond Traditional Radiography.
12:30 – 2:00 PM Lunch
Tuesday 17 January PLENARY: Embedded Gain Maps for Adaptive Display of High Dynamic Range Images
Session Chair: Robin Jenkin, NVIDIA Corporation (United States)
2:00 PM – 3:00 PM
Cyril Magnin I/II/III
Images optimized for High Dynamic Range (HDR) displays have brighter highlights and more detailed shadows, resulting in an increased sense of realism and greater impact. However, a major issue with HDR content is the lack of consistency in appearance across different devices and viewing environments. There are several reasons, including varying capabilities of HDR displays and the different tone mapping methods implemented across software and platforms. Consequently, HDR content authors can neither control nor predict how their images will appear in other apps.
We present a flexible system that provides consistent and adaptive display of HDR images. Conceptually, the method combines both SDR and HDR renditions within a single image and interpolates between the two dynamically at display time. We compute a Gain Map that represents the difference between the two renditions. In the file, we store a Base rendition (either SDR or HDR), the Gain Map, and some associated metadata. At display time, we combine the Base image with a scaled version of the Gain Map, where the scale factor depends on the image metadata, the HDR capacity of the display, and the viewing environment.
Eric Chan, Fellow, Adobe Inc. (United States)
Eric Chan is a Fellow at Adobe, where he develops software for editing photographs. Current projects include Photoshop, Lightroom, Camera Raw, and Digital Negative (DNG). When not writing software, Chan enjoys spending time at his other keyboard, the piano. He is an enthusiastic nature photographer and often combines his photo activities with travel and hiking.
Paul M. Hubel, director of Image Quality in Software Engineering, Apple Inc. (United States)
Paul M. Hubel is director of Image Quality in Software Engineering at Apple. He has worked on computational photography and image quality of photographic systems for many years on all aspects of the imaging chain, particularly for iPhone. He trained in optical engineering at University of Rochester, Oxford University, and MIT, and has more than 50 patents on color imaging and camera technology. Hubel is active on the ISO-TC42 committee Digital Photography, where this work is under discussion, and is currently a VP on the IS&T Board. Outside work he enjoys photography, travel, cycling, coffee roasting, and plays trumpet in several bay area ensembles.
3:00 – 3:30 PM Coffee Break
Computational Imaging using Fourier Ptychography and Phase Retrieval (T3)
Session Chairs:
Tony Allen, Purdue University (United States) and Andre Van Rynbach, U.S. Air Force (United States)
3:30 – 5:30 PM
Market Street
3:30COIMG-150
Scatter ptychography, David J. Brady, The University of Arizona (United States) [view abstract]
Phase retrieval on direct or secondary scatter enables remote object identification and imaging at resolution 10-100x finer than the single aperture diffraction limit. This talk presents theoretical analysis and experimental results exploring the limits of this technique. We describe experiments demonstrating <50 micron feature reconstruction at a range of 100 m, corresponding to sub-microradian resolution. We consider the feasibility and potential applications of nanoradian resolving systems. We discuss the limits of scatter imaging on terrestrial targets, drones, space debris and asteroids. We compare scatter imaging of laser and solar-illuminated targets and discuss resolution limits as a function of source coherence. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
3:50COIMG-151
Diffractive optical networks & computational imaging without a computer, Aydogan Ozcan, UCLA (United States) [view abstract]
I will discuss diffractive optical networks designed by deep learning to all-optically implement various complex functions as the input light diffracts through spatially-engineered surfaces. These diffractive processors designed by deep learning have various applications, e.g., all-optical image analysis, feature detection, object classification, computational imaging and seeing through diffusers, also enabling task-specific camera designs and new optical components for spatial, spectral and temporal beam shaping and spatially-controlled wavelength division multiplexing. These deep learning-designed diffractive systems can broadly impact (1) all-optical statistical inference engines, (2) computational camera and microscope designs and (3) inverse design of optical systems that are task-specific. In this talk, I will give examples of each group, enabling transformative capabilities for various applications of interest in e.g., autonomous systems, defense/security, telecommunications as well as biomedical imaging and sensing. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
4:10COIMG-152
Computational microscopy of scattering samples, Shwetadwip Chowdhury, University of Texas at Austin (United States) [view abstract]
Optical imaging is a major research tool in the basic sciences, and provides both morphological and molecular-specific imaging capabilities. Furthermore, it is the only imaging modality that routinely enables non-ionized imaging with subcellular spatial resolutions and high imaging speeds. In biological imaging applications, however, optical imaging is limited by tissue scattering to short imaging depths. This enables optical imaging of only outer superficial layers of an organism, or specific components isolated from within the organism and prepared in-vitro. I present recent developments in computational microscopy that enable 1) 3D phase and fluorescent super-resolution using optical scattering; and 2) 3D refractive-index imaging of heterogeneously scattering samples. I will discuss the computational frameworks that underpin these applications, which are based on large-scale nonlinear and nonconvex optimization. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
4:30COIMG-153
Practical phase retrieval using double deep image priors, Zhong Zhuang1, David Yang2, Felix Hofmann2, David Barmherzig3, and Ju Sun1; 1University of Minnesota, Twin Cities (United States), 2University of Oxford (United Kingdom), and 3Flatiron Institute (United States) [view abstract]
Phase retrieval (PR) consists of recovering complex-valued objects from their oversampled Fourier magnitudes and takes a central place in scientific imaging. A critical issue around PR is the typical nonconvexity in natural formulations and the associated bad local minimizers. The issue is exacerbated when the support of the object is not precisely known and hence must be overspecified in practice. Practical methods for PR hence involve convolved algorithms, e.g., multiple cycles of hybrid input-output (HIO) + error reduction (ER), to avoid the bad local minimizers and attain reasonable speed, and heuristics to refine the support of the object, e.g., the famous shrinkwrap trick. Overall, the convolved algorithms and the support-refinement heuristics induce multiple algorithm hyperparameters, to which the recovery quality is often sensitive. In this work, we propose a novel PR method by parameterizing the object as the output of a learnable neural network, i.e., deep image prior (DIP). For complex-valued objects in PR, we can flexibly parametrize the magnitude and phase, or the real and imaginary parts separately by two DIPs. We show that this simple idea, free from multi-hyperparameter tuning and support-refinement heuristics, can obtain superior performance than gold-standard PR methods. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
4:50COIMG-154
Synthetic wavelength imaging - Exploiting spectral diversity for absolute phase measurements through scattering scenes, Florian Willomitzer, University of Arizona (United States) [view abstract]
Scattering severely limits the visual acuity of an imaging system. This talk discusses how diversity in illumination wavelength can be utilized to circumvent the problem of phase randomization in scattered light fields. Amongst other applications, the introduced method allows for holographic measurements of hidden objects around corners and through scattering media, or for interferometric measurements of macroscopic objects with rough surfaces. This is possible as the technique interrogates the scene at two closely spaced optical wavelengths and computationally assembles a complex “synthetic field” at a “synthetic wave,” which is used for further processing. As the synthetic wavelength is the beat wavelength of the two optical wavelengths, it can be picked orders of magnitudes larger, and the computationally assembled synthetic field becomes immune to the deleterious effect of speckle. During the talk, different flavors of the technique will be introduced, using the examples of our latest experimental results. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
5:10COIMG-155
Commissioning the James Webb Space Telescope, Joseph M. Howard, NASA Goddard Space Flight Center (United States) [view abstract]
Astronomy is arguably in a golden age, where current and future NASA space telescopes are expected to contribute to this rapid growth in understanding of our universe. The most recent addition to our space-based telescopes dedicated to astronomy and astrophysics is the James Webb Space Telescope (JWST), which launched on 25 December 2021. This talk will discuss the first six months in space for JWST, which were spent commissioning the observatory with many deployments, alignments, and system and instrumentation checks. These engineering activities help verify the proper working of the telescope prior to commencing full science operations. For the session: Computational Imaging using Fourier Ptychography and Phase Retrieval.
5:30 – 7:00 PM EI 2023 Symposium Demonstration Session (in the Cyril Magnin Foyer)
Wednesday 18 January 2023
KEYNOTE: Processing at the Edge (W1)
Session Chairs: Stanley Chan, Purdue University (United States) and Boyd Fowler, OmniVision Technologies (United States)
8:45 – 10:20 AM
Market Street
This session is jointly sponsored by: Computational Imaging XXI, Imaging Sensors and Systems 2023, and the International Image Sensor Society (IISS).
8:45
COIMG/ISS Joint Sessions Welcome
8:50COIMG-177
KEYNOTE: Deep optics: Learning cameras and optical computing systems, Gordon Wetzstein, Stanford University (United States) [view abstract]
Gordon Wetzstein is an Associate Professor of Electrical Engineering and, by courtesy, of Computer Science at Stanford University. He is the leader of the Stanford Computational Imaging Lab and a faculty co-director of the Stanford Center for Image Systems Engineering. At the intersection of computer graphics and vision, artificial intelligence, computational optics, and applied vision science, Prof. Wetzstein's research has a wide range of applications in next-generation imaging, wearable computing, and neural rendering systems. Prof. Wetzstein is a Fellow of Optica and the recipient of numerous awards, including an NSF CAREER Award, an Alfred P. Sloan Fellowship, an ACM SIGGRAPH Significant New Researcher Award, a Presidential Early Career Award for Scientists and Engineers (PECASE), an SPIE Early Career Achievement Award, an Electronic Imaging Scientist of the Year Award, an Alain Fournier Ph.D. Dissertation Award as well as many Best Paper and Demo Awards.
Neural networks excel at a wide variety of imaging and perception tasks, but their high performance also comes at a high computational cost and their success on edge devices is often limited. In this talk, we explore hybrid optical-electronic strategies to computational imaging that outsource parts of the algorithm into the optical domain or into emerging in-pixel processing capabilities. Using such a co-design of optics, electronics, and image processing, we can learn application-domain-specific cameras using modern artificial intelligence techniques or compute parts of a convolutional neural network in optics with little to no computational overhead. For the session: Processing at the Edge (joint with ISS).
9:40COIMG-178
Computational photography on a smartphone, Michael Polley, Samsung Research America (United States) [view abstract]
Many of the recent advances in smartphone camera quality and features can be attributed to computational photography. However, the increased computational requirements must be balanced with cost, power, and other practical concerns. In this talk, we look at the embedded signal processing currently applied, including new AI-based solutions in the signal chain. By taking advantage of increasing computational performances of traditional processor cores, and additionally tapping into the exponentially increasing capabilities of the new compute engines such as neural processing units, we are able to deliver on-device computational imaging. For the session: Processing at the Edge (joint with ISS).
10:00COIMG-179
Analog in-memory computing with multilevel RRAM for edge electronic imaging application, Glenn Ge, Teramem Inc. (United States) [view abstract]
Conventional digital processors based on the von Neumann architecture have an intrinsic bottleneck in data transfer between processing and memory units. This constraint increasingly limits performance as data sets continue to grow exponentially for the various applications, especially for the Electronic Imaging Applications at the edge, for instance, the AR/VR wearable and automotive applications. TetraMem addresses this issue by delivering state-of-the-art in-memory computing using our proprietary non-volatile computing devices. This talk will discuss how TetraMem’s solution brings several orders of magnitude improvement in computing throughput and energy efficiency, ideal for those AI fusion sensing applications at the edge. For the session: Processing at the Edge (joint with ISS).
10:00 AM – 3:30 PM Industry Exhibition - Wednesday (in the Cyril Magnin Foyer)
10:20 – 10:50 AM Coffee Break
Processing at the Edge (W2.1)
Session Chairs:
Stanley Chan, Purdue University (United States) and Boyd Fowler, OmniVision Technologies (United States)
10:50 – 11:50 AM
Market Street
This session is jointly sponsored by: Computational Imaging XXI, Imaging Sensors and Systems 2023, and the International Image Sensor Society (IISS).
10:50COIMG-180
Processing of real time, bursty and high compute iToF data on the edge (Invited), Cyrus Bamji, Microsoft Corporation (United States) [view abstract]
In indirect time of flight (iToF), a depth frame is computed from multiple image captures (often 6-9 captures) which are composed together and processed using nonlinear filters. iToF sensor output bandwidth is high and inside the camera special purpose DSP hardware significantly improves power, cost and shuffling around of large amounts of data. Usually only a small percentage of depth frames need application specific processing and highest quality depth data both of which are difficult to compute within the limited hardware resources of the camera. Due to the sporadic nature of these compute requirements hardware utilization is improved by offloading this bursty compute to outside the camera. Many applications in the Industrial and commercial space have a real time requirement and may even use multiple cameras that need to be synchronized. These real time requirements coupled with the high bandwidth from the sensor makes offloading the compute purely into the cloud difficult. Thus, in many cases the compute edge can provide a goldilocks zone for this bursty high bandwidth and real-time processing requirement. For the session: Processing at the Edge (joint with ISS).
11:10COIMG-181
A distributed on-sensor compute system in AR/VR devices and neural architecture search (NAS) framework for optimal workload distribution (Invited), Chiao Liu1, Xin Dong2, Ziyun Li1, Barbara De Salvo3, and H. T. Kung2; 1Reality Labs, 2Harvard University, and 3Meta (United States) [view abstract]
Augmented Reality (AR) will be the next great wave of human oriented computing, dominating our relationship with the digital world for the next 50 years. The combined requirements of lowest power, best performance, and minimal form factor makes AR sensors the new frontier. Previously we presented a digital pixel sensor (DPS) that could be the optimal sensor architecture for AR applications. We further presented a distributed on-sensor compute architecture, coupled with new 3-layer sensor stacking technologies to enable the system to distribute the computation between sensors and main SoC in an AR system. In this talk, we study a deep neural network (DNN) as work load example and the network’s optimal splitting layer location to meet system performance requirements such as inference accuracy and latency under the given hardware resource constraint. We designed a split-aware neural architecture search (NAS) framework, SplitNets, to conduct model design, split, and communication reduction simultaneously. We validated SplitNets on ImageNet, and show that the SplitNets framework achieves state-of-the-art (SOTA) performance and system latency compared with existing approaches. For the session: Processing at the Edge (joint with ISS).
11:30ISS-182
A 2.2um three-wafer stacked back side illuminated voltage domain global shutter CMOS image sensor, Shimpei Fukuoka, OmniVision (Japan) [view abstract]
Due to the emergence of machine vision, augmented reality (AR), virtual reality (VR), and automotive connectivity in recent years, the necessity for chip miniaturization has grown. These emerging, next-generation applications, which are centered on user experience and comfort, require their constituent chips, devices, and parts to be smaller, lighter, and more accessible. AR/VR applications, especially demand smaller components due to their primary application towards wearable technology, in which the user experience would be negatively impacted by large features and bulk. Therefore, chips and devices intended for next-generation consumer applications must be small and modular, to support module miniaturization and promote user comfort. To enable the chip miniaturization required for technological advancement and innovation, we developed a 2.2μm pixel pitch Back Side Illuminated (BSI) Voltage Domain Global Shutter (VDGS) image sensor with the three-wafer stacked technology. Each wafer is connected by Stacked Pixel Level Connection (SPLC) and the middle and logic wafers are connected using a Back side Through Silicon Via (BTSV). The separation of the sensing, charge storage, and logic functions to different wafers allows process optimization in each wafer, improving overall chip performance. The peripheral circuit region is reduced by 75% compared to the previous product without degrading image sensor performance. For the session: Processing at the Edge (joint with COIMG).
HDR Imaging / Reflection Removal (W2.2)
Session Chair:
Gregery Buzzard, Purdue University (United States)
11:50 AM – 12:30 PM
Market Street
11:50COIMG-156
A lightweight exposure bracketing strategy for HDR imaging without access to camera raw, Jieyu Li1, Ruiwen Zhen2, and Robert L. Stevenson1; 1University of Notre Dame and 2SenseBrain Technology (United States) [view abstract]
A lightweight learning-based exposure bracketing strategy is proposed in this paper for high dynamic range (HDR) imaging without access to camera RAW. Some low-cost, power-efficient cameras, such as webcams, video surveillance cameras, sport cameras, mid-tier cellphone cameras, and navigation cameras on robots, can only provide access to 8-bit low dynamic range (LDR) images. Exposure fusion is a classical approach to capture HDR scenes by fusing images taken with different exposures into a 8-bit tone-mapped HDR image. A key question is what the optimal set of exposure settings are to cover the scene dynamic range and achieve a desirable tone. The proposed lightweight neural network predicts these exposure settings for a 3-shot exposure bracketing, given the input irradiance information from 1) the histograms of an auto-exposure LDR preview image, and 2) the maximum and minimum levels of the scene irradiance. Without the processing of the preview image streams, and the circuitous route of first estimating the scene HDR irradiance and then tone-mapping to 8-bit images, the proposed method gives a more practical HDR enhancement for real-time and on-device applications. Experiments on a number of challenging images reveal the advantages of our method in comparison with other state-of-the-art methods qualitatively and quantitatively.
12:10COIMG-157
Sparse x-ray phase contrast dark field tomography, Johnathan Mulcahy-Stanislawczyk and Amber L. Dagel, Sandia National Laboratories (United States) [view abstract]
X-Ray Phase Contrast Imaging (XPCI) augments absorption radiography with additional information related to the refractive and scattering properties of a sample. Grating-based XPCI allows broadband laboratory x-ray sources to be used, increasing the technique’s accessibility. However, grating-based techniques require repeatedly moving a grating and capturing an image at each location. Additionally, the gratings themselves are absorptive, reducing x-ray flux. As a result, data acquisition times and radiation dosages present a hurdle to practical application of XPCI tomography. We present a plug-and-play (PnP) reconstruction method for XPCI dark field tomographic reconstruction with sparse views. Dark field XPCI radiographs contain information about a sample’s microstructure and scatter. The dark field reveals subpixel sample properties, including crystalline structure, graininess, and material interfaces. This makes dark field images differently distributed from traditional absorption radiographs and natural imagery. PnP methods give greater control over reconstruction regularization compared to traditional iterative reconstruction techniques, which is especially useful given the dark field’s unique distribution. PnP allows us to collect dark field tomographic datasets with fewer projections, increasing XPCI’s practicality by reducing the amount of data needed for 3D reconstruction.
12:30 – 2:00 PM Lunch
Wednesday 18 January PLENARY: Bringing Vision Science to Electronic Imaging: The Pyramid of Visibility
Session Chair: Andreas Savakis, Rochester Institute of Technology (United States)
2:00 PM – 3:00 PM
Cyril Magnin I/II/III
Electronic imaging depends fundamentally on the capabilities and limitations of human vision. The challenge for the vision scientist is to describe these limitations to the engineer in a comprehensive, computable, and elegant formulation. Primary among these limitations are visibility of variations in light intensity over space and time, of variations in color over space and time, and of all of these patterns with position in the visual field. Lastly, we must describe how all these sensitivities vary with adapting light level. We have recently developed a structural description of human visual sensitivity that we call the Pyramid of Visibility, that accomplishes this synthesis. This talk shows how this structure accommodates all the dimensions described above, and how it can be used to solve a wide variety of problems in display engineering.
Andrew B. Watson, chief vision scientist, Apple Inc. (United States)
Andrew Watson is Chief Vision Scientist at Apple, where he leads the application of vision science to technologies, applications, and displays. His research focuses on computational models of early vision. He is the author of more than 100 scientific papers and 8 patents. He has 21,180 citations and an h-index of 63. Watson founded the Journal of Vision, and served as editor-in-chief 2001-2013 and 2018-2022. Watson has received numerous awards including the Presidential Rank Award from the President of the United States.
3:00 – 3:30 PM Coffee Break
Imaging with Coded Apertures (W3)
Session Chair:
Xiaogang Yang, Brookhaven National Laboratory (United States)
3:30 – 5:30 PM
Market Street
3:30COIMG-162
X-ray phase contrast imaging using apertures: From proof-of-concept experiments at synchrotrons to pre-commercial prototypes with conventional sources, Alessandro Olivo, University College London (United Kingdom) [view abstract]
X-ray phase contrast imaging using apertures: from proof-of-concept experiments at synchrotrons to pre-commercial prototypes with conventional sources. For the session: Imaging with Coded Apertures.
3:50COIMG-163
Deep regularization functions for coded-aperture design in computational imaging, Roman Jacome, Emmanuel Martinez, Jorge Bacca, and Henry Arguello Fuentes, Universidad Industrial de Santander (Colombia) [view abstract]
Deep regularization functions for coded-aperture design in computational imaging. For the session: Imaging with Coded Apertures.
4:10COIMG-160
CodEx: A modular framework for joint temporal de-blurring and tomographic reconstruction, Soumendu Majee1, Selin Aslan2, Doga Gursoy2, and Charles A. Bouman3; 1Samsung Research America, 2Argonne National Laboratory, and 3Purdue University (United States) [view abstract]
In many computed tomography (CT) imaging applications, it is important to rapidly collect data from an object that is moving or changing with time. Tomographic acquisition is generally assumed to be step-and-shoot, where the object is rotated to each desired angle, and a view is taken. However, step-and-shoot acquisition is slow and can waste photons, so in practice fly-scanning is done where the object is continuously rotated while collecting data. However, this can result in motion-blurred views and consequently reconstructions with severe motion artifacts. In this paper, we introduce CodEx, a modular framework for joint de-blurring and tomographic reconstruction that can effectively invert the motion blur introduced in sparse view fly-scanning. The method is a synergistic combination of a novel acquisition method with a novel non-convex Bayesian reconstruction algorithm. CodEx works by encoding the acquisition with a known binary code that the reconstruction algorithm then inverts. Using a well chosen binary code to encode the measurements can improve the accuracy of the inversion process. The CodEx reconstruction method uses the alternating direction method of multipliers (ADMM) to split the inverse problem into iterative deblurring and reconstruction sub-problems, making reconstruction practical to implement. We present reconstruction results on both simulated and binned experimental data to demonstrate the effectiveness of our method. For the session: Imaging with Coded Apertures.
4:30COIMG-158
First use of coded-apertures for depth-resolved Laue diffraction, Doga Gursoy, Dina Sheyfer, Michael J. Wojcik, Wenjun Liu, and Jon Tischler, Argonne National Laboratory (United States) [view abstract]
First use of coded-apertures for depth-resolved Laue diffraction. For the session: Imaging with Coded Apertures.
4:50COIMG-159
Deep learning image reconstruction for Laue microdiffraction with coded-apertures, Xiaogang Yang1, Esther Tsai1, and Doga Gursoy2; 1Brookhaven National Laboratory and 2Argonne National Laboratory (United States) [view abstract]
Scanning coded-apertures across the diffracted x-ray beams enabled the structural measurement of crystal materials such as orientation and strain with depth information. It resolves full diffraction information at sub-micrometer volume elements in bulk materials with a rapid process. The reconstruction from scanned signal to structural information is a typical inverse problem for linear equations. Specific sampling and noise conditions are required for a decent quality of reconstruction with conventional methods. However, these conditions may be varied in real measurements. We will present our study on deep learning reconstructions for coded-aperture measurements. A model-based image reconstruction with Deep Neural Networks will be applied for the coded-aperture modality. We will demonstrate the robust reconstruction process under undersampled and noisy data conditions. For the session: Imaging with Coded Apertures.
5:10COIMG-161
Coded aperture fabrication for x-ray experiments at the Advanced Photon Source, Michael J. Wojcik, Dina Sheyfer, Doga Gursoy, Jon Tischler, Ralu Divan, and David Czaplewski, Argonne National Laboratory Advanced Photon Source (United States) [view abstract]
The Advanced Photon Source (APS) is a third generation X-ray synchrotron with more than 50 operating beamlines using X-rays to study materials and systems. Beamlines at APS use coded apertures for many X-ray techniques and the APS Optics group has a Nanofabricated Optics section whose purpose is mostly to develop and fabricate these optics. The coded aperture optics include gratings, zone plates, random arrays, and coded bit arrays. Through utilizing the cleanroom at the Center for Nanoscale Materials a variety of recipes have been created for sub-20 nm smallest zone width zone plates to millimeter area 2-D gratings with micron pitch. The coded apertures have been used in experiments to measure coherence of the APS source, improve depth resolution measurements for Laue diffraction measurements, and many others. This paper will also discuss fabrication limitations and future possibilities for coded apertures. For the session: Imaging with Coded Apertures.
Computational Imaging XXI Interactive (Poster) Paper Session
5:30 – 7:00 PM
Cyril Magnin Foyer
The following work will be presented at the EI 2023 Symposium Interactive (Poster) Paper Session.
COIMG-164
Spectral recovery in a photograph with a hyperspectral color chart, Semin Kwon, Sang Mok Park, Yuhyun Ji, Jungwoo Leem, and Young L. Kim, Purdue University (United States) [view abstract]
Here we introduce an idea of photospectroscopy that can recover ambient spectral information from a photograph with a color reference chart taken by a smartphone camera. This color reference chart has an analogy with a hyperspectral filter array and provides the base information to recover the hidden spectral information in a photograph. As the number of colors configured in the color reference chart is abundant, an ill-posed problem of reconstructing the spectral information from RGB values can be formulated as an overdetermined problemWe test different scenarios and samples using several smartphone models and light conditions to evaluate the performance of spectral recovery. The spectral range of the photospectroscopy method covers a broad range of the visible light determined by the spectral response functions in the R, G, and B channels of the smartphone camera (also known as the sensitivity function of the camera). In conclusion, the combination of the specially designed color reference chart and the robust recovery algorithm enables a new concept of photospectroscopy, supporting the idea that a photograph is more than merely an image and actually contains rich spectral information.
5:30 – 7:00 PM EI 2023 Symposium Interactive (Poster) Paper Session (in the Cyril Magnin Foyer)
5:30 – 7:00 PM EI 2023 Meet the Future: A Showcase of Student and Young Professionals Research (in the Cyril Magnin Foyer)
PANEL: Next Generation Imaging-on-a-Chip Tech-Mixer Discussion (W4)
Hosts: Charles Bouman, Purdue University (United States) and Gregery Buzzard, Purdue University (United States)
Panelists: Stanley Chan, Purdue University (United States); Eiichi Funatsu, OmniVision Technologies, Inc. (United States); Sergio Goma, Qualcomm Inc. (United States); Michael Polley, Samsung Research America (United States); and Anton Tremsin, University of California, Berkeley (United States)
6:00 – 7:00 PM
Market Street
The need to both increase imaging capabilities and reduce cost is driving the need for extreme integration of sensing and processing. For example, in the future, analog sensors will be integrated with associated digital processing using methods such as 3D IC stacking. The function of this panel will be to facilitate discussions in the community on the future of imaging-on-a-chip solutions. What problems will these integrated imaging systems be uniquely suited to solve? How can the tight coupling of sensors and hardware be used to enhance capabilities and reduce cost? What should our community be doing to both enhance and exploit this emerging technology? Refreshments included, beer, wine, and snacks!
Thursday 19 January 2023
Computational Imaging Topics (R1)
Session Chair:
Charles Bouman, Purdue University (United States)
8:50 – 10:10 AM
Market Street
8:50COIMG-165
Generative Adversarial Linear Discriminant Analysis (GALDA) for spectroscopy classification and imaging, Ziyi Cao, Shijie Zhang, Youlin Liu, Casey Smith, Alex Sherman, and Garth Simpson, Purdue University (United States) [view abstract]
Generative adversarial linear discriminant analysis (GALDA) is formulated as a broadly applicable tool for increasing classification accuracy and reducing overfitting inherent in representative linear dimension reduction and classification algorithms. Although inspired by the successes of generative adversarial neural networks (GANs) for minimizing overfitting artifacts in artificial neural networks, GALDA was built around an independent linear algebra framework distinct from those in GANs. In contrast to feature extraction and data reduction approaches for minimizing overfitting, GALDA performs data augmentation by identifying and adversarially excluding the regions in spectral space in which genuine data do not reside. Classification accuracy was evaluated for GALDA together with other commonly available supervised and unsupervised methods for dimension reduction in simulated spectra generated using an open-source Raman database (Romanian Database of Raman Spectroscopy, RDRS). In one study, spectral analysis was performed for microscopy measurements of microsphereroids of the blood thinner clopidogrel bisulfate. In another study, THz Raman imaging of common constituents in aspirin tablets were assigned to generate and compare chemically-selective composition maps. From these collective results, the potential scope of use for GALDA is critically evaluated relative to alternative established spectral dimension reduction and classification methods, and GA based method are proved to increase Spectroscopy Imaging resolution on given THz raman imaging case.
9:10COIMG-166
Multi-agent consensus equilibrium (MACE) in electronic structure calculations, Jiayue Rong, Lyudmila Slipchenko, Charles A. Bouman, Gregery T. Buzzard, and Garth Simpson, Purdue University (United States) [view abstract]
A consensus equilibrium formalism is introduced for the integration of multiple quantum chemical calculations of electronic structure. In multi-agent consensus equilibrium, iterative updates in electronic structure optimization are intertwined, with the net output representing an equilibrium balance between multiple computational agents. MACE electronic structure calculations from integration of multiple low-level electronic structure calculations (HF and M06) compared favorably for urea and tryptophan with results evaluated through comparison with higher level electronic structure (PBE) using an identical basis set (6-31G*). Notably, MACE results differed substantially from the average of the independent computational agent outputs with molecular orbitals for the HOMO and LUMO, with MACE yielding improved agreement with higher-level PBE calculations. These results suggest potential promise for the use of MACE to improve accuracy of low-level electronic structure calculations through integration of multiple parallel methods.
9:30COIMG-167
Instrumentation and software development for parts-per-million characterization of pharmaceutical crystal forms using AF-PTIR microscopy, Aleksandr Razumtcev, Minghe Li, and Garth Simpson, Purdue University (United States) [view abstract]
Instrumentation and algorithms to perform label-free chemically-specific imaging of an active pharmaceutical ingredient (API) crystal forms using autofluorescence-detected photothermal mid-IR (AF-PTIR) microscopy are demonstrated. Limits of detection (LoDs) of the most common analytical techniques that enable bulk analysis of crystal polymorphism are insufficient for the detection of trace polymorph impurities. Chemically-specific imaging methods, such as Raman microscopy, enable lowering the LoDs by orders of magnitude. Herein, a complementary label-free technique based on fluorescence detection of mid-IR photothermal effect is shown to achieve parts per million detection limits in discrimination between two crystal forms of indomethacin API. Crystal form assignment confidence was improved by implementing a spectral masking approach designed for a random-access quantum cascade laser (QCL) array. Furthermore, algorithm development for per-particle-based segmentation of particles within a field-of-view and phase-sensitive signal detection is discussed.
9:50COIMG-168
Multivariate curve resolution with autoencoders for CARS microspectroscopy, Damien Boildieu1,2, David Helbert2, Amandine Magnaudeix3, Philippe Leproux1, and Philippe Carré2; 1XLIM, UMR CNRS 7252, University of Limoges, 2XLIM, UMR CNRS 7252, University of Poitiers, and 3IRCER, UMR CNRS 7315, University of Limoges (France) [view abstract]
Coherent anti-Stokes Raman scattering (CARS) microspectroscopy is a powerful tool for label-free cell imaging thanks to its ability to acquire a rich amount of information. An important family of operations applied to such data is multivariate curve resolution (MCR). It aims to find main components of a dataset and compute their spectra and concentrations in each pixel. Recently, autoencoders began to be studied to accomplish MCR with dense and convolutional models. However, many questions, like the results variability or the reconstruction metric, remain open and applications are limited to hyperspectral imaging. In this article, we present a nonlinear convolutional encoder combined with a linear decoder to apply MCR to CARS microspectroscopy. We conclude with a study of the result variability induced by the encoder initialization.
10:20 – 10:50 AM Coffee Break
Computational Imaging Topics (R2)
Session Chair:
Charles Bouman, Purdue University (United States)
10:50 AM – 12:30 PM
Market Street
10:50COIMG-169
BowTie Rasterization for extreme synthetic radiance image rendering, Thomas L. Burnett, Justin Halter, and Justin Jensen, FoVI3D (United States) [view abstract]
The light-field display (LfD) radiance image is a raster description of a light-field where every pixel in the image represents a unique ray within a 3D volume. The LfD radiance image can be projected through an array of micro-lenses to project a perspective-correct 3D aerial image visible for all viewers within the LfDs projection frustum. The synthetic LfD radiance image is comparable to the radiance image as captured by a plenoptic/light-field camera but is rendered from a 3D model or scene. Synthetic radiance image rasterization is an example of extreme multi-view rendering as the 3D scene must be rendered from many (1,000s to millions) viewpoints into small viewports per update of the light-field display. However, GPUs and their accompanying APIs (OpenGL, DirectX, Vulkan) generally expect to render a 3D scene from one viewpoint to a single large viewport/framebuffer. Therefore, LfD radiance image rendering is extremely time consuming and compute intensive. This paper reviews the novel, full-parallax, BowTie Radiance Image Rasterization algorithm which can be embedded within an LfD to accelerate light-field radiance image rendering for real-time update.
11:10COIMG-170
Automatic parameter tuning for plug-and-play algorithms using generalized cross validation and Stein's unbiased risk estimation for linear inverse problems in computational imaging, Canberk Ekmekci and Mujdat Cetin, University of Rochester (United States) [view abstract]
We propose an automatic parameter tuning method for Plug-and-Play (PnP) algorithms that use CNN denoisers. We focus on linear inverse problems and propose an iterative algorithm to calculate generalized cross-validation (GCV) and Stein's unbiased risk estimator (SURE) functions for a half-quadratic splitting based PnP algorithm that uses a state-of-the-art CNN denoiser. The proposed method leverages automatic differentiation to compute exact Jacobian-vector multiplications appearing in the update equations. The parameters can be tuned automatically by minimizing the GCV and SURE functions. Because linear inverse problems appear frequently in computational imaging, the proposed method can be applied to various problems. Furthermore, because the proposed method relies on GCV and SURE functions, it does not require access to the ground truth image and does not require collecting a training dataset, which is highly desirable for imaging applications for which acquiring data is costly and time-consuming. We evaluate the performance of the proposed parameter tuning strategy on a preliminary deblurring experiment and show that the reconstruction performance of the proposed method is comparable to that of the optimal tuning algorithm that adjusts the parameters by maximizing the structural similarity index between the ground truth image and the output of the PnP algorithm.
11:30COIMG-171
Ultrasound elasticity reconstruction with inaccurate forward model using integrated data-driven correction of data fidelity gradient, Narges Mohammadi, Marvin M. Doyley, and Mujdat Cetin, University of Rochester (United States) [view abstract]
Ultrasound elasticity images, which enable the visualization of quantitative maps of tissue stiffness, can be reconstructed by solving an inverse problem. Classical model-based approaches for ultrasound elastography use deterministic finite element methods (FEMs) to incorporate the governing physical laws leading to poor performance in low SNR conditions. Moreover, these approaches utilize approximate linear forward models discretized by FEMs to describe the underlying physics governed by partial differential equations (PDEs). To achieve highly accurate stiffness images, it is essential to compensate the error induced by noisy measurements and inaccurate forward models. In this regard, we propose a joint model-based and learning-based framework for estimating the elasticity distribution by solving a regularized optimization problem. To address noise, we introduce a statistical representation of the imaging system, which incorporates the noise statistics as a signal-dependent correlated noise model. Moreover, in order to compensate for the model errors, we introduce an explicit data-driven correction model, which can be integrated with any regularization term. This constrained optimization problem is solved using fixed-point gradient descent where the analytical gradient of the inaccurate data-fidelity term is corrected using a neural network, while regularization is achieved by data-driven unrolled regularization by denoising (RED). Both networks are jointly trained in an end-to-end manner.
11:50COIMG-172
A globally optimal fast-iterative linear maximum likelihood classifier, Prasanna Reddy Pulakurthi1, Sohail A. Dianat1, Majid Rabbani1, Suya You2, and Raghuveer M. Rao2; 1Rochester Institute of Technology and 2DEVCOM Army Research Laboratory (United States) [view abstract]
A novel iterative linear classification algorithm is developed from a maximum likelihood (ML) linear classifier. The main contribution of this paper is the discovery that a well-known maximum likelihood linear classifier is the solution to a contraction mapping for an acceptable range of values of the relaxation parameter. Hence, a novel iterative scheme is proposed that converges to a fixed point, the globally optimum solution. To the best of our knowledge, this formulation has not been discovered before. Furthermore, the proposed iterative solution converges to a fixed point at a rate faster than the traditional gradient-based techniques. Further variations of the iterative algorithm are presented to improve the constraint on the relaxation parameter. Finally, the performance of the proposed iterative solution is compared to conventional gradient descent methods on linear and non-linearly separable data in terms of both convergence speed and overall classification performance.
12:10COIMG-173
Multimodal contrastive learning for unsupervised video representation learning, Anup Hiremath and Avideh Zakhor, University of California, Berkeley (United States) [view abstract]
In this paper, we propose a multimodal unsupervised video learning algorithm designed to incorporate information from any number of modalities present in the data. We cooperatively train a network corresponding to each modality: at each stage of training, one of these networks is selected to be trained using the output of the other networks. To verify our algorithm, we train a model using RGB, optical flow, and audio. We then evaluate the effectiveness of our unsupervised learning model by performing action classification and nearest neighbor retrieval on a supervised dataset. We compare this triple modality model to contrastive learning models using one or two modalities, and find that using all three modalities in tandem provides a 1.5% improvement in UCF101 classification accuracy, a 1.4% improvement in R@1 retrieval recall, a 3.5% improvement in R@5 retrieval recall, and a 2.4% improvement in R@10 retrieval recall as compared to using only RGB and optical flow, demonstrating the merit of utilizing as many modalities as possible in a cooperative learning model.
12:30 – 2:00 PM Lunch
Computational Imaging Topics (R3)
Session Chair:
Charles Bouman, Purdue University (United States)
2:00 – 2:40 PM
Market Street
2:00COIMG-174
Hyperspectral learning for mHealth hemodynamic imaging, Yuhyun Ji, Sang Mok Park, Vidhya V. Nair, Yunjie Tong, and Young L. Kim, Purdue University (United States) [view abstract]
Hyperspectral imaging offers both spectral and spatial information, which is advantageous for understanding biological or physical signals. However, conventional hyperspectral imaging systems have intrinsic limitations including bulky instrument, high-cost optical components, slow data acquisition, and insufficient spectral resolution. Spectral learning can potentially overcome the limitations of conventional hyperspectral imaging systems. Spectral learning is a mapping from a sparse spectral space (RGB values) to a dense spectral space. However, hyperspectral image recovery from an RGB image is an ill-posed problem and requires a large amount of training data that hamper practical applications. Here we introduce a machine learning approach informed by locally sampled hyperspectral data to train a hyperspectral learning algorithm. The trained hyperspectral learning algorithm can recover a hypercube from an RGB image without complete hyperspectral scanning of the entire area. We apply learning-based hyperspectral imaging to vascular developmental models and peripheral microcirculation in a human to analyze and visualize dynamic changes in oxygen saturation and hemoglobin content. The key advantages of learning-based hyperspectral imaging include the hardware simplicity of using conventional cameras, high temporal resolution when a video is used, no tradeoff between spatial and spectral resolutions, and rich spectral information for a variety of machine learning methods.
2:20COIMG-175
Deep learning based image registration for 3D magnetic imaging at nanoscale, Srutarshi Banerjee, Junjing Deng, Joerg Strempfer, and Doga Gursoy, Argonne National Laboratory Advanced Photon Source (United States) [view abstract]
The availability of highly coherent X-ray flux with enhanced control of X-ray polarization from synchrotron facilities such as the Advanced Photon Source at Argonne National Laboratory provides basic resources for dichroic ptycho-tomographic based imaging of magnetic and electronic domains of magnetic materials at the nanoscale. As the resolution is pushed toward the nanoscale, imperfections associated with the stability of the beam and setup lead to jitter in the acquired data during ptycho-tomographic scans. These imperfections may arise from the relative motion between beam, sample and detector in the interferometric hardware setup. Since the dichroic ptycho-tomographic scans must be performed for many rotation angles and at least two tilted rotation axes, precise registration of the acquired data between projections is also critical. Without a proper image registration process, the quality of 3D reconstruction will be inaccurate. In this work, we present a deep learning based image registration process for addressing the misalignments in the experimental setup. We follow an approach of non-rigid registration in the deep learning framework. The algorithm is developed using synthetic and experimental data. Our approach allows a fast and effective model for image registration and is readily applicable to numerous other applications.