Computational Imaging XX

Monday 17 January 2022

08:10 – 08:40 EI 2022 Welcome Reception

Topics in Coherent Sensing

Session Chair: Andre Van Rynbach, U.S. Air Force (United States)
08:40 – 09:45
Blue Room

08:40
Conference Introduction

08:45COIMG-102
Coherent sensing performance using photon-counting arrays (Invited) [PRESENTATION-ONLY], Maureen E. Szymanski^1,2, Edward A. Watson^2,3, and David J. Rabb^1,2; ¹US Air Force Research Laboratory, ²University of Dayton, and ³Vista Applied Optics, LLC (United States) [view abstract]

Photon counting Geiger mode avalanche photodiode (GMAPD) arrays are typically used for high resolution 3D ranging. However, these detectors are also desirable for coherent sensing due to their high gain and bandwidth. This study explores the performance of a frequency modulated continuous wave (FMCW) heterodyne lidar system employing a GMAPD detector. The carrier-to-noise ratio and data rates of these unique systems are analyzed to demonstrate the tradeoffs between GMAPD readout architectures and other detector parameters. Ultimately, it is shown that the sparse coherent signal can be extracted from GMAPD outputs, using much less recorded data than a traditional coherent detector.

 

09:00COIMG-103
Adaptive deep learning for coherent imaging in scattering media (Invited) [PRESENTATION-ONLY], Lei Tian, Boston University (United States) [view abstract]

Deep learning has been broadly applied to coherent imaging in scattering applications. In this talk, I will discuss a new adaptive deep learning approach that can remove scattering artifacts across a continuum of scattering conditions regardless of whether the condition has been used for the training. Our dynamic synthesis framework may open a new paradigm for designing highly adaptive deep learning and computational imaging techniques.

 

09:15COIMG-104
Quantum limits for coherent-field estimation (Invited) [PRESENTATION-ONLY], Timothy J. Schulz¹ and David J. Brady²; ¹Michigan Technological University and ²The University of Arizona (United States) [view abstract]

In this presentation, we present the smallest lower bounds for parameter estimation from coherent-field data along with conditions for a measurement-system to induce those bounds, and we show how to use those bounds to infer an optimal measurement system. As examples, we discuss problems in source localization, source separation, and amplitude- and phase-estimation for arbitrary fields.

 

09:30COIMG-105
Compressive single shot synthetic aperture imaging (Invited) [PRESENTATION-ONLY], David J. Brady¹, Timothy J. Schulz², Chengyu Wang¹, and Minghao Hu¹; ¹The University of Arizona and ²Michigan Technological University (United States) [view abstract]

We use convolutional neural networks to synthesize aperture with arrays cameras observing coherently illuminated targets. We describe experimental demonstrations of >8x cross range resolution improvement relative to a single subaperture. We simulate diverse multiscale and quasi random sampling strategies to explore system optimization and we discuss coherence limitations and requirements. We also consider extensions to range-resolved imaging.

 

Recent Advances in Scientific CT I

Session Chairs: Doga Gursoy, Argonne National Laboratory Advanced Photon Source (United States); K. Aditya Mohan, Lawrence Livermore National Laboratory (United States); and Singanallur Venkatakrishnan, Oak Ridge National Laboratory (United States)
15:00 – 16:00
Blue Room

15:00COIMG-119
Weighted minimum norm algorithm for improved phase unwrapping, Tegan F. Lakshmanan¹, Kyle M. Champley², and K. Aditya Mohan²; ¹University of California, Berkeley and ²Lawrence Livermore National Lab (United States) [view abstract]

Since phase unwrapping is an integral part of multiple imaging techniques, a wide range of algorithms have been created over the years to unwrap phases. One such algorithm is the minimum norm phase unwrapping algorithm. This algorithm transforms the phase unwrapping problem into a minimization problem of a certain functional, which it solves through an iterative method. However, the problem is usually not convex, and when there are many structured discontinuities (lines and other artifacts with structure, not random noise) in the data to be unwrapped, the algorithm often produces a local minimum with new discontinuities in originally smooth areas. To prioritize solutions which minimize the functional better in smooth areas, we partially mask data lying along discontinuities. In the functional which we are minimizing, we multiply all terms which involve a pixel lying along a discontinuity by some positive number α < 1. After tuning the choice of α, we obtain much improved results on images with several discontinuities, at the expense of slower speed (less than twice as slow) than the original algorithm.

 

15:15COIMG-120
Nondestructive characterization systems to defend the homeland, with emphasis on an x-ray CT system independent feature space (Invited) [PRESENTATION-ONLY], Harry E. Martz, Steven M. Glenn, Kyle M. Champley, Isaac Seetho, Jeff Kallman, and K. Aditya Mohan, Lawrence Livermore National Laboratory (United States) [view abstract]

Nondestructive characterization systems are used to screen mail, luggage, cargo, vehicles, and people for prohibited items at U.S. land, sea, and air ports. U.S. Customs and Border Protection (CBP) screens for prohibited items at land, sea, and air (inbound) ports, while the Transportation Security Administration (TSA) screens at air (outbound) ports. The priority mission of CBP is to detect and prevent terrorists and terrorist weapons from entering the United States while facilitating legitimate trade and travel. To address the terrorist threat and help secure the U.S. border, CBP employs a multi-layered enforcement strategy. Non-Intrusive Inspection (NII) technology enables CBP to detect contraband (e.g., narcotics and weapons) and materials that pose potential nuclear and radiological threats. Technologies deployed by CBP include large-scale x-ray and gamma-ray radiographic imaging systems, as well as a variety of fixed and portable passive radiation measurement technologies. The TSA’s mission is to protect the U.S.'s transportation systems to ensure freedom of movement for people and commerce at airports. TSA uses a layered approach to security and screens for weapons, explosives, incendiaries, and other prohibited items, such as blunts, sharps, and arc lighters, to keep them from entering airport sterile areas and aircraft. Technologies employed by TSA include 3D imaging systems (e.g., millimeter wave and x-ray CT) and trace detection systems. The presentation will include an overview of the threat vectors, prohibited items that need to be detected, and NDC systems that detect them. Emphasis will be placed on an analytical technique developed for dual-energy x-ray CT systems called System Independent (ρe, Ze) (SIRZ), where ρe is electron density and Ze is effective atomic number.

 

15:30COIMG-122
Denoising advanced x-ray tomography data using deep learning without high-quality reference data (Invited) [PRESENTATION-ONLY], Daniel Pelt, Leiden University (the Netherlands) [view abstract]

In many tomographic experiments, there are unavoidable experimental constraints that impose dose and time limits, introducing noise in the reconstructed images. Recently, deep learning has shown promising results for denoising tomographic images. However, most deep learning techniques require the acquisition of high-quality high-dose reference data for accurate training. In practice, acquiring such high-quality data might be impossible, expensive, and/or time-consuming. Therefore, the requirement for high-quality training data presents a major obstacle for the application of deep learning in real-world experiments. In this talk, we present a recently developed deep learning approach, the Noise2Inverse method, that can accurately learn to denoise tomographic data without needing high-quality reference data. We show that there is mathematical proof that, under some mild assumptions, the method converges towards a noise-free reconstruction of the object. Reconstruction results are compared with other state-of-the-art methods, showing that Noise2Inverse is able to achieve accurate reconstruction results similar to deep learning methods trained on high-quality data. Finally, we show how the method can be applied to various advanced X-ray tomographic experiments that are performed at synchrotron facilities, including dynamic micro-tomography and X-ray diffraction tomography.

 

Recent Advances in Scientific CT II

Session Chairs: Doga Gursoy, Argonne National Laboratory Advanced Photon Source (United States); K. Aditya Mohan, Lawrence Livermore National Laboratory (United States); and Singanallur Venkatakrishnan, Oak Ridge National Laboratory (United States)
16:15 – 17:15
Blue Room

16:15COIMG-127
Grazing-incidence diffraction tomography with synchrotron x-rays (Invited) [PRESENTATION-ONLY], Esther Tsai, Masafumi Fukuto, and Ruipeng Li, Brookhaven National Laboratory (United States) [view abstract]

Characterization of thin films is essential for evaluating material processing outcomes and efficiency as well as establishing structure–performance relationships for applications in, for example, semiconductors and photovoltaics. We introduce grazing-incidence diffraction tomography (GID tomography), a computational method based on synchrotron X-ray scattering that quantitatively determines the dimension and orientation of crystalline domains. There are no restrictions on the beam coherence, substrate type, or film thickness for this method.

 

16:30COIMG-128
Hyperspectral neutron CT with material decomposition (Invited) [PRESENTATION-ONLY], Thilo Balke¹, Alexander M. Long¹, Sven C. Vogel¹, Brendt Wohlberg¹, and Charles A. Bouman²; ¹Los Alamos National Laboratory and ²Purdue University (United States) [view abstract]

Time-of-flight (TOF) neutron imaging is an advanced neutron radiography technique capable of generating hyperspectral radiographic image sequences. In combination with uniquely characteristic isotopic neutron cross-section spectra, isotopic areal densities can be determined on a per-pixel basis, thus resulting in a set of areal density images for each isotope present in the sample. By preforming measurements over several rotational views, an isotope decomposed 3D computed tomography is possible. However, large amounts of counting and scatter noise and large data sets severely impede proper calculation of attenuation spectra and fast estimation of isotopic densities.In this work we present a technique that estimates the isotopic densities directly in the Poisson counting domain through a non-linear modeling of the system and thus it avoids the irreversible mapping to the linearized domain. This approach allows for a significant computation time improvement, from weeks to a few hours compared to existing neutron evaluation tools while significantly improving reconstruction quality.

 

16:45COIMG-129
A metal artifact reduction scheme for accurate iterative dual-energy CT algorithms, Tao Ge¹, Maria Medrano¹, Rui Liao¹, Jeffrey F. Williamson¹, David Politte¹, Bruce R. Whiting², and Joseph A. O’Sullivan¹; ¹Washington University in St. Louis and ²University of Pittsburgh (United States) [view abstract]

CT images have been used to generate radiation therapy treatment plans for more than two decades. Dual-energy CT (DECT) has shown high accuracy in estimating electronic density or proton stopping power maps used in treatment planning. However, the presence of metal implants introduces severe streaking artifacts in the reconstructed images, affecting the diagnostic accuracy and treatment performance. In order to reduce the metal artifacts in DECT, we introduce a metal artifact reduction scheme for iterative DECT algorithms. The corrupt data is substituted with an estimation in each iteration. We utilize normalized metal artifact reduction (NMAR) composed with image domain decomposition to initialize the algorithm and speed up the convergence. A fully 3D joint statistical DECT algorithm, dual-energy alternating minimization (DEAM), with the proposed scheme is tested on experimental and clinical helical data acquired on a Philips Big Bore scanner. We compared DEAM with the proposed method to the original DEAM and vendor reconstructions with and without O-MAR. The visualization and quantitative analysis show that DEAM with the proposed method has the best performance in reducing streaking artifacts caused by metallic objects.

 

17:00COIMG-130
Non-paraxial multiple scattering model for multiplexed intensity diffraction tomography [PRESENTATION-ONLY], Jiabei Zhu¹, Hao Wang¹, Alex Matlock², and Lei Tian¹; ¹Boston University and ²Massachusetts Institute of Technology (United States) [view abstract]

Intensity diffraction tomography (IDT) is a novel 3D microscopy technique that can provide quantitative and large field-of-view information of biological samples in a label-free manner. The original IDT requires hundreds of intensity measurements to reconstruct a single volume, which is not applicable to dynamic samples. Multiplexed IDT (mIDT) provides an alternative illumination strategy that can accelerate this procedure by more than 60 times. The existing reconstruction algorithm for mIDT is based on the first-Born approximation, which cannot handle strongly scattering cases. In this work, we developed a multiple-scattering model-based algorithm based on the split-step non-paraxial (SSNP) method for mIDT reconstruction. Our algorithm can provide high accuracy for strongly scattering samples and using high-NA illumination angles. We demonstrate our approach can successfully reconstruct mIDT measurements on biological samples in experiments.

 

Tuesday 18 January 2022

Photon-Limited Imaging I

Session Chairs: Stanley Chan, Purdue University (United States) and Keigo Hirakawa, University of Dayton (United States)
07:00 – 08:00
Blue Room

07:00COIMG-135
Photon-starving and high-dynamic-range imaging with photon-counting quanta image sensors (Invited) [PRESENTATION-ONLY], Jiaju Ma, GigaJot Technology (United States) [view abstract]

This talk explores the fundamentals of quanta image sensors and photon counting pixels, beginning with the background knowledge necessary to effectively apply these devices. The discussion will include the key noise sources in CMOS image sensors as well as the common strategies used to mitigate these sources. It will also introduce the latest implementation in Gigajot sensor products to achieve photon counting read noise and high dynamic range. The presentation will further compare the sensor performance, in terms of read noise, dark current, quantum efficiency, and dynamic range. Finally, the presentation will outline how photon counting detectors can benefit different applications, especially for photon-starving and high dynamic range imaging.

 

07:15COIMG-136
Photon-limited object detection for CMOS cameras and quanta image sensors (Invited) [PRESENTATION-ONLY], Stanley Chan¹, Chengxi Li¹, Xiangyu Qu¹, Abhiram Gnanasambandam¹, Omar Elgendy², and Jiaju Ma²; ¹Purdue University and ²GigaJot Technology (United States) [view abstract]

Robust object detection under photon-limited conditions is crucial for applications such as night vision, surveillance, and microscopy, where the number of photons per pixel is low due to a dark environment and/or a short integration time. While the mainstream “low-light” image enhancement methods have produced promising results that improve the image contrast between the foreground and background through advanced coloring techniques, the more challenging problem of mitigating the photon shot noise inherited from the random Poisson process remains open. In this talk, I will present a photon-limited object detection framework by adding two ideas to state-of-the-art object detectors: 1) a space-time non-local module that leverages the spatial-temporal information across an image sequence in the feature space, and 2) knowledge distillation in the form of student-teacher learning to improve the robustness of the detector's feature extractor against noise. I will demonstrate the performance of the proposed method in comparison with the state-of-the-art baselines. When integrated with the latest Quanta Image Sensors, the algorithm achieves more than 50% mean average precision at a photon level of 1 photon per pixel.

 

07:30COIMG-137
High dynamic range single photon LiDAR (Invited) [PRESENTATION-ONLY], Robert K. Henderson, University of Edinburgh (United Kingdom) [view abstract]

Electronic interfaces to single photon detectors report photon arrival times rather than integrated photon numbers captured as charge or voltage (as for photodiode-based imaging). Single photon detectors such as single photon avalanche diodes (SPADs) have been traditionally deployed for low light sensing in scientific, radiation or microscopy applications. In the last few years, there has been a huge upsurge of interest in the use of SPAD detectors for low cost, mass market LiDAR for artificial reality (AR/VR) or automated driving assistance (ADAS). In these applications the detectors must tolerate high levels of solar background flux requiring a high dynamic range in the time domain rather than the charge or voltage domain. This requires the entire signal chain of the SPAD dToF sensors to be reconsidered from a legacy of scientific TCSPC electronics exploiting the sparsity of photon arrivals in low light conditions. A variety of implementations towards high dynamic range has resulted in pixel combining techniques and TDC architectures. We compare these approaches and demonstrate a technique capable of extending the dynamic range of LiDAR systems providing improved resilience to background conditions. A LiDAR system emulator using a reconfigurable SPAD array and FPGA interface is used to compare these different techniques.

 

07:45COIMG-138
Log-simplex denoising for color images (Invited) [PRESENTATION-ONLY], Sarah Miller¹, Keigo Hirakawa¹, and Chen Zhang²; ¹University of Dayton and ²OmniVision Technologies, Inc. (United States) [view abstract]

We propose a novel technique to determine the noise-free color at each pixel by estimating the ratio of the red, green, and blue (RGB) pixel values from their noisy version. In order to model the spatial statistics of the proportion of primary colors such as RGB components known to correspond to human perception of color, we interpret the simplex representation of color as an Aitchison geometry. Specifically, we develop a minimum mean square error (MMSE) estimator of log-color pixel values in the wavelet representation, with Poisson as its pixel domain likelihood function. We contrast this to most existing denoising techniques that are predominantly designed for single-channel/greyscale images that are then applied to YCbCr channels independently without regard for the RGB proportionality. In the extremely low photon regime, we verify experimentally that the proposed method yields state-of-the-art color denoising performance.

 

Photon-Limited Imaging II

Session Chairs: Stanley Chan, Purdue University (United States) and Keigo Hirakawa, University of Dayton (United States)
08:30 – 09:30
Blue Room

08:30COIMG-149
Computational imaging, one photon at a time (Invited) [PRESENTATION-ONLY], Mohit Gupta, University of Wisconsin, Madison (United States) [view abstract]

Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting and time-tagging individual photons with picosecond precision. Despite (or perhaps, due to) these capabilities, SPADs are considered specialized devices suitable only for photon-starved scenarios, and restricted to a limited set of niche applications. This raises the following questions: Can SPADs operate not just in low light, but in bright scenes as well? Can SPADs be used not just with precisely controlled active light sources such as pulsed lasers, but under passive, uncontrolled illumination like cellphone or machine vision cameras? I will describe our recent work on designing computational imaging techniques that (a) enable single-photon sensors to operate across the entire gamut of imaging conditions including high-flux scenes, and (b) leverages SPADs as passive imaging devices for ultra-low light photography. The overall goal is to transform SPADs into all-weather, general-purpose sensors capable of both active and passive imaging, across photon-starved and photon-flooded environments.

 

08:45COIMG-150
From a handful of photons (Invited) [PRESENTATION-ONLY], Hamid Sheikh, Samsung Research America (United States) [view abstract]

When CMOS image sensors started their journey in consumer photography, they were notoriously noisy, and unfit for low light photography. Nobody could have predicted that in only about 15 years, their performance in low light smartphone photography would be truly remarkable. In this talk I will describe the journey of low light consumer photography from the perspectives of sensors, algorithms, and compute innovations, and why a quantum leap is still needed to make our photography experience better.

 

09:00COIMG-151
An improved image enhancement algorithm using a statistical model for pixel value error, Henry G. Dietz, University of Kentucky (United States) [view abstract]

When an image is captured using an electronic sensor, statistical variations introduced by photon shot and other noise introduce errors in the raw value reported for each pixel sample. Earlier work found that modest improvements in raw image data quality reliably could be obtained by using empirically-determined pixel value error bounds to constrain texture synthesis. However, the prototype software implementation, KREMY (KentuckY Raw Error Modeler, pronounced “creamy”), was not effective in processing very noisy images. In comparison, the current work has reimplemented KREMY to make it capable of credibly improving far noisier raw DNG images. The key is a new approach that uses a simpler, but statistical, model for pixel value errors rather than simple bounds constraints.

 

09:15COIMG-152
Blind estimation of noise level based on pixels values prediction, Mykola Ponomarenko¹, Oleksandr Miroshnichenko², Vladimir Lukin², and Karen Egiazarian¹; ¹Tampere University (Finland) and ²National Aerospace University (Ukraine) [view abstract]

Noise parameters estimation is needed for many tasks of digital image processing. Many efficient algorithms of noise variance estimation were proposed during last two decades. However, most of those estimators are efficient only for a specific kind of noise for which they were designed. For example, methods of estimation of variance of white additive Gaussian noise (AWGN) fail in the case of additive colored Gaussian noise (ACGN) or for noises with other distributions. In this paper a new fully blind method of noise level estimation is proposed. For a given image, a distorted image with a removed part of pixels (around 10%) is generated. Then an inpainting (or impulse noise removal) method is used to recover missed pixels values. The difference between true and recovered values is used for a robust estimation of noise level. The algorithm is applied for different image scales to estimate noise spectrum. In the paper we propose a convolutional neural network PIXPNet for effective prediction of values of missing pixels. A comparative analysis shows that the proposed PIXPNet provides smallest error of recovered pixels values among all existing methods. A good efficiency of usage of the proposed approach in both AWGN and spatially correlated noise suppression is demonstrated.

 

Computational Chemical Imaging I

Session Chairs: Ji-Xin Cheng, Boston University (United States) and Garth Simpson, Purdue University (United States)
10:00 – 11:00
Blue Room

10:00COIMG-163
Computational chemical imaging with deep UV microscopy [PRESENTATION-ONLY], Francisco E. Robles, Georgia Tech and Emory University (United States) [view abstract]

In this presentation, I will give an overview of our recent work on UV microscopy and show how parallel efforts on computational microscopy can enhance nearly every aspect of deep-UV microscopy, for example increasing imaging speeds, efficiency, enhancing information (molecular/chemical) content, and providing new capabilities that would otherwise be out or reach. Results of UV microscopy applied to hematology, histopathology, and basic cell characterization will be presented.

 

10:15COIMG-164
Advances in diagnostics with mid-IR photothermal spectroscopic imaging [PRESENTATION-ONLY], Chalapathi Cajjela, Rupali Mankar, Ragib Ishrak, Sharmin Afrose, Xinyu Wu, Camille Artur, David Mayerich, and Rohith Reddy, University of Houston (United States) [view abstract]

Vibrational spectroscopy using mid-infrared (IR) absorption as the contrast mechanism has been effective at biochemical characterization in a wide array of applications, including cell and tissue classification. Biomedical samples such as cancerous tissue are chemically heterogeneous, and bulk spectroscopy is inadequate to understand such samples. Mid-infrared spectroscopic imaging (MIRSI) combines the molecular specificity of vibrational spectroscopy with the spatial detail provided by microscopy. Fourier transform infrared (FT-IR) imaging, the first MIRSI technology, is successful at tissue analysis using thin, dry tissue sections mounted on IR transparent substrates. We will discuss the benefits and challenges of such biomedical tissue analysis with FTIR. We will also contrast FTIR with emerging discrete frequency infrared (DFIR) technologies such as photothermal IR imaging for improved disease diagnosis. The combination of machine learning and MIRSI has facilitated the identification of tissue sub-type and cancer grade in a label-free and quantitative manner. Ovarian cancer is one of the deadliest cancers among women in the U.S., with over 22,000 women diagnosed with the disease every year. Early diagnosis of the disease is essential for improving survival. We present data to demonstrate the utility of MIRSI followed by machine learning in automating ovarian cancer diagnosis. Furthermore, we utilize the higher resolution provided by new technical advances in MIRSI to reduce the overall data collection time for MIRSI tissue analysis. We also extend these analyses to cervical cancer tissue. Bone disorders such as osteosclerosis and collagen deposition have spectroscopic signatures that can be identified using are MIRSI. We utilize the dichroic behavior of bone tissue sub-types to identify collagen fibers in clinical samples. We present the first study to demonstrate the ability to spectroscopically identify thin collagen fibers (≈1µm diameter) and their orientations, which is critical for accurate grading of human bone marrow fibrosis.

 

10:30COIMG-165
Computational label-free microscopy [PRESENTATION-ONLY], Lei Tian, Boston University (United States) [view abstract]

I will discuss our efforts of pushing the limit of label-free tomography using different computational imaging strategies. I will discuss Intensity Diffraction Tomography (IDT) for quantitative 3D phase imaging. IDT can be easily implemented in a standard microscope equipped with a programmable light source and requires no exogeneous contrast agents, making it easily accessible to the biological research community. I will present both physical model and deep learning strategies for improving the imaging capabilities of IDT for handling complex 3D objects. Finally, I will highlight the synergy of combining IDT and mid-infrared photothermal imaging to enable computational chemical phase tomography.

 

Computational Chemical Imaging II

Session Chairs: Ji-Xin Cheng, Boston University (United States) and Garth Simpson, Purdue University (United States)
15:00 – 16:00
Blue Room

15:00COIMG-175
Deep learning stimulated Raman scattering microscopy [PRESENTATION-ONLY], Haonan Lin and Ji-Xin Cheng, Boston University (United States) [view abstract]

Hyperspectral spectroscopic stimulated Raman scattering (hSRS) is a label-free chemical imaging modality enabling visualization of molecules in living systems with high specificity. However, its speed remains insufficient for studying high-motility samples. We present two works towards pushing the speed by synergistic integration between advanced instrumentation and deep learning. In the first work, we use a 50-kHz polygon scanner as a delay line tuner to rapidly scan a Raman spectrum, achieving a speed of 0.8 seconds per spectroscopic image stack. We further train a spatial-spectral residual net which, after training, can improve the signal level for images scanned at high speeds. Collectively, our design can generate high-fidelity spectroscopic SRS images with two orders of magnitude faster. In the second work, we exploited the data redundancy of hSRS in the spectral domain, and train a DenseNet to map a single-color femtosecond SRS image into chemical maps of four subcellular organelles (lipid droplet, endoplasmic reticulum, nuclei, cytoplasm) in living mammalian cells. Lipid droplet dynamics and cellular response to dithiothreitol in live MIA PaCa-2 cells are demonstrated using this computationally multiplex method.

 

15:15COIMG-176
Spectral super-resolution using device-informed machine learning [PRESENTATION-ONLY], Yuhyun Ji, Sang Mok Park, Yunsang Kwak, and Young L. Kim, Purdue University (United States) [view abstract]

Super-resolution means high-resolution reconstruction of digital images acquired with low-resolution systems. Our focus has been extending this concept to the frequency domain for spectroscopy and hyperspectral imaging. From a machine learning perspective, it is possible to recover hyperspectral or multispectral data from RGB images taken by a conventional camera (three-color sensors). Solving this ill-posed problem is often prohibitively expensive, requiring a large amount of training data. From a device perspective, almost all spectroscopy and hyperspectral systems rely heavily on complex and costly optical instrumentation. To overcome these limitations and improve the performance of learning algorithms without relying on data only, we introduce device-informed machine learning. In particular, we hybridize measurement devices (spectrometer and camera) and machine learning to obtain prior information and reduce an amount of training data. As an example, we apply this method to realize spectral super-resolution for noninvasive assessment of blood hemoglobin levels from RGB images of the eyelids (i.e. conjunctiva) using smartphones. This mHealth application can also serve as an example that device-informed statistical and deep learning can minimize hardware complexity, offer interpretable deep learning, and strengthen learning performance.

 

15:30COIMG-177
Multi-agent consensus equlibrium (MACE) for improving chemical structure determination [PRESENTATION-ONLY], Jiayue Rong, Garth Simpson, Gregery T. Buzzard, Lyudmila Slipchenko, and Charles A. Bouman, Purdue University (United States) [view abstract]

MACE is demonstrated for the integration of experimental observables as constraints in molecular structure determination and for the systematic merging of multiple computational architectures. MACE is founded on simultaneously determining the equilibrium point between multiple experimental and/or computational agents; the returned state description (e.g., atomic coordinates for molecular structure) represents the intersection of each manifold and is not equivalent to the average optimum state for each agent. The moment of inertia, determined directly from microwave spectroscopy measurements, serves to illustrate the mechanism through which MACE evaluations merge experimental and quantum chemical modeling. MACE results reported combine gradient descent optimization of each ab initio agent with an agent that predicts chemical structure based on root-mean-square deviation of the predicted inertia tensor with experimentally measured moments of inertia. Successful model fusion for several small molecules was achieved as well as the larger molecule solketal. Fusing a model of moment of inertia, an underdetermined predictor of structure, with low cost computational methods yielded structure determination performance comparable to standard computational methods such as MP2/cc-pVTZ, and greater agreement with experimental observables.

 

15:45COIMG-178
Multi-agent consensus equilibrium (MACE) in molecular spectral analysis [PRESENTATION-ONLY], Ziyi Cao, James Ulcickas, Charles A. Bouman, Lyudmila Slipchenko, Gregery T. Buzzard, and Garth Simpson, Purdue University (United States) [view abstract]

Multi-agent consensus equilibrium (MACE) algorithm has been successful inanalyzing computed tomographic (CT) on distributed systems. We recently discovered multi-agent consensus equilibrium (MACE) can also be demonstrated for chemical structure determination, as well as dimension reduction analysis. With respect to chemical structure determination, we discovered MACE can be used in integrating experimental observables as constraints in molecular structure determination and for the systematic merging of multiple computational architectures. As for dimension reduction analysis on vibrational spectra, we implemented MACE to combine different dimension reduction method (e.g. PCA,PCALDA,GALDA), and found out MACE can systematically merge multiple dimension reduction methods for vibrational spectra analysis.

 

Methods in Computational Imaging I

Session Chairs: Charles Bouman, Purdue University (United States) and Gregery Buzzard, Purdue University (United States)
16:15 – 17:15
Blue Room

16:15COIMG-179
Drone object detection using RGB/IR fusion, Lizhi Yang, Ruhang Ma, and Avideh Zakhor, University of California, Berkeley (United States) [view abstract]

Object detection using aerial drone imagery has received a great deal of attention in recent years. While visible light images are adequate in detecting objects in most scenarios, using thermal cameras, can extend the capabilities of object detection at nighttime or when objects of interest are occluded. As such, RGB and Infrared (IR) fusion methods for object detection are useful and important. One of the biggest challenges in applying deep learning methods to RGB/IR object detection is lack of available training data for drone IR imagery, especially at night. In this paper, we develop several strategies for creating synthetic IR images from RGB using AirSim simulation engine and CycleGAN. Furthermore, we develop an illumination aware fusion framework for fusing RGB and IR images to detect cars and people on the ground. We characterize and test our methods in both simulated and actual data. Our solution is implemented on an NVIDIA Jetson Xavier running on an actual drone, requiring about 28 milliseconds of processing per RGB/IR image pair.

 

16:30COIMG-416
Visual vibration tomography: Estimating interior material properties from monocular video [PRESENTATION-ONLY], Berthy Feng, Alexander Ogren, Chiara Darario, and Katherine L. Bouman, California Institute of Technology (United States) [view abstract]

An object's interior material properties, while invisible to the human eye, determine motion observed on its surface. We propose an approach that estimates heterogeneous material properties of an object directly from a monocular video of its surface vibrations. Specifically, we estimate Young's modulus and density throughout a 3D object with known geometry. Knowledge of how these values change across the object is useful for characterizing defects and simulating how the object will interact with different environments. Traditional non-destructive testing approaches, which generally estimate homogenized material properties or the presence of defects, are expensive and use specialized instruments. We propose an approach that leverages monocular video to (1) measure and object's sub-pixel motion and decompose this motion into image-space modes, and (2) directly infer spatially-varying Young's modulus and density values from the observed image-space modes. On both simulated and real videos, we demonstrate that our approach is able to image material properties simply by analyzing surface motion. In particular, our method allows us to identify unseen defects on a 2D drum head from real, high-speed video.

 

16:45COIMG-417
Deep radio interferometric imaging with POLISH: DSA-2000 and weak lensing [PRESENTATION-ONLY], Liam Connor, Gregg W. Hallinan, Vikram Ravi, and Katherine L. Bouman, California Institute of Technology (United States) [view abstract]

Radio interferometry allows astronomers to probe small spatial scales that are often inaccessible with single-dish instruments. However, recovering the radio sky from an interferometer is an ill-posed deconvolution problem that astronomers have worked on for half a century. More challenging still is achieving resolution below the array's diffraction limit, known as super-resolution imaging. To this end, we have developed a new learning-based approach for radio interferometric imaging, leveraging recent advances in the computer vision problems deconvolution and single-image super-resolution (SISR). We have developed and trained a high dynamic range residual neural network to learn the mapping between the dirty image and the true radio sky. We call this procedure POLISH, in contrast to the traditional CLEAN algorithm. The feed forward nature of learning-based approaches like POLISH is critical for analyzing data from the upcoming Deep Synoptic Array (DSA-2000). We show that POLISH achieves super-resolution, and we demonstrate its ability to deconvolve real observations from the Very Large Array (VLA). Super-resolution on DSA-2000 will allow us to measure the shapes and orientations of several hundred million star forming radio galaxies (SFGs), making it a powerful cosmological weak lensing survey and probe of dark energy. We forecast its ability to constrain the lensing power spectrum, finding that it will be complementary to next-generation optical surveys such as Euclid.

 

17:00COIMG-418
End-to-end sequential sampling and reconstruction for MR imaging [PRESENTATION-ONLY], Zihui Wu¹, Tianwei Yin², He Sun¹, Adrian Dalca³, Yisong Yue¹, and Katherine L. Bouman¹; ¹California Institute of Technology, ²The University of Texas at Austin, and ³Harvard Medical School (United States) [view abstract]

Accelerated MRI shortens acquisition time by subsampling in the measurement k-space. Recovering a high-fidelity anatomical image from subsampled measurements requires close cooperation between two components: (1) a sampler that chooses the subsampling pattern and (2) a reconstructor that recovers images from incomplete measurements. In this paper, we leverage the sequential nature of MRI measurements, and propose a fully differentiable framework that jointly learns a sequential sampling policy simultaneously with a reconstruction strategy. This co-designed framework is able to adapt during acquisition in order to capture the most informative measurements for a particular target. Experimental results on the fastMRI knee dataset demonstrate that the proposed approach successfully utilizes intermediate information during the sampling process to boost reconstruction performance. In particular, our proposed method outperforms the current state-of-the-art learned k-space sampling baseline on over 96% of test samples. We also investigate the individual and collective benefits of the sequential sampling and co-design strategies.

 

Wednesday 19 January 2022

Computational Imaging XX Posters

08:20 – 09:20
EI Symposium

Poster interactive session for all conferences authors and attendees.

COIMG-185
P-02: Improvement of aerial image by simulations, Katsunari Ashimine¹, Munemitsu Abe¹, and Kazuhiro Wako²; ¹Alps Alpine Co., Ltd. and ²National Institute of Technology, Sendai College (Japan) [view abstract]

We measured the contrast of standard charts using two different types of retro-reflectors in an AIRR (Aerial imaging by retro-reflection) system, and examined the results to be reproduced by optical simulation. As a result, it became possible to reproduce the effect of retro-reflector diffraction on the Aerial image quality of the AIRR system by optical simulation.

 

Neural Networks for Computational Imaging I

Session Chairs: Charles Bouman, Purdue University (United States) and Gregery Buzzard, Purdue University (United States)
09:30 – 10:30
Blue Room

09:30COIMG-217
Image denoising with control over deep network hallucination, Qiyuan Liang, Florian Cassayre, Haley Owsianko, Majed El Helou, and Sabine Süsstrunk, École Polytechnique Fédérale de Lausanne (EPFL) (Switzerland) [view abstract]

Deep image denoisers achieve state-of-the-art results but with a hidden and hazardous cost. As witnessed in recent literature, these deep networks are capable of overfitting their training distribution, causing inaccurate hallucinations and generalizing poorly to varying data. For better control and interpretability over a pretrained denoiser, we propose a novel framework surrounding a deep denoising network. We call it controllable confidence-based image denoising (CCID). In this framework, we exploit the outputs of deep denoising networks alongside an image processed with a simple reliable filter. We propose to fuse the two components with a frequency-domain approach that takes into account the reliability of deep network outputs. With our framework, the user can control the fusion of the two components, which is carried out smoothly in the frequency domain. We also provide a map estimating the spatial confidence of the hallucination output, as this is more user-friendly than the frequency domain. Results show that our CCID not only provides more interpretability and control, but can even outperform both the quantitative performance of the deep denoiser and that of the reliable filter. We show deep network hallucination is good ``at home'', when test data are similar to training data, but is otherwise detrimental.

 

09:45COIMG-218
FiveNet: Joint image demosaicing, denoising, deblurring, super-resolution, and clarity enhancement, Mykola Ponomarenko¹, Vladimir Marchuk², and Karen Egiazarian¹; ¹Tampere University (Finland) and ²Don State Technical University (Russian Federation) [view abstract]

In this paper, a convolutional neural network for joint image demosaicing, denoising, deblurring, super-resolution and clarity enhancement is proposed. The network inputs are four-channel Bayer CFA image (R, G, G, B) and three channels of the same size containing distortions maps (noise level map, blur level map, clarity degradation map). It is shown that the designed network FiveNet can effectively process images with the mix of five different distortions. It is also demonstrated that adding clarity enhancement into the processing chain can additionally increase image quality by up to 3-4 dB in PSNR. A small dataset ClarityDegr120 of color images with different clarity degradations and enhancements is designed using images processed by FiveNet. Mean opinion scores (MOS) for the test set are collected. The MOS prove that clarity enhancement can significantly increase image visual quality. A comparative analysis using the MOS demonstrates a low correspondence between image quality metrics and human perception for the clarity enhancement task.

 

10:00COIMG-219
Transfer learning for no-reference image quality metrics using large temporary image sets, Sheyda Ghanbaralizadeh Bahnemiri, Mykola Ponomarenko, and Karen Egiazarian, Tampere University (Finland) [view abstract]

One of the main problems of design of neural network based no-reference metrics for image visual quality assessment is small size of image databases with mean opinion scores (MOS). For large networks which can memorize key features of several thousands of images, usage of the databases for metrics training leads to overlearning. Since data augmentation for image quality assessment is limited by horizontal image flipping only, the main way to decrease overlearning is usage of transfer learning which can significantly speed up training process. In the paper we propose new technique of transfer learning between networks of different architectures using a large set of images without MOS. We implemented the technique for transfer learning between pre-trained KonCept512 metric and new IMQNet which architecture is proposed in the paper. Effectiveness of the transfer learning is estimated in a numerical analysis. It is shown that the trained IMQNet metric provides significantly better correlation with KonCept512 metric (0.89) than other modern metrics. It is also shown that IMQNet pre-trained by the proposed transfer learning shows better correlation with MOS of KonIQ-10k database (0.86) than IMQNet pre-trained using directly the MOS of KonIQ10k (0.73).

 

10:15COIMG-220
Recognition aware learned image compression (Invited), Maxime Kawawa-Beaudan, Ryan Roggenkemper, and Avideh Zakhor, University of California, Berkeley (United States) [view abstract]

Learned image compression methods generally optimize a rate-distortion loss, trading off improvements in visual distortion for added bitrate. Increasingly, however, compressed imagery is used as an input to deep learning networks for various tasks such as classification, object detection, and superresolution. We propose a recognition-aware learned compression method, which optimizes a rate-distortion loss alongside a task-specific loss, jointly learning compression and recognition networks. We augment a hierarchical autoencoder-based compression network with an EfficientNet recognition model and use two hyperparameters to trade off between distortion, bitrate, and recognition performance. We characterize the classification accuracy of our proposed method as a function of bitrate and find that for low bitrates our method achieves as much as 26% higher recognition accuracy at equivalent bitrates compared to traditional methods such as Better Portable Graphics (BPG). We present a joint approach to learned compression and recognition, training state-of-the-art models end-to-end to encourage the learning of complementary features. We demonstrate greater recognition accuracy results to those achieved by traditional methods like BPG, at equivalent bitrates.

 

Neural Networks for Computational Imaging II

Session Chairs: Charles Bouman, Purdue University (United States) and Gregery Buzzard, Purdue University (United States)
10:50 – 11:50
Blue Room

10:50COIMG-226
Deep generative priors for imaging ptycho-tomography [PRESENTATION-ONLY], Selin Aslan¹, Viktor Nikitin², Zhengchun Liu², Tekin Bicer², Sven Leyffer², and Doga Gursoy²; ¹Virginia Tech and ²Argonne National Laboratory (United States) [view abstract]

Joint ptycho-tomography is a powerful computational imaging framework that avoids the limitations introduced by conventional phase retrieval such as high probe overlap. For the solution of the problem, we use an alternating direction method of multipliers (ADMM) where we split the joint problem into three parts: ptychography, tomography, and a learned denoiser. Then, the PnP framework is proposed as a flexible way to add the state-of-the-art denoisers to the ADMM. While PnP has shown great performance in many inverse problems, these denoisers are not effective because of the different noise characteristics in the joint ptycho-tomography reconstructions. To address the unique challenges of solving the ptycho-tomography problem, we adopted a Poisson process to accurately model our measurements, and further improve reconstruction quality with deep generative models as priors. In our simulations, we demonstrate that our proposed framework with parameter tuning and learned priors generates high-quality reconstructions under limited and noisy measurement data.

 

11:05COIMG-227
Fully RNN for knee ligament tear classification and localization in MRI scans, Kaiyue Zhu¹, Ying Chen¹, Xu Ouyang¹, Gregory White², and Gady Agam¹; ¹Illinois Institute of Technology and ²Rush Medical College (United States) [view abstract]

Diagnosing ligament injuries using MRI scans is a labor-intensive task that requires an expert. In this paper, we propose a fully recurrent neural network (RNN) for detecting Anterior Cruciate Ligament (ACL) tears using MRI scans. The proposed network localizes the ACL and classifies it into several categories: ACL tear, normal tear, and healthy. Existing detection methods use deep learning networks based on single MRI sections, and in this way lose 3D spatial context. To address this, we propose a fully recurrent neural network that processes a sequence of 3D sections and so captures 3D spatial context. The proposed network is based on a YOLOv3 backbone and can produce a sequence of decisions which are then combined by majority voting. Experimental results show improvement over state-of-the-art methods.

 

11:20COIMG-228
Correction filter for single image super-resolution: Robustifying off-the-shelf deep super-resolvers [PRESENTATION-ONLY], Shady Abu-Hussein, Tel Aviv University (Israel) [view abstract]

The single image super-resolution task is one of the most examined inverse problems in the past decade. In the recent years, Deep Neural Networks (DNNs) have shown superior performance over alternative methods when the acquisition process uses a fixed known downsampling kernel-typically a bicubic kernel. However, several recent works have shown that in practical scenarios, where the test data mismatch the training data (e.g. when the downsampling kernel is not the bicubic kernel or is not available at training), the leading DNN methods suffer from a huge performance drop. Inspired by the literature on generalized sampling, in this work we propose a method for improving the performance of DNNs that have been trained with a fixed kernel on observations acquired by other kernels. For a known kernel, we design a closed-form correction filter that modifies the low-resolution image to match one which is obtained by another kernel (e.g. bicubic), and thus improves the results of existing pre-trained DNNs. For an unknown kernel, we extend this idea and propose an algorithm for blind estimation of the required correction filter. We show that our approach outperforms other super-resolution methods, which are designed for general downsampling kernels.

 

11:35COIMG-229
Multiresolution DECOLOR for camouflaged moving foreground detection using a redundant wavelet transform, Zoe Fowler¹, James Fowler², and Agnieszka Miguel³; ¹Georgia Institute of Technology, ²Mississippi State University, and ³Seattle University (United States) [view abstract]

Detection of moving foreground objects is essential to many image-sequence-analysis applications. However, preexisting methods tend to work best when the foreground is visually distinct from the background, suffering when objects are camouflaged. To address this shortcoming, a foreground-extraction algorithm resilient to camouflage is proposed by incorporating a redundant discrete wavelet transform into the well-known DECOLOR technique based on a sparse and low rank model of the foreground-extraction problem. Detection of camouflaged moving objects is enhanced as a result of the combination of multiple background estimates in independent wavelet subbands into an overall estimate of the background, leveraging the known robustness of redundant wavelet transforms to additive noise. Experimental results demonstrate that the proposed method offers robustness to camouflage superior to that of other competing methods for image sequences containing snow leopards in the wild.

 

Methods in Computational Imaging II

Session Chairs: Charles Bouman, Purdue University (United States) and Gregery Buzzard, Purdue University (United States)
15:00 – 16:00
Blue Room

15:00COIMG-238
Structural biology by cryo-EM: From in vitro to in situ structures [PRESENTATION-ONLY], Wen Jiang, Purdue University (United States) [view abstract]

Single particle cryo-electron microscopy (cryo-EM) imaging and 3D reconstruction methods are revolutionizing structural biology and drug developments. While stable structures that can be expressed/purified in large quantities can be solved routinely to 2-4 Angstrom resolutions, the dynamic compositions and conformations of many protein complexes (“machines”) that are essential for their functions pose serious challenges to current sample preparation/characterization methods and image processing/3D reconstruction. To fully understand how these macromolecular “machines” function in situ in the cells, thin sections of cells or tissues need to be imaged and 3D reconstructed using cryo-correlative light microscopy (CLEM) and electron tomography (cryo-ET). However, accurate targeting of the protein complexes in the large volume of cells and the limited power of electrons penetrating the thick cells pose serious challenges in sample preparation, data collection, computational analysis, and computing resources. In the Jiang Lab we utilize single particle cryo-EM and cryo-ET to study structures of protein filaments implicated in neurodegenerative disease, human viruses, phages, macromolecular complexes involved in cancer, nanobiology technologies, and drug discovery. We also aim to broaden the applicability of high resolution (2-4 Å) cryo-EM to targets of low abundance or high level of dynamics. In this presentation, I will describe our efforts in developing new methods in sample preparation, image data collection, and image analysis and 3D reconstruction that target at different stages of the cryo-EM workflow. Several cryo-EM structures enabled by these methods will be also presented. I will end the presentation with my views on the next phase of technical advances needed for high throughput in situ structural biology using cryo-EM.

 

15:15COIMG-239
Chemically selective imaging by fluorescence-detected photothermal mid-infrared microscopy [PRESENTATION-ONLY], Aleksandr Razumtcev, Minghe Li, and Garth Simpson, Purdue University (United States) [view abstract]

The instrumentation and software to perform fluorescence-detected photothermal mid-infrared (F-PTIR) microscopy is demonstrated and applied to characterize the chemical composition within binary mixtures of powdered and amorphous materials. F-PTIR relies on the temperature dependence of the fluorescence quantum efficiency as a sensitive “thermometer” for transducing local photothermal heating from infrared (IR) light absorption. However, photothermal imaging of multicomponent systems, such as pharmaceutical formulations, is complicated by the simultaneous presence of multiple IR-radiation absorbers with different spectral characteristics. Therefore, the development of software and hardware to perform autonomous chemical identification and classification supports optimizing F-PTIR and similar micro-spectroscopy methods for the analysis of commercially-relevant real-world biological or pharmaceutical samples. The software to perform multicomponent F-PTIR analysis based on spectral masking and non-linear matrix factorization refinement is demonstrated experimentally. Following proof-of-concept measurements on model molecular assemblies, F-PTIR was used to address questions of composition within phase-separated domains in pharmaceutical formulations. F-PTIR enabled the study of chemical composition of the domains formed upon phase separation within model ritonavir/copovidone ASDs. The results of F-PTIR suggest the formation of drug-enriched domains, which might potentially help in determining bioavailability of future drug candidates.

 

15:30COIMG-240
A Good RAP: Converting between mismatched backprojector and modified prior model [PRESENTATION-ONLY], Gregery T. Buzzard¹, Emma J. Reid², and Charles A. Bouman¹; ¹Purdue University and ²Oak Ridge National Laboratory (United States) [view abstract]

In the Bayesian formulation of many inverse problems, a linear forward model maps a candidate reconstruction to simulated measurements, and the transpose map converts measurements back to the reconstruction domain. In some applications, either for computational efficiency or improved results, the transpose is replaced by an approximation to the transpose, which is known as a mismatched backprojector or a relaxed adjoint projector (RAP). We describe a theorem to the effect that under appropriate hypotheses, the use of RAP with a given prior produces the same solution as the use of the standard backprojector (the transpose) with a suitably modified prior, thus giving an intuitive interpretation to a fairly common practice.

 

15:45COIMG-241
Computational field sensor for aperture synthesis [PRESENTATION-ONLY], Casey J. Pellizzari¹, Timothy Bate¹, David Strong², and Mark Spencer³; ¹United States Air Force Academy, ²Strong Inc., and ³Air Force Research Laboratory (United States) [view abstract]

Aperture synthesis requires measuring the phase of incident light to stitch together high-resolution images. However, conventional imaging systems are unable to measure phase at optical wavelengths. Therefore, to measure phase, we must use either heterodyne detection or phase-retrieval algorithms. Heterodyne detection requires complex hardware that limits its practical use. Furthermore, conventional phase-retrieval algorithms are not robust to measurement noise and laser speckle that results from coherent illumination and rough surface scattering. In this work, we present an approach for sensing the amplitude and phase of incident light that uses simple hardware and that is robust to noise, including speckle. Our approach leverages a customizable sensor that combines phase modulation with multi-plane measurements (i.e. image and pupil plane measurements). We couple this hardware with an algorithm that is uniquely designed for the regularized inversion of speckled fields. We compare our approach to existing methods in both simulation and laboratory experiments.

 

Methods in Computational Imaging III

Session Chairs: Charles Bouman, Purdue University (United States) and Gregery Buzzard, Purdue University (United States)
16:15 – 17:15
Blue Room

16:15COIMG-247
OldVSR: A model for the video super-resolution and restoration of old real-world TV series, Tony Nokap Park and Taeyoung Na, SK Telecom (Republic of Korea) [view abstract]

With the recent advance in video restoration and super-resolution techniques, there have been many requests for restoring real-world old analog TV series into high-definition digital content. As excellent classical series may receive little to no attention due to their poor video quality, restoring them would open new business opportunities for reusing old content. A problem with restoring real-world old TV series is in the complex artifacts introduced by the old interlaced scanning and compression artifacts during the digitalization of old analog videos. Though recent restoration methods perform nicely on clean videos, due to the artificial nature of interlacing and compression artifacts, recent restoration methods fail to restore old videos into a high-definition counterpart free from unwanted artifacts. In this work, we propose OldVSR for restoring old real-world TV series with artifacts of diverse artificial nature. The proposed model implements a bidirectional recurrent structure with first and second-order propagation where each recurrent layer implements two main functions, i.e., Feature alignment (FA) and Pyramid feature aggregation (PFA). The outputs of the forward and backward layers are merged and upsampled to produce a High-Definition (HD) restoration of the input standard-definition (SD) frame. We demonstrate through experiments that our proposed OldVSR can effectively remove diverse artifacts of artificial nature from old videos and successfully restores old tv-series.

 

16:30COIMG-248
Accurate measurement of charge density in nanoscale particles using an aperture optimization of Fourier based phase reconstruction, Takuma Okada, Yoshihiro Midoh, Koji Nakamae, and Noriyuki Miura, Osaka University (Japan) [view abstract]

Electron holography is used to observe minute electromagnetic field distributions in electronic and magnetic materials. The signal-to-noise ratio of electron hologram decreases when the electron beam irradiation is reduced to avoid unnecessary charging and damage. Noise in the hologram causes phase errors. In order to obtain accurate phase information, we propose an aperture optimization of Fourier based phase reconstruction. Our method effectively separates the signal from the noise using an extended Fourier ring correlation. From the experimental results using a simulated electron hologram with low signal-to-noise, it was found that the estimation error in the amount of charge on the specimen was reduced to about half that of the conventional method.

 

16:45COIMG-249
Jeweler: A Python module for searching binary sequences optimized for coded aperture [PRESENTATION-ONLY], Daniel J. Ching, Argonne National Laboratory (United States) [view abstract]

Coded aperture is a data-collection strategy where intentionally modulating or controlling the light source with a specially designed sequence converts various ill-posed imaging problems into well-posed problems. Examples of these imaging inverse problems include motion-deblurring, lensless imaging, and depth-of-field recovery. Jeweler is a Python-implemented library of functions designed to assist in efficiently checking sequences for coded-aperture against common fitness criteria. In a novel approach, Jeweler exploits the combinatoric equivalence classes such as Lyndon words because codes within each of these equivalence classes share a frequency response. Thus, systematically checking these equivalence classes instead of randomly searching all sequences of fixed-content significantly reduces the search space of potential codes.

 

17:00COIMG-250
What is the cost of applying a constraint in least squares? [PRESENTATION-ONLY], Ramakrishna Kakarala and Jun Wei, Omnivision Technologies (United States) [view abstract]

Although the theory of constrained least squares (CLS) estimation is well known, it is usually applied with the view that the constraints to be imposed are unavoidable. However, there are cases in which constraints are optional. For example, in camera color calibration, one of several possible color processing systems is obtained if a constraint on the row sums of a desired color correction matrix is imposed; in this example, it is not clear a priori whether imposing the constraint leads to better system performance. In this paper, we derive an exact expression connecting the constraint to the increase in fitting error obtained from imposing it. As another contribution, we show how to determine projection matrices that separate the measured data into two components: the first component drives up the fitting error due to imposing a constraint, and the second component is unaffected by the constraint. We demonstrate the use of these results in the color calibration problem.

 

Thursday 20 January 2022

Autonomous Science

Session Chairs: Doga Gursoy, Argonne National Laboratory Advanced Photon Source (United States) and Benji Maruyama, Air Force Research Laboratory (United States)
10:00 – 11:00
Blue Room

10:00COIMG-280
Physical discovery in automated scanning probe and electron microscopy (Invited) [PRESENTATION-ONLY], Sergei V. Kalinin, Oak Ridge National Laboratory (United States) [view abstract]

Machine learning and artificial intelligence (ML/AI) are rapidly becoming an indispensable part of physics research, with domain applications ranging from theory and materials prediction to high-throughput data analysis. The recent successes in applying ML/AI methods for autonomous systems from robotics through self-driving cars to organic and inorganic synthesis are generating enthusiasm for the potential of these techniques to enable automated and autonomous experiment (AE) in imaging. In this presentation, I will discuss recent progress in automated experiment in electron and scanning probe microscopy, ranging from feature to physics discovery via active learning. The applications of classical deep learning methods in streaming image analysis are strongly affected by the out of distribution drift effects, and the approaches to minimize though are discussed. I will then illustrate transition from post-experiment data analysis to active learning process. Here, the strategies based on simple Gaussian Processes often tend to produce sub-optimal results due to the lack of prior knowledge and very simplified (via learned kernel function) representation of spatial complexity of the system. Comparatively, deep kernel learning (DKL) methods allow to realize both the exploration of complex systems towards the discovery of structure-property relationship, and enable automated experiment targeting physics (rather than simple spatial feature) discovery. The latter is illustrated via experimental discovery of the edge plasmons in STEM/EELS and ferroelectric domain dynamics in PFM.

 

10:15COIMG-281
Domain-aware Gaussian processes and high-performance mathematical optimization for optimal autonomous data acquisition (Invited) [PRESENTATION-ONLY], Marcus M. Noack, Lawrence Berkeley National Laboratory (United States) [view abstract]

Gaussian processes and Gaussian-related stochastic processes have been shown to be powerful tools for stochastic function approximation and autonomous control of data acquisition due to their robustness, analytical tractability, and natural inclusion of uncertainty quantification. In this talk, I want to present our work on a general, flexible, and powerful GP-driven framework for autonomous data acquisition. The focus of this work lies on making Gaussian (related) processes more flexible and domain aware, how the added flexibility and domain-awareness can be used for decision-making, and the computational and mathematical challenges that come with these advancements. In particular, I will focus on the challenge of hyperparameter optimization and how it impacts on-the-fly autonomous decision-making.

 

10:30COIMG-282
Active learning for jump regression estimation with applications to materials discovery [PRESENTATION-ONLY], Chiwoo Park¹, Peihua Qiu¹, Jennifer Carpena-Nunez², Rahul Rao², Michael Susner², and Benji Maruyama²; ¹Florida State University and ²Air Force Research Laboratory (United States) [view abstract]

Selecting input variables or design points for machine learning models has been of great interest in adaptive design and active learning. Motivated by two scientific examples, we present a strategy of selecting the design points for a regression model when the underlying regression function is discontinuous. The first example we undertook was to accelerate imaging speed in high-resolution material imaging, and the second was to use sequential design for mapping a chemical phase diagram. In both examples, the underlying regression functions have discontinuities, and thus many existing design optimization approaches cannot be used because they assume a continuous regression function. Although some existing adaptive design strategies developed from the treed regression models can handle the discontinuities, the related Bayesian model estimation approaches come with computationally expensive Markov Chain Monte Carlo algorithms for posterior inferences and the subsequent design point selections, which may not be applicable for the first motivating example that requires the computation to be faster than the original imaging speed. We present a simple and effective adaptive design strategy for regression analysis with discontinuities. The suggested sequential design selection procedure is then demonstrated using two motivating examples.

 

10:45COIMG-283
Constrained matrix factorization enabling real-time insights of in situ and high-throughput experiments [PRESENTATION-ONLY], Phillip M. Maffettone¹, Aidan C. Daly², and Daniel Olds¹; ¹Brookhaven National Laboratory and ²Flatiron Institute (United States) [view abstract]

Non-negative matrix factorization (NMF) is an appealing class of methods for performing unsupervised learning on streaming spectral data, particularly in time-sensitive applications such as in situ characterization of materials. These methods seek to decompose a dataset into a small number of components and weights that can compactly represent the underlying signal while effectively reconstructing the observations with minimal error. However, canonical NMF methods have no underlying requirement that the reconstruction uses components or weights that are representative of the true physical processes. In this work, we demonstrate how constraining a subset of the NMF weights or components as rigid priors, provided as known or assumed values, can provide significant improvement in revealing true underlying phenomena. We present a PyTorch based method for efficiently applying this generally constrained matrix factorization (CMF) and demonstrate its application to several synthetic examples. Our implementation allows an expert researcher-in-the-loop to provide and dynamically adjust the constraints during a live experiment involving streaming spectral data. Such interactive priors allow researchers to specify known or identified independent components, as well as functional expectations about the mixing or transitions between the components. We further demonstrate the application of this method to measured synchrotron x-ray total scattering data from in situ beamline experiments. In such a context, CMF can result in a more interpretive and scientifically relevant decomposition than canonical NMF or other decomposition techniques. We provide the details of the open-source implementation at https://github.com/NSLS-II/constrained-matrix-factorization, and will discuss general guidance for employing CMF in the extraction of critical information and insights during time-sensitive experimental applications.

 

Latent Fields in Additive Manufacturing: From Sensing to Reconstruction I

Session Chairs: Megna Shah, Air Force Research Laboratory (United States); Jeff Simmons, Air Force Research Laboratory (United States); and Amir Ziabari, Oak Ridge National Laboratory (United States)
15:00 – 16:00
Blue Room

15:00COIMG-293
Opportunities and challenges in metallic additive manufacturing [PRESENTATION-ONLY], Edwin J. Schwalbach, Air Force Research Laboratory (United States) [view abstract]

The geometric capabilities and local processing control afforded by additive manufacturing (AM) processes have driven significant interest in this emerging class of manufacturing technologies, but many challenges remain. Imaging and characterization of AM processes and the materials they produce are critical to enabling AM's use in application domains with stringent quality requirements, as well as to realize the promise of location-specific material property control. AM methods relax many constraints associated with conventional manufacturing techniques such as machining, enabling them to produce complex geometries. This opens regions of design space that were historically either not cost-effective or simply not possible to manufacture conventionally, including designs enabled by topology optimization algorithms. Additionally, because metals develop different microstructures and therefore properties in response to manufacturing processing details such as thermal- or strain-history, AM methods open opportunities for local tailoring of material performance, a new set of design degrees of freedom. However, both the geometric complexity and local variability inherent to AM processes also present significant challenges. Inherent variability in welding processes introduces performance-limiting flaws such as keyhole and lack of fusion porosity. These events may be rare when using optimized process parameters, but a typical laser powder bed fusion process requires 200 m of weld bead per cubic centimeter of solid material produced, meaning the presence of some flaws is a near certainty. Additionally, geometric complexity, even in relatively simple shapes, induces unintended variation in local processing state when using constant processing parameters. This, in turn, can introduce microstructural and therefore performance heterogeneities that complicate design processes. The layer-by-layer nature of AM processes provides direct access to internal regions and intermediate states during fabrication. Monitoring and measurement during AM fabrication can therefore provide insight into processing state and formation of flaws and other microstructural features, and potentially even provide input to closed-loop process control systems. Additionally, post-fabrication characterization of the material produced is necessary to corroborate in situ indications. Finally, both in and ex situ data streams are valuable for calibrating and validating process models, which can be used to design and optimize new processes.

 

15:15COIMG-294
Computer vision techniques for additive manufacturing quality control [PRESENTATION-ONLY], Vincent Paquit, Luke Scieme, Michael Sprayberry, and James Haley, Oak Ridge National Laboratory (United States) [view abstract]

Additive manufacturing technologies use a sequence of material deposition events to manufacture objects in three dimensions, process having the advantage to provide more flexibility in fabricating complex geometries when compared to traditional manufacturing technologies. This gain in flexibility however is overshadowed by an increased number of issues in part quality and performance such as the occurrence of characteristic manufacturing defects, material properties inhomogeneity within the geometry, or increased residual stress deformations. Therefore, to be widely adopted by industry, a new approach for certification and qualification of additively manufactured components is required. Toward this goal, there is a growing research focus on developing new methodologies for certification and qualification that synthesize modeling and simulation, in-situ monitoring, and ex-situ characterization. In this presentation, we will illustrate how computer vision techniques are used to advance the field of advanced manufacturing by providing a series of examples on additive and hybrid manufacturing systems from the ORNL Manufacturing Demonstration Facility.

 

15:30COIMG-295
Physics based compressive sensing for melt pool monitoring in laser powder bed fusion [PRESENTATION-ONLY], Yanglong Lu and Yan Wang, Georgia Institute of Technology (United States) [view abstract]

Laser powder bed fusion (LPBF) as a metal additive manufacturing process enables the fabrication of complex parts. However, its potential of industry applications is limited by the variability of build qualities. The microstructures of the build are influenced by the thermal history and melt flow. Conventional sensing methods including x-ray imaging for melt flow measurement and thermal sensors such as pyrometers and infrared (IR) cameras for temperature monitoring are limited by their spatial and temporal resolution. To overcome these limitations, a novel technique called physics based compressive sensing (PBCS) is developed to efficiently monitor the melt pool in the LPBF process. PBCS is performed with two stages. At stage one, from some limited temperature measurements on the top surface of the melt pool, the load vector of the thermofluid model is recovered by solving an inverse problem. At stage two, the complete temperature field and melt flow are reconstructed directly from the forward model. A new constrained orthogonal matching pursuit algorithm is developed to solve the inverse problem. The PBCS method is validated with experimental data from traditional IR cameras and x-ray imaging.

 

15:45COIMG-296
Challenges and progress in physical tomographic reconstruction of light doses for additive manufacturing [PRESENTATION-ONLY], Vishal Bansal¹, Indrasen Bhattacharya¹, Kyle M. Champley², Erika Fong², Chi C. Li¹, Robert McLeod³, Charles Rackson³, Maxim Shusteff², Hayden Taylor¹, and Joseph Toombs¹; ¹University of California, Berkeley, ²Lawrence Livermore National Laboratory, and ³University of Colorado (United States) [view abstract]

Compared to layer-based additive manufacturing processes, volumetric additive manufacturing (VAM) increases the speed with which photopolymers can be processed and broadens the range of accessible material properties. VAM processes involve illuminating all points within a 3D target geometry simultaneously. In tomographic VAM, otherwise known as computed axial lithography (CAL), the light dose is delivered via the projection of a temporally evolving pattern of light into a rotating volume of photopolymer precursor solution. Where the cumulative dose exceeds a material-dependent threshold, the material polymerizes and the desired component is formed. The core computational challenge in tomographic VAM is delivering to the material a reconstructed dose distribution with acceptable contrast and geometrical fidelity, given the constraint of positive intensity that is inherent to light projection. Several algorithms developed by the authors for tomographic VAM will be described, and their geometrical accuracy and robustness to process variations will be evaluated. Moreover, tomographic reconstruction of the evolving 3D refractive index distribution of material within the printing volume has been achieved via color Schlieren imaging. This method is promising as a real-time process-monitoring technique. Challenges posed to light dose reconstruction by light scattering from dispersed particles and refractive index inhomogeneities will also be discussed.

 

Latent Fields in Additive Manufacturing: From Sensing to Reconstruction II

Session Chairs: Megna Shah, Air Force Research Laboratory (United States); Jeff Simmons, Air Force Research Laboratory (United States); and Amir Ziabari, Oak Ridge National Laboratory (United States)
16:15 – 17:15
Blue Room

16:15COIMG-304
Linking processing to microstructure development under additive manufacturing conditions [PRESENTATION-ONLY], Amy J. Clarke, Colorado School of Mines (United States) [view abstract]

Additive manufacturing (AM) typically produces large temperature gradients, high solidification rates and repeated cycles of heating and cooling. A common characteristic of as-printed metallic alloys is the growth of coarse columnar grains that may impact mechanical response, so ways to control microstructure evolution are of great technological interest. Combinations of thermal gradient and solid/liquid interface velocity are known to significantly impact microstructure (and defect) development. Here we perform in-situ synchrotron x-ray imaging during simulated AM at the Advanced Photon Source at Argonne National Laboratory to better understand metallic alloy solidification under AM conditions. We directly measure solid/liquid interface velocities from in-situ experiments and velocities and thermal gradients from modeling. Combined with complementary ex-situ microstructure characterization and solidification theory (e.g., columnar to equiaxed transition predictions), results such as these will enable the prediction and control of microstructure development during advanced manufacturing.

 

16:30COIMG-306
Inferring surface properties of oscillating fluids from video by inversion of physics models, Bob Price¹, Svyatoslav Korneev¹, Adrian Lew², Christoforos Somarakis¹, and Raja Bala³; ¹Palo Alto Research Center Incorporated, ²Stanford University, and ³Amazon (United States) [view abstract]

Measuring the shape, motion and physical properties of oscillating fluids is critical for understanding the physics of fluidic systems, as well as optimizing and controlling such systems in real time. Conventional surface measurement techniques such as profile analysis or stereo reconstruction are not effective for monitoring fluids in industrial process due to the presence of occluding structures, extreme heat, and complex light interactions at the fluid surface. We propose a video-based method comprising forward and inverse transforms. The forward transform employs a physics-based fluid surface model combined with a ray-traced renderer to map shape and motion parameters to synthetic video frames. The inverse transform uses machine learning models to recover surface parameters from video. The inverse models are trained on synthetic data generated by the forward transform. We illustrate the method on an industrial 3D printer for which we recover the motion and surface of a molten aluminum alloy oscillating inside a microscopic nozzle. The inverse transform is ill-posed, but can be regularized. We show that surface properties can be reliably inferred with either a suitably regularized non-parametric k-nearest neighbor regressor or a deep convolutional network whose results are less stable but faster to compute.