Monday 17 January 2022
IS&T Welcome & PLENARY: Quanta Image Sensors: Counting Photons Is the New Game in Town
07:00 – 08:10
The Quanta Image Sensor (QIS) was conceived as a different image sensor—one that counts photoelectrons one at a time using millions or billions of specialized pixels read out at high frame rate with computation imaging used to create gray scale images. QIS devices have been implemented in a CMOS image sensor (CIS) baseline room-temperature technology without using avalanche multiplication, and also with SPAD arrays. This plenary details the QIS concept, how it has been implemented in CIS and in SPADs, and what the major differences are. Applications that can be disrupted or enabled by this technology are also discussed, including smartphone, where CIS-QIS technology could even be employed in just a few years.
Eric R. Fossum, Dartmouth College (United States)
Eric R. Fossum is best known for the invention of the CMOS image sensor “camera-on-a-chip” used in billions of cameras. He is a solid-state image sensor device physicist and engineer, and his career has included academic and government research, and entrepreneurial leadership. At Dartmouth he is a professor of engineering and vice provost for entrepreneurship and technology transfer. Fossum received the 2017 Queen Elizabeth Prize from HRH Prince Charles, considered by many as the Nobel Prize of Engineering “for the creation of digital imaging sensors,” along with three others. He was inducted into the National Inventors Hall of Fame, and elected to the National Academy of Engineering among other honors including a recent Emmy Award. He has published more than 300 technical papers and holds more than 175 US patents. He co-founded several startups and co-founded the International Image Sensor Society (IISS), serving as its first president. He is a Fellow of IEEE and OSA.
08:10 – 08:40 EI 2022 Welcome Reception
Wednesday 19 January 2022
IS&T Awards & PLENARY: In situ Mobility for Planetary Exploration: Progress and Challenges
07:00 – 08:15
This year saw exciting milestones in planetary exploration with the successful landing of the Perseverance Mars rover, followed by its operation and the successful technology demonstration of the Ingenuity helicopter, the first heavier-than-air aircraft ever to fly on another planetary body. This plenary highlights new technologies used in this mission, including precision landing for Perseverance, a vision coprocessor, new algorithms for faster rover traverse, and the ingredients of the helicopter. It concludes with a survey of challenges for future planetary mobility systems, particularly for Mars, Earth’s moon, and Saturn’s moon, Titan.
Larry Matthies, Jet Propulsion Laboratory (United States)
Larry Matthies received his PhD in computer science from Carnegie Mellon University (1989), before joining JPL, where he has supervised the Computer Vision Group for 21 years, the past two coordinating internal technology investments in the Mars office. His research interests include 3-D perception, state estimation, terrain classification, and dynamic scene analysis for autonomous navigation of unmanned vehicles on Earth and in space. He has been a principal investigator in many programs involving robot vision and has initiated new technology developments that impacted every US Mars surface mission since 1997, including visual navigation algorithms for rovers, map matching algorithms for precision landers, and autonomous navigation hardware and software architectures for rotorcraft. He is a Fellow of the IEEE and was a joint winner in 2008 of the IEEE’s Robotics and Automation Award for his contributions to robotic space exploration.
Machine Learning for Scientific Imaging 2022 Posters
08:20 – 09:20
EI Symposium
Poster interactive session for all conferences authors and attendees.
MLSI-202
P-17: Advantage of machine learning over maximum likelihood in limited-angle low-photon x-ray tomography, Zhen Guo1, Jung Ki Song1, George Barbastathis1, Michael A. Glinsky2, Courtenay T. Vaughan2, Kurt W. Larson2, Bradley K. Alpert3, and Zachary H. Levine4; 1Massachusetts Institute of Technology, 2Sandia National Laboratory, 3Applied and Computational Mathematics Division, National Institute of Standards and Technology, and 4Quantum Measurement Division, National Institute of Standards and Technology (United States) [view abstract]
Limited-angle X-ray tomography reconstruction is an ill-posed inverse problem in general. Especially when the projection angles are limited, reconstructions from classical algorithms such as filtered backprojection may lose fidelity and acquire artifacts due to the missing cone problem. To obtain satisfactory reconstruction results, prior assumptions, such as total variation minimization and nonlocal image similarity, are usually incorporated within the reconstruction algorithm. In this work, we introduce deep neural networks to determine and apply a prior in the reconstruction process. Our neural networks learn the prior directly from the synthetic training samples. The neural nets obtain a prior distribution that is specific to the reconstructions we are interested in. In particular, we used deep generative models with 3D convolution and 3D attention which are trained on 3D synthetic integrated circuit (IC) data from a model dubbed CircuitFaker. We demonstrate that the priors from our deep generative models can drastically improve the IC reconstruction quality on synthetic data from a maximum likelihood algorithm when the projection angles and photon budgets are limited. The advantages of using machine learning in limited angle X-ray tomography may further enable its applications in low-photon nanoscale imaging.
MLSI-203
P-18: CNN to mitigate atmospheric turbulence effect on Shack-Hartmann Wavefront Sensing: A case study on the Magdalena Ridge Observatory Interferometer, Siavash Norouzi1, James J. Luis2, Ramyaa Ramyaa1, John S. Young2, Eugene B. Seneta2, Morteza Darvish Morshedi Hosseini3, and Edgar R. Ligon4; 1New Mexico Institute of Mining and Technology (United States), 2University of Cambridge (United Kingdom), 3Binghamton University (United States), and 4CHARA Array (United States) [view abstract]
The Magdalena Ridge Observatory Interferometer (MROI) utilizes Shack-Hartmann Wavefront Sensing (SH-WFS) for the back-end stability of its beam relay systems in a unique design. The SH-WFS hardware, however, is sensitive to atmospheric turbulence scintillation which can drastically affect its precision in calculating the position of the beam profile it sees. A large number of images are needed to counteract the turbulence effect. Here we use deep learning as an alternative to long averaging cycles. A CNN was trained to map a series of star frames to the average image of the entire series. The network was trained with different positioning scenarios of the beam profile. The results show that the network can determine the beam position that would be measured from 100 frames using only 10 frames as input. The mapping accuracy is within the permissible error margin of 0.1 pixels and the network can furnish proper generalization to the positions not seen during training. It can also outperform the averaging technique when both techniques operate on low numbers of input frames.
MLSI-204
P-19: ISP distillation [PRESENTATION-ONLY], Eli Schwartz1, Alex Bronstein2, and Raja Giryes1; 1Tel Aviv University and 2Technion (Israel) [view abstract]
Nowadays, many of the images captured are ”observed” by machines only and not by humans, for example, robots’ or autonomous cars’ cameras. High-level machine vision models, such as object recognition or semantic segmentation, assume images are transformed to some canonical image space by the camera ISP. However, the camera ISP is optimized for producing visually pleasing images to human observers and not for machines, thus, one may spare the ISP compute time and apply the vision models directly on the raw data. Yet, it has been shown that training such models directly on the RAW images results in a performance drop. To mitigate this drop in performance (without the need to annotate RAW data), we use a dataset of RAW and RGB image pairs, which can be easily acquired with no human labeling. We then train a model that is applied directly on the RAW data by using knowledge distillation such that the model predictions for RAW images will be aligned with the predictions of an off-the-shelf pre-trained model for processed RGB images. Our experiments show that our performance on RAW images for object classification and semantic segmentation are significantly better than a model trained on labeled RAW images. It also reasonably matches the predictions of a pre-trained model on processed RGB images, while saving the ISP compute overhead.
Monday 24 January 2022
KEYNOTE: Inverse Problems in Imaging
Session Chair: Marc Klasky, Los Alamos National Laboratory (United States)
07:00 – 08:05
Green Room
07:00
Conference Introduction
07:05MLSI-315
KEYNOTE: Tackling tough inverse problems in imaging using PINNs and DeepOnets [PRESENTATION-ONLY], George E. Karniadakis, Brown University (United States)
We will review physics-informed neural networks (PINNs) and operator regression networks (DeepOnets) with emphasis on discovering missing physics and system identification in diverse applications in fluid mechanics, solid mechanics, bioengineering, and beyond. We will demonstrate that we can use multimodality inputs from images and point measurements to discover effects in materials, obtain three-dimension fields, and improve greatly existing techniques such as particle tracking in fluid mechanics. The diverse problems we consider are ill-posed and cannot be solved with any traditional methods. For example, in one application in aortic dissections we identify from mechanical measurements the genotype that corresponds to the specific mouse tested among five different classes.
George E. Karniadakis received his SM (1984) and PhD (1987) from Massachusetts Institute of Technology. He was appointed Lecturer in the department of mechanical engineering at MIT in 1987 and subsequently he joined the Center for Turbulence Research at Stanford / Nasa Ames. He joined Princeton University as assistant professor in the department of mechanical and aerospace engineering and as associate faculty in the lrogram of applied and computational mathematics. He was a Visiting Professor at Caltech (1993) in the Aeronautics department. He joined Brown University as associate professor of applied mathematics in the Center for Fluid Mechanics on January 1, 1994. He became a full professor on July 1, 1996. He has been a Visiting Professor and Senior Lecturer of Ocean/Mechanical Engineering at MIT since September 1, 2000. He was Visiting Professor at Peking University (Fall 2007 & 2013). He has a joing appointment with PNNL since 2013. He is a Fellow of the Society for Industrial and Applied Mathematics (SIAM, 2010-), Fellow of the American Physical Society (APS, 2004-), Fellow of the American Society of Mechanical Engineers (ASME, 2003-) and Associate Fellow of the American Institute of Aeronautics and Astronautics (AIAA, 2006-). He received the SIAM CSE/ACM prize (2021), the SIAM Ralf E Kleinman award (2015), the (inaugural) J. Tinsley Oden Medal (2013), and the CFD award (2007) by the US Association in Computational Mechanics. His h-index is 112 and he has been cited more than 59,500 times.
07:45MLSI-316
Signal reconstruction of sparse, nano-scale metrology data using neural networks [PRESENTATION-ONLY], Eva Natinsky1 and Remi Dingreville2; 1The University of Texas at Austin and 2Sandia National Laboratories (United States) [view abstract]
In both large area nanomanufacturing and materials analysis the need is growing for real-time process control, feature adjustment and error correction. Macroscale atomic force microscopy (AFM) scans at nanoscale resolutions can take hours and produce datasets with millions of points, which are time consuming and computationally expensive to analyze. Using machine learning it is possible to reconstruct sparse metrology images into high-resolution scans with minimal time and computational requirements. In this project we propose developing a technique for intelligent, sparse sampling of data over large areas which can be processed through an existing machine learning framework to construct a higher resolution image. This two-step approach redesigns the scanning pattern based on identification of feature-rich areas and utilizes a reconstruction algorithm capable of extrapolating data between scan points using machine learning and image analysis. This allows a user to take scans more efficiently with the fewest data points, significantly reducing scan time and decreasing processing complexity. The new scanning technique will use a real-time gradient calculation performed during scanning that adjusts scanning density as feature density varies; preliminary results from the reconstruction algorithm demonstrate that a sparse, uniform AFM scan of 250x250 μm taken in several minutes could be ‘filled in’ within five minutes, and the output image would be comparable to a higher resolution scan requiring 30 minutes or more. With the gradient-based sampling pattern and further modification to the reconstruction algorithm, we can expect to achieve even greater resolution improvements.
Video -- Intelligent Manufacturing -- Dynamic Tomography
Session Chair:
Marc Klasky, Los Alamos National Laboratory (United States)
08:30 – 09:30
Green Room
08:30MLSI-320
Video from coded motion blur using dynamic phase coding [PRESENTATION-ONLY], Erez Yosef, Shay Elmalem, and Raja Giryes, Tel Aviv University (Israel) [view abstract]
Motion blur is a known issue in photography, caused by capturing moving objects during the exposure time. In this work, we aim to reconstruct a video of the scene during the exposure time from the inherent motion blur captured in a single image and generate a frame burst of the moving scene. As conventional motion blur is ambiguous, such a task is highly ill-posed. To circumvent this issue, a computational imaging approach is proposed and used for the task of video reconstruction from a motion blurred image. Using dynamic phase-coding in the lens aperture during the image acquisition, the trajectory of the motion is encoded in an intermediate optical image. These color cues serve as prior information for the reconstruction process, and they are interpreted to reconstruct the video-burst using a single time-dependent convolutional neural network (CNN). Using a single coded motion-blurred image, our model can be configured parametrically to generate a sharp frame burst of the scene during exposure at any frame rate. We show the advantage of our approach over previous solutions.
08:50MLSI-321
Feature anomaly detection system (FADS) for intelligent manufacturing [PRESENTATION-ONLY], Anthony Garland, Sandia National Laboratories (United States) [view abstract]
Anomaly detection is important for industrial automation and part quality assurance, and while humans are able to easily detect anomalies in components given a few examples, designing a generic automated system that can perform at human or above human capabilities remains a challenge. In this work, we present a simple new anomaly detection algorithm called FADS (feature based anomaly detection system) which leverages a pretrained convolutional neural network (CNN) to generate a statistical model of what a normal part should look like. By using a pretrained network, FADS demonstrates excellent performance similar to or better than other machine learning approaches to anomaly detection while at the same time FADS requires no tuning of the CNN weights. We demonstrate FADS’ ability by detecting process parameter changes on a custom dataset of additively manufactured lattices. In addition, we test FADS on benchmark datasets, such as MVTec, and report good results.
09:10MLSI-322
Learning from hydrodynamics simulations with mass constraints for density reconstruction in dynamic tomography [PRESENTATION-ONLY], Zhishen Huang1, Michael T. McCann2, Jennie Disterhaupt2, Marc Klasky2, and Saiprasad Ravishankar1; 1Michigan State University and 2Los Alamos National Laboratory (United States) [view abstract]
Object density reconstruction from projections containing scattered radiation and noise is of critical importance in many applications. Incorporating machine-learned models could prove beneficial for accurate density reconstruction particularly in dynamic imaging, where the time-evolution of the density fields could be captured by learning from hydrodynamics simulations. In this work, we demonstrate the ability of a learned generative adversarial network (GAN) for artifact removal in noisy density reconstructions, where the noise is imperfectly characterized. The generator in the GAN serves as a denoiser that removes artifacts in densities obtained from traditional reconstruction algorithms. We train the GAN from large density time-series datasets, with noise simulated according to parametric random distributions that may mimic noise in experiments. Essentially, the GAN is trained with noisy density frames as generator inputs, to match the generator output distribution to the distribution of clean densities (time-series) from simulations. We studied both the classic GAN and the Wasserstein-GAN in our experiments. In addition, we found that the use of physics-based constraints (e.g., mass conservation) during network training and application allows for more accurate density reconstructions. Primitive numerical results show that the models trained in our frameworks can remove significant portions of unknown noise in density time-series data.
KEYNOTE: Machine Learning - High Data Volume
Session Chair: Marc Klasky, Los Alamos National Laboratory (United States)
10:00 – 11:00
Green Room
10:00MLSI-327
KEYNOTE: Physics based machine learning [PRESENTATION-ONLY], David Rousseau, Université Paris-Saclay (France)
High Energy Physics experiments aim at establishing fundamental laws of physics by measuring the product of high energy particle collision with increasingly complex detectors. For example, tracking detectors deliver point clouds of micron precision over cubic meters, calorimeter detectors provide energy deposits in irregularly shapes voxels. Dedicated Machine Learning techniques are developed to deal with the specificities of the data, with the constraints of: dealing with large volume of data (many PetaBytes) within resources, maximise sensitivity to rare phenomenons, master unknowns along the full pipe line to be able to quote uncertainties. In addition, a long history of building accurate complex simulators is being supplemented by ML generators orders of magnitude faster but which have to demonstrate their ability to reproduce all details of the feature space.
David Rousseau is a High Energy Physicist at IJCLab, CNRS/IN2P3 and Université Paris-Saclay, currently working for the ATLAS experiment at CERN on the Large Hadron Collider. After a dozen years designing and implementing many pieces of the ATLAS experiment software, a chance meeting in 2013 with a machine learning computer scientist decided a new path in his career: develop the interface between high energy physics (and science in general) and machine learning (or artificial intelligence). Current research topics include: simulation based inference, machine learning for fast particle tracking at the large hadron collider, generator models for fast detector simulation, using an OPU for event classification, and uncertainty aware training. Rousseau received his PhD (1992) in high energy physics from the Université Aix-Marseille II. Since 2009, Rousseau has been senior researcher (Directeur de Recherche) at CNRS/IN2P3 LAL-Orsay, which merged into IJCLab-Orsay in 2020. He began his career at CERN as a CERN Fellow in 1997.
10:40MLSI-328
Learning optimal wavefront shaping for multi-channel imaging [PRESENTATION-ONLY], Elias Nehme1, Boris Ferdman1, Lucien E. Weiss1, Tal Naor1, Daniel Freedman2, Tomer Michaeli1, and Yoav Shechtman1; 1Technion -- Israel Institute of Technology and 2Google Machine Perception (Israel) [view abstract]
Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. In this work, we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled telomeres in dense environments.
Super Resolution -- Toroidal Magnetic Fields -- Nonlocal Kernel Network
Session Chairs:
Marc Klasky, Los Alamos National Laboratory (United States) and Jong Chul Ye, Korea Advanced Institute of Science and Technology (KAIST) (Republic of Korea)
15:00 – 16:00
Green Room
15:00MLSI-336
Fast neural Poincare maps for toroidal magnetic fields [PRESENTATION-ONLY], Joshua W. Burby, Los Alamos National Laboratory (United States) [view abstract]
Poincaré maps for toroidal magnetic fields are routinely employed to study gross confinement properties in devices built to contain hot plasmas. We show that a novel neural network architecture, the HénonNet, is capable of accurately learning realistic Poincaré maps from observations of a conventional field-line-following algorithm. The HénonNet architecture exactly reproduces the primary physics constraint imposed on field-line Poincaré maps: flux preservation.
15:20MLSI-337
Nonlocal Kernel Network (NKN): A stable and resolution-independent deep neural network [PRESENTATION-ONLY], Marta D'Elia, Sandia National Laboratories (United States) [view abstract]
Deep neural networks, often required in complex learning tasks such as image classification, are hard to train. We propose a novel formulation, inspired by graph kernel networks (GKNs), that allows for deep layers. Our nonlocal kernel network (NKN) stems from the interpretation of the neural network as a discrete nonlocal diffusion-reaction problem that, in the limit of infinite layers, is equivalent to a parabolic nonlocal equation, whose stability is proved via integral calculus. The resemblance with graph neural networks allows NKNs to capture long-range dependencies in the feature space, while the continuous treatment of node-to-node interactions makes NKNs resolution independent. Furthermore, the resemblance with neural ODEs, reinterpreted in a nonlocal sense, and the stable network dynamics between layers allow for generalization of NKN's optimal parameters from shallow to deep networks. This fact enables the use of shallow-to-deep initialization techniques. Our tests show that NKNs outperform baseline methods in both PDE learning tasks and image classification tasks and generalize well to different resolutions and depths.
KEYNOTE: Computational Imaging Pipelines
Session Chairs: Marc Klasky, Los Alamos National Laboratory (United States) and Jong Chul Ye, Korea Advanced Institute of Science and Technology (KAIST) (Republic of Korea)
16:15 – 17:15
Green Room
16:15MLSI-368
KEYNOTE: Learning to image the invisible [PRESENTATION-ONLY], Katherine L. Bouman, California Institute of Technology (United States)
As imaging requirements become more demanding, we must rely on increasingly sparse and/or noisy measurements that fail to paint a complete picture. Computational imaging pipelines, which replace optics with computation, have enabled image formation in situations that are impossible for conventional optical imaging. For instance, seismic and black hole imaging have only been made possible through the development of computational imaging pipelines. However, these computational "cameras" often suffer from (a) being difficult to analyze for image uncertainties, and (b) forward model mismatch. This talk will discuss how we are leveraging and building upon recent advances in machine learning in order to achieve more efficient uncertainty quantification and to better recover solutions in the presence of unknown model mismatch. The proposed approaches will be presented in the context of discussing the methods and procedures used to capture the first and future images of a black hole from the Event Horizon Telescope, as well as seismic localization and tomography.
Katherine L. Bouman is a Rosenberg Scholar and assistant professor of computing and mathematical sciences (CMS) and by courtesy in Electrical Engineering and Astronomy at Caltech in Pasadena, California. Bouman's research focuses on computational imaging: designing systems that tightly integrate algorithm and sensor design, making it possible to observe phenomena previously difficult or impossible to measure with traditional approaches. Her group at Caltech combines ideas from signal processing, computer vision, machine learning, and physics to find and exploit hidden signals for both scientific discovery and technological innovation. Prior to starting at Caltech, Bouman was a postdoctoral fellow with the Event Horizon Telescope, which published the first picture of a black hole in April of 2019. She received her PhD (2017) in electrical engineering and computer science from the Massachusetts Institute of Technology (MIT) and previously received a BSE (2011) in electrical engineering from the University of Michigan, Ann Arbor, MI, and an SM (2013) in electrical engineering and computer science from MIT. Bouman is a recipient of an NSF CAREER Award, the Electronic Imaging Scientist of the Year Award, an Okawa Research Grant, a Caltech faculty teaching award, a finalist for the AAAS Early Career Award for Public Engagement with Science, and is a co-recipient of the Breakthrough Prize.
16:55MLSI-369
Limited-angle CT with deep physics and image priors [PRESENTATION-ONLY], Semih Barutcu1, Selin Aslan2, Aggelos Katsaggelos1, and Doga Gursoy1,2; 1Northwestern University and 2Argonne National Laboratory (United States) [view abstract]
Priors are essential for having robust solutions to the limited-angle CT problem, in which the object cannot be illuminated uniformly from all directions. To address the problem, we consider a self-supervised learning approach that uses priors in the form of deep neural networks. We describe the algorithmic framework that combines the data likelihood and prior terms into a single deep network. We demonstrate that this approach is computationally more tractable and improves performance through an end-to-end training. In addition, we show that our approach can be used jointly with a total-variation regularization when needed. We describe the solver which is based on alternating direction method of multipliers, and present numerical results for various degrees of missing angle range and noise levels.
Tuesday 25 January 2022
IS&T Awards & PLENARY: Physics-based Image Systems Simulation
07:00 – 08:00
Three quarters of a century ago, visionaries in academia and industry saw the need for a new field called photographic engineering and formed what would become the Society for Imaging Science and Technology (IS&T). Thirty-five years ago, IS&T recognized the massive transition from analog to digital imaging and created the Symposium on Electronic Imaging (EI). IS&T and EI continue to evolve by cross-pollinating electronic imaging in the fields of computer graphics, computer vision, machine learning, and visual perception, among others. This talk describes open-source software and applications that build on this vision. The software combines quantitative computer graphics with models of optics and image sensors to generate physically accurate synthetic image data for devices that are being prototyped. These simulations can be a powerful tool in the design and evaluation of novel imaging systems, as well as for the production of synthetic data for machine learning applications.
Joyce Farrell, Stanford Center for Image Systems Engineering, Stanford University, CEO and Co-founder, ImagEval Consulting (United States)
Joyce Farrell is a senior research associate and lecturer in the Stanford School of Engineering and the executive director of the Stanford Center for Image Systems Engineering (SCIEN). Joyce received her BS from the University of California at San Diego and her PhD from Stanford University. She was a postdoctoral fellow at NASA Ames Research Center, New York University, and Xerox PARC, before joining the research staff at Hewlett Packard in 1985. In 2000 Joyce joined Shutterfly, a startup company specializing in online digital photofinishing, and in 2001 she formed ImagEval Consulting, LLC, a company specializing in the development of software and design tools for image systems simulation. In 2003, Joyce returned to Stanford University to develop the SCIEN Industry Affiliates Program.
PANEL: The Brave New World of Virtual Reality
08:00 – 09:00
Advances in electronic imaging, computer graphics, and machine learning have made it possible to create photorealistic images and videos. In the future, one can imagine that it will be possible to create a virtual reality that is indistinguishable from real-world experiences. This panel discusses the benefits of this brave new world of virtual reality and how we can mitigate the risks that it poses. The goal of the panel discussion is to showcase state-of-the art synthetic imagery, learn how this progress benefits society, and discuss how we can mitigate the risks that the technology also poses. After brief demos of the state-of-their-art, the panelists will discuss: creating photorealistic avatars, Project Shoah, and digital forensics.
Panel Moderator: Joyce Farrell, Stanford Center for Image Systems Engineering, Stanford University, CEO and Co-founder, ImagEval Consulting (United States)
Panelist: Matthias Neissner, Technical University of Munich (Germany)
Panelist: Paul Debevec, Netflix, Inc. (United States)
Panelist: Hany Farid, University of California, Berkeley (United States)
CT Reconstruction -- Decomposition
Session Chairs:
Marc Klasky, Los Alamos National Laboratory (United States) and Jong Chul Ye, Korea Advanced Institute of Science and Technology (KAIST) (Republic of Korea)
16:15 – 17:15
Red Room
16:15MLSI-378
Limited-view cone beam CT reconstruction using 3D patch-based supervised and adversarial learning: Validation using hydrodynamic simulations and experimental tomographic data [PRESENTATION-ONLY], Marc Klasky1, Anish Lahiri2, Jeff Fessler2, Saiprasad Ravishankar3, Michelle Espy1, Michael T. McCann1, Trevor Wilcox1, and Ajeeta Khatiwada1; 1Los Alamos National Laboratory, 2University of Michigan, and 3Michigan State University (United States) [view abstract]
We have recently developed and tested a machine learning architecture that enables highly accurate reconstructions using extremely limited views as well as very limited training data. In this presentation we utilize this architecture to perform tomographic reconstructions of complex hydrodynamic phenomena i.e. Richtmyer Meshkoff Instabilities (RMI) using training data generated from three-dimensional computational fluid dynamic simulations. The network is then tested using experimental data acquired from extended objects (objects generated with a nominal density but a path-length such that the transmission matches that of the actual image with elevated density).
16:35MLSI-379
Machine learning and spatial decomposition for large CT scans [PRESENTATION-ONLY], Gary J. Saavedra and Eric C. Cyr, Sandia National Laboratories (United States) [view abstract]
The application of machine learning for image segmentation and classification has become ubiquitous in areas such as medical imaging and manufacturing defect assessment. In particular, UNet has achieved state-of-the-art results on a variety of such tasks involving both 2D and 3D CT scan images. However, as the size of these images grows, the application of UNet becomes infeasible due to increasing computational demands. In this talk, we introduce spatial decomposition and parallelization as a method for addressing the computational limits of UNet for large images and also demonstrate the applicability of these methods for several datasets. Finally, we show some of the difficulties involved in the parallelization of UNet. In particular, we address the large memory and processor requirements that can often hinder the applicability of UNet to high resolution CT scans.
16:55MLSI-380
X-ray CT reconstruction leveraging CAD models, physics-based information and GANs [PRESENTATION-ONLY], Amir K. Ziabari1, Abhishek Dubey2, Singanallur Venkatakrishnan1, Paul Brackman3, Curtis Frederick3, Philip R. Bingham1, Ryan Dehoff1, and Vincent Paquit1; 1Oak Ridge National Laboratory, 2Center of Cancer Research (NCI-NIH), and 3Carl Zeiss Metrology LLC (United States) (United States) [view abstract]
Qualification/certification is a major bottleneck compromising the efficacy of the additive manufacturing (AM) processes in printing consistent and reliable parts. Non-destructive characterization (NDC) of complex parts using X-ray computed tomography (XCT) systems plays a vital role in qualification/certification of manufactured parts. With recent technological advancement in industrial computed tomography (CT), it is expected that there will be a breakthrough in enabling high-speed inspection of 100% AM parts, where CT systems can get integrated into the chain of manufacturing process and combined with the defect analysis and feedback virtually enable defect-free production. For dense and complex metal AM parts, beam hardening (BH), metal artifacts, noise and scattering pose challenges for traditional CT image reconstruction algorithms. Accelerating the scans for high-throughput NDC will further compromise the quality of the reconstructed images. In this talk, I will present our recent work on leveraging the CAD model of the parts and the physics-based information along with generative adversarial networks (GANs) and convolutional neural networks (CNNs) to produce fast and high-quality reconstruction of the parts while reducing the scan time and improving the qualification process. I will also present our promising results both on synthetic and real datasets demonstrating marked performance of our approach.
Wednesday 26 January 2022
KEYNOTE: 2D Unknown View Tomography -- Phase Imaging and Artificial Intelligence
Session Chairs: Marc Klasky, Los Alamos National Laboratory (United States) and Jong Chul Ye, Korea Advanced Institute of Science and Technology (KAIST) (Republic of Korea)
15:00 – 16:00
Green Room
15:00MLSI-405
KEYNOTE: An adversarial learning approach for 2D unknown view tomography [PRESENTATION-ONLY], Mona Zehni and Zhizhen J. Zhao, University of Illinois at Urbana-Champaign (United States)
The goal of 2D tomographic reconstruction is to recover an image given its projection lines from various views. It is often presumed that projection angles associated with the projections are known in advance. Under certain situations, however, these angles are known only approximately or are completely unknown, when imaging moving objects. It becomes challenging to reconstruct the image from a collection of random projection. We introduce an adversarial learning based approach to recover the image and the projection angle distribution by matching the empirical distribution of the measurements with the generated data. Fitting the distributions is achieved through solving a min-max game between a generator and a critic based on Wasserstein generative adversarial network structure. To accommodate the update of the projection angle distribution through gradient back propagation, we approximate the loss using the Gumbel-Softmax reparameterization of samples from discrete distributions. Our theoretical analysis verifies the unique recovery of the image and the projection distribution up to a rotation and reflection upon convergence. Our numerical experiments showcase the potential of the method to accurately recover the image and the projection angle distribution under noise contamination.
Zhizhen J. Zhao is an assistant professor in the department of electrical and computer engineering at the University of Illinois, Urbana-Champaign, with affiliation to the Coordinated Science Laboratory and the National Center Supercomputing Applications. She is also an affiliate assistant professor in the department of mathematics and the department of statistics. Her areas of research include geometric data analysis, dimensionality reduction, mathematical signal processing, scientific computing, and machine learning. Applications to imaging sciences and inverse problems, including cryo-electron microscopy image processing, data-driven methods for dynamical systems, and uncertainty quantification. Prior to joining ECE Illinois in 2016, she was a Courant Instructor at the Courant Institute of Mathematical Sciences, New York University. She received her PhD in physics from Princeton University (2013) working with Amit Singer and graduated from Trinity College, Cambridge University, with a BS and MS in physics (2008).
15:40MLSI-406
Quantitative phase imaging and artificial intelligence: Label-free 3D imaging, classification, and inference [PRESENTATION-ONLY], Yongkeun Park1,2; 1Korea Advanced Institute of Science and Technology (KAIST) and 2Tomocube, Inc. (Republic of Korea) [view abstract]
Quantitative phase imaging (QPI) exploits refractive index (RI) distributions in cells and tissues as intrinsic and quantitative imaging contrast Optical diffraction tomography (ODT) is one of the 3D QPI techniques. ODT is an optical analogous to X-ray computed tomography (CT). Multiple 2-D holograms, containing both the amplitude and phase information. of a sample are measured with various illumination angles, from which a 3-D RI distribution of the sample is reconstructed by inversely solving the wave equation. When label-free and quantitative 3D imaging capability of ODT is combined with machine learning, it can provide synergistic capability in bioimaging and clinical diagnosis. We will discuss the potentials and challenges of combining QPI and artificial intelligence in terms of various aspects of imaging and analysis, including phase retrieval, tomographic reconstruction, segmentation, imaging inference, and noise reduction. In particular, we discuss the segmentation of cellular and subcellular features, the classification of bacterial cell types, and the inference of molecular information from unlabeled RI tomogram of live cells.
Imaging Through Scattering Medium -- Hydrodynamics Simulations
Session Chairs:
Marc Klasky, Los Alamos National Laboratory (United States) and Jong Chul Ye, Korea Advanced Institute of Science and Technology (KAIST) (Republic of Korea)
16:15 – 17:15
Green Room
16:15MLSI-410
Physics-embedded deep learning for diffuser-based funduscope [PRESENTATION-ONLY], Yunzhe Li and Lei Tian, Boston University (United States) [view abstract]
People worldwide currently suffer from poor vision that could be improved with effective ocular diagnosis and treatment. Here, we demonstrate a diffuser-based ocular computational imaging platform that integrates novel DL techniques for high-resolution fundus reconstruction with known physical model. We built a low-cost diffuser-based computational funduscope that can recover pathological features of the model eye fundus over 33° FOV and is robust to ±4D refractive error. We then developed a novel physics-embedded deep learning framework for high quality fundus image reconstruction with two stages. In the inversion stage, intermediate fundus images are recovered by L2 norm reconstruction with different regularization parameters. In the enhancement stage, the intermediate images are enhanced by a holistic attention network. We show that our reconstruction framework significantly outperforms the classical L2 norm reconstruction with finer feature retrieved and suppressed noise artifacts. The proposed framework can have great potential on clinically relevant fundus imaging applications.
16:35MLSI-411
Imaging through scattering medium with deep phase retrieval [PRESENTATION-ONLY], Mooseok Jang, Hyungjin Chung, Hyeonggeon Kim, Gookho Song, and Jong Chul Ye, Korea Advanced Institute of Science and Technology (KAIST) (Republic of Korea) [view abstract]
Light scattering is a ubiquitous phenomenon in nature. Examples are including scattering in fog, biological tissues, and atmosphere. When the degree of scattering exceeds the ballistic regime, multiple light scattering impedes optical imaging and focusing inside turbid media. In the past decade, numerous hardware-based approaches have demonstrated the capability to deterministically control multiply scattered waves based on wavefront shaping and transmission matrix formalism. However, these approaches are inevitably slow and require an exacting measurement condition, thus making them not suitable for practical use. In this talk, we present a computational method based on deep learning techniques that enables imaging through scattering medium. Unlike the existing supervised approaches, we directly incorporated a physical theory that dictates some static statistical features emerged from multiply scattered light in learning process and enabled for the first time, to best of our knowledge, imaging of complex objects in an unsupervised manner. We further show that the presented scheme renders the unprecedented adaptability in scattering properties and scattering configuration. Our physics-based approach will pave the way to the use of deep learning techniques for imaging in practical scattering environments.
16:55MLSI-412
Physics based hydrodynamic learning from hydrodynamics simulations with mass constraints for density reconstruction in dynamic tomography [PRESENTATION-ONLY], Marc Klasky1, Balu Nadiga1, Soumi De1, Oleg Korobkin1, Jennie Disterhaupt1, Trevor Wilcox1, Maliha Hossain1,2, and Charles A. Bouman2; 1Los Alamos National Laboratory and 2Purdue University (United States) [view abstract]
The reconstruction of density objects via projections has a long history dating back to Radon. In many cases, however the presence of experimental artifacts i.e., noise, scattered radiation, and complex beam characteristics, do not enable sufficiently accurate density reconstructions to elucidate the underlying physics. Furthermore, the density fields obtained from a data driven radiographic approach of a dynamic sequence of images do not incorporate any notion of an underlying physics model describing the underlying evolution of the density fields. In this work, we demonstrate the inability of traditional radiographic reconstruction algorithms to capture allowable hydrodynamic paths using the discriminator capability of a trained Conditional Generative Adversarial Network (CGAN). In addition, we demonstrate that the use of physics-based features obtained from the radiographic images in conjunction with a CGAN allows for more accurate density reconstructions, which are consistent with the underlying hydrodynamics, to be obtained.