Monday 17 January 2022
IS&T Welcome & PLENARY: Quanta Image Sensors: Counting Photons Is the New Game in Town
07:00 – 08:10
The Quanta Image Sensor (QIS) was conceived as a different image sensor—one that counts photoelectrons one at a time using millions or billions of specialized pixels read out at high frame rate with computation imaging used to create gray scale images. QIS devices have been implemented in a CMOS image sensor (CIS) baseline room-temperature technology without using avalanche multiplication, and also with SPAD arrays. This plenary details the QIS concept, how it has been implemented in CIS and in SPADs, and what the major differences are. Applications that can be disrupted or enabled by this technology are also discussed, including smartphone, where CIS-QIS technology could even be employed in just a few years.
Eric R. Fossum, Dartmouth College (United States)
Eric R. Fossum is best known for the invention of the CMOS image sensor “camera-on-a-chip” used in billions of cameras. He is a solid-state image sensor device physicist and engineer, and his career has included academic and government research, and entrepreneurial leadership. At Dartmouth he is a professor of engineering and vice provost for entrepreneurship and technology transfer. Fossum received the 2017 Queen Elizabeth Prize from HRH Prince Charles, considered by many as the Nobel Prize of Engineering “for the creation of digital imaging sensors,” along with three others. He was inducted into the National Inventors Hall of Fame, and elected to the National Academy of Engineering among other honors including a recent Emmy Award. He has published more than 300 technical papers and holds more than 175 US patents. He co-founded several startups and co-founded the International Image Sensor Society (IISS), serving as its first president. He is a Fellow of IEEE and OSA.
08:10 – 08:40 EI 2022 Welcome Reception
Tuesday 18 January 2022
Computer Analysis of Fine Art
Session Chair:
Kurt Heumiller, Museum of Modern Art (United States)
10:20 – 11:25
Yellow Room
10:20
Conference Introduction
10:25CVAA-169
Improving semantic segmentation of fine art images using photographs rendered in a style learned from artworks, Thomas Heitzinger1 and David G. Stork2; 1TU Wien (Austria) and 2Consultant (United States) [view abstract]
Our central goal was to create automatic methods for semantic segmentation of human figures in images of fine art paintings. This is a difficult problem because the visual properties and statistics of artwork differ markedly from the natural photographs widely used in research in automatic segmentation. We used a deep neural network to transfer artistic style from paintings across several centuries to modern natural photographs in order to create a large data set of surrogate art images. We then used this data set to train a separate deep network for semantic image segmentation of genuine art images. Such data augmentation led to great improvement in the segmentation of difficult genuine artworks, revealed both qualitatively and quantitatively. Our unique technique of creating surrogate artworks should find wide use in many tasks in the growing field of computational analysis of fine art.
10:45CVAA-170
Extracting associations and meanings of objects depicted in artworks through bi-modal deep networks, Gregory Kell1, Ryan Rhys Griffiths2, Anthony Bourached3, and David G. Stork4; 1King's College London (United Kingdom), 2University of Cambridge (United Kingdom), 3University College London (United Kingdom), and 4Consultant (United States) [view abstract]
We present a novel bi-modal system based on deep networks to address the problem of learning associations and simple meanings of objects depicted in "authored" images, such as ne art paintings and drawings. Our overall system processes both the images and associated texts in order to learn associations between images of individual objects, their identities and the abstract meanings they signify. Unlike past deep net that describe depicted objects and infer predicates, our system identies meaning-bearing objects ("signifiers") and their associations ("signifieds") as well as basic overall meanings for target artworks. Our system had precision of 48% and recall of 78% with an F1 metric of 0.6 on a curated set of Dutch vanitas paintings, a genre celebrated for its concentration on conveying a meaning of great import at the time of their execution. We developed and tested our system on ne art paintings but our general methods can be applied to other authored images.
11:05CVAA-171
Improved identification of portraiture of the Julio-Claudian period with mobile apps (JIST-first), Dmitri A. Gusev, Purdue University (United States) [view abstract]
We present the results of our image analysis of portrait art from the Roman Empire's Julio-Claudian dynastic period. Our novel approach involves processing pictures of ancient statues, cameos, altar friezes, bas-reliefs, frescoes, and coins using modern mobile apps, such as Reface and FaceApp, to improve identification of the historical subjects depicted. In particular, we have discovered that the Reface app has limited, but useful capability to restore the approximate appearance of damaged noses of the statues. We confirm many traditional identifications, propose a few identification corrections for items located in museums and private collections around the world, and discuss the advantages and limitations of our approach. For example, Reface may make aquiline noses appear wider or shorter than they should be. This deficiency can be partially corrected if multiple views are available. We demonstrate that our approach can be extended to analyze portraiture from other cultures and historical periods. The article is intended for a broad section of the readers interested in how the modern AI-based solutions for mobile imaging merge with humanities to help improve our understanding of the modern civilization's ancient past and increase appreciation of our diverse cultural heritage.
Wednesday 19 January 2022
IS&T Awards & PLENARY: In situ Mobility for Planetary Exploration: Progress and Challenges
07:00 – 08:15
This year saw exciting milestones in planetary exploration with the successful landing of the Perseverance Mars rover, followed by its operation and the successful technology demonstration of the Ingenuity helicopter, the first heavier-than-air aircraft ever to fly on another planetary body. This plenary highlights new technologies used in this mission, including precision landing for Perseverance, a vision coprocessor, new algorithms for faster rover traverse, and the ingredients of the helicopter. It concludes with a survey of challenges for future planetary mobility systems, particularly for Mars, Earth’s moon, and Saturn’s moon, Titan.
Larry Matthies, Jet Propulsion Laboratory (United States)
Larry Matthies received his PhD in computer science from Carnegie Mellon University (1989), before joining JPL, where he has supervised the Computer Vision Group for 21 years, the past two coordinating internal technology investments in the Mars office. His research interests include 3-D perception, state estimation, terrain classification, and dynamic scene analysis for autonomous navigation of unmanned vehicles on Earth and in space. He has been a principal investigator in many programs involving robot vision and has initiated new technology developments that impacted every US Mars surface mission since 1997, including visual navigation algorithms for rovers, map matching algorithms for precision landers, and autonomous navigation hardware and software architectures for rotorcraft. He is a Fellow of the IEEE and was a joint winner in 2008 of the IEEE’s Robotics and Automation Award for his contributions to robotic space exploration.
Computer Vision and Image Analysis of Art 2022 Poster
08:20 – 09:20
EI Symposium
Poster interactive session for all conferences authors and attendees.
CVAA-186
P-03: Artist-specific style transfer for semantic segmentation of paintings: The value of large corpora of surrogate artworks, Matthias Wödlinger1, Thomas Heitzinger1, and David G. Stork2; 1TU Wien (Austria) and 2Consultant (United States) [view abstract]
Deep neural networks for semantic segmentation have recently outperformed other methods for natural images, partly due to the abundance of training data for this case. However, applying these networks to pictures from a different domain often leads to a significant drop in accuracy. Fine art paintings for highly stylized works, such as from Cubism or Expressionism, in particular, are challenging due to large deviations in shape and texture of certain objects when compared to natural images. In this paper, we demonstrate that style transfer can be used as a form of data augmentation during the training of CNN based semantic segmentation models to improve the accuracy of semantic segmentation models in art pieces of a specific artist. For this, we pick a selection of paintings from a specific style for the painters Egon Schiele, Vincent Van Gogh, Pablo Picasso and Willem de Kooning, create stylized training dataset by transferring artist-specific style to natural photographs and show that training the same segmentation network on surrogate artworks improves the accuracy for fine art paintings. We also provide a dataset with pixel-level annotation of 60 fine art paintings to the public and for evaluation of our method.