UPDATED
Perceptual Metrics for Image and Video Quality: From Perceptual Transparency to Structural Equivalence
Sunday 26 January • 13:30 – 17:45
Course Length: 4 hours
Course Level: Intermediate
Instructors: Thrasyvoulos N. Pappas, Northwestern University, and Sheila Hemami, Draper Lab

Learning Outcomes
This course enables the attendee to:

• Gain a basic understanding of the properties of the human visual system and how current applications (image and video compression, restoration, retrieval, etc.) attempt to exploit these properties.
• Gain an operational understanding of existing perceptually-based and structural similarity metrics, the types of images/artifacts on which they work, and their failure modes.
• Understand current distortion models for different applications, and how they can be used to modify or develop new metrics for specific contexts.
• Understand the differences between sub-threshold and supra-threshold artifacts, the HVS responses to these two paradigms, and the differences in measuring that response.
• Understand criteria by which to select and interpret a particular metric for a particular application.
• Understand the capabilities and limitations of full-reference, limited-reference, and no-reference metrics, and why each might be used in a particular application.

We examine objective criteria for the evaluation of image quality that are based on models of visual perception. Our primary emphasis is on image fidelity, i.e., how close an image is to a given original or reference image, but we broaden the scope of image fidelity to include structural equivalence. We also discuss no-reference and limited-reference metrics. We examine a variety of applications with special emphasis on image and video compression. We examine near-threshold perceptual metrics, which explicitly account for human visual system (HVS) sensitivity to noise by estimating thresholds above which the distortion is just-noticeable, and supra-threshold metrics, which attempt to quantify visible distortions encountered in high compression applications or when there are losses due to channel conditions. We also consider metrics for structural equivalence, whereby the original and the distorted image have visible differences but both look natural and are of equally high visual quality. We also take a close look at procedures for evaluating the performance of quality metrics, including database design, models for generating realistic distortions for various applications, and subjective procedures for metric development and testing. Throughout the course we discuss both the state of the art and directions for future research.

Intended Audience

• Image and video compression specialists who wish to gain an understanding of how performance can be quantified.
• Engineers and scientists who wish to learn about objective image and video quality evaluation.
• Managers who wish to gain a solid overview of image and video quality evaluation.
• Students who wish to pursue a career in digital image processing.
• Intellectual property and patent attorneys who wish to gain a more fundamental understanding of quality metrics and the underlying technologies.
• Government laboratory personnel who work in imaging.

Thrasyvoulos N. Pappas received aSB, SM, and PhD in electrical engineering and computer science from MIT in 1979, 1982, and 1987, respectively. From 1987 until 1999, he was a member of the Technical Staff at Bell Laboratories, Murray Hill, NJ. He is currently a professor in the department of electrical and computer engineering at Northwestern University, which he joined in 1999. His research interests are in image and video quality and compression, image and video analysis, content-based retrieval, perceptual models for multimedia processing, model-based halftoning, and tactile and multimodal interfaces. Pappas has served as Vice-President Publications, IEEE Signal Processing Society (2015-1017), editor-in-chief of the IEEE Transactions on Image Processing (2010-12), elected member of the Board of Governors of the Signal Processing Society of IEEE (2004-06), chair of the IEEE Image and Multidimensional Signal Processing (now IVMSP) Technical Committee (2002-03), technical program co-chair of ICIP-01 and ICIP-09, and co-chair of the 2011 IEEE IVMSP Workshop on Perception and Visual Analysis. He has also served as co-chair of the 2005 SPIE/IS&T Electronic Imaging Symposium and co-chair of the SPIE/IS&T Conference on Human Vision and Electronic Imaging (1997-2018). He is currently co-editor-in-chief of the IS&T Journal of Perceptual Imaging. Pappas is a Fellow of IEEE, SPIE, and IS&T.

Sheila S. Hemami received a BSEE from the University of Michigan (1990), and a MSEE and PhD from Stanford University (1992 and 1994). She was with Hewlett-Packard Laboratories in Palo Alto, California in 1994 and was with the School of Electrical Engineering at Cornell University from 1995-2013. From 2013 to 2016 she was professor and chair of the department of electrical and computer engineering at Northeastern University in Boston, MA. She is currently Director of Strategic Technical Opportunities at Draper Lab. Hemami's research interests broadly concern communication of visual information from the perspectives of both signal processing and psychophysics. She was elected a Fellow of the IEEE in 2009 for her for contributions to robust and perceptual image and video communications. Hemami has held various visiting positions, most recently at the University of Nantes, France and at Ecole Polytechnique Federale de Lausanne, Switzerland. She has received numerous university and national teaching awards, including Eta Kappa Nu's C. Holmes MacDonald Award. She served as Vice-President Publications Products and Services, IEEE (2015-2016). She was a Distinguished Lecturer for the IEEE Signal Processing Society in 2010-11, was editor-in-chief for the IEEE Transactions on Multimedia from 2008-10. She has held various technical leadership positions in the IEEE.