Excel to HTML
WEDNESDAY 4 MARCH 2026
Applications of Multimodal Models
Session Chair: Rui Shen, Nvidia
11:00 - 12:20
Grand Peninsula F
11:00GENAI-174
Seeing more with less: Variational prompting for vision foundation models, Xi Xiao, University of Alabama at Birmingham (US); Gaofei Chen, University of Alabama at Birmingham (US); Zhuxuanzi Wang, Cornell University (US); Tianyang Wang, University of Alabama at Birmingham (US); Xiao Wang, Oak Ridge National Laboratory (US) [view abstract]
Parameter-efficient fine-tuning (PEFT) has emerged as a crucial approach for adapting large vision transformers to downstream tasks without the prohibitive computational costs of full fine-tuning. While existing visual prompt tuning (VPT) methods have made significant strides, they predominantly rely on static, domain-specific prompts that fail to capture the rich visual diversity within individual instances. This paper introduces V2APT (Visual Variational Autoencoder Prompt Tuning), a novel framework that generates dynamic, input-dependent prompts using a variational autoencoder architecture. By learning a latent representation of image-specific features and decoding them into customized prompts, V2APT adapts to the unique visual characteristics of each input. Extensive experiments on FGVC, HTA, and VTAB-1k benchmarks demonstrate that our approach consistently outperforms state-of-the-art PEFT methods. Notably, V2APT achieves +3.2% improvement over VPT-Deep on HTA, with an average performance gain of +2.0% across all three datasets.
11:20GENAI-175
Vision-language learning for wireless capsule endoscopy: Diagnostic captioning with CLIP, Lu Xu, Norwegian University of Science and Technology (Norway); Anuja Vats, Norwegian University of Science and Technology (Norway); Marius Pedersen, Norwegian University of Science and Technology (Norway); Kiran Raja, Norwegian University of Science and Technology (Norway) [view abstract]
Wireless Capsule Endoscopy (WCE) is a minimally invasive diagnostic tool for examining the gastrointestinal tract, but the interpretation of large amounts of WCE image data demands extensive manual efforts and expert knowledge. Deep learning offers a promising approach to automate WCE data analysis, but training robust models is hindered by the scarcity of large-scale, high-quality labeled data in the WCE domain. This study explores the use of Contrastive Language-Image Pre-training (CLIP), a vision-language model pre-trained on extensive image-text pairs, to address these challenges in deep learning for WCE. We focus on caption retrieval and pathology classification tasks, using the CAPTIV8 dataset, a multi-modal WCE dataset containing image-diagnostic text pairs. After customizing the dataset for deep learning tasks, we conducted experiments comparing CLIP with state-of-the-art vision models. The results demonstrated that CLIP performs better than vision-only models, particularly in small-sample regimes such as one-shot and few-shot setups. By replacing the original CLIP loss with a KL-divergence loss, we further enhanced the model s ability to handle multiple positive pairs in a mini-batch during the training, to further attune learning for this specific medical domain.
11:40GENAI-176
From joints to bones: Expanding motion representations for comprehensive human movement understanding in language models, Lawrence Amadi, Futurewei Technologies (US); Chih-Hsien Chou, Futurewei Technologies (US); Ning Lu, Futurewei Technologies (US) [view abstract]
Large language models (LLMs) have advanced in reasoning across complex domains and are increasingly extended beyond text to multimodal learning. Recent efforts to integrate motion understanding into LLMs typically rely on a single representation of human movement, most often based on the SMPL model encoding of joint positions and rotations. While effective, this representation limits versatility in motion generation and restricts fine-grained understanding of compound movements. We propose a multimodal approach that combines complementary motion representations to provide LLMs with richer perspectives on human motion. In particular, we introduce a representation based on changes in bone orientations and displacements, enabling motion data to be sourced not only from cameras and motion-capture systems but also from inertial sensors. This approach enhances the interpretability and versatile compositionality of generated motions while broadening the suite of motion modalities accessible to LLMs. Our work expands the capacity of LLMs to understand and generate human motion, with implications for downstream tasks established in related works, in addition to promising new downstream tasks such as exercise form analysis and coaching.
12:00GENAI-177
Shelfie: A vision-language system for structured product understanding in real-world consumer environments, Ankur Purwar, Procter and Gamble (Singapore); Alexander Hollingworth, Procter and Gamble (Singapore); Laveena Satwani, Big Vision LLC (India); Eu Jack Tan, Procter and Gamble (Singapore); Nandita Mishra, Big Vision LLC (India); Pranav Mishra, Big Vision LLC (India) [view abstract]
Personalized consumer experiences increasingly depend on understanding actual product usage in everyday settings. We present Shelfie, a consumer-centric vision-language system that extracts structured product metadata from user-submitted images of lifestyle care products arranged in real-world contexts - such as shelves, countertops, and vanity spaces. Unlike conventional systems developed for controlled retail environments and dependent on barcode scanning, Shelfie is intentionally designed to operate effectively in cluttered, unconstrained home settings and is fully barcode independent. Shelfie integrates object detection, instance segmentation, and large language model (LLM)-based reasoning to infer rich metadata for each visible product. This includes product brand, name, category, form, package type, ingredients, benefits, and size. The Shelfie system is trained and validated on a diverse user-sourced dataset covering personal care to home lifestyle products, demonstrating strong generalization in producing high-accuracy, highly structured output across packaging styles and product categories. Shelfie establishes a vision-language foundation for real-world consumer-facing product understanding and discovery systems. It can enable downstream applications such as community-driven recommendation systems, ingredient sensitivity tracking, and in-depth consumer behaviour analysis, all while keeping consumer habits, needs, and convenience at the center. By bridging visual input with structured metadata output, Shelfie can enable more informed, personalized decisions through peer-driven insights.
Image Generation
Session Chair: Rui Shen, Nvidia
15:30 - 16:30
Grand Peninsula F
15:30GENAI-181
Motion-adaptive temporal attention for lightweight video generation with stable diffusion, Rui Hong, George Mason University (US); Tianma Shen, Santa Clara University; Shuxue Quan, Samsung Research America (US) [view abstract]
Generating temporally consistent videos from text prompts using image diffusion models remains challenging due to flickering artifacts and motion inconsistencies. While recent video diffusion models improve temporal coherence through large-scale retraining, lightweight extensions of pretrained models like Stable Diffusion are highly desirable. We propose a motion-adaptive temporal framework with three key contributions: (1) parameter-efficient temporal attention layers trained via LoRA, adding less than 1% trainable parameters; (2) motion-adaptive attention that dynamically adjusts temporal receptive fields based on estimated motion magnitude using long-range dependencies for static scenes and short-range adaptation for dynamic content; and (3) self-supervised frame interpolation during training, where randomly masked frames provide additional temporal smoothness supervision. By adapting temporal modeling to motion characteristics, our method generates 8-16 frame animations with significantly improved consistency compared to frame-independent generation. Experiments on WebVid demonstrate that motion-adaptive attention outperforms fixed-window approaches, while our lightweight design achieves comparable quality to heavier video diffusion models with substantially reduced computational requirements.
15:50GENAI-182
When synthetic faces fail: Exploring LoRA fine-tuning limitations for age-specific face generation, Julian Goetzinger, Fraunhofer SIT | ATHENE Center (Germany); Raphael Frick, Fraunhofer SIT | ATHENE Center (Germany) [view abstract]
Nowadays, a large quantity of the material circulating online are in the form of images and videos. Some of this media contains explicit material. To determine if minors appear in such content, age estimation systems can be used to analyze facial features and assess the likely age of individuals depicted. Facial age estimation systems require diverse training data across all age groups, yet existing datasets exhibit significant demographic biases and pose various privacy and copyright concerns. In this paper, we evaluate the trustworthiness of Low-Rank Adaptation (LoRA) fine-tuned text-to-image models for generating age-and gender-specific synthetic faces to address these limitations. Adapters trained on a small subset of images derived from age-estimation datasets are used to generate a balanced synthetic dataset consisting of 29,850 images. Our experiments revealed that across multiple datasets synthetic-only training yields markedly higher error (MAE 15 20 years) than real-image baselines. This is likely caused by severe imbalances in the training data distribution, combined with the limitations of low-rank adaptation techniques, such as LoRA, to learn the nuanced patterns of facial aging. These findings caution against current synthetic-face pipelines in privacy-sensitive deployments and motivate methods that improve privacy, while being reliable in terms of accuracy and robustness.
16:10GENAI-183
Texture learning for semantically consistent unpaired image translation, Sherif Nekkah, (Singapore); Artem Savkin; Federico Tombari [view abstract]
Deep learning has transformed computer vision but struggles with safety-critical applications due to data limitations. This work tackles the challenge with a cost-effective approach, leveraging unsupervised domain adaptation using procedurally generated synthetic depth and semantic information instead of expensive rendering pipelines. A novel constraint improves semantic consistency by ensuring uniform textures within semantic classes, even across significant domain gaps. This approach enables scalable generation of realistic and semantically consistent traffic scenes.
Image Quality
Session Chair: Shuxue Quan, Samsung (US)
16:30 - 17:30
Grand Peninsula F
16:30GENAI-178
Psychovisual experimentation using LLMs as observers, Robin Jenkin, NVIDIA (US); Preeti Pillai, KLE Tech (India); Aruna Nayak, KLE Tech (India); Abhishek Joshi, KLE Tech (India); Vasudhaika S, KLE Tech (India); Sinchana C, KLE Tech (India); Abhishek Patil, KLE Tech (India) [view abstract]
With the rapid progress of large language models (LLMs), there is growing interest in whether such models can act as judges of image quality. A fundamental question exists however, as to the ability of such models to distinguish between various levels of image quality attributes, such as sharpness and noise. This work represents one of the first systematic investigations of LLMs as evaluators in classical paired comparison IQA experiments. Prior work in LLM-based vision has focused on captioning or recognition tasks, whereas our study explicitly frames the Gemini 2.0 Flash as a proxy human subject in psychovisual testing to establish JNDs for sharpness and noise using the Kodak image quality ruler dataset as stimulus.For both sharpness and noise, the magnitude of JNDs were found to be proportional to the absolute quality of the stimulus. Additionally, judgements of individual pairs of images were found to be somewhat probabilistic rather than absolute, with more uncertainty observed for sharpness discrimination than noise. Prompt engineering is detailed as is the statistical analysis of results.Understanding the extent to which LLMs can act as reliable perceptual proxies offers transformative implications for automated IQA, dataset labeling, and adaptive imaging pipelines.
16:50GENAI-179
Adapting DeQA-score for attribute-specific portrait quality assessment, Yujin Cho, DXOMARK Image Labs (France); Minh Khang Tran, (France); Benoit Pochon, DXOMARK (France); Jean-Michel Morel, Lingnan University (France); Gabriele Facciolo, ENS Paris-Saclay (France); Sira Ferradans, DXOMARK [view abstract]
With the growing adoption of multimodal large language models (MLLMs) for image quality assessment, Vision Language IQA systems such as DeQA-Score have demonstrated strong correlation with human judgments on natural images. However, current MLLM-based quality predictors primarily provide global image quality scores and therefore lack the ability to quantitively assess specific perceptual attributes such as noise, texture, contrast, and color factors that are essential for explainability and camera tuning. In this work, we extend DeQA-Score from global Mean Opinion Score (MOS)-based quality prediction to attribute-specific, Just-Objectionable-Difference (JOD)-based portrait assessment. Our study investigates how a MOS-trained model behaves when exposed to pairwise-annotated data and how lightweight adaptation can achieve perceptual alignment at the attribute level. Using a controlled mannequin dataset, we analyze the model s baseline behavior under different prompt strategies and spatial input configurations, revealing limited attribute sensitivity. We then apply LoRA fine-tuning on realistic portrait data annotated for texture and noise quality. The adapted model achieved correlation of SRCC = 0.91/0.93, PLCC = 0.91/0.90 with JOD scores for noise and texture, respectively. Subsequent analysis confirms that the vision encoder is the main contributor to perceptual learning. The proposed framework establishes an efficient path for converting global VLM-based IQA models into attribute-aware, perceptually aligned assessors for real-world photography.
17:10GENAI-180
A quantitative framework for evaluating color-name understanding in generative AI models, Robin Jenkin, NVIDIA (US); Vijayalaxmi M, KLE Tech (India); Shailesh Pawale, KLE Tech (India); Francis Fernandes, KLE Tech (India); Ashutosh Naryagol, KLE Tech (India); Salman Sanadi, KLE Tech (India) [view abstract]
With the proliferation of text-to-image generative AI, understanding the fidelity of their output is critical. While these models can generate visually stunning images, their interpretation of nuanced, subjective concepts like color names remains largely unquantified. This paper introduces a systematic framework to evaluate how accurately leading generative AI models (including Flux, Ideogram, Kandinsky, and Stable Diffusion) understand and reproduce colors from textual prompts. We prompted these models with both one-word (e.g., "blue") and two-word (e.g., "sky blue") color names to generate uniform color fields. The resulting images were analyzed by converting them to the perceptually uniform CIE Lab color space. An adaptive k-means clustering algorithm was employed to extract the dominant color, mitigating issues of non-uniformity in the generated images. By calculating the perceptual color difference (?E) between the AI-generated colors and standardized ground-truth values, we provide a quantitative benchmark of each model's color accuracy. Our findings reveal significant performance variations among models and highlight systemic challenges, such as higher color deviation for ambiguous one-word prompts compared to more specific two-word prompts. This work provides a robust methodology for auditing and improving color fidelity in future generative models.