Research map / Vision-language and multimodal / Multimodal LLMs

SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SIF, a non-intrusive LVLM fingerprinting method that distills decoding-time text watermark signals into trigger images (plus worst-case representation perturbation training) so fingerprint responses stay in-distribution and survive quantization and fine-tuning, and introduces a Semantic Divergence Attack to expose prior fingerprints.

Problem

Ownership verification for publicly released large vision-language models: existing fingerprints use semantically abnormal queries or out-of-distribution responses that are easily detected and filtered, and they are fragile to quantization and fine-tuning.

Approach

Distill a unigram text-watermark (green-list logit bias) into an adversarially optimized input image using a watermark alignment loss on top-K truncated student distributions plus a cross-entropy semantic anchor to the watermarked teacher response, and harden it with gradient-aligned worst-case activation perturbations (RFO), so the frozen LVLM naturally emits coherent, watermarked answers.

Key results

• On LLaVA-1.5-7B under 4-bit and 8-bit quantization, SIF reaches FMR 0.49 and 0.89 versus 0.40 and 0.79 for the best prior baseline PLA.
• Under SDA, SIF retains FMR 0.86 on LLaVA-1.5-7B and 0.85 on Qwen2.5-VL-7B, while Proflingo and IF drop to 0 and PLA to 0.07-0.11.
• GPT-4.1 judge flags 86% of PLA interactions as suspicious versus only 4% of SIF interactions.

Surprising findings

• Existing fingerprints such as PLA, Proflingo and IF are largely removed by a simple reference-model-based semantic divergence filter, with near-zero FMR under SDA.
• PLA achieves FMR 0 on LlavaMix fine-tuning, suggesting adversarial shortcut fingerprints are erased by re-alignment on instruction data.

Limitations

• Fingerprint is a screening tool and not sufficient as legal proof on its own.
• Robustness to input perturbations is limited; JPEG and resize give FMR of only 0.18 and 0.23 on LLaVA-1.5-7B.
• Evaluation covers only two LVLM families and 7B-scale models; FMR under strong fine-tuning remains moderate (e.g., 0.31 on LlavaMix for LLaVA).

Experimental setup

Datasets: ImageNet 2012 validation AMBER VisionArena LlavaMix TikZ GUI-Actor ARC-AGI-1 V7W TextVQA Paintingform MathV

Benchmarks: LLaVA-1.5-7B and Qwen2.5-VL-7B under 4/8-bit quantization full fine-tuning variants (LlavaMix, TikZ, MathV, TextVQA, V7W, Paintingform, GUI-Actor, ARC-AGI-1) SDA stealthiness unrelated LVLMs reliability

Metrics: Fingerprint Matching Rate (FMR) perplexity FMR under SDA

Baselines: Ordinary attack DIM CroPA Proflingo IF RNA PLA

Builds on: PLA Proflingo Instruction Fingerprint Zhao et al. unigram watermark PGD

Closest papers by meaning

• AttnDiff: Attention-based Differential Fingerprinting for Large Language Models (NeurIPS 2026 · similarity 0.87)
• Vision-Language Model IP Protection via Prompt-based Learning (CVPR 2025 · similarity 0.86)
• WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.85)
• Scalable Fingerprinting of Large Language Models (NeurIPS 2025 · similarity 0.85)
• HuRef: HUman-REadable Fingerprint for Large Language Models (NeurIPS 2024 · similarity 0.85)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.85)