💽 CDPruner

Beyond Attention or Similarity: Maximizing Conditional
Diversity for Token Pruning in MLLMs

Qizhe Zhang1,2, Mengzhen Liu1, Lichen Li1, Ming Lu1,
Yuan Zhang1, Junwen Pan2, Qi She2, Shanghang Zhang1,✉
✉Corresponding Author
1 School of Computer Science, Peking University
2 ByteDance
Paper Code

Abstract

In multimodal large language models (MLLMs), the length of input visual tokens is often significantly greater than that of their textual counterparts, leading to a high inference cost. Many works aim to address this issue by removing redundant visual tokens. However, current approaches either rely on attention-based pruning, which retains numerous duplicate tokens, or use similarity-based pruning, overlooking the instruction relevance, consequently causing suboptimal performance. In this paper, we go beyond attention or similarity by proposing a novel visual token pruning method named CDPruner, which maximizes the conditional diversity of retained tokens. We first define the conditional similarity between visual tokens conditioned on the instruction, and then reformulate the token pruning problem with determinantal point process (DPP) to maximize the conditional diversity of the selected subset. The proposed CDPruner is training-free and model-agnostic, allowing easy application to various MLLMs. Extensive experiments across diverse MLLMs show that CDPruner establishes new state-of-the-art on various vision-language benchmarks. By maximizing conditional diversity through DPP, the selected subset better represents the input images while closely adhering to user instructions, thereby preserving strong performance even with high reduction ratios. When applied to LLaVA, CDPruner reduces FLOPs by 95% and CUDA latency by 78%, while maintaining 94% of the original accuracy. Our code is available at https://github.com/Theia-4869/CDPruner.

Maximizing Conditional Diversity for Token Pruning

Attention-based methods retain numerous duplicate tokens, failing to achieve effective visual token compression. Similarity-based methods neglect user instructions, always pruning the same tokens and paying insufficient attention to relevant regions. Our CDPruner considers the conditional diversity of the selected subset, dynamically adjusting pruning according to the user instructions and retaining maximal visual information.

Instruction Relevance

We introduce instruction relevance as a condition to achieve dynamic pruning, which is calculated as the cosine similarity between each visual token and the user instruction. The visualization of relevance scores is shown as follows:

Conditional DPP

We first calculate the similarity between visual tokens conditioned on their relevance to the current instruction. Then, CDPruner uses a DPP to select the subset to keep. As a training-free and model-agnostic method, it ensures both the diversity and quality of the selected token subset, significantly reducing computational cost while maintaining considerable performance.

Performance Comparison

We validate CDPruner against different types of existing methods across various MLLM architectures on comprehensive multi-modal benchmarks, including general VQA, text-oriented VQA and video understanding tasks.

LLaVA-1.5-7B

LLaVA-1.5-13B

LLaVA-NeXT-7B

LLaVA-NeXT-13B

LLaVA-Video-7B

Qwen2.5-VL-7B

Efficiency Comparison

To demonstrate the efficiency of CDPruner, we conduct a comparative analysis against other pruning methods in terms of FLOPs, CUDA latency, KV cache, and GPU memory on the high-resolution MLLM LLaVA-NeXT. All experiments are performed on a single NVIDIA A100-80GB GPU.

LLaVA-NeXT-7B

LLaVA-NeXT-13B

Contact

If you have any questions, please feel free to contact us:

BibTeX


@article{zhang2025cdpruner,
  title={Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs},
  author={Zhang, Qizhe and Liu, Mengzhen and Li, Lichen and Lu, Ming and Zhang, Yuan and Pan, Junwen and She, Qi and Zhang, Shanghang},
  journal={arXiv preprint arXiv:2506.10967},
  year={2025}
}