VoCo-LLaMA: Towards Vision Compression with Large Language Models

TL;DR

1. We propose VoCo-LLaMA, the first approach to compress vision tokens using LLMs. By fully utilizing the LLMs' understanding paradigm of vision tokens, our method can compress hundreds of vision tokens into a single VoCo token, while minimizing visual information loss.
2. VoCo-LLaMA demonstrates the ability to understand video through continuous training using time-series compressed token sequences of video frames.
3. VoCo-LLaMA presents a promising way to unlock the full potential of VLMs' contextual window.

Abstract

We propose VoCo-LLaMA, the first approach to compress vision tokens using LLMs. We strive to fully utilize the LLMs' understanding paradigm of vision tokens during the compression learning process. By introducing Vision Compression tokens during the vision instruction tuning phase and leveraging attention distillation, our method distill how LLMs comprehend vision tokens into their processing of VoCo tokens. VoCo-LLaMA facilitates effective vision compression and improves the computational efficiency during the inference stage. Specifically, our method achieves minimal performance loss with a compression ratio of 576 x, resulting in up to 94.8% fewer FLOPs and 69.6% acceleration in inference time. Furthermore, through continuous training using time-series compressed token sequences of video frames, VoCo-LLaMA demonstrates the ability to understand temporal correlations, outperforming previous methods on popular video question-answering benchmarks. Our approach presents a promising way to unlock the full potential of VLMs' contextual window, enabling more scalable multi-modal applications.

Vision Compression Design

Illustration of the VoCo-LLaMA framework. Based on standard VLMs (a), VoCo-LLaMA (b) first isolate visual and text tokens by injecting VoCo tokens, and then establishes a dedicated interaction pathway between the two modalities via VoCo tokens, enabling the distillation and compression of vision tokens into the transformer activations upon the compact VoCo tokens.
The compression process of VoCo-LLaMA can be elegantly implemented by strategically modifying the attention mask and learned through standard visual instruction tuning.

Vision Compression Results

Temporal Modeling

To further investigate the efficacy of our proposed method in handling video input, we utilize the time-series VoCo token sequence obtained by compressing video frames to further explore the temporal modeling capabilities of VoCo-LLaMA. Without any additional designs, VoCo-LLaMA outperforms existing vision compression methods on common video question-answering benchmarks, with each video frame compressed into an equal number of tokens.

Visualization

BibTeX

@InProceedings{Ye_2025_CVPR,
    author    = {Ye, Xubing and Gan, Yukang and Huang, Xiaoke and Ge, Yixiao and Tang, Yansong},
    title     = {VoCo-LLaMA: Towards Vision Compression with Large Language Models},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {29836-29846}
}

VoCo-LLaMA: Towards Vision Compression withLarge Language Models

CVPR 2025