163
0
v1v2 (latest)

EffOWT: Transfer Visual Language Models to Open-World Tracking Efficiently and Effectively

Main:9 Pages
5 Figures
Bibliography:2 Pages
5 Tables
Abstract

Open-World Tracking (OWT) aims to track every object of any category, which requires the model to have strong generalization capabilities. Trackers can improve their generalization ability by leveraging Visual Language Models (VLMs). However, challenges arise with the fine-tuning strategies when VLMs are transferred to OWT: full fine-tuning results in excessive parameter and memory costs, while the zero-shot strategy leads to sub-optimal performance. To solve the problem, EffOWT is proposed for efficiently transferring VLMs to OWT. Specifically, we build a small and independent learnable side network outside the VLM backbone. By freezing the backbone and only executing backpropagation on the side network, the model's efficiency requirements can be met. In addition, EffOWT enhances the side network by proposing a hybrid structure of Transformer and CNN to improve the model's performance in the OWT field. Finally, we implement sparse interactions on the MLP, thus reducing parameter updates and memory costs significantly. Thanks to the proposed methods, EffOWT achieves an absolute gain of 5.5% on the tracking metric OWTA for unknown categories, while only updating 1.3% of the parameters compared to full fine-tuning, with a 36.4% memory saving. Other metrics also demonstrate obvious improvement.

View on arXiv
@article{wang2025_2504.05141,
  title={ EffOWT: Transfer Visual Language Models to Open-World Tracking Efficiently and Effectively },
  author={ Bingyang Wang and Kaer Huang and Bin Li and Yiqiang Yan and Lihe Zhang and Huchuan Lu and You He },
  journal={arXiv preprint arXiv:2504.05141},
  year={ 2025 }
}
Comments on this paper

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.