Congratualtions!

Our paper has been accepted to the IEEE Trans. Circuits and Systems for Video Technology (TCSVT) (JCR Top 5.5%, Impact Factor: 8.3) [LINK]

  • Title: MSCoTDet: Language-Driven Multi-Modal Fusion for Improved Multispectral Pedestrian Detection

  • Authors: Taeheon Kim, Sangyun Chung, Damin Yeom, Youngjoon Yu, and Hak Gu Kim, and Yong Man Ro

  • Abstract: Multispectral pedestrian detection is attractive for around-the-clock applications due to the complementary information between RGB and thermal modalities. However, current models often fail to detect pedestrians in certain cases (e.g., thermal-obscured pedestrians), particularly due to the modality bias learned from statistically biased datasets. In this paper, we investigate how to mitigate modality bias in multispectral pedestrian detection using a Large Language Model (LLM). Accordingly, we design a Multispectral Chain-of-Thought (MSCoT) prompting strategy, which prompts the LLM to perform multispectral pedestrian detection. Moreover, we propose a novel Multispectral Chain-of-Thought Detection (MSCoTDet) framework that integrates MSCoT prompting into multispectral pedestrian detection. To this end, we design a Language-driven Multi-modal Fusion (LMF) strategy that enables fusing the outputs of MSCoT prompting with the detection results of vision-based multispectral pedestrian detection models. Extensive experiments validate that MSCoTDet effectively mitigates modality biases and improves multispectral pedestrian detection.