RM-MLLM: Robust Scene Understanding in Adverse Weather with RGB-Multispectral Fusion

Results in Engineering, 2026


Hyeongjin Ju, Sanghyeop Yeo, Incheol Park, Youngwan Jin, Yagiz Nalcakan, Shiho Kim

Seamless Trans-X Lab
Yonsei University
Paper
DENSE-QA example with synchronized RGB and multispectral images for six scene-understanding tasks.

DENSE-QA: An example of the DENSE-QA dataset for scene understanding in adverse driving conditions. (a) RGB and multispectral images are used as input, and (b) a total of six scene understanding tasks are structured in an instruction-response format.


RM-MLLM architecture with separate RGB and multispectral vision modules, fusion modules, a projector, and a language model.

RM-MLLM: An overview of the proposed RM-MLLM architecture. The model is trained using a progressive strategy consisting of three stages: pre-training, fusion, and instruction tuning. Synchronized RGB and multispectral images are passed through separate vision modules to be converted into visual tokens, which are then fused in N Fusion Modules using Multi-Head Cross Attention and MoE-FFN. The resulting fused tokens are passed through a Projector module and fed into a Large Language Model (LLM) fine-tuned with LoRA to generate a reasoning response.

Abstract

Multimodal large language models (MLLMs) for autonomous driving often become less reliable in adverse weather, partly because training data are dominated by clear, well-lit scenes, and most models primarily rely on RGB imagery. We address these issues with two contributions. First, we introduce DENSE-QA, a large-scale benchmark for adverse-weather scene understanding comprising 12,997 scenes and over 77,000 instruction-response pairs, built on synchronized RGB and multispectral imagery. Second, we propose RM-MLLM, an MLLM architecture with modality-specific encoders and an adaptive fusion module, trained with a three-stage progressive strategy to better integrate complementary sensor cues. Across multiple LLM backbones, RM-MLLM improves QA performance under adverse conditions. Using only RGB input, the proposed architecture and training already outperform all evaluated baselines, including prompt-augmented models, by over 11 percentage points. Incorporating RGB-multispectral fusion provides an additional 3-point gain concentrated on weather-sensitive tasks. We also observe that fine-tuning a strong RGB-only vision-language model on the same training data yields performance comparable to its zero-shot evaluation, suggesting that DENSE-QA fine-tuning alone does not close the gap to RM-MLLM and that multispectral input provides complementary value.

Details of DENSE-QA


Percentage distributions of DENSE-QA attributes for time of day, road-surface condition, weather, infrastructure, and illumination.

Percentage distributions for the five attribute groups used in our scene-understanding tasks: Time-of-Day (TOD), Road-Surface Condi- tion (RSC), Weather (WEA), Infrastructure (INF), and Illumination (ILL). The vertical and horizontal axes represent various tasks and class ratios, respectively.

Table of label counts for each task in the DENSE-QA dataset. Numeric values are shown in the figure.

Numerical distribution of labels for each task in the DENSE-QA dataset.

Quantitative Comparison with State-of-the-Art Models on the DENSE-QA Benchmark

Quantitative comparison with state-of-the-art models on the DENSE-QA benchmark.
We evaluate RM-MLLM against general-purpose VLMs, driving-specific MLLMs, a fine-tuned baseline (Qwen2-VL trained on the same DENSE-QA data), and an RGB-only ablation across six scene understanding tasks. An asterisk (*) denotes results from prompts augmented with a class list. All models are evaluated using identical category-based keyword matching. Average is computed over the six per-task accuracies (OBJ Total, TOD, RSC, WEA, INF, ILL).

Table comparing RM-MLLM with other models on the six DENSE-QA scene-understanding tasks. Numeric values are shown in the figure.

Qualitative Results of SOTA Models on the DENSE-QA Benchmark

Qualitative comparison on challenging DENSE-QA examples.
Two scenarios each for (a) RSC and (b) Weather. Colors: blue =RM-MLLM (ours), orange =fine-tuned (Qwen2-VL), gray =zero-shot VLMs.
RGB-only models are misled by ambiguous visual cues, while RM-MLLM correctly interprets the scenes using multispectral information.

Qualitative comparison on DENSE-QA road-surface and weather examples. Qualitative comparison on DENSE-QA time-of-day and illumination examples.

(c) Time-of-Day (TOD) and (d) Illumination (ILL)

Qualitative comparison on a DENSE-QA infrastructure example.

(e) Infrastructure (INF)


BibTeX

@article{JU2026RMMLLM,
  title = {RM-MLLM: Robust scene understanding in adverse weather with RGB-multispectral fusion},
  journal = {Results in Engineering},
  volume = {32},
  pages = {112317},
  year = {2026},
  issn = {2590-1230},
  doi = {10.1016/j.rineng.2026.112317},
  url = {https://doi.org/10.1016/j.rineng.2026.112317},
  author = {Hyeongjin Ju and Sanghyeop Yeo and Incheol Park and Youngwan Jin and Yagiz Nalcakan and Shiho Kim}
  }