Research Highlight | Vision–Language-Model-Guided Differentiable Ray Tracing – Future Communications Research & Development Programme

We are pleased to share that FCP research project: Vision–Language-Model-Guided Differentiable Ray Tracing for Fast and Accurate Multi-Material RF Parameter Estimation has been selected for support by the NVIDIA Academic Grant Program.

The fidelity of ray tracing (RT) based wireless digital twins depends on accurate material parameters. However, measuring and continuously updating these parameters is costly and difficult to scale. In our recent work, presented at AAAI’26 ML4Wireless, we investigate how vision language models (VLMs) can address this challenge.

By providing scene images with ITU-R P.2040 material parameter recommendation in context, a pre-trained VLM (without any fine tuning) can estimate multiple material parameters in parallel, refine them, and recommend additional measurement locations that maximize target reflections in NVIDIA Sionna RT and Isaac Sim. This enables scalable and efficient calibration of wireless digital twins.

Building on these results, our project has been selected for support by the NVIDIA Academic Grant Program, with two RTX Pro 6000 GPUs awarded to further advance this direction and extend it toward Physical AI and AI-RAN applications.