DMFuser: Distilled Multi-Task Learning for Transformer-Based Multi-modal Fusion
DMFuser applies knowledge distillation to multi-modal sensor fusion for autonomous driving, demonstrating that complex multi-task architectures can be compressed without proportional loss in accuracy. The systematic evaluation across multiple task heads - waypoint prediction, semantic segmentation, object detection - validates that shared representations can be maintained reliably across heterogeneous outputs. This work underpins Fact AI Lab's approach to reliability engineering for multi-output AI systems.
DMFuser addresses a fundamental challenge in end-to-end autonomous driving: how to fuse heterogeneous sensor inputs while simultaneously optimizing for multiple driving tasks without catastrophic forgetting.
Problem
Multi-task learning for autonomous driving faces a tension: tasks like waypoint prediction, semantic segmentation, and object detection benefit from shared representations, but naive joint training often causes one task to dominate gradient updates at the expense of others. This gets worse with heterogeneous sensor modalities, where the optimal representation for RGB features differs from the optimal representation for 3D point cloud features.
Architecture
DMFuser introduces a distilled multi-task transformer with three key components:
Cross-modal attention: Transformer attention blocks that align camera and LiDAR features at multiple spatial scales, producing a fused representation that preserves modality-specific information while enabling cross-modal reasoning.
Task-specific heads with shared backbone: A single transformer backbone processes the fused features, with lightweight task-specific decoders for each output. This limits parameter count while maintaining task performance.
Knowledge distillation: A teacher model trained with privileged information (ground-truth 3D annotations) guides the student model during training, improving data efficiency and generalization to sensor noise.
Results
On the CARLA autonomous driving benchmark, DMFuser achieves state-of-the-art performance on driving score while reducing model parameters by 40% compared to the teacher. The distillation step is particularly effective for tasks with sparse training signal (e.g., rare event detection), where the teacher can transfer structure that would take much longer to learn from data alone.