LeTFuser for Autonomous Driving with Multi-Task Learning
LeTFuser presents an end-to-end autonomous driving architecture that fuses camera and LiDAR through a transformer-based late fusion strategy with multi-task supervision for robust urban navigation. The systematic evaluation across multiple task heads - waypoint prediction, traffic light recognition, hazard detection - demonstrates how multi-modal systems can be benchmarked for reliability across heterogeneous outputs. Published at CVPR Workshop on Autonomous Vehicles 2023.
End-to-end autonomous driving systems must simultaneously handle perception (what is in the scene?), prediction (what will other agents do?), and planning (what should the vehicle do?). LeTFuser addresses this by combining transformer-based sensor fusion with multi-task supervision across all three capabilities.
Architecture
LeTFuser uses a late-fusion strategy: camera and LiDAR streams are processed by separate encoder branches before being fused at the feature level by a cross-modal transformer. This preserves modality-specific structure (spatial resolution for cameras, 3D geometry for LiDAR) while enabling the model to learn which modality is more informative for each sub-task.
Multi-task heads:
- Waypoint prediction: future trajectory points for the vehicle path planner
- Traffic light recognition: state classification (red/green/yellow/off) with spatial grounding
- Hazard detection: pedestrian and vehicle bounding boxes with uncertainty estimates
All three heads share the fused backbone but are trained with task-specific losses, weighted by a gradient normalization scheme that prevents any single task from dominating the shared representation.
Evaluation
Evaluated on the CARLA urban driving simulator using the standard Town benchmark. LeTFuser achieves competitive route completion and infraction rates compared to prior state-of-the-art methods while providing richer intermediate outputs - the multi-task heads produce interpretable intermediate representations that can be logged and audited separately from the final driving action.
This interpretability property is operationally important: a system that can explain which perception sub-task influenced a particular decision supports the kind of post-hoc incident analysis required in safety-critical deployment.