EcoLight: Reward Shaping in DRL for Environment Friendly Traffic Signal Control
EcoLight introduces reward shaping for traffic signal control that balances vehicle throughput with CO2 emissions, demonstrating that multi-objective real-world constraints can be embedded directly in the training objective rather than patched in post-hoc. Presented at the NeurIPS 2021 Workshop on Tackling Climate Change with Machine Learning, this work established the reward shaping methodology that subsequent work scaled to multi-intersection networks.
Traffic signal control is a domain where AI decisions have direct, measurable environmental consequences. Standard DRL approaches optimize for throughput metrics - queue length, wait time, average speed - that correlate with emissions but do not directly measure them. EcoLight asks: what happens when you put the actual metric you care about (CO2 reduction) directly in the reward function?
Reward Shaping Approach
EcoLight combines a throughput term with an emissions term in the training reward. The emissions estimate uses the HBEFA (Handbook of Emission Factors for Road Transport) model integrated into the SUMO traffic simulator, providing per-vehicle CO2 estimates at each timestep based on vehicle type, speed, and acceleration.
The combined reward: R = (1 - alpha) * throughput_reward + alpha * (-emissions_penalty)
where alpha is a weighting parameter that allows explicit configuration of the emissions-throughput trade-off at deployment time.
Why Explicit Metrics Matter
The core contribution is methodological: embedding the target metric (emissions) directly in training, rather than optimizing a proxy (throughput) and hoping emissions improve as a side effect. The alpha parameter makes the trade-off explicit and configurable - operators can set it based on local regulatory requirements without retraining.
This principle - optimize for what you actually measure, make trade-offs explicit and configurable - is central to how Fact AI Lab designs AI systems for regulated industries.
Results
On a single-intersection SUMO simulation, EcoLight achieves 18% reduction in CO2 emissions compared to fixed-time baselines and 12% compared to throughput-only DRL, with a throughput reduction of 3% at alpha=0.5. The explicit alpha parameter allowed these exact numbers to be reported, rather than conflating the two objectives.