SUMO-RL Environment

Integration of traffic signal control with the OpenEnv framework via SUMO (Simulation of Urban MObility) and SUMO-RL.

Overview

This environment enables reinforcement learning for traffic signal control using SUMO, a microscopic traffic simulation package. Train RL agents to optimize traffic light timing and minimize vehicle delays.

Key Features:

• Realistic traffic simulation via SUMO

• Single-agent mode for single intersection control

• Configurable rewards (waiting time, queue, pressure, speed)

• Multiple networks supported (custom .net.xml and .rou.xml files)

• Docker-ready with pre-bundled example network

Quick Start

Using Docker (Recommended)

from envs.sumo_rl_env import SumoRLEnv, SumoAction

# Automatically starts container
env = SumoRLEnv.from_docker_image("sumo-rl-env:latest")

# Reset environment
result = env.reset()
print(f"Observation shape: {result.observation.observation_shape}")
print(f"Available actions: {result.observation.action_mask}")

# Take action (select next green phase)
result = env.step(SumoAction(phase_id=1))
print(f"Reward: {result.reward}, Done: {result.done}")

# Get state
state = env.state()
print(f"Simulation time: {state.sim_time}")
print(f"Total vehicles: {state.total_vehicles}")
print(f"Mean waiting time: {state.mean_waiting_time}")

# Cleanup
env.close()

Building the Docker Image

cd OpenEnv

# Build base image first (if not already built)
docker build -t envtorch-base:latest -f src/openenv/core/containers/images/Dockerfile .

# Build SUMO-RL environment
docker build -f envs/sumo_rl_env/server/Dockerfile -t sumo-rl-env:latest .

Running with Different Configurations

# Default: single-intersection
docker run -p 8000:8000 sumo-rl-env:latest

# Longer simulation
docker run -p 8000:8000 \
  -e SUMO_NUM_SECONDS=50000 \
  sumo-rl-env:latest

# Different reward function
docker run -p 8000:8000 \
  -e SUMO_REWARD_FN=queue \
  sumo-rl-env:latest

# Custom seed for reproducibility
docker run -p 8000:8000 \
  -e SUMO_SEED=123 \
  sumo-rl-env:latest

Observation

The observation is a vector containing:

• Phase one-hot: Current active green phase (one-hot encoded)

• Min green flag: Binary indicator if minimum green time has passed

• Lane densities: Number of vehicles / lane capacity for each incoming lane

• Lane queues: Number of queued vehicles / lane capacity for each incoming lane

Observation size varies by network topology (depends on number of phases and lanes).

Default (single-intersection):

• 4 green phases

• 8 incoming lanes

• Observation size: ~21 elements

Action Space

The action space is discrete and represents selecting the next green phase to activate.

• Action type: Discrete

• Action range: [0, num_green_phases - 1]

• Default (single-intersection): 4 actions (one per green phase)

When a phase change is requested, SUMO automatically inserts a yellow phase before switching.

Rewards

Default reward function is change in cumulative waiting time:

reward = -(total_waiting_time_now - total_waiting_time_previous)

Positive rewards indicate waiting time decreased (good).

Available Reward Functions

Set via SUMO_REWARD_FN environment variable:

• diff-waiting-time (default): Change in cumulative waiting time

• average-speed: Average speed of all vehicles

• queue: Negative total queue length

• pressure: Pressure metric (incoming - outgoing vehicles)

Configuration

Environment Variables

Variable	Default	Description
SUMO_NET_FILE	/app/nets/single-intersection.net.xml	Network topology file
SUMO_ROUTE_FILE	/app/nets/single-intersection.rou.xml	Vehicle routes file
SUMO_NUM_SECONDS	20000	Simulation duration (seconds)
SUMO_DELTA_TIME	5	Seconds between agent actions
SUMO_YELLOW_TIME	2	Yellow phase duration (seconds)
SUMO_MIN_GREEN	5	Minimum green time (seconds)
SUMO_MAX_GREEN	50	Maximum green time (seconds)
SUMO_REWARD_FN	diff-waiting-time	Reward function name
SUMO_SEED	42	Random seed (use for reproducibility)

Using Custom Networks

To use your own SUMO network:

from envs.sumo_rl_env import SumoRLEnv

env = SumoRLEnv.from_docker_image(
    "sumo-rl-env:latest",
    volumes={
        "/path/to/your/nets": {"bind": "/nets", "mode": "ro"}
    },
    environment={
        "SUMO_NET_FILE": "/nets/my-network.net.xml",
        "SUMO_ROUTE_FILE": "/nets/my-routes.rou.xml",
    }
)

Your network directory should contain:

• .net.xml - Network topology (roads, junctions, traffic lights)

• .rou.xml - Vehicle routes (trip definitions, flow rates)

API Reference

SumoAction

@dataclass
class SumoAction(Action):
    phase_id: int  # Green phase to activate (0 to num_phases-1)
    ts_id: str = "0"  # Traffic signal ID (for multi-agent)

SumoObservation

@dataclass
class SumoObservation(Observation):
    observation: List[float]  # Observation vector
    observation_shape: List[int]  # Shape for reshaping
    action_mask: List[int]  # Valid action indices
    sim_time: float  # Current simulation time
    done: bool  # Episode finished
    reward: Optional[float]  # Reward from last action
    metadata: Dict  # System metrics

SumoState

@dataclass
class SumoState(State):
    episode_id: str  # Unique episode ID
    step_count: int  # Steps taken
    net_file: str  # Network file path
    route_file: str  # Route file path
    sim_time: float  # Current simulation time
    total_vehicles: int  # Total vehicles in simulation
    total_waiting_time: float  # Cumulative waiting time
    mean_waiting_time: float  # Mean waiting time
    mean_speed: float  # Mean vehicle speed
    # ... configuration parameters

Example Training Loop

from envs.sumo_rl_env import SumoRLEnv, SumoAction
import numpy as np

# Start environment
env = SumoRLEnv.from_docker_image("sumo-rl-env:latest")

# Training loop
for episode in range(10):
    result = env.reset()
    episode_reward = 0
    steps = 0

    while not result.done and steps < 1000:
        # Random policy (replace with your RL agent)
        action_id = np.random.choice(result.observation.action_mask)

        # Take action
        result = env.step(SumoAction(phase_id=int(action_id)))

        episode_reward += result.reward or 0
        steps += 1

        # Print progress every 100 steps
        if steps % 100 == 0:
            state = env.state()
            print(f"Step {steps}: "
                  f"reward={result.reward:.2f}, "
                  f"vehicles={state.total_vehicles}, "
                  f"waiting={state.mean_waiting_time:.2f}")

    print(f"Episode {episode}: total_reward={episode_reward:.2f}, steps={steps}")

env.close()

Performance Notes

Simulation Speed

• Reset time: 1-5 seconds (starts new SUMO simulation)

• Step time: ~50-200ms per step (depends on network size)

• Episode duration: Minutes (20,000 sim seconds with delta_time=5 → ~4,000 steps)

Optimization

For faster simulation:

1. Reduce SUMO_NUM_SECONDS for shorter episodes

2. Increase SUMO_DELTA_TIME for fewer decisions

3. Use simpler networks with fewer vehicles

Architecture

┌─────────────────────────────────┐
│ Client: SumoRLEnv               │
│  .step(phase_id=1)              │
└──────────────┬──────────────────┘
               │ HTTP
┌──────────────▼──────────────────┐
│ FastAPI Server (Docker)         │
│   SumoEnvironment               │
│     ├─ Wraps sumo_rl           │
│     ├─ Single-agent mode       │
│     └─ No GUI                  │
└──────────────┬──────────────────┘
               │
┌──────────────▼──────────────────┐
│ SUMO Simulator                  │
│  - Reads .net.xml (network)     │
│  - Reads .rou.xml (routes)      │
│  - Simulates traffic flow       │
│  - Provides observations        │
└─────────────────────────────────┘

Bundled Network

The default single-intersection network is a simple 4-way intersection with:

• 4 incoming roads (North, South, East, West)

• 4 green phases (NS straight, NS left, EW straight, EW left)

• Vehicle flow: Continuous stream with varying rates

Limitations

• No GUI in Docker: SUMO GUI requires X server (not available in containers)

• Single-agent only: Multi-agent (multiple intersections) coming in future version

• Fixed network per container: Each container uses one network topology

• Memory usage: ~500MB for small networks, 2-4GB for large city networks

Troubleshooting

Container won’t start

# Check logs
docker logs <container-id>

# Verify network files exist
docker run sumo-rl-env:latest ls -la /app/nets/

“SUMO_HOME not set” error

This should be automatic in Docker. If running locally:

export SUMO_HOME=/usr/share/sumo

Slow performance

• Reduce simulation duration: SUMO_NUM_SECONDS=5000

• Increase action interval: SUMO_DELTA_TIME=10

• Use smaller networks with fewer vehicles

References

• SUMO Documentation

• SUMO-RL GitHub

• SUMO-RL Paper

• RESCO Benchmarks

Citation

If you use SUMO-RL in your research, please cite:

@misc{sumorl,
    author = {Lucas N. Alegre},
    title = {{SUMO-RL}},
    year = {2019},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/LucasAlegre/sumo-rl}},
}

License

This integration is licensed under the BSD-style license. SUMO-RL and SUMO have their own licenses.