Demo
Demo Video

Autonomous navigation in dynamic environments is a significant challenge for robotics and AI, with applications in self-driving vehicles, delivery drones, and robotic exploration. Traditional Reinforcement Learning (RL) approaches require carefully designed reward functions and extensive labeled data, which can lead to undesired behaviors and hinder scalability.

In contrast, semi-supervised and self-supervised learning approaches can efficiently leverage partially labeled or unlabeled data to extract robust representations for downstream RL tasks. This project integrates self-supervised representation learning with curiosity-driven RL to address policy learning in dynamic environments, specifically using the CARLA simulator.

Learning effective driving policies in urban environments is complex due to:

• Multi-modal sensor inputs (semantic, depth, LiDAR, metadata).
• Limited labeled datasets for training.
• Environmental variability in weather, lighting, and dynamic obstacles.

Our solution leverages a semi-supervised RL framework that incorporates curiosity-driven exploration and self-supervised learning to improve adaptability and decision-making efficiency.

The CARLA simulator is used to generate a multi-modal dataset with synchronized sensor data and diverse environmental conditions:

• Input Modalities:
  • RGB images (semantic segmentation).
  • Depth images (normalized to 1 channel).
  • LiDAR data (top-down 3-channel projection).
  • Metadata (autopilot commands, weather, GPS).
• Dynamic Scenarios:
  • Weather variations (rain, fog, night, sunny).
  • Obstacles, traffic lights, and pedestrians.
  • Multiple urban maps for testing generalization.

The dataset is preprocessed to align multi-modal inputs, making it suitable for representation learning and policy optimization.

System Overview
System Overview

flowchart TD
 subgraph Inputs
 sem[/"Semantic (1, 420, 680)"/]
 depth[/"Depth (1, 420, 680)"/]
 lidar[/"LiDAR (3, 200, 200)"/]
 meta[/"Metadata (vehicle state, weather)"/]
 end
 subgraph "Semantic Encoder"
 sem --> sem1["Conv2D(1→64) + BN + ReLU"]
 sem1 --> sem2["MaxPool"]
 sem2 --> sem3["ResBlock(64)"]
 sem3 --> sem4["Conv2D(64→128) + BN + ReLU"]
 sem4 --> sem5["ResBlock(128)"]
 sem5 --> sem6["Conv2D(128→256) + BN + ReLU"]
 sem6 --> sem7["ResBlock(256)"]
 sem7 --> sem8["AdaptiveAvgPool + Flatten"]
 sem8 --> sem9["Linear(256→feature_dim)"]
 end
 subgraph "Depth Encoder"
 depth --> d1["Conv2D(1→64) + BN + ReLU"]
 d1 --> d2["MaxPool"]
 d2 --> d3["ResBlock(64)"]
 d3 --> d4["Conv2D(64→128) + BN + ReLU"]
 d4 --> d5["ResBlock(128)"]
 d5 --> d6["Conv2D(128→256) + BN + ReLU"]
 d6 --> d7["ResBlock(256)"]
 d7 --> d8["AdaptiveAvgPool + Flatten"]
 d8 --> d9["Linear(256→feature_dim)"]
 end
 subgraph "LiDAR Encoder"
 lidar --> l1["Conv2D(3→64) + BN + ReLU"]
 l1 --> l2["ResBlock(64)"]
 l2 --> l3["Conv2D(64→128) + BN + ReLU"]
 l3 --> l4["ResBlock(128)"]
 l4 --> l5["Conv2D(128→256) + BN + ReLU"]
 l5 --> l6["ResBlock(256)"]
 l6 --> l7["AdaptiveAvgPool + Flatten"]
 l7 --> l8["Linear(256→feature_dim)"]
 end
 subgraph "Metadata Encoder"
 meta --> m1["Linear(30→256)"]
 m1 --> m2["LayerNorm + ReLU + Dropout"]
 m2 --> m3["Linear(256→512)"]
 m3 --> m4["LayerNorm + ReLU + Dropout"]
 m4 --> m5["Linear(512→feature_dim)"]
 end
 subgraph "Temporal Processing"
 sem9 --> t1["GRU + LayerNorm"]
 d9 --> t2["GRU + LayerNorm"]
 l8 --> t3["GRU + LayerNorm"]
 m5 --> t4["GRU + LayerNorm"]
 end
 subgraph "Feature Fusion"
 t1 & t2 & t3 & t4 --> f1["Concatenate"]
 f1 --> f2["Linear(×ばつfeature_dim→feature_di×ばつ2)"]
 f2 --> f3["LayerNorm + ReLU + Dropout"]
 f3 --> f4["Linear(feature_di×ばつ2→feature_dim)"]
 f4 --> f5["LayerNorm"]
 end
 subgraph "Temporal Fusion"
 f5 --> tf["GRU + LayerNorm"]
 end
 subgraph "Decoders"
 tf --> dec1["Semantic Decoder\nLinear→LayerNorm→ReLU→Linear→Unflatten"]
 tf --> dec2["Depth Decoder\nLinear→LayerNorm→ReLU→Linear→Unflatten"]
 tf --> dec3["LiDAR Decoder\nLinear→LayerNorm→ReLU→Linear→Unflatten"]
 end
 dec1 --> out1["Semantic Reconstruction"]
 dec2 --> out2["Depth Reconstruction"]
 dec3 --> out3["LiDAR Reconstruction"]

flowchart TD
 %% Inputs
 input[/"Encoder Input\n(semantic, depth, lidar, metadata)"/]---enc[("Observation\nTemporal Encoder")]
 enc---state["Encoded State\n(feature_dim)"]
 %% Policy Network Branch
 state---p1["Linear(feature_dim→hidden_dim)"]
 p1---p2["LayerNorm"]
 p2---p3["Mish"]
 p3---p4["Linear(hidden_dim→hidden_dim)"]
 p4---p5["LayerNorm"]
 p5---p6["Mish"]
 p6---p7["Linear(hidden_dim→action_dim)"]
 p7---p8["Softmax"]
 p8---actions["Action Probabilities"]
 %% Value Network Branch
 state---v1["Linear(feature_dim→hidden_dim)"]
 v1---v2["LayerNorm"]
 v2---v3["Mish"]
 v3---v4["Linear(hidden_dim→hidden_dim)"]
 v4---v5["LayerNorm"]
 v5---v6["Mish"]
 v6---v7["Linear(hidden_dim→1)"]
 v7---value["State Value"]
 %% Adaptive Entropy
 state---ent["Entropy Coefficient\n(learnable parameter)"]
 ent-.->actions
 %% Outputs and Losses
 actions---dist["Categorical Distribution"]
 dist---samp["Sample Actions"]
 dist---lp["Log Probabilities"]
 value---returns["Value Estimates"]
 %% Losses
 samp & lp & returns---loss["PPO Loss Components:\n- Policy Loss\n- Value Loss\n- Entropy Bonus"]
 %% Styling
 classDef input fill:#e1f5fe,stroke:#01579b
 classDef output fill:#e8f5e9,stroke:#1b5e20
 classDef encoder fill:#fff3e0,stroke:#e65100
 classDef loss fill:#fce4ec,stroke:#880e4f
 class input input
 class actions,value output
 class enc encoder
 class loss loss
 %% Layout hints
 subgraph "Policy Network"
 p1
 p2
 p3
 p4
 p5
 p6
 p7
 p8
 end
 subgraph "Value Network"
 v1
 v2
 v3
 v4
 v5
 v6
 v7
 end

flowchart TD
 %% Inputs
 s1[/"Current State\n(state_dim)"/]---norm1["LayerNorm"]
 s2[/"Next State\n(state_dim)"/]---norm2["LayerNorm"]
 act[/"Action\n(action_dim)"/]---emb["Action\nEmbedding"]
 %% Forward Model Path
 norm1 & emb---concat1["Concatenate"]
 concat1---f1["Linear(state+action→hidden)"]
 f1---f2["LayerNorm"]
 f2---f3["ReLU"]
 f3---f4["Linear(hidden→hidden)"]
 f4---f5["LayerNorm"]
 f5---f6["ReLU"]
 f6---f7["Linear(hidden→state)"]
 f7---pred["Predicted Next State"]
 %% Inverse Model Path
 norm1 & norm2---concat2["Concatenate"]
 concat2---i1["Linear(×ばつstate→hidden)"]
 i1---i2["LayerNorm"]
 i2---i3["ReLU"]
 i3---i4["Linear(hidden→hidden)"]
 i4---i5["LayerNorm"]
 i5---i6["ReLU"]
 i6---i7["Linear(hidden→action)"]
 i7---pred_act["Predicted Action"]
 %% Loss Computation
 pred & norm2---fwd_loss["Forward Loss (MSE)"]
 pred_act & act---inv_loss["Inverse Loss (CrossEntropy)"]
 fwd_loss & inv_loss---total["Total Loss\n×ばつforward + (1-β)×ばつinverse"]
 %% Reward Generation
 pred & norm2---rew["Intrinsic Reward\n×ばつprediction error"]
 %% Styling
 classDef input fill:#e1f5fe,stroke:#01579b
 classDef output fill:#e8f5e9,stroke:#1b5e20
 classDef loss fill:#fce4ec,stroke:#880e4f
 classDef reward fill:#f3e5f5,stroke:#4a148c
 class s1,s2,act input
 class pred,pred_act output
 class fwd_loss,inv_loss,total loss
 class rew reward
 %% Layout hints
 subgraph "Forward Model"
 direction TB
 f1
 f2
 f3
 f4
 f5
 f6
 f7
 end
 subgraph "Inverse Model"
 direction TB
 i1
 i2
 i3
 i4
 i5
 i6
 i7
 end

We use an ObservationTemporalEncoder that processes input sequences across semantic, depth, LiDAR, and metadata channels. The encoder consists of:

• Convolutional and Residual blocks for sensor data.
• Temporal GRU blocks for spatial-temporal features.
• A self-supervised loss combining reconstruction, consistency, compactness, and temporal smoothness.

We initialize the RL policy using Proximal Policy Optimization (PPO) combined with an Intrinsic Curiosity Module (ICM):

• PPO: Ensures stable policy updates using clipped objectives.
• ICM: Generates intrinsic rewards based on prediction errors in state transitions, encouraging exploration of less predictable states.

The pre-training phase uses Behavioral Cloning (BC) loss to accelerate initial learning.

After pre-training, agents are fine-tuned in altered CARLA environments (e.g., new maps, weather conditions). The PPO-ICM framework continues to optimize the policy through real-time interactions with dynamic conditions.

The PPO objective is defined as: [ L^{CLIP}(\theta) = \mathbb{E}_t[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)], ] where ( \hat{A}_t ) is the Generalized Advantage Estimate.

We use the Driving Score Metric defined as: [ \text{Driving Score} = R_i P_i, ] where:

• ( R_i ): Route completion percentage.
• ( P_i ): Weighted infraction penalties (e.g., pedestrian collisions, stop sign violations).

• The ObservationTemporalEncoder achieved a total test loss of 0.0118.
• Pre-training was intentionally halted at 40% BC loss to focus on exploration efficiency.
• After 1000 episodes of RL training, the average Driving Score reached 61.6% on unseen test maps with altered conditions, calculated using CARLA Leaderboard metrics.

Observation Temporal Encoder Loss Curve
Observation Temporal Encoder Loss Curve

Pre Training Loss Curve
Pre Training Loss Curve

Pre Training Loss Curve
Pre Training Loss Curve

Our findings demonstrate that self-supervised representation learning, combined with curiosity-driven RL, enables agents to adapt to dynamic environments with improved generalization. Key contributions include:

1. Semi-supervised latent feature learning for multi-modal sensor data.
2. Curiosity-driven exploration that reduces reliance on external rewards.
3. Robust performance in unseen environments with dynamic obstacles.

• Explore deeper temporal models for improved sequence processing.
• Incorporate additional intrinsic reward schemes (e.g., count-based exploration).
• Scale experiments to larger CARLA maps with greater variability.

1. Codevilla, Felipe, et al. "End-to-end driving via conditional imitation learning." ICRA 2018.
2. Jing, Longlong, and Yann LeCun. "Self-supervised learning for video: A survey." TPAMI 2020.
3. Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction." CVPR 2017 Workshop.
4. Schulman, John, et al. "Proximal policy optimization algorithms." arXiv:1707.06347, 2017.
5. Pomerleau, Dean A. "ALVINN: An autonomous land vehicle in a neural network." NIPS 1989.
6. Kendall, Alex, et al. "Learning to drive in a day." ICRA 2019.
7. Dosovitskiy, Alexey, et al. "CARLA: An open urban driving simulator." CoRL 2017.
8. Liang, Xiaodan, et al. "CIRL: Controllable imitative reinforcement learning for vision-based self-driving." ECCV 2018.
9. Chen, Dian, et al. "Learning to drive from a world on rails." ICCV 2020.
10. Yu, Tianhe, et al. "COMBO: Conservative offline model-based policy optimization." NeurIPS 2021.
11. CARLA Leaderboard Case Study - ALPHA DRIVE, 2024.
12. CARLA Documentation: Driving Benchmark Performance Metrics, 2024.
13. CARLA Autonomous Driving Leaderboard Documentation, 2024.
14. Papers With Code - CARLA Leaderboard Benchmark (Autonomous Driving). Papers With Code, 2024.
15. Yeh, Paul. "Motion Planning on CARLA." GitHub repository, 2020.
16. Pathak, Deepak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. "Curiosity-driven Exploration by Self-supervised Prediction." PMLR, 2017.
17. Zhang, X., et al. "Trajectory-Guided Control Prediction for End-to-End Autonomous Driving." arXiv preprint arXiv:2206.08129v2, 2022.
18. Wu, Penghao, et al. "Trajectory-guided Control Prediction for End-to-end Autonomous Driving." GitHub repository, 2022.
19. Shaikh, Idrees. "Autonomous Driving in CARLA using Deep Reinforcement Learning." GitHub repository, 2021.
20. RobeSafe-UAH. "DQN-CARLA." GitHub repository, 2021.
21. Minelli, John. "Carla-Gym: Multi-Agent RL Interface for CARLA." GitHub repository, 2022.
22. Luca96. "CARLA Driving RL Agent." GitHub repository, 2021.