Autonomous Driving System

📋 Table of Contents

1. Project Overview
2. System Architecture
3. Module Breakdown
4. Installation
5. Dataset Setup
6. Running the System
7. Training the RL Agent
8. Evaluation
9. Dashboard Usage
10. Evaluation Metrics
11. Results
12. Project Structure
13. Future Improvements
14. References

🎯 Project Overview

This project implements a complete, modular autonomous driving pipeline capable of:

🏗 System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        CARLA Simulator                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐  │
│  │  RGB Camera  │  │  Depth Cam   │  │  LiDAR / GNSS / IMU  │  │
│  └──────┬───────┘  └──────┬───────┘  └──────────┬───────────┘  │
└─────────┼─────────────────┼────────────────────  ┼─────────────┘
          │                 │                       │
          ▼                 ▼                       ▼
┌─────────────────────────────────────────────────────────────────┐
│                      PERCEPTION LAYER                            │
│  ┌───────────────┐ ┌──────────────┐ ┌──────────┐ ┌──────────┐ │
│  │ Lane Detection│ │Object Detect │ │Sign Detec│ │  Depth   │ │
│  │ (Hough+CV2)   │ │ (YOLOv11)    │ │(YOLO+CNN)│ │ (MiDaS)  │ │
│  └───────┬───────┘ └──────┬───────┘ └────┬─────┘ └────┬─────┘ │
└──────────┼────────────────┼──────────────┼─────────────┼───────┘
           │                │              │             │
           └────────────────┼──────────────┼─────────────┘
                            ▼              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      PLANNING LAYER                              │
│  ┌──────────────────┐      ┌──────────────────────────────────┐ │
│  │  Decision Engine │      │      Behavior Planner (FSM)      │ │
│  │  (Rule-based)    │      │  IDLE→LANE_FOLLOWING→INTERSECT.. │ │
│  └────────┬─────────┘      └──────────────┬───────────────────┘ │
│           │        ┌───────────────────────┘                     │
│           ▼        ▼                                             │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  Path Planner: A* Global + Pure Pursuit Local            │   │
│  └────────────────────────────┬─────────────────────────────┘   │
└───────────────────────────────┼─────────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      CONTROL LAYER                               │
│  ┌───────────────────┐  ┌──────────────────┐  ┌─────────────┐  │
│  │ Steering (PID+    │  │  Throttle (PID)  │  │  AEB        │  │
│  │ Stanley)          │  │  Speed control   │  │  Emergency  │  │
│  └───────────────────┘  └──────────────────┘  └─────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                ▼
                    ┌───────────────────────┐
                    │   RL Agent (PPO)      │ ← Optional end-to-end
                    │   Stable-Baselines3   │
                    └───────────────────────┘
                                ▼
                    ┌───────────────────────┐
                    │ Streamlit Dashboard   │
                    │ (Live monitoring)     │
                    └───────────────────────┘

🔩 Module Breakdown

Perception (perception/)

Planning (planning/)

Control (control/)

Simulation (simulation/)

RL Agent (rl_agent/)

⚙️ Installation

Prerequisites

• Ubuntu 20.04+ or Windows 10/11
• Python 3.11+
• NVIDIA GPU with CUDA 11.8+ (recommended; CPU fallback available)
• CARLA 0.9.15 (download)

Step 1 — Clone & Create Environment

git clone https://github.com/yourusername/autonomous-driving-system.git
cd autonomous-driving-system

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

Step 2 — Install Python Dependencies

pip install --upgrade pip
pip install -r requirements.txt

Step 3 — Install CARLA Python Client

# For CARLA 0.9.15
pip install carla==0.9.15

# OR install from the CARLA PythonAPI egg (if server is local)
cd /opt/carla/PythonAPI/carla/dist
pip install carla-0.9.15-cp311-cp311-linux_x86_64.whl

Step 4 — Download YOLO Weights

# YOLOv8n (auto-downloads on first run)
python -c "from ultralytics import YOLO; YOLO('yolov8n.pt')"

# Or for YOLOv11:
python -c "from ultralytics import YOLO; YOLO('yolo11n.pt')"

Step 5 — Verify Installation

python main.py --mode benchmark

📦 Dataset Setup

CARLA — No dataset download needed

The CARLA simulator generates synthetic sensor data in real-time. Start the server and the system auto-collects training data.

Optional: Pre-trained Sign Classifier

For enhanced traffic-sign classification, train the CNN on:

# Prepare dataset
mkdir -p datasets/traffic_signs
# Download and extract to datasets/traffic_signs/
# Expected structure:
#   datasets/traffic_signs/
#     train/  (class_0/, class_1/, ...)
#     val/
#     test/

Train the sign classifier:

python scripts/train_sign_classifier.py \
  --data datasets/traffic_signs \
  --epochs 50 \
  --output models/sign_classifier.pt

🚀 Running the System

1. Start CARLA Server (separate terminal)

# Linux
cd /opt/carla
./CarlaUE4.sh -carla-server -fps=20 -ResX=1280 -ResY=720

# Windows
CarlaUE4.exe -carla-server -fps=20

2. Rule-Based Agent

# Default straight road scenario
python main.py --mode rule --scenario STRAIGHT_ROAD

# Urban traffic scenario
python main.py --mode rule --scenario DENSE_URBAN --steps 3000

# Night driving
python main.py --mode rule --scenario NIGHT_DRIVING --device cuda

3. RL Agent Inference

python main.py --mode rl \
  --model models/ppo_autonomous_driver_final \
  --scenario INTERSECTION

4. All 5 Scenarios (Sequential)

for SCENARIO in STRAIGHT_ROAD INTERSECTION PEDESTRIAN_CROSS DENSE_URBAN NIGHT_DRIVING; do
  python main.py --mode rule --scenario $SCENARIO --steps 1000
done

🧠 Training the RL Agent

# Quick training (testing)
python main.py --mode train --timesteps 50000 --scenario STRAIGHT_ROAD

# Full training
python main.py --mode train --timesteps 500000 --scenario DENSE_URBAN

# OR via the training script directly (more options)
python rl_agent/train.py \
  --timesteps 500000 \
  --scenario INTERSECTION

# Resume from checkpoint
python rl_agent/train.py \
  --resume models/ppo_autonomous_driver_100000_steps.zip \
  --timesteps 400000

# Monitor with TensorBoard
tensorboard --logdir results/rl_logs

Reward Function

📊 Evaluation

# Evaluate all 5 scenarios (5 episodes each)
python main.py --mode eval \
  --model models/ppo_autonomous_driver_final \
  --episodes 5

# Evaluate single scenario
python rl_agent/evaluate.py \
  --model models/ppo_autonomous_driver_final \
  --scenario INTERSECTION \
  --episodes 10

# Perception latency benchmark
python main.py --mode benchmark

# View results
cat results/evaluation_report.json

Evaluation output: results/evaluation_report.json, results/evaluation_metrics.csv,
results/episode_details.csv

📺 Dashboard Usage

streamlit run dashboard/app.py
# Opens at http://localhost:8501

Dashboard Features

📈 Evaluation Metrics

📁 Project Structure

autonomous_driving/
│
├── perception/                 # Sensing & interpretation
│   ├── lane_detection.py       # Hough-transform lane detector
│   ├── object_detection.py     # YOLOv11 multi-class detector
│   ├── traffic_sign_detection.py # YOLO + CNN sign classifier
│   └── depth_estimation.py    # MiDaS monocular depth
│
├── planning/                   # Intelligence layer
│   ├── decision_engine.py      # Rule-based priority engine
│   ├── behavior_planner.py     # FSM behavior planner
│   └── path_planner.py         # A* + Pure Pursuit planner
│
├── control/                    # Actuation
│   ├── pid_controller.py       # Generic PID controller
│   └── steering_control.py     # Steering + Throttle + AEB
│
├── simulation/                 # CARLA interface
│   ├── carla_env.py            # Gymnasium environment
│   └── sensors.py              # Sensor manager
│
├── rl_agent/                   # Reinforcement learning
│   ├── train.py                # PPO training pipeline
│   └── evaluate.py             # Evaluation & benchmarking
│
├── dashboard/                  # Monitoring
│   └── app.py                  # Streamlit dashboard
│
├── configs/
│   └── config.yaml             # Central configuration
│
├── models/                     # Trained weights
├── datasets/                   # Training data
├── results/                    # Logs, reports, checkpoints
│
├── main.py                     # System orchestrator + CLI
├── requirements.txt
└── README.md

🔮 Future Improvements

• Semantic Segmentation — Replace Hough lanes with DeepLabV3+/SegFormer drivable-area mask
• Transformer Perception — DETR / BEVFormer for multi-view object detection
• Multi-Camera Fusion — 360° surround-view with BEV projection
• LiDAR Integration — PointPillar / VoxelNet for LiDAR object detection
• HD Map Integration — OpenDRIVE map parsing for precise path planning
• V2X Communication — Vehicle-to-infrastructure signal simulation
• Driver Monitoring — Drowsiness / distraction detection
• Imitation Learning — Pre-training from expert CARLA demonstrations
• Uncertainty Estimation — Bayesian detection confidence for safety
• Real-world Deployment — ROS2 + Autoware.Universe integration

📚 References

1. Dosovitskiy et al. (2017) — CARLA: An Open Urban Driving Simulator
2. Redmon et al. — YOLO: Real-Time Object Detection
3. Ranftl et al. (2021) — Vision Transformers for Dense Prediction (MiDaS)
4. Schulman et al. (2017) — Proximal Policy Optimization
5. Coulter (1992) — Pure Pursuit Path Tracking
6. Hart, Nilsson, Raphael (1968) — A* Search Algorithm

👨‍💻 Author

Built as an AI/Autonomous Systems Internship Capstone Project.
Demonstrates mastery of: deep learning · computer vision · robotics · reinforcement learning · simulation · software engineering.

Tech Stack: Python · PyTorch · OpenCV · Ultralytics YOLOv11 · MiDaS · Stable-Baselines3 · CARLA · Streamlit · NumPy · Pandas

⭐ Star this repo if it helped you! · 🍴 Fork it for your own project