Robot Navigation using Reinforcement Learning

Thu, 01 Dec 2022 00:00:00 +0000

Built a custom OpenAI Gym environment inspired by Turtlebot3 to train robot navigation policies from scratch, and implemented and compared three reinforcement learning algorithms (DQN, DDPG, and TD3) across both discrete and continuous action spaces.

Key Contributions:

Custom Environment Design: Built a Gym-compatible navigation environment (world.py) from scratch, including a map-to-grayscale conversion pipeline (via OpenCV) to distinguish navigable space from walls and obstacles, and registered it as a proper Gym environment (World-v1 with obstacles, World-v2 without).
Reward Shaping: Designed a two-component reward function combining a distance reward (inverse of Euclidean distance to target) and a bearing reward (alignment between robot heading and target direction), balancing goal-seeking behavior with orientation control.
Algorithm Implementation & Comparison: Implemented and trained three RL algorithms, DQN (discrete action space) and DDPG/TD3 (continuous action space, controlling linear and angular velocity), to solve the navigation task under identical environment conditions.

Environment Details:

Observation space: robot pose (x, y, θ) and target position (x, y)
Action space: linear velocity (v) and angular velocity (w), continuous for DDPG/TD3
Randomized start/target poses per episode, constrained to avoid walls, obstacles, and overlapping start/goal positions
Motion modeled via an integration approximation for pose updates given (v, w)

Methods Used:

Deep Q-Network (DQN) for discrete action-space navigation
Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3) for continuous action-space navigation
Custom Gym environment with distance + bearing reward shaping
Computer vision-based map preprocessing (grayscale conversion for navigable-space detection)

Software Used: Python, OpenAI Gym, Stable-Baselines3, OpenCV, Pillow

Results (validated in simulation):

This video shows the robot navigatin to a random target location, trained using RL policy.

Human Iris Detection

Wed, 01 Dec 2021 00:00:00 +0000

Implemented a CNN-based pupil localization pipeline in Python, reproducing the methodology from the paper “Accurate Eye Pupil Localization Using Heterogeneous CNN Models”.

Key Contributions:

Model Implementation: Reproduced a heterogeneous CNN architecture for pupil detection, adapting the paper’s design to a practical Python/PyTorch workflow.
Data Pipeline: Built preprocessing and augmentation routines to prepare eye image datasets for training and evaluation.
Evaluation: Tracked train/test loss across training and validated predicted vs. ground-truth iris centers on held-out test images.

Methods Used:

Dilation and Gaussian blurring for target (label) preprocessing
Convolutional autoencoder as the model architecture
Grid search across optimizers, loss functions, and activations to find the best combination:
- optimizers: [SGD, Adam, Adamax, RMSprop]
- loss functions: [MSE, MAE]
- activations: [Tanh, ReLU, Sigmoid]
- best combination found: Adam + MSE + Tanh

Software Used: OpenCV, PyTorch, NumPy, Pandas, scikit-learn

Sample Result:

These images show the heat map and predections generated by the model.

CV |

Robot Navigation using Reinforcement Learning

Human Iris Detection