Inventory Control with Deep Reinforcement Learning

Optimizing single-product inventory management using DQN and PPO in a custom Gymnasium environment.

Inventory management is a classic operations problem: a business needs to decide how much to order at each time period to meet uncertain customer demand, while keeping costs low. Order too little and you face stockouts (lost sales, penalties). Order too much and you pay for holding excess stock.

Traditional approaches like Economic Order Quantity (EOQ) rely on fixed assumptions about demand. This project takes a different approach — training a Reinforcement Learning agent that learns an ordering policy directly from experience, without needing to know the demand distribution in advance.

Two deep RL algorithms are implemented and compared:

• DQN (Deep Q-Network) — a value-based method that learns which action has the best long-term payoff
• PPO (Proximal Policy Optimization) — a policy-gradient method that directly learns a probability distribution over actions

Both agents are trained in a custom simulation environment built from scratch using the Gymnasium API, and evaluated on their ability to minimize total inventory costs over time.

At each time step, the agent observes the current state of the inventory system and decides how many units to order. Here is what happens in one step:

Agent observes state → [current inventory, last period demand]
 ↓
Agent picks an action → order quantity (0 to 50 units)
 ↓
Inventory is replenished, demand is sampled from Poisson(λ=20)
 ↓
Sales happen (up to available inventory)
 ↓
Costs are computed → holding cost + stockout penalty + order cost
 ↓
Reward = negative total cost (agent wants to maximize reward = minimize cost)
 ↓
Next state is returned → [new inventory, demand just seen]

This loop runs for 100 steps per episode. The agent improves its ordering strategy over many episodes by learning which states call for larger or smaller orders.

The inventory system is modeled as a Markov Decision Process:

The agent's goal is to find a policy that maximizes expected discounted return, which is equivalent to minimizing long-run inventory costs.

inventory-rl/
├── inventory_env.py # Custom Gymnasium environment (InventoryEnv)
├── train_dqn.py # DQN training script
├── train_ppo.py # PPO training script
├── evaluate.py # Model evaluation script
├── utils.py # Evaluation helper functions
├── test_env.py # Quick environment sanity check
├── requirements.txt # Python dependencies
└── results/ # Saved models & TensorBoard logs (auto-generated)
 ├── dqn/
 └── ppo/

A custom single-product, periodic-review inventory system with stochastic Poisson demand.

State: [current_inventory, last_period_demand]
Action: Order quantity ∈ {0, 1, ..., 50}
Reward: Negative total cost at each step

DQN learns a Q-function — an estimate of the total future reward for taking each action in a given state. It picks the action with the highest Q-value.

• Uses an experience replay buffer to decorrelate training samples
• Maintains a target network for stable Q-value targets
• Explores via ε-greedy, decaying from ε=1.0 to ε=0.05 over 20k steps
• Well-suited here because the action space (order quantities) is discrete

PPO is an actor-critic algorithm that directly learns a policy (actor) and a value function (critic). It uses a clipped objective to prevent overly large policy updates.

• Uses Generalized Advantage Estimation (GAE, λ=0.95) to reduce variance
• Clip range of 0.2 limits how much the policy can change in one update
• More stable across random seeds due to its conservative update mechanism
• Exploration is implicit through the stochastic policy output

git clone https://github.com/vpriyareddy12/inventory-rl.git
cd inventory-rl

python -m venv venv
source venv/bin/activate # Linux/macOS
venv\Scripts\activate # Windows

pip install -r requirements.txt

Run this first to verify the environment works correctly before training:

python test_env.py

python train_dqn.py --seed 0 --timesteps 100000 --outdir results/dqn

python train_ppo.py --seed 0 --timesteps 100000 --outdir results/ppo

Both scripts automatically save:

• best_model.zip — checkpoint with highest evaluation return during training
• final_model.zip — model at the end of training
• TensorBoard logs under results/<algo>/tensorboard/

# Evaluate DQN
python evaluate.py --model-path results/dqn/best_model --algo dqn --episodes 50
# Evaluate PPO
python evaluate.py --model-path results/ppo/best_model --algo ppo --episodes 50

Output includes mean return, standard deviation, and min/max across episodes.

tensorboard --logdir results --port 6006

Then open http://localhost:6006 in your browser. You can inspect reward curves, loss trends, exploration rate (DQN), and policy metrics (PPO).

Evaluation over 50 independent episodes:

Returns are negative costs — values closer to zero mean lower total cost and better performance.

Key findings:

• DQN achieves better final returns and converges faster, benefiting from its fit with discrete action spaces
• PPO's best checkpoint approaches DQN's performance with lower variance across seeds, indicating more consistent behavior
• Both agents successfully learn to balance ordering, holding, and stockout costs under stochastic demand

1. Mnih et al. (2015). Human-level control through deep reinforcement learning. Nature.
2. Schulman et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
3. Gijsbrechts et al. (2022). Can Deep Reinforcement Learning Improve Inventory Management? MSOM.
4. Alvo et al. (2023). Deep RL for Inventory Networks. arXiv:2306.11246.
5. Fan et al. (2019). A Theoretical Analysis of Deep Q-Learning. arXiv:1901.00137.