Policy Gradient Reinforcement Learning

Policy Gradient Methods are a class of reinforcement learning algorithms that directly optimize the policy function by performing gradient ascent on the expected return. Unlike value-based methods (like Q-learning), policy gradient methods directly parameterize and learn the policy, making them well-suited for continuous action spaces and stochastic policies.

Key Algorithms:

• REINFORCE (Monte Carlo Policy Gradient): The most basic policy gradient algorithm that uses complete episode returns to update the policy. The core update is based on the likelihood ratio policy gradient:
  ∇_θJ(θ) = E_τ~p(τ|θ)[∑_t ∇_θlog π_θ(a_t|s_t) · R_t]
  where τ is a trajectory, θ are the policy parameters, π_θ is the policy, and R_t is the return at time t.
• Actor-Critic: Combines policy gradient with value function approximation to reduce variance. The critic (value network) estimates the value function, while the actor (policy network) learns the policy using advantage estimates:
  ∇_θJ(θ) = E_τ~p(τ|θ)[∑_t ∇_θlog π_θ(a_t|s_t) · A_t]
  where A_t is the advantage estimate, typically calculated as r_t + γV(s_t+1) - V(s_t).
• Proximal Policy Optimization (PPO): A family of policy gradient methods that use a clipped objective function to ensure policy updates aren't too large, improving stability:
  L^CLIP(θ) = E_t[min(r_t(θ)A_t, clip(r_t(θ), 1-ε, 1+ε)A_t)]
  where r_t(θ) is the probability ratio between the new and old policy.

Key Components:

1. Policy Network (Actor): Neural network that maps states to action probabilities, defining the agent's behavior
2. Value Network (Critic): In actor-critic methods, estimates the value function to reduce variance in policy updates
3. Trajectory Collection: Sample actions from the policy and collect state-action-reward sequences
4. Return Calculation: Compute discounted returns for each step in the trajectory
5. Policy Gradient Update: Update policy parameters to increase probability of actions that led to high returns
6. Entropy Regularization: Encourage exploration by adding an entropy bonus to the objective

Advantages of Policy Gradient Methods:

• Naturally handle continuous action spaces
• Can learn stochastic policies, which are important for exploration and games with hidden information
• Policy updates are more stable with appropriate optimization techniques
• Can directly optimize for the objective of interest
• Convergence properties are often better understood theoretically

Challenges and Solutions:

• High Variance: REINFORCE suffers from high variance in gradient estimates. Actor-critic methods reduce variance through value function bootstrapping.
• Sample Efficiency: Policy gradient methods can be sample-inefficient. Solutions include importance sampling and off-policy learning.
• Step Size Selection: The policy update can be sensitive to step size. Trust region methods like PPO constrain update sizes.
• Credit Assignment: It can be difficult to attribute rewards to specific actions. Techniques like reward shaping and advantage functions help.