Human-level control through deep reinforcement learning

Reinforcement learning is a type of machine learning where an agent learns to make decisions by interacting with its environment and receiving feedback in the form of rewards or penalties. Just like a child learns not to touch fire by experiencing the consequences, or a dog learns to balance a ball for a treat, reinforcement learning focuses on learning from actions and outcomes. Through trial and error, the agent refines its behavior to maximize rewards, ultimately developing strategies to solve complex tasks.

Reinforcement learning can be broken down into key components: The agent is the decision-maker that interacts with the environment. The environment provides feedback in response to the agent’s actions. The state represents the current situation observed by the agent, and actions are the possible decisions the agent can take in any given state. Rewards are the feedback signals indicating the quality of the action. The policy defines the agent's strategy for selecting actions based on the current state, with the overall goal being to maximize cumulative rewards over time. This interaction forms a feedback loop where actions influence the environment and, in turn, affect the agent's learning process.

In reinforcement learning, the agent's goal is to choose actions in each state that maximize the cumulative future reward. This is represented mathematically as the sum of immediate and discounted future rewards, where the discount factor determines the weight of future rewards compared to immediate ones. The action-value function, denoted as Q-star, represents the maximum expected cumulative reward achievable from a given state and action when following the optimal policy. The policy itself, denoted by pi, maps states to actions and defines the agent's strategy for decision-making.

In traditional reinforcement learning, experts manually design features based on their understanding of the problem. However, this approach has significant limitations. Handcrafted features are often task-specific and struggle to generalize to new problems or environments. Designing these features is also time-consuming and labor-intensive, making the process inefficient. Moreover, handcrafted features may fail to capture the complexity of high-dimensional data, such as raw pixels or unstructured data, limiting the agent's ability to effectively learn and perform in complex scenarios.

The Deep Q-Network, or DQN, introduced in this paper provides a solution to the limitations of traditional reinforcement learning. It eliminates the need for handcrafted features by directly processing raw sensory inputs, such as raw pixel images. The DQN leverages deep convolutional neural networks to automatically learn hierarchical feature representations that are relevant to the task, enabling more efficient and generalized learning from complex, high-dimensional data.

In traditional reinforcement learning, the action-value function, Q of s and a, represents the expected cumulative reward for starting in a given state, taking a specific action, and then following a specific policy. This Q-value is calculated and stored as a table for every possible state-action pair.

The Q-learning algorithm updates these Q-values using the Bellman equation. However, this tabular representation faces a major limitation known as state explosion. As the state-action space grows large, such as with continuous states or high-dimensional inputs like images, maintaining and updating the table becomes computationally infeasible.

The Deep Q-Network, or DQN, addresses this scalability challenge by approximating Q-values using a neural network, making it suitable for more complex and higher-dimensional problems.

First, in normal reinforcement learning, Q-values are stored explicitly in a table for all state-action pairs, while DQNs use a neural network to approximate Q-values with weights, allowing for more efficient representation.

Second, scalability is a major distinction. Traditional RL methods are limited to small state-action spaces, whereas DQNs can scale to handle large and high-dimensional spaces, such as image-based inputs.

Third, normal RL relies on handcrafted features, requiring domain expertise to design them. DQNs, on the other hand, automatically learn features directly from raw sensory inputs, like pixels.

Finally, the learning process differs. Normal RL updates Q-values directly using the Bellman equation. In contrast, DQNs minimize a loss function during neural network training to optimize their performance.

In reinforcement learning, using neural networks can lead to instability for several reasons.

First, correlations in observations arise because the agent interacts with the environment step by step. For example, in a video game, consecutive frames are highly similar, leading to strongly correlated data, which can hinder effective training.

Second, action-value instability occurs because the policy, which decides actions based on Q-values, changes as the neural network learns and updates Q-values. This constant change in the data distribution destabilizes the training process, as the network must continuously adapt to shifting data.

Lastly, prediction instability stems from the Q-learning update process. The target value, calculated using the Bellman equation, relies on the same neural network being updated. This feedback loop, where the network predicts and updates its own targets, can cause instability during training.

The main solution to the instability problem in reinforcement learning is to use two techniques: Experience Replay and the Target Network. Experience Replay ensures that the neural network is not trained on consecutive, correlated experiences. Instead, all experiences—state, action, reward, and next state—are stored in a replay buffer. During training, experiences are sampled randomly, which breaks the correlation and ensures diverse training data. This way, the network gets a broader perspective on the environment, leading to more stable updates. The Target Network addresses instability by introducing a separate, less frequently updated network for generating target Q-values. Unlike the main network, which updates frequently, the target network is updated only after a fixed number of steps. This separation means that the predicted Q-values and the target Q-values are calculated using different networks, preventing the network from chasing a constantly moving target and stabilizing training. Together, these techniques ensure smoother and more stable learning, effectively addressing the common instabilities in reinforcement learning with neural networks.

The Deep Q-Network optimizes its learning process using a loss function, which measures the error between the predicted Q-value and the target value. This loss is minimized using stochastic gradient descent to update the network weights. The target value, denoted as 'y,' is calculated as the immediate reward (r) plus the discounted maximum Q-value of the next state, which is determined using the target network. The discount factor (gamma) balances the importance of immediate versus future rewards. By using two networks—the main network for predictions and the target network for computing stable target values—this approach avoids the feedback loop issues. These networks are periodically synchronized, ensuring stability and reliable updates. This process ensures that the network converges to more accurate Q-values while maintaining robustness in learning.

The algorithm incorporates a balance between exploration and exploitation to ensure effective learning. During the training process, a random action may be selected with a certain probability, which is part of the exploration strategy. Exploration allows the agent to try actions that might not appear optimal at the moment but could lead to better results in the long run. This approach ensures the agent doesn’t get stuck in local optima by always exploiting known good actions.

At every step, the agent records the state, action, reward, and next state into the replay buffer. Random minibatches from this buffer are then sampled to train the network. This randomness helps break correlations and provides diverse experiences for better learning.

After a fixed number of steps, the weights of the main network are transferred to the target network. This periodic update ensures the stability of the learning process, as the target network provides consistent Q-values for the training of the main network. This loop of exploration, recording, learning, and weight transfer enables the agent to continuously improve its policy.

In this project, teamwork played a significant role. Rishab and I collaborated closely to create the discussion slides, ensuring clarity and coherence. Rishab also provided valuable assistance in refining and enhancing the presentation slides