# 1. DQN Algorithms

PPO is motivated by the following question: how can we take the biggest possible improvement step on a policy using the data we currently have, without stepping so far that we accidentally cause performance collapse? The main idea is that after an update, the new policy should be not too far form the old policy. For that, PPO (as implemented in Stable Baselines) uses clipping to avoid too large update.

### Quick Facts

* DQN is an off-policy algorithm.
* DQN can only be used for environments with discrete action spaces.
* The Stable Baselines implementation of PPO follows the OpenAI made for GPU. For multiprocessing, it uses vectorized environment.

### Key Equations

Our aim will be to train a policy that tries to maximize the discounted, cumulative reward
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, where $R_{t_0}$ is also known as the *return*. The discount, $\gamma$, should be a constant between $0$ and $1$ that ensures the sum converges. It makes rewards from the uncertain far future less important for our agent than the ones in the near future that it can be fairly confident about.

The main idea behind Q-learning is that if we had a function $Q^*: State \times Action \rightarrow \mathbb{R}$, that could tell us what our return would be, if we were to take an action in a given state, then we could easily construct a policy that maximizes our
rewards:

\begin{equation}
 \pi^*(s) = \arg\!\max_a \ Q^*(s, a)
\end{equation}

However, we don't know everything about the world, so we don't have access to $Q^*$. But, since neural networks are universal function approximators, we can simply create one and train it to resemble $Q^*$.

For our training update rule, we'll use a fact that every $Q$ function for some policy obeys the Bellman equation:

\begin{equation}
 Q^{\pi}(s, a) = r + \gamma Q^{\pi}(s', \pi(s'))
\end{equation}

The difference between the two sides of the equality is known as the temporal difference error, $\delta$:

\begin{equation}
 \delta = Q(s, a) - (r + \gamma \max_a Q(s', a))
\end{equation}

To minimise this error, we will use the `Huber loss` (https://en.wikipedia.org/wiki/Huber_loss). The Huber loss acts like the mean squared error when the error is small, but like the mean absolute error when the error is large - this makes it more robust to outliers when the estimates of $Q$ are very noisy. We calculate this over a batch of transitions, $B$, sampled from the replay memory:

\begin{equation}
\mathcal{L} = \frac{1}{|B|}\sum_{(s, a, s', r) \ \in \ B} \mathcal{L}(\delta)
\end{equation}

where

\begin{equation}
\quad \mathcal{L}(\delta) = \begin{cases}
 \frac{1}{2}{\delta^2}  & \text{for } |\delta| \le 1, \\
 |\delta| - \frac{1}{2} & \text{otherwise.}
  \end{cases}
\end{equation}

### Replay Buffer
We’ll be using experience replay memory for training our DQN. It stores the transitions that the agent observes, allowing us to reuse this data later. By sampling from it randomly, the transitions that build up a batch are decorrelated. It has been shown that this greatly stabilizes and improves the DQN training procedure

### Exploration vs. Exploitation

Stable Baselines DQN implementation uses a trick to improve exploration at the start of training. For a fixed number of steps at the beginning, the agent takes actions which are sampled from a uniform random distribution over valid actions. After that, it returns to normal DQN exploration.

### Implementation Notes (Stable Baselines)

By default, the DQN class has double q learning and dueling extensions enabled.

#### Can I use?

-  Recurrent policies: ❌
-  Multi processing: ❌
-  Gym spaces:

| Space         | Action | Observation |
| --- | --- | --- |
|Discrete |      ✔|      ✔ |
|Box      |      ❌ |      ✔ |
|MultiDiscrete | ❌ |      ✔ |
|MultiBinary |   ❌ |      ✔ |

## 1.1 Import Required Libraries

```python
import warnings
warnings.filterwarnings("ignore")
import gym
import time
import numpy as np

from stable_baselines.common.vec_env import DummyVecEnv
from stable_baselines.deepq.policies import MlpPolicy
from stable_baselines import DQN
import matplotlib.pyplot as plt
```

    WARNING:tensorflow:
    The TensorFlow contrib module will not be included in TensorFlow 2.0.
    For more information, please see:
      * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
      * https://github.com/tensorflow/addons
      * https://github.com/tensorflow/io (for I/O related ops)
    If you depend on functionality not listed there, please file an issue.

## 1.2 Training Configuration
### Parameters

* **ENV_NAME**: the name of the Gym Environment for training
* **EVAL_STEPS**: number of samples to be used for policy evaluation. Here, the horizon $H$ is 200 steps, so the evaulation is 2000 / 200 = 10 episodes
* **ITERS**: number of simulation time-steps during training. Here corresponds to 250 episodes

```python
ENV_NAME = 'CartPole-v0'
EVAL_STEPS = 2000
ITERS = 50000
```

## 1.3 Logging
### Logging Directories

* **PPO2_Cartpole**: the directory used for the tensorboard
* **Results**: the directory where the intermediate trained agents are stored

```python
def logs_gen(net_size):
    tensorboard_log = "./DQN_Cartpole/2x" + str(net_size)
    log_dir = "./Results/"
    return log_dir, tensorboard_log
```

## 1.4 Environment Definition
### Vectorized Environments

Vectorized Environments are a method for stacking multiple independent environments into a single environment. Instead of training an RL agent on 1 environment per step, it allows us to train it on n environments per step. Because of this, actions passed to the environment are now a vector (of dimension n). It is the same for observations, rewards and end of episode signals (dones). In the case of non-array observation spaces such as Dict or Tuple, where different sub-spaces may have different shapes, the sub-observations are vectors (of dimension n).

```python
env = gym.make('CartPole-v0')
env = DummyVecEnv([lambda: env])
```

## 1.5 Agent Definition
### Agent Parameters

* **MlpPolicy**: The policy model to use (MlpPolicy, CnnPolicy, CnnLstmPolicy, …)
* **policy_kwargs**: the Network Architecture (here 2 hiden layers with size 256)
* **env**: the Gym Environment
* **learning_rate**: learning rate for adam optimizer
* **buffer_size**: size of the replay buffer
* **gamma**: Discount factor
* **exploration_fraction**:  fraction of entire training period over which the exploration rate is annealed
* **exploration_final_eps**: final value of random action probability
* **exploration_initial_eps**: initial value of random action probability
* **train_freq**: update the model every train_freq steps
* **batch_size**: size of a batched sampled from replay buffer for training
* **double_q**: Whether to enable Double-Q learning or not (always do this!)
* **learning_starts**: how many steps of the model to collect transitions for before learning starts
* **target_network_update_freq**: update the target network every target_network_update_freq steps
* **prioritized_replay**: if True prioritized replay buffer will be used
* **tensorboard_log**: the log location for tensorboar

```python
hidden_layer_size = 256
policy_kwargs = dict(layers=[hidden_layer_size, hidden_layer_size])
log_dir, tensorboard_log = logs_gen(hidden_layer_size)

model = DQN(MlpPolicy, env, gamma=0.99, learning_rate=0.001, buffer_size=50000, policy_kwargs=policy_kwargs,
            exploration_fraction=0.1, exploration_final_eps=0.02, exploration_initial_eps=1.0,
            train_freq=1, batch_size=32, double_q=True,
            learning_starts=1000, target_network_update_freq=500,
            prioritized_replay=False, verbose=1, tensorboard_log=tensorboard_log)
```

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/common/tf_util.py:191: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/common/tf_util.py:200: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/dqn.py:129: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:358: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:359: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:139: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/policies.py:109: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use keras.layers.flatten instead.
    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/layers/core.py:332: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use `layer.__call__` method instead.
    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:147: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:149: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where
    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:372: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:372: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:372: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:415: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/build_graph.py:449: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/common/tf_util.py:241: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/common/tf_util.py:242: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

## 1.6 Define a callback function
We can define a custom callback function that will be called inside the agent. This could be useful when we want to monitor training, for instance display live learning curves in Tensorboard or save the best agent.

```python
best_mean_reward, n_steps = -np.inf, 0
evaluation_rewards = []

def callback(_locals, _globals):
    """
  Callback called at each step
  :param _locals: (dict)
  :param _globals: (dict)
  """
    global n_steps, best_mean_reward
    # Print stats every 1000 calls
    if (n_steps + 1) % 1000 == 0:
        print("-----------------------------------------------------")
        print("Evaluating Model: " + str(n_steps))
        rew = evaluate(model, num_steps=EVAL_STEPS, render=False)
        print("Best mean reward: {:.15f} - Last mean reward per episode: {:.15f}".format(best_mean_reward, rew))
        _locals['self'].save(log_dir + 'model_' + str(n_steps))
        if (rew > best_mean_reward):
            best_mean_reward = rew
            print("Saving new best model")
            _locals['self'].save(log_dir + 'best_model')
    n_steps += 1
    return True
```

## 1.7 Define an evaluation function

During training the action selection is subject to the exploration scheme defined. This implies that policy A could be better than policy B but appear to perform worse due to the effect of the exploration.

Due to this, in order to determine the best policy throughout the entire training phase, in each Callback we evaluate the current policy without exploration (deterministic actions); if the current policy is better than the best policy found so far, we simply override the best policy.

```python
def evaluate(model, num_steps=300, render=False):
    global evaluation_rewards
    print("EVALUATION!!!")
    """
    Evaluate a RL agent
    :param model: (BaseRLModel object) the RL Agent
    :param num_steps: (int) number of timesteps to evaluate it
    :return: (float) Mean reward
    """
    episode_rewards = [[0.0] for _ in range(env.num_envs)]
    obs = env.reset()
    # print(obs)
    for i in range(num_steps - 1):
        # _states are only useful when using LSTM policies
        actions, _ = model.predict(obs, deterministic=True)
        # here, action, rewards and dones are arrays
        # because we are using vectorized env
        obs, rewards, dones, info = env.step(actions)

        if (render):
            env.render()

        for j in range(env.num_envs):
            episode_rewards[j][-1] += rewards[j]
            if dones[j]:
                episode_rewards[j].append(0.0)

    mean_rewards = [0.0 for _ in range(env.num_envs)]
    n_episodes = 0
    for i in range(env.num_envs):
        mean_rewards[i] = np.mean(episode_rewards[i])
        n_episodes += len(episode_rewards[i])

    # Compute mean reward
    mean_reward = round(np.mean(mean_rewards), 1)
    print("Mean reward:", mean_reward, "Num episodes:", n_episodes)
    evaluation_rewards.append(mean_reward)

    return mean_reward
```

## 1.8 Evaluate the Agent before training

```python
rew = evaluate(model, num_steps=EVAL_STEPS, render=False)
```

    EVALUATION!!!
    Mean reward: 29.0 Num episodes: 69

## 1.9 Train the DQN Agent

```python
start_time = time.time()
model.learn(total_timesteps=ITERS, callback=callback)
elapsed_time = time.time() - start_time
print("-----------------------------------------------------")
print("Training Time: " + str(elapsed_time))
```

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/common/base_class.py:1143: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/common/tf_util.py:322: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/common/tf_util.py:502: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.

    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 28.2 Num episodes: 71
    Best mean reward: -inf - Last mean reward per episode: 28.199999999999999
    Saving new best model
    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/dqn.py:283: The name tf.RunOptions is deprecated. Please use tf.compat.v1.RunOptions instead.

    WARNING:tensorflow:From /Users/mut/workspace/anaconda3/envs/py36/lib/python3.6/site-packages/stable_baselines/deepq/dqn.py:284: The name tf.RunMetadata is deprecated. Please use tf.compat.v1.RunMetadata instead.

    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 28.199999999999999 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 58       |
    | episodes                | 100      |
    | mean 100 episode reward | 21.6     |
    | steps                   | 2139     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 28.199999999999999 - Last mean reward per episode: 105.200000000000003
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 105.200000000000003 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 105.200000000000003 - Last mean reward per episode: 133.300000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 87.6     |
    | steps                   | 10899    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 20.8 Num episodes: 96
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 20.800000000000001
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 60.6 Num episodes: 33
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 60.600000000000001
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 60.6 Num episodes: 33
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 60.600000000000001
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 113      |
    | steps                   | 22170    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 58.8 Num episodes: 34
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 58.799999999999997
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 74.0 Num episodes: 27
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 74.000000000000000
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 35.1 Num episodes: 57
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 35.100000000000001
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 90.8     |
    | steps                   | 31253    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 83.3 Num episodes: 24
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 83.299999999999997
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 74.0 Num episodes: 27
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 74.000000000000000
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 37.7 Num episodes: 53
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 37.700000000000003
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 18.3 Num episodes: 109
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 18.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 92.8     |
    | steps                   | 40531    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 16.9 Num episodes: 118
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 16.899999999999999
    -----------------------------------------------------
    Training Time: 215.4886450767517

## 1.10 Plot the total reward of the policies evaluated in the Callback (deterministic)

```python
plt.plot(evaluation_rewards)
plt.ylabel('Total Reward')
plt.xlabel('Evaluation Iteration')
plt.show()
```

## 1.11 Evaluate the best Agent after training

```python
model = DQN.load(log_dir + 'best_model', env=env)
r = evaluate(model, num_steps=EVAL_STEPS, render=True)
```

    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10

## 1.12 Close environment

```python
env.close()
```

## 1.13 How each parameter affects the performance of the Algorithm?

```python
hid_layer_height_list = [2, 64, 128, 256]
hid_layer_list = [1,2]

lr_list = [0.00001, 0.001, 0.1]
buf_list = [32, 5000, 50000]
solved = []
```

```python
# network layer tests
for num_layers in hid_layer_list:
    for hidden_layer_size in hid_layer_height_list:
        layer_str = str(num_layers) + "x" + str(hidden_layer_size)
        print("---------------------------------------------")
        print("Layer Size = " + layer_str)
        print("---------------------------------------------")
        _layers = []
        for l in range(num_layers):
            _layers.append(hidden_layer_size)
        policy_kwargs = dict(layers=_layers)

        tensorboard_log = "./DQN_Cartpole_Param/network_size_" + layer_str
        log_dir = "./Results/"

        model = DQN(MlpPolicy, env, gamma=0.99, learning_rate=0.001, buffer_size=50000, policy_kwargs=policy_kwargs,
                    exploration_fraction=0.1, exploration_final_eps=0.02, exploration_initial_eps=1.0,
                    train_freq=1, batch_size=32, double_q=True,
                    learning_starts=1000, target_network_update_freq=500,
                    prioritized_replay=False, verbose=1, tensorboard_log=tensorboard_log)

        best_mean_reward, n_steps = -np.inf, 0
        evaluation_rewards = []

        start_time = time.time()
        model.learn(total_timesteps=ITERS, callback=callback)
        elapsed_time = time.time() - start_time
        print("Training Time: " + str(elapsed_time))

        model = DQN.load(log_dir + 'best_model', env=env)
        r = evaluate(model, num_steps=EVAL_STEPS, render=False)

        solved.append(r)
```

    ---------------------------------------------
    Layer Size = 1x2
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: -inf - Last mean reward per episode: 9.300000000000001
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 61       |
    | episodes                | 100      |
    | mean 100 episode reward | 19.9     |
    | steps                   | 1966     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.300000000000001 - Last mean reward per episode: 9.400000000000000
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 34       |
    | episodes                | 200      |
    | mean 100 episode reward | 13.7     |
    | steps                   | 3333     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 13       |
    | episodes                | 300      |
    | mean 100 episode reward | 11.1     |
    | steps                   | 4438     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 9.6      |
    | steps                   | 5395     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 9.9 Num episodes: 202
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.900000000000000
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 10.6     |
    | steps                   | 6454     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 10.1 Num episodes: 198
    Best mean reward: 9.900000000000000 - Last mean reward per episode: 10.100000000000000
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 600      |
    | mean 100 episode reward | 9.6      |
    | steps                   | 7413     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 700      |
    | mean 100 episode reward | 9.7      |
    | steps                   | 8378     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 800      |
    | mean 100 episode reward | 9.5      |
    | steps                   | 9331     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 900      |
    | mean 100 episode reward | 9.5      |
    | steps                   | 10280    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 11233    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1100     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 12162    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1200     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 13111    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1300     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 14051    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 14995    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1500     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 15944    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1600     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 16900    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1700     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 17852    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1800     |
    | mean 100 episode reward | 9.7      |
    | steps                   | 18817    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1900     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 19778    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2000     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 20735    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2100     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 21685    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2200     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 22636    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2300     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 23564    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2400     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 24513    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2500     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 25459    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2600     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 26396    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2700     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 27334    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2800     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 28274    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2900     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 29210    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3000     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 30140    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 31076    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3200     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 32017    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3300     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 32969    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 33912    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3500     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 34864    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3600     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 35805    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3700     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 36742    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3800     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 37682    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3900     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 38618    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 39564    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 40506    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4200     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 41441    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4300     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 42375    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 43313    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4500     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 44252    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4600     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 45194    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4700     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 46152    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4800     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 47113    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4900     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 48068    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 5000     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 49006    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 5100     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 49960    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 10.100000000000000 - Last mean reward per episode: 9.300000000000001
    Training Time: 119.86329483985901
    EVALUATION!!!
    Mean reward: 10.0 Num episodes: 200
    ---------------------------------------------
    Layer Size = 1x64
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 62.5 Num episodes: 32
    Best mean reward: -inf - Last mean reward per episode: 62.500000000000000
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 45.4 Num episodes: 44
    Best mean reward: 62.500000000000000 - Last mean reward per episode: 45.399999999999999
    --------------------------------------
    | % time spent exploring  | 47       |
    | episodes                | 100      |
    | mean 100 episode reward | 27.2     |
    | steps                   | 2697     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 62.500000000000000 - Last mean reward per episode: 133.300000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 181.699999999999989
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 141      |
    | steps                   | 16773    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 86.9 Num episodes: 23
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 86.900000000000006
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 128      |
    | steps                   | 29567    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 129      |
    | steps                   | 42497    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    Training Time: 118.5438551902771
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    ---------------------------------------------
    Layer Size = 1x128
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 15.4 Num episodes: 130
    Best mean reward: -inf - Last mean reward per episode: 15.400000000000000
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.8 Num episodes: 205
    Best mean reward: 15.400000000000000 - Last mean reward per episode: 9.800000000000001
    --------------------------------------
    | % time spent exploring  | 54       |
    | episodes                | 100      |
    | mean 100 episode reward | 23.2     |
    | steps                   | 2298     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 15.400000000000000 - Last mean reward per episode: 111.099999999999994
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 111.099999999999994 - Last mean reward per episode: 181.699999999999989
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 144      |
    | steps                   | 16656    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 149      |
    | steps                   | 31518    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 174      |
    | steps                   | 48900    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    Training Time: 124.84450674057007
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    ---------------------------------------------
    Layer Size = 1x256
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 9.9 Num episodes: 201
    Best mean reward: -inf - Last mean reward per episode: 9.900000000000000
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 83.3 Num episodes: 24
    Best mean reward: 9.900000000000000 - Last mean reward per episode: 83.299999999999997
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 55       |
    | episodes                | 100      |
    | mean 100 episode reward | 23.2     |
    | steps                   | 2293     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 44.4 Num episodes: 45
    Best mean reward: 83.299999999999997 - Last mean reward per episode: 44.399999999999999
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 83.299999999999997 - Last mean reward per episode: 153.800000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 166.599999999999994
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 181.699999999999989
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 102      |
    | steps                   | 12462    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 140      |
    | steps                   | 26428    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 62.5 Num episodes: 32
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 62.500000000000000
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 133      |
    | steps                   | 39745    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    Training Time: 127.40379309654236
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    ---------------------------------------------
    Layer Size = 2x2
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 50.0 Num episodes: 40
    Best mean reward: -inf - Last mean reward per episode: 50.000000000000000
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 60       |
    | episodes                | 100      |
    | mean 100 episode reward | 20.2     |
    | steps                   | 1995     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 34       |
    | episodes                | 200      |
    | mean 100 episode reward | 13.4     |
    | steps                   | 3331     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 13       |
    | episodes                | 300      |
    | mean 100 episode reward | 11       |
    | steps                   | 4428     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 9.7      |
    | steps                   | 5398     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 9.5      |
    | steps                   | 6347     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 600      |
    | mean 100 episode reward | 9.6      |
    | steps                   | 7310     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 700      |
    | mean 100 episode reward | 9.3      |
    | steps                   | 8245     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 800      |
    | mean 100 episode reward | 9.4      |
    | steps                   | 9188     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 900      |
    | mean 100 episode reward | 9.3      |
    | steps                   | 10116    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1000     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 11050    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 11990    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1200     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 12934    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1300     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 13883    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 14828    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1500     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 15766    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1600     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 16705    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1700     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 17642    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 9.2 Num episodes: 217
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.199999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1800     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 18571    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1900     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 19522    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 20474    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2100     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 21421    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2200     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 22352    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2300     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 23293    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 24235    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2500     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 25174    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2600     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 26127    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2700     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 27073    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2800     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 28015    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2900     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 28971    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3000     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 29905    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3100     |
    | mean 100 episode reward | 9.7      |
    | steps                   | 30870    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3200     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 31819    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3300     |
    | mean 100 episode reward | 9.7      |
    | steps                   | 32785    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3400     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 33734    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3500     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 34673    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3600     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 35617    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3700     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 36574    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3800     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 37533    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3900     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 38466    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 39412    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4100     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 40368    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4200     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 41320    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4300     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 42266    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 43208    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4500     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 44142    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4600     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 45074    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4700     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 46022    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4800     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 46952    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4900     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 47900    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 5000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 48850    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 9.5 Num episodes: 211
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.500000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 5100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 49787    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 50.000000000000000 - Last mean reward per episode: 9.300000000000001
    Training Time: 128.05865216255188
    EVALUATION!!!
    Mean reward: 55.5 Num episodes: 36
    ---------------------------------------------
    Layer Size = 2x64
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 66.6 Num episodes: 30
    Best mean reward: -inf - Last mean reward per episode: 66.599999999999994
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 66.599999999999994 - Last mean reward per episode: 9.300000000000001
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 66.599999999999994 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 86.9 Num episodes: 23
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 86.900000000000006
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 100      |
    | mean 100 episode reward | 60.2     |
    | steps                   | 5962     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 147      |
    | steps                   | 20634    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 17.4 Num episodes: 115
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 17.399999999999999
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 124      |
    | steps                   | 33080    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 51.3 Num episodes: 39
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 51.299999999999997
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 19.4 Num episodes: 103
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 19.399999999999999
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 80.0 Num episodes: 25
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 80.000000000000000
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 131      |
    | steps                   | 46166    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 14.1 Num episodes: 142
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 14.100000000000000
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 166.599999999999994
    Training Time: 137.02858328819275
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    ---------------------------------------------
    Layer Size = 2x128
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 9.5 Num episodes: 210
    Best mean reward: -inf - Last mean reward per episode: 9.500000000000000
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 64       |
    | episodes                | 100      |
    | mean 100 episode reward | 18.1     |
    | steps                   | 1795     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 11.2 Num episodes: 179
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 11.199999999999999
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 11.199999999999999 - Last mean reward per episode: 105.200000000000003
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 105.200000000000003 - Last mean reward per episode: 142.800000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 142.800000000000011 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 142.800000000000011 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 142.800000000000011 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 142.800000000000011 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 142.800000000000011 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 142.800000000000011 - Last mean reward per episode: 153.800000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 124.900000000000006
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 98.2     |
    | steps                   | 11614    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 166.599999999999994
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 60.6 Num episodes: 33
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 60.600000000000001
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 83.3 Num episodes: 24
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 83.299999999999997
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 100.000000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 86.6     |
    | steps                   | 20273    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 181.699999999999989
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 95.200000000000003
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 101      |
    | steps                   | 30357    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 68.9 Num episodes: 29
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 68.900000000000006
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 76.9 Num episodes: 26
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 76.900000000000006
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 52.6 Num episodes: 38
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 52.600000000000001
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 34.5 Num episodes: 58
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 34.500000000000000
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 58.8 Num episodes: 34
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 58.799999999999997
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 91.5     |
    | steps                   | 39511    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    Training Time: 145.1732199192047
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    ---------------------------------------------
    Layer Size = 2x256
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 30.3 Num episodes: 66
    Best mean reward: -inf - Last mean reward per episode: 30.300000000000001
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 30.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 58       |
    | episodes                | 100      |
    | mean 100 episode reward | 21.2     |
    | steps                   | 2098     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 30.300000000000001 - Last mean reward per episode: 124.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 76.9 Num episodes: 26
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 76.900000000000006
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 100.000000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 81.9     |
    | steps                   | 10290    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 124.900000000000006 - Last mean reward per episode: 133.300000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 29.4 Num episodes: 68
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 29.399999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 110      |
    | steps                   | 21333    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 83.3 Num episodes: 24
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 83.299999999999997
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 68.9 Num episodes: 29
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 68.900000000000006
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 80.0 Num episodes: 25
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 80.000000000000000
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 86.9 Num episodes: 23
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 86.900000000000006
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 116      |
    | steps                   | 32923    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 86.9 Num episodes: 23
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 86.900000000000006
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 18.5 Num episodes: 108
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 18.500000000000000
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 14.2 Num episodes: 141
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 14.199999999999999
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 101      |
    | steps                   | 43048    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    Training Time: 191.87267184257507
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10

```python
# learning rate tests
for ii in lr_list:
    hidden_layer_size = 128
    policy_kwargs = dict(layers=[hidden_layer_size])
    learning_rate = ii
    print("---------------------------------------------")
    print("Learning Rate = " + str(learning_rate))
    print("---------------------------------------------")

    tensorboard_log = "./DQN_Cartpole_Param/learning_rate_" + str(learning_rate)
    log_dir = "./Results/"

    model = DQN(MlpPolicy, env, gamma=0.99, learning_rate=learning_rate, buffer_size=50000,
                policy_kwargs=policy_kwargs,
                exploration_fraction=0.1, exploration_final_eps=0.02, exploration_initial_eps=1.0,
                train_freq=1, batch_size=32, double_q=True,
                learning_starts=1000, target_network_update_freq=500,
                prioritized_replay=False, verbose=1, tensorboard_log=tensorboard_log)

    best_mean_reward, n_steps = -np.inf, 0
    evaluation_rewards = []

    start_time = time.time()
    model.learn(total_timesteps=ITERS, callback=callback)
    elapsed_time = time.time() - start_time
    print("Training Time: " + str(elapsed_time))

    model = DQN.load(log_dir + 'best_model', env=env)
    r = evaluate(model, num_steps=EVAL_STEPS, render=False)

    solved.append(r)
```

    ---------------------------------------------
    Learning Rate = 1e-05
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 21.3 Num episodes: 94
    Best mean reward: -inf - Last mean reward per episode: 21.300000000000001
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 10.2 Num episodes: 196
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 10.199999999999999
    --------------------------------------
    | % time spent exploring  | 57       |
    | episodes                | 100      |
    | mean 100 episode reward | 22.1     |
    | steps                   | 2183     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 32       |
    | episodes                | 200      |
    | mean 100 episode reward | 12.6     |
    | steps                   | 3439     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 11       |
    | episodes                | 300      |
    | mean 100 episode reward | 10.6     |
    | steps                   | 4499     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 9.8      |
    | steps                   | 5477     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 9.4      |
    | steps                   | 6415     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 600      |
    | mean 100 episode reward | 9.4      |
    | steps                   | 7358     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 700      |
    | mean 100 episode reward | 9.4      |
    | steps                   | 8294     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 800      |
    | mean 100 episode reward | 9.4      |
    | steps                   | 9231     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 900      |
    | mean 100 episode reward | 9.4      |
    | steps                   | 10171    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 11117    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 12061    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1200     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 13007    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1300     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 13948    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 14890    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1500     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 15824    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1600     |
    | mean 100 episode reward | 9.7      |
    | steps                   | 16795    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1700     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 17739    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1800     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 18684    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1900     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 19625    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2000     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 20558    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 21496    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2200     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 22435    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2300     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 23383    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 24327    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2500     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 25271    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2600     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 26213    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2700     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 27162    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2800     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 28109    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2900     |
    | mean 100 episode reward | 9.1      |
    | steps                   | 29022    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 29972    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3100     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 30922    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 9.5 Num episodes: 211
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.500000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3200     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 31851    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3300     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 32808    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 9.2 Num episodes: 217
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.199999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3400     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 33753    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3500     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 34717    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3600     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 35667    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3700     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 36606    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3800     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 37550    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3900     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 38495    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4000     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 39437    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4100     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 40385    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4200     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 41328    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4300     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 42278    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 9.2 Num episodes: 218
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.199999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4400     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 43232    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4500     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 44181    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4600     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 45122    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4700     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 46067    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4800     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 47002    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4900     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 47942    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 5000     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 48883    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 9.5 Num episodes: 211
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.500000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 5100     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 49817    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 21.300000000000001 - Last mean reward per episode: 9.400000000000000
    Training Time: 125.50925898551941
    EVALUATION!!!
    Mean reward: 14.9 Num episodes: 134
    ---------------------------------------------
    Learning Rate = 0.001
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 12.8 Num episodes: 156
    Best mean reward: -inf - Last mean reward per episode: 12.800000000000001
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 62       |
    | episodes                | 100      |
    | mean 100 episode reward | 19.4     |
    | steps                   | 1921     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 29.0 Num episodes: 69
    Best mean reward: 12.800000000000001 - Last mean reward per episode: 29.000000000000000
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 29.000000000000000 - Last mean reward per episode: 142.800000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 142.800000000000011 - Last mean reward per episode: 166.599999999999994
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 181.699999999999989
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 101      |
    | steps                   | 12048    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 133      |
    | steps                   | 25372    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 76.9 Num episodes: 26
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 76.900000000000006
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 144      |
    | steps                   | 39731    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    Training Time: 123.7341251373291
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    ---------------------------------------------
    Learning Rate = 0.1
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 9.2 Num episodes: 217
    Best mean reward: -inf - Last mean reward per episode: 9.199999999999999
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 61       |
    | episodes                | 100      |
    | mean 100 episode reward | 20       |
    | steps                   | 1980     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.199999999999999 - Last mean reward per episode: 9.300000000000001
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 9.300000000000001 - Last mean reward per episode: 9.400000000000000
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 26       |
    | episodes                | 200      |
    | mean 100 episode reward | 17.8     |
    | steps                   | 3763     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 12.2     |
    | steps                   | 4988     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 10       |
    | steps                   | 5989     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 10       |
    | steps                   | 6992     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 600      |
    | mean 100 episode reward | 9.8      |
    | steps                   | 7976     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 700      |
    | mean 100 episode reward | 9.6      |
    | steps                   | 8936     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 800      |
    | mean 100 episode reward | 9.6      |
    | steps                   | 9897     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 900      |
    | mean 100 episode reward | 9.6      |
    | steps                   | 10859    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 9.5 Num episodes: 211
    Best mean reward: 9.400000000000000 - Last mean reward per episode: 9.500000000000000
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1000     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 11797    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 12741    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1200     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 13683    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1300     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 14633    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1400     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 15562    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1500     |
    | mean 100 episode reward | 9.2      |
    | steps                   | 16483    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1600     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 17418    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 9.2 Num episodes: 217
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.199999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1700     |
    | mean 100 episode reward | 9.2      |
    | steps                   | 18340    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1800     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 19279    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1900     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 20230    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 21177    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2100     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 22137    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2200     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 23086    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2300     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 24015    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2400     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 24968    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2500     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 25919    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2600     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 26850    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2700     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 27790    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2800     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 28748    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 2900     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 29696    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3000     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 30639    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3100     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 31596    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3200     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 32551    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3300     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 33494    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 9.2 Num episodes: 217
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.199999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3400     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 34456    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3500     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 35405    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3600     |
    | mean 100 episode reward | 9.2      |
    | steps                   | 36329    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3700     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 37276    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3800     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 38234    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 3900     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 39179    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 40127    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4100     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 41068    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4200     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 42028    |
    --------------------------------------
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4300     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 42973    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4400     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 43932    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4500     |
    | mean 100 episode reward | 9.4      |
    | steps                   | 44872    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4600     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 45806    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4700     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 46739    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 214
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4800     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 47690    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 216
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 4900     |
    | mean 100 episode reward | 9.3      |
    | steps                   | 48621    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 5000     |
    | mean 100 episode reward | 9.5      |
    | steps                   | 49570    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 9.5 Num episodes: 211
    Best mean reward: 9.500000000000000 - Last mean reward per episode: 9.500000000000000
    Training Time: 126.10655283927917
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215

```python
# replay buffer tests
for ii in buf_list:
    hidden_layer_size = 128
    policy_kwargs = dict(layers=[hidden_layer_size])
    buffer_size = ii
    print("---------------------------------------------")
    print("Buffer Size = " + str(buffer_size))
    print("---------------------------------------------")

    tensorboard_log = "./DQN_Cartpole_Param/buffer_size_" + str(buffer_size)
    log_dir = "./Results/"

    model = DQN(MlpPolicy, env, gamma=0.99, learning_rate=0.001, buffer_size=buffer_size,
                policy_kwargs=policy_kwargs,
                exploration_fraction=0.1, exploration_final_eps=0.02, exploration_initial_eps=1.0,
                train_freq=1, batch_size=32, double_q=True,
                learning_starts=1000, target_network_update_freq=500,
                prioritized_replay=False, verbose=1, tensorboard_log=tensorboard_log)

    best_mean_reward, n_steps = -np.inf, 0
    evaluation_rewards = []

    start_time = time.time()
    model.learn(total_timesteps=ITERS, callback=callback)
    elapsed_time = time.time() - start_time
    print("Training Time: " + str(elapsed_time))

    model = DQN.load(log_dir + 'best_model', env=env)
    r = evaluate(model, num_steps=EVAL_STEPS, render=False)

    solved.append(r)
```

    ---------------------------------------------
    Buffer Size = 32
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: -inf - Last mean reward per episode: 90.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 35.7 Num episodes: 56
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 35.700000000000003
    --------------------------------------
    | % time spent exploring  | 55       |
    | episodes                | 100      |
    | mean 100 episode reward | 22.9     |
    | steps                   | 2272     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 10.5 Num episodes: 191
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 10.500000000000000
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 29.4 Num episodes: 68
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 29.399999999999999
    --------------------------------------
    | % time spent exploring  | 20       |
    | episodes                | 200      |
    | mean 100 episode reward | 17.8     |
    | steps                   | 4047     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 55.5 Num episodes: 36
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 55.500000000000000
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 25.6 Num episodes: 78
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 25.600000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 22.8     |
    | steps                   | 6324     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 9.5      |
    | steps                   | 7271     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 21.3 Num episodes: 94
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 21.300000000000001
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 90.900000000000006 - Last mean reward per episode: 105.200000000000003
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 71.4 Num episodes: 28
    Best mean reward: 105.200000000000003 - Last mean reward per episode: 71.400000000000006
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 32.2     |
    | steps                   | 10491    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 105.200000000000003 - Last mean reward per episode: 111.099999999999994
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 90.9 Num episodes: 22
    Best mean reward: 111.099999999999994 - Last mean reward per episode: 90.900000000000006
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 111.099999999999994 - Last mean reward per episode: 133.300000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 10.7 Num episodes: 187
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 10.699999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 600      |
    | mean 100 episode reward | 35.2     |
    | steps                   | 14013    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 95.200000000000003
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 16.5 Num episodes: 121
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 16.500000000000000
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 25.0 Num episodes: 80
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 25.000000000000000
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 19.2 Num episodes: 104
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 19.199999999999999
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 700      |
    | mean 100 episode reward | 51.6     |
    | steps                   | 19170    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 41.6 Num episodes: 48
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 41.600000000000001
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 153.800000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 12.4 Num episodes: 161
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 12.400000000000000
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 142.800000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 800      |
    | mean 100 episode reward | 64.9     |
    | steps                   | 25661    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 62.5 Num episodes: 32
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 62.500000000000000
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 153.800000000000011 - Last mean reward per episode: 181.699999999999989
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 32.8 Num episodes: 61
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 32.799999999999997
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 900      |
    | mean 100 episode reward | 66.7     |
    | steps                   | 32326    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 9.9 Num episodes: 202
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 9.900000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1000     |
    | mean 100 episode reward | 10.6     |
    | steps                   | 33383    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 9.300000000000001
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1100     |
    | mean 100 episode reward | 11.2     |
    | steps                   | 34505    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 213
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1200     |
    | mean 100 episode reward | 9.6      |
    | steps                   | 35467    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 48.8 Num episodes: 41
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 48.799999999999997
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 10.8 Num episodes: 185
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 10.800000000000001
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 13.8 Num episodes: 145
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 13.800000000000001
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1300     |
    | mean 100 episode reward | 79.1     |
    | steps                   | 43376    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 9.3 Num episodes: 215
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 9.300000000000001
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 1400     |
    | mean 100 episode reward | 47.8     |
    | steps                   | 48156    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    Training Time: 127.3192892074585
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    ---------------------------------------------
    Buffer Size = 5000
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 12.0 Num episodes: 166
    Best mean reward: -inf - Last mean reward per episode: 12.000000000000000
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 48.8 Num episodes: 41
    Best mean reward: 12.000000000000000 - Last mean reward per episode: 48.799999999999997
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 48.799999999999997 - Last mean reward per episode: 111.099999999999994
    Saving new best model
    --------------------------------------
    | % time spent exploring  | 23       |
    | episodes                | 100      |
    | mean 100 episode reward | 39.6     |
    | steps                   | 3916     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 111.099999999999994 - Last mean reward per episode: 166.599999999999994
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 166.599999999999994 - Last mean reward per episode: 199.900000000000006
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 153      |
    | steps                   | 19259    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 199.900000000000006
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 12.7 Num episodes: 157
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 12.699999999999999
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 54.0 Num episodes: 37
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 54.000000000000000
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 112      |
    | steps                   | 30460    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 181.699999999999989
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 95.200000000000003
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 121      |
    | steps                   | 42592    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 80.0 Num episodes: 25
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 80.000000000000000
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 199.900000000000006 - Last mean reward per episode: 105.200000000000003
    Training Time: 123.69381213188171
    EVALUATION!!!
    Mean reward: 199.9 Num episodes: 10
    ---------------------------------------------
    Buffer Size = 50000
    ---------------------------------------------
    -----------------------------------------------------
    Evaluating Model: 999
    EVALUATION!!!
    Mean reward: 11.4 Num episodes: 176
    Best mean reward: -inf - Last mean reward per episode: 11.400000000000000
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 1999
    EVALUATION!!!
    Mean reward: 9.4 Num episodes: 212
    Best mean reward: 11.400000000000000 - Last mean reward per episode: 9.400000000000000
    --------------------------------------
    | % time spent exploring  | 51       |
    | episodes                | 100      |
    | mean 100 episode reward | 24.8     |
    | steps                   | 2459     |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 2999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 11.400000000000000 - Last mean reward per episode: 95.200000000000003
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 3999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 95.200000000000003 - Last mean reward per episode: 133.300000000000011
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 4999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 5999
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11
    Best mean reward: 133.300000000000011 - Last mean reward per episode: 181.699999999999989
    Saving new best model
    -----------------------------------------------------
    Evaluating Model: 6999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 7999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 8999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 9999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 10999
    EVALUATION!!!
    Mean reward: 142.8 Num episodes: 14
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 142.800000000000011
    -----------------------------------------------------
    Evaluating Model: 11999
    EVALUATION!!!
    Mean reward: 86.9 Num episodes: 23
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 86.900000000000006
    -----------------------------------------------------
    Evaluating Model: 12999
    EVALUATION!!!
    Mean reward: 95.2 Num episodes: 21
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 95.200000000000003
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 200      |
    | mean 100 episode reward | 107      |
    | steps                   | 13149    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 13999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 14999
    EVALUATION!!!
    Mean reward: 166.6 Num episodes: 12
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 166.599999999999994
    -----------------------------------------------------
    Evaluating Model: 15999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 16999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 17999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 18999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 19999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 20999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 21999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 22999
    EVALUATION!!!
    Mean reward: 153.8 Num episodes: 13
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 153.800000000000011
    -----------------------------------------------------
    Evaluating Model: 23999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 300      |
    | mean 100 episode reward | 114      |
    | steps                   | 24526    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 24999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 25999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 26999
    EVALUATION!!!
    Mean reward: 86.9 Num episodes: 23
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 86.900000000000006
    -----------------------------------------------------
    Evaluating Model: 27999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 28999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 29999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 30999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 31999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 32999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 33999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 34999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 400      |
    | mean 100 episode reward | 114      |
    | steps                   | 35943    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 35999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 36999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 37999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 38999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 39999
    EVALUATION!!!
    Mean reward: 100.0 Num episodes: 20
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 100.000000000000000
    -----------------------------------------------------
    Evaluating Model: 40999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 41999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 42999
    EVALUATION!!!
    Mean reward: 105.2 Num episodes: 19
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 105.200000000000003
    -----------------------------------------------------
    Evaluating Model: 43999
    EVALUATION!!!
    Mean reward: 117.6 Num episodes: 17
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 117.599999999999994
    -----------------------------------------------------
    Evaluating Model: 44999
    EVALUATION!!!
    Mean reward: 124.9 Num episodes: 16
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 124.900000000000006
    -----------------------------------------------------
    Evaluating Model: 45999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    -----------------------------------------------------
    Evaluating Model: 46999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    --------------------------------------
    | % time spent exploring  | 2        |
    | episodes                | 500      |
    | mean 100 episode reward | 117      |
    | steps                   | 47626    |
    --------------------------------------
    -----------------------------------------------------
    Evaluating Model: 47999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 48999
    EVALUATION!!!
    Mean reward: 111.1 Num episodes: 18
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 111.099999999999994
    -----------------------------------------------------
    Evaluating Model: 49999
    EVALUATION!!!
    Mean reward: 133.3 Num episodes: 15
    Best mean reward: 181.699999999999989 - Last mean reward per episode: 133.300000000000011
    Training Time: 124.09976410865784
    EVALUATION!!!
    Mean reward: 181.7 Num episodes: 11

```python
env.close()

print("Final reward (Evaluation) for all hyperparameter combinations")
print(solved)
```

    Final reward (Evaluation) for all hyperparameter combinations
    [10.0, 181.7, 199.9, 181.7, 55.5, 199.9, 199.9, 199.9, 199.9, 199.9, 14.9, 199.9, 9.3, 199.9, 199.9, 181.7]

```python

```