TODO:

Comparison with incremental - should get same plot with fixed seeds

UCB derivation + experiment

Application to Opengym env (Contextual bandits)

Application to netpipe experiment

```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

```python
import torch
import torch.nn as nn
import torch.optim as optim

import numpy as np
import pandas as pd
import gym

import matplotlib.pylab as plt
```

# Introduction to the OpenAI gym

```python
env = gym.make('CartPole-v1')
```

```python
state = env.reset()
print(f'Current State = {state}')
```

    Current State = [ 0.0250601   0.00374017 -0.04489948  0.03326008]

```python
env.render()
```

    True

```python
print(f'Action Space = {env.action_space}')
```

    Action Space = Discrete(2)

```python
print(f'Observation Space = {env.observation_space}')
```

    Observation Space = Box(4,)

```python
action = env.action_space.sample()
print(f'Randomly chosen action = {action}')
```

    Randomly chosen action = 0

```python
state, reward, done, info = env.step(action)
```

```python
print(f'Action taken = {action}')
print(f'New State = {state}')
print(f'Reward received = {reward}')
print(f'Simulation done? = {done}')
print(f'Auxiliary info = {info}')
```

    Action taken = 0
    New State = [ 0.02513491 -0.1907101  -0.04423427  0.31144553]
    Reward received = 1.0
    Simulation done? = False
    Auxiliary info = {}

```python
#run N steps
env = gym.make('BipedalWalker-v2')

state = env.reset()
N = 1000 #run 1000 time-steps
for t in range(N):
    env.render()
    action = env.action_space.sample()
    state, reward, done, info = env.step(action)

    if done:
        state = env.reset()
```

```python
env.close()
```

## What other environments are possible?

https://gym.openai.com/envs/#classic_control

# Multi-armed Bandits

The material below is mostly based on the book by Sutton and Barto. I have tried to keep the notation consistent with the book and also inserted a few exercises from the book below.

### Comparison to Supervised Model

In a supervised model, the model tells us **what action to take**. This is often called ***instruction***. The action you actually take might be different.

In a reinforcement learning setup, the model tells us **how good the action is**. This is often called ***evaluation***.

Multi-armed bandits let us study a simplified version of the reinforcement learning problem. We will see both how to evaluate an action as well as how to use the evaluative part to take ("instruct") an action.

### Core Idea

Suppose you are at a slot machine with k slots. So you have k levers that can be pulled. On pulling each lever, you get a reward (some money) according to a stationary probability distribution that depends on the lever. Stationary means that the distribution for any lever doesn't change with time. For example lever 1 might have a normal distribution with mean \\$100 and standard deviation $10 (if you get a negative number, you have to pay that much to the casino).

This might sound like an artificial problem but applies to any setup where there are k discrete options with stationary distributions.

The catch is that you do not know the distributions in advance. If you did and you had enough time, you would just keep pulling on the lever with the highest mean (even if the standard deviation is high) and in the long run, i.e. in $T$ time-steps, you'll get a total reward of $T * \text{mean}$.

**Mathematical Aside**:

If you are given N independent and identically distributed random variables (outcomes of a probabilistic experiments):

$$X_1, X_2, \ldots, X_N$$

each with mean = $\mu$ and standard deviation = $\sigma$ (variance = $\sigma^2$ by definition).

then we are interested in looking at the total (each $X$ is a pull of the lever and there are $N$ pulls):

$$Y = X_1 + X_2 + \ldots + X_N$$

We have:

$$\text{mean of Y} = N\mu$$

$$\text{variance of Y} = N\sigma^2 \rightarrow \text{standard deviation of Y} = \sqrt{N}\sigma$$

So, after $N$ pulls, you'll get an average reward of $N\mu$ which is good because the higher the $\mu$, the more reward you'll get. Also, you might think that the "error" (standard deviation) of $Y$ also increases with $N$ which is true but the relevant quantity is the relative error:

$$\frac{\text{standard deviation of Y}}{\text{mean of Y}} = \frac{\sqrt{N} \sigma}{N \mu} \rightarrow_{N\rightarrow\infty} 0$$

In other words, your total reward goes up with the number of steps ($N$) and the relative deviation in your total reward goes to 0.

**End of Mathematical Aside**:

You have a choice of k actions - which lever to pull. Each action has a mean reward which is called the **value of the action**. Let's define a few quantities:

$$A_t = \text{action taken at time t}$$

$$R_t = \text{reward received at time t}$$

Note that the reward at time t is a number drawn from the stationary distribution for the lever pulled at time t.

So, another way of writing the value of an action (or a lever in our case) is:

$$q_*(a) = \mathbb{E}[R_t | A_t = a]$$

$\mathbb{E}$ means expected value or the mean of the variable $R_t$. The notation $[R_t | A_t = a]$ should be read as "the reward at time t **given** that the action at time t was a particular action a". In other words, $q_{*}(a)$ is the mean or expected reward for the action a. It is the unknown quantity that we would like to learn because if we did, then we could always take the action (i.e. pull the lever) with the highest $q_{*}(a)$.

The real challenge of course, is that we don't know anything about the values of the levers when we start the game. More formally, the challenge is:

**Given T time-steps, get the maximum money possible from the slot machine**

So our task is to try estimating the value of each action where there are k actions i.e. the act of pulling each of one of the k levers. These value estimates will change in time so let's denote them by

$$Q_t(a) = \text{estimate of the value of action a at time t}$$

Hopefully over time, our estimates will get better and we'll converge to $q_{*}(a)$ i.e. with time:

$$Q_t(a) \rightarrow_{t\rightarrow\infty} q_*(a)$$

Some more definitions:

Suppose at any given time t, you look at your estimates of the values, Q_t(a) and pick the action a with the highest possible Q_t(a). This would be called a ***greedy action***. Note that the greedy action isn't necessarily globally the best option because we don't have access to the true value, $q_*(a)$. The process of always using greedy actions is called ***exploitation***. On the other hand, if you don't pick the greedy action, the process is called ***exploration***.

In reinforcement learning, there's always a tradeoff between exploration and exploitation. In our particular case, you could try actions randomly for 10 steps and then always pick the greedy action. This would give you pretty bad results for say k = 10 since you either have no estimate or bad estimates for the actions. In the short-term it might sound desirable but in the long-term, you almost certainly are picking bad actions. Or you could be more patient and wait for 100 steps, get better estimates of the values and then pick the greedy actions. In this case, you spent more time exploring and potentially getting lower rewards for the first 100 steps but then get much higher rewards with lesser uncertainty.

```python

```

### Estimating Action Values

So our central problem is to estimate the value of each action in as few time steps as possible. This would then let us use the greedy action and maximize our rewards.

There's an obvious way to estimate these values:

$$Q_t(a) \equiv \frac{\text{sum of rewards when action a is taken till time t}}{\text{number of times action a is taken till time t}} = \frac{\Sigma_{i=1}^{t-1}R_i \mathbb{1}_{A_i=a}}{\Sigma_{i=1}^{t-1} \mathbb{1}_{A_i=a}}$$

The second equation is a more formal way of writing the average reward when action a is taken. $\mathbb{1}_{A_i=a}$ is a so-called indicator variables such that:

$$\mathbb{1}_{A_i=a} = \begin{cases}
1 \text{ if } A_i = a \\
0 \text{ if } A_i \neq a \\
\end{cases}$$

In other words, we are just keeping a running average of rewards for each action in a table. What can go wrong? At a given time t, there might be actions that we never got around to taking. Then we should set $Q_t(a)$ for such actions to be a default low value, say 0. With enough time, our estimates will keep getting better as we take each action more often and in the long-run, $Q_t(a) \rightarrow q_*(a)$.

### Taking actions

Once we have our estimates, $Q_t(a)$, what should we do? As mentioned before, one could take the greedy action:

$$A_t = arg\,max_{a} Q_t(a)$$

What can go wrong with this rule? If done too soon ($t$ is small enough to not get good estimates for the values, $Q_t(a)$), then we could get stuck just picking a few actions and getting more precise estimates for those actions but completely ignoring other actions that might lead to more reward in the long-run. The extreme case is where the best action, the one with the highest $q_*(a)$ was never sampled and has $Q_t(a) = 0$ (our default value) which means it'll never get picked greedily and will never get its value updated.

There's a simple solution to this problem. We use the so-called $\epsilon\text{-greedy}$ method. $\epsilon$ is used as a proxy for a small number greater than 0 in mathematics. What this method does is to pick a small number, say $\epsilon = 10^-2$ and with probability $\epsilon$, pick a random action and with probability $1-\epsilon$, pick the greedy action according to the value estimates $Q_t(a)$.

Why does this help? Because just by chance we'll randomly take other actions and update their value estimates $Q_t(a)$ which gives every action a chance to get sampled.

### Exercise 2.1

In $\epsilon\text{-greedy}$ action selection, for the case of two actions and $\epsilon=0.5$, what is the probability that the greedy action is selected?

There are two ways to get to the greedy action:

1. Pick greedy action with probability $1-\epsilon$.

2. With probability $\epsilon$, pick a random action. The probability of picking the greedy action is $\frac{1}{k}$ where $k$ = number of actions.

So, total probability of picking the greedy action is:

$$1-\epsilon + \epsilon \frac{1}{k} = 1 - \epsilon(1 - \frac{1}{k})$$

For $\epsilon = 0.5$ and $k = 2$, we get:

$$1 - 0.5*(1-\frac{1}{2}) = \frac{3}{4}$$

### Sutton-Barto - 2.3 The 10-armed Testbed (Simulations)

```python
def generate_instance(seed = None):
    '''Generate an instance with 10 arms
    where each arms has rewards distributed according
    to a normal distribution with some mean and variance=1.
    The mean of each arm is also chosen according to a normal
    distribution which itself has a mean=0.0 and variance=1.

    with means selected
    also according to a normal distribution between 0 and 1
    and variance = 1
    '''
    if seed is not None:
        np.random.seed(seed)

    #means = np.random.uniform(low=0.0, high=1.0, size=(10))

    means = np.random.normal(loc=0.0, scale=1.0, size=(10))
    variances = np.ones_like(means)

    return means, variances

def run_experiment(means, variances, time_steps=1000, epsilon=1e-2, seed=None):
    '''Given an instance of the k-armed bandit problem,
    run time_steps time steps with epsilon-greedy actions.
    Update Q (value of an action) at each time-step
    '''
    #seed=None -> change state
    np.random.seed(seed)

    K = len(means) #number of arms
    Q_estimates = np.zeros_like(means) #same shape as means or number of arms
    sums = np.zeros_like(means)
    counters = np.zeros_like(means)

    total_reward = 0
    average_reward_timeseries = []

    for t in range(time_steps):
        if np.random.random() < epsilon:
            #random selection
            arm_selected = np.random.randint(K)

        else:
            #greedy selection
            arm_selected = np.argmax(Q_estimates)

        m = means[arm_selected]
        var = variances[arm_selected]

        reward = np.random.normal(loc=m, scale=var) #rewards are normally distributed
        total_reward += reward
        average_reward_timeseries.append(total_reward / (t+1))

        #update Q
        sums[arm_selected] += reward
        counters[arm_selected] += 1

        Q_estimates[arm_selected] = sums[arm_selected] / counters[arm_selected]

    assert(np.allclose(np.sum(sums), total_reward, rtol=1e-5)) #comparison within tolerance rtol

    return total_reward, average_reward_timeseries, Q_estimates, means, variances, sums, counters
```

```python
means, variances = generate_instance(seed=0)
total_reward, avg_reward_ts, Q_estimates, means, variances, sums, counters = run_experiment(means,
                                                                                variances,
                                                                                time_steps=1000,
                                                                                epsilon=1e-2)
```

```python
fig, ax = plt.subplots()

ax.plot(means, Q_estimates, 'p')
ax.plot(means, means)
plt.xlabel('Actual Values $q_*(a)$')
plt.ylabel('Estimated Values $Q_{1000}(a)$')
plt.title("Estimated Values vs True Values Annotated with Number of Samples")

for index, number in enumerate(counters):
        ax.annotate(int(number), (means[index], Q_estimates[index]))
```

#### Experiment 1: Effect of $\epsilon$

```python
#Let's run with epsilon 1e-2
means, variances = generate_instance(seed=0)
total_reward, _, Q_estimates, means, variances, sums, counters = run_experiment(means,
                                                                                variances,
                                                                                time_steps=1000,
                                                                                epsilon=1e-2,
                                                                                seed=0)
```

```python
fig, ax = plt.subplots()

ax.plot(means, Q_estimates, 'p')
ax.plot(means, means)
plt.xlabel('Actual Values $q_*(a)$')
plt.ylabel('Estimated Values $Q_{1000}(a)$')
plt.title("Estimated Values vs True Values Annotated with Number of Samples")

for index, number in enumerate(counters):
        ax.annotate(int(number), (means[index], Q_estimates[index]))
```

In the plot above, the x-axis is the actual value of each action (generated uniformly between 0-1) while the y-value is the estimated value of each action which is a running average of the actions we took during our experiment. The diagonal line is the line $y = x$ where we expect all the points to line i.e. $Q_t(a) = q_*(a)$.

While a few points are close to the line, there are many that are pretty far away. We don't have good value estimates for those points. Why does that happen? Each point is annotated with the number of times we visited that lever/action during the experiment. As expected, a few actions were taken multiple times leading to good estimates of their values but others were barely visited (or not visited at all) which leads to bad estimates.

This suggests that if we were to increase $\epsilon$, then we'll randomly sample other states more often leading to better value estimates. Let's try this by increasing $\epsilon$ by a factor of 10.

```python
#Let's run with epsilon 1e-1
means, variances = generate_instance(seed=0)
total_reward, _, Q_estimates, means, variances, sums, counters = run_experiment(means,
                                                                                variances,
                                                                                time_steps=1000,
                                                                                epsilon=1e-1,
                                                                                seed=0)
```

```python
fig, ax = plt.subplots()

ax.plot(means, Q_estimates, 'p')
ax.plot(means, means)
plt.xlabel('Actual Values $q_*(a)$')
plt.ylabel('Estimated Values $Q_{1000}(a)$')
plt.title("Estimated Values vs True Values Annotated with Number of Samples")

for index, number in enumerate(counters):
        ax.annotate(int(number), (means[index], Q_estimates[index]))
```

Now we have many more points near the diagonal. We still have a couple of points which weren't visited much but the situation is much better than the previous case with $\epsilon = 1e-2$.

#### Experiment 2: Average reward vs Number of Steps for Epsilon = (0.0, 0.1, 0.01)

```python
#run each particular instance with particular means a total of 2000 times
N_experiments = 2000
time_steps = 1000
```

```python
# Epsilon = 0.0
def get_avg_rewards(epsilon):
    avg_rewards = np.zeros((N_experiments, time_steps))
    for exp in range(N_experiments):
        means, variances = generate_instance(seed=exp)
        _, avg_reward_ts, _, _, _, _, _ = run_experiment(means,
                                                         variances,
                                                         time_steps=time_steps,
                                                         epsilon=epsilon)
        avg_rewards[exp, :] = avg_reward_ts

    return avg_rewards
```

```python
avg_rewards_eps0 = get_avg_rewards(0.0)
avg_rewards_epsd1 = get_avg_rewards(1e-1)
avg_rewards_epsdd1 = get_avg_rewards(1e-2)
```

```python
plt.figure(figsize=(9,9))
plt.plot(avg_rewards_eps0.mean(axis=0), label='epsilon = 0.0')
plt.plot(avg_rewards_epsdd1.mean(axis=0), label='epsilon=0.01')
plt.plot(avg_rewards_epsd1.mean(axis=0), label='epsilon=0.1')
plt.legend(loc=4)
plt.xlabel('Time Steps')
plt.ylabel('Average reward at time t = Total reward / t')
```

Let's understand the plot above. The x-axis is time with a total of 1000 time-steps for each trial. The y-axis is the average reward per time-step as a function of time. In other words, at time t, we take the total reward accumulated till then, and divide by the time elapsed. This is a measure of the efficiency of our learning mechanism. If we learn which lever has the highest reward rapidly, we can keep pulling it thus increasing our average reward per unit time.

One problem with making a plot like this is that we might get biased results if we choose just one set of levers and rewards (the parameters of the distributions). To account for this, we generate 2000 experiments, each with 10 levers but with different parameters and run our experiment for each one. For each experiment, we get a time-series of average reward as a function of time. We average these across the 2000 experiments to get the value plotted on the y-axis. This way we can be pretty certain that we are looking at general trends and not something particular to one set of lever reward values.

$\epsilon = 0$: This is the $\color{blue}{\text{blue}}$ line. Since $\epsilon = 0$, at every time-step, we greedily pick the best action or lever i.e. the one with the highest $Q_t(a)$.

**Note**: Since we start out with $Q_0(a) = 0,  \forall a$, np.argmax will pick index = 0 as the first action and update its value, $Q_t(a = 0)$. If this value is positive, action = 0 will be picked again while if it's negative, index = 1 will be picked since $Q_1(a=1) = 0$.

In this regime, we are purely exploiting the knowledge we have and very quickly we saturate to an average reward which indicates we stopped learning anything more about our actions.

$\epsilon = 0.01$: This is the $\color{orange}{\text{orange}}$ line. In this regime, the average reward keeps climbing because with a small probability ($ = \epsilon$), we pick a random action that lets us stumble upon more information about the non-greedy actions and get better estimates of their reward. This suggests that we should increase $\epsilon$ to gather information about other actions more quickly.

$\epsilon = 0.1$: This is the $\color{green}{\text{green}}$ line. This is exactly what we expected. With a larger $\epsilon$, we pick a non-greedy action more frequently and estimate its value with more samples and less uncertainty. This leads to a much faster route to finding the most optimal action i.e. the one with the highest $q_*(a)$.

#### Experiment 3: Percentage of time optimal action was picked

### Sutton-Barto - 2.4 Incremental Computation

In the function run_experiments above, we keep track of two lists: sums and counters, each of which is indexed by the action to keep track of the average reward value when action $i$ was taken. In other words:

$$Q_t(a) = \frac{\text{sums}[a]}{\text{counters}[a]}$$

and $\text{sums[a]}, \text{counters}[a]$ is updated each time we take action $a$.

In more complicated problems, the action space can be enormous. It pays to compress this computation to just a single array. Can we do that? Yes!

Suppose, we have an estimate of an action's value at time t:

$$Q_t(a) = \frac{R_1 + R_2 + \ldots + R_{t-1}}{t-1}$$

This is a bit sloppy since it implies the first $t-1$ steps all took the same action. This is only for notational convenience.)

Suppose, we take the same action $a$ at time-step $t$ and get a reward $R_t$. Then,

$$Q_{t+1}(a) = \frac{R_1 + R_2 + \ldots + R_{t-1} + R_t}{t}$$

by definition. We'll try writing this in terms of $Q_t(a)$.

\begin{equation}
\begin{split}
Q_{t+1}(a) &= \frac{R_1 + R_2 + \ldots + R_{t-1}}{t} + \frac{R_t}{t} \\
& = \frac{R_1 + R_2 + \ldots + R_{t-1}}{t-1} \frac{t-1}{t} + \frac{R_t}{t} \\
& = Q_t(a) \frac{t-1}{t} + \frac{R_t}{t} \\
& = Q_t(a) (1 - \frac{1}{t}) + \frac{R_t}{t} \\
& = Q_t(a) + \frac{1}{t} [R_t - Q_t(a)] \\
\end{split}
\end{equation}

This is of a form:

$$Q_{\text{new}} = Q_{\text{old}} + (\text{update weight}) \text{ update}$$

where $\text{update weight} = \frac{1}{t}$ and $\text{update} = R_t - Q_t(a)$.

If you have seen gradient descent, this should look oddly familiar. Also note that we can now update $Q_t$ with just a direct addition.

```python
def run_experiment_incremental(means, variances, time_steps=1000, epsilon=1e-2, seed=None):
    '''Given an instance of the k-armed bandit problem,
    run time_steps time steps with epsilon-greedy actions.
    Update Q (value of an action) at each time-step
    '''
    #seed=None -> change state
    np.random.seed(seed)

    K = len(means) #number of arms
    Q_estimates = np.zeros_like(means) #same shape as means or number of arms

    total_reward = 0
    average_reward_timeseries = []

    for t in range(time_steps):
        if np.random.random() < epsilon:
            #random selection
            arm_selected = np.random.randint(K)

        else:
            #greedy selection
            arm_selected = np.argmax(Q_estimates)

        m = means[arm_selected]
        var = variances[arm_selected]

        reward = np.random.normal(loc=m, scale=var) #rewards are normally distributed
        total_reward += reward
        average_reward_timeseries.append(total_reward / (t+1))

        #incremental update here
        Q_estimates[arm_selected] = Q_estimates[arm_selected] + (1/(t+1)) * (reward - Q_estimates[arm_selected])

    return total_reward, average_reward_timeseries, Q_estimates, means, variances
```

```python
means, variances = generate_instance(seed=0)
total_reward, avg_reward_ts, Q_estimates, means, variances, sums, counters = run_experiment(means,
                                                                                variances,
                                                                                time_steps=1000,
                                                                                epsilon=1e-2,
                                                                                seed=0)
```

```python
print(means)
print(variances)
```

    [ 1.76405235  0.40015721  0.97873798  2.2408932   1.86755799 -0.97727788
      0.95008842 -0.15135721 -0.10321885  0.4105985 ]
    [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

```python
fig, ax = plt.subplots()

ax.plot(means, Q_estimates, 'p')
ax.plot(means, means)
plt.xlabel('Actual Values $q_*(a)$')
plt.ylabel('Estimated Values $Q_{1000}(a)$')
plt.title("Estimated Values vs True Values Annotated with Number of Samples")

for index, number in enumerate(counters):
        ax.annotate(int(number), (means[index], Q_estimates[index]))
```

```python
means, variances = generate_instance(seed=0)
total_reward, avg_reward_ts, Q_estimates, means, variances = run_experiment_incremental(means,
                                                                                variances,
                                                                                time_steps=1000,
                                                                                epsilon=1e-2,
                                                                                seed=0)
```

```python
print(means)
print(variances)
```

    [ 1.76405235  0.40015721  0.97873798  2.2408932   1.86755799 -0.97727788
      0.95008842 -0.15135721 -0.10321885  0.4105985 ]
    [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

```python
fig, ax = plt.subplots()

ax.plot(means, Q_estimates, 'p')
ax.plot(means, means)
plt.xlabel('Actual Values $q_*(a)$')
plt.ylabel('Estimated Values $Q_{1000}(a)$')
plt.title("Estimated Values vs True Values Annotated with Number of Samples")

for index, number in enumerate(counters):
        ax.annotate(int(number), (means[index], Q_estimates[index]))
```

### Generalizations

At this stage, we can think of generalizing the problem. We made a few assumptions which can each be relaxed.

**Assumption 1**: The reward distributions are stationary i.e. the probability distribution that each action/lever samples is constant in time. This is generally not true in the real world.

**Assumption 2**: Each action taken is completely independent of the other one. In other words, there are no ***dynamics***. The act of taking action $i$ vs action $j$ at time $t$ has no effect on the slot machine itself. This is generally also not true in the real world. A person or a robot has to change their strategy completely if they are walking normally vs if they slip on a certain step. What is done next is highly dependent on the fact that you slipped or didn't slip on the previous step.  One could account for this in our problem by saying that the act of taking a particular action ***changes*** the probability distributions of all other actions which leads to non-stationary distributions too.

What would go wrong in our strategy of estimating action values by running averages if the distributions were not stationary? It's easiest to demonstrate with an example. Suppose, we have numbers coming from a uniform random distribution $\mathcal{U}[0,1]$. This means we pick numbers between 0 and 1 with equal weight given to every number (technically, the real numbers are uncountable so one can't assign a probability to picking a number, say 0.3182. Instead, a uniform distribution means that probability of picking a number between $x$ and $y$ is proportional to $|x-y|$ i.e. the length of the interval as long as $x,y\in[0,1]$).

Now suppose the the interval shifts to the right at a constant speed. In other words, at time t, we are sampling uniformly from the interval $[t, t+1]$.

At any given time, the mean should be the mid-point $\frac{t + t+1}{2} = t + \frac{1}{2}$. But if we just take all the numbers we recorded till time t, we'll get a misleading answer.

At time $t=0$: sampled number is between 0 and 1, say 0.3.

At time $t=1$: sampled number is between 1 and 2, say 1.7.

At time $t=2$: sampled number is between 2 and 3, say 2.1.

So even though, at time $t=3$, we would expect the mean of the last 3 numbers to be somewhere reasonably close to the mean $3.5$, in actuality, we measured numbers completely below this range which would skew the average to:

$$\frac{0.3 + 1.7 + 2.1}{3} = 1.366$$

which is completely outside the interval $[3,4]$ at $t=3$!!!!!!

The problem is that our past measurements are from completely different distributions because the distribution has been moving in time.

How do we deal with this issue? There's an obvious solution. Weight recent measurements more in the running average than past measurements. Instead of the running average being:

$$\frac{x_1 + x_2 + \ldots + x_n}{n}$$

it would be:

$$\frac{w_1 x_1 + w_2 x_2 + \ldots + w_n x_n}{\Sigma_{t} w_t}$$

where $w_t$ are positive numbers that denote weights. We expect $w_t$ to monotonically increase in time to give more weight to measurements at later times t. In the special case, $w_t = 1, \forall t$, we get back our usual definition of the unweighted average.

So the big question is how should we weigh new updates so that we are biased towards more recent measurements. Recall our update equation for $Q_t(a)$:

$$Q_{t+1}(a) = Q_{t}(a) + \underbrace{\frac{1}{t}}_{\text{update weight}} [R_t - Q_t(a)]$$

In this case, as time goes on, the update weight $\frac{1}{t}$ decreases over time which has the effect that each reward value is weighted equally in the running average:

$$\frac{R_1 + R_2 + \ldots R_t}{t}$$

If we were to increase $\frac{1}{t}$, we'll hopefully get to a situation:

$$\frac{R_1 + R_2 + \ldots R_t}{t} \rightarrow \frac{w_1 R_1 + w_2 R_2 + \ldots w_t R_t}{t}$$

where $w_t$ increases with $t$.

We could pick the update weight to be anything of the form $t^p$ where $p$ is a power. Currently we have $p=-1$. The next higher we could go is $p=0$ i.e. $t^0 = 1$. So maybe we could try a constant (instead of just 1), $\alpha$.

$$Q_{t+1}(a) \stackrel{?}{=} Q_{t}(a) + \underbrace{\alpha}_{\text{try this?}} [R_t - Q_t(a)]$$

Let's see what the consequences of this are by unpacking this equation. First note:

$$Q_{t+1}(a) = Q_t(a) + \alpha [R_t - Q_t(a)] \implies Q_{t+1}(a) = \alpha R_t + (1-\alpha) Q_t(a)$$

\begin{equation}
\begin{split}
Q_{t+1}(a) &= \alpha R_t + (1 - \alpha) Q_t(a)\\
&= \alpha R_t + (1 - \alpha) [\alpha R_{t-1} + (1 - \alpha) Q_{t-1}(a)] \\
&= \alpha R_t + (1-\alpha)\alpha R_{t-1} + (1-\alpha)^2 Q_{t-1}(a) \\
&= \alpha R_t + (1-\alpha)\alpha R_{t-1} + (1-\alpha)^2 [\alpha R_{t-2} + (1 - \alpha) Q_{t-2}(a)] \\
&= \alpha R_t + (1-\alpha)\alpha R_{t-1} + (1-\alpha)^2 \alpha R_{t-2} + (1-\alpha)^3 Q_{t-2}(a)\\
&= \alpha R_t + (1-\alpha)\alpha R_{t-1} + (1-\alpha)^2 \alpha R_{t-2} + \ldots + (1-\alpha)^{t-1} \alpha R_{t-(t-1)} + (1-\alpha)^{t} \underbrace{Q_{1}(a)}_{R_1}\\
\end{split}
\end{equation}

We can see a pattern forming. The most recent reward $R_t$ has a weight of $\alpha$, the next more recent has a weight of $(1-\alpha)\alpha$ and every subsequent reward is multiplied by another factor of $1-\alpha$. If we select $\alpha \in [0,1]$, we'll get a suppression of reward earlier in time with $R_{t-k}$ getting suppressed by $\alpha (1-\alpha)^k$

This is often called exponential suppression since for small $\alpha$, $1-\alpha \approx e^{-\alpha}$ (look at the first order Taylor expansion) to give:

$Q_{t+1}(a) \approx \alpha R_t + \alpha e^{-\alpha} R_{t-1} + \ldots + \alpha e^{-k\alpha} R_{t-k} + \ldots + \alpha e^{-(t-1)\alpha} R_1 + e^{-t\alpha} R_1$.

By the way, at this point, you should worry about the weights being normalized i.e. adding to 1. Otherwise we have a weighted sum, not a weighted average. Let's quickly confirm this:

\begin{equation}
\begin{split}
\alpha + \alpha (1-\alpha) + \alpha (1-\alpha)^2 + \ldots + \alpha (1-\alpha)^{t-1} + (1-\alpha)^t &= \alpha [1 +  (1-\alpha) + (1-\alpha)^2 + \ldots + (1-\alpha)^{t-1}] + (1 - \alpha)^t \\
&= \alpha \frac{(1-\alpha)^t-1}{1-\alpha-1} + (1-\alpha)^t
&= 1
\end{split}
\end{equation}

### Application to OpenGym Env

#### CartPole Environment

The goal in the cartpole environment is to balance the pole and keep it vertical as long as possible. More precisely, we have a cart that can move left/right on a frictionless track. There's an upside-down pole that is balanced at the beginning. The state of the setup can be described by 4 numbers at any given point in time:

Position of Cart: -2.4 to 2.4

Velocity of Cart: -Inf to Inf

Pole Angle: -41.8 deg to 41.8 deg

Pole Velocity at Tip: -Inf to Inf

[Reference Link for Environment]("https://github.com/openai/gym/wiki/CartPole-v0">)

We can take one of two actions:

0: Push cart to left
1: Push cart to right

For every time-step that the game is active, we get a reward of +1. Our task is to maximize this reward.

The trial terminates when:

1. The pole angle is greater than 12 deg in either direction.

2. Cart reaches the edge of the display.

3. Time-steps exceeds 200 steps.

#### k-armed Bandits

We would expect k-armed bandits to be a very poor fit for this problem but let's try anyway.

Instead of k-actions, now we have two actions - go left or go right. But, we have a much bigger problem. Instead of just one environment i.e. one set of means and variances for each action/lever, we have one such experiment for ***every possible value in the state space***. In addition, depending on the action we take at time $t$, we end up in a different experiment.

```python
#run N steps
env = gym.make('CartPole-v1')

state = env.reset()
N = 1000 #run 1000 time-steps
for t in range(N):
    env.render()
    action = env.action_space.sample()
    state, reward, done, info = env.step(action)

    if done:
        state = env.reset()
```

### Application to netpipe data

Let's first try a very simple implementation of contextual bandits. In the multi-armed bandit approach above, each arm had fixed reward distributions. Even when we discussed the non-stationary case, the assumption is that the distribution is changing slowly enough.

What if we were randomly put in front of different slot machines, each with k-arms with different underlying parameters for the reward distribution. If we know which machine we are in front of, this problem is context the **contextual multi-arm bandit problem*** or just **contextual bandits**.

In the netpipe experiment, if the message size was fixed, we would have a multi-armed bandit problem where we choose between 1 of k discrete RX_DELAY values. The task then is to try out different RX_DELAYS (pull different levers) and follow the algorithm above to find the optimal RX_DELAY for THAT PARTICULAR message value.

Since we can have many different message values, we can treat each message value as its own experiment. In the language of data-structures, we have a dictionary of dictionaries. The first dictionary uses the message size as the key and the second dictionary uses the RX_DELAY as a key.

```python
df = pd.read_csv('../data/netpipe.log')
```

```python
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>MSG</th>
      <th>RXD</th>
      <th>DTXMXSZRQ</th>
      <th>WTHRESH</th>
      <th>PTHRESH</th>
      <th>HTHRESH</th>
      <th>TPUT</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>61535</td>
      <td>6</td>
      <td>610</td>
      <td>10</td>
      <td>30</td>
      <td>16</td>
      <td>3639.59</td>
    </tr>
    <tr>
      <th>1</th>
      <td>15571</td>
      <td>57</td>
      <td>3275</td>
      <td>18</td>
      <td>22</td>
      <td>36</td>
      <td>1087.24</td>
    </tr>
    <tr>
      <th>2</th>
      <td>54051</td>
      <td>52</td>
      <td>3212</td>
      <td>1</td>
      <td>39</td>
      <td>11</td>
      <td>2874.62</td>
    </tr>
    <tr>
      <th>3</th>
      <td>37847</td>
      <td>50</td>
      <td>2021</td>
      <td>21</td>
      <td>19</td>
      <td>31</td>
      <td>2237.28</td>
    </tr>
    <tr>
      <th>4</th>
      <td>6288</td>
      <td>6</td>
      <td>695</td>
      <td>34</td>
      <td>6</td>
      <td>26</td>
      <td>744.12</td>
    </tr>
  </tbody>
</table>
</div>

#### Step 1: We don't have access to the real netpipe environment in this notebook. So we'll simulate it by sampling from the dataframe df.

In other words, for each MSG (message size), we get (a) either sample from TPUT values (this is the reward) by choosing one of the available RXD (actions) or (b) infer a distribution with the sample mean and variance and sample from the distribution.

```python
plt.figure(figsize=(8,8))
plt.plot(np.sort(df.MSG.unique()),'p')
plt.xlabel('Index in Array of Uniques')
plt.ylabel('MSG value')
plt.title('Different slopes signifiy different granularities of sampling in different ranges')
```

```python
df.groupby('MSG').count().max()

# First problem: have at most 8 values for any given MSG (context)
# Solution: Let's bin the MSG values by percentiles
```

    RXD          8
    DTXMXSZRQ    8
    WTHRESH      8
    PTHRESH      8
    HTHRESH      8
    TPUT         8
    dtype: int64

##### Mapping MSG -> Index in percentiles

```python
percentiles = np.percentile(df.MSG, np.arange(0, 100, 5))
print(percentiles)
```

    [   501.    2836.7   5182.4   7471.1   9779.   12254.   14688.4  17136.9
      19620.2  24143.   29203.   38188.   47775.6  67427.2  89869.4 112952.
     136439.2 159970.  183040.  250595.3]

```python
def find_percentile_index(value, percentiles):
    if value < percentiles[0]:
        return -1
    if value > percentiles[-1]:
        return len(percentiles)-1

    return np.where(np.histogram([value], bins=percentiles)[0]==1)[0][0]
```

```python
print(find_percentile_index(500, percentiles))
print(find_percentile_index(2840, percentiles))
print(find_percentile_index(250595, percentiles))
print(find_percentile_index(250596, percentiles))
```

    -1
    1
    18
    19

```python
df['bin'] = df.MSG.apply(lambda x: find_percentile_index(x, percentiles))
```

```python
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>MSG</th>
      <th>RXD</th>
      <th>DTXMXSZRQ</th>
      <th>WTHRESH</th>
      <th>PTHRESH</th>
      <th>HTHRESH</th>
      <th>TPUT</th>
      <th>bin</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>61535</td>
      <td>6</td>
      <td>610</td>
      <td>10</td>
      <td>30</td>
      <td>16</td>
      <td>3639.59</td>
      <td>12</td>
    </tr>
    <tr>
      <th>1</th>
      <td>15571</td>
      <td>57</td>
      <td>3275</td>
      <td>18</td>
      <td>22</td>
      <td>36</td>
      <td>1087.24</td>
      <td>6</td>
    </tr>
    <tr>
      <th>2</th>
      <td>54051</td>
      <td>52</td>
      <td>3212</td>
      <td>1</td>
      <td>39</td>
      <td>11</td>
      <td>2874.62</td>
      <td>12</td>
    </tr>
    <tr>
      <th>3</th>
      <td>37847</td>
      <td>50</td>
      <td>2021</td>
      <td>21</td>
      <td>19</td>
      <td>31</td>
      <td>2237.28</td>
      <td>10</td>
    </tr>
    <tr>
      <th>4</th>
      <td>6288</td>
      <td>6</td>
      <td>695</td>
      <td>34</td>
      <td>6</td>
      <td>26</td>
      <td>744.12</td>
      <td>2</td>
    </tr>
  </tbody>
</table>
</div>

```python
df.groupby('bin').count()

#We essentially merged together a bunch of MSG values to create 20 environments
#where each environment is defined for 1/20th of the message sizes
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>MSG</th>
      <th>RXD</th>
      <th>DTXMXSZRQ</th>
      <th>WTHRESH</th>
      <th>PTHRESH</th>
      <th>HTHRESH</th>
      <th>TPUT</th>
    </tr>
    <tr>
      <th>bin</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>3</th>
      <td>2565</td>
      <td>2565</td>
      <td>2565</td>
      <td>2565</td>
      <td>2565</td>
      <td>2565</td>
      <td>2565</td>
    </tr>
    <tr>
      <th>4</th>
      <td>2570</td>
      <td>2570</td>
      <td>2570</td>
      <td>2570</td>
      <td>2570</td>
      <td>2570</td>
      <td>2570</td>
    </tr>
    <tr>
      <th>5</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>6</th>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>8</th>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
    </tr>
    <tr>
      <th>9</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>10</th>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
    </tr>
    <tr>
      <th>11</th>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
    </tr>
    <tr>
      <th>12</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>13</th>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
    </tr>
    <tr>
      <th>14</th>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
    </tr>
    <tr>
      <th>15</th>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
      <td>2569</td>
    </tr>
    <tr>
      <th>16</th>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
      <td>2567</td>
    </tr>
    <tr>
      <th>17</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>18</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
    <tr>
      <th>19</th>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
      <td>2568</td>
    </tr>
  </tbody>
</table>
</div>

```python
#some exploratory plots
df.hist(column='RXD', by='bin', figsize=(12,12));

#looking at the plots below, the distribution of arms/RXD values is pretty uniform i.e.
#we have enough statistics for each arm
```

```python
#reward distribution
plt.hist(df[(df.bin==1) & (df.RXD==40)].TPUT)
```

Let's spend some time discussing what was done above? I'll use the column names in the dataframe below.

MSG -> Message size

RXD -> Time delay

TPUT -> Throughput

Ignore the rest for now

For the netpipe experiment, we want to learn what the appropriate RXD value is for a given MSG. In particular, the exploratory plots (not in this notebook) show that there are RXD values that maximize TPUT (throughput) or power efficiency for a fixed MSG. The k-armed bandits way of phrasing this problem is the following:

* Fix a message size

* There are k-arms - one for each possible RXD

* When we pull on an arm i.e. we select a RXD, we get a certain reward (TPUT, power efficiency)

* The reward is stochastic (many other factors are not in our control) and our task is to find the RXD with the maximum expected value of reward (maximum mean).

* We can follow the learning algorithm for k-armed bandits above.

* But we don't have just one message size.

* We can treat this problem as one k-armed bandit problem for each message size.

* In real-life, we would implement the learning algorithm and let the experiment run. But we don't have the experiment in this notebook and we also want to use the already collected data.

* We can reconstruct the experimental setup by calculating the properties of the distribution of the rewards for each (MSG, RXD) pair. As one would expect, this leads to a loss of statistics i.e. each such pair has very few measurements giving us very noisy and unusable reward values.

* To fix this, we group together consecutive MSG values into one bucket where we define each bucket to be the 5% percentiles. This might be increased (7-10%) to increase the statistics but still maintain a fine granularity in MSG. We could also have combined RXD values but let's not do that for now.

```python
df.RXD.unique()
```

    array([ 6, 57, 52, 50, 58,  5, 45, 37, 22,  4, 51, 36, 46, 14, 55, 38, 20,
           41, 49, 29, 33, 18, 11, 54, 17, 31, 13, 44, 12, 47, 30,  7, 40, 48,
           10, 59, 39,  8, 26, 32, 27, 15, 35,  3, 34, 53, 24, 19, 21,  9, 28,
           42, 60, 23, 43, 25, 56, 16,  2,  1])

```python
np.sort(df.RXD.unique())
```

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
           18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
           35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
           52, 53, 54, 55, 56, 57, 58, 59, 60])

```python
def netpipe_bandits(df, epsilon=1e-1, seed=None):
    '''Given an instance of the k-armed bandit problem,
    run time_steps time steps with epsilon-greedy actions.
    Update Q (value of an action) at each time-step
    '''

    #seed=None -> change state
    np.random.seed(seed)

    #dictionaries mapping bins -> lists
    Q_estimates = {}
    sums = {}
    counters = {}

    bin_list = df.bin.unique()

    #data
    bin_values = df.bin
    rxd_values = df.RXD
    tput_values = df.TPUT

    N_time_steps = len(bin_values)

    #number of arms
    unique_arms = np.sort(df.RXD.unique())
    K = len(df.RXD.unique()) #should discretize arms

    #initialize lists
    for bin_val in bin_list:
        Q_estimates[bin_val] = np.zeros(K)
        sums[bin_val] = np.zeros(K)
        counters[bin_val] = np.zeros(K)

    total_reward = 0
    average_reward_timeseries = []

    #Start experiment
    for t in range(N_time_steps): #loop over df
        bin_val = bin_values[t]

        if np.random.random() < epsilon:
            #random selection
            arm_index_selected = np.random.randint(K)

        else:
            #greedy selection
            arm_index_selected = np.argmax(Q_estimates[bin_val])

        arm_selected = unique_arms[arm_index_selected]

        #get reward - empirical sampling
        print(f'Bin = {bin_val}')
        print(f'Rxd = {arm_selected}')
        print(df[(df.bin==bin_val) & (df.RXD==arm_selected)].shape)
        reward = np.random.choice(df[(df.bin==bin_val) & (df.RXD==arm_selected)].TPUT)
        #note: this step above will be realllly slow -> replace df by np.array

        total_reward += reward
        average_reward_timeseries.append(total_reward / (t+1))

        #update Q
        sums[bin_val][arm_index_selected] += reward
        counters[bin_val][arm_index_selected] += 1

        Q_estimates[bin_val][arm_index_selected] = sums[bin_val][arm_index_selected] / counters[bin_val][arm_index_selected]

    #assert(np.allclose(np.sum(sums), total_reward, rtol=1e-5)) #comparison within tolerance rtol

    return total_reward, average_reward_timeseries, Q_estimates, sums, counters
```

```python
total_reward, average_reward_ts, Q_estimates, sums, counters = netpipe_bandits(df)
```

BrnoTeaching2019/notebooks/Reinforcement Learning - Multi-armed Bandits.ipynb


# Duboko učenje

### 2. laboratorijska vježba - Konvolucijski modeli

*Zagreb, 20.04.2020.*

## Izjava

Tekstovi zadataka se koriste samo u edukativne svrhe, te njihova prava još uvijek pripadaju autorima. Tekstovi zadatka preuzeti su sa [sljedeće poveznice](https://dlunizg.github.io/lab2/). Također, bilo kakve izmjene su isključivo radi estetike, i ne mijenjaju intelektualnog vlasnika na mene ili bilo kog tko uređuje ovu datoteku.

## Sadržaj

- [Zadatak 1](#Zadatak-1-%C2%A0%C2%A0%C2%A0-(25%))
- [Zadatak 2](#Zadatak-2-%C2%A0%C2%A0%C2%A0-(25%))
- [Zadatak 3](#Zadatak-3-%C2%A0%C2%A0%C2%A0-(25%))
- [Zadatak 4](#Zadatak-4-%C2%A0%C2%A0%C2%A0-(25%))

# Učitavanje resursa

```python
import json
import os

from matplotlib import pyplot as plt
import torch

from IPython.core.display import display, HTML, Markdown, SVG

from task_3 import prepare_MNIST, CNNT3
from task_4 import prepare_CIFAR, CNNT4, analyze_evaluation, cifar_labels
from task_4 import SAVE_DIR as t4_save_dir
```

# Zadatak 1     (25%)

### Uvod u konvolucijske mreže

#### [Zadatak 2 ->](#Zadatak-2-%C2%A0%C2%A0%C2%A0-(25%))

## Postavke zadatka

```python
t1_do_train = False  # True ako želite pokrenuti trening u podzadatku 8, False inače.
```

### Podzadatak 1

Dovršite implementacije potpuno povezanog sloja, sloja nelinearnosti te funkcije gubitka u razredima `FC`, `ReLU` i `SoftmaxCrossEntropyWithLogits`.

**Odgovor**:

Implementacije su ponuđene u `layers.py`

### Podzadatak 2

Kako biste bili sigurni da ste ispravno napisali sve slojeve testirajte gradijente pozivom skripte `check_grads.py`. Zadovoljavajuća relativna greška bi trebala biti manja od $10^{-5}$ ako vaši tenzori imaju dvostruku preciznost.

**Odgovor**:

Prvo treba izgraditi modul:

```python
!python3 setup_cython.py build_ext --inplace
```

    running build_ext

Zatim treba pokrenuti program:

```python
!python3 check_grads.py
```

    Convolution
    Check grad wrt input
    Relative error =  1.387713497584426e-09
    Error norm =  2.708619411758534e-10
    Check grad wrt params
    Check weights:
    Relative error =  3.960080099515749e-11
    Error norm =  2.606048252692708e-10
    Check biases:
    Relative error =  2.2273920054210374e-12
    Error norm =  2.6553929139417377e-11

    MaxPooling
    Check grad wrt input
    Relative error =  3.2756455233944856e-12
    Error norm =  9.520047457286901e-11

    ReLU
    Check grad wrt input
    Relative error =  3.275620779981156e-12
    Error norm =  5.448211281921029e-11

    FC
    Check grad wrt input
    Relative error =  1.5526408894548135e-09
    Error norm =  7.755267280874444e-10
    Check grad wrt params
    Check weights:
    Relative error =  7.022404000356437e-10
    Error norm =  7.706474875193658e-10
    Check biases:
    Relative error =  8.472355095041804e-10
    Error norm =  1.4085252676109427e-10

    SoftmaxCrossEntropyWithLogits
    Relative error =  1.4583572252144326e-06
    Error norm =  4.934061917260249e-10

    L2Regularizer
    Check grad wrt params
    Relative error =  7.289869002753903e-06
    Error norm =  2.336430571622429e-09

**Komentar**:

Ponekad greška `L2Regularizer` može premašiti $10^{-5}$. Valja napomenuti da se uvijek koristi jednostruka (`float32`) preciznost, pa je to za očekivati.

### Podzadatak 3

Sada prevedite Cython modul `im2col_cython.pyx` pozivom `python3 setup_cython.py build_ext --inplace` te po potrebi izmijenite varijable `DATA_DIR` i `SAVE_DIR`.

**Odgovor**: Ovo smo već napravili u prethodnom koraku.

### Podzadatak 4

Proučite i skicirajte model zadan objektom `net` u skripti `train.py`.

**Odgovor**:

Navedena mreža ilustrirana je sljedećom shemom:

```python
SVG(filename="res/task1_schematic.svg")
```

Ova shema je izrađena uz pomoć alata [NN SVG](https://alexlenail.me/NN-SVG/).

### Podzadatak 5

Odredite veličine tenzora te broj parametara u svakom sloju.

**Odgovor**:

- Prvi konvulucijski sloj
  - ulaz: $N \times 1 \times 28 \times 28$
  - $16$ filtera, $1$ kanal, dimenzija $5 \times 5$ uz pomak od $16$ elemenata: $416$ parametara.
  - izlaz: $N \times 16 \times 28 \times 28$
- Prvi udruživački sloj po maksimumu
  - ulaz: $N \times 16 \times 28 \times 28$
  - nema parametara
  - izlaz: $N \times 16 \times 14 \times 14$
- Drugi konvolucijski sloj
  - ulaz: $N \times 16 \times 14 \times 14$
  - $32$ filtera, $16$ kanala, dimenzija $5 \times 5$ uz pomak od $32$ elementa tj. $12832$ parametara
  - izlaz: $N \times 32 \times 14 \times 14$
- Drugi udruživački sloj po maksimumu
  - ulaz: $N \times 32 \times 14 \times 14$
  - nema parametara
  - izlaz: $N \times 32 \times 7 \times 7$

- Sloj spljošćivanja
  - ulaz: $N \times 32 \times 7 \times 7$
  - nema parametara
  - izlaz: $N \times 1568$

- Prvi potpuno povezani sloj
  - ulaz: $N \times 1568$
  - matrica $512 \times 1568$ i pomak od $512$ elemenata, tj. $803328$ parametara
  - izlaz: $N \times 512$
- Drugi potpuno povezani sloj
  - ulaz: $N \times 512$
  - matrica $10 \times 512$ i pomak od $10$ elemenata, tj. $5130$ parametara
  - izlaz: $N \times 10$

### Podzadatak 6

Odredite veličinu receptivnog polja značajki iz posljednjeg (drugog) konvolucijskog sloja.

**Odgovor**:

Po algoritmu pronađenom [ovdje](https://shawnleezx.github.io/blog/2017/02/11/calculating-receptive-field-of-cnn/):

- $l = 1, f = 1$: input
- $l = 5, f = 1$: conv_1
- $l = 6, f = 2$: maxpool_1
- $l = 14, f = 2$: conv_2

Dakle, receptivno polje posljednjeg konvolucijskog sloja je $14$.

### Podzadatak 7

Procijenite ukupnu količinu memorije za pohranjivanje aktivacija koje su potrebne za provođenje backpropa ako učimo s mini-grupama od $50$ slika.

**Odgovor**:

Treba pohraniti $50$ gradijenata za sve naučive parametre. Pobrojimo ih po slojevima:

- Prvi konvolucijski: $416$ parametara
- Drugi konvolucijski: $12832$ parametara
- Prvi potpuno povezani: $803328$ parametara
- Drugi potpuno povezani: $5130$ parametara

Ubacimo sad ovo u izračun:

```python
t1_s7_conv_1_size = 416
t1_s7_conv_2_size = 12832
t1_s7_fc_1_size = 803328
t1_s7_fc_2_size = 5130

t1_s7_total_size = t1_s7_conv_1_size + t1_s7_conv_2_size + \
                   t1_s7_fc_1_size + t1_s7_fc_2_size

t1_s7_total_size_50_batches = 50 * t1_s7_total_size

t1_s7_total_size_bytes = t1_s7_total_size_50_batches * 4
```

```python
Markdown(f"<br>Ukupno nam treba **{t1_s7_total_size_bytes} B**, "
         f"tj. **{t1_s7_total_size_bytes / 1e6:.02f} MB** memorije.")
```

Ukupno nam treba **164341200 B**, tj. **164.34 MB** memorije.

**Komentar**: Naravno, s ovo je samo procjena donje granice. Tijekom izvršavanja je potrebna dodatna memorija jer nije kao da se samo pamte gradijenti koji se mijenjaju. Međutim, kako je alokacija i dealokacija ove memorije ovisna o implementaciji knjižnice i programskog jezika, pružamo procjenu samo za naučive gradijente, jer ćemo njih uvijek morati pamtiti. Isto tako, pretpostavilo se da nema kompresije podataka. Uz kompresiju (npr. rijetke matrice) bi ovi brojevi mogli biti manji.

### Podzadatak 8

Napokon, pokrenite učenje modela pozivom skripte `train.py`.

```python
t1_s8_display = ""

if t1_do_train:
    !python3 train.py
else:
    t1_s8_display = "<br>Postavljeno je da se treniranje **ne pokreće** " +\
                    "(`t1_do_train == False`)."

Markdown(t1_s8_display)
```

Postavljeno je da se treniranje **ne pokreće** (`t1_do_train == False`).

### Podzadatak 9

Odredite vezu između početnog iznosa funkcije gubitka i broja razreda $C$.

**Odgovor**:

Što je veći broj $C$, to nasumično inicijalizirana mreža, za koju pretpostavljamo da predviđa rezultate kao da izvlači klasifikacije iz uniformne razdiobe, ima više šanse pogriješiti. Stoga, porastom broja $C$ će početni iznos funkcije gubitka rasti.

# Zadatak 2     (25%)

### L2 regularizacija

#### [<- Zadatak 1](#Zadatak-1-%C2%A0%C2%A0%C2%A0-(25%))          [Zadatak 3 ->](#Zadatak-3-%C2%A0%C2%A0%C2%A0-(25%))

## Postavke zadatka

```python
t2_do_train = False  # True ako želite pokrenuti trening u podzadatku 1, False inače.
                     # Upozorenje - to traje skoro 1 sat!

t2_root_path = "out_l2reg"
t2_results_path = os.path.join(t2_root_path, "results.json")
t2_weight_decays = [1e-3, 1e-2, 1e-1]
```

U ovom zadatku trebate dodati podršku za L2 regularizaciju parametara.

### Podzadatak 1

Dovršite implementaciju sloja `L2Regularizer` te naučite regularizirani model iz prethodnog zadatka koji se nalazi u `train_l2reg.py`.

**Odgovor**:

Implementacija `L2Regularizer` nalazi se u `layers.py`.

```python
t2_s1_display = ""

if t2_do_train:
    !python3 train_l2reg.py
else:
    t2_s1_display = "<br>Postavljeno je da se treniranje **ne pokreće** " +\
                    "(`t2_do_train == False`)."

Markdown(t2_s1_display)
```

Postavljeno je da se treniranje **ne pokreće** (`t2_do_train == False`).

### Podzadatak 2

Proučite efekte regularizacijskog hiper-parametra tako da naučite tri različita modela s $\lambda = 10^{-3}$, $\lambda = 10^{-2}$, $\lambda = 10^{-1}$ te usporedite naučene filtre u prvom sloju i dobivenu točnost.

**Odgovor**:

Spremili smo sliku u datoteke naziva `out_l2reg/lambda0.001`, `out_l2reg/lambda0.010` i `out_l2reg/lambda0.100`. Ako učitamo posljednje datoteke, možemo prikazati sljedeće slike:

```python
t2_s2_folder_paths = [os.path.join(t2_root_path, x) for x in sorted(os.listdir(t2_root_path))]
t2_s2_image_paths = list()
```

```python
t2_s2_figure, t2_s2_axes = plt.subplots(3, 1, figsize=(12, 9))

for i, folder_path in enumerate(t2_s2_folder_paths[:3]):
    current_image_path = os.path.join(folder_path, sorted(os.listdir(folder_path))[-1])

    t2_s2_axes[i].imshow(plt.imread(current_image_path), interpolation="nearest")
    t2_s2_axes[i].set_title(f"λ = {t2_weight_decays[i]}")
    t2_s2_axes[i].axis("off")
```

Konsekventno, čitanjem datoteke `out_l2reg/results.json` možemo usporediti dobivene točnosti:

```python
with open(t2_results_path) as file:
    t2_s2_results = json.load(file)
```

```python
t2_s2_string = "<br>"

for i, weight_decay in enumerate(sorted(t2_weight_decays)):
    t2_s2_string += f"\n<br>Točnost za λ = {weight_decay}:   " +\
                    f"".join(["  " for _ in range(i)]) +\
                    f"<strong>{t2_s2_results[str(weight_decay)]:.02f}%</strong>"

HTML(t2_s2_string)
```

Točnost za λ = 0.001:   99.16%

Točnost za λ = 0.01:     98.74%

Točnost za λ = 0.1:       96.53%

**Komentar**:

Na prvi pogled se čini kako je najmanji iznos regularizacije dao najbolju mrežu radi najveće točnosti. Ovo možda i jest istina. Međutim, ako pogledamo slike, možemo uočiti da su za $\lambda = 10^{-2}$ prvi konvolucijski slojevi daleko interpretabilniji od ostalih - vidimo linije, krivulje, horizontalne i dijagonalne praznine i sl. Dok u dubokom učenje interpretabilnost ne rezultira bolji rezultatima, tj. obično bude i suprotno, radi se u prvom sloju, koji izvlači značajke iz slike. Prema tome, vjerojatno ćemo, ako se radi o slikama koje su jednostavne kao MNIST, imati bolje rezultate ako su i filteri prvog sloja interpretabilni.

# Zadatak 3     (25%)

### Usporedba s PyTorchem

#### [<- Zadatak 2](#Zadatak-2-%C2%A0%C2%A0%C2%A0-(25%))          [Zadatak 4 ->](#Zadatak-4-%C2%A0%C2%A0%C2%A0-(25%))

## Postavke zadatka

```python
t3_do_train = False  # True ako želite pokrenuti trening u podzadatku 1, False inače.
                     # Upozorenje - to traje skoro 1 sat!

t3_root_path = "out_task3"
t3_results_filename = "results.json"
t3_weight_decays = [1e-3, 1e-2, 1e-1]
```

U PyTorchu definirajte i naučite model koji je ekvivalentan regulariziranom modelu iz [2. zadatka](#Zadatak-2). Koristite identičnu arhitekturu i parametre učenja da biste reproducirali rezultate. Konvoluciju zadajte operacijama [`torch.nn.Conv2d`](https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d) ili [`torch.nn.functional.conv2d`](https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.conv2d).

### Podzadatak 1

Tijekom učenja vizualizirajte filtre u prvom sloju kao u prethodnoj vježbi. Nakon svake epohe učenja pohranite filtre i gubitak u datoteku (ili koristite [Tensorboard](https://pytorch.org/docs/stable/tensorboard.html)).

```python
t3_s1_display = ""

if t3_do_train:
    t3_s1_mnist_dict = prepare_MNIST()

    for weight_decay in [1e-3, 1e-2, 1e-1]:
        model = CNNT3()
        model.float()

        model.tr(t3_s1_mnist_dict["x"][0], t3_s1_mnist_dict["y"][0],
                 t3_s1_mnist_dict["x"][1], t3_s1_mnist_dict["y"][1],
                 weight_decay=weight_decay)
else:
    t3_s1_display = "<br>Postavljeno je da se treniranje <strong>ne pokreće</strong> " +\
                    "(<code>t3_do_train == False</code>)."

HTML(t3_s1_display)
```

Postavljeno je da se treniranje ne pokreće (t3_do_train == False).

**Komentar**

Umjesto svakih $5000$ epoha, ovo ćemo raditi jednom po epohi. Ne vidim razlog zašto bi se to radilo češće jer je trening ovakve mreže volatilan samo na početku, pa bi češće spremanje slika bilo opravdano jedino u prvoj epohi.

### Podzadatak 2

Na kraju učenja prikažite kretanje gubitka kroz epohe (Matplotlib).

**Odgovor**:

Prvo ćemo učitati sve puteve do datoteka:

```python
t3_s2_folder_paths = [os.path.join(t3_root_path, f"lambda{x:.03f}") for x in t3_weight_decays]

t3_s2_image_paths = list()
t3_s2_results_paths = list()

for folder_path in t3_s2_folder_paths:
    last_filter_path = sorted([x for x in os.listdir(folder_path) if x.endswith(".png")],
                               key=lambda x: x.lower())[-1]

    t3_s2_image_paths.append(os.path.join(folder_path, last_filter_path))
    t3_s2_results_paths.append(os.path.join(folder_path, t3_results_filename))
```

Zatim možemo prikazati posljednje filtre za različite parametre $\lambda$:

```python
t3_s2_figure, t3_s2_axes = plt.subplots(3, 1, figsize=(12, 9))

for i, image_path in enumerate(t3_s2_image_paths):
    t3_s2_axes[i].imshow(plt.imread(image_path), interpolation="nearest")
    t3_s2_axes[i].set_title(f"λ = {t2_weight_decays[i]}")
    t3_s2_axes[i].axis("off")
```

Konačno, možemo prikazati ovisnost gubitaka o epohama za sve 3 instance učenja:

```python
t3_s2_figure_l, t3_s2_axes_l = plt.subplots(1, 3, figsize=(16,5), sharey=True)

for i, result_path in enumerate(t3_s2_results_paths):
    with open(result_path) as file:
        loss_dict = json.load(file)

    loss, val_loss = [loss_dict[x] for x in ["loss", "val_loss"]]

    t3_s2_axes_l[i].set_title(f"Gubitak za λ = {t3_weight_decays[i]}")
    t3_s2_axes_l[i].set_xlabel(f"Epohe")
    t3_s2_axes_l[i].set_ylabel(f"Gubitak")

    t3_s2_axes_l[i].plot(range(len(loss)), loss, label="Gubitak učenja", marker="o")
    t3_s2_axes_l[i].plot(range(len(val_loss)), val_loss, label="Gubitak validacije", marker="o")

    t3_s2_axes_l[i].legend()
```

# Zadatak 4     (25%)

### Klasifikacija na skupu CIFAR-10

#### [<- Zadatak 3](#Zadatak-3-%C2%A0%C2%A0%C2%A0-(25%))

## Postavke zadatka

```python
t4_do_train = False  # True ako želite pokrenuti trening u podzadatku 2, False inače.
                     # Upozorenje - to traje 30 minuta, ali sama validacija zauzima
                     # oko 33 GB RAMa - pripremite veliki SWAP!

t4_root_path = "out_task4"

t4_results_tr_file_path = os.path.join(t4_root_path, "results_tr.json")
t4_results_val_file_path = os.path.join(t4_root_path, "results_val.json")
t4_learning_rates_file_path = os.path.join(t4_root_path, "learning_rates.json")

t4_model_path = os.path.join(t4_root_path, "model.pt")

t4_s5_path = os.path.join(t4_root_path, "t4_s5")
t4_s5_best_classes_path = os.path.join(t4_s5_path, "best_classes.json")
t4_s5_worst_classes_path = os.path.join(t4_s5_path, "worst_classes.json")
```

Skup podataka [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) sastoji se od $50000$ slika za učenje i validaciju te $10000$ slika za testiranje dimenzija $32 \times 32$ podijeljenih u $10$ razreda.

### Podzadatak 1

Najprije skinite dataset pripremljen za Python [odavde](https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz) ili korištenjem [`torchvision.datasets.CIFAR10`](https://pytorch.org/docs/stable/torchvision/datasets.html#cifar).

**Odgovor**:

Skup podataka je inicijalno preuzet s [sljedeće poveznice](https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz), raspakiran i smješten u `datasets/CIFAR`.

### Podzadatak 2

Vaš zadatak je da u PyTorchu naučite konvolucijski model na ovom skupu.

**Odgovor**:

Ovo ćemo učiniti s nešto drukčijom arhitekturom od predložene:

$
\begin{align}
    & \mathrm{Conv2D \space 64 @ 7 \times 7}  \\
    & \mathrm{MaxPool(2,2)}                   \\
    & \mathrm{PReLU}                          \\
    & \mathrm{BatchNorm2D}                    \\\\
%
    & \mathrm{Conv2D \space 64 @ 5 \times 5}  \\
    & \mathrm{MaxPool(2,2)}                   \\
    & \mathrm{PReLU}                          \\
    & \mathrm{BatchNorm2D}                    \\\\
%
    & \mathrm{Conv2D \space 64 @ 3 \times 3}  \\
    & \mathrm{PReLU}                          \\
    & \mathrm{BatchNorm2D}                    \\
    & \mathrm{Conv2D \space 128 @ 3 \times 3} \\
    & \mathrm{MaxPool(2,2)}                   \\
    & \mathrm{PReLU}                          \\
    & \mathrm{BatchNorm2D}                    \\\\
%
    & \mathrm{Flatten}                        \\\\
%
    & \mathrm{Dense(512)}                     \\
    & \mathrm{PReLU}                          \\
    & \mathrm{BatchNorm}                      \\\\
%
    & \mathrm{Dense(256)}                     \\
    & \mathrm{PReLU}                          \\
    & \mathrm{BatchNorm}                      \\\\
%
    & \mathrm{Dense(10)}                      \\
\end{align}
$

**Komentar**:

Razlog za ovoliko drukčiji model su bolje performanse, i u smislu brže konvergencije, i u smislu postizanja bolje točnost bez augmentacije.

```python
t4_s2_cifar_dict = prepare_CIFAR()
```

```python
t4_s2_display = ""

if t4_do_train:
    model = CNNT4()
    model.float()

    model.tr(t4_s2_cifar_dict["x"][0], t4_s2_cifar_dict["y"][0],
             t4_s2_cifar_dict["x"][1], t4_s2_cifar_dict["y"][1],
             n_epochs=16, batch_size=16, weight_decay=1e-4)

    torch.save(model, os.path.join(t4_root_path, "model.pt"))
else:
    t4_s2_display = "<br>Postavljeno je da se treniranje **ne pokreće**" +\
                    "(`t4_do_train == False`)."

Markdown(t4_s2_display)
```

Postavljeno je da se treniranje **ne pokreće**(`t4_do_train == False`).

### Podzadatak 3

Napišite funkciju `evaluate` koja na temelju predviđenih i točnih indeksa razreda određuje pokazatelje klasifikacijske performanse: ukupnu točnost klasifikacije, matricu zabune (engl. confusion matrix) u kojoj retci odgovaraju točnim razredima a stupci predikcijama te mjere preciznosti i odziva pojedinih razreda. U implementaciji prvo izračunajte matricu zabune, a onda sve ostale pokazatelje na temelju nje. Tijekom učenja pozivajte funkciju `evaluate` nakon svake epohe na skupu za učenje i validacijskom skupu te na grafu pratite sljedeće vrijednosti: prosječnu vrijednost funkcije gubitka, stopu učenja te ukupnu točnost klasifikacije. Preporuka je da funkciji provedete samo unaprijedni prolaz kroz dane primjere koristeći `torch.no_grad()` i pritom izračunati matricu zabune. Pazite da slučajno ne pozovete i operaciju koja provodi učenje tijekom evaluacije. Na kraju funkcije možete izračunati ostale pokazatelje te ih isprintati.

**Odgovor**:

Implementacija je ostvarena unutar klase `CNNT4` koja je unutar modula `task_4.py`:

```python
with open(t4_results_tr_file_path) as file:
    t4_s3_results_tr = json.load(file)

with open(t4_results_val_file_path) as file:
    t4_s3_results_val = json.load(file)

with open(t4_learning_rates_file_path) as file:
    t4_s3_learning_rates = json.load(file)
```

```python
t4_s3_metric_fig, t4_s3_metric_ax = plt.subplots(1, 2, figsize=(16,6), sharey=True)
t4_s3_tr_metric_fig, t4_s3_tr_metric_ax = plt.subplots(1, 2, figsize=(16,6))

t4_s3_metric_ax[0].set_title("Točnost po epohama")
t4_s3_metric_ax[0].set_xlabel("Epohe")
t4_s3_metric_ax[0].set_ylabel("%")

t4_s3_metric_ax[0].plot(range(len(t4_s3_results_tr["acc"])),
                        [x * 100 for x in t4_s3_results_tr["acc"]],
                        label="Točnost učenja",
                        marker="o")
t4_s3_metric_ax[0].plot(range(len(t4_s3_results_val["acc"])),
                        [x * 100 for x in t4_s3_results_val["acc"]],
                        label="Točnost validacije",
                        marker="o")

t4_s3_metric_ax[1].set_title("F1-mjera po epohama")
t4_s3_metric_ax[1].set_xlabel("Epohe")
t4_s3_metric_ax[1].set_ylabel("%")

t4_s3_metric_ax[1].plot(range(len(t4_s3_results_tr["f1"])),
                        [x * 100 for x in t4_s3_results_tr["f1"]],
                        label="F1-mjera učenja",
                        marker="o")
t4_s3_metric_ax[1].plot(range(len(t4_s3_results_val["f1"])),
                        [x * 100 for x in t4_s3_results_val["f1"]],
                        label="F1-mjera validacije",
                        marker="o")

t4_s3_tr_metric_ax[0].set_title("Unakrsna entropija po epohama")
t4_s3_tr_metric_ax[0].set_xlabel("Epohe")
t4_s3_tr_metric_ax[0].set_ylabel("Gubitak")

t4_s3_tr_metric_ax[0].plot(range(len(t4_s3_results_tr["loss"])),
                           t4_s3_results_tr["loss"],
                           label="Gubitak učenja",
                           marker="o")
t4_s3_tr_metric_ax[0].plot(range(len(t4_s3_results_val["loss"])),
                           t4_s3_results_val["loss"],
                           label="Gubitak validacije",
                           marker="o")

t4_s3_tr_metric_ax[1].set_title("Stopa učenja po epohama")
t4_s3_tr_metric_ax[1].set_xlabel("Epohe")
t4_s3_tr_metric_ax[1].set_ylabel("Stopa učenja")
t4_s3_tr_metric_ax[1].set_yscale("log")

t4_s3_tr_metric_ax[1].plot(range(len(t4_s3_learning_rates)),
                           t4_s3_learning_rates,
                           label="Stopa učenja",
                           marker="o")

for i in range(2):
    t4_s3_metric_ax[i].legend()
    t4_s3_tr_metric_ax[i].legend()
```

**Komentar**:

Gornji red grafova dijeli y-os (jer se radi o metrikama koje su koliko-toliko srodne). Primijetite da donji desni graf ima logaritamsku y-os.

### Podzadatak 4

Vizualizirajte slučajno inicijalizirane težine konvolucijskog sloja.

**Odgovor**:

Prikazat ćemo prvi konvolucijski sloj na početku učenja (nasumično inicijaliziran) i na kraju učenja:

```python
t4_s4_image_names = sorted([x for x in os.listdir(t4_root_path) if x.endswith(".png")],
                           key=lambda x: x.lower())
t4_first_last = t4_s4_image_names[0], t4_s4_image_names[-1]
```

```python
t4_s4_figure, t4_s4_axes = plt.subplots(1, 2, figsize=(16, 8))

for i, (image_name, title) in enumerate(zip(t4_first_last,
                                            ["Početak učenja", "Kraj učenja"])):
    t4_s4_axes[i].imshow(plt.imread(os.path.join(t4_root_path, image_name)), interpolation="nearest")
    t4_s4_axes[i].set_title(title)
    t4_s4_axes[i].axis("off")
```

**Komentar**:

Jako je teško vidjeti razliku ova dva sloja. Razlog za to je što smo koristili grupnu normalizaciju (engl. *batch normalization*), pa nam L2 regularizacija nije bila potrebna. No, zbog toga što grupna normalizacija ne sređuje težine, već ima svoje parametre koje utječu na podatke koji protječu kroz mrežu, taj utjecaj nije vidljiv u samim filtrima 1. konvolucijskog sloja, već bi se nad njim trebala napraviti transformacija koja bi onda bila ekvivalentna 1. filtru konvolucijskog sloja **bez** grupne normalizacije.

Izrada takve transformacije je izvan okvira ove vježbe.

### Podzadatak 5

Prikažite $20$ netočno klasificiranih slika s najvećim gubitkom te ispišite njihov točan razred, kao i $3$ razreda za koje je model dao najveću vjerojatnost.

**Odgovor**:

Ovo ćemo postići koristeći metodu `analyze_evaluation` u modulu `task_4.py`. Također, prije snimanja normalizirat ćemo slike po uputama. To možemo učiniti koristeći rječnik koji nam vraća metoda `prepare_CIFAR` jer će ona vratiti rječnik s unosima `mean` i `std` koji predstavljaju očekivanu vrijednost i standardnu devijaciju.

Prvo moramo učitati model i prebaciti ga u mod evaluacije:

```python
t4_s5_model = torch.load(t4_model_path)
t4_s5_model.eval();
```

Zatim možemo pozvati `analyze_evaluation` nad njim i testnim skupom (testni skup je 3. član trojke koje vraća metoda `prepare_CIFAR` za `x` i `y`:

```python
analyze_evaluation(model=t4_s5_model,
                   x=t4_s2_cifar_dict["x"][2],
                   y=t4_s2_cifar_dict["y"][2],
                   mean=t4_s2_cifar_dict["mean"],
                   std=t4_s2_cifar_dict["std"])
```

    100%|██████████| 10000/10000 [00:00<00:00, 67816.98it/s]

Imena labela nalaze se u listi `cifar_labels`. Pomoću njih možemo prikazati $20$ najgorih slika:

```python
with open(t4_s5_best_classes_path) as file:
    t4_s5_best_classes = json.load(file)

with open(t4_s5_worst_classes_path) as file:
    t4_s5_worst_classes = json.load(file)
```

```python
t4_s5_n_cols = 5
t4_s5_figure, t4_s5_axes = plt.subplots(20 // t4_s5_n_cols, t4_s5_n_cols, figsize=(14, 12))

for i, image_path in enumerate(sorted([x for x in os.listdir(t4_s5_path) if x.endswith(".png")])):
    a = i // t4_s5_n_cols
    b = i % t4_s5_n_cols

    t4_s5_axes[a][b].imshow(plt.imread(os.path.join(t4_s5_path, image_path)), interpolation="nearest")
    t4_s5_axes[a][b].set_title(f"{cifar_labels[t4_s5_worst_classes[i][1]]} " +\
                               f"umjesto {cifar_labels[t4_s5_worst_classes[i][0]]}")
    t4_s5_axes[a][b].axis("off")
    t4_s5_axes[a][b].title.set_fontsize(10)
```

**Komentar**:

Vidimo da je klasifikator najnesigurniji za neke nejednoznačne slike. Neke slike je teško opravdati kao pogreške:
- $7$ (Zrakoplov umjesto Mačka)
- $9$ (Automobil umjesto Ptica)
- $12$ (Ptica umjesto Jelen)
- $13$ (Jelen umjesto Ptica)
- $15$ (Pas umjesto Konj)

Kod ovih slika bi vjerojatno pomogao veći raspon rezolucija jer se radi o slikama koje objekt prikazuju iz blizine, dakle u detaljima koji se vjerojatno veći nega za ostatak slika.

Finalno, prikazat ćemo $3$ razreda za koje mreža najbolje rasuđuje:

```python
t4_t5_display = f"<br>Mreža najtočnije rasuđuje sljedeće razrede:\n"

for class_id, prob in t4_s5_best_classes:
    t4_t5_display += f"- **{cifar_labels[class_id]}**: {prob * 100:.02f}%\n"

Markdown(t4_t5_display)
```

Mreža najtočnije rasuđuje sljedeće razrede:
- **Automobil**: 85.80%
- **Brod**: 85.40%
- **Kamion**: 83.80%


# A short & practical introduction to Tensor Flow!

Part 4

The goal of this notebook is to train a LSTM character prediction model over [Text8](http://mattmahoney.net/dc/textdata) data.

This is a personal wrap-up of all the material provided by [Google's Deep Learning course on Udacity](https://www.udacity.com/course/deep-learning--ud730), so all credit goes to them.

Author: Pablo M. Olmos (olmos@tsc.uc3m.es)

** This notebook cerntainly needs more comments to make it self-contained**

Date: March 2017

```python
# These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
from __future__ import print_function
import os
import numpy as np
import random
import string
import tensorflow as tf
import zipfile
from six.moves import range
from six.moves.urllib.request import urlretrieve
```

```python
# Lets check what version of tensorflow we have installed. The provided scripts should run with tf 1.0 and above

print(tf.__version__)
```

    1.3.0

```python
url = 'http://mattmahoney.net/dc/'

def maybe_download(filename, expected_bytes):
    """Download a file if not present, and make sure it's the right size."""
    if not os.path.exists(filename):
        filename, _ = urlretrieve(url + filename, filename)
    statinfo = os.stat(filename)
    if statinfo.st_size == expected_bytes:
        print('Found and verified %s' % filename)
    else:
        print(statinfo.st_size)
        raise Exception(
            'Failed to verify ' + filename + '. Can you get to it with a browser?')
    return filename

filename = maybe_download('../../DataSets/textWordEmbeddings/text8.zip', 31344016) ## Change according to the folder where you saved the dataset provided
```

    Found and verified ../../DataSets/textWordEmbeddings/text8.zip

```python
def read_data(filename):
    with zipfile.ZipFile(filename) as f:
        name = f.namelist()[0]
        data = tf.compat.as_str(f.read(name))
    return data

text = read_data(filename)
print('Data size %d' % len(text))
```

    Data size 100000000

```python
text[0:20]
```

    ' anarchism originate'

Create a small validation set

```python
valid_size = 1000
valid_text = text[:valid_size]
train_text = text[valid_size:]
train_size = len(train_text)
print(train_size, train_text[:64])
print(valid_size, valid_text[:64])
```

    99999000 ons anarchists advocate social relations based upon voluntary as
    1000  anarchism originated as a term of abuse first used against earl

Utility functions to map characters to vocabulary IDs and back

```python
vocabulary_size = len(string.ascii_lowercase) + 1 # [a-z] + ' '
first_letter = ord(string.ascii_lowercase[0])

def char2id(char):
    if char in string.ascii_lowercase:
        return ord(char) - first_letter + 1
    elif char == ' ':
        return 0
    else:
        print('Unexpected character: %s' % char)
        return 0
```

```python
def id2char(dictid):
    if dictid > 0:
        return chr(dictid + first_letter - 1)
    else:
        return ' '

print(char2id('a'), char2id('z'), char2id(' '), char2id('ï'))
print(id2char(1), id2char(26), id2char(0))
```

    Unexpected character: ï
    1 26 0 0
    a z

Function to generate a training batch for the LSTM model.

```python
batch_size=64      ## Number of batches, but also number of segments in which we divide the text. We read batch_size
                   ## batches in parallel, each read from a different segment. The implementation is not obvious, the
                   ## key seems to be the zip function inside the for loop below

num_unrollings=10  ## Each sequence is num_unrolling character long

### NOW I GET IT!! Every batch is a batch_size times 27 (num letters) matrix. Every row correspond to a letter. Each letter
### comes from a different sequence of (num_unrollings) so that the 64 letters cannot be read together.
## In the next batch, we have the following letter for each of the 64 training sequences!!

class BatchGenerator(object):

    def __init__(self, text, batch_size, num_unrollings):
        self._text = text
        self._text_size = len(text)
        self._batch_size = batch_size
        self._num_unrollings = num_unrollings
        segment = self._text_size // batch_size #We split the text into batch_size pieces
        self._cursor = [ offset * segment for offset in range(batch_size)]    #Cursor pointing every piece
        self._last_batch = self._next_batch()

    #
    def _next_batch(self):
        """Generate a single batch from the current cursor position in the data."""
        batch = np.zeros(shape=(self._batch_size, vocabulary_size), dtype=np.float)
        for b in range(self._batch_size):
            batch[b, char2id(self._text[self._cursor[b]])] = 1.0     #One hot encoding
            #print(self._text[self._cursor[b]])
            self._cursor[b] = (self._cursor[b] + 1) % self._text_size
        return batch

    def next(self):
        """Generate the next array of batches from the data. The array consists of
        the last batch of the previous array, followed by num_unrollings new ones.
        """
        batches = [self._last_batch]
        for step in range(self._num_unrollings):
            batches.append(self._next_batch())
        self._last_batch = batches[-1]
        return batches

def characters(probabilities):
    """Turn a 1-hot encoding or a probability distribution over the possible
    characters back into its (mostl likely) character representation."""
    return [id2char(c) for c in np.argmax(probabilities, 1)]

def batches2string(batches):
    """Convert a sequence of batches back into their (most likely) string
    representation."""
    s = [''] * batches[0].shape[0]
    for b in batches:
        s = [''.join(x) for x in zip(s, characters(b))]            #Clever! The ZIP is the key function here!
    return s

train_batches = BatchGenerator(train_text, batch_size, 10)
valid_batches = BatchGenerator(valid_text, 1, 1)

```

```python
print(batches2string(train_batches.next()))
print(batches2string(train_batches.next()))
```

    ['ons anarchi', 'when milita', 'lleria arch', ' abbeys and', 'married urr', 'hel and ric', 'y and litur', 'ay opened f', 'tion from t', 'migration t', 'new york ot', 'he boeing s', 'e listed wi', 'eber has pr', 'o be made t', 'yer who rec', 'ore signifi', 'a fierce cr', ' two six ei', 'aristotle s', 'ity can be ', ' and intrac', 'tion of the', 'dy to pass ', 'f certain d', 'at it will ', 'e convince ', 'ent told hi', 'ampaign and', 'rver side s', 'ious texts ', 'o capitaliz', 'a duplicate', 'gh ann es d', 'ine january', 'ross zero t', 'cal theorie', 'ast instanc', ' dimensiona', 'most holy m', 't s support', 'u is still ', 'e oscillati', 'o eight sub', 'of italy la', 's the tower', 'klahoma pre', 'erprise lin', 'ws becomes ', 'et in a naz', 'the fabian ', 'etchy to re', ' sharman ne', 'ised empero', 'ting in pol', 'd neo latin', 'th risky ri', 'encyclopedi', 'fense the a', 'duating fro', 'treet grid ', 'ations more', 'appeal of d', 'si have mad']
    ['ists advoca', 'ary governm', 'hes nationa', 'd monasteri', 'raca prince', 'chard baer ', 'rgical lang', 'for passeng', 'the nationa', 'took place ', 'ther well k', 'seven six s', 'ith a gloss', 'robably bee', 'to recogniz', 'ceived the ', 'icant than ', 'ritic of th', 'ight in sig', 's uncaused ', ' lost as in', 'cellular ic', 'e size of t', ' him a stic', 'drugs confu', ' take to co', ' the priest', 'im to name ', 'd barred at', 'standard fo', ' such as es', 'ze on the g', 'e of the or', 'd hiver one', 'y eight mar', 'the lead ch', 'es classica', 'ce the non ', 'al analysis', 'mormons bel', 't or at lea', ' disagreed ', 'ing system ', 'btypes base', 'anguages th', 'r commissio', 'ess one nin', 'nux suse li', ' the first ', 'zi concentr', ' society ne', 'elatively s', 'etworks sha', 'or hirohito', 'litical ini', 'n most of t', 'iskerdoo ri', 'ic overview', 'air compone', 'om acnm acc', ' centerline', 'e than any ', 'devotional ', 'de such dev']

```python
#OK with this one
def logprob(predictions, labels):
    """Log-probability of the true labels in a predicted batch."""
    predictions[predictions < 1e-10] = 1e-10
    return np.sum(np.multiply(labels, -np.log(predictions))) / labels.shape[0]

#OK with this one
def sample_distribution(distribution):
    """Sample one element from a distribution assumed to be an array of normalized
        probabilities.
    """

    r = random.uniform(0,1)
    s = 0
    for i in range(len(distribution)):
        s += distribution[i]
        if s >= r:
            return i
    return len(distribution) - 1

#OK with this one
def sample(prediction):
    """Turn a (column) prediction into 1-hot encoded samples."""
    p = np.zeros(shape=[1, vocabulary_size], dtype=np.float)
    p[0, sample_distribution(prediction[0])] = 1.0
    return p

def random_distribution():
    """Generate a random column of probabilities."""
    b = np.random.uniform(0.0, 1.0, size=[1, vocabulary_size])
    return b / np.sum(b, 1)[:, None]

```

```python
train_batches.next()[0].shape
```

    (64, 27)

# Simple LSTM Model

Recall the fundamental model

Also, the un-regularized cost function is

\begin{align}
J(\boldsymbol{\theta})=\frac{1}{N}\sum_{n=1}^N\sum_{t=1}^{T_n}d(\boldsymbol{y}_t^{(n)},\sigma(\boldsymbol{h}_t^{(n)}))
\end{align}
where $d(\cdot,\cdot)$ is the cross-entropy loss function.

About the TF implementation below, see the following excellent [post](http://www.thushv.com/sequential_modelling/long-short-term-memory-lstm-networks-implementing-with-tensorflow-part-2/)

>
Now calculating logits for softmax is a little bit tricky. This a temporal (time-based) network. So after each processing each num_unrolling batches through the LSTM cell, we update h_{t-1}=h_t and c_{t-1}=c_t before calculating logits and the loss. This is done by using tf.control_dependencies. What this does is that, logits will not be calculated until saved_output and saved_states are updated. Finally, as you can see, num_unrolling acts as the amount of history we are remembering.

In other words, in the computation graph everytime something is updated, all the dependent op nodes are updated and this is propagated through the graph. If we want to wait until the very end to compute the loss, we wait using the command tf.control_dependencies.

About the zip() and zip(*) operators, see this [post](https://docs.python.org/2/library/functions.html#zip)

```python
num_nodes = 64

graph = tf.Graph()
with graph.as_default():

    # Parameters:

    #i(t) parameters
    # Input gate: input, previous output, and bias.
    ix = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1))   ##W^ix
    im = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1)) ## W^ih
    ib = tf.Variable(tf.zeros([1, num_nodes])) ##b_i

    #f(t) parameters
    # Forget gate: input, previous output, and bias.
    fx = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1)) ##W^fx
    fm = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1)) ##W^fh
    fb = tf.Variable(tf.zeros([1, num_nodes])) ##b_f

    #g(t) parameters
    # Memory cell: input, state and bias.
    cx = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1)) ##W^gx
    cm = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1)) ##W^gh
    cb = tf.Variable(tf.zeros([1, num_nodes]))  ##b_g

    #o(t) parameters
    # Output gate: input, previous output, and bias.
    ox = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1))  ##W^ox
    om = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))  ##W^oh
    ob = tf.Variable(tf.zeros([1, num_nodes])) ##b_o

    # Variables saving state across unrollings.
    saved_output = tf.Variable(tf.zeros([batch_size, num_nodes]), trainable=False) #h(t)
    saved_state = tf.Variable(tf.zeros([batch_size, num_nodes]), trainable=False) #s(t)

    # Classifier weights and biases (over h(t) to labels)
    w = tf.Variable(tf.truncated_normal([num_nodes, vocabulary_size], -0.1, 0.1))
    b = tf.Variable(tf.zeros([vocabulary_size]))

    # Definition of the cell computation.
    def lstm_cell(i, o, state):
        """Create a LSTM cell. See e.g.: http://arxiv.org/pdf/1402.1128v1.pdf
        Note that in this formulation, we omit the various connections between the
        previous state and the gates."""
        input_gate = tf.sigmoid(tf.matmul(i, ix) + tf.matmul(o, im) + ib)
        forget_gate = tf.sigmoid(tf.matmul(i, fx) + tf.matmul(o, fm) + fb)
        update = tf.matmul(i, cx) + tf.matmul(o, cm) + cb
        state = forget_gate * state + input_gate * tf.tanh(update)    #tf.tanh(update) is g(t)
        output_gate = tf.sigmoid(tf.matmul(i, ox) + tf.matmul(o, om) + ob)
        return output_gate * tf.tanh(state), state      #h(t) is output_gate * tf.tanh(state)

    # Input data. Now it makes sense!!!

    train_data = list()
    for _ in range(num_unrollings + 1):
        train_data.append(tf.placeholder(tf.float32, shape=[batch_size,vocabulary_size]))
    train_inputs = train_data[:num_unrollings]
    train_labels = train_data[1:]  # labels are inputs shifted by one time step.

    # Unrolled LSTM loop.

    outputs = list()
    output = saved_output
    aux = output
    state = saved_state
    for i in train_inputs:
        output, state = lstm_cell(i, output, state)
        outputs.append(output)

    # State saving across unrollings.
    with tf.control_dependencies([saved_output.assign(output),saved_state.assign(state)]):
        #Classifier.
        logits = tf.nn.xw_plus_b(tf.concat(axis=0,values=outputs), w, b)
        loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=tf.concat(axis=0, values=train_labels),logits=logits))

    # Optimizer.

    """Next, we are implementing the optimizer. Remember! we should use “gradient clipping” (tf.clip_by_global_norm)
    to avoid “Exploding gradient” phenomenon. Also, we decay the learning_rate over time."""
    global_step = tf.Variable(0)

    learning_rate = tf.train.exponential_decay(10.0, global_step, 5000, 0.1, staircase=True)
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)

    """ optimizer.compute_gradients(loss) yields (gradient, value) tuples. gradients, v = zip(*optimizer.compute_gradients(loss))
    performs a transposition, creating a list of gradients and a list of values.
    gradients, _ = tf.clip_by_global_norm(gradients, 1.25)
    then clips the gradients, and optimizer = optimizer.apply_gradients(zip(gradients, v), global_step=global_step)
    re-zips the gradient and value lists back into an iterable of (gradient, value)
    tuples which is then passed to the optimizer.apply_gradients method."""

    gradients, v = zip(*optimizer.compute_gradients(loss))
    gradients, _ = tf.clip_by_global_norm(gradients, 1.25)
    optimizer = optimizer.apply_gradients(zip(gradients, v), global_step=global_step)

    # Predictions.
    train_prediction = tf.nn.softmax(logits)

    # Sampling and validation eval: batch 1, no unrolling.
    sample_input = tf.placeholder(tf.float32, shape=[1, vocabulary_size])
    saved_sample_output = tf.Variable(tf.zeros([1, num_nodes]))
    saved_sample_state = tf.Variable(tf.zeros([1, num_nodes]))
    # Create an op that groups multiple operations.
    reset_sample_state = tf.group(saved_sample_output.assign(tf.zeros([1, num_nodes])),
                                  saved_sample_state.assign(tf.zeros([1, num_nodes])))

    sample_output, sample_state = lstm_cell(sample_input, saved_sample_output, saved_sample_state)

    with tf.control_dependencies([saved_sample_output.assign(sample_output),saved_sample_state.assign(sample_state)]):
        sample_prediction = tf.nn.softmax(tf.nn.xw_plus_b(sample_output, w, b))

```

```python
num_steps = 1001
summary_frequency = 100

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    print('Initialized')
    mean_loss = 0
    for step in range(num_steps):
        batches = train_batches.next()
        feed_dict = dict()
        for i in range(num_unrollings + 1):
            feed_dict[train_data[i]] = batches[i]
        _, l, predictions, lr = session.run(
            [optimizer, loss, train_prediction, learning_rate], feed_dict=feed_dict)
        mean_loss += l
        if step % summary_frequency == 0:
            if step > 0:
                mean_loss /= summary_frequency
            # The mean loss is an estimate of the loss over the last few batches.
            print(
                'Average loss at step %d: %f learning rate: %f' % (step, mean_loss, lr))
            mean_loss = 0
            labels = np.concatenate(list(batches)[1:])
            print('Minibatch perplexity: %.2f' % float(
                np.exp(logprob(predictions, labels))))
            if step % (summary_frequency * 10) == 0:
                # Generate some samples.
                print('=' * 80)
                for _ in range(5):
                    feed = sample(random_distribution())
                    sentence = characters(feed)[0]
                    reset_sample_state.run()
                    for _ in range(79):
                        prediction = sample_prediction.eval({sample_input: feed})
                        feed = sample(prediction)
                        sentence += characters(feed)[0]
                    print(sentence)
                print('=' * 80)
            # Measure validation set perplexity.
            reset_sample_state.run()
            valid_logprob = 0
            for _ in range(valid_size):
                b = valid_batches.next()
                predictions = sample_prediction.eval({sample_input: b[0]})
                valid_logprob = valid_logprob + logprob(predictions, b[1])
            print('Validation set perplexity: %.2f' % float(np.exp(
                valid_logprob / valid_size)))
```

    Initialized
    Average loss at step 0: 3.294209 learning rate: 10.000000
    Minibatch perplexity: 26.96
    ================================================================================
    mxovr oqde yfxkdo lgo slt r jyl  uqoizrtvtz rozih o  giopvwtirvvfji ioisixishdut
    rteew vmekfznhts utnotduqtnvitnip niki eb e pv dbopsfvpm  el ittceecbnryravioe j
    ltvbgee kzaauloap jdlvnh vthfvkvqtisnr oqnr yc  gs om icihhdfriiaradwnooezke v c
    b zowgrzzlfsfin qtpnbleqteoiss pf tta jn wd ohsv  zgwsictzraulredxokf os ev qwab
    lpeottcin ffinel hxcylubmceenohedvin cm nbp tednlatrxipat  yqvwnosbgnlwmviaeruox
    ================================================================================
    Validation set perplexity: 20.10
    Average loss at step 100: 2.598391 learning rate: 10.000000
    Minibatch perplexity: 10.37
    Validation set perplexity: 10.75
    Average loss at step 200: 2.253659 learning rate: 10.000000
    Minibatch perplexity: 9.72
    Validation set perplexity: 9.11
    Average loss at step 300: 2.097530 learning rate: 10.000000
    Minibatch perplexity: 7.71
    Validation set perplexity: 7.79
    Average loss at step 400: 1.997222 learning rate: 10.000000
    Minibatch perplexity: 7.69
    Validation set perplexity: 7.62
    Average loss at step 500: 1.932652 learning rate: 10.000000
    Minibatch perplexity: 6.26
    Validation set perplexity: 7.12
    Average loss at step 600: 1.907986 learning rate: 10.000000
    Minibatch perplexity: 6.14
    Validation set perplexity: 6.84
    Average loss at step 700: 1.857451 learning rate: 10.000000
    Minibatch perplexity: 5.60
    Validation set perplexity: 6.70
    Average loss at step 800: 1.818764 learning rate: 10.000000
    Minibatch perplexity: 6.20
    Validation set perplexity: 6.71
    Average loss at step 900: 1.830710 learning rate: 10.000000
    Minibatch perplexity: 7.22
    Validation set perplexity: 6.35
    Average loss at step 1000: 1.824062 learning rate: 10.000000
    Minibatch perplexity: 5.82
    ================================================================================
    h is notsowifs wite is engliaily hurds ninekture that finch ebord of pearist sub
    riced frvmic in in hidu of the ind d finlibul the leate and nothorlod and trever
    pily hlil and ros is inferrick in llils thhirai erlic brozg o redortining clihe
    s in the vire of the ifterible frog kid a and will priling madiep precents to so
    c lith severneric sever fill listration of the impress jo the mycri five zeres c
    ================================================================================
    Validation set perplexity: 6.17

```python
batches = train_batches.next()
```

```python
batches[0]
```

    array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  1.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.]])

```python

```