```python
from sympy.core.symbol import symbols
from sympy.solvers.solveset import nonlinsolve
```

```python
x, y, z = symbols('x, y, z', real=True)
nonlinsolve([x*y - 1, 4*x**2 + y**2 - 5], [x, y])
```

    {(-1, -1), (-1/2, -2), (1/2, 2), (1, 1)}

```python
from sympy.solvers import solve, nonlinsolve
from sympy import Symbol, symbols
from sympy import sin, cos, limit
from mpmath import *
from sympy import *
```

```python
x = Symbol('x')
eq = x**2 - 1
solve([x >= 0.5, x <= 3, eq ], x)
```

    Eq(x, 1.00000000000000)

```python
x, theta, gamma = symbols('x, theta, gamma', real=True)

eq1 = sin(theta)/x

solve(eq1, [x, theta])
```

    [{theta: 0}, {theta: pi}]

```python
# theta = 0 --> cos(0) = 1
eq2 = 1 + gamma * (1 - x)

# theta = pi --> cos(pi) = -1
eq3 = -1 + gamma * (1 - x)

print(solve(eq2, [x, gamma]))
print('\n')
print(solve(eq3, [x, gamma]))
```

    [{x: (gamma + 1)/gamma}]

    [{x: (gamma - 1)/gamma}]

```python
sol3 = solve(eq3, [x, gamma])
```

```python
sol3[0][x].evalf(subs={gamma:3})
```

    0.666666666666667

```python

```


# Symbolic Quantum Computing

## Imports

```python
%matplotlib inline
```

```python
from IPython.display import display
```

```python
from sympy import init_printing
init_printing(use_latex=True)
```

```python
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin, srepr
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

## Qubits

```python
alpha, beta = symbols('alpha beta', real=True)
```

```python
psi = alpha*Qubit('00') + beta*Qubit('11'); psi
```

```python
Dagger(psi)
```

```python
qapply(Dagger(Qubit('00'))*psi)
```

```python
for state, prob in measure_all(psi):
    display(state)
    display(prob)
```

Qubits can be represented in the computational basis.

```python
represent(psi)
```

## Gates

Gate objects are the operators which act on a quantum state.

```python
g = X(0)
g
```

```python
represent(g, nqubits=2)
```

```python
c = H(0)*Qubit('00')
c
```

```python
qapply(c)
```

```python
for gate in [H,X,Y,Z,S,T]:
    for state in [Qubit('0'),Qubit('1')]:
        lhs = gate(0)*state
        rhs = qapply(lhs)
        display(Eq(lhs,rhs))
```

Symbolic gate rules and circuit simplification

```python
for g1 in (Y,Z,H):
    for g2 in (Y,Z,H):
        e = Commutator(g1(0),g2(0))
        if g1 != g2:
            display(Eq(e,e.doit()))
```

```python
c = H(0)*X(1)*H(0)**2*CNOT(0,1)*X(1)**3*X(0)*Z(1)**2
c
```

```python
circuit_plot(c, nqubits=2);
```

This performs a commutator/anticommutator aware bubble sort algorithm to simplify a circuit:

```python
gate_simp(c)
```

```python
circuit_plot(gate_simp(c),nqubits=2)
```


## Mirroring: Reversing Order of a Vector's Components

Given an $N$-dimensional euclidean vector space, the reversal of a vectors components can be produced by $k$-reflections in hyperplanes normal to $n_i$

$$ a^{'} = -(n_1 n_{2}\cdots n_{k }) a (n_k \cdots n_2 n_1 )$$

where
$$ k =  N/2 $$

and

$$ n_i = e_i-e_{N-i} $$

Note that for vector spaces with an odd number of dimensions the middle component cancels it self out. Bellow are some explicit examples, illustrating the operation for various dimensions.

```
from sympy import *
from galgebra.ga import Ga
import sympy

```

### 2D

```
(ga,e1,e2) = Ga.build('e1 e2',g=[1,1])
a=ga.mv('a','vector')

n = sqrt(2)/2*(e1-e2)
-n*a*n
```

\begin{equation*} a^{2} e_{1} + a^{1} e_{2} \end{equation*}

### 3D

```
(ga,e1,e2,e3) = Ga.build('e1 e2 e3',g=[1,1,1])
a=ga.mv('a','vector')

n = sqrt(2)/2*(e1-e3)
-n*a*n
```

\begin{equation*} a^{3} e_{1} + a^{2} e_{2} + a^{1} e_{3} \end{equation*}

### 4D

```
(ga,e1,e2,e3,e4) = Ga.build('e1 e2 e3 e4',g=[1,1,1,1])
a=ga.mv('a','vector')
n = sqrt(2)/2*(e1-e4)
m = sqrt(2)/2*(e2-e3)

m*n*a*n*m
```

\begin{equation*} a^{4} e_{1} + a^{3} e_{2} + a^{2} e_{3} + a^{1} e_{4} \end{equation*}

### 5D

```
(ga,e1,e2,e3,e4, e5) = Ga.build('e1 e2 e3 e4 e5',g=[1,1,1,1,1])
a=ga.mv('a','vector')
n = sqrt(2)/2*(e1-e5)
m = sqrt(2)/2*(e2-e4)

a_flip = m*n*a*n*m
a_flip
```

\begin{equation*} a^{5} e_{1} + a^{4} e_{2} + a^{3} e_{3} + a^{2} e_{4} + a^{1} e_{5} \end{equation*}

### 6D

```
(ga,e1,e2,e3,e4, e5, e6) = Ga.build('e1 e2 e3 e4 e5 e6',g=[1,1,1,1,1,1])
a=ga.mv('a','vector')
n = sqrt(2)/2*(e1-e6)
m = sqrt(2)/2*(e2-e5)
o = sqrt(2)/2*(e3-e4)

a_flip = -m*n*o*a*o*n*m
a_flip
```

\begin{equation*} a^{6} e_{1} + a^{5} e_{2} + a^{4} e_{3} + a^{3} e_{4} + a^{2} e_{5} + a^{1} e_{6} \end{equation*}

### Ok, lets write a function for n-D

```

def mirror(a):
    e = a.Ga.mv_basis  # vector basis
    N = len(e)       # dimension of space
    n = range(N-1)   # reflection vectors
    a_mir = a        # outputs
    for i in range(len(e)/2):
        n[i] = sqrt(2)/2* (e[i] - e[N-1-i])
        a_mir = -n[i]*a_mir*n[i]
    return a_mir

a_mir = mirror(a)
a_mir
```

\begin{equation*} a^{6} e_{1} + a^{5} e_{2} + a^{4} e_{3} + a^{3} e_{4} + a^{2} e_{5} + a^{1} e_{6} \end{equation*}

## Even and Odd modes

A discrete function of length $N$ can be thought of as an $N$-dimensional vector. Any such function may broken up into *even* and *odd* modes.

\begin{eqnarray*}
a &=& {\frac{1}{2} (a+a^{'})} + \frac{1}{2} (a-a^{'})\\
a &=&  a_e + a_o
\end{eqnarray*}

where $a^{'}$ is the mirrored version of $a$, as described above.

```
half = Rational(1,2)
even = lambda a: half*(a+mirror(a))
odd = lambda a: half*(a-mirror(a))
```

### 2D

```
(ga,e1,e2) = Ga.build('e1 e2',g=[1,1])

a = ga.mv('a','vector')
a_e = even(a)
a_e
```

\begin{equation*} \left ( \frac{a^{1}}{2} + \frac{a^{2}}{2}\right ) e_{1} + \left ( \frac{a^{1}}{2} + \frac{a^{2}}{2}\right ) e_{2} \end{equation*}

```
a_o = odd(a)
a_o
```

\begin{equation*} \left ( \frac{a^{1}}{2} - \frac{a^{2}}{2}\right ) e_{1} + \left ( - \frac{a^{1}}{2} + \frac{a^{2}}{2}\right ) e_{2} \end{equation*}


### Markov decision process

This week's methods are all built to solve __M__arkov __D__ecision __P__rocesses. In the broadest sense, an MDP is defined by how it changes states and how rewards are computed.

State transition is defined by $P(s' |s,a)$ - how likely are you to end at state $s'$ if you take action $a$ from state $s$. Now there's more than one way to define rewards, but we'll use $r(s,a,s')$ function for convenience.

_This notebook is inspired by the awesome_ [CS294](https://github.com/berkeleydeeprlcourse/homework/blob/36a0b58261acde756abd55306fbe63df226bf62b/hw2/HW2.ipynb) _by Berkeley_

For starters, let's define a simple MDP from this picture:

```python
import sys, os
if 'google.colab' in sys.modules and not os.path.exists('.setup_complete'):
    !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/master/setup_colab.sh -O- | bash
    !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/master/week02_value_based/mdp.py
    !touch .setup_complete

# This code creates a virtual display to draw game images on.
# It will have no effect if your machine has a monitor.
if type(os.environ.get("DISPLAY")) is not str or len(os.environ.get("DISPLAY")) == 0:
    !bash ../xvfb start
    os.environ['DISPLAY'] = ':1'
```

```python
transition_probs = {
    's0': {
        'a0': {'s0': 0.5, 's2': 0.5},
        'a1': {'s2': 1}
    },
    's1': {
        'a0': {'s0': 0.7, 's1': 0.1, 's2': 0.2},
        'a1': {'s1': 0.95, 's2': 0.05}
    },
    's2': {
        'a0': {'s0': 0.4, 's2': 0.6},
        'a1': {'s0': 0.3, 's1': 0.3, 's2': 0.4}
    }
}
rewards = {
    's1': {'a0': {'s0': +5}},
    's2': {'a1': {'s0': -1}}
}

from mdp import MDP
mdp = MDP(transition_probs, rewards, initial_state='s0')
```

We can now use MDP just as any other gym environment:

```python
print('initial state =', mdp.reset())
next_state, reward, done, info = mdp.step('a1')
print('next_state = %s, reward = %s, done = %s' % (next_state, reward, done))
```

    initial state = s0
    next_state = s2, reward = 0.0, done = False

but it also has other methods that you'll need for Value Iteration

```python
print("mdp.get_all_states =", mdp.get_all_states())
print("mdp.get_possible_actions('s1') = ", mdp.get_possible_actions('s1'))
print("mdp.get_next_states('s1', 'a0') = ", mdp.get_next_states('s1', 'a0'))
print("mdp.get_reward('s1', 'a0', 's0') = ", mdp.get_reward('s1', 'a0', 's0'))
print("mdp.get_transition_prob('s1', 'a0', 's0') = ", mdp.get_transition_prob('s1', 'a0', 's0'))
```

    mdp.get_all_states = ('s0', 's1', 's2')
    mdp.get_possible_actions('s1') =  ('a0', 'a1')
    mdp.get_next_states('s1', 'a0') =  {'s0': 0.7, 's1': 0.1, 's2': 0.2}
    mdp.get_reward('s1', 'a0', 's0') =  5
    mdp.get_transition_prob('s1', 'a0', 's0') =  0.7

### Optional: Visualizing MDPs

You can also visualize any MDP with the drawing fuction donated by [neer201](https://github.com/neer201).

You have to install graphviz for system and for python.

1. * For ubuntu just run: `sudo apt-get install graphviz`
   * For OSX: `brew install graphviz`
2. `pip install graphviz`
3. restart the notebook

__Note:__ Installing graphviz on some OS (esp. Windows) may be tricky. However, you can ignore this part alltogether and use the standart vizualization.

```python
from mdp import has_graphviz
from IPython.display import display
print("Graphviz available:", has_graphviz)
```

    Graphviz available: True

```python
if has_graphviz:
    from mdp import plot_graph, plot_graph_with_state_values, plot_graph_optimal_strategy_and_state_values
    display(plot_graph(mdp, graph_size="50,50"))
```

### Value Iteration

Now let's build something to solve this MDP. The simplest algorithm so far is __V__alue __I__teration

Here's the pseudo-code for VI:

---

`1.` Initialize $V^{(0)}(s)=0$, for all $s$

`2.` For $i=0, 1, 2, \dots$

`3.` $ \quad V_{(i+1)}(s) = \max_a \sum_{s'} P(s' | s,a) \cdot [ r(s,a,s') + \gamma V_{i}(s')]$, for all $s$

---

First, let's write a function to compute the state-action value function $Q^{\pi}$, defined as follows

$$Q_i(s, a) = \sum_{s'} P(s' | s,a) \cdot [ r(s,a,s') + \gamma V_{i}(s')]$$

```python
def get_action_value(mdp, state_values, state, action, gamma):
    """ Computes Q(s,a) as in formula above """

    v = 0
    for next_state in mdp.get_next_states(state, action):
        v += (mdp.get_transition_prob(state, action, next_state)
              * (mdp.get_reward(state, action, next_state) + gamma * state_values[next_state]))

    return v
```

```python
import numpy as np
test_Vs = {s: i for i, s in enumerate(sorted(mdp.get_all_states()))}
assert np.isclose(get_action_value(mdp, test_Vs, 's2', 'a1', 0.9), 0.69)
assert np.isclose(get_action_value(mdp, test_Vs, 's1', 'a0', 0.9), 3.95)
```

Using $Q(s,a)$ we can now define the "next" V(s) for value iteration.
 $$V_{(i+1)}(s) = \max_a \sum_{s'} P(s' | s,a) \cdot [ r(s,a,s') + \gamma V_{i}(s')] = \max_a Q_i(s,a)$$

```python
def get_new_state_value(mdp, state_values, state, gamma):
    """ Computes next V(s) as in formula above. Please do not change state_values in process. """
    if mdp.is_terminal(state):
        return 0

    v = -1e30
    action = None
    for possible_action in mdp.get_possible_actions(state):
        cur_v = get_action_value(mdp, state_values, state, possible_action, gamma)
        if cur_v > v:
            v = cur_v
            action = possible_action

    return v
```

```python
test_Vs_copy = dict(test_Vs)
assert np.isclose(get_new_state_value(mdp, test_Vs, 's0', 0.9), 1.8)
assert np.isclose(get_new_state_value(mdp, test_Vs, 's2', 0.9), 1.08)
assert np.isclose(get_new_state_value(mdp, {'s0': -1e10, 's1': 0, 's2': -2e10}, 's0', 0.9), -13500000000.0), \
    "Please ensure that you handle negative Q-values of arbitrary magnitude correctly"
assert test_Vs == test_Vs_copy, "Please do not change state_values in get_new_state_value"
```

Finally, let's combine everything we wrote into a working value iteration algo.

```python
# parameters
gamma = 0.9            # discount for MDP
num_iter = 100         # maximum iterations, excluding initialization
# stop VI if new values are this close to old values (or closer)
min_difference = 0.001

# initialize V(s)
state_values = {s: 0 for s in mdp.get_all_states()}

if has_graphviz:
    display(plot_graph_with_state_values(mdp, state_values))

for i in range(num_iter):

    # Compute new state values using the functions you defined above.
    # It must be a dict {state : float V_new(state)}
    new_state_values = {s: get_new_state_value(mdp, state_values, s, gamma) for s in state_values.keys()}

    assert isinstance(new_state_values, dict)

    # Compute difference
    diff = max(abs(new_state_values[s] - state_values[s])
               for s in mdp.get_all_states())
    print("iter %4i   |   diff: %6.5f   |   " % (i, diff), end="")
    print('   '.join("V(%s) = %.3f" % (s, v) for s, v in state_values.items()))
    state_values = new_state_values

    if diff < min_difference:
        print("Terminated")
        break
```

    iter    0   |   diff: 3.50000   |   V(s0) = 0.000   V(s1) = 0.000   V(s2) = 0.000
    iter    1   |   diff: 0.64500   |   V(s0) = 0.000   V(s1) = 3.500   V(s2) = 0.000
    iter    2   |   diff: 0.58050   |   V(s0) = 0.000   V(s1) = 3.815   V(s2) = 0.645
    iter    3   |   diff: 0.43582   |   V(s0) = 0.581   V(s1) = 3.959   V(s2) = 0.962
    iter    4   |   diff: 0.30634   |   V(s0) = 0.866   V(s1) = 4.395   V(s2) = 1.272
    iter    5   |   diff: 0.27571   |   V(s0) = 1.145   V(s1) = 4.670   V(s2) = 1.579
    iter    6   |   diff: 0.24347   |   V(s0) = 1.421   V(s1) = 4.926   V(s2) = 1.838
    iter    7   |   diff: 0.21419   |   V(s0) = 1.655   V(s1) = 5.169   V(s2) = 2.075
    iter    8   |   diff: 0.19277   |   V(s0) = 1.868   V(s1) = 5.381   V(s2) = 2.290
    iter    9   |   diff: 0.17327   |   V(s0) = 2.061   V(s1) = 5.573   V(s2) = 2.481
    iter   10   |   diff: 0.15569   |   V(s0) = 2.233   V(s1) = 5.746   V(s2) = 2.654
    iter   11   |   diff: 0.14012   |   V(s0) = 2.389   V(s1) = 5.902   V(s2) = 2.810
    iter   12   |   diff: 0.12610   |   V(s0) = 2.529   V(s1) = 6.042   V(s2) = 2.950
    iter   13   |   diff: 0.11348   |   V(s0) = 2.655   V(s1) = 6.168   V(s2) = 3.076
    iter   14   |   diff: 0.10213   |   V(s0) = 2.769   V(s1) = 6.282   V(s2) = 3.190
    iter   15   |   diff: 0.09192   |   V(s0) = 2.871   V(s1) = 6.384   V(s2) = 3.292
    iter   16   |   diff: 0.08272   |   V(s0) = 2.963   V(s1) = 6.476   V(s2) = 3.384
    iter   17   |   diff: 0.07445   |   V(s0) = 3.045   V(s1) = 6.558   V(s2) = 3.467
    iter   18   |   diff: 0.06701   |   V(s0) = 3.120   V(s1) = 6.633   V(s2) = 3.541
    iter   19   |   diff: 0.06031   |   V(s0) = 3.187   V(s1) = 6.700   V(s2) = 3.608
    iter   20   |   diff: 0.05428   |   V(s0) = 3.247   V(s1) = 6.760   V(s2) = 3.668
    iter   21   |   diff: 0.04885   |   V(s0) = 3.301   V(s1) = 6.814   V(s2) = 3.723
    iter   22   |   diff: 0.04396   |   V(s0) = 3.350   V(s1) = 6.863   V(s2) = 3.771
    iter   23   |   diff: 0.03957   |   V(s0) = 3.394   V(s1) = 6.907   V(s2) = 3.815
    iter   24   |   diff: 0.03561   |   V(s0) = 3.434   V(s1) = 6.947   V(s2) = 3.855
    iter   25   |   diff: 0.03205   |   V(s0) = 3.469   V(s1) = 6.982   V(s2) = 3.891
    iter   26   |   diff: 0.02884   |   V(s0) = 3.502   V(s1) = 7.014   V(s2) = 3.923
    iter   27   |   diff: 0.02596   |   V(s0) = 3.530   V(s1) = 7.043   V(s2) = 3.951
    iter   28   |   diff: 0.02336   |   V(s0) = 3.556   V(s1) = 7.069   V(s2) = 3.977
    iter   29   |   diff: 0.02103   |   V(s0) = 3.580   V(s1) = 7.093   V(s2) = 4.001
    iter   30   |   diff: 0.01892   |   V(s0) = 3.601   V(s1) = 7.114   V(s2) = 4.022
    iter   31   |   diff: 0.01703   |   V(s0) = 3.620   V(s1) = 7.133   V(s2) = 4.041
    iter   32   |   diff: 0.01533   |   V(s0) = 3.637   V(s1) = 7.150   V(s2) = 4.058
    iter   33   |   diff: 0.01380   |   V(s0) = 3.652   V(s1) = 7.165   V(s2) = 4.073
    iter   34   |   diff: 0.01242   |   V(s0) = 3.666   V(s1) = 7.179   V(s2) = 4.087
    iter   35   |   diff: 0.01117   |   V(s0) = 3.678   V(s1) = 7.191   V(s2) = 4.099
    iter   36   |   diff: 0.01006   |   V(s0) = 3.689   V(s1) = 7.202   V(s2) = 4.110
    iter   37   |   diff: 0.00905   |   V(s0) = 3.699   V(s1) = 7.212   V(s2) = 4.121
    iter   38   |   diff: 0.00815   |   V(s0) = 3.708   V(s1) = 7.221   V(s2) = 4.130
    iter   39   |   diff: 0.00733   |   V(s0) = 3.717   V(s1) = 7.230   V(s2) = 4.138
    iter   40   |   diff: 0.00660   |   V(s0) = 3.724   V(s1) = 7.237   V(s2) = 4.145
    iter   41   |   diff: 0.00594   |   V(s0) = 3.731   V(s1) = 7.244   V(s2) = 4.152
    iter   42   |   diff: 0.00534   |   V(s0) = 3.736   V(s1) = 7.249   V(s2) = 4.158
    iter   43   |   diff: 0.00481   |   V(s0) = 3.742   V(s1) = 7.255   V(s2) = 4.163
    iter   44   |   diff: 0.00433   |   V(s0) = 3.747   V(s1) = 7.260   V(s2) = 4.168
    iter   45   |   diff: 0.00390   |   V(s0) = 3.751   V(s1) = 7.264   V(s2) = 4.172
    iter   46   |   diff: 0.00351   |   V(s0) = 3.755   V(s1) = 7.268   V(s2) = 4.176
    iter   47   |   diff: 0.00316   |   V(s0) = 3.758   V(s1) = 7.271   V(s2) = 4.179
    iter   48   |   diff: 0.00284   |   V(s0) = 3.762   V(s1) = 7.275   V(s2) = 4.183
    iter   49   |   diff: 0.00256   |   V(s0) = 3.764   V(s1) = 7.277   V(s2) = 4.185
    iter   50   |   diff: 0.00230   |   V(s0) = 3.767   V(s1) = 7.280   V(s2) = 4.188
    iter   51   |   diff: 0.00207   |   V(s0) = 3.769   V(s1) = 7.282   V(s2) = 4.190
    iter   52   |   diff: 0.00186   |   V(s0) = 3.771   V(s1) = 7.284   V(s2) = 4.192
    iter   53   |   diff: 0.00168   |   V(s0) = 3.773   V(s1) = 7.286   V(s2) = 4.194
    iter   54   |   diff: 0.00151   |   V(s0) = 3.775   V(s1) = 7.288   V(s2) = 4.196
    iter   55   |   diff: 0.00136   |   V(s0) = 3.776   V(s1) = 7.289   V(s2) = 4.197
    iter   56   |   diff: 0.00122   |   V(s0) = 3.778   V(s1) = 7.291   V(s2) = 4.199
    iter   57   |   diff: 0.00110   |   V(s0) = 3.779   V(s1) = 7.292   V(s2) = 4.200
    iter   58   |   diff: 0.00099   |   V(s0) = 3.780   V(s1) = 7.293   V(s2) = 4.201
    Terminated

```python
if has_graphviz:
    display(plot_graph_with_state_values(mdp, state_values))
```

```python
print("Final state values:", state_values)

assert abs(state_values['s0'] - 3.781) < 0.01
assert abs(state_values['s1'] - 7.294) < 0.01
assert abs(state_values['s2'] - 4.202) < 0.01
```

    Final state values: {'s0': 3.7810348735476405, 's1': 7.294006423867229, 's2': 4.202140275227048}

Now let's use those $V^{*}(s)$ to find optimal actions in each state

 $$\pi^*(s) = argmax_a \sum_{s'} P(s' | s,a) \cdot [ r(s,a,s') + \gamma V_{i}(s')] = argmax_a Q_i(s,a)$$

The only difference vs V(s) is that here we take not max but argmax: find action such with maximum Q(s,a).

```python
def get_optimal_action(mdp, state_values, state, gamma=0.9):
    """ Finds optimal action using formula above. """
    if mdp.is_terminal(state):
        return None

    v = -1e30
    action = None
    for possible_action in mdp.get_possible_actions(state):
        cur_v = get_action_value(mdp, state_values, state, possible_action, gamma)
        if cur_v > v:
            v = cur_v
            action = possible_action

    return action
```

```python
assert get_optimal_action(mdp, state_values, 's0', gamma) == 'a1'
assert get_optimal_action(mdp, state_values, 's1', gamma) == 'a0'
assert get_optimal_action(mdp, state_values, 's2', gamma) == 'a1'

assert get_optimal_action(mdp, {'s0': -1e10, 's1': 0, 's2': -2e10}, 's0', 0.9) == 'a0', \
    "Please ensure that you handle negative Q-values of arbitrary magnitude correctly"
assert get_optimal_action(mdp, {'s0': -2e10, 's1': 0, 's2': -1e10}, 's0', 0.9) == 'a1', \
    "Please ensure that you handle negative Q-values of arbitrary magnitude correctly"
```

```python
if has_graphviz:
    display(plot_graph_optimal_strategy_and_state_values(mdp, state_values, get_action_value))
```

```python
# Measure agent's average reward

s = mdp.reset()
rewards = []
for _ in range(10000):
    s, r, done, _ = mdp.step(get_optimal_action(mdp, state_values, s, gamma))
    rewards.append(r)

print("average reward: ", np.mean(rewards))

assert(0.40 < np.mean(rewards) < 0.55)
```

    average reward:  0.442

### Frozen lake

```python
from mdp import FrozenLakeEnv
mdp = FrozenLakeEnv(slip_chance=0)

mdp.render()
```

    *FFF
    FHFH
    FFFH
    HFFG

```python
def value_iteration(mdp, state_values=None, gamma=0.9, num_iter=1000, min_difference=1e-5, verbose=True):
    """ performs num_iter value iteration steps starting from state_values. Same as before but in a function """
    state_values = state_values or {s: 0 for s in mdp.get_all_states()}
    it = 0
    for i in range(num_iter):

        # Compute new state values using the functions you defined above. It must be a dict {state : new_V(state)}
        new_state_values = {s: get_new_state_value(mdp, state_values, s, gamma) for s in state_values.keys()}

        assert isinstance(new_state_values, dict)

        # Compute difference
        diff = max(abs(new_state_values[s] - state_values[s])
                   for s in mdp.get_all_states())

        if verbose:
            print("iter %4i   |   diff: %6.5f   |   V(start): %.3f " %
                  (i, diff, new_state_values[mdp._initial_state]))

        state_values = new_state_values
        it += 1
        if diff < min_difference:
            break

    return state_values, it
```

```python
state_values, _ = value_iteration(mdp)
```

    iter    0   |   diff: 1.00000   |   V(start): 0.000
    iter    1   |   diff: 0.90000   |   V(start): 0.000
    iter    2   |   diff: 0.81000   |   V(start): 0.000
    iter    3   |   diff: 0.72900   |   V(start): 0.000
    iter    4   |   diff: 0.65610   |   V(start): 0.000
    iter    5   |   diff: 0.59049   |   V(start): 0.590
    iter    6   |   diff: 0.00000   |   V(start): 0.590

```python
s = mdp.reset()
mdp.render()
for t in range(100):
    a = get_optimal_action(mdp, state_values, s, gamma)
    print(a, end='\n\n')
    s, r, done, _ = mdp.step(a)
    mdp.render()
    if done:
        break
```

    *FFF
    FHFH
    FFFH
    HFFG

    down

    SFFF
    *HFH
    FFFH
    HFFG

    down

    SFFF
    FHFH
    *FFH
    HFFG

    right

    SFFF
    FHFH
    F*FH
    HFFG

    down

    SFFF
    FHFH
    FFFH
    H*FG

    right

    SFFF
    FHFH
    FFFH
    HF*G

    right

    SFFF
    FHFH
    FFFH
    HFF*

### Let's visualize!

It's usually interesting to see what your algorithm actually learned under the hood. To do so, we'll plot state value functions and optimal actions at each VI step.

```python
import matplotlib.pyplot as plt
%matplotlib inline

def draw_policy(mdp, state_values):
    plt.figure(figsize=(3, 3))
    h, w = mdp.desc.shape
    states = sorted(mdp.get_all_states())
    V = np.array([state_values[s] for s in states])
    Pi = {s: get_optimal_action(mdp, state_values, s, gamma) for s in states}
    plt.imshow(V.reshape(w, h), cmap='gray', interpolation='none', clim=(0, 1))
    ax = plt.gca()
    ax.set_xticks(np.arange(h)-.5)
    ax.set_yticks(np.arange(w)-.5)
    ax.set_xticklabels([])
    ax.set_yticklabels([])
    Y, X = np.mgrid[0:4, 0:4]
    a2uv = {'left': (-1, 0), 'down': (0, -1), 'right': (1, 0), 'up': (0, 1)}
    for y in range(h):
        for x in range(w):
            plt.text(x, y, str(mdp.desc[y, x].item()),
                     color='g', size=12,  verticalalignment='center',
                     horizontalalignment='center', fontweight='bold')
            a = Pi[y, x]
            if a is None:
                continue
            u, v = a2uv[a]
            plt.arrow(x, y, u*.3, -v*.3, color='m',
                      head_width=0.1, head_length=0.1)
    plt.grid(color='b', lw=2, ls='-')
    plt.show()
```

```python
state_values = {s: 0 for s in mdp.get_all_states()}

for i in range(10):
    print("after iteration %i" % i)
    state_values, _ = value_iteration(mdp, state_values, num_iter=1)
    draw_policy(mdp, state_values)
# please ignore iter 0 at each step
```

```python
from IPython.display import clear_output
from time import sleep
mdp = FrozenLakeEnv(map_name='8x8', slip_chance=0.1)
state_values = {s: 0 for s in mdp.get_all_states()}

for i in range(30):
    clear_output(True)
    print("after iteration %i" % i)
    state_values, _ = value_iteration(mdp, state_values, num_iter=1)
    draw_policy(mdp, state_values)
    sleep(0.5)
# please ignore iter 0 at each step
```

Massive tests

```python
mdp = FrozenLakeEnv(slip_chance=0)
state_values, _ = value_iteration(mdp)

total_rewards = []
for game_i in range(1000):
    s = mdp.reset()
    rewards = []
    for t in range(100):
        s, r, done, _ = mdp.step(
            get_optimal_action(mdp, state_values, s, gamma))
        rewards.append(r)
        if done:
            break
    total_rewards.append(np.sum(rewards))

print("average reward: ", np.mean(total_rewards))
assert(1.0 <= np.mean(total_rewards) <= 1.0)
print("Well done!")
```

    iter    0   |   diff: 1.00000   |   V(start): 0.000
    iter    1   |   diff: 0.90000   |   V(start): 0.000
    iter    2   |   diff: 0.81000   |   V(start): 0.000
    iter    3   |   diff: 0.72900   |   V(start): 0.000
    iter    4   |   diff: 0.65610   |   V(start): 0.000
    iter    5   |   diff: 0.59049   |   V(start): 0.590
    iter    6   |   diff: 0.00000   |   V(start): 0.590
    average reward:  1.0
    Well done!

```python
# Measure agent's average reward
mdp = FrozenLakeEnv(slip_chance=0.1)
state_values, _ = value_iteration(mdp)

total_rewards = []
for game_i in range(1000):
    s = mdp.reset()
    rewards = []
    for t in range(100):
        s, r, done, _ = mdp.step(
            get_optimal_action(mdp, state_values, s, gamma))
        rewards.append(r)
        if done:
            break
    total_rewards.append(np.sum(rewards))

print("average reward: ", np.mean(total_rewards))
assert(0.8 <= np.mean(total_rewards) <= 0.95)
print("Well done!")
```

    iter    0   |   diff: 0.90000   |   V(start): 0.000
    iter    1   |   diff: 0.72900   |   V(start): 0.000
    iter    2   |   diff: 0.62330   |   V(start): 0.000
    iter    3   |   diff: 0.50487   |   V(start): 0.000
    iter    4   |   diff: 0.40894   |   V(start): 0.000
    iter    5   |   diff: 0.34868   |   V(start): 0.349
    iter    6   |   diff: 0.06529   |   V(start): 0.410
    iter    7   |   diff: 0.05832   |   V(start): 0.468
    iter    8   |   diff: 0.01139   |   V(start): 0.480
    iter    9   |   diff: 0.00764   |   V(start): 0.487
    iter   10   |   diff: 0.00164   |   V(start): 0.489
    iter   11   |   diff: 0.00094   |   V(start): 0.490
    iter   12   |   diff: 0.00022   |   V(start): 0.490
    iter   13   |   diff: 0.00011   |   V(start): 0.490
    iter   14   |   diff: 0.00003   |   V(start): 0.490
    iter   15   |   diff: 0.00001   |   V(start): 0.490
    iter   16   |   diff: 0.00000   |   V(start): 0.490
    average reward:  0.873
    Well done!

```python
# Measure agent's average reward
mdp = FrozenLakeEnv(slip_chance=0.25)
state_values, _ = value_iteration(mdp)

total_rewards = []
for game_i in range(1000):
    s = mdp.reset()
    rewards = []
    for t in range(100):
        s, r, done, _ = mdp.step(
            get_optimal_action(mdp, state_values, s, gamma))
        rewards.append(r)
        if done:
            break
    total_rewards.append(np.sum(rewards))

print("average reward: ", np.mean(total_rewards))
assert(0.6 <= np.mean(total_rewards) <= 0.7)
print("Well done!")
```

    iter    0   |   diff: 0.75000   |   V(start): 0.000
    iter    1   |   diff: 0.50625   |   V(start): 0.000
    iter    2   |   diff: 0.39867   |   V(start): 0.000
    iter    3   |   diff: 0.26910   |   V(start): 0.000
    iter    4   |   diff: 0.18164   |   V(start): 0.000
    iter    5   |   diff: 0.14013   |   V(start): 0.140
    iter    6   |   diff: 0.07028   |   V(start): 0.199
    iter    7   |   diff: 0.06030   |   V(start): 0.260
    iter    8   |   diff: 0.02594   |   V(start): 0.285
    iter    9   |   diff: 0.01918   |   V(start): 0.305
    iter   10   |   diff: 0.00858   |   V(start): 0.313
    iter   11   |   diff: 0.00560   |   V(start): 0.319
    iter   12   |   diff: 0.00260   |   V(start): 0.321
    iter   13   |   diff: 0.00159   |   V(start): 0.323
    iter   14   |   diff: 0.00076   |   V(start): 0.324
    iter   15   |   diff: 0.00045   |   V(start): 0.324
    iter   16   |   diff: 0.00022   |   V(start): 0.324
    iter   17   |   diff: 0.00012   |   V(start): 0.325
    iter   18   |   diff: 0.00006   |   V(start): 0.325
    iter   19   |   diff: 0.00003   |   V(start): 0.325
    iter   20   |   diff: 0.00002   |   V(start): 0.325
    iter   21   |   diff: 0.00001   |   V(start): 0.325
    average reward:  0.638
    Well done!

```python
# Measure agent's average reward
mdp = FrozenLakeEnv(slip_chance=0.2, map_name='8x8')
state_values, _ = value_iteration(mdp)

total_rewards = []
for game_i in range(1000):
    s = mdp.reset()
    rewards = []
    for t in range(100):
        s, r, done, _ = mdp.step(
            get_optimal_action(mdp, state_values, s, gamma))
        rewards.append(r)
        if done:
            break
    total_rewards.append(np.sum(rewards))

print("average reward: ", np.mean(total_rewards))
assert(0.6 <= np.mean(total_rewards) <= 0.8)
print("Well done!")
```

    iter    0   |   diff: 0.80000   |   V(start): 0.000
    iter    1   |   diff: 0.57600   |   V(start): 0.000
    iter    2   |   diff: 0.41472   |   V(start): 0.000
    iter    3   |   diff: 0.29860   |   V(start): 0.000
    iter    4   |   diff: 0.24186   |   V(start): 0.000
    iter    5   |   diff: 0.19349   |   V(start): 0.000
    iter    6   |   diff: 0.15325   |   V(start): 0.000
    iter    7   |   diff: 0.12288   |   V(start): 0.000
    iter    8   |   diff: 0.09930   |   V(start): 0.000
    iter    9   |   diff: 0.08037   |   V(start): 0.000
    iter   10   |   diff: 0.06426   |   V(start): 0.000
    iter   11   |   diff: 0.05129   |   V(start): 0.000
    iter   12   |   diff: 0.04330   |   V(start): 0.000
    iter   13   |   diff: 0.03802   |   V(start): 0.033
    iter   14   |   diff: 0.03332   |   V(start): 0.058
    iter   15   |   diff: 0.02910   |   V(start): 0.087
    iter   16   |   diff: 0.01855   |   V(start): 0.106
    iter   17   |   diff: 0.01403   |   V(start): 0.120
    iter   18   |   diff: 0.00810   |   V(start): 0.128
    iter   19   |   diff: 0.00555   |   V(start): 0.133
    iter   20   |   diff: 0.00321   |   V(start): 0.137
    iter   21   |   diff: 0.00247   |   V(start): 0.138
    iter   22   |   diff: 0.00147   |   V(start): 0.139
    iter   23   |   diff: 0.00104   |   V(start): 0.140
    iter   24   |   diff: 0.00058   |   V(start): 0.140
    iter   25   |   diff: 0.00036   |   V(start): 0.141
    iter   26   |   diff: 0.00024   |   V(start): 0.141
    iter   27   |   diff: 0.00018   |   V(start): 0.141
    iter   28   |   diff: 0.00012   |   V(start): 0.141
    iter   29   |   diff: 0.00007   |   V(start): 0.141
    iter   30   |   diff: 0.00004   |   V(start): 0.141
    iter   31   |   diff: 0.00003   |   V(start): 0.141
    iter   32   |   diff: 0.00001   |   V(start): 0.141
    iter   33   |   diff: 0.00001   |   V(start): 0.141
    average reward:  0.756
    Well done!

# HW Part 1: Value iteration convergence

### Find an MDP for which value iteration takes long to converge  (0.5 pts)

When we ran value iteration on the small frozen lake problem, the last iteration where an action changed was iteration 6--i.e., value iteration computed the optimal policy at iteration 6. Are there any guarantees regarding how many iterations it'll take value iteration to compute the optimal policy? There are no such guarantees without additional assumptions--we can construct the MDP in such a way that the greedy policy will change after arbitrarily many iterations.

Your task: define an MDP with at most 3 states and 2 actions, such that when you run value iteration, the optimal action changes at iteration >= 50. Use discount=0.95. (However, note that the discount doesn't matter here--you can construct an appropriate MDP with any discount.)

Note: value function must change at least once after iteration >=50, not necessarily change on every iteration till >=50.

```python
transition_probs = {
    's0': {
        'a0': {'s0': 0.999, 's1': 0.001}
    },
    's1': {
        'a0': {'s1': 1},
        'a1': {'s1': 0.98435, 's2': 0.01565}  # very carefully trial and error :)
    },
    's2': {
        'a0': {'s2': 1}
    }
}
rewards = {
    's0': {'a0': {'s0': -1, 's1': +1e3}},
    's1': {'a0': {'s1': +1e3}, 'a1': {'s1': -5e2, 's2': +1e4}},
    's2': {'a0': {'s2': +1e4}}
}

gamma = 0.95

from mdp import MDP
from numpy import random
mdp_bonus = MDP(transition_probs, rewards, initial_state='s0')
# Feel free to change the initial_state
```

```python
if has_graphviz:
    display(plot_graph(mdp_bonus, graph_size="50,50"))
```

```python
state_values = {s: 0 for s in mdp_bonus.get_all_states()}
policy = np.array([get_optimal_action(mdp_bonus, state_values, state, gamma)
                   for state in sorted(mdp_bonus.get_all_states())])

for i in range(100):
    print("after iteration %i" % i)
    state_values, _ = value_iteration(mdp_bonus, state_values, num_iter=1)

    print(policy)
    new_policy = np.array([get_optimal_action(mdp_bonus, state_values, state, gamma)
                           for state in sorted(mdp_bonus.get_all_states())])

    n_changes = (policy != new_policy).sum()
    print("N actions changed = %i \n" % n_changes)
    policy = new_policy

# please ignore iter 0 at each step
```

    after iteration 0
    iter    0   |   diff: 10000.00000   |   V(start): 0.001
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 1
    iter    0   |   diff: 9000.00000   |   V(start): 0.902
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 2
    iter    0   |   diff: 8100.00000   |   V(start): 2.522
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 3
    iter    0   |   diff: 7290.00000   |   V(start): 4.707
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 4
    iter    0   |   diff: 6561.00000   |   V(start): 7.329
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 5
    iter    0   |   diff: 5904.90000   |   V(start): 10.276
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 6
    iter    0   |   diff: 5314.41000   |   V(start): 13.457
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 7
    iter    0   |   diff: 4782.96900   |   V(start): 16.795
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 8
    iter    0   |   diff: 4304.67210   |   V(start): 20.228
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 9
    iter    0   |   diff: 3874.20489   |   V(start): 23.701
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 10
    iter    0   |   diff: 3486.78440   |   V(start): 27.172
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 11
    iter    0   |   diff: 3138.10596   |   V(start): 30.607
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 12
    iter    0   |   diff: 2824.29536   |   V(start): 33.978
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 13
    iter    0   |   diff: 2541.86583   |   V(start): 37.263
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 14
    iter    0   |   diff: 2287.67925   |   V(start): 40.445
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 15
    iter    0   |   diff: 2058.91132   |   V(start): 43.512
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 16
    iter    0   |   diff: 1853.02019   |   V(start): 46.455
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 17
    iter    0   |   diff: 1667.71817   |   V(start): 49.268
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 18
    iter    0   |   diff: 1500.94635   |   V(start): 51.947
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 19
    iter    0   |   diff: 1350.85172   |   V(start): 54.491
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 20
    iter    0   |   diff: 1215.76655   |   V(start): 56.899
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 21
    iter    0   |   diff: 1094.18989   |   V(start): 59.175
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 22
    iter    0   |   diff: 984.77090   |   V(start): 61.319
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 23
    iter    0   |   diff: 886.29381   |   V(start): 63.335
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 24
    iter    0   |   diff: 797.66443   |   V(start): 65.227
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 25
    iter    0   |   diff: 717.89799   |   V(start): 67.001
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 26
    iter    0   |   diff: 646.10819   |   V(start): 68.660
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 27
    iter    0   |   diff: 581.49737   |   V(start): 70.210
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 28
    iter    0   |   diff: 523.34763   |   V(start): 71.656
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 29
    iter    0   |   diff: 471.01287   |   V(start): 73.003
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 30
    iter    0   |   diff: 423.91158   |   V(start): 74.256
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 31
    iter    0   |   diff: 381.52042   |   V(start): 75.421
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 32
    iter    0   |   diff: 343.36838   |   V(start): 76.503
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 33
    iter    0   |   diff: 309.03154   |   V(start): 77.507
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 34
    iter    0   |   diff: 278.12839   |   V(start): 78.437
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 35
    iter    0   |   diff: 250.31555   |   V(start): 79.299
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 36
    iter    0   |   diff: 225.28400   |   V(start): 80.096
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 37
    iter    0   |   diff: 202.75560   |   V(start): 80.833
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 38
    iter    0   |   diff: 182.48004   |   V(start): 81.513
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 39
    iter    0   |   diff: 164.23203   |   V(start): 82.142
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 40
    iter    0   |   diff: 147.80883   |   V(start): 82.722
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 41
    iter    0   |   diff: 133.02795   |   V(start): 83.256
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 42
    iter    0   |   diff: 119.72515   |   V(start): 83.749
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 43
    iter    0   |   diff: 107.75264   |   V(start): 84.203
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 44
    iter    0   |   diff: 96.97737   |   V(start): 84.620
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 45
    iter    0   |   diff: 87.27964   |   V(start): 85.005
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 46
    iter    0   |   diff: 78.55167   |   V(start): 85.358
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 47
    iter    0   |   diff: 70.69650   |   V(start): 85.683
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 48
    iter    0   |   diff: 63.62685   |   V(start): 85.981
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 49
    iter    0   |   diff: 57.26417   |   V(start): 86.255
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 50
    iter    0   |   diff: 51.53775   |   V(start): 86.507
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 51
    iter    0   |   diff: 46.38398   |   V(start): 86.737
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 52
    iter    0   |   diff: 41.74558   |   V(start): 86.949
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 53
    iter    0   |   diff: 37.57102   |   V(start): 87.143
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 54
    iter    0   |   diff: 33.81392   |   V(start): 87.321
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 55
    iter    0   |   diff: 30.43253   |   V(start): 87.484
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 56
    iter    0   |   diff: 27.38927   |   V(start): 87.633
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 57
    iter    0   |   diff: 24.65035   |   V(start): 87.770
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 58
    iter    0   |   diff: 22.18531   |   V(start): 87.895
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 59
    iter    0   |   diff: 19.96678   |   V(start): 88.009
    ['a0' 'a0' 'a0']
    N actions changed = 0

    after iteration 60
    iter    0   |   diff: 17.97010   |   V(start): 88.114
    ['a0' 'a0' 'a0']
    N actions changed = 1

    after iteration 61
    iter    0   |   diff: 16.17309   |   V(start): 88.210
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 62
    iter    0   |   diff: 14.55578   |   V(start): 88.297
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 63
    iter    0   |   diff: 13.10021   |   V(start): 88.377
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 64
    iter    0   |   diff: 11.79018   |   V(start): 88.450
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 65
    iter    0   |   diff: 10.61117   |   V(start): 88.517
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 66
    iter    0   |   diff: 9.55005   |   V(start): 88.578
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 67
    iter    0   |   diff: 8.59504   |   V(start): 88.634
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 68
    iter    0   |   diff: 7.73554   |   V(start): 88.685
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 69
    iter    0   |   diff: 6.96199   |   V(start): 88.731
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 70
    iter    0   |   diff: 6.26579   |   V(start): 88.774
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 71
    iter    0   |   diff: 5.63921   |   V(start): 88.812
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 72
    iter    0   |   diff: 5.07529   |   V(start): 88.848
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 73
    iter    0   |   diff: 4.56776   |   V(start): 88.880
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 74
    iter    0   |   diff: 4.11098   |   V(start): 88.909
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 75
    iter    0   |   diff: 3.69988   |   V(start): 88.936
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 76
    iter    0   |   diff: 3.32990   |   V(start): 88.960
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 77
    iter    0   |   diff: 2.99691   |   V(start): 88.982
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 78
    iter    0   |   diff: 2.69722   |   V(start): 89.003
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 79
    iter    0   |   diff: 2.42749   |   V(start): 89.021
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 80
    iter    0   |   diff: 2.18475   |   V(start): 89.038
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 81
    iter    0   |   diff: 1.96627   |   V(start): 89.053
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 82
    iter    0   |   diff: 1.76964   |   V(start): 89.067
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 83
    iter    0   |   diff: 1.59268   |   V(start): 89.080
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 84
    iter    0   |   diff: 1.43341   |   V(start): 89.091
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 85
    iter    0   |   diff: 1.29007   |   V(start): 89.102
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 86
    iter    0   |   diff: 1.16106   |   V(start): 89.111
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 87
    iter    0   |   diff: 1.04496   |   V(start): 89.120
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 88
    iter    0   |   diff: 0.94046   |   V(start): 89.128
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 89
    iter    0   |   diff: 0.84641   |   V(start): 89.135
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 90
    iter    0   |   diff: 0.76177   |   V(start): 89.142
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 91
    iter    0   |   diff: 0.68560   |   V(start): 89.148
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 92
    iter    0   |   diff: 0.61704   |   V(start): 89.153
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 93
    iter    0   |   diff: 0.55533   |   V(start): 89.158
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 94
    iter    0   |   diff: 0.49980   |   V(start): 89.163
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 95
    iter    0   |   diff: 0.44982   |   V(start): 89.167
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 96
    iter    0   |   diff: 0.40484   |   V(start): 89.170
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 97
    iter    0   |   diff: 0.36435   |   V(start): 89.174
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 98
    iter    0   |   diff: 0.32792   |   V(start): 89.177
    ['a0' 'a1' 'a0']
    N actions changed = 0

    after iteration 99
    iter    0   |   diff: 0.29513   |   V(start): 89.180
    ['a0' 'a1' 'a0']
    N actions changed = 0

### Value iteration convervence proof (0.5 pts)
**Note:** Assume that $\mathcal{S}, \mathcal{A}$ are finite.

Update of value function in value iteration can be rewritten in a form of Bellman operator:

$$(TV)(s) = \max_{a \in \mathcal{A}}\mathbb{E}\left[ r_{t+1} + \gamma V(s_{t+1}) | s_t = s, a_t = a\right]$$

Value iteration algorithm with Bellman operator:

---
   Initialize $V_0$

   **for** $k = 0,1,2,...$ **do**

     $V_{k+1} \leftarrow TV_k$

  **end for**

---

In [lecture](https://docs.google.com/presentation/d/1lz2oIUTvd2MHWKEQSH8hquS66oe4MZ_eRvVViZs2uuE/edit#slide=id.g4fd6bae29e_2_4) we established contraction property of bellman operator:

$$
||TV - TU||_{\infty} \le \gamma ||V - U||_{\infty}
$$

For all $V, U$

Using contraction property of Bellman operator, Banach fixed-point theorem and Bellman equations prove that value function converges to $V^*$ in value iterateion$

**Proof.**

Consider $||V_{k+1} - V^*||_{\infty}$. As $V^*$ is a fixed point of operator $T$, $TV^* = V^*$.
$$\begin{align}
||V_{k+1} - V^*|| = ||TV_k - TV^*|| \le&\ \gamma ||V_k - V^*|| = \gamma^{k+1}||V_0-V^*|| \implies \\
0 \le \lim\limits_{k \to +\infty} ||V_{k+1} - V^*|| \le&\ \lim\limits_{k \to +\infty} \gamma^{k+1}||V_0-V^*|| = [0 < \gamma < 1] = 0 \implies \\
\lim\limits_{k \to +\infty} ||V_{k+1} - V^*|| =&\ 0
\end{align}$$

**Q.E.D.**

### Bonus. Asynchronious value iteration (2 pts)

Consider the following algorithm:

---

Initialize $V_0$

**for** $k = 0,1,2,...$ **do**

     Select some state $s_k \in \mathcal{S}$

     $V(s_k) := (TV)(s_k)$

**end for**

---

Note that unlike common value iteration, here we update only a single state at a time.

**Homework.** Prove the following proposition:

If for all $s \in \mathcal{S}$, $s$ appears in the sequence $(s_0, s_1, ...)$ infinitely often, then $V$ converges to $V*$

**Proof.**

Consider vector $\Delta^{t+1} = |V_{t+1} - V^*| = [|V_{t+1}(s_i) - V^*(s_i)|]_{i = \overline{1,|\mathcal{S}|}}$.

Let's split sequence $S$ of $s_i$-s into all subsequences with same index $S_i$. It's guaranteed that all of them are infinite, that is $|S_i| \to +\infty$. Then, we take a look at $||V_{t+1}-V^*||_{\infty}$. It's obvious that $||V_{t+1}-V^*||_{\infty} \le ||V_{t+1}-V^*||_1 = \sum\limits_{i=1}^{|\mathcal{S}|} \Delta_i^{t+1}$.

Taking a limit on both sides:
$$0 \le \lim\limits_{t \to +\infty} ||V_{t+1}-V^*||_{\infty} \le \lim\limits_{t \to +\infty}\sum\limits_{i=1}^{|\mathcal{S}|} \Delta_i^{t+1} = \sum\limits_{i=1}^{|\mathcal{S}|}\lim\limits_{t \to +\infty} \Delta_i^{t+1}$$

Last equality is valid for finite $|\mathcal{S}|$ iff $\lim\limits_{t \to +\infty} \Delta_i^{t+1}$ exists for all $i$. But
$$\lim\limits_{t \to +\infty} \Delta_i^{t+1} = \lim\limits_{t \to +\infty} |V_{t+1}(s_i) - V^*(s_i)|$$

Introducing $S_i(t) = \sum\limits_{j=1}^{t} \mathcal{I}(S[j] = s_i)$ where $\mathcal{I}$ is the indicator function. Then, in terms of $S_i(t)$ $s_k$ appearing infinitely many times in $S$ means that $S_i(t) \to +\infty$ as $t \to +\infty$. And limit above can be expressed in terms of $S_i(t)$ as well!

$$\lim\limits_{t \to +\infty} |V_{t+1}(s_i) - V^*(s_i)| \le \lim\limits_{t \to +\infty} \gamma^{S_i(t)} |V_0(s_i) - V^*(s_i)| = 0$$

Substituting this gives us:
$$0 \le \lim\limits_{t \to +\infty} ||V_{t+1}-V^*||_{\infty} \le \sum\limits_{i=1}^{|\mathcal{S}|}\lim\limits_{t \to +\infty} \gamma^{S_i(t)} |V_0(s_i) - V^*(s_i)| = [0 < \gamma < 1] = 0$$

Then $\lim\limits_{t \to +\infty} ||V_{t+1}-V^*||_{\infty} = 0$. **Q.E.D.**

# HW Part 2: Policy iteration

## Policy iteration implementateion (2 pts)

Let's implement exact policy iteration (PI), which has the following pseudocode:

---
Initialize $\pi_0$   `// random or fixed action`

For $n=0, 1, 2, \dots$
- Compute the state-value function $V^{\pi_{n}}$
- Using $V^{\pi_{n}}$, compute the state-action-value function $Q^{\pi_{n}}$
- Compute new policy $\pi_{n+1}(s) = \operatorname*{argmax}_a Q^{\pi_{n}}(s,a)$
---

Unlike VI, policy iteration has to maintain a policy - chosen actions from all states - and estimate $V^{\pi_{n}}$ based on this policy. It only changes policy once values converged.

Below are a few helpers that you may or may not use in your implementation.

```python
transition_probs = {
    's0': {
        'a0': {'s0': 0.5, 's2': 0.5},
        'a1': {'s2': 1}
    },
    's1': {
        'a0': {'s0': 0.7, 's1': 0.1, 's2': 0.2},
        'a1': {'s1': 0.95, 's2': 0.05}
    },
    's2': {
        'a0': {'s0': 0.4, 's1': 0.6},
        'a1': {'s0': 0.3, 's1': 0.3, 's2': 0.4}
    }
}
rewards = {
    's1': {'a0': {'s0': +5}},
    's2': {'a1': {'s0': -1}}
}

from mdp import MDP
mdp = MDP(transition_probs, rewards, initial_state='s0')
```

Let's write a function called `compute_vpi` that computes the state-value function $V^{\pi}$ for an arbitrary policy $\pi$.

Unlike VI, this time you must find the exact solution, not just a single iteration.

Recall that $V^{\pi}$ satisfies the following linear equation:
$$V^{\pi}(s) = \sum_{s'} P(s,\pi(s),s')[ R(s,\pi(s),s') + \gamma V^{\pi}(s')]$$

You'll have to solve a linear system in your code. (Find an exact solution, e.g., with `np.linalg.solve`.)

```python
def compute_vpi(mdp, policy, gamma):
    """
    Computes V^pi(s) FOR ALL STATES under given policy.
    :param policy: a dict of currently chosen actions {s : a}
    :returns: a dict {state : V^pi(state) for all states}
    """
    states = mdp.get_all_states()
    state_to_idx = dict(zip(states, np.arange(len(states))))
    system = np.diag(np.ones(len(states)))
    rhs = np.zeros(len(states))
    for (i, state) in enumerate(states):
        policy_action = policy[state]
        if policy_action:  # if state has no actions, policy_action will be an empty string
            for next_state in mdp.get_next_states(state, policy_action):
                p = mdp.get_transition_prob(state, policy_action, next_state)
                r = mdp.get_reward(state, policy_action, next_state)
                rhs[i] += p * r
                system[i][state_to_idx[next_state]] -= gamma * p

    vpi = np.linalg.solve(system, rhs).tolist()
    return dict(zip(states, vpi))
```

```python
test_policy = {s: np.random.choice(
    mdp.get_possible_actions(s)) for s in mdp.get_all_states()}
new_vpi = compute_vpi(mdp, test_policy, gamma)

print(new_vpi)

assert type(
    new_vpi) is dict, "compute_vpi must return a dict {state : V^pi(state) for all states}"
```

    {'s0': 17.245615232375616, 's1': 20.350781409950176, 's2': 18.153279191974335}

Once we've got new state values, it's time to update our policy.

```python
def compute_new_policy(mdp, vpi, gamma):
    """
    Computes new policy as argmax of state values
    :param vpi: a dict {state : V^pi(state) for all states}
    :returns: a dict {state : optimal action for all states}
    """
    states = mdp.get_all_states()
    q_pi = lambda state, action: np.sum([mdp.get_transition_prob(state, action, next_state)
                                         * (mdp.get_reward(state, action, next_state) + gamma * vpi[next_state])
                                         for next_state in mdp.get_next_states(state, action)])
    policy = {}
    for state in states:
        actions = mdp.get_possible_actions(state)
        if actions:
            policy[state] = actions[np.argmax([q_pi(state, action) for action in actions])]
        else:
            policy[state] = ''

    return policy
```

```python
new_policy = compute_new_policy(mdp, new_vpi, gamma)

print(new_policy)

assert type(
    new_policy) is dict, "compute_new_policy must return a dict {state : optimal action for all states}"
```

    {'s0': 'a1', 's1': 'a0', 's2': 'a0'}

__Main loop__

```python
def policy_iteration(mdp, policy=None, gamma=0.9, num_iter=1000, min_difference=1e-5, iter_callback=None):
    """
    Run the policy iteration loop for num_iter iterations or till difference between V(s) is below min_difference.
    If policy is not given, initialize it at random.
    """
    def get_random_policy():
        policy = {}
        states = mdp.get_all_states()
        for state in states:
            actions = mdp.get_possible_actions(state)
            if actions:
                policy[state] = actions[np.random.randint(0, len(actions))]
            else:
                policy[state] = ''
        return policy

    def diff(vpi, new_vpi):
        vpi_np = np.array(list(vpi.values()))
        new_vpi_np = np.array(list(new_vpi.values()))
        return np.linalg.norm(vpi_np - new_vpi_np, ord=np.inf)

    if policy is None:
        policy = get_random_policy()
    state_values = compute_vpi(mdp, policy, gamma)
    it = 0
    for i in range(num_iter):
        new_policy = compute_new_policy(mdp, state_values, gamma)
        new_state_values = compute_vpi(mdp, new_policy, gamma)
        d = diff(state_values, new_state_values)
        policy_is_optimal = d < min_difference
        state_values = new_state_values
        policy = new_policy
        it += 1
        if iter_callback:
            iter_callback(it=it, diff=d, policy=policy, vpi=state_values)
        if policy_is_optimal:
            break

    return state_values, policy, it
```

__Your PI Results__

```python
gamma = 0.95
```

```python
def play_vi_game(mdp, gamma=0.9, verbose=True):
    if verbose:
        print('Value iteration log')
    state_values, total_iters = value_iteration(mdp, gamma=gamma, verbose=verbose)

    total_rewards = []
    for game_i in range(1000):
        s = mdp.reset()
        rewards = []
        for t in range(100):
            s, r, done, _ = mdp.step(
                get_optimal_action(mdp, state_values, s, gamma))
            rewards.append(r)
            if done:
                break
        total_rewards.append(np.sum(rewards))
    return np.mean(total_rewards), total_iters

def play_pi_game(mdp, gamma=0.9, verbose=True):
    pi_callback = lambda it, diff, policy, vpi: 1
    if verbose:
        print('Policy iteration log')
        pi_callback = lambda it, diff, policy, vpi:\
            print(f'iter {it:4d}   |   diff: {diff:.5f}   |   V(start): {vpi[(0,0)]:.3f}')
    vpi, policy, total_iters = policy_iteration(mdp, gamma=gamma, iter_callback=pi_callback)

    total_rewards = []
    for game_i in range(1000):||
        s = mdp.reset()
        rewards = []
        for t in range(100):
            s, r, done, _ = mdp.step(policy[s])
            rewards.append(r)
            if done:
                break
        total_rewards.append(np.sum(rewards))
    return np.mean(total_rewards), total_iters
```

```python
from tqdm import tqdm

def print_mdp_comparison(mdp, mdp_name, gamma=0.9, verbose=False):
    print('###############################################')
    print(f'Playing MDP {mdp_name}')
    print('###############################################')

    vi_history = []
    vi_iters = []
    pi_history = []
    pi_iters = []

    for i in tqdm(range(25)):
        vi_r, vi_it = play_vi_game(mdp, gamma, verbose)
        pi_r, pi_it = play_pi_game(mdp, gamma, verbose)
        vi_history.append(vi_r)
        vi_iters.append(vi_it)
        pi_history.append(pi_r)
        pi_iters.append(pi_it)

    print('Average VI reward stats')
    print(f'Mean: {np.mean(vi_history)}, std: {np.std(vi_history)}')
    print(f'Mean iter: {np.mean(vi_iters)}')
    print('Average PI reward stats')
    print(f'Mean: {np.mean(pi_history)}, std: {np.std(pi_history)}')
    print(f'Mean iter: {np.mean(pi_iters)}')

MDPs = [mdp_bonus,
        FrozenLakeEnv(slip_chance=0.25),
        FrozenLakeEnv(slip_chance=0.2, map_name='8x8')]
MDP_names = ['Bonus 1',
             'Small lake',
             'Big lake'
             ]

for (mdp, mdp_name) in zip(MDPs, MDP_names):
    print_mdp_comparison(mdp, mdp_name, gamma)
```

      0%|          | 0/25 [00:00<?, ?it/s]

    ###############################################
    Playing MDP Bonus 1
    ###############################################

    100%|██████████| 25/25 [02:24<00:00,  5.80s/it]
      0%|          | 0/25 [00:00<?, ?it/s]

    Average VI reward stats
    Mean: 16500.61112, std: 2816.309295078682
    Mean iter: 406.0
    Average PI reward stats
    Mean: 16856.87312, std: 2260.2912720156983
    Mean iter: 1.52
    ###############################################
    Playing MDP Small lake
    ###############################################

    100%|██████████| 25/25 [00:14<00:00,  1.68it/s]
      0%|          | 0/25 [00:00<?, ?it/s]

    Average VI reward stats
    Mean: 0.6535599999999999, std: 0.013896992480389428
    Mean iter: 24.0
    Average PI reward stats
    Mean: 0.6488, std: 0.013401492454200775
    Mean iter: 4.48
    ###############################################
    Playing MDP Big lake
    ###############################################

    100%|██████████| 25/25 [00:38<00:00,  1.54s/it]

    Average VI reward stats
    Mean: 0.77256, std: 0.015365103318884661
    Mean iter: 42.0
    Average PI reward stats
    Mean: 0.7697200000000001, std: 0.010148970391128363
    Mean iter: 6.56

**Tricky MDP:**

For tricky MDP policy iteration achieves better values in only a few iterations. Value iteration has difficulties converging because the probability of getting into very profitable state is rather low and discounting takes a lot of time to have an effect.

**Frozen lake envs:**

Stats above suggest that policy iteration converges to something slightly worse than optimal value iteration policy. Even though we calculate average reward estimate based on 25 runs of both algorithms, it still may differ quite significantly between runs. If we actually run some statistical test, I guess it'll tell us that they actually have the same average reward. This might be the case because environments are quite straight-forward and both algorithms converge to globally optimal policy. Apart from that, PI seems to be more stable than VI overall (its std is lower than that of VI).

## Policy iteration convergence (3 pts)

**Note:** Assume that $\mathcal{S}, \mathcal{A}$ are finite.

We can define another Bellman operator:

$$(T_{\pi}V)(s) = \mathbb{E}_{r, s'|s, a = \pi(s)}\left[r + \gamma V(s')\right]$$

And rewrite policy iteration algorithm in operator form:

---

Initialize $\pi_0$

**for** $k = 0,1,2,...$ **do**

     Solve $V_k = T_{\pi_k}V_k$

     Select $\pi_{k+1}$ s.t. $T_{\pi_{k+1}}V_k = TV_k$

**end for**

---

To prove convergence of the algorithm we need to prove two properties: contraction an monotonicity.

#### Monotonicity (0.5 pts)

For all $V, U$ if $V(s) \le U(s)$   $\forall s \in \mathcal{S}$ then $(T_\pi V)(s) \le (T_\pi U)(s)$   $\forall s \in  \mathcal{S}$

**Proof.**

$$\begin{align}
V(s) \le U(s) \implies V(s) - U(s) \le&\ 0 \ \forall s \in \mathcal{S} \\
T_\pi V(s) - T_\pi U(s) =&\ \mathbb{E}_{r, s'|s, a = \pi(s)}\left[r + \gamma V(s')\right] - \mathbb{E}_{r, s'|s, a = \pi(s)}\left[r + \gamma U(s')\right] \\
=&\ \mathbb{E}_{r, s'|s, a = \pi(s)}\left[\gamma(V(s') - U(s'))\right] \\
=&\ \sum\limits_{s'} \underbrace{P(r, s' | s, a = \pi(s))\gamma}_{\ge 0}\underbrace{(V(s') - U(s'))}_{\le 0} \le 0
\end{align}
$$

**Q.E.D.**

#### Contraction (1 pts)

$$
||T_\pi V - T_\pi U||_{\infty} \le \gamma ||V - U||_{\infty}
$$

for all $V, U$.

**Proof.**

$$\begin{align}
||T_\pi V - T_\pi U||_{\infty} =&\ \max\limits_s \left|\sum\limits_{s'} P(r, s' | s, a = \pi(s))\gamma(V(s') - U(s'))\right| \le \\
\le&\ \max\limits_s \sum\limits_{s'} \gamma\left|P(r, s' | s, a = \pi(s))(V(s') - U(s'))\right| \le \\
\le&\ \max\limits_s \sum\limits_{s'} \gamma\max\limits_{s'}|V(s') - U(s')|P(r, s' | s, a = \pi(s)) = \\
=&\ \gamma||V - U||_{\infty}\max\limits_s \underbrace{\sum\limits_{s'} P(r, s' | s, a = \pi(s))}_{=1} = \\
=&\ \gamma||V - U||_{\infty}
\end{align}$$

**Q.E.D.**

#### Convergence (1.5 pts)

Prove that there exists iteration $k_0$ such that $\pi_k = \pi^*$ for all $k \ge k_0$

**Proof.**

It's easy to see that $V_k \le V_{k+1} \le V^*$. Using monotonicity, we conclude that $T_{\pi_{k+1}} V_k \le T_{\pi_{k+1}}V_{k+1} = V_{k+1}$. Now, consider

$$|V_{k+1}-V^*| = V^* - V_{k+1} \le [-V_{k+1} \le -T_{\pi_{k+1}}V_k] \le V^* - T_{\pi_{k+1}}V_k = T_{\pi_{k+1}}V^* - T_{\pi_{k+1}}V_k = |T_{\pi_{k+1}}V^* - T_{\pi_{k+1}}V_k|$$

Then
$$||V_{k+1}-V^*||_{\infty} \le ||T_{\pi_{k+1}}V^* - T_{\pi_{k+1}}V_k||_{\infty} \le \gamma||V_k-V^*||_{\infty} \le \gamma^k||V_0-V^*||_{\infty}$$

Taking limit on both sides, we prove that $V_{k+1}$ converges to $V^*$ as $k$ approaches infinity. Using $\epsilon-\delta$ formalism, this means that
$$\forall \epsilon > 0 \ \exists k_0(\epsilon): \forall k \ge k_0(\epsilon) \ ||V_k-V^*||_{\infty} \le \epsilon$$

Because $\mathcal{S}$, $\mathcal{A}$ are finite, there exists a finite sequence $E = ||V_{\pi_i} - V^*||_{\infty},  i=\overline{1,|\mathcal{S}||\mathcal{A}|}$. Denote $\epsilon_0 = \min\{E_i | E_i \ne 0\}$ (this minimum always exists because sequence is finite) and take $\epsilon = \frac{\epsilon_0}{2}$. The only possible value for $||V_k-V^*||_{\infty}$ is then $0$ (optimal policy always exists and $||V_{\pi^*} - V^*||_{\infty} = 0$ because $V_{\pi^*} = V^*$). And due to definition of the limit, there has to exist some $k_0$ such that for all $k \ge k_0$ $||V_k-V^*||_{\infty} = 0$. Then $\pi_k = \pi^*$. **Q.E.D.**

```python

```


**Exercise set 7**
==============

>The goal of this exercise is to perform
>**least squares regression** for a case where we have
>**correlations** between variables. We will also see how
>we can use a **training set** and a **test set** to
>assess the quality of a model, and we will do our first
>**principal component analysis**.

**Exercise 7.1:**

In this exercise, we will make a least squares model for a case where we have several
variables which may be correlated.
The file [bloodpress.txt](Data/bloodpress.txt) ('Data/bloodpress.txt') contains data about 20 individuals with high blood pressure.
The data columns present in the file are given in table 1.

| Column | Description              |             Unit |
|:-------|:-------------------------|-----------------:|
| PT     | Label for person         |              --- |
| BP     | Blood pressure           |             mmHg |
| Age    | Age                      |            years |
| Weight | Weight                   |               kg |
| BSA    | Body surface area        |            m$^2$ |
| DUR    | Duration of hypertension |            years |
| Pulse  | Basal heart rate         | beats per minute |
| Stress | Stress index             |              --- |
||**Table 1:** *Data columns present in the file [bloodpress.txt](Data/bloodpress.txt)*|

**(a)** Begin by exploring the data. Plot pairs of variables against each other
to visually inspect which variables might be correlated. Based on your plots, which pairs
of variables seem to be correlated?

```python
# Your code here
```

**Your answer to 7.1(a):** (double click here)

**(b)** To quantitatively investigate possible correlations, we can calculate
the [Pearson correlation
coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) ($\rho_{X,Y}$),
\begin{equation*}
\rho_{X,Y} = \frac{\mathrm{cov}(X,Y)}{S_X S_Y},
\end{equation*}
where $\mathrm{cov}(X,Y)$ is the covariance between the variables $X$ and $Y$, and
$S_i$ is the estimated standard deviation for variable $i$.
In `scipy`, this coefficient is a part of the [`stats` package](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html)
and can be imported by:
```Python
from scipy.stats import pearsonr
```
This function returns two values:
 * The correlation coefficient itself. It is close to $1$ for positively correlated distributions
   and close to $-1$ for negatively correlated distributions.
 * A "p-value" which (to quote the documentation of `scipy`):

> roughly indicates the probability of an uncorrelated system
> producing datasets that have a Pearson correlation at least as extreme
> as the one computed from these datasets.

Calculate this coefficient for each of the pairs of variables and
use it to judge if the variables
are correlated.

```python
# Your code here
```

**Your answer to 7.1(b):** (double click here)

**(c)** Create a linear model in which you predict the blood pressure ($y$) from
all 6 available variables (Age, Weight, BSA, DUR, Pulse, and Stress).
For this, make use of the `statsmodels` package and the
ordinary least squares (OLS) estimation. (For an example of the usage, see: [here](https://www.statsmodels.org/dev/examples/notebooks/generated/ols.html))
Here, it might be beneficial to scale your variables so that they have a
mean of $0$ and a variance of $1$. This can, for instance, be done using the `sklearn` package.(Please see the [`scale` method](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html))
Compare the results of the fitting using both scaled and non-scaled variables.

**Note:** One motivation for using the `statsmodels` package is that it will do
some additional statistics in connection with the fitting. To print out
the results of a fitting, we can do the following:
```Python
import statsmodels.api as sm
# load X data...
# load y data...
# Do fitting:
model = sm.OLS(y, X).fit()
# Print out statistics:
print(model.summary())
```
Some information on the interpretation of this summary can be found [here](https://blog.datarobot.com/ordinary-least-squares-in-python).

```python
# Your code here
```

**Your answer to 7.1(c):** (double click here)

**(d)** Your results so far should indicate that some of the variables
are correlated. Create a new model where you remove variables
that are correlated. That is, if a variable, say $u$, is correlated with
another variable, say $v$, pick either $u$ or $v$ for your new model.

Fit your new model to the blood pressure data
and compare it with the model where you included all variables.
How many variables can you remove and still get a good model?
Which variables seem
to be more important for predicting if a person will have high blood pressure?

```python
# Your code here
```

**Your answer to 7.1(d):** (double click here)

**(e) (Optional exercise)** One strategy to assess the quality of a model is to split the raw data
into a **training set** and a **test set**. Then the **training set**
is used to fit the data and the **test set** is used to test the quality of
the model.

Consider a data set containing $n$ samples. If the data set is
relatively small, we can use the "Leave-one-out cross-validation (LOOCV)"
approach. Here, we pick one sample, say measurement number $j$, which we
keep out of the fitting and we fit the model using the $n-1$ other points.
Now, we can *predict* the value of measurement $j$ using the fitted model,
let us call the predicted value $\tilde{y}_j$. The difference between the predicted
value from the model and the measured value $y_j$ can then be taken as a measure
of the error,
\begin{equation*}
r_j = y_{j} - \tilde{y}_j,
\end{equation*}
where $r_j$ is the so-called predicted residual.
There is nothing special about picking point $j$, and we can try all possibilities
of leaving one point out, and refitting the model using the remaining $n-1$
measurements. After doing this for all possibilities, we have fitted the model
$n$ times and the mean squared error (obtained from the squared
residuals), $\mathrm{MSE}_{\mathrm{CV}}$, can then be used
to estimate the error in the model,
\begin{equation*}
\mathrm{MSE}_{\mathrm{CV}} = \frac{1}{n} \sum_{i=1}^{n} r_i^2 =  \frac{1}{n} \sum_{i=1}^{n} (y_i - \tilde{y}_i)^2,
\end{equation*}
where $y_i$ is the measured $y$ in experiment $i$, and $\tilde{y}_i$ is the
predicted $y$, using a model which was fitted using all point *except* $y_i$.

*(i)* Implement the LOOCV approach and calculate the mean squared error for a linear fit
using all variables, and for a linear fit using just the weight and age as
variables.

**Hint:** The splitting into training and test sets can be done with the method `LeaveOneOut`
from `sklearn`:
```Python
from sklearn.model_selection import LeaveOneOut
# load X data...
# load y data...
loo = LeaveOneOut()
for train_index, test_index in loo.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    # Do fitting using X_train and y_train.
    # Calculate error using y_test and predicted y from the model.
```

*(ii)* One can show that for polynomial fitting, the mean squared error can
be obtained in a simpler way,
\begin{equation*}
\mathrm{MSE}_{\mathrm{CV}} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \tilde{y}_i)^2 =
\frac{1}{n}\sum_{i=1}^{m} \left(\frac{y_i - \hat{y}_i}{1 - h_{ii}} \right)^2,
\end{equation*}
where the $\hat{y}_i$'s are predicted values using the
model fitted with *all data points*,
and $h_{ii}$ is the $i$'th diagonal element of the
$\mathbf{H}$ matrix (the projection matrix,
see Eq. (2) in exercise 3),
\begin{equation}
\mathbf{H} =
\mathbf{X}
\left(
  \mathbf{X}^\mathrm{T} \mathbf{X}
\right)^{-1}
\mathbf{X}^\mathrm{T}
\end{equation}
Using the equations given above, recalculate $\mathrm{MSE}_{\mathrm{CV}}$ and compare
with your previous answers.
Note the difference between $\hat{y}_i$ and $\tilde{y}_i$, and the
fact that we actually don't have to do the
refitting(!) in order to obtain the $\mathrm{MSE}_{\mathrm{CV}}$.

```python
# (Optional) Your code here
```

**(Optional) Your answer to 7.1(e):** (double click here)

**Exercise 7.2**

In this exercise, we will investigate a data set using PCA and
our aim is to classify objects.
The data set is contained in the file
[data_exercise7.txt](Data/data_exercise7.txt) ('Data/data_exercise7.txt')
where you will find columns as described in table 2.

| Column | Description                       |    Unit |
|:-------|:----------------------------------|--------:|
| $x$    | Values of a measured quantity $x$ | Unknown |
| $y$    | Values of a measured quantity $y$ | Unknown |
| class  | Classification of objects         |     --- |
||**Table 2:** *Data columns present in the file [data_exercise7.txt](Data/data_exercise7.txt)*

Here, we have done a series of measurements of the variables
$x$ and $y$ for some objects, and we have labeled these objects as
belonging to one of two possible classes: "foo" or "bar".
Your task is now to investigate if you can separate
data points into these two classes, by performing a principal component analysis.

**(a)** As stated above, we are attempting to separate the data into two
classes. Should you include the labels (i.e. the "class" column)
in the data matrix you will run PCA on?

**Your answer to 7.2(a):** (double click here)

**(b)** Begin by plotting the raw data.
Below you will find some Python
code to get you started:

```python
# Needed to import matplotlib and generate plot in the same cell
%matplotlib inline
from matplotlib import pyplot as plt # Plotting
import numpy as np # Matrix operations
import pandas as pd # Reading the data set
from sklearn.preprocessing import StandardScaler # Scaling (if needed )
from sklearn.decomposition import PCA # Performing PCA
# Load data:
data = pd.read_csv('Data/data_exercise7.txt', delim_whitespace =True)
# Extract the two classes for plotting :
class1 = data[data['class'] == 'foo']
class2 = data[data['class'] == 'bar']
# We can plot the raw data as follows :
# Note: Here you could style the plot to your liking .
# Adding x- labels and y- labels is , for instance , a good idea.
plt.scatter(class1['x'], class1['y'], label='Class foo')
plt.scatter(class2['x'], class2['y'], label='Class bar')
plt.legend();
```

**(c)** Run the principal component analysis. Should you scale the data in this case?
Below you will find some Python
code to get you started:

```python
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
plt.style.use('seaborn-talk')
# Load data:
data = pd.read_csv('Data/data_exercise7.txt', delim_whitespace = True)
variables = ['x','y'] # Insert the variables you will use here
X = data[variables]
# Uncomment the following lines in order to scale the data:
# scaler = StandardScaler() # Initialise the scaler
# scaler.fit(X) # "fit" the scaler with the X data
# X = scaler.transform(X) # Transform X data with the fit
# To run PCA:
pca = PCA()
scores = pca.fit_transform(X)
```

**Your answer to 7.2(c):** (double click here)

**(d)** The `pca` object defined above contains the
results of the principal component analysis.

*(i)* How many principal components were
used here? Hint: Inspect `pca.n_components_`

*(ii)* Plot the explained variance by these components.
Hint: Inspect `pca.explained_variance_ratio_`

```python
# Your code here
```

**Your answer to 7.2(d):** (double click here)

**(e)** The principal components themselves are contained as row vectors in
the matrix `pca.components_`:

*(i)* Obtain the principal component vectors.

*(ii)* Verify that they are normalized (Hint: dot products can be done with `np.dot`)

*(iii)* Verify that they are orthogonal to each other.

*(iv)* Plot them together with the raw data.
Do they point in the directions you would expect?

```python
# Your code here
```

**Your answer to 7.2(e):** (double click here)

**(f)** Next, we will investigate the contributions from the
original variables to the principal components. The contribution from
a variable can be found as a column vector in the matrix `pca.components_`.
Make a plot where you show the contributions from each of the original variables to principal components 1 and 2.

```python
# Your code here
```

**(g)** Plot the scores of the data points for
principal components 1 and 2. Color these points by the
class information.

*(i)* Which of the principal components seems more important for
separating between the two classes?

*(ii)*
For the following two points,

 * Point 1: $x = 4$, $y=6$
 * Point 2: $x = 6$, $y=4$

which class would you
predict?

*(iii)* Can you, based on this plot, make a simple
rule for determining if a point will belong to class "foo"  or class "bar"?
What would your rule be?

*(iv)* Can you "translate" your rule from the principal
component space to the original ($x$, $y$) space? What would the corresponding
rule be in the original variable space?

```python
# Your code here
```

**Your answer to 7.2(g):** (double click here)

**(h)** PCA belongs to a class of methods that are called latent variable methods.
Latent variable methods typically discover new
variables using the original ones, intending to uncover
"hidden" relations in the data.
In PCA, we find
such latent variables by taking linear combinations of our
original variables, and we make the new latent variables point
in the directions of the largest variance in our data.

Another example of a latent variable method is
Linear discriminant analysis (LDA).
LDA is similar to PCA, and in LDA we also
find new variables as linear combinations of our original variables.
However, the way we find them is different. In LDA, we do not look for
directions in which the variance is largest, but we rather look for
directions that *separate* the classes best. This means that class information
is needed as input when training the model.

Based on this would you say that PCA is a supervised or unsupervised method?
What about LDA? Is LDA a supervised or unsupervised method?

**Your answer to 7.2(h):** (double click here)


# 微積分の計算について　　　N0.4　定積分の内容－3　＜定積分の部分積分＞

###  学籍番号[＿＿＿＿＿＿＿＿＿]クラス[＿＿＿＿＿]　クラス番号[＿＿＿＿＿] 名前[＿＿＿＿＿＿＿＿＿＿＿＿＿＿＿]

##### 部分積分

###### 部分積分の公式　　$ [a,b]$で $f(x)$および $g'(x)$が連続で、$f(x)$の原始関数を$F(x)$とする

　$$   \int_{a}^{b} f(x)g(x)dx =  \left[ F(x)g(x) \right]_{a}^{b}- \int_{a}^{b} F(x)g'(x)dx $$

###### ワリスの公式（Wallis）
$$  \int_{0}^{\frac{\pi}{2}}\sin^n x dx = \int_{0}^{\frac{\pi}{2}}\cos^n x dx  $$
    n:奇数の時
    $$＝(\frac{n-1}{n})(\frac{n-3}{n-2})・・・(\frac{2}{3})(1) $$
    n:偶数の時
    $$＝(\frac{n-1}{n})(\frac{n-3}{n-2})・・・(\frac{1}{2})(\frac{\pi}{2}) $$

```python
from sympy import *
x, n , y, a = symbols('x n y a')
init_printing()
m ='4//3'
i =0
```

```python
expr = x*cos(x)
itg = Integral(expr,(x,0,pi/2))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = x*exp(x)
itg = Integral(expr,(x,0,1))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = x**2*exp(x)
itg = Integral(expr,(x,0,1))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = exp(x)*cos(x)
itg = Integral(expr,(x,0,pi/2))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr =(log(x))**2
itg = Integral(expr,(x,1,exp(1)))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = sin(x)**3
itg = Integral(expr,(x,0,pi/2))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = cos(x)**4
itg = Integral(expr,(x,0,pi/2))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = (x-1)**2*exp(-x)
itg = Integral(expr,(x,0,1))
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```


### Problem 77: Prime summations

It is possible to write ten as the sum of primes in exactly five different ways:

7 + 3

5 + 5

5 + 3 + 2

3 + 3 + 2 + 2

2 + 2 + 2 + 2 + 2

What is the first value which can be written as the sum of primes in over five thousand different ways?

```python
from sympy import primefactors

def prime_summations(no_of_partitions):
    """
    Gives n such that the number of prime partitions
    of n is >= no_of_partitions

    n has a limit of 1000.
    i.e. no_of_partitions <= 10 quadrillion
    """
    l = [1, 0]
    for n in range(2, 1000):
        prime_partitions = sum([sum(primefactors(k))*l[n - k] for k in range(1, n + 1)])/n
        l.append(prime_partitions)
        if prime_partitions > no_of_partitions:
            return n, prime_partitions

print(prime_summations(5000))
```

    (71, 5007.0)

```python

```


# 2. Clasificación:

x=Features, variables de predicción, variables independientes.

y=Targets, variables objetivo, variables dependientes, variables de respuesta.

# 2.1 Modelos lineales para clasificación:

El objetivo en clasificación es tomar un vector de entrada $x$ y asignarlo a una de las clases discretas $C_k$, donde $k=1,..., K$. En el esceneario más común las clases se asumen como disjuntas. El espacio de entrada entonces, es dividido en fronteras de decisión o superficies de decisión. En los modelos lineales de clasificación, se considera que las superficies de decisión (o fronteras) son hiperplanos lineales dentro del espacio D-dimensional de entrada. También se asume que los datos pueden ser separados exactamente por una supercicie linealmente separable.

Para problemas de regresión, la variable objetivo $t$ es simplemente un vector de números reales que se desea predecir. En el caso de clasificación, es posible ver los variables objetivo como formas de representar una clase. En el caso de clasificación binaria es posible ver la variable objetivo como $t\in {0,1}$ tal que $t=1$ para representar a $C_1$ y $t=0$ para representar a $C_2$. Para K>2, es conveniente utilizar un esquema de código que permita identificar las clases, por ejemplo, en el caso de $K=5$, la clase 2 se puede representar como $t=(0,1,0,0,0)^T$.

A diferencia del caso de regresión. En esta ocasión se desea obtener un modelo en el cual transformemos la función lineal de $w$ usando una función no lineal $f(\cdot)$ tal que

\begin{equation}
y(x)=f(w^T x+w_0)
\end{equation}

Esta función $f$ se conoce como la función de activación. Debido a la linealidad de esta función, este tipo de funciones, se conocen en general como funciones de clasificación lineales o modelos lineales de clasificación.

El análisis y las características que se describirán a continuación también son aplicables para casos en los que se realicen transformaciones no-lineales a las variables de entrada a través de un vector base de funciones $\phi(x)$.

## Funciones discriminantes:

Una función discriminante toma un vector de entrada y lo asigna a una de las K clases, denotada como $C_k$.

```python
from sklearn import datasets

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
plt.style.use('ggplot')

iris=datasets.load_iris()

X=iris.data
y=iris.target

#print(X)
#print(y)

df=pd.DataFrame(X,columns=iris.feature_names)

_=pd.plotting.scatter_matrix(df,c=y,figsize=[8,8],s=150,marker='D')
plt.show()
```

    <Figure size 800x800 with 16 Axes>

# 2.2 k-nearest neighbors:

La idea básica de k-nearest neighbors es encontrar la etiqueta de un punto nuevo a partir de un conjunto de datos ya etiquetados. Para ello se mirarn los "k" vecinos más cercanos y se decide por mayoría.

A partir de un conjunto de datos, obtengamos un clasificador y saquemos un score de la clasificación

```python
from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.3,random_state=21,stratify=y)

k=np.arange(1,100)
sk=[]

for k_ in k:
    knn=KNeighborsClassifier(n_neighbors=k_)
    knn.fit(X_train,y_train)
    y_pred=knn.predict(X_test)
    sk.append(knn.score(X_test,y_test))

plt.plot(k,np.array(sk))
plt.xlabel('K')
plt.ylabel('score(K)')
plt.title('score(K) vs K')
plt.show()
```

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn import neighbors, datasets

n_neighbors=15

iris=datasets.load_iris()

X=iris.data[:,:2]
y=iris.target

h=.02

cmap_light=ListedColormap(['#FFAAAA','#AAFFAA','#AAAAFF'])
cmap_bold=ListedColormap(['#FF0000','#00FF00','#0000FF'])

for weights in ['uniform','distance']:
    clf=neighbors.KNeighborsClassifier(n_neighbors,weights=weights)
    clf.fit(X,y)

    x_min,x_max=X[:,0].min()-1,X[:,0].max()+1
    y_min,y_max=X[:,1].min()-1,X[:,1].max()+1

    xx,yy=np.meshgrid(np.arange(x_min,x_max,h),np.arange(y_min,y_max,h))
    Z=clf.predict(np.c_[xx.ravel(),yy.ravel()])
    Z=Z.reshape(xx.shape)

    plt.figure()

    plt.pcolormesh(xx,yy,Z,cmap=cmap_light)

    plt.scatter(X[:,0],X[:,1],c=y,cmap=cmap_bold,edgecolor='k',s=20)
    plt.xlim(xx.min(),xx.max())
    plt.ylim(yy.min(),yy.max())

    plt.title("3-class classification(k=%i,pesos='%s')"%(n_neighbors,weights))

plt.show()
```

# 2.3 Regresión logística

Es posibleexpresar la probabilidad a posteriori de la clase $C_1$ como una sigmoide logística actuando sobre una función lineal en un vector de características $\phi$.

\begin{equation}
p(C_1|\phi)=y(\phi)=\sigma(w^T \phi)
\end{equation}

con $p(C_2|\phi)=1-p(C_1|\phi)$. Aquí, $\sigma$ es la función logística de activación, definida como:

\begin{equation}
\sigma=\frac{1}{1+e^{-a}}
\end{equation}

Utilizando maximum likelihood para determinar los parámetros del modelo de regresión logística, es posible sacar la derivada de la función de activación logística sigmoidal que puede ser expresada como

\begin{equation}
\frac{d\sigma}{da}=\sigma (1-\sigma)
\end{equation}

A partir de esta formulación podemos obtener que

\begin{equation}
E(w)=-ln(p(t|w))=-\sum_{n=1}^{N}{t_n ln(y_n)+(1-t_n)ln(1-y_n)}
\end{equation}

donde $y_n=\sigma(a_n)$ y $a_n=w^T\phi_n$, por lo que

\begin{equation}
\nabla E(w)=\sum_{n=1}^{N}{(y_n-t_n)\phi_n}
\end{equation}

Con lo que tenemos suficiente para construir un algoritmo secuencial para lograr clasificación logística.

```python
from sklearn.linear_model import LogisticRegression
from sklearn import datasets

iris=datasets.load_iris()
X=iris.data[:,:2]
Y=iris.target

logreg=LogisticRegression(C=1e5,solver='lbfgs',multi_class='ovr')

logreg.fit(X,Y)

x_min,x_max=X[:,0].min()-.5,X[:,0].max()+.5
y_min,y_max=X[:,1].min()-.5,X[:,1].max()+.5

h=.02

xx,yy=np.meshgrid(np.arange(x_min,x_max,h),np.arange(y_min,y_max,h))

Z=logreg.predict(np.c_[xx.ravel(),yy.ravel()])
Z=Z.reshape(xx.shape)

plt.figure(1,figsize=(9,9))
plt.pcolormesh(xx,yy,Z)

plt.scatter(X[:,0],X[:,1],c=Y,edgecolors='k',cmap=plt.cm.Paired)
plt.xlabel('Sepal L')
plt.ylabel('Sepal W')

plt.xlim(xx.min(),xx.max())
plt.ylim(yy.min(),yy.max())
plt.xticks()
plt.yticks()
plt.show()
```

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets, svm

iris = datasets.load_iris()
X = iris.data
y = iris.target

X = X[y != 0, :2]
y = y[y != 0]

n_sample = len(X)

np.random.seed(0)
order = np.random.permutation(n_sample)
X = X[order]
y = y[order].astype(np.float)

X_train = X[:int(.9 * n_sample)]
y_train = y[:int(.9 * n_sample)]
X_test = X[int(.9 * n_sample):]
y_test = y[int(.9 * n_sample):]

for kernel in ('linear', 'rbf', 'poly'):
    clf = svm.SVC(kernel=kernel, gamma=10,C=1e5)
    clf.fit(X_train, y_train)

    plt.figure()
    plt.clf()
    plt.scatter(X[:, 0], X[:, 1], c=y, zorder=10, cmap=plt.cm.Paired,
                edgecolor='k', s=20)

    plt.scatter(X_test[:, 0], X_test[:, 1], s=80, facecolors='none',
                zorder=10, edgecolor='k')

    plt.axis('tight')
    x_min = X[:, 0].min()
    x_max = X[:, 0].max()
    y_min = X[:, 1].min()
    y_max = X[:, 1].max()

    XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]
    Z = clf.decision_function(np.c_[XX.ravel(), YY.ravel()])

    Z = Z.reshape(XX.shape)
    plt.pcolormesh(XX, YY, Z > 0, cmap=plt.cm.Paired)
    plt.contour(XX, YY, Z, colors=['k', 'k', 'k'],
                linestyles=['--', '-', '--'], levels=[-.5, 0, .5])

    plt.title(kernel)
plt.show()
```

# Referencias:

- https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html
- https://scikit-learn.org/stable/auto_examples/exercises/plot_iris_exercise.html#sphx-glr-auto-examples-exercises-plot-iris-exercise-py
- https://scikit-learn.org/stable/supervised_learning.html

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

h = .02  # step size in the mesh

names = ["Nearest Neighbors", "Linear SVM", "RBF SVM", "Gaussian Process",
         "Decision Tree", "Random Forest", "Neural Net", "AdaBoost",
         "Naive Bayes", "QDA"]

classifiers = [
    KNeighborsClassifier(3),
    SVC(kernel="linear", C=0.025),
    SVC(gamma=2, C=1),
    GaussianProcessClassifier(1.0 * RBF(1.0)),
    DecisionTreeClassifier(max_depth=5),
    RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
    MLPClassifier(alpha=1, max_iter=1000),
    AdaBoostClassifier(),
    GaussianNB(),
    QuadraticDiscriminantAnalysis()]

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2,
                           random_state=1, n_clusters_per_class=1)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [make_moons(noise=0.3, random_state=0),
            make_circles(noise=0.2, factor=0.5, random_state=1),
            linearly_separable
            ]

figure = plt.figure(figsize=(27, 9))
i = 1
# iterate over datasets
for ds_cnt, ds in enumerate(datasets):
    # preprocess dataset, split into training and test part
    X, y = ds
    X = StandardScaler().fit_transform(X)
    X_train, X_test, y_train, y_test = \
        train_test_split(X, y, test_size=.4, random_state=42)

    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))

    # just plot the dataset first
    cm = plt.cm.RdBu
    cm_bright = ListedColormap(['#FF0000', '#0000FF'])
    ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
    if ds_cnt == 0:
        ax.set_title("Input data")
    # Plot the training points
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright,
               edgecolors='k')
    # Plot the testing points
    ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright, alpha=0.6,
               edgecolors='k')
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    i += 1

    # iterate over classifiers
    for name, clf in zip(names, classifiers):
        ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
        clf.fit(X_train, y_train)
        score = clf.score(X_test, y_test)

        # Plot the decision boundary. For that, we will assign a color to each
        # point in the mesh [x_min, x_max]x[y_min, y_max].
        if hasattr(clf, "decision_function"):
            Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])
        else:
            Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]

        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, cmap=cm, alpha=.8)

        # Plot the training points
        ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright,
                   edgecolors='k')
        # Plot the testing points
        ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright,
                   edgecolors='k', alpha=0.6)

        ax.set_xlim(xx.min(), xx.max())
        ax.set_ylim(yy.min(), yy.max())
        ax.set_xticks(())
        ax.set_yticks(())
        if ds_cnt == 0:
            ax.set_title(name)
        ax.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score).lstrip('0'),
                size=15, horizontalalignment='right')
        i += 1

plt.tight_layout()
plt.show()

```

```python

```


# Naive Bayes

```{note}
Naive Bayes classifiers are a family of simple "probabilistic classifiers" based on applying Bayes' theorem with simple(naive) independence assumptions between the features.
```

## Discriminative and Generative Algorithms

Discriminative：try to learn $p(y|x)$ directly, such as Logistic Regression.

Generative：try to learn $p(x|y)$ and $p(y)$, such as Naive Bayes.

When predicting, use bayes rule：

$$p(y|x)=\frac{p(x|y)p(y)}{p(x)}$$

then：

$$\hat{y}=\underset{y}{\mbox{argmax}}\ \frac{p(x|y)p(y)}{p(x)}= \underset{y}{\mbox{argmax}}\ {p(x|y)p(y)}$$

## Algorithm

Suppose $x=(x_{1}, x_{2},..., x_{d})$ and $x_{j}$  is binary, then：

$$
p(x|y) = p(x_{1},x_{2},...,x_{d}|y) = p(x_{1}|y)p(x_{2}|y,x_{1})...p(x_{d}|y,x_{1},...,x_{d-1})
$$

Naive bayes independent assumption：

$$
p(x_{1}|y)p(x_{2}|y,x_{1})...p(x_{d}|y,x_{1},...,x_{d-1})
= p(x_{1}|y)p(x_{2}|y)...p(x_{d}|y)
$$

Parameterized by:

$$y\sim Bernoulli(\phi)$$
$$x_{j}|y=1 \sim Bernoulli(\phi_{j|y=1})$$
$$x_{j}|y=0 \sim Bernoulli(\phi_{j|y=0})$$

note:

$$k = \sum_{i=1}^{n}1\left\{y^{(i)}=1\right\}$$
$$s_{j} = \sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=1\right\}$$
$$t_{j} = \sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=0\right\}$$

We can write down the joint log likelihood of the data:

$$
\begin{equation}
\begin{split}
\mathcal{L}(\phi, \phi_{j|y=1}, \phi_{j|y=0}) &= log\prod_{i=1}^{n}p(x^{(i)},y^{(i)})\\
&=\sum_{i=1}^{n}log(p(x^{(i)}, y^{(i)})) \\
&=\sum_{i=1}^{n}log(p(y^{(i)})\prod_{j=1}^{d}p(x_{j}^{(i)}|y^{(i)})) \\
&=\sum_{y^{(i)}=1}(log(\phi) + \sum_{j=1}^{d}log(p(x_{j}^{(i)}|y=1))) + \sum_{y^{(i)}=0}(log(1 - \phi) + \sum_{j=1}^{d}log(p(x_{j}^{(i)}|y=0))) \\
&=k\ log(\phi) + (n-k)log(1 - \phi) + \sum_{j=1}^{d}(s_{j}\ log(\phi_{j|y=1}) + (k-s_{j})log(1 - \phi_{j|y=1}) + t_{j}\ log(\phi_{j|y=0}) + (n -k - t_{j})log(1 - \phi_{j|y=0})
\end{split}
\end{equation}
$$

Maximum likelihood estimate result:

$$\phi=\frac{k}{n}$$

$$\phi_{j|y=1} = \frac{s_{j}}{k}$$

$$\phi_{j|y=0} = \frac{t_{j}}{n-k}$$

## Laplace Smoothing

To predict unseen data, we add 1 to the numerator, add 2 to the denominator:

$$\phi_{j|y=1} = \frac{s_{j} + 1}{k + 2}$$

$$\phi_{j|y=0} = \frac{t_{j} + 1}{n-k + 2}$$

It's actually Binary Bernoulli Naive Bayes and implemented in *BernoulliNB*.

It assumes each feature is binary-valued, if handed any other kind of data, a *BernoulliNB* instance may binarize its input.

```python
import numpy as np

rng = np.random.RandomState(1)
X = rng.randint(5, size=(6, 100))
y = np.array([1, 2, 3, 4, 4, 5])
```

```python
from sklearn.naive_bayes import BernoulliNB

clf = BernoulliNB()
clf.fit(X, y)

clf.predict(X[2:3])
```

    array([3])

## Multinomial Naive Bayes

*MultinomialNB* implements the naive Bayes algorithm for multinomially distributed data, and is one of the two classic naive Bayes variants used in text classification.

This distribution is parametrized by vectors $\theta_{y} = (\theta_{y1},...,\theta_{yn})$ for each class $y$, where $n$ is the number of features(in text classification, the size of the vocabulary) and $\theta_{yi} = p(x_{i}|y)$.

The parameters $\theta_{y}$ is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting:

$$\hat{\theta}_{yi} = \frac{N_{yi} + \alpha}{N_{y} + \alpha{n}}$$

where $N_{yi}$ is the number of times feature $i$ appears in a sample of class $y$, and $N_{y} = \sum_{i=1}^{n}N_{yi}$, $\alpha \ge 0$ is smooth hyperparameter.

```python
from sklearn.naive_bayes import MultinomialNB

clf = MultinomialNB()
clf.fit(X, y)

clf.predict(X[2:3])
```

    array([3])

## Gaussian Naive Bayes

When $x_{j}$ is real-valued, we can assume the likelihood of the features is Gaussian:

$$p(x_{j}|y) = \frac{1}{\sqrt{2\pi\sigma_{y}^2}}\exp\left(-\frac{(x_{j} - \mu_{y})^2}{2\sigma_{y}^2}\right)$$

Just like the binary case, parameters $\mu_{y}$ and $\sigma_{y}$ are estimated using maximum likelihood.

This is Gaussian Naive Bayes and implemented in *GaussianNB*.

```python
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)

gnb = GaussianNB()
y_pred = gnb.fit(X_train, y_train).predict(X_test)
print("Number of mislabeled points out of a total %d points : %d"
      % (X_test.shape[0], (y_test != y_pred).sum()))
```

    Number of mislabeled points out of a total 75 points : 4