# Calculus and Machine learning

In deep learning, we typically choose **loss functions** that are **differentiable** with respect to our modelʼs parameters. Put simply, this means that for each parameter, we can determine how rapidly the loss would increase or decrease, were we to increase or decrease that parameter by an infinitesimally small amount.

1. Getting better on training datasets means minimizing a loss function.
2. Producing a model that performs well on data that we have never seen before, that means what's performance of the model on testing dataset.

Thus we can decompose the task of fitting models into two key concerns:
1. **Optimization**:  the process of fitting our models to observed data (like training and valid datasets)
2. **Generalization**: produce models whose validity extends beyond the exact set of data examples used to train them, like testing datasets

Suppose that we have a deep neural network where the weights are $w=(w_1, w_2, \ldots, w_n)$. Given a training dataset, we consider to minimize the loss of our neural network on this dataset: $\mathcal{L}(w)$:

**Solution**:
1. we often start by initializing our weights randomly.
2. We iteratively take small steps in the direction which makes the loss decrease as rapidly as possible

# Differential Calculus (微分)

## Derivative(导数) of a function

Suppose that we have a function ($y=f(x)$) $f: \mathbb{R} \rightarrow \mathbb{R}$, whose input and output are both scalars. The **derivative** of f is defined as:
\begin{equation}
\begin{aligned}
f^{'}(x) &= \lim_{h \rightarrow 0} \frac{f(x+h) - f(x)}{h} = \lim_{\Delta_x \rightarrow 0} \frac{f(x+\Delta_x) -f(x)}{\Delta_x}\;\textrm{if this limit exists} \\
&=\frac{dy}{dx} = \frac{df}{dx} = \frac{d}{dx}f(x) = Df(x) = D_xf(x)
\end{aligned}
\end{equation} where symbols $\frac{d}{dx}$ and $D$ are **differentiation** operators that indicate operation of differentiation.

If $f^{'}(a)$ exists, $f$ is said to be **differentiable** at $a$, that is defined as:
\begin{equation}
f^{'}(a) = \lim_{x \rightarrow a} \frac{f(x) - f(a)}{x-a}
\end{equation}

If $f$ is **differentiable** at every number of an interval ($x\in [a,b]$), then this function is differentiable on this interval ($[a,b]$).

## Linear Approximation

In particular, note that the following equation approximates the value of f by a line which passes through the point (x, f(x)) and has slope $f'$. In this way we say that the derivative gives a linear approximation to the function f.
\begin{equation}
\begin{aligned}
\frac{df(x)}{dx} = \lim_{\epsilon \rightarrow 0} \frac{f(x+\epsilon) - f(x))}{\epsilon } &\Rightarrow \frac{df(x)}{dx} \approx \frac{f(x+\epsilon) - f(x))} {\epsilon } \\
\Rightarrow \epsilon \frac{df(x)}{dx} \approx f(x+\epsilon) - f(x) \\
\Rightarrow f(x+\epsilon) \approx f(x) + \epsilon \frac{df(x)}{dx} = f(x) + \epsilon f^{'}
\end{aligned}
\end{equation}

```python
# f(x) = 3x^2 - 4x
def f(x):
    return 3*x**2 - 4*x

# Functions to calcuate limits
def numerical_limit(f, x, h):
    return (f(x + h) - f(x)) / h

h = 0.0001
x= 1
for i in range(5):
    print(f'h={h:.5f}, numerical limit={numerical_limit(f, x, h):.5f}')
    h *= 0.1
```

    h=0.00010, numerical limit=2.00030
    h=0.00001, numerical limit=2.00003
    h=0.00000, numerical limit=2.00000
    h=0.00000, numerical limit=2.00000
    h=0.00000, numerical limit=2.00000

## Common Derivatives

- $(x^n)' = nx^{n-1}$ (the power rule, $n$ is any real number)
- $(e^x)' = e^x$
- $(\sin(x))' = \cos(x)$
- $(\cos(x))' = -\sin(x)$
- $\tan(x)' = \sec^2(x)$
- $\cot(x)' = -\csc^2(x)$
- $\sec(x)' = \sec x \tan x$
- $\csc x' = -\csc x \cot x$
- $(a^x)' = a^x\ln x (a \gt 0, a \neq 1)$
- $(\log_ax)' = \frac{1}{x\ln a} (a \gt 0, a \neq 1)$
- $(\ln(x))' = \frac{1}{x}$

## Derivative Rules

Assume $u=u(x), v=v(x)$ are differentiable at an interval $x \in [a,b]$, and $C$ is a constant number
1. Constant Multiple Rule: (Cu)^' = Cu'
2. Sum Rule: $(u \pm v) = u' \pm v'$
3. Product Rule $(uv)' = u'v + uv'$
4. Quotient Rule: $(\frac{u}{v})' = \frac{u'v - uv'}{v^2} \; (v \neq 0 )$

## Partial Derivatives

So far we have dealt with the differentiation of functions with just **one variable**. In deep learning, functions often depend on **many variables**. Thus, we need to extend the ideas of differentiation to these **multivariate functions**.

Let $y = f(x_1, x_2, \ldots, x_n)$ be a function with $n$ variables. The partial derivative of $y$ with respect to its $i^{th}$ parameter $x_i$ is:
\begin{equation}
\frac{\partial y}{\partial x_i} = \frac{\partial f}{\partial x_i} = f_{x_i} = f_i = D_if = D_{x_i} f = \lim_{\epsilon \rightarrow 0} \frac{f(x_1, x_2, \ldots, x_i + \epsilon, \ldots, x_n) - f(x_1, x_2, \ldots, x_i, \ldots, x_n)}{\epsilon}
\end{equation}

To calculate $\frac{\partial y}{\partial x_i}$, we can simply treat all variables except $x_i$ as constants and calculate the derivative of $y$ with respect to $x_i$.

\begin{equation}
f(x+\epsilon) \approx f(x) + \epsilon  \frac{df}{dx}(x) \\ \Downarrow \\
L(x_1 + \epsilon_1, x_2, \ldots, x_n) \approx L(x_1, x_2, \ldots, x_n) + \epsilon_1 \frac{\partial L}{\partial x_1} (x_1, x_2, \ldots, x_n) \\ \Downarrow \\
L(x_1 + \epsilon_1, x_2 + \epsilon_2, \ldots, x_n + \epsilon_n) \approx L(x_1, x_2, \ldots, x_n) + \sum_{i=1}^{n} \epsilon_i \frac{\partial L}{\partial x_i} (x_1, x_2, \ldots, x_n)
\end{equation}

This may look like a mess, but we can make this more familiar by noting that the sum on the right looks exactly like a dot product, so if we let:

\begin{equation}
\epsilon = [\epsilon_1, \ldots, \epsilon_n]^T \; \textrm{and} \; \Delta_x L = [\frac{\partial L}{\partial x_1}, \ldots, \frac{\partial L}{\partial x_n}]  \\ \Downarrow \\
L(X + \epsilon) \approx L(X) + \epsilon \cdot \Delta_x L(X)
\end{equation}
We will call the vector $\Delta_x L$ the gradient of $L$. This equation allows us to tell approximately how the function L will change given any perturbation to the input.

```python
%matplotlib inline
import numpy as np
import torch
from IPython import display
def f(x, y):
    return torch.log(torch.exp(x) + torch.exp(y))
def grad_f(x, y):
    return torch.tensor([torch.exp(x) / (torch.exp(x) + torch.exp(y)), torch.exp(y) / (torch.exp(x) + torch.exp(y))])

epsilon = torch.tensor([0.01, -0.03])

grad_approx = f(torch.tensor([0.]), torch.log(torch.tensor([2.]))) + epsilon.dot(grad_f(torch.tensor([0.]), torch.log(torch.tensor(2.))))

true_value = f(torch.tensor([0.]) + epsilon[0], torch.log(torch.tensor([2.])) + epsilon[1])

f'approximation: {grad_approx}, true Value: {true_value}'
```

    'approximation: tensor([1.0819]), true Value: tensor([1.0821])'

## Gradients

We can concatenate partial derivatives of a multivariate function with respect to all its variables to obtain the **gradient** vector of the function

Suppose that the input of function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ is an n-dimensional vector $x = [x_1, x_n, \ldots, x_n]^T$ and the output is a scalar. The gradient of the function $f(x)$ with respect to $x$ is a vector of n partial derivatives:
\begin{equation}
\Delta_xf(x) = \Delta f(x) = [\frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots, \frac{\partial y}{\partial x_n}]^T  = [\frac{\partial f(x)}{\partial x_1}, \frac{\partial f(x)}{\partial x_2}, \ldots, \frac{\partial f(x)}{\partial x_n}]^T
\end{equation}

Let x be an n-dimensional vector $x = [x_1, x_n, \ldots, x_n]^T$, The following rules are often used when differentiating multivariate functions:

- For all $A \in \mathbb{R}^{m \times n}$, $\Delta_x A x = A^T$
- For all $A \in \mathbb{R}^{n \times m}$, $\Delta_x x^T A = A$
- For all $A \in \mathbb{R}^{n \times n}$, $\Delta_x x^T A x = (A + A^T) x$
- $\Delta_x ||x||^2 = \Delta_x x^T x = 2x$
- $\forall X, \Delta_X ||X||^2_F = \Delta_x x^T x = 2X$

## Geometry of Gradients and Gradient Descent
Consider an input vecotr $X=[x_1, \ldots, x_n]$ and its corresponding loss function $L(X)$, $L$ can be approxmated and minimized by following equation:
\begin{equation}
L(X + \epsilon) \approx L(X) + \epsilon \cdot \Delta_x L(X)
\end{equation}
where $\epsilon = [\epsilon_1, \ldots, \epsilon_n]^T$ and gradients $\Delta_x L = [\frac{\partial L}{\partial x_1}, \ldots, \frac{\partial L}{\partial x_n}]$.

Let us suppose that I want to use this to help minimize our loss L. Let us understand geometrically the algorithm of gradient descent first:

1. Start with a random choice for the initial parameters $X$
2. Find the direction $v$ that makes $L$ decrease the most rapidly at $X$.
3. Take a small step in that direction: $X \rightarrow X + \epsilon v$
4. Repeat

The only thing we do not know exactly how to do is to compute the vector $v$ in the second step. We will call such a direction **the direction of steepest descent**.

Using the geometric understanding of dot products $\cos(\theta) = \frac{u \cdot v}{ ||u||*||v||}$ where $\theta$ is the angle between two vectors $u$ and $v$, we can rewrite the above equation as:
\begin{equation}
\begin{aligned}
L(X + v) {}&\approx L(X) + v \cdot \Delta_x L(X) \\
&= L(X) + ||v|| * || \Delta_x L(X)|| \cos(\theta) \\
&= L(X) + || \Delta_x L(X)|| \cos(\theta)
\end{aligned}
\end{equation} Note that we have taken our direction to have length one for convenience, and used $\theta$ for the angle
between $v$ and $\Delta_XL(X)$.

Intuitively, we want to find the direction that decreases L as rapidly as possible, that means $L(X + v) - L(X) = || \Delta_x L(X)|| \cos(\theta) \leq 0$. Therefore, the only way the direction we pick enters into this equation is through $\cos(\theta)$, and thus we wish to make this cosine as negative as possible. Now, recalling the shape of cosine, we can make this as negative as possible by making $\cos(\theta) = -1$ or equivalently making the angle between the gradient and our chosen direction to be $\pi$ radians or equivalently 180 degrees.
- The only way to achieve this is to head in the exact opposite direction: pick $v$ to point in the exact opposite direction to $\Delta_XL(X)$.
- That means the direction of steepest decent points in the direction of $-\Delta_XL(X)$.

Thus our informal algorithm can be rewritten as follows.

1. Start with a random choice for the initial parameters $X$
2. Find the direction $v$ that makes $L$ decrease the most rapidly at $X$, that's $v = -\Delta_XL(X)$.
3. Take a small step in the opposite of that direction: $X \rightarrow X - \epsilon \Delta_XL(X)$
4. Repeat

**Summary**

 - Use the gradient to find the direction that decreases the loss as rapidly as possible, and update the parameters to take a step in that direction
 - The only possible points where we can minimize (or maximize) a function will have gradient equal to zero, however, not every point with gradient zero is the true global minimum (or maximum).

## Multivariate Chain Rule

The chain rule enables us to differentiate composite functions.

Let us first consider functions of a single variable. Suppose that functions $y = f(u)$ and $u = g(x)$ are both differentiable, then the chain rule states that:
\begin{equation}
\frac{dy}{dx} = \frac{dy}{du}*\frac{du}{dx}
\end{equation}

Next, suppose that the differentiable function y has variables $y = f(x_1, x_2, \ldots, x_n) = f(u_1, u_2, \ldots, u_m)$, where each differentiable function $u_i = g_i(x_1, x_2, \ldots, x_n)$. Then the chain rule gives for a variable $x_i$:
\begin{equation}
\begin{aligned}
\frac{dy}{d{x_i}} &= \sum_{j = 1}^m \frac{dy}{d{u_j}}*\frac{d{u_j}}{d{x_i}}\\
&=\frac{dy}{d{u_1}}*\frac{d{u_1}}{d{x_i}} + \frac{dy}{d{u_2}}*\frac{d{u_2}}{d{x_i}} + \ldots + \frac{dy}{d{u_m}}*\frac{d{u_m}}{d{x_i}}
\end{aligned}
\end{equation}

### Example
Let us suppose that we have a function of four variables $(w, x, y$ and $z)$, as shown in following figure

we can make by composing many terms:
\begin{equation}
y = f(u, v) = (u + v)^2 \\
u = u(a, b) = (a + b)^2, \; v = v(a,b) = (a - b)^2 \\
a = a(w, x, y, z) = (w +x + y + z)^2, b = b(w, x, y, z) = (w + x - y - z)^2
\end{equation}

Such chains of equations are common when working with neural networks, so trying to understand how to compute gradients of such functions is key.

If we want to compute say $\frac{\partial f}{\partial w}$, we may apply the multi-variate chain rule to see:
\begin{equation}
\frac{\partial f}{\partial w} = \frac{\partial f}{\partial u}\frac{\partial u}{\partial w} + \frac{\partial f}{\partial v}\frac{\partial v}{\partial w} \\
\frac{\partial u}{\partial w} = \frac{\partial u}{\partial a}\frac{\partial a}{\partial w} + \frac{\partial u}{\partial b}\frac{\partial b}{\partial w} \\
\frac{\partial v}{\partial w} = \frac{\partial v}{\partial a}\frac{\partial a}{\partial w} + \frac{\partial v}{\partial b}\frac{\partial b}{\partial w} \\
\end{equation}

Let us try using this decomposition to compute $\frac{\partial f}{\partial w}$. Notice that all we need here are the various single step partials:
\begin{equation}
\frac{\partial f}{\partial u} = 2(u+v), \frac{\partial f}{\partial v} = 2(u+v) \\
\frac{\partial u}{\partial a} = 2(a+b), \frac{\partial u}{\partial b} = 2(a+b) \\
\frac{\partial v}{\partial a} = 2(a-b), \frac{\partial v}{\partial b} = -2(a+b) \\
\frac{\partial a}{\partial w} = 2(w+x+y+z), \frac{\partial b}{\partial w} = 2(w+x-y-z)
\end{equation}

If we write this out into code this becomes a fairly manageable expression.

```python
# Compute the value of the function from inputs to outputs
w, x, y, z = -1, 0, -2, 1
a, b = (w + x + y + z)**2, (w + x - y - z)**2
u, v = (a + b)**2, (a - b)**2
f = (u + v)**2
print(f' f at {w}, {x}, {y}, {z} is {f}')
```

     f at -1, 0, -2, 1 is 1024

```python
# Compute the single step partials
df_du, df_dv = 2 * (u + v), 2 * (u + v)
du_da, du_db = 2 * (a + b), 2 * (a + b)
dv_da, dv_db = 2 * (a - b), -2 * (a -b)

da_dw, db_dw = 2 * (w + x + y + z), 2 * (w + x - y - z)
da_dx, db_dx = 2 * (w + x + y + z), 2 * (w + x - y - z)
da_dy, db_dy = 2 * (w + x + y + z), -2 * (w + x - y - z)
da_dz, db_dz = 2 * (w + x + y + z), -2 * (w + x - y - z)

# Compute the final result from inputs to outputs
du_dw = du_da * da_dw + du_db * db_dw
dv_dw = dv_da * da_dw + dv_db * db_dw
df_dw = df_du * du_dw + df_dv * dv_dw

print(f'df/dw at {w}, {x}, {y}, {z} is {df_dw}')
```

    df/dw at -1, 0, -2, 1 is -4096

```python
# Now compute how f changes when we change any value from output to input
df_da = df_du * du_da + df_dv * dv_da
df_db = df_du * du_db + df_dv * dv_db

df_dw = df_da * da_dw + df_db * db_dw
df_dx = df_da * da_dx + df_db * db_dx
df_dy = df_da * da_dy + df_db * db_dy
df_dz = df_da * da_dz + df_db * db_dz

print(f'df/dw at {w}, {x}, {y}, {z} is {df_dw}')
print(f'df/dx at {w}, {x}, {y}, {z} is {df_dx}')
print(f'df/dy at {w}, {x}, {y}, {z} is {df_dy}')
print(f'df/dz at {w}, {x}, {y}, {z} is {df_dz}')
```

    df/dw at -1, 0, -2, 1 is -4096
    df/dx at -1, 0, -2, 1 is -4096
    df/dy at -1, 0, -2, 1 is -4096
    df/dz at -1, 0, -2, 1 is -4096

The fact that we compute derivatives from f back towards the inputs rather than from the inputs forward to the outputs (as we did in the first code snippet above) is what gives this algorithm its name: **backpropagation**. Note that there are two steps:
1. Compute the value of the function, and the single step partials from front to back. While not done above, this can be combined into a single forward pass.
2. Compute the gradient of f from back to front. We call this the backwards pass.

This is precisely what every deep learning algorithm implements to allow the computation of the gradient of the loss with respect to every weight in the network at one pass. It is an astonishing fact that we have such a decomposition.

```python
# Initialize as ndarrays, then attach gradients
w = torch.tensor([-1.], requires_grad=True)
x = torch.tensor([0.], requires_grad=True)
y = torch.tensor([-2.], requires_grad=True)
z = torch.tensor([1.], requires_grad=True)
# Do the computation like usual, tracking gradients
a, b = (w + x + y + z)**2, (w + x - y - z)**2
u, v = (a + b)**2, (a - b)**2
f = (u + v)**2
print(f' f: {f} at \n w: {w}, \n x: {x}, \n y: {y}, \n z: {z}')
```

     f: tensor([1024.], grad_fn=<PowBackward0>) at
     w: tensor([-1.], requires_grad=True),
     x: tensor([0.], requires_grad=True),
     y: tensor([-2.], requires_grad=True),
     z: tensor([1.], requires_grad=True)

```python
# Execute backward pass
#All of what we did above can be done automatically by calling f.backwards().
f.backward()

print(f'df/dw at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {w.grad.data.item()}')
print(f'df/dx at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {x.grad.data.item()}')
print(f'df/dy at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {y.grad.data.item()}')
print(f'df/dz at {w.data.item()}, {x.data.item()}, {y.data.item()}, '
      f'{z.data.item()} is {z.grad.data.item()}')
```

    df/dw at -1.0, 0.0, -2.0, 1.0 is -4096.0
    df/dx at -1.0, 0.0, -2.0, 1.0 is -4096.0
    df/dy at -1.0, 0.0, -2.0, 1.0 is -4096.0
    df/dz at -1.0, 0.0, -2.0, 1.0 is -4096.0

## How to calucate differentiation in Pytorch: Automatic Differentiation

### Backward for a scalar variable returned value y

Suppose a differentiating function $y = f(x) = 2*x^T*x$ where $x$ is a n-dimensional vectors:

\begin{equation}
y = f(x) = 2*x^T*x = 2 (x^2_1 + x^2_2 + \ldots + x^2_n)
\end{equation}

The gradients of this function is:
\begin{equation}
\Delta_x f(x) = [\frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots, \frac{\partial y}{\partial x_n}]^T = [4x_1, 4x_2, \ldots, 4x_n]^T
\end{equation}

```python
import torch

# A differentiating function f(x) = 2X^T * X with a scalar value returned
def f(x):
    return 2*torch.dot(x, x)

# let us create the variable x and assign it an initial value
x = torch.arange(4.0)
x
```

    tensor([0., 1., 2., 3.])

```python
# It is important that we do not allocate new memory every time we take a derivative with respect
# to a parameter because we will often update the same parameters thousands or millions of times and
# could quickly run out of memory.
# The fact that gradients need to be computed for a Tensor do not mean that the grad attribute will be populated
x.requires_grad_(True) # Same as `x = torch.arange(4.0, requires_grad=True)`
x.grad
```

```python
# Now let us calculate y.
# Since x is a vector of length 4, an dot product of x and x is performed, yielding the scalar output that we assign to y.
y = f(x)
y
```

    tensor(28., grad_fn=<MulBackward0>)

```python
# At this time, there is no gradients are calcuated for x
x.grad
```

```python
# Next, we can automatically calculate the gradient of y with respect to each
# component of x by calling the function for backpropagation and printing the gradient.
y.backward()
x.grad
```

    tensor([ 0.,  4.,  8., 12.])

```python
# The gradient of the function y = 2x^⊤*x with respect to x should be 4x.
x.grad == 4 * x
```

    tensor([True, True, True, True])

```python
# PyTorch accumulates the gradient in default, we need to clear the previous values
x.grad.zero_()
x.grad
```

    tensor([0., 0., 0., 0.])

```python
# Now let us calculate another function of y = sum(x) = x_1 + x_n + ... + x_n.
y = x.sum()
y.backward()
x.grad
```

    tensor([1., 1., 1., 1.])

### Backward for Non-Scalar returned Variables Y

The most natural interpretation of the differentiation of a vector y with respect to a vector x is a matrix. For higher-order and higher-dimensional y and x, the differentiation result could be a high-order tensor.

```python

import torch

# A differentiating function f(x) = 2X^T * X with a scalar value returned
def f(x):
    return x * x

# let us create the variable x and assign it an initial value
x = torch.arange(4.0, requires_grad=True)
print(f"The input x: {x}")

y = f(x)
print(f"The output Y: {y}")

# Invoking `backward` on a non-scalar requires passing in a `gradient` argument
# which specifies the gradient of the differentiated function w.r.t `self`.
# In our case, we simply want to sum the partial derivatives, so passing
# in a gradient of ones is appropriate
y.sum().backward() # or y.backward(torch.ones(len(x))) equivalent to the below

print(f"The x grad: {x.grad}")
```

    The input x: tensor([0., 1., 2., 3.], requires_grad=True)
    The output Y: tensor([0., 1., 4., 9.], grad_fn=<MulBackward0>)
    The x grad: tensor([0., 2., 4., 6.])

### Computing the Gradient of Python Control Flow

```python
def f(a):
    b = a * 2
    while b.norm() < 1000:
        b = b * 2
    if b.sum() > 0:
        c = b
    else:
        c = 100 * b
    return c
```

```python
a = torch.randn(size=(), requires_grad=True)
d = f(a)
d.backward()
print (f"a: {a}")
print (f"a grad: {a.grad}")
print (f"d: {d}")
```

    a: -0.18225865066051483
    a grad: 819200.0
    d: -149306.28125

```python

a.grad == d / a
```

    tensor(True)

# Integral Calculus （积分）

```python

```


```python
"""Header cell, contains modules and functions to make the whole notebook experience better"""
%matplotlib inline
# plots graphs within the notebook

from IPython.display import display,Image, Latex
from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')
from IPython.display import clear_output

import time

from IPython.display import display,Image, Latex

from IPython.display import clear_output

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

font = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 12,
        }
fontlabel = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 12,
        }

from matplotlib.ticker import FormatStrFormatter
plt.rc('font', **font)

class PDF(object):
  def __init__(self, pdf, size=(200,200)):
    self.pdf = pdf
    self.size = size

  def _repr_html_(self):
    return ''.format(self.pdf, self.size)

  def _repr_latex_(self):
    return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

```

```python
PDF('bot-sketch.pdf',size = (550,400))
```

# Differential Drive Robot - 1

Alphabot 2 is a differential drive robot: Two independent motors drive the left and right wheels and the differential speed between the two wheels control the speed and direction of the robot. The following code is copied from http://enesbot.me/kinematic-model-of-a-differential-drive-robot.html

## Objective
This notebook focuses on the control of a simulated differential drive robot for prescribed simple tasks.

## Theoretical Background

The robot parameters are:

 Wheel radius: $r$.

 Length between wheels: $L$

 Angular velocity of the left and right wheels: $\omega_L$, $\omega_R$, respectively.

 Angle from horizontal: $\alpha$.

 Position vector of the robot: $(x,y)$.

 Velocity vector of the robot: $V=(\dot{x},\dot{y})$.

Hereafter $\dot{a}$ refers to the time derivative of variable $a$:

$$
\dot{a}=\frac{da}{dt}
$$

The velocities of the wheels are therefore defined as:

$$
V_L=\omega_Lr\text{ and }V_R=\omega_Rr.
$$

The velocity of the robot, taken at the center of the wheels, is simply:

$$
\vec{V}=\frac{V_R+V_L}{2}(\cos(\alpha)\vec{e}_x+\sin(\alpha\vec{e}_y),
$$

yielding the following equations of motions:

$$
\dot{x}=r\frac{\omega_R+\omega_L}{2}\cos{\alpha}
$$

$$
\dot{y}=r\frac{\omega_R+\omega_L}{2}\sin{\alpha}
$$

and

$$
\dot{\alpha}=\frac{r}{L}(\omega_R-\omega_L)
$$

Numerically, these solutions of these equations requires their discretization. Remember that the derivative $da/dt$ of a variable $a=f(t)$ is the slope $s$ of the local tangent. Over a small enough time interval $\Delta t$, any continuous function $f$ can be approximated by a linear function based on the slope $a(t+\Delta t$)=a(t)+s(t)\Delta t$. The derivative is then estimated as:

$$
\frac{da}{dt}\approx\frac{\Delta a}{\Delta t}
$$

where $\Delta a=a(t+\Delta t$)-a(t)$. The discretized equations of motion may be recast as an integration of the trajectory of the robot:

$$
x(t+\Delta t) = x(t) +\Delta t\left(r\frac{\omega_R+\omega_L}{2}\cos{\alpha}\right),
$$

$$
y(t+\Delta t) = y(t) +\Delta t\left(r\frac{\omega_R+\omega_L}{2}\sin{\alpha}\right),
$$

$$
\alpha(t+\Delta t) = \alpha(t) +\Delta t\left(r\frac{\omega_R-\omega_L}{L}\right).
$$

As long as, the initial conditions are prescribed at $t=0$, the motion of the robot is predictable providing that $\Delta t$ is not too large, as we shall see later. The present discretization is the simplest one, and also the most rudimentary. Even though the above equations carry fairly large numerical errors, no robot is perfect and these numerical errors will not be discussed here.

## Code description

The code below defines a class, named Robot, which includes the robot dimensions, position variables ($x,y,\alpha$), the time step $\Delta t$, the left/right angular velocities and various functions necessary to integrate the robot motion. First execute the cell below.

```python
import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

from math import sin, cos

class Robot(object):
    """Defines basic mobile robot properties"""
    def __init__(self):
        self.pos_x  = 0.0
        self.pos_y  = 0.0
        self.angle  = 0.0
        self.plot   = False
        self._delta = 0.01
        self.step_plot = int(5)
        self.mag_plot = 1.

    # Movement
    def step(self):
        """ updates the x,y and angle """
        self.deltax()
        self.deltay()
        self.deltaa()

    def move(self, seconds):
        """ Moves the robot for an 's' amount of seconds"""
        for i in range(int(seconds/self._delta)):
            self.step()
            if i % self.step_plot == 0 and self.plot: # plot path every 3 steps
                self.plot_xya()

    # Printing-and-plotting:
    def print_xya(self):
        """ prints the x,y position and angle """
        print ("x = " + str(self.pos_x) +" "+ "y = " + str(self.pos_y))
        print ("a = " + str(self.angle))

    def plot_robot(self):
        """ plots a representation of the robot """
        plt.arrow(self.pos_x, self.pos_y, 0.001
                  * cos(self.angle), 0.001 * sin(self.angle),
                  head_width=self.mag_plot*self.length, head_length=self.mag_plot*self.length,
                  fc='k', ec='k')

    def plot_xya(self):
        """ plots a dot in the position of the robot """
        plt.scatter(self.pos_x, self.pos_y, c='r', edgecolors='r')

class DDRobot(Robot):
    """Defines a differential drive robot"""

    def __init__(self):
        Robot.__init__(self)
        self.radius = 0.1
        self.length = 0.4

        self.rt_spd_left = 0.0
        self.rt_spd_right = 0.0

    def deltax(self):
        """ update x depending on l and r angular speeds """
        self.pos_x += self._delta * (self.radius*0.5) \
        * (self.rt_spd_right + self.rt_spd_left)*cos(self.angle)

    def deltay(self):
        """ update y depending on l and r angular speeds """
        self.pos_y += self._delta * (self.radius*0.5) \
        * (self.rt_spd_right + self.rt_spd_left)*sin(self.angle)

    def deltaa(self):
        """ update z depending on l and r angular speeds """
        self.angle += self._delta * (self.radius/self.length) \
        * (self.rt_spd_right - self.rt_spd_left)
# function to convert degrees to radians
def D2R(a):
    return math.pi*a/180
def R2D(a):
    return 180*a/math.pi
# mybot = DDRobot()            # robot called 'enesbot'

# mybot.angle = 3.1416/4        # 45 degrees
# mybot.plot = True             # plot the robot!
# mybot.plot_robot()

# mybot.rt_spd_left = 10
# mybot.rt_spd_right = 10       # straight line
# mybot.move(2)                 # move for 2 seconds

# mybot.rt_spd_left = 12.5664
# mybot.rt_spd_right = 18.8496  # (2m diameter circle)
# mybot.move(1)                 # move for 1 second

# mybot.rt_spd_left = 18.8496
# mybot.rt_spd_right = 12.5664  # (2m diameter circle)
# mybot.move(2.5)               # move for 2.5 second

# mybot.rt_spd_left = 12.5664
# mybot.rt_spd_right = 18.8496  # (2m diameter circle)
# mybot.move(3.5)               # move for 2.5 second

# mybot.plot_robot()

# plt.xlim([-1, 6])               # axis limits
# plt.ylim([-1, 6])

# plt.show()
```

## Quick Tutorial

Now you can create your first robot.

```python
mybot = DDRobot()
```

For simplicity, assume that all units are based on the metric system. Your new robot, mybot, is by default located at $(x=0,y=0)$, its angle from the horizontal axis $x$ is $0$ and the wheel radius and length between wheel are $r=0.1$m and $L=0.4$m (see class DDRobot). The default time step is $\Delta t=0.01$s. All these variables and more can be easily customized to your problem:

```python
mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = D2R(45) #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r
mybot.rt_spd_left = 10 # omega_L
mybot.rt_spd_right = 10 # omega_R

```

Now that our initial conditions and rotational speed is set, let's move our robot in a straight line and plot its trajectory.

```python
mybot.plot = True #True if you want to plot the robot's trajectory
mybot.plot_robot() #draw an arrow for the location of the robot at t=0

mybot.move(10) #move the robot for 10s
mybot.plot_robot() #draw the new location

plt.xlim(-1,11)
plt.ylim(-1,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

You may change the size of the arrow and the spacing of the dots making the trajectory using the following variables:

```python
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = D2R(45) #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r
mybot.rt_spd_left = 10 # omega_L
mybot.rt_spd_right = 10 # omega_R

mybot.plot = True #True if you want to plot the robot's trajectory
mybot.plot_robot() #draw an arrow for the location of the robot at t=0

mybot.move(10) #move the robot for 10s
mybot.plot_robot() #draw the new location

plt.xlim(-1,11)
plt.ylim(-1,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

You may print the location of your robot using:

```python
mybot.print_xya()
```

    x = 7.071067811865343 y = 7.071067811865343
    a = 0.7853981633974483

which you can compare to the theoretical location of the robot. Since the robot's wheel radius is $0.1$m, the rotational speed is $\omega=10$ (1/s), therefore the robot speed is $1$m/s. Over $10$s of travel time, starting at an angle of $45^\circ$, the location of the robot should be $x=10\cos(\pi/4)$ and $y=10\sin(\pi/4)$.

```python
print('absolute error x: %1.1e m' %(abs(mybot.pos_x - 10*math.cos(math.pi/4))))
print('absolute error y: %1.1e m' %(abs(mybot.pos_y - 10*math.sin(math.pi/4))))
print('absolute error alpha: %1.1e rad' %(abs(mybot.angle - math.pi/4)))
```

    absolute error x: 1.3e-13 m
    absolute error y: 1.3e-13 m
    absolute error alpha: 0.0e+00 rad

To turn, the left and right angular velocities must be different, as shown below

```python

mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 10 # plot location every 100 iterations

mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = D2R(45) #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r

mybot.plot = True #True if you want to plot the robot's trajectory
mybot.plot_robot() #draw an arrow for the location of the robot at t=0

# move straight for 10m
mybot.rt_spd_left = 10 # omega_L
mybot.rt_spd_right = 10 # omega_R
mybot.move(10) #move the robot for 10s

# 2m radius turn
mybot.rt_spd_left = 18.8496
mybot.rt_spd_right = 12.5664
mybot.move(2.5)

mybot.plot_robot() #draw the new location

plt.xlim(-1,11)
plt.ylim(-1,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

## Assignment

### 1 Iterative turn
In this first assignment the robot must start at $(0,0)$, drive horizontally to $(10,0)$, then vertically to $(10,10)$, horizontally to $0,10$ and finally vertically to $(0,0)$. Write a code using a for loop to move your robot. Within the for loop, turn your robot using a while loop. Use the same robot dimensions as above and an absolute angular velocity $\omega = 10$.

```python
mybot = DDRobot()            # robot called 'enesbot'
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 50 # plot location every 100 iterations

mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = 0 #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r
omega = 10
rot_time_increment = 0.01 #time increment for rotation
rotation_angle = D2R(90)

mybot.plot = True #True if you want to plot the robot's trajectory
mybot.plot_robot() #draw an arrow for the location of the robot at t=0

for i in range(4):
    print(i)
    # move straight for 10m
    mybot.rt_spd_left = omega # omega_L
    mybot.rt_spd_right = omega # omega_R
    mybot.move(10) #move the robot for 10s
    # turn 90 degrees to the left
    mybot.rt_spd_left = -omega # omega_L
    mybot.rt_spd_right = omega # omega_R
    alpha_0 = mybot.angle
    while (mybot.angle - alpha_0) < rotation_angle:
        mybot.move(rot_time_increment)
        print(R2D(mybot.angle - alpha_0))

    mybot.plot_robot() #draw the new location

plt.xlim(-1,11)
plt.ylim(-1,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()

```

As you can see, the robot does not return to its origin. Quantify the error.

```python
print('absolute error x: %1.1e m' %(math.sqrt(mybot.pos_x**2 +mybot.pos_y**2)))
```

    absolute error x: 8.1e-01 m

### 2 Exact rotation

Derive and implement an equation to determine the exact time needed for the robot to achieve a rotation of a given angle around its center. Quantify the error

```python
# your code here
```

### 3 Circular motion

Derive and implement an equation to determine the paramters (angular velocities and motion time) needed for the robot to achieve an arc of a given radius and angle (from origin to end).

```python
# your code here
```


# Hello world for linear regression in PyMC3

This is meant to be a first notebook for exploring how to use PyMC3 to do:

* MAP estimation
* MCMC sampling using the No-U-Turn (NUTS) extension of Hamlitonian Monte Carlo

Meant to allow side-by-side comparison with Stan

Inspired by:

* Bob Carpenter's blog post:

https://statmodeling.stat.columbia.edu/2017/05/31/compare-stan-pymc3-edward-hello-world/

* The "hello world" example in the PyMC3 journal article:

Salvatier, Wiecki, and Fonnesbeck. "Probabilistic programming in Python using PyMC3".
https://peerj.com/articles/cs-55.pdf

```python
import numpy as np
import pandas as pd
```

```python
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style('whitegrid');
sns.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 2.0})

%matplotlib inline
```

# Import PyMC3

If you need to install it first, using the `spr_2020s_env` conda environment just do:

```console
$ conda activate spr_2020s_env
$ conda install -c conda-forge pymc3
```

```python
import pymc3
```

# Model as MATH

Likelihood:
\begin{align}
 p(y_{1:N} | x_{1:N}, \alpha, \beta, \sigma) &= \text{NormalPDF}(\alpha + \beta_1 x_{n1} + \beta_2 x_{n2}, \sigma^2)
\end{align}

Priors on the variance of the likelihood:
\begin{align}
 p(\sigma) &= \text{HalfNormalPDF}(\sigma | 0, 1) =
 \begin{cases}
     2 \cdot \text{NormalPDF}(\sigma | 0, 1) & \sigma > 0
\\
     0.0  & \sigma \leq 0
 \end{cases}
\end{align}

Priors on regression coefficients:
\begin{align}
 p(\alpha) &= \text{NormalPDF}(0, 10)
\\
 p(\beta_1) &= \text{NormalPDF}(0, 10)
\\
 p(\beta_2) &= \text{NormalPDF}(0, 10)
\\
\end{align}

# Simulate data from the model

```python
# Initialize random number generator
np.random.seed(123)

# True regression coefficients
alpha = 1.0
beta = np.asarray([1.0, 1.337])

# True regression std dev
sigma = 0.3

# Size of dataset
N = 100

# Data Features (input variable)
x1_N = np.random.randn(N)
x2_N = np.random.randn(N)
x_N2 = np.hstack([x1_N[:,np.newaxis], x2_N[:,np.newaxis]])

# Simulate outcome variable
y_N = alpha + beta[0]*x1_N + beta[1]*x2_N + np.random.randn(N) * sigma

true_params = dict(
    alpha=alpha,
    beta=beta,
    sigma=sigma)
```

# Plot of observed data

```python
_, axgrid = plt.subplots(nrows=1, ncols=2, figsize=(8,3), sharex=True, sharey=True);

axgrid[0].plot(x1_N, y_N, 'k.'); axgrid[0].set_xlabel('x1'); axgrid[0].set_ylabel('y');
axgrid[1].plot(x2_N, y_N, 'b.'); axgrid[1].set_xlabel('x2'); axgrid[1].set_ylabel('y');
```

# Model specified as a PyMC3 object

```python
my_model_N10 = pymc3.Model()

with my_model_N10:
    # Priors for unknown model parameters
    alpha = pymc3.Normal('alpha', mu=0, sd=10)
    beta = pymc3.Normal('beta', mu=0, sd=10, shape=2)
    sigma = pymc3.HalfNormal('sigma', sd=1)

    # Likelihood (sampling distribution) of observations
    Y_obs = pymc3.Normal('Y_obs',
        mu=(alpha + beta[0] * x1_N[:10] + beta[1] * x2_N[:10]),
        sd=sigma,
        observed=y_N[:10])
```

```python
my_model_N100 = pymc3.Model()

with my_model_N100:
    # Priors for unknown model parameters
    alpha = pymc3.Normal('alpha', mu=0, sd=10)
    beta = pymc3.Normal('beta', mu=0, sd=10, shape=2)
    sigma = pymc3.HalfNormal('sigma', sd=1)

    # Likelihood (sampling distribution) of observations
    Y_obs = pymc3.Normal('Y_obs',
        mu=(alpha + beta[0] * x1_N + beta[1] * x2_N),
        sd=sigma,
        observed=y_N)
```

```python
print(my_model_N100.basic_RVs)
```

    [alpha, beta, sigma_log__, Y_obs]

# Setup: Prepare for plotting results

```python
def plot_traces(fit):
    fig, ax_grid = plt.subplots(nrows=2, ncols=2, sharex=True, sharey=True);
    for ax, var_name, dim in zip(
            ax_grid.flatten(), ['alpha', 'sigma', 'beta', 'beta'], [None, None, 0, 1]):
        var_samples_S = fit[var_name]
        true_val = true_params[var_name]

        if dim is not None:
            var_samples_S = var_samples_S[:, dim]
            true_val = true_val[dim]
            var_name = var_name + "[%d]" % dim

        S = var_samples_S.size
        ax.plot(var_samples_S, 'b.', markersize=4, alpha=0.3, label='samples')
        ax.plot(true_val * np.ones(S), 'r-', label='true_value')
        ax.set_ylabel(var_name)

    ax.legend(bbox_to_anchor=(1.1, 1.0))
    ax.set_ylim([0, 2.0]);
```

# Run MAP estimator for N=10 dataset

```python
map_N10 = pymc3.find_MAP(model=my_model_N10, maxeval=1000)
```

    /Users/mhughes/miniconda3/envs/spr_2020s_env/lib/python3.7/site-packages/pymc3/tuning/starting.py:61: UserWarning: find_MAP should not be used to initialize the NUTS sampler, simply call pymc3.sample() and it will automatically initialize NUTS in a better way.
      warnings.warn('find_MAP should not be used to initialize the NUTS sampler, simply call pymc3.sample() and it will automatically initialize NUTS in a better way.')
    logp = -8.0697, ||grad|| = 0.71175: 100%|██████████| 28/28 [00:00<00:00, 2509.63it/s]

```python
for key, arr in map_N10.items():
    print(key, arr)
```

    alpha 1.069344037625922
    beta [1.00373515 1.44519906]
    sigma_log__ -1.607230117606145
    sigma 0.2004420467603088

# Run MAP estimator for N=100 dataset

```python
map_N100 = pymc3.find_MAP(model=my_model_N100, maxeval=1000)
```

    /Users/mhughes/miniconda3/envs/spr_2020s_env/lib/python3.7/site-packages/pymc3/tuning/starting.py:61: UserWarning: find_MAP should not be used to initialize the NUTS sampler, simply call pymc3.sample() and it will automatically initialize NUTS in a better way.
      warnings.warn('find_MAP should not be used to initialize the NUTS sampler, simply call pymc3.sample() and it will automatically initialize NUTS in a better way.')
    logp = -28.138, ||grad|| = 0.0017975: 100%|██████████| 22/22 [00:00<00:00, 2838.26it/s]

```python
for key, arr in map_N100.items():
    print(key, arr)
```

    alpha 0.9720043305924299
    beta [0.98456717 1.34380165]
    sigma_log__ -1.237493384514688
    sigma 0.290110502779077

```python
for key, arr in true_params.items():
    print(key, arr)
```

    alpha 1.0
    beta [1.    1.337]
    sigma 0.3

```python

```

# Run MCMC sampler for N=10 dataset

```python
with my_model_N10:
    start = map_N10

    # instantiate sampler
    mcmc_step = pymc3.NUTS(scaling=start)

    # draw 1000 posterior samples
    trace = pymc3.sample(
        1000, mcmc_step, start=start, chains=1, tune=500, discard_tuned_samples=True)

```

    Sequential sampling (1 chains in 1 job)
    NUTS: [sigma, beta, alpha]
    100%|██████████| 1500/1500 [00:01<00:00, 940.68it/s]
    Only one chain was sampled, this makes it impossible to run some convergence checks

```python
plot_traces(trace)
```

# Run MCMC sampler for N=100 dataset

```python
with my_model_N100:
    start = map_N10

    # instantiate sampler
    mcmc_step = pymc3.NUTS(scaling=start)

    # draw 1000 posterior samples
    trace_N100 = pymc3.sample(
        1000, mcmc_step, start=start, chains=1, tune=500, discard_tuned_samples=True)
```

    Sequential sampling (1 chains in 1 job)
    NUTS: [sigma, beta, alpha]
    100%|██████████| 1500/1500 [00:01<00:00, 1371.11it/s]
    Only one chain was sampled, this makes it impossible to run some convergence checks

```python
plot_traces(trace_N100)
```

```python

```


# Inference implementation tests

```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
import re
import subprocess
import datetime
from planar_ising import PlanarIsingModelGenerator, InferenceAndSampling

np.random.seed(45)

from scipy import special
```

# Problem statement

Let $V = \{ 1, ..., N \}, E \subseteq \binom{V}{2}, G = (V, E)$ be an undirected planar graph. For each $v \in V$ we assign a random binary variable (a spin) $s_v \in \{ -1, +1 \}$, $S = (s_1, ..., s_N)$. For each $e \in E$ we define a number $J_e \in \mathbb{R}$. We define a probability of the assignment $X = (x_1, ..., x_N) \in \{ -1, +1 \}^N$ as follows:

\begin{equation}
\mathbb{P} (S = X) = \frac{1}{Z} \exp{ \sum_{e = \{i, j\} \in E} J_e x_i x_j }
\label{eq:distr}
\end{equation}

Where

\begin{equation}
Z = \sum_{X \in \{ -1, +1 \}^N} \exp{ \sum_{e = \{i, j\} \in E} J_e x_i x_j }
\end{equation}

Given $G$'s plane embedding and a set of $J_e$, a goal is to find $Z$ efficiently.

In all the experiments conducted further we work with the following model generating parameters (check `PlanarIsingModelGenerator` docstrings):

```python
graph_density = 1.0
interaction_values_std = 0.1
```

# Correctness of implementation

We will test two slightly different inference/sampling algorithms - `EfficientInferenceAndSampling` and `WilsonInferenceAndSampling`. Both are numerically correct, but the first one tends to draw samples faster on practice.

To prove correctness of partition function, we test the computed value against Kac-Ward approach [1, Theorem 1]. It is relatively simple to implement, and this is what the next cell does. The disadvantage of the approach is that it results in $O(| V |^3)$ time complexity, while our algorithm is $O(| V |^{\frac32})$.

Kac-Ward approach requires a straight line embedding of $G$. To find that, we use `straightline` executable borrowed from [here](https://github.com/jeffythedragonslayer/lipton-tarjan).

```python
def compute_log_partition_function_kac_ward(ising_model):

    embedding_points = get_embedding_points(ising_model.graph)
    interactions_matrix = get_interactions_matrix(ising_model)

    spins_count = ising_model.graph.size

    kac_ward_matrix = get_kac_ward_matrix(embedding_points, interactions_matrix)

    _, kac_ward_det_log = np.linalg.slogdet(kac_ward_matrix)

    pairwise_interaction_coshs = np.cosh(interactions_matrix[np.triu_indices(spins_count)])

    return np.log(2)*spins_count + kac_ward_det_log/2 + \
            np.log(pairwise_interaction_coshs).sum()

def get_embedding_points(graph):

    edge_list_filename = 'edgelist.tmp'
    embedding_filename = 'embedding.tmp'
    embedding_command = './straightline {}'.format(edge_list_filename)

    edge_list_lines = []

    for vertex in range(graph.size):
        for adjacent_vertex in graph.get_adjacent_vertices(vertex):
            if vertex < adjacent_vertex:
                edge_list_lines.append('{0}, {1}'.format(vertex, adjacent_vertex))

    with open(edge_list_filename, 'w') as edge_list_file:
        edge_list_file.write('\n'.join(edge_list_lines))

    with open(embedding_filename, 'w') as embedding_file:
        subprocess.run(embedding_command.split(), stdout=embedding_file)

    pattern = re.compile(r'^\d+\[pos\=\"(\d+)\,(\d+)\!\"\]\;')

    embedding_points = []

    with open(embedding_filename, 'r') as embedding_file:
        for line in embedding_file:

            match = pattern.match(line)

            if match is None:
                continue

            embedding_points.append([float(match[1]), float(match[2])])

            if len(embedding_points) == graph.size:
                break

    return np.asarray(embedding_points)

def get_interactions_matrix(ising_model):

    graph = ising_model.graph
    interaction_values = ising_model.interaction_values

    interactions_matrix = np.zeros((graph.size, graph.size), dtype=np.float64)

    interactions_matrix[graph.edges.vertex1, graph.edges.vertex2] = interaction_values
    interactions_matrix[graph.edges.vertex2, graph.edges.vertex1] = interaction_values

    return interactions_matrix

def get_kac_ward_matrix(embedding_points, interactions_matrix):

    spins_count = len(embedding_points)

    directed_edge_indices = {edge:i for i, edge in \
            enumerate(zip(*np.where(interactions_matrix != 0)))}
    directed_edges_count = len(directed_edge_indices)

    kac_ward_matrix = np.zeros((directed_edges_count, directed_edges_count),
                               dtype=np.complex128)

    for spin_index1 in range(spins_count):

        for spin_index2 in range(spins_count):

            if interactions_matrix[spin_index1, spin_index2] == 0:
                continue

            directed_edge_index1 = directed_edge_indices[(spin_index1, spin_index2)]

            kac_ward_matrix[directed_edge_index1, directed_edge_index1] = 1

            for spin_index3 in range(spins_count):

                if interactions_matrix[spin_index2, spin_index3] == 0:
                    continue

                if spin_index1 == spin_index3:
                    continue

                point1 = embedding_points[spin_index1]
                point2 = embedding_points[spin_index2]
                point3 = embedding_points[spin_index3]

                half_rotation_number = get_half_rotation_complex_number(point1, point2,
                        point3)

                directed_edge_index2 = directed_edge_indices[(spin_index2, spin_index3)]

                kac_ward_matrix[directed_edge_index1, directed_edge_index2] = \
                        -half_rotation_number*\
                        np.tanh(interactions_matrix[spin_index2, spin_index3])

    return kac_ward_matrix

def get_half_rotation_complex_number(point1, point2, point3):

    vector1 = point2 - point1
    vector2 = point3 - point2

    dot_product = (vector1*vector2).sum()
    cross_product_z = vector1[0]*vector2[1] - vector1[1]*vector2[0]

    angle_cos = dot_product/(np.linalg.norm(vector1)*np.linalg.norm(vector2))
    half_angle_cos = np.sqrt((1 + angle_cos)/2)
    half_angle_sin = -np.sign(cross_product_z)*np.sqrt((1 - angle_cos)/2)

    return half_angle_cos + 1j*half_angle_sin
```

For a given model let $Z$ be a partition function computed by our algorithm and $Z'$ - by Kac-Ward approach. Then we define a relative error of our approach as $\frac{| \log Z - \log Z' |}{| \log Z' |}$.

The next cell generates Planar Ising Models of different size and computes relative error for them. Then errors are drawn as a scatter plot depending on the model size. One can observe that the error values are small enough to consider partition function values as correct.

Runtime warnings in the output are related to determinant computations in Kac-Ward approach, which suffers from numerical instabilities.

```python
graph_sizes = np.arange(5, 105, 5)
samples_count = 20

graph_sizes_per_sample = []
relative_errors_per_sample = []

for graph_size in graph_sizes:

    for sample_index in range(samples_count):

        graph_sizes_per_sample.append(graph_size)

        ising_model = PlanarIsingModelGenerator.generate_random_model(graph_size,
                graph_density, interaction_values_std)

        kac_ward_log_partition_function = \
                compute_log_partition_function_kac_ward(ising_model)

        ising_inference = InferenceAndSampling(ising_model)
        ising_inference.prepare()
        log_partition_function = ising_inference.compute_logpf()

        relative_error = np.absolute((kac_ward_log_partition_function - \
                float(log_partition_function))/kac_ward_log_partition_function)

        relative_errors_per_sample.append(relative_error)

    if graph_size%10 == 0:
        print('Done with size', graph_size)

plt.scatter(graph_sizes_per_sample, np.log10(relative_errors_per_sample), s=10,
            c='b')
plt.title('Relative error w.r.t. Kac-Ward')
plt.ylabel('$\log_{10} (error)$')
plt.xlabel('size')

plt.show()
```

# Execution time

The following cell generates Planar Ising Models of different size and measures the time of partition function computation. Then benchmarks are drawn on the scatter plot with model's size on x axis. Logarithmic axes clearly illustrate $O(N^{\frac32})$ time complexity of the algorithm.

```python
log2_graph_sizes = np.arange(2, 13)
samples_count = 5

plt.figure(figsize=(9, 6))

log2_graph_sizes_per_sample = []
execution_times_per_samples = []

for log2_graph_size in log2_graph_sizes:

    graph_size = 2**log2_graph_size

    for sample_index in range(samples_count):

        log2_graph_sizes_per_sample.append(log2_graph_size)

        ising_model = PlanarIsingModelGenerator.generate_random_model(graph_size,
                graph_density, interaction_values_std)

        start_moment = datetime.datetime.now()

        ising_inference = InferenceAndSampling(ising_model)
        ising_inference.prepare()
        ising_inference.compute_logpf()

        execution_time = (datetime.datetime.now() - start_moment).total_seconds()

        execution_times_per_samples.append(execution_time)

    if log2_graph_size > 8:
        print('Done with size {}'.format(graph_size))

plt.scatter(log2_graph_sizes_per_sample, np.log2(execution_times_per_samples),
        c='b', s=8)

plt.plot(log2_graph_sizes, 3*log2_graph_sizes - 10, '--', label='$O(N^3)$')
plt.plot(log2_graph_sizes, 1.5*log2_graph_sizes - 10, '--', label='$O(N^{1.5})$')

plt.title('Execution time')
plt.xlabel('$\log_2 N$')
plt.ylabel('$\log_2 (sec.)$')
plt.legend()

plt.legend()
plt.show()
```

# References

[1] - J. K. Johnson, D. Oyen, M. Chertkov, and P. Netrapalli, "Learning planar ising models", J. Mach. Learn. Res., vol. 17, pp. 7539–7564, Jan. 2016.


# Python Practice Lecture 16 MATH 342W Queens College - K-fold Cross Validation
## Author: Amir ElTabakh
## Date: April 5, 2022

## Agenda:
* The K Tradeoff
* Reducing variance with Cross Validation (i.e. K-fold CV)

## The K Tradeoff

$K$ determines how large the training set is relative to the test set when you're doing honest validation for an algorithm. For now, let's not use K-fold CV, but only examine one split at a time. Consider this simulated dataset with 50 observations:

```python
# importing dependencies
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error # calculates RMSE
import pandas as pd
import random

# ignore warnings
import warnings
warnings.filterwarnings("ignore")

# Data viz
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_palette(sns.color_palette("colorblind")) # setting color palette
sns.set(rc={"figure.figsize":(10, 6)}) #width=10, #height=6
```

```python
# setting seed
np.random.seed(2022)

# constants
n = 50
xmin = 0
xmax = 4

# data vecs
x = np.random.uniform(low = xmin, high = xmax, size = n)
y = [2 + 3 * i ** 2 + np.random.normal(0, 0.8) for i in x]

# convert to df
Xy = pd.DataFrame({'y': y, 'x': x})

# snapshot
Xy.head()
```

      y
      x

      0
      1.475622
      0.037434

      1
      13.308573
      1.996231

      2
      3.327321
      0.453535

      3
      1.945917
      0.199896

      4
      23.797992
      2.741630

```python
from plotnine import ggplot, aes, geom_point

# plot the data
data_plot = ggplot(Xy) + aes(x = x, y = y) + geom_point()
data_plot
```

Note how $f(x)$ is quadratic and there is random noise which is "ignorance error". The random noise will be part of generalization error and can never go away.

If we use OLS with no derived features, then we can at most get $h*(x)$. Let's see what $h^*(x) = \beta_0 + \beta_1 x$ truly is. To do this, we imagine we see an absolute ton of data and run OLS on it.

```python
# setting seed
np.random.seed(1738)

n_hidden = 1e6

# data vecs
x_hidden = np.random.uniform(low = xmin, high = xmax, size = n)
y_hidden = [2 + 3 * i ** 2 + np.random.normal(0, 0.8) for i in x_hidden]

# convert to dataframes
x_hidden = pd.DataFrame(x_hidden)
y_hidden = pd.DataFrame(y_hidden)

# h_*
h_star_model = LinearRegression().fit(x_hidden, y_hidden)

print(h_star_model.intercept_[0], h_star_model.coef_[0][0])
```

    -5.7301681875688715 11.925324444538758

The fact that $\beta = [-6~12]^\top$ can actually be solved with calculus: $\int_0^4 ((2 + 3x^2) - (b_0 + b_1 x))^2 dx$ and solve for $b_0$ and $b_1$ explicitly by minimizing.

Plotting that over $\mathbb{D}$ we obtain

```python
from plotnine import geom_abline

data_plot + geom_abline(
    intercept = h_star_model.intercept_[0],
    slope = h_star_model.coef_[0][0],
    color = 'green')
```

That is the best we're going to get. However, $g_{final}$ falls far short of it:

```python
# h_*
g_final_model = LinearRegression().fit(Xy[['x']], Xy[['y']])

print(g_final_model.intercept_[0], g_final_model.coef_[0][0])
```

    -5.779880916597136 12.040200380319078

The actual standard error of g_final can be estimated by imagining tons of future observations:

```python
y_hat_g_final = g_final_model.predict(x_hidden)

gen_error_true = np.std(y_hidden - y_hat_g_final)
gen_error_true[0]
```

    3.895685515342658

The model $g$ can vary quite a bit as we subsample $\mathbb{D}$ which is what happens when you do train-test splits. It varies a lot because there is large misspecification error. If the model was correctly specified, the results of everything that follows will be less impressive. But in the real world - is your model ever correctly specified? is $f \in \mathcal{H}$?? NO. So this is more realistic.

Now let's let K be small. Let K = 2 meaning even 50-50 split of test and train.

Sets, initialized with paranthesis, not square brackets, allow us to reduce one list from the other using the subtract operator.

```python
# setting seed
random.seed(17388) # two different libraries to set seeds for

# constants
K = 2
prop_train = (K - 1) / K
n_train = int(prop_train * n)

# train test split indices
indices = set([i for i in range(n)])
index_train = set(random.sample(indices, n_train))
index_test = indices - index_train

# split based on indices
x_train = pd.DataFrame([x[i] for i in index_train]) # cast from set to list
y_train = pd.DataFrame([y[i] for i in index_train])

x_test = pd.DataFrame([x[i] for i in index_test])
y_test = pd.DataFrame([y[i] for i in index_test])

# fitting model
g_mod = LinearRegression().fit(x_train, y_train)

# yhat via test set
y_hat_g_test = g_mod.predict(x_test)

# yhat via train set
y_hat_g_train = g_mod.predict(x_train)

# standard error
gen_error_K_2_test = np.std(y_test - y_hat_g_test) # test
gen_error_K_2_train = np.std(y_train - y_hat_g_train) # train

print(gen_error_true[0]) # true standard error
print(gen_error_K_2_test[0]) # tested standard error
```

    3.895685515342658
    3.866651151341103

Although I cooked the books by setting the seed, this realization makes sense. If K=2, I build the model g with half the data than the model g_final. Less data to train on => higher generalization error. How about if K is large. Let's say $K = \frac{n}{2}$ meaning $n_{train} = 48$ and $n_{test} = 2$.

```python
# setting seed
np.random.seed(7777)
random.seed(7777) # two different libraries to set seeds for

# constants
K = n / 2
prop_train = (K - 1) / K
n_train = int(prop_train * n)

# train test split indices
indices = set([i for i in range(n)])
index_train = set(random.sample(indices, n_train))
index_test = indices - index_train

# split based on indices
x_train = pd.DataFrame([x[i] for i in index_train]) # cast from set to list
y_train = pd.DataFrame([y[i] for i in index_train])

x_test = pd.DataFrame([x[i] for i in index_test])
y_test = pd.DataFrame([y[i] for i in index_test])

# fitting model
g_mod = LinearRegression().fit(x_train, y_train)

# yhat via test set
y_hat_g_test = g_mod.predict(x_test)

# yhat via train set
y_hat_g_train = g_mod.predict(x_train)

# standard error
gen_error_K_2_test = np.std(y_test - y_hat_g_test) # test
gen_error_K_2_train = np.std(y_train - y_hat_g_train) # train

print(gen_error_true[0]) # true standard error
print(gen_error_K_2_test[0]) # tested standard error
print(gen_error_K_2_train[0]) # train standard error
```

    3.895685515342658
    4.721271620787243
    3.9750058046527084

Although I cooked the books again by setting the seed, this also makes sense. More data to train on means less error but still more error than all the data. In reality, there is massive variance over specific splits! Let's run the simulation with these two K values many times.

While we're at it, let's do all K's! Well, what are all the valid K's? If you want to keep the sizes the same, any factorization of n except the trivial 1 since n = 1 * n. A $K = 1$ would mean there's no split! How to find divisors? Of course a package for this.

```python
# import divisors() method from sympy
from sympy import divisors

Kuniques = list(set(divisors(n)) - set([1]))
Kuniques
```

    [2, 5, 10, 50, 25]

But should we also include the trivial n? Yes K = n is indeed a valid divisor. And this type of CV is called the "leave one out cross validation" (LOOCV). Now we compute the errors over K:

```python
# constants
Nsim_per_K = 2000 # simulations per K fold
num_Kuniques = len(Kuniques)
Ks = Kuniques * Nsim_per_K

results = pd.DataFrame({'standard_error': [0 for i in range(Nsim_per_K * num_Kuniques)],
                        'K': [0 for i in range(Nsim_per_K * num_Kuniques)]})

for i in range(len(Ks)):
    # constants
    K = Ks[i]
    prop_train = (K - 1) / K
    n_train = int(prop_train * n)

    # train test split indices
    indices = set([i for i in range(n)])
    index_train = set(random.sample(indices, n_train))
    index_test = indices - index_train

    # split based on indices
    X_train = pd.DataFrame([x[i] for i in index_train]) # cast from set to list
    y_train = pd.DataFrame([y[i] for i in index_train])

    X_test = pd.DataFrame([x[i] for i in index_test])
    y_test = pd.DataFrame([y[i] for i in index_test])

    # fitting model
    g_mod = LinearRegression().fit(X_train, y_train)

    # get predictions
    y_hat_g = g_mod.predict(X_test)

    # get error metric
    if len(y_test) == 1:
        g_s_e = np.absolute([float(y_test.iloc[i]) - y_hat_g[i] for i in range(len(y_test))])
    else:
        g_s_e = np.std([float(y_test.iloc[i]) - y_hat_g[i] for i in range(len(y_test))])

    # update results
    results.loc[i] = np.asarray([g_s_e, K])

# output results
results
```

      standard_error
      K

      0
      4.854482
      2.0

      1
      2.285150
      5.0

      2
      3.105590
      10.0

      3
      7.310184
      50.0

      4
      0.852891
      25.0

      ...
      ...
      ...

      9995
      4.322564
      2.0

      9996
      2.757709
      5.0

      9997
      3.843814
      10.0

      9998
      3.889604
      50.0

      9999
      0.186950
      25.0

10000 rows × 2 columns

What are the variabilities? Let's take the average error over each simulated split.

```python
# mean of the standard errors grouped by K
results_summary = pd.DataFrame({'K': list(results.groupby(['K']).mean().index),
                                 'K_avg': list(results.groupby(['K']).mean()['standard_error']),
                                 'K_SE': list(results.groupby(['K']).std()['standard_error'])
                                }).set_index('K')

results_summary
```

      K_avg
      K_SE

      K

      2.0
      4.138852
      0.349815

      5.0
      3.923679
      0.598949

      10.0
      3.586957
      0.995828

      25.0
      2.351176
      1.800594

      50.0
      3.716647
      1.875211

```python
sns.displot(results,
             x = 'standard_error',
             hue = 'K',
             kind = 'kde',
             multiple="stack",
             palette = 'bright',
             alpha = 0.9,
             aspect = 1.5) # width = height * aspect

plt.axvline(label='Mean 2.0', x=results_summary.iloc[0, 0], color='blue')
plt.axvline(label='Mean 5.0', x=results_summary.iloc[1, 0], color='orange')
plt.axvline(label='Mean 10.0', x=results_summary.iloc[2, 0], color='green')
plt.axvline(label='Mean 25.0', x=results_summary.iloc[3, 0], color='red')
plt.axvline(label='Mean 50.0', x=results_summary.iloc[4, 0], color='purple')
```

The main takeaways are

1. The std err of generalization error estimate is much lower for low K than high K

With high K, the test set is small meaning the estimate has high variance; with low K, the test set is large meaning you can measure it with low variance.

2. The average of generalization error estimate is lower for high K than low K

With high K, the training set is large meaning $g$ is closer to g_final and thus has higher expected accuracy; with low K, the training set is small meaning $g$ is further from g_final and thus has lower expected accuracy.

Thus, the tradeoff is bias vs. variance. There are many similar tradeoffs in statistics. We will see one later when we do machine learning.

Is the estimates' accuracy for what we really care about? No, the generalization error of g_final which we picture below:

```python
sns.displot(results,
             x = 'standard_error',
             hue = 'K',
             kind = 'kde',
             multiple="stack",
             palette = 'bright',
             alpha = 0.9,
             aspect = 1.5) # width = height * aspect

plt.axvline(label='Mean 2.0', x=results_summary.iloc[0, 0], color='blue')
plt.axvline(label='Mean 5.0', x=results_summary.iloc[1, 0], color='orange')
plt.axvline(label='Mean 10.0', x=results_summary.iloc[2, 0], color='green')
plt.axvline(label='Mean 25.0', x=results_summary.iloc[3, 0], color='red')
plt.axvline(label='Mean 50.0', x=results_summary.iloc[4, 0], color='purple')
plt.axvline(label='CV', x=gen_error_true[0], color='white')

```

Remember, g_final's error should be lower than both averages since it uses all the data. But we see above it's higher!

So what happened? Simple... we are mixing apples and oranges. We calculated that white line by looking at one million future observations. We calculated the red and blue distributions by looking at our data only which is a random realization of many such datasets! Thus, our generalization errors are biased based on the specific n observations in D we received. We will see that K-fold helps a bit with this. But there is nothing we can do about it beyond that (besides collect more observations). If you get a weird sample, you get a weird sample!

How would we be able to generate the picture we really want to see? We would run this simulation over many datasets and average. That would be a giant simulation. To show that this is the case, go back and change the seed in the first chunk and rerun. You'll see a different white bar.

What is the main takeaway? K matters because it induces a tradeoff. It shouldn't be too large or too small (as we believe at the moment). And, generalization error estimation is very variable in low n. To see this, go back and increase n.

## Reducing variance with Cross Validation (i.e. K-fold CV)

We saw previous there was a lot of variance in generalization error estimation. We can reduce some of this variance by using a very simple trick. We can rotate the train-test split so that each observation will be in the test set once. How many times is this done? K. Now we see the reason for the definition of K as it tells you how many times you validate. Why is it called "cross"? Because the training set crosses over as it does the rotation. Each observation is inside a training set K-1 times. This point will become important later. Why is it called K-fold? Because a fold is one set of training-test and there are K unique folds during the whole procedure.

How does this work? Well, let's say K=10, a typical value. This means in each "fold", 90% of the data is in the training set and 10% of the data is in the test set. As we run through the K folds, we train a model on the training set and predict on the test set and compute oos residuals We aggregate those oos residuals over the folds to result in n oos residuals. We then run our error metric on all n.

Let's begin with the dataset from the previous demo. Here is that code that will create the folds by specifying the K=10 test sets by index. The training sets can then be found by the set difference function.

We're gonna jump right into using sci-kit learn's KFold module.

```python
# importing dependencies
from sklearn.model_selection import KFold

# initializing model
model = LinearRegression() # intercept will fit by default

# 10 folds
K = 10
kf = KFold(n_splits=K)

# list to hold residuals
oos_cross_validation_residuals = []

for train_index , test_index in kf.split(x):
    X_train = pd.DataFrame([x[i] for i in train_index])
    y_train = pd.DataFrame([y[i] for i in train_index])

    X_test = pd.DataFrame([x[i] for i in test_index])
    y_test = pd.DataFrame([y[i] for i in test_index])

    model.fit(X_train,y_train)
    y_hat_g = model.predict(X_test)

    oos_cross_validation_residuals += [np.std(y_test - y_hat_g)]

oos_cross_validation_residual_average = np.std(oos_cross_validation_residuals)
oos_cross_validation_residual_average
```

    0.8584252859259888

How does this CV error look over K?

```python
# import divisors() method from sympy
from sympy import divisors

Kuniques = list(set(divisors(n)) - set([1]))
Kuniques
```

    [2, 5, 10, 50, 25]

```python
# constants
Nsim_per_K = 1 # simulations per K fold
num_Kuniques = len(Kuniques)
Ks = Kuniques * Nsim_per_K
n = len(y)

oos_residuals_CV = pd.DataFrame({'standard_error': [0 for i in range(Nsim_per_K * num_Kuniques)],
                        'K': [0 for i in range(Nsim_per_K * num_Kuniques)]})

for i in range(len(Ks)):
    # constants
    K = Ks[i]
    prop_train = (K - 1) / K
    n_train = int(prop_train * n)

    # train test split indices
    indices = set([i for i in range(n)])
    train_index = set(random.sample(indices, n_train))
    test_index = indices - train_index

    # cast as lists
    train_index = list(train_index)
    test_index = list(test_index)

    # split based on indices
    X_train = pd.DataFrame([x[i] for i in train_index])
    y_train = pd.DataFrame([y[i] for i in train_index])

    X_test = pd.DataFrame([x[i] for i in test_index])
    y_test = pd.DataFrame([y[i] for i in test_index])

    # fitting model
    g_mod = LinearRegression().fit(X_train, y_train)

    # get predictions
    y_hat_g = g_mod.predict(X_test)

    # get error metric
    if len(y_test) == 1:
        g_s_e = np.absolute([float(y_test.iloc[i]) - y_hat_g[i] for i in range(len(y_test))])
    else:
        g_s_e = np.std([float(y_test.iloc[i]) - y_hat_g[i] for i in range(len(y_test))])

    # update results
    oos_residuals_CV.loc[i] = np.asarray([g_s_e, K])

# output results
oos_residuals_CV
```

      standard_error
      K

      0
      3.672747
      2.0

      1
      3.954330
      5.0

      2
      3.224956
      10.0

      3
      7.383997
      50.0

      4
      1.272782
      25.0

There is still an effect of the one random fold. Let's do this many times and look at the distribution just like before.

```python
# constants
Nsim_per_K = 500 # simulations per K fold
num_Kuniques = len(Kuniques)
Ks = Kuniques * Nsim_per_K
n = len(y)

oos_residuals = pd.DataFrame({'standard_error': [0 for i in range(Nsim_per_K * num_Kuniques)],
                        'K': [0 for i in range(Nsim_per_K * num_Kuniques)]})

for i in range(len(Ks)):
    # constants
    K = Ks[i]
    prop_train = (K - 1) / K
    n_train = int(prop_train * n)

    # train test split indices
    indices = set([i for i in range(n)])
    train_index = set(random.sample(indices, n_train))
    test_index = indices - train_index

    # cast as lists
    train_index = list(train_index)
    test_index = list(test_index)

    # split based on indices
    X_train = pd.DataFrame([x[i] for i in train_index])
    y_train = pd.DataFrame([y[i] for i in train_index])

    X_test = pd.DataFrame([x[i] for i in test_index])
    y_test = pd.DataFrame([y[i] for i in test_index])

    # fitting model
    g_mod = LinearRegression().fit(X_train, y_train)

    # get predictions
    y_hat_g = g_mod.predict(X_test)

    # get error metric
    if len(y_test) == 1:
        g_s_e = np.absolute([float(y_test.iloc[i]) - y_hat_g[i] for i in range(len(y_test))])
    else:
        g_s_e = np.std([float(y_test.iloc[i]) - y_hat_g[i] for i in range(len(y_test))])

    # update results
    oos_residuals.loc[i] = np.asarray([g_s_e, K])

# output results
oos_residuals
```

      standard_error
      K

      0
      4.185499
      2.0

      1
      4.193835
      5.0

      2
      3.317849
      10.0

      3
      2.990152
      50.0

      4
      0.089897
      25.0

      ...
      ...
      ...

      2495
      4.207371
      2.0

      2496
      3.761627
      5.0

      2497
      5.222147
      10.0

      2498
      5.706351
      50.0

      2499
      3.677768
      25.0

2500 rows × 2 columns

What is the variability?

```python
# mean of the standard errors grouped by K
oos_results_summary = pd.DataFrame({'K': list(oos_residuals.groupby(['K']).mean().index),
                                 'K_avg': list(oos_residuals.groupby(['K']).mean()['standard_error']),
                                 'K_SE': list(oos_residuals.groupby(['K']).std()['standard_error'])
                                }).set_index('K')

oos_results_summary
```

      K_avg
      K_SE

      K

      2.0
      4.146039
      0.395974

      5.0
      3.877967
      0.579897

      10.0
      3.546045
      1.018892

      25.0
      2.368054
      1.752468

      50.0
      3.727671
      1.949647

```python
sns.displot(oos_residuals,
             x = 'standard_error',
             hue = 'K',
             kind = 'kde',
             multiple="stack",
             palette = 'bright',
             alpha = 0.9,
             aspect = 1.5) # width = height * aspect

plt.axvline(label='Mean 2.0', x=oos_results_summary.iloc[0, 0], color='blue')
plt.axvline(label='Mean 5.0', x=oos_results_summary.iloc[1, 0], color='orange')
plt.axvline(label='Mean 10.0', x=oos_results_summary.iloc[2, 0], color='green')
plt.axvline(label='Mean 25.0', x=oos_results_summary.iloc[3, 0], color='red')
plt.axvline(label='Mean 50.0', x=oos_results_summary.iloc[4, 0], color='purple')
plt.axvline(label='CV', x=oos_cross_validation_residual_average, color='black')

```

Admittedly, I don't know the properties of CV estimates as well as I should. Thus, there will be only procedural questions on the next exam. I do know that selecting K "optimally" for general datasets is an open question.

There is one other nice thing about having folds, you can estimate the standard error in your generalization estimate by pretending you have K iid samples and pretending the normal theory applies. For example, let's say K = 5. Instead of aggregating all residuals, we leave them separate and get K = 5 difference estimates for generalization error.

```python
# importing dependencies
from sklearn.model_selection import KFold

# initializing model
model = LinearRegression() # intercept will fit by default

# 10 folds
K = 10
kf = KFold(n_splits=K)

# list to hold residuals
oos_cross_validation_residuals = []

for train_index , test_index in kf.split(x):
    X_train = pd.DataFrame([x[i] for i in train_index])
    y_train = pd.DataFrame([y[i] for i in train_index])

    X_test = pd.DataFrame([x[i] for i in test_index])
    y_test = pd.DataFrame([y[i] for i in test_index])

    model.fit(X_train,y_train)
    y_hat_g = model.predict(X_test)

    oos_cross_validation_residuals += [np.std(y_test - y_hat_g)]

oos_cross_validation_residual_mean = np.mean(oos_cross_validation_residuals)
print(oos_cross_validation_residual_mean)

oos_cross_validation_residual_std = np.std(oos_cross_validation_residuals)
print(oos_cross_validation_residual_std)
```

    3.950922846645746
    0.8584252859259888

```python
# calculating confidence interval
import scipy.stats as st

confidence_interval = st.t.interval(alpha = 0.95,
                                   df = len(y) - 1,
                                   loc = oos_cross_validation_residual_mean,
                                   scale = oos_cross_validation_residual_std)

print(confidence_interval)
print(min(y), max(y))
```

    (2.2258526513895607, 5.675993041901931)
    1.2627430676861833 48.69439853538999

Although this is technically nonsense since they're not iid samples since the training set is crossed over containing mostly the same observations, at least it's something. In the above example, we've managed to capture the true generalization error.

Coverage in this confidence interval is over D. So I wouldn't gain much insight by simulating different splits with the same K.

I believe confidence intervals for generalization error is an open problem or maybe proved that you can't find them in general situations.

Here's a real data example with the `breast_cancer` dataset.

```python
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer(as_frame = True)
df = data.frame
X = df.iloc[:,:-1]
y = df.iloc[:,-1]

X
```

      mean radius
      mean texture
      mean perimeter
      mean area
      mean smoothness
      mean compactness
      mean concavity
      mean concave points
      mean symmetry
      mean fractal dimension
      ...
      worst radius
      worst texture
      worst perimeter
      worst area
      worst smoothness
      worst compactness
      worst concavity
      worst concave points
      worst symmetry
      worst fractal dimension

      0
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.30010
      0.14710
      0.2419
      0.07871
      ...
      25.380
      17.33
      184.60
      2019.0
      0.16220
      0.66560
      0.7119
      0.2654
      0.4601
      0.11890

      1
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.08690
      0.07017
      0.1812
      0.05667
      ...
      24.990
      23.41
      158.80
      1956.0
      0.12380
      0.18660
      0.2416
      0.1860
      0.2750
      0.08902

      2
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.19740
      0.12790
      0.2069
      0.05999
      ...
      23.570
      25.53
      152.50
      1709.0
      0.14440
      0.42450
      0.4504
      0.2430
      0.3613
      0.08758

      3
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.24140
      0.10520
      0.2597
      0.09744
      ...
      14.910
      26.50
      98.87
      567.7
      0.20980
      0.86630
      0.6869
      0.2575
      0.6638
      0.17300

      4
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.19800
      0.10430
      0.1809
      0.05883
      ...
      22.540
      16.67
      152.20
      1575.0
      0.13740
      0.20500
      0.4000
      0.1625
      0.2364
      0.07678

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      564
      21.56
      22.39
      142.00
      1479.0
      0.11100
      0.11590
      0.24390
      0.13890
      0.1726
      0.05623
      ...
      25.450
      26.40
      166.10
      2027.0
      0.14100
      0.21130
      0.4107
      0.2216
      0.2060
      0.07115

      565
      20.13
      28.25
      131.20
      1261.0
      0.09780
      0.10340
      0.14400
      0.09791
      0.1752
      0.05533
      ...
      23.690
      38.25
      155.00
      1731.0
      0.11660
      0.19220
      0.3215
      0.1628
      0.2572
      0.06637

      566
      16.60
      28.08
      108.30
      858.1
      0.08455
      0.10230
      0.09251
      0.05302
      0.1590
      0.05648
      ...
      18.980
      34.12
      126.70
      1124.0
      0.11390
      0.30940
      0.3403
      0.1418
      0.2218
      0.07820

      567
      20.60
      29.33
      140.10
      1265.0
      0.11780
      0.27700
      0.35140
      0.15200
      0.2397
      0.07016
      ...
      25.740
      39.42
      184.60
      1821.0
      0.16500
      0.86810
      0.9387
      0.2650
      0.4087
      0.12400

      568
      7.76
      24.54
      47.92
      181.0
      0.05263
      0.04362
      0.00000
      0.00000
      0.1587
      0.05884
      ...
      9.456
      30.37
      59.16
      268.6
      0.08996
      0.06444
      0.0000
      0.0000
      0.2871
      0.07039

569 rows × 30 columns

```python
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold

k = 5
kf = KFold(n_splits=k, random_state=None)

model = LinearRegression()

residuals = []

for train_index , test_index in kf.split(X):
    X_train , X_test = X.iloc[train_index,:],X.iloc[test_index,:]
    y_train , y_test = y[train_index] , y[test_index]

    model.fit(X_train,y_train)
    y_hat_g = model.predict(X_test)

    residuals += [np.std(y_test - y_hat_g)]

print(np.mean(residuals))
print(np.std(residuals))
```

    0.24089771870604665
    0.03227633906367752

```python
# calculating confidence interval
import scipy.stats as st

confidence_interval = st.t.interval(alpha = 0.95,
                                   df = len(X) - 1,
                                   loc = np.mean(residuals),
                                   scale = np.std(residuals))

print(confidence_interval)
```

    (0.17750217054311385, 0.30429326686897945)

Practice Notes/16 - Python Practice Lecture 16 MATH 342W Queens College - K-fold Cross Validation.ipynb


# Model project

```python
import numpy as np
import scipy as sp
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm

%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

In this project we analyze a consumer problem with a CES utility function and two goods, x1 and x2.

The utility function: alpha is a preference parameter and rho is the elasticity parameter
The income is 30 and the prices of the goods are: \\[ p_1=2, p_2=4\\].

First we investigate the jacobian and hessian matrix and how to use gradient based optimizers. In this section we use the *minimize_gradient_descent()* method.

In the next section the budget constraint is introduced and we ty to solve the consumer problem using a *Multi-dimensional constrained solver*.

After this the consumer problem is solved by using the optimizer **SLSQP**.

In the last section we introduce an extension to the problem where we impose a tax on good x2, which increases the price p2 with 0.5 for each consumed x2.

Consider the **CES utility function**:

\\[ f(\boldsymbol{x}) = u(x_1,x_2) =(\alpha x_{1}^{-\rho}+(1-\alpha) x_{2}^{-\rho})^{-1/\rho} \\]

We choose the parameters:

\\[ \alpha=0.2 \\]

\\[ \rho=-0.2 \\]

```python
sm.init_printing(use_unicode=True)
x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
f = (0.2*x1**(2) + (1-0.2)*x2**(2))**(1/2)
```

Find the Jacobian matrix:

```python
Df = sm.Matrix([sm.diff(f,i) for i in [x1,x2]])
Df
```

Find the Hessian matrix:

```python
Hf = sm.Matrix([[sm.diff(f,i,j) for j in [x1,x2]] for i in [x1,x2]])
Hf
```

I now define the function and the matrixes:

```python
def _ces(x1,x2):
    return (0.2*x1**(2) + (1-0.2)*x2**(2))**(1/2)
def ces(x):
    return _ces(x[0],x[1])
def ces_jac(x):
    return np.array([(0.2*x[0])/(0.2*x[0]**2+0.8*x[1]**2)**0.50,(0.8*x[1])/((0.2*x[0]**2+0.8*x[1]**2)**0.5)])
def ces_hess(x):
    return np.array([0.2/(0.2*x[0]**2+0.8*x[1]**2)**0.5-(0.04*x[0]**2)/(0.2*x[0]**2+0.8*x[1]**2)**1.5,-(0.16*x[0]*x[1])/(0.2*x[0]**2+0.8*x[1]**2)**1.5,-(0.16*x[0]*x[1])/(0.2*x[0]**2+0.8*x[1]**2)**1.5,0.8/(0.2*x[0]**2+0.8*x[1]**2)**0.5-(0.64*x[1]**2)/(0.2*x[0]**2+0.8*x[1]**2)**1.5])

```

I can now plot the function in 3D by using the definitions:

```python
# Grids:
x1_vec = np.linspace(-6,6,500)
x2_vec = np.linspace(-4,4,500)
x1_grid,x2_grid = np.meshgrid(x1_vec,x2_vec,indexing='ij')
ces_grid = _ces(x1_grid,x2_grid)

# Figure:
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(x1_grid,x2_grid,ces_grid,cmap=cm.jet)

# Labels:
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$u$')

ax.invert_xaxis()
ax.xaxis.pane.fill = False
ax.yaxis.pane.fill = False
ax.zaxis.pane.fill = False

#Color:
fig.colorbar(cs);
```

This figure shows the CES utility funtion. It shows that utility is smallest when x1 and x2 is equal to zero. Utility increases when x1 and x2 increases. Further it shows that x2 gives more utility than x1.

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
levels = [1e-6,5*1e-6,1e-5,5*1e-5,1e-4,5*1e-4,1e-3,5*1e-3,1e-2,5*1e-2,0.50,1.00,1.50,2.00,2.50,3.00,3.50,4.00,5.00]
cs = ax.contour(x1_grid,x2_grid,ces_grid,levels=levels,cmap=cm.jet)
fig.colorbar(cs);
```

Here we see a different kind of plot. It shows the same but in 2D. Utility is increased when x1 and x2 is increased. The circles are oval which shows that x2 give more utility than x1.

# Optimizing with gradient based optimizer

**Now we wish to optimize using the algorithm:** `minimize_gradient_descent()`

The idea is to make some guesses on x0, compute f(x0), find a step size and update guess. Keep doing this until it converges.

```python
def minimize_gradient_descent(f,x0,jac,alphas=[0.01,0.05,0.1,0.25,0.5,1],max_iter=5000,tol=1e-8):
    """ minimize function with gradient descent

    Args:

        f (callable): function
        x0 (float): initial value
        jac (callable): jacobian
        alpha (list): potential step sizes
        max_iter (int): maximum number of iterations
        tol (float): tolerance

    Returns:

        x (float): root
        n (int): number of iterations used

    """

    # step 1: initialize
    x = x0
    fx = f(x0)
    n = 1

    # step 2-6: iteration
    while n < max_iter:

        x_prev = x
        fx_prev = fx

        # step 2: evaluate gradient
        jacx = jac(x)

        # step 3: find good step size
        fx_ast = np.inf
        alpha_ast = np.nan
        for alpha in alphas:
            x = x_prev - alpha*jacx
            fx = f(x)
            if fx < fx_ast:
                fx_ast = fx
                alpha_ast = alpha

        # step 4: update guess
        x = x_prev - alpha_ast*jacx

        # step 5: check convergence
        fx = f(x)
        if abs(fx-fx_prev) < tol:
            break

        # d. update i
        n += 1

    return x,n
```

```python
x0 = np.array([5,4])
x,n = minimize_gradient_descent(ces,x0,ces_jac,alphas=[0.01,0.05,0.1,0.25,0.5,1])
print(n,x,ces(x))
```

    5000 [9.8813129e-324 4.2335895e-003] 0.0037866375632192124

**Solution:** We don't find any convergence and therefore no solution using the gradient based optimizer. Therefore we now try using the SLSQP optimizer, but still by iteration over gradient evaluations.

```python
def collect(x):

    global evals
    global x0
    global x1s
    global x2s
    global fs

    if evals == 0:
        x1s = [x0[0]]
        x2s = [x0[1]]
        fs = [ces(x0)]

    x1s.append(x[0])
    x2s.append(x[1])
    fs.append(ces(x))
    evals += 1
```

```python
def contour():

    global evals
    global x1s
    global x2s
    global fs

# Code for plotting the result:
    fig = plt.figure(figsize=(10,4))
    ax = fig.add_subplot(1,2,1)
    levels = [1e-6,5*1e-6,1e-5,5*1e-5,1e-4,5*1e-4,1e-3,5*1e-3,1e-2,5*1e-2,0.50,1.00,1.50,2.00,2.50,3.00,3.50,4.00,5.00]
    cs = ax.contour(x1_grid,x2_grid,ces_grid,levels=levels,cmap=cm.jet)
    fig.colorbar(cs)
    ax.plot(x1s,x2s,'-o',ms=4,color='black')
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')

    ax = fig.add_subplot(1,2,2)
    ax.plot(np.arange(evals+1),fs,'-o',ms=4,color='black')
    ax.set_xlabel('iteration')
    ax.set_ylabel('function value')
```

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(ces,x0,jac=ces_jac,
                           method='SLSQP',
                           bounds=((-2,2),(-2,2)),
                           callback=collect,
                           options={'disp':True})
contour()
```

We now see some convergence in the function with 23 iterations.

# Optimization problem

Optimization using the **SLSQP** method.

Consider now the consumer problem given by:

\\[
\begin{eqnarray*}
V(p_{1},p_{2},I) & = & \max_{x_{1},x_{2}}u(x_{1},x_{2})\\
 & \text{s.t.}\\
p_{1}x_{1}+p_{2}x_{2} & \leq & I\\
p_{1},p_{2},I>0\\
x_{1},x_{2} & \geq & 0
\end{eqnarray*}
\\]

where the utility is a CES funtion as described earlier.

We choose the parameters:

\\[ \alpha=0.20 \\]

\\[ \rho=-0.20 \\]

```python
def u_func(x1,x2,alpha=0.20,rho=-0.2):
    return (alpha*x1**(-rho)+(1-alpha)*x2**(-rho))**(-1/rho)
```

We now try to set a random value for x1 and x2 and calculate the utility:

```python
x1 = 2
x2 = 4
u = u_func(x1,x2)

print(f'x1 = {x1:.1f}, x2 = {x2:.1f} -> u = {u:.4f}')
```

    x1 = 2.0, x2 = 4.0 -> u = 3.5083

```python
print(u)
```

    3.508328497554137

The utility when the consumer buys two x1 and 4 x2, the utility is 3.5

Now we try to increase first x1 and then x2 to see how much it affects the utility:

```python
# Increase x1 to several random numbers:

x1_list = [2,4,6,8,10,12]
x2 = 3

for i,x1 in enumerate(x1_list):
    u = u_func(x1,x2,alpha=0.20,rho=-0.2)
    print(f'{i:2d}: x1 = {x1:<6.3f} x2 = {x2:<6.3f} -> u = {u:<6.3f}')
```

     0: x1 = 2.000  x2 = 3.000  -> u = 2.773
     1: x1 = 4.000  x2 = 3.000  -> u = 3.182
     2: x1 = 6.000  x2 = 3.000  -> u = 3.473
     3: x1 = 8.000  x2 = 3.000  -> u = 3.709
     4: x1 = 10.000 x2 = 3.000  -> u = 3.911
     5: x1 = 12.000 x2 = 3.000  -> u = 4.089

```python
# Increase x2 to several random numbers:

x2_list = [2,4,6,8,10,12]
x1 = 3

for i,x2 in enumerate(x2_list): # i is a counter
    u = u_func(x1,x2,alpha=0.20,rho=-0.2)
    print(f'{i:2d}: x1 = {x1:<6.3f} x2 = {x2:<6.3f} -> u = {u:<6.3f}')
```

     0: x1 = 3.000  x2 = 2.000  -> u = 2.175
     1: x1 = 3.000  x2 = 4.000  -> u = 3.781
     2: x1 = 3.000  x2 = 6.000  -> u = 5.262
     3: x1 = 3.000  x2 = 8.000  -> u = 6.673
     4: x1 = 3.000  x2 = 10.000 -> u = 8.036
     5: x1 = 3.000  x2 = 12.000 -> u = 9.362

This shows that in the beginning more of x1 gives much more utility, but after 10 units of x1, the marginal utility is very small, and more x1 doesn't give much more utility. But when x2 increases, the utility is also increased.

# Solve the consumer problem

 We now solve the consumer problem using a **Multi-dimensional constrained solver**.

The idea is to loop through a grid of \\(N_1 \times N_2\\) possible solutions for \\(x_1\\) and \\(x_2\\). This is a way of solving.

The problem is given by:

\\[
\begin{eqnarray*}
V(p_{1},p_{2},I) & = & \max_{x_{1}\in X_1}(\alpha x_{1}^{-\rho}+(1-\alpha) x_{2}^{-\rho})^{-1/\rho}\\
 & \text{s.t.}\\
 X_1 & = & \left\{0,\frac{1}{N-1}\frac{}{p_1},\frac{2}{N-1}\frac{I}{p_1},\dots,\frac{I}{p_1}\right\} \\
x_{2} & = & \frac{I-p_{1}x_{1}}{p_2}\\
\end{eqnarray*}
\\]

We choose Income and prices:

I=30, p1=2, p2=4

```python
def find_best_choice(alpha,rho,I,p1,p2,N1,N2,do_print=True):

    shape_tuple = (N1,N2)
    x1_values = np.empty(shape_tuple)
    x2_values = np.empty(shape_tuple)
    u_values = np.empty(shape_tuple)

    x1_best = 0
    x2_best = 0
    u_best = u_func(0,0,alpha=alpha,rho=rho)

    for i in range(N1):
        for j in range(N2):

            x1_values[i,j] = x1 = (i/(N1-1))*I/p1
            x2_values[i,j] = x2 = (j/(N2-1))*I/p2
            if p1*x1+p2*x2 <= I:
                u_values[i,j] = u_func(x1,x2,alpha=alpha,rho=rho)
            else:
                u_values[i,j] = u_func(0,0,alpha=alpha,rho=rho)

            if u_values[i,j] > u_best:
                x1_best = x1_values[i,j]
                x2_best = x2_values[i,j]
                u_best = u_values[i,j]

    if do_print:
        print_solution(x1_best,x2_best,u_best,I,p1,p2)

    return x1_best,x2_best,u_best,x1_values,x2_values,u_values

def print_solution(x1,x2,u,I,p1,p2):
    print(f'x1 = {x1:.8f}')
    print(f'x2 = {x2:.8f}')
    print(f'u  = {u:.8f}')
    print(f'I-p1*x1-p2*x2 = {I-p1*x1-p2*x2:.8f}')
```

```python
solution = find_best_choice(alpha=0.20,I=30,p1=2,p2=4,N1=200,N2=400,rho=-0.2)
```

    x1 = 2.63819095
    x2 = 6.16541353
    u  = 5.26100494
    I-p1*x1-p2*x2 = 0.06196396

We wish to find the best solution with high utility and low left-over income: $I-p_1 x_1-p_2 x_2$

This is not optimal and not the best way to solve it as the left-over income is greater than zero.

# Solve by using the SLSQP method

Now we use the SLSQP method to solve the consumer problem with a budget constraint. We use the same income and prices as before.

```python
from scipy import optimize
```

```python
alpha = 0.20 # preference parameter
rho = -0.20 # elasticity parameter
I = 30 # income
p1 = 2 # price on good x1
p2 = 4 # price on good x2
```

```python
def value_of_choice(x,alpha,rho,I,p1,p2):

    x1 = x[0]
    x2 = x[1]
    return -u_func(x1,x2,alpha,rho)

constraints = (
    {'type': 'ineq', 'fun': lambda x:  I-p1*x[0]-p2*x[1]}
)
bounds = (
    (0,I/p1),
    (0,I/p2)
)

# Call the SLSQP solver:
initial_guess = [I/p1/2,I/p2/2]
sol_optimize = optimize.minimize(
    value_of_choice,initial_guess,args=(alpha,rho,I,p1,p2),
    method='SLSQP',bounds=bounds,constraints=constraints)

x1 = sol_optimize.x[0]
x2 = sol_optimize.x[1]
u = u_func(x1,x2,alpha,rho)
print_solution(x1,x2,u,I,p1,p2)
```

    x1 = 2.60552384
    x2 = 6.19723808
    u  = 5.27198775
    I-p1*x1-p2*x2 = -0.00000000

Now the utility is increased and the left-over income is equal to zero. Therefore this is a better way to solve this consumer problem.

The maximum utility is 5.27 and the optimal consumption is 2.6 of good x1 and 6.2 of good x2.

**Plot the solution**

The plot shows utility as a function of consumed x1. From this we can see maximum utility

```python
fig = plt.figure(figsize=(12,4),dpi=100)
ax = fig.add_subplot(1,2,1)

ax.plot(x1_values,u_values, 'bo')

ax.set_title('value of choice, $u(x_1,x_2)$')
ax.set_xlabel('$x_1$')
ax.set_ylabel('$u(x_1,(I-p_1 x_1)/p_2)$')
ax.grid(True)
```

# Extension of the model

Change in budget constraint.

Now we introduce a tax on good x2, which increases when consuming more of x2. The price is increased by 0.5 for each consumed x2.

\\[
\begin{eqnarray*}
V(p_{1},p_{2},I) & = & \max_{x_{1}\in X_1}(\alpha x_{1}^{-\rho}+(1-\alpha) x_{2}^{-\rho})^{-1/\rho}\\
 & \text{s.t.}\\
 p_{1}*x_{1}+(p_{2}+0.5*x_{2})*x_{2} & \leq & I,\,\,\,p_{1},p_{2},I>0\\
x_{1},x_{2} & \geq & 0
\end{eqnarray*}
\\]

```python
alpha = 0.20 # preference parameter
rho = -0.20 # elasticity parameter
I = 30 # income
p1 = 2 # price 1
p2 = 4 # price 2
```

Again we use the SLSQP optimizer as it was the best one.

```python
def value_of_choice(x,alpha,rho,I,p1,p2):
    x1 = x[0]
    x2 = x[1]
    return -u_func(x1,x2,alpha,rho)

# Constraint:
constraints = (
    {'type': 'ineq', 'fun': lambda x:  I-p1*x[0]-(p2+0.5*x[1])*x[1]}
)
bounds = (
    (0,I/p1),
    (0,I/p2)
)

# Call the SLSQP optimizer:
initial_guess = [I/p1/2,I/p2/2]
sol_optimize = optimize.minimize(
    value_of_choice,initial_guess,args=(alpha,rho,I,p1,p2),
    method='SLSQP',bounds=bounds,constraints=constraints)

x1 = sol_optimize.x[0]
x2 = sol_optimize.x[1]
u = u_func(x1,x2,alpha,rho)
print_solution(x1,x2,u,I,p1,p2)
```

    x1 = 3.71257356
    x2 = 3.81982781
    u  = 3.79818113
    I-p1*x1-p2*x2 = 7.29554164

**Plot the solution**

```python
fig = plt.figure(figsize=(12,3),dpi=100)
ax = fig.add_subplot(1,2,2)

ax.scatter(x1,x2)

ax.set_title('Optimum')
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.grid(True)
```

```python
def _objective(x1,x2):
    return (0.2*x1**(2) + (1-0.2)*x2**(2))**(1/2)

def objective(x):
    return _objective(x[0],x[1])

def ineq_constraint(x):
    return I-p1*x[0]-(p2+0.5*x[1])*x[1]

def eq_constraint(x):
        income = 30 - p1*x[0]-(p2+0.5*x[1])*x[1]
        return income
```

**Solving by SLSQP optimizer and iterating gradient evaluations:**

```python
bound = (1.0,5.0)
bounds = (bound, bound)
ineq_con = {'type': 'ineq', 'fun': ineq_constraint}
eq_con = {'type': 'eq', 'fun': eq_constraint}

x0=(1.0,1.0)
result = optimize.minimize(objective,x0,
                             method='SLSQP',
                             bounds=bounds,
                             constraints=[ineq_con,eq_con],
                             options={'disp':True})

print('\nx = ',result.x)
contour()
```

We now define the values for x1, x2 and u with the new budget constraint in order to plot the solution:

```python
def values(alpha,rho,I,p1,p2,N1,N2,do_print=True):

    shape_tuple = (N1,N2)
    x1_values = np.empty(shape_tuple)
    x2_values = np.empty(shape_tuple)
    u_values = np.empty(shape_tuple)

    x1_best = 0
    x2_best = 0
    u_best = u_func(0,0,alpha=alpha,rho=rho)

    for i in range(N1):
        for j in range(N2):

            x1_values[i,j] = x1 = (i/(N1-1))*I/p1
            x2_values[i,j] = x2 = (j/(N2-1))*I/p2

            #Budget constraint:
            if p1*x1+(p2+0.5*x2)*x2 <= I:
                u_values[i,j] = u_func(x1,x2,alpha=alpha,rho=rho)
            else:
                u_values[i,j] = u_func(0,0,alpha=alpha,rho=rho)

            if u_values[i,j] > u_best:
                x1_best = x1_values[i,j]
                x2_best = x2_values[i,j]
                u_best = u_values[i,j]

    if do_print:
        print_solution(x1_best,x2_best,u_best,I,p1,p2)

    return x1_best,x2_best,u_best,x1_values,x2_values,u_values

def print_solution(x1,x2,u,I,p1,p2):
    print(f'x1 = {x1:.8f}')
    print(f'x2 = {x2:.8f}')
    print(f'u  = {u:.8f}')
    print(f'I-p1*x1-(p2+0.5*x2)*x2 = {I-p1*x1-(p2+0.5*x2)*x2:.8f}')
```

```python
solution2 = values(alpha=0.20,rho=-0.2,I=30,p1=2,p2=4,N1=400,N2=600)
```

    x1 = 3.75939850
    x2 = 3.80634391
    u  = 3.79691751
    I-p1*x1-(p2+0.5*x2)*x2 = 0.01170041

**Plot the solution**

```python
fig = plt.figure(figsize=(12,4),dpi=100)

#Unpack solution:
x1_best,x2_best,u_best,x1_values,x2_values,u_values = solution2

# Left plot
ax_left = fig.add_subplot(1,2,1)
ax_left.plot(x1_values,u_values, 'ob')
ax_left.set_title('$u(x_1,x_2)$')
ax_left.set_xlabel('$x_1$')
ax_left.set_ylabel('$u(x_1,(I-p_1 x_1)/p_2)$')
ax_left.grid(True)

# Right plot
ax_right = fig.add_subplot(1,2,2)
ax_right.scatter(x1_best,x2_best)
ax_right.set_title('Optimum')
ax_right.set_xlabel('$x_1$')
ax_right.set_ylabel('$x_2$')
ax_right.grid(True)
```

From this we see that the plot has changed and the utillity is smaller than before introdusing the tax. The consumer now wish to consume almost as much of x1 as x2. The optimal amount of x1 is increased, but the utility curve is more flat than before.