# Almgren and Chriss Model For Optimal Execution of Portfolio Transactions

### Introduction

We consider the execution of portfolio transactions with the aim of minimizing a combination of risk and transaction costs arising from permanent and temporary market impact. As an example, assume that you have a certain number of stocks that you want to sell within a given time frame. If you place this sell order directly to the market as it is, transaction costs may rise due to temporary market impact. On the other hand, if you split up into pieces in time, cost may rise due to volatility in the stock price.

[Almgren and Chriss](https://cims.nyu.edu/~almgren/papers/optliq.pdf) provided a solution to this problem by assuming the permanent and temporary market impact functions are linear functions of the rate of trading, and that stock prices follow a discrete arithmetic random walk.

In this notebook, we will take a look at the model used by Almgren and Chriss to solve the optimal liquidation problem. We will start by stating the formal definitions of *trading trajectory*, *trading list*, and *trading strategy* for liquidating a single stock.

### Trading Trajectory, Trading List, and Trading Strategy

We define trading trajectory, trading list, and trading strategy just as Almgren and Chriss did in their [paper](https://cims.nyu.edu/~almgren/papers/optliq.pdf). Suppose we hold $X$ shares of a stock that we want to liquidate before time $T$. Divide $T$ into $N$ intervals of length $\tau=\frac{T}{N}$ and define:

- $t_k = k\tau$ to be discrete times, where $k = 0,..,N$.

- A **trading trajectory** to be the list $(x_0,..,x_N)$, where $x_k$ is the number of shares we plan to hold at time $t_k$. We require that our initial position $x_0 = X$, and that at liquidation time $T$, $x_N = 0$.

- A **trading list** to be $(n_1,..,n_N)$, $n_k =  x_{k-1} - x_k$ as the number of shares that we will sell between times $t_{k-1}$ and $t_k$.

- A **trading strategy** as a rule for determining $n_k$ from the information available at time $t_{k-1}$.

Below, we can see a visual example of a trading trajectory, for $N = 12$.

## Price Dynamics

We will assume that the stock price evolves according to a discrete arithmetic random walk:

\begin{equation}
S_k = S_{k-1} + \sigma \tau^{1/2} \xi_k
\end{equation}

for $k = 1,..,N$ and where:

\begin{equation}
S_k = \text{ stock price at time $k$}\hspace{21.6cm}\\
\sigma = \text{ standard deviation of the fluctuations in stock price}\hspace{16.3cm}\\
\tau = \text{ length of discrete time interval}\hspace{20.2cm}\\
\xi_k = \text{ draws from independent random variables}\hspace{17.8cm}
\end{equation}

We will denote the initial stock price as $S_0$. The role of $\xi_k$ is to simulate random price fluctuations using random numbers drawn from a Normal Gaussian distribution with zero mean and unit variance. The code below shows us what this price model looks like, for an initial stock price of $S_0 =$ \$50 dollars, a standard deviation of price fluctuations of $\sigma = 0.379$, and a discrete time interval of $\tau = 1$.

```python
import statsmodels.compat.python as statpy
```

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Set the number of days to follow the stock price
n_days = 100

# Plot the stock price as a function of time
utils.plot_price_model(seed = 0, num_days = n_days)
```

## Market Impact

As we learned previously the price of a stock is affected by market impact that occurs every time we sell a stock. In their model, Almgren and Chriss distinguish between two types of market impact, permanent and temporary market impact. We will now add these two factors into our price model.

### Permanent Impact

Permanent market impact refers to changes in the equilibrium price of a stock as a direct function of our trading. Permanent market impact is called *permanent* because its effect persists for the entire liquidation period, $T$. We will denote the permanent price impact as $g(v)$, and will add it to our price model:

\begin{equation}
S_k = S_{k-1} + \sigma \tau^{1/2} \xi_k - \tau g\left(\frac{n_k}{\tau}\right)
\end{equation}

Here, we assumed the permanent impact function, $g(v)$, is a linear function of the trading rate, $v = n_k / \tau$. We will take $g(v)$ to have the form:

\begin{equation}
g(v) = \gamma \left(\frac{n_k}{\tau}\right)
\end{equation}

where $\gamma$ is a constant and has units of (\$/share${}^2$). Replacing this in the above equation we get:

\begin{equation}
S_k = S_{k-1} + \sigma \tau^{1/2} \xi_k - \gamma n_k
\end{equation}

With this form, we can see that for each $n$ shares that we sell, we will depress the stock price permanently by $n\gamma$, regardless of the time we take to sell the stocks.

### Temporary Impact

Temporary market impact refers to temporary imbalances in supply and demand caused by our trading. This leads to temporary price movements away from equilibrium. Temporary market impact is called *temporary* because its effect
dissipates by the next trading period. We will denote the temporary price impact as $h(v)$. Given this, the actual stock price at time $k$ is given by:

\begin{equation}
\tilde{S_k} = S_{k-1} - h\left(\frac{n_k}{\tau}\right)
\end{equation}

Where, we have again assumed the temporary impact function, $h(v)$, is a linear function of the trading rate, $v = n_k / \tau$. We will take $h(v)$ to have the form:

\begin{equation}
h(v) = \epsilon \mbox{ sign}(n_k) + \eta \left(\frac{n_k}{\tau}\right)
\end{equation}

where $\epsilon$ and $\eta$ are constants with units (\$/share) and (\$ time/share${}^2$), respectively. It is important to note that $h(v)$ does not affect the price $S_k$.

## Capture

We define the **Capture** to be the total profits resulting from trading along a particular trading trajectory, upon completion of all trades. We can compute the capture via:

\begin{equation}
\sum\limits_{k=1}^{N} n_k \tilde{S_k} = X S_0 + \sum\limits_{k=1}^{N} \left(\sigma \tau^{1/2} \xi_k - \tau g\left(\frac{n_k}{\tau}\right)\right) x_k - \sum\limits_{k=1}^{N} n_k h\left(\frac{n_k}{\tau}\right)
\end{equation}

As we can see this is the sum of the product of the number of shares $n_k$ that we sell in each time interval, times the effective price per share $\tilde{S_k}$ received on that sale.

## Implementation Shortfall

We define the **Implementation Shortfall** as the total cost of trading and is given by:

\begin{equation}
I_s = X S_0 - \sum_{k = 1}^N n_k \tilde{S_k}
\end{equation}

This is what we seek to minimize when determining the best trading strategy!

Note that since $\xi_k$ is random, so is the implementation shortfall. Therefore, we have to frame the minimization problem in terms of the expectation value of the shortfall and its corresponding variance. We'll refer to $E(x)$ as the expected shortfall and $V(x)$ as the variance of the shortfall. Simplifying the above equation for $I_s$, is easy to see that:

\begin{equation}
E(x) = \sum\limits_{k=1}^{N} \tau x_k g\left(\frac{n_k}{\tau}\right) + \sum\limits_{k=1}^{N} n_k h\left(\frac{n_k}{\tau}\right)
\end{equation}

and

\begin{equation}
V(x) = \sigma^2 \sum\limits_{k=1}^{N} \tau {x_k}^2
\end{equation}

The units of $E(x)$ are dollars and the units of $V(x)$ are dollars squared. So now, we can reframe our minimization problem in terms of $E(x)$ and $V(x)$.
For a given level of variance of shortfall, $V(x)$, we seek to minimize the expectation of shortfall, $E(x)$. In the next section we will see how to solve this problem.

## Utility Function

Our goal now is to find the strategy that has the minimum expected shortfall $E(x)$ for a given maximum level of variance $V(x) \ge 0$. This constrained optimization problem can be solved by introducing a Lagrange multiplier $\lambda$. Therefore, our problem reduces to finding the trading strategy that minimizes the **Utility Function** $U(x)$:

\begin{equation}
U(x) = E(x) + \lambda V(x)
\end{equation}

The parameter $\lambda$ is referred to as **trader’s risk aversion** and controls how much we penalize the variance relative to the expected shortfall.

The intuition of this utility function can be thought of as follows. Consider a stock which exhibits high price volatility and thus a high risk of price movement away from the equilibrium price. A risk averse trader would prefer to trade a large portion of the volume immediately, causing a known price impact, rather than risk trading in small increments at successively adverse prices. Alternatively, if the price is expected to be stable over the liquidation period, the trader would rather split the trade into smaller sizes to avoid price impact. This trade-off between speed of execution and risk of price movement is ultimately what governs the structure of the resulting trade list.

# Optimal Trading Strategy

Almgren and Chriss solved the above problem and showed that for each value
of risk aversion there is a uniquely determined optimal execution strategy. The details of their derivation is discussed in their [paper](https://cims.nyu.edu/~almgren/papers/optliq.pdf). Here, we will just state the general solution.

The optimal trajectory is given by:

\begin{equation}
x_j = \frac{\sinh \left( \kappa \left( T-t_j\right)\right)}{ \sinh (\kappa T)}X, \hspace{1cm}\text{ for } j=0,...,N
\end{equation}

and the associated trading list:

\begin{equation}
n_j = \frac{2 \sinh \left(\frac{1}{2} \kappa \tau \right)}{ \sinh \left(\kappa T\right) } \cosh \left(\kappa \left(T - t_{j-\frac{1}{2}}\right)\right) X, \hspace{1cm}\text{ for } j=1,...,N
\end{equation}

where $t_{j-1/2} = (j-\frac{1}{2}) \tau$. The expected shortfall and variance of the optimal trading strategy are given by:

In the above equations $\kappa$ is given by:

\begin{align*}
&\kappa = \frac{1}{\tau}\cosh^{-1}\left(\frac{\tau^2}{2}\tilde{\kappa}^2 + 1\right)
\end{align*}

where:

\begin{align*}
&\tilde{\kappa}^2 = \frac{\lambda \sigma^2}{\tilde{\eta}} = \frac{\lambda \sigma^2}{\eta \left(1-\frac{\gamma \tau}{2 \eta}\right)}
\end{align*}

```python

```


# 원형 단면의 관성 모멘트
Area Moment of Inertia of a Circular Section

```python
# 그래프, 수학 기능 추가
# Add graph and math features
import pylab as py
import numpy as np
import numpy.linalg as nl
# 기호 연산 기능 추가
# Add symbolic operation capability
import sympy as sy

```

참고문헌 :

* Pytel 외 저, 이주성 외 역, 재료역학, 2판, 한티미디어, 2013.

* 위키백과 기여자, '단면 이차 모멘트', 위키백과, , 2018년 3월 4일, 13:19 UTC, <https://ko.wikipedia.org/wiki/%EB%8B%A8%EB%A9%B4_%EC%9D%B4%EC%B0%A8_%EB%AA%A8%EB%A9%98%ED%8A%B8> [2018년 7월 31일에 접근]

Ref:

* Pytel, Kiusalaas, Sharma, Mechanics of Materials, 2nd Ed., Cengage Learning, 2013.

* Wikipedia contributors, 'Second moment of area', Wikipedia, The Free Encyclopedia, 16 June 2018, 14:15 UTC, <https://en.wikipedia.org/w/index.php?title=Second_moment_of_area&oldid=846126944> [accessed 31 July 2018]

다음과 같은 원형 단면의 2차 모멘트를 구해 보자.

Let's try to find the second moment of area of following circular section.

반지름 $r=10mm$
Radius of the section $r=10mm$

```python
r_mm = 10

```

## 단면 2차 모멘트의 정의
Definition of a Second Moment of Area

$$
I_x=\int_A y^2 dA
$$

여기서 $dA$는 다음과 같다. (원점은 원의 중심에 위치)

Here, $dA$ is as follows. (The origin is at the center of the circle)

$$
dA=S_x(y)dy=2\sqrt{r^2-y^2}dy
$$

Python 언어로는 다음과 같이 구현할 수 있다.
We can implement in python as follows.

```python
def sx(y_mm):
    if abs(y_mm) <= r_mm :
        result = 2 * (r_mm * r_mm - y_mm * y_mm) ** 0.5
    else:
        result = 0

    return result

```

이 함수의 그래프를 그려 보자
Let's plot this.

```python
y_mm_array = py.arange(-r_mm, r_mm+0.05, 0.1)
sx_mm_array = py.array([sx(y_mm) for y_mm in y_mm_array])

py.plot(sx_mm_array * 0.5, y_mm_array)
py.plot(sx_mm_array * (-0.5), y_mm_array)

py.axis('equal')
py.grid(True)

py.xlabel('x(mm)')
py.ylabel('y(mm)')

```

## 정적분 계산
Numerical Integration

0차 적분 함수를 이용해 보자
Let's use 0'th order numerical integration function.

```python
def get_delta_x(xi, xe, n):
    return (xe - xi) / n

```

```python
def num_int_0(f, xi, xe, n, b_verbose=False):
    x_array = py.linspace(xi, xe, n+1)

    delta_x = x_array[1] - x_array[0]

    assert 1e-3 > (abs(delta_x - get_delta_x(xi, xe, n)) / get_delta_x(xi, xe, n)), f"delta_x = {delta_x}"

    integration_result = 0.0

    for k in range(n):
        x_k = x_array[k]
        F_k = f(x_k) * delta_x

        if b_verbose:
            print('k = %2d, F_k = %g' % (k, F_k))

        integration_result += F_k

    return integration_result

```

### 단면 2차 모멘트
Second moment of area

```python
def y2sx(y_mm):
    return y_mm * y_mm * sx(y_mm)

```

```python
I_y_mm4 = num_int_0(y2sx, -r_mm, r_mm, int(r_mm * 10))

```

```python
I_y_mm4

```

확인해 보자.
Let's verify.

```python
I_y_exact_mm4 = np.pi * (r_mm ** 4) * 0.25

```

```python
I_y_exact_mm4

```

```python
abs(I_y_exact_mm4 - I_y_mm4)

```

어떻게 하면 위 오차를 줄일 수 있을 것인가?
How can we make the error above smaller?

```python
error = (I_y_exact_mm4 - I_y_mm4)

```

```python
try :
    assert (1e-6 > abs(error)), "Error too large"
except AssertionError as e:
    print(e)

```

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```


```python
%pylab inline
pylab.rcParams['figure.figsize'] = (16.0, 8.0)
```

    Populating the interactive namespace from numpy and matplotlib

# Summarizing the Monte Carlo output

The result of a Monte Carlo simulation is a set of samples from the probability distribution associated with the measurand
$$ \{\mathbf{y}^{(k)},\,k=1,\ldots,M\} $$

The aim is to derive characteristic information about the measurand from this set:

1) best estimate

2) uncertainty associated with the best estimate

3) intervals/regions with a prescribed coverage probability

## Univariate measurand

1) best estimate
\begin{align}
y =& mean \{ y^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M} \sum_{k=1}^M y^{(k)}
\end{align}
2) uncertainty associated with the best estimate
\begin{align}
u_y =& std\{ y^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M-1} \sum_{k=1}^M (y-y^{(k)})^2
\end{align}

### Exercise 4.1

Draw randomly from the normal distribution with mean $\mu=1.3$ and standard deviation $\sigma=0.4$ and calculate best estimate and its uncertainy using 100, 200, 300, ..., 100 000 draws. Compare estimate and uncertainty with mean and standard deviation of the original distribution for the different number of draws.

```python
from scipy.stats import norm

```

Intervals with a prescribed coverage probability can be calculated from the Monte Carlo outcome as follows

1) Sort the Monte Carlo outcome in ascending order
``` python
    sort(Y)
```
2) For propability $P$ calculate the corresponding fraction of Monte Carlo draws $q=int(PM)$

3) Set lower bound of interval as $r=int(0.5(M-q))$ for a probabilistically symmetrical interval

4) Calculate upper bound of interval as $r+q$

### Exercise 4.2

Draw randomly from the distribution from Exercise 4.1 and calculate the 95% probabilistally symetrical coverage interval from 1000, 10000 and 100000 draws. Compare the result to the exact 95% interval.

```python
from scipy.stats import norm

P = 0.95 # sought probability of coverage interval

```

## Multivariate measurand

1) best estimate
\begin{align}
\mathbf{y} =& mean \{ \mathbf{y}^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M} \sum_{k=1}^M \mathbf{y}^{(k)}
\end{align}
2) uncertainty associated with the best estimate
\begin{align}
U_\mathbf{y} =& cov\{ \mathbf{y}^{(k)}, k=1,\ldots,M\} \\
=& \frac{1}{M-1} \sum_{k=1}^M (\mathbf{y}-\mathbf{y}^{(k)})(\mathbf{y}-\mathbf{y}^{(k)})^T
\end{align}

### Exercise 4.3

Draw randomly from the normal distribution with mean
$$\mathbf{\mu}=\left( \begin{array}{c}
0.4 \\ -1.5
\end{array}\right)
$$
and covariance
$$
\Sigma=\left(\begin{array}{cc}
0.09 & -0.2 \\ -0.2 & 1.44
\end{array}\right)
$$
and calculate best estimate and its uncertainy using 1000, 10000 and 100 000 draws. Compare estimate and uncertainty with mean and covariance of the original distribution for the different number of draws.

```python
from scipy.stats import multivariate_normal

```

Regions with a prescribed coverage probability can be calculated from the multivariate Monte Carlo outcome as follows

1) Calculate the Cholesky decomposition of the sample covariance matrix $U_{\mathbf{y}}=\mathbf{LL}^T$

2) Transform the Monte Carlo outcomes
$$ \mathbf{y}_{(k)} = \mathbf{L}^{-1}(\mathbf{y}^{(k)}-\mathbf{y})$$
and sort according to the distance measure
$$ d^2_{(k)} = \mathbf{y}_{(k)}^T\mathbf{y}_{(k)} $$

3) calculate $k_P$ such that a fraction $P$ of all Monte Carlo outcomes satisfies $d_{(k)}<k_P$

This defines the ellipsoidal region $(\mathbf{\eta}-\mathbf{y})^TU_{\mathbf{y}}^{-1}(\mathbf{\eta}-\mathbf{y})<k^2_P$

For a bivariate normal distribution, the factor for a 95% coverage ellipsoidal region is given as the 95% quantile of the $\chi^2$ distribution with 2 degrees of freedom.

### Exercise 4.4

Calculate 100 000 random draws from the distribution from Exercise 4.3 and calculate the 95% coverage region. Compare to the true 95% coverage region.

```python
def calculate_ellipse(mu, Sigma, kP):
    vals, vecs = linalg.eigh(Sigma)
    order = vals.argsort()[::-1]
    vals = vals[order]
    vecs = vecs[:,order]
    theta = degrees(np.arctan2(*vecs[:,0][::-1]))
    width, height = kP * sqrt(vals)
    return width, height, theta

```

```python
from scipy.stats import multivariate_normal, chi2
from matplotlib.patches import Ellipse

mu = array([0.4, -1.5])
Sigma = array([[0.09, -0.2],[-0.2, 1.44]])

```

.ipynb_checkpoints/05 Summarizing the Monte Carlo output-checkpoint.ipynb


#AG2 - Actividad Guiada 2
Nombre: Daniel Bayerri

Link:   https://colab.research.google.com/drive/1DYnWOlMjFibZoTwrllCtyPp9nvf_R6dx?usp=sharing

Github: https://github.com/xxxxx/AlgoritmosOptimizacion

```
import math
```

##Programación Dinámica. Viaje por el rio
* **Definición**: Es posible dividir el problema en subproblemas más pequeños, guardando las soluciones para ser utilizadas más adelante.
* **Características** que permiten identificar problemas aplicables:

  -Es posible almacenar soluciones de los subproblemas para ser utilizados más adelante

  -Debe verificar el principio de optimalidad de Bellman: “en una secuencia optima de decisiones, toda sub-secuencia también es óptima” (*)

  -La necesidad de guardar la información acerca de las soluciones parciales unido a la recursividad provoca la necesidad de preocuparnos por la complejidad espacial (cuantos recursos de espacio usaremos)

###Problema

En un río hay **n** embarcaderos y debemos desplazarnos río abajo desde un embarcadero a otro. Cada embarcadero tiene precios diferentes para ir de un embarcadero a otro situado más abajo. Para ir del embarcadero i al j, puede ocurrir que sea más barato hacer un trasbordo por un embarcadero intermedio k. El problema consiste en determinar la combinación más barata.

*Consideramos una tabla TARIFAS(i,j) para almacenar todos los precios que nos ofrecen los embarcaderos.

*Si no es posible ir desde i a j daremos un valor alto para garantizar que ese trayecto no se va a elegir en la ruta óptima(modelado habitual para restricciones)

```
#Viaje por el rio - Programación dinámica
################################################################################

TARIFAS = [
[0,5,4,3,float("inf"),999,999],   #desde nodo 0
[999,0,999,2,3,999,11],  #desde nodo 1
[999,999, 0,1,999,4,10], #desde nodo 2
[999,999,999, 0,5,6,9],
[999,999, 999,999,0,999,4],
[999,999, 999,999,999,0,3],
[999,999,999,999,999,999,0]
]

#999 se puede sustituir por float("inf") del modulo math
TARIFAS
```

    [[0, 5, 4, 3, inf, 999, 999],
     [999, 0, 999, 2, 3, 999, 11],
     [999, 999, 0, 1, 999, 4, 10],
     [999, 999, 999, 0, 5, 6, 9],
     [999, 999, 999, 999, 0, 999, 4],
     [999, 999, 999, 999, 999, 0, 3],
     [999, 999, 999, 999, 999, 999, 0]]

```
#Calculo de la matriz de PRECIOS y RUTAS
#  PRECIOS - contiene la matriz del mejor precio para ir de un nodo a otro
#  RUTAS   - contiene los nodos intermedios para ir de un nodo a otro
################################################################
def Precios(TARIFAS):
################################################################
  #Total de Nodos
  N = len(TARIFAS[0])

  #Inicialización de la tabla de precios
  PRECIOS = [ [9999]*N for i in [9999]*N]  #n x n
  RUTA = [ [""]*N for i in [""]*N]

  #Se recorren todos los nodos con dos bucles(origen - destino)
  #  para ir construyendo la matriz de PRECIOS
  for i in range(N-1):
    for j in range(i+1, N):
      MIN = TARIFAS[i][j]
      RUTA[i][j] = i

      for k in range(i, j):
        if PRECIOS[i][k] + TARIFAS[k][j] < MIN:
            MIN = min(MIN, PRECIOS[i][k] + TARIFAS[k][j] )
            RUTA[i][j] = k
        PRECIOS[i][j] = MIN

  return PRECIOS,RUTA
```

```
PRECIOS,RUTA = Precios(TARIFAS)
#print(PRECIOS[0][6])

print("PRECIOS")
for i in range(len(TARIFAS)):
  print(PRECIOS[i])

print("\nRUTA")
for i in range(len(TARIFAS)):
  print(RUTA[i])
```

    PRECIOS
    [9999, 5, 4, 3, 8, 8, 11]
    [9999, 9999, 999, 2, 3, 8, 7]
    [9999, 9999, 9999, 1, 6, 4, 7]
    [9999, 9999, 9999, 9999, 5, 6, 9]
    [9999, 9999, 9999, 9999, 9999, 999, 4]
    [9999, 9999, 9999, 9999, 9999, 9999, 3]
    [9999, 9999, 9999, 9999, 9999, 9999, 9999]

    RUTA
    ['', 0, 0, 0, 1, 2, 5]
    ['', '', 1, 1, 1, 3, 4]
    ['', '', '', 2, 3, 2, 5]
    ['', '', '', '', 3, 3, 3]
    ['', '', '', '', '', 4, 4]
    ['', '', '', '', '', '', 5]
    ['', '', '', '', '', '', '']

```
#Calculo de la ruta usando la matriz RUTA
def calcular_ruta(RUTA, desde, hasta):
  if desde == RUTA[desde][hasta]:
  #if desde == hasta:
    #print("Ir a :" + str(desde))
    return desde
  else:
    return (str(calcular_ruta(RUTA, desde, RUTA[desde][hasta])) +  ',' + str(RUTA[desde][hasta]))

print("\nLa ruta es:")
calcular_ruta(RUTA, 0,6)
```

    La ruta es:

    '0,2,5'

##Problema de Asignacion de tarea

```
#Asignacion de tareas - Ramificación y Poda
################################################################################
#    T A R E A
#   A
#   G
#   E
#   N
#   T
#   E

COSTES=[[11,12,18,40],
        [14,15,13,22],
        [11,17,19,23],
        [17,14,20,28]]

```

```
#Calculo del valor de una solucion parcial
def valor(S,COSTES):
  VALOR = 0
  for i in range(len(S)):
    VALOR += COSTES[S[i]][i]
  return VALOR

valor((0, 1, 2, 3  ),COSTES)
```

    73

```
#Coste inferior para soluciones parciales
#  (1,3,) Se asigna la tarea 1 al agente 0 y la tarea 3 al agente 1

def CI(S,COSTES):
  VALOR = 0
  #Valores establecidos
  for i in range(len(S)):
    VALOR += COSTES[i][S[i]]

  #Estimacion
  for i in range( len(S), len(COSTES)   ):
    VALOR += min( [ COSTES[j][i] for j in range(len(S), len(COSTES))  ])
  return VALOR

def CS(S,COSTES):
  VALOR = 0
  #Valores establecidos
  for i in range(len(S)):
    VALOR += COSTES[i][S[i]]

  #Estimacion
  for i in range( len(S), len(COSTES)   ):
    VALOR += max( [ COSTES[j][i] for j in range(len(S), len(COSTES))  ])
  return VALOR

CI((0,1),COSTES)
```

    68

```
#Genera tantos hijos como como posibilidades haya para la siguiente elemento de la tupla
#(0,) -> (0,1), (0,2), (0,3)
def crear_hijos(NODO, N):
  HIJOS = []
  for i in range(N ):
    if i not in NODO:
      HIJOS.append({'s':NODO +(i,)    })
  return HIJOS
```

```
crear_hijos((0,) , 4)
```

    [{'s': (0, 1)}, {'s': (0, 2)}, {'s': (0, 3)}]

```
def ramificacion_y_poda(COSTES):
#Construccion iterativa de soluciones(arbol). En cada etapa asignamos un agente(ramas).
#Nodos del grafo  { s:(1,2),CI:3,CS:5  }
  #print(COSTES)
  DIMENSION = len(COSTES)
  MEJOR_SOLUCION=tuple( i for i in range(len(COSTES)) )
  CotaSup = valor(MEJOR_SOLUCION,COSTES)
  #print("Cota Superior:", CotaSup)

  NODOS=[]
  NODOS.append({'s':(), 'ci':CI((),COSTES)    } )

  iteracion = 0

  while( len(NODOS) > 0):
    iteracion +=1

    nodo_prometedor = [ min(NODOS, key=lambda x:x['ci']) ][0]['s']
    #print("Nodo prometedor:", nodo_prometedor)

    #Ramificacion
    #Se generan los hijos
    HIJOS =[ {'s':x['s'], 'ci':CI(x['s'], COSTES)   } for x in crear_hijos(nodo_prometedor, DIMENSION) ]

    #Revisamos la cota superior y nos quedamos con la mejor solucion si llegamos a una solucion final
    NODO_FINAL = [x for x in HIJOS if len(x['s']) == DIMENSION  ]
    if len(NODO_FINAL ) >0:
      #print("\n********Soluciones:",  [x for x in HIJOS if len(x['s']) == DIMENSION  ] )
      if NODO_FINAL[0]['ci'] < CotaSup:
        CotaSup = NODO_FINAL[0]['ci']
        MEJOR_SOLUCION = NODO_FINAL

    #Poda
    HIJOS = [x for x in HIJOS if x['ci'] < CotaSup   ]

    #Añadimos los hijos
    NODOS.extend(HIJOS)

    #Eliminamos el nodo ramificado
    NODOS =  [  x for x in NODOS if x['s'] != nodo_prometedor    ]

  print("La solucion final es:" ,MEJOR_SOLUCION , " en " , iteracion , " iteraciones" , " para dimension: " ,DIMENSION  )

ramificacion_y_poda(COSTES)
```

    La solucion final es: [{'s': (1, 2, 0, 3), 'ci': 64}]  en  10  iteraciones  para dimension:  4

##Descenso del gradiente

```
import math                      #Funciones matematicas
import matplotlib.pyplot as plt  #Generacion de gráficos (otra opcion seaborn)
import numpy as np               #Tratamiento matriz N-dimensionales y otras (fundamental!)
#import scipy as sc

import random
```

Vamos a buscar el minimo de la funcion paraboloide : $$f(x) =  x² + y²$$

Obviamente se encuentra en (x,y)=(0,0) pero probaremos como llegamos a él a través del descenso del gradiante.

```
#Definimos la funcion
#Paraboloide
f  = lambda X:      X[0]**2 + X[1]**2    #Funcion
df = lambda X: [2*X[0] , 2*X[1]]         #Gradiente

df([1,2])
```

    [2, 4]

```
from sympy import symbols
from sympy.plotting import plot
from sympy.plotting import plot3d
x,y = symbols('x y')
plot3d(x**2 + y**2,
       (x,-5,5),(y,-5,5),
       title='x**2 + y**2',
       size=(10,10))
```

```
#Prepara los datos para dibujar mapa de niveles de Z
resolucion = 100
rango=2.5

X=np.linspace(-rango,rango,resolucion)
Y=np.linspace(-rango,rango,resolucion)
Z=np.zeros((resolucion,resolucion))
for ix,x in enumerate(X):
  for iy,y in enumerate(Y):
    Z[iy,ix] = f([x,y])

#Pinta el mapa de niveles de Z
plt.contourf(X,Y,Z,resolucion)
plt.colorbar()

#Generamos un punto aleatorio inicial y pintamos de blanco
P=[random.uniform(-2,2  ),random.uniform(-2,2 ) ]
plt.plot(P[0],P[1],"o",c="white")

#Tasa de aprendizaje. Fija. Sería más efectivo reducirlo a medida que nos acercamos.
TA=.1

#Iteraciones:500
for _ in range(50):
  grad = df(P)
  #print(P,grad)
  P[0],P[1] = P[0] - TA*grad[0] , P[1] - TA*grad[1]
  plt.plot(P[0],P[1],"o",c="red")

#Dibujamos el punto final y pintamos de verde
plt.plot(P[0],P[1],"o",c="green")
plt.show()
print("Solucion:" , P , f(P))

```

**¿Te atreves a optimizar la función?:**

$$f(x)=sin(1/2 * x^2 - 1/4 * y^2 + 3) * cos(2*x + 1 - e^y)$$

```
#Definimos la funcion
f= lambda X: math.sin(1/2 * X[0]**2 - 1/4 * X[1]**2 + 3) *math.cos(2*X[0] + 1 - math.exp(X[1]) )

```


## Linear Regression

linear regression is a linear approach for **modelling** the relationship between a scalar **dependent variable y** and one or more **independent variables denoted X**. Using exsting data to estimate unknown data.

Linear regression model assumes that the relationship between the dependent variable **yi** and the p-vector of regressors **xi** is linear.

$$y_{i} = \beta _{0}1 + \beta _{1}x_{i1} + ... + \beta _{p}x_{ip} + \varepsilon _{i} = x_{i}^{T}\beta + \varepsilon _{i}$$

where $^{T}$ denotes the transpose, so that $x_{i}^{T}$ is the inner product between vectors $x_{i}$ and C. The equation can stacked together and written in vector form as:
$$ y = X\beta + \varepsilon $$
where,

$ y=\begin{pmatrix} y_{1}\\ y_{2}\\ ...\\ y_{n} \end{pmatrix} $ ,
$X=\begin{pmatrix} x_{1}^{T}\\  x_{2}^{T}\\ ...\\ x_{n}^{T}\end{pmatrix}
=\begin{pmatrix}
 1 &x _{11} & ... &x _{1p} \\
 1& x _{21} & ... & x _{2p}\\
 ...& ... & ... & ...\\
 1&  x _{n1}& ... & x _{np}
\end{pmatrix}$ ,

$\beta  = \begin{pmatrix} \beta _{0}\\ \beta _{1}\\ \beta _{2}\\ ...\\ \beta _{p} \end{pmatrix}$ , and
$\varepsilon = \begin{pmatrix} \varepsilon _{0}\\ \varepsilon _{1}\\ ...\\ \varepsilon _{n} \end{pmatrix}$

- $y_{i}$ is called dependent variable.
- $x_{i}$ is called independent variable.
- $\beta$ is called intercept, a (p+1)-dimension **parameter vector**, where $\beta _{0}$ is the constant (offset) term.  Its elements are also called effects, and the estimates of it are called "estimated effects" or regression coefficients.
- $\varepsilon _{i}$ is called the _error term_ or _disturbance term_.which is diffence between estimate value $y_{i}$ and real value $x_{i}\beta$,  $$\varepsilon _{i} = y_{i} - x_{i}\beta$$
  1. they are independently
  2. they are normal distribution

#### Probability density function:
$$p(\varepsilon _{i}) = \frac{1}{\sqrt{2\pi \sigma  ^{2}}} exp(-\frac{\varepsilon _{i}^{2}}{2\sigma  ^{2}})$$
$$\Rightarrow p(y_{i}|x_{i},\beta) = \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$

#### Likelihood estimation：using data to esitimate function paramters to let argmin($\varepsilon$)
$$Likelihood(\beta ) = \prod_{i=1}^{m}p(y_{i}|x_{i},\beta) =\prod_{i=1}^{m} \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$

#### Logarithm likelihood estimation
$$log\,  L(\beta ) = log\, \prod_{i=1}^{m} \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$
$$=\sum_{i=1}^{m} log\, \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$
$$= m\, log\, \frac{1}{\sqrt{2\pi \sigma^{2}}} - \frac{1}{\sigma^{2}}\cdot \frac{1}{2}\sum_{i=1}^{m}(y_{i}-x_{i}\beta)^{2}$$

#### Least squares
$$J(\beta ) = \frac{1}{2}\sum_{i=1}^{m}(y_{i}-x_{i}\beta)^{2}$$

#### objective function

$$
\begin{equation}
\begin{split}
 J(\beta )&= \frac{1}{2}\sum_{i=1}^{m}(x_{i}\beta-y_{i})^{2}   \\
          &= \frac{1}{2}(X\beta -y)^{T}(X\beta -y) \\
          &= \frac{1}{2}(\beta ^{T}X ^{T} - y ^{T})(X\beta -y) \\
          &= \frac{1}{2}(\beta ^{T}X ^{T}X\beta -\beta ^{T}X ^{T}y- y ^{T}X\beta+y ^{T}y) \\
\end{split}
\end{equation}
$$

#### partial derivatives $\triangledown_{\beta} J(\beta)$
$$
\begin{equation}
\begin{split}
 \triangledown_{\beta } J(\beta ) &=\triangledown_{\beta } (\frac{1}{2}(\beta ^{T}X ^{T}X\beta -\beta ^{T}X ^{T}y- y ^{T}X\beta+y ^{T}y)) \\
&=  \frac{1}{2}(2X ^{T}X\beta - X ^{T}y - (y^{T}X)^{T}) \\
&= X^{T}X\beta - X^{T}y
\end{split}
\end{equation}
$$

when $\triangledown_{\beta} J(\beta) = 0$:
$$\beta = (X^{T}X)^{-1}\,X^{T}y$$

#### Evaluation:
In statistics, the coefficient of determination, denoted R2 or r2 and pronounced "R squared", is the proportion of the variance in the dependent variable that is predictable from the independent variable(s)

$$R^{2} : 1 - \frac{\sum_{i=1}^{m}(\hat{y_{i}}-y_{i})^{2}}{\sum_{i=1}^{m}(y_{i} - \overline{y})^{2}}$$

where,

$\hat{y_{i}}$ is estimated y

$\overline{y}$ is mean y

```python

```

Yudi TANG/Linear Regression/Linear Regression.ipynb


Least Squares Support Vector Classifier

## Summary:
1. [Introduction](#introduction)

2. [Using the classifier](#using_classifier)

    2.1 [CPU/Numpy version](#cpu_version)

    2.2 [GPU/PyTorch version](#gpu_version)

# 1. Introduction

The Least Squares Support Vector Machine (LSSVM) is a variation of the original Support Vector Machine (SVM) in which we have a slight change in the objective and restriction functions that results in a big simplification of the optimization problem.

First, let's see the optimization problem of an SVM:

$$
\begin{align}
    minimize && f_o(\vec{w},\vec{\xi})=\frac{1}{2} \vec{w}^T\vec{w} + C \sum_{i=1}^{n} \xi_i &&\\
    s.t. && y_i(\vec{w}^T\vec{x}_i+b)\geq 1 - \xi_i, && i = 1,..., n \\
         && \xi_i \geq 0,                            && i = 1,..., n
\end{align}
$$

In this case, we have a set of inequality restrictions and when solving the optimization problem by it's dual we find a discriminative function, adding the kernel trick, of the type:

$$ f(\vec{x}) = sign \ \Big( \sum_{i=1}^{n} \alpha_i^o y_i K(\vec{x}_i,\vec{x}) + b_o \Big) $$

Where $\alpha_i^o$ and $b_o$ denote optimum values. Giving enough regularization (smaller values of $C$) we get a lot of $\alpha_i^o$ nulls, resulting in a sparse model in which we only need to save the pairs $(\vec{x}_i,y_i)$ which have the optimum dual variable not null. The vectors $\vec{x}_i$ with not null $\alpha_i^o$ are known as support vectors (SV).

In the LSSVM case, we change the inequality restrictions to equality restrictions. As the $\xi_i$ may be negative we square its values in the objective function:

$$
\begin{align}
    minimize && f_o(\vec{w},\vec{\xi})=\frac{1}{2} \vec{w}^T\vec{w} + \gamma \frac{1}{2}\sum_{i=1}^{n} \xi_i^2 &&\\
    s.t. && y_i(\vec{w}^T\vec{x}_i+b) = 1 - \xi_i, && i = 1,..., n
\end{align}
$$

The dual of this optimization problem results in a system of linear equations, a set of Karush-Khun-Tucker (KKT) equations:

$$
\begin{bmatrix}
    0 & \vec{d}^T \\
    \vec{y} & \Omega + \gamma^{-1} I
\end{bmatrix}
\
\begin{bmatrix}
    b  \\
    \vec{\alpha}
\end{bmatrix}
=
\begin{bmatrix}
    0 \\
    \vec{1}
\end{bmatrix}
$$

Where, with the kernel trick,   $\Omega_{i,j} = y_i y_j K(\vec{x}_i,\vec{x}_j)$,     $\vec{y} = [y_1 \ y_2 \ ... \ y_n]^T$,   $\vec{\alpha} = [\alpha_1 \ \alpha_2 \ ... \ \alpha_n]^T$    e   $\vec{1} = [1 \ 1 \ ... \ 1]^T$.

The discriminative function of the LSSVM has the same form of the SVM but the $\alpha_i^o$ aren't usually null, resulting in a bigger model. The big advantage of the LSSVM is in finding it's parameters, which is reduced to solving the linear system of the type:

$$ A\vec{z} = \vec{b} $$

A well-known solution of the linear system is when we minimize the square of the residues, that can be written as the optimization problem:

$$
\begin{align}
    minimize && f_o(\vec{z})=\frac{1}{2}||A\vec{z} - \vec{b}||^2\\
\end{align}
$$

And have the analytical solution:

$$ \vec{z} = A^{\dagger} \vec{b} $$

Where $A^{\dagger}$ is the pseudo-inverse defined as:

$$ A^{\dagger} = (A^T A)^{-1} A^T$$

# 2. Using the classifier

```python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_digits
from sklearn.preprocessing import MinMaxScaler

from lssvm import LSSVC, LSSVC_GPU
from utils.encoding import dummie2multilabel
```

```python
# Preprocessing

# Import digits recognition dataset (from sklearn)
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=2020)

# Scaling features (from sklearn)
scaler = MinMaxScaler()
scaler.fit(X_train)
X_tr_norm = scaler.transform(X_train)
X_ts_norm = scaler.transform(X_test)

# Get information about input and outputs
print(f"X_train.shape: {X_train.shape}")
print(f"X_test.shape:  {X_test.shape}")
print(f"y_train.shape: {y_train.shape}")
print(f"y_test.shape:  {y_test.shape}")
print(f"np.unique(y_train): {np.unique(y_train)}")
print(f"np.unique(y_test):  {np.unique(y_test)}")
```

    X_train.shape: (1078, 64)
    X_test.shape:  (719, 64)
    y_train.shape: (1078,)
    y_test.shape:  (719,)
    np.unique(y_train): [0 1 2 3 4 5 6 7 8 9]
    np.unique(y_test):  [0 1 2 3 4 5 6 7 8 9]

## 2.1 CPU/Numpy version

```python
# Use the classifier with different kernels

print('Gaussian kernel:')
lssvc = LSSVC(gamma=1, kernel='rbf', sigma=.5) # Class instantiation
lssvc.fit(X_tr_norm, y_train) # Fitting the model
y_pred = lssvc.predict(X_ts_norm) # Making predictions with the trained model
acc = accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)) # Calculate Accuracy
print('acc_test = ', acc, '\n')

print('Polynomial kernel:')
lssvc = LSSVC(gamma=1, kernel='poly', d=2)
lssvc.fit(X_tr_norm, y_train)
y_pred = lssvc.predict(X_ts_norm)
acc = accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred))
print('acc_test = ', acc, '\n')

print('Linear kernel:')
lssvc = LSSVC(gamma=1, kernel='linear')
lssvc.fit(X_tr_norm, y_train)
y_pred = lssvc.predict(X_ts_norm)
acc = accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred))
print('acc_test = ', acc, '\n')
```

    Gaussian kernel:
    acc_test =  0.9680111265646731

    Polynomial kernel:
    acc_test =  0.9944367176634215

    Linear kernel:
    acc_test =  0.9791376912378303

If there are any questions about a specific method, the user can ask for help:

```python
help(LSSVC.fit)
```

    Help on function fit in module lssvm.LSSVC:

    fit(self, X, y)
        Fits the model given the set of X attribute vectors and y labels.
        - X: ndarray of shape (n_samples, n_attributes)
        - y: ndarray of shape (n_samples,) or (n_samples, n)
            If the label is represented by an array of n elements, the y
            parameter must have n columns.

The user can also have an overview of the class:

```python
help(LSSVC)
```

    Help on class LSSVC in module lssvm.LSSVC:

    class LSSVC(builtins.object)
     |  LSSVC(gamma=1, kernel='rbf', **kernel_params)
     |
     |  A class that implements the Least Squares Support Vector Machine
     |  for classification tasks.
     |
     |  It uses Numpy pseudo-inverse function to solve the dual optimization
     |  problem with ordinary least squares. In multiclass classification
     |  problems the approach used is one-vs-all, so, a model is fit for each
     |  class while considering the others a single set of the same class.
     |
     |  # Parameters:
     |  - gamma: float, default = 1.0
     |      Constant that control the regularization of the model, it may vary
     |      in the set (0, +infinity). The closer gamma is to zero, the more
     |      regularized the model will be.
     |  - kernel: {'linear', 'poly', 'rbf'}, default = 'rbf'
     |      The kernel used for the model, if set to 'linear' the model
     |      will not take advantage of the kernel trick, and the LSSVC maybe only
     |      useful for linearly separable problems.
     |  - kernel_params: **kwargs, default = depends on 'kernel' choice
     |      If kernel = 'linear', these parameters are ignored. If kernel = 'poly',
     |      'd' is accepted to set the degree of the polynomial, with default = 3.
     |      If kernel = 'rbf', 'sigma' is accepted to set the radius of the
     |      gaussian function, with default = 1.
     |
     |  # Attributes:
     |  - All hyperparameters of section "Parameters".
     |  - alpha: ndarray of shape (1, n_support_vectors) if in binary
     |           classification and (n_classes, n_support_vectors) for
     |           multiclass problems
     |      Each column is the optimum value of the dual variable for each model
     |      (using the one-vs-all approach we have n_classes == n_classifiers),
     |      it can be seen as the weight given to the support vectors
     |      (sv_x, sv_y). As usually there is no alpha == 0, we have
     |      n_support_vectors == n_train_samples.
     |  - b: ndarray of shape (1,) if in binary classification and (n_classes,)
     |       for multiclass problems
     |      The optimum value of the bias of the model.
     |  - sv_x: ndarray of shape (n_support_vectors, n_features)
     |      The set of the supporting vectors attributes, it has the shape
     |      of the training data.
     |  - sv_y: ndarray of shape (n_support_vectors, n)
     |      The set of the supporting vectors labels. If the label is represented
     |      by an array of n elements, the sv_y attribute will have n columns.
     |  - y_labels: ndarray of shape (n_classes, n)
     |      The set of unique labels. If the label is represented by an array
     |      of n elements, the y_label attribute will have n columns.
     |  - K: function, default = rbf()
     |      Kernel function.
     |
     |  Methods defined here:
     |
     |  __init__(self, gamma=1, kernel='rbf', **kernel_params)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  dump(self, filepath='model', only_hyperparams=False)
     |      This method saves the model in a JSON format.
     |      - filepath: string, default = 'model'
     |          File path to save the model's json.
     |      - only_hyperparams: boolean, default = False
     |          To either save only the model's hyperparameters or not, it
     |          only affects trained/fitted models.
     |
     |  fit(self, X, y)
     |      Fits the model given the set of X attribute vectors and y labels.
     |      - X: ndarray of shape (n_samples, n_attributes)
     |      - y: ndarray of shape (n_samples,) or (n_samples, n)
     |          If the label is represented by an array of n elements, the y
     |          parameter must have n columns.
     |
     |  predict(self, X)
     |      Predicts the labels of data X given a trained model.
     |      - X: ndarray of shape (n_samples, n_attributes)
     |
     |  ----------------------------------------------------------------------
     |  Class methods defined here:
     |
     |  load(filepath, only_hyperparams=False) from builtins.type
     |      This class method loads a model from a .json file.
     |      - filepath: string
     |          The model's .json file path.
     |      - only_hyperparams: boolean, default = False
     |          To either load only the model's hyperparameters or not, it
     |          only has effects when the dump of the model as done with the
     |          model's parameters.
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)

The user may also save the model in JSON format:

```python
lssvc.dump('model')
loaded_model = LSSVC.load('model')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(lssvc.predict(X_ts_norm))
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(loaded_model.predict(X_ts_norm))
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303

## 2.2 GPU/PyTorch version

It has the same functionalities and syntax of the CPU version, the difference is the use of PyTorch to run the operations on the GPU.

```python
# Use the classifier with different kernels

print('Gaussian kernel:')
lssvc_gpu = LSSVC_GPU(gamma=1, kernel='rbf', sigma=.5)
lssvc_gpu.fit(X_tr_norm, y_train)
y_pred = lssvc_gpu.predict(X_ts_norm).cpu()
print('acc_test = ', accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)), '\n')

print('Polynomial kernel:')
lssvc_gpu = LSSVC_GPU(gamma=1, kernel='poly', d=2)
lssvc_gpu.fit(X_tr_norm, y_train)
y_pred = lssvc_gpu.predict(X_ts_norm).cpu()
print('acc_test = ', accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)), '\n')

print('Linear kernel:')
lssvc_gpu = LSSVC_GPU(gamma=1, kernel='linear')
lssvc_gpu.fit(X_tr_norm, y_train)
y_pred = lssvc_gpu.predict(X_ts_norm).cpu()
print('acc_test = ', accuracy_score(dummie2multilabel(y_test), dummie2multilabel(y_pred)), '\n')
```

    Gaussian kernel:
    acc_test =  0.9680111265646731

    Polynomial kernel:
    acc_test =  0.9944367176634215

    Linear kernel:
    acc_test =  0.9791376912378303

The user may also save the model in JSON format:

```python
lssvc_gpu.dump('model')
loaded_model = LSSVC_GPU.load('model')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(lssvc_gpu.predict(X_ts_norm).cpu())
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(loaded_model.predict(X_ts_norm).cpu())
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303

The `model.json` is the same for the CPU and GPU version, giving the developer the possibility to train a model in GPU, dumping it in a .json, and loading in CPU version (the other way around is also possible).

```python
lssvc.dump('model_from_cpu')
lssvc_gpu = LSSVC_GPU.load('model_from_cpu')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(lssvc.predict(X_ts_norm))
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(lssvc_gpu.predict(X_ts_norm).cpu())
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303

```python
lssvc_gpu.dump('model_from_gpu')
lssvc = LSSVC.load('model_from_gpu')

# Showing the same results
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(lssvc_gpu.predict(X_ts_norm).cpu())
    )
)
print('acc_test = ', accuracy_score(
        dummie2multilabel(y_test),
        dummie2multilabel(lssvc.predict(X_ts_norm))
    )
)
```

    acc_test =  0.9791376912378303
    acc_test =  0.9791376912378303


## Demonstration of Lorenz Coefficient for Quantifying Spatial, Subsurface Heterogeneity

#### Alan Scherman, Rice University, UT PGE 2020 SURI

#### Supervised by:

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

***
#### Introduction

This is a demonstration of how to calculate the Lorenz Coefficient of a subsurface sample from well log porosity and permeability measurements. The Lorenz Coefficient is an important heuristic imported form the sphere of macroeconomics used to quantify the spatial heterogeneity of a subsurface sample. The Lorenz Coefficient is obtained by doubling the area between the Lorenz Curve and the Homogeneous Line. It ranges from 0 to 1 where, by convention, a coefficient of less than 0.3 suggests low heterogeneity and a coefficient of more than 0.6 indicates high heterogeneity. In practical terms, a low spatial heterogeneity allows for simple displacement of subsurface fluids and a high recovery factor.
***
#### Objective

To understand and apply the methodology to calculate the Lorenz Curve and Coefficient from porosity and permeability measurements through Python functionalities.

***
#### Calculation procedure

The following list contains the necessary steps to determine the Lorenz Coefficient of a subsurface sample:

**(1)** Sort porosity ($\phi$) and permeability (**_k_**) in descending order of ratio **_k_**/$\phi$;

**(2)** Calculate storage (**_S.C._**) and flow capacity (**_F.C._**) of each layer (region between depth measurements) with:

\begin{equation}
S.C. = \phi*h
\end{equation}
\begin{equation}
F.C. = k*h
\end{equation}

where **_h_** is the layer thickness;

**(3)** Calculate the _cumulative_ storage (**_C.S.C._**) and flow capacities (**_C.F.C._**) of each layer with:

\begin{equation}
C.S.C. = \sum^{current layer}_{i = 1} \phi*h
\end{equation}
\begin{equation}
C.F.C. = \sum^{current layer}_{i = 1} k*h
\end{equation}

**(4)** Normalize the cumulative storage and flow capacities by dividing them by the largest cumulative storage and flow capacities (i.e. the last cumulative capacities calculated), respectively;

**(5)** _(Optional)_ Plot the normalized cumulative flow capacities against the normalized cumulative storage capacities (i.e. the Lorenz Curve);

**(6)** Find a curve fit for the normalized capacities in the Cartesian plane (usually a 3rd degree polynomial is sufficient);

**(7)** Integrate to find the area between the Lorenz Curve and the Homogenous Line;

**(8)** Divide the result found in **(7)** by 0.5 to obtain the Lorenz Coefficient:

\begin{equation}
LorenzCoefficient = \frac{\int^{1}_{0} (LorenzCurve - HomogeneousLine)}{0.5}
\end{equation}
***
#### Load the required libraries

The program below utilizes some standard Python packages. These should be previously installed if you have Anaconda or other similar software.

```python
import pandas as pd                                                  # To import data from .xlsx or .csv file
import numpy as np                                                   # For numerical array management
from matplotlib import pyplot as plt                                 # For graphical display of Lorenz Curve
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.
***
#### Importing porosity and permeability data

The data file for this demonstration is availbale at [here](https://github.com/GeostatsGuy/GeoDataSets/blob/master/WellPorPermSample_3.xlsx).

The well log data will be imported from an Excel (.xlsx) file. There are similar Pandas methods which allow the importation of data from .csv and other common file types. First, the Excel data is stored as a Pandas data frame. Then, the depth, porosity, and permeability data are placed into separate data frames by means of their Excel column names. Make sure that the strings match every character of the column titles in the Excel file.

```python
df = pd.read_excel('WellPorPermSample_3.xlsx')                # Include file directory with os.chdir() method if necessary
depth = pd.DataFrame(df, columns = ['Depth (m)']);
poros = pd.DataFrame(df, columns = ['Por (%)']);
perme = pd.DataFrame(df, columns = ['Perm (mD)']);
```

To certify that the subsurface data have been imported appropriately:

```python
df.head()
```

      Depth (m)
      Por (%)
      Perm (mD)

      0
      0.25
      12.993634
      265.528738

      1
      0.50
      13.588011
      116.891220

      2
      0.75
      8.962625
      136.920016

      3
      1.00
      17.634879
      216.668629

      4
      1.25
      9.424404
      131.594114

***
#### Convert data frames to NumPy arrays

In order to facilitate data management and allow the use of helpful mathematical methods, we'll convert the Pandas data frames to NumPy arrays. Also, let's transpose the arrays to row format for conventionality.

```python
depth = np.transpose(depth.to_numpy()[:,0])
poros = np.transpose(poros.to_numpy()[:,0])
perme = np.transpose(perme.to_numpy()[:,0])
```

Once again, let's take a look at the resulting arrays:

```python
print('depth =', depth[:5], '\nporos =', poros[:5], '\nperme =', perme[:5])   # To visualize the first 5 elements of each array
```

    depth = [0.25 0.5  0.75 1.   1.25]
    poros = [12.99363384 13.58801077  8.96262538 17.63487852  9.42440351]
    perme = [265.52873839 116.89122026 136.92001631 216.66862895 131.5941136 ]

***
#### Determine thicknesses of layers

The layer thickness is found by subtracting a shallower depth from a deeper depth. For most well logs, the layers will have a uniform thickness. However, irregular thicknesses are also possible and should be accounted for. Whichever is the case, the layer thicknesses array can be calculated in the following manner:

```python
layer_thick = depth - np.concatenate(([0],depth[:-1]))
```

Note that the layer_thick array should have the same dimensions as the porosity and permeability arrays.
***
#### Step (1) - Sort data in decreasing order of k/$\phi$

Recall that the first step to calculate the Lorenz Coefficient is to sort all layers in descending order of their ratio of permeability by porosity. Because both the porosity and permeability values are expressed as NumPy arrays, these ratios can be easily obtained by:

```python
ratio = perme/poros                                                           # Equivalent to np.multiply(perme,poros)
```

Now, it's possible to sort the layer thickness, porosity and permeability arrays in the desired pattern:

```python
layer_thick = np.flip(layer_thick[np.argsort(ratio)], axis = 0);
poros = np.flip(poros[np.argsort(ratio)], axis = 0)
perme = np.flip(perme[np.argsort(ratio)], axis = 0)
```

***
#### Step (2) - Calculate storage and flow capacities

The storage and flow capacities are found by performing element-wise multiplication of the porosity and permeability arrays by the thickness array. Hence:

```python
storage_cap = poros*layer_thick
flow_cap = perme*layer_thick
```

***
#### Step (3) - Calculate the cumulative storage and flow capacities

The next step is to compute the cumulative capacities for each layer. An easy solution to this assignment is provided by a for-loop. First, we pre-allocate the cumulative storage and flow capacities arrays. Then, we set the cumulative capacities of the first layer equal to its own capacities. Finally, we run a for-loop to compute the remaining cumlative capacities for the sequential layers.

```python
cumul_storage_cap = np.zeros(len(flow_cap)+1)
cumul_flow_cap = np.zeros(len(flow_cap)+1)

cumul_storage_cap[1] = storage_cap[0]
cumul_flow_cap[1] = flow_cap[0]

for i in range(2,len(flow_cap)+1):
    cumul_storage_cap[i] = cumul_storage_cap[i-1] + storage_cap[i-1]
    cumul_flow_cap[i] = cumul_flow_cap[i-1] + flow_cap[i-1]
```

Note that the length of the cumulative capacities arrays are one index greater than the flow and storage capacity arrays. In fact, the first entry of both arrays is set to 0 (zero) in order to ensure convergence of the future Lorenz Curve to the origin.

***
#### Step (4) - Normalize the cumulative storage and flow capacities

To normalize the cumulative storage and flow capacities (i.e. limiting the range to [0,1]), we'll perform element-wise division of the cumulative storage and flow capacities by the sum of the all layers' storage and flow capacities, respectively. Thus, the normalized (also known as fractional) storage and flow capacities are given by:

```python
frac_storage_cap = cumul_storage_cap/cumul_storage_cap[-1]
frac_flow_cap = cumul_flow_cap/cumul_flow_cap[-1]
```

***
#### Step (5) - Plot the Lorenz Curve

Even though we say plot the Lorenz Curve, we'll indeed plot the fractional flow capacities against the fractional storage capacities (i.e. we'll plot some of the points on the Lorenz Curve).

Let's instantiate a figure to hold our Lorenz Curve points:

```python
plt.figure(figsize = (16,12))
plt.style.use('fivethirtyeight')                                             # Just a plot style I find pretty

plt.plot([0,1], [0,1], color = 'red', label = 'Homogeneous Line')            # Homogeneous line
plt.plot(frac_storage_cap, frac_flow_cap, color = 'black', \
         label = 'Lorenz Curve points', linestyle = 'dotted')                # Sample points of Lorenz Curve

plt.title('Lorenz Curve', size = 50)
plt.xlabel('Fraction of Total Storage Capacity (m)', size = 36)
plt.ylabel('Fraction of Total Flow Capacity (mD*m)', size = 36)
plt.legend(loc = 4, facecolor = '#C0C0C0', fontsize = 25)

plt.show()
```

***
#### Step (6) - Find a curve fit

Now that we have some of the data points of the Lorenz Curve, we can apply a curve fit to predict the behavior of this entire subsurface sample. We'll use a 3rd degree polynomial to fit the Lorenz Curve.

```python
weights = np.ones(len(frac_storage_cap))
weights[0] = 1000; weights[-1] = 1000                                           # To ensure the fit converges to (0,0) and (1,1)

poly_fit = np.polyfit(frac_storage_cap, frac_flow_cap, deg=3, w=weights)
print('poly_fit = ', poly_fit)
```

    poly_fit =  [ 2.82712856e-01 -1.00095169e+00  1.71823878e+00  1.62950901e-07]

The resulting array contains the coefficients of the polynomial fit. To better visualize this polynomial in standard equation form:

```python
poly_fit = np.poly1d(poly_fit)
print(poly_fit)
```

            3         2
    0.2827 x - 1.001 x + 1.718 x + 1.63e-07

***
#### Step (7) - Integrate to find the area between the Lorenz Curve and the Homogenenous LIne

The advantage of having applied the _numpy.poly1d()_ method on Step (6) is that we now have a callable object for which we can evaluate it over any domain (here [0,1]). Then:

```python
integral = np.polyint(poly_fit)
print(integral)
```

             4          3          2
    0.07068 x - 0.3337 x + 0.8591 x + 1.63e-07 x

The result above can be easily validated since the fit has polynomial behavior. Now, we can evaluate the definite integral for the area under the Lorenz Curve and then subtract the area under the homogeneous line (i.e. area of isosceles triangle of side length 1 = 0.5):

```python
lorenz_area = integral(1) - integral(0) - 0.5
print(lorenz_area)
```

    0.09614720008195232

Note that for practical purposes the integral(0) term could have been ignored since the integration constant is assumed to be 0.
***
#### Step (8) - Obtain the Lorenz Coefficient

The final step of this procedure is to divide the _lorenz_area_ calculated on Step (7) by 0.5, which is the total area between the lines **y = 1** and **y = x** in the domain [0,1]:

```python
lorenz_coefficient = lorenz_area/0.5
print('Lorenz Coefficient = %.3f' %lorenz_coefficient)
```

    Lorenz Coefficient = 0.192

Because the Lorenz Coefficient sits below the conventional threshold of 0.3, we can assume that the subsurface sample near the well exhibits simple displacement of fluids and a potentially high recovery factor.

Let's replicate the plot from Step (5) with shaded areas to better illustrate this Lorenz Coefficient:

```python
fig = plt.figure(figsize = (16,12))

plt.style.use('fivethirtyeight')

plt.plot([0,1],[0,1], label='Homogeneous Curve', color = 'red', linewidth = '3')                # Homogeneous line
plt.plot(frac_storage_cap, frac_flow_cap, label = 'Lorenz Curve points', \
         linestyle = 'dotted', color = 'black')
plt.plot(np.linspace(0,1,100), poly_fit(np.linspace(0,1,100)), linestyle = 'dashed',\
         linewidth = '3', label = 'Polynomial Fit', color = 'white')                            # Polynomial fit
plt.fill_between(np.linspace(0,1,100), poly_fit(np.linspace(0,1,100)),\
                 np.linspace(0,1,100), alpha = 0.75)                                            # Blue area
plt.fill_between(np.linspace(0,1,100), poly_fit(np.linspace(0,1,100)), 1,\
                 alpha = 0.75, color = 'orange')                                                # Orange area

plt.title('Lorenz Curve and Coefficient', size = 50)
plt.xlabel('Fraction of Total Storage Capacity (m)', size = 36)
plt.ylabel('Fraction of Total Flow Capacity (mD*m)', size = 36)
plt.legend(loc = 4, facecolor = '#C0C0C0', fontsize = 25)
plt.text(0.3, 0.8, 'Lorenz Coefficient = \n %.3f' %lorenz_coefficient, size = 40, horizontalalignment = 'center', \
         bbox=dict(facecolor='none', edgecolor='black'))

plt.show()
```

It's now clear that, graphically, the Lorenz Coefficient is equivalent to the blue area divided by the sum of the blue and orange areas.
***
#### Conclusion

This exercise detailed the procedure towards calculating the Lorenz Coefficient of a subsurface sample from its porosity and permeability measurements. We also covered some of the Python resources available to improve management of well data, perform the necessary mathematical transformations, and visualize the solution curve and the Lorenz Coefficient.

If you use the GeostatsPy package, the calculation and display of the Lorenz Coefficient are available through the **lorenz_curve()** and **lorenz_display()** functions. Please check the documentation on those for more detail.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


```python
from miscpy.utils.sympyhelpers import *
init_printing()
from sympy.utilities.codegen import codegen
```

## Set up rotation matrices representing a 3-1-3 $(\psi,\theta,\phi)$ Euler angle set.

```python
aCi = rotMat(3,psi)
cCa = rotMat(1,th)
bCc = rotMat(3,ph)
aCi,cCa,bCc
```

```python
bCi = bCc*cCa*aCi; bCi #3-1-3 rotation
```

```python
bCi_dot = difftotalmat(bCi,t,{th:thd,psi:psid,ph:phd});
bCi_dot
```

## $\tilde{\omega} = {}^\mathcal{B}C^{\mathcal{I}}  {}^\mathcal{B}{\dot{C}}^{\mathcal{I}}$

```python
omega_tilde = bCi*bCi_dot.T; omega_tilde
```

## $\left[{}^\mathcal{I}\boldsymbol{\omega}^{\mathcal{B}}\right]_\mathcal{B} = \left[ {}^\mathcal{B}C^{\mathcal{I}}_{32} \quad  {}^\mathcal{B}C^{\mathcal{I}}_{13} \quad {}^\mathcal{B}C^{\mathcal{I}}_{21} \right]$

```python
omega = simplify(Matrix([omega_tilde[2,1],omega_tilde[0,2],omega_tilde[1,0]]))
omega
```

```python
w1,w2,w3 = symbols('omega_1,omega_2,omega_3')
```

```python
s0 = solve(omega - Matrix([w1,w2,w3]),[psid,thd,phd]); s0
```

## Find EOM (second derivatives of Euler Angles)

```python
I1,I2,I3 = symbols("I_1,I_2,I_3",real=True,positive=True)
iWb_B = omega
I_G_B = diag(I1,I2,I3)
I_G_B
```

```python
diffmap = {th:thd,psi:psid,ph:phd,thd:thdd,psid:psidd,phd:phdd}
diffmap
```

```python
t1 = I_G_B*difftotalmat(iWb_B,t,diffmap)
t2 = skew(iWb_B)*I_G_B*iWb_B
t1,t2
```

```python
dh_G_B = t1+t2
dh_G_B
```

```python
t3 = expand(dh_G_B[0]*cos(ph)*I2 - dh_G_B[1]*sin(ph)*I1)
```

```python
sol_thdd = simplify(solve(t3,thdd))
sol_thdd
```

```python
t4= expand(dh_G_B[0]*sin(ph)*I2 + dh_G_B[1]*cos(ph)*I1)
t4
```

```python
sol_psidd = simplify(solve(t4,psidd))
sol_psidd
```

```python
sol_phdd = solve(dh_G_B[2],phdd)
sol_phdd
```

```python

```

## Find initial orientation such that $\mathbf h$ is down-pointing

```python
h = sqrt(((I_G_B*Matrix([w1,w2,w3])).transpose()*(I_G_B*Matrix([w1,w2,w3])))[0]);h
```

```python
eqs1 = simplify(bCi.transpose()*I_G_B*Matrix([w1,w2,w3]) - Matrix([0,0,-h])); eqs1 #equal 0
```

```python
simplify(solve(simplify(eqs1[0]*cos(psi) + eqs1[1]*sin(psi)),ph)) #phi solution
```

```python
solve(simplify(expand(simplify(-eqs1[0]*sin(psi) + eqs1[1]*cos(psi)).subs(ph,atan(I1*w1/I2/w2)))),th) #th solution
```

```python
simplify(eqs1[2].subs(ph,atan(I1*w1/I2/w2)))
```

```python

```

```python

```

### Generate MATLAB Code

```python
out = codegen(("eom1",sol_psidd[0]), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3]);out
```

    [('eom1.m',
      "function out1 = eom1(theta, thetadot, psi, psidot, phi, phidot, I_1, I_2, I_3)\n  %EOM1  Autogenerated by sympy\n  %   Code generated with sympy 1.3\n  %\n  %   See http://www.sympy.org/ for more information.\n  %\n  %   This file is part of 'project'\n\n  out1 = (-I_1.^2.*phidot.*psidot.*cos(2*phi - theta)/4 + I_1.^2.*phidot.*psidot.*cos(2*phi + theta)/4 - I_1.^2.*phidot.*thetadot.*cos(2*phi)/2 - I_1.^2.*phidot.*thetadot/2 - I_1.^2.*psidot.^2.*cos(2*phi - 2*theta)/8 + I_1.^2.*psidot.^2.*cos(2*phi + 2*theta)/8 - I_1.^2.*psidot.*thetadot.*cos(theta)/2 - I_1.^2.*psidot.*thetadot.*cos(2*phi - theta)/4 - I_1.^2.*psidot.*thetadot.*cos(2*phi + theta)/4 + I_1.*I_2.*phidot.*thetadot - I_1.*I_2.*psidot.*thetadot.*cos(theta) + I_1.*I_3.*phidot.*psidot.*cos(2*phi - theta)/4 - I_1.*I_3.*phidot.*psidot.*cos(2*phi + theta)/4 + I_1.*I_3.*phidot.*thetadot.*cos(2*phi)/2 + I_1.*I_3.*phidot.*thetadot/2 + I_1.*I_3.*psidot.^2.*cos(2*phi - 2*theta)/8 - I_1.*I_3.*psidot.^2.*cos(2*phi + 2*theta)/8 + I_1.*I_3.*psidot.*thetadot.*cos(theta)/2 + I_1.*I_3.*psidot.*thetadot.*cos(2*phi - theta)/4 + I_1.*I_3.*psidot.*thetadot.*cos(2*phi + theta)/4 + I_2.^2.*phidot.*psidot.*cos(2*phi - theta)/4 - I_2.^2.*phidot.*psidot.*cos(2*phi + theta)/4 + I_2.^2.*phidot.*thetadot.*cos(2*phi)/2 - I_2.^2.*phidot.*thetadot/2 + I_2.^2.*psidot.^2.*cos(2*phi - 2*theta)/8 - I_2.^2.*psidot.^2.*cos(2*phi + 2*theta)/8 - I_2.^2.*psidot.*thetadot.*cos(theta)/2 + I_2.^2.*psidot.*thetadot.*cos(2*phi - theta)/4 + I_2.^2.*psidot.*thetadot.*cos(2*phi + theta)/4 - I_2.*I_3.*phidot.*psidot.*cos(2*phi - theta)/4 + I_2.*I_3.*phidot.*psidot.*cos(2*phi + theta)/4 - I_2.*I_3.*phidot.*thetadot.*cos(2*phi)/2 + I_2.*I_3.*phidot.*thetadot/2 - I_2.*I_3.*psidot.^2.*cos(2*phi - 2*theta)/8 + I_2.*I_3.*psidot.^2.*cos(2*phi + 2*theta)/8 + I_2.*I_3.*psidot.*thetadot.*cos(theta)/2 - I_2.*I_3.*psidot.*thetadot.*cos(2*phi - theta)/4 - I_2.*I_3.*psidot.*thetadot.*cos(2*phi + theta)/4)./(I_1.*I_2.*sin(theta));\n\nend\n")]

```python
codegen(("eom1",sol_thdd[0]), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3])
```

    [('eom1.m',
      "function out1 = eom1(theta, thetadot, psi, psidot, phi, phidot, I_1, I_2, I_3)\n  %EOM1  Autogenerated by sympy\n  %   Code generated with sympy 1.3\n  %\n  %   See http://www.sympy.org/ for more information.\n  %\n  %   This file is part of 'project'\n\n  out1 = (I_1.^2.*phidot.*psidot.*sin(phi).^2.*sin(theta) + I_1.^2.*phidot.*thetadot.*sin(2*phi)/2 + I_1.^2.*psidot.^2.*sin(phi).^2.*sin(theta).*cos(theta) + I_1.^2.*psidot.*thetadot.*(sin(2*phi - theta) + sin(2*phi + theta))/4 - I_1.*I_2.*phidot.*psidot.*sin(phi).^2.*sin(theta) - I_1.*I_2.*phidot.*psidot.*sin(theta).*cos(phi).^2 - I_1.*I_3.*phidot.*psidot.*sin(phi).^2.*sin(theta) - I_1.*I_3.*phidot.*thetadot.*sin(2*phi)/2 - I_1.*I_3.*psidot.^2.*sin(phi).^2.*sin(theta).*cos(theta) - I_1.*I_3.*psidot.*thetadot.*(sin(2*phi - theta) + sin(2*phi + theta))/4 + I_2.^2.*phidot.*psidot.*sin(theta).*cos(phi).^2 - I_2.^2.*phidot.*thetadot.*sin(2*phi)/2 + I_2.^2.*psidot.^2.*sin(theta).*cos(phi).^2.*cos(theta) - I_2.^2.*psidot.*thetadot.*(sin(2*phi - theta) + sin(2*phi + theta))/4 - I_2.*I_3.*phidot.*psidot.*sin(theta).*cos(phi).^2 + I_2.*I_3.*phidot.*thetadot.*sin(2*phi)/2 - I_2.*I_3.*psidot.^2.*sin(theta).*cos(phi).^2.*cos(theta) + I_2.*I_3.*psidot.*thetadot.*(sin(2*phi - theta) + sin(2*phi + theta))/4)./(I_1.*I_2);\n\nend\n")]

```python
codegen(("eom1",sol_phdd[0]), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])
```

    [('eom1.m',
      "function out1 = eom1(theta, thetadot, psi, psidot, phi, phidot, I_1, I_2, I_3, psiddot)\n  %EOM1  Autogenerated by sympy\n  %   Code generated with sympy 1.3\n  %\n  %   See http://www.sympy.org/ for more information.\n  %\n  %   This file is part of 'project'\n\n  out1 = (I_1.*psidot.^2.*sin(phi).*sin(theta).^2.*cos(phi) - 2*I_1.*psidot.*thetadot.*sin(phi).^2.*sin(theta) + I_1.*psidot.*thetadot.*sin(theta) - I_1.*thetadot.^2.*sin(phi).*cos(phi) - I_2.*psidot.^2.*sin(phi).*sin(theta).^2.*cos(phi) + 2*I_2.*psidot.*thetadot.*sin(phi).^2.*sin(theta) - I_2.*psidot.*thetadot.*sin(theta) + I_2.*thetadot.^2.*sin(phi).*cos(phi) - I_3.*psiddot.*cos(theta) + I_3.*psidot.*thetadot.*sin(theta))./I_3;\n\nend\n")]

```python
codegen(("eom1",[s0[psid],s0[thd],s0[phd]]), 'Octave', argument_sequence=[w1,w2,w3,th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])
```

    [('eom1.m',
      "function [out1, out2, out3] = eom1(omega_1, omega_2, omega_3, theta, thetadot, psi, psidot, phi, phidot, I_1, I_2, I_3, psiddot)\n  %EOM1  Autogenerated by sympy\n  %   Code generated with sympy 1.3\n  %\n  %   See http://www.sympy.org/ for more information.\n  %\n  %   This file is part of 'project'\n\n  out1 = (omega_1.*sin(phi) + omega_2.*cos(phi))./sin(theta);\n  out2 = omega_1.*cos(phi) - omega_2.*sin(phi);\n  out3 = -omega_1.*sin(phi)./tan(theta) - omega_2.*cos(phi)./tan(theta) + omega_3;\n\nend\n")]

```python
codegen(("eom1",bCi), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])
```

    [('eom1.m',
      "function out1 = eom1(theta, thetadot, psi, psidot, phi, phidot, I_1, I_2, I_3, psiddot)\n  %EOM1  Autogenerated by sympy\n  %   Code generated with sympy 1.3\n  %\n  %   See http://www.sympy.org/ for more information.\n  %\n  %   This file is part of 'project'\n\n  out1 = [-sin(phi).*sin(psi).*cos(theta) + cos(phi).*cos(psi) sin(phi).*cos(psi).*cos(theta) + sin(psi).*cos(phi) sin(phi).*sin(theta); -sin(phi).*cos(psi) - sin(psi).*cos(phi).*cos(theta) -sin(phi).*sin(psi) + cos(phi).*cos(psi).*cos(theta) sin(theta).*cos(phi); sin(psi).*sin(theta) -sin(theta).*cos(psi) cos(theta)];\n\nend\n")]

```python
codegen(("eom1",omega), 'Octave', argument_sequence=[w1,w2,w3,th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])
```

    [('eom1.m',
      "function out1 = eom1(omega_1, omega_2, omega_3, theta, thetadot, psi, psidot, phi, phidot, I_1, I_2, I_3, psiddot)\n  %EOM1  Autogenerated by sympy\n  %   Code generated with sympy 1.3\n  %\n  %   See http://www.sympy.org/ for more information.\n  %\n  %   This file is part of 'project'\n\n  out1 = [psidot.*sin(phi).*sin(theta) + thetadot.*cos(phi); psidot.*sin(theta).*cos(phi) - thetadot.*sin(phi); phidot + psidot.*cos(theta)];\n\nend\n")]

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

Notebooks/Torque Free 3-1-3 Body Dynamics.ipynb


# TensorFlow Basics

Note: If running this notebook directly, make sure you are running your Jupyter kernel in an environment with TensorFlow installed. Some useful packages to install/import first, if you didn't in notebook 00A:

```python
# Beginning a line with "!" allows you to execute a command in your terminal
# You'll need to install these packages for certain lines of this notebook if you don't already have them.
!conda install -y matplotlib
!pip install -U tqdm
```

    Fetching package metadata .......
    Solving package specifications: ..........

    # All requested packages already installed.
    # packages in environment at C:\Users\kevin_000\Anaconda3\envs\tensorflow:
    #
    matplotlib                2.0.2               np113py36_0
    Requirement already up-to-date: tqdm in c:\users\kevin_000\anaconda3\envs\tensorflow\lib\site-packages

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
from tqdm import trange
```

### Basic TensorFlow test from the installation instructions

```python
import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))
```

    b'Hello, TensorFlow!'

### Define a simple TensorFlow graph

The example from the TensorFlow Introduction slides:

```python
a = tf.constant(3.0, dtype=tf.float32)
b = tf.constant(4.0, dtype=tf.float32)
sum_a_b = tf.add(a, b)

# Using the Python "with" as a context manager
with tf.Session() as sess:
    print(sess.run(sum_a_b)) # Prints “7.0” to the screen
```

    7.0

The previous graph only produces a constant output. Not particularly interesting. To create a similar graph that can acccept variable inputs:

```python
a = tf.placeholder(tf.float32)
b = tf.placeholder(tf.float32)
sum_a_b = tf.add(a, b)

with tf.Session() as sess:
    feed_dict = {a: 3.0, b: 4.0}
    print('1st Result: {0}'.format(sess.run(sum_a_b, feed_dict=feed_dict)))

    feed_dict = {a: 2015, b: 2020}
    print('2nd Result: {0}'.format(sess.run(sum_a_b, feed_dict=feed_dict)))
```

    1st Result: 7.0
    2nd Result: 4035.0

# MNIST Example

The MNIST dataset is very popular machine learning dataset, consisting of 70000 grayscale images of handwritten digits, of dimensions 28x28. We'll be using it as our example for this section of the tutorial, with the goal being to predict which the digit is in each image.

Since it's such a common (and small) dataset, TensorFlow has commands for downloading and formatting the dataset conveniently baked in already:

```python
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
```

    Extracting MNIST_data/train-images-idx3-ubyte.gz
    Extracting MNIST_data/train-labels-idx1-ubyte.gz
    Extracting MNIST_data/t10k-images-idx3-ubyte.gz
    Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

Let's take a look at how the data is organized:

```python
# Dataset statistics
print('Training image data: {0}'.format(mnist.train.images.shape))
print('Validation image data: {0}'.format(mnist.validation.images.shape))
print('Testing image data: {0}'.format(mnist.test.images.shape))
print('28 x 28 = {0}'.format(28*28))

print('\nTest Labels: {0}'.format(mnist.test.labels.shape))
labels = np.arange(10)
num_labels = np.sum(mnist.test.labels, axis=0, dtype=np.int)
print('Label distribution:{0}'.format(list(zip(labels, num_labels))))

# Example image
print('\nTrain image 1 is labelled one-hot as {0}'.format(mnist.train.labels[1,:]))
image = np.reshape(mnist.train.images[1,:],[28,28])
plt.imshow(image, cmap='gray')
```

### Logistic Regression Model

Define the graph input: this is where we feed in our training images into the model. Since MNIST digits are pretty small and the model we're using is very simple, we'll feed them in as flat vectors.

```python
# Define input placeholder
x = tf.placeholder(tf.float32, [None, 784])
```

To get our predicted probabilities of each digit, let's first start with the probability of a digit being a 3 like the image above. For our simple model, we start by applying a linear transformation. That is, we multiply each value of the input vector by a weight, sum them all together, and then add a bias. In equation form:

\begin{align}
y_3 = \sum_i w_{i,3} x_i + b_3
\end{align}

The magnitude of this result $y_3$, we'll take as being correlated to our belief in how likely we think the input digit was a 3. The higher the value of $y_3$, the more likely we think the input image $x$ was a 3 (ie, we'd hope we'd get a relatively large value for $y_3$ for the above image). Remember though, our original goal was to identify all 10 digits, so we also have:

\begin{align*}
y_0 =& \sum_i w_{i,0} x_i + b_0 \\
&\vdots \\
y_9 =& \sum_i w_{i,9} x_i + b_9
\end{align*}

We can express this in matrix form as:

\begin{align}
y = W x + b
\end{align}

To put this into our graph in TensorFlow, we need to define some Variables to hold the weights and biases:

```python
# Define linear transformation
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.matmul(x, W) + b
```

We can interpret these values (aka logits) $y$ as probabilities if we normalize them to be positive and add up to 1. In logistic regression, we do this with a softmax:

\begin{align}
p(y_i) = \text{softmax}(y_i) = \frac{\text{exp}(y_i)}{\sum_j\text{exp}(y_j)}
\end{align}

Notice that because the range of the exponential function is always non-negative, and since we're normalizing by the sum, the softmax achieves the desired property of producing values between 0 and 1 that sum to 1. If we look at the case with only 2 classes, we see that the softmax is the multi-class extension of the binary sigmoid function:

Computing a softmax in TensorFlow is pretty easy, sort of*:

**More on this later*

```python
# Softmax to probabilities
py = tf.nn.softmax(y)
```

That defines our forward pass of our model! We now have a graph that performs a forward pass: given an input image, the graph returns the probabilities the model thinks the input is each of the 10 classes. Are we done?

Not quite. We don't know the values of $W$ and $b$ yet. We're going to learn those by defining a loss and using gradient descent to do backpropagation. Essentially, we'll be taking the derivative with respect to each of the elements in $W$ and $b$ and wiggling them in a direction that reduces our loss.

The loss we commonly use in classification is cross-entropy. Cross-entropy is a concept from information theory:

\begin{align}
H_{y'}(y)=-\sum_i y'_i \text{log}(y_i)
\end{align}

Cross-entropy not only captures how *correct* (max probability corresponds to the right answer) the model's answers are, it also accounts for how *confident* (high confidence in correct answers) they are. This encourages the model to produce very high probabilities for correct answers while driving down the probabilities for the wrong answers, instead of merely be satisfied with it being the argmax.

In supervised models, we need labels to learn, so we create a placeholder for the labels in our training data:

```python
# Define labels placeholder
y_ = tf.placeholder(tf.float32, [None, 10])
```

The cross-entropy loss is pretty easy to implement:

```python
# Loss
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(py), reduction_indices=[1]))
```

In the old days, we would have to go through and derive all the gradients ourselves, then code them into our program. Nowadays, we have libraries to compute all the gradients automatically. Not only that, but TensorFlow comes with a whole suite of optimizers implementing various optimization algorithms. I'm not going to go into the details of why you should appreciate that right now, because I know that Prof David Carlson has an entire day's worth of material on optimization.

```python
# Optimizer
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(cross_entropy)
```

To train, we simply call the optimizer op we defined above. First though, we need to start a session and initialize our variables:

```python
# Create a session object and initialize all graph variables
sess = tf.Session()
sess.run(tf.global_variables_initializer())
```

There are much cleverer ways to design a training regimen that stop training once the model is converged and before it starts overfitting, but for this demo, we'll keep it simple:

```python
# Train the model
# trange is a tqdm function. It's the same as range, but adds a pretty progress bar
for _ in trange(1000):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
```

    100%|█████████████████████████████████████| 1000/1000 [00:01<00:00, 571.94it/s]

Notice, because of the way the dependency links are connected in our graph, running the optimizer requires an input to both the training image placeholder `x` and the training label placeholder `y_` (as it should). The values of all variables (`W` and `b`) are updated in place automatically by the optimizer.

Now let's see how we did! For every image in our test set, we run the data through the model, and take the digit in which we have the highest confidence as our answer. We then compute an accuracy by seeing how many we got correct:

```python
# Test trained model
correct_prediction = tf.equal(tf.argmax(py, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))
```

    Test accuracy: 0.902899980545044

Not bad for a simple model and a few lines of code.  Before we close the session, there's one more interesting thing we can do. Normally, it can be difficult to inspect exactly what the filters in a model are doing, but since this model is so simple, and the weights transform the data directly to their logits, we can actually visualize what the model's learning by simply plotting the weights. The results look pretty reasonable:

```python
# Get weights
weights = sess.run(W)

fig, ax = plt.subplots(1, 10, figsize=(20, 2))

for digit in range(10):
    ax[digit].imshow(weights[:,digit].reshape(28,28), cmap='gray')

# Close session to finish
sess.close()
```

The entire model, with the complete model definition, training, and evaluation (but minus the weights visualization), is below. Note the slight difference when calculating the softmax; this is done for numerical stability purposes.

```python
import tensorflow as tf
from tqdm import trange
from tensorflow.examples.tutorials.mnist import input_data

# Import data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Create the model
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.matmul(x, W) + b

# Define loss and optimizer
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

# Create a Session object, initialize all variables
sess = tf.Session()
sess.run(tf.global_variables_initializer())

# Train
for _ in trange(1000):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

# Test trained model
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))

sess.close()
```

    Extracting MNIST_data/train-images-idx3-ubyte.gz
    Extracting MNIST_data/train-labels-idx1-ubyte.gz
    Extracting MNIST_data/t10k-images-idx3-ubyte.gz
    Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

    100%|█████████████████████████████████████| 1000/1000 [00:01<00:00, 572.50it/s]

    Test accuracy: 0.9190000295639038

Note: The accuracy from the full version directly above might return a slightly different test accuracy from the step-by-step version we first went through. This is because mnist.train.next_batch by default shuffles the order of the training data, so we're seeing the data in a different order.

*Acknowledgment: Material adapted from the TensorFlow tutorial: https://www.tensorflow.org/get_started/*