# Numerical Methods in Scientific Computing
# Assignment 2

# Q1.

We know that for cubic spline interpolation:

\begin{equation}
    g_i\ (x)=\frac{g_i^{\prime \prime}(x_i)}{6}(\frac{(x_{i+1}-x)^3}{h}-h(x_{i+1}-x))+\frac{g_{i+1}^{\prime \prime} (x_{i+1})}{6}(\frac{(x-x_i)^3}{h}-h(x-x_i\ ))+f_i(\frac{x_{i+1}-x}{h})+f_{i+1}(\frac{x-x_i}{h})
\end{equation}

So for the interpolating polynomial in the first interval and with parabolic run-out condition $g^{\prime \prime}(x_0)=g^{\prime \prime}(x_1)$ we have,

\begin{equation}
    g_0\ (x)=\frac{g^{\prime \prime}(x_0)}{6}(\frac{(x_{1}-x)^3}{h}-h(x_{1}-x))+\frac{g^{\prime \prime} (x_{0})}{6}(\frac{(x-x_0)^3}{h}-h(x-x_0\ ))+f_0(\frac{x_{1}-x}{h})+f_{1}(\frac{x-x_0}{h})
\end{equation}

$\Rightarrow$ The coefficients of $x^3$ in the above equation $-\frac{g^{\prime \prime}(x_0)}{6h}$ and $\frac{g^{\prime \prime}(x_0)}{6h}$ cancel out each other and we have a parabola(quadratic)

Similarly for the interpolating polynomial in the last interval and with parabolic run-out condition $g^{\prime \prime}(x_{n-1})=g^{\prime \prime}(x_n)$ we have,

\begin{equation}
    g_{n-1}\ (x)=\frac{g^{\prime \prime}(x_{n-1})}{6}(\frac{(x_{n}-x)^3}{h}-h(x_{n}-x))+\frac{g^{\prime \prime}(x_{n-1})}{6}(\frac{(x-x_{n-1})^3}{h}-h(x-x_{n-1}\ ))+f_{n-1}(\frac{x_{n}-x}{h})+f_{n}(\frac{x-x_{n-1}}{h})
\end{equation}

$\Rightarrow$ The coefficients of $x^3$ in the above equation $-\frac{g^{\prime \prime}(x_{n-1})}{6h}$ and $\frac{g^{\prime \prime}(x_{n-1})}{6h}$ cancel out each other and we have a parabola(quadratic)

Hence interpolating polynomials are parabolas at the first and last interval for cubic spline interpolation with parabolic run-out cinditions

# Q2.

Quadratic Spline Interpolation:

Say, for given set of points $\{x_i, f_i\}_{i=0}^n$ we have piecewise quadratic polynomials as
\begin{equation}
  g(x) =
\begin{cases}
      g_0(x) & , x \in [x_0,x_1] \\
      g_1(x) & , x \in [x_1,x_2] \\
      ... \\
      g_{n-1}(x) & , x \in [x_{n-1},x_n] \\
\end{cases}
\end{equation}

- Suitable joint conditions for quadratic spline

\begin{equation}
    g_i^\prime(x_{i+1})=g_{i+1}^\prime(x_{i+1}) \quad , \quad \forall x \in \{0,1,...,n-2\}
\end{equation}
\begin{equation}
    g_i(x_{i})=f_i \quad , \quad \forall x \in \{0,1,...,n-1\}
\end{equation}
\begin{equation}
    g_i(x_{i+1})=f_{i+1} \quad , \quad \forall x \in \{0,1,...,n-1\}
\end{equation}

In total there are (n-1) + (n) + (n) equations.

- Obtaining coefficients of the spline

The first derivative of the piecewise polynomial for Quadratic spline interpolation are linear. Hence, we have

\begin{equation}
    g_i^\prime(x)=g_i^\prime(x_i)\frac{(x_{i+1}-x)}{h}+g_i^\prime(x_{i+1})\frac{(x-x_{i})}{h}
\end{equation}

Integrating the above equation,

\begin{equation}
    g_i(x)=-g_i^\prime(x_i)\frac{(x_{i+1}-x)^2}{2h}+g_i^\prime(x_{i+1})\frac{(x-x_{i})^2}{2h}+C
\end{equation}

Using the conditions $g_i(x_{i})=f_i$ and $g_i(x_{i+1})=f_{i+1}$ to obtain two equations and subtracting one from the other to eliminate C we have over the entire range,

\begin{equation}
    (g^\prime(x_i) + g^\prime(x_{i+1})) = \frac{2}{h}(f_{i+1}-f_i) \quad \forall i \in \{0,1,...,n-1\}
\end{equation}

This form is similar to what was obtained in Cubic spline interpolation. In total there are n piecewise quadratic polynomials with 3n unknowns and we have already got 3n-1 unknowns. With another extra constraint we can solve the set of linear equations as a Linear system to obtain the first derivatives which are substituted back to get $g(x)$.

Say if the initial condition is given as $g_i^\prime(x_0) = 0$, then the form of equation reduces to

\begin{equation}
    g^\prime(x_{i+1}) = \frac{2}{h}(f_{i+1}-f_i)-g^\prime(x_i) \quad \forall i \in \{0,1,...,n-1\}
\end{equation}

With all the first derivatives computed at $\{x_i\}_{i=0}^n$ the piecewise quadratic equations are given by,

\begin{equation}
    g_i(x)=g^\prime(x_i)(-\frac{(x_{i+1}-x)^2}{2h}+\frac{h}{2})+g^\prime(x_{i+1})\frac{(x-x_{i})^2}{2h}+f_i \quad \forall i \in \{0,1,...,n-1\}
\end{equation}

- Suitable end Conditions

We require an additional constraint explicitly mentioned to solve the above system of equations. It could be one of the following:

\begin{equation}
    g^\prime(x_0)=0
\end{equation}

\begin{equation}
    g^\prime(x_n)=0
\end{equation}

\begin{equation}
    g^\prime(x_0)=g^\prime(x_1)
\end{equation}

\begin{equation}
    g^\prime(x_n)=g^\prime(x_{n-1})
\end{equation}

- Computational Efforts

    * In the case of Cubic we are required to solve the Linear system for second derivatives by computing inverse of the tridiagonal matrix whose computational cost is given by O(n) and then obtaining the piecewise cubic polynomial for the entire range is O(n). So, totally O(n)

    * In the case of Quadratic splines we are required to solve the Linear system for second derivatives by computing directly from n-1 equations without requirement of calculating inverse of a matrix although still the computational cost is given by O(n) and then obtaining the piecewise quadratic polynomial for the entire range is O(n). So, totally O(n)

```python
# Import the necessary libraries
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import interact
```

```python
# Function to compute the piecewise Quadratic splines
def  quadratic_spline(a, b, n):
    # Initializing required variables
    h = (b-a)/n
    x_runge = np.linspace(a,b,n+1)
    f_runge = 1/(1+25*x_runge**2)
    g_prime = np.zeros((n+1))

    # Initial condition
    g_prime[0] = 0

    # Calculating g_prime values at all nodes
    for i in range(0,n):
        g_prime[i+1] = (2/h)*(f_runge[i+1]-f_runge[i]) - g_prime[i]

    # Compute piecewise values for each of g(Interpolant), f(Runge Function), x(values to plot)
    g_piecewise = [0 for _ in range(n)]
    f_piecewise = [0 for _ in range(n)]
    x_piecewise = [0 for _ in range(n)]
    for i in range(0,n):
        x = np.linspace(a+i*h, a+(i+1)*h, 10)
        g_piecewise[i] = g_prime[i]*( ((-(x_runge[i+1]-x)**2)/(2*h)) + (h/2) ) + g_prime[i+1]*( (((x-x_runge[i])**2)/(2*h)) ) + f_runge[i]
        f_piecewise[i] = 1/(1+25*x**2)
        x_piecewise[i] = x

    return g_piecewise, f_piecewise, x_piecewise

# Function to convert peicewise interpolant to global quadratic spline interpolant
def piecewise_to_global(g_piecewise, f_piecewise, x_piecewise):
    n_pieces = np.shape(g_piecewise)[0]
    n_plot = np.shape(g_piecewise)[1]
    g = []
    f = []
    x = []

    for i in range(n_pieces):
        for j in range(n_plot):
            if j == n_plot-1 and i != n_pieces-1: continue
            g.append(g_piecewise[i][j])
            f.append(f_piecewise[i][j])
            x.append(x_piecewise[i][j])

    return g, f, x
```

```python
# Set Interval boundaries and No. of pieces in the overall interval
a = 0
b = 1
n = 8

# Call the functions defined in previous steps
g_piecewise, f_piecewise, x_piecewise = quadratic_spline(a, b, n)
g, f, x = piecewise_to_global(g_piecewise, f_piecewise, x_piecewise)

# Next few lines to model the plot of obtained results
plt.plot(x,f,color='r', label="Runge Function", linestyle='dashdot')
plt.plot(x,g,color='b', label="Interpolant")
plt.legend()
xcoords = np.linspace(a,b,n+1)
xcoords = xcoords[1:-1]
for xc in xcoords:
    plt.axvline(x=xc, linestyle='dashed')
plt.xlabel('x', fontsize=15)
plt.ylabel('f(x)/g(x)', fontsize=15)
plt.title('Quadratic Spline Interpolation - Runge Function \n (Note: Vertical Lines indicate each interval) \nn=%i' %n, fontsize=15)
plt.xlim(a, b)
plt.rcParams["figure.figsize"] = [10,10]
plt.show()
```

```python
# Interactive Version
# [Need to run this cell again when opening the jupyter notebook each time]

@interact
def inter(a=(-1,1,0.1),b=(-1,1,0.1), n=(1,20,1)):
    if b<=a : return print('Set b greater than a !')
    g_piecewise, f_piecewise, x_piecewise = quadratic_spline(a, b, n)
    g, f, x = piecewise_to_global(g_piecewise, f_piecewise, x_piecewise)
    plt.plot(x,f,color='r', label="Runge Function", linestyle='dashdot')
    plt.plot(x,g,color='b', label="Interpolant")
    plt.legend()
    xcoords = np.linspace(a,b,n+1)
    xcoords = xcoords[1:-1]
    for xc in xcoords:
        plt.axvline(x=xc, linestyle='dashed')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.title('Quadratic Spline Interpolation - Runge Function\n a =%1.2f' % a + ', b =%1.2f' % b +', n =%i' % n, fontsize=15)
    plt.xlim(a, b)
    plt.rcParams["figure.figsize"] = [10,10]
    plt.show()
```

    interactive(children=(FloatSlider(value=0.0, description='a', max=1.0, min=-1.0), FloatSlider(value=0.0, descr…

# Q3.

- fourth order Pade approximation for the second derivative at the ith node

Say,

\begin{equation}
    y_i^{\prime\prime} = a_{-1}y_{i-1} + a_0y_{i} + a_1y_{i+1} + b_{-1}y_{i-1}^{\prime\prime} + b_{1}y_{i+1}^{\prime\prime}
\end{equation}

By taylor series expansion we have,

\begin{equation}
    y_{i\pm1} = y_i \pm hy_i^{\prime} + \frac{h^2}{2!}y_i^{\prime\prime} \pm \frac{h^3}{3!}y_i^{\prime\prime\prime}+\frac{h^4}{4!}y_i^{\prime\prime\prime\prime} + ...
\end{equation}

\begin{equation}
    y_{i\pm1}^{\prime\prime} = y_i^{\prime\prime} \pm hy_i^{\prime\prime\prime} + \frac{h^2}{2!}y_i^{\prime\prime\prime\prime} \pm \frac{h^3}{3!}y_i^{\prime\prime\prime\prime\prime} + ...
\end{equation}

Substituting back in the first equation and segregating by derivatives of y we have,
\begin{equation}
    y_i^{\prime\prime} = y_i(a_{1}+a_0+a_{-1}) + y_i^{\prime}(ha_{1}-ha_{-1}) + y_i^{\prime\prime}(\frac{h^2}{2!}a_{1}+\frac{h^2}{2!}a_{-1} + b_{1}+b_{-1}) + y_i^{\prime\prime\prime}(\frac{h^3}{3!}a_{1}-\frac{h^3}{3!}a_{-1} + hb_{1}-hb_{-1})+ y_i^{\prime\prime\prime\prime}(\frac{h^4}{4!}a_{1}+\frac{h^4}{4!}a_{-1} + \frac{h^2}{2!}b_{1}+\frac{h^2}{2!}b_{-1})+...
\end{equation}

Equating the coefficients of the derivatives of y from the above equation with the first equation. From the coefficients of $y_i^{\prime}$ and $y_i^{\prime\prime\prime}$ we can see that $a_1=a_{-1}$ and $b_1=b_{-1}$. The equations now reduce to,
\begin{equation}
    2a_1+a_0=0
\end{equation}

\begin{equation}
    h^2a_1+2b_1=1
\end{equation}

\begin{equation}
    \frac{2h^2}{4!}a_1+b_1=0
\end{equation}

Solving them and obtaining all the coefficients yields the following equation,

\begin{equation}
    f_{i-1}^{\prime\prime} + 10f_{i}^{\prime\prime} + f_{i+1}^{\prime\prime} = 12\frac{f_{i+1}-2f_{i}+f_{i-1}}{h^2} \quad \forall i \in \{2,...,N-1\}
\end{equation}

\begin{equation}
    \Rightarrow \frac{f_{i-1}^{\prime\prime}}{10} + f_{i}^{\prime\prime} + \frac{f_{i+1}^{\prime\prime}}{10} = \frac{12}{10}\frac{f_{i+1}-2f_{i}+f_{i-1}}{h^2} \quad \forall i \in \{2,...,N-1\}
\end{equation}

The nodes are,
\begin{equation}
    x_i = (i-0.5)h, \quad i \in \{1,2,...,N\}, \quad h=\frac{1-0}{N}
\end{equation}

The order of trailing terms is given by
\begin{equation}
    y_i^{\prime\prime\prime\prime\prime}(\frac{h^5}{5!}(a_1-a_{-1}) + \frac{h^3}{3!}(b_1-b_{-1})) + y_i^{\prime\prime\prime\prime\prime\prime}(\frac{h^6}{6!}(a_1+a_{-1}) + \frac{h^4}{4!}(b_1+b_{-1})) + ... = O(\frac{h^6}{h^2}) = O(h^4)
\end{equation}
\begin{equation}
    \Rightarrow y_i^{\prime\prime\prime\prime\prime\prime}(\frac{h^6}{6!}(a_1+a_{-1}) + \frac{h^4}{4!}(b_1+b_{-1})) + ... = O(\frac{h^6}{h^2}) = O(h^4)
\end{equation}

- For the left boundary
\begin{equation}
    y_1^{\prime\prime}+b_2y_2^{\prime\prime} = a_1y_1 + a_2y_2 + a_3y_3 + a_4y^{\prime}(0)+O(h^3)
\end{equation}

Writing as Taylor series expansions

\begin{equation}
    y_{2} = y_1 + hy_1^{\prime} + \frac{h^2}{2!}y_1^{\prime\prime} + \frac{h^3}{3!}y_1^{\prime\prime\prime}+\frac{h^4}{4!}y_1^{\prime\prime\prime\prime} + \frac{h^5}{5!}y_1^{\prime\prime\prime\prime\prime} + ...
\end{equation}

\begin{equation}
    y_{3} = y_1 + 2hy_1^{\prime} + \frac{(2h)^2}{2!}y_1^{\prime\prime} + \frac{(2h)^3}{3!}y_1^{\prime\prime\prime}+\frac{(2h)^4}{4!}y_1^{\prime\prime\prime\prime} + \frac{(2h)^5}{5!}y_1^{\prime\prime\prime\prime\prime} + ...
\end{equation}

\begin{equation}
    y^{\prime}(0) = y_1^{\prime} - \frac{h}{2}y_1^{\prime\prime} + \frac{(\frac{h}{2})^2}{2!}y_1^{\prime\prime\prime} - \frac{(\frac{h}{2})^3}{3!}y_1^{\prime\prime\prime\prime}+\frac{(\frac{h}{2})^4}{4!}y_1^{\prime\prime\prime\prime\prime} - \frac{(\frac{h}{2})^5}{5!}y_1^{\prime\prime\prime\prime\prime\prime} + ...
\end{equation}

\begin{equation}
    y_{2}^{\prime\prime} = y_1^{\prime\prime} + hy_1^{\prime\prime\prime} + \frac{h^2}{2!}y_1^{\prime\prime\prime\prime} + \frac{h^3}{3!}y_1^{\prime\prime\prime\prime\prime} + ...
\end{equation}

Substituting back and comparing the coefficients of the derivatives yield the following values for the coefficients,
\begin{equation}
    b_2 = \frac{-11}{23}
\end{equation}

\begin{equation}
    a_1 = \frac{-36}{23h^2}
\end{equation}

\begin{equation}
    a_2 = \frac{48}{23h^2}
\end{equation}

\begin{equation}
    a_3 = \frac{-12}{23h^2}
\end{equation}

\begin{equation}
    a_4 = \frac{-24}{23h}
\end{equation}

- Similarly for the right boundary.
\begin{equation}
    y_n^{\prime\prime}+b_2y_{n-1}^{\prime\prime} = a_1y_n + a_2y_{n-1} + a_3y_{n-2} + a_4y^{\prime}(1)+O(h^3)
\end{equation}

Writing as Taylor series expansions

\begin{equation}
    y_{n-1} = y_n - hy_n^{\prime} + \frac{h^2}{2!}y_n^{\prime\prime} - \frac{h^3}{3!}y_n^{\prime\prime\prime}+\frac{h^4}{4!}y_n^{\prime\prime\prime\prime} - \frac{h^5}{5!}y_n^{\prime\prime\prime\prime\prime} + ...
\end{equation}

\begin{equation}
    y_{n-2} = y_n - 2hy_1^{\prime} + \frac{(2h)^2}{2!}y_n^{\prime\prime} - \frac{(2h)^3}{3!}y_n^{\prime\prime\prime}+\frac{(2h)^4}{4!}y_n^{\prime\prime\prime\prime} - \frac{(2h)^5}{5!}y_n^{\prime\prime\prime\prime\prime} + ...
\end{equation}

\begin{equation}
    y^{\prime}(1) = y_n^{\prime} + \frac{h}{2}y_n^{\prime\prime} + \frac{(\frac{h}{2})^2}{2!}y_n^{\prime\prime\prime} + \frac{(\frac{h}{2})^3}{3!}y_n^{\prime\prime\prime\prime}+\frac{(\frac{h}{2})^4}{4!}y_n^{\prime\prime\prime\prime\prime} + \frac{(\frac{h}{2})^5}{5!}y_n^{\prime\prime\prime\prime\prime\prime} + ...
\end{equation}

\begin{equation}
    y_{n-1}^{\prime\prime} = y_n^{\prime\prime} - hy_n^{\prime\prime\prime} + \frac{h^2}{2!}y_n^{\prime\prime\prime\prime} - \frac{h^3}{3!}y_n^{\prime\prime\prime\prime\prime} + ...
\end{equation}

By Observation we can see that the above 5 equations can be conoverted to the 5 equations presented previously for left boundary condition by changing h with -h,
\begin{equation}
    b_2 = \frac{-11}{23}
\end{equation}

\begin{equation}
    a_1 = \frac{-36}{23h^2}
\end{equation}

\begin{equation}
    a_2 = \frac{48}{23h^2}
\end{equation}

\begin{equation}
    a_3 = \frac{-12}{23h^2}
\end{equation}

\begin{equation}
    a_4 = \frac{24}{23h}
\end{equation}

\begin{equation}
\end{equation}

- Linear System

Formulating in the form Ax=b,

$$ A= \left(\begin{matrix}1&\frac{-11}{23}&0&.&.&.&0\\\frac{1}{10}&1&\frac{1}{10}&0&.&.&0\\0&\frac{1}{10}&1&\frac{1}{10}&0&.&0\\.&.&.&.&.&.&.\\.&.&.&.&.&.&.\\.&.&.&.&\frac{1}{10}&1&\frac{1}{10}\\0&.&.&.&0&\frac{-11}{23}&1\end{matrix}\right) $$

$$ x= \left(\begin{matrix}y_1^{\prime\prime}\\y_2^{\prime\prime}\\.\\.\\y_{n-1}^{\prime\prime}\\y_{n}^{\prime\prime}\end{matrix}\right) $$

$$ b= \left(\begin{matrix}\frac{(-36y_1+48y_2-12y_3-24hy^\prime(0))}{23h^2}\\\frac{12(y_1-2y_2+y_3)}{10h^2}\\\frac{12(y_2-2y_3+y_4)}{10h^2}\\.\\.\\\frac{12(y_{n-2}-2y_{n-1}+y_n)}{10h^2}\\\frac{(-36y_n+48y_{n-1}-12y_{n-2}+24hy^\prime(1))}{23h^2}\end{matrix}\right) $$

\begin{equation}
    \Rightarrow \left(\begin{matrix}1&\frac{-11}{23}&0&.&.&.&0\\\frac{1}{10}&1&\frac{1}{10}&0&.&.&0\\0&\frac{1}{10}&1&\frac{1}{10}&0&.&0\\.&.&.&.&.&.&.\\.&.&.&.&.&.&.\\.&.&.&.&\frac{1}{10}&1&\frac{1}{10}\\0&.&.&.&0&\frac{-11}{23}&1\end{matrix}\right) \left(\begin{matrix}y_1^{\prime\prime}\\y_2^{\prime\prime}\\.\\.\\y_{n-1}^{\prime\prime}\\y_{n}^{\prime\prime}\end{matrix}\right) = \left(\begin{matrix}\frac{(-36y_1+48y_2-12y_3-24hy^\prime(0))}{23h^2}\\\frac{12(y_1-2y_2+y_3)}{10h^2}\\\frac{12(y_2-2y_3+y_4)}{10h^2}\\.\\.\\\frac{12(y_{n-2}-2y_{n-1}+y_n)}{10h^2}\\\frac{(-36y_n+48y_{n-1}-12y_{n-2}+24hy^\prime(1))}{23h^2}\end{matrix}\right)
\end{equation}

\begin{equation}
    \Rightarrow \left(\begin{matrix}1&\frac{-11}{23}&0&.&.&.&0\\\frac{1}{10}&1&\frac{1}{10}&0&.&.&0\\0&\frac{1}{10}&1&\frac{1}{10}&0&.&0\\.&.&.&.&.&.&.\\.&.&.&.&.&.&.\\.&.&.&.&\frac{1}{10}&1&\frac{1}{10}\\0&.&.&.&0&\frac{-11}{23}&1\end{matrix}\right) \left(\begin{matrix}y_1^{\prime\prime}\\y_2^{\prime\prime}\\.\\.\\y_{n-1}^{\prime\prime}\\y_{n}^{\prime\prime}\end{matrix}\right) = \left(\begin{matrix}\frac{-36}{23h^2}&\frac{48}{23h^2}&\frac{-12}{23h^2}&0&.&.&0\\\frac{12}{10h^2}&\frac{-24}{10h^2}&\frac{12}{10h^2}&0&.&.&0\\0&\frac{12}{10h^2}&\frac{-24}{10h^2}&\frac{12}{10h^2}&0&.&0\\.&.&.&.&.&.&.\\.&.&.&\frac{12}{10h^2}&\frac{-24}{10h^2}&\frac{12}{10h^2}&0\\0&.&.&0&\frac{12}{10h^2}&\frac{-24}{10h^2}&\frac{12}{10h^2}\\0&.&.&0&\frac{-12}{23h^2}&\frac{48}{23h^2}&\frac{-36}{23h^2}\end{matrix}\right)\left(\begin{matrix}y_1\\y_2\\.\\.\\.\\y_{n-1}\\y_n\end{matrix}\right)
\end{equation}

\begin{equation}
    Ay^{\prime\prime}=By
\end{equation}

\begin{equation}
    y^{\prime\prime}=x^3-y
\end{equation}

Substituting $y^{\prime\prime}$ as a function of $y$ we can solve the ODE with,
\begin{equation}
    y = (A+B)^{-1}Ax^3
\end{equation}

- Exact Solution

For the ODE given by,
\begin{equation}
    y^{\prime\prime}+y=x^3
\end{equation}

The solution is given by the equation
\begin{equation}
    y = c_1\cos{x}+c_2\sin{x}+x^3-6x
\end{equation}

Using the boundary conditions,
\begin{equation}
    y^{\prime}(0)=y^{\prime}(1)=0
\end{equation}

The coefficients $c_1$ and $c_2$ are,
\begin{equation}
    c_2=6 ; \quad c_1=\frac{6\cos{1}-3}{\sin{1}}=0.2873
\end{equation}
\begin{equation}
    \Rightarrow y = 0.2873\cos{x}+6\sin{x}+x^3-6x
\end{equation}

```python
# Function to find the numerical solution to ODE
def numerical_ODE(n_nodes):
    # Initialize the required parameters
    n =n_nodes
    h = 1/n
    x = np.array([(i-0.5)*h for i in range(1,n+1)])
#     y_exact = 0.2873*np.cos(x)+6*np.sin(x)+x**3-6*x
    y_exact = 3*(-1+(2*np.cos(1)))*np.cos(x)*(1/np.sin(1))+6*np.sin(x)+x**3-6*x

    # Formulate the linear system derived earlier
    A = np.eye(n)
    A[0][1] = -11/23
    A[n-1][n-2] = -11/23
    for i in range(1,n-1):
        A[i][i+1] = 1/10
        A[i][i-1] = 1/10

    B = np.eye(n)
    B[0][0] = -36/(23*h**2)
    B[0][1] = 48/(23*h**2)
    B[0][2] = -12/(23*h**2)
    B[n-1][n-1] = -36/(23*h**2)
    B[n-1][n-2] = 48/(23*h**2)
    B[n-1][n-3] = -12/(23*h**2)
    for i in range(1,n-1):
        B[i][i+1] = 12/(10*h**2)
        B[i][i-1] = 12/(10*h**2)
        B[i][i] = -24/(10*h**2)

    # Solve the linear system to obtain second derivatives
    y_num = np.dot(np.dot(np.linalg.inv(A+B), A), np.power(x, 3))

    return y_num, y_exact, x
```

```python
np.cos(90)
```

    -0.4480736161291701

```python
# Plot the results to compare between Numerical and Exact solutions to the ODE for different values of n
fig = plt.figure(figsize=(20,20))

# N = [10, 20, 50, 100, 200, 500, 1000]
N = [10, 20, 50, 100]

for i, n in enumerate(N):
    plt.subplot(3, 3, i+1)
    y_num, y_exact, x = numerical_ODE(n)
    plt.plot(x,y_exact, marker='o',color='r', label="Exact Value", linestyle='dashdot')
    plt.plot(x,y_num,color='b', label="Numerical Value")
    plt.legend()
    plt.xlabel('x')
    plt.ylabel('y')
    plt.title('Numerical vs Exact solution for the ODE\n n =%i' %n)
    plt.xlim(0, 1)
```

The numerical and exact solutions are similar as seen in the plots for all values of n which indicates and verifies the derivation for numerical approximation of the ODE.

```python
# Plot for the max norm error in numerical solution as a function of n in log-log plot.
N = [10, 20, 50, 100, 200, 500, 1000]
error = []
for i, n in enumerate(N):
    y_num, y_exact, x = numerical_ODE(n)
#     error.append(max((np.abs(y_exact-y_num))))
    error.append(np.linalg.norm(y_exact-y_num, np.inf))
plt.figure(figsize=(5,5))
plt.loglog(N, error, marker='o',color='r', label="Max norm error", linestyle='dashed')
plt.xlabel('n (no. of nodes)', fontsize=12)
plt.ylabel('max norm Error', fontsize=12)
plt.title('Error in computation as a function of number of nodes\n n=[10, 20, 50, 100, 200, 500, 1000]', fontsize=15)
plt.legend()
plt.show()

# Plot for the max norm error in numerical solution as a function of n in log-log plot.
N = [50, 100, 200, 500, 1000, 1250, 1500, 1750]
error = []
for i, n in enumerate(N):
    y_num, y_exact, x = numerical_ODE(n)
    error.append(max((np.abs(y_exact-y_num))))
plt.figure(figsize=(5,5))
plt.loglog(N, error, marker='o',color='r', label="Max norm error", linestyle='dashed')
plt.xlabel('n (no. of nodes)', fontsize=12)
plt.ylabel('max norm Error', fontsize=12)
plt.title('Error in computation as a function of number of nodes\n n=[50, 100, 200, 500, 1000, 1250, 1500, 1750]', fontsize=15)
plt.legend()
plt.show()
```

There is an initial decrease in the error in log-log plot and then followed by a slight increase as number of nodes increases.
Plotting again with a different set of number of nodes we see a kind of quadratic increase in error.

- Neumann Boundary conditions

The Neumann Boundary conditions are enforced into the discretized boundary value problem when we used a third order Pade scheme to approximate $y_1^{\prime\prime}$ and $y_n^{\prime\prime}$ where we modeled each as a function of other parameters including $y^{\prime}(0)$ and $y^{\prime}(1)$ respectively.

```python

```


# Non-trivial Band Topology and the Chern Number
## Graduate

## Overview

A Chern number tells us whether something non-trivial is going on in the wavefunction and lets us distinguish between different topological phases.

Now let me clarify what I mean when I say "non-trivial".

Normally, when we are studying materials, we move from a spatial dependence for the wavefunction to a momentum dependence across a Brillouin Zone.

When we are in a non-trivial phase, we can't define a wavefunction across the entire Brillouin Zone at the same time. We can rewrite the wavefunction to cover the area that didn't work before, but then some other section isn't well-defined.

To be clear, the physics is well defined everywhere, and every way we write the wavefunction gives the same physics. The problem lies in our inability to write down a single "chart" for the whole Zone. This conundrum is similar to the problem with plotting a globe in 2 dimensions. We always have to make cuts, but the entire globe can be covered by "charts" that make up an "atlas".

## Our Model

A page of a Review of Modern Physics behemoth, "Classification of topological quantum matter with symmetries" [1], presented this model below. It inspired me to dig deeper and fill in the details.   They attributed this model as describing the Quantum Anomolous Hall Effect, QAHE, but this form describes a wide variety topological phenomena.

\begin{equation}
H(k) = R_0(k) \sigma_0 + \vec{R}(k) \cdot \vec{\sigma}
\end{equation}
\begin{equation}
= R_0(k) \sigma_0 + R_1(k) \sigma_1 + R_2(k) \sigma_2 + R_3(k) \sigma_3
\end{equation}
where $\sigma_0$ is the identity matrix and $\sigma_i$ are simply the Pauli matrices.
Combining the terms, $H(k)$ is a 2x2 matrix.
\begin{equation}=
\begin{pmatrix}
R_0 + R_3 & R_1 -i R_2 \\
R_1 + i R_2 & R_0 - R_3
\end{pmatrix}
\end{equation}
The wavefunction will then be 2-valued.  This could denote two sublattices, or two different types of particles.

While we could use any values for $\vec{R}(k)$, we will use
\begin{equation}
\vec{R}(k) = \begin{pmatrix}
-2 \sin k_x \\
-2 \sin k_y \\
\mu +2 \sum_{x,y} \cos k_i \\
\end{pmatrix}
\end{equation}
as it's a fairly simple form that gives us the physics we want and exhibits phase transitions depending on the value of $\mu$.  I set $\mu$ as $1$ early on.  Go through the notebook with that value, then go through the notebook with $\mu = -5, -1, 1, 5$.

```julia
using Plots
plotlyjs()
```

```julia
μ=1

labels=["-π","-π/2","0","π/2","π"]
ticks=[-π,-π/2,0,π/2,π];

ks=linspace(-π,π,314)
l=length(ks)

ka=Array{Array{Float64},2}(l,l)
for ii in 1:l
    x=ks[ii]
    for jj in 1:l
        ka[ii,jj]=[x,ks[jj]]
    end
end
```

```julia
function R0(k::Array)
    return 0
end

function R1(k::Array)
    return -2*sin(k[1])
end

function R2(k::Array)
    return -2*sin(k[2])
end

function R3(k::Array)
    return μ+2*cos(k[1])+2*cos(k[2])
end
```

```julia
function R(k::Array)
    return sqrt(R1(k)^2+R2(k)^2+R3(k)^2)
end
```

## Band Diagram

First, let's just take a look at the energy spectrum.  For each $k$, we calculate the eigenvalues of the 2x2 matrix Hamiltonian.

To make the calculation simpler, I denote the function $R$ as
\begin{equation}
R=\sqrt{R_1^2+R_2^2+R_3^2}
\end{equation}
With that, energy can simply be written as
\begin{equation}
\lambda= R_0 \pm R
\end{equation}
$R_0$ moves the entire spectrum up and down but doesn't effect the gap, and won't affect the physics.  That term won't even factor into the eigenvectors.

```julia
function λp(k::Array{Float64})
    return R0(k)+R(k)
end

function λm(k::Array{Float64})
    return R0(k)-R(k)
end
```

Notation Note:  I denote the Array evaluation of a function as the function name followed by 'a'.

```julia
λpa=zeros(Float64,l,l)
λma=zeros(Float64,l,l)

for ii in 1:l
    for jj in 1:l
        λpa[ii,jj]=λp(ka[ii,jj])
        λma[ii,jj]=λm(ka[ii,jj])
    end
end
```

```julia
surface(ks,ks,λpa)
surface!(ks,ks,λma)
plot!(xticks= (ticks,labels),
yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
zlabel="Energy",
title="Band Diagram")
```

## Eigenvectors

To find the eigenvectors, we find the nullspace of the following matrix,
\begin{equation}
\begin{vmatrix}
R_0 +R_3 - R_0 \mp R & R_1 - i R_2 \\
R_1 + i R_2 & R_0 - R_3 -R_0 \mp R \\
\end{vmatrix}
\end{equation}

Take the bottom row. Second column value multiples first column and first column value multiples second column.
\begin{equation}
\begin{pmatrix}
\pm R + R_3 \\
R_1 + i R_2
\end{pmatrix}
\end{equation}

And now we need to normalize the state
\begin{equation}
\frac{1}{\sqrt{2 R \left( R \pm R_3 \right) }}
\begin{pmatrix}
\pm R + R_3\\
R_1 + i R_2
\end{pmatrix}
\end{equation}
In low energy, just the minus state will be occupied (lower energy), but it has a singularity at $\vec{R} = ( 0 ,0, R) $.

We have two complex numbers in this vector, and thus four things to plot $r_1, \theta_1, r_2, \theta_2$,
\begin{equation}
|\psi \rangle =
\begin{pmatrix}
r_1 e^{i \theta_1} \\
r_2 e^{i \theta_2}
\end{pmatrix}
\end{equation}

If we chose to create our vector from the top row instead,
\begin{equation}
\begin{pmatrix}
R_1 -i R_2 \\
- R_3 \mp R
\end{pmatrix}
\end{equation}
and normalized
\begin{equation}
\frac{1}{\sqrt{2 R \left( R \mp R_3 \right) }}
\begin{pmatrix}
R_1 - i R_2 \\
\pm R - R_3
\end{pmatrix}
\end{equation}
But the minus state for this one has a singularity at $\vec{R} = (0,0, -R )$.

We can move where the singularity is, but we can't get rid of it.  The problem-point doesn't show up in the physics, only in the wavefunction representation of it.  We cannot well represent the state across the entire Brillouin Zone at the same time.  This problem occurs because we have a topologically non-trivial state and a non-zero Chern number.

```julia
function up1(k::Array)
    front=1/sqrt(2*R(k)*(R(k)+R3(k)))
    return front*(R(k)+R3(k))
end

function up2(k::Array)
    front=1/sqrt(2*R(k)*(R(k)+R3(k)))
    return front*(R1(k)+im*R2(k))
end

up=Function[]
push!(up,up1)
push!(up,up2)
```

```julia
function um1(k::Array)
    front=1/sqrt(2*R(k)*(R(k)-R3(k)))
    return front*(-R(k)+R3(k))
end

function um2(k::Array)
    front=1/sqrt(2*R(k)*(R(k)-R3(k)))
    return front*(R1(k)+im*R2(k))
end

um=Function[]
push!(um,um1)
push!(um,um2)
```

```julia
upa=zeros(Complex{Float64},2,l,l)
uma=zeros(Complex{Float64},2,l,l)

for ii in 1:l
    for jj in 1:l
        upa[1,ii,jj]=up[1](ka[ii,jj])
        upa[2,ii,jj]=up[2](ka[ii,jj])
        uma[1,ii,jj]=um[1](ka[ii,jj])
        uma[2,ii,jj]=um[2](ka[ii,jj])
    end
end
```

```julia
# Plotting θ2
heatmap(ks,ks,angle.(uma[2,:,:])-angle.(uma[1,:,:]))
plot!(xticks= (ticks,labels),
yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="difference in phase, lower energy")
```

```julia
surface(ks,ks,norm.(uma[2,:,:]))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="magnitude,second component, lower energy")
```

```julia
surface(ks,ks,norm.(uma[1,:,:]))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="magnitude,first component, lower energy")
```

## Calculating the Connection

The first step in calculating the Chern number is evaluating the Berry Connection.

\begin{equation}
\mathcal{A}^{i} = \langle u (k,r) | d_{i} u (k,r) \rangle
\end{equation}

Though $\mathcal{A}^i$ looks like a vector, it is not invariant under gauge tranformation.  If a wavefunction transforms as
\begin{equation}
| u(k,r) \rangle \rightarrow e^{-i \phi} | u(k,r) \rangle
\end{equation}
then the connection transforms as
\begin{equation}
\mathcal{A}^i \rightarrow \mathcal{A}^i -i \partial_i \phi
\end{equation}

Near the Dirac points in the Brillouin Zone, the wavefunction has quite a high curvature, which makes the numerical computation of the derivative prone to errors.  I tried numerical derivation but was unable to arrive at a correct, stable answer.  Therefore, I'm using the 'ForwardDiff' Package to take derivatives analytically.

```julia
using ForwardDiff
```

Before dealing with the physics, let's just look at the syntax for the package.

Here's just an example of taking the gradient of $x^2$.

```julia
ex=x->ForwardDiff.gradient(t->t[1]^2,x)
ex([1])
```

Now let's apply that syntax to our wavefunctions.

The 'ForwardDiff' package seems to only work on purely real functions, so we have to take the derivative of the real and imaginary parts separately.

```julia
dum1=kt->ForwardDiff.gradient(um1,kt)

Rdum2=kt->ForwardDiff.gradient(t->real(um2(t)),kt)
Idum2=kt->ForwardDiff.gradient(t->imag(um2(t)),kt)
dum2(k)=Rdum2(k)+im*Idum2(k)
```

With the derivatives, we can now calculate the connection.

```julia
Amkx(k)=conj(um1(k))*dum1(k)[1]+conj(um2(k))*dum2(k)[1]
Amky(k)=conj(um1(k))*dum1(k)[2]+conj(um2(k))*dum2(k)[2]
```

```julia
Akxa=Array{Complex{Float64}}(l,l)
Akya=Array{Complex{Float64}}(l,l)
for ii in 1:l
    for jj in 1:l
        Akxa[ii,jj]=Amkx(ka[ii,jj])
        Akya[ii,jj]=Amky(ka[ii,jj])
    end
end
```

```julia
heatmap(ks,ks,log.(norm.(Akxa)))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="Akx Berry Connection")
```

```julia
heatmap(ks,ks,log.(norm.(Akya)))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="Aky Berry Connection")
```

```julia
heatmap(ks,ks,angle.(Akxa))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="Akx Berry Connection")
```

```julia
heatmap(ks,ks,angle.(Akya))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="Aky Berry Connection")
```

## Curvature

I previously had a section here about Berry phases, but I need to fix it.  Until then, please believe the mass of physics literature that from the guage-dependent Berry connection, we can get a gauge independent quantity called the <i>Berry Curvature</i>.  Hopefully I can clarify this mathematical magic when I understand it myself.

Of course it's called Berry.  Every thing in this calculation is named after Berry.  At least its not Euler.  Done with side note, back to equations.
Here's our version of Gauss's Theorem:
\begin{equation}
\oint \mathcal{A} \cdot ds  = \iint F_{n}^{xy} d^2 k
\end{equation}

And $F_n^{xy}$ will take on this form:
\begin{equation}
F_{n}^{xy} = \nabla \times \mathcal{A} = \partial_{k_x} \mathcal{A}^y_{n} - \partial_{k_y} \mathcal{A}^x_{n}
\end{equation}
\begin{equation}
= \partial_{k_x} \langle u | \partial_{k_y} u \rangle - \partial_{k_y} \langle u | \partial_{k_x} u \rangle
\end{equation}

TIME FOR MORE DERIVATIVES!

```julia
DRAmkx=kt->ForwardDiff.gradient(t->(real(Amkx(t))),kt )
DImAmkx=kt->ForwardDiff.gradient(t->(imag(Amkx(t))),kt )

DRAmky=kt->ForwardDiff.gradient(t->(real(Amky(t))),kt )
DImAmky=kt->ForwardDiff.gradient(t->(imag(Amky(t))),kt )
```

```julia
F(k)=DRAmky(k)[1]+im*DImAmky(k)[1]-DRAmkx(k)[2]-im*DImAmkx(k)[2]
```

```julia
Fa=Array{Complex{Float64}}(l,l)
for ii in 1:l
    for jj in 1:l
        Fa[ii,jj]=F(ka[ii,jj])
    end
end
```

```julia
heatmap(ks,ks,norm.(Fa))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="F Berry curvature norm")
```

```julia
heatmap(ks,ks,angle.(Fa))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="F Berry Curvature phase")
```

## Chern Number

Last Step!

Easy calculation now in terms of code, but this number has a great deal of significance.  I'm still trying to wrap my head around it.  The Chern number seems to pop up in a variety of obscure mathematical stuff over this physicist's head, but hopefully none of that is necessary to grasp its incredible mind-blowing usefullness.

This single integer not only seperates out topological phases from topologically trivial phases, but seperates different topological phases from each other.  And always evaluates to an integer.

\begin{equation}
Ch = \frac{1}{2 \pi i} \iint_{BZ} F^{xy}_n d^2 k
\end{equation}

If you don't get approximately an integer, try a finer mesh.

```julia
sum(Fa)*(ks[2]-ks[1])^2/(2π*im)
```

## Victory !!!!!!!!!!

You made it!

Even if you didn't understand everything, or really understand much at all, pat yourself on the back.  This is the very frontier of science.

[1] Ching Kai Chiu, Jeffrey C.Y. Teo, Andreas P. Schnyder, and Shinsei Ryu, Classification of topological quantum matter with symmetries, Reviews of Modern Physics 88 2016, no. 3, 1–70.

@article{RevelsLubinPapamarkou2016,
    title = {Forward-Mode Automatic Differentiation in Julia},
   author = {{Revels}, J. and {Lubin}, M. and {Papamarkou}, T.},
  journal = {arXiv:1607.07892 [cs.MS]},
     year = {2016},
     url = {https://arxiv.org/abs/1607.07892}
}

```julia

```


```python
from devito import *
from examples.seismic.source import WaveletSource, TimeAxis
from examples.seismic import plot_image
import numpy as np

from sympy import init_printing, latex
init_printing(use_latex=True)
```

```python
# Initial grid: 1km x 1km, with spacing 100m
extent = (2000., 2000.)
shape = (81, 81)
x = SpaceDimension(name='x', spacing=Constant(name='h_x', value=extent[0]/(shape[0]-1)))
z = SpaceDimension(name='z', spacing=Constant(name='h_z', value=extent[1]/(shape[1]-1)))
grid = Grid(extent=extent, shape=shape, dimensions=(x, z))
```

```python
class DGaussSource(WaveletSource):

    def wavelet(self, f0, t):
        a = 0.004
        return -2.*a*(t - 1./f0) * np.exp(-a * (t - 1./f0)**2)

# Timestep size from Eq. 7 with V_p=6000. and dx=100
t0, tn = 0., 200.
dt = 1e2*(1. / np.sqrt(2.)) / 60.
time_range = TimeAxis(start=t0, stop=tn, step=dt)

src = DGaussSource(name='src', grid=grid, f0=0.01, time_range=time_range)
src.coordinates.data[:] = [1000., 1000.]
```

```python
#NBVAL_SKIP

src.show()
```

```python
# Now we create the velocity and pressure fields
p = TimeFunction(name='p', grid=grid, staggered=NODE, space_order=2, time_order=1)
v = VectorTimeFunction(name='v', grid=grid, space_order=2, time_order=1)
```

```python
from devito.finite_differences.operators import div, grad
t = grid.stepping_dim
time = grid.time_dim

# We need some initial conditions
V_p = 4.0
density = 1.

ro = 1/density * dt
l2m = V_p*V_p*density * dt

# The source injection term
src_p = src.inject(field=p.forward, expr=src)

# 2nd order acoustic according to fdelmoc
u_v_2 = Eq(v.forward, v + ro * grad(p))
u_p_2 = Eq(p.forward, p + l2m * div(v.forward))
```

```python
u_v_2
```

```python
u_p_2
```

```python
op_2 = Operator([u_v_2, u_p_2] + src_p)
```

```python
#NBVAL_IGNORE_OUTPUT

# Propagate the source
op_2(time=src.time_range.num-1)
```

    Operator `Kernel` run in 0.04 s

```python
#NBVAL_SKIP

# Let's see what we got....
plot_image(v[0].data[0])
plot_image(v[1].data[0])
plot_image(p.data[0])
```

```python
norm_p = norm(p)
assert np.isclose(norm_p, .35098, atol=1e-4, rtol=0)
```

```python
# # 4th order acoustic according to fdelmoc
# # Now we create the velocity and pressure fields
p4 = TimeFunction(name='p', grid=grid, staggered=NODE, space_order=4, time_order=1)
v4 = VectorTimeFunction(name='v', grid=grid, space_order=4, time_order=1)
u_v_4 = Eq(v4.forward, v4 + ro * grad(p4))
u_p_4 = Eq(p4.forward, p4 + l2m * div(v4.forward))
```

```python
#NBVAL_IGNORE_OUTPUT
op_4 = Operator([u_v_4, u_p_4] + src_p)
# Propagate the source
op_4(time=src.time_range.num-1)
```

    Operator `Kernel` run in 0.04 s

```python
#NBVAL_SKIP

# Let's see what we got....
plot_image(v4[0].data[-1])
plot_image(v4[1].data[-1])
plot_image(p4.data[-1])
```

```python
norm_p = norm(p)
assert np.isclose(norm_p, .35098, atol=1e-4, rtol=0)
```


# Lecture 17: Moment Generating Functions (MGFs), hybrid Bayes' rule, Laplace's rule of succession

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## $\operatorname{Expo}(\lambda)$ and the  Memorylessness Property

#### Theorem: If $X$ is a positive, continuous r.v. with the memorylessness property, then $X \sim \operatorname{Expo}(\lambda)$ for some $\lambda$.

Let $F$ be the CDF of $X$, $G(x) = P(X \ge x) = 1 - F(x)$.

With the memoryless property, $G(s + t) = G(s) \, G(t)$.

This time, rather than trying to solve for $s$ or $t$, we are going to solve for the function $G$, in order to show that it is only the exponential function that has the memorylessness property (in the continuous case).

\begin{align}
  & \text{let } s = t & \quad \\
  & \Rightarrow & G(2t) &= G(t + t) = G(t) \, G(t) = G(t)^{2} & \quad \\
  & & G(3t) &= G(2t) \, G(t) = G(t)^{2} \, G(t) = G(t)^{3} & \quad \\
  & &\dots & \quad \\
  & & G(kt) &= G(t)^{k} & \quad  \\
  \\
  & \text{case where } k = \frac{1}{n} & \quad \\
  & \Rightarrow & G\left(2 \, \frac{t}{2}\right) &= G\left(\frac{t}{2}\right)^{2} \text{ so } G\left(\frac{t}{2}\right) = \sqrt{G(t)} = G(t)^{1/2} & \quad \\
  & & G\left(\frac{t}{3}\right) &=  G(t)^{1/3} & \quad \\
  & &\dots & \quad \\
  & & G\left(\frac{t}{k}\right) &= G(t)^{1/k} & \quad \\
  \\
  & \text{case where } k = \frac{m}{n} & \quad \\
  & \Rightarrow & G\left(\frac{m}{n} \, t \right) &= G(t)^{m/n} & \quad \\
  \\
  & \text{let } x \in \mathbb{Q} & \quad \\
  & \Rightarrow & G(x \, t) &= G(t)^{x} ~~~~ \text{ for all } x \ge 0\\
  \\
  \\
  & \text{now let } t =1 & \quad \\
  & \Rightarrow & G(x) &= G(1)^{x} & \quad \\
  &             &      &= e^{x \, ln \, G(1)} ~~~~ \text{ where } ln \, G(1) \text{ is some negative real number } \\
  &             &      &= e^{-\lambda x} & \quad \blacksquare \\
\end{align}

And so now we see that in the continuous case, $\operatorname{Expo}(\lambda)$ is the only distribution with the memorylessness property.

## Moment Generating Function (MGF)

Moment generating functions are an alternative way to describe a distribution.

#### Definition

A random variable $X$ has MGF $M(t) = \mathbb{E}(e^{tX})$, as a function of $t$, if this is finite on some $(-a, a)$ where $a > 0$.

Note that any function of a random variable _is itself a random variable_, so it makes some sense that we can obtain the expected value $\mathbb{E}(e^{tX})$

But why is this called _moment-generating_?

\begin{align}
  \mathbb{E}(e^{tX}) &= \mathbb{E}\left(\sum_{n=0}^{\infty} \frac{X^{n} \, t^{n}}{n!} \right) &\quad \text{Taylor expand e} \\
  &= \sum_{n=0}^{\infty} \left( \frac{\mathbb{E}(X^{n}) \, t^{n}}{n!}\right) &\quad \text{ where } \mathbb{E}(X^{n}) \text{ is called the } n^{th} \text{ moment} \\
\end{align}

#### Moments

* the average value for a random variable $X$ $\mathbb{E}(X)$ is known as the _first moment_
* the _second moment_ of $X$ is $\mathbb{E}(X^{2})$ which helps use derive $\operatorname{Var}(X)$
* higher moments are easily generated (derived), as well

### 3 reasons why MGF is important

Let $X$ have MGF $M(t)$.

1. The first moment $\mathbb{E}(^{n})$ is the coeficient of $\frac{t^{n}}{n!}$  in the Taylor series of $M$, i.e., $M^{n}(0) = \mathbb{E}(X^{n})$
1. MGF determines the distribution, i.e., if $X$ and $Y$ have the same MGF, then they have the same CDF
1. sums of random variables (convolutions) are difficult; but if we have MGFs, they are _easy_

### Sums of MGFs

If we have _independent_ r.v. $X$ and $Y$, and we know their respective moment generating functions, then we can easily find the moment generating function for $X + Y$

\begin{align}
  M(X + Y) &= \mathbb{E}(e^{t(X+Y)}) \\
           &= \mathbb{E}(e^{tX}) \, \mathbb{E}(e^{tY}) &\quad \text{ by independence}
\end{align}

### MGF for $Bern(p)$

Given $X \sim \operatorname{Bern}(p)$, we obtain the MGF with

\begin{align}
  M(t) &= \mathbb{E}(e^{tX}) \\
       &= p \, e^t * q &\quad \text{ where } q = 1-p
\end{align}

### MGF for $\operatorname{Bin}(p)$

Given $X \sim Bin(n,p)$, we obtain the MGF with

\begin{align}
  M(t) &= \mathbb{E}(e^{tX}) \\
       &= \left( p \, e^t + q \right)^n &\quad \text{ by applying } G(kt) = G(t)^{k}
\end{align}

### MGF for standard normal $Z \sim \mathcal{N}(0,1)$

Given standard normal $Z \sim \mathcal{N}(0,1)$, we obtain the MGF with

\begin{align}
  M(t) &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{tZ - Z^2/2} \, dZ \\
       &= \frac{1}{\sqrt{2\pi}} ~~ e^{t^2/2} \int_{-\infty}^{\infty} e^{-\frac{1}{2}\,(Z-t)^2} \, dZ &\quad \text{ completing the square} \\
       &= \frac{1}{\sqrt{2\pi}} ~~ e^{t^2/2} ~~ \sqrt{2\pi} &\quad \text{ recall the PDF of standard normal (Lec. 13)}  \\
       &= e^{t^2/2}
\end{align}

\*And just in case you've forgotten how to [complete the square](https://www.youtube.com/watch?v=bclm1tJB-3g)...

### MGF for normal $X \sim \mathcal{N}(\mu, \sigma^2)$

\begin{align}
  M(t) &= \mathbb{E}(e^{tX}) \\
       &= \int_{-\infty}^{\infty} e^{tx} \, \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2} \left(\frac{x - \mu}{\sigma}\right)^2} \, dx \\
       &= \int_{-\infty}^{\infty} \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{\left( x^2 - 2x\mu + \mu^2 \right)}{2\sigma^2} + tx} \, dx \\
       &= \int_{-\infty}^{\infty} \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{x^2 - 2x\mu - 2\sigma^{2}tx + \mu^2}{2\sigma^2}} \, dx \\
       &= \int_{-\infty}^{\infty} \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2}x^2 - 2x(\mu + \sigma^{2}t) + \mu^2} \, dx \\
       &= \int_{-\infty}^{\infty} \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2} (x - (\mu + \sigma^{2}t))^2 - (\mu + \sigma^{2}t)^2 + \mu^2} \, dx \\
       &= e^{-\frac{1}{2\sigma^2} - (\mu + \sigma^{2}t)^2 + \mu^2} \int_{-\infty}^{\infty} \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2} \left( \frac{(x - (\mu + \sigma^{2}t))}{\sigma} \right)^2}  \, dx \\
       &= e^{-\frac{1}{2\sigma^2} (- \mu^2 - 2 \mu \sigma^{2}t - \sigma^{4}t^2 + \mu^2)} \\
       &= e^{\frac{2 \mu \sigma^{2}t + \sigma^{4}t^2}{2\sigma^2} } \\
       &= e^{\mu t + \frac{\sigma^2 t^2}{2} }
\end{align}

----

## Laplace's Rule of Succession

_If we have observed the sun rising for the past $n$ days in succession, then what is the probability that the sun will rise tomorrow?_

Given $p$ is the probability that the sun will rise on any given day $X_k$, we can consider a consecutive string of days $X_1, X_2, \dots \text{ i.i.d. } \operatorname{Bern}(p)$ which is conditional on $p$. But for the question above, we do not know what $p$ is. Bayesians treat $p$ as an r.v.

### Problem structure

* Let $p \sim \operatorname{Unif}(0,1)$ be our _prior_; we choose $\operatorname{Unif}(0,1)$ since $p$ could be _anything_
* Let $S_n = X_1 + X_2 + \cdots + X_n$
* So we then assume $S_n | p \sim \operatorname{Bin}(n,p) \text{, } p \sim \operatorname{Unif}(0,1)$

### Questions

1. What is the _posterior_ $p | S_n$?
1. What is $P(X_{n+1} | S_n)$, the probability that the sun will rise tomorrow given that we have observed so for the past $n$ days?

### Solution

We use $f$ as a simple stand-in for the PDF $p$. We start with the general case:

\begin{align}
  f( p | S_n=k) &= \frac{P(S_n=k | p) f(p)}{P(S_n=k)} &\quad \text{ from Bayes' Rule} \\
                &\propto p^k \, (1-p)^{n-k}
\end{align}

But since
* the _prior_ $f(p) = 1$ since it is Uniform.
* $P(S_n = k)$ does not depend on $p$
* the binomial coefficient of $P(p | S_n=k) = \binom{n}{k} p^k \, (1-p)^{n-k}$ also does not depend on $p$, and can be treated as a constant

we can consider $f(p | S_n=k) $ with proportionality.

Now let's consider the case of our question, where the sun has risen for $n$ days straight:

\begin{align}
  \text{since } f(p) &= \int_{0}^{1} p^n \, dp \\
       &= \frac{1}{n+1}
  \\
  \\
  \text{so } (p | S_n=n) &= \boxed{(n+1) \, p^n} &\quad \text{ normalizing for a valid PDF}\\
  \\
  \text{and } P(X_{n+1}=1 | S_n=n) &= \int_{0}^{1} (n+1) \, p \, p^n \, dp &\quad \text{ Fundamental Bridge, } \mathbb{E}(p | S_n=n) \\
       &= \int_{0}^{1} (n+1) \, p^{n+1} \, dp \\
       &= \boxed{\frac{n+1}{n+2}}
\end{align}

----

View [Lecture 17: Moment Generating Functions | Statistics 110](http://bit.ly/2CxVsgR) on YouTube.


## Subsurface Data Analytics

### Multidimensional Scaling for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Subsurface Machine Learning: Multidimensional Scaling for Subsurface Data Analytics

Here's a demonstration of mulidimensional scaling for subsurface modeling in Python. This is part of my Subsuface Machine Learning Course at the Cockrell School of Engineering at the University of Texas at Austin.

#### Multidimensional Scaling

A powerful ordination method in inferential statistics / information visualization for exploring / visualizing the similarity (conversely the difference) between individual samples from a high dimensional dataset.

* beyond 2 or 3 features it is difficult to visualize the relationship between samples

* for 2 features we can easily visualize the relationships between samples with a scatter plot

* for 3 features we can either visualize in 3D or include color or matrix scatter plots

Multidimensional scaling projects the $m$ dimensional data to $p$ dimensions such that $p << m$.

* ideally we are able to project to $p=2$ to easily explore the relationships between the samples

While principal component analysis (PCA) operates with the covariance matrix, multidimensional scaling operates with the distance / dissimilarity matrix.

* you don't need to know the actual feature values, just the distance or dissimilarity between the samples

* as with any distance in feature space, we consider feature standardization and weighting

* we may also work with a variety of dissimilarity measures

#### Classical Multidimensional Scaling

Based on Euclidian distance between samples.

The Steps:

1. calculate the square distance matrix, $D^{(2)} = \left[ \delta_{i,j}^2 \right]$.

2. apply double centering $B = - \frac{1}{2} J D^{(2)} J$

3. solve for the Eigen Values, $\lambda_1,\ldots,\lambda_p$

4. solve for the projected coordinates, $x^{'}_1, \dots ,x^{'}_p$

\begin{equation}
X^{'} = E_m \land^{\frac{1}{2}}_m
\end{equation}

where $E_m$ is the matrix of Eigen vectors and $\land_m$ is the diagonal matrix of Eigen values.

General comments about classical multidimensional scaling:

* nonlinear dimensionality reduction

* no distribution assumption

* the transform may not be unique, and may be arbitrarily be translated, rotated and transformed (these do not change the pairwise distances $||x_i - x_j||$.

#### Metric Multidimensional Scaling

A generalization of classical multidimensional scaling with a variety of metrics and a loss function optimization.

* formulated as an optimization problem to minimize the squate difference between the original and projected pairwise distances

\begin{equation}
min_{x_1,\ldots,x_m} \sum_{i<j} \left( ||x_i - x_j|| - \delta_{i,j} \right)^2
\end{equation}

where $||x_i - x_j||$ are the pairwise distances in the projected space ($p$ dimensional) and $\delta_{i,j}$ are the pairwise distances in the original feature space.

General comments about metric multidimensional scaling:

* dissimilarity measure must be meaningful

* dimensionality reduction is performed such that the error in the sample pairwise distance is minimized

* there is a variant known as Nonmetric Multidimensional Scaling for ordinal features (categorical with ordering).

#### Checking Multidimensional Scaling Results

The multidimensional scaling approach minimizes the square difference of the pairwise distances between all of the data samples and eachother between the projected, lower dimensional, and original feature space.

* **stress** is defined as:

\begin{equation}
Stress_P(x_1,\ldots,x_n) = \left( \sum_{i \ne j = 1,\ldots,n} \left( ||x_i - x_j|| - \delta_{i,j} \right)^2 \right)^{\frac{1}{2}}
\end{equation}

* it is also useful to visualize the scatterplot of projected vs. original pairwise distances

#### Comparison with Principal Component Analysis

Principal component analysis takes the covariance matrix ($m \times m$) between all the features and finds the orthogonal rotation such that the **variance is maximized** over the ordered principle components.

Multidimensional scaling takes the matrix of the pairwise distances ($n \times n$) between all the samples in feature space and finds the nonlinear projects such that the **error in the pairwise distances is minimized**.

#### Random Projection

An alternative multidimensional scaling that relies on an random ($p \times n$) projection matrix, with columns with unit length.

* first row is a random unit vector

* second row is orthogonal to the first row

* continue

This random approach works practically because of the The Johnson-Lindenstrauss lemma:

*In mathematics, the Johnson-Lindenstrauss lemma is a result concerning low-distortion embeddings of points from high-dimensional into low-dimensional Euclidean space. The lemma states that a small set of points in a high-dimensional space can be embedded into a space of much lower dimension in such a way that distances between the points are nearly preserved. The map used for the embedding is at least Lipschitz, and can even be taken to be an orthogonal projection.* - Wikipedia article.

#### Objective

In the Stochastic Machine Learning class, I want to provide hands-on experience with solving complicated subsurface modeling problems with data analytics, machine learning. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - unconv_MV_v4.csv at https://git.io/fhHLT.

There are examples below with GeostatsPy functions. You can go here to see a list of the available functions, https://git.io/fh4eX, and for other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # for randon numbers
import seaborn as sns                     # for matrix scatter plots
from scipy import linalg                  # for linear regression

from sklearn.random_projection import johnson_lindenstrauss_min_dim
from sklearn.random_projection import SparseRandomProjection
from sklearn.metrics.pairwise import euclidean_distances
```

Here's a couple of functions that I found online for metrics that we will calculate below.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('unconv_MV_v4.csv')      # load our data table
df['TOC'] = np.where(df['TOC']<0.0, 0.0, df['TOC']) # set TOC < 0.0 as 0.0, otherwise leave the same
```

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head(n=13)                             # we could also use this command for a table preview
```

      Well
      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod

      0
      1
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819

      1
      2
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063

      2
      3
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259

      3
      4
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854

      4
      5
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624

      5
      6
      14.53
      4.81
      2.69
      53.60
      0.94
      1.67
      4017.374438

      6
      7
      13.49
      3.60
      2.93
      63.71
      0.80
      1.85
      2952.812773

      7
      8
      11.58
      3.03
      3.25
      53.00
      0.69
      1.93
      2670.933846

      8
      9
      12.52
      2.72
      2.43
      65.77
      0.95
      1.98
      2474.048178

      9
      10
      13.25
      3.94
      3.71
      66.20
      1.14
      2.65
      2722.893266

      10
      11
      15.04
      4.39
      2.22
      61.11
      1.08
      1.77
      3828.247174

      11
      12
      16.19
      6.30
      2.29
      49.10
      1.53
      1.86
      5095.810104

      12
      13
      16.82
      5.42
      2.80
      66.65
      1.17
      1.98
      4091.637316

This dataset has features from 200 unconventional wells including:

0. well index
1. well average porosity (%)
2. permeability (mD)
3. accoustic impedance (kg/m2s*10^6)
4. brittness ratio (%)
5. total organic carbon (%)
6. vitrinite reflectance (%)
8. normalized initial production 90 day average (MCFPD).

Note, the dataset is synthetic, but has realistic ranges and general multivariate relationships.

Ranking features is really an effort to understand the features and their relationships with eachother.  We will start with basic data visualization and move to more complicated methods such are partial correlation and recursive feature elimination.

#### Summary Statistics

Let's check the summary statistics of our data.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Well
      200.0
      100.500000
      57.879185
      1.000000
      50.750000
      100.500000
      150.250000
      200.000000

      Por
      200.0
      14.991150
      2.971176
      6.550000
      12.912500
      15.070000
      17.402500
      23.550000

      Perm
      200.0
      4.330750
      1.731014
      1.130000
      3.122500
      4.035000
      5.287500
      9.870000

      AI
      200.0
      2.968850
      0.566885
      1.280000
      2.547500
      2.955000
      3.345000
      4.630000

      Brittle
      200.0
      48.161950
      14.129455
      10.940000
      37.755000
      49.510000
      58.262500
      84.330000

      TOC
      200.0
      0.991950
      0.478264
      0.000000
      0.617500
      1.030000
      1.350000
      2.180000

      VR
      200.0
      1.964300
      0.300827
      0.930000
      1.770000
      1.960000
      2.142500
      2.870000

      Prod
      200.0
      3864.407081
      1553.277558
      839.822063
      2686.227611
      3604.303507
      4752.637556
      8590.384044

Summary statistics are a critical first step in data checking.

* this includes the number of valid (non-null) values for each feature (count removes all np.NaN from the totals for each variable).

* we can see the general behavoirs such as central tendency, mean, and dispersion, variance.

* we can identify issue with negative values, extreme values, and values that are outside the range of plausible values for each property.

* We can also establish the feature ranges for plotting.  We could calculate the feature range directly from the data with code like this:

```p
Pormin = np.min(df['Por'].values)          # extract ndarray of data table column
Pormax = np.max(df['Por'].values)          # and calculate min and max
```

but, this would not result in easy to understand color bars and axis scales, let's pick convenient round numbers. We will also declare feature labels for ease of plotting.

```python
pormin = 6.0; pormax = 24.0; porname = 'Porosity (%)'; portitle = 'Porosity' # user specified min and max values, and labels for plotting
permmin = 0.0; permmax = 10; permname = 'Permeability (mD)'; permtitle = 'Permeability'
AImin = 1.0; AImax = 5.0; AIname = 'Acoustic Impedance (kg/m2s*10^6)'; AItitle = 'Acoustic Impedance'
brmin = 10.0; brmax = 85.0; brname = 'Brittleness Ratio (%)'; brtitle = 'Brittleness'
TOCmin = 0.0; TOCmax = 2.2; TOCname = 'Total Organic Carbon (%)'; TOCtitle = 'Total Organic Carbon'
VRmin = 0.9; VRmax = 2.9; VRname = 'Vitrinite Reflectance (%)'; VRtitle = 'Vitrinite Reflectance'
prodmin = 500.0; prodmax = 9000.0; prodname = 'Normalized Initial Production (MCFPD)'; prodtitle = 'Normalized Initial Production'
```

The data looks to be in pretty good shape and for brevity we skip outlier detection. Let's look at the distributions with a martix scatter plot from the Seaborn package.

```python
sns.pairplot(df,vars=['Por','Perm','AI','Brittle','TOC','VR','Prod'],markers='o',plot_kws={'alpha': 0.2})
plt.subplots_adjust(left=0.0, bottom=0.0, right=0.6, top=0.6, wspace=0.3, hspace=0.2)
plt.show()
```

#### Data Preparation

Let's make an ordinal feature from the continuous production:

1. low
2. medium
3. high
4. very high

production rates.  This will help us visualize the results as we proceed, we can look at wells with different levels of production projected into a variety of lower dimensional spaces with multidimensional scaling.

```python
bins = [0,2500,5000,7500,10000]                # assign the production bins (these are the fence posts)
labels = ['low', 'med', 'high', 'vhigh']       # assign the labels
category = pd.cut(df['Prod'],bins,labels=labels)     # make the 1D array with the labels for our data
df['tProd'] = category                                # add the new ordinal production feature to our DataFrames
df.head()

```

      Well
      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod
      tProd

      0
      1
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819
      low

      1
      2
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063
      med

      2
      3
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259
      med

      3
      4
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854
      high

      4
      5
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624
      med

Let's take a look at the matrix scatter plot of our 3 features and the production levels.

```python
sns.pairplot(df[['Por','AI','TOC','tProd']],markers='o',hue = 'tProd')
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1.5, wspace=0.3, hspace=0.2)
plt.show()
```

#### Multidimensional Scaling

The multidimensional scaling method follows the sample pattern as other scikit-learn methods, we instantiate, fit and then apply or transform.

Let's run multidimensional scaling on our subset of features ($m = 3$) and project to only 2 features ($p = 2$).

```python
from sklearn.manifold import MDS
n_components = 2
embedding = MDS(n_components=2)
MDS_transformed = embedding.fit_transform(df[['Por','AI','TOC']])
MDS_transformed.shape
```

    (200, 2)

The output is 2 multidimensional scaling components.  We have projected our 3 features to 2 features to minimize the error in pairwise distance between the samples.  Let's add the 2 components to our DataFrame.

```python
df['MDS1'] = MDS_transformed[:,0]
df['MDS2'] = MDS_transformed[:,1]
df.head()
```

      Well
      Por
      Perm
      AI
      Brittle
      TOC
      VR
      Prod
      tProd
      MDS1
      MDS2

      0
      1
      12.08
      2.92
      2.80
      81.40
      1.16
      2.31
      1695.360819
      low
      -1.463746
      2.531615

      1
      2
      12.38
      3.53
      3.22
      46.17
      0.89
      1.88
      3007.096063
      med
      -0.877417
      2.474143

      2
      3
      14.02
      2.59
      4.01
      72.80
      0.89
      2.72
      2531.938259
      med
      0.576820
      1.295791

      3
      4
      17.67
      6.75
      2.63
      39.81
      1.08
      1.88
      5288.514854
      high
      0.820377
      -2.573599

      4
      5
      17.52
      4.57
      3.18
      10.94
      1.51
      1.90
      2859.469624
      med
      1.061472
      -2.338752

Let's take a look at the samples projected into the new 2 dimensional feature space.

* note the rotation, translation is arbitrary in this space, only the sample pairwise distances are relevant.

```python
pairplot = sns.scatterplot(x = df['MDS1'],y = df['MDS2'],hue = df['tProd'],markers='o')
plt.subplots_adjust(left=0.0, bottom=0.0, right=1., top=1.3, wspace=0.3, hspace=0.2)
plt.show()
```

Some comments:

* the general relationship between samples is preserved.

* the general production transition from low to very high production is preserved.

Let's check our model:

* we will calculate the original and projected pairwise distances between all the samples

* we will cross plot the original vs the projects pairwise distances

* we will plot the distribution of the ratio between projects / original pariwise distances

```python
dists = euclidean_distances(df[['Por','AI','TOC']], squared=False).ravel()
nonzero = dists != 0   # select only non-identical samples pairs
dists = dists[nonzero]
projected_dists = euclidean_distances(MDS_transformed, squared=False).ravel()[nonzero]

plt.subplot(221)
plt.scatter(dists,projected_dists,c='red',alpha=0.2,edgecolor = 'black')
plt.arrow(0,0,200,200,width=0.02,color='black',head_length=0.0,head_width=0.0)
plt.xlim(0,15); plt.ylim(0,15)
plt.xlabel("Pairwise Distance: original space")
plt.ylabel("Pairwise Distance: projected space")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

rates = projected_dists / dists
print("Distance Ratio, mean: %0.2f, standard deviation %0.2f." % (np.mean(rates), np.std(rates)))

plt.subplot(222)
plt.hist(rates, bins=50, range=(0.5, 1.5),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Distance Ratio: projected / original")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplot(223)
plt.hist(dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Original Data")

plt.subplot(224)
plt.hist(projected_dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.7, top=2.3, wspace=0.2, hspace=0.3)
plt.show()
```

We projected to a 2 dimensional feature space and did a pretty good job preserving the pairwise distances between the samples.

#### Observing Specific Samples

Let's reduce the number of wells and actually post the well numbers and observe their spacings in the original and projected features spaces.

```python
df_subset = df.iloc[:30,:]
```

Let's look at out dataset, we will visualize the 3 possible scatter plots between the three features with the sample points labeled by well number (1 through 30).

```python
plt.subplot(131)
pairplot = sns.scatterplot(x = df_subset['Por'],y = df_subset['AI'],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset['Por'][i], df_subset['AI'][i]))

plt.subplot(132)
pairplot = sns.scatterplot(x = df_subset['AI'],y = df_subset['TOC'],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset['AI'][i], df_subset['TOC'][i]))

plt.subplot(133)
pairplot = sns.scatterplot(x = df_subset['Por'],y = df_subset['TOC'],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset['Por'][i], df_subset['TOC'][i]))

plt.subplots_adjust(left=0.0, bottom=0.0, right=2., top=1., wspace=0.3, hspace=0.2)
plt.show()
```

We can now see our 30 wells with their indices plots on each of the 3 possible bivariate plots.  Let's perform multidimensional scaling down to 2 components.

```python
from sklearn.manifold import MDS

embedding = MDS(n_components=2)
MDS_transformed = embedding.fit_transform(df_subset[['Por','AI','TOC']])
MDS_transformed.shape
```

    (30, 2)

Now let's visualize the wells in the projected feature space.

```python
plt.subplot(221)
pairplot = sns.scatterplot(x = df_subset['Por'],y = df_subset['AI'],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset['Por'][i], df_subset['AI'][i]))
pairplot.legend(loc='upper left')

plt.subplot(222)
pairplot = sns.scatterplot(x = df_subset['AI'],y = df_subset['TOC'],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset['AI'][i], df_subset['TOC'][i]))
pairplot.legend(loc='lower right')

plt.subplot(223)
pairplot = sns.scatterplot(x = df_subset['Por'],y = df_subset['TOC'],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (df_subset['Por'][i], df_subset['TOC'][i]))
pairplot.legend(loc='upper left')

plt.subplot(224)
pairplot = sns.scatterplot(x = MDS_transformed[:,0],y = MDS_transformed[:,1],hue = df_subset['tProd'],markers='o')
for i, txt in enumerate(df_subset['Well']):
    pairplot.annotate(txt, (MDS_transformed[i,0], MDS_transformed[i,1]))
pairplot.set_xlabel('MDS1'); pairplot.set_ylabel('MDS2')
pairplot.legend(loc='lower right')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.7, top=1.7, wspace=0.3, hspace=0.2)
plt.show()
```

Some observations:

* the transformation is not a rotation (like PCA)

* the intersample distance is approximately preserved

\begin{equation}
\delta_{i,j} = \left( (\delta Por_{i,j}^{2}) + (\delta AI_{i,j}^{2}) + (\delta TOC_{i,j}^{2}) \right)^{\frac{1}{2}}
\end{equation}

#### Applications

The main benefit with multiple dimensional scaling is the ability to inspect high dimensional feature spaces for relationships between samples.

* we may observe that specific wells cluster together
* we may observe systematic transitions

#### Random Projection

Let's demonstrate the use of random projection.

* given the low dimensionalilty you will see quite poor performance for some randon seeds

```python
from sklearn.random_projection import GaussianRandomProjection
n_components = 2; seed = 73058
rp = GaussianRandomProjection(n_components=n_components,random_state = seed)

dists = euclidean_distances(df[['Por','AI','TOC']], squared=False).ravel()
nonzero = dists != 0   # select only non-identical samples pairs
dists = dists[nonzero]

projected_data = rp.fit_transform(df[['Por','AI','TOC']])
projected_dists = euclidean_distances(projected_data, squared=False).ravel()[nonzero]

plt.subplot(221)
plt.scatter(dists,projected_dists,c='red',alpha=0.2,edgecolor = 'black')
plt.arrow(0,0,200,200,width=0.02,color='black',head_length=0.0,head_width=0.0)
plt.xlim(0,15); plt.ylim(0,15)
plt.xlabel("Pairwise Distance: original space")
plt.ylabel("Pairwise Distance: projected space")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

rates = projected_dists / dists
print("Distance Ratio, mean: %0.2f, standard deviation %0.2f." % (np.mean(rates), np.std(rates)))

plt.subplot(222)
plt.hist(rates, bins=50, range=(0., 2.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Distance Ratio: projected / original")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplot(223)
plt.hist(dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Original Data")

plt.subplot(224)
plt.hist(projected_dists, bins=50, range=(0., 15.),color = 'red', alpha = 0.2, edgecolor='k')
plt.xlabel("Pairwise Distance")
plt.ylabel("Frequency")
plt.title("Pairwise Distance: Projected to %d Components" % n_components)

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.7, top=2.3, wspace=0.2, hspace=0.3)
plt.show()
```

We should see improved performance (lower stress) for high dimensional problems (large $m$).

#### Comments

This was a basic demonstration of multidimensional scaling. A lot more could be done:

* comparison to principal components analysis
* use of a dataset with larger number of features
* use of a dataset with natural clusters in multivariate space

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Programación lineal

> La programación lineal es el campo de la optimización matemática dedicado a maximizar o minimizar (optimizar) funciones lineales, denominada función objetivo, de tal forma que las variables de dicha función estén sujetas a una serie de restricciones expresadas mediante un sistema de ecuaciones o inecuaciones también lineales.

**Referencias:**
- https://es.wikipedia.org/wiki/Programaci%C3%B3n_lineal
- https://docs.scipy.org/doc/scipy-0.18.1/reference/optimize.html

## 1. Apuntes históricos

- 1826:	Joseph Fourier anticipa la programación lineal. Carl Friedrich Gauss resuelve ecuaciones lineales por eliminación "gaussiana".
- 1902:	Gyula Farkas concibe un método para resolver sistemas de inecuaciones.
- Es hasta la Segunda Guerra Mundial que se plantea la programación lineal como un modelo matemático para planificar gastos y retornos, de modo que se reduzcan costos de guerra y aumentar pérdidas del enemigo. Secreto hasta 1947 (posguerra).
- 1947:	George Dantzig publica el algoritmo simplex y John von Neumann desarrolló la teoría de la dualidad. Se sabe que Leonid Kantoróvich también formuló la teoría en forma independiente.
- Fue usado por muchas industrias en la planificación diaria.

**Hasta acá, tiempos exponenciales de solución. Lo siguiente, tiempo polinomial.**

- 1979: Leonid Khachiyan, diseñó el llamado Algoritmo del elipsoide, a través del cual demostró que el problema de la programación lineal es resoluble de manera eficiente, es decir, en tiempo polinomial.
- 1984: Narendra Karmarkar introduce el método del punto interior para resolver problemas de programación lineal.

**Mencionar complejidad computacional.**

## 2. Motivación

Ya la clase pasada habíamos mencionado que cuando se quería optimizar una función de varias variables con restricciones, se podía aplicar siempre el método de Multiplicadores de Lagrange. Sin embargo, este método es computacionalmente muy complejo conforme crece el número de variables.

Por tanto, cuando la función a optimizar y las restricciones son de caracter lineal, los métodos de solución que se pueden desarrollar son computacionalmente eficientes, por lo que es útil realizar la distinción.

## 3. Problemas de programación lineal

### 3.1. Ejemplo básico
Una compañía produce dos productos ($X_1$ y $X_2$) usando dos máquinas ($A$ y $B$). Cada unidad de $X_1$ que se produce requiere 50 minutos en la máquina $A$ y 30 minutos en la máquina $B$. Cada unidad de $X_2$ que se produce requiere 24 minutos en la máquina $A$ y 33 minutos en la máquina $B$.

Al comienzo de la semana hay 30 unidades de $X_1$ y 90 unidades de $X_2$ en inventario. El tiempo de uso disponible de la máquina $A$ es de 40 horas y el de la máquina $B$ es de 35 horas.

La demanda para $X_1$ en la semana actual es de 75 unidades y de $X_2$ es de 95 unidades. La política de la compañía es maximizar la suma combinada de unidades de $X_1$ e $X_2$ en inventario al finalizar la semana.

Formular el problema de decidir cuánto hacer de cada producto en la semana como un problema de programación lineal.

#### Solución

Sean:
- $x_1$ la cantidad de unidades de $X_1$ a ser producidas en la semana, y
- $x_2$ la cantidad de unidades de $X_2$ a ser producidas en la semana.

Notar que lo que se quiere es maximizar $x_1+x_2$.

Restricciones:
1. El tiempo de uso disponible de la máquina $A$ es de 40 horas: $50x_1+24x_2\leq 40(60)\Rightarrow 50x_1+24x_2\leq 2400$.
2. El tiempo de uso disponible de la máquina $B$ es de 35 horas: $30x_1+33x_2\leq 35(60)\Rightarrow 30x_1+33x_2\leq 2100$.
3. La demanda para $X_1$ en la semana actual es de 75 unidades: $x_1+30\geq 75\Rightarrow x_1\geq 45\Rightarrow -x_1\leq -45$.
4. La demanda para $X_2$ en la semana actual es de 95 unidades: $x_2+90\geq 95\Rightarrow x_2\geq 5\Rightarrow -x_2\leq -5$.

Finalmente, el problema puede ser expresado en la forma explicada como:
\begin{equation}
\begin{array}{ll}
\min_{x_1,x_2} & -x_1-x_2 \\
\text{s. a. }  & 50x_1+24x_2\leq 2400 \\
               & 30x_1+33x_2\leq 2100 \\
               & -x_1\leq -45 \\
               & -x_2\leq -5,
\end{array}
\end{equation}

o, eqivalentemente
\begin{equation}
\begin{array}{ll}
\min_{\boldsymbol{x}} & \boldsymbol{c}^T\boldsymbol{x} \\
\text{s. a. }        & \boldsymbol{A}_{eq}\boldsymbol{x}=\boldsymbol{b}_{eq} \\
                     & \boldsymbol{A}\boldsymbol{x}\leq\boldsymbol{b},
\end{array}
\end{equation}
con
- $\boldsymbol{c}=\left[-1 \quad -1\right]^T$,
- $\boldsymbol{A}=\left[\begin{array}{cc}50 & 24 \\ 30 & 33\\ -1 & 0\\ 0 & -1\end{array}\right]$, y
- $\boldsymbol{b}=\left[2400\quad 2100\quad -45\quad -5\right]^T$.

Preferiremos, en adelante, la notación vectorial/matricial.

### 3.2. En general
De acuerdo a lo descrito anteriormente, un problema de programación lineal puede escribirse en la siguiente forma:

\begin{equation}
\begin{array}{ll}
\min_{x_1,\dots,x_n} & c_1x_1+\dots+c_nx_n \\
\text{s. a. }        & a^{eq}_{j,1}x_1+\dots+a^{eq}_{j,n}x_n=b^{eq}_j \text{ para } 1\leq j\leq m_1 \\
                     & a_{k,1}x_1+\dots+a_{k,n}x_n\leq b_k \text{ para } 1\leq k\leq m_2,
\end{array}
\end{equation}
donde:
- $x_i$ para $i=1,\dots,n$ son las incógnitas o variables de decisión,
- $c_i$ para $i=1,\dots,n$ son los coeficientes de la función a optimizar,
- $a^{eq}_{j,i}$ para $j=1,\dots,m_1$ e $i=1,\dots,n$, son los coeficientes de la restricción de igualdad,
- $a_{k,i}$ para $k=1,\dots,m_2$ e $i=1,\dots,n$, son los coeficientes de la restricción de desigualdad,
- $b^{eq}_j$ para $j=1,\dots,m_1$ son valores conocidos que deben ser respetados estrictamente, y
- $b_k$ para $k=1,\dots,m_2$ son valores conocidos que no deben ser superados.

Equivalentemente, el problema puede escribirse como

\begin{equation}
\begin{array}{ll}
\min_{\boldsymbol{x}} & \boldsymbol{c}^T\boldsymbol{x} \\
\text{s. a. }        & \boldsymbol{A}_{eq}\boldsymbol{x}=\boldsymbol{b}_{eq} \\
                     & \boldsymbol{A}\boldsymbol{x}\leq\boldsymbol{b},
\end{array}
\end{equation}
donde:
- $\boldsymbol{x}=\left[x_1\quad\dots\quad x_n\right]^T$,
- $\boldsymbol{c}=\left[c_1\quad\dots\quad c_n\right]^T$,
- $\boldsymbol{A}_{eq}=\left[\begin{array}{ccc}a^{eq}_{1,1} & \dots & a^{eq}_{1,n}\\ \vdots & \ddots & \vdots\\ a^{eq}_{m_1,1} & \dots & a^{eq}_{m_1,n}\end{array}\right]$,
- $\boldsymbol{A}=\left[\begin{array}{ccc}a_{1,1} & \dots & a_{1,n}\\ \vdots & \ddots & \vdots\\ a_{m_2,1} & \dots & a_{m_2,n}\end{array}\right]$,
- $\boldsymbol{b}_{eq}=\left[b^{eq}_1\quad\dots\quad b^{eq}_{m_1}\right]^T$, y
- $\boldsymbol{b}=\left[b_1\quad\dots\quad b_{m_2}\right]^T$.

**Nota:** el problema $\max_{\boldsymbol{x}}\boldsymbol{g}(\boldsymbol{x})$ es equivalente a $\min_{\boldsymbol{x}}-\boldsymbol{g}(\boldsymbol{x})$.

#### Bueno, y una vez planteado, ¿cómo se resuelve el problema?

Este problema está sencillo pues solo es en dos variables. La solución gráfica es válida.

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
```

```python
def res1(x1):
    return (2400-50*x1)/24
def res2(x1):
    return (2100-30*x1)/33
```

```python
x1 = np.linspace(40, 50)
r1 = res1(x1)
r2 = res2(x1)
```

```python
plt.figure(figsize = (8,6))
plt.plot(x1, res1(x1), 'b--', label = 'res1')
plt.plot(x1, res2(x1), 'r--', label = 'res2')
plt.plot([45, 45], [0, 25], 'k', label = 'res3')
plt.plot([40, 50], [5, 5], 'm', label = 'res4')
plt.fill_between(np.array([45.0, 45.6]), res1(np.array([45.0, 45.6])), 5*np.ones(2))
plt.text(44,4,'$(45,5)$',fontsize=10)
plt.text(45.1,6.35,'$(45,6.25)$',fontsize=10)
plt.text(45.6,4,'$(45.6,5)$',fontsize=10)
plt.legend(loc = 'best')
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.axis([44, 46, 4, 7])
plt.show()
```

**Actividad.** Mónica hace aretes y cadenitas de joyería. Es tan buena, que todo lo que hace lo vende.

Le toma 30 minutos hacer un par de aretes y una hora hacer una cadenita, y como Mónica también es estudihambre, solo dispone de 10 horas a la semana para hacer las joyas. Por otra parte, el material que compra solo le alcanza para hacer 15 unidades (el par de aretes cuenta como unidad) de joyas por semana.

La utilidad que le deja la venta de las joyas es \$15 en cada par de aretes y \$20 en cada cadenita.

¿Cuántos pares de aretes y cuántas cadenitas debería hacer Mónica para maximizar su utilidad?

Formular el problema en la forma explicada y obtener la solución gráfica (puede ser a mano).

**Diez minutos: quien primero lo haga, pasará a explicarlo al tablero y le subiré la nota de alguna tarea a 100. Debe salir a explicar el problema en el pizarrón.**

## 5. ¿Cómo se resuelve en python?

### 5.1 Librería `SciPy`

`SciPy` es un softwar de código abierto basado en `Python` para matemáticas, ciencia e ingeniería.

En particular, los siguientes son algunos de los paquetes básicos:
- `NumPy`
- **Librería `SciPy`**
- `SymPy`
- `matplotlib`
- `pandas`

La **Librería `SciPy`** es uno de los paquetes principales y provee varias rutinas numéricas eficientes. Entre ellas, para integración numérica y optimización.

En esta clase, y en lo que resta del módulo, estaremos utilizando el módulo `optimize` de la librería `SciPy`.

**Importémoslo**

```python
# Importar el módulo optimize de la librería scipy

```

El módulo `optimize` que acabamos de importar contiene varias funciones para optimización y búsqueda de raices ($f(x)=0$). Entre ellas se encuentra la función `linprog`

```python
# Función linprog del módulo optimize

```

la cual resuelve problemas como los que aprendimos a plantear.

### 5.2 Solución del ejemplo básico con linprog
Ya hicimos la solución gráfica. Contrastemos con la solución que nos da `linprog`...

```python
# Importar numpy para crear las matrices

```

```python
# Crear las matrices para resolver el problema

```

```python
# Mostrar las matrices

```

```python
# Resolver utilizando linprog

```

```python
# Mostrar el resultado

```

```python
# Extraer el vector solución

```

**Conclusión**
-
-

**Otra forma:** poner las cotas de las variables a parte

```python
# Escribir matrices y cotas

```

```python
# Resolver

```

```python
# Mostrar el resultado

```

**Actividad.** Resolver el ejemplo de Mónica y sus tiliches con `linprog`

```python
# Resolver acá

```

```python

```

```python

```

```python

```

         fun: -250.0
     message: 'Optimization terminated successfully.'
         nit: 2
       slack: array([ 0.,  0.])
      status: 0
     success: True
           x: array([ 10.,   5.])

## 6. Problema de transporte 1
- **Referencia**: https://es.wikipedia.org/wiki/Programaci%C3%B3n_lineal

Este es un caso curioso, con solo 6 variables (un caso real de problema de transporte puede tener fácilmente más de 1.000 variables) en el cual se aprecia la utilidad de este procedimiento de cálculo.

Existen tres minas de carbón cuya producción diaria es:
- la mina "a" produce 40 toneladas de carbón por día;
- la mina "b" produce 40 t/día; y,
- la mina "c" produce 20 t/día.

En la zona hay dos centrales termoeléctricas que consumen:
- la central "d" consume 40 t/día de carbón; y,
- la central "e" consume 60 t/día.

Los costos de mercado, de transporte por tonelada son:
- de "a" a "d" = 2 monedas;
- de "a" a "e" = 11 monedas;
- de "b" a "d" = 12 monedas;
- de "b" a "e" = 24 monedas;
- de "c" a "d" = 13 monedas; y,
- de "c" a "e" = 18 monedas.

Si se preguntase a los pobladores de la zona cómo organizar el transporte, tal vez la mayoría opinaría que debe aprovecharse el precio ofrecido por el transportista que va de "a" a "d", porque es más conveniente que los otros, debido a que es el de más bajo precio.

En este caso, el costo total del transporte es:
- transporte de 40 t de "a" a "d" = 80 monedas;
- transporte de 20 t de "c" a "e" = 360 monedas; y,
- transporte de 40 t de "b" a "e" = 960 monedas,

Para un total 1.400 monedas.

Sin embargo, formulando el problema para ser resuelto por la programación lineal con
- $x_1$ toneladas transportadas de la mina "a" a la central "d"
- $x_2$ toneladas transportadas de la mina "a" a la central "e"
- $x_3$ toneladas transportadas de la mina "b" a la central "d"
- $x_4$ toneladas transportadas de la mina "b" a la central "e"
- $x_5$ toneladas transportadas de la mina "c" a la central "d"
- $x_6$ toneladas transportadas de la mina "c" a la central "e"

se tienen las siguientes ecuaciones:

Restricciones de la producción:

- $x_1 + x_2 \leq 40$
- $x_3 + x_4 \leq 40$
- $x_5 + x_6 \leq 20$

Restricciones del consumo:

- $x_1 + x_3 + x_5 \geq 40$
- $x_2 + x_4 + x_6 \geq 60$

La función objetivo será:

$$\min_{x_1,\dots,x_6}2x_1 + 11x_2 + 12x_3 + 24x_4 + 13x_5 + 18x_6$$

Resolver con `linprog`

```python
# Matrices y cotas

```

```python
# Resolver

```

```python
# Mostrar resultado

```

**Conclusión**

-

## 7. Optimización de inversión en bonos

**Referencia:**

```python
from IPython.display import YouTubeVideo
YouTubeVideo('gukxBus8lOs')
```

El objetivo de este problema es determinar la mejor estrategia de inversión, dados diferentes tipos de bono, la máxima cantidad que puede ser invertida en cada bono, el porcentaje de retorno y los años de madurez. También hay una cantidad fija de dinero disponible ($\$750,000$). Por lo menos la mitad de este dinero debe ser invertido en bonos con 10 años o más para la madurez. Se puede invertir un máximo del $25\%$ de esta cantidad en cada bono. Finalmente, hay otra restricción que no permite usar más de $35\%$ en bonos de alto riesgo.

Existen seis (6) opciones de inversión con las letras correspondientes $A_i$

1. $A_1$:(Tasa de retorno=$8.65\%$; Años para la madurez=11, Riesgo=Bajo)
1. $A_2$:(Tasa de retorno=$9.50\%$; Años para la madurez=10, Riesgo=Alto)
1. $A_3$:(Tasa de retorno=$10.00\%$; Años para la madurez=6, Riesgo=Alto)
1. $A_4$:(Tasa de retorno=$8.75\%$; Años para la madurez=10, Riesgo=Bajo)
1. $A_5$:(Tasa de retorno=$9.25\%$; Años para la madurez=7, Riesgo=Alto)
1. $A_6$:(Tasa de retorno=$9.00\%$; Años para la madurez=13, Riesgo=Bajo)

Lo que se quiere entonces es maximizar el retorno que deja la inversión.

Este problema puede ser resuelto con programación lineal. Formalmente, puede ser descrito como:

$$\max_{A_1,A_2,...,A_6}\sum^{6}_{i=1} A_iR_i,$$

donde $A_i$ representa la cantidad invertida en la opción, y $R_i$ representa la tasa de retorno respectiva.

Plantear restricciones...

```python
# Matrices y cotas

```

```python
# Resolver

```

```python
# Mostrar resultado

```

Recordar que en el problema minimizamos $-\sum^{6}_{i=1} A_iR_i$. El rendimiento obtenido es entonces:

```python

```

**Conclusión**

-

## 8. Tarea

### 1. Diseño de la Dieta Óptima

Se quiere producir comida para gatos de la manera más barata, no obstante se debe también asegurar que se cumplan los datos requeridos de analisis nutricional. Por lo que se quiere variar la cantidad de cada ingrediente para cumplir con los estandares nutricionales. Los requisitos que se tienen es que en 100 gramos, se deben tener por lo menos 8 gramos de proteína y 6 gramos de grasa. Así mismo, no se debe tener más de 2 gramos de fibra y 0.4 gramos de sal.

Los datos nutricionales se pueden obtener de la siguiente tabla:

Ingrediente|Proteína|Grasa|Fibra|Sal
:----|----
Pollo|  10.0%|08.0%|00.1%|00.2%
Carne|  20.0%|10.0%|00.5%|00.5%
Cordero|15.0%|11.0%|00.5%|00.7%
Arroz|  00.0%|01.0%|10.0%|00.2%
Trigo|  04.0%|01.0%|15.0%|00.8%
Gel|    00.0%|00.0%|00.0%|00.0%

Los costos de cada producto son:

Ingrediente|Costo por gramo
:----|----
Pollo|$\$$0.013
Carne|$\$$0.008
Cordero|$\$$0.010
Arroz|$\$$0.002
Trigo|$\$$0.005
Gel|$\$$0.001

Lo que se busca optimizar en este caso es la cantidad de productos que se debe utilizar en la comida de gato, para simplificar la notación se van a nombrar las siguientes variables:

$x_1:$ Gramos de pollo
$x_2:$ Gramos de carne
$x_3:$ Gramos de cordero
$x_4:$ Gramos de arroz
$x_5:$ Gramos de trigo
$x_6:$ Gramos de gel

Con los datos, se puede plantear la función objetivo, está dada por la siguiente expresión:

$$\min 0.013 x_1 + 0.008 x_2 + 0.010 x_3 + 0.002 x_4 + 0.005 x_5 + 0.001 x_6$$

Las restricciones estarían dadas por el siguiente conjunto de ecuaciones:

$x_1+x_2+x_3+x_4+x_5+x_6=100$

$(10.0 x_1+ 20.0 x_2+ 15.0 x_3+ 00.0 x_4+ 04.0 x_5+ 00.0 x_6)/100 \geq 8.0$

$(08.0 x_1+ 10.0 x_2+ 11.0 x_3+ 01.0 x_4+ 01.0 x_5+ 00.0 x_6)/100 \geq 6.0$

$(00.1 x_1+ 00.5 x_2+ 00.5 x_3+ 10.0 x_4+ 15.0 x_5+ 00.0 x_6)/100 \leq 2.0$

$(00.2 x_1+ 00.5 x_2+ 00.7 x_3+ 00.2 x_4+ 00.8 x_5+ 00.0 x_6)/100 \leq 0.4$

La primer condición asegura que la cantidad de productos que se usará cumple con los 100 gramos. Las siguientes sólo siguen los lineamientos planteados para cumplir con los requisitos nutrimentales.

### 2. Otro problema de transporte

Referencia: https://relopezbriega.github.io/blog/2017/01/18/problemas-de-optimizacion-con-python/

Supongamos que tenemos que enviar cajas de cervezas de 2 cervecerías (Modelo y Cuauhtémoc Moctezuma) a 5 bares de acuerdo al siguiente gráfico:

Asimismo, supongamos que nuestro gerente financiero nos informa que el costo de transporte por caja de cada ruta se conforma de acuerdo a la siguiente tabla:

```python
import pandas as pd
info = pd.DataFrame({'Bar1': [2, 3], 'Bar2': [4, 1], 'Bar3': [5, 3], 'Bar4': [2, 2], 'Bar5': [1, 3]}, index = ['CerveceriaA', 'CerveceriaB'])
info
```

      Bar1
      Bar2
      Bar3
      Bar4
      Bar5

      CerveceriaA
      2
      4
      5
      2
      1

      CerveceriaB
      3
      1
      3
      2
      3

Y por último, las restricciones del problema, van a estar dadas por las capacidades de oferta y demanda de cada cervecería (en cajas de cerveza) y cada bar, las cuales se detallan en el gráfico de más arriba.

Sean:
- $x_i$ cajas transportadas de la cervecería A al Bar $i$,
- $x_{i+5}$ cajas transportadas de la cervecería B al Bar $i$.

La tarea consiste en plantear el problema de minimizar el costo de transporte de la forma vista y resolverlo con `linprog`.

Deben crear un notebook de jupyter (archivo .ipynb) y llamarlo Tarea4_ApellidoNombre, y subirlo a moodle.

**Definir fecha**

Created with Jupyter by Esteban Jiménez Rodríguez.


## Question: Forgetting about power laws, do eqns 17.x actually describe a well-defined distribution?

```python
# bloody dependencies
import matplotlib.pyplot as plt
from nltk.corpus import gutenberg
import numpy as np
import pandas as pd
from scipy.special import zeta, zetac
from scipy.optimize import fsolve

# custom classes
from legomena import Corpus, LogModel
```

### Choosing a Book

Select a book for use as our fitting corpus.

```python
# # moby dick
# words = gutenberg.words("melville-moby_dick.txt")
# corpus = Corpus(words)
# corpus.dimension = 20
# TTR = corpus.TTR
# model = LogModel().fit(TTR.m_tokens, TTR.n_types)
# print("Optimum (M_z,N_z) =", model.params)

# plot TTR curve
plt.plot(TTR.m_tokens, TTR.n_types)
plt.scatter(*model.params, color="red")
plt.title("Type-Token Relation")
plt.xlabel("tokens")
plt.ylabel("types")
plt.show()

# take an optimum sample
optimum = corpus.sample(model.M_z)

# plot WFD
xmax = 50
H = np.sum(np.arange(xmax))
df = optimum.fdist.reset_index().head(xmax)
df["pred"] = optimum.N/df["rank"]
plt.bar(df["rank"], df["freq"])
plt.plot(df["rank"], df["pred"], color="red")
plt.title("Word Frequency Distribution")
plt.xlabel("rank")
plt.ylabel("frequency")
plt.show()

# plot the k-vector
i = np.arange(1,50)
plt.bar(i, optimum.k[i])
plt.plot(i, optimum.N/i/(i+1), color = 'r')
plt.title("n-Legomena Distribution")
plt.xlabel("n = word occurences")
plt.ylabel("words appearing n times")
plt.show()
```

## The $n$-Legomena Curves

Each $n$-legomena model curve can be derived from the TTR curve, resulting in the following series of equations for ascending values of $n = 0, 1, 2, ...$. They are already "horizontally" normalized such that $x = m/M_z$ and $k_n$ gives the _count_ of the $n$th legomena. By "vertically" normalizing, setting $y(x) = E(x)/N_z, k'_n = k_n/N_z$ we'll obtain legomena _fractions_ rather than counts. Therefore, this series of equations answers the question "what fraction of types $k'_n$ do we expect to observe given $x$ fraction of tokens?"

$$
\begin{equation}
y(x) = \frac{\ln(x)x}{x-1} \tag{17}\label{eq:17}
\end{equation}
$$

$$
\begin{equation}
k'_0(x) = 1 - y(x) = \frac{x-\ln(x)x-1}{x-1} \tag{17.0}\label{eq:17.0}
\end{equation}
$$

$$
\begin{equation}
k'_1(x) = \frac{x^2-\ln(x)x-x}{(x-1)^2} \tag{17.1}\label{eq:17.1}
\end{equation}
$$

$$
\begin{equation}
k'_2(x) = \frac{x^3-2\ln(x)x^2-x}{2(x-1)^3} \tag{17.2}\label{eq:17.2}
\end{equation}
$$

$$
\begin{equation}
k'_3(x) = \frac{2x^4+3x^3-6\ln(x)x^3-6x^2+x}{6(x-1)^4} \tag{17.3}\label{eq:17.3}
\end{equation}
$$

$$
\begin{equation}
k'_4(x) = \frac{3x^5+10x^4-12x^4\ln(x)-18x^3+6x^2-x}{12(x-1)^5} \tag{17.4}\label{eq:17.4}
\end{equation}
$$

$$
\begin{equation}
k'_5(x) = \frac{12x^6+65x^5-60x^5\ln(x)-120x^4+60x^3-20x^2+3x}{60(x-1)^6} \tag{17.5}\label{eq:17.5}
\end{equation}
$$

We've already determined one useful ratio, the hapax ratio, in the original paper: $H(x) = \frac{k_1(x)}{E(x)} = \frac{k'_1(x)}{y(x)} = \frac{1}{\ln(x)}+\frac{1}{1-x}$. Let's exhaust some others...

## General Form

The ratio of $n$-legomena with respect to types as the corpus grows follows this beautiful and mysterious formula:

$$
y(x) = \phi\bigg(\frac{x-1}{x}, 1, 1\bigg) \\
\hat{k}_0(x) = 1 - y(x) = 1 - \phi\bigg(\frac{x-1}{x}, 1, 1\bigg) \\
\hat{k}_n(x) = (-1)^{n+1}\frac{x^n}{n!}y^{(n)}(x) = \frac{1}{n} - \frac{1}{x}\phi\bigg(\frac{x-1}{x}, 1, n+1\bigg)
$$

## Hypothesis

1. Since $k_n(1) = \frac{1}{n(n+1)} \approx n^{-2}$ by definition, the question then becomes: Does $k_n(x) \approx n^{\gamma(x)}$ for some function $\gamma(x)$, thus "imitating" a power law distribution?

2. Is this distribution already discovered, and if so, what is its name, history, and context? Might there be some dualism between text analytics and some other field?


```python
# G3: Standard 3D Model

# Make SymPy available to this program:
import sympy
from sympy import *

# Make GAlgebra available to this program:
from galgebra.ga import *
from galgebra.mv import *
from galgebra.printer import Fmt, GaPrinter, Format
    # Fmt:       sets the way that a multivector's basis expansion is output.
    # GaPrinter: makes GA output a little more readable.
    # Format:    turns on latex printer.
from galgebra.gprinter import gFormat, gprint
gFormat()
```

```python
# g3: The geometric algebra G^3.
g3coords = (x,y,z) = symbols('x y z', real=True)
g3 = Ga('e', g=[1,1,1], coords=g3coords)
(ex, ey, ez) = g3.mv()

grad = g3.grad
from galgebra.dop import *
pdx = Pdop(x)
pdy = Pdop(y)
pdz = Pdop(z)
```

```python

```