# Quantitative exploratory data analysis
> A Summary of lecture "Statistical Thinking in Python (Part 1)", via datacamp

- toc: true
- badges: true
- comments: true
- author: Chanseok Kang
- categories: [Python, Datacamp, Data_Science, Statistics]
- image: images/petal-ecdf.png

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

sns.set()

df = pd.read_csv('./dataset/iris.csv')
renamed_columns = ['sepal length (cm)', 'sepal width (cm)',
                   'petal length (cm)', 'petal width (cm)', 'species']
df.columns = renamed_columns
versicolor_petal_length = df[df['species'] == 'Versicolor']['petal length (cm)']
setosa_petal_length = df[df['species'] == 'Setosa']['petal length (cm)']
virginica_petal_length = df[df['species'] == 'Virginica']['petal length (cm)']
versicolor_petal_width = df[df['species'] == 'Versicolor']['petal width (cm)']
```

## Introduction to summary statistics: The sample mean and median

### Computing means
The mean of all measurements gives an indication of the typical magnitude of a measurement. It is computed using ```np.mean()```.

```python
# compute the mean: mean_length_vers
mean_length_vers = np.mean(versicolor_petal_length)

# Print the result with some nice formatting
print('I. versicolor:', mean_length_vers, 'cm')
```

    I. versicolor: 4.26 cm

## Percentiles, outliers, and box plots

### Compute percentiles
In this exercise, you will compute the percentiles of petal length of Iris versicolor.

```python
# Specify array of percentiles: percentiles
percentiles = np.array([2.5, 25, 50, 75, 97.5])

# Compute percentiles: ptiles_vers
ptiles_vers = np.percentile(versicolor_petal_length, percentiles)

# Print the result
print(ptiles_vers)
```

    [3.3    4.     4.35   4.6    4.9775]

### Comparing percentiles to ECDF
To see how the percentiles relate to the ECDF, you will plot the percentiles of Iris versicolor petal lengths you calculated in the last exercise on the ECDF plot you generated in chapter 1.

 Note that to ensure the Y-axis of the ECDF plot remains between 0 and 1, you will need to rescale the percentiles array accordingly - in this case, dividing it by 100.

```python
def ecdf(data):
    """Compute ECDF for a one-dimensional array of measurements."""
    # Number of data points: n
    n = len(data)

    # x-data for the ECDF: x
    x = np.sort(data)

    # y-data for the ECDF: y
    y = np.arange(1, n + 1) / n

    return x, y
```

```python
x_vers, y_vers = ecdf(versicolor_petal_length)
```

```python
# Plot the ECDF
_ = plt.plot(x_vers, y_vers, '.')
_ = plt.xlabel('petal length (cm)')
_ = plt.ylabel('ECDF')

# Overlay percentiles as red diamonds
_ = plt.plot(ptiles_vers, percentiles/100, marker='D', color='red', linestyle='none')
plt.savefig('../images/petal-ecdf.png')
```

### Box-and-whisker plot
Making a box plot for the petal lengths is unnecessary because the iris data set is not too large and the bee swarm plot works fine. However, it is always good to get some practice. Make a box plot of the iris petal lengths. You have a pandas DataFrame, df, which contains the petal length data, in your namespace. Inspect the data frame df in the IPython shell using ```df.head()``` to make sure you know what the pertinent columns are.

For your reference, the code used to produce the box plot in the video is provided below:
```python
_ = sns.boxplot(x='east_west', y='dem_share', data=df_all_states)
_ = plt.xlabel('region')
_ = plt.ylabel('percent of vote for Obama')
```
In the IPython Shell, you can use ```sns.boxplot?``` or ```help(sns.boxplot)``` for more details on how to make box plots using seaborn.

```python
# Create box plot with Seaborn`s default settings
_ = sns.boxplot(x='species', y='petal length (cm)', data=df)

# Label the axes
_ = plt.xlabel('species')
_ = plt.ylabel('petal length (cm)')
```

## Variance and standard deviation

$$ variance = \dfrac{1}{n}\sum^{n}_{i=1}(x_i - \bar{x})^2 $$

### Computing the variance
It is important to have some understanding of what commonly-used functions are doing under the hood. Though you may already know how to compute variances, this is a beginner course that does not assume so. In this exercise, we will explicitly compute the variance of the petal length of Iris veriscolor using the equations discussed in the videos. We will then use ```np.var()``` to compute it.

```python
# Array of differnces to mean: differences
differences = np.array(versicolor_petal_length - np.mean(versicolor_petal_length))

# Square the differences: diff_sq
diff_sq = differences ** 2

# Compute the mean square differences: variance_explicit
variance_explicit = np.mean(diff_sq)

# Compute the variance using NumPy: variance_np
variance_np = np.var(differences)

# Print the results
print(variance_explicit, variance_np)
```

    0.21640000000000004 0.21640000000000004

### The standard deviation and the variance
As mentioned in the video, the standard deviation is the square root of the variance. You will see this for yourself by computing the standard deviation using ```np.std()``` and comparing it to what you get by computing the variance with ```np.var()``` and then computing the square root.

```python
# Compute the variance: variance
variance = np.var(versicolor_petal_length)

# Print the square root of the variance
print(np.sqrt(variance))

# Print the standard deviation
print(np.std(versicolor_petal_length))
```

    0.4651881339845204
    0.4651881339845204

## Covariance and the Pearson correlation coefficient

$$ covariance = \dfrac{1}{n}\sum^{n}_{i=1}(x_i - \bar{x})(y_i - \bar{y})$$
$$ \begin{align} \rho &= \text{Pearson correlation} = \dfrac{\text{covariance}}{(\text{std of x})(\text{std of y})} \\ &= \dfrac{\text{variability due to codependence}}{\text{independent variability}} \end{align}$$

### Scatter plots
When you made bee swarm plots, box plots, and ECDF plots in previous exercises, you compared the petal lengths of different species of iris. But what if you want to compare two properties of a single species? This is exactly what we will do in this exercise. We will make a scatter plot of the petal length and width measurements of Anderson's Iris versicolor flowers. If the flower scales (that is, it preserves its proportion as it grows), we would expect the length and width to be correlated.

For your reference, the code used to produce the scatter plot in the video is provided below:
```python
_ = plt.plot(total_votes/1000, dem_share, marker='.', linestyle='none')
_ = plt.xlabel('total votes (thousands)')
_ = plt.ylabel('percent of vote for Obama')
```

```python
# Make a scatter plot
_ = plt.plot(versicolor_petal_length, versicolor_petal_width, marker='.', linestyle='none')

# Label the axes
_ = plt.xlabel('petal length')
_ = plt.ylabel('petal width')
```

### Computing the covariance
The covariance may be computed using the Numpy function ```np.cov()```. For example, we have two sets of data ```x``` and ```y```, ```np.cov(x, y)``` returns a 2D array where entries ```[0,1]``` and ```[1,0]``` are the covariances. Entry ```[0,0]``` is the variance of the data in x, and entry ```[1,1]``` is the variance of the data in y. This 2D output array is called the covariance matrix, since it organizes the self- and covariance.

```python
# Compute the covariance matrix: covariance_matrix
covariance_matrix = np.cov(versicolor_petal_length, versicolor_petal_width)

# Print covariance matrix
print(covariance_matrix)

# Extract covariance of length and width of petals: petal_cov
petal_cov = covariance_matrix[0, 1]

# Print the length/width covariance
print(petal_cov)

```

    [[0.22081633 0.07310204]
     [0.07310204 0.03910612]]
    0.07310204081632653

### Computing the Pearson correlation coefficient
In this exercise, you will write a function, ```pearson_r(x, y)``` that takes in two arrays and returns the Pearson correlation coefficient. You will then use this function to compute it for the petal lengths and widths of I. versicolor.

```python
def pearson_r(x, y):
    """Compute Pearson correlation coefficient between two arrays

    Args:
        x: arrays
        y: arrays

    returns:
        r: int
    """
    # Compute correlation matrix: corr_mat
    corr_mat = np.corrcoef(x, y)

    # Return entry[0, 1]
    return corr_mat[0, 1]

# Compute Pearson correlation coefficient for I. versicolor: r
r = pearson_r(versicolor_petal_length, versicolor_petal_width)

# Print the result
print(r)
```

    0.7866680885228169


## Project 2: Model Analysis Project
$Raphael\space Tran$

$Antoine\space Jouan$

For this model project, we have an economic model in which one company is producing.

This company's production is affected by only two inputs: capital (K) and labour (L) which affects its output production.

We will find the optimal values  $K*$ and $L*$ for which the company minimises the costs of production and optimises its production.

We will solve this problem using the analytic and numerical method.

The production follows a Cobb-Douglas function :

$Q = K^{\alpha}L^{\beta}$

And Budget constraint :

$Y = wL + rK$

In this model there is also a rental rate of capital $r$ and a rate of wages $w$ ( no values are assigned to them but we will assign values during the project)
###### What is the cost-minimizing capital-to-labor ratio?
To answer this question we will use the Marginal Rate of Substituion technique $(MRS)$.

In a second part, we will tackle another optimisation issue but from a Cost perspective.

```python
import scipy as sp
import numpy as np
import sympy as sm
import matplotlib.pyplot as plt
from math import *
```

```python
# Variables for the first part of the project
P = sm.symbols('P')
Y = sm.symbols('Y')
K = sm.symbols('K')
L = sm.symbols('L')
r = sm.symbols('r')
w = sm.symbols('w')
alpha = sm.symbols('alpha')
beta = sm.symbols('beta')

# Second part
TC = sm.symbols('TC')
VC = sm.symbols('VC')
FC = sm.symbols('FC')
MC = sm.symbols('MC')
q = sm.symbols('q')
p = sm.symbols('p')
```

```python
# We create our Cobb-Douglas function
Cobb = K**alpha*L**beta
Cobb
```

$\displaystyle K^{\alpha} L^{\beta}$

```python
# We create our budget constraint
budget_constraint= sm.Eq(w*L + r*K, Y)
budget_constraint
```

$\displaystyle K r + L w = Y$

```python
# We find what is the value of L based on the budget constraint
L_p = sm.solve(budget_constraint,L)
```

```python
# Now we substitute its value inside the Cobb-Douglas function
Cobb_subs = Cobb.subs(L,L_p[0])
Cobb_subs
```

$\displaystyle K^{\alpha} \left(\frac{- K r + Y}{w}\right)^{\beta}$

```python
# We differenciate for K
diff_K = sm.diff(Cobb, K)
diff_K
```

$\displaystyle \frac{K^{\alpha} L^{\beta} \alpha}{K}$

```python
# We differenciate for L
diff_L = sm.diff(Cobb, L)
diff_L
```

$\displaystyle \frac{K^{\alpha} L^{\beta} \beta}{L}$

```python
# We compute the MRS
MRS = diff_K/diff_L
```

We now have the value of our Marginal rate of substitution which is :

### $MRS = \frac{L\alpha}{K\beta}$

At $q^*$ which maximises output under the budget constraint, the MRS is equal to the differentiate of the constraint :

Here, $w/r$

```python
# Solve for K
a = sm.solve(sm.Eq(MRS, w/r), K)
a
```

    [L*alpha*r/(beta*w)]

```python
# We now substitue it in the budget constraint and we compute L*
b_con = budget_constraint.subs(K, a[0])
Lstar = sm.solve(b_con, L)
Lstar
```

    [Y*beta*w/(alpha*r**2 + beta*w**2)]

```python
# We now compute K*
b = sm.solve(sm.Eq(MRS, w/r), L)
Kstar = sm.solve(budget_constraint.subs(L, b[0]), K)
Kstar
```

    [Y*alpha*r/(alpha*r**2 + beta*w**2)]

```python
#We now substitute the values of K* and L* in our Cobb-Douglas function
c1 = Cobb.subs(K, Kstar[0])
c2 = c1.subs(L, Lstar[0])
c2
```

$\displaystyle \left(\frac{Y \alpha r}{\alpha r^{2} + \beta w^{2}}\right)^{\alpha} \left(\frac{Y \beta w}{\alpha r^{2} + \beta w^{2}}\right)^{\beta}$

```python
# We create the frontier of production curve
sol_eq = sm.Eq(c2, Cobb)
prod_frontier = sm.solve(sol_eq, K)
prod_frontier # Equation of the production frontier curve
```

    [(L**(-beta)*(Y*alpha*r/(alpha*r**2 + beta*w**2))**alpha*(Y*beta*w/(alpha*r**2 + beta*w**2))**beta)**(1/alpha)]

###### Let's now solve this model analytically, by assigning values to our variables.

```python
# Setting up the parameters
# We have constant returns to scale, so alpha + beta = 1
# Our model economy relies more on K than L, so alpha > beta
alpha = 2/3
beta = 1/3

# Now for the rate of return of K and wages of L
r = 1
w = 1

# Total output of the firm is 100
Y = 100
```

```python
# Budget constraint curve
def b_constraint(L, r=r, w=w, Y=Y) :
    y = (Y-w*L)/r
    return y

# Production frontier curve
def prod(L, Y=Y, alpha=alpha, beta=beta) :
    y = (L**(-beta)*(Y*alpha*r/(alpha*r**2 + beta*w**2))**alpha*(Y*beta*w/(alpha*r**2 + beta*w**2))**beta)**(1/alpha)
    return y

# L* value
print(round(Y*beta*w/(alpha*r**2 + beta*w**2), 1))

# L- in the budget constraint (K*
print(round(b_constraint(Y*beta*w/(alpha*r**2+beta*w**2)), 1))

# L* in the production function (K*)
print(round(prod(Y*beta*w/(alpha*r**2 + beta*w**2)), 1))
```

    33.3
    66.7
    66.7

We have the value of $L*$ and the value of $K*$ when we replace $L*$ by its value in both the budget constraint and the production function.

We can see that the value are identical which means that we have the right value of $K*$.

Now that we have our optimum values for Labor and Capital, we can create a graph to have a clearer vison of the model.

```python
# setting the x - coordinates
x = np.arange(10, 70, 1)

# setting the corresponding y - coordinates
y1 = b_constraint(x)
y2 = prod(x)

# potting the points
plt.plot(x, y1)
plt.plot(x, y2)
plt.grid(True)
plt.xlabel('Labour (L)')
plt.ylabel('Capital (K)')
plt.title('Capital and Labour Equilibrium')
yi = b_constraint(Y*beta*w/(alpha*r**2+beta*w**2))
xi = Y*beta*w/(alpha*r**2 + beta*w**2)
plt.scatter(xi,yi)
plt.axvline(x=xi,color='orange',linestyle='--')
plt.axhline(y=yi,color='blue',linestyle='--')
plt.text(xi+1,yi+2,r'(K*,L*)')
axes = plt.gca()
axes.set_xlim([10,60])
axes.set_ylim([40,100])

plt.subplots_adjust(top=1.5,right=1.5)

# function to show the plot
plt.show()
```

This graph is showing the point where the production frontier ($orange\space curve$) is tangent to the budget constraint ($blue\space curve$).

This point is the values of $K$ and $L$ $(K*,L*)$ that the company should chose to maximize it's profit and minimise the costs.

From the results that we have, the company has to use two times more inputs of Capital ($K$) than of Labour ($L$) in order to maximise its profit

We can now move on to the second part of our project, the production optimisation for a firm, given a Cost function.

# Part 2

Our cost function is made up of two parts :

    First, the Variable costs are costs depending on the quantity of goods produced by the firm. As such, it is a function of q.

    Second is the Fixed costs. We are on the short term, so the firm has basic production costs that it cannot get rid of as it could on the long term.

```python
# Setting up a variable costs function
variable_cost = q**2 + q     # We need values as it is a function of q

# Fixed costs
fixed_cost = FC

# Total costs
total_cost = fixed_cost + variable_cost
total_cost
```

$\displaystyle FC + q^{2} + q$

We want to find the values $q^*$ and $p^*$, with which the firm reaches its optimal level of production.

To that end, we need to set up some sub-functions of the total cost :

The Average cost, Marginal cost and Average Variable cost.

```python
# Average cost
average_cost = total_cost / q
average_cost
```

$\displaystyle \frac{FC + q^{2} + q}{q}$

```python
# Marginal cost
marginal_cost = sm.diff(total_cost, q)
marginal_cost
```

$\displaystyle 2 q + 1$

```python
# Average Variable cost
average_vcost = variable_cost / q
sm.simplify(average_vcost)
```

$\displaystyle q + 1$

The point of finding out these functions is to be able to cross them.

   The quantity of the crossing point between the Marginal cost and the Average cost is the wanted $q^*$ value.

   The quantity of the crossing point between the Marginal cost and the Average Variable cost is the quantity at which the firm doesn't gain money but only loses their Fixed costs, which is acceptable in the short term.

 But first, let's find $q^*$ and $p^*$ analytically.

```python
f = sm.Eq(average_cost, marginal_cost)
qstar = sm.solve(f, q)
qstar
```

    [-sqrt(FC), sqrt(FC)]

We can't have a negative value for quantity of goods, so we keep the positive result.

```python
pstar = marginal_cost.subs(q, qstar[1])
pstar
```

$\displaystyle 2 \sqrt{FC} + 1$

Our results are then :

                        $(q^*, p^*) = (\sqrt{FC},\space2\sqrt{FC}+1)$

Now for the graphs.

```python
# Functions in order to graph

# For graphing purposes, we will now use FC=100
FC = 100

# Marginal cost
def m_cost(q) :
    y = 2*q + 1
    return y

# Average cost
def a_cost(q) :
    y = (FC + q**2 + q)/q
    return y

# Average Variable cost
def a_vcost(q) :
    y = q + 1
    return y

```

```python
# Graph

# Setting the x - coordinates
x = np.arange(1, 20, 0.1)

# Setting the functions
y1 = m_cost(x)
y2 = a_cost(x)
y3 = a_vcost(x)

# plotting the points
plt.plot(x, y1)
plt.plot(x, y2)
plt.plot(x, y3)
plt.grid(True)
plt.xlabel('Quantity (q)')
plt.ylabel('Price (p)')
plt.title('Costs')

# Highlight
xi = 10
yj = 2 * sqrt(100) + 1
plt.axvline(x=xi,color='black',linestyle='--')
plt.scatter(xi, yj)

# function to show the plot
plt.subplots_adjust(top=1.5,right=1.5)
plt.show()
```

Crossing the marginal and average cost curves gives us the optimal values for quantity $q^*$ and prices $p^*$.

$q^*$ is in the Cobb-Douglas model also the value for the Production frontier.

The Production frontier is the quantity a firm in pure and perfect competition wants to attain.
It yields a profit equal to 0, the highest the profit curve can reach.

Let's now bring up the profit curve to see if it matches.

```python
# Setting up profit
income = pstar * q
profit = income - total_cost
profit
```

$\displaystyle - FC - q^{2} + q \left(2 \sqrt{FC} + 1\right) - q$

```python
# Graphing Profit and the aggregate of curves

def profit_f(q):
    y = -FC - q**2 + q*(2*sqrt(FC)+1) - q
    return y

# Setting the x - coordinates
x = np.arange(5, 15, 0.1)

# Setting the functions
y = profit_f(x)
y1 = m_cost(x)
y2 = a_cost(x)
y3 = a_vcost(x)

# Plotting the profit curve
plt.subplot(221)
plt.plot(x, y)
plt.grid(True)
plt.xlabel('Quantity (q)')
plt.ylabel('Profit')
plt.title('Profit curve')

# Putting it all together
plt.subplot(222)
plt.plot(x, y)
plt.plot(x, y1)
plt.plot(x, y2)
plt.plot(x, y3)
plt.grid(True)
plt.xlabel('Quantity (q)')
plt.ylabel('Price (p)')
plt.title('Production Frontier')

# Setting up the coordinates
xi = 10
yi = 0
plt.axvline(x=xi,color='black',linestyle='--')
plt.axhline(y=yi,color='blue',linestyle='--')
plt.text(xi+0.2,yi+2,r'Optimal profit = 0')
plt.scatter(xi, yi)

yj = 2 * sqrt(FC) + 1
plt.axhline(y=yj,color='orange',linestyle='--')
plt.text(xi-1.2,yj+2,r'(q*, p*)')
plt.scatter(xi, yj)

# function to show the plot
plt.subplots_adjust(top=2.5,right=2)
plt.show()
```

The quantity $q^*$ is indeed the optimal choice for the company, since it yields a profit of 0.

This means that the amount of money earned by the firm is equal to the amount spent for production, investment etc.

    It then doesn't have any useless money lying around, which would be unthinkable if you want to compete in a pure and perfect competition environment.

# Conclusion

In this project, we solved the Cobb-Douglas model from the firm's point of view, first from a production function with a budget constraint then from a cost function.

    By solving it analytically then assigning real values in order to graph the results, we went about the problem step by step.

    As such, we hope you understood the intricates of the Cobb-Douglas model and found our project clear and interesting.

 ##### Thank you for reading.

```python

```


# 16 PDEs: waves in 2D

## 2D wave equation

The wave equation in 2D can be derived in the same fashion as for the 1D case and reads

$$
\left(\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}\right)u(x, y, t) - \frac{1}{c^2} \frac{\partial^2 u}{\partial t^2} = 0
$$

or in general

$$
\boldsymbol{\nabla}^2 u(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2}{\partial t^2}u(\mathbf{x}, t) = 0.
$$

For the case of a string or a membrane with areal density $\rho$ and constant tension $T$, the wave velocity is

$$
c = \sqrt{\frac{T}{\rho}}.
$$

## Numerical solution: Finite difference discretization

As before, we replace the derivatives with their finite difference approximations, hence the name *finite difference method* (FDM) for this approach.

\begin{align}
\left(\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}\right)u(x, y, t) &\approx
 \frac{u(x+\Delta x, y, t) - 2u(x, y, t) + u(x-\Delta x, y, t)}{\Delta x^2} \notag\\
 &+ \frac{u(x, y + \Delta y, t) - 2u(x, y, t) + u(x, y-\Delta y, t)}{\Delta y^2}
\end{align}

Setting $\Delta x = \Delta y = \delta$ (same spacing in $x$ and $y$) and $u_{ijn} := u(i\delta, j\delta, n\Delta t)$ we obtain the discretized 2D wave equation

$$
u_{i,j,n+1} = 2(1 - 2\beta^2)u_{i,j,n} - u_{i,j,n-1} + \beta^2(u_{i+1,j,n} + u_{i-1,j,n} + u_{i,j+1,n} + u_{i,j-1,n})\\
\beta := \frac{c}{\delta / \Delta t}
$$

## Implementation
Packaged as a function.

```python
import matplotlib.pyplot as plt
import numpy as np

plt.style.use('ggplot')
```

```python
def wave2D(L=0.5, N=50, Dt=1e-4, Nt=100, step=1,
           rho=1.5e-3, tension=5., aspect=1):

    Lx = L
    Ly = aspect*L
    Nx = N
    Ny = int(aspect*N)
    delta = L/N

    c = np.sqrt(tension/rho)

    beta = c*Dt/delta
    beta2 = beta**2

    print("c = {0} m/s".format(c))
    print("delta = {0} m,  Dt = {1} s, delta/Dt = {2} m/s".format(delta, Dt, delta/Dt))
    print("beta = {}".format(beta))
    print("beta * sqrt(2) = {}".format(beta * np.sqrt(2)))

    x = np.linspace(0, Lx, Nx+1)  # need N+1!
    y = np.linspace(0, Ly, Ny+1)
    # note: we use u[i*delta, j*delta] = u(x, y)
    #       (which is the transpose of the numpy row-column layout so we transpose X and Y)
    X, Y = np.meshgrid(x, y)
    X, Y = X.T, Y.T

    def gaussian(x, u0=0.05, x0=None, sigma=0.1*L):
        x0 = np.mean(x) if x0 is None else x0
        g = u0/np.sqrt(2*np.pi*sigma**2) * np.exp(-(x-x0)**2 / (2*sigma**2))
        return g

    # u[i*delta, j*delta] = u(x, y)
    # displacements at j-1, j, j+1
    u0 = np.zeros((Nx+1, Ny+1))
    u1 = np.zeros_like(u0)
    u2 = np.zeros_like(u0)

    # save array
    u_t = np.zeros((int(np.ceil(Nt/step)), Nx+1, Ny+1))

    # use a boolean mask to define regions
    inside = np.zeros_like(u0, dtype=bool)
    inside[1:-1, 1:-1] = True
    boundary = np.logical_not(inside)

    # boundary conditions
    def set_boundary_conditions(u):
        u[boundary] = 0

    set_boundary_conditions(u0)
    set_boundary_conditions(u1)
    set_boundary_conditions(u2)

    # initial conditions: velocity 0, i.e. no difference between y0 and y1
    u0[inside] = u1[inside] = (gaussian(X)*gaussian(Y))[inside]

    # save initial
    t_index = 0
    u_t[t_index, :, :] = u0
    if step == 1:
        t_index += 1
        u_t[t_index, :, :] = u1

    for nt in range(2, Nt):
        u2[1:-1, 1:-1] = 2*(1-2*beta2)*u1[1:-1, 1:-1] - u0[1:-1, 1:-1] + beta2*(u1[2:, 1:-1]
                                            + u1[:-2, 1:-1] + u1[1:-1, 2:] + u1[1:-1, :-2])
        u0[:, :], u1[:, :] = u1, u2

        if nt % step == 0 or nt == Nt-1:
            t_index += 1
            u_t[t_index, :, :] = u2
            print("Iteration {0:5d}".format(nt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(nt, nt*Dt))

    return u_t, X, Y, delta, Dt, step

```

In 2D, the Courant criterion for stability is

$$
\beta = \frac{c}{\delta/\Delta t} ≤ \frac{1}{\sqrt{2}}
$$

or

$$
\sqrt{2} \beta ≤ 1.
$$

If the solution becomes unstable, check $\sqrt{2}\beta$!

```python
u_t, X, Y, delta, Dt, step = wave2D(Nt=1000)
```

    c = 57.735026918962575 m/s
    delta = 0.01 m,  Dt = 0.0001 s, delta/Dt = 100.0 m/s
    beta = 0.5773502691896257
    beta * sqrt(2) = 0.816496580927726
    Completed   999 iterations: t=0.0999 s

```python
times = Dt * np.arange(0, len(u_t))
```

Look at the minimum and maximum values for each frame:

```python
fig = plt.figure(figsize=(6,3))
ax = fig.add_subplot(111)
ax.plot(times, u_t.min(axis=(1, 2)),
         times, u_t.max(axis=(1, 2)))
ax.set_xlabel("time (s)")
fig.tight_layout()
plt.show()
```

The min/max values do not appear to grow, indicating stable integration.

#### 2D Animation
For 2D animation to work in a Jupyter notebook, use

```python
%matplotlib widget
```

If no animations are visible, restart kernel and execute the `%matplotlib widget` cell as the very first one in the notebook.

We use `matplotlib.animation` to look at movies of our solution:

```python
import matplotlib.animation as animation
```

The `update_wave()` function simply re-draws our image for every `frame`.

```python
u_limits = 0.5*u_t.min(), 0.5*u_t.max()

fig1 = plt.figure(figsize=(5,5))
ax = fig1.add_subplot(111)
ax.set_aspect(1)

def update_wave(frame, data):
    global ax, Dt, u_limits, X, Y
    ax.clear()
    ax.set_xlabel("x")
    ax.set_ylabel("y")
    ax.contour(X, Y, data[frame], 30, cmap=plt.cm.viridis, norm=plt.Normalize(*u_limits))
    ax.contourf(X, Y, data[frame], 30, cmap=plt.cm.viridis, norm=plt.Normalize(*u_limits), alpha=0.8)
    ax.text(0.1, 0.9, "t = {0:g} s".format(frame*Dt), transform=ax.transAxes)

wave_anim = animation.FuncAnimation(fig1, update_wave, len(u_t), fargs=(u_t, ),
                                    interval=5, blit=False, repeat_delay=1000)

```

If you have `ffmpeg` installed then you can export to a MP4 movie file:

```python
wave_anim.save("square_drum_contours.mp4", fps=30, dpi=300)
```

#### 3D animation

```python
from mpl_toolkits.mplot3d import Axes3D
```

```python
def plot_wireframe(X, Y, u, limits=None, ax=None):
    if ax is None:
        fig = plt.figure()
        ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X, Y, u)
    ax.set_xlabel(r"position $x$ (m)")
    ax.set_ylabel(r"position $y$ (m)")
    ax.set_zlabel(r"displacement $u$ (m)")
    ax.set_zlim(limits)
    ax.figure.tight_layout()
    return ax
```

```python
plot_wireframe(X, Y, u_t[0])
```

```python
u_limits = 0.5*u_t.min(), 0.5*u_t.max()

fig2 = plt.figure(figsize=(7,7))
ax = fig2.add_subplot(111, projection="3d")
#ax.set_aspect(1)

def update_wire(frame, data):
    global ax, u_limits, X, Y
    ax.clear()
    plot_wireframe(X, Y, data[frame], limits=u_limits, ax=ax)
    ax.text(0.1, 0.9, 0.9, "t = {0:3.1f} ms".format(frame*Dt*1000), transform=ax.transAxes)

wire_anim = animation.FuncAnimation(fig2, update_wire, len(u_t), fargs=(u_t,),
                                    interval=30, blit=False, repeat_delay=1000)

```

```python
wire_anim.save("square_drum_wire.mp4", fps=30, dpi=300)
```

#### Making pretty graphics

We want to us a surface representation where the color indicates the displacement. We also want to show our animation on a [webpage](http://asu-compmethodsphysics-phy494.github.io/ASU-PHY494/2017/04/20/17_PDEs_waves/#leap-frog-algorithm-for-solving-the-wave-equation).

```python
import matplotlib.colors
```

```python
def plot_surface(X, Y, u, limits=None, ax=None, **kwargs):
    if ax is None:
        fig = plt.figure()
        ax = fig.add_subplot(111, projection="3d")
    ax.plot_surface(X, Y, u, rstride=1, cstride=1, shade=True,
                    cmap=plt.cm.coolwarm, norm=norm, **kwargs)
    ax.set_xlabel(r"$x$ (m)")
    ax.set_ylabel(r"$y$ (m)")
    ax.set_zlabel(r"$u$ (m)")

    # fix scaling see https://stackoverflow.com/a/19933125
    #scaling = np.array([getattr(ax, 'get_{}lim'.format(dim))() for dim in 'xyz'])
    scaling = np.array([X.min(), X.max(), Y.min(), Y.max(), min(limits), max(limits)])
    ax.auto_scale_xyz(*[[np.min(scaling), np.max(scaling)]]*3)

    ax.set_zlim(limits)

    ax.figure.tight_layout()
    return ax

u_limits = 0.5*u_t.min(), 0.5*u_t.max()
umax = 0.4*np.max(np.abs(u_limits))
norm = matplotlib.colors.Normalize(vmin=-umax, vmax=umax)

fig3 = plt.figure(figsize=(3.5, 3.5))
ax = fig3.add_subplot(111, projection="3d")
# ax.set_aspect(1)

def update_surface(frame, data):
    global ax, u_limits, X, Y
    ax.clear()
    plot_surface(X, Y, data[frame], limits=u_limits, ax=ax, alpha=0.9)
    ax.text(0.2, 0.9, 0.9, "$t = {0:3.1f}$ ms".format(frame*Dt*1000), transform=ax.transAxes)
    ax.figure.tight_layout()

wire_anim = animation.FuncAnimation(fig3, update_surface, len(u_t), fargs=(u_t,),
                                    interval=30, blit=False, repeat_delay=1000)

```

The whole video can be incorporated into an html page (uses the HTML5 `<video>` tag).

```python
tag = wire_anim.to_html5_video()
with open("square_drum_redblue_2d.html", "w") as html:
    html.write(tag)
```

or exported as a MPEG4 movie (uses `ffmpeg`):

```python
wire_anim.save("square_drum_surface.mp4", fps=30, dpi=300)
```

```python

```


```python
import numpy as np
import sympy as sy
sy.init_printing()
```

#  Similarity

If $A = PBP^{-1}$, we say $A$ is similar to $B$, decomposing $A$ into $PBP^{-1}$ is also called a similarity transformation.

If $n\times n$ matrices $A$ and $B$ are similar, they have the same eigenvalues.

The diagnoalization, which we will explain,  is a process of finding similar matrices.

#  Diagonalizable Matrix

Let $A$ be an $n\times n$ matrix. If there exists an $n\times n$ invertible matrix $P$ and a diagonal matrix $D$, such that

$$
A=PDP^{-1}
$$

then matrix $A$ is called a diagonalizable matrix.

And futher, the columns of $P$ are linearly independent eigenvectors of $A$, and its corresponding eigenvalues are on the diagonal of $D$. In other words, $A$ is diagonalizable if and only if the dimension of eigenspace basis is $n$.

It's easy to show why this equation holds.

Let

$$
P = \left[\begin{array}{llll}
{v}_{1} & {v}_{2} & \cdots & {v}_{n}
\end{array}\right]\\
$$

$$
D = \left[\begin{array}{cccc}
\lambda_{1} & 0 & \cdots & 0 \\
0 & \lambda_{2} & \cdots & 0 \\
\vdots & \vdots & & \vdots \\
0 & 0 & \cdots & \lambda_{n}
\end{array}\right]
$$

where $v_i, i \in (1, 2, ...n)$ is an eigenvector of $A$, $\lambda_i, i \in (1, 2, ...n)$ is an eigenvalue of $A$.

$$
AP = A\left[\begin{array}{llll}
{v}_{1} & {v}_{2} & \cdots & {v}_{n}
\end{array}\right]=\left[\begin{array}{llll}
A {v}_{1} & A {v}_{2} & \cdots & A {v}_{n}
\end{array}\right]
$$

$$P D=P\left[\begin{array}{cccc}
\lambda_{1} & 0 & \cdots & 0 \\
0 & \lambda_{2} & \cdots & 0 \\
\vdots & \vdots & & \vdots \\
0 & 0 & \cdots & \lambda_{n}
\end{array}\right]=\left[\begin{array}{lllll}
\lambda_{1} {v}_{1} & \lambda_{2} {v}_{2} & \cdots & \lambda_{n} {v}_{n}
\end{array}\right]$$

We know that $A{v}_i = \lambda_i{v}_i$, i.e.

$$
AP = PD
$$

Since $P$ has all independent eigenvectors, then

$$
A = PDP^{-1}
$$

#  Diagonalizing a Matrix

Consider a matrix

$$A=\left[\begin{array}{rrr}
1 & 3 & 3 \\
-3 & -5 & -3 \\
3 & 3 & 1
\end{array}\right]$$

diagonalize the matrix $A$.

Following these steps:

1. Find the eigenvalues of $A$
2. Find the eigenvectors of $A$
3. Construct $P$.
4. Construct $D$ from the corresponding columns of $P$.

```python
A = sy.Matrix([[1,3,3], [-3, -5, -3], [3,3,1]])
eig = sy.matrices.MatrixEigen.eigenvects(A)
eig
```

Construct $P$

```python
P = sy.zeros(3, 3)
P[:, 0] = eig[0][2][0]
P[:, 1] = eig[0][2][1]
P[:, 2] = eig[1][2][0]
P
```

Construct $D$

```python
D = sy.diag(eig[0][0], eig[0][0], eig[1][0])
D
```

We can verify if $PDP^{-1}=A$ holds:

```python
P * D * P.inv() == A
```

Of course we don't need to go through this process seperately. There is ```diagonalize``` method in SymPy.

```python
P, D = A.diagonalize()
```

```python
P
```

```python
D
```

Sometimes we just want to test if a matrix is diagonalizable, then use ```is_diagonalizable``` in SymPy.

```python
A.is_diagonalizable()
```

    True


# Algorithms Exercise 2

## Imports

```python
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
```

## Peak finding

Write a function `find_peaks` that finds and returns the indices of the local maxima in a sequence. Your function should:

* Properly handle local maxima at the endpoints of the input array.
* Return a Numpy array of integer indices.
* Handle any Python iterable as input.

```python
def find_peaks(x):
    a = np.array(x)
    l = []
    for i in range(len(a)):
        if i == 0 and a[i] > a[i+1]:
            l.append(i)
        elif i == len(a)-1 and a[i]> a[i-1]:
            l.append(i)
        elif i > 0 and i < len(a)-1:
            if a[i]>a[i-1] and a[i] > a[i+1]:
                l.append(i)
    return l
```

```python
p1 = find_peaks([2,0,1,0,2,0,1])
assert np.allclose(p1, np.array([0,2,4,6]))
p2 = find_peaks(np.array([0,1,2,3]))
assert np.allclose(p2, np.array([3]))
p3 = find_peaks([3,2,1,0])
assert np.allclose(p3, np.array([0]))
```

Here is a string with the first 10000 digits of $\pi$ (after the decimal). Write code to perform the following:

* Convert that string to a Numpy array of integers.
* Find the indices of the local maxima in the digits of $\pi$.
* Use `np.diff` to find the distances between consequtive local maxima.
* Visualize that distribution using an appropriately customized histogram.

```python
from sympy import pi, N
pi_digits_str = str(N(pi, 10001))[2:]
```

```python
a = [i for i in pi_digits_str]
b = find_peaks(a)
plt.hist(np.diff(b),10, align = 'left')
plt.title("Local Max");
plt.xlabel("Difference");
plt.ylabel("Freq");
```

```python
assert True # use this for grading the pi digits histogram
```


```python
from sympy import *
init_printing()
```

```python
x, y = symbols('x y', cls=Function)
t = symbols('t', real=True)
R = ((x(t).diff(t)**2 + y(t).diff(t)**2)**(3/2) \
     / (x(t).diff(t)*y(t).diff(t, 2) - y(t).diff(t)*x(t).diff(t,2)))
```

```python
a, t0, t1 = symbols('a t0 t1', real=True)

curve = Matrix([
    a*sqrt(pi)*fresnelc((t)/sqrt(pi)),
    a*sqrt(pi)*fresnels((t)/sqrt(pi))
])
display(curve)

K = 1 / R.subs({x(t): curve[0], y(t): curve[1]}).doit().simplify()
display(Eq(Symbol('K'), K))

L = integrate(sqrt(curve[0].diff(t)**2 + curve[1].diff(t)**2), (t, t0, t1)).simplify()
display(Eq(Symbol('L'), L))
```

```python
k0, k1, l = symbols('k0 k1 l', real=True)

with assuming(Q.positive_definite(k0), Q.positive_definite(k1), Q.is_true(k1 > k0)):
    sn = solve([
        Eq(K.subs(t, t0), abs(k0)),
        Eq(K.subs(t, t1), abs(k1)),
        Eq(L, l),
    ], (a, t0, t1), dict=True)
    display(sn[0])
```

```python
xy = MatrixSymbol('xy', 2, 1)
th = symbols('theta')
d = {
    k0: 0.0,
    k1: 5.9114369622925669e-02,
    l: 5.7125709578755304e+00,

    xy: Matrix([
        34.008391843292344e+00,
        -4.3421826556979193e+00,
    ]),
    th: 7.9998293398872988e-03,
}

d2 = {
    xy: Matrix([
        39.707082592335631e+00,
        -4.6174670591660050e+00,
    ]),
    th: -1.6084768620939371e-01,
}

d3 = {
    k0: -5.9114369622925669e-02,
    k1: -0.0,
    l: 5.7125709578755304e+00,
    xy: Matrix([
        4.5392930162567524e+01,
        -6.6292251856718263e+00
    ]),
    th: -5.1929742028899106e-01,
}
```

```python
# Take into account the curvature sign
curve2 = Matrix([[-1, 0], [0, -sign(k1)]]) * curve

plotting.plot_parametric(
    curve2[0].subs(sn[0]).subs(d),
    curve2[1].subs(sn[0]).subs(d),
    (t, sn[0][t0].subs(d), sn[0][t1].subs(d3))
)
```

```python
# Rotate by the heading
rot = Matrix([[cos(th), -sin(th)], [sin(th), cos(th)]])
curve3 = (rot * curve2)

plotting.plot_parametric(
    curve3[0].subs(sn[0]).subs(d),
    curve3[1].subs(sn[0]).subs(d),
    (t, sn[0][t0].subs(d), sn[0][t1].subs(d))
)

p0 = (curve3 + xy).subs({t: t0}).subs(sn[0]).subs(d).doit().evalf()
p1 = (curve3 + xy).subs({t: t1}).subs(sn[0]).subs(d).doit().evalf()
display(p0)
display(p1)
display(d2[xy])
```

```python
print(sn[0])
```

    {a: -k1*sqrt(-l/(Abs(k0) - Abs(k1)))/Abs(k1), t0: k1*sqrt(-l/(Abs(k0) - Abs(k1)))*Abs(k0)/Abs(k1), t1: k1*sqrt(-l/(Abs(k0) - Abs(k1)))}


# Señales de tiempo continuo

## Señales estándar

Algunas [señales](https://en.wikipedia.org/wiki/Signal_%28electrical_engineering%29) desempeñan un rol importante en las aplicaciones del **procesamiento de señales**, **telecomunicaciones**, **control automático** y se usan para caracterizar las propiedades de los sistemas Lineales e Invariantes en el Tiempo (LTI). A continuación se introducen e ilustran estas señales estándar.

### Exponencial compleja

La señal exponencial compleja en el dominio del tiempo $t$ se define matemáticamente con la [función exponencial compleja](https://en.wikipedia.org/wiki/Exponential_function#Complex_plane)

\begin{equation}
x(t) = e^{s t}
\end{equation}

donde $s = \sigma + j \omega$ describe una frecuencia compleja con $\sigma, \omega \in \mathbb{R}$ y $j$ tal que $(j^2=-1)$.

Esta señal se usa para representar señales armónicas. Usando la [formula de Euler](https://es.wikipedia.org/wiki/F%C3%B3rmula_de_Euler), la expresión anterior se puede reescribir como:

\begin{equation}
x(t) = e^{(\sigma + j \omega) t}  = e^{(\sigma) t} \cdot e^{(j \omega) t} = e^{\sigma t} \cos(\omega t) + j e^{\sigma t} \sin(\omega t)
\end{equation}

Ahora, las partes real e imaginaria de la señal son ponderados por funciones conseno y seno de frecuencia $\omega = 2 \pi f$. Para $t>0$, el peso en función del tiempo  $e^{\sigma t}$ es

* exponencialmente decreciente si $\sigma < 0$,
* constante (1) si $\sigma = 0$,
* exponencialmente creciente si  $\sigma > 0$,

Así, la señal exponencial compleja se usa para modelar señales armónicas con amplitudes constantes o exponencialmente crecientes o decrecientes.

**Ejemplo**

Las variables que se requieren son $t$, $\omega$ and $\sigma$ y se definen como variables simbólicas reales.

```python
import sympy as sym
import matplotlib.pyplot as plt
%matplotlib inline
sym.init_printing()

t, sigma, omega = sym.symbols('t sigma omega', real=True)
s = sigma + 1j*omega
x = sym.exp(s*t)
x
```

Ahora se definen valores para $s = \sigma + j \omega$.

```python
y = x.subs({omega: 5, sigma: -1})
plt.rcParams['figure.figsize'] = 9, 3
sym.plot(sym.re(y), (t, 0, 20), ylabel=r'Re{$e^{st}$}')
sym.plot(sym.im(y), (t, 0, 20), ylabel=r'Im{$e^{st}$}')
```

**Ejercicio**

* Defina valores para `omega` y `sigma` de manera que pueda crear señales decrecientes, constantes y crecientes con distinta frecuencia angular.

```python
## Escriba su código aquí

```

### Impulso de Dirac

Esta es una de las señales más importantes. Se usa para caracterizar y modelar los sistemas LTI. Se define según la [función delta de Dirac](https://es.wikipedia.org/wiki/Delta_de_Dirac). Se denota como $\delta(t)$ y se describe completamente por el efecto en otra función. A continuación las propiedades más relevantes:

1.  \begin{equation}
    \int_{-\infty}^{\infty} \delta(t) \cdot x(t) \cdot dt = x(0)
    \end{equation} donde $x(t)$ debe ser diferenciable en $t=0$. Esto implica que $\int_{-\infty}^{\infty} \delta(t) \cdot dt= 1$.

2.  \begin{equation}
    x(t) \cdot \delta(t) = x(0) \cdot \delta(t)
    \end{equation}
    donde $x(t)$ debe ser diferenciable en $t=0$.

3.  \begin{equation}
    a \cdot \delta(t) + b \cdot \delta(t) = (a+b) \cdot \delta(t)
    \end{equation}

4. \begin{equation}
    \delta(a t) = \frac{1}{|a|} \delta(t)
    \end{equation}
    donde $a \in \mathbb{R} \setminus 0$. Esto implica que el impulso de Dirac tiene simetría par.

5. \begin{equation}
    \int_{-\infty}^{\infty} \frac{d \delta(t)}{dt} \cdot x(t) \; dt = - \frac{d x(t)}{dt} \bigg\vert_{t = 0}
    \end{equation}

6.  \begin{equation}
    \int_{-\infty}^{\infty} \delta(\tau) \cdot x(t - \tau)  \, d\tau = x(t)
    \end{equation}

La última función es la [convolución](https://es.wikipedia.org/wiki/Convoluci%C3%B3n) y se tratará después.

**Ejemplo**

Primero se define la señal delta de Dirac.

```python
delta = sym.DiracDelta(t)
delta
```

Observe la primera propiedad con una función arbitraria $f(t)$.

```python
f = sym.Function('f')(t)
sym.integrate(delta*f, (t, -sym.oo, sym.oo))
```

**Ejercicio**

* Configure la función $\delta(t-\tau)$ y verifique el ejemplo anterior.

```python
## Escriba su código aquí
tau = sym.symbols('tau', real=True)
deltaTau = sym.DiracDelta(t-tau)
deltaTau*f
```

```python
sym.integrate(deltaTau*f, (t, -sym.oo, sym.oo))
```

### Escalón

La señal escalón se define según la [función escalón](https://es.wikipedia.org/wiki/Funci%C3%B3n_escal%C3%B3n_de_Heaviside)

\begin{equation}
\epsilon(t) = \begin{cases} 0 & t<0 \\  1 & t > 0 \end{cases}
\end{equation}

El escalón y el impulso de dirac se relacionan mediante:

\begin{equation}
\epsilon(t) = \int_{-\infty}^{t} \delta(\tau) \; d\tau
\end{equation}

**Ejemplo**

Se define y grafica la función escalón. `Sympy` llama Heaviside a la función y la denota como $\theta(t)$.

```python
step = sym.Heaviside(t)
step
```

```python
plt.rcParams['figure.figsize'] = 9, 3
sym.plot(step, (t, -2, 2), ylim=[-0.2, 1.2], ylabel=r'$\epsilon(t)$');
```

**Ejercicio**

Construya una señal armónica $\cos(\omega t)$ con $\omega=2$ que se "enciende" en el instante $t=0$.

\begin{equation}
x(t) = \cos(\omega t) \cdot \epsilon(t)
\end{equation}

```python
step = sym.Heaviside(t)
x = sym.cos(omega*t) * step
plt.rcParams['figure.figsize'] = 9, 3
sym.plot(x.subs(omega,2), (t, -10, 20), ylim=[-1.2, 1.2], ylabel=r'$x(t)$',xlabel=r'$t_{seg}$')
```

**Ejemplo**

Basado en la función escalón, se define un pulso rectangular de ancho $a$.

```python
a = sym.symbols('a', real=True)
t = sym.symbols('t', real=True)

def rect(t,a):
    Esc1 = sym.Heaviside(t+a)
    Esc2 = sym.Heaviside(t-a)
    return Esc1 - Esc2
r = rect(t,a)
plt.rcParams['figure.figsize'] = 9, 3
sym.plot(sym.Heaviside(t+a).subs(a,0.5), (t, -2, 2), ylim=[-1.2, 1.2], ylabel=r'$Esc1(t)$')
sym.plot(sym.Heaviside(t-a).subs(a,0.5), (t, -2, 2), ylim=[-1.2, 1.2], ylabel=r'$Esc2(t)$')
sym.plot(r.subs(a,0.5), (t, -2, 2), ylim=[-1.2, 1.2], ylabel=r'$rect(t)$')

```

**Ejercicio**

Defina y grafique una función armónica $\cos(\omega t)$ with $\omega=5$ que se "encienda" en $t=-1$ y se "apague" en $t=1$.

```python
## Escriba su código aquí

```

## Operaciones con señales

Las señales estándar permiten formar señales de estructuras más complejas mediante operaciones como la superposición, el desplazamiento temporal, el escalamiento, etc.

### Superposición

Considere que $x(t)$ es la superposición ponderada de las señales $x_\text{A}(t)$ y $x_\text{B}(t)$, es decir, la relación está dada por:

\begin{equation}
x(t) = A \cdot x_\text{A}(t) + B \cdot x_\text{B}(t)
\end{equation}

donde $A, B \in \mathbb{C}$.

**Ejercicio**

Suponga que $x(t) = A \cdot cos(\omega_{A}t) + B \cdot cos(\omega_{B}t)$. Puede observarse que se superponen dos señales coseno con distintas amplitudes y frecuencias.

```python
t = sym.symbols('t', real=True)

A = 0.25
omA = 13
B = .75
omB = 3.5

xA = sym.cos(omA*t)
xB = sym.cos(omB*t)
x = A*xA + B*xB

plt.rcParams['figure.figsize'] = 9, 3
pA=sym.plot(A*xA, (t, -5, 5), ylim=[-1.2, 1.2], xlabel=r't', show=True, line_color='red')
pB=sym.plot(B*xB, (t, -5, 5), ylim=[-1.2, 1.2], show=True, line_color='green')
pA.extend(pB)
pA.show()
sym.plot(x,(t, -5, 5), ylim=[-1.2, 1.2], show=True, line_color='blue')
```

### Desplazamiento temporal

Considere que $y(t)$ es una señal con la misma forma y valores de la señal $x_(t)$ pero están corridas en el tiempo.

\begin{equation}
y(t) = x(t-\tau)
\end{equation}
donde $\tau \in \mathbb{R}$.

En este caso, se dice que la señal $x(t)$ se ha:

* desplazado a la derecha (*retardo*) si $\tau > 0$
* desplazado a la izquierda (*adelanto*) si $\tau < 0$

**Ejercicio**

Suponga que se combinan 3 pulsos rectangulares para formar una "escalera" de pocos nivele.

\begin{equation}
x(t) = \text{rect}\left(t - \frac{1}{2} \right) + \frac{2}{3} \cdot \text{rect}\left(t-\frac{3}{2} \right) + \frac{1}{3} \cdot \text{rect} \left(t-\frac{5}{2} \right)
\end{equation}

```python
rect = sym.Heaviside(t + 1/2) - sym.Heaviside(t - 1/2)

xA = rect.subs(t, t-1/2)
xB = 2/3*rect.subs(t, t-3/2)
xC = 1/3*rect.subs(t, t-5/2)
x = xA + xB + xC

plt.rcParams['figure.figsize'] = 9, 3

sym.plot(rect, (t, -1, 5), ylim=[-0.2, 1.2], ylabel='$x(t)$');
sym.plot(xA, (t, -1, 5), ylim=[-0.2, 1.2], ylabel='$x(t)$');
sym.plot(xB, (t, -1, 5), ylim=[-0.2, 1.2], ylabel='$x(t)$');
sym.plot(xC, (t, -1, 5), ylim=[-0.2, 1.2], ylabel='$x(t)$');
sym.plot(x, (t, -1, 5), ylim=[-0.2, 1.2], ylabel='$x(t)$');
```

### Escalamiento temporal

Esta operación permite alterar la rapidez aparente del tiempo dada la forma de una señal. Se define como:

\begin{equation}
y(t) = x(a \cdot t)
\end{equation}

donde $a \in \mathbb{R}$.

En este caso, se dice que la señal $x(t)$ se ha:

* estirado si $ 0 < a < 1$
* comprimido si $ 1 < a $
* invertido si $ a < 0$

**Ejercicio**

Tome la señal anterior y aplique una escala temporal $a$ tal que $y(t) = x(a \cdot t)$.

```python

```

**Ejercicio**

Modifique el ejemplo anterior para que la señal tenga:
- Compresión temporal y retardo
- Inversión temporal y adelanto

Realice distintas pruebas con otras formas de señal.


# Variational Mixture of Gaussians

Gaussian mixture models are widely used to model complex intractable probability distributions using a mixture of Gaussians. In a [previous post](https://chandrusuresh.github.io/MyNotes/files/DensityEstimation/GaussianMixtureModels.html), GMMs were discussed together with a Maximum Likelihood approach to fit a GMM to the Palmer Penguins dataset using the EM algorithm.

Here we descrive the variational inference algorithm to the same dataset. The following plate model is assumed for the variational approximation.
The following plate model in Fig 10.5 from [1] is assumed.

The variational approach resolves many of the limitations of the MLE approach.

- Variational approach uses a Bayesian model that enables determining parameter uncertainty
- Prevents over-fitting of data to model
- The cardinality/dimensionality of the latent variable can be determined or inferred by the algorithm.

## Theory
Given a dataset for random variable $x$, we introduce a $K$ dimensional binary random variable $z$ having a 1-of-$K$ representation in which a particular element $z_k = 1$  with $z_i = 0 \text{ } \forall i\ne k$ i.e., $\sum{z_k} = 1$. Assume the dataset has $N$ points. The dataset of observed and latent variables are denoted by $X$ and $Z$ respectively.

The marginal distribution $p(Z,X)$ is defined as $ p(Z,X) = p(X|Z)\cdot p(Z)$.

If $p(z_k = 1) = \pi_k$ and $p(x|z_k=1) = \mathcal{N}(x|\mu_k,\Lambda_k^{-1})$ then, $$\begin{align} p(Z) &= \prod_{n=1}^N \prod_{k=1}^K \pi_k^{z_{nk}} \\
p(X|Z) &= \prod_{n=1}^N \prod_{k=1}^K \Bigg(\mathcal{N}(x|\mu_k,\Lambda_k^{-1})\Bigg)^{z_{nk}} \end{align}$$

The marginal distribution of $X$ is therefore,

$$ \begin{align}  p(X) &= \sum_{z,\pi,\mu,\Lambda} p(X,Z,\pi,\mu,\Lambda) \\
&= \sum_{z,\pi,\mu,\Lambda} p(X|Z,\mu,\Lambda) p(Z|\pi) p(\pi) p(\mu|\Lambda) p(\Lambda) \end{align}$$

We now consider a variational distribution that factorizes the latent variables and parameters as:

$$ q(Z,\pi,\mu,\Lambda) = q(Z)q(\pi,\mu,\Lambda)$$

### Conjugate Priors
We introduce conjugate priors for $\pi$,$\mu$,$\Lambda$ as follows.

For $\pi$ we choose a Dirichlet prior with the same parameter $\alpha_0$ for each component.
$$ p(\pi) = \text{Dir}(\pi|\alpha_0) = C(\alpha_0)\prod_{k=1}^K{\pi_k^{\alpha_0-1}}$$
where $C(\alpha_0)$ is the normalization constant.

For $\mu$ and $\Lambda$, a Gaussian-Wishart prior is chosen for the mean and precision of each component.
$$ \begin{align} p(\mu,\Lambda) &= p(\mu|\Lambda) p(\Lambda) \\
&= \prod_{k=1}^K\mathcal{N}\Big(\mu_k|m_0,(\beta_0\Lambda_k)^{-1})\Big) \mathcal{W}(\Lambda_k|W_0,\nu_0) \end{align}$$

### Optimal factor for $q(Z)$
The update equation for the latent variable is given by,
$$ \begin{align} \ln{q^*(Z)} &= \mathbb{E}_{\pi,\mu,\Lambda}\Big[ \ln{p(X,Z,\pi,\mu,\Lambda}\Big] \\
&= \mathbb{E}_{\pi,\mu,\Lambda}\Big[\ln\Big\{p(X|Z,\mu,\Lambda) p(Z|\pi) p(\pi) p(\mu|\Lambda) p(\Lambda) \Big\}\Big]\end{align}$$

By combining the terms not including $Z$ in the above expression into a constant term,
$$ \begin{align} \ln{q^*(Z)} &= \mathbb{E}_{\pi}\Big[\ln p(Z|\pi)\Big] + \mathbb{E}_{\mu,\Lambda}\Big[\ln{p(X|Z,\pi,\mu)}\Big] + \text{const.}\end{align}$$

$$ \begin{align}\mathbb{E}_{\pi}\Big[\ln p(Z|\pi)\Big] &= \mathbb{E}_{\pi}\Big[\sum_{n=1}^N{\sum_{k=1}^K{z_{nk}\pi_k}}\Big] \\
&= \sum_{n=1}^N{\sum_{k=1}^K{z_{nk}\mathbb{E}\Big[\pi_k\Big]}} \end{align}$$

$$ \begin{align}\mathbb{E}_{\mu,\Lambda}\Big[\ln{p(X|Z,\pi,\mu)}\Big] &= \mathbb{E}_{\mu,\Lambda}\Bigg[\sum_{n=1}^N{\sum_{k=1}^K{z_{nk}\Big(-\frac{D}{2}\ln(2\pi) + \frac{1}{2}\ln{|\Lambda_k|} \\
- \frac{1}{2}(x_n-\mu_k)^T \Lambda_k (x_n-\mu_k)\Big)\Bigg]}} \\
&= \sum_{n=1}^N{\sum_{k=1}^K{z_{nk}\Bigg(\frac{1}{2}\mathbb{E}\Big[\ln{|\Lambda_k|}\Big] - \frac{D}{2}\ln(2\pi) \\
- \frac{1}{2}\mathbb{E}\Big[(x_n-\mu_k)^T \Lambda_k (x_n-\mu_k)\Big]\Bigg)}} \end{align}$$

Substituting the above 2 expressions above,

$$ \begin{align} \ln{q^*(Z)} &= \sum_{n=1}^N{\sum_{k=1}^K{z_{nk} \rho_{nk}}} + \text{const.}\end{align}$$

where $\rho_{nk}$ is given by,
$$ \rho_{nk} = \mathbb{E}\Big[\pi_k\Big] + \frac{1}{2}\mathbb{E}\Big[\ln{|\Lambda_k|}\Big] - \frac{D}{2}\ln(2\pi) - \frac{1}{2}\mathbb{E}\Big[(x_n-\mu_k)^T \Lambda_k (x_n-\mu_k)\Big]$$

The distribution $q^*(Z)$ is given by,

$$ \Rightarrow q^*(Z) = \prod_{n=1}^N{\prod_{k=1}^K{r_{nk}^{z_{nk}}}}$$

where $r_{nk}$ are the responsibilities with
$$ r_{nk} = \frac{\rho_{nk}}{\sum_{j=1}^K\rho_{nj}}$$

For the discrete distribution $q^*(Z)$, we have $\mathbb{E}\Big[q(z_{nk})\Big] = r_{nk}$

The following provide expressions for the terms in the above expression for $\rho_{nk}$.
$$\begin{align} \mathbb{E}\Big[(x_n-\mu_k)^T \Lambda_k (x_n-\mu_k)\Big] &= D \beta_k^{-1} + \nu_k \mathbb{E}\Big[(x_n-m_k)^T W_k (x_n-m_k)\Big] \\ \mathbb{E}\Big[\ln{|\Lambda_k|}\Big] &= \sum_{i=1}^D \psi\Bigg( \frac{\nu_k+1-i}{2}\Bigg) + D \ln 2 + \ln |W_k| \\
\mathbb{E}\Big[\pi_k\Big] &= \psi(\alpha_k) - \psi(\hat{\alpha}) \end{align}$$

where $\psi(\cdot)$ is the digamma function and $\hat{\alpha} = \sum_k \alpha_k$.

### Optimal factor for $q(\pi,\mu,\Lambda)$
Given the expression for $q^*(Z)$ and $r_{nk}$, the following quantities are defined.

$$ \begin{align} N_k &= \sum_{n=1}^N{r_{nk}} \\
\bar{x}_k&= \frac{1}{N_k}\sum_{n=1}^N{r_{nk}x_n}\\
S_k &= \frac{1}{N_k}\sum_{n=1}^N{r_{nk}(x_n-\bar{x}_k)(x_n-\bar{x}_k)^T} \end{align}$$

Now the optimal factor $q(\pi,\mu,\Lambda)$ is given by

$$ \begin{align} \ln{q^*(\pi,\mu,\Lambda)} &=  \mathbb{E}_{Z}\Big[\ln\Big\{p(X|Z,\mu,\Lambda) p(Z|\pi) p(\pi) p(\mu|\Lambda) p(\Lambda) \Big\}\Big] \\
&= \ln{p(\pi)} + \mathbb{E}_{Z}\Bigg[\sum_{k=1}^K{\ln p(\mu_k,\Lambda_k)} + \ln p(Z|\pi) \\
&+ \sum_{n=1}^N{\sum_{k=1}^K{z_{nk}\ln \mathcal{N}(x_n|\mu_k,\Lambda_k)}} \Bigg] + \text{const.} \\
&= \ln{p(\pi)} + \mathbb{E}_{Z}\Big[\ln p(Z|\pi)\Big] + \sum_{k=1}^K{\ln p(\mu_k,\Lambda_k)}  \\
&+ \sum_{n=1}^N{\sum_{k=1}^K{\mathbb{E}_{Z}\Big[z_{nk}\Big]\ln \mathcal{N}(x_n|\mu_k,\Lambda_k)}} + \text{const.}\end{align}$$

The expression above factorizes into terms involving just $\pi$ and $\mu,\Lambda$ thereby implying independence of these variables. Note that this result is not based on any prior assumption but a direct result of the model.

$$ \begin{align} q^*(\pi,\mu,\Lambda) &= q(\pi) \prod_{k=1}^K{q(\mu,\Lambda)} \\
\text{where}\quad \ln q^*(\pi) &= \ln{p(\pi)} + \mathbb{E}_{Z}\Big[\ln p(Z|\pi)\Big]  + \text{const.}\\
\text{and}\quad \ln q^*(\mu,\Lambda) &= \sum_{k=1}^K{\ln p(\mu_k,\Lambda_k)}
+ \sum_{n=1}^N{\sum_{k=1}^K{\mathbb{E}_{Z}\Big[z_{nk}\Big]\ln \mathcal{N}(x_n|\mu_k,\Lambda_k)}} + \text{const.}\end{align}$$

### Optimal factor for $q(\pi)$
$$\begin{align} \ln q^*(\pi) &= \ln{p(\pi)} + \mathbb{E}_{Z}\Big[\ln p(Z|\pi)\Big]  + \text{const.} \\
&= (\alpha_0-1)\sum_{k=1}^K{\pi_k} + \sum_{n=1}^N{\sum_{k=1}^K{\mathbb{E}_{Z}[z_{nk}]\pi_k}} + \text{const.} \\
&= (\alpha_0-1)\sum_{k=1}^K{\pi_k} + \sum_{n=1}^N{\sum_{k=1}^K{r_{nk}\pi_k}} + \text{const.} \\
&= (\alpha_0-1)\sum_{k=1}^K{\pi_k} + \sum_{k=1}^K{N_k\pi_k} + \text{const.} \\ &= (\alpha_0+N_k-1)\sum_{k=1}^K{\pi_k} + \text{const.} \\ \end{align}$$

The posterior $q^*(\pi)$ is also a Dirichlet distribution with parameter $\alpha_0+N_k$. i.e. $q^*(\pi) = \text{Dir}(\pi|\alpha_0+N_k)$

### Optimal factor for $q(\mu,\Lambda)$

The derivation for the parameters of the posterior distribution for $q(\mu,\Lambda)$ is very involved, so only the results are presented here. More details can be found in section 10.2.1. in [1].

$$\begin{align} q(\mu_k,\Lambda_k) &= \mathcal{N}\Big(\mu_k|m_k,(\beta_k \Lambda_k)^{-1}\Big)\cdot \mathcal{W}(\Lambda_k|W_k,\nu_k) \\
\text{where} \quad \beta_k &= \beta_0 + N_k \\
m_k &= \frac{1}{\beta_k}(\beta_0 m_0 + N_k \bar{x}_k) \\
W_k^{-1} &= W_0^{-1} + N_k S_k + \frac{\beta_0 N_k}{\beta_0+N_k}(\bar{x}_k-m_0)(\bar{x}_k-m_0)^T \\
\nu_k &= \nu_0 + N_k\end{align}$$

## Example
The palmer penguins dataset released by [2] and obtained from [3] is used as an example. Two features - Flipper Length & Culmen Length are used as the features to cluster the dataset. We set K = 6 and demonstrate that 3 of these components are redundant while correctly identifying the 3 correct categories of penguins - Adelie, Chinstrap and Gentoo. This same dataset was used in the GMM section. The dataset is plotted below. This is the same dataset used to demonstrate [Gaussian Mixture Models using the EM algorithm](https://chandrusuresh.github.io/MyNotes/files/DensityEstimation/GaussianMixtureModels.html).

```python
import pandas as pd
import requests
import io
import numpy as np
from scipy.stats import multivariate_normal as gaussian
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from matplotlib.patches import Ellipse
import matplotlib.transforms as transforms
import matplotlib.colors as mcolors
from scipy.special import digamma
from scipy.stats import multivariate_normal as gaussian
from scipy.stats import wishart,dirichlet
from scipy.special import softmax as softmax

def getCSV(url):
    download = requests.get(url).content
    df = pd.read_csv(io.StringIO(download.decode('utf-8')))
    return df

file = "https://raw.githubusercontent.com/mcnakhaee/palmerpenguins/master/palmerpenguins/data/penguins-raw.csv"
df = getCSV(file)
```

```python
txt_labels = np.unique(df['Species'])
lbl = txt_labels[0]
fig,ax = plt.subplots(1,2,figsize=(10,5))
df_data = [None]*len(txt_labels)
img = mpimg.imread('../../img/lter_penguins.png')
ax[0].imshow(img)
ax[0].axis('off')
color = ['tomato','mediumorchid','seagreen','aqua','black','magenta']
for i,lbl in enumerate(txt_labels):
    df_data[i] = df[df['Species'] ==  lbl]
#     print(df_data[i].columns)
    ax[1].scatter(df_data[i]['Flipper Length (mm)'],df_data[i]['Culmen Length (mm)'],color=color[i])
# ax[1].axis('off')
ax[1].set_xlabel('Flipper Length');
ax[1].set_ylabel('Culmen Length');
```

```python
## Number of classes
K = 10

flp_len = np.mean(df['Flipper Length (mm)'])
clm_len = np.mean(df['Culmen Length (mm)'])

df = df[df['Flipper Length (mm)'].notna()]
df = df[df['Culmen Length (mm)'].notna()]
data = np.matrix(np.c_[df['Flipper Length (mm)'],df['Culmen Length (mm)']].T)
# print(data)

x_mean = np.array([[flp_len],[clm_len]])
d = data - np.reshape(x_mean,(2,1))
cov = np.matmul(d,d.T)/float(data.shape[1])
prec = np.linalg.inv(cov)#
## Init
pts = data.shape[1]
m_init = np.mean(data[:,:pts//K],axis=1)
prev = pts//K
for k in range(1,K):
    m_init = np.c_[m_init,np.mean(data[:,prev+1:prev+1+(pts//K)],axis=1)]
    prev = prev+1+(pts//K)
m_init = np.matrix(m_init)
print(m_init)
# m_init = data[:,-K:]
# m_init = data[:,10:10+K]
# m_init = data[:,:K]
# m_init = np.matrix(np.c_[data[:,0],data[:,200],data[:,-1]])
# m_init = data[:,:K]
# m_init = np.matrix(np.random.randn(2,K))
beta_init  = np.array([1. for k in range(K)])
W_init  = [prec for k in range(K)]
nu_init = np.array([2. for k in range(K)])
alpha_init = np.array([0.001 for k in range(K)])
```

    [[186.35294118 188.55882353 191.85294118 192.88235294 206.82352941
      215.82352941 219.58823529 214.38235294 194.38235294 197.51851852]
     [ 38.64411765  38.92941176  38.51764706  39.44117647  43.96176471
       46.76470588  48.07058824  48.46176471  48.84411765  48.88518519]]

```python
## Variational Approximation
def getZ(X,m,beta,W,nu,alpha):
    dig_alpha = digamma(np.sum(alpha))
    D = X.shape[0]
    N = X.shape[1]
    rho = np.zeros((N,K))
    r = np.zeros((N,K))
    for k in range(K):
        E_ln_pi_k  = digamma(alpha[k]) - dig_alpha

        E_ln_sig_k = float(D)*np.log(2) + np.log(np.linalg.det(W[k]))
        for i in range(1,D+1):
            E_ln_sig_k = E_ln_sig_k + digamma(0.5*(nu[k]+1-i))

        tmpSum = E_ln_pi_k + 0.5*E_ln_sig_k - 0.5*float(D)*np.log(2*np.pi)
        for n in range(N):
            dx = X[:,n] - m[:,k]
            E_mu_sig = float(D)/beta[k] + nu[k]*np.matmul(dx.T,np.matmul(W[k],dx))
            rho[n,k] = tmpSum - 0.5*E_mu_sig
    for n in range(N):
        rho[n,:] = softmax(rho[n,:])
    return rho

def getHelperVariables(X,r):
    Nk = np.sum(r,axis=0)

    x = np.matrix(np.zeros((X.shape[0],K)))
    for k in range(K):
        if Nk[k] != 0:
            for n in range(X.shape[1]):
                x[:,k] = x[:,k] + r[n,k]/Nk[k]*X[:,n]

    S = [0*np.eye(X.shape[0]) for k in range(K)]
    for k in range(K):
        if Nk[k] != 0:
            for n in range(X.shape[1]):
                dx = X[:,n]-x[:,k]
                S[k] = S[k] + r[n,k]*np.matmul(dx,dx.T)/Nk[k]
    return Nk,x,S

def getMu(m0,beta0,Nk,x):
    beta = beta0+Nk
#     m = (beta0*m0 + Nk*x)/beta
    m = (np.multiply(beta_init,m_init) + np.multiply(Nk,x))/beta
    return m,beta

def getPi(alpha0,Nk):
    return alpha0+Nk

def getSigma(m0,beta0,W0_inv,nu0,Nk,x,S):
    nu = nu0+Nk
    Wk = [0*np.eye(2) for k in range(K)]
    for k in range(K):
        dx = x[:,k]-m0[:,k]
        Wk_inv = W0_inv[k] + Nk[k]*S[k] + beta0[k]*Nk[k]/(beta0[k]+Nk[k])*np.matmul(dx,dx.T)
        Wk[k] = np.linalg.inv(Wk_inv)
    return Wk,nu

def getLogLikelihood(X,r,Nk,x,S,m,beta,W,nu,alpha):
    pi = Nk/float(X.shape[1])
    logLikelihood = 0
    eps = 1E-20*np.ones(pi.shape)
    pi_new = np.maximum(pi,eps)
    pi_new = pi_new/np.sum(pi_new)
    logLikelihood = 0#dirichlet.logpdf(pi_new,alpha)
    pi1 = np.array([])
    alpha1 = np.array([])
    for k in range(K):
        if pi[k] == 0:
            continue
        pi1 = np.append(pi1,np.array([pi[k]]))
        alpha1 = np.append(alpha1,np.array([alpha[k]]))
        prec = np.linalg.inv(S[k])
        cv_mat = S[k]/beta[k]
        log_mu = np.log(gaussian.pdf(np.ravel(x[:,k]),mean=np.ravel(m[:,k]),cov=cv_mat))
        log_sig = np.log(wishart.pdf(prec,df=nu[k],scale=W[k]))
        logLikelihood = logLikelihood + log_mu + log_sig

    for n in range(X.shape[1]):
        prob = 0
        for k in range(K):
#             if np.linalg.det(S[k]) == 0:
            if pi[k] == 0:
                continue
            prob = prob + pi[k]*gaussian.pdf(np.ravel(X[:,n]),mean=np.ravel(x[:,k]),cov=S[k])
        logLikelihood = logLikelihood + np.log(prob)
    logLikelihood = logLikelihood + dirichlet.logpdf(pi1,alpha1)#dirichlet.logpdf(pi_new,alpha)#
    return logLikelihood
```

```python
def VariationalGMM(X,m0,beta0,W0,nu0,alpha0,max_iter=500,tol=1E-6):
    m1    = m0
    beta1 = beta0
    W1    = W0.copy()
    nu1   = nu0
    alpha1=alpha0

    W0_inv = []
    for k in range(K):
        W0_inv += [np.linalg.inv(W0[k])]

    c = 0
    logLikelihood = []
    while c < max_iter:
        r = getZ(data,m1,beta1,W1,nu1,alpha1)

        Nk,x,S = getHelperVariables(data,r)

        m,beta = getMu(m0,beta0,Nk,x)

        alpha = getPi(alpha0,Nk)

        W,nu = getSigma(m0,beta0,W0_inv,nu0,Nk,x,S)

        logLikelihood.append(getLogLikelihood(X,r,Nk,x,S,m,beta,W,nu,alpha))
#         print(c,logLikelihood[-1],np.round(Nk,3))
        max_diff = np.max(np.abs(beta-beta1))
        max_diff = max(max_diff,np.max(np.abs(alpha-alpha1)))
        max_diff = max(max_diff,np.max(np.abs(nu-nu1)))
        max_diff = max(max_diff,np.max(np.abs(m-m1)))
        for k in range(K):
            max_diff = max(max_diff,np.max(np.abs(W[k]-W1[k])))
        m1    = m
        beta1 = beta
        W1    = W.copy()
        nu1   = nu
        alpha1=alpha
        if max_diff <= tol:
            print("Algorithm converged after iteration:",c)
            break
        c = c+1
    print("Final Log Likelihood:",logLikelihood[-1])
    print("Effective cluster size:",np.round(Nk,3))
    return m,beta,W,nu,alpha,logLikelihood

def confidence_ellipse(ax, mu, cov, n_std=3.0, facecolor='none', **kwargs):
    """
    Create a plot of the covariance confidence ellipse of `x` and `y`

    Parameters
    ----------
    cov : Covariance matrix
        Input data.

    ax : matplotlib.axes.Axes
        The axes object to draw the ellipse into.

    n_std : float
        The number of standard deviations to determine the ellipse's radiuses.

    Returns
    -------
    matplotlib.patches.Ellipse

    Other parameters
    ----------------
    kwargs : `~matplotlib.patches.Patch` properties
    """
#     if cov != cov.T:
#         raise ValueError("Not a valid covariance matrix")

#     cov = np.cov(x, y)
    pearson = cov[0, 1]/np.sqrt(cov[0, 0] * cov[1, 1])
    # Using a special case to obtain the eigenvalues of this
    # two-dimensionl dataset.
    ell_radius_x = np.sqrt(1 + pearson)
    ell_radius_y = np.sqrt(1 - pearson)
    ellipse = Ellipse((0, 0),
        width=ell_radius_x * 2,
        height=ell_radius_y * 2,
        facecolor=facecolor,
        **kwargs)

    # Calculating the stdandard deviation of x from
    # the squareroot of the variance and multiplying
    # with the given number of standard deviations.
    scale_x = np.sqrt(cov[0, 0]) * n_std
    mean_x = mu[0]

    # calculating the stdandard deviation of y ...
    scale_y = np.sqrt(cov[1, 1]) * n_std
    mean_y = mu[1]

    transf = transforms.Affine2D() \
        .rotate_deg(45) \
        .scale(scale_x, scale_y) \
        .translate(mean_x, mean_y)

    ellipse.set_transform(transf + ax.transData)
    return ax.add_patch(ellipse)
```

```python
m,beta,W,nu,alpha,logLikelihood = VariationalGMM(data,m_init,beta_init,W_init,nu_init,alpha_init)#,max_iter=1000)
```

    Algorithm converged after iteration: 148
    Final Log Likelihood: -2199.653956297383
    Effective cluster size: [  0.      0.    148.792   0.      0.    130.389   0.      0.      1.132
      61.688]

#### Determining optimal number of clusters

```python
k_idx = []
dist_tol = 1E-6
r = getZ(data,m,beta,W,nu,alpha)
Nk,x,S = getHelperVariables(data,r)
m_final = None
beta_final = None
alpha_final = None
W_final = None
nu_final = None
N_final = None
x_final = None
S_final = None
for k in range(K):
    dist = np.linalg.norm(m[:,k]-m_init[:,k])
    if dist >= 1E-6 and Nk[k] >= 1:#0.01*data.shape[1]:
        if m_final is None:
            m_final = m[:,k]
            beta_final = np.array([beta[k]])
            alpha_final = np.array([alpha[k]])
            nu_final = np.array([nu[k]])
            W_final = [W[k]]
            N_final = np.array([Nk[k]])
            x_final = x[:,k]
            S_final = [S[k]]
        else:
            m_final = np.c_[m_final,m[:,k]]
            beta_final = np.append(beta_final,np.array([beta[k]]))
            alpha_final = np.append(alpha_final,np.array([alpha[k]]))
            nu_final = np.append(nu_final,np.array([nu[k]]))
            W_final += [W[k]]
            N_final = np.append(N_final,np.array([Nk[k]]))
            x_final = np.c_[x_final,x[:,k]]
            S_final += [S[k]]

K_final = x_final.shape[1]
print("Number of actual clusters:",K_final)
print()
print("Cluster Means:")
print(x_final)
print()
print("Cluster Covariance")
for k in range(K_final):
    print(S_final[k])
```

    Number of actual clusters: 4

    Cluster Means:
    [[189.3128161  216.55057185 182.07307154 196.19772708]
     [ 38.75547148  47.27299022  56.98587604  49.06080624]]

    Cluster Covariance
    [[35.94386692  3.55981486]
     [ 3.55981486  6.92690337]]
    [[47.61652171 15.77126996]
     [15.77126996 10.59665065]]
    [[17.21662508 -7.1001078 ]
     [-7.1001078   8.94857704]]
    [[35.18552793 10.10415175]
     [10.10415175  7.84172856]]

```python
fig,ax = plt.subplots(1,3,figsize=(20,5))
K = K_final
r = getZ(data,m_final,beta_final,W_final,nu_final,alpha_final)
Nk,x,S = getHelperVariables(data,r)
Nk = np.sum(r,axis=0)
for n in range(data.shape[1]):
    rgb = np.array([0,0,0])
    for k in range(K):
        rgb = rgb+r[n,(k+2)%K]*np.array(mcolors.to_rgb(color[k]))
    ax[1].scatter(data[0,n],data[1,n],color=rgb)
ax[1].set_title('Classification as a function of responsibilities')
for k in range(3):
    ax[0].scatter(df_data[k]['Flipper Length (mm)'],df_data[k]['Culmen Length (mm)'],color=color[k],alpha=0.3)

for k in range(K):
    ki = (k+2)%K
    ax[0].plot(x_final[0,ki],x_final[1,ki],'kx')
    for i in range(3):
        confidence_ellipse(ax[0],x_final[:,ki],S_final[ki],i+1,edgecolor=color[k],linestyle='dashed')

ax[0].set_title('Variational Approximation')
ax[0].set_xlabel('Flipper Length (mm)');
ax[0].set_ylabel('Culmen Length (mm)');

ax[2].plot(range(len(logLikelihood)),logLikelihood);
ax[2].set_title('Learning curve');
ax[2].set_ylabel('Log Likehood');
ax[2].set_xlabel('Iteration Number');
```

## References
[1]: Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.

[2]: Horst AM, Hill AP, Gorman KB (2020). palmerpenguins: Palmer Archipelago (Antarctica) penguin data. R package version 0.1.0. https://allisonhorst.github.io/palmerpenguins/.

[3]: CSV data downloaded from https://github.com/mcnakhaee/palmerpenguins

[4]: Code for plotting confidence ellipses from https://matplotlib.org/3.1.0/gallery/statistics/confidence_ellipse.html


# Lab 1 - Vectors and Matrices
This notebook demonstrates the use of vectors and matrices in IPython. Note that the basis is not explicit in any of these operations. You must keep track of the basis yourself (using variable names, or notes etc).

```python
from numpy import array, dot, outer, sqrt, matrix
from numpy.linalg import eig, eigvals
from matplotlib.pyplot import hist
```

```python
%matplotlib inline
```

```python
rv = array([1,2])  # a row vector
rv
```

    array([1, 2])

```python
cv = array([[3],[4]])  # a column vector
cv
```

    array([[3],
           [4]])

Two kinds of vector products we'll see: inner product (dot product) and outer product

### 1) Use the function _dot(vector1, vector2)_ to find the dot product of rv and cv. Does the order of the arguments matter?

```python
dot(rv,cv)
```

    array([11])

```python
dot(cv,rv)
```

### 2) Use the function _outer(vector1, vector2)_ to find the outer product of rv and cv. Does the order of the arguments matter?

```python
outer(rv,cv)
```

    array([[3, 4],
           [6, 8]])

```python
outer(cv,rv)
```

    array([[3, 6],
           [4, 8]])

## II. Complex vectors

```python
# Complex numbers in python have a j term:
a = 1+2j
```

```python
v1 = array([1+2j, 3+2j, 5+1j, 4+0j])
```

The complex conjugate changes the sign of the imaginary part:

```python
v1.conjugate()
```

    array([1.-2.j, 3.-2.j, 5.-1.j, 4.-0.j])

### 3) Use _dot()_ and _.conjugate()_ to find the dot product of v1 and it's own conjugate:

```python
dot(v1.conjugate(),v1)
```

    (60+0j)

## III. Matrices

```python
# a two-dimensional array
m1 = array([[2,1],[2,1]])
m1
```

    array([[2, 1],
           [2, 1]])

```python
# can find transpose with the T method:
m1.T
```

    array([[2, 2],
           [1, 1]])

```python
# find the eigenvalues and eigenvectors of a matrix:
eig(m1)
```

    (array([3., 0.]), array([[ 0.70710678, -0.4472136 ],
            [ 0.70710678,  0.89442719]]))

Can also use the `matrix` type which is like array but restricts to 2D. Also, `matrix` adds `.H` and `.I` methods for hermitian and inverse, respectively. For more information, see Stack Overflow question [#4151128](http://stackoverflow.com/questions/4151128/what-are-the-differences-between-numpy-arrays-and-matrices-which-one-should-i-u)

```python
m2 = matrix( [[2,1],[2,1]])
```

```python
m2.H
```

    matrix([[2, 2],
            [1, 1]])

```python
eig(m2)
```

    (array([3., 0.]), matrix([[ 0.70710678, -0.4472136 ],
             [ 0.70710678,  0.89442719]]))

```python
# use a question mark to get help on a command
eig?
```

# Examples:
## Example 1.4
Find the eigenvalues and eigenvectors of M = ([0,1],[-2,3]])

```python
M14 = array([[0,1],[-2,3]])
```

```python
eig(M14)
```

    (array([1., 2.]), array([[-0.70710678, -0.4472136 ],
            [-0.70710678, -0.89442719]]))

Interpret this result:
the two eigenvalues are 1 and 2
the eigenvectors are strange decimals, but we can check them against the stated solution:

```python
1/sqrt(2)  # this is the value for both entries in the first eigenvector
```

    0.7071067811865475

```python
1/sqrt(5)  # this is the first value in the second eigenvector
```

    0.4472135954999579

```python
2/sqrt(5)  # this is the second value in the second eigenvector
```

    0.8944271909999159

```python
eigvals(M14)
```

    array([1., 2.])

Signs are opposite compared to the book, but it turns out that (-) doesn't matter in the interpretation of eigenvectors: only "direction" matters (the relative size of the entries).

## Example: Problem 1.16 using Python functions

```python
M16 = array([[0,-1j],[1j,0]])
```

```python
evals, evecs = eig(M16)
```

```python
evecs
```

    array([[-0.        -0.70710678j,  0.70710678+0.j        ],
           [ 0.70710678+0.j        ,  0.        -0.70710678j]])

```python
evecs[:,0]
```

    array([-0.        -0.70710678j,  0.70710678+0.j        ])

```python
evecs[:,1]
```

    array([0.70710678+0.j        , 0.        -0.70710678j])

```python
dot(evecs[:,0].conjugate(),evecs[:,1])
```

    -1.6653345369377348e-16j

# Part 2: Using QuTiP
Keeping track of row and column vectors in Ipython is somewhat artificial and tedious. The QuTiP library is designed to take care of many of these headaches

```python
from qutip import *
```

```python
# Create a row vector:
qv = Qobj([[1,2]])
qv
```

Quantum object: dims = [[1], [2]], shape = (1, 2), type = bra\begin{equation*}\left(\begin{array}{*{11}c}1.0 & 2.0\\\end{array}\right)\end{equation*}

```python
# Find the corresponding column vector
qv.dag()
```

Quantum object: dims = [[2], [1]], shape = (2, 1), type = ket\begin{equation*}\left(\begin{array}{*{11}c}1.0\\2.0\\\end{array}\right)\end{equation*}

```python
qv2 = Qobj([[1+2j,4-1j]])
qv2
```

Quantum object: dims = [[1], [2]], shape = (1, 2), type = bra\begin{equation*}\left(\begin{array}{*{11}c}(1.0+2.0j) & (4.0-1.0j)\\\end{array}\right)\end{equation*}

```python
qv2.dag()
```

Quantum object: dims = [[2], [1]], shape = (2, 1), type = ket\begin{equation*}\left(\begin{array}{*{11}c}(1.0-2.0j)\\(4.0+1.0j)\\\end{array}\right)\end{equation*}

## Vector products in QuTiP
Only need to know one operator: "\*"
The product will depend on the order, either inner or outer

```python
qv2*qv2.dag()  # inner product (dot product)
```

Quantum object: dims = [[1], [1]], shape = (1, 1), type = bra\begin{equation*}\left(\begin{array}{*{11}c}22.0\\\end{array}\right)\end{equation*}

```python
qv2.dag()*qv2  # outer product
```

Quantum object: dims = [[2], [2]], shape = (2, 2), type = oper, isherm = True\begin{equation*}\left(\begin{array}{*{11}c}5.0 & (2.0-9.0j)\\(2.0+9.0j) & 17.0\\\end{array}\right)\end{equation*}

## Matrix in QuTiP

```python
qm = Qobj([[1,2],[2,1]])
qm
```

Quantum object: dims = [[2], [2]], shape = (2, 2), type = oper, isherm = True\begin{equation*}\left(\begin{array}{*{11}c}1.0 & 2.0\\2.0 & 1.0\\\end{array}\right)\end{equation*}

```python
qm.eigenenergies()  # in quantum (as we will learn) eigenvalues often correspond to energy levels
```

    array([-1.,  3.])

```python
evals, evecs = qm.eigenstates()
```

```python
evecs
```

    array([Quantum object: dims = [[2], [1]], shape = (2, 1), type = ket
    Qobj data =
    [[-0.70710678]
     [ 0.70710678]],
           Quantum object: dims = [[2], [1]], shape = (2, 1), type = ket
    Qobj data =
    [[0.70710678]
     [0.70710678]]], dtype=object)

```python
evecs[1]
```

Quantum object: dims = [[2], [1]], shape = (2, 1), type = ket\begin{equation*}\left(\begin{array}{*{11}c}0.707\\0.707\\\end{array}\right)\end{equation*}

# Some additional examples we have already done by hand:

## Problem 1.2 using the _hist()_ function.
Create a histogram of the data and from it estimate the probability distribution. Use min and max values for the range of 5 and 14, and use 5 bins (5 and 6 go in the first bin).

```python
# Solution
data = [10,13,14,14,6,8,7,9,12,14,13,11,10,7,7]
# Fill in the hist() function:
n, bins, patches = hist(data, bins=9, range=(5,14))
```

## Problem 1.8
Find the constant $c$ that normalizes the probability density:
$$p(x) =
\begin{cases}
ce^{-ax},  & x \geq 0 \\[2ex]
0 & x < 0
\end{cases}
$$

**Hint:** using sympy, we can calculate the relevant integral. The conds='none' asks the solver to ignore any strange conditions on the variables in the integral. This is fine for most of our integrals. Usually the variables are real and well-behaved numbers.

```python
# Partial Solution:
from sympy import *
c,a,x = symbols("c a x")
Q.positive((c,a))
first = integrate( c*exp(-a*x) ,(x,0,oo),conds='none')
first
```

$\displaystyle \frac{c}{a}$

```python
check = integrate( a*exp(-a*x) ,(x,0,oo),conds='none')
check
```

$\displaystyle 1$

```python

```


# 2 Step Adam Bashforth
#### John S Butler
john.s.butler@tudublin.ie
[Course Notes](https://johnsbutler.netlify.com/files/Teaching/Numerical_Analysis_for_Differential_Equations.pdf)    [Github](https://github.com/john-s-butler-dit/Numerical-Analysis-Python)

This notebook implements the 2 step Adams Bashforth method for three different population intial value problems.

# Formula
The general 2 step Adams-Bashforth method for the first order differential equation
\begin{equation} y^{'} = f(t,y), \end{equation}
numerical approximates $y$ the at time point $t_i$ as $w_i$
with the  formula:
\begin{equation} w_{i+1}=w_i+\frac{h}{2}\big[3f(t_i,w_i)-f(t_{i-1},w_{i-1})\big],\end{equation}
for $i=0,...,N-1$, where

and $h$ is the stepsize.

To illustrate the method we will apply it to three intial value problems:
## 1. Linear
Consider the linear population Differential Equation
\begin{equation} y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## 2. Non-Linear Population Equation
Consider the non-linear population Differential Equation
\begin{equation} y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## 3. Non-Linear Population Equation with an oscillation
Consider the non-linear population Differential Equation with an oscillation
\begin{equation} y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

#### Setting up Libraries

```python
## Library
import numpy as np
import math
import pandas as pd

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")
```

## Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation} h=\frac{b-a}{N}.\end{equation}
Here the interval is $2000\leq t \leq 2020,$
\begin{equation} h=\frac{2020-2000}{200}=0.1.\end{equation}
This gives the 201 discrete points:
\begin{equation} t_0=2000, \ t_1=2000.1, \ ... t_{200}=2020. \end{equation}
This is generalised to
\begin{equation} t_i=2000+i0.1, \ \ \ i=0,1,...,200.\end{equation}
The plot below shows the discrete time steps:

```python
### DISCRETE TIME
N=20
t_end=2020.0
t_start=2000.0
h=((t_end-t_start)/N)
t=np.arange(t_start,t_end+h/2,h)

## PLOTS TIME
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.title('Illustration of discrete time points for h=%s'%(h))
plt.show()
len(t)
```

# 1. Linear Population Equation
## Exact Solution
The linear population equation
\begin{equation} y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
has a known exact (analytic) solution
\begin{equation} y(t)=6e^{0.1(t-2000)}. \end{equation}

## Specific 2 step Adams Bashforth
The specific 2 step Adams Bashforth for the linear population equation is:

\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}\big[3(f(t_i,w_i))-f(t_{i-1},w_{i-1}))\big] \end{equation}
where
\begin{equation}f(t,y)=0.1y.\end{equation}

```python
## THIS IS THE RIGHT HANDSIDE OF THE LINEAR POPULATION DIFFERENTIAL
## EQUATION
def linfun(t,w):
    ftw=0.1*w
    return ftw
```

this gives

\begin{equation} w_{i+1}=w_{i}+\frac{0.1}{2}\big[ 3(0.1w_i)-0.1w_{i-1} \big]\end{equation}
for $i=1,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}

```python
### INSERT METHOD HERE
w=np.zeros(N+1) # a list of 2000+1 zeros
w[0]=6 # INITIAL CONDITION
w[1]=6.06
for i in range(1,N):
    w[i+1]=w[i]+h/2*(3*linfun(t[i],w[i])-linfun(t[i-1],w[i-1]))

```

## Plotting Results

```python
## PLOTTING METHOD
y=6*np.exp(0.1*(t-2000)) # EXACT SOLUTION
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Taylor')
plt.plot(t,y,'s:',color='black',label='Exact')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time, the numerical approximation, $w$,  the exact solution, $y$, and the exact error $|y(t_i)-w_i|$ for the linear population equation:

```python

d = {'time t_i': t, 'Adams w': w,'Exact y(t_i)':y,'Exact Error |w_i-y_i|':np.abs(y-w)}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Adams w
      Exact y(t_i)
      Exact Error |w_i-y_i|

      0
      2000.0
      6.000000
      6.000000
      0.000000

      1
      2001.0
      6.060000
      6.631026
      0.571026

      2
      2002.0
      6.669000
      7.328417
      0.659417

      3
      2003.0
      7.366350
      8.099153
      0.732803

      4
      2004.0
      8.137852
      8.950948
      0.813096

      5
      2005.0
      8.990213
      9.892328
      0.902115

      6
      2006.0
      9.931852
      10.932713
      1.000861

      7
      2007.0
      10.972119
      12.082516
      1.110397

      8
      2008.0
      12.121345
      13.353246
      1.231901

      9
      2009.0
      13.390940
      14.757619
      1.366678

      10
      2010.0
      14.793514
      16.309691
      1.516177

      11
      2011.0
      16.342994
      18.024996
      1.682002

      12
      2012.0
      18.054768
      19.920702
      1.865934

      13
      2013.0
      19.945833
      22.015780
      2.069947

      14
      2014.0
      22.034970
      24.331200
      2.296230

      15
      2015.0
      24.342924
      26.890134
      2.547211

      16
      2016.0
      26.892614
      29.718195
      2.825581

      17
      2017.0
      29.709360
      32.843684
      3.134325

      18
      2018.0
      32.821133
      36.297885
      3.476752

      19
      2019.0
      36.258835
      40.115367
      3.856532

      20
      2020.0
      40.056603
      44.334337
      4.277733

## 2. Non-Linear Population Equation
\begin{equation} y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
## Specific 2 step Adams-Bashforth method for the Non-Linear Population Equation
The specific Adams-Bashforth difference equation for the non-linear population equations is:

\begin{equation}w_{i+1}=w_{i}+\frac{0.1}{2}\big[3(0.2w_{i}-0.01w_{i}^2)-
(0.2w_{i-1}-0.01w_{i-1}^2)\big], \end{equation}

for $i=1,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}
To solve the 2 step method we need a value for $w_1$, here, we will use the approximation from the 2nd order Taylor method (see other notebook),
\begin{equation}w_1=6.084.\end{equation}

```python
def nonlinfun(t,w):
    ftw=0.2*w-0.01*w*w
    return ftw
```

```python
### INSERT METHOD HERE
w=np.zeros(N+1)
w[0]=6
w[1]=6.084 # FROM THE THE TAYLOR METHOD
for n in range(1,N):
    w[n+1]=w[n]+h/2*(3*nonlinfun(t[n],w[n])-nonlinfun(t[n-1],w[n-1]))
```

## Results
The plot below shows the numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='2 step Adams Method ')
plt.title('Non Linear Population Equation')
plt.legend(loc='best')
plt.xlabel('time (yrs)')
plt.ylabel('Population in billions')
plt.show()
```

## Table
The table below shows the time and the numerical approximation, $w$,  for the non-linear population equation:

```python

d = {'time t_i': t, 'Adams approx of non-linear w': w}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Adams approx of non-linear w

      0
      2000.0
      6.000000

      1
      2001.0
      6.084000

      2
      2002.0
      6.933974

      3
      2003.0
      7.869642

      4
      2004.0
      8.848568

      5
      2005.0
      9.851373

      6
      2006.0
      10.857671

      7
      2007.0
      11.846747

      8
      2008.0
      12.799268

      9
      2009.0
      13.698782

      10
      2010.0
      14.532747

      11
      2011.0
      15.292965

      12
      2012.0
      15.975461

      13
      2013.0
      16.579947

      14
      2014.0
      17.109042

      15
      2015.0
      17.567443

      16
      2016.0
      17.961143

      17
      2017.0
      18.296777

      18
      2018.0
      18.581128

      19
      2019.0
      18.820774

      20
      2020.0
      19.021862

## 3. Non-Linear Population Equation with an oscilation
\begin{equation} y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## Specific 2 Step Adams Bashforth for the Non-Linear Population Equation with an oscilation
To write the specific

\begin{equation} w_{i+1}=w_{i}+\frac{0.1}{2} \big[ 3(0.2w_{i}-0.01w_{i}^2+\sin(2\pi t_{i}))- (0.2w_{i-1}-0.01w_{i-1}^2+\sin(2\pi t_{i-1}))\big] \end{equation}

for $i=1,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}
 As $w_1$ is required for the method but unknown we will use the numerical solution of a one step method to approximate the value. Here, we use the 2nd order Runge Kutta approximation (see [Runge Kutta notebook](https://github.com/john-s-butler-dit/Numerical-Analysis-Python/blob/master/Chapter%2003%20-%20Runge%20Kutta/01_2nd%20Order%20Runge%20Kutta%20Population%20Equations.ipynb) )
\begin{equation}w_1=6.11.\end{equation}

```python
def nonlin_oscfun(t,w):
    ftw=0.2*w-0.01*w*w+np.sin(2*np.math.pi*t)
    return ftw
```

```python
## INSERT METHOD HERE
w=np.zeros(N+1)
w[0]=6
w[1]=6.11
for n in range(1,N):
    w[n+1]=w[n]+h/2*(3*nonlin_oscfun(t[n],w[n])
                     -nonlin_oscfun(t[n-1],w[n-1]))
```

## Results
The plot below shows the  numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Adams-Bashforth')
plt.title('Population Equation with seasonal oscilation')
plt.xlabel('time (yrs)')
plt.ylabel('Population in Billions')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time and the numerical approximation, $w$,  for the non-linear population equation with oscilations:

```python
d = {'time t_i': t, 'Adams Approx w': w}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Adams Approx w

      0
      2000.0
      6.000000

      1
      2001.0
      6.110000

      2
      2002.0
      6.963018

      3
      2003.0
      7.900330

      4
      2004.0
      8.880317

      5
      2005.0
      9.883555

      6
      2006.0
      10.889620

      7
      2007.0
      11.877816

      8
      2008.0
      12.828881

      9
      2009.0
      13.726473

      10
      2010.0
      14.558187

      11
      2011.0
      15.315964

      12
      2012.0
      15.995957

      13
      2013.0
      16.597982

      14
      2014.0
      17.124739

      15
      2015.0
      17.580977

      16
      2016.0
      17.972719

      17
      2017.0
      18.306611

      18
      2018.0
      18.589435

      19
      2019.0
      18.827758

      20
      2020.0
      19.027711

```python

```

Chapter 04 - Multistep Methods/402_Adams Bashforth Population Equations.ipynb


$$ \newcommand{\pd}[2]{ \frac{\partial #1}{\partial #2} }
\newcommand{\od}[2]{\frac{d #1}{d #2}}
\newcommand{\td}[2]{\frac{D #1}{D #2}}
\newcommand{\ab}[1]{\langle #1 \rangle}
\newcommand{\bss}[1]{\textsf{\textbf{#1}}}
\newcommand{\ol}{\overline}
\newcommand{\olx}[1]{\overline{#1}^x}
$$

# Homework 4: Equation Derivations

In this homework, we will simply re-derive equations that we have already derived in class. The point of doing it yourself is to make sure you understand each step of the derivation.

The starting point for all derivations is the Boussinesq horizontal equations of motion at low Rossby number:

$$ \begin{align} - f v &= -\pd{\phi}{x} + \nu \nabla^2 u \\
 f u &= -\pd{\phi}{y} + \nu \nabla^2 v
\end{align} $$

Plus the hydrostatic balance

$$ \pd{\phi}{z} = b $$

and the Boussinesq continuity equation:

$$ \pd{u}{x} + \pd{v}{y} + \pd{w}{z} = 0 \ . $$

Make sure your notation is precise enough to destiguish between

- Ordinary vs. partial derivatives
- Vectors vs. scalars
- Lower case and capital letters

For every problem, write _very neatly_ and _explain each step_. Each problem is worth 25%.

### 1) Derive the Thermal Wind Balance

$$ \begin{align} - f \pd{v_g}{z} &=  -\pd{b}{x}\\
f \pd{u_g}{z} &=  -\pd{b}{y} \ .
\end{align} $$

and explain what it means.

### 2) Derive the formula for Ekman pumping

$$ w_{Ek} =
  \pd{}{x}\left( \frac{\tau_y}{\rho_0 f} \right)
- \pd{}{y}\left( \frac{\tau_x}{\rho_0 f} \right)  \ . $$

### 3) Derive the formula for Sverdrup transport

$$
\beta V_{Sv} = \frac{1}{\rho_0}  \mbox{curl} ( \boldsymbol{\tau}_{surf}) \ .
$$

Explain in words what the equation means.

### 4) Derive the formula for the Stommel Aarons deep flow

\begin{equation*}
V=\frac{f Q_s}{\beta A}
\end{equation*}

Explain what $Q_s$ and $A$ represent, and explain what the equation means in words.


------------------------------
# Gadient Descent
> Prerequisites to increase your understanding
> - Programming knowledge
> - Mathematical background
> - Jupyter notebook knowledge

I will try to explain gradient descent a little in this assignment. I am going to abbreviate gradient descent as GD in this notebook now on.

GD is an algorithm that is used to solve the machine learning problems such as linear regression.

Simply, the goal of linear regression is to fit a line of corresponding given dataset or points.

Let's consider that we want to fit a line for the points showed with the graphic below.

| x (input)| y (output)|
| :----- | :----- |
| 10.0   | 8.04   |
| 8.0    | 6.95   |
| 13.0   | 7.58   |
| 9.0    | 8.81   |
| 11.0   | 8.33   |
| 14.0   | 9.96   |
| 6.0    | 7.24   |
| 4.0    | 4.26   |
| 12.0   | 10.84  |
| 7.0    | 4.82   |
| 5.0    | 5.68   |

```python
# dataset
y = [8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]
x = [10.0, 8.00, 13.0, 9.00, 11.0, 14.0, 6.00, 4.00, 12.00, 7.00, 5.00]
```

As you can guess, we are talking about a line and a line has an equation like $y = mx + b$. Here the coefficient of $x$, $m$ is the slope of the line and $b$ is y-intersect. We can simply say that the fittest line (the solution of linear regression) is strictly related to the values of $m$ and $b$. That's why it is obvious that we have to find the best pair of $m$ and $b$ values.

The common approach to finding the best values is to use a function named _error function_ or _cost function_ to measure how fit the line is.

\begin{equation*}
Error_{m, b} = \frac{1}{N} \sum_{i=1}^N \left( y_i - \left(m x_i + b \right) \right)^2
\end{equation*}

The squared error function makes us enable to calculate the error value positive and differentiate the error function.

```python
def computeError(m, b):
    # initialize total error
    error_m_b = 0.0

    # loop within the dataset
    for i in range(len(x)):
        x_i = x[i]
        y_i = y[i]

        # compute the error for a single point
        # and add it to the total error
        error_m_b += (y_i - (m * x_i + b)) ** 2
    # end of for loop

    # compute the mean of the error for all dataset
    error_m_b = error_m_b / float(len(x))
    return error_m_b
# end of def computeError

# compute error for the line corresponded by given m and b values
print("total error:", computeError(0.9, 0.1))
```

    total error: 3.3391545454545453

It is simple to compute the total error for a certain line. But we need to find the direction of total error to determine what we are going to do. Are we going to reduce the slope and y-intersect or raise them? Is total error increasing or decreasing? So we have to calculate the changes of the total error. As you know this means the first derivative of the error function.

However, the error function has two separate parameters. Which parameter are we going to differentiate it with respect to? We are going to differentiate it with respect to two parameters separately by the partial derivative. Partial derivative means that simply we approach the other parameter (variable) as a constant (for more information [wiki](https://en.wikipedia.org/wiki/Partial_derivative)). When we differentiate the error function with respect to two separate parameters partially, we will have two equations such below.

\begin{equation}
\frac{\partial E}{\partial m} = \frac{2}{N} \sum_{i=1}^N -x_i \left( y_i - \left( m x_i + b \right) \right)
\end{equation}

----------------------
\begin{equation}
\frac{\partial E}{\partial b} = \frac{2}{N} \sum_{i=1}^N - \left( y_i - \left( m x_i + b \right) \right)
\end{equation}

Let's look at how the code of these equations looks like.

```python
# definition of the function to compute the gradient of m.
def gradientM(m, b):
    # setting default value of gradient
    gradient_m = 0.0

    # loop within the dataset
    for i in range(len(x)):
        x_i = x[i]
        y_i = y[i]

        # compute the first derivative of the error funtion
        # with respect to m
        gradient_m += -x_i * (y_i - (m * x_i + b))
    # end of for loop

    # mean of total error
    gradient_m = 2 * gradient_m / float(len(x))
    return gradient_m
# end of def gradientM

# definition of the function to compute the gradient of b.
def gradientB(m, b):
    # setting default value of gradient
    gradient_b = 0.0

    # loop within the dataset
    for i in range(len(x)):
        x_i = x[i]
        y_i = y[i]

        # compute the first derivative of the error funtion
        # with respect to b
        gradient_b += -(y_i - (m * x_i + b))
    # end of for loop

    # mean of total error
    gradient_b = 2 * gradient_b / float(len(x))
    return gradient_b
# end of def gradientB

print("gradient_m: ", gradientM(1, 0))
print("gradient_b: ", gradientB(1, 0))
```

    gradient_m:  36.98181818181817
    gradient_b:  2.998181818181818

Now we have enough information how to compute the gradient of the line and determine how fit it is. So what is next that we are going to do? We are going to use these gradient results that we computed to determine the next values of $m$ and $b$. That's why we need a new equation to determine new values of $m$ and $b$ parameters.

\begin{align}
& m_0 = \text{initial value} \\
& \text{for} \; \tau = 1, 2,\dots \\
& m_{\tau} = m_{\tau-1} - \eta \nabla E(m_{\tau-1})
\end{align}

-------------------------------
\begin{align}
& b_0 = \text{initial value} \\
& \text{for} \; \tau = 1, 2,\dots \\
& b_{\tau} = b_{\tau-1} - \eta \nabla E(b_{\tau-1})
\end{align}

* $\eta$: Learning rate is a parameter of the algorithm
* $m$ and $b$: the variable is the parameters of the Model. _They represent scalar variables._
* $y$: targets (outputs)
* $x$: inputs

Yes, above you can see the calculation methods for new $m$ and $b$ values. As you know $\nabla E(b_{\tau-1})$ and $\nabla E(m_{\tau-1})$ mean derivative. Thus, we can write these equations like;

\begin{align}
\nabla E(m_{\tau-1}) = \frac{2}{N} \sum_{i=1}^N -x_i \left( y_i - \left( m_{\tau-1} x_i + b_{\tau-1} \right) \right)
\end{align}

-------------------------------
\begin{align}
\nabla E(b_{\tau-1}) = \frac{2}{N} \sum_{i=1}^N - \left( y_i - \left( m_{\tau-1} x_i + b_{\tau-1} \right) \right)
\end{align}

```python
def gradientStep(eta, m, b):
    # compute the derivate with respect to m and b
    dM = gradientM(m, b)
    dB = gradientB(m, b)

    # compute new value of m and b
    m = m - eta * dM
    b = b - eta * dB

    return m, b
# end of def gradientStep

# iteration count
N = 101

# learning rate
eta = 0.01

# initial values
m = b = 0

# Let's start iterating
for i in range(N):
    if i % 10 == 0:
        print(i, "\t error:", computeError(m, b))
    m, b = gradientStep(eta, m, b)
# end of for loop
```

    0 	 error: 60.01569999999999
    10 	 error: 3.8288460786805696
    20 	 error: 2.159837161759056
    30 	 error: 2.0758079629206465
    40 	 error: 2.039328445552182
    50 	 error: 2.005681412537383
    60 	 error: 1.9735062215911647
    70 	 error: 1.942704101310751
    80 	 error: 1.9132154562486745
    90 	 error: 1.884984247675222
    100 	 error: 1.8579568559211126

As you can see that we accomplished finding the fittest line out. Let's how it looks like :)

```python
# python library for plotting
import matplotlib.pylab as plt

# iteration count
N = 100

# learning rate
eta = 0.01

# initial values
m = b = 0

# lists to keep y-values of line
line = []

# Let's start iterating
for i in range(N):
    m, b = gradientStep(eta, m, b)
# end of for loop

for i in range(len(x)):
    line.append(m * x[i] + b)
# end of for loop

print("Parameters of the fittest line:")
print("m:", m)
print("b:", b)

# drawing of graph.
plt.title("Fittest Line")
plt.plot(x, y, "go")
plt.plot(x, line)
plt.show()
```

Let's see how the error level changes through the iterations.

```python
# python library for plotting
import matplotlib.pylab as plt

# iteration count
N = 30

# learning rate
eta = 0.01

# initial values
m = b = 0

# lists to keep the error and the iteration values.
errors = []
Ns = []

# Let's start iterating
for i in range(N):
    errors.append(computeError(m, b))
    Ns.append(i)
    m, b = gradientStep(eta, m, b)
# end of for loop

# drawing of graph.
plt.title("Error distribution for %s iteration" % N)
plt.ylabel("Error")
plt.xlabel("Iteration")
plt.plot(Ns, errors)
plt.show()
```

It is obvious that we can carry out a few optimization for the GD algorithm. For example, extending the steps of GD. This idea leads us to the notion of momentum.

## Gradient Descent with momentum

We are going to denote $\eta$ as $\alpha$ now on and for momentum calculation, we have to do a few modification in GD equation.

$p$ will denote the momentum;

\begin{align}
& p_{\tau} = \nabla E(m_{\tau-1}) + \beta p_{\tau-1} \\
& m_{\tau} = m_{\tau-1} - \alpha p_{\tau} \\
& m_{\tau} = m_{\tau-1} - \alpha \left( \nabla E(m_{\tau-1}) + \beta p_{\tau-1} \right)
\end{align}

------------------------------------------
\begin{align}
& p_{\tau} = \nabla E(b_{\tau-1}) + \beta p_{\tau-1} \\
& b_{\tau} = b_{\tau-1} - \alpha p_{\tau} \\
& b_{\tau} = b_{\tau-1} - \alpha \left( \nabla E(b_{\tau-1}) + \beta p_{\tau-1} \right) \\
\end{align}

$\beta$ will be our momentum coefficient. When $\beta=0$, we recover GD.

```python
def momentumM(m, b, p_m, beta):
    gradient_m = gradientM(m, b)
    momentum_m = gradient_m + beta * p_m
    return momentum_m
# end of def momentumM

def momentumB(m, b, p_b, beta):
    gradient_b = gradientB(m, b)
    momentum_b = gradient_b + beta * p_b
    return momentum_b
# end of def momentumB

def gradientStepWithMomentum(alpha, m, b, p_m0, p_b0, beta):
    # compute the momentum values of m and b
    p_m = momentumM(m, b, p_m0, beta)
    p_b = momentumB(m, b, p_b0, beta)

    # compute new values of m and b
    m = m - alpha * p_m
    b = b - alpha * p_b

    return m, b, p_m, p_b
# end of def gradientStepWithMomentum

# iteration count
N = 101

# learning rate
alpha = 0.01

# momentum coefficient
beta = 0.9

# initial values
m = b = 0
p_m = p_b = 1

# lists to keep the error and the iteration values.
errors = []
Ns = []

# Let's start iterating
for i in range(N):
    # append these value for plotting grapghic
    errors.append(computeError(m, b))
    Ns.append(i)

    if i % 10 == 0:
        print(i, "\t error:", computeError(m, b))

    m, b, p_m, p_b = gradientStepWithMomentum(alpha, m, b, p_m, p_b, beta)
# end of for loop

# drawing of graph.
plt.title("Error distribution for %s iteration" % N)
plt.ylabel("Error")
plt.xlabel("Iteration")
plt.plot(Ns, errors)
plt.show()
```

## Gradient Descent with adaptive alpha

```python
import math

def momentumM(m, b, p_m, beta):
    gradient_m = gradientM(m, b)
    momentum_m = gradient_m + beta * p_m
    return momentum_m
# end of def momentumM

def momentumB(m, b, p_b, beta):
    gradient_b = gradientB(m, b)
    momentum_b = gradient_b + beta * p_b
    return momentum_b
# end of def momentumB

def gradientStepWithMomentum(alpha, m, b, p_m0, p_b0, beta):
    # compute the momentum values of m and b
    p_m = momentumM(m, b, p_m0, beta)
    p_b = momentumB(m, b, p_b0, beta)

    # compute new values of m and b
    m = m - alpha * p_m
    b = b - alpha * p_b

    return m, b, p_m, p_b
# end of def gradientStepWithMomentum

def adaptAlpha(currentError, prevError, alpha):
    newAlpha = alpha
    if currentError > prevError:
        newAlpha = alpha * 0.9
    else:
        newAlpha = alpha * 1.1
    return newAlpha
# end of def adaptAlpha

# iteration count
N = 101

# learning rate
alpha = 0.001

# momentum coefficient
beta = 0.9

# initial values
m = b = 0
p_m = p_b = 1

# lists to keep the error and the iteration values.
errors = []
Ns = []

# keeps previous error value.
prevError = math.inf

# Let's start iterating
for i in range(N):
    # compute current error value
    currentError = computeError(m, b)

    # append these value for plotting grapghic
    errors.append(currentError)
    Ns.append(i)

    # adapt alpha
    if i > 0:
        alpha = adaptAlpha(currentError, prevError, alpha)
    prevError = currentError

    if i % 10 == 0:
        print(i, "\t error:", currentError)

    m, b, p_m, p_b = gradientStepWithMomentum(alpha, m, b, p_m, p_b, beta)
# end of for loop

# drawing of graph.
plt.title("Error distribution for %s iteration" % N)
plt.ylabel("Error")
plt.xlabel("Iteration")
plt.plot(Ns, errors)
plt.show()
```

### References
* [github notes](https://github.com/atcemgil/notes/blob/master/GradientDescent.ipynb), Ali Taylan Cemgil, Bogaziçi University, İstanbul, Turkey

GDAlgos/Gradient Descent- Vanila- syntheticData.ipynb