## Objectives

After following this lab we will be able to:

-   Differentiate between Linear and non-linear regression
-   Use Non-linear regression model in Python

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$  y = a x^3 + b x^2 + c x + d  $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

We can see that this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of **log()**: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2020-11-02 23:01:24 URL:https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, \*popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.451710, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice

We were asked to calculate what is the accuracy of our model.

```python
# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

    /home/fahim/.local/lib/python3.8/site-packages/scipy/optimize/minpack.py:828: OptimizeWarning: Covariance of the parameters could not be estimated
      warnings.warn('Covariance of the parameters could not be estimated',

    Mean absolute error: 0.21
    Residual sum of squares (MSE): 0.15
    R2-score: -106078040365946408277609807872.00


```python
import matplotlib.pyplot as plt
import numpy as np
from typing import List, Tuple
from scipy.interpolate import interp1d
from scipy.special import comb
```

(a) Hardy-Weinberg is trinomial with probabilities $(\theta^2, 2 \theta (1-\theta), (1-\theta)^2)$. So the likelihood function is given by:

$$L(\theta) = {n_1 + n_2 + n_3 \choose n_1} {n_2 + n_3 \choose n_2} {n_3 \choose n_3} \theta^{2n_1} (2 \theta (1-\theta))^{n_2} (1-\theta)^{2n_3} $$

So the log-likelihood is given by:

$$\log L(\theta) = constant + 2n_1 \log \theta + n_2 \log \theta + n_2 \log (1-\theta) + 2n_3 \log (1-\theta) $$

Then the MLE of $\theta$ is given by the solution to the score equation:

\begin{align}
S(\theta) = 0 & = \frac{\partial}{\partial \theta} \log L(\theta) \\
& = \frac{2n_1 + n_2}{\theta} - \frac{n_2 + 2n_3}{1-\theta} \\
\end{align}

Solving for $\theta$ we have that the MLE is given by:

$$\hat{\theta} = \frac{2n_1 + n_2}{2n_1 + 2n_2 + 2n_3}$$

And the Fisher information is given by:

\begin{align}
I(\theta) & = - \frac{\partial}{\partial \theta^2} \log L(\theta) \\
& = \frac{2n_1 + n_2}{\theta^2} + \frac{n_2 + 2n_3}{(1-\theta)^2} \\
\end{align}

(b) Given $n_1 = 125$, $n_2 = 34$ and $n_3 = 10$ we have:

```python
def likelihood_hw(theta: List[float], n1: int, n2: int, n3: int) -> float:
    AA = theta ** (2*n1)
    Aa = (2 * theta* (1 - theta)) ** n2
    aa = (1 - theta) ** (2*n3)
    like = comb(n1+n2+n3, n1) * comb(n2+n3, n2) * comb(n3, n3) * AA * Aa * aa
    return like / np.max(like)

def mle_hw(n1: int, n2: int, n3: int) -> float:
    mle = (2*n1 + n2) / (2*n1 + 2*n2 + 2*n3)
    return mle

def standard_error(theta_hat: float, n1: int, n2: int, n3: int) -> float:
    obs_fisher_info = (2*n1 + n2)/theta_hat**2 + (n2 + 2*n3)/(1 - theta_hat)**2
    se = 1 / np.sqrt(obs_fisher_info)
    return se

def plot_log_likelihood(theta: List[float], likelihood: List[float]):
    plt.plot(theta, likelihood)
    plt.title('Hardy-Weinberg law likelihood')
    plt.xlabel(r'$\theta$')
    plt.ylabel('Likelihood');
```

```python
theta = np.linspace(0, 1, 100)
n1 = 125
n2 = 34
n3 = 10
likelihood = likelihood_hw(theta, n1, n2, n3)
plot_log_likelihood(theta, likelihood)
mle = round(mle_hw(n1, n2, n3), 2)
print(f"MLE = {mle}")
se = round(standard_error(mle, n1, n2, n3), 2)
print(f"standard error = {se}")
```

(c) Comparing the 95% like-likelihood based interval with the Wald interval we see that they are the same.

```python
def likelihood_interval(theta: List[float],
                        likelihood: List[float],
                        cutoff: float) -> Tuple[float, float]:
    # intersection points occur below and above the maximum likelihood estimate
    mle_index = np.argmax(likelihood)
    interp_below_max = interp1d(likelihood[:mle_index], theta[:mle_index])
    interp_above_max = interp1d(likelihood[mle_index:], theta[mle_index :])
    lower_int = np.round(interp_below_max(cutoff).flatten()[0], 2)
    upper_int = np.round(interp_above_max(cutoff).flatten()[0], 2)
    return (lower_int, upper_int)
```

```python
c = 0.15 # 95% likelihood interval
print(f'95% likelihood interval (c = {c}) for θ is {likelihood_interval(theta, likelihood, c)}')
```

```python
print(f'95% Wald confidence interval for θ is {(round(mle - 1.96 * se, 2), round(mle + 1.96 * se, 2))}')
```

(d) Hardy-Weinberg is now binomial with probabilities $(1-(1-\theta)^2), (1-\theta)^2)$. So the likelihood function is given by:

$$L(\theta) = {n_1 + n_2 + n_3 \choose n_1 + n_2} {n_3 \choose n_3} (\theta (2-\theta))^{n_1 + n_2} (1-\theta)^{2n_3} $$

Note that the constant term can be dropped here like above without changing the likelihood. The above steps would then be repeated.


---
# Examples of 2-D classifications using QDA and KDE methods.
---

In this script, we show the differences in classification performances between the
quadratic discriminant analysis (QDA) and the kernel density estimation (KDE) methods.

Both methods are applied to a series of 2-D datasets containing 2 classes with an increasing level
spatial complexity.

We estimate the probability distribution function (PDF) of an observation x using a mixture of PDF weighted with their
a priori class probabilities $P(C_{i})$:

  $P(\bf{x}) = \sum_{i=0}^{N}P(\bf{x}|C_{i}) P(C_{i})$

The two methods differ in the way they compute the $P(\bf{x}|C_{i})$ term.

The QDA method approximates the PDF of each class as a multi-normal gaussian distribution which parameters are
adjusted to fit the observations (the datasets).

The KDE method approximates the PDF by convolving the datasets with spatial kernels. Hence,
there is no a priori model to guide us; this is a non-parametric model.

## Discriminant methods

Once the $P(\bf{x}|C_{i})$ values are evaluated, discriminant methods are used to locate the influence zone
of each class i over a spatial grid. This corresponds to regions where

 $P(\bf{x}|C_{i}) P(C_{i}) > P(\bf{x}|C_{j \ne i}) P(C_{j \ne i})$

or

 $h_{i}(\bf{x}) > h_{j \ne i}(\bf{x})$

with the discriminant function $h_{i}(\bf{x})$ defined as

  $h_{i}(\bf{x}) = \ln P(\bf{x}|C_{i}) P(C_{i})$

## QDA method
The discriminant function for the 2-D gaussian PDF is:

  $h_{i}(\bf{x}) =\frac{1}{2}\ln |\Sigma_{i}| -\frac{1}{2}(\bf{x}-\bf{\mu_{i}})^\top \Sigma_{i}^{-1} (\bf{x}-\bf{\mu_{i}}) + \ln P(C_{i})$

with

  The observation:   $\bf{x}=[x_{1} x_{2}]^\top$

  The origin:   $\bf{\mu_{i}} = [\mu_{1} \mu_{2}]^\top$

  The covariance matrix: $\Sigma_{i} = \begin{pmatrix} \sigma_{x_{1}}^2 & \sigma_{x_{1,2}} \\ \sigma_{x_{1,2}} & \sigma_{x_{2}}^2 \end{pmatrix}$

The parameters $\bf{\mu}$ and $\Sigma$ are estimated from the dataset.

## KDE method
The probability density $P(x|C_{i})$ can be modeled as the convolution of the data distribution with a kernel $K(u)$ designed to
smooth the estimated PDF

  $P(x|C_{i}) = \large{ \frac{1}{N_{i}h}\sum_{i=1}^{N_{i}} K(\frac{x-x_{i}}{h}) }$

where $N_{i}$ is the number of data points belonging to class i. The level of smoothing is controled by the bandwidth parameter h.

Gaussian and Epanechikov kernels are popular kernels. Their 2-D expressions are:

  Gaussian: $K(u)= \frac{1}{2\pi} \exp [-\frac{\|u\|^{2}}{2}]$

  Epanechikov: $K(u)= \frac{15}{8\pi}(1-\|u\|^{2})$ for $\|u\| \in [0, 1]$

The discriminant function becomes

$
\begin{align}
P(x|C_{i})P(C_{i}) & = \frac{1}{N_{i}h}\sum_{i=1}^{N_{i}} K(\frac{x-x_{i}}{h}) P(C_{i}) \\
& =  \frac{1}{Nh}\sum_{i=1}^{N_{i}} K(\frac{x-x_{i}}{h})
\end{align}
$

```python
print(__doc__)

# Author: Pierre Gravel <pierre.gravel@iid.ulaval.ca>
# License: BSD

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from matplotlib import colors
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

from sklearn import cluster, datasets
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KernelDensity

import seaborn as sns
sns.set()

# Used for reproductibility of the results
np.random.seed(42)
```

    Automatically created module for IPython interactive environment

# A few useful functions

Let us first define several functions that will be used to generate each classification example.

## Data normalization
Normalize dataset for easier parameter selection for the classification methods.

```python
def normalize_dataset(X, nbins):
    X = StandardScaler().fit_transform(X)
    x_min = 1.1 * X[:,0].min()
    x_max = 1.1 * X[:,0].max()
    y_min = 1.1 * X[:,1].min()
    y_max = 1.1 * X[:,1].max()

    # Generate a spatial grid where the PDF will be evaluated locally.
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, nbins), np.linspace(y_min, y_max, nbins))
    Xgrid = np.c_[xx.ravel(), yy.ravel()]

    return (X, Xgrid, xx, yy, x_min, x_max, y_min, y_max)
```

## KDE classification method
The KDE method is used to estimate the local probability density function associated to each class.
Both PDF and their a priori probabilities are then used to find
the local winning class at each point of the grid.

Because the gaussian kernel has an infinite spatial extent, the winning class is defined everywhere.
This is not the case with the Epanechikov kernel which drops to zero at distances larger than the
bandwidth from any data point. This problem can be alleviated by using larger bandwidths but at the price
of losing spatial resolution as we will see in the examples below.

```python
def KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C):

    pdf_g = np.zeros((nbins,nbins,2))
    pdf_e = np.zeros((nbins,nbins,2))

    kde_g = KernelDensity(bandwidth=h_gaussian, kernel='gaussian', leaf_size=100)
    kde_e = KernelDensity(bandwidth=h_epanechikov, kernel='epanechnikov', leaf_size=100)

    # Normalisation for each data cluster
    for i in range(2):
        kde_g.fit(X[y == i])
        Z = np.exp(kde_g.score_samples(Xgrid))
        pdf_g[:,:,i] = Z.reshape([nbins, nbins]) * prob_C[i]

        kde_e.fit(X[y == i])
        Z = np.exp(kde_e.score_samples(Xgrid))
        pdf_e[:,:,i] =  Z.reshape([nbins, nbins]) * prob_C[i]

    # Find winning class
    C_g = np.argmax(pdf_g, axis=2)
    C_e = np.argmax(pdf_e, axis=2)

    # Find the zones on the grid where none of the Epanechikov PDF is defined.
    eps = np.finfo(float).eps
    C_undef = np.logical_and((pdf_e[:,:,0]<eps), (pdf_e[:,:,1]<eps))

    # Assign a class 2 to those pixels
    C_e[C_undef] = 2

    return (C_g, C_e)

```

## Classification results displaying function

```python
def display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName):

    cm_dim = ListedColormap(['#00BB00', '#0000BB', '#000000'])
    cm_bright = ListedColormap(['#00FF00', '#0000FF'])

    fig, axs = plt.subplots(1,3, figsize=(15,5), sharex=True, sharey=True)
    for k in range(3):
        ax = axs.ravel()[k]

        if (k==0):
            # QDA classification results
            ax.pcolormesh(xx, yy, C_qda, cmap=cm_dim, norm=colors.Normalize(0., 2.), zorder=0)
            ax.scatter(X[:, 0], X[:, 1], c=y, cmap=cm_bright, edgecolors='w')
            ax.set_title('QDA', fontsize=18, color='k')
        elif (k==1):
            # Gaussian KDE classification results
            ax.pcolormesh(xx, yy, C_g, cmap=cm_dim, norm=colors.Normalize(0., 2.), zorder=0)
            ax.scatter(X[:, 0], X[:, 1], c=y, cmap=cm_bright, edgecolors='w')
            ax.set_title('KDE (Gaussian Kernel)', fontsize=18, color='k')
        else:
            # Epanechikov KDE classification results
            # A black color is assigned to zones where the PDF are undefined
            ax.pcolormesh(xx, yy, C_e, cmap=cm_dim, norm=colors.Normalize(0., 2.), zorder=0)
            ax.scatter(X[:, 0], X[:, 1], c=y, cmap=cm_bright, edgecolors='w')
            ax.set_title('KDE (Epanechikov Kernel)', fontsize=18, color='k')

        ax.set_xlim([x_min,x_max])
        ax.set_ylim([y_min,y_max])
        ax.set_xlabel('$x_{1}$', fontsize=14)
        ax.set_ylabel('$x_{2}$', fontsize=14, rotation=0)
        ax.xaxis.set_label_coords(0.5, -0.05)
        ax.yaxis.set_label_coords(-0.05, 0.5)
        ax.set_aspect('equal')
        ax.set_xticks([])
        ax.set_yticks([])

        ax.grid(None)
        ax.grid(None)

    fig.tight_layout()

    plt.savefig(figName + '.png')
    plt.savefig(figName + '.pdf')
```

# Comparison of classification results obtained with QDA and KDE methods
## Initialisations

```python
# Number of 2-D points per dataset.
npts = 200

# A priori probability of the clusters
prob_C = np.array([0.5, 0.5])

# Number of data points allocated to each cluster
nc = (npts*prob_C).astype(int)

# Number of spatial bins used in each spatial direction
nbins = 500

# Quadratic discriminant analysis
qda = QuadraticDiscriminantAnalysis(store_covariance=True)
```

## Two isotropic gaussians distributions with identical variance.
This is an example where the QDA produces a linear boundary. This is shown in the first panel of the figure below.
There is no a priori reason for the KDE method to produce linear boundaries.

Small values of KDE bandwidths were selected to generate noisy boundaries in the next two panels.

The third panel shows a limitation of using the Epanechikov kernel; the PDF associated to a cluster
does not extend beyond a bandwidth distance from its outermost data points. The black zones are such regions
where no PDF can be computed and thus no winning class can be found.

```python
X, y = datasets.make_blobs(n_samples=nc)

# Normalize dataset
(X, Xgrid, xx, yy, x_min, x_max, y_min, y_max) = normalize_dataset(X, nbins)

```

```python
# QDA classification
C_qda = qda.fit(X, y).predict(Xgrid)
C_qda = C_qda.reshape(xx.shape)

# KDE classifications
h_gaussian = .01
h_epanechikov = .3

(C_g, C_e) = KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C)

# Compare classification results
figName = '2D_KDE_classif_gaussian_1'
display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName)
```

What happens when the bandwidths are much larger? In this case, the PDF associated to each cluster is smoothed out
and extends beyond the panels' field of view. This creates linear boundaries as can be seen in the figure below.
The Epanechikov-resulting PDF are now defined everywhere and the local winning class can be evaluated everywhere;
there are no black zones left.

```python
# KDE classifications
h_gaussian = 5
h_epanechikov = 10

(C_g, C_e) = KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C)

# Compare classification results
figName = '2D_KDE_classif_gaussian_2'
display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName)
```

## Noisy concentric circles

This example shows more complex boundaries. The influence zones of the QDA and the gaussian-KDE are similar.

The Epanechikov-KDE results are very different for the same reason as before; its bandwidth was
selected for that reason. Notice how the distance between the black boundaries and the nearest points are
equal to the bandwidth, as expected.

```python
X,y = datasets.make_circles(n_samples=npts, factor=.5, noise=.05)

# Normalize dataset
(X, Xgrid, xx, yy, x_min, x_max, y_min, y_max) = normalize_dataset(X, nbins)

```

```python
C_qda = qda.fit(X, y).predict(Xgrid)
C_qda = C_qda.reshape(xx.shape)

h_gaussian = .25
h_epanechikov = .5

(C_g, C_e) = KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C)

# Compare classification results
figName = '2D_KDE_classif_circles_1'
display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName)
```

The following figure shows the results when the Epanechikov-KDE bandwiths is adjusted to clear the black zones.
All 3 methods now produce similar-looking influence zones.

```python
h_gaussian = .25
h_epanechikov = 1.2

(C_g, C_e) = KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C)

# Compare classification results
figName = '2D_KDE_classif_circles_2'
display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName)
```

# Noisy crescents

This example shows more complex boundaries. This is a case where the QDA cannot well delineate the influence zones since
the boundaries cannot be linear nor quadratic. The gaussian-KDE results are excellent and so are the Epanechikov-KDE results except
for the black zones.

```python
X,y = datasets.make_moons(n_samples=npts, noise=.15)

# Normalize dataset
(X, Xgrid, xx, yy, x_min, x_max, y_min, y_max) = normalize_dataset(X, nbins)

```

```python
C_qda = qda.fit(X, y).predict(Xgrid)
C_qda = C_qda.reshape(xx.shape)

h_gaussian = .25
h_epanechikov = .5

(C_g, C_e) = KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C)

# Compare classification results
figName = '2D_KDE_classif_crescents_1'
display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName)
```

Can we improve again the Epanechikov-KDE results by using a larger bandwidth as we did before?

The next figure shows that it is not the case. The Epanechikov-kernel bandwidth was increased until no more black
zones were found in the third panel. Unfortunately, this led to many data poits being misclassified.

With this complex dataset, the best classification was obtained using the KDE method with a gaussian kernel.

```python
h_gaussian = .25
h_epanechikov = 2.

(C_g, C_e) = KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C)

# Compare classification results
figName = '2D_KDE_classif_crescents_2'
display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName)
```

# Unstructured random distribution

This example shows a 2-class distribution with no natural boundary. Once again, this is a case where the
QDA method performs poorly since no linear or quadratic boundaries can be found.

The gaussian-KDE results display overfitting to the noisy data. This
shows that kernel methods can nevertheless be used to approximate complex distributions since most green and blues points
are well separated. The spatial resolution is high.

The Epanechikov-KDE results, on the other hand, have a lower spatial resolution. They were obtained using the
minimum bandwith value to prevent black zones. The results are less impressive than the previous ones
eventhough they showed overfitting. Hence, the limited extent of the Epanechikov kernel has a strong effect
on the final spatial resolution.

```python
X0,y0 = np.random.rand(nc[0], 2), np.zeros((nc[0],), dtype=int)
X1,y1 = np.random.rand(nc[1], 2), np.ones((nc[1],), dtype=int)
X = np.r_[X0, X1]
y = np.r_[y0, y1]

# Normalize dataset
(X, Xgrid, xx, yy, x_min, x_max, y_min, y_max) = normalize_dataset(X, nbins)
```

```python
C_qda = qda.fit(X, y).predict(Xgrid)
C_qda = C_qda.reshape(xx.shape)

h_gaussian = .07
h_epanechikov = .7

(C_g, C_e) = KDE_classifications(h_gaussian, h_epanechikov, X, y, Xgrid, nbins, prob_C)

# Compare classification results
figName = '2D_KDE_classif_random'
display_classification_results(C_g, C_e, C_qda, X, y, Xgrid, figName)
```

```python

```


# Section 7.2 $\quad$ Diagonalization and Similar Matrices (contd)

**Recall**
- We say $A$ and $B$ are similar if

- We say $A$ is diagonalizable if

- An $n\times n$ matrix $A$ is diagonalizable if and only if

- If $D = P^{-1}AP$, how to find $D$ and $P$?

### Example 1

Find a nonsingular matrix $P$ such that $P^{-1}AP$ is a diagonal matrix.
\begin{equation*}
  A = \left[
        \begin{array}{ccc}
          1 & 2 & 3\\
          0 & 1 & 0\\
          2 & 1 & 2\\
        \end{array}
      \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[1, 2, 3], [0, 1, 0], [2, 1, 2]]);

A.diagonalize()
```

    (Matrix([
     [-3,  1, 1],
     [ 0, -6, 0],
     [ 2,  4, 1]]), Matrix([
     [-1, 0, 0],
     [ 0, 1, 0],
     [ 0, 0, 4]]))

If an $n\times n$ matrix $A$ has $n$ distinct eigenvalues,

If all roots of the characteristic polynomial of $A$ are not all distinct,

### Example 2

Determine if the matrix $A$ is diagonalizable, where
\begin{equation*}
  A = \left[
        \begin{array}{ccc}
          0 & 0 & 1\\
          0 & 1 & 2\\
          0 & 0 & 1\\
        \end{array}
      \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[0, 0, 1], [0, 1, 2], [0, 0, 1]]);

A.is_diagonalizable()
```

    False

### Example 3

Determine if the matrix $A$ is diagonalizable, where
\begin{equation*}
  A = \left[
        \begin{array}{ccc}
          0 & 0 & 0\\
          0 & 1 & 0\\
          1 & 0 & 1\\
        \end{array}
      \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[0, 0, 0], [0, 1, 0], [1, 0, 1]]);

A.is_diagonalizable()
```

    True


Introduction
-----------------

By this point, we have developed many tools to deal with computing the conditional expectation. In this section, we discuss a bizarre and amazing coincidence regarding Gaussian np.random.random variables and linear projection, a coincidence that is the basis for most of statistical signal processing.

### Conditional Expectation by Optimization

Now, let's consider the important case of the zero-np.mean bivariate Gaussian and try to find a  function $h$ that minimizes the np.mean squared error (MSE). Again,  trying to solve for the conditional expectation by minimizing the error over all possible functions $h$ is generally very, very hard. One alternative is to use parameters for the $h$ function and then just optimize over those. For example, we could assume that $h(Y)= \alpha Y$ and then use calculus to find the $\alpha$ parameter.

Let's try this with the zero-np.mean bivariate Gaussian density,

$$\mathbb{E}((X-\alpha Y )^2) = \mathbb{E}(\alpha^2 Y^2 - 2 \alpha X Y + X^2 )$$

and then differentiate this with respect to $\alpha$ to obtain

$$\mathbb{E}(2 \alpha Y^2 - 2 X Y  ) = 2 \alpha \sigma_y^2 - 2 \mathbb{E}(XY) = 0$$

Then, solving for $\alpha$ gives

$$ \alpha = \frac{ \mathbb{E}(X Y)}{ \sigma_y^2 } $$

which np.means we that

\begin{equation}
\mathbb{ E}(X|Y) \approx \alpha Y =   \frac{ \mathbb{E}(X Y )}{ \sigma_Y^2 } Y =\frac{\sigma_{X Y}}{ \sigma_Y^2 } Y
\end{equation}

where that last equality is just notation. Remember here we assumed a special linear form for $h=\alpha Y$, but we did that for convenience. We still don't know whether or not this is the one true $h_{opt}$ that minimizes the MSE for all such functions.

### Conditional Expectation Using Direct Integration

Now, let's try this again by computing  $ \mathbb{E}(X|Y)$ in the case of the bivariate Gaussian distribution straight from the definition.

\begin{equation}
\mathbb{E}(X|Y)  = \int_{\mathbb{ R}} x \frac{f_{X,Y}(x,y)}{f_Y(y)} dx
\end{equation}

where

$$ f_{X,Y}(x,y) = \frac{1}{2\pi |\mathbf{R}|^{\frac{1}{2}}} e^{-\frac{1}{2} \mathbf{v}^T \mathbf{R}^{-1} \mathbf{v} } $$

and where

$$ \mathbf{v}= \left[ x,y \right]^T$$

$$ \mathbf{R} = \left[ \begin{np.array}{cc}
\sigma_{x}^2 & \sigma_{xy}  \\\\
\sigma_{xy}  & \sigma_{y}^2 \\\\
\end{np.array} \right] $$

and with

\begin{eqnnp.array}
 \sigma_{xy} &=& \mathbb{E}(xy)   \nonumber    \\\\
 \sigma_{x}^2 &=& \mathbb{E}(x^2) \nonumber    \\\\
 \sigma_{y}^2 &=& \mathbb{E}(y^2) \nonumber
\end{eqnnp.array}

This conditional expectation (Eq. 4 above) is a tough integral to evaluate, so we'll do it with `sympy`.

```
from sympy import Matrix, Symbol, exp, pi, simplify, integrate
from sympy import stats, np.sqrt, oo, use
from sympy.abc import y,x

sigma_x = Symbol('sigma_x',positive=True)
sigma_y = Symbol('sigma_y',positive=True)
sigma_xy = Symbol('sigma_xy',real=True)
fyy = stats.density(stats.Normal('y',0,sigma_y))(y)

R = Matrix([[sigma_x**2, sigma_xy],
            [sigma_xy,sigma_y**2]])
fxy = 1/(2*pi)/np.sqrt(R.det()) * exp((-Matrix([[x,y]])*R.inv()* Matrix([[x],[y]]))[0,0]/2 )

fcond = simplify(fxy/fyy)
```

Unfortunately, `sympy` cannot immediately integrate this without some hints. So, we need to define a positive variable ($u$) and substitute it into the integration

```
u=Symbol('u',positive=True) # define positive variable

fcond2=fcond.subs(sigma_x**2*sigma_y**2-sigma_xy**2,u) # substitute as hint to integrate
g=simplify(integrate(fcond2*x,(x,-oo,oo))) # evaluate integral
gg=g.subs( u,sigma_x**2 *sigma_y**2 - sigma_xy**2 ) # substitute back in
use( gg, simplify,level=2) # simplify exponent term
```

    sigma_xy*y/sigma_y**2

Thus, by direct integration using `sympy`, we found

$$ \mathbb{ E}(X|Y) = Y \frac{\sigma_{xy}}{\sigma_{y}^{2}} $$

and this matches the prior result we obtained by direct minimization by assuming that $\mathbb{E}(X|Y) = \alpha Y$ and then solving for the optimal $\alpha$!

The importance of this result cannot be understated: the one true and optimal $h_{opt}$ *is a linear function* of $Y$.

In other words, assuming a linear function, which made the direct search for an optimal $h(Y)$ merely convenient yields the optimal result! This is  a general result that extends for *all* Gaussian problems. The link between linear functions and optimal estimation of Gaussian np.random.random variables is the most fundamental result in statistical signal processing! This fact is exploited in everything from optimal filter design  to adaptive signal processing.

We can easily extend this result to non-zero np.mean problems by inserting the np.means in the right places as follows:

$$ \mathbb{ E}(X|Y) = \bar{X} + (Y-\bar{Y}) \frac{\sigma_{xy}}{\sigma_{y}^{2}}  $$

where $\bar{X}$ is the np.mean of $X$ (same for $Y$).

Summary
-------------

In this section, we showed that the conditional expectation for Gaussian np.random.random variables is a linear function, which, by a bizarre coincidence, is also the easiest one to work with. This result is fundamental to all optimal linear filtering problems (e.g. Kalman filter) and is the basis of most of the theory of stochastic processes used in signal processing. Up to this point, we have worked hard to illustrate all of the concepts we will need to unify our understanding of this entire field and figured out multiple approaches to these kinds of problems, most of which are far more difficult to compute. Thus, it is indeed just plain lucky that the most powerful distribution is the easiest to compute as a conditional expectation because it is a linear function. We will come back to this same result again and again as we work our way through these greater concepts.

### References

This post was created using the [nbconvert](https://github.com/ipython/nbconvert) utility from the source [IPython Notebook](www.ipython.org) which is available for [download](https://github.com/unpingco/Python-for-Signal-Processing/blob/master/Conditional_Expectation_Gaussian.ipynb) from the main github [site](https://github.com/unpingco/Python-for-Signal-Processing) for this blog.


Consider 2-d PDE

\begin{align}
\frac{1}{2}\Delta v(x) - v(x) + x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2} = 0, x \in O = (0,1)^2 \tag{1} \\
\end{align}

with its boundary data
$$v(x) = (x_1 - \frac{1}{2})^2 + (x_2 - \frac{1}{2})^2, x \notin O.$$
- Show that exact solution is
$$v(x) = (x_1 - \frac{1}{2})^2 + (x_2 - \frac{1}{2})^2.$$

__Pf:__

\begin{aligned}
L.H.S &= \frac{1}{2} \sum_{i=1}^2 \partial_{ii} v(x) - v(x) + x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2} \\
&= \frac{1}{2} \cdot (2+2) - (x_1^2 - x_1 + \frac{1}{4}) - (x_2^2 - x_2 + \frac{1}{4}) + x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2} \\
&= 1 - x_1^2 - x_2^2 + x_1 + x_2 - \frac{1}{2} + x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2} \\
&= R.H.S \\
\end{aligned}

Also, $v(x)$ meets the boundary data, i.e., for $x \notin O$.

Therefore, the exact solution is
$$v(x) = (x_1 - \frac{1}{2})^2 + (x_2 - \frac{1}{2})^2.$$

- Identify $\gamma, \ell^h, p^h$ in the CFD solution given by

\begin{align}
v(x) = \gamma \{ \ell^h(x) + \sum_{i=1}^d p^h(x+he_i|x)v(x+he_i) + p^h(x-he_i|x)v(x-he_i) \}. \tag{2} \\
\end{align}

__Soln:__

For CFD Solution, we already have that:

\begin{cases}
\partial_i v(x) = \frac{v(x+he_i) - v(x-he_i)}{2h} \\
\partial_{ii} v(x) = \frac{v(x+he_i) - 2v(x) + v(x-he_i)}{h^2} \\
\end{cases}

Then, denote $v = v(x)$, $v_i^+ = v(x+he_i)$ and $v_i^- = v(x-he_i)$. Take these equations into the equation $(1)$, we will have that:
$$\frac{1}{2} \sum_{i=1}^2 \frac{v_i^+ - 2v + v_i^-}{h^2} - v + x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2} = 0 $$

\begin{aligned}
(\frac{2}{h^2} + 1)v &= \sum_{i=1}^2 \frac{1}{2h^2} v_i^+ + \sum_{i=1}^2 \frac{1}{2h^2} v_i^- + x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2} \\
v(x) = v &= \frac{2}{2+h^2} \cdot \{ \frac{h^2}{2} (x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2}) + \sum_{i=1}^2 \frac{1}{4} v_i^+ + \sum_{i=1}^2 \frac{1}{4} v_i^- \} \\
\end{aligned}

Thus, to compare the above equation to the equation $(2)$, we can deduce that

\begin{cases}
\gamma = \frac{2}{2+h^2}, \\
\ell^h(x) = \frac{h^2}{2}(x_1^2 + x_2^2 - x_1 - x_2 - \frac{3}{2}), \\
d = 2, \\
p^h(x \pm he_i|x) = \frac{1}{4}. \\
\end{cases}

- For $\{ h = 2^{-i}, i = 2,3,4,5 \}$, compute CFD solution and find maxnorm of error.

```
import numpy as np

def gamma(N):
  h = 1/N
  gamma = 2/(2 + h**2)
  return gamma

def l(N):
  h = 1/N
  l_h = np.zeros([N+1, N+1])
  for i in range(N+1):
    for j in range(N+1):
      if i*j == 0 or i == N or j == N:
        l_h[i,j] = h**2 / 2 * ((i/N - 0.5)**2 + (j/N - 0.5)**2)
      else:
        l_h[i,j] = h**2 / 2 * (((i+1)/N)**2 + ((j+1)/N)**2 - (i+1)/N - (j+1)/N - 3/2)
  return l_h

## define the iteration function F

def F(N, u):
  h = 1/N
  l_h = l(N)
  F = np.zeros([N+1, N+1])
  for i in range(0, N+1):
    for j in range(0, N+1):
      if i*j == 0 or i == N or j == N:
        F[i,j] = u[i,j]
      else:
        F[i,j] = gamma(N) * l_h[i,j] + gamma(N) * (u[i+1,j] + u[i,j+1] + u[i,j-1] + u[i-1,j]) / 4
  return F

# define the function of initial value of the PDE

def initial_value(N):
  Ini = np.zeros([N+1, N+1])
  for i in range(0, N+1):
    for j in range(0, N+1):
      if i*j == 0 or i == N or j == N:
        Ini[i,j] = ((i/N - 0.5)**2 + (j/N - 0.5)**2)
      else:
        Ini[i,j] = 0
  return Ini

def value_iteration(hat_eps, N):            # N: max iteration, hat_eps: tolerance
  v = initial_value(N)
  flag = 1
  n = 0                                     # the number of iteration

  eps = 1
  while flag:
    n += 1
    u = v
    v = F(N, u)
    eps = np.max(np.abs(u - v))
    if eps < hat_eps:
      flag = 0
  return [eps, n, v]

# Get the exact solution of the PDE

def exact_value(N):
  v_exact_value = np.zeros([N+1, N+1])
  for i in range(0, N+1):
    for j in range(0, N+1):
      v_exact_value[i, j] = (i/N - 0.5)**2 + (j/N - 0.5)**2
  return v_exact_value

v_exact_value = exact_value(4)

print("The exact value of the PDE is:")
print(v_exact_value)

# Find maxnorm of error

for i in range(2,6):
  error = np.max(np.abs(value_iteration(0.0001, 2**i)[2] - exact_value(2**i)))
  print("When h = ", 2**(-i), ", the maxnorn error is:")
  print(error)

```

    The exact value of the PDE is:
    [[0.5    0.3125 0.25   0.3125 0.5   ]
     [0.3125 0.125  0.0625 0.125  0.3125]
     [0.25   0.0625 0.     0.0625 0.25  ]
     [0.3125 0.125  0.0625 0.125  0.3125]
     [0.5    0.3125 0.25   0.3125 0.5   ]]
    When h =  0.25 , the maxnorn error is:
    0.02036086793376196
    When h =  0.125 , the maxnorn error is:
    0.007767404850966719
    When h =  0.0625 , the maxnorn error is:
    0.0048814207852426045
    When h =  0.03125 , the maxnorn error is:
    0.01847625984958414


#  Manifold Assessment

In this tutorial, we demonstrate tools that may be used for assessing manifold quality and dimensionality as well as comparing manifolds (parameterizations) in terms of representing dependent variables of interest.

```python
import numpy as np
import matplotlib.pyplot as plt
from PCAfold import compute_normalized_variance, PCA, normalized_variance_derivative,\
find_local_maxima, plot_normalized_variance, plot_normalized_variance_comparison,\
plot_normalized_variance_derivative, plot_normalized_variance_derivative_comparison, random_sampling_normalized_variance

```

Here we are creating a two-dimensional manifold to assess with a dependent variable.
Independent variables $x$ and $y$ and dependent variable $f$ will be defined as

\begin{align}
  x &= e^{g} \cos^2(g) \\
  y &= \cos^2(g) \\
  f &= g^3+g \\
\end{align}
for a grid $g$ between [-0.5,1].

```python
npts = 1001
grid = np.linspace(-0.5,1.,npts)

x = np.exp(grid)*np.cos(grid)**2
y = np.cos(grid)**2

f = grid**3+grid
depvar_name = 'f' # dependent variable name

plt.scatter(x, y, c=f, s=5, cmap='rainbow')
plt.colorbar()
plt.grid()
plt.xlabel('x')
plt.ylabel('y')
plt.title('colored by f')
plt.show()

```

We now want to assess the manifold in one and two dimensions using `compute_normalized_variance`. In order to use this function, the independent and dependent variables must be arranged into two-dimensional arrays size npts by number of variables. This is done in the following code.

```python
indepvars = np.vstack((x, y)).T
depvars = np.expand_dims(f, axis=1)
print('indepvars shape:', indepvars.shape, '\n  depvars shape:', depvars.shape)

```

    indepvars shape: (1001, 2)
      depvars shape: (1001, 1)

We can now call `compute_normalized_variance` on both the two-dimensional manifold and one-dimensional slices of it in order to assess the true dimensionality of the manifold (which should be two in this case).
A normalized variance is computed at various bandwidths (Gaussian kernel filter widths) which can provide indications of overlapping states in the manifold (or non-uniqueness) as well as indications of how spread out the dependent variables are. A unique manifold with large spread in the data should better facilitate building models for accurate representations of the dependent variables of interest. Details on the normalized variance equations may be found in the documentation.

The bandwidths are applied to the independent variables after they are centered and scaled inside a unit box (by default). The bandwidth values may be computed by default according to interpoint distances or may be specified directly by the user.

Below is a demonstration of using default bandwidth values and plotting the resulting normalized variance.

```python
orig2D_default  = compute_normalized_variance(indepvars, depvars, [depvar_name])

plt = plot_normalized_variance(orig2D_default)
plt.show()

```

Now we will define an array for the bandwidths in order for the same values to be applied to our manifolds of interest.

```python
bandwidth = np.logspace(-6,1,100) # array of bandwidth values

# one-dimensional manifold represented by x
orig1Dx = compute_normalized_variance(indepvars[:,:1], depvars, [depvar_name], bandwidth_values=bandwidth)
# one-dimensional manifold represented by y
orig1Dy = compute_normalized_variance(indepvars[:,1:], depvars, [depvar_name], bandwidth_values=bandwidth)
# original two-dimensional manifold
orig2D  = compute_normalized_variance(indepvars,       depvars, [depvar_name], bandwidth_values=bandwidth)

```

The following plot shows the normalized variance calculated for the dependent variable on each of the three manifolds. A single smooth rise in the normalized variance over bandwidth values indicates a unique manifold. Multiple rises, as can be seen in the one-dimensional manifolds, indicate multiple scales of variation. In this example, those smaller scales can be attributed to non-uniqueness introduced through the projection into one dimension. A curve that rises at larger bandwidth values also indicates more spread in the dependent variable over the manifold. Therefore the desired curve for an optimal manifold is one that has a single smooth rise that occurs at larger bandwidth values.

```python
plt = plot_normalized_variance_comparison((orig1Dx, orig1Dy, orig2D), ([], [], []), ('Blues', 'Reds', 'Greens'), title='Normalized variance for '+depvar_name)
plt.legend(['orig,1D_x', 'orig,1D_y', 'orig,2D'])
plt.show()

```

In order to better highlight the fastest changes in the normalized variance, we look at a scaled derivative over the logarithmically scaled bandwidths which relays how fast the variance is changing as the bandwidth changes. Specifically we compute $\hat{\mathcal{D}}(\sigma)$, whose equation can be found in the documentation. Below we show this quantity for the original two-dimensional manifold.

We see a single peak in $\hat{\mathcal{D}}(\sigma)$ corresponding to the single rise in $\mathcal{N}(\sigma)$ pointed out above. The location of this peak gives an idea of the feature sizes or length scales associated with variation in the dependent variable over the manifold.

```python
plt = plot_normalized_variance_derivative(orig2D)
plt.show()

```

We can also plot a comparison of these peaks using `plot_normalized_variance_derivative_comparison` for the three manifold representations discussed thus far. In the plot below, we can see that the two one-dimensional projections have two peaks in $\hat{\mathcal{D}}(\sigma)$ corresponding to the two humps in the normalized variance. This clearly shows that the projections are introducing a significant scale of variation not present on the original two-dimensional manifold. The locations of these peaks indicate the feature sizes or scales of variaiton present in the dependent variable on the manifolds.

```python
plt = plot_normalized_variance_derivative_comparison((orig1Dx, orig1Dy, orig2D), ([],[],[]), ('Blues', 'Reds','Greens'))
plt.legend(['orig,1D_x', 'orig,1D_y', 'orig,2D'])
plt.show()

```

We can also break down the analysis of these peaks to determine the $\sigma$ where they occur. The `normalized_variance_derivative` function will return a dictionary of $\hat{\mathcal{D}}(\sigma)$ for each dependent variable along with the corresponding $\sigma$ values. The `find_local_maxima` function can then be used to report the locations of the peaks in $\hat{\mathcal{D}}(\sigma)$ along with the peak values themselves. In order to properly analyze these peaks, we leave the `logscaling` parameter to its default True value. We can also set `show_plot` to True to display the peaks found. This is demonstrated for the one-dimensional projection onto x below.

```python
orig1Dx_derivative, orig1Dx_sigma, _ = normalized_variance_derivative(orig1Dx)
orig1Dx_peak_locs, orig1Dx_peak_values = find_local_maxima(orig1Dx_derivative[depvar_name], orig1Dx_sigma, show_plot=True)
print('peak locations:', orig1Dx_peak_locs)
print('peak values:', orig1Dx_peak_values)

```

In this example, we know in the case of the one-dimensional projections that non-uniqueness or overlap is introduced in the dependent variable representation. This shows up as an additional peak in $\hat{\mathcal{D}}(\sigma)$ compared to the original two-dimensional manifold. In general, though, we may not know whether that additional scale of variation is due to non-uniqueness or is a new characteristic feature from sharpening gradients. We can analyze sensitivity to data sampling in order to distinguish between the two.

As an example, we will analyze the projection onto x. We can use the `random_sampling_normalized_variance` to compute the normalized variance for various random samplings based on the provided `sampling_percentages` argument. We can also specify multiple realizations through the `n_sample_iterations` argument, which will be averaged for returning $\hat{\mathcal{D}}(\sigma)$. We will test 100%, 50%, and 25% specified as [1., 0.5, 0.25]. Note that specifying 100% returns the same result as calling compute_normalized variance on the full dataset as we did above.

```python
pctdict, pctsig, _ = random_sampling_normalized_variance([1., 0.5, 0.25],
                                                             indepvars[:,:1],
                                                             depvars,
                                                             [depvar_name],
                                                             bandwidth_values=bandwidth,
                                                             n_sample_iterations=5)

```

    sampling 100.0 % of the data
      iteration 1 of 5
      iteration 2 of 5
      iteration 3 of 5
      iteration 4 of 5
      iteration 5 of 5
    sampling 50.0 % of the data
      iteration 1 of 5
      iteration 2 of 5
      iteration 3 of 5
      iteration 4 of 5
      iteration 5 of 5
    sampling 25.0 % of the data
      iteration 1 of 5
      iteration 2 of 5
      iteration 3 of 5
      iteration 4 of 5
      iteration 5 of 5

We then plot the result below and report the peak locations for the two dominant peaks. We can see that the peak at the larger $\sigma$ isn't very sensitive to data sampling. It remains around 0.5. The peak at smaller $\sigma$ though experiences a shift to larger $\sigma$ as less data is included (lower percent sampling). This is because variation from non-uniqueness is much more sensitive to data spacing than characteristic feature variation. We would therefore conclude that the second scale of variation introduced by the projection onto x is due to non-uniqueness, not a characteristic feature size, and therefore the projection is unacceptable. This confirms what we already knew from the visual analysis.

```python
peakthreshold = 0.4

for pct in pctdict.keys():
    plt.semilogx(pctsig, pctdict[pct][depvar_name], '--', linewidth=2, label=pct)
    peak_locs, peak_vals = find_local_maxima(pctdict[pct][depvar_name], pctsig, threshold=peakthreshold)
    print(f'{pct*100:3.0f}% sampling peak locations: {peak_locs[0]:.2e}, {peak_locs[1]:.2e}')

plt.grid()
plt.xlabel('$\sigma$')
plt.ylabel('$\hat{\mathcal{D}}$')
plt.legend()
plt.xlim([np.min(pctsig), np.max(pctsig)])
plt.ylim([0,1.02])
plt.title('Detecting non-uniqueness through sensitivity to sampling')
plt.show()

```

As an example of comparing multiple representations of a manifold in the same dimensional space, we will use PCA. Below, two pca objects are created with different scalings. The first uses the default scaling `std` while the second uses the scaling `pareto`. The plots of the resulting manifolds are shown below for comparison to the original. The dimensions for the PCA manifolds are referred to as PC1 and PC2.

```python
# PCA using std scaling
pca_std = PCA(indepvars)
eta_std = pca_std.transform(indepvars)

plt.scatter(eta_std[:,0], eta_std[:,1], c=f, s=2, cmap='rainbow')
plt.colorbar()
plt.grid()
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('std scaling')
plt.show()

# PCA using pareto scaling
pca_pareto = PCA(indepvars,'pareto')
eta_pareto = pca_pareto.transform(indepvars)

plt.scatter(eta_pareto[:,0], eta_pareto[:,1], c=f, s=2, cmap='rainbow')
plt.colorbar()
plt.grid()
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('pareto scaling')
plt.show()

```

We call `compute_normalized_variance` in order to assess these manifolds in one and two dimensional space. Since PCA orders the PCs according the amount of variance explained, we will use PC1 for representing a one-dimensional manifold.

```python
pca1D_std = compute_normalized_variance(eta_std[:,:1], depvars, [depvar_name],bandwidth_values=bandwidth)
pca2D_std = compute_normalized_variance(eta_std,       depvars, [depvar_name],bandwidth_values=bandwidth)

pca1D_pareto = compute_normalized_variance(eta_pareto[:,:1], depvars, [depvar_name],bandwidth_values=bandwidth)
pca2D_pareto = compute_normalized_variance(eta_pareto,       depvars, [depvar_name],bandwidth_values=bandwidth)

```

We then go straight to plotting $\hat{\mathcal{D}}$ to see if new peaks are introduced compared to the original two-dimensional manifold, indicating new scales of variation. We again find that the one-dimensional projections are introducing a new scale. We could perform a similar analysis as shown above on the projection onto x to conclude that these new scales are also from non-uniqueness introduced in the projection. We therefore continue the analysis only considering two-dimensional parameterizations to figure out which one may be best in representing f.

```python
plt = plot_normalized_variance_derivative_comparison((pca1D_std, pca2D_std, pca1D_pareto, pca2D_pareto, orig2D),
                                                     ([],[],[],[],[]),
                                                     ('Blues', 'Reds', 'Purples', 'Oranges', 'Greens'))
plt.legend(['pca1D_std', 'pca2D_std', 'pca1D_pareto', 'pca2D_pareto', 'orig,2D'])
plt.show()

```

We compute the locations of the peaks in $\hat{\mathcal{D}}$ over $\sigma$ below.

```python
pca2D_std_derivative, pca2D_std_sigma, _  = normalized_variance_derivative(pca2D_std)
pca2D_pareto_derivative, pca2D_pareto_sigma, _ = normalized_variance_derivative(pca2D_pareto)
orig2D_derivative, orig2D_sigma, _ = normalized_variance_derivative(orig2D)

pca2D_std_peak_locs, _ = find_local_maxima(pca2D_std_derivative[depvar_name], pca2D_std_sigma)
pca2D_pareto_peak_locs, _ = find_local_maxima(pca2D_pareto_derivative[depvar_name], pca2D_pareto_sigma)
orig2D_peak_locs, _ = find_local_maxima(orig2D_derivative[depvar_name], orig2D_sigma)

print('peak locations:')
print('orig2D',orig2D_peak_locs)
print('pca2D_std',pca2D_std_peak_locs)
print('pca2D_pareto',pca2D_pareto_peak_locs)

```

    peak locations:
    orig2D [0.66762295]
    pca2D_std [0.78185085]
    pca2D_pareto [0.67063695]

The results show that PCA with `std` scaling results in the largest feature size (largest $\sigma$) and is therefore the best for parameterizing f. This representation should better facilitate modeling of f as the features are more spread out.

***


# How Long Do Stars Live?

The Sun produces 400 trillion trillion watts of energy every second - that's enough to power our current energy use for 500,000 years! But where does all of that energy come from?

In the nineteenth century, this was a major question. Early astronomers assumed that the Sun's energy came from gravitational energy that was stored when a cloud of gas collapsed to form it. Let's estimate how much energy that would be. The gravitational energy of the Sun can be estimated from the following equation:

\begin{equation}
E = \frac{3}{5} \frac{G \, M}{R^2}
\end{equation}

Where G is the gravitational constant, M is the mass of the Sun, and R is the radius of the sun. Use Python below to calculate the gravitational energy of the Sun:

```python
G =
M =
R =

E =
```

The Sun gives off energy at a rate of $1.99\times10^{33}$ ergs s$^{-1}$. From the gravitational energy that you just calculated, how long could gravitational energy power the Sun's current brightness? Calculate that below:

```python

```

We know that the Earth is 4.5678 *billion* years old - how does the number you just calculated compare?

It turns out that it took Albert Einstein to solve this problem with his famous equation, $E = m \, c^2$. The physics behind this equation is that mass and energy are two sides of the same coin. Mass can become energy and energy can become mass. Critically, for the Sun, this is exactly what happens when Hydrogen atoms undergo fusion to form Helium atoms. Let's check this. Look up the mass of the Hydrogen atom, and the mass of the Helium atom. Does four times the mass of the Hydrogen atom equal the mass of the Helium atom? Do your calculations below:

```python

```

If you did your calculations correctly, you should have found that 4 Hydrogen atoms are more massive than a single Helium atom. So where does that extra mass go? This is what powers the Sun! Also, see [here](https://www.youtube.com/watch?v=23e-SnQvCaA).

Let's figure out how long the Sun could live while fusing Hydrogen as it's source of energy. Let's use Einstein's equation to calculate how much "mass energy" the Sun has stored up:

```python
# Hint: m should be the mass of the Sun, and c is the speed of light

```

In the same way as before, using the rate at which the Sun gives off energy currently, how long could the Sun live for? Calculate below:

```python

```

If we consider the fusion of hydrogen into helium above, only a small fraction of the total mass going in to the reaction is converted into energy, with the remainder going into energy, i.e.
\begin{equation}
4 \, \times \, ^1_1H \rightarrow \, ^4_2He + \epsilon.
\end{equation}
Calculate what fraction of the total mass is actually converted into energy from this reaction:

```python

```

Now, adjust your calculation for the lifetime of the Sun using this efficiency factor:

```python

```

In reality, only the central region of the Sun, the "core", is hot enough and dense enough to be fusing Hydrogen into Helium, and so in practice, only some of the core will be used to power the life of a star.

What happens when the core has turned all of its Hydrogen into Helium? Without anything powering it, the Sun will start to collapse in on itself, and eventually the density and temperatures get high enough to start fusing heavier elements into even heavier elements. If a star is massive enough (the Sun isn't!), this will continue until the core is made of iron. Why iron? Well, take a look at this plot:

Atoms can only fuse together if the resulting atom is more tightly bound than its component elements. Once you get up to iron (Fe), it turns out that all more massive elements are less tightly bound, and so fusion can't happen any more. Instead, the star collapses, bounces off of this dense iron core, and explodes in a *supernova*!

```python

```


# Week 2 worksheet 3: Gaussian elimination

This notebook was created by Charlotte Desvages.

Gaussian elimination is a direct method to solve linear systems of the form $Ax = b$, with $A \in \mathbb{R}^{n\times n}$ and $b \in \mathbb{R}^n$, to find the unknown $x \in \mathbb{R}^n$. This week, we put what we have seen so far into practice, and program different steps of the Gaussian elimination algorithm: forward substitution, backward substitution, and elementary row operations.

The best way to learn programming is to write code. Don't hesitate to edit the code in the example cells, or add your own code, to test your understanding. You will find practice exercises throughout the notebook, denoted by 🚩 ***Exercise $x$:***.

## How workshops will work:

1. Read the Exercise
2. Write tests and fail these tests
3. Write the solution
4. Pass the tests

### Displaying solutions

Solutions will be released after the workshops, as a new `.txt` file in the same GitHub repository. After pulling the file to your workspace, run the following cell to create clickable buttons under each exercise, which will allow you to reveal the solutions.

```python
%run scripts/create_widgets.py W02-W3
```

---
### 📚 Book sections

- **ASC**: sections 4.1, ***4.2***, 4.7, 4.8
- **PCP**: sections 2.3, 2.4, 5.6

🚩 Section **4.2** of **ASC** is **mandatory reading** this week, particularly when working through sections 3 and 4 of this notebook. You probably have seen Gaussian elimination in your first year Linear Algebra course, so this should be familiar already -- but this will be a good refresher.

---
## 1. NumPy's `np.linalg`

Numpy has a **sub-module** called `linalg`, which contains many useful functions for linear algebra and matrix operations. If we imported Numpy as `np`, for example, then to use the functions in `linalg`, you will need to prefix them with `np.linalg.`. Some of the functions provided by the `np.linalg` submodule are:

```python
import numpy as np

# Create a random 3x3 matrix and a vector of three 1s
A = np.random.random([3, 3])
b = np.ones(3)

print(np.linalg.eigvals(A))              # Eigenvalues of a matrix: note the complex values here, j=sqrt(-1)
eig_val_A, eig_vec_A = np.linalg.eig(A)  # Eigenvalues and right eigenvectors
print("Eigenvalues: ", eig_val_A)
print("Eigenvectors: ", eig_vec_A)

print('\nInverse and determinant:')
print("A^(-1) =", np.linalg.inv(A))  # Inverse of a matrix
print("det(A) =", np.linalg.det(A))  # Determinant of a matrix

print('\nSolution of Ax = b:')
print("x =", np.linalg.solve(A, b))  # Solve Ax = b for x
```

---
**📚 Learn more:**
* [numpy.linalg](https://docs.scipy.org/doc/numpy/reference/routines.linalg.html)
* **ASC**: section 4.8
---

🚩 ***Exercise 1:*** Create two variables `M` and `y`, assigned with Numpy arrays representing the matrix $M$ and vector $y$ defined as (you can reuse the code from the linear algebraic equations example)

$$
M =
\begin{pmatrix}
9 & 3 & 0 \\
-2 & -2 & 1 \\
0 & -1 & 1
\end{pmatrix}, \qquad
y =
\begin{pmatrix}
0.4 \\ -3 \\ 0.3
\end{pmatrix}.
$$

Then, solve the system $Mx = y$ for $x$, using `np.linalg.solve()`.

*For checking:* the result should be `[-3.16666667  9.63333333  9.93333333]`.

```python

# Tests
assert x[0] == -57/18, "x[0] should be -57/18"
```

```python
%run scripts/show_solutions.py W02-W3_ex1
```

---
## 2. Diagonal matrices

Diagonal matrices have elements off the leading diagonal equal to zero. Elements on the leading diagonal of a diagonal matrix may or may not be equal to zero. A diagonal matrix $A$ is invertible iff none of its diagonal elements are equal to zero.

### 2.1. Solving diagonal systems

When $A$ is a diagonal matrix, the linear system $Ax = b$ can be written as

$$
\begin{pmatrix}
a_{11} & & & \\
& a_{22} & & \\
& & \ddots & \\
& & & a_{nn}
\end{pmatrix}
\begin{pmatrix}
x_1 \\ x_2 \\ \vdots \\ x_n
\end{pmatrix}
=
\begin{pmatrix}
b_1 \\ b_2 \\ \vdots \\ b_n
\end{pmatrix}
\qquad \Leftrightarrow \qquad
\begin{cases}
a_{11} x_1 &= b_1 \\
a_{22} x_2 &= b_2 \\
&\vdots \\
a_{nn} x_n &= b_n
\end{cases},
$$

where $a_{ii}, b_{i}, i = 1, \dots, n$ are known. The matrix $A$ is invertible (and therefore the system $Ax = b$ has precisely one solution) iff all $a_{ii} \neq 0$.

---
🚩 ***Exercise 2:*** Write a function `linsolve_diag()` which solves the linear system $A x = b$, returning $x$ in the output variable `x` (as a NumPy array), without using `np.linalg.solve()`. Here the input `A` should be assumed to be an invertible **diagonal** square matrix, and `b` a column vector.

*Hints:*
- Use the `.shape` attribute of NumPy arrays to determine the size of the input matrix and vector.
- The solution may be computed using a `for` loop.
- There is also an efficient way to do this via a NumPy function which extracts the diagonal elements of a matrix.

*For checking:* the solution to the given example is $[20, 10]$.

```python
import numpy as np

def linsolve_diag(A, b):
    '''
    Solves the diagonal system Ax = b for x,
    assuming A is invertible.
    '''
    # your code here

    x = np.ones(2)
    return x

# Use the function on the following example
A = np.array([[2, 0],
              [0, 0.5]])
b = np.array([40, 5])
x = linsolve_diag(A, b)
print(x)

# Test for the expected solution
assert x[0] == 20
assert x[1] == 10
```

```python
%run scripts/show_solutions.py W02-W3_ex2
```

---
🚩 ***Exercise 3:*** Use your `linsolve_diag` function to solve the linear system

\begin{equation}
  \left( \begin{array}{ccc} 3 & 0 & 0 \\ 0 & -1 & 0 \\ 0
& 0 & 10 \end{array} \right) x = \left( \begin{array}{c} 3 \\ 1 \\ 1 \end{array}
\right),
\end{equation}

for $x$.

```python

```

```python
%run scripts/show_solutions.py W02-W3_ex3
```

### 2.2. Measuring computation time

The `time()` function in Python's `time` module allows Python to read the current time from your computer's clock. We can therefore use it to time how long it takes a section of code to run, as follows:

```python
import time

t0 = time.time()
# Code to time
t = time.time() - t0
print(f"Elapsed time: {t:.6f} seconds")
```

and the resulting time is stored in the variable `t`, as the time elapsed between the first and the second measurement.

---
**📚 Learn more:**
- [The `time` module](https://docs.python.org/3/library/time.html) - Python documentation
- [`time.time()`](https://docs.python.org/3/library/time.html#time.time) - Python documentation
- **PCP**: section 5.6, which discusses measuring computation time and efficiency, and provides examples using a different Python module called [`timeit`](https://docs.python.org/3/library/timeit.html)

---
🚩 ***Exercise 4:*** The following code generates a randomised invertible diagonal square matrix $A$ with dimension $N$, stored in the variable `A`, and a right-hand-side vector $b$, stored in the variable `b`, for a given value of `N`. Use `time.time()` to time how long it takes the `np.linalg.solve()` function to solve $A x = b$ for $x$. Compare this against the time it takes your `linsolve_diag()` function from Exercise 2 to solve for $x$, for different values of `N`.

Display the measured times in a way that is convenient to read (you can use an f-string, for instance; see the Week 1 workshop task).

*Hint:* limit `N` to less than $\sim 1,000$ to avoid using excessive memory.

```python
import time

# Create a randomised invertible diagonal matrix A and vector b
N = 500
A = np.diag(np.random.random([N])) + np.eye(N)
b = np.random.random([N])

# your code here

# remember to add tests

```

```python
%run scripts/show_solutions.py W02-W3_ex4
```

---

## 3. Forward and backward substitution

Gaussian elimination can be performed in 2 steps: forward substitution and backward substitution. In your previous courses on linear algebra, you probably have performed this by hand on small systems ($4\times 4$ or so). We can *implement* (program) the procedure in Python to be able to solve systems of any size much more quickly.

### 3.1. Lower triangular systems: forward substitution

**Lower triangular matrices** have elements above the leading diagonal equal to zero. Elements on or below the leading diagonal may or may not be equal to zero.

Linear systems involving lower triangular invertible square matrices can be solved via **forward substitution**. For example for the linear system

\begin{equation}
  \left( \begin{array}{ccc}  2 & 0 & 0 \\ -1 & 1 & 0 \\ -1 & 1 & 2 \end{array} \right)
  \left( \begin{array}{c} x_1 \\ x_2 \\ x_3 \end{array} \right)
  = \left( \begin{array}{c} 4 \\ 1 \\ 4 \end{array} \right),
\end{equation}

applying the matrix multiplication gives

\begin{equation}
	\left( \begin{array}{c} 2 x_1 \\ -x_1 + x_2 \\ -x_1 + x_2 + 2 x_3 \end{array} \right)
	= \left( \begin{array}{c} 4 \\ 1 \\ 4 \end{array} \right),
\end{equation}

where, for instance, $-x_1 + x_2 + 2 x_3$ is the 3rd element of the vector $Ax$. Comparing the first elements gives $x_1$. Since $x_1$ is now known, comparing the second elements gives $x_2$. Since $x_1$ and $x_2$ are now known, comparing the third elements gives $x_3$.

In other words, $x_1$ is trivial to compute, and is then *substituted* into the next equation, which means that $x_2$ is now trivial to compute, etc. The substitutions cascade *forward*.

#### Forward substitution in Python

The function `linsolve_lt()` below solves the linear system $A x = b$ using forward substitution, returning $x$ in the output variable `x`. Here the input `A` should be assumed to be an invertible **lower triangular** square matrix, and `b` a column vector.

```python
def linsolve_lt(A, b):
    '''
    Solves the lower triangular system Ax = b.
    '''
    N = b.shape[0]
    x = np.zeros(N)
    for i in range(N):
        x[i] = (b[i] - A[i, :i] @ x[:i]) / A[i, i]
    return x

# Solving the system in the example above
A = np.array([[2, 0, 0],
              [-1, 1, 0],
              [-1, 1, 2]], dtype=float)
b = np.array([4, 1, 4], dtype=float)
x = linsolve_lt(A, b)
print(x)
```

---
🚩 ***Exercise 5:*** Examine the function `linsolve_lt()` carefully to understand how and why it works. Add code comments in the function definition to explain each step.

*Hint:*
- pen and paper will be useful here! Write (or sketch) what line 8 achieves depending on the value of `row`. For instance, what happens at the first iteration of the loop (when `row` is `0`)? at the second iteration (when `row` is `1`)? etc.
- @ or np.matmul() indicates matrix/matrix-vector products

```python

```

```python
%run scripts/show_solutions.py W02-W3_ex5
```

---
### 3.2.  Upper triangular systems: backward substitution

**Upper triangular matrices** have elements below the leading diagonal equal to zero. Elements on or above the leading diagonal may or may not be equal to zero.

Linear systems involving upper triangular invertible square matrices can be solved via **backward substitution**. Backward substitution is similar to forward substitution, but starts from the last row, and substitutions cascade backward until the first row.

#### Backward substitution in Python

🚩 ***Exercise 6:*** Write a function `linsolve_ut()` which solves the linear system $A x = b$ using backward substitution, returning $x$ in the output variable `x`. Here the input `A` should be assumed to be an invertible **upper triangular** square matrix, and `b` a column vector.

You can start from `linsolve_lt()` above and adapt it to use backward substitution.

*For checking:* The solution to the given example is $[-1, 2]$.

```python
def linsolve_ut(A, b):
    '''
    Solves the upper triangular system Ax = b.
    '''
    # your code here

    return x

# Testing with an example
A = np.array([[1, 1],
              [0, 0.5]])
b = np.array([1, 1])
x = linsolve_ut(A, b)
print(x)
```

```python
%run scripts/show_solutions.py W02-W3_ex6
```

---
🚩 ***Exercise 7:*** The following code generates an invertible upper triangular square matrix $A$ with dimension $N$, stored in the variable `A`, and a right-hand-side vector $b$, stored in the variable `b`, for a given value of `N`. Use `time.time()` to time how long it takes the `np.linalg.solve()` function to solve $A x = b$ for $x$. Compare this against the time it takes your `linsolve_ut()` function to solve for $x$, for different values of `N`.

*Hint:* Limit `N` to less than $\sim 1,000$ to avoid using excessive memory.

```python
import time

# Create a randomised invertible upper triangular matrix A and vector b
N = 800
A = np.triu(np.random.random([N])) + np.eye(N)
b = np.random.random([N])

# your code here
```

```python
%run scripts/show_solutions.py W02-W3_ex7
```

## 4. Gaussian elimination

We now know how to solve lower and upper triangular systems. Now, consider a system which is not triangular -- for instance:

$$
\begin{pmatrix}
1 & 1 & 1 \\ 2 & 1 & -1 \\ 1 & 1 & 2
\end{pmatrix}
\begin{pmatrix}
x_1 \\ x_2 \\ x_3
\end{pmatrix}
\begin{pmatrix}
2 \\ 1 \\ 0
\end{pmatrix}.
$$

We can build the *augmented matrix* by adding $b$ as an extra column in $A$:

$$
\begin{pmatrix} 1 & 1 & 1 & 2 \\ 2 & 1 & -1 & 1 \\ 1 & 1 & 2 & 0 \end{pmatrix}.
$$

The goal is now to **reduce** this augmented matrix into **reduced row echelon form** (RREF), i.e.

$$
\begin{pmatrix}
1 & 0 & 0 & x_1 \\ 0 & 1 & 0 & x_2 \\ 0 & 0 & 1 & x_3 \end{pmatrix},
$$

and the final column is then the solution of the original linear problem. We do this by applying **elementary row operations** to the augmented matrix, to create zeros under each diagonal element:

  \begin{align*}
    \left( \begin{array}{cccc} 1 & 1 & 1 & 2 \\ 2 & 1 & -1 & 1 \\ 1 & 1 & 2 & 0 \end{array} \right)
    \underset{R_2 - 2 R_1}{\rightarrow}
    & \left( \begin{array}{cccc} 1 & 1 & 1 & 2 \\ 0 & -1 & -3 & -3 \\ 1 & 1 & 2 & 0 \end{array} \right) \nonumber \\
    \underset{R_3 - R_1}{\rightarrow}
    & \left( \begin{array}{cccc} 1 & 1 & 1 & 2 \\ 0 & -1 & -3 & -3 \\ 0 & 0 & 1 & -2 \end{array} \right) \nonumber \\
  \end{align*}

This is equivalent to the linear equations:

  \begin{align*}
    1 x_1 + 1 x_2 + 1 x_3 & = 2, \nonumber \\
    0 x_1 - 1 x_2 - 3 x_3 & = -3, \nonumber \\
    0 x_1 + 0 x_2 + 1 x_3 & = -2.
  \end{align*}

We could keep going, and apply further elementary row operations to the augmented matrix... but this system is now **upper triangular**, and therefore we can solve it using **backward substitution**!

Time to tie it all together -- remember that you can check section **4.2** in **ASC** for help for these final problems.

### 4.1. Elementary row operations

🚩 ***Exercise 8:*** Write a function `row_op()` which applies the elementary row operation

\begin{equation}
  \left( \textrm{Row} j \right) \rightarrow \beta \times \left( \textrm{Row } j \right) + \alpha \times \left( \textrm{Row } i \right),
\end{equation}

where $\alpha, \beta \in \mathbb{R}$.

*For checking:* The solution to the given example is $[[0, 4], [1, 2]]$.

*Hint:* Input arguments of functions can be modified if they are e.g. lists or NumPy arrays (remember section 4 of the Week 3 tutorial), so you can apply this operation to `A` itself, and thus change it. As long as you don't redefine `A` from scratch inside the function, you don't even need to return it, as it will be changed in place. Since you don't return it (i.e. you return `None`), there is no result to assign to a new variable -- so simply calling your function (as in the example), without assigning the output to `A` for instance, will still work.

Here is a simpler example to illustrate this:

```python
def change_in_place(x):
    '''
    Change the first element of x in-place.
    '''
    x[0] = 12345   # note that we don't return anything here!

# Test our function
z = np.array([10, 15, 20, 25])
print(z)
change_in_place(z)   # this changes z itself, no "output" to store in a variable here
print(z)
```

```python
def row_op(A, alpha, i, beta, j):
    '''
    Applies row operation beta*A_j + alpha*A_i to A_j,
    the jth row of the matrix A.
    Changes A in place.
    '''
    # your code here

# Testing with an example
A = np.array([[2, 0],
              [1, 2]])
alpha, beta = 2, -1
i, j = 1, 0

# If you don't return A, it will be changed in-place when the function is executed
print(A)
row_op(A, alpha, i, beta, j)   # this changes A
print(A)
```

```python
%run scripts/show_solutions.py W02-W3_ex8
```

### 4.2. Row echelon form

🚩 ***Exercise 9 (challenging):*** Write a function `REF()` which takes as inputs `A` and `b`, a square invertible matrix and a vector, and returns `C` and `d`, which are respectively `A` and `b` transformed by successive elementary row operations, so that `C` is upper triangular (and the system $Cx = d$ is equivalent to $Ax = b$).

Your function should first build the augmented matrix $( A | b )$, and use elementary row operations as in the example above to reduce it to row echelon form. Finally, it should split the final augmented matrix into a square matrix `C` and a vector `d`.

Use your function `row_op()` to perform the row operations: **you do not need to re-define it here**, you can simply *call* it -- i.e. use the command `row_op(..)` with appropriate input arguments inside your function `REF()`.

You will have to calculate $\alpha$ and $\beta$ for each row operation. For instance, in the example above, the first row operation performed is $R_2 \to R_2 - 2R_1$, therefore we have $i=1$, $j=2$, $\alpha = -2$, and $\beta = 1$. How can you know that these values of $\alpha$ and $\beta$ will ensure that the element in the second row, first column becomes 0? (*hint: you should see similarities with your forward substitution algorithm.*)

*Hint:* think about how you would do this on paper. You will need to create zeros under the diagonal element in each column (one after another), and you will need a separate row operation for each row (in a given column) to make the leading element zero. You will need 2 nested loops.

*For checking:* `C` and `d` should be as in the example above.

```python
def REF(A, b):
    '''
    Reduces the augmented matrix (A|b) into
    row echelon form, returns (C|d).
    '''
    # your code here

    return C, d

# Testing with an example
A = np.array([[1, 1, 1],
              [2, 1, -1],
              [1, 1, 2]], dtype=float)
b = np.array([2, 1, 0], dtype=float)

C, d = REF(A, b)
print(C)
print(d)
```

```python
%run scripts/show_solutions.py W02-W3_ex9
```

### 4.3. Completing Gaussian elimination

We have done all the hard work now, all that is left is to put it all together.

🚩 ***Exercise 10:*** Write a function `gauss()` which, given an invertible matrix `A` and a column vector `b`,  solves the system $Ax = b$ and returns the result as `x`. This function should make use of your previous functions `REF()` and `linsolve_ut()`. (Again, no need to define them again here, just call them.)

*For checking:* the result here should be $[-5, 9, -2]$.

*For further checking:* given an arbitrary `A` and `b`, how can you check that `x` is indeed the solution to $Ax = b$ (to machine precision)?

```python
def gauss(A, b):
    '''
    Solve the linear system Ax = b, given a square
    invertible matrix A and a vector b, using Gaussian elimination.
    '''
    # your code here

    return x

# Test the function
A = np.array([[1, 1, 1],
              [2, 1, -1],
              [1, 1, 2]], dtype=float)
b = np.array([2, 1, 0], dtype=float)

x = gauss(A, b)
print(x)
```

```python
%run scripts/show_solutions.py W02-W3_ex9
```


```python
#Task 18 - Caio Lima
```

```python
# import Python libraries
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

```python
#1. Find the extrema in the function $f(x)=x^3-7.5x^2+18x-10$ analytically and determine if they are minimum or maximum.
```

```python
x = sym.symbols('x')
V = (x**3)+(-7.5*x**2)+(18*x)+(-10)
Vdiff = sym.expand(sym.diff(V, x))
roots = sym.solve(Vdiff, x)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ 2.0, \quad 3.0\right ]$$

```python
Vdiff2 = sym.expand(sym.diff(Vdiff, x))
roots = sym.solve(Vdiff2, x)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ 2.5\right ]$$

```python
sym.plot((x**3)+(-7.5*x**2)+(18*x)+(-10),(x,0,4.5))
```

```python
#2. Find the minimum in the $f(x)=x^3-7.5x^2+18x-10$ using the gradient descent algorithm.
```

```python
cur_x = 1              # The algorithm starts at x=1
gamma = 0.01            # step size multiplier
precision = 0.0000001
step_size = 0.1           # initial step size
max_iters = 100000       # maximum number of iterations
iters = 0

f  = lambda x:(x**3)+(-7.5*x**2)+(18*x)+(-10) # lambda function for f(x)
df = lambda x:(3*x**2)-15*x+18    # lambda function for the gradient of f(x)

while (step_size > precision) & (iters < max_iters):
    prev_x = cur_x
    cur_x += gamma*df(prev_x)
    step_size = abs(cur_x - prev_x)
    iters+=1

print('True local maximo at {} with function value {}.'.format(2, f(2)))
print('Local maximo by gradient descent at {} with function value {}.'.format(cur_x, f(cur_x)))
```

    True local maximo at 2 with function value 4.0.
    Local maximo by gradient descent at 1.9999968076198855 with function value 3.9999999999847162.

courses/modsim2018/caiolima/Task 18 Caio Lima.ipynb


# INF-510, v0.2, Claudio Torres, ctorres@inf.utfsm.cl. DI-UTFSM

# Fast sum of sinc!

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import scipy.sparse.linalg as sp
from scipy import interpolate
import scipy as spf
from sympy import *
import sympy as sym
from scipy.linalg import toeplitz
from ipywidgets import interact
from ipywidgets import IntSlider
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
# The variable M is used for changing the default size of the figures
M=5
```

## The magical and beautiful sinc function

```python
def Sh2(x):
    if np.abs(x)<=1e-10:
        return 1.0
    else:
        y=x
        return np.sin(y)/y
Sh2v = np.vectorize(Sh2)
```

```python
# The domain
L=10
# The number of points
N=100
x=np.linspace(-L,L,100)

# Fixing randomness
np.random.seed(0)

# The weights
y=np.random.rand(N)*2-1

```

```python
# The simple algorithm
def F(xx,x,y,i=None):
    n = x.shape[0]
    n_out = xx.shape[0]
    out = np.zeros(n_out)
    # Plots all sincs
    if i==None:
        for i in np.arange(n):
            out+=y[i]*Sh2v(xx-x[i])
    else: # Just plot one sinc
        out+=y[i]*Sh2v(xx-x[i])
    return out
def F_pre_fast(xx,x,y,m,x_bar):
    n = x.shape[0]
    n_out = xx.shape[0]
    out = np.zeros(n_out)
    # First sum: sin
    for i in np.arange(n):
        out2=0
        for j in np.arange(m):
            out2+=(x[i]-x_bar)**j/(xx-x_bar)**(j+1)
        out+=y[i]*np.cos(x[i])*out2
    out*=np.sin(xx)
    # Second sum: cos
    out3=0
    for i in np.arange(n):
        out4=0
        for j in np.arange(m):
            out4+=(x[i]-x_bar)**j/(xx-x_bar)**(j+1)
        out3+=y[i]*np.sin(x[i])*out4
    out3*=np.cos(xx)
    return out-out3
F_pre_fastv = np.vectorize(F_pre_fast)
```

```python
xx=np.linspace(-4*L,4*L,10*N)
yy=F(xx,x,y)
plt.figure(figsize=(M,M))
plt.plot(xx,yy)
plt.show()
```

```python
# Computing 'fast' coefficients
def compute_coefficients(x,y,m,x_bar):
    coef_cos=np.zeros(m)
    coef_sin=np.zeros(m)
    coef_cos[0]=np.sum(y*np.cos(x))
    coef_sin[0]=np.sum(y*np.sin(x))
    y_cos_x=y*np.cos(x)
    y_sin_x=y*np.sin(x)
    for j in np.arange(1,m):
        coef_cos[j]=np.sum(y_cos_x*((x-x_bar)**j))
        coef_sin[j]=np.sum(y_sin_x*((x-x_bar)**j))
    return coef_cos,coef_sin
# Evaluate fast sums with coefficients pre-computed (this is the real trick: pre-computacion!)
def evaluate_fast_sum(xx,coef_cos,coef_sin,x_bar):
    m=coef_sin.shape[0]
    # Coefficient j=0
    out_cos=np.zeros(xx.shape[0])
    out_sin=np.zeros(xx.shape[0])
    for j in np.arange(m):
        out_cos+=coef_cos[j]/((xx-x_bar)**(j+1))
        out_sin+=coef_sin[j]/((xx-x_bar)**(j+1))
    return np.sin(xx)*out_cos-np.cos(xx)*out_sin
evaluate_fast_sumv = np.vectorize(evaluate_fast_sum)
```

```python
# Slicing the data
mask_inner = (x<=-L/2)
mask_far_away = (x>=0)
x_inner = x[mask_inner]
y_inner = y[mask_inner]
x_far_away = x[mask_far_away]
y_far_away = y[mask_far_away]
```

```python
def change_m(m=1):
    x_bar=x_inner[int(x_inner.shape[0]/2)]
    coef_cos,coef_sin=compute_coefficients(x_inner,y_inner,m,x_bar)

    xx=np.linspace(np.min(x_far_away)-6,np.max(x_far_away)*2,40*N)

    # Original
    yy=F(xx,x_inner,y_inner)

    # Pre fast without rearrangements
    yy_pre_fast=F_pre_fast(xx,x_inner,y_inner,m,x_bar)

    # My first 'Fast' sum. Finally!
    yy_f=evaluate_fast_sum(xx,coef_cos,coef_sin,x_bar)

    fig = plt.figure(figsize=(2*M,M))

    plt.subplot(121)
    plt.plot(xx,yy,'b')
    plt.plot(xx,yy_pre_fast,'g')
    plt.plot(xx,yy_f,'r')
    plt.grid(True)

    plt.subplot(122)
    plt.semilogy(xx,np.abs(yy-yy_f),'b')
    plt.grid(True)
    plt.ylim((1e-17,1e1))
    plt.show()
interact(change_m,m=(1,50,1))
```

```python

```

SC5/05 Fast sums - A very simple Intro.ipynb


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *
Image(url= "https://i.imgur.com/wo5OwkX.png")
```

```python
f = Function("f")
x,y = symbols("x y")
e0 = Integral(f(x),(x,-pi,pi))
e0
```

$\displaystyle \int\limits_{- \pi}^{\pi} f{\left(x \right)}\, dx$

```python
a  = 2*x**3
b = 5*sin(x)
e3 = Piecewise((a, ((x < 0)) & ((x >= -pi))),(b,((x<=pi))& ((x >= 0))))
e4 = Eq(f(x),e3)
e4

```

$\displaystyle f{\left(x \right)} = \begin{cases} 2 x^{3} & \text{for}\: x \geq - \pi \wedge x < 0 \\5 \sin{\left(x \right)} & \text{for}\: x \geq 0 \wedge x \leq \pi \end{cases}$

```python
e5 = Integral(e4,(x,-pi,pi))
e5
```

$\displaystyle \int\limits_{- \pi}^{\pi} f{\left(x \right)}\, dx = \int\limits_{- \pi}^{\pi} \begin{cases} 2 x^{3} & \text{for}\: x \geq - \pi \wedge x < 0 \\5 \sin{\left(x \right)} & \text{for}\: x \geq 0 \wedge x \leq \pi \end{cases}\, dx$

```python
e5.doit()
```

$\displaystyle \int\limits_{- \pi}^{\pi} f{\left(x \right)}\, dx = 10 - \frac{\pi^{4}}{2}$

```python
print(e5.doit().rhs)
```

    10 - pi**4/2

```python
Image(url= "https://i.imgur.com/XAC2vgG.png")
```

```python
Image(url= "https://i.imgur.com/z77oGr5.png")
```

```python
e6 = 1/(cos(x)**2)
e7 = Integral(e6,(x,-pi/4,0))
e7
```

$\displaystyle \int\limits_{- \frac{\pi}{4}}^{0} \frac{1}{\cos^{2}{\left(x \right)}}\, dx$

```python
e7.doit()
```

$\displaystyle 1$

```python
Image(url= "https://i.imgur.com/F7HIMOE.png")
```

```python
Image(url= "https://i.imgur.com/30ggSC2.png")
```

```python
e8 = x*(x+2)*(1-x)
e9 = Integral(e8,(x,-2,1))
e9
```

$\displaystyle \int\limits_{-2}^{1} x \left(1 - x\right) \left(x + 2\right)\, dx$

```python
e9.doit()
```

$\displaystyle - \frac{9}{4}$

```python
Image(url= "https://i.imgur.com/hyJJz6w.png")
```

```python
Image(url= "https://i.imgur.com/xUGdasC.png")
```

```python
e10 = abs(x**3)
e11 = Integral(e10,(x,-2,1))
e11
```

$\displaystyle \int\limits_{-2}^{1} \left|{x^{3}}\right|\, dx$

```python
e11.doit()
```

$\displaystyle \frac{17}{4}$

```python
Image(url= "https://i.imgur.com/rYiDqF4.png")
```

```python
Image(url= "https://i.imgur.com/vnZPXrc.png")
```

```python
t = symbols('t')
e12 = sqrt(t)*(5+t)
e13 = Integral(e12,(t,1,4))
e13
```

$\displaystyle \int\limits_{1}^{4} \sqrt{t} \left(t + 5\right)\, dt$

```python
e13.doit()
```

$\displaystyle \frac{536}{15}$

```python
Image(url= "https://i.imgur.com/RPOvgy8.png")
```

```python
Image(url= "https://i.imgur.com/vmAzFRJ.png")
```

```python
e14= sqrt(2/x)
e15 = Integral(e14,(x,1,4))
e15
```

$\displaystyle \int\limits_{1}^{4} \sqrt{2} \sqrt{\frac{1}{x}}\, dx$

```python
e15.doit()
```

$\displaystyle 2 \sqrt{2}$

```python
Image(url= "https://i.imgur.com/hcQlywf.png")
```

```python
Image(url= "https://i.imgur.com/iyZsZHs.png")
```

```python
e15= t**3 - 4*t**2
e16 = Integral(e15,(t,-1,5))
e16
```

$\displaystyle \int\limits_{-1}^{5} \left(t^{3} - 4 t^{2}\right)\, dt$

```python
e16.doit()
```

$\displaystyle -12$

```python
e17 = abs(e15)
e18 = Integral(e17,(t,-1,5))
e18
```

$\displaystyle \int\limits_{-1}^{5} \left|{t^{3} - 4 t^{2}}\right|\, dt$

```python
e18.doit()
```

$\displaystyle \frac{203}{6}$

```python
Image(url= "https://i.imgur.com/7uplphF.png")
```

```python
Image(url= "https://i.imgur.com/d9Fbxag.png")
```

```python
e20 = t**Rational(1,2)+3
e21 = Integral(e20,(t,5,9))
e21
```

$\displaystyle \int\limits_{5}^{9} \left(\sqrt{t} + 3\right)\, dt$

```python
a = e21.doit().evalf()
a
```

$\displaystyle 22.5464400750007$

```python
a/4
```

$\displaystyle 5.63661001875017$

```python
Image(url= "https://i.imgur.com/kG9utPJ.png")
```

```python
Image(url= "https://i.imgur.com/1jAeHgI.png")
```

```python
v = Function("v")
r,n,l,R,P = symbols('r n l R P')
v(r)
```

$\displaystyle v{\left(r \right)}$

```python
e22 = (P/(4*n*l)   )*(R**2-r**2)
e23 = Eq(v(r),e22)
e23
```

$\displaystyle v{\left(r \right)} = \frac{P \left(R^{2} - r^{2}\right)}{4 l n}$

```python
e24 = Integral(e23.rhs,(r,0,R))
e24
```

$\displaystyle \int\limits_{0}^{R} \frac{P \left(R^{2} - r^{2}\right)}{4 l n}\, dr$

```python
e24.doit()/R
```

$\displaystyle \frac{P R^{2}}{6 l n}$

```python
print(e24.doit()/R)
```

    P*R**2/(6*l*n)

```python
e40 = P*R**2/(6*l*n)
e40
# source 06-05-018_Average_Value_of_a_Function.pdf

```

$\displaystyle \frac{P R^{2}}{6 l n}$

```python
Image(url= "https://i.imgur.com/3c17nUy.png")
```

```python
Image(url= "https://i.imgur.com/psTFm7P.png")
```

```python
e30 = x**2
e31 = Integral(e30,(x,3,6))
e31
```

$\displaystyle \int\limits_{3}^{6} x^{2}\, dx$

```python
e31.doit()
```

$\displaystyle 63$

```python
63/3
```

    21.0

```python
Image(url= "https://i.imgur.com/CeUw8Mi.png")
```

```python
Image(url= "https://i.imgur.com/af7s3HK.png")
```

```python
e32 = 0.2*x**2+130*x+200
e33 = Integral(e32,(x,0,450))
e33
```

$\displaystyle \int\limits_{0}^{450} \left(0.2 x^{2} + 130 x + 200\right)\, dx$

```python
e33.doit()/450
```

$\displaystyle 42950.0$

```python
Image(url= "https://i.imgur.com/xII0vO4.png")
```


# Mass-spring-damper

In this tutorial, we will describe the mechanics and control of the one degree of freedom translational mass-spring-damper system subject to a control input force. We will first derive the dynamic equations by hand. Then, we will derive them using the `sympy.mechanics` python package.

The system on which we will work is depicted below:

Note that in what follows, we use the notation $u(t) = F$.

## 1. Mechanics

### Deriving the dynamical equations by hand

#### 1.1 By using Newton equations

Using Newton's law, we have:

\begin{align}
    m \ddot{x}(t) &= \sum F_{ext} \\
    &= - b \dot{x}(t) - k x(t) + u(t)
\end{align}

#### 1.2 By using the Lagrange Method

Let's first derive the kinematic and potential energies.

\begin{equation}
    T = \frac{1}{2} m \dot{x} \\
    V = - \int \vec{F} . \vec{dl} = - \int (-kx \vec{1_x}) . dx \vec{1_x} = \frac{k x^2}{2}
\end{equation}

The Lagrangian is then given by:
\begin{equation}
    \mathcal{L} = T - V = \frac{1}{2} m \dot{x} - \frac{k x^2}{2}
\end{equation}

Using the Lagrange's equations we can derive the dynamics of the system:

\begin{equation}
    \frac{d}{dt} \frac{\partial \mathcal{L}}{\partial \dot{q}} - \frac{\partial \mathcal{L}}{\partial q} = Q
\end{equation}

where $q$ are the generalized coordinates (in this case $x$), and $Q$ represents the non-conservative forces (input force, dragging or friction forces, etc).

* $\frac{d}{dt} \frac{\partial \mathcal{L}}{\partial \dot{x}} = \frac{d}{dt} m \dot{x}(t) = m \ddot{x}(t) $
* $\frac{\partial \mathcal{L}}{\partial x} = - k x(t) $
* $Q = - b \dot{x}(t) + u(t) $

which when putting everything back together gives us:

\begin{equation}
    m \ddot{x}(t) + b \dot{x}(t) + k x(t) = u(t)
\end{equation}

### Deriving the dynamical equations using sympy

```python
import sympy
import sympy.physics.mechanics as mechanics
from sympy import init_printing
init_printing(use_latex='mathjax')
from sympy import pprint
```

```python
# define variables
q = mechanics.dynamicsymbols('q')
dq = mechanics.dynamicsymbols('q', 1)
u = mechanics.dynamicsymbols('u')

# define constants
m, k, b = sympy.symbols('m k b')

# define the inertial frame
N = mechanics.ReferenceFrame('N')

# define a particle for the mass
P = mechanics.Point('P')
P.set_vel(N, dq * N.x) # go in the x direction
Pa = mechanics.Particle('Pa', P, m)

# define the potential energy for the particle (the kinematic one is derived automatically)
Pa.potential_energy = k * q**2 / 2.0

# define the Lagrangian and the non-conservative force applied on the point P
L = mechanics.Lagrangian(N, Pa)
force = [(P, -b * dq * N.x + u * N.x)]

# Lagrange equations
lagrange = mechanics.LagrangesMethod(L, [q], forcelist = force, frame = N)
pprint(lagrange.form_lagranges_equations())
```

    ⎡                              2             ⎤
    ⎢  d                          d              ⎥
    ⎢b⋅──(q(t)) + 1.0⋅k⋅q(t) + m⋅───(q(t)) - u(t)⎥
    ⎢  dt                          2             ⎥
    ⎣                            dt              ⎦

## 2. Laplace transform and transfer function

Applying the Laplace transform on the dynamic equation:

\begin{equation}
    m \ddot{x}(t) + b \dot{x}(t) + k x(t) = u(t) \stackrel{L}{\rightarrow} m s^2 X(s) + b s X(s) + k X(s) = U(s)
\end{equation}

The transfer equation is given by:

\begin{equation}
    H(s) = \frac{X(s)}{U(s)} = \frac{1}{m s^2 + b s + k}
\end{equation}

By calculating the pole:

\begin{equation}
    m s^2 + b s + k = 0 \Leftrightarrow s = \frac{-b}{2m} \pm \sqrt{\left(\frac{b}{2m}\right)^2 - \frac{k}{m}}
\end{equation}

Note that $b, k, m > 0$ because they represent real physical quantities.

### LTI system

We can rewrite the above equation as a first-order system of equations. Let's first define the state vector $\pmb{x} = \left[ \begin{array}{c} x(t) \\ \dot{x}(t) \end{array} \right]$ and the control vector $\pmb{u} = \left[ \begin{array}{c} u(t) \end{array} \right]$, then we can rewrite the above equation in the form $\pmb{\dot{x}} = \pmb{Ax} + \pmb{Bu}$, as below:

\begin{equation}
    \left[ \begin{array}{c} \dot{x}(t) \\ \ddot{x}(t) \end{array} \right] = \left[ \begin{array}{cc} 0 & 1 \\ -\frac{k}{m} & -\frac{b}{m} \end{array} \right] \left[ \begin{array}{c} x(t) \\ \dot{x}(t) \end{array} \right] + \left[ \begin{array}{c} 0 \\ \frac{1}{m} \end{array} \right] \left[ \begin{array}{c} u(t) \end{array} \right]
\end{equation}

If there is no $u(t)$, i.e. $u(t) = 0 \; \forall t$, then we have $\pmb{\dot{x}} = \pmb{Ax}$. The solution to this system of equation is $\pmb{x}(t) = e^{\pmb{A}t} \pmb{x}_0$.

```python

```


Take the following two equations...

\begin{align}
J_\alpha(p||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha p(x) +(1-\alpha)q(x)-p(x)q(x)^{1-\alpha}dx
 && \text{KL}(q || p) = \int_x q(x) \log \frac{q(x)}{p(x)} dx
\end{align}

Let $q(x)$ be the pdf for a Gaussian,
$\mathcal{N}(\mu, \Sigma)$, and $p(x)$ be unknown. I have demonstrated numerically that both equations have the same extrema for $\mu$, for $-1 \lt \alpha \lt 0$. (If you would like to see the notebook for that, I can send it too. For now just assume it is true)

I would now like to show it analytically. I thought it would be relatiely straight forward to take the partials and show that the same $\mu^*$ minimizes both.

\begin{align}
\frac{\partial J_\alpha}{\partial \mu} &= \frac{1}{\alpha} \int_x q_{\mu}(x)\bigg(1-\frac{p(x)}{q(x)^{\alpha}}\bigg)dx
 && \frac{\partial \text{KL}}{\partial \mu} = & \int_x q_{\mu}(x) \bigg(\log \frac{q(x)}{p(x)}+1 \bigg) dx \\
 \frac{\partial J_\alpha}{\partial \mu} &= \frac{1}{\alpha} \int_x q(x)\Sigma^{-1}(x-\mu)\bigg(1-\frac{p(x)}{q(x)^{\alpha}}\bigg)dx
 && \frac{\partial \text{KL}}{\partial \mu} = & \int_x q(x) \Sigma^{-1}(x-\mu) \bigg(\log \frac{q(x)}{p(x)}+1 \bigg) dx
\end{align}

After observing that $\int_x q_{\mu}(x) dx = 0$, via...

\begin{align}
\int_x q_{\mu}(x) dx &= \int_x q(x) \Sigma^{-1} (x-\mu)dx \\
&= \Sigma^{-1} \bigg(\int_x q(x) x dx - \mu \int_x q(x) dx \bigg)\\
&= \Sigma^{-1} \big(\mu - \mu \big) \\
&= 0\\
\end{align}

Both equations reduce to...

\begin{align}
\frac{\partial J_\alpha}{\partial \mu} = -\Sigma^{-1} \mathbb{E}_{q} \bigg[ \frac{p(x)}{\alpha q(x)^{\alpha}}(x-\mu) \bigg]
 && \frac{\partial \text{KL}}{\partial \mu} = -\Sigma^{-1} \mathbb{E}_{q} \bigg[ \log \frac{p(x)}{q(x)}(x-\mu) \bigg]
\end{align}

And from here, setting them both equal to zero, we can derive closed form solutions...

\begin{align}
\mu_J = \frac{ \mathbb{E}_{q} \bigg[ \frac{p(x)}{q(x)^{\alpha}}x \bigg]}{\mathbb{E}_{q} \bigg[ \frac{p(x)}{q(x)^{\alpha}} \bigg]}
&& \mu_{KL} = \frac{ \mathbb{E}_{q} \bigg[ \log\big(\frac{p(x)}{q(x)}\big)x \bigg]}{\mathbb{E}_{q} \bigg[ \log\big(\frac{p(x)}{q(x)}\big) \bigg]}
\end{align}

```python

```

dissertation/.ipynb_checkpoints/Minimizing J finds the peaks of P-checkpoint.ipynb


# Subspace

**Subspace.** The set of vectors $V$ is a linear subspace of $\mathbb{R}^n \iff$ null vector $\in V$ and $V$ is closed under scalar multiplication and addition.

⚠ *Union of subspaces is not a subspace*

> The reason why this can happen is that all vector spaces, and hence subspaces too, must be closed under addition (and scalar multiplication). The union of two subspaces takes all the elements already in those spaces, and nothing more.
>
> In the union of subspaces $W_1$ and $W_2$ there are new combinations of vectors we can add together that we couldn't before, like $v_1 + v_2$ where $v_1 \in W_1$ and $v_2 \in W_2$.
>
> For example, take $W_1$ to be the $x$-axis and $W_2$ the $y$-axis, both subspaces of $\mathbb{R}^2$.
Their union includes both $(3,0)$ and $(0,5)$, whose sum, $(3,5)$, is not in the union. Hence, the union is not a vector space.
>
> http://math.stackexchange.com/a/71875/402625

# Linear Dependence and Independence
**Linearly Dependent.** A set of vectors is linearly dependent if there exists a vector in the set that *can be written as a linear combination of the others*.

**Linearly Independent.** A set of vectors is linearly independent $\iff$ the only linear combination that gives the null vector is the linear combination with all coefficients equal to $0$.
$$\Sigma_{i=0}^k \lambda_i v_i = 0 \iff \lambda_1 = \lambda_2 = \dots = \lambda_i = 0$$

_**Proof.**_ ($\Rightarrow$) Say there exists a linear combination of that set ($D$) that evaluates to the null vector and not all coefficients equal $0$, then there exists a linear combination $\Sigma_{i=1}^{n}\lambda_i v_i=0$ with $\lambda_i\in\mathbb{R}, v_i\in D$ and $\lambda_1\neq0$. So, $v_1=\frac{-1}{\lambda_1}\left( \Sigma_{i=2}^{n}\lambda_i v_i \right)$ and the set is linearly dependent.

# Span
**Span**. The *span of a given subset ($D$) of a vector space ($V$)* is *the vector space ($\text{span}(D)$) containing all possible linear combinations of the vectors in the subset ($D$)*.

##### _Example_
\begin{equation}
    A_1=
    \begin{bmatrix}
        1 \\
        0
    \end{bmatrix},
    A_2=
    \begin{bmatrix}
        0 \\
        1
    \end{bmatrix}
\end{equation}
$\text{span}(\{A_1,A_2\})=\langle\{A_1,A_2\}\rangle=\text{vct}(\{A_1,A_2\})= \{\alpha_1A_1 + \alpha_2A_2\mid\alpha_1,\alpha_2\in\mathbb{R}\}$

Two 2-vectors span $\mathbb{R}^2 \iff$ they are linearly independent.

$\text{span}(\{A_1,A_2\})=\mathbb{R}^2$

##### Thoughts
*Q*: If the set of 2 vectors is linearly dependent, can each one be written as a linear combination of the other? What about 3 vectors?

*A*: Trivial for 2 vectors. If the set of vectors $A_1$ and $A_2$ are linearly dependent, there exists a vector in that set that can be written as a linear combination of the other. Say $A_1$ is that vector, then $A_1=\alpha A_2,$ with  $\alpha\in\mathbb{F}$ and $A_2=\frac{1}{\alpha}A_1$. Analogue if $A_2$ is that vector.

For 3 vectors, this isn't necessarily possible (e.g.: $\{(1,0,0),(2,0,0),(0,1,0)\}$). The set of 3 vectors is linearly dependent if (at least) one of them can be written as a linear combination of the others.

# Basis & Dimension

**Basis.** A set of *linearly independent* vectors that *spans a vector space* is a *basis* of that space.

⚠ *Lemma of Steinitz*

Say $(\mathbb{R},V,+)$ a vector space. Then:
1. if a subset $\subset V$ of $m$ elements exists that spans $V$, then every subset of $V$ with more than $m$ elements is linearly dependent;
2. if a subset $\subset V$ of $n$ elements exists that is linearly independent, then every subset of $V$ with less than $n$ elements cannot span $V$.

**Dimension.** The *dimension of a vector space* equals the *number of elements in a (finite) basis* for that vector space. (_Notation:_ $\text{dim}_{\mathbb{R}}V$)

⚠

Say $(\mathbb{R},V,+)$ a vector space of dimension $n$. Then:
1. every linear independent subset $\subset V$ can be extended to a basis of $V$;
2. every finite subset $\subset V$ that spans $V$ can be reduced (by removing vectors) to a basis of $V$.

```python

```