# Kernel Logistic Classifier

## Linear logistic classifier

Consider the multi-dimensional logistic classifier
\begin{equation}
P(c\mid{\bf x},\Theta) = \frac{e^{\alpha_c+{\bf\beta}_c^{T}{\bf x}}}
{\sum_{c'=1}^{C}e^{\alpha_{c'}+{\bf\beta}_{c'}^{T}{\bf x}}}\,,
\end{equation}
for feature vector ${\bf x}\in\mathbb{R}^{F}$.
We can, if we wish, notionally consider the prior of class $c$ to be $P(c\mid\Theta)\propto e^{\alpha_c}$, and
the class density of ${\bf x}$ to be
$p({\bf x}\mid c,\Theta)\propto e^{{\bf\beta}_c^{T}{\bf x}}$, although the latter assumption poses some normalisation issues.

Now consider supervised training data comprised of $N$ known class labels ${\bf C}=[c_1,c_2,\ldots,c_N]^{T}$
and feature (or design) matrix ${\bf X}=[{\bf x}_1,{\bf x}_2,\ldots,{\bf x}_N]^{T}$.
The discriminative likelihood is then given by
\begin{eqnarray}
P({\bf C}\mid{\bf X},\Theta) & = & \prod_{d=1}^{N}P(c_d\mid{\bf x}_d,\Theta)
~=~ \prod_{d=1}^{N}\prod_{c=1}^{C}P(c\mid{\bf x}_d,\Theta)^{\delta(c_d=c)}\,,
\end{eqnarray}
where $\delta(A)=1$ and $\delta(\neg A)=0$ if proposition $A$ is true.

For notational convenience, we let $z_{cd}\doteq\delta(c_d=c)$ and $\pi_{cd}\doteq P(c\mid{\bf x}_d,\Theta)$.
Then the discriminative log-likelihood is just
\begin{eqnarray}
L(\Theta) & = & \ln P({\bf C}\mid{\bf X},\Theta)
~=~\sum_{d=1}^{N}\sum_{c=1}^{C}z_{cd}\ln\pi_{cd}
\nonumber\\
& = & \sum_{d=1}^{N}\sum_{c=1}^{C}z_{cd}(\alpha_c+{\bf\beta}_c^{T}{\bf x}_d)
-\sum_{d=1}^{N}\ln\sum_{c=1}^{C}e^{\alpha_c+{\bf\beta}_c^{T}{\bf x}_d}\,.
\end{eqnarray}

Since discriminatively trained models are prone to overfitting, it is usual to regularise the parameters.
In the case of ridge regression, there is some dispute whether to penalise just the feature weights, proportional
$\|{\bf\beta}_c\|^2$, or to also penalise the bias $\alpha_c$, proportional to $\|{\bf\gamma}_c\|^2$,
where ${\bf\gamma}_c\doteq [\alpha_c]\oplus{\bf\beta}_c$ is the concatenate of all parameters for class $c$.

The former case is sometimes preferred on the basis that $\alpha_c$ controls the prior on class $c$, and probably shouldn't be constrained beyond what the data suggest.
In particular, if we do not regularise the class weights, then it can be shown that
\begin{eqnarray}
\frac{1}{N}\sum_{d=1}^{N}P(c\mid{\bf x}_d,\Theta) & = & \frac{N_c}{N}\,,
\end{eqnarray}
where $N_c$ is the number of training samples of class $c$.

Alternatively, it could be noted that the observed class proportions at best only approximate the true class priors, and at worst are artificially constrained (e.g. by balancing class sizes). Hence, we might prefer the latter case of regularising all parameters.

For now we consider the more general, quadratic penalty ${\bf\gamma}_c^{T}{\bf\Lambda}_c{\bf\gamma}_c$, which allows us to not only "turn off" regularisation of some parameters, but additionally to handle differently-scaled features and correlations between features. We define the modified feature vector $\tilde{\bf x}\doteq[1]\oplus{\bf x}$, such that
\begin{eqnarray}
\pi_{cd} & = & \frac{e^{{\bf\gamma}_c^{T}\tilde{\bf x}_d}}
{\sum_{c'=1}^{C}e^{{\bf\gamma}_{c'}^{T}\tilde{\bf x}_d}}\,.
\end{eqnarray}
The ridge-regularised discriminative log-likelihood is then
\begin{eqnarray}
\tilde{L}(\Theta) & = &
\sum_{d=1}^{N}\sum_{c=1}^{C}z_{cd}{\bf\gamma}_c^{T}\tilde{\bf x}_d
-\sum_{d=1}^{N}\ln\sum_{c=1}^{C}e^{{\bf\gamma}_c^{T}\tilde{\bf x}_d}
-\frac{1}{2}\sum_{c=1}^{C}{\bf\gamma}_c^T{\bf\Lambda}_c\mathbf{\gamma}_c
\,.
\end{eqnarray}

It can be shown that its class-specific gradient vector is given by
\begin{eqnarray}
{\bf\nabla}_{c}\tilde{L} & = & \frac{\partial\tilde{L}}{\partial{\bf\gamma}_c}
~=~\sum_{d=1}^{N}z_{cd}\tilde{\bf x}_d-\sum_{d=1}^{N}\pi_{cd}\tilde{\bf x}_d-{\bf\Lambda}_c{\bf\gamma}_c\,,
\end{eqnarray}
and the class-specific Hessian matrix is given by
\begin{eqnarray}
{\bf\nabla}_{c}^{T}{\bf\nabla}_{c}\tilde{L} & = &
\frac{\partial^2\tilde{L}}{\partial{\bf\gamma}_c^{T}\partial{\bf\gamma}_c}
~=~-\sum_{d=1}^{N}\pi_{cd}(1-\pi_{cd})\tilde{\bf x}_d\tilde{\bf x}_d^{T}-{\bf\Lambda}_c\,.
\end{eqnarray}
Note that, for simplicity, we are going to ignore the explicit cross-class dependencies
${\bf\nabla}_{c'}^{T}{\bf\nabla}_{c}\tilde{L}$.

In matrix notation, let ${\bf z}_c=[z_{c1},\ldots,z_{cN}]^T$, ${\bf\pi}_c=[\pi_{c1},\ldots,\pi_{cN}]^T$,
and $\tilde{\bf X}={\bf 1}\oplus{\bf X}$. Then the gradient becomes
\begin{eqnarray}
{\bf\nabla}_{c}\tilde{L} & = &
\tilde{\bf X}^{T}({\bf z}_c-{\bf\pi}_c)-{\bf\Lambda}_c{\bf\gamma}_c\,.
\end{eqnarray}
Similarly, define $w_{cd}\doteq\pi_{cd}(1-\pi_{cd})$, and let ${\bf w}_c=[w_{c1},\ldots,w_{cN}]^T$
and ${\bf W}_c={\tt diag}\{{\bf w}_c\}$. Then the Hessian becomes
\begin{eqnarray}
{\bf\nabla}_{c}^{T}{\bf\nabla}_{c}\tilde{L} & = &
-\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}-{\bf\Lambda}_c\,.
\end{eqnarray}

Now, the class-specific update for parameters ${\bf\gamma}_c$ takes the form of a single iteration of the Newton-Raphson method, namely
\begin{eqnarray}
{\bf\gamma}'_c & = & {\bf\gamma}_c - \left[{\bf\nabla}_{c}^{T}{\bf\nabla}_{c}\tilde{L}\right]^{-1}
{\bf\nabla}_{c}\tilde{L}
\nonumber\\
& = & {\bf\gamma}_c+\left[\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}+{\bf\Lambda}_c\right]^{-1}
\left[\tilde{\bf X}^{T}({\bf z}_c-{\bf\pi}_c)-{\bf\Lambda}_c{\bf\gamma}_c\right]
\nonumber\\
& = & \left[\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}+{\bf\Lambda}_c\right]^{-1}
\left[\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}{\bf\gamma}_c+{\bf\Lambda}_c{\bf\gamma}_c+
\tilde{\bf X}^{T}({\bf z}_c-{\bf\pi}_c)-{\bf\Lambda\gamma}_c\right]
\nonumber\\
& = & \left[\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}+{\bf\Lambda}_c\right]^{-1}\tilde{\bf X}^T{\bf W}_c
\left[\tilde{\bf X}{\bf\gamma}_c+{\bf W}_{c}^{-1}({\bf z}_c-{\bf\pi}_c)\right]\,.
\end{eqnarray}

## Dual optimisation

Observe from above that ${\bf z}_c-{\bf\pi}_c$ represents a vector of prediction errors for class $c$.
Hence, if we define the weighted errors
${\bf e}_c\doteq{\bf W}_c^{-1}({\bf z}_c-{\bf\pi}_c)$
and the linear system
\begin{equation}
{\bf y}_c = \tilde{\bf X}{\bf\gamma}_c+{\bf e}_c\,,
\end{equation}
then we note that the class-specific parameter update becomes
\begin{eqnarray}
\left[\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}+{\bf\Lambda}_c\right]{\bf\gamma}'_c
& = & \tilde{\bf X}^T{\bf W}_c{\bf y}_c\,.
\end{eqnarray}
It is of interest that this update corresponds to a regularised form of the iteratively reweighted least-squares (IRLS) algorithm, except that here ${\bf y}_c$ itself varies with each iteration.

To motivate this observation, note that if we knew ${\bf y}_c$ in advance, then we could simply find the optimal
parameters ${\bf\gamma}_c$ via a weighted least-squares (WLS)
minimisation of the square error $\|{\bf W}_c{\bf e}_c\|^2$. However, we instead must obtain ${\bf y}_c$ via the following steps:
1. Choose initial parameters, ${\bf\gamma}_c$, for all class $c=1,2,\ldots,C$.
2. Compute the linear projection, $\tilde{\bf X}{\bf\gamma}_c$.
3. Compute the posterior probabilities, ${\bf\pi}_c$.
4. Compute the weighted prediction errors, ${\bf e}_c$.
5. Compute the 'observations', ${\bf y}_c$.

At this juncture, we may now find update parameters ${\bf\gamma}'_c$ that minimise the
square error $\|{\bf e}'_c\|^2=\|{\bf y}_c-\tilde{\bf X}^T{\bf\gamma}'_c\|^2$, satisfying the system
${\bf y}_c = \tilde{\bf X}{\bf\gamma}'_c+{\bf e}'_c$. This gives rise to the IRLS update above.

In conclusion, Newton-Raphson maximisation of the discriminative log-likelihood (the primal problem)
corresponds to IRLS minimisation of the (weighted) prediction error (the dual problem).

## Representer theorem

Now, a consequence of this duality is that we may apply the representer theorem. Put simply, the ridge-penalised function $f_c(\cdot)$ that minimises the square error $\sum_{d=1}^{N}\|y_{cd}-f_c(\tilde{\bf x}_d)\|^2$
satisfies $f_c(\tilde{\bf x})=\sum_{d=1}^{N}\omega_{cd}k(\tilde{\bf x}_d,\tilde{\bf x})$ for some positive-definite kernel function $k(\cdot,\cdot)$. In other words, the least-squares function interpolates over the known data points.

Hence, since the parameter update chooses ${\bf\gamma}'_c$ to minimise the square error
$\|{\bf y}_c-\tilde{\bf X}{\bf\gamma}'_c\|^2$, then we may take
$f_c(\tilde{\bf x})\doteq{\bf\gamma}_c^{'T}\tilde{\bf x}$.

Let us now consider the scalar-product kernel $k({\bf x},{\bf y})\doteq {\bf x}^{T}{\bf y}$.
Then it follows that
\begin{equation}
{\bf\gamma}_c^{'T}\tilde{\bf x} = \sum_{d=1}^{N}\omega'_{cd}\tilde{\bf x}_d^{T}\tilde{\bf x}
= \left({\bf\omega}_c^{'T}\tilde{\bf X}\right)\tilde{\bf x}
\Rightarrow {\bf\gamma}'_c = \tilde{\bf X}^{T}{\bf\omega}'_c\,.
\end{equation}
Substituting this representation (for both the old and new parameter estimates) back into the parameter update then gives
\begin{eqnarray}
{\bf y}_c & = & \tilde{\bf X}\tilde{\bf X}^{T}{\bf\omega}_c+{\bf W}_{c}^{-1}({\bf z}_c-{\bf\pi}_c)\,,
\nonumber\\
\left[\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}+{\bf\Lambda}_c\right]\tilde{\bf X}^{T}{\bf\omega}'_c
&=&\tilde{\bf X}^T{\bf W}_c{\bf y}_c
\nonumber\\
\Rightarrow
\left[\tilde{\bf X}\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}\tilde{\bf X}^{T}
+\tilde{\bf X}{\bf\Lambda}_c\tilde{\bf X}^{T}\right]{\bf\omega}'_c
& = &\tilde{\bf X}\tilde{\bf X}^T{\bf W}_c{\bf y}_c\,.
\end{eqnarray}

In order to simplify these expressions further, we note that the general regulariser ${\bf\Lambda}_c$ now gives us a problem.
One way to avoid the problem is to reverse our generalisation, and take ${\bf\Lambda}_c=\lambda{\bf I}$ as per usual.
Hence, we obtain
\begin{eqnarray}
\left[\tilde{\bf X}\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}\tilde{\bf X}^{T}
+\lambda\tilde{\bf X}\tilde{\bf X}^{T}\right]{\bf\omega}'_c
&=&\tilde{\bf X}\tilde{\bf X}^T{\bf W}_c{\bf y}_c\,.
\end{eqnarray}
Note that this simplifying assumption now regularises both the class weights $\alpha_c$ and the feature weights
${\bf\beta}_c$. With some effort, we can avoid regularisation of $\alpha_c$ by instead choosing
${\bf\Lambda}_c\doteq\lambda\,{\tt diag}\{0,1,\ldots,1\}$, whereupon
\begin{eqnarray}
\left[\tilde{\bf X}\tilde{\bf X}^T{\bf W}_c\tilde{\bf X}\tilde{\bf X}^{T}
+\lambda{\bf X}{\bf X}^{T}\right]{\bf\omega}'_c
&=&\tilde{\bf X}\tilde{\bf X}^T{\bf W}_c{\bf y}_c\,,
\end{eqnarray}
since $\tilde{\bf X}={\bf 1}\oplus{\bf X}$.
We shall return to this point later.

## Kernel trick

Now, we define the kernel matrix $\tilde{\bf K}\doteq\tilde{\bf X}\tilde{\bf X}^{T}$,
where $\tilde{K}_{ij}=\tilde{\bf x}_i^{T}\tilde{\bf x}_j=k(\tilde{\bf x}_i,\tilde{\bf x}_j)$.
Consequently, we obtain
\begin{eqnarray}
{\bf y}_c & = &\tilde{\bf K}{\bf\omega}_c+{\bf W}_{c}^{-1}({\bf z}_c-{\bf\pi}_c)\,,
\nonumber\\
\left[\tilde{\bf K}{\bf W}_c\tilde{\bf K}
+\lambda\tilde{\bf K}\right]{\bf\omega}'_c
& = & \tilde{\bf K}{\bf W}_c{\bf y}_c
\nonumber\\
\Rightarrow \left[{\bf W}_c\tilde{\bf K}+\lambda{\bf I}\right]{\bf\omega}'_c & = & {\bf W}_c{\bf y}_c
={\bf W}_c\tilde{\bf K}{\bf\omega}_c+({\bf z}_c-{\bf\pi}_c)\,,
\end{eqnarray}
since $\tilde{\bf K}$ is invertible, from the definition that the kernel function $k(\cdot,\cdot)$
is positive-definite.
Note that we also assumed previously that the diagonal matrix
${\bf W}_c$ is invertible, hence we could simplify further to
\begin{eqnarray}
\left[\tilde{\bf K}+\lambda{\bf W}_c^{-1}\right]{\bf\omega}'_c & = &
\tilde{\bf K}{\bf\omega}_c+{\bf W}_{c}^{-1}({\bf z}_c-{\bf\pi}_c)\,.
\end{eqnarray}
We should note at this point that we have now traded an $F\times F$ inversion problem (in the parameter space) for an $N\times N$ one (in the data space).

We have also, as noted earlier, lost the propery of being able to *not* penalise the $\alpha_c$ bias (or class proportion) parameter.
To see what form the update would take if we did not regularise $\alpha_c$, we instead take the alternative
regularisation
${\bf\Lambda}_c\doteq\lambda\,{\tt diag}\{0,1,\ldots,1\}$. Then, from the derivation above,
we obtain the modified version
\begin{eqnarray}
\left[{\bf W}_c\tilde{\bf K}+\lambda\tilde{\bf K}^{-1}{\bf K}\right]{\bf\omega}'_c
&=&{\bf W}_c\tilde{\bf K}{\bf \omega}_c+({\bf z}_c-\mathbb{\pi}_c)\,,
\end{eqnarray}
where $\tilde{\bf K}={\bf K}+{\bf 1}{\bf 1}^T$.
We deduce from the Sherman-Morrison formula that $\tilde{\bf K}^{-1}{\bf K}={\bf I}-{\bf v}{\bf 1}^{T}$,
where ${\bf v}={\bf u}/(1+{\bf 1}^{T}{\bf u})$ and ${\bf K}{\bf u}={\bf 1}$.
Hence, the full version of the class-specific parameter update is
\begin{eqnarray}
\left[{\bf W}_c({\bf K}+{\bf 1}{\bf 1}^T)+\lambda({\bf I}-{\bf v}{\bf 1}^T)\right]{\bf\omega}'_c
&=&{\bf W}_c({\bf K}+{\bf 1}{\bf 1}^T){\bf \omega}_c+({\bf z}_c-\mathbb{\pi}_c)\,,
\end{eqnarray}
or
\begin{eqnarray}
\left[{\bf W}_c{\bf K}+\lambda{\bf I}+({\bf w}_c-\lambda{\bf v}){\bf 1}^T\right]{\bf\omega}'_c
&=&({\bf W}_c{\bf K}+{\bf w}_c{\bf 1}^T){\bf \omega}_c+({\bf z}_c-\mathbb{\pi}_c)\,,
\end{eqnarray}
since ${\bf W}_c{\bf 1}={\bf w}_c$.

Finally, we recall that ${\bf\gamma}_c=\tilde{\bf X}^{T}{\bf\omega}_c$, such that our original parameters are recovered as
\begin{eqnarray}
\alpha_c~=~{\bf 1}^{T}{\bf\omega}_c\,, && {\bf\beta}_c~=~{\bf X}^{T}{\bf\omega}_c\,.
\end{eqnarray}


# Ricci Tensor and Scalar Curvature calculations using Symbolic module

```python
import sympy
from einsteinpy.symbolic import RicciTensor, RicciScalar
from einsteinpy.symbolic.predefined import AntiDeSitter

sympy.init_printing()
```

### Defining the Anti-de Sitter spacetime Metric

```python
metric = AntiDeSitter()
metric.tensor()
```

### Calculating the Ricci Tensor(with both indices covariant)

```python
Ric = RicciTensor.from_metric(metric)
Ric.tensor()
```

### Calculating the Ricci Scalar(Scalar Curvature) from the Ricci Tensor

```python
R = RicciScalar.from_riccitensor(Ric)
R.simplify()
R.expr
```

The curavture is -12 which is in-line with the theoretical results

docs/source/examples/Ricci Tensor and Scalar Curvature symbolic calculation.ipynb


# The resilu linearity / non-linearity

The function $resilu(x)=\frac{x}{1-e^{-x}}$ can be written as the sum of a linear funciton and a function that limits to relu(x).

By using resilu(x) and 'non'-linearity in neural networks, the effect of a skip-connection should thus be included 'for free'.

```python
import copy
import numpy as np
import matplotlib.pyplot as plt
import math
import sympy
```

```python
x=np.arange(-20,20,0.01)
```

```python
def resilu(x):
    return x/(1.0-np.exp(x*-1.0))

def relu(x):
    y=copy.copy(x)
    y[y<0]=0.0
    return y

```

Comparing $resilu(x)=\frac{x}{1-e^{-x}}$ with the $relu(x)$ function.

By introducing a parameter $a$, $relu(x)$ is a limit of $resilu(x,a)$:

$$\lim_{a\to\infty}\frac{x}{1-e^{-x a}}=relu(x)$$

```python
fig=plt.figure()
ax=fig.add_axes([0,0,1,1])
ax.plot(x,resilu(x),label='resilu')
ax.plot(x,relu(x),linestyle=':',label='relu')
ax.legend()
```

```python
def resilu_nonlin(x):
    return x/(np.exp(x)-1.0)

def resilu_lin(x):
    return x
```

Resilu(x) can be split into a sum of two functions:

* linear part $f_1(x)=x$
* non-linear part $f_2(x)=\frac{x}{e^x-1}$, with exactly similar limit properties as the complete resilu function: $lim_{a\to\infty}\frac{x}{e^{x a}-1}=relu(-x)$

$$
resilu(x)=f_1(x)+f_2(x)=\frac{x}{e^x-1} + x = \frac{x}{1-e^{-x}}
$$

This should show that using resilu non-linearity should be äquivalent to using a 'standard' non-linearity and a skip connection that ommits the nonlinearity as additive residual connection. The only difference are arbitrary linear transformations which the net can easily cancel out.

```python
fig=plt.figure()
ax=fig.add_axes([0,0,1,1])
ax.plot(x,resilu_nonlin(x),label='resilu (non-lin part)')
ax.plot(x,resilu_lin(x),label='resilu (lin part)')
ax.plot(x,resilu_nonlin(x)+resilu_lin(x),label='non-lin + lin (= complete resilu)')
ax.plot(x,relu(x),linestyle=':',label='relu(x)')
ax.plot(x,relu(-x),linestyle=':',label='relu(-x)')
ax.legend()
```

## Numerical instability at $x=0$

$\frac{x}{1-e^{-x}}$ is obviously numerically unstable at $x=0$, which could be seen as a phase transition or an unstable fixed point.

In order to use the resilu(x) function, the function itself and it's derivate (which are both not numerically stable around $x=0$ are approximated with $\mathcal{O}(4)$ taylor expansions for a small interval $-h<0<h$ in the actual implementation. [See `Resilu` and `dResilu()` implementation](https://github.com/domschl/syncognite/blob/master/cpneural/layers/cpl-nonlinearity.h)

## Series expansion to work with instabilities

### Taylor series approximation for resilu(x)

```python
x=sympy.Symbol('x')
sympy.series(x/(1-sympy.exp(-x)),x,n=10)
```

$\displaystyle 1 + \frac{x}{2} + \frac{x^{2}}{12} - \frac{x^{4}}{720} + \frac{x^{6}}{30240} - \frac{x^{8}}{1209600} + O\left(x^{10}\right)$

### $d/dx$ $resilu(x)$ and it's Taylor approximation

```python
sympy.diff(x/(1-sympy.exp(-x)),x)
```

$\displaystyle - \frac{x e^{- x}}{\left(1 - e^{- x}\right)^{2}} + \frac{1}{1 - e^{- x}}$

```python
sympy.series(sympy.diff(x/(1-sympy.exp(-x)),x),x,n=10)
```

$\displaystyle \frac{1}{2} + \frac{x}{6} - \frac{x^{3}}{180} + \frac{x^{5}}{5040} - \frac{x^{7}}{151200} + \frac{x^{9}}{4790016} + O\left(x^{10}\right)$

```python
sympy.diff(sympy.series(x/(1-sympy.exp(-x)),x,n=10),x)
```

$\displaystyle \frac{1}{2} + \frac{x}{6} - \frac{x^{3}}{180} + \frac{x^{5}}{5040} - \frac{x^{7}}{151200} + O\left(x^{9}\right)$

```python

```


# Lecture 19

## Joint, Conditional and Marginal Distributions; 2D LOTUS; Expected Distance between Uniforms; Chicken-egg Problem

### Joint, Conditional and Marginal Distributions

#### Joint CDF
A joint CDF is simply where we are dealing with multiple random variables. As an example, a case where we have two random variables $X, Y$, the joint CDF of two random variables $X, Y$ can be expressed as:

\begin{align}
  F(x,y) &= P(X {\le} x, Y {\le} y)
\end{align}

Note that the random variables may be discrete, continuous, or a mixture of both.

#### Joint PDF
The joint PDF, in the case of _continuous_ random variables, is what you would _integrate_ to get the joint CDF. Continuing with our example of two (continous) random variables, we have:

\begin{align}
  f(x,y) &= \frac{\partial^2}{\partial{x}\partial{y}} F(x,y)
\end{align}

Conversely, if we want to know the probability of $X,Y$ in some set $A$, we _integrate_ the density to get that probability.

\begin{align}
  P\left((X,Y) \in A\right) &= \iint\limits_{A} f(x,y) \, dxdy
\end{align}

Integrate by holding one variable constant, and then do the other. The key thing is to be sure to get the _limits of integration_ correct.

#### Marginal PDF
The _marginal PDF of $X$_ is obtained by _integrating out the $Y$_. Recall the $X,Y$ contigency table and the definition of marginal probability.

\begin{align}
  \int_{-\infty}^{\infty} f(x,y) \, dy
\end{align}

Notice that by keeping $X$ constant and _integrating over all $Y$_, the marginal PDF of $X$ no longer depends on $Y$.

And we can do vice-versa for the marginal PDF of $Y$, but keeping $Y$ constant and _integrating over all $X$_.

Do not forget that taking the marginal PDF of $X$ and then integrating over all $X$, we should get 1.0.

\begin{align}
  \int\limits_{-\infty}^{\infty} \int\limits_{-\infty}^{\infty} f(x,y) \, dx dy &= 1.0
\end{align}

#### Conditional PDF

Given that we know $X$, what is the appropriate PDF for $Y$?

Well, we can apply what we know about _conditional probability_ to get a conditional PDF.

\begin{align}
  f_{Y|X} (y|x) &= \frac{f_{XY}(x,y)}{f_{X}(x)} \\
    &= \frac{f_{X|Y}(x|y) \, f_{Y}(y)}{f_{X}(x)}
\end{align}

This is completely analogous to conditional probability.

#### Independence

$X,Y$ are independent if

\begin{align}
  f_{X,Y}(x,y) &= f_{X}(x) \, f_{Y}(y) &\quad \text{for all }x, y \text{ from PDF p.o.v.} \\
  \\
  F(x,y) &= F(x) \, F(y) &\quad \text{for all }x, y \text{ from CDF p.o.v.}
\end{align}

These statements are equivalent, but in most cases it might be easier to work the PDFs.

----

### A Uniform example

Let's revisit that distribution that is uniform on the unit disc $x^2 + y^2 \le 1$.

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.xkcd()

fig = plt.figure(figsize = (5.0,5.0))
ax = fig.add_subplot(1, 1, 1)
ax.set_xlim([-1.5,1.5])
ax.set_ylim([-1.5,1.5])
circ = plt.Circle((0, 0), radius=1.0, color="b", alpha=0.2, lw=5)
ax.add_patch(circ)
plt.axhline(0, color="black", alpha=0.4)
plt.axvline(0, color="black", alpha=0.4)
plt.show()
```

#### Joint PDF

A valid PDF is required to integrate to 1.0, and since we are looking at a Uniform distribution over the unit circle with area equal to $\pi$,  we have

\begin{align}
  f_{XY}(x,y) &=
    \begin{cases}
      \frac{1}{\pi}  & \quad \text{if } x^2 + y^2 \le 1 \\
      0  & \quad \text{otherwise}\\
    \end{cases}
\end{align}

Understand that while this is uniformly distributed, $x$ and $y$ are closely related with $x^2 + y^2 \le 1$, and so $X,Y$ cannot be independent. More on that coming up.

#### Marginal PDF of $X$

\begin{align}
  f_{X}(x) &= \int_{-\sqrt{1-x^2}}^{\sqrt{1-x^2}} \frac{1}{\pi} dy \\
  \\
  &= \frac{2}{\pi} \sqrt{1-x^2} &\quad \text{where } -1 \le x \le 1
\end{align}

Notice that while the joint PDF is Uniform, the marginal PDF is not Uniform: it grows as $x$ approaches 0.

By _symmetry_ we can just replace $x$ with $y$ to get the marginal PDF of $Y$ $f_{Y}(y)$

#### Conditional PDF of $Y|X$

\begin{align}
  f_{Y|X}(y|x) &= \frac{f_{XY}(x,y)}{f_{X}(x)} \\
  &= \frac{\frac{1}{\pi}}{\frac{2}{\pi} \sqrt{1-x^2}} \\
  &= \frac{1}{2 \sqrt{1-x^2}} &\quad \text{if } -\sqrt{1-x^2} \le y \le \sqrt{1-x^2}
\end{align}

But since we are treating $x$ as a constant, and the above equation does not _depend on $y$_, the conditional PDF $f_{Y|X}(y|x)$ is actually Uniform.

\begin{align}
  Y|X &\sim Unif(-\sqrt{1-x^2}, \sqrt{1-x^2}) &\quad \text{or also } \\
  \\
  Y|X=x \, &\sim Unif(-\sqrt{1-x^2}, \sqrt{1-x^2})
\end{align}

#### Non-independence

Since in our example above

\begin{align}
  f_{XY}(x,y) \neq f_{X}(x) \, f_{Y}(y)
\end{align}

we can see that $X,Y$ are not independent.

Or, because the _unconditional_ distribution of $Y$ is not the same as the _conditional_ distribution of $Y|X$, it necessarily follows that $X,Y$ are not independent as learning $X$ gives us information about $Y$ (and vice-versa).

----

### 2D LOTUS

Let $X,Y$ have joint PDF $f(x,y)$, and let $g(x,y)$ be a real-valued function of $x,y$.

Then, without trying to find the PDF of $g(x,y)$, LOTUS lets us use joint PDF:

\begin{align}
  \mathbb{E} g(x,y) &= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x,y) \, f(x,y) \, dx dy
\end{align}

We will illustrate this by proving the following theorem.

#### Theorem
If $X,Y$ are independent, then $\mathbb{E}(XY) = \mathbb{E}(X)\mathbb{E}(Y)$.

Or, in other words, the _independence_ of r.v. $X$ and $Y$ implies that they are _uncorrelated_.

#### Proof

Consider the continuous case

\begin{align}
  \mathbb{E}(XY) &= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} xy \, f(x,y) \, dx dy &\quad \text{2D LOTUS} \\
  &= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} xy \, f_{X}(x) \, f_{Y}(y) \, dx dy &\quad \text{since } X,Y \text{ are independent} \\
  &= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} xy \, f_{X}(x) \, f_{Y}(y) \, dx \right) dy &\quad \text{this is a double integral, so we can group thusly} \\
  &= \int_{-\infty}^{\infty} y \, f_{Y}(y) \underbrace{ \left( \int_{-\infty}^{\infty} x \, f_{X}(x) \, dx \right)}_{\text{actually }\mathbb{E}(X) \text{, which is constant}} dy &\quad \text{since we are holding } y \text{ constant} \\
  &= \mathbb{E}(X)  \int_{-\infty}^{\infty} y \, f_{Y}(y) \, dy \\
  &= \mathbb{E}(X) \, \mathbb{E}(Y) &\quad \blacksquare
\end{align}

----

### Example: expected distance between 2 Uniformly distributed random points

Given _i.i.d._ $X,Y \sim Unif(0,1)$, find the expected distance between $X$ and $Y$, $\mathbb{E}|X-Y|$.

```python
plt.xkcd()

fig,ax = plt.subplots(1, figsize=(6.0,1.0))

plt.plot([1/3, 2/3],[0,0], 'ro')
ax.axhline(y=0, color='k', alpha=0.2)

ax.set_xticks([0.0, (1.0/3.0), (2.0/3.0), 1.0])
ax.set_xticklabels(['0', '1/3', '2/3', '1'])

ax.get_xaxis().set_ticks_position('bottom')
ax.get_yaxis().set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.spines['bottom'].set_visible(False)

plt.show()
```

We don't need to directly find the distribution of $|X-Y|$, because we are only interested in the _average distance between points $x,y$_. Using LOTUS, we have:

\begin{align}
  \mathbb{E}|X-Y| &= \int_{0}^{1} \int_{0}^{1} |x-y| \, dx dy &\quad \text{since }X,Y \text{ are i.i.d., the joint PDF }=1 \\
  &= \iint_\limits{x \gt y} (x-y) \, dx dy + \iint_\limits{x \le y} (y-x) \, dx dy &\quad \text{split into 2 integrals to deal with abs. value} \\
  &= 2 \int_{0}^{1} \int_{y}^{1} (x-y) \, dx dy  &\quad \text{by symmetry; and since } x \gt y \text{, inner integral starts from }y \\
  &= 2 \int_{0}^{1} \left( \frac{x^2}{2} - yx \right) \bigg|_{y}^{1} \, dy \\
  &= 2 \int_{0}^{1} \frac{1}{2} - y + \frac{y^2}{2} \, dy \\
  &= 2 \left( \frac{1}{6} \right) \\
  &= \boxed{ \frac{1}{3} }
\end{align}

----

But looking at that line-segment illustration above suggests another way of looking at this problem.

Let $M = max(X,Y)$

Let $L = min(X,Y)$

And so

\begin{align}
  |X-Y| &= M - L \\
  \\
  \mathbb{E}(M-L) &= \frac{1}{3} &\quad \text{from our previous proof} \\
  \mathbb{E}(M) - \mathbb{E}(L) &= \frac{1}{3} &\quad \text{since } X,Y \text{ are i.i.d.} \\
  \\
  \mathbb{E}(M+L)&= \mathbb{E}(M) + \mathbb{E}(L) &\quad \text{by linearity} \\
  &= \frac{1}{2} + \frac{1}{2} &\quad \text{since} X \sim Unif(0,1) \text{, } Y \sim Unif(0,1) \\
  &= 1 \\
  \\
  \mathbb{E}(M) &= \frac{2}{3} \\
  \mathbb{E}(L) &= \frac{1}{3}
\end{align}

----

### Chicken-egg Problem

Say we have $N$ eggs; let $N \sim Pois(\lambda)$

Some of the eggs hatch, while some don't. Each egg hatches with probability $p$. The event of an egg hatching is independent of the others.

Let $X$ be the number of eggs out of $N$ that do hatch, so $X|N \sim Bin(N,p)$.

Let $Y$ be the number of eggs that don't hatch, so $X+Y = N$.

Find the joint PMF of $X,Y$. Are they independent?

\begin{align}
  P(X{=}i, Y{=}j) &= \sum_{n=0}^{\infty} P(X{=}i, Y{=}j |N{=}n) \, P(N{=}n) \\
  \\
  &=  P(X{=}i, Y{=}j |N{=i+j}) \, P(N{=i+j})　&\quad \text{since we know that } i + j = n \\
  \\
  &=  P(X{=}i | N{=i+j}) \, P(N{=i+j})　&\quad Y{=}j \text{ is redundant} \\
  \\
  &= \binom{i+j}{i} p^i \, q^j \cdot \frac{e^{-\lambda} \lambda^{i+j}}{(i+j)!} &\quad \text{from definition of binomial and Poisson} \\
  \\
  &=  \frac{(i+j)!}{i! \, j!} p^i \, q^j \cdot \frac{e^{-\lambda} \lambda^{i+j}}{(i+j)!} \\
  \\
  &= \frac{(\lambda p)^i}{i!} \, \frac{(\lambda q)^j}{j!} \, e^{-\lambda} \\
  \\
  &= \left(e^{-\lambda p} \frac{(\lambda p)^i}{i!} \right) \left(e^{-\lambda q}  \frac{(\lambda q)^j}{j!} \right) &\quad \text{since } p + q = 1 \\
  \\
  &\Rightarrow X,Y \text{ are independent, } X \sim Pois(\lambda p), Y \sim Pois(\lambda q)
\end{align}

----


# Taylor integration of the Kepler problem

Here, we try to reproduce __exactly__ the [Kepler problem integration example](http://nbviewer.jupyter.org/github/JuliaDiff/TaylorSeries.jl/blob/master/examples/1-KeplerProblem.ipynb) made by Luis Benet in [JuliaDiff/TaylorSeries.jl](https://github.com/JuliaDiff/TaylorSeries.jl).

The Kepler problem is the basis of planetary motion; it describes the motion of a secondary body (e.g., a planet, asteroid, comet, etc.), around a primary body (e.g., the Sun). In cartesian coordinates over the orbital plane, the Hamiltonian for the Kepler problem reads:

$$
\begin{align}
H_{\mathrm{Kepler}} &= \frac{1}{2\mu}(p_x^2+p_y^2)-\frac{\mu}{\sqrt{x^2+y^2}}
\end{align}
$$

where $\mu=G(m_1+m_2)$, $G$ is the gravitational constant, $m_1$ is the mass of the primary body and $m_2$ is the mass of the secondary body. If we write $\vec r_1 = (x_1,y_1)$ and $\vec r_2 = (x_2,y_2)$, respectively, for the position of the primary and secondary body, then the vector $\vec r = \vec r_2-\vec r_1$, with coordinates $\vec r = (x, y)=(x_2-x_1,y_2-y_1)$, represents the position of the secondary body relative to the primary body; i.e., $\vec r$ represents the so-called relative coordinates. The primary body is at the origin.

Using Hamilton equations, we can obtain the equations of motion for the Kepler problem:

$$
\begin{align}
\dot x &= u \\
\dot y &= v \\
\dot u &= -\frac{\mu x}{(x^2+y^2)^{3/2}}\\
\dot v &= -\frac{\mu y}{(x^2+y^2)^{3/2}}
\end{align}
$$

Note that the canonical momenta are $p_x$ and $p_y$, while $u$ and $v$ are, respectively, the  $x$ and $y$ components of the velocity; i.e., $p_x = \mu u$ and $p_y = \mu v$.

First of all, we shall include all relevant packages:

```julia
using TaylorIntegration, PyPlot
```

Some parameters necessary for the integration:

+ $\mu$: the gravitational parameter
+ `q0`: the initial condition (we will select an initial condition which corresponds to elliptical motion)
+ `order`: the order of the Taylor expansion
+ `t_max`: the final time of the integration
+ `abs_tol`: the absolute tolerance
+ `n_iter`: the number of time-steps

```julia
const μ = 1.0
const q0 = [0.19999999999999996, 0.0, 0.0, 3.0] # a initial condition for elliptical motion
const order = 28
const t0 = 0.0
const t_max = 10000*(2π) # we are just taking a wild guess about the period ;)
const abs_tol = 1.0E-20
const steps = 500000
```

    500000

As usual, we write down the equations of motion into a `function`, which here we will name `kepler_problem`. `q` represents the system state, with the first component being the current value of the independent variable. Hence, its evolution is given by $\dot t = 1$. `params` represents the parameters of a given system of differential equations. In the Kepler problem case, the only parameter is $\mu$, the mass parameter.

```julia
function kepler_problem(t, q)
    r_p3d2 = (q[1]^2+q[2]^2)^(3/2)

    return [q[3], q[4], -μ*q[1]/r_p3d2, -μ*q[2]/r_p3d2]
end
```

    kepler_problem (generic function with 1 method)

The Taylor integration:

```julia
?taylorinteg
```

    search: taylorinteg TaylorIntegration

```
taylorinteg(f, x0, t0, tmax, order, abs_tol; keyword... )
```

This is a general-purpose Taylor integrator for the explicit ODE $\dot{x}=f(t,x)$ with initial condition specified by `x0` at time `t0`. It returns a vector with the values of time (independent variable), and a vector (of type `typeof(x0)`) with the computed values of the dependent variables. The integration stops when time is larger than `tmax`, or the number of saved steps is larger than `maxsteps`.

The integrator uses polynomial expansions on the independent variable of order `order` and the parameter `abs_tol` serves to define the time step using the last two Taylor coefficients of the expansions.

The current keyword argument is `maxsteps=500`.

```julia
t, q = taylorinteg(kepler_problem, q0, t0, t_max, order, abs_tol, maxsteps=2); #warm-up lap
@time t, q = taylorinteg(kepler_problem, q0, t0, t_max, order, abs_tol, maxsteps=steps);
```

    WARNING: Maximum number of integration steps reached; exiting.

     48.484457 seconds (621.08 M allocations: 39.178 GB, 8.69% gc time)

The final state:

```julia
t[end], q[end,:]
```

    (62831.853071795864,
    1x4 Array{Float64,2}:
     0.196132  0.0527212  -0.432649  2.94287)

Let's extract the values of $x$, $y$, $u$ and $v$ for each time-step:

```julia
x = q[:,1] #x position
y = q[:,2] #y position
u = q[:,3] #x velocity
v = q[:,4]; #y velocity
```

## Orbital motion

The initial conditions we selected correspond to a elliptical orbit with a relatively high eccentricity: $e=0.8$ for the initial condition `q0=[0.19999999999999996, 0.0, 0.0, 3.0]`. How does the motion of the planet/asteroid/comet looks like? Well, let's plot its orbits over the $x-y$ plane (the orbit is shown in blue; the position of the primary body is shown as a yellow dot):

```julia
title(L"\mathrm{The\,orbital\,motion}")
ylabel(L"y")
xlabel(L"x")
grid(true)
plot(0,0,"yo")
plot(x,y,",");
```

## Energy conservation

Below, we write the energy function of the Kepler problem; i.e., the Hamiltonian $H_{\mathrm{Kepler}}$ in terms of $x$, $y$, $u$ and $v$:

```julia
H(x_, y_, u_, v_) = 0.5.*(u_.*u_+v_.*v_)-μ./sqrt(x_.*x_+y_.*y_)
```

    H (generic function with 1 method)

Now, using the function `H` defined above, we calculate the energy during each time-step:

```julia
E = H(x, y, u, v);
```

$E_0=H_{\mathrm{Kepler}}(x_0,y_0,u_0,v_0)$, the initial value of the energy, is:

```julia
E0=E[1]
```

    -0.5000000000000009

We define $\delta E$ as the relative error in the energy; i.e.:

$$
\begin{align}
\delta E(t) &= \frac{E(t)-E_0}{E_0}
\end{align}
$$

```julia
δE = (E-E0)/(E0)
```

    436153-element Array{Float64,1}:
     -0.0
      1.77636e-15
      1.77636e-15
      8.88178e-16
      8.88178e-16
      2.66454e-15
      3.55271e-15
      2.66454e-15
      3.55271e-15
      2.66454e-15
      3.9968e-15
      2.66454e-15
      3.10862e-15
      ⋮
     -7.12763e-14
     -7.19425e-14
     -7.14984e-14
     -7.10543e-14
     -7.19425e-14
     -7.10543e-14
     -7.10543e-14
     -6.92779e-14
     -7.01661e-14
     -6.92779e-14
     -6.92779e-14
     -7.28306e-14

The analytical solution preserves the energy; i.e., $\delta E(t)=0$ for the analytical solution. Thus, we expect our solution to be close to zero. So, even if the $\delta E$ is not perfectly zero during the whole integration, it is comparable to Julia's `Float64` machine-epsilon.

Below, we plot $\delta E$ in units of Julia's `Float64` machine-epsilon as a function of time, $t$. In Julia, the machine-epsilon for the `Float64` type has a value

```julia
eps(Float64)
```

    2.220446049250313e-16

```julia
title(L"\mathrm{Energy\,relative\,error\,vs\,time}")
xlabel(L"t\mathrm{\,(orbital\,periods)}")
ylabel(L"\delta E \mathrm{\,(machine\,epsilons)}")
grid(true)
plot(t/(2π), δE/eps(Float64), ",");
```

Now, how does the energy error distribute around zero?

```julia
title(L"\mathrm{Distribution\,of\,energy\,relative\,error}")
xlabel(L"\delta E")
ylabel(L"\mathrm{Number\,of\,}\delta E\mathrm{\,values\,within\,bin\,range}")
hE = plt[:hist](δE/eps(Float64), 20);
```

The energy error behaves roughly as a random walk, which means that the numerical error in the energy is dominated by rounding errors.

## Angular momentum conservation

Analogously to the energy, we now focus on the angular momentum, which is preserved by the analytical solution too. The value of the angular momentum is given by
$$
\begin{align}
L &= xv-yu
\end{align}
$$
We write the angular momentum function as:

```julia
ang_mom(x_, y_, u_, v_) = x_.*v_-y.*u_
```

    ang_mom (generic function with 1 method)

So the angular momentum during each time-step is:

```julia
L = ang_mom(x, y, u, v);
```

$L_0=x_0v_0-y_0u_0$, the initial value of the angular momentum, is:

```julia
L0 = L[1]
```

    0.5999999999999999

We define $\delta L$ as the relative error in the angular momentum; i.e.:

$$
\begin{align}
\delta L(t) &= \frac{L(t)-L_0}{L_0}
\end{align}
$$

```julia
δL = (L-L0)/L0;
```

Just as we did for the energy, we now plot $\delta L$ in units of `eps(Float64)` vs $t$:

```julia
title(L"\mathrm{Angular\,momentum\,relative\,error\,vs\,time}")
xlabel(L"t\mathrm{\,(orbital\,periods)}")
ylabel(L"\delta L \mathrm{\,(machine\,epsilons)}")
grid(true)
plot(t/(2π), δL/eps(Float64), "g,");
```

Again, we see that the angular momentum relative error is comparable to `eps(Float64)`; the maximum variation is ~$150$ machine epsilons. How does the distribution of this error look like?

```julia
title(L"\mathrm{Distribution\,of\,angular\,momentum\,relative\,error}")
xlabel(L"\delta L")
ylabel(L"\mathrm{Number\,of\,}\delta L\mathrm{\,values\,within\,bin\,range}")
hL = plt[:hist]( δL/eps(Float64), 20 );
```

The distribution of $\delta L$ shows a peak near zero and roughly symmetrical around this value. This means that the error in the angular momentum is dominated also by rounding errors of floating-point arithmetic.

Lastly, we reproduce __exactly__ the last plot shown in the original version of this example, authored by Luis Benet and included in `JuliaDiff/TaylorSeries.jl`'s [Kepler problem integration example](http://nbviewer.jupyter.org/github/JuliaDiff/TaylorSeries.jl/blob/master/examples/1-KeplerProblem.ipynb) jupyter notebook.

A $\delta E$, $\delta L$ plot vs $t$:

```julia
title(L"\delta E\mathrm{\,(blue),\,}\delta L\mathrm{\,(green)\,vs\,time}")
ylabel(L"\delta E\mathrm{,\,}\delta L\,\mathrm{(machine\,epsilons)}")
xlabel(L"t\,\mathrm{(orbital\,periods)}")
grid(true)
plot(t/(2π), δE/eps(Float64), "b,", t/(2π), δL/eps(Float64), "g,");
```


# Least-Squares Regression Workbook
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

# Example 1
The height of a person vs femur length is given by the following data:

| femur length (cm) | height (cm) |
| :----------------: |:-----------: |
| 40	            | 163|
|41	|165|
|43	|167|
|43	|164|
|44	|168|
|44	|169|
|45	|170|
|45	|167|
|48	|170|
|51	|175|

Plot this data and then perform regression to a line and a quadratic using the standard derivation as well as the normal equations. Also use `numpy`'s `polyfit` and compare to you solutions.

Let's first get the boiler plate out of the way and import matplotlib and numpy

```python
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
import numpy as np
```

Now place the input data in `numpy` arrays. We can enter these manually or import from a text file.

```python
xi = np.array([40,41,43,43,44,44,45,45,48,51])
yi = np.array([163,165,167,164,168,169,170,167,170,175])
```

Now plot the data

```python
plt.plot(xi,yi,'ro')
plt.xlabel('Femur length (cm)')
plt.ylabel('Person\' height (cm)')
plt.title('Height vs femur length')
plt.grid()
```

## Regression to a straight line
Here, we regress the data to a straight line $a_0 + a_1 x$. Recall that, for regression to a straight line, we must solve the following system of equations:
\begin{equation}
\label{eq:regression-line}
\left[
\begin{array}{*{20}{c}}
N& \sum{x_i}\\
\sum{x_i} &\sum{x_i^2}
\end{array}
\right]
\left(
\begin{array}{*{20}{c}}
a_0\\
a_1
\end{array} \right)
= \left(
\begin{array}{*{20}{c}}
\sum{y_i}\\
\sum {x_i y_i}
\end{array}
\right)
\end{equation}

first build the coefficient matrix as a list of lists [ [a,b], [c,d]]. We will use `numpy.sum` to compute the various summations in the system \eqref{eq:regression-line}.

```python
N = len(xi)
A = np.array([[N, np.sum(xi)],
                 [np.sum(xi), np.sum(xi**2)]])
```

next, construct the right-hand-side using a 1D numpy array

```python
b = np.array([np.sum(yi), np.sum(xi*yi)])
```

now find the solution of $[\mathbf{A}]\mathbf{a} = \mathbf{b}$, where $a = (a_0, a_1)$ are the coefficients of the regressed line. Use `numpy`'s built-in linear solver.

```python
sol = np.linalg.solve(A,b)
# print the solution. Note that in this case, the solution should contain two values, a0 and a1, respectively.
print(sol)
```

    [123.20779221   1.004329  ]

The variable `sol` contains the solution of the system of equations. It is a list of two entries corresponding to the coefficients $a_0$ and $a_1$ of the regressed line.

We can now plot the line, $a_0 + a_1 x$ that we just fitted into the data

```python
# first get the coefficients a0 and a1 from the variable sol
a0 = sol[0]
a1 = sol[1]

# construct a function f(x) for the regressed line
y_line = lambda x: a0 + a1*x

# now plot the regressed line as a function of the input data xi
plt.plot(xi, y_line(xi), label='least-squares fit')
# plot the original data
plt.plot(xi, yi,'ro', label='original data')
plt.xlabel('Femur length (cm)')
plt.ylabel('Person\' height (cm)')
plt.title('Height vs femur length')
plt.legend()
plt.grid()
```

## Standard Error

The standard error of the model quantifies the spread of the data around the regression curve. It is given by
\begin{equation}
S_{y/x} = \sqrt{\frac{S_r}{n-2}} = \sqrt{\frac{\sum{(y_i - f_i)^2}}{n-2}}
\end{equation}

```python
ybar = np.average(yi)
print(ybar)
fi = y_line(xi)
stdev = np.sqrt(np.sum((yi - ybar)**2)/(N-1))
print(stdev - ybar)
Sr = np.sum((yi-fi)**2)
Syx = np.sqrt(Sr/(N-2))
print(Syx)
```

    167.8
    -164.3103327124527
    1.4317065166379408

### R2 Value

The $R^2$ value can be computed as:
\begin{equation}R^2 = 1 - \frac{\sum{(y_i - f_i)^2}}{\sum{(y_i - \bar{y})^2}}\end{equation}
where
\begin{equation}
\bar{y} = \frac{1}{N}\sum{y_i}
\end{equation}

```python
ybar = np.average(yi)
fi = y_line(xi)
rsq = 1 - np.sum( (yi - fi)**2 )/np.sum( (yi - ybar)**2 )
print(rsq)
```

    0.8503807627895221

To enable quick calculation of the R2 value for other functions, let's declare a function that computes the R2 value for an arbitrary model fit.

```python
def rsquared(xi,yi,ymodel):
    '''
    xi: vector of length n representing the known x values.
    yi: vector of length n representing the known y values that correspond to xi.
    ymodel: a python function (of x only) that can be evaluated at xi and represents a model fit of
            the data (e.g. a regressed curve).
    '''
    ybar = np.average(yi)
    fi = ymodel(xi)
    result = 1 - np.sum( (yi - fi)**2 )/np.sum( (yi - ybar)**2 )
    return result
```

```python
rsq = rsquared(xi,yi,y_line)
print(rsq)
```

    0.8503807627895221

## Regression to a quadratic
Here, we regress the data to a quadratic polynomial $a_0 + a_1 x + a_2 x^2$. Recall that, for regression to a quadratic, we must solve the following system of equations:
\begin{equation}
\label{eq:regression-line}
\left[
\begin{array}{*{20}{c}}
N& \sum{x_i} & \sum{x_i^2}\\
\sum{x_i} &\sum{x_i^2} & \sum{x_i^3} \\
\sum{x_i^2} & \sum{x_i^3} & \sum{x_i^4}
\end{array}
\right]
\left(
\begin{array}{*{20}{c}}
a_0\\
a_1 \\
a_2
\end{array} \right)
= \left(
\begin{array}{*{20}{c}}
\sum{y_i}\\
\sum {x_i y_i}\\
\sum {x_i^2 y_i}
\end{array}
\right)
\end{equation}

```python
A = np.array([[len(xi),    np.sum(xi)  , np.sum(xi**2)  ],
                 [np.sum(xi), np.sum(xi**2), np.sum(xi**3)],
                 [np.sum(xi**2), np.sum(xi**3), np.sum(xi**4)] ])
b = np.array([np.sum(yi), np.sum(xi*yi), np.sum(xi*xi*yi)])
print(A,b)
```

    [[      10      444    19806]
     [     444    19806   887796]
     [   19806   887796 39994422]] [   1678   74596 3331896]

```python
sol = np.linalg.solve(A,b)
print(sol)
```

    [1.28368173e+02 7.76379556e-01 2.50458155e-03]

```python
a0 = sol[0]
a1 = sol[1]
a2 = sol[2]
y_quad = lambda x: a0 + a1*x + a2*x**2
# now plot the regressed line as a function of the input data xi
plt.plot(xi, y_quad(xi), label='least-squares fit (quadratic)')
# plot the original data
plt.plot(xi, yi,'ko', label='original data')
plt.xlabel('Femur length (cm)')
plt.ylabel('Person\' height (cm)')
plt.title('Height vs femur length')
plt.legend()
plt.grid()
```

Let's now compute the R2 value for the quadratic fit

```python
rsq = rsquared(xi,yi,y_quad)
print(rsq)
```

    0.8504537705463819

This value of 85% is not much different than the one we got with a straight line fit. The data is very likely to be distributed linearly.

## Using the normal equations
Fit a straight line to the data using the normal equations

```python
A = np.array([np.ones(len(xi)),xi]).T
print(A)
ATA =  A.T @ A
sol = np.linalg.solve(ATA,A.T@yi)
print(sol)
```

    [[ 1. 40.]
     [ 1. 41.]
     [ 1. 43.]
     [ 1. 43.]
     [ 1. 44.]
     [ 1. 44.]
     [ 1. 45.]
     [ 1. 45.]
     [ 1. 48.]
     [ 1. 51.]]
    [123.20779221   1.004329  ]

For a quadratic fit using the normal equations

```python
A = np.array([np.ones(len(xi)),xi,xi**2]).T
ATA =  A.T @ A
sol = np.linalg.solve(ATA,A.T@yi)
print(sol)
```

    [1.28368173e+02 7.76379556e-01 2.50458155e-03]

## Using Numpy's Polyfit
It is possible to repeat the previous analysis using `numpy's` `polyfit` function. Let's try it out and compare to our results

We first compare a straight line fit

```python
coefs = np.polyfit(xi,yi,1)
print(coefs)
```

    [  1.004329   123.20779221]

Indeed, the coefficients are the same as the ones we obtained from our straight-line fit. Polyfit returns the coefs sorted in reverse order.

To use the data returned by `polyfit`, we can construct a polynomial using `poly1d` and simply pass the coefficients returned by `polyfit`

```python
p = np.poly1d(coefs)
```

no, simply call `p` on any value, say `p(43)`

```python
print(p(43))
```

    166.39393939393938

We can also plot the fit

```python
# now plot the regressed line as a function of the input data xi
plt.plot(xi, p(xi), label='least-squares fit (quadratic)')
# plot the original data
plt.plot(xi, yi,'ko', label='original data')
plt.xlabel('Femur length (cm)')
plt.ylabel('Person\' height (cm)')
plt.title('Height vs femur length')
plt.legend()
plt.grid()
```

We can do the same for the quadratic fit - simply change the degree of the polynomial in `polyfit`

```python
coefs = np.polyfit(xi,yi,2)
pquad = np.poly1d(coefs)
print(pquad(43))
```

    166.38346568926897

## Using numpy's Least Squares

```python
np.linalg.lstsq(A,yi,rcond=None)
```

    (array([1.28368173e+02, 7.76379556e-01, 2.50458155e-03]),
     array([16.39026675]),
     3,
     array([6.32567302e+03, 9.94238708e+00, 1.72620823e-02]))

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


# Sympy

```python
from sympy import *

# init_printing()
x, y, z = symbols("x y z")

```

```python
simplify(sin(x) ** 2 + cos(x) ** 2)
```

$\displaystyle 1$

```python
expand((x + 1) ** 3)
```

$\displaystyle x^{3} + 3 x^{2} + 3 x + 1$

```python
a = 3
b = 8
c = 2
y = a * x ** 2 + b * x + c

plot(y)
```

```python
solveset(y)
```

$\displaystyle \left\{- \frac{4}{3} - \frac{\sqrt{10}}{3}, - \frac{4}{3} + \frac{\sqrt{10}}{3}\right\}$

```python
dy = y.diff(x)

plot(dy)
```

```python
iy = integrate(y, x)

plot(iy)
```


# Нотация Денавита-Хартенберга

```python
from sympy import *
def rz(a):
    return Matrix([
        [cos(a), -sin(a), 0, 0],
        [sin(a), cos(a), 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1]
    ])

def ry(a):
    return Matrix([
        [cos(a), 0, sin(a), 0],
        [0, 1, 0, 0],
        [-sin(a), 0, cos(a), 0],
        [0, 0, 0, 1]
    ])

def rx(a):
    return Matrix([
        [1, 0, 0, 0],
        [0, cos(a), -sin(a), 0],
        [0, sin(a), cos(a), 0],
        [0, 0, 0, 1]
    ])

def trs(x, y, z):
    return Matrix([
        [1, 0, 0, x],
        [0, 1, 0, y],
        [0, 0, 1, z],
        [0, 0, 0, 1]
    ])

def vec(x, y, z):
    return Matrix([
        [x],
        [y],
        [z],
        [1]
    ])
```

Если соединить матрицы и винтовое исчисление, одним из результатов будет нотация Денавита-Хартенберга.
Она подразумевает четыре последовательных преобразовния:
$$
DH(\theta, d, \alpha, a) =
R_z(\theta) T_z(d) R_x(\alpha) T_z(a)
$$

```python
def dh(theta, d, alpha, a):
    return rz(theta) * trs(0, 0, d) * rx(alpha) * trs(a, 0, 0)
```

```python
theta, d, alpha, a = symbols("theta_i, d_i, alpha_i, a_i")
simplify(dh(theta, d, alpha, a))
```

DH-параметры описывают последовательные
- поворот вокруг оси $Z$ - $\theta$
- смещение вдоль оси $Z$ - $d$
- поворот вокруг новой оси $X$ - $\alpha$
- смещение вдоль новой оси $X$ - $r$

Обобщенные координаты выбираются так, чтобы попадать на вращение вокруг / смещение вдоль оси $Z$.


```python
import loader
from sympy import *
init_printing()
from root.solver import *
```

#### Solve $y'' - 4y' + 4y = 2t^2 e^{2t}$ using the method of undetermined coefficients

```python
coeffs = [1, -4, 4]
yc, p = nth_order_const_coeff(*coeffs)
p.display()
yp, p = undetermined_coefficients(yc, coeffs, 2 * t ** 2 * exp(2*t))
p.display()
```

$\displaystyle \text{Characteristic equation: }$

$\displaystyle r^{2} - 4 r + 4 = 0$

$\displaystyle \text{Roots: }\left\{ \begin{array}{ll}r_{1,2} = 2\\\end{array} \right.$

$\displaystyle \text{General Solution: }$

$\displaystyle y = \left(C_{1} + C_{2} t\right) e^{2 t}$

$\displaystyle \text{Find }Y(t)\text{ that mimics the form of }g(t)$

$\displaystyle Y{\left (t \right )} = A_{0} t^{2} e^{2 t} + A_{1} t^{4} e^{2 t} + A_{2} t^{3} e^{2 t}$

$\displaystyle \text{Compute successive derivatives of }Y(t)$

$\displaystyle \begin{align*}&Y{\left (t \right )} = t^{2} \left(A_{0} + A_{1} t^{2} + A_{2} t\right) e^{2 t}\\&\frac{d}{d t} Y{\left (t \right )} = t \left(2 A_{0} t + 2 A_{0} + 2 A_{1} t^{3} + 4 A_{1} t^{2} + 2 A_{2} t^{2} + 3 A_{2} t\right) e^{2 t}\\&\frac{d^{2}}{d t^{2}} Y{\left (t \right )} = 2 \left(2 A_{0} t^{2} + 4 A_{0} t + A_{0} + 2 A_{1} t^{4} + 8 A_{1} t^{3} + 6 A_{1} t^{2} + 2 A_{2} t^{3} + 6 A_{2} t^{2} + 3 A_{2} t\right) e^{2 t}\\\end{align*}$

$\displaystyle \text{Plug the derivatives into the LHS and equate coefficients}$

$\displaystyle \begin{align*}&4 t^{2} \left(A_{0} + A_{1} t^{2} + A_{2} t\right) e^{2 t} - 4 t \left(2 A_{0} t + 2 A_{0} + 2 A_{1} t^{3} + 4 A_{1} t^{2} + 2 A_{2} t^{2} + 3 A_{2} t\right) e^{2 t} + 2 \left(2 A_{0} t^{2} + 4 A_{0} t + A_{0} + 2 A_{1} t^{4} + 8 A_{1} t^{3} + 6 A_{1} t^{2} + 2 A_{2} t^{3} + 6 A_{2} t^{2} + 3 A_{2} t\right) e^{2 t} = 2 t^{2} e^{2 t}\\&2 A_{0} e^{2 t} + 12 A_{1} t^{2} e^{2 t} + 6 A_{2} t e^{2 t} = 2 t^{2} e^{2 t}\\\end{align*}$

$\displaystyle \left\{ \begin{array}{ll}12 A_{1} - 2 = 0\\0 = 0\\0 = 0\\6 A_{2} = 0\\2 A_{0} = 0\\\end{array} \right.$

$\displaystyle \text{Solve for the undetermined coefficients}$

$\displaystyle \left\{ \begin{array}{ll}A_{0} = 0\\A_{1} = \frac{1}{6}\\A_{2} = 0\\\end{array} \right.$

$\displaystyle \text{Substitute the coefficients to get the particular solution}$

$\displaystyle y_{p} = \frac{t^{4} e^{2 t}}{6}$

#### Given that ${1, \cos(t), \sin(t)}$ are the complementary solutions of $y''' + y' = 2\cos(2t) + 3\sin(t)$, find the particular solution.

```python
yp, p = undetermined_coefficients([1, cos(t), sin(t)], [1, 0, 1, 0], 2*cos(2*t) + 3*sin(t))
p.display()
```

$\displaystyle \text{Find }Y(t)\text{ that mimics the form of }g(t)$

$\displaystyle Y{\left (t \right )} = A_{0} t \cos{\left (t \right )} + A_{1} \cos{\left (2 t \right )} + A_{2} t \sin{\left (t \right )} + A_{3} \sin{\left (2 t \right )}$

$\displaystyle \text{Compute successive derivatives of }Y(t)$

$\displaystyle \begin{align*}&Y{\left (t \right )} = A_{0} t \cos{\left (t \right )} + A_{1} \cos{\left (2 t \right )} + A_{2} t \sin{\left (t \right )} + A_{3} \sin{\left (2 t \right )}\\&\frac{d}{d t} Y{\left (t \right )} = - A_{0} t \sin{\left (t \right )} + A_{0} \cos{\left (t \right )} - 2 A_{1} \sin{\left (2 t \right )} + A_{2} t \cos{\left (t \right )} + A_{2} \sin{\left (t \right )} + 2 A_{3} \cos{\left (2 t \right )}\\&\frac{d^{2}}{d t^{2}} Y{\left (t \right )} = - A_{0} t \cos{\left (t \right )} - 2 A_{0} \sin{\left (t \right )} - 4 A_{1} \cos{\left (2 t \right )} - A_{2} t \sin{\left (t \right )} + 2 A_{2} \cos{\left (t \right )} - 4 A_{3} \sin{\left (2 t \right )}\\&\frac{d^{3}}{d t^{3}} Y{\left (t \right )} = A_{0} t \sin{\left (t \right )} - 3 A_{0} \cos{\left (t \right )} + 8 A_{1} \sin{\left (2 t \right )} - A_{2} t \cos{\left (t \right )} - 3 A_{2} \sin{\left (t \right )} - 8 A_{3} \cos{\left (2 t \right )}\\\end{align*}$

$\displaystyle \text{Plug the derivatives into the LHS and equate coefficients}$

$\displaystyle \begin{align*}&- 2 A_{0} \cos{\left (t \right )} + 6 A_{1} \sin{\left (2 t \right )} - 2 A_{2} \sin{\left (t \right )} - 6 A_{3} \cos{\left (2 t \right )} = 3 \sin{\left (t \right )} + 2 \cos{\left (2 t \right )}\\&- 2 A_{0} \cos{\left (t \right )} + 6 A_{1} \sin{\left (2 t \right )} - 2 A_{2} \sin{\left (t \right )} - 6 A_{3} \cos{\left (2 t \right )} = 3 \sin{\left (t \right )} + 2 \cos{\left (2 t \right )}\\\end{align*}$

$\displaystyle \left\{ \begin{array}{ll}0 = 0\\- 2 A_{2} - 3 = 0\\0 = 0\\6 A_{1} = 0\\- 6 A_{3} - 2 = 0\\- 2 A_{0} = 0\\\end{array} \right.$

$\displaystyle \text{Solve for the undetermined coefficients}$

$\displaystyle \left\{ \begin{array}{ll}A_{0} = 0\\A_{1} = 0\\A_{2} = - \frac{3}{2}\\A_{3} = - \frac{1}{3}\\\end{array} \right.$

$\displaystyle \text{Substitute the coefficients to get the particular solution}$

$\displaystyle y_{p} = - \frac{3 t}{2} \sin{\left (t \right )} - \frac{\sin{\left (2 t \right )}}{3}$

```python

```


```python
from sympy import *
init_printing(use_latex='mathjax')
x, y, z = symbols('x,y,z')
r, theta = symbols('r,theta', positive=True)
```

## Matrices

The SymPy `Matrix` object helps us with small problems in linear algebra.

```python
rot = Matrix([[r*cos(theta), -r*sin(theta)],
              [r*sin(theta),  r*cos(theta)]])
rot
```

### Standard methods

```python
rot.det()
```

```python
rot.inv()
```

```python
rot.singular_values()
```

### Exercise

Find the inverse of the following Matrix:

$$ \left[\begin{matrix}1 & x\\y & 1\end{matrix}\right] $$

```python
# Create a matrix and use the `inv` method to find the inverse

```

### Operators

The standard SymPy operators work on matrices.

```python
rot * 2
```

```python
rot**2
```

```python
v = Matrix([[x], [y]])
v
```

```python
rot * v
```

### Exercise

In the last exercise you found the inverse of the following matrix

```python
M = Matrix([[1, x], [y, 1]])
M
```

```python
M.inv()
```

Now verify that this is the true inverse by multiplying the matrix times its inverse.  Do you get the identity matrix back?

```python
# Multiply `M` by its inverse.  Do you get back the identity matrix?

```

### Exercise

What are the eigenvectors and eigenvalues of `M`?

```python
# Find the methods to compute eigenvectors and eigenvalues. Use these methods on `M`

```

### NumPy-like Item access

```python
rot[0, 0]
```

```python
rot[:, 0]
```

```python
rot[1, :]
```

### Mutation

We can change elements in the matrix.

```python
rot[0, 0] += 1
rot
```

```python
simplify(rot.det())
```

```python
rot.singular_values()
```

### Exercise

Play around with your matrix `M`, manipulating elements in a NumPy like way.  Then try the various methods that we've talked about (or others).  See what sort of answers you get.

```python
# Play with matrices

```


### Example 3 , part B: Diffusion for non uniform material properties

In this example we will look at the diffusion equation for non uniform material properties and how to handle second-order derivatives. For this, we will reuse Devito's `.laplace` short-hand expression outlined in the previous example and demonstrate it using the examples from step 7 of the original tutorial. This example is an enhancement of `03_diffusion` in terms of having non-uniform viscosity as opposed to the constant $\nu$. This example introduces the use of `Function` in order to create this non-uniform space.

So, the equation we are now trying to implement is

$$\frac{\partial u}{\partial t} = \nu(x,y) \frac{\partial ^2 u}{\partial x^2} + \nu(x,y) \frac{\partial ^2 u}{\partial y^2}$$

In our case $\nu$ is not uniform and $\nu(x,y)$ represents spatially variable viscosity.
To discretize this equation we will use central differences and reorganizing the terms yields

\begin{align}
u_{i,j}^{n+1} = u_{i,j}^n &+ \frac{\nu(x,y) \Delta t}{\Delta x^2}(u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n) \\
&+ \frac{\nu(x,y) \Delta t}{\Delta y^2}(u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n)
\end{align}

As usual, we establish our baseline experiment by re-creating some of the original example runs. So let's start by defining some parameters.

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 100
ny = 100
nt = 1000

nu = 0.15 #the value of base viscosity

offset = 1 # Used for field definition

visc =  np.full((nx, ny), nu) # Initialize viscosity
visc[nx//4-offset:nx//4+offset, 1:-1] = 0.0001 # Adding a material with different viscosity
visc[1:-1,nx//4-offset:nx//4+offset ] = 0.0001
visc[3*nx//4-offset:3*nx//4+offset, 1:-1] = 0.0001

visc_nb = visc[1:-1,1:-1]

dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .25
dt = sigma * dx * dy / nu

# Initialize our field

# Initialise u with hat function
u_init = np.empty((nx, ny))
init_hat(field=u_init, dx=dx, dy=dy, value=1)
u_init[10:-10, 10:-10] = 1.5

zmax = 2.5 # zmax for plotting
```

We now set up the diffusion operator as a separate function, so that we can re-use if for several runs.

```python
def diffuse(u, nt ,visc):
    for n in range(nt + 1):
        un = u.copy()
        u[1:-1, 1:-1] = (un[1:-1,1:-1] +
                        visc*dt / dy**2 * (un[1:-1, 2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                        visc*dt / dx**2 * (un[2:,1: -1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))
        u[0, :] = 1
        u[-1, :] = 1
        u[:, 0] = 1
        u[:, -1] = 1
```

Now let's take this for a spin. In the next two cells we run the same diffusion operator for a varying number of timesteps to see our "hat function" dissipate to varying degrees.

```python
#NBVAL_IGNORE_OUTPUT

# Plot material according to viscosity, uncomment to plot
import matplotlib.pyplot as plt
plt.imshow(visc_nb, cmap='Greys',  interpolation='nearest')

# Field initialization
u = u_init

print ("Initial state")
plot_field(u, zmax=zmax)

diffuse(u, nt , visc_nb )
print ("After", nt, "timesteps")
plot_field(u, zmax=zmax)

diffuse(u, nt, visc_nb)
print ("After another", nt, "timesteps")
plot_field(u, zmax=zmax)
```

You can notice that the area with lower viscosity is not diffusing its heat as quickly as the area with higher viscosity.

```python
#NBVAL_IGNORE_OUTPUT

# Field initialization
u = u_init

diffuse(u, nt , visc_nb)
print ("After", nt, "timesteps")
plot_field(u, zmax=zmax)
```

Excellent. Now for the Devito part, we need to note one important detail to our previous examples: we now have a second-order derivative. So, when creating our `TimeFunction` object we need to tell it about our spatial discretization by setting `space_order=2`. We also use the notation `u.laplace` outlined previously to denote all second order derivatives in space, allowing us to reuse this code for 2D and 3D examples.

```python
from devito import Grid, TimeFunction, Eq, solve, Function
from sympy.abc import a
from sympy import nsimplify

# Initialize `u` for space order 2
grid = Grid(shape=(nx, ny), extent=(2., 2.))

# Create an operator with second-order derivatives
a = Function(name='a',grid = grid) # Define as Function
a.data[:]= visc  # Pass the viscosity in order to be used in the operator.

u = TimeFunction(name='u', grid=grid, space_order=2)

# Create an equation with second-order derivatives
eq = Eq(u.dt, a * u.laplace)
stencil = solve(eq, u.forward)
eq_stencil = Eq(u.forward, stencil)

print(nsimplify(eq_stencil))
```

    Eq(u(t + dt, x, y), dt*((-2*u(t, x, y)/h_y**2 + u(t, x, y - h_y)/h_y**2 + u(t, x, y + h_y)/h_y**2 - 2*u(t, x, y)/h_x**2 + u(t, x - h_x, y)/h_x**2 + u(t, x + h_x, y)/h_x**2)*a(x, y) + u(t, x, y)/dt))

Great. Now all that is left is to put it all together to build the operator and use it on our examples. For illustration purposes we will do this in one cell, including update equation and boundary conditions.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator, Constant, Eq, solve, Function

# Reset our data field and ICs
init_hat(field=u.data[0], dx=dx, dy=dy, value=1.)

# Field initialization
u.data[0] = u_init

# Create an operator with second-order derivatives
a = Function(name='a',grid = grid)
a.data[:]= visc

eq = Eq(u.dt, a * u.laplace, subdomain=grid.interior)
stencil = solve(eq, u.forward)
eq_stencil = Eq(u.forward, stencil)

# Create boundary condition expressions
x, y = grid.dimensions
t = grid.stepping_dim
bc = [Eq(u[t+1, 0, y], 1.)]  # left
bc += [Eq(u[t+1, nx-1, y], 1.)]  # right
bc += [Eq(u[t+1, x, ny-1], 1.)]  # top
bc += [Eq(u[t+1, x, 0], 1.)]  # bottom

op = Operator([eq_stencil] + bc)
op(time=nt, dt=dt, a = a)

print ("After", nt, "timesteps")
plot_field(u.data[0], zmax=zmax)

op(time=nt, dt=dt, a = a)
print ("After another", nt, "timesteps")
plot_field(u.data[0], zmax=zmax)
```

```python

```


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also http://splines.readthedocs.io/.

# Derivation of Non-Uniform Catmull--Rom Splines

Recursive algorithm developed by
Barry and Goldman (1988),
according to
Yuksel et al. (2011), figure 3.

```python
import sympy as sp
sp.init_printing()
```

```python
from utility import NamedExpression, NamedMatrix
```

```python
x_1, x0, x1, x2 = sp.symbols('xbm_-1 xbm:3')
```

```python
t, t_1, t0, t1, t2 = sp.symbols('t t_-1 t:3')
```

```python
p_10 = NamedExpression('pbm_-1,0', x_1 * (t0 - t) / (t0 - t_1) + x0 * (t - t_1) / (t0 - t_1))
p_10
```

```python
p01 = NamedExpression('pbm_0,1', x0 * (t1 - t) / (t1 - t0) + x1 * (t - t0) / (t1 - t0))
p01
```

```python
p12 = NamedExpression('pbm_1,2', x1 * (t2 - t) / (t2 - t1) + x2 * (t - t1) / (t2 - t1))
p12
```

```python
p_101 = NamedExpression('pbm_-1,0,1', p_10.name * (t1 - t) / (t1 - t_1) + p01.name * (t - t_1) / (t1 - t_1))
p_101
```

```python
p012 = NamedExpression('pbm_0,1,2', p01.name * (t2 - t) / (t2 - t0) + p12.name * (t - t0) / (t2 - t0))
p012
```

```python
p = NamedExpression('pbm', p_101.name * (t1 - t) / (t1 - t0) + p012.name * (t - t0) / (t1 - t0))
p
```

```python
p = p.subs([p_101, p012]).subs([p_10, p01, p12])
p
```

```python
p_normalized = p.expr.subs(t, t * (t1 - t0) + t0)
```

```python
M_CR = NamedMatrix(
    r'{M_\text{CR}}',
    sp.Matrix([[c.expand().coeff(x).factor() for x in (x_1, x0, x1, x2)]
               for c in p_normalized.as_poly(t).all_coeffs()]))
```

```python
deltas = [
    (t_1, -sp.Symbol('Delta_-1')),
    (t0, 0),
    (t1, sp.Symbol('Delta0')),
    (t2, sp.Symbol('Delta0') + sp.Symbol('Delta1'))
]
```

```python
M_CR.simplify().subs(deltas).factor()
```

```python
uniform = [
    (sp.Symbol('Delta_-1'), 1),
    (sp.Symbol('Delta0') , 1),
    (sp.Symbol('Delta1') , 1),
]
```

```python
M_CR.subs(deltas).subs(uniform).pull_out(sp.S.Half).expr
```

```python
velocity = p.expr.diff(t)
```

```python
velocity.subs(t, t0).subs(deltas).factor()
```

```python
velocity.subs(t, t1).subs(deltas).factor()
```

in general:

\begin{equation}
\boldsymbol{\dot{x}}_i =
\frac{
(t_{i+1} - t_i)^2 (\boldsymbol{x}_i - \boldsymbol{x}_{i-1}) +
(t_i - t_{i-1})^2 (\boldsymbol{x}_{i+1} - \boldsymbol{x}_i)
}{
(t_{i+1} - t_i)(t_i - t_{i-1})(t_{i+1} - t_{i-1})
}
\end{equation}

You might encounter another way to write the equation for $\boldsymbol{\dot{x}}_0$
(e.g. at https://stackoverflow.com/a/23980479/):

```python
(x0 - x_1) / (t0 - t_1) - (x1 - x_1) / (t1 - t_1) + (x1 - x0) / (t1 - t0)
```

... but this is equivalent to the equation shown above:

```python
_.subs(deltas).factor()
```

Yet another way to skin this cat -- sometimes referred to as Bessel--Overhauser -- is to define the velocity of the left and right chords:

```python
v_left = (x0 - x_1) / (t0 - t_1)
v_right = (x1 - x0) / (t1 - t0)
```

... and then combine them in this way:

```python
((t1 - t0) * v_left + (t0 - t_1) * v_right) / (t1 - t_1)
```

Again, that's the same as we had above:

```python
_.subs(deltas).factor()
```


```python
from sympy import *
```

```python
x,y,z = symbols('x y z')
init_printing(use_unicode=True)
```

### Basic *Calculus*

```python
print('------')

Limit((cos(x)-1),x,0)
Limit((cos(x)-1),x,0).doit()

print('------')

Derivative(5*x**2,x)
Derivative(5*x**2,x).doit()

print('------')

Integral(log(x)**2,x)
Integral(log(x)**2,x).doit()
```

```python
limit(sin(x)/x,x,0)
limit(sin(x)/x,x,oo)
```

```python
# use limit instead of subs (if there's a singularity)
(x/x**x).subs(x,oo)
limit((x/x**x),x,oo)

# 'limit' one side only
limit(1/x,x,0,'+')
limit(1/x,x,0,'-')
```

```python
diff(cos(x),x)

diff(5*x**3)
diff(5*x**3,x)
diff(5*x**3,x,1)

diff(x**2*y**2,y)
```

```python
diff(5*x**3,x,0)
diff(5*x**3*y**2,x,0,y,0)

diff(5*x**3*y**2,x,x,y,y)
diff(5*x**3*y**2,x,2,y,2)
```

```python
integrate(cos(x),x)

# use 'oo' to indicates 'infinity'
integrate(exp(-x),(x,0,oo))
integrate(exp(-x**2-y**2),(x,-oo,oo),(y,-oo,oo))

# oops!
integrate(x**x,x)
```

```python
exp(sin(x))
exp(sin(x)).series(x,0,5)

# hell yeah!
exp(x-5).series(x)
exp(x-5).series(x,x0=5)
```

### *Solver*

```python
# In Sympy, any expression not in an 'Eq'
#   is automatically assumed to equal 0 by solving funcs.

Eq(x,y)

# equals to 0? u can omit it!
solveset(Eq(x**2-5,0),x)
solveset(x**2-5,x)

# use Eq or not is FINE
solveset(Eq(x**2,x),x)
solveset(x**2-x,x)
```

```python
solveset(Eq(x**2,x),x,domain=S.Reals)
solveset(sin(x)-1,x,domain=S.Reals)
```

```python
# no solution exists
solveset(exp(x),x)

# not able to find solution
#   ( C代表虚数, 反V代表"与" )
solveset(cos(x)-x,x)
```

```python
linsolve([x+y+z-1,x+y+2*z-3],(x,y,z))
linsolve(Matrix(([1,1,1,1],[1,1,2,3])), (x,y,z))
```

```python
# nonlinear shit
a,b,c,d = symbols('a b c d',real=True)

nonlinsolve([a**2+a,a-b],[a,b])
nonlinsolve([x*y-1,x-2],x,y)

nonlinsolve([x**2+1,y**2+1],[x,y])
nonlinsolve([x**2-2*y**2-2,x*y-2],[x,y])
```

```python
# differential equations
f,g = symbols('f g',cls=Function)

f(x)
f(g(x))
```

```python
eq = Eq(f(x).diff(x,2) - 2*f(x).diff(x) + f(x),sin(x))

eq
dsolve(eq,f(x))

dsolve(Eq(f(x).diff(x)))
dsolve(f(x).diff(x)*(1-sin(f(x))),f(x))
```


# Getting Started with Equations
Equations are calculations in which one or more variables represent unknown values. In this notebook, you'll learn some fundamental techniques for solving simple equations.

## One Step Equations
Consider the following equation:

\begin{equation}x + 16 = -25\end{equation}

The challenge here is to find the value for **x**, and to do this we need to *isolate the variable*. In this case, we need to get **x** onto one side of the "=" sign, and all of the other values onto the other side. To accomplish this we'll follow these rules:
1. Use opposite operations to cancel out the values we don't want on one side. In this case, the left side of the equation includes an addition of 16, so we'll cancel that out by subtracting 16 and the left side of the equation becomes **x + 16 - 16**.
2. Whatever you do to one side, you must also do to the other side. In this case, we subtracted 16 from the left side, so we must also subtract 16 from the right side of the equation, which becomes **-25 - 16**.
Our equation now looks like this:

\begin{equation}x + 16 - 16 = -25 - 16\end{equation}

Now we can calculate the values on both side. On the left side, 16 - 16 is 0, so we're left with:

\begin{equation}x = -25 - 16\end{equation}

Which yields the result **-41**. Our equation is now solved, as you can see here:

\begin{equation}x = -41\end{equation}

It's always good practice to verify your result by plugging the variable value you've calculated into the original equation and ensuring that it holds true. We can easily do that by using some simple Python code.

To verify the equation using Python code, place the cursor in the following cell and then click the ►| button in the toolbar.

```python
x = -41
x + 16 == -25
```

## Two-Step Equations
The previous example was fairly simple - you could probably work it out in your head. So what about something a little more complex?

\begin{equation}3x - 2 = 10 \end{equation}

As before, we need to isolate the variable **x**, but this time we'll do it in two steps. The first thing we'll do is to cancel out the *constants*. A constant is any number that stands on its own, so in this case the 2 that we're subtracting on the left side is a constant. We'll use an opposite operation to cancel it out on the left side, so since the current operation is to subtract 2, we'll add 2; and of course whatever we do on the left side we also need to do on the right side, so after the first step, our equation looks like this:

\begin{equation}3x - 2 + 2 = 10 + 2 \end{equation}

Now the -2 and +2 on the left cancel one another out, and on the right side, 10 + 2 is 12; so the equation is now:

\begin{equation}3x = 12 \end{equation}

OK, time for step two - we need to deal with the *coefficients* - a coefficient is a number that is applied to a variable. In this case, our expression on the left is 3x, which means x multiplied by 3; so we can apply the opposite operation to cancel it out as long as we do the same to the other side, like this:

\begin{equation}\frac{3x}{3} = \frac{12}{3} \end{equation}

3x ÷ 3 is x, so we've now isolated the variable

\begin{equation}x = \frac{12}{3} \end{equation}

And we can calculate the result as 12/3 which is **4**:

\begin{equation}x = 4 \end{equation}

Let's verify that result using Python:

```python
x = 4
3*x - 2 == 10
```

## Combining Like Terms
Like terms are elements of an expression that relate to the same variable or constant (with the same *order* or *exponential*, which we'll discuss later). For example, consider the following equation:

\begin{equation}\textbf{5x} + 1 \textbf{- 2x} = 22 \end{equation}

In this equation, the left side includes the terms **5x** and **- 2x**, both of which represent the variable **x** multiplied by a coefficent. Note that we include the sign (+ or -) in front of the value.

We can rewrite the equation to combine these like terms:

\begin{equation}\textbf{5x - 2x} + 1 = 22 \end{equation}

We can then simply perform the necessary operations on the like terms to consolidate them into a single term:

\begin{equation}\textbf{3x} + 1 = 22 \end{equation}

Now, we can solve this like any other two-step equation. First we'll remove the constants from the left side - in this case, there's a constant expression that adds 1, so we'll use the opposite operation to remove it and do the same on the other side:

\begin{equation}3x + 1 - 1 = 22 - 1 \end{equation}

That gives us:

\begin{equation}3x = 21 \end{equation}

Then we'll deal with the coefficients - in this case x is multiplied by 3, so we'll divide by 3 on boths sides to remove that:

\begin{equation}\frac{3x}{3} = \frac{21}{3} \end{equation}

This give us our answer:

\begin{equation}x = 7 \end{equation}

```python
x = 7
5*x + 1 - 2*x == 22
```

## Working with Fractions
Some of the steps in solving the equations above have involved working wth fractions - which in themselves are actually just division operations. Let's take a look at an example of an equation in which our variable is defined as a fraction:

\begin{equation}\frac{x}{3} + 1 = 16 \end{equation}

We follow the same approach as before, first removing the constants from the left side - so we'll subtract 1 from both sides.

\begin{equation}\frac{x}{3} = 15 \end{equation}

Now we need to deal with the fraction on the left so that we're left with just **x**. The fraction is x/3 which is another way of saying *x divided by 3*, so we can apply the opposite operation to both sides. In this case, we need to multiply both sides by the denominator under our variable, which is 3. To make it easier to work with a term that contains fractions, we can express whole numbers as fractions with a denominator of 1; so on the left, we can express 3 as 3/1 and multiply it with x/3. Note that the notation for mutiplication is a **•** symbol rather than the standard *x* multiplication operator (which would cause confusion with the variable **x**) or the asterisk symbol used by most programming languages.

\begin{equation}\frac{3}{1} \cdot \frac{x}{3} = 15 \cdot 3 \end{equation}

This gives us the following result:

\begin{equation}x = 45 \end{equation}

Let's verify that with some Python code:

```python
x = 45
x/3 + 1 == 16
```

Let's look at another example, in which the variable is a whole number, but its coefficient is a fraction:

\begin{equation}\frac{2}{5}x + 1 = 11 \end{equation}

As usual, we'll start by removing the constants from the variable expression; so in this case we need to subtract 1 from both sides:

\begin{equation}\frac{2}{5}x = 10 \end{equation}

Now we need to cancel out the fraction. The expression equates to two-fifths times x, so the opposite operation is to divide by 2/5; but a simpler way to do this with a fraction is to multiply it by its *reciprocal*, which is just the inverse of the fraction, in this case 5/2. Of course, we need to do this to both sides:

\begin{equation}\frac{5}{2} \cdot \frac{2}{5}x = \frac{10}{1} \cdot \frac{5}{2} \end{equation}

That gives us the following result:

\begin{equation}x = \frac{50}{2} \end{equation}

Which we can simplify to:

\begin{equation}x = 25 \end{equation}

We can confirm that with Python:

```python
x = 25
2/5 * x + 1 ==11
```

## Equations with Variables on Both Sides
So far, all of our equations have had a variable term on only one side. However, variable terms can exist on both sides.

Consider this equation:

\begin{equation}3x + 2 = 5x - 1 \end{equation}

This time, we have terms that include **x** on both sides. Let's take exactly the same approach to solving this kind of equation as we did for the previous examples. First, let's deal with the constants by adding 1 to both sides. That gets rid of the -1 on the right:

\begin{equation}3x + 3 = 5x \end{equation}

Now we can eliminate the variable expression from one side by subtracting 3x from both sides. That gets rid of the 3x on the left:

\begin{equation}3 = 2x \end{equation}

Next, we can deal with the coefficient by dividing both sides by 2:

\begin{equation}\frac{3}{2} = x \end{equation}

Now we've isolated x. It looks a little strange because we usually have the variable on the left side, so if it makes you more comfortable you can simply reverse the equation:

\begin{equation}x = \frac{3}{2} \end{equation}

Finally, this answer is correct as it is; but 3/2 is an improper fraction. We can simplify it to:

\begin{equation}x = 1\frac{1}{2} \end{equation}

So x is 11/2 (which is of course 1.5 in decimal notation).
Let's check it in Python:

```python
x = 1.5
3*x + 2 == 5*x -1
```

## Using the Distributive Property
The distributive property is a mathematical law that enables us to distribute the same operation to terms within parenthesis. For example, consider the following equation:

\begin{equation}\textbf{4(x + 2)} + \textbf{3(x - 2)} = 16 \end{equation}

The equation includes two operations in parenthesis: **4(*x* + 2)** and **3(*x* - 2)**. Each of these operations consists of a constant by which the contents of the parenthesis must be multipled: for example, 4 times (*x* + 2). The distributive property means that we can achieve the same result by multiplying each term in the parenthesis and adding the results, so for the first parenthetical operation, we can multiply 4 by *x* and add it to 4 times +2; and for the second parenthetical operation, we can calculate 3 times *x* + 3 times -2). Note that the constants in the parenthesis include the sign (+ or -) that preceed them:

\begin{equation}4x + 8 + 3x - 6 = 16 \end{equation}

Now we can group our like terms:

\begin{equation}7x + 2 = 16 \end{equation}

Then we move the constant to the other side:

\begin{equation}7x = 14 \end{equation}

And now we can deal with the coefficient:

\begin{equation}\frac{7x}{7} = \frac{14}{7} \end{equation}

Which gives us our anwer:

\begin{equation}x = 2 \end{equation}

Here's the original equation with the calculated value for *x* in Python:

```python
x = 2
4*(x + 2) + 3*(x - 2) == 16
```


```python
import sympy
from IPython.display import display
import sympy as sp
from sympy import *
sympy.init_printing(use_latex=True)
xl, yl, d, x, y, theta, T, v_k, om_k, w_k, Q = symbols('xl yl d x y theta_k-1 T v_k om_k w_k Q')
X_k = sp.Matrix([[x],[y],[theta]]) + (T * sp.Matrix([[cos(theta),0], [sin(theta), 0], [0, 1]])@(sp.Matrix([[v_k], [om_k]])))
state = sp.Matrix([x, y, theta])
display(state)
F=X_k.jacobian(state)
display(F)
```

$$\left[\begin{matrix}x\\y\\\theta_{k-1}\end{matrix}\right]$$

$$\left[\begin{matrix}1 & 0 & - T v_{k} \sin{\left (\theta_{k-1} \right )}\\0 & 1 & T v_{k} \cos{\left (\theta_{k-1} \right )}\\0 & 0 & 1\end{matrix}\right]$$

```python
#xl, xk, theta, yl, yk, d = symbols('x_l x_k theta_k y_l y_k d')
z = sp.Matrix([[sqrt(((xl-x-d*cos(theta))**2) + ((yl-y-d*sin(theta))**2))], [atan2((yl-y-d*sin(theta)), (xl-x-d*cos(theta))) - theta]])
display(z)
H = z.jacobian(sp.Matrix([x, y, theta]))
display(H)
```

$$\left[\begin{matrix}\sqrt{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}}\\- \theta_{k-1} + \operatorname{atan_{2}}{\left (- d \sin{\left (\theta_{k-1} \right )} - y + yl,- d \cos{\left (\theta_{k-1} \right )} - x + xl \right )}\end{matrix}\right]$$

$$\left[\begin{matrix}\frac{d \cos{\left (\theta_{k-1} \right )} + x - xl}{\sqrt{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}}} & \frac{d \sin{\left (\theta_{k-1} \right )} + y - yl}{\sqrt{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}}} & \frac{- d \left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right) \cos{\left (\theta_{k-1} \right )} + d \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right) \sin{\left (\theta_{k-1} \right )}}{\sqrt{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}}}\\- \frac{d \sin{\left (\theta_{k-1} \right )} + y - yl}{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}} & - \frac{- d \cos{\left (\theta_{k-1} \right )} - x + xl}{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}} & \frac{d \left(d \sin{\left (\theta_{k-1} \right )} + y - yl\right) \sin{\left (\theta_{k-1} \right )}}{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}} - \frac{d \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right) \cos{\left (\theta_{k-1} \right )}}{\left(- d \sin{\left (\theta_{k-1} \right )} - y + yl\right)^{2} + \left(- d \cos{\left (\theta_{k-1} \right )} - x + xl\right)^{2}} - 1\end{matrix}\right]$$

```python

```


# Fourier Transforms in Python

A fundamental skill for anyone working on signal/image related data is the ability to analyze the frequencies (and strength of thsoe frequencies making up a signal). There are a few assumptions that we have to consider before taking a Fourier Transform.

1. The underlying signal is periodic.
2. The integral overal the entire input space (from $-\infty$ to $\infty$) is finite.

If you need a primer to remind you about Fourier Transforms, the [Wikipedia](https://en.wikipedia.org/wiki/Fourier_transform) and [Math World](https://mathworld.wolfram.com/FourierTransform.html) articles are a good place to start. The Fourier Transform and Inverse Fourier Transform are defined as

\begin{align}
    H(\omega) &=
    \mathcal{F}\left[h(t)\right] &=
    \int_{-\infty}^{\infty} h(t) e^{-i \omega t} dt \\
    h(t) &=
    \mathcal{F}^{-1}\left[H(\omega)\right] &=
    \frac{1}{2\pi} \int_{-\infty}^{\infty} H(\omega) e^{i \omega t} dt \\
\end{align}

respectively.

Now, when it comes to numerical programming and data analysis, we do not have a *continuous* signal to analyze (for which the equations above are derived). Instead, we have a *disccrete* signal for which we collect data at regular intervals. Therefore, we need likewise need a *discrete* Fourier Transform (DFT) which is defined as

\begin{align}
    F_n &=
    \sum_{k=0}^{N-1} f_k e^{-2 \pi i n k / N} \\
    f_k &=
    \frac{1}{N} \sum_{n=0}^{N-1} F_n e^{2 \pi i n k / N} \\
\end{align}

where $f_k$ and $F_n$ are the signals in the two different domains, respectively (such as time and frequency domains).

The final piece of information that we will need is the definition of the power spectrum which is what we will use to measure the strength of each given frequency. For the discreet transforms, the power spectrum is defined as

\begin{equation}
    S = F_n^* F_n.
\end{equation}

Perhaps this will be more convenient to understand with an example. Let's dive right in.

## Imports

```python
# Python Imports

# 3rd Party Imports
import numpy as np
import pandas as pd
from scipy.signal import periodogram
from matplotlib import pyplot as plt
```

## Fourier Transform Example

### Signal Creation

Let's begin by creating a signal to analyze. I'll define the underlying signal as

\begin{equation}
    x(t) = 5 \sin\left( 2 \pi f_1 t \right) + 7 \sin\left( 2 \pi f_2 t \right)
\end{equation}

where $f_1=2$ Hz and $f_2=5$ Hz. Again, since this is a *discrete* domain, we will also have to define the time step size which we will choose $\Delta t = 0.01$ s and we'll plot the underlying signal below.

```python
# Define the Variables
f1 = 2
f2 = 5
dt = 0.01
t = np.arange(0, 2, dt)
x = 5 * np.sin(2*np.pi*f1*t) + 7 * np.sin(2*np.pi*f2*t)

# Plot the Signal
_ = plt.plot(t, x, linewidth=2)
_ = plt.xlabel('Time (s)')
_ = plt.ylabel('Position (cm)')
_ = plt.title('Underlying Signal')
```

Now, to make this a little more realistic, let's add in some random Gaussian noise to this signal.

```python
# Get the Random Number Generator
rng = np.random.default_rng(0)

# Add the Random Numbers to the Signal
x += 4*rng.standard_normal(x.shape)

# Plot the Noisy Signal
_ = plt.plot(t, x, linewidth=2)
_ = plt.xlabel('Time (s)')
_ = plt.ylabel('Position (cm)')
_ = plt.title('Underlying Signal')
```

### Signal Analysis

At this point we are ready to start analyzing the signal. For this, we will use the Numpy Fast Fourier Transform (FFT) library.

```python
# Get the Fourier Transform
xT = np.fft.rfft(x)
```

Numpy provides several helper functions to parse through this data. We will use `rfftfreq` to get the frequencies of the transformed signal `xT`.

```python
# Get the measured frequencies
f = np.fft.rfftfreq(x.size, dt)
```

Now, if you attempted to plot this signal that has been transformed, you would receive a Numpy warning. This would arise due to the complex nature of the data. Due to the definition of the Fourier transform, the outputs are going to be, in general, complex. Therefore, we need a way to represent the overall magnitude of the transform. To do that, we will compute the square root of the power spectrum.

Now, the [rfft](https://docs.scipy.org/doc/numpy/reference/generated/numpy.fft.rfft.html) and [rfftfreq](https://docs.scipy.org/doc/numpy/reference/generated/numpy.fft.rfftfreq.html#numpy.fft.rfftfreq) have a few nuances that we have to consider.

1. The Fourier Transform is defined over all space (positive and negative frequencies), but each of these functions only returns values in the positive frequencies (i.e., half of the possible values). Therefore, we will have to multiply all of the non-zero frequencies by 2.
2. The DFT defined gets larger with the more data points we add. Therefore, we will have to divide the transformed signal by $N$ where is $N$ is the number of datapoints in $x$.

```python
# Get the Transform Magnitudes
xT[1:] *= 2  # Multiply the non-zero frequencies by 2.
magT = np.abs(xT/x.size)  # Get the Magnitude of the scaled transform.

# Plot the
_ = plt.plot(f, magT)
_ = plt.title('Signal Magnitude')
_ = plt.ylabel('Magnitude (cm)')
_ = plt.xlabel('Frequency (Hz)')
```

Scipy provides a convenient functon that calculates the RMS Power Spectrum. Therefore, we can use this function to wrap all the steps above into a single function call. However, since this is the *RMS* Power Spectrum, we will have to multiply this by two and take the square root to get the magnitudes we seek.

```python
# Get the Power Spectrum
f, spec = periodogram(x, 1/dt, scaling='spectrum')

# Plot the Magnitudes
_ = plt.plot(f, np.sqrt(spec*2))
_ = plt.title('Signal Magnitude')
_ = plt.ylabel('Magnitude (cm)')
_ = plt.xlabel('Frequency (Hz)')
```

Note that the signal we originally created was of the form

\begin{equation}
    x(t) = 5 \sin\left( 2 \pi f_1 t \right) + 7 \sin\left( 2 \pi f_2 t \right)
\end{equation}

where $f_1=2$ Hz and $f_2=5$ Hz. From the figure you can see that we recovered the frequencies and amplitudes that were used to create this signal. On both of the figures above, there is a peak approximately equal to 5 cm at $f=2$ Hz, and there is a peak approximately equal to 7 cm at $f=5$ Hz.

## Assignment

Your assignment is to study the periodicity of the total number of sunspots. I have provided the data, input lines to read in the data and the lines needed to clean the data below. I downloaded this [data](http://www.sidc.be/silso/INFO/sndtotcsv.php) from the [Sunspot Index and Long-term Solar Observations Website](http://sidc.be/silso/home).

```python
# Read in the Values as a Numpy array
ssDat = pd.read_csv(
    'SN_d_tot_V2.0.csv',
    sep=';',
    header=0,
    names=['Year', 'Month', 'Day', 'YearFraction', 'nSpots', 'std', 'nObs', 'Prov'],
    usecols=[3, 4],
    skiprows=6
).values

# Indicate -1 as missing data
ssN = ssDat[:, 1]
ssN[ssN == -1] = np.NaN
ssDat[:, 1] = ssN

# Interpolate Missing Data
msk = np.isfinite(ssDat[:, 1])
ssDat[:, 1] = np.interp(ssDat[:, 0], ssDat[msk, 0], ssDat[msk, 1])

# Get the Data into the form used above
dt = np.diff(ssDat[:, 0]).mean()
t  = ssDat[:, 0]
x  = ssDat[:, 1]

# Plot the Data
_ = plt.plot(t, x, linewidth=1)
_ = plt.xlabel('Year')
_ = plt.ylabel('Number of Sunspots')
_ = plt.title('Sunspot Data')
```

### Plot the Magnitude of the Fourier Transform

```python
# Get the Fourier Transform
xT = np.fft.rfft(x)

# Get the measured frequencies
f = np.fft.rfftfreq(x.size, dt)

# Get the Transform Magnitudes
xT[1:] *= 2  # Multiply the non-zero frequencies by 2.
magT = np.abs(xT/x.size)  # Get the Magnitude of the scaled transform.

# Plot the
_ = plt.plot(f[:100], magT[:100])
_ = plt.title('Sunspot Spectral Analysis')
_ = plt.ylabel('Magnitude')
_ = plt.xlabel('Frequency (Yr$^{-1}$)')
```

### Plot the Signal Magnitude using Scipy

```python
# Get the Power Spectrum
f, spec = periodogram(x, 1/dt, scaling='spectrum')

# Plot the Magnitudes
_ = plt.loglog(f[1:], np.sqrt(spec*2)[1:])
_ = plt.title('Signal Magnitude')
_ = plt.ylabel('Magnitude')
_ = plt.xlabel('Frequency (Yr$^{-1}$)')
```

In the cell below, insert the fundamental period (the inverse of the frequency with the highest magnitude) for the sunspot oscillations. If you are having a difficult time determining the correct frequency, you may want to plot a smaller window of data.

```python
11
```

    11


# Theoretical Question

## The algorithm ad its mechanism

We write the recursive function to check what kind of operation it implements on the array a.
We can apply this function to an array a = [1,2,3,4,5,6,7,8], that has a power of 2 as length, and see what kinds of
outputs it returns for different specific lenght n, as n=2, n=4, n=8.

```python
def splitSwap( a, l, n ):
    if n <= 1:
        return
    splitSwap(a, l, n//2)
    splitSwap(a, l+ n//2, n//2)
    swapList(a, l, n)

def swapList(a, l, n):
    for i in range(0,n//2):
        tmp = a[l + i]
        a[l + i] = a[l + n//2 + i]
        a[l + n//2 + i] = tmp

a = [1,2,3,4,5,6,7,8]

splitSwap(a, 0, 2)
print(a)

a = [1,2,3,4,5,6,7,8]

splitSwap(a, 0, 4)
print(a)

a = [1,2,3,4,5,6,7,8]

splitSwap(a, 0, 8)
print(a)
```

    [2, 1, 3, 4, 5, 6, 7, 8]
    [4, 3, 2, 1, 5, 6, 7, 8]
    [8, 7, 6, 5, 4, 3, 2, 1]

We can consider only the case when l = 0 as written in the homework text.
As shown in the previous code, the algorithm take an array a, an index l and a specific length n that could be
different from the total length of the array a, len(a): then it returns the first n components of the array a in a
swap order, where the first element of the first n components becomes the last and so on.

We can consider the first n components of the first array, a, as a new array, b.
The algorithm does the following things:

- it splits the array b into 2 arrays, 1 array composed by the first n/2 components and the second by the second
  half of the components; then it continue on the 2 new arrays and split again this 2 arrays, and so on till this
  split mechanism bring us to an array of only 1 component;

- now the algorithm can go back to the second last splitting and swap all the arrays with 2 components in the
  for loop of the swapList() function, then it can join the swapped arrays of 2 components and go back to the third     last splitting with the a new 4 components arrays and so on;

- on the last step it returns to the first splitting and it joins 2 arrays of n/2 components swapped respect to
  the starting 2 arrays, so we finally obtain the reversed b array applying swapList() on this couple of final couple
  of arrays.

Finally the algorithm returns the swapped first n components of the array a and it leaves the others unchanged.

## Big O analysis

We can imagine the described previous procedure like a tree with many layers and $2^i$ branches ( the
numbers of splitted arrays at the step $i$) for the layer $i$.

If we denote the input parameter as $n = n_0$, the algorithm performs $n_0/2$ steps for each layer of the tree and the big O calculus is th following:

\begin{equation}
T(n) = 2T(n/2) + n/2
\end{equation}

\begin{equation}
T(n) = 2( 2T(n/4) + n/4) + n/2 = 4T(n/4) + n/2 + n/2
\end{equation}

\begin{equation}
T(n) = 2^iT(n/2^i) + \sum_{k=1}^i n/2
\end{equation}

The algorithm finish to split the array when it arrives to:

\begin{equation}
n/2^i = 1   -->   i = log_2(n)
\end{equation}

so the total time is:

\begin{equation}
T(n) = nT(1) + {n\over 2}log_2(n).
\end{equation}

Finally, we have the asymptotic behavior for the splitSwap algorithm:

\begin{equation}
T(n) = O( nlog_2(n) ).
\end{equation}

because the linear term is asymptotically smaller than $nlog(n)$, so we can neglect the linear term.

## Is the algorithm optimal?

The proposed algorithm is not optimal because we can write one version with running time $T(n) = O(n)$, that is showen in the following cell.

```python
# The following version works in the case of interest, so with l = 0, and reproduce the same outputs of the
# given version

def splitSwap( a, l, n):

    for i in range( 0, n//2 ):
        tmp = a[i]
        a[i] = a[n-i-1]
        a[n-i-1] = tmp

a = [1,2,3,4,5,6,7,8]

splitSwap(a, 0, 2)
print(a)

a = [1,2,3,4,5,6,7,8]

splitSwap(a, 0, 4)
print(a)

a = [1,2,3,4,5,6,7,8]

splitSwap(a, 0, 8)
print(a)
```

    [2, 1, 3, 4, 5, 6, 7, 8]
    [4, 3, 2, 1, 5, 6, 7, 8]
    [8, 7, 6, 5, 4, 3, 2, 1]

```python

```