# Some basic discrete-time signals
Discrete-time signals are sequences, or functions with integer-valued arguments
$$ f:\, \mathbb{Z} \rightarrow \mathbb{R} $$
$$ f(k) \in \mathbb{k}, \; k \in \mathbb{Z} = \{\ldots, -2,-1,0,1,2,\ldots\}$$
Below are some important examples

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## The impulse function
$$ f(k) = \delta (k) = \begin{cases} 1 & k=0\\ 0 & \text{otherwise} \end{cases}$$

```python
k = np.arange(-4,5)
plt.figure(figsize=(12,1))
plt.stem(k, k==0)
plt.xticks([-4,-2,0,2,4])
plt.xlabel('$k$')
plt.ylabel('$\delta(k)$');
```

## The shifted and scaled impulse function
$$ f(k) = \mathrm{q}a\delta (k) = a\delta(k+1) = \begin{cases} a & k=-1\\ 0 & \text{otherwise} \end{cases}$$
Note the *shift operator* q, whose definition is that it shifts the sequence forward (advances it). The impulse function has the property that it is equal to one if and only if the argument is equal to zero. So in the one-step-ahead shifted sequence $\mathrm{q}\delta(k)$ the value is one when $k+1=0$.

```python
a = 1.4
k = np.arange(-4,5)
plt.figure(figsize=(12,1))
plt.stem(k, a*((k+1)==0))
plt.xticks(k); plt.yticks([0, a],['0', '$a$'])
plt.xlabel('$k$')
plt.ylabel('$\mathrm{q}\delta(k) = \delta(k+1)$');

```

## The unit step function
$$ f(k) = u_s (k) = \begin{cases} 1 & k\ge 0\\ 0 & k < 0 \end{cases}$$

```python
k = np.arange(-4,5)
plt.figure(figsize=(12,1));
plt.stem(k, k>=0)
plt.xticks(k)
plt.xlabel('$k$')
plt.ylabel('$u_s(k)$')
```

## Connection between the impulse and the step function
The impulse funktion is obtained by taking the first difference of the step function
$$ \delta(k) = (1-\text{q}^{-1})u_s(k) = u_s(k)-u_s(k-1). $$
Note the *shift operator* q, whose definition is that it shifts the sequence forward (advaneces it). Consequently the inverse shift operator $\text{q}^{-1}$ shifts the sequence backwards (delays it) one step.

```python
k = np.arange(-4,5)
us = k>=0
us_shift1 = k>=1 # The shifted sequence is one when k-1 >=0 <=> k>= 1
fig,axs = plt.subplots(3,1, sharex=True, figsize=(12,4))
axs[0].stem(k, us); axs[0].set(ylabel  = '$u_s(k)$')
axs[1].stem(k, us_shift1); axs[1].set(ylabel  = '$u_s(k-1)$')
axs[2].stem(k, k==0) ; axs[2].set(ylabel  = '$\delta(k)$');
```

Clearly the bottom sequence is obtained by subtracting the second sequence from the first.

The unit step function can be obtained by summing over the impulse funktion
$$ u_s(k) = \sum_{i=-\infty}^k \delta(i). $$

```python
k = np.arange(-4,5)
dlta = k==0
fig,axs = plt.subplots(2,1, sharex=True, figsize=(12,4))
axs[0].stem(k, dlta); axs[0].set(ylabel  = '$k < 0$')
axs[1].stem(k, dlta); axs[1].set(ylabel  = '$k > 1$')
axs[0].plot([-4,-1.9], [1.01, 1.01], 'r--')
axs[0].plot([-1.9,-1.9], [1.01, .01], 'r--')
axs[0].text(-4,0.5, '$k=-2,\;$ summing only zeros')
axs[1].plot([-4, 2.1], [1.01, 1.01], 'r--')
axs[1].plot([2.1, 2.1], [1.01, .01], 'r--')
axs[1].text(-4,0.5, '$k=2,\;$ summing zeros and a single one');
```

## Real exponential
The real exponential is
$$ f(k) = a^k. $$
Let's look at three cases
### Exampel 1 $a=0.5$

```python
a = 0.5
k = np.arange(-2,5)
plt.figure(figsize=(12,5))
plt.stem(k, np.power(a, k))
plt.xticks(k); plt.yticks(np.power(a,k[:-1]))
plt.xlabel('$k$')
plt.ylabel('$0.5^k$');
```

### Exampel 2 $a = 2$

```python
a = 2.0
k = np.arange(-2,5)
plt.figure(figsize=(12,5))
plt.stem(k, np.power(a, k))
plt.xticks(k); plt.yticks(np.power(a,k[1:]))
plt.xlabel('$k$')
plt.ylabel('$2^k$');
```

### Exampel 3 $a = -0.8$

```python
a = -.8
k = np.arange(-2,5)
plt.figure(figsize=(12,5))
plt.stem(k, np.power(a, k))
plt.xticks(k); plt.yticks(np.power(a,k[:]))
plt.xlabel('$k$')
plt.ylabel('$(-1)^k$');
```

## Sinusoid
$$ f(k) = a\sin(\omega_0 k + \phi), \quad k \in \mathbb{Z},\quad a,\omega_0, \phi \in \mathbb{R} $$

The discrete-time sinusoid has two important differences compared to the continuous-time sinusoid:

### It has infinitely many *aliases* at frequencies that are multiples of $2\pi$
$$\sin\big((\omega_0 + n2\pi)k + \phi\big) = \sin(\omega_0k + \phi), \quad n\in\mathbb{Z}$$
This means that the signal with frequency $\omega_0$ and the signal with frequency $\omega_0 + n2\pi$, where $n$ is an integer, are identical.

```python
a = 1.0; w0 = np.pi/8; phi = np.pi/3; n=3
k = np.arange(-14,15)
fig,axs = plt.subplots(2, 1, sharex=True,figsize=(12,5))
axs[0].stem(k, np.sin(w0*k+phi))
axs[1].stem(k, np.sin((w0+2*np.pi*n)*k+phi))
axs[0].set(yticks=[-1,1])
axs[1].set(yticks=[-1,1])
axs[0].set(ylabel='$\omega_0=\pi/8$')
axs[1].set(ylabel='$\omega_0=\pi/8 + 6\pi$')
plt.xlabel('$k$');
```

In the example above, wee see that we cannot know by just looking at a discrete-time sinusoid, which of the alias frequencies it has ($\omega_0=\pi/8$ and $\omega_0+n2\pi=\pi/8 + 6\pi$ in the example).

Since the discrete-time sinusoid is often obtained by sampling a continuous-time sinusoid, we reach the very important conclusion that **continous-time sinusoids of frequency $\omega_0$ and $\omega_0 + n\omega_s$ have the same sample values**, and so are not distinguishable when sampled. Here $\omega_s$ is the sampling frequency in radians per second. This has profound consequences when dealing with sampled signals and systems.

### The discrete-time sinusoid is not always periodic
This is due to the fact that the argument $\omega_0k + \phi$ to the sinusoid is only taking on discrete values, and not all real values as in the continuous-time case.
The discrete-time sinusoid is periodic when $f(k+N) = f(k)$ for some integer $N$. This requires that
$$\omega_0(k+N) = \omega_0k + 2\pi M, $$
$$ \omega_0k +  \omega_0N = \omega_0k + 2\pi M $$
for some integers $N$ and $M$. This gives
$$\omega_0N = 2\pi M \quad \Leftrightarrow \quad \frac{\omega_0}{2\pi} = \frac{M}{N},$$
which means that the ratio of $\omega_0$ to $2\pi$ must be a rational number. The periodicity will be $N$, and the sequence will repeat after $M$ whole wavelengths.

The above analysis show that we can have discrete-time sinusoids that are periodic where the period length is longer than one wavelength. To obtain a discrete-time sinusoid in which the period equals one wavelength we must require that
$$ \omega_0k + \omega_0N = \omega_0k + 2\pi \quad \Rightarrow \quad \omega_0 = \frac{2\pi}{N}. $$

#### Example 1:  Periodic with periodiciy equal to one wavelength $\omega_0 = \pi/8$, $\phi=\pi/3$

```python
w0 = np.pi/8; phi = np.pi/3
k = np.arange(-14,15)
plt.figure(figsize=(12,5))
plt.stem(k, np.sin(w0*k+phi))
plt.xticks(k); plt.yticks([-1,1])
plt.xlabel('$k$')
plt.ylabel('');
```

#### Example 2: Periodic with periodiciy equal to two wavelengths $\omega_0 = 4\pi/17$, $\phi=\pi/3$
This gives a sequence which has periodicity 17 and repeats after two wavelengths.

```python
N = 17; M = 2
w0 = M*2*np.pi/N; phi = np.pi/3
k = np.arange(-14,15)
plt.figure(figsize=(12,5))
plt.stem(k, np.sin(w0*k+phi))
plt.plot([-6, -6], [0.3, 0.75], 'r:')
plt.plot([-6+N, -6+N], [0.3, 0.75], 'r:')
plt.plot([-6, -6+17], [0.7, 0.7], 'r:')
plt.text(3, 0.8, '%d wavelengths' %M)
plt.xticks(k); plt.yticks([-1,1])
plt.xlabel('$k$');
```

#### Example 3: Aperiodic  $\omega_0 = 1$, $\phi = \pi/3$

```python
w0 = 1
phi = np.pi/3
k = np.arange(-14,15)
plt.figure(figsize=(12,5))
plt.stem(k, np.sin(w0*k+phi))
plt.xticks(k); plt.yticks([-1,1])
plt.xlabel('$k$')
plt.ylabel('');
```

#### Exercise: Construct an example with periodicity of three wavelengths

```python
#Your code here
```

## Discrete complex exponential
$$ f(k) = a^k, \quad a \in \mathbb{C}, \quad k \in \mathbb{Z} $$

Writing the complex number $a$ on polar form gives
$$ f(k) = \big(r\mathrm{e}^{i\omega_0}\big)^k = r^k\mathrm{e}^{ik\omega_0} =  r^k\cos(k\omega_0) + ir^k\sin(k\omega_0), $$
which gives a sequence that looks like a discrete spiral in the complex plane. If the magnitude of $a$ is less than one, $r<1$, then the spiral approached the origin. If the magnitude is greater than one it spirals outwards to infinity. If $r=1$, the complex sequence consists of points on the unit circle, each with the same phase distance to the previous point.

### Connection to sinusoids
Complex exponentials and sinusoids are closely connected through Euler's formula:
\begin{align}
     \sin(\omega_0 t) &= \frac{1}{2i} \big(\mathrm{e}^{i\omega_0 k} - \mathrm{e}^{-i\omega_0 k} \big) = \mathrm{Im} \{ \mathrm{e}^{i\omega_0 k} \}\\
     \cos(\omega_0 k) &= \frac{1}{2} \big(\mathrm{e}^{i\omega_0 k} + \mathrm{e}^{-i\omega_0 k} \big) = \mathrm{Re} \{ \mathrm{e}^{i\omega_0 k} \}
\end{align}
So, as for the case of sinusoids, we see that discrete-time complex exponentials will be periodic with period of $2\pi$ if and only if $\omega_0 can be written
$$ \omega_0 = \frac{2\pi}{N}, \quad N \in \mathbb{Z}. $$

More importantly, all discrete-time complex exponentials $f(k) = \mathrm{e}^{i\omega k}$ with
$$ \omega = \omega_0 + n2\pi, \quad n \in \mathbb{Z} $$
are indistinguishable, or aliases, since
$$ f(k) = \mathrm{e}^{i\omega k} = \mathrm{e}^{ik(\omega_0 + n2\pi)} = \mathrm{e}^{ik\omega_0 + ink2\pi} = \big(\mathrm{e}^{i2\pi}\big)^{(nk)} \mathrm{e}^{i\omega_0 k} = 1^{(nk)}\mathrm{e}^{i\omega_0 k} = \mathrm{e}^{i\omega_0 k}. $$
However, also $-\omega_0 + n2\pi$ are alias frequencies to $\omega_0$, since
\begin{align}
\sin(\omega_0 k) &= -\sin(-\omega_0k) = -\sin\big(-\omega_0 + 2n\pi)k\big) = - \frac{1}{2i} \big( \mathrm{e}^{i(-\omega_0+ n2\pi)k} -\mathrm{e}^{-i(-\omega_0 + 2\pi) k} \big).
 \end{align}

```python
w0 = np.pi/6
k = np.arange(-2,10)
plt.figure(figsize=(8,8))
for r in [0.8, 1.0, 1.2]:
    a = r*np.complex(np.cos(w0), np.sin(w0))
    f = np.power(a, k)
    plt.polar(np.angle(f), np.abs(f), '.', markersize=14, label='r=%0.2f' %r)
plt.legend(loc='right',fontsize='large', bbox_to_anchor=(1.1,1));
```

### Exercise
Plot the function
$$ f(k) = \mathrm{Im} \{ \big(1.2 \mathrm{e}^{i\frac{\pi}{12}}\big)^k\}$$

```python
w0 = np.pi/12; r=1.2
#a = # your code here
f = np.power(a, k)
k = np.arange(-2,20)
plt.figure(figsize=(12,4))
#plt.stem(k, ) # Your code here
```


```julia
using Symbolics
```

```julia
@variables t x y
```

\begin{equation}
\left[
\begin{array}{c}
t \\
x \\
y \\
\end{array}
\right]
\end{equation}

```julia
D = Differential(t)
```

    (::Differential) (generic function with 2 methods)

```julia
z = t + t^2
```

\begin{equation}
t + t^{2}
\end{equation}

```julia
D(z)
```

\begin{equation}
\mathrm{\frac{d}{d t}}\left( t + t^{2} \right)
\end{equation}

```julia
expand_derivatives(D(z))
```

\begin{equation}
1 + 2 t
\end{equation}

```julia
Symbolics.jacobian([x + x*y, x^2 + y], [x, y])
```

\begin{equation}
\left[
\begin{array}{cc}
1 + y & x \\
2 x & 1 \\
\end{array}
\right]
\end{equation}

```julia
B = simplify.([t^2 + t + t^2 2t + 4t; x + y + y + 2t x^2 - x^2 + y^2])
```

\begin{equation}
\left[
\begin{array}{cc}
t + 2 t^{2} & 6 t \\
x + 2 t + 2 y & y^{2} \\
\end{array}
\right]
\end{equation}

```julia
simplify.(substitute.(B, (Dict(x=>y^2),)))
```

\begin{equation}
\left[
\begin{array}{cc}
t + 2 t^{2} & 6 t \\
2 t + y^{2} + 2 y & y^{2} \\
\end{array}
\right]
\end{equation}

```julia
substitute.(B, (Dict(x => 2.0, y=>3.0, t=>4.0),))
```

\begin{equation}
\left[
\begin{array}{cc}
36.0 & 24.0 \\
16.0 & 9.0 \\
\end{array}
\right]
\end{equation}

# SymbolivUtils

```julia
import Pkg
Pkg.add("SymbolicUtils")
```

        Updating registry at `~/.julia/registries/General`
        Updating git-repo `https://github.com/JuliaRegistries/General.git`
       Resolving package versions...
       Installed SymbolicUtils ─ v0.18.2
        Updating `~/.julia/environments/v1.7/Project.toml`
      [d1185830] + SymbolicUtils v0.18.2
        Updating `~/.julia/environments/v1.7/Manifest.toml`
      [d1185830] ↑ SymbolicUtils v0.18.1 ⇒ v0.18.2
    Precompiling project...
      ✓ SymbolicUtils
      ✓ Symbolics
      2 dependencies successfully precompiled in 30 seconds (192 already precompiled)
      2 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions

```julia
using SymbolicUtils
```

```julia
SymbolicUtils.show_simplified[] = true
```

    true

```julia
@syms x::Real y::Real z::Complex f(::Number)::Real
```

    (x, y, z, f)

```julia
2x^2 - y + x^2
```

\begin{equation}
 - y + 3 x^{2}
\end{equation}

```julia
f(sin(x)^2 + cos(x)^2) + z
```

\begin{equation}
z + f\left( \cos^{2}\left( x \right) + \sin^{2}\left( x \right) \right)
\end{equation}

```julia
r = @rule sinh(im * ~x) => sin(~x)
```

    sinh(im * ~x) => sin(~x)

```julia
r(sinh(im*y))
```

\begin{equation}
\sin\left( y \right)
\end{equation}

```julia
simplify(cos(y)^2 + sinh(im*y)
```


# Maximum Likelihood and MAP continued...

## MLE of Mean of Gaussian

*  Let $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_N$ be samples from a multi-variance Normal distribution with known covariance matrix and an unknown mean.  Given this data, obtain the ML estimate of the mean vector.
	\begin{equation}
	p(\mathbf{x}_k| {{\mu}}) = \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_k - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_k - {\mu})\right)
	\end{equation}
* We can define our likelihood given the $N$ data points.  We are assuming these data points are drawn independently but from an identical distribution (i.i.d.):
	\begin{equation}
	\prod_{n=1}^N p(\mathbf{x}_n| {{\mu}}) = \prod_{n=1}^N \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)
	\end{equation}
*  We can apply our "trick" to simplify
	\begin{eqnarray}
	\mathscr{L} &=& \ln \prod_{n=1}^N p(\mathbf{x}_n| {{\mu}}) = \ln \prod_{n=1}^N \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)\\
	&=& \sum_{n=1}^N  \ln \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)\\
	&=& \sum_{n=1}^N  \left( \ln \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}} + \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right) \right) \\
	&=&  - N \ln (2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}} + \sum_{n=1}^N  \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu}) \right)
	\end{eqnarray}
* Now, lets maximize:
	\begin{eqnarray}
	\frac{\partial \mathscr{L}}{\partial \mu} &=& \frac{\partial}{\partial \mu} \left[- N \ln (2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}} + \sum_{n=1}^N  \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu}) \right)\right] = 0 \\
	&\rightarrow& \sum_{n=1}^N  \Sigma^{-1}(\mathbf{x}_n - {\mu}) = 0\\
	&\rightarrow& \sum_{n=1}^N  \Sigma^{-1}\mathbf{x}_n  = \sum_{n=1}^N  \Sigma^{-1} {\mu}\\
	&\rightarrow& \Sigma^{-1} \sum_{n=1}^N \mathbf{x}_n  = \Sigma^{-1} {\mu} N\\
	&\rightarrow& \sum_{n=1}^N \mathbf{x}_n  = {\mu} N\\
	&\rightarrow& \frac{\sum_{n=1}^N \mathbf{x}_n}{N} = {\mu}\\
	\end{eqnarray}
* So, the ML estimate of $\mu$ is the sample mean!

## MAP of Mean of Gaussian

* To get a MAP estimate of the mean of a Gaussian, we apply a prior distribution and maximize the posterior.
* Lets use a Gaussian prior on the mean (because it has a *conjugate prior* relationship)

\begin{eqnarray}
p(\mu|X, \mu_0, \sigma_0^2, \sigma^2) &\propto& \mathscr{N}(X|\mu, \sigma^2)\mathscr{N}(\mu|\mu_0, \sigma_0^2)\\
&=& \prod_{n=1}^N \left(\frac{1}{\sqrt{2\pi \sigma^2}} \exp\left\{-\frac{1}{2\sigma^2}\left(x_n-\mu\right)^2 \right\}\right)\frac{1}{\sqrt{2\pi \sigma_0^2}} \exp\left\{-\frac{1}{2\sigma_0^2}\left(\mu-\mu_0\right)^2 \right\}\nonumber\\
\mathscr{L} &=& -\frac{N}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{n=1}^N(x_n-\mu)^2 - \frac{1}{2}\ln(2\pi \sigma_0^2) - \frac{1}{2\sigma_0^2}(\mu - \mu_0)^2\\
\frac{\partial \mathscr{L}}{\partial \mu} &=& \frac{1}{\sigma^2}\sum_{n=1}^N x_n - \frac{N}{\sigma^2}\mu - \frac{1}{\sigma_0^2}\mu + \frac{1}{\sigma_0^2}\mu_0 = 0\\
\mu\left(\frac{N\sigma_0^2 + \sigma^2}{\sigma^2\sigma_0^2} \right) &=& \frac{1}{\sigma^2}\sum_{n=1}^N x_n + \frac{1}{\sigma_0^2}\mu_0 \\
\mu_{MAP} &=& \frac{\sigma_0^2}{N\sigma_0^2 + \sigma^2}\sum_{n=1}^N x_n + \frac{\mu_0\sigma^2}{N\sigma_0^2 + \sigma^2}
\end{eqnarray}

* *Does this result make sense?*

```python

```


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function
from __future__ import absolute_import

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Numerical Differentiation

**GOAL:**  Given a set of $N+1$ points $(x_i, y_i)$ compute the derivative of a given order to a specified accuracy.

**Approaches:**
 * Find the interpolating polynomial $P_N(x)$ and differentiate that.
 * Use Taylor-series expansions and the method of undetermined coefficients to derive finite-difference weights and their error estimates

**Issues:**  Order vs accuracy...how to choose

# Example 1:  how to approximate the derivative $f'(x)$ given a discrete sampling of a function $f(x)$

Here we will consider how to estimate $f'(x_k)$ given a $N$ point sampling of $f(x)=\sin(\pi x) + 1/2 \sin(2\pi x)$ sampled uniformly over the interval $x\in [ 0,1]$

```python
N = 11
x = numpy.linspace(0,1,N)
xfine = numpy.linspace(0,1,101)
f = lambda x:  numpy.sin(numpy.pi*x) + 0.5*numpy.sin(4*numpy.pi*x)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(xfine, f(xfine),'b',label='$f(x)$')
axes.plot(x, f(x), 'ro', markersize=12, label='$f(x_k)$')
axes.grid()
axes.set_xlabel('x')
p = numpy.polyfit(x,f(x),N-1)
axes.plot(xfine,numpy.polyval(p,xfine),'g--',label='$P_{{{N}}}$'.format(N=N-1))
axes.legend(fontsize=15)
plt.show()
```

### Example 2:  how to approximate derivative $f'(x)$ given a discrete sampling of a function $f(x)$

Here we will consider how to estimate $f'(x_k)$ given a $N$ point sampling of Runge's function sampled uniformly over the interval $x\in [ -1,1]$

```python
N = 11
x = numpy.linspace(-1,1,N)
xfine = numpy.linspace(-1,1,101)
f = lambda x:  1./(1. + 25*x**2)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(xfine, f(xfine),'b',label='$f(x)$')
axes.plot(x, f(x), 'ro', markersize=12, label='$f(x_k)$')
axes.grid()
axes.set_xlabel('x')
p = numpy.polyfit(x,f(x),N-1)
axes.plot(xfine,numpy.polyval(p,xfine),'g--',label='$P_{{{N}}}$'.format(N=N-1))
axes.legend(fontsize=15)
plt.show()
```

### The interpolating polynomial: review

From our previous lecture, we showed that we can approximate a function $f(x)$ over some interval in terms of a unique interpolating polynomial through $N+1$ points and a  remainder term

$$
    f(x) = P_N(x) + R_N(x)
$$

Where the Lagrange remainder term is

$$R_N(x) = (x - x_0)(x - x_1)\cdots (x - x_{N})(x - x_{N+1}) \frac{f^{(N+1)}(c)}{(N+1)!}$$

While there are multiple ways to represent the interpolating polynomial, both $P_N(x)$ and $R_N(x)$ are polynomials in $x$ and therefore differentiable.  Thus we should be able to calculate the first derivative and its error as

$$
    f'(x) = P'_N(x) + R'_N(x)
$$

and likewise for higher order derivatives up to degree $N$.

### Derivatives of the Lagrange Polynomials

The Lagrange basis, is a particularly nice basis for calculating numerical differentiation formulas because of their basic interpolating property that

$$
    P_N(x) = \sum_{i=0}^N f(x_i)\ell_i(x)
$$

where $f(x_i)$ is just the value of our function $f$ at node $x_i$ and all of the $x$ dependence is contained in the Lagrange Polynomials $\ell_i(x)$ (which only depend on the node coordinates $x_i$, $i=0,\ldots,N$).  Thus, the interpolating polynomial at any $x$ is simply a linear combination of the values at the nodes $f(x_i)$

Likewise its first derivative
$$
P'_N(x)  = \sum_{i=0}^N f(x_i)\ell'_i(x)
$$
is also just a linear combination of the values $f(x_i)$

## Examples

Given the potentially, highly oscillatory nature of the interpolating polynomial, in practice we only use a small number of data points around a given point $x_k$ to derive a differentiation formula for the derivative $f'(x_k)$.  In the context of differential equations we also often have $f(x)$ so that $f(x_k) = y_k$ and we can approximate the derivative of a known function $f(x)$.

```python
N = 9
f = lambda x:  1./(1. + 25*x**2)
#f = lambda x: numpy.cos(2.*numpy.pi*x)
```

```python
x = numpy.linspace(-1,1,N)
xfine = numpy.linspace(-1,1,101)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(xfine, f(xfine),'b',label='$f(x)$')
axes.plot(x, f(x), 'ro', markersize=12, label='$f(x_k)$')
x3 = x[5:8]
x3fine = numpy.linspace(x3[0],x3[-1],20)
p = numpy.polyfit(x3,f(x3),2)
axes.plot(x3,f(x3),'m',label = 'Piecewise $P_1(x)$')
axes.plot(x3fine,numpy.polyval(p,x3fine),'k',label = 'Piecewise $P_2(x)$')
axes.grid()
axes.set_xlabel('x')
p = numpy.polyfit(x,f(x),N-1)
axes.plot(xfine,numpy.polyval(p,xfine),'g--',label='$P_{{{N}}}$'.format(N=N-1))
axes.legend(fontsize=14,loc='best')
plt.show()
```

### Example: 1st order polynomial through 2 points $x=x_0, x_1$:

$$
    P_1(x)=f_0\ell_0(x) + f_1\ell_1(x)
$$

Or written out in full

$$
P_1(x) = f_0\frac{x-x_1}{x_0-x_1} + f_1\frac{x-x_0}{x_1-x_0}
$$

Thus the first derivative of this polynomial for all $x\in[x_0,x_1]$ is

$$
P'_1(x) = \frac{f_0}{x_0-x_1} + \frac{f_1}{x_1-x_0} = \frac{f_1 - f_0}{x_1 - x_0} = \frac{f_1 - f_0}{\Delta x}
$$

Where $\Delta x$ is the width of the interval.  This formula is simply the slope of the chord connecting the points $(x_0, f_0)$ and $(x_1,f_1)$.   Note also, that the estimate of the first-derivative is constant for all $x\in[x_0,x_1]$.

#### "Forward" and "Backward" first derivatives

Even though the first derivative by this method is the same at both $x_0$ and $x_1$, we sometime make a distinction between the "forward Derivative"

$$f'(x_n) \approx D_1^+ = \frac{f(x_{n+1}) - f(x_n)}{\Delta x}$$

and the "backward" finite-difference as

$$f'(x_n) \approx D_1^- = \frac{f(x_n) - f(x_{n-1})}{\Delta x}$$

Note these approximations should be familiar to use as the limit as $\Delta x \rightarrow 0$ these are no longer approximations but equivalent definitions of the derivative at $x_n$.

### Example: 2nd order polynomial through 3 points $x=x_0, x_1, x_2$:

$$
    P_2(x)=f_0\ell_0(x) + f_1\ell_1(x) + f_2\ell_2(x)
$$

Or written out in full

$$
P_2(x) = f_0\frac{(x-x_1)(x-x_2)}{(x_0-x_1)(x_0-x_2)} + f_1\frac{(x-x_0)(x-x_2)}{(x_1-x_0)(x_1-x_2)} + f_2\frac{(x-x_0)(x-x_1)}{(x_2-x_0)(x_2-x_1)}
$$

Thus the first derivative of this polynomial for all $x\in[x_0,x_2]$ is

$$
P'_2(x) = f_0\frac{(x-x_1)+(x-x_2)}{(x_0-x_1)(x_0-x_2)} + f_1\frac{(x-x_0)+(x-x_2)}{(x_1-x_0)(x_1-x_2)} + f_2\frac{(x-x_0)+(x-x_1)}{(x_2-x_0)(x_2-x_1)}
$$

**Exercise**: show that the second-derivative $P''_2(x)$ is a constant (find it!) but is also just a linear combination of the function values at the nodes.

### Special case of equally spaced nodes $x = [-h, 0, h]$ where $h=\Delta x$ is the grid spacing

General Case:
$$
P'_2(x) = f_0\frac{(x-x_1)+(x-x_2)}{(x_0-x_1)(x_0-x_2)} + f_1\frac{(x-x_0)+(x-x_2)}{(x_1-x_0)(x_1-x_2)} + f_2\frac{(x-x_0)+(x-x_1)}{(x_2-x_0)(x_2-x_1)}
$$

Becomes:
$$
P'_2(x) = f_0\frac{2x-h}{2h^2} + f_1\frac{-2x}{h^2} + f_2\frac{2x+h}{2h^2}
$$

which if we evaluate at the three nodes  $-h,0,h$ yields

$$
P'_2(-h) = \frac{-3f_0 + 4f_1 -1f_2}{2h}, \quad\quad P'_2(0) = \frac{-f_0 + f_2}{2h}, \quad\quad P'_2(h) = \frac{f_0 -4f_1 + 3f_2}{2h}
$$

Again, just  linear combinations of the values at the nodes $f(x_i)$

#### Quick Checks

In general,  all finite difference formulas can be written as linear combinations of the values of $f(x)$ at the nodes.  The formula's can be hard to remember, but they are easy to check.

* The sum of the coefficients must add to zero.  Why?
* The sign of the coefficients can be checked by inserting $f(x_i) = x_i$

##### Example

Given
$$
P'_2(-h) =\frac{-3f_0 + 4f_1 -1f_2}{2h}
$$

What is $P'_2(-h)$ if

* $$f_0=f_1=f_2$$
* $$f_0 = 0, ~f_1 = 1, ~f_2 = 2$$

### Error Analysis

In addition to calculating finite difference formulas, we can also estimate the error

From Lagrange's Theorem,  the remainder term looks like

$$R_N(x) = (x - x_0)(x - x_1)\cdots (x - x_{N})(x - x_{N+1}) \frac{f^{(N+1)}(c)}{(N+1)!}$$

Thus the derivative of the remainder term $R_N(x)$ is

$$R_N'(x) = \left(\sum^{N}_{i=0} \left( \prod^{N}_{j=0,~j\neq i} (x - x_j) \right )\right ) \frac{f^{(N+1)}(c)}{(N+1)!}$$

The remainder term contains a sum of $N$'th order polynomials and can be awkward to evaluate, however, if we restrict ourselves to the error at any given node $x_k$, the remainder simplifies to

$$R_N'(x_k) = \left( \prod^{N}_{j=0,~j\neq k} (x_k - x_j) \right) \frac{f^{(N+1)}(c)}{(N+1)!}$$

If we let $\Delta x = \max_i |x_k - x_i|$ we then know that the remainder term will be $\mathcal{O}(\Delta x^N)$ as $\Delta x \rightarrow 0$ thus showing that this approach converges and we can find arbitrarily high order approximations (ignoring floating point error).

### Examples

#### First order differences $N=1$

For our first order finite differences, the error term is simply

$$R_1'(x_0) = -\Delta x \frac{f''(c)}{2}$$
$$R_1'(x_1) = \Delta x \frac{f''(c)}{2}$$

Both of which are $O(\Delta x f'')$

#### Second order differences $N=2$

For general second order polynomial interpolation, the derivative of the remainder term is

$$\begin{aligned}
    R_2'(x) &= \left(\sum^{2}_{i=0} \left( \prod^{2}_{j=0,~j\neq i} (x - x_j) \right )\right ) \frac{f'''(c)}{3!} \\
    &= \left ( (x - x_{i+1}) (x - x_{i-1}) + (x-x_i) (x-x_{i-1}) + (x-x_i)(x-x_{i+1}) \right ) \frac{f'''(c)}{3!}
\end{aligned}$$

Again evaluating this expression at the center point $x = x_i$ and assuming evenly space points we have

$$R_2'(x_i) = -\Delta x^2 \frac{f'''(c)}{3!}$$

showing that our error is $\mathcal{O}(\Delta x^2)$.

### Caution

High order does not necessarily imply high-accuracy!

As always, the question remains as to whether the underlying function is well approximated by a high-order polynomial.

### Convergence

Nevertheless, we can always check to see if the error reduces as expected as $\Delta x\rightarrow 0$.  Here we estimate the 1st and 2nd order first-derivative for evenly spaced points

```python
def D1_p(func, x_min, x_max, N):
    """ calculate consistent 1st order Forward difference of a function func(x) defined on the interval [x_min,xmax]
    and sampled at N evenly spaced points"""

    x = numpy.linspace(x_min, x_max, N)
    f = func(x)
    dx = x[1] - x[0]
    f_prime = numpy.zeros(N)
    f_prime[0:-1] = (f[1:] - f[0:-1])/dx
    # and patch up the end point with a backwards difference
    f_prime[-1] = f_prime[-2]

    return f_prime

def D1_2(func, x_min, x_max, N):
    """ calculate consistent 2nd order first derivative of a function func(x) defined on the interval [x_min,xmax]
    and sampled at N evenly spaced points"""

    x = numpy.linspace(x_min, x_max, N)
    f = func(x)
    dx = x[1] - x[0]
    f_prime = numpy.zeros(N)
    f_prime[0] = f[:3].dot(numpy.array([-3, 4, -1]))/(2*dx)
    f_prime[1:-1] = (f[2:N] - f[0:-2])/(2*dx)
    f_prime[-1] = f[-3:].dot(numpy.array([1, -4, 3]))/(2*dx)

    return f_prime
```

#### Note:

This first derivative operator can also be written as a Matrix $D$ such that $f'(\mathbf{x}) = Df(\mathbf{x})$ where $\mathbf{x}$ is a vector of $x$ coordinates. (exercise left for the homework)

```python
N = 81
xmin = 0.
xmax = 1.
func = lambda x:  numpy.sin(numpy.pi*x) + 0.5*numpy.sin(4*numpy.pi*x)
func_prime = lambda x: numpy.pi*numpy.cos(numpy.pi*x) + 2.*numpy.pi * numpy.cos(4*numpy.pi*x)
D1f = D1_p(func, xmin, xmax, N)
D2f = D1_2(func, xmin, xmax, N)
```

```python
xa = numpy.linspace(xmin, xmax, 100)
xi = numpy.linspace(xmin, xmax, N)
fig = plt.figure(figsize=(16, 6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D1f, 'ko',label='$D^+_1(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()

axes = fig.add_subplot(1, 2, 2)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D2f, 'go',label='$D_1^2(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()
plt.show()
```

```python
N = 81
xmin = -1
xmax = 1.
func = lambda x:  1./(1 + 25.*x**2)
func_prime = lambda x: -50. * x / (1. + 25.*x**2)**2
D1f = D1_p(func, xmin, xmax, N)
D2f = D1_2(func, xmin, xmax, N)
```

```python
xa = numpy.linspace(xmin, xmax, 100)
xi = numpy.linspace(xmin, xmax, N)
fig = plt.figure(figsize=(16, 6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D1f, 'ko',label='$D^+_1(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()

axes = fig.add_subplot(1, 2, 2)
axes.plot(xa, func(xa), 'b', label="$f(x)$")
axes.plot(xa, func_prime(xa), 'k--', label="$f'(x)$")
axes.plot(xi, func(xi), 'ro')
axes.plot(xi, D2f, 'go',label='$D_1^2(f)$')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()
plt.show()
```

#### Computing Order of Convergence

Say we had the error $E(\Delta x)$ and we wanted to make a statement about the rate of convergence (note we can replace $E$ here with the $R$ from above).  Then we can do the following:
$$\begin{aligned}
    E(\Delta x) &= C \Delta x^n \\
    \log E(\Delta x) &= \log C + n \log \Delta x
\end{aligned}$$

The slope of the line is $n$ when modeling the error like this!  We can also match the first point by solving for $C$:

$$
    C = e^{\log E(\Delta x) - n \log \Delta x}
$$

```python
# Compute the error as a function of delta_x
N_range = numpy.logspace(1, 4, 10, dtype=int)
delta_x = numpy.empty(N_range.shape)
error = numpy.empty((N_range.shape[0], 4))
for (i, N) in enumerate(N_range):
    x_hat = numpy.linspace(xmin, xmax, N)
    delta_x[i] = x_hat[1] - x_hat[0]

    # Compute forward difference
    D1f = D1_p(func, xmin, xmax, N)

    # Compute 2nd order difference
    D2f = D1_2(func, xmin, xmax, N)

    # Calculate the infinity norm or maximum error
    error[i, 0] = numpy.linalg.norm(numpy.abs(func_prime(x_hat) - D1f), ord=numpy.inf)
    error[i, 1] = numpy.linalg.norm(numpy.abs(func_prime(x_hat) - D2f), ord=numpy.inf)

error = numpy.array(error)
delta_x = numpy.array(delta_x)

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.loglog(delta_x, error[:,0], 'ro', label='$D_1^+$')
axes.loglog(delta_x, error[:,1], 'bo', label='$D_1^2$')
axes.loglog(delta_x, order_C(delta_x[0], error[0, 0], 1.0) * delta_x**1.0, 'r--', label="1st Order")
axes.loglog(delta_x, order_C(delta_x[0], error[0, 1], 2.0) * delta_x**2.0, 'b--', label="2nd Order")
axes.legend(loc=4)
axes.set_title("Convergence of Finite Differences", fontsize=18)
axes.set_xlabel("$\Delta x$", fontsize=16)
axes.set_ylabel("$|f'(x) - \hat{f}'(x)|$", fontsize=16)
axes.legend(loc='best', fontsize=14)
axes.grid()

plt.show()
```

# Another approach:  The method of undetermined Coefficients

An alternative method for finding finite-difference formulas is by using Taylor series expansions about the point we want to approximate.  The Taylor series about $x_n$ is

$$f(x) = f(x_n) + (x - x_n) f'(x_n) + \frac{(x - x_n)^2}{2!} f''(x_n) + \frac{(x - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x - x_n)^4)$$

Say we want to derive the second order accurate, first derivative approximation that we just did, this requires the values $(x_{n+1}, f(x_{n+1})$ and $(x_{n-1}, f(x_{n-1})$.  We can express these values via our Taylor series approximation above as

\begin{aligned}
    f(x_{n+1}) &= f(x_n) + (x_{n+1} - x_n) f'(x_n) + \frac{(x_{n+1} - x_n)^2}{2!} f''(x_n) + \frac{(x_{n+1} - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x_{n+1} - x_n)^4) \\
\end{aligned}

or
\begin{aligned}
&= f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{aligned}

and

\begin{align}
f(x_{n-1}) &= f(x_n) + (x_{n-1} - x_n) f'(x_n) + \frac{(x_{n-1} - x_n)^2}{2!} f''(x_n) + \frac{(x_{n-1} - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x_{n-1} - x_n)^4)
\end{align}

\begin{align}
&= f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{align}

Or all together (for regularly spaced points),
\begin{align}
f(x_{n+1}) &= f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\\
f(x_n) &= f(x_n) \\
f(x_{n-1})&= f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{align}

Now to find out how to combine these into an expression for the derivative we assume our approximation looks like

$$
    f'(x_n) + R(x_n) = A f(x_{n+1}) + B f(x_n) + C f(x_{n-1})
$$

where $R(x_n)$ is our error.

Plugging in the Taylor series approximations we find

$$\begin{aligned}
    f'(x_n) + R(x_n) &= A \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right ) \\
    & + B  ~~~~f(x_n)  \\
    & + C \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4) \right )
\end{aligned}$$

Or
$$
f'(x_n) + R(x_n)= (A + B + C) f(x_n) + (A\Delta x +0B - C\Delta x)f'(x_n) + (A\frac{\Delta x^2}{2!} + C\frac{\Delta x^2}{2!})f''(x_n) + O(\Delta x^3)
$$

Since we want $R(x_n) = \mathcal{O}(\Delta x^2)$ we want all terms lower than this to cancel except for those multiplying $f'(x_n)$ as those should sum to 1 to give us our approximation.  Collecting the terms with common evaluations of the derivatives on $f(x_n)$ we get a series of expressions for the coefficients $A$, $B$, and $C$ based on the fact we want an approximation to $f'(x_n)$.  The $n=0$ terms collected are $A + B + C$ and are set to 0 as we want the $f(x_n)$ term to also cancel.

$$\begin{aligned}
    f(x_n):&  &A + B + C &= 0 \\
    f'(x_n): & &A \Delta x - C \Delta x &= 1 \\
    f''(x_n): & &A \frac{\Delta x^2}{2} + C \frac{\Delta x^2}{2} &= 0
\end{aligned} $$

Or as a linear algebra problem

$$\begin{bmatrix}
1 & 1 & 1 \\
\Delta x & 0 &-\Delta x \\
\frac{\Delta x^2}{2} & 0 & \frac{\Delta x^2}{2} \\
\end{bmatrix}
\begin{bmatrix} A \\ B\\ C\\\end{bmatrix} =
\begin{bmatrix} 0 \\ 1\\ 0\\\end{bmatrix}
$$

This last equation $\Rightarrow A = -C$, using this in the second equation gives $A = \frac{1}{2 \Delta x}$ and $C = -\frac{1}{2 \Delta x}$.  The first equation then leads to $B = 0$.

Putting this altogether then gives us our previous expression including an estimate for the error:

$$\begin{aligned}
    f'(x_n) + R(x_n) &= \quad \frac{1}{2 \Delta x} \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right ) \\
    & \quad + 0 \cdot f(x_n) \\
    & \quad - \frac{1}{2 \Delta x} \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4) \right ) \\
    &=  f'(x_n) + \frac{1}{2 \Delta x} \left ( \frac{2 \Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right )
\end{aligned}$$
so that we find
$$
    R(x_n) = \frac{\Delta x^2}{3!} f'''(x_n) + \mathcal{O}(\Delta x^3) = \mathcal{O}(\Delta x^2)
$$

#### Another way...

There is one more way to derive the second order accurate, first order finite-difference formula.  Consider the two first order forward and backward finite-differences averaged together:

$$\frac{D_1^+(f(x_n)) + D_1^-(f(x_n))}{2} = \frac{f(x_{n+1}) - f(x_n) + f(x_n) - f(x_{n-1})}{2 \Delta x} = \frac{f(x_{n+1}) - f(x_{n-1})}{2 \Delta x}$$

### Example 4: Higher Order Derivatives

Using our Taylor series approach lets derive the second order accurate second derivative formula.  Again we will use the same points and the Taylor series centered at $x = x_n$ so we end up with the same expression as before:

$$\begin{aligned}
    f''(x_n) + R(x_n) &= \quad A \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5)\right ) \\
    &+ \quad B \cdot f(x_n) \\
    &+ \quad C \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right )
\end{aligned}$$

except this time we want to leave $f''(x_n)$ on the right hand side.

Try out the same trick as before and see if you can setup the equations that need to be solved.

Doing the same trick as before we have the following expressions:

$$\begin{aligned}
    f(x_n): & & A + B + C &= 0\\
    f'(x_n): & & A \Delta x - C \Delta x &= 0\\
    f''(x_n): & & A \frac{\Delta x^2}{2} + C \frac{\Delta x^2}{2} &= 1
\end{aligned}$$

Or again

$$\begin{bmatrix}
1 & 1 & 1 \\
\Delta x & 0 &-\Delta x \\
\frac{\Delta x^2}{2} & 0 & \frac{\Delta x^2}{2} \\
\end{bmatrix}
\begin{bmatrix} A \\ B\\ C\\\end{bmatrix} =
\begin{bmatrix} 0 \\ 0\\ 1\\\end{bmatrix}
$$

Note,  the Matrix remains, the same, only the right hand side has changed

The second equation implies $A = C$ which combined with the third implies

$$A = C = \frac{1}{\Delta x^2}$$

Finally the first equation gives

$$B = -\frac{2}{\Delta x^2}$$

leading to the final expression

$$\begin{aligned}
    f''(x_n) + R(x_n) &= \quad \frac{1}{\Delta x^2} \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5)\right ) \\
    &+ \quad -\frac{2}{\Delta x^2} \cdot f(x_n) \\
    &+ \quad \frac{1}{\Delta x^2} \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right ) \\
    &= f''(x_n) + \frac{1}{\Delta x^2} \left(\frac{2 \Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right )
\end{aligned}
$$
so that

$$
    R(x_n) = \frac{\Delta x^2}{12} f^{(4)}(x_n) + \mathcal{O}(\Delta x^3)
$$

```python
def D2(func, x_min, x_max, N):
    """ calculate consistent 2nd order second derivative of a function func(x) defined on the interval [x_min,xmax]
    and sampled at N evenly spaced points"""

    x = numpy.linspace(x_min, x_max, N)
    f = func(x)
    dx = x[1] - x[0]
    D2f = numpy.zeros(x.shape)
    D2f[1:-1] = (f[:-2] - 2*f[1:-1] + f[2:])/(dx**2)
    # patch up end points to be 1 sided 2nd derivatives
    D2f[0] = D2f[1]
    D2f[-1] = D2f[-2]

    return D2f
```

```python
f = lambda x: numpy.sin(x)
f_dubl_prime = lambda x: -numpy.sin(x)

# Use uniform discretization
x = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, 1000)
N = 80
x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N)
delta_x = x_hat[1] - x_hat[0]

# Compute derivative
D2f  = D2(f, x_hat[0], x_hat[-1], N)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x,f(x),'b',label='$f(x)$')
axes.plot(x, f_dubl_prime(x), 'k--', label="$f'(x)$")
axes.plot(x_hat, D2f, 'ro', label='$D_2(f)$')
axes.set_xlim((x[0], x[-1]))
axes.set_ylim((-1.1, 1.1))
axes.legend(loc='best',fontsize=14)
axes.grid()
axes.set_title('Discrete Second derivative',fontsize=18)
axes.set_xlabel('$x$', fontsize=16)

plt.show()
```

### The general case

In the general case we can use any $N+1$ points to calculated consistent finite difference coefficients for approximating any derivative of order $k \leq N$.  Relaxing the requirement of equal grid spacing (or the expectation that the location where the derivative is evaluated  $\bar{x}$, is one of the grid points) the Taylor series expansions become

$$\begin{aligned}
    f^{(k)}(\bar{x}) + R(\bar{x}) &= \quad c_0 \left ( f(\bar{x}) + \Delta x_0 f'(\bar{x}) + \frac{\Delta x_0^2}{2!} f''(\bar{x}) + \frac{\Delta x_0^3}{3!} f'''(\bar{x}) + \frac{\Delta x_0^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_0^5)\right ) \\
    &+ \quad c_1 \left ( f(\bar{x}) + \Delta x_1 f'(\bar{x}) + \frac{\Delta x_1^2}{2!} f''(\bar{x}) + \frac{\Delta x_1^3}{3!} f'''(\bar{x}) + \frac{\Delta x_1^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_1^5)\right )\\
    &+ \quad c_2 \left ( f(\bar{x}) + \Delta x_2 f'(\bar{x}) + \frac{\Delta x_2^2}{2!} f''(\bar{x}) + \frac{\Delta x_2^3}{3!} f'''(\bar{x}) + \frac{\Delta x_2^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_2^5)\right ) \\
    &+ \quad \vdots\\
    &+ \quad c_N \left ( f(\bar{x}) + \Delta x_N f'(\bar{x}) + \frac{\Delta x_N^2}{2!} f''(\bar{x}) + \frac{\Delta x_N^3}{3!} f'''(\bar{x}) + \frac{\Delta x_N^4}{4!} f^{(4)}(\bar{x}) + \mathcal{O}(\Delta x_N^5)\right ) \\
\end{aligned}$$
where $\Delta\mathbf{x} = \bar{x} - \mathbf{x}$ is the distance between the point $\bar{x}$ and each grid point.

Equating terms of equal order reduces the problem to another Vandermonde matrix problem
$$\begin{bmatrix}
1 & 1 & 1 & \cdots & 1 \\
\Delta x_0 & \Delta x_1 & \Delta x_2 & \cdots & \Delta x_N\\
\frac{\Delta x_0^2}{2!} & \frac{\Delta x_1^2}{2!} & \frac{\Delta x_2^2}{2!} &\cdots &  \frac{\Delta x_N^2}{2!}\\
 &  & \vdots & \cdots &  \\
\frac{\Delta x_0^N}{N!} & \frac{\Delta x_1^N}{N!} & \frac{\Delta x_2^N}{N!} & \cdots &  \frac{\Delta x_N^N}{N!}\\
\end{bmatrix}
\begin{bmatrix} c_0 \\ c_1\\ c_2 \\ \vdots \\ c_N\\\end{bmatrix} =
\mathbf{b}_k
$$

where $\mathbf{b}_k$ is a vector of zeros with just a one in the $k$th position for the $k$th derivative.

By exactly accounting for the first $N+1$ terms of the Taylor series (with $N+1$ equations),  we can get any order derivative $0<k<N$ as well as an Error estimate for

$$R(\bar{x}) = O\left(\frac{\mathbf{c}^T\Delta\mathbf{x}^{N+1}}{(N+1)!}f^{(N+1)}\right) $$

This approach of "undetermined coefficients" can be efficiently coded up as a routine to provide consistent $Nth$ order finite difference coefficients for an arbitrarily spaced grid $\mathbf{x}$.

Here we present a python version of a matlab routine `fdcoeffV.m` from Randy Leveque's excellent book [Finite Difference Methods for ordinary and partial differential equations](https://faculty.washington.edu/rjl/fdmbook/)

```python
def fdcoeffV(k,xbar,x):
    """
    fdcoeffV routine modified from Leveque (2007) matlab function

    Params:
    -------

    k: int
        order of derivative
    xbar: float
        point at which derivative is to be evaluated
    x: ndarray
        numpy array of coordinates to use in calculating the weights

    Returns:
    --------
    c: ndarray
        array of floats of coefficients.

    Compute coefficients for finite difference approximation for the
    derivative of order k at xbar based on grid values at points in x.

    WARNING: This approach is numerically unstable for large values of n since
    the Vandermonde matrix is poorly conditioned.  Use fdcoeffF.m instead,
    which is based on Fornberg's method.

     This function returns a row vector c of dimension 1 by n, where n=length(x),
     containing coefficients to approximate u^{(k)}(xbar),
     the k'th derivative of u evaluated at xbar,  based on n values
     of u at x(1), x(2), ... x(n).

     If U is an array containing u(x) at these n points, then
     c.dot(U) will give the approximation to u^{(k)}(xbar).

     Note for k=0 this can be used to evaluate the interpolating polynomial
     itself.

    Requires len(x) > k.
    Usually the elements x(i) are monotonically increasing
    and x(1) <= xbar <= x(n), but neither condition is required.
    The x values need not be equally spaced but must be distinct.

    Modified rom  http://www.amath.washington.edu/~rjl/fdmbook/  (2007)
    """

    from  scipy.special import factorial

    n = x.shape[0]
    assert  k < n, " The order of the derivative must be less than the stencil width"

    # Generate the Vandermonde matrix from the Taylor series
    A = numpy.ones((n,n))
    xrow = (x - xbar)  # displacements x-xbar
    for i in range(1,n):
        A[i,:] = (xrow**(i))/factorial(i);

    b = numpy.zeros(n)    # b is right hand side,
    b[k] = 1              # so k'th derivative term remains

    c = numpy.linalg.solve(A,b)          # solve n by n system for coefficients

    return c

```

```python
N = 11
x = numpy.linspace(-2*numpy.pi, 2.*numpy.pi,  )
k = 2
scale = (x[1]-x[0])**k

print(fdcoeffV(k,x[0],x[:3])*scale)
for j in range(k,N-1):
    print(fdcoeffV(k,  x[j], x[j-1:j+2])*scale)
print(fdcoeffV(k,x[-1],x[-3:])*scale)
```

    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]
    [ 1. -2.  1.]

### Example: A variably spaced mesh

```python
N = 21
y = numpy.linspace(-.95, .95,N)
x = numpy.arctanh(y)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x,numpy.zeros(x.shape),'bo-')
axes.plot(x,y,'ro-')
axes.grid()
axes.set_xlabel('$x$')
axes.set_ylabel('$y$')
plt.show()
```

```python
k=1
fd = fdcoeffV(k,x[0],x[:3])
print('{}, sum={}'.format(fd,fd.sum()))
for j in range(1,N-1):
    fd = fdcoeffV(k,  x[j], x[j-1:j+2])
    print('{}, sum={}'.format(fd,fd.sum()))
fd = fdcoeffV(k,x[-1],x[-3:])
print('{}, sum={}'.format(fd,fd.sum()))

```

    [-2.99107041  5.38832143 -2.39725102], sum=0.0
    [-0.59748257 -1.79976846  2.39725102], sum=0.0
    [-1.4786644  -1.54760369  3.02626809], sum=0.0
    [-2.27379013 -1.34334775  3.61713788], sum=0.0
    [-2.97889933 -1.14769751  4.12659684], sum=0.0
    [-3.59168893 -0.95484098  4.54652991], sum=0.0
    [-4.11109435 -0.76316008  4.87425443], sum=0.0
    [-4.53657954 -0.57205069  5.10863023], sum=0.0
    [-4.86785915 -0.38124048  5.24909963], sum=0.0
    [-5.10478308 -0.19058641  5.29536948], sum=0.0
    [-5.24728627  0.          5.24728627], sum=0.0
    [-5.29536948  0.19058641  5.10478308], sum=0.0
    [-5.24909963  0.38124048  4.86785915], sum=0.0
    [-5.10863023  0.57205069  4.53657954], sum=0.0
    [-4.87425443  0.76316008  4.11109435], sum=0.0
    [-4.54652991  0.95484098  3.59168893], sum=0.0
    [-4.12659684  1.14769751  2.97889933], sum=-4.440892098500626e-16
    [-3.61713788  1.34334775  2.27379013], sum=0.0
    [-3.02626809  1.54760369  1.4786644 ], sum=-2.220446049250313e-16
    [-2.39725102  1.79976846  0.59748257], sum=0.0
    [ 2.39725102 -5.38832143  2.99107041], sum=0.0

### Application to Numerical PDE's

Given an efficent way to generate Finite Difference Coefficients these coefficients can be stored in a (usually sparse) matrix $D_k$ such that  given any discrete vector $\mathbf{f} = f(\mathbf{x})$,  We can calculate the approximate $k$th derivative as simply the matrix vector product

$$
    \mathbf{f}' = D_k\mathbf{f}
$$

This technique will become extremely useful when solving basic finite difference approximations to differential equations (as we will explore in future lectures and homeworks).

### The Bigger idea

More generally, using finite differences we can transform a continuous differential operator on a function space

$$
    v = \frac{d}{dx} u(x)
$$
which maps a function to a function,  to a discrete linear algebraic problem

$$
    \mathbf{v} = D\mathbf{u}
$$
where $\mathbf{v}, \mathbf{u}$ are discrete approximations to the continous functions $v,u$ and $D$ is a discrete differential operator (Matrix) which maps a vector to a vector.


# Linear Regression

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use(['ggplot'])
```

### Create Data

 Generate some data with:
\begin{equation} \theta_0= 4 \end{equation}
\begin{equation} \theta_1= 3 \end{equation}

Add some Gaussian noise to the data

```python
X = 2 * np.random.rand(100,1)
y = 4 +3 * X+np.random.randn(100,1)
```

Let's plot our data to check the relation between X and Y

```python
plt.plot(X,y,'b.')
plt.xlabel("$x$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
_ =plt.axis([0,2,0,15])
```

##  Analytical way of Linear Regression

```python
X_b = np.c_[np.ones((100,1)),X]
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
print(theta_best)
```

    [[3.78114801]
     [3.03518288]]

This is close to our real thetas 4 and 3. It cannot be accurate due to the noise

```python
X_new = np.array([[0],[2]])
X_new_b = np.c_[np.ones((2,1)),X_new]
y_predict = X_new_b.dot(theta_best)
y_predict
```

    array([[3.78114801],
           [9.85151377]])

Let's plot prediction line with calculated:theta

```python
plt.plot(X_new,y_predict,'r-')
plt.plot(X,y,'b.')
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([0,2,0,15])

```

## Gradient Descent

### Cost Function & Gradients

The equation for calculating cost function and gradients are as shown below. Please note the cost function is for Linear regression. For other algorithms the cost function will be different and the gradients would have  to be derived from the cost functions

Cost
\begin{equation}
J(\theta) = 1/2m \sum_{i=1}^{m} (h(\theta)^{(i)} - y^{(i)})^2
\end{equation}

Gradient

\begin{equation}
\frac{\partial J(\theta)}{\partial \theta_j} = 1/m\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_j^{(i)}
\end{equation}

Gradients
\begin{equation}
\theta_0: = \theta_0 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}
\begin{equation}
\theta_1: = \theta_1 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_1^{(i)})
\end{equation}
\begin{equation}
\theta_2: = \theta_2 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_2^{(i)})
\end{equation}

\begin{equation}
\theta_j: = \theta_j -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}

```python
def  cal_cost(theta,X,y):
    '''

    Calculates the cost for given X and Y. The following shows and example of a single dimensional X
    theta = Vector of thetas
    X     = Row of X's np.zeros((2,j))
    y     = Actual y's np.zeros((2,1))

    where:
        j is the no of features
    '''

    m = len(y)

    predictions = X.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost
```

```python
def gradient_descent(X,y,theta,learning_rate=0.01,iterations=100):
    '''
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)
    theta_history = np.zeros((iterations,2))
    for it in range(iterations):

        prediction = np.dot(X,theta)

        theta = theta -(1/m)*learning_rate*( X.T.dot((prediction - y)))
        theta_history[it,:] =theta.T
        cost_history[it]  = cal_cost(theta,X,y)

    return theta, cost_history, theta_history
```

#### Let's start with 1000 iterations and a learning rate of 0.01. Start with theta from a Gaussian distribution

```python
lr =0.01
n_iter = 1000

theta = np.random.randn(2,1)

X_b = np.c_[np.ones((len(X),1)),X]
theta,cost_history,theta_history = gradient_descent(X_b,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.599,
    Theta1:          3.184
    Final cost/MSE:  4600.829

 Let's plot the cost history over iterations

```python
fig,ax = plt.subplots(figsize=(12,8))

ax.set_ylabel('J(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(n_iter),cost_history,'b.')
```

#### After around 150 iterations the cost is flat so the remaining iterations  are not needed or will not result in any further optimization. Let us zoom in till iteration 200 and see the curve

```python
fig,ax = plt.subplots(figsize=(10,8))
_=ax.plot(range(200),cost_history[:200],'b.')
```

It is worth while to note that the cost drops faster initially and then the gain in cost reduction is not as much

It would be great to see the effect of different learning rates and iterations together

Let us  build a function which can show the effects together and also show how gradient decent actually is working

```python
def plot_GD(n_iter,lr,ax,ax1=None):
    """
     n_iter = no of iterations
     lr = Learning Rate
     ax = Axis to plot the Gradient Descent
     ax1 = Axis to plot cost_history vs Iterations plot

    """
    _ = ax.plot(X,y,'b.')
    theta = np.random.randn(2,1)

    tr =0.1
    cost_history = np.zeros(n_iter)
    for i in range(n_iter):
        pred_prev = X_b.dot(theta)
        theta,h,_ = gradient_descent(X_b,y,theta,lr,1)
        pred = X_b.dot(theta)

        cost_history[i] = h[0]

        if ((i % 25 == 0) ):
            _ = ax.plot(X,pred,'r-',alpha=tr)
            if tr < 0.8:
                tr = tr+0.2
    if not ax1== None:
        _ = ax1.plot(range(n_iter),cost_history,'b.')
```

#### Plot the graphs for different iterations and learning rates combination

```python
fig = plt.figure(figsize=(15,12))
fig.subplots_adjust(hspace=0.4, wspace=0.4)

it_lr =[(2000,0.001),(500,0.01),(200,0.05),(100,0.1)]
count =0
for n_iter, lr in it_lr:
    count += 1

    ax = fig.add_subplot(4, 2, count)
    count += 1

    ax1 = fig.add_subplot(4,2,count)

    ax.set_title("lr:{}".format(lr))
    ax1.set_title("Iterations:{}".format(n_iter))
    plot_GD(n_iter,lr,ax,ax1)

```

See how useful it is to visualize the effect of learning rates and iterations on gradient descent. The red lines show how the gradient descent starts and then slowly gets closer to the final value

We can always plot Indiviual graphs to zoom in

```python
_,ax = plt.subplots(figsize=(14,10))
plot_GD(100,0.1,ax)
```

## Stochastic Gradient Descent

```python
def stocashtic_gradient_descent(X,y,theta,learning_rate=0.01,iterations=10):
    '''
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)

    for it in range(iterations):
        cost =0.0
        for i in range(m):
            rand_ind = np.random.randint(0,m)
            X_i = X[rand_ind,:].reshape(1,X.shape[1])
            y_i = y[rand_ind].reshape(1,1)
            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta,X_i,y_i)
        cost_history[it]  = cost

    return theta, cost_history
```

```python
lr =0.5
n_iter = 50

theta = np.random.randn(2,1)

X_b = np.c_[np.ones((len(X),1)),X]
theta,cost_history = stocashtic_gradient_descent(X_b,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.807,
    Theta1:          2.948
    Final cost/MSE:  44.861

```python
fig,ax = plt.subplots(figsize=(10,8))

ax.set_ylabel('{J(Theta)}',rotation=0)
ax.set_xlabel('{Iterations}')
theta = np.random.randn(2,1)

_=ax.plot(range(n_iter),cost_history,'b.')
```

## Mini Batch Gradient Descent

```python
def minibatch_gradient_descent(X,y,theta,learning_rate=0.01,iterations=10,batch_size =20):
    '''
    X    = Matrix of X without added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)
    n_batches = int(m/batch_size)

    for it in range(iterations):
        cost =0.0
        indices = np.random.permutation(m)
        X = X[indices]
        y = y[indices]
        for i in range(0,m,batch_size):
            X_i = X[i:i+batch_size]
            y_i = y[i:i+batch_size]

            X_i = np.c_[np.ones(len(X_i)),X_i]

            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta,X_i,y_i)
        cost_history[it]  = cost

    return theta, cost_history
```

```python
lr =0.1
n_iter = 200

theta = np.random.randn(2,1)

theta,cost_history = minibatch_gradient_descent(X,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.728,
    Theta1:          3.080
    Final cost/MSE:  911.121

```python
fig,ax = plt.subplots(figsize=(10,8))

ax.set_ylabel('{J(Theta)}',rotation=0)
ax.set_xlabel('{Iterations}')
theta = np.random.randn(2,1)

_=ax.plot(range(n_iter),cost_history,'b.')
```

```python

```


### Example 2: Nonlinear convection in 2D

Following the initial convection tutorial with a single state variable $u$, we will now look at non-linear convection (step 6 in the original). This brings one new crucial challenge: computing a pair of coupled equations and thus updating two time-dependent variables $u$ and $v$.

The full set of coupled equations is now

\begin{aligned}
\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} + v \frac{\partial u}{\partial y} = 0 \\
\\
\frac{\partial v}{\partial t} + u \frac{\partial v}{\partial x} + v \frac{\partial v}{\partial y} = 0\\
\end{aligned}

and rearranging the discretized version gives us an expression for the update of both variables

\begin{aligned}
u_{i,j}^{n+1} &= u_{i,j}^n - u_{i,j}^n \frac{\Delta t}{\Delta x} (u_{i,j}^n-u_{i-1,j}^n) - v_{i,j}^n \frac{\Delta t}{\Delta y} (u_{i,j}^n-u_{i,j-1}^n) \\
\\
v_{i,j}^{n+1} &= v_{i,j}^n - u_{i,j}^n \frac{\Delta t}{\Delta x} (v_{i,j}^n-v_{i-1,j}^n) - v_{i,j}^n \frac{\Delta t}{\Delta y} (v_{i,j}^n-v_{i,j-1}^n)
\end{aligned}

So, for starters we will re-create the original example run in pure NumPy array notation, before demonstrating
the Devito version. Let's start again with some utilities and parameters:

```python
from examples.cfd import plot_field, init_hat
import numpy as np
import sympy
%matplotlib inline

# Some variable declarations
nx = 101
ny = 101
nt = 80
c = 1.
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .2
dt = sigma * dx
```

Let's re-create the initial setup with a 2D "hat function", but this time for two state variables.

```python
#NBVAL_IGNORE_OUTPUT

# Allocate fields and assign initial conditions
u = np.empty((nx, ny))
v = np.empty((nx, ny))

init_hat(field=u, dx=dx, dy=dy, value=2.)
init_hat(field=v, dx=dx, dy=dy, value=2.)

plot_field(u)
```

Now we can create the two stencil expression for our two coupled equations according to the discretized equation above. We again use some simple Dirichlet boundary conditions to keep the values on all sides constant.

```python
#NBVAL_IGNORE_OUTPUT
for n in range(nt + 1): ##loop across number of time steps
    un = u.copy()
    vn = v.copy()
    u[1:, 1:] = (un[1:, 1:] -
                 (un[1:, 1:] * c * dt / dy * (un[1:, 1:] - un[1:, :-1])) -
                  vn[1:, 1:] * c * dt / dx * (un[1:, 1:] - un[:-1, 1:]))
    v[1:, 1:] = (vn[1:, 1:] -
                 (un[1:, 1:] * c * dt / dy * (vn[1:, 1:] - vn[1:, :-1])) -
                 vn[1:, 1:] * c * dt / dx * (vn[1:, 1:] - vn[:-1, 1:]))

    u[0, :] = 1
    u[-1, :] = 1
    u[:, 0] = 1
    u[:, -1] = 1

    v[0, :] = 1
    v[-1, :] = 1
    v[:, 0] = 1
    v[:, -1] = 1

plot_field(u)
```

Excellent, we again get a wave that resembles the one from the oiginal examples.

Now we can set up our coupled problem in Devito. Let's start by creating two initial state variables $u$ and $v$, as before, and initialising them with our "hat function.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Grid, TimeFunction

# First we need two time-dependent data fields, both initialized with the hat function
grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)

v = TimeFunction(name='v', grid=grid)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

plot_field(u.data[0])
```

Using the two `TimeFunction` objects we can again derive our discretized equation, rearrange for the forward stencil point in time and define our variable update expression - only we have to do everything twice now! We again use forward differences for time via `u.dt` and backward differences in space via `u.dxl` and `u.dyl` to match the original tutorial.

```python
from devito import Eq, solve

eq_u = Eq(u.dt + u*u.dxl + v*u.dyl)
eq_v = Eq(v.dt + u*v.dxl + v*v.dyl)

# We can use the same SymPy trick to generate two
# stencil expressions, one for each field update.
stencil_u = solve(eq_u, u.forward)
stencil_v = solve(eq_v, v.forward)
update_u = Eq(u.forward, stencil_u, subdomain=grid.interior)
update_v = Eq(v.forward, stencil_v, subdomain=grid.interior)

print("U update:\n%s\n" % update_u)
print("V update:\n%s\n" % update_v)
```

    U update:
    Eq(u(t + dt, x, y), dt*(-(u(t, x, y)/h_x - u(t, x - h_x, y)/h_x)*u(t, x, y) - (u(t, x, y)/h_y - u(t, x, y - h_y)/h_y)*v(t, x, y) + u(t, x, y)/dt))

    V update:
    Eq(v(t + dt, x, y), dt*(-(v(t, x, y)/h_x - v(t, x - h_x, y)/h_x)*u(t, x, y) - (v(t, x, y)/h_y - v(t, x, y - h_y)/h_y)*v(t, x, y) + v(t, x, y)/dt))

We then set Dirichlet boundary conditions at all sides of the domain to $1$.

```python
x, y = grid.dimensions
t = grid.stepping_dim
bc_u = [Eq(u[t+1, 0, y], 1.)]  # left
bc_u += [Eq(u[t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(u[t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(u[t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(v[t+1, 0, y], 1.)]  # left
bc_v += [Eq(v[t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(v[t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(v[t+1, x, 0], 1.)]  # bottom
```

And finally we can put it all together to build an operator and solve our coupled problem.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our data field and ICs
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

op = Operator([update_u, update_v] + bc_u + bc_v)
op(time=nt, dt=dt)

plot_field(u.data[0])
```

Excellent, we have now a scalar implementation of a convection problem, but this can be written as a single vectorial equation:

$\frac{d U}{dt} + \nabla(U)U = 0$

Let's now use devito vectorial utilities and implement the vectorial equation

```python
from devito import VectorTimeFunction, grad

U = VectorTimeFunction(name='U', grid=grid)
init_hat(field=U[0].data[0], dx=dx, dy=dy, value=2.)
init_hat(field=U[1].data[0], dx=dx, dy=dy, value=2.)

plot_field(U[1].data[0])

eq_u = Eq(U.dt + grad(U)*U)
```

We now have a vectorial equation. Unlike in the previous case, we do not need to play with left/right derivatives
as the automated staggering of the vectorial function takes care of this.

```python
eq_u
```

$\displaystyle \left[\begin{matrix}\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \frac{\partial}{\partial t} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}\\\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \frac{\partial}{\partial t} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}\end{matrix}\right] = 0$

Then we set the nboundary conditions

```python
x, y = grid.dimensions
t = grid.stepping_dim
bc_u = [Eq(U[0][t+1, 0, y], 1.)]  # left
bc_u += [Eq(U[0][t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(U[0][t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(U[0][t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(U[1][t+1, 0, y], 1.)]  # left
bc_v += [Eq(U[1][t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(U[1][t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(U[1][t+1, x, 0], 1.)]  # bottom
```

```python
# We can use the same SymPy trick to generate two
# stencil expressions, one for each field update.
stencil_U = solve(eq_u, U.forward)
update_U = Eq(U.forward, stencil_U, subdomain=grid.interior)
```

And we have the updated (stencil) as a vectorial equation once again

```python
update_U
```

$\displaystyle \left[\begin{matrix}\operatorname{U_{x}}{\left(t + dt,x + \frac{h_{x}}{2},y \right)}\\\operatorname{U_{y}}{\left(t + dt,x,y + \frac{h_{y}}{2} \right)}\end{matrix}\right] = \left[\begin{matrix}dt \left(- \left(- \frac{\operatorname{U_{x}}{\left(t,x - \frac{h_{x}}{2},y \right)}}{h_{x}} + \frac{\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}}{h_{x}}\right) \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} - \left(\frac{\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}}{h_{y}} - \frac{\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y - h_{y} \right)}}{h_{y}}\right) \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \frac{\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}}{dt}\right)\\dt \left(- \left(\frac{\operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}}{h_{x}} - \frac{\operatorname{U_{y}}{\left(t,x - h_{x},y + \frac{h_{y}}{2} \right)}}{h_{x}}\right) \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} - \left(- \frac{\operatorname{U_{y}}{\left(t,x,y - \frac{h_{y}}{2} \right)}}{h_{y}} + \frac{\operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}}{h_{y}}\right) \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \frac{\operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}}{dt}\right)\end{matrix}\right]$

We finally run the operator

```python
#NBVAL_IGNORE_OUTPUT
op = Operator([update_U] + bc_u + bc_v)
op(time=nt, dt=dt)

# The result is indeed the expected one.
plot_field(U[0].data[0])
```

```python
from devito import norm
assert np.isclose(norm(u), norm(U[0]), rtol=1e-5, atol=0)
```

```python

```


# Método de punto fijo
J.J
---
Dado que se busca resolver $f(x) = 0$, la función $f$ puede ser reescrita como $f(x) = g(x) - x = 0$. El método de punto fijo está dado por las iteraciones:

\begin{equation}
x_{k+1} = g(x_{k}).
\end{equation}

El método converge a una raíz si los valores de $x_{k}$ en cada iteración cumplen $|g'(x)| \leq 1$.

Ejemplo: $f(x) = 2x^{3} - 9x^{2} + 7x + 6 = 0$
\begin{equation}
x = \frac{1}{7} \{ -2x^{3} + 9x^{2} - 6 \}
\end{equation}

```python
import numpy as np
from sympy import *
from sympy.utilities.lambdify import lambdify
import matplotlib.pyplot as plt
init_printing(use_unicode=True)
```

```python
#calcular la derivada de f
x = symbols('x')
funcion = 2*x**3 - 9*x**2 + 7*x + 6
gfuncion = (-2*x**3 + 9*x**2 - 6)/7 #escribir la función aqui
dgfuncion = diff(gfuncion, x)
print(str(dgfuncion))
```

    -6*x**2/7 + 18*x/7

```python
f = lambdify(x, funcion)
g = lambdify(x, gfuncion)
dg = lambdify(x, dgfuncion)
```

```python
X = np.linspace(-1, 4, 100)
plt.plot(X,dg(X), label = 'g´(x)')
plt.ylim(-1,1)
plt.legend()
plt.show()
```

La desigualdad se cumple aproximadamente entre en los intervalos $[-0.44, 0.46]$ y $[2.5, 3.34]$, aun que en realidad, en el primer intervalo no converge.

```python
e = 0.0001 #error
maxit = 100 #iteraciones máximas
```

```python
def PuntoFijo(x0, func = g, error = e, iterations = maxit):
    it = 0
    while (abs(f(x0)) > e) and (it < maxit):
        it += 1
        xk = g(x0)
        x0 = xk
    return x0
```

```python
sol = PuntoFijo(2.6)
print(sol)
```

    2.9999926857948673

```python
plt.plot(X, f(X), label='f(x)')
plt.plot(sol,f(sol),'ro')
plt.legend()
plt.show()
```

```python

```


Table of Contents

1  Question

1.1  Answer

2  Question

2.1  Answer

3  Question

3.1  Answer

# Question

**Question 1 :** Write a program which will find all such numbers which are divisible by 7 but are not a multiple of 5, between 2000 and 3200 (both included). The numbers obtained should be printed in a comma-separated sequence on a single line.

**BreakDown Points:**
1) Get the Numbers between 2000 and 3200

2) Find all Numbers divisible by 7

3) From the result exclude Numbers divisible by 5

4) Print in comma-separated sequence

## Answer

```python
for i in range(2000,3200+1):
    if i%7 == 0 and i%5 != 0:
        print(i, end=',')
```

    2002,2009,2016,2023,2037,2044,2051,2058,2072,2079,2086,2093,2107,2114,2121,2128,2142,2149,2156,2163,2177,2184,2191,2198,2212,2219,2226,2233,2247,2254,2261,2268,2282,2289,2296,2303,2317,2324,2331,2338,2352,2359,2366,2373,2387,2394,2401,2408,2422,2429,2436,2443,2457,2464,2471,2478,2492,2499,2506,2513,2527,2534,2541,2548,2562,2569,2576,2583,2597,2604,2611,2618,2632,2639,2646,2653,2667,2674,2681,2688,2702,2709,2716,2723,2737,2744,2751,2758,2772,2779,2786,2793,2807,2814,2821,2828,2842,2849,2856,2863,2877,2884,2891,2898,2912,2919,2926,2933,2947,2954,2961,2968,2982,2989,2996,3003,3017,3024,3031,3038,3052,3059,3066,3073,3087,3094,3101,3108,3122,3129,3136,3143,3157,3164,3171,3178,3192,3199,

# Question

**Question 2 :** Write a Python program to accept the user's first and last name and then getting them printed in the the reverse order with a space between first name and last name.

**BreakDown Points:**
1) Read First Name

2) Read Last Name

3) Print in reverse order

## Answer

```python
FullName = input('Please Enter First & Last Name : ').split()

for i in FullName[::-1]:
    print(i, end=' ')
```

    Please Enter First & Last Name : V P
    P V

# Question

**Question 3 :** Write a Python program to find the volume of a sphere with diameter 12 cm

\begin{align}
Formula : V = \frac{4}{3} * {\pi} * (\frac{d}{2})^3\\
\end{align}

**BreakDown Points:**
1) Replce r with d/2

2) Value of pi is availble in math module

## Answer

```python
import math
volume = (4/3) * math.pi  * (12/2)**3
print(volume)
```

    904.7786842338603

```python
volume = (4/3) * 3.141  * (12/2)**3
print(volume)
```

    904.608


Traçar um esboço do gráfico e obter uma equação da parábola que satisfaça as condições dadas.

19. Vértice: $V(2,-1)$; Foco: $F(5,-1)$

Como o vértice da parábola se encontra fora da origem e está paralela ao eixo $x$ a sua equação é dada por $(y-k)^2 = 2p(x-h)$

Substituindo os pontos do vértice por $x = 2$ e $y = -1$

$(y-(-1))^2 = 2p(x-2))$

$(y+1)^2 = 2p(x-2)$

Achando o valor de $p$

$\frac{p}{2} = \sqrt{(5-2)^2 + (-1-(-1))^2}$

$\frac{p}{2} = \sqrt{3^2 + 0}$

$\frac{p}{2} = \pm \sqrt{9}$

$\frac{p}{2} = 3$

$p = 6$

Substituindo $p$ na fórmula

$(y+1)^2 = 2 \cdot 6 (x-2)$

$(y+1)^2 = 12(x-2)$

$y^2 + 2y + 1 = 12x - 24$

$y^2 + 2y -12x +1 + 24 = 0$

$y^2 + 2y -12x + 25 = 0 $

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((y+1)**2, 6*(x-2)), (x,-20,20), (y,-10,10),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/19.ipynb


#Taylor Series Expansion with Python from Data Science Fabric

recovered from **[Data Science Fabric](https://dsfabric.org/taylor-series-expansion-with-python)**

Note: for ease and organization, titles were placed on the notebook for quick reading

##Libraries

```
from sympy import series, Symbol
from sympy.functions import sin, cos, exp
from sympy.plotting import plot
import matplotlib.pyplot as plt
```

```
from sympy.functions import ln
```

```
# Define symbol
x = Symbol('x')
```

```
# Function for Taylor Series Expansion

def taylor(function, x0, n):
    """
    Parameter "function" is our function which we want to approximate
    "x0" is the point where to approximate
    "n" is the order of approximation
    """
    return function.series(x,x0,n).removeO()
```

##First's Cases of Use

```
print('sin(x) ≅', taylor(sin(x), 0, 4))

print('cos(x) ≅', taylor(cos(x), 0, 4))

print('e(x) ≅', taylor(exp(x), 0, 4))
```

    sin(x) ≅ -x**3/6 + x
    cos(x) ≅ 1 - x**2/2
    e(x) ≅ x**3/6 + x**2/2 + x + 1

```
print("Ejercicio")
print('ln(x+1) ≅', taylor(ln(x+1), 0, 4))
```

    Ejercicio
    ln(x+1) ≅ x**3/3 - x**2/2 + x

```
print('sin(1) =', taylor(sin(x), 0, 4).subs(x,1))

print('cos(1) =', taylor(cos(x), 0, 4).subs(x,1))

print('e(1) =', taylor(exp(x), 0, 4).subs(x,1))
```

    sin(1) = 5/6
    cos(1) = 1/2
    e(1) = 8/3

```
print("Ejercicio")

print('ln((1)+1) =', taylor(ln(x+1), 0, 4).subs(x,1))
```

    Ejercicio
    ln((1)+1) = 5/6

##Tests of Taylor's Series

```
print('Taylor 0 exp(x) ≅', taylor(exp(x), 0, 0))
print('Taylor 1 exp(x) ≅', taylor(exp(x), 0, 1))
print('Taylor 2 exp(x) ≅', taylor(exp(x), 0, 2))
print('Taylor 3 exp(x) ≅', taylor(exp(x), 0, 3))
print('Taylor 4 exp(x) ≅', taylor(exp(x), 0, 4))
print('Taylor 5 exp(x) ≅', taylor(exp(x), 0, 5))
print('Taylor 6 exp(x) ≅', taylor(exp(x), 0, 6))
print('Taylor 7 exp(x) ≅', taylor(exp(x), 0, 7))
print('Taylor 8 exp(x) ≅', taylor(exp(x), 0, 8))
```

    Taylor 0 exp(x) ≅ 0
    Taylor 1 exp(x) ≅ 1
    Taylor 2 exp(x) ≅ x + 1
    Taylor 3 exp(x) ≅ x**2/2 + x + 1
    Taylor 4 exp(x) ≅ x**3/6 + x**2/2 + x + 1
    Taylor 5 exp(x) ≅ x**4/24 + x**3/6 + x**2/2 + x + 1
    Taylor 6 exp(x) ≅ x**5/120 + x**4/24 + x**3/6 + x**2/2 + x + 1
    Taylor 7 exp(x) ≅ x**6/720 + x**5/120 + x**4/24 + x**3/6 + x**2/2 + x + 1
    Taylor 8 exp(x) ≅ x**7/5040 + x**6/720 + x**5/120 + x**4/24 + x**3/6 + x**2/2 + x + 1

```
print("Ejercicio")

for i in range(1,10):
  print('Taylor', i,'ln(x+1) ≅', taylor(ln(x+1), 0, i))
```

    Ejercicio
    Taylor 1 ln(x+1) ≅ 0
    Taylor 2 ln(x+1) ≅ x
    Taylor 3 ln(x+1) ≅ -x**2/2 + x
    Taylor 4 ln(x+1) ≅ x**3/3 - x**2/2 + x
    Taylor 5 ln(x+1) ≅ -x**4/4 + x**3/3 - x**2/2 + x
    Taylor 6 ln(x+1) ≅ x**5/5 - x**4/4 + x**3/3 - x**2/2 + x
    Taylor 7 ln(x+1) ≅ -x**6/6 + x**5/5 - x**4/4 + x**3/3 - x**2/2 + x
    Taylor 8 ln(x+1) ≅ x**7/7 - x**6/6 + x**5/5 - x**4/4 + x**3/3 - x**2/2 + x
    Taylor 9 ln(x+1) ≅ -x**8/8 + x**7/7 - x**6/6 + x**5/5 - x**4/4 + x**3/3 - x**2/2 + x

```
print("Ejercicio")

for i in range(1,10):
  print('Taylor', i,'sin(x) ≅', taylor(sin(x), 0, i))
```

    Ejercicio
    Taylor 1 sin(x) ≅ 0
    Taylor 2 sin(x) ≅ x
    Taylor 3 sin(x) ≅ x
    Taylor 4 sin(x) ≅ -x**3/6 + x
    Taylor 5 sin(x) ≅ -x**3/6 + x
    Taylor 6 sin(x) ≅ x**5/120 - x**3/6 + x
    Taylor 7 sin(x) ≅ x**5/120 - x**3/6 + x
    Taylor 8 sin(x) ≅ -x**7/5040 + x**5/120 - x**3/6 + x
    Taylor 9 sin(x) ≅ -x**7/5040 + x**5/120 - x**3/6 + x

```
print('Taylor 0 sin(x) ≅', taylor(sin(x), 0, 0).subs(x,2),' = ',taylor(sin(x), 0, 0).subs(x,2).evalf())
print('Taylor 1 cos(x) ≅', taylor(cos(x), 0, 1).subs(x,2),' = ',taylor(cos(x), 0, 1).subs(x,2).evalf())
print('Taylor 2 exp(x) ≅', taylor(exp(x), 0, 2).subs(x,2),' = ',taylor(exp(x), 0, 2).subs(x,2).evalf())
print('Taylor 3 exp(x) ≅', taylor(exp(x), 0, 3).subs(x,2),' = ',taylor(exp(x), 0, 3).subs(x,2).evalf())
print('Taylor 4 exp(x) ≅', taylor(exp(x), 0, 4).subs(x,2),' = ',taylor(exp(x), 0, 4).subs(x,2).evalf())
print('Taylor 5 exp(x) ≅', taylor(exp(x), 0, 5).subs(x,2),' = ',taylor(exp(x), 0, 5).subs(x,2).evalf())
print('Taylor 6 exp(x) ≅', taylor(exp(x), 0, 6).subs(x,2),' = ',taylor(exp(x), 0, 6).subs(x,2).evalf())
print('Taylor 7 exp(x) ≅', taylor(exp(x), 0, 8).subs(x,2),' = ',taylor(exp(x), 0, 7).subs(x,2).evalf())
```

    Taylor 0 sin(x) ≅ 0  =  0
    Taylor 1 cos(x) ≅ 1  =  1.00000000000000
    Taylor 2 exp(x) ≅ 3  =  3.00000000000000
    Taylor 3 exp(x) ≅ 5  =  5.00000000000000
    Taylor 4 exp(x) ≅ 19/3  =  6.33333333333333
    Taylor 5 exp(x) ≅ 7  =  7.00000000000000
    Taylor 6 exp(x) ≅ 109/15  =  7.26666666666667
    Taylor 7 exp(x) ≅ 155/21  =  7.35555555555556

```
print("Ejercicio")
print('Taylor 0 sin(x) ≅', taylor(sin(x), 0, 0).subs(x,2),' = ',taylor(sin(x), 0, 0).subs(x,2).evalf())
print('Taylor 1 sin(x) ≅', taylor(sin(x), 0, 1).subs(x,2),' = ',taylor(sin(x), 0, 1).subs(x,2).evalf())
print('Taylor 2 sin(x) ≅', taylor(sin(x), 0, 2).subs(x,2),' = ',taylor(sin(x), 0, 2).subs(x,2).evalf())
print('Taylor 3 sin(x) ≅', taylor(sin(x), 0, 3).subs(x,2),' = ',taylor(sin(x), 0, 3).subs(x,2).evalf())
print('Taylor 4 ln(x+1) ≅', taylor(ln(x+1), 0, 4).subs(x,2),' = ',taylor(ln(x+1), 0, 4).subs(x,2).evalf())
print('Taylor 5 ln(x+1) ≅', taylor(ln(x+1), 0, 5).subs(x,2),' = ',taylor(ln(x+1), 0, 5).subs(x,2).evalf())
print('Taylor 6 ln(x+1) ≅', taylor(ln(x+1), 0, 6).subs(x,2),' = ',taylor(ln(x+1), 0, 6).subs(x,2).evalf())
print('Taylor 7 ln(x+1) ≅', taylor(ln(x+1), 0, 8).subs(x,2),' = ',taylor(ln(x+1), 0, 7).subs(x,2).evalf())
```

    Ejercicio
    Taylor 0 sin(x) ≅ 0  =  0
    Taylor 1 sin(x) ≅ 0  =  0
    Taylor 2 sin(x) ≅ 2  =  2.00000000000000
    Taylor 3 sin(x) ≅ 2  =  2.00000000000000
    Taylor 4 ln(x+1) ≅ 8/3  =  2.66666666666667
    Taylor 5 ln(x+1) ≅ -4/3  =  -1.33333333333333
    Taylor 6 ln(x+1) ≅ 76/15  =  5.06666666666667
    Taylor 7 ln(x+1) ≅ 444/35  =  -5.60000000000000

##Comparison between methods

```
import math
print('sympy exp(x)subs(x,2) =', exp(x).subs(x,2))
print('sympy exp(x).subs(x,2).evalf() =', exp(x).subs(x,2).evalf())
print('math.exp(2) =', math.exp(2))
```

    sympy exp(x)subs(x,2) = exp(2)
    sympy exp(x).subs(x,2).evalf() = 7.38905609893065
    math.exp(2) = 7.38905609893065

```
print("Ejercicio")
import math
print('sympy ln(x+1)subs(x,2) =', ln(x+1).subs(x,2))
print('sympy ln(x+1).subs(x,2).evalf() =', ln(x+1).subs(x,2).evalf())
print('math.ln(2+1) =', math.log1p(2))
```

    Ejercicio
    sympy ln(x+1)subs(x,2) = log(3)
    sympy ln(x+1).subs(x,2).evalf() = 1.09861228866811
    math.ln(2+1) = 1.0986122886681096

```
print("Ejercicio")
import math
print('sympy sin(x)subs(x,2) =', sin(x).subs(x,2))
print('sympy sin(x).subs(x,2).evalf() =', sin(x).subs(x,2).evalf())
print('math.sin(2) =', math.sin(2))
```

    Ejercicio
    sympy sin(x)subs(x,2) = sin(2)
    sympy sin(x).subs(x,2).evalf() = 0.909297426825682
    math.sin(2) = 0.9092974268256817

##Plots of `exp()`

```
import math
import numpy as np
import matplotlib.pyplot as plt

values = np.arange(-5,5,0.1)
p_exp = np.exp(values)
t_exp1 = [taylor(exp(x), 0, 1).subs(x,v) for v in values]
legends = ['exp() ','Taylor 1 (constant)']

fig, ax = plt.subplots()
ax.plot(values,p_exp, color ='red')
ax.plot(values,t_exp1)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_exp = np.exp(values)
t_exp2 = [taylor(exp(x), 0, 2).subs(x,v) for v in values]
legends = ['exp() ','Taylor 2 (linear)']

fig, ax = plt.subplots()
ax.plot(values,p_exp, color ='red')
ax.plot(values,t_exp2)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_exp = np.exp(values)
t_exp3 = [taylor(exp(x), 0, 3).subs(x,v) for v in values]
legends = ['exp() ','Taylor 3 (quadratic)']

fig, ax = plt.subplots()
ax.plot(values,p_exp, color ='red')
ax.plot(values,t_exp3)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_exp = np.exp(values)
t_exp4 = [taylor(exp(x), 0, 4).subs(x,v) for v in values]
legends = ['exp() ','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_exp, color ='red')
ax.plot(values,t_exp4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_exp = np.exp(values)
t_exp1 = [taylor(exp(x), 0, 1).subs(x,v) for v in values]
t_exp2 = [taylor(exp(x), 0, 2).subs(x,v) for v in values]
t_exp3 = [taylor(exp(x), 0, 3).subs(x,v) for v in values]
t_exp4 = [taylor(exp(x), 0, 4).subs(x,v) for v in values]
legends = ['exp() ','Taylor 1 (constant)','Taylor 3 (linear)','Taylor 3 (quadratic)','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_exp)
ax.plot(values,t_exp1)
ax.plot(values,t_exp2)
ax.plot(values,t_exp3)
ax.plot(values,t_exp4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

##Plots of $\ln(x+1)$

```
import math
import numpy as np
import matplotlib.pyplot as plt

values = np.arange(0,5,0.1)
p_ln = [math.log1p(value) for value in values]
t_ln1 = [taylor(ln(x+1), 0, 1).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 1 (constant)']

fig, ax = plt.subplots()
ax.plot(values,p_ln, color ='red')
ax.plot(values,t_ln1)

ax.set_ylim([-5,5])
#ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
#ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
print("Note that the blue line is in y=0")
```

```
import math
import numpy as np
import matplotlib.pyplot as plt

values = np.arange(0,5,0.1)
p_ln = [math.log1p(value) for value in values]
t_ln2 = [taylor(ln(x+1), 0, 2).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 2 (Lineal)']

fig, ax = plt.subplots()
ax.plot(values,p_ln, color ='red')
ax.plot(values,t_ln2)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(0,5,0.1)
p_ln = [math.log1p(value) for value in values]
t_ln3 = [taylor(ln(x+1), 0, 3).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 3 (Quadratic)']

fig, ax = plt.subplots()
ax.plot(values,p_ln, color ='red')
ax.plot(values,t_ln3)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt

values = np.arange(0,5,0.1)
p_ln = [math.log1p(value) for value in values]
t_ln4 = [taylor(ln(x+1), 0, 4).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 4 (Cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_ln, color ='red')
ax.plot(values,t_ln4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(0,5,0.1)
p_ln = [math.log1p(value) for value in values]
t_ln1 = [taylor(ln(x+1), 0, 1).subs(x,v) for v in values]
t_ln2 = [taylor(ln(x+1), 0, 2).subs(x,v) for v in values]
t_ln3 = [taylor(ln(x+1), 0, 3).subs(x,v) for v in values]
t_ln4 = [taylor(ln(x+1), 0, 4).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 1 (constant)','Taylor 3 (linear)','Taylor 3 (quadratic)','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_ln)
ax.plot(values,t_ln1)
ax.plot(values,t_ln2)
ax.plot(values,t_ln3)
ax.plot(values,t_ln4)

ax.set_ylim([-2,3])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

##Plots of $\sin(x)$

```
import math
import numpy as np
import matplotlib.pyplot as plt

values = np.arange(-2*math.pi,2*math.pi,0.1)
p_sin = [math.sin(value) for value in values]
t_sin1 = [taylor(sin(x), 0, 1).subs(x,v) for v in values]
legends = ['sin() ','Taylor 1 (constant)']

fig, ax = plt.subplots()
ax.plot(values,p_sin, color ='red')
ax.plot(values,t_sin1)

ax.set_ylim([-5,5])
#ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
#ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-2*math.pi,2*math.pi,0.1)
p_sin = [math.sin(value) for value in values]
t_sin2 = [taylor(sin(x), 0, 2).subs(x,v) for v in values]
legends = ['sin() ','Taylor 2 (linear)']

fig, ax = plt.subplots()
ax.plot(values,p_sin, color ='red')
ax.plot(values,t_sin2)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt

values = np.arange(-2*math.pi,2*math.pi,0.1)
p_sin = [math.sin(value) for value in values]
t_sin3 = [taylor(sin(x), 0, 3).subs(x,v) for v in values]
legends = ['sin()','Taylor 3 (quadratic)']

fig, ax = plt.subplots()
ax.plot(values,p_sin, color ='red')
ax.plot(values,t_sin3)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-2*math.pi,2*math.pi,0.1)
p_sin = [math.sin(value) for value in values]
t_sin4 = [taylor(sin(x), 0, 4).subs(x,v) for v in values]
legends = ['sin() ','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_sin, color ='red')
ax.plot(values,t_sin4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```
import math
import numpy as np
import matplotlib.pyplot as plt

values = np.arange(-2*math.pi,2*math.pi,0.1)
p_sin = [math.sin(value) for value in values]
t_sin1 = [taylor(sin(x), 0, 1).subs(x,v) for v in values]
t_sin2 = [taylor(sin(x), 0, 2).subs(x,v) for v in values]
t_sin3 = [taylor(sin(x), 0, 3).subs(x,v) for v in values]
t_sin4 = [taylor(sin(x), 0, 4).subs(x,v) for v in values]
legends = ['sin() ','Taylor 1 (constant)','Taylor 3 (linear)','Taylor 3 (quadratic)','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_sin)
ax.plot(values,t_sin1)
ax.plot(values,t_sin2)
ax.plot(values,t_sin3)
ax.plot(values,t_sin4)

ax.set_ylim([-5,5])
#ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
#ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```


# Bead on a rotating circle

We consider a movement of a material point (bead) with the mass $m$ in a gravitational field, along a vertical circle with a radius $l$ rotating with the frequency $\omega$ around a vertical axis, passing through the center of the circle.

Turning circle creates a virtual sphere with a $R$ radius and a bead moves in this sphere. Therefore, it is most convenient to choose a reference system in $(r, \theta, \phi)$ spherical variables:

\begin{eqnarray}
x&=&r \sin \theta \cos \phi \\
y&=&r \sin \theta \sin \phi \\
z&=&r \cos \theta
\end{eqnarray}

It is a system with constraints and therefore it is convenient to analyze this issue in the framework of Lagrange mechanics.

The Lagrange function is the difference in the kinetic energy of the bead and its potential energy (particles in the gravitational field):

$$L=L(r, \theta, \phi, \dot{r}, \dot{\theta}, \dot{\phi}) = E_k -E_p$$

The kinetic energy of the bead is given by the formula:

\begin{equation}
\label{eq:Ek_spherical}
E_k =\frac{m v^2}{2} = \frac{m}{2} [\dot{x}^2 + \dot{y}^2 + \dot{z}^2] =  \frac{m}{2}[\dot{r}^2 + r^2 \dot{\theta}^2 + r^2 \sin^2 (\theta) \dot{\phi}^2]
\end{equation}

The potential energy of the bead has the form
$$E_p = mgz = mgr \cos(\theta)$$

## Lagrange aproach in spherical coordinates

We can  write Lagrangian of the system in   spherical coordinates. Contraints in spherical coordinates have simple form:

 - $r = l$ - radius is constant.
 - $\phi(t)=\omega_0 t$ - the azimuthal angle is forced by the  constraint.

Effectively, the only degree of freedom is a polar angle $\theta$.

We will used CAS to derive both the form of the Lagrangian as well as equation of motion.

```python
load('cas_utils.sage')
```

```python
var('t')
var('l g w0')
xy_names = [('x','x'),('y','y'),('z','z')]
uv_names = [('r','r'),('phi',r'\phi'),('theta',r'\theta')]

load('cas_utils.sage')

to_fun, to_var = make_symbols(xy_names,uv_names)
x2u = {x:r*sin(theta)*cos(phi),
       y: r*sin(theta)*sin(phi),
       z: r*cos(theta)}
_  = transform_virtual_displacements(xy_names,uv_names,suffix='_uv')
```

```python
Ek = 1/2*sum([x_.subs(x2u).subs(to_fun).diff(t).subs(to_var)^2 \
              for x_ in [x,y,z]])
```

The symbolic variable is kinetic energy in spherical coordinates \ref{eq:Ek_spherical},

```python
showmath(Ek.trig_simplify())
```

At this point we can substitute conditions resulting from constraints. Note that since we have already time derivatives $\dot r$ and $\dot phi$, we have to also substitute them explicitely.

```python
Ek = Ek.subs({phi:w0*t,phid:w0,rd:0,r:l}).trig_simplify()
showmath(Ek)
```

The potential energy can be computer in a similar way:

```python
Ep = g*z.subs(x2u).subs({r:l})
showmath(Ep)
```

Langrange function is a difference between $E_k$ and $E_{pot}$:

```python
L = Ek-Ep
showmath(L.trig_simplify())
```

We have single Euler-Lagrange equation in generalized coodinate: polar angle $\theta$.

```python
EL = L.diff(thetad).subs(to_fun).diff(t).subs(to_var) - L.diff(theta)
```

```python
showmath(EL.trig_simplify())
```

## Analysis of the system

### Effective potential

Fist, we can notice that the form of Lagrangian contains kinetic part, with term contatining $\dot \theta$ and two terms dependent only on $\theta$. We can interpret the latter as an effective potential of 1d motion in $\theta$ coordinate.  The term comes from kinetic energy.

We can extract it symbolically in SageMath by setting $\dot \theta=0$ in Lagrangian:

```python
Ueff = -L.subs(thetad==0)
showmath(Ueff)
```

```python
plot?
```

```python
w0s = [0,3,4,5]
p_u1 = plot([Ueff.subs({w0:w0_,l:1,g:9.81}) for w0_ in w0s],\
            (theta,-3*pi/2-0.01,pi/2),
           legend_label=[r'$\omega_0=%0.1f$'%w_ for w_ in w0s],
         tick_formatter=pi, gridlines=[[-pi],None])
p_u1.show(figsize=(6,3))
```

\newpage


# Multivariate Normal Distribution - Mahalanobis Distance

> This document is written in *R*.
>
> ***GitHub***: https://github.com/czs108

## Question A

> Generate a sample of **1000** random points from a *Multivariate Normal Distribution* with *means* **0.0**, **0.0** and *standard deviations* of **1.5** along the $x$ axis and **0.5** along the $y$ axis.
>
> Plot the points on a graph with limits on the $x$ axis of $(-10,\, 10)$ and limits on the $y$ axis of $(-10,\, 10)$.

Use the `rnorm` function.

```R
total <- 1000
X.1 <- rnorm(n=total, mean=0, sd=1.5)
Y.1 <- rnorm(n=total, mean=0, sd=0.5)

plot(Y.1 ~ X.1, xlim=c(-10, 10), ylim=c(-10, 10), xlab="X", ylab="Y")
```

## Question B

> Generate a sample of **1000** random points from a *Multivariate Normal Distribution* with *means* **-4.0**, **0.0** and *standard deviations* of **0.5** along the $x$ axis and **1.5** along the $y$ axis.
>
> Plot the points on the current graph with color *blue*.

```R
X.2 <- rnorm(n=total, mean=-4.0, sd=0.5)
Y.2 <- rnorm(n=total, mean=0, sd=1.5)

plot(Y.1 ~ X.1, xlim=c(-10, 10), ylim=c(-10, 10), xlab="X", ylab="Y")
points(Y.2 ~ X.2, col="blue")
```

## Question C

> Compute the *Mahalanobis Distance* of each of the **2000** points from each of the distributions.

\begin{equation}
d^{2} = \frac{(x - m_x)^2}{\sigma^2_x} + \frac{(y - m_y)^2}{\sigma^2_y}
\end{equation}

```R
mahadist <- function(point, mean, sd) {
    d.x <- ((point[1] - mean[1]) ^ 2) / (sd[1] ^ 2)
    d.y <- ((point[2] - mean[2]) ^ 2) / (sd[2] ^ 2)
    return (d.x + d.y)
}
```

```R
dist.1 <- NULL
for (i in c(1:total)) {
    dist.1[i] <- mahadist(c(X.1[i], Y.1[i]), c(0, 0), c(1.5, 0.5))
}

dist.2 <- NULL
for (i in c(1:total)) {
    dist.2[i] <- mahadist(c(X.2[i], Y.2[i]), c(-4.0, 0), c(0.5, 1.5))
}

dist.1[c(1:5)]
dist.2[c(1:5)]
```

0.0835216640048668

0.525807369423523

2.89716097154469

2.76312320046508

0.570738446675666

0.849843382279618

0.782329010614814

0.0402248932660116

3.79224983124425

1.7885729380896

## Question D

> Classify each point using the *Mahalanobis Distance*. How accurate is your classifier?

```R
correct.1 <- NULL
for (i in c(1:total)) {
    if (dist.1[i] <= mahadist(c(X.1[i], Y.1[i]), c(-4.0, 0), c(0.5, 1.5))) {
        correct.1[i] = TRUE
    } else {
        correct.1[i] = FALSE
    }
}

correct.2 <- NULL
for (i in c(1:total)) {
    if (dist.2[i] <= mahadist(c(X.2[i], Y.2[i]), c(0, 0), c(1.5, 0.5))) {
        correct.2[i] = TRUE
    } else {
        correct.2[i] = FALSE
    }
}
```

For the *black* distribution, the accuracy is:

```R
sum(correct.1) / total
```

0.972

For the *blue* distribution, the accuracy is:

```R
sum(correct.2) / total
```

0.994

## Question E

> In order to visualise the feature space, plot a grid of points on the current graph covering the whole space. Compute the *Mahalanobis Distances* at each point.
>
> Plot a *black* circle if the point is closer to the *black* distribution and *blue* point if it is closer to the *blue* distribution.

```R
loc.1.x <- NULL
loc.1.y <- NULL
loc.2.x <- NULL
loc.2.y <- NULL

for (x in c(-10:10)) {
    for (y in c(-10:10)) {
        dist.1 <- mahadist(c(x, y), c(0, 0), c(1.5, 0.5))
        dist.2 <- mahadist(c(x, y), c(-4.0, 0), c(0.5, 1.5))
        if (dist.1 <= dist.2) {
            loc.1.x[length(loc.1.x) + 1] <- x
            loc.1.y[length(loc.1.y) + 1] <- y
        } else {
            loc.2.x[length(loc.2.x) + 1] <- x
            loc.2.y[length(loc.2.y) + 1] <- y
        }
    }
}

plot(loc.1.y ~ loc.1.x, xlim=c(-10, 10), ylim=c(-10, 10), xlab="X", ylab="Y")
points(loc.2.y ~ loc.2.x, col="blue")
```

exercises/Multivariate Normal Distribution - Mahalanobis Distance.ipynb


# 测试文件

在此对编写的各个函数进行测试，检查是否可以正常运行，首先加载base文件里面定义的各种函数。

```python
%run base.py
```

```python
from sympy import init_printing
init_printing()
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
```

## 基本梯度功能

在此可以求梯度与Hessian矩阵，例子使用

$$
x_1^2+x_2\times cos(x_2)
$$

```python
xvec = symbols('x1:3')  # 定义变量x1,x2
fexpr = xvec[0]**2+xvec[1]*cos(xvec[1])
gexpr = get_g(fexpr, xvec, display=True)
print('-'*40)
Gexpr = get_G(fexpr, xvec, display=True)
```

```python
g = get_fun(gexpr,xvec)
g.__name__
g(1, 1)
```

## 黄金分割法测试

目标函数为：

$$
1-x\times e^{-x^2}
$$

在$\sqrt{2}/2$取到解析解

```python
%%time
def foo(x): return 1-x*np.exp(-x**2)
print('0.618法',
    golden_section_search(foo, 0.01, interval=[0, 1], printout=True),
    '解析解', np.sqrt(0.5))
```

## 修正牛顿法在正定二次函数情况下的测试

函数形式为：

$$
x_1^2+x_2^2+x_1\cdot x_2
$$

在(0,0)取到最小值

```python
%%time
xvec = symbols('x1:3')
fexpr = xvec[0]**2+xvec[1]**2+1*xvec[0]*xvec[1]  # 正定的二次函数
pprint(fexpr)
x = modified_newton(fexpr, xvec, (1, 1), eps=1e-5)
print('结果：', x)
```

## 阻尼牛顿法在正定二次函数上的测试

函数形式同上

```python
%%time
fexpr = xvec[0]**2+xvec[1]**2+1*xvec[0]*xvec[1]  # 正定的二次函数
pprint(fexpr)
x = damped_newton(fexpr, xvec, (1, 1), eps=1e-5)
print('结果：', x)
```