Sascha Spors,
Professorship Signal Theory and Digital Signal Processing,
Institute of Communications Engineering (INT),
Faculty of Computer Science and Electrical Engineering (IEF),
University of Rostock,
Germany

# Tutorial Digital Signal Processing

**DFT Fundamentals**,
Winter Semester 2021/22 (Course #24505)

- lecture: https://github.com/spatialaudio/digital-signal-processing-lecture
- tutorial: https://github.com/spatialaudio/digital-signal-processing-exercises

Feel free to contact lecturer frank.schultz@uni-rostock.de

```python
import numpy as np
import timeit

from matplotlib import pyplot as plt
from numpy.fft import fft, ifft
#from scipy.fft import fft, ifft
from scipy.special import diric
```

These exercises accompany the [dft_windowing_tutorial.pdf](dft_windowing_tutorial.pdf)

## Exercise 1: My Own DFT / IDFT Implementation

Write Python functions `X = my_dft(x)` and `x = my_idft(X)` that calculate the DFT pair

\begin{align}
X[\mu] = \mathrm{DFT}_N\{x[k]\} = &\sum_{k=0}^{N-1} x[k] \, (\mathrm{e}^{-\mathrm{j}\frac{2\pi}{N}})^{k \cdot \mu}
\\
x[k] = \mathrm{IDFT}_N\{X[\mu]\} = \frac{1}{N} &\sum_{\mu=0}^{N-1} X[\mu] \, (\mathrm{e}^{+\mathrm{j}\frac{2\pi}{N}})^{k \cdot \mu}
\end{align}
without using the pre-built functions `fft()` and `ifft()`. Check validity and performance against the built-in functions, especially try large $N$. We might consider the matrix operation approach rather than a for-loop implementation, given that the $N \times N$ matrix fits into our RAM.

```python
def get_fourier_matrix(N):
    k = np.arange(N)
    return np.exp(+1j*2*np.pi/N*np.outer(k, k))

def my_dft(x):
    N = x.size
    return np.conj(get_fourier_matrix(N))@x

def my_idft(X):
    N = X.size
    return 1/N*get_fourier_matrix(N)@X
```

```python
N = 2**8
```

Check `my_dft` against numpy's `fft` with numpy's `allclose`

```python
# create random complex valued signal
x = np.random.randn(N) + 1j * np.random.randn(N)
# check against numpy fft
np.allclose(my_dft(x), fft(x))
```

Check `my_idft` against numpy's `ifft` with numpy's `allclose`

```python
# create random complex valued spectrum
X = np.random.randn(N) + 1j * np.random.randn(N)
# check against numpy ifft
np.allclose(my_idft(X), ifft(X))
```

check processing time as `dur`

```python
N_rep = 2**8  # play around with N and N_rep to see that my_xxx() functions
# are very slow for large N, i.e. our matrix multiplication generates very high
# computing load compared to the FFT / IFFT algorithm used in numpy.fft package

dur = timeit.timeit(
    'my_dft(x)', 'from __main__ import my_dft, x', number=N_rep)
print('my dft:', dur)

dur = timeit.timeit(
    'fft(x)', 'from __main__ import fft, x', number=N_rep)
print('numpy fft:', dur)

dur = timeit.timeit(
    'my_idft(X)', 'from __main__ import my_idft, X', number=N_rep)
print('my idft:', dur)

dur = timeit.timeit(
    'ifft(X)', 'from __main__ import ifft, X', number=N_rep)
print('numpy ifft', dur)
```

We see, that in practice we should better use the `fft()` and `ifft()` rourines rather than our own implementations  that exhibit comparably huge computational load.

## Exercise 2: IDFT as Analytic Calculus / as Linear Vector Combination

The discrete-time signal
\begin{equation}
x[k]=-2\cdot\sin\left(\frac{2\pi}{4}k\right)+3\cdot\cos\left(\frac{2\pi}{4}\cdot2k\right)+1
\hspace{5mm}\text{for}\,\,0\leq k\leq3
\end{equation}
with $k\in\mathbb{Z}$ is given.

**a)**

Calculate the resulting values of $x[k]$ for $0\leq k\leq3$.

**Solution**

\begin{align}
x[0]&=-2\cdot\sin\left(2\pi\frac{1}{4}\cdot0\right)+3\cdot\cos\left(2\pi\frac{2}{4}\cdot0\right)+1=3+1=4\\
x[1]&=-2\cdot\sin\left(2\pi\frac{1}{4}\cdot1\right)+3\cdot\cos\left(2\pi\frac{2}{4}\cdot1\right)+1=-2-3+1=-4\\
x[2]&=-2\cdot\sin\left(2\pi\frac{1}{4}\cdot2\right)+3\cdot\cos\left(2\pi\frac{2}{4}\cdot2\right)+1=3+1=4\\
x[3]&=-2\cdot\sin\left(2\pi\frac{1}{4}\cdot3\right)+3\cdot\cos\left(2\pi\frac{2}{4}\cdot3\right)+1=2-3+1=0
\end{align}

```python
N = 4
k = np.arange(N)
x = -2*np.sin(2*np.pi/N*k) + 3*np.cos(2*np.pi/N*2*k) + 1
print('x=', x)
```

**b)**

Show analytically that the given values of $X[\mu]$, $\mu\in\mathbb{Z}$:
\begin{equation}
X[\mu=0]=4,\quad X[\mu=1]=4\mathrm{j},\quad  X[\mu=2]=12
\end{equation}
are the DFT coefficients of $x[k]$ stemming from our used DFT equation
\begin{equation}
X[\mu]=\sum_{k=0}^{N-1}x[k]\cdot\mathrm{e}^{-\mathrm{j}\frac{2\pi}{N}k\mu}
\end{equation}
with $N=4$.

The following procedure is suggested: Set up the spectral
coefficients $X[\mu]$ in the form

\begin{equation}
X[\mu]=A[\mu]\cdot\mathrm{e}^{\mathrm{j}\phi[\mu]}
\end{equation}

and specify the missing value $X[\mu=3]$ so that the IDFT results in
$x[k]\in\mathbb{R}$. Certain symmetries in the spectrum hold.

Then calculate the IDFT analytically as
\begin{equation}
x[k]=\frac{1}{N}\sum_{\mu=0}^{N-1}X[\mu]\cdot\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}k\mu}
\end{equation}
showing that this corresponds to the given signal $x[k]$.

Also, check the results numerically. Store the DFT spectrum in `X` for future reference.

**Solution**

For manual calculus see [dft_windowing_tutorial.pdf](dft_windowing_tutorial.pdf).

```python
mu = np.arange(N)
W = np.exp(+1j*2*np.pi/N*np.outer(k, mu))  # Fourier matrix
with np.printoptions(precision=3, suppress=True):
    print('Fourier matrix W =\n', W)

X = fft(x)

# DFT as matrix mul
X1 = np.conj(W)@x

# DFT as individual complex inner products using Fourier matrix column vectors
X2 = np.zeros(N, dtype='complex')
for col in range(N):
    X2[col] = np.vdot(W[:, col], x)

with np.printoptions(precision=3, suppress=True):
    print('X = ', X)
    print('X1 == X2:', np.allclose(X1, X2))
    print('X1 == X:', np.allclose(X1, X))
```

**c)**

Plot the real and imaginary part as well as the magnitude and the phase of $X[\mu]$ over $\mu$.

```python
plt.figure(figsize=(7, 4.375))
plt.subplot(2, 2, 1)
plt.plot(mu, X.real, 'C0o', ms=7)
plt.xlabel(r'$\mu$')
plt.ylabel(r'$\Re(X)$')
plt.grid(True)
plt.subplot(2, 2, 2)
plt.plot(mu, X.imag, 'C0o', ms=7)
plt.xlabel(r'$\mu$')
plt.ylabel(r'$\Im(X)$')
plt.grid(True)
plt.subplot(2, 2, 3)
plt.plot(mu, np.abs(X), 'C0o', ms=7)
plt.xlabel(r'$\mu$')
plt.ylabel(r'$|X|$')
plt.grid(True)
plt.subplot(2, 2, 4)
plt.plot(mu, np.angle(X), 'C0o', ms=7)
plt.xlabel(r'$\mu$')
plt.ylabel(r'$\angle X$ / rad')
plt.grid(True)
```

**d)**

Remove the DC component in the DFT spectrum `X` and from that spectrum synthesise the signal $x_r[k]$ via IDFT.

Check that the synthesised signal corresponds to
\begin{equation}
x_r[k]=-2\cdot\sin\left(\frac{2\pi}{4}k\right)+3\cdot\cos\left(\frac{2\pi}{4}\cdot2k\right)
\hspace{5mm}\text{for}\,\,0\leq k\leq3
\end{equation}

by help of a linear combination using the Fourier matrix.

```python
X[0] = 0  # remove DC

# IDFT as matrix mul
xr1 = 1/N * W@X

# IDFT as linear combination of the Fourier matrix column vectors, i.e. DFT signals
xr2 = 1/N * (X[0]*W[:, 0] + X[1]*W[:, 1] + X[2]*W[:, 2] + X[3]*W[:, 3])

xr_check = -2*np.sin(2*np.pi/N*k) + 3*np.cos(2*np.pi/N*2*k)
print(np.allclose(xr1, xr_check))
print(np.allclose(xr2, xr_check))
```

**e)**

A DFT-based audio analyser shall exhibit a frequency resolution of
$\Delta f=0.5$ Hz for a sampling frequency $f_s=44100$ Hz using a rectangular
window.

Determine the minimum required DFT length $N$ when only lengths $N=2^M$
($M\in\mathbb{N}$) are allowed.

What is the resulting frequency resolution then?

**Solution**

\begin{align}
\Delta f=\frac{f_s}{N}=\frac{f_s}{2^M}\stackrel{!}{=}0.5\,\text{Hz}\\
M=\left\lceil\log_{2}\left(\frac{f_s}{\Delta f}\right)\right\rceil_{\in\mathbb{N}}\\
=\left\lceil\log_{2}\left(\frac{44100\,\text{Hz}}{0.5\,\text{Hz}}\right)\right\rceil_{\in\mathbb{N}}\\
=\left\lceil16.4285...\right\rceil_{\in\mathbb{N}}\\
=17\\
\Rightarrow\hspace{5mm} N=2^M=2^{17}=131072
\end{align}
The resulting frequency resolution is thus
\begin{equation}
\Delta f=\frac{f_s}{N}=\frac{44100\,\text{Hz}}{131072}\approx0.3365\,\text{Hz}.\nonumber
\end{equation}

```python
df = 0.5
fs = 44100
M = (np.ceil(np.log10(fs/df) / np.log10(2)))
Ntmp = 2**M
print('N =', Ntmp.astype(int))
print('df =', fs/Ntmp.astype(int), 'Hz')
```

## Exercise 3: DFT Analysis Using a Rectangular Window / DTFT Interpolation

The DFT spectrum $X[\mu]$ can used for interpolation towards the DTFT spectrum $X(\Omega)$ with

\begin{equation}
X(\Omega)=\sum_{\mu=0}^{N-1}X[\mu]\cdot\mathrm{e}^{-\mathrm{j}\frac{\left(\Omega-\frac{2\pi}{N}\mu\right)(N-1)}{2}}\cdot\text{psinc}_N\left(\Omega-\frac{2\pi}{N}\mu\right).
\end{equation}

The interpolation kernel utilises the so-called **periodic sinc function**

\begin{align}
\text{psinc}_N(\Omega)=\begin{cases}\frac{1}{N}\cdot\frac{\sin\left(\frac{N}{2}\Omega\right)}{\sin\left(\frac{1}{2}\Omega\right)}&\text{for }\Omega\neq2\pi m\\
(-1)^{m(N-1)}&\text{for }\Omega=2\pi m\end{cases},\,\,m\in\mathbb{Z},
\end{align}

which is also known as aliased sinc and Dirichlet function.
This interpolation implies:
- the DFT $X[\mu]$ stems from a signal $x[k]$ for which periodicity of $N$ is inherent, we assume the first period at $0\leq k\leq N-1$
- the DFT spectrum is discrete and $N$ periodic
- the DTFT $X(\Omega)$ stems from the signal $x[k]$ for $0\leq k\leq N-1$, for all other $k$ is $x[k]=0$, i.e. considering only one period
- the DTFT spectrum is continuous and $2\pi$ periodic, which means that the signal $x[k]$ can not be periodic

**Task**

A cosine signal $x[k]=\cos(\Omega k)$ with $0\leq k \leq N-1$ is to be analysed with the DFT using a rectangular window and assuming sampling frequency of $f_s=48$ kHz.

For the parameters

- $\Omega=2\cdot\frac{2\pi}{N}$, $N=8$
- $\Omega=2.5\cdot\frac{2\pi}{N}$, $N=8$
- $\Omega=2\cdot\frac{2\pi}{N}$, $N=9$
- $\Omega=2.5\cdot\frac{2\pi}{N}$, $N=9$

perform the following subtasks:

**a)**

Calculate the spectrum $X[\mu]$ of $x[k]$ and visualise the real and
imaginary part as well as the magnitude and the phase of $X[\mu]$ over
$0\leq\mu\leq N-1$.

**b)**

Check the expected axial and point symmetries.

**c)**

Implement the DTFT interpolation equation above and visualise the $|X[\mu]|$ and $|X(\Omega)|$ over the frequency axes $\mu$, $\Omega$ and $f$.

How and why the results are different?

**Solution**

It is meaningful to code two functions for repeated calculations, the first is the DFT to DTFT interpolation, the second function creates the signal and calculates the DFT, DTFT and plots the requested graphs.

```python
def dft2dtft(X, Om):
    # DFT spectrum X
    # digital angular freqency Om for which DTFT shall be evaluated
    # DTFT Xi
    N = len(X)
    Xi = np.zeros(len(Om), dtype='complex')
    for i, Omega in enumerate(Om):
        for mu_dft in range(N):
            Xi[i] += X[mu_dft] *\
                np.exp(-1j/2*(Omega - 2*np.pi/N*mu_dft)*(N-1)) *\
                diric(Omega - 2*np.pi/N*mu_dft, N)
    return Xi
```

```python
def dft2dtft_playground_plot(N=8, Om_sig=2,
                             Om=np.linspace(0, 2*np.pi, 2**8), fs=48000):
    mu = np.arange(N)
    k = mu
    x = np.cos(Om_sig*k)  # indicated symmetry holds for real input signals!
    dOm = 2*np.pi/N
    X_dft = fft(x)
    X_dtft = dft2dtft(X_dft, Om)

    plt.figure(figsize=(7, 4.375))
    plt.subplot(2, 2, 1)
    plt.plot(mu, X_dft.real, 'C0o', ms=7)
    plt.plot([N/2, N/2], [-N/2, N/2], 'C3:')  # indicate axial symmetry
    plt.xlim(0, N-1)
    plt.ylim(-N/2, N/2)
    plt.xlabel(r'$\mu$')
    plt.ylabel(r'$\Re(X)$')
    plt.title('axial symmetry wrt red line')
    plt.grid(True)
    plt.subplot(2, 2, 2)
    plt.plot(mu, X_dft.imag, 'C0o', ms=7)
    plt.plot([N/2], [0], 'C3o')  # indicate point symmetry
    plt.xlim(0, N-1)
    plt.ylim(-N/2, N/2)
    plt.xlabel(r'$\mu$')
    plt.ylabel(r'$\Im(X)$')
    plt.title('point symmetry wrt red point')
    plt.grid(True)
    plt.subplot(2, 2, 3)
    plt.plot(mu, np.abs(X_dft), 'C0o', ms=7)
    plt.plot([N/2, N/2], [0, N/2], 'C3:')  # indicate axial symmetry
    plt.xlim(0, N-1)
    plt.ylim(0, N/2)
    plt.xlabel(r'$\mu$')
    plt.ylabel(r'$|X|$')
    plt.grid(True)
    plt.subplot(2, 2, 4)
    plt.plot(mu, np.angle(X_dft), 'C0o', ms=7)
    plt.plot([N/2], [0], 'C3o')  # indicate point symmetry
    plt.xlim(0, N-1)
    plt.ylim(-np.pi, np.pi)
    plt.xlabel(r'$\mu$')
    plt.ylabel(r'$\angle X$ / rad')
    plt.grid(True)

    plt.figure(figsize=(7, 9))
    plt.subplot(3, 1, 1)
    plt.plot(mu, np.abs(X_dft), 'C0o', ms=7, label='DFT')
    plt.plot(Om/dOm, np.abs(X_dtft), 'C1', label='DTFT')
    plt.xlim(0, N)
    plt.xlabel(r'$\mu$')
    plt.title('magnitude spectrum |X|')
    plt.legend()
    plt.grid(True)
    plt.subplot(3, 1, 2)
    plt.plot(mu*dOm, np.abs(X_dft), 'C0o', ms=7)
    plt.plot(Om, np.abs(X_dtft), 'C1')
    plt.xlim(0, 2*np.pi)
    plt.xlabel(r'$\Omega$ / rad')
    plt.grid(True)
    plt.subplot(3, 1, 3)
    plt.plot(mu*fs/N, np.abs(X_dft), 'C0o', ms=7)
    plt.plot(Om*fs/2/np.pi, np.abs(X_dtft), 'C1')
    plt.xlim(0, fs)
    plt.xlabel(r'$f$ / Hz with fs='+str(fs/1000)+'kHz')
    plt.grid(True)
```

```python
N = 8
Om_sig = 2*2*np.pi/N
dft2dtft_playground_plot(N=N, Om_sig=Om_sig)
```

```python
N = 9
Om_sig = 2*2*np.pi/N
dft2dtft_playground_plot(N=N, Om_sig=Om_sig)
```

```python
N = 8
Om_sig = 2.5*2*np.pi/N
dft2dtft_playground_plot(N=N, Om_sig=Om_sig)
```

```python
N = 9
Om_sig = 2.5*2*np.pi/N
dft2dtft_playground_plot(N=N, Om_sig=Om_sig)
```

## Exercise 4: DFT / DTFT of an Finite Impulse Response (FIR)

The finite length impulse response (FIR) $h[k]$ of an LTI system is given
as

\begin{equation}
h[k]=\frac{1}{8}\cdot\left(11 \delta[k]-5 \delta[k-1]+7 \delta[k-2]-9 \delta[k-3]\right).
\end{equation}

Obviously, the LTI system is non-recursive.

The magnitude and phase spectrum is to be evaluated from
- DFT
- DTFT
- DFT of zerpadded impulse response

Discuss the different approaches.

```python
h = np.array([11, -5, 7, -9]) / 8  # FIR

# DFT
Nh = h.size  # length of FIR
H_dft = fft(h)  # DFT of FIR
mu = np.arange(Nh)  # DFT frequency vector mu

with np.printoptions(precision=3, suppress=True):
    print('at DFT frequencies:')
    print('level', 20*np.log10(np.abs(H_dft)), 'dB')
    print('phase', np.angle(H_dft)*180/np.pi, 'deg')

# interpolation towards DTFT
NOm = 2**8  # DTFT frequencies to be plotted
dOm = 2*np.pi/NOm  # resolution
Om = np.arange(NOm)*dOm
H_dtft = dft2dtft(fft(h), Om)

# interpolation via zeropadding and DFT
hz = np.zeros(NOm)  # zero signal
hz[0:4] = h  # insert h yields zeropadded FIR in hz
Hz_dft = fft(hz)  # DFT of zeropadded FIR

plt.figure(figsize=(7,4.375))
plt.subplot(2,1,1)
plt.plot(mu*2*np.pi/Nh, np.abs(H_dft), 'C0o', label=r'DFT of $h[k]$')
plt.plot(Om, np.abs(H_dtft), 'C1', label=r'DTFT of $h[k]$')
plt.plot(Om, np.abs(Hz_dft), 'C3-.', label=r'DFT of zeropadded IR $h_z[k]$')
plt.xlim(0, 2*np.pi)
plt.ylim(0)
#plt.xlabel(r'$\Omega$ / rad')
plt.ylabel(r'magnitude |H|')
plt.legend()
plt.grid(True)
plt.subplot(2,1,2)
plt.plot(mu*2*np.pi/Nh, np.angle(H_dft)*180/np.pi, 'C0o', label=r'DFT of $h[k]$')
plt.plot(Om, np.angle(H_dtft)*180/np.pi, 'C1', label=r'DTFT of $h[k]$')
plt.plot(Om, np.angle(Hz_dft)*180/np.pi, 'C3-.', label=r'DFT of zeropadded IR $h_z[k]$')
plt.xlim(0, 2*np.pi)
plt.xlabel(r'$\Omega$ / rad')
plt.ylabel(r'phase $\angle H$ in deg')
plt.grid(True)
```

**Copyright**

The notebooks are provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebooks for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Frank Schultz, Digital Signal Processing - A Tutorial Featuring Computational Examples* with the URL https://github.com/spatialaudio/digital-signal-processing-exercises


# 13 Root Finding

An important tool in the computational tool box is to find roots of equations for which no closed form solutions exist:

We want to find the roots $x_0$ of

$$
f(x_0) = 0
$$

## Problem: Projectile range
The equations of motion for the projectile with linear air resistance (see *12 ODE applications*) can be solved exactly.

As a reminder: the linear drag force is
$$
\mathbf{F}_1 = -b_1 \mathbf{v}\\
b := \frac{b_1}{m}
$$

Equations of motion with force due to gravity $\mathbf{g} = -g \hat{\mathbf{e}}_y$

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= - g \hat{\mathbf{e}}_y -b \mathbf{v}
\end{align}

### Analytical solution of the equations of motion
(Following Wang Ch 3.3.2)

Solve $x$ component of the velocity

$$
\frac{dv_x}{dt} = -b v_x
$$

by integration:

$$
v_x(t) = v_{0x} \exp(-bt)
$$

The drag force reduces the forward velocity to 0.

Integrate again to get the $x(t)$ component

$$
x(t) = x_0 + \frac{v_{0x}}{b} \left[1 - \exp(-bt)\right]
$$

Integrating the $y$ component of the velocity

$$
\frac{dv_y}{dt} = -g - b v_y
$$

gives

$$
v_y = \left(v_{0y} + \frac{g}{b}\right) \exp(-bt) - \frac{g}{b}
$$

and integrating again

$$
y(t) = y_0 + \frac{v_{0y} + \frac{g}{b}}{b} \left[1 - \exp(-bt)\right] - \frac{g}{b} t
$$

(Note: This shows immediately that the *terminal velocity* is

$$
\lim_{t\rightarrow\infty} v_y(t) = - \frac{g}{b},
$$

i.e., the force of gravity is balanced by the drag force.)

#### Analytical trajectory

To obtain the **trajectory $y(x)$** eliminate time (and for convenience, using the origin as the initial starting point, $x_0 = 0$ and $y_0 = 0$. Solve $x(t)$ for $t$

$$
t = -\frac{1}{b} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

and insert into $y(t)$:

$$
y(x) = \frac{x}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

#### Plot

Plot the analytical solution $y(x)$ for $\theta = 30^\circ$ and $v_0 = 100$ m/s.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
```

The function `y_lindrag()` should compute $y(x)$.

```python
def y_lindrag(x, v0, b1=0.2, g=9.81, m=0.5):
    b = b1/m
    v0x, v0y = v0
    return x/v0x * (v0y + g/b) + g/(b*b) * np.log(1 - b*x/v0x)

def initial_v(v, theta):
    x = np.deg2rad(theta)
    return v * np.array([np.cos(x), np.sin(x)])
```

The analytical function drops *very* rapidly towards the end ($> 42$ m – found by manual trial-and-error plotting...) so in order to nicely plot the function we use a fairly coarse sampling of points along $x$ for the range $0 \le x < 42$ and very fine sampling for the last 2 m ($42 \le x < 45$):

```python
X = np.concatenate([np.linspace(0, 42, 100), np.linspace(42, 45, 10000)])
```

Evaluate the function for all $x$ values:

```python
Y = y_lindrag(X, initial_v(100, 30), b1=1)
```

    /Users/oliver/anaconda3/envs/phy494/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in log
      after removing the cwd from sys.path.

(The warning can be ignored, it just means that some of our `X` values were not approriate and outside the range of validity – when the argument of the logarithm becomes ≤0).

To indicate the ground we also plot a dashed black line: note that the analytical solution goes below the dashed line.

```python
plt.plot(X, Y)
plt.xlabel("$x$ (m)")
plt.ylabel("$y$ (m)")
plt.hlines([0], X[0], X[-1], colors="k", linestyles="--");
```

Compare to the numerical solution (from **12 ODE Applications**):

```python
import ode

def simulate(v0, h=0.01, b1=0.2, g=9.81, m=0.5):
    def f(t, y):
        # y = [x, y, vx, vy]
        return np.array([y[2], y[3], -b1/m * y[2], -g - b1/m * y[3]])

    vx, vy = v0
    t = 0
    positions = []
    y = np.array([0, 0, vx, vy], dtype=np.float64)

    while y[1] >= 0:
        positions.append([t, y[0], y[1]])  # record t, x and y
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)
```

```python
r = simulate(initial_v(100, 30), h=0.01, b1=1)
```

```python
plt.plot(X, Y, lw=2, label="analytical")
plt.plot(r[:, 1], r[:, 2], '--', label="RK4")
plt.legend(loc="best")
plt.xlabel("$x$ (m)"); plt.ylabel("$y$ (m)")
plt.hlines([0], X[0], X[-1], colors="k", linestyles="--");
```

The RK4 solution tracks the analytical solution perfectly (and we also programmed it to not go below ground...)

OPTIONAL: Show the residual

$$
r = y_\text{numerical}(x) - y_\text{analytical}
$$

```python
residual = r[:, 2] - y_lindrag(r[:, 1], initial_v(100, 30), b1=1)
```

```python
plt.plot(r[:, 1], residual)
plt.xlabel("$x$ (m)"); plt.ylabel("residual $r$ (m)");
```

### Predict the range $R$
How far does the ball or projectile fly, i.e., that value $x=R$ where $y(R) = 0$:

$$
\frac{R}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bR}{v_{0x}}\right) = 0
$$

This *transcendental equation* can not be solved in terms of elementary functions.

Use a **root finding** algorithm.

## Root-finding with the Bisection algorithm
**Bisection** is the simplest (but very robust) root finding algorithm that uses trial-and-error:

* bracket the root
* refine the brackets
* see [13_Root-finding-algorithms (PDF)](13_Root-finding-algorithms.pdf)

More specifically:
1. determine a bracket that contains the root: $a < x_0 < b$ (i.e., an interval $[a, b]$ with $f(a) > 0$ and $f(b) < 0$ or $f(a) < 0$ and $f(b) > 0$)
2. cut bracket in half: $x' = \frac{1}{2}(a + b)$
3. determine in which half the root lies: either in $[a, x']$ or in $[x', b]$: If $f(a) f(x') > 0$ then the root lies in the right half $[x', b]$, otherwise the left half $[a, x']$.
4. Change the boundaries $a$ or $b$.
5. repeat until $|f(x')| < \epsilon$.

### Implementation of Bisection

- Test that the initial bracket contains a root; if not, return `None` (and possibly print a warning).
- If either of the bracket points is a root then return the bracket point.
- Allow `Nmax` iterations or until the convergence criterion `eps` is reached.
- Bonus: print a message if no root was found after `Nmax` iterations, but print the best guess and the error (but return `None`).

```python
def bisection(f, a, b, Nmax=100, eps=1e-14):
    fa, fb = f(a), f(b)
    if (fa*fb) > 0:
        print("bisect: Initial bracket [{0}, {1}] "
              "does not contain a single root".format(a, b))
        return None
    if np.abs(fa) < eps:
        return a
    if np.abs(fb) < eps:
        return b

    for iteration in range(Nmax):
        x = (a + b)/2
        fx = f(x)
        if f(a) * fx > 0:
            # root is not between a and x
            a = x
        else:
            b = x
        if np.abs(fx) < eps:
            break
    else:
        print("bisect: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x
```

### Finding the range with the bisection algorithm

Define the trial function `f`.

Note that our `y_lindrag()` function depends on `x` **and** `v` but `bisect()` only accepts functions `f` that depend on a *single variable*, $f(x)$. We therefore have to wrap `y_lindrag(x, v)` into a function `f(x)` that sets `v` already to a value *outside* the function: [Python's scoping rules](https://stackoverflow.com/questions/291978/short-description-of-the-scoping-rules#292502) say that inside the function `f(x)`, the variable `x` has the value assigned to the argument of `f(x)` but any other variables such as `v` or `b1`, which were *not defined inside `f`*, will get the value that they had *outside `f`* in the *enclosing code*.

```python
def f(x):
    v0 = initial_v(100, 30)
    b1 = 1.
    return y_lindrag(x, v0, b1=b1)
```

The initial bracket $[a_\text{initial}, b_\text{initial}]$ is a little bit difficult for this function: choose the right bracket near the point where the argument of the logarithm becomes 0 (which is actually the maximum $x$ value $\lim_{t\rightarrow +\infty} x(t) = \frac{v_{0x}}{b}$):

$$
b_\text{initial} = \frac{v_{0x}}{b} - \epsilon'
$$

where $\epsilon'$ is a small number.

```python
v = initial_v(100, 30)

b1 = 1.
m = 0.5
b = b1/m
bisection(f, 0.1, v[0]/b - 1e-12, eps=1e-6)
```

    43.30067423347077

Note that this solution is *not* the maximum value $\lim_{t\rightarrow +\infty} x(t) = \frac{v_{0x}}{b}$:

```python
v[0]/b
```

    43.30127018922194

### Find the range as a function of the initial angle

```python
b1 = 1.
m = 0.5
b = b1/m
v0 = 100
u = []
for theta in np.arange(1, 90):
    v = initial_v(v0, theta)
    def f(x):
        return y_lindrag(x, v, b1=b1)
    R = bisection(f, 0.1, v[0]/b - 1e-16, eps=1e-5)
    if R is not None:
        u.append((theta, R))
u = np.array(u)
```

    /Users/oliver/anaconda3/envs/phy494/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: divide by zero encountered in log
      after removing the cwd from sys.path.

```python
plt.plot(u[:, 0], u[:, 1])
plt.xlabel(r"launch angle $\theta$ ($^\circ$)")
plt.ylabel(r"range $R$ (m)");
```

Write a function `find_range()` to calculate the range for a given initial velocity $v_0$ and plot $R(\theta)$ for $10\,\text{m/s} ≤ v_0 ≤ 100\,\text{m/s}$.

```python
def find_range(v0, b1=1, m=0.5):
    b = b1/m
    u = []
    for theta in np.arange(1, 90):
        v = initial_v(v0, theta)
        def f(x):
            return y_lindrag(x, v, b1=b1)
        R = bisection(f, 0.1, v[0]/b - 1e-16, eps=1e-5)
        if R is not None:
            u.append((theta, R))
    return np.array(u)
```

```python
for v0 in (10, 25, 50, 75, 100):
    u = find_range(v0)
    plt.plot(u[:, 0], u[:, 1], label="{} m/s".format(v0))
plt.legend(loc="best")
plt.xlabel(r"$\theta$ (degrees)")
plt.ylabel(r"range $R$ (m)");
```

As a bonus, find the dependence of the *optimum launch angle* on the initial velocity, i.e., that angle that leads to the largest range:

```python
np.argmax(u[:, 1])
```

    10

```python
velocities = np.linspace(5, 100, 100)
results = []  # (v0, theta_opt)
for v0 in velocities:
    u = find_range(v0)
    thetas, ranges = u.transpose()
    # find index for the largest range and pull corresponding theta
    theta_opt = thetas[np.argmax(ranges)]
    results.append((v0, theta_opt))
results = np.array(results)

plt.plot(results[:, 0], results[:, 1])
plt.xlabel(r"velocity $v_0$ (m/s)")
plt.ylabel(r"$\theta_\mathrm{best}$ (degrees)");
```

The launch angle decreases with the velocity.

The steps in the graph are an artifact of choosing to only calculate the trajectories for integer angles (see `for theta in np.arange(1, 90)` in `find_range()`).

## Newton-Raphson algorithm
(see derivation in class and in the PDF or [Newton's Method](http://mathworld.wolfram.com/NewtonsMethod.html) on MathWorld)

### Activity: Implement Newton-Raphson
1. Implement the Newton-Raphson algorithm
2. Test with $g(x)$.

   $$
   g(x) = 2 \cos x - x
   $$

3. Bonus: test performance of `newton_raphson()` against `bisection()`.

```python
def g(x):
    return 2*np.cos(x) - x
```

```python
xvals = np.linspace(0, 7, 30)
plt.plot(xvals, np.zeros_like(xvals), 'k--')
plt.plot(xvals, g(xvals))
```

```python
def newton_raphson(f, x, h=1e-3, Nmax=100, eps=1e-14):
    """Find root x0 so that f(x0)=0 with the Newton-Raphson algorithm"""
    for iteration in range(Nmax):
        fx = f(x)
        if np.abs(fx) < eps:
            break
        df = (f(x + h/2) - f(x - h/2))/h
        Delta_x = -fx/df
        x += Delta_x
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x
```

```python
x0 = newton_raphson(g, 2)
print(x0)
```

```python
g(x0)
```

But note that the algorithm only converges well near the root. With other values it might not converge at all:

```python
newton_raphson(g, 3)
```

```python
newton_raphson(g, 10)
```

```python
newton_raphson(g, 15)
```

Let's look how Newton-Raphson iterates: also return all intermediate $x$ values:

```python
def newton_raphson_with_history(f, x, h=1e-3, Nmax=100, eps=1e-14):
    xvals = []
    for iteration in range(Nmax):
        fx = f(x)
        if np.abs(fx) < eps:
            break
        df = (f(x + h/2) - f(x - h/2))/h
        Delta_x = -fx/df
        x += Delta_x
        xvals.append(x)
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x, np.array(xvals)
```

```python
x = {}

x0, xvals = newton_raphson_with_history(g, 1.5)
x[1.5] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))

x0, xvals = newton_raphson_with_history(g, 5)
x[5] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))

x0, xvals = newton_raphson_with_history(g, 10)
x[10] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))
```

```python
for xstart in sorted(x.keys()):
    plt.semilogx(x[xstart], label=str(xstart))
plt.legend(loc="best")
plt.xlabel("iteration")
plt.ylabel("current guess for root $x_0$");
```


# Problem Set 1
# 1D Finite Element Method Consistency

This notebook is available on [GitHub](https://github.com/giovaniceotto/Spectral-hp-Element-Methods-for-CFD/blob/main/Problem%20Set%201/1D%20Finite%20Element%20Method%20Consistency.ipynb).

You can run it interactively using [Google Colab](https://colab.research.google.com/github/giovaniceotto/Spectral-hp-Element-Methods-for-CFD/blob/main/Problem%20Set%201/PSet%201.ipynb). Keep in mind that equation numbering is not yet supported by Colab.

## Problem Description

Consider the differential equation

\begin{equation}
    \frac{d^2 u}{dx^2} + 5u = 3.5,
    \label{eq:differential_equation}
\end{equation}

defined in the domain $\Omega = \left\{x | 0 \le x \le 10\right\}$, subject to the following boundary conditions,

\begin{equation}
    u(x=0) =1.2, \quad \left.\frac{du}{dx}\right\rvert_{x=10} = 0.5.
    \label{eq:boundary_conditions}
\end{equation}

## Item A - Analytical Solution

### Finding the general solution

The differential equation given above is a second-order linear non-homogeneous differential equation.
It's solution is given as the sum of a particular solution and the solution to it's homogeneous form.

To find the homogeneous solution, the following *ansatz* is used:

$$ u_h (x) = e^{sx}, $$

where $s$ is a constant. Substituting the *ansatz* into the homogeneous version of the differential equation results in:

$$ s^2 u_h(x) + 5 u_h(x) = 0 \Rightarrow s = \pm i \sqrt{5}. $$

The particular solution can be the constant function that follows:

$$ u_p = \frac{3.5}{5}. $$

Thus, the general solution is given by the linear combination of the solutions to the homogeneous equations plus the particular solution:

$$ u(x) = A\cdot u_{h_1}(x) + B\cdot u_{h_2}(x) + u_p $$

\begin{equation}
    u(x) = A e^{\left(i\sqrt{5}\right) x} + B e^{-\left(i\sqrt{5}\right) x} + \frac{3.5}{5}
    \label{eq:general_solution}
\end{equation}

Where the complex constants $A$ and $B$ are set by boundary conditions.

### Finding the constants $A$ and $B$ using boundary conditions

Substituting the general solution $\eqref{eq:general_solution}$ found above in the first boundary condition gives the following equation for $A$ and $B$:

$$ u(x=0) = A e^{\left(i\sqrt{5}\right) \cdot 0} + B e^{-\left(i\sqrt{5}\right) \cdot 0} + \frac{3.5}{5} = 1,2 $$

\begin{equation}
    \Rightarrow A + B = 0.5.
    \label{eq:system1}
\end{equation}

Then, substituting the general solution $\eqref{eq:general_solution}$ in the second boundary condition results in another equation relating $A$ and $B$:

$$ \left.\frac{du}{dx}\right\rvert_{x=10} = A \left(i\sqrt{5}\right) e^{\left(i\sqrt{5}\right) \cdot 10} - B \left(i\sqrt{5}\right) e^{-\left(i\sqrt{5}\right) \cdot 10} = 0.5 $$

\begin{equation}
    \Rightarrow A e^{\left(i\sqrt{5}\right) \cdot 10} - B e^{-\left(i\sqrt{5}\right) \cdot 10} = -i\frac{0.5}{\sqrt{5}}.
    \label{eq:system2}
\end{equation}

Solving the system of equations formed by $\eqref{eq:system1}$ and $\eqref{eq:system2}$ for $A$ and $B$ results in:

\begin{equation}
    A = \frac{1}{4} - i \frac{\sin{10\sqrt{5}} + 1/\sqrt{5}}{4\cos{10\sqrt{5}}} \approx \frac{1}{4} + 0.23i
\end{equation}

\begin{equation}
    B = \frac{1}{4} + i \frac{\sin{10\sqrt{5}} + 1/\sqrt{5}}{4\cos{10\sqrt{5}}} \approx \frac{1}{4} - 0.23i
\end{equation}

Plugging the solution for $A$ and $B$ into the general solution $\eqref{eq:general_solution}$ results in the final solution for the differential equation $\eqref{eq:differential_equation}$:

\begin{equation}
    u(x) =  \left(
                \frac{1}{4} - i \frac{\sin{10\sqrt{5}} + 1/\sqrt{5}}{4\cos{10\sqrt{5}}}
            \right) e^{\left(i\sqrt{5}\right) x}
          + \left(
                \frac{1}{4} + i \frac{\sin{10\sqrt{5}} + 1/\sqrt{5}}{4\cos{10\sqrt{5}}}
            \right) e^{-\left(i\sqrt{5}\right) x}
          + \frac{7}{10}
    \label{eq:solution}
\end{equation}

Equation $\eqref{eq:solution}$ can be simplified so that the imaginary terms clearly vanish:

\begin{equation}
    u(x) = \frac{1}{2} \cos{\left(x\sqrt{5}\right)}
         + \left(\frac{\sin{\left(10\sqrt{5}\right)} + 1/\sqrt{5}}{2\cos{\left(10\sqrt{5}\right)}}\right) \sin{\left(x\sqrt{5}\right)}
         + \frac{7}{10}
\end{equation}

### Plotting the analytical solution

```python
import numpy as np
from matplotlib import pyplot as plt

# Parameters
x_min = 0
x_max = 10
number_of_points = 1000

# Define x mesh
x_mesh = np.linspace(x_min, x_max, number_of_points)

# Calculate solution for all points in the x mesh
def u_analytical(x):
    return 0.5*np.cos(x*np.sqrt(5)) + ( (np.sin(10*np.sqrt(5)) + 1/np.sqrt(5)) / (2*np.cos(10*np.sqrt(5))) )*np.sin(x*np.sqrt(5)) + 0.7

# Plot the solution
plt.figure(figsize=(8,4), dpi=200)
plt.plot(x_mesh, u_analytical(x_mesh), color='k')
plt.title(r'Analytical Solution of $\frac{d^2 u}{dx^2} + 5u = 3.5$')
plt.xlabel(r'$x$')
plt.ylabel(r'$u(x)$')
plt.show()
```

## Item B - Numerical Solution using the Galerkin's Weighted Residual Method (Linear)

### Deriving the Equations for Galerkin's Weighted Residual Method (Linear)

Let the differential equation $\eqref{eq:differential_equation}$ be written as:

\begin{equation}
    \frac{d^2 u}{dx^2} + a\cdot u + b = 0, \quad a = 5, \quad b = -3.5
    \label{eq:diff_eq_simplified}
\end{equation}

The weighted residual method (WRM) can be applied to $\eqref{eq:diff_eq_simplified}$ as follows:

\begin{equation}
    \int_{\Omega_e} \varphi_i (x) \cdot \left( \frac{d^2 u}{dx^2} + a\cdot u(x) + b \right) dx = 0,
    \label{eq:wrm_strong}
\end{equation}

where $\Omega_e$ represents the domain of a mesh element and $\varphi_i (x)$ the testing function.
To achieve the weak formulation of the WRM, equation $\eqref{eq:wrm_strong}$ can be integrated by parts, resulting in:

\begin{equation}
    \int_{\Omega_e} \left( \varphi_i (x) \cdot a\cdot u(x) \right) dx
    + \int_{\Omega_e} \left( \varphi_i (x) \cdot b \right) dx
    +  \left. \varphi_i \frac{du}{dx} \right\rvert_{\partial \Omega_e}
    = \int_{\Omega_e} \left( \frac{d\varphi_i}{dx} \frac{du}{dx} \right) dx,
    \label{eq:wrm_weak}
\end{equation}

The numerical solution of $u(x)$ is approximated at each mesh element by $\hat{u}(x)$ such that:

\begin{equation}
    \hat{u}(x|x \in \Omega_e) = \phi_0^e \hat{u}_0^e + \phi_1^e \hat{u}_1^e,
\end{equation}

where $\phi_0^e$ and $\phi_1^e$ are linear base functions and $\hat{u}_i^e$ are their weight. Furthermore, by considering Galerkin's method, the testing functions are equal to the base functions. Therefore, for the element $e$, equation $\eqref{eq:wrm_weak}$ becomes:

\begin{equation}
    \int_{\Omega_e} \left( \phi_i (x) \cdot a\cdot \hat{u}(x) \right) dx
    + \int_{\Omega_e} \left( \phi_i (x) \cdot b \right) dx
    + \left. \phi_i \frac{d\hat{u}}{dx} \right\rvert_{\partial \Omega_e}
    = \int_{\Omega_e} \left( \frac{d\phi_i}{dx} \frac{d\hat{u}}{dx} \right) dx,
    \label{eq:wrm_weak_numerical}
\end{equation}

### Working in Local Coordinates

It is useful to work with equation $\eqref{eq:wrm_weak_numerical}$ in local coordinates so that the integrals only need to be evaluated once for all mesh elements. The mapping function, for a certain mesh element $e$, from the local coordinate $\xi \in [-1, 1]$ to the global coordinate $x \in [x_{e-1}, x_e]$ is given by:

\begin{equation}
    x = \chi^e (\xi) = x_{e-1}\frac{1-\xi}{2} + x_e \frac{1+\xi}{2},
    \label{eq:mapping}
\end{equation}

where $x_{e-1}$ and $x_e$ are the $x$-coordinates of the element's vertices.

In the local coordinate system of each mesh element, the linear base functions are given by:

\begin{equation}
    \phi_0^e (\xi) = \frac{1-\xi}{2}, \quad \phi_1^e (\xi) = \frac{1+\xi}{2}.
    \label{eq:linear_base_functions}
\end{equation}

### Manipulating and Simplifying each Term

#### First Term

The first term in equation $\eqref{eq:wrm_weak_numerical}$ can be written in the local coordinate system of the element $e$ as follows:

\begin{equation}
    \int_{\Omega_e} \left( \phi_i (x) \cdot a\cdot \hat{u}(x) \right) dx
    =
    \int_{-1}^{1} \left( \phi_i (\xi) \cdot a\cdot \left( \hat{u}_0^{e} \phi_0 (\xi) + \hat{u}_1^{e} \phi_1 (\xi) \right) \right) \left( \frac{d\chi}{d\xi} \right) d\xi
\end{equation}

Expanding the base functions as given in equation $\eqref{eq:linear_base_functions}$ and using the fact that $d\chi/d\xi = \ell_{e}/2$, where $\ell_{e} = x_{e} - x_{e-1}$ is the length of element $e$, results in:

\begin{equation}
    a \frac{\ell_e}{2} \int_{-1}^{1} \left( \phi_i (\xi)\left( \hat{u}_0^{e} \phi_0 (\xi) + \hat{u}_1^{e} \phi_1 (\xi) \right) \right) d\xi
\end{equation}

Denoting the vector $\vec{\phi} = [\phi_0, \phi_1]^T$ and the vector $\vec{u} = [\hat{u}_0, \hat{u}_1]^T$, this term can be written in matrix form:

\begin{equation}
    \int_{\Omega_e} \left( \phi_i (x) \cdot a\cdot \hat{u}(x) \right) dx
    =
    a \frac{\ell_e}{2} \left( \int_{-1}^{1} \vec{\phi} \cdot \vec{\phi}^T d\xi \right) \vec{u}.
\end{equation}

#### Second Term

In a similar manner, the second term in equation $\eqref{eq:wrm_weak_numerical}$ can be written in the local coordinate system using vector notation as follows:

\begin{equation}
    \int_{\Omega_e} \left( \phi_i (x) \cdot b \right) dx
    =
    b \frac{\ell_e}{2} \int_{-1}^{1} \vec{\phi} d\xi.
\end{equation}

#### Third Term

Once again, the third term in equation $\eqref{eq:wrm_weak_numerical}$ can be written in the local coordinate system using vector notation as follows:

\begin{equation}
    \left. \phi_i \frac{d\hat{u}}{dx} \right\rvert_{\partial \Omega_e}
    =
    \left( \vec{\phi} \frac{d\hat{u}}{dx} \right)_{\xi=-1}^{\xi=1}.
\end{equation}

#### Fourth Term

Finally, the last term of equation $\eqref{eq:wrm_weak_numerical}$ can be written as:

\begin{equation}
    \int_{\Omega_e} \left( \frac{d\phi_i}{dx} \frac{d\hat{u}}{dx} \right) dx
    =
    \frac{2}{\ell_e} \left( \int_{-1}^{1} \frac{d\vec{\phi}}{d\xi} \frac{d\vec{\phi}^T}{d\xi} d\xi \right) \vec{u},
\end{equation}

where the fact that $d\chi^{-1}/dx = (d\chi/d\xi)^{-1} = 2/\ell_e$ was used.

### Final Discretized Equation

Combining all the simplified terms, equation $\eqref{eq:wrm_weak_numerical}$ is expressed in local coordinates using vector notation by:

\begin{equation}
    a \frac{\ell_e}{2} \left( \int_{-1}^{1} \vec{\phi} \cdot \vec{\phi}^T d\xi \right) \vec{u}
    + b \frac{\ell_e}{2} \int_{-1}^{1} \vec{\phi} d\xi
    + \left( \vec{\phi} \frac{d\hat{u}}{dx} \right)_{\xi=-1}^{\xi=1}
    = \frac{2}{\ell_e} \left( \int_{-1}^{1} \frac{d\vec{\phi}}{d\xi} \frac{d\vec{\phi}^T}{d\xi} d\xi \right) \vec{u}.
\end{equation}

Rearranging the terms so that all terms with $\vec{u}$ are on the left side and all terms without are on the right side:

\begin{equation}
    \left(
        \left( \frac{2}{\ell_e} \int_{-1}^{1} \frac{d\vec{\phi}}{d\xi} \frac{d\vec{\phi}^T}{d\xi} d\xi \right)
        - a \frac{\ell_e}{2} \left( \int_{-1}^{1} \vec{\phi} \cdot \vec{\phi}^T d\xi \right)
    \right) \vec{u}
    = b \frac{\ell_e}{2} \int_{-1}^{1} \vec{\phi} d\xi + \left( \vec{\phi} \frac{d\hat{u}}{dx} \right)_{\xi=-1}^{\xi=1} .
\end{equation}

#### Calculating the Integrals

The definite integrals can be readily evaluated analytically, giving the following results:

\begin{equation}
    \left(
        \frac{1}{\ell_e} \begin{bmatrix}
             1 & -1\\
            -1 & 1
        \end{bmatrix}
        - a \frac{\ell_e}{6} \begin{bmatrix}
            2 & 1\\
            1 & 2
        \end{bmatrix}
    \right) \vec{u}
    = b \frac{\ell_e}{2} \begin{bmatrix}
            1 \\
            1
        \end{bmatrix}
      + \begin{bmatrix}
            \phi_0 (1) \left.\frac{d\hat{u}}{dx}\right\rvert_{\xi=1} - \phi_0 (-1) \left.\frac{d\hat{u}}{dx}\right\rvert_{\xi=-1} \\
            \phi_1 (1) \left.\frac{d\hat{u}}{dx}\right\rvert_{\xi=1} - \phi_1 (-1) \left.\frac{d\hat{u}}{dx}\right\rvert_{\xi=-1}
        \end{bmatrix}
\end{equation}

Further simplification results in the final form of the local algebraic equation where $\hat{u}_0$ and $\hat{u}_1$ are the variables and $d\hat{u}/dx$ is an unknown which will be dealt with using boundary conditions:

\begin{equation}
    \left(
        \frac{1}{\ell_e} \begin{bmatrix}
             1 & -1\\
            -1 & 1
        \end{bmatrix}
        - a \frac{\ell_e}{6} \begin{bmatrix}
            2 & 1\\
            1 & 2
        \end{bmatrix}
    \right) \begin{bmatrix}
            \hat{u}_0 \\
            \hat{u}_1
        \end{bmatrix}
    = b \frac{\ell_e}{2} \begin{bmatrix}
            1 \\
            1
        \end{bmatrix}
      + \begin{bmatrix}
            - \left.\frac{d\hat{u}}{dx}\right\rvert_{x_{e-1}} \\
            \left.\frac{d\hat{u}}{dx}\right\rvert_{x_e}
        \end{bmatrix}
    \label{eq:local_algebraic_equation}
\end{equation}

### Assembling the Global System

Each finite element has two unknowns, $\hat{u}_0^e$ and $\hat{u}_1^e$, governed by equation $\eqref{eq:local_algebraic_equation}$. In order to guarantee a continuos solution, the following condition must be satisfied:

\begin{equation}
    \hat{u}_1^e = \hat{u}_0^{e+1}
    \label{eq:continuity_condition}
\end{equation}

This means the algebraic equations which result by applying $\eqref{eq:local_algebraic_equation}$ to all elements form a coupled linear system of equations with unknowns $\hat{u}_0^0$, $\hat{u}_1^e, e=1,...,N$ and $d\hat{u}/dx$.

The complete system of equations is, then, given by:

\begin{equation}
    [K][u] = [f],
\end{equation}

where $[K]$ is usually called the stiffness matrix and $[f]$ is the force vector.

The derivative terms, $d\hat{u}/dx$, will mostly dissapper in the complete system of equations. Only the boundary derivatives will be kept. This are dealt with using boundary conditions.

### Code Implementation

```python
import numpy as np
from scipy import linalg
from matplotlib import pyplot as plt
from sympy import Symbol, Matrix, init_printing
init_printing()

# Differential equation parameters
a = 5
b = -3.5

# Boundary conditions
dirichlet_boundary_left = 1.2
neumann_boundary_right = 0.5

# Mesh settings
x_min = 0
x_max = 10
number_of_elements = 10
element_length = (x_max-x_min)/number_of_elements

# Calculate mesh nodes (length = number_of_elements + 1)
x_mesh_nodes = np.linspace(x_min, x_max, number_of_elements+1)

# Initialize solution vector, global system matrix A and global force vector f
u_values = np.zeros(number_of_elements+1)
global_matrix = np.zeros((number_of_elements+1, number_of_elements+1))
global_force_vector = np.zeros(number_of_elements+1)

# Assemble the global matrix
for e in range(1, number_of_elements+1):
    # Initialize local matrix
    local_matrix = np.zeros((2,2))
    # Compute local matrix
    local_matrix += (1/element_length) * np.array(
        [[ 1, -1],
         [-1,  1]]
    )
    local_matrix += -(a*element_length/6) * np.array(
        [[2, 1],
         [1, 2]]
    )
    # Add local matrix to global matrix
    global_matrix[e-1:e+1, e-1:e+1] += local_matrix

# Assemble the global force vector
for e in range(1, number_of_elements+1):
    # Compute local force vector
    local_force_vector = (b*element_length/2) * np.array([1, 1])
    # Add local force vector to global force vector
    global_force_vector[e-1:e+1] += local_force_vector

# Apply Dirichlet boundary condition to the left-most element
## Adjust global matrix
global_matrix[0, :] *= 0
global_matrix[0, 0] += 1
## Adjust force vector
global_force_vector[0] = dirichlet_boundary_left

# Apply Neumann boundary condition to the right-most element
global_force_vector[-1] += neumann_boundary_right

# Solve linear system for u solution vector
u_values = linalg.solve(global_matrix, global_force_vector)

# Print results
print('Global Matrix')
sympy_global_matrix = Matrix(global_matrix)
display(sympy_global_matrix.applyfunc(lambda x: Symbol('{:.2f}'.format(x))))

print('Force Vector')
display(Matrix(global_force_vector))

print('Solution Vector')
sympy_solution_vector = Matrix(u_values)
display(sympy_solution_vector.applyfunc(lambda x: Symbol('{:.3f}'.format(x))))

# Plot results
plt.figure(figsize=(8,4), dpi=200)
plt.plot(x_mesh, u_analytical(x_mesh), color='k', label='Analytical')
plt.plot(x_mesh_nodes, u_values.T, 'o--', color='r', label='Numerical')
plt.title(r'Solution of $\frac{d^2 u}{dx^2} + 5u = 3.5$')
plt.xlabel(r'$x$')
plt.ylabel(r'$u(x)$')
plt.legend()
plt.show()
```

## Item C, D and E - Numerical Solution for Different Mesh Sizes

In item B, the differential equation was solved by applying Galerkin's Weighted Residual Method (Linear) and using only 10 finite elements. The small amount of elements justify the disagreement between the analytical and numerical solution.

Here, the differential equation will be solved numerically using the same method as in item B, however, with 20, 50 and 100 elements.

### Implementing the numerical solution as a function

```python
import numpy as np
from scipy import linalg
from matplotlib import pyplot as plt
from sympy import Symbol, Matrix, init_printing
init_printing()

def solve_differential_equation(
    x_min=0,
    x_max=10,
    number_of_elements=10,
    dirichlet_boundary_left=1.2,
    neumann_boundary_right=0.5):
    """Solves the differential equation below using Galerkin's Weighted Residual
    Method with linear base functions.

        $$ \frac{d^2 u}{dx^2} + a\cdot u + b = 0, \quad a = 5, \quad b = -3.5. $$

    The differential equation is subjected to the following boundary conditions:

        $$ u(x=0) =1.2, \quad \left.\frac{du}{dx}\right\rvert_{x=10} = 0.5. $$

    Parameters
    ----------
    x_min : float, optional
        Left edge of the solution domain. Default is 0.
    x_max : float, optional
        Right edge of the solution domain. Default is 10
    number_of_elements: int, optional
        Number of elements to discretize the solution domain. Default is 10.
    dirichlet_boundary_left: float, optional
        Dirichlet boundary condition applied at the left edge of the domain.
        Default value is 1.2.
        Example: u(x=x_min) = 1.2.
    neumann_boundary_right: float, optional
        Neumann boundary condition applied at the right edge of the domain.
        Default value is 0.5.
        Example: du(x=x_max)/dx = 0.5.

    Returns
    -------
    x_mesh_nodes : array of floats
        Mesh nodes for x between x_min and x_max with length equal to
        number_of_elements + 1.
    u_values : array of floats
        Values of the numerical solution for each mesh node.
    """

    # Differential equation parameters
    a = 5
    b = -3.5

    # Mesh settings
    element_length = (x_max-x_min)/number_of_elements

    # Calculate mesh nodes (length = number_of_elements + 1)
    x_mesh_nodes = np.linspace(x_min, x_max, number_of_elements+1)

    # Initialize solution vector, global system matrix A and global force vector f
    u_values = np.zeros(number_of_elements+1)
    global_matrix = np.zeros((number_of_elements+1, number_of_elements+1))
    global_force_vector = np.zeros(number_of_elements+1)

    # Assemble the global matrix
    for e in range(1, number_of_elements+1):
        # Initialize local matrix
        local_matrix = np.zeros((2,2))
        # Compute local matrix
        local_matrix += (1/element_length) * np.array(
            [[ 1, -1],
             [-1,  1]]
        )
        local_matrix += -(a*element_length/6) * np.array(
            [[2, 1],
             [1, 2]]
        )
        # Add local matrix to global matrix
        global_matrix[e-1:e+1, e-1:e+1] += local_matrix

    # Assemble the global force vector
    for e in range(1, number_of_elements+1):
        # Compute local force vector
        local_force_vector = (b*element_length/2) * np.array([1, 1])
        # Add local force vector to global force vector
        global_force_vector[e-1:e+1] += local_force_vector

    # Apply Dirichlet boundary condition to the left-most element
    ## Adjust global matrix
    global_matrix[0, :] *= 0
    global_matrix[0, 0] += 1
    ## Adjust force vector
    global_force_vector[0] = dirichlet_boundary_left

    # Apply Neumann boundary condition to the right-most element
    global_force_vector[-1] += neumann_boundary_right

    # Solve linear system for u solution vector
    u_values = linalg.solve(global_matrix, global_force_vector)

    return x_mesh_nodes, u_values.T
```

### Executing the function for different meshes

```python
number_of_elements_options = [10, 20, 50, 100]

# Initialize plot
plt.figure(figsize=(8,4), dpi=200)
plt.title(r'Solution of $\frac{d^2 u}{dx^2} + 5u = 3.5$')
plt.xlabel(r'$x$')
plt.ylabel(r'$u(x)$')

# Plot analytical solution
plt.plot(x_mesh, u_analytical(x_mesh), color='k', linewidth=1, label='Analytical')

# Run and plot each option of number_of_elements
for number_of_elements in number_of_elements_options:
    x_mesh_nodes, u_values = solve_differential_equation(number_of_elements=number_of_elements)
    plt.scatter(x_mesh_nodes, u_values, s=6)
    plt.plot(x_mesh_nodes, u_values, label=f'Numerical - ${number_of_elements}$ elements', linestyle='--')

# Finish plot
plt.ylim(-1.3, 1.7)
plt.xlim(0, 10)
plt.legend()
plt.show()
```

### Interpretation of the result

As expected, with the increasing number of elements, the numerical solution appears to approach the analytical solution.
While for 10 and 20 elements the numerical solution is significantly off, the numerical results for 50 elements are close to the analytical solution. For 100 elements, it is hard to distinguish the difference between the analytical and numerical results.

In the next item, the error norm for each numerical solution will be analysed.

## Item F - Mesh Convergence

In this item, the error norm of each solution shown above will be calculated and plotted in a graph of error norm by degrees of freedom. The degrees of freedom correspond to the number of mesh nodes, which in this 1 dimensional case equals the number of elements plus 1.

Here, the error norm is defined as:

\begin{equation}
    \|E\|_{2}=\left[\int_{\Omega}\left(u-\hat{u}\right)^{2} \mathrm{~d} x\right]^{1 / 2},
\end{equation}

where $u$ is the analytical solution and $\hat{u}$ is the numerical solution. Since the numerical solution is discrete, this integral is calculated as a summation of the mesh nodes:

\begin{equation}
    \|E\|_{2}=\left[\sum_{x_i}\left(u(x_i)-\hat{u}(x_i)\right)^{2}\right]^{1 / 2}.
\end{equation}

### Code implementation

```python
def error_norm(x_mesh, analytical_solution, numerical_solution):
    return sum((analytical_solution(x_mesh) - numerical_solution)**2)

# Select options
number_of_elements_options = [10, 20, 50, 100, 200, 400, 800, 1600, 3200]

# Initialize variables to store results
degrees_of_freedom = []
errors = []

# Run and plot each option of number_of_elements
for number_of_elements in number_of_elements_options:
    x_mesh_nodes, u_values = solve_differential_equation(number_of_elements=number_of_elements)
    error = error_norm(x_mesh_nodes, u_analytical, u_values)

    # Save results for this case
    degrees_of_freedom += [len(x_mesh_nodes)]
    errors += [error]

# Plot
plt.figure(figsize=(8,4), dpi=200)
plt.plot(degrees_of_freedom, errors, 'o--', color='k')
plt.xscale('log')
plt.yscale('log')
plt.title(r'Mesh Convergence')
plt.xlabel(r'Degrees of freedom')
plt.ylabel(r'$\|E\|_{2}$')
plt.show()
```

### Result interpretation

The $L^2$ norm used in this item falls as the number of degrees of freedom increases, exponentially fast. The plot above, which uses logarithmic scales, displays this behaviour clearly.

## Plot Style

```python
# plt.rcParams['text.usetex'] = False
plt.rcParams['font.family'] = 'serif'
plt.rcParams['font.size'] = 9
# plt.rcParams['mathtext.fontset'] = 'dejavuserif'

plt.rcParams['text.usetex'] = True
plt.rcParams['text.latex.preamble'] = r'\usepackage{amsmath, amssymb}'

plt.rcParams['savefig.bbox'] = 'tight'
plt.rcParams['savefig.pad_inches'] = 0.05

plt.rcParams['xtick.direction'] = 'in'
plt.rcParams['xtick.major.size'] = 3
plt.rcParams['xtick.major.width'] = 0.5
plt.rcParams['xtick.minor.size'] = 1.5
plt.rcParams['xtick.minor.width'] = 0.5
plt.rcParams['xtick.minor.visible'] = True
plt.rcParams['xtick.top'] = True

plt.rcParams['ytick.direction'] = 'in'
plt.rcParams['ytick.major.size'] = 3
plt.rcParams['ytick.major.width'] = 0.5
plt.rcParams['ytick.minor.size'] = 1.5
plt.rcParams['ytick.minor.width'] = 0.5
plt.rcParams['ytick.minor.visible'] = True
plt.rcParams['ytick.right'] = True

plt.rcParams['axes.linewidth'] = 0.5
plt.rcParams['grid.linewidth'] = 0.5
plt.rcParams['lines.linewidth'] = 1.

# Grid lines
plt.rcParams['axes.grid'] = True
plt.rcParams['axes.axisbelow'] = True
plt.rcParams['grid.linestyle'] = '--'
plt.rcParams['grid.color'] = 'k'
plt.rcParams['grid.alpha'] = 0.5
plt.rcParams['grid.linewidth'] = 0.5

# Legend
plt.rcParams['legend.frameon'] = True
plt.rcParams['legend.framealpha'] = 1.0
plt.rcParams['legend.fancybox'] = True
plt.rcParams['legend.numpoints'] = 1
```

Problem Set 1/1D Finite Element Method Consistency.ipynb


# SymPy: Python for Symbolic Computations
### Tanmoy Dasgupta

### Initialization

```python
from sympy import *
from sympy.abc import *
from sympy import pi, I
```

### Basic Operations

#### Declaring variables

```python
expr = cos(x) + 1
expr
```

$\displaystyle \cos{\left(x \right)} + 1$

```python
(expr + 2)**2
```

$\displaystyle \left(\cos{\left(x \right)} + 3\right)^{2}$

#### Variable substitution

```python
expr.subs(x, y)
```

$\displaystyle \cos{\left(y \right)} + 1$

```python
expr.subs(x, pi)
```

$\displaystyle 0$

```python
expr = x**3 + 4*x*y - z
expr
```

$\displaystyle x^{3} + 4 x y - z$

```python
expr.subs([(x, 2), (y, 4), (z, 0)])
```

$\displaystyle 40$

#### Rationals

```python
1/3
```

    0.3333333333333333

```python
Rational(1, 3)
```

$\displaystyle \frac{1}{3}$

```python
Rational(3,2)*pi + exp(-I*z) / (z**2 + y)
```

$\displaystyle \frac{3 \pi}{2} + \frac{e^{- i z}}{y + z^{2}}$

### Converting Strings to SymPy Expressions

```python
str_expr = "x**2 + 3*x - 1/2"
expr = sympify(str_expr)
expr
```

$\displaystyle x^{2} + 3 x - \frac{1}{2}$

```python
strexp = '2/5'
sympify(strexp)
```

$\displaystyle \frac{2}{5}$

```python
expr.subs(x, 2)
```

$\displaystyle \frac{19}{2}$

### Evaluation

```python
expr = sqrt(8)
expr
```

$\displaystyle 2 \sqrt{2}$

```python
expr.evalf()
```

$\displaystyle 2.82842712474619$

#### Specifying the number of digits

```python
expr.evalf(100)
```

$\displaystyle 2.828427124746190097603377448419396157139343750753896146353359475981464956924214077700775068655283145$

```python
pi.evalf(1000)
```

$\displaystyle 3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117067982148086513282306647093844609550582231725359408128481117450284102701938521105559644622948954930381964428810975665933446128475648233786783165271201909145648566923460348610454326648213393607260249141273724587006606315588174881520920962829254091715364367892590360011330530548820466521384146951941511609433057270365759591953092186117381932611793105118548074462379962749567351885752724891227938183011949129833673362440656643086021394946395224737190702179860943702770539217176293176752384674818467669405132000568127145263560827785771342757789609173637178721468440901224953430146549585371050792279689258923542019956112129021960864034418159813629774771309960518707211349999998372978049951059731732816096318595024459455346908302642522308253344685035261931188171010003137838752886587533208381420617177669147303598253490428755468731159562863882353787593751957781857780532171226806613001927876611195909216420199$

#### Sometimes you'd have to call `evalf` with `subs`

```python
expr = sin(2*x) + cos(2*x)
expr.subs(x, 2)
```

$\displaystyle \sin{\left(4 \right)} + \cos{\left(4 \right)}$

```python
expr = sin(2*x) + cos(2*x)
expr.evalf(subs={x: 2})
```

$\displaystyle -1.41044611617154$

### Simplifications

```python
simplify(sin(x)**2 + cos(x)**2)
```

$\displaystyle 1$

```python
expr = (x**3 + x**2 - x - 1)/(x**2 + 2*x + 1)
expr
```

$\displaystyle \frac{x^{3} + x^{2} - x - 1}{x^{2} + 2 x + 1}$

```python
simplify(expr)
```

$\displaystyle x - 1$

```python
expand((a + 2*b + 3*c)**10)
```

$\displaystyle a^{10} + 20 a^{9} b + 30 a^{9} c + 180 a^{8} b^{2} + 540 a^{8} b c + 405 a^{8} c^{2} + 960 a^{7} b^{3} + 4320 a^{7} b^{2} c + 6480 a^{7} b c^{2} + 3240 a^{7} c^{3} + 3360 a^{6} b^{4} + 20160 a^{6} b^{3} c + 45360 a^{6} b^{2} c^{2} + 45360 a^{6} b c^{3} + 17010 a^{6} c^{4} + 8064 a^{5} b^{5} + 60480 a^{5} b^{4} c + 181440 a^{5} b^{3} c^{2} + 272160 a^{5} b^{2} c^{3} + 204120 a^{5} b c^{4} + 61236 a^{5} c^{5} + 13440 a^{4} b^{6} + 120960 a^{4} b^{5} c + 453600 a^{4} b^{4} c^{2} + 907200 a^{4} b^{3} c^{3} + 1020600 a^{4} b^{2} c^{4} + 612360 a^{4} b c^{5} + 153090 a^{4} c^{6} + 15360 a^{3} b^{7} + 161280 a^{3} b^{6} c + 725760 a^{3} b^{5} c^{2} + 1814400 a^{3} b^{4} c^{3} + 2721600 a^{3} b^{3} c^{4} + 2449440 a^{3} b^{2} c^{5} + 1224720 a^{3} b c^{6} + 262440 a^{3} c^{7} + 11520 a^{2} b^{8} + 138240 a^{2} b^{7} c + 725760 a^{2} b^{6} c^{2} + 2177280 a^{2} b^{5} c^{3} + 4082400 a^{2} b^{4} c^{4} + 4898880 a^{2} b^{3} c^{5} + 3674160 a^{2} b^{2} c^{6} + 1574640 a^{2} b c^{7} + 295245 a^{2} c^{8} + 5120 a b^{9} + 69120 a b^{8} c + 414720 a b^{7} c^{2} + 1451520 a b^{6} c^{3} + 3265920 a b^{5} c^{4} + 4898880 a b^{4} c^{5} + 4898880 a b^{3} c^{6} + 3149280 a b^{2} c^{7} + 1180980 a b c^{8} + 196830 a c^{9} + 1024 b^{10} + 15360 b^{9} c + 103680 b^{8} c^{2} + 414720 b^{7} c^{3} + 1088640 b^{6} c^{4} + 1959552 b^{5} c^{5} + 2449440 b^{4} c^{6} + 2099520 b^{3} c^{7} + 1180980 b^{2} c^{8} + 393660 b c^{9} + 59049 c^{10}$

```python
expand((x + 2)*(x - 3))
```

$\displaystyle x^{2} - x - 6$

```python
expand((x + 1)*(x - 2) - (x - 1)*x)
```

$\displaystyle -2$

```python
factor(x**3 - x**2 + x - 1)
```

$\displaystyle \left(x - 1\right) \left(x^{2} + 1\right)$

```python
factor(x**2*z + 4*x*y*z + 4*y**2*z)
```

$\displaystyle z \left(x + 2 y\right)^{2}$

```python
factor_list(x**2*z + 4*x*y*z + 4*y**2*z)
```

    (1, [(z, 1), (x + 2*y, 2)])

```python
expr = (x + 1) * (y**3 + 3*x + 7*x*y) * (17*z + 2)
expr1 = expand(expr)
expr1
```

$\displaystyle 119 x^{2} y z + 14 x^{2} y + 51 x^{2} z + 6 x^{2} + 17 x y^{3} z + 2 x y^{3} + 119 x y z + 14 x y + 51 x z + 6 x + 17 y^{3} z + 2 y^{3}$

```python
factor(expr1)
```

$\displaystyle \left(x + 1\right) \left(17 z + 2\right) \left(7 x y + 3 x + y^{3}\right)$

#### Collecting the like powers

```python
expr = x*y + x - 3 + 2*x**2 - z*x**2 + x**3
factor(expr)
```

$\displaystyle x^{3} - x^{2} z + 2 x^{2} + x y + x - 3$

```python
collected_expr = collect(expr, x)
collected_expr
```

$\displaystyle x^{3} + x^{2} \left(2 - z\right) + x \left(y + 1\right) - 3$

#### Cancel common factors

```python
cancel((x**2 + 2*x + 1)/(x**2 + x))
```

$\displaystyle \frac{x + 1}{x}$

```python
expr = 1/x + (3*x/2 - 2)/(x - 4)
expr
```

$\displaystyle \frac{\frac{3 x}{2} - 2}{x - 4} + \frac{1}{x}$

```python
cancel(expr)
```

$\displaystyle \frac{3 x^{2} - 2 x - 8}{2 x^{2} - 8 x}$

```python
expr = (x*y**2 - 2*x*y*z + x*z**2 + y**2 - 2*y*z + z**2)/(x**2 - 1)
expr
```

$\displaystyle \frac{x y^{2} - 2 x y z + x z^{2} + y^{2} - 2 y z + z^{2}}{x^{2} - 1}$

```python
cancel(expr)
```

$\displaystyle \frac{y^{2} - 2 y z + z^{2}}{x - 1}$

#### Partial fraction expansion

```python
expr = (4*x**3 + 21*x**2 + 10*x + 12)/(x**4 + 5*x**3 + 5*x**2 + 4*x)
expr
```

$\displaystyle \frac{4 x^{3} + 21 x^{2} + 10 x + 12}{x^{4} + 5 x^{3} + 5 x^{2} + 4 x}$

```python
apart(expr)
```

$\displaystyle \frac{2 x - 1}{x^{2} + x + 1} - \frac{1}{x + 4} + \frac{3}{x}$

#### Trig simplifications

```python
expr = sin(x)**4 - 2*cos(x)**2*sin(x)**2 + cos(x)**4
expr
```

$\displaystyle \sin^{4}{\left(x \right)} - 2 \sin^{2}{\left(x \right)} \cos^{2}{\left(x \right)} + \cos^{4}{\left(x \right)}$

```python
trigsimp(expr)
```

$\displaystyle \frac{\cos{\left(4 x \right)}}{2} + \frac{1}{2}$

```python
trigsimp(sin(x)*tan(x)/sec(x))
```

$\displaystyle \sin^{2}{\left(x \right)}$

```python
trigsimp(cosh(x)**2 + sinh(x)**2)
```

$\displaystyle \cosh{\left(2 x \right)}$

```python
expand(sin(x + y))
```

$\displaystyle \sin{\left(x + y \right)}$

```python
expand_trig(sin(x + y))
```

$\displaystyle \sin{\left(x \right)} \cos{\left(y \right)} + \sin{\left(y \right)} \cos{\left(x \right)}$

```python
expand_trig(sin(x + y)**2)
```

$\displaystyle \left(\sin{\left(x \right)} \cos{\left(y \right)} + \sin{\left(y \right)} \cos{\left(x \right)}\right)^{2}$

```python
expand(expand_trig(sin(x + y)**2))
```

$\displaystyle \sin^{2}{\left(x \right)} \cos^{2}{\left(y \right)} + 2 \sin{\left(x \right)} \sin{\left(y \right)} \cos{\left(x \right)} \cos{\left(y \right)} + \sin^{2}{\left(y \right)} \cos^{2}{\left(x \right)}$

### Calculus

#### Derivative

```python
diff(cos(x), x)
```

$\displaystyle - \sin{\left(x \right)}$

```python
diff(exp(x**2), x)
```

$\displaystyle 2 x e^{x^{2}}$

```python
expr = exp(x*y*z)
expr
```

$\displaystyle e^{x y z}$

```python
diff(expr, x, y, 2, z, 4)
```

$\displaystyle x^{3} y^{2} \left(x^{3} y^{3} z^{3} + 14 x^{2} y^{2} z^{2} + 52 x y z + 48\right) e^{x y z}$

#### Creating an un-evaluated derivative ...

```python
deriv = Derivative(expr, x, y, 2, z, 4)
deriv
```

$\displaystyle \frac{\partial^{7}}{\partial z^{4}\partial y^{2}\partial x} e^{x y z}$

#### ... and evaluating it

```python
deriv.doit()
```

$\displaystyle x^{3} y^{2} \left(x^{3} y^{3} z^{3} + 14 x^{2} y^{2} z^{2} + 52 x y z + 48\right) e^{x y z}$

#### Integration

```python
integrate(cos(x), x)
```

$\displaystyle \sin{\left(x \right)}$

#### Definite integrals

```python
integrate(exp(-x), (x, 0, oo))
```

$\displaystyle 1$

```python
expr = Integral(exp(-x), (x, 0, oo))
expr
```

$\displaystyle \int\limits_{0}^{\infty} e^{- x}\, dx$

```python
expr.doit()
```

$\displaystyle 1$

```python
integrate(exp(-x**2), (x, -oo, oo))
```

$\displaystyle \sqrt{\pi}$

```python
expr = Integral(exp(-x**2), (x, -oo, oo))
expr
```

$\displaystyle \int\limits_{-\infty}^{\infty} e^{- x^{2}}\, dx$

```python
expr.doit()
```

$\displaystyle \sqrt{\pi}$

#### Definite double-integrals

```python
expr = Integral(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
expr
```

$\displaystyle \int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty} e^{- x^{2} - y^{2}}\, dx\, dy$

```python
expr.doit()
```

$\displaystyle \pi$

#### Creating an un-evaluated integral ...

```python
integ = Integral((x**4 + x**2*exp(x) - x**2 - 2*x*exp(x) - 2*x -
                  exp(x))*exp(x)/((x - 1)**2*(x + 1)**2*(exp(x) + 1)), x)
integ
```

$\displaystyle \int \frac{\left(x^{4} + x^{2} e^{x} - x^{2} - 2 x e^{x} - 2 x - e^{x}\right) e^{x}}{\left(x - 1\right)^{2} \left(x + 1\right)^{2} \left(e^{x} + 1\right)}\, dx$

#### ... and evaluating it

```python
integ.doit()
```

$\displaystyle \log{\left(e^{x} + 1 \right)} + \frac{e^{x}}{x^{2} - 1}$

```python
integ = Integral(sin(x**2), x)
integ
```

$\displaystyle \int \sin{\left(x^{2} \right)}\, dx$

```python
integ.doit()
```

$\displaystyle \frac{3 \sqrt{2} \sqrt{\pi} S\left(\frac{\sqrt{2} x}{\sqrt{\pi}}\right) \Gamma\left(\frac{3}{4}\right)}{8 \Gamma\left(\frac{7}{4}\right)}$

#### Results can have conditions

```python
integ = Integral(x**(n-1)*exp(-x), (x, 0, oo))
integ
```

$\displaystyle \int\limits_{0}^{\infty} x^{n - 1} e^{- x}\, dx$

```python
integ.doit()
```

$\displaystyle \begin{cases} \Gamma\left(n\right) & \text{for}\: \operatorname{re}{\left(n\right)} - 1 > -1 \\\int\limits_{0}^{\infty} x^{n - 1} e^{- x}\, dx & \text{otherwise} \end{cases}$

```python
m = Symbol('m', integer=True, positive=True)
n = Symbol('n', integer=True, positive=True)
expr = Integral(sin(m*x) * sin(n*x), (x, -pi, pi))
expr
```

$\displaystyle \int\limits_{- \pi}^{\pi} \sin{\left(m x \right)} \sin{\left(n x \right)}\, dx$

```python
expr.doit()
```

$\displaystyle \begin{cases} 0 & \text{for}\: m \neq n \\\pi & \text{otherwise} \end{cases}$

#### Limits

```python
limit(sin(x)/x, x, 0)
```

$\displaystyle 1$

```python
expr = Limit((cos(x) - 1)/x, x, 0)
expr
```

$\displaystyle \lim_{x \to 0^+}\left(\frac{\cos{\left(x \right)} - 1}{x}\right)$

```python
expr.doit()
```

$\displaystyle 0$

```python
limit(1/x, x, 0, '+')
```

$\displaystyle \infty$

```python
limit(1/x, x, 0, '-')
```

$\displaystyle -\infty$

#### Series expansions

```python
expr = exp(x) + sin(x)
expr.series(x, 0, 8)
```

$\displaystyle 1 + 2 x + \frac{x^{2}}{2} + \frac{x^{4}}{24} + \frac{x^{5}}{60} + \frac{x^{6}}{720} + O\left(x^{8}\right)$

```python
exp(x - 6).series(x, x0=6)
```

$\displaystyle -5 + \frac{\left(x - 6\right)^{2}}{2} + \frac{\left(x - 6\right)^{3}}{6} + \frac{\left(x - 6\right)^{4}}{24} + \frac{\left(x - 6\right)^{5}}{120} + x + O\left(\left(x - 6\right)^{6}; x\rightarrow 6\right)$

```python
expr1 = sin(x).series(x, 0, 10)
expr1
```

$\displaystyle x - \frac{x^{3}}{6} + \frac{x^{5}}{120} - \frac{x^{7}}{5040} + \frac{x^{9}}{362880} + O\left(x^{10}\right)$

```python
expr2 = cos(x).series(x, 0, 10)
expr2
```

$\displaystyle 1 - \frac{x^{2}}{2} + \frac{x^{4}}{24} - \frac{x^{6}}{720} + \frac{x^{8}}{40320} + O\left(x^{10}\right)$

```python
expr3 = exp(I * x).series(x, 0, 10)
expr3
```

$\displaystyle 1 + i x - \frac{x^{2}}{2} - \frac{i x^{3}}{6} + \frac{x^{4}}{24} + \frac{i x^{5}}{120} - \frac{x^{6}}{720} - \frac{i x^{7}}{5040} + \frac{x^{8}}{40320} + \frac{i x^{9}}{362880} + O\left(x^{10}\right)$

```python
expr4 = expand(expr2 + I*expr1)
expr4
```

$\displaystyle 1 + i x - \frac{x^{2}}{2} - \frac{i x^{3}}{6} + \frac{x^{4}}{24} + \frac{i x^{5}}{120} - \frac{x^{6}}{720} - \frac{i x^{7}}{5040} + \frac{x^{8}}{40320} + \frac{i x^{9}}{362880} + O\left(x^{10}\right)$

```python
expr3 == expr4
```

    True

### Equation Solvers

#### Equality in `Sympy` cannot be presented by `=` or `==`. It can only be presented with the function `Eq`.

```python
Eq(x, y)
```

$\displaystyle x = y$

```python
expr = Eq(x**2, 1)
expr
```

$\displaystyle x^{2} = 1$

```python
solveset(expr, x)
```

$\displaystyle \left\{-1, 1\right\}$

```python
solveset(Eq(x**2 - 1, 0), x)
```

$\displaystyle \left\{-1, 1\right\}$

```python
solveset(Eq(sin(x) - 1, 0), x)
```

$\displaystyle \left\{2 n \pi + \frac{\pi}{2}\; \middle|\; n \in \mathbb{Z}\right\}$

```python
eq = Eq(x**3 + 2*x**2 + 4*x + 8, 0)
solve(eq, x)
```

    [-2, -2*I, 2*I]

```python
solveset(Eq(sin(x) + cos(x), 1), x)
```

$\displaystyle \left\{2 n \pi\; \middle|\; n \in \mathbb{Z}\right\} \cup \left\{2 n \pi + \frac{\pi}{2}\; \middle|\; n \in \mathbb{Z}\right\}$

### ODE Solvers

```python
f = symbols('f', cls=Function)
g = symbols('g', cls=Function)
f(x).diff(x)
```

$\displaystyle \frac{d}{d x} f{\left(x \right)}$

```python
diffeq = Eq(f(x).diff(x, 2) - 2*f(x).diff(x) + 3*f(x), sin(x))
diffeq
```

$\displaystyle 3 f{\left(x \right)} - 2 \frac{d}{d x} f{\left(x \right)} + \frac{d^{2}}{d x^{2}} f{\left(x \right)} = \sin{\left(x \right)}$

```python
dsolve(diffeq, f(x))
```

$\displaystyle f{\left(x \right)} = \left(C_{1} \sin{\left(\sqrt{2} x \right)} + C_{2} \cos{\left(\sqrt{2} x \right)}\right) e^{x} + \frac{\sin{\left(x \right)}}{4} + \frac{\cos{\left(x \right)}}{4}$

```python
eqn = Eq(Derivative(f(x),x,x) + 9*f(x), 1)
eqn
```

$\displaystyle 9 f{\left(x \right)} + \frac{d^{2}}{d x^{2}} f{\left(x \right)} = 1$

```python
dsolve(eqn, f(x))
```

$\displaystyle f{\left(x \right)} = C_{1} \sin{\left(3 x \right)} + C_{2} \cos{\left(3 x \right)} + \frac{1}{9}$

### Laplace Transforms

```python
laplace_transform(sin(t), t, s, noconds=True)
```

$\displaystyle \frac{1}{s^{2} + 1}$

```python
integrate(sin(t)*exp(-s*t), (t, 0, oo))
```

$\displaystyle \begin{cases} \frac{1}{s^{2} + 1} & \text{for}\: 2 \left|{\arg{\left(s \right)}}\right| < \pi \\\int\limits_{0}^{\infty} e^{- s t} \sin{\left(t \right)}\, dt & \text{otherwise} \end{cases}$

```python
laplace_transform(t * exp(-a*t) * sin(omega * t), t, s, noconds=True)
```

$\displaystyle \frac{2 \omega \left(a + s\right)}{\left(\omega^{2} + \left(a + s\right)^{2}\right)^{2}}$

```python
G = 1/(s**2 + s + 1)
X = 1/s
Y = G*X
y = inverse_laplace_transform(Y, s, t)
simplify(y)
```

$\displaystyle \frac{\left(- \sqrt{3} \sqrt{e^{t}} \sin{\left(\frac{\sqrt{3} t}{2} \right)} - 3 \sqrt{e^{t}} \cos{\left(\frac{\sqrt{3} t}{2} \right)} + 3 e^{t}\right) e^{- t} \theta\left(t\right)}{3}$

```python
plot(y)
```

```python

```


```python
import sympy as s
import matplotlib

x, x0, y, z = s.symbols('x x0 y z', type=s.S.Reals)
my_func = s.Rational(1, 2) * x ** 3
print(my_func)

integral = s.integrate(my_func, x)
print(integral)
cdf = integral

inv = s.solveset(
    s.Eq(my_func.subs(x, y), x), y, domain=s.S.Reals)

inv_cdf = s.solveset(
    s.Eq(cdf.subs(x, y), x), y, domain=s.S.Reals)

print(inv)
print(inv_cdf)
```

    x**3/2
    x**4/8
    Intersection(FiniteSet(2**(1/3)*Abs(x)**(1/3)*sign(x)), Reals)
    Intersection(FiniteSet(2**(3/4)*x**(1/4), -2**(3/4)*x**(1/4)), Reals)

```python
type(cdf), type(inv_cdf)
```

    (sympy.core.mul.Mul, sympy.sets.sets.Intersection)

```python
def evaluate_func(value):
    return my_func.subs({x: value}).evalf()

def evaluate_inverse(value):
    return inv_cdf.subs({x: value}).evalf()
```

```python
def array_map(f, x):
    return np.array(list(map(f, x)))
```

```python
from matplotlib import pyplot as plt
import numpy as np

MIN, MAX = -evaluate_func(0), evaluate_func(10)
print(MIN, MAX)

xs, y1s, y2s = [], [], []

for _ in range(10):
    x0 = MIN + np.random.random() * (MAX - MIN)
    y0 = evaluate_inverse(x0)

    xs.append(x0)
    y2s.append(y0)
    y1s.append(evaluate_func(y0))

def map_func(y2s, y1s):
    x = np.arange(-10, 10, 0.1)
    y= array_map(evaluate_func, x)
    plt.plot(x,y)

    plt.scatter(y2s, y1s)
    plt.show()

def map_inv(xs, y2s):
    x = np.arange(MIN, MAX, 0.01 * (MAX - MIN))
    y1, y2 = zip(*array_map(evaluate_inverse, x))
    plt.plot(x,y1)
    plt.plot(x,y2)

    plt.scatter(xs, y2s)
    plt.show()

map_func(y2s, y1s)
map_inv(xs, y2s)
```

```python

```


# Session 13: Mixtures of Bernoulli distributions

------------------------------------------------------
*Introduction to Data Science & Machine Learning*

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

In this notebook we will implement the EM algorithm for mixtures of Bernoulli distributions. This model is also known as [*latent class analysis*](https://en.wikipedia.org/wiki/Latent_class_model). As well as being of practical importance on its own right, understading this model and its learning also lay the fundation for **hidden Markov models (HMMs)** over discrete variables. HMMs will be dicussed in future course sessions.

### Mixtures of Bernoulli distributions

Consider a set of i.i.d. $D$-dimensional binary (0-1) vectors. Examples of this kind of data are binary images, binary detection results, or genetic markers. Consider also a mixture of multinomials (or multivariate Bernoullis) model for each of the vectors, $\mathbf{x}$,

$$\displaystyle p(\mathbf{x} | \boldsymbol{\Theta},\mathbf{\pi} ) = \sum_{k=1}^K \pi_k p_k(\mathbf{x}^{(i)}|\boldsymbol{\theta}_k) = \sum_{k=1}^K \pi_k \prod_{j=1}^D \theta_{jk}^{x_{j}} (1-\theta_{jk})^{1-x_{j}},$$
where $\boldsymbol{\Theta}=[\boldsymbol{\theta}_1,\ldots,\boldsymbol{\theta}_K]$. If we are given a data set $\mathbf{X}=\{\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(N)}\}$ then the log likelihood function for this model is given by

\begin{align}
\log p(\mathbf{X}|\boldsymbol{\Theta},\mathbf{\pi}) = \sum_{i=1}^N \log\left(\sum_{k=1}^D \pi_k p(\mathbf{x}^{(i)}|\boldsymbol{\theta}_k)\right)
\end{align}

To prevent overfitting, we will also use a **prior distribution** for the model parameters. For $\boldsymbol{\Theta}$ we have

\begin{align}
\theta_{jk}&\sim\text{Beta}(\alpha,\beta)\\
p(\boldsymbol{\Theta}) &= \prod_{k=1}^{K} \prod_{j=1}^D p(\theta_{jk})\\
p(\theta_{jk}) &= \frac{\theta_{jk}^{\alpha-1}(1-\theta_{jk})^{\beta-1}}{\text{B}(\alpha,\beta)},
\end{align}
where $\text{B}(\alpha,\beta)$ is the [Beta function](https://en.wikipedia.org/wiki/Beta_function). Check [here](https://en.wikipedia.org/wiki/Beta_distribution) for more details about the Beta distribution.

For $\mathbf{\pi}$ we use a uniform [Dirichlet distribution](https://en.wikipedia.org/wiki/Dirichlet_distribution):
\begin{align}
\mathbf{\pi}&\sim \text{Dir}(\frac{1}{K}, \ldots, \frac{1}{K}) \Rightarrow p(\mathbf{\pi}) =\frac{1}{\text{B}(\frac{1}{K}, \ldots, \frac{1}{K})}\prod_{k=1}^{K}\pi_k^{\frac{1}{K}-1},
\end{align}
where $\text{B}(\frac{1}{K}, \ldots, \frac{1}{K})$ is the multivariate Beta function.

### EM learning

We now derive the EM algorithm for maximizing the posterior distribution $p(\mathbf{\Theta},\mathbf{\pi}|\mathbf{X})$. To do this, we introduce an explict discrete latent variable $z\in\{1,\ldots,K\}$ assoacted to each data point $\mathbf{x}$:
\begin{align}
p(\mathbf{x},z) = \prod_{k=1}^{K} \left(\pi_k ~p_{k}(\mathbf{x})\right)^{\mathbb{1} [z==k]}, ~~ p(z)=\prod_{k=1}^K \pi_k^{\mathbb{1} [z==k]}
\end{align}

#### Complete log-likelihood

We write the complete data log-likelihood as follows:
\begin{align}
\log p(\mathbf{X},\mathbf{z}|\boldsymbol{\Theta},\mathbf{\pi})=\sum_{i=1}^{N}\sum_{k=1}^{K}\mathbb{1} [z^{(i)}==k]\left(\log \pi_k + \sum_{j=1}^D \left[x_{j}^{(i)}\log\theta_k+(1-x_{j}^{(i)})\log(1-\theta_k)\right]\right)
\end{align}

#### Posterior distribution of $\mathbf{z}$ given $\boldsymbol{\Theta},\mathbf{\pi}$

In the $E$, we compute the expected complete data log-likelihood w.r.t. posterior distribution of $\mathbf{z}$ given the current values of $\boldsymbol{\Theta},\mathbf{\pi}$:
\begin{align}
p(z^{(i)}=k|\mathbf{x}^{(i)},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)}) \triangleq r_{ik} = \frac{\pi_{(k,t-1)} p_k(\mathbf{x}|\boldsymbol{\theta}_k) }{\sum_{q=1}^K \pi_{(q,t-1)} p_k(\mathbf{x}^{(i)}|\boldsymbol{\theta}_q)}, ~~~ k=1,\ldots, K
\end{align}

#### E-step
It is easy to show that
\begin{align}
\mathcal{Q}(\boldsymbol{\Theta},\mathbf{\pi},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)})&=\mathbb{E}_{p(\mathbf{z}|\mathbf{X}^{(i)},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)})}[\log p(\mathbf{X},\mathbf{z}|\boldsymbol{\Theta},\mathbf{\pi})]\\
&=\sum_{i=1}^{N}\sum_{k=1}^K r_{ik} \left(\log \pi_k + \sum_{j=1}^D \left[x_{j}^{(i)}\log\theta_k+(1-x_{j}^{(i)})\log(1-\theta_k)\right]\right)
\end{align}

#### M-step

We have to find
\begin{align}
\boldsymbol{\Theta}_t,\mathbf{\pi}_t = \arg \max_{\boldsymbol{\Theta},\mathbf{\pi}} ~~\mathcal{Q}(\boldsymbol{\Theta},\mathbf{\pi},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)})+ \log p(\boldsymbol{\Theta})+\log p(\mathbf{\pi})
\end{align}

As a result, one can prove that the maximum is attained at

\begin{align}
r_k &\triangleq \sum_{i=1}^N r_{ik} \\\\
\pi^t_k &= \frac{r_k+\frac{1}{K}-1}{N+1-K}\\\\
\boldsymbol{\theta}_k &= \frac{\sum_{i=1}^N r_{ik}\mathbf{x}^{(i)}+\alpha-1}{r_k+\alpha+\beta-2}
\end{align}

To see details about the derivation of this result, check out chapter 9 of Bishop's book. Also Chapter 11 of Murphy's book.

## Implementation of the E-step

We first write a Python function that evaluates the responsibilities $r_{ik}$, $i=1,\ldots,N$, $k=1,\ldots,K$ for each data point. It should take as input the current values of $\boldsymbol{\pi}$, $\boldsymbol{\Theta}$, and the matrix $\mathbf{X}$ of observations.

```python
import matplotlib.pyplot as plt
import numpy as np
from IPython.display import Image
import matplotlib.mlab as mlab
from mpl_toolkits.mplot3d import Axes3D

%matplotlib inline
```

```python

# THE FOLLOWING FUNCTION takes the whole matrix of data points and the vector
# of probabilites of a single cluster. The function returns the probability of each data
# point given cluster parameters p(x|theta_k)

def eval_bern_pdf(X,Thetak):
    #Your code here
    M = np.exp(X*np.log(Thetak)+(1-X)*np.log(1-Thetak))
    pmf = np.prod(M,1)
    return pmf

# THE FOLLOWING FUNCTION takes the whole matrix of data points, the vector
# of probabilites of a single cluster and the cluster probabilites. The function  returns
# the matrix of responsibilites

def responsibilities(X,P,Theta,K):
    N,D = X.shape
    R = np.zeros([N,K])

    for k in range(K):
        #Your code here

        R[:,k] = eval_bern_pdf(X,Theta[k,:])

    R *= P.T
    R /= np.sum(R,1).reshape([-1,1])

    return R

```

```python
#We test the code with the following example

N = 2
D = 5
K = 3

np.random.seed(10)

X = np.random.randint(0,2,[N,D])
Theta = np.random.rand(K,D)

print(Theta)

P = np.random.rand(K,1)
P /= np.sum(P)

print(P)

R = responsibilities(X,P,Theta,K)

# R should be a [N,K] matrix with the following values
# [[  2.20811374e-02   2.19873976e-01   7.58044887e-01]
#  [  7.49839314e-04   3.34167429e-01   6.65082732e-01]]

print(X)

print(R)
```

    [[0.22479665 0.19806286 0.76053071 0.16911084 0.08833981]
     [0.68535982 0.95339335 0.00394827 0.51219226 0.81262096]
     [0.61252607 0.72175532 0.29187607 0.91777412 0.71457578]]
    [[0.51277512]
     [0.13436922]
     [0.35285566]]
    [[1 1 0 1 0]
     [1 1 0 1 1]]
    [[2.20811374e-02 2.19873976e-01 7.58044887e-01]
     [7.49839314e-04 3.34167429e-01 6.65082732e-01]]

## Implementation of the M step

```python
# THE FOLLOWING FUNCTION computes the the soft number of points
# associated to each cluster given the matrix R

def points_cluster(R):
    #Your code here
    return np.sum(R,0)

# THE FOLLOWING FUNCTION updates the value of the cluster probabilites

def new_P(R,N):
    #Your code here
    Rk = points_cluster(R)
    return Rk/N

def new_Theta(R,X,K,alpha,beta):

    Theta = np.zeros([K,D])
    Rk = points_cluster(R)

    for k in range(K):
        #Your code here
        Theta[k,:] = (np.sum(X*R[:,k].reshape([-1,1]),0)+alpha-1) / (Rk[k]+alpha+beta-2)

    return Theta

```

```python
alpha = 2.
beta = 2.

Theta = new_Theta(R,X,K,alpha,beta)

print(Theta)

# Theta should be a [K,D] matrix with the following values
# [[ 0.50564332  0.50564332  0.49435668  0.50564332  0.49472737]
# [ 0.60846367  0.60846367  0.39153633  0.60846367  0.52237502]
#  [ 0.70786949  0.70786949  0.29213051  0.70786949  0.48642146]]

```

    [[0.50564332 0.50564332 0.49435668 0.50564332 0.49472737]
     [0.60846367 0.60846367 0.39153633 0.60846367 0.52237502]
     [0.70786949 0.70786949 0.29213051 0.70786949 0.48642146]]

## Computing the log-likelihood

```python
# THE FOLLOWING FUNCTION computes the model log-likelihood
# given the model parameters

def log_lik(X,P,Theta,K):

    LL = 0.

    for k in range(K):
        #Your code here
        LL += eval_bern_pdf(X,Theta[k,:])*P[k]

    return np.sum(np.log(LL)) #Your code here

```

```python
#For the toy example the LL should be -5.31111037373

print(log_lik(X,P,Theta,K))
```

    -5.311110373732397

## EM for a synthetic dataset

Let's generate samples from a mixtures of Bernoulli distributions and run the EM algorithm to recover the true model parameters.

```python
## True model parameters
K = 5
D = 100
N = 5000
X = []

Theta_true = np.random.rand(K,D)
P_true = np.random.rand(K,1)
P_true /= np.sum(P_true)

N_Z_true = np.random.multinomial(N,P_true.reshape([-1,]))

for k in range(K):
    X.append(np.random.rand(N_Z_true[k],D)<=Theta_true[k,:])
    print(X[k].shape)

X = np.concatenate(X)
```

    (443, 100)
    (1445, 100)
    (973, 100)
    (1135, 100)
    (1004, 100)

```python
## EM initialization
K = 5

P = np.random.rand(K,1)
P /= np.sum(P_true)

Theta = 0.5/K*np.ones([K,D])+0.5*np.random.rand(K,D)   # This initialization ensures all clusters begin with
                                                       # non-negligible probability

Num_Iter = 500

alpha = 2.1
beta = 2.1

LL = np.zeros([Num_Iter,1])

for i in range(Num_Iter):

    LL[i] = log_lik(X,P,Theta,K)

    R = responsibilities(X,P,Theta,K)

    P = new_P(R,N)

    Theta = new_Theta(R,X,K,alpha,beta)

```

```python
# Plot the evolution of the negative LL. It should always decrease!

plt.semilogy(-LL,'-*')
```

```python
plt.plot(P_true,label='true probs', color = [0, 0.5, 0])
plt.plot(P,label='EM probs', color = [1, 0, 0])
plt.legend()
plt.title('Cluster Probabilites')
```

## EM for a data set of handwritten digits

Load and run the EM over the [sklearn dataset of digits](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits). Each datapoint is a 8x8 image of a digit.

```python
from sklearn.datasets import load_digits
digits = load_digits()    #Gray scale, we have to binarize
print(digits.data.shape)

plt.gray()
plt.matshow(digits.images[15])
plt.show()

#Binarization

Bin_Images = np.copy(digits.images)

val_min=np.min(Bin_Images)
val_max=np.max(Bin_Images)

Bin_Images = (Bin_Images - val_min) / (val_max - val_min)

Bin_Images = np.round(Bin_Images)

plt.gray()
plt.matshow(Bin_Images[15])
plt.show()

```

```python
# EM

K = 10
D = 64
X = Bin_Images.reshape([-1,D])

P = np.random.rand(K,1)
P /= np.sum(P_true)

Theta = 0.5/K*np.ones([K,D])+0.5*np.random.rand(K,D)   # This initialization ensures all clusters begin with
                                                       # non-negligible probability

Num_Iter = 1000

alpha = 2.
beta = 2.

LL = np.zeros([Num_Iter,1])

for i in range(Num_Iter):

    LL[i] = log_lik(X,P,Theta,K)

    R = responsibilities(X,P,Theta,K)

    P = new_P(R,N)

    Theta = new_Theta(R,X,K,alpha,beta)

# Plot the evolution of the negative LL. It should always decrease!
plt.semilogy(-LL,'-*')

```

```python
# Plot as a 8x8 gray-scale image each theta_k vector
# (it represents the probability of each pixel to take value 1)

for k in range(K):
    plt.gray()
    plt.matshow(Theta[k,:].reshape([8,8]))
    plt.show()
```

## EM for the MNIST Database

Repeat the experiment for the MNIST Database. Take $10^4$ images **at random**, and binarize the database.

```python
from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')
```

```python
images = mnist.data  #70000 images

plt.gray()
plt.matshow(images[0,:].reshape([28,28]))
plt.show()

val_min=np.min(images)
val_max=np.max(images)

Bin_Images = (images - val_min) / (val_max - val_min)

Bin_Images = np.round(Bin_Images)

plt.gray()
plt.matshow(Bin_Images[0,:].reshape([28,28]))
plt.show()

mask = np.random.permutation(Bin_Images.shape[0])

Bin_Images = Bin_Images[mask,:]
```

```python
# EM

K = 10
D = 784
N = 1000

X = Bin_Images[:N,:]

P = np.random.rand(K,1)
P /= np.sum(P_true)

Theta = 0.5/K*np.ones([K,D])+0.5*np.random.rand(K,D)

Num_Iter = 1000

alpha = 2.1
beta = 2.1

LL = np.zeros([Num_Iter,1])

for i in range(Num_Iter):

    LL[i] = log_lik(X,P,Theta,K)

    R = responsibilities(X,P,Theta,K)

    P = new_P(R,N)

    Theta = new_Theta(R,X,K,alpha,beta)

    if(i % 20 ==0):
        print('Iteration ',i)

```

    Iteration  0
    Iteration  20
    Iteration  40
    Iteration  60
    Iteration  80
    Iteration  100
    Iteration  120
    Iteration  140
    Iteration  160
    Iteration  180
    Iteration  200
    Iteration  220
    Iteration  240
    Iteration  260
    Iteration  280
    Iteration  300
    Iteration  320
    Iteration  340
    Iteration  360
    Iteration  380
    Iteration  400
    Iteration  420
    Iteration  440
    Iteration  460
    Iteration  480
    Iteration  500
    Iteration  520
    Iteration  540
    Iteration  560
    Iteration  580
    Iteration  600
    Iteration  620
    Iteration  640
    Iteration  660
    Iteration  680
    Iteration  700
    Iteration  720
    Iteration  740
    Iteration  760
    Iteration  780
    Iteration  800
    Iteration  820
    Iteration  840
    Iteration  860
    Iteration  880
    Iteration  900
    Iteration  920
    Iteration  940
    Iteration  960
    Iteration  980

```python
# Plot the evolution of the negative LL. It should always decrease!
plt.plot(-LL,'-*')
```

```python
# Plot as a 8x8 gray-scale image each theta_k vector
# (it represents the probability of each pixel to take value 1)

for k in range(K):
    plt.gray()
    plt.matshow(Theta[k,:].reshape([28,28]))
    plt.show()
```

```python

```

Notebooks/Session 13 Mixture of Bernoullis/Session_13_EM_mix_Bern.ipynb


## Simple qubit rotation old version of TFQ, with manual GD

In this jupyter file we define a variational quantum circuit $V(\theta)$ that rotates an initial state $|0000\rangle$ into a target state with equal superposition $\sum_{\sigma_i} | \sigma_i \rangle$. The aim is that  $\langle 1111 | V(\theta) | 0000\rangle = 1$.

```python
import tensorflow as tf
import tensorflow_quantum as tfq

import cirq
import sympy
import numpy as np

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
```

### Generate symbols

I did not figure out, how I can do the gradients in TFQ without using symbols, this seems to be mandatory for tfq. I don't reallay see the advantage so far. Especially the evaluation with the resolver function seems a bit odd and unnecessary.

The definition of the circuit is almost the same as in Pennylane.

There is no option to define a projections operator to calculate the overlap with a target state because they are not unitary. This gives a bit less room to play with TFQ. I assume the idea was, that these gates are not really feasible on a real quantum device.

% Instaed of defining a hermitian matrix that gives the overlap with the target state, we can simply measure the operator $M = 1/4*(X_1 + X_2 + X_3 + X_4)$ and minimize the loss $1-\langle M \rangle$.

```python
def generate_circuit(nr_of_qubits, layers):
    qubits = cirq.GridQubit.rect(1, nr_of_qubits) # Define qubit grid. In this case
    nr_parameters = 3*nr_of_qubits*layers # 3 params for each qubit and layer

    symb = sympy.symbols('theta0:'+str(nr_parameters))
    symbols = np.array(symb)
    symbols = symbols.reshape(layers, nr_of_qubits, 3)
    circuit = cirq.Circuit()

    for l in range(layers):
        # Add a series of single qubit rotations.
        for i, qubit in enumerate(qubits):
            circuit += cirq.rz(symbols[l][i][0])(qubit)
            circuit += cirq.rx(symbols[l][i][1])(qubit)
            circuit += cirq.rz(symbols[l][i][2])(qubit)

        circuit += cirq.CZ(qubits[0], qubits[1])
        circuit += cirq.CZ(qubits[2], qubits[3])
        circuit += cirq.CZ(qubits[1], qubits[2])

    op = 1/4*(cirq.X(qubits[0]) + cirq.X(qubits[1]) + cirq.X(qubits[2]) + cirq.X(qubits[3]))
    return circuit, op, list(symb)

nr_of_qubits = 4
layers = 2
tf_circuit, op, (symbols) = generate_circuit(nr_of_qubits, layers)
SVGCircuit(tf_circuit)
```

### Training

This part took me very long to figure out, because the TFQ documentation is mostly focus on training with data. I tried to use the keras.model.fit() functions, but I did not manage to make them work without input data. There is probably some way to do it, but after a few hours I gave up and I do the gradient update manually.

The key in the following part is the function `tfq.layers.Expectation()`. We can give it as an argumnet our circuit, which has to be converted to a tf tensor, the operator that we want to optimize the expectation value of (in our case this is $1-M$). We also need to feed the list of symbols and their current values.

TF finds the grad of this circuit with respect to the parameters in 'symbols'.

The gradient descent update rule is $\theta_i^{t+1} = \theta_i^t - \eta \partial_{\theta_i} f(\theta)$

```python
circuit_tensor = tfq.convert_to_tensor([tf_circuit])

expectation = tfq.layers.Expectation()

values_tensor = tf.convert_to_tensor(np.random.uniform(0, 2 * np.pi, (1, layers* nr_of_qubits*3 )).astype(np.float32))

eta = 0.1

for i in range(200):
    with tf.GradientTape() as g:
        g.watch(values_tensor)
        forward = expectation(circuit_tensor,
                              operators=1-op,
                              symbol_names=symbols,
                              symbol_values=values_tensor)

        if i%10==0:
            print(forward.numpy()[0][0])
        # Return variance of gradients across all circuits.
        grads = g.gradient(forward, values_tensor)
        values_tensor -= eta*grads
        del grads
```

    1.1165314
    0.8423661
    0.5715227

Like in the Hello World example we can extract the wave function we see that we get the qual superposition state with some global phase.

```python
simulator = cirq.Simulator()
dictionary = {}
for i in range(len(symbols)):
    symb = symbols[i]
    dictionary[symb] = values_tensor.numpy()[0][i]
resolver = cirq.ParamResolver(dictionary)
resolved_circuit = cirq.resolve_parameters(tf_circuit, resolver)
output_state_vector = simulator.simulate(tf_circuit, resolver).final_state
output_state_vector
```

```python

```


Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2019-07-15 14:50:54 URL:https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.453017, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here

# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

    Mean absolute error: 0.04
    Residual sum of squares (MSE): 0.00
    R2-score: 0.97

Double-click __here__ for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.