# Spectral Analysis of Deterministic Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Zero-Padding

### Concept

Let's assume a signal $x_N[k]$ of finite length $N$, for instance a windowed signal $x_N[k] = x[k] \cdot \text{rect}_N[k]$. The discrete Fourier transformation (DFT) of $x_N[k]$ reads

\begin{equation}
X_N[\mu] = \sum_{k=0}^{N-1} x_N[k] \; w_N^{\mu k}
\end{equation}

where $w_N = \mathrm{e}^{-\mathrm{j} \frac{2 \pi}{N}}$ denotes the kernel of the DFT. For a sampled time-domain signal, the distance in frequency between two neighboring coefficients is given as $\Delta f = \frac{f_s}{N}$, where $f_s = \frac{1}{T}$ denotes the sampling frequency. Hence, if $N$ is increased the distance between neighboring frequencies is decreased. This leads to the concept of zero-padding in spectral analysis. Here the signal $x_N[k]$ of finite length is filled up with (M-N) zero values to a total length $M \geq N$

\begin{equation}
x_M[k] = \begin{cases}
x_N[k] & \mathrm{for} \; k=0,1,\dots,N-1 \\
0 & \mathrm{for} \; k=N,N+1,\dots,M-1
\end{cases}
\end{equation}

Appending zeros does not change the contents of the signal itself. However, the DFT $X_M[\mu]$ of $x_M[k]$ has now a decreased distance between neighboring frequencies $\Delta f = \frac{f_s}{M}$.

The question arises what influence zero-padding has on the spectrum and if it can enhance spectral analysis. On first sight it seems that the frequency resolution is higher, however do we get more information on the signal? In order to discuss this, a short numerical example is evaluated followed by a derivation of the mathematical relations between the spectrum $X_M[k]$ with zero-padding and $X_N[k]$ without zero-padding.

#### Example - Zero-Padding

The following example computes and plots the magnitude spectra $|X[\mu]|$ of a truncated complex exponential signal $x_N[k] = \mathrm{e}^{\,\mathrm{j}\,\Omega_0\,k} \cdot \text{rect}_N[k]$ and its zero-padded version $x_M[k]$.

```python
import matplotlib.pyplot as plt
import numpy as np

N = 16  # length of the signal
M = 32  # length of zero-padded signal
Om0 = 5.33*(2*np.pi/N)  # frequency of exponential signal

# DFT of the exponential signal
xN = np.exp(1j*Om0*np.arange(N))
XN = np.fft.fft(xN)
# DFT of the zero-padded exponential signal
xM = np.concatenate((xN, np.zeros(M-N)))
XM = np.fft.fft(xM)

# plot spectra
plt.figure(figsize=(10, 6))

plt.subplot(121)
plt.stem(np.arange(N), np.abs(XN), use_line_collection=True)
plt.title(r'$\mathrm{{DFT}}_{{{0}}}$ of $e^{{j \Omega_0 k}}$ without zero-padding'.format(N))
plt.xlabel(r'$\mu$')
plt.ylabel(r'$|X_N[\mu]|$')
plt.axis([0, N, 0, 18])
plt.grid()

plt.subplot(122)
plt.stem(np.arange(M), np.abs(XM), use_line_collection=True)
plt.title(r'$\mathrm{{DFT}}_{{{0}}}$ of $e^{{j \Omega_0 k}}$ with zero-padding'.format(M))
plt.xlabel(r'$\mu$')
plt.ylabel(r'$|X_M[\mu]|$')
plt.axis([0, M, 0, 18])
plt.grid()
```

**Exercise**

* Check the two spectra carefully for relations. Are there common coefficients for the case $M = 2 N$?
* Increase the length `M` of the zero-padded signal $x_M[k]$. Can you gain additional information from the spectrum?

Solution: Every second (because the DFT length has been doubled) coefficient has been added, the other coefficients stay the same. With longer zero-padding, the maximum of the main lobe of the window gets closer to its true maximum.

### Interpolation of the Discrete Fourier Transformation

Lets step back to the discrete-time Fourier transformation (DTFT) of the finite-length signal $x_N[k]$ without zero-padding

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k = -\infty}^{\infty} x_N[k] \, \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k} = \sum_{k=0}^{N-1} x_N[k] \,\mathrm{e}^{-\,\mathrm{j}\,\Omega\,k}
\end{equation}

The discrete Fourier transformation (DFT) is derived by sampling $X_N(\mathrm{e}^{\mathrm{j}\,\Omega})$ at $\Omega = \mu \frac{2 \pi}{N}$

\begin{equation}
X_N[\mu] = X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega}) \big\vert_{\Omega = \mu \frac{2 \pi}{N}} = \sum_{k=0}^{N-1} x_N[k] \, \mathrm{e}^{\,-\mathrm{j}\, \mu \frac{2\pi}{N}\,k}
\end{equation}

Since the DFT coefficients $X_N[\mu]$ are sampled equidistantly from the DTFT $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$, we can reconstruct the DTFT of $x_N[k]$ from the DFT coefficients by interpolation. Introduce the inverse DFT of $X_N[\mu]$

\begin{equation}
x_N[k] = \frac{1}{N} \sum_{\mu = 0}^{N-1} X_N[\mu] \; \mathrm{e}^{\,\mathrm{j}\,\frac{2 \pi}{N} \mu \,k}
\end{equation}

into the DTFT

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k=0}^{N-1} x_N[k] \; \mathrm{e}^{-\,\mathrm{j}\, \Omega\, k} =
\sum_{\mu=0}^{N-1} X_N[\mu] \cdot \frac{1}{N} \sum_{k=0}^{N-1} \mathrm{e}^{-\mathrm{j}\, k \,(\Omega - \frac{2 \pi}{N} \mu)}
\end{equation}

reveals the relation between $X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $X_N[\mu]$. The last sum over $k$ constitutes a [geometric series](https://en.wikipedia.org/wiki/Geometric_series) and can be rearranged to

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\mu=0}^{N-1} X_N[\mu] \cdot \frac{1}{N} \cdot \frac{1-\mathrm{e}^{-\mathrm{j}(\Omega-\frac{2\pi}{N}\mu)N}}{1-\mathrm{e}^{-\mathrm{j}(\Omega-\frac{2\pi}{N}\mu)}}
\end{equation}

By factorizing the last fraction to

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\mu=0}^{N-1} X_N[\mu] \cdot \frac{1}{N} \cdot \frac{\mathrm{e}^{-\mathrm{j}\frac{(\Omega-\frac{2\pi}{N}\mu)N}{2}}}{\mathrm{e}^{-\mathrm{j}\frac{\Omega-\frac{2\pi}{N}\mu}{2}}} \cdot \frac{\mathrm{e}^{\mathrm{j}\frac{(\Omega-\frac{2\pi}{N}\mu)N}{2}}-\mathrm{e}^{-\mathrm{j}\frac{(\Omega-\frac{2\pi}{N}\mu)N}{2}}}{\mathrm{e}^{\mathrm{j}\frac{\Omega-\frac{2\pi}{N}\mu}{2}}-\mathrm{e}^{-\mathrm{j}\frac{\Omega-\frac{2\pi}{N}\mu}{2}}}
\end{equation}

and making use of [Euler's identity](https://en.wikipedia.org/wiki/Euler%27s_identity) $2\mathrm{j}\cdot\sin(x)=\mathrm{e}^{\mathrm{j} x}-\mathrm{e}^{-\mathrm{j} x}$ this can be simplified to

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\mu=0}^{N-1} X_N[\mu] \cdot \mathrm{e}^{-\mathrm{j}\frac{(\Omega-\frac{2\pi}{N}\mu)(N-1)}{2}} \cdot \frac{1}{N} \cdot \frac{\sin(N\frac{\Omega-\frac{2\pi}{N}\mu}{2})}{\sin(\frac{\Omega-\frac{2\pi}{N}\mu}{2})}
\end{equation}

The last fraction can be written in terms of the $N$-th order periodic sinc function (aliased sinc function, [Dirichlet kernel](https://en.wikipedia.org/wiki/Dirichlet_kernel)), which is defined as

\begin{equation}
\text{psinc}_N (\Omega) = \frac{1}{N} \frac{\sin(\frac{N}{2} \Omega)}{  \sin(\frac{1}{2} \Omega)}
\end{equation}

According to this definition, the periodic sinc function is not defined at $\Omega = 2 \pi \,n$ for $n \in \mathbb{Z}$. This is resolved by applying [L'Hôpital's rule](https://en.wikipedia.org/wiki/L%27H%C3%B4pital%27s_rule) which results in $\text{psinc}_N (2 \pi \,n) = 1$ for $n \in \mathbb{Z}$.

Using the periodic sinc function, the DTFT $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ of a finite-length signal $x_N[k]$ can be derived from its DFT $X_N[\mu]$ by

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega}) =  \sum_{\mu=0}^{N-1} X_N[\mu] \cdot \mathrm{e}^{-\,\mathrm{j}\, \frac{( \Omega - \frac{2 \pi}{N} \mu ) (N-1)}{2}} \cdot \text{psinc}_N ( \Omega - \frac{2 \pi}{N} \mu )
\end{equation}

#### Example - Periodic sinc function

This example illustrates the

1. periodic sinc function, and
2. interpolation of $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ from $X_N[\mu]$ for an exponential signal using above relation.

```python
N = 16  # order of periodic sinc function
M = 1024  # number of frequency points
Om = np.linspace(-np.pi, np.pi, M)

def psinc(x, N):
    '''Periodic sinc function.'''
    x = np.asanyarray(x)
    y = np.where(x == 0, 1.0e-20, x)
    return 1/N * np.sin(N/2*y)/np.sin(1/2*y)

# plot psinc
plt.figure(figsize=(10, 8))
plt.plot(Om, psinc(Om, 16))
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\mathrm{psinc}_N (\Omega)$')
plt.grid()
```

```python
N = 16  # length of the signal
M = 1024  # number of frequency points for DTFT
Om0 = 5.33*(2*np.pi/N)  # frequency of exponential signal

# DFT of the exponential signal
xN = np.exp(1j*Om0*np.arange(N))
XN = np.fft.fft(xN)

# interpolation of DTFT from DFT coefficients
Xi = np.asarray(np.zeros(M), dtype=complex)
for mu in np.arange(M):
    Omd = 2*np.pi/M*mu-2*np.pi*np.arange(N)/N
    interpolator = psinc(Omd, N) * np.exp(-1j*Omd*(N-1)/2)
    Xi[mu] = np.sum(XN * interpolator)

# plot spectra
plt.figure(figsize=(10, 8))
ax1 = plt.gca()

plt.plot(np.arange(M)*2*np.pi/M, abs(Xi), label=r'$|X_N(e^{j \Omega})|$')
plt.stem(np.arange(N)*2*np.pi/N, abs(XN), basefmt=' ', linefmt='C1',
         markerfmt='C1o', label=r'$|X_N[\mu]|$', use_line_collection=True)
plt.title(r'DFT $X_N[\mu]$ and interpolated DTFT $X_N(e^{j \Omega})$', y=1.08)
plt.ylim([-0.5, N+2])
plt.legend()

ax1.set_xlabel(r'$\Omega$')
ax1.set_xlim([0, 2*np.pi])
ax1.grid()

ax2 = ax1.twiny()
ax2.set_xlim([0, N])
ax2.set_xlabel(r'$\mu$', color='C1')
ax2.tick_params('x', colors='C1')
```

### Relation between Discrete Fourier Transformations with and without Zero-Padding

It was already outlined above that the DFT is related to the DTFT by sampling. Hence, the DFT $X_M[\mu]$ is given by sampling the DTFT $X_M(\mathrm{e}^{\mathrm{j}\, \Omega})$ at $\Omega = \frac{2 \pi}{M} \mu$. Since the zero-padded signal $x_M[k]$ differs from $x_N[k]$ only with respect to the additional zeros, the DTFTs of both are equal

\begin{equation}
X_M(\mathrm{e}^{\mathrm{j}\, \Omega}) = X_N(\mathrm{e}^{\mathrm{j}\, \Omega})
\end{equation}

The desired relation between the DFTs $X_N[\mu]$ and $X_M[\mu]$ of the signal $x_N[k]$ and its zero-padded version $x_M[k]$ can be found by sampling the interpolated DTFT $X_N(\mathrm{e}^{\mathrm{j}\, \Omega})$ at $\Omega = \frac{2 \pi}{M} \mu$

\begin{equation}
X_M[\mu] =  \sum_{\eta=0}^{N-1} X_N[\eta] \cdot \mathrm{e}^{\,-\mathrm{j}\, \frac{( \frac{2 \pi}{M} \mu - \frac{2 \pi}{N} \eta ) (N-1)}{2}} \cdot \text{psinc}_N \left( \frac{2 \pi}{M} \mu - \frac{2 \pi}{N} \eta \right)
\end{equation}

for $\mu = 0, 1, \dots, M-1$.

Above equation relates the spectrum $X_N[\mu]$ of the original signal $x_N[k]$ to the spectrum $X_M[\mu]$ of the zero-padded signal $x_M[k]$. It essentially constitutes a bandlimited interpolation of the coefficients $X_N[\mu]$.

All spectral information of a signal of finite length $N$ is already contained in its spectrum derived from a DFT of length $N$. By applying zero-padding and a longer DFT, the frequency resolution is only virtually increased. The additional coefficients are related to the original ones by bandlimited interpolation. In general, zero-padding does not bring additional insights in spectral analysis. It may bring a benefit in special applications, for instance when estimating the frequency of an isolated harmonic signal from its spectrum. This is illustrated in the following example.

Zero-padding is also used to make a circular convolution equivalent to a linear convolution. However, there is a different reasoning behind this. Details are discussed in a [later section](../nonrecursive_filters/fast_convolution.ipynb#Linear-Convolution-by-Periodic-Convolution).

#### Example - Interpolation of the DFT

The following example shows that the coefficients $X_M[\mu]$ of the spectrum of the zero-padded signal $x_M[k]$ can be derived by interpolation from the spectrum $X_N[\mu]$.

```python
N = 16  # length of the signal
M = 32  # number of points for interpolated DFT
Om0 = 5.33*(2*np.pi/N)  # frequency of exponential signal

# periodic sinc function
def psinc(x, N):
    x = np.asanyarray(x)
    y = np.where(x == 0, 1.0e-20, x)
    return 1/N * np.sin(N/2*y)/np.sin(1/2*y)

# DFT of the exponential signal
xN = np.exp(1j*Om0*np.arange(N))
XN = np.fft.fft(xN)

# interpolation of DFT coefficients
XM = np.asarray(np.zeros(M), dtype=complex)
for mu in np.arange(M):
    Omd = 2*np.pi/M*mu-2*np.pi*np.arange(N)/N
    interpolator = psinc(Omd, N) * np.exp(-1j*Omd*(N-1)/2)
    XM[mu] = np.sum(XN * interpolator)

# plot spectra
plt.figure(figsize=(10, 6))

plt.subplot(121)
plt.stem(np.arange(N), np.abs(XN), use_line_collection=True)
plt.title(r'$\mathrm{{DFT}}_{{{0}}}$ of $e^{{j \Omega_0 k}}$ without zero-padding'.format(N))
plt.xlabel(r'$\mu$')
plt.ylabel(r'$|X_N[\mu]|$')
plt.axis([0, N, 0, 18])
plt.grid()

plt.subplot(122)
plt.stem(np.arange(M), np.abs(XM), use_line_collection=True)
plt.title(r'Interpolated spectrum')
plt.xlabel(r'$\mu$')
plt.ylabel(r'$|X_M[\mu]|$')
plt.axis([0, M, 0, 18])
plt.grid()
```

**Exercise**

* Compare the interpolated spectrum to the spectrum with zero padding from the first example.
* Estimate the frequency $\Omega_0$ of the exponential signal from the interpolated spectrum. How could you further increase the accuracy of your estimate?

Solution: The interpolated spectrum is the same as the spectrum with zero padding from the first example. The estimated frequency from the interpolated spectrum is $\Omega_0=\frac{2\pi}{M}\mu=\frac{2\pi}{32}\cdot11$. A better estimate can be obtained by increasing the number of points for the interpolated DFT or by further zero-padding of the time domain signal.

#### Example - Estimation of Frequency and Amplitude of a Harmonic Signal

The estimation of the normalized frequency $\Omega_0$ and amplitude $A$ of a single exponential signal $x_N[k] = A \cdot e^{j \Omega_0 k}$ by the DFT of the zero-padded signal (or interpolated DFT) is illustrated in the following example. The frequency is estimated from the DFT of the zero-padded signal by finding the maximum in the magnitude spectrum

\begin{equation}
\hat{\mu_0} = \underset{\mu}{\mathrm{argmax}} \{ |X_M[\mu]| \}
\end{equation}

The amplitude is estimated by taking the magnitude at the maximum $\hat{A} = | X_M[\hat{\mu}_0] |$.

First a function is defined which estimates the frequency for a given number of zeros appended to the signal before calculating the DFT. Without loss of generality is is assumed that $A=1$.

```python
N = 128  # length of the signal
Om0 = 5.33*(2*np.pi/N)  # frequency of exponential signal

# generate harmonic signal
k = np.arange(N)
x = np.exp(1j*Om0*np.arange(N))

def estimate_frequency_amplitude(x, P):
    '''Estimate frequency and amplitude of an exponential signal.'''

    # perform zero-padding and DFT
    xM = np.concatenate((x, np.zeros(P)))
    XM = np.fft.fft(xM)

    # estimate frequency/amplitude of harmonic signal
    mu_max = np.argmax(abs(XM))
    amplitude = 1/N * abs(XM[mu_max])

    # print results
    Om = np.fft.fftfreq(N+P, 1/(2*np.pi))
    print('Normalized frequency of signal: {0:1.4f} (real) / {1:1.4f} (estimated) / {2:1.4f} (absolute error)'.format(
        Om0, Om[mu_max], abs(Om0 - Om[mu_max])))
    print('Amplitude of signal: {0:1.4f} (real) / {1:1.4f} (estimated) / {2:2.2f} dB (magnitude error)'.format(
        1, amplitude, 20*np.log10(abs(1/amplitude))))
```

First the estimation is performed without zero-padding

```python
estimate_frequency_amplitude(x, 0)
```

    Normalized frequency of signal: 0.2616 (real) / 0.2454 (estimated) / 0.0162 (absolute error)
    Amplitude of signal: 1.0000 (real) / 0.8303 (estimated) / 1.62 dB (magnitude error)

Then the signal is zero-padded to a total length of eight times its original length

```python
estimate_frequency_amplitude(x, 7*N)
```

    Normalized frequency of signal: 0.2616 (real) / 0.2638 (estimated) / 0.0022 (absolute error)
    Amplitude of signal: 1.0000 (real) / 0.9967 (estimated) / 0.03 dB (magnitude error)

**Exercise**

* What is the maximum error that can occur when estimating the frequency from the maximum of the (zero-padded) magnitude spectrum?

Solution: The maximum absolute error occurs if the maximum in the DTFT of the signal is in between two adjacent bins $\mu$ of the DFT. Since the DTFT is sampled at $\Omega = \frac{2 \pi}{M}$ to derive the DFT, the maximum absolute error is given by $\frac{\pi}{M}$ where $M$ denotes the length of the zero-padded signal/DFT.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# Word Embeddings: Intro to CBOW model, activation functions and working with Numpy

In this lecture notebook you will be given an introduction to the continuous bag-of-words model, its activation functions and some considerations when working with Numpy.

Let's dive into it!

```python
import numpy as np
```

# The continuous bag-of-words model

The CBOW model is based on a neural network, the architecture of which looks like the figure below, as you'll recall from the lecture.

<div style="width:image width px; font-size:100%; text-align:center;"> Figure 1 </div>

## Activation functions

Let's start by implementing the activation functions, ReLU and softmax.

### ReLU

ReLU is used to calculate the values of the hidden layer, in the following formulas:

\begin{align}
 \mathbf{z_1} &= \mathbf{W_1}\mathbf{x} + \mathbf{b_1}  \tag{1} \\
 \mathbf{h} &= \mathrm{ReLU}(\mathbf{z_1})  \tag{2} \\
\end{align}

Let's fix a value for $\mathbf{z_1}$ as a working example.

```python
# Define a random seed so all random outcomes can be reproduced
np.random.seed(10)

# Define a 5X1 column vector using numpy
z_1 = 10*np.random.rand(5, 1)-5

# Print the vector
z_1
```

    array([[ 2.71320643],
           [-4.79248051],
           [ 1.33648235],
           [ 2.48803883],
           [-0.01492988]])

Notice that using numpy's `random.rand` function returns a numpy array filled with values taken from a uniform distribution over [0, 1). Numpy allows vectorization so each value is multiplied by 10 and then substracted 5.

To get the ReLU of this vector, you want all the negative values to become zeros.

First create a copy of this vector.

```python
# Create copy of vector and save it in the 'h' variable
h = z_1.copy()
```

Now determine which of its values are negative.

```python
# Determine which values met the criteria (this is possible because of vectorization)
h < 0
```

    array([[False],
           [ True],
           [False],
           [False],
           [ True]])

You can now simply set all of the values which are negative to 0.

```python
# Slice the array or vector. This is the same as applying ReLU to it
h[h < 0] = 0
```

And that's it: you have the ReLU of $\mathbf{z_1}$!

```python
# Print the vector after ReLU
h
```

    array([[2.71320643],
           [0.        ],
           [1.33648235],
           [2.48803883],
           [0.        ]])

**Now implement ReLU as a function.**

```python
# Define the 'relu' function that will include the steps previously seen
def relu(z):
    result = z.copy()
    result[result < 0] = 0
    return result
```

**And check that it's working.**

```python
# Define a new vector and save it in the 'z' variable
z = np.array([[-1.25459881], [ 4.50714306], [ 2.31993942], [ 0.98658484], [-3.4398136 ]])

# Apply ReLU to it
relu(z)
```

    array([[0.        ],
           [4.50714306],
           [2.31993942],
           [0.98658484],
           [0.        ]])

Expected output:

    array([[0.        ],
           [4.50714306],
           [2.31993942],
           [0.98658484],
           [0.        ]])

### Softmax

The second activation function that you need is softmax. This function is used to calculate the values of the output layer of the neural network, using the following formulas:

\begin{align}
 \mathbf{z_2} &= \mathbf{W_2}\mathbf{h} + \mathbf{b_2}   \tag{3} \\
 \mathbf{\hat y} &= \mathrm{softmax}(\mathbf{z_2})   \tag{4} \\
\end{align}

To calculate softmax of a vector $\mathbf{z}$, the $i$-th component of the resulting vector is given by:

$$ \textrm{softmax}(\textbf{z})_i = \frac{e^{z_i} }{\sum\limits_{j=1}^{V} e^{z_j} }  \tag{5} $$

Let's work through an example.

```python
# Define a new vector and save it in the 'z' variable
z = np.array([9, 8, 11, 10, 8.5])

# Print the vector
z
```

    array([ 9. ,  8. , 11. , 10. ,  8.5])

You'll need to calculate the exponentials of each element, both for the numerator and for the denominator.

```python
# Save exponentials of the values in a new vector
e_z = np.exp(z)

# Print the vector with the exponential values
e_z
```

    array([ 8103.08392758,  2980.95798704, 59874.1417152 , 22026.46579481,
            4914.7688403 ])

The denominator is equal to the sum of these exponentials.

```python
# Save the sum of the exponentials
sum_e_z = np.sum(e_z)

# Print sum of exponentials
sum_e_z
```

    97899.41826492078

And the value of the first element of $\textrm{softmax}(\textbf{z})$ is given by:

```python
# Print softmax value of the first element in the original vector
e_z[0]/sum_e_z
```

    0.08276947985173956

This is for one element. You can use numpy's vectorized operations to calculate the values of all the elements of the $\textrm{softmax}(\textbf{z})$ vector in one go.

**Implement the softmax function.**

```python
# Define the 'softmax' function that will include the steps previously seen
def softmax(z):
    e_z = np.exp(z)
    sum_e_z = np.sum(e_z)
    return e_z / sum_e_z
```

**Now check that it works.**

```python
# Print softmax values for original vector
softmax([9, 8, 11, 10, 8.5])
```

    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])

Expected output:

    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])

Notice that the sum of all these values is equal to 1.

```python
# Assert that the sum of the softmax values is equal to 1
np.sum(softmax([9, 8, 11, 10, 8.5])) == 1
```

    True

## Dimensions: 1-D arrays vs 2-D column vectors

Before moving on to implement forward propagation, backpropagation, and gradient descent in the next lecture notebook, let's have a look at the dimensions of the vectors you've been handling until now.

Create a vector of length $V$ filled with zeros.

```python
# Define V. Remember this was the size of the vocabulary in the previous lecture notebook
V = 5

# Define vector of length V filled with zeros
x_array = np.zeros(V)

# Print vector
x_array
```

    array([0., 0., 0., 0., 0.])

This is a 1-dimensional array, as revealed by the `.shape` property of the array.

```python
# Print vector's shape
x_array.shape
```

    (5,)

To perform matrix multiplication in the next steps, you actually need your column vectors to be represented as a matrix with one column. In numpy, this matrix is represented as a 2-dimensional array.

The easiest way to convert a 1D vector to a 2D column matrix is to set its `.shape` property to the number of rows and one column, as shown in the next cell.

```python
# Copy vector
x_column_vector = x_array.copy()

# Reshape copy of vector
x_column_vector.shape = (V, 1)  # alternatively ... = (x_array.shape[0], 1)

# Print vector
x_column_vector
```

    array([[0.],
           [0.],
           [0.],
           [0.],
           [0.]])

The shape of the resulting "vector" is:

```python
# Print vector's shape
x_column_vector.shape
```

    (5, 1)

So you now have a 5x1 matrix that you can use to perform standard matrix multiplication.

**Congratulations on finishing this lecture notebook!** Hopefully you now have a better understanding of the activation functions used in the continuous bag-of-words model, as well as a clearer idea of how to leverage Numpy's power for these types of mathematical computations.

In the next lecture notebook you will get a comprehensive dive into:

- Forward propagation.

- Cross-entropy loss.

- Backpropagation.

- Gradient descent.

**See you next time!**

NLP/Natural Language Processing with Probabilistic Models/4/NLP_C2_W4_lecture_notebook_model_architecture.ipynb


## Chapter 4 -  Training Models

### Regularised Linear Models

A good way to reduce overfitting is to regularise the model (to constrain it). The fewer degrees of freedom the model has, the harder it will be to overfit the data. As seen earlier, one way to regularise a model is to reduce the number of polynomial degrees.

There are other ways to regularise the model. We can fit a model containing all $p$ predictors using a technique to constraing or regularise the estimates, or shrink the coefficient estimates towards zero. This can significantly reduce the variance. Three ways to do so are Ridge regression, Lasso regression and Elastic Net.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
```

#### Least Squares Recap

Recall that for least squares, to obtain the parameter estimates, we minimise the cost function, which is the MSE or RSS:

$$\begin{align}J(\Theta) = \text{RSS} (\mathbf X) &= \sum_{i=1}^n \begin{pmatrix} y_i -  \hat{y_i} \end{pmatrix}^2
\\&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2\end{align}$$

Letting $\Theta = \begin{pmatrix}\beta_0, \beta_1, \cdots, \beta_p\end{pmatrix}^T$ and $\mathbf x_i = \begin{pmatrix}1, x_{i1}, \cdots, x_{ip}\end{pmatrix}^T$, then $\text{RSS} = \sum_{i=1}^n \begin{pmatrix} y_i - \Theta\mathbf x_i\end{pmatrix}^2$

#### Ridge Regression

Ridge regression modifies the cost function by adding a penalty term:

$$\lambda \sum_{j=1}^p \beta^2_j$$

and the RSS now becomes:

$$\begin{align}J(\Theta)
&= \sum_{i=1}^n\begin{pmatrix} y_i - \hat{y_i} \end{pmatrix}^2 + \lambda \sum_{j=1}^p \beta^2_j\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2 + \lambda \sum_{j=1}^p\beta^2_j\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \Theta\mathbf x_i\end{pmatrix}^2 + \lambda \sum_{j=1}^p\beta^2_j\\
&= \text{RSS} + \lambda \sum_{j=1}^p\beta^2_j\end{align}$$

The hyperparameter $\lambda$ controls how much to regularise the model. The penalty term $\lambda \sum_{j=1}^p \beta_j^2$ is called the <u>shrinkage penalty</u>. This shrinkage penalty has an effect of shrinking the coefficients towards zero.

The tuning parameter $\lambda$ controls the relative impact of the terms from the RSS and the shrinkage penalty on the coefficient estimate.
- When $\lambda=0$, $\lambda \sum_{j=1}^p \beta_j^2=0$. There is no shrinkage effect and the estimates for $\hat{\Theta}$ are optimised at the lowest RSS. This means ridge regression is the same as linear regression.
- As $\lambda$ grows, the shrinkage penalty kicks in. The values of $\beta_j$ starts to shrink. When $\lambda \rightarrow \infty$, the impact of the shrinkage penalty becomes very large, causing the ridge regression coefficient estimates to shrink zero. Consequently, different values of $\lambda$ will produce different $\hat{\Theta}$ estimates and the ideal $\lambda$ can be found using cross-validation. Note that the bias term $\beta_0$ is not regularized.

Before applying ridge regression, it is ideal to standardize the predictors before performing any model fitting.

```python
# Generate y = 4X with noise
m = 100
X = np.random.normal(30,10,m).flatten()
X_b = np.c_[np.ones((m,1)), X]
y = 4*X + np.random.normal(0,25,m).flatten()
df_Xy = pd.DataFrame({'X' : X, 'y' : y})
```

```python
fig, ax = plt.subplots(figsize=(10,4))
df_Xy.plot(kind='scatter', x='X', y='y', ax=ax)
ax.set_xlim(0,60)
ax.set_ylim(0,300)
plt.show()
```

```python
# Test the model using new X_test
lin_X = np.linspace(X.min(), X.max(),m)
print(y.mean())
```

    126.72999947464523

```python
# Linear Regression
reg1 = LinearRegression()
reg1.fit(X_b, y) # Train
print(reg1.intercept_, reg1.coef_)
```

    7.618618837359307 [0.         3.83369237]

```python
# Ridge with different alpha values
reg2 = Ridge(alpha=10000, solver='cholesky')
reg2.fit(X_b, y)
print(reg2.intercept_, reg2.coef_)
```

    73.56739841053806 [0.         1.71107964]

```python
# Ridge with different alpha values
reg3 = Ridge(alpha=100000, solver='cholesky')
reg3.fit(X_b, y)
print(reg3.intercept_, reg3.coef_)
```

    117.84447742977554 [0.         0.28598743]

```python
lin_X_1 = np.c_[np.ones((m,1)), lin_X]
pred1 = reg1.predict(lin_X_1)
df_results1 = pd.DataFrame({'X' : lin_X, 'y' : pred1})
# print(df_results1.head()) # Test
```

```python
lin_X_2 = np.c_[np.ones((m,1)), lin_X]
pred2 = reg2.predict(lin_X_2)
df_results2 = pd.DataFrame({'X' : lin_X, 'y' : pred2})
# print(df_results2.head()) # Test
```

```python
lin_X_3 = np.c_[np.ones((m,1)), lin_X]
pred3 = reg3.predict(lin_X_3)
df_results3 = pd.DataFrame({'X' : lin_X, 'y' : pred3})
# print(df_results3.head()) # Test
```

```python
# Plotting for different alpha values. Observe how as alpha increases, the coefficient shrinks.
fig, ax = plt.subplots(figsize=(12,6))
df_Xy.plot(kind='scatter', x='X', y='y', ax=ax, color='SKYBLUE')

df_results1.plot(kind='line', x='X', y='y', ax=ax, color='DARKKHAKI')
df_results2.plot(kind='line', x='X', y='y', ax=ax, color='DARKORANGE', style='--')
df_results3.plot(kind='line', x='X', y='y', ax=ax, color='ORCHID', style='--')
ax.legend(['linreg, alpha=0','alpha=10000', 'alpha=100000', 'sample'])
ax.set_xlim(0,60)
ax.set_ylim(0,300)
plt.show()
```

Ridge regression's advantage over least-squares is due to the bias-variance trade-off. As $\lambda$ increases, the flexibility of the ridge regression decreases, leading to a decrease in variance but increase in bias.

#### Lasso Regression

Ridge regression still includes all $p$ predictors in the model. The penalty $\lambda \sum \beta_j^2$ will shrink all coefficients towards zero but not set any of them exactly to zero. This can create a challenge in model interpretability.

We introduce the lasso regression method as a way to overcome this disadvantage. Least Absolute Shrinkage and Selection Operator Regression (Lasso Regression) is another regularized form of Linear Regression. Like Ridge regression, it adds a penalty term, now $\lambda\sum_{j=1}^p |\beta_j|$ to the cost function:

$$\begin{align}J(\Theta)
&= \sum_{i=1}^n\begin{pmatrix} y_i - \hat{y_i} \end{pmatrix}^2 + \lambda \sum_{j=1}^p |\beta_j|\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2 + \lambda \sum_{j=1}^p|\beta_j|\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \Theta\mathbf x_i\end{pmatrix}^2 + \lambda \sum_{j=1}^p|\beta_j|\\
&= \text{RSS} + \lambda \sum_{j=1}^p|\beta_j|\end{align}$$

Consider that ridge regression involves the $\beta_j^2$ term while lasso regression involves the $|\beta_j|$ term. We say that the ridge uses the $l_2$ penalty while the lasso uses the $l_1$ penalty. Recall that the norm is the sum of all vectors in a space. So the $l_1$ norm is simply $\sum_j |\beta_j|$ while the $l_2$ norm is $\sum_j \beta^2_j$

Similar to ridge regression, lasso shrinks the coefficient estimates towards zero. However, the $l_1$ penalty forces some of the coefficient estimates to be exactly equal to zero when $\lambda$ is sufficiently large. So the lasso also performs variable selection. Hence, models from the lasso are generally much easier to interpret than those produced by ridge regression. The lasso yields sparse models, which are models that involve only a subset of the variables.

To select good values for $\lambda$ for both ridge and lasso regression, we use cross-validation.

```python
regb1 = Lasso(alpha=0.1)
regb1.fit(X_b, y)
```

    Lasso(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=1000,
          normalize=False, positive=False, precompute=False, random_state=None,
          selection='cyclic', tol=0.0001, warm_start=False)

```python
regb2 = Lasso(alpha=100)
regb2.fit(X_b, y)
```

    Lasso(alpha=100, copy_X=True, fit_intercept=True, max_iter=1000,
          normalize=False, positive=False, precompute=False, random_state=None,
          selection='cyclic', tol=0.0001, warm_start=False)

```python
lin_X_b1 = np.c_[np.ones((m,1)), lin_X]
predb1 = regb1.predict(lin_X_b1)
df_resultsb1 = pd.DataFrame({'X' : lin_X, 'y' : predb1})

predb2 = regb2.predict(lin_X_b1)
df_resultsb2 = pd.DataFrame({'X' : lin_X, 'y' : predb2})
```

```python
# Plotting for different alpha values. Observe how as alpha increases, the coefficient shrinks.
fig, ax = plt.subplots(figsize=(12,6))
df_Xy.plot(kind='scatter', x='X', y='y', ax=ax, color='SKYBLUE')

df_results1.plot(kind='line', x='X', y='y', ax=ax, color='DARKKHAKI')
df_resultsb2.plot(kind='line', x='X', y='y', ax=ax, color='DARKORANGE', style='--')
ax.legend(['linreg, alpha=0','alpha=100', 'sample'])
ax.set_xlim(0,60)
ax.set_ylim(0,300)
plt.show()
```

#### Elastic Net

Elastic Net is a middle ground between Ridge Regression and Lasso Regression. The regularisation term is a mix of the both ridge and lasso regularisation term and you can control the mix ratio $r$. When $r=0$ Elastic net is equivalent to Ridge regression while when $r=1$ then Elastic net is equivalent to Lasso regression:

The cost function , $J(\Theta)$ in terms of RSS is:

$$\begin{align}J(\Theta) &= \text{RSS} + r\lambda \sum_{j=1}^p\beta^2_j + (1-r)\lambda \sum_{j=1}^p|\beta_j|\\&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2 + r\lambda \sum_{j=1}^p\beta^2_j + (1-r)\lambda \sum_{j=1}^p|\beta_j|\end{align}$$

```python
regc1 = ElasticNet(alpha=0.1, l1_ratio=0.5)
regc1.fit(X_b, y)
```

    ElasticNet(alpha=0.1, copy_X=True, fit_intercept=True, l1_ratio=0.5,
               max_iter=1000, normalize=False, positive=False, precompute=False,
               random_state=None, selection='cyclic', tol=0.0001, warm_start=False)


## This notebook is meant to capture experience with Jeffery's Priors
https://en.wikipedia.org/wiki/Jeffreys_prior

```python
import pymc3 as pm
import theano
import numpy as np
import scipy.stats as ss
import seaborn as sns

sns.set(font_scale=1.5)
```

# Guassian mean

$f(x\mid\mu) = \frac{e^{-(x - \mu)^2 / 2\sigma^2}}{\sqrt{2 \pi \sigma^2}}$

with $\sigma$ fixed, the Jeffreys prior for the mean $\mu$ is
: $\begin{align} p(\mu) & \propto \sqrt{I(\mu)}
= \sqrt{\operatorname{E}\!\left[ \left( \frac{d}{d\mu} \log f(x\mid\mu) \right)^2\right]}
= \sqrt{\operatorname{E}\!\left[ \left( \frac{x - \mu}{\sigma^2} \right)^2 \right]} \\\\
& = \sqrt{\int_{-\infty}^{+\infty} f(x\mid\mu) \left(\frac{x-\mu}{\sigma^2}\right)^2 dx}
=  \sqrt{1/\sigma^2} \propto 1.\end{align}

# Gaussian sd

$f(x\mid\sigma) = \frac{e^{-(x - \mu)^2 / 2 \sigma^2}}{\sqrt{2 \pi \sigma^2}}$,

with $\mu$ fixed, the Jeffreys prior for the standard deviation $\sigma > 0$ is
: $\begin{align}p(\sigma) & \propto \sqrt{I(\sigma)}
= \sqrt{\operatorname{E}\!\left[ \left( \frac{d}{d\sigma} \log f(x\mid\sigma) \right)^2\right]}
= \sqrt{\operatorname{E}\!\left[ \left( \frac{(x - \mu)^2-\sigma^2}{\sigma^3} \right)^2 \right]} \\\\
& = \sqrt{\int_{-\infty}^{+\infty} f(x\mid\sigma)\left(\frac{(x-\mu)^2-\sigma^2}{\sigma^3}\right)^2 dx}
= \sqrt{\frac{2}{\sigma^2}}
\propto \frac{1}{\sigma}.
\end{align}

### Example model

```python
data = np.random.normal(10, 2, size=20)
sns.distplot(data)
```

```python
def jeffreys(value):
    return -theano.tensor.log(value)

with pm.Model() as model:
    sigma = pm.DensityDist('sigma', jeffreys, testval=1)
    mu = pm.DensityDist('mu', jeffreys, testval=1)

```

# Poisson

```python
data = np.random.poisson(20, size=10)
sns.distplot(data)
```

```python
def jeffreys(value):
    return -theano.tensor.log(value)

with pm.Model() as model:
    mean = pm.DensityDist('mean', jeffreys, testval=1)
    poi = pm.Poisson('Poisson', mean, observed=data)

    mean_flat = pm.Uniform('mean_flat', 0, 100)
    poi_flat = pm.Poisson('Poisson_flat', mean_flat, observed=data)

    trace = pm.sample(10000, tune=1000)

```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [mean_flat, mean]
    Sampling 4 chains, 0 divergences: 100%|██████████| 44000/44000 [00:08<00:00, 5069.74draws/s]

```python
pm.summary(trace)
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      mean
      18.900
      1.377
      16.334
      21.474
      0.007
      0.005
      35207.0
      34612.0
      35565.0
      27445.0
      1.0

      mean_flat
      19.015
      1.389
      16.434
      21.613
      0.007
      0.005
      34386.0
      34217.0
      34498.0
      28372.0
      1.0

```python
pm.traceplot(trace)
```

```python
pm.plot_posterior(trace)
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python
# Make custom Jeffreys prior, returns log prior
def jeffreys(value):
    return -theano.tensor.log(value)

# Instantiate model
model = pm.Model()
with model:
    # Priors for unknown model parameters
    sigma = pm.DensityDist('sigma', jeffreys, testval=1)
    slope = pm.Uniform('slope', lower=-1, upper=1, testval=0.5)
    intercept = pm.Uniform('intercept', lower=-10, upper=10, testval=0)

    # Expected value of outcome
    mu = slope * df.parent_bd + intercept

    # Likelihood (sampling distribution) of observations
    y_obs = pm.Normal('y_obs', mu=mu, sd=sigma, observed=df.offspring_bd)

    # draw 10000 posterior samples for burn-in
    trace = pm.sample(10000)

```

```python
pm.summary(trace, ['slope', 'intercept', 'sigma'])
```

```python
pm.traceplot(trace, ['slope', 'intercept', 'sigma']);
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

Jeffreys prior/Jeffreys prior examples.ipynb


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *; x,h,t  = symbols("x h t")
Image(url= "https://i.imgur.com/p8p6QeA.png")
```

```python
expr = -0.8 / sqrt(6+4*x)
plot(expr)
```

```python
dexpr = (diff(expr))  #getting derivative of our expression
dexpr.subs(x,4)
```

$\displaystyle 0.00330578512396694 \sqrt{22}$

```python
import numpy as np
MofTan = 0.0033057851239669422*np.sqrt(22)
```

```python
-0.170560573084488 + MofTan*(x-4)
```

$\displaystyle 0.0155055066440444 x - 0.232582599660666$

```python
Image(url= "https://i.imgur.com/LPbplPW.png")
```


In this notebook, we'll explore how images are represented by the computer. We'll learn how to load, examine and manipulate images, and to perform basic pre-processing actions.

# Basic Operations and Image Examination

We start by loading the packages.
We will use:

*    **matplotlib** - a package for plotting graphs and images
*    **numpy** - a package for numerical processing of matrices, and
*    **scikit-learn image** - a package for processing scientific images, which also includes scientific images examples.

```python
import cv2 as cv
import matplotlib.pyplot as plt
import numpy as np

from skimage import data
```

```python
# skimage.data has many different image samples. Check them out here: https://scikit-image.org/docs/stable/api/skimage.data.html
# You can play with this notebook, by changing the data image to other images and examin the result.
img = data.brain()[0]
plt.imshow(img)
```

```python
# we can also plot it in black & white by setting the colormap (cmap)
plt.imshow(img, cmap=plt.cm.gray)
```

```python
# The shape of the matrix is the width and height in Pixels
print(img.shape)
img_width, img_height = img.shape
```

    (256, 256)

```python
# We can print the matrix itself:
img
```

    array([[4, 4, 4, ..., 4, 4, 4],
           [4, 4, 4, ..., 4, 4, 4],
           [4, 4, 4, ..., 4, 4, 4],
           ...,
           [4, 4, 4, ..., 4, 4, 4],
           [4, 4, 4, ..., 4, 4, 4],
           [4, 4, 4, ..., 4, 4, 4]], dtype=uint16)

```python
# or see and use parts of it using python slices
img[100:120, 50:58]
```

    array([[19044, 16900, 13456,  9604,  7744,  7225,  6889,  7056],
           [18496, 15625, 10816,  7921,  7569,  7225,  7056,  6889],
           [17424, 15376, 11025,  8100,  7396,  7225,  7396,  7056],
           [17161, 15129, 11449,  8464,  7225,  7056,  7225,  7225],
           [17689, 15129, 10609,  7921,  7056,  6889,  6889,  7225],
           [17956, 14161,  9409,  7396,  7056,  6889,  6889,  7225],
           [17689, 12321,  8100,  7225,  7056,  7056,  7056,  7056],
           [18496, 11025,  7569,  7056,  7056,  6889,  7225,  7056],
           [18769, 10201,  7225,  7056,  6889,  6889,  7056,  7056],
           [18225,  9604,  7225,  7056,  7056,  6889,  6724,  6889],
           [15625,  8649,  7056,  7056,  7056,  6889,  6889,  7056],
           [12321,  7921,  6889,  6889,  6724,  6724,  7056,  7056],
           [ 9801,  7396,  6889,  6889,  6561,  6724,  6889,  6889],
           [ 7921,  7056,  6724,  6889,  6724,  6724,  6889,  6889],
           [ 7225,  7056,  6889,  6889,  6889,  6724,  6724,  6889],
           [ 7225,  6889,  7056,  6889,  6889,  6724,  6889,  6889],
           [ 7225,  7056,  6889,  6889,  6724,  6889,  6889,  7056],
           [ 7056,  6889,  6889,  6889,  6889,  6889,  6889,  7056],
           [ 7056,  7056,  7056,  6889,  6889,  6889,  6889,  7056],
           [ 6889,  6724,  7056,  6889,  6889,  7056,  7225,  6889]],
          dtype=uint16)

```python
# the values of the matrix corresponds to the color value of the pixel at that point
print(img.max())
print(img.min())
```

    49284
    0

## Histogram
A histogram is sort of a graph or plot, which can give you an overall idea about the intensity distribution of an image. It is a plot with pixel values (ranging from 0 to 255, though not always) in X-axis and corresponding number of pixels in the image on Y-axis.

```python
# Let's get the histogram using numpy:
hist, bins = np.histogram(img, bins=range(img.max()), range=[0, img.max()])
hist
```

    array([ 595, 3208,    0, ...,    0,    0,    0])

```python
# And plot it using matplotlib
plt.title("Grayscale Histogram")
plt.xlabel("grayscale value")
plt.ylabel("amt. of pixels")

plt.plot(hist)
plt.show()
```

Another way to think or imagine of the color values is as a 3D surface map: black (or 0) is a low valley, while brighter colors (up to white) are mountains:

```python
# create the x and y coordinate arrays (here we just use pixel indices)
xx, yy = np.mgrid[0:img_width, 0:img_height]

# create the figure
fig = plt.figure()

# gca = Get the Current Axes (or create new if)
ax = fig.gca(projection='3d')
ax.plot_surface(xx, yy, img ,rstride=1, cstride=1, cmap=plt.cm.gray, linewidth=0)

# show it
plt.show()

```

# Image Processing

Image processing is done thorugh a function that takes one or more images as an input and produces an output image.

Image transforms are generally divided to two:
* Point operators: Operators that transform pixels values individually. For example, changing the brightness, adjusting contrast or correcting the image colors.
* Neighborhood (area-based) operators - transform a pixel based on its neighbor pixels

## Pixel Operations
Since images are represented as 2D matrices, we can manupulate or generate images by "turning on or off" pixels in the image.

```python
img_width, img_height = img.shape

mask = np.zeros(img.shape[:2], np.uint8)
mask_width, mask_height = img_width//4, img_height//4
mask[mask_width:img_width-mask_width, mask_height:img_height-mask_height] = img.max()
plt.imshow(mask, cmap='gray')
```

```python
# And we can fuse two images together with byte operations. For example, AND operator would result in '0's where either the mask or the image was black (0).
masked_img = cv.bitwise_and(img,img,mask = mask)
plt.imshow(masked_img, cmap='gray')
```

###Linear blend operator
Images can also be blnded linearly, giving more weight to one image over the other.

$g(x) = (1-\alpha) f_0(x) + \alpha f_1(x)$

By varying α from 0→1 this operator can be used to perform a temporal cross-dissolve between two images or videos, as seen in slide shows and film productions.

```python
alpha = 0.5
beta = (1.0 - alpha)
img1 = data.brain()[0]
img2 = data.brain()[5]
dst = cv.addWeighted(img1, alpha, img2, beta, 0.0)

plt.figure(1)
plt.subplot(221)
plt.imshow(img1)
plt.subplot(222)
plt.imshow(img2)
plt.subplot(223)
plt.imshow(dst)
```

## Contrast

### Linear manipulation
Two commonly used point processes are a multiplication by a factor and an addition with a constant:
$g(x) = \alpha f(x) + \beta$

The parameters $alpha>0$ and $\beta$ are often called the **gain** and **bias** parameters; sometimes these parameters are said to control *contrast* and *brightness* respectively.
You can think of $f(x)$ as the source image pixels and $g(x)$ as the output image pixels. Then, more conveniently we can write the expression as:

$g(i,j)=\alpha⋅f(i,j)+\beta$

where $i$ and $j$ indicates that the pixel is located in the $i$-th row and $j$-th column.

```python
def set_contrast_brightness(img, alpha = 1.0, beta = 0):
  out_img = np.zeros(img.shape, img.dtype)
  ############################## YOUR TURN: ##############################
  """
  Implement the function g(x)=αf(x)+β with pixel operations and numpy.
  Try implementing it first using two loops, for the rows and columns.
  Then, see how you can use Numpy function for matrix operations.
  Which method was faster?

  Attention: pixels color values must be integers, and in the range of 0 to 255.
  The numpy functions `clip` and `rint` might be useful.
  """

  ### END ###

  return out_img

alpha = 2.2  # Simple contrast control
beta = 50.    # Simple brightness control

orig_img = data.human_mitosis()
%time new_img = set_contrast_brightness(orig_img, alpha, beta)

plt.figure(1)
plt.subplot(121)
plt.imshow(orig_img)
plt.subplot(122)
plt.imshow(new_img)
plt.show()
```

```python
################################# YOUR TURN: #################################
### Plot the histogram of the original and the altered human_mitosis image ###

##############################################################################
```

```python
# The OpenCV package has implemented such method - convertScaleAbs.
# Ensure your method operates correctly:

alpha, beta = 2.0, 50
assert (cv.convertScaleAbs(orig_img, alpha=alpha, beta=beta) == set_contrast_brightness(orig_img, alpha, beta)).all()

alpha, beta = 1.0, 0
assert (cv.convertScaleAbs(orig_img, alpha=alpha, beta=beta) == set_contrast_brightness(orig_img, alpha, beta)).all()

alpha, beta = 3.0, 250
assert (cv.convertScaleAbs(orig_img, alpha=alpha, beta=beta) == set_contrast_brightness(orig_img, alpha, beta)).all()
```

### Gamma Correction
Gamma Correction can be used to correct the brightness of an image by using a **non linear** transformation between the input values and the mapped output values using this function:

\begin{align}
        O = \left ( \frac{I}{255} \right ) ^\gamma × 255
    \end{align}

As this relation is non linear, the effect will not be the same for all the pixels and will depend to their original value.

When $γ<1$, the original dark regions will be brighter and the histogram will be shifted to the right whereas it will be the opposite with $γ>1$.

Let's see it in action:

```python
def gamma(img, gamma=1.0):
  return np.clip(pow(img / 255.0, gamma) * 255.0, 0, 255)

def plot_rgb_hist(img):
    histr = cv.calcHist([img],[0],None,[256],[0,256])
    plt.plot(histr)
    plt.xlim([0,256])

orig_img = data.human_mitosis()

plt.figure(1)
plt.subplot(231)
plt.imshow(orig_img, cmap='gray')
plt.subplot(232)
plt.imshow(gamma(orig_img, 0.5), cmap='gray')
plt.subplot(233)
plt.imshow(gamma(orig_img, 0.1), cmap='gray')

plt.subplot(234)
plt.hist(orig_img.ravel(),256,[0,256]);
plt.subplot(235)
plt.hist(gamma(orig_img, 0.5).ravel(),256,[0,256]);
plt.subplot(236)
plt.hist(gamma(orig_img, 0.5).ravel(),256,[0,256]);

plt.figure(2)
plt.subplot(231)
plt.imshow(gamma(orig_img, 1.1), cmap='gray')
plt.subplot(232)
plt.imshow(gamma(orig_img, 1.7), cmap='gray')
plt.subplot(233)
plt.imshow(gamma(orig_img, 2.0), cmap='gray')
plt.subplot(234)
plt.hist(gamma(orig_img, 1.1).ravel(),256,[0,256]);
plt.subplot(235)
plt.hist(gamma(orig_img, 1.7).ravel(),256,[0,256]);
plt.subplot(236)
plt.hist(gamma(orig_img, 2.0).ravel(),256,[0,256]);

plt.show()
```

### Contrast Stretching

As we've seen before, contrast Enhancement methods can be divided into Linear and Non-Linear ones. Contrast Stretching belongs to the **Linear** family. Other non-linear methods to adjust the contrast automatically includes [Histogram Equilisation](https://en.wikipedia.org/wiki/Histogram_equalization) and [Gaussian Stretch](https://www.l3harrisgeospatial.com/docs/backgroundstretchtypes.html).

Contrast enhancement is represented well in the following image:

Contrast stretching as the name suggests is an image enhancement technique that tries to improve the contrast by stretching the intensity values of an image to fill the entire dynamic range. The transformation function used is always linear and monotonically increasing.

A typical transformation function for contrast stretching looks something like this:

By changing the location of points (r1, s1) and (r2, s2), we can control the shape of the transformation function. For example,

* When $r1 =s1$ and $r2=s2$, transformation is a **Linear** function.
* When $r1=r2$, $s1=0$ and $s2=L-1$, transformation becomes a **thresholding** function.
* When $(r1, s1) = (rmin, 0)$ and $(r2, s2) = (rmax, L-1)$, this is known as **Min-Max** Stretching.
* When $(r1, s1) = (rmin + c, 0)$ and $(r2, s2) = (rmax – c, L-1)$, this is known as **Percentile** Stretching.

Let’s understand Min-Max and Percentile Stretching in detail.

The general formula for Contrast Stretching is
\begin{align}
        S = (r - r_{min}) \left ( \frac{C_{max} - C_{min}}{r_{max} - r_{min}} \right ) + C_{min}
    \end{align}

Where $C_{max}$ and $C_{min}$ are the maxmimum and minimum **possible** color values (normally 255 and 0) and the $r_{max}$ and $r_{min}$ are the maximal and minimal color values that appear in the image itself.

For the normal scale of colors (0 - 255) the formula can be simplified to:
\begin{align}
        S = 255 \times \left ( \frac{r - r_{min}}{r_{max} - r_{min}} \right ) + C_{min}
    \end{align}

This method is also known as a **linear scaling** or **normalization** method, and is also widely used in Machine Learning for any other sort of data (e.g. tabular).

```python
def contrast_stretch(img):
  new_img = np.zeros(img.shape, img.dtype)
  ### YOUR TURN ###
  ### Implement the contrast stretching method

  ### END ###

  return new_img

orig_img = data.microaneurysms()
streched_img = contrast_stretch(orig_img)

plt.figure(1, figsize=[10,12])
plt.subplot(221)
plt.imshow(orig_img, cmap='gray')
plt.subplot(222)
plt.imshow(streched_img, cmap='gray')

plt.subplot(223)
plt.hist(orig_img.ravel(),256);
plt.subplot(224)
plt.hist(streched_img.ravel(),256);

```

```python

```


## Decision Making

The focus of pymoo is on optimization methods itself. However, some basic decision making methods are available:

### Compromise Programming

We can use any scalarization method and use it for post-processing. Let us assume our algorithm has converged to the Pareto-front:

```python
from pymoo.factory import get_problem

F = get_problem("zdt1").pareto_front()
```

Then, we initialize weights and our decomposition function:

```python
from pymoo.factory import get_problem, get_visualization, get_decomposition
import numpy as np

weights = np.array([0.5, 0.5])
decomp = get_decomposition("asf")
```

We apply the decomposition and retrieve the best value (here minimum):

```python
I = get_decomposition("asf").do(F, weights).argmin()
print("Best regarding decomposition: Point %s - %s" % (I, F[I]))
```

    Best regarding decomposition: Point 38 - [0.38383838 0.38045308]

Visualize it:

```python
from pymoo.factory import get_visualization

plot = get_visualization("scatter")
plot.add(F, color="blue", alpha=0.2, s=10)
plot.add(F[I], color="red", s=30)
plot.do()
plot.apply(lambda ax: ax.arrow(0, 0, 0.5, 0.5, color='black',
                               head_width=0.01, head_length=0.01, alpha=0.4))
plot.show()
```

### Pseudo-Weights

A simple way to chose a solution out of a solution set in the context of multi-objective optimization is the pseudo-weight vector approach proposed in . Respectively, the pseudo weight $w_i$ for the i-ith objective function can be calculated by:

\begin{equation}
w_i = \frac{(f_i^{max} - f_i {(x)}) \, /\,  (f_i^{max} - f_i^{min})}{\sum_{m=1}^M (f_m^{max} - f_m (x)) \, /\,  (f_m^{max} - f_m^{min})}
\end{equation}

This equation calculates the normalized distance to the worst solution regarding each objective $i$. Please note that for non-convex Pareto fronts the pseudo weight does not correspond to the result of an optimization using the weighted sum. However, for convex Pareto-fronts the pseudo weights are an indicator of the location in the objective space.

```python
from pymoo.factory import get_decision_making, get_reference_directions

ref_dirs = get_reference_directions("das-dennis", 4, n_partitions=12)
F = get_problem("dtlz1").pareto_front(ref_dirs)

weights = np.array([0.25, 0.25, 0.25, 0.25])
a, pseudo_weights = get_decision_making("pseudo-weights", weights).do(F, return_pseudo_weights=True)

weights = np.array([0.4, 0.20, 0.15, 0.25])
b, pseudo_weights = get_decision_making("pseudo-weights", weights).do(F, return_pseudo_weights=True)

plot = get_visualization("petal", bounds=(0, 0.5), reverse=True)
plot.add(F[[a, b]])
plot.show()
```

### High Trade-off Points

Furthermore, high trade-off points are usually of interest. We have implemented the trade-off metric proposed in . An example for 2 and 3 dimensional solution is given below:

```python
import os

import numpy as np

from pymoo.configuration import get_pymoo
from pymoo.factory import get_decision_making
from pymoo.visualization.scatter import Scatter

pf = np.loadtxt(os.path.join(get_pymoo(), "pymoo", "usage", "decision_making", "knee-2d.out"))
dm = get_decision_making("high-tradeoff")

I = dm.do(pf)

plot = Scatter()
plot.add(pf, alpha=0.2)
plot.add(pf[I], color="red", s=100)
plot.show()
```

```python
pf = np.loadtxt(os.path.join(get_pymoo(), "pymoo", "usage", "decision_making", "knee-3d.out"))
dm = get_decision_making("high-tradeoff")

I = dm.do(pf)

plot = Scatter(angle=(10, 140))
plot.add(pf, alpha=0.2)
plot.add(pf[I], color="red", s=100)
plot.show()
```

```python
import os

import numpy as np

from pymoo.configuration import get_pymoo
from pymoo.decision_making.high_tradeoff_inverted import HighTradeoffPointsInverted
from pymoo.visualization.scatter import Scatter

pf = np.loadtxt(os.path.join(get_pymoo(), "pymoo", "usage", "decision_making", "knee-2d.out"))
dm = HighTradeoffPointsInverted()

I = dm.do(pf)

plot = Scatter()
plot.add(pf, alpha=0.2)
plot.add(pf[I], color="red", s=100)
plot.show()
```


```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
```

# Unsupervised learning: Hierarchical and density-based clustering algorithms

In a previous notebook, "08 Unsupervised Learning - Clustering.ipynb", we introduced one of the essential and widely used clustering algorithms, K-means. One of the advantages of K-means is that it is extremely easy to implement, and it is also computationally very efficient compared to other clustering algorithms. However, we've seen that one of the weaknesses of K-Means is that it only works well if the data can be grouped into a globular or spherical shape. Also, we have to assign the number of clusters, *k*, *a priori* -- this can be a problem if we have no prior knowledge about how many clusters we expect to find.

In this notebook, we will take a look at 2 alternative approaches to clustering, hierarchical clustering and density-based clustering.

# Hierarchical Clustering

One nice feature of hierachical clustering is that we can visualize the results as a dendrogram, a hierachical tree. Using the visualization, we can then decide how "deep" we want to cluster the dataset by setting a "depth" threshold. Or in other words, we don't need to make a decision about the number of clusters upfront.

**Agglomerative and divisive hierarchical clustering**

Furthermore, we can distinguish between 2 main approaches to hierarchical clustering: Divisive clustering and agglomerative clustering. In agglomerative clustering, we start with a single sample from our dataset and iteratively merge it with other samples to form clusters -- we can see it as a bottom-up approach for building the clustering dendrogram.
In divisive clustering, however, we start with the whole dataset as one cluster, and we iteratively split it into smaller subclusters -- a top-down approach.

In this notebook, we will use **agglomerative** clustering.

**Single and complete linkage**

Now, the next question is how we measure the similarity between samples. One approach is the familiar Euclidean distance metric that we already used via the K-Means algorithm. As a refresher, the distance between 2 m-dimensional vectors $\mathbf{p}$ and $\mathbf{q}$ can be computed as:

\begin{align} \mathrm{d}(\mathbf{q},\mathbf{p}) & = \sqrt{(q_1-p_1)^2 + (q_2-p_2)^2 + \cdots + (q_m-p_m)^2} \\[8pt]
& = \sqrt{\sum_{j=1}^m (q_j-p_j)^2}.\end{align}

However, that's the distance between 2 samples. Now, how do we compute the similarity between subclusters of samples in order to decide which clusters to merge when constructing the dendrogram? I.e., our goal is to iteratively merge the most similar pairs of clusters until only one big cluster remains. There are many different approaches to this, for example single and complete linkage.

In single linkage, we take the pair of the most similar samples (based on the Euclidean distance, for example) in each cluster, and merge the two clusters which have the most similar 2 members into one new, bigger cluster.

In complete linkage, we compare the pairs of the two most dissimilar members of each cluster with each other, and we merge the 2 clusters where the distance between its 2 most dissimilar members is smallest.

To see the agglomerative, hierarchical clustering approach in action, let us load the familiar Iris dataset -- pretending we don't know the true class labels and want to find out how many different follow species it consists of:

```python
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data[:, [2, 3]]
y = iris.target
n_samples, n_features = X.shape

plt.scatter(X[:, 0], X[:, 1], c=y);
```

First, we start with some exploratory clustering, visualizing the clustering dendrogram using SciPy's `linkage` and `dendrogram` functions:

```python
from scipy.cluster.hierarchy import linkage
from scipy.cluster.hierarchy import dendrogram

clusters = linkage(X,
                   metric='euclidean',
                   method='complete')

dendr = dendrogram(clusters)

plt.ylabel('Euclidean Distance');
```

Next, let's use the `AgglomerativeClustering` estimator from scikit-learn and divide the dataset into 3 clusters. Can you guess which 3 clusters from the dendrogram it will reproduce?

```python
from sklearn.cluster import AgglomerativeClustering

ac = AgglomerativeClustering(n_clusters=3,
                             affinity='euclidean',
                             linkage='complete')

prediction = ac.fit_predict(X)
print('Cluster labels: %s\n' % prediction)
```

    Cluster labels: [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1
     1 1 1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0]

```python
plt.scatter(X[:, 0], X[:, 1], c=prediction);
```

# Density-based Clustering - DBSCAN

Another useful approach to clustering is *Density-based Spatial Clustering of Applications with Noise* (DBSCAN). In essence, we can think of DBSCAN as an algorithm that divides the dataset into subgroup based on dense regions of point.

In DBSCAN, we distinguish between 3 different "points":

- Core points: A core point is a point that has at least a minimum number of other points (MinPts) in its radius epsilon.
- Border points: A border point is a point that is not a core point, since it doesn't have enough MinPts in its neighborhood, but lies within the radius epsilon of a core point.
- Noise points: All other points that are neither core points nor border points.

A nice feature about DBSCAN is that we don't have to specify a number of clusters upfront. However, it requires the setting of additional hyperparameters such as the value for MinPts and the radius epsilon.

```python
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=400,
                  noise=0.1,
                  random_state=1)
plt.scatter(X[:,0], X[:,1])
plt.show()
```

```python
from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.2,
            min_samples=10,
            metric='euclidean')
prediction = db.fit_predict(X)

print("Predicted labels:\n", prediction)

plt.scatter(X[:, 0], X[:, 1], c=prediction);
```

# Exercise

    EXERCISE:

      Using the following toy dataset, two concentric circles, experiment with the three different clustering algorithms that we used so far: `KMeans`, `AgglomerativeClustering`, and `DBSCAN`.

Which clustering algorithms reproduces or discovers the hidden structure (pretending we don't know `y`) best?

Can you explain why this particular algorithm is a good choice while the other 2 "fail"?

```python
from sklearn.datasets import make_circles

X, y = make_circles(n_samples=1500,
                    factor=.4,
                    noise=.05)

plt.scatter(X[:, 0], X[:, 1], c=y);
```

```python
# %load solutions/20_clustering_comparison.py
```


## Expectation Maximization

Expectation Maximization (EM) is a powerful technique for creating maximum likelihood estimators when the variables are difficult to separate. Here, we set up a Gaussian mixture experiment with two Gaussians and derive the corresponding estimators of their means using EM.

### Experiment: Measuring from Unseen Groups

Let's investigate the following experiment:

Suppose we have a population with two distinct groups of individuals with different heights. If we randomly pick an individual from the population, assume we don't know which group the individual is from. So, we measure that individual's height and choose another individual. The goal is to estimate the mean heights of the two distinct groups when we have an unlabeled distribution of heights sampled from both groups.

Group **a** is normally distributed as

$$ \mathcal{N}_a(x) =\mathcal{N}(x; \mu_a,\sigma) $$

and likewise for group **b**

$$ \mathcal{N}_b(x) =\mathcal{N}(x; \mu_b,\sigma) $$

Note that we fix the standard deviation $\sigma$ to be the same for both groups, but the means ($\mu_a,\mu_b$) are different. The problem is to estimate the means given that you can't directly know which group you are picking from.

Then we can write the joint density for this experiment as the following:

$$ f_{\mu_a,\mu_b}(x,z)=  \frac{1}{2} \mathcal{N}_a(x) ^z \mathcal{N}_b(x) ^{1-z} $$

where $z=1$ if we pick from group **a** and $z=0$ for group **b**. Note that the $1/2$ comes from the 50/50 chance of picking either group.  Unfortunately, since we do not measure the $z$ variable, we have to integrate it out of our density function to account for this handicap. Thus,

$$ f_{\mu_a,\mu_b}(x)=  \frac{1}{2}  \mathcal{N}_a(x)+\frac{1}{2}  \mathcal{N}_b(x)$$

Now, since $n$ trials are independent, we can write out the likelihood:

$$ \mathcal{L}(\mu_a,\mu_b|\mathbf{x})= \prod_{i=1}^n f_{\mu_a,\mu_b}(x_i)$$

This is basically notation. We have just substituted everything into $ f_{\mu_a,\mu_b}(x)$ under the independent-trials assumption. Recal that the independent trials assumptions means that the joint probability is just the product of the individual probabilities. The idea of *maximum likelihood* is to maximize this as the function of $\mu_a$ and $\mu_b$ after plugging in all of the $x_i$ data.  The problem is we don't know which group we are measuring at each trial so this is trickier than just estimating the parameters for each group separately.

### Simulating the Experiment

We need the following code to setup the experiment of randomly a group and then picking an individual from that group.

```
from __future__ import division
from numpy import array, linspace, random
from scipy.stats import bernoulli, norm
from matplotlib import cm
from matplotlib.pylab import figure, subplots
#random.seed(101) # set random seed for reproducibility
mua_true=4 # we are trying to estimate this from the data
mub_true=7 # we are trying to estimate this from the data
fa=norm(mua_true,1) # distribution for group A
fb=norm(mub_true,1) # distribution for group B
fz=bernoulli(0.25) # each group equally likely

def sample(n=10):
    'simulate picking from each group n times'
    tmp=fz.rvs(n) # choose n of the coins, A or B
    return tmp*(fb.rvs(n))+(1-tmp)*fa.rvs(n) # flip it n times

xs = sample(1000) # generate some samples
```

Here's a quick look at the density functions of each group and a histogram of the samples

```
f,ax = subplots()
x = linspace(mua_true-2,mub_true+2,100)
ax.plot(x,fa.pdf(x),label='group A')
ax.plot(x,fb.pdf(x),label='group B')
ax.hist(xs,bins=50,normed=1,label='Samples');
ax.legend(loc=0);
```

Just from looking at this plot, we suspect that we will have to reconcile the samples in the overlap region since these could have come from either group. This is where the *Expectation Maximization* algorithm enters.

## Expectation maximization

The key idea of expectation maximization is that we can somehow pretend we know the unobservable $z$ value and the proceed with the usual maximum likelihood estimation process.

The idea behind expectation-maximization is that we want to use a maximum likelihood estimate (this is the *maximization* part of the algorithm) after computing the expectation over the missing variable (in this case, $z$).

The following code uses `sympy` to setup the functions symbolically and convert them to `numpy` functions that we can quickly evaluate. Because it's easier and more stable to evaluate, we will work with the `log` of the likelihood function. It is useful to keep track of the *incomplete log-likelihood* ($\log\mathcal{L}$) since it can be proved that it is monotone increasing and good way to identify coding errors. Recall that this was the likelihood in the case where we integrated out the $z$ variable to reconcile as its absence.

```
import sympy
from sympy.abc import x, z
from sympy import stats

mu_a,mu_b = sympy.symbols('mu_a,mu_b')
na=stats.Normal( 'x', mu_a,1)
nb=stats.Normal( 'x', mu_b,1)

L=(stats.density(na)(x)+stats.density(nb)(x))/2 # incomplete likelihood function
```

Next, we need to compute the expectation step. To avoid notational overload, we will just use $\Theta$ to denote the $\mu_b$ and $\mu_a$ parameters and the data $x_i$. This means that the density function of $z$ and $\Theta$ can be written as the following:

$$ \mathbb{P}(z,\Theta) = \frac{1}{2} \mathcal{N}_a(\Theta) ^ z \mathcal{N}_b(\Theta) ^ {(1-z)} $$

For the expectation part we have to compute $\mathbb{E}(z|\Theta)$ but since $z\in \lbrace 0,1 \rbrace$, this simplifies easily

$$ \mathbb{E}(z|\Theta) = 1 \cdot \mathbb{P}(z=1|\Theta) + 0 \cdot \mathbb{P}(z=0|\Theta) =  \mathbb{P}(z=1|\Theta)  $$

Now, the only thing left is to find $  \mathbb{P}(z=1|\Theta) $ which we can do using Bayes rule:

$$  \mathbb{P}(z=1|\Theta)  = \frac{ \mathbb{P}(\Theta|z=1)\mathbb{P}(z=1)}{\mathbb{P}(\Theta)} $$

The term in the denominator comes from summing (integrating) out the $z$ items in the full joint density $ \mathbb{P}(z,\Theta) $

$$ \mathbb{P}(\Theta) = (\mathcal{N}_a(\Theta) + \mathcal{N}_b(\Theta))\frac{1}{2} $$

and since $\mathbb{P}(z=1)=1/2$, we finally obtain

$$  \mathbb{E}(z|\Theta) =\mathbb{P}(z=1|\Theta)  = \frac{\mathcal{N}_a(\Theta)}{\mathcal{N}_a(\Theta) + \mathcal{N}_b(\Theta)} $$

and which is coded below.

```
def ez(x,mu_a,mu_b): # expected value of hidden variable
  return norm(mu_a).pdf(x) / ( norm(mu_a).pdf(x) + norm(mu_b).pdf(x) )
```

Now, given we we have this estimate for $z_i$,  $\hat{z}_i=\mathbb{E(z|\Theta_i)}$, we can go back and compute the log likelihood estimate of

$$ J= \log\prod_{i=1}^n \mathbb{P}(\hat{z}_i,\Theta_i) = \sum_{i=1}^n \hat{z}_i\log \mathcal{N}_a(\Theta_i) +(1-\hat{z}_i)\log \mathcal{N}_b(\Theta_i) +\log(1/2)  $$

by maximizing it using basic calculus. The trick is to remember that $\hat{z}_i$ is *fixed*, so we only have to maximize the $\log$ parts. This leads to

$$ \hat{\mu_a} = \frac{\sum_{i=1}^n \hat{z}_i x_i}{\sum_{i=1}^n  \hat{z}_i } $$

and for $\mu_b$

$$ \hat{\mu_b} = \frac{\sum_{i=1}^n (1-\hat{z}_i) x_i}{\sum_{i=1}^n  1-\hat{z}_i } $$

Now, we finally have the *maximization* step ( above ) and the *expectation* step ($\hat{z}_i$) from earlier. We're ready to simulate the algorithm and plot its performance!

```
Lf=sympy.lambdify((x,mu_a,mu_b), sympy.log(abs(L)),'numpy') # convert to numpy function from sympy

def run():
    out, lout = [], []
    mu_a_n=random.random() * 10 # initial guess
    mu_b_n=random.random() * 10 # initial guess
    for i in range(20): # iterations of expectation and maximization
        tau=ez(xs,mu_a_n,mu_b_n)                 # expected value of z-variable
        lout.append( sum(Lf(xs,mu_a_n,mu_b_n)) ) # save incomplete likelihood value (should be monotone)
        out.append((mu_a_n, mu_b_n))             # save of (pa,pb) steps
        mu_a_n=( sum(tau*xs) / sum(tau) )        # new maximum likelihood estimate of pa
        mu_b_n=( sum((1-tau) * xs) / sum(1-tau) )
    return out, lout

out, lout = run()

fig=figure()
fig.set_figwidth(12)
ax=fig.add_subplot(121)
ax.plot(array(out),'o-')
ax.legend(('mu_a','mu_b'),loc=0)
ax.hlines([mua_true,mub_true],0,len(out),['r','g'])
ax.set_xlabel('iteration',fontsize=18)
ax.set_ylabel('$\mu_a,\mu_b$ values',fontsize=24)
ax=fig.add_subplot(122)
ax.plot(array(lout),'o-')
ax.set_xlabel('iteration',fontsize=18)
ax.set_title('Incomplete likelihood',fontsize=16)
```

The figure on the left shows the estimates for both $\mu_a$ and $\mu_b$ for each iteration and the figure on the right shows the corresponding incomplete likelihood function. The horizontal lines on the left-figure show the true values we are trying to estimate. Notice the EM algorithm converges very quickly, but because each group is equally likely to be chosen, the algorithm cannot distinguish one from the other. The code below constructs a error surface to see this effect. The incomplete likelihood function is monotone which tells us that we have not made a coding error. We're omitting the proof of this monotonicity.

```
out, lout = run()
mua_step=linspace(0,10,30)
mub_step=linspace(0,10,20)
z=Lf(xs,mua_step[:,None],mub_step[:,None,None]).sum(axis=2) # numpy broadcasting
fig=figure(figsize=(8,5))
ax=fig.add_subplot(111)
p=ax.contourf(mua_step,mub_step,z,30,cmap=cm.gray)
xa,xb=zip(*out) # unpack the container from the previous block
ax.plot(xa,xb,'ro')                                    # points per iteration in red
ax.plot(mua_true,mub_true,'bs')                        # true values in blue
ax.plot(xa[0],xb[0],'gx',ms=15.,mew=2.)                # starting point in green
ax.text(xa[0],xb[0],'   start',color='g',fontsize=11.)
ax.set_xlabel('$\mu_a$',fontsize=24)
ax.set_ylabel('$\mu_b$',fontsize=24)
ax.set_title('Incomplete Likelihood',fontsize=18)
fig.colorbar(p);
```

The figure shows the incomplete likelihood function that the algorithm is exploring. Note that the algorithm can get to the maximizer but since the surface has symmetric maxima, it has no way to pick between them and ultimately just picks the one that is closest to the starting point. This is because each group is equally likely to be chosen. I urge you to download this notebook and try different initial points and see where the maximizer winds up.

## Summary

Expectation maximization is a powerful algorithm that is especially useful when it is difficult to de-couple the variables involved in a standard maximum likelihood estimation. Note that convergence to the "correct" maxima is not guaranteed, as we observed here. This is even more pronounced when there are more parameters to estimate. There is a nice [applet](http://www.cs.cmu.edu/~alad/em/) you can use to investigate this effect and a much more detailed mathematical derivation [here](http://crow.ee.washington.edu/people/bulyko/papers/em.pdf).

As usual, the IPython notebook corresponding to this post can be found [here](https://github.com/unpingco/Python-for-Signal-Processing/blob/master/Expectation_Maximization.ipynb). I urge you to try these calculations on your own. Try changing the sample size and making the choice between the two groups no longer equal to 1/2 (equally likely).

Note you will need at least `sympy` version 0.7.2 to run this notebook.

Comments appreciated!

```

```


```python
import numpy as np
import math
import sympy
import sympy as sp
from sympy import latex, Eq, IndexedBase, symbols, Idx, Indexed, Sum, S, diff
from sympy.functions.special.tensor_functions import KroneckerDelta
import matplotlib.pyplot as plt
from sympy.vector import Vector, CoordSys3D, AxisOrienter, BodyOrienter, Del, curl, divergence, gradient, is_conservative, is_solenoidal, scalar_potential, Point, scalar_potential_difference
from sympy.physics.vector import ReferenceFrame
from sympy.vector import express
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
from IPython.display import display, display_latex
```

```python
a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z = sympy.symbols('a:z')
alpha, beta, gamma, theta, phi, rho, omega = sympy.symbols('alpha beta gamma theta phi rho omega')
Alpha, Beta, Gamma, Theta, Phi, Rho, Omega = sympy.symbols('Alpha Beta Gamma Theta Phi Rho Omega')
```

## Setup

```python
sympy.init_printing()
np.set_printoptions(precision=3)
```

### Symbols

Cross product: ^
Dot product: &
Outer product: |

```python
a, b, c, d, e, f = symbols('a:f')
u1, u2, u3, v1, v2, v3 = symbols('u_1 u_2 u_3 v_1 v_2 v_3')
E = ReferenceFrame('E')
u = u1 * E.x + u2 * E.y + u3 * E.z
v = v1 * E.x + v2 * E.y + v3 * E.z
u
```

#### Dot Product

```python
u & v
```

#### Cross Product

```python
u ^ v
```

```python
N = CoordSys3D('N')
```

```python
v = 2 * N.i + N.j
v
```

```python
Vector.zero
```

```python
N.origin
```

```python
P = N.origin.locate_new('P', a*N.i + b*N.j + c*N.k)
Q = P.locate_new('Q', -b*N.j)
P
```

```python
P.position_wrt(Q)
```

```python
Q.express_coordinates(N)
```

#### Dyadics

```python
dyad = N.i.outer(N.k)
dyad
```

```python
A = CoordSys3D('A')
B = A.orient_new_axis('B', theta, A.k)
B
```

$\displaystyle CoordSys3D\left(B, \left( \left[\begin{matrix}\cos{\left(\theta \right)} & \sin{\left(\theta \right)} & 0\\- \sin{\left(\theta \right)} & \cos{\left(\theta \right)} & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

```python
axis_orienter = AxisOrienter(theta, A.k)
B = A.orient_new('B', axis_orienter)
B
```

$\displaystyle CoordSys3D\left(B, \left( \left[\begin{matrix}\cos{\left(\theta \right)} & \sin{\left(\theta \right)} & 0\\- \sin{\left(\theta \right)} & \cos{\left(\theta \right)} & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

#### Chain Rotations

```python
A = CoordSys3D('A')
body_orienter = BodyOrienter(a, b, c, 'XYZ')
B = A.orient_new('B', (axis_orienter, body_orienter)).subs(theta, 0)
B
```

$\displaystyle CoordSys3D\left(B, \left( \left[\begin{matrix}\cos{\left(b \right)} \cos{\left(c \right)} & \sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)} & \sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\\- \sin{\left(c \right)} \cos{\left(b \right)} & - \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)} & \sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\\\sin{\left(b \right)} & - \sin{\left(a \right)} \cos{\left(b \right)} & \cos{\left(a \right)} \cos{\left(b \right)}\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

#### Use Multiple Orienters

```python
C = A.orient_new('C', (axis_orienter, body_orienter))
C
```

$\displaystyle CoordSys3D\left(C, \left( \left[\begin{matrix}- \sin{\left(c \right)} \sin{\left(\theta \right)} \cos{\left(b \right)} + \cos{\left(b \right)} \cos{\left(c \right)} \cos{\left(\theta \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)} & \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)}\\- \sin{\left(c \right)} \cos{\left(b \right)} \cos{\left(\theta \right)} - \sin{\left(\theta \right)} \cos{\left(b \right)} \cos{\left(c \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} - \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)} & - \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)}\\\sin{\left(b \right)} & - \sin{\left(a \right)} \cos{\left(b \right)} & \cos{\left(a \right)} \cos{\left(b \right)}\end{matrix}\right], \  \mathbf{\hat{0}}\right), CoordSys3D\left(A, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)\right)$

#### Get Rotation Matrix W.R.T

```python
B.rotation_matrix(A)
```

$\displaystyle \left[\begin{matrix}- \sin{\left(c \right)} \sin{\left(\theta \right)} \cos{\left(b \right)} + \cos{\left(b \right)} \cos{\left(c \right)} \cos{\left(\theta \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)} & \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)}\\- \sin{\left(c \right)} \cos{\left(b \right)} \cos{\left(\theta \right)} - \sin{\left(\theta \right)} \cos{\left(b \right)} \cos{\left(c \right)} & \left(- \sin{\left(a \right)} \sin{\left(b \right)} \sin{\left(c \right)} + \cos{\left(a \right)} \cos{\left(c \right)}\right) \cos{\left(\theta \right)} - \left(\sin{\left(a \right)} \sin{\left(b \right)} \cos{\left(c \right)} + \sin{\left(c \right)} \cos{\left(a \right)}\right) \sin{\left(\theta \right)} & - \left(\sin{\left(a \right)} \sin{\left(c \right)} - \sin{\left(b \right)} \cos{\left(a \right)} \cos{\left(c \right)}\right) \sin{\left(\theta \right)} + \left(\sin{\left(a \right)} \cos{\left(c \right)} + \sin{\left(b \right)} \sin{\left(c \right)} \cos{\left(a \right)}\right) \cos{\left(\theta \right)}\\\sin{\left(b \right)} & - \sin{\left(a \right)} \cos{\left(b \right)} & \cos{\left(a \right)} \cos{\left(b \right)}\end{matrix}\right]$

#### Rotate and Translate

```python
C = A.orient_new_axis('C', a, A.k, location=2*A.j)
C.position_wrt(A)
```

#### Express : Vector instances can be expressed in user defined systems using express.

```python
M = N.orient_new_axis('M', a, N.k)
v1 = N.i + N.j + N.k
express(v1, M)
```

```python
v = N.i + N.j + N.k
v.to_matrix(M)
```

$\displaystyle \left[\begin{matrix}\sin{\left(a \right)} + \cos{\left(a \right)}\\- \sin{\left(a \right)} + \cos{\left(a \right)}\\1\end{matrix}\right]$

### Scalar and Vector Fields and Base Scalar

potential field = $2x^2y$

```python
R = CoordSys3D('R')
electric_potential = 2*R.x**2 * R.y
electric_potential
```

```python
diff(electric_potential, R.x)
```

```python
v = R.x**2*R.i + 2*R.x*R.z*R.k
v
```

#### Gradient

```python
C = CoordSys3D('C')
delop = Del()
gradient_field = delop(C.x*C.y*C.z)
gradient_field
```

```python
gradient_field.doit()
```

```python
gradient(C.x*C.y*C.z)
```

```python
delop.gradient(C.x*C.y*C.z).doit()
```

#### Curl

##### Method 1

```python
delop = Del()
delop.cross(C.x*C.y*C.z*C.i)
```

```python
delop.cross(C.x*C.y*C.z*C.i).doit()
```

##### Method 2

```python
curl(C.x*C.y*C.z*C.i)
```

#### Divergence

```python
divergence(C.x*C.y*C.z*(C.i + C.j + C.k))
```

```python
delop.dot(C.x*C.y*C.z*(C.i + C.j + C.k)).doit()
```

#### Directional Derivative

```python
from sympy.vector import directional_derivative
directional_derivative(C.x*C.y*C.z, 3*C.i + 4*C.j + C.k)
```

#### Conservative Field
* Curl = 0
* Irrotational
* Conservation of Energy
* Gradient of some scalar field

```python
R = CoordSys3D('R')
field = R.y*R.z*R.i + R.x*R.z*R.j + R.x*R.y*R.k
is_conservative(field)
```

    True

```python
curl(field)
```

#### Solenoidal Field
* Div = 0

```python
R = CoordSys3D('R')
field = R.y*R.z*R.i + R.x*R.z*R.j + R.x*R.y*R.k
is_solenoidal(field)
```

    True

```python
divergence(field)
```

#### Scalar Potential Functions

```python
R = CoordSys3D('R')
conservative_field = 4*R.x*R.y*R.z*R.i + 2*R.x**2*R.z*R.j + 2*R.x**2*R.y*R.k
scalar_potential(conservative_field, R)
```

```python
R = CoordSys3D('R')
P = R.origin.locate_new('P', 1*R.i + 2*R.j + 3*R.k)
vectfield = 4*R.x*R.y*R.i + 2*R.x**2*R.j
scalar_potential_difference(vectfield, R, R.origin, P)
```

```python

```


```python
import matplotlib.pyplot as plt
import matplotlib as mpl
import numpy as np
import sympy as sy
sy.init_printing()
```

```python
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

#  Eigenvalue and Eigenvector

An  eigenvector of an $n \times n$ matrix $A$ is a nonzero vector $x$ such that $Ax = \lambda x$ for some scalar $\lambda$. A scalar $\lambda$ is called an  eigenvalue of $A$ if there is a nontrivial solution $x$ of $Ax = \lambda x$, such an $x$ is called an eigenvector corresponding to $\lambda$.

Rewrite the equation,

$$
(A-\lambda I)x = 0
$$

Since the eigenvector should be a nonzero vector, which means:

1. The column or rows of $(A-\lambda I)$ are linearly dependent
2. $(A-\lambda I)$ is not full rank, $Rank(A)<n$.
3. $(A-\lambda I)$ is not invertible.
4. $\text{det}(A-\lambda I)=0$, which is called characteristic equation.

Consider a matrix $A$

$$
A=\left[\matrix{1 & 0 & 0\cr 1 & 0 & 1\cr 2 & -2 & 3}\right]
$$

Set up the characteristic equation,

$$
\text{det}\left(
\left[\matrix{1 & 0 & 0\cr 1 & 0 & 1\cr 2 & -2 & 3}\right]-
\lambda
\left[\matrix{1 & 0 & 0\cr 0 & 1 & 0\cr 0 & 0 & 1}\right]
\right)=0
$$

Use SymPy ```charpoly``` and ```factor```, we can have straightforward solutions for eigenvalues.

```python
lamda = sy.symbols('lamda') # 'lamda' withtout 'b' is reserved for SymPy, lambda is reserved for Python
```

```charpoly``` returns characteristic equation.

```python
A = sy.Matrix([[1, 0, 0], [1, 0, 1], [2, -2, 3]])
p = A.charpoly(lamda); p
```

Factor the polynomial such that we can see the solution.

```python
sy.factor(p)
```

From the factored characteristic polynomial, we get the eigenvalue, and $\lambda =1$ has algebraic multiplicity of $2$, because there are two $(\lambda-1)$. If not factored, we can use ```solve``` instead.

```python
sy.solve(p,lamda)
```

Or use ```eigenvals``` directly.

```python
sy.matrices.MatrixEigen.eigenvals(A)
```

To find the eigenvector corresponding to $\lambda$, we substitute the eigenvalues back into $(A-\lambda I)x=0$ and solve it. Construct augmented matrix with $\lambda =1$ and perform rref.

```python
(A - 1*sy.eye(3)).row_join(sy.zeros(3,1)).rref()
```

The null space is the solution set of the linear system.

$$
\left[
\begin{matrix}
x_1 \\ x_2 \\ x_3
\end{matrix}
\right]=
\left[
\begin{matrix}
x_2-x_3 \\ x_2 \\ x_3
\end{matrix}
\right]=
x_2\left[
\begin{matrix}
1 \\ 1 \\ 0
\end{matrix}
\right]
+x_3\left[
\begin{matrix}
-1 \\ 0 \\ 1
\end{matrix}
\right]
$$

This is called  eigenspace  for $\lambda = 1$, which is a subspace in $\mathbb{R}^3$. All eigenvectors are inside the eigenspace.

We can proceed with $\lambda = 2$ as well.

```python
(A - 2*sy.eye(3)).row_join(sy.zeros(3,1)).rref()
```

The null space is the solution set of the linear system.

$$
\left[
\begin{matrix}
x_1 \\ x_2 \\ x_3
\end{matrix}
\right]=
\left[
\begin{matrix}
0\\ \frac{1}{2}x_3\\ x_3
\end{matrix}
\right]=
x_3\left[
\begin{matrix}
0 \\ \frac{1}{2} \\ 1
\end{matrix}
\right]
$$

To avoid troubles of solving back and forth, SymPy has ```eigenvects``` to calcuate eigenvalues and eigenspaces (basis of eigenspace).

```python
eig = sy.matrices.MatrixEigen.eigenvects(A)
eig
```

To clarify what we just get, write

```python
print('Eigenvalue = {0}, Multiplicity = {1}, Eigenspace = {2}'.format(eig[0][0], eig[0][1], eig[0][2]))
```

    Eigenvalue = 1, Multiplicity = 2, Eigenspace = [Matrix([
    [1],
    [1],
    [0]]), Matrix([
    [-1],
    [ 0],
    [ 1]])]

```python
print('Eigenvalue = {0}, Multiplicity = {1}, Eigenspace = {2}'.format(eig[1][0], eig[1][1], eig[1][2]))
```

    Eigenvalue = 2, Multiplicity = 1, Eigenspace = [Matrix([
    [  0],
    [1/2],
    [  1]])]

##  NumPy Functions for Eigenvalues and Eigenspace

Convert SymPy matrix into NumPy float array.

```python
A = np.array(A).astype(float); A
```

    array([[ 1.,  0.,  0.],
           [ 1.,  0.,  1.],
           [ 2., -2.,  3.]])

```.eigvals()``` and ```.eig(A)``` are handy functions for eigenvalues and eigenvectors.

```python
np.linalg.eigvals(A)
```

    array([2., 1., 1.])

```python
np.linalg.eig(A) #return both eigenvalues and eigenvectors
```

    (array([2., 1., 1.]), array([[ 0.   ,  0.   ,  0.408],
            [ 0.447,  0.707, -0.408],
            [ 0.894,  0.707, -0.816]]))

##  An Example

Consider a matrix $A$

```python
A = sy.Matrix([[-4, -4, 20, -8, -1],
               [14, 12, 46, 18, 2],
               [6, 4, -18, 8, 1],
               [11, 7, -37, 17, 2],
               [18, 12, -60, 24, 5]])
A
```

Find eigenvalues.

```python
eig = sy.matrices.MatrixEigen.eigenvals(A)
eig
```

Or use NumPy functions, show the eigenvalues.

```python
A = np.array(A)
A = A.astype(float)
eigval, eigvec = np.linalg.eig(A)
eigval
```

    array([ 21.69, -16.69,   3.  ,   2.  ,   2.  ])

And corresponding eigenvectors.

```python
eigvec
```

    array([[-0.124, -0.224, -0.   ,  0.816, -0.056],
           [ 0.886, -0.543, -0.894, -0.408,  0.263],
           [ 0.124,  0.224,  0.   , -0.   , -0.   ],
           [ 0.216,  0.392,  0.447, -0.408, -0.207],
           [ 0.371,  0.672, -0.   ,  0.   ,  0.941]])

##  A Visualization Example

Let

$$
A=
\left[
\begin{matrix}
1 & 6\\
5 & 2
\end{matrix}
\right]
$$

find the eigenvalues and vectors, then visualize in $\mathbb{R}^2$

Use characteristic equation $|A - \lambda I|=0$

$$
\left|
\left[
\begin{matrix}
1 & 6\\
5 & 2
\end{matrix}
\right]
-
\left[
\begin{matrix}
\lambda & 0\\
0 & \lambda
\end{matrix}
\right]\right|=0
$$

```python
lamda = sy.symbols('lamda')
A = sy.Matrix([[1,6],[5,2]])
I = sy.eye(2)
```

```python
A - lamda*I
```

```python
p = A.charpoly(lamda);p
```

```python
sy.factor(p)
```

There are two eigenvalues: $7$ and $4$. Next we calculate eigenvectors.

```python
(A - 7*sy.eye(2)).row_join(sy.zeros(2,1)).rref()
```

The eigenspace for $\lambda = 7$ is

$$
\left[
\begin{matrix}
x_1\\
x_2
\end{matrix}
\right]=
x_2\left[
\begin{matrix}
1\\
1
\end{matrix}
\right]
$$

Any vector is eigenspace as long as $x \neq 0$ is an eigenvector. Let's find out eigenspace for $\lambda = 4$.

```python
(A + 4*sy.eye(2)).row_join(sy.zeros(2,1)).rref()
```

The eigenspace for $\lambda = -4$ is

$$
\left[
\begin{matrix}
x_1\\
x_2
\end{matrix}
\right]=
x_2\left[
\begin{matrix}
-\frac{6}{5}\\
1
\end{matrix}
\right]
$$

Let's plot both eigenvectors as $(1, 1)$ and $(-6/5, 1)$ and multiples with eigenvalues.

```python
fig, ax = plt.subplots(figsize = (12, 12))
arrows = np.array([ [[0,0,1,1]],
                  [[0,0,-6/5,1]],
                  [[0,0,7,7]],
                  [[0,0,24/5,-4]] ])
colors = ['darkorange','skyblue','r','b']
for i in range(arrows.shape[0]):
    X,Y,U,V = zip(*arrows[i,:,:])
    ax.arrow(X[0], Y[0], U[0],V[0], color = colors[i], width = .08,
             length_includes_head = True,
             head_width = .3, # default: 3*width
             head_length = .6,
             overhang = .4, zorder = -i)

################################### Eigenspace #################################
x = np.arange(-10, 10.6, .5)
y = x
ax.plot(x, y, lw = 2, color = 'red', alpha = .3, label = 'Eigenspace for $\lambda = 7$')

x = np.arange(-10, 10.6, .5)
y = -5/6*x
ax.plot(x, y, lw = 2, color = 'blue', alpha = .3,  label = 'Eigenspace for $\lambda = -4$')

######################## Annotation Arrows ################################

style="Simple, tail_width=0.5, head_width=5, head_length=10"
kw = dict(arrowstyle=style, color="k")

a = mpl.patches.FancyArrowPatch((1,1), (7,7),connectionstyle="arc3,rad=.4", **kw, alpha = .3)
plt.gca().add_patch(a)

a = mpl.patches.FancyArrowPatch((-6/5,1), (24/5,-4),connectionstyle="arc3,rad=.4", **kw, alpha = .3)
plt.gca().add_patch(a)

############################ Legend ###############################

leg = ax.legend(fontsize = 15, loc = 'lower right')
leg.get_frame().set_alpha(0.5)

###################### Axis, Spines, Ticks ##########################
ax.axis([-10, 10.1, -10.1, 10.1])
ax.spines['left'].set_position('center')
ax.spines['right'].set_color('none')
ax.spines['bottom'].set_position('center')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')

ax.minorticks_on()
ax.tick_params(axis = 'both', direction = 'inout', length=12, width=2, which='major')
ax.tick_params(axis = 'both', direction = 'inout', length=10, width=1, which='minor')
ax.grid()
plt.show()
```

#  Geometric Intuition

Eigenvector has a special property that preserves the pointing direction after linear transformation.To illustrate the idea, let's plot a 'circle' and arrows touching edges of circle.

Start from one arrow. If you want to draw a smoother circle, you can use parametric function rather two quadratic functions, because cicle can't be draw with one-to-one mapping.But this is not the main issue, we will live with that.

```python
x = np.linspace(-4, 4)
y_u = np.sqrt(16 - x**2)
y_d = -np.sqrt(16 - x**2)

fig, ax = plt.subplots(figsize = (8, 8))
ax.plot(x, y_u, color = 'b')
ax.plot(x, y_d, color = 'b')

ax.scatter(0, 0, s = 100, fc = 'k', ec = 'r')

ax.arrow(0, 0, x[5], y_u[5], head_width = .18,
         head_length= .27, length_includes_head = True,
         width = .03, ec = 'r', fc = 'None')
plt.show()
```

Now, the same 'circle', but more arrows.

```python
x = np.linspace(-4, 4, 50)
y_u = np.sqrt(16 - x**2)
y_d = -np.sqrt(16 - x**2)

fig, ax = plt.subplots(figsize = (8, 8))
ax.plot(x, y_u, color = 'b')
ax.plot(x, y_d, color = 'b')

ax.scatter(0, 0, s = 100, fc = 'k', ec = 'r')

for i in range(len(x)):
    ax.arrow(0, 0, x[i], y_u[i], head_width = .08,
             head_length= .27, length_includes_head = True,
             width = .01, ec = 'r', fc = 'None')
    ax.arrow(0, 0, x[i], y_d[i], head_width = .08,
             head_length= .27, length_includes_head = True,
             width = .008, ec = 'r', fc = 'None')
```

Now we will perform linear transformation on the circle. Technically, we can only transform the points - the arrow tip - that we specify on the circle.

We create a matrix

$$
A =
\left[\begin{matrix}
3 & -2\\
1 & 0
\end{matrix}\right]
$$

Align all the coordinates into two matrices for upper and lower half respectively.

$$
V_u =
\left[\begin{matrix}
x_1^u & x_2^u & \ldots & x_m^u\\
y_1^u & y_2^u & \ldots & y_m^u
\end{matrix}\right]\\
V_d =
\left[\begin{matrix}
x_1^d & x_2^d & \ldots & x_m^d\\
y_1^d & y_2^d & \ldots & y_m^d
\end{matrix}\right]
$$

The matrix multiplication $AV_u$ and $AV_d$ are linear transformation of the circle.

```python
A = np.array([[3, -2], [1, 0]])

Vu = np.hstack((x[:, np.newaxis], y_u[:, np.newaxis]))
AV_u = (A@Vu.T)

Vd = np.hstack((x[:, np.newaxis], y_d[:, np.newaxis]))
AV_d = (A@Vd.T)
```

The circle becomes an ellipse. However, if you watch closely, you will find there are some arrows still pointing the same direction after linear transformation.

```python
fig, ax = plt.subplots(figsize = (8, 8))

for i in range(len(x)):
    ax.arrow(0, 0, Av_1[0, i], Av_1[1, i], head_width = .18,
             head_length= .27, length_includes_head = True,
             width = .03, ec = 'darkorange', fc = 'None')
    ax.arrow(0, 0, Av_2[0, i], Av_2[1, i], head_width = .18,
             head_length= .27, length_includes_head = True,
             width = .03, ec = 'darkorange', fc = 'None')
ax.axis([-15, 15, -5, 5])
ax.grid()
plt.show()
```

We can plot the cirle and ellipse together, those vectors pointing the same direction before and after the linear transformation are eigenvector of $A$, eigenvalue is the length ratio between them.

```python
k = 50
x = np.linspace(-4, 4, k)
y_u = np.sqrt(16 - x**2)
y_d = -np.sqrt(16 - x**2)

fig, ax = plt.subplots(figsize = (8, 8))

ax.scatter(0, 0, s = 100, fc = 'k', ec = 'r')

for i in range(len(x)):
    ax.arrow(0, 0, x[i], y_u[i], head_width = .18,
             head_length= .27, length_includes_head = True,
             width = .03, ec = 'r', alpha = .5, fc = 'None')
    ax.arrow(0, 0, x[i], y_d[i], head_width = .18,
             head_length= .27, length_includes_head = True,
             width = .03, ec = 'r', alpha = .5, fc = 'None')

A = np.array([[3, -2], [1, 0]])

v = np.hstack((x[:, np.newaxis], y_u[:, np.newaxis]))
Av_1 = (A@v.T)

v = np.hstack((x[:, np.newaxis], y_d[:, np.newaxis]))
Av_2 = (A@v.T)

for i in range(len(x)):
    ax.arrow(0, 0, Av_1[0, i], Av_1[1, i], head_width = .18,
             head_length= .27, length_includes_head = True,
             width = .03, ec = 'darkorange', fc = 'None')
    ax.arrow(0, 0, Av_2[0, i], Av_2[1, i], head_width = .18,
             head_length= .27, length_includes_head = True,
             width = .03, ec = 'darkorange', fc = 'None')
n = 7
ax.axis([-n, n, -n, n])
plt.show()
```

Chapter 12 - Eigenvalues and Eigenvectors.ipynb


Grover's Algorithm

```
%load_ext sympyprinting
```

```
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

```
nqubits = 3

```

Grover's algorithm is a quantum algorithm which searches an unordered database (inverts a function).

Provides polynomial speedup over classical brute-force search ($O(\sqrt{N}) vs. O(N))$

Define a black box function that returns True if it is passed the state we are searching for.

```
def black_box(qubits):
    return True if qubits == IntQubit(1, qubits.nqubits) else False

```

Build a uniform superposition state to start the search.

```
psi = superposition_basis(nqubits); psi

```

```
v = OracleGate(nqubits, black_box)

```

Perform two iterations of Grover's algorithm.  Each iteration, the amplitude of the target state increases.

```
iter1 = qapply(grover_iteration(psi, v)); iter1

```

```
iter2 = qapply(grover_iteration(iter1, v)); iter2

```

A single shot measurement is performed to retrieve the target state.

```
measure_all_oneshot(iter2)

```


```python
import numpy as np
import sympy as sp
```

```python
def newton_1d_step(f_str, x_prev):
    x = sp.symbols('x') # We will assume that you use the symbol x
    formula = sp.sympify(f_str)
    df = sp.diff(formula, x)
    ddf = sp.diff(df, x)
    return x_prev - df.subs(x, x_prev) / ddf.subs(x, x_prev)
```

```python
# Test case
tol = 10**-7
expected = 0.239999999617
actual = newton_1d_step('4*sin(x**6)', 0.3)
assert abs(expected-actual) < tol
```