### Authors’ Note:

This is the 1st of 3 notebooks prepared on [2020 Winter School on Synchronization](https://complex-systems-turkey.github.io/).

If you have any questions or comments please contact us at [GitHub repository](https://github.com/complex-systems-turkey/winter-2020-synchronization).

* [Oğuz Kaan Yüksel](https://github.com/okyksl)
* [Enis Simsar](https://github.com/enisimsar)
* [Galip Ümit Yolcu](https://github.com/gumityolcu)
* [Suzan Üsküdarlı](https://github.com/uskudarli)

# Introduction to Dynamical Systems

Dynamical systems are rules that determine how a particular state moves to another state in time. To understand quantative behaviour of these systems over time, first we need to define numerical integration and approximation methods.

```python
import numpy as np
import matplotlib.pyplot as plt
%config InlineBackend.figure_format = 'retina'
```

```python
def integrate(estimator, f, x, t=0, dt=0.01, n=100):
    """Numerical approximation of a trajectory in a vector field

    Parameters
    ----------
    estimator : function
        A function that estimates the change of value in trajectory from a given point.
    f         : function
        A function that returns a vector field (or a gradient) on a given point.
    x         : array_like
        Initial value point of the trajectory.
    t         : float | tuple | array_like
        Initial time point | start and end time values | whole time points of the integration.
    dt        : float
        Time step will be used in the integration.
        Will only be used if `t` is not given as array_like.
    n         : int
        Number of steps will be used in the integration.
        Will only be used together with `dt` when `t` denotes the initial time value.

    Returns
    -------
    xs        : array_like
        a list of value points calculated with integration that start from given `x`.
    ts        : array
        a list of time points that corresponds to given value points.
    """

    # calculate time points that will be used in the integration
    if isinstance(t, float): # if given as (start) time point
        ts = np.arange(t, t + dt * n, dt) # start integrating from t using `dt` and `n`
    elif isinstance(t, tuple): # if given (start, end) time points
        ts = np.arange(t[0], t[1]+dt, dt) # utilize dt to find intermediate points
    else: # if given as an array of time points
        ts = t

    xs = [ x ]
    for i in range(1, len(ts)):
        dt = ts[i] - ts[i-1] # calculate time diff
        x = x + estimator(f, x, ts[i], dt) # calculate next point
        xs.append(x)
    return np.stack(xs), ts
```

### [Euler Method](https://en.wikipedia.org/wiki/Euler_method) (Tangent Line Method)

The tangent line approximation of $y$ at $t$ can be written as:
\begin{align}
y(t_1) \approx y(t_0) + f(y(t_0), t_0)(t_1-t_0)
\end{align}

where $f(y) = \frac{dy}{dt}$. When the distance $\|t_1-t_0\|$ is very small, this is a good approximation.

To define a discrete approximation method, we can fix a very small time interval $d$ and iteratively approximate the values of $y$ around some neighborhood of $t_0$.

\begin{align}
y_0 &= y(t_0) \\
y_i &= y_{i-1} + f(y_{i-1}) \cdot d
\end{align}

where $y_i \approx y(t_0 + i \cdot d)$.

```python
def euler(f, x, t=0, dt=0.01):
    """Estimates a change in trajectory using Euler's Method.

    Parameters
    ----------
    f         : function
        A function that returns a vector field (or a gradient) on a given point.
    x         : array_like
        Value point of the estimation.
    t         : float | tuple | array
        Time point of the estimation.
    dt        : float
        Time step will be used in estimation.

    Returns
    -------
    dx        : array_like
        Change estimated by Euler's Method.
    """

    return f(x, t) * dt
```

### [Heun's Method](https://en.wikipedia.org/wiki/Heun%27s_method) (Runge-Kutta $2^{nd}$ order Method)

Heun method is improved version of Euler's, a predictor-corrector method. The method uses dynamics to predict the slope at next step, then corrects Euler's method by averaging two consecutive tangent lines.

\begin{align}
y(t+h) \approx y(t) + \frac{f(y(t), t)+ f(\tilde{y}(t,h), t+h)}{2} \cdot h
\end{align}

where $\tilde{y}(t,h) = y(t) + f(y(t), t) \cdot h$. This method is a $2^{nd}$ order accurate method which means local error is on the order of $\mathcal{0}(h^3)$.

\begin{align}
y_0 &= y(t_0) \\
t_{i+1} &= t_0 + i \cdot d \\
y_{i+1} &= y_i + \frac{f(y_i, t_i)+ f(\tilde{y}_{i+1}, t_{i+1})}{2} \cdot d
\end{align}

where $\tilde{y}_{i+1}$ is calculated with Euler's Method.

```python
def heun(f, x, t=0, dt=0.01):
    """Estimates a change in trajectory using Heun's Method.

    Parameters
    ----------
    f         : function
        A function that returns a vector field (or a gradient) on a given point.
    x         : array_like
        Value point of the estimation.
    t         : float | tuple | array
        Time point of the estimation.
    dt        : float
        Time step will be used in estimation.

    Returns
    -------
    dx        : array_like
        Change estimated by Heun's Method.
    """

    d_euler = euler(f, x, t, dt) # calculate change with Euler's
    x_euler = x + d_euler # calculate point estimated with Euler's
    return (d_euler + f(x_euler, t + dt) * dt) * 0.5 # apply Heun's correction
```

### [Runge-Kutta $4^{th}$ order Method](https://en.wikipedia.org/wiki/Runge–Kutta_methods)
RK4 method, the most famous member of the Runge-Kutta family, is a $4^{th}$ order accurate method which means local error is on the order of $\mathcal{0}(h^5)$. Here is the numerical integration procedure defined recursively on time points $t_i$s and associated value points $y_{i}s$.

\begin{align}
h &= t_i-t_{i-1}\\
y_{i} &= y_{i-1} + \frac{1}{6}(k_1+2k_2+2k_3+k_4)\\
k_1 &= h f(t_{i-1},y_{t_{i-1}})\\
k_2 &= h f(t_{i-1}+\frac{h}{2}, y_{i-1}+\frac{k_1}{2})\\
k_3 &= h f(t_{i-1}+\frac{h}{2}, y_{i-1}+\frac{k_2}{2})\\
k_4 &= h f(t_{i-1}+\frac{h}{2}, y_{i-1}+k_3)\\
\end{align}

```python
def rk4(f, x, t, dt):
    """Estimates a change in trajectory using Runge-Kutta 4th order Method.

    Parameters
    ----------
    f         : function
        A function that returns a vector field (or a gradient) on a given point.
    x         : array_like
        Value point of the estimation.
    t         : float | tuple | array
        Time point of the estimation.
    dt        : float
        Time step will be used in estimation.

    Returns
    -------
    dx        : array_like
        Change estimated by RK4 Method.
    """

    k1 = f(x, t) * dt
    k2 = f(x + k1 * 0.5, t + dt * 0.5) * dt
    k3 = f(x + k2 * 0.5, t + dt * 0.5) * dt
    k4 = f(x + k3, t + dt * 0.5) * dt
    return (k1 + 2.0 * k2 + 2.0 * k3 + k4) / 6.0
```

### 1-d Differential Equation
We define a $1^{st}$ order differential equation with the general form $\dot{y}=f(y,t)$ where $\dot{y}$ is the derivative of the function $y(t)$ with respect to time $t$.<br>
For a simple example, let us choose the equation $\dot{y}=2y$
with the initial condition that $y(0)=1$.

```python
# define the derivative function
def f(x, t):
    return 2 * x

# define initial conditions and parameters of the integration
x0 = 1
t = (0,5)
dt = 0.1

# calculate trajectory of the initial point with numerical integration
x_euler, ts = integrate(euler, f=f, x=x0, t=t, dt=dt)
x_heun, _ = integrate(heun, f=f, x=x0, t=ts, dt=dt)
x_rk4, _ = integrate(rk4, f=f, x=x0, t=ts, dt=dt)

# calculate analytical solution of the problem
x_crt = np.exp(2 * ts)

# plot trajectories together
plt.figure(figsize=(12,6))
plt.plot(ts, x_crt, linewidth=2, linestyle='--')
plt.plot(ts, x_euler, linewidth=2, alpha=0.75)
plt.plot(ts, x_heun, linewidth=2, alpha=0.75)
plt.plot(ts, x_rk4, linewidth=2, alpha=0.75)
plt.ylabel('x(t)', fontsize=24)
plt.xlabel('t', fontsize=24)
plt.legend(['Analytical', 'Euler\'s', 'Heun\'s', 'Runge-Kutta'], fontsize=24);
```

### 2-d Differential Equation
Now, we are going to take a look at an application of $2^{nd}$ order differential equation, in particular mass-spring problem.
\begin{align}
m \frac{d^{2}y}{dt^2} &= -k y
\end{align}
Assume for simplicity $\frac{k}{m} = 1$ where $k$ is the spring constant and $m$ is the mass. Our problem turns into:
\begin{align}
\frac{d^{2}y}{dt^2} &= -y
\end{align}

Assume the initial condition is $y(0)=1$ and $\dot{y}(0)=0$

```python
# define the derivative function
def f(x, t):
    return np.asarray([ x[1], -x[0] ])

# define initial conditions and parameters of the integration
x0 = np.asarray([ 1, 0 ])
t = (0,100)
dt = 0.1

# calculate trajectory of the initial point with numerical integration
x_euler, ts = integrate(euler, f=f, x=x0, t=t, dt=dt)
x_heun, _ = integrate(heun, f=f, x=x0, t=t, dt=dt)
x_rk4, _ = integrate(rk4, f=f, x=x0, t=t, dt=dt)

# calculate analytical solution of the problem
x_crt = np.transpose(np.asarray([np.cos(ts), -np.sin(ts)]))

# plot trajectories together
fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(22,6))
axs[0].plot(ts, x_crt[:, 0], linewidth=2, linestyle='--')
axs[0].plot(ts, x_euler[:, 0], linewidth=2, alpha=0.8)
axs[0].plot(ts, x_heun[:, 0], linewidth=2, alpha=0.8)
axs[0].plot(ts, x_rk4[:, 0], linewidth=2, alpha=0.8)
axs[0].set_ylabel('x(t)', fontsize=24)
axs[0].set_xlabel('t', fontsize=24)
axs[0].set_title('Position', fontsize=24)

axs[1].plot(ts, x_crt[:, 1], linewidth=2, linestyle='--')
axs[1].plot(ts, x_euler[:, 1], linewidth=2, alpha=0.8)
axs[1].plot(ts, x_heun[:, 1], linewidth=2, alpha=0.8)
axs[1].plot(ts, x_rk4[:, 1], linewidth=2, alpha=0.8)
axs[1].set_ylabel('$\dot{x}$(t)', fontsize=24)
axs[1].set_xlabel('t', fontsize=24)
axs[1].set_title('Velocity', fontsize=24)

axs[2].plot(x_crt[:, 1], x_crt[:, 0], linewidth=2, linestyle='--')
axs[2].plot(x_euler[:, 1], x_euler[:, 0], linewidth=2, alpha=0.8)
axs[2].plot(x_heun[:, 1], x_heun[:, 0], linewidth=2, alpha=0.8)
axs[2].plot(x_rk4[:, 1], x_rk4[:, 0], linewidth=2, alpha=0.8)
axs[2].set_ylabel('$\dot{x}$(t)', fontsize=24)
axs[2].set_xlabel('x(t)', fontsize=24)
axs[2].set_title('Phase Space', fontsize=24)

fig.legend(['Analytical', 'Euler\'s', 'Heun\'s', 'Runge-Kutta'], fontsize=24);
```

# Synchronization of Linear Systems

### Synchronization of two linearly coupled linear systems

Analytical investigation predicts that these two systems synchronize at:

\begin{align}
\alpha_c = \frac{a}{2}
\end{align}

So, as long as

\begin{align}
\alpha_c \geq \frac{a}{2}
\end{align}

synchronization should occur. Let our systems be by defined by the equations:

\begin{aligned}
\dot{y_1} &= ay_1 + \alpha (y_2 - y_1)\\
\dot{y_2} &= ay_2 + \alpha (y_1 - y_2)
\end{aligned}

```python
"""
1. TODO: The task given here is this "Tune the alpha parameter to see the synchronization case".
An interactive plot that you can play with a and alpha would be great here.
"""

# define a function for two coupled linear system
def f(x, t, a=-0.4, alpha=0.5):
    return np.asarray([
        a * x[0] + alpha * (x[1] - x[0]),
        a * x[1] + alpha * (x[0] - x[1])
    ])

# define initial conditions and parameters of the integration
x0 = np.asarray([1000, 5])
t = (0, 12)
dt = 0.01

# calculate trajectory of the initial point with numerical integration
x_rk4, _ = integrate(rk4, f=f, x=x0, t=t, dt=dt)

# plot trajectories of both systems
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(18,6))
axs[0].plot(x_rk4[:, 0], linewidth=2, alpha=0.8)
axs[0].plot(x_rk4[:, 1], linewidth=2, alpha=0.8)
axs[0].set_xlabel('t', fontsize=24)
axs[0].set_title('Time Evolution', fontsize=24)
axs[0].legend(['$x_1(t)$', '$x_2(t)$'], fontsize=24)

axs[1].plot(x_rk4[:, 0], x_rk4[:, 1], linewidth=2, alpha=0.8)
axs[1].set_xlabel('$x_1(t)$', fontsize=24)
axs[1].set_ylabel('$x_2(t)$', fontsize=24)
axs[1].plot(x0[0], x0[1], marker='o', markersize=10, color='r')
axs[1].plot(x_rk4[-1, 0], x_rk4[-1, 1], marker='o', markersize=10, color='g')
axs[1].set_title('Phase Space', fontsize=24)
axs[1].legend(['Trajectory', 'Initial Point', 'Final Point'], fontsize=24);
```

The synchronization error can be defined by:
\begin{align}
E = \frac{1}{N(N-1)}\sum_{i,j\gt 1} |x_i - x_j|
\end{align}
But as $\|x_i - x_j\| = \|x_j - x_i\| $, this sum could be written as:

\begin{align}
E = \frac{2}{N(N-1)}\sum_{i = 1}^{N} \sum_{j=1}^{i} |x_i - x_j|
\end{align}

```python
# define a function for synchronization error
def error(x):
    n = x.shape[1] # assume x is shape (t x n)
    x_row = np.repeat(x[..., np.newaxis], n, axis=2) # construct t times (n x n) matrices with same values in rows
    x_col = np.transpose(x_row, (0, 2, 1)) # construct t times (n x n) matrices with same values in cols
    x_err = np.abs(x_col - x_row) # calculate error between all (i, j) pairs over matrices
    x_err = np.sum(x_err, axis=(1,2)) # sum errors in matrices and collapse to (t)
    x_err = x_err / (n * (n-1)) # only have n * (n-1) in denominator bcs matrices count every (i,j) twice
    return x_err

# calculate trajectory of the initial point with numerical integration and error
x_rk4, ts = integrate(rk4, f=f, x=x0, t=t, dt=dt)
err = error(x_rk4)

# plot synchronization error
plt.figure(figsize=(12,6))
plt.plot(ts, err, linewidth=2, alpha=0.8)
plt.ylabel('Error', fontsize=24)
plt.xlabel('t', fontsize=24);
```

### Synchronization of N linearly coupled linear systems

Analytical investigation predicts that these N systems synchronize at:

\begin{align}
\alpha_c = \frac{a}{N}
\end{align}

where $N$ is the number of variables. So, as long as

\begin{align}
\alpha_c \geq \frac{a}{N}
\end{align}

systems are expected to synchronize.

Things to note:

* As $\alpha$ is increased synchronization happens much fast. Is that expected?
* At $\alpha = \frac{a}{N}$ systems are neutrally stable?
* What happens as N tends to infinity?

\begin{align}
\dot{y_i} = ay_i + \alpha \sum_{j=1}^{N} (y_j - y_i)
\end{align}

```python
"""
1. TODO: The task given here is this "Tune the alpha parameter around its critical value and see the behaviour".
An interactive plot that you can play with a and alpha would be great here.
"""

# define a function for N linearly coupled linear systems
def f(x, t, a=0.4, alpha=0.1):
    n = x.shape[0]
    x_row = np.repeat(x[..., np.newaxis], n, axis=1) # duplicate over rows
    x_col = np.transpose(x_row) # duplicate over cols
    dx_coup = alpha * (x_col - x_row) # calculate coupling matrix
    dx_diag = a * np.diag(x) # calculate internal dynamics (or diagonal of laplacian)
    return np.sum(dx_diag + dx_coup, axis=1) # sum over columns to get derivatives for each system

# define initial conditions and parameters of the integration
x0 = np.asarray([30, 15, -5, 5])
t = (0, 4)
dt = 0.01

# calculate trajectory of the initial point with numerical integration and error
x_rk4, _ = integrate(rk4, f=f, x=x0, t=t, dt=dt)
err = error(x_rk4)

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(20,6))
axs[0].plot(x_rk4, linewidth=2, alpha=0.8)
axs[0].set_ylabel('$x_i(t)$', fontsize=24)
axs[0].set_xlabel('t', fontsize=24)
axs[0].legend([('$x_%d(t)$' % i) for i in range(1, x_rk4.shape[1]+1)], fontsize=16)

axs[1].plot(err)
axs[1].set_ylabel('Error', fontsize=24)
axs[1].set_xlabel('t', fontsize=24);
```

Day #1 - Numerical Integration and Linear Synchronization.ipynb


## Problem
You are faced repeatedly with a choice among n different options, or actions. After each choice you receive a
numerical reward chosen from a stationary probability distribution that depends on the action you selected. Your objective is to maximize the expected total reward over some time period, for example, over 1000 action selections,
or time steps.

## Exploring and exploiting
Let expected or mean of reward of each action be called its *value*. At any time step if we choose best possible action as highest of our observed values of all actions. We call such action *greedy* and this process is called *exploiting*. If we choose one of the nongreedy action, this process will be called *exploration* because this may improve observed values of nongreedy actions which may be optimal.

## Action value method
we denote the true (actual) value of action a as $q∗(a)$, and the estimated value on the tth time step as $Q_t(a)$. If by the $t^{th}$ time step action $a$ has been chosen $K_a$ times prior to $t$, yielding rewards $R_1, R_2,...R_{K_a}$, then its value is estimated to be:
$$Q_t(a) = \frac{R_1 + R_2 + .... + R_{K_a}}{K_a}$$

We choose explore with probability $\epsilon$ and choose greedy action with probability $(1-\epsilon)$ i.e Action with max $Q_t(a)$. This way of near greedy selection is called $\epsilon-$greedy method

#### Optimized way to update mean
Let mean at time t is $m_t$. Let new observation at time $t+1$ is $x_{t+1}$.
$$
\begin{align}
m_t &= \frac{x_1 + x_2 + ....... + x_t}{t} \tag{1} \\
m_{t+1} &= \frac{x_1 + x_2 + ....... + x_t + x_{t+1}}{t+1} \\
&= \frac{m_t*t + x_{t+1}}{t+1} && \text{by (1)} \\
&= \frac{m_t*t + m_t -m_t + x_{t+1}}{t+1}  \\
&= \frac{m_t(t + 1) + (x_{t+1}-m_t)}{t+1}  \\
m_{t+1} &= m_t + \frac{x_{t+1}-m_t}{t+1}
\end{align}
$$

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sb
%matplotlib inline
```

```python
#np.random.seed(seed=2017)

# Environment code
class Bandit:
    def __init__(self,n):
        self.n = n
        self.q_star_arr = np.random.normal(size=self.n)

    # get reward R
    def reward(self,a):
        return np.random.normal(loc=self.q_star_arr[a])

    def getOptimal(self):
        return self.q_star_arr.argmax()
```

```python
class ActionValue:
    def __init__(self,n,epsilon,bandit,max_t = 1000):
        self.n = n
        self.bandit = bandit
        self.max_t = max_t
        self.Qta = np.zeros(n)
        self.Ka = np.zeros(n,dtype=int)
        self.optimalHistory = np.zeros(max_t)
        self.epsilon = epsilon
        self.rewardHistory = np.zeros(max_t)
        self.algoRun()

    def algoRun(self):
        for t in range(self.max_t):
            greedyAction = self.Qta.argmax()

            if np.random.rand() < self.epsilon:#exploring
                curAction = np.random.randint(self.n -1)
                #not to use greedyAction so
                if curAction>=greedyAction:
                    curAction += 1
            else:#exploiting
                curAction = greedyAction

            self.optimalHistory[t] = 1 if curAction == self.bandit.getOptimal() else 0
            curActionReward = self.bandit.reward(curAction)
            self.rewardHistory[t] = curActionReward
            self.Ka[curAction] += 1
            #update Qt(a)
            self.Qta[curAction] += (curActionReward - self.Qta[curAction])/self.Ka[curAction]

# n =10
# bandit = Bandit(10)
# print(bandit.q_star_arr)
# max_t = 1000
# a1 = ActionValue(n,0.1,bandit,max_t=max_t)
# a1.algoRun()
# print(a1.Ka)
# a2 = ActionValue(n,0.01,bandit,max_t=max_t)
# a2.algoRun()
# print(a2.Ka)
# a3 = ActionValue(n,0.0,bandit,max_t=max_t)
# a3.algoRun()
# print(a3.Ka)
```

```python
class TestBed:
    def __init__(self, n, epsilon, max_t = 1000):
        self.n = n
        self.maxExp = 2000
        self.max_t = max_t
        self.epsilon = epsilon
        self.averageRewardHistory = np.zeros(self.max_t)
        self.averageOptimalHistory = np.zeros(self.max_t)

        for i in range(self.maxExp):
            bandit = Bandit(n)
            exp = ActionValue(self.n,self.epsilon,bandit,max_t=self.max_t)
            self.averageRewardHistory += exp.rewardHistory
            self.averageOptimalHistory += exp.optimalHistory

        self.averageRewardHistory /= self.maxExp
        self.averageOptimalHistory *= 100.0/self.maxExp
```

```python
n =10
max_t =1000
a1 = TestBed(n,0.1,max_t)
a2 = TestBed(n,0.01,max_t)
a3 = TestBed(n,0.0,max_t)
```

```python
t = range(max_t)
plt.plot(t,a1.averageRewardHistory,label='epsilon=0.1')
plt.plot(t,a2.averageRewardHistory,label='epsilon=0.01')
plt.plot(t,a3.averageRewardHistory,label='epsilon=0.0')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.show()
```

```python
plt.plot(t,a1.averageOptimalHistory,label='epsilon=0.1')
plt.plot(t,a2.averageOptimalHistory,label='epsilon=0.01')
plt.plot(t,a3.averageOptimalHistory,label='epsilon=0.0')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.show()
```


```python
%matplotlib inline
```

```python
import numpy as np
import numpy.polynomial.polynomial as p
import matplotlib.pyplot as plt
from turtle import *
```

# Basic Algebra Exercise
## Functions, Polynomials, Complex Numbers. Applications of Abstract Algebra

### Problem 1. Polynomial Interpolation
We know that if we have a set of $n$ data points with coordinates $(x_1; y_1), (x_2; y_2), \dots, (x_n; y_n)$, we can try to figure out what function may have generated these points.

Please note that **our assumptions about the data** will lead us to choosing one function over another. This means that our results are as good as our data and assumptions. Therefore, it's extremely important that we write down our assumptions (which sometimes can be difficult as we sometimes don't realize we're making them). It will be better for our readers if they know what those assumptions and models are.

In this case, we'll state two assumptions:
1. The points in our dataset are generated by a polynomial function
2. The points are very precise, there is absolutely no error in them. This means that the function should pass **through every point**

This method is called *polynomial interpolation* (*"polynomial"* captures assumption 1 and *"interpolation"* captures assumption 2).

It can be proved (look at [Wikipedia](https://en.wikipedia.org/wiki/Polynomial_interpolation) for example) that if we have $n$ data points, there is only one polynomial of degree $n-1$ which passes through them. In "math speak": "the vector spaces of $n$ points and polynomials of degree $n-1$ are isomorphic (there exists a bijection mapping one to the other)".

There are a lot of ways to do interpolation. We can also write the function ourselves if we want but this requires quite a lot more knowledge than we already covered in this course. So we'll use a function which does this for us. `numpy.polyfit()` is one such function. It accepts three main parameters (there are others as well, but they are optional): a list of $x$ coordinates, a list of $y$ coordinates, and a polynomial degree.

Let's say we have these points:
```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
```

First, we need to "extract" the coordinates:
```python
x = points[:, 0]
y = points[:, 1]
```

Then, we need to calculate the interpolating polynomial. For the degree, we'll set $n-1$:
```python
coefficients = np.polyfit(x, y, len(points) - 1)
poly = np.poly1d(coefficients)
```

After that, we need to plot the function. To do this, we'll create a range of $x$ values and evaluate the polynomial at each value:
```python
plot_x = np.linspace(np.min(x), np.max(x), 1000)
plot_y = poly(plot_x)
```

Finally, we need to plot the result. We'll plot both the fitting polynomial curve (using `plt.plot()`) and the points (using `plt.scatter`). It's also nice to have different colors to make the line stand out from the points.
```python
plt.plot(plot_x, plot_y, c = "green")
plt.scatter(x, y)
plt.xlabel("x")
plt.ylabel("y")
plt.show()
```
Don't forget to label the axes!

Your task now is to **wrap the code in a function**. It should accept a list of points, the polynomial degree, min and max value of $x$ used for plotting. **Be extremely careful to ensure that the function uses its parameters!**

We'll use this function to try some other cases.

```python
def interpolate_polynomial(points, degree, min_x, max_x):
    """
    Interpolates a polynomial of the specified degree through the given points and plots it
    points - a list of points (x, y) to plot
    degree - the polynomial degree
    min_x, max_x - range of x values used to plot the interpolating polynomial
    """
    x = points[:, 0]
    y = points[:, 1]
    coefficients = np.polyfit(x, y, degree)
    poly = np.poly1d(coefficients)
    plot_x = np.linspace(min_x, max_x, 1000)
    plot_y = poly(plot_x)
    plt.plot(plot_x, plot_y, c = "green")
    plt.scatter(x, y)
    plt.xlabel("x")
    plt.ylabel("y")
    plt.show()
    pass
```

```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

We see this is a very nice fit. This is expected, of course. Let's try to expand our view a little. Let's try to plot other values of $x$, further than the original ones. This is **extrapolation**.

```python
interpolate_polynomial(points, len(points) - 1, -5, 10)
```

Hmmm... it seems our polynomial goes a little wild outside the original range. This is to show how **extrapolation can be quite dangerous**.

Let's try a lower polynomial degree now. We used 4, how about 3, 2 and 1?
**Note:** We can add titles to every plot so that we know what exactly we're doing. Te title may be passed as an additional parameter to our function.

```python
interpolate_polynomial(points, 3, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 2, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

We see the fitting curves (or line in the last case) struggle more and more and they don't pass through every point. This breaks our assumptions but it can be very useful.

Okay, one more thing. How about increasing the degree? Let's try 5, 7 and 10. Python might complain a little, just ignore it, everything is fine... sort of :).

```python
interpolate_polynomial(points, 5, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 7, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 10, np.min(points[:, 0]), np.max(points[:, 0]))
```

Those graphs look pretty much the same. But that's the point exactly. I'm being quite sneaky here. Let's try to expand our view once again and see what our results really look like.

```python
interpolate_polynomial(points, 5, -10, 10)
interpolate_polynomial(points, 7, -10, 10)
interpolate_polynomial(points, 10, -10, 10)
```

Now we see there are very wild differences. Even though the first two plots look quite similar, look at the $y$ values - they're quite different.

So, these are the dangers of interpolation. Use a too high degree, and you get "the polynomial wiggle". These are all meant to represent **the same** data points but they look insanely different. Here's one more comparison.

```python
interpolate_polynomial(points, len(points) - 1, -2, 7)
interpolate_polynomial(points, len(points) + 1, -2, 7)
```

Now we can see what big difference even a small change in degree can make. This is why we have to choose our interpolating functions very carefully. Generally, a lower degree means a simpler function, which is to be preferred. See [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor).

And also, **we need to be very careful about our assumptions**.

```python
points = np.array([(-5, 0.03846), (-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882), (5, 0.03846)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

This one definitely looks strange. Even stranger, if we remove the outermost points... ($x = \pm 5$), we get this

```python
points = np.array([(-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882)])
interpolate_polynomial(points, len(points - 1), np.min(points[:, 0]), np.max(points[:, 0]))
```

This is because the generating function is not a polynomial. It's actually:
$$ y = \frac{1}{1 + x^2} $$

Plot the polynomial interpolation and the real generating function **on the same plot**. You may need to modify the original plotting function or just copy its contents.

```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
x = points[:, 0]
y = points[:, 1]
coefficients = np.polyfit(x, y, len(points) - 1)
poly = np.poly1d(coefficients)
plot_x = np.linspace(np.min(x), np.max(x), 1000)
plot_y = poly(plot_x)
plt.plot(plot_x, plot_y, c = "green")
x = np.linspace(0, 5, 10)
y = 1 / (1 + x*x)
plt.plot(x, y, c = "red")
plt.scatter(x, y)
plt.xlabel("x")
plt.ylabel("y")
plt.show()
```

### Problem 2. Complex Numbers as Vectors
We saw that a complex number $z = a + bi$ is equivalent to (and therefore can be represented as) the ordered tuple $(a; b)$, which can be plotted in a 2D space. So, complex numbers and 2D points are equivalent. What is more, we can draw a vector from the origin of the coordinate plane to our point. This is called a point's **radius-vector**.

Let's try plotting complex numbers as radius vectors. Don't forget to label the real and imaginary axes. Also, move the axes to the origin. Hint: These are called "spines"; you'll need to move 2 of them to the origin and remove the other 2 completely. Hint 2: You already did this in the previous lab.

We can use `plt.quiver()` to plot the vector. It can behave a bit strangely, so we'll need to set the scale of the vectors to be the same as the scale on the graph axes:
```python
plt.quiver(0, 0, z.real, z.imag, angles = "xy", scale_units = "xy", scale = 1)
```

Other than that, the main parameters are: $x_{begin}$, $y_{begin}$, $x_{length}$, $y_{length}$ in that order.

Now, set the aspect ratio of the axes to be equal. Also, add grid lines. Set the axis numbers (called ticks) to be something like `range(-3, 4)` for now.
```python
plt.xticks(range(-3, 4))
plt.yticks(range(-3, 4))

```

If you wish to, you can be a bit more clever with the tick marks. Find the minimal and maximal $x$ and $y$ values and set the ticks according to them. It's a good practice not to jam the plot too much, so leave a little bit of space. That is, if the actual x-range is $[-2; 2]$, set the plotting to be $[-2.5; 2.5]$ for example. Otherwise, the vector heads (arrows) will be "jammed" into a corner or side of the plot.

```python
def plot_complex_number(z):
    """
    Plots the complex number z as a radius vector in the 2D space
    """
    plt.quiver(0, 0, z.real, z.imag, angles = "xy", scale_units = "xy", scale = 1)
    plt.xticks(range(-5, 8))
    plt.yticks(range(-5, 8))
    plt.gca().set_aspect("equal")
    plt.show()
    pass

plot_complex_number(2 + 3j)
```

How about many numbers? We'll need to get a little bit more creative. First, we need to create a 2D array, each element of which will be a 4-element array: `[0, 0, z.real, z.imag]`. Next, `plt.quiver()` can accept a range of values. Look at [this StackOverflow post](https://stackoverflow.com/questions/12265234/how-to-plot-2d-math-vectors-with-matplotlib) for details and adapt your code.

```python
def plot_complex_numbers(numbers, colors):
    """
    Plots the given complex numbers as radius vectors in the 2D space
    """
    start = [0] * len(numbers)
    reals = [number.real for number in numbers]
    imags = [number.imag for number in numbers]
    plt.quiver(start, start, reals, imags, angles = "xy", scale_units = "xy", scale = 1, color = colors)
    plt.xticks(range(-5, 8))
    plt.yticks(range(-5, 8))
    plt.gca().set_aspect("equal")
    plt.show()
    pass
plot_complex_numbers([2 + 3j, -2 - 1j, -3, 2j], ["green", "red", "blue", "orange"])
```

Now let's see what the operations look like. Let's add two numbers and plot the result.

```python
z1 = 2 + 3j
z2 = 1 - 1j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

We can see that adding the complex numbers is equivalent to adding vectors (remember the "parallelogram rule"). As special cases, let's try adding pure real and pure imaginary numbers:

```python
z1 = 2 + 3j
z2 = 2 + 0j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

```python
z1 = 2 + 3j
z2 = 0 + 2j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

How about multiplication? First we know that multiplying by 1 gives us the same vector and mulpiplying by -1 gives us the reversed version of the same vector. How about multiplication by $\pm i$?

```python
z = 2 + 3j
plot_complex_numbers([z, z * 1], ["red", "blue"])
plot_complex_numbers([z, z * -1], ["red", "blue"])
plot_complex_numbers([z, z * 1j], ["red", "blue"])
plot_complex_numbers([z, z * -1j], ["red", "blue"])
```

So, multiplication by $i$ is equivalent to 90-degree rotation. We can actually see the following equivalence relationships between multiplying numbers and rotation about the origin:

| Real | Imaginary | Result rotation |
|------|-----------|-----------------|
| 1    | 0         | $0^\circ$       |
| 0    | 1         | $90^\circ$      |
| -1   | 0         | $180^\circ$     |
| 0    | -1        | $270^\circ$     |

Once again, we see the power of abstraction and algebra in practice. We know that complex numbers and 2D vectors are equivalent. Now we see something more: addition and multiplication are equivalent to translation (movement) and rotation!

Let's test the multiplication some more. We can see the resulting vector is the sum of the original vectors, but *scaled and rotated*:

```python
z1 = 2 + 3j
z2 = 1 - 2j
plot_complex_numbers([z1, z2, z1 * z2], ["red", "blue", "green"])
```

### Problem 3. Recursion and Fractals

> "To understand recursion, you first need to understand recursion."

There are three main parts to a recursive function:
1. Bottom - when the recursion should finish
2. Operation - some meaningful thing to do
3. Recursive call - calling the same function
4. Clean-up - returning all data to its previous state (this reverses the effect of the operation)

Let's do one of the most famous recursion examples. And I'm not talking about Fibonacci here. Let's draw a tree using recursive functions.

The figure we're going to draw is called a **fractal**. It's self-similar, which means that if you zoom in on a part of it, it will look the same. You can see fractals everywhere in nature, with broccoli being one of the prime examples. Have a look:

First, we need to specify the recursive part. In order to draw a tree, we need to draw a line of a given length (which will be the current branch), and then draw two more lines to the left and right. By "left" and "right", we should mean "rotation by a specified angle".

So, this is how to draw a branch: draw a line and prepare to draw two more branches to the left and right. This is going to be our recursive call.

To make things prettier, more natural-looking (and have a natural end to our recursion), let's draw each "sub-branch" a little shorter. If the branch becomes too short, it won't have "child branches". This will be the bottom of our recursion.

There's one more important part of recursion, and this is **"the clean-up"**. After we did something in the recursive calls, it's very important to return the state of everything as it was **before** we did anything. In this case, after we draw a branch, we go back to our starting position.

Let's first import the most import-ant (no pun intended...) Python drawing library: `turtle`! In order to make things easier, we'll import all methods directly.
```python
from turtle import *
```

You can look up the docs about turtle if you're more interested. The basic things we're going to use are going forward and backward by a specified number of pixels and turning left and right by a specified angle (in degrees).

Let's now define our recursive function:
```python
def draw_branch(branch_length, angle):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle)
        left(2 * angle)
        draw_branch(branch_length - 15, angle)
        right(angle)
        backward(branch_length)
```

And let's call it:
```python
draw_branch(100, 20)
```

We need to start the tree not at the middle, but toward the bottom of the screen, so we need to make a few more adjustments. We can wrap the setup in another function and call it. Let's start one trunk length below the center (the trunk length is the length of the longest line).

**Note:** It's important to call `done()` after the drawing is finished. If you miss it, the window with the turtle drawing will freeze and throw an exception.

```python
def draw_tree(trunk_length, angle):
    speed("fastest")
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

Note that the graphics will show in a separate window. Also note that sometimes you might get bugs. If you do, go to Kernel > Restart.

```python
def draw_tree(trunk_length, angle):
    speed("fastest")
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

```python
draw_tree(100, 20)
```

Experiment with different lengths and angles. Especially interesting angles are $30^\circ$, $45^\circ$, $60^\circ$ and $90^\circ$.

```python
draw_tree(100, 30)
```

```python
draw_tree(100, 45)
```

```python
draw_tree(100, 90)
```

Now modify the original function a little. Draw the lines with different thickness. Provide the trunk thickness at the initial call. Similar to how branches go shorter, they should also go thinner.

```python
def draw_branch(branch_length, angle, thickness):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle, thickness / 2)
        left(2 * angle)
        draw_branch(branch_length - 15, angle, thickness / 2)
        right(angle)
        width(thickness)
        backward(branch_length)
def draw_tree(trunk_length, angle, thickness = 10):
    speed("fastest")
    width(thickness)
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle, thickness / 2)
    done()
draw_tree(100, 20, 8)
```

#### * Optional problem
Try to draw another kind of fractal graphic using recursion and the `turtle` library. Two very popular examples are the "Koch snowflake" and the "Sierpinski triangle". You can also modify the original tree algorithm to create more natural-looking trees. You can, for example, play with angles, number of branches, lengths, and widths. The Internet has a lot of ideas about this :). Hint: Look up **"L-systems"**.

### Problem 4. Run-length Encoding
One application of algebra and basic math can be **compression**. This is a way to save data in less space than it originally takes. The most basic form of compression is called [run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding).

Write a function that encodes a given text. Write another one that decodes.

We can see that RLE is not very useful in the general case. But it can be extremely useful if we have very few symbols. An example of this can be DNA and protein sequences. DNA code, for example, has only 4 characters.

Test your encoding and decoding functions on a DNA sequence (you can look up some on the Internet). Measure how much your data is compressed relative to the original.

```python
import re
def encode(text):
    """
    Returns the run-length encoded version of the text
    (numbers after symbols, length = 1 is skipped)
    """
    counts = {}
    for c in text:
        if counts.get(c) is None:
            counts[c] = text.count(c)
        output = ''
        for key,value in counts.items():
            output = output + key
            if value > 1:
                output = output + str(value)
    return output

def decode(text):
    """
    Decodes the text using run-length encoding
    """
    output = ''
    for (char, num) in re.findall(r'([a-zA-Z])([0-9]*)', text):
        if num == '':
            output = output + char
        else:
            output = output + (char * int(num))
    return output

```

```python
# Tests
# Test that the functions work on their own
assert encode("AABCCCDEEEE") == "A2BC3DE4"
assert decode("A2BC3DE4") == "AABCCCDEEEE"

# Test that the functions really invert each other
assert decode(encode("AABCCCDEEEE")) == "AABCCCDEEEE"
assert encode(decode("A2BC3DE4")) == "A2BC3DE4"
```

### * Problem 5. Function Invertibility and Cryptography
As we already saw, some functions are able to be inverted. That is, if we know the output, we can see what input generated it directly. This is true if the function is **one-to-one correspondence** (bijection).

However, not all functions are created the same. Some functions are easy to compute but their inverses are extremely difficult. A very important example is **number factorization**. It's relatively easy (computationally) to multiply numbers but factoring them is quite difficult. Let's run an experiment.

We'll need a function to generate random n-bit numbers. One such number can be found in the `Crypto` package
```python
from Crypto.Util import number
random_integer = number.getRandomNBitInteger(n_bits)
```

We could, of course, write our factorization by hand but we'll use `sympy`
```python
from sympy.ntheory import factorint
factorint(1032969399047817906432668079951) # {3: 2, 79: 1, 36779: 1, 7776252885493: 1, 5079811103: 1}
```

This function returns a `dict` where the keys are the factors, and the values - how many times they should be multiplied.

We'll also need a tool to accurately measure performance. Have a look at [this one](https://docs.python.org/3/library/time.html#time.time) for example.

Specity a sequence of bit lengths, in increasing order. For example, you might choose something like `[10, 20, 25, 30, 32, 33, 35, 38, 40]`. Depending on your computer's abilities you can go as high as you want. For each bit length, generate a number. See how much time it takes to factor it. Then see how much time it takes to multiply the factors. Be careful how you measure these. You shouldn't include the number generation (or any other external functions) in your timing.

In order to have better accuracy, don't do this once per bit length. Do it, for example, five times, and average the results.

Plot all multiplication and factorization times as a function of the number of bits. You should see that factorization is much, much slower. If you don't see this, just try larger numbers :D.

```python
# Write your code here
```

### * Problem 6. Diffie - Hellman Simulation
As we already saw, there are functions which are very easy to compute in the "forward" direction but really difficult (computationally) to invert (that is, determine the input from the output). There is a special case: the function may have a hidden "trap door". If you know where that door is, you can invert the function easily. This statement is at the core of modern cryptography.

Look up **Diffie - Hellman key exchange** (here's a [video](https://www.youtube.com/watch?v=cM4mNVUBtHk) on that but feel free to use anything else you might find useful).

Simulate the algorithm you just saw. Generate large enough numbers so the difference is noticeable (say, factoring takes 10-15 seconds). Simulate both participants in the key exchange. Simulate an eavesdropper.

First, make sure after both participants run the algotihm, they have *the same key* (they generate the same number).

Second, see how long it takes for them to exchange keys.

Third, see how long it takes the eavesdropper to arrive at the correct shared secret.

You should be able to see **the power of cryptography**. In this case, it's not that the function is irreversible. It can be reversed, but it takes a really long time (and with more bits, we're talking billions of years). However, if you know something else (this is called a **trap door**), the function becomes relatively easy to invert.

```python
# Write your code here
```

### ** Problem 7. The Galois Field in Cryptography
Research about the uses of the Galois field. What are its properties? How can it be used in cryptography? Write a simple cryptosystem based on the field.

You can use the following questions to facilitate your research:
* What is a field?
* What is GF(2)? Why is it an algebraic field?
* What is perfect secrecy? How does it relate to the participants in the conversation, and to the outside eavesdropper?
* What is symmetrical encryption?
* How to encrypt one-bit messages?
* How to extend the one-bit encryption system to many buts?
* Why is the system decryptable? How do the participants decrypt the encrypted messages?
* Why isn't the eavesdropper able to decrypt?
* What is a one-time pad?
    * How does the one-time pad achieve perfect secrecy?
* What happens if we try to use a one-time pad many times?
    * Provide an example where you break the "many-time pad" security
* What are some current enterprise-grade applications of encryption over GF(2)?
* Implement a cryptosystem based on GF(2). Show correctness on various test cases

### ** Problem 8. Huffman Compression Algorithm
Examine and implement the **Huffman algorithm** for compressing data. It's based on information theory and probiability theory. Document your findings and provide your implementation.

This algorithm is used for **lossless compression**: compressing data without loss of quality. You can use the following checklist:

* What is the difference betwenn lossless and lossy compression?
* When can we get away with lossy compression?
* What is entropy?
* How are Huffman trees constructed?
    * Provide a few examples
* How can we get back the uncompressed data from the Huffman tree?
* How and where are Huffman trees stored?
* Implement the algorithm. Add any other formulas / assumptions / etc. you might need.
* Test the algorithm. A good meaure would be percentage compression: $$\frac{\text{compressed}}{\text{uncompressed}} * 100\%$$
* How well does Huffman's algorithm perform compared to other compression algorithms (e.g. LZ77)?


```python
import numpy as np
import scipy
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
```

The general functional dependence of a signal vs. pH can be expressed as follows.

$$
\begin{equation}
\label{ee}
S = \frac{S_B + S_A 10^{pK_a - pH}}{1 + 10^{pK_a - pH}}
\end{equation}
$$

```python
def S(pH, pK, SA, SB):
    return ((SA + SB * 10 ** (pK -pH)) / (1 + 10 ** (pK -pH)))
```

If we consider 2 signals S1 and S2:

$$
\begin{align}
E \equiv 10^{pK_a - pH} &&
A1 \equiv S_A^1 &&
A2 \equiv S_A^2 \\
&&
B1 \equiv S_B^1 &&
B2 \equiv S_B^2 \\
\end{align}
$$

It is always possible to build an isosbestic signal:

$$
(B1 - A1) (S2 - A2 + 1) - (B2- A2) (S1 - A1 + 1)
$$

```python
pK = 7
ph = np.linspace(4,9,30)

def iso(a1, a2, b1, b2):
    s1 = S(ph, pK, b1, a1)
    s2 = S(ph, pK, b2, a2)
    plt.plot(ph, s1)
    plt.plot(ph, s2)
    plt.legend(['S1','S2'])
    plt.figure()
    iso = (b1 - a1) * (s2 - a2 + 1) - (b2 - a2) * (s1 - a1 + 1)
    iso = abs(iso)/iso.max()
    norm_sum = (s1 - a1) / (b1 - a1) + (s2 - a2) / (b2 - a2)
    plt.plot(ph, iso, 'b')
    plt.plot(ph, norm_sum, 'r')
    plt.legend(['isosbestic', 'normalized sum'], loc=2)
    plt.figure()
    plt.plot(ph, s1 / s2)
    plt.legend(['S1 / S2'], loc=2)
    plt.figure()
    plt.plot(ph, s1 * s2)
    plt.legend(['S1 * S2'], loc=2)

iso(5, 20, 10, 3)
```

```python
n = np.random.poisson(19, size=10)
d = np.random.poisson(9, size=10)
n/d, 19/9, np.exp(np.log(n) - np.log(d))
```

    (array([ 1.85714286,  2.11111111,  1.875     ,  1.77777778,  2.42857143,
             2.3       ,  1.54545455,  2.18181818,  1.61538462,  1.41666667]),
     2.111111111111111,
     array([ 1.85714286,  2.11111111,  1.875     ,  1.77777778,  2.42857143,
             2.3       ,  1.54545455,  2.18181818,  1.61538462,  1.41666667]))


# Python and NumPy Programming Philosophy

If you have experience with other programming languages like C/C++, it my take a while for you to adapt to the Python nd NumPy programming style.The take aways from this tutorial are:
- Python programming style is different than other traditional languages like C/C++. By using built-in commands and functions, you should be able to write clear and concise code.
- Explicit Python loops are slow. You should avoild explicit Python loops at all costs, and let the compiled NumPy functions do these loops for you.
- For most machine learning applications, using single-precision floating point numbers  is enough and saves you twice the memory.

```python
import numpy as np
import matplotlib.pylab as plt
```

## Array Initialization

Let's say we want to initialize an one-dimensional array of size  **N** to a constant value **c**. Let's compare the time performance using a traditional programming style like C/C++ and others against the Pythonic way.

```python
N = 100000000
c = 33
```

```python
%%timeit # built-in magic command https://ipython.readthedocs.io/en/stable/interactive/magics.html

a = np.empty(N)
counter = 0

while counter < N:
    a[counter] = c
    counter+=1
```

    10.8 s ± 327 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%%timeit

a = np.empty(N)
a[:] = c
```

    117 ms ± 1.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

## Counting Positive Values in an Array

Let's count the positive values in an array using an explicit loop and an implicit NumPy loop.

```python
M = 1000000
b = np.random.randn(M) # Random array following normal distribution
```

```python
%%timeit
nz = 0
for ii in range(b.size):
    if b[ii] > 0:
        nz+=1
```

    289 ms ± 8.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%%timeit
nz = (b > 0).sum()
```

    1.02 ms ± 9.86 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

## Accesing Different Axes of the Array

Now, let's count the number of positive values for each row of a two-dimensional array.

```python
W = 1000
c = np.random.randn(W,W) # Random array following normal distribution
print(c.shape)
```

    (1000, 1000)

```python
%%timeit
ax = 0 # 0 is the first axis. By convention the first axis are the rows of a 2D array
nz2 = np.empty(c.shape[ax])
counter = 0
# Initialize array to zeros
while counter < W:
    nz2[counter] = 0
    counter+=1

# count positive values
for ii in range(c.shape[0]):
    for jj in range(c.shape[1]):
        if c[ii,jj] > 0:
            nz2[ii]+=1
```

    485 ms ± 9.95 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%%timeit
nz2 = (c>0).sum(axis = 1)
```

    1.01 ms ± 6.29 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

## Implementing a Function to Compute the Local Binary Pattern

Local Binary Pattern (LBP) is a simple yet very efficient texture feature commonly used in image processing.

\begin{align}
LBP(p) = \sum_{i=0}^{7} [f(n_i) > f(p)]2^i$
\end{align}

```python
# First student to propose new implementation that is more efficient and still elegant gets 1 point in the final grade
def local_binary_pattern(img):
    neighbors = [(0,2),(1,2),(2,2),\
                 (2,1), (2,0), (1,0),\
                 (0,0),(0,1)]
    H,W = img.shape
    # np.pad -> can do the padding for you
    aux_img = np.zeros((H+2,W+2))
    aux_img[1:-1,1:-1] = img
    lbp = np.zeros(img.shape, dtype = np.uint8) #uint8
    for i, (x,y) in enumerate(neighbors):
        lbp+= (aux_img[x:H+x,y:W+y] > img)*2**i
    return lbp
```

```python
from PIL import Image

img = np.array(Image.open('../Figures/lena.png').convert('L'))
lbp = local_binary_pattern(img)
print(lbp.dtype)
print(img.shape)
plt.figure()
plt.subplot(121)
plt.imshow(img, cmap = "gray")
plt.axis("off")
plt.subplot(122)
plt.imshow(lbp, cmap = "gray")
plt.axis("off")
plt.show()
```

## Standardizing your data channel-wise

Image dimensions: $(H,W,3)$

\begin{align}
f_R = \frac{f_R - mean(f_R)}{std(f_R)}
\end{align}

\begin{align}
f_G = \frac{f_G - mean(f_G)}{std(f_G)}
\end{align}

\begin{align}
f_B = \frac{f_B - mean(f_B)}{std(f_B)}
\end{align}

```python
def standardize_img(f):
    f_new = (f - f.mean(axis = (0,1)))/f.std(axis = (0,1))
    return f_new
```

```python
img = np.array(Image.open('../Figures/lena.png').convert('RGB'))[:510,:,:]
print(img.shape)
print(img.mean())

print(img.mean(axis = (0),keepdims = True).shape)

img2 = standardize_img(img)

# Before scaling
print(img.mean(axis = (0,1)))
print(img.std(axis = (0,1)))

# After scaling
print(img2.mean(axis = (0,1)))
print(img2.std(axis = (0,1)))
```

    (510, 512, 3)
    89.10454197303922
    (1, 512, 3)
    [89.10454197 89.10454197 89.10454197]
    [60.79930718 60.79930718 60.79930718]
    [-7.64950297e-15 -7.64950297e-15 -7.64950297e-15]
    [1. 1. 1.]

## Floating point precision

```python
print(img2.nbytes)
print(img2.dtype)
```

    6266880
    float64

```python
img3 = img2.astype(np.float32)
print(img3.nbytes)
print(img3.dtype)
```

    3133440
    float32

## Loading Datasets and Splitting the Data

```python
import tensorflow as tf

# Loading the data using the Keras function
(X_dev, Y_dev), (X_test, Y_test) = tf.keras.datasets.mnist.load_data() # The data comes already split
                                                                        # in dev and test sets
print("Development set")
print("Images: ",X_dev.shape)
print("Labels shape:",Y_dev.shape)
print("\nNumber of classes:",np.unique(Y_dev).size)
print("\nClasses:",np.unique(Y_dev))
print("\nTest set")
print("Images: ",X_test.shape)
print("Labels shape: ",Y_test.shape)
```

    Development set
    Images:  (60000, 28, 28)
    Labels shape: (60000,)

    Number of classes: 10

    Classes: [0 1 2 3 4 5 6 7 8 9]

    Test set
    Images:  (10000, 28, 28)
    Labels shape:  (10000,)

```python
# X -> (#samples, H, W) -> (#samples, H,W, #channels)
X_dev2 = X_dev[:,:,:, np.newaxis]
X_dev3 = X_dev[:,:,np.newaxis,: ]
print(X_dev.shape)
print(X_dev2.shape)
print(X_dev3.shape)
```

    (60000, 28, 28)
    (60000, 28, 28, 1)
    (60000, 28, 1, 28)

```python
H,W,Z = X_dev.shape
print(X_dev4.shape)

```

    (60000, 28, 28, 1)

```python
X_dev = X_dev.astype(np.float32)
X_dev4 = X_dev.reshape(H,W,Z,1)
print(X_dev[0,0,0])
print(X_dev4[0,0,0,0])
X_dev4[0,0,0,0] = -200
print(X_dev[0,0,0])
print(X_dev4[0,0,0,0])
```

    246.0
    246.0
    -200.0
    -200.0

```python
print(X_dev.dtype) # 0 -255
```

    uint8

```python

```


```python
import numpy as np
import jhu2016_labs.plotting as plotting
import jhu2016_labs.models as models

%matplotlib inline
```

## Training data

In real scenario, we  observed a set of values, often called features, and we try to learn some model that can "explain" the observed features. Generally, we do not know the true distribution of the features. For this work we will simulate a set of single dimensional features sampled from a GMM. We will call this GMM the "true model" and in this notebook we will see different strategies to approximate the true model from the simulated training data.

```python
true_model = models.GMM([-4, 0, 2], [1, 2, .7], [0.2, 0.2, 0.6])
X = true_model.sampleData(50)
Y = np.zeros_like(X) # The random variable has a single dimension !
x_min = -8
x_max = 7
y_min = -0.01
y_max = .4
fig, ax = plotting.plotGMM(true_model, x_min=x_min, x_max=x_max, y_min=y_min, y_max=y_max,
                           label='True model')
ax.plot(X, Y, '+', color='b', label='data')
ax.legend(loc='upper left')
```

## Gaussian density

We can first try to model the observed data with a Gaussian density function. The Gaussian density function is defined as:
$$
\DeclareMathOperator{\Norm}{\mathcal{N}}
\DeclareMathOperator{\Gam}{Gam}
\DeclareMathOperator{\e}{exp}
p(x \mid \mu, \sigma^2) = \Norm(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \e\{ \frac{-(x - \mu)^2}{2\sigma^2} \}
$$
* $\mu$ is the **mean** of the Gaussian density
* $\sigma^2$ is the **variance** of the Gaussian density

##### TODO
Try to play with the mean and the variance of the Gaussian. How each parameter change the density ?

```python
gaussian = models.Gaussian(0., 1.) # <- 1st parameter is mean, second variance0
fig, ax = plotting.plotGaussian(gaussian)
```

Alternatively, the Gaussian density function can be parameterized with a precision $\lambda$ which is the inverse of the variance:
$$
    \DeclareMathOperator{\e}{exp}
    p(x | \mu, \lambda) = \Norm(x \mid \mu, \lambda^{-1}) = \frac{\sqrt{\lambda}}{\sqrt{2\pi}} \e\{ \frac{-\lambda(x - \mu)^2}{2} \}
$$
* $\mu$ is the **mean** of the Gaussian density
* $\lambda = \frac{1}{\sigma^2}$ is the **precision** of the Gaussian density

This alternative parameterization will prove to be useful when dealing with Bayesian inference for the Gaussian/GMM density.

### Maximum Likelihood estimation

We will first try to model our data with a simple Gaussian. We need to find the parameters (mean and variance) that fit the best the data.

$$
\begin{align}
\mu &= \frac{1}{N} \sum_i x_i \\
\sigma^2 &= \frac{1}{N} \sum_i x_i^2
\end{align}
$$

##### TODO
From the above equations, implement the maximum-likelihood solution for the Gaussian. Does the Gaussian seems to be a good model for the true density ?

```python
##########################
# Write your code here.
mean_ml =
var_ml =
##########################

# Check your solutions uncomment the following code
test_gaussian_ml = models.Gaussian.maximumLikelihood(X)
assert np.isclose(test_gaussian_ml.mean, mean_ml), 'incorrect mean'
assert np.isclose(test_gaussian_ml.var, var_ml), 'incorrect variance'
print('correct solution !')
gaussian_ml = models.Gaussian(mean_ml, var_ml)
llh = gaussian_ml.logLikelihood(X)
print('log-likelihood:', llh)

y_min = -.01
y_max = .4
fig, ax = plotting.plotGMM(true_model, y_min=y_min, y_max=y_max, label='True model')
plotting.plotGaussian(gaussian_ml, fig=fig, ax=ax, color='red', label='Gaussian ML')
ax.plot(X, Y, '+', color='b', label='data')
ax.legend(loc='upper left')
```

### Bayesian Inference

We can work within the Bayesian framework with the Gaussian density by putting a prior over the mean and precision. A common an convenient choice of prior for the Gaussian is the **Normal-Gamma** prior:

$$
p(\mu, \lambda \mid m_0, \kappa_0, a_0, b_0) = \Norm(\mu \mid m_0, (\kappa_0 \lambda)^{-1}) \Gam(\lambda \mid a_0, b_0)
$$

where:

$$
\Gam(\lambda \mid a_0, b_0) = \frac{1}{\Gamma(a_0)} b_0^{a_0} \lambda^{a_0 - 1} \e \{ -b_0 \lambda\}
$$

$m_0$, $\kappa_0$, $a_0$ and $b_0$ are called **hyper-parameters**. They are the parameters of the prior distribution.

##### TODO
Play with the parameters of the Normal-Gamma. How each parameter affects the density ?

```python
ng_prior = models.NormalGamma(0, 2, 5, 6)
x_min =-2.
x_max = 2.
y_min = 0.
y_max = 2
plotting.plotNormalGamma(ng_prior, x_min=x_min, x_max=x_max, y_min=y_min, y_max=y_max)
```

Because the Normal-Gamma is the **conjugate prior** of the Normal density, the posterior distribution $p(\mu, \lambda \mid \mathbf{x})$ has a closed form solution:

$$
p(\mu, \lambda \mid \mathbf{x}) = \Norm(\mu \mid m_n, (\kappa_n \lambda)^{-1}) \Gam(\lambda \mid a_n, b_n)
$$

where:

$$
\begin{align}
m_n &= \frac{\kappa_0 m_0 + N \bar{x}} {\kappa_0 +  N} \\
\kappa_n &= \kappa_0 + N \\
a_n &= a_0 + \frac{N}{2} \\
b_n &= b_0 + \frac{N}{2} ( s + \frac{\kappa_0 (\bar{x} - m_0)^2}{\kappa_0 + N} ) \\
\bar{x} &= \frac{1}{N} \sum_i x_i \\
s &= \frac{1}{N} \sum_i (x_i - \bar{x})^2
\end{align}
$$

$N$ is the total number of point in the the training data and $m_n$, $\kappa_n$, $a_n$ and $b_n$ are the parameters of the posterior. Note that they are different from the hyper-parameters !!

##### TODO
Compute the posterior distribution with 1, 5, 10, 20 and 50  data points from the training data. What do you observe ?

```python
ng_prior = models.NormalGamma(0, 2, 5, 6)
ng_posterior = ng_prior.posterior(X[:1]) # <- Change '1' to the number of sample you want to use
x_min =-2
x_max = 2
y_min = 0.
y_max = 2
plotting.plotNormalGamma(ng_posterior, x_min=x_min, x_max=x_max, y_min=y_min, y_max=y_max)
```

### Predictive probability

Now that we have our posterior distibution we can predict the probability of a new data point given the training data:

$$
p(x' \mid \mathbf{x}) = \int p(x' \mid \theta) p(\theta \mid \mathbf{x}) d \theta
$$

For the Gaussian with Normal-Gamma prior the marginal predictive distribution is the Student's t distribution:

$$
\newcommand{\diff}{\mathop{}\!d}
\DeclareMathOperator{\St}{\mathcal{St}}
p(x' \mid \mathbf{x}) = \int_{-\infty}^{\infty} p(x|\mu, \lambda) p(\mu, \lambda \mid \mathbf{x}) \diff \mu \diff \lambda  = \St(x' \mid \mu_n, \nu, \gamma)
$$

where:

$$
\begin{align}
\nu &= 2a_n \\
\gamma &= \frac{a_n \kappa_n}{b_n(\kappa_n + 1)} \\
\St(x' \mid \mu_n, \nu, \gamma) &= \frac{\Gamma(\frac{\nu}{2} + \frac{1}{2})}{\Gamma(\frac{\nu}{2})} \Big( \frac{\gamma}{\pi \nu} \Big)^{\frac{1}{2}} \Big[ 1 + \frac{\gamma (x - \mu_n)^2}{\nu} \Big]^{-\frac{\nu}{2} - \frac{1}{2}}
\end{align}
$$

##### TODO
Compute the log-likelihood of the predictive distribution derived from a posterior trained with 1, 5, 10, 20  and 50 data points. How does it compare to the log-likelihood of the Gaussian trained with maximum likelihood ?

```python
ng_posterior = ng_prior.posterior(X[:1]) # <- Change '1' by the number of sample you want to use.
predict_pdf = ng_posterior.predictiveDensity()
llh = predict_pdf.logLikelihood(X)
print('log-likelihood:', llh)
```

## Gaussian Mixture Model (GMM)

The Gaussian density is a very simple function, however, in most cases of interest the density we try to model has a complex shape that cannot be expressed with a simple formula. A solution is to assume that our complex density is made of $K$ Gaussian densities. This is called a Gaussian Mixture Model (GMM) and it is defined as:
$$
    p(x|\boldsymbol{\mu}, \boldsymbol{\lambda}, \boldsymbol{\pi}) = \sum_{k=1}^{K} \pi_k \Norm(x|\mu_k, (\lambda_k)^{-1})
$$
* $\boldsymbol{\mu}$ is the vector of $K$ means
* $\boldsymbol{\lambda}$ is the vector of $K$ precisions
* $\boldsymbol{\pi}$ is the vector of $K$ weights such that $\sum_{k=1}^K \pi_k = 1$

##### TODO
Observe the influence of each parameters on the density and add/remove some components. Does this model suffer of the same drawbacks of the Gaussian density ?

```python
# Change the initial values of the parameters.
# you can add a component by removing a mean, a variance
# and a weight. All arrays shall always have the same number
# of elements ! Be careful, make sure that the sum of weight
# always sum up to one.
means = [-4.0, 0.0, 4.0, 5]
variances = [1.0, 2.0, 1.4, 1]
weights = [0.1, 0.4, 0.2, 0.3]
gmm = models.GMM(means, variances, weights)

y_min = 0.
y_max = .3
plotting.plotGMM(gmm, show_components=True, y_min=y_min, y_max=y_max)
```

## Maximum Likelihood estimation

The GMM parameters can be estimated with the **Expectation-Maximization** (EM) algorithm. The EM algorithm is an iterative algorithm that converges toward a (local) maximum of the log-likelihood of the data given the model. The EM training is as follows:
* initialize the parameters of the GMM
* iterate until convergence:
    * Expectation (E-step): compute the probability of the latent variable for each data point
    * Maximization (M-step): update the parameters from the statistics of the E-step.

##### TODO
* We have implemented a simple version of the EM algorithm that iterates 10 times. Change the following code to iterates until convergence of the log-likelihood. We will consider that the log-liklihood has converged if $ \log p(\mathbf{x} | \theta^{new}) - \log p(\mathbf{x} | \theta^{old}) \le 0.01$.
* Run the EM algorithm with different initialization: is the final log-likelihood always the same ? What can you conclude ? Is the log-likelihood greater than the one from the Gaussian estimated with Maximum-Likelihood ?

```python
# Initialization of the model
means = [-4.0, 0.0, 4.0]
variances = [1.0, 1.0, 1.]
weights = [0.3, 0.4, 0.3]
gmm = models.GMM(means, variances, weights)

## EM algorithm
print('initial log-likelihood:', gmm.logLikelihood(X))
############################################
# Change this loop to check the convergence of the EM .
for i in range(10):
    # E-step
    Z = gmm.EStep(X)

    # M-step
    gmm.MStep(X, Z)

    llh = gmm.logLikelihood(X)
    print('log-likelihood:', llh)
##############################################

y_min = -0.01
y_max = .4
fig, ax = plotting.plotGMM(true_model, y_min=y_min, y_max=y_max, label='True model')
plotting.plotGMM(gmm, fig=fig, ax=ax, color='r', label='GMM EM')
ax.plot(X, Y, '+', color='b', label='data')
ax.legend(loc='upper left')
```

## Bayesian GMM

We can also work the Bayesian inference for GMM by putting a prior over the GMM parameters. Let $\Theta$ be the set of parameters of the GMM:
$$
\Theta = \{ \boldsymbol{\mu}, \boldsymbol{\lambda}, \boldsymbol{\pi} \}
$$

The prior over the weights $\boldsymbol{\pi}$ will be a Dirichlet distribution:

$$
\DeclareMathOperator{\Dir}{Dir}
p(\boldsymbol{\pi}) = \Dir(\boldsymbol{\pi} \mid \boldsymbol{\alpha}_0)
$$

and the prior of the mean and precision of the component $k$ of the mixture will be a Normal-Gamma distribution:

$$
p(\mu_k, \lambda_k) = \Norm(\mu_k \mid m_0, (\kappa_0 \lambda_k)^{-1}) \Gam(\lambda_k \mid a_0, b_0)
$$

The joint distribution of the data, the latent variables and the parameters can be written as:

$$
\begin{align}
p(\mathbf{x}, \mathbf{z}, \Theta) &= p(\mathbf{x}, \mathbf{z} \mid \Theta)p(\Theta) \\
                      &= \Bigg[ \prod_{i=0}^{N} p(x_i \mid \boldsymbol{\mu}, \boldsymbol{\lambda}, \boldsymbol{\pi}) \Bigg] \Bigg[ \Dir(\boldsymbol{\pi} \mid \boldsymbol{\alpha}_0) \prod_{k=0}^{K} \Norm(\mu_k \mid m_0, (\kappa_0 \lambda_k)^{-1}) \Gam(\lambda_k \mid a_0, b_0) \Bigg]
\end{align}
$$

### Gibbs Sampling:

The EM algorithm cannot be applied directly to train the Bayesian GMM. Instead, we will use Gibbs Sampling (GS) to learn the posterior distribution of the parameters. Gibbs Sampling is a simple way to sample values from a complex distribution. Let's say that we want to sample values for the random variable $A$ and $B$ conditioned on $C$. Unfortunately, $p(A, B \mid C)$ is too complex to be sampled from directly. Alternatively we can sample in turn $a$ from $p(A | B = b, C)$ and $b$ from $p(B | A=a, C)$. It can be proven that if we keep sampling  long enough, the set of  $a$ and $b$  will be distributed according to $p(A, B | C)$. This is the Gibbs Sampling algorithm.

NOTE: Gibbs Sampling is not always applicable as we cannot always sample from the conditional distributions (it may be intractable).

In the following we just show the 3 conditional distributions we need for the Gibbs Sampling of the Bayesian GMM.

##### latent variable $z_i$
$$
\begin{align}
p(z_i \mid \mathbf{x}, \Theta) &= p(z_i \mid x_i, \boldsymbol{\pi}, \boldsymbol{\mu}, \boldsymbol{\lambda}) \\
p(z_i = k \mid x_i, \boldsymbol{\pi}, \boldsymbol{\mu}, \boldsymbol{\lambda}) &= \frac{\pi_k \Norm(x_i \mid \mu_k, \lambda_k)} {\sum_{j=0}^{K} \pi_j \Norm(x_i \mid \mu_j, \lambda_j)}
\end{align}
$$

###### mean and variance
We define the set of all data point $x_i$  that are assigned to the component $k$ of the mixture as follows:
$$
\mathbf{x}_{(k)} = \{ x_i : z_i = k, \forall i \in \{1,... , N \} \}
$$
and similarly for the latent variables $\mathbf{z}$:
$$
\mathbf{z}_{(k)} = \{ z_i : z_i = k, \forall i \in \{1,... , N \} \}
$$

$$
\begin{align}
p(\mu_k, \lambda_k \mid \mathbf{x}, \mathbf{z}, \Theta_{\smallsetminus \{ \mu_k, \lambda_k \} } ) &= p(\mu_k, \lambda_k \mid \mathbf{x}_{(k)}, \mathbf{z}_{(k)}, \Theta_{\smallsetminus \{ \mu_k, \lambda_k \} } ) \\
&= \Norm(\mu_k \mid m_{n,k}, (\kappa_{n,k} \lambda_k)^{-1}) \Gam(\lambda_k \mid a_{n,k}, b_{n,k})
\end{align}
$$

where:

$$
\begin{align}
m_{n,k} &= \frac{\kappa_0 m_0 + N_k \bar{x}_k} {\kappa_0 +  N_k} \\
\kappa_{n,k} &= \kappa_0 + N_k \\
a_{n,k} &= a_0 + \frac{N_k}{2} \\
b_{n,k} &= b_0 + \frac{N_k}{2} ( s + \frac{\kappa_0 (\bar{x}_k - m_0)^2}{\kappa_0 + N_k} ) \\
N_k &= \left\vert \mathbf{x}_{(k)} \right\vert \\
\bar{x}_k &= \frac{1}{N_k} \sum_{\forall x \in \mathbf{x}_{(k)}} x \\
s_n &= \frac{1}{N} \sum_{\forall x \in \mathbf{x}_{(k)}} (x_i - \bar{x})^2
\end{align}
$$

NOTE: these equations are very similar to the Bayesian Gaussian estimate. However, it remains some difference

##### weights

$$
\begin{align}
p( \boldsymbol{\pi} \mid \mathbf{x}, \mathbf{z}, \Theta_{\smallsetminus \{ \boldsymbol{\pi} \} } ) &= p( \boldsymbol{\pi} \mid \mathbf{z}) \\
&= \Dir(\boldsymbol{\pi} \mid \boldsymbol{\alpha})
\end{align}
$$
where:
$$
\alpha_{n,k} = \alpha_{0,k} + N_k \; ; \; \forall \, k = 1\dots K
$$

##### TODO
* Compare the following implementation of the Gibbs Sampling to the EM algorithm. What are their differences/similarities ?
* Run the Gibbs Sampling algorithm for 1, 2, 5, 10, 20, ... iterations. How many iterations do we need to get a reasonable estimate of the true density ?
* Change the hyper-parameters and run the Gibbs Sampling algorithm for 3 iterations. Are the initial sampled GMM very similar ? Run more iterations, does the Gibbs Sampling converges to the same solution regardless of the initialization ?

```python
# hyper-parameters [pi_1, pi_2, ...], m, kappa, a, b
# you may try to change this.
bgmm = models.BayesianGMM([1, 1, 1], 0, 1, 2, 1)

y_min = -0.01
y_max = .4
fig, ax = plotting.plotGMM(true_model, y_min=y_min, y_max=y_max, label='True model')

for i in range(5):
    # Sample the latent variables
    Z = bgmm.sampleLatentVariables(X)

    # Update the parameters
    bgmm.sampleMeansVariances(X, Z)
    bgmm.sampleWeights(Z)

    # Just for plotting, this is not part of the Gibbs Sampling algorithm.
    plotting.plotGMM(bgmm.gmm, fig=fig, ax=ax, color='b', lw=.5, label='sampled GMM')

gmm_avg = bgmm.averageGMM()
plotting.plotGMM(gmm_avg, fig=fig, ax=ax, color='r', lw=3, label='avg GMM')
ax.plot(X, Y, '+', color='b', label='data')
handles, labels = ax.get_legend_handles_labels()
labels, ids = np.unique(labels, return_index=True)
handles = [handles[i] for i in ids]
ax.legend(handles, labels, loc='upper left')
```

```python

```


# Numbers

The `numpy` array is the foundation of essentially all numerical computing in Python, so it is important to understand the array and how to use it well.

## Learning objectives

1. Attributes of an array
2. How to create vectors, matrices, tensors
3. How to index and slice arrays
4. Generating random arrays and sampling
5. Universal functions, vectorization and matrix multiplication
6. Array axes and marginal calculations
7. Broadcasting
8. Masking
9. Combining and splitting arrays
10. Vectorizing loops

```python
import numpy as np
```

## The `ndarray`: Vectors, matrices and tenosrs

dtype, shape, strides

### Vector

```python
x = np.array([1,2,3])
x
```

```python
type(x)
```

```python
x.dtype
```

```python
x.shape
```

```python
x.strides
```

### Matrix

```python
x = np.array([[1,2,3], [4,5,6]], dtype=np.int32)
x
```

```python
x.dtype
```

```python
x.shape
```

```python
x.strides
```

### Tensor

```python
x = np.arange(24).reshape((2,3,4))
```

```python
x
```

## Creating `ndarray`s

### From a file

```python
%%file numbers.txt
a,b,c # can also skip headers
1,2,3
4,5,6
```

```python
np.loadtxt('numbers.txt', dtype='int', delimiter=',',
           skiprows=1, comments='#')
```

### From Python lists or tuples

```python
np.array([
    [1,2,3],
    [4,5,6]
])
```

### From ranges

arange, linspace, logspace

```python
np.arange(1, 7).reshape((2,3))
```

```python
np.linspace(1, 10, 4)
```

```python
np.logspace(0, 4, 5, dtype='int')
```

### From a function

`fromfunciton`

```python
np.fromfunction(lambda i, j: i*3 + j + 1, (2,3))
```

```python
np.fromfunction(lambda i, j: (i-2)**2 + (j-2)**2, (5,5), dtype='int')
```

#### How to visualize `fromfunction`

```python
j = np.repeat([np.arange(5)], 5, axis=0)
i = j.T
```

```python
i
```

```python
j
```

```python
(i-2)**2 + (j-2)**2
```

#### Using element-wise functions in `fromfunction`

```python
np.fromfunction(lambda i, j: np.where(i==j,0, -1), (5,5))
```

```python
np.fromfunction(lambda i, j: np.where(i<j, 1, np.where(i==j,0, -1)), (5,5))
```

```python
np.fromfunction(lambda i, j: np.minimum(i,j), (5,5), dtype='int')
```

```python
np.fromfunction(lambda i, j: np.maximum(i,j), (5,5), dtype='int')
```

### From special constructors

zeros, ones, eye, diag

```python
np.zeros((2,3))
```

```python
np.ones((2,3))
```

```python
np.eye(3)
```

```python
np.eye(3, 4)
```

```python
np.eye(4, k=-1)
```

```python
np.diag([1,2,3,4])
```

```python
np.diag([1,2,3,4], k=1)
```

### From random variables

#### Convenience functions

rand, randn

```python
np.random.rand(2,3)
```

```python
np.random.randn(2,3)
```

#### Distributions

uniform, normal, randint, poisson, multinomial, multivariate_ normal

```python
np.random.uniform(0, 1, (2,3))
```

```python
np.random.normal(0, 1, (2,3))
```

```python
np.random.randint(0, 10, (4,5))
```

```python
np.random.poisson(10, (4,5))
```

```python
np.random.multinomial(n=5, pvals=np.ones(5)/5, size=8)
```

```python
np.random.multivariate_normal(mean=[10,20,30], cov=np.eye(3), size=4)
```

### Sampling using `choice`

Works much like the R `sample` function.

```python
x = np.random.permutation(list('ABCDEF'))
```

```python
x
```

```python
np.random.choice(x, 3)
```

```python
np.random.choice(x, 10)
```

```python
try:
    np.random.choice(x, 10, replace=False)
except ValueError as e:
    print(e)
```

## Indexing

```python
x = np.arange(20).reshape((4,5))
x
```

### Extracing a scalar

```python
x[1,1]
```

### Extracting a vector

```python
x[1]
```

### Using slices

```python
x[1,:]
```

```python
x[:,1]
```

```python
x[1:3,1:3]
```

### Using slices with strides

```python
x[::2,::2]
```

### Extrcting blocks with arbitrary row and column lists (fancy indexing)

`np.ix_`

```python
x[:, [0,3]]
```

Warning: Fancy indexing can only be used for 1 dimension at a time.

In the example below, `numpy` treats the arguments as *paired* coordinates, and returns the values at (0,0) and (2,3).

```python
x[[0,2],[0,3]]
```

Use the helper `np.ix_` to extract arbitrary blocks.

```python
x[np.ix_([0,2], [0,3])]
```

### A slice is a view, not a copy

**Warning**

```python
b = a[:]
```

makes a copy if `a` is a list but not if `a` is a numpy array

```python
a1 = list(range(3))
a2 = np.arange(3)
```

```python
b = a1[:]
b[1] = 9
a1
```

```python
b = a2[:]
b[1] = 9
a2
```

```python
x
```

```python
y = x[1:-1, 1:-1]
y
```

```python
y *= 10
```

```python
y
```

```python
x
```

Use the copy method to convert a view to a copy

```python
z = x[1:-1, 1:-1].copy()
```

```python
z
```

```python
z[:] = 0
```

```python
z
```

```python
x
```

### Boolean indexing

```python
x[x % 2 == 0]
```

```python
x [x > 3]
```

### Functions that return indexes

```python
idx = np.nonzero(x)
idx
```

```python
x[idx]
```

```python
idx = np.where(x > 3)
idx
```

```python
x[idx]
```

## Universal functions

```python
x
```

Operations

```python
x + x
```

Element-wise functions

```python
np.log1p(x)
```

```python
x.clip(10, 100)
```

Scans

```python
np.cumsum(x, axis=1)
```

Reductions

```python
np.sum(x)
```

```python
x.prod()
```

## Margins and the `axis` argument

```python
x
```

The 0th axis has 4 items, the 1st axis has 5 items.

```python
x.shape
```

```python
x.mean()
```

### Marginalizing out the 0th axis = column summaries

```python
x.mean(axis=0)
```

### Marginalizing out the 1st axis = row summaries

```python
x.mean(axis=1)
```

Note marginalizing out the last axis is a common default.

```python
x.mean(axis=-1)
```

### Marginalization works for higher dimensions in the same way

```python
x = np.random.random((2,3,4))
x
```

```python
x.shape
```

```python
x.mean(axis=0).shape
```

```python
x.mean(axis=1).shape
```

```python
x.mean(axis=2).shape
```

```python
x.mean(axis=(0,1)).shape
```

```python

x.mean(axis=(0,2)).shape
```

```python
x.mean(axis=(1,2)).shape
```

## Broadcasting

Broadcasting is what happens when `numpy` tries to perform binary operations on two arrays with different shapes. In general, shapes are *promoted* to make the arrays compatible using the following rule

- For each axis from highest to lowest
    - If both dimensions are the same, do nothing
    - If one of the dimensions is 1 or None and the other is $k$, promote to $k$
    - Otherwise print error message

```python
x = np.zeros((3,2))
x.shape
```

    (3, 2)

```python
x
```

    array([[0., 0.],
           [0., 0.],
           [0., 0.]])

Shapes are compatible

```python
y = np.ones(2)
y.shape
```

    (2,)

```python
x + y
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

Shapes are compatible

```python
y = np.ones((1,2))
y.shape
```

```python
x + y
```

Shapes are incompatible but can be made compaible by adding empty dimension

```python
y = np.ones(3)
y.shape
```

    (3,)

```python
try:
    x + y
except ValueError as e:
    print(e)
```

    operands could not be broadcast together with shapes (3,2) (3,)

```python
y[:, None].shape
```

    (3, 1)

```python
x + y[:, None]
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

Shapes are incompatible

```python
y = np.ones((2,2))
y.shape
```

```python
try:
    x + y
except ValueError as e:
    print(e)
```

### More examples of broadcasting

```python
x1 = np.arange(12)
```

```python
x1
```

```python
x1 * 10
```

```python
x2 = np.random.randint(0,10,(3,4))
```

```python
x2
```

```python
x2 * 10
```

```python
x2.shape
```

### Column-wise broadcasting

```python
mu = np.mean(x2, axis=0)
mu.shape
```

```python
x2 - mu
```

```python
(x2 - mu).mean(axis=0)
```

### Row wise broadcasting

```python
mu = np.mean(x2, axis=1)
mu.shape
```

```python
try:
    x2 - mu
except ValueError as e:
    print(e)
```

### We can add a "dummy" axis using None or `np.newaxis`

```python
mu[:, None].shape
```

```python
x2 - mu[:, None]
```

```python
x2 - mu[:, np.newaxis]
```

```python
np.mean(x2 - mu[:, None], axis=1)
```

#### Reshaping works too

```python
x2 - mu.reshape((-1,1))
```

#### Exercise in broadcasting

Creating a 12 by 12 multiplication table

```python
x = np.arange(1, 13)
x[:,None] * x[None,:]
```

Scaling to have zero mean and unit standard devation for each feature.

```python
x = np.random.normal(10, 5,(3,4))
x
```

Scaling column-wise

```python
(x - x.mean(axis=0))/x.std(axis=0)
```

Scaling row-wise

```python
(x - x.mean(axis=1)[:, None])/x.std(axis=1)[:, None]
```

## Masking

- [Ref](https://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html)

```python
import numpy.ma as ma
```

```python
x = np.arange(20).reshape(4,5)
```

```python
x
```

```python
mask = x % 2 == 0
mask
```

- Note that values that are True in the mask are not used in the array
- Values that are False are *not* masked and so remain
- So the above mask keeps only the *odd* numbers in the array `x`

```python
m = ma.masked_array(x, mask)
```

```python
m
```

```python
m.data
```

```python
m.mask
```

```python
m.sum(axis=0).data
```

```python
m.sum(axis=1).data
```

### Often used with missing value sentinels

```python
import warnings

with warnings.catch_warnings():
    warnings.simplefilter('ignore', RuntimeWarning)
    x1 = x / mask
```

```python
x1
```

```python
x1.sum()
```

```python
x2 = ma.masked_invalid(x1)
x2
```

```python
x2.data
```

```python
x2.mask
```

```python
x2.sum()
```

```python
x2.filled(0)
```

## Combining `ndarray`s

```python
x1 = np.zeros((3,4))
x2 = np.ones((3,5))
x3 = np.eye(4)
```

```python
x1
```

```python
x2
```

```python
x3
```

### Binding rows when number of columns is the same

```python
np.r_[x1, x3]
```

### Binding columns when number of rows is the same

```python
np.c_[x1, x2]
```

### You can combine more than 2 at a time

```python
np.c_[x1, x2, x1]
```

### Stacking

```python
np.vstack([x1, x3])
```

```python
np.hstack([x1, x2])
```

```python
np.dstack([x2, 2*x2, 3*x2])
```

### Generic stack with axis argument

```python
np.stack([x2, 2*x2, 3*x2], axis=0)
```

```python
np.stack([x2, 2*x2, 3*x2], axis=1)
```

```python
np.stack([x2, 2*x2, 3*x2], axis=2)
```

### Repetition and tiling

#### For a vector

```python
x = np.array([1,2,3])
```

```python
np.repeat(x, 3)
```

```python
np.tile(x, 3)
```

#### For a matrix

```python
x = np.arange(6).reshape((2,3))
x
```

```python
np.repeat(x, 3)
```

```python
np.repeat(x, 3, axis=0)
```

```python
np.repeat(x, 3, axis=1)
```

```python
np.tile(x, (3,2))
```

## Splitting `ndarray`s

```python
x = np.arange(32).reshape((4,8))
```

```python
x
```

```python
np.split(x, 4)
```

```python
np.split(x, 4, axis=1)
```

## Saving and loading arrays

```python
x = np.arange(16).reshape(4,4)
y = np.arange(20).reshape(-1,4)
```

```python
np.save('x.npy', x)
```

```python
np.load('x.npy')
```

```python
np.savez('xy.npz', x=x, y=y)
```

```python
arr = np.load('xy.npz')
```

```python
arr['x']
```

```python
arr['y']
```

```python
import os
```

```python
os.remove('x.npy')
```

```python
os.remove('xy.npz')
```

### Einstein summation notation

See https://obilaniu6266h16.wordpress.com/2016/02/04/einstein-summation-in-numpy/ for a comprehensible explanation before reading the `numpy.einsum` help documentation. Then try out several examples.

On the first try, ignore implicit outputs and broadcasting `...`.

```python
x = np.arange(1,10).reshape(3,3)
x
```

```python
np.einsum('ii', x)
```

```python
np.einsum('ji', x)
```

```python
np.einsum('ii->i', x)
```

```python
np.einsum('ij->i', x)
```

```python
np.einsum('ij->j', x)
```

```python
np.einsum('mn, np -> mp', x, x)
```

```python
np.einsum('...i,...i', x, x)
```

```python
np.einsum('j...,j...', x,x)
```

## Vectorization

### Example 1

The operators and functions (ufuncs) in Python are vectorized, and will work element-wise over all entries in an `ndarray`.

```python
xs = np.zeros(10, dtype='int')
for i in range(10):
    xs[i] = i**2
xs
```

```python
xs = np.arange(10)**2
xs
```

Using ufuncs

```python
np.sqrt(xs)
```

```python
np.log1p(xs)
```

### Example 2

Scalar product.

```python
n = 10

xs = np.random.rand(n)
ys = np.random.rand(n)

s = 0
for i in range(n):
    s += xs[i] * ys[i]
s
```

```python
np.dot(xs, ys)
```

```python
xs @ ys
```

```python
np.einsum('i,i->', xs, ys)
```

### Example 3

\begin{align}
y_0 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
y_1 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
y_2 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
\end{align}

```python
m = 3
n = 2

alpha = np.random.rand(1)
betas = np.random.rand(n,1)
xs = np.random.rand(m,n)
```

```python
alpha
```

```python
betas
```

```python
xs
```

### Using loops

```python
ys = np.zeros((m,1))
for i in range(m):
    ys[i] = alpha
    for j in range(n):
        ys[i] += betas[j] * xs[i,j]
ys
```

### Removing inner loop

```python
ys = np.zeros((m,1))
for i in range(m):
    ys[i] = alpha + xs[i,:].T @ betas
ys
```

### Removing all loops

```python
ys = alpha + xs @ betas
ys
```

### Alternative approach

The calculaiton with explicit intercepts and coefficients is common in deep learning, where $\alpha$ is called the bias ($b$) and $\beta$ are called the weights ($w$), and each equation is $y[i] = b + w[i]*x[i]$.

It is common in statisiics to use an augmented matrix in which the first column is all ones, so that all that is needed is a single matrix multiplicaiotn.

```python
X = np.c_[np.ones(m), xs]
X
```

```python
alpha
```

```python
betas
```

```python
betas_ = np.concatenate([[alpha], betas])
betas_
```

```python
ys = X @ betas_
ys
```

```python
X.shape, betas_.shape
```

```python
np.einsum('ij,jk->ik', X, betas_)
```

### Simulating diffusion

See how we use views to avoid making (slow) copies of the matrix at each step.

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
w = 100
h = 100
x = np.zeros((w+2,h+2), dtype='float')
x[(w//2-1):(w//2+2), (h//2-1):(h//2+2)] = 1

wts = np.ones(5)/5

for i in range(41):
    if i % 10 == 0:
        plt.figure()
        plt.imshow(x[1:-1, 1:-1], interpolation='nearest')

    center = x[1:-1, 1:-1]
    left = x[:-2, 1:-1]
    right = x[2:, 1:-1]
    bottom = x[1:-1, :-2]
    top = x[1:-1, 2:]
    nbrs = np.dstack([center, left, right, bottom, top])
    x = np.sum(wts * nbrs, axis=-1)
```

```python

```


This notebook requires latex_env nbextension for a correct visualization. Download and Installation's instructions available at
http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/latex_envs/README.html

# Reducing row echelon form #

\begin{definition}
two $n \times m$ matrices $A,B$ are row equivalent if there exists an invertible matrix $M$ with $B = MA$
\end{definition}

Recall the following fact:

\begin{proposition}
Proposition: Two matrices are row equivalent if and only if they represent the same linear map V \to W up to a choice of basis for W.
\end{proposition}

Any  (possibly not square)  finite matrix  $B$  can be reduced in many ways by a finite sequence
of   Elementary Row-Operations $E_1, E_2, ...,E_m$, each one invertible,  to a Reduced Row-Echelon Form (RREF)
$$U :=  E_m \cdot\cdot\cdot E_2 \cdot E_1 \cdot B $$

characterized by three properties:
1. The first nonzero element in any nonzero row is  “1” .
2. Each nonzero row's leading  “1”  comes in a column whose every other element is  “0” .
3. Each such leading  “1”  comes in a column after every preceeding row's leading zeros.

The entries which contain leading '1' so defined are called pivots.

```python
import numpy as np
import sympy

M = np.array([[1,2,3,4,5],[1,2,3,4,6],[1,0,0,2,3]])
print(M)
print(sympy.Matrix(M).rref())
```

    [[1 2 3 4 5]
     [1 2 3 4 6]
     [1 0 0 2 3]]
    (Matrix([
    [1, 0,   0, 2, 0],
    [0, 1, 3/2, 1, 0],
    [0, 0,   0, 0, 1]]), (0, 1, 4))

https://www.csun.edu/~panferov/math262/262_rref.pdf

https://www.math.purdue.edu/~shao92/documents/Algorithm%20REF.pdf

http://www.math.jhu.edu/~bernstein/math201/RREF.pdf

The reduced row echelon form is canonical, meaning it is uniquely determined for any matrix (this is not obvious, but for brevity we refer the reader to an external proof).
https://people.eecs.berkeley.edu/~wkahan/MathH110/RREF1.pdf

https://www.di-mgt.com.au/matrixtransform.html, indeed the algorithm is gauss jordan elimination (check also this link
https://gist.github.com/num3ric/1357315)
We can deduce many useful facts about a matrix once it has been transformed to its canonical form. Let A denote a general matrix and let E

denote its RREF form.

Every matrix A

is row equivalent to a unique matrix E
in row canonical (RREF) form.
    This means the form and elements of E are uniquely determined by A. So, if two matrices M and N both reduce to the same row canonical form E, then M∼N
    .
The rank of a matrix A
is equal to the number of pivots in E
.
    This is one definition of rank. We write this as rank(A) or rk(A)
    . The rank of a matrix is equal to the dimension of both the row space and the column space.
All invertible matrices reduce to the identity matrix I
.
    If the n×n matrix A is invertible, then E is always the n×n identity matrix.

## Elementary row operations: ##

Recall the elementary row operations:

https://www.khanacademy.org/math/precalculus/precalc-matrices/elementary-matrix-row-operations/a/matrix-row-operations

 1. switch any two rows,
 2. multiply any row by a nonzero constant,
 3. add a nonzero (multiple of one) row to another row.

Elementary row operation corresponds to left multiplication of an appropriate matrix (elementary Matrix). Therefore. let $M$ be an $n \times m$ matrix,
1. $R_i \leftrightarrow R_j$ := $M \cdot I$ swapped
2. $R_i \leftrightarrow cR_i$ := $M \cdot I$ with c on the ith entry
3. $R_i + R_j \to R_k$ := $M \cdot I$ with 1 (or c) on the ith and jth entries

***
The identity Matrix is, elementary, an elementary matrix.
Elementary Matrices are invertible, and the inverse is an elementary matrix.
And every product of elementary matrices is invertible.

Therefore we have the structure of a group, called the general linear group. (See Lie Groups)
***

https://www.youtube.com/watch?v=l69YjkuUym0

Is this the case?

Echelon
    (military) a formation in which units follow one another but are offset sufficiently to allow each unit a clear line of fire ahead.
    A step-like formation of troops.
    Any structure or group of structures arranged in a steplike form.

In other words, in echelon formation we arrange things so the row behind you won't shoot you.

```python
import numpy as np

def rref(matrix):
    # Check matrix dimension (useless in numpy)
   numRows = len(matrix)
   numCols = len(matrix[0])
   nonzeroRow = 0

    # Initialize to 0,0
   i,j = 0,0

   while True:
        # Loop forever, and break when exceed indexes
        # Why this choice?
      if i >= numRows or j >= numCols:
         break

     # We are working over a field, any non 0 entry can be a pivot
     # Search column wise for a non zero element

      if matrix[i][j] == 0:
         nonzeroRow = i
         while nonzeroRow < numRows and matrix[nonzeroRow][j] == 0:
            nonzeroRow += 1
      # if none is found, check the next column
      if nonzeroRow == numRows:
         j += 1
        # Go back at the beginning of the while loop
         continue

    # swap the two rows
      temp = matrix[i]
      matrix[i] = matrix[nonzeroRow]
      matrix[nonzeroRow] = temp
   # save the pivot value
      pivot = matrix[i][j]
    # Once we have found a pivot, we simply need to eliminate the remaining entries in the column.
    #We know this won’t affect any previously inspected columns,
    #because by the inductive hypothesis any entries which are to the left of our pivot are zero.???
    # normalize, so the first is one is what is going on
      matrix[i] = [x / pivot for x in matrix[i]]

      for otherRow in range(0, numRows):
         if otherRow == i:
             continue
         if matrix[otherRow][j] != 0:
            matrix[otherRow] = [y - matrix[otherRow][j]*x
             for (x,y) in zip(matrix[i], matrix[otherRow])]

      i += 1; j+= 1

   return matrix

# Note this works with python lists
# To Rewrite it in numpy for God's sake
bd1 = np.array([[-1,-1,-1,-1,0,0,0,0], [1,0,0,0,-1,-1,0,0],
     [0,1,0,0,1,0,-1,-1], [0,0,1,0,0,1,1,0], [0,0,0,1,0,0,0,1]])
test = np.array([[-2,-2,4,-2],[-2,1,10,7],[-4,-4,-8,4],[4,-1,14,6]])
M = np.array([[1,2,3,4,5],[1,2,3,4,6],[1,0,0,2,3]])
toReduce = test.T.tolist()
print(np.array(rref(toReduce)))
print(test)
```

    [[-9.          0.          0.          0.        ]
     [ 0.45454545  1.          0.          0.        ]
     [ 0.54545455  0.          1.          0.        ]
     [ 0.27272727  0.          0.          1.        ]]
    [[-2 -2  4 -2]
     [-2  1 10  7]
     [-4 -4 -8  4]
     [ 4 -1 14  6]]

```python

```


# Fourier smoothing

Notebook for AMath 570, Autumn Quarter 2015, R.J. LeVeque

See: http://faculty.washington.edu/rjl/classes/am570a2015/codes.html

### Aliasing in the DFT

For a periodic function $u(x)$ we can write
$$
u(x) = \frac{1}{2\pi}\sum_{k=-\infty}^\infty e^{ikx} \hat u_k, \quad\quad
\text{where}\quad \hat u_k = \int_0^{2\pi} e^{-ikx}u(x)\,dx.
$$
The Discrete Fourier Transform can be viewed as an approximation to this using $N$ points $x_j=jh$ with $h=2\pi/N$,
$$
v_j = u(x_j) = \frac{1}{2\pi} \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\, \hat v_k, \quad\quad
\text{where}\quad \hat v_k = \frac 1 N \sum_{j=1}^N e^{-ikx_j}\,v_j.
$$

Similar to Theorem 2 in SMM Chapter 4, there is an aliasing theorem that relates these:

**Theorem:**
$\quad\displaystyle \hat v_k = \sum_{\ell=-\infty}^\infty \hat u_{k + \ell N}.$

**Proof:**

$$
\begin{align}
v_j = u(x_j) &= \frac{1}{2\pi}\sum_{k=-\infty}^\infty e^{ikx} \hat u_k \\
&= \frac{1}{2\pi} \left( \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\, \hat u_k + \sum_{k=N/2+1}^{3N/2} e^{ikx_j}\, \hat u_k + \cdots \right) \\
\end{align}
$$
Now note that
$$
\sum_{k=N/2+1}^{3N/2} e^{ikx_j}\, \hat u_k = \sum_{k=-N/2+1}^{N/2} e^{i(k+N)x_j}\, \hat u_{k+N}
= \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\, \hat u_{k+N}
$$
since $e^{ikNx_j} = 1$ for all $j$.  Similarly for all other terms in the sum above, and so
$$
v_j = \frac{1}{2\pi} \sum_{k=-N/2+1}^{N/2} e^{ikx_j}\,\left( \sum_{\ell=-\infty}^\infty \hat u_{k + \ell N} \right)
$$

We thus see how the Fourier series that *interpolates* at $N$ points is related to the full Fourier series.

### Trucation / projection

Another approach to getting a finite Fourier series would be to take the full series and *truncate* it to $M$ terms, defining
$$
u_M(x) = \frac{1}{2\pi} \sum_{k=-M/2+1}^{M/2} e^{ikx}\, \hat u_k, \quad\quad
\text{where}\quad \hat u_k = \int_0^{2\pi} e^{-ikx}u(x)\,dx.
$$
How does this relate to $v(x)$?  If we take $M=N$, then the function $u_N(v)$ has the same number of terms as $v(x)$, but different coefficients.  It no longer *interpolates* at equally spaced points.  Instead it gives the *best approximation* in the 2-norm (i.e. the *least squares* approximation.

If we let $F_M$ be the space of all Fourier sums with $M$ terms, then $u_M$ solves the problem
$$
\min_{p\in F_M} \|p - u\|_2, \quad\quad \text{where} \quad
\|w\|_2 = \left(\int_0^{2\pi} |w(x)|^2\, dx \right)^{1/2}.
$$
Recall that for complex valued functions, $|w(x)|^2 = w(x)\bar w(x)$ and also that the inner product of two functions can be defined by
$$
\langle w_1, w_2\rangle = \int_0^{2\pi} w_1(x) \bar w_2(x)\, dx
$$
We say two functions are *orthogonal* if their inner product is 0.  Note that $e^{ik_1x}$ and $e^{ik_2 x}$ are orthogonal for any two integers $k_1 \neq k_2$, since their inner product is
$$
\int_0^{2\pi} e^{ik_1x}e^{-ik_2x}\, dx = \int_0^{2\pi} e^{i(k_1-k_2)x}\,dx
$$
If $k_1 = k_2$ then the integral is $2\pi$, otherwise it is 0.

In general the solution to a least squares problem has the property that the residual $p(x) - u(x)$ must be orthogonal to the space $F_M$.  This can be seen by noting that the square of the error is $\langle p-u, p-u\rangle$, writing $p$ as a Fourier sum in terms of unknown coefficients $\hat p_k$, differentiating with respect to each $\hat p_k$ and setting these derivatives to zero.  You should find the requirement that $\langle p-u, e^{ikx}\rangle = 0$ for each wave number occuring in the sum.  Now writing out $p$ and $u$ as Fourier sums in this expression and using the orthogonality of the Fourier modes, this condition reduces to $\hat p_k = \hat u_k$ for the wave numbers in the finite sum $p$.

### Approximating $u_M$ by a finite sum

Truncating the full Fourier series requires being able to compute the integrals that define the $\hat u_k$.  If this can't be done, we could use a finite sum.  If we use $M$ function values then we are back to the discrete Fourier transform $v(x)$ defined above, which we might call $v_M(x)$.  But another approach is to compute $v_N(x)$ for some $N > M$ and then truncate this to $M$ terms.  If the original function is smooth enough that the coefficents decay quickly then $v_N$ will be very close to $u$.  This is often done as a way of smoothing noisy data.

## Exploration:

Below we start with a Gaussian function and add some high frequencies as a way of generating a function to experiment with.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

The next line makes it possible to zoom in on plots, only works with matplotlib 1.4 or later.

```python
# %matplotlib notebook   # uncomment to try this
```

```python
xfine = linspace(0, 2*pi, 10000)
```

```python
omega = [16]   # list of high frequencies to add (integers so noise is periodic)
m = len(omega)
a = m*[0.1]   # amplitudes

def u(x):
    u = exp(-(x-pi)**2)     # Gaussian
    for k in range(m):
        u = u + a[k] * cos(omega[k]*x)
    return u

figure(figsize=(10,5))
plot(xfine,u(xfine))
print 'wave numbers:', omega
print 'amplitudes:', a
```

```python
from scipy import fft,ifft
N = 30;
h = 2*pi/N
xj = linspace(0, 2*pi-h, N)  # note: fft assumes points are 0, h, 2h, ..., (N-1)h   (not h, 2h, ..., Nh)

M = N  # M = N ==> interpolation,  M < N gives smoothing  N and M shold be even

ik = 1j*hstack((range(0,N/2+1), range(-N/2+1,0)));   # i * wave number vector (matlab ordering)

vj = u(xj)
vj_hat = fft(vj) * h

def v(x):
    v = vj_hat[0] # constant term
    v += vj_hat[N/2] * cos(x*N/2) # N/2 term needs special handling: see SMM (3.4)

    for k in range(1, M/2):
        v += exp(1j*k*x)*vj_hat[k] + exp(-1j*k*x)*vj_hat[N-k]

        # alternatively, for real vj can instead write this as:
        # v += 2*real(vj_hat[k])*cos(k*x) - 2*imag(vj_hat[k])*sin(k*x)

    v = v / (2.*pi)
    v = real(v)  # should be real already up to rounding errors!
    return v

figure(figsize=(12,5))
subplot(1,2,1)
plot(xfine,u(xfine),'b')
plot(xj, u(xj), 'bo')
plot(xj, v(xj), 'r.')
plot(xfine,v(xfine),'r')
xlim(0, 2*pi)
title('Functions u(x) and v_M(x)')

subplot(1,2,2)
plot(abs(vj_hat[:N/2+1]))
title('Fourier transform')

```

## Things to try:

Try changing the cells above and re-executing.  Try to explain what you see in each case.

 - `omega = [16]` and $N = 34$ or $N = 30$ with $M=N$.  Note that in the latter case wave number 16 is aliased to 14.
 - `omega = [16]` and $N = 10$.  Now what aliasing do you see?
 - `omega = [16]` and $N = 34$ and $M = 10$.  Now there is no aliasing of wave number 16 and the smoothed function $v_M(x)$ is very close to the Gaussian.
 - `omega = [16, 25]` and $N = 64$ and $M = 12$.
 - `omega = [16, 25]` and $N = 30$ and $M = 12$.
 - `omega = [16, 25]` and $N = 30$ and $M = 30$.
 - `omega = [16, 25]` and $N = 12$ and $M = 12$.

```python

```


# Algebra

## Real Numbers

Commutative
```
a + b = b + a
```

Associative
```
a + (b + c) = (a + b) + c
```

Distributive
```
(a + b)c = ac + bc
```

|Operation|Commutative|Associative|Distributive|
|-|-|-|-|
|Addition|✓|✓||
|Multiplication|✓|✓||

----

## Law of Exponents

$\displaystyle \begin{equation} \label{eq1}
\begin{split}
a^0 & = 1 \\
(a^m)(a^n) & = a^{m+n} \\
(ab)^m & = a^mb^m \\
(a^m)^n & = a^{mn} \\
\frac{a^m}{a^n} & = a^{m-n} \\
a^m & = \frac{1}{a^{-m}} \\
a^{-m} & = \frac{1}{a^{m}} \\
a^b & = e^{ln(a)b} \\
\end{split}
\end{equation}$

----

## Formulas
https://byjus.com/algebra-formulas

$\displaystyle \begin{equation} \label{eq1}
\begin{split}
a^2 - b^2 & = (a+b)(a-b) & \\
a^2 + 2ab + b^2 & = (a+b)^2 & \\
a^2 + b^2 & = (a-b)^2\;+ && 2ab \\
(a + b + c)^2 & = & a^2\;+\;b^2\;+\;c^2\;+\;& 2ab\;+\;& 2ac\;+\;& 2bc \\
(a\;–\;b\;–\;c)^2 & = & a^2\;+\;b^2\;+\;c^2\;–\;& 2ab\;–\;& 2ac\;+\;& 2bc \\
\end{split}
\end{equation}$

$\displaystyle \begin{equation} \label{eq2}
\begin{split}
(a + b)^3 & = & a^3\;+\;& 3a^2b\;+\;& 3ab^2\;+\;& b^3 \\
 & = 3ab(a + b) & a^3\;+\;&&& b^3 \\
(a – b)^3 & = & a^3 \;–\;& 3a^2b\;+\;& 3ab^2\;–\;& b^3 \\
a^3 – b^3 & = (a – b)(a^2 + ab + b^2) & \\
a^3 + b^3 & = (a + b)(a^2 – ab + b^2) & \\
(a + b)^3 & = & a^3\;+\;& 3a^2b\;+\;& 3ab^2\;+\;& b^3 \\
(a – b)^3 & = & a^3\;–\;& 3a^2b\;+\;& 3ab^2\;-\;& b^3 \\
\end{split}
\end{equation}$

$\displaystyle \begin{equation} \label{eq3}
\begin{split}
(a + b)^4 & = & a^4\;+\;& 4a^3b\;+\;& 6a^2b^2\;+\;& 4ab^3\;+\;& b^4 \\
(a – b)^4 & = & a^4\;–\;& 4a^3b\;+\;& 6a^2b^2\;–\;& 4ab^3\;+\;& b^4 \\
a^4 – b^4 & = (a – b)(a + b)(a^2 + b^2) & \\
\end{split}
\end{equation}$

$\displaystyle \begin{equation} \label{eq4}
\begin{split}
a5 – b5 = (a – b)(a4 + a3b + a2b2 + ab3 + b4) \\
\end{split}
\end{equation}$

```python

```