# Image reconstruction in X-ray tomography

**Authors**:

* Alexandre Zouaoui (alexandre.zouaoui@ens-paris-saclay.fr)

## X-ray tomography

We are interested in the following inverse problem:

\begin{align}
y = H \overline{x} + w
\end{align}

where $y \in \mathbb{R}^M$ are the measurements from a set of projections at different angles of a X-ray tomography, $\overline{x} \in \mathbb{R}^N$  is the sought absorption image and $w \in \mathbb{R}^M$ is the measurement noise assumed to be i.i.d. Gaussian with variance $\sigma^2$. $H$ is a sparse tomography matrix that encodes the geometry of the measurements.

In our case $H$ models parallel projections of a 2-D object $\overline{x}$. Here the tomography measures are acquired at fixed and regularly sampled rotational positions between the sample and the detector so that $H_{mn}$ models the intersection length between the $m$-th light-ray and the $n$-th pixel. We denote by $N_\theta$ the number of different angular positions of the detector and by $L$ the linear size of the detector, hence the number of measurements are $M = L \times N_\theta$.

We study reconstruction approaches that do not require the linear system to be sufficiently determined for good results (i.e. $N_\theta \sim L$). We must then overcome the under-determinacy of the problem and make it robust to the presence of noise in the measurements.

### Imports

```python
import numpy as np
from scipy.io import loadmat
from scipy import sparse
from scipy import linalg
import matplotlib.pyplot as plt
import seaborn as sns
import os
import time
from itertools import product
```

### Style

```python
sns.set_style("darkgrid")
```

### Paths

```python
CWD = os.getcwd()
DATA_DIR = os.path.join(CWD, "data")
```

### Question 1

Download the projection matrix $H$ and the image $\overline{x}$ available on the website.

---

```python
# Load files
x = loadmat(os.path.join(DATA_DIR, "x.mat"))["x"]
H = loadmat(os.path.join(DATA_DIR, "H.mat"))["H"]

# Squeeze x
x = x.squeeze()

# Print type and shape information
print(f"Type of x: {type(x)}")
print(f"Shape of x: {x.shape}")
print(f"Type of H: {type(H)}")
print(f"Shape of H: {H.shape}")

# Global variables setup
N = 90 * 90
M = 90 * 180

# Sanity check
assert x.shape[0] == N
assert H.shape == (M, N)
```

    Type of x: <class 'numpy.ndarray'>
    Shape of x: (8100,)
    Type of H: <class 'scipy.sparse.csc.csc_matrix'>
    Shape of H: (16200, 8100)

### Question 2

Construct y, according to the linear system defined above, using $\sigma = 1$.

---

```python
# Set a random seed for reproducibility
np.random.seed(0)

# Noise setup
σ = 1
noise = σ ** np.random.randn(M)

# Construct y
y = H.dot(x) + noise

# Sanity check
assert noise.shape[0] == M
assert y.shape[0] == M
```

### Question 3

Here $N = 90 \times 90$ pixels and $M = 90 \times 180$ measurements. Display a 2D version of $x$ and a 2D version of $y$, also known as sinogram. Bare in mind that $x$ and $y$ were constructed by concatenating the flattened the columns values.

---

```python
# Display 2D signals
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Display a 2D version of x
axes[0].imshow(x.reshape((90, 90), order="F"))
axes[0].set_axis_off()
axes[0].set_title("Original absorption image");
# Display a 2D version of y
axes[1].imshow(y.reshape((90, 180), order="F"))
axes[1].set_axis_off()
axes[1].set_title("Measurements image");
```

## Optimization problem

An efficient strategy to address the reconstruction problem is to define $x$ as a minimizer of an appropriate cost function $f$. More specifically, we focus on the regularized least-squares criterion:

$$\forall x \in \mathbb{R}^N, \quad f(x) = \frac{1}{2} \| Hx - y \|^2 + \lambda r(x)$$

where $r$ is a regularization function incorporating a priori assumptions to guarantee the robustness of the solution with respect to noise. In order to promote images formed by smooth regions separated by sharp edges, we use:

$$\forall x \in \mathbb{R}^N, \quad r(x) = \sum_{i=1}^{2N} \phi([Gx]^{(i)})$$

where $G \in \mathbb{R}^{2N \times N}$ is a sparse matrix such that $Gx \in \mathbb{R}^{2N}$ is the concatenation of the horizontal and vertical gradients of the image, and $\phi$ is a potential function defined as:

$$\forall u \in \mathbb{R}, \quad \phi(u) = \sqrt{1 + u^2 / \delta^2}$$
with some parameter $\delta > 0$ to guarantee the differentiability of $r$.

In the following, we will set $(\lambda, \delta) = (0.13, 0.02)$.

```python
# Setup
λ = 0.13
δ = 0.02
```

### Question 1

Download the gradient operator $G$

---

```python
# Load gradient operator
G = loadmat(os.path.join(DATA_DIR, "G.mat"))["G"]

# Print type and shape information
print(f"Type of x: {type(G)}")
print(f"Shape of x: {G.shape}")

# Sanity check
assert G.shape == (2 * N, N)
```

    Type of x: <class 'scipy.sparse.csc.csc_matrix'>
    Shape of x: (16200, 8100)

### Question 2

* Give the expression of the gradient $\nabla f$ at some point $x \in \mathbb{R}^N$.

* Create a function which gives as an output the gradient of $f$ at some input vector $x$.

---

* We have:
$$\forall x \in \mathbb{R}^N, \quad \nabla f(x) = H^T(Hx - y) + \lambda \nabla r(x)$$

* Moreover:

\begin{align*}
\forall x \in \mathbb{R}^N, \, \forall k \in \{1, \ldots, N\}, \quad \frac{d r(x)}{d x_k}
& = \sum_{i=1}^{2N} \frac{d (\sum_{j=1}^N G_{i, j} x_j)}{d x_k} \phi'([Gx]_i) \\
& = \sum_{i=1}^{2N} G_{i, k} \phi'([Gx]_i) \\
& = \sum_{i=1}^{2N} G_{k, i}^T \phi'([Gx]_i) \\
& = \big[ G^T \big( \phi'([Gx]_i)\big)_{i \in \{1, \ldots, 2N\}} \big]_k
\end{align*}

* Therefore:

$$\forall x \in \mathbb{R}^N, \quad \nabla r(x) = G^T \big( \phi'([Gx]_i)\big)_{i \in \{1, \ldots, 2N\}}$$

* Where:

$$\forall u \in \mathbb{R}, \quad \phi'(u) = \frac{u}{\delta^2 \sqrt{1 + u^2 / \delta^2}}$$

* Finally:

$$\boxed{\forall x \in \mathbb{R}^N, \quad \nabla f(x) = H^T (Hx - y) + \lambda G^T \big( \phi'([Gx]_i)\big)_{i \in \{1, \ldots, 2N\}}}$$

```python
def grad_f(x, λ=λ, δ=δ):
    """
    Compute the gradient of f (defined above)
    at some input x

    Parameters
    -----------
    - x : numpy.array
        Input vector

    - λ : float (optional)
        Relaxation parameter
        Default: 0.13

    - δ : float (optional)
        Differentiability parameter
        Default: 0.02

    Returns
    -----------
    - g : numpy.array
        Gradient output vector
    """

    # Compute the ϕ' part
    ϕ_grad = G.dot(x) / (δ**2 * np.sqrt(1 + G.dot(x)**2 / δ**2))

    # Compute the gradient
    grad = H.T.dot(H.dot(x) - y) + λ * G.T.dot(ϕ_grad)
    return grad
```

### Question 3

Show that a Lipschitz constant of $\nabla f$ is

$$L = \|H\|^2 + \frac{\lambda}{\delta^2} \|G\|^2$$

Calculate it for the $(\lambda, \delta)$ values given above. In Python we can use ``scipy.sparse.linalg.svds`` that outputs the singular values of a sparse matrix, the norm of the matrix being the maximal singular value.

---

First of all, we have:

$$\forall x \in \mathbb{R}^N, \quad \nabla^2 f (x) = H^T H + \lambda G^T G \big( \phi''([Gx]_i)\big)_{i \in \{1, \ldots, 2N\}}$$

In addition:

$$\forall u \in \mathbb{R}, \quad \frac{d^2 \phi(u)}{du^2} = \frac{1 - \frac{u^2}{\delta^2(1 + \frac{u^2}{\delta^2})}}{\delta^2 \sqrt{1 + \frac{u^2}{\delta^2}}}$$

Therefore:

$$\forall u \in \mathbb{R}, \quad \frac{d^2 \phi(u)}{du^2} \leq \frac{1}{\delta^2}$$

As a result:

$$\forall u, v \in \mathbb{R}, \quad |\phi'(u) - \phi'(v)| \leq \frac{1}{\delta^2} |u - v |$$

Combining the previous results:

$$\boxed{L = \|H\|^2 + \frac{\lambda}{\delta^2} \|G\|^2}$$

Where $\|H\|^2$ (respectively $\|G\|^2$) denotes the squared largest singular value of $H$ (respectively $G$).

```python
# Compute norms
norm_H = np.max(sparse.linalg.svds(H, return_singular_vectors=False))
norm_G = np.max(sparse.linalg.svds(G, return_singular_vectors=False))
# Compute Lipschitz constant
L = (norm_H ** 2) + λ / (δ**2) * (norm_G**2)
```

## Optimization algorithms

### Gradient descent algorithm

#### Question 1

Create $x_0 \in \mathbb{R}^N$ a vector with all entries equal to 0. This will be our initialization for all tested algorithms.

---

```python
# Create xO as the null vector
x0 = np.zeros_like(x)
```

#### Question 2

Implement a gradient descent algorithm to minimize $f$

---

```python
# Define ϕ and f
ϕ = lambda u : np.sqrt(1 + u**2 / δ ** 2)
f = lambda x : 1 / 2 * np.sum((H.dot(x) - y)**2) + λ * np.sum(ϕ(G.dot(x)))
```

```python
class Solver:
    """
    Generic Solver class serving as a parent class
    to GradientDescent, MM quadratic, 3MG, Block Coordinate MMQ

    Implements the display method that prints information
    and display both the reconstruction and the iterations evolution
    """
    def __init__(self):
        pass

    def solve(self, x0=x0, tol=1e-2, λ=λ, δ=δ):
        pass

    def display(self):
        # Display reconstruction
        fig, axes = plt.subplots(1, 2, figsize=(14, 5))
        axes[0].imshow(x.reshape((90, 90), order="F"))
        axes[0].set_axis_off()
        axes[0].set_title("Original absorption image");
        axes[1].set_axis_off()
        axes[1].imshow(self.x_hat.reshape((90, 90), order="F"))
        axes[1].set_title("Reconstruction image")

        # Display iterates evolution
        plt.figure(figsize=(12, 5))
        plt.title(f"Value of $f$ along the {self.name} iterations")
        plt.xlabel("Iterations time (in seconds)")
        plt.ylabel("Value of $f$")
        plt.plot(self.iterations_time, self.f_values,
                 linestyle="--", marker="o", linewidth=3,
                 label=f"{self.short} iterates")
        plt.yscale("log")
        plt.legend(frameon=True);

    def __repr__(self):
        if self.counter is None:
            raise ValueError("Cannot call print before using ``solve``")
        return f"{self.name}: {self.counter} steps in {round(self.total_time, 2)} seconds"

    def get_info(self):
        """
        Returns
        ----------
        - x_hat : numpy.array
            Approximation result
        - counter : int
            Number of iterations
        - iterations_time : list of float
            List of computation time at regular intervals
        - f_values : list of float
            objectif function values at regular intervals
        - total_time : float
            Total computational time
        """
        return self.x_hat, self.counter, self.iterations_time, self.f_values, self.total_time
```

```python
class GradientDescent(Solver):

    def __init__(self):
        self.name = "Gradient Descent"
        self.short = "GD"

    def solve(self, x0=x0, tol=1e-2):
        # Setup
        self.x0 = x0
        self.tol = tol
        self.counter = 0
        self.f_values = list()
        self.iterations_time = list()

        # Initialization
        t0 = time.time()
        x_curr = np.copy(self.x0)

        # Main loop
        while np.linalg.norm(grad_f(x_curr)) > np.sqrt(N) * self.tol:
            # First order approximation update
            x_curr = x_curr - 1 / L * grad_f(x_curr)
            # Save iterates
            if self.counter % 100 == 0:
                self.f_values.append(f(x_curr))
                self.iterations_time.append(time.time() - t0)
            self.counter += 1

        # Compute total time in seconds
        self.total_time = time.time() - t0
        # Store solution
        self.x_hat = x_curr
```

```python
%%time
solver = GradientDescent()
solver.solve(x0=x0, tol=1e-2)
print(solver)
solver.display()
```

### Majorization Minimization (MM) quadratic algorithm

#### Question 1

Construct, for all $x \in \mathbb{R}^N$, a quadratic majorant function of $f$ at $x$.

Create a function which gives, as an output, the curvature $A(x)$ of the majorant function at an input vector $x$.

Hint: In Python, we can use ``scipy.sparse.diags(d[:, 0]).tocsc()`` to create a sparse matrix from a diagonal vector $d \in \mathbb{R}^{n \times 1}$ using the compressed sparse column (csc) format.

---

The function $r: x \in \mathbb{R}^N \mapsto r(x)$ is a separable function whose entries $r_i$ are even differentiable functions. As a result, we derive a majorant of $r_i$ using $w: u \in \mathbb{R} \mapsto \frac{r'(|u|)}{u} = \frac{1}{\delta^2 \sqrt{1 + u^2 / \delta^2}}$.

```python
def curvature_A(x, λ=λ, δ=δ):
    """
    Define the curvature of the majorant function
    at input vector x

    Parameters
    ------------
    - x : numpy.array
        Current iterate

    Returns
    -----------
    A function that takes an input vector $v$
    and outputs $Av$
    """

    # Define a diagonal sparse matrix
    d = 1 / (δ**2 * np.sqrt(1 + G.dot(x) ** 2 / δ ** 2))

    D_G = sparse.diags(d).tocsc()

    # Combine intermediate results using sparse operations
    return lambda v: H.T.dot(H.dot(v)) + λ * G.T.dot(D_G.dot(G.dot(v)))
```

#### Question 2

Deduce a MM quadratic algorithm to minimize $f$. Implement it.

Hint: To invert the majorant matrix at each iteration, use ``bicg`` from ``scipy.sparse.linalg``.

---

Since $f$ is a differentiable function from $\mathbb{R}^N$ to $\mathbb{R}$, we can derive the Majoration Minimization algorithm updates:

$$(\forall n \in \mathbb{N}) \quad x_{n+1} = x_n - \theta_n A(x_n)^{-1} \nabla f(x_n)$$

where for all $n \in \mathbb{N}, \; \theta_n \in \; ]0, 2[$. In the following, we take for all $n \in \mathbb{N}, \; \theta_n = 1$.

```python
class MMQuadratic(Solver):

    def __init__(self):
        self.name = "MM quadratic"
        self.short = "MMQ"

    def solve(self, x0=x0, tol=1e-2):
        # Setup
        self.x0 = x0
        self.tol = tol
        self.counter = 0
        self.f_values = list()
        self.iterations_time = list()

        # Initialization
        t0 = time.time()
        x_curr = np.copy(self.x0)

        # Main loop
        while np.linalg.norm(grad_f(x_curr)) > np.sqrt(N) * self.tol:
            # Instanciate Linear Operator
            A = sparse.linalg.LinearOperator(shape=(N, N),
                                             matvec=curvature_A(x_curr),
                                             rmatvec=curvature_A(x_curr))
            # Solve Linear System
            x_curr = x_curr - sparse.linalg.bicg(A=A, b=grad_f(x_curr))[0]
            # Save iterates
            if self.counter % 10 == 0:
                self.f_values.append(f(x_curr))
                self.iterations_time.append(time.time() - t0)
            self.counter += 1

        # Compute total time in seconds
        self.total_time = time.time() - t0
        # Store solution
        self.x_hat = x_curr
```

```python
%%time
solver = MMQuadratic()
solver.solve(x0=x0, tol=1e-2)
print(solver)
solver.display()
```

### Majoration Minimization Memory Gradient (3MG) algorithm

In this section we use a subspace strategy acceleration. We focus on the 3MG approach which consists in defining the iterate $x_{k+1}$ as the minimizer of the quadratic majorant function at $x_k$ within a subspace spanned by the following directions:

$$\forall k \in \mathbb{N}, \quad D_k = \big[ - \nabla f(x_k) \; | \; x_k - x_{k-1} \big]$$

Using the convention $D_0 = - \nabla f (x_0)$. An iterate of 3MG reads:

$$\forall k \in \mathbb{N}, \quad x_{k+1} = x_k + D_k u_k$$

with

$$\forall k \in \mathbb{N}, \quad u_k = - \big( D_k^T A(x_k) D_k \big)^{\dagger} \big(D_k^T \nabla f(x_k) \big)$$

where $A(x_k) \in \mathbb{R}^{N \times N}$ is the curvature of the majorant matrix at $x_k$ and $\dagger$ denotes the pseudo-inverse operation.

#### Question 1

Implement the 3MG algorithm using ``scipy.linalg.pinv`` and be mindful of the matrix operations given the size of the matrices.

---

```python
def compute_directions(x_curr, x_previous=None, first_iterate=False, λ=λ, δ=δ):
    """
    Compute the set of directions that span the acceleration subspace

    Parameters
    ---------
    - x_curr : numpy.array
        Current iterate

    - x_previous : numpy.array (optional)
        Previous iterate
        Default: None.

    - first_iterate : boolean (optional)
        Whether we are running the first iteration
        as it modifies the directions computation
        Default: False.
    """
    if first_iterate:
        return - grad_f(x_curr, λ=λ, δ=δ).reshape(-1, 1)
    else:
        return np.hstack((-grad_f(x_curr, λ=λ, δ=δ).reshape(-1, 1), (x_curr - x_previous).reshape(-1, 1)))
```

```python
def compute_u(x_curr, D_curr, first_iterate=False, λ=λ, δ=δ):
    """
    Compute current directions minimizer

    Parameters
    ---------------
    - x_curr : numpy.array
        Current iterate

    - D_curr : numpy.ndarray
        Current directions

    - first_iterate : boolean (optional)
        Whether this is the first iterate
        Default: False.

    """

    # Intermediate computations
    tmp1 = H.dot(D_curr)
    tmp2 = G.dot(D_curr)
    # Compute Diag(Gx)
    d = 1 / (δ**2 * np.sqrt(1 + G.dot(x_curr) ** 2 / δ ** 2))
    D_G = sparse.diags(d).tocsc()
    # Matrix to be inverted
    tmp3 = tmp1.T.dot(tmp1) + λ + tmp2.T.dot(D_G.dot(tmp2))
    # Compute u using Moore Penrose pseudo inverse
    if first_iterate:
        return - linalg.pinv(tmp3).dot(D_curr.T.dot(grad_f(x_curr, λ=λ, δ=δ).reshape(-1, 1)))
        #return - linalg.inv(tmp3).dot(D_curr.T.dot(grad_f(x_curr).reshape(-1, 1)))
    else:
        return - linalg.pinv(tmp3).dot(D_curr.T.dot(grad_f(x_curr, λ=λ, δ=δ)))
        #return - linalg.inv(tmp3).dot(D_curr.T.dot(grad_f(x_curr)))
```

```python
class MMMG(Solver):

    def __init__(self):
        self.name = "MM Memory Gradient"
        self.short = "3MG"

    def solve(self, x0=x0, tol=1e-2):
        # Setup
        self.x0 = x0
        self.tol = tol
        self.counter = 0
        self.f_values = list()
        self.iterations_time = list()

        # Initialization
        t0 = time.time()
        x_curr = np.copy(self.x0)

        # First iterate
        D0 = compute_directions(x_curr, first_iterate=True)
        u0 = compute_u(x_curr, D0, first_iterate=True)
        x_next = x_curr + D0.dot(u0).squeeze()
        self.counter += 1

        # Main loop
        while np.linalg.norm(grad_f(x_next)) > np.sqrt(N) * self.tol:
            x_previous = x_curr
            x_curr = x_next
            # Compute directions
            D_curr = compute_directions(x_curr, x_previous)
            # Compute minimizer
            u_curr = compute_u(x_curr, D_curr)
            # Next iterate using subspace strategy
            x_next = x_curr + D_curr.dot(u_curr).squeeze()

            # Store
            if self.counter % 50 == 1: # first step outside loop
                self.f_values.append(f(x_next))
                self.iterations_time.append(time.time() - t0)
            self.counter += 1

        # Compute total time in seconds
        self.total_time = time.time() - t0
        # Store solution
        self.x_hat = x_curr
```

```python
%%time
solver = MMMG()
solver.solve(x0=x0, tol=1e-2)
print(solver)
solver.display()
```

### Block-coordinate MM quadratic algorithm

Another acceleration strategy consists in applying a block alternation technique. The vector $x$ is divided into $J \geq 1$ blocks, with size $1 \leq N_j \leq N$. At each iteration $k \in \mathbb{N}$, a block index $j \subset \{1, \ldots, J\}$ is chosen, and the corresponding components of $x$, denoted $x^{(j)}$ are updated according to a MM quadratic rule. Here we will assume that the blocks are selected in a cyclic manner, that is:

$$\forall k \in \mathbb{N}, \quad j = k \text{mod} \; J + 1$$

For a given block index $j$, the corresponding pixel indexes are updated in the image:

$$n \in \mathbb{J}_j = \{(j - 1) N_j + 1, \ldots, j N_j\}$$

#### Question 1.

Create a function which gives, as an output, matrix $A_j(x) \in \mathbb{R}^{N_j \times N_j}$ containing only the lines and rows of $A(x)$ with indexes $\mathbb{J}_j$.

---

```python
# Get A block
def A_block(x, j, Nj, δ=δ, λ=λ):
    """
    Compute the matrix $Aj(x) \in \mathbb{R}^{N_j \times N_j}$
    containing only the lines and rows of $A(x)$ with indexes $\mathbb{J}_j$

    Parameters
    ------------
    - x : numpy.array
        Vector x

    - j : int
        Index to select block coordinates

    - Nj : int
        Size of submatrix
    """
    # Select relevant block in H and G
    sub_H = H[:, (j - 1) * Nj : j * Nj]
    sub_G = G[:, (j - 1) * Nj : j * Nj]

    # Compute Diag(Gx)
    d = 1 / (δ**2 * np.sqrt(1 + G.dot(x) ** 2 / δ ** 2))
    D_G = sparse.diags(d).tocsc()

    return sub_H.T.dot(sub_H) + λ * sub_G.T.dot(D_G.dot(sub_G))
```

##### Question 2.

Deduce an implementation of a block coordinate MM quadratic algorithm for minimizing $f$.

Test if for $N_j = N / K$ with $K \in \{1, 2, 3, 5, 6, 9\}$.

---

```python
class BCMMQ(Solver):

    def __init__(self, K):
        self.name = "Block Coordinate MMQ"
        self.short = "BCMMQ"
        self.K = K

    def solve(self, x0=x0, tol=1e-2):
        # Setup
        self.x0 = x0
        self.tol = tol
        self.counter = 0
        self.f_values = list()
        self.iterations_time = list()

        # Initialization
        t0 = time.time()
        x_curr = np.copy(self.x0)

        # Implement MMQ algorithm
        # Use stopping criterion
        while np.linalg.norm(grad_f(x_curr)) > np.sqrt(N) * self.tol:
            # Create block
            j = self.counter % self.K + 1
            Nj = N // self.K
            lower, upper = (j - 1) * Nj , j * Nj
            A_j = A_block(x_curr, j, N // self.K)

            # Update block
            x_curr[lower:upper] += - sparse.linalg.bicg(A=A_j, b=grad_f(x_curr)[lower:upper])[0]

            if self.counter % 10 == 0:
                self.f_values.append(f(x_curr))
                self.iterations_time.append(time.time() - t0)
            self.counter += 1

        # Compute total time in seconds
        self.total_time = time.time() - t0
        # Store solution
        self.x_hat = x_curr

    def display(self):
        # Display reconstruction
        fig, axes = plt.subplots(1, 2, figsize=(14, 5))
        axes[0].imshow(x.reshape((90, 90), order="F"))
        axes[0].set_axis_off()
        axes[0].set_title("Original absorption image");
        axes[1].set_axis_off()
        axes[1].imshow(self.x_hat.reshape((90, 90), order="F"))
        axes[1].set_title(f"Reconstruction image (K={self.K})")
        # Display iterates
        plt.figure(figsize=(12, 5))
        plt.title(f"Value of $f$ along the iterations of {self.name} algorithm (K={self.K})")
        plt.xlabel("Iterations time (in seconds)")
        plt.ylabel("Value of $f$")
        plt.plot(self.iterations_time, self.f_values,
                 linestyle="--", marker="o", label=f"{self.short} iterates")
        plt.yscale("log")
        plt.legend(frameon=True);

    def __repr__(self):
        if self.counter is None:
            raise ValueError("Cannot call print before using ``solve``")
        return (f"{self.name}: {self.counter} steps "
               f"in {round(self.total_time, 2)} seconds (K={self.K})")

```

```python
%%time
solver = BCMMQ(K=5)
solver.solve(x0=x0, tol=1e-2)
print(solver)
solver.display()
```

```python
# Setup
K_range = [5, 6, 9] # I was getting MemoryError using K=1 or K=2 or K=3
reconstructions = list()
computation_times = list()
f_iterates = list()

for K in K_range:
    solver = BCMMQ(K=K)
    solver.solve(x0=x0, tol=1e-1)
    print(solver)
    x_hat, _ , iter_time, f_values, _ = solver.get_info()
    reconstructions.append(x_hat)
    computation_times.append(iter_time)
    f_iterates.append(f_values)

# Grid plot for reconstruction images
fig, axes = plt.subplots(1, len(K_range) + 1, figsize=(12, 5))
axes[0].imshow(x.reshape((90, 90), order="F"))
axes[0].set_axis_off()
axes[0].set_title("Original absorption image");
for i in range(1, len(K_range) + 1):
    axes[i].imshow(reconstructions[i - 1].reshape((90, 90), order="F"))
    axes[i].set_axis_off()
    axes[i].set_title(f"Reconstruction image (K={K_range[i - 1]})")

# Single plot for iterates evolution
plt.figure(figsize=(12, 5))
plt.title(f"Value of $f$ along the iterations of the BCMMQ algorithm for different K")
plt.xlabel("Iterations time (in seconds)")
plt.ylabel("Value of $f$")
for i in range(len(K_range)):
    plt.plot(computation_times[i], f_iterates[i],
         linestyle="--", marker="o", label=f"K={K_range[i]}")
plt.yscale("log")
plt.legend(frameon=True);
```

**Comment:**

* It looks like the higher ``K`` is, faster is the convergence. In the last section, we will evaluate the Block Coordinate MM Quadratic algorithm using ``K=9``.

### 3.5 Parallel MM quadratic algorithm (Bonus)

### 3.6 Comparison of the methods

#### Question 1.

Create a function that computes the value of the criterion f along the iterations of the algorithm

---

We did that previously using ``f = lambda x : 1 / 2 * np.sum((H.dot(x) - y)**2) + λ * np.sum(ϕ(G.dot(x)))``

where $\phi$ is defined as ``ϕ = lambda u : np.sqrt(1 + u**2 / δ ** 2)``

#### Question 2.

We will consider that the convergence is reached when the following stopping criterion is fulfilled:

$$\|\nabla f(x_k) \| \leq \sqrt{N} \times 10^{-4}$$

* What is the required time for each method to achieve this condition?
* For each method, plot the evolution of $\big(f(x_k)\big)_{k \in \mathbb{N}}$ until the stopping criterion is satisfied.

---

```python
# Setup
tol=1e-4
solvers = [GradientDescent(),
           MMQuadratic(),
           MMMG(),
           BCMMQ(K=9)]

reconstructions = list()
iterates_times = list()
iterates_values = list()
short_names = list()
total_times = dict()

# Loop over solvers
for solver in solvers:
    solver.solve(x0=x0, tol=tol)
    print(solver)
    x_hat, _ , iter_time, f_values, total_time = solver.get_info()
    # Store results
    reconstructions.append(x_hat)
    iterates_times.append(iter_time)
    iterates_values.append(f_values)
    short_names.append(solver.short)
    total_times[solver.short] = total_time

# Grid plot for reconstruction images
fig, axes = plt.subplots(1, len(solvers) + 1, figsize=(12, 5))
axes[0].imshow(x.reshape((90, 90), order="F"))
axes[0].set_axis_off()
axes[0].set_title("Original");
for i in range(1, len(solvers) + 1):
    axes[i].imshow(reconstructions[i - 1].reshape((90, 90), order="F"))
    axes[i].set_axis_off()
    axes[i].set_title(f"{short_names[i - 1]}")

# Single plot for iterates evolution
plt.figure(figsize=(12, 5))
plt.title(f"Comparing the iterates of $f$ using different solvers (tol={tol})")
plt.xlabel("Iterations time (in seconds)")
plt.ylabel("Value of $f$")
for i in range(len(solvers)):
    plt.plot(iterates_times[i], iterates_values[i],
        linewidth=2, linestyle="--",
        marker="o", markersize=6,
        label=f"{short_names[i]}")
plt.yscale("log")
plt.xscale("log")
plt.legend(frameon=True);

# Print sorted convergence times
print("-"*10)
print("Sorted convergence times")
for i, solver in enumerate(sorted(total_times, key=total_times.get)):
    print(f"{i+1}) {solver} in {round(total_times[solver], 2)}s")
print("-"*10)
```

#### Question 3.

The Signal to Noise Ratio (SNR) of a restored image $\hat{x}$ is defined as:

$$\text{SNR} = 10 \log_{10} \big( \frac{\|\overline{x}\|^2}{\|\overline{x} - \hat{x}\|^2} \big)$$

* Using the fastest method, look for the parameters $(\lambda, \delta)$ that optimize the SNR.

---

Let us redefine our ``MMQuadratic`` solver so that we can incorporate $\lambda$ and $\delta$ seamlessly.

```python
class MMQuadraticFinal(Solver):

    def __init__(self):
        self.name = "MM quadratic"
        self.short = "MMQ"

    def solve(self, x0=x0, tol=1e-2, λ=λ, δ=δ):
        # Setup
        self.x0 = x0
        self.tol = tol
        self.counter = 0
        self.f_values = list()
        self.iterations_time = list()

        # Redefine f
        ϕ = lambda u : np.sqrt(1 + u**2 / δ ** 2)
        f = lambda x : 1 / 2 * np.sum((H.dot(x) - y)**2) + λ * np.sum(ϕ(G.dot(x)))

        # Initialization
        t0 = time.time()
        x_curr = np.copy(self.x0)

        # Main loop
        while np.linalg.norm(grad_f(x_curr, λ=λ, δ=δ)) > np.sqrt(N) * self.tol:
            # Instanciate Linear Operator
            A = sparse.linalg.LinearOperator(shape=(N, N),
                                             matvec=curvature_A(x_curr, λ=λ, δ=δ),
                                             rmatvec=curvature_A(x_curr, λ=λ, δ=δ))
            # Solve Linear System
            x_curr = x_curr - sparse.linalg.bicg(A=A, b=grad_f(x_curr, λ=λ, δ=δ))[0]
            # Save iterates
            if self.counter % 10 == 0:
                self.f_values.append(f(x_curr))
                self.iterations_time.append(time.time() - t0)
            self.counter += 1

        # Compute total time in seconds
        self.total_time = time.time() - t0
        # Store solution
        self.x_hat = x_curr
```

```python
%%time
# Compute the Signal to Noise Ratio between restored and original images
SNR = lambda x_hat : 10 * np.log10(np.sum(x ** 2) / np.sum((x - x_hat) ** 2))

# Setup
tol = 1e-3
λ_range = np.linspace(0.5, 2, 4, endpoint=True)
δ_range = np.linspace(0.5, 2, 4, endpoint=True)

settings = list(product(λ_range, δ_range))
n_settings = len(settings)

best_SNR = - np.inf
best_approx = None
best_λ = None
best_δ = None

solver = MMQuadraticFinal()

# Main loop
for i, setup in enumerate(settings):
    λ, δ = setup
    print("-"*30)
    print(f"Run {i+1}/{n_settings}: λ={λ}, δ={δ}")
    solver.solve(x0=x0, tol=tol, λ=λ, δ=δ)
    approx = solver.x_hat
    score = SNR(approx)

    # Update if new print
    if score > best_SNR:
        best_SNR = score
        best_approx = approx
        best_λ = λ
        best_δ = δ
        print(f"New best: SNR={round(best_SNR, 4)} (λ={best_λ}, δ={best_δ})")

print(f"Best SNR: {round(best_SNR, 4)} using λ={best_λ}, δ={best_δ}")

# Display reconstruction
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].imshow(x.reshape((90, 90), order="F"))
axes[0].set_axis_off()
axes[0].set_title("Original absorption image")
axes[1].set_axis_off()
axes[1].imshow(best_approx.reshape((90, 90), order="F"))
axes[1].set_title(f"Best reconstruction image (SNR={round(best_SNR, 4)})");
```

**Comment:**

* We recover nicely the details compared to previous approximations that were using different values for $\lambda$ and $\delta$.


# Least-Squares Fitting

### Prof. Robert Quimby
© 2018 Robert Quimby

## In this tutorial you will...

* find the best-fit model when you have more data than model parameters
* learn how to use `numpy.matrix` objects to do least-square fits
* estimate the uncertainty in the best-fit model parameters

## Plot some Data

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (7, 5)

x = [1.3, 3.4]
y = [2.1, 5.9]

plt.plot(x, y, 'ro')
plt.xlim(0, 8)
plt.ylim(0, 15)
```

## Fit a Line to these Data

$y = mx + b$

$$
\begin{align}
  5.9 & =  3.4  m  +  b \\
-( 2.1 & = 1.3  m  +  b) \\
\hline
  3.8 & =  2.1  m
\end{align}
$$

```python
m =
b =
```

## Plot the linear relation

```python
plt.plot(x, y, 'ro')
plt.xlim(0, 8)
plt.ylim(0, 15)

import numpy as np
modelx = np.linspace(0, 8, 10)
modely = m * modelx + b
plt.plot(modelx, modely, '--')
```

## What if we want to fit a line to three points?

```python
x = [1.3, 3.4, 6.4]
y = [2.1, 5.9, 13.5]
plt.plot(x, y, 'ro')

# overlay the model
plt.plot(modelx, modely, '--')
plt.xlim(0, 8)
plt.ylim(0, 15)
```

## Dealing with Overdetermined Data

* Real world measurements are imperfect!
* Your best attempts to measure something will still have some error
* The quantity you are trying to measure may deviate from the predictions of your model

#### Therefore...
* Three or more data points will usually **not** fix *exactly* on a line, even if you think they should

## Fitting Models to Overdetermined Data

* **IF** we can assume that the deviations from our ideal model are random and follow a Gaussian distribution
* **THEN** there is an ideal method for determining the best fitting model

## Least-Squares Fitting

* minimize the sum of the square of the deviations from the model

### Fit a line to the data in the least-squares sense

data:
$$x = [1.3, 3.4, 6.4]$$
$$y = [2.1, 5.9, 13.5]$$

model: $$y = mx + b$$

#### The sum of the squares of the deviations as a function of $m$ and $b$ is:
$$S(m, b) = \Sigma (y_i - \theta_i)^2$$

where $\theta_i$ is the **model prediction** for the $i^{\rm th}$ data point. With $\theta_i = mx_i + b$, we have:

$$S(m, b) =  (2.1 - (1.3m + b))^2 \\ + (5.9 - (3.4m + b))^2 \\ + (13.5 - (6.4m + b))^2$$

This simplifies to:

$$ S(m, b) = 54.21 m^2 + 22.2  m  b - 218.38  m + 3 b^2 - 43  b + 221.47$$

### Now we just have to minimize this...

### Find the partial derivatives, $\delta S / \delta m$ and $\delta S / \delta b$

$$ \delta S / \delta m  = 108.42 m + 22.2 b - 218.38 $$
$$ \delta S / \delta b = 6 b + 22.2 m - 43 $$

### ...set these to zero and solve for $m$ and $b$

```python
m = 494. / 219.
b = (43 - 22.2 * m) / 6
print(m, b)
```

```python
# now we can plot the best fit line with our data
modely = m * modelx + b
plt.plot(x, y, 'ro')
plt.plot(modelx, modely, '--')
plt.xlim(0, 8)
plt.ylim(0, 15)
```

## That was for a two parameter model with 3 data points...

* think about doing this for, say, 100 data points

## Matricies to the Rescue!

We can take the three equations:

$$
\begin{align}
2.1   &  =  1.3m + b\\
5.9   &  =  3.4m + b\\
13.5  &  =  6.4m + b\\
\end{align}
$$

and turn them into a single matrix equation:

$$ Y = Xp $$

$Y =
\left[ \begin{array}{c}
2.1  \\
5.9  \\
13.5  \end{array} \right] $,
$X =
\left[ \begin{array}{cc}
1.3 & 1 \\
3.4 & 1 \\
6.4 & 1 \end{array} \right]
$, and $p = \left[ \begin{array}{c}
m \\
b \end{array} \right] $

### Now, just solve for $p$!

$$ Xp = Y $$

$$ X^T X p = X^T Y $$

$$ (X^T X)^{-1} (X^T X) p = (X^T X)^{-1} X^T Y $$

$$ p = (X^T X)^{-1} X^T Y $$

## A quick intro to `numpy.matrix` objects

### `numpy.matrix` is NOT the same as `numpy.array`

```python
# create an array and a matrix for testing
a1 =
m1 =
print("the array is:\n", a1)
print("the matrix is:\n", m1)
```

```python
# you can multiply them by scalars
print("the scaled array is: \n", a1 * 2)
print("the scaled matrix is: \n", m1 * 2)
```

```python
# you can add them
print("the array sum is:\n", a1 + a1)
print("the matrix sum is:\n", m1 + m1)
```

### Key difference between `numpy.matrix` and `numpy.array`: multiplication!

```python
# note that array and matrix multiplacation is diferent
print("the array product is:\n", a1 * a1)
print("the matrix product is:\n", m1 * m1)
```

## Recall matrix multiplication

$$
\left[ \begin{array}{cc}
a & b \\
c & d \\
\end{array} \right]
\left[ \begin{array}{cc}
w & x \\
y & z \\
\end{array} \right]
=
\left[ \begin{array}{cc}
aw + by & ax + bz \\
cw + dy & cx + dz \\
\end{array} \right] $$

for more see:
 * https://en.wikipedia.org/wiki/Matrix_multiplication
 * http://mathworld.wolfram.com/MatrixMultiplication.html

## Matrix Math Makes Least-Squares Fitting a Snap!

Express the model, $y = mx + b$ as:
$$ Y = Xp $$

where $Y =
\left[ \begin{array}{c}
y_0  \\
y_1  \\
 \vdots  \\
y_{N-1}  \end{array} \right] $,
$X =
\left[ \begin{array}{cc}
x_0 & 1 \\
x_1 & 1 \\
 \vdots & \vdots \\
x_{N-1} & 1 \end{array} \right]
$, and $p = \left[ \begin{array}{c}
m \\
b \end{array} \right] $

### here's what that looks like with `numpy.matrix` objects

```python
# set up the x in a 3 row by two column (N x 2) matrix
X =
```

```python
# set up the y in a single column matrix
Y =
```

```python
# solve for p
p =
```

## What if we have additional parameters in our model?

As long as you can express the model as a linear equation with independant model parameters
$$ Y = Xp $$
will work.

### for example, what about a quadratic equation...

$$ y = ax^2 + bx + c $$

no problem!

$$
\left[ \begin{array}{c}
y_0  \\
y_1  \\
 \vdots  \\
y_N  \end{array} \right] = \left[ \begin{array}{ccc}
x_0^2 & x_0 & 1 \\
x_1^2 & x_1 & 1 \\
 \vdots & \vdots \\
x_N^2 & x_N & 1 \end{array} \right]
\left[ \begin{array}{c}
a \\
b \\
c \end{array} \right] $$

## Sample Variance

* How close are the data points to the model?

```python
# predicted y values
modelY =

# residuals from observed values

```

### Variance (${\rm var} = \sigma^2$) of the data from the model

```python
M = ???? # number of data samples
N = ???? # number of model parameters
sample_var = ????
print(sample_var)
```

```python
# plot residuals
plt.errorbar(x, y, np.sqrt(sample_var[0,0]), ls='None', marker='o', color='red', capsize=3)
plt.plot(modelx, modely, '--')
```

## Uncertainty in the model parameters

```python
# now for the model parameter uncertainties
pvar =
psig =
```

```python
# best fit parameters
m, b =
msig, bsig =

# now for the parameter errors
print("m is {:.3f} +/- {:.3f}".format(m, msig))
print("b is {:.3f} +/- {:.3f}".format(b, bsig))
```

## For more details see...

["Least-Squares and Chi-Square for the Budding Aficionado: Art and Practice"](http://ugastro.berkeley.edu/radio/2015/handout_links/lsfit_2008.pdf) by Carl Heiles (UC Berkeley)


# Sinais exponenciais

Neste notebook avaliaremos os sinais exponenciais do tipo

\begin{equation}
x(t) = A \ \mathrm{e}^{a \ t}
\end{equation}

Estamos interessados em 3 casos:

1. $A \ \in \ \mathbb{R}$ e $a \ \in \ \mathbb{R}$  - As exponenciais reais.

2. $A \ \in \ \mathbb{C}$ e $a \ \in \ \mathbb{C}, \ \mathrm{Re}\left\{a\right\} = 0$   - As exponenciais complexas.

3. $A \ \in \ \mathbb{C}$ e $a \ \in \ \mathbb{C}, \ \mathrm{Re}\left\{a\right\} < 0$

```python
# importar as bibliotecas necessárias
import numpy as np
import matplotlib.pyplot as plt
```

## 1. $A \ \in \ \mathbb{R}$ e $a \ \in \ \mathbb{R}$  - As exponenciais reais.

```python
# Sinal
t = np.linspace(-1, 5, 1000) # vetor temporal
A = 1
a = -5

xt = A*np.exp(a*t)

# Figura
plt.figure()
plt.title('Exponencial real')
plt.plot(t, xt, '-b', linewidth = 2, label = 'Exp. Real')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.tight_layout()
plt.show()
```

## 2. $A \ \in \ \mathbb{C}$ e $a \ \in \ \mathbb{C}, \ \mathrm{Re}\left\{a\right\} = 0$   - As exponenciais complexas.

Temos grande interesse neste caso. Ele está relacionado ao sinal

\begin{equation}
x(t) = A \ \mathrm{cos}(2 \pi f t - \phi) = A \ \mathrm{cos}(\omega t - \phi).
\end{equation}

A relação de Euler nos diz que:

\begin{equation}
\mathrm{cos}(\omega t - \phi) = \mathrm{Re}\left\{\mathrm{e}^{\mathrm{j}(\omega t-\phi)}    \right\} .
\end{equation}

Assim, o sinal $x(t)$ torna-se:

\begin{equation}
x(t) = A \ \mathrm{cos}(\omega t - \phi) = A\mathrm{Re}\left\{\mathrm{e}^{\mathrm{j}(\omega t-\phi)}    \right\} = \mathrm{Re}\left\{A \mathrm{e}^{\mathrm{j}(\omega t-\phi)}    \right\}
\end{equation}

\begin{equation}
x(t) = \mathrm{Re}\left\{A\mathrm{e}^{-\mathrm{j}\phi} \ \mathrm{e}^{\mathrm{j}\omega t}    \right\}
\end{equation}
em que $\tilde{A} = A\mathrm{e}^{-\mathrm{j}\phi}$ é a amplitude complexa do cosseno e contêm as informações de magnitude, $A$, e fase, $\phi$.

```python
# Sinal
t = np.linspace(-2, 2, 1000) # vetor temporal
A = 1.5
phi = 0

A = A*np.exp(-1j*phi)

f=1
w = 2*np.pi*f
a = 1j*w

xt = np.real(A*np.exp(a*t))

# Figura
plt.figure()
plt.title('Exponencial complexa')
plt.plot(t, xt, '-b', linewidth = 2, label = 'Exp. complexa')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.ylim((-2, 2))
plt.xlim((t[0], t[-1]))
plt.tight_layout()
plt.show()
```

3. $A \ \in \ \mathbb{C}$ e $a \ \in \ \mathbb{C}, \ \mathrm{Re}\left\{a\right\} < 0$

Neste caso, se $a \ \in \ \mathbb{C}$ e  $\mathrm{Re}\left\{a\right\} < 0$, teremos um sinal oscilatório, cuja amplitude decai com o tempo. É típico de um sistema massa-mola-amortecedor.

```python
# Sinal
t = np.linspace(0, 5, 1000) # vetor temporal
A = 1.5
phi = 0.3
A = A*np.exp(-1j*phi)

f=1
w = 2*np.pi*f
a = -0.5+1j*w

xt = np.real(A*np.exp(a*t))

# Figura
plt.figure()
plt.title('Exponencial complexa vom decaimento')
plt.plot(t, xt, '-b', linewidth = 2, label = 'Exp. complexa com decaimento')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.ylim((-2, 2))
plt.xlim((0, t[-1]))
plt.tight_layout()
plt.show()
```

## Exponenciais complexas em sinais discretos

Retomamos agora as exponenciais complexas e discretas. Lembramos que para um sinal contínuo,  $x(t)=\mathrm{e}^{\mathrm{j} \omega t}$, a taxa de oscilação aumenta quando $\omega$ aumenta. Para sinais discretos do tipo

\begin{equation}
x[n] = \mathrm{e}^{\mathrm{j}  \omega  n}
\end{equation}
veremos um aumento da taxa de oscilação para $0 \leq \omega < \pi$ e uma diminuição da taxa de oscilação para  $\pi \leq \omega \leq 2 \pi$. Isto se relacionará depois com a amostragem de sinais. Veremos que para amostrar um sinal corretamente (representar bem suas componentes de frequência), precisaremos usar uma taxa de amostragem que seja pelo menos o dobro da maior frequência contida no sinal a ser amostrado.

```python
omega = [0, np.pi/8, np.pi/4, np.pi/2, np.pi,
         3*np.pi/2, 7*np.pi/4, 15*np.pi/8, 2*np.pi]#np.linspace(0, 2*np.pi, 9) # Frequencias angulares
n = np.arange(50) # amostras

plt.figure(figsize=(15,10))
for jw,w in enumerate(omega):
    xn = np.real(np.exp(1j*w*n))

    plt.subplot(3,3,jw+1)
    plt.stem(n, xn, '-b', label = r'$\omega$ = {:.3} [rad/s]'.format(float(w)), basefmt=" ", use_line_collection=  True)
    plt.legend(loc = 'upper right')
    plt.grid(linestyle = '--', which='both')
    plt.xlabel('Amostra [-]')
    plt.ylabel('Amplitude [-]')
    plt.ylim((-1.5, 1.5))
    plt.tight_layout()

plt.show()
```

Aula 6 - Sinais exponenciais/sinais exponenciais.ipynb


```python
import numpy as np
import matplotlib.pyplot as plt
import math
import scipy.integrate as inte
```

### ODE
$$F(x,y,y',...,y^{(n-1)})=y^{(n)}$$
where $y^{(n)}$ is the n-th derivative.
- Order of an ODE is the highest order derivtive that appears
- Mechanical systems are usually second order
- Recall Newton's second $\mathbf{F=ma=m\ddot{x}}$

### State-space for 2nd order systems
Define vector equation

$$x = \begin{pmatrix} x_1\\x_2\end{pmatrix}=\begin{pmatrix}\chi\\\dot{\chi}\end{pmatrix}\Longrightarrow \dot{x}=\begin{pmatrix}x_2\\\frac{F}{m}\end{pmatrix}$$

### Numerical ODE integration
- Consider $\dot{x}=\alpha$
- Could use fixed timestep, $dt$
- Set $x(t_{k+1})=x(t_k)+\alpha dt$
- If $\alpha(t)$ is not fixed, the results will be inaccurate
- ode estimates how the right side is changing and picks the best timestep $dt$.
- Only works when right hand side is smooth

### Examples from MATLAB to python
Try to solve the following ode problems from https://uk.mathworks.com/help/matlab/ref/ode45.html using python.

### Solve ODE with Single Solution Component
$$y'=2t$$

```python
def odeSSC(t,y):
    return 2*t

t0, t1 = 0, 5
t = np.linspace(t0,t1,100)
y0 = 0
y = np.zeros((len(t),1))
y[0,:] = y0
r = inte.ode(odeSSC).set_integrator("dopri5")  # choice of method
r.set_initial_value(y0, t0)   # initial values
for i in range(1, t.size):
   y[i, :] = r.integrate(t[i]) # get one more value, add it to the array
   if not r.successful():
       raise RuntimeError("Could not integrate")
plt.plot(t, y)
```

### Solve Nonstiff equation
The van der Pol equation is a second order ODE:
$$y_1''-\mu(1-y_1^2)y_1'+y_1=0$$
where $\mu>0$ is a scalar parameter. Rewrite this equation as first order ODEs by making the substitution $y_2=y_1'$. The resulting system of first-order ODEs is,

\begin{align}
y_1' &= y_2 \\
y_2' &= \mu(1-y_1^2)y_2-y_1
\end{align}

```python
def vdp1(t, y):
    return np.array([y[1], (1 - y[0]**2)*y[1] - y[0]])

t0, t1 = 0, 20                # start and end
t = np.linspace(t0, t1, 100)  # the points of evaluation of solution
y0 = [2, 0]                   # initial value
y = np.zeros((len(t), len(y0)))   # array for solution
y[0, :] = y0
r = inte.ode(vdp1).set_integrator("dopri5")  # choice of method
r.set_initial_value(y0, t0)   # initial values
for i in range(1, t.size):
   y[i, :] = r.integrate(t[i]) # get one more value, add it to the array
   if not r.successful():
       raise RuntimeError("Could not integrate")
plt.plot(t, y)
```

### Pass Extra Parameters to ODE Function
Solve the ODE,
$$y''=\frac{A}{B}ty$$
Rewriting the equation as a 1st order ODE yields,

\begin{align}
y_1' &= y_2\\
y_2' &= \frac{A}{B}ty_1
\end{align}

```python
def odefcn(t,y,A,B):
    return np.array([y[1],A/B*t*y[0]])
A,B = 1,2
t0, t1 = 0, 5
t = np.linspace(t0,t1,100)
y0 = [0,0.01]                   # initial value
y = np.zeros((len(t), len(y0)))   # array for solution
r = inte.ode(lambda t,y:odefcn(t,y,A,B)).set_integrator("dopri5")  # choice of method
r.set_initial_value(y0, t0)   # initial values
for i in range(1, t.size):
   y[i, :] = r.integrate(t[i]) # get one more value, add it to the array
   if not r.successful():
       raise RuntimeError("Could not integrate")
plt.plot(t, y)
```

### ODE with Time-Dependent Terms
Consider the following ODE with time-dependent parameters,
$$y'(t)+f(t)y(t) = g(t)$$

```python
ft = np.linspace(0,5,25)
gt = np.linspace(1,6,25)
f = ft**2-ft-3
g = 3*np.sin(gt-.25)
t0, t1, ic, y0 = 1, 5, 1, 1
t= np.linspace(t0,t1,100)
```

```python
def myode(t,y,ft,f,gt,g):
    f = np.interp(t,ft,f)
    g = np.interp(t,gt,g)
    dydt = -f*y+g
    return dydt
```

```python
y = np.zeros((len(t), 1))   # array for solution
r = inte.ode(lambda t,y:myode(t,y,ft,f,gt,g)).set_integrator("dopri5",atol=1e-4,rtol=1e-2)  # choice of method
r.set_initial_value(y0, t0)   # initial values
for i in range(1, t.size):
    tmp = r.integrate(t[i]) # get one more value, add it to the array
    y[i, :] = tmp
    if not r.successful():
        raise RuntimeError("Could not integrate")
plt.plot(t, y)
```

### Evaluate and Extend Solution Structure
The van der Pol equation is a second order ODE
$$y_1''-\mu(1-y_1^2)y_1'+y_1=0$$

```python
t0, t1 = 0, 20                # start and end
t = np.linspace(t0, t1, 100)  # the points of evaluation of solution
y0 = [2, 0]                   # initial value
y = np.zeros((len(t), len(y0)))   # array for solution
y[0, :] = y0
r = inte.ode(vdp1).set_integrator("dopri5")  # choice of method
r.set_initial_value(y0, t0)   # initial values
for i in range(1, t.size):
   y[i, :] = r.integrate(t[i]) # get one more value, add it to the array
   if not r.successful():
       raise RuntimeError("Could not integrate")
```

```python
plt.plot(t,y[:,0])
```

```python
x0, x1 = 20,35
x = np.linspace(x0,x1,350)
yx0 = y[-1,:]
yx = np.zeros((len(x), len(yx0)))   # array for solution
yx[0,:] = yx0

r.set_initial_value(yx0, x0)   # initial values
for i in range(1, x.size):
   yx[i, :] = r.integrate(x[i]) # get one more value, add it to the array
   if not r.successful():
       raise RuntimeError("Could not integrate")

plt.plot(t,y[:,0])
plt.plot(x,yx[:,0])
```

### Robotics Capstone
#### 1. Solving ODE
$$\dot{x}=\begin{pmatrix}\dot{x_1}\\\dot{x_2}\end{pmatrix}=\begin{pmatrix} x_2 \\ \gamma\cos(\omega t)-\alpha x_1-\delta x_2 -\beta x_1^3\end{pmatrix}$$

```python
def dyn(t,X,beta):
    delta,alpha,gamma,omega = 1,1,1,1
    x = X[0]; xd = X[1]
    return np.array([xd,gamma*np.cos(omega*t)-alpha*x-delta*xd-beta*x**3])
```

```python
def simode(beta,tend):
    t0, t1 = 0, tend                # start and end
    t = np.linspace(t0, t1, 100)  # the points of evaluation of solution
    y0 = [1, 1]                   # initial value
    y = np.zeros((len(t), len(y0)))   # array for solution
    y[0, :] = y0
    r = inte.ode(lambda t,X: dyn(t,X,beta)).set_integrator("dopri5")
    r.set_initial_value(y0, t0)   # initial values
    for i in range(1, t.size):
       y[i, :] = r.integrate(t[i]) # get one more value, add it to the array
       if not r.successful():
           raise RuntimeError("Could not integrate")

    return t,y
```

```python
t,y = simode(1,50)
plt.figure(figsize=(20,5))
plt.subplot(1,2,1); plt.plot(t,y)
plt.subplot(1,2,2); plt.plot(y[:,0], y[:,1])
```

```python
t,y = simode(5,50)
plt.figure(figsize=(20,5))
plt.subplot(1,2,1); plt.plot(t,y)
plt.subplot(1,2,2); plt.plot(y[:,0], y[:,1])
```

```python
t,y = simode(10,50)
plt.figure(figsize=(20,5))
plt.subplot(1,2,1); plt.plot(t,y)
plt.subplot(1,2,2); plt.plot(y[:,0], y[:,1])
```

```python

```

Capstone/ODE solver/Ordinary Differential Equation.ipynb


# Práctica 1 - Introducción a Jupyter lab y libreria ```robots```

## Introducción al calculo simbólico numérico y librería ```sympy```

En este documento se describe el proceso de obtención de la dinámica de un robot manipulador (pendulo simple) por medio de la ecuación de Euler-Lagrange, empecemos importando las librerias necesarias:

```python
from sympy import var, sin, cos, pi, Matrix, Function, Rational
from sympy.physics.mechanics import mechanics_printing
mechanics_printing()
```

Una vez que hemos importado las funciones necesarias, podemos empezar definiendo las variables a utilizar dentro del calculo:

```python
var("l1")
```

Cuando se definen las variables, se pueden mandar a llamar con el mismo nombre:

```python
l1
```

Definimos de una vez todas las variables necesarias:

```python
var("m1 J1 t g")
```

Y definimos las variables que dependen de otra variable, especificamente en este calculo, todo lo anterior es constante y solo $q_1$ es una variable dependiente del tiempo:

```python
q1 = Function("q1")(t)
```

Ya con las variables definidas, puedo empezar a definir la posición del centro de masa del primer (y único eslabon):

```python
x1 = l1*cos(q1)
y1 = l1*sin(q1)
```

```python
x1
```

```python
y1
```

De manera que si necesitamos calcular la derivada con respecto del tiempo de $x_1$, tenemos que hacer:

```python
x1.diff(t)
```

Calculamos el cuadrado de la velocidad traslacional del primer centro de masa:

```python
v1c = x1.diff(t)**2 + y1.diff(t)**2
```

```python
v1c
```

Pero como se puede ver, no necesariamente se va a simplificar completamente la expresión calculada; por lo que podemos decir explicitamente que trate de simplificar más, el motor de algebra simbolica:

```python
v1c.simplify()
```

Guardando esta expresion simplificada:

```python
v1c = v1c.simplify()
```

Y calculando la altura y velocidad rotacional del eslabon:

```python
h1 = y1
ω1 = q1.diff(t)
```

Calculando la energía cinética y potencial:

```python
K = Rational(1,2)*m1*v1c + Rational(1,2)*J1*ω1**2
```

```python
U = m1*g*h1
```

Con estas energias se puede calcular el Lagrangiano:

```python
L = K - U
```

```python
L
```

Y una vez obtenido el Lagrangiano, podemos empezar a derivar, $\frac{\partial L}{\partial \dot{q}_1}$, $\frac{d}{dt}\left( \frac{\partial L}{\partial \dot{q}_1} \right)$ y $\frac{\partial L}{\partial q_1}$

```python
L.diff(q1.diff(t))
```

```python
L.diff(q1.diff(t)).diff(t)
```

```python
L.diff(q1)
```

O bien, agrupandolo en la ecuación de Euler - Lagrange:

$$
\tau_1 =
\frac{d}{dt}\left( \frac{\partial L}{\partial \dot{q}_1} \right) - \frac{\partial L}{\partial q_1}
$$

```python
L.diff(q1.diff(t)).diff(t) - L.diff(q1)
```

En este caso, podemos utilizar el metodo collect para factorizar con respecto a ciertos terminos, en este caso $\ddot{q}_1$:

```python
τ1 = (L.diff(q1.diff(t)).diff(t) - L.diff(q1)).collect(q1.diff(t).diff(t))
```

```python
τ1
```

Una vez que hemos concluido este proceso, podemos pasar al documento llamado ```numerico.ipynb```


# Solution {-}

```python
from sympy import Matrix, symbols, sqrt

xA, yA, xB, yB, r1, r2, x, xdot, y, ydot = symbols('xA, yA, xB, yB, r1 r2 x xdot y ydot')

# Measurement equation
r1 = sqrt((xA - x)**2 + (yA - y)**2)
r2 = sqrt((xB - x)**2 + (yB - y)**2)

# State vector
x = Matrix([[x],
            [xdot],
            [y],
            [ydot]])

H = Matrix([[r1],
            [r2]])
dH = H.jacobian(x)
display(dH)
```

$\displaystyle \left[\begin{matrix}\frac{x - xA}{\sqrt{\left(- x + xA\right)^{2} + \left(- y + yA\right)^{2}}} & 0 & \frac{y - yA}{\sqrt{\left(- x + xA\right)^{2} + \left(- y + yA\right)^{2}}} & 0\\\frac{x - xB}{\sqrt{\left(- x + xB\right)^{2} + \left(- y + yB\right)^{2}}} & 0 & \frac{y - yB}{\sqrt{\left(- x + xB\right)^{2} + \left(- y + yB\right)^{2}}} & 0\end{matrix}\right]$

```python
from numpy import array, exp, sqrt, arange, eye, kron, diag, zeros, column_stack
from numpy.linalg import inv, norm
from lib.vanloan import numeval
import matplotlib.pyplot as plt

dt = 1
samples = 200
d = 10000

# Initial standard deviation
sigmap = sqrt(2000)  # Position [meter]
sigmav = 30          # Velocity [meter/second]
sigmar = 15          # Range [meter]

# Gauss-Markov process parameters
beta = 0.01  # Time constant [rad/second]
sigma = 30   # [meter/second]

# DME station coordinates (x, y)
dme1 = [-10000, 0]
dme2 = [ 10000, 0]
dme = column_stack([dme1, dme2])  # Make columnvector

# Dynamic matrix (2D)
F0 = array([[0, 1],
            [0, -beta]])
F = kron(eye(2), F0)

# White noise coefficients (2D)
G0 = array([[0],
           [sqrt(2*sigma**2*beta)]])
G = kron(eye(2), G0)

# Numerical evaluation (Van Loan)
phi, Q = numeval(F, G, dt)

# Inital state covariance matrix
P0 = array([[sigmap**2, 0],
            [0, sigmav**2]])
P = kron(eye(2), P0)

# Initial measurement covariance matrix
R = sigmar**2*diag([2, 2])

# Linearized design matrix
def dH(xs, xnom):

    dH = zeros([2, 4])

    dx = xs - xnom

    dH[0] = [-dx[0, 0]/norm(dx[:, 0]), 1, -dx[1, 0]/norm(dx[:, 0]), 0]
    dH[1] = [-dx[0, 1]/norm(dx[:, 1]), 0, -dx[1, 1]/norm(dx[:, 1]), 1]

    return dH

# Initialize plot vectors
P_all = []

# Main loop
for k in range(0, samples):

    # Nominal trajectory
    xnom = array([[0],
                  [-10000 + 100*k]])

    H = dH(dme, xnom)

    # Time update
    Pp = phi@P@phi.T + Q

    # Design matrix
    #h11 = d/sqrt(d**2 + ynom**2)
    #h12 = ynom/sqrt(d**2 + ynom**2)
    #h21 = -d/sqrt(d**2 + ynom**2)
    #h22 = ynom/sqrt(d**2 + ynom**2)
    #
    #H = array([[h11, 1, h12, 0],
    #           [h21, 0, h22, 1]])

    # Measurement update
    K = Pp@H.T@inv(H@Pp@H.T + R)
    P = (eye(4) - K@H)@Pp

    # Accumulate plot vectors
    P_all.append(P)

# Time vector
time = arange(0, samples)

# Extract plot vectors
std_x = [sqrt(P[0, 0]) for P in P_all]  # Standard deviation x-direction
std_y = [sqrt(P[2, 2]) for P in P_all]  # Standard deviation y-direction

# Plot results
plt.title('Error Analysis')
plt.plot(time, std_x, 'r', label='std_x')
plt.plot(time, std_y, 'g', label='std_y')
plt.xlabel('Time [second]')
plt.ylabel('Standard deviation [meter]')
plt.legend()
plt.grid()
plt.show()
```

### Comment
The increase in variance in y-direction is due to the lack of observability of the y-component of from the range measurements close to the origin. However, the geometry of the x-component is good throughout the whole flight.


# charging of capacitor

\begin{equation}
\frac{Q}{C}+R\frac{dQ}{dt}=V
\end{equation}

\begin{equation}
\frac{dQ}{dt}=\frac{V}{R}-\frac{Q}{CR}
\end{equation}

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
```

```python
# function that returns dy/dt
def model(y,t,C,R,V):
    dydt = V/R-y/(C*R)
    return dydt
```

```python
V=10 #V
C=1e-6 #F
R1=100 #ohm
R2 = 200
R3=400
```

```python
# time points
t = np.arange(0,0.002,0.0001) #
```

```python
# Charging
# initial condition
y0 = 0
y1 = odeint(model,y0,t,args=(C,R1,V))
y2 = odeint(model,y0,t,args=(C,R2,V))
y3 = odeint(model,y0,t,args=(C,R3,V,))

# plot results
plt.plot(t,y1,'r-',linewidth=2,label='R1=100$\Omega$')
plt.plot(t,y2,'b--',linewidth=2,label='R2=200$\Omega$')
plt.plot(t,y3,'g:',linewidth=2,label='R3=300$\Omega$')
plt.xlabel('time(sec)')
plt.xticks(rotation=45)
plt.ylabel('Q(C)')
plt.legend()
plt.show()

```

```python
#discharging

y0 = V*C
y11 = odeint(model,y0,t,args=(C,R1,0))
y12 = odeint(model,y0,t,args=(C,R2,0))
y13 = odeint(model,y0,t,args=(C,R3,0,))

# plot results
plt.plot(t,y11,'r-',linewidth=2,label='R1=100$\Omega$')
plt.plot(t,y12,'b--',linewidth=2,label='R2=200$\Omega$')
plt.plot(t,y13,'g:',linewidth=2,label='R3=300$\Omega$')
plt.xlabel('time(sec)')
plt.xticks(rotation=45)
plt.ylabel('Q(C)')
plt.legend()
plt.show()
```

```python

```


# Assignment 1

The goal of this assignment is to supply you with machine learning models and algorithms. In this notebook, we will cover linear and nonlinear models, the concept of loss functions and some optimization techniques. All mathematical operations should be implemented in **NumPy** only.

## Table of contents
* [1. Logistic Regression](#1.-Logistic-Regression)
    * [1.1 Linear Mapping](#1.1-Linear-Mapping)
    * [1.2 Sigmoid](#1.2-Sigmoid)
    * [1.3 Negative Log Likelihood](#1.3-Negative-Log-Likelihood)
    * [1.4 Model](#1.4-Model)
    * [1.5 Simple Experiment](#1.5-Simple-Experiment)
* [2. Decision Tree](#2.-Decision-Tree)
    * [2.1 Gini Index & Data Split](#2.1-Gini-Index-&-Data-Split)
    * [2.2 Terminal Node](#2.2-Terminal-Node)
    * [2.3 Build the Decision Tree](#2.3-Build-the-Decision-Tree)
* [3. Experiments](#3.-Experiments)
    * [3.1 Decision Tree for Heart Disease Prediction](#3.1-Decision-Tree-for-Heart-Disease-Prediction)
    * [3.2 Logistic Regression for Heart Disease Prediction](#3.2-Logistic-Regression-for-Heart-Disease-Prediction)

### Note
Some of the concepts below have not (yet) been discussed during the lecture. These will be discussed further during the next lectures.

### Before you begin

To check whether the code you've written is correct, we'll use **automark**. For this, we created for each of you an account with the username being your student number.

```python
import automark as am

# fill in you student number as your username
username = 'Your Username'

# to check your progress, you can run this function
am.get_progress(username)
```

So far all your tests are 'not attempted'. At the end of this notebook you'll need to have completed all test. The output of `am.get_progress(username)` should at least match the example below. However, we encourage you to take a shot at the 'not attempted' tests!

```
---------------------------------------------
| Your name / student number                |
| your_email@your_domain.whatever           |
---------------------------------------------
| linear_forward           | not attempted  |
| linear_grad_W            | not attempted  |
| linear_grad_b            | not attempted  |
| nll_forward              | not attempted  |
| nll_grad_input           | not attempted  |
| sigmoid_forward          | not attempted  |
| sigmoid_grad_input       | not attempted  |
| tree_data_split_left     | not attempted  |
| tree_data_split_right    | not attempted  |
| tree_gini_index          | not attempted  |
| tree_to_terminal         | not attempted  |
---------------------------------------------
```

```python
from __future__ import print_function, absolute_import, division # You don't need to know what this is.
import numpy as np # this imports numpy, which is used for vector- and matrix calculations
```

This notebook makes use of **classes** and their **instances** that we have already implemented for you. It allows us to write less code and make it more readable. If you are interested in it, here are some useful links:
* The official [documentation](https://docs.python.org/3/tutorial/classes.html)
* Video by *sentdex*: [Object Oriented Programming Introduction](https://www.youtube.com/watch?v=ekA6hvk-8H8)
* Antipatterns in OOP: [Stop Writing Classes](https://www.youtube.com/watch?v=o9pEzgHorH0)

# 1. Logistic Regression

We start with a very simple algorithm called **Logistic Regression**. It is a generalized linear model for 2-class classification.
It can be generalized to the case of many classes and to non-linear cases as well. However, here we consider only the simplest case.

Let us consider a data with 2 classes. Class 0 and class 1. For a given test sample, logistic regression returns a value from $[0, 1]$ which is interpreted as a probability of belonging to class 1. The set of points for which the prediction is $0.5$ is called a *decision boundary*. It is a line on a plane or a hyper-plane in a space.

Logistic regression has two trainable parameters: a weight $W$ and a bias $b$. For a vector of features $X$, the prediction of logistic regression is given by

$$
f(X) = \frac{1}{1 + \exp(-[XW + b])} = \sigma(h(X))
$$
where $\sigma(z) = \frac{1}{1 + \exp(-z)}$ and $h(X)=XW + b$.

Parameters $W$ and $b$ are fitted by maximizing the log-likelihood (or minimizing the negative log-likelihood) of the model on the training data. For a training subset $\{X_j, Y_j\}_{j=1}^N$ the normalized negative log likelihood (NLL) is given by

$$
\mathcal{L} = -\frac{1}{N}\sum_j \log\Big[ f(X_j)^{Y_j} \cdot (1-f(X_j))^{1-Y_j}\Big]
= -\frac{1}{N}\sum_j \Big[ Y_j\log f(X_j) + (1-Y_j)\log(1-f(X_j))\Big]
$$

There are different ways of fitting this model. In this assignment we consider Logistic Regression as a one-layer neural network. We use the following algorithm for the **forward** pass:

1. Linear mapping: $h=XW + b$
2. Sigmoid activation function: $f=\sigma(h)$
3. Calculation of NLL: $\mathcal{L} = -\frac{1}{N}\sum_j \Big[ Y_j\log f_j + (1-Y_j)\log(1-f_j)\Big]$

In order to fit $W$ and $b$ we perform Gradient Descent ([GD](https://en.wikipedia.org/wiki/Gradient_descent)). We choose a small learning rate $\gamma$ and after each computation of forward pass, we update the parameters

$$W_{\text{new}} = W_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial W}$$

$$b_{\text{new}} = b_{\text{old}} - \gamma \frac{\partial \mathcal{L}}{\partial b}$$

We use Backpropagation method ([BP](https://en.wikipedia.org/wiki/Backpropagation)) to calculate the partial derivatives of the loss function with respect to the parameters of the model.

$$
\frac{\partial\mathcal{L}}{\partial W} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial W} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial W}
$$

$$
\frac{\partial\mathcal{L}}{\partial b} =
\frac{\partial\mathcal{L}}{\partial h} \frac{\partial h}{\partial b} =
\frac{\partial\mathcal{L}}{\partial f} \frac{\partial f}{\partial h} \frac{\partial h}{\partial b}
$$

## 1.1 Linear Mapping
First of all, you need to implement the forward pass of a linear mapping:
$$
h(X) = XW +b
$$

**Note**: here we use `n_out` as the dimensionality of the output. For logisitc regression `n_out = 1`. However, we will work with cases of `n_out > 1` in next assignments. You will **pass** the current assignment even if your implementation works only in case `n_out = 1`. If your implementation works for the cases of `n_out > 1` then you will not have to modify your method next week. All **numpy** operations are generic. It is recommended to use numpy when is it possible.

```python
def linear_forward(x_input, W, b):
    """Perform the mapping of the input
    # Arguments
        x_input: input of the linear function - np.array of size `(n_objects, n_in)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the output of the linear function
        np.array of size `(n_objects, n_out)`
    """
    #################
    ### YOUR CODE ###
    #################
    return output
```

Let's check your first function. We set the matrices $X, W, b$:
$$
X = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix} \quad
W = \begin{bmatrix}
4 \\
2 \\
\end{bmatrix} \quad
b = \begin{bmatrix}
3 \\
\end{bmatrix}
$$

And then compute
$$
XW = \begin{bmatrix}
1 & -1 \\
-1 & 0 \\
1 & 1 \\
\end{bmatrix}
\begin{bmatrix}
4 \\
2 \\
\end{bmatrix} =
\begin{bmatrix}
2 \\
-4 \\
6 \\
\end{bmatrix} \\
XW + b =
\begin{bmatrix}
5 \\
-1 \\
9 \\
\end{bmatrix}
$$

```python
X_test = np.array([[1, -1],
                   [-1, 0],
                   [1, 1]])

W_test = np.array([[4],
                   [2]])

b_test = np.array([3])

h_test = linear_forward(X_test, W_test, b_test)
print(h_test)
```

```python
am.test_student_function(username, linear_forward, ['x_input', 'W', 'b'])
```

Now you need to implement the calculation of the partial derivative of the loss function with respect to the parameters of the model. As this expressions are used for the updates of the parameters, we refer to them as gradients.
$$
\frac{\partial \mathcal{L}}{\partial W} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial W} \\
\frac{\partial \mathcal{L}}{\partial b} =
\frac{\partial \mathcal{L}}{\partial h}
\frac{\partial h}{\partial b} \\
$$

```python
def linear_grad_W(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to W parameter of the function
        dL / dW = (dL / dh) * (dh / dW)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the dense layer (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss
        with respect to W parameter of the function
        np.array of size `(n_in, n_out)`
    """
    #################
    ### YOUR CODE ###
    #################
    return grad_W
```

```python
am.test_student_function(username, linear_grad_W, ['x_input', 'grad_output', 'W', 'b'])
```

```python
def linear_grad_b(x_input, grad_output, W, b):
    """Calculate the partial derivative of
        the loss with respect to b parameter of the function
        dL / db = (dL / dh) * (dh / db)
    # Arguments
        x_input: input of a dense layer - np.array of size `(n_objects, n_in)`
        grad_output: partial derivative of the loss functions with
            respect to the ouput of the linear function (dL / dh)
            np.array of size `(n_objects, n_out)`
        W: np.array of size `(n_in, n_out)`
        b: np.array of size `(n_out,)`
    # Output
        the partial derivative of the loss
        with respect to b parameter of the linear function
        np.array of size `(n_out,)`
    """
    #################
    ### YOUR CODE ###
    #################
    return grad_b
```

```python
am.test_student_function(username, linear_grad_b, ['x_input', 'grad_output', 'W', 'b'])
```

```python
am.get_progress(username)
```

## 1.2 Sigmoid
$$
f = \sigma(h) = \frac{1}{1 + e^{-h}}
$$

Sigmoid function is applied element-wise. It does not change the dimensionality of the tensor and its implementation is shape-agnostic in general.

```python
def sigmoid_forward(x_input):
    """sigmoid nonlinearity
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
    # Output
        the output of relu layer
        np.array of size `(n_objects, n_in)`
    """
    #################
    ### YOUR CODE ###
    #################
    return output
```

```python
am.test_student_function(username, sigmoid_forward, ['x_input'])
```

Now you need to implement the calculation of the partial derivative of the loss function with respect to the input of sigmoid.

$$
\frac{\partial \mathcal{L}}{\partial h} =
\frac{\partial \mathcal{L}}{\partial f}
\frac{\partial f}{\partial h}
$$

Tensor $\frac{\partial \mathcal{L}}{\partial f}$ comes from the loss function. Let's calculate $\frac{\partial f}{\partial h}$

$$
\frac{\partial f}{\partial h} =
\frac{\partial \sigma(h)}{\partial h} =
\frac{\partial}{\partial h} \Big(\frac{1}{1 + e^{-h}}\Big)
= \frac{e^{-h}}{(1 + e^{-h})^2}
= \frac{1}{1 + e^{-h}} \frac{e^{-h}}{1 + e^{-h}}
= f(h) (1 - f(h))
$$

Therefore, in order to calculate the gradient of the loss with respect to the input of sigmoid function you need
to
1. calculate $f(h) (1 - f(h))$
2. multiply it element-wise by $\frac{\partial \mathcal{L}}{\partial f}$

```python
def sigmoid_grad_input(x_input, grad_output):
    """sigmoid nonlinearity gradient.
        Calculate the partial derivative of the loss
        with respect to the input of the layer
    # Arguments
        x_input: np.array of size `(n_objects, n_in)`
        grad_output: np.array of size `(n_objects, n_in)`
            dL / df
    # Output
        the partial derivative of the loss
        with respect to the input of the function
        np.array of size `(n_objects, n_in)`
        dL / dh
    """
    #################
    ### YOUR CODE ###
    #################
    return grad_input
```

```python
am.test_student_function(username, sigmoid_grad_input, ['x_input', 'grad_output'])
```

## 1.3 Negative Log Likelihood

$$
\mathcal{L}
= -\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]
$$

Here $N$ is the number of objects. $Y_j$ is the real label of an object and $\dot{Y}_j$ is the predicted one.

```python
def nll_forward(target_pred, target_true):
    """Compute the value of NLL
        for a given prediction and the ground truth
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the value of NLL for a given prediction and the ground truth
        scalar
    """
    #################
    ### YOUR CODE ###
    #################
    return output
```

```python
am.test_student_function(username, nll_forward, ['target_pred', 'target_true'])
```

Now you need to calculate the partial derivative of NLL with with respect to its input.

$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}}
=
\begin{pmatrix}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0} \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_1} \\
\vdots \\
\frac{\partial \mathcal{L}}{\partial \dot{Y}_N}
\end{pmatrix}
$$

Let's do it step-by-step

\begin{equation}
\begin{split}
\frac{\partial \mathcal{L}}{\partial \dot{Y}_0}
&= \frac{\partial}{\partial \dot{Y}_0} \Big(-\frac{1}{N}\sum_j \Big[ Y_j\log \dot{Y}_j + (1-Y_j)\log(1-\dot{Y}_j)\Big]\Big) \\
&= -\frac{1}{N} \frac{\partial}{\partial \dot{Y}_0} \Big(Y_0\log \dot{Y}_0 + (1-Y_0)\log(1-\dot{Y}_0)\Big) \\
&= -\frac{1}{N} \Big(\frac{Y_0}{\dot{Y}_0} - \frac{1-Y_0}{1-\dot{Y}_0}\Big)
= \frac{1}{N} \frac{\dot{Y}_0 - Y_0}{\dot{Y}_0 (1 - \dot{Y}_0)}
\end{split}
\end{equation}

And for the other components it can be done in exactly the same way. So the result is the vector where each component is given by
$$\frac{1}{N} \frac{\dot{Y}_j - Y_j}{\dot{Y}_j (1 - \dot{Y}_j)}$$

Or if we assume all multiplications and divisions to be done element-wise the output can be calculated as
$$
\frac{\partial \mathcal{L}}{\partial \dot{Y}} = \frac{1}{N} \frac{\dot{Y} - Y}{\dot{Y} (1 - \dot{Y})}
$$

```python
def nll_grad_input(target_pred, target_true):
    """Compute the partial derivative of NLL
        with respect to its input
    # Arguments
        target_pred: predictions - np.array of size `(n_objects, 1)`
        target_true: ground truth - np.array of size `(n_objects, 1)`
    # Output
        the partial derivative
        of NLL with respect to its input
        np.array of size `(n_objects, 1)`
    """
    #################
    ### YOUR CODE ###
    #################
    return grad_input
```

```python
am.test_student_function(username, nll_grad_input, ['target_pred', 'target_true'])
```

```python
am.get_progress(username)
```

## 1.4 Model

Here we provide a model for your. It consist of the function which you have implmeneted above

```python
class LogsticRegressionGD(object):

    def __init__(self, n_in, lr=0.05):
        super().__init__()
        self.lr = lr
        self.b = np.zeros(1, )
        self.W = np.random.randn(n_in, 1)

    def forward(self, x):
        self.h = linear_forward(x, self.W, self.b)
        y = sigmoid_forward(self.h)
        return y

    def update_params(self, x, nll_grad):
        # compute gradients
        grad_h = sigmoid_grad_input(self.h, nll_grad)
        grad_W = linear_grad_W(x, grad_h, self.W, self.b)
        grad_b = linear_grad_b(x, grad_h, self.W, self.b)
        # update params
        self.W = self.W - self.lr * grad_W
        self.b = self.b - self.lr * grad_b
```

## 1.5 Simple Experiment

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Generate some data
def generate_2_circles(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = 1.1 * np.array([np.sin(phi), np.cos(phi)])
    X2 = 3.0 * np.array([np.sin(phi), np.cos(phi)])
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.hstack([X1,X2]).T
    return X, Y

def generate_2_gaussians(N=100):
    phi = np.linspace(0.0, np.pi * 2, 100)
    X1 = np.random.normal(loc=[1, 2], scale=[2.5, 0.9], size=(N, 2))
    X1 = X1.dot(np.array([[0.7, -0.7], [0.7, 0.7]]))
    X2 = np.random.normal(loc=[-2, 0], scale=[1, 1.5], size=(N, 2))
    X2 = X2.dot(np.array([[0.7, 0.7], [-0.7, 0.7]]))
    Y = np.concatenate([np.ones(N), np.zeros(N)]).reshape((-1, 1))
    X = np.vstack([X1,X2])
    return X, Y

def split(X, Y, train_ratio=0.7):
    size = len(X)
    train_size = int(size * train_ratio)
    indices = np.arange(size)
    np.random.shuffle(indices)
    train_indices = indices[:train_size]
    test_indices = indices[train_size:]
    return X[train_indices], Y[train_indices], X[test_indices], Y[test_indices]

f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 4))

X, Y = generate_2_circles()
ax1.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax1.set_aspect('equal')

X, Y = generate_2_gaussians()
ax2.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'none')
ax2.set_aspect('equal')

```

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_gaussians(), 0.7)
```

```python
# let's train our model
model = LogsticRegressionGD(2, 0.05)

for step in range(30):
    Y_pred = model.forward(X_train)

    loss_value = nll_forward(Y_pred, Y_train)
    accuracy = ((Y_pred > 0.5) == Y_train).mean()
    print('Step: {} \t Loss: {:.3f} \t Acc: {:.1f}%'.format(step, loss_value, accuracy * 100))

    loss_grad = nll_grad_input(Y_pred, Y_train)
    model.update_params(X_train, loss_grad)

print('\n\nTesting...')
Y_test_pred = model.forward(X_test)
test_accuracy = ((Y_test_pred > 0.5) == Y_test).mean()
print('Acc: {:.1f}%'.format(test_accuracy * 100))
```

```python
def plot_model_prediction(prediction_func, X, Y, hard=True):
    u_min = X[:, 0].min()-1
    u_max = X[:, 0].max()+1
    v_min = X[:, 1].min()-1
    v_max = X[:, 1].max()+1

    U, V = np.meshgrid(np.linspace(u_min, u_max, 100), np.linspace(v_min, v_max, 100))
    UV = np.stack([U.ravel(), V.ravel()]).T
    c = prediction_func(UV).ravel()
    if hard:
        c = c > 0.5
    plt.scatter(UV[:,0], UV[:,1], c=c, edgecolors= 'none', alpha=0.15)
    plt.scatter(X[:,0], X[:,1], c=Y.ravel(), edgecolors= 'black')
    plt.xlim(left=u_min, right=u_max)
    plt.ylim(bottom=v_min, top=v_max)
    plt.axes().set_aspect('equal')
    plt.show()

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, False)

plot_model_prediction(lambda x: model.forward(x), X_train, Y_train, True)
```

```python
# Now run the same experiment on 2 circles
```

# 2. Decision Tree
The next model we look at is called **Decision Tree**. This type of model is non-parametric, meaning in contrast to **Logistic Regression** we do not have any parameters here that need to be trained.

Let us consider a simple binary decision tree for deciding on the two classes of "creditable" and "Not creditable".

Each node, except the leafs, asks a question about the the client in question. A decision is made by going from the root node to a leaf node, while considering the clients situation. The situation of the client, in this case, is fully described by the features:
1. Checking account balance
2. Duration of requested credit
3. Payment status of previous loan
4. Length of current employment

In order to build a decision tree we need training data. To carry on the previous example: we need a number of clients for which we know the properties 1.-4. and their creditability.
The process of building a decision tree starts with the root node and involves the following steps:
1. Choose a splitting criteria and add it to the current node.
2. Split the dataset at the current node into those that fullfil the criteria and those that do not.
3. Add a child node for each data split.
4. For each child node decide on either A. or B.:
    1. Repeat from 1. step
    2. Make it a leaf node: The predicted class label is decided by the majority vote over the training data in the current split.

## 2.1 Gini Index & Data Split
Deciding on how to split your training data at each node is dominated by the following two criterias:
1. Does the rule help me make a final decision?
2. Is the rule general enough such that it applies not only to my training data, but also to new unseen examples?

When considering our previous example, splitting the clients by their handedness would not help us deciding on their creditability. Knowning if a rule will generalize is usually a hard call to make, but in practice we rely on [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor) principle. Thus the less rules we use, the better we believe it to generalize to previously unseen examples.

One way to measure the quality of a rule is by the [**Gini Index**](https://en.wikipedia.org/wiki/Gini_coefficient).
Since we only consider binary classification, it is calculated by:
$$
Gini = \sum_{n\in\{L,R\}}\frac{|S_n|}{|S|}\left( 1 - \sum_{c \in C} p_{S_n}(c)^2\right)\\
p_{S_n}(c) = \frac{|\{\mathbf{x}_{i}\in \mathbf{X}|y_{i} = c, i \in S_n\}|}{|S_n|}, n \in \{L, R\}
$$
with $|C|=2$ being your set of class labels and $S_L$ and $S_R$ the two splits determined by the splitting criteria.
The lower the gini score, the better the split. In the extreme case, where all class labels are the same in each split respectively, the gini index takes the value of $0$.

```python
def tree_gini_index(Y_left, Y_right, classes):
    """Compute the Gini Index.
    # Arguments
        Y_left: class labels of the data left set
            np.array of size `(n_objects, 1)`
        Y_right: class labels of the data right set
            np.array of size `(n_objects, 1)`
        classes: list of all class values
    # Output
        gini: scalar `float`
    """
    gini = 0.0
    #################
    ### YOUR CODE ###
    #################
    return gini
```

```python
am.test_student_function(username, tree_gini_index, ['Y_left', 'Y_right', 'classes'])
```

At each node in the tree, the data is split according to a split criterion and each split is passed onto the left/right child respectively.
Implement the following function to return all rows in `X` and `Y` such that the left child gets all examples that are less than the split value and vice versa.

```python
def tree_split_data_left(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is less than `split_value` then return it as part of the left group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    X_left, Y_left = None, None
    #################
    ### YOUR CODE ###
    #################
    XY_left = np.concatenate([X_left, Y_left], axis=-1)
    return XY_left

def tree_split_data_right(X, Y, feature_index, split_value):
    """Split the data `X` and `Y`, at the feature indexed by `feature_index`.
    If the value is greater or equal than `split_value` then return it as part of the right group.

    # Arguments
        X: np.array of size `(n_objects, n_in)`
        Y: np.array of size `(n_objects, 1)`
        feature_index: index of the feature to split at
        split_value: value to split between
    # Output
        (XY_left): np.array of size `(n_objects_left, n_in + 1)`
    """
    X_right, Y_right = None, None
    #################
    ### YOUR CODE ###
    #################
    XY_right = np.concatenate([X_right, Y_right], axis=-1)
    return XY_right
```

```python
am.test_student_function(username, tree_split_data_left, ['X', 'Y', 'feature_index', 'split_value'])
```

```python
am.test_student_function(username, tree_split_data_right, ['X', 'Y', 'feature_index', 'split_value'])
```

```python
am.get_progress(username)
```

Now to find the split rule with the lowest gini score, we brute-force search over all features and values to split by.

```python
def tree_best_split(X, Y):
    class_values = list(set(Y.flatten().tolist()))
    r_index, r_value, r_score =  float("inf"),  float("inf"), float("inf")
    r_XY_left, r_XY_right = (X,Y), (X,Y)
    for feature_index in range(X.shape[1]):
        for row in X:
            XY_left = tree_split_data_left(X, Y, feature_index, row[feature_index])
            XY_right = tree_split_data_right(X, Y, feature_index, row[feature_index])
            XY_left, XY_right = (XY_left[:,:-1], XY_left[:,-1:]), (XY_right[:,:-1], XY_right[:,-1:])
            gini = tree_gini_index(XY_left[1], XY_right[1], class_values)
            if gini < r_score:
                r_index, r_value, r_score = feature_index, row[feature_index], gini
                r_XY_left, r_XY_right = XY_left, XY_right
    return {'index':r_index, 'value':r_value, 'XY_left': r_XY_left, 'XY_right':r_XY_right}
```

## 2.2 Terminal Node
The leaf nodes predict the label of an unseen example, by taking a majority vote over all training class labels in that node.

```python
def tree_to_terminal(Y):
    """The most frequent class label, out of the data points belonging to the leaf node,
    is selected as the predicted class.

    # Arguments
        Y: np.array of size `(n_objects)`

    # Output
        label: most frequent label of `Y.dtype`
    """
    label = None
    #################
    ### YOUR CODE ###
    #################
    return label
```

```python
am.test_student_function(username, tree_to_terminal, ['Y'])
```

```python
am.get_progress(username)
```

## 2.3 Build the Decision Tree
Now we recursively build the decision tree, by greedily splitting the data at each node according to the gini index.
To prevent the model from overfitting, we transform a node into a terminal/leaf node, if:
1. a maximum depth is reached.
2. the node does not reach a minimum number of training samples.

```python
def tree_recursive_split(X, Y, node, max_depth, min_size, depth):
    XY_left, XY_right = node['XY_left'], node['XY_right']
    del(node['XY_left'])
    del(node['XY_right'])
    # check for a no split
    if XY_left[0].size <= 0 or XY_right[0].size <= 0:
        node['left_child'] = node['right_child'] = tree_to_terminal(np.concatenate((XY_left[1], XY_right[1])))
        return
    # check for max depth
    if depth >= max_depth:
        node['left_child'], node['right_child'] = tree_to_terminal(XY_left[1]), tree_to_terminal(XY_right[1])
        return
    # process left child
    if XY_left[0].shape[0] <= min_size:
        node['left_child'] = tree_to_terminal(XY_left[1])
    else:
        node['left_child'] = tree_best_split(*XY_left)
        tree_recursive_split(X, Y, node['left_child'], max_depth, min_size, depth+1)
    # process right child
    if XY_right[0].shape[0] <= min_size:
        node['right_child'] = tree_to_terminal(XY_right[1])
    else:
        node['right_child'] = tree_best_split(*XY_right)
        tree_recursive_split(X, Y, node['right_child'], max_depth, min_size, depth+1)

def build_tree(X, Y, max_depth, min_size):
    root = tree_best_split(X, Y)
    tree_recursive_split(X, Y, root, max_depth, min_size, 1)
    return root
```

By printing the split criteria or the predicted class at each node, we can visualise the decising making process.
Both the tree and a a prediction can be implemented recursively, by going from the root to a leaf node.

```python
def print_tree(node, depth=0):
    if isinstance(node, dict):
        print('%s[X%d < %.3f]' % ((depth*' ', (node['index']+1), node['value'])))
        print_tree(node['left_child'], depth+1)
        print_tree(node['right_child'], depth+1)
    else:
        print('%s[%s]' % ((depth*' ', node)))

def tree_predict_single(x, node):
    if isinstance(node, dict):
        if x[node['index']] < node['value']:
            return tree_predict_single(x, node['left_child'])
        else:
            return tree_predict_single(x, node['right_child'])

    return node

def tree_predict_multi(X, node):
    Y = np.array([tree_predict_single(row, node) for row in X])
    return Y[:, None]  # size: (n_object,) -> (n_object, 1)
```

Let's test our decision tree model on some toy data.

```python
X_train, Y_train, X_test, Y_test = split(*generate_2_circles(), 0.7)

tree = build_tree(X_train, Y_train, 4, 1)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

We print the decision tree in [pre-order](https://en.wikipedia.org/wiki/Tree_traversal#Pre-order_(NLR)).

```python
print_tree(tree)
```

```python
plot_model_prediction(lambda x: tree_predict_multi(x, tree), X_test, Y_test)
```

# 3. Experiments
The [Cleveland Heart Disease](https://archive.ics.uci.edu/ml/datasets/Heart+Disease) dataset aims at predicting the presence of heart disease based on other available medical information of the patient.

Although the whole database contains 76 attributes, we focus on the following 14:
1. Age: age in years
2. Sex:
    * 0 = female
    * 1 = male
3. Chest pain type:
    * 1 = typical angina
    * 2 = atypical angina
    * 3 = non-anginal pain
    * 4 = asymptomatic
4. Trestbps: resting blood pressure in mm Hg on admission to the hospital
5. Chol: serum cholestoral in mg/dl
6. Fasting blood sugar: > 120 mg/dl
    * 0 = false
    * 1 = true
7. Resting electrocardiographic results:
    * 0 = normal
    * 1 = having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV)
    * 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria
8. Thalach: maximum heart rate achieved
9. Exercise induced angina:
    * 0 = no
    * 1 = yes
10. Oldpeak: ST depression induced by exercise relative to rest
11. Slope: the slope of the peak exercise ST segment
    * 1 = upsloping
    * 2 = flat
    * 3 = downsloping
12. Ca: number of major vessels (0-3) colored by flourosopy
13. Thal:
    * 3 = normal
    * 6 = fixed defect
    * 7 = reversable defect
14. Target: diagnosis of heart disease (angiographic disease status)
    * 0 = < 50% diameter narrowing
    * 1 = > 50% diameter narrowing

The 14. attribute is the target variable that we would like to predict based on the rest.

We have prepared some helper functions to download and pre-process the data in `heart_disease_data.py`

```python
import heart_disease_data
```

```python
X, Y = heart_disease_data.download_and_preprocess()
X_train, Y_train, X_test, Y_test = split(X, Y, 0.7)
```

Let's have a look at some examples

```python
print(X_train[0:2])
print(Y_train[0:2])

# TODO feel free to explore more examples and see if you can predict the presence of a heart disease
```

## 3.1 Decision Tree for Heart Disease Prediction
Let's build a decision tree model on the training data and see how well it performs

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance

tree = build_tree(X_train, Y_train, 5, 4)
Y_pred = tree_predict_multi(X_test, tree)
test_accuracy = (Y_pred == Y_test).mean()
print('Test Acc: {:.1f}%'.format(test_accuracy * 100))
```

How did changing the hyper parameters affect the test performance? Usually hyper parameters are tuned using a hold-out [validation set](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets#Validation_dataset) instead of the test set.

## 3.2 Logistic Regression for Heart Disease Prediction

Instead of manually going through the data to find possible correlations, let's try training a logistic regression model on the data.

```python
# TODO: you are free to make use of code that we provide in previous cells
# TODO: play around with different hyper parameters and see how these impact your performance
```

How well did your model perform? Was it actually better then guessing? Let's look at the empirical mean of the target.

```python
Y_train.mean()
```

So what is the problem? Let's have a look at the learned parameters of our model.

```python
print(model.W, model.b)
```

If you trained sufficiently many steps you'll probably see how some weights are much larger than others. Have a look at what range the parameters were initialized and how much change we allow per step (learning rate). Compare this to the scale of the input features. Here an important concept arises, when we want to train on real world data:
[Feature Scaling](https://en.wikipedia.org/wiki/Feature_scaling).

Let's try applying it on our data and see how it affects our performance.

```python
# TODO: Rescale the input features and train again
```

Notice that we did not need any rescaling for the decision tree. Can you think of why?


# How do micromorts add up?

```python
import numpy as np
import matplotlib.pyplot as plt
```

Micromorts are a convenient way to measure the dealiness of human activity.
They are defined as the amount of activity resulting in a 1:1 000 000 chance of dying.

These low probablities can be added up, due to the taylor expansion

$\begin{align}
p_{not\_dead} &= (1 - p)^N \\
              &=  1 - N \cdot (1 -0) \cdot p + O(p^2)\\
              &\approx 1 - Np
\end{align}$

But this is clearly not valid for $N\approx \frac{1}{p}$, as $p_{not\_dead} = 0$ in this case.

So, lets simulate some numbers!

```python
n_long = np.logspace(1, 6.5, 100)
p_not_death = np.power(1 - 10**-6, n_long)
plt.plot(n_long, p_not_death, label='exact')

n = np.logspace(1, 6, 100)
taylored_not_death = 1 - n * 10**-6
plt.plot(n, taylored_not_death, label='taylored')
plt.legend()
plt.show()
```

As we can see, the exact probablity to zero asymptotically, and is about zero at $N\approx \frac{3}{p}$, while the taylored formula is exactly zero at $N = \frac{1}{p}$.

But for which quantities of N can we use the first-order taylored formula?

```python
n = np.logspace(1, 5.7, 1000)
p_not_death = np.power(1 - 10**-6, n)
plt.plot(n, p_not_death, label='exact')

taylored_not_death = 1 - n * 10**-6
plt.plot(n, taylored_not_death, label='taylored')
plt.legend()
plt.show()
```

It's probably safe for $N<\frac{1}{p}/10$, and roughly correct for $N<\frac{1}{p}/5$, 100 000 and 500 000 for the micromorts in this case.

What if we add up bigger lumps of micromorts?

```python
n_long = np.logspace(1, 4.5, 100)
p_not_death = np.power(1 - 10**-4, n_long)
plt.plot(n_long, p_not_death, label='exact')

n = np.logspace(1, 4, 100)
taylored_not_death = 1 - n * 10**-4
plt.plot(n, taylored_not_death, label='taylored')
plt.legend()

plt.figure()
n = np.logspace(1, 3.7, 1000)
p_not_death = np.power(1 - 10**-4, n)
plt.plot(n, p_not_death, label='exact')

taylored_not_death = 1 - n * 10**-4
plt.plot(n, taylored_not_death, label='taylored')
plt.figure()
plt.show()
```

For bigger lumps, the results are the same, just shifted downwards - i.e., we can add up 1000 doses of 100 micromorts safely.


# Solving ODEs with the Euler integrator

**Ordinary differential equations** ([ODE](http://mathworld.wolfram.com/OrdinaryDifferentialEquation.html)s) describe many phenomena in physics. They describe the changes of a **dependent variable** $y(t)$ as a function of a **single independent variable** (e.g. $t$ or $x$).

- An ODE of order $n$ contains $\frac{d^n y}{dt^n}$ as the highest derivative.

  For example, **Newton's equations of motion**

  $$
  F = m \frac{d^2 x(t)}{dt^2}
  $$

  are second order ODEs.

- An ODE of order $n$ requires $n$ initial conditions to uniquely determine a solution $y(t)$.
  For Newton: we need initial position $x(t=0)$ and velocity $v(t=0)$.

- Linear ODEs contain no higher powers than 1 of any of the derivatives (including the 0-th derivative $y$ term).

- Non-linear ODEs can contain any powers in the dependent variable and its derivatives.

## Integrating ODEs with Euler's algorithm
First order ODE:

$$
\frac{dy}{dt} = f(t, y)
$$

Basic idea:
1. Start with initial conditions, $y_0 \equiv y(t=0)$
2. Use $\frac{dy}{dt} = f(t, y)$ (the RHS!) to advance solution a small step $h$ forward in time: $y(t=h) \equiv y_1$
3. Repeat with $y_1$ to obtain $y_2 \equiv y(t=2h)$... and for all future values of $t$.

### Euler's algorithm
Use the forward difference approximation for the derivative:

$$
f(t, y) = \frac{dy(t)}{dt} \approx \frac{y(t_{n+1}) - y(t_n)}{h}
$$

Solve for the position in the future $y(t_{n+1})$, based on present *and known* values $y(t_n)$ and  $f\big(t_n, y(t_n)\big)$:

$$
y_{n+1} \approx y_n + h f(t_n, y_n) \quad \text{with} \quad y_n := y(t_n)
$$

### Convert 2nd order ODE to 2 coupled 1st order ODEs
The 2nd order ODE is
$$
\frac{d^2 y}{dt^2} = f(t, y)
$$

Introduce "dummy" dependent variables $y_i$ with $y_0 \equiv y$ and

\begin{alignat}{1}
\frac{dy}{dt} &= \frac{dy_0}{dt} &=   y_1\\
\frac{d^2y}{dt^2} &= \frac{dy_1}{dt} &= {} f(t, y_0).
\end{alignat}

The first equation defines the velocity $y_1 = v$ and the second one is the original ODE.

## Bouncing ball

Problem: Integrate the equations of a bouncing ball under gravity
* Drop from height $y_0 = 2$ within initial velocity $v_0 = 0$.
* The ball bounces elastically off the ground at $y=0$.

We have to solve the *second order ODE* (Newton's equations of motion with constant acceleration)

$$
\frac{d^2 y}{dt^2} = -g.
$$

The Euler scheme for any *first order ODE*

$$
\frac{dy}{dt} = f(y, t)
$$

is
$$
y(t + h) = y(t) + h f(y(t), t).
$$

In order to solve the original 2nd order equation of motion we make use of the fact that one $n$-th order ODE can be written as $n$ coupled first order ODEs, namely

\begin{align}
\frac{dy}{dt} &= v\\
\frac{dv}{dt} &= -g.
\end{align}

Solve each of the first order ODEs with Euler:

\begin{align}
y(t + h) &= y(t) + h v(t)\\
v(t + h) &= v(t) - h g.
\end{align}

### Free fall

Start with free fall as an even simpler problem.

```python
import numpy as np
```

```python
# parameters
g = -9.81
# initial conditions
y = 2.0
v = 0.0

t = 0
dt = 0.01

# record initial conditions
data = [[t, y, v]]

# start at first step
t = dt
while t < 10:
    y = y + v*dt
    v = v + g*dt
    data.append([t, y, v])
    t += dt

data = np.array(data)
```

```python
data.shape
```

    (1001, 3)

Look at the first few values:

```python
data[:4]
```

    array([[ 0.      ,  2.      ,  0.      ],
           [ 0.01    ,  2.      , -0.0981  ],
           [ 0.02    ,  1.999019, -0.1962  ],
           [ 0.03    ,  1.997057, -0.2943  ]])

To make it more convenient to get `t = data[:, 0]`, `y = data[:, 1]`, and `v = data[:, 2]` we use transposition to change the array from time x coordinates to coordinates x time and then use tuple assignment:

```python
data = data.transpose()
data.shape  # t, y, v
```

    (3, 1001)

```python
t, y, v = data
```

Plot the trajectory $y(t)$ with matplotlib.

(Using `t = data[0]` for time and `y = data[1]` for position, after the transposition!)

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.plot(data[0], data[1])
plt.xlabel("time (s)")
plt.ylabel("position (m)");
```

### Bouncing
Add a floor at $y = 0$.

What happens at the floor? – The velocity changes (elastic collision).

```python
# parameters
g = -9.81
y_floor = 0

# initial conditions
y = 2.0
v = 0.0

t = 0
dt = 0.01

# record initial conditions
data = [[t, y, v]]

# start at first step
t = dt
while t < 10:
    y += v*dt
    if y > y_floor:
        v += g*dt
    else:
        v = -v   # bounce off floor
    data.append([t, y, v])
    t += dt

data = np.array(data).transpose()
```

```python
plt.plot(data[0], data[1])
plt.xlabel("time (s)")
plt.ylabel("position (m)");
```

## Summary: Euler integrator

1. If the order of the ODE > 1 then write the ODE as a coupled system of n first order ODEs.

   For Newton's EOM ($F = m\frac{d^2}{dt^2}$):

   \begin{align}
   \frac{dx}{dt} &= v\\
   \frac{dv}{dt} &= m^{-1}F
   \end{align}

   Note that $F$ typically depends on $x$, e.g., $F(x) = -\frac{\partial U}{\partial x}$ when the force can be derived from a potential energy function $U(x)$.
2. Solve all first order ODEs with the forward Euler algorithm for time step $\Delta t$:

   \begin{align}
   x_{t+1} &= x_t + v \Delta t\\
   v_{t+1} &= v_t + m^{-1} F(x_t) \Delta t
   \end{align}

The time step of the Euler algorithm has to be chosen small because the error in $x(t)$ will go like $\Delta t^2$ – Euler is really a terrible algorithm but for this introductory class it is good enough. Better algorithms exist and are not much more difficult (see, e.g., PHY432 Computational Methods).


# 4.1 Introduction

- Our predictor G(x) takes values in a discrete set $\mathcal{G}$.

- We can divide the input space into regions labeled according to the classification.

- Decision boundaries of regions are linear; this is what we'll mean by *linear methods for classification*.

Suppose there are K classes and the fitted linear model: $\hat{f}_k(x)=\hat{\beta}_{k0} + \hat{\beta}_k^Tx$; then the decision boundary between class *k* and *l* is $\hat{f}_k(x)=\hat{f}_l(x)$ that is  $\{ x : (\hat{\beta}_{k0}-\hat{\beta}_{l0}) + (\hat{\beta}_{k}-\hat{\beta}_{l})^Tx = 0 \}$. This regression approach is a member of a class of methods that model *discriminant functions $\delta_k(x)$* for each class, and then classify x to the class with the largest value for its discriminant function. Methods that model the posterior probabilities Pr(G = k | X = x) are also in this class. Clearly, if either the $\delta_k(x)$ or Pr(G = k| X = x) are linear in x, then the decision boundaries will be linear.

For example, a popular model for the posterior probabilities for two classes are (4.1):

$$
\begin{equation}
Pr(G = 1|X=x)=\cfrac{exp(\beta_0 + \beta^Tx)}{1+exp(\beta_0 + \beta^Tx)}\\
Pr(G = 2|X=x)=\cfrac{1}{1+exp(\beta_0 + \beta^Tx)}
\end{equation}
$$

Here the monotone transformation is the *logit* (or *log-odds*) transformation: log[p/(1-p)], in fact (4.2):

$$
log \frac{Pr(G = 1|X=x)}{Pr(G = 2|X=x)} = \beta_0 + \beta^Tx
$$

The decision boundary defined by $\{x : \beta_0 + \beta^Tx = 0\}$