---
# Section 1.4: Positive Definite Systems; Cholesky Decomposition
---

After triangular linear systems, the next easiest type of linear system to solve is the **symmetric positive definite** linear system.

A matrix $A \in \mathbb{R}^{n \times n}$ is **symmetric** if $A = A^T$.

For example,

$$
A =
\begin{bmatrix}
1 & 2 & 3 \\
2 & 4 & 5 \\
3 & 5 & 6
\end{bmatrix}
$$

is a symmetric matrix.

A matrix $A \in \mathbb{R}^{n \times n}$ is called **positive definite** if $A$ is symmetric and

$$
x^T A x > 0, \qquad \text{for all nonzero $x \in \mathbb{R}^n$.}
$$

---

### Exercise:

Let $A = \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}$ and $x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \neq 0$. Prove that $x^T A x > 0$.

```julia
A = [ 2 -1; -1 2 ]
```

```julia
x = randn(2)
@show x
@show x'*A*x;
```

Let $x$ be a nonzero vector. Then,

\begin{eqnarray}
x^T A x
&=&
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix}^T
\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \\
&=&
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix}^T
\begin{bmatrix} 2x_1 - x_2 \\ -x_1 + 2x_2 \end{bmatrix} \\
&=& x_1(2x_1 - x_2) + x_2(-x_1 + 2x_2) \\
&=& 2x_1^2 - 2 x_1 x_2 + 2x_2^2 \\
&=& x_1^2 + (x_1^2 - 2 x_1 x_2 + x_2^2) + x_2^2 \\
&=& x_1^2 + (x_1 - x_2)^2 + x_2^2 \\
&\ge& x_1^2 + x_2^2 \\
&>& 0,
\end{eqnarray}

since $x$ is a nonzero vector. Therefore, the symmetric matrix $A$ is positive definite.

```julia
x'*A*x
```

```julia
x[1]^2 + (x[1] - x[2])^2 + x[2]^2
```

---

## Properties of positive definite matrices

(1) If $A$ is positive definite, then $A$ is nonsingular.

(2) If $A$ is positive definite, then $Ax = b$ has a unique solution.

(3) If $A = M^TM$ for some nonsingular $M \in \mathbb{R}^{n \times n}$, then $A$ is positive definite.

(4) $A$ is positive definite $\Longleftrightarrow$ all eigenvalues of $A$ are positive.

```julia
using LinearAlgebra
```

```julia
eigvals(A)
```

---
### Exercise:

Prove property (1) and deduce property (2).

#### Proof:

Let $A$ be a symmetric positive definite matrix. Suppose, for the sake of contradiction, that $A$ is singular. Then there is a nonzero vector $x$ such that $Ax = 0$. So we have that $x^T A x = x^T 0 = 0$, which contradicts our assumption that $A$ is positive definite. Therefore, $A$ is nonsingular.

Since $A$ is nonsingular, any linear system $Ax = b$ has a unique solution. $\blacksquare$

---
### Exercise:

Based on property (3), write a `Julia` function `A = randspd(n)` that randomly generates an $n \times n$ symmetric positive definite matrix $A$. Check that the matrix $A$ is positive definite by using property (4).

```julia
function randspd(n)
    M = randn(n,n)
    A = M'M
end
```

```julia
A = randspd(5)
```

```julia
eigvals(A)
```

```julia
n = 5
M = randn(n,n)
eigvals(M)
```

---

## The Cholesky Decomposition

It turns out that every positive definite matrix $A$ can be written as $A = M^TM$, for some nonsingular matrix $M$.

> ### Cholesky Decomposition Theorem
> Let $A$ be positive definite.
> Then there is a **unique upper-triangular matrix** $R$ with **positive diagonal entries** such that
> $$A = R^T R.$$

We will see a proof of this theorem later.

---

### Exercise:

Computing the Cholesky factor of a random positive definite matrix $A$ using the `Julia` function `cholesky`. Check that the output is correct.

```julia
?cholesky
```

```julia
n = 5
A = randspd(n)
```

```julia
F = cholesky(A)
```

```julia
R = F.U
```

```julia
A - R'R
```

---

### Exercise:

The function `cholesky` will detect if $A$ is not positive definite. In fact, this is the **most efficient** method for testing if a symmetric matrix is positive definite or not.

See what happens when you give the `cholesky` function a matrix that is symmetric but not positive definite.

```julia
n = 5
M = randn(n,n)
A = M + M'
```

```julia
F = cholesky(A)
```

```julia
eigvals(A)
```

```julia
isposdef(A)
```

```julia
isposdef(randspd(n))
```

---

### Exercise:

Let $R$ be the Cholesky factor of a positive definite matrix $A$.

1. Propose an algorithm for solving $Ax = b$ using the Cholesky factorization $A = R^TR$. Your algorithm should have **two** steps, a forward-substitution and a backward-substitution.

2. What is the flop count of your algorithm?

3. Implement your algorithm in `Julia`. Check that the output is correct.

#### Part 1

Suppose $A$ is symmetric positive definite and that $A = R^TR$ is its Cholesky decomposition. Suppose also that $Ax = b$. Then

$$
R^T R x = b.
$$

Let $z = R x$. Then $R^T z = b$ is a lower triangular linear system, and since the diagonal entries of $R$ are all positive, the matrix $R^T$ is nonsingular. So our algorithm is:

1. Solve $R^T z = b$ for the vector $z$ using forward substitution.
2. Solve $R x = z$ for the vector $x$ using backward substitution.

#### Part 2

Suppose that $A$ is $n \times n$. Then both $R$ and $R^T$ are $n \times n$. Then we have:

1. Solving $R^T z = b$ requires $n^2$ flops.
2. Solving $R x = z$ requires $n^2$ flops.

So, in total, solving $A x = b$ given the Cholesky decomposition of $A$ requires $2 n^2$ flops. Note this is the same flop count as multiplying a $n \times n$ matrix times a vector.

### Part 3

```julia
n = 5
A = randspd(n)
xtrue = randn(n)
b = A*xtrue

F = cholesky(A)
R = F.U

# Step 1 (forward substitution)
z = R'\b

# Step 2 (backward substitution)
x = R\z
```

```julia
x - xtrue
```

---
### Exercise:

Compute the Cholesky factor of $A$ by hand by writing $A = R^T R$ as follows.

$$
\begin{bmatrix}
 4 & -2 &  4 \\
-2 & 10 & -2 \\
 4 & -2 &  8 \\
\end{bmatrix} =
\begin{bmatrix}
r_{11} \\
r_{12} & r_{22} \\
r_{13} & r_{23} & r_{33} \\
\end{bmatrix}
\begin{bmatrix}
r_{11} & r_{12} & r_{13} \\
       & r_{22} & r_{23} \\
       &        & r_{33} \\
\end{bmatrix}
$$

Multiplying the two matrices on the rhs gives us

$$
\begin{bmatrix}
 4 & -2 &  4 \\
-2 & 10 & -2 \\
 4 & -2 &  8 \\
\end{bmatrix} =
\begin{bmatrix}
r_{11}^2 & r_{11} r_{12} & r_{11} r_{13} \\
r_{11} r_{12} & r_{12}^2 + r_{22}^2 & r_{12} r_{13} + r_{22} r_{23} \\
r_{11} r_{13} & r_{12} r_{13} + r_{22} r_{23} & r_{13}^2 + r_{23}^2 + r_{33}^2
\end{bmatrix}.
$$

Then we have:

1. $r_{11}^2 = 4 \implies r_{11} = 2$
2. $r_{11} r_{12} = -2 \implies 2 r_{12} = -2 \implies r_{12} = -1$
3. $r_{11} r_{13} = 4 \implies 2 r_{13} = 4 \implies r_{13} = 2$
4. $r_{12}^2 + r_{22}^2 = 10 \implies 1 + r_{22}^2 = 10 \implies r_{22} = 3$
5. $r_{12} r_{13} + r_{22} r_{23} = -2 \implies -2 + 3 r_{23} = -2 \implies r_{23} = 0$
6. $r_{13}^2 + r_{23}^2 + r_{33}^2 = 8 \implies 4 + 0 + r_{33}^2 = 8 \implies r_{33} = 2$

Therefore,

$$
R = \begin{bmatrix}
2 & -1 & 2 \\
  &  3 & 0 \\
  &    & 2
  \end{bmatrix}.
  $$

```julia
R = [ 2 -1 2; 0 3 0; 0 0 2]
```

```julia
R'R
```

```julia
A = [
    4 -2 4
    -2 10 -2
    4 -2 8 ]
```

---

### General Inner-Product Cholesky Formula

The general formulas for the entries $r_{ij}$ of the Cholesky factor $R$ of a positive definite matrix $A$ are given by

$$
\begin{align}
r_{ii} &= \sqrt{a_{ii} - \sum_{k=1}^{i-1} r_{ki}^2}, \\
r_{ij} &= \left(a_{ij} - \sum_{k=1}^{i-1} r_{ki} r_{kj}\right)\bigg/r_{ii}, \qquad j = i+1, \ldots, n. \\
\end{align}
$$

#### Example
$$
\begin{bmatrix}
 4 & -2 &  4 \\
-2 & 10 & -2 \\
 4 & -2 &  8 \\
\end{bmatrix} =
\begin{bmatrix}
r_{11}^2 & r_{11} r_{12} & r_{11} r_{13} \\
r_{11} r_{12} & r_{12}^2 + r_{22}^2 & r_{12} r_{13} + r_{22} r_{23} \\
r_{11} r_{13} & r_{12} r_{13} + r_{22} r_{23} & r_{13}^2 + r_{23}^2 + r_{33}^2
\end{bmatrix}.
$$

```julia
R = zeros(size(A))

# i = 1
R[1,1] = sqrt(A[1,1])
# j = 2
R[1,2] = (A[1,2])/R[1,1]
# j = 3
R[1,3] = (A[1,3])/R[1,1]

# i = 2
R[2,2] = sqrt(A[2,2] - R[1,2]^2)
# j = 3
R[2,3] = (A[2,3] - R[1,2]*R[1,3])/R[2,2]

# i = 3
R[3,3] = sqrt(A[3,3] - R[1,3]^2 - R[2,3]^2)

R
```

```julia
A - R'R
```

---

### Exercise:

Complete the following `Julia` function for the inner-product version of Cholesky's Algorithm.

```julia
function chol_ip(A::Symmetric)

    # Initialize R to be the upper-triangular part of A
    R = UpperTriangular(Matrix(A))

    n = size(A, 1)

    for i = 1:n
        # R[i,i] = sqrt(A[i,i] - sum_{k=1}^{i-1} R[k,i]^2)
        tmp = A[i,i] - sum([R[k,i]^2 for k=1:i-1])
        if tmp <= 0.0
            error("Matrix is not positive definite.")
        end
        R[i,i] = sqrt(tmp)

        for j = i+1:n
            # R[i,j] = (A[i,j] - sum_{k=1}^{i-1} R[k,i]*R[k,j]) / R[i,i]
            R[i,j] = (A[i,j] - sum([R[k,i]*R[k,j] for k=1:i-1]))/R[i,i]
        end
    end

    return R
end
```

```julia
A = Float64.(A)
```

```julia
A = Symmetric(A)
```

```julia
R = UpperTriangular(Matrix(A))
```

```julia
R = chol_ip(A)
```

```julia
B = copy(A)
```

```julia
B[1,1] = 3
```

```julia
chol_ip(B)
```

```julia
A = Symmetric(randn(3,3))
eigvals(A)
```

```julia
chol_ip(A)
```

---

### Exercise:

Develop the **Outer-Product Form of Cholesky's Algorithm** by partitioning $A = R^TR$ as:

$$
\begin{bmatrix}
a_{11} & b^T \\
b & \hat{A} \\
\end{bmatrix} =
\begin{bmatrix}
r_{11} & 0 \\
s & \hat{R}^T \\
\end{bmatrix}
\begin{bmatrix}
r_{11} & s^T \\
0 & \hat{R} \\
\end{bmatrix}.
$$

1. Use block-matrix multiplication to multiply the matrices $R^T$ and $R$.
2. Based on the equations from part (1), derive a recursive algorithm for computing $R$.

\begin{eqnarray}
\begin{bmatrix}
a_{11} & b^T \\
b & \hat{A} \\
\end{bmatrix} &=&
\begin{bmatrix}
r_{11}^2 & r_{11} s^T + 0 \hat{R} \\
s r_{11} + \hat{R}^T 0 & s s^T + \hat{R}^T \hat{R}
\end{bmatrix} \\
&=&
\begin{bmatrix}
r_{11}^2 & r_{11} s^T  \\
r_{11} s & s s^T + \hat{R}^T \hat{R}
\end{bmatrix} \\
\end{eqnarray}

1. $r_{11}^2 = a_{11} \implies r_{11} = \sqrt{a_{11}}$
2. $r_{11} s^T = b^T \implies s^T = b^T/r_{11}$
3. $s s^T + \hat{R}^T \hat{R} = \hat{A} \implies \hat{R}^T \hat{R} = \hat{A} - s s^T$

---

### Exercise:

Complete the following `Julia` function for the recursive version of Cholesky's Algorithm.

```julia
function chol_rop(A)

    if !issymmetric(A)
        error("Matrix must be symmetric.")
    end
    n = size(A, 1)

    # Initialize R = 0 and the same size and datatype as A
    R = UpperTriangular(zeros(n, n))

    if A[1,1] <= 0.0
        error("Matrix is not positive definite")
    end

    R[1,1] = sqrt(A[1,1])
    if n > 1
        b = A[2:n,1]
        s = b/R[1,1]

        R[1,2:n] = s

        Â = A[2:n,2:n]
        R̂ = chol_rop(Â - s*s')

        R[2:n,2:n] = R̂
    end

    return R
end
```

```julia
n = 4
A = randspd(n)

R = chol_rop(A)
```

```julia
A - R'R
```

```julia
A = Symmetric(randn(n,n))
eigvals(A)
```

```julia
R = chol_rop(A)
```

---

### Exercise:

Develop the **Bordered Form of Cholesky's Algorithm** by partitioning $A = R^TR$ as:

$$
\begin{bmatrix}
\hat{A} & c \\
c^T & a_{nn} \\
\end{bmatrix} =
\begin{bmatrix}
\hat{R}^T & 0 \\
h^T & r_{nn} \\
\end{bmatrix}
\begin{bmatrix}
\hat{R} & h \\
0 & r_{nn} \\
\end{bmatrix}
$$

1. Use block-matrix multiplication to multiply the matrices $R^T$ and $R$.
2. Based on the equations from part (1), derive a recursive algorithm for computing $R$.
3. Based on your recursive algorithm, determine the number of flops for computing the Cholesky decomposition of an $n \times n$ positive definite matrix $A$.

#### Part 1

$$
\begin{bmatrix}
\hat{A} & c \\
c^T & a_{nn} \\
\end{bmatrix} =
\begin{bmatrix}
\hat{R}^T \hat{R} & \hat{R}^T h \\
h^T \hat{R} & h^T h + r_{nn}^2
\end{bmatrix}
$$

#### Part 2

1. $\hat{R}^T \hat{R} = \hat{A}$ (recursive call)
2. Solve $\hat{R}^T h = c$ for $h$ (forward substitution)
3. $h^T h + r_{nn}^2 = a_{nn} \implies r_{nn} = \sqrt{a_{nn} - h^T h}$

#### Part 3

Let $T_n$ be the number of flops to compute the Cholesky factor of an $n \times n$ symmetric positive definite matrix.

1. Requires $T_{n-1}$ flops.
2. Requires $(n-1)^2$ flops.
3. Computing $a_{nn} - h^T h$ requires $2(n-1)$ flops. Plus one more flop for the square root.

Therefore,

$$T_{n} = T_{n-1} + (n-1)^2 + 2(n-1) + 1$$

and $T_1 = 1$ (or $T_0 = 0$).

First note that

$$T_{n} = T_{n-1} + n^2 - 2n + 1 + 2n - 2 + 1.$$

Thus,

$$T_{n} = T_{n-1} + n^2.$$

Then

\begin{align}
T_{n}
&= T_{n-1} + n^2 \\
&= T_{n-2} + (n-1)^2 + n^2 \\
&= T_{n-3} + (n-2)^2 + (n-1)^2 + n^2 \\
&\vdots \\
&= T_0 + (1)^2 + \cdots + (n-2)^2 + (n-1)^2 + n^2 \\
&= 1 + 2^2 + \cdots + (n-1)^2 + n^2 \\
&= \sum_{k=1}^n k^2 \\
&= \frac{n(n+1)(2n+1)}{6} \\
&= \frac{1}{3} n^3 + O(n^2).
\end{align}

---

## Proof of the Cholesky Decomposition Theorem

> ### Cholesky Decomposition Theorem
> Let $n$ be a positive integer. If $A$ is an $n \times n$ positive definite matrix, then there is a **unique $n \times n$ upper-triangular matrix** $R$ with **positive diagonal entries** such that
>
> $$A = R^T R.$$

We will prove this theorem using mathematical induction, based on the following partitioning.

$$
\begin{bmatrix}
\hat{A} & c \\
c^T & a_{nn} \\
\end{bmatrix}
=
\begin{bmatrix}
\hat{R}^T & 0 \\
h^T & r_{nn} \\
\end{bmatrix}
\begin{bmatrix}
\hat{R} & h \\
0 & r_{nn} \\
\end{bmatrix}
$$

First we need to prove the following lemma.

> ### Lemma
> Let $A$ be positive definite and partitioned as
>
> $$
A =
\begin{bmatrix}
A_{11} & A_{12} \\
A_{21} & A_{22}
\end{bmatrix}
$$
>
>where $A_{11} \in \mathbb{R}^{n_1 \times n_1}$ and $A_{22} \in \mathbb{R}^{n_2 \times n_2}$. Then $A_{11}$ and $A_{22}$ are positive definite.

---
### Exercise:

Prove the lemma.

#### Proof.

Suppose, for the sake of contradiction, that $A_{11}$ is **not** positive definite. Then there is a nonzero vector $y \in \mathbb{R}^{n_1}$ such that $y^T A_{11} y \le 0$. Let

$$
x = \begin{bmatrix} y \\ 0 \end{bmatrix} \in \mathbb{R}^n.
$$

Since $y \ne 0$, we have that $x \ne 0$. Then,

$$
x^T A x =
\begin{bmatrix} y \\ 0 \end{bmatrix}^T
\begin{bmatrix}
A_{11} & A_{12} \\
A_{21} & A_{22}
\end{bmatrix}
\begin{bmatrix} y \\ 0 \end{bmatrix} =
\begin{bmatrix} y \\ 0 \end{bmatrix}^T
\begin{bmatrix} A_{11} y \\ A_{21} y \end{bmatrix} =
y^T A_{11} y \le 0.
$$

Thus, $A$ is **not** positive definite. This contradicts our assumption that $A$ is positive definite. Therefore, it is not possible for $A_{11}$ to not be positive definite. So, $A_{11}$ is positive definite.

Similarly, $A_{22}$ can be shown to be positive definite as well. $\blacksquare$

---
### Exercise:

Use the following steps to prove the theorem.

1. Prove that the theorem is true for $n = 1$. This is the _base case_.

2. Suppose that the theorem is true for $n = k$. This is the _induction hypothesis_.

3. Let $n = k+1$ and let $A$ be an $n \times n$ positive definite matrix.

4. Show that there is a unique $n \times n$ upper-triangular matrix $R$ with positive diagonal entries such that $A = R^T R$ by considering the following partitioning of $A = R^T R$,

    $$
    \begin{bmatrix}
    \hat{A} & c \\
    c^T & a_{nn} \\
    \end{bmatrix} =
    \begin{bmatrix}
    \hat{R}^T & 0 \\
    h^T & r_{nn} \\
    \end{bmatrix}
    \begin{bmatrix}
    \hat{R} & h \\
    0 & r_{nn} \\
    \end{bmatrix},
    $$

    and showing that there exist unique $\hat{R}$, $h$, and $r_{nn}$ that satisfy this equation.

### Proof.

#### Step 1: (base case)

Let $A = [a_{11}]$ be a $1 \times 1$ positive definite matrix. Let $x = [1]$, then $x^T A x = a_{11}$. Thus, $a_{11}$ must be a positive number.

Let $r_{11} = \sqrt{a_{11}}$, and let $R = [r_{11}]$. Then $R$ is an upper-triangular matrix with positive diagonal entries and $A = R^T R$.

Moreover, if $S = [s_{11}]$ with $s_{11} > 0$ and $A = S^T S$, then $a_{11} = s_{11}^2$. But then $s_{11} = \sqrt{a_{11}} = r_{11}$, so $S = R$.

#### Step 2: (induction hypothesis)

We now assume that if $A$ is $k \times k$ and symmetric positive definite, then there exists a unique upper-triangular matrix $R$ with positive diagonal entries such that $A = R^T R$.

#### Step 3: (start of induction step)

Let $n = k+1$ and let $A$ be a $n \times n$ symmetric positive definite matrix.

#### Step 4:

If $A = R^T R$, then we have

$$
\begin{bmatrix}
\hat{A} & c \\
c^T & a_{nn} \\
\end{bmatrix} =
\begin{bmatrix}
\hat{R}^T & 0 \\
h^T & r_{nn} \\
\end{bmatrix}
\begin{bmatrix}
\hat{R} & h \\
0 & r_{nn} \\
\end{bmatrix}
$$

where $\hat{A}$ and $\hat{R}$ are $k \times k$ matrices, $c$ and $h$ are vectors of length $k$, and $a_{nn}$ and $r_{nn}$ are real numbers.

Then

$$
\begin{bmatrix}
\hat{A} & c \\
c^T & a_{nn} \\
\end{bmatrix} =
\begin{bmatrix}
\hat{R}^T \hat{R} & \hat{R}^T h \\
h^T \hat{R} & h^T h + r_{nn}^2
\end{bmatrix}.
$$

This implies that

1. $\hat{A} = \hat{R}^T \hat{R}$,
2. $\hat{R}^T h = c$, and
3. $a_{nn} = h^T h + r_{nn}^2$.

Note that $\hat{A}$ is a $k \times k$ symmetric matrix and by the lemma $\hat{A}$ is also positive definite.

Therefore, by the induction hypothesis, there is a unique upper-triangular matrix $\hat{R}$ with positive diagonal entries such that $\hat{A} = \hat{R}^T \hat{R}$.

Since $\hat{R}$ is upper-triangular with positive diagonal entries, we know that it is nonsingular (i.e., invertible) since $\det(\hat{R})$ is the product of the diagonal entries of $\hat{R}$, so $\det(\hat{R}) \ne 0$.

Therefore, $\hat{R}^T h = c$ has a unique solution $h$ and

$$h = \hat{R}^{-T} c.$$

Next, we need to have $r_{nn}^2 = a_{nn} - h^T h$ and $r_{nn} > 0$. That means that we need to show that

$$a_{nn} - h^T h > 0.$$

To do this, we will use the fact that $x^T A x > 0$ for all nonzero vectors $x$.

First note that

\begin{align}
a_{nn} - h^T h
&= a_{nn} - \left(\hat{R}^{-T} c\right)^T \left(\hat{R}^{-T} c\right) \\
&= a_{nn} - c^T \hat{R}^{-1} \hat{R}^{-T} c \\
&= a_{nn} - c^T \left( \hat{R}^{T} \hat{R} \right)^{-1} c \\
&= a_{nn} - c^T \hat{A}^{-1} c. \\
\end{align}

We want to find a nonzero vector $x$ such that $x^T A x = a_{nn} - c^T \hat{A}^{-1} c$. To do this, we observe that

\begin{align}
x^T A x &=
\begin{bmatrix}
\hat{x} \\ x_n
\end{bmatrix}^T
\begin{bmatrix}
\hat{A} & c \\
c^T & a_{nn} \\
\end{bmatrix}
\begin{bmatrix}
\hat{x} \\ x_n
\end{bmatrix} \\
&=
\begin{bmatrix}
\hat{x} \\ x_n
\end{bmatrix}^T
\begin{bmatrix}
\hat{A}\hat{x} + c x_n \\
c^T \hat{x} + a_{nn} x_n \\
\end{bmatrix} \\
&=
\hat{x}^T \hat{A} \hat{x} + \hat{x}^T c x_n +
x_n c^T \hat{x} + a_{nn} x_n^2 \\
&=
\hat{x}^T \hat{A} \hat{x} + 2 x_n c^T \hat{x} + a_{nn} x_n^2.
\end{align}

Let $x_n = 1$. Then

$$
x^T A x = \hat{x}^T \hat{A} \hat{x} + 2 c^T \hat{x} + a_{nn}.
$$

Let $\hat{x} = -\hat{A}^{-1} c.$ Then

\begin{align}
x^T A x
&= c^T \hat{A}^{-1} \hat{A} \hat{A}^{-1} c - 2 c^T \hat{A}^{-1} c + a_{nn} \\
&= c^T \hat{A}^{-1} I c - 2 c^T \hat{A}^{-1} c + a_{nn} \\
&= -c^T \hat{A}^{-1} c + a_{nn} \\
&= a_{nn} - c^T \hat{A}^{-1} c, \\
\end{align}

which is what we wanted.

Since $x_n = 1$, the vector $x$ is nonzero, so $x^T A x = a_{nn} - c^T \hat{A}^{-1} c = a_{nn} - h^T h > 0$.

Therefore, there is a unique positive $r_{nn}$ that satisfies $r_{nn}^2 = a_{nn} - h^T h$, and it is $r_{nn} = \sqrt{a_{nn} - h^T h}$.

Thus, $R$ is uniquely defined since $\hat{R}$, $h$, and $r_{nn}$ are all uniquely defined. $\blacksquare$

---

Section 1.4 - Positive Definite Systems; Cholesky Decomposition.ipynb


# Optimization

> Marcos Duarte
> Laboratory of Biomechanics and Motor Control ([http://demotu.org/](http://demotu.org/))
> Federal University of ABC, Brazil

If there occur some changes in nature, the amount of action necessary for this change must be as small as possible.

Maupertuis (sec XVIII)

**Optimization is the process of finding the best value from possible alternatives with regards to a certain criteria** ([Wikipedia](http://en.wikipedia.org/wiki/Mathematical_optimization)).

Typically, such best value is the value that maximizes or minimizes the criteria. In this context, to solve a (mathematical) optimization problem is to find the maximum or minimum (a.k.a., a stationary point) of a function (and we can use maximum or minimum interchangeably because the maximum of a function is the minimum of the negative of that function).
To solve an optimization problem, we first have to model the problem and define the objective, the variables, and the constraints of the problem. In optimization, these terms are usually defined as:

1. Objective function (or also, cost, loss, utility, or fitness function): a function describing what we want to optimize.
2. Design variable(s): variables that will be manipulated to optimize the cost function.
3. Constraint functions: a set of constraints, equalities or inequalities that constrains the possible solutions to possible values of the design variables (candidate solutions or feasible solutions or feasible set).

A feasible solution that minimizes (or maximizes) the objective function is called an optimal solution.

The optimization problem is the calculation of the minimum or maximum values of an objective function over a set of **unknown** possible values of the design variables.
Even in case of a finite number of possible values of the objective function and design variables (e.g., after discretization and a manual or a grid search), in general the evaluation of the objective function is computationally expensive and should be avoided.
Of note, even if there is no other option, a random search is in fact more efficient than a manual or a grid search! See [Bergstra, Bengio (2012)](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf).

A typical problem of optimization: [Knapsack problem](https://en.wikipedia.org/wiki/Knapsack_problem).

Read more about that in [Introduction to Optimization](http://neos-guide.org/content/optimization-introduction) from the  [NEOS Guide](http://neos-guide.org/).

## Some jargon in mathematical optimization

 - **Linear versus nonlinear optimization**: linear optimization refers to when the objective function and the constraints are linear mathematical functions. When the objective function is linear, an optimal solution is always found at the constraint boundaries and a local optimum is also a global optimum. See [Wikipedia 1](https://en.wikipedia.org/wiki/Linear_programming) and [Wikipedia 2](https://en.wikipedia.org/wiki/Nonlinear_programming).
 - **Constrained versus unconstrained optimization**: in constrained optimization there are no constraints.
 - **Convex optimization**: the field of optimization that deals with finding the minimum of convex functions (or the maximum of concave functions) over a convex constraint set. The convexity of a function facilitates the optimization because a local minimum must be a global minimum and first-order conditions (the first derivatives) are sufficient conditions for finding the optimal solution. Note that although convex optimization is a particular case of nonlinear optimization, it is a relatively simple optimization problem, with robust and mature methods of solution. See [Wikipedia](https://en.wikipedia.org/wiki/Convex_optimization).
 - **Multivariate optimization**: optimization of a function of several variables.
 - **Multimodal optimization**: optimization of a function with several local minima to find the multiple (locally) optimal solutions, as opposed to a single best solution.
 - **Multi-objective optimization**: optimization involving more than one objective function to be optimized simultaneously.
 - **Optimal control**: finding a control law for a given system such that a certain optimality criterion is achieved. See [Wikipedia](https://en.wikipedia.org/wiki/Optimal_control).
 - **Quadratic programming**: optimization of a quadratic function subject to linear constraints. See [Wikipedia](https://en.wikipedia.org/wiki/Quadratic_programming).
 - **Simplex algorithm**: linear optimization algorithm that begins at a starting vertex and moves along the edges of the polytope (the feasible region) until it reaches the vertex of the optimum solution. See [Wikipedia](https://en.wikipedia.org/wiki/Simplex_algorithm).

## Maxima and minima

In mathematics, the maximum and minimum of a function are the largest and smallest values that the function takes at a point either within a neighborhood (local) or on the function entire domain  (global) ([Wikipedia](http://en.wikipedia.org/wiki/Maxima_and_minima)).

For a function of one variable, if the maximum or minimum of a function is not at the limits of the domain and if at least the first and second derivatives of the function exist, a maximum and minimum can be found as the point where the first derivative of the function is zero. If the second derivative on that point is positive, then it's a minimum, if it is negative, it's a maximum.

Figure. Maxima and minima of a function of one variable.

 - Note that the requirement that the second derivative on the extremum to be positive for a minimum or negative for a maximum is sufficient but not a necessary condition. For instance, the function $f(x)=x^4$ has an extremum in $x=0$ since $f'(x)=4x^3$ and $f'(0)=0$, but its second derivative at $x=0$ is also zero: $f''(x)=12x^2;\: f''(0)=0$. In fact, the requirement is that the first non-zero derivative on that point should be positive for a minimum or negative for a maximum: $f''''(0)=24$; the extremum is a minimum.

Let's now apply optimization to solve a problem with a univariate function.

```python
# import Python libraries
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

### Example 1: Maximum volume of a cardboard box

We want to make a box from a square cardboard with side $a$ such that its volume should be maximum.
What is the optimal distance where the square cardboard should be cut and folded to make a box with maximum volume?

Figure. A box to be made from a cardboard such that its volume should be maximum. Where we should cut?

If the distance where to cut and fold the cardboard is $b$, see figure above, the volume of the box will be:

\begin{equation}
\begin{array}{l l}
V(b) = b(a-2b)(a-2b) \\
\\
V(b) = a^2b - 4ab^2 + 4b^3
\end{array}
\label{}
\end{equation}

In the context of optimization:
**The expression for $V$ is the cost function, $b$ is the design variable, and the constraint is that feasible values of $b$ are in the interval $]0, \dfrac{a}{2}[$, i.e., $b>0$ and $b<\dfrac{a}{2}$.**

The first and second derivatives of $V$ w.r.t. $b$ are:

\begin{equation}
\begin{array}{l l}
\dfrac{\mathrm{d}V}{\mathrm{d}b} = a^2 - 8ab + 12b^2 \\
\\
\dfrac{\mathrm{d}^2 V}{\mathrm{d}b^2} = - 8a + 24b
\end{array}
\label{}
\end{equation}

We have to find the values for $b$ where the first derivative of $V$ is zero (the extrema) and then use the expression for the second derivative of $V$ to find whether each of these extrema is a minimum (positive value) or a maximum (negative value).
Let's use Sympy for that:

```python
a, b = sym.symbols('a b')
V = b*(a - 2*b)*(a - 2*b)
Vdiff = sym.expand(sym.diff(V, b))
roots = sym.solve(Vdiff, b)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ \frac{a}{6}, \quad \frac{a}{2}\right ]$$

```python
roots
```

    [a/6, a/2]

Discarding the solution $b=\dfrac{a}{2}$ (where $V=0$, which is a minimum), $b=\dfrac{a}{6}$ results in the maximum volume.
We can check that by plotting the volume of the cardboard box for $a=1$ and $b: [0,\:0.5]$:

```python
plot(V.subs({a: 1}), (b, 0, .5), xlabel='b', ylabel='V')
display(Math(sym.latex('V_{a=1}^{max}(b=%s)=%s'
                       %(roots[0].evalf(n=4, subs={a: 1}), V.evalf(n=3, subs={a: 1, b: roots[0]})))))
```

 - Note that although the problem above is a case of nonlinear constrained optimization, because the objective function is univariate, well-conditioned and the constraints are linear inequalities, the optimization is simple. Unfortunately, this is seldom the case.

## Curve fitting as an optimization problem

Curve fitting is the process of fitting a model, expressed in terms of a mathematical function, that depends on adjustable parameters to a series of data points and once adjusted, that curve has the best fit to the data points.

The general approach to the fitting procedure involves the definition of a merit function that measures the agreement between data and model. The model parameters are then adjusted to yield the best-fit parameters as a problem of minimization (an optimization problem, where the merit function is the cost function).

A classical solution, termed least-squares fitting, is to find the best fit by minimizing the sum of the squared differences between data points and the model function (the sum of squared residuals as the merit function).

For more on curve fitting see the video below and the notebook [Curve fitting](http://nbviewer.jupyter.org/github/demotu/BMC/blob/master/notebooks/CurveFitting.ipynb).

```python
from IPython.display import YouTubeVideo
YouTubeVideo('Rxp7o7_RxII', width=480, height=360, rel=0)
```

## Gradient descent

Gradient descent is a first-order iterative optimization algorithm for finding the minimum of a function ([Wikipedia](https://en.wikipedia.org/wiki/Gradient_descent)).
In the gradient descent algorithm, a local minimum of a function is found starting from an initial point and taking steps proportional to the negative of the derivative of the function (gradient) at the current point and we evaluate if the current point is lower than then the previous point until a local minimum in reached (hopefully).

It follows that, if

\begin{equation}
x_{n+1} = x_n - \gamma \nabla f(x)
\label{}
\end{equation}

for $\gamma$ small enough, then $f(x_{n}) \geq f(x_{n+1})$.

This process is repeated iteratively until the step size (which is proportional to the gradient!) is below a required precision (hopefully the sequence $x_{n}$ converges to the desired local minimum).

### Example 2: Minimum of a function by gradient descent

From https://en.wikipedia.org/wiki/Gradient_descent:
Calculate the minimum of $f(x)=x^4-3x^3+2$.

```python
# From https://en.wikipedia.org/wiki/Gradient_descent
# The local minimum of $f(x)=x^4-3x^3+2$ is at x=9/4

cur_x = 6               # The algorithm starts at x=6
gamma = 0.01            # step size multiplier
precision = 0.00001
step_size = 1           # initial step size
max_iters = 10000       # maximum number of iterations
iters = 0               # iteration counter

f  = lambda x: x**4 - 3*x**3 + 2  # lambda function for f(x)
df = lambda x: 4*x**3 - 9*x**2    # lambda function for the gradient of f(x)

while (step_size > precision) & (iters < max_iters):
    prev_x = cur_x
    cur_x -= gamma*df(prev_x)
    step_size = abs(cur_x - prev_x)
    iters+=1

print('True local minimum at {} with function value {}.'.format(9/4, f(9/4)))
print('Local minimum by gradient descent at {} with function value {}.'.format(cur_x, f(cur_x)))
```

## Multivariate optimization

When there is more than one design variable (the cost function depends on more than one variable), it's a multivariate optimization. The general idea of finding minimum and maximum values where the derivatives are zero still holds for a multivariate function. The second derivative of a multivariate function can be described by the Hessian matrix:

\begin{equation}
\mathbf{H} = \begin{bmatrix}{\dfrac  {\partial ^{2}f}{\partial x_{1}^{2}}}&{\dfrac  {\partial ^{2}f}{\partial x_{1}\,\partial x_{2}}}&\cdots &{\dfrac  {\partial ^{2}f}{\partial x_{1}\,\partial x_{n}}}\\[2.2ex]{\dfrac  {\partial ^{2}f}{\partial x_{2}\,\partial x_{1}}}&{\dfrac  {\partial ^{2}f}{\partial x_{2}^{2}}}&\cdots &{\dfrac  {\partial ^{2}f}{\partial x_{2}\,\partial x_{n}}}\\[2.2ex]\vdots &\vdots &\ddots &\vdots \\[2.2ex]{\dfrac  {\partial ^{2}f}{\partial x_{n}\,\partial x_{1}}}&{\dfrac  {\partial ^{2}f}{\partial x_{n}\,\partial x_{2}}}&\cdots &{\dfrac  {\partial ^{2}f}{\partial x_{n}^{2}}}
\end{bmatrix}
\label{}
\end{equation}

Let's see now a classical problem in biomechanics where optimization is useful and there is more than one design variable.

## The distribution problem in biomechanics

Using the inverse dynamics approach in biomechanics, we can determine the net force and torque acting on a joint if we know the external forces on the segments and the kinematics and inertial properties of the segments. But with this approach we are unable to determine the individual muscles forces that  created such torque, as expressed in the following equation:

\begin{equation}
M_{total} = M_1 + M_2 + \dots + M_n = r_1F_1 + r_2F_2 + \dots + r_nF_n
\label{}
\end{equation}

where $r_i$ is the moment arm of the force $F_i$ that generates a torque $M_i$, a parcel of the (known) total torque $M_{total}$.

Even if we know the moment arm of each muscle (e.g., from cadaveric data or from image analysis), the equation above has $n$ unknowns. Because there is more than one muscle that potentially created such torque, there are more unknowns than equations, and the problem is undetermined. So, the problem is how to find how the torque is distributed among the muscles of that joint.

One solution is to consider that we (biological systems) optimize our effort in order to minimize energy expenditure, stresses on our tissues, fatigue, etc. The principle of least action, stated in the opening of this text, is an allusion that optimization might be ubiquitous in nature. With this rationale, let's solve the distribution problem in biomechanics using optimization and find the minimum force of each muscle necessary to complete a given task.

The following cost functions have been proposed to solve the distribution problem in biomechanics:

\begin{equation}
\begin{array}{l l}
\displaystyle\sum_{i=1}^N F_i \quad &\text{e.g., Seireg and Arkivar (1973)}
\\
\displaystyle\sum_{i=1}^N F_i^2 \quad &
\\
\displaystyle\sum_{i=1}^N \left(\dfrac{F_i}{pcsa_i}\right)^2 \quad &\text{e.g., Crowninshield and Brand (1981)}
\\
\displaystyle\sum_{i=1}^N \left(\dfrac{F_i}{M_{max,i}}\right)^3 \quad &\text{e.g., Herzog (1987)}
\end{array}
\label{}
\end{equation}

Where $pcsa_i$ is the physiological cross-sectional area of muscle $i$ and $M_{max,i}$ is the maximum torque muscle $i$ can produce.
Each muscle force $F_i$ is a design variable and the following constraints must be satisfied:

\begin{equation}
\begin{array}{l l}
0 \leq F_i \leq F_{max}
\\
\displaystyle\sum_{i=1}^N r_i \times F_i = M
\end{array}
\label{}
\end{equation}

Let's apply this concept to solve a distribution problem in biomechanics.

### Muscle force estimation

Consider the following main flexors of the elbow joint (see figure below): biceps long head, biceps short head, and brachialis. Suppose that the elbow net joint torque determined using inverse dynamics is 20 Nm (flexor). How much each of these muscles contributed to the net torque?

Figure. A view in OpenSim of the arm26 model showing three elbow flexors (Biceps long and short heads and Brachialis).

For the optimization, we will need experimental data for the moment arm, maximum moment, and *pcsa* of each muscle. Let's import these data from the OpenSim arm26 model:

```python
# time elbow_flexion BIClong BICshort BRA
r_ef = np.loadtxt('./../data/r_elbowflexors.mot', skiprows=7)
f_ef = np.loadtxt('./../data/f_elbowflexors.mot', skiprows=7)
```

The maximum isometric force of these muscles are defined in the arm26 model as: Biceps long head: 624.3 N, Biceps short head: 435.56 N, and Brachialis: 987.26 N. Let's compute the mamimum torques that each muscle could produce considering a static situation at the different elbow flexion angles:

```python
m_ef = r_ef*1
m_ef[:, 2:] = r_ef[:, 2:]*f_ef[:, 2:]
```

And let's visualize these data:

```python
labels = ['Biceps long head', 'Biceps short head', 'Brachialis']
fig, ax = plt.subplots(nrows=1, ncols=3, sharex=True, figsize=(10, 4))
ax[0].plot(r_ef[:, 1], r_ef[:, 2:])
#ax[0].set_xlabel('Elbow angle $(\,^o)$')
ax[0].set_title('Moment arm (m)')
ax[1].plot(f_ef[:, 1], f_ef[:, 2:])
ax[1].set_xlabel('Elbow angle $(\,^o)$', fontsize=16)
ax[1].set_title('Maximum force (N)')
ax[2].plot(m_ef[:, 1], m_ef[:, 2:])
#ax[2].set_xlabel('Elbow angle $(\,^o)$')
ax[2].set_title('Maximum torque (Nm)')
ax[2].legend(labels, loc='best', framealpha=.5)
ax[2].set_xlim(np.min(r_ef[:, 1]), np.max(r_ef[:, 1]))
plt.tight_layout()
plt.show()
```

These data don't have the *pcsa* value of each muscle. We will estimate the *pcsa* considering that the amount of maximum muscle force generated per area is constant and equal to 50N/cm$^2$. Consequently, the *pcsa* (in cm$^2$) for each muscle is:

```python
a_ef = np.array([624.3, 435.56, 987.26])/50  # 50 N/cm2
print(a_ef)
```

### Static versus dynamic optimization

In the context of biomechanics, we can solve the distribution problem separately for each angle (instant) of the elbow; we will refer to that as static optimization. However, there is no guarantee that when we analyze all these solutions across the range of angles, they will be the best solution overall. One reason is that static optimization ignores the time history of the muscle force. Dynamic optimization refers to the optimization over a period of time. For such, we will need to input a cost function spanning the entire period of time at once. Dynamic optimization usually has a higher computational cost than static optimization.

For now, we will solve the present problem using static optimization.

### Solution of the optimization problem

For the present case, we are dealing with a problem of minimization, multidimensional (function of several variables), nonlinear, constrained, and we can't assume that the cost function is convex. Numerical optimization is hardly a simple task. There are many different algorithms and public and commercial software for performing optimization. For instance, look at [NEOS Server](http://www.neos-server.org/neos/), a free internet-based service for solving numerical optimization problems.
We will solve the present problem using the [scipy.optimize](http://docs.scipy.org/doc/scipy/reference/optimize.html#module-scipy.optimize) package which provides several optimization algorithms. We will use the function `minimize`:

```python
scipy.optimize.minimize(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None)
"""Minimization of scalar function of one or more variables."""
```

Now, let's write Python functions for each cost function:

```python
from scipy.optimize import minimize
```

```python
def cf_f1(x):
    """Cost function: sum of forces."""
    return x[0] + x[1] + x[2]

def cf_f2(x):
    """Cost function: sum of forces squared."""
    return x[0]**2 + x[1]**2 + x[2]**2

def cf_fpcsa2(x, a):
    """Cost function: sum of squared muscle stresses."""
    return (x[0]/a[0])**2 + (x[1]/a[1])**2 + (x[2]/a[2])**2

def cf_fmmax3(x, m):
    """Cost function: sum of cubic forces normalized by moments."""
    return (x[0]/m[0])**3 + (x[1]/m[1])**3 + (x[2]/m[2])**3
```

Let's also define the Jacobian for each cost function (which is an optional parameter for the optimization):

```python
def cf_f1d(x):
    """Derivative of cost function: sum of forces."""
    dfdx0 = 1
    dfdx1 = 1
    dfdx2 = 1
    return np.array([dfdx0, dfdx1, dfdx2])

def cf_f2d(x):
    """Derivative of cost function: sum of forces squared."""
    dfdx0 = 2*x[0]
    dfdx1 = 2*x[1]
    dfdx2 = 2*x[2]
    return np.array([dfdx0, dfdx1, dfdx2])

def cf_fpcsa2d(x, a):
    """Derivative of cost function: sum of squared muscle stresses."""
    dfdx0 = 2*x[0]/a[0]**2
    dfdx1 = 2*x[1]/a[1]**2
    dfdx2 = 2*x[2]/a[2]**2
    return np.array([dfdx0, dfdx1, dfdx2])

def cf_fmmax3d(x, m):
    """Derivative of cost function: sum of cubic forces normalized by moments."""
    dfdx0 = 3*x[0]**2/m[0]**3
    dfdx1 = 3*x[1]**2/m[1]**3
    dfdx2 = 3*x[2]**2/m[2]**3
    return np.array([dfdx0, dfdx1, dfdx2])
```

Let's define initial values:

```python
M = 20  # desired torque at the elbow
iang = 69  # which will give the closest value to 90 degrees
r  = r_ef[iang, 2:]
f0 = f_ef[iang, 2:]
a  = a_ef
m  = m_ef[iang, 2:]
x0 = f_ef[iang, 2:]/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

Inequality constraints (such as boundaries in our problem) can be entered with the parameter `bounds` to the `minimize` function:

```python
bnds = ((0, f0[0]), (0, f0[1]), (0, f0[2]))
```

Equality constraints (such as the sum of torques should equals the desired torque in our problem), as well as inequality constraints, can be entered with the parameter `constraints` to the `minimize` function (and we can also opt to enter the Jacobian of these constraints):

```python
# use this in combination with the parameter bounds:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)})
```

```python
# to enter everything as constraints:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[0]-x[0],
         'jac' : lambda x, r, f0, M: np.array([-1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[1]-x[1],
         'jac' : lambda x, r, f0, M: np.array([0, -1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[2]-x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, -1]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[0],
         'jac' : lambda x, r, f0, M: np.array([1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[1],
         'jac' : lambda x, r, f0, M: np.array([0, 1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, 1]), 'args': (r, f0, M)})
```

Although more verbose, if all the Jacobians of the constraints are also informed, this alternative seems better than informing bounds for the optimization process (less error in the final result and less iterations).

Given the characteristics of the problem, if we use the function `minimize` we are limited to the SLSQP (Sequential Least SQuares Programming) solver.

Finally, let's run the optimization for the four different cost functions and find the optimal muscle forces:

```python
f1r = minimize(fun=cf_f1, x0=x0, args=(), jac=cf_f1d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

```python
f2r = minimize(fun=cf_f2, x0=x0, args=(), jac=cf_f2d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

```python
fpcsa2r = minimize(fun=cf_fpcsa2, x0=x0, args=(a,), jac=cf_fpcsa2d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

```python
fmmax3r = minimize(fun=cf_fmmax3, x0=x0, args=(m,), jac=cf_fmmax3d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

Let's compare the results for the different cost functions:

```python
dat = np.vstack((np.around(r*100,1), np.around(a,1), np.around(f0,0), np.around(m,1)))
opt = np.around(np.vstack((f1r.x, f2r.x, fpcsa2r.x, fmmax3r.x)), 1)
er = ['-', '-', '-', '-',
      np.sum(r*f1r.x)-M, np.sum(r*f2r.x)-M, np.sum(r*fpcsa2r.x)-M, np.sum(r*fmmax3r.x)-M]
data = np.vstack((np.vstack((dat, opt)).T, er)).T

rows = ['$\text{Moment arm}\;[cm]$', '$pcsa\;[cm^2]$', '$F_{max}\;[N]$', '$M_{max}\;[Nm]$',
        '$\sum F_i$', '$\sum F_i^2$', '$\sum(F_i/pcsa_i)^2$', '$\sum(F_i/M_{max,i})^3$']
cols = ['Biceps long head', 'Biceps short head', 'Brachialis', 'Error in M']
df = pd.DataFrame(data, index=rows, columns=cols)
print('\nComparison of different cost functions for solving the distribution problem')
df
```

## Comments

The results show that the estimations for the muscle forces depend on the cost function used in the optimization. Which one is correct? This is a difficult question and it's dependent on the goal of the actual task being modeled. Glitsch and Baumann (1997) investigated the effect of different cost functions on the optimization of walking and running and the predicted muscles forces were compared with the electromyographic activity of the corresponding muscles of the lower limb. They found that, among the analyzed cost functions, the minimization of the sum of squared muscle stresses resulted in the best similarity with the actual electromyographic activity.

In general, one should always test different algorithms and different initial values before settling for the solution found. Downey (2011), Kitchin (2013), and Kiusalaas (2013) present more examples on numerical optimization. The [NEOS Guide](http://neos-guide.org/) is a valuable source of information on this topic and [OpenOpt](http://openopt.org/) is a good alternative software for numerical optimization in Python.

## Exercises

1. Find the extrema in the function $f(x)=x^3-7.5x^2+18x-10$ analytically and determine if they are minimum or maximum.
2. Find the minimum in the $f(x)=x^3-7.5x^2+18x-10$ using the gradient descent algorithm.
2. Regarding the distribution problem for the elbow muscles presented in this text:
    a. Test different initial values for the optimization.
    b. Test other values for the elbow angle where the results are likely to change.

3. In an experiment to estimate forces of the elbow flexors, through inverse dynamics it was found an elbow flexor moment of 10 Nm.
Consider the following data for maximum force (F0), moment arm (r), and pcsa (A) of the brachialis, brachioradialis, and biceps brachii muscles: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A (cm$^2$): 33, 8, 23, respectively (data from Robertson et al. (2013)).
    a. Use static optimization to estimate the muscle forces.
    b. Test the robustness of the results using different initial values for the muscle forces.
    c. Compare the results for different cost functions.

## References

- Bergstra B, Bengio Y (2012) [Random Search for Hyper-Parameter Optimization](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf). Journal of Machine Learning Research, 13, 281-305.
- Crowninshield RD, Brand RA (1981) [A physiologically based criterion of muscle force prediction in locomotion](http://www.ncbi.nlm.nih.gov/pubmed/7334039). Journal of Biomechanics, 14, 793–801.
- Downey AB (2014) [Physical Modeling in MATLAB](http://greenteapress.com/wp/physical-modeling-in-matlab-2e/). 2nd edition. Green Tea Press.
- Herzog W (1987) [Individual muscle force estimations using a non-linear optimal design](http://www.ncbi.nlm.nih.gov/pubmed/3682873). J Neurosci Methods, 21, 167-179.
- Glitsch U, Baumann W (1997) [The three-dimensional determination of internal loads in the lower extremity](http://www.ncbi.nlm.nih.gov/pubmed/9456380). Journal of Biomechanics, 30, 1123–1131.
- Kitchin J (2013) [pycse - Python Computations in Science and Engineering](http://kitchingroup.cheme.cmu.edu/pycse/).
- Kiusalaas (2013) [Numerical methods in engineering with Python 3](http://books.google.com.br/books?id=aJkXoxxoCoUC). 3rd edition. Cambridge University Press.
- Nigg BM and Herzog W (2006) [Biomechanics of the Musculo-skeletal System](https://books.google.com.br/books?id=hOIeAQAAIAAJ&dq=editions:ISBN0470017678). 3rd Edition. Wiley.
- Robertson G, Caldwell G, Hamill J, Kamen G (2013) [Research Methods in Biomechanics](http://books.google.com.br/books?id=gRn8AAAAQBAJ). 2nd Edition. Human Kinetics.
- Seireg A, Arvikar RJ (1973) [A mathematical model for evaluation of forces in lower extremeties of the musculo-skeletal system](http://www.ncbi.nlm.nih.gov/pubmed/4706941). Journal of Biomechanics, 6,  313–322, IN19–IN20, 323–326.


```python
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.gridspec import GridSpec
from matplotlib.ticker import ScalarFormatter
import math
```

This notebook assumes you have completed the notebook [Introduction of sine waves](TDS_Introduction-sine_waves.ipynb). This notebook follows the same pattern of time domain waveform generation: instantaneous frequency -> angle step -> total angle -> time domain waveform.

Our goal is to track features of different acoustic impedance in material using a low power time domain waveform. Time delay spectrometry (TDS) is one implementation of this goal. To understand TDS we need to understand the waveform which is used by TDS called a chirp. A chirp is a sinusoid that is constantly varying in frequency. The chirp is generated by integrating a varying angle step which is derived from an instantaneous frequency profile. We will generate a chirp in this notebook. An overview of this technique is given [here](https://www.youtube.com/watch?v=RQplkt0bw_c).
The angle of the chirp can be found by integrating the instantaneous frequency:

\begin{equation}
	f(t)=\frac{f_{end}-f_{start}}{T_c}t + f_{start}
\end{equation}

\begin{equation}
\Delta\phi(t) = 2\pi f(t)\Delta t
\end{equation}

\begin{equation}
\phi (t)=\int_{}^{} \Delta\phi(t) = \int_{}^{} 2\pi f(t) dt = \int_{}^{}\frac{f_{end}-f_{start}}{T_c}tdt + \int_{}^{}f_{start}dt
\end{equation}

\begin{equation}
 \phi (t)= \frac{f_{end}-f_{start}}{T_c}\int_{}^{}tdt + f_{start}\int_{}^{}dt
\end{equation}

\begin{equation}
 \phi (t)= \frac{f_{end}-f_{start}}{T_c}\frac{t^2}{2} + f_{start}t
\end{equation}

This gives the time series value of

\begin{equation}
x(t) = e^{j\phi (t)} = e^{j(\frac{f_{end}-f_{start}}{T_c}\frac{t^2}{2} + f_{start}t)}
\end{equation}

But the formula for angle requires squaring time which will cause numeric errors as the time increases. Another approach is to implement the formula for angle as a cummulative summation.

\begin{equation}
\phi_{sum} (N)=\sum_{k=1}^{N} \Delta\phi(k) = \sum_{k=1}^{N} 2\pi f(k) t_s = \sum_{k=1}^{N}(\frac{f_{end}-f_{start}}{T_c}k + f_{start})t_s
\end{equation}

This allow for the angle always stay between 0 and two pi by subtracting two phi whenever the angle exceeds the value. We will work with the cummlative sum of angle, but then compare it to the integral to determine how accurate the cummulative sum is.

```python

#Tc is the max depth we are interested in
Tc_sec=5
#Tc_sec=2

f_start_Hz=10e3
#talk about difference and similarity of sine wave example, answer why not 32 samples
f_stop_Hz=990e3

#We choose 8 samples per cycle at the maximum frequency to not require steep pulse shaping filter profiles on the output of the
#digital to analog converter
#fs=16e6
fs=2e6
samplesPerCycle=fs/f_stop_Hz

ts=1/fs

total_samples= math.ceil(fs*Tc_sec)
n = np.arange(0,total_samples, step=1, dtype=np.float64)
t_sec=n*ts
t_usec = t_sec *1e6

#This is the frequency of the chirp over time. We assume linear change in frequency
chirp_freq_slope_HzPerSec=(f_stop_Hz-f_start_Hz)/Tc_sec

#Compute the instantaneous frequency which is a linear function
chirp_instantaneous_freq_Hz=chirp_freq_slope_HzPerSec*t_sec+f_start_Hz
chirp_instantaneous_angular_freq_radPerSec=2*np.pi*chirp_instantaneous_freq_Hz

#Since frequency is a change in phase the we can plot it as a phase step
chirp_phase_step_rad=chirp_instantaneous_angular_freq_radPerSec*ts

#The phase step can be summed (or integrated) to produce the total phase which is the phase value
#for each point in time for the chirp function
chirp_phase_rad=np.cumsum(chirp_phase_step_rad)
#The time domain chirp function
chirp = np.exp(1j*chirp_phase_rad)
```

```python
#We can see, unlike the complex exponential, the chirp's instantaneous frequency is linearly increasing.
#This corresponds with the linearly increasing phase step.
fig, ax = plt.subplots(2, 1, sharex=True,figsize = [8, 8])
lns1=ax[0].plot(t_usec,chirp_instantaneous_freq_Hz,linewidth=4, label='instantanous frequency');
ax[0].set_title('Comparing the instantaneous frequency and phase step')
ax[0].set_ylabel('instantaneous frequency (Hz)')

axt = ax[0].twinx()
lns2=axt.plot(t_usec,chirp_phase_step_rad,linewidth=2,color='black', linestyle=':', label='phase step');
axt.set_ylabel('phase step (rad)')

#ref: https://stackoverflow.com/questions/5484922/secondary-axis-with-twinx-how-to-add-to-legend
lns = lns1+lns2
labs = [l.get_label() for l in lns]
ax[0].legend(lns, labs, loc=0)

#We see that summing or integrating the linearly increasing phase step gives a quadratic function of total phase.
ax[1].plot(t_usec,chirp_phase_rad,linewidth=4,label='chirp');
ax[1].plot([t_usec[0], t_usec[-1]],[chirp_phase_rad[0], chirp_phase_rad[-1]],linewidth=1, linestyle=':',label='linear (x=y)');
ax[1].set_title('Cumulative quandratic phase function of chirp')
ax[1].set_xlabel('time ($\mu$sec)')
ax[1].set_ylabel('total phase (rad)')
ax[1].legend();

```

    <IPython.core.display.Javascript object>

```python
#We now what to load this into GNURadio. So we will only write the raw file and then load that into GNURadio. This will cause the array size to be lost
chirp.astype('complex64').tofile(f'../data/TDS_Part_1_chirp_compare_fs_{int(fs)}_samplesPerCycle_{int(samplesPerCycle)}_Tc_sec_{Tc_sec}_complex64.bin')
print(f'The sample rate is {int(fs)}')
print(f'The samples per cycle is {samplesPerCycle}')
```

    The sample rate is 2000000
    The samples per cycle is 2.0202020202020203

```python

```


# Custom Gradient

This is a brief demonstration of tensorflow [custom gradients](https://www.tensorflow.org/api_docs/python/tf/custom_gradient)

## Chain rule

Lets say we have a function $f(x) = x^2$. If we now compose this function such that $y = f(f(f(x)))$. Now we want to find the gradient $\frac{dy}{dx}$.

We first decompose this into

\begin{align*}
  y &= x_0^2\\
  x_0 &= x_1^2\\
  x_1 &= x^2
\end{align*}

On taking first order derivative, we get

\begin{align*}
  \frac{dy}{dx_0} &= 2x_0\\
  \frac{dx_0}{dx_1} &= 2x_1\\
  \frac{dx_1}{dx} &= 2x
\end{align*}

Using chain rule

\begin{equation*}
  \frac{dy}{dx} = \frac{dy}{dx_0}  \frac{dx_0}{dx_1}  \frac{dx_1}{dx}
\end{equation*}

To generalize

\begin{equation*}
  \frac{dy}{dx} = \frac{dy}{dx_{0}}  ...  \frac{dx_i}{dx_{i+1}}  ...  \frac{dx_n}{dx}
\end{equation*}

In tensorflow the `upstream` gradient is passed as an argument to the inner function `grad`.

\begin{equation*}
  upstream = \frac{dx_{i+1}}{dx_{i+2}}  ...  \frac{dx_n}{dx}
\end{equation*}

Now we can multiply this upstream gradient to the gradient of the current function $\frac{dx_{i}}{dx_{i+1}}$ and pass it downstream.

\begin{equation*}
  downstream = \frac{dx_{i}}{dx_{i+1}}  * upstream
\end{equation*}

```python
import tensorflow as tf
```

```python
# @tf.function
@tf.custom_gradient
def foo(x):
    tf.debugging.assert_rank(x, 0)

    def grad(dy_dx_upstream):
        dy_dx = 2 * x
        dy_dx_downstream = dy_dx * dy_dx_upstream
        tf.print(f'x={x}\tupstream={dy_dx_upstream}\tcurrent={dy_dx}\t\tdownstream={dy_dx_downstream}')
        return dy_dx_downstream

    y = x ** 2
    tf.print(f'x={x}\ty={y}')

    return y, grad

x = tf.constant(2.0, dtype=tf.float32)

with tf.GradientTape(persistent=True) as tape:
    tape.watch(x)
    y = foo(foo(foo(x))) # y = x ** 8

tf.print(f'\nfinal dy/dx={tape.gradient(y, x)}')
```

    x=2.0	y=4.0
    x=4.0	y=16.0
    x=16.0	y=256.0
    x=16.0	upstream=1.0	current=32.0		downstream=32.0
    x=4.0	upstream=32.0	current=8.0		downstream=256.0
    x=2.0	upstream=256.0	current=4.0		downstream=1024.0

    final dy/dx=1024.0

### Gradients with multiple variables

If the function takes multiple variables, then the gradient for each variable has to be returned as demonstrated in the example.

```python
@tf.custom_gradient
def bar(x, y):
    tf.debugging.assert_rank(x, 0)
    tf.debugging.assert_rank(y, 0)

    def grad(upstream):
        dz_dx = y
        dz_dy = x
        return upstream * dz_dx, upstream * dz_dy

    z = x * y

    return z, grad

x = tf.constant(2.0, dtype=tf.float32)
y = tf.constant(3.0, dtype=tf.float32)

with tf.GradientTape(persistent=True) as tape:
    tape.watch(x)
    tape.watch(y)
    z = bar(x, y)

tf.print(z)
tf.print(tape.gradient(z, x))
tf.print(tape.gradient(z, y))
tf.print(tape.gradient(x, y))
```

    6
    3
    2
    None

## Application of custom gradients

### Toy example: Differentiable approximation of non-differentiable functions

We take the sign function as an example

\begin{equation}
sign(x)= \\
\begin{cases}
  -1, & \text{if}\ x<0 \\
  0, & \text{if}\ x=0 \\
  1, & \text{if}\ x>0 \\
\end{cases}\end{equation}

By implementing a custom gradient, we can continue to have the $sign(x)$ function in forward pass but a differentiable approximation in the backward pass. In this case we approximate $sign(x)$ with the sigmoid function $ \sigma(x)$

\begin{equation}
\frac{dsign_{approx}(x)}{dx} = \sigma(x) (1 - \sigma(x)) \\
sign_{approx}(x) = \sigma(x) + C \\
\end{equation}

```python
# @tf.function
@tf.custom_gradient
def differentiable_sign(x):
    tf.debugging.assert_rank(x, 0)

    def grad(upstream):
        dy_dx = tf.math.sigmoid(x) * (1 - tf.math.sigmoid(x))
        return upstream * dy_dx

    if x > tf.constant(0.0):
        return tf.constant(1.0), grad
    else:
        return tf.constant(-1.0), grad

x = tf.constant(3.0, dtype=tf.float32)

with tf.GradientTape(persistent=True) as tape:
    tape.watch(x)
    y = differentiable_sign(x)
    loss = tf.nn.l2_loss(y - tf.constant(-1.0))

tf.print(y)
tf.print(tape.gradient(y, x))
tf.print(loss)
tf.print(tape.gradient(loss, x))
```

    1
    0.0451766551
    2
    0.0903533101

```python
x = tf.Variable(-1.0)
opt = tf.keras.optimizers.Adam(1e-1)
# opt = tf.keras.optimizers.SGD(1)

def train_step():
    with tf.GradientTape() as tape:
        y = differentiable_sign(x)
        loss = tf.nn.l2_loss(y - tf.constant(1.0))
    grads = tape.gradient(loss, x)
    opt.apply_gradients(zip([grads], [x]))
    return loss, y, grads

for i in range(100):
    loss, y, grads = train_step()
    if i % 10 == 0:
        tf.print(i, loss, grads, x, y)
```

    0 2 -0.393223882 -0.89999783 -1
    10 0 0 0.0995881185 1
    20 0 0 0.6450876 1
    30 0 0 0.855591536 1
    40 0 0 0.938390434 1
    50 0 0 0.970632374 1
    60 0 0 0.983009696 1
    70 0 0 0.987700462 1
    80 0 0 0.989459515 1
    90 0 0 0.990113616 1

```python

```


```python
# https://www.khanacademy.org/math/ap-calculus-ab/ab-diff-analytical-applications-new/ab-5-11/v/minimizing-the-cost-of-a-storage-container
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *; x,h,y,n,t,d,c   = symbols("x h y n t d c"); C, D = symbols("C D", real=True)
Image(url= "https://i.imgur.com/iU00Tki.png")
```

```python
cost_of_base = 10*x*(2*x)
cb = cost_of_base
cb
```

$\displaystyle 20 x^{2}$

```python
cost_of_side_pair0 = 2*5*x*h
cs0 = cost_of_side_pair0
cs0
```

$\displaystyle 10 h x$

```python
cost_of_side_pair1 = 2*5*2*x*h
cs1 = cost_of_side_pair1
cs1
```

$\displaystyle 20 h x$

```python
total_cost = cb + cs0 + cs1
tc = total_cost
tc
```

$\displaystyle 30 h x + 20 x^{2}$

```python
# now lets optimize this in respect to x
```

```python
Eq(x*2*x*h,16)
```

$\displaystyle 2 h x^{2} = 16$

```python
# to make Eq(x*2*x*h,16) a function of h(x) then solve for h
solve(Eq(x*2*x*h,16),h)
```

    [8/x**2]

```python
8/x**2
```

$\displaystyle \frac{8}{x^{2}}$

```python
#let us substitute 8/x**2 for h in our variable tc
```

```python
tc
```

$\displaystyle 30 h x + 20 x^{2}$

```python
tc.subs(h, 8/x**2)
```

$\displaystyle 20 x^{2} + \frac{240}{x}$

```python
def C(x):
    return tc.subs(h, 8/x**2)
C(x)
```

$\displaystyle 20 x^{2} + \frac{240}{x}$

```python
# now let us optomize C(x) and
# find the critical points to find min/max values
diff(C(x))  # find the derivative

```

$\displaystyle 40 x - \frac{240}{x^{2}}$

```python
# find when diff(C(x)) is 0
Eq(diff(C(x)) ,0)
```

$\displaystyle 40 x - \frac{240}{x^{2}} = 0$

```python
solve(diff(C(x)) ,x)
```

    [6**(1/3),
     -6**(1/3)/2 - 2**(1/3)*3**(5/6)*I/2,
     -6**(1/3)/2 + 2**(1/3)*3**(5/6)*I/2]

```python
# 6**(1/3) is our critical point
critical_point = 6**(1/3)
cp = critical_point
cp
```

    1.8171205928321397

```python
simplify(diff(C(x),x,2)) #second derivative to see concave upward
```

$\displaystyle 40 + \frac{480}{x^{3}}$

```python
C(x).subs(x,cp)
```

$\displaystyle 198.115634933678$

```python
Image(url= "https://i.imgur.com/hLdDjpC.png")
```


Example 3
=========

The next example demonstrates the multi-parameter capabilities of ``cgn``. We are going to solve a nonlinear least-squares problem that depends on two parameters $x$ and $y$, where $x$ has to satisfy a nonlinear equality constraint, while $y$ is only nonnegativity constrained.

$
\begin{align}
\min_{x \in \mathbb{R}^2, y \in \mathbb{R}} \quad & ||F(x, y)||_2^2 + \beta ||R(x - m)||_2^2\\
\text{s. t.} \quad  & x_1 + x_1^3 + x_2 + x_2^2 = 0, \quad y > 0, \\
\text{where } \quad & F(x) = \left(\begin{matrix}
x_1 + e^{-x_2} + \sqrt{y} \\
x_1^2 + 2 x_2 + 1 - \sqrt{y}
\end{matrix} \right), \\
& R = \left( \begin{matrix}
1 & 2 \\
3 & 4
\end{matrix} \right), \quad m = \left(\begin{matrix}
1 \\ 1 \\
\end{matrix} \right), \quad \beta = 0.1.
\end{align}
$

Let us start by implementing the required functions and their derivatives:

```python
from math import exp, sqrt
import numpy as np
import cgn

def F(x, y):
    out = np.array([x[0] + exp(-x[1] + sqrt(y[0])),
                    x[0] ** 2 + 2 * x[1] + 1 - sqrt(y[0])])
    return out

def DF(x, y):
    jac = np.array([[1., -exp(-x[1]), 0.5 / sqrt(y[0])],
                    [2 * x[0], 2., - 0.5 / sqrt(y[0])]])
    return jac
```

Next, we set up the inequality constraint, which only depends on $x$:

```python
def g(x):
    out = x[0] + x[0] ** 3 + x[1] + x[1] ** 2
    return np.array([out])

def Dg(x):
    jac = np.array([1 + 3 * x[0] ** 2, 1. + 2 * x[1]]).reshape((1, 2))
    return jac
```

Next, we set up our ``cgn.Parameter`` objects. For the initial guess, let's just try $x = [0, 0]$ and $y = 1$:

```python
x = cgn.Parameter(name="x", start=np.zeros(2))
y = cgn.Parameter(name="y", start=np.ones(1))
```

For ``x``, we have to specify the regularization term $\beta ||R(x-m)||_2^2$:

```python
x.regop = np.array([[1., 2.], [3., 4.]])
x.mean = np.array([1., 1.])
x.beta = 0.1
```

The second parameter ``y`` is not regularized, but it has to satisfy the lower-bound constraint $y > 0$. We implement this strict inequality constraint as an equality constraint
$y \geq \epsilon$ for a small, positive $\epsilon > 0$.

```python
eps = 1e-10
y.lb = np.array([eps])
```

With this setup, we can finally create our optimization problem and solve it with ``cgn``.

```python
inequality_constraint = cgn.NonlinearConstraint(parameters=[x], fun=g, jac=Dg, ctype="ineq")

problem = cgn.Problem(parameters=[x, y], fun=F, jac=DF, constraints=[inequality_constraint])
solver = cgn.CGN()

solution = solver.solve(problem=problem)
```

Seems to work without problems. Let us look at the minimizers:

```python

x_min = solution.minimizer("x")
y_min = solution.minimizer("y")
print(f"x_min = {x_min}")
print(f"y_min = {y_min}")
```

    x_min = [-0.00522749  0.83669584]
    y_min = [1.02461374]


```python
# SymPy/IPython Initialization

from sympy import *
from IPython.display import display
from IPython.display import Latex
from sympy.abc import x, y, z, s, t

init_printing(use_latex=True)

# def eq(sym):
#     return Eq(sym, sym.doit())
```

```python
# Symbols

alpha_n, alpha_nm1 = symbols(r'alpha_n \alpha_{n-1}')
display(Eq(alpha_n, 2 * alpha_nm1 + 1))
```

```python
# Calculation

x, y = symbols('x y')

print(Rational(3,2) + 4)
print((x**2).subs(x, y + 1).subs(y, 4))
```

```python
# Limits

x = Symbol('x')

sin_x = Limit(sin(x)/x, x, 0)
recip = Limit(1/x, x, oo)

display(Eq(sin_x, sin_x.doit()))
display(Eq(recip, recip.doit()))
```

```python
# Derivatives

x, y = symbols('x y')

deriv = Derivative(x**3 * y, x, 2)
display(Eq(deriv, deriv.doit()))
```

```python
# Real integration

# integrate() evaluates
# Integral() creates unevaluated

r, u = symbols('r u', real=True)

indefinite = Integral(exp(u), u)
display(Eq(indefinite, indefinite.doit()))

definite = Integral(exp(-r**2), (r, -oo, oo))
display(Eq(definite, definite.doit()))
```

```python
# Integral transforms

a = Symbol('a', positive=True)
k, s, x = symbols('k s x')

lp = laplace_transform(exp(a*x), x, s)
inv_lp = inverse_laplace_transform(lp[0], s, x)
fourier = fourier_transform(exp(-abs(x)), x, k)

display(lp)
display(inv_lp)
display(fourier)
```

```python
# Complex

x = symbols('x', real=True)

display(Eq(exp(I*x), exp(I*x).expand(complex=True)))

# residues??
#TODO
```

```python
# ODEs

f = Function('f')
x, t = symbols('x, t')

# f'' + f = 0
display(dsolve(f(t).diff(t, t) + f(t), f(t)))
```

```python
# PDEs

u, X, T = map(Function, 'uXT')

#TODO

```

```python
# Recurrence relation

y = Function('y')
n = symbols('n', integer=True)

fib = y(n) - y(n-1) - y(n-2)
rsolve(fib, y(n), {y(0):0, y(1):1})
```

```python
# Latex

x = symbols('x')

print(latex(exp(2*I*pi*x)))

display(Latex(r"""\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}"""))
```


```python
import numpy as np
from scipy import optimize
```

Starting with an initial flow over a horizontal surface where $M_1 = 2.2$, an oblique shock forms at an angle of $\theta = 35^{\circ}$, which deflects the flow by the angle $\delta$. The flow now has Mach number $M_2$.

To satisfy the boundary condition of the surface, the oblique shock reflects at an angle $\beta$ from the surface, which deflects the flow back to the horizontal direction with a flow at a Mach number of $M_3$.

What is the angle of reflection ($\beta$)? How do the strengths of each shock compare?

First, let's determine the deflection angle of the first oblique shock using
\begin{equation}
\tan \delta = 2 (\cot \theta) \left[ \frac{M_1^2 \sin^2 \theta - 1}{M_1^2 (\gamma + \cos 2\theta) + 2}\right]
\end{equation}

```python
# known values
gamma = 1.4
M1 = 2.2
theta = np.radians(35.0) # convert from degrees to radians

delta = np.arctan(
    2 * (1 / np.tan(theta)) * (M1**2 * np.sin(theta)**2 - 1) / (M1**2 * (gamma + np.cos(2*theta)) + 2)
)
print(f'Deflection angle: {np.degrees(delta):.3}')
```

    Deflection angle: 9.21

Now we can determine the conditions after the oblique shock, using
\begin{equation}
M_{1n} = M_1 \sin \theta
\end{equation}
and
\begin{equation}
M_{2n}^2 = \frac{M_{1n}^2 + \frac{2}{\delta-1}}{\frac{2\gamma}{\gamma-1} M_{1n}^2 - 1}
\end{equation}

```python
M1n = M1 * np.sin(theta)
M2n = np.sqrt((M1n**2 + (2/(gamma - 1))) / (M1n**2 * (2*gamma)/(gamma - 1) - 1))

M2 = M2n / np.sin(theta - delta)
print(f'M_2 = {M2:.3}')
```

    M_2 = 1.85

Now, the reflected oblique shock must turn the flow back to the horizontal direction to satisfy flow tangency with the wall, so $\delta_2 = \delta$. We can thus determine the angle of the second shock, $\theta_2$:

```python
def oblique(theta, M1, delta, gamma):
    return (
        np.tan(delta) -
        2 * (1 / np.tan(theta)) * (M1**2 * np.sin(theta)**2 - 1) / (M1**2 * (gamma + np.cos(2*theta)) + 2)
    )

root = optimize.root_scalar(oblique, # function we are solving
                            args=(M2, delta, gamma),
                            # give range for weak shocks
                            bracket=[np.radians(0.0001), np.radians(45.0)]
                           )
theta_2 = root.root
print(f'theta_2 = {np.degrees(theta_2):.3} degrees')
```

    theta_2 = 41.7 degrees

 **But** this shock angle is defined with respect to the flow direction in the middle region, so $\beta = \theta_2 - \delta$:

```python
beta = theta_2 - delta
print(f'beta = {np.degrees(beta):.3} degrees')
```

    beta = 32.5 degrees

Thus, the angle of reflection $\beta$ is **smaller** than the angle of incidence $\theta = 35^{\circ}$. Regarding the shock strength, we can compare the normal Mach numbers of the flow prior to each shock:

```python
print(f'M1n = {M1n:.4}')
M3n = M2 * np.sin(theta_2)
print(f'M2n = {M3n:.4}')
```

    M1n = 1.262
    M2n = 1.233

We can see that the second shock occurs at a smaller Mach number, so it is **weaker**.


# Models, Data, Learning Problems

In this lab we start our first data analysis on a concrete problem. We are using Fisher's famous Iris data set. The goal is to classify flowers from the Iris family into one of three species, that look as follows:

Iris Setosa
Iris Versicolor
Iris Virginica

Our data set contains 50 flowers from each class, thus 150 in total. There are four features, the length and width of the petal (dt. Kronblatt) and sepal (dt. Kelchblatt) in centimeters.

Your goal is to go through the notebook, understand premade code and text as well as filling blanks and exercises left for you. You may also edit the notebook as you wish. A good way to learn is to add comments (lines starting with #) or modifying the code and see what changes.

The data set is distributed with sci-kit learn, the only thing we have to do is to important a function and call it.

```python
from sklearn.datasets import load_iris

data = load_iris()
X = data.data
y = data.target
print(type(X))
print(X.shape)

print(f"First three rows of data\n{X[:3]}")
print(f"First three labels: {y[:3]}")
```

    <class 'numpy.ndarray'>
    (150, 4)
    First three rows of data
    [[5.1 3.5 1.4 0.2]
     [4.9 3.  1.4 0.2]
     [4.7 3.2 1.3 0.2]]
    First three labels: [0 0 0]

 Not only do we get the input matrix $X \in \mathbb{R}^{150 \times 4}$ and target $y \in \mathbb{R}^{150}$, but also meta information such as what the class labels $0, 1, 2$ stand for and what the features (i.e. columns of $X$) correspond to.

```python
print(data.target_names)
print(data.feature_names)
```

    ['setosa' 'versicolor' 'virginica']
    ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']

As a first step we focus our analysis on the first two variables, the sepal length and sepal width. Since we obtain a representation of the data in two dimensions, we are able to plot it.

```python
X_2 = X[:, :2]
y_2 = y
```

```python
# Configures Jupyter to show graphics in the notebook
%matplotlib inline
from matplotlib import pyplot as plt # standard import

# We write a function so we can reuse it later.
def generate_scatter_plot(X, y):
    class_names = data.target_names
    class_colors = ['blue','yellow','green']

    fig = plt.figure(figsize=(12, 6)) # increase size of plot

    for i, class_color in enumerate(class_colors):
        # plot the points only of this class label
        plt.scatter(X[y == i, 0], X[y == i, 1], c=class_color, label=class_names[i])

    plt.xlabel(data.feature_names[0]) # label the axis
    plt.ylabel(data.feature_names[1])
    plt.legend(loc="best") # with legend

generate_scatter_plot(X_2, y)
```

We see that we could discriminate the iris setosa linearly from other two species. The linear function could even have a slope of about $1$. Let us substitute the first feature with the difference of the two features.

```python
import numpy as np
x_new = X[:, 0] - X[:, 1]
X_new = np.column_stack((x_new, X[:, 1]))
print(X_new.shape)
generate_scatter_plot(X_new, y)
plt.xlabel("sepal length - sepal width")
```

Remember that our main goal is to find a model,

$$ y_\theta: X \rightarrow Y $$

such that $y_\theta(x)$ models the knowledge we got from our training data plus the inductive bias. The plot gives the decision rule (or part of):

"If sepal length - sepal width $\leq$ 2.2 $\rightarrow$ Classify iris setosa"

Exercise 1:

Implement the naive decision rule as given above. If the condition for iris setosa is not fulfilled, classify the result as 'iris versicolor'.

```python
def naive_decision_rule(x):
    # X matrix with 4 dimensional columns
    # returns the expected class label for each row of X (0 = setosa, 1 = versicolor, 2 = virginica)

    # FILL IN
    x_new = x[0] - x[1]
    pred_class =  0 if (x_new <= 2.2) else 1
    return pred_class

```

The following function takes a decision rule (or model) and a matrix of data points to generate the predictions for this matrix.

```python
def predict(model, X):
    """Builds prediction on a matrix X given a model for each data point in a row.
    Returns a flat vector of predictions.
    """
    return np.apply_along_axis(model, axis=1, arr=X)

y_pred = predict(naive_decision_rule, X)
print(y_pred[:50]) # print first 50 predictions
print(y_pred) # print all 150
```

    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0]
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1]

The predictions of the first 50 numbers should be zero and one for all others. Now we have to judge the quality of our model, we do this by using the zero-loss function of the lecture.

Exercise 2:

Implement the zero loss function as defined in the lecture,
$$
\begin{align}
l(x_i, y_i; \theta) &= l_{0/1}(y_\theta(x_i), y_i) = \begin{cases} 0, & \mbox{ if } y_\theta(x_i) = y_i \\ 1, & \mbox{ otherwise } \end{cases} \\
l(X, y; \theta) &= \sum_i{ l(x_i, y_i; \theta). }
\end{align}
$$
In lay-man terms one counts how often the label predicted differed from the observed label.

```python
def zero_one_loss(y_pred, y_true,loss):
    # FILL IN
    #print (y_pred)
    #print (y_true)

    for i in range(len(y_pred)):
        if y_pred[i] - y_true[i] != 0:
            loss= loss+1

    return loss

```

```python
print(f"The 0-1-loss of the naive decision rule is {zero_one_loss(y_pred, y,loss=0)} (should be 50)")
```

    The 0-1-loss of the naive decision rule is 50 (should be 50)

Exercise 3:

Improve the decision rule to have a maximum number of missclassifications of $10$. As an informal constraint use "Occams Razor" as an inductive bias, i.e. as simplest as possible.

Discussion topic: Why could a complex model with zero missclassifications perform worse in reality (we got out and measure new flowers) than a simple model with more misclassifications?

```python
import numpy as np
x_new1 = X[:, 2]- X[:, 3]
X_new1 = np.column_stack((x_new1, X[:, 3]))
print(X_new.shape)
generate_scatter_plot(X_new1, y)
plt.xlabel("P length - P width")
```

```python
# Place for your analysis.
def my_decision_rule(x):
    # Take 'petal length (cm)', 'petal width (cm)' into account
     # 'petal width (cm)' at Y axis
    x_new = x[2] - x[3]
    if (x_new <= 1.8):
        pred_class =  0
    elif ((x_new >= 1.8) & (x_new <= 3.6)) & (x[3] <= 1.6) :
        pred_class = 1
    else:
        pred_class =  2
    return pred_class
    pass
```

```python
# Evaluation script
y_pred = predict(my_decision_rule, X)
print(y_pred)
loss = zero_one_loss(y_pred, y,loss=0)
print(f"Your loss {loss}.")

if loss <= 10:
    print("You have made it!")
else:
    print("Uhm, try again. Maybe you have flipped some class?")
```

    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1
     1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2]
    Your loss 4.
    You have made it!

```python

```

Lab2/.ipynb_checkpoints/Models, Data, Learning Problems-checkpoint.ipynb


## Simple neural network in plain Python

This notebook implements a simple neural network architecture that can map $2$ dimensional input vectors onto binary output values. Our network will have $2$ input neurons, one hidden layer with $6$ hidden neurons and an output layer with $1$ output neuron.

We will represent the architecture by means of the weight matrices between the layers. In our example, the weight matrix between the input and hidden layer will be denoted as $W_h$, the weight matrix between the hidden and output layer as $W_o$. In addition to the weights connecting the neurons, each hidden and output neuron will have a bias weight with a constant input of $+1$.

Our training set consists of $m = 750$ examples. Therefore, we will have the following matrix shapes:
- Training set shape: $X = (750, 2)$
- Targets shape: $Y = (750, 1)$
- $W_h$ shape: $(n_{features}, n_{hidden}) = (2, 6)$
- $b_h$ shape (bias vector): $(1, n_{hidden}) = (1, 6)$
- $W_o$ shape: $(n_{hidden}, n_{outputs}) = (6, 1)$
- $b_o$ shape (bias vector): $(1, n_{outputs}) = (1, 1)$

### Loss Function

We will use the same loss function as in logistic regression:

\begin{equation}
J(\boldsymbol{w},b) = - \frac{1}{m} \sum_{i=1}^m \Big[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \big(1 - \log(\hat{y}^{(i)})\big) \Big]
\end{equation}

For a classification task with more than two classes, we would use a generalization of this function, namely the categorical cross-entropy.

### Training

We will train our network with gradient descent and we will use backpropagation to compute the required partial derivatives. The training procedure has the following steps:
1. Initialize the parameters (i.e. the weights and biases)
2. Repeat until convergence:
2.1. Propagate the current input batch forward through the network. To do so, compute the activations and outputs of all hidden and output units.
2.2 Compute the partial derivatives of the loss function with respect to each parameter
2.3 Update the parameters

### Forward Pass

We start by computing the activation and output of each unit in our network. To speed up the implementation, we won't do this for each input example individually but for all examples at once, using vectorization. We will use the following notation:

- $\boldsymbol{A}_h$: matrix with activations of all hidden units for all training examples
- $\boldsymbol{O}_h$: matrix with outputs of all hidden units for all training examples

The hidden neurons will have $\tanh$ as their activation function:
\begin{equation}
\tanh(x) = \frac{sinh(x)}{cosh(x)} = \frac{\exp(x) - exp(-x)}{\exp(x) + exp(-x)}
\end{equation}
\begin{equation}
\tanh'(x) = 1 - tanh^2(x)
\end{equation}

The output neurons will have the $\textit{sigmoid}$ activation function:
\begin{equation}
\sigma(x) = \frac{1}{1 + \exp(-x)}
\end{equation}
\begin{equation}
\sigma'(x) = 1 - (1 + \sigma(x))
\end{equation}

The activations and outputs can then be computed as follows ($\cdot$ denotes the dot product):

\begin{equation}
\boldsymbol{A}_h = \boldsymbol{X} \cdot \boldsymbol{W}_h + \boldsymbol{b}_h, \text{shape: } (750, 6)
\end{equation}
\begin{equation}
\boldsymbol{O}_h = \sigma(\boldsymbol{A}_h), \text{shape: } (750, 6)
\end{equation}

\begin{equation}
\boldsymbol{A}_o = \boldsymbol{O}_h \cdot \boldsymbol{W}_o + b_o, \text{shape: } (750, 1)
\end{equation}
\begin{equation}
\boldsymbol{O}_o = \sigma(\boldsymbol{A}_o), \text{shape: } (750, 1)
\end{equation}

### Backward pass

To compute the weight updates we need the partial derivatives of the loss function with respect to each unit. I won't give the derivation of these equations here, you will find plenty of good explanations on other websites (for example [here](https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/)).

For the output neurons, the gradients are given by (matrix notation):

$\frac{\partial L}{\partial \boldsymbol{A}_o} = d\boldsymbol{A}_o = (\boldsymbol{O}_o - \boldsymbol{Y})$

$\frac{\partial L}{\partial \boldsymbol{W}_o} = \frac{1}{m} (\boldsymbol{O}_h^T \cdot d\boldsymbol{A}_o)$

$\frac{\partial L}{\partial \boldsymbol{b}_o} = \frac{1}{m} \sum d\boldsymbol{A}_o$

For the weight matrix between input and hidden layer we have:
$\frac{\partial L}{\partial \boldsymbol{A}_h} = d\boldsymbol{A}_h = (\boldsymbol{W}_o^T \cdot d\boldsymbol{A}_o) * (1 - \tanh^2 (\boldsymbol{A}_h))$

$\frac{\partial L}{\partial \boldsymbol{W}_h} = \frac{1}{m} (\boldsymbol{X}^T \cdot d\boldsymbol{A}_h)$

$\frac{\partial L}{\partial \boldsymbol{b}_h} = \frac{1}{m} \sum d\boldsymbol{A}_h$

### Weight Update

$\boldsymbol{W}_h = \boldsymbol{W}_h - \eta * \frac{\partial L}{\partial \boldsymbol{W}_h}$

$\boldsymbol{b}_h = \boldsymbol{b}_h - \eta * \frac{\partial L}{\partial \boldsymbol{b}_h} $

$\boldsymbol{W}_o = \boldsymbol{W}_o - \eta *   \frac{\partial L}{\partial \boldsymbol{W}_o} $

$\boldsymbol{b}_o = \boldsymbol{b}_o - \eta *  \frac{\partial L}{\partial \boldsymbol{b}_o} $

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
np.random.seed(123)

% matplotlib inline
```

    UsageError: Line magic function `%` not found.

## Neural Network Class

Some parts of this implementation are inspired by the exercises of Andrew Ng's [coursera course](https://www.coursera.org/learn/neural-networks-deep-learning)

```python
class NeuralNet():

    def __init__(self, n_inputs, n_outputs, n_hidden):
        self.n_inputs = n_inputs
        self.n_outputs = n_outputs
        self.hidden = n_hidden

        # Initialize weight matrices and bias vectors
        self.W_h = np.random.randn(self.n_inputs, self.hidden)
        self.b_h = np.zeros((1, self.hidden))
        self.W_o = np.random.randn(self.hidden, self.n_outputs)
        self.b_o = np.zeros((1, self.n_outputs))

    def sigmoid(self, a):
        return 1 / (1 + np.exp(-a))

    def relu(self, a):
        return np.maximum(a, 0)

    def forward_pass(self, X):
        """
        Propagates the given input X forward through the net.

        Returns:
            A_h: matrix with activations of all hidden neurons for all input examples
            O_h: matrix with outputs of all hidden neurons for all input examples
            A_o: matrix with activations of all output neurons for all input examples
            O_o: matrix with outputs of all output neurons for all input examples
        """
        # Compute activations and outputs of hidden units
        A_h = np.dot(X, self.W_h) + self.b_h
        O_h = np.tanh(A_h)

        # Compute activations and outputs of output units
        A_o = np.dot(O_h, self.W_o) + self.b_o
        O_o = self.sigmoid(A_o)

        outputs = {
                "A_h": A_h,
                "A_o": A_o,
                "O_h": O_h,
                "O_o": O_o,
                }

        return outputs

    def cost(self, y_true, y_predict, n_samples):
        """
        Computes and returns the cost over all examples
        """
        # same cost function as in logistic regression
        cost = (- 1 / n_samples) * np.sum(y_true * np.log(y_predict) + (1 - y_true) * (np.log(1 - y_predict)))
        cost = np.squeeze(cost)
        assert isinstance(cost, float)

        return cost

    def backward_pass(self,  X, Y, n_samples, outputs):
        """
        Propagates the errors backward through the net.

        Returns:
            dW_h: partial derivatives of loss function w.r.t hidden weights
            db_h: partial derivatives of loss function w.r.t hidden bias
            dW_o: partial derivatives of loss function w.r.t output weights
            db_o: partial derivatives of loss function w.r.t output bias
        """

        dA_o = (outputs["O_o"] - Y)
        dW_o = (1 / n_samples) * np.dot(outputs["O_h"].T, dA_o)
        db_o = (1 / n_samples) * np.sum(dA_o)

        dA_h = (np.dot(dA_o, self.W_o.T)) * (1 - np.power(outputs["O_h"], 2))
        dW_h = (1 / n_samples) * np.dot(X.T, dA_h)
        db_h = (1 / n_samples) * np.sum(dA_h)

        gradients = {
                "dW_o": dW_o,
                "db_o": db_o,
                "dW_h": dW_h,
                "db_h": db_h,
                }

        return gradients

    def update_weights(self, gradients, eta):
        """
        Updates the model parameters using a fixed learning rate
        """
        self.W_o = self.W_o - eta * gradients["dW_o"]
        self.b_o = self.b_o - eta * gradients["db_o"]
        self.W_h = self.W_h - eta * gradients["dW_h"]
        self.b_h = self.b_h - eta * gradients["db_h"]

    def train(self, X, y, n_iters=500, eta=0.3):
        """
        Trains the neural net on the given input data
        """
        n_samples, _ = X.shape

        for i in range(n_iters):
            outputs = self.forward_pass(X)
            cost = self.cost(y, outputs["O_o"], n_samples=n_samples)
            gradients = self.backward_pass(X, y, n_samples, outputs)

            if i % 100 == 0:
                print(f'Cost at iteration {i}: {np.round(cost, 4)}')

            self.update_weights(gradients, eta)

    def predict(self, X):
        """
        Computes and returns network predictions for given dataset
        """
        outputs = self.forward_pass(X)
        y_pred = [1 if elem >= 0.5 else 0 for elem in outputs["O_o"]]

        return np.array(y_pred).reshape(-1, 1)
```

## Dataset

```python
X, y = make_circles(n_samples=1000, factor=0.5, noise=.1)

fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.title("Dataset")
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.show()
```

```python
# reshape targets to get column vector with shape (n_samples, 1)
y_true = y[:, np.newaxis]
# Split the data into a training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y_true)

print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape}')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
```

    Shape X_train: (750, 2)
    Shape y_train: (750, 1)
    Shape X_test: (250, 2)
    Shape y_test: (250, 1)

## Initializing and training the neural network

```python
nn = NeuralNet(n_inputs=2, n_hidden=6, n_outputs=1)
print("Shape of weight matrices and bias vectors:")
print(f'W_h shape: {nn.W_h.shape}')
print(f'b_h shape: {nn.b_h.shape}')
print(f'W_o shape: {nn.W_o.shape}')
print(f'b_o shape: {nn.b_o.shape}')
print()

print("Training:")
nn.train(X_train, y_train, n_iters=2000, eta=0.7)
```

    Shape of weight matrices and bias vectors:
    W_h shape: (2, 6)
    b_h shape: (1, 6)
    W_o shape: (6, 1)
    b_o shape: (1, 1)

    Training:
    Cost at iteration 0: 0.8033
    Cost at iteration 100: 0.3248
    Cost at iteration 200: 0.1751
    Cost at iteration 300: 0.1285
    Cost at iteration 400: 0.1069
    Cost at iteration 500: 0.0938
    Cost at iteration 600: 0.0847
    Cost at iteration 700: 0.0778
    Cost at iteration 800: 0.0724
    Cost at iteration 900: 0.068
    Cost at iteration 1000: 0.0643
    Cost at iteration 1100: 0.0611
    Cost at iteration 1200: 0.0582
    Cost at iteration 1300: 0.0557
    Cost at iteration 1400: 0.0534
    Cost at iteration 1500: 0.0513
    Cost at iteration 1600: 0.0494
    Cost at iteration 1700: 0.0476
    Cost at iteration 1800: 0.046
    Cost at iteration 1900: 0.0445

## Testing the neural network

```python
n_test_samples, _ = X_test.shape
y_predict = nn.predict(X_test)
print(f"Classification accuracy on test set: {(np.sum(y_predict == y_test)/n_test_samples)*100} %")
```

    Classification accuracy on test set: 99.2 %

## Visualizing the decision boundary

In the lowermost plot we can see which parts of the input space are classified as positive and which are classified as negative by the trained network.

```python
X_temp, y_temp = make_circles(n_samples=60000, noise=.5)
y_predict_temp = nn.predict(X_temp)
y_predict_temp = np.ravel(y_predict_temp)
```

```python
fig = plt.figure(figsize=(8,12))
ax = fig.add_subplot(2,1,1)
plt.scatter(X[:,0], X[:,1], c=y)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.title("Training and test set")

ax = fig.add_subplot(2,1,2)
plt.scatter(X_temp[:,0], X_temp[:,1], c=y_predict_temp)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.title("Decision boundary")
```

```python

```


# Cálculo de estabilidad a partir de valores propios y matriz de Jordan

## Caracterización de estabilidad ecuaciones lineales de coenficientes constantes

Sea $A \in M_d( \mathbb R)$,  $\sigma(A) = \{ \lambda_1, ..., \lambda_d \}$ el radio espectral, es decir el conjunto de valores própios de $A$.
Denotaremos por $m(\lambda_i)$ a la multiplicidad algebraica, (el número de veces que aparece cierto vector en el radio espectral.

La dimensión geomtrica es la dimensión del espacio asociado, calculada como: $dim E_{\lambda _i} = dim_ker( A - \lambda_i I) - rango(A - \lambda_i I)$, a efectos prácticas para nosotros el número de filas que sean cero salvo el elemento de la diagonal y que coincidad con ese valor propio.

Llamaremos a $\mu(A)$ al máximo de las partes reales del radio espectral.

Tenemos pues que
1. Si $\mu(A) < 0$ entonces EDO lineal es **asintóticamente estable**.
2. Si $\mu(A) > 0$ entonces EDO lineal es **inestable**.
3. Si $\mu(A) =  0$ y la dimensión algebraica coincide con la geométrica para todo valor propio, entonces EDO lineal es **estable** (No asintóticamente estable).
4. Si $\mu(A) =  0$ y la dimensión algebraica NO coincide con la geométrica para algún valor propio, entonces EDO lineal es **inestable**.

### Documentación:
- Valores propios: https://docs.sympy.org/latest/tutorial/matrices.html

- Matriz de jordan: https://docs.sympy.org/latest/modules/matrices/matrices.html

- Para cálculo numérico: https://www.math.ubc.ca/~pwalls/math-python/linear-algebra/eigenvalues-eigenvectors/

Análisis para la matriz $A$:

$\mu(A) = 0$ y para $i y -i$ la multiplicidad algebraica no coinciden con la geométrica, luego no es estable.

```python
import sympy as sp

A = sp.Matrix (
    [
        [2,    2,  3, -3, -1],
        [1,    3,  4, -5, 0],
        [-10, -6,-13, 10, 4],
        [-7, -2, -7, 4, 3],
        [-8, -4, -10, 6, 3]
    ]
)
P , J = A.jordan_form() # A = P J P^{-1}

print('Jordan Matrix: ')
display(J)

print('valores propios:', A.eigenvals()) # dicionario con valor propios: multiplicidad algebraica

t = sp.symbols("t")
(A*t).sp.exp()
```

```python
# Por si acaso el cálculo simbólico no funciona utilizar numpy:

import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as la

# Matriz

M = np.array(
    [
        [2,    2,  3, -3, -1],
        [1,    3,  4, -5, 0],
        [-10, -6,-13, 10, 4],
        [-7, -2, -7, 4, 3],
        [-8, -4, -10, 6, 3]
    ]
)

eigvals, eigvecs = la.eig(M) # almacena en 0 los valores propios, en 1 lso vectoris propios

#np.set_printoptions(precision=2)
print( np.around(eigvals, decimals = 5))

```

    [-1.+0.j  0.+1.j  0.-1.j -0.+1.j -0.-1.j]

```python
A = sp.Matrix (
    [
        [2,   6,  -5],
        [ 2,  10,  -7],
        [4,  18 , -13]
    ]
)
A = sp.Matrix (
    [
        [ 6,  18, -13],
        [4, 16, -11],
        [8, 30, -21]
    ]
)

P , J = A.jordan_form() # A = P J P^{-1}

print('Jordan Matrix: ')
display(J)

print('valores propios:', A.eigenvals()) # dicionario con valor propios: multiplicidad algebraica

```

    Jordan Matrix:

$\displaystyle \left[\begin{matrix}-2 & 0 & 0\\0 & 1 & 0\\0 & 0 & 2\end{matrix}\right]$

    valores propios: {2: 1, 1: 1, -2: 1}