# Solving equations

You will learn about working with matrices and linear algebra (**scipy.linalg**), including solving systems of linear equations. You will learn to find roots of linear and non-linear equations both numerically (**scipy.optimize**) and symbolically (**sympy**).

**Note:** The algorithms written here is to illustrate the underlying algorithms. The scipy implementations are always both the *fastest* and the *safest* choice.

**Links:**

1. **scipy.linalg:** [overview](https://docs.scipy.org/doc/scipy/reference/linalg.html) + [tutorial](https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html)
2. **sympy:** [overview](https://docs.sympy.org/latest/index.html) + [tutorial](https://docs.sympy.org/latest/tutorial/index.html#tutorial)
3. **scipy.optimize:** [overview](https://docs.scipy.org/doc/scipy/reference/optimize.html) + [turtorial](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html)

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')
import ipywidgets as widgets
import time
from scipy import linalg
from scipy import optimize
import sympy as sm

# local module for linear algebra
%load_ext autoreload
%autoreload 2
import numecon_linalg
```

# Systems of linear equations

## Introduction

We consider **matrix equations** with $n$ equations and $n$ unknowns:

$$
\begin{aligned}
Ax = b \Leftrightarrow
\begin{bmatrix}a_{11} & a_{12} & \cdots & a_{1n}\\
a_{21} & a_{22} & \cdots & a_{2n}\\
\vdots & \vdots & \ddots & \vdots\\
a_{n1} & a_{n2} & \cdots & a_{nn}
\end{bmatrix}\cdot\begin{bmatrix}x_{1}\\
x_{2}\\
\vdots\\
x_{n}
\end{bmatrix} & = & \begin{bmatrix}b_{1}\\
b_{2}\\
\vdots\\
b_{n}
\end{bmatrix}
\end{aligned}
$$

where $A$ is a square parameter matrix, $b$ is a parameter vector, and $x$ is the vector of unknowns.

A specific **example** could be:

$$
\begin{aligned}
Ax = b \Leftrightarrow
\begin{bmatrix}
3 & 2 & 0 \\
1 & -1 & 0 \\
0 & 5 & 1
\end{bmatrix} \cdot
\begin{bmatrix}
x_1 \\
x_2 \\
x_3
\end{bmatrix} \,=\,
\begin{bmatrix}
2 \\
4 \\
-1
\end{bmatrix}
\end{aligned}
$$

**How to solve this?**

```python
A = np.array([[3.0, 2.0, 0.0], [1.0, -1.0, 0], [0.0, 5.0, 1.0]])
b = np.array([2.0, 4.0, -1.0])
```

Trial-and-error:

```python
Ax = A@[2,-2,9] # @ is matrix multiplication
print('A@x: ',Ax)

if np.allclose(Ax,b):
    print('solution found')
else:
    print('solution not found')
```

    A@x:  [ 2.  4. -1.]
    solution found

**Various matrix operations:**

```python
A.T # transpose
```

    array([[ 3.,  1.,  0.],
           [ 2., -1.,  5.],
           [ 0.,  0.,  1.]])

```python
np.diag(A) # diagonal
```

    array([ 3., -1.,  1.])

```python
np.tril(A) # lower triangular matrix
```

    array([[ 3.,  0.,  0.],
           [ 1., -1.,  0.],
           [ 0.,  5.,  1.]])

```python
np.triu(A) # upper triangular matrix
```

    array([[ 3.,  2.,  0.],
           [ 0., -1.,  0.],
           [ 0.,  0.,  1.]])

```python
B = A.copy()
np.fill_diagonal(B,0) # fill diagonal with zeros
print(B)
```

    [[0. 2. 0.]
     [1. 0. 0.]
     [0. 5. 0.]]

```python
linalg.inv(A) # inverse
```

    array([[ 2.00000000e-01,  4.00000000e-01,  1.85037171e-18],
           [ 2.00000000e-01, -6.00000000e-01, -7.40148683e-18],
           [-1.00000000e+00,  3.00000000e+00,  1.00000000e+00]])

```python
linalg.eigvals(A) # eigen values
```

    array([ 1.        +0.j, -1.44948974+0.j,  3.44948974+0.j])

## Direct solution with Gauss-Jordan elimination

Consider the column stacked matrix:

$$
X=[A\,|\,b]=\begin{bmatrix}a_{11} & a_{12} & \cdots & a_{1n} & b_{1}\\
a_{21} & a_{22} & \cdots & a_{2n} & b_{2}\\
\vdots & \vdots & \ddots & \vdots & \vdots\\
a_{n1} & a_{n2} & \cdots & a_{nn} & b_{n}
\end{bmatrix}
$$

Find the **row reduced echelon form** by performing row operations, i.e.

1. Multiply row with constant
2. Swap rows
3. Add one row to another row,

until the $A$ part of the matrix is the identity matrix.

**Manually:**

```python
# a. stack
X = np.column_stack((A,b))
print('stacked:\n',X)

# b. row operations
X[0,:] += 2*X[1,:]
X[0,:] /= 5.0
X[1,:] -= X[0,:]
X[1,:] *= -1
X[2,:] -= 5*X[1,:]
print('row reduced echelon form:\n',X)

# c. print result (the last column in X in row reduced echelon form)
print('solution',X[:,-1])
```

    stacked:
     [[ 3.  2.  0.  2.]
     [ 1. -1.  0.  4.]
     [ 0.  5.  1. -1.]]
    row reduced echelon form:
     [[ 1.  0.  0.  2.]
     [-0.  1. -0. -2.]
     [ 0.  0.  1.  9.]]
    solution [ 2. -2.  9.]

**General function:**

```python
Y = np.column_stack((A,b))
numecon_linalg.gauss_jordan(Y)
print('solution',Y[:,-1])
```

    solution [ 2. -2.  9.]

which can also be used to find the inverse if we stack with the identity matrix instead,

```python
# a. construct stacked matrix
Z = np.hstack((A,np.eye(3)))
print('stacked:\n',Z)

# b. apply gauss jordan elimination
numecon_linalg.gauss_jordan(Z)

# b. find inverse
inv_Z = Z[:,3:] # last 3 columns of Z in row reduced echelon form
print('inverse:\n',inv_Z)

assert np.allclose(Z[:,3:]@A,np.eye(3))
```

    stacked:
     [[ 3.  2.  0.  1.  0.  0.]
     [ 1. -1.  0.  0.  1.  0.]
     [ 0.  5.  1.  0.  0.  1.]]
    inverse:
     [[ 0.2  0.4  0. ]
     [ 0.2 -0.6  0. ]
     [-1.   3.   1. ]]

## Iteative Gauss-Seidel (+)

We can always decompose $A$ into additive lower and upper triangular matrices,

$$
A=L+U=\begin{bmatrix}a_{11} & 0 & \cdots & 0\\
a_{21} & a_{22} & \cdots & 0\\
\vdots & \vdots & \ddots & \vdots\\
a_{n1} & a_{n2} & \cdots & a_{nn}
\end{bmatrix}+\begin{bmatrix}0 & a_{12} & \cdots & a_{1n}\\
0 & 0 & \cdots & a_{2n}\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & 0
\end{bmatrix}
$$

such that

$$
Ax=b\Leftrightarrow Lx=b-Ux
$$

**Algorithm:** `gauss_seidel()`

1. Choose tolerance $\epsilon > 0$, guess on $x_0$, and set $n=1$.
2. Find $x_n$ by solving \\( Lx_n = y \equiv (b-Ux_{n-1}) \\).
3. If $|x_n-x_{n-1}|_{\infty} < \epsilon$ stop, else $n = n +1 $ and return to step 2.

> **Note:** Step 2 is very easy because the equation can be solved directly by *forward substitution*:
>
> $x_1 = \frac{y_1}{a_{11}}$
>
> $x_2 = \frac{(y_2 - a_{21} x_1)}{a_{22}}$
>
> $x_3 = \frac{(y_3 - a_{31} x_1 - a_{32} x_2)}{a_{33}}$
>
> etc.

**Apply Gauss-Seidel:**

```python
x0 = np.array([1,1,1])
x =  numecon_linalg.gauss_seidel(A,b,x0)
print('solution',x)
```

    solution [ 2. -2.  9.]

> **Note:** Convergence is not ensured unless the matrix is *diagonally dominant* or *symmetric* and *positive definite*.

```python
x =  numecon_linalg.gauss_seidel(A,b,x0,do_print=True)
```

       [1 1 1]
     0 [ 0. -4. 19.]
     1 [ 3.33333333 -0.66666667  2.33333333]
     2 [ 1.11111111 -2.88888889 13.44444444]
     3 [ 2.59259259 -1.40740741  6.03703704]
     4 [ 1.60493827 -2.39506173 10.97530864]
     5 [ 2.26337449 -1.73662551  7.68312757]
     6 [ 1.82441701 -2.17558299  9.87791495]
     7 [ 2.11705533 -1.88294467  8.41472337]
     8 [ 1.92196312 -2.07803688  9.39018442]
     9 [ 2.05202459 -1.94797541  8.73987705]
    10 [ 1.96531694 -2.03468306  9.1734153 ]
    11 [ 2.02312204 -1.97687796  8.8843898 ]
    12 [ 1.98458531 -2.01541469  9.07707347]
    13 [ 2.01027646 -1.98972354  8.94861769]
    14 [ 1.99314903 -2.00685097  9.03425487]
    15 [ 2.00456732 -1.99543268  8.97716342]
    16 [ 1.99695512 -2.00304488  9.01522439]
    17 [ 2.00202992 -1.99797008  8.98985041]
    18 [ 1.99864672 -2.00135328  9.00676639]
    19 [ 2.00090219 -1.99909781  8.99548907]
    20 [ 1.99939854 -2.00060146  9.00300729]
    21 [ 2.00040097 -1.99959903  8.99799514]
    22 [ 1.99973269 -2.00026731  9.00133657]
    23 [ 2.00017821 -1.99982179  8.99910895]
    24 [ 1.99988119 -2.00011881  9.00059403]
    25 [ 2.0000792  -1.9999208   8.99960398]
    26 [ 1.9999472  -2.0000528   9.00026401]
    27 [ 2.0000352  -1.9999648   8.99982399]
    28 [ 1.99997653 -2.00002347  9.00011734]
    29 [ 2.00001565 -1.99998435  8.99992177]
    30 [ 1.99998957 -2.00001043  9.00005215]
    31 [ 2.00000695 -1.99999305  8.99996523]
    32 [ 1.99999536 -2.00000464  9.00002318]
    33 [ 2.00000309 -1.99999691  8.99998455]
    34 [ 1.99999794 -2.00000206  9.0000103 ]
    35 [ 2.00000137 -1.99999863  8.99999313]
    36 [ 1.99999908 -2.00000092  9.00000458]
    37 [ 2.00000061 -1.99999939  8.99999695]
    38 [ 1.99999959 -2.00000041  9.00000203]
    39 [ 2.00000027 -1.99999973  8.99999864]
    40 [ 1.99999982 -2.00000018  9.0000009 ]
    41 [ 2.00000012 -1.99999988  8.9999994 ]
    42 [ 1.99999992 -2.00000008  9.0000004 ]
    43 [ 2.00000005 -1.99999995  8.99999973]
    44 [ 1.99999996 -2.00000004  9.00000018]
    45 [ 2.00000002 -1.99999998  8.99999988]
    46 [ 1.99999998 -2.00000002  9.00000008]
    47 [ 2.00000001 -1.99999999  8.99999995]
    48 [ 1.99999999 -2.00000001  9.00000004]
    49 [ 2.         -2.          8.99999998]
    50 [ 2.         -2.          9.00000002]
    51 [ 2.         -2.          8.99999999]
    52 [ 2.         -2.          9.00000001]
    53 [ 2. -2.  9.]
    54 [ 2. -2.  9.]

## Scipy functions

**Option 1:** Use `.solve()` (scipy chooses what happens).

```python
x1 = linalg.solve(A, b)
print(x1)
assert np.all(A@x1 == b)
```

    [ 2. -2.  9.]

**Option 2:** Compute `.inv()` first and then solve.

```python
Ainv = linalg.inv(A)
x2 = Ainv@b
print(x2)
```

    [ 2. -2.  9.]

> **Note:** Computing the inverse is normally not a good idea due to numerical stability.

**Option 3:** Compute LU decomposition and then solve.

```python
LU,piv = linalg.lu_factor(A) # decomposition (factorization)
x3 = linalg.lu_solve((LU,piv),b)
print(x3)
```

    [ 2. -2.  9.]

**Detail:** `piv` contains information on a numerical stable reordering.

## Comparisons

1. `linalg.solve()` is the best choice for solving once.
2. `linalg.lu_solve()` is the best choice when solving for multipe $b$'s for a fixed $A$ (the LU decomposition only needs to be done once).
3. Gauss-Seidel is an alternative when e.g. only an approximate solution is needed.

## Details on LU factorization (+)

When $A$ is *regular* (invertible), we can decompose it into a *lower unit triangular matrix*, $L$, and an *upper triangular matrix*, $U$:

$$
A= L\cdot U = \begin{bmatrix}1 & 0 & \cdots & 0\\
l_{21} & 1 & \cdots & 0\\
\vdots & \vdots & \ddots & \vdots\\
l_{n1} & l_{n2} & \cdots & 1
\end{bmatrix}\cdot\begin{bmatrix}u_{11} & u_{12} & \cdots & u_{1n}\\
0 & u_{22} & \cdots & u_{2n}\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & u_{nn}
\end{bmatrix}
$$

where it can be shown that we can compute the elements by

$$
\begin{aligned}
u_{ij} &= a_{ij} - \sum_{k=1}^{i-1} u_{kj} l_{ik} \\
l_{ij} &= \frac{1}{u_{jj}} \big( a_{ij} - \sum_{k=1}^{j-1} u_{kj} l_{ik} \big)
\end{aligned}
$$

This implies that the equation system can be written

$$
L(Ux) = b
$$

**Algorithm:** `lu_solve()`

1. Perform LU decomposition (factorization)
2. Solve $Ly = b$ for $y$ (by *forward substitution*) where $y = Ux$
3. Solve $Ux = y$ for $x$ (by *backward substitution*)

```python
L,U = numecon_linalg.lu_decomposition(A) # step 1
y = numecon_linalg.solve_with_forward_substitution(L,b) # step 2
x = numecon_linalg.solve_with_backward_substitution(U,y) # step 3
print('L:\n',L)
print('\nU:\n',U)
print('\nsolution:',x)
```

    L:
     [[ 1.          0.          0.        ]
     [ 0.33333333  1.          0.        ]
     [ 0.         -3.          1.        ]]

    U:
     [[ 3.          2.          0.        ]
     [ 0.         -1.66666667  0.        ]
     [ 0.          0.          1.        ]]

    solution: [ 2. -2.  9.]

**Relation to scipy:**

1. Scipy use pivoting to improve numerical stability.
2. Scipy is implemented much much better than here.

## Sparse matrices (+)

**Sparse matrix:** A matrix with many zeros. Letting the computer know where they are is extremely valuable.

**Documentation:** [basics](https://docs.scipy.org/doc/scipy/reference/sparse.html) + [linear algebra](https://docs.scipy.org/doc/scipy/reference/sparse.linalg.html#module-scipy.sparse.linalg)

**Create a sparse matrix**, where most elements are on the diagonal:

```python
from scipy import sparse
import scipy.sparse.linalg

S = sparse.lil_matrix((1000, 1000)) # 1000x1000 matrix with zeroes
S.setdiag(np.random.rand(1000)) # some values on the diagonal
S[200, :100] = np.random.rand(100) # some values in a row
S[200:210, 100:200] = S[200, :100] # and the same value in some other rows
```

Create a plot of the values in the matrix:

```python
S_np = S.toarray() # conversion to numpy
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.matshow(S_np,cmap=plt.cm.binary);
```

**Solve it in four different ways:**

1. Like it was not sparse
2. Using the sparsity
3. Using the sparsity + explicit factorization
4. Iterative solver (similar to Gauss-Seidel)

```python
k = np.random.rand(1000) # random RHS

# a. solve
t0 = time.time()
x = linalg.solve(S_np,k)
print(f'{"solve":12s}: {time.time()-t0:.5f} secs')

# b. solve with spsolve
t0 = time.time()
x_alt = sparse.linalg.spsolve(S.tocsr(), k)
print(f'{"spsolve":12s}: {time.time()-t0:.5f} secs')
assert np.allclose(x,x_alt)

# c. solve with explicit factorization
t0 = time.time()
S_solver = sparse.linalg.factorized(S.tocsc())
x_alt = S_solver(k)
print(f'{"factorized":12s}: {time.time()-t0:.5f} secs')
assert np.allclose(x,x_alt)

# d. solve with iterative solver (bicgstab)
t0 = time.time()
x_alt,_info = sparse.linalg.bicgstab(S,k,x0=1.001*x,tol=10**(-8))
print(f'{"bicgstab":12s}: {time.time()-t0:.5f} secs')
assert np.allclose(x,x_alt),x-x_alt
```

    solve       : 0.04418 secs
    spsolve     : 0.00400 secs
    factorized  : 0.00400 secs
    bicgstab    : 0.27044 secs

**Conclusion:**

1. Using the sparsity can be very important.
2. Iterative solvers can be very very slow.

# Symbolically

```python
sm.init_printing(use_unicode=True) # for pretty printing
from IPython.display import display
```

## Solve consumer problem

Consider solving the following problem:

$$
\max_{x_1,x_2} x_1^{\alpha} x_2^{\beta} \text{ s.t. } p_1x_1 + p_2x_2 = I
$$

Define all symbols:

```python
x1 = sm.symbols('x_1') # x1 is a Python variable representing the symbol x_1
x2 = sm.symbols('x_2')
alpha = sm.symbols('alpha')
beta = sm.symbols('beta')
p1 = sm.symbols('p_1')
p2 = sm.symbols('p_2')
I = sm.symbols('I')
```

Define objective and budget constraint:

```python
objective = x1**alpha*x2**beta
display(objective)
```

```python
budget_constraint = sm.Eq(p1*x1+p2*x2,I)
display(budget_constraint)
```

Solve in **four steps**:

1. **Isolate** $x_2$ from the budget constraint
2. **Substitute** in $x_2$
3. **Take the derivative** wrt. $x_1$
4. **Solve the FOC** for $x_1$

**Step 1: Isolate**

```python
x2_from_con = sm.solve(budget_constraint,x2)
display(x2_from_con)
```

**Step 2: Substitute**

```python
objective_subs = objective.subs(x2,x2_from_con[0])
display(objective_subs)
```

**Step 3: Take the derivative**

```python
foc = sm.diff(objective_subs,x1)
display(foc)
```

**Step 4: Solve the FOC**

```python
sol = sm.solve(sm.Eq(foc,0),x1)
display(sol)
```

> An alternative is `sm.solveset()`, which will be the default in the future, but it is still a bit immature in my view.

**Task:** Solve the consumer problem with quasi-linear preferences,

$$ \max_{x_1,x_2} \sqrt{x_1} + \gamma x_2 \text{ s.t. } p_1x_1 + p_2x_2 = I $$

```python
# write your code here
```

```python
gamma = sm.symbols('gamma')
objective_alt = sm.sqrt(x1) + gamma*x2
objective_alt_subs = objective_alt.subs(x2,x2_from_con[0])
foc_alt = sm.diff(objective_alt_subs,x1)
sol_alt = sm.solve(foc_alt,x1)
sol_alt[0]
```

## Use solution

**LaTex:** Print in LaTex format:

```python
print(sm.latex(sol[0]))
```

    \frac{I \alpha}{p_{1} \left(\alpha + \beta\right)}

**Turn into Python function:**

```python
f = lambda x: x**2
```

```python
_sol_func = sm.lambdify((p1,I,alpha,beta),sol[0])
def sol_func(p1,I=10,alpha=1,beta=1):
    return _sol_func(p1,I,alpha,beta)

# test
p1_vec = np.array([1.2,3,5,9])
demand_p1 = sol_func(p1_vec)
print(demand_p1)
```

    [4.16666667 1.66666667 1.         0.55555556]

**Is demand always positive?**

Give the computer the **information** we have. I.e. that $p_1$, $p_2$, $\alpha$, $\beta$, $I$ are all strictly positive:

```python
for var in [p1,p2,alpha,beta,I]:
    sm.assume.global_assumptions.add(sm.Q.positive(var))
sm.assume.global_assumptions
```

**Ask** the computer a **question**:

```python
answer = sm.ask(sm.Q.positive(sol[0]))
print(answer)
```

    True

We need the assumption that $p_1 > 0$:

```python
sm.assume.global_assumptions.remove(sm.Q.positive(p1))
answer = sm.ask(sm.Q.positive(sol[0]))
print(answer)
```

    None

To clear all assumptions we can use:

```python
sm.assume.global_assumptions.clear()
```

## Solving matrix equations (+)

$$ Ax = b $$

**Remember:**

```python
print('A:\n',A)
print('b:',b)
```

    A:
     [[ 3.  2.  0.]
     [ 1. -1.  0.]
     [ 0.  5.  1.]]
    b: [ 2.  4. -1.]

**Construct symbolic matrix:**

```python
A_sm = numecon_linalg.construct_sympy_matrix(['11','12','21','22','32','33']) # somewhat complicated function
display(A_sm)
```

**Find the inverse symbolically:**

```python
A_sm_inv = A_sm.inv()
display(A_sm_inv)
```

**Fill in the numeric values:**

```python
A_inv_num = numecon_linalg.fill_sympy_matrix(A_sm_inv,A) # somewhat complicated function
x = A_inv_num@b
print('solution:',x)
```

    solution: [ 2. -2.  9.]

**Note:** The inverse multiplied by the determinant looks nicer...

```python
A_sm_det = A_sm.det()
display(A_sm_det)
```

```python
A_sm_inv_raw = sm.simplify(A_sm_inv*A_sm_det)
display(A_sm_inv_raw)
```

## More features (mixed goodies)

```python
x = sm.symbols('x')
```

**Derivatives:** Higher order derivatives are also availible

```python
sm.Derivative('x**4',x,x)
```

```python
sm.diff('x**4',x,x)
```

Alternatively,

```python
expr = sm.Derivative('x**4',x,x)
expr.doit()
```

**Integrals:**

```python
sm.Integral(sm.exp(-x), (x, 0, sm.oo))
```

```python
sm.integrate(sm.exp(-x), (x, 0, sm.oo))
```

**Limits:**

```python
c = sm.symbols('c')
rho = sm.symbols('rho')
sm.Limit((c**(1-rho)-1)/(1-rho),rho,1)
```

```python
sm.limit((c**(1-rho)-1)/(1-rho),rho,1)
```

**Integers:**

```python
X = sm.Integer(7)/sm.Integer(3)
Y = sm.Integer(3)/sm.Integer(8)
Z = 3
sm.simplify((X*Y)**Z)
```

**Simplify:**

```python
expr = sm.sin(x)**2 + sm.cos(x)**2
display(expr)
```

```python
sm.simplify(expr)
```

**Solve multiple equations at once:**

```python
x = sm.symbols('x')
y = sm.symbols('y')
Eq1 = sm.Eq(x**2+y-2,0)
Eq2 = sm.Eq(y**2-4,0)
sol = sm.solve([Eq1,Eq2], [x,y])

# print all solutions
for xy in sol:
    print(f'(x,y) = ({xy[0]},{xy[1]})')
```

    (x,y) = (-2,-2)
    (x,y) = (0,2)
    (x,y) = (0,2)
    (x,y) = (2,-2)

## Stop pretty printing

```python
sm.init_printing(pretty_print=False)
```

# Non-linear equations - one dimensional

## Introduction

We consider **solving non-linear equations** on the form,

$$
f(x) = 0, x \in \mathbb{R}
$$

This is also called **root-finding**.

A specific **example** is:

$$
f(x) = 10x^3 - x^2 -1
$$

## Derivative based methods

**Newton methods:** Assume you know the function value and derivatives at $x_0$.

A **first order** approximate value of the function at $x_1$ then is:

$$
f(x_1) \approx f(x_0) + f^{\prime}(x_0)(x_1-x_0)
$$

implying

$$
f(x_1) = 0 \Leftrightarrow x_1 = x_0 - \frac{f(x_0)}{f^{\prime}(x_0)}
$$

This is called **Newtons method**.

An alternative is **Halleys method**, which uses

$$
x_1 = x_0 - \frac{f(x_0)}{f^{\prime}(x_0)} \Big[ 1-\frac{f(x_0)}{f^{\prime}(x_0)}\frac{f^{\prime\prime}(x_0)}{2f^{\prime}(x_0)} \Big]^{-1}
$$

making use of information from the **second derivative**.

**Algorithm:** `find_root()`

1. Choose tolerance $\epsilon > 0$, guess on $x_0$ and set $n = 0$.
2. Calculate $f(x_n)$, $f^{\prime}(x_n)$, and perhaps $f^{\prime\prime}(x_n)$.
3. If $|f(x_n)| < \epsilon$ stop.
4. Calculate $x_{n+1}$ using Newtons or Halleys formula (see above).
5. Set $n = n + 1$ and return to step 2.

```python
def find_root(x0,f,fp,fpp=None,method='newton',max_iter=500,tol=1e-8,full_info=False):
    """ find root

    Args:

        x0 (float): initial value
        f (callable): function
        fp (callable): derivative
        fp (callable): second derivative
        method (str): newton or halley
        max_iter (int): maximum number of iterations
        tol (float): tolerance
        full_info (bool): controls information returned

    Returns:

        x (float/ndarray): root (if full_info, all x tried)
        i (int): number of iterations used
        fx (ndarray): function values used (if full_info)
        fpx (ndarray): derivative values used (if full_info)
        fppx (ndarray): second derivative values used (if full_info)

    """

    # initialize
    x = np.zeros(max_iter)
    fx = np.zeros(max_iter)
    fpx = np.zeros(max_iter)
    fppx = np.zeros(max_iter)
    x[0] = x0
    i = 0

    # iterate
    while True:

        # step 2: evaluate function and derivatives
        fx[i] = f(x[i])
        fpx[i] = fp(x[i])
        if method == 'halley':
            fppx[i] = fpp(x[i])

        # step 3: check convergence
        if abs(fx[i]) < tol or i >= max_iter:
            break

        # step 4: update x
        if method == 'newton':
            x[i+1] = x[i] - fx[i]/fpx[i]
        elif method == 'halley':
            a = fx[i]/fpx[i]
            b = a*fppx[i]/(2*fpx[i])
            x[i+1] = x[i] - a/(1-b)

        # step 5: increment counter
        i += 1

    # return
    if full_info:
        return x,i,fx,fpx,fppx
    else:
        return x[i],i
```

**Note:** The cell below contains a function for plotting the convergence.

```python
def plot_find_root(x0,f,fp,fpp=None,method='newton',xmin=-8,xmax=8,xn=100):

    # a. find root and return all information
    x,max_iter,fx,fpx,fppx = find_root(x0,f,fp,fpp=fpp,method=method,full_info=True)

    # b. compute function on grid
    xvec = np.linspace(xmin,xmax,xn)
    fxvec = f(xvec)

    # c. figure
    def _figure(i):

        # i. approximation
        if method == 'newton':
            fapprox = fx[i] + fpx[i]*(xvec-x[i])
        elif method == 'halley':
            fapprox = fx[i] + fpx[i]*(xvec-x[i]) + fppx[i]*(xvec-x[i])**2

        # ii. figure
        fig = plt.figure()
        ax = fig.add_subplot(1,1,1)

        ax.plot(xvec,fxvec,label='function') # on grid
        ax.plot(x[i],fx[i],'o',color='black',label='current') # now

        if i < max_iter:
            ax.plot(xvec,fapprox,label='approximation') # approximation
            ax.axvline(x[i+1],ls='--',lw=1,color='black') # cross zero
            ax.plot(x[i+1],fx[i+1],'o',color='black',mfc='none',label='next')# next

        ax.legend(loc='lower right',facecolor='white',frameon=True)
        ax.set_ylim([fxvec[0],fxvec[-1]])

    widgets.interact(_figure,
        i=widgets.IntSlider(description="iterations", min=0, max=max_iter, step=1, value=0)
    );
```

## Example

```python
f = lambda x: 10*x**3 - x**2 -1
fp = lambda x: 30*x**2 - 2*x
fpp = lambda x: 60*x - 2
```

```python
x,i = find_root(-5,f,fp,method='newton')
print(i,x,f(x))
```

    17 0.5000000000000073 4.773959005888173e-14

```python
x,i = find_root(-5,f,fp,fpp,method='halley')
print(i,x,f(x))
```

    12 0.4999999999675154 -2.1114998638438465e-10

```python
plot_find_root(-5,f,fp,method='newton')
```

    interactive(children=(IntSlider(value=0, description='iterations', max=17), Output()), _dom_classes=('widget-i…

```python
plot_find_root(-5,f,fp,fpp,method='halley')
```

    interactive(children=(IntSlider(value=0, description='iterations', max=12), Output()), _dom_classes=('widget-i…

## Numerical derivative

Sometimes, you might not have the **analytical derivative**. Then, you can instead use the **numerical derivative**.

```python
# a. function
f = lambda x: 10*x**3 - x**2 -1

# b. numerical derivative (forward)
stepsize = 1e-8
fp_approx = lambda x: (f(x+stepsize)-f(x))/stepsize

# b. find root
x0 = -5
x,i = find_root(x0,f,fp_approx,method='newton')
print(i,x,f(x))
```

    17 0.5000000000000091 5.928590951498336e-14

**Question:** What happens if you increase the stepsize?

## Another example

```python
g = lambda x: np.sin(x)
gp = lambda x: np.cos(x)
gpp = lambda x: -np.sin(x)

x0 = -4.0
plot_find_root(x0,g,gp,gpp,method='newton')
```

    interactive(children=(IntSlider(value=0, description='iterations', max=4), Output()), _dom_classes=('widget-in…

**Question:** Is the initial value important?

**Sympy** can actually tell us that there are many solutions:

```python
sm.init_printing(use_unicode=True) # for nice printing
x = sm.symbols('x')
sm.solveset(sm.sin(x),)
```

## Derivative free methods: Bisection

**Algorithm:** `bisection()`

1. Set $a_0 = a$ and $b_0 = b$.
2. Compute $f(m_0)$ where $m_0 = (a_0 + b_0)/2$ is the midpoint.
3. Determine the next sub-interval $[a_1,b_1]$:
  * If $f(a_0)f(m_0) < 0$ (different signs) then $a_1 = a_0$ and $b_1 = m_0$ (i.e. focus on the range $[a_0,m_0]$).
  * If $f(m_0)f(b_0) < 0$ (different signs) then $a_1 = m_0$ and $b_1 = b_0$ (i.e. focus on the range $[m_0,b_0]$).
4. Repeat step 2 and step 3 until $f(m_n) < \epsilon$.

```python
def bisection(f,a,b,max_iter=500,tol=1e-6,full_info=False):
    """ bisection

    Solve equation f(x) = 0 for a <= x <= b.

    Args:

        f (function): function
        a (float): left bound
        b (float): right bound
        tol (float): tolerance on solution

    Returns:

        m (float): root

    """

    # test inputs
    if f(a)*f(b) >= 0:
        print("bisection method fails.")
        return None

    # step 1: initialize
    _a = a
    _b = b
    a = np.zeros(max_iter)
    b = np.zeros(max_iter)
    m = np.zeros(max_iter)
    fm = np.zeros(max_iter)
    a[0] = _a
    b[0] = _b

    # step 2-4: main
    i = 0
    while i < max_iter:

        # step 2: midpoint and associated value
        m[i] = (a[i]+b[i])/2
        fm[i] = f(m[i])

        # step 3: determine sub-interval
        if abs(fm[i]) < tol:
            break
        elif f(a[i])*fm[i] < 0:
            a[i+1] = a[i]
            b[i+1] = m[i]
        elif f(b[i])*fm[i] < 0:
            a[i+1] = m[i]
            b[i+1] = b[i]
        else:
            print("bisection method fails.")
            return None

        i += 1

    if full_info:
        return m,i,a,b,fm
    else:
        return m[i],i
```

**Same result** as before, but **trade-off** between more iterations and no evaluation of derivatives.

```python
m,i = bisection(f,-8,7)
print(i,m,f(m))
```

    23 0.4999999403953552 -3.8743014130204756e-07

**Note:** The cell below contains a function for plotting the convergence.

```python
def plot_bisection(f,a,b,xmin=-8,xmax=8,xn=100):

    # a. find root and return all information
    res = bisection(f,a,b,full_info=True)
    if res == None:
        return
    else:
        m,max_iter,a,b,fm = res

    # b. compute function on grid
    xvec = np.linspace(xmin,xmax,xn)
    fxvec = f(xvec)

    # c. figure
    def _figure(i):

        # ii. figure
        fig = plt.figure()
        ax = fig.add_subplot(1,1,1)

        ax.plot(xvec,fxvec) # on grid
        ax.plot(m[i],fm[i],'o',color='black',label='current') # mid
        ax.plot([a[i],b[i]],[fm[i],fm[i]],'--',color='black',label='range') # range
        ax.axvline(a[i],ls='--',color='black')
        ax.axvline(b[i],ls='--',color='black')

        ax.legend(loc='lower right',facecolor='white',frameon=True)
        ax.set_ylim([fxvec[0],fxvec[-1]])

    widgets.interact(_figure,
        i=widgets.IntSlider(description="iterations", min=0, max=max_iter, step=1, value=0)
    );

plot_bisection(f,-8,3)
```

    interactive(children=(IntSlider(value=0, description='iterations', max=24), Output()), _dom_classes=('widget-i…

**Note:** Bisection is not good at the final convergence steps. Generally true for methods not using derivatives.

## Scipy

Scipy, naturally, has better implementations of the above algorithms.

**Newton:**

```python
result = optimize.root_scalar(f,x0=-4,fprime=fp,method='newton')
print(result)
```

          converged: True
               flag: 'converged'
     function_calls: 30
         iterations: 15
               root: 0.5

```python
x,result = optimize.newton(f,x0=-4,fprime=fp,full_output=True)
print(result)
```

          converged: True
               flag: 'converged'
     function_calls: 30
         iterations: 15
               root: 0.5

**Halley:**

```python
result = optimize.root_scalar(f,x0=-4,fprime=fp,fprime2=fpp,method='halley')
print(result)
```

          converged: True
               flag: 'converged'
     function_calls: 27
         iterations: 9
               root: 0.5

**Bisect:**

```python
result = optimize.root_scalar(f,bracket=[-8,1],method='bisect')
print(result)
```

          converged: True
               flag: 'converged'
     function_calls: 45
         iterations: 43
               root: 0.5000000000001137

The **best choice** is the more advanced **Brent-method**:

```python
result = optimize.root_scalar(f,bracket=[-8,1],method='brentq')
print(result)
```

          converged: True
               flag: 'converged'
     function_calls: 15
         iterations: 14
               root: 0.5000000000000082

# Solving non-linear equations (multi-dimensional)

## Introduction

We consider **solving non-linear equations** on the form,

$$
f(\boldsymbol{x}) = f(x_1,x_2,\dots,x_k) = 0, \boldsymbol{x} \in \mathbb{R}^k
$$

A specific **example** is:

$$
h(\boldsymbol{x})=h(x_{1,}x_{2})=\begin{bmatrix}h_{1}(x_{1},x_{2})\\
h_{2}(x_{1},x_{2})
\end{bmatrix}=\begin{bmatrix}x_{1}+0.5(x_{1}-x_{2})^{3}-1\\
x_{2}+0.5(x_{1}-x_{2})^{3}
\end{bmatrix}\in\mathbb{R}^{2}
$$

where the **Jacobian** is

$$
\nabla h(\boldsymbol{x})=\begin{bmatrix}\frac{\partial h_{1}}{\partial x_{1}} & \frac{\partial h_{1}}{\partial x_{2}}\\
\frac{\partial h_{2}}{\partial x_{1}} & \frac{\partial h_{2}}{\partial x_{2}}
\end{bmatrix}=\begin{bmatrix}1+1.5(x_{1}-x_{2})^{2} & -1.5(x_{1}-x_{2})^{2}\\
-1.5(x_{2}-x_{1})^{2} & 1+1.5(x_{2}-x_{1})^{2}
\end{bmatrix}
$$

```python
def h(x):
    y = np.zeros(2)
    y[0] = x[0]+0.5*(x[0]-x[1])**3-1.0
    y[1] = x[1]+0.5*(x[1]-x[0])**3
    return y

def hp(x):
    y = np.zeros((2,2))
    y[0,0] = 1+1.5*(x[0]-x[1])**2
    y[0,1] = -1.5*(x[0]-x[1])**2
    y[1,0] = -1.5*(x[1]-x[0])**2
    y[1,1] = 1+1.5*(x[1]-x[0])**2
    return y
```

## Newton's method

Same as Newton's method in one dimension, but with the following **update step**:

$$
\boldsymbol{x}_{n+1} = \boldsymbol{x_n} - [ \nabla h(\boldsymbol{x_n})]^{-1} f(\boldsymbol{x_n})
$$

```python
def find_root_multidim(x0,f,fp,max_iter=500,tol=1e-8):
    """ find root

    Args:

        x0 (float): initial value
        f (callable): function
        fp (callable): derivative
        max_iter (int): maximum number of iterations
        tol (float): tolerance

    Returns:

        x (float): root
        i (int): number of iterations used

    """

    # initialize
    x = x0
    i = 0

    # iterate
    while i < max_iter:

        # step 2: function and derivatives
        fx = f(x)
        fpx = fp(x)

        # step 3: check convergence
        if max(abs(fx)) < tol:
            break

        # step 4: update x
        fpx_inv = linalg.inv(fpx)
        x = x - fpx_inv@fx

        # step 5: increment counter
        i += 1

    return x,i
```

**Test algorithm:**

```python
x0 = np.array([0,0])
x,i = find_root_multidim(x0,h,hp)
print(i,x,h(x))
```

    5 [0.8411639 0.1588361] [ 1.41997525e-10 -1.41997469e-10]

## Scipy

There exist a lot of efficient algorithms for finding roots in multiple dimensions. The default scipy choice is something called *hybr*.

**With the Jacobian:**

```python
result = optimize.root(h,x0,jac=hp)
print(result)
print('\nx =',result.x,', h(x) =',h(result.x))
```

        fjac: array([[ 0.89914291, -0.43765515],
           [ 0.43765515,  0.89914291]])
         fun: array([-1.11022302e-16,  0.00000000e+00])
     message: 'The solution converged.'
        nfev: 10
        njev: 1
         qtf: array([-1.19565972e-11,  4.12770392e-12])
           r: array([ 2.16690469, -1.03701789,  1.10605417])
      status: 1
     success: True
           x: array([0.8411639, 0.1588361])

    x = [0.8411639 0.1588361] , h(x) = [-1.11022302e-16  0.00000000e+00]

**Without the Jacobian:**

```python
result = optimize.root(h,x0)
print(result)
print('\nx =',result.x,', h(x) =',h(result.x))
```

        fjac: array([[-0.89914291,  0.43765515],
           [-0.43765515, -0.89914291]])
         fun: array([-1.11022302e-16,  0.00000000e+00])
     message: 'The solution converged.'
        nfev: 12
         qtf: array([ 1.19565972e-11, -4.12770392e-12])
           r: array([-2.16690469,  1.03701789, -1.10605417])
      status: 1
     success: True
           x: array([0.8411639, 0.1588361])

    x = [0.8411639 0.1588361] , h(x) = [-1.11022302e-16  0.00000000e+00]

# Summary

**This lecture:**

1. Solving matrix equations (directly, decomposition, iterative)
2. Symbollic solutions (substitution, derivative, solution)
3. Root-finding (one dimension, multiple dimensions, Newton's method)

**Your work:** Play around with the code in this notebook before solving the problem set. Especially, try out the various scipy functions used.

**Next lecture:** Numerical optimization.


Euler Problem 58
================

Starting with 1 and spiralling anticlockwise in the following way, a square spiral with side length 7 is formed.

    37 36 35 34 33 32 31
    38 17 16 15 14 13 30
    39 18  5  4  3 12 29
    40 19  6  1  2 11 28
    41 20  7  8  9 10 27
    42 21 22 23 24 25 26
    43 44 45 46 47 48 49

It is interesting to note that the odd squares lie along the bottom right diagonal, but what is more interesting is that 8 out of the 13 numbers lying along both diagonals are prime; that is, a ratio of 8/13 ≈ 62%.

If one complete new layer is wrapped around the spiral above, a square spiral with side length 9 will be formed. If this process is continued, what is the side length of the square spiral for which the ratio of primes along both diagonals first falls below 10%?

```python
from sympy import isprime
corners = [3, 5, 7, 9]
squares = 5
primes = 3
delta = 8
sidelength = 3
while primes*10 >= squares:
    sidelength += 2
    for i in range(4):
        delta += 2
        corners[i] += delta
        primes += isprime(corners[i])
    squares += 4
print(sidelength)
```

    26241

```python

```


# Music machine learning - Introduction

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A [first definition](#definition) on the concept of machine learning
2. An introduction to a simple problem of [linear regression](#regression)
3. An explanation on [model capacity and overfitting](#capacity)
4. An introduction to the [audio datasets](#audio) that we will use

## Defining machine learning

In all natural process, there exists complex relations between sets $\mathcal{X} \mapsto \mathcal{Y}$. This can relate some objects with their names, or a cause to a consequence. In most cases, _we do not know the precise relations_ between these sets, all we have is _observations_ such as pairs $(x,y)$, composed of input data $x \in \mathcal{X}$, which have a corresponding expected output $y \in \mathcal{Y}$. The overarching goal of machine learning is to approximate such _unknown processes_ as a function $\mathcal{F}_{\theta}$, which _transforms_ input data $x$ into output data $y$.

Hence, machine learning aims to understand and model the relationship between some (usually complex and high-dimensional) inputs $\mathbf{x}\in\mathcal{X}\subset\mathbb{R}^{\mathcal{X}}$ and outputs $\mathbf{y}\in\mathcal{Y}\subset\mathbb{R}^{\mathcal{Y}}$, given by a set of data examples $\mathcal{D}=\left\{(x_1,y_1),\cdots,(x_N,y_N)\right\}$. This is achieved by defining a parametric model $f_{\mathbf{\theta}}\in\mathcal{F}$ inside a family of functions $\mathcal{F}$, which depends on parameters $\mathbf{\theta} \in \mathbf{\Theta}$ and that could approximate the underlying relationship. The _learning_ aspect refers to the adjustment of the parameters $\mathbf{\theta}$ in order to obtain the best approximation of the given task
$$
\begin{equation}
f_{\mathbf{\theta}}(\mathbf{x}) = \hat{\mathbf{y}}\approx \mathbf{y}.
\end{equation}
$$

To observe this idea in simple setups, we are going to use the `numpy` library and `matplotlib` for plotting. We also set some fixed random generator, to ensure reproducibility

```python
# Base imports
import numpy as np
import matplotlib.pyplot as plt
from helper_plot import prep_plots, finalize_plots, hdr_plot_style
rng = np.random.RandomState(1)
```

## Simple learning problem

Imagine that a certain process somewhere follows the form of a quadratic relationship

$$
 y = a x^{2} + bx + c
$$

In this case, all the **unknown parameters** are that of a polynomial model, therefore we have $\theta = \{a, b, c\}$. However, this is clearly an ideal (clean) case, whereas in natural observations, there might be some noise in our observations
$$
 y = a x^{2} + bx + c +\epsilon \quad \mbox{with} \quad \epsilon \in [-0.1, 0.1]
$$

An example of such noisy observations for different parameters is given below

```python
noise_level = 0.1
a, b, c = 3, 0, 1
# Generating
x = np.linspace(0, 1, 100)
poly = np.poly1d([a, b, c])
epsilon = np.random.uniform(-noise_level,noise_level,x.shape)
y = poly(x) + epsilon
ax1, = prep_plots(["Simple learning problem"], fig_size=(10,8), fig_num=1)
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
#plt.xlim([-1, 2])
```

Now our main problem is that this function can follow different types of parameters

```python
params = [[3, 0, 1], [-2, 1, 0], [0.1, 1, 1]]
ax = prep_plots(["Problem 1", "Problem 2", "Problem 3"], fig_size=(20,6), fig_num=1)
# Generating
x = np.linspace(0, 1, 100)
for p in range(len(params)):
    poly = np.poly1d(params[p])
    epsilon = np.random.uniform(-noise_level,noise_level,x.shape)
    y = poly(x) + epsilon
    ax[p].scatter(x, y, color='green', s=50, marker='o', edgecolor='k', label="training samples")
finalize_plots(ax, fig_title="Different observations")
```

In real-life settings, this function can also have different levels of noise, as exemplified in the following code.

```python
params = [3, 0, 1]
noise_levels = [0.1, 1.0, 8.0]
ax = prep_plots(["Problem 1", "Problem 2", "Problem 3"], fig_size=(20,6), fig_num=1)
# Generating
x = np.linspace(0, 1, 100)
for p in range(len(noise_levels)):
    poly = np.poly1d(params)
    epsilon = np.random.uniform(-noise_levels[p],noise_levels[p],x.shape)
    y = poly(x) + epsilon
    ax[p].scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots(ax, fig_title="Different amounts of noise")
```

To summarize, we will have some observations of a function, and we would like to optimize a function that gets as close as possible to the real function that generated this data. Here, we plot the real function and also _subsample_ our number of observations (having only a few points to find the corresponding function)

```python
# Generating the data and subsampling
x_all = np.linspace(0, 1, 100); x_plot = np.linspace(0, 1, 100)
rng.shuffle(x_all); x = np.sort(x_all[:10])
poly = np.poly1d([3,0,1])
# Adding some external
epsilon = np.random.uniform(-0.1,0.1,x.shape)
y = poly(x)+ epsilon
# We keep the standard deviation for later
nnstd = np.std(epsilon); lw = 2
ax1, = prep_plots(["Simple learning problem"], fig_size=(10,8), fig_num=1)
ax1.plot(x_plot, poly(x_plot), color='green', linewidth=4, label='true function')
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
```

To get a first grip on what machine learning does, we will rely on the `scikit-learn` library. This contains already coded models and learning procedure, that will allow us to _learn_ the parameters of this unknown function.

Here we already know that we want to use a `PolynomialFeatures` model to perfom `LinearRegression` and that this polynomial should be of degree 2.

```python
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
# Our data to fit
X = x[:, np.newaxis]
# Degree of our polynomial
degree = 2;
# Create our polynomial model for regression
model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
# Fit the parameters of this model
model.fit(X, y)
```

Now that we have trained the model, we can perform _predictions_ from it, meaning that we can infer the output of the function at values that we did not observe originally.

```python
# Inference points (not observed)
X_plot = x_plot[:, np.newaxis]
# Predict the values
y_plot = model.predict(X_plot)
# Compute the error of our model at observed points
Y_model_err = np.sqrt(np.mean(np.square(y-model.predict(X))))
# Plot the result
ax1, = prep_plots(["Trained model"], fig_size=(10,8), fig_num=1)
ax1.plot(x_plot, y_plot, color='red', linewidth=4, label='trained model')
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
print(f'Model error : {Y_model_err}')
```

## Understanding model capacity and selection

In real-life problem, we are aiming to find the parameters of a model, but we do not really know what is the _real_ function underlying this process. So what we can decide to select _any_ function of _any_ **capacity** (complexity of the function). One of the problem with that, is that if we have a too simple function, it will _underfit_ (it is not complex enough for our observations). On the opposite end, if we have a function which is too complex, it might be able to _fit through all training points exactly_ ... even though there is noise in our observations ! This is examplified in the following

We can observe this idea and play with it directly by trying to find a function approximating our previous observations with a polynomial function chosen to have a degree inside \([1,2,8]\).

```python
X_plot = x_plot[:, np.newaxis]
ax1, ax2 = prep_plots(["training error", "model generalization"], fig_size=(15,4), fig_num=1)
ax1.plot([1,8], [nnstd, nnstd], label="noise level")
ax2.scatter(x, y, color='orange', edgecolor='w', s=50, marker='o', label="training samples")
colors = ['blue', 'green', 'red']; labels = ['underfitting', 'good fit', 'overfitting']
axes_sub = prep_plots(labels, fig_size=(15,4), fig_num=2)
# Optimize different degree polynomials
for count, degree in enumerate([1,2,8]):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X, y)
    Y_model_err = np.sqrt(np.mean(np.square(y-model.predict(X))))
    y_plot = model.predict(X_plot)
    ax1.scatter(degree, Y_model_err, s=50, marker="o", edgecolor='w', label=labels[count])
    ax2.plot(x_plot, y_plot, color=colors[count], linewidth=2,label=labels[count])
    ax2.set_ylim((0,4))
    axes_sub[count].scatter(x, y, color='orange', edgecolor='w', s=50, marker='o', label="Training samples")
    axes_sub[count].plot(x_plot, y_plot, color=colors[count], linewidth=2,label=labels[count])
    axes_sub[count].set_ylim((0,4))
finalize_plots(axes_sub, fig_title="Linear regression")
finalize_plots([ax1,ax2], fig_title="Linear regression")
```

Depending on the _capacity_ of the model, what we can observe is that

- `capacity too low   -> underfitting   : prediction variance >  noise variance`
- `adequate capacity  -> good fit       : prediction variance == noise variance`
- `capacity too high  -> overfitting    : prediction variance <  noise variance`

A similar example can be given for a classification problem in two dimensions as follows

## Audio applications

In order to test our algorithms on audio and music data, we will work with several datasets that should be downloaded on your local computer first from this [link ](https://nuage.ircam.fr/index.php/s/FTsaaAMFV1jEwsk)

  |**Type**|*Origin*|
  |-------:|:---------|
  |**Classification**|[*MuscleFish*](http://knight.cis.temple.edu/~vasilis/Courses/CIS750/Papers/muscle_fish.pdf) dataset|
  |**Music-speech**|[*MIREX Recognition*](http://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection) set|
  |**Source separation**|[*SMC Mirum*](http://smc.inesctec.pt/research/data-2/) dataset|
  |**Speech recognition**|[*CMU Arctic*](http://festvox.org/cmu_arctic/) dataset|

**Unzip the file and place the `data` folder along with the other code folders**
For the first parts of the tutorial, we will mostly rely solely on the classification dataset. In order to facilitate the interactions, we provide the function `import_dataset` that will allow to import all audio datasets along the tutorials.

```Python
def importDataset(class_path, type):
    """
    Helper function to import datasets
    % class_path  : Path to the dataset (string)
    % type       : Type of dataset (string: 'classify', 'plain', 'metadata')
    """
    # Returns the data_struct structure with
    data_struct["filenames"]  # Cell containing the list of audio files
    data_struct["classes"]    # Vector of indexes assigning each file to a class
    data_struct["class_names"] % Cell of class names
    return data_struct
```

***

**_Exercise_**

  1. Launch the import procedure  and check the corresponding structure
  2. Code a count function that prints the name and number of examples for each classes

***

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from helper_data import import_dataset
# 0.1 - Import the classification dataset
class_path = 'data/classification';
data_struct = import_dataset(class_path, 'classification');
```

```python
#%% Q-0.1.2 - Count function to print the number of examples

######################
# YOUR CODE GOES HERE
######################

```

### Preprocessing

We will rely on a set of spectral transforms that allow to obtain a more descriptive view over the audio information. As most of these are out of the scope of the machine learning course, we redirect you to a [signal processing course](https://ccrma.stanford.edu/~jos/sasp/) proposed by [Julius O. Smith](https://ccrma.stanford.edu/~jos/).

The following functions to compute various types of transforms are given as part of the basic package, in the `helper_data.py` file

  |**File**|*Transform*|
  |-------:|:----------|
  |`stft.m`       |[Short-term Fourier transform](https://en.wikipedia.org/wiki/Short-time_Fourier_transform)|
  |`fft2barkmx.m` |[Bark scale](https://en.wikipedia.org/wiki/Bark_scale) transform|
  |`fft2melmx.m`  |[Mel scale](https://en.wikipedia.org/wiki/Mel_scale) transform|
  |`fft2chromamx` |[Chromas vector](https://en.wikipedia.org/wiki/Harmonic_pitch_class_profiles)|
  |`spec2cep.m`   |[Cepstrum](https://en.wikipedia.org/wiki/Cepstrum) transform|
  |`cqt.m`        |[Constant-Q](https://en.wikipedia.org/wiki/Constant_Q_transform) transform|

In order to perform the various computations, we provide the following function, which performs the different transforms on a complete dataset.

``` Python
def computeTransforms(data_struct)
    """ data_struct   : Dataset structure with filenames """
    return data_struct

# Returns the data_struct structure with
data_struct["spectrum_power"]     # Power spectrum (STFT)
data_struct["spectrum_mel"]       # Spectrum in Mel scale
data_struct["spectrum_chroma"]    # Chroma vectors
data_struct["spectrum_CQT"]       # Constant-Q transform
```

***

**Exercise**

  1. Launch the transform computation procedure and check the corresponding structure
  2. For each class, select a random element and plot its various transforms on a single plot. You should obtain plots similar to those shown afterwards.
  3. For each transform, try to spot major pros and cons of their representation.

***

```python
# 0.2 - Pre-process the audio to obtain spectral transforms
# (may take around a minute)
from helper_data import compute_transforms
data_struct = compute_transforms(data_struct);
```

```python
#%% Q-0.2.2 - Plot the various transforms

# Just a little helper to make your figures pretty
from helper_plot import hdr_plot_style
hdr_plot_style()

######################
# YOUR CODE GOES HERE
######################

```

### Features

As you might have noted from the previous exercice, most spectral transforms have a very high dimensionality, and might not be suited to exhibit the relevant structure of different classes. To that end, we provide a set of functions for computing several spectral features in the `helper_data` folder, we redirect interested readers to this [exhaustive article](http://recherche.ircam.fr/anasyn/peeters/ARTICLES/Peeters_2003_cuidadoaudiofeatures.pdf) on spectral features computation.

  |**File**|*Transform*|
  |-------:|:----------|
  |`spectral_centroid`|Spectral centroid|
  |`spectral_bandwidth`|Spectral bandwidth|
  |`spectral_contrast`|Spectral contrast|
  |`spectral_flatness`|Spectral flatness|
  |`spectral_rolloff`|Spectral rolloff|

Once again, we provide a function to perform the computation of different features on a complete set. Note that for each feature, we compute the temporal evolution in a vector along with the mean and standard deviation of each feature. We only detail the resulting data structure for a single feature (`SpectralCentroid`).

``` Python
def data_struct = computeFeatures(data_struct)
     """ data_struct   : Dataset structure with filenames """
    return data_struct

% Returns the data_struct structure with
data_struct["spectral_centroid"]      # Temporal value of a feature
data_struct["spectral_centroid_mean"] # Mean value of that feature
data_struct["spectral_centroid_std"]  # Standard deviation
```

***

**Exercise**

  1. Launch the feature computation procedure and check the corresponding structure
  2. This time for each class, superimpose the plots of various features on a single plot, along with a boxplot of mean and standard deviations. You should obtain plots similar to those shown afterwards.
  3. What conclusions can you make on the discriminative power of each feature ?
  4. Perform scatter plots of the mean features for all the dataset, while coloring different classes.
  5. What conclusions can you make on the discriminative power of mean features ?

***

```python
#%%
# 0.3 - Compute a set of temporal and spectral features
# (may take around 1-2 minutes)
from helper_data import compute_features
data_struct = compute_features(data_struct);
```

```python
#%% Q-0.3.2 - Plot the various features

# Just a little helper to make your figures pretty
from helper_plot import hdr_plot_style
hdr_plot_style()
# Use these styles for boxplot
boxprops=dict(linewidth=3, color='white')
whiskerprops=dict(linewidth=3, color='white')
medianprops=dict(linewidth=2.5, color='firebrick')
flierprops = dict(markeredgecolor='white', markerfacecolor='firebrick')

######################
# YOUR CODE GOES HERE
######################

```

```python
#%% Q-0.3.4 - Observe the distribution of classes for different features

# This allows to use 3D rendering in matplotlib
from mpl_toolkits.mplot3d import Axes3D
plt.figure(figsize=(12,8))
# Create a vector of random colors for each class
colorVect = np.zeros((3, len(data_struct["class_names"])));
for c in range(len(data_struct["class_names"])):
    colorVect[:,c] = np.random.rand(3);

######################
# YOUR CODE GOES HERE
######################

```

That's it for this tutorial, now remember that we can use any form of description (features) as a basis for learning algorithms. We will see in the next tutorial what we an do with these features.


# Simulating a Predator and Prey Relationship

Without a predator, rabbits will reproduce until they reach the carrying capacity of the land. When coyotes show up, they will eat the rabbits and reproduce until they can't find enough rabbits. We will explore the fluctuations in the two populations over time.

# Using Lotka-Volterra Model

## Part 1: Rabbits without predators

According to [Mother Earth News](https://www.motherearthnews.com/homesteading-and-livestock/rabbits-on-pasture-intensive-grazing-with-bunnies-zbcz1504), a rabbit eats six square feet of pasture per day. Let's assume that our rabbits live in a five acre clearing in a forest: 217,800 square feet/6 square feet = 36,300 rabbit-days worth of food. For simplicity, let's assume the grass grows back in two months. Thus, the carrying capacity of five acres is 36,300/60 = 605 rabbits.

Female rabbits reproduce about six to seven times per year. They have six to ten children in a litter.  According to [Wikipedia](https://en.wikipedia.org/wiki/Rabbit), a wild rabbit reaches sexual maturity when it is about six months old and typically lives one to two years. For simplicity, let's assume that in the presence of unlimited food, a rabbit lives forever, is immediately sexually mature, and has 1.5 children every month.

For our purposes, then, let $x_t$ be the number of rabbits in our five acre clearing on month $t$.
$$
\begin{equation*}
  R_t = R_{t-1} + 1.5\frac{605 - R_{t-1}}{605} R_{t-1}
\end{equation*}
$$

The formula could be put into general form
$$
\begin{equation*}
  R_t = R_{t-1} + growth_{R} \times \big( \frac{capacity_{R} - R_{t-1}}{capacity_{R}} \big) R_{t-1}
\end{equation*}
$$

By doing this, we allow users to interact with growth rate and the capacity value visualize different interaction

```python
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
from IPython.display import display, clear_output
import ipywidgets as widgets
import matplotlib.pyplot as plt
import numpy as np
```

```python
%matplotlib inline
style = {'description_width': 'initial'}
capacity_R = widgets.FloatText(description="Capacity", value=605)
growth_rate_R = widgets.FloatText(description="Growth rate", value=1.5)
initial_R = widgets.FloatText(description="Initial population",style=style, value=1)
button_R = widgets.Button(description="Plot Graph")
display(initial_R, capacity_R, growth_rate_R, button_R)

def plot_graph_r(b):
    print("helo")
    clear_output()
    display(initial_R, capacity_R, growth_rate_R, button_R)
    fig = plt.figure()
    ax = fig.add_subplot(111)
    t = np.arange(0, 20, 1)
    s = np.zeros(t.shape)
    R = initial_R.value
    for i in range(t.shape[0]):
        s[i] = R
        R = R + growth_rate_R.value * (capacity_R.value - R)/(capacity_R.value) * R

    ax.plot(t, s)
    ax.set(xlabel='time (months)', ylabel='rabbits',
       title='Rabbits Without Predators')
    ax.grid()

button_R.on_click(plot_graph_r)
```

## Part 2: Coyotes without preys
According to [Huntwise](https://www.besthuntingtimes.com/blog/2020/2/3/why-you-should-coyote-hunt-how-to-get-started), coyotes need to consume about 2-3 pounds of food per day. Their diet is 90 percent mammalian. The perfect adult cottontail rabbits weigh 2.6 pounds on average. Thus, we assume the coyote eats one rabbit per day.

For coyotes, the breeding season is in February and March. According to [Wikipedia](https://en.wikipedia.org/wiki/Coyote#Social_and_reproductive_behaviors), females have a gestation period of 63 days, with an average litter size of 6, though the number fluctuates depending on coyote population density and the abundance of food. By fall, the pups are old enough to hunt for themselves.

In the absence of rabbits, the number of coyotes will drop, as their food supply is scarce.
The formula could be put into general form:

$$
\begin{align*}
  C_t & \sim (1 - death_{C}) \times C_{t-1}\\
  &= C_{t-1} - death_{C} \times C_{t-1}
\end{align*}
$$

```python
%matplotlib inline
style = {'description_width': 'initial'}
initial_C=widgets.FloatText(description="Initial Population",style=style,value=200.0)
declining_rate_C=widgets.FloatText(description="Death rate",value=0.5)
button_C=widgets.Button(description="Plot Graph")
display(initial_C, declining_rate_C, button_C)

def plot_graph_c(b):
    clear_output()
    display(initial_C, declining_rate_C, button_C)
    fig = plt.figure()
    ax = fig.add_subplot(111)
    t1 = np.arange(0, 20, 1)
    s1 = np.zeros(t1.shape)
    C = initial_C.value
    for i in range(t1.shape[0]):
        s1[i] = C
        C = (1 - declining_rate_C.value)*C

    ax.plot(t1, s1)
    ax.set(xlabel='time (months)', ylabel='rabbits',
       title='Rabbits Without Predators')
    ax.grid()

button_C.on_click(plot_graph_c)

```

## Part 3: Interaction Between Coyotes and Rabbit
With the simple interaction from the first two parts, now we can combine both interaction and come out with simple interaction.
$$
\begin{align*}
  R_t &= R_{t-1} + growth_{R} \times \big( \frac{capacity_{R} - R_{t-1}}{capacity_{R}} \big) R_{t-1} - death_{R}(C_{t-1})\times R_{t-1}\\\\
  C_t &= C_{t-1} - death_{C} \times C_{t-1} + growth_{C}(R_{t-1}) \times C_{t-1}
\end{align*}
$$

In equations above, death rate of rabbit is a function parameterized by the amount of coyote. Similarly, the growth rate of coyotes is a function parameterized by the amount of the rabbit.

The death rate of the rabbit should be $0$ if there are no coyotes, while it should approach $1$ if there are many coyotes. One of the formula fulfilling this characteristics is hyperbolic function.

$$
\begin{equation}
death_R(C) = 1 - \frac{1}{xC + 1}
\end{equation}
$$

where $x$ determines how quickly $death_R$ increases as the number of coyotes ($C$) increases. Similarly, the growth rate of the coyotes should be $0$ if there are no rabbits, while it should approach infinity if there are many rabbits. One of the formula fulfilling this characteristics is a linear function.

$$
\begin{equation}
growth_C(R) = yC
\end{equation}
$$

where $y$ determines how quickly $growth_C$ increases as number of rabbit ($R$) increases.

Putting all together, the final equtions are

$$
\begin{align*}
  R_t &= R_{t-1} + growth_{R} \times \big( \frac{capacity_{R} - R_{t-1}}{capacity_{R}} \big) R_{t-1} - \big( 1 - \frac{1}{xC_{t-1} + 1} \big)\times R_{t-1}\\\\
  C_t &= C_{t-1} - death_{C} \times C_{t-1} + yR_{t-1}C_{t-1}
\end{align*}
$$

```python
%matplotlib inline
initial_rabbit = widgets.FloatText(description="Initial Rabbit",style=style, value=1)
initial_coyote = widgets.FloatText(description="Initial Coyote",style=style, value=1)
capacity = widgets.FloatText(description="Capacity rabbits", style=style,value=5)
growth_rate = widgets.FloatText(description="Growth rate rabbits", style=style,value=1)
death_rate = widgets.FloatText(description="Death rate coyotes", style=style,value=1)
x = widgets.FloatText(description="Death rate ratio due to coyote",style=style, value=1)
y = widgets.FloatText(description="Growth rate ratio due to rabbit",style=style, value=1)
button = widgets.Button(description="Plot Graph")
display(initial_rabbit, initial_coyote, capacity, growth_rate, death_rate, x, y, button)
def plot_graph(b):
    clear_output()
    display(initial_rabbit, initial_coyote, capacity, growth_rate, death_rate, x, y, button)
    fig = plt.figure()
    ax = fig.add_subplot(111)
    t = np.arange(0, 20, 0.5)
    s = np.zeros(t.shape)
    p = np.zeros(t.shape)
    R = initial_rabbit.value
    C = initial_coyote.value
    for i in range(t.shape[0]):
        s[i] = R
        p[i] = C
        R = R + growth_rate.value * (capacity.value - R)/(capacity.value) * R - (1 - 1/(x.value*C + 1))*R
        C = C - death_rate.value * C  + y.value*s[i]*C

    ax.plot(t, s, label="rabit")
    ax.plot(t, p, label="coyote")
    ax.set(xlabel='time (months)', ylabel='rabbits',
       title='Coyotes Rabbit Predator Prey Relationship')
    ax.grid()
    ax.legend()

button.on_click(plot_graph)
```

x: number of preys(rabbits)\
y: number of predators(coyotes)\
a: natural growing rate of rabbits, when there is no coyotes\
b: natural dying rate of rabbits, which is killed by coyotes per unit of time\
c: natural dying rate of coyotes, when ther is not rabbits\
d: natural growing rate of coyotes with which consumed prey is converted to predator

```python
from numpy import *
from matplotlib import pyplot as plt
from scipy import integrate

# seterr(divide='ignore', invalid='ignore')
a = growth_rate.value
b = x.value
c = death_rate.value
d = y.value
# Define the system of ODEs
# P[0] is prey, P[1] is predator
def dP_dt(P,t=0):
    return array([a*P[0]-b*P[0]*P[1], d*b*P[0]*P[1]-c*P[1]])
# Define the Equilibrium points
eq1 = array([0. , 0.])
eq2 = array([c/(d*b),a/b])
```

```python
values = linspace(0.1, 3, 10)
# Colors for each trajectory
vcolors = plt.cm.autumn_r(linspace(0.1, 1., len(values)))
f = plt.figure(figsize=(10,6))
t = np.linspace(0, 150, 1000)
for v, col in zip(values, vcolors):
    # Starting point
    P0 = v*eq2
    P = integrate.odeint(dP_dt, P0, t)
    plt.plot(P[:,0], P[:,1],
             lw= 1.5*v, # Different line width for different trajectories
             color=col, label='P0=(%.f, %.f)' % ( P0[0], P0[1]) )
ymax = plt.ylim(ymin=0)[1]
xmax = plt.xlim(xmin=0)[1]
nb_points   = 20
x = np.linspace(0, xmax, nb_points)
y = np.linspace(0, ymax, nb_points)
X1,Y1  = meshgrid(x, y)
DX1, DY1 = dP_dt([X1, Y1])
M = (hypot(DX1, DY1))
M[M == 0] = 1.
DX1 /= M
DY1 /= M
plt.title('Trajectories and direction fields')
Q = plt.quiver(X1, Y1, DX1, DY1, M, pivot='mid', cmap=plt.cm.plasma)
plt.xlabel('Number of rabbits')
plt.ylabel('Number of coyotes')
plt.legend()
plt.grid()
plt.xlim(0, xmax)
plt.ylim(0, ymax)
plt.show()

```

## Multiple Predators and Preys Relationship

The previous relationship could be extended to multiple predators and preys relationship

```python
%matplotlib inline
initial_rabbit2 = widgets.FloatText(description="Initial Rabbit", style=style,value=2)
initial_coyote2 = widgets.FloatText(description="Initial Coyote",style=style, value=2)
initial_deer2 = widgets.FloatText(description="Initial Deer", style=style,value=1)
initial_wolf2 = widgets.FloatText(description="Initial Wolf", style=style,value=1)
population_capacity = widgets.FloatText(description="capacity",style=style, value=10)
population_capacity_rabbit = widgets.FloatText(description="capacity rabbit",style=style, value=3)
growth_rate_rabbit = widgets.FloatText(description="growth rate rabbit",style=style, value=1)
death_rate_coyote = widgets.FloatText(description="death rate coyote",style=style, value=1)
growth_rate_deer = widgets.FloatText(description="growth rate deer",style=style, value=1)
death_rate_wolf = widgets.FloatText(description="death rate wolf",style=style, value=1)
x1 = widgets.FloatText(description="death rate ratio due to coyote",style=style, value=1)
y1 = widgets.FloatText(description="growth rate ratio due to rabbit", style=style,value=1)
x2 = widgets.FloatText(description="death rate ratio due to wolf",style=style, value=1)
y2 = widgets.FloatText(description="growth rate ratio due to deer", style=style,value=1)
plot2 = widgets.Button(description="Plot Graph")
display(initial_rabbit2, initial_coyote2,initial_deer2, initial_wolf2, population_capacity,
        population_capacity_rabbit, growth_rate_rabbit, growth_rate_deer, death_rate_coyote,death_rate_wolf,
        x1, y1,x2, y2, plot2)
def plot_graph(b):
    clear_output()
    display(initial_rabbit2, initial_coyote2,initial_deer2, initial_wolf2, population_capacity,
        population_capacity_rabbit, growth_rate_rabbit, growth_rate_deer, death_rate_coyote,death_rate_wolf,
        x1, y1,x2, y2, plot2)
    fig = plt.figure()
    ax = fig.add_subplot(111)
    t_m = np.arange(0, 20, 0.5)
    r_m = np.zeros(t_m.shape)
    c_m = np.zeros(t_m.shape)
    d_m = np.zeros(t_m.shape)
    w_m = np.zeros(t_m.shape)
    R_m = initial_rabbit2.value
    C_m = initial_coyote2.value
    D_m = initial_deer2.value
    W_m = initial_wolf2.value
    population_capacity_deer = population_capacity.value - population_capacity_rabbit.value
    for i in range(t_m.shape[0]):
        r_m[i] = R_m
        c_m[i] = C_m
        d_m[i] = D_m
        w_m[i] = W_m

        R_m = R_m + growth_rate_rabbit.value * (population_capacity_rabbit.value - R_m)\
            /(population_capacity_rabbit.value) * R_m - (1 - 1/(x1.value*C_m + 1))*R_m - (1 - 1/(x2.value*W_m + 1))*R_m
        D_m = D_m + growth_rate_deer.value * (population_capacity_deer - D_m) \
            /(population_capacity_deer) * D_m - (1 - 1/(x1.value*C_m + 1))*D_m - (1 - 1/(x2.value*W_m + 1))*D_m

        C_m = C_m - death_rate_coyote.value * C_m  + y1.value*r_m[i]*C_m + y2.value*d_m[i]*C_m
        W_m = W_m - death_rate_wolf.value * W_m  + y1.value*r_m[i]*W_m + y2.value*d_m[i]*W_m

    ax.plot(t_m, r_m, label="rabit")
    ax.plot(t_m, c_m, label="coyote")
    ax.plot(t_m, d_m, label="deer")
    ax.plot(t_m, w_m, label="wolf")
    ax.set(xlabel='time (months)', ylabel='population',
       title='Multiple Predator Prey Relationship')
    ax.grid()
    ax.legend()

plot2.on_click(plot_graph)
```


# Nonlinear vibrations
Part 2

## Van der Pol equation

### Limiting cycle

Method of averaging:

Van der Pol equation for autonomous system with negative damping

$$\ddot{x}-\epsilon(1-x^2)\dot{x}+x=0$$

Use method of averaging to estimate amplitude in case of small nonlinearity $\epsilon \ll 1$:

\begin{aligned}
x&=r\cos(t+\phi)\\
\dot{x}&=-r\sin(t+\phi)\\
\ddot{x}&=-\dot{r}\sin(t+\phi)-r\dot{\phi}\cos(t+\phi)-x
\end{aligned}

where equation for $\dot{x}$ holds if
$$\dot{r}\cos(t+\phi)-r\dot{\phi}\sin(t+\phi)=0$$

Substituting these equations into van der Pol equation yields

$$\dot{r}\sin(t+\phi)+r\dot{\phi}\cos(t+\phi)=-\epsilon(1-x^2)\dot{x}$$

The system can be solved for $\dot{r}$ and $\dot{\phi}$

\begin{aligned}
\dot{r}&=-\epsilon\sin(t+\phi)(1-x^2)\dot{x}\\
\dot{\phi}&=-\frac{1}{r}\epsilon\cos(t+\phi)(1-x^2)\dot{x}
\end{aligned}

Let $\tau=t+\phi$ and $\phi$ is considered almost constant in comparison with $t$ on period of $\sin t$. In this case

\begin{aligned}
\dot{r}&\approx-\epsilon r\frac{1}{2\pi}\int_0^{2\pi}\sin\tau\cdot(1-r^2\cos^2\tau)\sin\tau d\tau=\epsilon\frac{r}{2} \left(1-\frac{r^2}{4}\right)\\
\dot{\phi}&\approx-\epsilon\frac{1}{2\pi}\int_0^{2\pi}\cos\tau\cdot(1-r^2\cos^2\tau)\sin\tau d\tau=0
\end{aligned}

Staedy state solutions to $\dot{r}=0$ are $r=0$ and $r=2$ amplitudes. The first one is unstable singular point and $r=2$ corresponds to a stable limiting cycle.

$$\dot{r}=\epsilon\frac{r}{2} \left(1-\frac{r^2}{4}\right)\quad\implies\quad\frac{dr}{\frac{r}{2}\left(1-\left(\frac{r}{2}\right)^2\right)}=\epsilon dt$$

$$z=\frac{r}{2}, \quad \frac{2 dz}{z(1-z^2)}=\epsilon dt$$

$$\frac{2}{z(1-z^2)}=\frac{2}{z}-\left(\frac{1}{z-1}+\frac{1}{z+1}\right)$$

$$2\int_{z_0}^z\frac{dz}{z}-\left(\int_{z_0}^z\frac{dz}{z-1}+\int_{z_0}^z\frac{dz}{z+1}\right)=\epsilon t$$

$$2\log\frac{z}{z_0}-\left(\log\frac{z-1}{z_0-1}+\log\frac{z+1}{z_0+1}\right)=\epsilon t$$

$$\frac{z_0^2}{z_0^2-1}\cdot\frac{z^2-1}{z^2}=e^{-\epsilon t}$$

$$1-\frac{4}{r^2}=\left(1-\frac{4}{r_0^2}\right)e^{-\epsilon t}$$

$$r=\frac{2}{\sqrt{1-\left(1-4/r_0^2\right)e^{-\epsilon t}}}$$

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint
import sympy as sp
```

```python
r, phi, eps, t = sp.symbols('r, \phi, \epsilon, t')
```

```python
x  =  r * sp.cos(t)
dx = -r * sp.sin(t)

F = eps * (1 - x**2) * dx

dr   = -sp.sin(t) * F
dphi = -sp.cos(t) * F
```

```python
F
```

$\displaystyle - \epsilon r \left(- r^{2} \cos^{2}{\left(t \right)} + 1\right) \sin{\left(t \right)}$

```python
eq1 = (sp.integrate(dr, (t, 0, 2*sp.pi))/2/sp.pi).simplify()
eq1
```

$\displaystyle \frac{\epsilon r \left(4 - r^{2}\right)}{8}$

```python
sp.integrate(dphi, (t,0,2*sp.pi))
```

$\displaystyle 0$

## Relaxation oscillations

### Rayleigh's equation to van de Pol relation

$$\ddot{u}-\epsilon\left(\dot{u}-\frac{1}{3}\dot{u}^3\right)+u=0$$

differentiate Rayleigh's equation:

$$\dddot{u}-\epsilon\left(\ddot{u}-\dot{u}^2\ddot{u}\right)+\dot{u}=0$$

and sybstituying $\dot{u} = v$ results in van der Pol equation:

$$\ddot{v}-\epsilon(1-v^2)\dot{v}+v=0$$

### limiting cycle

substitution

$$\dot{u} = v,\quad\xi=u/\epsilon,\quad\dot{\xi}=v/\epsilon$$

into Rayleigh's eqyation gives 1st order system of equations:

\begin{aligned}
\frac{1}{\epsilon}\dot{v}&=v-\frac{1}{3}v^3-\xi\\
\dot{\xi}&=\frac{v}{\epsilon}
\end{aligned}

Equation for the trajectories on phase plane:

$$\frac{1}{\epsilon^2}\frac{dv}{d\xi}=\frac{v-v^3/3-\xi}{v}$$

### Period
Limiting cycle $\xi=v-\frac{1}{3}v^3$. From $\dot{\xi}=v/\epsilon$ find
$$dt=\epsilon\frac{d\xi}{v}$$

Period can be estimates as integral over the limiting circle.
The symmetry of the curve can be taken into account as well:

$$T=\epsilon\oint\frac{d\xi}{v}=2\epsilon\int_{v_1}^{v_2}\frac{d\xi}{v}=2\epsilon\int_{v_1}^{v_2}\left(\frac{1}{v}-v\right)dv$$

Find the integral limits. Point $v_2$ corresponds to equation $\dot{\xi}=1-v_2^2=0$, thus $v_2=\pm 1$.

Another point can be found from equation
$$\xi(-1)=-2/3=v_1-v_1^3/3$$

$$v_1^3-3v_1-2=0\quad\implies\quad v_1=2$$

$$T=2\epsilon\int_2^1\left(\frac{1}{v}-v\right)dv=2\epsilon\left(\frac{3}{2}-\log 2\right)\approx 1.614\epsilon$$

## Numerical solution

```python
# for numerical integration
def vdp(z, t, e):
    return [ z[1], -z[0] + e*(1-z[0]**2)*z[1] ]
```

```python
eps = 0.3
r0  = 1e-3

t = np.linspace(0, 100, 1000)
sol = odeint(vdp, [r0, 0], t, args=(eps,))

plt.figure(figsize=(12,5))
plt.subplot(1,2,1)
plt.plot(t, sol[:,0], 'b',
         t, 2/np.sqrt(1-(1-4/r0**2)*np.exp(-eps*t)), 'm')
plt.subplot(1,2,2)
plt.plot(sol[:,0], sol[:,1])
plt.grid(True)
plt.show()
```

```python
t = np.linspace(0, 100, 1000)

sol1 = odeint(vdp, [0.1, 0], t, args=(0.1,))
sol2 = odeint(vdp, [3.0, 0], t, args=(0.1,))

plt.figure(figsize=(7,7))
plt.plot(sol1[:,0],sol1[:,1],'b',sol2[:,0],sol2[:,1],'r')
plt.grid(True)
plt.xlabel('$x$')
plt.ylabel('$\dot{x}$')
plt.title('Limiting cycle')
plt.show()
```

```python
r0 = 1e-3
eps = 1

t = np.linspace(0, 30, 300)
sol = odeint(vdp, [r0, 0], t, args=(eps,))

plt.figure(figsize=(12,5))
plt.subplot(1,2,1)
plt.plot(t, sol[:,0], 'b',
         t, 2/np.sqrt(1-(1-4/r0**2)*np.exp(-eps*t)), 'm')
plt.subplot(1,2,2)
plt.plot(sol[:,0], sol[:,1])
plt.grid(True)
plt.show()
```

Relaxation oscillations similar to multivibrator at $\epsilon \gg 1$.

Period depends on $\epsilon$.

```python
r0 = 0.01
eps = 10

t = np.linspace(0, 60, 2000)
sol = odeint(vdp, [r0, 0], t, args=(eps,))

plt.figure(figsize=(12,5))
plt.subplot(1,2,1)
plt.plot(t, sol[:,0])
plt.subplot(1,2,2)
plt.plot(sol[:,0], sol[:,1])
plt.grid(True)
plt.show()
```

## Notes

"Dynamics of a System Exhibiting the Global Bifurcation of a Limit Cycle at Infinity" by W.L.Keith and R.H.Rand, Int. J. Non-Linear Mechanics, 20:325-338 (1985)

```python

```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
import yahoofinancials
import seaborn as sn
from scipy.stats import anderson,kstest,shapiro, probplot
```

```python
stock = 'TLKM.JK'
tlkm = yf.Ticker(stock).history(period = '2y', interval = '1mo',actions=False)
tlkm.dropna(inplace = True)
tlkm['log_return'] = np.log(tlkm['Close']) - np.log(tlkm['Close'].shift(1))
tlkm['log_return'].dropna(inplace = True)

stock = 'BBNI.JK'
bbni = yf.Ticker(stock).history(period = '2y', interval = '1mo',actions=False)
bbni.dropna(inplace = True)
bbni['log_return'] = np.log(bbni['Close']) - np.log(bbni['Close'].shift(1))
bbni['log_return'].dropna(inplace = True)
bbni['log_return']

stock = 'ASII.JK'
asii = yf.Ticker(stock).history(period = '2y', interval = '1mo',actions=False)
asii.dropna(inplace = True)
asii['log_return'] = np.log(asii['Close']) - np.log(asii['Close'].shift(1))
asii['log_return'].dropna(inplace = True)
asii['log_return']

df = pd.concat([tlkm['log_return'],bbni['log_return'],asii['log_return']], axis = 1, keys =['tlkm', 'bbni', 'asii']).dropna()
df = df[:-1]
```

```python
df.mean()
```

    tlkm    0.019182
    bbni    0.032105
    asii    0.017229
    dtype: float64

```python
df.cov()
```

      tlkm
      bbni
      asii

      tlkm
      0.006196
      0.004734
      -0.000863

      bbni
      0.004734
      0.011494
      0.001603

      asii
      -0.000863
      0.001603
      0.007762

# TLKM dan BBNI

```python
df.mean().iloc[:2]
```

    tlkm    0.019182
    bbni    0.032105
    dtype: float64

```python
Q = df.cov().iloc[:2][['tlkm','bbni']]
Q
```

      tlkm
      bbni

      tlkm
      0.006196
      0.004734

      bbni
      0.004734
      0.011494

```python
import sympy as sym
x = sym.Symbol('x')
```

```python
u = np.array([[0],[1]])
v = np.array([[1],[-1]])
y1 = sym.solve(sym.expand((u + v*x).T.dot(Q).dot((u + v*x)).reshape((1))[0]).diff(x),x)
```

```python
sym.expand((u + v*x).T.dot(Q).dot((u + v*x)).reshape((1))[0]).diff(x).diff(x)
```

$\displaystyle 0.0164429521011939$

```python
print('Diperoleh besar proporsi untuk saham pertama adalah : ',y1[0])
```

    Diperoleh besar proporsi untuk saham pertama adalah :  0.822258088558763

```python
retur = y1[0]*df.mean().iloc[0] + (1-y1[0])*df.mean().iloc[1]
var = y1[0]**2*Q.iloc[0][0]**2 + (1-y1[0])**2*Q.iloc[1][1]**2 + 2*y1[0]*(1-y1[0])*Q.iloc[0][1]
print('Diperoleh total return', retur*100, 'dan total variansi', var*100)
```

    Diperoleh total return 2.14790463192366 dan total variansi 0.141393865913339


```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import time
from scipy import stats
from sklearn.datasets.samples_generator import make_regression
```

```python
%matplotlib inline
```

```python
def timeit(method):
    def timed(*args, **kw):
        ts = time.time()
        result = method(*args, **kw)
        te = time.time()
        if 'log_time' in kw:
            name = kw.get('log_name', method.__name__.upper())
            kw['log_time'][name] = int((te - ts) * 1000)
        else:
            print('%r  %2.2f ms' % \
                  (method.__name__, (te - ts) * 1000))
        return result
    return timed
```

```python
X, y = make_regression(n_samples = 10000,
                       n_features=1,
                       n_informative=1,
                       noise=20,
                       random_state=2019)
```

```python
plt.scatter(X,y)
```

```python
x = X.flatten()
slope, intercept,_,_,_ = stats.linregress(x,y)
print (slope)
print (intercept)
```

    10.650659469662912
    0.2873381752645907

```python
best_fit = np.vectorize(lambda x: x * slope + intercept)
plt.plot(x,y, 'o', alpha=0.5)
grid = np.arange(-3,3,0.1)
plt.plot(grid,best_fit(grid), '.')
```

```python
y = y.reshape(-1,1)
```

# Gradient Descent

## Cost Function & Gradients

 The equation for calculating cost function and gradients are as shown below. Please note the cost function is for Linear regression. For other algorithms the cost function will be different and the gradients would have  to be derived from the cost functions

Cost
\begin{equation}
J(\theta) = 1/2m \sum_{i=1}^{m} (h(\theta)^{(i)} - y^{(i)})^2
\end{equation}

Gradient

\begin{equation}
\frac{\partial J(\theta)}{\partial \theta_j} = 1/m\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_j^{(i)}
\end{equation}

Gradients
\begin{equation}
\theta_0: = \theta_0 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}
\begin{equation}
\theta_1: = \theta_1 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_1^{(i)})
\end{equation}
\begin{equation}
\theta_2: = \theta_2 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_2^{(i)})
\end{equation}

\begin{equation}
\theta_j: = \theta_j -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}

```python
def cal_cost(theta,X,y):
    m = len(y)
    predictions = X.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost
```

```python
@timeit
def gradient_descent(X, y, theta, learning_rate=0.01, iterations=100):
    m = len(y)
    cost_history = np.zeros(iterations)
    theta_history = np.zeros((iterations, 2))
    for it in range(iterations):
        prediction = np.dot(X, theta)
        theta = theta - (1 / m) * learning_rate * (X.T.dot((prediction - y)))
        theta_history[it, :] = theta.T
        cost_history[it] = cal_cost(theta, X, y)
    return theta, cost_history, theta_history
```

 Let's start with 1000 iterations and a learning rate of 0.05. Start with theta from a Gaussian distribution

```python
lr = 0.05
n_iter = 1000
theta = np.random.randn(2, 1)
X_b = np.c_[np.ones((len(X), 1)), X]
theta, cost_history, theta_history = gradient_descent(X_b, y, theta, lr, n_iter)
print("Theta0: {:0.3f},\nTheta1:{:0.3f}".format(theta[0][0], theta[1][0]))
print("Final cost/MSE:  {:0.3f}".format(cost_history[-1]))
```

    'gradient_descent'  156.16 ms
    Theta0: 0.287,
    Theta1:10.651
    Final cost/MSE:  19910372815.825

 Let's plot the cost history over iterations

```python
fig,ax = plt.subplots(figsize=(12,8))
ax.set_ylabel('J(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(n_iter),cost_history,'b.')
```

 After around 60 iterations the cost is flat so the remaining iterations  are not needed or will not result in any further optimization. Let us zoom in till iteration 100 and see the curve

```python
fig,ax = plt.subplots(figsize=(10,8))
_=ax.plot(range(100),cost_history[:100],'b.')
```

# Stochastic Gradient Descent

```python
@timeit
def stocashtic_gradient_descent(X, y, theta, learning_rate=0.01, iterations=10):
    m = len(y)
    cost_history = np.zeros(iterations)
    for it in range(iterations):
        cost = 0.0
        for i in range(m):
            rand_ind = np.random.randint(0, m)
            X_i = X[rand_ind, :].reshape(1, X.shape[1])
            y_i = y[rand_ind].reshape(1, 1)
            prediction = np.dot(X_i, theta)
            theta = theta - (1 / m) * learning_rate * (X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta, X_i, y_i)
        cost_history[it] = cost
    return theta, cost_history
```

```python
lr = 0.05
n_iter = 1000
theta = np.random.randn(2, 1)
X_b = np.c_[np.ones((len(X),1)),X]
theta,cost_history = stocashtic_gradient_descent(X_b,y,theta,lr,n_iter)
print("Theta0: {:0.3f},\nTheta1:{:0.3f}".format(theta[0][0], theta[1][0]))
print("Final cost/MSE:  {:0.3f}".format(cost_history[-1]))
```

    'stocashtic_gradient_descent'  193933.83 ms
    Theta0: 0.321,
    Theta1:10.681
    Final cost/MSE:  2016254.635

```python
fig, ax = plt.subplots(figsize=(10, 8))
ax.set_ylabel("{J(Theta)}", rotation=0)
ax.set_xlabel("{Iterations}")
theta = np.random.randn(2, 1)
_ = ax.plot(range(n_iter), cost_history, "b.")
```

# Mini Batch Gradient Descent

```python
@timeit
def minibatch_gradient_descent(X,y,theta,learning_rate=0.01, iterations=10, batch_size =20):
    m = len(y)
    cost_history = np.zeros(iterations)
    n_batches = int(m/batch_size)
    for it in range(iterations):
        cost =0.0
        indices = np.random.permutation(m)
        X = X[indices]
        y = y[indices]
        for i in range(0,m,batch_size):
            X_i = X[i:i+batch_size]
            y_i = y[i:i+batch_size]
            X_i = np.c_[np.ones(len(X_i)),X_i]
            prediction = np.dot(X_i,theta)
            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta,X_i,y_i)
        cost_history[it]  = cost
    return theta, cost_history
```

```python
lr = 0.05
n_iter = 1000
theta = np.random.randn(2, 1)
theta,cost_history = minibatch_gradient_descent(X,y,theta,lr,n_iter)
print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    'minibatch_gradient_descent'  24359.26 ms
    Theta0:          0.288,
    Theta1:          10.651
    Final cost/MSE:  39820314.310

```python
fig, ax = plt.subplots(figsize=(10, 8))
ax.set_ylabel("{J(Theta)}", rotation=0)
ax.set_xlabel("{Iterations}")
theta = np.random.randn(2, 1)
_ = ax.plot(range(n_iter), cost_history, "b.")
```

```python
fig,ax = plt.subplots(figsize=(10,8))
_=ax.plot(range(100),cost_history[:100],'b.')
```

```python

```


The [Fourier convolution theorem](https://en.wikipedia.org/wiki/Convolution_theorem) states that convolution in real space is equivalent to multiplication in Fourier space. Mathematically:
\begin{equation}
\mathcal{F}(f * g) = \mathcal{F}(f) \times \mathcal{F}(g)
\end{equation}
When used numerically one has to wonder what happens to the edges of the data. As the FFT implicitly assumes that the data repeats forever this means that Fourier convolution done with an FFt is equivalent to convolving in real space with _wrapped_ data.

```python
import numpy as np
from numpy.fft import *

%matplotlib inline
from matplotlib import pyplot as plt
from scipy.ndimage import convolve
from scipy.signal import fftconvolve
```

```python
plt.set_cmap("inferno")
```

The basic plan will be to generate some simulation data and a kernel to convolve it with and then test the various methods against one another.

```python
sim_size = 128
# make kernel
kernel = np.zeros((sim_size, sim_size))
kernel[sim_size // 2 - 2 : sim_size // 2 + 3, sim_size // 2 - 2 : sim_size // 2 + 3] = 1
# make data
data = np.random.randn(sim_size, sim_size)
plt.matshow(kernel)
plt.matshow(data)
```

```python
# do the fourier convolution, "matlab" style
k_kernel = rfftn(ifftshift(kernel), data.shape)
k_data = rfftn(data, data.shape)
convolve_data0 = irfftn(k_kernel * k_data, data.shape)
plt.matshow(convolve_data0)
```

```python
# make sure that the kernel is placed in the right place (imaginary part should be zero)
plt.matshow(k_kernel.real)
plt.colorbar()
plt.matshow(k_kernel.imag)
plt.colorbar()
```

```python
# check that real space convolution works as expected
np.allclose(convolve(data, np.ones((5, 5))), convolve(data, kernel))
```

```python
# check reflection mode
convolve_data_reflect = convolve(data, np.ones((5, 5)))
plt.matshow(convolve_data_realspace - convolve_data0)
np.allclose(convolve_data_realspace, convolve_data0)
```

```python
# check wrap mode
convolve_data_wrap = convolve(data, np.ones((5, 5)), mode="wrap")
plt.matshow(convolve_data_wrap - convolve_data0)
plt.colorbar()
np.allclose(convolve_data_wrap, convolve_data0)
```

It's clear from this that Fourier convolution (using the FFT) is equivalent to real space convolution when the edges are wrapped around.

Now let's take a look at SciPy's `fftconvolve` function.

```python
# scipy's FFT convolution doesn't work quite the same way, it will pad
# out the data first with zeros so that the convolution doesn't wrap
# around, this leads to some shifting.
convolve_data_sp = fftconvolve(data, np.ones((5, 5)), "same")
plt.matshow(convolve_data_sp - convolve_data0)
plt.colorbar()
np.allclose(convolve_data_sp, convolve_data0)
```

```python
# note that if we had used a kernel (which was already fft_padded) we'd
# have to reverse it because of the way fftconvolve pads the data internally
convolve_data_sp = fftconvolve(data, kernel[::-1, ::-1], "same")
plt.matshow(convolve_data_sp - convolve_data0)
plt.colorbar()
np.allclose(convolve_data_sp, convolve_data0)
```

```python
# But if we pad with zeros then the convolutions agree
convolve_data_zeros = convolve(data, np.ones((5, 5)), mode="constant")
plt.matshow(convolve_data_zeros - convolve_data_sp)
plt.colorbar()
np.allclose(convolve_data_zeros, convolve_data_sp)
```

The main findings are:
- Simple straight forward fft convolve is equivalent to convolution with wrapping the edges.
- SciPy's version of fft convolve is equivalent to convoltuion with padding with zeros at the edges.

Now the question is: when does the fftshifting really have to happen in the straight forward version?

```python
# need new data for this
from skimage.draw import circle_perimeter

data = np.zeros((sim_size, sim_size))
data[circle_perimeter(sim_size // 2, sim_size // 2, sim_size // 4)] = 1

fig, axs = plt.subplots(2, 3, sharex=True, sharey=True, figsize=(12, 8))
axs = axs.ravel()

k_kernel = rfftn(ifftshift(kernel), data.shape)
k_data = rfftn(data, data.shape)
convolve_data0 = irfftn(k_kernel * k_data, data.shape)
axs[0].matshow(convolve_data0)
axs[0].set_title("Matlab Method")

k_kernel = rfftn(kernel, data.shape)
k_data = rfftn(data, data.shape)
convolve_data1 = irfftn(k_kernel * k_data, data.shape)
axs[1].matshow(convolve_data1)
axs[1].set_title("No shifting")

k_kernel = rfftn(kernel, data.shape)
k_data = rfftn(ifftshift(data), data.shape)
convolve_data2 = irfftn(k_kernel * k_data, data.shape)
axs[2].matshow(convolve_data2)
axs[2].set_title("Shift Data")

k_kernel = rfftn(kernel, data.shape)
k_data = rfftn(data, data.shape)
convolve_data3 = irfftn(fftshift(k_kernel * k_data), data.shape)
axs[3].matshow(convolve_data3)
axs[3].set_title("Shift product")

k_kernel = rfftn(kernel, data.shape)
k_data = rfftn(data, data.shape)
convolve_data4 = irfftn(fftshift(k_kernel) * k_data, data.shape)
axs[4].matshow(convolve_data4)
axs[4].set_title("Shift k_kernel")

k_kernel = rfftn(ifftshift(kernel), data.shape)
k_data = rfftn(ifftshift(data), data.shape)
convolve_data5 = fftshift(irfftn(k_kernel * k_data, data.shape))
axs[5].matshow(convolve_data5)
axs[5].set_title("fftshift result")
```

Shifting has to happen to either the data or the kernel which makes sense as they're mathematically equivalent in this case. Or we can just fftshift the result.

All of these make sense because if we don't shift the kernel then it's equivalent to convolving with a shifted (or split) kernel. All of which doesn't reall matter because the FFT assumes that the data is wrapped around (repeating infinitely).

```python

```