```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

# Numerical differentiation

Suppose we want to apply Newton's method to a function that we know how to evaluate, but don't have code to differentiate.  This is often because it's difficult/error-prone to write or because the interface by which we call it does not support derivatives.  (Commercial packages often fall in this category.)  So we just have a black box function $f(x)$ and wish to approximate its derivative,
$$ f'(x) = \lim_{h\to 0} \frac{f(x+h) - f(x)}{h} .$$

```python
def diff(f, x, h=1e-5):
    return (f(x + h) - f(x)) / h

diff(np.sin, 0.7, 1e-8) - np.cos(0.7)
```

    2.297793622041411e-09

```python
x = .5
diff(np.tan, x, 1e-8) - 1/np.cos(x)**2
```

    1.77456063177317e-08

This formula works pretty well for the functions above.  It isn't always so nice, however:

```python
x = 3.14/2
[(h, diff(np.tan, x, h) - 1/np.cos(x)**2)
 for h in [1e-14, 1e-12, 1e-10, 1e-8, 1e-6, 1e-4]]
```

    [(1e-14, -1271.3873432741966),
     (1e-12, 140.14843387738802),
     (1e-10, 0.32726590032689273),
     (1e-08, 19.79343111347407),
     (1e-06, 1982.7670766180381),
     (0.0001, 226466.6387994727)]

Recall that the derivative is ill conditioned for this function
$$ \kappa(f') = |f''| \frac{|x|}{|f'|} $$
but also that our best accuracy is worse than

```python
y = 3.14/2;
print(
    np.tan(y), # f
    np.cos(y)**(-2), # f'
    np.cos(y)**(-2) * y / np.tan(y), # cond(f)
    2*np.cos(y)**(-3)*np.sin(y) * y * np.cos(y)**2, # cond(f')
)
```

    1255.7655915007897 1576948.220797328 1971.5532288895718 3943.1039573124795

In other cases, we see excellent accuracy so long as the size of $h$ is chosen appropriately.

```python
x = 1e4
[(h, diff(np.log, x, h) - 1/x)
 for h in [1e-14, 1e-12, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2]]
```

    [(1e-14, -0.0001),
     (1e-12, -0.0001),
     (1e-10, -1.1182158029987482e-05),
     (1e-08, 8.89005823409637e-09),
     (1e-06, 8.274037095116864e-12),
     (0.0001, -9.489531298885641e-12),
     (0.01, -5.0168102921151377e-11)]

Asking the user to choose a good value of $h$ is a leaky abstraction and unmanageable complexity in many applications.

### Automatically choosing a suitable $h$
This is one simple and popular choice.

```python
def diff_wp(f, x, eps=1e-8):
    """Numerical derivative with Walker and Pernice (1998) choice of step"""
    h = eps * (1 + abs(x))
    return (f(x+h) - f(x)) / h

x = 1e4
[(eps, diff_wp(np.log, x, eps) - 1/x) for eps in [1e-14, 1e-12, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2]]
```

    [(1e-14, -1.1191038926094873e-05),
     (1e-12, -1.109830782817004e-09),
     (1e-10, -1.109830782817004e-09),
     (1e-08, -8.599665508239422e-12),
     (1e-06, -5.0162259288282114e-11),
     (0.0001, -5.000167712764913e-09),
     (0.01, -4.967408090441846e-07)]

```python
x = 0
[(eps, diff_wp(np.exp, x, eps) - np.exp(x)) for eps in [1e-14, 1e-12, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2]]
```

    [(1e-14, -0.0007992778373591136),
     (1e-12, 8.890058234101161e-05),
     (1e-10, 8.274037099909037e-08),
     (1e-08, -6.07747097092215e-09),
     (1e-06, 4.999621836532242e-07),
     (0.0001, 5.0001667140975314e-05),
     (0.01, 0.005016708416794913)]

In both of these experiments, `eps = 1e-8` (that is, $\sqrt{\epsilon_{\text{machine}}}$) produces good results.
This isn't always the case; consider $\log x$ for small values of $x$:

```python
x = 1e-4
[(eps, diff_wp(np.log, x, eps) - 1/x) for eps in [1e-14, 1e-12, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2]]
```

    [(1e-14, -0.11098307828251563),
     (1e-12, -0.0008599665507063037),
     (1e-10, -0.005016225928557105),
     (1e-08, -0.5000167712751136),
     (1e-06, -49.67408090441677),
     (0.0001, -3068.721334766654),
     (0.01, -9538.52419539635)]

This algorithm is imperfect, leaving some scaling responsibility to the user.
It is the default in PETSc's "matrix-free" Newton-type solvers; it's cheap and works well when problems are well scaled.
It's worth considering why we use
$$ h_{WP} = \sqrt{\epsilon_{\text{machine}}} (1 + \lvert x \rvert) $$
instead of $h_? = \epsilon_{\text{machine}} \lvert x \rvert$.
This choice would be scale-invariant, but problematic when $f(0)$ is not small, as in the example $f(x) = e^x$.

### Accuracy of numerical differentiation

#### Taylor expansion

Classical accuracy analysis assumes that functions are sufficiently smooth, meaning that derivatives exist and Taylor expansions are valid within a neighborhood.  In particular,
$$ f(x+h) = f(x) + f'(x) h + f''(x) \frac{h^2}{2!} + \underbrace{f'''(x) \frac{h^3}{3!} + \dotsb}_{O(h^3)} . $$

The big-$O$ notation is meant in the limit $h\to 0$.  Specifically, a function $g(h) \in O(h^p)$ (sometimes written $g(h) = O(h^p)$) when
there exists a constant $C$ such that
$$ g(h) \le C h^p $$
for all sufficiently *small* $h$.

**Note:** When analyzing algorithms, we will refer to cost being $O(n^2)$, for example, where $n\to \infty$.
In this case, the above definition applies for sufficiently *large* $n$.  Whether the limit is small ($h\to 0$) or large ($n \to\infty$) should be clear from context.

#### Discretization error
The `diff` and `diff_wp` functions use a "forward difference" formula
$$ \tilde f'(x) := \frac{f(x+h) - f(x)}{h}.$$
Using the Taylor expansion of $f(x+h),$ we compute the discretization error
$$ \begin{split} \frac{f(x+h) - f(x)}{h} - f'(x) &= \frac{f(x) + f'(x) h + f''(x) h^2/2 + O(h^3) - f(x)}{h} - f'(x) \\
&= \frac{f'(x) h + f''(x) h^2/2 + O(h^3)}{h} - f'(x) \\
&= f''(x) h/2 + O(h^2) .
\end{split} $$

This is the *discretization error* caused by choosing a finite (not infinitesimal) differencing parameter $h$, and the leading order term depends linearly on $h$.
This is also called *truncation error*, due to truncating the Taylor series; the terms are interchangeable.

#### Rounding error
We have an additional source of error, *rounding error*, which comes from not being able to compute $f(x)$ or $f(x+h)$ exactly, nor subtract them exactly.  Suppose that we can, however, compute these functions with a relative error on the order of $\epsilon_{\text{machine}}$.  This leads to
$$ \begin{split}
\tilde f(x) &= f(x)(1 + \epsilon_1) \\
\tilde f(x \oplus h) &= \tilde f((x+h)(1 + \epsilon_2)) \\
&= f((x + h)(1 + \epsilon_2))(1 + \epsilon_3) \\
&= [f(x+h) + f'(x+h)(x+h)\epsilon_2 + O(\epsilon_2^2)](1 + \epsilon_3) \\
&= f(x+h)(1 + \epsilon_3) + f'(x+h)x\epsilon_2 + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}} h)
\end{split}
$$

where each $\epsilon_i$ is an independent relative error on the order of $\epsilon_{\text{machine}}$ and we have used a Taylor expansion at $x+h$ to approximate $f(x \oplus h)$.
We thus write the rounding error in the forward difference approximation as
$$ \begin{split}
\left\lvert \frac{\tilde f(x+h) \ominus \tilde f(x)}{h} - \frac{f(x+h) - f(x)}{h} \right\rvert &=
  \left\lvert \frac{f(x+h)(1 + \epsilon_3) + f'(x+h)x\epsilon_2 + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}} h) - f(x)(1 + \epsilon_1) - f(x+h) + f(x)}{h} \right\rvert \\
  &\le \frac{|f(x+h)\epsilon_3| + |f'(x+h)x\epsilon_2| + |f(x)\epsilon_1| + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}}h)}{h} \\
  &\le \frac{(2 \max_{[x,x+h]} |f| + \max_{[x,x+h]} |f' x| \epsilon_{\text{machine}} + O(\epsilon_{\text{machine}}^2 + \epsilon_{\text{machine}} h)}{h} \\
  &= (2\max|f| + \max|f'x|) \frac{\epsilon_{\text{machine}}}{h} + O(\epsilon_{\text{machine}}) \\
\end{split} $$
where we have assumed that $h \ge \epsilon_{\text{machine}}$.
This error becomes large (relative to $f'$ -- we are concerned with relative error after all)
* $f$ is large compared to $f'$
* $x$ is large
* $h$ is too small

#### Total error and optimal $h$

Suppose we would like to choose $h$ to minimize the combined discretization and rounding error,
$$ h^* = \arg\min_h | f''(x) h/2 | + (2\max|f| + \max|f'x|) \frac{\epsilon_{\text{machine}}}{h} $$
(dropping the higher order terms), which we can compute by differentiating with respect to $h$ and setting the result equal to zero
$$ |f''|/2 - (2\max|f| + \max|f'x|) \frac{\epsilon_{\text{machine}}}{h^2} = 0 $$
which can be rearranged as
$$ h^* = \sqrt{\frac{4\max|f| + 2\max|f'x|}{|f''|}} \sqrt{\epsilon_{\text{machine}}} .$$
Of course this formula is of little use for computing $h$ because all this is to compute $f'$, which we obviously don't know yet, much less $f''$.
However, it does have value:
* It explains why `1e-8` (i.e., $\sqrt{\epsilon_{\text{machine}}}$) was empirically found to be about optimal for well-behaved/scaled functions.
* It explains why even for the best behaved functions, our best attainable accuracy with forward differencing is $\sqrt{\epsilon_{\text{machine}}}$.
* If we have some special knowledge about the class of functions we need to differentiate, we might have bounds on these quantities and thus an ability to use this formula to improve accuracy.  Alternatively, we could run a parameter sweep to empirically choose a suitable $h$, though we would have to re-tune in response to parameter changes in the class of functions.
* If someone claims to have a simple and robust rule for computing $h$ then this formula tells us how to build a function that breaks their rule.  There are no silver bullets.
* If our numerical differentiation routine produces a poor approximation for some function that we run into in the wild, this helps us explain what happened and how to fix it.

### Centered difference

Instead of the forward difference approximation
$$ \frac{f(x+h) - f(x)}{h} $$
we could use the centered difference formula,
$$ \frac{f(x+h) - f(x-h)}{2h} . $$
(One way to derive this formula is to average a forward and backward difference.  We will learn a more general method later in the course when we do interpolation.)
We can compute the discretization error by Taylor expansion,
\begin{split} \frac{f(x) + f'(x)h + f''(x)h^2/2 + f'''(x)h^3/6 - f(x) + f'(x)h - f''(x)h^2/2 + f'''(x) h^3/6 + O(h^4)}{2h} \\
= f'(x) + f'''(x)h^2/6 + O(h^3) \end{split}
showing that the leading error term is of order $h^2$, versus order $h$ for forward differences.
A similar computation including rounding error will find that the optimal $h$ is now of order $\sqrt[3]{\epsilon_{\text{machine}}}$ so the best attainable accuracy is $\epsilon_{\text{machine}}^{2/3}$.
This accuracy improvement (versus $\sqrt{\epsilon_{\text{machine}}}$) is significant, but we'll also see that it is twice as expensive when computing derivatives of multi-variate functions.

# Symbolic differentiation

We've been differentiating basic mathematical functions, for which there is a formula for the derivative.
Symbolic differentiation is a tool that can compute those expressions (and generate code to evaluate the expressions numerically).

```python
import sympy
from sympy.abc import x

f = sympy.cos(x**sympy.pi) * sympy.log(x)
f
```

$\displaystyle \log{\left(x \right)} \cos{\left(x^{\pi} \right)}$

```python
sympy.diff(f, x)
```

$\displaystyle - \frac{\pi x^{\pi} \log{\left(x \right)} \sin{\left(x^{\pi} \right)}}{x} + \frac{\cos{\left(x^{\pi} \right)}}{x}$

```python
sympy.ccode(f, 'y')
```

    'y = log(x)*cos(pow(x, M_PI));'

```python
sympy.fcode(f, 'y')
```

    '      y = log(x)*cos(x**3.1415926535897932d0)'

```python
f.evalf(40, subs={x: 1.9})
```

$\displaystyle 0.215513413838041906745231945917755720873$

```python
def g(x, m=np):
    y = x
    for i in range(2):
        # a = m.log(y)
        # b = y ** m.pi
        # c = m.cos(b)
        # y = c * a
        y = m.cos(y**m.pi) * m.log(y)
    return y

gexpr = g(x, m=sympy)
gexpr
```

$\displaystyle \log{\left(\log{\left(x \right)} \cos{\left(x^{\pi} \right)} \right)} \cos{\left(\left(\log{\left(x \right)} \cos{\left(x^{\pi} \right)}\right)^{\pi} \right)}$

```python
sympy.diff(gexpr, x)
```

$\displaystyle - \frac{\pi \left(\log{\left(x \right)} \cos{\left(x^{\pi} \right)}\right)^{\pi} \left(- \frac{\pi x^{\pi} \log{\left(x \right)} \sin{\left(x^{\pi} \right)}}{x} + \frac{\cos{\left(x^{\pi} \right)}}{x}\right) \log{\left(\log{\left(x \right)} \cos{\left(x^{\pi} \right)} \right)} \sin{\left(\left(\log{\left(x \right)} \cos{\left(x^{\pi} \right)}\right)^{\pi} \right)}}{\log{\left(x \right)} \cos{\left(x^{\pi} \right)}} + \frac{\left(- \frac{\pi x^{\pi} \log{\left(x \right)} \sin{\left(x^{\pi} \right)}}{x} + \frac{\cos{\left(x^{\pi} \right)}}{x}\right) \cos{\left(\left(\log{\left(x \right)} \cos{\left(x^{\pi} \right)}\right)^{\pi} \right)}}{\log{\left(x \right)} \cos{\left(x^{\pi} \right)}}$

# Hand-coding derivatives

The size of these expressions grow exponentially in the number of loop iterations, yet one can write efficient code for computing the derivative by hand.  We use the variational notation

$$ \operatorname{d} f = f'(x) \operatorname{d} x $$

which allows us to break a large computation into simple pieces that we can compute incrementally, instead of trying to build up expressions for complicated functions.  That is, we can differentiate a composition $h(g(f(x)))$ as

\begin{align}
  \operatorname{d} h &= h' \operatorname{d} g \\
  \operatorname{d} g &= g' \operatorname{d} f \\
  \operatorname{d} f &= f' \operatorname{d} x.
\end{align}
Consider our example above.

```python
def gprime(x):
    y = x
    dy = 1
    for i in range(2):
        a = np.log(y)
        da = 1/y * dy
        b = y ** np.pi
        db = np.pi * y ** (np.pi - 1) * dy
        c = np.cos(b)
        dc = -np.sin(b) * db
        y = c * a
        dy = dc * a + c * da
    return y, dy

print('by hand', gprime(1.9))
print('numerical', diff_wp(g, 1.9))
```

    by hand (-1.5346823414986814, -34.03241959914048)
    numerical -34.032439961925064

* This code is pretty mechanical to write
* It's hard to maintain as you add new features
* It's hard to debug
  * You can test using finite differencing
  * You can take apart pieces for unit testing and/or debugging
* If you know you'll be writing this sort of code, plan ahead!

### Variational notation is handy (an example)

We'll differentiate the expression

$$ I = A^{-1} A $$
applying the product rule

$$ 0 = A^{-1} (\operatorname dA) + (\operatorname dA^{-1}) A $$
and collect terms

$$ \operatorname dA^{-1} = - A^{-1} (\operatorname dA) A^{-1}. $$

This expression for the derivative $\operatorname d A^{-1}$ in direction $\operatorname d A$ is useful when differentiating algorithmn that involve linear algebra.

## Reverse-mode

What we've done above is called "forward mode", and amounts to placing the parentheses in the chain rule like

$$ \operatorname d h = \frac{dh}{dg} \left(\frac{dg}{df} \left(\frac{df}{dx} \operatorname d x \right) \right) .$$

The expression means the same thing if we rearrange the parentheses,

$$ \operatorname d h = \left( \left( \left( \frac{dh}{dg} \right) \frac{dg}{df} \right) \frac{df}{dx} \right) \operatorname d x .$$

```python
def gprime_rev(x):
    # First compute all the values by going through the iteration forwards
    # I'm unrolling two iterations here for clarity ("static single assignment" form)
    # It is possible to write code that keeps the loop structure.
    a1 = np.log(x)
    b1 = x ** np.pi
    c1 = np.cos(b1)
    y1 = c1 * a1
    a2 = np.log(y1)
    b2 = y1 ** np.pi
    c2 = np.cos(b2)
    y = c2 * a2 # Result
    # Now go backwards computing dy/d_ for each variable
    y_ = 1
    y_c2 = y_ * a2
    y_a2 = c2 * y_
    y_b2 = -y_c2 * np.sin(b2) # dy/db2 = dy/dc2 dc2/db2
    y_y1 = y_b2 * np.pi * y1 ** (np.pi - 1) + y_a2 / y1
    y_c1 = y_y1 * a1
    y_a1 = c1 * y_y1
    y_b1 = -y_c1 * np.sin(b1)
    y_x = y_b1 * np.pi * x ** (np.pi - 1) + y_a1 / x
    return y, y_x

print('forward', gprime(1.9))
print('reverse', gprime_rev(1.9))
```

    forward (-1.5346823414986814, -34.03241959914048)
    reverse (-1.5346823414986814, -34.03241959914049)

* This is fairly mechanical, similar to forward-mode
* It is more complicated than forward-mode
* This sort of code is tricky to debug
  * You can test using forward-mode or finite differencing
* We need the results of intermediate computation in reverse order
  * We have to store those values somewhere ("taping" in the literature)
  * Or we have to recompute them (see "hierarchical checkpointing")
* Reverse-mode is also known as the "adjoint" method and "back-propagation".

### Why reverse?

If all we had was scalar functions of scalar inputs, we would never use reverse mode.  But let's suppose we are given a dot product with a constant vector.

$$ f(\mathbf x) = \mathbf c^T \mathbf x = \begin{pmatrix} c_0 & c_1 & c_2 & \dotsb \end{pmatrix} \begin{pmatrix} x_0 \\ x_1 \\ x_2 \\ \vdots \end{pmatrix} $$
and wish to compute the gradient
$$ \nabla_{\mathbf x} f = \frac{\partial f}{\partial \mathbf x} = \begin{pmatrix} \frac{\partial f}{\partial x_0} & \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} & \dotsb \end{pmatrix} . $$

```python
def dot(c, x):
    n = len(c)
    sum = 0
    for i in range(n):
        sum += c[i] * x[i]
    return sum

n = 20
c = np.random.randn(n)
x = np.random.randn(n)
f = dot(c, x)
f
```

    -0.7881902865938712

If we use forward mode, we can only compute one direction at a time, effectively
$$ \left(\nabla_{\mathbf x} f\right) \cdot \operatorname d x $$
for one value of the vector $\operatorname d x$ at a time.
We can compute the full gradient by choosing $\operatorname d x$ to be each column of the identity.

```python
def dot_x(c, x, dx):
    """Compute derivative in direction dx"""
    n = len(c)
    dsum = 0
    for i in range(n):
        dsum += c[i] * dx[i]
    return dsum

def grad_dot(c, x):
    n = len(c)
    I = np.eye(n)
    grad = np.zeros(n)
    for j in range(n):
        dx = I[:,j]
        grad[j] = dot_x(c, x, dx)
    return grad

grad_dot(c, x)
```

    array([ 0.41288309, -0.02618   , -1.63423663,  0.3006944 , -0.25078029,
           -0.45382014,  1.94208405,  0.30730437,  0.45064477, -0.89218614,
           -1.25475455, -0.76033123, -0.54544687, -1.40029722, -0.74489052,
           -0.18788402, -1.00248867,  1.5845576 , -0.29049147,  0.98277278])

We've now traversed the loop with our work as many times as there are components in the vector.  The forward evaluation for `dot` costs $O(n)$ and computing the gradient costs $O(n^2)$ because we have to do $O(n)$ for for each direction and there are $n$ directions.

Compare with reverse-mode

```python
def grad_dot_rev(c, x):
    n = len(c)
    sum_ = np.zeros(n)
    for i in range(n):
        sum_[i] = c[i]
    return sum_

grad_dot_rev(c, x)
```

    array([ 0.41288309, -0.02618   , -1.63423663,  0.3006944 , -0.25078029,
           -0.45382014,  1.94208405,  0.30730437,  0.45064477, -0.89218614,
           -1.25475455, -0.76033123, -0.54544687, -1.40029722, -0.74489052,
           -0.18788402, -1.00248867,  1.5845576 , -0.29049147,  0.98277278])

* We get the same values in only $O(n)$ work!
* The astute reader may recall that we already worked out this case,
$$ \frac{\partial \mathbf c^T \mathbf x}{\partial \mathbf x} = \mathbf c^T .$$

## Shape of the gradient (Jacobian)

Suppose we have a vector-valued function of vector-valued input, $\mathbf f(\mathbf x)$ where $\mathbf f$ has length $m$ and $\mathbf x$ has length $n$.
* The gradient (Jacobian) matrix $J = \nabla_{\mathbf x} \mathbf f$ has shape $m\times n$.
* Usually in optimization, $m=1$ because we only have one objective
* If $m\ll n$ then finite differencing and forward-mode differentiation will be much more expensive than reverse-mode differentiation
  * Find a way to use reverse-mode!
* If $m \approx n$ then either is about as efficient, but forward-mode is simpler.
* If $m \gg n$ then forward-mode is the ticket.
* In real computations, there may be expensive stages that have lower dimension inputs or outputs, in which case those can be captured. An example is
$$ \mathbf f(\mathbf x) = \mathbf q \sigma(\mathbf q^T \mathbf x) $$
where $\sigma$ is an expensive nonlinear function.
The Jacobian $J = \nabla_{\mathbf x} \mathbf f$ is a square matrix, but naive forward- and reverse-mode would both require $n$ evaluations of $\sigma$.
Since $\sigma$ is a scalar-valued function of a scalar argument, $\sigma'(\mathbf q^T \mathbf x)$ is just one number, and thus $J = (\sigma') \mathbf q \mathbf q^T$ is readily available (and you know it's rank-1 so don't need to store all $n^2$ entries). Models of this sort show up frequently in physical modeling.

# Algorithmic (automatic) differentiation

Next, we'll consider ways to have libraries/compilers generate by-hand code such as we see above.
We'll use the [JAX](https://jax.readthedocs.io/en/latest/) library, which offers differentiation of NumPy computations (and offload to GPUs, which we won't use now).
Uncomment the line below if you need to install `jax` and `jaxlib`.

```python
# ! pip install jax jaxlib
```

```python
import jax
import jax.numpy as jnp

def g_jax(x):
    """Same function as before, but using jnp in place of np."""
    y = x
    for i in range(2):
        y = jnp.cos(y**jnp.pi) * jnp.log(y)
    return y

gprime_jax = jax.grad(g_jax)
print(gprime_jax(1.9))
print(gprime(1.9)[1])
```

    -34.03244
    -34.03241959914048

```python
jax.grad(dot)(c, x), x # Differentiates with respect to argument zero
```

    (DeviceArray([-1.9289212 ,  2.097858  ,  0.20503117,  0.22631234,
                  -1.2956989 ,  0.5156321 , -0.5992178 ,  0.56441456,
                  -0.23191658,  0.07927939,  0.451759  ,  1.7861412 ,
                  -1.1517502 ,  0.4325543 ,  0.26122522, -0.44151178,
                  -1.85014   ,  0.8240289 ,  0.69433916,  0.46808633],            dtype=float32),
     array([-1.92892122,  2.09785787,  0.20503117,  0.22631233, -1.29569892,
             0.51563212, -0.59921775,  0.56441458, -0.23191657,  0.07927939,
             0.451759  ,  1.78614113, -1.15175023,  0.4325543 ,  0.26122522,
            -0.44151178, -1.85014002,  0.82402888,  0.69433915,  0.46808634]))

```python
jax.grad(dot, argnums=1)(c, x) - c
```

    DeviceArray([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
                 0., 0., 0., 0., 0.], dtype=float32)

## Software

* Algorithmic differentiation (AD) software has been around for over 40 years
* There are two classical approaches
  * Source transformation: AD tool emits Fortran (or C, etc.) code, which is compiled by a normal compiler
  * Operator overleading: each basic operation is overloaded to transform objects holding values + derivatives
* Source transformation is usually more efficient, retaining loop structure, etc.
* Implementations tend to have poor ergonomics, odd restrictions on use, poor composition.
* Vectorization has been poor with most classical tools.
* AD *implementations* have come a long way in the past few years (despite the math being old)
* Just-in-time compilation and extensive software engineering
* Exemplars:
  * [JAX](https://jax.readthedocs.io/en/latest/) for Python
  * [Zygote.jl](http://fluxml.ai/Zygote.jl/latest/) for Julia
* AD is great within its domain, but is still intrusive (especially for multi-language projects, languages with poor AD tooling, etc.).  Even in JAX, you'll see [various constraints](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html), such as that you can't in-place update an array.

```python
z = jnp.zeros(3)
z[1] = 1
```

* If you work in this space, you'll eventually learn to judge when to use AD and when to hand-code a derivative.  This type of decision lies at the intersection of numerical analysis and software engineering.


# Profit Maximizing Input Choices

Consider the profit maximization problem a firm facing production technology $A\cdot F(K, L)$ with inputs $K,L$ having market prices of $r, w > 0$ respectively and can sell its product at price $p>0$.

\begin{align}
    \max_{K,L \geq 0} \; A\cdot F(K, L) p - rK - wL
\end{align}

Assume that the technology is convex and captial instensive such that output can be described from inputs as

\begin{align}
    A \cdot F(K, L) = A \cdot K^{2/3}\cdot L^{1/3}
\end{align}

where $A > 0$ is a constant representing a total factor productivity.

```python
from sympy import *
init_printing()
```

```python
# Declare the symbols we need to setup
# the firm's profit maximization problem.
K, L, w, r, A, p = symbols('K L w r A p',
                             positive=True,
                             real=True,
                             finite=True)

a = Rational(1,2)
b = Rational(1,3)
```

```python
# Assume Cobb-Douglas production technology
F = A * (K**a) * (L**b)

# Firm faces input prices:
#    w <-- wage rate of labor
#    r <-- rental rate on capital
cost = r * K + w * L

# Firm profit is revenue - price
profit = p * F - cost

# display profit function
Eq(Symbol('\pi(K,L)'), profit)
```

Now that we have setup the profit function we want to identify the pair of input choices $(K, L)$ that will maximize profits.  This means we want to find at the very least a local maxima.  We have to satisfy a set of first order conditions and a second order condition.  In particular we will need the gradiant $\nabla \pi(K,L) = \mathbf{0}$ to be zero at $(K,L)$ to identify a critical point.  Next, we will determine whether such critical points are a maximum, minimum or saddle point.

```python
# Differentiate the profit function
# w.r.t. K and L.
DK = profit.diff(K)
DL = profit.diff(L)

DK, DL
```

Both of the first order conditions must be satisfied simultaneously for a critical point.  So we solve the system of two nonlinear equations,

\begin{align}
    \frac{\partial \pi(K,L)}{\partial K} &= 0 \\
    \frac{\partial \pi(K,L)}{\partial L} &= 0
\end{align}

```python
sols = solve([DK, DL],[K, L], dict=True)[0]
sols
```

The above solutions for $(K, L)$ give values in terms of constant parameters that satisfy the first order condition, making $(K, L)$ a critical point.  Next, we need to test the second order condition by deriving the matrix of second partial derivatives, known as a Hessian matrix.

```python
# Derive the second and cross-partial
# derivatives of the profit function
# and substitute in the values (K, L)
# we found as critical points.
pi_KK = DK.diff(K).subs({K: sols[K], L: sols[L]})
pi_KL = DK.diff(L).subs({K: sols[K], L: sols[L]})
pi_LL = DL.diff(L).subs({K: sols[K], L: sols[L]})
pi_LK = DL.diff(K).subs({K: sols[K], L: sols[L]})

# Place the derivatives into a matrix
# in the proper order.
hessian = Matrix([
    [pi_KK, pi_KL],
    [pi_LK, pi_LL]
])

hessian
```

Now that we have the Hessian matrix for this problem, we need to determine whether it is negative-definite.  In a univariate optimization problem with maximand $f(x)$ the second order test for a maximum is $f''(x^*) < 0$ where $x^*$ solves $f'(x^*) = 0$ (or its a critical point).  The analog to this in multivariate optimization is to identify the signs of principal minors of the matrix or its eigenvalues to determine whether the matrix is "positive" (positive definite) or "negative" (negative definite) or zero.

In this case, the $2\times 2$ Hessian matrix will be negative definite (a maximum) if when evaluated at the $(K^*,L^*)$ that satisfy the first order condition the following holds.

1. $\pi_{KK}(K^*, L^*) < 0$, and
2. $\det(\text{Hessian}(K^*,L^*)) > 0$

```python
# For the first condition we substitute in
# the optimal choices (K, L) into the derivative
# function obtained earlier and test its sign.
pm1 = pi_KK.subs({K: sols[K], L: sols[L]})
pm1.is_negative
# Is it negative?  YES.
```

    True

```python
# Calculate the determinate of the Hessian
# evaluated at the critical point and test
# its sign.
detH = hessian.det()
detH.is_positive
# Is it positive? YES.
```

    True


# Introduction

This notebook gives examples of operations that can be performed using the **Numpy** and **Sympy** module. The documentation for **Sympy** can be found [here](https://docs.sympy.org/latest/index.html).
A drafting space is available at the end of the notebook

```python
import numpy as np
from sympy import *
```

```python
init_printing()  # Init pretty print as default

X = symbols('X') # Use for polynomials variables
x = symbols('x') # Declare new variable
y = symbols('y')
a = symbols('a', commutative=False)
b = symbols('b', commutative=False)
a = symbols('a')
b = symbols('b')
c = symbols('c')
d = symbols('d')
```

# Equations

Solve system $$\begin{cases} x + 5y -2 = 0 \\ -3x + 6y - 15 = 0 \end{cases}$$

```python
solve([x + 5*y - 2, -3*x + 6*y - 15], [x, y])
```

# Integration

```python
integrate(x ** 2 + x + 1, x)  # Give the primitive
```

```python
integrate(x ** 2 + x + 1, (x, 0, 1))  # Give the value of the integral
```

# Matrix
The reference of **Sympy** is [here](http://docs.sympy.org/latest/modules/matrices/matrices.html). See also this [tutorial](https://docs.sympy.org/latest/tutorial/matrices.html) for a quick introduction to **Sympy** matrices.

### Basics

```python
n, m = 3, 4 # Used later for some matrices
```

```python
M = Matrix([
    [a, b],
    [c, d]
])

U = Matrix([
    [x],
    [y]
])

M * U
```

```python
U = Matrix([[x, y]])

U * M
```

```python
CNOT = Matrix([
[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 0, 1],
[0, 0, 1, 0]
])

H = 1/np.sqrt(2) * np.matrix([
    [1, 1],
    [1, -1]
])

psi = 1/np.sqrt(2) * np.matrix([[1], [1]])
```

```python
CNOT * Matrix([0, 1, a, b])
```

```python
ket = H.dot(psi)
ket
```

    matrix([[1.],
            [0.]])

```python
bra = ket.transpose()
bra
```

    matrix([[1., 0.]])

### Special matrices

```python
I_3 = eye(n) # Identity matrix
I_3
```

```python
M, M2 = zeros(2, 3), zeros(n) # Zero matrix
M, M2
```

```python
M, M2 = ones(1, 2), ones(n) # Matrix of ones
M, M2
```

# Reduction

```python
M = Matrix([[1, 1], [1, 1]])
M
```

```python
P, D = M.diagonalize()
P, D
```

```python
P * D * P.inv()
```

```python
M.eigenvals(), M.eigenvects(), M.charpoly(x) # Eigenvalues, associated eigenvectors and characteristic polynomial
```

```python
np.linalg.eig(H) # Eigenvalues and associated eigenvectors of the Hadamard gate
```

    (array([ 1., -1.]), matrix([[ 0.92387953, -0.38268343],
             [ 0.38268343,  0.92387953]]))

# Polynomials

```python
P = M.charpoly(x)
P, P.factor()
```

# Draft space

```python
rho = Matrix([
    [Rational('1/2'), Rational('-1/6')],
    [Rational('-1/6'), Rational('1/2')]
])
rho.eigenvals(), rho.eigenvects(), rho.charpoly(X)
```

```python
colors = ["r", "g", "b"]
n = ["red", "green", "blue"]

m = {key: color for key,color in zip(colors, n)}
m
```

    {'r': 'red', 'g': 'green', 'b': 'blue'}


# Extended Kalman filter with Parameter Identification ($T_1$ and $K$) for Nomoto model
An Extended Kalman filter with a Nomoto model as the predictor will be developed.
The filter should also estimate the parameters in the Nomoto model. This is done by setting up a system where the parameters are defined as some of the states in the model
The filter is run on simulated data as well as real model test data.

```python
%load_ext autoreload
%autoreload 2

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv
import sympy as sp

import src.visualization.book_format as book_format
book_format.set_style()
from src.substitute_dynamic_symbols import lambdify
from sympy import Matrix
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify
from sympy.physics.vector.printing import vpprint, vlatex
from src.data import mdl
from src.kalman_filter import extended_kalman_filter
```

## Nomoto model for ship manoeuvring dynamics
The Nomoto model can be written as:

```python
r,r1d,r2d = sp.symbols('r \dot{r} \ddot{r}')
psi,psi1d = sp.symbols('psi \dot{\psi}')
h,u = sp.symbols('h u')
x, x1d = sp.symbols('x \dot{x}')
A,B,C,D,E, Phi = sp.symbols('A B C D E Phi')
w = sp.symbols('w')

K, delta, T_1, T_2 = sp.symbols('K delta T_1 T_2')

eq_nomoto = sp.Eq(K*delta,
                 r + T_1*r1d + T_2*r2d)
Math(vlatex(eq_nomoto))
```

where $r$ is yaw rate with its time derivatives and $\delta$ is the rudder angle. $K$, $T_{1}$
 and $T_{1}$ are the coefficients describing the hydrodynamics of the ship.

For slow manoeuvres this equation can be further simplified by removing the $\ddot{r}$ term into a first order Nomoto model:

```python
eq_nomoto_simple = eq_nomoto.subs(r2d,0)
Math(vlatex(eq_nomoto_simple))
```

### Simulation model

```python
f_hat = sp.Function('\hat{f}')(x,u,w)
eq_system = sp.Eq(x1d, f_hat)
eq_system
```

Where the state vector $x$:

```python
eq_x = sp.Eq(x, sp.UnevaluatedExpr(Matrix([psi,r])))
eq_x
```

and input vector $u$:

and $w$ is zero mean Gausian process noise

For the nomoto model the time derivatives for the states can be expressed as:

```python
eq_psi1d = sp.Eq(psi1d,r)
eq_psi1d
```

```python
eq_r1d = sp.Eq(r1d,sp.solve(eq_nomoto_simple,r1d)[0])
eq_r1d
```

```python
def lambda_f_constructor(K, T_1):
    def lambda_f(x, u):
        delta = u
        f = np.array([[x[1], (K*delta-x[1])/T_1]]).T
        return f

    return lambda_f
```

## Simulation
Simulation with this model where rudder angle shifting between port and starboard

```python
T_1_ = 1.8962353076056344
K_ = 0.17950970687951323
h_ = 0.01

lambda_f = lambda_f_constructor(K=K_, T_1=T_1_)
```

```python
def simulate(E, ws, t, us):

    simdata = []
    x_=np.deg2rad(np.array([[0,0]]).T)

    for u_,w_ in zip(us,ws):

        x_=x_ + h_*lambda_f(x=x_.flatten(), u=u_)

        simdata.append(x_.flatten())

    simdata = np.array(simdata)
    df = pd.DataFrame(simdata, columns=["psi","r"], index=t)
    df['delta'] = us

    return df
```

```python
N_ = 8000

t_ = np.arange(0,N_*h_,h_)

us = np.deg2rad(np.concatenate((-10*np.ones(int(N_/4)),
                                10*np.ones(int(N_/4)),
                                -10*np.ones(int(N_/4)),
                                10*np.ones(int(N_/4)))))

np.random.seed(42)
E = np.array([[0, 1]]).T
process_noise = np.deg2rad(0.01)
ws = process_noise*np.random.normal(size=N_)
df = simulate(E=E, ws=ws, t=t_, us=us)

measurement_noise = np.deg2rad(0.5)
df['epsilon'] = measurement_noise*np.random.normal(size=N_)
df['psi_measure'] = df['psi'] + df['epsilon']
df['psi_deg'] = np.rad2deg(df['psi'])
df['psi_measure_deg'] = np.rad2deg(df['psi_measure'])
df['delta_deg'] = np.rad2deg(df['delta'])

```

```python
fig,ax=plt.subplots()
df.plot(y='psi_deg', ax=ax)
df.plot(y='psi_measure_deg', ax=ax, zorder=-1)
df.plot(y='delta_deg', ax=ax, zorder=-1)
df.plot(y='r')
ax.set_title('Simulation with measurement and process noise')
ax.set_xlabel('Time [s]');

```

## Kalman filter
Implementation of the Kalman filter. The code is inspired of this Matlab implementation: [ExEKF.m](https://github.com/cybergalactic/MSS/blob/master/mssExamples/ExEKF.m).

```python
jac = sp.eye(4,4) + Matrix([r,eq_r1d.rhs,0,0]).jacobian([psi,r,T_1,K])*h
jac
```

```python
def lambda_f_constructor2():
    def lambda_f(x, u):
        delta = u
        T_1 = x[2]  # Note! T_1 is the third state now!
        K = x[3]  # Note! K is the fourth state!
        r = x[1]

        f = np.array([[r, (K*delta-r)/T_1, 0, 0]]).T
        return f

    return lambda_f
```

```python
def lambda_jacobian_constructor(h):
    def lambda_jacobian(x, u):

        T_1 = x[2]  # Note! T_1 is the third state now!
        K = x[3]  # Note! K is the fourth state!
        delta = u
        r = x[1]

        jac = np.array(
            [
                [1, h, 0, 0],
                [0, 1 - h / T_1, -h * (K * delta - r) / T_1 ** 2, delta*h/T_1],
                [0, 0, 1, 0],
                [0, 0, 0, 1]
            ]
        )
        return jac

    return lambda_jacobian
```

```python
lambda_jacobian = lambda_jacobian_constructor(h=h_)
lambda_f = lambda_f_constructor2()
```

```python
lambda_jacobian(x=[0,0,0.1,0.1], u=0)
```

```python
lambda_f(x=[0,0,0.1, 0.1], u=0)
```

```python
def plot(df,time_steps):

    x_hats = np.array([time_step["x_hat"] for time_step in time_steps]).T
    time = np.array([time_step["time"] for time_step in time_steps]).T
    Ks = np.array([time_step["K"] for time_step in time_steps]).T
    stds = np.sqrt(np.array([[time_step["P_hat"][0,0],
                          time_step["P_hat"][1,1],
                          time_step["P_hat"][2,2],
                          time_step["P_hat"][3,3],
                         ] for time_step in time_steps]).T)

    n=len(P_prd)
    fig,axes=plt.subplots(nrows=n)

    df['T_1'] = T_1_
    df['K'] = K_
    keys = ['psi','r','T_1','K']
    labels = {
        'psi' : '\Psi',
    }
    for i,key in enumerate(keys):

        ax=axes[i]

        if key=='psi':
            df.plot(y='psi_measure', ax=ax, label="Measured", zorder=-1)
        else:
            ax.plot([])

        df.plot(y=key, ax=ax, label="True")

        ax.plot(time, x_hats[i, :], "-", label="kalman")

        ylims = np.array(ax.get_ylim())

        std_top = x_hats[i, :] + stds[i, :]
        std_btm = x_hats[i, :] - stds[i, :]
        ax.plot(time, std_top, linestyle=':', color='k', lw=1, alpha=0.4)
        ax.plot(time, std_btm, linestyle=':', color='k', lw=1, alpha=0.4)
        ax.fill_between(time, std_top, std_btm,
                         facecolor='yellow', alpha=0.2, interpolate=True, label='+/- std')

        if np.all(ylims < 0):
            ax.set_ylim(ylims[0],0)
        elif np.all(ylims > 0):
            ax.set_ylim(0,ylims[1])
        else:
            ax.set_ylim(ylims)

        ylabel=labels.get(key,key)
        ax.set_ylabel(f'${ylabel}$')

        if i<2:
            ax.legend(loc='upper left')
        else:
            ax.legend().set_visible(False)

```

```python
x0=np.deg2rad(np.array([[0,0,3, 0.2]]).T)
P_prd = np.diag([np.deg2rad(1), np.deg2rad(0.1), 0.1, 0.5])

Qd = np.diag([np.deg2rad(0), 2, 1])

Rd = np.deg2rad(0.1)

ys = df['psi_measure'].values

E_ = np.array(
    [
     [0,0,0],
     [1,0,0],
     [0,1,0],
     [0,0,1],

    ],
)

C_ = np.array([[1, 0, 0, 0]])

Cd_ = C_
Ed_ = h_ * E_

time_steps = extended_kalman_filter(x0=x0, P_prd=P_prd, lambda_f=lambda_f,
                                    lambda_jacobian=lambda_jacobian,h=h_, us=us, ys=ys, E=E_, Qd=Qd, Rd=Rd, Cd=Cd_)

x_hats = np.array([time_step["x_hat"] for time_step in time_steps]).T
time = np.array([time_step["time"] for time_step in time_steps]).T
Ks = np.array([time_step["K"] for time_step in time_steps]).T
stds = np.sqrt(np.array([[time_step["P_hat"][0,0],
                      time_step["P_hat"][1,1],
                      time_step["P_hat"][2,2],
                      time_step["P_hat"][3,3],
                     ] for time_step in time_steps]).T)

```

```python
plot(df=df, time_steps=time_steps)
```

```python
fig,axes=plt.subplots(nrows=2)
for i, key in enumerate(['T_1','K']):
    ax = axes[i]
    ax.set_ylabel(key)
    ax.set_xlabel('Time [s]')
    ax.plot(time,x_hats[2+i,:],label='Kalman')
    df.plot(y=key, ax=ax, label='True')
    ax.legend()

```

```python
fig,ax=plt.subplots()
keys = ['psi','r','T_1','K']
for i,key in enumerate(keys):
    ax.plot(time,Ks[i,:],label=key)
ax.set_title('Kalman gains')
ax.legend();
ax.set_ylim(-1,1);

```

```python
fig,axes=plt.subplots(nrows=2)

ax=axes[0]
for i,key in enumerate(keys):
    ax.plot(time,stds[i,:]**2,label=key)
ax.set_title('Variances')
ax.legend();

ax=axes[1]
df.plot(y='delta',ax=ax)

```

```python
x0=np.deg2rad(np.array([[0,0,3, 0.2]]).T)
P_prd = np.diag([np.deg2rad(1), np.deg2rad(0.1), 0.1, 0.1])

Qd = np.diag([np.deg2rad(0), 2, 1])

Rd = np.deg2rad(0.1)

ys = df['psi_measure'].values

E_ = np.array(
    [
     [0,0,0],
     [1,0,0],
     [0,1,0],
     [0,0,1],

    ],
)

C_ = np.array([[1, 0, 0, 0]])

Cd_ = C_
Ed_ = h_ * E_

for i in range(5):
    time_steps = extended_kalman_filter(x0=x0, P_prd=P_prd, lambda_f=lambda_f,
                                        lambda_jacobian=lambda_jacobian,h=h_, us=us, ys=ys, E=E_, Qd=Qd, Rd=Rd, Cd=Cd_)

    time_step = time_steps[-1]
    x0[2:]=np.array([time_step['x_hat'][2:]]).T
    P_prd = time_step['P_hat']
```

```python
plot(df=df, time_steps=time_steps)
```


# Item 04

A *fix-point-iteration review*. See Numerical Analysis, 2nd edition by Timothy Sauer.

* Definition 1: The real number $r$ is a fix-point of the function $g(x)$ if $g(r)=r$.
* Algorithm 1: Fixed-Point-Iteration: Let $x_0$ be the initial guess. Compute $x_{i+1} = g(x_i)$, for $i=0,1,2,3,\dots$. Notice that this fixed-point-iteration may or may not converge to $r$.
* Definition 2: Let $e=|r-x_i|$ be the error at iteration $i$. If $0 < \lim_{i \rightarrow \infty} \frac{e_{i+1}}{e_i} = S < 1$, the fixed-point-iteration $x_{i+1} = g(x_i)$ is said to obey linear convergence with rate $S$.
* Theorem 1: Assume that $g(x)$ is continuously differentiable and that $S=|g'(x)|<1$. Then the fixed-point-iteration $x_{i+1}=g(x_i)$ converges at least linearly with rate $S$ to the fixed point $r$ for the initial guesses sufficiently close to $r$.

---

* **Question 1**: Prove that a continuously differentiable function $g(x)$ satisfying $|g'(x)|<1$ on a closed interval cannot have two fixed points on that interval.

__Dem__:

By contradiction, let's asume that $g(x)$ has two fixed points, $x_1$ and $x_2$, with $x_1 \neq x_2$ on the interval $[a,b]$, and that $|g'(x)|< 1 \, \forall (x \in [a,b])$, then we have that:
$$
g(x_1) = x_1 \qquad g(x_2) = x_2 \,.
$$

Given, this two points, by the [mean value theorem](https://en.wikipedia.org/wiki/Mean_value_theorem), there should exist a point $c \in [x_1,x_2]$ so that:
$$
g'(c) = \frac{g(x_1)-g(x_2)}{x_1-x_2} = \frac{x_1-x_2}{x_1-x_2} = 1
$$
as $c \in [x_1,x_2] \Rightarrow c \in [a,b]$, we have a contradiction, since we said that $|g'(x)|<1 \, \forall (x \in [a,b])$.

---

* **Question 2**: Given that $f(x)$ has a root near $x_0$. Derive three different fix-point-iterations that may converge to $f(r)=0$ and state the restrictions of $f(x)$ needed, if any. Assume that $f(x)$ has as many derivatives as you may need.

(1) We have the trivial options (as $x \rightarrow r$):
\begin{align}
0 &= f(x)
\\ x &= \underbrace{\pm f(x) + x}_{g(x)}
\end{align}
it requires that $|g'(r)| = |\pm f'(r) + 1| \leq 1$, we may choose the sign of $\pm$.

(2) We expand the Taylor series arround $x_0$:
\begin{align}
f(x) &= f(x_0) + f'(x_0)(x-x_0) + \dots
\\  0 &= f(x_0) + f'(x_0)(x-x_0)  \qquad \qquad \text{as $x\rightarrow r$}
\\ f'(x_0)x &= -f(x_0) + f'(x_0) x_0
\\ x &= -\frac{f(x_0)}{f'(x_0)} + x_0
\\ x &= \underbrace{x_0 - \frac{f(x_0)}{f'(x_0)}}_{g(x_0)}
\end{align}
Which corresponds to the Newton's method. It doens't have restrictions.

(2) We expand the Taylor series arround $x_0$:
\begin{align}
f(x) &= f(x_0) + f'(x_0)(x-x_0) + \frac{1}{2}f''(x_0)(x-x_0)^2
\\ 0 &= f(x_0) + f'(x_0)(x-x_0) + \frac{1}{2}f''(x_0)(x-x_0)^2 &\text{as $x\rightarrow r$}
\\ 0 &= f'(x_0) + f''(x_0)x - f''(x_0)x_0 &\text{after $\frac{d(\cdot)}{dx}$}
\\ f''(x_0)x &= f''(x_0)x_0 - f'(x_0)
\\ x &= \underbrace{x_0 - \frac{f'(x_0)}{f''(x_0)}}_{g(x_0)}
\end{align}
Which is the [Newton's method in optimization](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization).

It requires $f''(r)$ to exist, and if we make $|g'(r)| <1$ that results in:
$$
|f'(r) f'''(r)| < |f''(r)|
$$

---

* **Question 3**: Derive a unsuccessful and a successful fix-point-iteration for finding the root of $x^3+x=1$ near $x_0=1$.

Our unsuccessful fix-point-iteration would be:
\begin{align}
x^3+x &= 1
\\ x &= 1-x^3
\\ \text{we make } g(x) &= 1-x^3
\end{align}
it won't converge as $g'(x) = -3x^2$ and $g'(1) = -3$.

Our successful one:
\begin{align}
x^3+x &= 1
\\ x(x^2+1) &= 1
\\ x^2+1 &= \frac{1}{x}
\\ x^2 &= \frac{1}{x}-1
\\ x &= \sqrt{\frac{1}{x}-1}
\\ \text{we make } g(x) &= \sqrt{\frac{1}{x}-1}
\end{align}
it will converge as $g'(x) = -\frac{1}{2} \left( x^{-1}-1 \right)^{-\frac{1}{2}}x^{-2}$ and $g'(1) = 0$.

---

* Algorithm 2: Newton's method: Let $x_0$ be the initial guess. Compute $x_{i+1} = \hat{g}(x_i)$, for $i=0,1,2,3,\dots$, where $\hat{g}=x-(f'(x))^{-1}f(x)$ and $(f'(x))^{-1}$ is the inverse of $f'(x)$, i.e. $(f'(x))^{-1} = 1/f'(x)$.
* Definition 3: Let $e = |r-x_i|$ be the error at iteration $i$. If $\lim_{i \rightarrow \infty} \frac{e_{i+1}}{e_i^2} = M < \infty$, the method is said to be quadratically convergent.

---

* **Question 4**: Prove that Newton's method is quadratically convergent as long as $f'(r) \neq 0$.

**Dem**:

We have that $g(x) = x - \frac{f(x)}{f'(x)}$. Expanding a Taylor's series around $r$:
\begin{align}
g(x_i) &= g(r) + g'(r)(x_i-r) + \frac{1}{2}g''(r)(x_i-r)^2 + \dots
\\ x_{i+1} &= r + g'(r)(x_i-r) + \frac{1}{2}g''(r)(x_i-r)^2 + \dots
\end{align}

We can see that:
\begin{align}
g'(x) &= 1 - \frac{f'(x)f'(x)-f(x)f''(x)}{(f'(x))^2}
\\ g'(r) &= 1 - \frac{f'(r)f'(r)-f(r)f''(r)}{(f'(r))^2}
\\ &= 1 - \frac{f'(r)f'(r)}{(f'(r))^2} = 1-1 = 0
\end{align}

So, retaking the previous equation:
\begin{align}
g(x_i) &= g(r) + g'(r)(x_i-r) + \frac{1}{2}g''(r)(x_i-r)^2 + \dots
\\ x_{i+1} &= r \frac{1}{2}g''(r)(x_i-r)^2 + \dots
\\ x_{i+1}-r &= \frac{1}{2}g''(r)(x_i-r)^2 + \dots
\\ \frac{x_{i+1}-r}{(x_i-r)^2} &= \frac{1}{2}g''(r) + \dots
\\ \frac{e_{i+1}}{e_i^2} &= \frac{1}{2}g''(r) + \dots
\end{align}

And the method has quadratic convergence at rate $M = \frac{1}{2}g''(r)$.

---

* **Question 5**: Explain when Newton's method show linear convergence and also explain how it can be fixed.

When $f'(r)=0$ we have that:
\begin{align}
g'(x) &= 1 - \frac{f'(x)f'(x)-f(x)f''(x)}{(f'(x))^2}
\\ &= 1-\frac{{(f'(x))^2}}{{(f'(x))^2}}+\frac{f(x)f''(x)}{(f'(x))^2}
\\ &= 1-1+\frac{f(x)f''(x)}{(f'(x))^2}
\\ &= \frac{f(x)f''(x)}{(f'(x))^2}
\end{align}
using L'Hopital (because $f(r)=0$ too).
\begin{align}
g(r) &= \frac{f'(r)f''(r)+f(r)f'''(r)}{2f'(r)f''(r)}
\\ &= \frac{1}{2}+\frac{f(r)f'''(r)}{2f'(r)f''(r)}
\\ &= \frac{1}{2}+\frac{f(r)f'''(r)}{2f'(r)f''(r)}
\\ &= \frac{1}{2}+\frac{f'(r)f'''(r)+f(r)f^{(4)}(r)}{2f''(r)f''(r)+2f'(r)f'''(r)}
\\ &= \frac{1}{2}+' \qquad \text{as long as $f''(x)\neq 0$}
\end{align}
we see that the method has linear convergence because $|g'(r)|=\frac{1}{2} \neq 0$.

If we make our $g(x)= x - \alpha \frac{f'(x)}{f''(x)}$, we will see that the derivate will be:
\begin{align}
g'(x) &= 1 - \alpha\frac{f'(x)f'(x)-f(x)f''(x)}{(f'(x))^2}
\\g'(r) &= 1 - \alpha \left(1-\frac{1}{2} \right)
\end{align}
and we just have to make $\alpha = 2$ so that $g'(x)=0$. In general, we have to make $a$ equal to the multiplicity of the root

```python

```