### Quadratic Interpolation

Given a smooth, well-behaved function $f(t)$ that is known to have no more than one root within the range $t_a \le t \le t_b$, we want to find an iterative procedure to quickly narrow in on a value $t_0$ in that range such that $f(t_0) \approx 0$. The function $f$ may be expensive to compute, so we would like to evaluate it as few times as possible. We will use quadratic interpolation, where we approximate $f$ as a parabola.

First find the midpoint time $t_m$ between $t_a$ and $t_b$:

\begin{align}
\\
t_m = \frac {t_b + t_a} {2}
\end{align}

Define a parameter $x$ that has the linear range $-1 \le x \le +1$ over the search interval:

\begin{align}
\\
x = \frac {t - t_m} {t_b - t_m}
\end{align}

A generic parabolic function over this range is

\begin{align}
\\
p(x) = Q x^2 + R x + S
\end{align}

Evaluate $f$ at the left, middle, and right points. These correspond to $x=-1$, $x=0$, and $x=+1$.

\begin{align}
\\
f_a &= f(t_a) \\
f_m &= f(t_m) \\
f_b &= f(t_b)
\end{align}

Now we can relate these 3 values to a unique parabola that passes exactly through the points $(t_a, f_a)$, $(t_m, f_m)$, and $(t_b, f_b)$ as

\begin{align}
\\
p(-1) &= f_a = Q - R + S \\
p(0)  &= f_m = S\\
p(+1) &= f_b = Q + R + S
\end{align}

Solving this system of equations in terms of the known values $f_a$, $f_m$, and $f_b$, we obtain

\begin{align}
\\
Q &= \frac{f_b + f_a}{2} - f_m \\
R &= \frac{f_b - f_a}{2} \\
S &= f_m
\end{align}

Let's test these formulas:

```python
import math

class Parabola:
    def __init__(self, f, ta, tb):
        tm = (tb + ta)/2
        fa = f(ta)
        fb = f(tb)
        fm = f(tm)
        self.Q = (fb + fa)/2 - fm
        self.R = (fb - fa)/2
        self.S = fm

    def Eval(self, x):
        return self.Q*x*x + self.R*x + self.S

    def Slope(self, x):
        return 2*self.Q*x + self.R

def Func(t):
    return math.cos(t) - t

ta = 0.6
tb = 0.8
tm = (ta + tb)/2
p = Parabola(Func, ta, tb)
print('Q={:0.6f}, R={:0.6f}, S={:0.6f}'.format(p.Q, p.R, p.S))
print('p(-1)={:0.6f}, f(ta)={:0.6f}'.format(p.Eval(-1), Func(ta)))
print('p( 0)={:0.6f}, f(tm)={:0.6f}'.format(p.Eval( 0), Func(tm)))
print('p(+1)={:0.6f}, f(tb)={:0.6f}'.format(p.Eval(+1), Func(tb)))

```

    Q=-0.003821, R=-0.164314, S=0.064842
    p(-1)=0.225336, f(ta)=0.225336
    p( 0)=0.064842, f(tm)=0.064842
    p(+1)=-0.103293, f(tb)=-0.103293

Now that we have a parabola that passes through the 3 points, let's solve for the value of $x$ such that $p(x)=0$. The quadratic formula gives us two values of $x$:

\begin{align}
\\
x = \frac {-R \pm \sqrt{R^2 - 4QS}} {2Q}
\end{align}

For our purposes, the solution must have a real value and must lie in the range $-1 \le x \le +1$. We expect exactly one such solution. If there are zero or two such values of $x$, our search will return a null value.

```python
def Solve(p):
    if p.Q == 0:
        # This is not a parabola but a straight line
        # p(x) = Rx + S = 0
        # Therefore x = -S/R
        if p.R == 0:
            return None
        x = -p.S / p.R
        if -1 <= x <= +1:
            return x
        return None

    # Let u = the quantity inside the square root.
    u = p.R*p.R - 4*p.Q*p.S
    if u <= 0:
        # If u<0, then both solutions are complex-valued.
        # If u=0, the parabola is tangent to the axis, not crossing it.
        return None

    ru = math.sqrt(u)
    x1 = (-p.R + ru) / (2 * p.Q)
    x2 = (-p.R - ru) / (2 * p.Q)

    if -1 <= x1 <= +1:
        if -1 <= x2 <= +1:
            return None
        return x1

    if -1 <= x2 <= +1:
        return x2

    return None
```

```python
x = Solve(p)
print(x)
```

    0.39106619198014525

Convert the value $x$ back into a time value $t$.

```python
t = tm + x*(tb-tm)
print(t)
```

    0.7391066191980145

See how close this is to the correct solution.

```python
f = Func(t)
print(f)
```

    -3.595936996025895e-05

We want to iterate and do another parabolic interpolation, but the problem is we don't know where to choose a new time range $\bar{t}_a \le t \le \bar{t}_b$. We want this new range to be as small as possible while definitely bracketing the place where $f(t)$ crosses the $t$-axis. If we knew where that was, we would have already solved the problem and wouldn't need to do this!

We are actually more interested in finding $t_0$ within a sufficiently small window $\Delta t$ that definitely brackets the root, than we are in finding a very small value $\left|{f(t)}\right|$.

We can keep track of the bounding values of $t$ that most closely bracket where the root must be, and iterate with quadratic interpolation around each successive approximation of $t_0$. The trick might be to just create another parabola through the $t$ value we just found by picking a new value of $t_a$ and $t_b$ around it, without worrying about whether the parabola crosses the $t$-axis between the new $t_a$ and $t_b$, so long as it stays inside the original range.

```python

```


# Finding Roots of Equations

## Calculus review

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy as scipy
from scipy.interpolate import interp1d
```

Let's review the theory of optimization for multivariate functions. Recall that in the single-variable case, extreme values (local extrema) occur at points where the first derivative is zero, however, the vanishing of the first derivative is not a sufficient condition for a local max or min.  Generally, we apply the second derivative test to determine whether a candidate point is a max or min (sometimes it fails - if the second derivative either does not exist or is zero).  In the multivariate case, the first and second derivatives are *matrices*.  In the case of a scalar-valued function on $\mathbb{R}^n$, the first derivative is an $n\times 1$ vector called the *gradient* (denoted $\nabla f$). The second derivative is an $n\times n$ matrix called the *Hessian* (denoted $H$)

Just to remind you, the gradient and Hessian are given by:

$$\nabla f(x) = \left(\begin{matrix}\frac{\partial f}{\partial x_1}\\ \vdots \\\frac{\partial f}{\partial x_n}\end{matrix}\right)$$

$$H = \left(\begin{matrix}
  \dfrac{\partial^2 f}{\partial x_1^2} & \dfrac{\partial^2 f}{\partial x_1\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_1\,\partial x_n} \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_2\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_2^2} & \cdots & \dfrac{\partial^2 f}{\partial x_2\,\partial x_n} \\[2.2ex]
  \vdots & \vdots & \ddots & \vdots \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_n\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_n\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_n^2}
\end{matrix}\right)$$

One of the first things to note about the Hessian - it's symmetric. This structure leads to some useful properties in terms of interpreting critical points.

The multivariate analog of the test for a local max or min turns out to be a statement about the gradient and the Hessian matrix.  Specifically, a function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ has a critical point at $x$ if $\nabla f(x) = 0$ (where zero is the zero vector!).  Furthermore, the second derivative test at a critical point is as follows:

* If $H(x)$ is positive-definite ($\iff$ it has all positive eigenvalues), $f$ has a local minimum at $x$
* If $H(x)$ is negative-definite ($\iff$ it has all negative eigenvalues), $f$ has a local maximum at $x$
* If $H(x)$ has both positive and negative eigenvalues, $f$ has a saddle point at $x$.

If you have $m$ equations with $n$ variables, then the $m \times n$ matrix of first partial derivatives is known as the Jacobian $J(x)$. For example, for two equations $f(x, y)$ and $g(x, y)$, we have

$$
J(x) = \begin{bmatrix}
\frac{\delta f}{\delta x} & \frac{\delta f}{\delta y} \\
\frac{\delta g}{\delta x} & \frac{\delta g}{\delta y}
\end{bmatrix}
$$

We can now express the multivariate form of Taylor polynomials in a familiar format.

$$
f(x + \delta x) = f(x) + \delta x \cdot J(x) + \frac{1}{2} \delta x^T H(x) \delta x + \mathcal{O}(\delta x^3)
$$

## Main Issues in Root Finding in One Dimension

* Separating close roots
* Numerical Stability
* Rate of Convergence
* Continuity and Differentiability

## Bisection Method

The bisection method is one of the simplest methods for finding zeros of a non-linear function.  It is guaranteed to find a root - but it can be slow. The main idea comes from the intermediate value theorem:  If $f(a)$ and $f(b)$ have different signs and $f$ is continuous, then $f$ must have a zero between $a$ and $b$.  We evaluate the function at the midpoint, $c = \frac12(a+b)$. $f(c)$ is either zero, has the same sign as $f(a)$ or the same sign as $f(b)$.  Suppose $f(c)$ has the same sign as $f(a)$ (as pictured below).  We then repeat the process on the interval $[c,b]$.

```python
def f(x):
    return x**3 + 4*x**2 -3

x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

a = -3.0
b = -0.5
c = 0.5*(a+b)

plt.text(a,-1,"a")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([a,b,c], [f(a), f(b),f(c)], s=50, facecolors='none')
plt.scatter([a,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

```python
x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

d = 0.5*(b+c)

plt.text(d,-1,"d")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([d,b,c], [f(d), f(b),f(c)], s=50, facecolors='none')
plt.scatter([d,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

We can terminate the process whenever the function evaluated at the new midpoint is 'close enough' to zero. This method is an example of what are known as 'bracketed methods'. This means the root is 'bracketed' by the end-points (it is somewhere in between). Another class of methods are 'open methods' - the root need not be somewhere in between the end-points (but it usually needs to be close!)

## Secant Method

The secant method also begins with two initial points, but without the constraint that the function values are of opposite signs.  We use the secant line to extrapolate the next candidate point.

```python
def f(x):
    return (x**3-2*x+7)/(x**4+2)

x = np.arange(-3,5, 0.1);
y = f(x)

p1=plt.plot(x, y)
plt.xlim(-3, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
t = np.arange(-10, 5., 0.1)

x0=-1.2
x1=-0.5
xvals = []
xvals.append(x0)
xvals.append(x1)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--']
while (notconverge==1 and count <  3):
    slope=(f(xvals[count+1])-f(xvals[count]))/(xvals[count+1]-xvals[count])
    intercept=-slope*xvals[count+1]+f(xvals[count+1])
    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(f(nextval)) < 0.001:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

plt.show()
```

The secant method has the advantage of fast convergence.  While the bisection method has a linear convergence rate (i.e. error goes to zero at the rate that $h(x) = x$ goes to zero, the secant method has a convergence rate that is faster than linear, but not quite quadratic (i.e. $\sim x^\alpha$, where $\alpha = \frac{1+\sqrt{5}}2 \approx 1.6$) however, the trade-off is that the secant method is not guaranteed to find a root in the brackets.

A variant of the secant method is known as the **method of false positions**. Conceptually it is identical to the secant method, except that instead of always using the last two values of $x$ for linear interpolation, it chooses the two most recent values that maintain the bracket property (i.e $f(a) f(b) < 0$). It is slower than the secant, but like the bisection, is safe.

## Newton-Raphson Method

We want to find the value $\theta$ so that some (differentiable) function $g(\theta)=0$.
Idea: start with a guess, $\theta_0$.  Let $\tilde{\theta}$ denote the value of $\theta$ for which $g(\theta) = 0$ and define $h = \tilde{\theta} - \theta_0$.  Then:

$$
\begin{eqnarray*}
g(\tilde{\theta}) &=& 0 \\\\
&=&g(\theta_0 + h) \\\\
&\approx& g(\theta_0) + hg'(\theta_0)
\end{eqnarray*}
$$

This implies that

$$ h\approx \frac{g(\theta_0)}{g'(\theta_0)}$$

So that

$$\tilde{\theta}\approx \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

Thus, we set our next approximation:

$$\theta_1 = \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

and we have developed an iterative procedure with:

$$\theta_n = \theta_{n-1} - \frac{g(\theta_{n-1})}{g'(\theta_{n-1})}$$

#### Example

Let $$g(x) = \frac{x^3-2x+7}{x^4+2}$$

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Example Function')
plt.show()
```

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Good Guess')
t = np.arange(-5, 5., 0.1)

x0=-1.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

```

From the graph, we see the zero is near -2.  We make an initial guess of $$x=-1.5$$

We have made an excellent choice for our first guess, and we can see rapid convergence!

```python
funval
```

In fact, the Newton-Raphson method converges quadratically.  However, NR (and the secant method) have a fatal flaw:

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Bad Guess')
t = np.arange(-5, 5., 0.1)

x0=-0.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge = 0
    else:
        xvals.append(nextval)
    count = count+1
```

We have stumbled on the horizontal asymptote.  The algorithm fails to converge.

### Convergence Rate

The following is a derivation of the convergence rate of the NR method:

Suppose $x_k \; \rightarrow \; x^*$ and $g'(x^*) \neq 0$.  Then we may write:

$$x_k = x^* + \epsilon_k$$.

Now expand $g$ at $x^*$:

$$g(x_k) = g(x^*) + g'(x^*)\epsilon_k + \frac12 g''(x^*)\epsilon_k^2 + ...$$
$$g'(x_k)=g'(x^*) + g''(x^*)\epsilon_k$$

We have that

\begin{eqnarray}
\epsilon_{k+1} &=& \epsilon_k + \left(x_{k-1}-x_k\right)\\
&=& \epsilon_k -\frac{g(x_k)}{g'(x_k)}\\
&\approx & \frac{g'(x^*)\epsilon_k + \frac12g''(x^*)\epsilon_k^2}{g'(x^*)+g''(x^*)\epsilon_k}\\
&\approx & \frac{g''(x^*)}{2g'(x^*)}\epsilon_k^2
\end{eqnarray}

## Gauss-Newton

For 1D, the Newton method is
$$
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
$$

We can generalize to $k$ dimensions by
$$
x_{n+1} = x_n - J^{-1} f(x_n)
$$
where $x$ and $f(x)$ are now vectors, and $J^{-1}$ is the inverse Jacobian matrix. In general, the Jacobian is not a square matrix, and we use the generalized inverse $(J^TJ)^{-1}J^T$ instead, giving
$$
x_{n+1} = x_n - (J^TJ)^{-1}J^T f(x_n)
$$

In multivariate nonlinear estimation problems, we can find the vector of parameters $\beta$ by minimizing the residuals $r(\beta)$,
$$
\beta_{n+1} = \beta_n - (J^TJ)^{-1}J^T r(\beta_n)
$$
where the entries of the Jacobian matrix $J$ are
$$
J_{ij} = \frac{\partial r_i(\beta)}{\partial \beta_j}
$$

## Inverse Quadratic Interpolation

Inverse quadratic interpolation is a type of polynomial interpolation.  Polynomial interpolation simply means we find the polynomial of least degree that fits a set of points.  In quadratic interpolation, we use three points, and find the quadratic polynomial that passes through those three points.

```python

def f(x):
    return (x - 2) * x * (x + 2)**2

x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,-1.0])
y0=f(x0)
f2 = interp1d(x0, y0,kind='quadratic')

#Plot parabola
xs = np.linspace(-3, -1, num=10000, endpoint=True)
plt.plot(xs, f2(xs))

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');

#New x value
xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]

plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='black');

#New triplet
x1=np.append([-3,-2.5],xnew)
y1=f(x1)
f2 = interp1d(x1, y1,kind='quadratic')

#New Parabola
xs = np.linspace(min(x1), max(x1), num=100, endpoint=True)
plt.plot(xs, f2(xs))

xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]
plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='green');

```

So that's the idea behind quadratic interpolation. Use a quadratic approximation, find the zero of interest, use that as a new point for the next quadratic approximation.

Inverse quadratic interpolation means we do quadratic interpolation on the *inverse function*.  So, if we are looking for a root of $f$, we approximate $f^{-1}(x)$ using quadratic interpolation. This just means fitting $x$ as a function of $y$, so that the quadratic is turned on its side and we are guaranteed that it cuts the x-axis somewhere. Note that the secant method can be viewed as a *linear* interpolation on the inverse of $f$. We can write:

$$f^{-1}(y) = \frac{(y-f(x_n))(y-f(x_{n-1}))}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{(y-f(x_n))(y-f(x_{n-2}))}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{(y-f(x_{n-2}))(y-f(x_{n-1}))}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n-1}$$

We use the above formula to find the next guess $x_{n+1}$ for a zero of $f$ (so $y=0$):

$$x_{n+1} = \frac{f(x_n)f(x_{n-1})}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{f(x_n)f(x_{n-2})}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{f(x_{n-2})f(x_{n-1})}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n}$$

We aren't so much interested in deriving this as we are understanding the procedure:

```python
x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Inverse Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,1])
y0=f(x0)
f2 = interp1d(y0, x0,kind='quadratic')

#Plot parabola
xs = np.linspace(min(f(x0)), max(f(x0)), num=10000, endpoint=True)
plt.plot(f2(xs), xs)

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');
```

Convergence rate is approximately $1.8$. The advantage of the inverse method is that we will *always* have a real root (the parabola will always cross the x-axis). A serious disadvantage is that the initial points must be very close to the root or the method may not converge.

That is why it is usually used in conjunction with other methods.

## Brentq Method

Brent's method is a combination of bisection, secant and inverse quadratic interpolation.  Like bisection, it is a 'bracketed' method (starts with points $(a,b)$ such that $f(a)f(b)<0$.

Roughly speaking, the method begins by using the secant method to obtain a third point $c$, then uses inverse quadratic interpolation to generate the next possible root. Without going into too much detail, the algorithm attempts to assess when interpolation will go awry, and if so, performs a bisection step. Also, it has certain criteria to reject an iterate. If that happens, the next step will be linear interpolation (secant method).

To find zeros, use

```python
x = np.arange(-5,5, 0.1);
p1=plt.plot(x, f(x))
plt.xlim(-4, 4)
plt.ylim(-10, 20)
plt.xlabel('x')
plt.axhline(0)
pass
```

```python
from scipy import optimize
```

```python
scipy.optimize.brentq(f,-1,.5)
```

```python
scipy.optimize.brentq(f,.5,3)
```

## Roots of polynomials

One method for finding roots of polynomials converts the problem into an eigenvalue one by using the **companion matrix** of a polynomial. For a polynomial

$$
p(x) = a_0 + a_1x + a_2 x^2 + \ldots + a_m x^m
$$

the companion matrix is

$$
A = \begin{bmatrix}
-a_{m-1}/a_m & -a_{m-2}/a_m & \ldots & -a_0/a_m \\
1 & 0 & \ldots & 0 \\
0 & 1 & \ldots & 0 \\
\vdots & \vdots & \ldots & \vdots \\
0 & 0 & \ldots & 0
\end{bmatrix}
$$

The characteristic polynomial of the companion matrix is $\lvert \lambda I - A \rvert$ which expands to

$$
a_0 + a_1 \lambda + a_2 \lambda^2 + \ldots + a_m \lambda^m
$$

In other words, the roots we are seeking are the eigenvalues of the companion matrix.

For example, to find the cube roots of unity, we solve $x^3 - 1 = 0$. The `roots` function uses the companion matrix method to find roots of polynomials.

```python
# Coefficients of $x^3, x^2, x^1, x^0$

poly = np.array([1, 0, 0, -1])
```

Manual construction

```python
A = np.array([
    [0,0,1],
    [1,0,0],
    [0,1,0]
])
```

```python
scipy.linalg.eigvals(A)
```

Using built-in function

```python
x = np.roots(poly)
x
```

```python
plt.scatter([z.real for z in x], [z.imag for z in x])
theta = np.linspace(0, 2*np.pi, 100)
u = np.cos(theta)
v = np.sin(theta)
plt.plot(u, v, ':')
plt.axis('square')
pass
```

## Using `scipy.optimize`

### Finding roots of univariate equations

```python
def f(x):
    return x**3-3*x+1
```

```python
x = np.linspace(-3,3,100)
plt.axhline(0, c='red')
plt.plot(x, f(x))
pass
```

```python
from scipy.optimize import brentq, newton
```

#### `brentq` is the recommended method

```python
brentq(f, -3, 0), brentq(f, 0, 1), brentq(f, 1,3)
```

#### Secant method

```python
newton(f, -3), newton(f, 0), newton(f, 3)
```

#### Newton-Raphson method

```python
fprime = lambda x: 3*x**2 - 3
newton(f, -3, fprime), newton(f, 0, fprime), newton(f, 3, fprime)
```

### Finding fixed points

Finding the fixed points of a function $g(x) = x$ is the same as finding the roots of $g(x) - x$. However, specialized algorithms also exist - e.g. using `scipy.optimize.fixedpoint`.

```python
from scipy.optimize import fixed_point
```

```python
x = np.linspace(-3,3,100)
plt.plot(x, f(x), color='red')
plt.plot(x, x)
pass
```

```python
fixed_point(f, 0), fixed_point(f, -3), fixed_point(f, 3)
```

### Mutlivariate roots and fixed points

Use `root` to solve polynomial equations. Use `fsolve` for non-polynomial equations.

```python
from scipy.optimize import root, fsolve
```

Suppose we want to solve a sysetm of $m$ equations with $n$ unknowns

\begin{align}
f(x_0, x_1) &= x_1 - 3x_0(x_0+1)(x_0-1) \\
g(x_0, x_1) &= 0.25 x_0^2 + x_1^2 - 1
\end{align}

Note that the equations are non-linear and there can be multiple solutions. These can be interpreted as fixed points of a system of differential equations.

```python
def f(x):
    return [x[1] - 3*x[0]*(x[0]+1)*(x[0]-1),
            .25*x[0]**2 + x[1]**2 - 1]
```

```python
sol = root(f, (0.5, 0.5))
sol.x
```

```python
fsolve(f, (0.5, 0.5))
```

```python
r0 = root(f,[1,1])
r1 = root(f,[0,1])
r2 = root(f,[-1,1.1])
r3 = root(f,[-1,-1])
r4 = root(f,[2,-0.5])

roots = np.c_[r0.x, r1.x, r2.x, r3.x, r4.x]
```

```python
Y, X = np.mgrid[-3:3:100j, -3:3:100j]
U = Y - 3*X*(X + 1)*(X-1)
V = .25*X**2 + Y**2 - 1

plt.streamplot(X, Y, U, V, color=U, linewidth=2, cmap=plt.cm.autumn)
plt.scatter(roots[0], roots[1], s=50, c='none', edgecolors='k', linewidth=2)
pass
```

#### We can also give the Jacobian

```python
def jac(x):
    return [[-6*x[0], 1], [0.5*x[0], 2*x[1]]]
```

```python
sol = root(f, (0.5, 0.5), jac=jac)
sol.x, sol.fun
```

#### Check that values found are really roots

```python
np.allclose(f(sol.x), 0)
```

#### Starting from other initial conditions, different roots may be found

```python
sol = root(f, (12,12))
sol.x
```

```python
np.allclose(f(sol.x), 0)
```


# Calculate Jensen-Shannon Divergence Between Luke and John

The KL Divergence between two discrete distributions $P$ and $Q$ with pdfs $p$ and $q$, defined over the same sample space $X=\{x_0,x_1, \dots, x_N\}$, is given by

\begin{equation}
KL(P||Q) = \sum_{i=0}^N p(x_i) \ln \Big( \frac{p(x_i)}{q(x_i)} \Big)
\end{equation}

This divergence is not a metric because it is not symmetric, i.e. it is often the case that $KL(P||Q) \ne KL(Q||P)$. To address this, we will use the Jensen-Shannon Divergence which is a true metric and is defined as

\begin{equation}
JSD(P||Q) = \frac{1}{2}KL(P||R) + \frac{1}{2}KL(Q||R)
\end{equation}

where $R$ is defined as the average of the two distributions $R=\frac{P+Q}{2}$

---

```python
from PIL import Image
import numpy as np
```

```python
luke = Image.open("/home/nathan/Downloads/Luke_Van_Poppering.jpeg")
luke.thumbnail((300,300)) # Thanks for this, John....
john = Image.open("/home/nathan/Downloads/John_Abascal.jpg")
john.thumbnail((300,300))
```

```python
luke
```

```python
john
```

---
If we assume that our histograms are exact, then we can trivally calculate the Jensen-Shannon Divergence between them by normalizing the histograms and summing up the terms on the RHS of the JSD equation...

```python
def KL(p: np.array, q: np.array):
    val = 0
    for i,pi in enumerate(p):
        if pi == 0:
            continue
        val += pi*np.log2(pi/q[i])
    return val

def JSD(p: np.array, q: np.array):
    r = (p+q)/2
    p = p[r != 0] # If r_i is zero, then it is in neither p nor q and can be ignored
    q = q[r != 0]
    r = r[r != 0]
    val = 0.5*(KL(p,r)+KL(q,r))
    return val

def hist_loss(im1, im2):
    im1_channels = im1.split()
    im2_channels = im2.split()
    loss = []
    for im1_c, im2_c in zip(im1_channels,im2_channels):
        hist1 = np.array(im1_c.histogram())
        hist2 = np.array(im2_c.histogram())
        loss.append(JSD(hist1/hist1.sum(), hist2/hist2.sum()))
    return np.mean(loss)
```

---
It is symmetric...

```python
hist_loss(luke, john) == hist_loss(john, luke)
```

    True

and returns zero when operating on the same image...

```python
hist_loss(john, john)
```

    0.0

```python
hist_loss(luke, luke)
```

    0.0

```python
hist_loss(np.zeros(255),np.ones(255))
```

```python
JSD(np.zeros(255),np.ones(255)/255)
```

    0.49999999999999845


```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from scipy.stats import pearsonr
from scipy.stats import t as tdist
```

# DIY t-test

As a first exercise lets try to mimic a t-test using simulated data. As the t-test is just a theoretic representation of what we are doing here, our empirical result from the simulation should be the very close to the theoretical one. To generate a so called null-distribution for the correlation coefficient we repeatedly draw __uncorrelated__ samples from a normal distribution and correlate them with each other saving the result.

```python
# Total number of samples
nsample = 10000
# Number of observations in each sample
nobs = 100

# Initialization of a variable for all our samples
r_sample= np.zeros(nsample)

# Draw samples and correlate
for i in range(nsample):
    x = np.random.randn(nobs)
    y = np.random.randn(nobs)
    r, p = pearsonr(x, y)
    r_sample[i] = r
```

We can take a look at the samples and how they are distributed using a histogram. Note that they have a mean value of 0. The width of the distribution changes with the number of observations we correlate in each of the samples. Go ahead and give that a try, by changing the value of `nobs` above.

```python
plt.hist(r_sample, histtype='step')
plt.xlabel('$r$')
```

For the t-test we use the value of $r$ and the number of observations to calculate the test statistic, the t-value using the following formula:

\begin{align}
    t = r \frac{\sqrt{n - 2}}{\sqrt{1 - r^2}}
\end{align}

below this formula is implemented in a function and than applied to the sample of $r$'s that we have produced above.

```python
def calc_t(r, n):
    """Calculate t statistic for Pearsons r"""
    t = r * np.sqrt(n - 2) / np.sqrt(1 - r**2)
    return t
```

```python
t_sample = calc_t(r_sample, nobs)
```

To be able to compare the sample with the theoretical distribution of this variable, the t-distribution we need to have a range of t-values that spans the range of our samples:

```python
t_plot = np.linspace(np.min(t_sample), np.max(t_sample), 100)
```

Now we have everything to compare our empirical distribution with the theoretical one:

```python
plt.hist(t_sample, histtype='step', density=True, label='empirical')
plt.plot(t_plot, tdist.pdf(t_plot, nobs - 2), label='theoretical')
plt.legend()
```

They should be a pretty close match as the theory is exactly describing what we have done before, just for infinitly many samples.

Lets now compare the $r$ value of the time series in the slides to both the theoretical and empirical ones. Recall. that $r=0.27$ and we had 99 observations in the correlation.

```python
r_data = 0.27
nobs = 99
```

First lets test the value using the classical t-test:

```python
t_data = calc_t(r_data, nobs)
print('t = %.2f' % t_data)

p_theoretical = 2 * (1 - tdist.cdf(t_data, nobs - 2))

print('Theoretical p-value: %.3f' % p_theoretical)
```

Now lets use the code from above to generate a sample for r and for t that we can test these values against.

```python
# Total number of samples
nsample = 10000
# Number of observations in each sample
# Initialization of a variable for all our samples
r_sample= np.zeros(nsample)
# Draw samples and correlate
for i in range(nsample):
    x = np.random.randn(nobs)
    y = np.random.randn(nobs)
    r, p = pearsonr(x, y)
    r_sample[i] = r

t_sample = calc_t(r_sample, nobs)
```

We can test both the fraction of simulated t-values that are larger than the t-value for the data as well as the r-value directly to obtain an empirical p-value for our data.

```python
np.mean(np.abs(t_sample) >= t_data)
```

```python
np.mean(np.abs(r_sample) >= r_data)
```

Again, these values should be very close to the theoretical one as they are essenitally generated the same way, just with a finite number of samples.

So why was this important, when the result is the same as in the t-test anyway?

The t-test as per the theory is not always applicable, depending on the data or the type of statistic that we are looking at. The same is true for any other statistical test. However, as long as you can generate an empirical null-distribution, you will have something to test against. You will not be limited to the special cases that the usual tests are usefull for! So this a very important tool to have.


# Logistic Regression
[source](https://www.youtube.com/watch?v=hSXFuypLukA)
### Step 1: Function Set
We want to find $P_{w,b}(C1|x)$
if $P_{w,b}(C_1|x) \geq 0.5$, output $C_1$, otherwise, output $C_2$

$P_{w,b}(C_1|x) = \sigma(z)$
$$z = w \cdot x + b = \sum_{i}wi xi + b \\
\sigma(z) = \frac{1}{1+exp(-z)}$$

$\begin{equation}
\large{f_{w,b}(x) = \sigma(\sum_{i}w_i x_i + b)}
\end{equation}$
Output should between 0 and 1

### Step 2: Goodness of a Function
Training Data$
\ x^1 \ x^2 \ x^3 \ ... \ x^N \\
\ C_1 \ C_2 \ C_1 \ ... \ C_1
$
Assume the data is generated based on $f_{w,b}(x) = P_{w,b}(C_1|x)$
Given a set of w and b, what is its probability of generating the data?
$\begin{equation}
\large L(w,b) = f_{w,b}(x^)(1 - f_{w,b}(x^2))f_{w,b}(x^3)...f_{w,b}(x^N)
\end{equation}$

The most likely $w^*$ and $b^*$ is the one with the largest $L(w,b)$.

$$w^*,b^* = arg\ max_{w,b}L(w,b) =
w^*, b^* = arg\min_{w,b}-lnL(w,b)$$
$$
x^1 \qquad x^2 \qquad x^3 \qquad ... \\
\hat{y}^1=1 \quad \hat{y}^2=0 \quad \hat{y}^3=1 ... \\
\hat{y}^n : 1 \text{ for class 1, 0 for class 2}
$$

右側省略w,b

左右兩側相等, 故可寫成下列公式
$$
\large{\sum_{n}-[\hat{y}^n ln f_{w,b}(x^n) + (1-\hat{y}^n) ln (1- f_{w,b}(x^n))]} \\
\text{Cross entropy between two Bernoulli distribution}
$$

### Step 3: Find the best function


# Linear Programming

## Introduction
* Widely used
* Used to represent many practical problems
* Elements
    * A linear objective function
    * Linear (in)equalities

## The standard Form
\begin{align}
\text{minimize}\  & f(x) \\
\text{subject to } & \\
& a_1x &&\geq b_1 \\
& a_2x + c && \geq b_2 \\
& x &&\geq 0
\end{align}

# Gurobi Basics: Linear Model
## Mathematical Model
\begin{align}
\text{minimize}\  & 5x + 4y \\
\text{subject to } & \\
& \ \ x+\ \ y &&\geq \ \ 8 \\
& 2x + \ \ y &&\geq 10 \\
& \ \ x + 4y &&\geq 11 \\
& \ \ x &&\geq \ \ 0 \\
& \ \ \ \ \ \ \ \ \ \ \ y &&\geq \ \ 0
\end{align}

## Graphical representation of the problem

```python
from IPython.display import Image
from IPython.display import display
graphs = ['fr', 'fr_o1', 'fr_o3']
for g in graphs:
    display(Image(filename = g+'.png'))
```

# Code in Python using gurobipy
## Step 1: Importing gurobipy package

```python
from gurobipy import *
```

## Step 2: Create an optimization model
Model constructor. Initially, no variables or constraints.
``` python
Model(name = '')
```

```python
opt_mod = Model(name = "linear program")
```

## Step 3: Add decision variables
Add a decision variable to a model.
``` python

Model.addVar(lb = 0.0,               #(optional) lower bound
             ub = float('inf'),      #(optional) upper bound
             obj = 0.0,              #(optional) objective coefficient
             vtype = GRB.CONTINUOUS, #(optional) variable type
             name = "")              #(optional) name

```

```python
x = opt_mod.addVar(name = 'x', vtype = GRB.CONTINUOUS, lb = 0)
y = opt_mod.addVar(name = 'y', vtype = GRB.CONTINUOUS, lb = 0)
```

## Step 4: Define the objective function
Set the model objective equal to a expression
``` python

Model.setObjective(expr,            #New objective expression
                   sense = None)    #GRB.MINIMIZE for minimization,
                                    #GRB.MAXIMIZE for maximization
```

```python
obj_fn = 5*x + 4*y
opt_mod.setObjective(obj_fn, GRB.MINIMIZE)
```

## Step 5: Add the constraints
Add a constraint to a model.
```python
Model.addConstr(constr,  # constraint object
                name="") # name of the constraint
```

```python
c1 = opt_mod.addConstr(  x +   y >=  8, name = 'c1')
c2 = opt_mod.addConstr(2*x +   y >= 10, name = 'c2')
c3 = opt_mod.addConstr(  x + 4*y >= 11, name = 'c3')
```

## Step 6: Solve the model

``` python
Model.optimize() # optimize the model

Model.write(filename) # write model to a file
```

```python
opt_mod.optimize() # solve the model
opt_mod.write("linear_model.lp") # output the LP file of the model
```

## Step 7: Output the result

```python
print('Objective Function Value: %f' % opt_mod.objVal)
# Get values of the decision variables
for v in opt_mod.getVars():
    print('%s: %g' % (v.varName, v.x))
```


```python
from sympy import *
init_printing(use_latex='mathjax')
```

```python
def get_diff(symbol, expression):
    return diff(expression, symbol)
```

```python
x = symbols('x')
get_diff(x, (x + 2) * (3 * x - 3))
```

$$6 x + 3$$

```python
x = symbols('x')
get_diff(x, exp(x) * ( x ** 2 + 7 * x - 3 - sin(x) ))
```

$$\left(2 x - \cos{\left (x \right )} + 7\right) e^{x} + \left(x^{2} + 7 x - \sin{\left (x \right )} - 3\right) e^{x}$$

```python
f =  (x**2 - 2 * x + 5)
```

```python
g = (sin(x) + cos(x))
```

```python
x = symbols('x')
fp=get_diff(x, f)
```

```python
x = symbols('x')
gp=get_diff(x, g)
```

```python
f * gp + g * fp
```

$$\left(2 x - 2\right) \left(\sin{\left (x \right )} + \cos{\left (x \right )}\right) + \left(- \sin{\left (x \right )} + \cos{\left (x \right )}\right) \left(x^{2} - 2 x + 5\right)$$

```python
x = symbols('x')
fp = get_diff(x, exp(x) * (3 * x - 2 ) * sin(x))
factor(simplify(fp))
```

$$\left(3 x \sin{\left (x \right )} + 3 x \cos{\left (x \right )} + \sin{\left (x \right )} - 2 \cos{\left (x \right )}\right) e^{x}$$

```python
x = symbols('x')
fp = get_diff(x, - 1 / 3 * ( exp(x) - 1) ** 2 + (exp(x) - 1) + 1/5)
simplify(fp)
```

$$\left(- 0.666666666666667 e^{x} + 1.66666666666667\right) e^{x}$$

```python
x = symbols('x')
fp = get_diff(x, sqrt(x ** 3 - 2 * x ) )
fp
```

$$\frac{\frac{3 x^{2}}{2} - 1}{\sqrt{x^{3} - 2 x}}$$

```python
x = symbols('x')
fp = get_diff(x, exp(x ** 3 - 3 ))
fp
```

$$3 x^{2} e^{x^{3} - 3}$$

```python
x = symbols('x')
fp = get_diff(x, sqrt(exp(x + 2)))
simplify(fp)
```

$$\frac{1}{2} \sqrt{e^{x + 2}}$$

```python
x = symbols('x')
fp = get_diff(x, x ** (3/2) + pi * x ** 2 + sqrt(7))
fp
```

$$1.5 x^{0.5} + 2 \pi x$$

```python
x = symbols('x')
fp = get_diff(x, x ** 3 * cos(x) * exp(x))
fp
```

$$- x^{3} e^{x} \sin{\left (x \right )} + x^{3} e^{x} \cos{\left (x \right )} + 3 x^{2} e^{x} \cos{\left (x \right )}$$

```python
x = symbols('x')
fp = get_diff(x, exp((x + 1) ** 2))
fp
```

$$\left(2 x + 2\right) e^{\left(x + 1\right)^{2}}$$

```python
x = symbols('x')
fp = get_diff(x, x ** 2 * cos(x ** 3))
fp
```

$$- 3 x^{4} \sin{\left (x^{3} \right )} + 2 x \cos{\left (x^{3} \right )}$$

```python
x = symbols('x')
fp = get_diff(x, sin(x) * exp(cos(x)))
fp
```

$$- e^{\cos{\left (x \right )}} \sin^{2}{\left (x \right )} + e^{\cos{\left (x \right )}} \cos{\left (x \right )}$$

```python
fp.evalf(subs={x: pi})
```

$$-0.367879441171442$$

```python
nsimplify(6.28, [pi], tolerance=0.01)
```

$$2 \pi$$

```python
(1 / exp(1)).evalf()
```

$$0.367879441171442$$

```python

```

Certification 2/Week1.1 - Differential.ipynb


Linear Regression with Gradient Descent Using a Made Up Example

## Gradient descent algorithm
From our [video](https://youtu.be/fkS3FkVAPWU) on linear regression, we derived the equation to update the linear model parameters as:

\begin{equation}
\theta^{+} = \theta^{-} + \frac{\alpha}{m} (y_{i} - h(x_{i}) )\bar{x}
\end{equation}

This minimizes the following cost function

\begin{equation}
J(x, \theta, y) = \frac{1}{2m}\sum_{i=1}^{m}(h(x_i) - y_i)^2
\end{equation}

where
\begin{equation}
h(x_i) = \theta^T \bar{x}
\end{equation}

### Batch gradient descent
```FOR j FROM 0 -> max_iteration:
    FOR i FROM 0 -> m:
        theta += (alpha / m) * (y[i] - h(x[i])) * x_bar
    ENDLOOP
ENDLOOP
```

### Stochastic gradient descent
```shuffle(x, y)
FOR i FROM 0 -> m:
    theta += (alpha / m) * (y[i] - h(x[i])) * x_bar
ENDLOOP
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
"""Generate data"""
true_slope = 10.889
true_intercept = 3.456
input_var = np.arange(0.0,100.0)
output_var = true_slope * input_var + true_intercept + 500.0 * np.random.rand(len(input_var))
```

```python
%matplotlib notebook
plt.figure()
plt.scatter(input_var, output_var)
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

```python
def compute_cost(input_var, output_var, params):
    "Compute linear regression cost"
    num_samples = len(input_var)
    cost_sum = 0.0
    for x,y in zip(input_var, output_var):
        y_hat = np.dot(params, np.array([1.0, x]))
        cost_sum += (y_hat - y) ** 2

    cost = cost_sum / (num_samples * 2.0)

    return cost
```

```python
def lin_reg_batch_gradient_descent(input_var, output_var, params, alpha, max_iter):
    """Compute the params for linear regression using batch gradient descent"""
    iteration = 0
    num_samples = len(input_var)
    cost = np.zeros(max_iter)
    params_store = np.zeros([2, max_iter])

    while iteration < max_iter:
        cost[iteration] = compute_cost(input_var, output_var, params)
        params_store[:, iteration] = params

        print('--------------------------')
        print(f'iteration: {iteration}')
        print(f'cost: {cost[iteration]}')

        for x,y in zip(input_var, output_var):
            y_hat = np.dot(params, np.array([1.0, x]))
            gradient = np.array([1.0, x]) * (y - y_hat)
            params += alpha * gradient/num_samples

        iteration += 1

    return params, cost, params_store

```

```python
"""Train the model"""
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(input_var, output_var, test_size=0.20)

params_0 = np.array([20.0, 80.0])

alpha_batch = 1e-3
max_iter = 500
params_hat_batch, cost_batch, params_store_batch =\
    lin_reg_batch_gradient_descent(x_train, y_train, params_0, alpha_batch, max_iter)
```

```python
def lin_reg_stoch_gradient_descent(input_var, output_var, params, alpha):
    """Compute the params for linear regression using stochastic gradient descent"""
    num_samples = len(input_var)
    cost = np.zeros(num_samples)
    params_store = np.zeros([2, num_samples])

    i = 0
    for x,y in zip(input_var, output_var):
        cost[i] = compute_cost(input_var, output_var, params)
        params_store[:, i] = params

        print('--------------------------')
        print(f'iteration: {i}')
        print(f'cost: {cost[i]}')

        y_hat = np.dot(params, np.array([1.0, x]))
        gradient = np.array([1.0, x]) * (y - y_hat)
        params += alpha * gradient/num_samples

        i += 1

    return params, cost, params_store
```

```python
alpha = 1e-3
params_0 = np.array([20.0, 80.0])
params_hat, cost, params_store =\
lin_reg_stoch_gradient_descent(x_train, y_train, params_0, alpha)
```

```python
plt.figure()
plt.scatter(x_test, y_test)
plt.plot(x_test, params_hat_batch[0] + params_hat_batch[1]*x_test, 'g', label='batch')
plt.plot(x_test, params_hat[0] + params_hat[1]*x_test, '-r', label='stochastic')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
print(f'batch      T0, T1: {params_hat_batch[0]}, {params_hat_batch[1]}')
print(f'stochastic T0, T1: {params_hat[0]}, {params_hat[1]}')
rms_batch = np.sqrt(np.mean(np.square(params_hat_batch[0] + params_hat_batch[1]*x_test - y_test)))
rms_stochastic = np.sqrt(np.mean(np.square(params_hat[0] + params_hat[1]*x_test - y_test)))
print(f'batch rms:      {rms_batch}')
print(f'stochastic rms: {rms_stochastic}')
```

```python
plt.figure()
plt.plot(np.arange(max_iter), cost_batch, 'r', label='batch')
plt.plot(np.arange(len(cost)), cost, 'g', label='stochastic')
plt.xlabel('iteration')
plt.ylabel('normalized cost')
plt.legend()
plt.show()
print(f'min cost with BGD: {np.min(cost_batch)}')
print(f'min cost with SGD: {np.min(cost)}')
```

```python

```


#Matrice
Rappresentazione di un insieme di vettori si può utilizzare una visualizzaione in tabella.
$$
v_0 = (30, 10, 10) \\
v_1 = (20, 20, 5) \\
v_2 = (15, 15 ,10) \\
\begin{bmatrix}
30 & 10 & 10 \\
20 & 20 & 5  \\
15 & 15 & 10 \\
\end{bmatrix}
$$
Questa rappresentazione in tabella è definita matrice.
Il numero di righe e colonne dipende dall'applicazione ma non esiste un limite predefinito.
$$
\begin{pmatrix}
0 & 3 & 4 & 1 \\
0 & 1 & 1 & 1 \\
1 & -1 & 0 & 0
\end{pmatrix}
$$
Anche la scelta delle parentesi non ha un significato a priori.
Quindi una matrice $A$ con $m$ righe e $n$ colonne avrà il seguente schema
$$
A =
\begin{bmatrix}
a_{11} & \cdots & a_{1n} \\
\vdots & \ddots & \vdots \\
a_{m1} & \cdots & a_{mn}
\end{bmatrix}
$$
Secondo questa notazione l'elemento di valore 3 della tabella precedente sarà l'elemento $a_{12}$.
Ogni elemento quindi può essere identificato da una riga e una colonna.
##Spazio delle righe e delle colonne
Guardando la tabella, guardando le righe, ogni riga è una n-upla. Quindi vedendo ogni riga come un vettore queste generano uno spazio vettoriale in $R^n$.
Allo stesso modo le colonne, vista ognuna come un vettore, generano un'altro spazio vettoriale questa volta sottospazio di $R^m$.
Per esempio data la matrice:
$$
A =
\begin{bmatrix}
1 & 1 & 1 \\
2 & 1 & 0
\end{bmatrix}
$$
*Spazio delle righe*
$$
r_0 = (1, 1, 1) \\
r_1 = (2, 1, 0) \\
\subseteq R^3
$$
*Spazio delle colonne*
$$
c_1 = (1, 2) \\
c_2 = (1, 1) \\
c_3 = (1, 0) \\
\subseteq R^2
$$
Quindi una stessa matrice può dare origini a spazi di righe e colonne molto diversi.
#Rango di una matrice
Quando lavoro con una matrice posso identificare la dimensione degli spazi delle righe e degli spazi delle colonne. Questi coincidono e coincidono anche con il rango della matrice stessa.
$$
\dim{L(R_1, \dots, R_m)} \\
= \\
\dim{L(C_1, \dots, C_n)} \\
= \\
\text{Rango di }A = \rho(A)
$$
Data la matrice;
$$
A =
\begin{pmatrix}
1 & 2 & 1 \\
0 & 1 & 3 \\
\end{pmatrix}
$$
Spazio delle righe
$$
R_1 = (1, 2, 1) \\
R_2 = (0 ,1, 3) \\
l.i \Rightarrow \dim{L(R_1, R_2)} = 2 \\
\rho(A) = 2
$$
Spazio delle colonne
$$
C_1 = (1, 0) \\
C_2 = (2, 1) \\
C_3 = (1, 3) \\
\dim{L(C_1, C_2, C_3)} = 2 \\
\rho(A) = 2
$$
Questo modo di procedere per quanto semplice non è ottimale man mano che la matrice cresce di dimensioni.
$$
A =
\begin{pmatrix}
1 & 2 & 1 \\
0 & 1 & 3 \\
0 & 5 & 0 \\
\end{pmatrix}
$$
Questa matrice è gia più complessa ma posso notare facilmente che
 * La prima riga non è nulla
 * La seconda riga, avendo uno 0 dove la prima ha un 1, non può essere multipla della prima.
 * Lo stesso vale per la terza riga.

La presenza di 0 in posizioni strategiche permette di capire facilmente quando due vettori sono indipendenti e di conseguenza capire la dimensione dello spazio vettoriale e di conseguenza del campo della matrice.
##Matrici ridotte per righe
$$
A =
\begin{pmatrix}
\diamond & \square & \diamond & \diamond & \diamond \\
\square  & 0       & \diamond & \diamond & \diamond \\
0        & 0       & \square  & \diamond & \diamond \\
0        & 0       & 0        & \diamond & \square  \\
0        & 0       & 0        & \diamond & 0        \\
\end{pmatrix} \\
\begin{align}
\square & = \text{elemento speciale != 0} \\
\diamond & = \text{elemento qualsiasi}\\
0 & = \text{zero}
\end{align}
$$
I numeri speciali sono quei numeri sotto i quali, nella stessa colonna, si trovono solo 0.
Una matrice di questi tipo ha un rango pari il numero delle righe non nulle.

Il medesimo ragionamento può essere fatto per le colonne.
$$
A =
\begin{pmatrix}
\diamond & \square  & 0        & 0        & 0        \\
\square  & 0        & 0        & 0        & 0        \\
\diamond & \diamond & \square  & 0        & 0        \\
\diamond & \diamond & \diamond & \diamond & \diamond \\
\diamond & \diamond & \diamond & \square  & 0        \\
\end{pmatrix} \\
$$

Il problema è che raramente le matrici su cui lavorare sono ridotte per righe o per colonne.
Devo cercare quindi un'operazione che mi permette di trovare una matrice $A^1$ ridotta che abbia lo stesso rango della matrice di origine.
Mi servno delle operazioni che conservano il rango della matrice.
#Trasformazione elementare (sulle righe)
 1. Moltiplicazione di una riga
Posso moltiplicare tutti i valori di una riga per un valore $a \neq 0$ e il rango della matrice non cambia.
 2. Scambiare di posto le righe
Posso scambare di posto la riga $R_i$ e la righa $R_j$ con $i \neq j$ e il rango della matrica non cambia
 3. Somma al multiplo
Posso prendere una riga $R_i$ e una riga $R_j$ con $i \neq j$.
Quindi sostituire la riga $R_i$ con una nuova riga alla quale sommo a ogni valore di $R_i$ i valori di $R_j$ moltiplicati per un $a \neq 0$.
Anche in questo caso il rango non cambia

###Esempio delle operazioni
$$
A =
\begin{pmatrix}
1     & 1     & 1     \\
2     & 1     & 1     \\
3     & 1     & -1
\end{pmatrix}
\rightarrow R_2 - 1R_1 \\ \rightarrow
\begin{pmatrix}
1     & 1     & 1     \\
1     & 0     & 0     \\
3     & 1     & -1
\end{pmatrix}
\rightarrow R_3 + R_1 \\ \rightarrow
\begin{pmatrix}
1     & 1     & 1     \\
1     & 0     & 0     \\
4     & 2     & 0
\end{pmatrix}
\rightarrow R_3 - 4R_2 \\ \rightarrow
\begin{pmatrix}
1     & 1     & 1     \\
1     & 0     & 0     \\
0     & 2     & 0
\end{pmatrix}
\rightarrow \text{ Matrice ridotta per righe } \rho(a) = 3
$$

Tutto questo vale anche per le colonne.

##Applicazione agli spazi vettoriali
Creando una matrice che ha come righe i generatori di uno spazio vettoriale e calcolando il rango della matrice in realtà sto calcolando la dimensione dello spazio vettoriale dato dai generatori utilizzati.
Dato uno spazio vettoriale generato da:
$$
R_0 = (1, 1, 2, 1) \\
R_1 = (2, 1, 0, 3) \\
R_3 = (4, 4, 1, 0) \\
$$
Genero la matrice:
$$
\begin{pmatrix}
1 & 1 & 2 & 1 \\
2 & 1 & 0 & 3 \\
4 & 4 & 1 & 0
\end{pmatrix}
$$
Tramite le trasformazioni elementari la riduco
$$
\begin{pmatrix}
1 & 1 & 2 & 1 \\
2 & 1 & 0 & 3 \\
4 & 4 & 1 & 0
\end{pmatrix} \\
\downarrow \\
\begin{pmatrix}
1 & 1 & 2 & 1 \\
2 & 1 & 0 & 3 \\
7 & 7 & 0 & -1
\end{pmatrix} \\
\downarrow \\
\begin{pmatrix}
1  & 1 & 2 & 1 \\
2  & 1 & 0 & 3 \\
-7 & 0 & 0 & -22 \\
\end{pmatrix} \\
$$
Matrice di rango 3, quindi lo spazio vettoriale è di dimensione 3

Algebra 20 Matrici 1 rango e riduzione.ipynb


# Homework 1: Solving an ODE

Given an ordinary differential equation:

$$
\frac{d}{dt}x(t) = cos(t)
$$

with initial condition $x(0)=0.0$

## Part 1: Julia's ODE solver

Please **solve** the ODE using `DifferentialEquations.jl` for $t \in [0.0, 4 \pi]$ and **plot** the time series. **Compare** it to the analytical solution *in one plot*.

## Part 2: The forward Euler method

Please **try** a range of dts to **solve** the ODE using the (home-made) forward Euler method for $t \in [0.0, 4 \pi]$, **plot** the time series, and **compare** them to the analytical solution *in one plot*.

**About the math**

We treat the solution curve as a straight line locally. In each step, the next state variables ($\vec{u}_{n+1}$) is accumulated by time step (dt) multiplied the derivative (tangent slope) at the current state ($\vec{u}_{n}$):

$$
\vec{u}_{n+1} = \vec{u}_{n} + dt \cdot f(\vec{u}_{n}, t_{n})
$$

```julia
# The ODE model. Exponential decay in this example
# The input/output format is compatible to Julia DiffEq ecosystem AND YOU SHOULD KEEP IT.
model(u, p, t) = cos(t)

# Forward Euler stepper
step_euler(model, u, p, t, dt) = u .+ dt .* model(u, p, t)

# In house ODE solver
function mysolve(model, u0, tspan, p; dt=0.1, stepper=step_euler)
    # Time points
    ts = tspan[1]:dt:tspan[end]
    # State variable at those time points
    us = zeros(length(ts), length(u0))
    # Initial conditions
    us[1, :] .= u0
    # Iterations
    for i in 1:length(ts)-1
        us[i+1, :] .= stepper(model, us[i, :], p, ts[i], dt)
    end
    # Results
    return (t = ts, u = us)
end

tspan = (0.0, 4.0π)
p = nothing
u0 = 0.0

sol = mysolve(model, u0, tspan, p, dt=0.2, stepper=step_euler)

# Visualization
using Plots
Plots.gr(lw=2)

# Numerical solution
plot(sol.t, sol.u, label="FE method")
plot!(sin, tspan..., label = "Analytical solution", linestyle=:dash)
```

## Part 3: The RK4 method

1. Please **try** a range of dts to **solve** the ODE using the (home-made) fourth order Runge-Kutta ([RK4](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods)) method for $t \in [0.0, 4 \pi]$, **plot** the time series, and **compare** them to the analytical solution *in one plot*.
2. Compared to the forward Eular method, which one is more efficient in terms of
time step needed for the same accuracy? You could make a visual comparison by plotting the analytical and numerical solutions together.

**About the math**

We use more steps to eliminate some of the nonlinear terms of error. In each iteration, the next state is calculated in 5 steps.

$$
\begin{align}
k_1 &= dt \cdot f(\vec{u}_{n}, t_n)  \\
k_2 &= dt \cdot f(\vec{u}_{n} + 0.5k_1, t_n + 0.5dt)  \\
k_3 &= dt \cdot f(\vec{u}_{n} + 0.5k_2, t_n + 0.5dt)  \\
k_4 &= dt \cdot f(\vec{u}_{n} + k_3, t_n + dt)  \\
u_{n+1} &= \vec{u}_{n} + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)
\end{align}
$$

Hint: replace the Euler stepper with the RK4 one

```julia
# Forward Euler stepper
step_euler(model, u, p, t, dt) = u .+ dt .* model(u, p, t)
# Your RK4 version
step_rk4(model, u, p, t, dt) = """TODO"""
```


# Vector operators in Cartesian coordinates

``continuum_mechanics`` support major vector operators such as:

- gradient of a scalar function;

- divergence of a vector function;

- curl of a vector function;

- gradient of a vector function;

- divergence of a tensor;

- Laplace operator of a scalar function;

- Laplace operator of a vector function; and

- Biharmonic operator of a scalar function.

All these operators are in the module `vector`.

```python
from sympy import *
from continuum_mechanics import vector
```

By default Cartesian coordinates are given by $x$, $y$ and $z$.
If these coordinates are used there is not necessary to specify
them when calling the vector operators

```python
init_printing()
x, y, z = symbols("x y z")
```

Following, we have some examples of vector operators in Cartesian coordinates.

## Gradient of a scalar function

The gradient takes as input a scalar and returns a vector,
represented by a 3 by 1 matrix.

```python
f = 2*x + 3*y**2 - sin(z)
f
```

```python
vector.grad(f)
```

## Divergence of a vector function

The divergence takes as input a vector (represented by a 3 by 1 matrix) and returns a scalar.

```python
vector.div(Matrix([x, y, z]))
```

```python
vector.div(Matrix([
    x**2 + y*z,
    y**2 + x*z,
    z**2 + x*y]))
```

## Divergence of a tensor function

The divergence of a tensor (represented by a 3 by 3 matrix)
returns a vector.

```python
Axx, Axy, Axz = symbols("A_xx A_xy A_xz", cls=Function)
Ayx, Ayy, Ayz = symbols("A_yx A_yy A_yz", cls=Function)
Azx, Azy, Azz = symbols("A_zx A_zy A_zz", cls=Function)
```

```python
tensor = Matrix([
    [Axx(x, y, z), Axy(x, y, z), Axz(x, y, z)],
    [Ayx(x, y, z), Ayy(x, y, z), Ayz(x, y, z)],
    [Azx(x, y, z), Azy(x, y, z), Azz(x, y, z)]])
tensor
```

```python
vector.div_tensor(tensor)
```

## Curl of a vector function

Let us check the identity

$$\nabla \times \nabla f(x, y, z) = \mathbf{0}\, .$$

```python
fun = symbols("fun", cls=Function)
vector.curl(vector.grad(fun(x, y, z)))
```


# Neural Networks - Representation

## 1. Model Representation

Neural Networks were developed as simulating networks of neurons in the brain. So, to start understanding the representation of these hypotheses, let's start by understanding how a single neuron in the brain works:

The components of the neuron are:

- A cell body;
- Input wires (dendrites);
- Output wire (axon).

The axon often goes to the dendrites of other neurons, forming a network.

The neurons communicate via pulses of electricity.

### Neuron model: Logistic unit

Given the above, we're going to use a very simple model of the neuron:

where

$$
h_{\theta}(x) = \frac{1}{1 + e^{-\theta^T x}} = g(\theta^T x)
$$

with $x=[x_0 \quad x_1 \quad x_2 \quad x_3]^T$ and $\theta=[\theta_0 \quad \theta_1 \quad \theta_2 \quad \theta_3]^T$.

Here, we are using the sigmoid (logistic) function as the **activation function**. Other functions such as $\tanh(\cdot)$ or $\mathrm{ReLU(\cdot)}=\max\{0, \cdot\}$ are also used often.

### Neural network

A neural networks is a group of these neurons acting together:

In this network:

- $a_i^{(j)}$ is the activation of unit $i$ in layer $j$.
- $\Theta^{(j)}$ is the matrix of weights controlling function mapping from layer $j$ to layer $j+1$.

Thus:

\begin{align}
a_1^{(2)} &= g(\Theta_{10}^{(1)} x_0 + \Theta_{11}^{(1)} x_1 + \Theta_{12}^{(1)} x_2 + \Theta_{13}^{(1)} x_3) \\
a_2^{(2)} &= g(\Theta_{20}^{(1)} x_0 + \Theta_{21}^{(1)} x_1 + \Theta_{22}^{(1)} x_2 + \Theta_{23}^{(1)} x_3) \\
a_3^{(2)} &= g(\Theta_{30}^{(1)} x_0 + \Theta_{31}^{(1)} x_1 + \Theta_{32}^{(1)} x_2 + \Theta_{33}^{(1)} x_3) \\
& \\
h_{\Theta}(x) &= a_1^{(3)} = g(\Theta_{10}^{(3)} a_0^{(2)} + \Theta_{11}^{(2)} a_1^{(2)} + \Theta_{32}^{(2)} a_2^{(2)} + \Theta_{33}^{(2)} a_3^{(2)})
\end{align}

In this setting $\Theta^{(1)} \in \mathbb{R}^{3 \times 4}$ and $\Theta^{(2)} \in \mathbb{R}^{1 \times 4}$.

In general, if a network has $s_j$ units in layer $j$, and $s_{j+1}$ units in layer $j+1$, then $\Theta^{(j)}$ will be of dimension $s_{j+1} \times (1 + s_j)$.

In the above setting, we can define intermediate variables

$$
z^{(j+1)}_i = \Theta_{i0}^{(j)} a_0^{(j)} + \Theta_{i1}^{(j)} a_1^{(j)} + \Theta_{i2}^{(j)} a_2^{(j)} + \Theta_{i3}^{(j)} a_3^{(j)},
$$

and in terms of $z^{(j+1)}_i$ we can define $a^{(j+1)}_i$ as:

$$
a^{(j+1)}_i = g(z^{(j+1)}_i).
$$

Moreover, we can write the above in a vectorized efficient form as:

\begin{align}
a^{(1)} = x\\
z^{(2)} &= \Theta^{(1)} \left[\begin{array}{c} 1 \\ a^{(1)} \end{array}\right] \\
a^{(2)} &= g(z^{(2)}) \\
z^{(3)} &= \Theta^{(2)} \left[\begin{array}{c} 1 \\ a^{(2)} \end{array}\right] \\
a^{(3)} &= g(z^{(3)}).
\end{align}

where

$$
x = \left[\begin{array}{c} x_1 \\ x_2 \\ x_3 \end{array}\right], \qquad z^{(2)} = \left[\begin{array}{c} z^{(2)}_1 \\ z^{(2)}_2 \\ z^{(2)}_3 \end{array}\right], \qquad a^{(2)} = \left[\begin{array}{c} a^{(2)}_1 \\ a^{(2)}_2 \\ a^{(2)}_3 \end{array}\right], \qquad z^{(3)} = z^{(3)}_1, \qquad a^{(3)} = a^{(3)}_1,
$$

and

$$
\Theta^{(1)} = \left[
\begin{array}{cccc}
\Theta_{10}^{(1)} & \Theta_{11}^{(1)} & \Theta_{12}^{(1)} & \Theta_{13}^{(1)}  \\
\Theta_{20}^{(1)} & \Theta_{21}^{(1)} & \Theta_{22}^{(1)} & \Theta_{23}^{(1)}  \\
\Theta_{30}^{(1)} & \Theta_{31}^{(1)} & \Theta_{32}^{(1)} & \Theta_{33}^{(1)}
\end{array}\right],
\qquad
\Theta^{(2)} = \left[
\begin{array}{cccc}
\Theta_{10}^{(2)} & \Theta_{11}^{(2)} & \Theta_{12}^{(2)} & \Theta_{13}^{(2)}
\end{array}\right],
$$

This algorithm is called **forward propagation**.

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez. Based on the content of the Machine Learning course offered through coursera by Prof. Andrew Ng.
</footer>


# Systems of Equations
Imagine you are at a casino, and you have a mixture of £10 and £25 chips. You know that you have a total of 16 chips, and you also know that the total value of chips you have is £250. Is this enough information to determine how many of each denomination of chip you have?

Well, we can express each of the facts that we have as an equation. The first equation deals with the total number of chips - we know that this is 16, and that it is the number of £10 chips (which we'll call ***x*** ) added to the number of £25 chips (***y***).

The second equation deals with the total value of the chips (£250), and we know that this is made up of ***x*** chips worth £10 and ***y*** chips worth £25.

Here are the equations

\begin{equation}x + y = 16 \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

Taken together, these equations form a *system of equations* that will enable us to determine how many of each chip denomination we have.

## Graphing Lines to Find the Intersection Point
One approach is to determine all possible values for x and y in each equation and plot them.

A collection of 16 chips could be made up of 16 £10 chips and no £25 chips, no £10 chips and 16 £25 chips, or any combination between these.

Similarly, a total of £250 could be made up of 25 £10 chips and no £25 chips, no £10 chips and 10 £25 chips, or a combination in between.

Let's plot each of these ranges of values as lines on a graph:

```R
library(ggplot2)
library(repr)
options(repr.plot.width=4, repr.plot.height=4)

## Create a data frames with the extreems of the possible values of chips
chips = data.frame(x = c(16,0), y = c(0,16))

## A second data frame with the extreems of the values of the chips
values = data.frame(x = c(25,0), y = c(0,10))

ggplot() + geom_line(data = chips, aes(x,y), color = 'blue', size = 1) +
           geom_line(data = values, aes(x,y), color = 'orange', size = 1)
```

Looking at the graph, you can see that there is only a single combination of £10 and £25 chips that is on both the line for all possible combinations of 16 chips and the line for all possible combinations of £250. The point where the line intersects is (10, 6); or put another way, there are ten £10 chips and six £25 chips.

### Solving a System of Equations with Elimination
You can also solve a system of equations mathematically. Let's take a look at our two equations:

\begin{equation}x + y = 16 \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

We can combine these equations to eliminate one of the variable terms and solve the resulting equation to find the value of one of the variables. Let's start by combining the equations and eliminating the x term.

We can combine the equations by adding them together, but first, we need to manipulate one of the equations so that adding them will eliminate the x term. The first equation includes the term ***x***, and the second includes the term ***10x***, so if we multiply the first equation by -10, the two x terms will cancel each other out. So here are the equations with the first one multiplied by -10:

\begin{equation}-10(x + y) = -10(16) \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

After we apply the multiplication to all of the terms in the first equation, the system of equations look like this:

\begin{equation}-10x + -10y = -160 \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

Now we can combine the equations by adding them. The ***-10x*** and ***10x*** cancel one another, leaving us with a single equation like this:

\begin{equation}15y = 90 \end{equation}

We can isolate ***y*** by dividing both sides by 15:

\begin{equation}y = \frac{90}{15} \end{equation}

So now we have a value for ***y***:

\begin{equation}y = 6 \end{equation}

So how does that help us? Well, now we have a value for ***y*** that satisfies both equations. We can simply use it in either of the equations to determine the value of ***x***. Let's use the first one:

\begin{equation}x + 6 = 16 \end{equation}

When we work through this equation, we get a value for ***x***:

\begin{equation}x = 10 \end{equation}

So now we've calculated values for ***x*** and ***y***, and we find, just as we did with the graphical intersection method, that there are ten £10 chips and six £25 chips.

You can run the following R code to verify that the equations are both true with an ***x*** value of 10 and a ***y*** value of 6.

```R
x = 10
y = 6
print((x + y == 16) & ((10*x) + (25*y) == 250))
```

    [1] TRUE


Lagrange's work on Kepler equation suggests that an equation of the form
\begin{equation}
y=x+\alpha \phi(y)
\end{equation}
Its solution is approximated by the series expansion:
\begin{equation}
y=x+\sum_{n=1}^{\infty} \frac{\alpha^{n}}{n !} \frac{d^{n-1}}{d x^{n-1}} \phi(x)^{n}
\end{equation}

```python
import numpy as np  # calculations
import pathlib  # needed to create folder
import matplotlib.pyplot as plt  # needed for graphs
import sympy as sp
from sympy.abc import E, e, n, M, o
```

Kepler's equation is
\begin{equation}
E=M+e \sin E
\end{equation}
Its solution is approximated by the series expansion:
\begin{equation}
y=M+\sum_{n=1}^{\infty} \frac{e^{n}}{n !} \frac{d^{n-1}}{d E^{n-1}} \sin(M)^{n}
\end{equation}
We plan to evaluate the series expression E(M) up to order n=3 and n=10 for eccentrities e=0.3 and e=0.9.

We're using python's package Sympy to do our symbolic interpretation.
From sympy.abc we're importing these symbols. Their usage is:
E = Eccentric anomaly of Kepler's Equation
M = Mean Anomaly of Kepler's Equation
e = Eccentricity
n = index of summation
o = upper bound of summation

Some starting setup

```python
dpisiz=100
def savim(dir,name):
    path = pathlib.Path(f"./{dir}")
    path.mkdir(exist_ok=True,parents=True)
    plt.savefig(f'./{dir}/{name}.png',dpi=dpisiz)
```

```python
fract = e ** n / sp.factorial(n)
func = (sp.sin(M)) ** n
dif_func = (sp.Derivative(func, (M, n - 1)))
sumo = sp.Sum(fract * dif_func, (n, 1, o))
sum3 = (sumo.subs(o, 3)).doit() #The form asked for n=3. Could add .simplify() as well
sum10 = (sumo.subs(o, 10)).doit() #The form asked for n=3. Could add .simplify() as well
E3 = sum3 + M
E10 = sum10 + M
En3_e3 = sp.trigsimp(E3.subs(e, 0.3))
En10_e3 = sp.trigsimp(E10.subs(e, 0.3))
En3_e9 = sp.trigsimp(E3.subs(e, 0.9))
En10_e9 = sp.trigsimp(E10.subs(e, 0.9))
```

Use sp.lambdify to create our lambda functions and plot our graphs.
Could have used sp.plot but I prefer having more options with matplotlib.

```python

numberoftries = 10000
xx = np.linspace(0, 2 * np.pi, 10000)
En3_e3f = sp.lambdify(M, En3_e3)(xx)
En10_e3f = sp.lambdify(M, En10_e3)(xx)
En3_e9f = sp.lambdify(M, En3_e9)(xx)
En10_e9f = sp.lambdify(M, En10_e9)(xx)

fig = plt.figure()
ax = plt.axes()

data = np.genfromtxt(fname="data.csv", delimiter=',') #Importing data from task 1
ax.set_ylabel('E')
ax.set_xlabel('M')
ax.plot(data[:, 0], data[:, 2], label='E for e=0.3', color='tab:pink')
ax.plot(xx, En3_e3f, label='E for n=3, e=0.3', linestyle='dashed', color='tab:red')
ax.plot(xx, En10_e3f, label='E for n=10, e=0.3', linestyle='dotted', color='tab:blue')
ax.legend()
savim('pr1_task2','e_03')
fig1 = plt.figure()
ax1 = plt.axes()
ax1.plot(data[:, 0], data[:, 5], label='E for e=0.9', color='black' )
ax1.plot(xx, En3_e9f, label='E for n=3, e=0.9', color='tan')
ax1.plot(xx, En10_e9f, label='E for n=10, e=0.9', color='fuchsia')
ax1.legend()
ax1.set_ylabel('E')
ax1.set_xlabel('M')
savim('pr1_task2','e_09')
```

As we can see from our results, for lower eccentricities both n=3 and n=10 are nearly identical compared to the Newton-Raphson method.
However, for 0.9, this is no longer the case. In this case there are some sinusoidal curves appearing except the middle of the curve (There are still some curvature showing at the middle, however it's close to the Newton-Raphson method).

Regarding the form \begin{equation}
E=M+\sum_{n} \Pi_{n}(e) \sin (n M)
\end{equation}
Lagrange's theorem states that for any f
\begin{equation}
\begin{aligned}
f(y)=f(z) &+\frac{x}{1 !} F(z) f^{\prime}(z) \\
&+\frac{x^{2}}{2 !} \frac{d}{d z}\left[\{F(z)\}^{2} f^{\prime}(z)\right] \\
&+\frac{x^{3}}{3 !} \frac{d^{2}}{d z^{2}}\left[\{F(z)\}^{3} f^{\prime}(z)\right] \\
&+\cdots \\
&+\frac{x^{n}}{n !} \frac{d^{n-1}}{d z^{n-1}}\left[\{F(z)\}^{n} f^{\prime}(z)\right] \\
&+\cdots
\end{aligned}
\end{equation}
Applying the above theorem to Kepler's equation
\begin{equation}
E=M+e \sin E
\end{equation}
it yields
\begin{equation}
\begin{aligned}
E=M+e \sin M &+\frac{e^{2}}{2 !} \frac{d}{d M}\left[\sin ^{2} M\right]+\frac{e^{3}}{3 !} \frac{d^{2}}{d M^{2}}\left[\sin ^{3} M\right]+\frac{e^{4}}{4 !} \frac{d^{3}}{d M^{3}}\left[\sin ^{4} M\right]+\cdots \\
&+\frac{e^{n}}{n !} \frac{d^{n-1}}{d M^{n-1}}\left[\sin ^{n} M\right]+\cdots
\end{aligned}
\end{equation}
Neglecting higher orders of derivatives
\begin{equation}
\begin{aligned}
\frac{d}{d M} \sin ^{2} M &=2 \cos M \sin M=\sin 2 M \\
\frac{d^{2}}{d M^{2}} \sin ^{3} M &=6 \cos ^{2} M \sin M-3 \sin ^{3} M=\frac{1}{4}(9 \sin 3 M-3 \sin M) \\
&=\frac{1}{2^{2}}\left(3^{3} \sin 3 M-3 \sin M\right) \\
\frac{d^{3}}{d M^{3}} \sin ^{4} M &=24 \cos ^{3} M \sin M-40 \cos M \sin ^{3} M=8 \sin 4 M-4 \sin 2 M \\
&=\frac{1}{2^{3}}\left(4^{3} \sin 4 M-2^{3} \cdot 4 \sin 2 M\right) \\
\frac{d^{4}}{d M^{4}} \sin ^{5} M &=120 \cos ^{4} M \sin M-440 \cos ^{2} M \sin ^{3} M+65 \operatorname{coin}^{5} M \\
&=\frac{1}{16}(625 \sin 5 M-405 \sin 3 M+10 \sin M) \\
&=\frac{1}{2^{4}}\left(5^{4} \sin 5 M-3^{4} \cdot 5 \sin 3 M+10 \sin M\right) \\
\frac{d^{5}}{d M^{5}} \sin ^{6} M &=720 \cos ^{5} M \sin M-4800 \cos ^{3} M \sin ^{3} M+2256 \cos M \sin ^{5} M \\
&=243 \sin 6 M-192 \sin 4 M+15 \sin 2 M \\
&=\frac{1}{2^{5}}\left(6^{5} \sin 6 M-4^{5} \cdot 6 \sin 4 M+2^{5} \cdot 15 \cos 2 M\right)
\end{aligned}
\end{equation}
Inserting the above derivatives to our series expression we get
\begin{equation}
\begin{aligned}
E=M &+e \sin M+\frac{e^{2}}{2 !} \sin 2 M+\frac{e^{3}}{3 ! 2^{2}}\left(3^{2} \sin 3 M-3 \sin M\right)+\frac{e^{4}}{4 ! 2^{3}}\left(4^{3} \sin 4 M-2^{3} \cdot 4 \sin 2 M\right) \\
&+\frac{e^{5}}{5 ! 2^{4}}\left(5^{4} \sin 5 M-3^{4} \cdot 5 \sin 3 M+10 \sin M\right)+\frac{e^{6}}{6 ! 2^{5}}\left(6^{5} \sin 6 M-4^{5} \cdot 6 \sin 4 M+2^{5} \cdot 15 \sin 2 M\right)+\cdots
\end{aligned}
\end{equation}
which simplifies to
\begin{equation}
\begin{aligned}
E=M &+\left(e-\frac{1}{8} e^{3}+\frac{1}{192} e^{5}+\cdots\right) \sin M+\left(\frac{1}{2} e^{2}-\frac{1}{6} e^{4}+\frac{1}{48} e^{6}+\cdots\right) \sin 2 M \\
&+\left(\frac{3}{8} e^{3}-\frac{27}{128} e^{5}+\cdots\right) \sin 3 M+\left(\frac{1}{3} e^{4}-\frac{4}{15} e^{6}+\cdots\right) \sin 4 M \\
&+\left(\frac{125}{384} e^{5}+\cdots\right) \sin 5 M+\left(\frac{27}{80} e^{6}+\cdots\right) \sin 6 M+\cdots
\end{aligned}
\end{equation}


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
```

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
alpha = symbols('alpha')
beta = symbols('beta')
t = symbols('t')
f = Function('f')
p_0 = symbols('p_0')
C1 = symbols('C1')
```

```python
a_eq = Eq(diff(f(t),t), alpha*f(t)+beta*f(t)**2)
```

```python
a_sol = dsolve(a_eq)
```

```python
a_gen = a_sol.rhs
```

```python
c1_particular = solve(Eq(a_gen.subs(t, 0), p_0), C1)
```

```python
a_part = a_gen.subs(C1, c1_particular)
```

```python
a_sol_full = simplify(a_gen + a_part)
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```


# About Feature Scaling
Scale down or scale up the data in order to standardise the range of features/ variable

## Sections

- [Min Max normalization](#Min-Max-normalization)
- [Z Score normalization](#Z-Score-normalization)
- [Decimal Scaling normalization](#Z-Score-normalization)

Let us define a list with random numbers as our independent variable

```python
import random
random.seed(332)

data = []

for x in range(100):
  data.append(random.randint(1,101))

print (data)
```

## Min-Max-scaling-normalization
Min-Max normalization is a simple technique where the technique can specifically fit the data in a pre-defined boundary.

\begin{equation} X_{norm} = \frac{X - X_{min}}{X_{max}-X_{min}} \end{equation}

In the cell below we will use the *data* list and use the min-max scaling normalisation to normalise it
- *min_max_normal* is the list of scaled output
- *X_min* is the minimum value of the data
- *x_max* is the maximum value of the data
- *D* upper boundry of the predefined range
- *C* lower boundry of the predefined range

```python
min_max_normal = []
X_min = min(data)
X_max = max(data)
D = 1
C = 0

for element in data:
    X_norm = (float(element - X_min)/(X_max - X_min))*(D - C) + C
    min_max_normal.append(X_norm)

print (min_max_normal)
```

## Z-Score-normalization
We scale the feature so that transformed features are with an mean of zero and standard deviation of one.

\begin{equation} z = \frac{x - \mu}{\sigma}\end{equation}

In the cell below we will use the *data* list and use the min-max scaling normalisation to normalise it
- *z_score_normal* is the list of scaled output
- *mean* is the minimum value of the data
- *std* is the maximum value of the data

```python
import math

def mean(column):
    """
    takes input the list of variables from the data
    returns mean of the variables in the list
    """
    sum_ = 0
    for element in column:
        sum_ = sum_ + element

    return float(sum_)/len(column)

def std(column):
    """
    takes input the list of variables from the data
    returns standard deviation of the variables in the list
    """
    if len(column) <= 1:
        return 0.0

    mean_data, sd = mean(column), 0.0

    # calculate stan. dev.
    for el in column:
        sd += (float(el) - mean_data)**2
    sd = math.sqrt(sd / float(len(column)-1))

    return sd
```

```python
z_normal = []

mean_data = mean(data)
std_data = std(data)

for element in data:
    z_norm = float(element - mean_data)/std_data
    z_normal.append(z_norm)

print (z_normal)
```

## Decimal Scaling Normalisation

We normalize by moving the decimal point of values of features. The number of decimal points moved depends on the maximum absolute value in the features, it provides the range between -1 and 1

- Take the maximum number of digits. For eg. 3031, then maximum digits is 4
- Calculate power of 10. 10^4 = 10000.
- Divide each number by 10000.

```python
ds_normal =[]
max_digits = 10**len(str(max(data)))

for element in data:
    ds_norm = float(element)/max_digits
    ds_normal.append(ds_norm)

print (ds_normal)
```

```python

```


# Enzyme kinetics #
###  Mathematical Biology | University of Cambridge |  Lent 2020 ###
Ronojoy Adhikari (ra413)

----

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint
```

### 1. Model ###

The catalysis of substrate $S$ into product $P$ by enzyme $E$ is determined by the three reactions

$$
\begin{align}
S + E &\rightarrow C\quad\quad\,\,\,\,(\text{rate  $k_1$})\\
C &\rightarrow S + E\quad (\text{rate  $k_{-1}$})\\
C &\rightarrow P + E\quad (\text{rate  $k_{2}$})\\
\end{align}
$$

comprising of the reversible binding of the substrate and enzyme into a complex $C$ and the irreversible unbinding of the complex into product and regenerated enzyme. Denoting the concentration of substrate, enzyme, complex and product by $s$, $e$, $c$ and $p$ respectively, the equation of motion is

$$
\begin{align}
\frac{ds}{dt} &= aN - bNP = N(a-bP)\\
\frac{de}{dt} &= cNP - dp = P(cN-d)\\
\frac{dc}{dt} &= cNP - dp = P(cN-d)\\
\frac{dp}{dt} &= cNP - dp = P(cN-d)\\
\end{align}
$$

Here $a$, $d$ are growth rates for prey and predators and $c$, $d$ are coefficients determining the intensity of the effects of predation. We rescale the model to bring into the working form

$$
\begin{equation}
\frac{du}{dt} = u(1-v),\quad\frac{dv}{dt} = -\alpha v(1-u)
\end{equation}
$$

We make parameter choices and set up the right hand side of the differential equation for numerical solution.

### 2. Numerical solution ##

```python
# Non-dimensionalised Lotka-Volterra system

alpha = 1.0               # predation coefficient

F = lambda x, t :[
    x[0]*(1-x[1]),        # x[0] -> u - equation for prey
    x[1]*(x[0]-1)*alpha   # x[1] -> v - equation for predators
]
```

```python
# Numerical integration of the system

# time step, final time, and time points of the solution
dt = 0.01; tf  = 10; t = np.arange(0,tf,dt)

# intial values of prey and predators: prey half as much as predators
x0 = [1,2]

# integrate
x = odeint(F,x0,t)
```

```python
# Plot of the solution

plt.plot(t, x[:, 0], label='prey')
plt.plot(t, x[:, 1], label='predator')
plt.xlabel('time'); plt.ylabel('population')
plt.grid(linestyle=':');plt.legend();plt.show()
```

Ths solution shows that the prey and predator populations oscillate in time. Initially, the predators outnumber the prey, which leads to a decline of both populations (the prey because they are being predated upon, the predators because there is not enough prey). The reduction in the number of predators allows the prey to increase, which leads to an increase in the predators and a repeating cycle.

### 3. Orbits and the constant of motion ###

The Lotka-Volterra system has the constant of motion

$$ H(u, v) = \alpha(\log u - u)  + (\log u - v) $$

as is easily verified by computing its time derivative and using the working form of Lotka-Volterra equations. The constant of motion determines the *orbits*, that is the curve obtained by plotting $u(t)$ against $v(t)$, for each $t$. In other words, the orbit is a curve in the $u-v$ plane parametrised by time. For the numerical solution, we obtain the orbit below, with the center of the dynamics indicated by a circle.

```python
# Plot of  the orbit
plt.plot(x[:,0], x[:, 1])
plt.plot(1,1,'ro')
plt.xlabel('prey'); plt.ylabel('predators')
plt.grid(linestyle=':');plt.show()
```

The orbit is closed implying that the oscillations are periodic.

### 4. Phase portrait ###

The phase portrait of the system is obtained from the plotting the vector field $\boldsymbol F$ whose components are

$$
F_1 = u(1-v),\quad F_2 = -\alpha(1-u)v
$$

The solution "flows" along the vector field and the streamlines of the vector field are solution curves.

```python
# Plot of the vector field

# generate a grid of points in the plane
u, v = np.meshgrid(np.arange(0,2.2,0.15), np.arange(0,2.2,0.15))

# compute the components of the vector field at those points
F1 = u*(1-v); F2 = -alpha*(1-u)*v

# plot
fig, ax = plt.subplots(figsize=(6,6))
clr = np.sqrt(F1*F1 + F2*F2)
ax.quiver(u,v, Fu, Fv, clr, scale=30)
ax.axis([0,2,0,2])
ax.xaxis.set_ticks([]); ax.xaxis.set_label_text('prey')
ax.yaxis.set_ticks([]); ax.yaxis.set_label_text('$v$')
ax.plot(1,1,'ro')
plt.show()
```

### Exercises ###

* Examine the effect of changing the value of $\alpha$ on the solution, paying attention to the change in period of the oscillations and the lag in time between the maxima of the prey and predators.
* Examine the effect of changing the initial condition, covering the cases where (a) the prey are initially more numerous (b) the predators are initially more numerous and (c) both prey and predators are equally numerous.
* Examine the effect of distance $d = \sqrt{u_0^2 + v_0^2}$ of the initial condition from the center at $(1,1)$ on the shape of the oscillations. What happens when the distance is zero ?
* Examine, by modifying the codes above, the effect of small amounts of competition within each population on the resulting dynamics.


# Week 1 worksheet: Introduction to elliptic PDEs

This worksheet contains a number of exercises covering both the analytical and numerical aspects of the course. This means that some parts require you to solve the problem by hand, i.e. with pen and paper, while other parts require you to write code. It should usually be obvious which parts require which.

#### Suggested reading

You will see lists of links to further reading and resources throughout the worksheets, in sections titled **Learn more:**. These will include links to the Python documentation on the topic at hand, or links to relevant book sections or other online resources. Unless explicitly indicated, these are not mandatory reading, although of course we strongly recommend that you consult them!

#### Displaying solutions

Solutions will be released after the workshop, as a new `.txt` file in the same GitHub repository. After pulling the file to Noteable, **run the following cell** to create clickable buttons under each exercise, which will allow you to reveal the solutions.

```python
%run scripts/create_widgets.py W01
```

    <IPython.core.display.Javascript object>

    Buttons created!

*How it works: You will see cells located below each exercise, each containing a command starting with `%run scripts/show_solutions.py`. You don't need to run those yourself; the command above runs a script which automatically runs these specific cells for you. The commands in each of these cells each create the button for the corresponding exercise. The Python code to achieve this is contained in `scripts/show_solutions.py`, and relies on [IPython widgets](https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20Basics.html) --- feel free to take a look at the code if you are curious.*

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

## Exercise 1

Consider the the function
$$
u(x,y) = x^4 - 6x^2y^2 + y^4
$$

### Part a)

Verify that $u(x,y)$ satisfies the Laplace equation.

```python
import sympy as sp
x,y = sp.symbols('x y')
u = x**4-6*x**2*y**2+y**4
dx = sp.diff(u,x)
dx2 = sp.diff(dx,x)
dy = sp.diff(u,y)
dy2 = sp.diff(dy,y)
print(f'{dx2}  and {dy2}')
if dx2+dy2 ==0:
    print(f'The function {u} satisfies the elliptical Laplacian Partial Differential Equation')
```

    12*x**2 - 12*y**2  and -12*x**2 + 12*y**2
    The function x**4 - 6*x**2*y**2 + y**4 satisfies the elliptical Laplacian Partial Differential Equation

### Part b)

Generate a surface plot of $u(x,y)$ in the square given by $0\le x\le1$ and $0\le y\le 1$.

- Check the [numpy.meshgrid](https://numpy.org/doc/stable/reference/generated/numpy.meshgrid.html) documentation

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
def u(x,y):
    return x**4-6*x**2*y**2+y**4

values_x = np.linspace(0,1,100)
values_y = np.linspace(0,1,100)
X,Y = np.meshgrid(values_x,values_y)
Z = u(X, Y)
fig = plt.figure()
ax = plt.axes(projection='3d')
#ax.contour3D(X, Y, Z, 50, cmap='binary')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
surf = ax.plot_surface(X, Y, Z,cmap = cm.coolwarm,linewidth=0, antialiased=False)
plt.show()

def y_zero_x_in_range_from_zero_to_one():
    y_zero = 0
    x_zero = np.linspace(0,1,10)
    z_values = []
    for i in x_zero:
        z_values.append(u(i,y_zero))
    plt.plot(x_zero,z_values, label = 'y = 0')
    print('The boundary condition for u(x,0) = x**4 on the range 0<=x<=1')
    plt.xlabel('x-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
y_zero_x_in_range_from_zero_to_one()

def y_one_x_in_range_from_zero_to_one():
    y_one = 1
    x_one= np.linspace(0,1,10)
    z_values = []
    for i in x_one:
        z_values.append(u(i,y_one))
    plt.plot(x_one,z_values,label = 'y = 1')
    print('The boundary condition for u(x,1) = x**4-6*x**2*+1 on the range 0<=x<=1')
    plt.xlabel('x-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
y_one_x_in_range_from_zero_to_one()

def x_one_y_in_range_from_zero_to_one():
    y_one = np.linspace(0,1,10)
    x_one= 1
    z_values = []
    for i in y_one:
        z_values.append(u(x_one,i))
    plt.plot(y_one,z_values,label = 'x = 1')
    print('The boundary condition for u(1,y) = x**4-6*x**2*+1 on the range 0<=y<=1')
    plt.xlabel('y-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
x_one_y_in_range_from_zero_to_one()

def x_zero_y_in_range_from_zero_to_one():
    y_one = np.linspace(0,1,10)
    x_one= 0
    z_values = []
    for i in y_one:
        z_values.append(u(x_one,i))
    plt.plot(y_one,z_values, label = 'x = 0')
    print('The boundary condition for u(0,y) = x**4 on the range 0<=y<=1')
    plt.xlabel('y-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
x_zero_y_in_range_from_zero_to_one()

def e():
    x_value = np.linspace(0,1,100)
    y_value = []
    for i in x_value:
        y_value.append(i**4-6*i**2+1)
    plt.plot(x_value,y_value, label = 'e^-x')
    plt.xlabel('x_axis')
    plt.ylabel('y_axis')
    return plt.show()
```

### Part c)

Give the boundary conditions fulfilled by $u(x,y)$ on the boundaries of the square given by $0\le x\le1$ and $0\le y\le 1$.

```python
%run scripts/show_solutions.py W01-ex1_partc
```

## Exercise 2

Consider the the function
$$
u(x,y) = e^{-x} \sin(y)
$$

### Part a)

Verify that $u(x,y)$ satisfies the Laplace equation.

```python
import sympy as sp
x,y = sp.symbols('x y')
u = sp.exp(-x)*sp.sin(y)
dx = sp.diff(u,x)
dx2 = sp.diff(dx,x)
dy = sp.diff(u,y)
dy2 = sp.diff(dy,y)
print(f'{dx2}  and {dy2}')
if dx2+dy2 ==0:
    print(f'The function {u} satisfies the elliptical Laplacian Partial Differential Equation')
```

    exp(-x)*sin(y)  and -exp(-x)*sin(y)
    The function exp(-x)*sin(y) satisfies the elliptical Laplacian Partial Differential Equation

### Part b)

Generate a surface plot of $u(x,y)$ in the square given by $0\le x\le1$ and $0\le y\le \pi$.

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
def u(x,y):
    return np.exp(-x)*np.sin(y)

values_x = np.linspace(0,1,100)
values_y = np.linspace(0,1,100)
X,Y = np.meshgrid(values_x,values_y)
Z = u(X, Y)
fig = plt.figure()
ax = plt.axes(projection='3d')
#ax.contour3D(X, Y, Z, 50, cmap='binary')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
ax.plot_surface(X, Y, Z,cmap = cm.coolwarm,linewidth=0, antialiased=False)

plt.show()

def y_zero_x_in_range_from_zero_to_one():
    y_zero = 0
    x_zero = np.linspace(0,1,10)
    z_values = []
    for i in x_zero:
        z_values.append(u(i,y_zero))
    plt.plot(x_zero,z_values, label = 'y = 0')
    print('The boundary condition for u(x,0) = 0 on the range 0<=x<=1')
    plt.xlabel('x-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
y_zero_x_in_range_from_zero_to_one()

def y_one_x_in_range_from_zero_to_one():
    y_one = 1
    x_one= np.linspace(0,1,10)
    z_values = []
    for i in x_one:
        z_values.append(u(i,y_one))
    plt.plot(x_one,z_values,label = 'y = 1')
    print('The boundary condition for u(x,1) = exp(-x)*sin(1) on the range 0<=x<=1')
    plt.xlabel('x-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
y_one_x_in_range_from_zero_to_one()

def x_one_y_in_range_from_zero_to_one():
    y_one = np.linspace(0,1,10)
    x_one= 1
    z_values = []
    for i in y_one:
        z_values.append(u(x_one,i))
    plt.plot(y_one,z_values,label = 'x = 1')
    print('The boundary condition for u(1,y) = exp(-1)*sin(x) on the range 0<=y<=1')
    plt.xlabel('y-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
x_one_y_in_range_from_zero_to_one()

def x_zero_y_in_range_from_zero_to_one():
    y_one = np.linspace(0,1,10)
    x_one= 0
    z_values = []
    for i in y_one:
        z_values.append(u(x_one,i))
    plt.plot(y_one,z_values, label = 'x = 0')
    print('The boundary condition for u(0,y) = sin(x) on the range 0<=y<=1')
    plt.xlabel('y-axis')
    plt.ylabel('z_axis')
    plt.legend()
    return plt.show()
x_zero_y_in_range_from_zero_to_one()

def e():
    x_value = np.linspace(0,1,100)
    y_value = []
    for i in x_value:
        y_value.append(np.exp(-i)*np.sin(1))
    plt.plot(x_value,y_value, label = 'e^-x')
    plt.xlabel('x_axis')
    plt.ylabel('y_axis')
    return plt.show()

```

```python
%run scripts/show_solutions.py W01-ex2_partb
```

### Part c)

Give the boundary conditions fulfilled by $u(x,y)$ on the boundaries of the square given by $0\le x\le1$ and $0\le y\le \pi$.

```python

```

```python

```

```python
%run scripts/show_solutions.py W01-ex2_partc
```

## Exercise 3

Show that the function
$$
u(x,t) = e^{-kt} \cos(mx) \cos(nt)
$$
is a solution to the partial differential equation
$$
c^2 \pdderiv{u}{x} = \pdderiv{u}{t} + 2k \pderiv{u}{t}
$$
if the constants with parameters $k$, $m$, $n$ and $c$ are related by the equation
$$
n^2 + k^2 = c^2 m^2
$$

```python
import math
import numpy as np
import matplotlib.pyplot as plt
x,t,k,n,m,c = sp.symbols('x t k n m c')
u = sp.exp(-k*t)*sp.cos(m*x)*sp.cos(n*t)
dx = sp.diff(u,x)
dx2 = sp.diff(dx,x)
dt = sp.diff(u,t)
dt2 = sp.diff(dy,t)
#print(f'{dt2}  and {dt} and {dx2}')

print( dt2 +2*k*dt)
print(c**2*dx2)

```

    2*k*(-k*exp(-k*t)*cos(m*x)*cos(n*t) - n*exp(-k*t)*sin(n*t)*cos(m*x))
    -c**2*m**2*exp(-k*t)*cos(m*x)*cos(n*t)

```python
%run scripts/show_solutions.py W01-ex3
```


Loading libraries and setting up the environment

```python
import sympy
sympy.init_printing()
```

# Newtonian Case

Equation of motion

```python
m = sympy.Symbol('m', positive=True) # Mass of the bullet
v = sympy.Symbol('v', positive=True) # Velocity
t = sympy.Symbol('t', positive=True) # Time
A = sympy.Symbol('A', positive=True) # Bullet cross section
p = sympy.Symbol('p') # Pressure
eqn_of_motion = sympy.Eq(v(t).diff(t)*m,A*p)
eqn_of_motion
```

Conservation of the Riemann invariant

```python
c_0 = sympy.Symbol('c_0', positive=True) # Initial speed of sound
c = sympy.Symbol('c', positive=True) # Speed of sound
eta = sympy.Symbol('eta', positive=True) # Adiabatic index (I'm not using gamma to avoid confusion with the Lorentz factor)
riemann_invariant_conservation = sympy.Eq(v+2*c/(eta-1),2*c_0/(eta-1))
riemann_invariant_conservation
```

Isentropic relation

```python
rho = sympy.Symbol('rho') # Density
rho_0 = sympy.Symbol('rho_0', positive=True) # Initial density
p_0 = sympy.Symbol('p_0', positive=True) # Initial pressure
entropy_conservation = sympy.Eq(p/rho**eta,p_0/rho_0**eta)
entropy_conservation
```

It will be more useful to relate the pressure to the speed of sound

```python
temp = entropy_conservation
temp = temp.subs(rho, eta*p/c**2)
temp = temp.subs(rho_0, eta*p_0/c_0**2)
temp = sympy.expand_power_base(temp,force=True).simplify()
entropy_p_vs_c = temp
entropy_p_vs_c
```

Pressure as a function of velocity

```python
temp = entropy_conservation
temp = temp.subs(rho,eta*p/c**2)
temp = temp.subs(rho_0,eta*p_0/c_0**2)
temp = temp.subs(sympy.solve(riemann_invariant_conservation,c,dict=True)[0])
temp = sympy.expand_power_base(temp,force=True)
temp = sympy.solve(temp,p)[0]
temp = sympy.expand_power_base(temp, force=True).simplify()
p_vs_v = temp
sympy.Eq(p,p_vs_v)
```

Terminal velocity

```python
temp = riemann_invariant_conservation
temp = sympy.solve(temp.subs(c,0),v)[0]
terminal_velocity = temp
terminal_velocity
```

Solving the equation of motion

```python
y = sympy.Symbol('y', positive=True)
v_t = sympy.Symbol('v_t', positive=True)
temp = eqn_of_motion
temp = temp.subs(p, p_vs_v.subs(v,terminal_velocity*(1-y)))
temp = temp.subs(v(t), -terminal_velocity*(1-y(t)))
temp = temp.doit()
temp = sympy.expand_power_base(temp, force=True)
temp = temp.simplify()
temp = temp.subs(y(t).diff(t),y/t)
temp = sympy.solve(temp,y)[0]
temp = (temp*terminal_velocity).subs(c_0,v_t*(eta-1)/2)
temp = sympy.expand_power_base(temp,force=True).simplify()
asymptotic_velocity = temp
asymptotic_velocity
```

The thickness of the bullet (size along the direction of motion) increases with time according to

```python
w_0 = sympy.Symbol('w_0', positive=True) # Initial thickness
temp = w_0*(p/p_0)**(-1/eta)
temp = temp.subs(sympy.solve(entropy_p_vs_c,p,dict=True)[0])
temp = temp.subs(c_0, v_t)
temp = temp.subs(c, asymptotic_velocity)
temp = sympy.expand_power_base(temp).simplify()
thickness_history = temp
thickness_history
```

Time when the bullet is broken apart by the Rayleigh Taylor instability

```python
w = sympy.Symbol('w', positive=True) # Current thickness of the bullet
a = sympy.Symbol('a', positive=True) # Acceleration
temp = sympy.Eq(t*sympy.sqrt(a/w),1)
temp = temp.subs(a,-asymptotic_velocity.diff(t)).simplify()
temp = temp.subs(w, thickness_history)
sympy.expand_power_base(temp).simplify()
```

We have two timescales in this problem: the sound crossing time of the bullet $w_0/v_t$, and the acceleration time $m v_t/A p_0$. If the acceleration time is larger than the sound crossing time, then the bullet disintegrates right at the beginning. If not, then it never will.

# Relativistic Case

Equation of motion

```python
gamma = sympy.Symbol('gamma', positive=True) # Lorentz factor
C = sympy.Symbol('C', positive=True) # Speed of light
ur_eqn_of_motion = sympy.Eq(C*m*gamma(t).diff(t), A*p)
ur_eqn_of_motion
```

Conservation of the relativistic Riemann invariant

```python
ur_riemann_invariant_conservation = sympy.Eq(p,p_0*gamma**(-sympy.sqrt(eta-1)/eta))
ur_riemann_invariant_conservation
```

Solving the equation of motion

```python
temp = ur_eqn_of_motion.subs(p,ur_riemann_invariant_conservation.rhs)
temp = temp.subs(gamma(t).diff(t),gamma/t)
lf_history = sympy.solve(temp,gamma)[0]
lf_history
```

Pressure history

```python
temp = ur_riemann_invariant_conservation.rhs
ur_pressure_history = temp.subs(gamma, lf_history).simplify()
ur_pressure_history
```

Density history

```python
eta2 = sympy.Symbol('eta2', positive=True)
temp = rho_0*(p/p_0)**(1/eta)
temp = temp.subs(p,ur_pressure_history)
temp = temp.subs(eta,eta2+1)
ur_density_history = sympy.expand_power_base(temp,force=True).simplify().subs(eta2, eta-1)
ur_density_history
```

Now, let us turn our attention to the Riemann problem. On the left (negative) side, there's a photon gas with pressure $p_l$. On the right, there's a cold baryonic matter with mass density $\rho_r$. Both fluids are stationary. In the case where $p_l/\rho_r \ll c^2$, then the first shock is non relativistic, and what we get is basically the non relativistic problem, boosted to a relativistic velocity. If, on the other hand $p_l/\rho_r \gg c^2$, then the first shock is relativistic. We call the second case the genuinely relativistic case.

## Boosted Newtonian Case

Thickness of the bullet in the fluid frame

```python
temp = w_0*rho_0/rho
temp = temp.subs(rho, ur_density_history)
ff_bullet_thickness = temp
ff_bullet_thickness
```

Breakup time

```python
temp = (t/gamma)**2*a/ff_bullet_thickness
temp = temp.subs(gamma, lf_history)
temp = temp.subs(a, lf_history/t)
temp = temp.subs(eta, eta2+1)
temp = sympy.expand_power_base(temp, force=True).simplify().subs(eta2, eta-1).simplify()
tentative_growth_factor=temp
tentative_growth_factor
```

```python
t*sympy.log(tentative_growth_factor).diff(t).simplify()
```

```python
bn_t_breakup = sympy.solve(tentative_growth_factor-1,t)[0]
[bn_t_breakup,
 sympy.expand_power_base(bn_t_breakup, force=True).simplify().subs(eta,sympy.Rational(4,3)).simplify()]
```

## Genuinely Relativistic Case

In this case the initial pressure in the bullet will be different from the initial pressure in the barrel. To determine this pressre, we need to find the intersection between the relative Hugoniot (Taub) curve of the bullet and the rarefaction curve of the barrel

```python
taub_curve = sympy.Eq(p, rho_0*C**2*gamma**2)
taub_curve
```

```python
rel_riemann_problem_intersection = sympy.solve([taub_curve, ur_riemann_invariant_conservation],[p,gamma])[1]
rel_riemann_problem_intersection
```

When calculating the growth factor, one has to take into account the contribution of the pressure to the intertia

```python
temp = (t/gamma)*(a/w)*(rho*C**2/p)
temp = temp.subs(w,w_0*(rho_0/rho))
temp = temp.subs(rho, rho_0*(p/p_0)**(1/eta))
temp = temp.subs(p,rel_riemann_problem_intersection[1]*(gamma/rel_riemann_problem_intersection[0])**(sympy.sqrt(eta-1)/eta))
temp = temp.subs(a, gamma/t)
temp = temp.subs(gamma, lf_history)
temp = temp.subs(eta,eta2+1)
temp = sympy.expand_power_base(temp, force=True)
temp = temp.simplify()
temp = temp.subs(eta2, eta-1).simplify()
gr_growth_factor = temp
gr_growth_factor
```

Breakup time

```python
temp = sympy.solve(gr_growth_factor-1,t)[0]
temp = temp.subs(eta, eta2+1)
gr_breakup_time = sympy.expand_power_base(temp, force=True).simplify().subs(eta2, eta-1)
gr_breakup_time
```

For an equation of state with $\eta=\frac{4}{3}$

```python
gr_breakup_time.subs(eta,sympy.Rational(4,3)).simplify()
```

We note that the relativistic Rayleigh Taylor rate is approximately $\sqrt{k a\frac{\rho_2-\rho_1}{\rho_1+\rho_2+p/c^2}}$. As one might expect, the difference with respect to the newtonian rate is the inclusion of the pressure term as part of the inertia (denominator). It does not appear in the driving term (numerator) because it is the same of both sides.

```python

```