## In-Class Demonstration and Visualization

```python
%matplotlib inline
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
```

```python
# You can change the default figure size to be a bit larger if you want,
# uncomment the next line for that:
#plt.rc('figure', figsize=(10, 6))
```

The function `plot_taylor_approximations` included here was written by Fernando Perez and was part of work on the original `IPython` project.  Although attribution seems to have been lost over time, we gratefully acknowledge FP and thank him for this code!

```python
def plot_taylor_approximations(func, x0=None, orders=(2, 4), xrange=(0,1), yrange=None, npts=200):
    """Plot the Taylor series approximations to a function at various orders.

    Parameters
    ----------
    func : a sympy function
    x0 : float
      Origin of the Taylor series expansion.  If not given, x0=xrange[0].
    orders : list
      List of integers with the orders of Taylor series to show.  Default is (2, 4).
    xrange : 2-tuple or array.
      Either an (xmin, xmax) tuple indicating the x range for the plot (default is (0, 1)),
      or the actual array of values to use.
    yrange : 2-tuple
      (ymin, ymax) tuple indicating the y range for the plot.  If not given,
      the full range of values will be automatically used.
    npts : int
      Number of points to sample the x range with.  Default is 200.
    """
    if not callable(func):
        raise ValueError('func must be callable')
    if isinstance(xrange, (list, tuple)):
        x = np.linspace(float(xrange[0]), float(xrange[1]), npts)
    else:
        x = xrange
    if x0 is None: x0 = x[0]
    xs = sp.Symbol('x')
    # Make a numpy-callable form of the original function for plotting
    fx = func(xs)
    f = sp.lambdify(xs, fx, modules=['numpy'])
    # We could use latex(fx) instead of str(), but matploblib gets confused
    # with some of the (valid) latex constructs sympy emits.  So we play it safe.
    plt.plot(x, f(x), label=str(fx), lw=2)
    # Build the Taylor approximations, plotting as we go
    apps = {}
    for order in orders:
        app = fx.series(xs, x0, n=order).removeO()
        apps[order] = app
        # Must be careful here: if the approximation is a constant, we can't
        # blindly use lambdify as it won't do the right thing.  In that case,
        # evaluate the number as a float and fill the y array with that value.
        if isinstance(app, sp.numbers.Number):
            y = np.zeros_like(x)
            y.fill(app.evalf())
        else:
            fa = sp.lambdify(xs, app, modules=['numpy'])
            y = fa(x)
        tex = sp.latex(app).replace('$', '')
        plt.plot(x, y, label=r'$n=%s:\, %s$' % (order, tex) )

    # Plot refinements
    if yrange is not None:
        plt.ylim(*yrange)
    plt.grid()
    plt.legend(loc='best').get_frame().set_alpha(0.8)

# For an expression made from elementary functions, we must first make it into
# a callable function, the simplest way is to use the Python lambda construct.
# plot_taylor_approximations(lambda x: 1/sp.cos(x), 0, [2,4,6,8], (0, 2*sp.pi), (-5,5))
```

```python
plot_taylor_approximations(sp.sin, 0, [2, 4, 6, 8], (0, 2*sp.pi), (-2,2))
```

# Lecture 10:  Taylor's Series and Discrete Calculus

## Reading and Reference

* Essential Mathematical Methods for Physicists, H. Weber and G. Arfken, Academic Press, 2003
* Advanced engineering Mathematics, E. Kreyszig, John wiley and Sons, 2010
* Numerical Recipes, W. Press, Cambridge University Press, 1986
* Numerical Methods, R. Hornbeck, Quantum Publishers, 1975

## What to Learn?

* Definition of an infinite series and power series
* Identify the Taylor series
* How to derive the central and forward difference formulae from Taylor's series
* Learn the SymPy functions to compute series
* Learn how to represent discrete data as a matrix

## What to Do?

* Derive Taylor's series
* Use SymPy functions to produce series representations
* Compute numerical derivatives using data provided
* Advanced Challenge:  Vectorize the numerical derivative calculations

### Introduction
----

It is common in physics and engineering to represent transcendental functions and other nonlinear expressions using a few terms from a Taylor series.  This series provides a fast and efficient way to compute quantities such as $\mathrm{sin}(x)$ or $e^x$ to a prescribed error.  Learning how to calculate the series representation of these functions will provide practical experience with the Taylor series and help the student understand the results of Python methods designed to accelerate and simplify computations.  The series can be written generally as:

$$f(x) = f{\left (0 \right )} + x \left. \frac{d}{d x} f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{2}}{2} \left. \frac{d^{2}}{d x^{2}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{3}}{6} \left. \frac{d^{3}}{d x^{3}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{4}}{24} \left. \frac{d^{4}}{d x^{4}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{5}}{120} \left. \frac{d^{5}}{d x^{5}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \mathcal{O}\left(x^{6}\right)$$

Of equal importance the Taylor series permits discrete representation of derivatives and is a common way to perform numerical integration of partial and ordinary differential equations.  Expansion of a general function $f(x)$ about a point, coupled with algebraic manipulation, will produce expressions that can be used to approximate derivative quantities.  Although any order of derivative can be computed, this lesson will focus on first and second derivatives that will be encountered in the diffusion equation.

### Infinite Sequences
----

Ideas relating to sequences, series, and power series are used in the formulation of integral calculus and in the construction of polynomial representations of functions.  The limit of functions will also be investigated as boundary conditions for differential equations.  For this reason understanding concepts related to sequences and series are important to review.

A sequence is an ordered list of numbers.  A list such as the following represents a sequence:

$$a_1, a_2, a_3, a_4, \dots, a_n, \dots $$

The sequence maps one value $a_n$ for every integer $n$.  It is typical to provide a formula for construction of the nth term in the sequence.  While _ad-hoc_ strategies could be used to develop sequences using SymPy and lists in Python, SymPy has a sequence class that can be used.  A short demonstration is provided next:

```python
import sympy as sp
#from __future__ import division
x, y, z, t, a = sp.symbols('x y z t a')
k, m, n = sp.symbols('k m n', integer=True)
f, g, h = sp.symbols('f g h', cls=sp.Function)
sp.var('a1:6')
sp.init_printing()
```

It is important to read about `SymPy` symbols at this time.

We can generate a sequence using `SeqFormula`.

```python
a1 = sp.SeqFormula(n**2, (n,0,5))
list(a1)
```

if we want the limit of the sequence at infinity:

$$[0, 1, 4, 9, \ldots ]$$

we can use `limit_seq`:

```python
sp.limit_seq(a1.formula, n)
```

### DIY:  Determine if the following sequences are convergent or divergent.  If convergent, what is the limit?

$$
\begin{aligned}
a_n = & \frac{1}{n} \\
a_n = & 1 - (0.2)^n \\
a_n = & \frac{1}{2n+1}
\end{aligned}
$$

```python
# Your code here.
```

### Infinite Series
----

A series is the sum of a sequence.  An infinite series will converge if the partial sums of the series has a finite limit.  For example, examine the partial sums of the series:

$$
\sum^{\infty}_{n=1} \frac{1}{2^n}
$$

```python
a2 = sp.Sum(1/2**n, (n,0,1))
a2
```

```python
a2.doit()
```

```python
a4 = sp.Sum(n**2, (n,0,5))
a4
```

```python
a5 = sp.Sum(k**2, (k, 1, m))
a5
```

```python
a4.doit()
```

```python
a5.doit()
```

A power series is of the form:

$$
\sum_{n=0}^{\infty} M_{n} x^{n} = M_0 + M_1 x + M_2 x^2 + \cdots
$$

```python
M = sp.IndexedBase('M')
```

```python
sp.Sum(M[n]*x**n, (n,0,m))
```

We can define the series about the point $a$ as follows:

```python
sp.Sum(M[n]*(x-a)**n, (n,0,m))
```

SymPy has a function that can take SymPy expressions and represent them as power series:

```python
sp.series(f(x), x, x0=0)
```

### DIY:  Use SymPy to determine series approximations to $e^x$, $sin(x)$, and $cos(x)$ about the point $x=0$.

$$
\begin{aligned}
a_n = & \frac{1}{n} \\
a_n = & 1 - (0.2)^n \\
a_n = & \frac{1}{2n+1}
\end{aligned}
$$

```python
# Your code here.
```

### Taylor's Series
----

Below we present a derivation of Taylor's series and small algebraic argument for series representations of functions.  In contrast to the ability to use `sympy` functions without any deeper understanding, these presentations are intended to give you insight into the origin of the series representation and the factors present within each term.  While the algebraic presentation isn't a general case, the essential elements of a general polynomial representation are visible.

The function $f(x)$ can be expanded into an infinite series or a finite series plus an error term.  Assume that the function has a continuous nth derivative over the interval $a \le x \le b$.  Integrate the nth derivative n times:

$$\int_a^x f^n(x) dx = f^{(n-1)}(x) - f^{(n-1)}(a)$$

The power on the function $f$ in the equation above indicates the order of the derivative.  Do this n times and then solve for f(x) to recover Taylor's series.  One of the key features in this derivation is that the integral is definite.  This derivation is outlined on [Wolfram’s Mathworld](http://mathworld.wolfram.com/TaylorSeries.html).

As a second exercise, assume that we wish to expand sin(x) about x=0.  First, assume that the series exists and can be written as a power series with unknown coefficients.  As a first step, differentiate the series and the function we are expanding.  Next, let the value of x go to the value of the expansion point and it will be possible to evaluate the coefficients in turn:

$$
\sin x = A+Bx+Cx^2+Dx^3+Ex^4
$$

We can choose an expansion point (e.g. $x = 0$) and differentiate to get a set of simultaneous equations permitting determination of the coefficients.  The computer algebra system can help us with this activity:

```python
import sympy as sp
sp.init_printing()
```

```python
x, A, B, C, D, E = sp.symbols('x, A, B, C, D, E')
```

To help us get our work done we can use `sympy`'s `diff` function.  Testing this function with a known result, we can write:

```python
sp.diff(sp.sin(x),x)
```

A list comprehension is used to organize the results.  In each iteration the exact function and the power series are differentiated and stored as an element of a list.  The list can be inspected and a set of simultaneous equations can be written down and solved to determine the values of the coefficients.  Casting the list as a `sympy` `Matrix` object clarifies the correspondance between entries in the list.

```python
orderOfDifferentiation = 1
powerSeries = A+B*x+C*x**2+D*x**3+E*x**4
# Differentiate, element by element, the list [sp.sin(x),powerSeries]
[sp.diff(a,x,orderOfDifferentiation) for a in [sp.sin(x),powerSeries]]
```

A list comprehension can be used to organize and extend the results further.  We can wrap the list above into another list that changes the order of differentiation each iteration.

```python
maximumOrder = 5
funcAndSeries = [[sp.diff(a,x,order) for a in [sp.sin(x),powerSeries]] for order in range(maximumOrder)]
funcAndSeries
```

Casting the results as a `sympy` `Matrix` object the list is more easily viewed in the Jupyter notebook:

```python
sp.Matrix(funcAndSeries)
```

### DIY:  Determine the coefficients in the above power series.
----

You don't necessarily need to write code to complete this DIY problem.

```python
# Your code here if you feel you need it.
```

`Your markdown here.`

### Computing a Taylor's Series Symbolically
---

Using `sympy` the Taylor's series can be computed symbolically.

```python
from sympy import init_printing, symbols, Function
init_printing()

x, c = symbols("x,c")
f = Function("f")

f(x).series(x, x0=c, n=3)
```

One of the major uses of Taylor's series in computation is for the evaluation of derivatives.  Take note of the fact that the derivatives of a function appear in the evaluation of the series.

### Computing Derivatives of Discrete Data
---

It may be straightforward to compute the derivative of some functions.  For example:

$$f(x) = x^2$$
$$f'(x) = 2x$$

In numerical computing situations there is no analytical solution to the problem being solved and therefore no function to integrate or differentiate.  The approximate solution is available as a list of discrete points in the domain of the problem's independent variables (e.g. space, time).  The values could be represented as a list of numbers:

$$\{f(x_0), f(x_1), f(x_2), ...\}$$

The neighboring points $f(x_0)$ and $f(x_1)$ are seperated by a distance $\Delta x = x_1 - x_0$ in the independent variable.  Although this will not be apparent from the values, it is implicit in the structure of the data.  Taylor's series can be used to compute approximate derivatives of the unknown function directly from the list of points in this situation.  We are going to compute a series expansion for an unknown function $f(x)$ in the vicinity of the point $c$ and then examine the relationship between that function and it's derivative quantities at a point $c\pm h$.  The goal of the activity is to see if we can find expressions for the derivatives using the data point of interest ($c$) and its neighbors ($c \pm h$).  We are going to use the idea of _forward_ and _backward_ differences.

1. Forward differences are computed by expanding an unknown function in a Taylor series about a point “x=c” and then letting x go to c+h.
1. Then, for backward differences, let x go to c-h.

### Symbolically Computing Forward and Backward Differences
---

In the figure below we indicate the following:

* the unknown function $f(x)$ as a dashed line
* the point about which the unknown function is expanded at $x=c$
* the distance between successive points is shown as $h$
* the approximate values of the function given at the filled squares

Imagine that we take the above series expansion and use it to compute the value of the function near the point $c$.  Let us evaluate this series by adding and subtracting to the independent varable the quantity $h$.  To accomplish this we write down the series expansion for our function about the point $c$, then we let the independent variable $x \rightarrow c+h$ and $c-h$.

```python
x, h, c = sp.symbols("x,h,c")
f = sp.Function("f")

# the .subs() method replaces occurences of 'x' with somethine else
taylorExpansionPlus = f(x).series(x, x0=c, n=3).removeO().subs(x,c+h)
taylorExpansionMinus = f(x).series(x, x0=c, n=3).removeO().subs(x,c-h)
```

```python
taylorExpansionPlus
```

Meaning that:

$$
f(c+h) = \frac{h^{2}}{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + h \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + f{\left (c \right )}
$$

```python
taylorExpansionMinus
```

Meaning that:

$$
f(c-h) = \frac{h^{2}}{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} - h \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + f{\left (c \right )}
$$

### Solving for First and Second Derivatives
----

Inspection of the results shows that the signs on the terms containing the first derivative are different between the two expressions.  We can use this to our advantage in solving for the derivative terms explicitly.  Note that each grouped expression is equal to zero as is the default in `sympy`.

_Find the first derivative in this expression:_

```python
(taylorExpansionMinus-f(c-h))-(taylorExpansionPlus-f(c+h))
```

Remember that `sympy` expressions are zero by default.  So this is true:

$$
- 2 h \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} - f{\left (c - h \right )} + f{\left (c + h \right )} = 0
$$

_Find the second derivative in this expression:_

```python
(taylorExpansionMinus-f(c-h))+(taylorExpansionPlus-f(c+h))
```

Similarly:

$$
h^{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + 2 f{\left (c \right )} - f{\left (c - h \right )} - f{\left (c + h \right )} = 0
$$

Solve each of the equations above for the derivative quantites and you will have recovered the formula for discrete derivatives.

### DIY:  Compute Numerical Derivatives

Generate a list of values that approximate the function $f(x)=x^8$ on the domain $\{x | 0 \leq x \leq 1\}$.  Using these values, numerically compute the derivative at your selected grid points and compare it to the analytical solution.  Using this technique, examine how the observed error changes as the number of grid points is varied.  Visualize and explain the results.

### Challenge DIY:  Vectorized Computations

A _list_ is one of the fundamental data structures within Python.  Numpy (a Python library) and other parts of Python libraries use _vectorized_ computations.  From Wikipedia, vectorization is "a style of computer programming where operations are applied to whole arrays instead of individual elements."

With this in mind, we certainly _can_ iterate over our list of points and apply the function that you will soon write in an element by element fashion, however, it is a more common practice in Python and other modern languages to write vectorized code.  If this is your first exposure to vectorized computation, I recommend two initial strategies:  write out your algorithms and use "classic" flow control and iteration to compute the results.  From that point you will more easily see the strategy you should use to write vectorized code.

Using the discrete forms of the first and second derivatives (based on central differences) can you devise a vectorized operation that computes the derivative without looping in Python?


Mathematical Optimization for Engineers

Lab 2

Basic math

$$\newcommand{\mr}[1]{\mathrm{#1}}
\newcommand{\D}{\displaystyle}
\newcommand{\bm}[1]{\text{\mathbf $#1$}}
\newcommand{\bx}{\mathbf{ x}}
\newcommand{\f}{\mathbf{{f}}}
\newcommand{\g}{\mathbf{ g}}
\newcommand{\h}{\mathbf{ h}}
\newcommand{\R}{\mathbb R}
\newcommand{\A}{\mathbf{ A}}
\newcommand{\br}{\boldsymbol{r}}
\newcommand{\bp}{\boldsymbol{p}}
\newcommand{\bnabla}{\mathbf{\nabla}}
$$
In this lab, we will learn about Jacobians, gradients and Hessian matrices.

Notation: Please note that throughout the course, we will denote matrices and vectors with boldface letters. Their components will be denoted by normal letters with subscripts. For example,
$$
\bx = \left(\begin{array}{c}
x_1 \\
\vdots \\
x_n
\end{array} \right)
$$

## Jacobian
Let $\ \mathbf \f:\R^n \rightarrow \R^m,\,\bx \mapsto \f(\bx)$ be a continuously differentiable function, where
$$
\bx = \left(\begin{array}{c}
x_1 \\
\vdots \\
x_n
\end{array} \right) , \qquad \f(\bx)  = \left(\begin{array}{c}
f_1(\bx) \\
\vdots \\
f_m(\bx)
\end{array} \right).
$$
The Jacobian $\f'(\bx)$ is defined by the matrix
$$
\f'(\bx) = \left(  \begin{array}{ccc}
\frac{\partial f_1(\bx)}{\partial x_1} & \cdots & \frac{\partial f_1(\bx)}{\partial x_n} \\
\vdots & \ddots & \vdots \\
\frac{\partial f_m(\bx)}{\partial x_1} & \cdots & \frac{\partial f_m(\bx)}{\partial x_n}
\end{array}  \right).
$$

## Gradient

Now, let $f:\R^n \rightarrow \R,\,\bx \mapsto f(\bx)$ be a scalar-valued continuously differentiable function
with vector-valued arguments.
The gradient $\bnabla f(\bx)$ and Jacobian $f'(\bx)$ are defined as the column vector and row vector, respectively,

$$
\bnabla f(\bx)  = \left(\begin{array}{c}
	\frac{\partial f(\bx)}{\partial x_1} \\
	\vdots \\
	\frac{\partial f(\bx)}{\partial x_n}
\end{array} \right), \qquad  f'(\bx)  = \left(\begin{array}{ccc}
\frac{\partial f(\bx)}{\partial x_1} &
\cdots &
\frac{\partial f(\bx)}{\partial x_n}
\end{array}\right).
$$

Note that  $\bnabla f(\bx)^T=f'(\bx)$.

This is also valid for any general $\ \mathbf \f:\R^n \rightarrow \R^m,\,\bx \mapsto \f(\bx)$

### Gradient of the scalar product of two functions

Let $\g,\h:\R^n\rightarrow\R^n$ be two continuously differentiable functions. We want to compute the
gradient of $f:\R^n\rightarrow \R,\;\bx \mapsto f(\bx) = \g(\bx)^T\h(\bx)$, where $\bx\in\R^n$.
We have

$$
f(\bx) = \g(\bx)^T\h(\bx) = \sum_{i=1}^{n} g_i(\bx)h_i(\bx).
$$

The derivative with respect to $x_j$ ($1 \le j \le n$) is computed by the application of the product rule

\begin{equation}\label{eq:1}
\frac{\partial f(\bx)}{\partial x_j} = \sum_{i=1}^{n} \left(\frac{\partial g_i(\bx)}{\partial x_j}h_i(\bx)  +g_i(\bx)\frac{\partial h_i(\bx)}{\partial x_j}\right).
\end{equation}

With the notations

$$
\frac{\partial \g(\bx)}{\partial x_j}   = \left(\begin{array}{c}
\frac{\partial g_1(\bx)}{\partial x_j} \\
\vdots \\
\frac{\partial g_n(\bx)}{\partial x_j}
\end{array} \right) \quad \text{and} \quad
\frac{\partial \h(\bx)}{\partial x_j}
= \left(\begin{array}{c}
\frac{\partial h_1(\bx)}{\partial x_j} \\
\vdots \\
\frac{\partial h_n(\bx)}{\partial x_j}
\end{array} \right), \text{ respectively},
$$

we can rewrite the equation as

$$
\frac{\partial f(\bx)}{\partial x_j} = \frac{\partial \g(\bx)}{\partial x_j} ^T \h(\bx) +
\g(\bx)^T \frac{\partial \h(\bx)}{\partial x_j}  = \h(\bx)^T\frac{\partial \g(\bx)}{\partial x_j}
+\g(\bx)^T \frac{\partial \h(\bx)}{\partial x_j}
$$

Finally,

$$
\bnabla f(\bx)^T = f'(\bx) = \h(\bx)^T \g'(\bx) + \g(\bx)^T\h'(\bx).
$$

$$
\implies \bnabla f(\bx) = \bnabla g(\bx) \ \h(\bx) + \bnabla h(\bx) \ \g(\bx).
$$

### Derivative of quadratic form
If $\A\in\R^{n\times n}$ is a symmetric matrix, the function

$$
f:\R^n\rightarrow \R,\quad\bx\mapsto f(\bx) = \bx^T\,\A\,\bx
$$ is called a quadratic form.

With the definitions,

$$
\g(\bx) := \bx \ \text{and } \ \h(\bx):= \A\,\bx,
$$

we have $f(\bx) = \g(\bx)^T\h(\bx)$, i.e., exactly the situation as above.
With $\g'(\bx) = \mathbf{ I}$, where $\mathbf{ I}$ denotes the unity matrix, and $\h'(\bx) = \A$, it is more or less easy to see that the gradient of $f$ is given by

$$
\bnabla f(\bx) = 2\,\A\,\bx.
$$

### Example 1
Let the function $f:\R^2 \rightarrow \R$ be defined as
$ f(x,y) = \frac{1}{2} (x^2 + \alpha y^2), \alpha \in \R $

Task 1: Find all stationary points of the function $f$.

Task 2: Calculate the Hessian of the function $f$ for arbitrary $x$ and $y$.

Task 3: What are the eigenvalues of the Hessian of the function $f$ with respect to $x$ and $y$?

Task 4: Characterize the stationary points for positive and negative $\alpha$.

Task 5: Characterize the convexity of the function for every $\alpha$.

### Example 2: Rosenbrock function
The Rosenbrock function is a famous test function used in optimization.

It is defined by:
$f:\R^2 \rightarrow \R$, $ f(x,y) = (a-x)^2 + b(y-x^2)^2 $ with $a=1, b=100$

Task 1: Find all stationary points of the function $f$.

Task 2: Calculate the Hessian of the function $f$ for the stationary points found. (Hint: use Python)

Task 3: What are the eigenvalues of the Hessian of the function $f$? (Hint: use Python)

```python
def rosenbrock(x):
    return ((x[0]-1)**2 + 100*(x[1]-x[0]**2)**2)
```

```python
# compute hessian using autograd
import autograd
x0 = # stationary point
hessian_rosenbrock = # add your code here
```

```python
# compute the eigenvalues using numpy
import numpy as np
# add your code here
```


(Other_Activation_Functions)=
# Chapter 16 -- Other Activation Functions

The other solution for the vanishing gradient is to use other activation functions. We like the old activation function sigmoid $\sigma(h)$ because first, it returns $0.5$ when $h=0$ (i.e. $\sigma(0)$) and second, it gives a higher probability when the input value is positive and vice versa. This makes it the perfect activation function for predicting the probability. However, the vanishing gradient is a major problem we cannot ignore. In DNN (Deep Neural Network), fortunately, we can use the sigmoid function for only the output layer, and use other activation functions for hidden layers. Here are some alternatives for the activation function that do not share the same vanishing gradient problem.

$$
tanh(x)=\frac{e^h-e^{-h}}{e^h+e^{-h}}
$$ (eq16_1)

where $h=w*x+b$.

```python
# make the figure be plotted at the centre
from IPython.core.display import HTML
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

```python
import numpy as np
import matplotlib.pyplot as plt
N  = 100

def main():
    h = np.linspace(-5, 5, N)
    tanh = (np.exp(h)-np.exp(-h))/(np.exp(h)+np.exp(-h))

    plt.figure()
    plt.plot(h, tanh)
    plt.xlabel('$h$')
    plt.ylabel('$tanh(h)$')
    plt.title('Figure 1.4 Tanh function')

    plt.show()

if __name__ == '__main__':
    main()
```

$tanh(x)$ is centred at $0$, which means the probability would be negative if the input value is negative and vice versa. The possibility of having a negative value allows the weight to update better than sigmoid. For sigmoid function that only produces positive values, all weights to the same neuron must either increase together or decrease together. That's a problem, since some of the weights may need to increase while others need to decrease. That can only happen if some of the input activations have different signs. That suggests replacing the sigmoid by an activation function, such as tanh, which allows both positive and negative activations. Of course, the tanh has slightly steeper gradient than the sigmoid but it still faces the vanishing gradient problem.

\begin{equation}
ReLU(h)=max(0,h)
\end{equation}

```python
#ReLu function
def relu(X):
    return np.maximum(0,X)

N  = 100

def main():
    h = np.linspace(-5, 5, N)
    Relu = relu(h)

    plt.figure()
    plt.plot(h, Relu)
    plt.xlabel('$h$')
    plt.ylabel('$Relu(h)$')
    plt.title('Figure 1.4 Relu function')

    plt.show()

if __name__ == '__main__':
    main()
```

This activation function is wildly used in CNN (Convolutional Neural Network) because of the two characters: it is easy to compute and it does not have a vanishing gradient problem at all. Nevertheless, it has the biggest problem, which is there is no derivative at the point ($x=0$). We can avoid this by keep our learning rate low. On the other hand, when the weighted input to a rectified linear unit is negative, the gradient vanishes, and so the neuron stops learning entirely. Some recent work on image recognition has found considerable benefit in using rectified linear units through much of the network. However, as with tanh neurons, we do not yet have a really deep understanding of when, exactly, rectified linear units are preferable, nor why.

The following figure 16.1 is a MLP/DNN model with modified activation functions. In this model, the activation functions are changed to ReLU from sigmoid for all hidden layers but the output layer (in order to predict the probability).

<center> Figure 16.1

notebooks/e_extra/pytorch_image_filtering_ml/Chapter 16 -- Other Activation Functions.ipynb


# Imports

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
```

```python
plt.style.use(["ggplot"])
```

# Create Data

 Generate some data with:
\begin{equation} \theta_0= 4 \end{equation}
\begin{equation} \theta_1= 3 \end{equation}

Add some Gaussian noise to the data

Here we have $$y=\theta_0 + \theta_1 *X+ random\ noise$$

```python
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)
```

Let's plot our data to check the relation between X and Y

## plot generated 2d data points

```python
plt.plot(X, y, "b.")
plt.xlabel("$x$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
ax = plt.axis([0, 2, 0, 15])
```

#  Analytical way of Linear Regression

```python
# ?? np.c_
```

```python
X_b = np.c_[np.ones((100, 1)), X]
X_b
```

    array([[1.        , 0.61161301],
           [1.        , 1.72430602],
           [1.        , 1.85534309],
           [1.        , 1.76073836],
           [1.        , 0.0572145 ],
           [1.        , 1.84703953],
           [1.        , 0.23746452],
           [1.        , 1.66383103],
           [1.        , 0.2041836 ],
           [1.        , 1.51621319],
           [1.        , 1.40221905],
           [1.        , 1.68842929],
           [1.        , 0.23088307],
           [1.        , 1.96224888],
           [1.        , 1.38270468],
           [1.        , 0.45367799],
           [1.        , 1.51284161],
           [1.        , 1.57549713],
           [1.        , 0.6783873 ],
           [1.        , 1.89365556],
           [1.        , 0.21850822],
           [1.        , 1.85162873],
           [1.        , 0.76138162],
           [1.        , 1.21935501],
           [1.        , 1.52439915],
           [1.        , 0.80818048],
           [1.        , 0.70451694],
           [1.        , 0.88903564],
           [1.        , 0.32331054],
           [1.        , 1.64909732],
           [1.        , 1.7124657 ],
           [1.        , 0.91713771],
           [1.        , 0.57914975],
           [1.        , 0.29721128],
           [1.        , 1.10640219],
           [1.        , 1.90267708],
           [1.        , 0.52570261],
           [1.        , 0.76683326],
           [1.        , 1.60819434],
           [1.        , 1.78457635],
           [1.        , 1.17548247],
           [1.        , 0.50681977],
           [1.        , 0.04839011],
           [1.        , 1.71238211],
           [1.        , 1.17437814],
           [1.        , 1.35064057],
           [1.        , 1.46373702],
           [1.        , 0.42525494],
           [1.        , 0.82589216],
           [1.        , 1.09337039],
           [1.        , 1.53121379],
           [1.        , 1.74522245],
           [1.        , 0.33265673],
           [1.        , 0.70568204],
           [1.        , 0.55856574],
           [1.        , 1.92412633],
           [1.        , 0.38815032],
           [1.        , 1.33906861],
           [1.        , 0.82516803],
           [1.        , 1.5456871 ],
           [1.        , 0.71522878],
           [1.        , 0.08590715],
           [1.        , 0.58955981],
           [1.        , 0.61951299],
           [1.        , 0.97332899],
           [1.        , 0.44926547],
           [1.        , 1.94176703],
           [1.        , 0.18386009],
           [1.        , 0.15639567],
           [1.        , 1.76878058],
           [1.        , 0.4371604 ],
           [1.        , 0.90612205],
           [1.        , 0.20664235],
           [1.        , 1.58972011],
           [1.        , 1.62836902],
           [1.        , 1.75144256],
           [1.        , 0.40163699],
           [1.        , 1.67312172],
           [1.        , 1.81273512],
           [1.        , 0.44030981],
           [1.        , 1.60587802],
           [1.        , 1.79572153],
           [1.        , 0.16385152],
           [1.        , 0.00740951],
           [1.        , 1.09224804],
           [1.        , 1.45951573],
           [1.        , 1.27301069],
           [1.        , 0.49603731],
           [1.        , 0.01205083],
           [1.        , 1.0129211 ],
           [1.        , 1.57774042],
           [1.        , 0.83671178],
           [1.        , 0.35653958],
           [1.        , 1.00719932],
           [1.        , 1.10596479],
           [1.        , 1.11281235],
           [1.        , 1.19792334],
           [1.        , 0.46265133],
           [1.        , 1.81943243],
           [1.        , 0.77343146]])

Normal Equation

$\hat{\boldsymbol\beta} = (\mathbf{X}^\mathsf{T}\mathbf{X})^{-1} \mathbf{X}^\mathsf{T} \mathbf{y}$

```python
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
theta_best
```

    array([[4.2682626 ],
           [2.78901042]])

This is close to our real thetas 4 and 3. It cannot be accurate due to the noise I have introduced in data

```python
X_new = np.array([[0], [2]])
X_new_b = np.c_[np.ones((2, 1)), X_new]
y_predict = X_new_b.dot(theta_best)
y_predict
```

    array([[4.2682626 ],
           [9.84628344]])

Let's plot prediction line with calculated:theta

```python
plt.plot(X_new, y_predict, "r-")
plt.plot(X, y, "b.")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([0, 2, 0, 15])
```

# Gradient Descent

## Cost Function & Gradients

 The equation for calculating cost function and gradients are as shown below. Please note the cost function is for Linear regression. For other algorithms the cost function will be different and the gradients would have  to be derived from the cost functions

Cost
\begin{equation}
J(\theta) = 1/2m \sum_{i=1}^{m} (h(\theta)^{(i)} - y^{(i)})^2
\end{equation}

Gradient

\begin{equation}
\frac{\partial J(\theta)}{\partial \theta_j} = 1/m\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_j^{(i)}
\end{equation}

Gradients
\begin{equation}
\theta_0: = \theta_0 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}
\begin{equation}
\theta_1: = \theta_1 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_1^{(i)})
\end{equation}
\begin{equation}
\theta_2: = \theta_2 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_2^{(i)})
\end{equation}

\begin{equation}
\theta_j: = \theta_j -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_j^{(i)})
\end{equation}

```python
def cal_cost(theta, X, y):
    """

    Calculates the cost for given X and Y. The following shows and example of a single dimensional X
    theta = Vector of thetas
    X     = Row of X's np.zeros((2,j))
    y     = Actual y's np.zeros((2,1))

    where:
        j is the no of features
    """

    m = len(y)

    predictions = X.dot(theta)
    cost = (1 / 2 * m) * np.sum(np.square(predictions - y))
    return cost
```

```python
def gradient_descent(X, y, theta, learning_rate=0.01, iterations=100):
    """
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    """
    m = len(y)
    cost_history = np.zeros(iterations)
    theta_history = np.zeros((iterations, 2))
    for it in range(iterations):

        prediction = np.dot(X, theta)

        theta = theta - (1 / m) * learning_rate * (X.T.dot((prediction - y)))
        theta_history[it, :] = theta.T
        cost_history[it] = cal_cost(theta, X, y)

    return theta, cost_history, theta_history
```

 Let's start with 1000 iterations and a learning rate of 0.01. Start with theta from a Gaussian distribution

```python
lr = 1e-2
n_iter = 1000

theta = np.random.randn(2, 1)

X_b = np.c_[np.ones((len(X), 1)), X]
theta, cost_history, theta_history = gradient_descent(X_b, y, theta, lr, n_iter)

print(
    "Theta0:          {:0.3f},\nTheta1:          {:0.3f}".format(
        theta[0][0], theta[1][0]
    )
)
print("Final cost/MSE:  {:0.3f}".format(cost_history[-1]))
```

    Theta0:          3.969,
    Theta1:          3.033
    Final cost/MSE:  3869.407

 Let's plot the cost history over iterations

```python
fig, ax = plt.subplots(figsize=(12, 8))

ax.set_ylabel("J(Theta)")
ax.set_xlabel("Iterations")
_ = ax.plot(range(n_iter), cost_history, "b.")
```

 After around 150 iterations the cost is flat so the remaining iterations  are not needed or will not result in any further optimization. Let us zoom in till iteration 200 and see the curve

```python
fig, ax = plt.subplots(figsize=(10, 8))
_ = ax.plot(range(200), cost_history[:200], "b.")
```

It is worth while to note that the cost drops faster initially and then the gain in cost reduction is not as much

### It would be great to see the effect of different learning rates and iterations together

### Let us  build a function which can show the effects together and also show how gradient decent actually is working

```python
def plot_GD(n_iter, lr, ax, ax1=None):
    """
     n_iter = no of iterations
     lr = Learning Rate
     ax = Axis to plot the Gradient Descent
     ax1 = Axis to plot cost_history vs Iterations plot

     """
    _ = ax.plot(X, y, "b.")
    theta = np.random.randn(2, 1)

    tr = 0.1
    cost_history = np.zeros(n_iter)
    for i in range(n_iter):
        pred_prev = X_b.dot(theta)
        theta, h, _ = gradient_descent(X_b, y, theta, lr, 1)
        pred = X_b.dot(theta)

        cost_history[i] = h[0]

        if i % 25 == 0:
            _ = ax.plot(X, pred, "r-", alpha=tr)
            if tr < 0.8:
                tr = tr + 0.2
    if not ax1 == None:
        _ = ax1.plot(range(n_iter), cost_history, "b.")
```

### Plot the graphs for different iterations and learning rates combination

```python
fig = plt.figure(figsize=(30, 25), dpi=200)
fig.subplots_adjust(hspace=0.4, wspace=0.4)

it_lr = [(2000, 0.001), (500, 0.01), (200, 0.05), (100, 0.1)]
count = 0
for n_iter, lr in it_lr:
    count += 1

    ax = fig.add_subplot(4, 2, count)
    count += 1

    ax1 = fig.add_subplot(4, 2, count)

    ax.set_title("lr:{}".format(lr))
    ax1.set_title("Iterations:{}".format(n_iter))
    plot_GD(n_iter, lr, ax, ax1)
```

 See how useful it is to visualize the effect of learning rates and iterations on gradient descent. The red lines show how the gradient descent starts and then slowly gets closer to the final value

## You can always plot Indiviual graphs to zoom in

```python
_, ax = plt.subplots(figsize=(14, 10))
plot_GD(100, 0.1, ax)
```

# Stochastic Gradient Descent

```python
def stocashtic_gradient_descent(X, y, theta, learning_rate=0.01, iterations=10):
    """
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    """
    m = len(y)
    cost_history = np.zeros(iterations)

    for it in range(iterations):
        cost = 0.0
        for i in range(m):
            rand_ind = np.random.randint(0, m)
            X_i = X[rand_ind, :].reshape(1, X.shape[1])
            y_i = y[rand_ind].reshape(1, 1)
            prediction = np.dot(X_i, theta)

            theta = theta - (1 / m) * learning_rate * (X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta, X_i, y_i)
        cost_history[it] = cost

    return theta, cost_history
```

```python
lr = 0.5
n_iter = 50

theta = np.random.randn(2, 1)

X_b = np.c_[np.ones((len(X), 1)), X]
theta, cost_history = stocashtic_gradient_descent(X_b, y, theta, lr, n_iter)

print(
    "Theta0:          {:0.3f},\nTheta1:          {:0.3f}".format(
        theta[0][0], theta[1][0]
    )
)
print("Final cost/MSE:  {:0.3f}".format(cost_history[-1]))
```

    Theta0:          4.160,
    Theta1:          2.860
    Final cost/MSE:  38.051

```python
fig, ax = plt.subplots(figsize=(10, 8))

ax.set_ylabel("{J(Theta)}", rotation=0)
ax.set_xlabel("{Iterations}")
theta = np.random.randn(2, 1)

_ = ax.plot(range(n_iter), cost_history, "b.")
```

# Mini Batch Gradient Descent

```python
def minibatch_gradient_descent(
    X, y, theta, learning_rate=0.01, iterations=10, batch_size=20
):
    """
    X    = Matrix of X without added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    """
    m = len(y)
    cost_history = np.zeros(iterations)
    n_batches = int(m / batch_size)

    for it in range(iterations):
        cost = 0.0
        indices = np.random.permutation(m)
        X = X[indices]
        y = y[indices]
        for i in range(0, m, batch_size):
            X_i = X[i : i + batch_size]
            y_i = y[i : i + batch_size]

            X_i = np.c_[np.ones(len(X_i)), X_i]

            prediction = np.dot(X_i, theta)

            theta = theta - (1 / m) * learning_rate * (X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta, X_i, y_i)
        cost_history[it] = cost

    return theta, cost_history
```

```python
lr = 0.1
n_iter = 200

theta = np.random.randn(2, 1)

theta, cost_history = minibatch_gradient_descent(X, y, theta, lr, n_iter)

print(
    "Theta0:          {:0.3f},\nTheta1:          {:0.3f}".format(
        theta[0][0], theta[1][0]
    )
)
print("Final cost/MSE:  {:0.3f}".format(cost_history[-1]))
```

    Theta0:          4.201,
    Theta1:          2.844
    Final cost/MSE:  750.136

```python
fig, ax = plt.subplots(figsize=(10, 8))

ax.set_ylabel("{J(Theta)}", rotation=0)
ax.set_xlabel("{Iterations}")
theta = np.random.randn(2, 1)

_ = ax.plot(range(n_iter), cost_history, "b.")
```


```python
import sympy as sm
import sympy.physics.mechanics as me
me.init_vprinting()
```

```python
k, m, l, g = sm.symbols('k, m, l, g')

q1, q2 = me.dynamicsymbols('q1, q2')

u1, u2 = me.dynamicsymbols('u1, u2')
```

```python
N, A, B = sm.symbols('N, A, B', cls=me.ReferenceFrame)
```

```python
O, Ao, Bo = sm.symbols('O, Ao, Bo', cls=me.Point)
```

```python
A.orient_axis(N, q1, N.z)
B.orient_axis(A, q2, A.x)
```

```python
A.set_ang_vel(N, u1*N.z)
B.set_ang_vel(A, u2*A.x)
```

```python
O.set_vel(N, 0)
```

```python
Ao.set_pos(O, l/2*A.x)
Bo.set_pos(O, l*A.x)
```

```python
Ao.v2pt_theory(O, N, A)
```

```python
Bo.v2pt_theory(O, N, A)
```

```python
A.ang_vel_in(N)
```

```python
B.ang_vel_in(N)
```

```python
me.partial_velocity([Ao.vel(N), Bo.vel(N),
                     A.ang_vel_in(N), B.ang_vel_in(N)], [u1, u2], N)
```

```python
v_Ao_1 = Ao.vel(N).diff(u1, N)
v_Ao_2 = Ao.vel(N).diff(u2, N)
v_Bo_1 = Bo.vel(N).diff(u1, N)
v_Bo_2 = Bo.vel(N).diff(u2, N)
w_A_1 = A.ang_vel_in(N).diff(u1, N)
w_A_2 = A.ang_vel_in(N).diff(u2, N)
w_B_1 = B.ang_vel_in(N).diff(u1, N)
w_B_2 = B.ang_vel_in(N).diff(u2, N)
```

```python
R_Ao = m*g*N.x
R_Bo = m*g*N.x
T_A = -q1*k*N.z + q2*k*B.x
T_B = -q2*k*B.x
```

```python
F1 = v_Ao_1.dot(R_Ao) + v_Bo_1.dot(R_Bo) + w_A_1.dot(T_A) + w_B_1.dot(T_B)
F1
```

```python
F2 = v_Ao_2.dot(R_Ao) + v_Bo_2.dot(R_Bo) + w_A_2.dot(T_A) + w_B_2.dot(T_B)
F2
```

```python
Rs_Ao = -m*Ao.acc(N)
Rs_Bo = -m*Bo.acc(N)
```

```python
I = m*l**2/12
I_A_Ao = I*me.outer(A.y, A.y) + I*me.outer(A.z, A.z)
I_B_Bo = I*me.outer(B.x, B.x) + I*me.outer(B.z, B.z)
```

```python
Ts_A = -(A.ang_acc_in(N).dot(I_A_Ao) +
         me.cross(A.ang_vel_in(N), I_A_Ao).dot(A.ang_vel_in(N)))
Ts_B = -(B.ang_acc_in(N).dot(I_B_Bo) +
         me.cross(B.ang_vel_in(N), I_B_Bo).dot(B.ang_vel_in(N)))
```

```python
Fs1 = v_Ao_1.dot(Rs_Ao) + v_Bo_1.dot(Rs_Bo) + w_A_1.dot(Ts_A) + w_B_1.dot(Ts_B)
Fs1
```

```python
Fs2 = v_Ao_2.dot(Rs_Ao) + v_Bo_2.dot(Rs_Bo) + w_A_2.dot(Ts_A) + w_B_2.dot(Ts_B)
Fs2
```

```python
Fr = sm.Matrix([F1, F2])
Fr
```

```python
Frs = sm.Matrix([Fs1, Fs2])
Frs
```

```python

```


```python
from scipy import linalg as sla
import numpy as np
import sympy as sy
sy.init_printing()
import matplotlib.pyplot as plt
```

```python
A=sy.Matrix([[3,-2],[-3,5]])
A
```

```python
A.rank()
```

```python
A.condition_number().simplify()
```

```python
sy.N(_)
```

```python
A.norm()
```

# confirm matrix $A$ property using numpy

```python
A=np.array([[3,2],[-3,5]])
```

```python
np.linalg.matrix_rank(A)
```

    2

```python
np.linalg.cond(A)
```

```python
np.linalg.norm(A)
```

# solve linear equation $A\mathbf{x}=\mathbf{b}$ using sympy

```python
A=sy.Matrix([[3,2],[-3,5]])
b=sy.Matrix([8,-1])
A,b
```

```python
x=A.inv()@b
x
```

```python
x=A.inv()*b
x
```

```python
#check solve
b_=A*x
b_
```

```python
#you can also express as follow using solve
x=A.solve(b)
x
```

# LU-decomposition

```python
A=sy.Matrix([[3,2],[3,-5]])
A
```

```python
L,U,_=A.LUdecomposition()
L,U,_
```

```python
#check A=LU
L*U
```

```python
%timeit x=A.solve(b)
```

    100 loops, best of 3: 844 µs per loop

```python
%timeit x=A.LUsolve(b)
```

    1000 loops, best of 3: 293 µs per loop

# solve linear eq $A\mathbf{x}=\mathbf{b}$ using numpy and scipy

```python
A=np.array([[3,2],[-3,5]])
b=np.array([[8,-1]]).T
x=sla.solve(A,b)
print(x)
```

    [[ 2.]
     [ 1.]]

```python
#check
A @ x-b
```

    array([[ 0.],
           [ 0.]])

```python
P,L,U=sla.lu(A)
print(P)
print(L)
print(U)
```

    [[ 1.  0.]
     [ 0.  1.]]
    [[ 1.  0.]
     [-1.  1.]]
    [[ 3.  2.]
     [ 0.  7.]]

```python
#check PLU=A
P @ L @ U
```

    array([[ 3.,  2.],
           [-3.,  5.]])

```python
LU=sla.lu_factor(A)
LU
```

    (array([[ 3.,  2.],
            [-1.,  7.]]), array([0, 1], dtype=int32))

```python
x=sla.lu_solve(LU,b)
x
```

    array([[ 2.],
           [ 1.]])

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
```

# 8-3. Activation functions

## Sigmoid function

$$
\sigma (x) = \frac{1}{1 + \exp(-x)}
$$

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```

```python
sigmoid(0)
```

    0.5

```python
sigmoid(-10)
```

    4.5397868702434395e-05

```python
sigmoid(10)
```

    0.99995460213129761

```python
x = np.arange(-5, 5, 0.1)
y = sigmoid(x)

plt.plot(x, y)
plt.show()
```

## Derivative of sigmoid function

\begin{align}
\sigma (x) = \frac{1}{1 + \exp(-x)}
\end{align}

Let

\begin{align}
f(x) &= 1 + \exp(-x) \\
g(u) &= u^{-1}
\end{align}

Now we can write the sigmoid function using $f(x)$ and $g(u)$

\begin{align}
\sigma (x) = g(f(x))
\end{align}

As derivatives of $f(x)$ and $g(u)$ are as follows,

\begin{align}
f'(x) &= - \exp(-x) \\
g'(u) &= - u^{-2}
\end{align}

and by using the chain rule

\begin{align}
\sigma' (x) &= g'(f(x)) \cdot f'(x) \\
&= g'(1 + \exp(-x)) \cdot (- \exp(-x)) \\
&= -(1 + \exp(-x))^{-2} \cdot (- \exp(-x)) \\
&= - \frac{- \exp(-x)}{(1 + \exp(-x))^{-2}} \\
&= \frac{\exp(-x)}{(1 + \exp(-x))^{-2}} \\
&= \frac{1}{1 + \exp(-x)} \cdot \frac{\exp(-x)}{1 + \exp(-x)} \\
&= \frac{1}{1 + \exp(-x)} \cdot \left(  1 - 1 + \frac{\exp(-x)}{1 + \exp(-x)} \right) \\
&= \frac{1}{1 + \exp(-x)} \cdot \left( 1 - \frac{1 + \exp(-x)}{1 + \exp(-x)} + \frac{\exp(-x)}{1 + \exp(-x)} \right) \\
&= \frac{1}{1 + \exp(-x)} \cdot \left( 1 - \frac{1}{1 + \exp(-x)} \right)
\end{align}

Becase $\sigma (x) = \frac{1}{1 + \exp(-x)}$

\begin{align}
\sigma' (x) &= \sigma(x) (1 - \sigma(x))
\end{align}

```python
def sigmoid_prime(x):
    s = sigmoid(x)
    return s * (1 - s)
```

```python
sigmoid_prime(0)
```

    0.23500371220159449

```python
sigmoid_prime(-1)
```

    0.19661193324148185

```python
sigmoid_prime(1)
```

    0.19661193324148185

```python
x = np.arange(-5, 5, 0.1)
y = sigmoid_prime(x)

plt.plot(x, y)
plt.show()
```

## Tanh function

Tanh function or hyperbolic tangent function is defined as follows

$$
\tanh (x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}
$$

Also as other hyperbolic functions (hyperbolic sine and hyperbolic cosine) defined as following,

\begin{align}
\sinh (x) = \frac{\exp(x) - \exp(-x)}{2} \\
\cosh (x) = \frac{\exp(x) + \exp(-x)}{2}
\end{align}

tanh function can be express as below (that is the origina definition)

$$
\tanh (x) = \frac{\sinh (x)}{\cosh (x)}
$$

```python
def tanh(x):
    return np.tanh(x)
```

```python
tanh(0)
```

    0.0

```python
tanh(-10)
```

    -0.99999999587769273

```python
tanh(5)
```

    0.99990920426259511

```python
x = np.arange(-5, 5, 0.1)
y = tanh(x)

plt.plot(x, y)
plt.show()
```

## Derivative of tanh function

$$
\tanh (x) = \frac{\sinh (x)}{\cosh (x)}
$$

Because derivatives of sinh and cosh are as below

\begin{align}
\sinh'(x) = \cosh (x)
\cosh'(x) = \sinh (x)
\end{align}

and by using quotient rule

\begin{align}
\tanh'(x) &= \frac{\sinh' (x) \cosh (x) - \sinh (x) \cosh' (x)}{\cosh^2 (x)} \\
&= \frac{\cosh^2 (x) - \sinh^2 (x)}{\cosh^2 (x)} \\
&= 1 - \tanh^2 (x)
\end{align}

```python
def tanh_prime(x):
    t = tanh(x)
    return 1 - np.power(t, 2)
```

```python
tanh_prime(0)
```

    1.0

```python
tanh_prime(-1)
```

    0.41997434161402614

```python
tanh_prime(1)
```

    0.41997434161402614

```python
x = np.arange(-5, 5, 0.1)
y = tanh_prime(x)

plt.plot(x, y)
plt.show()
```

## ReLU function

ReLU or rectified linear function is defined as following:

$$
f(x) = \max(0, x)
$$

```python
def relu(x):
    return np.maximum(0, x)
```

```python
relu(2)
```

    2

```python
relu(10)
```

    10

```python
relu(-1.5)
```

    0.0

```python
x = np.arange(-5, 5, 0.1)
y = relu(x)

plt.plot(x, y)
plt.show()
```

## Derivative of ReLU function

$$
f'(x)= \begin{cases}
      1 & (x > 0) \\
      0 & (x \leq 0)
\end{cases}
$$

```python
def relu_prime(x):
    return 1 * (x > 0)
```

```python
relu_prime(2)
```

    1

```python
relu_prime(10)
```

    1

```python
relu_prime(-1.5)
```

    0

```python
x = np.arange(-5, 5, 0.1)
y = relu_prime(x)

plt.plot(x, y)
plt.show()
```

## Leaky ReLU

$$
f(x) = max(\alpha x, x)
$$

where $\alpha$ is a small constnt such as 0.01.

```python
def lrelu(x, alpha=0.05):
    return np.maximum(alpha * x, x)
```

```python
lrelu(2)
```

    2.0

```python
lrelu(10)
```

    10.0

```python
lrelu(-1.5)
```

    -0.075000000000000011

```python
x = np.arange(-5, 5, 0.1)
y = lrelu(x)

plt.plot(x, y)
plt.show()
```

## Derivative of Leaky ReLU function

$$
f'(x)= \begin{cases}
      1 & (x > 0) \\
      \alpha & (x \leq 0)
\end{cases}
$$

```python
def lrelu_prime(x, alpha=0.05):
    return 1 * (x > 0) + alpha * (x <= 0)
```

```python
lrelu_prime(2)
```

    1.0

```python
lrelu_prime(10)
```

    1.0

```python
lrelu_prime(-1.5)
```

    0.05

```python
x = np.arange(-5, 5, 0.1)
y = lrelu_prime(x)

plt.plot(x, y)
plt.show()
```

## Comparison

```python
x = np.arange(-3, 3, 0.1)
y_sigmoid = sigmoid(x)
y_tanh = tanh(x)
y_relu = relu(x)
y_lrelu = lrelu(x)

plt.plot(x, y_sigmoid, 'b', label='sigmoid')
plt.plot(x, y_tanh, 'r', label='tanh')
plt.plot(x, y_relu, 'g', label='relu')
plt.plot(x, y_lrelu, 'c', label='lrelu')
plt.legend()
plt.show()
```

```python
x = np.arange(-6, 6, 0.1)
y_sigmoid = sigmoid_prime(x)
y_tanh = tanh_prime(x)
y_relu = relu_prime(x)
y_lrelu = lrelu_prime(x)

plt.plot(x, y_sigmoid, 'b', label="sigmoid'")
plt.plot(x, y_tanh, 'r', label="tanh'")
plt.plot(x, y_relu, 'g', label="relu'")
plt.plot(x, y_lrelu, 'c', label="lrelu'")
plt.legend()
plt.show()
```

```python

```


# 推导自然梯度下降的多项分布形式（logit为自变量）

```python
import sympy
import numpy as np
sympy.init_printing()
```

```python
theta = [sympy.symbols(f'phi_{i}') for i in range(1, 4)]
```

```python
theta
```

```python
exps = [sympy.exp(theta_i) for theta_i in theta]
```

```python
thetas = [exp/sum(exps) for exp in exps]
```

```python
thetas
```

```python
xs = [sympy.symbols(f'x_{i}') for i in range(1, 4)]
```

```python
ll = sympy.log(thetas[0]**xs[0]*thetas[1]**xs[1]*thetas[2]**xs[2])
```

```python
sympy.diff(ll, theta[2]).simplify()
```

$$\displaystyle \frac{x_{1} e^{\phi_{2}} + x_{1} e^{\phi_{3}} - x_{2} e^{\phi_{1}} - x_{3} e^{\phi_{1}}}{e^{\phi_{1}} + e^{\phi_{2}} + e^{\phi_{3}}}$$
$$\displaystyle \frac{- x_{1} e^{\phi_{2}} + x_{2} e^{\phi_{1}} + x_{2} e^{\phi_{3}} - x_{3} e^{\phi_{2}}}{e^{\phi_{1}} + e^{\phi_{2}} + e^{\phi_{3}}}$$
$$\displaystyle \frac{- x_{1} e^{\phi_{3}} - x_{2} e^{\phi_{3}} + x_{3} e^{\phi_{1}} + x_{3} e^{\phi_{2}}}{e^{\phi_{1}} + e^{\phi_{2}} + e^{\phi_{3}}}$$

# 计算信息矩阵

```python
-sympy.diff(ll, theta[2], 2).simplify().subs(xs[0], thetas[0]).subs(xs[1], thetas[1]).subs(xs[2], thetas[2]).simplify()

```

```python
-sympy.diff(ll, theta[1], 2).simplify().subs(xs[0], thetas[0]).subs(xs[1], thetas[1]).subs(xs[2], thetas[2]).simplify()

```

```python
-sympy.diff(ll, theta[0], 2).simplify().subs(xs[0], thetas[0]).subs(xs[1], thetas[1]).subs(xs[2], thetas[2]).simplify()
```

$$\displaystyle \frac{\left(e^{\phi_{2}} + e^{\phi_{3}}\right) e^{\phi_{1}}}{e^{2 \phi_{1}} + e^{2 \phi_{2}} + e^{2 \phi_{3}} + 2 e^{\phi_{1} + \phi_{2}} + 2 e^{\phi_{1} + \phi_{3}} + 2 e^{\phi_{2} + \phi_{3}}}$$

$$\displaystyle \frac{\left(e^{\phi_{1}} + e^{\phi_{3}}\right) e^{\phi_{2}}}{e^{2 \phi_{1}} + e^{2 \phi_{2}} + e^{2 \phi_{3}} + 2 e^{\phi_{1} + \phi_{2}} + 2 e^{\phi_{1} + \phi_{3}} + 2 e^{\phi_{2} + \phi_{3}}}$$

$$\displaystyle \frac{\left(e^{\phi_{1}} + e^{\phi_{2}}\right) e^{\phi_{3}}}{e^{2 \phi_{1}} + e^{2 \phi_{2}} + e^{2 \phi_{3}} + 2 e^{\phi_{1} + \phi_{2}} + 2 e^{\phi_{1} + \phi_{3}} + 2 e^{\phi_{2} + \phi_{3}}}$$

# 计算 $\partial_\theta\log P_\theta(x)$

```python
sympy.diff(ll, theta[0]).simplify()
```

$$\displaystyle \frac{x_{1} e^{\phi_{2}} + x_{1} e^{\phi_{3}} - x_{2} e^{\phi_{1}} - x_{3} e^{\phi_{1}}}{e^{\phi_{1}} + e^{\phi_{2}} + e^{\phi_{3}}}$$

```python
sympy.diff(ll, theta[1]).simplify()
```

$$\displaystyle \frac{- x_{1} e^{\phi_{2}} + x_{2} e^{\phi_{1}} + x_{2} e^{\phi_{3}} - x_{3} e^{\phi_{2}}}{e^{\phi_{1}} + e^{\phi_{2}} + e^{\phi_{3}}}$$

```python
sympy.diff(ll, theta[2]).simplify()
```

$$\displaystyle \frac{- x_{1} e^{\phi_{3}} - x_{2} e^{\phi_{3}} + x_{3} e^{\phi_{1}} + x_{3} e^{\phi_{2}}}{e^{\phi_{1}} + e^{\phi_{2}} + e^{\phi_{3}}}$$

```python
sympy.diff(ll, theta[0]).simplify()
```


```python
import numpy as np
from prettytable import PrettyTable
from scipy.interpolate import lagrange
import matplotlib.pyplot as plt
from sympy import Symbol
def Lagrange(x, L_x, L_y,y = 0):
  m = len(L_x)
  for i in range(0, m):
    prod_i = 1
    for j in range(m):
      if i==j:
        continue
      else:
        prod_i = prod_i*(x - L_x[j])/(L_x[i] - L_x[j])
    y = y + L_y[i]*prod_i
  return y
def Lagrange_inverse(y,L_x,L_y,x=0):
    return Lagrange(y,L_y,L_x,x=0)
def inbuilt(x, L_x, L_y):
    poly = lagrange(L_x, L_y)
    L=poly(x)
    return L
L1 = [0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0]
L2 = [1.0, 0.99, 0.96, 0.91, 0.85, 0.76, 0.67, 0.57, 0.46, 0.34, 0.22, 0.11, 0.00, -0.10, -0.18, -0.26]

I = [2.81, 3.24, 3.80, 4.30, 4.37, 5.29, 6.03]
V = [0.5, 1.2, 2.1, 2.9, 3.6, 4.5, 5.7]
mytable = PrettyTable(["Problem" , "inbuilt", "my function","Relative Error"])
mytable.add_rows([["3(a)(i)",inbuilt(2.3, L1, L2),Lagrange(2.3, L1, L2),
                   (inbuilt(2.3, L1, L2)-Lagrange(2.3, L1, L2))/inbuilt(2.3, L1, L2)],
                  ["3(a)(ii)",inbuilt(0.5, L2,L1),Lagrange(0.5, L2, L1,y = 0),
                   (inbuilt(0.5, L2,L1)-Lagrange(0.5, L1, L2,y = 0))/inbuilt(0.5, L2,L1)],
                  ["3(b)",inbuilt(2.4, V,I),Lagrange(2.4, V,I),
                   (inbuilt(2.4, V,I)-Lagrange(2.4, V,I))/inbuilt(2.4, V,I)]])
print(mytable, end="\n\n")
xrange = np.linspace(0.0,3.0,100,float)
bes_inb,bes_inb_inv,lag_bes,lag_bes_inv,lin_inb,lin_inb_inv,lin_lag,lin_lag_inv= [],[],[],[],[],[],[],[]
for i in xrange:
  bes_inb.append(inbuilt(i, L1, L2))
  lag_bes.append(Lagrange(i, L1, L2))
x1range = np.linspace(0.0,6.0,100,float)
for i in x1range:
  lin_inb.append(inbuilt(i, V,I))
  lin_lag.append(Lagrange(i, V,I))
x2range = np.linspace(1.0,-0.3,100,float)
for i in x2range:
    bes_inb_inv.append(inbuilt(i, L2,L1))
    lag_bes_inv.append(Lagrange(i, L2, L1,y = 0))
x3range = np.linspace(2.0,6.5,100,float)
for i in x3range:
    lin_inb_inv.append(inbuilt(i, I,V))
    lin_lag_inv.append(Lagrange(i,I,V,y = 0))

fig,axs=plt.subplots(2,2,figsize=(15,8))
ax11,ax12,ax21,ax22=axs[0][0],axs[0][1],axs[1][0],axs[1][1]
ax11.plot(xrange,lag_bes,label="lagrange interpolation"),ax11.plot(xrange,bes_inb,label="inbuilt")
ax11.scatter(L1,L2,label="Given Data Points"),ax11.scatter(2.3,Lagrange(2.3, L1, L2),label="Interpolation Point",c='red')
ax11.set_title("Bessel Interpolation"),ax11.set_ylabel("f(x)")
ax12.plot(x2range,lag_bes_inv,label="Inverse lagrange interpolation"),ax12.plot(x2range,bes_inb_inv,label="inbuilt")
ax12.scatter(L2,L1,label="Given Data Points"),ax12.set_ylabel("x")
ax12.set_ylim([-0.5,3.5]), ax12.set_title("Bessel Inverse Interpolation")
ax21.plot(x1range,lin_lag,label="lagrange interpolation"),ax21.plot(x1range,lin_inb,label="inbuilt")
ax21.scatter(V,I,label="Given Data Points"),ax21.scatter(2.4,Lagrange(2.4, V, I),label="Interpolation Point",c='red')
ax21.set_title("V vs I Interpolation"),ax21.set_ylabel("V")
ax22.plot(x3range,lin_inb_inv,label="Inverse lagrange interpolation"),ax22.plot(x3range,lin_lag_inv,label="inbuilt")
ax22.scatter(I,V,label="Given Data Points"),ax22.set_title("I vs V Inverse Interpolation")
ax22.set_ylim([-10,10]),ax22.set_ylabel("I")
ax11.legend(),ax11.grid(True),ax12.legend(),ax12.grid(True),ax21.legend(),ax21.grid(True),ax22.legend(),ax22.grid(True)
plt.show()
```

```python

```

```python

```

```python

```

```python

```

```python

```


# [Learn Quantum Computing with Python and Q#](https://www.manning.com/books/learn-quantum-computing-with-python-and-q-sharp?a_aid=learn-qc-granade&a_bid=ee23f338)
Chapter 3 Exercise Solutions
----
> Copyright (c) Sarah Kaiser and Chris Granade.
> Code sample from the book "Learn Quantum Computing with Python and Q#" by
> Sarah Kaiser and Chris Granade, published by Manning Publications Co.
> Book ISBN 9781617296130.
> Code licensed under the MIT License.

### Preamble

```python
import numpy as np
```

### Exercise 3.1

**In Chapter 2, we saw that unitary matrices play the same role in quantum computing that _truth tables_ play in classical computing.
We can use that to figure out what the matrix 𝑋 has to look like in order to represent the quantum NOT operation, `x`.
Let's start by making a table of what the matrix 𝑋 has to do to each input state in order to represent what the `x` instruction does:**

| Input | Output |
|---|---|
| \|0⟩ | \|1⟩
| \|1⟩ | \|0⟩

**This table tells us that if we multiply the matrix 𝑋 by the vector |0⟩, we need to get |1⟩, and similarly that 𝑋|1⟩ = |0⟩.
Either by using NumPy or by hand, check that the matrix**

\begin{align}
X = \left(\begin{matrix}
    0 & 1 \\
    1 & 0
\end{matrix}\right)
\end{align}

**matches what we have in our truth table above.**

We can start by defining two variables, `ket0` and `ket1`, to store the vectors representing |0⟩ and |1⟩, respectively.

```python
ket0 = np.array([
    [1],
    [0]
])
ket1 = np.array([
    [0],
    [1]
])
```

Next, we can define a variable to store the matrix $X$.

```python
X = np.array([
    [0, 1],
    [1, 0]
])
```

Now, we can verify that `X @ ket0` is the same as `ket1` and vice versa:

```python
X @ ket0
```

    array([[0],
           [1]])

```python
X @ ket1
```

    array([[1],
           [0]])

----
### Exercise 3.2

**Try using what we learned about vectors in the previous chapter to verify that |0⟩ = (|\+⟩ + |−⟩) / √2, either by hand or using Python.**

*HINT*: recall that |+⟩ = (|0⟩ + |1⟩) / √2 and that |−⟩ = (|0⟩ − |1⟩) / √2.

As before,it's helpful to start by defining variables to represent the vectors |+⟩ and |−⟩.

```python
ket_plus = np.array([
    [1],
    [1]
]) / np.sqrt(2)
ket_minus = np.array([
    [1],
    [-1]
]) / np.sqrt(2)
```

Using these new variables, it's easy to verify that $|0\rangle = (|+\rangle + |-\rangle) / \sqrt{2}$.

```python
(ket_plus + ket_minus) / np.sqrt(2)
```

    array([[1.],
           [0.]])

----
### Exercise 3.3

Calculate what the probability of getting the measurement outcome |−⟩ when measuring the |0⟩ state in the |−⟩ direction.

Also calculate what the probability of getting the |−⟩ measurement outcome with the input state of |1⟩.

```python
ket0 = np.array([
    [1],
    [0]
])
ket1 = np.array([
    [0],
    [1]
])
```

```python
ket_plus = np.array([
    [1],
    [1]
]) / np.sqrt(2)
ket_minus = np.array([
    [1],
    [-1]
]) / np.sqrt(2)
```

With these variables defined, it's easy to compute Born's rule, $\Pr(- | 0) = |\langle - | 0 \rangle|^2$ by computing the inner product of $|-\rangle$ with $|0\rangle$ and taking the absolute value squared of the result.

```python
np.abs(ket_minus.conj().transpose() @ ket0) ** 2
```

    array([[0.5]])

Similarly, we can compute $\Pr(- | 1)$ using the same technique.

```python
np.abs(ket_minus.conj().transpose() @ ket1) ** 2
```

    array([[0.5]])

----
### Exercise 3.4

**If you had the ciphertext `10100101` and the key `00100110`, what was the message that was originally sent?**

```python
ciphertext = [bit == '1' for bit in '10100101']
key = [bit == '1' for bit in '00100110']
```

```python
plaintext = [cipher_bit ^ key_bit for (cipher_bit, key_bit) in zip(ciphertext, key)]
```

```python
"".join("1" if bit else "0" for bit in plaintext)
```

    '10000011'


# Chapter 10 - Position and Momentum
We can start using sympy to handle symbolic math (integrals and other calculus):

```python
from sympy import *
```

```python
init_printing(use_unicode=True)
```

```python
# SymPy works better if you specify what letters are symbols:
x, y, z = symbols('x y z', real=True)

# notice we can also put some restrictions on the symbols:
a, c = symbols('a c', nonzero=True, real=True)
```

```python
integrate?
```

There are two ways to use the `integrate` function. In one line, like `integrate(x,(x,0,1))` or by naming an expression and then integrating it over a range:

```
A = (c*cos((pi*x)/(2.0*a)))**2
A.integrate((x,-a,a),conds='none')
```

We'll use both, at different times. For longer expressions, the second form can be easier to read and write.

First, just try the following, then we'll re-create some examples in the book.

```python
integrate(x,(x,0,1))
```

```python
integrate(x**2,(x,0,1))
```

```python
A = (c*cos((pi*x)/(2.0*a)))**2
A.integrate((x,-a,a))
```

So this tells us the normalization constant should be $c=\frac{1}{\sqrt{a}}$. Check that it is normalized if we do that:

```python
psi = 1/sqrt(a)*cos((pi*x)/(2.0*a))  # notice we can name the expression something useful.
B = psi**2
B.integrate( (x,-a,a), conds='none')
```

Because `psi` is a real function, we can calculate expectation values by integrating over $x$ or $x^2$ with `psi**2`:

```python
C = x*psi**2
C.integrate( (x,-a,a), conds='none')
```

```python
D = x**2 * psi**2
E = D.integrate( (x,-a,a), conds='none')
E
```

```python
E.simplify()  # this is a useful method!
```

```python
E.n()  # the .n() method approximates the numerical part. You can look at the full expression below.
```

## Example 10.2

```python
h = Symbol('hbar', real=True, positive=True)
```

Use the `diff` function to take a derivative of a symbolic expression. For example:

```python
diff(x**2, x)
```

```python
# Solution
-1j*h*diff( 1/a*cos((pi*x)/(2*a)) ,x)
```

```python
# Solution
B1 = (pi*h/(2*a))**2 * (cos((pi*x)/(2*a)))**2
B1.integrate( (x,-a,a), conds='none' )
```

## Example 10.3

```python
p = Symbol('p', real=True)
```

```python
# Solution
A = integrate(1/sqrt(2*pi*a*h)*exp(-I*p*x/h)*cos((pi*x)/(2*a)),(x,-a,a), conds='none')
```

```python
# Solution
A
```

```python
psi_p = sqrt(2*a*pi/h) * 2/(pi**2 - (2*p*a/h)**2) * cos(p*a/h)
psi_p
```

```python
psi_p == sqrt(2*a*pi/h)*2/(pi**2 - (2*p*a/h)**2) * cos(p*a/h)
```

    True

Which agrees with the book.

This is about as far as we can go in sympy. Unfortunately, many other momentum integrals choke. There are a few hints to get through the rest here:

## Problem 10.3

```python
x, y, z = symbols('x y z', real=True)
a, c = symbols('a c', nonzero=True, real=True, positive=True)

psi = c*1/(a**2 + x**2)  # define the wavefunction with c constant
int1 = integrate(psi*psi,(x,-oo,oo), conds='none')  # integrate psi^2
solutions = solve(int1 - 1,c)  # solve for c, this returns a list of solutions
c2 = simplify(solutions[0])  # simplify the solution for c:
c2
```

```python
psi2 = c2/c*psi
psi2
```

```python
integrate(psi2 * x * psi2,(x,-oo,oo))
```

```python
integrate(psi2 * x**2 * psi2,(x,-oo,oo))
```

So $\Delta x^2 = a^2 - 0^2$ therefore $\Delta x = a$

## Problem 10.17:
Now find the momentum representation of the state from 10.3

```python
p = symbols('p', nonzero=True, real=True, positive=True)
B = integrate(sqrt(1/(2*pi*h))*exp(-I*p*x/h)*psi2,(x,-oo,oo))
B
```

```python
B.simplify()
```

This agrees with the book after we notice that we had to force $p$ to be positive in order to get the integral to converge. The book has $|p|$ in the argument of the exponent to reflect this constraint.

```python

```


# Lecture 12

## Linear Algebra I:

### Introduction to Matrices

```python
import numpy as np
import sympy as sp

##################################################
##### Matplotlib boilerplate for consistency #####
##################################################
from ipywidgets import interact
from ipywidgets import FloatSlider
from matplotlib import pyplot as plt

%matplotlib inline

from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')

global_fig_width = 10
global_fig_height = global_fig_width / 1.61803399
font_size = 12

plt.rcParams['axes.axisbelow'] = True
plt.rcParams['axes.edgecolor'] = '0.8'
plt.rcParams['axes.grid'] = True
plt.rcParams['axes.labelpad'] = 8
plt.rcParams['axes.linewidth'] = 2
plt.rcParams['axes.titlepad'] = 16.0
plt.rcParams['axes.titlesize'] = font_size * 1.4
plt.rcParams['figure.figsize'] = (global_fig_width, global_fig_height)
plt.rcParams['font.sans-serif'] = ['Computer Modern Sans Serif', 'DejaVu Sans', 'sans-serif']
plt.rcParams['font.size'] = font_size
plt.rcParams['grid.color'] = '0.8'
plt.rcParams['grid.linestyle'] = 'dashed'
plt.rcParams['grid.linewidth'] = 2
plt.rcParams['lines.dash_capstyle'] = 'round'
plt.rcParams['lines.dashed_pattern'] = [1, 4]
plt.rcParams['xtick.labelsize'] = font_size
plt.rcParams['xtick.major.pad'] = 4
plt.rcParams['xtick.major.size'] = 0
plt.rcParams['ytick.labelsize'] = font_size
plt.rcParams['ytick.major.pad'] = 4
plt.rcParams['ytick.major.size'] = 0
##################################################
```

## Simultaneous equations

Consider 2 simultaneous equations:
\begin{eqnarray}
a_1x+b_1y &=& c_1, \quad (1)\\
a_2x+b_2y &=& c_2, \quad (2)
\end{eqnarray}

where the values $\;x\;$ and $\;y\;$ are to be found, and $\;a_1, \;b_1, \;a_2, \;b_2, \;c_1\;$ and $\;c_2\;$ are given constants.

\begin{eqnarray}
 (1) \times b_2:~~~~~~~~~~~~~~~ b_2a_1x+b_2b_1y &=& b_2c_1, \quad (3)\\
  (2) \times b_1:~~~~~~~~~~~~~~~ b_1a_2x+b_1b_2y &=& b_1c_2, \quad (4)\\
  (3) - (4):~~~~~~~~~~~~~~~ b_2a_1x-b_1a_2x &=& b_2c_1-b_1c_2.
\end{eqnarray}

Thus:

$$x=\frac{b_2c_1-b_1c_2}{b_2a_1-b_1a_2},$$

and similarly:

$$y=\frac{a_1c_2-a_2c_1}{a_1b_2-a_2b_1}.$$

This works, provided that $a_1b_2-a_2b_1\neq 0.$

While the algebraic manipulation is straightforward when solving two equations, it can get really messy when solving large systems.

What we want is a way to be able to easily manipulate **linear** systems, regardless of how big they are.

## The Matrix
Matrices are a structure that allow us to more easily manipulate linear systems.

Consider the original system

\begin{align}
a_1x+b_1y &= c_1, \\
a_2x+b_2y &= c_2.
\end{align}

We rewrite this, in the form of a matrix as:

\begin{equation}
\left(\begin{matrix}a_1&b_1\\ a_2&b_2\end{matrix}\right)
\left(\begin{matrix}x\\y\end{matrix}\right)
=\left(\begin{matrix}c_1\\ c_2 \end{matrix}\right).
\label{eq:mat}
\end{equation}

Think about how this form relates to the original linear system.

## What is a matrix?

A matrix is an array of numbers such as:

$$\left(\begin{matrix}a&b&c\\ d&e&f\\ g&h&i\end{matrix}\right).$$

$3\times3$ is the **size** of the matrix.

A $3\times3$ matrix is said to be **square** and have **order** (dimension) 3.

### Addition, subtraction, and scalar multiplication

We can add or subtract two matrices as long as they have the **same** size:

$$\left(\begin{matrix} 2&1  \\ 3&-4 \end{matrix}\right)
  +\left(\begin{matrix} 6&-5 \\ 1&-7 \end{matrix}\right)=
   \left(\begin{matrix} 8&-4 \\ 4&-11\end{matrix}\right).$$

Multiply a matrix by a scalar:

$$5\times\left(\begin{matrix} 2&1\\ 3&-4 \end{matrix}\right)
    =\left(\begin{matrix}10&5\\ 15&-20\end{matrix}\right).$$

### Matrix multiplication
To multiply two matrices, we multiply each **row** in the first matrix by each **column** in the second one, and put the results into a new matrix.

A row and column are multiplied by summing up each element in the row, multiplied by the corresponding element in the column.

$$\left(\begin{matrix} 1&2 \\ 3&4 \end{matrix}\right) \left(\begin{matrix} 5&6\\7&8\end{matrix}\right) = \left(\begin{matrix} 1 \times 5 + 2 \times 7 & 1 \times 6 + 2 \times 8 \\ 3 \times 5 + 4 \times 7 & 3 \times 6 + 4\times 8\end{matrix}\right) = \left(\begin{matrix} 19&22\\43&46\end{matrix}\right)$$

$$\left(\begin{matrix} 1&2&3&4\\ 5&6&7&8                \end{matrix}\right)
    \left(\begin{matrix} 1&2&3\\ 4&5&6\\ 7&8&9\\ 10&11&12 \end{matrix}\right)
=   \left(\begin{matrix} 70&80&90\\ 158&184&210            \end{matrix}\right),$$

$$
 \left(\begin{matrix} a & b & c                \end{matrix}\right)
 \left(\begin{matrix} p & q  \\ r & s \\ v & w \end{matrix}\right)
= \left(\begin{matrix} ap+br+cv & aq+bs+cw \end{matrix}\right).
$$

If the number of **columns** in the first matrix doesn't match the number of **rows** in the second, they **cannot** be multiplied.

$$\left(\begin{matrix} 2 & 3 & 1 \\ 2 & -1 & 3\end{matrix}\right)\left(\begin{matrix} 1 & 0 \\ -1 & -4\end{matrix}\right) =\;?\;?\;?$$

### Matrix multiplication is not commutative
This can be easily seen from the fact that multiplying different sized matrices doesn't always work:

$(3 x 2 \rm{matrix}) \times (2 x 2 \rm{matrix}) = (3 x 2 \rm{matrix})$

$(2 x 2 \rm{matrix}) \times (3 x 2 \rm{matrix}) = ???$

However, even when sizes match, the product is usually not the same.

### Example: Show that

$\left(\begin{matrix} 4 & 0 & 2 & 3 \\ 1 & 0 & 5 & 2\end{matrix}\right)\left(\begin{matrix} 6 & 0 \\ 2 & 1 \\ 3 & 0 \\ 3 & 10 \end{matrix}\right) \neq \left(\begin{matrix} 6 & 0 \\ 2 & 1 \\ 3 & 0 \\ 3 & 10 \end{matrix}\right)\left(\begin{matrix} 4 & 0 & 2 & 3 \\ 1 & 0 & 5 & 2\end{matrix}\right)$

### The identity matrix
**I** is the identity matrix, which has the property that:

$A I = I A = A$

for a square matrix $A$. It is the matrix equivalent of multiplying by 1.

The 2x2 identity matrix is:

$I_2 = \left(\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}\right).$

The 3x3 identity matrix is:

$I_3 = \left(\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix}\right).$

and so on for higher dimensions.

### The determinant
If

$A = \left(\begin{matrix} p & q \\ r & s\end{matrix}\right)$

then the **determinant** of A is:

$|A| = ps-qr$

That is, (top left $\times$ bottom right) - (top right $\times$ bottom left).

If $|A| = 0$, A is said to be **singular** (have no inverse).

### Inverting 2x2 matrices

If $AB = I$, what is B?

A is called the inverse of B, and vice versa. I.e.
$A = B^{-1}, B = A^{-1}$.

If
$A = \left(\begin{matrix} p & q \\ r & s\end{matrix}\right)$

then:

$A^{-1} = \frac{1}{ps-qr} \left(\begin{matrix} s & -q \\ -r & p\end{matrix}\right)$.

## Example - Inverting a 2x2 matrix

$$A=\left(\begin{matrix}2&-3\\ -2&4\end{matrix}\right).$$

$$|A|=(2\times 4)-(-3\times-2)=8-6=2.$$

 so

$$A^{-1}={1\over 2}\left(\begin{matrix}4&3\\ 2&2\end{matrix}\right)$$

As a check, calculate $A^{-1}A$:

$$A^{-1}A= \frac{1}{2}\left(\begin{matrix}4&3\\ 2&2\end{matrix}\right)\left(\begin{matrix}2&-3\\ -2&4\end{matrix}\right) $$

$$= \frac{1}{2}\left(\begin{matrix}2&0\\ 0&2\end{matrix}\right)$$

$$= \left(\begin{matrix}1&0\\ 0&1\end{matrix}\right)$$

$$=I_2.$$

## The transpose of a Matrix

$A^T$ is the transpose of $A$.

 Swap elements across the leading diagonal so that $A^T_{ij}= A_{ji}$.

 $$A=\left(\begin{matrix}2&1&2\\ 1&4&6\\ 1&-1&2\end{matrix}\right)$$

 $$A^T=\left(\begin{matrix}2&1&1\\ 1&4&-1\\ 2&6&2\end{matrix}\right)$$

## Solving a linear system using matrices

To solve a matrix system $A {\bf x} = {\bf b}$ for an unknown left-hand side ${\bf x}$.

- If it's of order 2 then use the formula to write $A^{-1}$ and hence ${\bf x} = A^{-1}{\bf b}$.

- If it's larger $(3\times3)$ then there's still a formula for  $A^{-1}$ (not in this course).

- Use an analytical method (Gaussian elimination) to find the inverse (not in this course).

- Use a numerical scheme to find an approximation to ${\bf x}$, such as Newton's method (not in this course).

- Solve using a linear algebra package.

### Example. Solving a 2x2 system

$$A^{-1}A{\bf x}=A^{-1}{\bf b}$$

 and  $${\bf x}=A^{-1}{\bf b}$$

 e.g.

$$x+5y  = 11,$$

$$-x+5y = 9.$$

In matrix form, this gives:

$$\left(\begin{matrix}1 &5\\ -1&5\end{matrix}\right) \left(\begin{matrix}x\\ y\end{matrix}\right) = \left(\begin{matrix}11\\ 9\end{matrix}\right)$$

 We have:

$$A^{-1}= \frac{1}{10} \left(\begin{matrix}5 &-5\\ 1&1 \end{matrix}\right)$$

 Thus:

$$\left(\begin{matrix}x\\ y\end{matrix}\right) = \frac{1}{10}\left(\begin{matrix}5 &-5\\ 1&1\end{matrix}\right)\left(\begin{matrix}11\\ 9\end{matrix}\right) =\frac{1}{10} \left(\begin{matrix}10\\ 20\end{matrix}\right)$$

$$=\left(\begin{matrix}1\\ 2\end{matrix}\right)$$

 Thus $x=1, y=2$

This process seems like more effort than its worth for small systems.

But it allows for a much more systematic approach when dealing with large systems.

As the size of the matrix grows, this process can be easily performed with Python (or other tools).

### Example. Solving a 4x4 system in Python

$ x + 5y + 3z - w = 5,$

$ x - 2y + z + 4w = 2,$

$ -3x + y -z + 2w = -5,$

$ x + y + z = 0.$

This gives

$\left(\begin{matrix} 1 & 5 & 3 & -1 \\ 1 & -2 & 1 & 4 \\ -3 & 1 & -1 & 2\\ 1 & 1 & 1 & 0 \end{matrix}\right) \left(\begin{matrix} x \\ y \\ z \\ w\end{matrix}\right) = \left(\begin{matrix} 5 \\ 2 \\ -5 \\ 0\end{matrix}\right).$

```python
## In python, we use numpy arrays to store the needed matrices
## the procedure linalg.solve, solves the system Ax = b
## We could also calculate the inverse of A (linalg.inv), and then multiply.
## But this is faster
A = np.array([[1,5,3,-1],[1,-2,1,4],[-3,1,-1,2],[1,1,1,0]])

b = np.array([5, 2, -5, 0])

x = np.linalg.solve(A,b)

print(x)

print(np.matmul(A,x))
```

```python
A = sp.Matrix([[1,5,3,-1],[1,-2,1,4],[-3,1,-1,2],[1,1,1,0]])

A.inv() * sp.Matrix([5, 2, -5, 0])

```


```
from sympy import pi, var, S, init_printing, log, Function, sqrt
init_printing(use_latex=True)
```

# Exchange and Correlation Potential

The basic relation for any LDA potential is the contribution to the total/free energy:

$$
    F_{xc}[n_e]
        = \int n_e(\mathbf{x}) \epsilon_{xc}^{LD}(n_e) d^3 x\,.
$$
From this we calculate the XC potential $V_{xc}(\mathbf{x})$ as a functional derivative (this follows from minimizing the total/free energy):
$$
V_{xc}(\mathbf{x}) \equiv {\delta F_{xc}[n_e] \over \delta n_e} = \epsilon_{xc}^{LD}(n_e) + n_e(\mathbf{x})
            {d \epsilon_{xc}^{LD}(n_e) \over d n_e}
= {d \over d n_e}\left(n_e \epsilon_{xc}^{LD}(n_e)\right)
$$
The $n_e(\mathbf{x})$ is the (positive) electronic particle density, $\epsilon_{xc}^{LD}(n_e)$ is the XC energy density and $V_{xc}(\mathbf{x})$ is the XC potential.

Sometimes the potential can be decomposed into exchange and correlation parts as follows:
$$\epsilon_{xc}^{LD}(n_e)= \epsilon_{x}^{LD}(n_e) + \epsilon_{c}^{LD}(n_e)$$

## Exchange part

$$\epsilon_{x}^{LD}(n_e) = -{3\over 4\pi} (3 \pi^2 n_e)^{1\over 3}$$

```
var("n")
ex = -3/(4*pi) * (3*pi**2*n)**(S(1)/3)
```

The corresponding $V_x$ is then calculated as:

```
Vx = (n*ex).diff(n)
Vx
```

Finally, we notice:

```
Vx/ex
```

So we have:
$$V_{x}(\mathbf{x}) = {4\over 3}\epsilon_{x}^{LD}(n_e)$$
So this show that the exchange part can be calculated using the following Fortran statements:

    ex = -3/(4*pi) * (3*pi**2*n)**(1.0_dp/3)
    Vx = 4*ex/3

## Correlation part of Perdew and Zunger (PZ)

The following is taken from [1]. The constants are:

$$
\begin{eqnarray*}
\gamma &=& -0.1423 \\
\beta_1 &=& 1.0529 \\
\beta_2 &=& 0.3334 \\
A &=&  0.0311 \\
B &=& -0.048 \\
C &=&  0.0020 \\
D &=& -0.0116 \\
\end{eqnarray*}
$$

[1] Perdew, J. P., & Zunger, A. (1981). Self-interaction correction to
density-functional approximations for many-electron systems. Physical Review
B, 23(10), 5048–5079.

```
rs = (3/(4*pi*n))**(S(1)/3)
rs
```

### Case $r_s < 1$

```
var("A B C D")
ec = A*log(rs) + B + C*rs*log(rs) + D*rs
ec
```

```
Vc = (n*ec).diff(n).expand()
Vc
```

Which is a little complicated, so we try the following approach:

```
rs.diff(n) / rs
```

So ${d r_s \over d n} = - {1\over 3n} r_s$.

```
rs = Function("rs")(n)
ec = A*log(rs) + B + C*rs*log(rs) + D*rs
ec
```

```
Vc = (n*ec).diff(n).expand()
Vc
```

Now we substitute for the derivative:

```
Vc = Vc.subs(rs.diff(n), (-1/(3*n))*rs)
Vc
```

So this part of $V_c$ can be calculated using the following Fortran statements:

    rs = (3/(4*pi*n))**(1.0_dp/3)
    ec = A*log(rs) + B + C*rs*log(rs) + D*rs
    Vc = A*log(rs) + (B-A/3) + 2*C*rs*log(rs)/3 + (2*D-C)*rs/3

### Case $r_s \ge 1$

```
var("gamma beta1 beta2")
ec = gamma / (1+beta1*sqrt(rs)+beta2*rs)
ec
```

```
Vc = (n*ec).diff(n)
Vc
```

```
Vc = Vc.subs(rs.diff(n), (-1/(3*n))*rs)
Vc
```

We express $V_c$ as $V_c = e_c \cdot C$. Arguably the simplest expression for $C$ is:

```
C = (Vc / ec).simplify()
C
```

But in literature the following is used (both numerator and denumerator must be divided by 6 below):

```
C.normal()
```

So the following Fortran statements implement that:

    ec = gam / (1+beta1*sqrt(rs)+beta2*rs)
    Vc = ec * (1+7*beta1*sqrt(rs)/6 + 4*beta2*rs/3) / (1+beta1*sqrt(rs) + beta2*rs)

## Fortran Subroutine

The final Fortran subroutine is:

    subroutine xc_pz(n, exc, Vxc)
    ! Calculates XC LDA density and potential from the charge density "n".
    ! Uses the Perdew Zunger [1] parametrization.
    !
    ! [1] Perdew, J. P., & Zunger, A. (1981). Self-interaction correction to
    ! density-functional approximations for many-electron systems. Physical Review
    ! B, 23(10), 5048–5079.
    real(dp), intent(in) :: n ! charge density
    real(dp), intent(out) :: exc ! XC density
    real(dp), intent(out) :: Vxc ! XC potential

    real(dp), parameter :: gam = -0.1423_dp
    real(dp), parameter :: beta1 = 1.0529_dp
    real(dp), parameter :: beta2 = 0.3334_dp
    real(dp), parameter :: A =  0.0311_dp
    real(dp), parameter :: B = -0.048_dp
    real(dp), parameter :: C =  0.0020_dp
    real(dp), parameter :: D = -0.0116_dp
    real(dp) :: ex, ec, Vx, Vc, rs

    if (n == 0) then
        exc = 0
        Vxc = 0
        return
    end if

    ex = -3/(4*pi) * (3*pi**2*n)**(1.0_dp/3)
    Vx = 4*ex/3

    rs = (3/(4*pi*n))**(1.0_dp/3)
    if (rs >= 1) then
        ec = gam / (1+beta1*sqrt(rs)+beta2*rs)
        Vc = ec * (1+7*beta1*sqrt(rs)/6 + 4*beta2*rs/3) / &
            (1+beta1*sqrt(rs) + beta2*rs)
    else
        ec = A*log(rs) + B + C*rs*log(rs) + D*rs
        Vc = A*log(rs) + (B-A/3) + 2*C*rs*log(rs)/3 + (2*D-C)*rs/3
    end if

    exc = ex + ec
    Vxc = Vx + Vc
    end subroutine


```python
import sympy as sp
sp.init_printing()
```

```python
eps = sp.Symbol("epsilon")
lam = sp.Symbol("lambda")
```

```python
m = sp.Matrix([[1, eps], [eps, 1]])
m
```

```python
m.charpoly(lam)
```

Observe the occurrence of $(1-\epsilon^2)$ above.

eigenvalue/Rounding in characteristic polynomial using SymPy.ipynb


# A simple epidemic model for the spread of SARS-CoV-2

This is a simple box model that projects the infections, recoveries, and death toll of an epidemic. It is a simple 'box model' that is a member of the family of models known in epidemiology as SIR-models (for Susceptible, Infected, Recovered). We will also calculate the projected death toll, hence making this a SIRD-model. It can be (cautiously) applied to the current coronavirus pandemic by using current case counts and best estimate infection, recover, hospitalisation, and mortality rates. The model is meant to help build public awareness why __flattening the curve__ by __social distancing__ is effective and therefore important to heed. It comes with the warning that the following script was developed by a non-expert and does _NOT_ claim to make accurate predictions for the ongoing crisis.

The model solves the following set of non-linear ordinary differential equations:

\begin{align}
    \dfrac{dS(t)}{dt} &= - \beta S(t) I(t) \ , \tag{1a} \\
    \dfrac{dI(t)}{dt} &= \hspace{6pt} \beta S(t) I(t) - \gamma I(t) \ , \tag{1b} \\
    \dfrac{dR(t)}{dt} &= \hspace{49pt} \gamma I(t) (1-\delta) \ , \tag{1c} \\
    \dfrac{dD(t)}{dt} &= \hspace{49pt} \gamma I(t) \hspace{18pt} \delta \ , \tag{1d}
\end{align}
where $S$ is the population size of people susceptible to infection (not previously immunised and within social contact with infected population), $I$ is the currently infected population (some asymptomatic, some severe requiring hospitalisation), $R$ is the recovered population that is now immune and will not further contribute to infections, and $D$ the death toll. All variables are normalised by the total population size, $P$, such that $S+I+R+D = 1$. The epidemic model is governed by three parameters, the infection rate, $\beta$, the recovery rate, $\gamma$, and the mortality rate parameter, $\delta$.

The system can be characterised by the epidemic number $\mathrm{R}$,
\begin{equation}
    \mathrm{R}_E = \dfrac{\beta S}{\gamma} \ , \tag{2}
\end{equation}
a dimensionless number which is $>1$ when infection rates exceed recovery and mortality rates and therefore the epidemic is spreading, and $<0$ when the situation is reversed and the epidemic is retreating.

Two further dimensionless numbers can be derived from it, the recovery number $\mathrm{R}_R$, and the mortality number $\mathrm{R}_D$,
\begin{align}
    \mathrm{R}_R &= (1-\delta)\mathrm{R}^{-1} = \dfrac{\gamma (1-\delta)}{\beta S} \ , \tag{3a} \\
    \mathrm{R}_I &= \hspace{18pt} \delta \ \mathrm{R}^{-1} = \dfrac{\gamma \delta}{\beta S} \ , \tag{3b} \\
\end{align}
which quantify the rates of recovery and mortality relative to new infections, respectively.

We will implement the equations using a simple explicit finite-difference scheme, where we discretise continuous time $t$ into $n$ discrete points in time, $t^k$, $k=1,...,n$, spaced at equal time steps $\Delta t = t^k - t^{k-1} = 1$ day. The time derivatives are discretised following the pattern,
\begin{equation}
    \dfrac{d(\cdot)}{dt} = \dfrac{(\cdot)^k - (\cdot)^{k-1}}{\delta t} \ . \tag{4}
\end{equation}
The discretised equations then are rearranged to yield explicit updates for the solution variables,
\begin{align}
    S^k &= S^{k-1} - \beta S^{k-1} I^{k-1} \Delta t \ , \tag{5a} \\
    I^k &= I^{k-1} + (\beta S^{k-1} I^{k-1} - \gamma I^{k-1}) \Delta t \ , \tag{5b} \\
    R^k &= R^{k-1} + \gamma I^{k-1} \Delta t \ , \tag{5c} \\
    D^k &= D^{k-1} + \gamma I^{k-1} \Delta t \ . \tag{5d}
\end{align}

```python
# import modules for numerical calculations and for plotting
import numpy as np
import matplotlib.pyplot as plt

# tell python to allow plotting within the page
%matplotlib inline

# customise figure style to use font size 16
from matplotlib import rcParams
rcParams['font.size'] = 16
```

```python
# create function for corona virus epidemic model
def SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, f_dst, f_rst, f_snl, f_str, f_dcy, t_end):

    # set duration of model and create time line
    Time     = np.arange(0,int(t_end+1))  # model time line in days

    # create arrays to store results
    S      = np.zeros(np.size(Time))  # susceptible population
    I      = np.zeros(np.size(Time))  # infected population
    R      = np.zeros(np.size(Time))  # recovered population
    D      = np.zeros(np.size(Time))  # death toll
    RI     = np.zeros(np.size(Time))  # infection rate
    RO     = np.zeros(np.size(Time))  # outcome rate (recovery or death)
    RM     = np.zeros(np.size(Time))  # mortality rate

    # set initial conditions
    S[0]   = (P0*f_rst - I0)/P0  # initial susceptible population
    I[0]   = I0/P0               # initial infected population
    R[0]   = R0/P0               # initial recovered population
    D[0]   = D0/P0               # initial death toll
    RI[0]  = beta*f_dst * (1 - f_snl*np.sin(0))  # initial infection rate
    RO[0]  = gamma                               # initial outcome rate
    RM[0]  = delta*(1 + f_str*I[0]/(C0/P0))      # initial mortality rate

    # calculate epidemic SIRD-model
    # dS/dt = -RI.S.I
    # dI/dt =  RI.S.I - RR.I
    # dR/dt =           RR.I.(1-R)
    # dD/dt =           RR.I.   R
    for t in Time[1:]:
        # update variables
        S[t] = S[t-1] - RI[t-1]*S[t-1]*I[t-1]                              # update susceptible population
        I[t] = I[t-1] + RI[t-1]*S[t-1]*I[t-1] - RO[t-1]*I[t-1]             # update infected population
        R[t] = R[t-1] +                         RO[t-1]*I[t-1]*(1-RM[t-1]) # update recovered population
        D[t] = D[t-1] +                         RO[t-1]*I[t-1]*   RM[t-1]  # update death toll

        # update rates of change
        RI[t] = beta*f_dst * (1 + f_snl*(np.cos(2*np.pi/365*t)-1))  # update infection rate
        RO[t] = gamma                                                # update outcome rate
        RM[t] = delta*(1 + f_str*I[t-1]/(C0/P0) - f_dcy/365*t)       # update mortality rate

    return Time, S, I, R, D, RI, RO, RM

mega  = 1e6
kilo  = 1e3
shift = 30-0
```

## Model projection for UK

This projection is based on the latest case count and best estimate growth and mortality rates for the United Kingdom.

```python
# set initial condition and parameter values for the model
I0    = 25150*10# current infected count
R0    = 135*20 # current recovered count
D0    = 1789   # current fatality count
P0    = 66.4e6 # population size
C0    = 1.5e5  # bed capacity of health care system
beta  = 0.14   # estimated infection rate parameter
gamma = 1/14   # estimated recovery rate parameter
delta = 0.005  # estimated mortality rate parameter
r_hsp = 0.05   # estimated hospitalisation rate
f_dst = 1.00   # factor for reduction in infection rate due to social distancing
f_rst = 1.00   # factor for reduction in effective population size due to travel restrictions
f_snl = 0.00   # increase seasonality of infection rate (>0 for norther, <0 for southern hemisphere)
f_str = 0.02   # factor for increase in mortality rate for health service at/over capacity
f_dcy = 0.50   # factor for increase in yearly decay of mortality rate (weaker form of virus spread further)
t_end = 365    # model duration in days

# run the model with the above choices
Time, S_1, I_1, R_1, D_1, RI_1, RO_1, RM_1 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 1.00, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_2, I_2, R_2, D_2, RI_2, RO_2, RM_2 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.80, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_3, I_3, R_3, D_3, RI_3, RO_3, RM_3 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.60, f_rst, f_snl, f_str, f_dcy, t_end)

# calculate dimensionless numbers
RE_1 = RI_1*S_1/RO_1
RE_2 = RI_2*S_2/RO_2
RE_3 = RI_3*S_3/RO_3

RR_1 = (1-RM_1)*RO_1/RI_1/S_1
RR_2 = (1-RM_2)*RO_2/RI_2/S_2
RR_3 = (1-RM_3)*RO_3/RI_3/S_3

RD_1 =    RM_1 *RO_1/RI_1/S_1
RD_2 =    RM_2 *RO_2/RI_2/S_2
RD_3 =    RM_3 *RO_3/RI_3/S_3

fig1 = plt.figure(1,figsize=(10,12))  # create figure to plot in

plt.subplot(3,1,1)
plt.plot(Time,S_1*P0/mega,'k-' ,Time,I_1*P0/mega,'b-' ,Time,R_1*P0/mega,'g-' , \
         Time,S_2*P0/mega,'k--',Time,I_2*P0/mega,'b--',Time,R_2*P0/mega,'g--', \
         Time,S_3*P0/mega,'k-.',Time,I_3*P0/mega,'b-.',Time,R_3*P0/mega,'g-.')
plt.ylabel('Cases [million]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['susceptible','infected','recovered'],loc=7)
plt.title('United Kingdom')

plt.subplot(3,1,2)
plt.plot(Time,I_1*r_hsp*P0/mega,'m-' ,Time,D_1*P0/mega,'r-' ,[Time[0],Time[-1]],[C0/mega,C0/mega],'k:', \
         Time,I_2*r_hsp*P0/mega,'m--',Time,D_2*P0/mega,'r--', \
         Time,I_3*r_hsp*P0/mega,'m-.',Time,D_3*P0/mega,'r-.')
plt.ylabel('Cases [million]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['severe','fatal','capacity'],loc=7);

plt.subplot(3,1,3)
plt.plot(Time,RE_1,'c-' ,Time,RM_1*100,'y-' , \
         Time,RE_2,'c--',Time,RM_2*100,'y--', \
         Time,RE_3,'c-.',Time,RM_3*100,'y-.')
plt.ylabel('Dim.less number [1]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['epidemic number','mortality x 100'],loc=7);

plt.savefig("UK-1April.png",dpi=600)
```

## Model projection for CH

This projection is based on the latest case count and best estimate growth and mortality rates for Switzerland.

```python
# set initial condition and parameter values for the model
I0    = 16605*5 # current infected count
R0    = 2967*5  # current recovered count
D0    = 433     # current fatality count
P0    = 8.7e6   # population size
C0    = 2e4     # bed capacity of health care system
beta  = 0.12    # estimated infection rate parameter
gamma = 1/14    # estimated recovery rate parameter
delta = 0.005   # estimated mortality rate parameter
r_hsp = 0.05    # estimated hospitalisation rate
f_dst = 1.00    # factor for reduction in infection rate due to social distancing
f_rst = 1.00    # factor for reduction in effective population size due to travel restrictions
f_snl = 0.00    # increase seasonality of infection rate (>0 for norther, <0 for southern hemisphere)
f_str = 0.02    # factor for increase in mortality rate for health service at/over capacity
f_dcy = 0.50    # factor for increase in yearly decay of mortality rate (weaker form of virus spread further)
t_end = 365     # model duration in days

# run the model with the above choices
Time, S_1, I_1, R_1, D_1, RI_1, RO_1, RM_1 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 1.00, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_2, I_2, R_2, D_2, RI_2, RO_2, RM_2 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.80, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_3, I_3, R_3, D_3, RI_3, RO_3, RM_3 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.625, f_rst, f_snl, f_str, f_dcy, t_end)

# calculate dimensionless numbers
RE_1 = RI_1*S_1/RO_1
RE_2 = RI_2*S_2/RO_2
RE_3 = RI_3*S_3/RO_3

RR_1 = (1-RM_1)*RO_1/RI_1/S_1
RR_2 = (1-RM_2)*RO_2/RI_2/S_2
RR_3 = (1-RM_3)*RO_3/RI_3/S_3

RD_1 =    RM_1 *RO_1/RI_1/S_1
RD_2 =    RM_2 *RO_2/RI_2/S_2
RD_3 =    RM_3 *RO_3/RI_3/S_3

fig1 = plt.figure(1,figsize=(10,12))  # create figure to plot in

plt.subplot(3,1,1)
plt.plot(Time,S_1*P0/mega,'k-' ,Time,I_1*P0/mega,'b-' ,Time,R_1*P0/mega,'g-' , \
         Time,S_2*P0/mega,'k--',Time,I_2*P0/mega,'b--',Time,R_2*P0/mega,'g--', \
         Time,S_3*P0/mega,'k-.',Time,I_3*P0/mega,'b-.',Time,R_3*P0/mega,'g-.')
plt.ylabel('Cases [million]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['susceptible','infected','recovered'],loc=7)
plt.title('Switzerland')

plt.subplot(3,1,2)
plt.plot(Time,I_1*r_hsp*P0/kilo,'m-' ,Time,D_1*P0/kilo,'r-' ,[Time[0],Time[-1]],[C0/kilo,C0/kilo],'k:', \
         Time,I_2*r_hsp*P0/kilo,'m--',Time,D_2*P0/kilo,'r--', \
         Time,I_3*r_hsp*P0/kilo,'m-.',Time,D_3*P0/kilo,'r-.')
plt.ylabel('Cases [thousand]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['severe','fatal','capacity'],loc=7);

plt.subplot(3,1,3)
plt.plot(Time,RE_1,'c-' ,Time,RM_1*100,'y-' , \
         Time,RE_2,'c--',Time,RM_2*100,'y--', \
         Time,RE_3,'c-.',Time,RM_3*100,'y-.')
plt.ylabel('Dim.less number [1]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['epidemic number','mortality x 100'],loc=7);

plt.savefig("CH-1April.png",dpi=600)
```

## Model projection for IT

This projection is based on the latest case count and best estimate growth and mortality rates for Italy.

```python
# set initial condition and parameter values for the model
I0    = 105792*8 # current infected count
R0    = 15729*8  # current recovered count
D0    = 12428    # current fatality count
P0    = 60.5e6   # population size
C0    = 2e5      # bed capacity of health care system
beta  = 0.12     # estimated infection rate parameter
gamma = 1/14     # estimated recovery rate parameter
delta = 0.005    # estimated mortality rate parameter
r_hsp = 0.05     # estimated hospitalisation rate
f_dst = 1.00     # factor for reduction in infection rate due to social distancing
f_rst = 1.00     # factor for reduction in effective population size due to travel restrictions
f_snl = 0.00     # increase seasonality of infection rate (>0 for norther, <0 for southern hemisphere)
f_str = 0.02     # factor for increase in mortality rate for health service at/over capacity
f_dcy = 0.50     # factor for increase in yearly decay of mortality rate (weaker form of virus spread further)
t_end = 365      # model duration in days

# run the model with the above choices
Time, S_1, I_1, R_1, D_1, RI_1, RO_1, RM_1 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 1.00, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_2, I_2, R_2, D_2, RI_2, RO_2, RM_2 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.80, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_3, I_3, R_3, D_3, RI_3, RO_3, RM_3 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.60, f_rst, f_snl, f_str, f_dcy, t_end)

# calculate dimensionless numbers
RE_1 = RI_1*S_1/RO_1
RE_2 = RI_2*S_2/RO_2
RE_3 = RI_3*S_3/RO_3

RR_1 = (1-RM_1)*RO_1/RI_1/S_1
RR_2 = (1-RM_2)*RO_2/RI_2/S_2
RR_3 = (1-RM_3)*RO_3/RI_3/S_3

RD_1 =    RM_1 *RO_1/RI_1/S_1
RD_2 =    RM_2 *RO_2/RI_2/S_2
RD_3 =    RM_3 *RO_3/RI_3/S_3

fig1 = plt.figure(1,figsize=(10,12))  # create figure to plot in

plt.subplot(3,1,1)
plt.plot(Time,S_1*P0/mega,'k-' ,Time,I_1*P0/mega,'b-' ,Time,R_1*P0/mega,'g-' , \
         Time,S_2*P0/mega,'k--',Time,I_2*P0/mega,'b--',Time,R_2*P0/mega,'g--', \
         Time,S_3*P0/mega,'k-.',Time,I_3*P0/mega,'b-.',Time,R_3*P0/mega,'g-.')
plt.ylabel('Cases [million]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['susceptible','infected','recovered'],loc=7)
plt.title('Italy')

plt.subplot(3,1,2)
plt.plot(Time,I_1*r_hsp*P0/mega,'m-' ,Time,D_1*P0/mega,'r-' ,[Time[0],Time[-1]],[C0/mega,C0/mega],'k:', \
         Time,I_2*r_hsp*P0/mega,'m--',Time,D_2*P0/mega,'r--', \
         Time,I_3*r_hsp*P0/mega,'m-.',Time,D_3*P0/mega,'r-.')
plt.ylabel('Cases [million]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['severe','fatal','capacity'],loc=7);

plt.subplot(3,1,3)
plt.plot(Time,RE_1,'c-' ,Time,RM_1*100,'y-' , \
         Time,RE_2,'c--',Time,RM_2*100,'y--', \
         Time,RE_3,'c-.',Time,RM_3*100,'y-.')
plt.ylabel('Dim.less number [1]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['epidemic number','mortality x 100'],loc=7);

plt.savefig("IT-1April.png",dpi=600)
```

## Model projection for USA

This projection is based on the latest case count and best estimate growth and mortality rates for the United States.

```python
# set initial condition and parameter values for the model
I0    = 188647*10# current infected count
R0    = 7251*10  # current recovered count
D0    = 4059     # current fatality count
P0    = 327e6    # population size
C0    = 5e5      # bed capacity of health care system
beta  = 0.16     # estimated infection rate parameter
gamma = 1/14     # estimated recovery rate parameter
delta = 0.005    # estimated mortality rate parameter
r_hsp = 0.05     # estimated hospitalisation rate
f_dst = 1.00     # factor for reduction in infection rate due to social distancing
f_rst = 1.00     # factor for reduction in effective population size due to travel restrictions
f_snl = 0.00     # increase seasonality of infection rate (>0 for norther, <0 for southern hemisphere)
f_str = 0.02     # factor for increase in mortality rate for health service at/over capacity
f_dcy = 0.50     # factor for increase in yearly decay of mortality rate (weaker form of virus spread further)
t_end = 365      # model duration in days

# run the model with the above choices
Time, S_1, I_1, R_1, D_1, RI_1, RO_1, RM_1 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 1.00, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_2, I_2, R_2, D_2, RI_2, RO_2, RM_2 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.80, f_rst, f_snl, f_str, f_dcy, t_end)
Time, S_3, I_3, R_3, D_3, RI_3, RO_3, RM_3 = SIRD(I0, R0, D0, P0, C0, beta, gamma, delta, 0.60, f_rst, f_snl, f_str, f_dcy, t_end)

# calculate dimensionless numbers
RE_1 = RI_1*S_1/RO_1
RE_2 = RI_2*S_2/RO_2
RE_3 = RI_3*S_3/RO_3

RR_1 = (1-RM_1)*RO_1/RI_1/S_1
RR_2 = (1-RM_2)*RO_2/RI_2/S_2
RR_3 = (1-RM_3)*RO_3/RI_3/S_3

RD_1 =    RM_1 *RO_1/RI_1/S_1
RD_2 =    RM_2 *RO_2/RI_2/S_2
RD_3 =    RM_3 *RO_3/RI_3/S_3

fig1 = plt.figure(1,figsize=(10,12))  # create figure to plot in

plt.subplot(3,1,1)
plt.plot(Time,S_1*P0/mega,'k-' ,Time,I_1*P0/mega,'b-' ,Time,R_1*P0/mega,'g-' , \
         Time,S_2*P0/mega,'k--',Time,I_2*P0/mega,'b--',Time,R_2*P0/mega,'g--', \
         Time,S_3*P0/mega,'k-.',Time,I_3*P0/mega,'b-.',Time,R_3*P0/mega,'g-.')
plt.ylabel('Cases [million]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['susceptible','infected','recovered'],loc=7)
plt.title('United States')

plt.subplot(3,1,2)
plt.plot(Time,I_1*r_hsp*P0/mega,'m-' ,Time,D_1*P0/mega,'r-' ,[Time[0],Time[-1]],[C0/mega,C0/mega],'k:', \
         Time,I_2*r_hsp*P0/mega,'m--',Time,D_2*P0/mega,'r--', \
         Time,I_3*r_hsp*P0/mega,'m-.',Time,D_3*P0/mega,'r-.')
plt.ylabel('Cases [million]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['severe','fatal','capacity'],loc=7);

plt.subplot(3,1,3)
plt.plot(Time,RE_1,'c-' ,Time,RM_1*100,'y-' , \
         Time,RE_2,'c--',Time,RM_2*100,'y--', \
         Time,RE_3,'c-.',Time,RM_3*100,'y-.')
plt.ylabel('Dim.less number [1]');
plt.grid(True)
plt.xticks(np.arange(0,t_end,30)+shift, ('May','June','July','Aug','Sept','Oct','Nov','Dec','Jan','Feb','Mar','Apr'), rotation=25)
plt.legend(['epidemic number','mortality x 100'],loc=7);

plt.savefig("US-1April.png",dpi=600)
```

```python

```