# Introduction to Neural Networks

## The Perceptron

To get an intuitive idea about Neural Networks, we will code an elementary perceptron. In this example we will illustrate some of the concepts you have just seen, build a small perceptron and make a link between Perceptron and linear classifier.

### Generating some data

Before working with the MNIST dataset, you'll first test your perceptron implementation on a "toy" dataset with just a few data points. This allows you to test your implementations with data you can easily inspect and visualise without getting lost in the complexities of the dataset itself.

Start by loading two basic libraries: `matplotlib` and numpy

```python
# Load the libraries ...

```

Then let us generate some points in a 2D space that will form our dataset (you can add points later if you'd like)

```python
crosses = np.array([[0.5, 1.0], [1.0, 1.5], [1.5, 1.5], [2.0, 1.2], [3.0, 1.7], [1.5, 1.1],[2.1, 1.7]])
circles = np.array([[3.0, 0.5], [4.0, 1.0], [5.0, 0.7], [4.0, 0.2], [5.1, 0.3], [4.2, 0.7]])
```

### Visualising the data

Using `matploblib`, you can display the crosses as crosses (use `marker='x'`) and the circles as circles (use `marker='o'`). You will need to specify that you don't want a line using `linestyle='none'`. You can observe that the points are very easily separable.

```python
# add your code here to visualise the points

```

### Computing the output of a Perceptron

Let us consider the problem of building a classifier that for a given **new** point will return whether it belongs to the crosses (class 1) or circles (class 0). So for example it would take `(2, 1.5)` and return `1`.

Define a function `outPerceptron` which takes a 2d vector `x`, a 2d weight vector `w` and a bias `b` and returns the output following the step rule:

$$
\text{output} = \left\{\begin{align} 1\,\, &\text{if}\,\, \langle x, w\rangle -b \, >\,0 \\ 0\,\, &\text{otherwise}\end{align}\right.
$$

```python
# add your code here...

```

You can then enrich the function so that it can take a **sequence of inputs** (in the form of a matrix where each line of the matrix is one input vector) and return the corresponding **sequence of outputs**.

One way of doing this is to loop over the rows of `X` and for each of them, use the function `outPerceptron` that you just wrote. Store the results in an array `outputs` and return that. Call that function `multiOutPerceptron`/

Once you have that, you can try optimising the function by using a matrix-vector product; call the resulting function `multiOutPerceptron2` (and make sure it leads to the same results!)

```python
# add your code here to implement multiOutPerceptron

# (bonus) add your code here to implement multiOutPerceptron2

```

## = checkpoint 1 =

here, you should copy-paste the following code. If it returns `True` you're good to go on.

```python
np.random.seed(1234)
X = np.random.randn(10, 5)
w = np.random.randn(5)
b = np.random.randn()
np.all(multiOutPerceptron2(X, w, b) == np.array([ 1.,  1.,  0.,  0.,  0.,  1.,  0.,  0.,  1.,  0.]))
```

### Trying different weights and biases

You now have a method that can compute the outputs predicted by an **untrained** perceptron. Can you try picking different weights and biases and see how well you can classify the crosses and circles?

**Note**: to join the crosses and circles into one `instances` matrix, you can use `np.concatenate((crosses, circles), axis=0)`.

You can maybe start with `w=[1, 1]` and `b=1` and output the result of `multiOutPerceptron`. What is your analysis?

```python
# your code here...

```

With the suggested weights and biases (`([1, 1],1)`), you should see something like

> `[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]`

which is clearly not great! Now try with `w=[-0.5, 1]` and `b=-0.2`, what do you observe?

```python
# your code here...

```

### How did we get there?

This is much better (100% correct on the training data).
To obtain these values, we found a **separating hyperplane** (here a line) between the points.
The equation of the line is

$ y = 0.5x-0.2 $

**Quiz**
- **Can you explain why this line corresponds to the weights and bias we used?**
- **Is this separating line unique? does it matter?**

### Illustrating the output of the Perceptron and the separating line

Copy-paste your code to visualise the crosses and circles above and overlay the separating line in red.

Can you modify the parameters of the line a little bit and still find a separating line that "works"?

```python
# your code here..

```

### Testing a few new points

Can you add the following `testPoints` on the plot and discuss what happens to them?

```python
testPoints = np.array([[1, 0.5],[5, 1.5],[3, 1.1]])
```

```python
# your code here to visualise the situation

```

## = checkpoint 2 =

# Gradient Descent (remember?)

## Coding a simple gradient descent

### Considering some function

Let's consider the following arbitrary function and its gradient

$f(x) = \exp(-\sin(x))x^2$

$f'(x) = -x \exp(-\sin(x)) (x\cos(x)-2)$

It is convenient to define python functions which return the value of the function and its gradient at an arbitrary point $x$. Can you define a function `function` and a function `gradient`?

* use `np.exp`, `np.sin`, `np.cos` and remember that `x**2` is the squared of `x`

```python
# your code here...

```

### Visualising the function

Can you write a simple code that shows what the function looks like over the interval `[-10,10]`? use at least `100` points in order to have a high enough definition of the line.

```python
# your code here...

```

### Implementing a simple GD

Now let us implement a simple Gradient Descent that uses constant stepsizes. Define two functions:

1. simplest version which doesn't store the intermediate steps that are taken.
2. a version which does store the steps (useful to visualize what is going on and explain some of the typical behaviour of GD).

Let's call them `simpleGD` and `simpleGD2`. The parameters of both functions will be the initial point `x0`, the stepsize, and the number of steps to be taken.

```python
# your code for simpleGD and simpleGD2

```

### Testing different situations

Try your algorithm `simpleGD` in the following cases:

* $x_0=1, \delta=0.1, n=100$
* $x_0=6, \delta=0.1, n=100$
* $x_0=8, \delta=0.01, n=100$

Can you discuss the results you obtained by having a look at the plot of the function?

### Visualising the cases

We suggest below a function `viz` which shows the path taken by the gradient descent when computed using `simpleGD2`.

Use it in the different cases above in order to see what the Gradient Descent does. Try to interpret the different cases.

```python
def viz(x, a=-10, b=10):
    xx  = np.linspace(a, b, 100)
    yy  = function(xx)
    ygd = function(x)
    plt.plot(xx, yy)
    plt.plot(x, ygd, color='red')
    plt.plot(x[0], ygd[0], marker='o', color='green', markersize=10)
    plt.plot(x[len(x)-1], ygd[len(x)-1], marker='o', color='red', markersize=10)
    plt.show()

```

```python
# your code to visualise the different situations here

```


# Interact Basics

## Import

```python
from ipywidgets import interact, interactive, fixed
from IPython.display import display
```

## Adding numbers

Write a `print_sum` function that `prints` the sum of its arguments `a` and `b`.

```python
def print_sum(a, b):
    """Print the sum of the arguments a and b."""
    print(a+b)
```

Use the `interact` function to interact with the `print_sum` function.

* `a` should be a floating point slider over the interval `[-10., 10.]` with step sizes of `0.1`
* `b` should be an integer slider the interval [-8, 8] with step sizes of `2`.

```python
interact(print_sum, a=(-10.0,10.0,0.1), b=(-8,8,2));
```

## Optional flags
Write a function named `print_string` that prints a string and additionally prints the length of that string if a boolean parameter is `True`.

```python
def print_string(s, length=False):
    """Print the string s and optionally its length."""
    print(s)
    if length:
        print(len(s))
```

Use the `interact` function to interact with the `print_string` function.

* `s` should be a textbox with the initial value `"Hello World!"`.
* `length` should be a checkbox with an initial value of `True`.

```python
interact(print_string, s='Hello World', length=True);
```

# Using SymPy to play with Polynomials

We now show that IPython's `interact` capabilities are not limited to basic Python types, with an exercise that uses [SymPy](http://sympy.org/en/index.html) to factor polynomials.

We begin by initializing SymPy to display its results with proper mathematical formatting in the browser:

```python
from sympy import Symbol, Eq, factor, init_printing
init_printing(use_latex='mathjax')
```

Now, we create a symbolic variable, $x$, to build symbolic polynomials with:

```python
x = Symbol('x')
```

We can then, for example, factor a polynomial:

```python
factor(x**5-1)
```

## Exercise
Write a function that takes an argument $n$, and will display both the polynomial $x^n-1$ and its factorization.

Then, use IPython's `interact` to provide a control to visualize how the result varies with a slider control that allows $n$ to vary over the range from 2 to 40.

*Hint:* Look up the help for the sympy function `Eq`.

```python
def factorit(n):
    display(Eq(x**n-1, factor(x**n-1)))
```

Notice how the output of the `factorit` function is properly formatted LaTeX.

```python
factorit(12)
```

```python
interact(factorit, n=(2,40));
```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's choose an example formula whose antiderivative we will compute.

```python
var( 'x' )
formula = 3*sqrt(x)
formula
```

$\displaystyle 3 \sqrt{x}$

Use the `Integral` function to build a definite integral without evaluating it.
The second parameter is the variable with respect to which you're integrating.

```python
Integral( formula, x )
```

$\displaystyle \int 3 \sqrt{x}\, dx$

Use the `integrate` function to perform the integration, showing the answer.

```python
integrate( formula, x )
```

$\displaystyle 2 x^{\frac{3}{2}}$

```python
integrate( formula, x ) + var('C')  # same, but with a constant of integration
```

$\displaystyle C + 2 x^{\frac{3}{2}}$

database/tasks/How to write and evaluate indefinite integrals/Python, using SymPy.ipynb


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's re-use here the code from how to write an ordinary differential equation,
to write $\frac{dy}{dx}=y$.

```python
var( 'x' )
y = Function('y')(x)
dydx = Derivative( y, x )
ode = dydx - y
ode
```

$\displaystyle - y{\left(x \right)} + \frac{d}{d x} y{\left(x \right)}$

You can solve an ODE by using the `dsolve` command.

```python
solution = dsolve( ode )
solution
```

$\displaystyle y{\left(x \right)} = C_{1} e^{x}$

If there are initial conditions that need to be substituted in for $x$ and $y$,
it is crucial to substitute for $y$ first and then $x$.  Let's assume we have the
initial condition $(3,5)$.  We might proceed as follows.

```python
with_inits = solution.subs( y, 5 ).subs( x, 3 )
with_inits
```

$\displaystyle 5 = C_{1} e^{3}$

```python
solve( with_inits )
```

$\displaystyle \left[ \frac{5}{e^{3}}\right]$

To substitute $C_1=\frac{5}{e^3}$ into the solution, note that $C_1$ is written as `var('C1')`.

```python
solution.subs( var('C1'), 5/E**3 )
```

$\displaystyle y{\left(x \right)} = \frac{5 e^{x}}{e^{3}}$

database/tasks/How to solve an ordinary differential equation/Python, using SymPy.ipynb


# Problem set 1

##Strassen Algorithm

Let $C = AB$, where $A$ and $B$ are squared matrices of the same size.
Direct computation of $C$ requires $\mathcal{O}(n^3)$ arithmetic operations.
Fortunately, this complexity can be reduced even for arbitrary matrices $A$ and $B$.
The following approach which has $\mathcal{O}(n^{\log_2 7})$ is called Strassen algorithm.
Its idea is based on the fact that elements of $2\times 2$ matrix
$$
\begin{bmatrix} c_{11} & c_{12} \\ c_{21} & c_{22}  \end{bmatrix} =
\begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22}  \end{bmatrix}
\begin{bmatrix} b_{11} & b_{12} \\ b_{21} & b_{22}  \end{bmatrix}
$$
can be computed using only 7 multiplications:
\begin{equation}\begin{split}
c_{11} &= f_1 + f_4 - f_5 + f_7, \\
c_{12} &= f_3 + f_5, \\
c_{21} &= f_2 + f_4, \\
c_{22} &= f_1 - f_2 + f_3 + f_6,
\end{split}\end{equation}
where
\begin{equation}\begin{split}
f_1 &= (a_{11} + a_{22}) (b_{11} + b_{22}),\\
f_2 &= (a_{21} + a_{22}) b_{11},\\
f_3 &= a_{11} (b_{12} - b_{22}),\\
f_4 &= a_{22} (b_{21} - b_{11}),\\
f_5 &= (a_{11} + a_{12}) b_{22},\\
f_6 &= (a_{21} - a_{11}) (b_{11} + b_{12}),\\
f_7 &= (a_{12} - a_{22}) (b_{21} + b_{22}).
\end{split}\end{equation}

Formulas above hold for the case when $a_{ij}, b_{ij}, c_{ij}$ ($i$ and $j=1,2$) are blocks.
Therefore, spliting matrices $A$ and $B$ of abitrary sizes into 4 blocks and applying described procedure recursively for blocks one will get $\mathcal{O}(n^{\log_2 7})$ complexity.

**Tasks**

- (4 pts)  Prove that Strassen alogorithm has $\mathcal{O}(n^{\log_2 7})$ complexity
- (4 pts) Implement Strassen algorithm in Python. **Note**: for simplicity consider that $n$ is a power of 2
- (3 pts) Compare the result with direct matrix-by-matrix multiplication and $\verb|numpy.dot|$ procedure by ploting timings as a function of $n$. **Note**: use logarithmic scale

```

```

##Fast Fourier Transform

Let $y = Ax$ (matvec operation), where $A \in \mathbb{C}^{m\times n}$ and $x \in \mathbb{C}^{n\times 1}$.
Direct computation of $y$ requires $\mathcal{O}(n^2)$.
Since $A$ contains $n^2$ elements, this complexity can not be reduced for an arbitrary matrix $A$.
There are certain classes of matrices for which matvec requires less operations.
For instance, sparse, Toeplitz, lowrank, etc.
Another important example of structured matrix which arises in a huge amount of applications (signal and image processing, fast PDE solvers) is Fourier matrix
$$
F_n = \{ \omega^{kl} \}_{k,l=0}^{n-1}, \quad \text{where} \quad \omega = e^{-\frac{2\pi i}{n}}.
$$
Matvec operation with Fourier matrix is called discrete Fourier transform (DFT) and has $\mathcal{O}(n \log n)$ complexity.
The simplest way to get this complexity is to spilt odd and even rows in Fourier matrix:
\begin{equation}
P_n F_n = \begin{bmatrix} F_{n/2} & F_{n/2} \\ F_{n/2} W_{n/2} & -F_{n/2} W_{n/2} \end{bmatrix}, \quad (1)
\end{equation}
where $P_n$ is a permutaion matrix which permutes odd and even rows, and $W_{n/2}=\text{diag}(1,\omega,\omega^2,\dots,\omega^{n/2-1})$.
Thus, multiplication by $F_n$ is reduced to several multiplications by $F_{n/2}$ and linear operations such as multiplication by the diagonal matrix $W_{n/2}$.
Continuing this procedure recursively for $F_{n/2}$ we will get $\mathcal{O}(n \log n)$ operations.

**Tasks**

- (4 pts) Prove expression (1)
- (4 pts) Implement the described fft algorithm in Python. **Note**: for simplicity consider that $n$ is a power of 2
- (3 pts) Compare the result with $\verb|numpy.dot|$ and $\verb|numpy.fft.fft|$ procedures by ploting timings as a function of $n$. **Note**: use logarithmic scale

```

```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's say we want to write the equation $x^2+y^2=2$.
We must first define $x$ and $y$ as mathematical variables,
then use SymPy's `Eq` function to build an equation.
This helps SymPy distinguish a mathematical equation
from a Python assignment statement.

```python
var( 'x y' )
Eq( x**2 + y**2, 2 )    # Two parameters: left and right sides of equation
```

$\displaystyle x^{2} + y^{2} = 2$

You can make a system of equations just by placing several equations in a Python list.

```python
system = [
    Eq( x + 2*y, 1 ),
    Eq( x - 9*y, 5 )
]
system
```

$\displaystyle \left[ x + 2 y = 1, \  x - 9 y = 5\right]$

database/tasks/How to write symbolic equations/Python, using SymPy.ipynb


# Exponentials, Radicals, and Logs
Up to this point, all of our equations have included standard arithmetic operations, such as division, multiplication, addition, and subtraction. Many real-world calculations involve exponential values in which numbers are raised by a specific power.

## Exponentials
A simple case of of using an exponential is squaring a number; in other words, multipying a number by itself. For example, 2 squared is 2 times 2, which is 4. This is written like this:

\begin{equation}2^{2} = 2 \cdot 2 = 4\end{equation}

Similarly, 2 cubed is 2 times 2 times 2 (which is of course 8):

\begin{equation}2^{3} = 2 \cdot 2 \cdot 2 = 8\end{equation}

In R, you use the ****** operator, like this example in which **x** is assigned the value of 5 raised to the power of 3 (in other words, 5 x 5 x 5, or 5-cubed):

```R
x = 5**3
print(x)
```

    [1] 125

Multiplying a number by itself twice or three times to calculate the square or cube of a number is a common operation, but you can raise a number by any exponential power. For example, the following notation shows 4 to the power of 7 (or 4 x 4 x 4 x 4 x 4 x 4 x 4), which has the value:

\begin{equation}4^{7} = 16384 \end{equation}

In mathematical terminology, **4** is the *base*, and **7** is the *power* or *exponent* in this expression.

## Radicals (Roots)
While it's common to need to calculate the solution for a given base and exponential, sometimes you'll need to calculate one or other of the elements themselves. For example, consider the following expression:

\begin{equation}?^{2} = 9 \end{equation}

This expression is asking, given a number (9) and an exponent (2), what's the base? In other words, which number multipled by itself results in 9? This type of operation is referred to as calculating the *root*, and in this particular case it's the *square root* (the base for a specified number given the exponential **2**). In this case, the answer is 3, because 3 x 3 = 9. We show this with a **√** symbol, like this:

\begin{equation}\sqrt{9} = 3 \end{equation}

Other common roots include the *cube root* (the base for a specified number given the exponential **3**). For example, the cube root of 64 is 4 (because 4 x 4 x 4 = 64). To show that this is the cube root, we include the exponent **3** in the **√** symbol, like this:

\begin{equation}\sqrt[3]{64} = 4 \end{equation}

We can calculate any root of any non-negative number, indicating the exponent in the **√** symbol.

The R **sqrt** function calculates the square root of a number. To calculate other roots, you need to reverse the exponential calculation by raising the given number to the power of 1 divided by the given exponent:

```R
## calculate and display the square root of 25
x = sqrt(25)
print(x)

## calculate and display the cube root of 64
cr = 64**(1/3)
print(cr)
```

    [1] 5
    [1] 4

The code used in R to calculate roots other than the square root reveals something about the relationship between roots and exponentials. The exponential root of a number is the same as that number raised to the power of 1 divided by the exponential. For example, consider the following statement:

\begin{equation} 8^{\frac{1}{3}} = \sqrt[3]{8} = 2 \end{equation}

Note that a number to the power of 1/3 is the same as the cube root of that number.

Based on the same arithmetic, a number to the power of 1/2 is the same as the square root of the number:

\begin{equation} 9^{\frac{1}{2}} = \sqrt{9} = 3 \end{equation}

You can see this for yourself with the following R code:

```R
print(9**0.5)
print(sqrt(9))
```

    [1] 3
    [1] 3

## Logarithms
Another consideration for exponential values is the requirement occassionally to determine the exponent for a given number and base. In other words, how many times do I need to multiply a base number by itself to get the given result. This kind of calculation is known as the *logarithm*.

For example, consider the following expression:

\begin{equation}4^{?} = 16 \end{equation}

In other words, to what power must you raise 4 to produce the result 16?

The answer to this is 2, because 4 x 4 (or 4 to the power of 2) = 16. The notation looks like this:

\begin{equation}log_{4}(16) = 2 \end{equation}

In R, you can calculate the logarithm of a number of a specified base using the **logb** function, indicating the number and the base:

```R
x = logb(16, 4)
print(x)
```

    [1] 2

The final thing you need to know about exponentials and logarithms is that there are some special logarithms:

The *common* logarithm of a number is its exponential for the base **10**. You'll occassionally see this written using the usual *log* notation with the base omitted:

\begin{equation}log(1000) = 3 \end{equation}

Another special logarithm is something called the *natural log*, which is a exponential of a number for base ***e***, where ***e*** is a constant with the approximate value 2.718. This number occurs naturally in a lot of scenarios, and you'll see it often as you work with data in many analytical contexts. For the time being, just be aware that the natural log is sometimes written as ***ln***:

\begin{equation}log_{e}(64) = ln(64) = 4.1589 \end{equation}

The **log** function in R returns the natural log (base ***e***) when no base is specified. To return the base 10 or common log in R, use the **log10** function:

```R
## Natural log of 29
log(29)

## Base 10 log of 100
log10(100)
```

3.36729582998647

2

## Solving Equations with Exponentials
OK, so now that you have a basic understanding of exponentials, roots, and logarithms; let's take a look at some equations that involve exponential calculations.

Let's start with what might at first glance look like a complicated example, but don't worry - we'll solve it step-by-step and learn a few tricks along the way:

\begin{equation}2y = 2x^{4} ( \frac{x^{2} + 2x^{2}}{x^{3}} ) \end{equation}

First, let's deal with the fraction on the right side. The numerator of this fraction is x2 + 2x2 - so we're adding two exponential terms. When the terms you're adding (or subtracting) have the same exponential, you can simply add (or subtract) the coefficients. In this case, x2 is the same as 1x2, which when added to 2x2 gives us the result 3x2, so our equation now looks like this:

\begin{equation}2y = 2x^{4} ( \frac{3x^{2}}{x^{3}} ) \end{equation}

Now that we've condolidated the numerator, let's simplify the entire fraction by dividing the numerator by the denominator. When you divide exponential terms with the same variable, you simply divide the coefficients as you usually would and subtract the exponential of the denominator from the exponential of the numerator. In this case, we're dividing 3x2 by 1x3: The coefficient 3 divided by 1 is 3, and the exponential 2 minus 3 is -1, so the result is 3x-1, making our equation:

\begin{equation}2y = 2x^{4} ( 3x^{-1} ) \end{equation}

So now we've got rid of the fraction on the right side, let's deal with the remaining multiplication. We need to multiply 3x-1 by 2x4. Multiplication, is the opposite of division, so this time we'll multipy the coefficients and add the exponentials: 3 multiplied by 2 is 6, and -1 + 4 is 3, so the result is 6x3:

\begin{equation}2y = 6x^{3} \end{equation}

We're in the home stretch now, we just need to isolate y on the left side, and we can do that by dividing both sides by 2. Note that we're not dividing by an exponential, we simply need to divide the whole 6x3 term by two; and half of 6 times x3 is just 3 times x3:

\begin{equation}y = 3x^{3} \end{equation}

Now we have a solution that defines y in terms of x. We can use R to plot the line created by this equation for a set of arbitrary *x* and *y* values:

```R
# Create a dataframe with an x column containing values from -10 to 10
df = data.frame(x = seq(-10, 10))

# Add a y column by applying the slope-intercept equation to x
df$y = 3*df$x**3

#Display the dataframe
print(df)

# Plot the line
library(ggplot2)
library(repr)
options(repr.plot.width=4, repr.plot.height=4)
ggplot(df, aes(x,y)) +
       geom_line(color = 'magenta', size = 1) +
       geom_hline(yintercept=0) + geom_vline(xintercept=0)
```

Note that the line is curved. This is symptomatic of an exponential equation: as values on one axis increase or decrease, the values on the other axis scale *exponentially* rather than *linearly*.

Let's look at an example in which x is the exponential, not the base:

\begin{equation}y = 2^{x} \end{equation}

We can still plot this as a line:

```R
# Create a dataframe with an x column containing values from -10 to 10
df = data.frame(x = seq(-10, 10))

# Add a y column by applying the slope-intercept equation to x
df$y = 2.0**df$x

## Plot the line
ggplot(df, aes(x,y)) +
       geom_line(color = 'magenta', size = 1) +
       geom_hline(yintercept=0) + geom_vline(xintercept=0)
```

Note that when the exponential is a negative number, R reports the result as 0. Actually, it's a very small fractional number, but because the base is positive the exponential number will always positive. Also, note the rate at which y increases as x increases - exponential growth can be be pretty dramatic.

So what's the practical application of this?

Well, let's suppose you deposit $100 in a bank account that earns 5% interest per year. What would the balance of the account be in twenty years, assuming you don't deposit or withdraw any additional funds?

To work this out, you could calculate the balance for each year:

After the first year, the balance will be the initial deposit ($100) plus 5% of that amount:

\begin{equation}y1 = 100 + (100 \cdot 0.05) \end{equation}

Another way of saying this is:

\begin{equation}y1 = 100 \cdot 1.05 \end{equation}

At the end of year two, the balance will be the year one balance plus 5%:

\begin{equation}y2 = 100 \cdot 1.05 \cdot 1.05 \end{equation}

Note that the interest for year two, is the interest for year one multiplied by itself - in other words, squared. So another way of saying this is:

\begin{equation}y2 = 100 \cdot 1.05^{2} \end{equation}

It turns out, if we just use the year as the exponent, we can easily calculate the growth after twenty years like this:

\begin{equation}y20 = 100 \cdot 1.05^{20} \end{equation}

Let's apply this logic in R to see how the account balance would grow over twenty years:

```R
# Create a dataframe with an x column containing values from -10 to 10
df = data.frame(Year = seq(1, 20))

# Calculate the balance for each year based on the exponential growth from interest
df$Balance = 100 * (1.05**df$Year)

## Plot the line
ggplot(df, aes(Year, Balance)) +
       geom_line(color = 'green', size = 1) +
       geom_hline(yintercept=0) + geom_vline(xintercept=0)
```

R/Module01/01-04-Exponentials Radicals and Logarithms.ipynb


The Cholesky Decomposition exists when a matrix is hermitian and positive-definite. It expresses the matrix $\mathbf{A}$ as:

\begin{equation}
\mathbf{A} = \mathbf{L}\mathbf{L^\dagger}
\end{equation}

Where $\mathbf{L}$ is a lower-triangular matrix with positive, real diagonal entries. When $\mathbf{A}$ is real, then so is $\mathbf{L}$. The Cholesky decomposition enables fast solution of a linear system, but it can also be used to create correlated random variables in Monte Carlo simulations.

### Creating Correlated Random Variables
Let $\mathbf{u}_t$ be a vector of uncorrelated samples with unit standard deviation. If the covariance matrix of the system to be simulated is  $\mathbf{\Sigma}$ with Cholesky decomposition $\mathbf{\Sigma} = \mathbf{LL}^\dagger$, then the vector $\mathbf{v}_t = \mathbf{Lu}_t$ has the desired covariance.

```python
import numpy as np
import matplotlib.pyplot as plt

np.random.seed = 42

T = 10000
n = 5

M = np.random.randn(*(n,n))
Covar = np.matmul(M.T,M) # random symmetric matrix

L = np.linalg.cholesky(Covar)

print('covar:\n',Covar)
print('L:\n',L)
print('L*L^T:\n',np.matmul(L,L.T))
print('covar - L*L^T = 0',np.allclose(np.matmul(L,L.T)-Covar,0))
```

    covar:
     [[ 5.83329982 -2.7988554  -1.221951    0.1995683   2.31184273]
     [-2.7988554  12.06818258  1.35675522 -0.99876349 -4.46688883]
     [-1.221951    1.35675522  2.52907446 -2.10538897  1.58002341]
     [ 0.1995683  -0.99876349 -2.10538897  3.50034403 -2.21280123]
     [ 2.31184273 -4.46688883  1.58002341 -2.21280123  5.34210571]]
    L:
     [[ 2.41522252  0.          0.          0.          0.        ]
     [-1.15883956  3.27494633  0.          0.          0.        ]
     [-0.50593723  0.23525733  1.4892132   0.          0.        ]
     [ 0.08262936 -0.27573255 -1.34212858  1.27129023  0.        ]
     [ 0.95719658 -1.02525392  1.54813515 -0.39077877  0.90846262]]
    L*L^T:
     [[ 5.83329982 -2.7988554  -1.221951    0.1995683   2.31184273]
     [-2.7988554  12.06818258  1.35675522 -0.99876349 -4.46688883]
     [-1.221951    1.35675522  2.52907446 -2.10538897  1.58002341]
     [ 0.1995683  -0.99876349 -2.10538897  3.50034403 -2.21280123]
     [ 2.31184273 -4.46688883  1.58002341 -2.21280123  5.34210571]]
    covar - L*L^T = 0 True

```python
u = np.random.randn(*(n,T))
v = np.matmul(L,u)
```

```python
fig, ax = plt.subplots(2,1,figsize=(12,8))
for i in range(n):
    ax[0].plot(np.cumsum(u[i],axis=0))
    ax[1].plot(np.cumsum(v[i],axis=0))

ax[0].set_title('Uncorrelated Gaussian Random Walk (mu=0,std=1)')
ax[1].set_title('Correlated')

plt.savefig('img/cholesky1.png')
```

```python
fig, ax = plt.subplots(2,2,figsize=(8,8))

ax[0][0].imshow(Covar,vmin=np.min(Covar),vmax=np.max(Covar))
ax[0][0].set_title('Covariance: Target')
ax[0][1].imshow(np.cov(u))
ax[0][1].set_title('Covariance: Uncorrelated u')
ax[1][0].imshow(np.cov(v),vmin=np.min(Covar),vmax=np.max(Covar))
ax[1][0].set_title('Covariance: Cholesky Constructed v=Lu')
ax[1][1].imshow(np.log10(np.abs(Covar-np.cov(v))),vmin=np.min(Covar),vmax=np.max(Covar))
ax[1][1].set_title('Error (log10)')

plt.savefig('img/cholesky2.png')
```

Simulation - Cholesky Decomposition, Correlated Random Variables.ipynb


# 2D heat conduction

The equation of heat conduction in 2D is:

$$
\begin{equation}
\rho c_p \frac{\partial T}{\partial t} = \frac{\partial}{\partial x} \left( \kappa_x \frac{\partial T}{\partial x} \right) + \frac{\partial}{\partial y} \left(\kappa_y \frac{\partial T}{\partial y} \right)
\end{equation}
$$

where $\rho$ is the density, $c_p$ is the heat capacity and $\kappa$ is the thermal conductivity.

If the thermal conductivity $\kappa$ is constant, then we can take it outside of the spatial derivative and the equation simplifies to:

$$
\begin{equation}
\frac{\partial T}{\partial t} = \alpha \left(\frac{\partial^2 T}{\partial x^2} + \frac{\partial^2 T}{\partial y^2} \right)
\end{equation}
$$

where $\alpha = \frac{\kappa}{\rho c_p}$ is the thermal diffusivity.  The thermal diffusivity describes the ability of a material to conduct heat vs. storing it.

Let's write this out discretized using forward difference in time, and central difference in space,  using an explicit scheme. You should be able write this out yourself, without looking—if you need to look, it means you still need to write more difference equations by your own hand!

$$
\begin{equation}
\frac{T^{n+1}_{i,j} - T^n_{i,j}}{\Delta t} = \alpha \left( \frac{T^n_{i+1, j} - 2T^n_{i,j} + T^n_{i-1,j}}{\Delta x^2} + \frac{T^n_{i, j+1} - 2T^n_{i,j} + T^n_{i,j-1}}{\Delta y^2}\right)
\end{equation}
$$
suppose we have
$$
\begin{equation}
{\Delta x^2}={\Delta y^2}
\end{equation}
$$
Now our discrtized equation will simpler

$$
\begin{equation}
\frac{T^{n+1}_{i,j} - T^n_{i,j}}{\Delta t} =  \alpha( \frac{T^n_{i+1, j} - 4T^n_{i,j} + T^n_{i-1,j} + T^n_{i, j+1} + T^n_{i,j-1}} {\Delta x^2})
\end{equation}
$$
Rearranging the equation to solve for the value at the next time step, $T^{n+1}_{i,j}$, yields

$$
\begin{equation}
T^{n+1}_{i,j}= T^n_{i,j} + \alpha \left( \frac{\Delta t}{\Delta x^2} (T^n_{i+1, j} - 2T^n_{i,j} + T^n_{i-1,j}) + \\\frac{\Delta t}{\Delta y^2} (T^n_{i, j+1} - 2T^n_{i,j} + T^n_{i,j-1})\right) = T^n_{i,j} + \frac{\Delta t}{\Delta x^2}\alpha (T^n_{i+1, j} - 4T^n_{i,j} + T^n_{i-1,j} + T^n_{i, j+1} + T^n_{i,j-1})
\end{equation}
$$

### Stability analysis

Before doing any coding, let's revisit stability constraints. We saw in the 1D explicit discretization of the diffusion equation was stable as long as $\alpha \frac{\Delta t}{(\Delta x)^2} \leq \frac{1}{2}$. In 2D, this constraint is even tighter, as we need to add them in both directions:

$$
\begin{equation}
\alpha \frac{\Delta t}{(\Delta x)^2} + \alpha \frac{\Delta t}{(\Delta y)^2} < \frac{1}{2}.
\end{equation}
$$

Say that the mesh has the same spacing in $x$ and $y$, $\Delta x = \Delta y = \delta$. In that case, the stability condition is:

$$
\begin{equation}
\alpha \frac{\Delta t}{\delta^2} < \frac{1}{4}
\end{equation}
$$

### Problem statement

We understand how heat is conducted through water . we consider 2D water body of size $1{\rm m}\times 1{\rm m}$, with a thermal diffusivity $\alpha \approx 0.146.10^{-6}{\rm m}^2{/\rm s}$.
We're going to set up a somewhat artificial problem, just to demonstrate an interesting numerical solution. water is in contact at left side with a heat source which has a constant temperature of $T=100°C $, and for the rest of the edges it is in contact with insulating material, Initially, the water temperature is at $(25{\rm C})$. *How long does it take for the water tempreture to be fully warmed by the heat source ?*

### Boundary Conditions

Whenever we reach a point that interacts with the boundary, we apply the boundary condition,if the boundary has Dirichlet conditions, we simply impose the prescribed temperature $T=100°C $ at $y=0$ at the left side. If the boundary has Neumann conditions, we approximate them with a finite-difference scheme.

Remember, Neumann boundary conditions prescribe the derivative in the normal direction. For example, in the problem described above, we have $\frac{\partial T}{\partial y} = q_y$ in the top boundary and $\frac{\partial T}{\partial x} = q_x$ in the right boundary, with $q_y = q_x = 0$ (insulation).

Thus, at every time step, we need to enforce

$$
\begin{equation}
T_{i,end} = q_y\cdot\Delta y + T_{i,end-1}
\end{equation}
$$

and

$$
\begin{equation}
T_{end,j} = q_x\cdot\Delta x + T_{end-1,j}
\end{equation}
$$

```python
from mpl_toolkits import mplot3d
import numpy as np
from matplotlib import animation
import matplotlib.pyplot as plt
from IPython.display import HTML
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
plt.rcParams['font.family'] = 'serif'
plt.rcParams['font.size'] = 16
```

```python
Lx = 1  # length of the plate in the x direction
Ly = 1  # height of the plate in the y direction
nx = 41  # number of points in the x direction
ny = 41  # number of points in the y direction
dx = Lx / (nx - 1)  # grid spacing in the x direction
dy = Ly / (ny - 1)  # grid spacing in the y direction
alpha = 0.146e-6 # thermal diffusivity of the water

# Define the locations along a gridline.
x = np.linspace(0.0, Lx, num=nx)
y = np.linspace(0.0, Ly, num=ny)

# Compute the initial temperature distribution.
Tb = 100.0  # temperature at the left boundaries
T0 = 25.0 * np.ones((ny, nx))
#apply Dirichlet boundary conditions
T0[:, 0] = Tb #here T[j,i] because in python j represnts the rows which corresponds to y axis  and i represnts the colums which corresponds to x axis

```

```python
sigma = 0.24
dt = (sigma * dx**2 )/ alpha  # time-step size
nt = 100
c=(dt*alpha)/dx**2
```

```python
T=T0.copy()
for i in range (nt):
    Tn=T.copy()
    T[1:-1, 1:-1] = (Tn[1:-1, 1:-1] +
                     c * (Tn[1:-1, 2:] - 4.0 * Tn[1:-1, 1:-1] + Tn[1:-1, :-2]
                    + Tn[2:, 1:-1] + Tn[:-2, 1:-1]))
#apply Neumann boundary conditions
    T[-1,:]=T[-2,:]
    T[:, -1] = T[:, -2]
    T[0,:] = T[1,:]
```

```python
# Plot the filled 2d contour of the temperature.
plt.figure(figsize=(8.0, 5.0))
plt.xlabel('x [m]')
plt.ylabel('y [m]')
levels= np.linspace(25.0, 100.0, num=40)
contf = plt.contourf(x, y, T, levels=levels)
cbar = plt.colorbar(contf)
cbar.set_label('Temperature [C]')
```

```python
# Plot the filled 3d contour of the temperature.
plt.figure(figsize=(8.0, 5.0))
axis=plt.gca(projection= "3d")
plt.xlabel('x [m]')
plt.ylabel('y [m]')
levels= np.linspace(25.0, 100.0, num=40)
contf = plt.contourf(x, y, T, levels=levels)
cbar = plt.colorbar(contf)
cbar.set_label('Temperature [C]')

```

```python
def ftcs(T0,nt, c):
    T=T0.copy()
    for i in range (nt):
        Tn=T.copy()
        T[1:-1, 1:-1] = (Tn[1:-1, 1:-1] +
                     c * (Tn[1:-1, 2:] - 4.0 * Tn[1:-1, 1:-1] + Tn[1:-1, :-2]
                    + Tn[2:, 1:-1] + Tn[:-2, 1:-1]))
#apply Neumann boundary conditions
        T[-1,:]=T[-2,:]
        T[:, -1] = T[:, -2]
        T[0,:] = T[1,:]
    return T
```

```python
nt = 1000
c=(dt*alpha)/dx**2
T1=ftcs(T0, nt,c)
```

```python
# Plot the filled 2d contour of the temperature.
fig=plt.figure(figsize=(8.0, 5.0))
plt.xlabel('x [m]')
plt.ylabel('y [m]')
levels= np.linspace(25.0, 100.0, num=40)
contf = plt.contourf(x, y, T1, levels=levels)
cbar = plt.colorbar(contf)
cbar.set_label('Temperature [C]')
```

```python
#def animate(n):
    #fig.suptitle('Time step {:0>2}'.format(n))

```

```python
#anim = animation.FuncAnimation(fig, animate,frames=nt , interval=100)
```

__3D heat conduction__

Now we are going to expand our problem into 3D , adding a z dimension with the same height , thus we are going to have $1{\rm m}\times 1{\rm m}\times 1{\rm m}$ water with the same initial conditions as before
our new equation will be if
$$
\begin{equation}
{\Delta x^2}={\Delta y^2}={\Delta z^2}
\end{equation}
$$
$$
\begin{equation}
\frac{T^{n+1}_{i,j,k} - T^n_{i,j,k}}{\Delta t} =  \alpha( \frac{T^n_{i+1, j,k} - 6T^n_{i,j,k} + T^n_{i-1,j,k} + T^n_{i, j+1,k} + T^n_{i,j-1,k}+ T^n_{i, j,k+1} +T^n_{i, j,k-1}}{\Delta x^2})
\end{equation}
$$
Rearranging the equation to solve for the value at the next time step, $T^{n+1}_{i,j}$, yields

$$
\begin{equation}
T^{n+1}_{i,j,k} = T^n_{i,j,k} + \frac{\Delta t}{\Delta x^2}\alpha (T^n_{i+1, j,k} - 6T^n_{i,j,k} + T^n_{i-1,j,k} + T^n_{i, j+1,k} + T^n_{i,j-1,k}+ T^n_{i, j,k+1} +T^n_{i, j,k-1})
\end{equation}
$$

__Stability analysis__

Now let's examine the stability constraint of 3D dimensions

$$
\begin{equation}
\alpha \frac{\Delta t}{(\Delta x)^2} + \alpha \frac{\Delta t}{(\Delta y)^2} + \alpha \frac{\Delta t}{(\Delta z)^2} < \frac{1}{2}.
\end{equation}
$$

Say that the mesh has the same spacing in $x$,$y$,$z$ $\Delta x = \Delta y = \Delta z = \delta $. In that case, the stability condition is:

$$
\begin{equation}
\alpha \frac{\Delta t}{\delta^2} < \frac{1}{6}
\end{equation}
$$

```python
Lx = 1  # length of the plate in the x direction
Ly = 1 # height of the plate in the y direction
Lz = 1 # height of the plate in the z direction
nx = 41  # number of points in the x direction
ny = 41 # number of points in the y direction
nz= 41 # number of points in the z direction
dx = Lx / (nx - 1)  # grid spacing in the x direction
dy = Ly / (ny - 1) # grid spacing in the y direction
dz = Lz / (nz - 1) # grid spacing in the z direction
alpha = 0.146e-6 # thermal diffusivity of the water

# Define the locations along a gridline.
x = np.linspace(0.0, Lx, num=nx)
y = np.linspace(0.0, Ly, num=ny)
z = np.linspace(0.0, Lz, num=nz)

#initial conditions
T0=25.0*np.ones((nx,ny,nz))
Tb=100.0
T0[:,0,:]= Tb

```

```python
sigma = 0.16
dt = (sigma * dx**2 )/ alpha  # time-step size
nt = 500
c=(dt*alpha)/dx**2
```

```python
%%time
T=T0.copy()
for i in range (nt):
        Tn=T.copy()
        T[1:-1, 1:-1,1:-1] = (Tn[1:-1, 1:-1,1:-1] + c * (Tn[1:-1, 2:,1:-1] - 6.0 * Tn[1:-1, 1:-1,1:-1]
                                                         + Tn[1:-1, :-2,1:-1] + + Tn[2:, 1:-1,1:-1] + Tn[:-2, 1:-1,1:-1]+ Tn[1:-1,1:-1,2:]
                                                         + Tn[ 1:-1,1:-1,:-2] ))

      #apply Neumann boundary conditions
        T[-1,:,:]=T[-2,:,:]   #delta t/ delta y = 0 at y=ly
        T[:, -1,:] = T[:, -2,:] #delta t/ delta x = 0 at x=lx
        T[0,:,:] = T[1,:,:] #delta t/ delta y = 0 at y=0
        T[:,:,-1] = T[:,:,-2] #delta t/ delta z = 0 at z=Lz
        T[:,:,0] = T[:,:,1] #delta t/ delta z = 0 at z=0
```

    Wall time: 1.1 s

```python
from mayavi import mlab
mlab.init_notebook()
```

    Notebook initialized with ipy backend.

```python
X,Y,Z=np.mgrid[0:1:41j,0:1:41j,0:1:41j]
```

```python
mlab.clf()
s=mlab.volume_slice(T,plane_orientation='z_axes')
mlab.title('time step is {}' .format(nt))
mlab.colorbar(object=None, title='Temperature',orientation='horizontal')
s
```

    Image(value=b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01\x90\x00\x00\x01^\x08\x02\x00\x00\x00$?\xde_\x00\…

```python

```

```python

```


# Multiples in Number Range

Repo: https://github.com/Andrewnetwork/MathematicalProgrammingProblems

## 0.) Definitions
A *range* of natural numbers, positive integers, can be defined by the notation $[x,y]$ where $x$ is the starting number and $y$ is the ending number. Example: $[0,10] = [0,1,2,3,4,5,6,7,8,9,10]$.

In the range $[x,y]$ there are $y-x = \lvert[x,y]\lvert$ numbers. Example: $\lvert[2,5]\lvert= 5 - 2 = 3.$ There are three numbers in the range $[2,5]$: $[2,3,4]$

A number $m$ is a *multiple* of some natural number $n$ if there is some integer $i$ such that $n = im$.

## 1.) How Many Multiples in a Range

Let $x,y,z \in \mathbb{N}.$

In the number range $[x,y]$ given $x<y$, how many multiples of $z$ are there?

### Solution
Haskell Code:
```Haskell
multInRange strt stop mult = floor((stop-strt)/mult)
```
Mathematics:
$$\large \lfloor \frac{\lvert[x,y]\lvert}{z} \rfloor$$

$\lvert[x,y]\lvert = y-x$ gives us the total number of natural numbers in this range. $\lfloor \frac{\lvert[x,y]\lvert}{z}\rfloor$ gives us how many times z partitions we can get in this range. I.e. multiples.

### Problem Examples

#### 1: Within the range of natural numbers $[1,10]$ how many multiples of $3$ are there?

Answer: $3$.

Given $x=0,y=10,z=3$

$\large \lfloor \frac{(y-x)}{z} \rfloor = 3$

In haskell: $[1,2..10] = [1,2,3,4,5,6,7,8,9,10]$

Multiples of 3: $[3,6,9]$.

Sum: $3 + 3*2 + 3*3 = 6*3 = 18$

#### 2:  Within the range of natural numbers $[1,20]$ how many multiples of $3$ are there?
3,6,9,12,15,18

$3 + 3*2 + 3*3 + 3*4 + 3*5 + 3*6 = 7*3*3 = 7*6 = 21*3 = 63$

## 2.) Sum of Multiples in a Range
Let $x,y,z \in \mathbb{N}.$

In the range $[x,y]$ what is the sum of the multiples of z?

### Solution
Haskell Code:
```Haskell
multInRange strt stop mult = floor((stop-strt)/mult)

sumNat n = (n*(n+1))/2

sumMult strt stop mult = mult * (sumNat (fromInteger (multInRange strt stop mult)))
```
Mathematics:
$$\large\begin{align}
n = \lfloor \frac{(y-x)}{z} \rfloor\\
z*\frac{n*(n+1)}{2}
\end{align}$$

### Problem Examples
#### 1: In the range [0,100] what's the sum of the multiples of 5?
Multiples of $5$ in $[0,100] = [5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100]$

How many multiples of $5$ are there in $[0,100]$?
$$n = \lfloor \frac{(y-x)}{z} \rfloor == \lfloor \frac{100}{5} \rfloor = 20$$

Haskell Code:
```Haskell
multInRange 0 100 5
-- Result: 20
```

Sum of these multiples:
$$5 + (5+5) + (5+5+5) + \ldots + (5 + 5 + \ldots n) = 5*1 + 5*2 + 5*3 + \dots + 5*n =  5*\sum_{i=1}^{n}{i} = 5 $$

As we see here, to compute the sum of multiples of $5$ in the range $[0,100]$, we compute the sum of integers up to how many multiples are in that range multiplied by $5$.

$$
\large\begin{align}
z*\frac{n*(n+1)}{2} = 5 * \frac{20*(20+1)}{2} = 1050
\end{align}$$

Haskell Code:
```Haskell
sumMult 0 100 5
-- Result: 1050.0
```

#### 2: In the range [0,100] what's the sum of the multiples of 5 and 6?
Multiples of $5$ and $6$ in $[0,100] = $

$$[5,6,10,12,15,18,20,24,25,30,35,36,40,42,45,48,50,54,55,60,65,66,70,72,75,78,80,84,85,90,95,96,100]$$

We can extend the results from the previous question to multiple multiples on a range.

We can find the sum of the multiples of $5$ and $6$ on the range $[0,100]$. Independently by:
Haskell Code:
```Haskell
a = (sumMult 0 100 5)
b = (sumMult 0 100 6)
```
But alas, $a + b$ is not our answer! We need to take into consideration the multiples $5$ and $6$ share in common so we do not include the twice in our calculation. The multiples of $5$ and $6$ are multiples of $5*6 = 30$. Therefore we achieve the result by computing:

Haskell Code:
```Haskell
(sumMult 0 100 5) + (sumMult 0 100 6) - (sumMult 0 100 (5*6) )
-- Result: 1686.0
```

MathematicalProgrammingProblems/Notebooks/Multiples in Number Range.ipynb


# Problem definition

## Assume 2 players are playing tennis. The probability of Player 1 winning a point is $p$. What is the probability that Player 1 will win the game?

#### Imports

```python
import math

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
```

### Problem solution

To solve this problem we will consider two outcomes:
1. The probability of Player 1 winning the game without the game going to a draw.
1. The probability of Player 1 winning the game given the game went to a draw.

The final solution will be the total probability of the two above events as per the total probability law:
\begin{equation}
    P(win) = P(win|draw^C)(1-P(draw)) + P(win|draw)P(draw)
\end{equation}

with

$P(draw)$: Probability of draw,

$P(win)$: Probability of Player 1 winning.

\begin{equation}
    P(win|draw^C) = comb(3,3)p^3p + comb(4,3)p^3(1-p)p + comb(5,3)p^3(1-p)^2p
\end{equation}

Justification. Player 1 has to win 3 games and Player 2 at least 1 and then the last game has to be won by Player 1.

\begin{equation}
    P(draw) = comb(6,3)p^3(1-p)^3
\end{equation}

\begin{equation}
    P(win|draw) = p^2 + 2p(1-p)P(win|draw)
\end{equation}

```python
def nCk(n, k):
    return math.factorial(n) / math.factorial(k) / math.factorial(n-k)
```

```python
def get_prob_win_game(p, verbose=True):
    pWin_nDraw = 0
    for i in range(3):
        pWin_nDraw += nCk(3+i, 3)*p**4*(1-p)**i
    pDraw = nCk(6,3)*p**3*(1-p)**3
    pWin_Draw = p**2 / (1-2*p + 2*p**2)

    pWin = pWin_nDraw*(1-pDraw) + pWin_Draw*pDraw

    if verbose:
        print(f"Probability of winning without draw: \t {pWin_nDraw:.2f}")
        print(f"Probability of winning with draw: \t{pWin_Draw:.2f}")
        print(f"Probability of draw: \t{pDraw:.2f}")
        print(f"Probability of winning the game: \t{pWin:.2f}")

    return pWin
```

# Solution

```python
ps = np.arange(0,1,0.001)
Pgame = []
for p in ps:
    pWin = get_prob_win_game(p, verbose=False)
    Pgame.append(pWin)
```

### And lo and behold, it's a sigmoid centered at 0.5!!!

```python
plt.plot(ps, Pgame)
```

### Badly scaled :(

```python
def sigmoid(x):
    return 1 / (1+np.exp(-x))
sx = np.arange(-10,10,0.001)
sps = sigmoid(sx)
plt.plot(ps, Pgame)
plt.plot(sx, sps)
```


# One-dimensional Lagrange Interpolation

The problem of interpolation or finding the value of a function at an arbitrary point $X$ inside a given domain, provided we have discrete known values of the function inside the same domain is at the heart of the finite element method. In this notebooke we use Lagrange interpolation where the approximation $\hat f(x)$ to the function $f(x)$ is built like:

\begin{equation}
\hat f(x)={L^I}(x)f^I
\end{equation}

In the expression above $L^I$ represents the $I$ Lagrange Polynomial of order $n-1$ and $f^1, f^2,,...,f^n$ are the $n$ known values of the function. Here we are using the summation convention over the repeated superscripts.

The $I$ Lagrange polynomial is given by the recursive expression:

\begin{equation}
{L^I}(x)=\prod_{J=1, J \ne I}^{n}{\frac{{\left( {x - {x^J}} \right)}}{{\left( {{x^I} - {x^J}} \right)}}}
\end{equation}

in the domain $x\in[-1.0,1.0]$.

We wish to interpolate the function $ f(x)=x^3+4x^2-10 $ assuming we know its value at points $x=-1.0$, $x=1.0$ and $x=0.0$.

```python
from __future__ import division
import numpy as np
from scipy import interpolate
import sympy as sym
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
%matplotlib notebook
sym.init_printing()
```

First we use a function to generate the Lagrage polynomial of order $order$ at point $i$

```python
def basis_lagrange(x_data, var, cont):
    """Find the basis for the Lagrange interpolant"""
    prod = sym.prod((var - x_data[i])/(x_data[cont] - x_data[i])
                    for i in range(len(x_data)) if i != cont)
    return sym.simplify(prod)
```

we now define the function $ f(x)=x^3+4x^2-10 $:

```python
fun = lambda x: x**3 + 4*x**2 - 10
```

```python
x = sym.symbols('x')
x_data = np.array([-1, 1, 0])
f_data = fun(x_data)
```

And obtain the Lagrange polynomials using:

```python
basis = []
for cont in range(len(x_data)):
    basis.append(basis_lagrange(x_data, x, cont))
    sym.pprint(basis[cont])

```

    x⋅(x - 1)
    ─────────
        2
    x⋅(x + 1)
    ─────────
        2
       2
    - x  + 1

which are shown in the following plots/

```python
npts = 101
x_eval = np.linspace(-1, 1, npts)
basis_num = sym.lambdify((x), basis, "numpy")  # Create a lambda function for the polynomials
```

```python
plt.figure(figsize=(6, 4))
for k in range(3):
    y_eval = basis_num(x_eval)[k]
    plt.plot(x_eval, y_eval)
```

    <IPython.core.display.Javascript object>

```python
y_interp = sym.simplify(sum(f_data[k]*basis[k] for k in range(3)))
y_interp
```

Now we plot the complete approximating polynomial, the actual function and the points where the function was known.

```python
y_interp = sum(f_data[k]*basis_num(x_eval)[k] for k in range(3))
y_original = fun(x_eval)

plt.figure(figsize=(6, 4))
plt.plot(x_eval, y_original)
plt.plot(x_eval, y_interp)
plt.plot([-1, 1, 0], f_data, 'ko')
plt.show()
```

    <IPython.core.display.Javascript object>

The next cell change the format of the Notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```


# Item I

What is a...

### a) Singular matrix

A square matrix ($n\times n$) $A$ is singular if there doesn't exist a square matrix ($n \times n$) $B$ such that:
$$
BA = I_n \, \wedge \, AB = I_n
$$
where $I_n$ is the $n \times n$ identity matrix. $B$ is denoted as $A^{-1}$.

The following conditions for $A$ are equivalent:
* $A$ is singular.
* Its determinant is 0.
* At least one of its eigenvalues is 0.
* $\text{rank}(A) < n $.
* The equation $A\vec{x} = \vec{0}$ has infinite solutions.

Properties:
* The equation $A\vec{x} = \vec{b}$, with $\vec{b}\neq \vec{0}$, can have zero or infinite solutions.
* $(A^{-1})^{-1} = A$.
* $(kA)^{-1} = k^{-1} A^{-1}$, if $k \neq 0$.
* $(A^T)^{-1} = (A^{-1})^T$
* $\text{det}(A^{-1}) = \text{det}(A)^{-1}$
* $A^{-1} = Q \Lambda^{-1} Q^{-1}$, where $Q$ and $\Lambda$ are matrices obtained from the [eigendecomposition](https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix) of the matrix: $A = Q \Lambda Q^{-1}$.

### b) Vandermonde matrix

Is a $m \times n$ matrix with the following structure:
$$
V = \begin{bmatrix}1&\alpha _{1}&\alpha _{1}^{2}&\dots &\alpha _{1}^{n-1}\\1&\alpha _{2}&\alpha _{2}^{2}&\dots &\alpha _{2}^{n-1}\\1&\alpha _{3}&\alpha _{3}^{2}&\dots &\alpha _{3}^{n-1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&\alpha _{m}&\alpha _{m}^{2}&\dots &\alpha _{m}^{n-1}\end{bmatrix}
$$

* It evaluates a polynomial at a set of points. It maps the coefficients of a polynomial to the value it acquires at the $\alpha_i$'s.
* $\det(V) = \prod_{1\leq i < j \leq n} (\alpha_j-\alpha_i)$.
* A $m \times n$ rectangular Vandermonde matrix such that $m \leq n$ has maximum rank $m$ iff all $x_i$ are distinct.
* A $m \times n$ rectangular Vandermonde matrix such that $n \leq m$ has maximum rank $n$ iff there are $n$ of the $\alpha_i$ that are distinct.
* A $n \times n$ Vandermonde matrix is invertible iff the $\alpha_i$ are distinct. [A direct formula to compute it is known](https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19660023042.pdf).
* The discrete Fourier transform is defined by the [DFT matrix](https://en.wikipedia.org/wiki/DFT_matrix), which is a specific Vandermonde matrix where the numbers $\alpha_i$ are chosen to be roots of unity.

### c) Symmetric matrix

A square matrix $A$ so that $A = A^T$.

* Every square diagonal matrix is symmetric.
* With $A,B$ symmetric, $A+B$ is symmetric.
* With $A,B$ symmetric, iff $AB=BA$, then $AB$ is symmetric.
* Given $n$ integer, $A^n$ is symmetric.
* If $A^{-1}$ exists, it is symmetric.
* If $A \in \mathbb{R}^{n\times n}$, then $\langle A\vec{x}, \vec{y} \rangle = \langle \vec{x}, A\vec{y} \rangle \quad \forall \vec{x},\vec{y}, \in \mathbb{R}^n$
* If $A$ is congruent with $R$, i.e. there exists an invertible matrix $P$ so that $P^T AP = B$, then $B$ is also symmetric.
* Every squared real matrix can be docmposed into two real symmetric matrices.
* If $A \in \mathbb{R}$, it is also hermitian (and has their properties!).

### d) Hermitian matrix

A square matrix $A$ so that $A = A^*$, where $A^* = \overline{A^T}$.
* $a_{ij} = \overline{a_{ji}}$
* All the eigenvalues $A$ are real.
* $A$ is normal, i.e. $A^*A = AA^*$.
* $\langle \vec{v}, A\vec{v} \rangle \in \mathbb{R}$
* If $A$ is also positive-definite (i.e. $\vec{z}^* A \vec{z} > 0, \forall z \neq \vec{0}$), then $A=LL^*$ where $L$ is a lower-triangular matrix.
    * This is the [Cholesky decomposition](https://en.wikipedia.org/wiki/Cholesky_decomposition), more efficient than $LU$ decomposition for solving systems of linear equations.
* $\langle \vec{v}, A \vec{w} \rangle = \langle A\vec{v}, \vec{w} \rangle$
* With $A,B$ hermitian, $A+B$ is hermitian.
* With $A,B$ hermitian, if $AB=BA$, then $AB$ is hermitian.
* Given $n$ integer, $A^n$ is hermitian.
* $A$ can be diagonalized: $A= Q\Lambda Q^T$, where $Q$ is a unitary matrix.
    * $Q$ is composed of orthonormal eigenvectors of $A$ and $\Lambda$ is diagonal, having the eigenvalues of $A$.
    * It holds that $$
    A = \sum_j \lambda_j u_j u_j^* \,,
    $$ where the $u_j$'s are the orthonormal eigenvectors and the $\lambda_j$'s are the eigenvalues.
    * This is the [eigendecomposition](https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix) of $A$.
* $B + B^*$ is Hermitian, for a square matrix $B$.

### e) Skew-hermitian matrix

A square matrix $A$ so that $A^* = -A$, where $A^* = \overline{A^T}$.

The following conditions are equivalent:
* The real part $\mathfrak{R}(A)$ is skew-symmetric and the imaginary part $\mathfrak{I}(A)$ is symmetric.
* $iA$ is Hermitian.
* $-iA$ is Hermitian.
* $x^*Ay=-y^*Ax$ for all vectors $x,y$.

Properties:
* $a_{ij} = -\overline{a_{ji}}$
* All the eigenvalues of a $A$ are purely imaginary (possibly zero).
* It is normal (i.e. $A^*A = AA^*$).
  * $\Rightarrow$ it is diagonalizable ($A = VDV^*$ with $V$ unitary and $D$ diagonal).
  * Its eigenvectors for distinct eigenvalues are orthogonal.
* All $a_{ii}$ (values in the diagonal) have to be purely imaginary (possibly zero).
* With $A,B$ skew-Hermitian, $A+B$ is skew-hermitian.
* $A^k$ is Hermitian if $k$ is even and skew-Hermitian if $k$ is odd.
* $B - B^*$ is skew-Hermitian, for a square matrix $B$.
* An arbitrary square matrix $X$ can be writeen as the sum of a Hermitian matrix $H$ and a skew-Hermitian matrix $S$:
$$
X = H + S \quad \text{with} \quad H = \tfrac{1}{2}(X+X^*) \quad \text{and} \quad S = \tfrac{1}{2}(X-X^*)
$$

### f) Unitary matrix

A square matrix $U$ is unitary if $U^*$ is also its inverse $U^{-1}$:
$$
U^* U = U U^* = I
$$

The following conditions for $U$ are equivalent:
* $U^*$ is unitary.
* $U^{-1} = U^*$
* $U$ is normal and their eigenvalues are in the unit circle.
* The columns of $U$ form an orthonormal basis of $\mathbb{C}^n$.
* The rows of $U$ form an orthonormal basis of $\mathbb{C}^n$.

Properties:
* By definition, it is normal (i.e. $U^*U = UU^*$).
  * $\Rightarrow$ it is diagonalizable ($A = VDV^*$ with $V$ unitary and $D$ diagonal).
  * Its eigenvectors for distinct eigenvalues are orthogonal (orthonormal in this case).
* $|\text{det}(U)| = 1$
* Vector norms are preserved on multiplication.

### g) Jacobian matrix

A matrix $J$ that contains all the first-order partial derivates of a vector-valued function $\mathbf{f}:\mathbb{R}^n \rightarrow \mathbb{R}^m$ :

$$
J(x_1,\dots,x_n) = \left[
\begin{matrix}
\frac{\partial \mathbf{f}}{\partial x_1}
& \cdots
& \frac{\partial \mathbf{f}}{\partial x_n}
\end{matrix} \right]
= \left[
\begin{matrix}
\frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n}
\\ \vdots & \ddots & \vdots
\\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n}
\end{matrix} \right]
$$

* The best linear approximation of $\mathbf{f}$ near a point $\mathbf{p}$ is:
$$
\mathbf{f}(\mathbf{x})-\mathbf{f}(\mathbf{p}) = J(\mathbf{p})(\mathbf{x}-\mathbf{p})
$$
* If $J(\mathbf{x})$ is non-singular, $\mathbf{f}$ is locally invertible near $\mathbf{x}$.
* The inverse of the Jacobian matrix $J$ of $\mathbf{f}$ is the Jacobian matrix of $\mathbf{f}^{-1}$.
* It satiesfies the chain rule:
$$
J_{\mathbf{g} \circ \mathbf{f}}(\mathbf{x}) = J_{\mathbf{g}}(\mathbf{f}(\mathbf{x})) J_{\mathbf{f}}(\mathbf{x})
$$

### h) Projection matrix

A matrix $P$ such that $P^2 = P$ (idempotent), that defines a transformation between the vector space $W$ to itself.

* Its eigenvalues are $0$ or $1$.
* Being $U = \text{Rank}(P)$ and $V = \text{Kern}(P)$:
  * $\forall x \in U : Px = x$, i.e. $P$ is equivalent to the identity on the subspace $U$.
  * Every vector $x \in W$ may be decomposed uniquely as $x = u + v$, with $u = Px$ and $v = x - Px = (I-P) x$, where $u \in U, v \in V$.
* If two projections commute, their product is a projection.
* If $P$ projects onto a line with unit vector $u$, then $P=uu^T$.
* If $P$ projects onto a subspace $U$ with $u_1,\dots,u_k$ an arthonormal basis, then
  $P = AA^T$, where $A$ is the $n \times k$ matrix whose columns are $u_1,\dots,u_k$.

### i) Companion matrix

A square matrix defined for the polinomial:
$$
p(t) = c_0 + c_1 t + \cdots + c_{n-1} t^{n-1} + t^n
$$
as
$$
C(p) = \left[
\begin{matrix}
0 & 0 & \cdots & 0 & -c_0 \\
1 & 0 & \cdots & 0 & -c_1 \\
0 & 1 & \cdots & 0 & -c_2 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & \cdots & 1 & -c_{n-1}
\end{matrix}
\right]
$$

* A matrix $A$ is similar to the companion matrix $C$ of its characteristic polynomial (i.e. $\exists P: C=P^{-1}AP$).
  * Similar matrices share several properties, in that sense, analyzing $C$ could be more easy that analyzing $B$.
  * Rank
  * Characteristic polynomial
    * Determinant
    * Trace
    * Eigenvalues

### j) Jacobi matrix

A square tridiagonal matrix:
$$
A = \left[\begin{matrix}
a_1 & b_1 & 0 & \cdots & 0 & 0 \\
c_1 & a_2 & b_2 & \cdots & 0 & 0 \\
0 & c_2 & a_3 & \cdots & 0 & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & 0 & \cdots & a_{n-1} & b_{n-1} \\
0 & 0 & 0 & \cdots & c_{n-1} & a_{n}
\end{matrix}\right]
$$
so that $a_i \neq 0 , \forall i$.

* The determinant can be computed from a three-term recurrence relation:
  \begin{align}
  f_m &= a_m f_{m-1} - c_{m-1}b_{m-1}f_{m-2} \\
  f_{-1} &= 0 \\
  f_{0} &= 1
  \end{align}
  where each $f_m$ is the determinant of the top-left $m\times m$ submatrix and, thus, $\text{det}(A) = f_n$.
* The inverse can be obtained by a [known recurrence](https://en.wikipedia.org/w/index.php?title=Tridiagonal_matrix&oldid=905913546#Inversion).
* If it is also Toeplitz (i.e. all values in the same diagonal are equal), which means $a_i =a \, \forall i$ ; $b_i =b \, \forall i$ ; $c_i =c \, \forall i$; the eigenvalues are:
$$
\lambda_k = a+2\sqrt{bc} \cos\left(\frac{k\pi}{n+1}\right), \quad k=1,\dots,n
$$
* If $A$ is real and symmetric, then its eigenvalues are real. If all offdiagonal elements are nonzero, then they also are distinct.
* [Numerous methods](https://www.sciencedirect.com/science/article/pii/S1063520312001042?via%3Dihub) exist for the numerical computation of the eigenvalues of a real symmetric tridiagonal matrix.
* With [Lanczos algorithm](https://en.wikipedia.org/wiki/Lanczos_algorithm) it is possible to transform a Hermitian matrix to tridiagonal (and compute its eigenvalues).

### k) Defective matrix

A $n \times n$ matrix $A$ that doesn't have $n$ linearly independent eigenvectors.

* $A$ has fewer than $n$ distinct eigenvalues. Repeated eigenvalues are called *defective*.
* $A$ cannot be diagonalized.
* Normal matrices are never defective.

### l) Toeplitz matrix

A matrix $A$ of size $n \times n$ in which each descending diagonal from left to right is constant.
$$
A = \left[ \begin{matrix}
a_0 & a_{-1} & a_{-2} & \cdots & a_{-(n-1)} \\
a_{1} & a_0 & a_{-1} & \ddots & \vdots \\
a_{2} & a_{1} & a_0 & \ddots & a_{-2} \\
\vdots & \ddots & \ddots & \ddots & a_{-1} \\
a_{n-1} & \cdots & a_2 & a_1 & a_0
\end{matrix}\right]
$$
i.e. it is determined by the constants $c_{-(n{-}1)},\dots,c_{n-1}$, so that
$$
a_{i,j} = c_{i-j}
$$

* The system $Ax = b$ (called Toeplitz system) can be solved using the [Levinson algorithm](https://en.wikipedia.org/wiki/Levinson_recursion) on $\Theta(n^2)$ time.
* $A$ can be decomposed ($LU$) in $O(n^2)$ time.
* If $A$ is symmetric, it can be decomposed as:
$$
\tfrac{1}{a_0} A = G G^T - (G-I) (G-I)^T
$$
where $G$ is the lower triangular part of $\tfrac{1}{a_0}A$.
* If $A$ is nonsingular and symmetric, we have that:
$$
A^{-1} = \tfrac{1}{a_0}(B B^T - C C^T)
$$
where $B$ and $C$ are lower triangular Toeplitz matrices and $C$ is strictly lower.
* It can be used to express a [discrete convolution](https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convolution) as matrix multiplication.

Note: The definition sometimes is extended to include $n \times m$ matrices but most properties don't apply.

### m) Circulant matrix

A Toeplitz matrix with the aditional condition that $a_i = a_{i+m}$.

$$
A = \left[ \begin{matrix}
a_0 & a_{n-1} & a_{n-2} & \cdots & a_{1} \\
a_1 & a_0 & a_{n-1} & \ddots & \vdots \\
a_2 & a_{1} & a_0 & \ddots & a_{n-2} \\
\vdots & \ddots & \ddots & \ddots & a_{n-1} \\
a_{n-1} & \cdots & a_2 & a_1 & a_0
\end{matrix}\right]
$$

The polynomial $f(x) = a_0 + a_1 x + a_2 x^2 + \dots + a_{n-1}x^{n-1}$ is called the *asociated polynomial* of $A$.

* It is fully specified by a vector $\mathbf{a} = a_0,\dots,a_{n-1}$.
* They are diagonalized by a discrete Fourier transform.
  * Equations that contain them may be quickly solved useing a FFT.
  * The following matrix $U_n$ is composed of the eigenvectors of $A$:
  $$
  U_n^* = \tfrac{1}{\sqrt{n}}F_n \quad \text{and} \quad U_n = \sqrt{n}F_n^{-1} \,,
  $$
  where $F_n = [f_{jk}]$ with $f_{jk} = e^{-2jk\pi i/n},\quad 0\leq j,k < n$.
  * $A = U_n \text{diag}(F_n \mathbf{a}) U_n^* = F_n^{-1} \text{diag}(F_n \mathbf{a})F_n.$
  The eigenvalues of $A$ are given by $F_n \mathbf{a}$ which can be calculated using a FFT.
* The normalized eigenvectors are given by:
$$
v_j = \tfrac{1}{\sqrt{n}}(1,\omega_j,\omega_j^2,\dots,\omega_j^{n-1}), \qquad \forall j \in [0,n{-}1] \,,
$$
where $\omega_j = \text{exp}\left( i \tfrac{2 \pi j}{n} \right)$ (the $n$-th roots of unity).
* The eigenvalues are given by:
$$
\lambda_j = f(w_j), \qquad \forall j \in [0,n{-}1] \,.
$$
* $\text{Rank}(A) = n-d$, where $d$ is the degree of the $\text{gcd}(f(x),x^n-1)$.
* If $A$ and $B$ are circulant, $A+B$ is circulant, $AB=BA$ is also circulant.
* Given an equation $A\mathbf{x} = \mathbf{b}$, it can be written as a circular convolution:
  $$
  \mathbf{a} \star \mathbf{x} = \mathbf{b}
  $$
  Then $F_n(\mathbf{a} \star \mathbf{x}) = F_n(\mathbf{a})F_n(\mathbf{x}) = F_n(\mathbf{b})$ and then
  $$
  \mathbf{x} = F_n^{-1}\left[ \left(\frac{(F_n(\mathbf{b}))_v}{(F_n(\mathbf{a}))_v} \right)_{v \in Z} \right]^T
  $$

### n) Hankel matrix

A matrix $A$ of size $n \times n$ in which each **ascending** diagonal from left to right is constant.
$$
A = \left[ \begin{matrix}
a_0 & a_1 & a_2 & \cdots & a_{n-1} \\
a_1 & a_2 & a_3 & \cdots & a_n  \\
a_2 & a_3 & a_4 & \cdots & a_{n+1} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
a_{n-1} & a_{n} & a_{n+1} & \cdots & a_{2n-2}
\end{matrix}\right]
$$
in other words
$$
A = [A_{ij} = a_{i+j-2}]
$$

* It is symmetric.
* The determinant of the particular Hankel matrix:
$$
H_n = \left[h_{ij} = \begin{cases}
0 & \text{ if } i+j-1 > n \\
i+j-1 & \text{otherwise}
\end{cases}\right] = \left[
\begin{matrix}
1 & 2 & 3 & \cdots &  n{-}2 & n{-}1 & n \\
2 & 3 & 4 & \cdots &  n{-}1 & n & 0 \\
3 & 4 & 5 & \cdots &  n  & 0 & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots \\
n{-}2 & n{-}1 & n & \cdots & 0 & 0 & 0\\
n{-}1 & n & 0  & \cdots & 0 & 0 & 0 \\
n  & 0 & 0  & \cdots & 0 & 0 & 0 \\
\end{matrix}
\right]
$$
is given by $\text{det}(H_n) = (-1)^{\lfloor n/2 \rfloor} n^n$.

### o) Hilbert matrix

The Hilbert matrix is an specific $n \times n$ matrix with the following form:
$$
H = \left[h_{ij} = \frac{1}{i+j-1}\right] = \left[\begin{matrix}
1 & \tfrac{1}{2} & \tfrac{1}{3} & \cdots & \tfrac{1}{n} \\
\tfrac{1}{2} & \tfrac{1}{3} & \tfrac{1}{4} & \cdots & \tfrac{1}{n{+}1} \\
\tfrac{1}{3} & \tfrac{1}{4} & \tfrac{1}{5} & \cdots & \tfrac{1}{n{+}2} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\tfrac{1}{n} & \tfrac{1}{n{+}1} & \tfrac{1}{n{+}2} & \cdots & \tfrac{1}{2n{-}1}
\end{matrix}\right]
$$

* They are canonical examples of ill-conditioned matrices.
* It is a Hankel matrix.
  * It is symmetric.
* The determinant is:
$$
\text{det}(H) = \frac{c_n^4}{c_{2n}} \,,
$$
where
$$
c_n = \prod_{i=1}^{n-1} i^{n-i} = \prod_{i=1}^{n-1} i! \, .
$$
* The inverse is given by:
$$
H^{-1} = \left[h'_{ij} = (-1)^{i+j}(i+j-1)\binom{n+i-1}{n-j} \binom{n+j-1}{n-i} \binom{i+j-2}{i-1}^2 \right]
$$
  * All entries are integers.
  * The signs form a checkerboard matrix, with the principal diagonal positive.
* The condition number grows as $O\left((1+\sqrt{2})^{4n}\middle/\sqrt{n}\right)$.

### p) Markov matrix

A matrix that describes the transitions of a Markov chain, where each entry represents a probability (non-negative), there are 3 kinds:
* **right stochastic matrix**: each rows sums 1.
* **left stochastic matrix**: each column sums 1.
* **doubly stochastic matrix**: the whole matrix sums 1.

For the first one we have that
$$
P = [p_{ij}] \, ,
$$
with the condition that $\sum_{j=1}^{n} p_{ij} = 1$, which is equivalent to:
$$
P \mathbf{1} = \mathbf{1}
$$

* The entry $p_{ij}$ represents the probability of transitioning from state $i$ to state $j$, where there are $n$ possible states. State probabilities should be multiplied by the left in order to advance the system.
* The product of two right stochastic matrices is also stochastic: $P' P'' \mathbf{1} = \mathbf{1}$.
* A probability vector $\mathbf{\pi}$  that's also a row eigenvector asociated to the eigenvalue $1$ represents a stationary distribution (that doesn't change under the application of the transition matrix):
$$
\mathbf{\pi} P = \mathbf{\pi} \,.
$$
  * The system evolves over time to a static state:
  $$
  \lim_{k \rightarrow \infty} P^k = \Pi,
  $$
  where al $\Pi$ rows are equal, and coincide with the vector $\mathbf{\pi}$.
* The vector $\mathbf{1}$ is a column eigenvector of $P$.
* The spectral radius is at most $1$, moreover:
$$
|\lambda - \omega| \leq 1-\omega, \quad \text{where } \omega = \min_{1\leq i \leq n} p_{ii}.
$$

### q) Differentiation matrices

Its a matrix that when multiplied with a discrete approximation for a function $y(x)$, at some fixed points $x_j, j \in 0,\dots,n$ retrieves a numerical discrete approximation a derivate of $y$.

These matrices are equivalent to fiting a function $p$ to the data, differentiating this approximation and evaluating this differentiation.

For instance:
$$
D = \frac{1}{h^2}\left[\begin{matrix}
1 & -1 \\
-1 & 2 & -1 \\
& -1 & 2 & -1 \\
& & \ddots & \ddots & \ddots \\
& & & -1 & 2 & -1 \\
& & & & -1 & 2 & -1 \\
& & & & & -1 & 1 \\
\end{matrix}\right]
$$
performs an approximation of $y''(x)$ using second-order central difference.

The approximation is performed:
$$
D \mathbf{y} = \mathbf{y''}
$$
where $\mathbf{y} = [y(x_j)]$ and $\mathbf{y''}$ is the approximation of $[y''(x_j)]$.

* Spectral methods use basis functions that are nonzero over the whole domain (they are **global**), for instance, $p(x)$ being a polynomial of degree $n$. Differentiation matrices are dense.
* Finite element methods use basis functions that are nonzero only on small subdomains (they are **local**), for instance, using $p(x)$ as splines. Differentiation matrices are sparse.
* The spectral element method chooses high degree piecewise polynomials as basis functions, also achieving a very high order of accuracy. Such polynomials are usually orthogonal Chebyshev polynomials or very high order Legendre polynomials over non-uniformly spaced nodes.

These matrices are used to represent the differentiation operator in order to solve differential equations.

Powers $D^n$ can be used to represent the $(n)$ derivate of a function discretization.

### r) Spectral differentiation matrices

Differentiation matrices that get better global approximations for derivates. These approximations have "*spectral accuracy*", i.e. error diminises exponentially with $n$.

On multiplication, retrieves a vector of points $p'(x_j)$ (or a higher order derivate) where $p$ is a single function so that $p(x_j)=y_j$ and this function $p$ is (generally) nonzero over the domain.

* For periodic grids: Fourier methods can be used, $p$ is chosen to be a sum of trigonometric functions.
* For non periodic grids: Chebyshev methods can be used, $p$ is chosen to be a polynomial.

### s) Chebyshev differentiation matrices

A $(n{+}1) \times (n{+}1)$ spectral differentiation matrix that evaluates the function at the Chebyshev points:
$$
x_j = \cos(j\pi/N),\quad j = 0,\dots,N
$$

On multiplication, retrieves a vector of the points $p'(x_j)$ where $p(x)$ is the unique polynomial of degree $N$ that interpolates the input points.

For instance, for $n=2$:
$$
D = \left[
\begin{matrix}
\tfrac{3}{2} & -2 & \tfrac{1}{2} \\
\tfrac{1}{2} & 0 & -\tfrac{1}{2} \\
-\tfrac{1}{2} & 2 & -\tfrac{3}{2}
\end{matrix}
\right]
$$

```python

```


# Chapter 9

*Modeling and Simulation in Python*

Copyright 2021 Allen Downey

License: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)

```python
# check if the libraries we need are installed

try:
    import pint
except ImportError:
    !pip install pint
    import pint

try:
    from modsim import *
except ImportError:
    !pip install modsimpy
    from modsim import *
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
import sympy as sp

t = sp.symbols('t')
t
```

$\displaystyle t$

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
expr
```

$\displaystyle t + 1$

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

$\displaystyle 3$

`f` is a special class of symbol that represents a function.

```python
f = sp.Function('f')
f
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

$\displaystyle f{\left(t \right)}$

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = sp.diff(f(t), t)
dfdt
```

$\displaystyle \frac{d}{d t} f{\left(t \right)}$

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = sp.symbols('alpha')
alpha
```

$\displaystyle \alpha$

Now we can write the differential equation for proportional growth.

```python
eq1 = sp.Eq(dfdt, alpha*f(t))
eq1
```

$\displaystyle \frac{d}{d t} f{\left(t \right)} = \alpha f{\left(t \right)}$

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = sp.dsolve(eq1)
solution_eq
```

$\displaystyle f{\left(t \right)} = C_{1} e^{\alpha t}$

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = sp.symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
particular
```

$\displaystyle f{\left(t \right)} = p_{0} e^{\alpha t}$

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = sp.symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = sp.Eq(sp.diff(f(t), t), r * f(t) * (1 - f(t)/K))
eq2
```

$\displaystyle \frac{d}{d t} f{\left(t \right)} = r \left(1 - \frac{f{\left(t \right)}}{K}\right) f{\left(t \right)}$

And solve it.

```python
solution_eq = sp.dsolve(eq2)
solution_eq
```

$\displaystyle f{\left(t \right)} = \frac{K e^{C_{1} K + r t}}{e^{C_{1} K + r t} - 1}$

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
general
```

$\displaystyle \frac{K e^{C_{1} K + r t}}{e^{C_{1} K + r t} - 1}$

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
at_0
```

$\displaystyle \frac{K e^{C_{1} K}}{e^{C_{1} K} - 1}$

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = sp.solve(sp.Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
value_of_C1
```

$\displaystyle \frac{\log{\left(- \frac{p_{0}}{K - p_{0}} \right)}}{K}$

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
particular
```

$\displaystyle - \frac{K p_{0} e^{r t}}{\left(K - p_{0}\right) \left(- \frac{p_{0} e^{r t}}{K - p_{0}} - 1\right)}$

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = sp.simplify(particular)
particular
```

$\displaystyle \frac{K p_{0} e^{r t}}{K + p_{0} e^{r t} - p_{0}}$

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

$\displaystyle p_{0}$

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
A
```

$\displaystyle \frac{K - p_{0}}{p_{0}}$

```python
logistic = K / (1 + A * sp.exp(-r*t))
logistic
```

$\displaystyle \frac{K}{1 + \frac{\left(K - p_{0}\right) e^{- r t}}{p_{0}}}$

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
sp.simplify(particular - logistic)
```

$\displaystyle 0$

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
beta = sp.symbols('beta')
beta
```

$\displaystyle \beta$

```python
eq3 = sp.Eq(sp.diff(f(t), t), alpha * f(t) + beta * f(t)**2)
eq3
```

$\displaystyle \frac{d}{d t} f{\left(t \right)} = \alpha f{\left(t \right)} + \beta f^{2}{\left(t \right)}$

```python
solution_eq3 = sp.dsolve(eq3)
solution_eq3
```

$\displaystyle f{\left(t \right)} = \frac{\alpha e^{\alpha \left(C_{1} + t\right)}}{\beta \left(1 - e^{\alpha \left(C_{1} + t\right)}\right)}$

```python
general3 = solution_eq3.rhs
general3
```

$\displaystyle \frac{\alpha e^{\alpha \left(C_{1} + t\right)}}{\beta \left(1 - e^{\alpha \left(C_{1} + t\right)}\right)}$

```python
at_03 = general3.subs(t, 0)
at_03
```

$\displaystyle \frac{\alpha e^{C_{1} \alpha}}{\beta \left(1 - e^{C_{1} \alpha}\right)}$

```python
solutions3 = sp.solve(sp.Eq(at_03, p_0), C1)
solutions3[0]
```

$\displaystyle \frac{\log{\left(\frac{\beta p_{0}}{\alpha + \beta p_{0}} \right)}}{\alpha}$

```python
particular3 = sp.simplify(general3.subs(C1, solutions3[0]))
particular3
```

$\displaystyle \frac{\alpha p_{0} e^{\alpha t}}{\alpha - \beta p_{0} e^{\alpha t} + \beta p_{0}}$

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python
Please see the solution above
```


# Gradient Descent Optimizations

Mini-batch and stochastic gradient descent is widely used in deep learning, where the large number of parameters and  limited memory make the use of more sophisticated optimization methods impractical. Many methods have been proposed to accelerate gradient descent in this context, and here we sketch the ideas behind some of the most popular algorithms.

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
```

## Smoothing with exponentially weighted averages

```python
n = 50
x = np.arange(n) * np.pi
y = np.cos(x) * np.exp(x/100) - 10*np.exp(-0.01*x)
```

### Exponentially weighted average

The exponentially weighted average adds a fraction $\beta$ of the current value to a leaky running sum of past values. Effectively, the contribution from the $t-n$th value is scaled by

$$
\beta^n(1 - \beta)
$$

For example, here are the contributions to the current value after 5 iterations (iteration 5 is the current iteration)

| iteration | contribution |
| --- | --- |
| 1 | $\beta^4(1 - \beta)$ |
| 2 | $\beta^3(1 - \beta)$ |
| 3 | $\beta^2(1 - \beta)$ |
| 4 | $\beta^1(1 - \beta)$ |
| 5 | $(1 - \beta)$ |

Since $\beta \lt 1$, the contribution decreases exponentially with the passage of time. Effectively, this acts as a smoother for a function.

```python
def ewa(y, beta):
    """Exponentially weighted average."""

    n = len(y)
    zs = np.zeros(n)
    z = 0
    for i in range(n):
        z = beta*z + (1 - beta)*y[i]
        zs[i] = z
    return zs
```

### Exponentially weighted average with bias correction

Since the EWA starts from 0, there is an initial bias. This can be corrected by scaling with

$$
\frac{1}{1 - \beta^t}
$$

where $t$ is the iteration number.

```python
def ewabc(y, beta):
    """Exponentially weighted average with hias correction."""

    n = len(y)
    zs = np.zeros(n)
    z = 0
    for i in range(n):
        z = beta*z + (1 - beta)*y[i]
        zc = z/(1 - beta**(i+1))
        zs[i] = zc
    return zs
```

```python
beta = 0.9

plt.plot(x, y, 'o-')
plt.plot(x, ewa(y, beta), c='red', label='EWA')
plt.plot(x, ewabc(y, beta), c='orange', label='EWA with bias correction')
plt.legend()
pass
```

## Momentum in 1D

Momentum comes from physics, where the contribution of the gradient is to the velocity, not the position. Hence we create an accessory variable $v$ and increment it with the gradient. The position is then updated with the velocity in place of the gradient. The analogy is that we can think of the parameter $x$ as a particle in an energy well with potential energy $U = mgh$ where $h$ is given by our objective function $f$. The force generated is a function of the rat of change of potential energy $F \propto \nabla U \propto \nabla f$, and we use $F = ma$ to get that the acceleration $a \propto \nabla f$. Finally, we integrate $a$ over time to get the velocity $v$ and integrate $v$ to get the displacement $x$. Note that we need to damp the velocity otherwise the particle would just oscillate forever.

We use a version of the update that simply treats the velocity as an exponentially weighted average popularized by Andrew Ng in his Coursera course. This is the same as the momentum scheme motivated by physics with some rescaling of constants.

```python
def f(x):
    return x**2
```

```python
def grad(x):
    return 2*x
```

```python
def gd(x, grad, alpha, max_iter=10):
    xs = np.zeros(1 + max_iter)
    xs[0] = x
    for i in range(max_iter):
        x = x - alpha * grad(x)
        xs[i+1] = x
    return xs
```

```python
def gd_momentum(x, grad, alpha, beta=0.9, max_iter=10):
    xs = np.zeros(1 + max_iter)
    xs[0] = x
    v = 0
    for i in range(max_iter):
        v = beta*v + (1-beta)*grad(x)
        vc = v/(1+beta**(i+1))
        x = x - alpha * vc
        xs[i+1] = x
    return xs
```

### Gradient descent with moderate step size

```python
alpha = 0.1
x0 = 1
xs = gd(x0, grad, alpha)
xp = np.linspace(-1.2, 1.2, 100)
plt.plot(xp, f(xp))
plt.plot(xs, f(xs), 'o-', c='red')
for i, (x, y) in enumerate(zip(xs, f(xs)), 1):
    plt.text(x, y+0.2, i,
             bbox=dict(facecolor='yellow', alpha=0.5), fontsize=14)
pass
```

### Gradient descent with large step size

When the step size is too large, gradient descent can oscillate and even diverge.

```python
alpha = 0.95
xs = gd(1, grad, alpha)
xp = np.linspace(-1.2, 1.2, 100)
plt.plot(xp, f(xp))
plt.plot(xs, f(xs), 'o-', c='red')
for i, (x, y) in enumerate(zip(xs, f(xs)), 1):
    plt.text(x*1.2, y, i,
             bbox=dict(facecolor='yellow', alpha=0.5), fontsize=14)
pass
```

### Gradient descent with momentum

Momentum results in cancellation of gradient changes in opposite directions, and hence damps out oscillations while amplifying consistent changes in the same direction. This is perhaps clearer in the 2D example below.

```python
alpha = 0.95
xs = gd_momentum(1, grad, alpha, beta=0.9)
xp = np.linspace(-1.2, 1.2, 100)
plt.plot(xp, f(xp))
plt.plot(xs, f(xs), 'o-', c='red')
for i, (x, y) in enumerate(zip(xs, f(xs)), 1):
    plt.text(x, y+0.2, i,
             bbox=dict(facecolor='yellow', alpha=0.5), fontsize=14)
pass
```

## Momentum and RMSprop in 2D

```python
def f2(x):
    return x[0]**2 + 100*x[1]**2
```

```python
def grad2(x):
    return np.array([2*x[0], 200*x[1]])
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
pass
```

```python
def gd2(x, grad, alpha, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0,:] = x
    for i in range(max_iter):
        x = x - alpha * grad(x)
        xs[i+1,:] = x
    return xs
```

```python
def gd2_momentum(x, grad, alpha, beta=0.9, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0, :] = x
    v = 0
    for i in range(max_iter):
        v = beta*v + (1-beta)*grad(x)
        vc = v/(1+beta**(i+1))
        x = x - alpha * vc
        xs[i+1, :] = x
    return xs
```

### Gradient descent with large step size

We get severe oscillations.

```python
alpha = 0.01
x0 = np.array([-1,-1])
xs = gd2(x0, grad2, alpha, max_iter=75)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Vanilla gradient descent')
pass
```

### Gradient descent with momentum

The damping effect is clear.

```python
alpha = 0.01
x0 = np.array([-1,-1])
xs = gd2_momentum(x0, grad2, alpha, beta=0.9, max_iter=75)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Gradieent descent with momentum')
pass
```

### Gradient descent with RMSprop

RMSprop scales the learning rate in each direction by the square root of the exponentially weighted sum of squared gradients. Near a saddle or any plateau, there are directions where the gradient is very small - RMSporp encourages larger steps in those directions, allowing faster escape.

```python
def gd2_rmsprop(x, grad, alpha, beta=0.9, eps=1e-8, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0, :] = x
    v = 0
    for i in range(max_iter):
        v = beta*v + (1-beta)*grad(x)**2
        x = x - alpha * grad(x) / (eps + np.sqrt(v))
        xs[i+1, :] = x
    return xs
```

```python
alpha = 0.1
x0 = np.array([-1,-1])
xs = gd2_rmsprop(x0, grad2, alpha, beta=0.9, max_iter=10)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Gradient descent with RMSprop')
pass
```

### ADAM

ADAM (Adaptive Moment Estimation) combines the ideas of momentum, RMSprop and bias correction. It is probably the most popular gradient descent method in current deep learning practice.

```python
def gd2_adam(x, grad, alpha, beta1=0.9, beta2=0.999, eps=1e-8, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0, :] = x
    m = 0
    v = 0
    for i in range(max_iter):
        m = beta1*m + (1-beta1)*grad(x)
        v = beta2*v + (1-beta2)*grad(x)**2
        mc = m/(1+beta1**(i+1))
        vc = v/(1+beta2**(i+1))
        x = x - alpha * m / (eps + np.sqrt(vc))
        xs[i+1, :] = x
    return xs
```

```python
alpha = 0.1
x0 = np.array([-1,-1])
xs = gd2_adam(x0, grad2, alpha, beta1=0.9, beta2=0.9, max_iter=10)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Gradient descent with RMSprop')
pass
```

## Implementing a custom optimization routine for `scipy.optimize`

Gradient descent is not one of the methods available in `scipy.optimize`. However we can implement our own version by following the API of the `minimize` function.

```python
import scipy.optimize as opt
import scipy.linalg as la
```

```python
def custmin(fun, x0, args=(), maxfev=None, alpha=0.0002,
        maxiter=100000, tol=1e-10, callback=None, **options):
    """Implements simple gradient descent for the Rosen function."""
    bestx = x0
    bestf = fun(x0)
    funcalls = 1
    niter = 0
    improved = True
    stop = False

    while improved and not stop and niter < maxiter:
        niter += 1
        # the next 2 lines are gradient descent
        step = alpha * rosen_der(bestx)
        bestx = bestx - step

        bestf = fun(bestx)
        funcalls += 1

        if la.norm(step) < tol:
            improved = False
        if callback is not None:
            callback(bestx)
        if maxfev is not None and funcalls >= maxfev:
            stop = True
            break

    return opt.OptimizeResult(fun=bestf, x=bestx, nit=niter,
                              nfev=funcalls, success=(niter > 1))
```

```python
def reporter(p):
    """Reporter function to capture intermediate states of optimization."""
    global ps
    ps.append(p)
```

### Test on Rosenbrock banana function

We will use the [Rosenbrock "banana" function](http://en.wikipedia.org/wiki/Rosenbrock_function) to illustrate unconstrained multivariate optimization. In 2D, this is
$$
f(x, y) = b(y - x^2)^2 + (a - x)^2
$$

The function has a global minimum at (1,1) and the standard expression takes $a = 1$ and $b = 100$.

#### Conditioning of optimization problem

With these values for $a$ and $b$, the problem is ill-conditioned. As we shall see, one of the factors affecting the ease of optimization is the condition number of the curvature (Hessian). When the condition number is high, the gradient may not point in the direction of the minimum, and simple gradient descent methods may be inefficient since they may be forced to take many sharp turns.

For the 2D version, we have

$$
f(x) = 100(y - x^2)^2 + (1 - x)^2
$$

and can calculate the Hessian to be

$$
\begin{bmatrix}
802 & -400 \\
-400 & 200
\end{bmatrix}
$$

```python
H = np.array([
    [802, -400],
    [-400, 200]
])
```

```python
np.linalg.cond(H)
```

    2508.009601277298

```python
U, s, Vt = np.linalg.svd(H)
s[0]/s[1]
```

    2508.0096012772983

#### Function to minimize

```python
def rosen(x):
    """Generalized n-dimensional version of the Rosenbrock function"""
    return sum(100*(x[1:]-x[:-1]**2.0)**2.0 +(1-x[:-1])**2.0)
```

```python
def rosen_der(x):
    """Derivative of generalized Rosen function."""
    xm = x[1:-1]
    xm_m1 = x[:-2]
    xm_p1 = x[2:]
    der = np.zeros_like(x)
    der[1:-1] = 200*(xm-xm_m1**2) - 400*(xm_p1 - xm**2)*xm - 2*(1-xm)
    der[0] = -400*x[0]*(x[1]-x[0]**2) - 2*(1-x[0])
    der[-1] = 200*(x[-1]-x[-2]**2)
    return der
```

#### Why is the condition number so large?

```python
x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)
Z = rosen(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
```

```python
# Note: the global minimum is at (1,1) in a tiny contour island
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.text(1, 1, 'x', va='center', ha='center', color='red', fontsize=20)
pass
```

#### Zooming in to the global minimum at (1,1)

```python
x = np.linspace(0, 2, 100)
y = np.linspace(0, 2, 100)
X, Y = np.meshgrid(x, y)
Z = rosen(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
```

```python
plt.contour(X, Y, Z, [rosen(np.array([k, k])) for k in np.linspace(1, 1.5, 10)], cmap='jet')
plt.text(1, 1, 'x', va='center', ha='center', color='red', fontsize=20)
pass
```

#### We will use our custom gradient descent to minimize the banana function

#### Helpful Hint

One of the most common causes of failure of optimization is because the gradient  or Hessian function is specified incorrectly. You can check for this using `check_grad` which compares the analytical gradient with one calculated using finite differences.

```python
from scipy.optimize import check_grad

for x in np.random.uniform(-2,2,(10,2)):
    print(x, check_grad(rosen, rosen_der, x))
```

    [ 1.48907302 -1.42171331] 3.04092560512107e-05
    [-0.48887404  1.21108406] 1.0685858595103707e-06
    [-1.1971752   1.28731448] 9.014975946452588e-06
    [-0.74543228  0.77030133] 3.1276750146010007e-06
    [ 0.24111646 -1.65737227] 7.678560202206516e-06
    [-1.37253259 -1.06738594] 2.2075707844094206e-05
    [-0.16354856  0.61330733] 2.526511024327562e-06
    [-1.20574923  0.20626545] 1.3748984984515249e-05
    [ 1.73282778 -1.84865849] 3.4902945985726736e-05
    [ 0.4104856  -1.81645528] 4.023697388337955e-06

```python
# Initial starting position
x0 = np.array([4,-4.1])
ps = [x0]
opt.minimize(rosen, x0, method=custmin, callback=reporter)
```

         fun: 1.060466347344834e-08
        nfev: 100001
         nit: 100000
     success: True
           x: array([0.9998971 , 0.99979381])

```python
x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)
Z = rosen(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
```

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T))
pass
```

### Comparison with standard algorithms

Note that all these methods take far fewer function iterations and function evaluations to find the minimum compared with vanilla gradient descent.

Many of these are based on estimating the Newton direction. Recall Newton's method for finding roots of a univariate function

$$
x_{K+1} = x_k - \frac{f(x_k)}{f'(x_k)}
$$

When we are looking for a minimum, we are looking for the roots of the *derivative* $f'(x)$, so

$$
x_{K+1} = x_k - \frac{f'(x_k}{f''(x_k)}
$$

Newton's method can also be seen as a Taylor series approximation

$$
f(x+h) = f(x) + h f'(x) + \frac{h^2}{2}f''(x)
$$

At the function minimum, the derivative is 0, so
\begin{align}
\frac{f(x+h) - f(x)}{h} &= f'(x) + \frac{h}{2}f''(x) \\
0 &= f'(x) + \frac{h}{2}f''(x)
\end{align}

and letting $\Delta x = \frac{h}{2}$, we get that the Newton step is

$$
\Delta x = - \frac{f'(x)}{f''(x)}
$$

The multivariate analog replaces $f'$ with the Jacobian and $f''$ with the Hessian, so the Newton step is

$$
\Delta x = -H^{-1}(x) \nabla f(x)
$$

Slightly more rigorously, we can optimize the quadratic multivariate Taylor expansion

$$
f(x + p) = f(x) + p^T\nabla f(x) + \frac{1}{2}p^TH(x)p
$$

Differentiating with respect to the direction vector $p$ and setting to zero, we get

$$
H(x)p = -\nabla f(x)
$$

giving

$$
p = -H(x)^{-1}\nabla f(x)
$$

```python
from scipy.optimize import rosen, rosen_der, rosen_hess
```

#### Nelder-Mead

There are some optimization algorithms not based on the Newton method, but on other heuristic search strategies that do not require any derivatives, only function evaluations. One well-known example is the Nelder-Mead simplex algorithm.

```python
ps = [x0]
opt.minimize(rosen, x0, method='nelder-mead', callback=reporter)
```

     final_simplex: (array([[0.99998846, 0.99997494],
           [0.99994401, 0.99989075],
           [1.0000023 , 1.0000149 ]]), array([5.26275688e-10, 3.87529507e-09, 1.06085894e-08]))
               fun: 5.262756878429089e-10
           message: 'Optimization terminated successfully.'
              nfev: 162
               nit: 85
            status: 0
           success: True
                 x: array([0.99998846, 0.99997494])

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T));
```

#### BFGS

As calculating the Hessian is computationally expensive, sometimes first order methods that only use the first derivatives are preferred. Quasi-Newton methods use functions of the first derivatives to approximate the inverse Hessian. A well know example of the Quasi-Newoton class of algorithjms is BFGS, named after the initials of the creators. As usual, the first derivatives can either be provided via the `jac=` argument or approximated by finite difference methods.

```python
ps = [x0]
opt.minimize(rosen, x0, method='Newton-CG', jac=rosen_der, hess=rosen_hess, callback=reporter)
```

         fun: 1.3642782750354208e-13
         jac: array([ 1.21204353e-04, -6.08502470e-05])
     message: 'Optimization terminated successfully.'
        nfev: 38
        nhev: 26
         nit: 26
        njev: 63
      status: 0
     success: True
           x: array([0.99999963, 0.99999926])

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T))
pass
```

#### Newton-CG

Second order methods solve for $H^{-1}$ and so require calculation of the Hessian (either provided or approximated using finite differences). For efficiency reasons, the Hessian is not directly inverted, but solved for using a variety of methods such as conjugate gradient. An example of a second order method in the `optimize` package is `Newton-GC`.

```python
ps = [x0]
opt.minimize(rosen, x0, method='Newton-CG', jac=rosen_der, hess=rosen_hess, callback=reporter)
```

         fun: 1.3642782750354208e-13
         jac: array([ 1.21204353e-04, -6.08502470e-05])
     message: 'Optimization terminated successfully.'
        nfev: 38
        nhev: 26
         nit: 26
        njev: 63
      status: 0
     success: True
           x: array([0.99999963, 0.99999926])

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T))
pass
```