Chapter 4 of [A Guided Tour of Mathematical Methods for the Physical Sciences](http://www.cambridge.org/9780521542616) concerns different coordinate systems and how to navigate between these. To illustrate the power of coordinate transformations, let us consider the Gaussian function. Gaussians play an important role in probablity and inverse theory, and as we'll see, its integral comes up in often in Physical problems.

In its most general form, the Gaussian is $$ f(x)=ae^{-{\frac {(x-b)^{2}}{2c^{2}}}}.$$ First, we will define a function for Gaussians:

```python
import numpy as np

def gaussian(x,a,b,c):
    return a*np.exp(-(x-b)**2/(2*c**2))
```

Let us plot a specific Gaussian for $a=c=1, b=0$. In other words, $f(x)=e^{-x^{2}}$

```python
import matplotlib.pyplot as plt

a=1
b=0
c=np.sqrt(1/2)
dx=0.1
x = np.arange(-10,10,dx)

plt.plot(x,gaussian(x,a,b,c))
plt.xlabel('x')
plt.ylabel('f(x)')
plt.show()
```

It may or may not be obvious from the graph and/or the equation, but $a$ controls the height of the bell curve, $b$ determines the horizontal position, and $c$ the width of the curve.

The area under this curve, the integral: $$\int _{-\infty }^{\infty }e^{-x^{2}}dx$$ is of interest in many Physics problems. In our book, we need to determine this integral to find the Green's function for the heat equation, for example. The solution is not so simple, because there is no analytic anti-derivative for this problem. One option is always to approximate an integral numerically. Let's try that, and in the process learn some new python tricks!

The Gausian integral is the area under this curve. One option is turn this area into skinny rectangles of width $dx$ and add the areas of each rectangle. A function to compute the area of all the rectangles made up of height $f(x)$ and width $dx$ is

```python
def g(a,b,c,x,dx):
    return np.sum(gaussian(x,a,b,c)*dx)
```

```python
print(g(a,b,c,x,dx))
```

    1.7724538509055225

Apparently, this is the numerical estimate of the aread under our Gaussian. To confirm, we can take advantage of the numerical integration capabilities in Python, with the [quad function](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.quad.html) in the scipy package. (And numpy even knows about infinity!)

```python
from scipy.integrate import quad
ans, err = quad(gaussian, -np.inf, np.inf,args =(a,b,c))
print(ans)
```

    1.772453850905516

Our two results agree to many significant digits, despite one of the methods being our "crude" summation of rectangles, limited to the region between -10 and 10. But what does this 1.77245 value mean?

Let's see if we cannot get an analytic solution to shed some light on this.
$$
\int _{-\infty }^{\infty }e^{-x^{2}}dx =
\sqrt{\int _{-\infty }^{\infty }\int _{-\infty }^{\infty }e^{-x^{2}-y^2}dx dy}$$

It appears we made the problem harder, not easier, by going from a 1D integral to one in two dimensions, but $x^2+y^2 = r^2$, the radius of this 2-dimensional Gaussian, and we can solve the integral in polar coordinates:

\begin{align}
\int _{-\infty }^{\infty }e^{-x^{2}}dx =
\sqrt{\int _{-\infty }^{\infty }\int _{-\infty }^{\infty }e^{-x^{2}-y^2}dx dy} \\
\sqrt{\int _{-\infty }^{\infty }\int_0^{2\pi}e^{-r^{2}} rdr d\theta}
\end{align}

How the integration area dxdy in polar coordinates becomes $rdrd\theta$ (with an "extra" $r$) is explained in Chapter 4 of our book. This factor forms the "Jacobian" of the transfer from Cartesian to Polar coordidates. We can, however, present some intuitive insight to the elementary area of Cartesian and polar coordinates. The Cartesian element is $dA= dx dy$:

In polar coordinates, the elementary area is a small arc length times a small but of radius: $dA = rd\theta dr$, as you can see in the drawing below. If the arc length is mysterious to you, consider putting so many of these together that you get a full circle: $\int_{2\pi}rd\theta = 2\pi r$:

After the conversion to polar coordinates, we can determine the anti-derivative after a simple change of variables, and solve our original Gaussian integral:

\begin{align}
\iint_{-\infty}^\infty e^{-(x^{2}+y^{2})}dx dy &=\int _{0}^{2\pi }\int _{0}^{\infty }e^{-r^{2}}r\,dr\,d\theta \\&=2\pi \int _{0}^{\infty }re^{-r^{2}}\,dr\\&=2\pi \int _{-\infty }^{0}{\tfrac {1}{2}}e^{s}\,ds&&s=-r^{2}\\&=\pi \int _{-\infty }^{0}e^{s}\,ds\\&=\pi (e^{0}-e^{-\infty })\\&=\pi.
\end{align}

This means that our original problem is
$$\int _{-\infty }^{\infty }e^{-x^{2}}dx = \sqrt{\iint_{-\infty}^\infty e^{-(x^{2}+y^{2})}dx dy}  = \sqrt{\pi}.$$
And let us check that against our numerical answer:

```python
np.sqrt(np.pi)
```

    1.7724538509055159

Lo and behold, our (two) numerical and analytic solutions agree! Chapter 4 explains how to go from 3D Cartesian to Spherical coordinates, which will prove handy when we get to 3D problems with such a spherical symmetry. Also, we may revisit the Gaussian function in our python notebook about the Dirac Delta function of Chapter 13.


```python
# setup by importing some good modules
import sympy as sp
# All calls to sympy require sp. at the beginning
# One could use "from sympy import *", but then it's difficult to know what is sympy
# and what is another module when importing multiple modules.
# print things all pretty
from sympy.abc import *
sp.init_printing(use_latex='mathjax')
```

```python
sp.integrate(sp.cos(x-t)**2, x)
```

$$\frac{x}{2} \sin^{2}{\left (t - x \right )} + \frac{x}{2} \cos^{2}{\left (t - x \right )} - \frac{1}{2} \sin{\left (t - x \right )} \cos{\left (t - x \right )}$$

```python

```


```python
from sympy import *

pprint("\nDistributed load:")
q, l, xi = var("q, l, xi")

N1 = -xi**3 + 2*xi**2 - xi
N2 = 2*xi**3 - 3*xi**2 + 1
N3 = -xi**3 + xi**2
N4 = -2*xi**3 + 3*xi**2

pprint("\nq₁:")
q1 = 2*q*xi
pprint(q1)
pprint("\nq₂:")
q2 = 2*q*(1 - xi)
pprint(q2)

M1 =  integrate( N1*q1*l *l, (xi, 0, S(1)/2))
M1 += integrate( N1*q2*l *l, (xi, S(1)/2, 1))

F1 =  integrate( N2*q1 *l, (xi, 0, S(1)/2))
F1 += integrate( N2*q2 *l, (xi, S(1)/2, 1))

M2 =  integrate( N3*q1*l *l, (xi, 0, S(1)/2))
M2 += integrate( N3*q2*l *l, (xi, S(1)/2, 1))

F2 =  integrate( N4*q1 *l, (xi, 0, S(1)/2))
F2 += integrate( N4*q2 *l, (xi, S(1)/2, 1))

f = Matrix([M1, F1, M2, F2])
pprint("\nNodal loads:")
pprint(f)

# Distributed load:
#
# q₁:
# 2⋅q⋅ξ
#
# q₂:
# 2⋅q⋅(-ξ + 1)
#
# Nodal loads:
# ⎡    2   ⎤
# ⎢-5⋅l ⋅q ⎥
# ⎢────────⎥
# ⎢   96   ⎥
# ⎢        ⎥
# ⎢  l⋅q   ⎥
# ⎢  ───   ⎥
# ⎢   4    ⎥
# ⎢        ⎥
# ⎢    2   ⎥
# ⎢ 5⋅l ⋅q ⎥
# ⎢ ────── ⎥
# ⎢   96   ⎥
# ⎢        ⎥
# ⎢  l⋅q   ⎥
# ⎢  ───   ⎥
# ⎣   4    ⎦

```


**Brief Honor Code**. Do the homework on your own. You may discuss ideas with your classmates, but DO NOT copy the solutions from someone else or the Internet. If stuck, discuss with TA.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

**1**. (20 points)

Find the gradient and Hessian for the following equation

$$
f(x, y) = 1 + 2x + 3y + 4x^2 + 2xy + y^2
$$

- Plot the contours of this function using `matplotlib` in the box $-5 \le x \le 5$ and $-5 \le y \le 5$ using a $100 \times 100$ grid.
- Then plot the gradient vectors using the `quiver` function on top of the contour plot using a $10 \times 10$ grid. Are the gradients orthogonal to the contours?

Hint: Use `numpy.meshgrid`, `matplotlib.contour` and `matplotllib.quiver`.

Gradient is

$$
\begin{bmatrix}
2 + 8x + 2y \\
3 + 2x + 2y
\end{bmatrix}
$$

Hessian is
$$
\begin{bmatrix}
8 & 2 \\
2 & 2
\end{bmatrix}
$$

```python
x = np.linspace(-10, 10, 100)
y = np.linspace(-10, 10, 100)
X, Y = np.meshgrid(x, y)
Z = 1 + 2*X + 3*Y + 4*X**2 + 2*X*Y + Y**2
plt.contour(X, Y, Z, 15)

x = np.linspace(-10, 10, 10)
y = np.linspace(-10, 10, 10)
X, Y = np.meshgrid(x, y)
U = 2 + 8*X + 2*Y
V = 3 + 2*X + 2*Y
plt.quiver(X, Y, U, V, edgecolor='k', facecolor='r', linewidth=.5, minlength=5)

plt.axis('square')
pass
```

**2**. (30 points)

This exercise is about using Newton's method to find the cube roots of unity - find $z$ such that $z^3 = 1$. From the fundamental theorem of algebra, we know there must be exactly 3 complex roots since this is a degree 3 polynomial.

We start with Euler's equation
$$
e^{ix} = \cos x + i \sin x
$$

Raising $e^{ix}$ to the $n$th power where $n$ is an integer, we get from Euler's formula with $nx$ substituting for $x$
$$
(e^{ix})^n = e^{i(nx)} = \cos nx + i \sin nx
$$

Whenever $nx$ is an integer multiple of $2\pi$, we have
$$
\cos nx + i \sin nx = 1
$$

So
$$
e^{2\pi i \frac{k}{n}}
$$
is a root of 1 whenever $k/n = 0, 1, 2, \ldots$.

So the cube roots of unity are $1, e^{2\pi i/3}, e^{4\pi i/3}$.

While we can do this analytically, the idea is to use Newton's method to find these roots, and in the process, discover some rather perplexing behavior of Newton's method.

```python
from sympy import Symbol, exp, I, pi, N, expand
```

```python
expand(exp(2*pi*I/3), complex=True)
```

    -1/2 + sqrt(3)*I/2

```python
expand(exp(4*pi*I/3), complex=True)
```

    -1/2 - sqrt(3)*I/2

```python
plt.figure(figsize=(4,4))
roots = np.array([[1,0], [-0.5, np.sqrt(3)/2], [-0.5, -np.sqrt(3)/2]])
plt.scatter(roots[:,0], roots[:,1], s=50, c='red')
xp = np.linspace(0, 2*np.pi, 100)
plt.plot(np.cos(xp), np.sin(xp), c='blue');
```

Newton's method for functions of complex variables - stability and basins of attraction. (30 points)

1. Write a function with the following function signature `newton(z, f, fprime, max_iter=100, tol=1e-6)` where
    - `z` is a starting value (a complex number e.g.  ` 3 + 4j`)
    - `f` is a function of `z`
    - `fprime` is the derivative of  `f`
The function will run until either max_iter is reached or the absolute value of the Newton step is less than tol. In either case, the function should return the number of iterations taken and the final value of `z` as a tuple (`i`, `z`).

2. Define the function `f` and `fprime` that will result in Newton's method finding the cube roots of 1. Find 3 starting points that will give different roots, and print both the start and end points.

Write the following two plotting functions to see some (pretty) aspects of Newton's algorithm in the complex plane.

3. The first function `plot_newton_iters(f, fprime, n=200, extent=[-1,1,-1,1], cmap='hsv')` calculates and stores the number of iterations taken for convergence (or max_iter) for each point in a 2D array. The 2D array limits are given by `extent` - for example, when `extent = [-1,1,-1,1]` the corners of the plot are `(-i, -i), (1, -i), (1, i), (-1, i)`. There are `n` grid points in both the real and imaginary axes. The argument `cmap` specifies the color map to use - the suggested defaults are fine. Finally plot the image using `plt.imshow` - make sure the axis ticks are correctly scaled. Make a plot for the cube roots of 1.

4. The second function `plot_newton_basins(f, fprime, n=200, extent=[-1,1,-1,1], cmap='jet')` has the same arguments, but this time the grid stores the identity of the root that the starting point converged to. Make a plot for the cube roots of 1 - since there are 3 roots, there should be only 3 colors in the plot.

```python
def newton(z, f, fprime, max_iter=100, tol=1e-6):
    """The Newton-Raphson method."""
    for i in range(max_iter):
        step = f(z)/fprime(z)
        if abs(step) < tol:
            return i, z
        z -= step
    return i, z
```

```python
def plot_newton_iters(p, pprime, n=200, extent=[-1,1,-1,1], cmap='hsv'):
    """Shows how long it takes to converge to a root using the Newton-Rahphson method."""
    m = np.zeros((n,n))
    xmin, xmax, ymin, ymax = extent
    for r, x in enumerate(np.linspace(xmin, xmax, n)):
        for s, y in enumerate(np.linspace(ymin, ymax, n)):
            z = x + y*1j
            m[s, r] = newton(z, p, pprime)[0]
    plt.imshow(m, cmap=cmap, extent=extent)
```

```python
def plot_newton_basins(p, pprime, n=200, extent=[-1,1,-1,1], cmap='jet'):
    """Shows basin of attraction for convergence to each root using the Newton-Raphson method."""
    root_count = 0
    roots = {}

    m = np.zeros((n,n))
    xmin, xmax, ymin, ymax = extent
    for r, x in enumerate(np.linspace(xmin, xmax, n)):
        for s, y in enumerate(np.linspace(ymin, ymax, n)):
            z = x + y*1j
            root = np.round(newton(z, p, pprime)[1], 1)
            if not root in roots:
                roots[root] = root_count
                root_count += 1
            m[s, r] = roots[root]
    plt.imshow(m, cmap=cmap, extent=extent)
```

```python
plt.grid('off')
plot_newton_iters(lambda x: x**3 - 1, lambda x: 3*x**2)
```

```python
plt.grid('off')
m = plot_newton_basins(lambda x: x**3 - 1, lambda x: 3*x**2)
```

**3**. (20 points)

Consider the following function on $\mathbb{R}^2$:

$$
f(x_1,x_2) = -x_1x_2e^{-\frac{(x_1^2+x_2^2)}{2}}
$$

- Find the minimum under the constraint
$$g(x) = x_1^2+x_2^2 \leq 10$$
and
$$h(x) = 2x_1 + 3x_2 = 5$$ using `scipy.optimize.minimize`.
- Plot the function contours using `matplotlib`, showing the constraints $g$ and $h$ and indicate the constrained minimum with an `X`.

```python
import scipy.optimize as opt
```

```python
def f(x):
    return -x[0] * x[1] * np.exp(-(x[0]**2+x[1]**2)/2)

cons = ({'type': 'eq',
         'fun' : lambda x: np.array([2.0*x[0] + 3.0*x[1] - 5.0]),
         'jac' : lambda x: np.array([2.0,3.0])},
        {'type': 'ineq',
         'fun' : lambda x: np.array([-x[0]**2.0 - x[1]**2.0 + 10.0])})

x0 = [1.5,1.5]
cx = opt.minimize(f, x0, constraints=cons)
```

```python
x = np.linspace(-5, 5, 200)
y = np.linspace(-5, 5, 200)
X, Y = np.meshgrid(x, y)
Z = f(np.vstack([X.ravel(), Y.ravel()])).reshape((200,200))
plt.contour(X, Y, Z)

# g constraint
plt.plot(x, (5-2*x)/3, 'k:', linewidth=1)

# h constraint
theta = np.linspace(0, 2*np.pi, 100)
x = np.sqrt(10) * np.cos(theta)
y = np.sqrt(10) * np.sin(theta)
plt.plot(x, y, 'k:', linewidth=1)
plt.fill_between(x,y,alpha=0.15)
plt.text(cx['x'][0], cx['x'][1], 'x', va='center', ha='center', size=20, color='red')
plt.axis([-5,5,-5,5])
plt.title('Contour plot of f(x) subject to constraints g(x) and h(x)')
plt.xlabel('x1')
plt.ylabel('x2')
pass
```

**4** (30 points)

Find solutions to $x^3 + 4x^2 -3 = x$.

- Write a function to find brackets, assuming roots are always at least 1 unit apart and that the roots lie between -10 and 10
- For each bracket, find the enclosed root using
    - a bisection method
    - Newton-Raphson (no guarantee to stay within brackets)
- Use the end points of the bracket as starting points for the bisection methods and the midpoint for Newton-Raphson.
- Use the companion matrix and characteristic polynomial to find the solutions
- Plot the function and its roots (marked with a circle) in a window just large enough to contain all roots.

Use a tolerance of 1e-6.

```python
def f(x):
    """Fucntion to find zeros for."""

    return x**3 + 4*x**2 -x - 3
```

```python
def fp(x):
    """Derivative of f."""
    return 3*x**2 + 8*x -1
```

### Bracketing function

```python
def bracket(f, start, stop, step):
    """Find brackets where end points have different signs."""

    brackets = []
    for a in np.arange(start, stop, step):
        b = a + step
        if f(a) * f(b) < 0:
            brackets.append([a, b])
    return brackets
```

```python
brackets = bracket(f, -10, 10, 1)
brackets
```

    [[-5, -4], [-1, 0], [0, 1]]

### Bisection

```python
def bisect(f, a, b, tol=1e-6):
    """Bisection method."""

    while np.abs(b - a) >= tol:
        c = (a + b)/2
        if (f(a) * f(c )) < 0:
            b = c
        else:
            a = c
    return c
```

```python
for bracket in brackets:
    a, b = bracket
    x = bisect(f, a, b)
    print(x)
```

    -4.064435005187988
    -0.827519416809082
    0.8919553756713867

### Newton-Raphson

```python
def newton(f, fp, x, tol=1e-6):
    """Newton Raphson method."""

    while np.abs(f(x)) >= tol:
        x = x - f(x)/fp(x)
    return x
```

```python
for bracket in brackets:
    a, b = bracket
    x = (a + b)/2
    x = newton(f, fp, x)
    print(x)
```

    -4.064434533802053
    -0.8275200557849137
    0.8919544406642576

### Companion matrix

```python
A = np.array([
    [-4, 1, 3],
    [1,0,0],
    [0,1,0]
])
```

```python
roots = np.linalg.eigvals(A)
roots
```

    array([-4.06443453,  0.89195444, -0.82751991])

```python
x = np.linspace(-5, 2, 100)
plt.plot(x, f(x))
plt.axhline(0, c='black')
plt.scatter(roots, np.zeros_like(roots), s=50, c='red')
pass
```

```python

```


# Information Theoretic Quantities

The following presents key concepts of Information Theory that will be used later to train generative models

## How can we measure information?

Information Theory is the mathematical study of the quantification and transmission of information proposed by **Claude Shannon** on this seminal work: *A Mathematical Theory of Communication*, 1948

Shannon considered the output of a noisy source as a random variable $X$

- The RV takes $M$ possible values $\mathcal{A} = \{x_1, x_2, x_3, \ldots, x_M\}$
- Each value $x_i$ have an associated probability $P(X=x_i) = p_i$

Consider the following question: What is the amount of information carried by $x_i$?

Shannon defined the amount of information as

$$
I(x_i) = \log_2 \frac{1}{p_i},
$$

which is measured in **bits**

:::{note}

One bit is the amount of information needed to choose between two **equiprobable** states

:::

**Example:** A meteorological station in 1920 that sends tomorrow's weather prediction from Niebla to [Valdivia](https://en.wikipedia.org/wiki/Valdivia) via telegraph

Tomorrows weather is a random variable

- The dictionary of messages: (1) Rainy, (2) Cloudy, (3) Partially cloudy, (4) Sunny
- Assume thta their probabilities are: $p_1=1/2$, $p_2=1/4$, $p_3=1/8$, $p_4=1/8$

What is the minimum number of yes/no questions (equiprobable) needed to guess tomorrow's weather?

- Is it going to rain?
- No: Is it going to be cloudy?
- No: Is it going to be sunny?

What is then the amount amount of information of each message?

- Rainy: $\log_2 \frac{1}{p_1} = \log_2 2 = 1$ bits
- Cloudy: $2$ bits
- Partially cloudy and Sunny: $3$ bits

:::{important}

The larger the probability the smallest information it carries

:::

## Shannon's entropy

After defining the amount of information for a state Shannon's defined the average information of the source $X$ as

$$
\begin{align}
H(X) &= \mathbb{E}_{x\sim X}\left [\log_2 \frac{1}{P(x)} \right] \nonumber \\
&= - \sum_{x\in \mathcal{A}} P(x) \log_2 P(X)  \nonumber \\
&= - \sum_{i=1}^M p_i \log_2 p_i  ~ \text{[bits]} \nonumber
\end{align}
$$

and called it the **entropy** of the source

:::{note}

Entropy is the "average information of the source"

:::

**Properties:**

- Entropy is nonnegative: $H(X)>0$
- Entropy is equal to zero when $p_j = 1 \wedge p_i = 0, i \neq j$
- Entropy is maximum when $X$ is uniformly distributed $p_i = \frac{1}{M}$, $H(X) = \log_2(M)$

:::{note}

The more random the source is the larger its entropy

:::

For continuous variables the Differential entropy is defined as

$$
H(p) = - \int p(x) \log p(x) \,dx ~ \text{[nats]}
$$

where $p(x)$ is the probability density function (pdf) of $X$

## Relative Entropy: Kullback-Leibler (KL) divergence

Consider a continuous random variable $X$ and two distributions $q(x)$ and $p(x)$ defined on its probability space

The relative entropy between these distributions is

$$
\begin{align}
D_{\text{KL}} \left [ p(x) || q(x) \right] &= \mathbb{E}_{x \sim p(x)} \left [ \log \frac{p(x)}{q(x)} \right ] \nonumber \\
&= \mathbb{E}_{x \sim p(x)} \left [ \log p(x) \right ]  - \mathbb{E}_{x \sim p(x)} \left [ \log q(x) \right ],  \nonumber \\
&= \int p(x) \log p(x) \,dx  - \int p(x) \log q(x) \,dx  \nonumber
\end{align}
$$

which is also known as the **[Kullback](https://en.wikipedia.org/wiki/Solomon_Kullback)- [Leibler](https://en.wikipedia.org/wiki/Richard_Leibler) divergence**

- The left hand side term is the negative entropy of $p(x)$
- The right hand side term is called the **cross-entropy of $q(x)$ relative to $p(x)$**

**Intepretations of KL**

- Coding: Expected number of "extra bits" needed to code $p(x)$ using a code optimal for $q(x)$
- Bayesian modeling: Amount of information lost when $q(x)$ is used as a model for $p(x)$

**Property**: Non-negativity

The KL divergence is non-negative

$$
D_{\text{KL}} \left [ p(x) || q(x) \right] \geq 0
$$

with the equality holding for $p(x) \equiv q(x)$

This is given by the [Gibbs inequality](https://en.wikipedia.org/wiki/Gibbs%27_inequality)

$$
- \int p(x) \log p(x) \,dx  \leq - \int p(x) \log q(x) \,dx
$$

:::{note}

The entropy of $p(x)$ is equal or less than the cross-entropy of $q(x)$ relative to $p(x)$

:::

**Property**: Asymmetry

The KL divergence is asymmetric

$$
D_{\text{KL}} \left [ p(x) || q(x) \right] \neq D_{\text{KL}} \left [ q(x) || p(x) \right]
$$

- The KL is not a proper distance (no triangle inequility either)
- Forward and Reverse KL have different meanings (we will explore them soon)

**Property**: Relation with mutual information

The KL is related to the mutual information between random variables as

$$
\text{MI}(X, Y) = D_{\text{KL}} \left [ p(x, y) || p(x)p(y) \right]
$$

:::{seealso}

See D. Mackays' book chapter 1 for more details on Information Theory

:::


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import Symbol, integrate
%matplotlib notebook
```

### Smooth local paths
We will use cubic spirals to generate smooth local paths. Without loss of generality, as $\theta$ smoothly changes from 0 to 1, we impose a condition on the curvature as follows

$\kappa = f'(x) = K(x(1-x))^n $

This ensures curvature vanishes at the beginning and end of the path. Integrating, the yaw changes as
$\theta = \int_0^x f'(x')dx'$

With $n = 1$ we get a cubic spiral, $n=2$ we get a quintic spiral and so on. Let us use the sympy package to find the family of spirals

1. Declare $x$ a Symbol

2. You want to find Integral of $f'(x)$

3. You can choose $K$ so that all coefficients are integers

Verify if $\theta(0) = 0$ and $\theta(1) = 1$

```python
K =  30 #choose for cubic/quintic
n =  2 #choose for cubic/ quintic
x =  Symbol('x') #declare as Symbol
print(integrate(K*(x*(1-x))**n, x)) # complete the expression
```

    6*x**5 - 15*x**4 + 10*x**3

```python
x = np.linspace(0, 1, num=100)
thetas = -2*x**3 + 3*x**2
plt.figure()
plt.plot(x, thetas,'.')
thetas = 6*x**5 - 15*x**4 + 10*x**3
plt.plot(x, thetas,'.')
```

```python
#write function to compute a cubic spiral
#input can be any theta_i and theta_f (not just 0 and 1)
def cubic_spiral(theta_i, theta_f, n=10):
    x = np.linspace(0, 1, num=n)
    #-2*x**3 + 3*x**2
    return (theta_f-theta_i)*(-2*x**3 + 3*x**2) + theta_i

def quintic_spiral(theta_i, theta_f, n=10):
    x = np.linspace(0, 1, num=n)
    #6*x**5 - 15*x**4 + 10*x**3
    return (theta_f-theta_i)*(6*x**5 - 15*x**4 + 10*x**3) + theta_i
```

### Plotting
Plot cubic, quintic spirals along with how $\theta$ will change from $\pi/2$ to $0$ when moving in a circular arc. Remember circular arc is when  $\omega $ is constant

```python
num_pts = 100
plt.figure()
plt.plot(np.pi/2*(1-np.linspace(0,1,num_pts)), label='Circular')
plt.plot(cubic_spiral(np.pi/2, 0, num_pts), label='Cubic')
plt.plot(quintic_spiral(np.pi/2, 0, num_pts),label='Quintic')
plt.grid()
plt.legend()
```

## Trajectory

Using the spirals, convert them to trajectories $\{(x_i,y_i,\theta_i)\}$. Remember the unicycle model

$dx = v\cos \theta dt$

$dy = v\sin \theta dt$

$\theta$ is given by the spiral functions you just wrote. Use cumsum() in numpy to calculate {(x_i, y_i)}

What happens when you change $v$?

```python
num_pts = 50
v = 1
dt = 0.1

#cubic
theta = cubic_spiral(np.pi/2, np.pi, num_pts)
x = np.cumsum(v*np.cos(theta)*dt)
y = np.cumsum(v*np.sin(theta)*dt)

#Quintic
theta = quintic_spiral(np.pi/2, np.pi, num_pts+2)
xq = np.cumsum(v*np.cos(theta)*dt)
yq = np.cumsum(v*np.sin(theta)*dt)

#Circular
theta = np.pi/2*(1+np.linspace(0,1,num_pts-2))
xc = np.cumsum(v*np.cos(theta)*dt)
yc = np.cumsum(v*np.sin(theta)*dt)

# plot trajectories for circular/ cubic/ quintic
plt.figure()
plt.plot(xc, yc, label='Circular')
plt.plot(x, y, label='Cubic')
plt.plot(xq, yq, label='Quintic')
plt.legend()
plt.grid()
```

## Symmetric poses

We have been doing only examples with $|\theta_i - \theta_f| = \pi/2$.

What about other orientation changes? Given below is an array of terminal angles (they are in degrees!). Start from 0 deg and plot the family of trajectories

```python
dt = 0.1
thetas = np.deg2rad([15, 30, 45, 60, 90, 120, 150, 180]) #convert to radians
plt.figure()
for tf in thetas:
    t = cubic_spiral(0, tf,50)
    x = np.cumsum(np.cos(t)*dt)
    y = np.cumsum(np.sin(t)*dt)
    plt.plot(x, y)

# On the same plot, move from 180 to 180 - theta
thetas = np.pi - np.deg2rad([15, 30, 45, 60, 90, 120, 150, 180])
for tf in thetas:
    t = cubic_spiral(np.pi, tf, 50)
    x = np.cumsum(np.cos(t)*dt)
    y = np.cumsum(np.sin(t)*dt)
    plt.plot(x, y)

plt.grid()
```

Modify your code to print the following for the positive terminal angles $\{\theta_f\}$
1. Final x, y position in corresponding trajectory: $x_f, y_f$
2. $\frac{y_f}{x_f}$ and $\tan \frac{\theta_f}{2}$

What do you notice?
What happens when $v$ is doubled?

```python
dt = 0.05
v = 2.0
thetas = np.deg2rad([15, 30, 45, 60, 90, 120, 150, 180]) #convert to radians
for tf in thetas:
    t = cubic_spiral(0, tf,100)
    x = np.cumsum(v*np.cos(t)*dt)
    y = np.cumsum(v*np.sin(t)*dt)
    print(f"tf:{np.rad2deg(tf):0.1f} xf:{x[-1]:0.3f} yf:{y[-1]:0.3f} yf/xf:{y[-1]/x[-1]:0.3f} tan(theta/2):{np.tan(tf/2):0.3f}")
```

    tf:15.0 xf:9.873 yf:1.300 yf/xf:0.132 tan(theta/2):0.132
    tf:30.0 xf:9.497 yf:2.545 yf/xf:0.268 tan(theta/2):0.268
    tf:45.0 xf:8.892 yf:3.683 yf/xf:0.414 tan(theta/2):0.414
    tf:60.0 xf:8.087 yf:4.669 yf/xf:0.577 tan(theta/2):0.577
    tf:90.0 xf:6.041 yf:6.041 yf/xf:1.000 tan(theta/2):1.000
    tf:120.0 xf:3.743 yf:6.484 yf/xf:1.732 tan(theta/2):1.732
    tf:150.0 xf:1.610 yf:6.010 yf/xf:3.732 tan(theta/2):3.732
    tf:180.0 xf:-0.000 yf:4.812 yf/xf:-7880729543884461.000 tan(theta/2):16331239353195370.000

These are called *symmetric poses*. With this spiral-fitting approach, only symmetric poses can be reached.

In order to move between any 2 arbitrary poses, you will have to find an intermediate pose that is pair-wise symmetric to the start and the end pose.

What should be the intermediate pose? There are infinite possibilities. We would have to formulate it as an optimization problem. As they say, that has to be left for another time!

```python

```


# Cálculos en ingeniería con Sympy y Jupyter Notebooks

## Cuadros de texto con ecuaciones

El editor permite escribir ecuaciones en [$\LaTeX$](). Al dar "`Doble click`", puede ver cómo se escriben tales expresiones.

$$ f(a) = \int_\infty^0 \frac{1}{a+2} \mathrm{d}a$$

## Símbolos especiales en los nombres

Para nombrar una variable con un símbolo especial, use  "`\`" seguido del nombre del símbolo y presione "`tab`".

**Ejemplo**

Escriba `\alpha` y presione "`tab`" para usar α en el código.

```python
α = 1
print(α)

```

    1

## SymPy

Se importa el módulo [SymPy](http://docs.sympy.org/latest/index.html) para desarrollar operaciones matemáticas de forma simbólica, debe importase. Además se inicia un proceso de "impresión estética" para la visualización de expresiones y uno de "gráficas incorporadas".

```python
import sympy as sym
sym.init_printing()
%matplotlib inline
```

Se definen símbolos para operarlos matemáticamente.

```python
x = sym.Symbol('x')
```

```python
x
```

Se realizan operaciones matemáticas con los símbolos creados.

```python
polinomio = (x + 2)**3
polinomio.factor()
```

"`polinomio`" se creó como una operación que usa el símbolo "`x`". Después, se usó el método "`.expand()`" para expandir el "`polinomio`".

Puede tener un listado de métodos que puede usar sobre "`polinomio`" escribiendo "`polinomio.`" y presionando "`tab`"

```python
polinomio.
```

Puede obtener ayuda de un método con "`?`".

```python
polinomio.expand?
```

También puede obtener ayuda de un método al ubicar el cursor entre el paréntesis y presionar "`Shift+Tab`"

```python
polinomio.factor()
```

## Cálculo

Puede resolver integrales y derivadas.

```python
expresionA = sym.tan(sym.log(x**2 + 1)) # define una expresión
expresionA
```

```python
expresionA.diff(x) # deriva "expresionA" respecto a "x"
```

```python
polinomio.diff(x) # primera derivada
```

```python
polinomio
```

```python
polinomio.diff(x, 2) # segunda derivada
```

```python
polinomio.integrate(x) # integral indefinida sin constante de integración
```

```python
polinomio.integrate((x, 0, 1))  # integral definida. Observe la tupla como argumento
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## Límites

Puede resolver límites.

```python
expresionB = (2*sym.sin(x) - sym.sin(2*x))/(x - sym.sin(x))
```

```python
expresionB
```

```python
a = sym.symbols('a')
```

```python
lim_a = sym.limit(expresionB, x, a)
lim_a
```

```python
lim_a.subs(sym.sin(a), 1)
```

```python
lim_0 = sym.limit(expresionB, x, 0)
lim_0
```

```python
sym.plot(expresionB)
```

## Aproximaciones

Puede aproximar funciones con la serie de Taylor.

```python
expresionC = sym.sin(x)
sym.series(expresionC, x, 1.5, 3)
# Serie de Taylor en términos de x, al rededor de x=1.5, orden 2.
```

Use `.removeO()`

```python
temp = sym.series(expresionC, x, 1.5, 2)
temp.removeO()
```

SymPy mantiene la representación exacta de los números, aunque pueden aproximarse con "`float`" o "`sympy.N`" si hay variables.

```python
c = sym.sqrt(2)*sym.pi
c
```

```python
round(float(c)*1000)/1000
```

```python
sym.N(c*x)
```

## Solución de ecuaciones

Soluciona ecuaciones con la función "`solve`".

\begin{align}
               2x^2 + 2 &= 4 \\
           2x^2 + 2 - 2 &= 4 - 2 \\
                  2x^2  &= 2 \\
        \frac{2x^2}{2}  &= \frac{2}{2} \\
                    x^2 &= 1 \\
 \end{align}

```python
sym.plot(2*x**2 + 2)
```

```python
sym.plot(2*x**2 + 2,(x, -2, 2))
```

```python
soluciones = sym.solve(2*x**2 + 2 - 4)
soluciones
```

```python
soluciones
```

Con "`sympy.Eq`" se pueden construir ecuaciones.

```python
eqA = sym.Eq(2*x**2 + 2, 4)
eqA
```

```python
sym.roots(eqA)
```

```python
eqA.lhs  # lado izquierdo de la ecuación
```

```python
eqA.rhs  # lado derecho de la ecuación
```

```python
g1 = sym.plot(eqA.lhs,(x, -2, 2), show = False,line_color='blue')
g2 = sym.plot(eqA.rhs,(x, -2, 2), show = False,line_color='red')
g1.extend(g2)
g1.show()
```

Puede resolverse un sistema de ecuaciones.

```python
x, y = sym.symbols('x, y')
sym.solve([x + y - 2,
           x - y - 0], [x, y])
```

```python
a, b, c = sym.symbols('a, b, c')
sym.solve([a*x + b*y - 2,
           a*x - b*y - c], [x, y])
```

```python

```


## Simple neural network in plain Python

This notebook implements a simple neural network architecture that can map $2$ dimensional input vectors onto binary output values. Our network will have $2$ input neurons, one hidden layer with $6$ hidden neurons and an output layer with $1$ output neuron.

We will represent the architecture by means of the weight matrices between the layers. In our example, the weight matrix between the input and hidden layer will be denoted as $W_h$, the weight matrix between the hidden and output layer as $W_o$. In addition to the weights connecting the neurons, each hidden and output neuron will have a bias weight with a constant input of $+1$.

Our training set consists of $m = 750$ examples. Therefore, we will have the following matrix shapes:
- Training set shape: $X = (750, 2)$
- Targets shape: $Y = (750, 1)$
- $W_h$ shape: $(n_{features}, n_{hidden}) = (2, 6)$
- $b_h$ shape (bias vector): $(1, n_{hidden}) = (1, 6)$
- $W_o$ shape: $(n_{hidden}, n_{outputs}) = (6, 1)$
- $b_o$ shape (bias vector): $(1, n_{outputs}) = (1, 1)$

### Loss Function

We will use the same loss function as in logistic regression:

\begin{equation}
J(\boldsymbol{w},b) = - \frac{1}{m} \sum_{i=1}^m \Big[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \big(1 - \log(\hat{y}^{(i)})\big) \Big]
\end{equation}

For a classification task with more than two classes, we would use a generalization of this function, namely the categorical cross-entropy.

### Training

We will train our network with gradient descent and we will use backpropagation to compute the required partial derivatives. The training procedure has the following steps:
1. Initialize the parameters (i.e. the weights and biases)
2. Repeat until convergence:
2.1. Propagate the current input batch forward through the network. To do so, compute the activations and outputs of all hidden and output units.
2.2 Compute the partial derivatives of the loss function with respect to each parameter
2.3 Update the parameters

### Forward Pass

We start by computing the activation and output of each unit in our network. To speed up the implementation, we won't do this for each input example individually but for all examples at once, using vectorization. We will use the following notation:

- $\boldsymbol{A}_h$: matrix with activations of all hidden units for all training examples
- $\boldsymbol{O}_h$: matrix with outputs of all hidden units for all training examples

The hidden neurons will have $\tanh$ as their activation function:
\begin{equation}
\tanh(x) = \frac{sinh(x)}{cosh(x)} = \frac{\exp(x) - exp(-x)}{\exp(x) + exp(-x)}
\end{equation}
\begin{equation}
\tanh'(x) = 1 - tanh^2(x)
\end{equation}

The output neurons will have the $\textit{sigmoid}$ activation function:
\begin{equation}
\sigma(x) = \frac{1}{1 + \exp(-x)}
\end{equation}
\begin{equation}
\sigma'(x) = 1 - (1 + \sigma(x))
\end{equation}

The activations and outputs can then be computed as follows ($\cdot$ denotes the dot product):

\begin{equation}
\boldsymbol{A}_h = \boldsymbol{X} \cdot \boldsymbol{W}_h + \boldsymbol{b}_h, \text{shape: } (750, 6)
\end{equation}
\begin{equation}
\boldsymbol{O}_h = \sigma(\boldsymbol{A}_h), \text{shape: } (750, 6)
\end{equation}

\begin{equation}
\boldsymbol{A}_o = \boldsymbol{O}_h \cdot \boldsymbol{W}_o + b_o, \text{shape: } (750, 1)
\end{equation}
\begin{equation}
\boldsymbol{O}_o = \sigma(\boldsymbol{A}_o), \text{shape: } (750, 1)
\end{equation}

### Backward pass

To compute the weight updates we need the partial derivatives of the loss function with respect to each unit. I won't give the derivation of these equations here, you will find plenty of good explanations on other websites (for example [here](https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/)).

For the output neurons, the gradients are given by (matrix notation):

$\frac{\partial L}{\partial \boldsymbol{A}_o} = d\boldsymbol{A}_o = (\boldsymbol{O}_o - \boldsymbol{Y})$

$\frac{\partial L}{\partial \boldsymbol{W}_o} = \frac{1}{m} (\boldsymbol{O}_h^T \cdot d\boldsymbol{A}_o)$

$\frac{\partial L}{\partial \boldsymbol{b}_o} = \frac{1}{m} \sum d\boldsymbol{A}_o$

For the weight matrix between input and hidden layer we have:
$\frac{\partial L}{\partial \boldsymbol{A}_h} = d\boldsymbol{A}_h = (\boldsymbol{W}_o^T \cdot d\boldsymbol{A}_o) * (1 - \tanh^2 (\boldsymbol{A}_h))$

$\frac{\partial L}{\partial \boldsymbol{W}_h} = \frac{1}{m} (\boldsymbol{X}^T \cdot d\boldsymbol{A}_h)$

$\frac{\partial L}{\partial \boldsymbol{b}_h} = \frac{1}{m} \sum d\boldsymbol{A}_h$

### Weight Update

$\boldsymbol{W}_h = \boldsymbol{W}_h - \eta * \frac{\partial L}{\partial \boldsymbol{W}_h}$

$\boldsymbol{b}_h = \boldsymbol{b}_h - \eta * \frac{\partial L}{\partial \boldsymbol{b}_h} $

$\boldsymbol{W}_o = \boldsymbol{W}_o - \eta *   \frac{\partial L}{\partial \boldsymbol{W}_o} $

$\boldsymbol{b}_o = \boldsymbol{b}_o - \eta *  \frac{\partial L}{\partial \boldsymbol{b}_o} $

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
np.random.seed(123)

% matplotlib inline
```

## Dataset

```python
X, y = make_circles(n_samples=1000, factor=0.5, noise=.1)

fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.title("Dataset")
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.show()
```

```python
# reshape targets to get column vector with shape (n_samples, 1)
y_true = y[:, np.newaxis]
# Split the data into a training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y_true)

print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape}')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
```

    Shape X_train: (750, 2)
    Shape y_train: (750, 1)
    Shape X_test: (250, 2)
    Shape y_test: (250, 1)

## Neural Network Class

Some parts of this implementation are inspired by the exercises of Andrew Ng's [coursera course](https://www.coursera.org/learn/neural-networks-deep-learning)

```python
class NeuralNet():

    def __init__(self, n_inputs, n_outputs, n_hidden):
        self.n_inputs = n_inputs
        self.n_outputs = n_outputs
        self.hidden = n_hidden

        # Initialize weight matrices and bias vectors
        self.W_h = np.random.randn(self.n_inputs, self.hidden)
        self.b_h = np.zeros((1, self.hidden))
        self.W_o = np.random.randn(self.hidden, self.n_outputs)
        self.b_o = np.zeros((1, self.n_outputs))

    def sigmoid(self, a):
        return 1 / (1 + np.exp(-a))

    def forward_pass(self, X):
        """
        Propagates the given input X forward through the net.

        Returns:
            A_h: matrix with activations of all hidden neurons for all input examples
            O_h: matrix with outputs of all hidden neurons for all input examples
            A_o: matrix with activations of all output neurons for all input examples
            O_o: matrix with outputs of all output neurons for all input examples
        """
        # Compute activations and outputs of hidden units
        A_h = np.dot(X, self.W_h) + self.b_h
        O_h = np.tanh(A_h)

        # Compute activations and outputs of output units
        A_o = np.dot(O_h, self.W_o) + self.b_o
        O_o = self.sigmoid(A_o)

        outputs = {
                "A_h": A_h,
                "A_o": A_o,
                "O_h": O_h,
                "O_o": O_o,
                }

        return outputs

    def cost(self, y_true, y_predict, n_samples):
        """
        Computes and returns the cost over all examples
        """
        # same cost function as in logistic regression
        cost = (- 1 / n_samples) * np.sum(y_true * np.log(y_predict) + (1 - y_true) * (np.log(1 - y_predict)))
        cost = np.squeeze(cost)
        assert isinstance(cost, float)

        return cost

    def backward_pass(self,  X, Y, n_samples, outputs):
        """
        Propagates the errors backward through the net.

        Returns:
            dW_h: partial derivatives of loss function w.r.t hidden weights
            db_h: partial derivatives of loss function w.r.t hidden bias
            dW_o: partial derivatives of loss function w.r.t output weights
            db_o: partial derivatives of loss function w.r.t output bias
        """

        dA_o = (outputs["O_o"] - Y)
        dW_o = (1 / n_samples) * np.dot(outputs["O_h"].T, dA_o)
        db_o = (1 / n_samples) * np.sum(dA_o)

        dA_h = (np.dot(dA_o, self.W_o.T)) * (1 - np.power(outputs["O_h"], 2))
        dW_h = (1 / n_samples) * np.dot(X.T, dA_h)
        db_h = (1 / n_samples) * np.sum(dA_h)

        gradients = {
                "dW_o": dW_o,
                "db_o": db_o,
                "dW_h": dW_h,
                "db_h": db_h,
                }

        return gradients

    def update_weights(self, gradients, eta):
        """
        Updates the model parameters using a fixed learning rate
        """
        self.W_o = self.W_o - eta * gradients["dW_o"]
        self.W_h = self.W_h - eta * gradients["dW_h"]
        self.b_o = self.b_o - eta * gradients["db_o"]
        self.b_h = self.b_h - eta * gradients["db_h"]

    def train(self, X, y, n_iters=500, eta=0.3):
        """
        Trains the neural net on the given input data
        """
        n_samples, _ = X.shape

        for i in range(n_iters):
            outputs = self.forward_pass(X)
            cost = self.cost(y, outputs["O_o"], n_samples=n_samples)
            gradients = self.backward_pass(X, y, n_samples, outputs)

            if i % 100 == 0:
                print(f'Cost at iteration {i}: {np.round(cost, 4)}')

            self.update_weights(gradients, eta)

    def predict(self, X):
        """
        Computes and returns network predictions for given dataset
        """
        outputs = self.forward_pass(X)
        y_pred = [1 if elem >= 0.5 else 0 for elem in outputs["O_o"]]

        return np.array(y_pred)[:, np.newaxis]
```

## Initializing and training the neural network

```python
nn = NeuralNet(n_inputs=2, n_hidden=6, n_outputs=1)
print("Shape of weight matrices and bias vectors:")
print(f'W_h shape: {nn.W_h.shape}')
print(f'b_h shape: {nn.b_h.shape}')
print(f'W_o shape: {nn.W_o.shape}')
print(f'b_o shape: {nn.b_o.shape}')
print()

print("Training:")
nn.train(X_train, y_train, n_iters=2000, eta=0.7)
```

    Shape of weight matrices and bias vectors:
    W_h shape: (2, 6)
    b_h shape: (1, 6)
    W_o shape: (6, 1)
    b_o shape: (1, 1)

    Training:
    Cost at iteration 0: 1.0872
    Cost at iteration 100: 0.2723
    Cost at iteration 200: 0.1712
    Cost at iteration 300: 0.1386
    Cost at iteration 400: 0.1208
    Cost at iteration 500: 0.1084
    Cost at iteration 600: 0.0986
    Cost at iteration 700: 0.0907
    Cost at iteration 800: 0.0841
    Cost at iteration 900: 0.0785
    Cost at iteration 1000: 0.0739
    Cost at iteration 1100: 0.0699
    Cost at iteration 1200: 0.0665
    Cost at iteration 1300: 0.0635
    Cost at iteration 1400: 0.061
    Cost at iteration 1500: 0.0587
    Cost at iteration 1600: 0.0566
    Cost at iteration 1700: 0.0547
    Cost at iteration 1800: 0.0531
    Cost at iteration 1900: 0.0515

## Testing the neural network

```python
n_test_samples, _ = X_test.shape
y_predict = nn.predict(X_test)
print(f"Classification accuracy on test set: {(np.sum(y_predict == y_test)/n_test_samples)*100} %")
```

    Classification accuracy on test set: 98.4 %

## Visualizing the decision boundary

In the lowermost plot we can see which parts of the input space are classified as positive and which are classified as negative by the trained network.

```python
X_temp, y_temp = make_circles(n_samples=60000, noise=.5)
y_predict_temp = nn.predict(X_temp)
y_predict_temp = np.ravel(y_predict_temp)
```

```python
fig = plt.figure(figsize=(8,12))
ax = fig.add_subplot(2,1,1)
plt.scatter(X[:,0], X[:,1], c=y)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.title("Training and test set")

ax = fig.add_subplot(2,1,2)
plt.scatter(X_temp[:,0], X_temp[:,1], c=y_predict_temp)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.title("Decision boundary")
```

```python

```


# Taylor problem 2.40

Consider an object that is coasting horizontally (positive $x$ direction) subject to a drag force $f = -bv -c v^2$.  Your first job is to solve Newton's 2nd law equation for $v(t)$ by separating variables.  You should find:

$\begin{align}
  v(t) &= \frac{b A e^{-bt/m}}{1 - c A e^{-bt/m}} \\
  A &\equiv \frac{v_0}{b + c v_0}
\end{align}$

Now we want to plot $v(t)$ as analyze the behavior for large $t$.

**Go through and fill in the blanks where ### appears.**

```python
import numpy as np

def v_of_t(t, b, c, v0, m=1):
    A = v0/(b + c*v0)
    return   ### fill in the equation here
```

Next we make a plot in the standard way:

```python
%matplotlib inline
import matplotlib.pyplot as plt

t_pts =  ### determine a set of t points such that you see the decay
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(t_pts, v_of_t(t_pts, 1., 1., 1.))
ax.set_xlabel(r'$t$')
ax.set_ylabel(r'$v(t)$')

```

Now we add another plot and check if it is an exponential decay.  **What kind of plot is this?  (Google the name along with 'matplotlib'.)**

```python
%matplotlib inline
import matplotlib.pyplot as plt

plt.rcParams.update({'font.size': 18})

t_pts = np.arange(0., 3., 0.1)
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,2,1)
ax.plot(t_pts, v_of_t(t_pts, 1., 1., 1.))
ax.set_xlabel(r'$t$')
ax.set_ylabel(r'$v(t)$')
ax.grid(True)

ax2 = fig.add_subplot(1,2,2)
ax2.semilogy(t_pts, v_of_t(t_pts, 1., 1., 1.))
ax2.set_xlabel(r'$t$')
ax2.set_ylabel(r'$v(t)$')
ax2.grid(True)

fig.tight_layout()  # make the spacing of subplots nicer

```

```python
fig.savefig('Taylor_prob_2.40.png', bbox_inches='tight')
### Find the figure file and display it in your browser, then save or print.
### What do you learn from the second graph?
```

```python

```


```
from sympy import init_session
init_session()
```

    IPython console for SymPy 0.7.7.dev (Python 2.7.8-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/dev

## Quadratic

We want to construct a quadratic polynomal through that points $x_{i-1}$, $x_i$, $x_{i+1}$ that gives the correct averages,
$f_{i-1}$, $f_i$, and $f_{i-1}$ when integrated over the volume, e.g.

$$\frac{1}{\Delta x} \int_{x_{i-1/2}}^{x_{i+1/2}} f(x) dx = f_i$$

There are 3 unknowns in the quadratic and three constraints, so this is a linear system we can solve.

Define the quadratic polynomial

```
a, b, c = symbols("a b c")
x0 = symbols("x0")
f = a*(x-x0)**2 + b*(x-x0) + c
f
```

```
dx = symbols("\Delta")
```

### constraints

Define the 3 constraint equations---here we set them up construct $A$, $B$, and $C$ as the integrals over the 3 control volumes

```
fm, f0, fp = symbols("f_{i-1} f_i f_{i+1}")
#xm32, xm12, xp12, xp32 = symbols("x_{i-3/2} x_{i-1/2} x_{i+1/2} x_{i+3/2}")
xm32 = x0 - Rational(3,2)*dx
xm12 = x0 - Rational(1,2)*dx
xp12 = x0 + Rational(1,2)*dx
xp32 = x0 + Rational(3,2)*dx
```

interfaces

```
xm32, xm12, xp12, xp32
```

```
A = simplify(integrate(f/dx, (x, xm32, xm12)))
B = simplify(integrate(f/dx, (x, xm12, xp12)))
C = simplify(integrate(f/dx, (x, xp12, xp32)))
```

The analytic forms of the integrals

```
A, B, C
```

Our linear system is now:

$$A = f_{i-1}$$
$$B = f_i$$
$$C = f_{i+1}$$

Now find the coefficients of the polynomial

```
coeffs = solve([A-fm, B-f0, C-fp], [a,b,c])
coeffs
```

And in pretty form, here's the polynomial

```
f.subs(a,coeffs[a]).subs(b,coeffs[b]).subs(c,coeffs[c])
```

### Cubic

We want to construct a cubic polynomal through that points $x_{i-2}$, $x_{i-1}$, $x_i$, $x_{i+1}$ that gives the correct averages,
$f_{i-2}$, $f_{i-1}$, $f_i$, and $f_{i-1}$ when integrated over the volume of each zone

```
a, b, c, d = symbols("a b c d")
f = a*(x-x0)**3 + b*(x-x0)**2 + c*(x-x0) + d
f
```

Now perform the integals of $f(x)$ over each zone

```
fm2, fm, f0, fp = symbols("f_{i-2} f_{i-1} f_i f_{i+1}")
xm52 = x0 - Rational(5,2)*dx
xm32 = x0 - Rational(3,2)*dx
xm12 = x0 - Rational(1,2)*dx
xp12 = x0 + Rational(1,2)*dx
xp32 = x0 + Rational(3,2)*dx
```

interfaces

```
xm52, xm32, xm12, xp12, xp32
```

```
A = simplify(integrate(f/dx, (x, xm52, xm32)))
B = simplify(integrate(f/dx, (x, xm32, xm12)))
C = simplify(integrate(f/dx, (x, xm12, xp12)))
D = simplify(integrate(f/dx, (x, xp12, xp32)))
```

```
A, B, C, D
```

```
coeffs = solve([A-fm2, B-fm, C-f0, D-fp], [a,b,c,d], check=False)
coeffs
```

and the pretty form of the polynomial

```
fc = f.subs(a,coeffs[a]).subs(b,coeffs[b]).subs(c,coeffs[c]).subs(d,coeffs[d])
fc
```

this interpolant is symmetric about the $i-1/2$ interface---let's see the value there

```
fc.subs(x,x0-Rational(1,2)*dx)
```

Note that this is the interpolating polynomial used to find the interface states in PPM (Colella & Woodward 1984)

## Quartic

Now we define a quartic polynomial that gives the correct averages over 5 zones, $x_{i-2}$, $x_{i-1}$, $x_i$, $x_{i+1}$, $x_{i+2}$,
with zone averages $f_{i-2}$, $f_{i-1}$, $f_i$, $f_{i+1}$, $f_{i+2}$

```
a, b, c, d, e = symbols("a b c d e")
x0 = symbols("x0")
f = a*(x-x0)**4 + b*(x-x0)**3 + c*(x-x0)**2 + d*(x-x0) + e
f
```

Now we perform the integrals of $f(x)$ over each zone

```
fm2, fm, f0, fp, fp2 = symbols("f_{i-2} f_{i-1} f_i f_{i+1} f_{i+2}")
#xm32, xm12, xp12, xp32 = symbols("x_{i-3/2} x_{i-1/2} x_{i+1/2} x_{i+3/2}")
xm52 = x0 - Rational(5,2)*dx
xm32 = x0 - Rational(3,2)*dx
xm12 = x0 - Rational(1,2)*dx
xp12 = x0 + Rational(1,2)*dx
xp32 = x0 + Rational(3,2)*dx
xp52 = x0 + Rational(5,2)*dx
```

interfaces

```
xm52, xm32, xm12, xp12, xp32, xp52
```

```
A = simplify(integrate(f/dx, (x, xm52, xm32)))
B = simplify(integrate(f/dx, (x, xm32, xm12)))
C = simplify(integrate(f/dx, (x, xm12, xp12)))
D = simplify(integrate(f/dx, (x, xp12, xp32)))
E = simplify(integrate(f/dx, (x, xp32, xp52)))
```

The analytic form of the constraints

```
A, B, C, D, E
```

Now find the coefficients

```
coeffs = solve([A-fm2, B-fm, C-f0, D-fp, E-fp2], [a,b,c,d,e], check=False)
coeffs
```

and the pretty form of the polynomial

```
f.subs(a,coeffs[a]).subs(b,coeffs[b]).subs(c,coeffs[c]).subs(d,coeffs[d]).subs(e,coeffs[e])
```

```

```


# "Logistic Regression from Scratch"
> "Implementing logistic regression in Python using numpy library"

- toc: true
- badges: true
- comments: true
- categories: [ML, jupyter]

## Introduction
Logistic Regression is a statistical method to predict qualitative response using one or more independent variables. Instead of predicting the response directly, Logistic Regression models probability that the response belongs to a particular category. In logistic regression, we use logistic function.

The logistic function will always producr an S shaped curve, so we always get a sensible prediction. $p(X)$ is sometimes also called sigmoid function.

$$ \log\Big(\frac{p(X)}{1-p(X)}\Big) = w^TX + c $$

$$z = w^TX + c $$

$$ p(X) = \frac{1}{1 + e^{-z}} $$

```python
#collapse-hide
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import normalize
```

```python
#collapse-hide
def sigmoid(z):
    '''Sigmoid function maps from a value between 0 and 1'''
    return 1/(1+np.exp(-z))

x = np.linspace(-10,10,100)
y = sigmoid(x)
plt.plot(x,y)
plt.title("Sigmoid Function")
plt.show()
```

## Estimate Regression Coefficients

A method called **maximum likelihood** is to estimate the unknown coefficients. The idea behind this method is to estimate coefficients such that predicted probability $\hat{p}(x_i)$ of each observation corresponds closely as possible to the observed value of response variable for the same observation. In other words, we estimate coefficients such that $p(X)$ yields a number close to 1 for the observations with actual response value 1 and a number close to 0 for the observations with actual response value 0. The mathematical eequation for the likelihood function is

$$ \ell(W) = \prod_{i;y_i = 1} p(x_i) \prod_{i;y_i = 0} ( 1 - p(x_i)) $$

The $W$ estimates are chosen to maximize the likelihood function.

Manipulating the above constraint, we arrive at Cross Entropy Loss for binary classification case.
\begin{equation}
L(y,\hat{y}) = -\frac{1}{N} \sum_{i=1}^N y_i \log(\hat{y_i}) + (1-y_i)\log(\hat{y_i})
\end{equation}

```python
def cross_entropy_loss(w,x,y):
    '''Cross entropy loss function'''
    z = np.dot(x,w)
    h = sigmoid(z)
    total_loss = np.sum(-y*np.log(h) - (1-y)*np.log(1-h)).mean()
    return total_loss
```

Use chain rule to calculate gradient of loss

\begin{equation}
\frac{\partial{L}}{\partial{w_i}} = \frac{\partial{L}}{\partial{\hat{y}}} \frac{\partial{\hat{y}}}{\partial{z}} \frac{\partial{z}}{\partial{w}}
\end{equation}

Examining each factor in turn

\begin{equation}
\frac{\partial{L}}{\partial{\hat{y_i}}} = \frac{-y_i}{\hat{y_i}} + \frac{1-y_i}{1-\hat{y_i}}
\end{equation}

\begin{equation}
\frac{\partial{L}}{\partial{\hat{y_i}}} =  \frac{\hat{y_i}-y_i}{\hat{y_i}(1-\hat{y_i})}
\end{equation}

\begin{equation}
\frac{\partial{\hat{y}}}{\partial{z}}  =  \hat{y_i}(1-\hat{y_i})
\end{equation}

\begin{equation}
\frac{\partial{z}}{\partial{w}}  =  x_i
\end{equation}

Multiplying all the indivdual terms, you get

\begin{equation}
\frac{\partial{L}}{\partial{w_i}} = (\hat{y_i}-y_i)x_i
\end{equation}

```python
def gradient(w,x,y):
    '''Gradient for cross entropy loss'''
    z = np.dot(x,w)
    h = sigmoid(z)
    gradient = np.dot(x.T,h - y)
    return gradient
```

We are gonnna use a algorithm called **batch gradient descent** to find the optimal weights.

$$ \Delta w_{i} = -\eta  \frac{\partial L}{\partial w_{i}} $$

$$ w_{i} := w_{i} + \Delta w_{i} $$

```python
def update_weights(learning_rate = 0.01, n_iters = 1000, loss_threshold = 0.001):
    ## Initialize the weights with zero values
    w = np.random.rand(x.shape[1])

    for epoch in range(n_iters):
        loss = cross_entropy_loss(w,x,y)
        grad = gradient(w,x,y)
        w = w - learning_rate * grad
        if epoch % 1000 == 0:
            print(f"Loss after iteration {epoch} is {round(loss,2)}")
        if loss < loss_threshold:
            break
    return(w)
```

```python
def accuracy(true, probs, threshold = 0.5):
    predicted = (probs > threshold).astype(int)
    return 100*(predicted == true).mean()

def predict(w,x):
    return sigmoid(np.dot(x,w))
```

Lets load some binary classification data from `sklearn.datasets` and split the data into train test split.

```python
cancer_data = load_breast_cancer()
x, x_test, y, y_test = train_test_split(cancer_data.data,
                                                    cancer_data.target,
                                                    test_size=0.25,
                                                    random_state=0)
x = normalize(x)
```

Lets call the update weights function to find the optimal weights that minimizes cross entropy loss

```python
w = update_weights(learning_rate = 0.05, n_iters = 10000, loss_threshold = 0.001)
```

    Loss after iteration 0 is 281.55
    Loss after iteration 1000 is 87.93
    Loss after iteration 2000 is 83.59
    Loss after iteration 3000 is 81.78
    Loss after iteration 4000 is 80.63
    Loss after iteration 5000 is 79.76
    Loss after iteration 6000 is 79.06
    Loss after iteration 7000 is 78.46
    Loss after iteration 8000 is 77.94
    Loss after iteration 9000 is 77.47

Lets use the model to generate predictions for the test data and see what kind of accuracies are we reaching.

```python
preds = predict(w,normalize(x_test))
acc = accuracy(y_test, preds)
print(f"Accuracy on test data is {round(acc,3)}")
```

    Accuracy on test data is 93.706


## Plain Python implementation of K-Means

K-Means is a very simple clustering algorithm (clustering belongs to unsupervised learning). Given a fixed number of clusters and an input dataset the algorithm tries to partition the data into clusters such that the clusters have high intra-class similarity and low inter-class similarity.

### Algorithm

1. Initialize the cluster centers, either randomly within the range of the input data or (recommended) with some of the existing training examples

2. Until convergence

   2.1. Assign each datapoint to the closest cluster. The distance between a point and cluster center is measured using the Euclidean distance.

   2.2. Update the current estimates of the cluster centers by setting them to the mean of all instance belonging to that cluster

### Objective function

The underlying objective function tries to find cluster centers such that, if the data are partitioned into the corresponding clusters, distances between data points and their closest cluster centers become as small as possible.

Given a set of datapoints ${x_1, ..., x_n}$ and a positive number $k$, find the clusters $C_1, ..., C_k$ that minimize

\begin{equation}
J = \sum_{i=1}^n \, \sum_{j=1}^k \, z_{ij} \, || x_i - \mu_j ||_2
\end{equation}

where:
- $z_{ij} \in \{0,1\}$ defines whether of not datapoint $x_i$ belongs to cluster $C_j$
- $\mu_j$ denotes the cluster center of cluster $C_j$
- $|| \, ||_2$ denotes the Euclidean distance

### Disadvantages of K-Means
- The number of clusters has to be set in the beginning
- The results depend on the inital cluster centers
- It's sensitive to outliers
- It's not suitable for finding non-convex clusters
- It's not guaranteed to find a global optimum, so it can get stuck in a local minimum
- ...

### How to solve major disadvantages?
However, the k-means algorithm has at least two major theoretic shortcomings:

- First, it has been shown that the worst case running time of the algorithm is super-polynomial in the input size.
- Second, the approximation found can be arbitrarily bad with respect to the objective function compared to the optimal clustering.

The k-means++ algorithm addresses the second of these obstacles by specifying a procedure to initialize the cluster centers before proceeding with the standard k-means optimization iterations. With the k-means++ initialization, the algorithm is guaranteed to find a solution that is O(log k) competitive to the optimal k-means solution.
Ref: https://datasciencelab.wordpress.com/2014/01/15/improved-seeding-for-clustering-with-k-means/

```python
import numpy as np
import matplotlib.pyplot as plt
import random
from sklearn.datasets import make_blobs
np.random.seed(123)

# % matplotlib inline
```

## K-Means class

```python
class MyKMeansCluster():
    def __init__(self, n_cluster: int = 4):
        self.k = n_cluster

    def fit(self, data: np.ndarray):
        """
        Fits the k-means model to the given dataset
        """
        n_samples, _ = data.shape
        # init the centers using python random.sample(population, k)
        self.centers = np.array(random.sample(list(data), self.k))
        self.init_centers = np.copy(self.centers)

        # We will keep track of whether the assignment of data points to the clusters has changed.
        # If it stops changing, we are done fitting the model
        old_assign = None
        n_iter = 0

        while True:
            # step 1: assign all data points based on the current centers
            new_assign = [self.classify(data_point) for data_point in data]
            if new_assign == old_assign:
                print("Iteration is done: {}!".format(n_iter))
                break
            old_assign = new_assign

            # step 2: re-calculate the centers
            for cluster_id in range(self.k):
                data_indexes = np.where(np.array(new_assign) == cluster_id)
                data_pointers = data[data_indexes]
                self.centers[cluster_id] = data_pointers.mean(axis = 0)
            n_iter += 1

    def classify(self, data_point: np.array) -> np.array:
        """
        Given a datapoint, compute the cluster closest to the datapoint.
        Return the cluster ID of that cluster.
        """
        dists = self._l2_distance(data_point)
        return np.argmin(dists)

    def get_center(self):
        return self.centers

    def _l2_distance(self, data_point: np.array) -> np.ndarray:
        """ Calculate the l2 distances between current centers and an given data point"""
        dists = np.sqrt(np.sum((self.centers - data_point)**2, axis=1))
        return dists

    def _l1_distance(self, data_point: np.array) -> np.ndarray:
        dists = np.sum(np.abs(self.centers - data_point), axis=1)
        return dists

    def plot_clusters(self, data: np.array):
        plt.figure(figsize=(12,10))
        plt.title("Initial centers in black, final centers in red")
        plt.scatter(data[:, 0], data[:, 1], marker='.', c='y')
        plt.scatter(self.centers[:, 0], self.centers[:,1], c='r')
        plt.scatter(self.init_centers[:, 0], self.init_centers[:,1], c='k')
        plt.show()
```

## Test case

```python
data1 = np.random.normal(loc=(10, 4), scale=0.1, size=(10, 2))
data2 = np.random.normal(loc=(5, 5), scale=0.1, size=(10, 2))

# X = np.concatenate((data1, data2), axis=0)
X = np.vstack((data1, data2))

kmeans = MyKMeansCluster(n_cluster = 2)
# print("type: {}, shape: {}".format(type(X), X.shape))
kmeans.fit(X)
kmeans.plot_clusters(X)

print("centers: {}".format(kmeans.get_center()))
```

```python
class KMeans():
    def __init__(self, n_clusters=4):
        self.k = n_clusters

    def fit(self, data):
        """
        Fits the k-means model to the given dataset
        """
        n_samples, _ = data.shape
        # initialize cluster centers
        self.centers = np.array(random.sample(list(data), self.k))
        self.initial_centers = np.copy(self.centers)

        # We will keep track of whether the assignment of data points
        # to the clusters has changed. If it stops changing, we are
        # done fitting the model
        old_assigns = None
        n_iters = 0

        while True:
            new_assigns = [self.classify(datapoint) for datapoint in data]

            if new_assigns == old_assigns:
                print(f"Training finished after {n_iters} iterations!")
                return

            old_assigns = new_assigns
            n_iters += 1

            # recalculate centers
            for id_ in range(self.k):
                points_idx = np.where(np.array(new_assigns) == id_)
                datapoints = data[points_idx]
                self.centers[id_] = datapoints.mean(axis=0)

    def l2_distance(self, datapoint):
        dists = np.sqrt(np.sum((self.centers - datapoint)**2, axis=1))
        return dists

    def classify(self, datapoint):
        """
        Given a datapoint, compute the cluster closest to the
        datapoint. Return the cluster ID of that cluster.
        """
        dists = self.l2_distance(datapoint)
        return np.argmin(dists)

    def plot_clusters(self, data):
        plt.figure(figsize=(12,10))
        plt.title("Initial centers in black, final centers in red")
        plt.scatter(data[:, 0], data[:, 1], marker='.', c=y)
        plt.scatter(self.centers[:, 0], self.centers[:,1], c='r')
        plt.scatter(self.initial_centers[:, 0], self.initial_centers[:,1], c='k')
        plt.show()
```

## Dataset

```python
X, y = make_blobs(centers=4, n_samples=1000)
print(f'Shape of dataset: {X.shape}')

fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.title("Dataset with 4 clusters")
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.show()
```

## Initializing and fitting the model

```python
kmeans = KMeans(n_clusters=4)
kmeans.fit(X)
```

    datapoints size: (179, 2)
    datapoints size: (71, 2)
    datapoints size: (249, 2)
    datapoints size: (501, 2)
    datapoints size: (157, 2)
    datapoints size: (93, 2)
    datapoints size: (250, 2)
    datapoints size: (500, 2)
    datapoints size: (147, 2)
    datapoints size: (103, 2)
    datapoints size: (250, 2)
    datapoints size: (500, 2)
    datapoints size: (142, 2)
    datapoints size: (108, 2)
    datapoints size: (250, 2)
    datapoints size: (500, 2)
    datapoints size: (138, 2)
    datapoints size: (112, 2)
    datapoints size: (250, 2)
    datapoints size: (500, 2)
    datapoints size: (136, 2)
    datapoints size: (114, 2)
    datapoints size: (250, 2)
    datapoints size: (500, 2)
    Training finished after 6 iterations!

## Plot initial and final cluster centers

```python
kmeans.plot_clusters(X)
```

```python
class MyKMeansCluster:
    def __init__(self, n_cluster: int):
        assert(n_cluster > 0), "Invalid cluster number {}".format(n_cluster)
        self.k = n_cluster

    def fit(self, data: np.array):
        n_samples, _ = data.shape

        # init centers
        self.centers = np.array(random.sample(list(data), self.k))
        self.init_centers = np.copy(self.centers)

        old_assign = None
        n_iter = 0

        while True:
            # calculate new assigments
            new_assign = [self.classify(data_point) for data_point in data]

            if new_assign == old_assign:
                print("fit is done after {} iterations".format(n_iter))
                break

            # re-calculate the centers
            for cluster_id in range(self.k):
                assign_indexes = np.where(np.array(new_assign) == cluster_id)
                cluster_points = data[assign_indexes]
                self.centers[cluster_id] = cluster_points.mean(axis=0)

            old_assign = new_assign
            n_iter += 1

    def classify(self, data_point: np.array):
        distances = self._l2_distance(data_point)
        return np.argmin(distances)

    def plot(self, data):
        plt.figure(figsize=(12,10))
        plt.title("Initial centers in black, final centers in red")
        plt.scatter(data[:, 0], data[:, 1], marker='.', c=y)
        plt.scatter(self.centers[:, 0], self.centers[:,1], c='r')
        plt.scatter(self.init_centers[:, 0], self.init_centers[:,1], c='k')
        plt.show()

    def _l2_distance(self, data_point: np.array):
        return np.sqrt(np.sum((self.centers - data_point)**2, axis=1))
```

```python
knn_cluster = MyKMeansCluster(4)
knn_cluster.fit(X)
knn_cluster.plot(X)
```

```python

```


# Obtain the SELU parameters for arbitrary fixed points

*Author:* Guenter Klambauer, 2017

tested under Python 3.5

```python
import numpy as np
from scipy.special import erf,erfc
from sympy import Symbol, solve, nsolve
```

### Function to obtain the parameters for the SELU with arbitrary fixed point (mean variance)

```python
def getSeluParameters(fixedpointMean=0,fixedpointVar=1):
    """ Finding the parameters of the SELU activation function. The function returns alpha and lambda for the desired fixed point. """

    import sympy
    from sympy import Symbol, solve, nsolve

    aa = Symbol('aa')
    ll = Symbol('ll')
    nu = fixedpointMean
    tau = fixedpointVar

    mean =  0.5*ll*(nu + np.exp(-nu**2/(2*tau))*np.sqrt(2/np.pi)*np.sqrt(tau) + \
                        nu*erf(nu/(np.sqrt(2*tau))) - aa*erfc(nu/(np.sqrt(2*tau))) + \
                        np.exp(nu+tau/2)*aa*erfc((nu+tau)/(np.sqrt(2*tau))))

    var = 0.5*ll**2*(np.exp(-nu**2/(2*tau))*np.sqrt(2/np.pi*tau)*nu + (nu**2+tau)* \
                          (1+erf(nu/(np.sqrt(2*tau)))) + aa**2 *erfc(nu/(np.sqrt(2*tau))) \
                          - aa**2 * 2 *np.exp(nu+tau/2)*erfc((nu+tau)/(np.sqrt(2*tau)))+ \
                          aa**2*np.exp(2*(nu+tau))*erfc((nu+2*tau)/(np.sqrt(2*tau))) ) - mean**2

    eq1 = mean - nu
    eq2 = var - tau

    res = nsolve( (eq2, eq1), (aa,ll), (1.67,1.05))
    return float(res[0]),float(res[1])

```

```python
### To recover the parameters of the SELU with mean zero and unit variance
getSeluParameters(0,1)
```

    (1.6732632423543774, 1.0507009873554802)

```python
### To obtain new parameters for mean zero and variance 2
myFixedPointMean = -0.1
myFixedPointVar = 2.0
myAlpha, myLambda = getSeluParameters(myFixedPointMean,myFixedPointVar)
getSeluParameters(myFixedPointMean,myFixedPointVar)
```

    (1.9769021954241999, 1.073851239616047)

### Adjust the SELU function and Dropout to your new parameters

```python
def selu(x):
    with ops.name_scope('elu') as scope:
        alpha = myAlpha
        scale = myLambda
        return scale*tf.where(x>=0.0, x, alpha*tf.nn.elu(x))
```

```python
def dropout_selu(x, rate, alpha= -myAlpha*myLambda, fixedPointMean=myFixedPointMean, fixedPointVar=myFixedPointVar,
                 noise_shape=None, seed=None, name=None, training=False):
    """Dropout to a value with rescaling."""

    def dropout_selu_impl(x, rate, alpha, noise_shape, seed, name):
        keep_prob = 1.0 - rate
        x = ops.convert_to_tensor(x, name="x")
        if isinstance(keep_prob, numbers.Real) and not 0 < keep_prob <= 1:
            raise ValueError("keep_prob must be a scalar tensor or a float in the "
                                             "range (0, 1], got %g" % keep_prob)
        keep_prob = ops.convert_to_tensor(keep_prob, dtype=x.dtype, name="keep_prob")
        keep_prob.get_shape().assert_is_compatible_with(tensor_shape.scalar())

        alpha = ops.convert_to_tensor(alpha, dtype=x.dtype, name="alpha")
        keep_prob.get_shape().assert_is_compatible_with(tensor_shape.scalar())

        if tensor_util.constant_value(keep_prob) == 1:
            return x

        noise_shape = noise_shape if noise_shape is not None else array_ops.shape(x)
        random_tensor = keep_prob
        random_tensor += random_ops.random_uniform(noise_shape, seed=seed, dtype=x.dtype)
        binary_tensor = math_ops.floor(random_tensor)
        ret = x * binary_tensor + alpha * (1-binary_tensor)

        a = tf.sqrt(fixedPointVar / (keep_prob *((1-keep_prob) * tf.pow(alpha-fixedPointMean,2) + fixedPointVar)))

        b = fixedPointMean - a * (keep_prob * fixedPointMean + (1 - keep_prob) * alpha)
        ret = a * ret + b
        ret.set_shape(x.get_shape())
        return ret

    with ops.name_scope(name, "dropout", [x]) as name:
        return utils.smart_cond(training,
            lambda: dropout_selu_impl(x, rate, alpha, noise_shape, seed, name),
            lambda: array_ops.identity(x))

```

```python
import tensorflow as tf
import numpy as np

from __future__ import absolute_import, division, print_function
import numbers
from tensorflow.contrib import layers
from tensorflow.python.framework import ops
from tensorflow.python.framework import tensor_shape
from tensorflow.python.framework import tensor_util
from tensorflow.python.ops import math_ops
from tensorflow.python.ops import random_ops
from tensorflow.python.ops import array_ops
from tensorflow.python.layers import utils

x = tf.Variable(tf.random_normal([10000],mean=myFixedPointMean, stddev=np.sqrt(myFixedPointVar)))
w = selu(x)
y = dropout_selu(w,0.2,training=True)
init = tf.global_variables_initializer()

gpu_options = tf.GPUOptions(allow_growth=True)
with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
    sess.run(init)
    z,zz, zzz = sess.run([x, w, y])
    #print(z)
    #print(zz)
    print("mean/var should be at:", myFixedPointMean, "/", myFixedPointVar)
    print("Input data mean/var:  ", "{:.12f}".format(np.mean(z)), "/", "{:.12f}".format(np.var(z)))
    print("After selu:           ", "{:.12f}".format(np.mean(zz)), "/", "{:.12f}".format(np.var(zz)))
    print("After dropout mean/var", "{:.12f}".format(np.mean(zzz)), "/", "{:.12f}".format(np.var(zzz)))
```

    mean/var should be at: -0.1 / 2.0
    Input data mean/var:   -0.081669516861 / 1.985695838928
    After selu:            -0.086086399853 / 2.011691331863
    After dropout mean/var -0.080544397235 / 2.031569242477

### For completeness: These are the correct expressions for mean zero and unit variance

```python
myAlpha = -np.sqrt(2/np.pi) / (np.exp(0.5) * erfc(1/np.sqrt(2))-1 )
myLambda = (1-np.sqrt(np.exp(1))*erfc(1/np.sqrt(2)))  *  \
            np.sqrt( 2*np.pi/ (2 + np.pi -2*np.sqrt(np.exp(1))*(2+np.pi)*erfc(1/np.sqrt(2)) + \
            np.exp(1)*np.pi*erfc(1/np.sqrt(2))**2 + 2*np.exp(2)*erfc(np.sqrt(2))))
```

```python
print("Alpha parameter of the SELU: ", myAlpha)
print("Lambda parameter of the SELU: ", myLambda)
```

    Alpha parameter of the SELU:  1.67326324235
    Lambda parameter of the SELU:  1.05070098736


```python
import sympy as sp
from sympy import symbols
from numpy import linspace
from sympy import lambdify
from sympy import I as j
from sympy import re, im
import matplotlib.pyplot as mpl

from sympy.physics.mechanics import *
init_vprinting()
```

## 1.D-2

```python
rho_0 = symbols('rho_0')
c_0 = symbols('c_0')
r_0 = symbols('r_0')
k = symbols('k')
```

```python
Z = rho_0*c_0 / (1 + ( 1 / (j*k*r_0)) )
Z
```

```python
res = {c_0: 1, rho_0: 1, r_0: 1}

plot1 = sp.plot(re(Z.subs(res)), (k, 0, 10))
plot2 = sp.plot(im(Z.subs(res)), (k, 0, 10))
plot1.extend(plot2)
plot1.show()
```

```python
# Imaginary part check
Z2 = c_0*k*rho_0*r_0 / (k**2 * r_0**2 + 1)
sp.plot(Z2.subs(res), (k, 0, 10))
```

```python
# Real part check
Z3 = rho_0*c_0*((k*c_0) / (k*c_0 + 1))
sp.plot(Z3.subs(res), (k, 0, 10))
```

```python

```


Solution to: [Day 3: Drawing Marbles](https://www.hackerrank.com/challenges/s10-mcq-6/problem)

Table of Contents

- Table of Contents
- Math Solution
    - Facts
- Monte Carlo Solution
    - Imports
    - Constants
    - Auxiliary functions
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

This script contains 2 sections:
1. Math solution to the problem
2. Monte Carlo simulation of the problem

# Math Solution
A bag contains 3 red marbles and 4 blue marbles. Then, 2 marbles are drawn from the bag, at random, without replacement.
If the first marble drawn is red, what is the probability that the second marble is blue?

## Facts
- 7 marbles in the bag
- 1st one is always red
- P(B)?

If the first one is always red, we don't calculate the probability of the 1st draw.
Thus, we use the 6 remaining marbles.
\begin{equation}
\large
P(B) = \frac{4}{6} = \frac{2}{3}
\end{equation}

# Monte Carlo Solution

## Imports

```python
from typing import List
import random
```

## Constants

```python
MARBLE_DICT = {
	'r'	:	3,
	'b'	:	4
}
FIRST_MARBLE = 'r'
SECOND_MARBLE = 'b'
```

## Auxiliary functions

```python
def create_marble_bag(marbles: dict) -> List[str]:
	"""Returns list of marbles to draw from."""
	bag = []
	for k, v in marbles.items():
		m = [k for _ in range(v)]
		bag += m
	return bag
```

```python
def remove_first_marble(bag: List[str], marble: str) -> List[str]:
	"""Returns bag after removing marble."""
	bag.remove(marble)
	return bag
```

```python
def check_second_marble(bag: List[str], marble: str) -> bool:
	"""Returns boolean if sample from bag is the marble."""
	return random.choice(bag) == marble
```

```python
def get_ratio(bag: List[str], marble: str, iterations: int) -> float:
	"""Returns ratio of times sample from bag is marble."""
	was_marble = 0
	for _ in range(iterations):
		if check_second_marble(bag, marble):
			was_marble += 1
	return was_marble / iterations
```

## Main

```python
def main():
	bag = create_marble_bag(MARBLE_DICT)
	bag = remove_first_marble(bag, FIRST_MARBLE)

	iterations =  1000000
	ratio = get_ratio(bag, SECOND_MARBLE, iterations)
	print(ratio)
```

```python
if __name__ == "__main__":
	main()
```

    0.665724


Euler Problem 193
=================

A positive integer n is called squarefree, if no square of a prime divides n,
thus 1, 2, 3, 5, 6, 7, 10, 11 are squarefree, but not 4, 8, 9, 12.

How many squarefree numbers are there below 250?

```python
from sympy import sieve
N = 2**25
NN = N*N - 1
mobius = [0, 1, -1, 1] * ((N+3) // 4)
mobius[0] = 1
for p in sieve.primerange(3, N):
    for n in range(p, N, p):
        mobius[n] = -mobius[n]
    pp = p*p
    for n in range(pp, N, pp):
        mobius[n] = 0
print(sum(int(NN/(d*d)) * mobius[d]
          for d in range(1, N)))
```

    684465067343069

```python

```


Make sure you remove `raise NotImplementedError()` and fill in any place that says `# YOUR CODE HERE`, as well as your `NAME`, `ID`, and `LAB_SECTION` below:

```python
NAME = "Fahmid Bin Kibria"
ID = "19201063"
LAB_SECTION = "02"
```

---

# Part 1: Representing a Polynomial

Polynomials are function of the following format

$$p(x) = a_0 + a_1 x ^ 1 + a_2 x ^ 2 + ... + a_n x ^ n,$$

where, $[a_0, a_1, \cdots a_n]$ are called coefficients and $n$ (called the degree or order) is a non-negative integer.

This can also be written as:

$$y = f(x) = a_0 x^0 + a_1 x ^ 1 + a_2 x ^ 2 + ... + a_n x ^ n.$$

**Example**
$$ y = 1 + 2x^2 + 5x^4 $$ is a polynomial of order 4 ($ = n$) with $n+1$ coeffecients $a_0 = 1, a_1 = 0, a_2 = 2, a_3 = 0, a_4 = 5$

**Method 1**: Using List

---

```python
import numpy as np
import matplotlib.pyplot as plt

# numpy is used for efficient array (vector or matrix) operations
# pyplot is used for plotting
# Must read: [https://www.tutorialspoint.com/numpy/numpy_matplotlib.htm]
```

```python
a = [1, 0, 2, 0, 5] # coeffecients of the polynomial
n = len(a) - 1 # degree. Remember: number of coeff = degree + 1

'''
For a single value of x, finding p(x)

Note that this is an example of block comment in python. A block comment
starts with three ' and ends with three '.
'''

x = 5.0
p_x = 0.0

for i in range(n + 1):
    '''
    p_x = a[i] * x # WRONG, because no power
    p_x = a[i] * (x ** i) # WRONG, have to add the terms
    '''
    p_x += a[i] * (x ** i) # a ** b means pow(a, b) or a^b

'''
For an array of x, finding p(x) for each element
'''

x_arr = [1.0, 2.0, 3.0, 4.0, 5.0]
p_x_arr = []

'''
# naive way:
for i in range(len(x_arr)):
    print(x_arr[i])
'''

# better way: array traversing
for x in x_arr:
    temp = 0.0
    for i in range(n + 1):
        temp += a[i] * (x ** i)

    p_x_arr.append(temp) # array er last e insert kore dao

print(f"p({x_arr}) = ", p_x_arr)
# note how we formatted the string. A formatted string starts with 'f'.
```

    p([1.0, 2.0, 3.0, 4.0, 5.0]) =  [8.0, 89.0, 424.0, 1313.0, 3176.0]

```python
# Using numpy array for vectorization
import numpy as np
# numpy is used for efficient array (vector or matrix) operations
# Must read: [https://www.tutorialspoint.com/numpy/numpy_matplotlib.htm]

a = np.array([1, 0, 2, 0, 5])
x_arr = np.array([1, 2, 3, 4, 5])
p_x_arr = 0.0

# vectorized version. requires only one loop
for i in range(n + 1):
    p_x_arr += a[i] * (x_arr ** i) # a ** b means pow(a, b) or a^b

print(f"p({x_arr}) = ", p_x_arr)
```

    p([1 2 3 4 5]) =  [   8.   89.  424. 1313. 3176.]

**Method 2 (Better)**: Using a Class

---
Complete the implementation of the polynomial class as showed in the lecture

```python
'''
Lab task 1
Here we implement a Polynomial class with three methods: the constructor
__init__(), the toString method __repr__(), and a method to make the objects
of the class callable, __call__() method
'''

# Polynomial Class

class Polynomial:
  # Constructor, note that it starts and ends with two underscores
  def __init__(self, coeff):
    '''
    Every internal variable of the object must be saved and initialized
    in this method: self.variable = value
    '''
    self.coeff = coeff
    self.degree = len(coeff) - 1

  # Constructor to make the object callable
  def __call__(self, x_arr):
    '''
    Here we assumed x_arr is a numpy array. Remember that a numpy array acts
    like a vector (1D matrix). So an operation x + 1 would add 1 to each element
    of the matrix (unlike python's defaule list). Simlarly, x ** 2 would return
    element wise square of the array.

    Hence, this method would return an array, where the i'th element is the
    (polynomial) interpolated value of x[i], given the coeffecients a[i].
    '''
    p_x_arr = 0
    # --------------------------------------------
    # HINT: Should look like
    # for i in range(self.degree + 1):
    #     ????
    # --------------------------------------------

    # remember 1: length = degree + 1 for a polynomial
    # remember 2: range(0, a) is same as range(a)
    # remember 3: range(a, b) means a is inclusive, b is exclusive

    # --------------------------------------------
    # YOUR CODE HERE
    for i in range(self.degree + 1):
      p_x_arr +=  self.coeff[i] * (x_arr ** i)

    return p_x_arr
    # --------------------------------------------

  # String representation method of the object (similar to toString() of java)
  def __repr__(self):
    # remember: a formatted string must start with f.

    str_ret = f'Polynomial of degree {self.degree}\np(x) = '
    for i in range(self.degree + 1):
        a = self.coeff[i]
        if i != 0:
            if a >= 0:
                str_ret += f'+ {a}x^{i} '
            else:
                str_ret += f'- {-a}x^{i} '
        else:
            str_ret += f'{a}x^{i} '

    return str_ret

  # custom method 1: to get the degree of the polynomial
  def get_degree(self):
    # --------------------------------------------
    # YOUR CODE HERE
    return self.degree
    # --------------------------------------------

  # custom method 2: to get the coefficients of the polynomial
  def get_coeffs(self):
    # --------------------------------------------
    # YOUR CODE HERE
    return self.coeff
    # --------------------------------------------
```

```python
# test cases for your answer
x_arr = np.array([1, 2, 3, 4, 5, 6])

coeff = np.array([1.0, 0.0, 2.0, 0.0, 5.0])
p = Polynomial(coeff)

results = [8, 89, 424, 1313, 3176, 6553]
test = p(x_arr)

np.testing.assert_array_equal(results, test)
```

```python
# an example to see if our implementation works
coeff = np.array([1.0, 0.0, 2.0, 0.0, 5.0])
p = Polynomial(coeff)
print(p)  # check if printable
x_arr = np.array([1, 2, 3, 4, 5, 6])
print()
print(f"p({x_arr}) =", p(x_arr)) # check if the object is callable
# should print p([1 2 3 4 5]) =  [   8.   89.  424. 1313. 3176.]
```

    Polynomial of degree 4
    p(x) = 1.0x^0 + 0.0x^1 + 2.0x^2 + 0.0x^3 + 5.0x^4

    p([1 2 3 4 5 6]) = [   8.   89.  424. 1313. 3176. 6553.]

# Part 2: Polynomial Interpolation (Matrix Method)

If we have  $n+1$ nodes, that is,  $\{(x_0, y_0), (x_1, y_1), (x_2, y_2), (x_{n}, y_{n})\}$ that satisfies a polynomial of order $n$, it can be written as:

\begin{align}
&a_0 + a_1  x_0 + a_2  x_0^2 + \cdots a_n  + x_0^n = y_0\\
&a_0 + a_1  x_1 + a_2  x_1^2 + \cdots a_n  + x_1^n = y_1\\
&a_0 + a_1  x_2 + a_2  x_2^2 + \cdots a_n  + x_2^n = y_2\\
&\cdots\\
&a_0 + a_1  x_{n-1} + a_2  x_{n}^2 + \cdots + a_n  x_{n}^n = y_{n}\\
\end{align}

Here, $p(x) = a_0 + a_1x^1 + a_2x^2 + \cdots a_nx^n$ is called the fitted polynomial of the given data points (nodes). Using this polynomial to find the $y_k$ corresponding to an $x_k$ with the range of the given nodes is called polynomial interpolation.

In matrix form, the equations can be written as  $$\mathbf{Xa = y},$$

where $\mathbf{X} = $

\begin{bmatrix}
x_0^0 & x_0^1 & x_0^2 & \cdots & x_0^n\\
x_1^0 & x_1^1 & x_1^2 & \cdots & x_1^n\\
\vdots & \vdots & \vdots & \ddots & \vdots\\
x_n^0 & x_{n}^1 & x_n^2 & \cdots & x_n^n\\
\end{bmatrix}

$\mathbf{a} = $
\begin{bmatrix}
a_0\\
a_1\\
\vdots\\
a_n
\end{bmatrix}

and $\mathbf{y} = $
\begin{bmatrix}
y_0\\
y_1\\
\vdots\\
y_n
\end{bmatrix}

From this, we can solve for $\mathbf{a}$ using
$$\mathbf{a = X^{-1}y}.$$

```python
'''
Lab task 2
Here we implement a function which takes a discrete x and y array, and returns
a Polynomial object (the one we just implemented). This polynomial object can
be used to calculate y for any other value of x (not in that list) within the
range
'''
def get_poly(data_x, data_y):
    n_nodes = len(data_x)
    # np.zeors((a, b)) returns a (a x b) matrix, i.e., a rows and b columns
    X = np.zeros((n_nodes, n_nodes))

    # Populate the X matrix with appropriate values
    # See the lecture video how the matrix is formed
    # --------------------------------------------
    # Hint: The code will like like this:
    # for i in range(n_nodes):
    #   for j in range(n_nodes):
    #     X[i, j] = ????
    # --------------------------------------------
    # YOUR CODE HERE
    for i in range(n_nodes):
      for j in range(n_nodes):
        X[i,j] = data_x[i] ** j
    print(X)
    # --------------------------------------------
    # We could have also used np.linalg.inv to find the inverse
    # but pinv is more efficient
    X_inv = np.linalg.pinv(X) # pseudo inverse
    a = np.dot(X_inv, data_y)
    p = Polynomial(a)

    return p
```

```python
# test cases for your answer
data_x = np.array([-3., -2., -1., 0., 1., 3.])
data_y = np.array([-80., -13., 6., 1., 5., 16.])
p = get_poly(data_x, data_y)

results = np.array([-80, -74.60997689, -69.36169492, -64.26436346, -59.32622134,
                   -54.55456417, -49.95577177, -45.53533558, -41.297886, -37.24721982,
                   -33.38632762, -29.71742112, -26.24196062, -22.96068235, -19.87362589,
                   -16.98016156, -14.2790178, -11.76830857,  -9.44556072,  -7.30774144,
                   -5.35128559, -3.57212312, -1.96570645, -0.52703788, 0.74930302,
                   1.86913203, 2.83863291, 3.66432993, 4.35306058, 4.9119481,
                   5.34837412, 5.66995126, 5.88449574, 6, 6.02460529,
                   5.96657428, 5.83426368, 5.63609684, 5.38053634, 5.07605665,
                   4.73111668, 4.35413242, 3.95344953, 3.53731597, 3.1138546,
                   2.69103576, 2.27664993, 1.87828029, 1.50327536, 1.15872159,
                   0.85141596, 0.58783863, 0.3741255, 0.21604084, 0.1189499,
                   0.0877915, 0.12705066, 0.2407312, 0.43232834, 0.70480134,
                   1.06054604, 1.50136754, 2.02845277, 2.64234311, 3.34290699,
                   4.1293125, 5, 5.95265474, 6.98417944, 8.09066693,
                   9.26737272, 10.50868766, 11.80811047, 13.15822045, 14.55065,
                   15.97605727, 17.42409876, 18.88340192, 20.34153777, 21.78499351,
                   23.19914511, 24.56822994, 25.87531935, 27.1022913, 28.22980298,
                   29.23726338, 30.10280593, 30.80326108, 31.31412894, 31.60955188,
                   31.6622871, 31.4436793, 30.92363323, 30.07058634, 28.85148136,
                   27.23173894, 25.17523021, 22.64424943, 19.59948659, 16])

x_arr = np.linspace(-3, 3, 100)
test = p(x_arr)

np.testing.assert_array_almost_equal(test, results)
```

    [[   1.   -3.    9.  -27.   81. -243.]
     [   1.   -2.    4.   -8.   16.  -32.]
     [   1.   -1.    1.   -1.    1.   -1.]
     [   1.    0.    0.    0.    0.    0.]
     [   1.    1.    1.    1.    1.    1.]
     [   1.    3.    9.   27.   81.  243.]]

```python
data_x = np.array([-3., -2., -1., 0., 1., 3.])
data_y = np.array([-80., -13., 6., 1., 5., 16.])
p = get_poly(data_x, data_y)
'''
np.linspace(a, b, n) returns a numpy array of n points equally
spaced from a to b
'''
x_arr = np.linspace(-3, 3, 100)
# interpolated values
y_interp = p(x_arr)

# pyplot is used for plotting
# Must read: [https://www.tutorialspoint.com/numpy/numpy_matplotlib.htm]

plt.plot(x_arr, y_interp, 'r')
plt.plot(data_x, data_y, 'go')
plt.xlabel('x - axis')
plt.ylabel('y - axis')

plt.show()
```

```python

```


# Motivation for Implicit Time Integration
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
import numpy as np
from numpy import *
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
from scipy.integrate import odeint
from scipy.optimize import fsolve
```

# Solve the following initial value problem using the Forward Euler method
Solve the ODE:
\begin{equation}
\frac{\text{d}y}{\text{d}t} = -ay;\quad y(0) = y_0
\end{equation}
The analytical solution is
\begin{equation}
y(t) = y_0 e^{-a t}
\end{equation}

We first define the RHS for this ODE

```python
a = 1000.0
def rhs_sharp_transient(f, t):
    return - a * f
```

we now plot it

```python
yexact = lambda a,y0,t : y0 * np.exp(-a*t)
t = linspace(0,0.1,300)
y0 = 1
plt.plot(t,yexact(1000,y0,t),label='Exact')
plt.legend()
plt.grid()
# plt.savefig('sharp-transient-exact.pdf')
```

# Using Forward Euler
Using the forward Euler method, solve the above ODE using various timestep sizes, starting with $\Delta t = 0.003$.

```python
def forward_euler(rhs, f0, tend, dt):
    ''' Computes the forward_euler method '''
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        f[n+1] = f[n] + dt * rhs(f[n], time[n])
    return time, f
```

```python
f0 = 1.0
tend = 0.1
dt = 0.003
# dt = 2.0/a
# dt = 0.5*dt
print('dt',dt)
# a = 1000
t,ffe = forward_euler(rhs_sharp_transient,f0, tend, dt)
plt.plot(t,ffe,'r-o',label='Forward Euler, dt='+str(dt))

texact = np.linspace(0,tend,400)
plt.plot(texact,yexact(a,f0,texact),label='Exact')
plt.legend()
plt.grid()
plt.savefig('motivation for implicit dt='+str(dt)+'.pdf')
```

    dt 0.003

# Using Backward Euler

```python
def be_residual(fnp1, rhs, fn, dt, tnp1):
    '''
    Nonlinear residual function for the backward Euler implicit time integrator
    '''
    return fnp1 - fn - dt * rhs(fnp1, tnp1)

def backward_euler(rhs, f0, tend, dt):
    '''
    Computes the backward euler method
    :param rhs: a rhs function
    '''
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        fn = f[n]
        tnp1 = time[n+1]
        fnew = fsolve(be_residual, fn, (rhs, fn, dt, tnp1))
        f[n+1] = fnew
    return time, f
```

```python
f0 = 1.0
tend = 0.1
dt = 0.0025

t,fbe = backward_euler(rhs_sharp_transient, f0, tend, dt)
plt.plot(t,fbe, 'r*-',label='Backward Euler')

texact = np.linspace(0,tend,400)
plt.plot(texact,yexact(a,f0,texact),label='Exact')
```

    [<matplotlib.lines.Line2D at 0x151b590190>]

# Using Crank-Nicolson

```python
def cn_residual(fnp1, rhs, fn, dt, tnp1, tn):
    '''
    Nonlinear residual function for the Crank-Nicolson implicit time integrator
    '''
    return fnp1 - fn - 0.5 * dt * ( rhs(fnp1, tnp1) + rhs(fn, tn) )

def crank_nicolson(rhs,f0,tend,dt):
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        fn = f[n]
        tnp1 = time[n+1]
        tn = time[n]
        fnew = fsolve(cn_residual, fn, (rhs, fn, dt, tnp1, tn))
        f[n+1] = fnew
    return time, f
```

```python
f0 = 1.0
tend = 0.1
dt = 0.0025

t,fcn = crank_nicolson(rhs_sharp_transient, f0, tend, dt)
plt.plot(t,fcn, 'r*-',label='Backward Euler')

texact = np.linspace(0,tend,400)
plt.plot(texact,yexact(a,f0,texact),label='Exact')
```

    [<matplotlib.lines.Line2D at 0x151c892cd0>]

# Using odeint

```python
time = np.linspace(0,0.1,20)
sol = odeint(rhs_sharp_transient,f0,time)
texact = np.linspace(0,tend,400)
plt.plot(texact,y(a,f0,texact),label='Exact')
plt.plot(time,sol, 'k-o', label='ODEint')
plt.legend()
```

    <matplotlib.legend.Legend at 0x152621ed50>

topics/initial-value-problems/Motivation for Implicit Methods.ipynb


# Sampling - Balls in a Box

> ***GitHub***: https://github.com/czs108

## Background

> Consider a box containing **15** *red* balls, **10** *green* balls and **5** *blue* balls. Let **5** balls be drawn at random *without* replacement from the box.
>
> Please give the probabilities of the following events.

\begin{equation}
P(A \cup B) = P(A) + P(B) - P(A \cap B)
\end{equation}

\begin{equation}
\begin{split}
P(A \cap B) &= P(A) \cdot P(B \mid A) \\
    &= P(B) \cdot P(A \mid B)
\end{split}
\end{equation}

## Question A

### Question 1

> The *1st* ball is *red*.

\begin{equation}
P = \frac{15}{30}
\end{equation}

### Question 2

> The *1st* ball and *2nd* ball are *red*.

\begin{equation}
P = \frac{15}{30} \times \frac{14}{29}
\end{equation}

### Question 3

> The *2nd* ball is *red*.

There are **2** situations:

\begin{equation}
P(1st = red\, \cap\, 2nd = red) = \frac{15}{30} \times \frac{14}{29}
\end{equation}

\begin{equation}
P(1st \neq red\, \cap\, 2nd = red) = \frac{15}{30} \times \frac{15}{29}
\end{equation}

The total probability:

\begin{equation}
\begin{split}
P &= P(1st = red\, \cap\, 2nd = red) + P(1st \neq red\, \cap\, 2nd = red) \\
    &= \frac{1}{2}
\end{split}
\end{equation}

### Question 4

> The *2nd* ball is *red* given that the *1st* ball is *red*.

\begin{equation}
\begin{split}
P &= \frac{P(1st = red\, \cap\, 2nd = red)}{P(1st = red)} \\
    &= \frac{\frac{15}{30} \times \frac{14}{29}}{\frac{15}{30}} \\
    &= \frac{14}{29}
\end{split}
\end{equation}

### Question 5

> The *1st* ball is *red* given that the *2nd* ball is *red*.

\begin{equation}
\begin{split}
P &= \frac{P(1st = red\, \cap\, 2nd = red)}{P(2nd = red)} \\
    &= \frac{\frac{15}{30} \times \frac{14}{29}}{\frac{1}{2}} \\
    &= \frac{14}{29}
\end{split}
\end{equation}

### Question 6

> Either the *1st* ball or *2nd* ball or both are *red*.

\begin{equation}
\begin{split}
P &= P(1st = red) + P(2nd = red) - P(1st = red\, \cap\, 2nd = red) \\
    &= \frac{1}{2} + \frac{1}{2} - \frac{15}{30} \times \frac{14}{29}
\end{split}
\end{equation}

## Question B

> **5** balls are all the *same* color.

\begin{equation}
P = \frac{^{15}C_{5} +\, ^{10}C_{5} +\, ^{5}C_{5}}{^{30}C_{5}}
\end{equation}

## Question C

> Exactly **2** balls are *red*.

\begin{equation}
P = \frac{^{15}C_{2} \times\, ^{15}C_{3}}{^{30}C_{5}}
\end{equation}

## Question D

> The sequence is *red*, *green*, *blue*, *red*, *green*.

\begin{equation}
P = \frac{15 \times 10 \times 5 \times 14 \times 9}{^{30}P_{5}}
\end{equation}

## Question E

> Getting **2** *red* balls, **2** *green* balls and **1** *blue* ball in any order.

\begin{equation}
P = \frac{^{15}C_{2} \times\, ^{10}C_{2} \times\, ^{5}C_{1}}{^{30}C_{5}}
\end{equation}

exercises/Sampling - Balls in a Box.ipynb


# Chaotic model divergence: an ensemble of Lorenz 96 models using basic model interface (BMI)
The Lorenz 96 model is a model often used to demonstrate chaotic behavior and it is the de-facto standard benchmark model used in the field of data assimilation to test data assimilation methods. In this notebook I present how to easily interact with an implementation of the Lorenz 96 model through the Basic Model Interface (BMI). I show to run a complete ensemble of model-instances that illustrates the chaotic nature of the model.

## the Lorenz 96 model
The Lorenz 96 model is a dynamical sytem for $i=1,...,N$ defined by
\begin{equation}
\frac{dx_{i}}{dt}=\left(x_{i+1} - x_{i-2}\right)x_{i-1} - x_{i} + F
\end{equation}
where i is cyclical, ie. $x_{0}=x_{N}$ and $x_{-1} = x_{N-1}$. $F$ is an external force acting on the system. A value of $F=8$ is known to create chaotic bahavior and is often used. The dimension $N$ can be freely chosen and is typical $40$, but for testing very high dimension systems, higher values can be used. The Lorenz 96 model is a typical chaotic model where, although, the model is deterministic, slight variations in the input state will over time result in complete different states.

## Numerical implementation of the Lorenz 96 model
A fourth order Runga Kutta scheme is used to implement the Lorenz 96 model. Writing the entire state-vector as $\vec{x}$ and using $f\left(\vec{x}\right)$ as the right hand side of the model, ie:
\begin{eqnarray}
f\left(x_{i}\right) = \left(x_{i+1} - x_{i-2}\right)x_{i-1} - x_{i} + F
\\
f\left(\vec{x}\right) = \left\{f\left(x_{1}\right),...,f\left(x_{N}\right)\right\}
\end{eqnarray}
the implementation is given by:
\begin{eqnarray}
\vec{k}_{1}=f\left(\vec{x}\left(t\right)\right)
\\
\vec{k}_{2}=f\left(\vec{x}\left(t\right) + \frac{1}{2}\vec{k}_{1}\Delta t\right)
\\
\vec{k}_{3}=f\left(\vec{x}\left(t\right) + \frac{1}{2}\vec{k}_{2}\Delta t\right)
\\
\vec{k}_{4}=f\left(\vec{x}\left(t\right) + \vec{k}_{3}\Delta t\right)
\end{eqnarray}
and finally
\begin{equation}
\vec{x}\left(t + \Delta t\right) = \vec{x}\left(t\right) + \frac{1}{6}\left(\vec{k}_{1} + 2\vec{k}_{2} + 2 \vec{k}_{3} + \vec{k}_{4}\right)
\end{equation}

## The Basic Model Interface (BMI)
The basic model interface allows communicating with models in a generic fashion. It requires a few standard methods to be available such as 'initialize()' and 'update()'. Methods that are not relevant for the model need still be implemented, but can simply raise a one line exception. See [ref] for more information. Implementing the BMI allows easy interaction with the model. The cells below initiate one instance of the model. For reasons that will become clear we will call this instance "truthModel".

BMI models are typically initialized with a settings-file. This is overkill here, but for completeness, we generate the settings-file first and than pass it to the model.

```python
#required libraries and settings
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import yaml
import io

import BMILorenz
```

```python
#create an instance of the BMILorenz class
truthModel = BMILorenz.BMILorenz ()

#initialize the model
J = 40 #dimension of Lorenz Model

#make a starting state vector
commonStartState = np.zeros(J)
commonStartState[19]=0.01

# Define settings data
settings = {'J': J,
            'F': 8.0,
            'startTime': 0.0,
            'endTime': 20.0,
            'dt':1e-3,
            'startState': commonStartState}

# Write YAML file
with io.open('settings.yaml', 'w', encoding='utf8') as outfile:
    yaml.dump(settings, outfile, default_flow_style=False, allow_unicode=True)

#initialize model
truthModel.initialize('settings.yaml')
```

### Running the model
The model is now all set to run. We need to choose which variables to save every timestep. In the case of the Lorenz model we choose the 5th element of the state-vector, yet this is arbitrary.

The model us run for one timestep by calling `truthModel.update()`. We check if the model time is greater than the end time by using `truthModel.get_current_time()` and `truthModel.get_end_time()`

```python
output = pd.DataFrame(columns = ['truth'])

while truthModel.get_current_time() < truthModel.get_end_time():
    truthModel.update()
    output.loc[truthModel.get_current_time()] = truthModel.get_value_at_indices('state',5)

```

Let's plot the output of the model

```python
plt.plot(output)
plt.show()
```

## creating an ensemble of models
An ensemble of model instances can be easily created by making an array of model objects. Below I slightly change the starting state of the ensemble members by using the `model.set_value_at_indices()` function.

```python
N = 25 #numeber of ensemble members

#start with  an empty ensemble
ensemble = []

for n in range (N):
    #add an ensemble methods
    ensemble.append(BMILorenz.BMILorenz ())
    ensemble[n].initialize('settings.yaml')
    ensemble[n].set_value_at_indices('state',5,ensemble[n].get_value_at_indices('state',5) + np.random.randn(1)*0.01)

    #also add a column to the output dataframe to store the output
    output['ensemble' + str(n)]= np.nan

```

```python
#run the Ensemble. We will use the time of the first ensemble member to keep time, assuming that all
#models use the same time steps.

while ensemble[0].get_current_time()<ensemble[0].get_end_time():

    #loop through the ensemble members and store the state after each update
    for n in range (N):
        ensemble[n].update()
        output.at[ensemble[n].get_current_time(),'ensemble' + str(n)] = ensemble[n].get_value_at_indices('state',5)

```

### output of the ensemble
I plot the output of all ensemble members in black and the truth in red

```python
plt.plot(output.loc[:,'ensemble0':],'k')
plt.plot(output.loc[:,'truth'],'r')
plt.show()
```

It shows that after a certain moment in time the ensemble starts diverging from the truth: the little variations on the starting conditions are enough to trigger this behavior. Zooming in on the point where the divergence starts shows this behavior

```python
plt.plot(output.loc[1:3,'ensemble0':],'k')
plt.plot(output.loc[1:3,'truth'],'r')
plt.show()
```

.ipynb_checkpoints/Lorenz 96 model ensemble demonstration-checkpoint.ipynb

Lorenz 96 model ensemble demonstration.ipynb