# 反向传播(Backpropagation)笔记

反向传播是深度学习的基石。

## 导数

先回顾下导数：

$$\frac{df(x)}{dx}=\lim_{h->0}\frac{f(x+h)-f(x)}{h}$$

函数在每个变量的导数就是偏导数。

对于函数$f(x,y)=x+y$，$\frac{\partial f}{\partial x}=1$，同时，$\frac{\partial f}{\partial y}=1$

梯度就是偏导数组成的矢量。上述例子中，$\Delta f=[\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}]$。

## 链式法则

对于简单函数，我们可以根据公式直接计算出其导数。但是对于复杂的函数，我们就没那么容易直接写出导数。但是我们有**链式法则(chain rule)**。

定义不多说，咱们举个例子，感受一下链式法则的魅力。

我们熟悉的sigmoid函数$\sigma(x)=\frac{1}{1+e^{-x}}$，如果你记不住它的导数，我们怎么求解呢？

求解步骤如下：

* 将函数模块化，分成多个基本的部分，对于每一个部分都可以使用简单的求导法则进行求导
* 使用链式法则，将这些导数链接起来，计算出最终的导数

具体如下：

* 令$a=x$，则 $\frac{\partial a}{\partial x}=1$
* 令$b=-a$，则 $\frac{\partial b}{\partial a}=-1$
* 令$c=e^{b}$，则 $\frac{\partial c}{\partial b}=e^{b}$
* 令$d=1+c$，则 $\frac{\partial d}{\partial c}=1$
* 令$e=\frac{1}{d}$，则 $\frac{\partial e}{\partial d}=\frac{-1}{d^2}$

上面的e实际上就是我们的$\sigma(x)$，那么根据链式法则，有：

$$
\begin{align}
\frac{\partial \sigma(x)}{\partial x} &=\frac{\partial e}{\partial x} \\
                                                       &=\frac{\partial e}{\partial d}\cdot\frac{\partial d}{\partial c}\cdot\frac{\partial c}{\partial b}\cdot\frac{\partial b}{\partial a}\cdot\frac{\partial a}{\partial x} \\
                                                       &=\frac{-1}{d^2}\cdot1\cdot e^{b}\cdot-1\cdot1 \\
                                                       &=\frac{e^b}{d^{2}} \\
                                                       &=\frac{e^{-x}}{(1+e^{-x})^2} \\
                                                       &=(1-\sigma(x))\cdot\sigma(x)
\end{align}
$$

sigmoid函数的导数可以直接用自身表示，这也是很奇妙的性质了。这样的求导过程是不是很简单？

## 反向传播代码实现

求导和链式法则我都会了，那么具体的前向传播和反向传播的代码是怎么样的呢？

这次我们使用一个更复杂一点点的例子：

$$f(x,y)=\frac{x+\sigma(x)}{\sigma(x)+(x+y)^2}$$

我们先看下它地forward pass代码：

```python
import math

x = 3
y = -4

sigy = 1.0 / (1 + math.exp(-y)) # sigmoid function
num = x + sigy # 分子
sigx = 1.0 / (1 + math.exp(-x))
xpy = x + y
xpy_sqr = xpy**2
den = sigx + xpy_sqr # 分母
invden = 1.0 / den
f = num * invden # 函数
```

上述过程很简单对不对，就是把复杂的函数拆解成一个一个简单函数。

我们看看接下来的反向传播过程：

```python
dnum = invden
```

因为$f = num * invden$，所以有$\frac{\partial f}{\partial num} = invden$，也就是$dnum = invden$

```python
dinvden = num # 同理

dden = (-1.0 / (den**2)) * dinvden # 链式法则
```

展开来说：$\frac{\partial invden}{\partial den}=\frac{-1}{den^2}$，又$\frac{\partial f}{\partial invden}=num$，所以$dden=\frac{\partial f}{\partial den}=\frac{partial f}{\partial invden}\cdot \frac{\partial invden}{\partial den} = \frac{-1.0}{den^2}\cdot dinvden$

所以，同理，我们可以写出所有的导数：

```python
dsigx = (1) * dden
dxpy_sqr = (1) * dden

dxpy = (2 * xpy) * dxpy_sqr

# backprob xpy = x + y
dx = (1) * dxpy
dy = (1) * dxpy

# 这里开始，请注意使用的是"+="，而不是"=”
dx += ((1 - sigx) * sigx) * dsigx # dsigma(x) = (1 - sigma(x))*sigma(x)
dx += (1) * dnum

# backprob num = x + sigy
dsigy = (1) * dnum
# 注意“+=”
dy += ((1 - sigy) * sigy) * dsigy
```

问题：

* 上面计算过程中，为什么要用“+=”替代“=”呢？

如果变量x，y在前向传播的表达式中出现多次，那么进行反向传播的时候就要非常小心，使用+=而不是=来累计这些变量的梯度（不然就会造成覆写）。这是遵循了在微积分中的多元链式法则，该法则指出如果变量在线路中分支走向不同的部分，那么梯度在回传的时候，就应该进行累加。


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## 6 Optimizing Functions (Examples)

Chapter 6 in [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

By the end of this session, you should be able to:
- Understand general approaches to optimize functions either for minima or maxima (i.e. "extrema")
- Understand how to use derivatives in multiple dimensions to categorize extrema
- Using `scipy.optimize` to do numeric optimization of complicated functions
  - (We'll have more examples for both algebra/calculus and numerical optimization in recitation)

### Quick Summary

In chemistry and physics, we often want to determine the maximum or minimum value of a function of one or many variables. Examples include characterizing the minima, maxima, and saddle points on a potential energy surface.

Optimizing functions in one dimension is pretty easy, if sometimes tedious.
1. Take derivatives and find where the first derivative is zero
2. Look at the second derivatives, to categorize as a minima / maxima / inflection point
3. Then compare values of the function at those points to see if it's a local minima / max or the global minima / max.

### Many Variables

Not surprisingly, we can use a similar technique in multiple dimensions.

If we have a function $f(x,y)$ in two dimensions, then to have an extrema:

$$
\frac{\partial f}{\partial x}=0 \quad \frac{\partial f}{\partial y}=0
$$

In other words, we need to see the partial derivative with respect to **both / all** variables be zero.

We can then categorize the type of minima / maxima with the [Hessian]. (Later, we will see that this is the *determinant* of the Hessian matrix, for when we have more than 2 variables.)

$$
D=\left(\frac{\partial^{2} f}{\partial x^{2}}\right)\left(\frac{\partial^{2} f}{\partial y^{2}}\right)-\left(\frac{\partial^{2} f}{\partial x \partial y}\right)\left(\frac{\partial^{2} f}{\partial y \partial x}\right)
$$

$$
\left.\frac{\partial^{2} f}{\partial x^{2}}\right|_{\left(x^{*}, y^{*}\right)}=\left\{\begin{array}{ll}
<0 & D>0 & \text { maximum } \\
>0 & D>0 & \text { minimum } \\
 & D < 0 & \text { saddle-point }
\end{array}\right.
$$

### Example:

Let's try the so-called "butterfly potential"

$$
V(x, y)=\left((x-y)^{2}-1\right)^{2}+10 x^{2} y^{2}
$$

```python
from sympy import init_session
init_session()
```

```python
V = ((x-y)**2-1)**2 + 10*x**2*y**2
V
```

```python
# okay, let's see where the partial derivative with x is zero...
diff(V, x)
```

```python
# now let's do y
diff(V, y)
```

Okay, we can't do this so easily - we have two equations and two unknowns.

Fortunately, sympy offers some nice features:
- Eq(*expression*, value) sets up an equation like $x = 0$
  - We can set our derivatives as equal to zero
- solve() can take multiple equations and multiple unknowns

```python
eq1 = Eq(diff(V, x), 0) # partial derivative with x == 0
eq2 = Eq(diff(V, y), 0) # partial derivative with y == 0
solve((eq1,eq2), (x, y)) # solve two equations for two unknowns
```

```python
# here's the two-variable Hessian test...
D = diff(V, x, 2)*diff(V, y, 2) - diff(V, x, y)*diff(V, y, x)
D # print it
```

Okay, I like when the computer does math for me...

```python
# we can use the .subs() method to substitute values (-1, 0)
D.subs([ (x, -1), (y, 0) ])
```

```python
# check (0,0)
D.subs([ (x, 0), (y, 0) ])
```

The origin is inconclusive - because it's something like a saddle point, but there are four wells around it.

```python
# now check (+1, 0)
D.subs([ (x, 1), (y, 0) ])
```

### Water

We looked at a one-dimensional Lennard-Jones potential.

Here, I've calculated the O-H bond stretch in water, using the RI-MP2 method and the cc-pVTZ basis set. (Don't worry, you don't need to know what that means, but we're aiming for a relatively accurate quantum chemical method.)

Most bonds fit well to a [Morse Potential](https://en.wikipedia.org/wiki/Morse_potential) like this. There are reasons to use other functional forms (e.g., speed), but this relatively simple form captures a lot of how real covalent bonds behave.

I also calculated the H-O-H bond angle bend using the same method. Fortunately, this (like many angle bends) fit well to a quadratic function:

The total potential energy will be:

$$
V = V_{bond} + V_{angle}
$$

In principle, there are "cross-terms" (e.g., stretch-bend) involving both $r$ and $\theta$ but it's always good to start with a simple model and get more complex once we understand that.

```python
# We'll declare new symbols for now
r, theta = symbols("r theta")
bond = 265.79*(1 - exp((-2.2534)*(r - 0.96242)))**2
angle = 167.16 - 3.217*theta + 0.01548*theta**2
V = bond + angle
```

```python
# let's solve for the derivative w.r.t. theta equal to zero
solve( diff(angle, theta) )
```

```python
diff(bond, r)
```

Oof, that's a hard one to solve. We know that the minimum value is around 0.96Å, so we can try this numerically:

```python
lengths = [0.93, 0.94, 0.95, 0.96, 0.97, 0.98]
for l in lengths:
    print(l, round(bond.subs(r, l), 4), round(dr.subs(r, l), 4))
```

Clearly 0.96Å is very close. Notice that the partial derivative goes from negative to positive between 0.96Å and 0.97Å.

We can interpolate:

```python
for digit in range(0, 10):
    l = 0.96 + digit/1000
    print(l, round(bond.subs(r, l), 4), round(dr.subs(r, l), 4))
```

I can go further, but ~0.9625Å seems like a fairly accurate estimate of the optimal bond length.

I picked this example because sometimes we want to use several methods (e.g., both calculus and numerical methods) to solve problems.

Now to be completely honest, a better description of the potential energy function:

$$
V(r, \theta) = V_{bond}(r) + V_{angle}(\theta) + V_{stretch-bend}(r, \theta)
$$

Usually those stretch-bend terms look something like:

$$
V_{\mathrm{str} / \mathrm{bend}}=k^{A B C}\left(\theta^{A B C}-\theta_{0}^{A B C}\right)\left[\left(R^{A B}-R_{0}^{A B}\right)+\left(R^{B C}-R_{0}^{B C}\right)\right]
$$

-------
This notebook is from Prof. Geoffrey Hutchison, University of Pittsburgh
https://github.com/ghutchis/chem1000


# Matrix Multiplication

Author: Yoseph K. Soenggoro

```python
from random import random
from itertools import product
import numpy as np
```

```python
# Check you NumPy Version (I used 1.16.4).
# If the program is incompatible with your NumPy version, use pip or conda to set the appropriate version
np.__version__
```

    '1.16.4'

```python
# Choose the value of n, the dimension for Matrix X and Y
n = 3

# Choose d as the range of random value for Matrix X and Y.
# By choosing value d, the element of Matrix X and Y will be any real number between 0 and d, but never d.
d = 10
```

Before starting to multiply any two matrices, first define two different matrices $X$ and $Y$ using the `random` library.

```python
# Define Matrix X and Matrix Y
X = []
Y = []

for i in range(0, n):
    x_row = []
    for j in range(0, n):
        x_val = random() * d
        x_row.append(x_val)
    X.append(x_row)

for i in range(0, n):
    y_row = []
    for j in range(0, n):
        y_val = random() * d
        y_row.append(y_val)
    Y.append(y_row)
```

```python
# Function to print the matrices
def print_matrix(X):
    matrix_string = ''
    for i, j in product(range(0, n), range(0, n)):
        matrix_string += f'{X[i][j]}' + ('\t' if j != n - 1 else '\n')
    print(matrix_string)
```

```python
# Print X to Check
print_matrix(X)
```

    1.411243489014371	4.823307914767269	2.9619965649396427
    8.632221076600207	6.909318166288182	4.297076325685831
    6.741265901114342	9.161915445688102	6.166996253979611

```python
# Print Y to Check
print_matrix(Y)
```

    7.627644148989169	4.412768013974525	4.55820479886072
    7.864788107651906	2.491004876229417	1.2387049762153457
    1.6601811042693615	6.587244192044375	9.966139995991005

### Matrix Multiplication Formula (Linear Algebra)

Given a $n \times n$ matrices $X$ and $Y$, as follows:

\begin{align}
X =
\begin{bmatrix}
    x_{1, 1} & x_{1, 2} & \dots & x_{1, n} \\
    x_{2, 1} & x_{2, 2} & \dots & x_{2, n} \\
    \vdots & \vdots & \ddots & \vdots \\
    x_{n, 1} & x_{n, 2} & \dots & x_{n, n}
\end{bmatrix}
, \quad
Y =
\begin{bmatrix}
    y_{1, 1} & y_{1, 2} & \dots & y_{1, n} \\
    y_{2, 1} & y_{2, 2} & \dots & y_{2, n} \\
    \vdots & \vdots & \ddots & \vdots \\
    y_{n, 1} & y_{n, 2} & \dots & y_{n, n}
\end{bmatrix}
\end{align}

then the multiplication is defined by the following formula:
\begin{align}
    X \cdot Y = \left[\sum_{k = 1}^n x_{i, k} \cdot y_{j, k}\right]_{i, j = 1}^n
\end{align}

### Implementation \#1: Functional Paradigm

The simplest way to implement Matrix Multiplication is by using modular functions, that can be used and reused multiple times within a program. Given the formula above, the Python implementation will be as follows.

```python
# Function to implement Matrix Multiplication of Matrix X and Y
def matrix_mul(X, Y):
    Z = []

    for i in range(0, n):
        z_row = []
        for j in range(0, n):
            z_val = 0
            for k in range(0, n):
                z_val += X[i][k] * Y[k][j]
            z_row.append(z_val)
        Z.append(z_row)
    return Z
```

For the multiplication between $X$ and $Y$, the result will be kept in variable $Z$.

```python
Z = matrix_mul(X, Y)
print_matrix(Z)
```

    53.61620859740801	37.75376833274768	41.92706479366027
    127.31775885342397	83.60902536916785	90.7313025662656
    133.714831698932	93.1935288493318	103.53812885700651

### Check Validity on Matrix Multiplication Function

Despite having a working matrix multiplication implementation in functional form, we still have no idea whether the result from our implementation is right or wrong. Therefore, one method to validate the result will be doing a comparison with `NumPy`'s implementation of `matmul` API.

```python
# Function to compare the Matrix Multiplication Function to NumPy's matmul
def check_matrix_mul(X, Y):
    print('Starting Validation Process...\n\n\n')
    x = np.array(X)
    y = np.array(Y)
    z = np.matmul(x, y)
    Z = matrix_mul(X, Y)
    for i, j in product(range(0, n), range(0, n)):
        print(f'Checking index {(i, j)}... \t\t\t {round(z[i][j], 2) == round(Z[i][j], 2)}')

    print('\n')
    print('Validation Process Completed')
```

```python
a = check_matrix_mul(X, Y)
```

    Starting Validation Process...

    Checking index (0, 0)... 			 True
    Checking index (0, 1)... 			 True
    Checking index (0, 2)... 			 True
    Checking index (1, 0)... 			 True
    Checking index (1, 1)... 			 True
    Checking index (1, 2)... 			 True
    Checking index (2, 0)... 			 True
    Checking index (2, 1)... 			 True
    Checking index (2, 2)... 			 True

    Validation Process Completed

Since after checking all the results are True, then it can be confirmed that the implentation works sucessfully.

### Implementation \#2: Object-Oriented Paradigm

Another paradigm that can be used is OOP or Object-Oriented Programming, which represents a program as a set of Objects with various fields and methods to interact with the defined Object. In this case, first defined a generalized form of matrices, which is known as Tensors. The implementation of `Tensor` will be as follows:

```python
class Tensor:
    def __init__(self, X):
        validation = self.__checking_validity(X)

        self.__dim = 2
        self.tensor = X if validation else []
        self.__dimension = self.__get_dimension_private(X) if validation else -1

    def __get_dimension_private(self, X):
        if not check_child(X):
            return 1
        else:
            # Check whether the size of each child are the same
            for i in range(0, len(X)):
                if not check_child(X[i]):
                    return self.__dim
                else:
                    get_dimension(X[i])
            self.__dim += 1
            return self.__dim

    def __checking_validity(self, X):
        self.__dim = 2
        valid = True
        if not check_child(X):
            return valid
        else:
            dim_0 = get_dimension(X[0])
            # Check whether the size of each child are the same
            for i in range(1, len(X)):
                self.__dim = 2
                if get_dimension(X[i]) != dim_0:
                    valid &= False
                    break
        return valid

    # Getting the Value of Tensor Rank/Dimension (Not to be confused with Matrix Dimension)
    def get_dimension(self):
        return self.__dimension
```

Since Tensors are generalized form of matrices, it implies that it is possible to define `Matrix` class as a child class of `Tensor` with additional methods (some overrides the `Tensor`'s original methods). For operators, I only managed to override the multiplication operator for the sake of implementing Matrix Multiplication. Thus, other operator such as `+`, `-`, `/`, and others will not be available for the current implementation.

```python
class Matrix(Tensor):
    def __init__(self, X):
        super().__init__(X)
        self.__matrix_string = ''

    def __str__(self):
        return self.__matrix_string if self.__check_matrix_validation() else ''

    # Check whether the given input X is a valid Matrix
    def __check_matrix_validation(self):
        valid = True
        try:
            for i, j in product(range(0, n), range(0, n)):
                self.__matrix_string += f'{self.tensor[i][j]}' + ('\t' if j != n - 1 else '\n')
        except:
            valid = False
            print('Matrix is Invalid. Create New Instance with appropriate inputs.')

        return valid

    # Get Matrix Dimension: Number of Columns and Rows
    def get_dimension(self):
        print(f'Matrix Dimension: ({len(self.tensor)}, {len(self.tensor[0])})' if self.__check_matrix_validation() else -1)
        return [len(self.tensor), len(self.tensor[0])]

    # Overriding Multiplication Operator for Matrix Multiplication
    # and Integer-Matrix Multiplication
    def __mul__(self, other):
        if isinstance(other, Matrix):
            Z = []
            for i in range(0, n):
                z_row = []
                for j in range(0, n):
                    z_val = 0
                    for k in range(0, n):
                        z_val += self.tensor[i][k] * other.tensor[k][j]
                    z_row.append(z_val)
                Z.append(z_row)
            return Matrix(Z)
        elif isinstance(other, int):
            Z = []
            for i in range(0, n):
                z_row = []
                for j in range(0, n):
                    z_row.append(self.tensor[i][j] * other)
                Z.append(z_row)
            return Matrix(Z)
        else:
            return NotImplemented

    # Overriding Reverse Multiplication to support Matrix-Integer Multiplication
    def __rmul__(self, other):
        if isinstance(other, int):
            Z = []
            for i in range(0, n):
                z_row = []
                for j in range(0, n):
                    z_row.append(self.tensor[i][j] * other)
                Z.append(z_row)
            return Matrix(Z)
        else:
            return NotImplemented
```

```python
# Transform X and Y to Matrix Object
x_obj = Matrix(X)
y_obj = Matrix(Y)

# Implement Matrix Multiplication as follows
z_obj = x_obj * y_obj
print(z_obj)
```

    53.61620859740801	37.75376833274768	41.92706479366027
    127.31775885342397	83.60902536916785	90.7313025662656
    133.714831698932	93.1935288493318	103.53812885700651

### Check Validity on Matrix Multiplication using OOP

Similar to the previous section, we still have no idea whether the result from our implementation is right or wrong. Hence, validation is highly important. Therefore, one method to validate the result will be again doing a comparison with `NumPy`'s implementation of `matmul` API.

```python
# Function to compare the Matrix Multiplication Function to Numpy's matmul
def check_matrix_mul_oop(X, Y):
    print('Starting Validation Process...\n\n\n')
    x = np.array(X)
    y = np.array(Y)
    z = np.matmul(x, y)
    Z = Matrix(X) * Matrix(Y)
    for i, j in product(range(0, n), range(0, n)):
        print(f'Checking index {(i, j)}... \t\t\t {round(z[i][j], 2) == round(Z.tensor[i][j], 2)}')

    print('\n')
    print('Validation Process Completed')
```

```python
a = check_matrix_mul_oop(X, Y)
```

    Starting Validation Process...

    Checking index (0, 0)... 			 True
    Checking index (0, 1)... 			 True
    Checking index (0, 2)... 			 True
    Checking index (1, 0)... 			 True
    Checking index (1, 1)... 			 True
    Checking index (1, 2)... 			 True
    Checking index (2, 0)... 			 True
    Checking index (2, 1)... 			 True
    Checking index (2, 2)... 			 True

    Validation Process Completed

Since after checking all the results are True, then it can be confirmed that the implentation works sucessfully.

# Python Libraries

- [NumPy](https://numpy.org/)


# Visualizing stationary paths of a  functional

Last revised: 02-Feb-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

Consider the functional

$\begin{align}
  S = \int_{x_1}^{x_2} f[y(x), y'(x), x] \, dx
\end{align}$

with $y_1 = y(x_1)$ and $y_2 = y(x_2)$ fixed.  We denote by $y^*(x)$ the path that minimizes $S$ (or, more generally, makes it stationary).  Then we consider the class of candidate paths $y(x)$ given by

$\begin{align}
  y(x) = y^*(x) + \alpha \eta(x)
\end{align}$

where $\eta(x)$ is some function that vanishes at the endpoints: $\eta(x_1) = \eta(x_2) = 0$.  We can derive the Euler-Lagrange equations by minimizing $S(\alpha)$ with respect to $\alpha$.

Here we visualize this problem by considering a particular $S$, choosing among some possible $\eta(x)$ definitions, and seeing how $S$ is minimized with respect to $\alpha$.  We will also allow for an incorrect determination of $y^*(x)$, in which case we expect that the minimum alpha will give us a reasonable reproduction of the true $y^*(x)$.  The variation of $\alpha$ and the choice of functions will be made using widgets from `ipywidgets`.

## Looking at a plot of the functional evaluation versus $\alpha$

\[We'll use `%matplotlib notebook` so that we can modify figures without redrawing them.\]

```python
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt

import ipywidgets as widgets
from IPython.display import display

```

### Functional from Taylor problem 6.9

This problem states: "Find the equation of the path from the origin $O$ to the point $P(1,1)$ in the $xy$ plane that makes the integral $\int_O^P (y'^2 + y y' + y^2)$ stationary.  The answer from solving the Euler-Lagrange equation is $y^*(x) = \sinh(x)/\sinh(1)$.

```python
def y_star(x):
    """Path that minimizes the functional in Taylor problem 6.9."""
    return np.sinh(x) / np.sinh(1.)
```

```python
delta_x = 0.001
x_pts = np.arange(0., 1., delta_x)

fig = plt.figure(figsize=(6,3),
                 num='Visualizing stationary paths of a functional')
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)

def setup_figure():
    ax1.set_title('Show paths')
    ax1.plot(x_pts, y_star(x_pts), color='black', lw=2)
    ax1.set_xlabel('x')
    ax1.set_ylabel('y(x)')

    ax2.set_title('Evaluate functional')
    ax2.set_xlabel(r'$\alpha$')
    ax2.set_ylabel('functional')
    ax2.set_xlim(-0.4, 0.4)
    ax2.set_ylim(1.5, 3.)
    #ax2.axvline(0., color='black', alpha=0.3)
    ax2.axhline(evaluate_functional(x_pts, y_star(x_pts)),
                                    color='black', alpha=0.3)

    fig.tight_layout()

def evaluate_functional(x_pts, y_pts):
    """Given arrays of x and y points, evaluate the functional from 6.9."""
    # The numpy gradient function takes the derivative of an array y_pts
    #  that is a function of points x in x_pts.
    y_deriv_pts = np.gradient(y_pts, x_pts)
    f = y_deriv_pts**2 + y_pts * y_deriv_pts + y_pts**2

    # Use the numpy trapezoid rule (trapz) to do the integral over f.
    return np.trapz(f, x_pts)

def make_path(alpha, ax1_passed, ax2_passed,
              base_function='exact', eta_function='sine'):
    """Given a base function, which may be the exact y^*(x) or a guess that
       is not correct, generate and plot the path corresponding to adding
       alpha*eta(x) to the base function, with eta(x) chosen among some
       functions that vanish at the endpoints in x.
    """
    # map x_pts to zero to 1 (it may already be there)
    x_mapped_pts = (x_pts - x_pts[0]) / (x_pts[-1] - x_pts[0])

    # Choices for the base function
    if (base_function == 'exact'):
        base = lambda x : y_star(x)
    elif (base_function == 'guess 1'):
        base = lambda x : np.sinh(2.*x) / np.sinh(2.)
    elif (base_function == 'guess 2'):
        base = lambda x : x**3

    if (eta_function == 'sine'):
        eta = lambda x : np.sin(np.pi * x)
    elif (eta_function == 'parabola'):
        eta = lambda x : 4. * x * (1. - x)

    y_new_pts = base(x_pts) + alpha * eta(x_mapped_pts)

    ax1_passed.plot(x_pts, y_new_pts, color='red', lw=1)
    ax2_passed.plot(alpha, evaluate_functional(x_pts, y_new_pts), '.',
                    color='red')

def reset_graph(event):
    ax1.clear()
    ax2.clear()
    setup_figure()

button = widgets.Button(
    description='reset graph'
)
button.on_click(reset_graph)

widgets.interact(make_path,
                 alpha=widgets.FloatSlider(min=-1., max=1., step=.05,
                                           value=0.0, description=r'$\alpha$',
                                           continuous_update=False),
                 ax1_passed=widgets.fixed(ax1),
                 ax2_passed=widgets.fixed(ax2),
                 base_function=widgets.Dropdown(options=['exact', 'guess 1',
                                                         'guess 2'],
                                                value='exact',
                                                description='base function'),
                 eta_function=widgets.Dropdown(options=['sine', 'parabola'],
                                               value='sine',
                                               description=r'$\eta(x)$')
                )

setup_figure()
button
```

    <IPython.core.display.Javascript object>

    interactive(children=(FloatSlider(value=0.0, continuous_update=False, description='$\\alpha$', max=1.0, min=-1…

    Button(description='reset graph', style=ButtonStyle())

```python

```


# Exercises

 We have prepared five exercises in this chapter:

 1. Modify the HCM code to work for three groups. This exercise can be divded into four tasks:
     - modify the parameters,
     - modify the calculate_u function,
     - execute the clustering,
     - plot the results.
 2. For density clustering, plot the feature space with all element marked with different color, depending on the cluster that it's assigned to. You should do the following tasks:
     - fill the get_color method,
     - fill the plot code.
 3. Build a method that plot baed on dendrograms_history and pydot, a dendrogram for the divisive clustering method. You should base on agglomerative method, but keep in mind that it works top-down instead of bottom-up. This exercise need just one function to be implemented:
     - show_tree_divisive.
     You should loop over the dendrogram_history variable and loop over childs.
 4. Implement the $s_{2}$ metric
 5. Draw the borders between clusters in the output image

 ## Libraries

 To solve the exercises, we need the following libraries to load in the first place.

```python
import numpy
import random
import numpy as np
import pandas as pd
from math import sqrt

import matplotlib.image as img
import PIL.Image

from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

from IPython.display import Image

import pydot

```

 ## Load data

 ## Exercise 1: Modify the HCM code to work for three groups

 The obvious part is the variable ```groups```, but the most changes needs to be done here:

```python
get_ipython().run_line_magic('store', '-r data_set')

# change here:
groups = 3

error_margin = 0.01
m = 2
assignation = np.zeros((len(data_set), groups))

# centers = np.array([[0.01229673, 0.25183492],
#                     [0.3689626, 0.61904127],
#                     [0.95732769, 0.45059586]])

centers = np.array([[0.01229673, 0.25183492],
                    [0.3689626, 0.61904127],
                    [0.95732769, 0.45059586]])

def calculate_distance(x, v):
    return sqrt((x[0]-v[0])**2+(x[1]-v[1])**2)

def calculate_new_centers(u):
    new_centers = []
    for c in range(groups):
        u_x_vector = np.zeros(2)
        u_scalar = 0.0
        for i in range(len(data_set)):
            u_scalar = u_scalar+(u[i][c]**m)
            u_x_vector = np.add(
                u_x_vector, np.multiply(u[i][c]**m, data_set[i]))
        new_centers.append(np.divide(u_x_vector, u_scalar))
    return new_centers

def calculate_differences(new_assignation, assignation):
    return np.sum(np.abs(np.subtract(assignation, new_assignation)))

def cluster_hcm(assignation, centers):
    difference_limit_not_achieved = True
    new_centers = centers
    iter = 0
    while difference_limit_not_achieved:
        new_assignation = []
        for i in range(len(data_set)):
            new_assignation.append(calculate_u_three(data_set[i], new_centers))
        new_centers = calculate_new_centers(new_assignation)
        if iter > 0:
            if calculate_differences(new_assignation, assignation) < error_margin:
                difference_limit_not_achieved = False
        assignation = new_assignation
        iter = iter+1
    return new_assignation, new_centers

```

 ### Modify the ``calculate_u`` function

 Fill the gap below to make the function working for more groups than two. The goal here is to calculate the distance between ``x`` and the center of a given group and append the value to ``minimal_distance``.

```python

def calculate_u_three(x, centers):
    u_array = np.zeros(groups)
    minimal_distance = []
    for group in range(groups):
        distance = calculate_distance(x, centers[group])
        minimal_distance.append(distance)
    min_group_id = np.argmin(minimal_distance)
    u_array[min_group_id] = 1
    return u_array

```

 ### Execute the clustering

 As in the previous example we need to cluster it.

```python
new_assignation_hcm3, new_centers_hcm3 = cluster_hcm(assignation, centers)
pd.DataFrame(new_centers_hcm3)

```

      0
      1

      0
      0.078585
      0.178323

      1
      0.496071
      0.692516

      2
      0.958743
      0.955892

 ### Plot the results

```python
red = data_set[np.where(np.array(new_assignation_hcm3)[:, 0] == 1)]
blue = data_set[np.where(np.array(new_assignation_hcm3)[:, 1] == 1)]
green = data_set[np.where(np.array(new_assignation_hcm3)[:, 2] == 1)]

fig, ax = plt.subplots()

ax.scatter(blue[:, 0], blue[:, 1], c='blue')
ax.scatter(red[:, 0], red[:, 1], c='red')
ax.scatter(green[:, 0], green[:, 1], c='green')
ax.scatter(np.array(new_centers_hcm3)[:, 0], np.array(
    new_centers_hcm3)[:, 1], c='black')
ax.set(xlabel='Seats count', ylabel='Distance range (km)',
       title='Aircrafts (clusters)')
ax.grid()
plt.show()

```

 ## Exercise 2: Plot the density clusters

 Use the code below to plot the results. You can play with the max_distance variable to get more or less groups.

```python
get_ipython().run_line_magic('store', '-r new_assignation_density')
get_ipython().run_line_magic('store', '-r data_set')

# print(data_set)
# print(new_assignation_density)

```

 ### Fill the ``get_group_objects`` method

 Only one line needs to be updated. The ``get_group_objects`` function should return the objects of a given group.

```python
colors = ['red', 'blue', 'green', 'orange', 'black', 'yellow']

def get_group_objects(color_id):
    objects = []
    for idx, density in enumerate(new_assignation_density):
        if density == color_id:
            objects.append(data_set[idx])
    return np.array(objects)

```

 ### Fill the plot code

 If done properly the code below should return a plot of two clusters and the noise.

```python
colors = ['red', 'blue', 'green', 'orange', 'black', 'yellow']

fig, ax = plt.subplots()

assigned_groups = new_assignation_density
# print(assigned_groups)
for group in np.unique(assigned_groups):
    small_set = get_group_objects(group)
    ax.scatter(small_set[:, 0], small_set[:, 1], c=colors.pop(0))
    for circle in small_set:
        circle1 = plt.Circle((circle[0], circle[1]),
                             0.25, color='green', fill=False)
        ax.add_artist(circle1)

ax.set(xlabel='Seats count', ylabel='Distance range (km)',
       title='Aircrafts (clusters)')
ax.grid()
plt.show()

```

 ## Exercise 3: Build a dendrogram using dendrograms_history and pydot

 In this exercise we gonna use the variable dendrograms_history and pydot. Below we restore the variable and initialize the dendrogram graph.

```python
get_ipython().run_line_magic('store', '-r dendrogram_hist_diana')

print(dendrogram_hist_diana)

tree = pydot.Dot(graph_type='graph')

```

    [[{'acesor': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 'childs': [[6, 7, 8], [0, 1, 2, 3, 4, 5, 9]]}], [{'acesor': [6, 7, 8], 'childs': [[8], [6, 7]]}], [{'acesor': [0, 1, 2, 3, 4, 5, 9], 'childs': [[9], [0, 1, 2, 3, 4, 5]]}], [{'acesor': [0, 1, 2, 3, 4, 5], 'childs': [[0, 3, 4], [1, 2, 5]]}], [{'acesor': [0, 3, 4], 'childs': [[0], [3, 4]]}], [{'acesor': [1, 2, 5], 'childs': [[2], [1, 5]]}], [{'acesor': [1, 5], 'childs': [[1], [5]]}], [{'acesor': [6, 7], 'childs': [[6], [7]]}], [{'acesor': [3, 4], 'childs': [[3], [4]]}]]

 ### Fill ``show_tree_divisive`` function

 The function show_tree_divisive goes through each child node and build and edge between.

```python
def show_tree_divisive():
    for item in dendrogram_hist_diana:
        for child in item[0]["childs"]:
#             print(child)
            new_edge = pydot.Edge(str(item[0]["acesor"]), str(child))
            tree.add_edge(new_edge)
    tree.write('tree.png', format='png')

show_tree_divisive()
```

 Take a look if you did it properly:

```python
Image(filename='tree.png')
```

 ## Exercise 4: Implement the $s_{2}$ metric

 The $s_{2}$ metric gives a better understanding of the distances between centers.

 \begin{equation}
  s_{2}(c_{i},c_{j})=d(c_{i},c_{j}).
 \end{equation}

 Let's restore the centers from HCM grouped by two and initialize the values for three groups as below.

```python
get_ipython().run_line_magic('store', '-r new_centers_hcm')

new_centers_hcm = np.array(new_centers_hcm)
new_centers_hcm3 = np.array(
    [[0.42239686, 0.38503185], [0.07858546, 0.17832272], [0.82907662, 0.97059448]])

print(new_centers_hcm)
print(new_centers_hcm3)
```

    [[0.82907662 0.97059448]
     [0.12770138 0.20785259]]
    [[0.42239686 0.38503185]
     [0.07858546 0.17832272]
     [0.82907662 0.97059448]]

 Measure the distance between each center.

```python
def calculate_s_2(centers):
    s2 = []
    for center_1 in range(len(centers)):
        for center_2 in range(len(centers)):
            if center_1 == center_2:
                break
            s2.append(calculate_distance(centers[center_1], centers[center_2]))
            # fill the code here (3 lines)
    return s2
```

Test:

```python
print(calculate_s_2(new_centers_hcm))
print(calculate_s_2(new_centers_hcm3))
```

    [1.0361961303876361]
    [0.40116697670087065, 0.7129319889345509, 1.0912980907761376]

## Exercise 5: Modify the output image with borders between clusters

 We use the ``Segmentation`` class as in previous example.

```python

class Segmentation:

    def __init__(self, feature_matrix, groups):
        self.__data_set = feature_matrix
        self.__groups = groups
        self.__space = [[0, 255], [0, 255], [0, 255]]
        self.__error_margin = 0.5
        self.assignation = np.zeros((len(self.__data_set), self.__groups))
        self.centers = []
        self.select_centers()

    def select_centers(self):
        if len(self.centers) == 0:
            iter = 0
            while iter < self.__groups:
                self.centers.append(((random.randrange(0, 255)*1.0/255),
                                     (random.randrange(0, 255)*1.0/255),
                                     (random.randrange(0, 255)*1.0/255)))
                iter = iter+1

    def calculate_distance(self, x, v):
        return sqrt((x[0]-v[0])**2+(x[1]-v[1])**2+(x[2]-v[2])**2)

    def calculate_u(self, x, i):
        smallest_distance = float(self.calculate_distance(x, self.centers[0]))
        smallest_id = 0
        for i in range(1, self.__groups):
            distance = self.calculate_distance(x, self.centers[i])
            if distance < smallest_distance:
                smallest_id = i
                smallest_distance = distance
        distance = np.zeros(self.__groups)
        distance[smallest_id] = 1
        return distance

    def calculate_new_centers(self, u):
        new_centers = []
        for c in range(self.__groups):
            u_x_vector = np.zeros(len(self.centers[0]))
            u_scalar = 0
            for i in range(len(u)):
                u_scalar = u_scalar + u[i][c]
                u_x_vector = np.add(u_x_vector, np.multiply(
                    u[i][c], self.__data_set[i]))
            new_centers.append(np.divide(u_x_vector, u_scalar))
        self.centers = new_centers

    def calculate_differences(self, new_assignation):
        diff = 0
        for i in range(len(self.assignation)):
            for j in range(self.__groups):
                diff = diff + \
                    abs(float(new_assignation[i][j]) -
                        float(self.assignation[i][j]))
        return diff

    def do_segmentation(self):
        difference_limit_not_achieved = True
        iter = 0
        while difference_limit_not_achieved:
            new_assignation = []
            for i in range(len(self.__data_set)):
                new_assignation.append(
                    self.calculate_u(self.__data_set[i], iter))
            self.calculate_new_centers(new_assignation)

            if iter > 0:
                if self.calculate_differences(new_assignation) < self.__error_margin:
                    difference_limit_not_achieved = False
            self.assignation = new_assignation
            iter = iter + 1

    def get_results(self):
        return self.centers, self.assignation

```

 ### Change ``save_image`` method

 Add an if statement in the code below. It should consider the change of ``current_pixel`` variable. Please keep in mind that there should be three states considered.

```python
class ImageConversion:

    def get_image_from_url(self, img_url):
        image = open(img_url, 'rb')
        return img.imread(image)

    def get_unique_colours(self, image_matrix):
        feature_matrix = []
        for i in range(len(image_matrix)):
            for j in range(len(image_matrix[0])):
                feature_matrix.append(image_matrix[i, j])
        feature_matrix_np = np.array(feature_matrix)
        uniques, index = np.unique(
            [str(i) for i in feature_matrix_np], return_index=True)
        return feature_matrix_np[index], feature_matrix

    def save_image(self, size, pixel_matrix, unique_matrix, assignation_matrix, colours, output):
        image_out = PIL.Image.new("RGB", size)
        pixels = []
        current_pixel = None  # added
        for i in range(len(pixel_matrix)):
            pixel_list = pixel_matrix[i].tolist()
            for j in range(len(unique_matrix)):
                if (pixel_list == unique_matrix[j].tolist()):
                    for k in range(len(colours)):
                        if assignation_matrix[j][k] == 1:
                            segmented_colours = [int(i) for i in (colours[k] * 255)]
                            if current_pixel is not None and current_pixel != k:
                                segmented_colours = [0,0,0]
                            current_pixel = k
                            pixels.append(tuple(segmented_colours))
                            # fill here (about 6-7 lines)
        image_out.putdata(pixels)
        image_out.save(output)

    def do_segmentation(self):
        difference_limit_not_achieved = True
        iter = 0
        while difference_limit_not_achieved:
            new_assignation = []
            for i in range(len(self.__data_set)):
                new_assignation.append(
                    self.calculate_u(self.__data_set[i], iter))
            self.calculate_new_centers(new_assignation)
            if iter > 0:
                if self.calculate_differences(new_assignation) < self.__error_margin:
                    difference_limit_not_achieved = False
            self.assignation = new_assignation
            iter = iter + 1

    def get_results(self):
        return self.centers, self.assignation

```

 Execute segmentation without any changes:

```python
image_to_segment = "images/logo_krakow.png"
image_converter = ImageConversion()
image_data = image_converter.get_image_from_url(image_to_segment)
unique_image_data, image_data_list = image_converter.get_unique_colours(
    image_data)

groups = 3

segmentation = Segmentation(unique_image_data, groups)
segmentation.do_segmentation()
centers, assignation_matrix = segmentation.get_results()

image_size = (232, 258)
# image_converter.save_image(image_size, image_data_list, unique_image_data, assignation_matrix, centers, "images/segmented_hw.png")

```

```python
image_converter.save_image(image_size, image_data_list, unique_image_data, assignation_matrix, centers, "images/segmented_hw.png")
```

 The image should have black broders between one and the other segment.

```python
Image("images/segmented_hw.png")
```

```python

```


# PyCAS Tutorial
Welcome to the PyCAS Crash Course! Use SHIFT+ENTER to jump through the Notebook.
## Why PyCAS?
We wanted to make working with PyPropagate as easy as possible - even for inexperienced programmers - without loosing performance to inefficiently written python scripts. The solution was to use a symbolic library to define all simulation parameters. These are then optimized and compiled to low-level code on the fly. There is only one well-known symbolic library for python: `sympy`.

```python
import sympy as sp
```

However, working with `sympy` turned out to be quite complicated and soon some serious issues arose. For example, lets try to define a material with the complex refractive index $1+10^{-14}\;i$ in `sympy`:

```python
index_of_refraction = sp.S(1 + 10**-14 * 1j)
index_of_refraction.simplify()
```

    1.00000000000000

Yikes! It completely forgot about the imaginary part (at least in the current stable release: 0.7.6.1). As it turns out, this and other bugs make using `sympy` for serious tasks quite risky. Unfortunately, there are no known alternatives for python, so we had to write our own: the symbolic library `Expresso` and minimal computer algebra system `PyCAS` were born!

```python
import expresso.pycas as pc
```

```python
index_of_refraction = pc.S(1 + 10**-14 * 1j)
index_of_refraction.evaluate()
```

$$1+\frac{i }{10^{14}} \,  $$

## Working with symbols
We can define a `PyCAS` symbol using the `pc.Symbol` function. To define multiple symbols at once we can use `pc.symbols` with a comma-separated string.

```python
wavelength = pc.Symbol('lambda') # wavelength because lambda is a reserved keyword in python
x,y,z = pc.symbols('x,y,z')
```

We can create expressions by using some built-in python operators such as `+-*/` or by using predefined functions in `PyCAS`.

```python
pc.sqrt( pc.sin(x) + z / (wavelength + y) )
```

$$\sqrt{\frac{z}{\lambda +y} \,  +\text{sin}  \mathopen{} \left(x \right) \mathclose{} } '$$

We can also use numbers in our expressions, which will automatically be converted to PyCAS symbols. Be aware that operations on numbers will still first be evaluated in python before they are used in a `PyCAS` function. To explicitly convert a number to `PyCAS` we can use the `pc.S` function.

```python
x - z * 2 / 3
```

$$x-\frac{2z}{3} \,  $$

```python
x + 2 / 3 # Here the expression 1/3 is first evaluated by python (and integer division yields 0)
```

$$0+x$$

```python
x + pc.S(2) / 3 # The whole expression is converted to PyCAS
```

$$\frac{2}{3} \,  +x$$

### Exercises:
1. What happens when you try to convert a floating point number to an expression? Set the variable `n` below to different integer values (e.g. 1,2,3) and see what happens.
2. Can you explain this behavior?
3. Change the second line to preserve the exact representation of the number.

```python
n = ### <- insert integer here
pc.S(1./n)
```

## Manipulating expressions
`PyCAS` can evaluate and simplify expressions to some extent using the `expression.evalute()` method. Note that its power is still somewhat limited.

```python
expr = ((-2*x**(z)*(y+2*y/6)/x**(z-1)-y*(x+x/3)+42*3*x**2/(3*x*y**((x-2*x)/x)))+4*x*y)/(x*y)
expr
```

$$\frac{1}{x\, y} \, \left( 4x\, y+\frac{126}{3x\, {y}^{\frac{1}{x} \, \left( x-2x \right)  }} \, {x}^{2} +\frac{1}{{x}^{z-1}} \, \left( -2 \right) \, \left( \frac{2y}{6} \,  +y \right) \, {x}^{z} -\left( \frac{x}{3} \,  +x \right) \, y \right)  $$

```python
expr.evaluate()
```

$$42$$

```python
expr = pc.derivative(pc.exp(x*y)*x**2,x)
expr.evaluate()
```

$$\left( 2x+y\, {x}^{2} \right) \, \text{exp}  \mathopen{} \left(x\, y \right) \mathclose{} $$

To access the arguments of a function we can use the `expression.args` member of an expression. For commuatative functions the arguments are not necessarily in the same order as in the definition or when its printed. To get the callable function itself we can use the `expression.function` member.

```python
expr.evaluate().args
```

    [exp(x*y), x*2+y*x**2]

```python
expr.evaluate().args[1].function(x,y)
```

$$x+y$$

It is possible to define your own functions using the `pc.Function` class.

```python
f = pc.Function('f')
f(x**2+y)
```

$$f \mathopen{} \left(y+{x}^{2} \right) \mathclose{} $$

Values inside expressions can be substituted using the `expression.subs` method. It is possible to perform multiple substitutions at once.

```python
(x*y**2+y*wavelength).subs(y,wavelength).evaluate()
```

$$\left( 1+x \right) \, {\lambda }^{2}$$

```python
(x*(x+y)).subs([(x,y),(y,x)])
```

$$\left( x+y \right) \, y$$

### Exercises:
1. Define $a$ and $b$ as symbols in the cell below to create a piecewise defined function.
2. Change the substitution in two cells below so the result is $a$.
3. Can you chain the `expression.args` method multiple times to extract the $x>0$ condition from `piecewise_defined_function`? Note that in the special case of `piecewise` there is a hidden outer function.
4. Can a piecewise function contain more than two cases?
5. What happens when the derivative of a function is unknown? Define a function $g$ and calculate $\partial_x \; f(g(x))$ and $\partial_x \; f(x)\cdot g(x) $.

```python
a,b = ### <- define symbols here
piecewise_defined_function = pc.piecewise((a,x>0),(b,True))
piecewise_defined_function
```

```python
piecewise_defined_function.subs(x,0).evaluate()
```

```python
piecewise_defined_function.args[0]
```

```python

```

## Numerical evaluation and compilation
If an expression contains only numerical symbols and standard functions (functions which are defined in the python `mpmath` module) it can be evaluated numerically to an arbitrary precision. This is done with the `expression.N(precision)` function.

```python
(pc.sqrt(2)).N(100)
```

    mpf('1.414213562373095048801688724209698078569671875376948073176679737990732478462107038850387534327641572735')

```python
(4*pc.atan(1)).N(1000) # accurate to the first 1000 digits of pi
```

    mpf('3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117067982148086513282306647093844609550582231725359408128481117450284102701938521105559644622948954930381964428810975665933446128475648233786783165271201909145648566923460348610454326648213393607260249141273724587006606315588174881520920962829254091715364367892590360011330530548820466521384146951941511609433057270365759591953092186117381932611793105118548074462379962749567351885752724891227938183011949129833673362440656643086021394946395224737190702179860943702770539217176293176752384674818467669405132000568127145263560827785771342757789609173637178721468440901224953430146549585371050792279689258923542019956112129021960864034418159813629774771309960518707211349999998372978049951059731732816096318595024459455346908302642522308253344685035261931188171010003137838752886587533208381420617177669147303598253490428755468731159562863882353787593751957781857780532171226806613001927876611195909216420198931')

Expressions that contain symbols can be compiled to numpy (or c) functions that can operate on huge arrays very efficiently. The compiled function will be applied element-wise on the arrays. If the return value of the function cannot be deduced, the function result is casted to `complex` (this can be overwritten by defining the `restype` argument of `pc.numpyfy`). The compiled functions take the name of the symbols contained in the expression as keyword arguments.

```python
# import numpy and a pllotting library
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

    /usr/local/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

```python
func = pc.piecewise((pc.sin(2*x)**5,(pc.sin(x)>0) ),(pc.sin(4*x)**2,True))
func
```

$$\begin{cases} {\text{sin}  \mathopen{} \left(2x \right) \mathclose{} }^{5} & \text{if } \text{sin}  \mathopen{} \left(x \right) \mathclose{}  > 0 \\ {\text{sin}  \mathopen{} \left(4x \right) \mathclose{} }^{2} & \text{otherwise }  \end{cases}$$

```python
numpy_function = pc.numpyfy(func)
plt.plot(numpy_function(x = np.linspace(0,3*np.pi,1000)).real)
```

```python
func = pc.sin(x)*pc.sin(y)
func
```

$$\text{sin}  \mathopen{} \left(x \right) \mathclose{} \, \text{sin}  \mathopen{} \left(y \right) \mathclose{} $$

```python
numpy_function = pc.numpyfy(func,restype=float)
ny,nx = np.meshgrid(np.linspace(0,3*np.pi,1000),np.linspace(0,3*np.pi,1000))
plt.imshow(numpy_function(x = nx,y = ny))
plt.colorbar()
```

### Exercises:
1. Using the logical and function (`pc.And`) define a expression describing the refractive index of a square Waveguide (arbitrary size and material).
2. Plot the waveguide using `plt.imshow`.

```python

```

```python

```

## Array Access Operations
We can embed numpy arrays in our expressions using the `pc.array(name,array)` function definition. This will become useful in case an expression is too complicated or computationally intensive to be evaluated explicitly in the expression. Arra access operations are created by calling the array function just lika an ordinary function. If the argument is less than 0 or larger or equal ot the size of the array, the array access returns 0.

```python
rand = pc.array('rand',np.random.rand(100))
rand(x*100)
```

$$\text{rand}  \mathopen{} \left[ 10^{2}x \right] \mathclose{} $$

```python
plt.plot(pc.numpyfy(rand(x*10),restype=float)(x=np.linspace(-2,12,1000)))
```

### Exercises:
1. How could you define the refractive index $n(x,y)$ of [circular zoneplate](https://en.wikipedia.org/wiki/Zone_plate)?
2. How could you define the refractive index of a zoneplate with arbitrary zone distances using an array access operation?
3. **Bonus Points:** using the function below and an array access operation, define and plot the refractive index for a binary zoneplate with 100 zones.

```python
def zone_radius(i,wavelength = 1,F = 1):
    return (i*wavelength*F + i**2*wavelength**2/4)**0.5

i = pc.Symbol('i')
zone_radius(i).evaluate()
```

```python

```


# From transfer function to difference equation
In approximately the middle of Peter Corke's lecture [Introduction to digital control](https://youtu.be/XuR3QKVtx-g?t=34m56s), he explaines how to go from a transfer function description of a controller (or compensator) to a difference equation that can be implemented on a microcontroller.

The idea is to recognize that the term $$ sX(s) $$ in a transfer function is the laplace transform of the derivative of $x(t)$,
\begin{equation}
sX(s) + x(0) \quad \overset{\mathcal{L}}{\longleftrightarrow} \quad \frac{d}{dt} x(t),
\end{equation}
where the inital value $x(0)$ is often taken to be zero. We then make use of a discrete approximation of the derivative
$$ \frac{d}{dt}x(t) \approx \frac{x(t-h) - x(t)}{h}, $$
where $h$ is the time between the samples in the sampled version of signal $x(t)$.

The steps to convert the system on transfer function form
$$ Y(s) = F(s)U(s) = \frac{s+b}{s+a}U(s) $$
are to write
$$ (s+a)Y(s) = (s+b)U(s) $$
$$ sY(s) + aY(s) = sU(s) + bU(s), $$
take the inverse Laplace transform
$$ \frac{d}{dt} y + ay = \frac{d}{dt} u + bu$$
and use the discrete approximation of the derivative
$$ \frac{y_k - y_{k-1}}{h} + ay_k = \frac{u_k - u_{k-1}}{h} + bu_k $$
which can be written
$$ (1+ah) y_k = y_{k-1} + u_k - u_{k-1} + bh u_k,$$
or
$$ y_k = \frac{1}{1+ah} y_{k-1} + \frac{1+bh}{1+ah}u_k - \frac{1}{1+ah}u_{k-1}. $$

## Example
With the system
$$ F(s) =  \frac{s+1}{s+2} $$
and the sampling time
$$ h=0.1 $$
we get the difference equation
$$ y_k = \frac{1}{1.2}y_{k-1} + \frac{1.1}{1.2}u_k - \frac{1}{1.2} u_{k-1}. $$

Let's implement the system and see how the discrete approximation compares to the continuous-time system for the case of a step-response.

```python
import numpy as np
import scipy.signal as signal
import matplotlib.pyplot as plt
%matplotlib inline

# Define the continuous-time linear time invariant system F
a = 2
b = 1
num = [1, b]
den = [1, a]
F = signal.lti(num, den)

# Plot a step response
(t, y) = signal.step(F)
plt.figure(figsize=(14,6))
plt.plot(t, y, linewidth=2)

# Solve the difference equation y_k = c y_{k-1} + d_0 u_k + d_1 u_{k-1}
h = 0.1 # The sampling time

c = 1.0/(1 + a*h)
d0 = (1 + b*h) / (1 + a*h)
d1 = -c

td = np.arange(35)* h #The sampling time instants
ud = np.ones(35) # The input signal is a step, limited in time to 3.5 seconds
yd = np.zeros(35) # A vector to hold the discrete output signal
yd[0] = c*0 + d0*ud[0] - d1*0 # The first sample of the output signal
for k in range(1,35): # And then the rest
    yd[k] = c*yd[k-1] + d0*ud[k] + d1*ud[k-1]

plt.plot(td, yd, 'o', markersize=8)
plt.xlim([0, 3.5])
plt.ylim([0, 1])
plt.legend(('Continuous-time system', 'Discrete approximation'))
```

## Exercise
1. Why is the error in the discrete approximation larger in the beginning than at the end of the step-response?
2. Make a discrete approximation of the transfer function $$ F(s) = \frac{3}{s+3} $$ using the sampling time $$ h=0.2 $$ Then simulate and plot a step-response for the continuous- and discrete system, following the example above. *Hint*: Copy the python code for the example above into the code cell below and modify for the exercise.

```python
## Your python code goes here
```

# Recursively computing values of a polynomial using difference equations
In the lecture by Peter Corke, he talks about the historical importance of difference equations for computing values of a polynomial. Let's look at this in some more detail.

## A first order polynomial
Consider the polynomial
$$ p(x) =  4x + 2. $$
The first difference is
$$ \Delta p(x) = p(x) - p(x-h) = 4x + 2 - \big( 4(x-h) + 2 \big) = 4h, $$
and the second order difference is zero (as are all higher order differences):
$$ \Delta^2 p(x) = \Delta p(x) - \Delta p(x-h) = 4h - 4h = 0. $$

Using the firs order difference, we can also write the second order difference $ \Delta p(x) - \Delta p(x-h) = \Delta^2 p(x) $
as
$$ p(x) - p(x-h) - \Delta p(x-h) = \Delta^2p(x) $$
or
$$ p(x) = p(x-h) + \Delta p(x-h) + \Delta^2 p(x)$$
which for the first order polynomial above becomes
$$ p(x) = p(x-h) + \Delta p(x-h) = p(x-h) + 4h. $$

```python
def p1(x): return 4*x + 2 # Our first-order polynomial
# Compute values for x=[0,0.2, 0.4, ... 2] recursively using the difference equation
h = 0.2
x = h*np.arange(11) # Gives the array [0,0.2, 0.4, ... 2]
pd = np.zeros(11)
d1 = 4*h

# Need to compute the first value as the initial value for the difference equation,
pd[0] = p1(x[0])

for k in range(1,10): # Solve difference equation
    pd[k] = pd[k-1] + d1

plt.figure(figsize=(14,6))
plt.plot(x, p1(x), linewidth=2)
plt.plot(x, pd, 'ro')
```

## Second order polynomial
For a second order polynomial
$$ p(x) = a_0x^2 + a_1x + a_2 $$
we have
$$ p''(x) = 2a_0, $$
and the differences
$$ \Delta p(x) = p(x) - p(x-h) = a_0x^2 + a_1x + a_2 - \big( a_0(x-h)^2 + a_1(x-h) + a_2 \big) = h(2a_0x + a_1) -a_0h^2, $$
$$ \Delta^2 p(x) = \Delta p(x) - \Delta p(x-h) = h(2a_0x+a_1) - a_0h^2 - \big( h(2a_0(x-h) + a_1) - a_0 h^2 \big) =  h^22a_0 $$

Recall the difference equation using the second order difference
$$ p(x) = p(x-h) + \Delta p(x-h) + \Delta^2 p(x)$$
We now get
$$ p(x) = p(x-h) + \Delta p(x-h) + \Delta^2 p(x) = p(x-h) + \Delta p(x-h) + h^2 2 a_0,$$
or, using the definition of the first-order difference $\Delta p(x-h)$
$$ p(x) = 2p(x-h) - p(x-2h) + h^2 2 a_0,$$

Consider the second order polynomial
$$ p(x) = 2x^2 - 3x + 2, $$
and compute values using the difference equation.

```python
a0 = 2
a1 = -3
a2 = 2
def p2(x): return a0*x**2 + a1*x + a2 # Our second-order polynomial

# Compute values for x=[0,0.2, 0.4, ... 8] recursively using the difference equation
h = 0.2
x = h*np.arange(41) # Gives the array [0,0.2, 0.4, ... 2]
d1 = np.zeros(41) # The first differences
pd = np.zeros(41)
d2 = h**2*2*a0 # The constant, second difference

# Need to compute the first two values to get the initial values for the difference equation,
pd[0] = p2(x[0])
pd[1] = p2(x[1])

for k in range(2,41): # Solve difference equation
    pd[k] = 2*pd[k-1] - pd[k-2] + d2

plt.figure(figsize=(14,6))
plt.plot(x, p2(x), linewidth=2) # Evaluating the polynomial
plt.plot(x, pd, 'ro') # The solution using the difference equation
```

## Exercise
What order would the difference equation be for computing valuse of a third-order polynomial? What is the difference equation?

```python

```


# Baseball problem: Solution
Copyright © 2022 Oliver Beckstein.

This notebook is based on `11-ODE-applications.ipynb`. The whole problem was live-coded from scratch during class and this notebook is a cleaned-up and commented solution. The lesson showed how
1. to define the physical problem (obtain trajectory of a baseball with spin and air resistance)
2. to derive the underlying equations (Newton's equations of motions and the forces acting on the ball)
3. to adapt an algorithm to solve the equations (use RK4 for integrating the ODEs and express the problem in ODE standard form so that one can use `ode.rk4()`)
4. to visualize and discuss the results (plot trajectories and assess the influence of spin)

## Packages
We use numpy and our ODE solvers (`ode.py`):

```python
import numpy as np
import ode
```

For 3D graphs, install `ipympl` with `conda install ipympl` and then you can use
```
%matplotlib widget
```
but you have to do this as the very first thing (before loading `matplotlib`). You will also have to "close" active figures before you can plot into a new one.

```python
%matplotlib inline
#%matplotlib widget
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

## Simple Baseball physics

- quadratic air resistance (with velocity-independent drag coefficient)
- Magnus force due to spin

### Quadratic air resistance
Occurs at high Reynolds numbers, i.e., turbulent flow. Only approximate:

$$
\mathbf{F}_2 = -b_2 v \mathbf{v}
$$

with
$$
b_2 = \frac{1}{2} C_D \rho A
$$
where $C_D$ is the drag coefficient, $\rho$ the density of air, and $A = r^2 \pi$ the cross sectional area of the baseball.

### Magnus effect

**Magnus effect**: airflow is changed around a spinning object. The Magnus force is

$$
\mathbf{F}_M = \alpha \boldsymbol{\omega} \times \mathbf{v}
$$

where $\boldsymbol{\omega}$ is the ball's angular velocity in rad/s (e.g., 200/s for a baseball).

For a sphere the proportionality constant $\alpha$ can be written

$$
\mathbf{F}_M = \frac{1}{2} C_L \rho A \frac{v}{\omega} \boldsymbol{\omega} \times \mathbf{v}
$$

where $C_L$ is the lift coefficient, $\rho$ the air density, $A$ the ball's cross section. (Advantage of defining $C_L$ this way: when spin and velocity are perpendicular, the Magnus force is simply $F_M = \frac{1}{2} C_L \rho A v^2$.)

$C_L$ is mainly a function of the *spin parameter*

$$
S = \frac{r\omega}{v}
$$

with the radius $r$ of the ball. In general we write

$$
\mathbf{F}_M = \frac{1}{2} C_L(S)  \frac{\rho A r}{S} \boldsymbol{\omega} \times \mathbf{v}
$$

For a baseball, experimental data show approximately a power law dependence of $C_L$ on $S$

$$
C_L(S) = 0.62 \times S^{0.7}
$$

All together:

\begin{align}
\mathbf{F}_M &= \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
v &= \sqrt{\mathbf{v}\cdot\mathbf{v}}\\
S &= \frac{r\omega}{v}\\
C_L &= 0.62 \times S^{0.7}\\
\alpha &= \frac{1}{2} C_L  \frac{\rho A r}{S}
\end{align}

### Equations of motion

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= -g \hat{\mathbf{e}}_y -\frac{b_2}{m} v \mathbf{v} + \frac{\alpha}{m}\ \boldsymbol{\omega} \times \mathbf{v}
\end{align}

(quadratic drag $-\frac{b_2}{m} v \mathbf{v}$ included and dividing through by $m$ to get accelerations in the second equation.)

### Baseball simulation

Implement the full baseball equations of motions:
- gravity $a_\text{gravity}$
- quadratic drag $a_\text{drag}$
- Magnus effect $a_\text{Magnus}$

For the cross product you can look at [numpy.cross()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cross.html).

```python
def C_L(S):
    return 0.62 * S**0.7

def simulate_baseball(v0, omega, r0=None,
                      h=0.01, C_D=0.40, g=9.81, rho=1.225,
                      r=0.07468/2, m=0.14883,
                      R_homeplate=18.4):
    """simulate baseball pitch

    Parameters
    ----------
    v0 : array
         initial velocity (vx, vy, vz) in m/s
    omega : array
         angular velocity vector of the ball ("spin"), in rad/s
    r0 : array, optional
         initial position of the ball (in m) when it leaves the pitcher's hand
         as (x, y, z); the default is (0, 2, 0)
    h : float, optional
         integration time step in s, default is 0.01 s
    C_D : float, optional
         drag coefficient, default is 0.40
    g : float, optional
         acceleration due to gravity, default 9.81 kg/(m*s^2)
    rho : float, optional
         density of air, default 1.225 kg/m^3
    r : float, optional
         radius of the baseball
    m : float, optional
         mass of the baseball
    R_homeplate : float, optional
         distance of the catcher from the pitcher

    Returns
    -------

    positions : array
         The array contains an entry (time, x, y, z) for each time step.
    """
    # all SI units (kg, m)
    if r0 is None:
        r0 = np.array([0, 2.0, 0])  # pitching at 2m height

    omega = np.asarray(omega)

    domega = np.linalg.norm(omega)
    A = np.pi*r**2
    rhoArm = rho * A * r / m
    b2 = 0.5 * C_D * rho * A

    a_gravity = np.array([0, -g, 0])

    def f(t, y):
        # y = [x, y, z, vx, vy, vz]
        v = y[3:]
        dv = np.linalg.norm(v)
        S = r*domega/dv
        a_magnus = 0.5 * C_L(S) * rhoArm / S * np.cross(omega, v)
        a_drag = -b2/m * dv * v
        a = a_gravity + a_drag + a_magnus
        return np.array([y[3], y[4], y[5],
                         a[0], a[1], a[2]])

    t = 0
    # initialize 3D!
    y = np.array([r0[0], r0[1], r0[2], v0[0], v0[1], v0[2]], dtype=np.float64)
    positions = [[t, y[0], y[1], y[2]]] # record t, x and y, z

    while y[0] < R_homeplate and y[1] >= 0.2:
        t += h
        y[:] = ode.rk4(y, f, t, h)
        positions.append([t, y[0], y[1], y[2]])  # record t, x and y, z

    return np.array(positions)

```

#### Simulate throws

Simulate baseball throw for initial velocity $\mathbf{v} = (30\,\text{m/s}, 0)$.

Plot x vs y and x vs z (to see curving).

Try out different spins; a good value is $\boldsymbol{\omega} = 200\,\text{rad/s} \times (0, 1, 1)$.

200 rad/s is 32 revolutions per second or almost 2000 rpm.

```python
print(200/(2*np.pi), "revolutions per second")
print(200/(2*np.pi) * 60, "rpm")
```

    31.830988618379067 revolutions per second
    1909.859317102744 rpm

Simulate the baseball throw with
- almost no spin: $\omega = 0.001 \times (0, 0, 1)$  (our code does not handle $\omega = 0$ gracefully...)
- full upward spin: $\omega = 200 \times (0, 0, 1)$
- sideways spin: $\omega = 200 \times (0, 1, 1)$

```python
r = simulate_baseball([30, 0, 0], omega=0.001 * np.array([0,0,1]))
rz = simulate_baseball([30, 0, 0], omega=200. * np.array([0,0,1]))
rzy = simulate_baseball([30, 0, 0], omega=200. * 2**(-0.5) * np.array([0,1,1]))
```

#### Plotting

Plot the three scenarios in 2D planes: x-y (side view) and x-z (top view).

```python
idx = 2  # y
plt.plot(r[:,1], r[:,idx], label="no spin")
plt.plot(rz[:,1], rz[:,idx], label="upward spin")
plt.plot(rzy[:,1], rzy[:,idx], label="sideways spin")
plt.xlabel("$x$ (m)")
plt.ylabel("$y$ (m)")
plt.legend(loc="best")
plt.title("Side view");
```

Note how the balls with spin fly higher: the Magnus force lifts them up (due to the component of the spin along $z$).

```python
idx = 3  # z
plt.plot(r[:,1], r[:,idx], label="no spin")
plt.plot(rz[:,1], rz[:,idx], label="upward spin")
plt.plot(rzy[:,1], rzy[:,idx], label="sideways spin")
plt.xlabel("$x$ (m)")
plt.ylabel("$z$ (m)")
plt.legend(loc="best")
plt.title("Top-down view");
```

The ball with a component of the spin along the $y$ direction ($\boldsymbol{\omega} = \omega_0 \, \left(\begin{array}{c} 0\\1\\1\end{array}\right)$) has a 0.6 m deflection in the $-z$ direction.

#### 3D plot
Use simple `matplotlib` 3D plot. (BONUS: Make it work with vpython)

If we use the [`notebook` backend for matplotlib](http://ipython.readthedocs.io/en/stable/interactive/plotting.html) then we will be able to interactively rotate our [matplotlib 3D graphics](http://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html). (Note: If this does not seem to work, disable adblockers and allow javascript on the page.)

```python
%matplotlib widget
```

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(r[:,1], r[:,3], r[:,2], '.', label="no spin")
ax.plot(rz[:,1], rz[:,3], rz[:,2], '.', label="up spin")
ax.plot(rzy[:,1], rzy[:,3], rzy[:,2], '.', label="sideways spin")

# hand of the catcher, 0.2m above homeplate
ax.plot([18.4, 18.4], [0, 0], [0, 0.2], color="black", lw=6)

# drop line for last point
x,y,z = rzy[-1, 1:]
ax.plot([x,x], [z,z], [y,0], color="red")

ax.set_xlabel("$x$ (m)")
ax.set_ylabel("$z$ (m)")
ax.set_zlabel("$y$ (m)")
ax.legend(loc="upper left", numpoints=1)
ax.figure.tight_layout()
fig.savefig("baseball.png")
```

In the plot above we also added the hand of the catcher as a heavy black bar at 0.2 m above the home plate, and a drop-down bar in red to indicate the height of the ball thrown with *sideways spin* to make cleared that it also arrives at almost the same height as the ball with *up spin*.

```python

```


```python
import numpy as np
import scipy.constants as c
import math, cmath
from sympy import *
from math import e
import sympy as sp
r, x, a, theta = symbols('r x a theta')
init_printing(use_unicode=True)

def spherical_to_cartesian(r, theta, phi):
    x = r * np.sin(theta) * np.cos(phi)
    y = r * np.sin(theta) * np.sin(phi)
    z = r * np.cos(theta)
    return x, y, z

def cartesian_to_spherical(x, y, z):
    r = math.sqrt(x ** 2 + y ** 2 + z ** 2)
    theta = np.arccos(z / math.sqrt(x ** 2 + y ** 2 + z ** 2)) if r != 0 else 0
    if x == 0:
        phi = 0 if (y == 0) else 1.5708
    else:
        phi = np.arctan(y / x)
    return r, theta, phi

def absolute(number):
    return (number.real**2 + number.imag**2)**0.5

def P_l(l):
    ans = 1 / (2 ** l * factorial(l)) * sp.diff((x ** 2 - 1) ** l, x, int(l)) #integer l
    return ans

def Pm_l(m, l):
    ans = ((1 - x ** 2) ** (abs(m) / 2)) * sp.diff(P_l(l), x, abs(int(m))) #abs for int m
    return ans

def angular_wave_func(m, l, theta_value, phi):
    if m > 0:
        E = (-1)**m
    else:
        E = 1
    A_factor_1 = (2*l+1)/(4*math.pi)
    A_factor_2 = math.factorial(l-abs(m)) / math.factorial(l+abs(m))
    A = math.sqrt(A_factor_1*A_factor_2)
    B = cmath.exp(m*phi*1j)
    C = (Pm_l(m,l).subs(x,cos(theta)).subs(theta, theta_value)).evalf()
    long_ans = complex(E * A * B * C)
    ans = round(long_ans.real, 5) + round(long_ans.imag, 5)*1j

    return ans

def Lq(q):
    ans = e ** x * sp.diff(e ** -x * x ** q, x, int(q))
    return ans

def assoc_Lq(p, q):
    ans = (-1) ** p * sp.diff(Lq(q), x, int(p))
    return ans

def radial_wave_func(n, l, radius):
    if radius == 0:
        return np.nan
    else:
        bohr=c.physical_constants['Bohr radius'][0]
        A_factor_1 = (2/(n*a))**3
        A_factor_2 = (math.factorial(n-l-1)) / (2*n*(math.factorial(n+l))**3)
        A = (A_factor_1 * A_factor_2)**0.5
        B = e**(-radius/(n*a))
        C = ((2*radius)/(n*a))**l
        p = 2*l + 1
        q = n-l-1+p
        D = assoc_Lq(p,q)
        expression = A * B * C * D / (a**(-3/2))
        subbed = expression.subs(x, 2*r/(n*a)).subs(r, radius).subs(a, bohr)
        return round(subbed.evalf(), 5)

def linspace(start, stop, num=50):
    increment = (stop-start)/(num-1)
    current = start
    output = []
    for i in range(0,int((stop-start)/increment)+1):
        output.append(round(float(current),5))
        current+=increment
    return output

def meshgrid(x,y,z):
    output = [[],[],[]]
    output_0, output_1, output_2 = [],[],[]
    x_list, y_list, z_list = [],[],[]
    z_inner = []
    for k,z_i in enumerate(z):
        z_inner.append(z_i)
    for i,x_i in enumerate(x):
        x_list.append([x_i,x_i])
        z_list.append(z_inner)
    for j,y_i in enumerate(y):
        y_inner = [[y_i,y_i] for i in range(len(x))]
        y_list.append(y_inner)
        output_0.append(x_list)
        output_2.append(z_list)

    return output_0, y_list, output_2

cartesian_to_spherical_vector = np.vectorize(cartesian_to_spherical)
angular_wave_vector = np.vectorize(angular_wave_func)
radial_wave_vector = np.vectorize(radial_wave_func)
absolute_vector = np.vectorize(absolute)

def hydrogen_wave_func(n, l, m, roa, Nx, Ny, Nz):
    x_space = (linspace(-roa, roa, Nx))
    y_space = (linspace(-roa, roa, Ny))
    z_space = (linspace(-roa, roa, Nz))

    xx, yy, zz = np.meshgrid(y_space, x_space, z_space)

    r, theta, phi = (cartesian_to_spherical_vector(yy, xx, zz))

    if m == 0:
        angular = angular_wave_vector(m, l, theta, phi)
    elif m < 0:
        angular = (1j / math.sqrt(2)) * (angular_wave_vector(m, l, theta, phi) - (-1) ** m * angular_wave_vector(-m, l, theta, phi))
    elif m > 0:
        angular = (1 / math.sqrt(2)) * (angular_wave_vector(-m, l, theta, phi) + (-1) ** m * angular_wave_vector(m, l, theta, phi))

    radial = radial_wave_vector(n, l, r * a)

    mag = absolute_vector(radial * angular) ** 2

    return np.array(yy), np.array(xx), np.array(zz), np.round(mag, 5)
```

```python
hydrogen_wave_func (2,0,0,3,5,4,3)
```

    (array([[[-3. , -3. , -3. ],
             [-3. , -3. , -3. ],
             [-3. , -3. , -3. ],
             [-3. , -3. , -3. ]],

            [[-1.5, -1.5, -1.5],
             [-1.5, -1.5, -1.5],
             [-1.5, -1.5, -1.5],
             [-1.5, -1.5, -1.5]],

            [[ 0. ,  0. ,  0. ],
             [ 0. ,  0. ,  0. ],
             [ 0. ,  0. ,  0. ],
             [ 0. ,  0. ,  0. ]],

            [[ 1.5,  1.5,  1.5],
             [ 1.5,  1.5,  1.5],
             [ 1.5,  1.5,  1.5],
             [ 1.5,  1.5,  1.5]],

            [[ 3. ,  3. ,  3. ],
             [ 3. ,  3. ,  3. ],
             [ 3. ,  3. ,  3. ],
             [ 3. ,  3. ,  3. ]]]), array([[[-3., -3., -3.],
             [-1., -1., -1.],
             [ 1.,  1.,  1.],
             [ 3.,  3.,  3.]],

            [[-3., -3., -3.],
             [-1., -1., -1.],
             [ 1.,  1.,  1.],
             [ 3.,  3.,  3.]],

            [[-3., -3., -3.],
             [-1., -1., -1.],
             [ 1.,  1.,  1.],
             [ 3.,  3.,  3.]],

            [[-3., -3., -3.],
             [-1., -1., -1.],
             [ 1.,  1.,  1.],
             [ 3.,  3.,  3.]],

            [[-3., -3., -3.],
             [-1., -1., -1.],
             [ 1.,  1.,  1.],
             [ 3.,  3.,  3.]]]), array([[[-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.]],

            [[-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.]],

            [[-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.]],

            [[-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.]],

            [[-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.],
             [-3.,  0.,  3.]]]), array([[[5.60e-04, 7.20e-04, 5.60e-04],
             [7.10e-04, 5.70e-04, 7.10e-04],
             [7.10e-04, 5.70e-04, 7.10e-04],
             [5.60e-04, 7.20e-04, 5.60e-04]],

            [[6.90e-04, 6.40e-04, 6.90e-04],
             [6.80e-04, 6.00e-05, 6.80e-04],
             [6.80e-04, 6.00e-05, 6.80e-04],
             [6.90e-04, 6.40e-04, 6.90e-04]],

            [[7.20e-04, 5.00e-04, 7.20e-04],
             [5.70e-04, 3.66e-03, 5.70e-04],
             [5.70e-04, 3.66e-03, 5.70e-04],
             [7.20e-04, 5.00e-04, 7.20e-04]],

            [[6.90e-04, 6.40e-04, 6.90e-04],
             [6.80e-04, 6.00e-05, 6.80e-04],
             [6.80e-04, 6.00e-05, 6.80e-04],
             [6.90e-04, 6.40e-04, 6.90e-04]],

            [[5.60e-04, 7.20e-04, 5.60e-04],
             [7.10e-04, 5.70e-04, 7.10e-04],
             [7.10e-04, 5.70e-04, 7.10e-04],
             [5.60e-04, 7.20e-04, 5.60e-04]]]))

```python

```

```python

```


# Model project: Solow with natural ressources

```python
import numpy as np
from scipy import linalg
from scipy import optimize
import sympy as sm
import scipy as sc
import ipywidgets as widgets
import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, fixed, interact_manual
sm.init_printing(use_unicode=True)
```

The model considered in this project is a Solow model with a fixed natural ressource, X, which could be interpreted as land. The aim of the project is to describe the effect of the amount of land on the economy through numerical simulation of the model and visualizations. We illustrate the positive effects of savings and technological growth as well as the negative effect of population growth on the economy. Finally, we illustrate the diminishing returns to scale on land in the economy.

## The baseline model

(1) $ \quad Y_t = K_t^{\alpha} (A_tL_t)^{\beta} X^{\kappa} $

(2) $\quad K_{t+1} - K_t = s_K Y_t - \delta K_t $

(3) $\quad L_{t+1} = (1+n)L_t$

(4) $\quad A_{t+1} = (1+g)A_t$

We define $z_t$ as the capital/output ratio, $k_t/y_t$. By combining the model equations we can derive the transition equation for z which will be used for solving the model.

(5) $\quad z_{t+1} = (\frac{1}{(1+n)(1+g)} )^{\beta} (s + (1-\delta)z_t )^{1-\alpha} z_t^{\alpha}    $

(6) $\quad  y^* = (z^*)^{\frac{\alpha}{\beta + \kappa}} A_0^{\frac{\beta}{\beta + \kappa}} \big( \frac{X}{L_0} \big)^{\frac{\kappa}{\beta + \kappa}} (1+g)^{\frac{\beta}{\beta + \kappa}} (1+n)^{-\frac{\alpha}{\beta + \kappa}} $

# Solving the model

## The algorithm

The model is solved by numerically finding the steady state, defined as the point where $z_t = z_{t+1}$. We use equation (5) and construct a root-finding problem by subtracting $z_t$ from both sides:
$$ z^* = z_t = z_{t+1} \Leftrightarrow z_t - (\frac{1}{(1+n)(1+g)} )^{\beta} (s + (1-\delta)z_t )^{1-\alpha} z_t^{\alpha} = 0 $$
The root-finding problem is solved using scipys optimization function, in which we use the Brent-method. The Brent-method combines the bisection and the secant method making it a more robust and effective method than any of the two used seperately. It uses the secant method first, but turns to the bisectio method if  The secant method has a superlinear rate of convergence, but it does not have global convergence (can not always find a root). The bisection method does have global convergence, but only has a linear rate of convergence. Thus, by combing the two, we get a method which is faster than the bisection method, and more robust than the secant method.

## Numerical solution

### The effect of savings, technological growth and population growth.

To calculate the value of the capital/output ratio steady state $z^*$ numerically, we define the optimization function and the parameter values below. The optimization function is used to simulate the model, and the simulations are used to create an interactive plot of the steady state value of $z^*$. (Click dots below to view code for simulation and plotting.)

```python
#Define the parameters.
T = 200
k0 = 0.1
x = 100
A = 100
s = 0.2
g = 0.05
n = 0.05
alpha = 1/3
beta = 1/3
kappa = 1 - alpha - beta
delta = 0.05
```

```python
def get_steadystate(s,g,n,alpha,beta,kappa,delta):
    ### Function that finds the numerical steady state value for z by using Brent's method.
    obj_z = lambda z: z - (1/((1+n)*(1+g)))**beta *(s+(1-delta)*z)**(1-alpha)*z**alpha

    result = sc.optimize.root_scalar(obj_z,bracket=[0.1,100],method='brentq')
    return result.root

def solow_eq(z,x,A,alpha,beta,kappa,delta,s,n,g):

    ### Calculate z_t+1 given value of z_t, as defined by equation (XX)
    z_plus = (1/((1+n)*(1+g)))**beta *(s+(1-delta)*z)**(1-alpha)*z**alpha

    return z_plus

def simulate_solow_model(k0,x,A,alpha,beta,kappa,delta,s,n,g,T):
    ### Function that simulates the model by looping through all t from 1 to T.
    ### Returns a list holding the growth path for z.

    # a. initialize
    #k_path = [k0]
    z0 = k0**(1-alpha) * A**(-beta) * x**(-kappa)
    z_path = [z0]

    # b. simulate forward
    for t in range(1,T):

        # i. apply solow equation
        z_plus = solow_eq(t,x,A,alpha,beta,kappa,delta,s,n,g)

        # ii. save result
        z_path.append(z_plus)

    return z_path
```

```python
def _plot_zgrowth(k0,x,A,alpha,beta,kappa,delta,s,n,g,T):

    z_path = simulate_solow_model(k0,x,A,alpha,beta,kappa,delta,s,n,g,T)
    deg = np.linspace(0,0+T,T)

    #Create figure
    fig = plt.figure(figsize=(12, 6))
    ax = fig.add_subplot(1,1,1)
    ax.plot(z_path, label = 'Growth path for z')
    ax.plot(deg,lw=2,label='45 degree')

    # Calculate and insert st.st. indicator
    stst = get_steadystate(s,g,n,alpha,beta,kappa,delta)
    ax.scatter(stst,stst,s=20, marker = 'o', c='green', label= 'steady state')
    ax.text(stst + 0.2, stst, f'steady state for z = {stst:6.5f}', horizontalalignment='left', size='medium', color='black')

    # Format
    ax.set_title('Growth path and steady state for the capital/output ratio, $z_t$')
    ax.legend(loc='upper left')
    ax.set_xlim([0,10])
    ax.set_xlabel('$z_t$')
    ax.set_ylabel('$z_{t+1}$')
    ax.set_ylim([0,10])
    ax.grid(True)
    plt.show()
```

```python
interact(_plot_zgrowth
         ,k0=fixed(k0)
         ,x=fixed(x)
         ,A=fixed(A)
         ,alpha=fixed(alpha)#widgets.FloatSlider(description="alpha", min=0.05,max=1-alpha-kappa,step=0.05,value=0.3)
         ,beta=fixed(beta) #widgets.FloatSlider(description="beta", min=0.05,max=1-alpha-kappa,step=0.05,value=0.3)
         ,kappa=fixed(kappa) #widgets.FloatSlider(description="kappa", min=0.05,max=1-beta-alpha,step=0.05,value=0.3)
         ,delta=fixed(delta)
         ,s=widgets.FloatSlider(description="Savings, s", min=0.05,max=0.99,step=0.05,value=0.2)
         ,n=widgets.FloatSlider(description="Pop. growth, n", min=-0.3,max=1,step=0.001,value=0.05)
         ,g=widgets.FloatSlider(description="TFP growth, g", min=0,max=0.96,step=0.05,value=0.3)
         ,T=fixed(T));
```

    interactive(children=(FloatSlider(value=0.2, description='Savings, s', max=0.99, min=0.05, step=0.05), FloatSl…

Main (and well-known) takeaways from the interactive diagram of the numerically solved steady state value for the capital / output ratio:
 - Higher savings yield higher steady state of the capital output ratio.
 - There is a negative effect from higher population growth on the steady state capital output ratio.
 - The negative effect from a higher n can be made up for by a higher technological growth rate, g.

### The effect of the fixed amount of land per capita on income per capita

To examine the effect of land on the economy, we turn to the steady state level of income per capita, which is given by equation (6). As we see from the equation, the income per capita depends positively on the steady state capital/output ratio and on the fixed amount of land, X, that the country has. The positive relationship is illustrated in the plot below.

```python
def y(x):
    ### Function that evaluates the equation for y* given above.

    #Define parameter values (set to plausible) (Defining X/L as x)
    s = 0.2
    alpha = 1/3
    beta = 1/3
    kappa = 1/3
    delta = 0.05
    n = 0.02
    g = 0.05
    A = 100

    #Find z^*
    z = get_steadystate(s,g,n,alpha,beta,kappa,delta)

    #Calculate y
    y = z**(alpha/(beta+kappa)) * A**(beta/(beta+kappa)) * x**(kappa/(beta+kappa)) * (1+g)**(beta/(beta+kappa)) * (1+n)**(-kappa/(beta+kappa))

    return y
```

```python
# Define an array of x's and apply y function to x
x = np.linspace(0,200,100)
y_star = y(x)

#Create figure
fig = plt.figure(figsize=(12, 6))
ax = fig.add_subplot(1,1,1)
ax.plot(x,y_star, label='$y^*$')
ax.set_title('Steady state income per capita as a function of the amount of land per capita')
ax.legend(loc='upper left')
ax.set_xlabel('Amount of land per capita, x')
ax.set_ylabel('Steady state income per capita, $y^*$')
ax.grid(True)
```

The plot above shows the positive relationship between the amount of land and the steady state income, ceteris paribus. The graph displays as well the diminishing returns to scale on land through the stalling effect on the steady state income per capita as land gets bigger and bigger.

## Sources

- The model considered in this project as well as the intuition behind it is taken from Sørensen and Whitta, Introducing Advanced Macroeconomics, chapter 7.
- For this project we have used snippets from the NumEcon Snippets-2019 github repository (https://github.com/NumEconCopenhagen/snippets-2019 ). (Solow snippet for model simulation and Expenditure minimization snippet for the interactive graph.)

modelproject/20190510 - Model project.ipynb