# 2D Structure Extraction (Hough Transform)
In this exercise, we will implement a Hough transform in order to detect parametric curves, such as lines or circles.
In the following, we shortly review the motivation for this technique.

Consider the point $p=(\mathtt{x},\mathtt{y})$ and the equation for a line $y = mx+c$. What are the lines that could pass through $p$?
The answer is simple: all the lines for which $m$ and $c$ satisfy $\mathtt{y} = m\mathtt{x}+c$.
Regarding $(\mathtt{x},\mathtt{y})$ as fixed, the last equation is that of a line in $(m,c)$-space.
Repeating this reasoning, a second point $p'=(\mathtt{x}',\mathtt{y}')$ will also have an associated line in parameter space, and the two lines will intersect at the point $(\tilde{m},\tilde{c})$, which corresponds to the line connecting $p$ and $p'$.

In order to find lines in the input image, we can thus pursue the following approach.
We start with an empty accumulator array quantizing the parameter space for $m$ and $c$.
For each edge pixel in the input image, we then draw a line in the accumulator array and increment the corresponding cells.
Edge pixels on the same line in the input image will produce intersecting lines in $(m,c)$-space and will thus reinforce the intersection point.
Maxima in this array thus correspond to lines in the input image that many edge pixels agree on.

In practice, the parametrization in terms of $m$ and $c$ is problematic, since the slope $m$ may become infinite.
Instead, we use the following parametrization in polar coordinates:
\begin{equation}
	\mathtt{x}\cos\theta + \mathtt{y}\sin\theta = \rho \label{eq:hough_line}
\end{equation}
This produces a sinusoidal curve in $(\rho,\theta)$-space, but otherwise the procedure is unchanged.

The following sub-questions will guide you through the steps of building a Hough transform.

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
import imageio
import cv2
```

## Part a
Build up an accumulator array ``acc`` for the parameter space $(\rho, \theta)$. $\theta$ ranges from $-\pi/2$ to $\pi/2$, and $\rho$ ranges from $-D$ to $D$, where $D$ denotes the length of the image diagonal.
Use ``n_bins_rho`` and ``n_bins_theta`` as the number of bins in each direction.
Initially, the array should be filled with zeros.

For each edge pixel in the input image, create the corresponding curve in $(\rho, \theta)$ space by evaluating above line equation for all values of $\theta$ and increment the corresponding cells of the accumulator array.

```python
def hough_transform(edge_image, n_bins_rho, n_bins_theta):
    # Vote accumulator
    votes = np.zeros((n_bins_rho, n_bins_theta), dtype=np.int)

    # Create bins
    diag = np.linalg.norm(edge_image.shape)  # Length of image diagonal
    theta_bins = np.linspace(-np.pi / 2, np.pi / 2, n_bins_theta)
    rho_bins = np.linspace(-diag, diag, n_bins_rho)

    # YOUR CODE HERE
    raise NotImplementedError()
    return votes, rho_bins, theta_bins
```

Test the implementation on an example image. Visualize the resulting Hough space by displaying it as a 2D image.

```python
color_im = imageio.imread('gantrycrane.png')
gray_im = cv2.cvtColor(color_im, cv2.COLOR_RGB2GRAY)

# Get edges using Canny
blurred = cv2.GaussianBlur(gray_im, None, sigmaX=2)
edges = cv2.Canny(blurred, threshold1=30, threshold2=90)  # 30, 90 are manually tunned

hough_space, rho_bins, theta_bins = hough_transform(edges, n_bins_rho=300, n_bins_theta=300)

fig, axes = plt.subplots(1, 3, figsize=(12, 4))
axes[0].imshow(color_im)
axes[0].set_title('Image')
axes[1].imshow(edges, cmap='gray')
axes[1].set_title('Edges')
axes[2].imshow(hough_space)
axes[2].set_title('Hough space')
axes[2].set_xlabel('theta (index)')
axes[2].set_ylabel('rho (index)')
fig.tight_layout()
```

## Part b
Write a function ``nms2d`` which suppresses all points in the Hough space that are not local maxima.
This can be achieved by looking at the 8 direct neighbors of each pixel and keeping only pixels whose value is greater than all its neighbors.
This function is simpler than the non-maximum suppression from the Canny Edge Detector since it does not take into account local gradients.

```python
def nms2d(hough_array):
    hough_array_out = np.zeros_like(hough_array)

    # YOUR CODE HERE
    raise NotImplementedError()

    return hough_array_out
```

Write a function ``find_hough_peaks`` that takes the result of ``hough_transform`` as an argument, finds the extrema in Hough space using ``nms2d`` and returns the index of all points $(\rho_i, \theta_i)$ for which the corresponding Hough value is greater than ``threshold``.

```python
def find_hough_peaks(hough_space, threshold):
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
def plot_hough_lines(image, rho, theta):
    # compute start and ending point of the line x*cos(theta)+y*sin(theta)=rho
    x0, x1 = 0, image.shape[1] - 1
    y0 = rho / np.sin(theta)
    y1 = (rho - x1 * np.cos(theta)) / np.sin(theta)

    # Check out this page for more drawing function in OpenCV:
    # https://docs.opencv.org/3.1.0/dc/da5/tutorial_py_drawing_functions.html
    for yy0, yy1 in zip(y0, y1):
        cv2.line(image, (x0, int(yy0)), (x1, int(yy1)), color=(255, 0, 0), thickness=1)

    return image
```

Try your implementation on the images ``gantrycrane.png`` and ``circuit.png``.
Do you find all the lines?

YOUR ANSWER HERE

```python
# Find maximum
rho_max_idx, theta_max_idx = find_hough_peaks(hough_space, 200)
print(f'gantrycrane.png: found {len(rho_max_idx)} lines in the image.')
rho_max, theta_max = rho_bins[rho_max_idx], theta_bins[theta_max_idx]

color_image = imageio.imread('gantrycrane.png')
image_with_lines = plot_hough_lines(color_image, rho_max, theta_max)

# Plot
fig, ax = plt.subplots(figsize=(8, 4))
ax.imshow(image_with_lines)
```

```python
# Try another image
im = imageio.imread('circuit.png')

blurred = cv2.GaussianBlur(im, None, sigmaX=2)
edge = cv2.Canny(blurred, threshold1=30, threshold2=90)
hough_space, rho_bins, theta_bins = hough_transform(edge, n_bins_rho=300, n_bins_theta=300)

# Find maximum
rho_max_idx, theta_max_idx = find_hough_peaks(hough_space, 100)
print(f'circuit.png: found {len(rho_max_idx)} lines in the image.')
rho_max, theta_max = rho_bins[rho_max_idx], theta_bins[theta_max_idx]
color_image = cv2.cvtColor(im, cv2.COLOR_GRAY2RGB)
image_with_lines = plot_hough_lines(color_image, rho_max, theta_max)

# Plot
fig, ax = plt.subplots(figsize=(8, 4))
ax.imshow(image_with_lines)
fig.tight_layout()
```

## Part c (bonus)

The Hough transform is a general technique that can not only be applied to lines, but also to other parametric curves, such as circles.
In the following, we will show how the implementation can be extended to finding circles.

A circle can be parameterized by the following equation:
$$
    (\mathtt{x}-a)^2 + (\mathtt{y}-b)^2 = r^2. \label{eq:hough_circle}
$$

Unfortunately, the computation and memory requirements for the Hough transform increase exponentially with the number of parameters.
While a 3D search space is still just feasible, we can dramatically reduce the amount of computation by integrating the gradient direction in the algorithm.

Without gradient information, all values $a, b$ lying on the cone given by above equation are incremented.
With the gradient information, we only need to increment points on an arc centered at $(a, b)$:
$$
\begin{eqnarray}
	a &=& x + r\cos\phi\\
	b &=& y + r\sin\phi,
\end{eqnarray}
$$
where $\phi$ is the gradient angle returned by the edge operator.

Create a function ``hough_circle`` which implements the Hough transform for circles.
Try your implementation for a practical application of counting coins in an image.
You can use the images ``coins1.png`` and ``coins2.png`` for testing.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

## Pard d (bonus)
The same trick (as in **Part c**) of using the image gradient can be used for lines.
Modify the code from **Part a** to only vote for one line per edge pixel, instead of all the lines running through this pixel.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

## Part e (bonus)
Can you build an online coin classification and counting system?

You can take a look at the ``Haribo classification`` demo (MATLAB) in the Moodle for some ideas. Use the functions you wrote in the previous questions.
(Hint: you may need to include a reference shape in the picture in order to obtain the absolute scale).

```python
# YOUR CODE HERE
raise NotImplementedError()
```


# Optimization

Almost every problem in machine learning and data science starts
with a dataset $X$, a model $g(\theta)$, which is a function of the parameters $\theta$ and a cost function $C(X, g(\theta))$ that allows us to judge how well the
model $g(\theta)$ explains the observations $X$. The model is fit by finding the values of $\theta$ that minimize the cost function.

We will look at a class of methods for computing minima of functions known as gradient descent and its generalizations.

# Gradient Descent

The basic idea of gradient descent is that a function $F(\mathbf{x})$, $ \mathbf{x} \equiv (x_1,\cdots,x_n)$, decreases fastest if one goes from $\bf {x}$ in the direction of the negative gradient $-\nabla F(\mathbf{x})$.

It follows that, if
\begin{equation}
\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma \nabla F(\mathbf{x}_k), \ \ \gamma > 0
\end{equation}
for $\gamma$ small enough, then $F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k)$. This means that for a sufficiently small $\gamma$ we moves towards smaller function values, i.e a minimum.

This observation is the basis of the gradient descent (GD) method. One starts with an initial guess $\mathbf{x}_0$ for a minimum of $F$ and compute new approximations according to

\begin{equation}
\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma \nabla F(\mathbf{x}_k), \ \ k \geq 0.
\end{equation}

Ideally the sequence $\{ \mathbf{x}_k \}_{k=0}$ converges to a __global__ minimum of the function $F$. In general we do not know if we are in a global or local minimum.  In the special case when $F$ is a convex function, all local minima are also global minima, so in this case gradient descent can converge to the global solution. We will explore this further in the exercises.

In a practical implementation we have to choose a criterion for when to stop the iteration. One such condition would be to stop when

\begin{equation}
||\nabla F(\mathbf{x})|| < \epsilon,
\end{equation}
where $\epsilon$ is some small number.

The parameter $\gamma$ is referred to as the step size and is constant in the simplest Gradient Descent scheme. We will consider generalizations later where $\gamma$ is adaptet during each iteration in order to obtain faster convergence or esacpe local minima.

Another point worth noticing is that in order to use gradient descent all we need to know is the function and its gradient. Computing the gradient may be difficult depending on the complexity of the function we want to minimize.

## Exercise 1
We start by considering functions of a single variable since they are easy to work with and we can compute exact minima to compare our implementation. Furthermore, single variable functions captures much of the problems we experience computing minima of more complex, multi-variable functions.

a) Consider the function $f(x) = x^2+1$. Show that $f(x_{k+1}) \leq f(x_k)$ when $0 \leq \gamma \leq 1$.

Hint: Use that (GD step) $x_{k+1} = x_k - \gamma f'(x_k)$ and solve the resulting inequality.

Solution:

\begin{align}
x_k^2(1-2\gamma)^2 + 1 &\leq x_k^2+1 \\
(1-2\gamma)^2 &\leq 1 \\
\gamma(1-\gamma) &\leq 0 \\
\Rightarrow 0 \leq \gamma &\leq 1.
\end{align}

b) Show/convince yourself that f has its minimum at $x=0$ with $f(0) = 1$.

Write a function computes the minimum using the GD method, with $x_0 = -3$ as initial guess. Try to write the function in a general way such that you in principle can use it on any function.

Experiment with different step sizes in the range $0 < \gamma < 1$ and check how many iterations you need in order for the solution to converge within a precision of $10^{-6}$.

Since the derivative has to be zero in a minimum you can use $$|f'(x_k)| < \epsilon,$$ with $\epsilon = 10^{-6}$ as a criterion for convergence. (The choice of $10^{-6}$ as precision is arbitrary).

What happens if you choose $\gamma$ exactly equal to 1? What happens if $\gamma$ is slighly larger than 1? One way to get a better visual understanding of whats going on is to plot the function and the function values at each iteration, $f(x_k)$, within the same plot.

```python
#Program that solves exercise 1b.
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def gradient_descent(xk,dx_f,gamma):
    return xk-gamma*dx_f

def quadratic(a,b,c,x):
    return a*x**2+b*x+c

def dx_quadratic(a,b,x):
    return 2*a*x+b

#One variable examples
a,b,c   = 1,0,1
x       = np.linspace(-5,5,101)
quad    = quadratic(a,b,c,x)
dx_quad = dx_quadratic(a,b,x)

xk        = -3
xk_vec    = [xk]
fxk_vec   = [quadratic(a,b,c,xk)]
gamma     = 0.3
iters     = 0
max_iters = 1000
converged = False

while(abs(dx_quadratic(a,b,xk)) > 1e-6 and iters < max_iters):
    xk = gradient_descent(xk,dx_quadratic(a,b,xk),gamma)
    xk_vec.append(xk)
    fxk_vec.append(quadratic(a,b,c,xk))
    iters += 1

if(iters < max_iters):
    converged = True

print ("Converged: %s" % converged)
print ("Number of iterations for convergence: %d" % iters)

plt.figure(1)
plt.plot(x,quad)
plt.plot(xk_vec,fxk_vec,'o')
plt.legend(["f(x)","GD-iterates"])
plt.show()
```

c) Quadratic functions as the one in exercise b) are particularly forigiving to work with since they only have one minimum/maximum, which in turn is global. A third order polynomial can have a maximum and a minimum or a saddle point. A fourth order polynomial may have two local minima. The point of the following exercise is to investigate the how GD depends on the initial guess, $x_0$.

Consider the function
\begin{equation}
f(x) = \frac{(x+4)(x+1)(x-1)(x-3)}{14} + \frac{1}{2},
\end{equation}
with derivative
\begin{equation}
f'(x) = \frac{1}{14}\left( 4x^3 + 3x^2 - 26x -1 \right).
\end{equation}

Make a plot of the function for $x \in [-5,4]$. The function has a global minimum at $x \approx -2.9354$, a local maximum at $x \approx -0.038301$ and a local minimum at $x \approx 2.2237$. Choose $\gamma = 0.1$ as step length and use GD descent to compute the minimum.

* Expermient with different initial values $x_0 \in [-5,-0.1]$ and $x_0 \in [0.1, 4]$.
* What happens if you choose $x_0 = -0.038301$ (i.e at the local maximum)? Explain why this happens.
* What happens if you choose $x_0$ slightly smaller/larger than $-0.038301$?
* Furthermore you can experiment with different step sizes.

```python
def quartic(x):
    return (x+4)*(x+1)*(x-1)*(x-3)/14.0 + 0.5
def dx_quartic(x):
    return (1.0/14.0)*(4*x**3 + 3*x**2 - 26*x - 1)

x = np.linspace(-5,4,101)
quart    = quartic(x)
dx_quart = dx_quartic(x)

xk        = -0.02
xk_vec    = [xk]
fxk_vec   = [quartic(xk)]
gamma     = 0.1
iters     = 0
max_iters = 200
converged = False

while(abs(dx_quartic(xk)) > 1e-6 and iters < max_iters):
    xk = gradient_descent(xk,dx_quartic(xk),gamma)
    xk_vec.append(xk)
    fxk_vec.append(quartic(xk))
    iters += 1

if(iters < max_iters):
    converged = True

print ("Converged: %s" % converged)
print ("Number of iterations for convergence: %d" % iters)

plt.figure(1)
plt.plot(x,quart)
plt.plot(xk_vec,fxk_vec,'o')
plt.legend(["f(x)","GD-iterates"])
plt.show()
```

d) In this exercise we will look at function of two variables.

Consider the function
\begin{equation}
z(x,y) = x^2+10y^2-1.
\end{equation}

* Compute the gradient and show that $z(x,y)$ has its minimum at $x=0, y=0$.
* Extend your program such that it can compute the minimum of a multivariate function. The main difference now is that the points and derivative/gradient now are vectors/arrays and not just scalars. Also try to make relevant visualizations, such as surface or contour plots.
* As before, experiment with different step sizes $\gamma$ and intitial values $\mathbf{x}_0 = (x_0,y_0)$.

```python
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm

#Two variable example
def Z(x,y):
    return x**2+10*y**2-1

def grad_Z(x,y):
    gZ = np.zeros(2)
    gZ[0] = 2*x
    gZ[1] = 20*y
    return gZ

X = np.arange(-4,4,0.1)
Y = np.arange(-5,5,0.1)

X_, Y_ = np.meshgrid(X, Y)
Z_ = Z(X_,Y_)
fig = plt.figure(2)
ax = fig.gca(projection='3d')
surf = ax.plot_surface(X_, Y_, Z_, cmap=cm.coolwarm,linewidth=0, antialiased=False)
plt.show()
plt.figure(3)
plt.contour(X_,Y_,Z_,corner_mask=0)
plt.show()

xk = np.zeros(2)
xk[0] = 1.5
xk[1] = 2.3

gamma = 0.05
iters = 0
max_iters = 200
converged = False
while(abs(np.linalg.norm(grad_Z(xk[0],xk[1]))) > 1e-6 and iters < max_iters):
    xk = xk - gamma*grad_Z(xk[0],xk[1])
    iters += 1
print ("Number of iterations for convergence: %d" % iters)
print(xk)
```

## Exercise 2 (Linear regression)

In this exercise we will work out how Gradient Descent is used in the context of linear regression. The method is unchanged, however we have to minimize a different function, namely the cost function.

\begin{equation}
\hat{y} = \theta_0x_0 + \sum_{i=1}^n \theta_i x_i, \ \ \hat{y} = \theta^T \cdot \bar{x}
\end{equation}
where $x_0 \equiv 1$ by convention (?)

The normal equation
\begin{equation}
\hat{\theta} = (X^TX)^{-1} X^Ty
\end{equation}

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

#One variable example
N  = 100
x0 = np.ones(N)
x1 = 2*np.random.rand(N)
y = 4 + 3*x1 + np.random.randn(N)

#Compute theta and predicted y using normal equations
X = np.c_[x0,x1]
Xt_X_inv = np.linalg.inv(np.dot(X.transpose(),X))
Xt_y     = np.dot(X.transpose(),y)
theta_normeqs = np.dot(Xt_X_inv,Xt_y)

#Compute theta using gradient descent
eta = 0.1
max_iters = 100
theta = np.random.randn(2)

diff = 100
iters = 0
while(diff > 1e-10):
    gradient = 2.0/float(N) * np.dot(X.transpose(), np.dot(X,theta) - y)
    theta = theta-eta*gradient
    diff = np.linalg.norm(gradient)
    iters += 1

#Output number of iterations before convergence and compare theta computed with GD with theta computed
#using the Normal equations
print("Number of iterations before convergence: %d" % iters)
print(abs(theta_normeqs-theta))
print(theta_normeqs)
print(theta)

#Plot true y and y_predicted
plt.figure(4)
y_pred = theta[0] + theta[1]*x1
plt.plot(x1,y,'ro')
plt.plot(x1,y_pred,'-b')
plt.show()
```

```python
#Two variable example
N = 100
x0 = np.ones(N)
x1 = 2*np.random.rand(N)
x2 = 2*np.random.rand(N)

noise_scale = 2
g_noise = noise_scale*np.random.randn(N)

y = 4+3*x1+2*x2+g_noise

#Compute theta using normal equations
X = np.c_[x0,x1,x2]
Xt_X_inv = np.linalg.inv(np.dot(X.transpose(),X))
Xt_y     = np.dot(X.transpose(),y)
theta_normeqs = np.dot(Xt_X_inv,Xt_y)

#Compute theta using gradient descent
eta = 0.1
max_iters = 100
theta = np.random.randn(3)

diff = 100
iters = 0
while(diff > 1e-10):
    gradient = 2.0/float(N) * np.dot(X.transpose(), np.dot(X,theta) - y)
    theta = theta-eta*gradient
    diff = np.linalg.norm(gradient)
    iters += 1

#Output number of iterations before convergence and compare theta computed with GD with theta computed
#using the Normal equations
print("Number of iterations before convergence: %d" % iters)
print(abs(theta_normeqs-theta))
print(theta_normeqs)
print(theta)

#Plot true y and y_predicted
y_pred = theta[0] + theta[1]*x1+theta[2]*x2
fig = plt.figure(5)
ax = fig.gca(projection='3d')
scatter1 = ax.scatter(x1, x2, y,marker='^',c='r')
scatter2 = ax.scatter(x1, x2, y_pred,marker='o',c='b')
plt.show()
```

## Short summary so far

Summary...

```python

```


# Merge Sort

Many problems require sorting a collection of items by their keys, which are assumed to be comparable. Simple algorithums such as bubble sort and insertion sort can do this in $O(n^2)$ time, however using a divide and conquer approach, merge sort acheives $O(n \lg n)$.

Other than runtime, their are two additional characteristics we may be intrested in when comparing sorting algorithums
- __in-place__: Only $O(1)$ extra space is required to sort the array
- __stable__: For every $i$, $j$ where $i < j$ and $A[i].key = A[j].key$, $A[i]$ is guarenteed to proceed $A[j]$,
(the ordering of equal keys remains the same).

## Algorithum

Merge sort works by recursivly splitting the array into two, until we reach arrays of size 1, which by definition are sorted.

```
Algorithum mergeSort(A, i, j)
    mid <- floor((i + j) / 2)
    mergeSort(A, i, mid)
    mergeSort(A, mid+1, j)
    merge(A, i, mid, j)
```

Merging two sorted arrays is simple. Create a scratch array $B$, then look at the first element in each of the two halves. Copy the smallest element and repeat until all elements have been copyed. Finnaly copy array $B$ back into array $A$.

```
Algorithum merge(A, i, mid, j)
    initialize array B with length j - i + 1
    k <- i
    l <- mid + 1
    m <- 0

    while k <= mid and l <= j do
        if A[k].key <= A[l].key then
            B[m] <- A[l].key
            k <- k + 1
        else
            B[m] <- A[l]
            l <- l + 1
        m <- m + 1

    while k <= mid do
        B[m] <- A[l]
        k <- k + 1
        m <- m + 1

    while l <= j do
        B[m] <- A[l]
        l <- l + 1
        m <- m + 1

    for m = 0 to j - i do
        A[m + i] <- B[m]
```

Clearly mergo sort is not _in-place_ since it uses an auxillary array $B$, however it is _stable_ since we always merge $A$ first if the keys are equal, and since all elements of $A$ came before $B$ `mergeSort` will never swap equal keys.

## Runtime

The runtime of `mergeSort` is simple, some constant amount of work for the mid point, then the sum of the three function calls.

$$
T_{mergeSort} = \begin{cases}
    O(1) & n \leq 1 \\
    O(1) + T_{mergeSort}(\lfloor{\frac{n}{2}}\rfloor) + T_{mergeSort}(\lceil{\frac{n}{2}}\rceil) + T_{merge}(n) & n \geq 2 \\
\end{cases}
$$

The worst case runtime of `merge` is $O(n)$ since their are no nested loops and each loop runs at most $n$ times. The last loop always runs $n$ times thus it is also $\Omega(n)$, thus `merge` us $\Theta(n)$.

Lets assume $n = 2^k$ for some integer $k$, thus we dont need to consider the ceil and floor's, now to solve the rest of the recurence we can expand and collect terms:

$$
\begin{align}
T_{mergeSort}(n) &= 2 T_{mergeSort}\left(\frac{n}{2}\right) + \Theta(n) \\
                 &= 2 \left(2 T_{mergeSort}\left(\frac{n}{2^2}\right) + \Theta\left(\frac{n}{2}\right)\right) + \Theta(n) \\
                 &= 4 T_{mergeSort}\left(\frac{n}{2^2}\right) + 2 \Theta\left(\frac{n}{2}\right) + \Theta(n) \\
                 &= 4 \left( 2 T_{mergeSort}\left(\frac{n}{2^3}\right) + \Theta\left(\frac{n}{2^2}\right) \right) + 2 \Theta\left(\frac{n}{2}\right) + \Theta(n) \\
                 &=  8 T_{mergeSort}\left(\frac{n}{2^3}\right) + 4 \Theta\left(\frac{n}{2^2}\right) + 2 \Theta\left(\frac{n}{2}\right) + \Theta(n) \\
                 &\vdots \\
                 &= 2^k T_{mergeSort}\left(\frac{n}{2^k}\right) + \sum_{i=0}^{k-1}{2^i \Theta{\left(\frac{n}{2^i}\right)}} \\
                 &= n T_{mergeSort}\left(1\right) + \sum_{i=0}^{k-1}{2^i \Theta{\left(\frac{n}{2^i}\right)}} \\
                 &= n \Theta(1) + \sum_{i=0}^{k-1}{2^i \Theta{\left(\frac{n}{2^i}\right)}} \\
                 &= \Theta(n) + \sum_{i=0}^{\lg n -1}{2^i \Theta{\left(\frac{n}{2^i}\right)}} \\
                 &= \Theta(n) + \Theta(n \lg n) \\
                 &= \Theta(n \lg n) \\
\end{align}
$$

To complete this proof we would use induction to fill in the "gap" when unwinding the recurance. Also, to show this works for any $n$ we use induction to show $T(n) \leq T(n+1)$ then using the facts that $n \leq 2^k < 2n$ for some $k$ and $\frac{n}{2} \leq 2^{k'} < n$ for some $k'$ we get

$$
T(n) \leq T(2^k) = O(2^k \lg 2^k) = O(2n \lg 2n) = O(n \lg n)
$$

and

$$
T(n) = \Omega(2^{k'} \lg 2^{k'}) = \Omega\left(\frac{n}{2} \lg \frac{n}{2}\right) = \Omega(n \lg n)
$$

thus $T(n) = \Omega(n \lg n)$

## Masters Theorem

The proof above was from first princibles, but we could instead use _masters theorem_.

Let $n_0 \in \mathbb{N}$, $k \in \mathbb{N}$, and $a, b \in \mathbb{R}$ with $a > 0$ and $b > 1$, let $T: \mathbb{N} \rightarrow \mathbb{N}$ satisfy the following recurrence:

$$
T(n) = \begin{cases}
\Theta(1) & n < n_0 \\
a \Theta\left(\frac{n}{b}\right) + \Theta(n^k) & n \geq n_0
\end{cases}
$$

Let the critical component $e = \log_b(a)$, then:

$$
T(n) = \begin{cases}
\Theta(n^e) & k < e \\
\Theta(n^e \lg n) & k = e \\
\Theta(n^k) & k > e \\
\end{cases}
$$

For merge sort we have the terms $a = 2$, $b = 2$, $k = 1$, $n_0 = 2$, thus $e = \log_2(2) = 1$. $k = e$ thus by case $II$ of Masters Theorem merge sort is $\Theta(n \lg n)$.


# Lecture 14, Algebraic Modeling Languages

Algebraic Modeling Languages (AML) are high-level computer programming languages for describing and solving high complexity problems for large scale mathematical computation (i.e. large scale optimization type problems).  Their syntax mimics the mathematical notation of optimization problems, which allows one to express optimization problems in a familiar, concise and readable way.

**AMLs do not directly solve the problem, but they call appropriate external solvers to find the solution.**

Examples of AMLs are
* A Mathematical Programming Language (AMPL),
* General Algebraic Modeling System (GAMS),
* Optimization Programming Language (OPL),
* Advanced Interactive Multidimensional Modeling System (AIMMS), and
* Pyomo.

In addition to the ease of modelling, one of the advantages of AMLs is that you can model the problem once and then solve it with multiple solvers.

## Pyomo

On this course, we use Pyomo as an example of AMLs. Pyomo is a Python-based, open-source optimization modeling language with a diverse set of optimization capabilities.

Pyomo may not be a completely typical AML, because Pyomo's modeling objects are embedded within a full-featured high-level programming language providing a rich set of supporting libraries, which distinguishes Pyomo from other AMLs.

Pyomo supports a wide range of problem types, including:
* Linear programming
* Quadratic programming
* Nonlinear programming
* Mixed-integer linear programming
* Mixed-integer quadratic programming
* Mixed-integer nonlinear programming
* Stochastic programming
* Generalized disjunctive programming
* Differential algebraic equations
* Bilevel programming
* Mathematical programs with equilibrium constraints

# Installing Pyomo

The easiest way to install Pyomo is to call
```
pip install pyomo
```
when pip has been installed on your machine.

## Example 1, linear optimization

Let us start with a very simple linear problem
$$
\begin{align}
\min &\qquad   2x_1+3x_2\\
\text{s.t. }& \qquad 3x_1+4x_2\geq 1\\
& \qquad x_1,x_2\geq 0.
\end{align}
$$

```python
from pyomo.environ import *

model = ConcreteModel()

model.x = Var([1,2], domain=NonNegativeReals) #Non-negative variables x[1] and x[2]

model.OBJ = Objective(expr = 2*model.x[1] + 3*model.x[2]) #Objective function

model.Constraint1 = Constraint(expr = 3*model.x[1] + 4*model.x[2] >= 1) #Constraint

```

Once we have defined the problem, we can solve it. Let us start by using glpk, which is an open source linear programming program.

You need to have glpk installed on your system. For details, see https://www.gnu.org/software/glpk/#TOCdownloading. For many Linux distributions, you can install glpk from the repositories by typing
```
sudo yum install glpk
```
```
sudo apt-get install glpk,
```
or whatever your distribution needs.

```python
from pyomo.opt import SolverFactory #Import interfaces to solvers
opt = SolverFactory("glpk") #Use glpk
res = opt.solve(model, tee=True) #Solve the  problem and print the output
print "Solution:"
print "========="
model.x.display() #Print values of x
```

    GLPSOL: GLPK LP/MIP Solver, v4.52
    Parameter(s) specified in the command line:
     --write /tmp/tmpogS1Ue.glpk.raw --wglp /tmp/tmpA5Hc_F.glpk.glp --cpxlp /tmp/tmpkKjSRP.pyomo.lp
    Reading problem data from '/tmp/tmpkKjSRP.pyomo.lp'...
    2 rows, 3 columns, 3 non-zeros
    21 lines were read
    Writing problem data to `/tmp/tmpA5Hc_F.glpk.glp'...
    15 lines were written
    GLPK Simplex Optimizer, v4.52
    2 rows, 3 columns, 3 non-zeros
    Preprocessing...
    1 row, 2 columns, 2 non-zeros
    Scaling...
     A: min|aij| =  3.000e+00  max|aij| =  4.000e+00  ratio =  1.333e+00
    Problem data seem to be well scaled
    Constructing initial basis...
    Size of triangular part is 1
          0: obj =   0.000000000e+00  infeas =  1.000e+00 (0)
    *     1: obj =   7.500000000e-01  infeas =  0.000e+00 (0)
    *     2: obj =   6.666666667e-01  infeas =  0.000e+00 (0)
    OPTIMAL LP SOLUTION FOUND
    Time used:   0.0 secs
    Memory used: 0.0 Mb (40408 bytes)
    Writing basic solution to `/tmp/tmpogS1Ue.glpk.raw'...
    7 lines were written
    Solution:
    =========
    x : Size=2, Index=x_index, Domain=NonNegativeReals
        Key : Lower : Value          : Upper : Fixed : Stale
          1 :     0 : 0.333333333333 :  None : False : False
          2 :     0 :            0.0 :  None : False : False

Now, if you have other linear solvers installed on your system, you can use them too. Let us use Cplex, which is a commercial solvers (academic license available).

```python
opt = SolverFactory("cplex")
res = opt.solve(model, tee=True)
print "Solution:"
model.x.display()
```

    Welcome to IBM(R) ILOG(R) CPLEX(R) Interactive Optimizer 12.6.2.0
      with Simplex, Mixed Integer & Barrier Optimizers
    5725-A06 5725-A29 5724-Y48 5724-Y49 5724-Y54 5724-Y55 5655-Y21
    Copyright IBM Corp. 1988, 2015.  All Rights Reserved.

    Type 'help' for a list of available commands.
    Type 'help' followed by a command name for more
    information on commands.

    CPLEX> Logfile 'cplex.log' closed.
    Logfile '/tmp/tmpnxw9d9.cplex.log' open.
    CPLEX> Problem '/tmp/tmpEdgLVy.pyomo.lp' read.
    Read time = 0.00 sec. (0.00 ticks)
    CPLEX> Problem name         : /tmp/tmpEdgLVy.pyomo.lp
    Objective sense      : Minimize
    Variables            :       3
    Objective nonzeros   :       2
    Linear constraints   :       2  [Greater: 1,  Equal: 1]
      Nonzeros           :       3
      RHS nonzeros       :       2

    Variables            : Min LB: 0.000000         Max UB: all infinite
    Objective nonzeros   : Min   : 2.000000         Max   : 3.000000
    Linear constraints   :
      Nonzeros           : Min   : 1.000000         Max   : 4.000000
      RHS nonzeros       : Min   : 1.000000         Max   : 1.000000
    CPLEX> Tried aggregator 1 time.
    LP Presolve eliminated 2 rows and 3 columns.
    All rows and columns eliminated.
    Presolve time = 0.00 sec. (0.00 ticks)

    Dual simplex - Optimal:  Objective =  6.6666666667e-01
    Solution time =    0.00 sec.  Iterations = 0 (0)
    Deterministic time = 0.00 ticks  (2.72 ticks/sec)

    CPLEX> Solution written to file '/tmp/tmpjWjvnD.cplex.sol'.
    CPLEX> Solution:
    x : Size=2, Index=x_index, Domain=NonNegativeReals
        Key : Lower : Value          : Upper : Fixed : Stale
          1 :     0 : 0.333333333333 :  None : False : False
          2 :     0 :            0.0 :  None : False : False

We can use also gurobi, which is another commercial solver with academic license.

```python
opt = SolverFactory("gurobi")
res = opt.solve(model, tee=True)
print "Solution:"
model.x.display()
```

    Optimize a model with 2 rows, 3 columns and 3 nonzeros
    Coefficient statistics:
      Matrix range    [1e+00, 4e+00]
      Objective range [2e+00, 3e+00]
      Bounds range    [0e+00, 0e+00]
      RHS range       [1e+00, 1e+00]
    Presolve removed 2 rows and 3 columns
    Presolve time: 0.00s
    Presolve: All rows and columns removed
    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    6.6666667e-01   0.000000e+00   0.000000e+00      0s

    Solved in 0 iterations and 0.00 seconds
    Optimal objective  6.666666667e-01
    Solution:
    x : Size=2, Index=x_index, Domain=NonNegativeReals
        Key : Lower : Value          : Upper : Fixed : Stale
          1 :     0 : 0.333333333333 :  None : False : False
          2 :     0 :            0.0 :  None : False : False

## Example 2, nonlinear optimization

Let use define optimization problem
$$
\begin{align}
\max &\qquad c_b\\
\text{s.t. }& \qquad c_{af}s_v - s_vc_a-k_1c_a=0\\
&\qquad s_vc_b+k_1c_a-k_2c_b-2k_3c_a^2=0\\
&\qquad s_vc_c+k_2c_b=0\\
&\qquad s_vc_d+k_3c_a^2=0,\\
&\qquad s_v,c_a,c_b,c_c,c_d\geq0
\end{align}
$$
where $k_1=5/6$, $k_2=5/3$, $k_3=1/6000$, and $c_{af}=10000$.

```python
from pyomo.environ import *
# create the concrete model
model = ConcreteModel()
# set the data
k1 = 5.0/6.0
k2 = 5.0/3.0
k3 = 1.0/6000.0
caf = 10000.0
# create the variables
model.sv = Var(initialize = 1.0, within=PositiveReals)
model.ca = Var(initialize = 5000.0, within=PositiveReals)
model.cb = Var(initialize = 2000.0, within=PositiveReals)
model.cc = Var(initialize = 2000.0, within=PositiveReals)
model.cd = Var(initialize = 1000.0, within=PositiveReals)

# create the objective
model.obj = Objective(expr = model.cb, sense=maximize)
# create the constraints
model.ca_bal = Constraint(expr = (0 == model.sv * caf \
    - model.sv * model.ca - k1 * model.ca \
    - 2.0 * k3 * model.ca ** 2.0))
model.cb_bal = Constraint(expr=(0 == -model.sv * model.cb \
    + k1 * model.ca - k2 * model.cb))
model.cc_bal = Constraint(expr=(0 == -model.sv * model.cc \
    + k2 * model.cb))
model.cd_bal = Constraint(expr=(0 == -model.sv * model.cd \
    + k3 * model.ca ** 2.0))
```

## Solving with Ipopt

Install IPopt following http://www.coin-or.org/Ipopt/documentation/node10.html.

```python
opt = SolverFactory("ipopt",solver_io="nl")

opt.solve(model,tee=True)

print "Solution is "
model.sv.display()
model.ca.display()
model.cb.display()
model.cc.display()
model.cd.display()
```

    ******************************************************************************
    This program contains Ipopt, a library for large-scale nonlinear optimization.
     Ipopt is released as open source code under the Eclipse Public License (EPL).
             For more information visit http://projects.coin-or.org/Ipopt
    ******************************************************************************

    This is Ipopt version 3.12, running with linear solver mumps.
    NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

    Number of nonzeros in equality constraint Jacobian...:       11
    Number of nonzeros in inequality constraint Jacobian.:        0
    Number of nonzeros in Lagrangian Hessian.............:        5

    Total number of variables............................:        5
                         variables with only lower bounds:        5
                    variables with lower and upper bounds:        0
                         variables with only upper bounds:        0
    Total number of equality constraints.................:        4
    Total number of inequality constraints...............:        0
            inequality constraints with only lower bounds:        0
       inequality constraints with lower and upper bounds:        0
            inequality constraints with only upper bounds:        0

    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
       0 -2.0000000e+03 7.50e+03 6.25e-01  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
       1 -1.0801475e+03 5.54e+02 2.46e+00  -1.0 1.39e+03    -  5.54e-01 1.00e+00h  1
       2 -1.0763574e+03 8.86e+01 1.87e+02  -1.0 3.66e+02    -  9.31e-01 1.00e+00h  1
       3 -1.0727252e+03 1.07e+01 5.26e+00  -1.0 9.35e+01    -  9.71e-01 1.00e+00h  1
       4 -1.0726714e+03 4.01e+00 1.22e-01  -1.0 6.03e+01    -  9.68e-01 1.00e+00h  1
       5 -1.0724371e+03 3.44e-04 2.93e-05  -2.5 4.19e-01    -  1.00e+00 1.00e+00h  1
       6 -1.0724372e+03 1.63e-08 3.93e-09  -3.8 3.85e-03    -  1.00e+00 1.00e+00h  1
       7 -1.0724372e+03 5.68e-11 2.68e-11  -5.7 2.24e-04    -  1.00e+00 1.00e+00h  1
       8 -1.0724372e+03 4.55e-13 2.51e-14  -8.6 2.78e-06    -  1.00e+00 1.00e+00h  1

    Number of Iterations....: 8

                                       (scaled)                 (unscaled)
    Objective...............:  -1.0724372001086319e+03   -1.0724372001086319e+03
    Dual infeasibility......:   2.5080209371475748e-14    2.5080209371475748e-14
    Constraint violation....:   1.1368683772161604e-14    4.5474735088646412e-13
    Complementarity.........:   2.5059065225787342e-09    2.5059065225787342e-09
    Overall NLP error.......:   2.5059065225787342e-09    2.5059065225787342e-09

    Number of objective function evaluations             = 9
    Number of objective gradient evaluations             = 9
    Number of equality constraint evaluations            = 9
    Number of inequality constraint evaluations          = 0
    Number of equality constraint Jacobian evaluations   = 9
    Number of inequality constraint Jacobian evaluations = 0
    Number of Lagrangian Hessian evaluations             = 8
    Total CPU secs in IPOPT (w/o function evaluations)   =      0.002
    Total CPU secs in NLP function evaluations           =      0.000

    EXIT: Optimal Solution Found.

    Ipopt 3.12: Optimal Solution Found
    Solution is
    sv : Size=1, Index=None, Domain=PositiveReals
        Key  : Lower : Value         : Upper : Fixed : Stale
        None :     0 : 1.34381176107 :  None : False : False
    ca : Size=1, Index=None, Domain=PositiveReals
        Key  : Lower : Value         : Upper : Fixed : Stale
        None :     0 : 3874.25886723 :  None : False : False
    cb : Size=1, Index=None, Domain=PositiveReals
        Key  : Lower : Value         : Upper : Fixed : Stale
        None :     0 : 1072.43720011 :  None : False : False
    cc : Size=1, Index=None, Domain=PositiveReals
        Key  : Lower : Value         : Upper : Fixed : Stale
        None :     0 : 1330.09353341 :  None : False : False
    cd : Size=1, Index=None, Domain=PositiveReals
        Key  : Lower : Value         : Upper : Fixed : Stale
        None :     0 : 1861.60519963 :  None : False : False

# Black-box optimization using scipy.optimize

Often cases, you do not have algebraic formulations of the objective functions, but instead, you have an executable, which gives you the values and you do not know what is happening inside there.

### Example

Executable 'prob4' (which you need to compile before using) includes a script for two variable problem with three inequality constraints (where two of them involve only one variable). The problem is of the form
$$
\min \  f(x)
\\ \text{s.t. }g(x) <= 2
\\          x\geq 0.
$$
The executable reads in a file 'input.txt', which contains variable values on top of each other and outputs a file "output.txt", which contains on top of each other value of f, value of g, value of $x_1$, value of $x_2$, gradient of f and gradient of the contraints.

Let us solve this problem using *scipy.optimize*

```python
import csv
def evaluate_prob4(x):
    with open('input.txt','w') as f:
        f.write('%f\n%f'%(x[0],x[1])) #Write x[0] and x[1] to the input.txt file
    !./prob4 #Execute prob4
    val = []
    with open('output.txt','r') as f:
        valuereader = csv.reader(f)
        for row in valuereader:
            val.extend([float(i) for i in row])
    f_val = val[0]
    g_val = [0]*3
    g_val[0] = 2-val[1]
    g_val[1]=val[2]
    g_val[2]=val[3]
    grad_f=[val[4],val[5]]
    grad_g = [[0,0],[0,0],[0,0]]
    grad_g[0] = [-val[6],-val[7]]
    grad_g[1] = [val[8],val[9]]
    grad_g[2] = [val[10],val[11]]
    return f_val,g_val,grad_f,grad_g

```

```python
import math
evaluate_prob4([2,0.])
```

    (3.0, [-2.0, 2.0, 0.0], [-6.0, 2.0], [[-4.0, -0.0], [1.0, 1.0], [1.0, 1.0]])

```python
from scipy.optimize import minimize

constraint_tuple=(
    {'type':'ineq','fun':lambda x:evaluate_prob4(x)[1][0],\
     'jac':lambda x:evaluate_prob4(x)[3][0]},
    {'type':'ineq','fun':lambda x:evaluate_prob4(x)[1][1],\
     'jac':lambda x:evaluate_prob4(x)[3][1]},
    {'type':'ineq','fun':lambda x:evaluate_prob4(x)[1][2],\
     'jac':lambda x:evaluate_prob4(x)[3][2]}
)
```

```python
minimize(lambda x: evaluate_prob4(x)[0], [0,0], method='SLSQP'
                        , jac=lambda x: evaluate_prob4(x)[2],
         constraints = constraint_tuple,options = {'disp':True})
```

# Example 3, Nonlinear multiobjective optimization

Let us study optimization problem
$$
\begin{align}
\min \ & \left(\sum_{i=1}^{48}\frac{\sqrt{1+x_i^2}}{v_i},\sum_{i=1}^{48}\left(\left(\frac{x_iv_i}{\sqrt{1+x_i^2}}+v_w\right)^2+\frac{v_i^2}{1+x_i^2}\right)\right., \\
&\qquad\left.\sum_{i=1}^{47}\big|(x_{i+1}-x_i\big|\right)\\
\text{s.t. } & \sum_{i=1}^{j}x_i\leq -1\text{ for all }j=10,11,12,13,14\\
& \left|\sum_{i=1}^{j}x_i\right|\geq 2\text{ for all }j=20,21,22,23,24\\
& \sum_{i=1}^{j}x_i\geq 1\text{ for all }j=30,31,32,33,34\\
&\sum_{i=1}^{48}\frac{\sqrt{1+x_i^2}}{v_i} \leq 5\\
&\sum_{i=1}^{48}x_i=0\\
&-10\leq\sum_{i=1}^{j}x_i\leq10\text{ for all }j=1,\ldots,48
&0\leq v_i\leq 25\text{ for all }i=1,\ldots,48\\
&-10\leq x_i\leq 10\text{ for all }i=1,\ldots,48\\
\end{align}
$$

```python

from pyomo.environ import *
# create the concrete model9
def solve_ach(reference,lb,ub):
    model = ConcreteModel()

    vwind = 5.0
    min_speed = 0.01

    #f1, time used
    def f1(model):
        return sum([sqrt(1+model.y[i]**2)/model.v[i] for i in range(48)])
    #f2, wind drag, directly proportional to square of speed wrt. wind
    def f2(model):
        return sum([((model.y[i]*model.v[i])/sqrt(1+model.y[i]**2)+vwind)**2/
                    +model.v[i]**2*((1+model.y[i])**2) for i in range(48)])
    #f3, maximal course changes
    def f3(model):
        return sum([abs(model.y[i+1]-model.y[i]) for i in range(47)])

    def h1_rule(model,i):
        return sum(model.y[j] for j in range(i))<=-1
    def h2_rule(model,i):
        return abs(sum(model.y[j] for j in range(i)))>=2
    def h3_rule(model,i):
        return sum(model.y[j] for j in range(i))>=1
    def h4_rule(model):
        return sum([sqrt(1+model.y[i]**2)/model.v[i] for i in range(48)])<=25
    def h5_rule(model):
        return sum(model.y[i] for i in range(48))==0

    def f_rule(model):
        return t

    def y_init(model,i):
        if i==0:
            return -1
        if i==18:
            return -1
        if i==24:
            return 1
        if i==25:
            return 1
        if i==26:
            return 1
        if i==34:
            return -1
        return 0
    model.y = Var(range(48),bounds = (-10,10),initialize=y_init)
    model.v = Var(range(48),domain=NonNegativeReals,bounds=(min_speed,25),initialize=25)
    model.t = Var()
    model.h1=Constraint(range(9,14),rule=h1_rule)
    model.h2=Constraint(range(19,24),rule=h2_rule)
    model.h3=Constraint(range(29,34),rule=h3_rule)
    model.h4=Constraint(rule=h4_rule)
    model.h5=Constraint(rule=h5_rule)

    def h6_rule(model,i):
        return -10<=sum([model.y[j] for j in range(i)])<=10

    model.h6 = Constraint(range(1,48),rule=h6_rule)
    def t_con_f1_rule(model):
        return model.t>=(f1(model)-reference[0]-lb[0])/(ub[0]-lb[0])
    model.t_con_f1 = Constraint(rule = t_con_f1_rule)
    def t_con_f2_rule(model):
        return model.t>=(f2(model)-reference[1]-lb[1])/(ub[1]-lb[1])
    model.t_con_f2 = Constraint(rule = t_con_f2_rule)
    def t_con_f3_rule(model):
        return model.t>=(f3(model)-reference[2]-lb[2])/(ub[2]-lb[2])
    model.t_con_f3 = Constraint(rule = t_con_f3_rule)
    model.f = Objective(expr = model.t+1e-10*(f1(model)+f2(model)+f3(model)))
    tee =False
    opt = SolverFactory("ipopt",solver_io="nl")
    opt.options.max_iter=100000
    #opt.options.constr_viol_tol=0.01
    #opt.options.halt_on_ampl_error = "yes"

    opt.solve(model,tee=tee)
    return [[value(f1(model)),value(f2(model)),value(f3(model))],[model.y,model.v]]

```

```python
lb_ = [0,0,0]
ub_ = [1,1,1]
values =[]
for i in range(3):
    reference = [1e10,1e10,1e10]
    reference[i]=0
    values.append(solve_ach(reference,ub_,lb_)[0])
print values
```

    WARNING - Loading a SolverResults object with a warning status into model=unknown; message from solver=Ipopt 3.12\x3a Maximum Number of Iterations Exceeded.
    WARNING - Loading a SolverResults object with a warning status into model=unknown; message from solver=Ipopt 3.12\x3a Maximum Number of Iterations Exceeded.
    [[25133.93493837363, 759264160.1226324, 860.0], [25.000000247539568, 11188.12151236145, 147.55358533550918], [12673.93579526275, 289156195.6539606, 151.2406436251979]]

```python
lb = [0,0,0]
ub = [1,1,1]
for i in range(3):
    lb[i] = min([values[j][i] for j in range(3)])
    ub[i] = max([values[j][i] for j in range(3)])
print lb
print ub
```

```python
[f,x] = solve_ach([(a+b)/2 for (a,b) in zip(lb,ub)],lb,ub) #Compromise solution
#[f,x] = solve_ach([1e10,1e10,0],lb,ub) #Minimize the third objective

```

```python
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
plt.plot([sum(value(x[0][j]) for j in range(i)) for i in range(49)])
currentAxis = plt.gca()
currentAxis.add_patch(Rectangle((10, -1),4,10))
currentAxis.add_patch(Rectangle((20, -2),4,4))
currentAxis.add_patch(Rectangle((30, -10),4,11))
plt.show()
```

Lecture 10, Algebraic Modeling Languages, especially Pyomo.ipynb


# SymPy

[SymPy](https://es.wikipedia.org/wiki/SymPy) es una biblioteca de Python que permite realizar cálculos simbólicos.
Nos ofrece las capacidades de álgebra computacional, y se puede usar en línea a través de [SymPy Live](http://live.sympy.org/) o [SymPy Gamma](http://www.sympygamma.com/), este último es similar a
[Wolfram Alpha](https://www.wolframalpha.com/).

Si usas Anaconda este paquete ya viene instalado por defecto pero si se usa miniconda o pip debe instalarse.

````python
conda install sympy # Usando el gestor conda de Anaconda/Miniconda
pip install sympy # Usando el gestor pip (puede requerir instalar más paquetes)
````

Lo primero que debemos hacer, antes de usarlo, es importar el módulo, como con cualquier
otra biblioteca de Python.

Si deseamos usar SymPy de forma interactiva usamos

```python
from sympy import *
init_printing()
```

Para scripting es mejor importar la biblioteca de la siguiente manera

```python
import sympy as sym
```

Y llamar las funciones de la siguiente manera

```python
x = sym.Symbols("x")
expr = sym.cos(x)**2 + 3*x
deriv = expr.diff(x)
```

en donde calculamos la derivada de  $\cos^2(x) + 3x$,
que debe ser $-2\sin(x)\cos(x) + 3$.

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
```

```python
init_printing()
```

Definamos la variable $x$ como un símbolo matemático. Esto nos permite hacer uso de
esta variable en SymPy.

```python
x = symbols("x")
```

Empecemos con cálculos simples. Abajo, tenemos una _celda de código_ con una suma.
Ubica el cursor en ella y presiona SHIFT + ENTER para evaluarla.

```python
1 + 3
```

Realicemos algunos cálculos.

```python
factorial(5)
```

```python
1 // 3
```

```python
1 / 3
```

```python
S(1) / 3
```

Podemos evaluar esta expresión a su versión en punto flotante

```python
sqrt(2*pi)
```

```python
float(sqrt(2*pi))
```

También podemos almacenar expresiones como variables, como cualquier variable de Python.

```python
radius = 10
height = 100
area = pi * radius**2
volume = area * height
```

```python
volume
```

```python
float(volume)
```

Hasta ahora, hemos usado SymPy como una calculadora. Intentemos
algunos cálculos más avanzados. Por ejemplo, algunas integrales.

```python
integrate(sin(x), x)
```

```python
integrate(sin(x), (x, 0, pi))
```

Podemos definir una función, e integrarla

```python
f = lambda x: x**2 + 5
```

```python
f(5)
```

```python
integrate(f(x), x)
```

```python
y = symbols("y")
integrate(1/(x**2 + y), x)
```

Si asumimos que el denominador es positivo, esta expresión se puede simplificar aún más

```python
a = symbols("a", positive=True)
integrate(1/(x**2 + a), x)
```

Hasta ahora, aprendimos lo más básico. Intentemos algunos ejemplos
más complicados ahora.

**Nota:** Si quieres saber más sobre una función específica se puede usar
la función ``help()`` o el comándo _mágico_ de IPython ``??``

```python
help(integrate)
```

```python
integrate??
```

## Ejemplos

### Solución de ecuaciones algebraicas

Para resolver sistemas de ecuaciones algebraicos podemos usar:
[``solveset`` and ``solve``](http://docs.sympy.org/latest/tutorial/solvers.html).
El método preferido es ``solveset``, sin embargo, hay sistemas que
se pueden resolver usando ``solve`` y no ``solveset``.

Para resolver sistemas usando ``solveset``:

```python
a, b, c = symbols("a b c")
solveset(a*x**2 + b*x + c, x)
```

Debemos ingresar la expresión igualada a 0, o como una ecuación

```python
solveset(Eq(a*x**2 + b*x, -c), x)
```

``solveset`` no permite resolver sistemas de ecuaciones no lineales, por ejemplo

```python
solve([x*y - 1, x - 2], x, y)
```

### Álgebra lineal

Usamos ``Matrix`` para crear matrices. Las matrices pueden contener variables y expresiones matemáticas.

Usamos el método ``.inv()`` para calcular la inversa, y ``*`` para multiplicar matrices.

```python
A = Matrix([
        [1, -1],
        [1, sin(c)]
    ])
display(A)
```

```python
B = A.inv()
display(B)
```

```python
A * B
```

Esta expresión debería ser la matriz identidad, simplifiquemos la expresión.
Existen varias formas de simplificar expresiones, y ``simplify`` es la más general.

```python
simplify(A * B)
```

### Graficación

SymPy permite realizar gráficos 2D y 3D

```python
from sympy.plotting import plot3d
```

```python
plot(sin(x), (x, -pi, pi));
```

```python
monkey_saddle = x**3 - 3*x*y**2
p = plot3d(monkey_saddle, (x, -2, 2), (y, -2, 2))
```

### Derivadas y ecuaciones diferenciales

Podemos usar la función ``diff`` o el método ``.diff()`` para calcular derivadas.

```python
f = lambda x: x**2
```

```python
diff(f(x), x)
```

```python
f(x).diff(x)
```

```python
g = lambda x: sin(x)
```

```python
diff(g(f(x)), x)
```

Y sí, ¡SymPy sabe sobre la regla de la cadena!

Para terminar, resolvamos una ecuación diferencial de segundo orden

$$ u''(t) + \omega^2 u(t) = 0$$

```python
t = symbols("t")
u = symbols("u", cls=Function)
omega = symbols("omega", positive=True)
```

```python
ode = u(t).diff(t, 2) + omega**2 * u(t)
dsolve(ode, u(t))
```

## Convertir expresiones de SymPy en funciones de NumPy

``lambdify`` permite convertir expresiones de sympy en funciones para hacer cálculos usando NumPy.

Veamos cómo.

```python
f = lambdify(x, x**2, "numpy")
f(3)
```

```python
f(np.array([1, 2, 3]))
```

Intentemos un ejemplo más complejo

```python
fun = diff(sin(x)*cos(x**3) - sin(x)/x, x)
fun
```

```python
fun_numpy = lambdify(x, fun, "numpy")
```

y evalúemoslo en algún intervalo, por ejemplo, $[0, 5]$.

```python
pts = np.linspace(0, 5, 1000)
fun_pts = fun_numpy(pts + 1e-6) # Para evitar división por 0
```

```python
plt.figure()
plt.plot(pts, fun_pts)
```

## Ejercicios

1. Calcule el límite

  $$ \lim_{x \rightarrow 0} \frac{\sin(x)}{x}\, .$$

2. Resuelva la ecuación diferencial de Bernoulli

  $$x \frac{\mathrm{d} u(x)}{\mathrm{d}x}  + u(x) - u(x)^2 = 0\, .$$

## Recursos adicionales

- Equipo de desarrollo de SymPy. [SymPy Tutorial](http://docs.sympy.org/latest/tutorial/index.html), (2018). Consultado: Julio 23, 2018
- Ivan Savov. [Taming math and physics using SymPy](https://minireference.com/static/tutorials/sympy_tutorial.pdf), (2017). Consultado: Julio 23, 2018

```python

```


# Gibbs sampling in 2D

This is BONUS content related to Day 22, where we introduce Gibbs sampling

## Random variables

(We'll use 0-indexing so we have close alignment between math and python code)

* 2D random variable $z = [z_0, z_1]$
* each entry $z_d$ is a real scalar: $z_d \in \mathbb{R}$

## Target distribution

\begin{align}
p^*(z_0, z_1) =  \mathcal{N}\left(
    \left[ \begin{array}{c}
    0 \\ 0
    \end{array} \right],
    \left[
    \begin{array}{c c}
    1 & 0.8 \\
    0.8 & 2
    \end{array} \right] \right)
\end{align}

## Key takeaways

* New concept: 'Gibbs sampling', which just iterates between two conditional sampling distributions:

\begin{align}
    z^{t+1}_0 &\sim p^* (z_0 | z_1 = z^t_1) \\
    z^{t+1}_1 &\sim p^* (z_1 | z_0 = z^{t+1}_0)
\end{align}

## Things to remember

This is a simple example to illustrate the idea of how Gibbs sampling works.

There are other "better" ways of sampling from a 2d normal.

# Setup

```python
import numpy as np

```

```python
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("notebook", font_scale=2.0)
```

# Step 1: Prepare for Gibbs sampling

## Define functions to sample from target's conditionals

```python
def draw_z0_given_z1(z1, random_state):
    ## First, use Bishop textbook formulas to compute the conditional mean/var
    mean_01 = 0.4 * z1
    var_01 = 0.68

    ## Then, use simple transform to obtain a sample from this conditional
    ## Remember, if u ~ Normal(0, 1), a "standard" normal with mean 0 variance 1,
    ## then using transform: x <- T(u), with T(u) = \mu + \sigma * u
    ## we can say x ~ Normal(\mu, \sigma^2)
    u_samp = random_state.randn()
    z0_samp = mean_01 + np.sqrt(var_01) * u_samp
    return z0_samp
```

```python
def draw_z1_given_z0(z0, random_state):
    ## First, use Bishop textbook formulas to compute conditional mean/var
    mean_10 = 0.8 * z0
    var_10 = 1.36

    ## Then, use simple transform to obtain a sample from this conditional
    ## Remember, if u ~ Normal(0, 1), a "standard" normal with mean 0 variance 1,
    ## then using transform: x <- T(u), with T(u) = \mu + \sigma * u
    ## we can say x ~ Normal(\mu, \sigma^2)
    u_samp = random_state.randn()
    z1_samp = mean_10 + np.sqrt(var_10) * u_samp
    return z1_samp
```

# Step 2: Execute the Gibbs sampling algorithm

Perform 6000 iterations.

Discard the first 1000 as "not yet burned in".

```python
S = 6000
sample_list = list()
z_D = np.zeros(2)

random_state = np.random.RandomState(0) # reproducible random seeds

for t in range(S):
    z_D[0] = draw_z0_given_z1(z_D[1], random_state)
    z_D[1] = draw_z1_given_z0(z_D[0], random_state)

    if t > 1000:
        sample_list.append(z_D.copy()) # save copies so we get different vectors
```

```python
z_samples_SD = np.vstack(sample_list)
```

## Step 3: Compare to samples from built-in routines for 2D MVNormal sampling

```python
Cov_22 = np.asarray([[1.0, 0.8], [0.8, 2.0]])
true_samples_SD = random_state.multivariate_normal(np.zeros(2), Cov_22, size=S-1000)
```

```python
fig, ax_grid = plt.subplots(nrows=1, ncols=2, sharex=True, sharey=True, figsize=(10,4))

ax_grid[0].plot(z_samples_SD[:,0], z_samples_SD[:,1], 'k.')
ax_grid[0].set_title('Gibbs sampler')
ax_grid[0].set_aspect('equal', 'box');

ax_grid[1].plot(true_samples_SD[:,0], true_samples_SD[:,1], 'k.')
ax_grid[1].set_title('np.random.multivariate_normal')
ax_grid[1].set_aspect('equal', 'box');
ax_grid[1].set_xlim([-6, 6]);
ax_grid[1].set_ylim([-6, 6]);

```

```python

```


```
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats as st
```

## Univariate normal distribution

The normal distribution, also known as the **Gaussian distribution**, is so called because its based on the Gaussian function . This distribution is defined by two parameters: the **mean** $\mu$, which is the **expected value of the distribution**, and the **standard deviation** $\sigma$, which corresponds to the **expected deviation from the mean**. The square of the standard deviation is typically referred to as the **variance** $\sigma^2$. We denote this distribution as:

\begin{equation}
$\mathcal{N}=(\mu,\sigma)$
\end{equation}

Given this mean and variance we can calculate the **probility densitiy function** (**pdf**) of the normal distribution with the normalised Gaussian function. For a value $x$ the density is:

\begin{equation}
p(x \mid \mu, \sigma) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp{ \left( -\frac{(x - \mu)^2}{2\sigma^2}\right)}
\end{equation}

We call this distribution the **univariate** normal because it consists of **only one random normal variable**. Three examples of univariate normal distributions with different mean and variance are plotted in the next figure:

```
def univariate_normal(x, mean, std):
    """pdf of the univariate normal distribution."""
    return ((1. / np.sqrt(2 * np.pi * std**2)) * np.exp(-(x - mean)**2 / (2 * std**2)))
```

```
# Plot different Univariate Normals
x = np.linspace(-5, 5, num=150)
fig = plt.figure(figsize=(10, 5))
plt.plot(x, univariate_normal(x, mean=0, std=1),label="$\mathcal{N}(0, 1)$")
plt.xlabel('$x$', fontsize=13)
plt.ylabel('density: $p(x)$', fontsize=13)
plt.title('Univariate normal distributions')
plt.ylim([0, 1])
plt.xlim([-5, 5])
plt.legend(loc=1)

plt.show()
```

```
# Plot different Univariate Normals
x = np.linspace(-5, 5, num=150)
fig = plt.figure(figsize=(10, 7))
plt.plot(x, univariate_normal(x, mean=0, std=2),label="$\mathcal{N}(0, 2)$")
plt.plot(x, univariate_normal(x, mean=0, std=0.5),label="$\mathcal{N}(0, 0.5)$")
plt.plot(x, univariate_normal(x, mean=2, std=2),label="$\mathcal{N}(2, 2)$")
plt.plot(x, univariate_normal(x, mean=2, std=0.4),label="$\mathcal{N}(2, 0.4)$")
plt.xlabel('$x$', fontsize=13)
plt.ylabel('density: $p(x)$', fontsize=13)
plt.title('Univariate normal distributions')
plt.ylim([0, 1])
plt.xlim([-5, 5])
plt.legend(loc=1)

plt.show()
```

```
# Plot different Univariate Normals
x = np.linspace(-5, 5, num=150)
fig = plt.figure(figsize=(10, 7))
plt.plot(x, st.norm.pdf(x, loc=0, scale=1),label="$\mathcal{N}(0, 1)$")
plt.plot(x, st.norm.pdf(x, loc=2, scale=2),label="$\mathcal{N}(2, 3)$")
plt.plot(x, st.norm.pdf(x, loc=0, scale=0.5),label="$\mathcal{N}(0, 0.2)$")
plt.xlabel('$x$', fontsize=13)
plt.ylabel('density: $p(x)$', fontsize=13)
plt.title('Univariate normal distributions')
plt.ylim([0, 1])
plt.xlim([-5, 5])
plt.legend(loc=1)
plt.show()

```


#### Jupyter notebooks

This is a [Jupyter](http://jupyter.org/) notebook using Python.  You can install Jupyter locally to edit and interact with this notebook.

# Higher order finite difference methods

## Lagrange Interpolating Polynomials

Suppose we are given function values $u_0, \dotsc, u_n$ at the distinct points $x_0, \dotsc, x_n$ and we would like to build a polynomial of degree $n$ that goes through all these points.  This explicit construction is attributed to Lagrange (though he was not first):

$$ p(x) = \sum_{i=0}^n u_i \prod_{j \ne i} \frac{x - x_j}{x_i - x_j} $$

* What is the degree of this polynomial?
* Why is $p(x_i) = u_i$?
* How expensive (in terms of $n$) is it to evaluate $p(x)$?
* How expensive (in terms of $n$) is it to convert to standard form $p(x) = \sum_{i=0}^n a_i x^i$?
* Can we easily evaluate the derivative $p'(x)$?
* What can go wrong?  Is this formulation numerically stable?

A general derivation of finite difference methods for approximating $p^{(k)}(x)$ using function values $u(x_i)$ is to construct the Lagrange interpolating polynomial $p(x)$ from the function values $u_i = u(x_i)$ and evaluate it or its derivatives at the target point $x$.  We can do this directly from the formula above, but a more linear algebraic approach will turn out to be more reusable.

#### Uniqueness

Is the polynomial $p(x)$ of degree $m$ that interpolates $m+1$ points unique?  Why?

### Vandermonde matrices

We can compute a polynomial

$$ p(x) = c_0 + c_1 x + c_2 x^2 + \dotsb $$

that assumes function values $p(x_i) = u_i$ by solving a linear system with the Vandermonde matrix.

$$ \underbrace{\begin{bmatrix} 1 & x_0 & x_0^2 & \dotsb \\
    1 & x_1 & x_1^2 & \dotsb \\
    1 & x_2 & x_2^2 & \dotsb \\
    \vdots & & & \ddots \end{bmatrix}}_V \begin{bmatrix} c_0 \\ c_1 \\ c_2 \\ \vdots \end{bmatrix} = \begin{bmatrix} u_0 \\ u_1 \\ u_2 \\ \vdots \end{bmatrix} .$$

```python
%matplotlib inline
import numpy
from matplotlib import pyplot
pyplot.style.use('ggplot')

x = numpy.linspace(-2,2,4)
u = numpy.sin(x)
xx = numpy.linspace(-3,3,40)
c = numpy.linalg.solve(numpy.vander(x), u)
pyplot.plot(x, u, '*')
pyplot.plot(xx, numpy.vander(xx, 4).dot(c), label='p(x)')
pyplot.plot(xx, numpy.sin(xx), label='sin(x)')
pyplot.legend(loc='upper left');
```

Given the coefficients $c = V^{-1} u$, we find

$$ \begin{align} p(0) &= c_0 \\ p'(0) &= c_1 \\ p''(0) &= c_2 \cdot 2! \\ p^{(k)}(0) &= c_k \cdot k! . \end{align} $$

To compute the stencil coefficients $s_i^0$ for interpolation to $x=0$,
$$ p(0) = s_0^0 u_0 + s_1^0 u_1 + \dotsb = \sum_i s_i^0 u_i $$
we can write
$$ p(0) = e_0^T \underbrace{V^{-1} u}_c = \underbrace{e_0^T V^{-1}}_{(s^0)^T} u $$
where $e_0$ is the first column of the identity.  Evidently $s^0$ can also be expressed as
$$ s^0 = V^{-T} e_0 . $$
We can compute stencil coefficients for any order derivative $p^{(k)}(0) = (s^k)^T u$ by solving the linear system
$$ s^k = V^{-T} e_k \cdot k! . $$
Alternatively, invert the Vandermonde matrix $V$ and scale row $k$ of $V^{-1}$ by $k!$.

```python
def fdstencil(z, x):
    x = numpy.array(x)
    V = numpy.vander(x - z, increasing=True)
    scaling = numpy.array([numpy.math.factorial(i) for i in range(len(x))])
    return (numpy.linalg.inv(V).T * scaling).T

x = numpy.linspace(0,3,4)
S = fdstencil(0, x)
print(S)

hs = 2.**(-numpy.arange(6))
errors = numpy.zeros((3,len(hs)))
for i,h in enumerate(hs):
    z = 1 + .3*h
    S = fdstencil(z, 1+x*h)
    u = numpy.sin(1+x*h)
    errors[:,i] = S[:3].dot(u) - numpy.array([numpy.sin(z), numpy.cos(z), -numpy.sin(z)])

pyplot.loglog(hs, numpy.abs(errors[0]), 'o', label="$p(0)$")
pyplot.loglog(hs, numpy.abs(errors[1]), '<', label="$p'(0)$")
pyplot.loglog(hs, numpy.abs(errors[2]), 's', label="$p''(0)$")
for k in (1,2,3):
    pyplot.loglog(hs, hs**k, label='$h^{%d}$' % k)
pyplot.legend(loc='upper left');
```

### Notes on accuracy

* When using three points, we fit a polynomial of degree 2.  The leading error term for interpolation $p(0)$ is thus $O(h^3)$.
* Each derivative gives up one order of accuracy, therefore differencing to a general (non-centered or non-uniform grid) point is $O(h^2)$ for the first derivative and $O(h)$ for the second derivative.
* Centered differences on uniform grids can provide cancelation, raising the order of accuracy by one.  So our standard 3-point centered second derivative is $O(h^2)$ as we have seen in the Taylor analysis and numerically.
* The Vandermonde matrix is notoriously ill-conditioned when using many points.  For such cases, we recommend using a more numerically stable method from [Fornberg](https://doi.org/10.1137/S0036144596322507).

```python
-fdstencil(0, numpy.linspace(-1,4,6))[2]
```

    array([-0.83333333,  1.25      ,  0.33333333, -1.16666667,  0.5       ,
           -0.08333333])

### Solving BVPs

This `fdstencil` gives us a way to compute derivatives of arbitrary accuracy on arbitrary grids.  We will need to use uncentered rules near boundaries, usually with more points to maintain order of accuracy.  This will usually cost us symmetry.  Implementation of boundary conditions is the bane of high order finite difference methods.

### Discretization stability measures: $h$-ellipticity

Consider the test function $\phi(\theta, x) = e^{i\theta x}$ and apply the difference stencil centered at an arbitrary point $x$ with element size $h=1$:

$$ \begin{bmatrix} -1 & 2 & -1 \end{bmatrix} \begin{bmatrix} e^{i \theta (x - 1)} \\ e^{i \theta x} \\ e^{i \theta (x+1)} \end{bmatrix}
= \big( 2 - (e^{i\theta} + e^{-i\theta}) \big) e^{i\theta x}= 2 (1 - \cos \theta) e^{i \theta x} . $$

Evidently $\phi(\theta,x) = e^{i \theta x}$ is an eigenfunction of the discrete differencing operator on an infinite grid and the corresponding eigenvalue is
$$ L(\theta) = 2 (1 - \cos \theta), $$
also known as the "symbol" of the operator.  That $\phi(\theta,x)$ is an eigenfunction of the discrete differencing formula will generally be true for uniform grids.

The highest frequency that is distinguishable using this stencil is $\theta_{\max} = \pi$ which results in a wave at the Nyquist frequency.  If a higher frequency wave is sampled onto this grid, it will be aliased back into the interval $[-\pi, \pi)$.

```python
x = numpy.linspace(-1, 1, 3)
s2 = -fdstencil(0, x)[2]
print(s2)
theta = numpy.linspace(-numpy.pi, numpy.pi)
phi = numpy.exp(1j*numpy.outer(x, theta))
pyplot.plot(theta, numpy.abs(s2.dot(phi)), '.')
pyplot.plot(theta, 2*(1-numpy.cos(theta)))
pyplot.plot(theta, theta**2);
```

A measure of internal stability known as $h$-ellipticity is defined by

$$ E^h(L) = \frac{\min_{\pi/2 \le |\theta| \le \pi} L(\theta)}{\max_{|\theta| \le \pi} L(\theta)} . $$

* What is $E^h(L)$ for the second order "version 2" stencil?
* How about for uncentered formulas and higher order?

# Spectral collocation

Suppose that instead of using only a fixed number of neighbors in our differencing stencil, we use all points in the domain?

```python
n = 10
x = numpy.linspace(-1, 1, n)
L = numpy.zeros((n,n))
for i in range(n):
    L[i] = -fdstencil(x[i], x)[2]

u = numpy.cos(3*x)
pyplot.plot(x, L.dot(u), 'o')
pyplot.plot(x, 9*u);
```

We are suffering from two problems here.  The first is that the monomial basis is very ill-conditioned when using many terms.  This is true as continuous functions, not just when sampled onto a particular grid.

```python
x = numpy.linspace(-1, 1, 50)
V = numpy.vander(x, 15)
pyplot.plot(x, V)
numpy.linalg.cond(V)
```

## Chebyshev polynomials

Define $$ T_n(x) = \cos (n \arccos(x)) .$$
This turns out to be a polynomial, though it may not be obvious why.
Recall $$ \cos(a + b) = \cos a \cos b - \sin a \sin b .$$
Let $y = \arccos x$ and check
$$ \begin{split}
    T_{n+1}(x) &= \cos (n+1) y = \cos ny \cos y - \sin ny \sin y \\
    T_{n-1}(x) &= \cos (n-1) y = \cos ny \cos y + \sin ny \sin y
\end{split}$$
Adding these together produces a similar recurrence:
$$\begin{split}
T_0(x) &= 1 \\
T_1(x) &= x \\
T_{n+1}(x) &= 2 x T_n(x) - T_{n-1}(x)
\end{split}$$
which we can also implement in code

```python
def vander_chebyshev(x, n=None):
    if n is None:
        n = len(x)
    T = numpy.ones((len(x), n))
    if n > 1:
        T[:,1] = x
    for k in range(2,n):
        T[:,k] = 2 * x * T[:,k-1] - T[:,k-2]
    return T

x = numpy.linspace(-1, 1)
V = vander_chebyshev(x, 5)
pyplot.plot(x, V)
numpy.linalg.cond(V)
```

```python
# We can use the Chebyshev basis for interpolation
x = numpy.linspace(-2, 2, 4)
u = numpy.sin(x)
c = numpy.linalg.solve(vander_chebyshev(x), u)
pyplot.plot(x, u, '*')
pyplot.plot(xx, vander_chebyshev(xx, 4).dot(c), label='p(x)')
pyplot.plot(xx, numpy.sin(xx), label='sin(x)')
pyplot.legend(loc='upper left');
```

### Differentiation

We can differentiate Chebyshev polynomials using the recurrence

$$ \frac{T_n'(x)}{n} = 2 T_{n-1}(x) + \frac{T_{n-2}'(x)}{n-2} $$

which we can differentiate to evaluate higher derivatives.

```python
def chebeval(z, n=None):
    """Build matrices to evaluate the n-term Chebyshev expansion and its derivatives at point(s) z"""
    z = numpy.array(z, ndmin=1)
    if n is None:
        n = len(z)
    Tz = vander_chebyshev(z, n)
    dTz = numpy.zeros_like(Tz)
    dTz[:,1] = 1
    dTz[:,2] = 4*z
    ddTz = numpy.zeros_like(Tz)
    ddTz[:,2] = 4
    for n in range(3,n):
        dTz[:,n]  = n * (2*Tz[:,n-1] + dTz[:,n-2]/(n-2))
        ddTz[:,n] = n * (2*dTz[:,n-1] + ddTz[:,n-2]/(n-2))
    return [Tz, dTz, ddTz]

n = 44
x = numpy.linspace(-1, 1, n)
T = vander_chebyshev(x)
print('cond = {:e}'.format(numpy.linalg.cond(T)))
Tinv = numpy.linalg.inv(T)
L = numpy.zeros((n,n))
for i in range(n):
    L[i] = chebeval(x[i], n)[2].dot(Tinv)

u = numpy.cos(3*x)
pyplot.plot(x, L.dot(u), 'o')
xx = numpy.linspace(-1, 1, 100)
pyplot.plot(xx, -9*numpy.cos(3*xx));
```

### Runge Effect

Polynomial interpolation on equally spaced points is very ill-conditioned as the number of points grows.  We've seen that in the growth of the condition number of the Vandermonde matrix, both for monomials and Chebyshev polynomials, but it's also true if the polynomials are measured in a different norm, such as pointwise values or merely the eyeball norm.

```python
def chebyshev_interp_and_eval(x, xx):
    """Matrix mapping from values at points x to values
    of Chebyshev interpolating polynomial at points xx"""
    A = vander_chebyshev(x)
    B = vander_chebyshev(xx, len(x))
    return B.dot(numpy.linalg.inv(A))

def runge1(x):
    return 1 / (1 + 10*x**2)
x = numpy.linspace(-1,1,20)
xx = numpy.linspace(-1,1,100)
pyplot.plot(x, runge1(x), 'o')
pyplot.plot(xx, chebyshev_interp_and_eval(x, xx).dot(runge1(x)));
```

```python
x = cosspace(-1,1,8)
pyplot.plot(xx, chebyshev_interp_and_eval(x,xx))
```

```python
numpy.outer(numpy.arange(4), [10,20])
```

```python
ns = numpy.arange(5,20)
conds = [numpy.linalg.cond(chebyshev_interp_and_eval(numpy.linspace(-1,1,n),
                                                    numpy.linspace(-1,1,100)))
        for n in ns]
pyplot.semilogy(ns, conds);
```

This ill-conditioning cannot be fixed when using polynomial *interpolation* on equally spaced grids.

### Chebyshev nodes

The Chebyshev polynomials assume their maximum value of 1 at points where their derivatives are zero (plus the endpoints).  Choosing the roots of $T_n'(x)$ (plus endpoints) will control the polynomials and should lead to a well-conditioned formulation.

```python
def cosspace(a, b, n=50):
    return (a + b)/2 + (b - a)/2 * (numpy.cos(numpy.linspace(-numpy.pi, 0, n)))

conds = [numpy.linalg.cond(chebyshev_interp_and_eval(cosspace(-1,1,n),
                                                    numpy.linspace(-1,1,100)))
        for n in ns]
pyplot.figure()
pyplot.plot(ns, conds);
```

```python
x = cosspace(-1, 1, 7)
pyplot.plot(x, 0*x, 'o')
pyplot.plot(xx, chebeval(xx, 7)[1]);
```

```python
x = cosspace(-1,1,20)
xx = numpy.linspace(-1,1,100)
pyplot.figure()
pyplot.plot(x, runge1(x), 'o')
pyplot.plot(xx, chebyshev_interp_and_eval(x, xx).dot(runge1(x)));
```

## Chebyshev solution of Boundary Value Problems

If instead of an equally (or arbitrarily) spaced grid, we choose the Chebyshev nodes and compute derivatives in a stable way (e.g., via interpolating into the Chebyshev basis), we should have a very accurate method.  Let's return to our test equation

$$ -u''(x) = f(x) $$

subject to some combination of Neumann and Dirichlet boundary conditions.

```python
def laplacian_cheb(n, rhsfunc, left, right):
    """Solve the Laplacian boundary value problem on (-1,1) using n elements with rhsfunc(x) forcing.
    The left and right boundary conditions are specified as a pair (deriv, func) where
      * deriv=0 for Dirichlet u(x_endpoint) = func(x_endpoint)
      * deriv=1 for Neumann u'(x_endpoint) = func(x_endpoint)"""
    x = cosspace(-1, 1, n+1)  # n+1 points is n "elements"
    T = chebeval(x)
    L = -T[2]
    rhs = rhsfunc(x)
    for i,deriv,func in [(0, *left), (-1, *right)]:
        L[i] = T[deriv][i]
        rhs[i] = func(x[i])
    return x, L.dot(numpy.linalg.inv(T[0])), rhs

class exact_tanh:
    def __init__(self, k=1, x0=0):
        self.k = k
        self.x0 = x0
    def u(self, x):
        return numpy.tanh(self.k*(x - self.x0))
    def du(self, x):
        return self.k * numpy.cosh(self.k*(x - self.x0))**(-2)
    def ddu(self, x):
        return -2 * self.k**2 * numpy.tanh(self.k*(x - self.x0)) * numpy.cosh(self.k*(x - self.x0))**(-2)

ex = exact_tanh(5, 0.3)
x, L, rhs = laplacian_cheb(50, lambda x: -ex.ddu(x),
                           left=(0,ex.u), right=(0,ex.u))
uu = numpy.linalg.solve(L, rhs)
pyplot.plot(x, uu, 'o')
pyplot.plot(xx, ex.u(xx))
pyplot.plot(xx, chebeval(xx)[0][:,:51].dot(numpy.linalg.solve(chebeval(x)[0], uu)))
print(numpy.linalg.norm(numpy.linalg.solve(L, rhs) - ex.u(x), numpy.inf))
```

```python
def mms_error(n, discretize, sol):
    x, L, f = discretize(n, lambda x: -sol.ddu(x), left=(0,sol.u), right=(1,sol.du))
    u = numpy.linalg.solve(L, f)
    return numpy.linalg.norm(u - sol.u(x), numpy.inf)

ns = numpy.arange(10,60,2)
errors = [mms_error(n, laplacian_cheb, ex) for n in ns]
pyplot.figure()
pyplot.semilogy(ns, errors, 'o', label='numerical')
for p in range(1,5):
    pyplot.semilogy(ns, 1/ns**(p), label='$n^{-%d}$'%p)
pyplot.xlabel('n')
pyplot.ylabel('error')

pyplot.legend(loc='lower left');
```

# Homework 1: Due 2017-09-25

Use a Chebyshev method to solve the second order ordinary differential equation

$$ u''(t) + a u'(t) + b u(t) = f(t) $$

from $t=0$ to $t=1$ with initial conditions $u(0) = 1$ and $u'(0) = 0$.

1. Do a grid convergence study to test the accuracy of your method.
* Setting $f(t)=0$, experiment with the values $a$ and $b$ to identify two regimes with qualitatively different dynamics.


# Introduction to the exponential distribution

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
```

In queuing systems it is typically assumed that

1. customers arrive at random
2. service time varies randomly.

The notion of *random* is closely associated with the **exponential distribution** (note this sometimes called the *negative* exponential distribution).  Its density function is given by (1).

\begin{equation}
f(t) = \lambda e^{-\lambda t}
\tag{1}
\end{equation}

where
* $1/\lambda$ is the mean time between events.
* $\lambda$ is the rate at which events occur (the expected no. of events per time unit)

To put it another way $f(t)$ represents the probability of an event occuring in the next t time units.

## Illustration: $\lambda = 1.0$

The chart below plot $f(t)$ for $\lambda = 1.0$.  It can be observed that the most likely times are the small values close to zero.  The longer the time the less likely it is to occur.  In other words, exponentially distributed times are most likely to be small and below the mean, but there still will be the occational long time.

```python
LAMBDA = 1.0
```

```python
#create 100 equally spaced time points between 0 and 5
t = np.linspace(0, 5, 100)
```

```python
#calculate f_t
f_t = LAMBDA * np.power(np.e, -LAMBDA * t)
```

```python
#plot
ax = pd.DataFrame(f_t, index=t).plot(figsize=(9, 6))
ax.legend(['$\lambda = 1.0$']);
ax.figure.savefig('exponential_1.png', bbox_inches='tight', dpi=300)
```

# Varying $\lambda$ changes the shape of the distribution.

The area under each density curve must equal 1.0.  This has consequences for the shape of the distribution as $\lambda$ varies in size.
The code below plots $f(t)$ while varying the parameter $\lambda = 1.0, 2.0, 3.0$.  We observe the following behaviour:

* A larger value of $\lambda$ leads to a more rapid decrease an asymptotic decrease to zero.

```python
fig, ax = plt.subplots(1, 1, figsize=(9, 6))

labels = []
for _lambda in range(1, 4):
    f_t = _lambda * np.power(np.e, -_lambda * t)
    ax.plot(pd.DataFrame(f_t, index=t))
    labels.append(f'$\lambda = {float(_lambda)}$')

ax.set_xlabel('t')
ax.set_ylabel('f(t)')
ax.legend(labels);
fig.savefig('exponential.png', bbox_inches='tight', dpi=300)
```

```python

```


# Series Expansions: Fourier

Series expansions are used in a variety of circumstances:
- When we need a tractable approximation to some ugly equation
- To transform between equivalent ways of looking at a problem (e.g. time domain vs frequency domain)
- When they are (part of) a solution to a particular class of differential equation

For approximations, there is an important divide between getting the best fit *near a point* (e.g. Taylor series) and getting the best fit *over an interval* (e.g. Fourier series). This notebook deals with the latter; there is a separate notebook for Taylor expansions, and others for Bessel, Legendre, etc.

## Fitting over an interval

What is the best (tractable) series approximating my function across some range of values? What matters is an overall best fit (e.g. least-squares deviation) across the range, and we can't tolerate wild divergences as with the Taylor series.

There are various series which are useful in different contexts, but a common property is that the terms are *orthogonal* over some interval $[-L,L]$. If $f(t)$ is a real-valued function their *inner product* is defined as

$$ \langle f(m t),f(n t) \rangle   \colon =\int _{-L}^L f(m t) f(n t) \,  dt $$

For orthogonal functions, this is non-zero if $m=n$ and zero if $m \ne n$. If the inner product is $\delta_{mn}$ (the Kronecker delta), the fuctions are said to be orthonormal.

# Differential Equations

The ODE $y'' + n^2 y = 0$ is solved by all the period functions $\sin(n x)$, $\cos(n x)$ and $e^{\pm i n x}$. There is thus a close analogy to functions such as Bessel and Legendre, though sine and cosine have become much more familar to most of us for other (geometric) reasons. They have the nice property of evenly-spaced zeros, unlike Bessel functions. For example, $\sin(x)=0$ for $x = n \pi$ where n is any integer.

The use of sin/cos or complex exponentials is also exceptionally familiar in series expansions, mainly because they are so useful in engineering and communications.

## Fourier Series and Fourier Analysis

A periodic function $f$ of period $2L$ can be approximated by a Fourier Series of sines and cosines:

$$ f(t) = \frac{a_0}{2} + \sum _{n \ge 1} a_ n \cos \frac{n \pi t}{L} + \sum _{n \ge 1} b_ n \sin \frac{n \pi t}{L}  $$

To find the coefficients:
$$
\begin{align*}
	\frac{a_0}{2} &= \displaystyle \frac{1}{2L} \int _{-L}^{L} f(t) \,  dt = \frac{\langle f(t), 1 \rangle }{\langle 1, 1\rangle }\\[6pt]
	a_ n& = \frac{1}{L} \int _{-L}^{L} f(t) \cos \frac{n \pi t}{L} \,  dt = \frac{\langle f(t),\cos \left(\frac{n \pi }{L} t\right)\rangle }{\langle \cos \left(\frac{n \pi }{L} t\right), \cos \left(\frac{n \pi }{L} t\right)\rangle } \\[10pt]
	b_ n &= \displaystyle \frac{1}{L} \int _{-L}^{L} f(t) \sin \frac{n \pi t}{L} \,  dt = \frac{\langle f(t),\sin \left(\frac{n \pi }{L} t\right)\rangle }{\langle \sin \left(\frac{n \pi }{L} t\right), \sin \left(\frac{n \pi }{L} t\right)\rangle }
\end{align*}
$$

Equivalently, we can express the Fourier Series as complex exponentials:

$$ f\left(t\right) = \sum _{n = -\infty }^{\infty } c_{n} e^{i n t}, \qquad c_{n} \colon =\frac{a_{n} - i b_{n}}{2} \quad \text{ and } \quad c_{-n} \colon =\bar{c}_{n} = \frac{a_{n} + i b_{n}}{2} $$

Real-world situations tend not to give infinitely periodic functions, so Fourier Analysis can be thought of as the limit as $L$ goes to infinity of a periodic signal of period $2L$. As $L$ increases, the spacing between the frequencies in our sum are approaching zero. This turns the sum into an integral in the limit, and we have the equations:

$$ f(t)  = \int _{-\infty }^{\infty } \widehat{f}\left(k\right)e^{ i k t} \,  dk  \quad \text{where} \quad \widehat{f} = \frac{1}{2\pi }\int _{-\infty }^{\infty } f\left(t\right)e^{- i k t} \,  dt $$

We call $\widehat{f}$ the **Fourier transform** of $f(t)$.

Start with quite a lot of imports ready for calculation and plotting:

```python
import numpy as np
import matplotlib.pyplot as plt

from ipywidgets import interact, interactive, fixed, interact_manual, Layout
import ipywidgets as w

plt.rcParams.update({'font.size': 16})

from sympy import fourier_series, pi, init_printing, lambdify, integrate
init_printing()
from sympy.functions import sin, cos
from sympy.abc import x

from sympy.parsing.sympy_parser import parse_expr
from sympy.parsing.sympy_parser import standard_transformations, implicit_multiplication_application
transformations = (standard_transformations + (implicit_multiplication_application,))
```

SymPy has a FourierSeries class which looked like it would do what we need, very simply. The results were disappointing: `fourier_series()` returns a very complicated result for even simple functions, and attempts to lambdify this never terminated.

The `fourier_coeff()` function below calculates $a_n$ and $b_n$ by integration and returns them as NumPy arrays to 5 figure accuracy. Note that this is only for illustrating the effect of adding more terms: the calculation is slow and clunky, and nobody would do Fourier transforms this way for real problems.

```python
# Fourier series, to order n
def fourier_coeff(f, L, n):
    # f is a SymPy function of x
    a_n = []
    b_n = [0,]
    a_n.append((1/(2*L)*integrate(f, (x, -L, L))).evalf(5))
    for i in range(1, n+1):
        a_n.append((1/L*integrate(f*cos(i*pi/L*x), (x, -L, L))).evalf(5))
        b_n.append((1/L*integrate(f*sin(i*pi/L*x), (x, -L, L))).evalf(5))

    # SymPy is VERY reluctant to give us simple numbers rather than SymPy objects
    # Force the conversion to NumPy floats with np.array().astype()
    return (np.array(a_n).astype(np.float64), np.array(b_n).astype(np.float64))
```

```python
# Plot results
def plotFourier(f_sympy, L):

    max_terms = 20

    # get a NumPy-style function from the SymPy version
    f_np = lambdify(x, f_sympy, 'numpy')
    display(f_sympy) # display shows LaTex, print wouldn't

    # plot the starting function
    x_lims = [-L,L]
    x1 = np.linspace(x_lims[0], x_lims[1], 100)
    fig = plt.figure(figsize=(9, 20))

    ax1 = fig.add_subplot(311)
    ax1.plot(x1, f_np(x1), 'k.', label=f_sympy)

    # get some terms of a Fourier series
#     f_fourier = fourier_series(f_sympy, (x, -L, L))
#     f_fourier.truncate(4)
#     display(f_fourier)

    a_n, b_n = fourier_coeff(f_sympy, L, max_terms)

    ax2 = fig.add_subplot(312)
    x_int = range(0, len(a_n))
#     ax2.stem(a_n, 'k.', label='a_n')
    ax2.stem(x_int, a_n, markerfmt='C0o', label='a_n')
    ax2.stem(x_int, b_n, markerfmt='C1o', label='b_n')
    ax2.set_xlim(left=-0.5)
    ax2.set_ylabel('coefficients')
#     ax2.xaxis.set_major_locator(MaxNLocator(integer=True)) # fails
    ax2.legend()

    # plot the successive approximations
    for n in [0,1,2,3,5,10,20]:
        if n > max_terms:
            break
        y = np.zeros(len(x1))
        for i in range(n+1):
            cos_term = a_n[i]*np.cos(i*np.pi/L*x1)
            sin_term = b_n[i]*np.sin(i*np.pi/L*x1)
            y += (cos_term + sin_term)
        ax1.plot(x1, y, label='order ' + str(n))

    # graph housekeeping
    ax1.set_xlim(x_lims)
    ax1.set_xlabel('x')
    ax1.set_ylabel('y')
    ax1.legend()
    ax1.grid(True)
    plt.title('Fourier series approximation of ' + str(f_sympy))
```

```python
def parse_input(f_txt, L):
    f_sympy = parse_expr(f_txt, transformations=transformations)
    plotFourier(f_sympy, L)
```

Please wait patiently for each new calculation, which will take several seconds at best.

```python
style = {'description_width': 'initial'} # to avoid the labels getting truncated
interact(parse_input,
             f_txt = w.Text(description='f(x):',
                                            layout=Layout(width='80%'),
                                            continuous_update=False,
                                            value='x**2 - x**3'),
             L = w.FloatSlider(description="Limits $\pm L$", style=style,
                                            layout=Layout(width='80%'),
                                            continuous_update=False,
                                            min=1, max=10,
                                            value=np.pi),
);
```

    interactive(children=(Text(value='x**2 - x**3', continuous_update=False, description='f(x):', layout=Layout(wi…

```python

```

### Discrete Fourier transforms

The mathematics of Fourier analysis goes back to the early 19th century, but its use has exploded in the last few decades. A couple of factors collided to drive this:

- An efficient Fast Fourier Transform (FFT) algorithm, developed in the 1960s and implemented in both software and specialist hardware
- The spread of digital technology, for audio, video and many other sorts of discretized signals. These are all perfect inputs for FFT.

FFT gets away from complicated integrals and replaces them with a series of simple multiplications and additions. This gives a computation time of $\mathcal{O}(N \log N)$ for a signal with $N$ data points. Fast, as the name suggests! And your cellphone is doing millions of these calculations whenever you use it (for anything at all).

There are many FFT functions in the `numpy.fft` module, but for real input we will use the `rfft()` function and its inverse, `irfft()`. This avoids calculating and storing half the coefficients, which for real input are just complex conjugates of the other half.

Zeroing all but the first few coefficients before the inverse FFT simulates the effect of using few terms in the Fourier series.

```python
def plotFFT(f_sympy, L):

    n_pts = 100

    # get a NumPy-style function from the SymPy version
    f_np = lambdify(x, f_sympy, 'numpy')

    # discretize the function (for fft, not just for plotting)
    x_lims = [-L,L]
    x1 = np.linspace(x_lims[0], x_lims[1], n_pts)
    y = f_np(x1)

    # plot the starting function
    fig = plt.figure(figsize=(9, 20))
    ax1 = fig.add_subplot(311)
    ax1.plot(x1, f_np(x1), 'k.', label=f_sympy)

    display(f_sympy) # display shows LaTex, print wouldn't

    f_fft = np.fft.rfft(y)

    # get a_n and b_n from complex coefficients in f_fft
#     print(f_fft.shape)
#     print(f_fft)
    a_n = np.real(f_fft[:50] + np.conj(f_fft[:50]))
    a_n[0] = a_n[0]/2
    b_n = np.imag(f_fft[:50] - np.conj(f_fft[:50]))

    ax2 = fig.add_subplot(312)
    x_int = range(0, len(a_n))
    ax2.stem(x_int, a_n, markerfmt='C0o', label='real')
    ax2.stem(x_int, b_n, markerfmt='C1o', label='imag')
#     ax2.set_xlim(left=-0.5)
    ax2.set_ylabel('coefficients')
#     ax2.xaxis.set_major_locator(MaxNLocator(integer=True)) # fails
    ax2.legend()

#     # plot the successive approximations - FAILS
    for i in [0,1,2,3,10,20]:
        fft_i = np.zeros(len(f_fft), dtype=np.cfloat)
        np.put(fft_i, range(i+1), f_fft[:i+1])
        y_i = np.real(np.fft.irfft(fft_i))
        ax1.plot(x1, y_i, label='order ' + str(i))

    y_i = np.real(np.fft.irfft(f_fft))
    ax1.plot(x1, y_i, label='full inverse FFT')

    # graph housekeeping
    ax1.set_xlim(x_lims)
#     plt.ylim([-3,3])
    ax1.set_xlabel('x')
    ax1.set_ylabel('y')
    ax1.legend()
    ax1.grid(True)
    plt.title('FFT series approximation of ' + str(f_sympy))
```

```python
def parse_input_fft(f_txt, L):
    # we could just run eval(f_txt),
    # but parse_expr() followed by lambdify() is probably safer
    f_sympy = parse_expr(f_txt, transformations=transformations)
    plotFFT(f_sympy, L)
```

```python
style = {'description_width': 'initial'} # to avoid the labels getting truncated
interact(parse_input_fft,
             f_txt = w.Text(description='f(x):',
                                            layout=Layout(width='80%'),
                                            continuous_update=False,
                                            value='x**2 - x**3'),
             L = w.FloatSlider(description="Limits $\pm L$", style=style,
                                            layout=Layout(width='80%'),
                                            continuous_update=False,
                                            min=1, max=10,
                                            value=np.pi),
        );
```

    interactive(children=(Text(value='x**2 - x**3', continuous_update=False, description='f(x):', layout=Layout(wi…

```python

```

```python
def plotFFT_discontinuous(waveform='square', L=np.pi):

    n_pts = 100

    # discretize the function (for fft, not just for plotting)
    x_lims = [-L,L]
    x1 = np.linspace(x_lims[0], x_lims[1], n_pts)

    if waveform == 'square':
        y = np.sign(x1)
    elif waveform=='triangle':
        y = np.abs(x1)

    # plot the starting function
    fig = plt.figure(figsize=(9, 20))
    ax1 = fig.add_subplot(311)
    ax1.plot(x1, y, 'r.', label=waveform)

    f_fft = np.fft.rfft(y)

    # get a_n and b_n from complex coefficients in f_fft
#     print(f_fft.shape)
#     print(f_fft)
    a_n = np.real(f_fft[:50] + np.conj(f_fft[:50]))
    a_n[0] = a_n[0]/2
    b_n = np.imag(f_fft[:50] - np.conj(f_fft[:50]))

    ax2 = fig.add_subplot(312)
    x_int = range(0, len(a_n))
    ax2.stem(x_int, a_n, markerfmt='C0o', label='a_n')
    ax2.stem(x_int, b_n, markerfmt='C1o', label='b_n')
#     ax2.set_xlim(left=-0.5)
    ax2.set_ylabel('coefficients')
#     ax2.xaxis.set_major_locator(MaxNLocator(integer=True)) # fails
    ax2.legend()

    y_i = np.real(np.fft.irfft(f_fft))
    ax1.plot(x1, y_i, label='full inverse FFT')

    # graph housekeeping
    ax1.set_xlim(x_lims)
#     plt.ylim([-3,3])
    ax1.set_xlabel('x')
    ax1.set_ylabel('y')
    ax1.legend()
    ax1.grid(True)
    plt.title('FFT approximation of ' + waveform + ' wave')
```

```python
plotFFT_discontinuous('triangle')
```

```python
plotFFT_discontinuous('square')
```

```python

```

```python

```

```python

```

## References

Boas, "Mathematical methods in the physical sciences"

```python

```


# Code Samples

```python
import sympy as sp
import numpy
import pandas
```

```python
sp.init_printing()
```

```python
x, y, z, k1, k2, k3 = sp.symbols("x, y, z, k1, k2, k3")
```

```python
sp.solveset(sp.sin(x) - 1, x)
```

```python
matrix = sp.Matrix([sp.sin(x) -1, sp.cos(y) -1 ])
matrix
```

$\displaystyle \left[\begin{matrix}\sin{\left(x \right)} - 1\\\cos{\left(y \right)} - 1\end{matrix}\right]$

```python
sp.solve(matrix)
```

```python
kinetics = sp.Matrix([k1*x*y - 3, k2*x/(1 -x) - 4])
```

```python
sp.nonlinsolve(kinetics, [x,y])
```

```python
sp.plot(2*x**2 + 3*x)
```

```python
from sympy.plotting import plot3d_parametric_line

t = sp.symbols('t')
alpha = [sp.cos(t), sp.sin(t), t]
plot3d_parametric_line(*alpha)

```

```python
# Plots for the reaction flux
#  x + y -> z; k1*x*y
flux = sp.Matrix([x, y, k1*x*y])
flux_plot = flux.subs({k1: 3})
plot3d_parametric_line(x, x**2, 3*x**3)
```

```python
f, g = sp.symbols('f g', cls=sp.Function)
diffeq = sp.Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sp.sin(x))
diffeq
```

```python
result = sp.dsolve(diffeq, f(x))
```

```python
syms = list(result.free_symbols)
syms[0]
result1 = result.subs({syms[0]: 1, syms[1]: 1})
sp.plot(result1.rhs)
```

```python
sp.solve(x**2 - 2*x + 1, x)
```

```python
result1.rhs
```

# Workflow for Solving LTI Systems
1. Given $A, B, C$, find
   1. $e^{At}$
   1. $\int_0^t e^{A(t - \tau)} u(\tau) d \tau$ for
   $u(\tau) \in \{ \delta(t), 1(t), t \} $
   1. $x(t)$
   1. $y(t)$

1. Plot $x$, $y$

1. Solve for observability, controllability

# Workflow for Reaction Networks
1. Simulate the original model
1. Convert model to sympy
1. Get symbolic Jaccobian
1. Construct LTI models for different points in state space


# Fisher Discriminant Analysis

First off we import necessary libraries.

```python
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import random

from numpy.linalg import inv, eig
```

We define the number of points in each class, covariance matrix and mean of every class in accordance with the requirements of the exercise.

```python
n = 30
cov = np.array([[1, 0], [0, 1]])
means = {'A': np.array([-1, 1]), 'B': np.array([2, 4]), 'C': np.array([-2, 2])}
```

We generate the data from multivariate normal distribution and insert it into the dataframe

```python
data = pd.DataFrame(index=range(n * len(means)), columns=['x', 'y', 'label'])
for i, (label, mean) in enumerate(means.items()):
    data.loc[i*n:((i+1)*n)-1, ['x', 'y']] = np.random.multivariate_normal(mean, cov, n)
    data.loc[i*n:((i+1)*n)-1, ['label']] = label
```

Next we define a function that plots the points from the dataframe.

```python
def plot_points(data):
    fig = plt.figure(figsize=(8, 6))
    sns.set_style('white')
    sns.set_palette('muted')
    sns.scatterplot(data=data, x='x', y='y', hue='label', legend=False, edgecolor='black').set(xlim=(-5, 5), ylim=(-2, 6))
```

```python
plot_points(data)
```

## Determination of optimal direction

Before we move on to computing of optimal direction, we should update means of the classes and covariance matrix corresponding to the generated points.

```python
means = {label: np.array(np.mean(data.loc[data['label'] == label])) for label in means.keys()}
cov = {label: np.cov(data.loc[data['label'] == label][['x', 'y']].values.T.astype(float)) for label in means.keys()}
```

We compute a mean of all the points without distinguishing classes. Next we compute between class covariance matrix based on the equation:

\begin{equation}
\boldsymbol{B} = \frac{1}{g - 1}\sum_{k=1}^{g}n_{k}(\boldsymbol{m_{k}} - \boldsymbol{m})(\boldsymbol{m_{k}} - \boldsymbol{m})^{T}
\end{equation}

and within class covariance matrix based on the equation:

\begin{equation}
\boldsymbol{W} = \frac{1}{n - g}\sum_{k=1}^{g}(n_{k} - 1)\boldsymbol{S_{k}}.
\end{equation}

However we can simplify the above expression into the form:

\begin{equation}
\boldsymbol{W} = \frac{1}{g}\sum_{k=1}^{g}\boldsymbol{S_{k}}
\end{equation}

because number of points is identical for every class.

```python
mean = np.mean(np.array([means[i] for i in means]), axis=0)
outer_products = [np.outer(value - mean, value - mean) for value in means.values()]
bc_cov = sum(n * outer_products) / (len(means) - 1)
wc_cov = np.mean(np.array([cov[i] for i in cov]), axis=0)
```

We define the matrix $\boldsymbol{U}$:

\begin{equation}
\boldsymbol{U} = \boldsymbol{W^{-1}}\boldsymbol{B},
\end{equation}

determine eigenvalues and eigenvectors of that matrix and choose the eigenvector corresponding to the maximum eigenvalue.

```python
U = np.dot(inv(wc_cov), bc_cov)
eig_values, eig_vectors = eig(U)
a = eig_vectors[:, np.argmax(eig_values)]
```

## Projection of points on the line and decision boundary

Having vector $\boldsymbol{a}$ that maximizes the expression:

\begin{equation}
\boldsymbol{J} = \frac{\boldsymbol{a^{T}Ba}}{\boldsymbol{a^{T}Wa}}
\end{equation}

it's possible to figure out the parameters of discriminant hyperplane.

```python
slope = a[1] / a[0]
intercept = -0.5 * np.dot(a.T, np.sum(np.array([means[i] for i in means]), axis=0))
```

Now we can plot the line which the points will be projected on.

```python
x = np.arange(-5, 5, 0.1)
y = slope * x

plot_points(data)
sns.lineplot(x, y, color='red', linewidth=0.75);
```

After that we define a function that projects the points on the line.

```python
def projection(data, slope):
    data_cp = data.copy()
    data_cp['x'] = (data['x'] + slope * data['y']) / (slope**2 + 1)
    data_cp['y'] = slope * data_cp['x']
    return data_cp
```

```python
proj_data = projection(data, slope)
```

Finally we can plot the projected points and the decision boundary that satisfies the equation:

\begin{equation}
a_{0}x + a_{1}y + b = 0,
\end{equation}

where $a_{0}$ and $a_{1}$ are the elements of vector $\boldsymbol{a}$ and $b$ is the intercept.

```python
plot_points(data)
sns.lineplot(x, y, color='red', linewidth=0.75)
sns.scatterplot(data=proj_data, x='x', y='y', hue='label', legend=False, marker='P', s=75, edgecolor='black')
sns.lineplot(x, -1 / slope * x - intercept / a[1], color='black', linewidth=2);
```

As we can see it's impossible to separate all three classes satisfying the requirements of the exercise with Fisher Discriminant Analysis method.

Exercise 0/Fisher Discriminant Analysis.ipynb