**Problem 1** (7 pts)

In the finite space all norms are equivalent.
This means that given any two norms $\|\cdot\|_*$ and $\|\cdot\|_{**}$ over $\mathbb{C}^{n\times 1}$, inequality
$$
   c_1 \Vert x \Vert_* \leq  \Vert x \Vert_{**} \leq c_2 \Vert x \Vert_*
$$
holds for every $x\in \mathbb{C}^{n\times 1}$ for some constants $c_1, c_2$ which in general depend on the vector size $n$: $c_1 \equiv c_1(n)$, $c_2 \equiv c_2(n)$.
Norms equivalence means that for a certain process convergence in $\Vert \cdot \Vert_{**}$ is followed by $\Vert \cdot \Vert_{*}$ and vice versa. Note that practically convergence in a certain norm may be better than in another due to the strong dependence on $n$.

Consider
\begin{equation}
\begin{split}
c_1(n) \Vert x \Vert_\infty &\leqslant \Vert x \Vert_1 \leqslant c_2(n) \Vert x \Vert_\infty \\
c_1(n) \Vert x \Vert_\infty &\leqslant \Vert x \Vert_2 \leqslant c_2(n)\Vert x \Vert_\infty \\
c_1(n) \Vert x \Vert_2\  &\leqslant \Vert x \Vert_1 \leqslant    c_2(n) \Vert x \Vert_2
\end{split}
\end{equation}

- Generate random vectors and plot optimal constants $c_1$ and $c_2$ from inequalities above as a function of $n$.
- Find these optimal constants analytically and plot them together with constants found numerically

```

```

**Problem 2** (7 pts)

Given $A = [a_{ij}] \in\mathbb{C}^{n\times m}$

- prove that for operator matrix norms $\Vert \cdot \Vert_{1}$, $\Vert \cdot \Vert_{\infty}$ hold
$$ \Vert A \Vert_{1} = \max_{1\leqslant j \leqslant m} \sum_{i=1}^n |a_{ij}|, \quad \Vert A \Vert_{\infty} = \max_{1\leqslant i \leqslant n} \sum_{j=1}^m |a_{ij}|.
$$
**Hint**: show that
$$
\Vert Ax\Vert_{1} \leqslant \left(\max_{1\leqslant j \leqslant m} \sum_{i=1}^n |a_{ij}|\right) \Vert x\Vert_1
$$
and find such $x$ that this inequality becomes equality (almost the same hint is for $ \Vert A \Vert_\infty$).
- check that for randomly generated $x$ and for given analytical expressions for $\Vert \cdot \Vert_{1}$, $\Vert \cdot \Vert_{\infty}$ always hold $\|A\| \geqslant \|Ax\|/\|x\|$ (choose matrix $A$ randomly)
- prove that $ \Vert A \Vert_F = \sqrt{\text{trace}(A^{*} A)}$

```

```

**Problem 3** (6 pts)
- Prove Cauchy-Schwarz (Cauchy-Bunyakovsky) inequality $(x, y) \leqslant \Vert x \Vert \Vert y \Vert $, where $(\cdot, \cdot)$ is a dot product that induces norm $ \Vert x \Vert = (x,x)^{1/2}$.
- Show that vector norm $\|\cdot \|_2$ is unitary ivariant: $\|Ux\|_2\equiv \|x\|_2$, where $U$ is unitary
- Prove that matrix norms $\|\cdot \|_2$ and $\|\cdot \|_F$ are unitary invariant: $\|UAV\|_2 = \|A\|_2$ and $\|UAV\|_F = \|A\|_F$, where $U$ and $V$ are unitary

```

```

**Problem 4** (5 pts)

- Download [Lenna image](http://www.ece.rice.edu/~wakin/images/lenaTest3.jpg) and import it in Python as a 2D real array
- Find its SVD and plot singular values (use logarithmic scale)
- Plot compressed images for several accuracies (use $\verb|plt.subplots|$). Specify their compression rates

```

```

**Problem 5** (bonus tasks)
- The norm is called absolute if $\|x\|=\| \lvert x \lvert \|$ for any vector $x$, where $x=(x_1,\dots,x_n)^T$ and $\lvert x \lvert = (\lvert x_1 \lvert,\dots, \lvert x_n \lvert)^T$. Give an example of a norm which is not absolute.
- Prove that Frobenius norm is not an operator norm

```

```


# 4.3 Linear Discriminant Analysis

Suppose $f_k(x)$ is the class-conditional density of X, and let $\pi_k$ be the prior-probability, with $\sum \pi_k = 1$.
The Bayes theorem gives us (4.7):

$$
Pr(G = k, X = x) = \cfrac{f_k(x)\pi_k}{\sum_{l=1}^K f_l(x)\pi_l}
$$

Suppose that we model each class density as multivariate Gaussian (4.8):

$$
f_k(x) = \cfrac{1}{(2\pi)^{p/2} |\Sigma_k|^{1/2}} e^{-\frac{1}{2}(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)}
$$

Linear discriminant analysis (LDA) arises when $\Sigma_k = \Sigma \text{ }\forall k$. The log-ration between two classes $k \text{ and } l$ is (4.9):

$$
\begin{align}
log \cfrac{PR(G=k|X=x)}{PR(G=l|X=x)} &= log \cfrac{f_k(x)}{f_l(x)} + log \cfrac{\pi_k}{\pi_l}\\
&= log \cfrac{\pi_k}{\pi_l} - \frac{1}{2}(\mu_k+\mu_l)^T\Sigma^{-1}(\mu_k - \mu_l)
+ x^T\Sigma^{-1}(\mu_k-\mu_l),
\end{align}
$$

an equation is linear in x.

From (4.9) we see that the linear discriminant functions (4.10):

$$
\delta_k(x) = x^T\Sigma^{-1}\mu_k - \frac{1}{2}\mu_k^T\Sigma^{-1}\mu_k + log \pi_k
$$

are an equivalent description of the decision rule, with $G(x) = argmax_k \delta_k(x)$.

In practice we do not know the parameters of the Gaussian distributions, and will need to estimate them using the training data:

- $\hat{\pi_k} = N_k / N, N_k$ is the number of class-k observations;

- $\hat{\mu}_k = \sum_{g_i = k} x_i / N_k$

- $\hat{\Sigma} = \sum_{k=1}^K\sum_{g_i=k} (x_i - \hat{\mu}_k)(x_i-\hat{\mu}_k)^T / (N - K)$

With two classes, the LDA rule classifies to class 2 if (4.11):

$$
x^T\hat{\Sigma}^{-1}(\hat{\mu}_2 - \hat{\mu}_1) >
\frac{1}{2}\hat{\mu}_2^T\hat{\Sigma}^{-1}\hat{\mu}_2
- \frac{1}{2}\hat{\mu}_1^T\hat{\Sigma}^{-1}\hat{\mu}_1
+ log(N_1/N)
- log(N_2/N)
$$

Suppose we code the targets in the 2-classes as +1 and -1. It is easy to show that the coefficient vector from least squares is proportional to the LDA direction given in (4.11). However unless $N_1 = N_2$ the intercepts are different.
(**TODO**: solve exercise 4.11)

Since LDA direction via least squares does not use a Gaussian assumption, except the derivation of the intercept or cut-point via (4.11). Thus it makes sense to choose the cut-point that minimizes the training error.

With more than two classes, LDA is not the same as linear regression and it avoids the masking problems.

**Quadratic discriminant functions**

If $\Sigma_k$ are assumed to be equal, then we get *quadratic discriminant functions (QDA)* (4.12):
$$
\delta_k(x)=\cfrac{1}{2}log|\Sigma_k| - \cfrac{1}{2}(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)+log \pi_k
$$


# Interpolation for Squaring

```python
import numpy as np
import matplotlib.pyplot as plt

import os
import sys
sys.path.append(os.path.join(os.getcwd(), '../build/'))
import pydeft as deft

plt.rc('font', family='serif')
plt.rc('xtick', labelsize='x-small')
plt.rc('ytick', labelsize='x-small')
```

Consider samples of a continuous function
\begin{equation}
    w(\mathbf{r}) = \cos{p x},
\end{equation}
such that we have enough samples to reconstruct the continuous function using Fourier-based interpolations.

If we are interested in the quantitiy $w^2$, just squaring the values of the sampled points may not be enough to fully reconstruct the square of the continuous function. This is because larger frequency components have been introduced during the squaring, which might not be adequately described by the sampled points. Hence, the same set of sampled points that could successfully reconstruct $w$ may not adequately reconstruct $w^2$. This can be rectified by first interpolating the sampled points in $w$ before squaring the values on a denser grid.

```python
# create box and data array

dense_factor = 11
grd_pts = 7
dense_grd_pts = grd_pts*dense_factor

box_vectors = np.eye(3)
box = deft.Box(box_vectors)

x_sparse =  np.arange(grd_pts)/grd_pts
x_dense =  np.arange(dense_grd_pts)/dense_grd_pts
x,y,z = np.meshgrid(x_sparse, np.arange(5)/5, np.arange(5)/5,indexing='ij')
p = np.pi*4
data = deft.Double3D([grd_pts, 5, 5])
data[...] = np.cos(p*x)

data_dense = deft.fourier_interpolate(data, [dense_grd_pts, 5, 5])

# font
plt.rc('font', size=14)
fig = plt.figure(figsize=[9,3.5])
plt.plot(x_dense, np.cos(p*x_dense), color='0.5', ls = 'solid', linewidth = 2)
plt.plot(x_sparse, data[:,0,0], 'k+', markersize = 18)
plt.plot(x_dense, data_dense[:,0,0], 'k.', markersize = 8)
plt.legend(['$\cos(4\pi x)$', 'sampled points', 'interpolated points'], loc=(1.02,0.25))
plt.tight_layout()
plt.show()
```

```python
sparse_squared = deft.fourier_interpolate(data*data, [dense_grd_pts, 5, 5])

# plot
plt.rc('font', size=14)
fig = plt.figure(figsize=[9,3.5])
plt.plot(x_dense, np.cos(p*x_dense)**2, color='0.5', ls = 'solid', linewidth = 2)
plt.plot(x_dense, data_dense[:,0,0]**2, 'k.', markersize = 8)
plt.plot(x_sparse, data[:,0,0]**2, 'k+', markersize = 18)
plt.plot(x_dense, sparse_squared[:,0,0], color='0.2', ls = 'dashed', linewidth = 1)
plt.set_xlabel('$x$-axis [one lattice cell]')
plt.legend(['$\cos^2(4\pi x)$', 'interpolated before squaring', 'sampled points squared', 'squared before interpolating'], loc=(1.02,0.25))
plt.tight_layout()
plt.show()
```


# Linear Algebra using SymPy

## Introduction

This notebook is a short tutorial of Linear Algebra calculation using SymPy. For further information refer to SymPy official [tutorial](http://docs.sympy.org/latest/tutorial/index.html).

You can also check the [SymPy in 10 minutes](./SymPy_in_10_minutes.ipynb) tutorial.

```python
from sympy import *
init_session()
```

    IPython console for SymPy 1.0 (Python 2.7.13-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

A matrix $A \in \mathbb{R}^{m\times n}$ is a rectangular array of real number with $m$ rows and $n$ columns. To specify  a matrix $A$, we specify the values for its components as a list of lists:

```python
A = Matrix([
        [3, 2, -1, 1],
        [2, -2, 4, -2],
        [-1, S(1)/2, -1, 0]])
display(A)
```

$$\left[\begin{matrix}3 & 2 & -1 & 1\\2 & -2 & 4 & -2\\-1 & \frac{1}{2} & -1 & 0\end{matrix}\right]$$

We can access the matrix elements using square brackets, we can also use it for submatrices

```python
A[0, 1] # row 0, column 1
```

```python
A[0:2, 0:3] # top-left 2x3 submatrix
```

$$\left[\begin{matrix}3 & 2 & -1\\2 & -2 & 4\end{matrix}\right]$$

We can also create some common matrices. Let us create an identity matrix

```python
eye(2)
```

$$\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

```python
zeros(2, 3)
```

$$\left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$$

We can use algebraic operations like addition $+$, substraction $-$, multiplication $*$, and exponentiation $**$ with ``Matrix`` objects.

```python
B = Matrix([
        [2, -3, -8],
        [-2, -1, 2],
        [1, 0, -3]])
C = Matrix([
        [sin(x), exp(x**2), 1],
        [0, cos(x), 1/x],
        [1, 0, 2]])
```

```python
B + C
```

$$\left[\begin{matrix}\sin{\left (x \right )} + 2 & e^{x^{2}} - 3 & -7\\-2 & \cos{\left (x \right )} - 1 & 2 + \frac{1}{x}\\2 & 0 & -1\end{matrix}\right]$$

```python
B ** 2
```

$$\left[\begin{matrix}2 & -3 & 2\\0 & 7 & 8\\-1 & -3 & 1\end{matrix}\right]$$

```python
C ** 2
```

$$\left[\begin{matrix}\sin^{2}{\left (x \right )} + 1 & e^{x^{2}} \sin{\left (x \right )} + e^{x^{2}} \cos{\left (x \right )} & \sin{\left (x \right )} + 2 + \frac{e^{x^{2}}}{x}\\\frac{1}{x} & \cos^{2}{\left (x \right )} & \frac{1}{x} \cos{\left (x \right )} + \frac{2}{x}\\\sin{\left (x \right )} + 2 & e^{x^{2}} & 5\end{matrix}\right]$$

```python
tan(x) * B ** 5
```

$$\left[\begin{matrix}52 \tan{\left (x \right )} & 27 \tan{\left (x \right )} & - 28 \tan{\left (x \right )}\\- 2 \tan{\left (x \right )} & - \tan{\left (x \right )} & - 78 \tan{\left (x \right )}\\11 \tan{\left (x \right )} & 30 \tan{\left (x \right )} & 57 \tan{\left (x \right )}\end{matrix}\right]$$

And the ``transpose`` of the matrix, that flips the matrix through its main diagonal:

```python
A.transpose() # the same as A.T
```

$$\left[\begin{matrix}3 & 2 & -1\\2 & -2 & \frac{1}{2}\\-1 & 4 & -1\\1 & -2 & 0\end{matrix}\right]$$

## Row operations

```python
M = eye(4)
```

```python
M[1, :] = M[1, :] + 5*M[0, :]
```

```python
M
```

$$\left[\begin{matrix}1 & 0 & 0 & 0\\5 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$$

The notation ``M[1, :]`` refers to entire rows of the matrix. The first argument specifies the 0-based row index, for example the first row of ``M`` is ``M[0, :]``. The code example above implements the row operation $R_2 \leftarrow R_2 + 5R_1$. To scale a row by a constant $c$, use the ``M[1, :] = c*M[1, :]``. To swap rows $1$ and $j$, we can use the Python tuple-assignment syntax ``M[1, :], M[j, :] = M[j, :], M[1, :]``.

## Reduced row echelon form

The Gauss-Jordan elimination procedure is a sequence of row operations that can be performed on any matrix to bring it to its _reduced row echelon form_ (RREF). In Sympy, matrices have a ``rref`` method that compute it:

```python
A.rref()
```

$$\left ( \left[\begin{matrix}1 & 0 & 0 & 1\\0 & 1 & 0 & -2\\0 & 0 & 1 & -2\end{matrix}\right], \quad \left [ 0, \quad 1, \quad 2\right ]\right )$$

It return a tuple, the first value is the RREF of the matrix $A$, and the second tells the location of the leading ones (pivots). If we just want the RREF, we can just get the first entry of the matrix, i.e.

```python
A.rref()[0]
```

$$\left[\begin{matrix}1 & 0 & 0 & 1\\0 & 1 & 0 & -2\\0 & 0 & 1 & -2\end{matrix}\right]$$

## Matrix fundamental spaces

Consider the matrix $A \in \mathbb{R}^{m\times n}$. The fundamental spaces of a matrix are its column space $\mathcal{C}(A)$, its null space $\mathcal{N}(A)$, and its row space $\mathcal{R}(A)$. These vector spaces are importan when we consider the matrix product $A\mathbf{x} = \mathbf{y}$ as a linear transformation $T_A:\mathbb{R}^n\rightarrow \mathbb{R}^n$ of the input vector $\mathbf{x}\in\mathbb{R}^n$ to produce an output vector $\mathbf{y} \in \mathbb{R}^m$.

**Linear transformations** $T_A: \mathbb{R}^n \rightarrow \mathbb{R}^m$ can be represented as $m\times n$ matrices. The fundamental spaces of a matrix $A$ gives us information about the domain and image of the linear transformation $T_A$. The column space $\mathcal{C}(A)$ is the same as the image space $\mathrm{Im}(T_A)$ (the set of all possible outputs). The null space $\mathcal{N}(A)$ is also called kernel $\mathrm{Ker}(T_A)$, and is the set of all input vectors that are mapped to the zero vector. The row space $\mathcal{R}(A)$ is the orthogonal complement of the null space, i.e., the vectors that are mapped to vectors different from zero. Input vectors in the row space of $A$ are in a one-to-one correspondence with the output vectors in the column space of $A$.

Let us see how to compute these spaces, or a base for them!

The non-zero rows in the reduced row echelon form $A$ are a basis for its row space, i.e.

```python
[A.rref()[0][row, :] for row in A.rref()[1]]
```

$$\left [ \left[\begin{matrix}1 & 0 & 0 & 1\end{matrix}\right], \quad \left[\begin{matrix}0 & 1 & 0 & -2\end{matrix}\right], \quad \left[\begin{matrix}0 & 0 & 1 & -2\end{matrix}\right]\right ]$$

The column space of $A$ is the span of the columns of $A$ that contain the pivots.

```python
[A[:, col] for col in A.rref()[1]]
```

$$\left [ \left[\begin{matrix}3\\2\\-1\end{matrix}\right], \quad \left[\begin{matrix}2\\-2\\\frac{1}{2}\end{matrix}\right], \quad \left[\begin{matrix}-1\\4\\-1\end{matrix}\right]\right ]$$

We can also use the ``columnspace`` method

```python
A.columnspace()
```

$$\left [ \left[\begin{matrix}3\\2\\-1\end{matrix}\right], \quad \left[\begin{matrix}2\\-2\\\frac{1}{2}\end{matrix}\right], \quad \left[\begin{matrix}-1\\4\\-1\end{matrix}\right]\right ]$$

Note that we took columns from the original matrix and not from its RREF.

To find (a base for) the null space of $A$ we use the ``nullspace`` method:

```python
A.nullspace()
```

$$\left [ \left[\begin{matrix}-1\\2\\2\\1\end{matrix}\right]\right ]$$

## Determinants

The determinant of a matrix, denoted by $\det(A)$ or $|A|$, isis a useful value that can be computed from the elements of a square matrix. It can be viewed as the scaling factor of the transformation described by the matrix.

```python
M = Matrix([
        [1, 2, 2],
        [4, 5, 6],
        [7, 8, 9]])
```

```python
M.det()
```

## Matrix inverse

For invertible matrices (those with $\det(A)\neq 0$), there is an inverse matrix $A^{-1}$ that have the _inverse_ effect (if we are thinking about linear transformations).

```python
A = Matrix([
        [1, -1, -1],
        [0, 1, 0],
        [1, -2, 1]])
```

```python
A.inv()
```

$$\left[\begin{matrix}\frac{1}{2} & \frac{3}{2} & \frac{1}{2}\\0 & 1 & 0\\- \frac{1}{2} & \frac{1}{2} & \frac{1}{2}\end{matrix}\right]$$

```python
A.inv() * A
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
A * A.inv()
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

## Eigenvectors and Eigenvalues

To find the eigenvalues of a matrix, use ``eigenvals``. ``eigenvals`` returns a dictionary of ``eigenvalue:algebraic multiplicity``.

```python
M = Matrix([
        [3, -2,  4, -2],
        [5,  3, -3, -2],
        [5, -2,  2, -2],
        [5, -2, -3,  3]])

M
```

$$\left[\begin{matrix}3 & -2 & 4 & -2\\5 & 3 & -3 & -2\\5 & -2 & 2 & -2\\5 & -2 & -3 & 3\end{matrix}\right]$$

```python
M.eigenvals()
```

This means that ``M`` has eigenvalues -2, 3, and 5, and that the eigenvalues -2 and 3 have algebraic multiplicity 1 and that the eigenvalue 5 has algebraic multiplicity 2.

To find the eigenvectors of a matrix, use ``eigenvects``. ``eigenvects`` returns a list of tuples of the form ``(eigenvalue:algebraic multiplicity, [eigenvectors])``.

```python
M.eigenvects()
```

$$\left [ \left ( -2, \quad 1, \quad \left [ \left[\begin{matrix}0\\1\\1\\1\end{matrix}\right]\right ]\right ), \quad \left ( 3, \quad 1, \quad \left [ \left[\begin{matrix}1\\1\\1\\1\end{matrix}\right]\right ]\right ), \quad \left ( 5, \quad 2, \quad \left [ \left[\begin{matrix}1\\1\\1\\0\end{matrix}\right], \quad \left[\begin{matrix}0\\-1\\0\\1\end{matrix}\right]\right ]\right )\right ]$$

This shows us that, for example, the eigenvalue 5 also has geometric multiplicity 2, because it has two eigenvectors. Because the algebraic and geometric multiplicities are the same for all the eigenvalues, ``M`` is diagonalizable.

To diagonalize a matrix, use diagonalize. diagonalize returns a tuple $(P,D)$, where $D$ is diagonal and $M=PDP^{−1}$.

```python
P, D = M.diagonalize()
```

```python
P
```

$$\left[\begin{matrix}0 & 1 & 1 & 0\\1 & 1 & 1 & -1\\1 & 1 & 1 & 0\\1 & 1 & 0 & 1\end{matrix}\right]$$

```python
D
```

$$\left[\begin{matrix}-2 & 0 & 0 & 0\\0 & 3 & 0 & 0\\0 & 0 & 5 & 0\\0 & 0 & 0 & 5\end{matrix}\right]$$

```python
P * D * P.inv()
```

$$\left[\begin{matrix}3 & -2 & 4 & -2\\5 & 3 & -3 & -2\\5 & -2 & 2 & -2\\5 & -2 & -3 & 3\end{matrix}\right]$$

```python
P * D * P.inv() == M
```

    True

Note that since ``eigenvects`` also includes the ``eigenvalues``, you should use it instead of ``eigenvals`` if you also want the ``eigenvectors``. However, as computing the eigenvectors may often be costly, ``eigenvals`` should be preferred if you only wish to find the eigenvalues.

If all you want is the characteristic polynomial, use ``charpoly``. This is more efficient than ``eigenvals``, because sometimes symbolic roots can be expensive to calculate.

```python
lamda = symbols('lamda')
p = M.charpoly(lamda)
factor(p)
```

**Note:** ``lambda`` is a reserved keyword in Python, so to create a Symbol called λ, while using the same names for SymPy Symbols and Python variables, use ``lamda`` (without the b). It will still pretty print as λ.

Non-square matrices don’t have eigenvectors and therefore don’t
have an eigendecomposition. Instead, we can use the singular value
decomposition to break up a non-square matrix A into left singular
vectors, right singular vectors, and a diagonal matrix of singular
values. Use the singular_values method on any matrix to find its
singular values.

```python
A
```

$$\left[\begin{matrix}1 & -1 & -1\\0 & 1 & 0\\1 & -2 & 1\end{matrix}\right]$$

```python
A.singular_values()
```

## References

1. SymPy Development Team (2016). [Sympy Tutorial: Matrices](http://docs.sympy.org/latest/tutorial/matrices.html)
2. Ivan Savov (2016). [Taming math and physics using SymPy](https://minireference.com/static/tutorials/sympy_tutorial.pdf)

The following cell change the style of the notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
/* Based on Lorena Barba template available at: https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css*/
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```


# Simon's Algorithm

## Problem statement:

Given: a function $f$ acting on bit strings $f:\{0,1\}^n \rightarrow \{0,1\}^n$ and a promise that $f(x)=f(x \oplus s)$ for all $x$ (addition mod 2). The goal is to use Simon's algorithm to find the unknown string $s$.

### Example:

For example, if $n = 3$, then the following function is an example of a function that satisfies the required and just mentioned property:

|$x$|   $$f(x)$$|
|---|------|
|000|	101|
|001|	010|
|010|	000|
|011|	110|
|100|	000|
|101|	110|
|110|	101|
|111|	010|

Given $f$ is a two-to-one function i.e. it maps exactly two inputs to every unique output, and we find 2 values of input $x$ that have the same output, $f(x_1)=f(x_2)$ then it is guaranteed that $x_1 \oplus x_2 = s$

For example, the input strings $011$ and $101$ are both mapped by $f$ to the same output string $110$. If we XOR $011$ and $101$ we obtain $s$, that is:

$$011 \oplus 101 = 110$$

so for this example $s = 110$

## Problem hardness

To solve this classically, you need to find two different inputs $x$ and $y$ for which $f(x)=f(y)$. Given $f$ is a blackbox, we can discover something about $f$ (or what it does) only when, for two different inputs, we obtain the same output. In any case, we would need to guess $ \Omega ({\sqrt {2^{n}}})$ different inputs before being likely to find a pair on which $f$ takes the same output.

## Simon's Algorithm

The high-level idea behind Simon's algorithm is to "probe" a quantum circuit "enough times" to find $n-1$ (linearly independent) n-bit strings, that is

$$ y_{1},y_{2},\dots ,y_{n-1}\in \{0,1\}^{n} $$

such that the following equations are satisfied

$$ \begin{aligned}y_{1}\cdot s&=0\\y_{2}\cdot s&=0\\&\,\,\,\vdots \\y_{n-1}\cdot s&=0\end{aligned}$$

where $ y_{i}\cdot s$ is the modulo-2 dot product; that is, $ y_{i}\cdot s=y_{i1}s_{1}\oplus y_{i2}s_{2}\oplus \dots \oplus y_{in}s_{n} $

So, this linear system contains $n-1$ linear equations in $n$ unknowns (i.e. the bits of $s$, and the goal is to solve it to obtain $s$, and $s$ is fixed for a given function $f$.

### Simon's quantum circuit

The quantum circuit below is the implementation (and visualization) of the quantum part of Simon's algorithm.

The circuit acts on $2n$ qubits (where $n$ is the length of the bit string in question (i.e., $n=3$ for our example). Apply a Hadamard gate to the first $n$ qubits, then apply $U_f$ - which is an oracle (or "black box"), which knows how to compute $f$ , then apply a Hadamard gate to the first $n$ qubits.

For more details on Simon's algorithm refer to [Wikipedia](<https://en.wikipedia.org/wiki/Simon%27s_problem#Simon's_algorithm>)

## Import related lib

```python
from braket.devices import LocalSimulator
from braket.circuits import Circuit
import random
import matplotlib.pyplot as plt
```

```python
# the following are imported for matrix computing to resolve the final equation set.
from sympy import Matrix, pprint, MatrixSymbol, expand, mod_inverse
```

```python
# define the quantum device.
# use LocalSimulator so that it can be run locally with Braket SDK installed.
device = LocalSimulator()
```

```python
def generate_oracle(secret_s):

    # validating input secret s:
    first_1_bit_location = -1
    other_1_bit_location_list = list()

    for index,bit_value in enumerate(secret_s):
        if (bit_value != '0' and bit_value != '1'):
            raise Exception ('Incorrect char \'' + bit_value + '\' in secret string S:' + secret_s)
        else:
            if (bit_value == '1'):
                if (first_1_bit_location == -1):
                    first_1_bit_location = index
                else:
                    other_1_bit_location_list.append(index)

    if (first_1_bit_location == -1):
        raise Exception (' All 0 in secret string S')

    n = len(str(secret_s))

    oracle_circuit = Circuit()

    oracle_circuit.cnot(first_1_bit_location, first_1_bit_location+n)

    for other_1_bit_location in other_1_bit_location_list:
        oracle_circuit.cnot(first_1_bit_location, other_1_bit_location)

    for i in range(n):
        oracle_circuit.cnot(i, n+i)

    for other_1_bit_location in other_1_bit_location_list:
        oracle_circuit.cnot(first_1_bit_location, other_1_bit_location)

    return oracle_circuit
```

```python
def generate_input_circuit(source_list):

    input_circuit_list = list()

    for input_index, digit_string in enumerate(source_list):
        cur_circuit = Circuit()
        for reg_index, digit_value in enumerate(digit_string):
            if (digit_value == '0'):
                cur_circuit.i(reg_index)
            elif (digit_value == '1'):
                cur_circuit.x(reg_index)
            else:
                raise Exception('incorrect input value: \'' + digit_value + '\' in: ' + digit_string )

        input_circuit_list.append(cur_circuit)

    return input_circuit_list
```

```python

```

```python
def generate_simon_circuit(secret_s):

    bit_number = len(secret_s)

    oracle_circuit = generate_oracle(secret_s)

    input_circuit = Circuit()

    for i in range(bit_number):
        input_circuit.h(i)

    output_circuit = Circuit()

    for i in range(bit_number):
        output_circuit.h(i)

    simon_circuit = input_circuit + oracle_circuit + output_circuit

    return simon_circuit

```

```python
def generate_full_bit_string(bit_number):
    zero_string = '0' * bit_number
    result_list = list()
    for i in range(pow(2, bit_number)):
        cur_string = (zero_string + bin(i)[2:])[-bit_number:]
        result_list.append(cur_string)
    return result_list
```

```python
def generate_random_secret_s(start=2, end=8):
    random_bit_number = random.randint(start, end)

    secret_s_list = generate_full_bit_string(random_bit_number)[1:]

    candidate_number = len(secret_s_list)

    random_index = random.randint(0,candidate_number-1)

    selected_secret_s = secret_s_list[random_index]

    return selected_secret_s
```

```python
secret_s = generate_random_secret_s()
```

```python
secret_s
```

    '10101010'

```python
bit_number = len(secret_s)
```

```python
simon_circuit = generate_simon_circuit(secret_s)
```

```python
print(simon_circuit)
```

    T   : |0|    1    | 2 | 3 | 4 | 5 |6| 7 | 8 |9|

    q0  : -H-C---------C---C---C---C---C-C---C---H-
             |         |   |   |   |   | |   |
    q1  : -H-|-C-------|-H-|---|---|---|-|---|-----
             | |       |   |   |   |   | |   |
    q2  : -H-|-|-------X---|-C-|---|---X-|-H-|-----
             | |           | | |   |     |   |
    q3  : -H-|-|-C-----H---|-|-|---|-----|---|-----
             | | |         | | |   |     |   |
    q4  : -H-|-|-|---------X-|-|-C-|-----X---|-H---
             | | |           | | | |         |
    q5  : -H-|-|-|-C---H-----|-|-|-|---------|-----
             | | | |         | | | |         |
    q6  : -H-|-|-|-|---------|-X-|-|-C-------X---H-
             | | | |         |   | | |
    q7  : -H-|-|-|-|-C-H-----|---|-|-|-------------
             | | | | |       |   | | |
    q8  : ---X-|-|-|-|-------|---|-X-|-------------
               | | | |       |   |   |
    q9  : -----X-|-|-|-------|---|---|-------------
                 | | |       |   |   |
    q10 : -------|-|-|-------X---|---|-------------
                 | | |           |   |
    q11 : -------X-|-|-----------|---|-------------
                   | |           |   |
    q12 : ---------|-|-----------X---|-------------
                   | |               |
    q13 : ---------X-|---------------|-------------
                     |               |
    q14 : -----------|---------------X-------------
                     |
    q15 : -----------X-----------------------------

    T   : |0|    1    | 2 | 3 | 4 | 5 |6| 7 | 8 |9|

```python
# run 2 * bit_number times to get the y output
# according to Simon's algorithm, we only need bit_number-1 independent y to caculate s
# in real world, we may get y with all zeros or dependent y,
# running bit_number-1 time is not enough to get bit_number-1 independent y
# so we run 2 * bit_number times to generate y,
# the complexity is O(2*bit_number), which is still O(bit_number), aka O(n)

task = device.run(simon_circuit, shots=bit_number*2)
result = task.result()
print (result.measurement_counts)
```

    Counter({'0010011001101111': 1, '0001111100101100': 1, '1111010101111000': 1, '0001010001110001': 1, '1110000000100100': 1, '1011111000010011': 1, '1111101000100001': 1, '1011111001110101': 1, '0111100001000010': 1, '1001100000101111': 1, '0101000101001010': 1, '0100111100000011': 1, '1011111001110010': 1, '1000011001011001': 1, '1000100101011100': 1, '0010001001110100': 1})

```python
# Simulate partial measurement by seperating out first n bits
answer_plot = {}
for measresult in result.measurement_counts.keys():
    measresult_input = measresult[:bit_number]
    if measresult_input in answer_plot:
        answer_plot[measresult_input] += result.measurement_counts[measresult]
    else:
        answer_plot[measresult_input] = result.measurement_counts[measresult]

print(f"measurement_of_input_registers: {answer_plot}\n")
plt.bar(answer_plot.keys(), answer_plot.values())
plt.show()
```

```python
lAnswer = [ (k,v) for k,v in answer_plot.items() if k != "0"*bit_number  ] #excluding the trivial all-zero
Y = []
for k, v in lAnswer:
    Y.append( [ int(c) for c in k ] )

print('The output we got:')
for a in Y:
    print (a)
```

    The output we got:
    [0, 0, 1, 0, 0, 1, 1, 0]
    [0, 0, 0, 1, 1, 1, 1, 1]
    [1, 1, 1, 1, 0, 1, 0, 1]
    [0, 0, 0, 1, 0, 1, 0, 0]
    [1, 1, 1, 0, 0, 0, 0, 0]
    [1, 0, 1, 1, 1, 1, 1, 0]
    [1, 1, 1, 1, 1, 0, 1, 0]
    [0, 1, 1, 1, 1, 0, 0, 0]
    [1, 0, 0, 1, 1, 0, 0, 0]
    [0, 1, 0, 1, 0, 0, 0, 1]
    [0, 1, 0, 0, 1, 1, 1, 1]
    [1, 0, 0, 0, 0, 1, 1, 0]
    [1, 0, 0, 0, 1, 0, 0, 1]
    [0, 0, 1, 0, 0, 0, 1, 0]

```python
def mod(x,modulus):
    numer, denom = x.as_numer_denom()
    return numer*mod_inverse(denom,modulus) % modulus
```

```python
Y = Matrix(Y)
Y_transformed = Y.rref(iszerofunc=lambda x: x % 2==0)
Y_new = Y_transformed[0].applyfunc(lambda x: mod(x,2)) #must takecare of negatives and fractional values
for row_index in range(Y_new.shape[0]):
    print (Y_new.row(row_index))
```

    Matrix([[1, 0, 0, 0, 0, 0, 1, 0]])
    Matrix([[0, 1, 0, 0, 0, 0, 0, 0]])
    Matrix([[0, 0, 1, 0, 0, 0, 1, 0]])
    Matrix([[0, 0, 0, 1, 0, 0, 0, 0]])
    Matrix([[0, 0, 0, 0, 1, 0, 1, 0]])
    Matrix([[0, 0, 0, 0, 0, 1, 0, 0]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 1]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 0]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 0]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 0]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 0]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 0]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 0]])
    Matrix([[0, 0, 0, 0, 0, 0, 0, 0]])

```python

```

```python
print("The hidden bistring s[ 0 ], s[ 1 ]....s[",bit_number-1,"] is the one satisfying the following system of linear equations:")
rows, cols = Y_new.shape
result_s = ['0']*bit_number
for r in range(rows):

    location_list = list()
    for i,v in enumerate(list(Y_new[r,:])):
        if (v == 1):
            location_list.append(i)

    if (len(location_list) == 1):
        print ('s[ ' + str(location_list[0]) +' ] = 0')
    elif (len(location_list) > 1):
        for location in location_list:
            result_s[location] = '1'
        Yr = [ "s[ "+str(location)+" ]" for location in location_list ]
        tStr = " + ".join(Yr)
        print(tStr, "= 0")

result_s = ''.join(result_s)

print()
print ('Which is:  ' + result_s)
```

    The hidden bistring s[ 0 ], s[ 1 ]....s[ 7 ] is the one satisfying the following system of linear equations:
    s[ 0 ] + s[ 6 ] = 0
    s[ 1 ] = 0
    s[ 2 ] + s[ 6 ] = 0
    s[ 3 ] = 0
    s[ 4 ] + s[ 6 ] = 0
    s[ 5 ] = 0
    s[ 7 ] = 0

    Which is:  10101010

```python

```

```python
# check whether result_s is equal to secret_s:

if (result_s == secret_s):
    print ('We found the answer')
    print ('\tsecret string:' + secret_s)
    print ('\tresult string:' + result_s)
else:
    print ('Error, the answer is wrong')
```

    We found the answer
    	secret string:10101010
    	result string:10101010

```python

```

### Appendix

How did we design the Oracle circuit.

The following cells explain how we designed the Oracle circuit, using an specified secret_s: 0110.

#### Basic idea

Say that we have a secret string '0110', we want to generate an Oracle circuit with this secret string.

The bit bumber of '0110' is 4, so the input qbit number should be 4.

For all the possible inputs from '0000' to '1111', we can seperate them into two groups: x1 and x2, base on the rule of Simon's problem:

$$x1 \oplus x2 = s$$

Put x1 in column 1, x2 in column 2 and secret string in column 3, we got the following table:

|  x1   | x2  | s
|  ----  | ----  |----  |
| 0000  | 0110 | 0110 |
| 0001  | 0111 | 0110 |
| 0010  | 0100 | 0110 |
| 0011  | 0101 | 0110 |
| 1000  | 1110 | 0110 |
| 1001  | 1111 | 0110 |
| 1010  | 1100 | 0110 |
| 1011  | 1101 | 0110 |

After analyzing the table, we found that we can seperate all the inputs base on the second bit of x.

In column 1, all the second bit of x1 is 0, while in column 2, all the second bit of x2 is 1.

    x1[1] = 0
    x2[1] = 1

In fact, for any location $j$, if $s[j] = 1$, we can use bit number $j$ to seperate all the inputs into two groups which meet the requirement of Simon's Oracle.

In our example, when s is '0110', we can use the third bit too. Of course, we will seperate the intputs in different way if we use the third bit.

+ this is the first key thing we identified, use the location $j$ where we find the first 1 in secret string s. To seperate the inputs into two groups.

As all elements in x1 (which is column 1 in the table) are different from each other, we can use the x1 as the output of the ciruit, this will meet the requirement of Simon's Oracle cuircuit:

|  x1   | x2  | s | y (=x1)
|  ----  | ----  |----  |----  |
| 0000  | 0110 | 0110 |0000  |
| 0001  | 0111 | 0110 |0001  |
| 0010  | 0100 | 0110 |0010  |
| 0011  | 0101 | 0110 |0011  |
| 1000  | 1110 | 0110 |1000  |
| 1001  | 1111 | 0110 |1001  |
| 1010  | 1100 | 0110 |1010  |
| 1011  | 1101 | 0110 |1011  |

From the requirement: $x1 \oplus x2 = s$, we know that $x2 \oplus s = x1$

So, for all the x2, we can make $ y = x1 =  x2 \oplus s $

The basic logic of our circuit is:

    j = the location of first 1 we found in secret_s
    if ( input_x[j] == 0):
        y = input_x
    elif (input_x[j] == 1):
        y = input_x[j] XOR secret_s

+ This is the second key thing we indentified: XOR the input $x$ with $s$ if bit $j$ in $s$ is $1$

#### Implemetation

The question left to us is how to implement the basic logic in quantum circuit

```python
# we have the secret string '0110'
secret_s = '0110'

# the location j of first 1 found in secret_s is 1 (0 is the first bit, 1 is the second bit)
j = 1
```

If we already know that `x[1] = 0`, then the quantum circuit is very simple, we just need to copy all the input qubit to output qubit.

The circuit is like this:

```python
oracle_circuit = Circuit()
oracle_circuit.cnot(0,4).cnot(1,5).cnot(2,6).cnot(3,7)
print (oracle_circuit)
```

    T  : |   0   |

    q0 : -C-------
          |
    q1 : -|-C-----
          | |
    q2 : -|-|-C---
          | | |
    q3 : -|-|-|-C-
          | | | |
    q4 : -X-|-|-|-
            | | |
    q5 : ---X-|-|-
              | |
    q6 : -----X-|-
                |
    q7 : -------X-

    T  : |   0   |

On the other hand, if we know that `x[1] = 1`, we need to let the output be $ x \oplus s$

While, there is no other ways we can input secret s into the quantum circuit.

We need to caculate the result of $x \oplus s $ on the fly.

Look into the $\oplus$ operation bit by bit, we know that when `s[i] = 0` we keep `x[i]`, when `s[i] = 1` we flip `x[i]`.

For the secret s '0110' the circuit is like this:

```python
oracle_circuit = Circuit()
oracle_circuit.cnot(0,4).cnot(3,7).x(1).x(2).cnot(1,5).cnot(2,6)
print (oracle_circuit)
```

    T  : | 0 | 1 |

    q0 : -C-------
          |
    q1 : -|-X-C---
          |   |
    q2 : -|-X-|-C-
          |   | |
    q3 : -|-C-|-|-
          | | | |
    q4 : -X-|-|-|-
            | | |
    q5 : ---|-X-|-
            |   |
    q6 : ---|---X-
            |
    q7 : ---X-----

    T  : | 0 | 1 |

Now we have two seperated circuits for two cases: `x[1] = 0` and `x[1] = 1`.

We need to combine them into one circuit, that means we don't know whether `x[1] = 0` or `x[1] = 1`.

The key thing is that we need to flip qbit1 and qbit2 if `input_x[1] = 1`, that is what exactly cnot gate can do.

For qbit2, we can use qbit1 to control the flip.

For qbit1 itself, there is a little trouble, we do not have cnote gate can flip the control bit itself.

But if we look into the true table of qbit1, we found that we actually don't need to do anything, we just need to output 0. If the input of this bit is 0, that is `x[1] = 0`, we need to output `x[1]`, which is 0, if the input of this bit 1, that is `x[1] = 1`, we need to flip itself and then copy to output bit, which is still 0.

So, we don't need to do anything for qbit1, while, when we are designing quantum circuit, we are requested to generate continued qbit, we must assign some gates for this qbit. So, two cnot gate which always generate 0 is a good choice.

Now, the circuit is something like this:

```python
oracle_circuit = Circuit()
oracle_circuit.cnot(0,4).cnot(1,5).cnot(3,7).cnot(1,2).cnot(1,5).cnot(2,6)
print (oracle_circuit)
```

    T  : |  0  |1| 2 |

    q0 : -C-----------
          |
    q1 : -|-C---C-C---
          | |   | |
    q2 : -|-|---X-|-C-
          | |     | |
    q3 : -|-|-C---|-|-
          | | |   | |
    q4 : -X-|-|---|-|-
            | |   | |
    q5 : ---X-|---X-|-
              |     |
    q6 : -----|-----X-
              |
    q7 : -----X-------

    T  : |  0  |1| 2 |

```python

```

For quantum circuit, we need to make sure that the input `|x>` is equal to output `|x>`.

So we add another cnot gate to convert qbit2 to what it is originaly:

```python
oracle_circuit = Circuit()
oracle_circuit.cnot(0,4).cnot(1,5).cnot(3,7).cnot(1,2).cnot(1,5).cnot(2,6).cnot(1,2)
print (oracle_circuit)
```

    T  : |  0  |1| 2 |3|

    q0 : -C-------------
          |
    q1 : -|-C---C-C---C-
          | |   | |   |
    q2 : -|-|---X-|-C-X-
          | |     | |
    q3 : -|-|-C---|-|---
          | | |   | |
    q4 : -X-|-|---|-|---
            | |   | |
    q5 : ---X-|---X-|---
              |     |
    q6 : -----|-----X---
              |
    q7 : -----X---------

    T  : |  0  |1| 2 |3|

Now we have an Oracle circuit which meet the requirement of Simon's problem.

But there is still somthing missing, in this circuit, if `x[j] = 0` then ` f(x) = x`.

It doesn't look like a real "function", and the user of this Oracle circuit can find some clues of secret_s，then they may solve the problem in shorter time.

A simple solution is adding some x gates in the output qubit to flip some of the output bits, to make the output look different from the input when `x[j] = 0`.

Only if we use same x gates for all shots, it doesn't not impact the result of Simon's algorithm, as the output still meet the requirement of Simon's Oracle.

```python

```


#Snippets and Programs from Chapter 5: Playing with Sets and Probability

```python
%matplotlib inline
```

```python
#P125: Finding the power set of a set
>>> from sympy import FiniteSet
>>> s = FiniteSet(1, 2, 3)
>>> ps = s.powerset()
>>> ps
```

    {EmptySet(), {1}, {2}, {3}, {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}}

```python
#P126: Union of two Sets
>>> from sympy import FiniteSet
>>> s = FiniteSet(1, 2, 3)
>>> t = FiniteSet(2, 4, 6)
>>> s.union(t)
{1, 2, 3, 4, 6}
```

    {1, 2, 3, 4, 6}

```python
#P127: Intersection of two Sets
>>> from sympy import FiniteSet
>>> s = FiniteSet(1, 2)
>>> t = FiniteSet(2, 3)
>>> s.intersect(t)
```

    {2}

```python
#P127/128: Cartesian product of two Sets
>>> from sympy import FiniteSet
>>> s = FiniteSet(1, 2)
>>> t = FiniteSet(3, 4)
>>> p = s*t
>>> for elem in p:
        print(elem)
```

    (1, 3)
    (1, 4)
    (2, 3)
    (2, 4)

```python
#P130: Different gravity, different results
from sympy import FiniteSet, pi
def time_period(length, g):
    T = 2*pi*(length/g)**0.5
    return T

if __name__ == '__main__':
    L = FiniteSet(15, 18, 21, 22.5, 25)
    g_values = FiniteSet(9.8, 9.78, 9.83)
    print('{0:^15}{1:^15}{2:^15}'.format('Length(cm)', 'Gravity(m/s^2)', 'Time Period(s)'))
    for elem in L*g_values:
        l = elem[0]
        g = elem[1]
        t = time_period(l/100, g)
        print('{0:^15}{1:^15}{2:^15.3f}'.format(float(l), float(g), float(t)))

```

      Length(cm)   Gravity(m/s^2) Time Period(s)
         15.0           9.78           0.778
         15.0            9.8           0.777
         15.0           9.83           0.776
         18.0           9.78           0.852
         18.0            9.8           0.852
         18.0           9.83           0.850
         21.0           9.78           0.921
         21.0            9.8           0.920
         21.0           9.83           0.918
         22.5           9.78           0.953
         22.5            9.8           0.952
         22.5           9.83           0.951
         25.0           9.78           1.005
         25.0            9.8           1.004
         25.0           9.83           1.002

```python
#P132: Probability of a Prime number appearing when a 20-sided dice is rolled
def probability(space, event):
    return len(event)/len(space)

def check_prime(number):
    if number != 1:
        for factor in range(2, number):
            if number % factor == 0:
                return False
    else:
        return False
    return True

if __name__ == '__main__':
    space = FiniteSet(*range(1, 21))
    primes = []
    for num in space:
        if check_prime(num):
            primes.append(num)
    event= FiniteSet(*primes)
    p = probability(space, event)
    print('Sample space: {0}'.format(space))
    print('Event: {0}'.format(event))
    print('Probability of rolling a prime: {0:.5f}'.format(p))
```

    Sample space: {1, 2, 3, ..., 18, 19, 20}
    Event: {2, 3, 5, 7, 11, 13, 17, 19}
    Probability of rolling a prime: 0.40000

```python
#P134: Probability of event A or event B
>>> from sympy import FiniteSet
>>> s = FiniteSet(1, 2, 3, 4, 5, 6)
>>> a = FiniteSet(2, 3, 5)
>>> b = FiniteSet(1, 3, 5)
>>> e = a.union(b)
>>> len(e)/len(s)
```

    0.6666666666666666

```python
#P134: Probability of event A and event B
>>> from sympy import FiniteSet
>>> s = FiniteSet(1, 2, 3, 4, 5, 6)
>>> a = FiniteSet(2, 3, 5)
>>> b = FiniteSet(1, 3, 5)
>>> e = a.intersect(b)
>>> len(e)/len(s)
```

    0.3333333333333333

```python
#P135: Can you Roll that score?

'''
Roll a die until the total score is 20
'''
import matplotlib.pyplot as plt
import random

target_score = 20
def roll():
    return random.randint(1, 6)

if __name__ == '__main__':
    score = 0
    num_rolls = 0
    while score < target_score:
        die_roll = roll()
        num_rolls += 1
        print('Rolled: {0}'.format(die_roll))
        score += die_roll
    print('Score of {0} reached in {1} rolls'.format(score, num_rolls))
```

    Rolled: 5
    Rolled: 4
    Rolled: 2
    Rolled: 5
    Rolled: 2
    Rolled: 6
    Score of 24 reached in 6 rolls

```python
#P136: Is the target score possible?
from sympy import FiniteSet
import random
def find_prob(target_score, max_rolls):
    die_sides = FiniteSet(1, 2, 3, 4, 5, 6)
    # sample space
    s = die_sides**max_rolls
    # Find the event set
    if max_rolls > 1:
        success_rolls = []
        for elem in s:
            if sum(elem) >= target_score:
                success_rolls.append(elem)
    else:
        if target_score > 6:
            success_rolls = []
        else:
            success_rolls = []
            for roll in die_sides:
                if roll >= target_score:
                    success_rolls.append(roll)
    e = FiniteSet(*success_rolls)
    # calculate the probability of reaching target score
    return len(e)/len(s)
if __name__ == '__main__':
    target_score = int(input('Enter the target score: '))
    max_rolls  = int(input('Enter the maximum number of rolls allowed: '))
    p = find_prob(target_score, max_rolls)
    print('Probability:  {0:.5f}'.format(p))
```

    Enter the target score: 25
    Enter the maximum number of rolls allowed: 5
    Probability:  0.03241

```python
#P139: Simulate a fictional ATM
'''
Simulate a fictional ATM that dispenses dollar bills
of various denominations with varying probability
'''

import random
import matplotlib.pyplot as plt

def get_index(probability):
    c_probability = 0
    sum_probability = []
    for p in probability:
        c_probability += p
        sum_probability.append(c_probability)
    r = random.random()
    for index, sp in enumerate(sum_probability):
        if r <= sp:
            return index
    return len(probability)-1

def dispense():
    dollar_bills = [5, 10, 20, 50]
    probability = [1/6, 1/6, 1/3, 1/3]
    bill_index = get_index(probability)
    return dollar_bills[bill_index]

# Simulate a large number of bill withdrawls
if __name__ == '__main__':
    bill_dispensed = []
    for i in range(10000):
        bill_dispensed.append(dispense())
    # plot a histogram
    plt.hist(bill_dispensed)
    plt.show()

```

```python
#P140: Draw a Venn diagram for two sets
'''
Draw a Venn diagram for two sets
'''
from matplotlib_venn import venn2
import matplotlib.pyplot as plt
from sympy import FiniteSet
def draw_venn(sets):
    venn2(subsets=sets)
    plt.show()

if __name__ == '__main__':
    s1 = FiniteSet(1, 3, 5, 7, 9, 11, 13, 15, 17, 19)
    s2 = FiniteSet(2, 3, 5, 7, 11, 13, 17, 19)
    draw_venn([s1, s2])
```


# MS-E2121 Exercise session 9
### Problem 9.1: Uncapacitated Facility Location (UFL)

**a)**
Let $N = \{1,\dots,n\}$ be a set of potential facilities and $M = \{1,\dots,m\}$ a set of clients. Let $y_j = 1$ if facility $j$ is opened, and $y_j = 0$ otherwise. Moreover, let $x_{ij}$ be the fraction of client $i$'s demand satisfied from facility $j$. The UFL can be formulated as the mixed-integer problem (MIP):

$$\begin{align}
 \text{(UFL-W)} : \quad &\min_{x,y} \sum_{j\in N} f_jy_j + \sum_{i\in M}\sum_{j\in N} c_{ij}x_{ij} \\
       &\text{s.t.} \\
       &\quad \sum_{j\in N}x_{ij} = 1, &\forall i \in M,\\
       &\quad \sum_{i\in M}x_{ij} \leq my_j, &\forall  j \in N,\\
       &\quad x_{ij} \geq 0, &\forall i \in M, \forall j \in N,\\
       &\quad y_j \in \{0,1\}, &\forall j\in N,
\end{align}$$

where $f_j$ is the cost of opening facility $j$, and $c_{ij}$ is the cost of satisfying client $i$'s demand from facility $j$. Consider an instance of the UFL with opening costs $f=(4,3,4,4,7)$ and client costs

$$\begin{align*}
 (c_{ij}) = \left(
	\begin{array}{ccccc}
		12 & 13 & 6 & 0  & 1 \\
		8  & 4  & 9 & 1  & 2 \\
		2  & 6  & 6 & 0  & 1 \\
		3  & 5  & 2 & 1  & 8 \\
		8  & 0  & 5 & 10 & 8 \\
		2  & 0  & 3 & 4  & 1
	\end{array}
 \right)
\end{align*}$$

Implement (the model) and solve the problem with Julia using JuMP.

**b)**
An alternative formulation of the UFL is of the form

$$\begin{align}
 \text{(UFL-S)} : \quad &\min_{x,y} \sum_{j\in N}f_jy_j + \sum_{i\in M}\sum_{j\in N}c_{ij}x_{ij}\\
       &\text{s.t.}  \\
       &\quad \sum_{j\in N}x_{ij} = 1, &\forall i \in M,\\
       &\quad x_{ij} \leq y_j, &\forall  i\in M, \forall j \in N,\\
       &\quad x_{ij} \geq 0, &\forall i \in M, \forall j \in N,\\
       &\quad y_j \in \{0,1\}, &\forall j\in N.
\end{align}$$

Linear programming (LP) relaxations of these problems can be obtained by relaxing the binary constraints $y_j\in \{0,1\}$ to $0 \leq y_j \leq 1$ for all $j \in N$. For the same instance as in part (a), solve the LP relaxations of UFL-W and UFL-S with Julia using JuMP, and compare the optimal costs of the LP relaxations against the optimal integer cost obtained in part (a).

```julia
using JuMP, Cbc
```

Write down the problem data

```julia
f = [4 3 4 4 7] # Facility opening costs
c = [12 13 6 0 1; 8 4 9 1 2; 2 6 6 0 1; 3 5 2 1 8; 8 0 5 10 8; 2 0 3 4 1] # Cost of satisfying demand
(m, n) = size(c)
M = 1:m # Set of facilities
N = 1:n;# Set of clients
```

Implement the problem in JuMP

```julia
ufl_w = Model(Cbc.Optimizer)

@variable(ufl_w, x[M,N] >= 0) # Fraction of demand (client i) satisfied by facility j
@variable(ufl_w, y[N], Bin)   # Facility location

# Minimize total cost
@objective(ufl_w, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

# For each client, the demand must be fulfilled
@constraint(ufl_w, demand[i in M], sum(x[i,j] for j in N) == 1)
# A big-M style constraint stating that facility j can't send out anything if y[j]==0
@constraint(ufl_w, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])

optimize!(ufl_w)
```

```julia
println("UFL-W MILP:")
println("Optimal value $(objective_value(ufl_w))")
println("with y = $(value.(y).data)")
```

```julia
ufl_w_rel = Model(Cbc.Optimizer)

@variable(ufl_w_rel, x[M,N] >= 0) # Fraction of demand (client i) satisfied by facility j
@variable(ufl_w_rel, 0<=y[N]<=1)  # Facility location

# Minimize total cost
@objective(ufl_w_rel, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

# For each client, the demand must be fulfilled
@constraint(ufl_w_rel, demand[i in M], sum(x[i,j] for j in N) == 1)
# A big-M style constraint stating that facility j can't send out anything if y[j]==0
@constraint(ufl_w_rel, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])

optimize!(ufl_w_rel)
```

```julia
println("UFL-W LP:")
println("Optimal value $(objective_value(ufl_w_rel))")
println("with y = $(value.(y).data)")
```

```julia
ufl_s_rel = Model(Cbc.Optimizer)

@variable(ufl_s_rel, x[M,N] >= 0)
@variable(ufl_s_rel, 0<=y[N]<=1)

@objective(ufl_s_rel, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

@constraint(ufl_s_rel, demand[i in M], sum(x[i,j] for j in N) == 1)
# The difference between the models is that UFL-S has m constraints telling that nothing can be sent to client i from facility j if y[j]==0
# In UFL-W, there is a single constraint telling that nothing can be sent from facility j if y[j]==0
@constraint(ufl_s_rel, supply[i in M, j in N], x[i,j] <= y[j])

optimize!(ufl_s_rel)
```

```julia
println("UFL-S LP:")
println("Optimal value $(objective_value(ufl_s_rel))")
println("with y = $(value.(y).data)")
```

#### Branching
We see that the UFL-S relaxation produces an integer solution, meaning that we have an integer optimal solution and no branching needs to be done. However, if we used UFL-W instead, we would need to do B&B or something else to obtain the integer optimum. In the UFL-W LP relaxation solution (0, 1/3, 0, 2/3, 0), we have two fractional variables $y_2$ and $y_4$, and we can branch on one of them. Let's choose $y_2$ and see what happens if we set it to 0 or 1.

```julia
ufl_w_rel_y2_0 = Model(Cbc.Optimizer)

@variable(ufl_w_rel_y2_0, x[M,N] >= 0)
@variable(ufl_w_rel_y2_0, 0<=y[N]<=1)

@objective(ufl_w_rel_y2_0, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

@constraint(ufl_w_rel_y2_0, demand[i in M], sum(x[i,j] for j in N) == 1)
@constraint(ufl_w_rel_y2_0, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])
@constraint(ufl_w_rel_y2_0, y[2] == 0)

optimize!(ufl_w_rel_y2_0)
```

```julia
ufl_w_rel_y2_1 = Model(Cbc.Optimizer)

@variable(ufl_w_rel_y2_1, x[M,N] >= 0)
@variable(ufl_w_rel_y2_1, 0<=y[N]<=1)

@objective(ufl_w_rel_y2_1, Min, sum(f[j]*y[j] for j in N) + sum(c[i,j]*x[i,j] for i in M, j in N))

@constraint(ufl_w_rel_y2_1, demand[i in M], sum(x[i,j] for j in N) == 1)
@constraint(ufl_w_rel_y2_1, supply[j in N], sum(x[i,j] for i in M) <= m*y[j])
@constraint(ufl_w_rel_y2_1, y[2] == 1)

optimize!(ufl_w_rel_y2_1)
```

```julia
println("UFL-W LP with y2=0:")
println("Optimal value $(objective_value(ufl_w_rel_y2_0))")
println("with y = $(value.(ufl_w_rel_y2_0[:y]).data)")

println()

println("UFL-W LP with y2=1:")
println("Optimal value $(objective_value(ufl_w_rel_y2_1))")
println("with y = $(value.(ufl_w_rel_y2_1[:y]).data)")
```

Both branches have fractional solutions, and more branching is thus needed. You can practice that in the next exercise.
#'
### Problem 9.3: Solving Branch & Bound (B&B) graphically

*You can do this with pen and paper if you want to do it graphically, or solve the problems using JuMP instead if you don't feel like drawing.*

Consider the following integer programming problem $IP$:

$$\begin{matrix}
\text{max} &x_{1} &+&2x_{2} & \\
\text{s.t.}&-3x_{1} &+&4x_{2} &\le 4 \\
&3x_{1} &+&2x_{2} &\le 11 \\
&2x_{1} &-&x_{2} &\le 5 \\
&x_{1}, &x_{2} & \text{integer} &\\
\end{matrix}$$

Plot (or draw) the feasible region of the linear programming (LP) relaxation of the problem $IP$, then solve the problems using the figure. Recall that the LP relaxation of $IP$ is obtained by replacing the integrality constraints $x_1,x_2\in \mathbb{Z}_+$ by linear nonnegativity $x_1,x_2\geq 0$ and including (possible) upper bounds corresponding to the upper bounds of the integer variables ($x_1,x_2\leq 1$ for binary variables).

(a) What is the optimal cost $z_{LP}$ of the LP relaxation of the problem $IP$? What is the optimal cost $z$ of the problem $IP$?

(b) Draw the border of the convex hull of the feasible solutions of the problem $IP$. Recall that the convex hull represents the *ideal* formulation for the problem $IP$.

(c) Solve the problem $IP$ by LP-relaxation based Branch \& Bound (B\&B). You can solve the LP relaxations at each node of the B\&B tree graphically. Start the B\&B procedure without any primal bound.

Check your solutions using JuMP. Make sure to point out the optimal solutions in the figure, as well as giving their numerical values.

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia
# TODO: add your code here
```

```julia

```


# Exercise

```python
#%pylab inline
%pylab notebook
# pylab Populating the interactive namespace from numpy and matplotlib
# numpy for numerical computation
# matplotlib for ploting
import numpy as np
from numpy import append, array, diagonal, tril, triu
from numpy.linalg import inv
from scipy.linalg import lu
#from scipy.linalg import solve
from pprint import pprint
from numpy import array, zeros, diag, diagflat, dot
from sympy import *
import sympy as sym
init_printing()
```

    Populating the interactive namespace from numpy and matplotlib

    /opt/conda/lib/python3.5/site-packages/IPython/core/magics/pylab.py:161: UserWarning: pylab import has clobbered these variables: ['Polygon', 'log', 'sqrt', 'source', 'det', 'ones', 'power', 'zeros', 'sinc', 'solve', 'exp', 'interactive', 'product', 'sign', 'tanh', 'sinh', 'plotting', 'test', 'flatten', 'multinomial', 'poly', 'cos', 'Circle', 'cbrt', 'var', 'transpose', 'tan', 'diff', 'roots', 'mod', 'seterr', 'trace', 'plot', 'reshape', 'gamma', 'beta', 'pi', 'eye', 'cosh', 'trunc', 'take', 'diag', 'conjugate', 'floor', 'invert', 'vectorize', 'add', 'prod', 'binomial', 'nan', 'sin']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

```python

```

2.1 Solve Ax =b

```python
A = array([[54, 14, -11, 2],
               [14, 50, -4, 29],
               [-11, -4, 55, 22],
               [2, 29, 22, 95]])
b = array([1, 1, 1, 1])
x0 = [1, 1, 1,1]

```

## (a) L-U decomposition

```python
n = 20

p, l, u = lu(A)
```

```python
p
```

    array([[ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  1.]])

```python
l
```

    array([[ 1.        ,  0.        ,  0.        ,  0.        ],
           [ 0.25925926,  1.        ,  0.        ,  0.        ],
           [-0.2037037 , -0.02476038,  1.        ,  0.        ],
           [ 0.03703704,  0.61421725,  0.43831321,  1.        ]])

```python
u
```

    array([[ 54.        ,  14.        , -11.        ,   2.        ],
           [  0.        ,  46.37037037,  -1.14814815,  28.48148148],
           [  0.        ,   0.        ,  52.73083067,  23.11261981],
           [  0.        ,   0.        ,   0.        ,  67.30154198]])

```python
b
```

    array([1, 1, 1, 1])

```python
y = np.linalg.solve(l, b)
y

```

    array([ 1.        ,  0.74074074,  1.22204473, -0.02765113])

```python
x = np.linalg.solve(u, y)
x
```

    array([ 0.01893441,  0.01680508,  0.02335523, -0.00041085])

## (b) Gauss-Jacobi iteration

```python
def gjacobi(A, b, maxit=1000,tol = 1/1000, x0=None):
    """
    Solve a linear equation by the gauss jacobi iteration outlined in the book.
    Follows the eq:
        x = inv(D)(b - Rx)
    Where D is the diagonal matrix of A and R is the remainder s.t D + R = A
    """
    # If we have not provided an initial array for x make a new one
    if x0==None:
        x = array([1 for _ in range(A.shape[1])])
    else:
        x = x0
    d = np.diag(np.diag(A))

    for it in np.arange(maxit):
        dx =  inv(d).dot(b - A.dot(x))
        x = x + dx
        if np.linalg.norm(dx)<tol:
            break
    #Do not print the result for 1000 times
    #print("x:\n", x)
    return x

gjacobi(A,b, maxit = 10000, tol = 1/100000)
```

    array([ 0.01893373,  0.01680349,  0.02335483, -0.00041184])

## (c) Gauss-Seidel iteration
How many Gauss-Jacobi and Gauss-Seidel iterations are required to get answers
that agree with the L-U decomposition solution to four signicant digits?

```python
def gseidel(A, b, maxit=1000, tol=1/1000,lamb =1, x0=None):
    """
    Solve a linear equation by the gauss seidel iteration outlined in the book
    Follows the eq:

    """
    Q = np.tril(A)
    # If we have not provided an initial array for x make a new one
    if x0 == None:
        x = array([1 for _ in range(A.shape[1])])
    else:
        x = x0
    for it in np.arange(maxit):
        dx = np.linalg.inv(Q).dot(b -A.dot(x))
        x = x + lamb*dx
        if np.linalg.norm(dx)<tol:
            break
    #Do not print the result for 1000 times
    #print("x:\n", x)
    return x

gseidel(A,b, maxit = 10000, tol = 1/100000)
```

    array([ 0.01893538,  0.0168029 ,  0.02335414, -0.00040996])

### 2.3.

Use a random number generator create a random 10 by 10 matrix A and a
random 10 by 1 vector b. Then use the Matlab function flop to count the number
of floating point operations needed to solve the linear equation Ax = b 1, 10,
and 50 times for each of the following algorithms:

 - (a) $x = A \ b$
 - (b) $x = U^{-1}(L^{-1}b)$, computing the L-U factors of A only once using the Matlab
function lu.
 - (c) $x = A^{-1}b$, computing $A^{-1}$ only once using function that computes the inverse.

https://docs.python.org/3/library/timeit.html
https://stackoverflow.com/questions/29280470/what-is-timeit-in-python

using IPython’s timeit magic,

```python
%lsmagic
```

    Available line magics:
    %alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

    Available cell magics:
    %%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

    Automagic is ON, % prefix IS NOT needed for line magics.

```python
%%timeit
gjacobi(A,b, maxit = 10000, tol = 1/100000)

```

    1000 loops, best of 3: 575 µs per loop

```python
%%timeit

gseidel(A,b, maxit = 10000, tol = 1/100000)
```

    1000 loops, best of 3: 406 µs per loop

```python

```


Name: Julie Gardner-Hoag, Cynthia Parks
Student ID: 2299636, 2303535
Email: gardnerh@chapman.edu, cparks@chapman.edu
Course: CS510 Fall 2017
Assignment: Classwork 4

```python
import numpy as np    # Numeric python
import sympy as sp    # Symbolic python
import pandas as pd   # Data processing python
import primes

```

# The Sieve of Eratosthenes

The goal of this algorithm is to find all prime numbers up to a number n. This algorithm works by starting at the number 2 and removing all multiple of 2. Then we remove all multiples of 3, and so on.  Once multiples of 3 are removed, we check for the next available number.  This ends up being 5 since 4 was removed during the multiples of 2 process.  This is how we are able to get the primes left behind.

For this function we decided to go with a list because the removal process works well with generating a full list of integers up to n, then picking out all the composite numbers.

```python
def eratosthenes(n):
    primes = []
    if n<=0:
        print("Please enter a positive number")
        return
    count = 2
    while count<=n-1:
        primes.append(count)
        count +=1
    p = 2
    while p<=n:
        for i in range(2, n+1):
            if i*p <= n:
                if i*p not in primes:
                    continue
                else:
                    primes.remove(i*p)
        p +=1
    return primes
```

# Generating Prime Numbers

The function gen_eratosthenes() generates all prime numbers by running the eratosthenes function each implementation, upping the current variable until a new prime is added to the list.  Since this generator uses the original function, the implementation is very similar.

```python
def gen_eratosthenes():
    a = 2
    gen_primes = []
    while True:
        if a is in eratosthenes(a+1):
            gen_primes.append(a)
            yield a
            a += 1
        else:
            a += 1
    return gen_primes
```

# Benchmarking Implementations

Use %time and %timeit magic function to compare functions and explain which one is faster. By how much? Does it match your prediction about efficiency above? Speculate about what is being slow about the slower implementation.
Using the %timeit magic function, we can compare which function is faster.

```python
%timeit eratosthenes(100)
```

    766 µs ± 43.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
%timeit gen_eratosthenes()
```

This leads us to believe that the generator function works faster because it doesn't need a for loop, thus increasing the efficieny of the code.


## Fibonacci numbers

The Fibonacci numbers are defined recursively by the following difference equation:

\begin{equation}
\left\{
\begin{aligned}
    F_{n} & = F_{n-1} + F_{n-2} \\
    F_1 & = 1 \\
    F_0 & = 0 \\
\end{aligned}
\right.
\end{equation}

It is easy to compute the first few elements in the sequence:

$0, 1, 1, 2, 3, 5, 8, 13, 21, 34 \cdots $

## Derivation of the general formula

It is possible to derive a general formula for $F_n$ without computing all the previous numbers in the sequence. If a gemetric series (i.e. a series with a constant ratio between consecutive terms $r^n$) is to solve the difference equation, we must have

\begin{aligned}
    r^n = r^{n-1} + r^{n-2} \\
\end{aligned}

which is equivalent to

\begin{aligned}
    r^2 = r + 1 \\
\end{aligned}

This equation has two unique solutions
\begin{aligned}
    \varphi = & \frac{1 + \sqrt{5}}{2} \approx 1.61803\cdots \\
    \psi = & \frac{1 - \sqrt{5}}{2} = 1 - \varphi = - {1 \over \varphi} \approx -0.61803\cdots \\
\end{aligned}

In particular the larger root is known as the _golden ratio_
\begin{align}
\varphi = \frac{1 + \sqrt{5}}{2} \approx 1.61803\cdots
\end{align}

Now, since both roots solve the difference equation for Fibonacci numbers, any linear combination of the two sequences also solves it

\begin{aligned}
    a \left(\frac{1 + \sqrt{5}}{2}\right)^n + b \left(\frac{1 - \sqrt{5}}{2}\right)^n \\
\end{aligned}

It's not hard to see that all Fibonacci numbers must be of this general form because we can uniquely solve for $a$ and $b$ such that the initial conditions of $F_1 = 1$ and $F_0 = 0$ are met

\begin{equation}
\left\{
\begin{aligned}
    F_0 = 0 = a \left(\frac{1 + \sqrt{5}}{2}\right)^0 + b \left(\frac{1 - \sqrt{5}}{2}\right)^0 \\
    F_1 = 1 = a \left(\frac{1 + \sqrt{5}}{2}\right)^1 + b \left(\frac{1 - \sqrt{5}}{2}\right)^1 \\
\end{aligned}
\right.
\end{equation}

yielding

\begin{equation}
\left\{
\begin{aligned}
    a = \frac{1}{\sqrt{5}} \\
    b = \frac{-1}{\sqrt{5}} \\
\end{aligned}
\right.
\end{equation}

We have therefore derived the general formula for the $n$-th Fibonacci number

\begin{aligned}
    F_n = \frac{1}{\sqrt{5}} \left(\frac{1 + \sqrt{5}}{2}\right)^n - \frac{1}{\sqrt{5}} \left(\frac{1 - \sqrt{5}}{2}\right)^n \\
\end{aligned}

Since the second term has an absolute value smaller than $1$, we can see that the ratios of Fibonacci numbers converge to the golden ratio

\begin{aligned}
    \lim_{n \rightarrow \infty} \frac{F_n}{F_{n-1}} = \frac{1 + \sqrt{5}}{2}
\end{aligned}

## Various implementations in Python

Writing a function in Python that outputs the $n$-th Fibonacci number seems simple enough. However even in this simple case one should be aware of some of the computational subtleties in order to avoid common pitfalls and improve efficiency.

### Common pitfall #1: inefficient recursion

Here's a very straight-forward recursive implementation

```python
import math
from __future__ import print_function
```

```python
def fib_recursive(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib_recursive(n-1) + fib_recursive(n-2)
```

```python
print([fib_recursive(i) for i in range(20)])
```

    [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181]

this seems to work fine, however the recursion overhead is actually very significant when $n$ is just slightly large. Here I'm computing $F_{34}$ and it takes more than 3 seconds! (on a 2013 model Macbook Air)

```python
%timeit fib_recursive(34)
```

    1 loops, best of 3: 3.58 s per loop

The overhead incurred by creating a large number of stack frames is tremendous. Python by default does not perform what's known as tail recursion elimination http://stackoverflow.com/questions/13543019/why-is-recursion-in-python-so-slow, and therefore this is a very inefficient implemenation. In contrast, if we have an iterative implementation, the speed is dramatically faster

```python
def fib_iterative(n):
    a, b = 0, 1
    while n > 0:
        a, b = b, a + b
        n -= 1
    return a
```

```python
%timeit fib_iterative(34)
```

    100000 loops, best of 3: 4.59 µs per loop

Now, let's see if we can make it even faster by eliminating the loop altogether and just go straight to the general formula we derived earlier

```python
def fib_formula(n):
    golden_ratio = (1 + math.sqrt(5)) / 2
    val = (golden_ratio**n - (1 - golden_ratio)**n) / math.sqrt(5)
    return int(round(val))
```

```python
%timeit fib_formula(34)
```

    1000000 loops, best of 3: 1.36 µs per loop

Even faster, great! And since we are not looping anymore, we should expect to see the computation time to scale better as $n$ increases. That's indeed what we see:

```python
import pandas as pd
import numpy as np
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
from IPython.core.pylabtools import figsize
figsize(15, 5)
```

```python
elapsed = {}
elapsed['iterative'] = {}
elapsed['formula'] = {}
for i in range(34):
    result = %timeit -n 10000 -q -o fib_iterative(i)
    elapsed['iterative'][i] = result.best
    result = %timeit -n 10000 -q -o fib_formula(i)
    elapsed['formula'][i] = result.best
```

```python
elapased_ms = pd.DataFrame(elapsed) * 1000
elapased_ms.plot(title='time taken to compute the n-th Fibonaccis number')
plt.ylabel('time taken (ms)')
plt.xlabel('n')
```

Indeed as we expect, the iterative approach scales linearly, while the formula approach is basically constant time.

However we need to be careful with using a numerical formula like this for getting integer results.

### Common pitfall #2: numerical precision

Here we compare the actual values obtained by `fib_iterative()` and `fib_formula()`. Notice that it does not take a very large `n` for us to run into numerical precision issues.

When `n` is 71 we are starting to get different results from the two implementations!

```python
df = {}
df['iterative'] = {}
df['formula'] = {}
df['diff'] = {}

for i in range(100):
    df['iterative'][i] = fib_iterative(i)
    df['formula'][i] = fib_formula(i)
    df['diff'][i] = df['formula'][i] - df['iterative'][i]
df = pd.DataFrame(df, columns=['iterative', 'formula', 'diff'])
df.index.name = 'n-th Fibonacci'
df.ix[68:74]
```

      iterative
      formula
      diff

      n-th Fibonacci

      68
         72723460248141
         72723460248141
       0

      69
        117669030460994
        117669030460994
       0

      70
        190392490709135
        190392490709135
       0

      71
        308061521170129
        308061521170130
       1

      72
        498454011879264
        498454011879265
       1

      73
        806515533049393
        806515533049395
       2

      74
       1304969544928657
       1304969544928660
       3

You can see that `fib_iterative()` produces the correct result by eyeballing the sum of $F_{69}$ and $F_{70}$, while `fib_formual()` starts to have precision errors as the number gets larger. So, be mindful with precision issues when doing numerical computing. Here's a nice article on this topic http://www.codeproject.com/Articles/25294/Avoiding-Overflow-Underflow-and-Loss-of-Precision

Also notice that unlike C/C++, in Python there's technically no limit in the precision of its integer representation. In Python 2 any overflowing operation on `int` is automatically converted into `long`, and `long` has arbitrary precision. In Python 3 it is just `int`. More information on Python's arbitrary-precision integers can be found here  http://stackoverflow.com/questions/9860588/maximum-value-for-long-integer


# What is an optimization problem?

A general mathematical formulation for **the optimization problems studied on this course** is
$$
\begin{align} \
\min \quad &f(x)\\
\text{s.t.} \quad & g_j(x) \geq 0\text{ for all }j=1,\ldots,J\\
& h_k(x) = 0\text{ for all }k=1,\ldots,K\\
&x\in \mathbb R^n.
\end{align}
$$

The above problem can be expressed as
>Find an $x\in \mathbb R^n$ such that $g_j(x)\geq 0$ for all $j=1,\ldots,J$ and $h_k(x)=0$ for all $k=1,\ldots,K$, and there does not exist $x'\in \mathbb R^n$ such that $f(x')<f(x)$ and $g_j(x')\geq 0$ for all $j=1,\ldots,J$, $h_k(x')=0$ for all $k=1,\ldots,K$.

There are three main components to an optimization problem:
* the variables $x$ are called the **decision variables**,
* the equalities and inequalities $g_j(x)\geq 0$ and $h_k(x)=0$ are called the **constraints**,
* the funtion $f(x)$ is called the **objective function**.

Values of decision variables $x^*$ are called **solutions** and a solution is called
* **feasible** if $g_j(x^*)\geq 0$ for all $j=1,\ldots,J$, $h_k(x^*)=0$ for all $k=1,\ldots,K$,
* **locally optimal** if $x^*$ is feasible and there exists $r>0$ such that there does not exist a feasible solution $x'\in \operatorname{B}(x^*,r)$ such that $f(x')<f(x^*)$, and
* **optimal** if $x^*$ is feasible and there does not exist a feasible solution $x'$ such that $f(x')<f(x^*)$.

The problem is called
* **linear/nonlinear** if the objective function and the constraints of the problem are/are not affinely linear,
* **multi/unimodal** if the problem has/does not have more than one local optimum,
* **convex/nonconvex** if the objective and the constraints are that,
* **continuous/differentiable/twice-differentiable, etc** if the objective and the constraints are that.

## Why do we study optimization problems?

Because optimization problems arise in various fields engineering, finance, medicine etc.

### Mixing problem
Refinery produces 3 types of gasoline by mixing 3 different grude oil. Each grude oil can be purchased maximum of 5000 barrels per day. Let us assume that octane values and lead concentrations behave linearly in mixing. Refining costs are 4\$ per barrel and the capacity of the refinery is 14000 barrels per day. Demand of gasoline can be increased by (demand grows 10 barrels per day for each \$ used for advertizing). The details of the gasolines and Grude oils can be found in the following tables:

| |Gasoline 1|Gasoline 2|Gasoline 3|
|---|---|---|---|
|Sale price|70|60|50|
|Lower limit for octane|10|8|6|
|Upper limit for lead|0.01|0.02|0.01|
|Demand|3000|2000|1000|
|Refining cost|4|4|4|

| |Grude oil 1|Grude oil 2|Grude oil 3|
|---|---|---|---|
|Purchase price|45|35|25|
|Octane value|12|6|8|
|Lead concentration|0.005|0.02|0.03|
|Availability|5000|5000|5000|

Determine the production quantities of each type of gasoline, mixing ratios of different grude oil and advertising budget so that the profit is maximized and the demand is met exactly.

#### Modelling mixing problem as an optimization problem:
* decision variables
    * $x_{ij}$ = amount of grude oil $i$ used for producing gasoline $j$
    * $y_j$ = the amount of money used for advertizing gasoline $j$
* constraints
    * gasoline 1 demand: $x_{1,1}+x_{2,1}+x_{3,1}=3000+10y_1$
    * gasoline 2 demand: $x_{1,2}+x_{2,2}+x_{3,2}=2000+10y_2$
    * gasoline 3 demand: $x_{1,3}+x_{2,3}+x_{3,3}=1000+10y_3$
    * grude oil 1 availability: $x_{1,1}+x_{1,2}+x_{1,3}\leq 5000$
    * grude oil 2 availability: $x_{2,1}+x_{2,2}+x_{2,3}\leq 5000$
    * grude oil 3 availability: $x_{3,1}+x_{3,2}+x_{3,3}\leq 5000$
    * gasoline 1 octave value: $\frac{12x_{1,1}+6x_{2,1}+8x_{3,1}}{x_{1,1}+x_{2,1}+x_{3,1}}\geq 10$
    * gasoline 2 octave value: $\frac{12x_{1,2}+6x_{2,2}+8x_{3,2}}{x_{1,2}+x_{2,2}+x_{3,2}}\geq 8$
    * gasoline 3 octave value: $\frac{12x_{1,3}+6x_{2,3}+8x_{3,3}}{x_{1,3}+x_{2,3}+x_{3,3}}\geq 6$
    * gasoline 1 lead value: $\frac{0.005x_{1,1}+0.02x_{2,1}+0.03x_{3,1}}{x_{1,1}+x_{2,1}+x_{3,1}}\leq 0.01$
    * gasoline 2 lead value: $\frac{0.005x_{1,2}+0.02x_{2,2}+0.03x_{3,2}}{x_{1,2}+x_{2,2}+x_{3,2}}\leq 0.02$
    * gasoline 3 lead value: $\frac{0.005x_{1,3}+0.02x_{2,3}+0.03x_{3,3}}{x_{1,3}+x_{2,3}+x_{3,3}}\leq 0.01$
* objective function:
$$
70(x_{1,1}+x_{2,1}+x_{3,1})+60(x_{1,2}+x_{2,2}+x_{3,2})+50(x_{1,3}+x_{2,3}+x_{3,3})-45(x_{1,1}+x_{1,2}+x_{1,3})-35(x_{2,1}+x_{2,2}+x_{2,3})-25(x_{3,1}+x_{3,2}+x_{3,3})
$$

# How to solve optimization problems?

## Iterative vs. non-iterative methods

Optimal solutions to some optimization problems can be found by defining an explicit formula for it. For example, if the objective function is twice continuously differentiable and there are no constraints, the optimal solution (if exists) can be found by calculating all the zero-points of the gradient and finding the best one of those. In this kind of cases, the optimization problem **can be solved using non-iterative methods.**

**In this course we concentrate on the iterative methods.** Iterative methods are needed, if the problem has constraints, or the problem is in some other way not-well behaved (to be defined later, depending on the context). In iterative methods, the solving the optimizaiton problem starts from a so-called starting solution and then tries to improve the solution iteratively. The optimization algorithm chooses how the solution is changed at each iteration.

## What kind of methods will you learn in this course?
Different optimization problems require different methods. In this course, we study optimization problems, which are
* non-linear
* not hugely multimodal

Often the methods cannot guarantee a (global) optimum, but instead **we need to satisfy ourselves with a local optimum**. In addition, it is usually not possible to find the actual optimal solution, but instead **an approximation of the optimal solution**. A feasible solution $x^*$ is called an approximation of a local optimum $x^{**}$ with quality $L>0$, when $\|x^*-x^{**}\|\leq L$.

# Line search

Let us study optimization problem $\min_{x\in[a,b]} f(x)$, where $a,b\in\mathbb R$. Let us try to find an approximation of a local optimum to this problem.

```python
#Example objective function
def f(x):
    return 2+(1-x)**2
```

```python
print "The value of the objective function at 0 is " + str(f(0))
```

    The value of the objective function at 0 is 3

## Line search with fixed steps
**input:** the quality $L>0$ of the approximation of the local optimum.
**output:** an approximation of the local optimum with quality $L$.
```
start with x as the start point of the interval
loop until stops:
    if the value of the objective is increasing for x+L from x
        stop, because the approximation of the locally optimal solution is x
    increase x by L
```

```python
def fixed_steps_line_search(a,b,f,L):
    x = a
    while f(x)>f(x+L) and x+L<b:
        x=x+L
    return x
```

```python
%timeit fixed_steps_line_search(0.0,3.0,f,1e-3)
#x = fixed_steps_line_search(0,3,f,1e-3); print "optimum is "+str(x)
```

    1000 loops, best of 3: 587 µs per loop

## The method of bisection
**input:** the quality $L>0$ of the approximation of the local optimum.
**output:** an approximation of the local optimum with quality $L$.
```
Set x as the start point of interval and y as the end point
while y-x<2*L:
    if the function is increasing at the mid point between x and y:
        set y as the midpoint between y and x, because a local optimum is before the midpoint
    otherwise:
        set x as the midpoint, because a local optimum is after the midpoint
return midpoint between x and y
```

The following function is to be completed in class as an exercise

```python
def bisection_line_search(a,b,f,L,epsilon):
    x = a
    y = b
    while y-x>2*L:
        if f((x+y)/2+epsilon)>f((x+y)/2-epsilon):
            y=(x+y)/2+epsilon
        else:
            x = (x+y)/2-epsilon
    return (x+y)/2
```

This is what we should end up. The following function is not shown on the slides.

```python
def bisection_line_search(a,b,f,L,epsilon):
    x = a
    y = b
    while y-x<2*L:
        if f((x+y)/2-epsilon)<f((x+y)/2+epsilon):
            y = (x+y)/2+epsilon
        else:
            x = (x+y)/2-epsilon
    return (x+y)/2

```

```python
%timeit bisection_line_search(0.0,3.0,f,1e-3,1e-4)
#x = bisection_line_search(0.0,3.0,f,0.01,1e-4); print "x="+str(x)
```

    100000 loops, best of 3: 9.61 µs per loop

## Golden section search

### Golden section

Let $a<c<b$ be such that $\frac{b-a}{c-a}=\frac{c-a}{b-c}$. Then it is said that the point $c$ devides interval $[a,b]$ in the ratio of golden section (from the left, mirror from the right). Note that $c=a+\frac{\sqrt{5}-1}2(b-a)\approx a+0.618(b-a)$.

There is a theorem that if $a<c<d<b$ and both points divide the interval $[a,b]$ in the ratio of golden section (from right and left), then the point $c$ divides the interval $[a,d]$ in the ratio of golder ration from the left.

### Golden section search algorithm

**input:** the quality $L>0$ of the approximation of the local optimum.
**output:** an approximation of the local optimum with quality $L$.
```
Set x as the start point of interval and y as the end point
while y-x>2*L:
    Divide the interval [x,y] in the golden section from the letf and right and attain two division points
    If the greater of the division points has a greater function value
        set y as the rightmost division point, because a local optimum is before that
    otherwise:
        set x as the leftmost division point, because a local optimum is after that
return midpoint between x and y
```

The following function is to be completed in class as an exercise

```python
import math
def golden_section_line_search(a,b,f,L):
    x = a
    y = b
    while y-x>2*L:
        if f(x+(math.sqrt(5.0)-1)/2.0*(y-x))<f(y-(math.sqrt(5.0)-1)/2.0*(y-x)):
            x = y-(math.sqrt(5.0)-1)/2.0*(y-x)
        else:
            y = x+(math.sqrt(5.0)-1)/2.0*(y-x)
    return (x+y)/2
```

```python
%timeit golden_section_line_search(0.0,3.0,f,1e-3)
#x = golden_section_line_search(0.0,3.0,f,0.01); print "x="+str(x)
```

    10000 loops, best of 3: 20.1 µs per loop

Lecture 2, What is an optimization problem and how to solve them and line search.ipynb


# Module 2 - Example: Using Proofs in Algorithms

CSCI304: Analysis and Design of Algorithms

Nile University

Ammar Sherif (email-ID: ASherif)

## Problem Statement

Prove that the following algorithm is correct:

```python
def maxVal(A):
    """
    The Algorithm returns the maximum value in the list

    Input constraints: A is not empty
    """
    maxV = A[0]
    for v in A:
        if maxV < v:
            maxV = v
    return maxV
```

```python
maxVal([1,2,3,40,5,6])
```

    40

## Proof

We first try to try to formulate a hypothesis to help us in proving the correctness. It is as follows:

$$H(i): \text{ After the completion of the }i^{th} \text{ iteration,}$$
$$\boxed{\texttt{maxV} = \max(A[0],A[1],\cdots,A[i-1])}$$
Which is a compact representation of:
\begin{equation}
    \texttt{maxV} \in \{ A[0],A[1],\cdots,A[i-1]\}\\
    \land ( \texttt{maxV} \geq A[0] \land \texttt{maxV} \geq A[1] \land \cdots \land \texttt{maxV} \geq A[i-1] )
\end{equation}
To prove using Induction, we have two steps we have to show that it holds:

Base Case

Inductive Step

### Base case

After the completion of the first iteration, we know that $\texttt{maxV} = \max(A[0]) = A[0]$ because of the **initialization** at the first line before the loop. Therefore, we showed that $\boxed{H(1)}$ holds.

### Inductive step

We want to show that $$H(k) \implies H(k+1)$$
Therefore, assuming $H(k)$ is True:

$$H(k) \implies \texttt{maxV}\rvert_k = \max(A[0],A[1],\cdots,A[k-1])$$
For the next iteration $\texttt{v} = A[k]$; then, we have a condition as follows
\begin{align*}
H(k) &\implies\\
\texttt{maxV}\rvert_{k+1} &=
\begin{cases}
\texttt{v} = A[k], \qquad \texttt{maxV}\rvert_{k} < A[k]\\
\texttt{maxV}\rvert_{k}, \qquad \text{otherwise}
\end{cases}\\
&= \max(\texttt{maxV}\rvert_{k},A[k])\\
&= \max(\underbrace{\max(A[0],A[1],\cdots,A[k-1])}_{H(k)},A[k])\\
&= \max(A[0],A[1],\cdots,A[k])
\end{align*}
Therefore, after the end of this iteration the variable
$$\underbrace{\texttt{maxV} = \max(A[0],A[1],\cdots,A[k])}_{H(k+1)}$$
$$\boxed{H(k) \implies H(k+1)}$$

### Termination

Now, using the induction proof above: we know that after the last iteration, $m^{th}$ iteration, $H(m)$ holds. Therefore, $$\texttt{maxV} = \max(A[0],\cdots,A[m-1])$$
The remaining is that our algorithm terminates after the end of the loop, returning the maximum value, so the algorithm is *correct*.

Module 02: Introduction to Theoretical Analysis/Module 02: Example on Correctness Proofs.ipynb


```python
from sympy import *
from IPython.display import display
%matplotlib inline
init_printing(use_latex=True)
```

# Rayleigh Quotient MarkII

We want to mix the last two functions we saw in the exercise, the shape associated with a load applied to the tip and the shape associated with a uniform distributed load.

We start by defining a number of variables that point to `Symbol` objects,

```python
z, h , r0, dr, t, E, rho, zeta = symbols('z H r_0 Delta t E rho zeta')
```

We define the tip-load function starting from the expression of the bending moment, just a linear function that is 0 for $z=H$... we integrate two times and we get the displacements bar the constants of integration that, on the other hand, happen to be both equal to zero due to clamped end at $z=0$, implying that $\psi_1(0)=0$ and $\psi'_1(0)=0$

```python
f12 = h-z
f11 = integrate(f12,z)
f10 = integrate(f11,z)
```

We have no scaling in place... we have to scale correctly our function by evaluating it for $z=H$

```python
scale_factor = f10.subs(z,h)
```

Dividing our shape function (and its derivatives) by this particular scale factor we have, of course, an unit value of the tip displacement.

```python
f10 /= scale_factor
f11 /= scale_factor
f12 /= scale_factor
f10, f11, f12, f10.subs(z,h)
```

We repeat the same procedure to compute the shape function for a constant distributed load, here the constraint on the bending moment is that both the moment and the shear are zero for $z=H$, so the non-normalized expression for $M_b\propto \psi_2''$ is

```python
f22 = h*h/2 - h*z + z*z/2
```

The rest of the derivation is the same

```python
f21 = integrate(f22,z)
f20 = integrate(f21,z)
scale_factor = f20.subs(z,h)
f20 /= scale_factor
f21 /= scale_factor
f22 /= scale_factor
f20, f21, f22, f20.subs(z,h)
```

To combine the two shapes in the _right_ way we write

$$\psi = \alpha\,\psi_1+(1-\alpha)\,\psi_2$$

so that $\psi(H)=1$, note that the shape function depends on one parameter, $\alpha$, and we can minimize the Rayleigh Quotient with respect to $\alpha$.

```python
a = symbols('alpha')
f0 = a*f10 + (1-a)*f20
f2 = diff(f0,z,2)
f0.expand().collect(z), f2.expand().collect(z), f0.subs(z,h)
```

Working with symbols we don't need to formally define a Python function, it suffices to bind a name to a symbolic expression.  That's done for the different variable quantities that model our problem and using these named expressions we can compute the denominator and the numerator of the Rayleigh Quotient.

```python
re = r0 - dr * z/h
ri = re - t
A = pi*(re**2-ri**2)
J = pi*(re**4-ri**4)/4
fm = rho*A*f0**2
fs = E*J*f2**2
mstar = 80000+integrate(fm,(z,0,h))
kstar = integrate(fs,(z,0,h))
```

Our problem is characterized by a set of numerical values for the different basic variables:

```python
values = {E:30000000000,
          h:32,
          rho:2500,
          t:Rational(1,4),
          r0:Rational(18,10),
          dr:Rational(6,10)}

values
```

We can substitute these values in the numerator and denominator of the RQ

```python
display(mstar.subs(values))
display(kstar.subs(values))
```

Let's look at the RQ as a function of $\alpha$, with successive refinements

```python
rq = (kstar/mstar).subs(values)
plot(rq, (a,-3,3));
```

```python
plot(rq, (a,1,3));
```

```python
plot(rq, (a,1.5,2.0));
```

Here we do the following:

   1. Derive the RQ and obtain a numerical function (rather than a symbolic expression) using the `lambdify` function.
   2. Using a root finder function (here `bisect` from the `scipy.optimize` collection) we find the location of the minimum of RQ.
   3. Display the location of the minimum.
   4. Display the shape function as a function of $\zeta=z/H$.
   5. Display the minimum value of RQ.

Note that the eigenvalue we have previously found, for $\psi\propto1-\cos\zeta\pi/2$ was $\omega^2= 66.259\,(\text{rad/s})^2$

```python
rqdiff = lambdify(a, rq.diff(a))
from scipy.optimize import bisect
a_0 = bisect(rqdiff, 1.6, 1.9)
display(a_0)
display(f0.expand().subs(a,a_0).subs(z,zeta*h))
rq.subs(a,a_0).evalf()
```

Oh, we have (re)discovered the Ritz method! and we have the better solution so far...

```python
# usual incantation
from IPython.display import HTML
HTML(open('00_custom.css').read())
```

<style>
@font-face {
    font-family: 'Charis SIL';
    src: url('fonts/CharisSILEur-R.woff') format('woff');
}
@font-face {
    font-family: 'Architects Daughter';
    font-style: normal;
    src: url(https://fonts.gstatic.com/s/architectsdaughter/v6/RXTgOOQ9AAtaVOHxx0IUBM3t7GjCYufj5TXV5VnA2p8.woff2) format('woff2');
    unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215, U+E0FF, U+EFFD, U+F000;
}
.prompt{display: None;}
div.cell{margin: auto;width:900px;}

div #notebook { /* centre the content */
    background: #fffaf0;
    margin: auto;
    padding: 0em;
    padding-top: 1em;
}
div #notebook_container {width: 960px!important;}
#notebook li { /* More space between bullet points */
    margin-top:0.2em;
}

div.text_cell_render{
    font-family: "Charis SIL", Cambria,  serif;
    line-height: 155%;
    font-size: 130%;
}
.CodeMirror{
    font-family: Consolas, monospace;
    font-size: 140%;
    background-color:#fcffff!important;
}

.output_area {
    font-family: Consolas,monospace;font-size: 120%;
    background-color:#fcffff!important;
    margin-top:0.8em;
}

div.output_latex{overflow:hidden}

h1 {
    font-family: 'Architects Daughter', serif;
    font-size:   48pt!important;
    text-align:  center;
    text-shadow: 4px 4px 4px #aaa;
    padding-bottom: 48pt;
}

.warning{color: rgb( 240, 20, 20 )}
</style>