# Perceptron

## Note to my future self

The important concepts in this notebook are based off
- of Section 3.5, The perceptron,
- of Chapter 3, Single-layer Networks,
- of Bishop's Neural Networks for Pattern Recognition (1995).

Tasks accomplished in this notebook:
- Part I: Train a perceptron using the pattern-by-pattern gradient-descent update rule (Equation 3.68, Page 100)
- Part II: Demonstrate the Perceptron convergence theorem (Section 3.5.3, Page 100) in action
    - Did not like Bishop's explanations
    - Preferred: http://web.mit.edu/course/other/i2course/www/vision_and_learning/perceptron_notes.pdf
- Part III: When data set is not linearly separable (Second paragraph, Page 103)
    - Method 1: gradually reduce learning, by analogy with the Robbins-Monro procedure
    - Method 2: the pocket algorithm
- Part IV: Connection with single-layer networks with adaptive weights (Second and third paragraph, Page 104)

## Imports

```python
from tqdm import tqdm_notebook
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
```

```python
norm = np.linalg.norm
```

## Part I - Train a Perceptron

Generate samples from two clusters, each with a standard deviation of 0.3.

```python
samps, labels = make_blobs(
    n_samples=250,
    n_features=2,
    centers=np.array([[-0.4, -0.4], [0.5, 0.5]]),
    cluster_std=0.24,
    center_box=(-10.0, 10.0),
    shuffle=True,
    random_state=324
)
samps = samps + 10
# shifting the data set a bit away from the origin can increase the number of steps
# it takes to converge; otherwise the perceptron converges too quickly, which is good
# in practice but bad for addition analysis in this notebook
```

Change 0 labels to -1 labels.

```python
labels[labels == 0] = -1  # in accordance with the threshold activation function (Equation 3.66, Page 99)
```

```python
labels[:10]  # sanity check
```

    array([ 1, -1, -1,  1,  1,  1,  1, -1, -1, -1])

"Defining an extra basis function $\phi_0$ whose activation is permanently set to +1, together with a corresponding bias parameteres $w_0$." (Last paragraph, Page 98)

```python
F1, F2 = np.s_[:,0], np.s_[:,1]  # a simple way to slice more elegantly
```

Visualize. There is one thing to look out for. "One of the difficulties with the perceptron learning rule  is that, if the data set happens not to be linearly separable, then the learning algorithm will never terminate. Furthermore, if we arbitrarily stop the learning process there is no guarantee that the weight vector found will generalize well for new data." (2nd Paragraph, Page 103) For our case, the data set is clearly linearly separable and hence does not suffer from this problem.

```python
plt.scatter(samps[F1], samps[F2], c=labels)
plt.show()
```

```python
samps_with_ones = np.hstack([
    samps,
    np.ones(len(samps)).reshape(len(samps), 1)  # \phi_0's
])
```

```python
samps_with_ones[:3]  # sanity check
```

    array([[10.335539  , 10.56417591,  1.        ],
           [10.08057497,  9.25581449,  1.        ],
           [ 9.59036198,  9.29635402,  1.        ]])

Now, we discuss the training procedure.

Define the Perceptron class.

Search for the smallest margin under the teacher weight (a strictly positive quantity):

```python
class Perceptron:

    def __init__(self):
        # initial guess
        # I chose a strange initial guess so that the perceptron does not converge quickly \
        # so that I could actually see the loss decreasing over a dozens of (not just a few) weight updates
        self.w_init = np.array([[-10.0], [8.0], [-7.0]])
        self.w = self.w_init.copy()
        self.w_histories = []
        self.num_updates = 0

    @staticmethod
    def step_func(input):
        """
        Activation function used by the Perceptron. (Page 99)
        """
        if input >= 0: return 1
        elif input < 0: return -1

    def linear_combination(self, samp_vec):
        """
        Lenght of the projection of sample vector onto the weight vector.
        Equivalent to the orthogonal distance from the decision-boundary hyperplane.
        """
        return float(self.w.T @ samp_vec)

    def discriminant(self, samp_vec):
        """
        Output of the Perceptron.
        """
        activation = self.step_func(
            self.linear_combination(samp_vec)
        )
        return activation

    def increment_weight(self, delta):

        self.w = self.w.copy() + delta
        self.num_updates += 1
        self.w_histories.append(self.w)

    def hyperplane(self, xs, w):
        ys = (- w[0]) / w[1] * xs - w[2] / w[1]
        return ys

    def hyperplane_histories(self, xs, num_recent=10, num_stride=1):
        hyperplanes = []
        for w in self.w_histories[-num_recent::num_stride]:
            hyperplanes.append(self.hyperplane(xs, w))
        return hyperplanes

#     @staticmethod
#     def cosine_angle(v1, v2):
#         return (v1.T @ v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

#     def numerator(self):
#         numerators = []
#         for r in range(self.r):
#             numerator = w_best.T @ self.w_init + (r+1) * smallest_delta
#             numerators.append(float(numerator))
#         return numerators

#     def denominator(self):
#         denominators = []
#         for r in range(self.r):
#             denominators.append(
#                 float(np.sqrt(np.linalg.norm(self.w_init) ** 2 + (r+1)))
#             )
#         return denominators

#     def w_norms(self):
#         return [float(np.linalg.norm(w)) for w in self.w_histories]
```

Train the perceptron. Re-compute loss after each weight update.

```python
p = Perceptron()
losses = []
num_samples_seen = 0

for i in tqdm_notebook(range(5000)):

    for samp, label in zip(samps_with_ones, labels):

        num_samples_seen += 1
        samp_vec = samp.reshape(3, 1)

        if p.discriminant(samp_vec) * label < 0:  # misclassified

            # multiplying by label is just a compact way to express:
            # if
            # p.discriminant(samp_vec) > 0 and label < 0
            # or
            # p.discriminant(samp_vec) < 0 and label > 0

            p.increment_weight(samp_vec * label)  # guaranteed to reduce loss on this example

            # ========== proof (Equation 3.69, Page 100) ==========
            # w_new.T @ samp_vec : interpreted as the new magnitude of mistake
            #
            # if p.discriminant(samp_vec) > 0 and label < 0:
            #     w_new = w_old - samp_vec
            #     reduces error because:
            #     -> w_new.T @ samp_vec
            #     -> = (w_old - sample_vec).T @ sample_vec
            #     -> = w_old @ sample_vec - sample_vec.T @ sample_vec (positive)
            #     -> < w_old @ sample_vec (old magnitude of mistake)
            #
            # elif p.discriminant(samp_vec) < 0 and label > 0:
            #     ... works in a similar way ...

            # loss computation only when w is updated

            loss = 0.0

            for samp, label in zip(samps_with_ones, labels):

                samp_vec = samp.reshape(3, 1)

                if p.discriminant(samp_vec) * label < 0:  # misclassified
                    loss -= p.linear_combination(samp_vec) * label

            losses.append(loss)
```

    HBox(children=(IntProgress(value=0, max=5000), HTML(value='')))

```python
preds = np.zeros(len(labels))
for i, s in enumerate(samps_with_ones):
    preds[i] = p.discriminant(s)
acc = np.mean(preds == labels)
print('Accuracy:', acc)
```

    Accuracy: 1.0

```python
fig = plt.figure(figsize=(24, 6))

# figure 1

fig.add_subplot(131)

plt.scatter(samps_with_ones[F1], samps_with_ones[F2], c=labels)

y_lower, y_upper = samps_with_ones[F2].min(), samps_with_ones[F2].max()
x_lower, x_upper = samps_with_ones[F1].min(), samps_with_ones[F1].max()

xs = np.array([x_lower, x_upper])

kNUM_RECENT = 5
kNUM_STRIDE = 1
hyperplane_histories = p.hyperplane_histories(xs, num_recent=kNUM_RECENT, num_stride=kNUM_STRIDE)
num_histories = len(p.w_histories)

for i, hyperplane in enumerate(hyperplane_histories):
    plt.plot(xs, hyperplane, label=f'After {num_histories - kNUM_RECENT + (i+1) * kNUM_STRIDE} iters')

plt.xlim(x_lower, x_upper)
plt.ylim(y_lower, y_upper)

plt.legend()
plt.title('Hyperplanes during training')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

# figure 2

fig.add_subplot(132)

plt.scatter(samps_with_ones[F1], samps_with_ones[F2], c=labels)

xs = np.arange(x_lower, x_upper, 0.01)
ys = np.arange(y_lower, y_upper, 0.01)

preds_matrix = np.zeros((len(ys), len(xs)))

for i, x in enumerate(xs):
    for j, y in enumerate(ys):
        preds_matrix[j][i] = p.linear_combination(np.array([[x], [y], [1.0]]))

plt.contour(xs, ys, preds_matrix, np.arange(np.floor(preds_matrix.min()), np.ceil(preds_matrix.max()), 1), cmap='coolwarm')
plt.colorbar()

argmax_min_y, argmax_min_x = np.where(preds_matrix == np.abs(preds_matrix).min())

plt.arrow(float(xs[argmax_min_x]), float(ys[argmax_min_y]), float(p.w[0]) / norm(p.w), float(p.w[1]) / norm(p.w), width=0.03, length_includes_head=False, color='black')

plt.xlim(x_lower, x_upper)
plt.ylim(y_lower, y_upper)

plt.title(r'Converged $\vec{w}/{||\vec{w}||}$ and $\vec{w}^T \vec{x}$ (bias included)')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

# figure 3

fig.add_subplot(133)

plt.plot(losses, linewidth=0.05)

is_converged = acc == 1.0
if is_converged:
    plt.title('Loss over weight update - converged')
else:
    plt.title('Loss over weight update - not converged')

plt.xlabel(f'Weight update')
plt.ylabel('Perceptron citerion (loss function)')

plt.show()
```

The white arrow shown in the middle plot is the normalized weight vector at convergence.

## Part II - Perceptron convergence theorem

Before we start, let's define a few quantities that will prove to be useful in the formal proof presented later.

### Perceptron learning algorithm (PLA)

PLA is essentially the rule for weight update.

$$
\begin{align}
\vec{w}_n &= \vec{w}_{n-1} + \eta y_n \vec{x}_n \\
\vartriangle \vec{w} \equiv \vec{w}_n - \vec{w}_{n-1} &= \eta y_n \vec{x}_n
\end{align}
$$

### Margin

$$\delta_i = \frac{ y_i {\vec{t}}^T \vec{x}_i} {||\vec{t}||}$$

- $\vec{t}$ is the teacher weight vector that can correctly classify all data points. Formally, this is writte as ($\vec{t}^T \vec{x}_i) y_i > 0, \forall i$.
- $\delta_i$ is the signed Euclidean distance from the data point $\vec{x}_i$ to the hyperplane defined by $\vec{t}$.

The smallest margin is defined by $\delta^{*} = \min_{\vec{x}}\delta > 0$, the minimum margin that can be obtained by tuning $\vec{x}$. This quantity is useful in the formal proof and hence useful in verifying the proof, so let us compute it here.

Let us use the converged weight vector `p.w` as the teacher weight, since it leads to correct classifications of all data points.

```python
t = p.w
```

```python
min_margin = np.inf
argmin_margin = None

for x, y in zip(samps_with_ones, labels):
    this_margin = y * (t.T @ x) / norm(t)
    if this_margin < min_margin:
        min_margin = this_margin
        argmin_margin = x
```

```python
print(min_margin)
argmin_margin
```

    [4.32322195e-05]

    array([10.26797868,  9.96379245,  1.        ])

Visualize `argmin_margin`, the data point that has the smallest margin.

```python
plt.figure(figsize=(5, 5))

plt.scatter(samps_with_ones[F1], samps_with_ones[F2], c=labels)

converged_hyperplane = p.hyperplane_histories(xs, num_recent=1, num_stride=1)[0]
plt.plot(xs, converged_hyperplane, label='Converged hyperplane')

plt.arrow(float(xs[argmax_min_x]), float(ys[argmax_min_y]), float(p.w[0]) / norm(p.w), float(p.w[1]) / norm(p.w), width=0.03, length_includes_head=False, color='black')

plt.scatter(argmin_margin[0], argmin_margin[1], label='Argmin margin')

plt.xlim(x_lower, x_upper)
plt.ylim(y_lower, y_upper)

plt.legend()
plt.title(r'Argmin margin')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.show()
```

### Proof

Let $\vec{w}_n$ be the student's weight vector after $n$ weight updates. It forms an angle $theta(n)$ with $\vec{t}$. The cosine of $\theta(n)$, $c(n)$ can be computed by:

$$c(n)=\frac{\vec{w}_n^T \vec{t}}{||\vec{w}_n|| \space ||\vec{t}||}$$

**Numerator**

See the equation for margin to understand step 2 to 3.
$$
\begin{align}
\vec{w}_n^T \vec{t} &= \vec{w}_{n-1}^T \vec{t} + \vartriangle \vec{w}_n \vec{t}\\
&= \vec{w}_{n-1}^T \vec{t} + \eta y_n \vec{w}_n^T \vec{t}\\
&= \vec{w}_{n-1}^T \vec{t} + \eta \delta_n ||\vec{t}||\\
\vec{w}_n^T \vec{t} &\geq \vec{w}_{n-1}^T \vec{t} + \eta \delta^{*} ||\vec{t}||
\end{align}
$$

Applying this inequality $n$ times, obtain:

$$\vec{w}_n^T \vec{t} \geq \vec{w}_{0}^T \vec{t} + n \eta \delta^{*} ||\vec{t}||$$

**Denominator**

Since $||\vec{t}||$ is fixed, only examine $||\vec{w_n}||$:

$$
\begin{align}
||\vec{w}_n||^2 &= ||\vec{w}_{n-1} + \vartriangle \vec{w}_n||^2 \\
&= ||\vec{w}_{n-1}||^2 + \eta^2 ||\vec{x}_n||^2 + 2(\vec{w}_{n-1}^T \vartriangle \vec{w}_n)
\end{align}
$$

$\vec{w}_{n-1}^T \vartriangle \vec{w}_n$ is negative because (1) $\vartriangle \vec{w}_n = \eta y_n \vec{x}_n$ and $\vec{w}_{n-1}^T \vec{x}_n y_n < 0$ since a mistake was made.

$$
\begin{align}
||\vec{w}_n||^2 &\leq||\vec{w}_{n-1}||^2 + \eta^2 ||\vec{x}_n||^2 \\
&\leq||\vec{w}_{n-1}||^2 + \eta^2 \max_n ||\vec{x}_n||^2 \\
&\leq||\vec{w}_{n-1}||^2 + \eta^2 D^2
\end{align}
$$

where $D = \max_n ||\vec{x}_n||^2 $.

We can easily compute $D$ by searching for the $\vec{x}_n$ that maximizes $\max_n ||\vec{x}_n||^2$:

```python
D = -np.inf
for s in samps_with_ones:
    squared_norm = norm(s) ** 2
    if squared_norm > D: D = squared_norm
```

```python
squared_norm
```

    187.3688106141413

Applying this inequality $n$ times, obtain:

$$||\vec{w}_n||^2 \leq ||\vec{w}_{0}||^2 + n \eta^2 D^2$$
$$||\vec{w}_n|| \leq \sqrt{||\vec{w}_{0}||^2 + n \eta^2 D^2}$$

**Putting everything together**

$$c(n) \geq \frac{\vec{w}_{0}^T \vec{t} + n \eta \delta^{*} ||\vec{t}||}{\sqrt{||\vec{w}_{0}||^2 + n \eta^2 D^2}}$$

As $n$ approaches $\infty$, the numerator grows linearly while the denominator grows only by $n^{1/2}$. Therefore, it seems that, as $n$ approaches $\infty$, $c(n)$ approaches $\infty$. However, since $c(n)$ is bounded above by $1$ as a cosine function, $n$ can't grow indefinitely. Therefore, the perceptron converges in a finite number of steps.

**In terms of angle**

Since the inverse cosine function is an monotonically decreasing function defined over domain $[-1, 1]$, the inequality reverses when it is applied to both sides of the inequality.

$$A(n) \leq \arccos ( \frac{\vec{w}_{0}^T \vec{t} + n \eta \delta^{*} ||\vec{t}||}{\sqrt{||\vec{w}_{0}||^2 + n \eta^2 D^2}} ) $$

**Disclaimer**

The derived inequality above does not say when $n$ terminates, nor does it say how $A(n)$ would fluctuate during the perceptron training process.

**Compute upper bound of $A(n)$ as $n$ increases from $0$ to $10^{14}$**

```python
def upper_bound_A(n):
    numerator = p.w_init.T @ t + n * min_margin * norm(t)
    denominator = np.sqrt(norm(p.w_init) ** 2 + n * (D ** 2)) * norm(t)

    if numerator / denominator > 1.0:  # convergence (numerator / denominator <= c(n), not supposed to be greater than 1)
        return 0.0

    if np.abs(float(numerator / denominator) - 1) < 0.000001:  # account for numerical error
        return 0.0

    else:  # before convergence
        return float(np.rad2deg(np.arccos(numerator / denominator)))
```

```python
n_max = 10 ** 14
n_step = 10 ** 12
ns = np.arange(0, n_max, n_step)

upper_bound_As = [upper_bound_A(n+1) for n in ns]

plt.plot(ns, upper_bound_As, label=r'Upper bound on $A(n)$')

plt.grid()

plt.xlabel('Weight update')
plt.ylabel('Angle')
plt.title(r'Upper bound on $A(n)$ VS weight update')

plt.show()
```

However, as mentioned in the disclaimer, the upper bound does not necessarily converge to zero over a small number of $n$. In fact, as shown above, it converges at around $0.3 * 10^{14}$. As shown below, $A(n)$ has converged to zero much much earlier than this number.

```python
def deg_between(v1, v2):
    cosine_angle = (v1.T @ v2) / (norm(v1) * norm(v2))

    # ========== Rationale for the if statement below ==========
    # When v1 is equal to v2, we expect that cosine_angle
    # is equal to 1. However, it turns out that it is larger
    # than 1 by a tiny tiny bit. This causes np.arccos to raise
    # an error because it can only handle values within [-1, 1].

    if np.abs(float(cosine_angle) - 1) < 0.000001:  # account for numerical error
        return 0.0
    else:
        return float(np.rad2deg(np.arccos(cosine_angle)))
```

```python
actual_As = [deg_between(w, t) for w in p.w_histories]
```

```python
plt.plot(actual_As, label=r'$A(n)$')

plt.grid()

plt.legend()
plt.xlabel('Weight update')
plt.ylabel('Angle')
plt.title(r'Angle between $\vec{w}$ and $\vec{t}$ VS weight update')

plt.show()
```

## Part III: Non-linearly separable data

**Quote I'm pondering**
> "The issue of linear separability is a somewhat artificial one, and it is more important to develop learning algorithms which can be expected to give good performance across a wide range of problems, even if this means sacrificing the guarantee of perfect classification for linearly separable problems." (Second paragraph, Page 103)

### Load data

First, load an example data set that is not linearly separable - but almost linearly separable, i.e., we can still use a linear decision boundary to achieve a good but not perfect accuracy. The question is whether the PLA can find this decision boundary. Since this section is purely experimental, no proofs are involved.

```python
df = pd.read_csv('data_not_linearly_separable.csv')
df.head()
```

      x
      y
      Class

      0
      continuous
      continuous
      discrete

      1
      NaN
      NaN
      class

      2
      0.3004432888935764
      0.725100270232804
      C1

      3
      0.3975428521219788
      0.8118280265580967
      C1

      4
      0.340919199036685
      0.8067692594520467
      C1

```python
samps_2 = df[['x', 'y']][2:].convert_objects(convert_numeric=True).to_numpy()
```

    /Users/yangzhihan/anaconda/envs/deeplearning_googlecloud/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: convert_objects is deprecated.  To re-infer data dtypes for object columns, use DataFrame.infer_objects()
    For all other conversions use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric.
      """Entry point for launching an IPython kernel.

```python
labels_2 = df['Class'][2:].to_numpy()
```

```python
labels_2[np.where(labels_2 == 'C1')[0]] = -1
labels_2[np.where(labels_2 == 'C2')[0]] = 1
```

```python
plt.figure(figsize=(5, 5))
plt.scatter(samps_2[F1], samps_2[F2], c=labels_2)
plt.title('A data set that is not linearly separable')
plt.xlabel('Feature 1'); plt.ylabel('Feature 2')
plt.show()
```

```python
samps_with_ones_2 = np.hstack([
    samps_2,
    np.ones(len(samps_2)).reshape(-1, 1)
])

samps_with_ones_2[:5]  # sanity check
```

    array([[0.30044329, 0.72510027, 1.        ],
           [0.39754285, 0.81182803, 1.        ],
           [0.3409192 , 0.80676926, 1.        ],
           [0.28515358, 0.81346378, 1.        ],
           [0.35402109, 0.73023822, 1.        ]])

### Method 1: PLA

```python

```

### Method 2: PLA + learning rate decay

```python

```

```python

```

### Method 3: PLA + pocket algorithm

```python

```

## References

- Introduction: The Perception by Haim Sompolinsky @ MIT

```python

```


# The Party Problem

There are $n$ drunk kids at a party that is presumably getting shut down by the fun police in Cambridge, MA. They (meaning the kids, presumably) grab their coats at random, and the problem is built around thinking about how many people wind up with the correct coat.

### Lame Interview Question 1: Every one finds their coat
The common version of this problem asks "what is the probability that all of the kids wind up with the right coat". This is much easier than the general case. Imagine the list of party guests as a sequence $(1,2,3,4,...,n-1,n)$ and the coats they grab as a random permutation of that sequence, such as $\alpha = (6,1,40,21,9,...)_n$. There are $n!$ such permutations, and they are all equally likely. The probability of all guests getting the correct coat is the probability that the permutation happens to be the single correct one, i.e. $p(\alpha = (1,2,3,4,...,n-1,n)_n)$, which is the probality of one particular permutation, i.e. $p(\alpha = (1,2,3,4,...,n-1,n)_n) = \frac{1}{n!}$.

### Lame Interview Question 2: At least r people find their coat
This is still pretty easy, because the permutations that are correct are easily counted. If at least $r$ coats are correctly assigned, then there are $\left(\begin{array}{l}n\\r\end{array}\right)$ ways of choosing wich of the $r$ people wind up with the right coat. Then, while the location of $r$ indices in the sequence is fixed, $n-r$ indices can be assigned arbitrarily.

The number of permutations where at least $r$ are assigned correctly are then:

\begin{equation}
\left(
\begin{array}{l}
n\\
r
\end{array}
\right)(n-r)!
\end{equation}

And the probability of at least $r$ people finding their coat is the number of permutations multiplied with the probability of an individual permutation (that is, $\frac{1}{n!}$):

\begin{equation}
p(\mathrm{\#\ correct} \geq r)=\left(
\begin{array}{l}
n\\
r
\end{array}
\right)\frac{(n-r)!}{n!} = \frac{1}{r!}
\end{equation}

Where $r\leq n$.

### Not lame: Exactly r people find their coat
So, then, what's the probability that exactly nobody finds their coat? What's the probability that 2 people find their coat but nobody else does? What's the probability that $r$ out of $n$ people find their coat? After thinking quickly on your feet for two seconds, you realize that the answer is obviously:

\begin{equation}
p(\mathrm{\#\ correct} = r) = \frac{1}{r!}\sum_{s=0}^{n-r} \frac{(-1)^s}{s!}
\end{equation}

The interviewer grunts ambiguously. They never call you back. You never find out why. You can't sleep. You can't eat. You become an anarchist and you declare war on the system.

The first two versions here are what I've come across in interview prep-type materials. I find them a bit annoying, because they represent particular cases that are much simpler than the general case. Applicants in the habit of studying stupid interview questions are rewarded because those answers are easily memorized (false positive). Applicants who intuit the complexity of the general problem, and who don't know the answer beforehand, might become overwhelmed during an interview and fail (false-ish negative).

/ rant

My approach here follows the extraordinary lecture notes https://mast.queensu.ca/~stat455/ by Glen Takahara at the University of Queensland.

As so often, indicator variables help:

\begin{equation}
I_A = \begin{array}{l} 1 \mathrm{\ if\ outcome\ in\ A}\\0 \mathrm{\ if\ outcome\ in\ A^c} \end{array}
\end{equation}

Which means that:
\begin{equation}
\mathbb{E}(I_A) = 1\times p(A) + 0\times p(A^c)= p(A)
\end{equation}

To indicator variables for different events can also be combined:
\begin{equation}
I_{A\cap B\cap C\cap ...} = I_A I_B I_C ...
\end{equation}

And indicator variables for the complement can be constructed trivially:
\begin{equation}
I_{A^c} = 1 - I_A
\end{equation}

Let $A_i$ be the region of state space in which guest $i$ grabbed the right coat. Let's say a *particular* subset $\{i\}_r$ of $r$ guests grabs their correct coats (for example, $\{i\}_r = \{5,9,11,24,...\}_r)$, and that the set of remaining $n-r$ guests $\{j\}_{n-r}= \{1,2,3,...\}_n\setminus\{i\}_r$ grab the wrong coat. The area of state space that corresponds to this outcome is:

\begin{equation}
A_{\{i\}_r,\{j\}_{n-r}}=\bigcap_{i\in\{i\}_r}A_i\bigcap_{j\in\{j\}_{n-r}}A_j^c
\end{equation}

The event that the outcome lies within that region of configuration space can be described with an indicator function:

\begin{equation}
I_{A_{\{i\}_r,\{j\}_{n-r}}} = \prod_{\{i\}_r}I_{A_i}\prod_{\{j\}_{n-r}}(1-I_{A_j})
\end{equation}

And the probability of those *particular* $r$ people finding their coat is it's expectation value, $p(A_{\{i\}_r,\{j\}_{n-r}}) = \mathbb{E}(I_{A_{\{i\}_r,\{j\}_{n-r}}})$. Good stuff.

The expression above will consist of a bunch of products of indicator variables that describe whether a particular guest wound up with the correct coat. We know that the product of indicator variables corresponds to an indicator variable for the *intersection* of the corresponding subsets of the state space. Explicitly, if it is a product of $s$ indicator variables for some particular set of coats coat $\{k\}_s$:

\begin{equation}
\prod_{k} I_{A_k} = I_{\bigcap_{k}A_k}
\end{equation}

And $\mathbb{E}(I_{\bigcap_{k} A_k}) = p(\bigcap_{k} A_k)$ is the probability of a particular $s$ coats being picked up correctly, which is $(n-s)!/n!$. (There is no binomial factor, because it's one *specific* set of $s$ coats).

Products of the sort $\prod^n (1-x_i)$ can be expanded:

\begin{equation}
\prod^n (1-x_i) = \sum_{s=0}^n (-1)^s \sum_{1\leq i_1,...,i_s\leq n} x_{i_1}x_{i_2}...x_{i_s}
\end{equation}

The sum $\sum_{1\leq i_1,...,i_s\leq n}$ is over all possible sets of up to $s$ indices that can be drawn from $\{1,2,3,...,n\}$. A simple example with $n=3$:

\begin{equation}
(1-x_1)(1-x_2)(1-x_3) = \underbrace{1}_{s=0} - \underbrace{(x_1 + x_2 + x_3)}_{s=1} + \underbrace{(x_1x_2 + x_2x_3 + x_1x_3)}_{s=2} - \underbrace{(x_1x_2x_3)}_{s=3}
\end{equation}

The number of terms of order $s$ is the amount of ways that $s$ indices can be sampled from $n$ indices, $\left(\begin{array}{l}n\\s\end{array}\right)$.

Returning to the original problem, then:

\begin{equation}
\begin{array}{ll}
I_{A_{\{i\}_r,\{j\}_{n-r}}} &= \prod_{\{i\}_r}I_{A_i}\prod_{\{j\}_{n-r}}(1-I_{A_j})\\
&=\sum_{s=0}^{n-r} (-1)^s \underbrace{\sum_{n-r\leq j_1,...,j_s\leq n}}_{\mathrm{sum\ of\ }\left(\begin{array}{l}n-r\\s\end{array}\right)\mathrm{\ terms\ \ }} \underbrace{\prod_{\{i\}_r} I_{A_i}\prod_{\{j\}_s}I_{A_j}}_{\mathrm{product\ of\ r+s\ terms\ \ }}
\end{array}
\end{equation}

By linearity of expected value, and using the relationship of the expected value of indicator variables to their probabilities:

\begin{equation}
\begin{array}{l}
\mathbb{E}(I_{A_{\{i\}_r,\{j\}_{n-r}}}) &= \sum_{s=0}^{n-r} (-1)^s \underbrace{\sum_{n-r\leq j_1,...,j_s\leq n}}_{\mathrm{sum\ of\ }\left(\begin{array}{l}n-r\\s\end{array}\right)\mathrm{\ terms\ \ }} p\left(\underbrace{I_{\bigcap_{\{i\}_r}A_i\bigcap_{\{j\}_s}A_j}}_{\mathrm{r+s\ coats\ picked\ up\ correctly}}\right)\\
&=\sum_{s=0}^{n-r} (-1)^s \left(\begin{array}{c}n-r\\s\end{array}\right)\frac{(n-r-s)!}{n!}
\end{array}
\end{equation}

Now, this is already a pretty neat expression for the probability of a *particular* $r$ coats being picked up correctly (i.e. Joe, Mary, and "Hans" picked up the right coat). We don't really care which of the $r$ guests got lucky, though, so since there are $\left(\begin{array}{c}n\\r\end{array}\right)$ ways of $r$ coats having been picked out correctly, you sum over all of them:

\begin{equation}
\begin{array}{ll}
p(\mathrm{\#\ correct} = r) &= \left(\begin{array}{c}n\\r\end{array}\right) \sum_{s=0}^{n-r} (-1)^s \left(\begin{array}{c}n-r\\s\end{array}\right)\frac{(n-r-s)!}{n!}\\
&= \sum_{s=0}^{n-r} (-1)^s \frac{n!}{r!(n-r)!}\frac{(n-r)!}{s!(n-r-s)!}\frac{(n-r-s)!}{n!}\\
&= \frac{1}{r!}\sum_{s=0}^{n-r} \frac{(-1)^s}{s!}
\end{array}
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt

def party(n):
    """
    Simulate a party with n people who randomly pick their coats. Return number of correctly picked coats.
    """
    guests = list(range(n))
    coats = np.random.permutation(guests)
    return np.sum(guests == coats)

def factorial(x):
    if x == 0:
        return 1
    else:
        res = 1
        for i in range(1,x+1):
            res *= i
        return res

def p_r(r,n):
    return (1/factorial(r))*np.sum([(-1)**s / factorial(s) for s in range(n-r+1)])

n = 100 #rager
data = [party(n) for i in range(50000)]

rr = np.array(list(range(n)))
pr = [p_r(r,n) for r in rr]

plt.figure(figsize=(8,5))
_ = plt.plot(rr,pr,'d-',linewidth=2,color='navy')
_ = plt.hist(data,bins=rr,density=True,color='cornflowerblue')

plt.xlim([0,n/10])
plt.title('Simulated vs. Analytical Solution to the Party Problem (100 people)')
plt.legend('Analytical,Simulated'.split(','))
plt.xlabel('r people find their coats',fontsize=12)
plt.ylabel('p(#correct = r)',fontsize=12)

plt.savefig('./img/partyproblem.png',bbox='tight')
```

Probability - Party Problem with Indicator Variables.ipynb


# Calculus

Contains an overview of calculus.

## Common derivatives

The following derivatives must simply be memorised:

$$
\begin{align}
{\Large \text{Very common}}\\
\\\
\frac{d}{dx} [x^n] =\ & n \cdot x^{n-1}\\
\\\
\frac{d}{dx} [e^x] =\ & e^x\\
\\\
\frac{d}{dx} [sin\ x] =\ & cos\ x\\
\\\
\frac{d}{dx} [cos\ x] =\ & \text{-} sin\ x\\
\\\
\frac{d}{dx} [ln\ x] =\ & \frac{1}{x}\\
\\\
\\\
\\\
{\Large \text{Hyperbolic}}\\
\\\
\frac{d}{dx} [sinh\ x] =\ & cosh\ x\\
\\\
\frac{d}{dx} [cosh\ x] =\ & sinh\ x\\
\\\
\frac{d}{dx} [tanh\ x] =\ & \frac{1}{cosh^2\ x}\\
\\\
\end{align}
$$

## Differentiation rules

### Product rule

$$
\frac{d}{dx}(f \cdot g) = \frac{d}{dx}(f) \cdot g + f \cdot \frac{d}{dx}(g) \equiv f' g + f g'
$$

### Quotient rule

$$
\frac{d}{dx} \left ( \frac{f}{g} \right ) = \frac{f' g - f g'}{g^2}
$$

### Chain rule

$$
\frac{dy}{dx} = \frac{df}{dg} \frac{dg}{dx} = f'(g(x))\ g'(x)
$$

### Reciprocal rule

$$
\frac{dx}{dy} = \frac{1}{dy/dx}
$$

## Implicit differentiation

Sometimes not possible to solve for $y$ and then differentiate.

We can still do differentiation in this case by simply differentiating w.r.t. $x$ on all terms and then solve for $\frac{dy}{dx}$.

## Leibniz's formula

Gives _n_th derivative of a product of functions.

Looks like binomial expansion $(p+q)^n$:

$$
\begin{align}
\frac{d^n(f \cdot g)}{dx^n} &= \sum_{m=0}^{n} \begin{pmatrix} n \\ m \end{pmatrix}\ f^{(n - m)}\ g^{(m)}\\
\\\
                            &= f^{(n)} \cdot g^{(0)} + n f^{(n - 1)} \cdot g^{(1)} + \frac{n(n - 1)}{2!} f^{(n - 2)} \cdot g^{(2)} + \ldots + f^{(0)} \cdot g^{(n)}
\end{align}
$$


# Bernstein-Varzirani Algorithm
The Bernstein-Varzirani algorithm solves the following problem:
1. Given an n-bit secret number x
2. There is an oracle holding the secret key that one can query with a number y, and the oracle will return $x \bullet y$, the bit-wise dot product modulo 2 of the two numbers. For example, given a 4-bit secret number $x=13$ with binary representation 1101 and $y=15$ with binary representation 1111, then $x \bullet y =$ 3 mod 2, which is 1
3. The question is: how many oracle queries does one need to figure out x?

With the classical approach, if we query the oracle with 1, 2(10), 4(100) and 8(1000), then we can figure out each bit of the secret key and thus we need four queries. For a general n-bit number, we need n queries. Surprisingly, in the quantum computing paradigm, the Bernstein-Varzirani algorithm needs only one measurement, regardless of the size of the secret number

```python
import projectq
from projectq.backends import CircuitDrawer
from projectq.ops import H, Z, All, Measure, Barrier

from pathlib import Path
```

```python
n = 3 #No. of qubits
```

## The classical approach

```python
def query_oracle(y):
    """
    The oracle returns the bit-wise dot product modulo 2
    of the secret key and y
    """
    x = 6 #secret key
    s = bin(x & y).count('1') % 2

    return s
```

```python
b = ''.join([str(query_oracle(2**(k))) for k in reversed(range(n))])
print("Secret key is {} after {} tries".format(int(b, 2), n))
```

    Secret key is 6 after 3 tries

## The quantum computing approach
There are five steps in the Bernstein-Varzirani circuit construction:
1. Initialize all qubits to 0 (This is the default value anyway)
2. Apply Hadamard transform to all qubits
3. Implement the oracle
4. Apply Hadamard transform to all qubits
5. Measure the circuit

The workings of the Bernstein-Varzirani algorithm can only be understood by working through the mathematics. After step 2, the application of the Hadamard transform to all qubits results in
$$\begin{align} H^{\otimes n} |0\rangle = \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} | y \rangle \end{align}$$
where $N = 2^n$

The oracle seeks to implement the following operation in step 3:
$$\begin{align} \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} | y \rangle \rightarrow \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle \end{align}$$
The oracle effectively multiplies each superposition state $|y \rangle$ with -1 raised to the power of $x \bullet y$, the bit-wise inner product modulo 2. But why does the oracle implement this operation?

Using the identity (proof given below)
$$\begin{align} H^{\otimes n} |x\rangle = \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle \end{align}$$
If we further apply another Hadamard transform to all qubits in step 4, then $$ H^{\otimes n} H^{\otimes n} |x\rangle = H^{\otimes n} \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle $$
i.e. $$ |x\rangle = H^{\otimes n} \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle $$
since the Hadamard transform is symmetric and unitary (meaning that the transform is its own inverse). Thus measuring the circuit in step 5 will give the secret key.

Summarizing,
$$ H^{\otimes n} |0\rangle ^{\otimes n} \rightarrow \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} | y \rangle \rightarrow \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle \rightarrow H^{\otimes n} \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle = |x\rangle$$

### The oracle implementation
First note that in implementing
$$ \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle = \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{(x_{n-1} y_{n-1} \oplus \ldots \oplus x_0 y_0)} | y_{n-1}\ldots y_0\rangle $$
$$ = \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x_{n-1} y_{n-1}} (-1)^{x_{n-2} y_{n-2}} \ldots (-1)^{x_0 y_0}) | y_{n-1}\ldots y_0\rangle $$
the individual product ${x_{n-1} y_{n-1}, \ldots, x_0 y_0}$ only needs to be evaluated for bits in x that are 1.

For the bits in x that are 1, we only introduce a minus sign for the bits in y that are 1 i.e.
$ |0 \rangle \rightarrow |0 \rangle$ and $ |1 \rangle \rightarrow -|1 \rangle$. This is precisely what the Z gate does. Thus the oracle can be implemented as: if a bit in x is 1, apply the Z gate to the corresponding bit in y.

```python
def make_bv_circuit(engine, n):
    circuit = engine.allocate_qureg(n)
    All(H) | circuit
    Barrier | circuit

    # Oracle
    x = 6  # secret key
    for i in range(n):
        if x & 1:  # Only apply Z if the current bit of the secret key x is 1
            Z | circuit[i]
        x >>= 1  # Move the next bit to the 1 position
    Barrier | circuit

    All(H) | circuit
    All(Measure) | circuit
    engine.flush()

    return circuit
```

## Draw circuit diagram

```python
drawing_engine = CircuitDrawer()
main_engine = projectq.MainEngine(drawing_engine)
```

```python
circuit = make_bv_circuit(main_engine, n)
```

```python
p = Path('diagram')
if not p.exists(): #if the diagram directory doesn't exist, create it
    p.mkdir()
with open('diagram/bv.tex', 'w') as f:
    latex = drawing_engine.get_latex() #get circuit diagram as latex
    f.write(latex)
#Change the pdf scale to 1.8 from 0.8 to have better visual effect
!sed -i 's@tikzpicture\}\[scale=0.8@tikzpicture\}\[scale=1.8@g' diagram/bv.tex
!cd diagram; pdflatex bv.tex  > /dev/null #convert tex to latex, piping to /dev/null to silent output

#Wand package needed to convert pdf to image
from wand.image import Image as WImage
img = WImage(filename='diagram/bv.pdf')
img
```

## Run circuit

```python
main_engine = projectq.MainEngine()
circuit = make_bv_circuit(main_engine, n)
measurement = ''.join(str(int(c)) for c in reversed(circuit))
print('Measurement outcome:', measurement)
print('Secret key:', int(measurement, 2))
```

    Measurement outcome: 110
    Secret key: 6

## Proof of the key identity

$$\begin{align} H^{\otimes n} |x\rangle = \frac{1}{\sqrt{N}} \sum_{y=0}^{N-1} (-1)^{x \bullet y} | y \rangle \end{align}$$ where $ N = 2^n $ and the $ \bullet$ operator denotes bitwise dot product modulo 2 i.e. $ x \bullet y = x_0y_0 \oplus x_1y_1 \oplus \ldots x_{N-1}y_{N-1}$ where $\oplus$ denotes the sum modulo 2 operation or the XOR operation. We prove this identity by induction.

Base case, n = 1:
$$ H |x_0\rangle = \frac{1}{\sqrt{2}} (|0\rangle + (-1)^{x_0}|1\rangle) = \frac{1}{\sqrt{2}} \sum_{y=0}^{1} (-1)^{x_0y} | y \rangle$$

Assuming n-1 (n > 1) is true i.e. $$ H^{\otimes (n-1)} |x_{n-2} \ldots x_0\rangle = \frac{1}{\sqrt{2^{n-1}}} \sum_{y=0}^{2^{n-1} - 1} (-1)^{(x_{n-2} \ldots x_0) \bullet (y_{n-2} \ldots y_0)} | y \rangle$$

$$ = \frac{1}{\sqrt{2^{n-1}}} \sum_{y=0}^{2^{n-1} - 1} (-1)^{(x_{n-2} y_{n-2} \oplus \ldots \oplus x_0 y_0)} | y \rangle$$

where $y = y_{n-2} \ldots y_0$ is the binary representation

Then $$ H^{\otimes n} |x_{n-1} \ldots x_0 \rangle =  \frac{1}{\sqrt{2}} (|0\rangle + (-1)^{x_{n-1}}|1\rangle) \times \frac{1}{\sqrt{2^{n-1}}} \sum_{y=0}^{2^{n-1} - 1} (-1)^{(x_{n-2} y_{n-2} \oplus \ldots \oplus x_0 y_0)} | y \rangle $$

$$ \begin{align} = \frac{1}{\sqrt{2^n}} \sum_{y=0}^{2^{n-1} - 1} (-1)^{(x_{n-2} y_{n-2} \oplus \ldots \oplus x_0 y_0)} |0\rangle| y \rangle + \frac{1}{\sqrt{2^n}} \sum_{y=0}^{2^{n-1} - 1} (-1)^{x_{n-1}} (-1)^{(x_{n-2} y_{n-2} \oplus \ldots \oplus x_0 y_0)} |1\rangle| y \rangle \end{align} $$

$$ \begin{align} = \frac{1}{\sqrt{2^n}} \sum_{y=0}^{2^{n-1} - 1} (-1)^{(x_{n-1}0 \oplus x_{n-2}y_{n-2} \oplus \ldots \oplus x_0y_0)} | y \rangle + \frac{1}{\sqrt{2^n}} \sum_{y=2^{n-1}}^{2^{n} - 1} (-1)^{(x_{n-1}1 \oplus x_{n-2}y_{n-2}\oplus \ldots \oplus x_0y_0)} | y \rangle \end{align}$$

$$ = \frac{1}{\sqrt{2^n}} \sum_{y=0}^{2^{n-1} - 1} (-1)^{(x_{n-1}x_{n-2} \ldots x_0) \bullet (0y_{n-2} \ldots y_0)} | y \rangle + \frac{1}{\sqrt{2^n}} \sum_{y=2^{n-1}}^{2^{n} - 1} (-1)^{(x_{n-1} \ldots x_0) \bullet (1y_{{n}-2} \ldots y_0)} | y \rangle $$

$$ = \frac{1}{\sqrt{2^{n}}} \sum_{y=0}^{2^{n} - 1} (-1)^{(x_{n-1} \ldots x_0) \bullet (y_{n-1} \ldots y_0)} | y \rangle $$

```python

```


# Lecture 12:  Fourier Series

## Background

A summation of sines and cosines can be used to approximate periodic functions.  The Fourier Series is one such series and is given by:

$$
f(x) = \frac{a_0}{2} + \sum_{n=1}^{\infty} a_n cos(nx) + \sum_{m=1}^{\infty} b_m sin(mx)
$$

The coefficients are related to the original function by:

$$
a_n = \frac{1}{\pi} \int^{2\pi}_0 f(s) \cos ns \; ds, \;\; n = 0,1,2,...
$$

and

$$
b_m = \frac{1}{\pi} \int^{2\pi}_0 f(s) \sin ms \; ds, \;\; m = 1,2,...
$$

This series with coefficients determined by the integrals may be used in the solution of [ordinary differential](https://en.wikipedia.org/wiki/Ordinary_differential_equation) and [partial differential equations](https://en.wikipedia.org/wiki/Partial_differential_equation).  In materials engineering you will sometimes see diffusion problems use series solutions to describe the evolution of a concentration field where there is a factor composed of an infinite series and a factor containing an exponential in time.  Together they are selected to describe the diffusive evolution of a system.  A classic example of a [Fourier series](https://en.wikipedia.org/wiki/Fourier_series) in the solution to a diffusion problem is in Jackson and Hunt's paper on eutectic solidification.  In that paper the boundary condition was represented by a Fourier series to model the composition profile across eutectic lamellae.

## What Skills Will I Learn?

* A wave can be described by two parameters:  the frequency and amplitude.
* Arbitrary, periodic functions can be approximated from combinations of individual waves if the frequencies and amplitudes are chosen correctly.
* Sines and cosines are basis vectors and for appropriate definitions of the dot product, orthogonal in a particular (Fourier) space the same way the unit vectors $\hat{i}$, $\hat{j}$, and $\hat{k}$ are orthogonal in Eucledian space.
* A generalized inner (dot) product of functions can be used to compute the correct combinations of frequencies and amplitudes to approximate a function.

## What Steps Should I Take?

1. Compute Fourier coefficients using the inner product of functions.
1. Learn how to shift the functions represented to arbitrary center points and domain widths.
1. Demonstrate that Fourier basis vectors are orthogonal by showing the inner product is zero over some domain.
1. Use a Fourier series to represent a sawtooth wave.
1. Prepare a new notebook (not just modifications to this one) that describes your approach to computing the above items.

## A Sucessful Jupyter Notebook Will

* Demonstrate the student's capability to calculate Fourier Series approximations to functions chosen by the student.
* Identify the audience for which the work is intended;
* Run the code necessary to draw one of the plane groups;
* Provide a narrative and equations to explain why your approach is relevant to solving the problem;
* Provide references and citations to any others' work you use to complete the assignment;
* Be checked into your GitHub repository by the due date (one week from assignment).

A high quality communication provides an organized, logically progressing, blend of narrative, equations, and code that teaches the reader a particular topic or idea.  You will be assessed on:
* The functionality of the code (i.e. it should perform the task assigned).
* The narrative you present.  I should be able to read and learn from it.  Choose your audience wisely.
* The supporting equations and figures you choose to include.

If your notebook is just computer code your assignment will be marked incomplete.

## Reading and Reference

* Essential Mathematical Methods for Physicists, H. Weber and G. Arfken, Academic Press, 2003
* Advanced engineering Mathematics, E. Kreyszig, John wiley and Sons, 2010
* Numerical Recipes, W. Press, Cambridge University Press, 1986
* C. Hammond, The Basics of Crystallography and Diffraction, Oxford Science Publications, 4th ed.
* B. Gustafsson, Fundamentals of Scientific Computing, Springer, 2011
* S. Farlow, Partial Differential Equations for Scientists and Engineers, Dover, 1993

### Representations of a Wave
---

A wave:

*  is represented by a frequency and amplitude.
*  is periodic on some domain, usually 0 to 2$\pi$ but can also be $-\pi$ to $\pi$ or anything else.
*  can be summed in combination with other waves to construct more complex functions.

```python
# Note the form of the import statements.  Keep the namespaces from colliding.
%matplotlib inline
import numpy as np
import sympy as sp

def plotSine(amplitude=2.4, frequency=np.pi/3.0, npoints=200):
    """
    Plots a sine function with a user specified amplitude and frequency.

    Parameters
    ----------
    amplitude : amplitude of the sine wave.

    frequency : the frequency of the sine wave.

    npoints : the number of points to use when plotting the wave.

    Returns
    -------
    A plot.

    """

    import matplotlib.pyplot as plt
    import numpy as np

    t = np.linspace(0, 2*np.pi, npoints)
    f = amplitude*np.sin(2*np.pi*frequency*t)
    fname = r"$A(t) = A \sin(2 \pi f t)$"

    fig, ax = plt.subplots()
    ax.plot(t, f, label=fname)
    ax.legend(loc='upper right')
    ax.set_xlabel(r'$t$', fontsize=18)
    ax.set_ylabel(r'$A$', fontsize=18)
    ax.set_title('A Sine Wave');

    plt.show()

    return
```

```python
plotSine()
```

All the properties of the wave are specified with these three pieces of information:

* It is a sine wave
* It has amplitude 2.4
* It has frequency $\pi$/3

In the previous plot we know that the frequency of $2\pi/3$ and coefficient (amplitue) of $2.4$ were linked through the `sin` function.  So it isn't hard to extrapolate to a situation where we might have MANY functions each with their own amplitude.  We could also imagine having many `sin` functions each with a different frequency - so let us make a list of amplitudes and frequencies (numerically) that we can use for plotting.  The following histogram plots the amplitudes for each frequency.

```python
def plotPower(amplitudes=[0,0,1.0,2.0,0,0,0], period=2.0*np.pi, npoints=200):
    """
    Plots a power series and the associated function assuming that the amplitudes
    provided are equally divided over the period of 2\pi unless specified.  Can also
    change the number of points to represent the function if necessary.
    """
    import matplotlib.pyplot as plt
    import numpy as np

    fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,5))
    fig.subplots_adjust(bottom=0.2)

    frequencies = np.linspace(0, period, len(amplitudes))
    t = np.linspace(0, period, npoints)

    # Reminder: zip([1,2,3],[4,5,6]) --> [(1,4),(2,5),(3,6)]
    f = sum([amplitude*np.sin(2*np.pi*frequency*t) for (amplitude, frequency) in zip(amplitudes, frequencies)])

    ax[0].bar(frequencies, amplitudes)
    ax[0].set_xlabel(r'$f$', fontsize=12)
    ax[0].set_ylabel(r'$A$', fontsize=12)
    ax[0].set_title(r'Power Spectrum')

    ax[1].plot(t, f)
    ax[1].set_xlabel(r'$f$', fontsize=12)
    ax[1].set_ylabel(r'$A$', fontsize=12)
    ax[1].set_title(r'Constructed Function')

    plt.show()

    return
```

```python
plotPower()
```

```python
plotPower(amplitudes=[0,1,1,1,0.4,0,0])
```

The plot above is one common way of visualizing the amplitudes of each term in a series.  Each bar represents the amplitude of a particular frequency in the reconstructed function.

### A Vector Space and Dot Products
----

A vector is an element of a _vector space_.  A vector space is the set of all vectors having dimension N.

We are introduced to the Euclidian vectors $\hat{i}$, $\hat{j}$, and $\hat{k}$ in physical problems and we gain a physical intuition for orthogonality.  We also learn a mechanism for computing the [dot product](https://en.wikipedia.org/wiki/Dot_product) in Euclidian systems, but other generalizations are possible.  One such generalization is the dot product of functions.

This dot product of functions can be used to determine Fourier coefficients.

```python
t = sp.symbols('t')
sp.init_printing()

def signal(x):
    return (x*(2 - x)*(1 - x)**2)
```

```python
sp.plot(signal(t), (t,0,2));
```

Is there a way to approximate the function above?  For real functions, the dot product can be generalized by the inner product, defined as:

$$ < f(x) | g(x) > = \int_{-L}^{L} f(x) g(x) dx $$

If this quantity is zero, then the functions are orthogonal. If the functions are orthogonal then they form a function space and can be used to approximate other functions.

The dot product for vectors v and w in Euclidian space has a geometric interpretation:

$$
\mathbf{v} \cdot \mathbf{w} = |v||w| \cos{\theta}
$$

This scalar quantity tells you how much of the vector v points along w, i.e., the magnitude of a vector pointing in the direction of $\hat{v}$ you need to add to some other (mutually orthogonal) vectors in order to reproduce w as a summation. When generalized to functions we write:

$$
< f(x) | g(x) > = \int_{-L}^{L} f(x) g(x) dx
$$

This computes how much of function $f(x)$ is projected onto $g(x)$.  Using a point $x = a$, compute $f(a)$ and $g(a)$. $f(a)$ and $g(a)$ represent the height of each function above/below the x-axis, so a vector from (a, 0) to (a, f(a)) can be dotted with a vector from (a, 0) to (a, g(a)). They are necessarily parallel along the space that contains the x-axis, so their dot product is just the product of their magnitudes:  $f(a)$ times $g(a)$. Now, multiply this by dx to keep the contribution from position $x=a$ proportional to how many additional x-positions you'll do this for. Take this dot product over and over, at each x-position, always scaling by $dx$ to keep it all in proportion. The sum of these dot products is the projection of $f(x)$ onto $g(x)$ (or vice-versa).

### Interactive Visualization of the Dot Product of Functions
----

```python
import matplotlib as mpl
import matplotlib.pyplot as plt
from ipywidgets import interact, fixed

# Somehow we want to add this text to the plot...
# dot_prod_value = sp.integrate(sp.sin(2*x)*sp.sin(x), (x, 0, 2*sp.pi))

def npf(x):
    return np.sin(2*x)

def npg(x):
    return np.sin(x)

def spf(x):
    return sp.sin(2*x)

def spg(x):
    return sp.sin(x)

# Make ff and gg tuples of np/sp functions? - or we can lambdafy the sp functions.
def myfig(ff,gg,a):
    """
    This function's docstring explaining the function.
    """
    x = np.linspace(0, 2*np.pi, 100)
    y1 = ff(x)
    y2 = gg(x)
    y3 = ff(x)*gg(x)
    fig = plt.figure(figsize=(8,5))
    axes = fig.add_axes([0.1, 0.1, 0.8, 0.8])
    axes.plot(x, y1, 'r', label=r"$f(x)$")
    axes.arrow(a, 0, 0, ff(a), length_includes_head=True, head_length=0.1, head_width=0.1, color='r')
    axes.plot(x, y2, 'g', label=r"$g(x)$")
    axes.arrow(a, 0, 0, gg(a), length_includes_head=True, head_length=0.1, head_width=0.1, color='g')
    axes.plot(x, y3, 'b', label=r"$f(x) \cdot g(x)$")
    axes.arrow(a, 0, 0, ff(a)*gg(a), length_includes_head=True, head_length=0.1, head_width=0.1, color='b')
    axes.legend()
    axes.grid(True)
    plt.show()
    return
```

```python
interact(myfig, ff=fixed(npf), gg=fixed(npg), a=(0,np.pi*2,0.05))
```

    interactive(children=(FloatSlider(value=3.1, description='a', max=6.283185307179586, step=0.05), Output()), _d…

    <function __main__.myfig(ff, gg, a)>

Using `scipy` we can perform this and other integrations numerically.  Two examples are given for the following functions:

```python
from scipy import integrate
import numpy as np

def myfunc1(x):
    return np.sin(4*x)

def myfunc2(x):
    return np.sin(x)

def myfunc3(x):
    return myfunc1(x)*myfunc2(x)

def myfuncx2(x):
    return x**2
```

```python
[integrate.quad(myfuncx2, 0, 4), 4.0**3/3.0]
```

```python
integrate.quad(myfunc3, 0, 2*np.pi)
```

```python
import sympy as sp
sp.init_printing()

n, m = sp.symbols('n m', Integer=True)
x = sp.symbols('x')

def f(x):
    return sp.sin(n*x)

def g(x):
    return sp.sin(m*x)

# scope of variables in def is local.
def func_dot(f, g, lb, ub):
    return sp.integrate(f(x)*g(x), (x, lb, ub))

func_dot(f, g, 0, 2*sp.pi)
```

### DIY:  Demonstrate the Inner Product of Certain Functions are Zero
----

Identify the conditions under which the inner product of:

$$
<\sin{4x}, \sin{x}>
$$

and

$$
<\sin{nx}, \sin{mx}>
$$

are zero.

### The Fourier Series Definied on Arbitrary Ranges

This discussion is derived from Sean Mauch's open source Applied Mathematics textbook.  If $f(x)$ is defined over $c-L \leq x \leq c+L $ and $f(x+2L) = f(x)$ then $f(x)$ can be written as:

$$
f(x) = \frac{a_0}{2} + \sum_{n=1}^{\infty} a_n cos \left( \frac{n \pi (x+c)}{L} \right) + \sum_{m=1}^{\infty} b_m sin \left( \frac{m \pi (x+c)}{L} \right)
$$

and the coefficients:

$$ a_n = \langle f(x) | \cos \left( \frac{n\pi (x+c)}{L} \right) \rangle = \frac{1}{L}\int^{c+L}_{c-L} f(x) \cos \frac{n \pi (x+c)}{L} $$

and

$$ b_m = \langle f(x) | \sin \left( \frac{m\pi (x+c)}{L} \right) \rangle = \frac{1}{L}\int^{c+L}_{c-L} f(x) \sin \frac{m \pi (x+c)}{L} $$

Using our generalized dot product for functions as defined above we can compute the Fourier coefficients.  The code for this follows in functions `a_n_amplitudes` and `b_m_amplitudes`.

### Computing the Fourier Coefficients by Hand
----

Note:  These next couple of cells take a few seconds to run.

```python
import sympy as sp
import numpy as np

x = sp.symbols('x')
dum = sp.symbols('dum')
sp.init_printing()

lam = 2
center = 1

def signal(x):
    return (x*(2 - x)*(1 - x)**2)

def mySpecialFunction(x):
    return sp.sin(2*x)

def b_m_amplitudes(n, funToProject, center, lam):
    return (2/lam)*sp.integrate(funToProject(dum)*sp.sin(2*n*sp.pi*dum/lam), (dum,center-lam/2,center+lam/2))

def a_n_amplitudes(m, funToProject, center, lam):
    return (2/lam)*sp.integrate(funToProject(dum)*sp.cos(2*m*sp.pi*dum/lam), (dum,center-lam/2,center+lam/2))

def b_m_vectorspace_element(n, var, lam):
    return sp.sin(2*n*sp.pi*var/lam)

def a_n_vectorspace_element(m, var, lam):
    if m==0:
        return sp.Rational(1,2)
    elif m!=0:
        return sp.cos(2*m*sp.pi*var/lam)
```

```python
terms = 3
funToProject = signal

an_vectors = [a_n_vectorspace_element(n, x, lam) for n in range(terms)]
an_amplitudes = [a_n_amplitudes(n, funToProject, center, lam) for n in range(terms)]
bm_vectors = [b_m_vectorspace_element(m, x, lam) for m in range(terms)]
bm_amplitudes = [b_m_amplitudes(m, funToProject, center, lam) for m in range(terms)]
```

We use a list comprehension to collect the basis vectors and amplitudes into a useful data structure through the `zip` function.

```python
truncatedSeries = (sum([a*b for a,b in zip(an_vectors,an_amplitudes)])
                   + sum([c*d for c,d in zip(bm_vectors,bm_amplitudes)]))
truncatedSeries
```

We can now plot this series and see the comparison of the signal (blue) and the series representation (red).  We can quantitatively describe the accuracy between the approximation and the function.

```python
p = sp.plot(signal(x), truncatedSeries, (x, 0, 2), show=False, title=r'Comparison of Series (Red) and Function (Blue)')
p[0].line_color = 'blue'
p[1].line_color = 'red'
p.show()
```

It is also possible to unpack the series above and look at the plot of each individual term's contribution to the approximate function.

```python
test = [c*d for c,d in zip(an_vectors,an_amplitudes)]

p2 = sp.plot(test[0],(x,0,2), show=False)

#[p2.append(sp.plot(test[i], (x,0,2), show=False)[0]) for i in range(1,5,1)]

[p2.append(sp.plot(i, (x,0,2), show=False)[0]) for i in test]

for i in range(1,terms,1):
    #p = sp.plot(test[i], (x,0,2), show=False)
    #p2.append(p[0])
    p2[i].line_color = 1.0-i/5.0,i/5.0,0.3

[p2.append(sp.plot(test[i], (x,0,2), show=False)[0])]

p2.show()
```

### Computing the Fourier Coefficients using Sympy
----

Here we use `sympy`'s `fourier_series` function to build a truncated series. We plot the series so that you can explore what happens when you change the number of terms.  The `interact` command creates a widget you can use to explore the effect of changing the nubmer of terms.

```python
import sympy as sp
import matplotlib.pyplot as plt
from ipywidgets import interact, fixed
```

```python
sp.fourier_series(x**2)
```

```python
sp.init_printing()

x = sp.symbols('x')

def myAwesomeFunction(a):
    return a

def fsMyFunc(terms, var):
    return sp.fourier_series(myAwesomeFunction(var), (var, -sp.pi, sp.pi)).truncate(n=terms)

def plotMyFunc(terms):
    p1 = sp.plot(fsMyFunc(terms,x),(x,-sp.pi, sp.pi), show=False, line_color='r')
    p2 = sp.plot(myAwesomeFunction(x), (x,-sp.pi,sp.pi), show=False, line_color='b')
    p2.append(p1[0])
    p2.show()
    return None

plt.rcParams['lines.linewidth'] = 3
plt.rcParams['figure.figsize'] = 8, 6
```

```python
interact(plotMyFunc, terms=(1,10,1));
```

    interactive(children=(IntSlider(value=5, description='terms', max=10, min=1), Output()), _dom_classes=('widget…

### Homework:  Series for a Sawtooth Wave
----

Using a Fourier series, represent the following periodic function:

$$f(x) = \left\{
\begin{array}{ll}
 x, & 0 \leq x \leq \pi, \\
 x-2\pi, & \pi \leq x \leq 2\pi,
\end{array}
\right.$$

### Homework:  Compute Your Own
----

Compute a Fourier series for a function of your choosing.  Think about the restrictions on the use of the Fourier series.


```python
import numpy
from matplotlib import pyplot as plot
import matplotlib.cm as cm

def iter_count(C, max_iter):
	X = C
	for n in range(max_iter):
		if abs(X) > 2.:
			return n
		X = X ** 2 + C
	return max_iter

N = 32
max_iter = 64
xmin, xmax, ymin, ymax = -2.2, .8, -1.5, 1.5
X = numpy.linspace(xmin, xmax, N)
Y = numpy.linspace(ymin, ymax, N)
Z = numpy.empty((N, N))

for i, y in enumerate(Y):
	for j, x in enumerate(X):
		Z[i, j] = iter_count(complex(x, y), max_iter)
                                #선택적인 파라미터인 extent는 2D배열에 저장한 데이터에 대한 좌표계를 지정한다.
plot.imshow(Z, cmap = cm.binary, extent = (xmin, xmax, ymin, ymax), interpolation = 'bicubic')
plot.show()

```

```python
import numpy
from matplotlib import pyplot as plot
import matplotlib.cm as cm

def iter_count(C, max_iter):
	X = C
	for n in range(max_iter):
		if abs(X) > 2.:
			return n
		X = X ** 2 + C
	return max_iter

N = 512
max_iter = 64
xmin, xmax, ymin, ymax = -2.2, .8, -1.5, 1.5
X = numpy.linspace(xmin, xmax, N)
Y = numpy.linspace(ymin, ymax, N)
Z = numpy.empty((N, N))

for i, y in enumerate(Y):
	for j, x in enumerate(X):
		Z[i, j] = iter_count(complex(x, y), max_iter)

plot.imshow(Z,
            cmap = cm.binary,
            interpolation = 'bicubic',
            extent=(xmin, xmax, ymin, ymax))

cb = plot.colorbar(orientation='horizontal', shrink=.75)
cb.set_label('iteration count')

plot.show()
```

```python
import numpy
from numpy.random import uniform, seed

from matplotlib import pyplot as plot
from matplotlib.mlab import griddata
import matplotlib.cm as cm

def iter_count(C, max_iter):
	X = C
	for n in range(max_iter):
		if abs(X) > 2.:
			return n
		X = X ** 2 + C
	return max_iter

max_iter = 64
xmin, xmax, ymin, ymax = -2.2, .8, -1.5, 1.5

sample_count = 2 ** 12
A = uniform(xmin, xmax, sample_count)
B = uniform(ymin, ymax, sample_count)
C = [iter_count(complex(a, b), max_iter) for a, b in zip(A, B)]

N = 512
X = numpy.linspace(xmin, xmax, N)
Y = numpy.linspace(ymin, ymax, N)
Z = griddata(A, B, C, X, Y, interp = 'linear')

plot.scatter(A, B, color = (0., 0., 0., .5), s = .5)
plot.imshow(Z,
            cmap = cm.binary,
            interpolation = 'bicubic',
            extent=(xmin, xmax, ymin, ymax))
plot.show()
```

```python
import numpy, sympy
from sympy.abc import x, y
from matplotlib import pyplot as plot
import matplotlib.patches as patches
import matplotlib.cm as cm

def cylinder_stream_function(U = 1, R = 1):
	r = sympy.sqrt(x ** 2 + y ** 2)
	theta = sympy.atan2(y, x)
	return U * (r - R ** 2 / r) * sympy.sin(theta)

def velocity_field(psi):
	u = sympy.lambdify((x, y), psi.diff(y), 'numpy')
	v = sympy.lambdify((x, y), -psi.diff(x), 'numpy')
	return u, v

psi = cylinder_stream_function()
U_func, V_func = velocity_field(psi)

xmin, xmax, ymin, ymax = -3, 3, -3, 3
Y, X = numpy.ogrid[ymin:ymax:128j, xmin:xmax:128j]
U, V = U_func(X, Y), V_func(X, Y)

M = (X ** 2 + Y ** 2) < 1.
U = numpy.ma.masked_array(U, mask = M)
V = numpy.ma.masked_array(V, mask = M)

shape = patches.Circle((0, 0), radius = 1., lw = 2., fc = 'w', ec = 'k', zorder = 0)
plot.gca().add_patch(shape)

plot.streamplot(X, Y, U, V, color = U ** 2 + V ** 2, cmap = cm.binary)

plot.axes().set_aspect('equal')
plot.show()

```

```python
import numpy
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plot

# Dataset generation
a, b, c = 10., 28., 8. / 3.
def lorenz_map(X, dt = 1e-2):
	X_dt = numpy.array([a * (X[1] - X[0]),
	                    X[0] * (b - X[2]) - X[1],
	                    X[0] * X[1] - c * X[2]])
	return X + dt * X_dt

points = numpy.zeros((2000, 3))
X = numpy.array([.1, .0, .0])
for i in range(points.shape[0]):
    points[i], X = X, lorenz_map(X)

# Plotting
fig = plot.figure()
ax = fig.gca(projection = '3d')

ax.set_xlabel('X axis')
ax.set_ylabel('Y axis')
ax.set_zlabel('Z axis')
ax.set_title('Lorenz Attractor a=%0.2f b=%0.2f c=%0.2f' % (a, b, c))

'''
ax.scatter(points[:, 0], points[:, 1],  points[:, 2],
           marker = 's',
           edgecolor = '.5',
           facecolor = '.5')
'''
ax.scatter(points[:, 0], points[:, 1],  points[:, 2],
           zdir = 'z',
           c = '.5')
plot.show()
```

```python
import numpy
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plot

a, b, c = 10., 28., 8. / 3.
def lorenz_map(X, dt = 1e-2):
	X_dt = numpy.array([a * (X[1] - X[0]),
	                    X[0] * (b - X[2]) - X[1],
	                    X[0] * X[1] - c * X[2]])
	return X + dt * X_dt

points = numpy.zeros((10000, 3))
X = numpy.array([.1, .0, .0])
for i in range(points.shape[0]):
	points[i], X = X, lorenz_map(X)

fig = plot.figure()
ax = fig.gca(projection = '3d')

ax.plot(points[:, 0], points[:, 1],  points[:, 2], c = 'k')
plot.show()

```

```python
import numpy
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plot

x = numpy.linspace(-3, 3, 256)
y = numpy.linspace(-3, 3, 256)
X, Y = numpy.meshgrid(x, y)
Z = numpy.sinc(numpy.sqrt(X ** 2 + Y ** 2))

fig = plot.figure()
ax = fig.gca(projection = '3d')
#ax.plot_surface(X, Y, Z, color = 'w')
#ax.plot_surface(X, Y, Z, cmap=cm.gray)
ax.plot_surface(X, Y, Z, cmap=cm.gray, linewidth=0, antialiased=False)

plot.show()

```

```python

```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
```

```python
# create function
x = Symbol('x')
f = x**2 + 8*sin(x)

# derivative
df = f.diff(x)

# lambdify both
f = lambdify(x, f)
df = lambdify(x, df)

# evaluate
pts = np.linspace(-5, 8)
plt.plot(pts, f(pts), label='f(x)')
plt.plot(pts, df(pts), label='df/dx')
plt.legend()
plt.show()
```

```python
max_iter = 100
momentum = 0.8
moment = [0]
with_momentum = [False, True]

strategy_steps = []

for use_momentum in with_momentum:
    lr = 0.05
    X0 = 10
    cur_iter = 0
    steps = [X0]
    while True:
        # sum last 10 momentum
        m = sum(moment[-10:])
        # get closer to a local min
        grad = lr * df(X0)
        if use_momentum:
            X1 = X0 + m - grad
        else:
            X1 = X0 - grad
        # save step
        steps.append(X1)
        # save grad
        moment.append(-grad)
        # previous moment are : 0.9*(grad N), 0.9*0.9*(grad N-1), etc.
        moment = [momentum*x for x in moment]
        # calculate step size
        step_size = X1 - X0
        # save new step
        X0 = X1
        if cur_iter > max_iter or abs(step_size) < 0.00001:
            print("Convergence done")
            break
        elif cur_iter%40 == 0:
            lr = lr*0.9
            print("Iteration {} has x value {}".format(cur_iter, X1))
        cur_iter = cur_iter + 1
    strategy_steps.append(steps)

print("The local minimum occurs at", X1)
```

    Iteration 0 has x value 9.33562861163058
    Iteration 40 has x value 3.5953497384665702
    Convergence done
    Iteration 0 has x value 9.335540104209164
    Iteration 40 has x value -1.2902349357717582
    Iteration 80 has x value -1.2513700135960928
    Convergence done
    The local minimum occurs at -1.2523427452482645

```python
fig = plt.figure(num=None, figsize=(8, 10), dpi=80, facecolor='w', edgecolor='k')
ax1 = fig.add_subplot(2, 1, 1)
steps = strategy_steps[0]
pts = np.linspace(-5, 8)
# change color to see GD progress
color = [item/255 for item in list(range(len(steps)))]
ax1.plot(pts, f(pts), label='f(x)')
ax1.scatter(steps, [f(x) for x in steps], label='GD steps', c=color)
ax1.scatter(steps[0], f(steps[0]), label='Start GD', c='red', s=50)
ax1.scatter(steps[-1], f(steps[-1]), label='End GD', c='yellow', s=50)
plt.title('Gradient Descent without momentum stays in local minimum')
ax1.legend()

ax2 = fig.add_subplot(2, 1, 2)
steps = strategy_steps[1]
pts = np.linspace(-5, 8)
# change color to see GD progress
color = [item/255 for item in list(range(len(steps)))]
ax2.plot(pts, f(pts), label='f(x)')
ax2.scatter(steps, [f(x) for x in steps], label='GD steps', c=color)
ax2.scatter(steps[0], f(steps[0]), label='Start GD', c='red', s=50)
ax2.scatter(steps[-1], f(steps[-1]), label='End GD', c='yellow', s=50)
plt.title('Gradient descent with momentum can avoid local minimum')
ax2.legend()

plt.show()
fig.savefig('../plots/2D_GD.png')
```

# Animation

```python
from tqdm import tqdm
from PIL import Image
def fig2img(fig):
    """Convert a Matplotlib figure to a PIL Image and return it"""
    import io
    buf = io.BytesIO()
    fig.savefig(buf)
    buf.seek(0)
    img = Image.open(buf)
    return img
```

```python
images = []
fig = plt.figure(num=None, figsize=(8, 10), dpi=80, facecolor='w', edgecolor='k')
ax1 = fig.add_subplot(2, 1, 1)
ax1.plot(pts, f(pts), c='grey')
ax2 = fig.add_subplot(2, 1, 2)
ax2.plot(pts, f(pts), c='grey')
steps1 = strategy_steps[0]
x1, y1, intensity1 = [], [], []

steps2 = strategy_steps[1]
x2, y2, intensity2 = [], [], []
for i, step in enumerate(tqdm(steps1)):
    pts = np.linspace(-5, 8)
    x1.append(step)
    y1.append(f(step))
    intensity1.append(i)
    ax1.scatter(x1, y1, cmap = 'viridis', c=intensity1)
    ax1.set_title('Gradient Descent without momentum stays in local minimum')
    images.append(fig2img(fig))

    step = steps2[i]
    pts = np.linspace(-5, 8)
    x2.append(step)
    y2.append(f(step))
    intensity2.append(i)
    ax2.scatter(x2, y2, cmap = 'viridis', c=intensity2)
    ax2.set_title('Gradient descent with momentum can avoid local minimum')
    images.append(fig2img(fig))
```

```python
images[0].save(fp='../plots/2D_GD.gif', format='GIF', append_images=images,
               save_all=True, duration=100, loop=0)
```

```python

```


# Local search

In unconstrained optimization, we wish to solve problems of the form
\begin{align}
\text{minimize} & & E(w)
\end{align}

* The local search algorithms have the form :
\begin{align}
w_0 & = \text{some initial value} \\
\text{for}\;\; & \tau = 1, 2,\dots \\
& w_\tau = w_{\tau-1} + g_\tau
\end{align}

Here, $g_\tau$ is a search direction. The loop is executed until a convergence condition is satisfied or
the maximum number of iterations is reached. The algorithm iteratively search for solutions that achieve a lower objective value by moving in the search direction.

# Gradient Descent
* Gradient descent is a popular local search method with the search direction chosen as the negative gradient direction:
\begin{align}
g_\tau & =  - \eta \nabla E(w_{\tau-1})
\end{align}

* When the gradient vanishes, i.e., $\nabla E(w) = 0$, the algorithm does not make any progress. Such points are also called fixed points.

* The iterates, under certain conditions, converge to the minimum $w^* = \arg\min_{w} E(w)$. A natural question here finding the conditions for guaranteed convergence to a fixed point and the rate -- how fast convergence happens as a function of iterations

* The parameter $\eta$ is called the *learning rate*, to be chosen depending on the problem. If the learning rate is not properly chosen, the algorithm can (and will) diverge.

* There is a well developed theory on how to choose $\eta$ adaptively to speed up covergence.

* Even for minimizing quadratic objective functions, or equivalently for solving linear systems, gradient descent can have a quite poor converge properties: it takes a lot of iterations to find the minimum. However, it is applicable as a practical method in many problems as it requires only the calculation of the gradient.

* For maximization problems
\begin{align}
\text{maximize} & & E(w)
\end{align}
we just move in the direction of the gradient so the search direction is $g_\tau = \eta \nabla E(w_{\tau-1})$

```python
%matplotlib inline
import numpy as np
import matplotlib as mpl
import matplotlib.pylab as plt

from ipywidgets import interact, interactive, fixed
import ipywidgets as widgets
from IPython.display import clear_output, display, HTML
from matplotlib import rc

mpl.rc('font',**{'size': 20, 'family':'sans-serif','sans-serif':['Helvetica']})
mpl.rc('text', usetex=True)

import time
import numpy as np

y = np.array([7.04, 7.95, 7.58, 7.81, 8.33, 7.96, 8.24, 8.26, 7.84, 6.82, 5.68])
x = np.array(np.linspace(-1,1,11))
N = len(x)

# Design matrix
#A = np.vstack((np.ones(N), x, x**2, x**3)).T
degree = 9
A = np.hstack([np.power(x.reshape(N,1),i) for i in range(degree+1)])

# Learning rate
eta = 0.001

# initial parameters
w = np.array(np.random.randn(degree+1))
W = []
Err = []
for epoch in range(50000):
    # Error
    err = y-A.dot(w)

    # Total error
    E = np.sum(err**2)/N

    # Gradient
    dE = -2.*A.T.dot(err)/N

    if epoch%100 == 0:
        #print(epoch,':',E)
        # print(w)
        W.append(w)
        Err.append(E)

    # Perfom one descent step
    w = w - eta*dE
```

The following cell demonstrates interactively the progress of plain gradient descent
and how its solution differs from the optimum found by solving the corresponding least squares problem.

```python

fig = plt.figure(figsize=(5,5))

left = -1.5
right = 1.5
xx = np.linspace(left,right,50)
AA = np.hstack((np.power(xx.reshape(len(xx),1),i) for i in range(degree+1)))

# Find best
A_orth, R = np.linalg.qr(A)
w_orth, res, rank, s = np.linalg.lstsq(A_orth, y)
w_star = np.linalg.solve(R, w_orth)
yy = AA.dot(w_star)

#ax.set_xlim((2,15))

#dots = plt.Line2D(x,y, linestyle='', markerfacecolor='b',marker='o', alpha=0.5, markersize=5)
#ax.add_line(dots)
plt.plot(x,y, linestyle='', markerfacecolor='b',marker='o', alpha=0.5, markersize=5)
plt.plot(xx, yy, linestyle=':', color='k', alpha=0.3)
ln = plt.Line2D(xdata=[], ydata=[], linestyle='-',linewidth=2)

ax = fig.gca()
ax.add_line(ln)
plt.close(fig)

ax.set_xlim((left,right))
ax.set_ylim((5,9))

def plot_gd(iteration=0):
    w = W[iteration]
    f = AA.dot(w)
    #print(w)
    ln.set_ydata(f)
    ln.set_xdata(xx)

    ax.set_title('$E = '+str(Err[iteration])+'$')

    display(fig)

res = interact(plot_gd, iteration=(0,len(W)-1))

```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Plotting the Error Surface

```python
%matplotlib inline

import scipy as sc
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pylab as plt

df_arac = pd.read_csv(u'data/arac.csv',sep=';')
#df_arac[['Year','Car']]

BaseYear = 1995
x = np.matrix(df_arac.Year[0:]).T-BaseYear
y = np.matrix(df_arac.Car[0:]).T/1000000.

plt.plot(x+BaseYear, y, 'o-')
plt.xlabel('Yil')
plt.ylabel('Araba (Millions)')

plt.show()
```

```python
from itertools import product

def Error_Surface(y, A, left=0, right=1, bottom=0, top=1, step=0.1):
    W0 = np.arange(left,right, step)
    W1 = np.arange(bottom,top, step)

    ErrSurf = np.zeros((len(W1),len(W0)))

    for i,j in product(range(len(W1)), range(len(W0))):
        e = y - A*np.matrix([W0[j], W1[i]]).T
        ErrSurf[i,j] = e.T*e/2

    return ErrSurf
```

```python
BaseYear = 1995
x = np.matrix(df_arac.Year[0:]).T-BaseYear
y = np.matrix(df_arac.Car[0:]).T/1000000.

# Setup the vandermonde matrix
N = len(x)
A = np.hstack((np.ones((N,1)), x))

left = -5
right = 15
bottom = -4
top = 6
step = 0.05
ErrSurf = Error_Surface(y, A, left=left, right=right, top=top, bottom=bottom)

plt.figure(figsize=(10,10))
#plt.imshow(ErrSurf, interpolation='nearest',
#           vmin=0, vmax=10000,origin='lower',
#           extent=(left,right,bottom,top), cmap='jet')

plt.contour(ErrSurf,
            vmin=0, vmax=10000,origin='lower', levels=np.linspace(100,5000,10),
            extent=(left,right,bottom,top), cmap='jet')

plt.xlabel('$w_0$')
plt.ylabel('$w_1$')
plt.title('Error Surface')
#plt.colorbar(orientation='horizontal')
plt.show()
```

### Animation of Gradient descent

```python
%matplotlib inline
import matplotlib.pylab as plt

import time
from IPython import display
import numpy as np

# Setup the Design matrix
N = len(x)
A = np.hstack((np.ones((N,1)), x))

# Starting point
w = np.matrix('[15; -6]')

# Number of iterations
EPOCH = 200

# Learning rate: The following is the largest possible fixed rate for this problem
#eta = 0.0001696
eta = 0.0001696

fig = plt.figure()
ax = fig.gca()

plt.plot(x+BaseYear, y, 'o-')

plt.xlabel('x')
plt.ylabel('y')

f = A.dot(w)
ln = plt.Line2D(xdata=x+BaseYear, ydata=f, linestyle='-',linewidth=2,color='red')
ax.add_line(ln)

for epoch in range(EPOCH):
    f = A.dot(w)
    err = y-f

    ln.set_xdata(x)
    ln.set_ydata(f)

    E = np.sum(err.T*err)/2
    dE = -A.T.dot(err)

#    if epoch%1 == 0:
#        print(epoch,':',E)
        # print(w)

    w = w - eta*dE

    ax.set_title(E)
    display.clear_output(wait=True)
    display.display(plt.gcf())
    time.sleep(0.1)
```

```python
# An implementation of Gradient Descent for solving linear a system

# Setup the Design matrix
N = len(x)
A = np.hstack((np.ones((N,1)), x))

# Starting point
w = np.matrix('[15; -6]')

# Number of iterations
EPOCH = 5000

# Learning rate: The following is the largest possible fixed rate for this problem
#eta = 0.00016
eta = 0.000161

Error = np.zeros((EPOCH))
W = np.zeros((2,EPOCH))

for tau in range(EPOCH):
    # Calculate the error
    e = y - A*w

    # Store the intermediate results
    W[0,tau] = w[0]
    W[1,tau] = w[1]
    Error[tau] = (e.T*e)/2

    # Compute the gradient descent step
    g = -A.T*e
    w = w - eta*g
    #print(w.T)

w_star = w
plt.figure(figsize=(8,8))
plt.imshow(ErrSurf, interpolation='nearest',
           vmin=0, vmax=1000,origin='lower',
           extent=(left,right,bottom,top))
plt.xlabel('w0')
plt.ylabel('w1')

ln = plt.Line2D(W[0,:300:1], W[1,:300:1], marker='o',markerfacecolor='w')
plt.gca().add_line(ln)

ln = plt.Line2D(w_star[0], w_star[1], marker='x',markerfacecolor='w')
plt.gca().add_line(ln)
plt.show()

plt.figure(figsize=(8,3))
plt.semilogy(Error)
plt.xlabel('Iteration tau')
plt.ylabel('Error')
plt.show()

```

* The illustration shows the convergence of GD with learning rate near the limit where the convergence is oscillatory.

* $\eta$, Learning rate is a parameter of the algorithm

* $w$, the variable are the parameters of the Model

* $y$: Targets

* $x$: Inputs,

# Accelerating Gradient descent

## Momentum methods, a.k.a., heavy ball

\begin{align}
p(\tau) & =  \nabla E(w(\tau-1)) + \beta p(\tau-1) \\
w(\tau) & =  w(\tau-1) - \alpha p(\tau)
\end{align}

When $\beta=0$, we recover gradient descent.

```python
%matplotlib inline
import matplotlib.pylab as plt

from notes_utilities import pnorm_ball_line

import time
from IPython import display
import numpy as np

#y = np.array([7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73])
y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
#y = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74])
x = np.array([10., 8., 13., 9., 11., 14., 6., 4., 12., 7., 5.])
N = len(x)

# Design matrix
A = np.vstack((np.ones(N), x)).T

w_best, E, rank, s = np.linalg.lstsq(A, y)
err = y-A.dot(w_best)
E_min = np.sum(err**2)/N

def inspect_momentum(alpha = 0.005, beta = 0.97):
    ln = pnorm_ball_line(mu=w_best, A=np.linalg.cholesky(np.linalg.inv(A.T.dot(A))),linewidth=1)
    ln2 = pnorm_ball_line(mu=w_best, A=4*np.linalg.cholesky(np.linalg.inv(A.T.dot(A))),linewidth=1)

    # initial parameters
    w0 = np.array([2., 1.])
    w = w0.copy()
    p = np.zeros(2)

    EPOCHS = 100
    W = np.zeros((2,EPOCHS))
    for epoch in range(EPOCHS):
        # Error
        err = y-A.dot(w)
        W[:,epoch] = w
        # Mean square error
        E = np.sum(err**2)/N

        # Gradient
        dE = -2.*A.T.dot(err)/N
        p = dE + beta*p

#        if epoch%10 == 1:
#            print(epoch,':',E)
            # print(w)

        # Perfom one descent step
        w = w - alpha*p

#    print(E_min)

    plt.plot(W[0,:],W[1,:],'.-b')
    plt.plot(w_best[0],w_best[1],'ro')
    plt.plot(w0[0],w0[1],'ko')
    plt.xlim((1.8,4.3))
    plt.ylim((0,1.2))
    plt.title('$\\alpha = $'+str(alpha)+' $\\beta = $'+str(beta))
    plt.gca().add_line(ln)
    plt.gca().add_line(ln2)
    plt.show()

inspect_momentum(alpha=0.0014088, beta=0.95)

```

```python
%matplotlib inline
from __future__ import print_function
from ipywidgets import interact, interactive, fixed
import ipywidgets as widgets
import matplotlib.pylab as plt
from IPython.display import clear_output, display, HTML

interact(inspect_momentum, alpha=(0, 0.02, 0.001), beta=(0, 0.99, 0.001))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    <function __main__.inspect_momentum>

# Advanced Material

https://distill.pub/2017/momentum/

http://blog.mrtz.org/2013/09/07/the-zen-of-gradient-descent.html

A great talk by Ben Recht:
https://simons.berkeley.edu/talks/ben-recht-2013-09-04

Backpropagation
http://www.offconvex.org/2016/12/20/backprop/

## Analysis of convergence of Gradient descent for a quadratic function

Recall that the error function we minimize is
$$
E(w) = \frac{1}{2} (y-Aw)^T(y-Aw) = \frac{1}{2}(y^\top y - 2 y^\top A w + w^\top A^\top A w)
$$

The gradient at the point $w$ will be denoted as $\nabla E(w) = g(w)$ where
$$g(w) = -A^\top (y - Aw) = A^\top A w - A^\top y$$

Moreover, the gradient at the minimum will vanish:
$$
g(w_\star) = 0
$$
Indeed, we can solve
$$0 = A^\top (Aw_\star - y)$$
as
$$w_\star =  (A^\top A)^{-1}A^\top y  $$
but this is not our point.

For a constant learning rate $\eta$, gradient descent executes the following iteration
$$
w_t = w_{t-1} - \eta g(w_{t-1}) = w_{t-1} - \eta A^\top (Aw_{t-1} - y)
$$

$$
w_t  =  (I - \eta A^\top A) w_{t-1} + \eta A^\top y
$$

This is a fixed point equation of form
$$
w_t  =  T(w_{t-1})
$$
where $T$ is an affine transformation.

We will assume that $T$ is a contraction, i.e. for any two different
parameters $w$ and $w'$ in the domain we have
$$
\| T(w) - T(w') \| \leq L_\eta \|w-w' \|
$$

where $L_\eta < 1$, then the distance shrinks. Hence the mapping converges to a fixed point (this is a consequence of a deeper result in analysis called the Brouwer fixed-point theorem (https://en.0wikipedia.org/wiki/Brouwer_fixed-point_theorem))

We will consider in particular the distance between the optimum and the current point $w(t)$

$$
\| T(w_t) - T(w_\star) \| \leq L_\eta \|w_t - w_\star \|
$$
But we have
$T(w_\star) = w_\star$ and $w_t = T(w_{t-1})$ so
$\|w_t - w_\star \| = \|T(w_{t-1}) - T(w_\star) \|$.

\begin{align}
\| T(w_t) - T(w_\star) \| & \leq L_\eta \|T(w_{t-1}) - T(w_\star) \| \\
& \leq L^2_\eta \|T(w_{t-2}) - T(w_\star) \| \\
\vdots \\
& \leq L^{t+1}_\eta \| w_{0} - w_\star \|
\end{align}

$$
T(w)   =  (I - \eta A^\top A) w + \eta A^\top y
$$

$$
T(w_\star)   =  (I - \eta A^\top A) w_\star + \eta A^\top y
$$

$$
\| T(w) - T(w') \| = \| (I - \eta A^\top A) (w-w') \| \leq \| I - \eta A^\top A \| \| w-w' \|
$$

When the norm of the matrix $\| I - \eta A^\top A \| < 1$ we have convergence. Here we take the operator norm, i.e., the magnitude of the largest eigenvalue.

Below, we plot the absolute value of the maximum eigenvalues of $I - \eta A^\top A$ as a function of $\eta$.

```python

left = 0.0000
right = 0.015

N = 1000
ETA = np.linspace(left,right,N)

def compute_largest_eig(ETA, A):

    LAM = np.zeros(N)
    D = A.shape[1]
    n = A.shape[0]

    for i,eta in enumerate(ETA):
        #print(eta)
        lam,v = np.linalg.eig(np.eye(D) - 2*eta*A.T.dot(A)/n)
        LAM[i] = np.max(np.abs(lam))

    return LAM

# This number is L_\eta
LAM = compute_largest_eig(ETA, A)

plt.plot(ETA, LAM)
#plt.plot(ETA, np.ones((N,1)))
#plt.gca().set_ylim([0.98, 1.02])
plt.ylim([0.997,1.01])
plt.xlabel('eta')
plt.ylabel('absolute value of the largest eigenvalue')
plt.show()

```

```python
plt.semilogy(ETA,LAM)
plt.ylim([0.997,1])
plt.show()
```

If $E$ is twice differentiable, contractivity means that $E$ is convex.

For $t>0$
\begin{align}
\|T(x + t \Delta x) - T(x) \| & \leq \rho \|t \Delta x\| \\
\frac{1}{t} \|T(x + t \Delta x) - T(x) \| &\leq \rho \|\Delta x\|
\end{align}

If we can show that $\rho< 1$, then $T$ is a contraction.

By definitions
$$
T(x)  = x - \alpha \nabla E(x)
$$

$$
T(x + t \Delta x)  = x + t \Delta x - \alpha \nabla E(x + t \Delta x)
$$

\begin{align}
\frac{1}{t} \|T(x + t \Delta x) - T(x) \| & = \frac{1}{t} \|x + t \Delta x - \alpha \nabla E(x + t \Delta x) - x + \alpha \nabla E(x) \| \\
& = \| \Delta x - \frac{\alpha}{t}  (\nabla E(x + t \Delta x) - \nabla E(x) ) \|  \\
\end{align}

As this relation holds for all $t$, we take the limit when $t\rightarrow 0^+$

\begin{align}
\| \Delta x - \alpha \nabla^2 E(x) \Delta x  \| & =  \| (I - \alpha \nabla^2 E(x)) \Delta x  \| \\
& \leq \| I - \alpha \nabla^2 E(x) \| \| \Delta x  \|
\end{align}

If we can choose $\alpha$ for all $\xi$ in the domain such that
$$
\| I - \alpha \nabla^2 E(\xi) \| \leq \rho < 1
$$
is satisfied, we have a sufficient condition for a contraction.

Lemma:

Assume that for $0 \leq \rho < 1$, $\alpha> 0$ and $U(\xi)$ is a symmetric matrix valued function for all $\xi \in \mathcal{D}$ and we have
$$
\| I - \alpha U(\xi) \| \leq \rho
$$
then $U = U(\xi)$ is positive semidefinite with $$\frac{1 - \rho}{\alpha} I \preceq U $$ for every $\xi$.

Proof:

$$
\|I - \alpha U \| = \sup_{x\neq 0} \frac{x^\top(I - \alpha U )x }{x^\top x} \leq \rho
$$

$$
x^\top(I - \alpha U )x \leq \rho x^\top x
$$

$$
(1- \rho) x^\top x  \leq \alpha x^\top U x
$$

This implies that for all $x$ we have
$$
0 \leq x^\top (U - \frac{1 - \rho}{\alpha} I) x
$$
In other words, the matrix $U - \frac{1 - \rho}{\alpha} I$ is positive semidefinite, or:

$$
\frac{1 - \rho}{\alpha} I \preceq U
$$
We now see that $\rho<1$ we have the guarantee that $U$ is positive semidefinite.

$$
T(x) = M x + b
$$

$$
\|T(x) - T(x_\star) \| = \|Mx + b - M x_\star + b \| = \| M(x-x_\star) \|
$$

By Schwarz inequality

$$
\|T(x) - T(x_\star) \| \leq \|M\| \|x-x_\star\|
$$
If $\|M\| < 1$, we have a contraction. Assume the existence of a fixed point $x_\star$ such that $x_\star = T(x_\star)$. (Does a fixed point always exist for a contraction?)

```python
# Try to fit with GD to the original data
BaseYear2 = 0
x2 = np.matrix(df_arac.Year[31:]).T-BaseYear2

# Setup the vandermonde matrix
N = len(x2)
A = np.hstack((np.ones((N,1)), x2))

left = -8
right = -7.55
N = 100
ETA = np.logspace(left,right,N)

LAM = compute_largest_eig(ETA, A)

plt.plot(ETA, LAM)
plt.plot(ETA, np.ones((N,1)))
plt.gca().set_ylim([0.98, 1.02])
plt.xlabel('eta')
plt.ylabel('absolute value of the largest eigenvalue')
plt.show()

```

Analysis of Momentum

\begin{align}
p(\tau) & =  \nabla E(w(\tau-1)) + \beta p(\tau-1) \\
w(\tau) & =  w(\tau-1) - \alpha p(\tau) \\
w(\tau-1) & =  w(\tau-2) - \alpha p(\tau-1) \\
\end{align}

\begin{align}
\left(\begin{array}{c}
w(\tau) \\
w(\tau-1)
\end{array}
\right)
& = &\left(\begin{array}{cc}
\cdot & \cdot \\
\cdot & \cdot
\end{array}
\right)
\left(\begin{array}{c}
w(\tau-1) \\
w(\tau-2)
\end{array}
\right)
\end{align}

\begin{align}
\frac{1}{\alpha}(w(\tau-1) - w(\tau))  & =  p(\tau) =  \nabla E(w(\tau-1)) + \beta \frac{1}{\alpha}(w(\tau-2) - w(\tau-1)) = \\
\frac{1}{\alpha}(w(\tau-2) - w(\tau-1))  & =  p(\tau-1) \\
\end{align}

\begin{align}
\frac{1}{\alpha}(w(\tau-1) - w(\tau))  & = \nabla E(w(\tau-1)) + \beta \frac{1}{\alpha}(w(\tau-2) - w(\tau-1)) \\
  w(\tau)  & = -\alpha \nabla E(w(\tau-1)) - \beta w(\tau-2) + (\beta+1)  w(\tau-1)
\end{align}

* Note that GD is sensetive to scaling of data
* For example, if we would not have shifted the $x$ axis our original data, GD might not have worked. The maximum eigenvalue is very close to $1$ for all $\eta$ upto numerical precision

```python
w = 7
alpha = 0.7
EPOCH = 100
W = []
for tau in range(EPOCH):
    W.append(w)
    w = w - alpha*(2*w - 4)

plt.plot(W)
plt.show()
```

```python
w = 7
alpha = 0.1
beta = 0.95
p = 0
EPOCH = 1000
W = []
for tau in range(EPOCH):
    W.append(w)
    p = (2*w - 4) + beta*p
    w = w - alpha*p

plt.plot(W)
plt.show()
```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
import numpy as np
```

# Infinite Perimeter

## Question

We have a rectangle with an area of $16m^2$, express the the perimeter of the rectangle as a function of one side.

## Observation

If the rectangle has an area of $16m^2$ we know that for some width and height we have $w\cdot h=16m^2$.

* $4m \cdot 4m = 16m^2$
* $8m \cdot 2m = 16m^2$
* $16m \cdot 1m = 16m^2$

The formula for the perimeter is $2(w+h)$.

If we have a given side $L_1$, we can find the missing side $L_2 = \dfrac{16m}{L_1}$, because $L_1 \cdot L_2 = 16m^2$.

## Formula

Putting everything together we get:

$$\begin{align}P &= 2(L_1 + L_2) \\ &= 2\left(L_1 + \frac{16m}{L_1}\right) \\ &= 2L_1 + 2\frac{16m}{L_1} \\ &= \frac{2L_1^2+32}{L_1} \end{align}$$

This gives the function for the perimeter of the rectangle for any side $L$:

$$P(L)=\frac{2L^2+32}{L}$$

where $L \not= 0$.

```python
def P(L):
    return (2*L**2+32)/L
```

If we try this for the sides we specified above we will see:

```python
print("Perimeter 4*4: %i" % (2*(4+4)))
print("Perimeter 2*8: %i" % (2*(2+8)))
print("Perimeter 1*16: %i" % (2*(1+16)))
```

    Perimeter 4*4: 16
    Perimeter 2*8: 20
    Perimeter 1*16: 34

We can give either of the sides, because $P(2)=P(8)=20$, etc. Also notice that this function is not injective or surjective.

```python
P(4)
```

    16.0

```python
P(8)
```

    20.0

```python
P(2)
```

    20.0

```python
P(16)
```

    34.0

```python
P(1)
```

    34.0

These results shows us that our expectation is correct.

## Plot

Plotting this function yields:

```python
points = np.arange(1,16,0.01)
plot(points, [P(x) for x in points], c='b', lw=2)
xlabel('Length of one side (m)')
ylabel('Perimeter (m)');
```

Upon seeing this graph, I did not found it intuitive at all. However, upon closer inspection, we can see that for example at $L=4$ we have a square. And being a square, this gives us the shortest perimeter, as we can see in the graph.

Another thing to notice is that when we have $L<1$, the perimeter starts to grow rapidly.

```python
P(0.5)
```

    65.0

```python
P(0.25)
```

    128.5

```python
P(0.01)
```

    3200.02

Thinking about it, it is a pretty obvious result, but not so obvious upon first sight. Because the rectangle is bounded with an area of $16m^2$, the other side of the rectangle has to become really big to meet this criteria if the length of the input side is very small.

Obviously this also works in the other direction:

```python
P(32) == P(0.5) == 65
```

    True

## Differentiation

To see how much the perimeter grows relatively to the input side length we give, we can differentiate the function with respect to $L$. Doing so will give:

$$\begin{align} P(L) &= \frac{2L^2+32}{L} \\ \frac{dP}{dL} &= \dfrac{L \cdot \dfrac{d(2L^2+32)}{dL} + (2L^2+32)\dfrac{d(L)}{dL}}{L^2} \\ \frac{dP}{dL} &= \frac{L\cdot(4L) + (2L^2+32)\cdot(1)}{L^2} \\ \frac{dP}{dL} &= \frac{4L^2+2L^2+32}{L^2} \\ \frac{dP}{dL} &= \frac{6L^2+32}{L^2}  \end{align}$$

If we implement $P'(L) = \dfrac{6L^2+32}{L^2}$:

```python
def PPrime(L):
    return (6*L**2+32)/L**2
```

And plot it with $P(L)$:

```python
points = np.arange(1,16,0.01)
plot(points, [P(x) for x in points], c='b', lw=2)
plot(points, [PPrime(x) for x in points], c='r', lw=2)
xlabel('Length of one side (m)')
ylabel('Perimeter (m)')
legend(['P(L)', "P'(L)"]);
```

And if we take a closer look at $0<L<4$ on a logarithmic scale:

```python
points = np.arange(0.001,4,0.001)
plot(points, [np.log(P(x)) for x in points], c='b', lw=2)
plot(points, [np.log(PPrime(x)) for x in points], c='r', lw=2)
legend(['P(L)', "P'(L)"]);
```

We can see that $P'(L)$ is causing $P(L)$ to grow really fast as we are starting to approach $L=0$. Now let's take the second derivative of $P(L)$:

$$\begin{align} \frac{dP}{dL} &= \frac{4L^2+32}{L^2} \\ \frac{d^2P}{dL^2} &= \dfrac{L^2 \dfrac{d(6L^2+32)}{dL}+(6L^2+32)\dfrac{d(L^2)}{dL}}{(L^2)^2} \\ \frac{d^2P}{dL^2} &= \frac{L^2(12L) + (6L^2+32)2L}{L^4} \\ \frac{d^2P}{dL^2} &= \frac{24L^3+62L}{L^4}\end{align}$$

If we implement $P''(L)=\dfrac{24L^3+62L}{L^4}$:

```python
def PDoublePrime(L):
    return (24*L**3+62*L)/L**4
```

And plot it with $P(L)$, and $P'(L)$:

```python
points = np.arange(0.001,4,0.001)
plot(points, [np.log(P(x)) for x in points], c='b', lw=2)
plot(points, [np.log(PPrime(x)) for x in points], c='r', lw=2)
plot(points, [np.log(PDoublePrime(x)) for x in points], c='green', lw=2)
grid()
legend(['P(L)', "P'(L)", "P''(L)"]);
```

We can see that $P''(L)$ is causing $P'(L)$ to grow really big. If we keep differentiating I think we will keep finding derivatives that are tending faster to infinity. This observation probably has a name but I am currently unaware of it.

## Infinite perimeter

We can see that when we let $L$ approach $0$, the perimeter becomes infinite. Thus, we come to the assertion that:

$$ \lim_{L\rightarrow 0} \frac{2L^2+32}{L} = \infty$$

which makes the perimeter infinitely long! The proof is rather easy if we look at the domain of $P(L)$, which is $\mathbb{R}\setminus {0}$, because it is a rational function it means that there is a vertical asymptote that tends to infinity as $L$ approaches $0$.

## We must go deeper

If we graph the functions over a greater interval, we can see this behaviour more clearly:

```python
points = np.arange(-20,20, 0.01)
interval = np.arange(0,4, 0.01)
axes = plt.gca()
axes.set_ylim([-50,50])
plot(points, [P(x) for x in points], c='b', lw=1)
plot(points, [PPrime(x) for x in points], c='r', lw=1)
plot(points, [PDoublePrime(x) for x in points], c='green', lw=1);
plot(interval, np.zeros(len(interval)), c='black', ls='dotted', lw=3)
legend(['P(L)', "P'(L)", "P''(L)", '0<L<4'])
grid()
```

A cool observation is that $F''(L)$ has a horizontal asymptote on the horizontal axis because the degree of the nominator is $3$ and the degree of the denominator is $4$ (the largest power is in the denominator).

Because of that the slope of $F'(L)$ will decrease at an infinitely small rate, and we can see that it's horizontal asymptote is at $6$ because we have $6L^2$ in the numerator and $L^2$ in the denominator.

This horizontal asymptote causes that the value of $F'(L)$ is approaching a constant decreasement, which affects the slope of $F(L)$ and gives the oblique asymptote in $F(L)$.

If we apply long polynomial division to $\dfrac{2L^2+32}{L}$ we will get $2L$, and plotting this with $P(L)$:

```python
points = np.arange(-20,20, 0.05)
plot(points, [P(x) for x in points], c='b', lw=1)
plot(points, [2*x for x in points], c='r', lw=2)
axes = plt.gca()
axes.set_ylim([-50,50]);
```

which is the oblique asymptote!

## Continuing differentiation

The third derivative $\dfrac{d^3P}{dL^3}=\dfrac{168L^6+320L^4}{L^8}$ gives the graph:

```python
def PTriplePrime(L):
    return (168*L**6+320*L**4)/L**8

points = np.arange(-20,20, 0.01)
interval = np.arange(0,4, 0.01)
axes = plt.gca()
axes.set_ylim([-5,100])
plot(points, [PTriplePrime(x) for x in points], c='b')
grid()
```