# Data Visualization with PCA

## The Challenges of High-dimensional Data

Once we decide to measure more than three features per input vector, it can become challenging to understand how a network is learning to solve such a problem since we can no longer generate a plot or visualization of the feature vector space to which the network is being exposed. One-, two-, or three-dimensional vectors are easy enough to plot, and we could even color the corresponding points based on their class assignments to see how groups of similar items are in similar parts of the vector space. If we can see straight boundaries (lines or planes) between the differently colored point clouds, then we would understand why a linear (i.e. single-layer) network might be capable of producing a reasonable solution to the problem at hand. Also, if we see regions where a single line/plane will not suffice for separating the different classes, then we might have reason to suspect that a linear network will fail to solve the problem, and probably attempt to use a multilayer network instead.

Given the advantages of visualizing such relationships, a common trick when exploring high-dimensional data sets is to **project** the high-dimensional data vectors onto a low-dimensional space (two or three dimensions) where we can see if such relationships exist. Such projection is _risky_ in the sense that we will be throwing information away to perform the projection (similar to how neural networks throw information away when performing regression or classification), and we may no longer see some important relationships in the low-dimensional projection of the data. However, it is sometimes possible that the projection **will** preserve relationships between the original data vectors that are important for making an accurate classification or regression while also enabling visualization.

In this assignment, we will explore a commonly-used **linear** projection known as Principal Component Analysis (PCA) to visualize some data sets and see how such projections might be useful for understanding why our single-layer networks were able to effectively learn the functions that these data sets represent. Since PCA is a linear method, it will be limited in its ability to produce useful _projections_ in a manner roughly analogous to how single-layer neural networks are limited in their ability to solve _linearly separable_ problems. Since we will be producing low-dimensional projections, relative to the original dimensionality of the vectors, this technique is also a form of **dimensionality reduction** in that the new two- or three- dimensional vectors that we produce will still share some of the relationships between one another that the higher-dimensional vectors possessed. This might even mean that we could use these new vectors as inputs for a neural network instead of the original, high-dimensional vectors. This could lead to a significant reduction in the number of neural units and connection weights in a network, and hence reduce its computation time. Also, some of the original features (or combinations of features) may just not be very useful for the problem at hand, and removing them would allow the network to focus on only the more relevant features in the data. In some cases (albeit rarely with PCA), this can even lead to superior performance of the trained neural network overall. While we will focus on two-dimensional projections in this assignment, PCA can be used to reduce the dimensionality of a given set of input vectors to any chosen number less than or equal to the original dimensionality. The smaller the dimensionality of the projection: the more information is being projected away. Thus, two-dimensional projections are often too low to be of any real use on many large data sets. However, some data sets might be reduced from millions of features to thousands, or thousands to hundreds, while still preserving the vast majority of the information that they encode. Such large reductions can make certain problems far more tractable to learn than they would otherwise be.

## Gathering the Iris data set

Let's start out by grabbing some data that we are already a little familiar with, and see if we can use PCA to better understand why a linear network can learn to solve this function effectively.

Let's start by importing some tools for the job...

```python
# For reading data sets from the web.
import pandas

# For lots of great things.
import numpy as np

# To make our plots.
import matplotlib.pyplot as plt
%matplotlib inline

# Because sympy and LaTeX make
# everything look wonderful!
import sympy as sp
sp.init_printing(use_latex=True)
from IPython.display import display

# We will use this to check our implementation...
from sklearn.decomposition import PCA

# We will grab another data set using Keras
# after we finish up with Iris...
import tensorflow.keras as keras
```

Now let's grab the Iris data set and start projecting!

```python
iris_data = np.array(
        pandas.read_table(
            "https://www.cs.mtsu.edu/~jphillips/courses/CSCI4850-5850/public/iris-data.txt",
            delim_whitespace=True,
            header=None))
```

```python
# Remember the data is composed of feature
# vectors AND class labels...
X = iris_data[:,0:4] # 0,1,2,3
Y = iris_data[:,4]   # 4

# Pretty-print with display()!
display(X.shape)
display(Y.shape)
display(sp.Matrix(np.unique(Y)).T)
```

The Iris data set consists of 150 four-dimensional feature vectors, each one assigned to one of three class labels (0,1,2) corresponding to an iris species.

We could potentially use **four** dimensions to plot and understand this data. Namely, we could make 3D plots using the first three dimensions, and sort the points along the fourth dimension so that we could play them in-sequence like a movie. However, that can still be tricky to visualize since we may miss some relationships between frames in our "movie" if they are far apart in time. Potentially more useful would be to plot the first three dimensions in one plot, then the last three dimensions in another, where the two plots now share the middle two dimensions. Still, if relationships between the first and fourth dimensions were the most important, we might not see them very clearly using this presentation.

Let's see if a PCA projection down to just two dimensions would be more effective.

To do this we will be using some linear algebra that we have already seen before, but we need to process the data just a little before we can use those tools.

First, we will _mean-center_ the values of _each_ feature in the data. That is, we will find the _mean_ value of the first feature across all examples in the data set, and then subtract the mean value from this feature for all examples. We will perform the same operation for all four features as well, so that each feature will have its mean value effectively set to zero. You can think of this as moving the entire set of data vectors so that the average of the data vectors now lies at the value zero in all four dimensions. In other words, it's a _translation_ operation on the original data. The relative distances between all of the points is maintained, so all of the relationships between the data vectors important for classification is maintained as well.

We will use a custom function for this, that we apply to each of the columns using the `apply_along_axis()` function:

```python
# Mean center a vector
def mean_center(x):
    return x - np.mean(x)

# Call this function for each column in the data (move along axis 0 or the rows)
Xcentered = np.apply_along_axis(mean_center,0,X)
```

Now that we have a mean-centered data matrix, we will use singular value decomposition to extract the left-singular vectors and singular-values of this matrix.

```python
U,S,V = np.linalg.svd(Xcentered,full_matrices=True)

# Percent variance accounted for
plt.plot(100.0*S/np.sum(S))
plt.ylabel('% Var')
plt.xlabel('Singular Value')
plt.show()
```

Each of the singular values indicate some amount of variance present in the original data set that is captured by the corresponding left-singular vector (column of U). They are sorted in order from largest to smallest when returned by the `svd()` function so that you can see that the largest amount of variance is captured by the first left-singular vector, the second most variance by the second singular-vector, and so on. Often, the sum of the singular values is calculated to obtain the _total_ variance in the data, and then used to normalize the variance to obtain the percentage of variance captured by each left-singular vector. Given the data above, it is clear that the first two vectors alone will account for over 85% of the total variance in the data, and should form a reasonable projection for the data set.

```python
# Variance accounted for in the first two principal components
100.0*(S[0]+S[1])/np.sum(S)
```

The singular-values (S) are mapped into a rectangular, diagonal matrix which is then multiplied by the left-singular vectors (U). The vectors in U are all unit length, so this operation effectively scales the length of the first 4 vectors by each of the corresponding singular values. In the end, these operations will produce a rotated version of our original data set where the major orthogonal directions capturing the largest variance in the data lie along the principal axes (x and y for a 2D plot). Each of these so-called principal components is a linear combination of our original feature vectors, and allows us to produce a projection onto a smaller set of these components by simply throwing away vectors associated with small singular values. Thus, while we obtain all 4 of the principal components for the iris data set, we will throw away the last two as they capture less than 15% of the variance in the data.

```python
# Scale the singular vectors, resulting in a rotated form of our mean-centered data
D = np.zeros([X.shape[0],X.shape[1]])
np.fill_diagonal(D,S)
Xrotated = np.dot(U,D)

# Extract just the first two principal components!
PCs = Xrotated[:,0:2]
PCs.shape
```

Now that we have projected our data set into a low-dimensional space where we can better visualize it, let's make a plot to see how it looks. We will be careful to color the points by the associated class label.

```python
# The x and y values come from the two
# Principal Components and the colors for
# each point are selected based on the
# corresponding iris species for each point...
plt.scatter(PCs[:,0],PCs[:,1],
            color=[['red','green','blue'][i] for i in Y.astype(int)])
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.show()
```

You can see in the above plot that the data forms three fairly distinct groups of points. The red group on the left is easily linearly separable from the other two groups. Even the green and blue groups are fairly distinct. While you can almost draw a straight line between them, it appears that a few data points from each of these groups would lie on the opposite side, and not allow for perfect classification with a linear network. Nevertheless, we can now see why a linear network might work well on this data, and (perhaps more importantly) that an additional feature measurement may be needed to completely separate the green and blue species.

We can perform the same analysis (just like we performed PCA above from scratch with numpy) by using the SciKitLearn library:

```python
pca = PCA(2)
PCs = pca.fit_transform(X)[:,0:2]
```

```python
plt.scatter(PCs[:,0],PCs[:,1],
            color=[['red','green','blue'][i] for i in Y.astype(int)])
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.show()
```

The result is the same here, but you will sometimes notice that this data has been "flipped" or "mirrored" along the origin of one (or both) of the principal components. This commonly occurs since there is an equally valid rotation of the data 180 degrees along any axis that still preserves all of the variance and internal relationships between the data points. Either way, the same amount of variance is accounted for on each component, and decision boundaries can still be explored.

Let's perform a similar analysis with a much larger data set.

## Exploring MNIST with PCA

```python
# Load the MNIST data set using Keras
# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Subsample (there's a lot of data here!)
X = x_train[range(0,x_train.shape[0],10),:,:]
Y = y_train[range(0,y_train.shape[0],10)]

display(X.shape)
display(Y.shape)
```

We will only look at the training data for this data set for now (the test data is of the same kind so will not be needed for our purposes in this assignment). The training set consists of 60,000 images each 28x28 pixels in size. However, we have selected just 6000 of those images for this example to make the analysis more tractable (60,000 would take a long time and lots of memory to compute the PCs). Each pixel is represented by an integer intensity value between 0 and 255. For the sake of examining what these images look like, let's scale those intensities to be floating point values in the range [0,1]:

```python
X = X.astype('float32') / 255.0
```

Once normalized, the images can be easily plotted as a kind of "heatmap" where black pixels are represented by low intensities and white pixels by high intensities. The `imshow()` function will map high intensities to a dark colors (blue) and low intensities to light colors (yellow). Intermediate values are colored green with more or less blue/yellow depending on which intensity they favor more.

Let's take a look at the first five images in our subset of the MNIST data...

```python
# Plot some of the images
for i in range(5):
    plt.figure()
    plt.imshow(X[i,:,:])
plt.show()
```

```python
# What are their corresponding class labels?
display(sp.Matrix(Y[0:5]))
```

You can see that each of these images corresponds to a hand-written digit, each labeled with the appropriate number in the category labels, Y.

We will now **flatten** these images so that we can perform principal component analysis, so that each pixel is treated as a single measurement. The numpy function `reshape()` allows us to do this by providing a new shape for the data that has the same _total_ number of entries. Here we convert earh 28x28 matrix (image) into a 784 element vector (since 28x28=784). PCA can still detect relationships between the pixel values even when they are not arranged in a matrix form, so we treat each image as a vector.

```python
X = X.reshape(X.shape[0],X.shape[1]*X.shape[2])
X.shape
```

Each image has now been encoded as a feature vector in a 784-dimensional space. Even though the pixel intensities make sense to us when visualized as a 2D image, newly initialized neural networks only experience the vector space for the first time, and have to learn such relationships from scratch. However, let's see if PCA can provide some insight on the difficulty of this task.

We will apply the same approach as before:
1. mean-centering the features
2. calculating the SVD
3. examining the singular values
4. scaling the left-singular vectors
5. plotting the two-dimensional projection

NOTE: It may take a minute or so to compute the SVD for a data set of this size, so be patient on the steps below.

```python
# Mean-centering
Xcentered = np.apply_along_axis(mean_center,0,X)

# SVD
U,S,V = np.linalg.svd(Xcentered,full_matrices=True)

# Percent variance accounted for
plt.plot(100.0*S/np.sum(S))
plt.ylabel('% Var')
plt.xlabel('Singular Value')
plt.show()
```

```python
# Variance accounted for in the first two principal components
100.0*(S[0]+S[1])/np.sum(S)
```

You can see that the variance accounted for dips sharply (which is good, because having a few PCs which capture a lot of variance is a useful thing). However, the variance captured by the first __two__ components together is _less than 5% of the total variance_ in the data set, so PCA might __not__ be so useful for visualization.

However, let's take a quick look at one more thing before moving on:

```python
# Variance accounted for in the first two principal components
display(100.0*(np.sum(S[0:340]))/np.sum(S))

# Reduction?
display(100*340/len(S))
```

Notice that 90% of the **total** variance in the data can be captured using 340 principal components. 340 is only just over 43% of the original dimensionality, so 90% of the data set can be effectively represented using a space less than half of the original in size. Thus, PCA can also be thought of as a lossy linear data compression technique as well. While we can't visualize a 340-dimensional space, a much smaller network would be required to process this data, possibly without sacrificing generalization accuracy (but that's for a later time). Let's just take a look at the first two principal components for now:

```python
D = np.zeros([X.shape[0],X.shape[1]])
np.fill_diagonal(D,S)
Xrotated = np.dot(U,D)

# First two principal components!
PCs = Xrotated[:,0:2]
PCs.shape
```

```python
# Need a lot of colors for this one!
plt.scatter(PCs[:,0],PCs[:,1],
            color=[['red','green','blue',
                    'cyan','magenta','yellow',
                    'black','brown','grey',
                    'purple'][i] for i in Y.astype(int)])
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.show()
```

Note that the clutter of 6000 points is pretty bad in this space, and reducing the sampling may help in some ways. Let's try that now:

```python
plt.scatter(PCs[range(0,6000,10),0],PCs[range(0,6000,10),1],
            color=[['red','green','blue',
                    'cyan','magenta','yellow',
                    'black','brown','grey',
                    'purple'][i] for i in Y[range(0,6000,10)].astype(int)])
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.show()
```

Even with the subsampling, some interesting divisions between classes can be found, but others are not so clear. For example, the red points now correspond to images of zeros and green dots now correspond to images of ones, and these seem to lie on opposite ends of the first principal component. Also, the brown sevens seem very distinct from the cyan threes. However, others are more cluttered, like the purple nines, brown sevens, and magenta fours. However, this makes some sense because these numbers share some common featural similarities.

Because PCA is a **linear** technique, it is somewhat limited in its ability to capture some of the **non-linear** relationships between the data vectors. Also, two dimensions, while useful for visualization, is still often too low to capture all of the relevant relationships that allow for categorization. The 2D representation can be thought of as capturing the **lower-bound on the distances between the data vectors** since *adding more dimensions may cause points to move farther apart, but they could never cause them to move closer together*.

Even with its drawbacks, PCA is a useful technique for quickly creating projections of high-dimensional data onto lower-dimensional spaces for exploratory analysis and lossy compression. There is even a neural network architecture that we will look at briefly later in the semester which allows a single-layer network to learn to perform PCA on a set of data by using a specific learning rule (Oja's Rule). However, because we can solve linear problems like PCA in closed form just like we did above, it is atypical to utilize Oja's Rule since it requires training several many epochs and is therefore slower than closed-form solutions.


---
# Section 3.4: The Gram-Schmidt Process
---

Given a linearly independent set of $m$ vectors in $\mathbb{R}^n$,

$$
\{v_1, v_2, \ldots, v_m\} \subseteq \mathbb{R}^n,
$$

the **Gram-Schmidt process** produces an orthonormal set of vectors,

$$
\{q_1, q_2, \ldots, q_m\} \subseteq \mathbb{R}^n,
$$

such that

$$
\mathrm{span}\{v_1, v_2, \ldots, v_m\} = \mathrm{span}\{q_1, q_2, \ldots, q_m\}.
$$

---

## Orthonormal set of vectors

A set of vectors $\{q_1, q_2, \ldots, q_m\}$ is said to be _orthonormal_ if the vectors are pairwise orthogonal and each vector has Euclidean norm one; that is,

$$
\langle q_i, q_j \rangle =
\begin{cases}
0 & \text{if $i \ne j$} \\
1 & \text{if $i = j$},
\end{cases}
$$

where $\langle q_i, q_j \rangle = q_i^T q_j$ is the inner product (or dot product) of the vectors $q_i$ and $q_j$.

---

## Orthonormal and isometric matrices

A matrix $Q \in \mathbb{R}^{n \times m}$, with $n \ge m$, is called _isometric_ (or an _isometry_) if its columns are orthonormal; that is, if $Q^T Q = I$.

If $Q$ is a _square_ isometric matrix, then $Q$ is an orthogonal matrix since $Q^{-1} = Q^T$, so we also have $Q Q^T = I$.

However, if $Q$ is an isometric matrix that has more rows than columns, then $Q$ cannot have an inverse since it is not square. Moreover, $Q Q^T \ne I$.

---

## Example

```julia
using LinearAlgebra
```

```julia
"""
`isorand(n, m)` generates a random n-by-m isometric matrix.
"""
function isorand(n, m)
    F = qr(randn(n, m))
    return F.Q[:,1:m]
end

Q = isorand(5, 3)
```

```julia
?isorand
```

```julia
Q'Q
```

```julia
Q*Q'
```

---

## Exercise

Let $Q \in \mathbb{R}^{n \times m}$ ($n > m$) be an isometry with columns $q_1, q_2, \ldots, q_m$ and let

$$S = \mathrm{span}\{q_1,q_2,\ldots,q_m\}$$

be the subspace spanned by these vectors.

1. Show that $Q Q^T v = 0$ if $v$ is orthogonal to $q_1, q_2, \ldots, q_m$.

2. Show that $Q Q^T w = w$ if $w \in S$. Therefore, $Q Q^T$ behaves like the identity matrix on the subspace $S$.

3. Show that $(Q Q^T)^2 = Q Q^T$. Thus, $Q Q^T$ is a _projector_ (in fact, it is an _orthogonal projector_).

### Part 1

Suppose that $v \perp q_i$ for $i=1,\ldots,m$. In other words, $\langle v, q_i \rangle = 0$ for $i=1,\ldots,m$. Note that

$$
Q^T v =
\begin{bmatrix}
\langle q_1, v \rangle \\
\vdots \\
\langle q_m, v \rangle
\end{bmatrix} = 0.
$$

Thus, $Q Q^T v = Q \cdot 0 = 0$.

### Part 2

Let $w \in S$. Then $w$ is a linear combination of the vectors $q_1,\ldots,q_m$, so

$$
w = c_1 q_1 + \cdots + c_m q_m.
$$

Then, letting $e_i$ represent the $i$th column of the $n \times n$ identity matrix, we have

$$
\begin{align}
Q Q^T w
&= Q Q^T (c_1 q_1 + \cdots + c_m q_m) \\
&= Q (c_1 Q^T q_1 + \cdots + c_m Q^T q_m) \\
&= Q (c_1 e_1 + \cdots + c_m e_m) \\
&= Q \begin{bmatrix} c_1 \\ \vdots \\ c_m \end{bmatrix} \\
&= c_1 q_1 + \cdots + c_m q_m \\
&= w.
\end{align}
$$

### Part 3

Since $Q^T Q = I$, we have

$$
(Q Q^T)^2 = Q Q^T Q Q^T = Q I Q^T = Q Q^T.
$$

---

## Example

```julia
Q = isorand(5, 3)
```

```julia
x = randn(5)

# Project x onto S = span{q1,...,qm}
w = Q*(Q'x)
```

```julia
Q*(Q'w) ≈ w
```

```julia
# Since Q*Q' is an orthogonal projector,
# x - w is orthogonal to q1,...,qm.

v = x - w

Q'v
```

---

## Exercise

Show that if $Q \in \mathbb{R}^{n \times m}$ is an isometry, then

1. $\langle Q x, Q y \rangle = \langle x, y \rangle$ for all $x, y \in \mathbb{R}^{m}$,

2. $\|Q x\|_2 = \|x\|_2$ for all $x \in \mathbb{R}^m$.

### Part 1

Since $Q^T Q = I$, we have that

$$
\langle Q x, Q y \rangle = (Q x)^T (Q y) = x^T Q^T Q y = x^T y = \langle x, y \rangle.
$$

### Part 2

By part 1,

$$
\| Q x \|_2 = \sqrt{\langle Q x, Q x \rangle} = \sqrt{\langle x, x \rangle} = \| x \|_2.
$$

---

> ## Theorem: (Condensed $QR$ for $n \times m$ matrix)
>
> Let $A \in \mathbb{R}^{n \times m}$, $n \ge m$. Then there exists $\hat{Q} \in \mathbb{R}^{n \times m}$ isometric and $\hat{R} \in \mathbb{R}^{m \times m}$ upper-triangular such that
>
> $$ A = \hat{Q}\hat{R}. $$
>
> Moreover, if the columns of $A$ are linearly independent, then this factorization of $A$ is unique under the condition that $\hat{R}$ has positive diagonal entries.

---

## Example

We can obtain the condensed $QR$ factorization of a matrix $A$ using the Julia function `qr` as follows.

```julia
function condensed_qr(A)
    n, m = size(A)
    F = qr(A)
    Q̂ = Matrix(F.Q)
    R̂ = F.R
    return Q̂, R̂
end
```

```julia
n, m = 5, 3
A = randn(n, m)
Q̂, R̂ = condensed_qr(A);
```

```julia
Q̂
```

```julia
R̂
```

```julia
A ≈ Q̂*R̂
```

---

## Exercise

Let $q_1,\ldots,q_m$ be orthonormal vectors in $\mathbb{R}^n$. Prove that $q_1,\ldots,q_m$ are linearly independent.

### Proof.

Suppose we have a linear combination of the vectors $q_1,\ldots,q_m$ that gives us the zero vector. That is, $\exists c_1,\ldots,c_m \in \mathbb{R}$ such that

$$
c_1 q_1 + \cdots + c_m q_m = 0.
$$

To show that $q_1,\ldots,q_m$ are linearly independent, we need to show that $c_1 = \cdots = c_m = 0$.

Let's take the inner product of $q_1$ with $c_1 q_1 + \cdots + c_m q_m$, which gives us

$$
\begin{eqnarray}
\langle q_1, c_1 q_1 + \cdots + c_m q_m \rangle &=& \langle q_1, 0 \rangle \\
c_1 \langle q_1, q_1 \rangle + \cdots + c_m \langle q_1, q_m \rangle &=& 0 \\
c_1 \cdot 1 + c_2 \cdot 0 + \cdots + c_m \cdot 0 &=& 0 \\
c_1 &=& 0.
\end{eqnarray}
$$

Thus,

$$
c_2 q_2 + \cdots + c_m q_m = 0.
$$

Taking the inner product of this equation with $q_2$ leads us to conclude that $c_2 = 0$. And so on, we have that $c_1 = c_2 = \ldots = c_m = 0$.

---

## The Classical Gram-Schmidt Process

Let $S$ be a subspace of $\mathbb{R}^n$ and let $v_1,\ldots,v_m$ be a basis of $S$; that is, $v_1,\ldots,v_m$ are linearly independent and span $S$, which implies that $\dim S = m$.

Gram-Schmidt will produce orthonormal vectors $q_1,\ldots,q_m$ that form a basis of $S$ and satisfy

$$
\begin{eqnarray}
\mathrm{span}\{q_1\} &=& \mathrm{span}\{v_1\} \\
\mathrm{span}\{q_1,q_2\} &=& \mathrm{span}\{v_1,v_2\} \\
\mathrm{span}\{q_1,q_2,q_3\} &=& \mathrm{span}\{v_1,v_2,v_3\} \\
&\vdots& \\
\mathrm{span}\{q_1,q_2,\ldots,q_m\} &=& \mathrm{span}\{v_1,v_2,\ldots,v_m\}. \\
\end{eqnarray}
$$

---

### Step 1

Since $\mathrm{span}\{q_1\} = \mathrm{span}\{v_1\}$, we let

$$
r_{11} = \|v_1\|_2, \quad q_1 = \frac{v_1}{r_{11}}.
$$

---

### Step 2

Next, we let $w$ be the projection of $v_2$ onto $\mathrm{span}\{q_1\}$:

$$
w = \begin{bmatrix} q_1 \end{bmatrix} \begin{bmatrix} q_1 \end{bmatrix}^T v_2 = \langle q_1, v_2 \rangle q_1.
$$

Then, we let $\hat{q}_2 = v_2 - w$. Notice that $\hat{q}_2$ is orthogonal to $q_1$ since

$$
\begin{align}
\langle q_1, \hat{q}_2 \rangle
&= \langle q_1, v_2 - \langle q_1, v_2 \rangle q_1 \rangle \\
&= \langle q_1, v_2 \rangle - \langle q_1, v_2 \rangle \langle q_1, q_1 \rangle \\
&= \langle q_1, v_2 \rangle - \langle q_1, v_2 \rangle \\
&= 0.
\end{align}
$$

Therefore, we let

$$
r_{12} = \langle q_1, v_2 \rangle, \quad \hat{q}_2 = v_2 - r_{12} q_1, \quad r_{22} = \|\hat{q}_2\|_2, \quad q_2 = \frac{\hat{q}_2}{r_{22}},
$$

and we have $\mathrm{span}\{q_1, q_2\} = \mathrm{span}\{v_1, v_2\}$ (see text for details).

---

### Step $k$

Now suppose we have orthonormal vectors $q_1,\ldots,q_{k-1}$ that satisfy

$$
\mathrm{span}\{q_1,\ldots,q_i\} = \mathrm{span}\{v_1,\ldots,v_i\}, \quad i=1,\ldots,k-1.
$$

We let $w$ be the projection of $v_k$ onto $\mathrm{span}\{q_1,\ldots,q_{k-1}\}$,

$$
w =
\begin{bmatrix} q_1 & \cdots & q_{k-1} \end{bmatrix}
\begin{bmatrix} q_1 & \cdots & q_{k-1} \end{bmatrix}^T v_k =
\langle q_1, v_k \rangle q_1 + \cdots + \langle q_{k-1}, v_k \rangle q_{k-1},
$$

and we let $\hat{q}_k = v_k - w$. Then $\hat{q}_k$ is orthogonal to $q_1,\ldots,q_{k-1}$.

Therefore, letting

$$
r_{ik} = \langle q_i, v_k \rangle, \quad i=1,\ldots,k-1,
$$

we have

$$
\hat{q}_k = v_k - \sum_{i=1}^{k-1} r_{ik} q_i.
$$

Finally, we let

$$
r_{kk} = \|\hat{q}_k\|_2, \quad q_k = \frac{\hat{q}_k}{r_{kk}},
$$

and we have $\mathrm{span}\{q_1, \ldots, q_k\} = \mathrm{span}\{v_1, \ldots, v_k\}$.

---

## An implementation of Gram-Schmidt

```julia
function gram_schmidt(V)
    n, m = size(V)

    Q = copy(V)
    R = zeros(m, m)

    for k=1:m
        # R[i,k] = dot(Q[:,i],Q[:,k]), for i=1:k-1
        for i=1:k-1
            for j=1:n
                R[i,k] += Q[j,i]*Q[j,k]
            end
        end

        # Q[:,k] = Q[:,k] - sum(R[i,k]*Q[:,i] for i=1:k-1)
        for i=1:k-1
            for j=1:n
                Q[j,k] -= R[i,k]*Q[j,i]
            end
        end

        R[k,k] = norm(Q[:,k])
        Q[:,k] /= R[k,k]
    end

    return Q, R
end
```

```julia
n, m = 5, 3
A = randn(n, m)
Q, R = gram_schmidt(A);
```

```julia
Q
```

```julia
Q'Q
```

```julia
R
```

```julia
A ≈ Q*R
```

---

## Gram-Schmidt equals $QR$

Note that

$$
\begin{eqnarray}
v_1 &=& r_{11} q_1 \\
v_2 &=& r_{12} q_1 + r_{22} q_2 \\
&\vdots& \\
v_m &=& r_{1m} q_1 + r_{2m} q_2 + \cdots + r_{mm} q_m.
\end{eqnarray}
$$

Thus, if we let

$$
\begin{eqnarray}
V &=& \begin{bmatrix} v_1 & \cdots & v_m \end{bmatrix} \in \mathbb{R}^{n \times m}, \\\\
Q &=& \begin{bmatrix} q_1 & \cdots & q_m \end{bmatrix} \in \mathbb{R}^{n \times m}, \\\\
R &=&
\begin{bmatrix}
r_{11} & r_{12} & \cdots & r_{1m} \\
       & r_{22} & \cdots & r_{2m} \\
       &        & \ddots & \vdots \\
       &        &        & r_{mm}
\end{bmatrix} \in \mathbb{R}^{m \times m},
\end{eqnarray}
$$

we have that

$$
V = Q R.
$$

Therefore, we have another way to compute the $QR$ decomposition of a matrix, and any method for computing a $QR$ decomposition provides a way to orthogonalize a set of vectors.

---

## The Gram-Schmidt Process and Roundoff Errors

We expect $Q^T Q = I$ in exact arithmetic. Thus, we can measure the deviation from orthonormality due to roundoff errors by computing

$$
\|I - Q^T Q\|_2.
$$

```julia
# Create a Vandermonde matrix
vandermonde(n,m) = [(j/m)^(i-1) for i=1:n, j=1:m]

n, m = 5, 3
V = vandermonde(n, m)
```

```julia
using Printf

@printf("%3s %3s %10s %14s %14s\n", "n", "m", "cond(V)", "Gram-Schmidt", "Householder")
for (n, m) in [(6,4), (9,6), (12,8), (15,10), (18,12)]
    V = vandermonde(n,m)
    Q, R = gram_schmidt(V)
    Q̂, R̂ = condensed_qr(V)
    @printf("%3d %3d %10.1e %14.1e %14.1e\n", n, m, cond(V),
        opnorm(I - Q'Q), opnorm(I - Q̂'Q̂))
end
```

---

## Modified Gram-Schmidt

The computation

```julia
for i=1:k-1
    for j=1:n
        R[i,k] += Q[j,i]*Q[j,k]
    end
end
for i=1:k-1
    for j=1:n
        Q[j,k] -= R[i,k]*Q[j,i]
    end
end
```

produces the exact same output as the computation

```julia
for i=1:k-1
    for j=1:n
        R[i,k] += Q[j,i]*Q[j,k]
    end
    for j=1:n
        Q[j,k] -= R[i,k]*Q[j,i]
    end
end
```

in exact arithmetic. In the presence of roundoff errors, the latter computation does a better job in terms of orthogonality.

```julia
function modified_gram_schmidt(V)
    n, m = size(V)

    Q = copy(V)
    R = zeros(m, m)

    for k=1:m
        for i=1:k-1
            for j=1:n
                R[i,k] += Q[j,i]*Q[j,k]
            end
            for j=1:n
                Q[j,k] -= R[i,k]*Q[j,i]
            end
        end

        R[k,k] = norm(Q[:,k])
        Q[:,k] /= R[k,k]
    end

    return Q, R
end
```

```julia
using Printf

@printf("%3s %3s %10s %14s %14s %14s\n", "n", "m", "cond(V)",
        "Gram-Schmidt", "Modified G-S", "Householder")
for (n, m) in [(6,4), (9,6), (12,8), (15,10), (18,12)]
    V = vandermonde(n,m)
    Q, R = gram_schmidt(V)
    Qm, Rm = modified_gram_schmidt(V)
    Q̂, R̂ = condensed_qr(V)
    @printf("%3d %3d %10.1e %14.1e %14.1e %14.1e\n", n, m, cond(V),
        opnorm(I - Q'Q), opnorm(I - Qm'Qm), opnorm(I - Q̂'Q̂))
end
```

---

## Gram-Schmidt with Reorthogonalization

```julia
function orthogonalize!(s, Q, k, n)
    for i=1:k-1
        s[i] = 0
        for j=1:n
            s[i] += Q[j,i]*Q[j,k]
        end
    end
    for i=1:k-1
        for j=1:n
            Q[j,k] -= s[i]*Q[j,i]
        end
    end
end

function gram_schmidt(A; num_orthog=1)
    n, m = size(A)

    s = Vector{Float64}(undef, m-1)
    R = zeros(m, m)
    Q = copy(A)

    for k=1:m

        for j=1:num_orthog
            orthogonalize!(s, Q, k, n)
            R[1:k-1,k] += s[1:k-1]
        end

        R[k,k] = norm(Q[:,k])
        Q[:,k] /= R[k,k]
    end

    return Q, UpperTriangular(R)
end
```

```julia
n, m = 50, 30
A = randn(n, m)
Q, R = gram_schmidt(A, num_orthog=2)
A ≈ Q*R
```

```julia
using Printf

@printf("%3s %3s %10s %14s %14s %14s %14s\n", "n", "m", "cond(V)",
        "Gram-Schmidt", "Modified G-S", "Reorthog", "Householder")
for (n, m) in [(6,4), (9,6), (12,8), (15,10), (18,12)]
    V = vand(n,m)
    Q, R = gram_schmidt(V)
    Qm, Rm = modified_gram_schmidt(V)
    Qr, Rr = gram_schmidt(V, num_orthog=2)
    Q̂, R̂ = condensed_qr(V)
    @printf("%3d %3d %10.1e %14.1e %14.1e %14.1e %14.1e\n", n, m, cond(V),
        opnorm(I - Q'Q), opnorm(I - Qm'Qm), opnorm(I - Qr'Qr), opnorm(I - Q̂'Q̂))
end
```

---

## Flop counts

1. $QR$ by Householder reflectors and forming $Q$ requires about $4nm^2 - 4m^3/3$ flops.

2. Gram-Schmidt with reorthogonalization requires about $4nm^2$ flops.

Thus, the method by reflectors requires about $4m^3/3$ fewer flops than Gram-Schmidt.

However, Gram-Schmidt has the advantage that the vectors $v_1,\ldots,v_m$ do not all need to be known from the beginning; an example of this situation is the Arnoldi process for computing eigenvectors (see Chapter 6 in the text) in which the vector $v_k$ cannot be determined until after $q_1,\ldots,q_{k-1}$ have been computed.

---

Section 3.4 - The Gram-Schmidt Process.ipynb


```python
import numpy as np
import matplotlib as mpl
import sympy as sym
from matplotlib import pyplot as plt
import scipy.integrate as integrate
from scipy.integrate import quad
from scipy.special import ellipe
from scipy.special import ellipk

#parameters for Galaxy NGC 5907
G = 4.300e-6                                        #gravitational constant (kpc/solar mass*(km/s)^2)
h = 8.9                                           #radial scale-length (kpc) [FOR NGC 5533]
#d = 0.99                                            #cut-off length upper limits(kpc)
d = 0.2*h
mu0 = 1.85e9                                        #central surface density [FOR NGC 5533]
z0 = 0.2*h                                          #half-thickness (kpc)
R = 4*h                                             #cut-off radius (kpc)

#rho = 10                                            #madeup parameter until we figure out the partial derivative of rho(u,xi)

#definitions
def x(r,u,xi):
    return ((r**2)+(u**2)+(xi**2))/(2*r*u)
#x = lambda r,u,xi: ((r**2)+(u**2)+(xi**2))/(2*r*u)

def px(r,u,xi):
    return x(r,u,xi)-(np.sqrt((x(r,u,xi)**2)-1))
#p = lambda x: x-(np.sqrt((x**2)-1))

rho00 = mu0/(2*z0)
```

```python
#density piecewise function
def rho0(r, R, h, d):
    condlist = [r <= R, (r > R) & (r <= (R+d)), r > (R+d)]
    funclist = [lambda r: rho00*np.exp(-r/h), lambda r: rho00*np.exp(-R/h)*(1-((r-R)/d)), lambda r: 0]
    return np.piecewise(r, condlist, funclist)

ra = np.linspace(0.1, 25, 500)
#ra = np.linspace(0, 25, 500)

#plotting piecewise function
fig = plt.figure(figsize=(9.0,8.0))                #size of the plot
ax = plt.axes()
ax.grid(True)
ax.yaxis.grid(True,which='minor',linestyle='--')

plt.plot(ra, rho0(ra, R, h, d), linestyle='solid', label='$\\rho_0$(r)')

plt.axis('tight');                                 #xmin, xmax, ymin, ymax
                                                   #or use: plt.xlim(10, 0) plt.ylim(1.2, -1.2);
                                                   #or to tighten the bounds: plt.axis('tight')
                                                   #or use: plt.axis([0, 15000, 0.5e6, 2.5e6])

fig.suptitle('Density Distribution', fontsize=18)
ax.set(title='Piecewise Function')              #labeling axes
ax.set_xlabel('r (kpc)', fontsize=14)
ax.set_ylabel('$\\rho_0$ ()', fontsize=14)
ax.xaxis.set_tick_params(labelsize=14)
ax.yaxis.set_tick_params(labelsize=14);

#plt.legend(fontsize=16);                                   #adding legend

print(rho0(20.289, R, h, d))
print(rho0(20.291, R, h, d))
```

```python
#partial derivative of rho(u,xi)
def durho0(r, R, h, d):
    condlist = [r <= R, (r > R) & (r <= (R+d)), r > (R+d)]
    funclist = [lambda r: -(1/h)*rho00*np.exp(-r/h), lambda r: -(1/d)*rho00*np.exp(-R/h), lambda r: 0]
    return np.piecewise(r, condlist, funclist)

#plotting piecewise function derivative
fig = plt.figure(figsize=(9.0,8.0))                #size of the plot
ax = plt.axes()
ax.grid(True)
ax.yaxis.grid(True,which='minor',linestyle='--')

plt.plot(ra, durho0(ra, R, h, d), linestyle='solid', label='d$\\rho_0$(r)')

plt.axis('tight');                                 #xmin, xmax, ymin, ymax
                                                   #or use: plt.xlim(10, 0) plt.ylim(1.2, -1.2);
                                                   #or to tighten the bounds: plt.axis('tight')
                                                   #or use: plt.axis([0, 15000, 0.5e6, 2.5e6])

fig.suptitle('Density Distribution', fontsize=18)
ax.set(title='Piecewise Function Derivative')              #labeling axes
ax.set_xlabel('r (kpc)', fontsize=14)
ax.set_ylabel('d$\\rho_0$ ()', fontsize=14)
ax.xaxis.set_tick_params(labelsize=14)
ax.yaxis.set_tick_params(labelsize=14);

plt.legend(fontsize=16);                                   #adding legend
plt.show()

print(durho0(20.289, R, h, d))
print(durho0(20.291, R, h, d))
```

```python
#disk density distribution
rho_rz = lambda r,z: rho0(r, R, h, d)*(np.power(np.cosh(z/z0), (-2)))
drho_rz = lambda r,z: durho0(r, R, h, d)*(np.power(sym.cosh(z/z0), (-2)))

#z = np.linspace(0, 25, 500)
#print(np.shape(r3))
#print(np.shape(z3))
#print(np.shape(rho_rz(r3,z3)))

#plotting disk density distribution
fig = plt.figure(figsize=(9.0,8.0))                #size of the plot
ax = plt.axes()
ax.grid(True)
ax.yaxis.grid(True,which='minor',linestyle='--')

plt.plot(ra, rho_rz(ra,0), linestyle='solid', label='z=0')
plt.plot(ra, rho_rz(ra,1), linestyle='solid', label='z=1', color='r')

plt.axis('tight');                                 #xmin, xmax, ymin, ymax
                                                   #or use: plt.xlim(10, 0) plt.ylim(1.2, -1.2);
                                                   #or to tighten the bounds: plt.axis('tight')
                                                   #or use: plt.axis([0, 15000, 0.5e6, 2.5e6])

fig.suptitle('Density Distribution', fontsize=18)
ax.set(title='Piecewise Function Derivative')              #labeling axes
ax.set_xlabel('r (kpc)', fontsize=14)
ax.set_ylabel('d$\\rho_0$ ()', fontsize=14)
ax.xaxis.set_tick_params(labelsize=14)
ax.yaxis.set_tick_params(labelsize=14);

plt.legend(fontsize=16);                                   #adding legend
plt.show()

print(rho_rz(20.289,0))
print(rho_rz(20.291,0))
print(drho_rz(20.289,0))
print(drho_rz(20.291,0))
```

```python
#plotting disk density distribution in 3D
from mpl_toolkits import mplot3d

def rho_rz2(r, z):
    return rho0(r, R, h, d)*(np.power(np.cosh(z/z0), (-2)))

za = np.linspace(-1, 1, 500)

RA, ZA = np.meshgrid(ra, za)
RZ = rho_rz2(RA, ZA)

#fig = plt.figure()
ax = plt.axes(projection='3d')
ax.contour3D(RA, ZA, RZ, 100, cmap='RdGy')
plt.tight_layout()
ax.set_xlabel('r')
ax.set_ylabel('z')
ax.set_zlabel('rho_rz');

#print(rho_rz2(2,2))
```

```python
#plotting derivative of disk density distribution in 3D
def drho_rz2(r, z):
    return durho0(r, R, h, d)*(np.power(np.cosh(z/z0), (-2)))

zb = np.linspace(-2, 2, 500)

RB, ZB = np.meshgrid(ra, zb)
RZB = drho_rz2(RB, ZB)

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.contour3D(RB, ZB, RZB, 100, cmap='RdGy')

plt.tight_layout()
ax.set_xlabel('r')
ax.set_ylabel('z')
ax.set_zlabel('drho_rz');
```

```python
#complete elliptic integral
K = lambda r,u,xi: ellipk(px(r,u,xi)) - ellipe(px(r,u,xi))

print(K(1,1,1))
```

    0.35734399698214614

```python
#inner function
f = lambda r,u,z: drho_rz(u, z)*(2*K(r,u,z))/(np.pi*np.sqrt(r*u*px(r,u,z)))

#za = np.linspace(-1, 1, 500)

#plot inner function 3D
def f2(r, u, z):
    return drho_rz(r, z)*(2*K(r,u,z))/(np.pi*np.sqrt(r*u*px(r,u,z)))

RC, ZC = np.meshgrid(ra, za)
RZC = f2(RC, 1, ZC)

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.contour3D(RC, ZC, RZC, 100, cmap='RdGy')

plt.tight_layout()
ax.set_xlim(0,5)
ax.set_xlabel('r')
ax.set_ylabel('z')
ax.set_zlabel('inner function');

print(f(1,1,1))
```

```python
#integrate function
#intf = lambda xi: quad(f, 0, np.inf, args=(r,u,))
f3 = lambda z,r,u: f(r,u,z)
intf = lambda r,u: quad(f3, 0, np.inf, args=(r,u,))[0]
print(intf(25,5))
```

```python
#integrate outer function
intf3 = lambda u,r: intf(r,u)
intintf = lambda r: quad(intf3, 0.1, 100, args=(r,))[0]
print(intintf(1))
```

    C:\Users\Cirdan\Anaconda3\lib\site-packages\ipykernel_launcher.py:3: RuntimeWarning: overflow encountered in cosh
      This is separate from the ipykernel package so we can avoid doing imports until
    C:\Users\Cirdan\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in double_scalars

    C:\Users\Cirdan\Anaconda3\lib\site-packages\scipy\integrate\quadpack.py:385: IntegrationWarning: The occurrence of roundoff error is detected, which prevents
      the requested tolerance from being achieved.  The error may be
      underestimated.
      warnings.warn(msg, IntegrationWarning)

    nan

    C:\Users\Cirdan\Anaconda3\lib\site-packages\scipy\integrate\quadpack.py:385: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
      If increasing the limit yields no improvement it is advised to analyze
      the integrand in order to determine the difficulties.  If the position of a
      local difficulty can be determined (singularity, discontinuity) one will
      probably gain from splitting up the interval and calling the integrator
      on the subranges.  Perhaps a special-purpose integrator should be used.
      warnings.warn(msg, IntegrationWarning)

```python
#radial force function
F = lambda r: 4*np.pi*G*intintf(r)
print(F(20))
rd = np.linspace(0.1, 50, num=50)
Fv = np.vectorize(F)

Fvcalc = Fv(rd)
```

    C:\Users\Cirdan\Anaconda3\lib\site-packages\ipykernel_launcher.py:3: RuntimeWarning: overflow encountered in cosh
      This is separate from the ipykernel package so we can avoid doing imports until

    -1515.0127029357081

    C:\Users\Cirdan\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in double_scalars

```python
#plot radial force
fig = plt.figure(figsize=(9.0,8.0))                #size of the plot
ax = plt.axes()
ax.grid(True)
ax.yaxis.grid(True,which='minor',linestyle='--')

plt.plot(rd, np.sqrt(-rd*Fvcalc), linestyle='solid', label='Disk Velocity')

plt.axis('tight');                                 #xmin, xmax, ymin, ymax
                                                   #or use: plt.xlim(10, 0) plt.ylim(1.2, -1.2);
                                                   #or to tighten the bounds: plt.axis('tight')
                                                   #or use: plt.axis([0, 15000, 0.5e6, 2.5e6])

fig.suptitle('Disk Velocity', fontsize=18)
ax.set(title='NGC 5533')              #labeling axes
ax.set_xlim(0,50)
ax.set_xlabel('Radius (kpc)', fontsize=14)
ax.set_ylabel('Velocity (km/s)', fontsize=14)
ax.xaxis.set_tick_params(labelsize=14)
ax.yaxis.set_tick_params(labelsize=14);

plt.legend(fontsize=16);                                   #adding legend
plt.savefig('DiskVelocity50pts_ANV 5533.png')
plt.show()
```

```python

```


Considere una ciudad con un mercado perfectamente competitivo de taxis, es decir, el precio no se encuentra regulado por una autoridad estatal. El costo marginal de cada viaje es constante e igual a \$10 y cada taxi puede realizar, en promedio, 25 viajes diarios. Si la función de demanda de viajes en taxi es Q(p) = 2000 - 15p

```python
import sympy as sp
x,p, c = sp.symbols('x,p,c')
Q = 2000 - 15 * p
CMg = 10
CT = 10 * x + c
x_diario = 25

```

a) ¿Cuál es el precio de equilibrio de un viaje?

```python
peq = CMg
peq
```

    10

b) ¿Cuál es, en equilibrio, el número de viajes diarios?

```python
xeq = Q.subs(p, peq)
xeq
```

$\displaystyle 1850$

c) ¿Cuál es, en equilibrio, el número de taxis de la ciudad?

```python
xeq/x_diario
```

$\displaystyle 74$

```python
¿Cuál es el beneficio de un taxista por cada viaje?
```

```python
beneficio = (p*x - CT).subs({p: 10, x:25})
beneficio

```

$\displaystyle - c$

Si la curva de demanda se desplaza a Q(p) = 1220 - 20 p
e) ¿Cuál será ahora el precio de equilibrio de un viaje?

```python
peq = CMg
peq
```

    10

f) ¿Cuáles son ahora los beneficios de un taxista por cada viaje?

```python
Q = 1200 - 20* p

beneficio = (p*x - CT).subs({p: 10, x:25})
beneficio
```

$\displaystyle - c$

```python

```

Microeconomia/2do parcial/5- Ejercicio 6.ipynb


# COURSE: Master math by coding in Python
## SECTION: Calculus

#### https://www.udemy.com/course/math-with-python/?couponCode=MXC-DISC4ALL
#### INSTRUCTOR: sincxpress.com

Note about this code: Each video in this section of the course corresponds to a section of code below. Please note that this code roughly matches the code shown in the live recording, but is not exactly the same -- the variable names, order of lines, and parameters may be slightly different.

```python

```

```python
import sympy as sym
import matplotlib.pyplot as plt
import numpy as np
from scipy.signal import find_peaks
from IPython.display import display,Math
```

# VIDEO: Computing limits of a function

```python
x = sym.symbols('x')

# the function
fx = x**3

# limit
lim_pnt = 1.5
lim = sym.limit(fx,x,lim_pnt)

display(Math('\\lim_{x\\to %g} %s = %g' %(lim_pnt,sym.latex(fx),lim)))
```

```python
# evaluate the function in a range
xx = np.linspace(-5,5,200)
fxx = sym.lambdify(x,fx) # a function for evaluating a sympy function

# show it in a plot
plt.plot(xx,fxx(xx))

# make the plot look a bit nicer
plt.plot(lim_pnt,lim,'ro')
plt.xlim([-5,5])
plt.ylim([-10,10])
plt.xlabel('x')
plt.ylabel('f(x) = $%s$' %sym.latex(fx))

plt.show()
```

```python
# the function
fx = (x**2)/(x-2)
fxx = sym.lambdify(x,fx) # a function for evaluating a sympy function

xx = np.linspace(1,3,100)

# limit
lim_pnt = 2
lim = sym.limit(fx,x,lim_pnt,dir='+')

display(Math('\\lim_{x\\to %g^+} %s = %g' %(lim_pnt,sym.latex(fx),lim)))

# show it in a plot
plt.plot(xx,fxx(xx))

# make the plot look a bit nicer
plt.plot(lim_pnt,lim,'ro')
plt.xlim([1,3])
plt.xlabel('x')
plt.ylabel('f(x) = $%s$' %sym.latex(fx))

plt.show()
```

### Exercise

```python
fx = sym.sqrt(x+1)*sym.exp(-x)
gx = sym.cos(x + sym.sin(x))

## start by plotting the functions
fxx = sym.lambdify(x,fx)
gxx = sym.lambdify(x,gx)

xx = np.linspace(0,10,100)

# show it in a plot
plt.plot(xx,fxx(xx),label='f(x)')
plt.plot(xx,gxx(xx),label='g(x)')

# make the plot look a bit nicer
plt.xlabel('x')
plt.legend()

plt.show()
```

```python
# compute the limits separately at x=5
lim_pnt = 5
lim_fx = sym.limit(fx,x,lim_pnt)
lim_gx = sym.limit(gx,x,lim_pnt)

display(Math('\\frac{\\lim_{x\\to %g} f(x)}{\\lim_{x\\to %g} g(x)} = \\frac{%g}{%g} = %g' \
             %(lim_pnt,lim_pnt,lim_fx,lim_gx,lim_fx/lim_gx)))

# now compute limit of fx/gx
lim_fg = sym.limit(fx/gx,x,lim_pnt)

display(Math('\\lim_{x\\to %g} \\frac{f(x)}{g(x)} = %g' %(lim_pnt,lim_fg)))

```

# VIDEO: Piece-wise functions

```python
# list function pieces
from sympy.abc import x
piece1 = 0
piece2 = -2*x
piece3 = x**3/10

# put them together with conditions
fx = sym.Piecewise( (piece1,x<0),(piece2,(x>=0) & (x<3)),(piece3,x>=3) )

# evaluate the function in a range
xx = np.linspace(-3,5,1000)
fxx = sym.lambdify(x,fx)

# show it in a plot
plt.plot(xx,fxx(xx),'k')

# make the plot look a bit nicer
plt.xlabel('x')
plt.ylabel('f(x)')
plt.show()
```

### Exercise

```python
# function pieces
x = sym.symbols('x')

f = x**3
g = sym.log(x,2)

fx = sym.Piecewise( (f,x<=0),(g,x>0) )

# print out the function definition
display(Math('f(x) = ' + sym.latex(fx)))

# evaluate the function in a range
xx = np.linspace(-2,2,1000)
fxx = sym.lambdify(x,fx)

# show it in a plot
with plt.xkcd():
    plt.plot(xx,fxx(xx),'k')

# make the plot look a bit nicer
plt.xlim([-2,2])
plt.ylim([-10,3])
plt.xlabel('x')
plt.ylabel('y')

plt.show()
```

# VIDEO: Derivatives of polynomials

```python
x = sym.symbols('x')

fx = x**2

sym.diff(fx)
```

```python
# Leibniz notation
display(Math('f(x) = %s, \\quad \\frac{df}{dx} = %s' %(sym.latex(fx),sym.latex(sym.diff(fx)))))

# Lagrange notation
display(Math('f(x) = %s, \\quad f\' = %s' %(sym.latex(fx),sym.latex(sym.diff(fx)))))

# Newton notation
display(Math('f(x) = %s, \\quad \\ddot{f} = %s' %(sym.latex(fx),sym.latex(sym.diff(sym.diff(fx))))))
```

```python
import sympy.plotting.plot as symplot

# plot

fx = (3-x**3)

# generate the first plot
p = symplot(fx,(x,-5,5),show=False)
p[0].label = '$f(x) = %s$' %sym.latex(fx) # create a label for the legend

# extend to show the second plot as well
p.extend( symplot(sym.diff(fx),show=False) )
p[1].label = '$f(x)\' = %s$' %sym.latex(sym.diff(fx))

# some plotting adjustments
p.ylim = [-5,5]
p.xlim = [-3,3]
p[0].line_color = 'r'
p.legend = True # activate the legend

# and show the plot
p.show()

```

### Exercise

```python
f = 3 + 2*x - 5*x**2 + 7*x**4
g = 4*x**2 + x**5

df = sym.diff(f)
dg = sym.diff(g)

d_f_times_g = sym.diff(f*g)
df_times_dg = sym.diff(f) * sym.diff(g)

display(Math('\\text{Without applying the product rule:}'))
display(Math('\\quad (f\\times g)\' = %s' %sym.latex(sym.expand(d_f_times_g))))
display(Math('\\quad f\' \\times g\' = %s' %sym.latex(sym.expand(df_times_dg))))
```

```python
# apply the product rule
display(Math('\\text{With the product rule: }'))
display(Math('\\quad (f\\times g)\' = %s' %sym.latex(sym.expand(d_f_times_g))))
display(Math('\\quad f\'\\times g+f\\times g\' = %s' %sym.latex(sym.expand(df*g+f*dg))))
```

```python
# try again for addition
dfPlusg  = sym.diff(f+g)

display(Math('(f+ g)\' = %s' %sym.latex(sym.expand(dfPlusg))))
display(Math('f\' + g\' = %s' %sym.latex(sym.expand(df+dg))))
```

# VIDEO: Derivatives of trig functions

```python
q = sym.symbols('q')

print(sym.diff( sym.cos(q) ))
print(sym.diff( sym.sin(q) ))
```

```python
# show the cyclicity of sin/cos derivatives
f = sym.cos(x)

for i in range(0,8):
    display(Math('\\frac{d}{dx}%s = %s' %(sym.latex(f),sym.latex(sym.diff(f)))))
    f = sym.diff(f)
```

```python
import sympy.plotting.plot as symplot

f = sym.cos(x)

for i in range(0,4):
    if i==0:
        p = symplot(f,show=False,line_color=(i/5,i/4,i/5),label=sym.latex(f))
    else:
        p.extend( symplot(f,show=False,line_color=(i/5,i/4,i/5),label=sym.latex(f)) )
    f = sym.diff(f)

p.legend = True
p.xlim = [-3,3]
p.show()
```

### Exercise

```python
a = sym.symbols('a')

f = sym.cos(x + sym.sin(x)) + a

colors = 'brkm'

for ai in range(0,4):
    if ai==0:
        p = symplot(f.subs(a,ai),show=False,label='a=%s' %ai)
    else:
        p.extend( symplot(f.subs(a,ai),line_color=colors[ai],show=False,label='a=%s' %ai) )

p.title = 'The functions'
p.legend = True
p.show()

for ai in range(0,4):
    if ai==0:
        p = symplot(sym.diff(f.subs(a,ai)),show=False,label='a=%s' %ai)
    else:
        p.extend( symplot(sym.diff(f.subs(a,ai)),line_color=colors[ai],show=False,label='a=%s' %ai) )

p.title = 'Their derivatives'
p.legend = True
p.show()
```

# VIDEO: Graphing a function tangent line

```python
x = sym.symbols('x')

# define function and its derivative
f = x**2
df = sym.diff(f)

# select x point
xa = 1

# define function and derivative values at that point
fa = f.subs(x,xa)
df_a = df.subs(x,xa)

## evaluate function and its derivative
xx     = np.linspace(-2,2,200)
f_fun  = sym.lambdify(x,f)(xx)
df_fun = sym.lambdify(x,df)(xx)

# compute the tangent line
tanline = df_a * (xx - xa) + fa

# plot it!
plt.plot(xx,f_fun,label='f(x)')
plt.plot(xx,tanline,label='tangent')
plt.plot(xa,fa,'ro')

plt.axis('square')
plt.axis([-3,3,-3,3])
ax = plt.gca()
plt.plot(ax.get_xlim(),[0,0],'k--')
plt.plot([0,0],ax.get_xlim(),'k--')
plt.xlabel('X')
plt.ylabel('f(x)')
plt.legend()

plt.show()
```

### Exercise

```python
# make a function that computes the tangent line, loop through lots of points

def computetangent(f,xa,bnds):
    # define function and derivative values at that point
    df = sym.diff(f)
    fa = f.subs(x,xa)
    df_a = df.subs(x,xa)

    # evaluate the tangent line
    xx = np.linspace(bnds[0],bnds[1],200)
    return df_a * (xx - xa) + fa
```

```python
x = sym.symbols('x')
f = x**2
xx = np.linspace(-2,2,200)
f_fun  = sym.lambdify(x,f)(xx)

for i in xx:
    yy = computetangent(f,i,xx[[0,-1]])
    plt.plot(xx,yy,color=[abs(i)/3,abs(i)/4,abs(i)/2])

# plot it!
plt.plot(xx,f_fun,'k',linewidth=2)

plt.axis('square')
plt.axis([-2,2,-1,3])
plt.axis('off')
plt.show()
```

# VIDEO: Finding critical points of a function

```python
# The empirical method (useful for df=0; won't work for non-differentiable points)

# create a function
x = np.linspace(-5,5,1000) # vs 1001
fx = x**2 * np.exp(-x**2)

# extrema
localmax = find_peaks(fx)[0]
localmin = find_peaks(-fx)[0]
print('The critical points are ' + str(x[localmax]) + ' ' + str(x[localmin]))

# compute its derivative
dfx = np.diff(fx)/np.mean(np.diff(x)) # scale by dx!

# plot everything
plt.plot(x,fx,label='y')
plt.plot(x[0:-1],dfx,label='dy/dx')
plt.plot(x[localmax],fx[localmax],'ro',label='local max.')
plt.plot(x[localmin],fx[localmin],'gs',label='local min.')
plt.plot(x[[0,-1]],[0,0],'--',c=[.7,.7,.7])

plt.legend()
plt.xlim(x[[0,-1]])
plt.show()
```

```python
# The analytic (symbolic) method

x = sym.symbols('x')
fx = x**2 * sym.exp(-x**2)

# derivative in sympy, solve
dfx = sym.diff(fx,x)
critpoints = sym.solve(dfx)
print('The critical points are: ' + str(critpoints))

# some sympy plotting
p = sym.plot(fx,(x,-5,5),show=False)
p.extend( sym.plot(dfx,(x,-5,5),show=False,line_color='r') )

p[0].label = 'y'
p[1].label = 'dy/dx'
p.legend = True
p.show()
```

### Exercise

```python
# what values of 'a' give this function a critical point at x=1 or x=2?
a,x = sym.symbols('a,x')

baseexpr = x**2 * sym.exp(-a*x**2)
arange = np.arange(0,2.25,.25)
xrange = np.linspace(-3,3,100)

# setup plots
fig,ax = plt.subplots(1,2)

for ai in arange:

    fx = baseexpr.subs(a,ai)
    dfx = sym.diff(fx)
    critpnts = sym.solve( dfx )

    # also plot the function in subplot1 and its derivative in subplot2
    ax[0].plot(xrange,sym.lambdify(x,fx)(xrange))
    ax[1].plot(xrange,sym.lambdify(x,dfx)(xrange))

    if 1 in critpnts:
        display(Math('\\Rightarrow %s\\text{ has a critical point at x=1! Woohoo!!}' %sym.latex(fx)))
    elif 2 in critpnts:
        display(Math('\\Rightarrow %s\\text{ has a critical point at x=2! Woohoo!!}' %sym.latex(fx)))
    else:
        display(Math('\\quad %s\\text{ has NO critical point at x=2. :(}' %sym.latex(fx)))

# some adjustments to the function plot
ax[0].set_ylim([0,2])
ax[0].set_title('Function')
ax[0].plot([1,1],[0,2],'--',c='gray')
ax[0].plot([2,2],[0,2],'--',c='gray')

# adjustments to the derivative plot
ax[1].set_ylim([-1.5,1.5])
ax[1].plot(xrange[[0,-1]],[0,0],'--',c='gray')
ax[1].plot([1,1],[-1.5,1.5],'--',c='gray')
ax[1].plot([2,2],[-1.5,1.5],'--',c='gray')
# ax[1].set_xlim([.5,2.5])
ax[1].set_title('Its derivative')
fig.set_size_inches(8,4)

plt.show()
```

# VIDEO: Partial derivatives

```python
from sympy.abc import x,y

f = x**2 + x*y**2

display(Math('\\frac{\\partial f}{\\partial x} = %s' %sym.latex(sym.diff(f,x))))
display(Math('\\frac{\\partial f}{\\partial y} = %s' %sym.latex(sym.diff(f,y))))
```

### Exercise

```python
p = sym.plotting.plot3d(f,(x,-3,3),title='$f(x,y)=%s$' %sym.latex(f))
p = sym.plotting.plot3d(sym.diff(f,x),(x,-3,3),(y,-3,3),title='$f_x=%s$' %sym.latex(sym.diff(f,x)))
p = sym.plotting.plot3d(sym.diff(f,y),(x,-3,3),(y,-3,3),title='$f_y=%s$' %sym.latex(sym.diff(f,y)))
```

# VIDEO: Indefinite and definite integrals

```python
x = sym.symbols('x')

# a simple function
f = x

# indefinite integration
sym.integrate(f) # notice no constant term
```

```python
# definite integation (within bounds)
sym.integrate(f,(x,0,1))
```

```python
# plotting...

lowbnd = 2 # add later

f = x**3 / (x-lowbnd)
intf = sym.integrate(f)
p = sym.plotting.plot(f,show=False)
p.extend( sym.plotting.plot(intf,(x,-10,0),show=False,line_color='r') )

p.xlim = [-10,10]
p.ylim = [-200,200]

p[0].label = '$f(x)$'
p[1].label = '$\\int f(x)dx$'
p.legend = True
p.show()

display(Math('f(x) = %s' %sym.latex(f)))
display(Math('\\int f(x) dx = %s' %sym.latex(intf)))
```

### Exercise

```python
# show that the antiderivative (indefinite integral) of the derivative is the original function

f = 2*x**3 + sym.sin(x)

df = sym.diff(f)
idf = sym.integrate(df)

display(Math('f(x)=%s' %sym.latex(f)))
display(Math('f\'=%s' %sym.latex(df)))
display(Math('\\int (f\')dx=%s' %sym.latex(idf)))
```

# VIDEO: Area between two curves

```python
x = sym.symbols('x')

symf = x**2
symg = x

f = sym.lambdify(x,symf)
g = sym.lambdify(x,symg)

xx = np.linspace(-2,2,40)

plt.plot(xx,f(xx))
plt.plot(xx,g(xx),'r')

plt.legend(['$f(x)=%s$'%sym.latex(symf),'$g(x)=%s$'%sym.latex(symg)])
plt.axis([-.25,1.25,-.5,1.5])
plt.show()
```

```python
from matplotlib.patches import Polygon

xinter = np.linspace(0,1,100)
points = np.vstack((g(xinter),f(xinter))).T
p = Polygon(points,facecolor='k',alpha=.3)

fig, ax = plt.subplots()
ax.add_patch(p)

plt.plot(xx,f(xx))
plt.plot(xx,g(xx),'r')

plt.legend(['$f(x)=%s$'%sym.latex(symf),'$g(x)=%s$'%sym.latex(symg)])
plt.axis([-.25,1.25,-.5,1.5])
plt.show()
```

### Exercise

### Computing the area between two functions
The area between two functions is given by the formula $A = \int_{a}^{b}f(x) - g(x) dx$

In our example, $f(x)=x^2$ and $g(x)=x$

Therefore, $A = \int_{a}^{b}(x^2 - x) dx$

We will compute the area between the two crossing points, that is, where the two functions are equal. This is given by the two solutions to the equation
$$x^2=x$$

The two solutions are $x=0$ and $x=1$. This gives us the definite integral of
$$A = \int_{0}^{1}(x^2 - x) dx$$

The final answer will be given in the code below.

```python
# find intersection points
fg_intersect = sym.solve(symf-symg)
display(Math('f(x)\\text{ and }g(x) \\text{ intersect at } x=%s,%s' %(fg_intersect[0],fg_intersect[1])))

# compute area and print results!
A = sym.integrate(symf-symg,(x,fg_intersect[0],fg_intersect[1]))
display(Math('\\text{The area between the functions is } A = %s' %A))
```

# VIDEO: Calculus BUG HUNT!!

```python
# evaluate a function in a range
from sympy.abc import x

fx = (4*x**3 + 2*x**2 - x) / (-4*x**4 + 2*x**2)

xrange = np.linspace(-2,2,200)
fxx = sym.lambdify(x,fx)

plt.plot(xrange,fxx(xrange))
plt.ylim([-20,20])
plt.xlim(xrange[[0,-1]])
plt.show()
```

```python
# compute the limit
x = sym.symbols('x')
fx = 1/(x+3)

lim_pnt = -3
lim = sym.limit(fx,x,lim_pnt,dir='+')

display(Math('\\lim_{x\\to %g^+} %s = %s' %(lim_pnt,sym.latex(fx),sym.latex(lim))))
p = sym.plotting.plot(fx,show=False)
p.ylim = [-10,10]
p.show()
```

```python
# piece-wise function
from sympy.abc import x

piece1 = x**2
piece2 = 4*sym.exp(-x**2)

# put them together with conditions
fx = sym.Piecewise( (piece1,x<0) , (piece2,x>=0) )

# evaluate the function in a range
xx = np.linspace(-2,2,1000)
fxx = sym.lambdify(x,fx)

# show it in a plot
plt.plot(xx,fxx(xx),'k')

plt.show()
```

```python
# show the first and second derivatives of sin(x)
x = np.linspace(-2*np.pi,2*np.pi,200)
dt = np.diff(x[0:2])

y = np.sin(x)
dy = np.diff(y)
ddy = np.diff(y,2)

plt.plot(x,y,label='y')
plt.plot(x[0:-1],dy/dt,'--',label='dy',alpha=.6)
plt.plot(x[0:-2],ddy/dt**2,':',label='d$^2$y',alpha=.3)

plt.legend(framealpha=1)
plt.show()
```

```python
# Compute critical points using sympy

x = sym.symbols('x')
fx = x**2 * sym.exp(-x**2)

# derivative in sympy, solve
dfx = sym.diff(fx,x)
critpoints = sym.solve(dfx)
print('The critical points are: ' + str(critpoints))

# plot the function derivative and its critical points
y = sym.lambdify(x,dfx)
xx = np.linspace(-3,3,200)

plt.plot(xx,y(xx))
plt.plot([-3,3],[0,0],'k--')
plt.xlim([-3,3])

for i in critpoints:
    plt.plot(i,0,'ro')

plt.title('Function derivative')
plt.show()
```

```python
# Compute the area between two curves (not the same thing as Between Two Ferns)
from matplotlib.patches import Polygon

x = sym.symbols('x')
f1sym = sym.cos(x)
f2sym = x

xx = np.linspace(0,np.pi/3,100)
f1 = np.cos(xx)
f2 = xx

fintersect = np.argmin(abs(f1-f2))

# compute area
A = sym.integrate(f1sym-f2sym,(x,xx[0],xx[fintersect]))

traceX = np.concatenate((xx[0:fintersect],xx[fintersect:0:-1]))
traceY = np.concatenate((f1[0:fintersect],f2[fintersect:0:-1]))

points = np.vstack((traceX,traceY)).T
p = Polygon(points,facecolor='k',alpha=.3)

fig, ax = plt.subplots()
ax.add_patch(p)

plt.plot(xx,f1, xx,f2)
plt.title('The shaded area is %s' %A)
plt.show()

```


Numerical Methods  -- Assignment 6

## Problem 1 -- Minimisation

### (a) Brent's method

In numerical analysis, Brent's method is a root-finding algorithm combining the bisection method, the secant method and inverse quadratic interpolation. It has the reliability of bisection but it can be as quick as some of the less-reliable methods. Now given the following function:
\begin{equation}
f(x) = \frac{(x+3)^2}{2}+\frac{x-5}{3}-\frac{10}{4}
\end{equation}

```python
%config InlineBackend.figure_format = 'retina'
from scipy.optimize import brent
import numpy as np
import matplotlib.pyplot as plt

f = lambda x: 0.5*np.power(x+3,2)+(x-5)/3-10./4
minimum = brent(f,brack=(-100,100))
x = np.linspace(-100,100,100)
y = f(x)
print "The minimum of the function is found to be:",minimum
#Plot to check the shape of the function
plt.plot(x,y,'-')
plt.legend('f(x)')
plt.show()
```

### (b) Conjugate gradient method

$$f(x,y) = 0.26((x+0.5)^2+(y-1)^2)-0.48xy$$

```python
from scipy.optimize import fmin_cg,minimize

def func(x):
    return 0.26*((x[0]+0.5)**2+(x[1]-1)**2)-0.48*x[0]*x[1]

def fprime(x):
    return np.array((0.26*2*(x[0]+0.5)-0.48*x[1],0.26*2*(x[1]-1)-0.48*x[0]))
fc = lambda x: 0.26*(np.power(x[0]+0.5,2)+np.power(x[1]-1,2))-0.48*x[0]*x[1]
fp = lambda x: np.array((0.26*2*(x[0]+0.5)-0.48*x[1],0.26*2*(x[1]-1)-0.48*x[0]))
x0 = [2,5]
fmin = minimize(fc,x0,jac=fp,method='CG')
print "The minimum of the function is found at (x,y)=:",fmin.x
```

    The minimum of the function is found at (x,y)=: [ 2.86015324  3.64015324]

```python
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.pyplot import figure, show
import numpy as np

x = np.linspace(-100,100,100)
y = x
xx,yy = np.meshgrid(x,y)
X = [xx,yy]
zz = func(X)
#plot
fig = plt.figure(figsize=(10,6))
frame = fig.add_subplot(1,1,1,projection ='3d')
surf = frame.plot_surface(xx,yy,zz,cmap='Dark2')
frame.set_xlabel('x')
frame.set_ylabel('y')
frame.set_zlabel('z')
frame.set_title('f(x,y)')
fig.colorbar(surf,ax=frame)
show()
```

## Problem 2 -- x,y data

The $\textit{Maximum Likelihood}$ is done by minimizing the log-likelihood function.
$$ln\,L = -\frac{1}{2}\sum_{i=1}^{N}\left(\frac{(data-model)^2}{2\sigma^2}+ln\,(2\pi\sigma^2)\right)$$,
where the model is described by $y = kx+m$, and $\sigma^2 = yerr^2+f^2(kx+m)^2$.

First use $\chi^2$ fitting and reduced $\chi^2$ to get a reference of the values of parameters.

```python
from kapteyn import kmpfit

data = np.genfromtxt('Downloads/table_1.dat')
x = data[:,0]
y = data[:,1]
ey = data[:,2]
#define a model function, since assuming linear correlation
def model(params, x):
    a, b = params
    return a + b*x

# A residuals function
def residuals(p, data):  # Residuals function needed by kmpfit
    x, y, err= data
    return (y - model(p,x))/err

p0 = (1, 1)
fitobj = kmpfit.Fitter(residuals=residuals, data=(x,y,ey))
status = fitobj.fit(params0=p0)
print("Was fit successful?", bool(status))

a, b = fitobj.params
da, db = fitobj.stderr
s = u"Model parameters a={} \u00B1 {} and b={} \u00B1 {}".format(a, da, b, db)
print(s)
print("Chi squared, reduced Chi squared:", fitobj.chi2_min, fitobj.rchi2_min)
print("The errors derived from the covariance matrix:", fitobj.xerror)

```

    ('Was fit successful?', True)
    Model parameters a=3.63163709836 ± 0.657618505171 and b=0.528201941419 ± 0.106155936616
    ('Chi squared, reduced Chi squared:', 4033.3942262757655, 84.02904638074511)
    ('The errors derived from the covariance matrix:', array([ 0.07173966,  0.01158056]))

```python
from matplotlib.pyplot import figure, show

fig = figure()
frame = fig.add_subplot(1,1,1)
frame.errorbar(x, y, yerr=ey,marker='o', ls='')
xmin = x.min(); xmax = x.max()
p = (a,b)
ymin = model(p,xmin); ymax = model(p,xmax)
frame.plot([xmin, xmax], [ymin, ymax])
frame.set_xlabel('Distance (Mpc)')
frame.set_ylabel('Velocity (km/s)')
show()
```

### (a) -- Optimal parameters

```python
from math import *
import time

def log_like(params,x,y,ey):
    k,m,f = params
    sig = np.sqrt(ey**2+(f*(k*x+m))**2)
    L = ((k*x+m)-y)**2/(sig**2)+np.log(2*pi*sig**2)
    return 0.5*np.sum(L)
#minimize via three methods
t1 = time.time()
l_BFGS = minimize(log_like,x0=[0,0,0],args=(x,y,ey),method='BFGS')
t2 = time.time()
t3 = time.time()
l_NM = minimize(log_like,x0=[0,0,0],args=(x,y,ey),method='Nelder-Mead')
t4 = time.time()
t5 = time.time()
l_CG = minimize(log_like,x0=[0,0,0],args=(x,y,ey),method='CG')
t6 = time.time()
print '%6s %6s %6s %6s %12s %12s' % ("Methods","k","m","f","time","Success")
print '%6s %.7f %.7f %.7f %.6f %8s' % ("BFGS",l_BFGS.x[0],l_BFGS.x[1],l_BFGS.x[2],t2-t1,l_BFGS.success)
print '%6s %.7f %.7f %.7f %.6f %8s' % ("NM",l_NM.x[0],l_NM.x[1],np.abs(l_NM.x[2]),t4-t3,l_NM.success)
print '%6s %.7f %.7f %.7f %.6f %8s' % ("CG",l_CG.x[0],l_CG.x[1],l_CG.x[2],t6-t5,l_CG.success)
```

    Methods      k      m      f         time      Success
      BFGS 0.6850430 3.3064525 0.2998975 0.014389     True
        NM 0.6850535 3.3064455 0.2998962 0.021051     True
        CG 0.6850429 3.3064530 0.2998975 0.013389     True

The table printed above listed the optimization results via the three methods. In this case CG method performs the fastest. All three methods uses x,y and error in y to minimize the log-likelihood function.

### (b) -- Plot fitting

```python
fig = figure()
frame = fig.add_subplot(1,1,1)
frame.errorbar(x, y, yerr=ey,marker='*', fillstyle='none',ls='',color='indigo')
xmin = x.min(); xmax = x.max()
p1 = [l_BFGS.x[1],l_BFGS.x[0]]
p2 = [l_NM.x[1],l_NM.x[0]]
p3 = [l_CG.x[1],l_CG.x[0]]
ymin1 = model(p1,xmin); ymax1 = model(p1,xmax)
ymin2 = model(p2,xmin); ymax2 = model(p2,xmax)
ymin3 = model(p3,xmin); ymax3 = model(p3,xmax)
frame.plot([xmin, xmax], [ymin1, ymax1])
frame.plot([xmin, xmax], [ymin1, ymax1])
frame.plot([xmin, xmax], [ymin1, ymax1])
frame.set_xlabel('x')
frame.set_ylabel('y')
frame.legend(['BFGS','NM','CG','data'])
show()
```

Notice that because of small deviations the three fits overlay on each other and is not distinguashiable on the graph above.

## Problem 3 -- Dispersion relation

```python
data = np.genfromtxt('./Downloads/table_2.dat')
x = data[:,0]
y = data[:,1]
v = data[:,2]
ve = data[:,3]
```

### (a) -- Minimization

```python
def L(params, x,y,v,ve):
    e0, vsys = params
    v_model = vsys    #model without A
    sigma = np.sqrt(e0**2 + ve**2)
    L = ((v - v_model)**2/ (sigma**2) + np.log(2*pi*sigma**2))
    return 0.5 * np.sum(L)
log_BFGS = minimize(L,x0=[1,1],args=(x,y,v,ve))
print "Was minimization successful?",log_BFGS.success
print '%6s %6s' % ('sig0(rad)','v_sys(km/s)')
print '%.3f %.3f' % (log_BFGS.x[0],log_BFGS.x[1])

```

    Was minimization successful? True
    sig0(rad) v_sys(km/s)
    127.074 134.081

### (b) -- Minimization (with $A$ and $\theta_0$)

```python
def logL(params,x,y,v,ve):
    e0, vsys, A, theta0 = params
    theta = np.arctan2(y,x)
    v_model = vsys + A*np.sin(theta-theta0)    #model with A
    sigma = np.sqrt(e0**2 + ve**2)
    L = ((v - v_model)**2/ (sigma**2) + np.log(2*pi*sigma**2))
    return 0.5 * np.sum(L)

ln_BFGS = minimize(logL,x0=[1,1,1,1],args=(x,y,v,ve))
print "Was minimization successful?",ln_BFGS.success
print '%6s %6s %4s %8s' % ('sig0(rad)','v_sys(km/s)','A(km/s)','theta0(rad)')
print '%.6f %.6f %.6f %.6f' % (ln_BFGS.x[0],ln_BFGS.x[1],ln_BFGS.x[2],ln_BFGS.x[3])

```

    Was minimization successful? True
    sig0(rad) v_sys(km/s) A(km/s) theta0(rad)
    120.918200 132.610064 56.638726 0.812978

### (c) -- MCMC

```python
import emcee

def LMC(params,x,y,v,ve):
    e0, vsys, A, theta0 = params
    theta = np.arctan2(y,x)
    v_model = vsys + A*np.sin(theta-theta0)
    sigma = np.sqrt(e0**2 + ve**2)
    L = ((v - v_model)**2/ (sigma**2) + np.log(2*pi*sigma**2))
    if A > 0: #A stays positive
        return -0.5 * np.sum(L)
    else:
        return -np.inf
nwalkers, ndim = 500, 4
p0 = np.zeros(shape=(500,4))
for i in range(500):
    p0[i,0] = ln_BFGS.x[0]+1e-4*np.random.uniform(-ln_BFGS.x[0]/10, ln_BFGS.x[0]/10)
    p0[i,1] = ln_BFGS.x[1]+1e-4*np.random.uniform(-ln_BFGS.x[1]/10, ln_BFGS.x[1]/10)
    p0[i,2] = ln_BFGS.x[2]+1e-4*np.random.uniform(-ln_BFGS.x[2]/10, ln_BFGS.x[2]/10)
    p0[i,3] = ln_BFGS.x[3]+1e-6*np.random.uniform(-ln_BFGS.x[3]/10, ln_BFGS.x[3]/10)

l = emcee.EnsembleSampler(nwalkers, ndim, LMC, args=[x,y,v,ve])

MCMC = l.run_mcmc(p0, 500)
e0 = [i[0] for i in MCMC[0]]
vsys = [i[1] for i in MCMC[0]]
A = [i[2] for i in MCMC[0]]
theta0 = [i[3] for i in MCMC[0]]

#print the result
print '%6s %6s %4s %8s' % ('sig0(rad)','v_sys(km/s)','A(km/s)','theta0(rad)')
print '%.6f %.6f %.6f %.6f' % (np.mean(e0),np.mean(vsys),np.mean(A),np.mean(theta0))

```

    sig0(rad) v_sys(km/s) A(km/s) theta0(rad)
    124.025509 131.732557 54.471844 0.806364

### (d) -- Histogram

```python
plt.hist(e0, bins=100)
plt.xlabel(r"$\sigma_0(rad)$")
plt.ylabel('occurance')
plt.axvline(x=ln_BFGS.x[0],color='#9370db',ls='--')
plt.axvline(x=np.mean(e0),color='#b22222',ls='--')
plt.legend(['BFGS','MCMC'])
plt.show()
```

```python
plt.hist(vsys, bins=100)
plt.xlabel(r"$v_{sys}(km/s)$")
plt.ylabel('occurance')
plt.axvline(x=ln_BFGS.x[1],color='#9370db',ls='--')
plt.axvline(x=np.mean(vsys),color='#b22222',ls='--')
plt.legend(['BFGS','MCMC'])
plt.show()
```

```python
plt.hist(A, bins=100)
plt.xlabel(r"$A(km/s)$")
plt.ylabel('occurance')
plt.axvline(x=ln_BFGS.x[2],color='#9370db',ls='--')
plt.axvline(x=np.mean(A),color='#b22222',ls='--')
plt.legend(['BFGS','MCMC'])
plt.show()
```

```python
plt.hist(theta0, bins=100)
plt.xlabel(r"$\theta_0(rad)$")
plt.ylabel('occurance')
plt.axvline(x=ln_BFGS.x[3],color='#9370db',ls='--')
plt.axvline(x=np.mean(theta0),color='#b22222',ls='--')
plt.legend(['BFGS','MCMC'])
plt.show()
```

### (e) -- $v_{rot} \ vs \theta$

```python
from matplotlib.pyplot import figure,show

params1 = [ln_BFGS.x[1],ln_BFGS.x[2],ln_BFGS.x[3]]
params2 = [np.mean(vsys),np.mean(A),np.mean(theta0)]

def v_model(params,x,y):
    vsys, A, theta0 = params
    theta = np.arctan2(y,x)
    v_rot = vsys + A*np.sin(theta-theta0)
    return theta, v_rot
Theta1, V1 = v_model(params1,x,y)
Theta2, V2 = v_model(params2,x,y)

#plot data and fits
fig = figure(figsize=(12,9))
frame = fig.add_subplot(1,1,1)
frame.errorbar(np.arctan2(y,x),v,ve,marker = 'd',ls='',color='indigo',fillstyle='none')
frame.plot(Theta1,V1,'.',color = 'darkorange')
frame.plot(Theta2,V2,'.',color='#b22222')
frame.set_xlabel(r'$\theta(radian)$')
frame.set_ylabel(r'$v_{rot}(km/s)$')
frame.legend(['data','BFGS','MCMC'])
show()
```

### (f) -- Sampling method

emcee.EnsembleSampler uses Monte Carlo Markov Chain (MCMC) moethods with $affine\ invariance$. Unlike most single-variable MCMC meothds which force the perturbations of order $\sqrt{\epsilon}$ to slowly come to equilibration, EnsembleSampler would use perturbations of order $\sqrt{\epsilon}$ in the $(1,-1)$ direction and perturbations of order one in the $(1,1)$ direction.

A general MCMC sampler is of the form
$$X(t+1) = R(X(t),\xi(t),\pi),$$
where $X(t)$ is the sample after $t$ iterations, $\xi(t)$ is a sequence of iid(independent identically distributed) random variables, and $\pi$ is a probability density. The algorithm is $affine \ invariant$ if for any affine transformation $Ax+b$,
$$R(Ax+b,\xi(t),\pi_{A,b})=AR(x(t),\xi(t),\pi)+b,$$
for every $x$ and almost all $\xi(t)$.

```python

```


```python
%matplotlib widget
```

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
import matplotlib.cm as cm
from IPython.display import display, Math, clear_output

import sympy
from sympy import *
from sympy.physics.vector import ReferenceFrame, CoordinateSym
from sympy.vector import CoordSys3D, divergence, curl

import ipyvolume as ipv
import time
from ipywidgets import Output, interact
import ipywidgets as widgets
np.seterr(divide='ignore', invalid='ignore')
init_printing()
```

## Coaxial Cable Challenge:
* We will try to develop a coaxial cable that is perfect for our application. In the process we may discover that perfect depends on the context.

### Coaxial Cable Design
A coaxial cable consists of
* Two conductors:
    1. A centre core solid internal cylinder which "carries the signal".
    2. A metallic child hollow outer cylinder which is held at ground potential.
* A dielectric insulator between the two conductors.

By Tkgd2007 - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=4138900

***
### There are **three critical parameters** of the cable that we will use our electromagnetism skills to study.
1. The **capacitance**: $ C = \dfrac{Q}{V}$ which relates the amount of charge needed on the centre core to charge it to a certain voltage.
2. The **inductance**: $ L = \dfrac{V_{drop}}{(\dfrac{dI}{dt})}$
3. And the **resistance**.

* The voltage $V$ in the capacitance expression is the voltage of the centre core with respect to ground.
* The voltage $V_{drop}$ is the voltage difference between two segments of the centre core that is induced by a time change of current.

Referring to Griffiths problem 2.43 find the capacitance per unit length of two coaxial cylindrical tubes:

## Solve for the Capacitance
#### Step 1: ** Find the E-Field ** (use cylindrical coordinates):

$\int \mathbf{E}\cdot{\mathbf{dA}} = \dfrac{Q_{enc}}{\epsilon}$
By symmetry the E-field is parallel to the normal vector of the cylindrical surface (assume $Q$ charge over a length of $L$).

So: $E(2\pi s*L) = \dfrac{Q}{\epsilon} \Rightarrow$ $\mathbf{E} = \dfrac{Q}{2\pi s L  \epsilon} \hat{\mathbf{s}} $

#### Step 2: Find the potential. Capacitance depends on the voltage so let's find the potential between the inner and outer cylinders.

$-\int_a^b \mathbf{E}\cdot \mathbf{dl} = V(b) - V(a) = -\dfrac{Q}{2\pi L  \epsilon} \int_a^b \dfrac{ds}{s} \Rightarrow$

$ V(a) - V(b) = \dfrac{Q}{2\pi L  \epsilon} \ln{\left(\dfrac{b}{a} \right)} $

Capacitance ($C$) per unit length ($l$):
$ \dfrac{C}{l} = \dfrac{Q}{V} \dfrac{1}{l} \Rightarrow \dfrac{C}{l} = \dfrac{2 \pi \epsilon}{\ln{\left(\dfrac{b}{a} \right)}} $

## Solve for the Inductance
#### Step 1: ** Find the B-Field within the cable ** (use cylindrical coordinates and an Amperian Loop):

$\oint \mathbf{B}\cdot{\mathbf{dl}} = \mu I$
By symmetry the E-field is parallel to the circular Amperian loop of radius $r$.
$B(2\pi r) = \mu I$

So: $\mathbf{B} = \dfrac{\mu I}{2 \pi r} \hat{\mathbf{\phi}} $

#### Step 2: Find the magnetic flux through a loop between the two conductors.

$\Phi = \int \mathbf{B}\cdot \mathbf{dA} = \int_{r=a}^{r=b} \dfrac{\mu I}{2 \pi r} l dr = \dfrac{\mu I}{2 \pi} l \ln{\left(\dfrac{b}{a} \right)}$

Inductance ($L$) per unit length ($l$):
$ V_{drop} = \dfrac{d\Phi}{dt} = L \dfrac{dI}{dt} $

$ \dfrac{d\Phi}{dt} = \dfrac{\mu}{2 \pi} l \ln{\left(\dfrac{b}{a} \right) \dfrac{dI}{dt}}$

$ \dfrac{L}{l} = \dfrac{\mu}{2 \pi} \ln{\left(\dfrac{b}{a} \right)}$

## Plot the Capacitance and Inductance Versus the inner diameter. Use:
1. a slider to adjust the outer diameter and
2. a drop down box to adjust the  dielectric.

```python
plt.close('all')
a = np.linspace(0.05, 2, 1000) # inner radius [in mm]

fig = plt.figure(figsize = (11,8))

# constants and equation for capacitance
e0 = 8.85e-12
b = 3
cap = 2*np.pi*e0/np.log(b/a)*1e12
dielectrics = {'air': 1, 'polyethylene': 2.25, 'teflon_PTFE': 2.1}

# plot capacitance
ax1 = fig.add_subplot(1, 2, 1)
line, = ax1.plot(a, cap)
ax1.set_ylim([0, np.max(cap[np.isfinite(cap)])*1.1])
# label plot
ax1.set_title('Capacitance')
plt.ylabel('C [pF/m]')
plt.xlabel('a:inner diameter [mm]')
ax1.grid(True)

# constants and equation for inducatance
u0 = 4*np.pi*1e-7
ind = u0/(2*np.pi)*np.log(b/a)*1e9

ax2 = fig.add_subplot(1, 2, 2)
line2, = ax2.plot(a, ind)
ax2.set_ylim([0, np.max(ind[np.isfinite(ind)])*1.1])

# label plot
ax2.set_title('Inductance')
plt.ylabel('L [nH/m]')
plt.xlabel('a:inner diameter [mm]')
ax2.grid(True)

def update(b=widgets.FloatSlider(min=0.0,max=100.0,step=0.1,value=5),
           dielectric = ['air', 'polyethylene', 'teflon_PTFE']):

    # capacitance
    er = dielectrics[dielectric]
    cap = 2*np.pi*e0*er/np.log(b/a)*1e12

    # find negative values
    idx = cap<0
    line.set_ydata(cap)
    line.set_marker('*')
    fig.canvas.draw()
    ax1.set_ylim([0, np.max([np.max(cap[np.isfinite(cap)])*1.1, 0.5])])

    # inductance
    ind = u0/(2*np.pi)*np.log(b/a)*1e9

    # remove any negative values
    idx = ind<0
    line2.set_ydata(ind)
    line2.set_marker('*')
    fig.canvas.draw()
    ax2.set_ylim([0, np.max([np.max(ind[np.isfinite(ind)])*1.1, 0.5])])

interact(update);
```

    A Jupyter Widget

    A Jupyter Widget

## Plot the Resistance Versus the inner diameter (no slider needed in this case).

```python
a = np.linspace(0, 2, 1000) # inner radius [in mm]
fig = plt.figure(figsize = (11,8))
ax1 = fig.add_subplot(1, 1, 1)
ax1.set_title('Resistance -- no slider')
plt.ylabel('R [milli-Ohm]')
plt.xlabel('a:inner diameter [mm]')

# resistance
a = np.linspace(0.05, 2, 1000) # inner radius [in mm]
rho = 1.68e-8  # Ohm * m
data = rho/(np.pi * (a*1e-3)**2)
line, = ax1.plot(a, data)
line.set_marker('*')
plt.grid(True)
```

```python
plt.close('all')
```

```python

```