# CMSC6950 Assignment 3

Due on Tuesday, May 29, 2018

Solve all questions.

*Any two identical submitted assignments from different students will be considered as copies of each other.*

Save your assignment as an ipynb file with a filename ”userid_assignment3.ipynb”. [For userid, I mean your MUN user id/email .e.g. abc123 and not your student number]. Submit your assignment through the D2L dropbox.

### Name :
### Student Number:

## Question 1

Define the following vectors and matrices:

```
vec1 = np.array([ -1., 4., -9.])
mat1 = np.array([[ 1., 3., 5.], [7., -9., 2.], [4., 6., 8. ]]
```
1. You can multiply vectors by constants. Compute `vec2 = (np.pi/4) * vec1`
2. The cosine function can be applied to a vector to yield a vector of cosines.
    Compute `vec2 = np.cos(vec2)`
3. You can add vectors and multiply by scalars. Compute
    `vec3 = vec1 + 2 * vec2`
4. The Euclidean norm of a matrix or a vector is available using `la.norm`. Compute `la.norm(vec3)`
5. You can do row-column matrix multiplication. Compute the product of `mat1` and `vec3` and set `vec4` equal to the result.
6. Compute the transpose of `mat1`.
7. Compute the determinant of `mat1`.
8. Find the smallest element in `vec1`.
9. What function would you use to find the value of `j` so that `vec1[j]` is equal to the smallest element in `vec1`?
10. What expression would you use to find the smallest element of the matrix `mat1`?

```python

```

## Question 2
1. Generate and plot values (50,50) drawn from a normal distribution.  Make a plot as shown in (a).

2. Manipulate the array such that the lowest values are in the top-left corner, the highest in the bottom-right. Make a plot as shown in (b).

```python

```

## Question 3

Using the data available at [http://berkeleyearth.lbl.gov/auto/Regional/TAVG/Text/hawaii-TAVG-Trend.txt](http://berkeleyearth.lbl.gov/auto/Regional/TAVG/Text/hawaii-TAVG-Trend.txt), reproduce the following plot using Numpy, Matplotlib, and/or Scipy.

[High resolution version](http://berkeleyearth.lbl.gov/auto/Regional/TAVG/Figures/hawaii-TAVG-Trend.pdf)

Here's a nice way to ensure the data file is always available locally:

```python
import urllib.request
urllib.request.urlretrieve('http://berkeleyearth.lbl.gov/auto/Regional/TAVG/Text/hawaii-TAVG-Trend.txt',
                   'hawaii-TAVG-Trend.txt')
```

    ('hawaii-TAVG-Trend.txt', <http.client.HTTPMessage at 0x1035ff0b8>)

```python
!ls -l 'hawaii-TAVG-Trend.txt'
```

    -rw-------@ 1 jmunroe  staff  157295 22 May 11:44 hawaii-TAVG-Trend.txt

```python

```

## Question 4

1 . As you know, a magic square is a matrix all of whose row sums, column
sums and the sums of the two diagonals are the same. (One diagonal of
a matrix goes from the top left to the bottom right, the other diagonal
goes from top right to bottom left.) Show by direct computation that if
the matrix A is given by

```A=np.array([[17, 24, 1, 8, 15],
[23, 5, 7, 14, 16],
[ 4, 6, 13, 20, 22],
[10, 12, 19, 21, 3],
[11, 18, 25, 2, 9]])```

The matrix A has 5 row sums (one for each row), 5 column sums (one for
each column) and two diagonal sums. These 12 sums should all be exactly
the same, and you could verify that they are the same by printing them
and “seeing” that they are the same. It is easy to miss small differences
among so many numbers, though. Instead, verify that A is a magic
square by constructing the 5 column sums and computing the maximum
and minimum values of the column sums. Do the same for the 5 row sums,
and compute the two diagonal sums. Check that these six values are the
same. If the maximum and minimum values are the same, the flyswatter
principle says that all values are the same.

*Hints*: The function `np.diag` extracts the diagonal of a matrix, and the
function `np.fliplr` extracts the other diagonal.

```python

```

2 . From Wikipedia:
> A magic square is a $ n \times n$ square grid (where $n$ is the number of cells on each side) filled with distinct positive integers in the range {$1, 2, \ldots, n^2$)
such that each cell contains a different integer and the sum of the integers in each row, column and diagonal is equal. The sum is called the magic constant or magic sum of the magic square. A square grid with $n$ cells on each side is said to have order $n$.

Define a function that tests if a matrix A is a magic square.  Using this function, write a program that (naively) searches for all possible magic squares of order 3 by explicitly testing all possible combinations. Display all possible $3 \times 3$ magic squares.

```python

```

## Question 5

1 . Use `matplotlib.pyplot.plot` to produce a plot of the functions
$f(x) = e^{−x/10} \sin(\pi x)$ and $g(x) = x e^{−x/3}$ over the interval [0, 10].
Include labels for the x- and y-axes, and a legend explaining which line is which plot.

```python

```

2 . The shape of a limaçon  can be defined parametrically as

\begin{align}
r &= r_0 + \cos θ \\
x &= r \cos θ \\
y &= r \sin θ
\end{align}

When $r_0 = 1$, this curve is called a cardioid. Use this definition to plot
the shape of a limaçon for $r_0 = 0.8$, $r_0 = 1.0$, and $r_0 = 1.2$. Be sure to
use enough points that the curve is closed and appears smooth (except for
the cusp in the cardioid). Use a legend to identify which curve is which.

```python

```


```python
from sympy import *
```

```python
x = Symbol('x')
```

```python
init_printing(use_unicode=True)
```

```python
expr = exp(-x) * cos(x)
expr
```

```python
integrate(expr, x)
```

```python
Integral(expr, x)
```

```python
simplify(gamma(x)*gamma(1-x))
```

```python
simplify(beta(x, x))
```

```python
expand((x-3)*(x+1))
```

```python
e = (x+x**2+x**3+x**4+x**5+x**6)**3
expand(e).coeff(x, 10)
# collect(e, x).coeff(x, 10)
```

```python
beta(x, x).rewrite(gamma)
```

```python
y = Symbol('y')

```

```python
ig = Integral(exp(-x**2-y**2), (x, 0, oo), (y, 0, oo))
ig
```

```python
ig.doit()
```

```python
sin(x).series(x, 0, 5)
```

```python
f, g = symbols('f g', cls=Function)
eq = Eq(f(x).diff(x, x) + 9*f(x), 0)
eq
```

```python
dsolve(eq, f(x))
```

```python
e = 1 / (7-x**2)
e.series(x, 0, 10)
```


# Determinant formula from Cavalieri's principle

```python
# setup SymPy
from sympy import *
init_printing()
Vector = Matrix

# setup plotting
#%matplotlib inline
%matplotlib notebook
import matplotlib.pyplot as mpl
from util.plot_helpers import plot_vec, plot_vecs, plot_line, plot_plane, autoscale_arrows
```

## Two dimentions

```python
a,b, c,d = symbols('a b  c d')

# Consider the volume of the parallelegram with sides:
u1 = Vector([a,b])
u2 = Vector([c,d])

# We can compute the volume of the parallelpiped by computing the deteminant of
A = Matrix([[a,b],
            [c,d]])
A.det()
```

###  Cavalieri's principle

Mathematically, we have

$$
  D(\vec{u}_1, \  \vec{u}_2)
  =
  D(\vec{u}_1 - \alpha \vec{u}_2, \ \vec{u}_2).
$$

```python
# choose alpha so A's top right entry will be zero
alpha = symbols('alpha')

alpha = b/d
A[0,:] = A[0,:] - alpha*A[1,:]
A
```

Copmputing the area is the same as computing the product of the entries on the diagonal:

```python
A[0,0]*A[1,1]
```

```python
simplify( A[0,0]*A[1,1] )
```

**Intuition:** the coefficient $\alpha$ encodes something very important about the "alternating multi-linear" structure that deteminants and cross products embody. In words, the choice of $\alpha$ and the resulting expression $a-\frac{bc}{d}$ correponds to what happens to the first component of $\vec{u}_1$ when we make it's second component zero.

(Yeah, I know, handwavy like crazy, but better than nothing.)

## Three dimentions

```python
a,b,c, d,e,f, g,h,i = symbols('a b c  d e f  g h i')

# Consider the volume of the parellelpiped with sides:
u1 = Vector([a,b,c])
u2 = Vector([d,e,f])
u3 = Vector([g,h,i])

# We can compute the volume of the parallelpiped by computing the deteminant of
A = Matrix([[a,b,c],
            [d,e,f],
            [g,h,i]])
A.det()
```

###  Cavalieri's principle

This principle leads to the following property of the determinant

$$
  D(\vec{u}_1, \ \vec{u}_2, \ \vec{u}_3)
  =
  D(\vec{u}_1,  \ \vec{u}_2- k \vec{u}_3, \ \vec{u}_3)
  =
  D(\vec{u}_1 -\ell\vec{u}_2 - m\vec{u}_3  , \  \vec{u}_2-k \vec{u}_3, \ \vec{u}_3).
$$

In particuler we make the following particular choice for the cofficients

$$
  D(\vec{u}_1, \vec{u}_2, \vec{u}_3)
  =
  D(\vec{u}_1 - \beta \vec{u}_3 - \gamma(\vec{u}_2 - \alpha\vec{u}_3), \  \vec{u}_2 - \alpha\vec{u}_3, \ \vec{u}_3).
$$

Choosing the right coefficients $\alpha$, $\beta$, and $\gamma$ can transform the matrix $A$ into a lower triangular form, which will make computing the determinant easier.

```python
alpha, beta, gamma = symbols('alpha beta gamma')

A
```

```python
# first get rid of f by subtracting third row from second row
alpha = f/i
A[1,:]= A[1,:] - alpha*A[2,:]
A
```

```python
# second get rid of c by subtracting third row from first row
beta = c/i
A[0,:]= A[0,:] - beta*A[2,:]
A
```

```python
# third get rid of b-ch/i by subtracting second row from first row
gamma = A[0,1]/A[1,1]
A[0,:]= A[0,:] - gamma*A[1,:]
A
```

Stargting from the first row, the volume of the determinant is proporitonal to the coeffieicnt `A[0,0]`. The are of the parallelepiped formd by the first two rows is `A[0,0]*A[1,1]` (we can ignore `A[1,0]`) and the overall volume is

```python
A[0,0]*A[1,1]*A[2,2]
```

```python
simplify( A[0,0]*A[1,1]*A[2,2] )
```

```python
# I still don't know how to motivate the recusive formula except to say it turns out that way...

# I tried to think about decomposing the problem into subparts,
#  but i cannot motivate det(Ai + Aj + Ak) != det(Ai) + det(Aj) + det(Ak)
#  where Ai is the same as A but with A[0,1] and A[0,2] set to zero.

```

```python
u1 = Vector([3,0,0])
u2 = Vector([2,2,0])
u3 = Vector([3,3,3])

plot_vecs(u1,u2,u3)
plot_vec(u1, at=u2, color='k')
plot_vec(u2, at=u1, color='b')
plot_vec(u1, at=u2+u3, color='k')
plot_vec(u2, at=u1+u3, color='b')
plot_vec(u1, at=u3, color='k')
plot_vec(u2, at=u3, color='b')
plot_vec(u3, at=u1, color='g')
plot_vec(u3, at=u2, color='g')
plot_vec(u3, at=u1+u2, color='g')

autoscale_arrows()
```

    <IPython.core.display.Javascript object>

```python

```


## Exercise 1 (7 points): Kernel Functions

We already learned techniques to classify linearly separable data, e.g. with the help of an SVM. Unfortunately, not every dataset is linearly separable. There are cases where we need more complex decision surfaces. Ideally, we would still like to use the same algorithms as in the linear case. A common approach is to use so-called kernel functions which project the data
into another space so that the representations of the points in the new space become linearly separable. For example, we can project our data to a higher dimensional space.

In this exercise, we want to take a look at how this works in the case of SVMs. For this, we are going to work with a one-dimensional example consisting of four values $x_i \in \mathbb{R}$
\begin{equation} x_1 = −2, x_4 = 2 \in \omega_1 \hspace{0.5cm} and \hspace{0.5cm} x_2 = −1, x_3 = 0 \in \omega{−1} \end{equation}

assigned to either the set of positive samples $\omega_1 = {x_i | T_i = 1}$ or the set of negative samples $\omega{−1} = {x_i | T_i = −1}$. Further, we take the opportunity and analyse the decision function in a bit more detail.

1. [Python] Figure 1 shows a graphical representation of the dataset we are using here. Obviously, the data are not linearly separable. Find a projection function $\phi ∶ \mathbb{R} \rightarrow \mathbb{R}^2$, $\tilde{x} = \phi(x)$ which projects the data into the two-dimensional space so that the projected points $\tilde{x}$ become linearly separable. Apply $\phi(x)$ to all the values in the dataset, i.e. calculate $\tilde{x}_i = \phi(x_i)$ for 𝑖 = 1, 2, 3, 4

2. [Python] Once we have calculated all projections $\tilde{x}_i$, we can apply the same algorithm as in one of the previous exercises. Remember that we need the optimal Lagrange multipliers $\alpha_i^*$ and the threshold $w_0^*$ to classify new points. Train a linear SVM with the help of the svm object from the sklearn package and extract the relevant parameters, i.e.

\begin{equation}\alpha_1^∗ = ?,\hspace{0.2cm} \alpha_2^∗ = ?,\hspace{0.2cm} \alpha_3^∗ = ?,\hspace{0.2cm} \alpha_4^∗ = ? \hspace{0.5cm} and \hspace{0.5cm} w_0^∗ = ? \end{equation}
Note that you may have to take the absolute value of the extracted parameters.

3. [Python] Use the calculated parameters to classify the new point 𝑥5 = 1 based on the decision function
\begin{equation} f(\tilde{x}) = sig \left(\sum_{i=1}^4 \alpha_i^∗T_i ⟨\tilde{x}_i, \tilde{x}⟩ + w_0^∗\right)  \end{equation}
You can check your result by additionally using the predict method.

4. By now, we calculated the projection of each 𝑥 value and applied the same algorithm as usual. However, it turns out that projecting each data point is quite computational expensive when the number of points increases. Luckily, there is a so-called kernel trick which avoids the necessity of calculating any projections at all. The idea is to use a kernel function 𝑘(𝑥𝑖, 𝑥𝑗) which can be used as a replacement for the dot product of the projected data points

\begin{equation}(\tilde{x}_i, \tilde{x}⟩ = k(x_i, x_j)\end{equation}

Note that the kernel function uses the original data points $x$ and not the projections $\tilde{x}$.

    a) [Pen and Paper] Define the kernel $k(x_i, x_j)$ which calculates the dot product of your projection function $\phi(x)$.

    b) [Python] Compute the output of the value $x_5 = 1$ again but this time using only the one-dimensional values $x_i$. For this, you need to change Equation 1 so that it uses your kernel function.

5. [Python] We can apply the kernel trick not only to the decision function but also to the optimization problem itself. Remember that for this we need to optimize

\begin{equation}
L(w, w_0, \alpha_1, \alpha_2) = L(\alpha_1, \alpha_2) = \sum_{i=1}^2 \alpha_i - \frac{1}{2}\sum_{i=1}^2\sum_{j=1}^2\alpha_i\alpha_jT_iT_jx_i^Tx_j
\end{equation}

subject to the constraint
$$\sum_{i=1}^2\alpha_iT_i=0$$

Note how the dot product $\langle\tilde{x}_i, \tilde{x}_j\rangle$ appears here as well. Re-calculate the optimal parameters by using only the original values $x_i$. For this, write a custom kernel function according to the official interface which implements your previously defined kernel $k(x_i, x_j)$. When creating the svm object, you have the choice to set your own kernel. Check if the resulting parameters are indeed the same as before.

6. So far, we have only used the trained SVM to predict a label for a new data point based on the decision function of Equation 1. However, it is often useful to retrieve a prediction score as well, i.e. a value which tells us how good the prediction was. That is, we also want information about the uncertainty of the classifier.

 In the case of SVMs, we can use the signed distance to the hyperplane as an indicator. As closer a point is located to this hyperplane, as more uncertain is the classification since then the point is also in nearer proximity to the other class. On the other hand, a point far away from the hyperplane has a very clear class assignment. Remember that we can calculate the distance of a point 𝒑 to a hyperplane 𝐻 via

\begin{equation}d(p, H) = \frac{\langle w, p \rangle + w_0}{ ‖w‖_2 }\end{equation}

 using the weight vector 𝒘 and the bias $w_0$.

    a) [Pen and Paper] It turns out that we already dealt with distances before even though it was not obvious. Show that the decision function of Equation 1 without the sig(𝑥) function already calculates the unnormalized version of Equation 3, i.e. without dividing by $‖𝒘‖_2$
\begin{equation} d(\tilde{x}, H) ⋅ ‖𝒘‖_2 = \sum_{i=1}^4\alpha_i^*T_ik(x_i, x) + w_0^*\end{equation}
Hint: use the definition of 𝒘 from the script (p. 101). Note also that on the left side the projection ̃𝒙 and on the right side only the value 𝑥 is used. That is, we can calculate the (unnormalized) distance from the projected point to the hyperplane in
the projection space without actually entering this space.

    b) [Pen and Paper] In case we want the correct distance to the hyperplane, we need to divide Equation 4 by the norm ‖𝒘‖2. However, this is not an easy task if we use a kernel function where we cannot directly compute the projections (cf. Exercise 2). Show that we can use the alternative expression

\begin{equation}‖𝒘‖2 = \sqrt{\sum_{i=1}^4 \sum{j=1}^4T_iT_j\alpha_i^∗\alpha_j^∗k(x_i, x_j)}\end{equation}

instead which does not use any variable from the projection space.

    c) [Python] Calculate the signed distance of the point $\phi(x_5)$ to the hyperplane of your trained SVM without using $\phi(x_5)$ directly. Instead, use only the value $x_5 = 1$ and your kernel $k(x_i, x_j)$.

## Exercise 2 (3 points): RBF Kernel [Pen and Paper]

Kernel functions are a very neat way to handle non-linearly separable problems. They can be used in all algorithms which need only the dot product of the data points so that the kernel function can be used instead (Equation 2). The power of the kernels is that they operate implicitly in a higher-dimensional projection space without actually entering it. This gives us the
opportunity to make our data linearly separable without negative consequences in terms of computational efficiency.

There are also some standard kernels often used and one particularly famous example is a radial basis function
\begin{equation}k(x_i, x_j) = e^{−\gamma‖x_i−x_j‖_2^2} \end{equation}
controlled by the parameter $\gamma \in \mathbb{R}^+$. This kernel is a bit special. First, it does not contain any dot product so it may be a bit surprising that we can use it to replace the calculation of a dot product. Secondly, it is even more unclear what the projection space for this kernel is, i.e. how the corresponding projection function $\phi(x)$ is defined. Thirdly, it introduces a new hyperparameter $\gamma$ which is worth further investigations.

This exercise tries to bring some light into this darkness. While we are trying to achieve this,
we also take the opportunity and briefly discuss non-separable problems where the margin is
calculated by minimizing
\begin{equation} \varphi(x, \delta) = \frac{‖𝒘‖_2^2}{2} + C \sum_{i=1}^N \delta_i \end{equation}
with the slack variables $𝛿_i ≥ 0$ and the new hyperparameter 𝐶 ∈ ℝ+.

1. It turns out that Equation 5 uses an infinitely dimensional projection space. This is a bit complicated to see so we consider only the one-dimensional case where we reduce our data points to be values 𝑥 ∈ ℝ. For this case, show that the projection function is defined as

\begin{equation}\phi(x) = e^{−\gamma x^2} \left(1, \sqrt{\frac{2\gamma}{1!} 𝑥}, \sqrt{\frac{(2\gamma)^2}{2!} x_2}, \sqrt{\frac{(2\gamma)^3}{3!} x_3}, …\right) \end{equation}

Hint: use the Taylor series of the exponential function
\begin{equation}e^x = 1 + 𝑥 + \frac{x^2}{2!} + \frac{x^3}{3!} + \frac{x^4}{4!} + ... = \sum_{k=0}^\inf \frac{x^k}{k!} \end{equation}

2. We now want to turn our attention to the hyperparameters 𝛾 and 𝐶. For this, open this animation1 where you can adjust both parameters as well as the kernel type. It
shows a two-dimensional example of a non-linear separable dataset and plots the decision function of Equation 4 as a heat map.

    a) For the parameter 𝐶, fill out Table 1 by answering the following questions.
        i. How does the width of the margin change (linear kernel)? [small/large]
        ii. How does the complexity of the decision surface change? [simple/complex]
        iii. How does the accuracy on the (training) data set change? [low/high]

    b) For the parameter 𝛾, fill out Table 2 by answering the following
        i. How does the corresponding RBF function change? For this, map each function shown in Figure 2 to the corresponding parameter description. [A/B]
        ii. Each data point has a certain influence on the decision function. How strong is this influence? [small/high]


```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
```

# Unsupervised learning: Hierarchical and density-based clustering algorithms

In a previous notebook, "08 Unsupervised Learning - Clustering.ipynb", we introduced one of the essential and widely used clustering algorithms, K-means. One of the advantages of K-means is that it is extremely easy to implement, and it is also computationally very efficient compared to other clustering algorithms. However, we've seen that one of the weaknesses of K-Means is that it only works well if the data can be grouped into a globular or spherical shape. Also, we have to assign the number of clusters, *k*, *a priori* -- this can be a problem if we have no prior knowledge about how many clusters we expect to find.

In this notebook, we will take a look at 2 alternative approaches to clustering, hierarchical clustering and density-based clustering.

# Hierarchical Clustering

One nice feature of hierachical clustering is that we can visualize the results as a dendrogram, a hierachical tree. Using the visualization, we can then decide how "deep" we want to cluster the dataset by setting a "depth" threshold. Or in other words, we don't need to make a decision about the number of clusters upfront.

**Agglomerative and divisive hierarchical clustering**

Furthermore, we can distinguish between 2 main approaches to hierarchical clustering: Divisive clustering and agglomerative clustering. In agglomerative clustering, we start with a single sample from our dataset and iteratively merge it with other samples to form clusters -- we can see it as a bottom-up approach for building the clustering dendrogram.
In divisive clustering, however, we start with the whole dataset as one cluster, and we iteratively split it into smaller subclusters -- a top-down approach.

In this notebook, we will use **agglomerative** clustering.

**Single and complete linkage**

Now, the next question is how we measure the similarity between samples. One approach is the familiar Euclidean distance metric that we already used via the K-Means algorithm. As a refresher, the distance between 2 m-dimensional vectors $\mathbf{p}$ and $\mathbf{q}$ can be computed as:

\begin{align} \mathrm{d}(\mathbf{q},\mathbf{p}) & = \sqrt{(q_1-p_1)^2 + (q_2-p_2)^2 + \cdots + (q_m-p_m)^2} \\[8pt]
& = \sqrt{\sum_{j=1}^m (q_j-p_j)^2}.\end{align}

However, that's the distance between 2 samples. Now, how do we compute the similarity between subclusters of samples in order to decide which clusters to merge when constructing the dendrogram? I.e., our goal is to iteratively merge the most similar pairs of clusters until only one big cluster remains. There are many different approaches to this, for example single and complete linkage.

In single linkage, we take the pair of the most similar samples (based on the Euclidean distance, for example) in each cluster, and merge the two clusters which have the most similar 2 members into one new, bigger cluster.

In complete linkage, we compare the pairs of the two most dissimilar members of each cluster with each other, and we merge the 2 clusters where the distance between its 2 most dissimilar members is smallest.

To see the agglomerative, hierarchical clustering approach in action, let us load the familiar Iris dataset -- pretending we don't know the true class labels and want to find out how many different follow species it consists of:

```python
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data[:, [2, 3]]
y = iris.target
n_samples, n_features = X.shape

plt.scatter(X[:, 0], X[:, 1], c=y);
```

First, we start with some exploratory clustering, visualizing the clustering dendrogram using SciPy's `linkage` and `dendrogram` functions:

```python
from scipy.cluster.hierarchy import linkage
from scipy.cluster.hierarchy import dendrogram

clusters = linkage(X,
                   metric='euclidean',
                   method='complete')

dendr = dendrogram(clusters)

plt.ylabel('Euclidean Distance');
```

Next, let's use the `AgglomerativeClustering` estimator from scikit-learn and divide the dataset into 3 clusters. Can you guess which 3 clusters from the dendrogram it will reproduce?

```python
from sklearn.cluster import AgglomerativeClustering

ac = AgglomerativeClustering(n_clusters=3,
                             affinity='euclidean',
                             linkage='complete')

prediction = ac.fit_predict(X)
print('Cluster labels: %s\n' % prediction)
```

```python
plt.scatter(X[:, 0], X[:, 1], c=prediction);
```

# Density-based Clustering - DBSCAN

Another useful approach to clustering is *Density-based Spatial Clustering of Applications with Noise* (DBSCAN). In essence, we can think of DBSCAN as an algorithm that divides the dataset into subgroup based on dense regions of point.

In DBSCAN, we distinguish between 3 different "points":

- Core points: A core point is a point that has at least a minimum number of other points (MinPts) in its radius epsilon.
- Border points: A border point is a point that is not a core point, since it doesn't have enough MinPts in its neighborhood, but lies within the radius epsilon of a core point.
- Noise points: All other points that are neither core points nor border points.

A nice feature about DBSCAN is that we don't have to specify a number of clusters upfront. However, it requires the setting of additional hyperparameters such as the value for MinPts and the radius epsilon.

```python
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=400,
                  noise=0.1,
                  random_state=1)
plt.scatter(X[:,0], X[:,1])
plt.show()
```

```python
from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.2,
            min_samples=10,
            metric='euclidean')
prediction = db.fit_predict(X)

print("Predicted labels:\n", prediction)

plt.scatter(X[:, 0], X[:, 1], c=prediction);
```

# Exercise

    EXERCISE:

      Using the following toy dataset, two concentric circles, experiment with the three different clustering algorithms that we used so far: `KMeans`, `AgglomerativeClustering`, and `DBSCAN`.

Which clustering algorithms reproduces or discovers the hidden structure (pretending we don't know `y`) best?

Can you explain why this particular algorithm is a good choice while the other 2 "fail"?

```python
from sklearn.datasets import make_circles

X, y = make_circles(n_samples=1500,
                    factor=.4,
                    noise=.05)

plt.scatter(X[:, 0], X[:, 1], c=y);
```

```python
# %load solutions/20_clustering_comparison.py
```


```python
import sympy as sym
import symnum
import symnum.diffops.symbolic as sym_diffops
import symnum.numpy as snp
import numpy as np
import multiprocessing
```

```python
y = snp.array([1, 0])
σ = 1

def forward_func(x):
    return snp.array([[1, -1], [-2, 2]]) @ snp.array(
        [snp.cos(-x[1]**2 + 3 * x[0]), snp.sin(x[0] - 1)])

def neg_log_dens(x):
    return (
        snp.sum((y - forward_func(x))**2 / σ**2) +
        snp.sum(x**2)) / 2
```

## Symbolic evaluation and derivatives

```python
x = symnum.named_array('x', 2)
v = symnum.named_array('v', 2)
m = symnum.named_array('m', (2, 2))
```

```python
forward_func(x)
```

$\displaystyle \left[\begin{matrix}- \sin{\left(x[0] - 1 \right)} + \cos{\left(3 x[0] - x[1]^{2} \right)} & 2 \sin{\left(x[0] - 1 \right)} - 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\end{matrix}\right]$

```python
sym_diffops.jacobian(forward_func)(x)
```

$\displaystyle \left[\begin{matrix}- 3 \sin{\left(3 x[0] - x[1]^{2} \right)} - \cos{\left(x[0] - 1 \right)} & 2 x[1] \sin{\left(3 x[0] - x[1]^{2} \right)}\\6 \sin{\left(3 x[0] - x[1]^{2} \right)} + 2 \cos{\left(x[0] - 1 \right)} & - 4 x[1] \sin{\left(3 x[0] - x[1]^{2} \right)}\end{matrix}\right]$

```python
sym_diffops.vector_jacobian_product(forward_func)(x)(v)
```

$\displaystyle \left[\begin{matrix}v[0] \left(- 3 \sin{\left(3 x[0] - x[1]^{2} \right)} - \cos{\left(x[0] - 1 \right)}\right) + v[1] \left(6 \sin{\left(3 x[0] - x[1]^{2} \right)} + 2 \cos{\left(x[0] - 1 \right)}\right) & 2 v[0] x[1] \sin{\left(3 x[0] - x[1]^{2} \right)} - 4 v[1] x[1] \sin{\left(3 x[0] - x[1]^{2} \right)}\end{matrix}\right]$

```python
sym_diffops.jacobian_vector_product(forward_func)(x)(v)
```

$\displaystyle \left[\begin{matrix}v[0] \left(- 3 \sin{\left(3 x[0] - x[1]^{2} \right)} - \cos{\left(x[0] - 1 \right)}\right) + 2 v[1] x[1] \sin{\left(3 x[0] - x[1]^{2} \right)} & v[0] \left(6 \sin{\left(3 x[0] - x[1]^{2} \right)} + 2 \cos{\left(x[0] - 1 \right)}\right) - 4 v[1] x[1] \sin{\left(3 x[0] - x[1]^{2} \right)}\end{matrix}\right]$

```python
sym_diffops.matrix_hessian_product(forward_func)(x)(m)
```

$\displaystyle \left[\begin{matrix}m[0, 0] \left(\sin{\left(x[0] - 1 \right)} - 9 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) + 6 m[0, 1] x[1] \cos{\left(3 x[0] - x[1]^{2} \right)} + m[1, 0] \left(- 2 \sin{\left(x[0] - 1 \right)} + 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) - 12 m[1, 1] x[1] \cos{\left(3 x[0] - x[1]^{2} \right)} & 6 m[0, 0] x[1] \cos{\left(3 x[0] - x[1]^{2} \right)} + m[0, 1] \left(- 4 x[1]^{2} \cos{\left(3 x[0] - x[1]^{2} \right)} + 2 \sin{\left(3 x[0] - x[1]^{2} \right)}\right) - 12 m[1, 0] x[1] \cos{\left(3 x[0] - x[1]^{2} \right)} + m[1, 1] \left(8 x[1]^{2} \cos{\left(3 x[0] - x[1]^{2} \right)} - 4 \sin{\left(3 x[0] - x[1]^{2} \right)}\right)\end{matrix}\right]$

```python
neg_log_dens(x)
```

$\displaystyle \frac{x[0]^{2}}{2} + \frac{x[1]^{2}}{2} + \frac{\left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right)^{2}}{2} + \frac{\left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right)^{2}}{2}$

```python
sym_diffops.grad(neg_log_dens)(x)
```

$\displaystyle \left[\begin{matrix}x[0] + \frac{\left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(- 12 \sin{\left(3 x[0] - x[1]^{2} \right)} - 4 \cos{\left(x[0] - 1 \right)}\right)}{2} + \frac{\left(6 \sin{\left(3 x[0] - x[1]^{2} \right)} + 2 \cos{\left(x[0] - 1 \right)}\right) \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right)}{2} & 4 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + x[1]\end{matrix}\right]$

```python
sym_diffops.hessian(neg_log_dens)(x)
```

$\displaystyle \left[\begin{matrix}\left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(2 \sin{\left(x[0] - 1 \right)} - 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) + \left(- \sin{\left(x[0] - 1 \right)} + 9 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) + \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right)^{2} + \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right)^{2} + 1 & 12 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1] \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 6 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \cos{\left(3 x[0] - x[1]^{2} \right)}\\12 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1] \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 6 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \cos{\left(3 x[0] - x[1]^{2} \right)} & - 8 x[1]^{2} \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1]^{2} \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 20 x[1]^{2} \sin^{2}{\left(3 x[0] - x[1]^{2} \right)} + \left(- 8 \sin{\left(x[0] - 1 \right)} + 8 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)} - 2\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 1\end{matrix}\right]$

```python
sym_diffops.hessian_vector_product(neg_log_dens)(x)(v)
```

$\displaystyle \left[\begin{matrix}v[0] \left(\left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(2 \sin{\left(x[0] - 1 \right)} - 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) + \left(- \sin{\left(x[0] - 1 \right)} + 9 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) + \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right)^{2} + \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right)^{2} + 1\right) + v[1] \left(12 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1] \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 6 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \cos{\left(3 x[0] - x[1]^{2} \right)}\right) & v[0] \left(12 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1] \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 6 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \cos{\left(3 x[0] - x[1]^{2} \right)}\right) + v[1] \left(- 8 x[1]^{2} \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1]^{2} \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 20 x[1]^{2} \sin^{2}{\left(3 x[0] - x[1]^{2} \right)} + \left(- 8 \sin{\left(x[0] - 1 \right)} + 8 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)} - 2\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 1\right)\end{matrix}\right]$

```python
sym_diffops.matrix_tressian_product(neg_log_dens)(x)(m)
```

$\displaystyle \left[\begin{matrix}m[0, 0] \left(\left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(54 \sin{\left(3 x[0] - x[1]^{2} \right)} + 2 \cos{\left(x[0] - 1 \right)}\right) + \left(- 2 \sin{\left(x[0] - 1 \right)} + 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) + \left(- \sin{\left(x[0] - 1 \right)} + 9 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) + \left(2 \sin{\left(x[0] - 1 \right)} - 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) + \left(4 \sin{\left(x[0] - 1 \right)} - 36 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) + \left(- 27 \sin{\left(3 x[0] - x[1]^{2} \right)} - \cos{\left(x[0] - 1 \right)}\right) \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right)\right) + m[0, 1] \left(- 36 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(- \sin{\left(x[0] - 1 \right)} + 9 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1] \left(2 \sin{\left(x[0] - 1 \right)} - 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 24 x[1] \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 12 x[1] \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 18 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)}\right) + m[1, 0] \left(- 36 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(- \sin{\left(x[0] - 1 \right)} + 9 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1] \left(2 \sin{\left(x[0] - 1 \right)} - 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 24 x[1] \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 12 x[1] \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 18 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)}\right) + m[1, 1] \left(24 x[1]^{2} \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 8 x[1]^{2} \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1]^{2} \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 12 x[1]^{2} \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 120 x[1]^{2} \sin{\left(3 x[0] - x[1]^{2} \right)} \cos{\left(3 x[0] - x[1]^{2} \right)} + 3 \left(- 8 \sin{\left(x[0] - 1 \right)} + 8 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + \left(- 24 \sin{\left(3 x[0] - x[1]^{2} \right)} - 8 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 3 \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)} - 2\right) \cos{\left(3 x[0] - x[1]^{2} \right)}\right) & m[0, 0] \left(- 36 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(- \sin{\left(x[0] - 1 \right)} + 9 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1] \left(2 \sin{\left(x[0] - 1 \right)} - 18 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 24 x[1] \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 12 x[1] \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 18 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)}\right) + m[0, 1] \left(24 x[1]^{2} \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 8 x[1]^{2} \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1]^{2} \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 12 x[1]^{2} \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 120 x[1]^{2} \sin{\left(3 x[0] - x[1]^{2} \right)} \cos{\left(3 x[0] - x[1]^{2} \right)} + \left(- 24 \sin{\left(x[0] - 1 \right)} + 24 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + \left(- 24 \sin{\left(3 x[0] - x[1]^{2} \right)} - 8 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + \left(- 6 \sin{\left(x[0] - 1 \right)} + 6 \cos{\left(3 x[0] - x[1]^{2} \right)} - 6\right) \cos{\left(3 x[0] - x[1]^{2} \right)}\right) + m[1, 0] \left(24 x[1]^{2} \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 8 x[1]^{2} \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 4 x[1]^{2} \left(3 \sin{\left(3 x[0] - x[1]^{2} \right)} + \cos{\left(x[0] - 1 \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 12 x[1]^{2} \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 120 x[1]^{2} \sin{\left(3 x[0] - x[1]^{2} \right)} \cos{\left(3 x[0] - x[1]^{2} \right)} + \left(- 24 \sin{\left(x[0] - 1 \right)} + 24 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + \left(- 24 \sin{\left(3 x[0] - x[1]^{2} \right)} - 8 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + \left(- 6 \sin{\left(3 x[0] - x[1]^{2} \right)} - 2 \cos{\left(x[0] - 1 \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + \left(- 6 \sin{\left(x[0] - 1 \right)} + 6 \cos{\left(3 x[0] - x[1]^{2} \right)} - 6\right) \cos{\left(3 x[0] - x[1]^{2} \right)}\right) + m[1, 1] \left(- 16 x[1]^{3} \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \sin{\left(3 x[0] - x[1]^{2} \right)} + 8 x[1]^{3} \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \sin{\left(3 x[0] - x[1]^{2} \right)} - 120 x[1]^{3} \sin{\left(3 x[0] - x[1]^{2} \right)} \cos{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(- 8 \sin{\left(x[0] - 1 \right)} + 8 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 16 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)}\right) \cos{\left(3 x[0] - x[1]^{2} \right)} - 2 x[1] \left(- 2 \sin{\left(x[0] - 1 \right)} + 2 \cos{\left(3 x[0] - x[1]^{2} \right)} - 2\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 8 x[1] \left(\sin{\left(x[0] - 1 \right)} - \cos{\left(3 x[0] - x[1]^{2} \right)} + 1\right) \cos{\left(3 x[0] - x[1]^{2} \right)} + 60 x[1] \sin^{2}{\left(3 x[0] - x[1]^{2} \right)}\right)\end{matrix}\right]$

## NumPy function generation

```python
x_np = np.array([0.2, 1.1])
v_np = np.array([1., -1.])
m_np = np.array([[1., 2.], [3., 4.]])
```

```python
forward_func_np = symnum.numpify(2)(forward_func)
forward_func_np(x_np)
```

    array([ 1.53700411, -3.07400822])

```python
neg_log_dens_np = symnum.numpify(2)(neg_log_dens)
neg_log_dens_np(x_np)
```

    5.493949967002545

```python
symnum.grad(neg_log_dens_np)(x_np)
```

    array([ 7.03139355, -7.32518763])

```python
symnum.hessian(neg_log_dens_np)(x_np)
```

    array([[-47.88846093,  29.72428278],
           [ 29.72428278, -25.23749599]])

```python
symnum.hessian_vector_product(neg_log_dens_np)(x_np)(v_np)
```

    array([-77.61274371,  54.96177876])

```python
symnum.matrix_tressian_product(neg_log_dens_np)(x_np)(m_np)
```

    array([ 219.62500437, -142.62315559])

```python
symnum.jacobian(forward_func_np)(x_np)
```

    array([[ 1.02189567, -1.26030841],
           [-2.04379134,  2.52061682]])

```python
symnum.jacobian_vector_product(forward_func_np)(x_np)(v_np)
```

    array([ 2.28220408, -4.56440817])

```python
symnum.vector_jacobian_product(forward_func_np)(x_np)(v_np)
```

    array([ 3.06568701, -3.78092524])

```python
symnum.matrix_hessian_product(forward_func_np)(x_np)(m_np)
```

    array([8.01287975, 3.62860337])

## Compatibility with `multiprocessing`

```python
grad_neg_log_dens = symnum.grad(neg_log_dens_np)
```

```python
pool = multiprocessing.Pool(4)
```

```python
pool.map(grad_neg_log_dens, [x_np + 0.1 * i for i in range(8)])
```

    [array([ 7.03139355, -7.32518763]),
     array([ 5.36166701, -6.83240645]),
     array([ 4.05701734, -6.32916535]),
     array([ 3.1698736 , -5.90591748]),
     array([ 2.69164583, -5.61080858]),
     array([ 2.56667444, -5.4433092 ]),
     array([ 2.70213256, -5.34858144]),
     array([ 2.97340715, -5.21063384])]


### The goal for this tutorial

The goal will be to plot a variety of 1 dimensional electron density models of the solar atmosphere, both theoretical and empirical. The empirical models that we will use (Newkirk and Saito) are often used for estimates of radio burst altitude and speed in the corona. We'll be plotting electrons density as a function of height $n(r)$.

N.B. When I use the term height, altitude or distance I mean heliocnetric distance i.e., 1 R$_{\odot}$ is the solar surface.

### First let's start with the hydrostatic plane parallel model

Start with plotting the following equation from 1-10 solar radii
\begin{equation}
n(r) = n_0\mathrm{exp}\bigg(\frac{-r}{H}\bigg)
\end{equation}

where $n_0$ is the density at the base of the corona and $r$ is the distance into the solar atmosphere. $H$ is known as the scale height. And is given by

\begin{equation}
H = \frac{m_p g_{\odot}}{k_B T}
\end{equation}

where $k_B$ is Boltzmann's constant, $T$ is the temperature, $m_p$ is the proton mass, and $g_{\odot}$ is solar gravity. What is the significance of the scale height?

```python
import numpy as np
import matplotlib.pyplot as plt
```

### Let's first define some constants

```python
n0 = 1e9       # Electron density at 1 Rsun (cm^-3)
T = 3e6        # MK
kb = 1.38e-23  # Boltzman constant J/K
mp = 1.67e-27  # Proton mass kg
Msun = 2e30    # Solar mass kg
rsun = 6.96e8  # Solar radius m
G = 6.67e-11   # Universal gravitational constant (m3 kg-1 s-2)
gsun = 274     # Solar gravity (m/s/s)
```

### First build a 1D distance grid

```python
radius = np.arange(1,10,0.1) # Solar radii
print(radius)
```

    [1.  1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.  2.1 2.2 2.3 2.4 2.5 2.6 2.7
     2.8 2.9 3.  3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.  4.1 4.2 4.3 4.4 4.5
     4.6 4.7 4.8 4.9 5.  5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 6.  6.1 6.2 6.3
     6.4 6.5 6.6 6.7 6.8 6.9 7.  7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 8.  8.1
     8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 9.  9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9]

### Let's look at a plane-parallel hydrostatic model

```python
H = kb*T/(mp*gsun)
print('Hydrostatic scale height: %s Mm' %(H/1e6))

```

    Hydrostatic scale height: 90.47598234188558 Mm

```python
Hrsun = H/rsun
npp = n0*np.exp((-1.0/Hrsun)*(radius-1))
print(Hrsun)
```

    0.12999422750270917

```python
fig, ax = plt.subplots()
ax.plot(radius, npp)
ax.set_yscale('log')
ax.set_ylim(1e4,1e10)
ax.set_xlabel('Heliocentric Distance (R$_{\odot}$)')
ax.set_ylabel('Electron density (cm$^{-3})$')
plt.show()
```

### Increasing the temperature of the atmosphere

```python
fig, ax = plt.subplots()
def plothydrostat(radius, temp, label=None):
    H = kb*temp/(mp*gsun)
    Hrsun = H/rsun
    n = n0*np.exp((-1.0/Hrsun)*(radius-1))
    ax.plot(radius, n, label=label)

    return ax
plothydrostat(radius, 1e6, label='1 MK')
plothydrostat(radius, 5e6, label='5 MK')
ax = plothydrostat(radius, 10e6, label='10 MK')
ax.set_yscale('log')
ax.set_ylim(1e4,1e10)
ax.set_xlabel('Heliocentric Distance (R$_{\odot}$)')
ax.set_ylabel('Electron density (cm$^{-3})$')
ax.set_xlim(1,7)
plt.legend()
plt.show()
```

### Now let's plot the spherically symmetric solution

```python
# The difference here is the scale-height isn't just a single value
T = 1.4e6
H = (2*kb*T*rsun)/(G*Msun*mp*1.2)
nss = n0 * np.exp ( (1/H)*(1/(radius) - 1) )
plt.plot(radius, nss, label='Plane parallel')
plt.plot(radius, npp, label='Spherically symmetric')
plt.yscale('log')
plt.ylim(1e4,1e10)
plt.xlabel('Heliocentric Distance (R$_{\odot}$)')
plt.ylabel('Electron density (cm$^{-3})$')
plt.legend()
plt.show()
```

### What does this look like in terms of plasma frequency?

```python
fplasma_pp = 8980.0*np.sqrt(npp)/1e6
fplasma_ss = 8980.0*np.sqrt(nss)/1e6
plt.plot(radius, fplasma_ss)
plt.xlabel('Heliocentric Distance (R$_{\odot}$)')
plt.ylabel('Plasma Frequency (MHz)')
plt.yscale('log')
plt.ylim(1,300)
plt.show()

```

### Now let's plot some empirical density models

```python
# The Newkirk model
N=4.2e4
newkirk=N*10.0**(4.32/radius)

# The Saito QS model
c1, c2, d1, d2 = 1.36e6, 1.68e8, 2.14, 6.13
saito=c1*radius**(-1.0*d1) + c2*radius**(-1.0*d2)

plt.plot(radius, newkirk, label='Newkirk')
plt.plot(radius, saito, label='Saito')
plt.plot(radius, nss, label='Hydrostat SS')
plt.yscale('log')
plt.xscale('log')
plt.ylabel('Electron density (cm$^{-3}$)')
plt.xlabel('Heliocentric Distance (R$_{\odot}$)')
plt.legend()

```

    <matplotlib.legend.Legend at 0x10a594128>

### Now let's get the speed of the shock that caused the type II

```python
# Firstly we convert frequency to density
def freq2dens(freq):
    # freq in Hz
    emass = 9.11e-28       # grams (cgs)
    eq = 4.80e-10          # statcouloumb (cgs)
    ne = (np.pi*emass*freq**2)/eq**2
    return ne # cm^-3
```

```python
from astropy.time import Time
from datetime import datetime
from scipy import stats

# Read in the frequency time points from Pearse's tutorial.
result = np.load('/Users/eoincarley/Desktop/WP4-T1/STELLAR_SSW_tutorials/day2_radio_session/ft_coords.npz', allow_pickle=True)
time = result['arr_0']
time = Time(time).unix
tsec = time - time[0]
freq = result['arr_1']

# Read in the frequency time points from Pearse's tutorial.
#freq = np.array([150e6, 100e6, 40e6, 30e6, 20e6])
#tsec = np.array([0, 120, 200, 360, 520])

# Convert the frequencies to densities.
# density = freq2dens(freq)

density = freq2dens(freq*1e6)
print(density)
```

    [25901780.03874429 19883225.99254251 16583424.06190234 15212887.15286152
     13688626.54649108 12751224.72265591 10976156.71435402  9956857.27092373
      9073317.05990052  7351479.1213379 ]

```python
# Now use one of the density models to map density to radius
rpoints = np.interp(density, newkirk[::-1], radius[::-1])
```

```python
print(rpoints)
```

    [1.55511805 1.61809467 1.66927492 1.69053208 1.72254949 1.74568078
     1.78948228 1.82227452 1.85545876 1.92950182]

```python
from scipy import stats

# Plot the results and fit a line to the data points
speed, rstart, r_value, p_value, std_err = stats.linregress(tsec, rpoints)
speedkm = speed*rsun
speedstring = 'Speed: '+str(round(speedkm/1e3))+' km/s'

fig, ax = plt.subplots()
plt.plot(tsec, rpoints, 'o')
plt.ylabel('Heliocentric Distance (Rsun)')
plt.xlabel('Time (sec)')
plt.plot(tsec, rstart + speed*tsec, label=speedstring)
plt.legend()
plt.show()
```

```python
print(speedkm/1e3)
```

    952.6538827684496

### Now estimate the speed using the Saito model.

```python

```


[From the Stan Manual](https://mc-stan.org/docs/2_25/reference-manual/change-of-variables-section.html#multivariate-changes-of-variables):

> Suppose $X$ is a $K$-dimensional random variable with probability density function $p_X(x)$. A new random variable $Y = f ( X )$ may be defined by transforming $X$ with a suitably well-behaved function $f $. It suffices for what follows to note that if $f$ is one-to-one and its inverse $f ^ {− 1}$ has a well-defined Jacobian, then the density of $Y$ is
$$
p_Y ( y ) = p_X ( f ^ {− 1} ( y ) ) ∣ det J _ {f ^{ − 1}} ( y ) |
$$

Here we have $p_X(a, b) \sim \text{Uniform}$, with $a = \frac{\alpha}{\alpha + \beta}, b = (\alpha + \beta)^{-1/2}$

```python
from sympy import symbols, Matrix, simplify, solve, Eq, Abs, log
```

```python
alpha, beta = symbols('alpha, beta', positive=True)
a, b = symbols('a b')
f = solve((Eq(alpha / (alpha + beta), a),
           Eq((alpha + beta)**(-1/2), b)),
          alpha, beta)
f
```

    {alpha: a/b**2, beta: (1.0 - a)/b**2}

```python
f_inv = solve((Eq(alpha / (alpha + beta), a),
               Eq((alpha + beta)**(-1/2), b)),
              a, b)
f_inv
```

    {b: 1/sqrt(alpha + beta), a: alpha/(alpha + beta)}

```python
X = Matrix([f_inv[a], f_inv[b]])
Y = Matrix([alpha, beta])
det = X.jacobian(Y).det()
det = det.subs([
    (alpha, f[alpha]),
    (beta, f[beta])]).subs(b, (alpha + beta)**(-1/2))
p_a_b = simplify(Abs(det))
p_a_b
```

$\displaystyle \frac{0.5}{\left(\alpha + \beta\right)^{2.5}}$

```python
x, y = symbols('x y')
rewrited = solve((Eq(x, log(alpha / beta)),
                  Eq(y, log(alpha + beta))),
                 alpha, beta)
```

```python
rewrited
```

    {beta: exp(y)/(exp(x) + 1), alpha: exp(x + y)/(exp(x) + 1)}

```python
X = Matrix([rewrited[alpha], rewrited[beta]])
Y = Matrix([x, y])
det = X.jacobian(Y).det().subs([
  (x, log(alpha / beta)),
  (y, log(alpha + beta))])
simplify(Abs(det)) * p_a_b
```

$\displaystyle \frac{0.5 \alpha \beta}{\left(\alpha + \beta\right)^{2.5}}$

```python

```


# Modeling and Simulation in Python

Chapter 11: Rotation

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
from sympy import *

init_printing()
```

```python
T, a, alpha, I, m, g, r = symbols('T a alpha I m g r')
```

```python
eq1 = Eq(a, -r * alpha)
eq1
```

```python
eq2 = Eq(T - m * g, m * a)
eq2
```

```python
eq3 = Eq(T * r, I * alpha)
eq3
```

```python
soln = solve([eq1, eq2, eq3], [T, a, alpha])
```

```python
soln[T]
```

```python
soln[a]
```

```python
soln[alpha]
```

```python

```

```python

```


```python
%matplotlib inline
from IPython.display import display,Math
from sympy import *
init_session()
```

```python
%%time
# 平方根の第k位まで
p = 2
k = 2
q = 0
while ( q**2 <= p*(10**(2*k)) ):
    q = q+1

# 第k桁まで r*(10**k)
if q**2 == p*(10**(2*k)):
    r = q
else:
    r = q-1
print("{0:d} の平方根の小数第{1:d}位までは {2:.{digits}f}".format(p,k,r*(10**(-k)),digits=k))

```

```python
%%time
# 平方根の第k位まで(変数追加による改良)
p = 2
k = 2
q = 0
pk = p * (10**(2*k)) # 繰り返し使うので変数として設定
while ( q**2 <= pk ):
    q = q+1

# 第k桁まで r*(10**k)
if q**2 == pk:
    r = q
else:
    r = q-1
print("{0:d} の平方根の小数第{1:d}位までは {2:.{digits}f}".format(p,k,r*(10**(-k)),digits=k))

```

```python
%%time
# 平方根の第k位まで(和の公式を利用)
p = 2
k = 2
s = 0
q = 0
pk = p*(10**(2*k))
while ( s <= pk ):
    s = s+(2*q+1)
    q = q+1

# 第k桁まで r*(10**k)
if s == p:
    r = q
else:
    r = q-1
print("{0:d} の平方根の小数第{1:d}位までは {2:.{digits}f}".format(p,k,r*(10**(-k)),digits=k))

```

```python
%%time
# 平方根を用いる(ニュートン法)
p = 3
k = 2 # 誤差の精度
e = 10**(-k)  # 誤差
q = p
while ( (q**2-p)/q >= e):
    q = (p+q**2)/(2*q)
print("{0:d} の平方根の誤差{1:.{digits}f}の値は {2:.{digits}f}".format(p,e,q,digits=k))

```

```python
from ipywidgets import interact
from ipywidgets import BoundedIntText
import time
def mynewton(p,k,mstep=10**3):
    e = 10**(-k)
    q = p
    step = 0
    while ( (q**2-p)/q >= e) and (step < mstep):
        q = (p+q**2)/(2*q)
        step += 1
    return q,step
@interact
def _(p=BoundedIntText(value=2,min=1,max=1000,step=1,description="p"),
      k=BoundedIntText(value=1,min=1,max=100,step=1,description="精度")):
    p,k = int(p),int(k)
    q,step = mynewton(p,k)
    return display(Math("$\sqrt{{ {0:d} }}\\fallingdotseq {1:.{digits}f}\\\\ \
        \\text{{{2:d}step(s)}}".format(p,q,step,digits=k)))
```

```python

```


```python
from sympy import pi, sqrt, erf, integrate, var, S, exp, init_printing, oo, Symbol
```

```python
var("r")
alpha = Symbol("alpha", positive=True)
```

```python
g = alpha**3/pi**(S(3)/2)*exp(-alpha**2*r**2)
```

```python
V = erf(alpha*r)/r
```

```python
Ig = 4*pi*integrate(g*(1/r-V)*r**2, (r, 0, oo))
```

```python
Isph = 4*pi*integrate((1/r-V)*r**2, (r, 0, 1))
```

```python
alpha_code = 37
```

```python
Ig.subs(alpha, sqrt(alpha_code)).n(25)
```

    2.010320207622576710962755

```python
Isph.subs(alpha, sqrt(alpha_code)).n(25)
```

    0.08490790955648089702710017

```python
e = V.subs(alpha, sqrt(alpha_code))
for i in range(1, 18):
    print i, e.subs(r, S(1)/10**i).n(25)
```

    1 6.103391218853852058466783
    2 6.855206721888216457088578
    3 6.863577866679608029294916
    4 6.863661671059414495539896
    5 6.863662509112513937641552
    6 6.863662517493045862208506
    7 6.863662517576851181547190
    8 6.863662517577689234740586
    9 6.863662517577697615272520
    10 6.863662517577697699077840
    11 6.863662517577697699915893
    12 6.863662517577697699924273
    13 6.863662517577697699924357
    14 6.863662517577697699924358
    15 6.863662517577697699924358
    16 6.863662517577697699924358
    17 6.863662517577697699924358

```python
print "Ig = %.22f_dp / rc" % Ig.subs(alpha, sqrt(alpha_code)).n(25)
print "Isph = %.22f_dp * rc**2" % Isph.subs(alpha, sqrt(alpha_code)).n(25)
print "v0 = %.22f_dp / rc" % e.subs(r, S(1)/10**18).n(25)
```

    Ig = 2.0103202076225765715378_dp / rc
    Isph = 0.0849079095564808927765_dp * rc**2
    v0 = 6.8636625175776977414444_dp / rc


# 14 Linear Algebra: Singular Value Decomposition

One can always decompose a matrix $\mathsf{A}$

\begin{gather}
\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}\\
\mathsf{U}^T \mathsf{U} = \mathsf{U} \mathsf{U}^T = 1\\
\mathsf{V}^T \mathsf{V} = \mathsf{V} \mathsf{V}^T = 1
\end{gather}

where $\mathsf{U}$ and $\mathsf{V}$ are orthogonal matrices and the $w_j$ are the _singular values_ that are assembled into a diagonal matrix $\mathsf{W}$.

$$
\mathsf{W} = \text{diag}(w_j)
$$

The inverse (if it exists) can be directly calculated from the SVD:

$$
\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T
$$

## Solving ill-conditioned coupled linear equations

```python
import numpy as np
```

### Non-singular matrix
Solve the linear system of equations

$$
\mathsf{A}\mathbf{x} = \mathbf{b}
$$

Using the standard linear solver in numpy:

```python
A = np.array([
        [1, 2, 3],
        [3, 2, 1],
        [-1, -2, -6],
    ])
b = np.array([0, 1, -1])
```

```python
np.linalg.solve(A, b)
```

    array([ 0.83333333, -0.91666667,  0.33333333])

Using the inverse from SVD:

$$
\mathbf{x} = \mathsf{A}^{-1} \mathbf{b}
$$

```python
U, w, VT = np.linalg.svd(A)
print(w)
```

    [7.74140616 2.96605874 0.52261473]

First check that the SVD really factors $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python
U.dot(np.diag(w).dot(VT))
```

    array([[ 1.,  2.,  3.],
           [ 3.,  2.,  1.],
           [-1., -2., -6.]])

```python
np.allclose(A, U.dot(np.diag(w).dot(VT)))
```

    True

Now calculate the matrix inverse $\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T$:

```python
inv_w = 1/w
print(inv_w)
```

    [0.1291755  0.33714774 1.91345545]

```python
A_inv = VT.T.dot(np.diag(inv_w)).dot(U.T)
print(A_inv)
```

    [[-8.33333333e-01  5.00000000e-01 -3.33333333e-01]
     [ 1.41666667e+00 -2.50000000e-01  6.66666667e-01]
     [-3.33333333e-01 -1.08335035e-16 -3.33333333e-01]]

Check that this is the same that we get from `numpy.linalg.inv()`:

```python
np.allclose(A_inv, np.linalg.inv(A))
```

    True

Now, *finally* solve (and check against `numpy.linalg.solve()`):

```python
x = A_inv.dot(b)
print(x)
np.allclose(x, np.linalg.solve(A, b))
```

    [ 0.83333333 -0.91666667  0.33333333]

    True

```python
A.dot(x)
```

    array([-7.77156117e-16,  1.00000000e+00, -1.00000000e+00])

```python
np.allclose(A.dot(x), b)
```

    True

### Singular matrix

If the matrix $\mathsf{A}$ is *singular* (i.e., its rank (linearly independent rows or columns) is less than its dimension and hence the linear system of equation does not have a unique solution):

For example, the following matrix has the same row twice:

```python
C = np.array([
     [ 0.87119148,  0.9330127,  -0.9330127],
     [ 1.1160254,   0.04736717, -0.04736717],
     [ 1.1160254,   0.04736717, -0.04736717],
    ])
b1 = np.array([ 2.3674474,  -0.24813392, -0.24813392])
b2 = np.array([0, 1, 1])
```

```python
np.linalg.solve(C, b1)
```

NOTE: failure is not always that obvious: numerically, a matrix can be *almost* singular.

Try solving the linear system of equations

$$
\mathsf{D}\mathbf{x} = \mathbf{b}_1
$$
with matrix $\mathsf{D}$ below:

```python
D = C.copy()
D[2, :] = C[0] - 3*C[1]
D
```

    array([[ 0.87119148,  0.9330127 , -0.9330127 ],
           [ 1.1160254 ,  0.04736717, -0.04736717],
           [-2.47688472,  0.79091119, -0.79091119]])

```python
np.linalg.solve(D, b1)
```

    array([1.61493184e+00, 2.01760247e+16, 2.01760247e+16])

Note that some of the values are huge, and suspiciously like the inverse of machine precision? Sign of a nearly singular matrix.

**Note**: *Just because a function did not throw an exception it does not mean that the answer is correct.* **Always check your output!**

Now back to the example with $\mathsf{C}$:

#### SVD for singular matrices
If a matrix is *singular* or *near singular* then one can *still* apply SVD.

One can then compute the *pseudo inverse*

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

i.e., any singular $w_j = 0$ is being "augmented" by setting

$$
\frac{1}{w_j} \rightarrow 0 \quad\text{if}\quad w_j = 0
$$

in $\text{diag}(1/w_j)$.

Perform the SVD for the singular matrix $\mathsf{C}$:

```python
U, w, VT = np.linalg.svd(C)
print(w)
```

    [1.99999999e+00 1.00000000e+00 2.46519033e-32]

Note the third value $w_2 \approx 0$: sign of a singular matrix.

Test that the SVD really decomposes $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python
U.dot(np.diag(w).dot(VT))
```

    array([[ 0.87119148,  0.9330127 , -0.9330127 ],
           [ 1.1160254 ,  0.04736717, -0.04736717],
           [ 1.1160254 ,  0.04736717, -0.04736717]])

```python
np.allclose(C, U.dot(np.diag(w).dot(VT)))
```

    True

There are the **singular values** (let's say, $|w_i| < 10^{-12}$):

```python
singular_values = np.abs(w) < 1e-12
print(singular_values)
```

    [False False  True]

#### Pseudo-inverse

Calculate the **pseudo-inverse** from the SVD

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

Augment:

```python
inv_w = 1/w
inv_w[singular_values] = 0
print(inv_w)
```

    [0.5 1.  0. ]

```python
C_inv = VT.T.dot(np.diag(inv_w)).dot(U.T)
print(C_inv)
```

    [[-0.04736717  0.46650635  0.46650635]
     [ 0.5580127  -0.21779787 -0.21779787]
     [-0.5580127   0.21779787  0.21779787]]

#### Solution for $\mathbf{b}_1$

Now solve the linear problem with SVD:

```python
x1 = C_inv.dot(b1)
print(x1)
```

    [-0.34365138  1.4291518  -1.4291518 ]

```python
C.dot(x1)
```

    array([ 2.3674474 , -0.24813392, -0.24813392])

```python
np.allclose(C.dot(x1), b1)
```

    True

Thus, using the pseudo-inverse $\mathsf{C}^{-1}$ we can obtain solutions to the equation

$$
\mathsf{C} \mathbf{x}_1 = \mathbf{b}_1
$$

However, $\mathbf{x}_1$ is not the only solution: there's a whole line of solutions that are formed by the special solution and a combination of the basis vectors in the *null space* of the matrix:

The (right) *kernel* or *null space* contains all vectors $\mathbf{x^0}$ for which

$$
\mathsf{C} \mathbf{x^0} = 0
$$

(The dimension of the null space corresponds to the number of singular values.) You can find a basis that spans the null space. Any linear combination of null space basis vectors will also end up in the null space when $\mathbf{A}$ is applied to it.

Specifically, if $\mathbf{x}_1$ is a special solution and $\lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots$ is a vector in the null space then

$$
\mathbf{x} = \mathbf{x}_1 + ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots )
$$

is **also a solution** because

$$
\mathsf{C} \mathbf{x} = \mathsf{C} \mathbf{x_1} + \mathsf{C} ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots ) = \mathsf{C} \mathbf{x_1} + 0 = \mathbf{b}_1 + 0 = \mathbf{b}_1
$$

The $\lambda_i$ are arbitrary real numbers and hence there is an infinite number of solutions.

In SVD:

* The columns $U_{\cdot, i}$ of $\mathsf{U}$ (i.e. `U.T[i]` or `U[:, i]`) corresponding to non-zero $w_i$, i.e. $\{i : w_i \neq 0\}$, form the basis for the _range_ of the matrix $\mathsf{A}$.
* The columns $V_{\cdot, i}$ of $\mathsf{V}$ (i.e. `V.T[i]` or `V[:, i]`) corresponding to zero $w_i$, i.e. $\{i : w_i = 0\}$, form the basis for the _null space_ of the matrix $\mathsf{A}$.

```python
x1
```

    array([-0.34365138,  1.4291518 , -1.4291518 ])

The rank space comes from $\mathsf{U}^T$:

```python
U.T
```

    array([[-7.07106782e-01, -4.99999999e-01, -4.99999999e-01],
           [ 7.07106780e-01, -5.00000001e-01, -5.00000001e-01],
           [-2.47010760e-16, -7.07106781e-01,  7.07106781e-01]])

The basis vectors for the rank space (``~ bool_array`` applies a logical ``NOT`` operation to the entries in the boolean array so that we can pick out "not singular values"):

```python
U.T[~singular_values]
```

    array([[-0.70710678, -0.5       , -0.5       ],
           [ 0.70710678, -0.5       , -0.5       ]])

The null space comes from $\mathsf{V}^T$:

```python
VT
```

    array([[-0.8660254 , -0.35355339,  0.35355339],
           [-0.5       ,  0.61237244, -0.61237244],
           [-0.        , -0.70710678, -0.70710678]])

The basis vector for the null space:

```python
VT[singular_values]
```

    array([[-0.        , -0.70710678, -0.70710678]])

The component of $\mathbf{x}_1$ along the basis vector of the null space of $\mathsf{C}$ (here a 1D space) – note that this component is zero, i.e., the special solution lives in the rank space:

```python
x1.dot(VT[singular_values][0])
```

    2.220446049250313e-16

We can create a family of solutions by adding vectors in the null space to the special solution $\mathbf{x}_1$, e.g. $\lambda_1 = 2$:

```python
lambda_1 = 2
x1_1 = x1 + lambda_1 * VT[2]
print(x1_1)

np.allclose(C.dot(x1_1), b1)
```

    [-0.34365138  0.01493824 -2.84336536]

    True

Thus, **all** solutions are
```
x1 + lambda * VT[2]
```

#### Solution for  $\mathbf{b}_2$

The solution vector $x_2$ solves
$$
\mathsf{C}\mathbf{x}_2 = \mathbf{b}_2
$$

```python
b2
```

    array([0, 1, 1])

```python
x2 = C_inv.dot(b2)
print(x2)
print(C.dot(x2))
np.allclose(C.dot(x2), b2)
```

    [ 0.9330127  -0.43559574  0.43559574]
    [-4.4408921e-16  1.0000000e+00  1.0000000e+00]

    True

... and the general solution will again be obtained by adding any multiple of the null space basis vector.

#### Null space

The Null space is spanned by the following basis vectors (just one in this example):

```python
null_basis = VT[singular_values]
null_basis
```

    array([[-0.        , -0.70710678, -0.70710678]])

Show that
$$
\mathsf{C}\mathbf{x}^0 = 0
$$

```python
C.dot(null_basis.T)
```

    array([[ 0.0000000e+00],
           [-6.9388939e-18],
           [-6.9388939e-18]])

## SVD for fewer equations than unknowns
$N$ equations for $M$ unknowns with $N < M$:

* no unique solutions (underdetermined)
* $M-N$ dimensional family of solutions
* SVD: at least $M-N$ zero or negligible $w_j$: columns of $\mathsf{V}$ corresponding to singular $w_j$ span the solution space when added to a particular solution.

Same as the above [**Solving ill-conditioned coupled linear equations**](#Solving-ill-conditioned-coupled-linear-equations).

## SVD for more equations than unknowns
$N$ equations for $M$ unknowns with $N > M$:

* no exact solutions in general (overdetermined)
* but: SVD can provide best solution in the least-square sense
  $$
  \mathbf{x} = \mathsf{V}\, \text{diag}(1/w_j)\, \mathsf{U}^{T}\, \mathbf{b}
  $$
  where

  * $\mathbf{x}$ is a $M$-dimensional vector of the unknowns (parameters of the fit),
  * $\mathsf{V}$ is a $M \times N$ matrix
  * the $w_j$ form a square $M \times M$ matrix,
  * $\mathsf{U}$ is a $M \times N$ matrix (and $\mathsf{U}^T$ is a $N \times M$ matrix), and
  * $\mathbf{b}$ is the $N$-dimensional vector of the given values (data)

It can be shown that $\mathbf{x}$ minimizes the residual

$$
\mathbf{r} := |\mathsf{A}\mathbf{x} - \mathbf{b}|.
$$

where the matrix $\mathsf{A}$ will be described below and will contain the evaluation of the fit function for each data point in $\mathbf{b}$.

(For a $N \le M$, one can find $\mathbf{x}$ so that $\mathbf{r} = 0$ – see above.)

(In the following, we will switch notation and denote the vector of $M$ unknown parameters of the model as $\mathbf{a}$; this $\mathbf{a}$ corresponds to $\mathbf{x}$ above. $N$ is the number of observations.)

### Linear least-squares fitting

This is the *liner least-squares fitting problem*: Given $N$ data points $(x_i, y_i)$ (where $1 \le i \le N$), fit to a linear model $y(x)$, which can be any linear combination of $M$ functions of $x$.

For example, if we have $N$ functions $x^k$ with parameters $a_k$
$$
y(x) = a_1 + a_2 x + a_3 x^2 + \dots + a_M x^{M-1}
$$
or in general
$$
y(x) = \sum_{k=1}^M a_k X_k(x)
$$

The goal is to determine the $M$ coefficients $a_k$.

Define the **merit function**
$$
\chi^2 = \sum_{i=1}^N \left[ \frac{y_i - \sum_{k=1}^M a_k X_k(x_i)}{\sigma_i}\right]^2
$$
(sum of squared deviations, weighted with standard deviations $\sigma_i$ on the $y_i$).

Best parameters $a_k$ are the ones that *minimize $\chi^2$*.

*Design matrix* $\mathsf{A}$ ($N \times M$, $N \geq M$), vector of measurements $\mathbf{b}$ ($N$-dim) and parameter vector $\mathbf{a}$ ($M$-dim):

\begin{align}
A_{ij} &= \frac{X_j(x_i)}{\sigma_i}\\
b_i &= \frac{y_i}{\sigma_i}\\
\mathbf{a} &= (a_1, a_2, \dots, a_M)
\end{align}

The design matrix $\mathsf{A}$ contains the *predicted* values from the basis functions for all values $x_i$ of the independent variable $x$ for which we have measured data $y_i$.

Minimum occurs when the derivative vanishes:
$$
0 = \frac{\partial\chi^2}{\partial a_k} = \sum_{i=1}^N {\sigma_i}^{-2} \left[ y_i - \sum_{j=1}^M a_j X_j(x_i) \right] X_k(x_i), \quad 1 \leq k \leq M
$$
($M$ coupled equations)

To simplify the notation, define the $M \times M$ matrix
\begin{align}
\alpha_{kj} &= \sum_{i=1}^N \frac{X_k(x_i) X_j(x_i)}{\sigma_i^2}\\
\mathsf{\alpha} &= \mathsf{A}^T \mathsf{A}
\end{align}
and the vector of length $M$
\begin{align}
\beta_{k} &= \sum_{i=1}^N \frac{y_i X_k(x_i)}{\sigma_i^2}\\
\boldsymbol{\beta} &= \mathsf{A}^T \mathbf{b}
\end{align}

Then the $M$ coupled equations can be compactly written as
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \boldsymbol{\beta}
\end{align}

$\mathsf{\alpha}$ and  $\boldsymbol{\beta}$ are known, so we have to solve this matrix equation for the vector of the unknown parameters $\mathbf{a}$.

#### Error estimates for the parameters

The inverse of $\mathsf{\alpha}$ is related to the uncertainties in the parameters:
$$
\mathsf{C} := \mathsf{\alpha}^{-1}
$$
in particular
$$
\sigma(a_i) = C_{ii}
$$
(and the $C_{ij}$ are the co-variances).

#### Solution of the linear least-squares fitting problem with SVD
We need to solve the overdetermined system of $M$ coupled equations
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \boldsymbol{\beta}
\end{align}
We can solve the above equation with SVD.

SVD finds $\mathbf{a}$ that minimizes
$$
\chi^2 = |\mathsf{A}\mathbf{a} - \mathbf{b}|
$$
(proof in _Numerical Recipes_ Ch 2.) and so SVD is suitable to solve the equation above.

We can alternatively use SVD to directly solve the overdetermined system
$$
\mathsf{A}\mathbf{a} = \mathbf{b}
$$
and we should get the same answer.

The first approach might be preferrable because the matrix $\mathsf{\alpha}$ is a relatively small $M \times M$ matrix, i.e., its size depends on the number of parameters. The design matrix $\mathsf{A}$ is a $N \times M$ matrix and can be large (in one dimensions) because its size depends on the possibly large number $N$, the number of data points.

The errors are
$$
\sigma^2(a_j) = \sum_{i=1}^{M} \left(\frac{V_{ji}}{w_i}\right)^2
$$
(see also _Numerical Recipes_ Ch. 15) where $V_{ji}$ are elements of $\mathsf{V}$.

#### Example
Synthetic data

$$
y(x) = 3\sin x - 2\sin 3x + \sin 4x
$$

with noise $r$ added (uniform in range $-5 < r < 5$).

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import numpy as np
```

```python
def signal(x, noise=0):
    r = np.random.uniform(-noise, noise, len(x))
    return 3*np.sin(x) - 2*np.sin(3*x) + np.sin(4*x) + r
```

```python
X = np.linspace(-10, 10, 500)
Y = signal(X, noise=5)
```

```python
plt.plot(X, Y, 'r-', X, signal(X, noise=0), 'k--')
```

Define our fit function (the model) and the basis functions. We need the basis functions for setting up the problem and we will later use the fitfunction together with our parameter estimates to compare our fit to the true underlying function.

```python
def fitfunc(x, a):
    return a[0]*np.cos(x) + a[1]*np.sin(x) + \
           a[2]*np.cos(2*x) + a[3]*np.sin(2*x) + \
           a[4]*np.cos(3*x) + a[5]*np.sin(3*x) + \
           a[6]*np.cos(4*x) + a[7]*np.sin(4*x)

def basisfuncs(x):
    return np.array([np.cos(x), np.sin(x),
                     np.cos(2*x), np.sin(2*x),
                     np.cos(3*x), np.sin(3*x),
                     np.cos(4*x), np.sin(4*x)])
```

(Note that we could have used the `basisfuncs()` in `fitfunc()` – left as an exercise for the keen reader...)

Set up the $\mathsf{\alpha}$ matrix and the $\boldsymbol{\beta}$ vector (here we assume that all observations have the same error $\sigma = 1$):

```python
M = 8
sigma = 1.
alpha = np.zeros((M, M))
beta = np.zeros(M)
for x in X:
    Xk = basisfuncs(x)
    for k in range(M):
        for j in range(M):
            alpha[k, j] += Xk[k]*Xk[j]
for x, y in zip(X, Y):
    beta += y * basisfuncs(x)/sigma
```

Finally, solving the problem follows the same procedure as before:

Get the SVD:

```python
U, w, VT = np.linalg.svd(alpha)
V = VT.T
```

In this case, the singular values do not immediately show if any basis functions are superfluous (this would be the case for values close to 0).

```python
w
```

    array([296.92809624, 282.94804954, 243.7895787 , 235.7300808 ,
           235.15938555, 235.14838812, 235.14821093, 235.14821013])

... nevertheless, remember to routinely mask any singular values or close to singular values:

```python
w_inv = 1/w
w_inv[np.abs(w) < 1e-12] = 0
alpha_inv = V.dot(np.diag(w_inv)).dot(U.T)
```

Solve the system of equations with the pseudo-inverse:

```python
a_values = alpha_inv.dot(beta)
print(a_values)
```

    [-0.08436957  3.01646     0.30062685  0.03492872 -0.03711539 -1.70048432
      0.12217945  1.03404778]

Compare the fitted values to the original parameters $a_j = 0, +3, 0, 0, 0, -2, 0, +1$.

The original parameters show up as 3.15, -2.04 and 1.08 but the other parameters also have appreciable values. Given that the noise was sizable, this is not unreasonable.

Compare the plot of the underlying true function ("signal", dashed line) to the model ("fit", solid line):

```python
plt.plot(X, fitfunc(X, a_values), 'b-', label="fit")
plt.plot(X, signal(X, noise=0), 'k--', label="signal")
plt.legend(loc="best", fontsize="small")
```

We get some spurious oscillations but overall the result looks reasonable.

```python

```

#### Direct calculation
Instead of solving the compact $M \times M$ matrix equation $\mathsf{\alpha}\mathbf{a} = \boldsymbol{\beta}$, we can try to directly solve the overdetermined $M \times N$ equation

$$
\mathsf{A}\mathbf{a} = \mathbf{b}
$$

Creating the design matrix is straight-forward (but because of the way that `basisfuncs()` returns values, we need to transpose the output to get the proper $N \times M$ matrix $\mathsf{A}$):

```python
A = np.transpose(basisfuncs(X))
b = Y
```

```python
A.shape
```

    (500, 8)

Calculate the pseudo-inverse $\mathsf{A}^{-1}$.

Note that we need to explicitly construct the matrix with the inverses of the singular values by filling a $M \times N$ matrix with $\text{diag}(w_i)$.

```python
U, w, VT = np.linalg.svd(A)
V = VT.T
singular_values = np.abs(w) < 1e-12
winv = 1/w
winv[singular_values] = 0
winvmat = np.zeros((V.shape[0], U.shape[0]))
winvmat[:len(winv), :len(winv)] = np.diag(winv)
Ainv = V.dot(winvmat).dot(U.T)
```

The singular values are all well behaved:

```python
w
```

    array([17.23160167, 16.8210597 , 15.61376248, 15.35350386, 15.33490742,
           15.33454884, 15.33454306, 15.33454304])

```python
V.shape, winvmat.shape, U.T.shape
```

    ((8, 8), (8, 500), (500, 500))

```python
Ainv.shape
```

    (8, 500)

```python
A.shape
```

    (500, 8)

Now solve directly
$$
\mathsf{A}^{-1} \mathbf{b} = \mathbf{a}
$$

```python
a = Ainv.dot(b)
a
```

    array([-0.08436957,  3.01646   ,  0.30062685,  0.03492872, -0.03711539,
           -1.70048432,  0.12217945,  1.03404778])

The parameter estimates are the same as above.

```python
a_values - a
```

    array([-6.80011603e-16,  5.32907052e-15, -1.11022302e-16, -2.74780199e-15,
           -1.03389519e-15, -4.44089210e-16,  5.55111512e-17,  2.22044605e-15])

and hence the plot looks the same:

```python
plt.plot(X, fitfunc(X, a_values), 'b-', label="fit")
plt.plot(X, signal(X, noise=0), 'k--', label="signal")
plt.legend(loc="best", fontsize="small")
```

```python

```


```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sn

import pycollocation
```

Consider the following boundary value problem:
\begin{align}
    \frac{d T_1}{d_A} =& -q,\ T_1(0) = T_{1,0} \\
    \frac{d T_2}{d_A} =& -\frac{1}{2}q,\ T_2(A_{hx}) = T_{2,A_{hx}} \\
\end{align}
where $q = (T_1 - T_2)U$.

This boundary value problem describes a counter current heat exchanger; a hot liquid enters a device and exchanges heat across a metal plate with a cold liquid traveling through the device in the opposite direction. Here, $T_1$ is the temperature of the hot stream, $T_2$ is the temperature of the cold stream, $q$ is the rate of heat transfer from the hot fluid to the cold fluid, $U$ is the overall heat transfer coefficient and $A_hx$ is the total area of the heat exchanger.

```python
# provide numerical values for the parameters
params = {'T10': 130, 'T2Ahx': 70, 'U': 1.0}

def q(T1, T2, U):
    return (T1 - T2) * U

def rhs(A, T1, T2, U, **params):
    return [-q(T1, T2, U), -0.5 * q(T1, T2, U)]

def bcs_lower(A, T1, T2, T10, **params):
    return [T1 - T10]

def bcs_upper(A, T1, T2, T2Ahx, **params):
    return [T2 - T2Ahx]
```

```python
problem = pycollocation.problems.TwoPointBVP(bcs_lower, bcs_upper, 1, 2, params, rhs)
```

```python
def initial_mesh(domain, num, problem):
    As = np.linspace(domain[0], domain[1], num)
    T1s = np.repeat(0.5 * (problem.params['T10'] + problem.params['T2Ahx']), num)
    return As, T1s, T1s

```

```python
polynomial_basis = pycollocation.basis_functions.PolynomialBasis()
solver = pycollocation.solvers.Solver(polynomial_basis)

basis_kwargs = {'kind': 'Chebyshev', 'domain': [0, 5.0], 'degree': 15}

As, T1s, T2s = initial_mesh(basis_kwargs['domain'], 1000, problem)
T1_poly = polynomial_basis.fit(As, T1s, **basis_kwargs)
T2_poly = polynomial_basis.fit(As, T2s, **basis_kwargs)
initial_coefs = np.hstack([T1_poly.coef, T2_poly.coef])

solution = solver.solve(basis_kwargs, initial_coefs, problem)

```

```python
T1_soln, T2_soln = solution.evaluate_solution(As)
plt.plot(As, T1_soln)
plt.plot(As, T2_soln)
plt.show()
```

```python
T1_normalized_resids, T2_normalized_resids = solution.normalize_residuals(As)
plt.plot(As, np.abs(T1_normalized_resids))
plt.plot(As, np.abs(T2_normalized_resids))
plt.yscale('log')
plt.show()
```

```python

```