# Discriminative Classification

G. Richards (2016,2018), based on materials from Connolly, VanderPlas, and Ivezic.

Last time we talked about how to do classification by mapping the full pdf of your parameter space.  This time we will concentrate on methods that seek only to determine the **decision boundary**, so called [**discriminative classification**](https://en.wikipedia.org/wiki/Discriminative_model) methods.

As before, let's say that you have 2 blobs of data as shown below.  In many cases, you might say "just draw a line between those two blobs that are well separated".  So let's do exactly that in the example below.  There are clearly lots of different lines that you could draw that would work.  So, how do you do this *optimally*?  And what if the blobs are not perfectly well separated?

<!--  -->

```python
# Source: https://github.com/jakevdp/ESAC-stats-2014/blob/master/notebooks/04.1-Classification-SVMs.ipynb
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
from sklearn.datasets.samples_generator import make_blobs

X, y = make_blobs(n_samples=50, centers=2,
                  random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring' ,edgecolor='k');

Xgrid = np.linspace(-1, 3.5)
for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(Xgrid, m * Xgrid + b, '-k')

plt.xlim(-1, 3.5)
```

## Support Vector Machines
This is where [Support Vector Machines (SVM)](https://en.wikipedia.org/wiki/Support_vector_machine) come in.  We are going to define a hyperplane (a plane in $N-1$ dimensions) that maximizes the distance of the closest point from each class.  This distance is the "margin".  It is the width of the "cylinder" that you can put between the closest points that just barely touches the points in each class.  The points that touch the margin are called **support vectors**.  Obvious, right?

Once again, we have an algortihm that seems purposely named to frighten people away.  Though I don't know that "Data-Supported Hyperplane" classification would be any better...

```python
Xgrid = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring', edgecolor='k')

for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
    yfit = m * Xgrid+ b
    plt.plot(Xgrid, yfit, '-k')
    plt.fill_between(Xgrid, yfit - d, yfit + d, edgecolor='None', color='#AAAAAA', alpha=0.4)

plt.xlim(-1, 3.5)
```

To make life easier, we'll assume that the classes are separable by a straight line and that the decision boundary is at 0, with the two edges at $-1$ and $1$, and define $y \in \{-1,1\}$.

The hyperplane which maximizes the margin is given by finding

> \begin{equation}
\max_{\beta_0,\beta}(m) \;\;\;
  \mbox{subject to} \;\;\; \frac{1}{||\beta||} y_i ( \beta_0 + \beta^T x_i )
  \geq m \,\,\, \forall \, i.
\end{equation}

The constraints can be written as $y_i ( \beta_0 + \beta^T x_i ) \geq m ||\beta|| $.

Thus the optimization problem is equivalent to minimizing
> \begin{equation}
\frac{1}{2} ||\beta|| \;\;\; \mbox{subject to} \;\;\; y_i
  ( \beta_0 + \beta^T x_i ) \geq 1 \,\,\, \forall \, i.
\end{equation}

This optimization  is a _quadratic programming_ problem (quadratic objective function with linear constraints).

To make sure that we get through all the remaining classification algorithms, we'll skip over the mathematical details.  You can read about them in Ivezic $\S$ 9.6 or in Karen Leighly's [classification lecture notes](http://seminar.ouml.org/lectures/classification/).

For realistic data sets where the decision boundary is not obvious we relax the assumption that the classes are linearly separable.  This changes the minimization condition and puts bounds on the number of misclassifications (which we would obviously like to minimize).

Treating Scikit-Learn's agorithm as a black box, let's fit a Support Vector Machine Classifier to these points.

The Scikit-Learn implementation of SVM classification is [`SVC`](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC) which looks like:

```python
from sklearn.svm import SVC
svm = SVC(kernel='linear')
svm.fit(X,y)
```

In order to better visualize what SVM is doing, let's create a convenience function that will plot the decision boundaries:

```python
def plot_svc_decision_function(clf, ax=None):
    """Plot the decision function for a 2D SVC"""
    if ax is None:
        ax = plt.gca()
    u = np.linspace(plt.xlim()[0], plt.xlim()[1], 30)
    v = np.linspace(plt.ylim()[0], plt.ylim()[1], 30)
    yy, xx = np.meshgrid(v, u)

    P = np.zeros_like(xx)
    for i, ui in enumerate(u):
        for j, vj in enumerate(v):
            Xgrid = np.array([ui, vj])
            P[i, j] = clf.decision_function(Xgrid.reshape(1,-1))
    return ax.contour(xx, yy, P, colors='k',
                      levels=[-1, 0, 1], alpha=0.5,
                      linestyles=['--', '-', '--'])

    #GTR: Not clear why we need the loops and can't just
    # make the Xgrid array like we normally do.
```

Now let's plot the decision boundary and the support vectors, which are stored in the `support_vectors_` attribute of the classifier.

```python
plt.scatter(svm.support_vectors_[:, 0], svm.support_vectors_[:, 1], s=200, edgecolor='k', facecolor='w');
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring', edgecolor='k')
plot_svc_decision_function(svm)
```

Below is an example using the same data set from last time.

```python
%matplotlib inline
# Ivezic, Figure 9.10
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.svm import SVC
from astroML.decorators import pickle_results
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results

# SVM takes several minutes to run, and is order[N^2]
#  truncating the dataset can be useful for experimentation.
#X = X[::5]
#y = y[::5]

(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25], random_state=0)

N_tot = len(y)
N_stars = np.sum(y == 0)
N_rrlyrae = N_tot - N_stars
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rrlyrae

#----------------------------------------------------------------------
# Fit SVM
Ncolors = np.arange(1, X.shape[1] + 1)

#@pickle_results('SVM_rrlyrae.pkl')
def compute_SVM(Ncolors):
    y_class = []
    y_pred = []

    for nc in Ncolors:
        # perform support vector classification
        svm = SVC(kernel='linear', C=1, class_weight='balanced')
        svm.fit(X_train[:, :nc], y_train)

        y_pred.append(svm.predict(X_test[:, :nc]))
        y_class.append(svm)

    return y_class, y_pred

y_class, y_pred = compute_SVM(Ncolors)

completeness, contamination = completeness_contamination(y_pred, y_test)

print "completeness", completeness
print "contamination", contamination

#------------------------------------------------------------
# compute the decision boundary
svm = y_class[1]
w = svm.coef_[0]
a = -w[0] / w[1]
yy = np.linspace(-0.1, 0.4)
xx = a * yy - svm.intercept_[0] / w[1]

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(15, 7))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
ax.plot(xx, yy, '-k')
#Too many RR Lyrae to plot, so just show the last N_plot
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.binary, zorder=2)
im.set_clim(-0.5, 1)

ax.set_xlim(0.7, 1.35)
ax.set_ylim(-0.15, 0.4)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

Some comments on these results:

- The median of a distribution is unaffected by large perturbations of outlying points, as long as those perturbations do not cross the boundary.
- In the same way, once the support vectors are determined, changes to the positions or numbers of points beyond the margin will not change the decision boundary.  For this reason, SVM can be a very powerful tool for discriminative classification.
- This is why there is a high completeness compared to the other methods: it does not matter that the background sources outnumber the RR Lyrae stars by a factor of $\sim$200 to 1. It simply determines the best boundary between the small RR Lyrae clump and the large background clump.
- This completeness, however, comes at the cost of a relatively large contamination level.

Note that:
- SVM is not scale invariant so it often worth rescaling the data to [0,1] or to whiten it to have a mean of 0 and variance 1 (remember to do this to the test data as well!)
- The data don't need to be separable (we can put a constraint in minimizing the number of "failures")

### Kernel Methods
If the contamination is driven by non-linear effects (which isn't the case here), it may be worth implementing a non-linear decision boundary.  As before, we do that by *kernelization*.

Go to [Scikit-Learn SVM](http://scikit-learn.org/stable/modules/svm.html) (or better yet [this example](https://scikit-learn.org/stable/auto_examples/svm/plot_iris.html)) and see if you can figure out how to implement SVC with `kernel='rbf'` using the RR-Lyrae example above.  (As in Figure 9.11)  Also see what happens if you don't use `class_weight='balanced'`.

```python
%matplotlib inline
# Ivezic, Figure 9.10
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.svm import SVC
from astroML.decorators import pickle_results
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results

# SVM takes several minutes to run, and is order[N^2]
#  truncating the dataset can be useful for experimentation.
X = X[::5]
y = y[::5]

(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25], random_state=0)

N_tot = len(y)
N_stars = np.sum(y == 0)
N_rrlyrae = N_tot - N_stars
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rrlyrae

#----------------------------------------------------------------------
# Fit SVM
Ncolors = np.arange(1, X.shape[1] + 1)

#@pickle_results('SVM_rrlyrae.pkl')
def compute_SVM(Ncolors):
    y_class = []
    y_pred = []

    for nc in Ncolors:
        # perform support vector classification
        #svm = SVC(kernel='linear', C=1, class_weight='balanced')
        svm = SVC(___,gamma=___,C=___,class_weight=___) # Complete
        svm.fit(X_train[:, :nc], y_train)

        y_pred.append(svm.predict(X_test[:, :nc]))
        y_class.append(svm)

    return y_class, y_pred

y_class, y_pred = compute_SVM(Ncolors)

completeness, contamination = completeness_contamination(y_pred, y_test)

print "completeness", completeness
print "contamination", contamination

#------------------------------------------------------------
# compute the decision boundary
svm = y_class[1]
#w = svm.coef_[0]
#a = -w[0] / w[1]
#yy = np.linspace(-0.1, 0.4)
#xx = a * yy - svm.intercept_[0] / w[1]

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(15, 7))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
#ax.plot(xx, yy, '-k')
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.binary, zorder=2)

plot_svc_decision_function(svm)

im.set_clim(-0.5, 1)

ax.set_xlim(0.7, 1.35)
ax.set_ylim(-0.15, 0.4)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

Let's take a quick look at an example where the data are not linearly separable and where kernelization really makes a difference.

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring', edgecolor='k')
plot_svc_decision_function(clf);
```

But we can make a transform of the data to *make* it linearly separable, for example with a simple **radial basis function** as shown below.

```python
# Transform X using a radial basis function
z = np.exp(-(X[:, 0] ** 2 + X[:, 1] ** 2))

from IPython.html.widgets import interact
from mpl_toolkits import mplot3d

def plot_3D(elev=30, azim=30):
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], z, c=y, s=50, cmap='spring', edgecolor='k')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('z')

plot_3D()
# GTR: Or even fancier with
# interact(plot_3D, elev=[-90, 90], azip=(-180, 180));
```

Now we can trivially separate these populations as shown below!

```python
clf = SVC(kernel='rbf')
clf.fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring', edgecolor='k')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, facecolors='none');
```

## Decision Trees

A [**decision tree**](https://en.wikipedia.org/wiki/Decision_tree) is similar to the process of classification that you might do by hand: define some criteria to separate the sample into 2 groups (not necessarily equal).  Then take those sub-groups and do it again.  Keep going until you reach a stopping point such as not having a minimum number of objects to split again.  In short, we have done a hierarchical application of decision boundaries.

The tree structure is as follows:
- top node contains the entire data set
- at each branch the data are subdivided into two child nodes
- split is based on a predefined decision boundary (usually axis aligned)
- splitting repeats, recursively, until we reach a predefined stopping criteria

Below is a simple example of a decision tree.

```python
# Source: Jake VanderPlas, https://github.com/LocalGroupAstrostatistics2015/MachineLearning/blob/master/fig_code/figures.py
def plot_example_decision_tree():
    fig = plt.figure(figsize=(10, 4))
    ax = fig.add_axes([0, 0, 0.8, 1], frameon=False, xticks=[], yticks=[])
    ax.set_title('Example Decision Tree: Animal Classification', size=24)

    def text(ax, x, y, t, size=20, **kwargs):
        ax.text(x, y, t,
                ha='center', va='center', size=size,
                bbox=dict(boxstyle='round', ec='k', fc='w'), **kwargs)

    text(ax, 0.5, 0.9, "How big is\nthe animal?", 20)
    text(ax, 0.3, 0.6, "Does the animal\nhave horns?", 18)
    text(ax, 0.7, 0.6, "Does the animal\nhave two legs?", 18)
    text(ax, 0.12, 0.3, "Are the horns\nlonger than 10cm?", 14)
    text(ax, 0.38, 0.3, "Is the animal\nwearing a collar?", 14)
    text(ax, 0.62, 0.3, "Does the animal\nhave wings?", 14)
    text(ax, 0.88, 0.3, "Does the animal\nhave a tail?", 14)

    text(ax, 0.4, 0.75, "> 1m", 12, alpha=0.4)
    text(ax, 0.6, 0.75, "< 1m", 12, alpha=0.4)

    text(ax, 0.21, 0.45, "yes", 12, alpha=0.4)
    text(ax, 0.34, 0.45, "no", 12, alpha=0.4)

    text(ax, 0.66, 0.45, "yes", 12, alpha=0.4)
    text(ax, 0.79, 0.45, "no", 12, alpha=0.4)

    ax.plot([0.3, 0.5, 0.7], [0.6, 0.9, 0.6], '-k')
    ax.plot([0.12, 0.3, 0.38], [0.3, 0.6, 0.3], '-k')
    ax.plot([0.62, 0.7, 0.88], [0.3, 0.6, 0.3], '-k')
    ax.plot([0.0, 0.12, 0.20], [0.0, 0.3, 0.0], '--k')
    ax.plot([0.28, 0.38, 0.48], [0.0, 0.3, 0.0], '--k')
    ax.plot([0.52, 0.62, 0.72], [0.0, 0.3, 0.0], '--k')
    ax.plot([0.8, 0.88, 1.0], [0.0, 0.3, 0.0], '--k')
    ax.axis([0, 1, 0, 1])

plot_example_decision_tree()
```

<!-- For our RR Lyrae stars -->
<! --  -->

The "leaf (terminal) nodes" record the fraction of points that have one classification or the other

Application of the tree to classification is simple (a series of binary decisions). The  fraction of points from the training set classified as one class or the other (in the leaf node) defines the class associated with that leaf node.

The binary splitting makes this extremely efficient. Tthe trick is to ask the *right* questions.
So, decision trees are simple to interpret (just a set of questions).

Scikit-learn implements the [`DecisionTreeClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) as follows:

```python
import numpy as np
from sklearn.tree import DecisionTreeClassifier
X = np.random.random((100,2))
y = (X[:,0] + X[:,1] > 1).astype(int)
dtree = DecisionTreeClassifier(max_depth=6)
dtree.fit(X,y)
y_pred = dtree.predict(X)
```

An example with our data set of RR Lyrae stars shows that it has moderately good completenees and contamination, but that, for this data set, it is not the optimal choice.

```python
# Ivezic, Figure 9.13
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.tree import DecisionTreeClassifier
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25], random_state=0)

N_tot = len(y)
N_stars = np.sum(y == 0)
N_rrlyrae = N_tot - N_stars
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rrlyrae

#----------------------------------------------------------------------
# Fit Decision tree
Ncolors = np.arange(1, X.shape[1] + 1)

y_class = []
y_pred = []
Ncolors = np.arange(1, X.shape[1] + 1)
depths = [7, 12]

for depth in depths:
    y_class.append([])
    y_pred.append([])
    for nc in Ncolors:
        dt = DecisionTreeClassifier(random_state=0, max_depth=depth,
                                     criterion='entropy')
        dt.fit(X_train[:, :nc], y_train)
        y_pred[-1].append(dt.predict(X_test[:, :nc]))
        y_class[-1].append(dt)

completeness, contamination = completeness_contamination(y_pred, y_test)

print "completeness", completeness
print "contamination", contamination

#------------------------------------------------------------
# compute the decision boundary

dt = y_class[1][1]
xlim = (0.7, 1.35)
ylim = (-0.15, 0.4)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 101),
                     np.linspace(ylim[0], ylim[1], 101))
#GTR Whoa!?  Why is this yy, xx and not xx, yy????
#Ah, because the plot reverses the usual python order
xystack = np.vstack([yy.ravel(),xx.ravel()])
Xgrid = xystack.T

Z = dt.predict(Xgrid)
Z = Z.reshape(xx.shape)

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(13, 6))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.binary, zorder=2)
im.set_clim(-0.5, 1)

ax.contour(xx, yy, Z, [0.5], colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

ax.text(0.02, 0.02, "depth = %i" % depths[1],
        transform=ax.transAxes)

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness[0], 'o-k', ms=6, label="depth=%i" % depths[0])
ax.plot(Ncolors, completeness[1], '^--k', ms=6, label="depth=%i" % depths[1])

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination[0], 'o-k', ms=6, label="depth=%i" % depths[0])
ax.plot(Ncolors, contamination[1], '^--k', ms=6, label="depth=%i" % depths[1])
ax.legend(loc='lower right', bbox_to_anchor=(1.0, 0.79))

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

### Splitting Criteria

Now let's talk about the best ways to split the data.  This is actually a really hard problem, which you can read about more in Ivezic $\S$ 9.7.1.

One way is to use the information content or entropy, $E(x)$, of the data

$$ E(x) = -\sum_i p_i(x) \ln (p_i(x)),$$

where $i$ is the class and $p_i(x)$ is the probability of that class
given the training data.

Another commonly used "loss function" (especially for categorical classification) is the Gini coefficient:
$$ G = \sum_i^k p_i(1-p_i).$$

It essentially estimates the probability of incorrect classification by choosing both a point and (separately) a class randomly from the data.

Try changing the example above to use `criterion='gini'` and `min_samples_leaf=3`, see [here](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier) for all of the parameters.

Obviously in constructing a decision treee, if your choice of stopping criteria is too loose, further splitting just ends up adding noise.  So here is an example using cross-validation in order to optimize the depth of the tree (and to avoid overfitting).

Note that here we aren't classifying the objects into discrete categories, rather we are classifying them into a continuous category.  That is, we are doing regression.  In this particular case, we are using the colors of galaxies in order to predict their redshifts (distances).

```python
# Ivezic, Figure 9.14
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.tree import DecisionTreeRegressor
from astroML.datasets import fetch_sdss_specgals

#------------------------------------------------------------
# Fetch data and prepare it for the computation
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 separate points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the cross-validation scores for several tree depths
depth = np.arange(1, 21)
rms_test = np.zeros(len(depth))
rms_train = np.zeros(len(depth))
i_best = 0
z_fit_best = None

for i, d in enumerate(depth):
    clf = DecisionTreeRegressor(max_depth=d, random_state=0)
    clf.fit(mag_train, z_train)

    z_fit_train = clf.predict(mag_train)
    z_fit_test = clf.predict(mag_test)
    rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
    rms_test[i] = np.mean(np.sqrt((z_fit_test - z_test) ** 2))

    if rms_test[i] <= rms_test[i_best]:
        i_best = i
        z_fit_best = z_fit_test

best_depth = depth[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# first panel: cross-validation
ax = fig.add_subplot(121)
ax.plot(depth, rms_test, '-k', label='cross-validation')
ax.plot(depth, rms_train, '--k', label='training set')
ax.set_xlabel('depth of tree')
ax.set_ylabel('rms error')
ax.yaxis.set_major_locator(plt.MultipleLocator(0.01))
ax.set_xlim(0, 21)
ax.set_ylim(0.009,  0.04)
ax.legend(loc=1)

# second panel: best-fit results
ax = fig.add_subplot(122)
ax.scatter(z_test, z_fit_best, s=1, lw=0, c='k')
ax.plot([-0.1, 0.4], [-0.1, 0.4], ':k')
ax.text(0.04, 0.96, "depth = %i\nrms = %.3f" % (best_depth, rms_test[i_best]),
        ha='left', va='top', transform=ax.transAxes)
ax.set_xlabel(r'$z_{\rm true}$')
ax.set_ylabel(r'$z_{\rm fit}$')

ax.set_xlim(-0.02, 0.4001)
ax.set_ylim(-0.02, 0.4001)
ax.xaxis.set_major_locator(plt.MultipleLocator(0.1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.1))

plt.show()
```

That's doing the Cross Validation by hand, let's try it automatically using data like the first example that we started with today.

```python
X, y = make_blobs(n_samples=500, centers=3,
                  random_state=0, cluster_std=1.50)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='spring',edgecolor='k')
```

```python
#from sklearn.grid_search import GridSearchCV
from sklearn.model_selection import GridSearchCV

clf = DecisionTreeClassifier()
drange = np.arange(1,21)

grid = GridSearchCV(clf, param_grid={'max_depth': drange}, cv=5)
grid.fit(X, y)

best = grid.best_params_['max_depth']
print("best parameter choice:", best)
```

Now plot the decision boundary

```python
dt = DecisionTreeClassifier(random_state=0, max_depth=best, criterion='entropy')
dt.fit(X, y)

xlim = (-4, 8)
ylim = (-6, 10)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 51),
                     np.linspace(ylim[0], ylim[1], 51))
xystack = np.vstack([xx.ravel(),yy.ravel()])
Xgrid = xystack.T

Z = dt.predict(Xgrid)
Z = Z.reshape(xx.shape)
print Z

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(111)
im = ax.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap=plt.cm.spring, zorder=2, edgecolor='k')

ax.contour(xx, yy, Z, [0.5], colors='k')
```

Note that Decision Trees are the basis for the rest of the material today.  So it is useful to consider some of the [advantages and disadvantages](https://scikit-learn.org/stable/modules/tree.html).

## Ensemble Learning

You may have noticed that each of the classification methods that we have described so far has its strengths and weaknesses.  You might wonder if we could gain something by some sort of averaging of weighted "voting".  Such a process is what we call *ensemble learning*.  We'll discuss two such processes: [**bagging**](https://en.wikipedia.org/wiki/Bootstrap_aggregating) and [**random forests**](https://en.wikipedia.org/wiki/Random_forest).

### Bagging

Bagging (short for *bootstrap aggregation*--a name, unlike SVM, which actually makes some sense) can significantly improve the performance of decision trees.  In short, bagging averages the predictive results of a series of *bootstrap* samples.

Remember that instead of splitting the sample into training and test sets that do not overlap, bootstrap says to draw from the observed data set with replacement.  So we select indices $j$ from the range $i=1,\ldots,N$ and this is our new sample.  Some indices, $i$, will be repeated and we do this $B$ times.

For a sample of $N$ points in a training set, bagging generates $B$ equally sized bootstrap samples from which to estimate the function $f_i(x)$. The final estimator for $\hat{y}$, defined by bagging, is then
$$\hat{y} = f(x) = \frac{1}{B} \sum_i^B f_i(x).$$

### Random Forests
Random forests extend bagging by generating decision trees from the bootstrap samples.  A interesting aspect of random forests is that the features on which to generate the tree are selected at random from the full set of features in the data (the number of features selected per split level is typically the square root of the total number of attributes, $\sqrt{D}$).  The final classification from the random forest is based on the averaging of the classifications of each of the individual decision trees.  So, you can literally give it the kitchen sink (including attributes that you might not otherwise think would be useful for classification).

Random forests help to overcome some of the limitations of decision trees.

As before, cross-validation can be used to determine the optimal depth.  Generally the number of trees, $n$, that are chosen is the number at which the cross-validation error plateaus.

Below we give the same example as above for estimation of galaxy redshifts, where Scikit-Learn's [`RandomForestClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) call looks as follows:

```python
import numpy as np
from sklearn.ensemble import RandomForestClassifier
X = np.random.random((100,2))
y = (X[:,0] + X[:,1] > 1).astype(int)
ranfor = RandomForestClassifier(10)
ranfor.fit(X,y)
y_pred = ranfor.predict(X)
```

```python
# Ivezic, Figure 9.15
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.ensemble import RandomForestRegressor
from astroML.datasets import fetch_sdss_specgals

#------------------------------------------------------------
# Fetch and prepare the data
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 distinct points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the results
def compute_photoz_forest(depth):
    rms_test = np.zeros(len(depth))
    rms_train = np.zeros(len(depth))
    i_best = 0
    z_fit_best = None

    for i, d in enumerate(depth):
        clf = RandomForestRegressor(n_estimators=10,
                                    max_depth=d, random_state=0)
        clf.fit(mag_train, z_train)

        z_fit_train = clf.predict(mag_train)
        z_fit = clf.predict(mag_test)
        rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
        rms_test[i] = np.mean(np.sqrt((z_fit - z_test) ** 2))

        if rms_test[i] <= rms_test[i_best]:
            i_best = i
            z_fit_best = z_fit

    return rms_test, rms_train, i_best, z_fit_best

depth = np.arange(1, 21)
rms_test, rms_train, i_best, z_fit_best = compute_photoz_forest(depth)
best_depth = depth[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(13, 6))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# left panel: plot cross-validation results
ax = fig.add_subplot(121)
ax.plot(depth, rms_test, '-k', label='cross-validation')
ax.plot(depth, rms_train, '--k', label='training set')
ax.legend(loc=1)

ax.set_xlabel('depth of tree')
ax.set_ylabel('rms error')

ax.set_xlim(0, 21)
ax.set_ylim(0.009,  0.04)
ax.yaxis.set_major_locator(plt.MultipleLocator(0.01))

# right panel: plot best fit
ax = fig.add_subplot(122)
ax.scatter(z_test, z_fit_best, s=1, lw=0, c='k')
ax.plot([-0.1, 0.4], [-0.1, 0.4], ':k')
ax.text(0.03, 0.97, "depth = %i\nrms = %.3f" % (best_depth, rms_test[i_best]),
        ha='left', va='top', transform=ax.transAxes)

ax.set_xlabel(r'$z_{\rm true}$')
ax.set_ylabel(r'$z_{\rm fit}$')

ax.set_xlim(-0.02, 0.4001)
ax.set_ylim(-0.02, 0.4001)
ax.xaxis.set_major_locator(plt.MultipleLocator(0.1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.1))

plt.show()
```

How many attributes/features is the code currently using?  Looking at [`RandomForestClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), how might you use `max_features` to change this?

## Boosting

Boosting is an ensemble approach motivated by the idea that combining many weak classifiers can result in an improved classification. Boosting creates models that attempt to correct the errors of the ensemble so far. At the heart of boosting is the idea that we reweight the data based on how incorrectly the data were classified in the previous iteration.

We run the classification multiple times and each time reweight the data based on the previous performance of the classifier. At the end of this procedure we allow the classifiers to vote on the final classification. The most popular form of boosting is that of adaptive boosting.  In this case we take a weak classifier, $h(x)$, and create a strong classifier, $f(x)$, as
$$ f(x) = \sum_m^B\theta_m h_m(x),$$
where $m$ is the number of iterations and $\theta_m$ is the weight of the classifier in each iteration.

If we chose $\theta_m=1/B$, then we'd essentially have bagging.  For boosting the idea is to increase the weight of the misclassified data in each step.

A fundamental limitation of the boosted decision tree is the computation time for large data sets (they rely on a chain of classifiers which are each dependent on the last), whereas random forests can be easily parallelized.

The example given below is actually Scikit-Learn's [`GradientBoostingClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) where we approximate the steepest descent criterion after each simple evaluation.

```python
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
X = np.random.random((100,2))
y = (X[:,0] + X[:,1] > 1).astype(int)
gradboost = GradientBoostingClassifier()
gradboost.fit(X,y)
y_pred = gradboost.predict(X)
```

```python
# Ivezic, Figure 9.16
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.ensemble import GradientBoostingRegressor
from astroML.datasets import fetch_sdss_specgals

#------------------------------------------------------------
# Fetch and prepare the data
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 distinct points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the results
def compute_photoz_forest(N_boosts):
    rms_test = np.zeros(len(N_boosts))
    rms_train = np.zeros(len(N_boosts))
    i_best = 0
    z_fit_best = None

    for i, Nb in enumerate(N_boosts):
        try:
            # older versions of scikit-learn
            clf = GradientBoostingRegressor(n_estimators=Nb, learn_rate=0.1,
                                            max_depth=3, max_features='sqrt', random_state=0)
        except TypeError:
            clf = GradientBoostingRegressor(n_estimators=Nb, learning_rate=0.1,
                                            max_depth=3, max_features='sqrt', random_state=0)
        clf.fit(mag_train, z_train)

        z_fit_train = clf.predict(mag_train)
        z_fit = clf.predict(mag_test)
        rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
        rms_test[i] = np.mean(np.sqrt((z_fit - z_test) ** 2))

        if rms_test[i] <= rms_test[i_best]:
            i_best = i
            z_fit_best = z_fit

    return rms_test, rms_train, i_best, z_fit_best

N_boosts = (10, 100, 200, 300, 400, 500)
rms_test, rms_train, i_best, z_fit_best = compute_photoz_forest(N_boosts)
best_N = N_boosts[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(12, 6))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# left panel: plot cross-validation results
ax = fig.add_subplot(121)
ax.plot(N_boosts, rms_test, '-k', label='cross-validation')
ax.plot(N_boosts, rms_train, '--k', label='training set')
ax.legend(loc=1)

ax.set_xlabel('number of boosts')
ax.set_ylabel('rms error')
ax.set_xlim(0, 510)
ax.set_ylim(0.009,  0.032)
ax.yaxis.set_major_locator(plt.MultipleLocator(0.01))

ax.text(0.03, 0.03, "Tree depth: 3",
        ha='left', va='bottom', transform=ax.transAxes)

# right panel: plot best fit
ax = fig.add_subplot(122)
ax.scatter(z_test, z_fit_best, s=1, lw=0, c='k')
ax.plot([-0.1, 0.4], [-0.1, 0.4], ':k')
ax.text(0.04, 0.96, "N = %i\nrms = %.3f" % (best_N, rms_test[i_best]),
        ha='left', va='top', transform=ax.transAxes)

ax.set_xlabel(r'$z_{\rm true}$')
ax.set_ylabel(r'$z_{\rm fit}$')

ax.set_xlim(-0.02, 0.4001)
ax.set_ylim(-0.02, 0.4001)
ax.xaxis.set_major_locator(plt.MultipleLocator(0.1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.1))

plt.show()
```

## Alright, alright but what the @#%! should I use?

A convenient cop-out: no single model can be known in advance to be the best classifier!

In general the level of accuracy increases for parametric models as:
- <b>naive Bayes</b>,
- linear discriminant analysis (LDA),
- logistic regression,
- linear support vector machines,
- quadratic discriminant analysis (QDA),
- linear ensembles of linear models.

For non-parametric models accuracy increases as:
- decision trees
- $K$-nearest-neighbor,
- neural networks
- kernel discriminant analysis,
- <b> kernelized support vector machines</b>
- <b> random forests</b>
- boosting

See also Ivezic, Table 9.1.

Naive Bayes and its variants are by far the easiest to compute. Linear support vector machines are more expensive, though several fast algorithms exist. Random forests can be easily parallelized.

We saw before that Scikit-learn has tools for computing ROC curves, which is implemented as follows.

```python
import numpy as np
from sklearn.naive_bayes import GaussianNB
from sklearn import metrics
X = np.random.random((100,2))
y = (X[:,0] + X[:,1] > 1).astype(int)

gnb = GaussianNB().fit(X,y)
y_prob = gnb.predict_proba(X)

# Compute precision/recall curve
pr, re, thresh = metrics.precision_recall_curve(y, y_prob[:,0])

# Compute ROC curve
tpr, fpr, thresh = metrics.roc_curve(y, y_prob[:,0])
```

Let's remember what they had to say:

Here's an example with a different data set.  Here we are trying to distinguish quasars (in black) from stars (in grey)

```python
# Ivezic, 9.18
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from astroML.utils import split_samples

from sklearn.metrics import roc_curve
from sklearn.naive_bayes import GaussianNB
#from sklearn.lda import LDA
#from sklearn.qda import QDA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
#from astroML.classification import GMMBayes
from sklearn.mixture import GaussianMixture

#------------------------------------------------------------
# Fetch data and split into training and test samples
from astroML.datasets import fetch_dr7_quasar
from astroML.datasets import fetch_sdss_sspp

quasars = fetch_dr7_quasar()
stars = fetch_sdss_sspp()

# Truncate data for speed
quasars = quasars[::5]
stars = stars[::5]

# stack colors into matrix X
Nqso = len(quasars)
Nstars = len(stars)
X = np.empty((Nqso + Nstars, 4), dtype=float)

X[:Nqso, 0] = quasars['mag_u'] - quasars['mag_g']
X[:Nqso, 1] = quasars['mag_g'] - quasars['mag_r']
X[:Nqso, 2] = quasars['mag_r'] - quasars['mag_i']
X[:Nqso, 3] = quasars['mag_i'] - quasars['mag_z']

X[Nqso:, 0] = stars['upsf'] - stars['gpsf']
X[Nqso:, 1] = stars['gpsf'] - stars['rpsf']
X[Nqso:, 2] = stars['rpsf'] - stars['ipsf']
X[Nqso:, 3] = stars['ipsf'] - stars['zpsf']

y = np.zeros(Nqso + Nstars, dtype=int)
y[:Nqso] = 1

# split into training and test sets
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.9, 0.1],
                                                     random_state=0)

#------------------------------------------------------------
# Compute fits for all the classifiers
def compute_results(*args):
    names = []
    probs = []

    for classifier, kwargs in args:
        print classifier.__name__
        model = classifier(**kwargs)
        model.fit(X, y)
        y_prob = model.predict_proba(X_test)

        names.append(classifier.__name__)
        probs.append(y_prob[:, 1])

    return names, probs

LRclass_weight = dict([(i, np.sum(y_train == i)) for i in (0, 1)])

names, probs = compute_results((GaussianNB, {}),
                               (LDA, {}),
                               (QDA, {}),
                               (LogisticRegression,
                                dict(class_weight=LRclass_weight)),
                               (KNeighborsClassifier,
                                dict(n_neighbors=10)),
                               (DecisionTreeClassifier,
                                dict(random_state=0, max_depth=12,
                                     criterion='entropy')),
                               (GaussianMixture, dict(n_components=3, tol=1E-5,
                                               covariance_type='full')))

#------------------------------------------------------------
# Plot results
fig = plt.figure(figsize=(13, 7))
fig.subplots_adjust(left=0.1, right=0.95, bottom=0.15, top=0.9, wspace=0.25)

# First axis shows the data
ax1 = fig.add_subplot(121)
im = ax1.scatter(X_test[:, 0], X_test[:, 1], c=y_test, s=4,
                 linewidths=0, edgecolors='none',
                 cmap=plt.cm.binary)
im.set_clim(-0.5, 1)
ax1.set_xlim(-0.5, 3.0)
ax1.set_ylim(-0.3, 1.4)
ax1.set_xlabel('$u - g$')
ax1.set_ylabel('$g - r$')

labels = dict(GaussianNB='GNB',
              LinearDiscriminantAnalysis='LDA',
              QuadraticDiscriminantAnalysis='QDA',
              KNeighborsClassifier='KNN',
              DecisionTreeClassifier='DT',
              GaussianMixture='GMMB',
              LogisticRegression='LR')

# Second axis shows the ROC curves
ax2 = fig.add_subplot(122)
for name, y_prob in zip(names, probs):
    fpr, tpr, thresholds = roc_curve(y_test, y_prob)

    fpr = np.concatenate([[0], fpr])
    tpr = np.concatenate([[0], tpr])

    ax2.plot(fpr, tpr, label=labels[name])
ax2.legend(loc=4)
ax2.set_xlabel('false positive rate')
ax2.set_ylabel('true positive rate')
ax2.set_xlim(0, 0.15)
ax2.set_ylim(0.6, 1.01)
ax2.xaxis.set_major_locator(plt.MaxNLocator(5))

plt.show()
```

Curiously GMMBayes went from being one of the best to one of the worst after I changed from using the deprecated GMMBayes to GaussianMixture.  So, it is likely that the current input parameters are not optimal for that.

Add a `precision_recall` plot.

```python
# Ivezic, 9.18
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from astroML.utils import split_samples

from sklearn.metrics import roc_curve
from sklearn.metrics import ____  #Complete
from sklearn.naive_bayes import GaussianNB
#from sklearn.lda import LDA
#from sklearn.qda import QDA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
#from astroML.classification import GMMBayes
from sklearn.mixture import GaussianMixture

#------------------------------------------------------------
# Fetch data and split into training and test samples
from astroML.datasets import fetch_dr7_quasar
from astroML.datasets import fetch_sdss_sspp

quasars = fetch_dr7_quasar()
stars = fetch_sdss_sspp()

# Truncate data for speed
quasars = quasars[::5]
stars = stars[::5]

# stack colors into matrix X
Nqso = len(quasars)
Nstars = len(stars)
X = np.empty((Nqso + Nstars, 4), dtype=float)

X[:Nqso, 0] = quasars['mag_u'] - quasars['mag_g']
X[:Nqso, 1] = quasars['mag_g'] - quasars['mag_r']
X[:Nqso, 2] = quasars['mag_r'] - quasars['mag_i']
X[:Nqso, 3] = quasars['mag_i'] - quasars['mag_z']

X[Nqso:, 0] = stars['upsf'] - stars['gpsf']
X[Nqso:, 1] = stars['gpsf'] - stars['rpsf']
X[Nqso:, 2] = stars['rpsf'] - stars['ipsf']
X[Nqso:, 3] = stars['ipsf'] - stars['zpsf']

y = np.zeros(Nqso + Nstars, dtype=int)
y[:Nqso] = 1

# split into training and test sets
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.9, 0.1],
                                                     random_state=0)

#------------------------------------------------------------
# Compute fits for all the classifiers
def compute_results(*args):
    names = []
    probs = []

    for classifier, kwargs in args:
        print classifier.__name__
        model = classifier(**kwargs)
        model.fit(X, y)
        y_prob = model.predict_proba(X_test)

        names.append(classifier.__name__)
        probs.append(y_prob[:, 1])

    return names, probs

LRclass_weight = dict([(i, np.sum(y_train == i)) for i in (0, 1)])

names, probs = compute_results((GaussianNB, {}),
                               (LDA, {}),
                               (QDA, {}),
                               (LogisticRegression,
                                dict(class_weight=LRclass_weight)),
                               (KNeighborsClassifier,
                                dict(n_neighbors=10)),
                               (DecisionTreeClassifier,
                                dict(random_state=0, max_depth=12,
                                     criterion='entropy')),
                               (GaussianMixture, dict(n_components=3, tol=1E-5,
                                               covariance_type='full')))

#------------------------------------------------------------
# Plot results
fig = plt.figure(figsize=(18, 7))
fig.subplots_adjust(left=0.1, right=0.95, bottom=0.15, top=0.9, wspace=0.25)

# First axis shows the data
ax1 = fig.add_subplot(131)
im = ax1.scatter(X_test[:, 0], X_test[:, 1], c=y_test, s=4,
                 linewidths=0, edgecolors='none',
                 cmap=plt.cm.binary)
im.set_clim(-0.5, 1)
ax1.set_xlim(-0.5, 3.0)
ax1.set_ylim(-0.3, 1.4)
ax1.set_xlabel('$u - g$')
ax1.set_ylabel('$g - r$')

labels = dict(GaussianNB='GNB',
              LinearDiscriminantAnalysis='LDA',
              QuadraticDiscriminantAnalysis='QDA',
              KNeighborsClassifier='KNN',
              DecisionTreeClassifier='DT',
              GaussianMixture='GMMB',
              LogisticRegression='LR')

# Second axis shows the ROC curves
ax2 = fig.add_subplot(132)
ax3 = fig.add_subplot(133)
for name, y_prob in zip(names, probs):
    fpr, tpr, thresholds = roc_curve(y_test, y_prob)
    precision, recall, thresholds2 = ____(____, ____) # Complete

    fpr = np.concatenate([[0], fpr])
    tpr = np.concatenate([[0], tpr])

    precision = ___.___(___,___) # Complete
    recall = ___.___(___,___) # Complete

    ax2.plot(fpr, tpr, label=labels[name])
    ax3.plot(____, ____, label=labels[name]) # Complete

ax2.legend(loc=4)
ax2.set_xlabel('false positive rate')
ax2.set_ylabel('true positive rate')
ax2.set_xlim(0, 0.15)
ax2.set_ylim(0.6, 1.01)
ax2.xaxis.set_major_locator(plt.MaxNLocator(5))

ax3.set_xlim(0.5, 1.01)
ax3.set_ylim(0.5, 1.01)
ax3.set_xlabel('recall')
ax3.set_ylabel('precision')

plt.show()
```


# Crypto

## RSA Attacks

### Wiener Attack

>$\phi(n)$  = (p - 1) (q - 1)

 $\phi(n)$  = pq - (p + q) + 1

 $\phi(n)$  $\cong$  n

 ed = k * $\phi(n)$ + n

 ed - k * $\phi(n)$  = 1

 $\frac{e}{\phi(n))}$ - $\frac{k}{d}$ = $\frac1{d * \phi(n)}$

 Now $\frac1{ d * phi(n)}$ is very small

 $\therefore \frac{e}{n}$  $\cong$ $\frac{d}{n}$

 We use continued fractions to get approx of $\frac{k}{d}$

Few Observations:
1. As ed = 1 mod $\phi$(n) , and $\phi$(n) is even, Therefore e and d must be odd
2. $\phi$(n) is a whole number. Therefore (ed - 1) / k must be a whole number
3. If we get $\phi$(n) as a whole number,  we can check if it is indeed the correct $\phi$(n) by: \
$\phi$(n) = (p - 1) * (q - 1) = n - (p + q) + 1 \
p + q = n - $\phi$(n) + 1 \
Consider a quadratic equation :
    (x - p) (x - q) = 0

    ${x}^2$ - (p + q) + pq = 0

    ${x}^2$ - (n - $\phi$(n) + 1) + n = 0

If this equation has integral roots, then the $\phi$(n) is correct.

Ref : https://www.youtube.com/watch?v=OpPrrndyYNU

### Fermat factoring algorithm

The algorithm is based upon the being able to factor the difference of 2 squares

>n = ${x}^2$ - ${y}^2$

 n = (x - y) / (x + y)

 And we know that every positive odd integer can be written as the difference of two squares.

For n = pq

    n = ${(\frac{p + q}{2})} ^2$  - ${(\frac{p - q}{2})} ^2$

Let k be the smallest positive integer so that k > $\sqrt n$ \
Consider ${k}^2$ - n,

if it is a perfect square i.e.

   n = (k + h) * (k - h)

else:

      k = k + 1

Eventually we will find an h such that ${k + h}^2$ - n can be factored because:

    ${\frac{n + 1}{2}^2}$ - n = ${\frac{n - 1}{2}^2}$

Fermat’s factorization algorithm works well if the factors are roughly the ${\bf same \ size.}$ (p and q are close together)

### Hastads Broadcasting Attack

>c_1 = $m^{e}$ mod N1

c_2 = $m^{e}$ mod N2

c_3 = $m^{e}$ mod N3

Given GCD(N_i, N_j) = 1

c = c1 mod N1

c = c2 mod N2

c = c3 mod N3

Using CRT, we can find c

c = $m^e$ mod n1 * n2 * n3

$\therefore$  m =  $\sqrt[e]{c}$

Ref: https://www.youtube.com/watch?v=DKWnvyCsh9A

## Number Theory

### Fermat Little Theorem

if gcd(a, p) = 1, p is a prime number and a is an integer

        ${a}^{p - 1}$ = 1 mod p

We considering the first p - 1 positive multiples of a

        a , 2a , 3a , ... , (p - l)a

None of these numbers is congruent modulo p to any other nor is any congruent to zero. Because it it happened,

ra $\cong$ sa mod(p)   where 1 <= r < s <= p - 1

then a could be canceled to give r = s (mod p), which is impossible.

        ra - sa = kp
        a(r - s) = kp
        and p is prime.

$\therefore$ the previous set of integers must be congruent modulo 1, 2 , 3 , ... , p - 1

Multiplying,
    a· 2a · 3a · · · (p - l)a = 1 · 2 · 3 · · · (p - 1) (mod p)

    ${a}^{p - 1}$ = 1 mod p

### Chinese Remainder Theorem

    Let n1, n2, ... , nr be positive integers such that gcd(ni, nj) = 1 for i != j

        x = a1 (mod n1)
		x = a2 (mod n2)
		...
		...
		x = ar (mod nr)

	Then this has a simultaneous solution, which is unique modulo the integer n1n2...nr

	Proof(Intuition):
		Let N = n1*n2....*nr

		And Nk = N / nk

		Then  x = a1*N1*x1 + a2*N2*x2 .......... + ar*Nr*xr
		 is a simultaneous solution for this system.

		 Observe: Ni = 0 mod(nk) because nk divides Ni (As Ni = N/ni )

### Euler Theorem

if n > 1 and gcd(a, n) = 1
\begin{equation}
a^{\phi (n)} = 1 \ mod \ n
\end{equation}

### Extended Euclidean Algorithm:

    Extended Euclidean algorithm also finds integer coefficients x and y such that:
    ax + by = gcd(a, b)
    ...

### Other
1. Let p, q be prime and p != q

    if a $\cong$ M mod(p) and a $\cong$ M mod(q)

    then a $\cong$ M mod(p $\times$ q)

2. If p and q are distinct primes with $a^p$ = a mod(p) and $a^q$ = a mod(p)

    then $a^{pq}$ = a (mod pq)

3. The linear congruence ax $\cong$ b (mod n) has a solution if and only if d divides b,

   where d = gcd(a, n). If d divides b, then it has d mutually incongruent solutions modulo n.

4. Wilson Theorem:

    If p is a prime, then (p - 1)! $\cong$ -1 (mod p ).

## PRNG

 ### Linear Congruential Generators
 Link : https://tailcall.net/blog/cracking-randomness-lcgs/

```python
### Implementation

class prng_lcg:
    m = 672257317069504227  # the "multiplier"
    c = 7382843889490547368  # the "increment"
    n = 9223372036854775783  # the "modulus"

    def __init__(self, seed):
        self.state = seed  # the "seed"

    def next(self):
        self.state = (self.state * self.m + self.c) % self.n
        return self.state

def test():
    gen = prng_lcg(123)  # seed = 123
    print (gen.next())  # generate first value
    print (gen.next())  # generate second value
    print (gen.next())  # generate third value

```

```python
# Attacks
## unknown increment

def crack_unknown_increment(states, modulus, multiplier):
    increment = (states[1] - states[0]*multiplier) % modulus
    return modulus, multiplier, increment

## unknown increment and multiplier

def crack_unknown_multiplier(states, modulus):
    multiplier = (states[2] - states[1]) * modinv(states[1] - states[0], modulus) % modulus
    return crack_unknown_increment(states, modulus, multiplier)

## crack_unknown_modulus

def crack_unknown_modulus(states):
    diffs = [s1 - s0 for s0, s1 in zip(states, states[1:])]
    zeroes = [t2*t0 - t1*t1 for t0, t1, t2 in zip(diffs, diffs[1:], diffs[2:])]
    modulus = abs(reduce(gcd, zeroes))
    return crack_unknown_multiplier(states, modulus)
```


# Decomposition Methods

```
%matplotlib inline
```

```
import numpy as np
import numpy.linalg as la
```

In Gaussian elimination we performed row (or column) operations to reduce the linear system

\begin{equation}
  A {\bf x} = {\bf b}
\end{equation}

to a form that was easy to solve, essentially by transforming to an equivalent system where the matrix is triangular.

In *decomposition* methods we instead write the matrix as a product of other matrices in such a way that we have a set of easy-to-solve problems.

For example, suppose we can write

\begin{equation}
  A = L U
\end{equation}

where $L$ is a lower triangular matrix and $U$ upper triangular. In this case, the original linear system is equivalent to the two systems

\begin{equation}
  A {\bf x} = {\bf b} \qquad \Leftrightarrow \qquad \left\{ \begin{aligned} L {\bf y} & = {\bf b}, \\ U {\bf x} & = {\bf y}. \end{aligned} \right.
\end{equation}

As both $L$ and $U$ are triangular, the two systems they define can easily be solved using forwards and backwards substitution respectively. Therefore, *if* we can decompose the matrix $A$ as the product of triangular matrices, the linear system is easily solved.

## Example

Consider the system

\begin{equation}
  \begin{pmatrix} 2 & 1 & -1 \\ 4 & 1 & 0 \\ -2 & -3 & 8 \end{pmatrix} {\bf x} = \begin{pmatrix} 0 \\ 6 \\ 12 \end{pmatrix}.
\end{equation}

We will see later that the matrix can be decomposed as

\begin{equation}
  \begin{pmatrix} 2 & 1 & -1 \\ 4 & 1 & 0 \\ -2 & -3 & 8 \end{pmatrix} = \begin{pmatrix} 1 & 0 & 0 \\ 2 & 1 & 0 \\ -1 & 2 & 1 \end{pmatrix} \begin{pmatrix} 2 & 1 & -1 \\ 0 & -1 & 2 \\ 0 & 0 & 3 \end{pmatrix}.
\end{equation}

We can quickly check that.

```
A = np.array([[2.0, 1.0, -1.0] ,[4.0,  1.0, 0.0], [-2.0, -3.0, 8.0]])
L = np.array([[1.0, 0.0,  0.0] ,[2.0,  1.0, 0.0], [-1.0,  2.0, 1.0]])
U = np.array([[2.0, 1.0, -1.0] ,[0.0, -1.0, 2.0], [ 0.0,  0.0, 3.0]])
np.dot(L, U)
```

    array([[ 2.,  1., -1.],
           [ 4.,  1.,  0.],
           [-2., -3.,  8.]])

Then solve $L {\bf y} = {\bf b}$ by forward substitution:

\begin{equation}
  \begin{pmatrix}
    1 & 0 & 0 \\
    2 & 1 & 0 \\
   -1 & 2 & 1
  \end{pmatrix}
  \begin{pmatrix}
    y_1 \\ y_2 \\ y_3
  \end{pmatrix} =
  \begin{pmatrix}
    0 \\ 6 \\ -12
  \end{pmatrix}
  \Rightarrow \left\{
  \begin{aligned}
    y_1 & = 0, \\
    y_2 & = 6, \\
    y_3 & = -24.
  \end{aligned}
  \right.
\end{equation}

Finally solve $U {\bf x} = {\bf y}$ by backward substitution:

\begin{equation}
  \begin{pmatrix}
    2 & 1 & -1 \\
    0 & -1 & 2 \\
    0 & 0 & 3
  \end{pmatrix}
  \begin{pmatrix}
    x_1 \\ x_2 \\ x_3
  \end{pmatrix} =
  \begin{pmatrix}
    0 \\ 6 \\ -24
  \end{pmatrix}
  \Rightarrow \left\{
  \begin{aligned}
    x_1 & = 7, \\
    x_2 & = -22, \\
    x_3 & = -8.
  \end{aligned}
  \right.
\end{equation}

Again we can cross-check:

```
b = np.array([0.0, 6.0, -12.0])
la.solve(A, b)
```

    array([  7., -22.,  -8.])

## Uniqueness

  The factorisation is not unique: $L$ and $U$ together have $n^2+n$
  free coefficients; $A$ only has $n^2$.  Can freely choose $n$
  coefficients.   Two obvious choices are

1. Doolittle's factorisation: The diagonal entries of $L$ are 1.
2. Crout's factorisation: The diagonal entries of $U$ are 1.

  Then use the explicit matrix multiplication formula, and work from
  the top left corner. Each coefficient obeys

  \begin{equation}
    a_{i j} = \sum_{s=1}^n \ell_{i s} u_{s j} = \sum_{s=1}^{\min(i,j)}
     \ell_{i s} u_{s j}
  \end{equation}

  where the last equality holds because $L$ and $U$ are triangular:
  $\ell_{i s} = 0$ for $s > i$ and $u_{s j} = 0$ for $s > j$.

## Hand factorisation example

  The previous example had the matrix

  \begin{equation}
    A =
    \begin{pmatrix}
      2 & 1 & -1 \\
      4 & 1 & 0 \\
      -2 & -3 & 8
    \end{pmatrix}
  \end{equation}

  which we decompose using

  \begin{equation}
    a_{i j} = \sum_{s=1}^{\min(i, j)} \ell_{i s} u_{s j}.
  \end{equation}

  First coefficient, $i = j = 1$, obeys

  \begin{equation}
    a_{1 1}  = 2 = \ell_{1 1} u_{1 1}.
  \end{equation}

  Use our freedom to set some coefficients to choose $\ell_{1 1} = 1$, for example, which gives $u_{1 1} = a_{1 1} = 2$.

  Now consider the first row of $U$ ($i = 1$, $j$ free) and the first column of $L$ ($i$ free, $j = 1$). The first row of $U$ obeys

  \begin{equation}
    a_{1 j} = \ell_{1 1} u_{1 j} = u_{1 j}
  \end{equation}

  and the first column of $L$ obeys

  \begin{equation}
    a_{i 1} = \ell_{i 1} u_{1 1} = 2 \ell_{i 1}.
  \end{equation}

  Therefore we know that

  \begin{equation}
    L =
    \begin{pmatrix}
      1 & 0 & 0 \\
      \tfrac{4}{2} & ?? & 0 \\
      \tfrac{-2}{2} & ?? & ??
    \end{pmatrix}, \quad
    U =
    \begin{pmatrix}
      2 & 1 & -1 \\
      0 & ?? & ?? \\
      0 & 0 & ??
    \end{pmatrix}.
  \end{equation}

  Go to second row and column.

  \begin{equation}
    a_{2 2} = \ell_{2 1} u_{1 2} + \ell_{2 2} u_{2 2}.
  \end{equation}

  Already computed entries $\ell_{2 1} = 2$, $u_{1 2} = 1$. Again set $\ell_{2 2} = 1$, giving

  \begin{equation}
    a_{2 2} = 1 = 2 + u_{2 2} \Rightarrow u_{2 2} = -1.
  \end{equation}

  As in the previous step, consider the second row of $U$ and
  second column of $L$, finding

  \begin{align}
    u_{2 j}    & = a_{2 j} - \ell_{2 1} u_{1 j}, \\
    \ell_{i 2} & = \left(a_{i 2} - \ell_{i 1} u_{1 2} \right) / u_{2 2},
  \end{align}

  which implies that

  \begin{align}
    u_{2 3}    & = 0 - 2 \times (-1) = 2, \\
    \ell_{3 2} & = \left(-3 - (-1) \times 1\right) / (-1) = 2.
  \end{align}

  We now have the first two rows and columns:

  \begin{equation}
    L =
    \begin{pmatrix}
      1 & 0 & 0 \\
      2 & 1 & 0 \\
      -1 & 2 & ??
    \end{pmatrix}, \quad
    U =
    \begin{pmatrix}
      2 & 1 & -1 \\
      0 & -1 & 2 \\
      0 & 0 & ??
    \end{pmatrix}.
  \end{equation}

  Continuing as above, use last free choice to set $\ell_{3 3} =
  1$. Finally, as above, compute that $u_{3 3} = 3$. So we finally
  have

  \begin{equation}
    L =
    \begin{pmatrix}
      1 & 0 & 0 \\
      2 & 1 & 0 \\
      -1 & 2 & 1
    \end{pmatrix}, \quad
    U =
    \begin{pmatrix}
      2 & 1 & -1 \\
      0 & -1 & 2 \\
      0 & 0 & 3
    \end{pmatrix}.
  \end{equation}

## $LU$ Decomposition Algorithm

* Write down the explicit formula for the matrix coefficients.

  1. Work from the first row / column to the last.
  2. Look at the diagonal entry of $A$: use the freedom to choose the value of either $L$ or $U$'s diagonal entry.
  3. The appropriate row of $U$ and column of $L$ follows from the explicit formula as all other entries are known.

* If either $u_{k k}$ or $\ell_{k k}$ are zero the simple algorithm fails; however, an $LU$ decomposition may still exist.
* $U$ is the matrix that would be found using Gaussian elimination. $LU$ decomposition has the advantage that it holds for any ${\bf b}$ in $A {\bf x} = {\bf b}$, whereas with Gaussian elimination the whole algorithm needs repeating.
* As with Gaussian elimination, pivoting for accuracy is necessary for general matrices.

## Code example

Here is an example LU decomposition algorithm, taken from the solution to the worksheets. Note that, as it doesn't use pivoting, it may well fail.

```
def LU_decomposition(A):
    """Perform LU decomposition using the Doolittle factorisation."""

    L = np.zeros_like(A)
    U = np.zeros_like(A)
    N = np.size(A, 0)

    for k in range(N):
        L[k, k] = 1
        U[k, k] = (A[k, k] - np.dot(L[k, :k], U[:k, k])) / L[k, k]
        for j in range(k+1, N):
            U[k, j] = (A[k, j] - np.dot(L[k, :k], U[:k, j])) / L[k, k]
        for i in range(k+1, N):
            L[i, k] = (A[i, k] - np.dot(L[i, :k], U[:k, k])) / U[k, k]

    return L, U
```

Check this code against the example above.

```
L2, U2 = LU_decomposition(A)
print("Compare the code and hand answers for L:\n{},\n{}\nand for U:\n{},\n{}".format(L2, L, U2, U))
```

    Compare the code and hand answers for L:
    [[ 1.  0.  0.]
     [ 2.  1.  0.]
     [-1.  2.  1.]],
    [[ 1.  0.  0.]
     [ 2.  1.  0.]
     [-1.  2.  1.]]
    and for U:
    [[ 2.  1. -1.]
     [ 0. -1.  2.]
     [ 0.  0.  3.]],
    [[ 2.  1. -1.]
     [ 0. -1.  2.]
     [ 0.  0.  3.]]

## Conditions for factorisation

### Background

  We *assumed* an $LU$ decomposition is always possible. However,
  we then noted that the algorithm works only if both $u_{k k}$ and
  $\ell_{k k}$ are non-zero, which is not known in advance.

  Two key concepts used in determining if a factorisation exists.

1.  The *principal minor* of order $k$ of a matrix $A$ is the (sub-) matrix
    \begin{equation*}
      \begin{pmatrix}
        a_{1 1} & \cdots & a_{1 k} \\
        \vdots & \ddots & \vdots \\
        a_{k 1} & \cdots & a_{k k}
      \end{pmatrix} .
    \end{equation*}

2. A matrix is *strictly diagonally dominant* if
    \begin{equation*}
      | a_{i i} | > \sum_{\substack{j = 1 \\ j \ne i}}^n | a_{i j} |,
        \quad (1 \le i \le n).
    \end{equation*}

### Theorems

  Summarize the two key theorems; see references given in the notes:

  **Theorem 1.1**: If all $n - 1$ leading principal minors of the $n
  \times n$ matrix $A$ are nonsingular, then an $LU$ decomposition of
  $A$ exists.

  The relation between $LU$ decomposition and Gaussian elimination is
  exploited to prove this theorem. It also shows that, if the matrix
  is nonsingular, its rows can be permuted so that an $LU$
  decomposition of the permuted matrix can be found.

  **Theorem 1.2}**: Every strictly diagonally dominant matrix is
  nonsingular and has an $LU$ decomposition.

  Diagonal dominance is a very simple condition to check. Most
  matrices involved in numerical methods for solving PDEs are
  diagonally dominant, leading to the practical utility of this
  theorem.

## Alternative decomposition methods

### Cholesky factorisation - example

We want to decompose the symmetric, positive definite matrix

  \begin{equation}
    A =
    \begin{pmatrix}
      2 & 1 \\
      1 & 3
    \end{pmatrix}
  \end{equation}

  as $A = L L^T$ where $L$ is lower triangular with positive diagonal
  elements. Again we write the coefficients out explicitly and
  look at the diagonal elements first:

  \begin{align}
                && A & = L L^T \\
    \Rightarrow &&
    \begin{pmatrix}
      2 & 1 \\
      1 & 3
    \end{pmatrix}
    & =
    \begin{pmatrix}
      \ell_{1 1}^2 & \ell_{1 1} \ell_{2 1} \\
      \ell_{1 1} \ell_{2 1} & \ell_{2 1}^2 +  \ell_{2 2}^2
    \end{pmatrix} \\
    \Rightarrow && &\left\{
      \begin{aligned}
        \ell_{1 1} & = \sqrt{2} \\
        \ell_{2 1} & = 1 / \sqrt{2} \\
        \ell_{2 2} & = \sqrt{3 - \ell_{2 1}^2} \\
                   & = \sqrt{5/2}
      \end{aligned}\right. .
  \end{align}

Lectures/03 - Decomposition Methods.ipynb


Smallest sphere enclosing a set of points.
===========================

The aim of this tutorial is two-fold

1. Demostrate how to write a conic quadratic model in `Fusion` in a very simple and compact way.
2. Show how and way the dual formulation may solved more efficiently.

Our problem is the defined as:

**Find the smallest sphere that encloses a set of** $k$ **points** $p_i \in \mathbb{R}^n$.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import random

def plot_points(p, p0=[], r0=0.):
    n,k= len(p0), len(p)

    plt.rc('savefig',dpi=120)

    fig, ax = plt.subplots()
    ax.set_aspect('equal')
    ax.plot([ p[i][0] for i in range(k)], [ p[i][1] for i in range(k)], 'b*')

    if len(p0)>0:
        ax.plot(  p0[0],p0[1], 'r.')
        ax.add_patch( mpatches.Circle( p0,  r0 ,  fc="w", ec="r", lw=1.5) )
    plt.grid()
    plt.show()

n = 2
k = 500

p=  [ [random.gauss(0.,10.) for nn in range(n)] for kk in range(k)]

plot_points(p)
```

The problem boils down to determine the sphere center $p_0\in \mathbb{R}^n$ and its radius $r_0\geq 0$, i.e.

\begin{equation}
  \begin{aligned}
\min \max_{i=1,\dots,k} \| p_0 - p_i\|_2 \\
  \end{aligned}
\end{equation}

The maximum in the objective function can be easily, i.e.

\begin{equation}
  \begin{aligned}
\min r_0 & & &\\
s.t.& & &\\
& r_0 \geq \| p_0 - p_i\|_2 ,& \quad & i=1,\ldots,k\\
\end{aligned}
\end{equation}

The SOCP formulation reads

\begin{equation}
  \begin{aligned}
\min r_0 & & &\\
s.t.& & &\\
& \left[r_0,p_0 - p_i\right] \in Q^{(n+1)}, & \quad & i=1,\ldots,k.
\end{aligned}
\end{equation}

Before defining the constraints, we note that we can write

\begin{equation}
R_0 = \left( \begin{array}{c} r_0   \\ \vdots \\ r_0   \end{array} \right) \in \mathbb{R}^k          , \quad
P_0 = \left( \begin{array}{c} p_0^T \\ \vdots \\ p_0^T \end{array} \right) \in \mathbb{R}^{k\times n}, \quad
P   = \left( \begin{array}{c} p_1^T \\ \vdots \\ p_k^T \end{array} \right) \in \mathbb{R}^{k\times n}.
\end{equation}

so that

\begin{equation}
\left[r_0,p_i - p_0\right] \in Q^{(n+1)},  \quad  i=1,\ldots,k.
\end{equation}

can be compactly expressed as

\begin{equation}
\left[ R_0,P_0-P\right] \in \Pi Q^{(n+1)},
\end{equation}

that means, with a little abuse of notation, that each rows belongs to a quadratic cone of dimension $n+1$.

Now we are ready for a compact implementation in `Fusion`:

```python
from mosek.fusion import *
import mosek as msk

def primal_problem(P):

    print(msk.Env.getversion())

    k= len(P)
    if k==0: return -1,[]

    n= len(P[0])

    with Model("minimal sphere enclosing a set of points - primal") as M:

        r0 = M.variable(1    , Domain.greaterThan(0.))
        p0 = M.variable([1,n], Domain.unbounded())

        R0 = Var.repeat(r0,k)
        P0 = Var.repeat(p0,k)

        M.constraint( Expr.hstack( R0, Expr.sub(P0 , P) ), Domain.inQCone())

        M.objective(ObjectiveSense.Minimize, r0)
        M.setLogHandler(open('logp','wt'))

        M.solve()

        return r0.level()[0], p0.level()

```

We will also store the solver output in a file to use it later on. And then just solve the problem.

```python
r0,p0 = primal_problem(p)

print ("r0^* = ", r0)
print ("p0^* = ", p0)
```

    (8, 0, 0, 48)
    ('r0^* = ', 34.277010059679036)
    ('p0^* = ', array([ 0.9775514 ,  3.05181728]))

```python
plot_points(p,p0,r0)
```

Dual Formulation
-----------------------

The dual problem can be determined in few steps following basic rules. Introducing dual variables

\begin{equation}
 Y = \left( \begin{array}{c} y_1^T\\ \vdots \\y_k  \end{array}\right), \quad z = \left( \begin{array}{c} z_1\\ \vdots \\z_k  \end{array}\right),
\end{equation}

the dual is:

\begin{aligned}
    \max & \left\langle P, Y \right\rangle \\
    & e_k^T z = 1\\
    & Y^T e_k = \mathbf{0}_n \\
    & \left[z_i , y_i\right] \in \mathcal{Q}^{n+1}\\
    & z_i\in \mathbb{R}, y_i\in \mathbb{R}^n,
\end{aligned}

where $e_k\in \mathbb{R}^k$ is a vector of all ones.

The ``Fusion`` code is the following:

```python
def dual_problem(P):

    k= len(P)
    if k==0: return -1,[]

    n= len(P[0])

    with Model("minimal sphere enclosing a set of points - dual") as M:

        Y= M.variable([k,n], Domain.unbounded())
        z= M.variable(k    , Domain.unbounded())

        M.constraint(Expr.sum(z), Domain.equalsTo(1.) )

        e= [1.0 for i in range(k)]

        M.constraint(Expr.mul(Y.transpose(), Matrix.ones(k,1)), Domain.equalsTo(0.) )

        M.constraint( Var.hstack(z,Y), Domain.inQCone())

        M.objective( ObjectiveSense.Maximize, Expr.dot( P, Y ))

        M.setLogHandler(open('logd','wt'))

        M.solve()

        return

dual_problem(p)
```

Looking at the solver output for the primal and dual implementation we get

```python
!tail  logp
```

    11  4.7e-06  2.8e-04  1.9e-05  1.01e+00   3.429062907e+01   3.430435587e+01   5.6e-07  0.01
    12  1.3e-06  7.5e-05  1.1e-05  1.00e+00   3.427957004e+01   3.428275209e+01   1.5e-07  0.01
    13  2.2e-07  1.3e-05  4.8e-06  1.00e+00   3.427737473e+01   3.427786148e+01   2.7e-08  0.01
    14  2.1e-08  1.2e-06  1.6e-06  1.00e+00   3.427703453e+01   3.427707456e+01   2.5e-09  0.01
    15  2.8e-09  1.7e-07  5.7e-07  1.00e+00   3.427701143e+01   3.427701671e+01   3.3e-10  0.01
    16  5.3e-10  3.2e-08  2.7e-07  1.00e+00   3.427701016e+01   3.427701104e+01   6.3e-11  0.01
    17  3.1e-11  1.4e-09  4.5e-08  1.00e+00   3.427701006e+01   3.427701009e+01   2.7e-12  0.01
    Interior-point optimizer terminated. Time: 0.01.

    Optimizer terminated. Time: 0.10

```python
!tail logd
```

    11  2.8e-04  4.7e-06  1.9e-05  1.01e+00   3.430435567e+01   3.429062880e+01   5.6e-07  0.01
    12  7.5e-05  1.3e-06  1.1e-05  1.00e+00   3.428275207e+01   3.427956999e+01   1.5e-07  0.01
    13  1.3e-05  2.2e-07  4.8e-06  1.00e+00   3.427786147e+01   3.427737472e+01   2.7e-08  0.01
    14  1.2e-06  2.1e-08  1.6e-06  1.00e+00   3.427707456e+01   3.427703453e+01   2.5e-09  0.01
    15  1.7e-07  2.8e-09  5.7e-07  1.00e+00   3.427701671e+01   3.427701143e+01   3.3e-10  0.01
    16  3.2e-08  5.3e-10  2.7e-07  1.00e+00   3.427701104e+01   3.427701016e+01   6.3e-11  0.01
    17  1.4e-09  2.7e-11  5.8e-08  1.00e+00   3.427701009e+01   3.427701006e+01   2.7e-12  0.01
    Interior-point optimizer terminated. Time: 0.01.

    Optimizer terminated. Time: 0.01

**The solver has got the same result doing exacly the same steps!**

In fact, **MOSEK** has solve the same problem, thanks to the automatic dualizer that decides whether is more convenient to solve the primal or the dual of a given problem. To see that, just look at the output again:

```python
!grep Optimizer logp
```

    Optimizer started.
    Optimizer  - threads                : 2
    Optimizer  - solved problem         : the dual
    Optimizer  - Constraints            : 3
    Optimizer  - Cones                  : 500
    Optimizer  - Scalar variables       : 1501              conic                  : 1500
    Optimizer  - Semi-definite variables: 0                 scalarized             : 0
    Optimizer terminated. Time: 0.10

```python
!grep Optimizer logd
```

    Optimizer started.
    Optimizer  - threads                : 2
    Optimizer  - solved problem         : the primal
    Optimizer  - Constraints            : 3
    Optimizer  - Cones                  : 500
    Optimizer  - Scalar variables       : 1500              conic                  : 1500
    Optimizer  - Semi-definite variables: 0                 scalarized             : 0
    Optimizer terminated. Time: 0.01

In primal problem output the solver reports it is going to solve the `dual`, i.e. our second formulation. In the dual problem output it says it is going to solve the `primal`. Therefore in both cases we actually solve the very same formulation.

This work is licensed under a Creative Commons Attribution 4.0 International License. The **MOSEK** logo and name are trademarks of Mosek ApS. The code is provided as-is. Compatibility with future release of **MOSEK** or the `Fusion API` are not guaranteed. For more information contact our [support](mailto:support@mosek.com).

```python

```


# Physics 420/580 Final Exam
## December 19, 2019 2pm-5pm  L1-150

Do three of the four following problems.  If you attempt all four, your best three will be used for your mark.  Use the Jupyter notebook, inserting your code and any textual answers/explanations in cells between the questions. (Feel free to add additional cells!)  Marks will be given based on how clearly you demonstrate your understanding.

There are no restrictions on downloading from the internet, eclass,  or the use of books, notes, or any other widely available computing resources.  However, **you are not allowed** to communicate with each other or collaborate in any way and uploading to the internet or sending or receiving direct communications is not appropriate.  You are required to sit the exam in L1-150, and to upload it before you leave.

When you are finished, upload the jupyter notebook to eclass.

Also be careful to save the notebook periodically and double check **that you upload the file that has your answers in it!**

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import scipy
from scipy.integrate import odeint
from scipy.integrate import quad
from scipy.optimize import minimize
from scipy.optimize import fsolve
from scipy.optimize import least_squares
from scipy.interpolate import interp1d
```

```python
mpl.rc('figure',dpi=250)
mpl.rc('text',usetex=True)
```

```python
def add_labels(xlabel, ylabel, title):
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.title(title)
    plt.legend()
```

# The Normal Distribution

 Write a simple program that:

 a.  Calculates and plots the function
 $$G(x;\mu,\sigma)=1000e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$ with an average value $\mu=10$, width $\sigma=2$.  This is proportional to the normal distribution.

```python
"""
Plan:
-Here we just graph G, and plot it
"""
```

```python
mu = 10
sigma = 2
def G(x, mu, sigma, A):
    return A*np.exp(-(x - mu)**2/(2*sigma**2))
```

```python
x = np.linspace(-25, 25)
plt.plot(x, G(x, mu, sigma, 1000), label=f"mu = {mu}, sigma = {sigma}, A = {1000}")
add_labels('x', 'G(x)', 'G(x)')

```

The three dimensional normal distribution is the product of normal distributions in the three coordinates:
\begin{equation} G(\vec{x}; \vec{x_0},\sigma, A)=Ae^{-\frac{(\vec{x}-\vec{x_0})^2}{2\sigma^2}}\end{equation}

Set $\vec{x_0}=(6,8,0)$ and throw 100000 random points by throwing the three coordinates independently with $\sigma=2$ for each coordinate.

Histogram and plot the distributions for each coordinate.  Comment on how you pick the right number of bins for your histogram.

```python
"""
Plan:
We could use our normal function G, and sample from it: just plug it into generate samples. However, but we realize that in numpy this is already done for us:
Numpy.random.normal. It's the same thing, just that np.random.normal just much more efficient and I trust it more than myself. It runs faster too.

"""
```

    "\nPlan:\nWe could use our normal function G, and sample from it: just plug it into generate samples. However, but we realize that in numpy this is already done for us: \nNumpy.random.normal. It's the same thing, just that np.random.normal just much more efficient and I trust it more than myself. It runs faster too.\n\n\n"

```
def generate_samples(func, bounds, N):
    """
    Assignment 5: function for sampling from a custom 1d distribution.

    Input:  func - any function
            bounds - tuple (a, b)

    Output: 1d samples of size N
    """
    a, b = bounds
    x = np.linspace(a, b)

    f_max = max(func(x))

    x_rand = np.random.uniform(a, b, size=N)
    y_rand = np.random.uniform(0, f_max, size=N)

    samples = x_rand[y_rand < func(x_rand)]

    return samples
```

```python
x0 = np.array([6, 8, 0])
N = 100000
sigma = 2
```

```python
throws = np.zeros((N, 3))

for i in range(N):
    throws[i, 0] = np.random.normal(x0[0], sigma)
    throws[i, 1] = np.random.normal(x0[1], sigma)
    throws[i, 2] = np.random.normal(x0[2], sigma)

throws.shape

# I picked the bins based on the domain and through just visual comparison. The bins are 100 linearly spaced along the domain.
plt.figure()
plt.hist(throws[:, 0], bins=100, label=f'mu = {x0[0]}, sigma = 2')
add_labels('x', 'counts', 'distribution of x coordinate')
plt.figure()
plt.hist(throws[:, 1], bins=100, label=f'mu = {x0[1]}, sigma = 2')
add_labels('y', 'counts', 'distribution of y coordinate')
plt.figure()
plt.hist(throws[:, 2], bins=100, label=f'mu = {x0[2]}, sigma = 2')
add_labels('z', 'counts', 'distribution of z coordinate')
```

Now histogram and plot the distribution of $r=|\vec{x}-\vec{x_0}|$.

```python
radial = np.linalg.norm(throws - x0, axis = 1)
plt.hist(radial, bins=100)
add_labels('r', 'counts', 'distribution of radial coordinate with respect to mean')
```

This should look like the analytic 3d normal distribution in spherical coordinates:
\begin{equation}G(r;\sigma,A)=4\pi A r^2 e^{-\frac{r^2}{2\sigma^2}}.\end{equation}  We derived this equation by converting $G(\vec{x}; \vec{x_0},\sigma, A)$ into spherical coordinates and integrating over $\theta$ and $\phi.$

Determine A for 100000 points by numerically integrating:
\begin{equation}100000=\int_0^{20} G(r;\sigma,A)dr =A\int_0^{20} G(r;\sigma,1)dr\end{equation}

With a known A, you can now replot your histogram and overlay the calculation of $G(r;\sigma,A)$.  To have $G(r;\sigma,A)$ overlay properly you will need to include the bin widths, which you can do either by the exact integral or the approximation: $\int_{r_i}^{r_{i+1}}G(r;\sigma,A)dr \approx G(r;\sigma,A)\Delta r$, where $r_i$ is the start of the i-th bin and $r_{i+1}$ is the start of the next bin, and $\Delta r=r_{i+1}-r_i$.

```python
def radial_Gaussian(r, sigma):
    return 4*np.pi*r**2*np.exp(-r**2/(2*sigma**2))
```

```python
sigma = 2
integral = quad(radial_Gaussian, 0, 20, args=sigma)

A = 100000/integral[0]
print(f'A = {A} is the normalization factor')
```

    A = 793.6704491780121 is the normalization factor

```python
r = np.linspace(0, 10)
plt.plot(r, A*radial_Gaussian(r, sigma))
```

```python
plt.hist(radial, bins=100)
add_labels('r', 'counts', 'distribution of radial coordinate with respect to mean')
plt.plot(r, A*radial_Gaussian(r, sigma))
```

```python
"""I have to normalize it now, and I have to fudge with some bins but if I have time after the exam I will do that. Right now, I need to solve the other 3 questions"""
```

    'I have to normalize it now, and I have to fudge with some bins but if I have time after the exam I will do that. Right now, I need to solve the other 3 questions'

# Monte Carlo

A radioactive source is placed into a large water Cerenkov detector (which is able to detect high energy electrons/gammas in pure water by looking at visible photons from the Cerenkov process.)  Before the source is deployed, the detector measures background radiation at a rate of 200 events/second.  The source produces two types of events:
1.  single gamma rays with a rate of 100 events/second
2. events that produce both a gamma ray and a neutron at a rate of 100 events/second.  The gamma ray is detected immediately after creation with an efficiency of 50% (for both gamma rays from the source- the singles and the gamma/neutron coincidences).  The neutron is detected after a capture time, which is quantified with a mean of 250 microseconds. In math, this says that the time distribution of neutron captures is $$P(t)=250\times 10^{-6} e^{-t/(250\times 10^{-6})},$$ where t is the time difference (in seconds) between gamma/neutron creation and neutron capture. The neutron detection efficiency is. 40%.    The time distribution of neutron captures is $$P(t)=250\times 10^{-6} e^{-t/(250\times 10^{-6})},$$ where t is the time difference (in seconds) between gamma/neutron creation and neutron capture.

Write a simple MC to model 1000 seconds of running time.   Generate the times for background events, the single gamma rays from the source and the gamma/neutron coincidences.  Then generate the neutron capture times and finally apply the detection efficiencies for each type of event.

Histogram and plot:

1.  The times of each type of event (on an x-axis with a scale of 0-1000 seconds)
2.  The intervals between a detected event and the next detected event (from 0-10 ms).  To do this you might need to make an array with the times of all detections, sort it and then calculate the time from one event to the next.
3.  Repeat plots 1 and 2 for the case where the background rate is 1000 events/second.

```python
"""
Plan:
We need the times of each type of event.
Let's say the source produces events perfectly.
Sample from the uniform distribution then find the interavls
"""
```

    "\nPlan:\nWe need the times of each type of event. \nLet's say the source produces events perfectly. \nSample from the uniform distribution then find the interavls\n"

```python
N = 100000

efficiency_test = np.random.rand(N)
times = np.random.uniform(0, 1000, size=N)

plt.hist(times, bins=50, label='source/detector events')
add_labels('time of event', 'counts', 'times for event 1 (gamma)')
```

```python
intervals = []
times = np.sort(times)
for i in range(len(times) - 1):
    intervals.append(times[i+1] - times[i])

plt.hist(intervals, bins=50)
add_labels('interval between events', 'counts', 'time interval for event 1 (gamma)')
```

```python
N = 100000

efficiency_test = np.random.rand(N)
efficiency_test_2 = np.random.rand(N)
times = np.random.uniform(0, 1000, size=N) #times for event 2.
mask = efficiency_test > 0.5
mask2 = efficiency_test_2 > 0.4

plt.hist(times, bins=50, label='source events',alpha=0.3)
plt.hist(times[mask], bins=50, label='gamma detector events', alpha=0.7)
plt.hist(times[mask2], bins=50, label='neutron detector events', alpha=0.7)
add_labels('time of event', 'counts', 'times for event 2 (gamma+neutron)')
```

```python
meanTime = 250*10**-6

times = np.random.uniform(0, 1000, size=N) #times for event 2.
mask = efficiency_test > 0.5
mask2 = efficiency_test_2 > 0.4

gamma_events = np.sort(times[mask])
neutron_events = np.sort(times[mask2])

detector_events = neutron_events + np.random.exponential(meanTime, size = len(neutron_events))

intervals_gamma = []
intervals_neutron = []
detector_events = np.sort(detector_events)
for i in range(len(detector_events) - 1):
    intervals_neutron.append(detector_events[i+1] - detector_events[i])

for i in range(len(gamma_events) - 1):
    intervals_gamma.append(gamma_events[i+1] - gamma_events[i])

plt.hist(intervals_neutron, bins=50, label='neutron detector events', alpha = 0.7)
plt.hist(intervals_gamma, bins=50, label='gamma detector events', alpha=0.7)
add_labels('interval between events', 'counts', 'intervals between times for event 2 (gamma+neutron)')
```

```python
"""Will repeat for the background rate later. Gotta do the last two questions"""
```

    'Will repeat for the background rate later. Gotta do the last two questions'

```python

```

# Solving and Fitting

An object is viewed by 6 theodolites (telescopes that allow you to measure angles with respect to the vertical($\theta$) and with respect to a horizontal($\phi$).  The locations of the telescopes and the measured angles are given in the code cell below (all with respect to a single coordinate system,) which you can run in order to avoid cutting and pasting.

Since each theodolite gives you two quantities we have 12 measurements.  There are three unknowns- the x, y, and z coordinates of a point on an object.  Consequently the system of equations is overdetermined, and we would typically be unable to solve.

In a case like this, we "solve" the system of equations by fitting.

Write a function that calculates the theta and phi angles (spherical coordinates) given an arbitrary point position and an theodolite position.

Using this function, write a second function that calculates the residuals of the calculation with respect to the data. Then find the position of the target by fitting the calculations to the data.

Extract the uncertainties on the position.  In order to do this, you will need to estimate the size of the uncertainty in the angles to incorporate into the residual calculation.  One way to do this is by histogramming the residuals ((data-calculation)/uncertainty) and getting a sense of how big they are- if the uncertainties are correct the RMS/width of the residuals distribution should be about 1.

```python
import numpy as np
positions =  np.array([[-0.0114519 ,  9.99999344,  6.69130799],
       [ 6.33116335, -7.74056655,  3.59009715],
       [-9.85887257, -1.67410623,  4.56138703],
       [ 4.09466118,  9.12325325,  7.32437587],
       [-3.37815507, -9.41212348,  3.67927878],
       [-5.4274145 , -8.39899827,  6.8865273 ]])
thetaMeasures = np.array( [2.3164460126412245, 1.6313464103926718, 1.6970719353033004, 2.533624050255211, 1.6289243873579955, 1.8159707640506253] )
phiMeasures = np.array( [-0.9844385514247201, 1.8432675248986576, 0.5831709514637226, -2.132723211863188, 1.2215622389804748, 1.0853416750730511] )

print('Theodolite x  \t Theodolite y \t Theodolite z \t Meas.theta \t Meas. phi')
for i in range(0,6):
    print('%f  \t %f \t %f \t %f \t %f '%(positions[i,0],positions[i,1],positions[i,2],thetaMeasures[i],phiMeasures[i]))

```

    Theodolite x  	 Theodolite y 	 Theodolite z 	 Meas.theta 	 Meas. phi
    -0.011452  	 9.999993 	 6.691308 	 2.316446 	 -0.984439
    6.331163  	 -7.740567 	 3.590097 	 1.631346 	 1.843268
    -9.858873  	 -1.674106 	 4.561387 	 1.697072 	 0.583171
    4.094661  	 9.123253 	 7.324376 	 2.533624 	 -2.132723
    -3.378155  	 -9.412123 	 3.679279 	 1.628924 	 1.221562
    -5.427415  	 -8.398998 	 6.886527 	 1.815971 	 1.085342

```python
def xyz_to_spherical(vector):
    r = np.linalg.norm(vector)
    theta = np.arccos(vector[2]/r)
    phi = np.arctan2(vector[1], vector[0])
    return np.array([r, theta , phi])
```

```python
def find_theta_phi(point, telescope):
    difference = point - telescope
    return xyz_to_spherical(difference)

def find_residuals(point):
    theta_res = np.zeros((6))
    phi_res = np.zeros((6))
    for i in range(6):
        angles = find_theta_phi(point, positions[i, :3])
        theta_res[i] = thetaMeasures[i] - angles[1]
        phi_res[i] = phiMeasures[i] - angles[2]

    return np.concatenate([theta_res, phi_res], axis=0)
```

```python
from scipy.optimize import least_squares

params0 = np.array([1, 1, 1])
popt_minimization = least_squares(find_residuals, params0)
popt_minimization

```

     active_mask: array([0., 0., 0.])
            cost: 3.653545714200531e-19
             fun: array([-3.29253513e-10, -9.01634323e-12,  5.41442446e-11,  5.54386315e-10,
           -2.39801290e-10,  1.09330989e-10, -1.75378601e-10,  9.24476051e-11,
            7.91083865e-11,  3.49196672e-10,  6.83249013e-11,  2.65174105e-10])
            grad: array([-9.43717308e-11,  3.27579658e-11,  1.00605068e-11])
             jac: array([[ 0.06381229, -0.09606042,  0.12487658],
           [-0.00110712,  0.0039622 ,  0.06786023],
           [ 0.00711269,  0.00469231,  0.06712066],
           [-0.07762667, -0.12328795,  0.10138615],
           [ 0.00117931,  0.00323844,  0.05922442],
           [ 0.00656647,  0.01244677,  0.05624395],
           [-0.1927295 , -0.12802894,  0.        ],
           [ 0.06559699,  0.01832914,  0.        ],
           [ 0.03755727, -0.05692997,  0.        ],
           [-0.26295946,  0.16556903,  0.        ],
           [ 0.05583779, -0.02033393,  0.        ],
           [ 0.05285997, -0.027887  ,  0.        ]])
         message: '`gtol` termination condition is satisfied.'
            nfev: 6
            njev: 6
      optimality: 9.437173084897083e-11
          status: 1
         success: True
               x: array([2.38, 6.4 , 2.7 ])

```python
uncertainty = 5e-10
plt.hist(find_residuals(popt_minimization.x)/uncertainty)
print(f'the uncertainty is about {uncertainty}')
```

# Ordinary Differential Equations

Consider the mechanics problem shown in the diagram below.  We have a rigid bar $l=4$ meters long, with mass $m=20$ kg, at an angle $\theta$ to the floor and leaning against a wall.  The center of mass of the bar is at coordinates (x,y).  The normal force against the wall we designate as $N_1,$ while the normal force to the floor is $N_2.$  The coefficient of (sliding) friction with the floor is $\mu_2=0.2$, while the coefficient of friction with the wall is $\mu_1=0.4.$

Calculate how long it takes for the bar to fall to the floor.  The acceleration due to gravity is $g.$  Start with an initial condition of $\theta_0= 1.15$ radians, and $\omega_0=\dot{\theta}(0)=0.$

From the mechanics perspective, this involves setting up Newton's second law for the acceleration of the center of mass, and using the torque equation for the rotation of the bar around the center of mass:
\begin{align}\\
-mg+N_2+\mu_1 N_1&=m\ddot{y}\\
N_1-\mu_2 N_2=m\ddot{x}\\
(N_1+\mu_2 N_2)\frac{l}{2} \sin\theta - (N_2-\mu_1 N_1) \frac{l}{2}\cos\theta &=I\ddot{\theta}\\
\end{align}

For a bar, the moment of inertia $I=\frac{1}{12}ml^2.$

The geometry of the situation means that we can't simply plug the equation above into an ODE integrator.  Instead, we need to relate $x$ and $y$ to the angle $\theta:$
\begin{align}\\
x&=\frac{l}{2}\cos\theta\\
\dot{x}&=-\frac{l}{2}\sin\theta \dot{\theta}\\
\ddot{x}&=-\frac{l}{2}\sin\theta \ddot{\theta}-\frac{l}{2}\cos\theta \dot{\theta}^2\\
y&=\frac{l}{2}\sin\theta\\
\dot{y}&=\frac{l}{2}\cos\theta \dot{\theta}\\
\ddot{y}&=\frac{l}{2}\cos\theta \ddot{\theta}-\frac{l}{2}\sin\theta \dot{\theta}^2\\
\end{align}

Now substitute these equations for $\ddot{x}$ and $\ddot{y}$ into the original equation for Newton's laws.
Solve the ODE numerically.  For each step, you will need to solve the set of simultaneous equations for $\ddot{\theta}$, $N_1,$ and $N_2$.

Plot $\theta$ as a function of time $t$.  Graphically find the time the bar hits the floor ($\theta=0.$)
Plot $N_1$ and $N_2$ as a function of time $t$.  Discuss any mathematical assumptions in our model that might not reflect reality and how the model would need to change to accommodate them.

```python
m = 20
l = 4
mu2 = 0.2
mu1 = 0.4
I = (1/12)*m*l**2
g=9.81
```

```python
global N1
global N2
global ddtheta
```

```python
#x = l/2*np.cos(theta)
#y = l/2*np.sin(theta)

```

```python
"""This is a coupled system.
Hence we need to update both the linear newtons law
and the angular newtons law. we know the linear newtons law is 0, -mg.

I can't seem to get the height of the ground

"""
```

```python
N1 = []
N2 = []

def main(y, t):
    "Main thing to integrate"

    """
    y[0] - theta
    y[1] - omega
    y[2:4] - x,y
    y[4:6] - dx, dy
    """

    theta = y[0:2]

    positions = y[2:4]
    velocities = y[4:6]

    ddx = 0
    ddy = -m*g

    #Solving for N. Need ddx, ddy
    A = np.array([[mu1, -m*g],[1, -mu2]])
    b = np.array([m*ddy, m*ddx])
    N = np.linalg.solve(A, b)

    N1.append(N[0])
    N2.append(N[1])

    dydt = np.empty(6)

    dx = -l/2*np.sin(y[0])*y[1]
    dy = l/2*np.sin(y[0])*y[1]

    dydt[0] = y[1] #omega
    dydt[1] = (1/I)*((N[0] + mu2*N[1])*(l/2)*np.sin(y[0]) - (N[1] - mu1*N[0])*(l/2)*np.cos(y[0]))
    dydt[2:4] = np.array([dx, dy])
    dydt[4:6] = np.array([ddx, ddy])

    return dydt
```

```python
t = np.linspace(0, 10, 10000)
y0 = np.array([1.15, 0, l/2*np.cos(1.15), l/2*np.sin(1.15), 0, 0], dtype='float64')

y = odeint(main, y0, t)

"""Grab all physical numbers: the bar cannot fall into the floor"""
mask = (y[:, 3] > 0) & (y[:, 0] > 0)
t = t[mask]
y = y[mask]
```

```python

```

When you are finished, upload your completed exam onto eClass.  Double check that the uploaded exam contains your latest work!

```python

```

```python
plt.plot(t, y[:, 0]/np.pi*180)
add_labels('t','$\\theta$ (degrees)','angle of bar')
```

```python
t = np.linspace(0, 10, len(N1))
plt.plot(t, N1, label="N1")
plt.plot(t, N2, label="N2")
add_labels('time', 'N1 N2', 'N1, N2 as a function of time')
```

## Problems with the Model

N1, N2 are roughly constant which I think is not really correct, to get N1, N2 I had to let ddy = -mg which is gravity on the CoM. Otherwise there is no way to solve the problem due to recursion. We don't have enough variables.

I think that an good simulation should have friction be proportional to the velocity

If we plot the height of the bar, this is also not really correct as well. I think what's causing the problem is probably this assumption that friction is constant, when in reality, it's far from constant. So we know this model is wrong but I think I did the integration correctly.

```python
t = np.linspace(0, 10, 10000)
y0 = np.array([1.15, 0, l/2*np.cos(1.15), l/2*np.sin(1.15), 0, 0], dtype='float64')

y = odeint(main, y0, t)

"""Grab all physical numbers: the bar cannot fall into the floor"""
mask = (y[:, 3] > 0) & (y[:, 0] > 0)
t = t[mask]
y = y[mask]

plt.plot(t, y[:, 3]/np.pi*180)
add_labels('t','y','height of bar')
```

```python

```


```python
import sympy as sym
sym.init_printing()
%matplotlib inline
```

**Ejercicio**

A partir de este circuito encuentre un modelo que relacione el voltaje de salida $y(t)$ con el voltaje de entrada $x(t)$.

**Respuesta**

Desarrolle sus procedimientos

$$ v_{ca}(t) = v_{cb}(t)+v_{rb}(t) $$
$$ x = v_{ra}(t) + v_{cb}(t)+v_{rb}(t) $$

```python
# Celda para código si es necesario
```

Considerando que todas las resistencias son de $1 k\Omega$ y las capacitancias son de $100\mu F$:

¿cómo se comporta $y(t)$ ante una entrada escalón de amplitud 2?

```python
# Celda para código
```

Escriba su análisis

¿cómo se comporta $y(t)$ ante una entrada rampa?

Considerando que $Ra = 1 k\Omega$, $Rb = 0.5 k\Omega$ $Ca = 50\mu F$ y $Cb = 200\mu F$:

¿cómo se comporta $y(t)$ ante una entrada escalón unitario?

```python
# Celda para código
```

Escriba su análisis

Considerando que $Ra = 1 k\Omega$, $Rb = 0.5 k\Omega$ $Ca = 50\mu F$ y $Cb = 200\mu F$:

¿cómo se comporta $y(t)$ ante una entrada escalón unitario?

```python
# Celda para código
```

Escriba su análisis

¿cómo se comporta $y(t)$ ante una entrada rampa?

```python
# Celda para código
```

Escriba su análisis