# Kernels

In some cases the problem is that we are looking for a linear boundary when the boundary in our space is non-linear. However is does not mean that there is no space in which this problem have no linear boundary. If we find the mapping to such space the problem could be solved.

The question is if we can map the features space into more dimensions in such a way to make the problem linearly separable. Is is quite obvious that if we will use the function:
\begin{equation}
K(x) = (x_1^2, \sqrt(2) x_1 x_2, x_2^2)
\end{equation}
then we transfer our problem into $R^3$ space in which it is linearly separable.

Load the data set in the first place

```python
from sklearn.datasets import load_iris
import numpy as np
from sklearn.model_selection import train_test_split
import cvxopt

iris = load_iris()
data_set = iris.data
labels = iris.target

data_set = data_set[labels!=2]
labels = labels[labels!=2]

train_data_set, test_data_set, train_labels, test_labels = train_test_split(
    data_set, labels, test_size=0.2, random_state=15)

train_labels[train_labels<1] = -1
test_labels[test_labels<1] = -1

objects_count = len(train_labels)
```

The ``build_kernel`` function for building more than just the linear kernel needs to be modifier. The RBF kernel can be written as:
\begin{equation}
K=\exp(-\frac{|x-x'|^{2}}{nfeatures*\sigma^{2}})
\end{equation}
nfeatures is number of featurs

```python
def build_kernel(data_set, kernel_type='linear'):
    kernel = np.dot(data_set, data_set.T)
    if kernel_type == 'rbf':
        sigma = 1.0
        objects_count = len(data_set)
        b = np.ones((objects_count, 1))
        diag = np.diag(kernel)*np.ones((1, objects_count))
        kernel -= 0.5 * (np.dot(diag.T, b.T)
                         + np.dot(b, diag))
        kernel = np.exp(kernel / (2. * sigma ** 2))
    return kernel

def build_kernel_old(data_set, kernel_type='linear'):
    kernel = np.dot(data_set, data_set.T)
    if kernel_type == 'rbf':
        sigma = 1.0
        objects_count = len(data_set)
        b = np.ones((len(data_set), 1))
        kernel -= 0.5 * (np.dot((np.diag(kernel)*np.ones((1, objects_count))).T, b.T)
                         + np.dot(b, (np.diag(kernel) * np.ones((1, objects_count))).T.T))
        kernel = np.exp(kernel / (2. * sigma ** 2))
    return kernel
```

The training part is the same as in the case of linear kernel:

```python
def train(train_data_set, train_labels, kernel_type='linear', C=10, threshold=1e-5):
    kernel = build_kernel(train_data_set, kernel_type=kernel_type)

    P = train_labels * train_labels.transpose() * kernel
    q = -np.ones((objects_count, 1))
    G = np.concatenate((np.eye(objects_count), -np.eye(objects_count)))
    h = np.concatenate((C * np.ones((objects_count, 1)), np.zeros((objects_count, 1))))

    A = train_labels.reshape(1, objects_count)
    A = A.astype(float)
    b = 0.0

    sol = cvxopt.solvers.qp(cvxopt.matrix(P), cvxopt.matrix(q), cvxopt.matrix(G), cvxopt.matrix(h), cvxopt.matrix(A), cvxopt.matrix(b))

    lambdas = np.array(sol['x'])

    support_vectors_id = np.where(lambdas > threshold)[0]
    vector_number = len(support_vectors_id)
    support_vectors = train_data_set[support_vectors_id, :]

    lambdas = lambdas[support_vectors_id]
    targets = train_labels[support_vectors_id]

    b = np.sum(targets)
    for n in range(vector_number):
        b -= np.sum(lambdas * targets * np.reshape(kernel[support_vectors_id[n], support_vectors_id], (vector_number, 1)))
    b /= len(lambdas)

    return lambdas, support_vectors, support_vectors_id, b, targets, vector_number
```

The prediction part is a bit different as we need to take RBF kernel into consideration:

```python
def classify_rbf(test_data_set, train_data_set, lambdas, targets, b, vector_number, support_vectors, support_vectors_id):
    kernel = np.dot(test_data_set, support_vectors.T)
    sigma = 1.0
    c = (1. / sigma * np.sum(test_data_set ** 2, axis=1) * np.ones((1, np.shape(test_data_set)[0]))).T
    c = np.dot(c, np.ones((1, np.shape(kernel)[1])))
    sv = (np.diag(np.dot(train_data_set, train_data_set.T))*np.ones((1,len(train_data_set)))).T[support_vectors_id]
    aa = np.dot(sv,np.ones((1,np.shape(kernel)[0]))).T
    kernel = kernel - 0.5 * c - 0.5 * aa
    kernel = np.exp(kernel / (2. * sigma ** 2))

    y = np.zeros((np.shape(test_data_set)[0], 1))
    for j in range(np.shape(test_data_set)[0]):
        for i in range(vector_number):
            y[j] += lambdas[i] * targets[i] * kernel[j, i]
        y[j] += b
    return np.sign(y)
```

The prediction and accuracy is usually higher than the linear kernel.

```python
lambdas, support_vectors, support_vectors_id, b, targets, vector_number = train(train_data_set, train_labels, kernel_type='rbf')
predicted = classify_rbf(test_data_set, train_data_set, lambdas, targets, b, vector_number, support_vectors, support_vectors_id)
predicted = list(predicted.astype(int))

from sklearn.metrics import accuracy_score

print(accuracy_score(predicted, test_labels))
```

         pcost       dcost       gap    pres   dres
     0:  9.6305e+01 -1.2289e+03  2e+03  2e-01  2e-15
     1:  5.9143e+01 -1.2031e+02  2e+02  5e-03  2e-15
     2:  7.0898e+00 -1.6497e+01  2e+01  2e-16  2e-15
     3: -5.2057e-01 -3.7668e+00  3e+00  4e-16  9e-16
     4: -1.1712e+00 -1.8374e+00  7e-01  2e-16  3e-16
     5: -1.3952e+00 -1.6846e+00  3e-01  2e-16  2e-16
     6: -1.4671e+00 -1.5679e+00  1e-01  1e-16  2e-16
     7: -1.5060e+00 -1.5164e+00  1e-02  2e-16  2e-16
     8: -1.5105e+00 -1.5106e+00  1e-04  2e-16  2e-16
     9: -1.5105e+00 -1.5105e+00  1e-06  2e-16  2e-16
    Optimal solution found.
    0.85


# Import Python modules

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
```

# Generate data for Univariate Decision Tree Regression

```python
n = 100 #Number of observations in the training set

trueSplitPoint = 5

trueLowerLevel = 15
trueUpperLevel = 30

```

```python
x = np.random.uniform(0, 10, n)

def generateY(x, trueLowerLevel, trueUpperLevel, trueSplitPoint, sd):

    if x < trueSplitPoint:
        y = np.random.normal(trueLowerLevel, sd, 1)
    else:
        y = np.random.normal(trueUpperLevel, sd, 1)

    return y[0]

y = [generateY(i, trueLowerLevel, trueUpperLevel, trueSplitPoint, 3) for i in x]
```

```python
data = pd.DataFrame({'X':x, 'Y':y}) #Put the two arrays into a dataframe to make it easy to work with
data.head(10) #Inspect the first ten rows of our dataset
```

      X
      Y

      0
      9.701683
      30.927819

      1
      7.618556
      30.736159

      2
      4.572114
      12.845369

      3
      5.142518
      32.198665

      4
      4.966570
      18.501048

      5
      8.102828
      27.678129

      6
      0.371426
      12.520482

      7
      8.664975
      33.794388

      8
      9.785477
      29.627837

      9
      2.432104
      19.029805

# Quickly plot the data so we know what it looks like

```python
plt.scatter(data['X'], data['Y'])
plt.show()
```

Data appears to be generated by two distinct processes (as indeed it is).

For the simplest version of decision tree we want to identify the split point. When predicting the (continuous) label for a new observation, if the feature is less than the split point we assign one value and if it's greater than the split point we assign another value. Thus the set of possible values we can assign has size two. Clearly this is a very simple regression technique and it's unlikely we'd use this in a real scenario, but we've generated the data in a very contrived way so that it should perform reasonably here and we'll see that the decision tree can approximate non-linear functions by using techniques such as Bagging.

Mathematically, we want to learn a function $f: \mathbb{R} \rightarrow \mathbb{R}$, such that:

\begin{equation}
  f(x)=\begin{cases}
    c_0, & \text{if $x<c_{split}$}.\\
    c_1, & \text{otherwise}.
  \end{cases}
\end{equation}

Our aim is to find the best combination of values for $c_0, c_1 \& c_{split}$

# How we're going to fit the model

We can't use a Maximum-Likelihood style approach because we don't make any probabilistic assumptions. We're going to test lots of split points, then for each split point we're calculate $c_0$ and $c_1$ by taking all of the points on the relevant side of the split point and calculating the mean of those points.

That is

$$c_0|c_{split} = \frac{\sum_{x_i < c_{split}} x_i}{\sum_{x_i < c_{split}} 1}$$

$$c_1|c_{split} = \frac{\sum_{x_i > c_{split}} x_i}{\sum_{x_i > c_{split}} 1}$$

And then we'll choose the values of $c_0, c_1 \text{and} c_{split}$ which minimise the squared error on the training set

```python
class univariateDecisionTree:

    def __init__(self, data, target, feature, trainTestRatio = 0.9):
        #data - a pandas dataset
        #target - the name of the pandas column which contains the true labels
        #feature - the name of the pandas column which we will use to do the regression
        #trainTestRatio - the proportion of the entire dataset which we'll use for training
                    #   - the rest will be used for testing

        self.target = target
        self.feature = feature

        #Split up data into a training and testing set
        self.train, self.test = train_test_split(data, test_size=1-trainTestRatio)

        #Generate mesh of split points to try. We want the mesh to be of a decent size (say 1000 points)
        #Think about what reasonable end points of the mesh would be - what would happen if we considered
        #a point that was smaller than the smallest point in the training set as a split point?

        #np.linspace() is a good function for constructing this 1-d mesh
        meshMin = np.min(self.train[self.feature])
        meshMax = np.max(self.train[self.feature])
        self.splitPointMesh = np.linspace(meshMin, meshMax, 1000)

    def computeMeansGivenSplitPoint(self, splitPoint):
        #Given a split point, we want to split the training set in two
        #One containing all the points below the split point and one containing all the points above the split point
        #The means are then the means of the targets in those datasets and they are the values we want to return

        belowSplitPoint = self.train.loc[self.train[self.feature] < splitPoint][self.target].mean()
        aboveSplitPoint = self.train.loc[self.train[self.feature] >= splitPoint][self.target].mean()

        return belowSplitPoint, aboveSplitPoint

    def computeSquaredError(self, splitPoint):
        #Once we have a split point and a set of means, we need to have some way of identifying whether it's
        #a good split point or not

        #First apply compuuteMeansGivenSplitPoint to get the values for above and below the dataset
        #Then compute the sum of squared errors yielded by assigning the corresponding mean to each point in the training set
        #If we add these two sums of squares together then we have a single error number which indicates how good our split point is

        c0, c1 = self.computeMeansGivenSplitPoint(splitPoint)

        #To get the value of errorBelow, subset the training set to the points below the split points
        #Then calculate the squared difference between target and c0 for each observation in the subset
        #Then sum them up (This can all be done in one line)

        errorBelow = np.sum((self.train.loc[self.train[self.feature] < splitPoint][self.target] - c0)**2)

        #errorAbove works in the same way
        errorAbove = np.sum((self.train.loc[self.train[self.feature] >= splitPoint][self.target] - c1)**2)

        totalError = errorBelow + errorAbove

        return totalError

    def fitDT(self):
        #Calculate the squared error for each split point in the mesh,
        #and save the values (splitPoint and two means) obtained by the best one

        #Errors for each member of the mesh
        errors = [self.computeSquaredError(sp) for sp in self.splitPointMesh]

        #Find the member of the mesh which gives the smallest error
        bestSplitPoint = self.splitPointMesh[np.argmin(errors)]

        #Save as attributes of the class the split point and two means which give the smallest error
        self.c0, self.c1 = self.computeMeansGivenSplitPoint(bestSplitPoint)
        self.cSplit = bestSplitPoint

        return 0

    def predict(self, X):

        X = np.array(X).reshape((-1,1))
        return [self.c0 if x < self.cSplit else self.c1 for x in X]

```

```python
myModel = univariateDecisionTree(data, 'Y', 'X')
```

```python
myModel.fitDT()
```

    0

# Now lets see what the results look like

We'll plot the regression line we obtained against the values we observed in the training set

```python
regLine = myModel.predict(myModel.splitPointMesh)
```

```python
plt.scatter(myModel.train[myModel.feature], myModel.train[myModel.target], label = 'Training set')
plt.scatter(myModel.splitPointMesh, regLine, label = 'Regression Line')
plt.legend()
plt.show()

```

```python

```

Implementations/DecisionTree/.ipynb_checkpoints/Decision Tree - Regression - Single Variable - Complete-checkpoint.ipynb


# Lecture 11:  Scalar and Vector Fields, Accumulation

## Reading and Reference
* Essential Mathematical Methods for Physicists, H. Weber and G. Arfken, Academic Press, 2003

## What to Learn?

* Define a scalar and vector field
* Define and use the gradient and divergence operators
* Develop a geometric intuition for the divergence and gradient
* Learn or review the definition of the chemical potential and ideal solution models
* The assumptions required to write the diffusion equation in a simplified form

## What to do?

* Compute the gradient of a scalar field and the divergence of a vector field
* Develop visualization tools
* Derive the accumulation formula from the flux into/out of a control volume
* Derive Fick's first law starting from the chemical potential

### Scalar Fields
----

Engineers and scientists are typically interested in how certain quantities vary within a domain.  A scalar field is a mapping of values to positions within some coordinate system.  In physical problems it is common to map values to three dimensions or fewer.  For example, scalar fields may be generated from temperature measurements within a chemical reactor, pressure data over the Earth's surface, or x-ray intensity from telescope observations.  Any mapping of a numerical quantity to a spatial coordinate is a scalar field.

Materials engineers interested in phase transformations typically deal with scalar fields corresponding to concentration as a function of position.  This could be indicated using the symbols $c(x,y)$ where the $c$ corresponds to the field of interest and the $(x,y)$ indicating the dimensions of and position within the field.

$$
f(x,y) = \sin(2x) \cdot \sin(y)
$$

Note that this function takes an $(x,y)$ position as input and returns a single scalar value.

```python
%matplotlib notebook

import sympy as sp
from sympy.plotting import plot3d
import numpy as np
import matplotlib.pyplot as plt
x, y = sp.symbols('x y')
```

```python
# Plot our scalar function over a specified range.
plot3d(sp.sin(2*x)*sp.sin(y), (x, -5, 5), (y, -5, 5));
```

We use `plot3d()` to visualize this field so that we use the height of the surface as a proxy for the magnitude of the scalar field.

```python
# Here is another example using a more complex `sympy` expression:

plot3d(sp.series(sp.sin(2*x)*sp.sin(y), x, x0=0, n=7).removeO(), \
       (x, -5, 5), (y, -5, 5));
```

### Projections Onto Two Dimensions
----

* A field $f(x,y)$ contains three pieces of information:  f, x, y
* Colors and other glyphs can help access the additional information
* Contours and heatmaps are two such methods

An alternative to the three dimensional plot is to project the scalar values onto a two dimensional surface.  Two common options for this are _contour plots_ and _heat maps_.  Rather than using a third dimension to represent the scalar values a contour plot traces single-valued lines through the domain whereas a heat-map uses colors to represent the value of the scalar field.

### Visualizing the Scalar Field by Contours
----

As previously stated, _iso-lines_ are plotted within the domain and the color, annotations and position of the lines quantify the scalar field.  Topographic maps use this representation to help the reader understand the locations and incline of mountains and valleys.  `matplotlib` has a `.contour()` method that will generate contour plots.  We demonstrate this next.

```python
delta = 0.025
x = np.arange(-3.0, 3.0, delta)
y = np.arange(-2.0, 2.0, delta)
X, Y = np.meshgrid(x, y)
Z = np.sin(2*X)*np.sin(Y)
```

```python
contours = 10
plt.figure()
CS = plt.contour(X, Y, Z, contours)
plt.clabel(CS, inline=1, fontsize=10)
plt.title(r'A Simple Contour Plot of Your Scalar Field')
plt.show()
```

### Visualizing the Scalar Field by Value
----

Color choice requires consideration of the medium in which you distribute your data as well as the capabilities of the reader.  An [essay](http://blogs.nature.com/onyourwavelength/2018/07/18/heuristics-for-better-figures/) has been written on the topic and provides some guidance for preparation of figures.  `matplotlib` has a module called `cm` that provides some standard color maps.  User specified maps are possible.

We use our same scalar field and the `.meshgrid()` method to create an array $Z$ that holds our field data.  These data are passed to `imshow` to display the data as an image on the plot axes.  The relationship between the field coordinates and the data is lost when plotted this way so we must specify that directly using the `extent` keyword argument.

```python
print(ax)
```

```python
from matplotlib import cm

delta = 0.025
x = np.arange(-3.0, 3.0, delta)
y = np.arange(-2.0, 2.0, delta)
X, Y = np.meshgrid(x, y)
Z = np.sin(2*X)*np.sin(Y)

fig, ax = plt.subplots()
cax = ax.imshow(Z, interpolation='nearest', cmap=cm.coolwarm, extent=(-3,3,-2,2))
# Empirical hack to make colorbar not ugly.
fig.colorbar(cax, fraction=0.032, pad=0.04)
ax.set_title('Scalar Field')
plt.show()
```

### DIY:  Using Surface and Contour Plots
----

Plot the electric field around a point charge if the potential is given by:

$$
V(x,y) = \frac{1}{x^2+y^2}
$$

Use three dimensional surface plots and contour plots to help visualize this potential.

```python
# Your code here.
```

### Gradients
----

Scalar fields have associated vector fields.  One such vector field is known as the _gradient_ and is indicated with the symbol $\nabla$ and is called [nabla](https://en.wikipedia.org/wiki/Nabla_symbol).  The gradient is a type of directional derivative and is a vector quantity.  In a physical problem of three dimensions the gradient can be thought of as pointing in the direction of the maximum spatial rate of change.  Each basis vector magnitude is multiplied by the partial derivative of the field with respect to the basis vector's coordinate.

You can visualize the gradient operator as a vector with components:

$$\overrightarrow{\nabla} = \frac{\partial}{\partial x} \hat{i}  + \frac{\partial}{\partial y} \hat{j} + \frac{\partial}{\partial z} \hat{k}  $$

and then proceed to perform multiplication as per vector algebra using the dot product.

There are multiple ways to interact with Python and get at the gradient of a function.  In the first instance we can use the coordinate system capabilities of `sympy` so that we can access the built in method `.gradient()`.  We start by defining a coordinate system and then calling gradient on our scalar function.  Scalars and vectors are objects of the coordinate system.  [See this page](http://docs.sympy.org/latest/modules/vector/intro.html) for more information on the vector module.

For the purposes of numeric computing, `NumPy` has functions and operators that work as you would expect for vector operations.  What is presented below is for convenience, only.

```python
import sympy as sp
import sympy.vector as spv

C = spv.CoordSys3D('C')
spv.gradient(sp.sin(2*C.x)*sp.sin(C.y))
# The gradient function should return something that looks like u+v
# where u is a vector in the x direction and v is a vector in the
# y direction.
```

### Quiver Plots

One way to visualize a vector field in a physical problem is to use small oriented arrow to indicate direction and length to indicate magnitude.  These are known as quiver plots and they can be used in conjunction with heat maps to help visualize vector fields.  In the example below the scalar field is sampled with two different point densities.  A higher point density (100 points) for the filled contour plot and a lower point density (30 points) for the quivers.

```python
x0, x1 = (-3,3)
y0, y1 = (-2,2)

# Read the docstrings to understand why the numbers are given
# as complex quantities.  use: ?np.mgrid
Y, X =  np.mgrid[y0:y1:100j, x0:x1:100j]
Y1, X1 =  np.mgrid[y0:y1:30j, x0:x1:30j]

Z = np.sin(2*X)*np.sin(Y)

# u and v here are the results of applying the gradient operation
# to our scalar field.  Probably wise to check this in a seperate
# code block.
u = (2*np.sin(Y1)*np.cos(2*X1))
v = (np.sin(2*X1)*np.cos(Y1))
```

```python
fig, ax = plt.subplots()
filled_contour = plt.contourf(X,Y,Z,5)
fig.colorbar(filled_contour, ax=ax)
plt.quiver(X1,Y1,u,v, color='white')
plt.show()
```

### Divergence
----

If we interpret the gradient field as a measurement of the flow of some quantity through space, then the divergence of that vector field is a measurement of the accumulation of that quantity.  To compute the divergence we compute the dot product of _nabla_ and the vector field.  The resulting quantity is a scalar quantity:

$$
\overrightarrow{\nabla} \cdot \overrightarrow{F} = \frac{\partial F_x}{\partial x} + \frac{\partial F_y}{\partial y} + \frac{\partial F_z}{\partial z}
$$

```python
gradientOfOurFunction = spv.gradient(sp.sin(2*C.x)*sp.sin(C.y))
gradientOfOurFunction
```

```python
spv.divergence(gradientOfOurFunction)
```

### Accumulation and Diffusion
----

One of the key problems in science and engineering is the diffusion of an extensive quantity such as energy, charge, or mass.  The gradient of a potential provides a driving force for this diffusion.

Using the diffusion of mass as an example:

$$\overrightarrow{J} = -M \overrightarrow{\nabla} \cdot {\mu} $$

and then computing the accumulation based on the vector field:

$$\frac{\partial C(x,t)}{\partial t} = - \overrightarrow{\nabla} \cdot \overrightarrow{J} $$

where the minus sign indicates that accumulation occurs antiparallel to the gradient.

```python
# Define your example scalar field (a concentration like
# variable C(x,y).
exampleField = sp.sin(sp.pi*C.x)*sp.cos(sp.pi*C.y)
exampleField
```

We then use the built in `sympy` function `gradient` to compute the gradient:

```python
# Compute the gradient.
gradientOfField = -spv.gradient(exampleField)
gradientOfField
```

We then compute the divergence of the gradient.  Note the absence of `C.i` and `C.j` in the answer indicating that these are not components of a vector.  (Compare this to the last slide.)

```python
# Compute the divergence.
accumulation = -spv.divergence(gradientOfField)
accumulation
```

### Plotting the Flux (or Accumulation)
----

In this section we drop the vector descriptions as provided by `sympy` and we keep track of the vector components by hand using the mathematical definitions.  Writing this out we have a scalar field:

$$
F(x,y)
$$

the flux written as minus the gradient of the field (with $M=1$):

$$
\overrightarrow{J} = -\nabla F(x,y) = -\left(\frac{\partial F}{\partial x} \hat{i} + \frac{\partial F}{\partial y}\hat{j} \right)
$$

and the accumulation as minus the divergence of the flux:

$$
A = -\nabla \cdot \overrightarrow{J} = \left(\frac{\partial^2 F}{\partial x^2} + \frac{\partial^2 F}{\partial y^2}\right)
$$

```python
x, y = sp.symbols('x y')

concentrationFunction = sp.sin(sp.pi*x)*sp.cos(sp.pi*y)
fluxX = -sp.diff(concentrationFunction,x)
fluxY = -sp.diff(concentrationFunction,y)
accumulationFunction = sp.diff(concentrationFunction,x,2) + sp.diff(concentrationFunction,y,2)

# We use lambdify to permit the functions to take arguments and vectorize
# the computations.
myConcentrationFunction = sp.lambdify((x,y), concentrationFunction, 'numpy')
myFluxX = sp.lambdify((x,y), fluxX, 'numpy')
myFluxY = sp.lambdify((x,y), fluxY, 'numpy')
myAccumulationFunction = sp.lambdify((x,y), accumulationFunction, 'numpy')
```

We use our quiver plotting capability to plot:

* the scalar value for the concentration (or accumulation) as `z`
* the $\hat{i}$ component of the flux as `u`
* the $\hat{j}$ component of the flux as `v`

```python
import numpy as np

x0, x1 = (-1,1)
y0, y1 = (-1,1)
plotResolution = 200

Y, X =  np.mgrid[y0:y1:200j, x0:x1:200j]
# Quivers are on a seperate grid since they clutter things up.
Y1, X1 =  np.mgrid[y0:y1:20j, x0:x1:20j]

Z = myConcentrationFunction(X,Y)
#Z = myAccumulationFunction(X,Y)
u = myFluxX(X1,Y1)
v = myFluxY(X1,Y1)
```

```python
fig, ax = plt.subplots()

plt.contourf(X,Y,Z,20, cmap=cm.coolwarm)
plt.colorbar()
plt.quiver(X1,Y1,u,v, color='white')

plt.show()
```

In this example we use the contour plot to show locations of high (red) and low (blue) concentrations.  From this concentration field we compute the flux and show that as a quiver plot overlaid on the contour plot.  Examination of this figure confirms our physical understanding that mass flows from high to low concentration areas when the chemical potential is proportional to the concentration of species.

## DIY:  Using the ideal solution model for the Gibbs free energy, *derive* the accumulation of species in a diffusive problem.

The flux is the divergence of the chemical potential.  The chemical potential is the derivative of the Gibbs free energy with respect to composition.  Review the chain rule before you start this problem.  The ideal solution model in terms of mole fraction for a two component system is:

$$
G(X) = X_AG_A + X_BG_B + RT(X \ln X + (1-X) \ln (1-X))
$$

Compute the divergence of the gradient of this function.  Note that the concentration is a function of spatial coordinates.  In this problem assume that the mole fraction $X$ is only a function of $x$, i.e. $X(x)$.  $G_A$ and $G_B$ are constants that depend on the melting temperature and heat of fusion, $R$ is the gas constant and $T$ is the absolute temperature.

This is easiest if you use SymPy and ignore the coordinate system formalities above.

## DIY:  Read [this](http://blogs.nature.com/onyourwavelength/2018/07/18/heuristics-for-better-figures/) article and write a 200 word essay on choosing colors for heat maps.


Most of the material in this notebook is from the following textbook: [Pattern Recognition and Machine Learning-ch-12](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf)

### Mathematical Discussion of PCA - Content

1. [Notation](#notation)

2. [What is PCA?](#what_is_pca)

3. [The idea behind PCA](#the_idea_behind_pca)

4. [The mathematics of PCA](#mathematics_of_pca)

5. [Solving maximization problem](#solving_maximization_problem)

5. [Conclusion](#conclusion)

### Notation [↩](#content)
[](#notation)

- A vector in $\mathbb{R}^{n}$ is written as $\vec{u} = (u_1, \cdots, u_{n})$. So when we think a vector as an $n\times1$ matrix, we represent it as a column vector:

$$ \vec{u} = \begin{bmatrix}
    u_1 \\
    u_2 \\
    \vdots\\
    u_n
\end{bmatrix}_{ n\times1}$$

with this notation the transpose of a vector is a $1\times n$ matrix:

$$
\vec{u}^{\top} = \begin{bmatrix}
    u_1 & u_2 & \cdots & u_n
\end{bmatrix}_{ 1\times n}
$$

Note that for simplicity, we will omit '$\to$' symbol whenever the computations are complicated.
- For the dot-product we can use any of the following notations:

 - If we think of vectors abstractly: $u \cdot v = v \cdot u = u_1 v_1 + \cdots + u_n v_u$

 - If we consider vectors as $n\times 1$ matrices:

 $$
 u^{\top}\cdot v = \begin{bmatrix}
    u_1 & u_2 & \cdots & u_n
\end{bmatrix}\begin{bmatrix}
    v_1 \\
    v_2 \\
    \vdots\\
    v_n
\end{bmatrix} = v^{\top}\cdot u =  \begin{bmatrix}
    v_1 & v_2 & \cdots & v_n
\end{bmatrix}\begin{bmatrix}
    u_1 \\
    u_2 \\
    \vdots\\
    u_n
    \end{bmatrix}
     = u_1 v_1 + \cdots + u_n v_n
 $$
 - Note that the dot product of two vector coincides with the matrix multiplication of them when we consider these vector as $n\times 1$-matrices.

### What is PCA? [↩](#content)
[](#what_is_pca)

PCA is an algorithm that allows us to reduce the number of features by finding special directions in the dataset that explains the variation more efficiently. In this notebook we would like to understand why this is useful and how this algorithm works mathematically.

Suppose we would like to work with a dataset of dimension $N\times D$. This means we have N-observations and D-features. Note that such dataset can be represented as an $N$ by $D$ matrix $X$. Our goal is to find a direction (a vector of dimension $N$) such that the projection of the dataset $X$ onto the line through this direction will have maximum variance.

__Note 1__ Even though this algorithm is very similar to the least square method here we don't have any 'target' variable.

__Note 2__ This algorithm can be applied to the dataset iteratively to explain more variance in the dataset and then we can stop after we reach a certain criteria or a number of dimensions.

__Note 3__ Even though this is a dimension reduction algorithm, PCA is not a feature extraction technique. We will see that each 'special' direction will be expressed as the linear combinations of the all initial features. So at the end we will end up with less features but to create them we need all of the initial features.

__Note 4__ If you have linear algebra background, PCA is basically writing the dataset matrix in a new basis formed by the eigenvectors of the covariance matrix of $X$.

### The idea behind PCA - a very simple case: Dimension reduction from 2 --> 1[↩](#content)
[](#the_idea_behind_pca)

To understand the essence of PCA algorithm and dive into mathematics of it, let's consider the following simple case below. The data is two dimensional and each observation is marked with a pink dot. Here the green line is called __the first principal direction__ and the projections of the datapoints on this line is given by the dashed lines. Note that, we choose the green line among all other lines passing through the origin so that if we would choose another line variance of the projections on this line would be smaller.

After we find the first principal direction, we will choose the second principal direction as a direction which is perpendicular to the first and explains most of the remaining variance. Here since there are only two dimensions the other direction is determined immediately after the first direction. After we find principal directions, we will use these directions as 'axis' and the projections of the datapoints as the 'new' components of the dataset.

([Image source](http://www-bcf.usc.edu/~gareth/ISL/))

Now, if we want to reduce the dimension from two to one then we can drop the second principal direction and just work with the projection of data onto the first principal direction. PCA guarantees that this is the best choice in terms of capturing the variance in the dataset.

As we can see, this algorithm implicitly assumes that the information(signal) comes from variance of the data and small variance contains a little information hence can be considered as noise. With this motivation, this algorithm iteratively finds the directions that captures the most of the variance and also records how much variance is left after the projections. This allow us to reduce dimension with a control over the loss of information.

## Mathematics of PCA:[↩](#content)
[](#mathematics_of_pca)

### Maximum Variance Formulation

There are two different formulation that give us the algorithm of PCA. One of them is the one that we mentioned above and the other one is to minimum error formulation. We will first look at the mathematics behind the maximum variance formulation.

Suppose we are given a dataset with $N$-observations(rows) and $D$-features(columns). We will denote this dataset with $X$ and mathematicall $X$ is a matrix of shape $N \times D$. In this setting, each observation is represented by a $D$ dimensional vector which we will denote by $x_{i} = (x_{i1}, x_{i2}, \cdots x_{iD})$. Note that we have $N$ such vectors.

$$ X =
\begin{bmatrix}
    ----    &  x_{1}^{\top} &  ----   \\
    \vdots    &  \vdots    &  \vdots    \\
    ----    &  x_{i}^{\top} &  ----      \\
     \vdots    &  \vdots    &  \vdots     \\
     ----    &  x_{N}^{\top} &  ----
\end{bmatrix}_{N\times D}
$$

Before finding a direction that maximizes the variance of the projections, let's first understand how we will find the variance of projections of $x_i$ onto a given direction $v = (v_1, \cdots, v_{D})$. Recall that mathematically a direction is represented by a vector of unit lenght. So we assume that $\lVert v \rVert = 1$.

Again if we recall the scalar projection of a vector $\vec{a}$ onto a unit vector $\vec{b}$ is given by the dot product. More explicitly: $ \text{proj}_{b}a = \dfrac{a.b}{\lVert b \rVert}$

([Image sourse](https://mathinsight.org/dot_product))

If we look at the projections of each row-vectors $x_i$ onto a given vector $v$, we will get $N$-numbers each given by the scalar projection formula.

$$ \begin{align}
p_i &= v^{\top} \cdot x_{i} &  &\text{(projection of each row onto the vector)}\\
\overline{p} &= \frac{1}{N} \sum\limits_{i=1}^{N} p_{i}\\
             &= \frac{1}{N} \sum\limits_{i=1}^{N} v^{\top} \cdot x_{i} & &\text{(the mean of the projections)}\\
             &= v^{\top}\cdot\frac{1}{N} \sum\limits_{i=1}^{N} x_{i}               &     &\text{(recall that $ \vec{a}\cdot\vec{b}+\vec{a}\cdot\vec{c} = \vec{a}\cdot(\vec{b}+\vec{c}))$}
\end{align}
$$

Note that in the last equation we get the following vector $\frac{1}{N} \sum\limits_{i=1}^{N} x_{i}$ . This is summation over each row vector in $X$ and then dividing the corresponding vector by the number of rows. In other words, this is a vector consists of the means of the columns of $X$. We will denote this vector as $\overline{x}$. Then:

$$
\overline{p} = v^{\top} \cdot \overline{x}
$$

So the average of the projections of the rows $x_i$ onto an arbitrary vector $v$ is given by the dot product of this vector with the mean vector of $X$. Note that for the variance we need to subtract each projection from this and then sum their squares over all of the rows of $X$. Let's write this mathematically:

$$
\begin{align}
\sigma^{2} &= \frac{1}{N}\sum\limits_{i=1}^{N}  \left( \overline{p} - p_{i}\right)^{2} \\
       &= \frac{1}{N}\sum\limits_{i=1}^{N} \left( v^{\top}\cdot \overline{x} - v^{\top}\cdot x_{i}\right)^{2}\\
       &= \frac{1}{N}\sum\limits_{i=1}^{N} \left( v^{\top}\cdot\left( \overline{x} -  x_{i} \right) \right)^{2}
\end{align}
$$

Let's focus on the last equation again. Note that the dot product of two vectors is a number and it's square is good old second power in real numbers. However, we can write this as:

$$
\begin{align}
\left(a^{\top}\cdot b\right)^{2} &= \left(a^{\top}\cdot b\right)\left(a^{\top}\cdot b\right) \\
                                 &= \left(a^{\top}\cdot b\right)\left(b^{\top} \cdot a \right) && \text{with the trick $a^{\top}\cdot b = b^{\top} \cdot a$} \\
                                 &= a^{\top} \cdot \left( b\cdot b^{\top} \right) \cdot a \\
              \end{align}
$$

Note that the last equation is a little bit trickier. We used the fact that for vector the dot product and the matrix products are the same and the matrix product is associative (A.(B.C) = (A.B).C) ie. the order of multiplication doesn't matter. Note that this is not the case for the dot product.

Therefore and finally we get a nice expression for the variance of projections:

$$
\begin{align}
\sigma^{2}&= \frac{1}{N} \sum\limits_{i=1}^{N} \left( v^{\top}\cdot\left( \overline{x} -  x_{i} \right) \right)^{2}\\
          &= \frac{1}{N}\sum\limits_{i=1}^{N} v^{\top} \left( \overline{x} -  x_{i} \right)\left( \overline{x} -  x_{i} \right)^{\top} v
\end{align}
$$

Now one question is begging for an answer: Why should we do all this abracadabra. The answer is the following. In the last equation, the middle term would give us a familiar matrix: $ \frac{1}{N}\left( \overline{x} -  x_{i} \right)\left( \overline{x} -  x_{i} \right)^{\top} = \Sigma$ where $\Sigma $ denotes the covariance matrix of $X$. After this long journey, let's summarize what we get: for a given data matrix $X$ and a unit vector $v$ the variance of the projection of $X$ onto $v$ is given by:

$$
\sigma_{v}^{2} = v^{\top} \Sigma v
$$

where $\Sigma = \frac{1}{N}\left( \overline{x} -  x_{i} \right)\left( \overline{x} -  x_{i} \right)^{\top} $ the covarince matrix of the data matrix $X$.

[More on Covariance Matrix](https://datascienceplus.com/understanding-the-covariance-matrix/)

### Solving maximization problem [↩](#content)

[](#solving_maximization_problem)

Note that the formula above depends on the vector $v$. Our goal in PCA is to find the unit vector $u$ so that $\sigma_{u}^{2}$ is maximum compared to all $D$-dimensional unit vectors . In more technical terms, we would like to find the maximum of a function (let's call it $f(v) = \sigma_{v}^{2}$) with the constraint $\lVert v \rVert =1$. Since this is a maximization problem with a constraint we should use the Lagrange multiplier method [1](#f1)[2](#f2)[3](#f3).  According to this method, the maxima of the function $f(v) = \sigma_{v}^{2} = v^{\top} \Sigma v$ with constraint $\lVert v \rVert^{2} =1$ will be a critical point of the function $\mathcal{L}(v, \lambda) = f(v) - \lambda(\lVert v \rVert^{2}-1)$. Then we have to find the gradient $\nabla\mathcal{L} = (\frac{\partial\mathcal{L}}{\partial v}, \frac{\partial\mathcal{L}}{\partial \lambda})$. The partial derivative $\frac{\partial\mathcal{L}}{\partial \lambda}$ simply gives us the constraint condition. So only interesting part comes from $\frac{\partial\mathcal{L}}{\partial v}$. In our case we have the following:

$$
\begin{align}
\frac{\partial\mathcal{L}}{\partial v} = 0 &\iff& 2\Sigma v - 2\lambda v = 0 &&\text{(A short-hand notation for the partial derivatives)}\\
& \iff & \Sigma v = \lambda v &&\text{(Note that this means v is an eigenvector of $\Sigma$)}
\end{align}
$$

So note that any eigen-vector of $\Sigma$ is an candidate for the maxima of $\sigma^{2}$. Recall that for any such vector $u$ with the corresponding eigen-value $\lambda_{u}$ the value of $\sigma_{u}^{2} $ is:

$$
\begin{align}
\sigma_{u}^{2} &= u^{\top} \Sigma u \\
               &= \lambda_{u} \lVert u \rVert^{2}\\
               & = \lambda_{u}
\end{align}
$$
Therefore maximum of $\sigma^{2}$ is achived at the eigenvectors with highest eigenvalue.

### Conclusion and remarks [↩](#content)
[](#conclusion)

Notice that above computation showed that the first principal direction is an eigenvector of length 1 and whose eigenvector is the highest among other eigenvectors. For the second direction we will have the same maximization problem with an additional constraint that the second principal direction $u_2$ will be orthogonal to $u_1$. For more details of finding other directions you can look at: [Introduction to statistical learning - p377](http://www-bcf.usc.edu/~gareth/ISL/).

__remark 1__ In this notebook we didn't discuss the other but equivalent formulation of PCA which is minimum error formulation. This formulation is kind of inituitive after the approach we disscussed. The idea is the principal directions should be also 'closest' to the datapoints. For more on this formulation you can check [Patter recognition and Machine Learning - p563](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf).

__remark 2__ In application PCA's has some disadvantages. First of all, note that they are sensitive to the outliers. Also the principal directions depends on the scaling of the data. To address these problems usually it is advised that the data is standardized, that is mean zero and standard deviation of the columns are 1. For more on this, please check [Pattern recognition and Machine Learning - p568](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) and [Inroduction to statistical learning - p380](http://www-bcf.usc.edu/~gareth/ISL/).

#### Footnotes

 1 [Lagrange multiplier method - Khan Academy](https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/constrained-optimization/a/interpretation-of-lagrange-multipliers). [↩](#Lagrange_multiplier_1)

  2 [Lagrange multiplier method - Wikipedia](https://en.wikipedia.org/wiki/Lagrange_multiplier). [↩](#Lagrange_multiplier_2)

   3 [Lagrange multiplier method - A detailed blogpost](https://danstronger.wordpress.com/2015/08/08/lagrange-multipliers/). [↩](#Lagrange_multiplier_3)

# Ipython code

```python

import ipywidgets as widgets
from IPython.display import display, HTML

javascript_functions = {False: "hide()", True: "show()"}
button_descriptions  = {False: "Show code", True: "Hide code"}

def toggle_code(state):

    """
    Toggles the JavaScript show()/hide() function on the div.input element.
    """

    output_string = ""
    output_args   = (javascript_functions[state],)
    output        = output_string.format(*output_args)

    display(HTML(output))

def button_action(value):

    """
    Calls the toggle_code function and updates the button description.
    """

    state = value.new

    toggle_code(state)

    value.owner.description = button_descriptions[state]

state = False
toggle_code(state)

button = widgets.ToggleButton(state, description = button_descriptions[state])
button.observe(button_action, "value")

display(button)
```

    ToggleButton(value=False, description='Show code')


```python
%matplotlib inline
import numpy as np
import pylab as plt
import pandas as pd
from sklearn import svm
from sklearn.metrics import classification_report,confusion_matrix,accuracy_score
```

```python
np.random.seed(0)
X = np.r_[np.random.randn(20, 2) - [2, 2], np.random.randn(20, 2) + [2, 2]]
Y = [0] * 20 + [1] * 20
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.coolwarm, s=25)
```

```python
fig, ax = plt.subplots()
clf2 = svm.LinearSVC(C=1).fit(X, Y)

# get the separating hyperplane
w = clf2.coef_[0]
a = -w[0] / w[1]
xx = np.linspace(-5, 5)
yy = a * xx - (clf2.intercept_[0]) / w[1]

# create a mesh to plot in
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx2, yy2 = np.meshgrid(np.arange(x_min, x_max, .2),
                     np.arange(y_min, y_max, .2))
Z = clf2.predict(np.c_[xx2.ravel(), yy2.ravel()])

Z = Z.reshape(xx2.shape)
ax.contourf(xx2, yy2, Z, cmap=plt.cm.coolwarm, alpha=0.3)
ax.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.coolwarm, s=25)
ax.plot(xx,yy)

ax.axis([x_min, x_max,y_min, y_max])
plt.show()
```

**Supervised learning**:

  - Linear models (Ridge, Lasso, Elastic Net, ...)
  - Support Vector Machines
  - Tree-based methods (Random Forests, Bagging, GBRT, ...)
  - Nearest neighbors
  - Neural networks (basics)

**Unsupervised learning**:

  - Clustering (KMeans, Ward, ...)
  - Outlier detection

## Accuracy and precision

1. precision:
The fraction of relevant instances among the retrieved instances,
1. recall:
The fraction of relevant instances that have been retrieved over the total amount of relevant instances.
1. F-score

* The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.

* The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.

* The F-beta score can be interpreted as a weighted harmonic mean of the precision and recall, where an F-beta score reaches its best value at 1 and worst score at 0. The F-beta score weights recall more than precision by a factor of beta. beta == 1.0 means recall and precision are equally important.

* The support is the number of occurrences of each class in y_true.

Take a look at [HERE](https://en.wikipedia.org/wiki/F1_score) or [HERE](https://en.wikipedia.org/wiki/Precision_and_recall).

```python
# Generate data
from sklearn.datasets import make_blobs,make_circles,make_moons
# X, y = make_blobs(n_samples=1000, centers=2,
#                   cluster_std=1.5,
#                   center_box=(-4.0, 4.0))
# X, y = make_circles(n_samples=1000, noise=.1, factor=.5)
X,y = make_moons(n_samples=1000, noise=.2)

plt.scatter(X[:,0],X[:,1],c=y,)

```

```python
print(X[:3])
```

    [[-0.36188439  1.18939692]
     [ 1.00567821  0.91527408]
     [-1.13965547  0.29097234]]

```python
print(y[:3])
```

    [0 0 0]

```python
# X is a 2 dimensional array, with 1000 rows and 2 columns
print(X.shape)

# y is a vector of 1000 elements
print(y.shape)
```

    (1000, 2)
    (1000,)

```python
X_train, y_train = X[:700], y[:700]
X_test, y_test = X[700:], y[700:]
```

```python
fig,(ax1,ax2) = plt.subplots(1,2,figsize=(14,7))

ax1.scatter(X_train[:,0],X_train[:,1],c=y_train,)
ax2.scatter(X_test[:,0],X_test[:,1],c=y_test,)

```

## K-Nearest Neighbours

```python
# K-Nearest Neighbours
from sklearn.neighbors import KNeighborsClassifier

Model = KNeighborsClassifier(n_neighbors=8)
Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)

# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
# Accuracy score

print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.96      0.97      0.97       151
               1       0.97      0.96      0.97       149

       micro avg       0.97      0.97      0.97       300
       macro avg       0.97      0.97      0.97       300
    weighted avg       0.97      0.97      0.97       300

    [[147   4]
     [  6 143]]
    accuracy is 0.9666666666666667

```python
# Compute (approximate) class probabilities
print(Model.predict_proba(X_test[:5]))
```

    [[1. 0.]
     [1. 0.]
     [0. 1.]
     [0. 1.]
     [1. 0.]]

##  Radius Neighbors Classifier

```python
from sklearn.neighbors import  RadiusNeighborsClassifier
Model=RadiusNeighborsClassifier(radius=8.0)
Model.fit(X_train,y_train)
y_pred=Model.predict(X_test)
#summary of the predictions made by the classifier
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_test,y_pred))
#Accouracy score
print('accuracy is ', accuracy_score(y_test,y_pred))
```

                  precision    recall  f1-score   support

               0       1.00      0.32      0.48       158
               1       0.57      1.00      0.72       142

       micro avg       0.64      0.64      0.64       300
       macro avg       0.78      0.66      0.60       300
    weighted avg       0.80      0.64      0.60       300

    [[ 50 108]
     [  0 142]]
    accuracy is  0.64

## Naive Bayes

\begin{align}\begin{aligned}P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)\\\Downarrow\\\hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i \mid y),\end{aligned}\end{align}

```python
# Naive Bayes
from sklearn.naive_bayes import GaussianNB
Model = GaussianNB()
Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)

# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
# Accuracy score
print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.93      0.93      0.93       158
               1       0.92      0.92      0.92       142

       micro avg       0.93      0.93      0.93       300
       macro avg       0.93      0.93      0.93       300
    weighted avg       0.93      0.93      0.93       300

    [[147  11]
     [ 11 131]]
    accuracy is 0.9266666666666666

## SVM

```python
# Support Vector Machine
from sklearn.svm import SVC

Model = SVC(kernel='linear')
# Model = svm.LinearSVC(C=1)

Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)

# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
# Accuracy score

print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.93      0.94      0.94       158
               1       0.94      0.92      0.93       142

       micro avg       0.93      0.93      0.93       300
       macro avg       0.93      0.93      0.93       300
    weighted avg       0.93      0.93      0.93       300

    [[149   9]
     [ 11 131]]
    accuracy is 0.9333333333333333

```python
plt.scatter(X_test[:,0],X_test[:,1],c=y_pred,)
```

## Decision Tree

```python
# Decision Tree's
from sklearn.tree import DecisionTreeClassifier

Model = DecisionTreeClassifier()

Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)

# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
# Accuracy score
print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.88      0.87      0.88       158
               1       0.86      0.87      0.87       142

       micro avg       0.87      0.87      0.87       300
       macro avg       0.87      0.87      0.87       300
    weighted avg       0.87      0.87      0.87       300

    [[138  20]
     [ 18 124]]
    accuracy is 0.8733333333333333

## RandomForest

```python
from sklearn.ensemble import RandomForestClassifier
Model=RandomForestClassifier(max_depth=2)
Model.fit(X_train,y_train)
y_pred=Model.predict(X_test)
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_pred,y_test))
#Accuracy Score
print('accuracy is ',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.93      0.94      0.93       158
               1       0.93      0.92      0.93       142

       micro avg       0.93      0.93      0.93       300
       macro avg       0.93      0.93      0.93       300
    weighted avg       0.93      0.93      0.93       300

    [[148  11]
     [ 10 131]]
    accuracy is  0.93

    /home/gf/packages/anaconda3/lib/python3.6/site-packages/sklearn/ensemble/forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
      "10 in version 0.20 to 100 in 0.22.", FutureWarning)

## Neural network

```python
from sklearn.neural_network import MLPClassifier
Model=MLPClassifier()
Model.fit(X_train,y_train)
y_pred=Model.predict(X_test)
# Summary of the predictions
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_test,y_pred))
#Accuracy Score
print('accuracy is ',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.93      0.93      0.93       158
               1       0.92      0.92      0.92       142

       micro avg       0.93      0.93      0.93       300
       macro avg       0.93      0.93      0.93       300
    weighted avg       0.93      0.93      0.93       300

    [[147  11]
     [ 11 131]]
    accuracy is  0.9266666666666666

```python
help(MLPClassifier)
```

    Help on class MLPClassifier in module sklearn.neural_network.multilayer_perceptron:

    class MLPClassifier(BaseMultilayerPerceptron, sklearn.base.ClassifierMixin)
     |  Multi-layer Perceptron classifier.
     |
     |  This model optimizes the log-loss function using LBFGS or stochastic
     |  gradient descent.
     |
     |  .. versionadded:: 0.18
     |
     |  Parameters
     |  ----------
     |  hidden_layer_sizes : tuple, length = n_layers - 2, default (100,)
     |      The ith element represents the number of neurons in the ith
     |      hidden layer.
     |
     |  activation : {'identity', 'logistic', 'tanh', 'relu'}, default 'relu'
     |      Activation function for the hidden layer.
     |
     |      - 'identity', no-op activation, useful to implement linear bottleneck,
     |        returns f(x) = x
     |
     |      - 'logistic', the logistic sigmoid function,
     |        returns f(x) = 1 / (1 + exp(-x)).
     |
     |      - 'tanh', the hyperbolic tan function,
     |        returns f(x) = tanh(x).
     |
     |      - 'relu', the rectified linear unit function,
     |        returns f(x) = max(0, x)
     |
     |  solver : {'lbfgs', 'sgd', 'adam'}, default 'adam'
     |      The solver for weight optimization.
     |
     |      - 'lbfgs' is an optimizer in the family of quasi-Newton methods.
     |
     |      - 'sgd' refers to stochastic gradient descent.
     |
     |      - 'adam' refers to a stochastic gradient-based optimizer proposed
     |        by Kingma, Diederik, and Jimmy Ba
     |
     |      Note: The default solver 'adam' works pretty well on relatively
     |      large datasets (with thousands of training samples or more) in terms of
     |      both training time and validation score.
     |      For small datasets, however, 'lbfgs' can converge faster and perform
     |      better.
     |
     |  alpha : float, optional, default 0.0001
     |      L2 penalty (regularization term) parameter.
     |
     |  batch_size : int, optional, default 'auto'
     |      Size of minibatches for stochastic optimizers.
     |      If the solver is 'lbfgs', the classifier will not use minibatch.
     |      When set to "auto", `batch_size=min(200, n_samples)`
     |
     |  learning_rate : {'constant', 'invscaling', 'adaptive'}, default 'constant'
     |      Learning rate schedule for weight updates.
     |
     |      - 'constant' is a constant learning rate given by
     |        'learning_rate_init'.
     |
     |      - 'invscaling' gradually decreases the learning rate at each
     |        time step 't' using an inverse scaling exponent of 'power_t'.
     |        effective_learning_rate = learning_rate_init / pow(t, power_t)
     |
     |      - 'adaptive' keeps the learning rate constant to
     |        'learning_rate_init' as long as training loss keeps decreasing.
     |        Each time two consecutive epochs fail to decrease training loss by at
     |        least tol, or fail to increase validation score by at least tol if
     |        'early_stopping' is on, the current learning rate is divided by 5.
     |
     |      Only used when ``solver='sgd'``.
     |
     |  learning_rate_init : double, optional, default 0.001
     |      The initial learning rate used. It controls the step-size
     |      in updating the weights. Only used when solver='sgd' or 'adam'.
     |
     |  power_t : double, optional, default 0.5
     |      The exponent for inverse scaling learning rate.
     |      It is used in updating effective learning rate when the learning_rate
     |      is set to 'invscaling'. Only used when solver='sgd'.
     |
     |  max_iter : int, optional, default 200
     |      Maximum number of iterations. The solver iterates until convergence
     |      (determined by 'tol') or this number of iterations. For stochastic
     |      solvers ('sgd', 'adam'), note that this determines the number of epochs
     |      (how many times each data point will be used), not the number of
     |      gradient steps.
     |
     |  shuffle : bool, optional, default True
     |      Whether to shuffle samples in each iteration. Only used when
     |      solver='sgd' or 'adam'.
     |
     |  random_state : int, RandomState instance or None, optional, default None
     |      If int, random_state is the seed used by the random number generator;
     |      If RandomState instance, random_state is the random number generator;
     |      If None, the random number generator is the RandomState instance used
     |      by `np.random`.
     |
     |  tol : float, optional, default 1e-4
     |      Tolerance for the optimization. When the loss or score is not improving
     |      by at least ``tol`` for ``n_iter_no_change`` consecutive iterations,
     |      unless ``learning_rate`` is set to 'adaptive', convergence is
     |      considered to be reached and training stops.
     |
     |  verbose : bool, optional, default False
     |      Whether to print progress messages to stdout.
     |
     |  warm_start : bool, optional, default False
     |      When set to True, reuse the solution of the previous
     |      call to fit as initialization, otherwise, just erase the
     |      previous solution. See :term:`the Glossary <warm_start>`.
     |
     |  momentum : float, default 0.9
     |      Momentum for gradient descent update. Should be between 0 and 1. Only
     |      used when solver='sgd'.
     |
     |  nesterovs_momentum : boolean, default True
     |      Whether to use Nesterov's momentum. Only used when solver='sgd' and
     |      momentum > 0.
     |
     |  early_stopping : bool, default False
     |      Whether to use early stopping to terminate training when validation
     |      score is not improving. If set to true, it will automatically set
     |      aside 10% of training data as validation and terminate training when
     |      validation score is not improving by at least tol for
     |      ``n_iter_no_change`` consecutive epochs.
     |      Only effective when solver='sgd' or 'adam'
     |
     |  validation_fraction : float, optional, default 0.1
     |      The proportion of training data to set aside as validation set for
     |      early stopping. Must be between 0 and 1.
     |      Only used if early_stopping is True
     |
     |  beta_1 : float, optional, default 0.9
     |      Exponential decay rate for estimates of first moment vector in adam,
     |      should be in [0, 1). Only used when solver='adam'
     |
     |  beta_2 : float, optional, default 0.999
     |      Exponential decay rate for estimates of second moment vector in adam,
     |      should be in [0, 1). Only used when solver='adam'
     |
     |  epsilon : float, optional, default 1e-8
     |      Value for numerical stability in adam. Only used when solver='adam'
     |
     |  n_iter_no_change : int, optional, default 10
     |      Maximum number of epochs to not meet ``tol`` improvement.
     |      Only effective when solver='sgd' or 'adam'
     |
     |      .. versionadded:: 0.20
     |
     |  Attributes
     |  ----------
     |  classes_ : array or list of array of shape (n_classes,)
     |      Class labels for each output.
     |
     |  loss_ : float
     |      The current loss computed with the loss function.
     |
     |  coefs_ : list, length n_layers - 1
     |      The ith element in the list represents the weight matrix corresponding
     |      to layer i.
     |
     |  intercepts_ : list, length n_layers - 1
     |      The ith element in the list represents the bias vector corresponding to
     |      layer i + 1.
     |
     |  n_iter_ : int,
     |      The number of iterations the solver has ran.
     |
     |  n_layers_ : int
     |      Number of layers.
     |
     |  n_outputs_ : int
     |      Number of outputs.
     |
     |  out_activation_ : string
     |      Name of the output activation function.
     |
     |  Notes
     |  -----
     |  MLPClassifier trains iteratively since at each time step
     |  the partial derivatives of the loss function with respect to the model
     |  parameters are computed to update the parameters.
     |
     |  It can also have a regularization term added to the loss function
     |  that shrinks model parameters to prevent overfitting.
     |
     |  This implementation works with data represented as dense numpy arrays or
     |  sparse scipy arrays of floating point values.
     |
     |  References
     |  ----------
     |  Hinton, Geoffrey E.
     |      "Connectionist learning procedures." Artificial intelligence 40.1
     |      (1989): 185-234.
     |
     |  Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of
     |      training deep feedforward neural networks." International Conference
     |      on Artificial Intelligence and Statistics. 2010.
     |
     |  He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level
     |      performance on imagenet classification." arXiv preprint
     |      arXiv:1502.01852 (2015).
     |
     |  Kingma, Diederik, and Jimmy Ba. "Adam: A method for stochastic
     |      optimization." arXiv preprint arXiv:1412.6980 (2014).
     |
     |  Method resolution order:
     |      MLPClassifier
     |      BaseMultilayerPerceptron
     |      abc.NewBase
     |      sklearn.base.BaseEstimator
     |      sklearn.base.ClassifierMixin
     |      builtins.object
     |
     |  Methods defined here:
     |
     |  __init__(self, hidden_layer_sizes=(100,), activation='relu', solver='adam', alpha=0.0001, batch_size='auto', learning_rate='constant', learning_rate_init=0.001, power_t=0.5, max_iter=200, shuffle=True, random_state=None, tol=0.0001, verbose=False, warm_start=False, momentum=0.9, nesterovs_momentum=True, early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08, n_iter_no_change=10)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  fit(self, X, y)
     |      Fit the model to data matrix X and target(s) y.
     |
     |      Parameters
     |      ----------
     |      X : array-like or sparse matrix, shape (n_samples, n_features)
     |          The input data.
     |
     |      y : array-like, shape (n_samples,) or (n_samples, n_outputs)
     |          The target values (class labels in classification, real numbers in
     |          regression).
     |
     |      Returns
     |      -------
     |      self : returns a trained MLP model.
     |
     |  predict(self, X)
     |      Predict using the multi-layer perceptron classifier
     |
     |      Parameters
     |      ----------
     |      X : {array-like, sparse matrix}, shape (n_samples, n_features)
     |          The input data.
     |
     |      Returns
     |      -------
     |      y : array-like, shape (n_samples,) or (n_samples, n_classes)
     |          The predicted classes.
     |
     |  predict_log_proba(self, X)
     |      Return the log of probability estimates.
     |
     |      Parameters
     |      ----------
     |      X : array-like, shape (n_samples, n_features)
     |          The input data.
     |
     |      Returns
     |      -------
     |      log_y_prob : array-like, shape (n_samples, n_classes)
     |          The predicted log-probability of the sample for each class
     |          in the model, where classes are ordered as they are in
     |          `self.classes_`. Equivalent to log(predict_proba(X))
     |
     |  predict_proba(self, X)
     |      Probability estimates.
     |
     |      Parameters
     |      ----------
     |      X : {array-like, sparse matrix}, shape (n_samples, n_features)
     |          The input data.
     |
     |      Returns
     |      -------
     |      y_prob : array-like, shape (n_samples, n_classes)
     |          The predicted probability of the sample for each class in the
     |          model, where classes are ordered as they are in `self.classes_`.
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  partial_fit
     |      Update the model with a single iteration over the given data.
     |
     |      Parameters
     |      ----------
     |      X : {array-like, sparse matrix}, shape (n_samples, n_features)
     |          The input data.
     |
     |      y : array-like, shape (n_samples,)
     |          The target values.
     |
     |      classes : array, shape (n_classes), default None
     |          Classes across all calls to partial_fit.
     |          Can be obtained via `np.unique(y_all)`, where y_all is the
     |          target vector of the entire dataset.
     |          This argument is required for the first call to partial_fit
     |          and can be omitted in the subsequent calls.
     |          Note that y doesn't need to contain all labels in `classes`.
     |
     |      Returns
     |      -------
     |      self : returns a trained MLP model.
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes defined here:
     |
     |  __abstractmethods__ = frozenset()
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from sklearn.base.BaseEstimator:
     |
     |  __getstate__(self)
     |
     |  __repr__(self)
     |      Return repr(self).
     |
     |  __setstate__(self, state)
     |
     |  get_params(self, deep=True)
     |      Get parameters for this estimator.
     |
     |      Parameters
     |      ----------
     |      deep : boolean, optional
     |          If True, will return the parameters for this estimator and
     |          contained subobjects that are estimators.
     |
     |      Returns
     |      -------
     |      params : mapping of string to any
     |          Parameter names mapped to their values.
     |
     |  set_params(self, **params)
     |      Set the parameters of this estimator.
     |
     |      The method works on simple estimators as well as on nested objects
     |      (such as pipelines). The latter have parameters of the form
     |      ``<component>__<parameter>`` so that it's possible to update each
     |      component of a nested object.
     |
     |      Returns
     |      -------
     |      self
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors inherited from sklearn.base.BaseEstimator:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from sklearn.base.ClassifierMixin:
     |
     |  score(self, X, y, sample_weight=None)
     |      Returns the mean accuracy on the given test data and labels.
     |
     |      In multi-label classification, this is the subset accuracy
     |      which is a harsh metric since you require for each sample that
     |      each label set be correctly predicted.
     |
     |      Parameters
     |      ----------
     |      X : array-like, shape = (n_samples, n_features)
     |          Test samples.
     |
     |      y : array-like, shape = (n_samples) or (n_samples, n_outputs)
     |          True labels for X.
     |
     |      sample_weight : array-like, shape = [n_samples], optional
     |          Sample weights.
     |
     |      Returns
     |      -------
     |      score : float
     |          Mean accuracy of self.predict(X) wrt. y.

```python

```

```python
df = pd.read_csv('datasets//H`itters.csv').dropna()
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 263 entries, 1 to 321
    Data columns (total 21 columns):
    Unnamed: 0    263 non-null object
    AtBat         263 non-null int64
    Hits          263 non-null int64
    HmRun         263 non-null int64
    Runs          263 non-null int64
    RBI           263 non-null int64
    Walks         263 non-null int64
    Years         263 non-null int64
    CAtBat        263 non-null int64
    CHits         263 non-null int64
    CHmRun        263 non-null int64
    CRuns         263 non-null int64
    CRBI          263 non-null int64
    CWalks        263 non-null int64
    League        263 non-null object
    Division      263 non-null object
    PutOuts       263 non-null int64
    Assists       263 non-null int64
    Errors        263 non-null int64
    Salary        263 non-null float64
    NewLeague     263 non-null object
    dtypes: float64(1), int64(16), object(4)
    memory usage: 45.2+ KB

```python
X = df[['Years', 'Hits']].as_matrix()
y = np.log(df.Salary.as_matrix())
```

    /home/gf/packages/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
      """Entry point for launching an IPython kernel.
    /home/gf/packages/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.

```python
from sklearn.externals.six import StringIO
import pydot
from IPython.display import Image
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier, export_graphviz
regr = DecisionTreeRegressor(max_leaf_nodes=3)
regr.fit(X, y)
```

    DecisionTreeRegressor(criterion='mse', max_depth=None, max_features=None,
               max_leaf_nodes=3, min_impurity_decrease=0.0,
               min_impurity_split=None, min_samples_leaf=1,
               min_samples_split=2, min_weight_fraction_leaf=0.0,
               presort=False, random_state=None, splitter='best')

```python
# This function creates images of tree models using pydot
def print_tree(estimator, features, class_names=None, filled=True):
    tree = estimator
    names = features
    color = filled
    classn = class_names

    dot_data = StringIO()
    export_graphviz(estimator, out_file=dot_data, feature_names=features, class_names=classn, filled=filled)
    graph = pydot.graph_from_dot_data(dot_data.getvalue())
    return(graph)
```

```python
graph, = print_tree(regr, features=['Years', 'Hits'])
Image(graph.create_png())
```

```python

```


```python
import psi4
from psi4 import *
from psi4.core import *
import numpy as np
import os
sys.path.append('os.getcwd()')
from opt_helper import stre, bend, intcosMisc, linearAlgebra
```

## The Step Back-transformation
An optimization algorithm carried out in internal coordinates (see, e.g., the RFO tutorial) will generate a displacement step to be taken in internal coordinates.  The conversion of the step into Cartesian coordinates is here called the "back-transformation" ("back", since the original gradient was computed in Cartesians).

As shown in the tutorial on coordinates and the B-matrix,

$$\textbf {B} \Delta x = \Delta q $$

and the $\textbf A^T$ matrix defined by

$$ \textbf A^T \equiv (\textbf{B} \textbf{u} \textbf {B}^T)^{-1} \textbf {B} \textbf{u}$$

was shown to be the left inverse of $\textbf B^T$ where __u__ is an arbitrary symmetric matrix.  Attention must be paid to the non-square nature of __A__ and __B__.  Here, we have

\begin{align}
\textbf B \Delta x &= \Delta q \\
\\
\textbf B \Delta x &= \big( \textbf{BuB}^T \big) \big( \textbf{BuB}^T\big)^{-1} \Delta q \\
\\
\textbf B \Delta x &= \textbf B \big[ \textbf{uB}^T \big( \textbf{BuB}^T\big)^{-1}\big] \Delta q \\
\\
\Delta x &= \textbf{uB}^T \big( \textbf{BuB}^T\big)^{-1} \Delta q = \textbf A \Delta q \\
\end{align}

The __u__ matrix may be chosen to be the unit matrix which gives

$$\Delta x = \textbf B^T (\textbf B \textbf B^T)^{-1} \Delta q$$

where redundant coordinates can be accommodated simply by using the generalized inverse.  It is common to introduce $ \textbf{G} = \textbf B \textbf B^T $ and write the expression as

$$ \Delta x = \textbf{B}^T \textbf{G}^{-1} \Delta q$$

Note the __G__ matrix is a square matrix of dimension (number of internals) by (number of internals).  This equation is exact only for infinitesimal displacements, because the B-matrix elements depend upon the molecular geometry (i.e., the Cartesian coordinates).  Thus, the back-transformation is carried out iteratively.

To converge on a Cartesian geometry with the desired internal coordinate values, we repeatedly compute the difference between the current internal coordinate values and the desired ones (generating repeated $\Delta q$'s) and using the equation above to compute a new Cartesian geometry.

### Illustration of back-transformation
The back-transformation will now be demonstrated by taking a 0.2 au step increase in the bond lengths and a 5 degree increase in the bond angle of a water molecule.

```python
# Setup the water molecule and coordinates.
mol = psi4.geometry("""
O
H 1 1.7
H 1 1.7 2 104
unit au
""")
# We'll use cc-pVDZ RHF.
psi4.set_options({"basis": "cc-pvdz"})
mol.update_geometry()
xyz_0 = np.array( mol.geometry() )

# Generate the internal coordinates manually.  Show their values.
intcos = [stre.STRE(0,1), stre.STRE(0,2), bend.BEND(1,0,2)]
print("%15s%15s" % ('Coordinate', 'Value'))
for I in intcos:
  print("%15s = %15.8f %15.8f" % (I, I.q(xyz_0), I.qShow(xyz_0)))

# Handy variables for later.
Natom = mol.natom()
Nintco = len(intcos)
Ncart = 3*Natom
```

```python
# Create an internal coordinate displacement of +0.2au in bond lengths,
# and +5 degrees in the bond angle.
dq = np.array( [0.2, 0.2, 5.0/180*np.pi], float)

B = intcosMisc.Bmat(intcos, xyz_0)
G = np.dot(B, B.T)
G_inv = linearAlgebra.symmMatInv(G, redundant=True)

# Dx = B^T G^(-1) Dq
dx = np.dot(B.T, np.dot(G_inv, dq))
print("Displacement in Cartesians")
print(dx)

# Add Dx to original geometry.
xyz_1 = np.add(np.reshape(dx, (3, 3)), xyz_0)
print("New geometry in cartesians")
print(xyz_1)

# Compute internal coordinate values of new geometry.
print("\n%15s%15s" % ('Coordinate', 'Value'))
for I in intcos:
  print("%15s = %15.8f %15.8f" % (I, I.q(xyz_1), I.qShow(xyz_1)))
```

You see that the desired internal coordinate value is not _exactly_ achieved.  You can play with the desired displacement and observe more diverse behavior.  For water, if you displace only the bond lengths, then the result will be exact, because if the bond angle is fixed then the direction of the displacements (_s_-vectors on each atom) are constant wrt to the bond lengths.  On the other hand, the displacement directions for the bend depend upon the value of the angle.  So if you displace only along a bend, the result will not be exact.  In general, the result is reasonable but only approximate for small displacements.

### Illustration of iterative back-transformation
Finally, we demonstrate how convergence to the desired internal coordinate displacement can be achieved by an interative process.

```python
# Create array of target internal coordinate values.
dq_target = np.array( [0.2, 0.2, 5.0/180*np.pi], float)
q_target = np.zeros( (len(intcos)), float)
for i, intco in enumerate(intcos):
    q_target[i] = intco.q(xyz_0) + dq_target[i]

xyz = xyz_0.copy()
rms_dq = 1
niter = 1

while rms_dq > 1e-10:
    print("Iteration %d" % niter)
    dq = dq_target.copy()

    # Compute distance from target in internal coordinates.
    for i, intco in enumerate(intcos):
        dq[i] = q_target[i] - intco.q(xyz)
    rms_dq = np.sqrt(np.mean(dq**2))
    print("\tRMS(dq) = %10.5e" % rms_dq)

    # Dx = B^T G^(-1) Dq
    B = intcosMisc.Bmat(intcos, xyz)
    G = np.dot(B, B.T)
    G_inv = linearAlgebra.symmMatInv(G, redundant=True)
    dx = np.dot(B.T, np.dot(G_inv, dq))
    print("\tRMS(dx) = %10.5e" % np.sqrt(np.mean(dx**2)))

    # Compute new Cartesian geometry.
    xyz[:] += np.reshape(dx, (3,3))
    niter += 1

print("\nFinal converged geometry.")
print(xyz)

# Compute internal coordinate values of new geometry.
print("\n%15s%15s" % ('Coordinate', 'Value'))
for I in intcos:
  print("%15s = %15.8f %15.8f" % (I, I.q(xyz), I.qShow(xyz)))
```

The exact desired displacement is achieved.

Due to the non-orthogonal nature of the coordinates, the iterations may not always converge.  In this case, common tactics include using the Cartesian geometry generated by the first back-transformation step, or using the Cartesian geometry that was closest to the desired internal coordinates.  Hopefully, as a geometry optimization proceeds, the forces and displacements get smaller and convergence occurs.

A serious complication in procedures such as this one are discontinuities in the values of the internal coordinates.  In some way, the internal coordinate values must be canonicalized so that, e.g., an increase in a torsion from 179 degrees to -178 degrees is interpreted as an increase of 3 degrees.  Similar problems present for bond angles near 180 degrees.  (The consistent computation of these changes in values and forces is also critical in Hessian update schemes.)


```python
'''
    File name          : Math HW 1A -- Dimensional Analysis.ipynb
    Author             : Ming-Feng Ho
    Date created       : 9/30/2018
    Date last modified : 10/2/2018
    Python Version     : 3.6
    Sympy Version      : 1.3
'''

import sympy as sp

# pretty print the latex formula in notebook
sp.init_printing()

# this function is like the print function in Jupyter notebook,
# you can check the Jupyter display document.
from IPython.display import display
```

```python
# (a) use the Coulomb law in cgs unit
F, q1, q2, r = sp.symbols("F q1 q2 r", real=True)
coulomb_law = sp.Eq(F, q1 * q2 * r**-2)
print("(a) Coulomb law in cgs unit")
display(coulomb_law)

# use symbols to represent the dimensions: length, mass, time
L, M, T = sp.symbols("L M T")

# convert into dimension
coulomb_law_dimension = coulomb_law.subs({
    F : M * L * T**-2,
    q1: sp.symbols("[q]"),
    q2: sp.symbols("[q]"),
    r : L
})
print("Substitute the dimensions")
display(coulomb_law_dimension)

# solve for [q]
sol = sp.solve(coulomb_law_dimension, sp.symbols("[q]"), rational=False)[1]
coulomb_law_dimension = sp.Eq(sp.symbols("[q]"), sol)
print("Solve for [q]")
display(coulomb_law_dimension.expand(force=True))
```

```python
# (b) Lagrangian L = Kinetic energy - potential energy, so it is in the same unit of energy.
Lagrangian = M * (L * T**-1)**2
print("(b) Lagrangian is in the same unit of energy, so I use m*v**2 to express the dimension of Lagrangian: [Lagrangian] =")
display(Lagrangian)

# Action is [S] = [Lagrangian] * T
S = sp.symbols("[S]")
display(sp.Eq(S, Lagrangian * T, ))
```

```python
# (c) Recall the F = q (v x B) equation
B, v, q = sp.symbols("B v q")
print("(c) Recall the F = q (v x B) equation")

# damn ... sympy's cross product is in vetor form.
display(sp.Eq(
    sp.symbols("[B]"),
    (F * q**-1 * v**-1).subs({
        F : M * L * T**-2,
        q : coulomb_law_dimension.rhs,
        v : L * T**-1
    })
).expand(force=True))
```

```python
# (d) Energy is in the same unit of Largrangian
print("(d) Energy is in the same unit of Largrangian")
display(sp.Eq(
    sp.symbols("[E]"),
    Lagrangian
))
```

```python
# (a) dessert animal
```

(a) dessert animal

I played a video game called "Zelda : Breath of the Wild" on Nintendo switch before. It is a really nice game. "Zelda" allows players to explore the world of Hyrule Kingdom with limited constraints. Anyway, there's a town called Gerudo on the southern area of the Hyrule Kingdom. The town is located in a harsh dessert. While the players is running in the dessert area, because it is too hot, their life (or Health Point (HP)) will decrease until they die.

So, we may assume that the maximal distance an animal can run depends on the how many HPs they have and the rate of decreasing HPs under the hot desert weather.

```python
# maximal distance = velocity * time = velocity * (number of HPs) / (rate of decreasing HP)
HP, rate = sp.symbols("HP rate")
maximal_distance = v * HP / rate
print("maximal distance = velocity * time = velocity * (number of HPs) / (rate of decreasing HP) =\n= {}\n".format(maximal_distance))

# assume: number of HPs is proportional to the volume size of an animal.
HP_expr = L**3
print("Assume: number of HPs is proportional to the volume size of an animal. \nLarger animals get more HPs. \nIn the game, the Bosses are usually bigger and have more HPs, so there may have a correlation here.")
display(sp.Eq(sp.symbols("[HP]"), HP_expr))

# assume: rate of decreasing HPs is proportional to the surface size of an animal.
rate_expr = L**2
print("Assume: rate of decreasing HPs is proportional to the surface size of an animal. \nLarger surface area usually more easily suffer from dehydration. \nAnd when you dehydration in the dessert, you die.")
display(sp.Eq(sp.symbols("[rate]"), rate_expr))

# combine all information, and recall that time is proportional to (number of HPs) / (rate of decreasing HP)
maximal_distance_dimension = maximal_distance.subs({
    v    : L * T**-1,
    HP   : HP_expr,
    rate : rate_expr,
}).subs({
    T : HP_expr * rate_expr**-1
})
print("Combine all information, and recall that time is proportional to (number of HPs) / (rate of decreasing HP)")
display(sp.Eq(
    sp.symbols("[MaximalDistance]"),
    maximal_distance_dimension
))
```

(b) Height of the jump of a animal

```python
# (b) Height of the jump of a animal
# recall the equation of Work and Energy
dx, W, m, g, h = sp.symbols("dx W m g h")
work_energy_expr = sp.Eq(F * dx, m * g * h)
print("Recall the equation of Work and Energy: \nWork = force * distance = mass * gravitational_acceleration * height")
display(work_energy_expr)

h_expr = sp.Eq(h, sp.solve(work_energy_expr, h)[0])
display(h_expr)

# set the force proportional to the cross section
print("Force is proptional to the cross section, which means the force is proportional to the size square, ")
c = sp.symbols("c")
F_expr = sp.Eq(F, c * L**2)
display(F_expr)
print("where L is the size and c is just a constant.")

# set the mass is proportional to the size^3, distance for the force pushing to floor is proportional to size
print("Say if the density of any animal is just a constant, we can set the mass is just proportional to the size^3.")
print("And, the dx, a displacement in a straight line in the direction of the force, is just proportional to the size.")
print("We get, ")

display(h_expr.subs({
    F : F_expr.rhs,
    m : L**3,
    g : 1,
    dx : L
}).subs({
    c : sp.symbols("c'")
}))
print("Surprisingly, we get the height an animal can jump is a constant.")

```

I guess it is the problem of the body density. If we take density into account, we get

\begin{equation}
    h \propto \frac{1}{\rho},
\end{equation}

where $\rho$ is the body density. The above relation is more promising than $h$ is just a constant.


## Markov Network

A Markov network is a set of random variables having a Markov property described by an undirected graph.

**Markov property**: Any two subsets of variables are conditionally independent given a separating subset.

**Clique factorization:**

Let $C$ denote a clique and the set of variables in it as ${\bf x}_C$
$$p({\bf x})=\frac{1}{Z}\prod_C \psi_C({\bf x}_C)$$
where $\psi_C({\bf x}_C)$ is called potential function and $Z$ is the normalization factor.

The definition is equivalent if only maximal cliques are used.

Boltzmann distribution:
$$\psi_C({\bf x}_C)=\exp\{-E({\bf x}_C)\}$$
where $E({\bf x}_C)$ is called energy function.
$$p({\bf x})=\frac{1}{Z}\prod_C \psi_C({\bf x}_C)=\frac{1}{Z}\exp\{-\sum_C E({\bf x}_C)\}$$

## Image Denoising

**Problem Description**:

Suppose we have an image consisting of a 2-dimensional array of pixels, where each pixel value is binary. Assume it got corrupted by random noise, where each pixel in the image is flipped with 10% probability.
Our goal is to recover the original image.

```python
import numpy as np
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
```

```python
orig_img = np.array(mpimg.imread('../input/origimg.jpg')).astype(int)
orig_img[orig_img<128] = -1
orig_img[orig_img>=128] = 1

plt.figure()
plt.axis('off')
plt.imshow(orig_img, cmap='binary_r')

noisy_img = np.array(mpimg.imread('../input/noisyimg.jpg')).astype(int)
noisy_img[noisy_img<128] = -1
noisy_img[noisy_img>=128] = 1

plt.figure()
plt.axis('off')
plt.imshow(noisy_img, cmap='binary_r')
print ('Image:')

```

### Model

**Observed variables**: each pixel in the noisy image, $y_{ij}\in \{-1,+1\}$

**Unobserved variables**: each pixel in the ground truth image, $x_{ij}\in \{-1,+1\}$

**Assumptions**:

1. There is a strong correlation between $x_{ij}$ and $y_{ij}$

2. There is a strong correlation between neighboring pixels, i.e., $x_{ij}$ and $x_{i+1,j}$, $x_{i-1,j}$, $x_{i,j+1}$, $x_{i,j-1}$

Energy function:
$$E({\bf x},{\bf y})=-\beta\sum_{i,j}x_{ij}y_{ij}-\eta\sum_{\{ij,kl\}}x_{ij}x_{kl}$$
where (i,j) and (k,l) in the third term are adjacent pixels.

\begin{equation}
\begin{split}
\arg\max_{\bf x} p({\bf x}|{\bf y})&=\arg\max_{\bf x} p({\bf x},{\bf y})\\
&=\arg\max_{\bf x} \frac{1}{Z}\exp\{-E({\bf x},{\bf y})\}\\
&=\arg\min_{\bf x} E({\bf x},{\bf y})\\
&= \arg\min_{\bf x} -\beta\sum_{i,j}x_{ij}y_{ij}-\eta\sum_{\{ij,kl\}}x_{ij}x_{kl}\\
&= \arg\max_{\bf x} \beta\sum_{i,j}x_{ij}y_{ij}+\eta\sum_{\{ij,kl\}}x_{ij}x_{kl}
\end{split}
\end{equation}

####  Iterated Conditional Modes (ICM)
1. Initialize $x_{ij}$ to their noisy values $y_{ij}$.

2. Iterate through each $x_{ij}$ and check whether setting it’s value to +1 or -1 leads to a lower energy (higher probability).

```python
# compute the error rate between the noisy image and the original image
def compute_error_rate(img1, img2):
    err = abs(img1 - img2) / 2
    return np.sum(err) / np.size(img2) * 100

print ('Percentage of mismatched pixels in noisy image: %.6f%%' % compute_error_rate(orig_img, noisy_img))
```

    Percentage of mismatched pixels in noisy image: 9.938571%

```python
def compute_prob(X, Y, i, j, m, n, beta, eta, x_value):

    result = beta * Y[i][j] * x_value

    if i > 0:
        result += eta * x_value * X[i-1][j]
    if i < m-1:
        result += eta * x_value * X[i+1][j]
    if j > 0:
        result += eta * x_value * X[i][j-1]
    if j < n-1:
        result += eta * x_value * X[i][j+1]

    return result
```

```python
def denoise_image(Y, orig, beta, eta):
    m, n = np.shape(Y)
    X = np.copy(Y)
    max_iter = 5

    for k in range(max_iter):
        for i in range(m):
            for j in range(n):

                p_pos = compute_prob(X, Y, i, j, m, n, beta, eta, 1)
                p_neg = compute_prob(X, Y, i, j, m, n, beta, eta, -1)

                if p_pos > p_neg:
                    X[i][j] = 1
                else:
                    X[i][j] = -1

        print('Iteration number:', k+1)
        print ('Percentage of mismatched pixels: %.6f%%' % compute_error_rate(orig, X))
    return X
```

```python
beta = 1.0
eta = 2.1
denoised_img = denoise_image(noisy_img, orig_img, beta, eta)

plt.figure()
plt.axis('off')
plt.imshow(denoised_img, cmap='binary_r')
print ('Percentage of mismatched pixels in denoised image: %.6f%%' % compute_error_rate(orig_img, denoised_img))
print ('Denoised Image:')
```


```python
import numpy as np
```

Notation see [here](https://www.overleaf.com/project/5eb163d1d04a61000199ae5f).

A _deformation_ transformes lattice vectors ${\bf a}_i$ into new lattice vectors ${\bf a}_i'$:

$$
\begin{align}
    A' =
    \begin{pmatrix}
        -~ {\bf a'}^1 ~- \\
        -~ {\bf a'}^2 ~- \\
        \vdots \\
    \end{pmatrix}
    =
    D
    \begin{pmatrix}
        -~ {\bf a}^1 ~- \\
        -~ {\bf a}^2 ~- \\
        \vdots \\
    \end{pmatrix}
    = D A~.
\end{align}
$$

Write  in shape $Ax = B$ by transposing:
$$
\begin{align}
    (A')^{\rm t} = A^{\rm t} D^{\rm t}
\end{align}
$$

so that $D$ can be obtained by `D = np.linalg.solve(A1.T, A2.T).T`.

```python
def get_deformation(cell1, cell2):
    """Compute the matrix that deforms cell1 into cell2"""
    return np.linalg.solve(cell1.T, cell2.T).T
```

```python
# Example

A1 = np.array([[0, 1, 1], [1, 0, 1], [1, 1, 0]])
A2 = np.array([[0, 1, 2], [1, 0, 1], [1, 1, 0]])
```

```python
# verify
D = get_deformation(A1, A2)
D
```

    array([[ 1.5,  0.5, -0.5],
           [ 0. ,  1. , -0. ],
           [ 0. ,  0. ,  1. ]])

```python
np.linalg.norm(D @ A1 - A2)
```

    0.0

```python

```


```julia
using RowEchelon
using Latexify

rref([1 2 3; 4 5 6])

display(latexify(rref([1 2 3; 4 5 6])))

display(latexify(rref([1 2 3; 4 5 6])))
```

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & -1.0 \\
0.0 & 1.0 & 2.0 \\
\end{array}
\right]
\end{equation}

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & -1.0 \\
0.0 & 1.0 & 2.0 \\
\end{array}
\right]
\end{equation}

```julia
# The Four Subspaces for A
using RowEchelon
using Latexify

A = [1 3 5 0 7;
     0 0 0 1 2;
     1 3 5 1 9]

At = transpose(A)
display(A)
display(rref(A))

display("+++++++++++A Transpose+++++++++++")
display(At)
display(latexify(rref(At)))
```

    3×5 Matrix{Int64}:
     1  3  5  0  7
     0  0  0  1  2
     1  3  5  1  9

    3×5 Matrix{Float64}:
     1.0  3.0  5.0  0.0  7.0
     0.0  0.0  0.0  1.0  2.0
     0.0  0.0  0.0  0.0  0.0

    "+++++++++++A Transpose+++++++++++"

    5×3 transpose(::Matrix{Int64}) with eltype Int64:
     1  0  1
     3  0  3
     5  0  5
     0  1  1
     7  2  9

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & 1.0 \\
0.0 & 1.0 & 1.0000000000000002 \\
0.0 & 0.0 & 0.0 \\
0.0 & 0.0 & 0.0 \\
0.0 & 0.0 & 0.0 \\
\end{array}
\right]
\end{equation}

```julia
A = [1 2 3;
     1 2 3;
     2 5 8]
display(A)
display(latexify(rref(A)))
```

    3×3 Matrix{Int64}:
     1  2  3
     1  2  3
     2  5  8

\begin{equation}
\left[
\begin{array}{ccc}
1.0 & 0.0 & -1.0 \\
0.0 & 1.0 & 2.0 \\
0.0 & 0.0 & 0.0 \\
\end{array}
\right]
\end{equation}

```julia
using RowEchelon
using Latexify

A = [1 2 3 1;
     1 1 2 1;
     1 2 3 1]
display(transpose(A))
transpose(A) * [-1,0,1]

```

    4×3 transpose(::Matrix{Int64}) with eltype Int64:
     1  1  1
     2  1  2
     3  2  3
     1  1  1

    4-element Vector{Int64}:
     0
     0
     0
     0

```julia
using RowEchelon
using Latexify

A = [1 3 4;  5 2 7]

rref(transpose(A))
```

    3×2 Matrix{Float64}:
     1.0  0.0
     0.0  1.0
     0.0  0.0

```julia
A = [1 1 1;
     1 0 1;
     1 -1 1]

display(rref(A))

A * [-1, 0, 1]
```

    3×3 Matrix{Float64}:
     1.0  0.0   1.0
     0.0  1.0  -0.0
     0.0  0.0   0.0

    3-element Vector{Int64}:
     0
     0
     0

```julia
A = [1 0 0;0 1 0;0 0 0]
A*A*A*A
```

    3×3 Matrix{Int64}:
     1  0  0
     0  1  0
     0  0  0

```julia
A = [1 0 0;0 1 0;0 0 0]
```

    3×3 Matrix{Int64}:
     1  0  0
     0  1  0
     0  0  0

```julia
a = [1.0, 2.0, 2.0]
b = [1.0, 1.0, 1.0]

x = 5/9

p = x * a

e = b - p

transpose(e) * a
```

    3-element Vector{Float64}:
      0.4444444444444444
     -0.11111111111111116
     -0.11111111111111116

    -2.220446049250313e-16

```julia

A = [1 2 2; 2 4 4; 2 4 4]
A = 1/9 * A
display(latexify(A))
display(latexify(A * A))
```

\begin{equation}
\left[
\begin{array}{ccc}
0.1111111111111111 & 0.2222222222222222 & 0.2222222222222222 \\
0.2222222222222222 & 0.4444444444444444 & 0.4444444444444444 \\
0.2222222222222222 & 0.4444444444444444 & 0.4444444444444444 \\
\end{array}
\right]
\end{equation}

\begin{equation}
\left[
\begin{array}{ccc}
0.1111111111111111 & 0.2222222222222222 & 0.2222222222222222 \\
0.2222222222222222 & 0.4444444444444444 & 0.4444444444444444 \\
0.2222222222222222 & 0.4444444444444444 & 0.4444444444444444 \\
\end{array}
\right]
\end{equation}

```julia
using RowEchelon
using Latexify

A = [1 2 3; 4 5 6; 7 8 9]

display(latexify(A))
display(latexify(A*A))
display(latexify(A*A*A))
display(latexify(A*A*A*A))
display(latexify(A*A*A*A*A))
```

\begin{equation}
\left[
\begin{array}{ccc}
1 & 2 & 3 \\
4 & 5 & 6 \\
7 & 8 & 9 \\
\end{array}
\right]
\end{equation}

\begin{equation}
\left[
\begin{array}{ccc}
30 & 36 & 42 \\
66 & 81 & 96 \\
102 & 126 & 150 \\
\end{array}
\right]
\end{equation}

\begin{equation}
\left[
\begin{array}{ccc}
468 & 576 & 684 \\
1062 & 1305 & 1548 \\
1656 & 2034 & 2412 \\
\end{array}
\right]
\end{equation}

\begin{equation}
\left[
\begin{array}{ccc}
7560 & 9288 & 11016 \\
17118 & 21033 & 24948 \\
26676 & 32778 & 38880 \\
\end{array}
\right]
\end{equation}

\begin{equation}
\left[
\begin{array}{ccc}
121824 & 149688 & 177552 \\
275886 & 338985 & 402084 \\
429948 & 528282 & 626616 \\
\end{array}
\right]
\end{equation}

```julia
using LinearAlgebra

A = [0 1; 1 0]

eigvals(A)
```

    2-element Vector{Float64}:
     -1.0
      1.0

```julia
A = [.8 .3; .3 .7]
A*A*A*A*A*A*A*A*A*A
```

    2×2 Matrix{Float64}:
     0.986516  0.835441
     0.835441  0.708036

```julia

```