# Random Number Generation

This demonstrates simple random number generation using `numpy`, and shows how to make plots along the way.

For details about the API of modules used here, see the documentation for [numpy](https://numpy.org/) and [matplotlib](https://matplotlib.org/).

```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

mpl.rc('font', size=14)
```

## Set Random Number Seed

Set the state of numpy's random number generator. Can call this repeatedly to change the seed.

```python
# The same combination as my luggage...
np.random.seed(12345)
```

## Transforming a Uniform Random Variable

### Draws from a Uniform Distribution

Generate $u\sim U(0,1)$ and histogram the results.

```python
# Generate 1000 random draws from U(0,1).
u = np.random.uniform(0, 1, size=1000)

fig, ax = plt.subplots(1,1, figsize=(6,4), tight_layout=True)

# Histogram the result.
# ubins = bin edges in the histogram: [0, 0.05, 0.1, ..., 1]
ubins = np.linspace(0., 1., 21)
n, bin_edges, p = ax.hist(u, bins=ubins, histtype='step')

# Bin centers: mean of bin edges: [0.025, 0.075, ..., 0.975].
# Bin counts estimate Poisson mean; estimated uncertainties are sqrt(counts).
bin_center = 0.5*(bin_edges[1:] + bin_edges[:-1])
ax.errorbar(bin_center, n, yerr=np.sqrt(n), fmt='_')

ax.set(xlim=(0,1), xlabel='$u$', ylabel='count');
```

### Transform Uniform Numbers into Exponentials

Use the transformation/inversion method to convert $u\sim U(0,1)$ to $x\sim\mathrm{Exp}(-1)$, e.g., sample the distribution

$$
p(x|\beta,I) = \frac{1}{\beta}e^{-x/\beta}~\mathrm{where}~\beta=1.
$$

In practice you could directly generate these quantities by calling `numpy.random.exponential`.

```python
beta = 1.
x = -beta*np.log(u)

fig, ax = plt.subplots(1,1, figsize=(6,4), tight_layout=True)

# Histogram the result.
# Stretch the previous bin edges by 6x.
# Use a log-linear scale.
n, bin_edges, p = ax.hist(x, bins=6*ubins, histtype='step', log=True)

bin_center = 0.5*(bin_edges[1:] + bin_edges[:-1])
ax.errorbar(bin_center, n, yerr=np.sqrt(n), fmt='_')

ax.set(xlim=(bin_edges[0], bin_edges[-1]), xlabel='$x$', ylabel='count');
```

### Generate Two Gaussian Variables using the Box-Müller Algorithm

You could of course use `numpy.random.normal` to generate Gaussian variables but this nicely illustrates the algorithm:

$$
\begin{align}
  u &\sim U(0,1) & v &\sim U(0,1) \\
  x &= \sqrt{-2\ln{u}}\cos{2\pi v}\sim\mathcal{N}(0,1) & y &= \sqrt{-2\ln{u}}\sin{2\pi v}\sim\mathcal{N}(0,1)
\end{align}
$$

```python
# We already have u, so generate v ~ U(0,1).
v = np.random.uniform(0.,1., size=1000)
r = np.sqrt(-2*np.log(u))
x = r * np.cos(2*np.pi*v)
y = r * np.sin(2*np.pi*v)

# Draw a scatter plot with projected x, y histograms.
# Here I do it manually, but the seaborn package can do this automatically.
fig, axes = plt.subplots(2,2, figsize=(7,7),
                         gridspec_kw={'width_ratios':[2,1], 'height_ratios':[1,2]})

ax = axes[1,0]
ax.scatter(x, y, alpha=0.5)
ax.set(xlim=(-4,4), xlabel='$x$',
       ylim=(-4,4), ylabel='$y$',
       aspect='equal')

# Draw the x projection.
# Binning: [-4, -3.5, -3, ..., 3.5, 4]
hbins = np.linspace(-4,4,17)
ax = axes[0,0]
n, bin_edges, p = ax.hist(x, bins=hbins, histtype='step')
bin_center = 0.5*(bin_edges[1:] + bin_edges[:-1])
ax.errorbar(bin_center, n, yerr=np.sqrt(n), fmt='_')
ax.set(xlim=(-4,4), xticklabels=[], ylabel='count')

# Draw the y projection.
ax = axes[1,1]
n, bin_edges, p = ax.hist(y, bins=hbins, histtype='step', orientation='horizontal')
bin_center = 0.5*(bin_edges[1:] + bin_edges[:-1])
ax.errorbar(n, bin_center, xerr=np.sqrt(n), fmt='_')
ax.set(ylim=(-4,4), yticklabels=[], xlabel='count')

# Turn the upper right panel off.
ax = axes[0,1]
ax.set_axis_off()

fig.tight_layout();
```

## Random Numbers via Acceptance-Rejection

Demonstrate sampling from the Rayleigh scattering function
$$
f(x) = \frac{3}{8}(1+x^2)\ \mathrm{where}\ x\in[-1,1]
$$
using the acceptance-rejection technique. In this case the instrumental distribution will be a uniform box between $-1$ and $1$ with maximum value $3/4$.

```python
f = lambda x: 3/8 * (1 + x**2)
x = np.random.uniform(-1,1, size=1000)
y = np.random.uniform(0,0.75, size=1000)

# Boolean array to track accepted and rejected points.
accept = y < f(x)
reject = ~accept
```

```python
fig, axes = plt.subplots(1,2, figsize=(10,4), sharex=True, sharey=True, tight_layout=True)

# Draw the accepted points in red and rejected points in blue.
ax = axes[0]
ax.scatter(x[accept], y[accept], color='r')
ax.scatter(x[reject], y[reject], color='b')

# Draw the function.
_x = np.linspace(-1,1,101)
ax.plot(_x, f(_x), color='k')
ax.set(xlabel='$x$', xlim=(-1,1),
       ylabel='$f(x)=3/8~(1+x^2)$', ylim=(0,0.75))

# Histogram the accepted data.
ax = axes[1]
ax.hist(x[accept], density=True, histtype='step')
ax.plot(_x, f(_x), color='k');
```


# Information Retrieval in High Dimensional Data
## Lab #2, 26.10.2017
## Statistical Decision making

### Task 1

Consider the two dimensional, discrete random variable $X = [X_1\,X_2]^T$ subjected to the joint probability density $p_X$ as described in the following table.

$\begin{array}{c||cc} p_X(X_1,X_2) & X_2=0 & X_2=1\\ \hline
X_1 = 0 & 0.4 & 0.3 \\
X_1 = 1 & 0.2 & 0.1
\end{array}$

a) Compute the marginal probability densities $p_{X_1}, p_{X_2}$ and the coonditional probability $P(X_2=0|X_1=0)$ as well as the expected value $\mathbb{E}[X]$ and the covariance matrix $\mathbb{E}[(X-\mathbb{E}[X])(X-\mathbb{E}[X])^T]$.

```python
import numpy as np
```

```python
# Init needed variables
joint_prob = np.array([[0.4, 0.3], [0.2, 0.1]])
results = np.array([[0, 1, 0, 1],
                    [0, 0, 1, 1]])
```

```python
px1 = np.sum(joint_prob, axis=1)
px2 = np.sum(joint_prob, axis=0)

print ("Marginal distribution density px1: {}".format(px1))
print ("Marginal distribution density px2: ",px2)
```

    Marginal distribution density px1: [ 0.7  0.3]
    Marginal distribution density px2:  [ 0.6  0.4]

```python
cond_prob = joint_prob[0][0]/px1[0]
print ("Conditional probability P(X2=0|X1=0): {:.3f}".format(cond_prob))
```

    Conditional probability P(X2=0|X1=0): 0.571

```python
EX = np.dot(results, np.ravel(joint_prob, 'F'))
print ("Expected Value EX:", EX)
```

    Expected Value EX: [ 0.3  0.4]

```python
results_centered = results - np.reshape(EX, (2,1))
CovX = np.dot(np.dot(results_centered, np.diag(joint_prob.ravel('F'))), results_centered.T)
print ("Covariance Matrix: \n{}".format(CovX))
```

    Covariance Matrix:
    [[ 0.21 -0.02]
     [-0.02  0.24]]

b) Write a PYTHON function toyrnd that expects the positive integer parameter n as its input an returns a matrix $X$ of size (2,n), containing $n$ samples drawn independently from the distribution $p_X$, as its output.

```python
def toyrnd(n):
    x1 = np.random.choice([0, 1], (1, n), p=px1)
    x2 = np.random.choice([0, 1], (1, n), p=px2)
    return np.vstack((x1,x2))
```

    [[0 1 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 1 0 0 1 0 1 0 1 0 0 0 1 0 0 1 0 1 0 1 0
      0 0 1 0 1 0 1 1 0 0 0 0 0 0 1 0 0 1 1 1 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 1 0
      0 1 0 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0]
     [1 0 0 1 1 0 0 0 0 1 1 1 0 1 0 1 1 0 0 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 1 0
      0 0 1 0 0 1 1 1 1 0 1 1 1 1 1 1 0 0 1 0 1 1 0 0 1 0 0 0 0 0 0 0 1 1 1 0 0
      1 1 1 0 1 0 0 0 0 0 0 0 1 0 0 1 1 0 1 0 1 0 0 1 0 0]]

c) Verify your results in a) by generating 10000 samples with toyrnd and computing the respective empirical values

```python
samples = toyrnd(10000)

p_x1equ1 = (samples[0].sum()/len(samples[0]))
p_x2equ1 = (samples[1].sum()/len(samples[1]))
p_x1 = np.array([1-p_x1equ1, p_x1equ1])
p_x2 = np.array([1-p_x2equ1, p_x2equ1])

print ("Empirical Marginal distribution density p_x1: {}".format(p_x1))
print ("Empirical Marginal distribution density p_x2: ",p_x2)
```

    Empirical Marginal distribution density p_x1: [ 0.6986  0.3014]
    Empirical Marginal distribution density p_x2:  [ 0.5997  0.4003]

```python
cond_prob_empirical = samples[1, samples[0,:]==0]
np.sum(cond_prob_empirical==0)/len(cond_prob_empirical)
```

    0.5986258230747209

### Task 2
The MNS trainign set consists of handwritten digits from 0 to 9, stored as PNG files of size 28 x 28 an indexed by label. Download the provided ZIP file from Moodle and make yourself familiar with the directory structure.

a) Grayscale images are typically described as matrices of uint8 values. For numerical calculations, it is more sensible to work with floating point numbers. Load two (arbitrary) images from the database and convert them to matrices I1 and I2 of float64 values in the interval $[0, 1]$.

```python
import imageio
import os
```

```python
path = './mnist/d4/'
filenames = os.listdir(path)
im1 = imageio.imread(path + filenames[0])
I1 = im1/255
im2 = imageio.imread(path +  filenames[10])
I2 = im2/255
```

b) The matrix equivalent of te euclidean norm $\| \cdot \|_2$ is the $Frobenius$ norm. For any matrix $\mathbf{A} \in \mathbb{R}^{m\, \times \, n}$, it is defined as

\begin{equation} \|\mathbf{A}\|_F = \sqrt{tr(\mathbf{A}^T\mathbf{A})} \end{equation}

where tr denotes the trace of a matrix. Compute the distance $\|\mathbf{I}_1 - \mathbf{I}_2 \|_F$ between the images I1 and I2 using three different procedures in PYTHON:
- Running the numpy.linalg.norm function with the 'fro' parameter
- Directly applying formula (1)
- Computing the euclidean norm between the vectorized images

```python
D = I1 - I2
```

```python
dist1 = np.linalg.norm(D, ord='fro')
dist1
```

    9.0642043267400751

```python
dist2 = np.sqrt(np.trace(np.dot(D.T,D)))
dist2
```

    9.0642043267400751

```python
d = np.reshape(D, 784)
dist3 = np.linalg.norm(d)
dist3
```

    9.0642043267400751

c) In the following, we want to solve a simple classification problem by applying <i>k-Nearest Neighbors</i>. To this end, choose two digits classes, e.g. 0 and 1, and lod n_train = 500 images from each class to the workspace. Convert them according to subtask a) and store them in vectorized form in the matrix X_train of size (784, 2*n_train). Provide an indicator vector Y_train pf length 2*n_train that assigns the respective digit class label to each column of X_train.
From each of the two classes, choose another set of n_test=10 images and create according matrices X_test and Y_test. Now, for each sample in the test set, determine the k = 20 training samples with the smallest Frobenius distance to it and store their indices in the (2*n_test, k) matrix NN. Generate a vector Y_kNN containing the respective estimated class labels by performing a majority vote on NN. Compare the result with Y_test.

```python
X_train = np.zeros((784, 1000))
Y_train = np.zeros(1000, dtype='int64')
digitclass0 = 0
digitclass1 = 1
path0 = './mnist/d' + str(digitclass0) +'/'
path1 = './mnist/d' + str(digitclass1) +'/'
filenames0 = os.listdir(path0)
filenames1 = os.listdir(path1)

for i in range(500):
    im0 = imageio.imread(path0 + filenames0[i])
    im1 = imageio.imread(path1 + filenames1[i])
    X_train[:, i] = np.reshape(im0, 784)/255
    X_train[:, 500 + i] = np.reshape(im1, 784)/255
    Y_train[i] = digitclass0
    Y_train[500 + i] = digitclass1
```

```python
X_test = np.zeros((784, 20))
Y_test = np.zeros(20, dtype='int64')

for i in range(500, 510):
    im0 = imageio.imread(path0 + filenames0[i])
    im1 = imageio.imread(path1 + filenames1[i])
    X_test[:, i-500] = np.reshape(im0, 784)
    X_test[:, i-490] = np.reshape(im1, 784)
    Y_test[i - 500] = digitclass0
    Y_test[i - 490]= digitclass1
```

```python
distances = np.zeros(1000)
NN = np.zeros((20, 20))
Y_kNN = np.zeros(20, dtype='int64')
for i in range(20):
    for j in range(1000):
        DIFF = X_test[:, i] - X_train[:, j]
        distances[j] = np.linalg.norm(DIFF)
    indices = np.argsort(distances)
    labels = Y_train[indices[:20]]
    bins = np.bincount(labels)
    Y_kNN[i] = np.argsort(bins)[-1]
```

```python
print('Labels determined by k-Nearest Neighbors:')
print(Y_kNN)
print('\nLabels of training set')
print(Y_test)
```

    Labels determined by k-Nearest Neighbors:
    [0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1]

    Labels of training set
    [0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1]

```python

```


# Simple Linear Regression with NumPy

In school, students are taught to draw lines like the following.

$$ y = 2 x + 1$$

They're taught to pick two values for $x$ and calculate the corresponding values for $y$ using the equation.
Then they draw a set of axes, plot the points, and then draw a line extending through the two dots on their axes.

```python
# Import matplotlib.
import matplotlib.pyplot as plt
```

```python
# Draw some axes.
plt.plot([-1, 10], [0, 0], 'k-')
plt.plot([0, 0], [-1, 10], 'k-')

# Plot the red, blue and green lines.
plt.plot([1, 1], [-1, 3], 'b:')
plt.plot([-1, 1], [3, 3], 'r:')

# Plot the two points (1,3) and (2,5).
plt.plot([1, 2], [3, 5], 'ko')
# Join them with an (extending) green lines.
plt.plot([-1, 10], [-1, 21], 'g-')

# Set some reasonable plot limits.
plt.xlim([-1, 10])
plt.ylim([-1, 10])

# Show the plot.
plt.show()
```

Simple linear regression is about the opposite problem - what if you have some points and are looking for the equation?
It's easy when the points are perfectly on a line already, but usually real-world data has some noise.
The data might still look roughly linear, but aren't exactly so.

***

## Example (contrived and simulated)

#### Scenario
Suppose you are trying to weigh your suitcase to avoid an airline's extra charges.
You don't have a weighing scales, but you do have a spring and some gym-style weights of masses 7KG, 14KG and 21KG.
You attach the spring to the wall hook, and mark where the bottom of it hangs.
You then hang the 7KG weight on the end and mark where the bottom of the spring is.
You repeat this with the 14KG weight and the 21KG weight.
Finally, you place your case hanging on the spring, and the spring hangs down halfway between the 7KG mark and the 14KG mark.
Is your case over the 10KG limit set by the airline?

#### Hypothesis
When you look at the marks on the wall, it seems that the 0KG, 7KG, 14KG and 21KG marks are evenly spaced.
You wonder if that means your case weighs 10.5KG.
That is, you wonder if there is a *linear* relationship between the distance the spring's hook is from its resting position, and the mass on the end of it.

#### Experiment
You decide to experiment.
You buy some new weights - a 1KG, a 2KG, a 3Kg, all the way up to 20KG.
You place them each in turn on the spring and measure the distance the spring moves from the resting position.
You tabulate the data and plot them.

#### Analysis
Here we'll import the Python libraries we need for or investigations below.

```python
# numpy efficiently deals with numerical multi-dimensional arrays.
import numpy as np

# matplotlib is a plotting library, and pyplot is its easy-to-use module.
import matplotlib.pyplot as plt

# This just sets the default plot size to be bigger.
plt.rcParams['figure.figsize'] = (8, 6)
```

Ignore the next couple of lines where I fake up some data. I'll use the fact that I faked the data to explain some results later. Just pretend that w is an array containing the weight values and d are the corresponding distance measurements.

```python
w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
```

```python
# Let's have a look at w.
w
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
           13., 14., 15., 16., 17., 18., 19., 20.])

```python
# Let's have a look at d.
d
```

    array([ 15.99410275,  10.9911427 ,  10.53445534,  15.97058549,
            27.44753565,  39.51180328,  43.06251898,  47.47682646,
            52.27808676,  56.67322568,  59.16156245,  56.69603504,
            67.15387974,  80.96017265,  84.50699528,  87.26011678,
            90.77740446, 100.07004239, 107.93739931, 105.51085343,
           118.07859596])

Let's have a look at the data from our experiment.

```python
# Create the plot.

plt.plot(w, d, 'k.')

# Set some properties for the plot.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

#### Model
It looks like the data might indeed be linear.
The points don't exactly fit on a straight line, but they are not far off it.
We might put that down to some other factors, such as the air density, or errors, such as in our tape measure.
Then we can go ahead and see what would be the best line to fit the data.

#### Straight lines
All straight lines can be expressed in the form $y = mx + c$.
The number $m$ is the slope of the line.
The slope is how much $y$ increases by when $x$ is increased by 1.0.
The number $c$ is the y-intercept of the line.
It's the value of $y$ when $x$ is 0.

#### Fitting the model
To fit a straight line to the data, we just must pick values for $m$ and $c$.
These are called the parameters of the model, and we want to pick the best values possible for the parameters.
That is, the best parameter values *given* the data observed.
Below we show various lines plotted over the data, with different values for $m$ and $c$.

```python
# Plot w versus d with black dots.
plt.plot(w, d, 'k.', label="Data")

# Overlay some lines on the plot.
x = np.arange(0.0, 21.0, 1.0)
plt.plot(x, 5.0 * x + 10.0, 'r-', label=r"$5x + 10$")
plt.plot(x, 6.0 * x +  5.0, 'g-', label=r"$6x +  5$")
plt.plot(x, 5.0 * x + 15.0, 'b-', label=r"$5x + 15$")

# Add a legend.
plt.legend()

# Add axis labels.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

#### Calculating the cost
You can see that each of these lines roughly fits the data.
Which one is best, and is there another line that is better than all three?
Is there a "best" line?

It depends how you define the word best.
Luckily, everyone seems to have settled on what the best means.
The best line is the one that minimises the following calculated value.

$$ \sum_i (y_i - mx_i - c)^2 $$

Here $(x_i, y_i)$ is the $i^{th}$ point in the data set and $\sum_i$ means to sum over all points.
The values of $m$ and $c$ are to be determined.
We usually denote the above as $Cost(m, c)$.

Where does the above calculation come from?
It's easy to explain the part in the brackets $(y_i - mx_i - c)$.
The corresponding value to $x_i$ in the dataset is $y_i$.
These are the measured values.
The value $m x_i + c$ is what the model says $y_i$ should have been.
The difference between  the value that was observed ($y_i$) and the value that the model gives ($m x_i + c$), is $y_i - mx_i - c$.

Why square that value?
Well note that the value could be positive or negative, and you sum over all of these values.
If we allow the values to be positive or negative, then the positive could cancel the negatives.
So, the natural thing to do is to take the absolute value $\mid y_i - m x_i - c \mid$.
Well it turns out that absolute values are a pain to deal with, and instead it was decided to just square the quantity instead, as the square of a number is always positive.
There are pros and cons to using the square instead of the absolute value, but the square is used.
This is usually called *least squares* fitting.

```python
# Calculate the cost of the lines above for the data above.
cost = lambda m,c: np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)])

print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 10.0, cost(5.0, 10.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (6.0,  5.0, cost(6.0,  5.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 15.0, cost(5.0, 15.0)))
```

    Cost with m =  5.00 and c = 10.00:   567.04
    Cost with m =  6.00 and c =  5.00:  1073.18
    Cost with m =  5.00 and c = 15.00:   911.51

#### Minimising the cost
We want to calculate values of $m$ and $c$ that give the lowest value for the cost value above.
For our given data set we can plot the cost value/function.
Recall that the cost is:

$$ Cost(m, c) = \sum_i (y_i - mx_i - c)^2 $$

This is a function of two variables, $m$ and $c$, so a plot of it is three dimensional.
See the **Advanced** section below for the plot.

In the case of fitting a two-dimensional line to a few data points, we can easily calculate exactly the best values of $m$ and $c$.
Some of the details are discussed in the **Advanced** section, as they involve calculus, but the resulting code is straight-forward.
We first calculate the mean (average) values of our $x$ values and that of our $y$ values.
Then we subtract the mean of $x$ from each of the $x$ values, and the mean of $y$ from each of the $y$ values.
Then we take the *dot product* of the new $x$ values and the new $y$ values and divide it by the dot product of the new $x$ values with themselves.
That gives us $m$, and we use $m$ to calculate $c$.

Remember that in our dataset $x$ is called $w$ (for weight) and $y$ is called $d$ (for distance).
We calculate $m$ and $c$ below.

```python
# Calculate the best values for m and c.

# First calculate the means (a.k.a. averages) of w and d.
w_avg = np.mean(w)
d_avg = np.mean(d)

# Subtract means from w and d.
w_zero = w - w_avg
d_zero = d - d_avg

# The best m is found by the following calculation.
m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
# Use m from above to calculate the best c.
c = d_avg - m * w_avg

print("m is %8.6f and c is %6.6f." % (m, c))
```

    m is 5.395020 and c is 6.909486.

Note that numpy has a function that will perform this calculation for us, called polyfit.
It can be used to fit lines in many dimensions.

```python
np.polyfit(w, d, 1)
```

    array([5.39501974, 6.90948551])

#### Best fit line
So, the best values for $m$ and $c$ given our data and using least squares fitting are about $4.95$ for $m$ and about $11.13$ for $c$.
We plot this line on top of the data below.

```python
# Plot the best fit line.
plt.plot(w, d, 'k.', label='Original data')
plt.plot(w, m * w + c, 'b-', label='Best fit line')

# Add axis labels and a legend.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')
plt.legend()

# Show the plot.
plt.show()
```

Note that the $Cost$ of the best $m$ and best $c$ is not zero in this case.

```python
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (m, c, cost(m, c)))
```

    Cost with m =  5.40 and c =  6.91:   431.37

### Summary
In this notebook we:
1. Investigated the data.
2. Picked a model.
3. Picked a cost function.
4. Estimated the model parameter values that minimised our cost function.

### Advanced
In the following sections we cover some of the more advanced concepts involved in fitting the line.

#### Simulating data
Earlier in the notebook we glossed over something important: we didn't actually do the weighing and measuring - we faked the data.
A better term for this is *simulation*, which is an important tool in research, especially when testing methods such as simple linear regression.

We ran the following two commands to do this:

```python
w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
```

The first command creates a numpy array containing all values between 1.0 and 21.0 (including 1.0 but not including 21.0) in steps of 1.0.

```python
 np.arange(0.0, 21.0, 1.0)
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
           13., 14., 15., 16., 17., 18., 19., 20.])

The second command is more complex.
First it takes the values in the `w` array, multiplies each by 5.0 and then adds 10.0.

```python
5.0 * w + 10.0
```

    array([ 10.,  15.,  20.,  25.,  30.,  35.,  40.,  45.,  50.,  55.,  60.,
            65.,  70.,  75.,  80.,  85.,  90.,  95., 100., 105., 110.])

It then adds an array of the same length containing random values.
The values are taken from what is called the normal distribution with mean 0.0 and standard deviation 5.0.

```python
np.random.normal(0.0, 5.0, w.size)
```

    array([ 2.77924728, -7.36852008, -1.37782612,  9.98075174,  4.21995499,
           -2.60231604,  9.32075467,  2.83510953, -9.14550022,  6.52177982,
           11.11216945,  6.69576981,  6.84188875, -3.55449951,  6.14654406,
            0.97382891, -2.89697444,  3.99446255, -1.07846657, -9.81654276,
           -3.02725231])

The normal distribution follows a bell shaped curve.
The curve is centred on the mean (0.0 in this case) and its general width is determined by the standard deviation (5.0 in this case).

```python
# Plot the normal distrution.
normpdf = lambda mu, s, x: (1.0 / (2.0 * np.pi * s**2)) * np.exp(-((x - mu)**2)/(2 * s**2))

x = np.linspace(-20.0, 20.0, 100)
y = normpdf(0.0, 5.0, x)
plt.plot(x, y)

plt.show()
```

The idea here is to add a little bit of randomness to the measurements of the distance.
The random values are entered around 0.0, with a greater than 99% chance they're within the range -15.0 to 15.0.
The normal distribution is used because of the [Central Limit Theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) which basically states that when a bunch of random effects happen together the outcome looks roughly like the normal distribution. (Don't quote me on that!)

#### Plotting the cost function
We can plot the cost function for a given set of data points.
Recall that the cost function involves two variables: $m$ and $c$, and that it looks like this:

$$ Cost(m,c) = \sum_i (y_i - mx_i - c)^2 $$

To plot a function of two variables we need a 3D plot.
It can be difficult to get the viewing angle right in 3D plots, but below you can just about make out that there is a low point on the graph around the $(m, c) = (\approx 5.0, \approx 10.0)$ point.

```python
# This code is a little bit involved - don't worry about it.
# Just look at the plot below.

from mpl_toolkits.mplot3d import Axes3D

# Ask pyplot a 3D set of axes.
ax = plt.figure().gca(projection='3d')

# Make data.
mvals = np.linspace(4.5, 5.5, 100)
cvals = np.linspace(0.0, 20.0, 100)

# Fill the grid.
mvals, cvals = np.meshgrid(mvals, cvals)

# Flatten the meshes for convenience.
mflat = np.ravel(mvals)
cflat = np.ravel(cvals)

# Calculate the cost of each point on the grid.
C = [np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)]) for m, c in zip(mflat, cflat)]
C = np.array(C).reshape(mvals.shape)

# Plot the surface.
surf = ax.plot_surface(mvals, cvals, C)

# Set the axis labels.
ax.set_xlabel('$m$', fontsize=16)
ax.set_ylabel('$c$', fontsize=16)
ax.set_zlabel('$Cost$', fontsize=16)

# Show the plot.
plt.show()
```

#### Coefficient of determination
Earlier we used a cost function to determine the best line to fit the data.
Usually the data do not perfectly fit on the best fit line, and so the cost is greater than 0.
A quantity closely related to the cost is the *coefficient of determination*, also known as the *R-squared* value.
The purpose of the R-squared value is to measure how much of the variance in $y$ is determined by $x$.

For instance, in our example the main thing that affects the distance the spring is hanging down is the weight on the end.
It's not the only thing that affects it though.
The room temperature and density of the air at the time of measurment probably affect it a little.
The age of the spring, and how many times it has been stretched previously probably also have a small affect.
There are probably lots of unknown factors affecting the measurment.

The R-squared value estimates how much of the changes in the $y$ value is due to the changes in the $x$ value compared to all of the other factors affecting the $y$ value.
It is calculated as follows:

$$ R^2 = 1 - \frac{\sum_i (y_i - m x_i - c)^2}{\sum_i (y_i - \bar{y})^2} $$

Note that sometimes the [*Pearson correlation coefficient*](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used instead of the R-squared value.
You can just square the Pearson coefficient to get the R-squred value.

```python
# Calculate the R-squared value for our data set.
rsq = 1.0 - (np.sum((d - m * w - c)**2)/np.sum((d - d_avg)**2))

print("The R-squared value is %6.4f" % rsq)
```

    The R-squared value is 0.9811

```python
# The same value using numpy.
np.corrcoef(w, d)[0][1]**2
```

    0.9811159849088361

#### The minimisation calculations
Earlier we used the following calculation to calculate $m$ and $c$ for the line of best fit.
The code was:

```python
w_zero = w - np.mean(w)
d_zero = d - np.mean(d)

m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
c = np.mean(d) - m * np.mean(w)
```

In mathematical notation we write this as:

$$ m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \qquad \textrm{and} \qquad c = \bar{y} - m \bar{x} $$

where $\bar{x}$ is the mean of $x$ and $\bar{y}$ that of $y$.

Where did these equations come from?
They were derived using calculus.
We'll give a brief overview of it here, but feel free to gloss over this section if it's not for you.
If you can understand the first part, where we calculate the partial derivatives, then great!

The calculations look complex, but if you know basic differentiation, including the chain rule, you can easily derive them.
First, we differentiate the cost function with respect to $m$ while treating $c$ as a constant, called a partial derivative.
We write this as $\frac{\partial m}{ \partial Cost}$, using $\delta$ as opposed to $d$ to signify that we are treating the other variable as a constant.
We then do the same with respect to $c$ while treating $m$ as a constant.
We set both equal to zero, and then solve them as two simultaneous equations in two variables.

###### Calculate the partial derivatives
$$
\begin{align}
Cost(m, c)                       &= \sum_i (y_i - mx_i - c)^2      \\[1cm]
\frac{\partial Cost}{\partial m} &= \sum 2(y_i - m x_i -c)(-x_i) \\
                                 &= -2 \sum x_i (y_i - m x_i -c) \\[0.5cm]
\frac{\partial Cost}{\partial c} & = \sum 2(y_i - m x_i -c)(-1) \\
                                 & = -2 \sum (y_i - m x_i -c)   \\
\end{align}
$$

###### Set to zero
$$
\begin{align}
& \frac{\partial Cost}{\partial m}                         = 0 \\[0.2cm]
& \Rightarrow -2 \sum x_i (y_i - m x_i -c)                 = 0 \\
& \Rightarrow \sum (x_i y_i - m x_i x_i - x_i c)           = 0 \\
& \Rightarrow \sum x_i y_i - \sum_i m x_i x_i - \sum x_i c = 0 \\
& \Rightarrow m \sum x_i x_i                               = \sum x_i y_i - c \sum x_i \\[0.2cm]
& \Rightarrow m                                            = \frac{\sum x_i y_i - c \sum x_i}{\sum x_i x_i} \\[0.5cm]
& \frac{\partial Cost}{\partial c}               = 0 \\[0.2cm]
& \Rightarrow -2 \sum (y_i - m x_i - c)          = 0 \\
& \Rightarrow \sum y_i - \sum_i m x_i - \sum c   = 0 \\
& \Rightarrow \sum y_i - m \sum_i x_i            = c \sum 1 \\
& \Rightarrow c = \frac{\sum y_i - m \sum x_i}{\sum 1} \\
& \Rightarrow c = \frac{\sum y_i}{\sum 1} - m \frac{\sum x_i}{\sum 1} \\[0.2cm]
& \Rightarrow c = \bar{y} - m \bar{x} \\
\end{align}
$$

###### Solve the simultaneous equations
Here we let $n$ be the length of $x$, which is also the length of $y$.

$$
\begin{align}
& m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum x_i y_i - (\bar{y} - m \bar{x}) \sum x_i}{\sum x_i x_i} \\
& \Rightarrow m \sum x_i x_i = \sum x_i y_i - \bar{y} \sum x_i + m \bar{x} \sum x_i \\
& \Rightarrow m \sum x_i x_i - m \bar{x} \sum x_i = \sum x_i y_i - \bar{y} \sum x_i \\[0.3cm]
& \Rightarrow m = \frac{\sum x_i y_i - \bar{y} \sum x_i}{\sum x_i x_i - \bar{x} \sum x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x} - n \bar{y} \bar{x} + n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x} - n \bar{x} \bar{x} + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - \sum y_i \bar{x} - \sum \bar{y} x_i + n \bar{y} \bar{x}}{\sum (x_i x_i) - \sum x_i \bar{x} - \sum \bar{x} x_i + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \\
\end{align}
$$

<br>

#### Using sklearn neural networks

***

```python
import sklearn.neural_network as sknn

# Expects a 2D array of inputs.
w2d = w.reshape(-1, 1)

# Train the neural network.
regr = sknn.MLPRegressor(max_iter=10000).fit(w2d, d)

# Show the predictions.
np.array([d, regr.predict(w2d)]).T
```

    array([[ 15.99410275,  15.99136402],
           [ 10.9911427 ,  10.99025661],
           [ 10.53445534,   9.83175482],
           [ 15.97058549,  18.21173347],
           [ 27.44753565,  26.74674379],
           [ 39.51180328,  35.26744107],
           [ 43.06251898,  40.53315812],
           [ 47.47682646,  45.79886552],
           [ 52.27808676,  51.06457291],
           [ 56.67322568,  56.33028031],
           [ 59.16156245,  61.5959877 ],
           [ 56.69603504,  66.86169509],
           [ 67.15387974,  72.12740249],
           [ 80.96017265,  77.39310988],
           [ 84.50699528,  82.65881728],
           [ 87.26011678,  87.92452467],
           [ 90.77740446,  93.19023207],
           [100.07004239,  98.45593946],
           [107.93739931, 103.72164686],
           [105.51085343, 108.98735425],
           [118.07859596, 114.25306165]])

```python
# The score.
regr.score(w2d, d)
```

    0.9895666755071412

#### End


---
# Example of classification using two gaussian distributions.
---

In this example, we generate a global probability distribution function (PDF) that is the sum of two gaussian PDF.
Each one is weighted with its a priori class probability $P(C_{i})$:

  $P(\bf{x}) = P(\bf{x}|C_{1}) P(C_{1}) + P(\bf{x}|C_{2}) P(C_{2})$

Our goal is to locate the influence zone of each class i. This corresponds to the zone where

 $P(\bf{x}|C_{i}) P(C_{i}) > P(\bf{x}|C_{j}) P(C_{j})$

## The 2-D gausian PDF

In its matrix-form, the equation for the 2-D gausian PDF reads like this:

  $P(\bf{x}) = \frac{1}{2\pi |\Sigma|^{0.5}} \exp{[-\frac{1}{2}(\bf{x}-\bf{\mu})^\top \Sigma^{-1} (\bf{x}-\bf{\mu})]}$

where

$
\begin{align}
\bf{x} &= [x_{1} x_{2}]^\top \\
\bf{\mu} &= [\mu_{1} \mu_{2}]^\top  \\
\Sigma &= \begin{pmatrix} \sigma_{x_{1}}^2 & \rho\sigma_{x_{1}}\sigma_{x_{2}} \\ \rho\sigma_{x_{1}}\sigma_{x_{2}} & \sigma_{x_{2}}^2 \end{pmatrix}
\end{align}
$

where $\rho$ is the correlation factor between the $x_{1}$ and $x_{2}$ data.

## A useful trick to generate a covariance matrix $\Sigma$ with desired features

Instead of guessing the values of $\sigma_{x_{1}}$, $\sigma_{x_{2}}$ and $\rho$ to create a
covariance matrix $\Sigma$ for visualization purpose, we can design it with some desired characteristics.
Those are the principal axis variances $\sigma_{1}^2$ and $\sigma_{2}^2$, and the rotation angle $\theta$.

First, we generate the covariance matrix of a correlated PDF with its principal axes oriented along the
$x_{1}$ and $x_{2}$ axes.

  $\Sigma_{PA} = \begin{pmatrix} \sigma_{1}^2 & 0 \\ 0 & \sigma_{2}^2 \end{pmatrix}$

Next, we generate the rotation matrix for the angle $\theta$:

  $R = \begin{pmatrix} \cos{\theta}  & -\sin{\theta} \\ \sin{\theta} & \cos{\theta} \end{pmatrix}$

The covariance matrix we are looking for is

  $\Sigma = R \Sigma_{PA} R^\top $

Hence, we only have to specify the values of $\sigma_{1}$ and $\sigma_{2}$ along principal axes and the
rotation angle $\theta$. The correlation coefficient $\rho$ depends on the value of $\theta$:

  $\rho>0$ when $\theta>0$

  $\rho<0$ when $\theta<0$

N.B. For ease of reading, we use below the variables x and y instead of x1 and x2. The final results are shown with x1 and x2.

```python
print(__doc__)

# Author: Pierre Gravel <pierre.gravel@iid.ulaval.ca>
# License: BSD

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import colors
from matplotlib import cm
from matplotlib.ticker import NullFormatter
from mpl_toolkits.mplot3d import Axes3D

from scipy import stats
from scipy.stats import multivariate_normal

from skimage.transform import resize

import seaborn as sns
sns.set(color_codes=True)

# Used for reproductibility of the results
np.random.seed(43)
```

    Automatically created module for IPython interactive environment

Here are the characteristics of the 2-D PDF we want to generate:

```python
# Origins of the PDF
Mu = np.zeros((2,2))
Mu[0,:] = [6., 3.]
Mu[1,:] = [6., 7.]

# Standard deviations along the x1 and x2 directions (or x and y)
sigma = np.zeros((2,2))
sigma[0,:] = [2., 0.7]
sigma[1,:] = [1.2, 0.7]

# Rotation angles for the principal axis
theta = np.array([-30., 0.])

# A priori class probabilities
prob_C = np.array([0.5, 0.5])

```

Compute the covariance matrix $\Sigma$

```python
Sigma = np.zeros((2,2,2))
for i in range(2):
    angle = np.radians(theta[i])
    c, s = np.cos(angle), np.sin(angle)

    # Rotation matrix
    R = np.array(((c, -s), (s, c)))

    # Covariance matrix for a PDF with its principal axes oriented along the x and y directions
    sig = np.array([[sigma[i,0]**2, 0.],[0., sigma[i,1]**2]])

    # Covariance matrix after rotation
    Sigma[:,:,i] = R.dot( sig.dot(R.T) )

```

## Global PDF computation
Generate a spatial grid where the PDF will be evaluated locally.

```python
x_min, x_max = 0., 10.
y_min, y_max = 0., 10.
nx, ny = 300, 300
x = np.linspace(x_min, x_max, nx)
y = np.linspace(y_min, y_max, ny)
xx, yy = np.meshgrid(x,y)
pos = np.dstack((xx, yy))

```

Compute the global PDF

```python
# Weighted PDF
model = multivariate_normal(Mu[0,:], Sigma[:,:,0])
pdf0 = prob_C[0]*model.pdf(pos)

model = multivariate_normal(Mu[1,:], Sigma[:,:,1])
pdf1 = prob_C[1]*model.pdf(pos)

# Global PDF
pdf = pdf0 + pdf1

# Define a mask to identify the influence zone of each class: Mask=False for class 0 and Mask=True for class 1
mask = pdf1 > pdf0

```

## Display the global PDF

```python
z_offset = -0.1
view=[22., -30.]

fig = plt.figure(figsize = (15,10))
ax = fig.gca(projection='3d')

# ---------Generate overlying 3-D surface with colors indicating the influence zone of each class
rstride, cstride = 5, 5

s = ax.plot_surface(xx, yy, pdf, rstride=rstride, cstride=cstride, linewidth=.5, antialiased=True, color='gray', edgecolors='k')
a1 = s.__dict__['_original_facecolor']
b1 = s.__dict__['_facecolors']
c1 = s.__dict__['_facecolors3d']

s = ax.plot_surface(xx, yy, pdf, rstride=rstride, cstride=cstride, linewidth=.5, antialiased=True, color='w', edgecolors='k')
a2 = s.__dict__['_original_facecolor']
b2 = s.__dict__['_facecolors']
c2 = s.__dict__['_facecolors3d']

Lx = int(nx/rstride)
Ly = int(ny/cstride)

mask2 = resize(mask, (Lx,Ly), order=0)
indx = np.argwhere(mask2)
idx = indx[:,0]*Lx + indx[:,1]

a = a1
b = b1
c = c1
for i in idx:
    a[i,:] = a2[i,:]
    b[i,:] = b2[i,:]
    c[i,:] = c2[i,:]

s.__dict__['_original_facecolor'] = a
s.__dict__['_facecolors'] = b
s.__dict__['_facecolors3d'] = c

# --------- Generate underlying contours with borders between each class
cset = ax.contourf(xx, yy, pdf, zdir='z', offset=z_offset, cmap='viridis')
ax.contour(xx, yy, mask, [0.5], offset=z_offset, linewidths=2., colors='white')

# N.B. For convenience, we use classes 1 and 2 instead of classes 0 and 1 in the figure
ax.text(2, 1, z_offset,'$\mathcal{R}_{1}$', zdir='y', color='w',fontsize=18)
ax.text(9, 8, z_offset,'$\mathcal{R}_{2}$', zdir='y', color='w',fontsize=18)
ax.text(6, 2.5, 0.05, '$p(\mathbf{X}|C_{1})p(C_{1})$', zdir='y', horizontalalignment='right', verticalalignment='bottom',
        color='k',fontsize=18)
ax.text(6, 6., 0.08, '$p(\mathbf{X}|C_{2})p(C_{2})$', zdir='y', horizontalalignment='right', verticalalignment='bottom',
        color='gray',fontsize=18)

ax.set_zlim(z_offset, np.max(pdf))
ax.view_init(view[0], view[1])

ax.set_ylabel('$x_{2}$', fontsize=18)
ax.xaxis.set_rotate_label(False)
ax.set_xlabel('$x_{1}$', rotation=10, fontsize=18)
ax.set_zlabel('$Z$', rotation=0, fontsize=18)
ax.set_title('$Z = PDF(x_{1},x_{2})$', fontsize=20)

fig.tight_layout()

plt.savefig('Influence_zones_in_gaussian_PDF_classification.png')
plt.savefig('Influence_zones_in_gaussian_PDF_classification.pdf')

plt.show()

```

```python

```


We have a square matrix $R$.  We consider the error for $T = R'R$ where $R'$ is the transpose of $R$.

The elements of $R$ are $r_{i,j}$, where $i = 1 \dots N, j = 1 \dots N$.

$r_{i, *}$ is row $i$ of $R$.

Now let $R$ be a rotation matrix.  $T$ at infinite precision will be the identity matrix $I$

Assume the maximum error in the specification of values $r_{i, j}$ is constant, $\delta$. That is, any floating point value $r_{i, j}$ represents an infinite precision value between $r_{i, j} \pm \delta$

```
from sympy import Symbol, symarray, Matrix, matrices, simplify, nsimplify
R = Matrix(symarray('r', (3,3)))
R
```

    [r_0_0, r_0_1, r_0_2]
    [r_1_0, r_1_1, r_1_2]
    [r_2_0, r_2_1, r_2_2]

```
T = R.T * R
T
```

    [         r_0_0**2 + r_1_0**2 + r_2_0**2, r_0_0*r_0_1 + r_1_0*r_1_1 + r_2_0*r_2_1, r_0_0*r_0_2 + r_1_0*r_1_2 + r_2_0*r_2_2]
    [r_0_0*r_0_1 + r_1_0*r_1_1 + r_2_0*r_2_1,          r_0_1**2 + r_1_1**2 + r_2_1**2, r_0_1*r_0_2 + r_1_1*r_1_2 + r_2_1*r_2_2]
    [r_0_0*r_0_2 + r_1_0*r_1_2 + r_2_0*r_2_2, r_0_1*r_0_2 + r_1_1*r_1_2 + r_2_1*r_2_2,          r_0_2**2 + r_1_2**2 + r_2_2**2]

Now the same result with error $\delta$ added to each element

```
d = Symbol('d')
E = matrices.ones((3,3)) * d
RE = R + E
RE
```

    [d + r_0_0, d + r_0_1, d + r_0_2]
    [d + r_1_0, d + r_1_1, d + r_1_2]
    [d + r_2_0, d + r_2_1, d + r_2_2]

Calculate the result $T$ with error

```
TE = RE.T * RE
TE
```

    [                           (d + r_0_0)**2 + (d + r_1_0)**2 + (d + r_2_0)**2, (d + r_0_0)*(d + r_0_1) + (d + r_1_0)*(d + r_1_1) + (d + r_2_0)*(d + r_2_1), (d + r_0_0)*(d + r_0_2) + (d + r_1_0)*(d + r_1_2) + (d + r_2_0)*(d + r_2_2)]
    [(d + r_0_0)*(d + r_0_1) + (d + r_1_0)*(d + r_1_1) + (d + r_2_0)*(d + r_2_1),                            (d + r_0_1)**2 + (d + r_1_1)**2 + (d + r_2_1)**2, (d + r_0_1)*(d + r_0_2) + (d + r_1_1)*(d + r_1_2) + (d + r_2_1)*(d + r_2_2)]
    [(d + r_0_0)*(d + r_0_2) + (d + r_1_0)*(d + r_1_2) + (d + r_2_0)*(d + r_2_2), (d + r_0_1)*(d + r_0_2) + (d + r_1_1)*(d + r_1_2) + (d + r_2_1)*(d + r_2_2),                            (d + r_0_2)**2 + (d + r_1_2)**2 + (d + r_2_2)**2]

Subtract the true result to get the absolute error

```
TTE = TE-T
TTE.simplify()
TTE
```

    [                  d*(3*d + 2*r_0_0 + 2*r_1_0 + 2*r_2_0), d*(3*d + r_0_0 + r_0_1 + r_1_0 + r_1_1 + r_2_0 + r_2_1), d*(3*d + r_0_0 + r_0_2 + r_1_0 + r_1_2 + r_2_0 + r_2_2)]
    [d*(3*d + r_0_0 + r_0_1 + r_1_0 + r_1_1 + r_2_0 + r_2_1),                   d*(3*d + 2*r_0_1 + 2*r_1_1 + 2*r_2_1), d*(3*d + r_0_1 + r_0_2 + r_1_1 + r_1_2 + r_2_1 + r_2_2)]
    [d*(3*d + r_0_0 + r_0_2 + r_1_0 + r_1_2 + r_2_0 + r_2_2), d*(3*d + r_0_1 + r_0_2 + r_1_1 + r_1_2 + r_2_1 + r_2_2),                   d*(3*d + 2*r_0_2 + 2*r_1_2 + 2*r_2_2)]

```
TTE[0,0], TTE[1,1], TTE[2,2]
```

    (d*(3*d + 2*r_0_0 + 2*r_1_0 + 2*r_2_0),
     d*(3*d + 2*r_0_1 + 2*r_1_1 + 2*r_2_1),
     d*(3*d + 2*r_0_2 + 2*r_1_2 + 2*r_2_2))

Assuming $\delta$ is small ($\delta^2$ is near zero) then the diagonal values $TTE_{k, k}$ are approximately $2\delta \Sigma_i r_{i, k}$

$\Sigma_i r_{i, k}$ is the column sum for column $k$ of $R$.  We know that the column $L^2$ norms of $R$ are each 1.  We know $\|x\|_1 \leq \sqrt{n}\|x\|_2$ - http://en.wikipedia.org/wiki/Lp_space. Therefore the column sums must be $\le \sqrt{N}$.  Therefore the maximum error for the diagonal of $T$ is $\sqrt{N} 2\delta$.

More generally, the elements $k, m$ of $TTE$ are approximately $\delta (\Sigma_i{r_{i, k}} + \Sigma_i{r_{i, m}})$

So the error for each of the elements of $TTE$ is also bounded by $\sqrt{N} 2\delta$.

Now consider the floating point calculation error.  This depends on the floating point representation we use for the calculations. Let $\epsilon = x-1$ where $x$ is the smallest number greater than 1 that is representable in our floating point format (see http://matthew-brett.github.com/pydagogue/floating_error.html). The largest error for a calculation resulting in a value near 1 is $\frac{\epsilon}{2}$. For the diagonal values, the calculation error will be the error for the $r_{i,*} r_{i, *}'$ dot product.  This comprises $N$ scalar products with results each bounded by 1 ($r_{i, j} r_{i, j}$) followed by $N-1$ sums each bounded by 1.  Maximum error is therefore $(2N-1) \frac{\epsilon}{2}$ = $\frac{5}{2} \epsilon$ where $N=3$.

For the off-diagonal values $T_{k, m}$, we have the $r_{k,*} r_{m, *}'$ dot product.  Because $R$ is a rotation matrix, by definition this result must be zero.

Because the column and row $L^2$ norms of $R$ are 1, the values in $R$ cannot be greater than 1.  Therefore $r_{k,*} r_{m, *}'$ consists of the $N$ products with results each bounded by 1 ($r_{k, j} r_{m, j}$) followed by $N-2$ sums each bounded by 1 (the last of the sums must be approximately 0).  Maximum error is therefore $(2N-2) \frac{\epsilon}{2}$ = $2\epsilon$ where $N=3$.

So, assuming an initial error of $\delta$ per element, and $N=3$, the maximum error for diagonal elements is $\sqrt{3} 2 \delta + \frac{5}{3} \epsilon$. For the off-diagonal elements it is $\sqrt{3} 2 \delta + 2 \epsilon$.


# Simplification and Evaluation of mathematical expressions using Sympy

**Question 1**

\begin{align}
\frac{2x^{12}}{5y^{10}}
\end{align}

Solve for (a) $x = 2$ , $y = 2$  (b) Case 2 : $x = 1$ , $y = 1$

**Evaluating the expression using Sympy**

```python
from sympy import*

x = Symbol('x')
y = Symbol('y')

expression = "(2*x**12)/(5*y**10)"

simplified_expression = simplify(expression)
print("The simplified expression is {}". format(simplified_expression))

evaluated_result_a = simplified_expression.subs({x:2, y:2})
evaluated_result_b = simplified_expression.subs({x:1, y:1})

print("The Evaluated result for (a) is: {}".format(evaluated_result_a))
print("The Evaluated result for (b) is: {}".format(evaluated_result_b))
```

    The simplified expression is 2*x**12/(5*y**10)
    The Evaluated result for (a) is: 8/5
    The Evaluated result for (b) is: 2/5

**Question 2**

\begin{align}35
\left (\frac{4x}{5}\right)
\end{align}

Solve for (a) $x = 1$ and (b) $x = 5$

```python
from sympy import*

x = Symbol('x')

expression = "35*(4*x)/5"

simplified_expression = simplify(expression)
print("The simplified expression is {}". format(simplified_expression))

evaluated_result_a = simplified_expression.subs({x:1})
evaluated_result_b = simplified_expression.subs({x:5})

print("The Evaluated result for (a) is: {}".format(evaluated_result_a))
print("The Evaluated result for (b) is: {}".format(evaluated_result_b))

```

    The simplified expression is 28*x
    The Evaluated result for (a) is: 28
    The Evaluated result for (b) is: 140

**Question 3**

\begin{align}24
\left(\frac{y}{6}+ \frac{3}{8} \right)
\end{align}

Solve for (a) $y = 2$ and (b) $y = 1$

```python
from sympy import*

y = Symbol('y')

expression = "24*((y/6)+(3/8))"
simplified_expression = simplify(expression)

print("The simplified expression is {}". format(simplified_expression))

evaluated_result_a = simplified_expression.subs({y:2})
evaluated_result_b = simplified_expression.subs({y:1})

print("The Evaluated result for (a) is: {}".format(evaluated_result_a))
print("The Evaluated result for (b) is: {}".format(evaluated_result_b))

```

    The simplified expression is 4*y + 9
    The Evaluated result for (a) is: 17
    The Evaluated result for (b) is: 13

**Question 4**

Calculate the pressure P of the gas using Van der Waal’s Equation for real gas

\begin{align}P =
\left(\frac{RT}{V-b} - \frac{a}{V^2}\right)
\end{align}

1. Average attraction between particles (a) = 1.360
2. volume excluded by a mole of particles (b) = 0.03186
3. Universal Gas constant (R) = 8.31
4. Volume of gas(V) = 5
5. Temperature(T) = 275

```python
from sympy import*

a = Symbol('a')
b = Symbol('b')
R = Symbol('R')
V = Symbol('V')
T = Symbol('T')

expression = "(((R*T)/(V-b))-(a/V**2))"
simplified_expression = simplify(expression)
print("The simplified expression is: {}". format(simplified_expression))

evaluated_result = simplified_expression.subs({a:1.360, b:0.03186, R:8.31, V:5, T:275})
print("The solution of Van der Waal’s Equation for real gas under the given conditions is: {}".format(evaluated_result))
```

    The simplified expression is: (R*T*V**2 - a*(V - b))/(V**2*(V - b))
    The solution of Van der Waal’s Equation for real gas under the given conditions is: 459.926598925151


```python
%matplotlib inline

# Using gradient descent for linear regression
# Ideas from https://spin.atomicobject.com/2014/06/24/gradient-descent-linear-regression/

# We will attempt to predict the college admission test score based
# on the high school math score (following on Chapter 3 of "Doing Math with Python")

# Known data
x_data = [83, 85, 84, 96, 94, 86, 87, 97, 97, 85]
y_data = [85, 87, 86, 97, 96, 88, 89, 98, 98, 87]

from sympy import Symbol, Derivative
import matplotlib.pyplot as plt

# Assumed linear model
# x = math score in high school
# y = admission test score

# y = m*x + c
def estimate_y(x, m, c):
    y_estimated = m*x + c
    return y_estimated

def estimate_theta(m_current, c_current, max_iterations=50000):
    learning_rate = 0.0001
    m_gradient = 0
    c_gradient = 0
    N = len(x_data)

    m = Symbol('m')
    c = Symbol('c')
    y = Symbol('y')
    x = Symbol('x')
    # Error term
    error_term = (y - (m*x+c))**2
    # Error function = 1/n*sum(error_term)
    for i in range(max_iterations):
        for i in range(0, N):
            m_gradient += (1/N)*Derivative(error_term, m).doit().subs({x:x_data[i], y:y_data[i], m:m_current, c:c_current})
            c_gradient += (1/N)*Derivative(error_term, c).doit().subs({x:x_data[i], y:y_data[i], m:m_current, c:c_current})

        m_new = m_current - (learning_rate * m_gradient)
        c_new = c_current - (learning_rate * c_gradient)
        if abs(m_new - m_current) < 1e-5 or abs(c_new - c_current) < 1e-5:
            break
        else:
            m_current = m_new
            c_curret = c_new
    return m_new, c_new

m, c = estimate_theta(1, 1)

# Let's try and unknown set of data
# This data set is different and widely spread,
# but they are very similarly correlated
x_data = [63, 61, 98, 76, 74, 59, 40, 87, 71, 75]
y_data = [65, 62, 99, 78, 75, 60, 42, 89, 71, 77]

y_estimated = [estimate_y(x, m, c) for x in x_data]
plt.plot(x_data, y_data, 'ro')
plt.plot(x_data, y_estimated, 'b*')
plt.legend(['Actual', 'Estimated'], loc='best')
plt.show()

```

```python

```


```python
from sympy import *
init_printing(use_latex='mathjax')
```

```python
def get_diff(expressions, symbols):
    rows = len(expressions)
    columns = len(symbols)
    assert rows == columns , "Number of expression doesnt match number of symbols"

    print("Expressions:")
    for expression in expressions:
        display(expression)

    results = [[0 for x in range(rows)] for y in range(columns)]
    for row, expression in enumerate(expressions):
        for column, symbol in enumerate(symbols):
#             print('Row %d, column %d, expression: %s, symbol: %s' % (row, column, expression, symbol))
            df = diff(expression, symbol)
#             print("DF: %s" % df)
            results[row][column] = df
    return results
```

```python
x, y = symbols('x y')
get_diff([x ** 2 - y**2, 2 * x * y], [x, y])
```

    Expressions:

$$x^{2} - y^{2}$$

$$2 x y$$

$$\left [ \left [ 2 x, \quad - 2 y\right ], \quad \left [ 2 y, \quad 2 x\right ]\right ]$$

```python
x, y, z = symbols('x y z')
get_diff([
    2 * x + 3 * y,
    cos(x) * sin(z),
    exp(x) * exp(y) * exp(z)
], [x, y , z])
```

    Expressions:

$$2 x + 3 y$$

$$\sin{\left (z \right )} \cos{\left (x \right )}$$

$$e^{x} e^{y} e^{z}$$

$$\left [ \left [ 2, \quad 3, \quad 0\right ], \quad \left [ - \sin{\left (x \right )} \sin{\left (z \right )}, \quad 0, \quad \cos{\left (x \right )} \cos{\left (z \right )}\right ], \quad \left [ e^{x} e^{y} e^{z}, \quad e^{x} e^{y} e^{z}, \quad e^{x} e^{y} e^{z}\right ]\right ]$$

```python
x, y, a, b, c, d = symbols('x y a b c d')
get_diff([
    a * x + b * y,
    c * x + d * y
], [x, y])
```

    Expressions:

$$a x + b y$$

$$c x + d y$$

$$\left [ \left [ a, \quad b\right ], \quad \left [ c, \quad d\right ]\right ]$$

```python
def jacobian_at(jacobian, point):
    rows = len(jacobian)
    columns = len(jacobian[0])
    results = [[0 for x in range(rows)] for y in range(columns)]
    for rowIndex, row in enumerate(jacobian):
        for colIndex, cell in enumerate(row):
            results[rowIndex][colIndex] = cell.evalf(subs=point)

    return results
```

```python
x, y, z = symbols('x y z')
J = get_diff([
    9 * x ** 2 * y ** 2 + z * exp(x),
    x * y + x ** 2 * y ** 3 + 2 * z,
    cos(x) * sin(z) * exp(y)
], [x, y, z])
jacobian_at(J, {x: 0, y:0, z:0})
```

    Expressions:

$$9 x^{2} y^{2} + z e^{x}$$

$$x^{2} y^{3} + x y + 2 z$$

$$e^{y} \sin{\left (z \right )} \cos{\left (x \right )}$$

$$\left [ \left [ 0, \quad 0, \quad 1.0\right ], \quad \left [ 0, \quad 0, \quad 2.0\right ], \quad \left [ 0, \quad 0, \quad 1.0\right ]\right ]$$

```python
r, theta, phi = symbols('r theta phi')
get_diff([
    r * cos(theta) * sin(phi),
    r * sin(theta) * sin(phi),
    r * cos(phi)
], [r, theta, phi])
```

    Expressions:

$$r \sin{\left (\phi \right )} \cos{\left (\theta \right )}$$

$$r \sin{\left (\phi \right )} \sin{\left (\theta \right )}$$

$$r \cos{\left (\phi \right )}$$

$$\left [ \left [ \sin{\left (\phi \right )} \cos{\left (\theta \right )}, \quad - r \sin{\left (\phi \right )} \sin{\left (\theta \right )}, \quad r \cos{\left (\phi \right )} \cos{\left (\theta \right )}\right ], \quad \left [ \sin{\left (\phi \right )} \sin{\left (\theta \right )}, \quad r \sin{\left (\phi \right )} \cos{\left (\theta \right )}, \quad r \sin{\left (\theta \right )} \cos{\left (\phi \right )}\right ], \quad \left [ \cos{\left (\phi \right )}, \quad 0, \quad - r \sin{\left (\phi \right )}\right ]\right ]$$

```python

```

Certification 2/Week2.3 - Jacobian Matrix.ipynb


```python
import numpy as np

import scipy.linalg as la

import sympy as sp
```

Lecture 14. Orthogonality

Orthogonal vectors:

```python
x = np.array([1,0])
y = np.array([0,1])
np.dot(x.T,y)
```

    0

|x|^2 + |y|^2 = |x+y|^2

x.Tx - length squared

x.Tx+y.Ty = (x+y).T(x+y)

x.Tx+y.Ty = x.Tx+x.Ty+y.Tx+y.Ty

2x.Ty = 0

x.Ty = 0

```python
np.dot(x.T,x)
```

    1

```python
np.dot(y.T,y)
```

    1

```python
np.dot((x+y).T,(x+y))
```

    2

Dot product of orthogonal vectors is 0.

0 vector is orthogonal to everthing.

Now to subspaces

Supspace S is orthogonal to subspace T:

means every vector in S is orthogonal to every vector in T. They don't intersect (except of 0 vector).

Rowspace is orthogonal to a nullspace.

Ax = 0

Means the dot product of all the rows of A and x is 0, and x is a vector in nullspace as a definition.

c1(row1).Tx = 0

"+" = 0

c2(row2).Tx = 0

Nullspace and rowspace are orthogonal complements in R^n.

Nullspace contains all vectors perpedicular to rowspace.

Coming: Ax = b

"solve" when there is no solution.

m > n

b is not in acolumnspace

Practically too many equations with noise on a right side. E.g. finding the best solution.

Drop some equtions is not a solution, as by droping them we literally drop the measurements and loose information.

A.TA - is a square symmetric matrix (A.TA).T = A.TA

A.TAx' = A.Tb  is a way to go. x' is a best solution, not a solution to original system.

N(A.TA) = N(A)

rank of A.TA  = rank of A

A.TA is invertible if A has independant columns.

Lecture 15. Projections.

a.T(b-xa) = 0 - projection of a vector b to a vector a.

xa.Ta = a.Tb

x = (a.Tb)/(a.Ta)

p = ax

p = a(a.Tb)/(a.Ta)

Doubling b will double p - projection.

proj p = Pb

P = aa.T/a.Ta - n x n matrix

C(P)  = a line through a

rank(P) = 1

P.T = P

The projection of a projection is the same projection.

P^2 = P

Why to even project?

Because Ax = b may have no solution, thus we project b to a columnspace of A and this is exactly the closest and the best possible solution.

e = b-p  - error. The reason why we project, and projection sort of eliminates the error.

p = x1'a1 + x2'a2  = Ax'

p = Ax'

Find a solution, such as error vector is perpendicular to the plain. b = Ax'

a1.T(b-Ax' = 0 a2.T(b-Ax')

A.T(b-Ax') = 0

e in N(A.T)

e perpendicular to C(A) - YES!

Key: A.TAx' = A.Tb

Projection matrix.

x'  = (A.TA)^(-1)A.Tb

p = Ax' = A(A.TA)^(-1)A.T b

matrix P = A(A.TA)^(-1)A.T

for squared A; P  = AA^-1(A.T)^-1A.T = I, which will mean it covers the b, and P prohjects b to b.

You can't do this for rectangular.

P.T = P

P^2 = P

Least squares. Fitting 3 points with a best line. Practically a linear regression.

Lecture 16

Example:

C+D = 1

C+2D = 2

C+3D = 2

```python
A = np.array([[1,1],
             [1,2],
             [1,3]])

b = np.array([1,2,2])
```

```python
P = np.dot(np.dot(A, np.linalg.inv(np.dot(A.T,A))),A.T)
```

```python
P
```

    array([[ 0.83333333,  0.33333333, -0.16666667],
           [ 0.33333333,  0.33333333,  0.33333333],
           [-0.16666667,  0.33333333,  0.83333333]])

```python
newb = np.dot(P,b)
newb
```

    array([ 1.16666667,  1.66666667,  2.16666667])

Ax = newb

```python
A
```

    array([[1, 1],
           [1, 2],
           [1, 3]])

Practically just solve A.TAx' = A.Tb

```python
xhat = la.solve(np.dot(A.T,A),np.dot(A.T,b))
xhat
```

    array([ 0.66666667,  0.5       ])

```python
np.around(np.dot(A,xhat) - newb)
```

    array([-0., -0., -0.])

```python
la.inv(np.dot(A.T,A))
```

    array([[ 2.33333333, -1.        ],
           [-1.        ,  0.5       ]])

```python

```


# Revisão de conceitos estatísticos I

Vamos explorar alguns conceitos estatísticos aplicados à análise de sinais.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
import IPython.display as ipd # to play signals
import sounddevice as sd
```

# Exemplo 1: Três pessoas lançam uma moeda 10 vezes

Assim, teremos um registro "temporal" do lançamento da moeda. Vamos dizer que

- Ao conjunto de registros temporais chamamos de ***Ensemble***
- A cada $n$ em $n_{\text{eventos}}$, temos um ***evento***. Cada ***evento*** (cara ou coroa) tem associado a si uma ***variável aleatória***, definida como
    - Cara = 0
    - Coroa = 1

- O universo amostral é: $\Omega = (0, 1)$
- Podemos calcular a probavilidade do evento 'cara' tomando $p(\text{cara}) = n(\text{cara})/n_{\text{eventos}}$. Esta é uma visão frequentista do fenômeno.

```python
n_eventos = np.arange(1,11)
p1 = np.random.randint(2, size=len(n_eventos))
p2 = np.random.randint(2, size=len(n_eventos))
p3 = np.random.randint(2, size=len(n_eventos))

print("Prob. de caras em p1 é de: {:.2f}".format(len(p1[p1==1])/len(p1)))
print("Prob. de caras em p2 é de: {:.2f}".format(len(p2[p2==1])/len(p2)))
print("Prob. de caras em p3 é de: {:.2f}".format(len(p3[p3==1])/len(p3)))

plt.figure(figsize = (12,6))
plt.subplot(3,1,1)
plt.bar(n_eventos, p1, color = 'lightblue')
plt.xticks(n_eventos)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Eventos')
plt.ylabel('V.A. [-]')
plt.xlim((0, len(n_eventos)+1))
plt.ylim((0, 1.2))

plt.subplot(3,1,2)
plt.bar(n_eventos, p2, color = 'lightcoral')
plt.xticks(n_eventos)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Eventos')
plt.ylabel('V.A. [-]')
plt.xlim((0, len(n_eventos)+1))
plt.ylim((0, 1.2))

plt.subplot(3,1,3)
plt.bar(n_eventos, p3, color = 'lightgreen')
plt.xticks(n_eventos)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Eventos')
plt.ylabel('V.A. [-]')
plt.xlim((0, len(n_eventos)+1))
plt.ylim((0, 1.2))
plt.tight_layout();
```

# Exemplo 2: Ruído aleatório com distribuição normal

Neste exemplo nós consideramos um sinal gerado a partir do processo de obter amostras de uma distribuição normal, dada por:

\begin{equation}
p(x) = \mathcal{N}(\mu_x, \sigma_x) = \frac{1}{\sqrt{2\pi}\sigma_x}\mathrm{e}^{-\frac{1}{2\sigma_x^2}(x-\mu_x)^2}
\end{equation}
em que $\mu_x$ é a média e $\sigma_{x}$ é o desvio padrão.

Imaginemos então, que a cada instante de tempo $t$, nós sorteamos um valor da distribuição normal.

```python
fs = 44100
time = np.arange(0,2, 1/fs)

# sinal
mu_x = 0
sigma_x = 0.5
xt = np.random.normal(loc = mu_x, scale = sigma_x, size=len(time))

# Figura
fig, axs = plt.subplots(1, 2, gridspec_kw={'width_ratios': [8, 2]}, figsize = (12, 4))
axs[0].plot(time, xt, '-b', linewidth = 1, color = 'lightcoral')
axs[0].grid(linestyle = '--', which='both')
axs[0].set_xlabel('Tempo [s]')
axs[0].set_ylabel('Amplitude [-]')
axs[0].set_xlim((0, 0.05))
axs[0].set_ylim((-4, 4))

axs[1].hist(xt, density = True, orientation='horizontal', bins=np.linspace(-4*sigma_x, 4*sigma_x, 20), color = 'lightcoral')
axs[1].grid(linestyle = '--', which='both')
axs[1].set_xlabel('Densidade [-]')
axs[1].set_ylabel(r'Valores de $x(t)$ [-]')
axs[1].set_ylim((-4, 4))

plt.tight_layout()

ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python

```

Aula 49 - Rev conceitos estatisticos I/conceitos estatisticos I.ipynb


# Euler's Formula

[back to overview page](index.ipynb)

a proof: http://austinrochford.com/posts/2014-02-05-eulers-formula-sympy.html

```python
import sympy as sp
sp.init_printing()
```

```python
x = sp.symbols('x', real=True)
```

```python
exp1 = sp.exp(sp.I * x)
exp1
```

```python
exp2 = exp1.expand(complex=True)
exp2
```

```python
exp2.rewrite(sp.exp)
```

Euler's identity:

```python
sp.exp(sp.I * sp.pi) + 1
```

  To the extent possible under law,
  the person who associated CC0
  with this work has waived all copyright and related or neighboring
  rights to this work.


## Classical Mechanics - Week 9

### Last Week:
- We saw how a potential can be used to analyze a system
- Gained experience with plotting and integrating in Python

### This Week:
- We will study harmonic oscillations using packages
- Further develope our analysis skills
- Gain more experience wtih sympy

```python
# Let's import packages, as usual
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
sym.init_printing(use_unicode=True)
```

Let's analyze a spring using sympy. It will have mass $m$, spring constant $k$, angular frequency $\omega_0$, initial position $x_0$, and initial velocity $v_0$.

The motion of this harmonic oscillator is described by the equation:

eq 1.) $m\ddot{x} = -kx$

This can be solved as

eq 2.) $x(t) = A\cos(\omega_0 t - \delta)$, $\qquad \omega_0 = \sqrt{\dfrac{k}{m}}$

Use SymPy below to plot this function. Set $A=2$, $\omega_0 = \pi/2$ and $\delta = \pi/4$.

(Refer back to ***Notebook 7*** if you need to review plotting with SymPy.)

```python
# Plot for equation 2 here
```

## Q1.) Calculate analytically the initial conditions, $x_0$ and $v_0$, and the period of the motion for the given constants.  Is your plot consistent with these values?

✅ Double click this cell, erase its content, and put your answer to the above question here.

#### Now let's make plots for underdamped, critically-damped, and overdamped harmonic oscillators.
Below are the general equations for these oscillators:

- Underdamped, $\beta < \omega_0$ :

eq 3.)    $x(t) = A e^{-\beta t}cos(\omega ' t) + B e^{-\beta t}sin(\omega ' t)$ , $\omega ' = \sqrt{\omega_0^2 - \beta^2}$

 ___________________________________

- Critically-damped, $\beta = \omega_0$:

eq 4.)    $x(t) = Ae^{-\beta t} + B t e^{-\beta t}$

 ___________________________________

- Overdamped, $\beta > \omega_0$:

eq 5.)    $x(t) = Ae^{-\left(\beta + \sqrt{\beta^2 - \omega_0^2}\right)t} + Be^{-\left(\beta - \sqrt{\beta^2 - \omega_0^2}\right)t}$

 _______________________

In the cells below use SymPy to create the Position vs Time plots for these three oscillators.

Use $\omega_0=\pi/2$ as before, and then choose an appropriate value of $\beta$ for the three different damped oscillator solutions. Play around with the variables, $A$, $B$, and $\beta$, to see how different values affect the motion and if this agrees with your intuition.

```python
# Put your code for graphing Underdamped here
```

```python
# Put your code for graphing Critical here
```

```python
# Put your code for graphing Overdamped here
```

## Q2.) How would you compare the 3 different oscillators?

✅ Double click this cell, erase its content, and put your answer to the above question here.

# Here's another simple harmonic system, the pendulum.

The equation of motion for the pendulum is:

eq 6.) $ml\dfrac{d^2\theta}{dt^2} + mg \sin(\theta) = 0$, where $v=l\dfrac{d\theta}{dt}$ and $a=l\dfrac{d^2\theta}{dt^2}$

In the small angle approximation $\sin\theta\approx\theta$, so this can be written:

eq 7.) $\dfrac{d^2\theta}{dt^2} = -\dfrac{g}{l}\theta$

We then find the period of the pendulum to be $T = \dfrac{2\pi}{\sqrt{l/g}}$ and the angle at any given time
(if released from rest) is given by

$\theta = \theta_0\cos{\left(\sqrt{\dfrac{g}{l}} t\right)}$.

Let's use Euler's Forward method to solve equation (7) for the motion of the pendulum in the small angle approximation, and compare to the analytic solution.

First, let's graph the analytic solution for $\theta$. Go ahead and graph using either sympy, or the other method we have used, utilizing these variables:

- $t:(0s,50s)$
- $\theta_0 = 0.5$ radians
- $l = 40$ meters

```python
# Plot the analytic solution here
```

Now, use Euler's Forward method to obtain a plot of $\theta$ as a function of time $t$ (in the small angle approximation).  Use eq (7) to calculate $\ddot{\theta}$ at each time step.
Try varying the time step size to see how it affects the Euler's method solution.

```python
# Perform Euler's Method Here
```

You should have found that if you chose the time step size small enough, then the Euler's method solution was
indistinguishable from the analytic solution.

We can now trivially modify this, to solve for the pendulum **exactly**, without using the small angle approximation.
The exact equation for the acceleration is

eq 8.) $\dfrac{d^2\theta}{dt^2} = -\dfrac{g}{l}\sin\theta$.

Modify your Euler's Forward method calculation to use eq (8) to calculate $\ddot{\theta}$ at each time step in the cell below.

```python

```

# Q3.) What time step size did you use to find agreement between Euler's method and the analytic solution (in the small angle approximation)? How did the exact solution differ from the small angle approximation?

✅ Double click this cell, erase its content, and put your answer to the above question here.

### Now let's do something fun:

In class we found that the 2-dimensional anisotropic harmonic motion can be solved as

eq 8a.) $x(t) = A_x \cos(\omega_xt)$

eq 8b.) $y(t) = A_y \cos(\omega_yt - \delta)$

If $\dfrac{\omega_x}{\omega_y}$ is a rational number (*i.e,* a ratio of two integers), then the trajectory repeats itself after some amount of time. The plots of $x$ vs $y$ in this case are called Lissajous figures (after the French physicists Jules Lissajous).  If $\dfrac{\omega_x}{\omega_y}$ is not a rational number, then the trajectory does not repeat itself, but it still shows some very interesting behavior.

Let's make some x vs y plots below for the 2-d anisotropic oscillator.

First, recreate the plots in Figure 5.9 of Taylor.  (Hint: Let $A_x=A_y$. For the left plot of Figure 5.9, let $\delta=\pi/4$ and for the right plot, let $\delta=0$.)

Next, try other rational values of $\dfrac{\omega_x}{\omega_y}$ such as 5/6, 19/15, etc, and using different phase angles $\delta$.

Finally, for non-rational $\dfrac{\omega_x}{\omega_y}$, what does the trajectory plot look like if you let the length of time to be arbitrarily long?

\[For these parametric plots, it is preferable to use our original plotting method, *i.e.* using `plt.plot()`, as introduced in ***Notebook 1***.\]

```python
# Plot the Lissajous curves here
```

# Q4.) What are some observations you make as you play with the variables? What happens for non-rational $\omega_x/\omega_y$ if you let the oscillator run for a long time?

✅ Double click this cell, erase its content, and put your answer to the above question here.

# Notebook Wrap-up.
Run the cell below and copy-paste your answers into their corresponding cells.

```python
from IPython.display import HTML
HTML(
"""

"""
)
```

# Well that's that, another Notebook! It's now been 10 weeks of class

You've been given lots of computational and programing tools these past few months. These past two weeks have been practicing these tools and hopefully you are understanding how some of these pieces add up. Play around with the code and see how it affects our systems of equations. Solve the Schrodinger Equation for the Helium atom. Figure out the unifying theory. The future is limitless!


# 3.2.4 Multiple Outputs

Suppose we have multiple outputs $Y_1, Y_2, ..., Y_k$ that we wish to predict from our inputs $X_0, X_1, ..., X_p$. We assume a linear model for each output (3.34, 3.35):

$$
\begin{align}
Y_k &= \beta_{0k} + \sum_{j=1}^{p} X_j\beta_{jk} + \varepsilon_k\\
&= f_k(X) + \varepsilon_k
\end{align}
$$

We can write the model in matrix notation:

$$
\mathbf{Y} = \mathbf{XB} + \mathbf{E}
$$

Here **Y** is the $N\times K $ response matrix, **X** is the $N\times(p+1)$ input matrix, **B** is the $(p+1)\times K$ matrix and **E** is the $N\times K$ matrix of errors.

A straightforward generalization of the univariate loss functio is (3.37, 3.38):
$$
\begin{align}
RSS(\mathbf{B}) &= \sum_{k=1}^K{\sum_{i=1}^N (y_{ik} - f_k(x_i))^2}\\
&= tr\left[(\mathbf{Y}-\mathbf{XB})^T(\mathbf{Y}-\mathbf{XB})\right]
\end{align}
$$

The least squares estimates is (3.39):
$$\hat{\mathbf{B}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$$

If the errors $\varepsilon=(\varepsilon_1,...,\varepsilon_K)$ are correlated, then (3.40):

$$
RSS(B; \Sigma)=\sum_{i=1}^N(y_i - f(x_i))^T\Sigma^{-1}(y_i - f(x_i))
$$

arises from multivariate Gaussian theory.


# Resistor Cube Solutions

Quick symbolic and numeric solutions to the popular resistor-cube problem.

```python
import sympy as sp
import numpy as np
```

```python
IR,v1,v2,v3,v4,v5,v6,v7 = sp.symbols('IR,v1,v2,v3,v4,v5,v6,v7')
```

## Circuit Matrix

After a half-page or so of algebra, we can derive the system of seven node-equations.

$$\displaystyle \left[\begin{matrix}3 & -1 & -1 & -1 & 0 & 0 & 0\\-1 & 3 & 0 & 0 & -1 & 0 & -1\\-1 & 0 & 3 & 0 & -1 & -1 & 0\\-1 & 0 & 0 & 3 & 0 & -1 & -1\\0 & -1 & -1 & 0 & 3 & 0 & 0\\0 & 0 & -1 & -1 & 0 & 3 & 0\\0 & -1 & 0 & -1 & 0 & 0 & 3\end{matrix}\right]
\left[\begin{matrix}V1\\V2\\V3\\V4\\V5\\V6\\V7\end{matrix}\right]
=
\left[\begin{matrix}IR\\0\\0\\0\\0\\0\\0\end{matrix}\right]$$

We'll represent these in `sympy.Matrix` objects `A` and `b`:

```python
lhs = np.array([
          [ 3,-1,-1,-1, 0, 0, 0],
          [-1, 3, 0, 0,-1, 0,-1],
          [-1, 0, 3, 0,-1,-1, 0],
          [-1, 0, 0, 3, 0,-1,-1],
          [ 0,-1,-1, 0, 3, 0, 0],
          [ 0, 0,-1,-1, 0, 3, 0],
          [ 0,-1, 0,-1, 0, 0, 3]
    ],)
A = sp.Matrix(lhs)
A
```

$\displaystyle \left[\begin{matrix}3 & -1 & -1 & -1 & 0 & 0 & 0\\-1 & 3 & 0 & 0 & -1 & 0 & -1\\-1 & 0 & 3 & 0 & -1 & -1 & 0\\-1 & 0 & 0 & 3 & 0 & -1 & -1\\0 & -1 & -1 & 0 & 3 & 0 & 0\\0 & 0 & -1 & -1 & 0 & 3 & 0\\0 & -1 & 0 & -1 & 0 & 0 & 3\end{matrix}\right]$

```python
b = sp.Matrix([[IR],[0],[0],[0],[0],[0],[0]])
b
```

$\displaystyle \left[\begin{matrix}IR\\0\\0\\0\\0\\0\\0\end{matrix}\right]$

## Symbolic Solution

Most statements of the resistor-cube problem ask for the resistance between diagonal corners of the cube, relative to the unit-resistance per edge.  In other words, the desired quantity is:

```
V(1) / I
```

Conveniently the `sympy.linsolve` solver will give us just this - along with all of the other node voltages:

```python
sp.linsolve((A,b),(v1,v2,v3,v4,v5,v6,v7))
```

$\displaystyle \left\{\left( \frac{5 IR}{6}, \  \frac{IR}{2}, \  \frac{IR}{2}, \  \frac{IR}{2}, \  \frac{IR}{3}, \  \frac{IR}{3}, \  \frac{IR}{3}\right)\right\}$

These results square with our intuitive expectations.

## Numeric Solution

Most "real" circuit-simulations are not amenable to symbolic-solvers such as `sympy`, for a slew of reasons.

* They are often non-linear and piece-wise defined.
* They have (many) more nodes than numeric solvers can efficiently handle.
* Even if we generated symbolic results for complicated circuits, we'd often not know what to do with them - other than converting into numeric answers.

Instead nearly all practical circuit simulation uses numerical solutions.  We'll just have to choose a numeric value for our single parameter `IR` - say, 1.0V.  The resistor-cube is nice enough to be completely linear, and allows us to use a linear solver such as `numpy.linalg.solve`:

```python
rhs = np.transpose(np.array([1,0,0,0,0,0,0]))
np.linalg.solve(lhs, rhs)
```

    array([0.83333333, 0.5       , 0.5       , 0.5       , 0.33333333,
           0.33333333, 0.33333333])

Again this checks out, both with our intuitive expectations and the earlier symbolic results.

## Intuitive Solution

The resistor-cube is usually offered as a brain-teaser of sorts, whether as a job interview problem or just for shits & grins.  I have never seen anyone (but me!) work through it analytically as we have here.

Instead, a typical thought-process looks something like so:

* We can tell from looking at this circuit that there are a few symmetries between similar-looking nodes.  This may become particularly clear when we "flatten" the cube into a 2D visual form.
* From inspection (and our deep well of experience), we can discern that `V(2)=V(3)=V(4)`.  In other words, `I2=I3=I4 = I/3`.
* Similarly we can tell that `V(5)=V(6)=V(7)`, and that `I5=I6=I7 = I/3`.
* With those two realizations in hand, we can further discern that each of the six middle-row resistors has *half* the current of their top and bottom-row counterparts, or `(I/3)/2 = I/6`.
* Walking through any KVL loop including a top, middle, and bottom row resistor, we find that their voltages are `IR/3`, `IR/6`, and `IR/3`, making a total of `5*IR/6`.  The overall resistance `V(1)/I` is therefore `5*R/6`.

Although not usually asked, we can use the same insights to notice that the values of `V(5)=V(6)=V(7) = IR/3`, and that `V(2)=V(3)=V(4)` are `IR/3+IR/6 = IR/2`.  All checks out with the symbolic and numeric solutions.

```python

```


# Constrained ESP fitting

## using Horton's cost function

According to https://theochem.github.io/horton/2.1.0/user_postproc_espfit.html, cost function is constructed as

$$ c(\mathbf{q}, \Delta V_\text{ref}) = \int_V d\mathbf{r} \omega(\mathbf{r}) \cdot \left( V_\text{ab initio}(\mathbf{r}) - \sum_{i=1}^N \frac{q_i}{\mathbf{r} - \mathbf{R}_i} - \Delta V_\text{ref} \right)^2  $$

> $\Delta V_\text{ref}$: constant that may account for differences in reference between the ab initio ESP and the point charge ESP. The need for such a term was established in the REPEAT paper for ESP fitting in 3D periodic systems.

We look at an aperiodic system, neglect this term for now. The (unconstrained) cost function takes the general quadratic form

$$ c_u(\mathbf{x}) = \mathbf{x}^T A\ \mathbf{x} - 2 \mathbf{b}^T \mathbf{x} - C $$

with

$$ C = - \int_V d\mathbf{r} \omega(\mathbf{r}) \cdot \left[ V_\text{ab initio}(\mathbf{r}) \right]^2$$

entry i,j of matrix $A^{N\times N}$

$$  A_{ij} = \int_V d\mathbf{r}\ \omega(\mathbf{r}) \cdot \left( \frac{1}{|\mathbf{r} - \mathbf{R}_i|} \cdot \frac{1}{|\mathbf{r} - \mathbf{R}_j|} \right)  $$

and entry i of vector $\mathbf{b}$

$$  b_i = \int_V d\mathbf{r}\ \omega(\mathbf{r}) \cdot \frac{V_\text{ab initio}(\mathbf{r})}{|\mathbf{r} - \mathbf{R}_i|} $$

In the code below, first the miniumum of an unconstrained system is found by solving

$$ \frac{1}{2} \cdot \frac{\mathrm{d} c(\mathbf{x})}{\mathrm d  \mathbf{x}}  = \frac{1}{2} \cdot \nabla_\mathbf{x} c = A \mathbf{x} - \mathbf{b} = 0$$

with $\nabla (\mathbf{b}^T \mathbf{x}) = \mathbf{b}$ and
$\nabla \cdot (\mathbf{x}^T A \mathbf{x}) = (A + A^T) \mathbf{x}
= 2 A \mathbf{x} $ for symmetric A, as in our case.
The (unconstrained) solution

$$ \mathbf{x}_u = A^{-1} \mathbf{b} $$

is corrected for *one* total charge constraint of the form

$$ g(\mathbf{x}) = \mathbf{d} \cdot \mathbf{x} - q_\text{tot} = 0 $$

with all entries of $\mathbf{d}$ unity. Notice that in the code below, the whole system is normalized in order to have unity diagonal entries $A_{jj}$. We neglect this normalization here.

A Lagrange multiplier $\lambda$ is introduced into the *constrained* cost function

$$ c_c(\mathbf{x},\lambda) = \mathbf{x}^T A\ \mathbf{x} - 2 \mathbf{b}^T \mathbf{x} - C + \lambda \cdot  g(\mathbf{x}) $$

and the system

$$ \frac{1}{2} \cdot \nabla_\mathbf{x} c = A \mathbf{x} - \mathbf{b} + \lambda \cdot \nabla_\mathbf{x} g(\mathbf{x}) = A \mathbf{x} - \mathbf{b} +\lambda \cdot \mathbf{d} = 0 $$

$$  \nabla_\mathbf{\lambda} c = g(\mathbf{x}) = \mathbf{d} \cdot \mathbf{x} - q_\text{tot} = 0 $$

is solved by finding a correction for the unconstrained solution

$$ \mathbf{x} = \mathbf{x}_u - \lambda \cdot\delta \mathbf{x} = A^{-1} \mathbf{b} - \lambda \cdot \delta \mathbf{x} $$

as

$$ - \lambda A \delta \mathbf{x} + \lambda \cdot \mathbf{d} = 0
\Leftrightarrow \delta \mathbf{x} = A^{-1} \mathbf{d}$$

and the Lagrange multiplier by

$$ g(\mathbf{x})
    = \mathbf{d} \cdot \left( \mathbf{x}_u
        - \lambda \ \delta \mathbf{x} \right)- q_\text{tot}
    = \mathbf{d} \cdot A^{-1} \mathbf{b}
        - \lambda \ \mathbf{d} \cdot \delta \mathbf{x} - q_\text{tot}
    = 0 $$

$$ \lambda = \frac{\mathbf{d} \cdot A^{-1} \mathbf{b} - q_\text{tot}}
    { \mathbf{d} \cdot \delta \mathbf{x} }
    = \frac{\mathbf{b} \cdot \delta \mathbf{x} - q_\text{tot}}
    { \mathbf{d} \cdot \delta \mathbf{x} }$$

and thereby the constrained minimum at
$$ \mathbf{x}
    = \mathbf{x}_u
        - \frac{\mathbf{b} \cdot \delta \mathbf{x} - q_\text{tot}}
        { \mathbf{d} \cdot \delta \mathbf{x} }
        \cdot \delta \mathbf{x}$$

as implemented in HORTON

```python
# modified excerpt from horton/espfit/cost.py

import numpy as np
import h5py

def solve(_A, _B, _C, _natom, qtot=None, ridge=0.0):
    # apply regularization to atomic degrees of freedom
    A = _A.copy()
    A.ravel()[::len(A)+1][:_natom] += ridge*np.diag(A)[:_natom].mean()
    # construct preconditioned matrices
    norms = np.diag(A)**0.5
    print("Diagonal of A: \n {}".format(np.diag(A)))
    print("Vector norms: \n {}".format(norms))
    print("Matrix norms: \n {}".format(norms*norms.reshape(-1,1)))

    A = A/norms/norms.reshape(-1,1)
    print("Normed diagonal of A: \n {}".format(np.diag(A)))

    B = _B/norms
    print("Normed B: \n {}".format(B))

    x = np.linalg.solve(A, B)
    if qtot is not None:
        # Fix the total charge with a lagrange multiplier
        d = np.zeros(len(A))
        d[:_natom] = 1/norms[:_natom]
        d[_natom:] = 0.0
        aid = np.linalg.solve(A, d)
        lagrange = (np.dot(aid, B) - qtot)/np.dot(aid, d)
        x -= aid*lagrange
    x /= norms
    return x
```

```python
np.set_printoptions(precision=4)
```

```python
# Example: water
# load the cost function
cost_function = h5py.File('h2o.cost.h5')
cost = cost_function['cost']
A = cost['A'][:]
B = cost['B'][:]
C = cost['C'].value
N = cost['natom'].value
print("A: \n {}".format(A))
print("B: {}".format(B))
print("C: {}".format(C))
print("N: {}".format(N))
```

    A:
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B: [-0.0105 -0.0797 -0.0797]
    C: 0.0266115260155
    N: 3

```python
q1 = solve(A,B,C,N)
```

    Diagonal of A:
     [ 8.5387  9.0212  9.0212]
    Vector norms:
     [ 2.9221  3.0035  3.0035]
    Matrix norms:
     [[ 8.5387  8.7766  8.7766]
     [ 8.7766  9.0212  9.0212]
     [ 8.7766  9.0212  9.0212]]
    Normed diagonal of A:
     [ 1.  1.  1.]
    Normed B:
     [-0.0036 -0.0265 -0.0265]

```python
q1
```

    array([ 0.3378, -0.1699, -0.1699])

```python
sum(q1)
```

    -0.0019095241274412478

```python
q2 = solve(A,B,C,N,qtot=0)
```

    Diagonal of A:
     [ 8.5387  9.0212  9.0212]
    Vector norms:
     [ 2.9221  3.0035  3.0035]
    Matrix norms:
     [[ 8.5387  8.7766  8.7766]
     [ 8.7766  9.0212  9.0212]
     [ 8.7766  9.0212  9.0212]]
    Normed diagonal of A:
     [ 1.  1.  1.]
    Normed B:
     [-0.0036 -0.0265 -0.0265]

```python
q2
```

    array([ 0.3396, -0.1698, -0.1698])

```python
sum(q2)
```

    0.0

```python
q3 = solve(A,B,C,N,qtot=1)
```

    Diagonal of A:
     [ 8.5387  9.0212  9.0212]
    Vector norms:
     [ 2.9221  3.0035  3.0035]
    Matrix norms:
     [[ 8.5387  8.7766  8.7766]
     [ 8.7766  9.0212  9.0212]
     [ 8.7766  9.0212  9.0212]]
    Normed diagonal of A:
     [ 1.  1.  1.]
    Normed B:
     [-0.0036 -0.0265 -0.0265]

```python
q3
```

    array([ 1.2558, -0.1279, -0.1279])

```python
sum(q3)
```

    0.99999999999999989

# Arbitrary number of equality constraints

We modifiy the optimization in order to allow for an arbitrary amount of constraints.

M lagrange multipliers $\lambda_k$ are introduced into the *constrained* cost function

$$ c_c(\mathbf{x},\mathbf{\lambda}) = \mathbf{x}^T A\ \mathbf{x} - 2 \mathbf{b}^T \mathbf{x} - C + \sum_{k=1}^M \lambda_k \cdot  g_k(\mathbf{x}) $$

All our constraints (charge groups and symmetries) will be of linear form

$$ g_k(\mathbf{x}) = \mathbf{d}_k \cdot \mathbf{x} - q_k = 0 $$

and thus can be compacted into matrix form

$$ D^{(M \times N)} \mathbf{x} - \mathbf{q}^{(M\times 1)} = 0 $$

with

$$ D^T = [\mathbf{d}_1, \mathbf{d}_2, \dots , \mathbf{d}_M] $$

and hence
$$ c_c(\mathbf{x}^{(N\times1)},\mathbf{\lambda}^{(M\times1)}) = \mathbf{x}^T A\ \mathbf{x}
    - 2 \mathbf{b}^T \mathbf{x}
    - C + \mathbf{\lambda}^T \cdot \left( D \mathbf{x} - \mathbf{q} \right) $$

Derivative

$$
\begin{align}
    \nabla_\mathbf{x} \cdot c_c & = 2\ A\ \mathbf{x}
    + \sum_{k=1}^M \lambda_k \mathbf{d}_k - 2 \mathbf{b} & = 0\\
    \nabla_\mathbf{\lambda} \cdot c_c & = D\ \mathbf{x} - \mathbf{q} & = 0
\end{align}
$$

Identify

$$ D^T \mathbf{\lambda}= \sum_{k=1}^M \lambda_k \mathbf{d}_k  $$

and solve

$$ \tilde{A} \mathbf{y} - \tilde{\mathbf{b}} = 0$$

with generalized $\mathbf{y}^T = [\mathbf{x}^T, \mathbf{\lambda}^T ]$
as well as $(N+M)\times(N+M)$ matrix

$$ \tilde{A} =
    \begin{bmatrix}
        2 A & D^T \\
        D & 0
    \end{bmatrix} $$

and $(N+M)$ vector

$$ \tilde{\mathbf{b}} =
   \begin{bmatrix}
        2 \mathbf{b} \\
        \mathbf{q}
   \end{bmatrix} $$

```python
import numpy as np

def constrainedMinimize(A_matrix, b_vector, C_scalar, D_matrix = None, q_vector = np.array([0]), debug=False):
    N = b_vector.shape[0]
    M = q_vector.shape[0]

    if not isinstance(D_matrix,np.ndarray):
        D_matrix = np.atleast_2d( np.ones(N) )

    if debug:
        print("{:d} unknowns, {:d} equality constraints".format(N,M))
        print("A {}: \n {}".format(A_matrix.shape,A_matrix))
        print("B {}: \n {}".format(b_vector.shape,b_vector))
        print("C {}: \n {}".format(C_scalar.shape,C_scalar))
        print("D {}: \n {}".format(D_matrix.shape,D_matrix))
        print("q {}: \n {}".format(q_vector.shape,q_vector))

    A_upper = np.block([ 2*np.atleast_2d(A_matrix), np.atleast_2d(D_matrix).T ])
    A_lower = np.block([ np.atleast_2d(D_matrix), np.atleast_2d(np.zeros((M,M)))])
    if debug:
        print("upper A ({}): \n {}".format(A_upper.shape,A_upper))
        print("upper A ({}): \n {}".format(A_lower.shape,A_lower))

    A = np.block( [ [ A_upper ], [ A_lower ] ] )

    if debug:
        print("block A ({}): \n {}".format(A.shape,A))

    B = np.block( [2*np.atleast_1d(b_vector), np.atleast_1d(q_vector)] )

    if debug:
        print("block B ({}): \n {}".format(B.shape,B))

    C = C_scalar

    x = np.linalg.solve(A, B)

    return x
```

```python
# charges: list of charges to meet symmetry constraint,
#   indices beginning from 0
#   can also be list of list of group of charges to be symmetric,
#   indices beginning from 0, i.e. charges = [[0,1],[2,3,4]] requires
#   the cummulative charge of atom 0 and 1 to equal
#   the sum over charges 2 to 4
# N: total number of charges
# symmetry: scalar or list of symmetry requirements
#   default 1 causes equality, -1 antisymmetry,
#   values different from +/- unity result in relative scaling
def constructPairwiseSymmetryConstraints(charges, N, symmetry=1.0, debug=False):
    M = len(charges)-1

    symmetry = symmetry*np.ones(M)

    if debug:
        print("{:d} unknowns, {:d} pairwise equality constraints".format(N,M))
        print("symmetry list ({}):\n{}".format(symmetry.shape,symmetry))

    D = np.atleast_2d(np.zeros((M,N)))
    q = np.atleast_1d(np.zeros(M))
    D[:,charges[0]] = 1

    for j in range(M):
        D[j,charges[j+1]] = -1.0*symmetry[j]

    if debug:
        print("D ({}):\n{}".format(D.shape,D))
        print("q ({}):\n{}".format(q.shape,q))

    return D, q
```

```python
# chargeGroups: list of list of charges in charge groups,
#    indices beginning from 0, i.e. chargeGroups = [[0,1],[2,3,4]]
# N: total number of charges
# q: required charges per charge group

def constructChargegroupConstraints(chargeGroups, N, q=0, debug=False):
    M = len(chargeGroups)

    q = np.atleast_1d(q*np.ones(M))

    if debug:
        print("{:d} unknowns, {:d} pairwise equality constraints".format(N,M))

    D = np.atleast_2d(np.zeros((M,N)))
    #q = np.atleast_2d(np.zeros(M))

    for j in range(M):
        D[j,chargeGroups[j]] = 1.0

    if debug:
        print("D ({}):\n{}".format(D.shape,D))
        print("q ({}):\n{}".format(q.shape,q))

    return D, q
```

```python
q1u = solve(A,B,C,N)
```

    Diagonal of A:
     [ 8.5387  9.0212  9.0212]
    Vector norms:
     [ 2.9221  3.0035  3.0035]
    Matrix norms:
     [[ 8.5387  8.7766  8.7766]
     [ 8.7766  9.0212  9.0212]
     [ 8.7766  9.0212  9.0212]]
    Normed diagonal of A:
     [ 1.  1.  1.]
    Normed B:
     [-0.0036 -0.0265 -0.0265]

```python
q1c = constrainedMinimize(A,B,C,debug=True)
```

    3 unknowns, 1 equality constraints
    A (3, 3):
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B (3,):
     [-0.0105 -0.0797 -0.0797]
    C ():
     0.0266115260155
    D (1, 3):
     [[ 1.  1.  1.]]
    q (1,):
     [0]
    upper A ((3, 4)):
     [[ 17.0774  17.0433  17.0433   1.    ]
     [ 17.0433  18.0423  16.7912   1.    ]
     [ 17.0433  16.7912  18.0423   1.    ]]
    upper A ((1, 4)):
     [[ 1.  1.  1.  0.]]
    block A ((4, 4)):
     [[ 17.0774  17.0433  17.0433   1.    ]
     [ 17.0433  18.0423  16.7912   1.    ]
     [ 17.0433  16.7912  18.0423   1.    ]
     [  1.       1.       1.       0.    ]]
    block B ((4,)):
     [-0.021  -0.1594 -0.1594  0.    ]

```python
q1u
```

    array([ 0.3378, -0.1699, -0.1699])

```python
q1c
```

    array([ 0.3396, -0.1698, -0.1698, -0.0326])

```python
q2_horton = solve(A,B,C,N,qtot=1)
```

    Diagonal of A:
     [ 8.5387  9.0212  9.0212]
    Vector norms:
     [ 2.9221  3.0035  3.0035]
    Matrix norms:
     [[ 8.5387  8.7766  8.7766]
     [ 8.7766  9.0212  9.0212]
     [ 8.7766  9.0212  9.0212]]
    Normed diagonal of A:
     [ 1.  1.  1.]
    Normed B:
     [-0.0036 -0.0265 -0.0265]

```python
q2c = constrainedMinimize(A,B,C,q_vector=np.array([1]),debug=True)
```

    3 unknowns, 1 equality constraints
    A (3, 3):
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B (3,):
     [-0.0105 -0.0797 -0.0797]
    C ():
     0.0266115260155
    D (1, 3):
     [[ 1.  1.  1.]]
    q (1,):
     [1]
    upper A ((3, 4)):
     [[ 17.0774  17.0433  17.0433   1.    ]
     [ 17.0433  18.0423  16.7912   1.    ]
     [ 17.0433  16.7912  18.0423   1.    ]]
    upper A ((1, 4)):
     [[ 1.  1.  1.  0.]]
    block A ((4, 4)):
     [[ 17.0774  17.0433  17.0433   1.    ]
     [ 17.0433  18.0423  16.7912   1.    ]
     [ 17.0433  16.7912  18.0423   1.    ]
     [  1.       1.       1.       0.    ]]
    block B ((4,)):
     [-0.021  -0.1594 -0.1594  1.    ]

```python
q2_horton
```

    array([ 1.2558, -0.1279, -0.1279])

```python
q2c
```

    array([  1.2558,  -0.1279,  -0.1279, -17.1071])

```python
d1 = np.array([0,1,-2])
```

```python
q3c = constrainedMinimize(A,B,C,D_matrix=d1,q_vector=np.array([0]),debug=True)
```

    3 unknowns, 1 equality constraints
    A (3, 3):
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B (3,):
     [-0.0105 -0.0797 -0.0797]
    C ():
     0.0266115260155
    D (3,):
     [ 0  1 -2]
    q (1,):
     [0]
    upper A ((3, 4)):
     [[ 17.0774  17.0433  17.0433   0.    ]
     [ 17.0433  18.0423  16.7912   1.    ]
     [ 17.0433  16.7912  18.0423  -2.    ]]
    upper A ((1, 4)):
     [[ 0.  1. -2.  0.]]
    block A ((4, 4)):
     [[ 17.0774  17.0433  17.0433   0.    ]
     [ 17.0433  18.0423  16.7912   1.    ]
     [ 17.0433  16.7912  18.0423  -2.    ]
     [  0.       1.      -2.       0.    ]]
    block B ((4,)):
     [-0.021  -0.1594 -0.1594  0.    ]

```python
q3c
```

    array([ 0.2884, -0.1935, -0.0967,  0.0403])

```python
d1 = np.array([0,1,-2])
```

```python
d2 = np.array([1,1,1])
```

```python
D = np.block([[d1],[d2]])
```

```python
D
```

    array([[ 0,  1, -2],
           [ 1,  1,  1]])

```python
q = np.array([0,3])
```

```python
q4c = constrainedMinimize(A,B,C,D_matrix=D,q_vector=q,debug=True)
```

    3 unknowns, 2 equality constraints
    A (3, 3):
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B (3,):
     [-0.0105 -0.0797 -0.0797]
    C ():
     0.0266115260155
    D (2, 3):
     [[ 0  1 -2]
     [ 1  1  1]]
    q (2,):
     [0 3]
    upper A ((3, 5)):
     [[ 17.0774  17.0433  17.0433   0.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.    ]
     [ 17.0433  16.7912  18.0423  -2.       1.    ]]
    upper A ((2, 5)):
     [[ 0.  1. -2.  0.  0.]
     [ 1.  1.  1.  0.  0.]]
    block A ((5, 5)):
     [[ 17.0774  17.0433  17.0433   0.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.    ]
     [ 17.0433  16.7912  18.0423  -2.       1.    ]
     [  0.       1.      -2.       0.       0.    ]
     [  1.       1.       1.       0.       0.    ]]
    block B ((5,)):
     [-0.021  -0.1594 -0.1594  0.      3.    ]

```python
q4c
```

    array([  3.0755e+00,  -5.0328e-02,  -2.5164e-02,   1.0487e-02,  -5.1256e+01])

```python
sum(q4c)
```

    -48.245273601712007

```python
D2,q2 = constructPairwiseSymmetryConstraints([1,3,8],10,debug=True)
```

    10 unknowns, 2 pairwise equality constraints
    symmetry list ((2,)):
    [ 1.  1.]
    D ((2, 10)):
    [[ 0.  1.  0. -1.  0.  0.  0.  0.  0.  0.]
     [ 0.  1.  0.  0.  0.  0.  0.  0. -1.  0.]]
    q ((2,)):
    [ 0.  0.]

```python
D3,q3 = constructPairwiseSymmetryConstraints([1,3,8],10,symmetry=-1,debug=True)
```

    10 unknowns, 2 pairwise equality constraints
    symmetry list ((2,)):
    [-1. -1.]
    D ((2, 10)):
    [[ 0.  1.  0.  1.  0.  0.  0.  0.  0.  0.]
     [ 0.  1.  0.  0.  0.  0.  0.  0.  1.  0.]]
    q ((2,)):
    [ 0.  0.]

```python
D4,q4 = constructPairwiseSymmetryConstraints([1,3,8],10,symmetry=np.array([1,-1]),debug=True)
```

    10 unknowns, 2 pairwise equality constraints
    symmetry list ((2,)):
    [ 1. -1.]
    D ((2, 10)):
    [[ 0.  1.  0. -1.  0.  0.  0.  0.  0.  0.]
     [ 0.  1.  0.  0.  0.  0.  0.  0.  1.  0.]]
    q ((2,)):
    [ 0.  0.]

```python
D5,q5 = constructPairwiseSymmetryConstraints([[1,2],[3,4,5],[8,9]],10,symmetry=np.array([1,-1]),debug=True)
```

    10 unknowns, 2 pairwise equality constraints
    symmetry list ((2,)):
    [ 1. -1.]
    D ((2, 10)):
    [[ 0.  1.  1. -1. -1. -1.  0.  0.  0.  0.]
     [ 0.  1.  1.  0.  0.  0.  0.  0.  1.  1.]]
    q ((2,)):
    [ 0.  0.]

### full example
Water with antisymmetric hydrogens and +1 integer charge group of oxygen and 1 hydrogen

```python
# Example: water
import h5py
#load the cost function
cost_function = h5py.File('h2o.cost.h5')
cost = cost_function['cost']
A = cost['A'][:]
B = cost['B'][:]
C = cost['C'].value
N = cost['natom'].value
print("A: {}".format(A))
print("B: {}".format(B))
print("C: {}".format(C))
print("N: {}".format(N))

```

    A: [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B: [-0.0105 -0.0797 -0.0797]
    C: 0.0266115260155
    N: 3

```python
D1,q1 = constructPairwiseSymmetryConstraints(
    charges=[1,2],N=3,symmetry=-1,debug=True)
```

    3 unknowns, 1 pairwise equality constraints
    symmetry list ((1,)):
    [-1.]
    D ((1, 3)):
    [[ 0.  1.  1.]]
    q ((1,)):
    [ 0.]

```python
D1, q1
```

    (array([[ 0.,  1.,  1.]]), array([ 0.]))

```python
D2,q2 = constructChargegroupConstraints(
    chargeGroups=[[0,1]],N=3,q=1,debug=True)
```

    3 unknowns, 1 pairwise equality constraints
    D ((1, 3)):
    [[ 1.  1.  0.]]
    q ((1,)):
    [ 1.]

```python
D2, q2
```

    (array([[ 1.,  1.,  0.]]), array([ 1.]))

```python
D = np.vstack([D1,D2])
```

```python
D
```

    array([[ 0.,  1.,  1.],
           [ 1.,  1.,  0.]])

```python
q = np.hstack([q1,q2])
```

```python
q
```

    array([ 0.,  1.])

```python
X = constrainedMinimize(A,B,C,D,q,debug=True)
```

    3 unknowns, 2 equality constraints
    A (3, 3):
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B (3,):
     [-0.0105 -0.0797 -0.0797]
    C ():
     0.0266115260155
    D (2, 3):
     [[ 0.  1.  1.]
     [ 1.  1.  0.]]
    q (2,):
     [ 0.  1.]
    upper A ((3, 5)):
     [[ 17.0774  17.0433  17.0433   0.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.    ]
     [ 17.0433  16.7912  18.0423   1.       0.    ]]
    upper A ((2, 5)):
     [[ 0.  1.  1.  0.  0.]
     [ 1.  1.  0.  0.  0.]]
    block A ((5, 5)):
     [[ 17.0774  17.0433  17.0433   0.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.    ]
     [ 17.0433  16.7912  18.0423   1.       0.    ]
     [  0.       1.       1.       0.       0.    ]
     [  1.       1.       0.       0.       0.    ]]
    block B ((5,)):
     [-0.021  -0.1594 -0.1594  0.      1.    ]

```python
X
```

    array([ 0.1267,  0.8733, -0.8733, -1.2267, -2.1852])

```python
Q = X[:N]
```

```python
Lagrange = X[N:]
```

```python
Q # all constraints fulfilled
```

    array([ 0.1267,  0.8733, -0.8733])

```python
Q[0]+Q[1]
```

    1.0

```python
Q[1]+Q[2]
```

    1.1102230246251565e-16

```python
Lagrange
```

    array([-1.2267, -2.1852])

```python

```

```python
D3,q3 = constructPairwiseSymmetryConstraints(
    charges=[1,2],N=3,symmetry=1,debug=True)
```

    3 unknowns, 1 pairwise equality constraints
    symmetry list ((1,)):
    [ 1.]
    D ((1, 3)):
    [[ 0.  1. -1.]]
    q ((1,)):
    [ 0.]

```python
D4,q4 = constructPairwiseSymmetryConstraints(
    charges=[1,2],N=3,symmetry=-1,debug=True)
```

    3 unknowns, 1 pairwise equality constraints
    symmetry list ((1,)):
    [-1.]
    D ((1, 3)):
    [[ 0.  1.  1.]]
    q ((1,)):
    [ 0.]

```python
D5,q5 = constructChargegroupConstraints(chargeGroups=[[0,1,2],[0,1,2]],N=3,q=[0,1],debug=True)
```

    3 unknowns, 2 pairwise equality constraints
    D ((2, 3)):
    [[ 1.  1.  1.]
     [ 1.  1.  1.]]
    q ((2,)):
    [ 0.  1.]

```python
D = np.vstack([D3,D4,D5])
```

```python
D
```

    array([[ 0.,  1., -1.],
           [ 0.,  1.,  1.],
           [ 1.,  1.,  1.],
           [ 1.,  1.,  1.]])

```python
q = np.hstack([q3,q4,q5])
```

```python
q
```

    array([ 0.,  0.,  0.,  1.])

```python
X = constrainedMinimize(A,B,C,D,q,debug=True)
```

    3 unknowns, 4 equality constraints
    A (3, 3):
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B (3,):
     [-0.0105 -0.0797 -0.0797]
    C ():
     0.0266115260155
    D (4, 3):
     [[ 0.  1. -1.]
     [ 0.  1.  1.]
     [ 1.  1.  1.]
     [ 1.  1.  1.]]
    q (4,):
     [ 0.  0.  0.  1.]
    upper A ((3, 7)):
     [[ 17.0774  17.0433  17.0433   0.       0.       1.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.       1.       1.    ]
     [ 17.0433  16.7912  18.0423  -1.       1.       1.       1.    ]]
    upper A ((4, 7)):
     [[ 0.  1. -1.  0.  0.  0.  0.]
     [ 0.  1.  1.  0.  0.  0.  0.]
     [ 1.  1.  1.  0.  0.  0.  0.]
     [ 1.  1.  1.  0.  0.  0.  0.]]
    block A ((7, 7)):
     [[ 17.0774  17.0433  17.0433   0.       0.       1.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.       1.       1.    ]
     [ 17.0433  16.7912  18.0423  -1.       1.       1.       1.    ]
     [  0.       1.      -1.       0.       0.       0.       0.    ]
     [  0.       1.       1.       0.       0.       0.       0.    ]
     [  1.       1.       1.       0.       0.       0.       0.    ]
     [  1.       1.       1.       0.       0.       0.       0.    ]]
    block B ((7,)):
     [-0.021  -0.1594 -0.1594  0.      0.      0.      1.    ]

```python
X[:N]
```

    array([-0.004 ,  0.0258, -0.0218])

```python
X[N:]
```

    array([ -3.1395e-02,  -1.2500e-01,   1.4412e+17,  -1.4412e+17])

```python
X = constrainedMinimize(A,B,C,D5,q5,debug=True)
```

    3 unknowns, 2 equality constraints
    A (3, 3):
     [[ 8.5387  8.5216  8.5216]
     [ 8.5216  9.0212  8.3956]
     [ 8.5216  8.3956  9.0212]]
    B (3,):
     [-0.0105 -0.0797 -0.0797]
    C ():
     0.0266115260155
    D (2, 3):
     [[ 1.  1.  1.]
     [ 1.  1.  1.]]
    q (2,):
     [ 0.  1.]
    upper A ((3, 5)):
     [[ 17.0774  17.0433  17.0433   1.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.    ]
     [ 17.0433  16.7912  18.0423   1.       1.    ]]
    upper A ((2, 5)):
     [[ 1.  1.  1.  0.  0.]
     [ 1.  1.  1.  0.  0.]]
    block A ((5, 5)):
     [[ 17.0774  17.0433  17.0433   1.       1.    ]
     [ 17.0433  18.0423  16.7912   1.       1.    ]
     [ 17.0433  16.7912  18.0423   1.       1.    ]
     [  1.       1.       1.       0.       0.    ]
     [  1.       1.       1.       0.       0.    ]]
    block B ((5,)):
     [-0.021  -0.1594 -0.1594  0.      1.    ]

```python
X
```

    array([  3.5032e-01,  -1.6107e-01,  -1.8995e-01,   1.4412e+17,  -1.4412e+17])

```python

```