# Model Project - Consumer Problem

This project has to purpose to find the optimal consumption, for a consumer. We are given an utility function and the price of the two goods. The optimal consumption for the consumer is determined both numerically, algebraically. An interactive solution is also presented. In addition, this project will be using theory from Microeconomics I, and rather generous inspiration will be taken from exercise 2 on the Winter 2016 exam.

```python
#Importing packages
import sympy as sm
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy import optimize
```

## The Consumers Utility

This problem we will be looking at, is the way the consumer is utilizing their 2 given goods, x1 and x2

$$ u(x_1,x_2)=x_1x_2+2x_2 $$

This function is then defined

```python
def u(x1,x2):
    u = x1*x2 + 2 * x2
    return u
```

```python
def U(x1,x2):
    return x1x2 + 2*x2
```
The CPF is in this case (which is not the same as the PPF/PPC, since we are not under autarky), is given as

Furthermore, we are assuming that the following is true

$$ p1,p2, I>0 $$

Now, to illustrate the consumers utility which was just defined above, a 3d plot will be made

```python
# First things first
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')

# Vectors, range 0 to 15
x1_vec = np.linspace(0,15,150)
x2_vec = np.linspace(0,15,150)

# Grid
x1_grid, x2_grid = np.meshgrid(x1_vec, x2_vec)

# Utility function within the grid
u_grid = u(x1_grid, x2_grid)

# PLotting
ax.plot_surface(x1_grid, x2_grid, u_grid, color='g')

# Labels and inv axes
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$u$')
ax.invert_xaxis()
```

## The Utility Maximization Problem

The following utility maximization problem is considered

The variables given will be defined as sympy symbols, which will make it possible for us to do equations

```python
# From variables to sympy symbols
x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
p1 = sm.symbols('p_1')
p2 = sm.symbols('p_2')
I = sm.symbols('I')
U = sm.symbols('U')
```

Furthermore, a toggle will be enabled, so we can see the result after each equation

```python
sm.init_printing(use_unicode=True)
```

Now the utility function can be constructed using sympy

```python
U = sm.Eq(x1*x2+2*x2,U)
U
```

And the budget constraint

```python
BC= sm.Eq(p1*x1 + p2*x2,I)
BC
```

Now, the goal here is to isolate x1 and x2, as we wish to find the consumers demand

```python
# Isolating x1
x1cons = sm.solve(BC,x1)
x1cons[0]
```

Now, the result here will be substituted into the utility function

```python
# Substituting into utility function
Usubx1 = U.subs(x1, x1cons[0])
Usubx1
```

This enables us to find the derivative, with respect to x2

```python
FOCx2 = sm.diff(Usubx1, x2)
FOCx2
```

Now focx2 is set equal to 0, and solved for x2

```python
SOLx2 = sm.solve(sm.Eq(FOCx2,0),x2)
```

The reverse is now done to find x1

```python
x2cons=sm.solve(BC,x2)
x2cons
```

```python
Usubx2 = U.subs(x2,x2cons[0])
Usubx2
```

```python
FOCx1=sm.diff(Usubx2,x1)
FOCx1
```

```python
SOLx1=sm.solve(sm.Eq(0,FOCx1),x1)
```

## Applying the solutions for x1 and x2

In this case, the values will be as following

```python
I = 24
p1 = 6
p2 = 1.5
```

```python
# Defining x1 and x2
def x1function(p1,p2,I):
    return (I/2 - p1)/p1

def x2function(p1,p2,I):
    return (I/2 + p1)/p2
```

Now, the values of this case will be applied

```python
# Applying values
print('x1 =',x1function(6,1.5,24))
print('x2 =',x2function(6,1.5,24))
print('U =',u(x1function(6,1.5,24),x2function(6,1.5,24)))
```

    x1 = 1.0
    x2 = 12.0
    U = 36.0

Now, a graphic display will be made to show the results

```python
# Making it possible to plot in the same graph, by isolating x2 in the 2 functions
Step1 = sm.solve(BC,x2)
Step2 = sm.solve(U,x2)
```

```python
print(Step1)
```

    [(I - p_1*x_1)/p_2]

```python
print(Step2)
```

    [U/(x_1 + 2)]

Now, the curves that are gonna be used in making the graphs, will be defined

```python
# Defining the curves
def indiff (x1,u,p1,p2):
    return u/(x1+2)

def budget (x1,I,p1,p2):
    return(I-p1*x1)/p2
```

Now the graphs will be made

```python
# Plotting the curves and optimum point (OPT)
fig = plt.figure(dpi=130)
ax = fig.add_subplot(1,1,1)
ax.plot(x1_vec,budget(x1_vec,24,6,1.5),label='Budget Constraint', color='r')
ax.plot(x1_vec,indiff(x1_vec,36,6,1.5),label='Indifference Curve', color ='g')
ax.scatter(x1function(6,1.5,24),x2function(6,1.5,24),u(x1function(6,1.5,24),x2function(6,1.5,24)),color='b',label='OPT')

# Labels and titles
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
plt.title('Consumption and optimum')
ax.legend();
```

And the values

```python
print('x1 =',x1function(6,1.5,24))
print('x2 =',x2function(6,1.5,24))
print('U =', u(x1function(6,1.5,24),x2function(6,1.5,24)))
print('I-p1*x1-p2*x2 =',24-6*x1function(6,1.5,24)-1.5*x2function(6,1.5,24))
```

    x1 = 1.0
    x2 = 12.0
    U = 36.0
    I-p1*x1-p2*x2 = 0.0

## Interactive Solution

As we want to visualize the consumption bundle, for various prices and incomes, an interactive chart is plotted, where the buydget constraint, indiff curve and optimum all rely on prices and incomes.
Furthermore, the two indifference curves above and below the optimal curve is plotted, to visualize the given change.

```python
#Defining plot
def _figure(x1_vec, I, p2, p1):
    fig=plt.figure(dpi=60)
    ax = fig.add_subplot(1, 1, 1)
    x1_vec = np.linspace(0, 2.3*x1function(p1, p2, I), 40)
    x2_vec = np.linspace(0, 2.3*x2function(p1, p2, I), 40)
    ax.plot(x1_vec,budget(x1_vec,I,p1,p2),label="BC")
    x1 = x1function(p1,p2,I)
    x2 = x2function(p1,p2,I)

    #Defining utility
    u_plot = x1*x2 + 2*x2
    ax.plot(x1_vec,indiff(x1_vec,u_plot,p1,p2),label="Indiff curve in optimum")
    ax.plot(x1_vec,indiff(x1_vec,u_plot*0.5,p1,p2),label="Indiff Curve x0.5")
    ax.plot(x1_vec,indiff(x1_vec,u_plot*1.5,p1,p2),label="Indiff Curve x1.5")
    ax.scatter(x1,x2,color='red',label='Optimum')
    plt.title('Optimal Consumption Bundle')
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')
    ax.legend();
    plt.grid(which = 'both')

#Plotting p1, p2 and I
import ipywidgets as widgets
widgets.interact(_figure,
                x1_vec = widgets.fixed(x1_vec),
                I = widgets.FloatSlider(description="$I$", min=0.1, max=50, step=0.01, value=16),
                p2 = widgets.FloatSlider(description="$p_2$", min=0.01, max=16, step=0.01, value=1),
                p1 = widgets.FloatSlider(description="$p_1$", min=0.01, max=16, step=0.01, value=1));
```

    interactive(children=(FloatSlider(value=16.0, description='$I$', max=50.0, min=0.1, step=0.01), FloatSlider(va…

We notice in the graph, that for $I \leq 2p_1$, a negative value for $x_1$ is required for a solution.

## Solution by analysing

Here, scipy will be used to solve it numerically, for the same values as before,

```python
I = 24
p1 = 6
p2 = 1.5
```

Now, onto redefining the utility function

```python
# New utility function
def U_new(x1,x2):
    return x1*x2+2*x2
```

A new indifference curve is defined, with a minimizer

```python
# New indifference curve
def I_function(x1,I,p1,p2):
    x2 = (I-p1*x1)/p2
    return -U_new(x1,x2)
```

Now, onto using the scipy solver

```python
# Scipy Solve
ScipySolve = optimize.minimize_scalar(I_function,method='bounded',bounds=(0,I/p2),args=(I,p1,p2))
```

Taking and breaking down the solution

```python
# Simplyfying the scipy solution
x1 = ScipySolve.x
x2 = (I-p1*x1)/p2
U_func = U_new(x1,x2)
```

And then onto printing!

```python
# Printing the solution
print(f'x1={x1:.1f}')
print(f'x2={x2:.1f}')
print(f'U={U_func:.1f}')
print(f'I-p1*x1-p2*x2={I-p1*x1-p2*x2:.2f}')
```

    x1=1.0
    x2=12.0
    U=36.0
    I-p1*x1-p2*x2=0.00

This result is now compared to the one we found earlier

```python
print('x1 =',x1function(6,1.5,24))
print('x2 =',x2function(6,1.5,24))
print('U =', u(x1function(6,1.5,24),x2function(6,1.5,24)))
print('I-p1*x1-p2*x2 =',24-6*x1function(6,1.5,24)-1.5*x2function(6,1.5,24))
```

    x1 = 1.0
    x2 = 12.0
    U = 36.0
    I-p1*x1-p2*x2 = 0.0

And we rightfully see, that they are identical


```python
import numpy as np
import sympy as sy
import control.matlab as cm
sy.init_printing()
```

# Zero order hold sampling of a first order system

\begin{equation}
G(s) = \frac{1}{s-\lambda}
\end{equation}

```python
h, lam = sy.symbols('h, lambda', real=True, positive=True)
s, z = sy.symbols('s, z', real=False)
```

```python
G = 1/(s-lam)
Y = G/s
Yp = sy.apart(Y, s)
Yp
```

```python
from sympy.integrals.transforms import inverse_laplace_transform
from sympy.abc import t
```

```python
inverse_laplace_transform(Yp, s, t)
```

## Sampling and taking the z-transform of the step-response
\begin{equation}
Y(z) = \frac{1}{\lambda} \left( \frac{z}{z-\mathrm{e}^{\lambda h}} - \frac{z}{z-1} \right).
\end{equation}

## Dividing by the z-transform of the input signal
\begin{equation}
H(z) = \frac{z-1}{z}Y(z) = \frac{1}{\lambda} \left( \frac{ \mathrm{e}^{\lambda h} - 1 }{ z - \mathrm{e}^{\lambda h} } \right)
\end{equation}

## Verifying for specific value of lambda

```python
lam = -0.5
h = 0.1
G = cm.tf([1], [1, -lam])
Gd = cm.c2d(G, h)
Hd = 1/lam * cm.tf([np.exp(lam*h)-1],[1, np.exp(lam*h)])
```

```python
print(Gd)
print(Hd)
```

      0.09754
    ----------
    z - 0.9512

    dt = 0.1

      0.09754
    ----------
    s + 0.9512

```python

```


# Binary classification single feature

Classification using "raw" python or libraries (SciKit Learn, Tensorflow).

The classification is first on a single boundary defined by a continuous univariate function and added white noise

```python
import math
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as pltcolors
from sklearn import metrics as metrics
from sklearn.linear_model import LogisticRegression as SkLinReg
import scipy as sy
import seaborn as sns
import pandas

import tensorflow as tf
```

## Model

We want to measure or predict a value y to be above a threshold. E.g.: y is a temperature.

We know a feature x, y is related to x through a quadratic function we do not a priori know and some unknown

This unknown is modeled by a Gaussian noise

```python
# Single feature, Gaussian noise
nFeatures = 1
def generateBatch(N):
    #
    xMin = 0
    xMax = 1
    b = 0.2
    std = 0.2
    # Threshold from 0 to 1
    threshold = 1
    #
    x = np.random.uniform(xMin, xMax, N)
    # 4th degree relation between y and x
    yClean = 2*(x**4 + (x-0.3)**3 + b)
    labels = yClean + np.random.normal(0, std, N) > threshold
    return (x, yClean, labels)
```

The values of X are uniformly distributed and independent

```python
N = 2000
# x and y have 1 dim in R, label has 1 dim in B
xTrain, yCleanTrain, labelTrain = generateBatch(N)

colors = ['blue','red']

fig = plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plt.scatter(xTrain, yCleanTrain, c=labelTrain, cmap=pltcolors.ListedColormap(colors), marker=',', alpha=0.01)
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.subplot(1,2,2)
plt.scatter(xTrain, labelTrain, marker=',', alpha=0.01)
plt.xlabel('x')
plt.ylabel('label')
plt.grid()
```

```python
count, bins, ignored = plt.hist(labelTrain*1.0, 10, density=True, alpha=0.5)
p = np.mean(labelTrain)
print('Bernouilli parameter of the distribution:', p)
```

Note: The two values are not a priori equi probable. In theory, ressampling of the training values would be required to balance the a priori distribution.

```python
xTest, yTest, labelTest = generateBatch(N)
```

## Helpers

```python
def plotHeatMap(X, classes, title=None, fmt='.2g', ax=None, xlabel=None, ylabel=None):
    """ Fix heatmap plot from Seaborn with pyplot 3.1.0, 3.1.1
        https://stackoverflow.com/questions/56942670/matplotlib-seaborn-first-and-last-row-cut-in-half-of-heatmap-plot
    """
    ax = sns.heatmap(X, xticklabels=classes, yticklabels=classes, annot=True, fmt=fmt, cmap=plt.cm.Blues, ax=ax) #notation: "annot" not "annote"
    bottom, top = ax.get_ylim()
    ax.set_ylim(bottom + 0.5, top - 0.5)
    if title:
        ax.set_title(title)
    if xlabel:
        ax.set_xlabel(xlabel)
    if ylabel:
        ax.set_ylabel(ylabel)

def plotConfusionMatrix(yTrue, yEst, classes, title=None, fmt='.2g', ax=None):
    plotHeatMap(metrics.confusion_matrix(yTrue, yEst), classes, title, fmt, ax, xlabel='Estimations', \
                ylabel='True values');
```

### Logistic and log of Logistic functions

```python
def logistic(X):
    return (1+(np.exp(-(X))))**-1
```

```python
xx = np.linspace(-10, 10)
xlogistic = logistic(xx)
plt.figure(figsize=(10,5))
plt.subplot(1, 2, 1)
plt.plot(xx, xlogistic)
plt.grid()
plt.subplot(1, 2, 2)
plt.plot(xx, np.log(xlogistic))
plt.grid()
```

# Logistic regression

\begin{align}
y \in \left\{ 0, 1 \right\}
\end{align}

\begin{align}
p(Y=1 \mid x) & = \frac{1}{1+e^{-f_\theta(x)}} \\
f_\theta(x) & = b + w x \\
\theta &= \{b, w\}
\end{align}

We are looking for the value of w that maximize the likelyhood:
\begin{align}
\hat{\theta} & = \max_{\theta}{\prod_{i=0}^N{p(y_i \mid  x_i, w)}} \\
& = \max_{\theta}{\sum_{i=0}^N{log \left(p(y_i \mid x_i, w)\right)} } \\
& = \max_{\theta}{\sum_{i=0}^N{log \left(\left(\frac{1}{1+e^{-f_\theta(x_i)}}\right)^{y_i}\left(1-\frac{1}{1+e^{-f_\theta(x_i)}}\right)^{1-y_i}\right)} } \\
& = \max_{\theta}{\sum_{i=0}^N{log \left(y_i * \left(\frac{1}{1+e^{-f_\theta(x_i)}}\right) + \left(1-y_i\right) * \left(1-\frac{1}{1+e^{-f_\theta(x_I)}}\right) \right)} } \\
\end{align}

Using the fact that $y_i$ is either 0 or 1. The last formulation is avoiding logarithm of zero as one of the two terms within the sum is null.

Since the number of classes is 2, the maximum log likelyhood is also called binary cross entropy.

Reference:
- https://en.wikipedia.org/wiki/Logistic_regression

## Fitting of $b$ and then $w$

Suboptimal fitting:
- Taking some assumption on $w$ to fit $b$ as $\hat{b}$
- and then fitting $w$ with the $\hat{b}$ estimate

```python
b = np.linspace(-5, 5)
w = 1
px = np.zeros(len(b))
for i in range(len(b)):
    fx = logistic(b[i] + w*xTrain)
    px[i] = 1/N * np.sum(np.log(labelTrain*fx + (1-labelTrain)*(1-fx)))
plt.plot(b, px);
plt.xlabel('$b$')
plt.ylabel('l(b, X)')
plt.grid()
bHat = b[np.argmax(px)]
print('Estimate b =', bHat)
```

```python
w = np.linspace(-20, 20)
px = np.zeros(len(w))
for i in range(len(w)):
    fx = logistic(bHat + w[i]*xTrain)
    px[i] = 1/N * np.sum(np.log(labelTrain*fx + (1-labelTrain)*(1-fx)))
plt.plot(w, px);
plt.xlabel('w')
plt.ylabel('l(w, X)')
plt.grid()
wHat = w[np.argmax(px)]
print('Estimate w =', wHat)
```

```python
pXTest0 = logistic(bHat + wHat * xTest)
labelEst0 = pXTest0 > 0.5
```

```python
plt.scatter(xTest, pXTest0, c=labelEst0, cmap=pltcolors.ListedColormap(colors), marker=',', alpha=0.01);
plt.scatter(xTest, yTest/np.max(yTest), c = labelTest, cmap=pltcolors.ListedColormap(colors), marker='x', alpha=0.01);
plt.xlabel('x')
plt.legend(('Estimated probability', 'Normalized model'));
```

```python
plt.hist(labelEst0*1.0, 10, density=True)
print('Bernouilli parameter =', np.mean(labelEst0))
```

```python
accuracy0 = np.sum(labelTest == labelEst0)/N
print('Accuracy =', accuracy0)
```

    Accuracy = 0.9315

### Precision
$p(y = 1 \mid \hat{y} = 1)$

```python
print('Precision =', np.sum(labelTest[labelEst0 == 1])/np.sum(labelEst0))
```

    Precision = 0.8383152173913043

### Recall
$p(\hat{y} = 1 \mid y = 1)$

```python
print('Recall =', np.sum(labelTest[labelEst0 == 1])/np.sum(labelTest))
```

    Recall = 0.9716535433070866

### Confusion matrix

```python
plotConfusionMatrix(labelTest, labelEst0, np.array(['Blue', 'Red']));
```

```python
print(metrics.classification_report(labelTest, labelEst0))
```

                  precision    recall  f1-score   support

           False       0.99      0.91      0.95      1365
            True       0.84      0.97      0.90       635

        accuracy                           0.93      2000
       macro avg       0.91      0.94      0.92      2000
    weighted avg       0.94      0.93      0.93      2000

# SciKit Learn

References:
- SciKit documentation
- https://www.geeksforgeeks.org/ml-logistic-regression-using-python/

```python
model1 = SkLinReg(solver='lbfgs')
model1.fit(xTrain.reshape(-1,1), labelTrain)
model1.coef_
```

    array([[13.08470501]])

```python
labelEst1 = model1.predict(xTest.reshape(-1,1))
print('Accuracy =',model1.score(xTest.reshape(-1,1), labelTest))
```

    Accuracy = 0.948

```python
plt.hist(labelEst1*1.0, 10, density=True)
print('Bernouilli parameter =', np.mean(labelEst1))
```

### Confusion matrix (plot)

```python
plotConfusionMatrix(labelTest, labelEst1, np.array(['Blue', 'Red']))
```

### Classification report

```python
print(metrics.classification_report(labelTest, labelEst1))
```

                  precision    recall  f1-score   support

           False       0.97      0.96      0.96      1365
            True       0.91      0.93      0.92       635

        accuracy                           0.95      2000
       macro avg       0.94      0.94      0.94      2000
    weighted avg       0.95      0.95      0.95      2000

References :
- https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8
- https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression.get_params

### ROC curve

```python
logit_roc_auc = metrics.roc_auc_score(labelTest, labelEst1)
fpr, tpr, thresholds = metrics.roc_curve(labelTest, model1.predict_proba(xTest.reshape(-1,1))[:,1])
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right");
```

# Using TensorFlow 2.0

In TensorFlow 2.0 many possibilities are available to design a sequential layer. It could be based on high level API using Keras, down to function code close to the syntax of Tensorflow 1.0.

Following design is showing how to implement a custom layer within a Sequential pipeline of Keras, and how to implement a custom metric. This is the favoured method to implement custom code in TensorFlow 2.0.

```python
# Labels as floats {0., 1.}
labelTrainF = np.multiply(labelTrain, 1.0)
labelTrainF.dtype, labelTrainF.shape
```

    (dtype('float64'), (2000,))

```python
# (Mini) Batch size
nBatch = 100
# Number of batches per Epoch
nBatchPerEpoch =20
# Number of epochs
nEpochMax = 1000
```

```python
# Simple custom layer exposing the linear regression model
class MyLogisticRegressionLayer(tf.keras.layers.Layer):
    def __init__(self, *args, **kwargs):
        super(MyLogisticRegressionLayer, self).__init__(*args, **kwargs)

    def build(self, input_shape):
        self.w = self.add_weight(
            shape=input_shape[0],
            dtype=self.dtype,
            initializer=tf.keras.initializers.ones(),
            #regularizer=tf.keras.regularizers.l2(0.02),
            trainable=True)
        self.b = self.add_weight(
            shape=1,
            dtype=self.dtype,
            initializer=tf.keras.initializers.ones(),
            #regularizer=tf.keras.regularizers.l2(0.02),
            trainable=True)

    @tf.function
    def call(self, x, training=None):
        return tf.math.sigmoid(tf.math.add(tf.math.multiply(x, self.w), self.b))
```

```python
# Using TensorFlow 2.0 style of metrics to implement accuracy
class MyBinaryAccuracy(tf.keras.metrics.Metric):

    def __init__(self, name='my_accuracy', **kwargs):
        super(MyBinaryAccuracy, self).__init__(name=name, **kwargs)
        self.accuracySum = self.add_weight(name='accuracySum',
                                           initializer='zeros')
        self.accuracyCount = self.add_weight(name='accuracyCount',
                                             initializer='zeros')

    def update_state(self, labels, yEst):
        labels = tf.cast(labels, tf.bool)
        labelEst = tf.greater(yEst, 0.5)

        values = tf.cast(tf.equal(labels, labelEst), self.dtype)
        self.accuracySum.assign_add(tf.reduce_sum(values))
        self.accuracyCount.assign_add(values.get_shape()[0])

    def result(self):
        return self.accuracySum / self.accuracyCount
```

```python
# Model 1, instantiate the custom layer
model1 = tf.keras.Sequential([MyLogisticRegressionLayer(input_shape=[nFeatures], dtype="float64")])

# Stochastic Gradient Descent Optimizer
optim1 = tf.keras.optimizers.SGD(0.01)

# Perform a train step on a mini-batch
#  This function's code is rewritten by TensorFlow 2.0 and shall be compiled at every execution of the optimizer
@tf.function
def trainStep1(x, labels):
    with tf.GradientTape() as tape:
        predictions = model1(x, training=True)
        loss = -tf.reduce_sum(tf.math.log((labels * predictions) + ((1 - labels) * (1 - predictions))))
        #loss = tf.keras.losses.categorical_crossentropy(labels, predictions)

        gradients = tape.gradient(loss, model1.trainable_variables)
        optim1.apply_gradients(zip(gradients, model1.trainable_variables))
        return loss, predictions

# Initialize values and loop on epochs and mini batch
epoch = 0
cost_epoch = 1
histo = []
accuracy = MyBinaryAccuracy()
for epoch in range(nEpochMax):
    cost_cumul = 0
    accuracy.reset_states()
    for b in range(0, nBatchPerEpoch*nBatch, nBatch):
        cost, predictions = trainStep1(xTrain[b : b + nBatch], labelTrainF[b : b + nBatch])
        cost_cumul += cost
        accuracy.update_state(labelTrainF[b : b + nBatch], predictions)

    cost_epoch = cost_cumul / nBatchPerEpoch
    W = model1.get_weights()
    histo.append((cost_epoch.numpy(), accuracy.result().numpy(), W[1][0], W[0]))

print("Predicted model: {b:.3f} + {w:.3f} x, num epochs={c}".format(w=W[0], b=W[1][0], c=len(histo)))

# Save history as a Panda Data Frame
df = pandas.DataFrame(histo, columns = ('cost', 'accuracy', 'b', 'w0'))
```

    Predicted model: -17.421 + 25.190 x, num epochs=1000

SGD shows that there is not a single optimal value for b+w (intercept + slope) but a straight line  as shown on the graph below.
This is explained by the single feature: the decision boundary does not need to be a straight line, a single intercept point would be enough.

```python
plt.scatter(df['b'], df['w0'], marker='.', alpha=0.2);
plt.xlabel('intercept')
plt.ylabel('weight');
```

```python
fig, ax = plt.subplots(1,2, figsize=(16, 4))
ax[0].plot(df['cost'])
ax[0].grid()
ax[1].plot(df['accuracy'])
ax[1].grid()
```

# Where to go from here ?

__More complex models__ with the 2 feature [binary classification](ClassificationContinuous2Features.html) ([Notebook](ClassificationContinuous2Features.ipynb)) or the [K Nearest Neighbors classifier](ClassificationContinuous2Features-KNN.html) ([Notebook](ClassificationContinuous2Features-KNN.ipynb))

__Compare with the single feature linear regression__ [using simple algorithms](../linear/LinearRegressionUnivariate.html) ([Notebook](LinearRegressionUnivariate.ipynb])), [or using Tensorflow](LinearRegressionUnivariate-TensorFlow.html) ([Notebook](LinearRegressionUnivariate-TensorFlow.ipynb))


# Sample Mean vs. Mode vs. Expected Value

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import gamma
from scipy.stats import multivariate_normal
import textwrap
import math
%matplotlib inline

def sampleMeanEx():
	'''sampleMeanEx()'''
	nSamples = (5, 10, 100, 5000)
	a = 3
	b = 1
	for i in range(len(nSamples)):
		fig = plt.figure()
		ax = fig.add_subplot(*[1,1,1])
		draws = np.random.gamma(shape=a,scale=b,size=nSamples[i])
		mode = (a-1)*b
		expectedv = a*b
		samplemean = sum(draws)/len(draws)
		x = np.linspace(gamma.ppf(0.001, a, scale=b), gamma.ppf(0.999, a, scale=b), 100)
		ax.plot(x, gamma.pdf(x, a, scale=b), 'r-', lw=5)
		ax.scatter(draws, np.zeros(len(draws)), c='k')
		myTitle = 'Num Points: ' + str(nSamples[i]) + ' Mode: ' + str("%.2f"%mode) + ' E[x]: ' + str("%.2f"%expectedv) + ' Sample Mean: ' + str("%.2f"%samplemean)
		ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))
		plt.show()

sampleMeanEx()
```

# Conjugate Priors

* Last class we mentioned the concept of *conjugate priors*
* Two distributions have a conjugate prior relationship when the form of the posterior is the same as the form of the prior.
* For example, a Gaussian distribution is a conjugate prior for the mean of a Gaussian as shown in the following:

\begin{eqnarray}
p(\mu |\mathbf{X}) &\propto& p(\mathbf{X}|\mu)p(\mu)\\
&=& \prod_{i=1}^N\mathscr{N}(x_i|\mu, \sigma^2)\mathscr{N}(\mu|\mu_0, \sigma_0^2)\\
&=& \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left\{-\frac{1}{2}\frac{(x_i - \mu)^2}{\sigma^2} \right\}\frac{1}{\sqrt{2\pi\sigma_0^2}}\exp\left\{-\frac{1}{2}\frac{(\mu - \mu_0)^2}{\sigma_0^2} \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}}\frac{1}{\sqrt{2\pi\sigma_0^2}} \exp\left\{ \sum_{i=1}^N \left(-\frac{1}{2}\frac{(x_i - \mu)^2}{\sigma^2}\right) -\frac{1}{2}\frac{(\mu - \mu_0)^2}{\sigma_0^2} \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \sum_{i=1}^N\frac{(x_i - \mu)^2}{\sigma^2} + \frac{(\mu - \mu_0)^2}{\sigma_0^2} \right) \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{\sum_{i=1}^Nx_i^2 - 2\sum_{i=1}^Nx_i\mu + \mu^2N}{\sigma^2} + \frac{\mu^2 - 2\mu\mu_0 + \mu_0^2}{\sigma_0^2} \right) \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \mu^2\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right) - 2\mu\left( \frac{\sum_{i=1}^Nx_i}{\sigma^2} + \frac{\mu_0}{\sigma_0^2}  \right) +  \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2} \right) \right\}\\
&=& \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu^2 - 2\mu\left( \frac{\sum_{i=1}^Nx_i}{\sigma^2} + \frac{\mu_0}{\sigma_0^2}  \right)\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)^{-1} \right) \right\}\\
& & \exp\left\{ \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2}  \right\} \nonumber\\
&=& \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu - \left( \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{\sigma^2\sigma_0^2} \right)\left( \frac{N\sigma_0^2 + \sigma^2}{\sigma^2\sigma_0^2}\right)^{-1}  \right)^2 \right.\\
&+& \left. \frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)
\left(
\left( \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{\sigma^2\sigma_0^2} \right)\left( \frac{N\sigma_0^2 + \sigma^2}{\sigma^2\sigma_0^2}\right)^{-1} \right)^2  \right\}
\exp\left\{ \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2}  \right\} \nonumber\\
%
&=& \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu -  \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }  \right)^2  \right\} \\
& &\exp\left\{  \frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)
\left(\frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }\right)^2 + \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2}  \right\} \nonumber\\
%
&=& C \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu -  \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }  \right)^2  \right\} \\
&\propto& \mathscr{N}\left(\mu\left|\frac{\sum_{i=1}^N x_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }, \left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)^{-1}\right.\right)
\end{eqnarray}

* So, as shown above, the form of the posterior is also a Gaussian distribution.

* There are many conjugate prior relationships, e.g., Bernoulli-Beta, Gaussian-Gaussian, Gaussian-InverseWishart, Multinomial-Dirichlet

# Bayesian Regression

* Look back our polynomial regression:
\begin{equation}
\min E^{\ast}(\mathbf{w}) = \frac{1}{2}\sum_{n=1}^N\left( y(x_n, \mathbf{w}) - t_n \right)^2 + \frac{\lambda}{2}\left\| \mathbf{w} \right\|_2^2
\end{equation}
This is equivalent to:
\begin{equation}
\max \prod_{n=1}^N \exp\left\{-\frac{1}{2}\left( y(x_n, \mathbf{w}) - t_n \right)^2 \right\}\exp\left\{- \frac{\lambda}{2}\left\| \mathbf{w} \right\|_2^2 \right\}
\end{equation}
* As discussed, the first term is the likelihood and the second term is the prior on the weights
*  These are Gaussian distributions:
\begin{equation}
\mathscr{N}(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}}\exp\left\{ -\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2} \right\}
\end{equation}
* $\sigma^2$ is the variance OR $\frac{1}{\sigma^2}$ is the *precision*
* So, as $\lambda$ gets big, variance gets smaller/tighter.  As $\lambda$ gets small, variance gets larger/wider.

* Previously, we used:
\begin{equation}
y = \sum_{j=0}^M w_j x^j
\end{equation}
*  We can extend this, to make it more general and flexible:
\begin{equation}
y = \sum_{j=0}^M w_j \phi_j(\mathbf{x})
\end{equation}
where $\phi_j(\mathbf{x})$ is a *basis function*
* For example:
    * Basis function we were using previously: $\phi_j(x) = x^j$ (for univariate $x$)
    * Linear Basis Function: $\phi_j(\mathbf{x}) = x_j$
    * Radial Basis Function: $\phi_j(\mathbf{x}) = \exp\left\{ - \frac{(x - \mu_j)^2}{2s_j^2}\right\}$
    * Sigmoidal Basis Function: $\phi_j(\mathbf{x}) = \frac{1}{1 + \exp \left\{ \frac{\mathbf{x} - \mu_j}{s}\right\}}$

* As before:
\begin{equation}
t = y(\mathbf{x},\mathbf{w}) + \epsilon
\end{equation}
* However, now:
\begin{equation}
y = \mathbf{w}^T\boldsymbol{\Phi}(\mathbf{x}) = [w_0, w_1, \ldots, w_M][\phi_0(\mathbf{x}), \phi_1(\mathbf{x}), \ldots, \phi_M(\mathbf{x})]^T
\end{equation}
where $\epsilon \sim \mathscr{N}(\cdot|0, \beta^{-1})$

\begin{equation}
p(t | \mathbf{w}, \beta) = \prod_{n=1}^N \mathscr{N}(t_n | \mathbf{w}^T\boldsymbol{\Phi}(\mathbf{x}_n), \beta^{-1})
\end{equation}
* So, what is the ``trick'' to use to maximize this?
\begin{eqnarray}
\mathscr{L} = \frac{N}{2}\ln\beta - \frac{N}{2}\ln(2\pi) - \beta E(\mathbf{w})\\
\frac{\partial \mathscr{L}}{\partial \mathbf{w}} = \beta \sum_{n=1}^N(t_n - \mathbf{w}^T\boldsymbol{\Phi}(\mathbf{x}_n))\boldsymbol{\Phi}(\mathbf{x}_n)^T = 0
\end{eqnarray}
* This results in:
\begin{equation}
\mathbf{w}_{ML} = \left(\boldsymbol{\Phi}^T\boldsymbol{\Phi} \right)^{-1}\boldsymbol{\Phi}^T\mathbf{t}
\end{equation}
where
\begin{equation}
\boldsymbol{\Phi} = \left[ \boldsymbol{\Phi}(x_1), \boldsymbol{\Phi}(x_2), ...\right]
\end{equation}

* What would you do if you want to include a prior? get the MAP solution?  If assuming zero-mean Gaussian noise, then Regularized Least Squares!

### Bayesian Linear Regression

* Recall: $E_D(\mathbf{w}) + \lambda E_W(\mathbf{w})$ where $\lambda$ is the trade-off regularization parameter
* A simple regularizer (and the one we used previously) is: $E_W(\mathbf{w}) = \frac{1}{2}\mathbf{w}^T\mathbf{w}$
* If we assume zero-mean Gaussian noise: $E_D(\mathbf{w}) = \frac{1}{2}\sum_{n=1}^N \left\{ t_n - \mathbf{w}^T\boldsymbol{\phi}(\mathbf{x}_n) \right\}^2$
* Then, the total error becomes: $\frac{1}{2}\sum_{n=1}^N \left\{ t_n - \mathbf{w}^T\boldsymbol{\phi}(\mathbf{x}_n) \right\}^2 +  \frac{\lambda}{2}\mathbf{w}^T\mathbf{w}$
* We can take the derivative, set it equal to zero and solve for the weights.  When we do, we get:
\begin{equation}
\mathbf{w} = \left( \lambda \mathbf{I} + \boldsymbol{\Phi}^T\boldsymbol{\Phi}\right)^{-1}\boldsymbol{\Phi}^T\mathbf{t}
\end{equation}
* Recall, we can interpret this as:
\begin{eqnarray}
\min_{\mathbf{w}} E^{\ast} &=& \min_{\mathbf{w}}\left\{ E_D(\mathbf{w}) + \lambda E_W(\mathbf{w}) \right\}\\
 &=& \max_{\mathbf{w}}\left\{ -E_D(\mathbf{w}) - \lambda E_W(\mathbf{w}) \right\}\\
 &=& \max_{\mathbf{w}}\exp\left\{ -E_D(\mathbf{w}) - \lambda E_W(\mathbf{w}) \right\}\\
 &=& \max_{\mathbf{w}}\exp\left\{ -E_D(\mathbf{w})\right\} \exp\left\{- \lambda E_W(\mathbf{w}) \right\}\\
 &\propto& \max_{\mathbf{w}} \prod_{n=1}^N \mathscr{N}\left(t\left|\mathbf{w}^T\mathbf{\Phi}(\mathbf{x}_n),  \beta\mathbf{I}\right.\right) \mathscr{N}\left(\mathbf{w}\left|\mathbf{m}_0, \mathbf{S}_0\right.\right)\\
 &=& \max_{\mathbf{w}} p\left(\mathbf{t}\left|\mathbf{w}, \mathbf{X}\right.\right)p\left(\mathbf{w}\right)\\
 &\propto& \max_{\mathbf{w}} p(\mathbf{w}|\mathbf{m}_N, \mathbf{S}_N)= \mathscr{N}(\mathbf{w}|\mathbf{m}_N, \mathbf{S}_N)
\end{eqnarray}
where $\mathbf{m}_N = \mathbf{S}_N\left(\mathbf{S}_0\mathbf{m}_0 + \beta\boldsymbol{\Phi}^T\mathbf{t} \right)$ and $\mathbf{S}_N^{-1} = \mathbf{S}_0^{-1} + \beta\boldsymbol{\Phi}^T\boldsymbol{\Phi}$

* What happens with different values of $\beta$ and $\mathbf{S}_0$?

* To simplify, let us assume that $\mathbf{S}_0 = \alpha^{-1}\mathbf{I}$ and $\mathbf{m}_0 = \mathbf{0}$, thus, $\mathbf{m}_N = \beta\mathbf{S}_N\boldsymbol{\Phi}^T\mathbf{t}$ and $\mathbf{S}_N^{-1} = (\alpha^{-1}\mathbf{I})^{-1} + \beta\boldsymbol{\Phi}^T\boldsymbol{\Phi} =  \alpha\mathbf{I} + \beta\boldsymbol{\Phi}^T\boldsymbol{\Phi}$
* This results in the following Log Posterior:
\begin{eqnarray}
\ln p(\mathbf{w}|\mathbf{t}) = - \frac{\beta}{2}\sum_{n=1}^N\left( t_n - \mathbf{w}^T\mathbf{\Phi}(\mathbf{x}_n)\right)^2 - \frac{\alpha}{2}\mathbf{w}^T\mathbf{w} + const
\end{eqnarray}

* Let us suppose we are dealing with 1-D data, $\mathbf{X} = \left\{ x_1, \ldots, x_N \right\}$ and a linear form for y: $y(x, \mathbf{w}) = w_0 + w_1x$

* We are going to generate synthetic data from: $t = -0.3 + 0.5x + \epsilon$ where $\epsilon$ is from zero-mean Gaussian noise. The goal is to estimate the true values $w_0 = -0.3$ and $w_1 = 0.5$.

```python
def innerBasisFunc(dataX):
	return np.vstack([dataX[i,:]@dataX[i,:].T for i in range(dataX.shape[0])])

def prodBasisFunc(dataX):
	return np.vstack([dataX[i,0]*dataX[i,1] for i in range(dataX.shape[0])])

def xorExample():
	'''xorExample()'''
	class1X = np.vstack([np.array([-1,-1])+np.random.normal(0,.1,2 ) for i in range(100)])
	class1X = np.vstack((class1X,np.vstack([np.array([1,1])+np.random.normal(0,.1,2 ) for i in range(100)])))
	class2X = np.vstack([np.array([1,-1])+np.random.normal(0,.1,2 ) for i in range(100)])
	class2X = np.vstack((class2X,np.vstack([np.array([-1,1])+np.random.normal(0,.1,2 ) for i in range(100)])))
	phi1X = prodBasisFunc(class1X)
	phi2X = prodBasisFunc(class2X)

	fig = plt.figure()
	ax = fig.add_subplot(*[1,2,1])
	ax.scatter(class1X[:,0], class1X[:,1], c='r')
	ax.scatter(class2X[:,0], class2X[:,1])
	ax = fig.add_subplot(*[1,2,2])
	ax.scatter(phi1X, np.zeros(phi1X.shape)+0.001, c='r')
	ax.scatter(phi2X, np.zeros(phi2X.shape))
	plt.show()

xorExample()
```

```python
def likelihood_prior_func():
    fig = plt.figure(figsize=(18, 16), dpi= 80, facecolor='w', edgecolor='k')

    #set up variables
    a = -0.3
    b = 0.5
    rangeX = [-1, 1]
    step = 0.025
    X = np.mgrid[rangeX[0]:rangeX[1]:step]
    alpha = 30
    beta = 2
    S0 = (1/alpha)*np.eye(2)
    draw_num = (0,1,2,3,20)

    #initialize prior/posterior and sample data
    sigma = S0
    mean = [0,0]
    draws = np.random.uniform(rangeX[0],rangeX[1],size=draw_num[-1])
    T = a + b*draws + np.random.normal(loc=0, scale=math.sqrt(1/beta))

    for i in range(len(draw_num)):
        if draw_num[i]>0: #skip first image
            #Show data likelihood
            Phi = np.vstack((np.ones(draws[0:draw_num[i]].shape), draws[0:draw_num[i]]))
            t = T[0:draw_num[i]]
            sigma = np.linalg.inv(S0 + beta*Phi@Phi.T)
            mean = beta*sigma@Phi@t

            w0, w1 = np.mgrid[rangeX[0]:rangeX[1]:step, rangeX[0]:rangeX[1]:step]
            p = multivariate_normal(t[draw_num[i]-1], 1/beta)
            out = np.empty(w0.shape)
            for j in range(len(w0)):
                out[j] = p.pdf(w0[j]+w1[j]*draws[draw_num[i]-1])

            ax = fig.add_subplot(*[len(draw_num),3,(i)*3+1])
            ax.pcolor(w0, w1, out)
            ax.scatter(a,b, c='c')
            myTitle = 'data likelihood'
            ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))

        #Show prior/posterior
        w0, w1 = np.mgrid[rangeX[0]:rangeX[1]:step, rangeX[0]:rangeX[1]:step]
        pos = np.empty(w1.shape + (2,))
        pos[:, :, 0] = w0; pos[:, :, 1] = w1
        p = multivariate_normal(mean, sigma)

        ax = fig.add_subplot(*[len(draw_num),3,(i)*3+2])
        ax.pcolor(w0, w1, p.pdf(pos))
        ax.scatter(a,b, c='c')
        myTitle = 'Prior/Posterior'
        ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))

        #Show data space
        for j in range(6):
            w0, w1 = np.random.multivariate_normal(mean, sigma)
            t = w0 + w1*X
            ax = fig.add_subplot(*[len(draw_num),3,(i)*3+3])
            ax.plot(X,t)
            if draw_num[i] > 0:
                ax.scatter(Phi[1,:], T[0:draw_num[i]])
            myTitle = 'data space'
            ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))

    plt.show()

likelihood_prior_func()

```

```python

```

Lecture05_BayesianRegression/Lecture 05 - Conjugate Priors and Bayesian Linear Regression.ipynb


# Vortex tube

A vortex tube takes in high-pressure air at 650 kPa and 305 K, and splits it into two streams at a lower pressure, 100 kPa: one at a higher temperature of 325 K and one at a lower temperature. The fraction of mass entering that leaves at the cold outlet is $f = 0.25$. The vortex tube operates continuously at steady state, is adiabatic, and performs/experiences no work. Air should be modeled as an ideal gas with constant specific heat: $R = 287$ J/kg⋅K and $c_p = 1004$ J/kg⋅K.

Figure: Vortex tube

**Problem:** Determine the temperature at the cold end. Then, determine whether this device is physically possible.

```python
# Enter the known quantities
import numpy as np
from pint import UnitRegistry
ureg = UnitRegistry()
Q_ = ureg.Quantity

gas_constant = Q_(287, 'J/(kg K)')
cp = Q_(1004, 'J/(kg K)')

temp_1 = Q_(305, 'K')
pres_1 = Q_(650, 'kPa')

temp_2 = Q_(325, 'K')
pres_2 = Q_(100, 'kPa')

f = 0.25
pres_3 = Q_(100, 'kPa')
```

First, we can find the temperature at the cold outlet by performing an energy balance on the device:
\begin{align}
\dot{m}_1 u_1 &= \dot{m}_3 u_3 + \dot{m}_2 u_2 \\
\dot{m} c_p T_1 &= f \dot{m} c_p T_3 + (1-f) \dot{m} c_p T_2 \\
T_3 &= \frac{T_1 - (1-f) T_2}{f}
\end{align}

```python
temp_3 = (temp_1 - (1-f)*temp_2) / f
print(f'Temperature at cold outlet: {temp_3: .2f}')
```

    Temperature at cold outlet: 245.00 kelvin

Now, examine whether the device is physically possible by performing an entropy balance:
\begin{align}
\dot{m} s_1 + \dot{S}_{\text{gen}} &= f \dot{m} s_3 + (1-f) \dot{m} s_2 \\
\frac{\dot{S}_{\text{gen}}}{\dot{m}} &= f s_3 + (1-f) s_2 = f(s_3 - s_2) + (s_2 - s_1)
\end{align}

We can obtain the $\Delta s$ values by using the relationship for an ideal gas with constant specific heat:
\begin{equation}
\Delta s_{1-2} = c_p \ln \left(\frac{T_2}{T_1}\right) - R \ln \left( \frac{p_2}{p_1} \right)
\end{equation}

```python
delta_s_12 = (
    cp * np.log(temp_2/temp_1) -
    gas_constant * np.log(pres_2/pres_1)
    )
delta_s_23 = (
    cp * np.log(temp_3/temp_2) -
    gas_constant * np.log(pres_3/pres_2)
    )

entropy_gen = f * delta_s_23 + delta_s_12
print(f'Entropy generation rate: {entropy_gen: .2f}')
```

    Entropy generation rate: 530.05 joule / kelvin / kilogram

Since the rate of entropy generation is positive, this device can operate as described.


```python
from cime import *
import matplotlib.pyplot as plt
from sympy.physics.mechanics import dynamicsymbols, init_vprinting
init_vprinting()
```

```python
t = symbols("t")
r2 = 3
t2 = kinvars("theta_2")
r3 = 8
t3 = kinvars("theta_3")
r1 = kinvars("r_1")
t1 = pi
```

```python
v1 = Vector2D(r1,t1)
v2 = Vector2D(r2,t2)
v3 = Vector2D(r3,t3)

s = VectorLoop(v2,v3,v1)
s
```

$$ \begin{bmatrix} {- r_{1} + 3 \operatorname{cos}\left(\theta_{2}\right) + 8 \operatorname{cos}\left(\theta_{3}\right)} \\ {3 \operatorname{sin}\left(\theta_{2}\right) + 8 \operatorname{sin}\left(\theta_{3}\right)} \end{bmatrix} $$

```python
ssol = s.solve([r1,t3],{t2:rad(40).evalf()})
ssol
```

```python
ssol
```

```python
s.draw({r1:ssol[0][0], t3:ssol[0][1], t2:rad(40).evalf()})
s.draw({r1:ssol[1][0], t3:ssol[1][1], t2:rad(40).evalf()})
plt.axis("equal")
```

```python
r1 = Vector2D(x=10,y=10)
r2 = Vector2D(x=10,y=-10)
r3 = Vector2D(x=-20,y=0)
VectorLoop(r1,r2,r3).draw()
```

```python

```


# Harmonic oscillator

```python
import sympy
sympy.init_printing()
from IPython.display import display
import numpy
import matplotlib.pyplot as plt

import sys
sys.path.insert(0, './code')

from gauss_legendre import gauss_legendre
from evaluate_functional import evaluate_functional
```

```python
# state vector (symbolic)
q = sympy.Symbol('q')
p = sympy.Symbol('p')
x = [q, p]

# structure matrix
J = sympy.Matrix([[0, 1],
                  [-1, 0]])

# physical parameters of Hamiltonian function
# (system mass and spring stiffness)
m = sympy.Symbol('m')
k = sympy.Symbol('k')
params = {
    m: 0.1,
    k: 3.0,
}

# Hamiltonian function
H = sympy.Rational(1,2) * p * 1/m * p + sympy.Rational(1, 2) * q * k * q

# storage efforts
dH = sympy.Matrix([H.diff(x_i) for x_i in x])

# time evolution
xdot = J @ dH

# initial condition
x_0 = numpy.array([numpy.pi/6, 0.0])

# duration of simulation in seconds
t_f = 30.0

# time step in seconds
dt = 5e-2

# order of collocation polynomial
s = 2
```

```python
# run the simulation
%time time, solution = gauss_legendre(x, xdot, x_0, t_f, dt, s=s, params=params)
```

    CPU times: user 212 ms, sys: 3.56 ms, total: 215 ms
    Wall time: 215 ms

```python
fig, ax = plt.subplots(nrows=2, dpi=200)
fig.tight_layout(pad=1.5)

# upper plot shows q over t
ax[0].set_title("q")
ax[0].plot(time, solution[:, 0])

# plot analytical solution for q in dashed
ω = numpy.sqrt(params[k] / params[m])
ax[0].plot(time, solution[0,0]*numpy.cos(time*ω), '--')

# lower plot shows p over t
ax[1].set_title("p")
ax[1].plot(time, solution[:, 1])
ax[1].set_xlabel("time");
```

```python
# relative error of energy conservation
energy = evaluate_functional(x, H, solution, params=params)
abs(energy[0] - energy[-1]) / energy[0]
```


# Information Retrieval in High Dimensional Data
### Lab \#10, 25.01.2018

## CVXOPT

a) Go to cvxopt.org and follow the installation instructions for your distribution.
For conda, you need to run

conda install -c conda-forge cvxopt

b) Skim through the Examples section on cvxopt.org to get an overview of the functionality of the different solvers of CVXOPT.

c) Implement a function minsq which expects a NumPy array A of shape (m, n) and a NumPy array y of shape (m,) as its arguments and returns a NumPy array x of shape (n,) that solves the following problem

\begin{equation} \min\limits_x \| Ax-y\|. \end{equation}

Test your function by feeding it with appropriate inputs an comparing the results with the one syou get by using np.linalg.pinv. Experiment by adding white Gaussian noise to y.

```python
import numpy as np
from cvxopt import matrix, solvers
```

```python
def minsq(A,y):
    m, n = A.shape
    P = matrix(np.eye(n).astype(dtype=np.float_))
    q = matrix(np.zeros(n).astype(dtype=np.float_))
    G = matrix(np.zeros((m, n)))
    h = matrix(np.zeros((n, 1)))
    A = matrix(A.astype(dtype=np.float_))
    y = np.expand_dims(y, axis = 1)
    b = matrix(y.astype(dtype=np.float_))
    print(y.shape)
    sol = solvers.qp(P,q, G, h, A=A, b=b, solver='MOSEK')
    return sol['x']
```

```python
A = np.array([[1, 2, 4], [4, 9, 6], [7, 8, 4]], dtype='float64')
y = np.array([3, 5, 9], dtype='float64')
```

```python
A_inv = np.linalg.pinv(A)
```

```python
x = np.dot(A_inv, y)
x
```

    array([ 1.57142857, -0.57142857,  0.64285714])

```python
x = minsq(A, y)
print(x)
```

    (3, 1)
         pcost       dcost       gap    pres   dres
     0:  1.6046e+00  1.6046e+00  3e+00  2e+00  0e+00
     1:  1.6046e+00  1.6046e+00  3e-02  2e-02  2e-16
     2:  1.6046e+00  1.6046e+00  3e-04  2e-04  5e-16
     3:  1.6046e+00  1.6046e+00  3e-06  2e-06  1e-15
     4:  1.6046e+00  1.6046e+00  3e-08  2e-08  1e-15
    Optimal solution found.
    [ 1.57e+00]
    [-5.71e-01]
    [ 6.43e-01]

```python

```


# Chapter 8: Transformations

This Jupyter notebook is the Python equivalent of the R code in section 8.8 R, pp. 373 - 375, [Introduction to Probability, 2nd Edition](https://www.crcpress.com/Introduction-to-Probability-Second-Edition/Blitzstein-Hwang/p/book/9781138369917), Blitzstein & Hwang.

----

```python
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
```

## Beta and Gamma distributions

```python
from scipy.stats import beta, gamma

# to learn more about scipy.stats.beta, un-comment ouf the following line
#print(beta.__doc__)

# to learn more about scipy.stats.gamma, un-comment ouf the following line
#rint(gamma.__doc__)
```

The Beta and Gamma distributions are implemented in [`scipy.stats.beta`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.beta.html) and [`scipy.stats.gamma`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gamma.html), respectively.

* To evaluate the $Beta(a, b)$ PDF or CDF at $x$, we use `beta.pdf(x, a, b)` and `beta.cdf(x, a, b)`. To generate $n$ realizations from the $Beta(a, b)$ distribution, we use `beta.rvs(a, b, size=n)`.
* To evaluate the $Gamma(a, \lambda)$ PDF or CDF at $x$, we use`gamma.pdf(x, a, scale=1/lambd)` or `gamma.cdf(x, a, scale=1/lambd)`. To generate $n$ realizations from the $Gamma(a, \lambda)$ distribution, we use `gamma.rvs(a, scale=1/lambd, size=n)`.
   * The $\lambda$ parameter in $Gamma(a, \lambda)$ corresponds in `gamma` to using `scale` = $\frac{1}{\lambda}$ and default value of `loc` = 0.

For example, we can check that the $Gamma(3, 2)$ distribution has mean $\frac{3}{2}$ and variance $\frac{3}{4}$. To do this, we generate a large number of $Gamma(3, 2)$ random variables using `gamma.rvs`, then compute their mean and var using their corresponding methods in `numpy.array` (you could of course also use `numpy.mean` and `numpy.var`, passing in the array of r.v.):

```python
# seed the random number generator
np.random.seed(317811)

alpha = 3.0
lambd = 2.0

y = gamma.rvs(alpha, scale=1/lambd, size=10**5)

mean = y.mean()
#mean = np.mean(y)
print('mean of Gamma(3, 2) = {}'.format(mean))

var = y.var()
#var = np.var(y)
print('variance of Gamma(3, 2) = {}'.format(var))
```

    mean of Gamma(3, 2) = 1.5012880719327166
    variance of Gamma(3, 2) = 0.7512319256875035

**NOTE**: [`lambda`](https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions) in Python is a reserved keyword for declaring small, anonymous functions, and so we cannot used the name `lambda` for variable or functions.

Try changing the `numpy.random.seed` input value in the code block above, and hit SHIFT+ENTER to re-execute the code block. Did you get values that are close to 1.5 and 0.75, respectively?

## Convolution of Uniforms

Using [`scipy.stats.uniform`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.uniform.html) and [`matplotlib.pyplot.hist`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.hist.html), we can quickly verify that for $X, Y \stackrel{i.i.d.}{\sim} Unif(0, 1)$, the distribution of $T = X + Y$ is triangular in shape:

```python
np.random.seed(514229)

from scipy.stats import uniform

x = uniform.rvs(size=10**5)
y = uniform.rvs(size=10**5)

t = x + y

plt.hist(t, bins=20)

plt.title(r'Histogram of $T = X + Y$')
plt.xlabel(r'$x + y$')
plt.ylabel('Frequency')

plt.show()
```

The histogram looks like an ascending and then descending staircase, a discrete approximation to a triangle.

## Bayes' billiards

In the Bayes' billiards story, we have $n$ white balls and 1 gray ball, throw them onto the unit interval completely at random, and count the number of white balls to the left of the gray ball. Letting $p$ be the position of the gray ball and $X$ be the number of white balls to the left of the gray ball, we have

\begin{align}
  p &\sim Unif(0, 1) \\
  X|p &\sim Bin(n, p)
\end{align}

By performing this experiment a large number of times, we can verify the results we derived in this chapter about the marginal PMF of $X$ and the posterior PDF of $p \text{ given } X = x$. We'll let the number of simulations be called `nsims`, to avoid a name conflict with the number of white balls, `n`, which we set equal to 10:

```python
nsims = 10**5
n = 10
```

We simulate 105 values of $p$, then simulate 105 values from the conditional distribution of $X$ given $p$:

```python
np.random.seed(832040)

p = uniform.rvs(size=nsims)

from scipy.stats import binom

x = binom.rvs(n, p, size=nsims)
```

Notice that we feed the entire array `p` into `binom.rvs`. This means that the first element of `x` is generated using the first element of `p`, the second element of `x` is generated using the second element of `p`, and so forth. Thus, conditional on a particular element of `p`, the corresponding element of `x` is Binomial, but the elements of `p` are themselves Uniform, exactly as the model specifies.

According to the Bayes' billiards argument, the marginal distribution of $X$ should be Discrete Uniform on the integers 0 through $n$. Is this in fact the case? We can make a histogram of `x` to check! Because the distribution of $X$ is discrete, we can specify `bins=np.arange(0, n+1, 0.5), align='left'` in the call to `matplotlib.pyplot.hist` so that each bar is centered at an integer value:

```python
plt.hist(x, bins=np.arange(0, n+1, 0.5), align='left')

plt.title(r'Histogram of $x$')
plt.xlabel(r'$x$')
plt.ylabel('Frequency')
# also set the x-axis ticks to show integers 0, 1, 2, ..., 10
plt.xticks(range(0, 10+1))

plt.show()
```

Indeed, all the histogram bars are approximately equal in height, consistent with a Discrete Uniform distribution.

Now for the posterior distribution of $p$ given $X = x$. Conditioning is very simple since we are using `numpy.array`. To consider only the simulated values of $p$ where the value of $X$ was 3, we use [`numpy.where(x==3)`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html) to find the indices where `x` = 3, and then use that within the square brackets of `p` to index only those corresponding values. In particular, we can create a histogram of these values using `matplotlib.pyplot.hist` to see what the posterior distribution of $p$ given $X = 3$ looks like.

According to the Beta-Binomial conjugacy result, the true posterior distribution is $p|X = 3 \sim Beta(4, 8)$. We can plot the histogram of `p[numpy.where(x==3)]` next to a histogram of simulated values from the $Beta(4, 8)$ distribution to confirm that they look similar:

```python
np.random.seed(1346269)

fig = plt.figure(figsize=(14, 6))

p_given_x_3 = p[np.where(x==3)]

ax1 = fig.add_subplot(121)
ax1.hist(p_given_x_3, bins=20, color='#e66101')
ax1.set_title(r'Histogram of $\tt{p[numpy.where(x==3)]}$')
ax1.set_ylabel('Frequency')

brv = beta.rvs(4.0, 8.0, size=10**4)

ax2 = fig.add_subplot(122)
ax2.hist(brv, bins=20, color='#5e3c99')
ax2.set_title(r'Histogram of $Beta(4, \, 8)$, $10^4$ simulated values')
ax2.set_ylabel('Frequency')

plt.show()
```

For the side-by-side graph above, we obtain an instance of [`matplotlib.pyplot.figure`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.figure.html) from `pyplot`, specifying the figure's width and height. Next, we use [`figure.add_subplot(121)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplot.html) to create graph axes in a figure with 1 row and 2 columns, in the 1st column position from the left, for the histogram of `p[numpy.where(x==3)]`. Likewise, `figure.add_subplot(122)` creates the graph axes for the histogram for $Beta(4, 8)$ in the 2nd column position of the 1-row, 2-column figure.

## Simulating order statistics

Simulating order statistics in Python/NumPy/SciPy is easy: we simply simulate i.i.d. r.v.s and sort them in order. For example,

```python
np.random.seed(2178309)

from scipy.stats import norm

rv = np.sort(norm.rvs(size=10))
print(rv)
```

    [-2.05800632 -1.14300612 -0.09108752 -0.08188422  0.45810851  0.55556877
      0.59398037  0.9312065   0.98367818  1.12202827]

produces one realization of $X_{(1)}, \ldots, X_{(10)}$, where $X_{(1)}, \ldots, X_{(10)}$ are i.i.d. $N(0, 1)$. If we want to plot a histogram of realizations of, say, $X_{(9)}$, we'll need to use iteration. Let's try using [Python's list comprehensions](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions):

```python
np.random.seed(3524578)

rv_arr = [np.sort(norm.rvs(size=10)) for _ in range(10**4)]
order_stats = np.array(rv_arr)

print('order_stats has shape: {}'.format(order_stats.shape))
```

    order_stats has shape: (10000, 10)

The 104 iterations are done with

    rv_arr = [np.sort(norm.rvs(size=10)) for _ in range(10**4)]

This Python list comprehension does the following:
* loops 104 times with `for _ in range(10**4)`, where we ignore the iteration counter by using `_`
   * as we loop, we do not need to know which particular iteration we are on
* for each iteration, calculate `np.sort(norm.rvs(size=10))`
* gather up the results of all the iterations into a Python list (array), assigning this resulting list to `rv_arr`
* this one-liner list comprehension is equivalent to all this code:

    rv_arr = []
    for _ in range(10**4):
        rv = np.sort(norm.rvs(size=10))
        rv_arr.append(rv)

This creates a matrix, `order_stats`, with 104 rows and 10 columns. The _i_th column of the matrix contains 104 realizations of $X_{(i+1)}$ (don't forget that Python is zero-indexed, so the 8th column is $X_{(9)}$!). Now we can create a histogram of $X_{(9)}$, simply by selecting column 8 of the matrix:

```python
x9 = order_stats[:, 8]

plt.hist(x9, bins=20)

plt.xticks(np.arange(-1, 4, 1))
plt.title(r'Histogram of $X_{(9)}$')
plt.ylabel('Frequency')

plt.show()
```

We can also compute summaries like `numpy.mean(x9)` and `numpy.var(x9)`.

```python
mean_x9 = np.mean(x9)
print('mean of x9 = {}'.format(mean_x9))

var_x9 = np.var(x9)
print('variance of x9 = {}'.format(var_x9))
```

    mean of x9 = 1.0092010380841254
    variance of x9 = 0.2108867888683199

----

Joseph K. Blitzstein and Jessica Hwang, Harvard University and Stanford University, © 2019 by Taylor and Francis Group, LLC


# Some uses for SVD

**Unsupervised learning: dimensionality reduction**

**Abstract:** Singular value decomposition (SVD) is referred to by many as kind of a linear algebra "Swiss army knife". The motivation for this notebook is to show that SVD can in fact find versitile applications in analytics. This notebook gives a short overview of what SVD is and demonstrates two uses for it, namely (1) solving the einproblem that comes with PCA and, (2) compressing, say, images. Be warned, the PCA part includes some mathematical explanations to connect some dots. If math notation doesn't really rock your boat, feel free to skip that part.

**Topics covered:** singular value decomp (SVD), principal component analysis (PCA), compression.

**So, what's SVD?** We can rewrite any $n \times p$ matrix $X$, with $n > p$, as the product of three components, namely two orthogonal matrices and one diagonal matrix. These components together is what we refer to as the _singular value_ of the matrix in question.

Take for instance the real-valued matrix $X \in \mathbb{R}^{n \times p}$ with $n$ rows and $p$ columns. Let $s$ be the smaller of two numbers, i.e. $s = \min(n, p)$. The SVD of $X$ is the factorization

$$
X = U \cdot \Sigma \cdot V^T
$$

where $U \in \mathbb{R}^{n \times s}$, $\Sigma \in \mathbb{R}^{s \times s}$, and $V^T \in \mathbb{R}^{s \times p}$.

Let's unpack these three components.

$U$ and $V$ are both orthogonal, unitary matrices which means their transpose are also their inverse. In other words, $U^T U = I$ and $V^T V = I$ where $I$ is the identity matrix.

The columns of $U$ are known as the _left singular vectors_ and the rows of $V^T$ as the _right singular vectors_ of $X$. If we denote these vectors by $u_i$ and $v_i$, then

$$
\begin{array}{rcl}
  U & = & \begin{bmatrix} u_0 & u_1 & \cdots & u_{s-1} \end{bmatrix} \\
  V^T & = & \begin{bmatrix} v_0 & v_1 & \cdots & v_{s-1} \end{bmatrix}^T.
\end{array}
$$

$\Sigma$ is a diagonal matrix. Its diagonal entries are, collectively, called the _singular values_ of $X$, and are sorted from highest to lowest, i.e. $\sigma_0 \geq \sigma_1 \geq .... \geq \sigma_{s-1}$

$$
\begin{equation}
\Sigma =
\begin{bmatrix}
  \sigma_0 &          &        & \\
           & \sigma_1 &        & \\
           &          & \ddots & \\
           &          &        & \sigma_{s-1}
\end{bmatrix}
\end{equation}.
$$

**Aha, so what?** From these definitions we imply that

$$
X V = U \Sigma.
$$

This is a compact way of writing a system of independent vector equations, namely

$$
\begin{array}{rcl}
  X v_i & = & \sigma_i u_i.
\end{array}
$$

It kind of looks like an eigenproblem already, doesn't it?

The neat thing about SVD, in contrast to other orthogonal transformations like QR decomposition, is that it deals with _both_ the rows and columns of $X$ in the same way. This symmetry provides us with more information about $X$. SVD also 'orders' the information in $X$, letting us see the more salient information first.

**And... what's it good for?** Well, for one, dimensionality reduction! We can use SVD to come up with a smaller representation of $X$ that still preserves its "mass". Two cool applications for it are:

* We can use the rows of $V^T$ (the right singular vectors) as a solution to the eigenproblem that comes with PCA.
* We can use the entries of $\Sigma$ (the singular values) for compressing, say, images. This provides us with a way of measuring the relative reconstruction error compared to the original image.

We'll demonstrate these two applications next.

# Demo No. 1: PCA

Principal component analysis lets us come up with a lower-dimensional, but equally informative representation of the data. Essentially we transform the data into orthogonal dimensions ranking them by variance. This is useful for visualization and discovering hidden structures in the data. Here an attempt to prove why we can use the rows of $V^T$ as the principal components (PCs).

(If you don't want to get into any "nasty" math, start to skip here).

In PCA we want to find the projection-maximizing set of axes. This computation can be reduced to an eigenproblem. We can use SVD to find the eigenpair(s) that solve the PCA eigenproblem.

The PCA cost function is

$$
\begin{array}{rcl}
  \underset{\varphi, \lambda}{\arg\max} \, \hat{F}(\varphi, \lambda)
  = F(\varphi) + \frac{\lambda}{2} (1 - \varphi^T \varphi)
\end{array}
$$

and $\varphi$ is the unit-length vector we use to project the centered data: $X - \mu_X$. Computing the partial derivatives and solving this Lagrange objective function for $(\varphi^*, \lambda^*)$ yields

$$
\begin{array}{rcl}
  \frac{1}{m} X^T X \varphi & = & \lambda \varphi \\
  \| \varphi \|_2^2 & = & 1.
\end{array}
$$

Note that $\frac{1}{m} X^T X$ is nothing but the sample covariance matrix.

If we insert the SVD into the first equation above, we end up with something really neat, namely

$$
X^T X V = V \Sigma^2,
$$

or, as a system of vector equations,

$$
\begin{array}{rcl}
  X^T X v_i & = & \sigma_i^2 v_i.
\end{array}
$$

As we can see, $v_i$ is the eigenvector of the covariance matrix of $X$ and $\sigma_i^2$ its eigenvalue. And every eigenpair $(v_i, \sigma_i^2)$ is a solution to the eigenproblem. The pair with the largest eigenvalue is $(v_0, \sigma_0^2)$, i.e. the first row of $V^T$ and the first entry of $\Sigma$.

(If you skipped, let's reconvene here).

**In a nutshell.** The right singular vectors $v_0, v_1, \ldots, v_{s-1}$, the rows of $V^T$, are the directions of the PCs. The columns of $U \Sigma$, $u_0\sigma_0, u_1\sigma_1, \ldots, u_s\sigma_s$, are the PCs themselves. Say we want to understand the nature in the variation of the first $k$ PCs, we can simply examine the first $k$ rows of $V^T$ to see what stands out.

Let's see how this looks in practice. We take an example from this cool blog http://setosa.io/ev/principal-component-analysis/ which provides a visual motivation for PCA using some "old" data from a nutrition study done in the UK. For more details on the data and the analysis just go to this site!

**Imports.**

```python
import numpy as np
from numpy.linalg import svd
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
```

**Load data & visualize.**

```python
df = pd.read_csv('data/uk-nutrition-data.csv')
print('DataFrame with {} rows x {} columns:'.format(df.shape[0], df.shape[1]))
display(df)
```

    DataFrame with 17 rows x 5 columns:

      Product
      England
      Northern Ireland
      Scotland
      Wales

      0
      Alcoholic drinks
      375
      135
      458
      475

      1
      Beverages
      57
      47
      53
      73

      2
      Meat
      245
      267
      242
      227

      3
      Cereals
      1472
      1494
      1462
      1582

      4
      Cheese
      105
      66
      103
      103

      5
      Sweets
      54
      41
      62
      64

      6
      Fats and oils
      193
      209
      184
      235

      7
      Fish
      147
      93
      122
      160

      8
      Fresh fruit
      1102
      674
      957
      1137

      9
      Fresh potatoes
      720
      1033
      566
      874

      10
      Fresh veggies
      253
      143
      171
      265

      11
      Other meat
      685
      586
      750
      803

      12
      Other veggies
      488
      355
      418
      570

      13
      Processed potatoes
      198
      187
      220
      203

      14
      Processed veggies
      360
      334
      337
      365

      15
      Soft drinks
      1374
      1506
      1572
      1256

      16
      Sugars
      156
      139
      147
      175

```python
prods = df['Product']
ctries = df.columns.difference(['Product'])

fig, ax = plt.subplots(1, 4, figsize=(12, 6), sharey=True)
for i in range(len(ctries)):
    sns.barplot(x=ctries[i], y='Product', data=df, ax=ax[i])
    ax[i].set_ylabel('')
fig.suptitle('Grams per Week per Person')
```

Can you spot the differences between the four countries? If you stare at this graph long enough, well... you'll go crazy. :) Instead we can work with a lower-dimensional representation of this data, say 2 PCs instead of 17 features, and see if that helps.

**Compute the SVD of X.**

```python
# Extract numpy ndarray from DataFrame
X_raw = df[ctries].values.T # n=4, p=17
# Center the data
X = X_raw - np.mean(X_raw, axis=0)
n, p = X.shape
print('X:', X.shape)
# Compute s
s = min(X.shape[0], X.shape[1])
print('s:', s)
```

    X: (4, 17)
    s: 4

```python
# Compute SVD
U, S, VT = svd(X, full_matrices=False)
# Inspect SVD components
print('U:', U.shape)
print('Sigma:', S.shape)
print('VT:', VT.shape)
```

    U: (4, 4)
    Sigma: (4,)
    VT: (4, 17)

**Compute & plot first two right singular vectors.** As discussed above, we take the first two rows of $V^T$ here, as these decribe the direction (or movement) of the first two PCs. Examining $v_0$ and $v_1$ will provides clues as to what features explain most of the variation in the data. And by plotting the linear combinations $Xv_0$ and $Xv_1$ we'll be able to see differences in the variances between the countries.

Aside: Another 'goodie' from thinking of PCA in terms of SVD is that this uncovers interesting properties about the PCs. For instance, the linear combination $Xv_0$ has the highest variance among all linear combinations of the features. $Xv_1$ has the highest variance among all linear combinations satisfying $v_1$ orthogonal to $v_0$, and so on.

```python
# Extract first two singular vectors
v_0 = VT[0, :].T
v_1 = VT[1, :].T
v_s = [v_0, v_1]
print('0th right singular vector:', '\n', v_0, '\n')
print('1st right singular vector:', '\n', v_1)
```

    0th right singular vector:
     [-0.46396817 -0.02618776  0.04792763 -0.04770286 -0.05695538 -0.0296502
     -0.00519362 -0.08441498 -0.6326409   0.40140206 -0.15184994 -0.25891666
     -0.24359373 -0.02688623 -0.03648827  0.23224414 -0.03762098]

    1st right singular vector:
     [-0.11353652  0.03056054 -0.01391582  0.21259968 -0.01601285 -0.00594992
      0.09538866  0.05075495  0.17774074  0.71501708  0.14490027  0.01533114
      0.22545092 -0.04285076  0.0454518  -0.55512431  0.0430217 ]

Each right singular vector is of length $p=17$. Each entry in each vector $v_k$ tells us the magnitude each product has on the respective PC. We can inspect the absolute numbers to see which products have the greatest magnitude.

```python
# Plot
fig, ax = plt.subplots(1, 2, figsize=(13,5), sharey=True, sharex=True)
for i in range(2):
    ax[i].bar(np.arange(v_s[i].shape[0]), height=np.absolute(v_s[i]), color='c')
    ax[i].set_xticks(range(len(prods)))
    ax[i].set_title(f'v_{i}: feature magnitudes over {i}-th PC')
```

In case of $v_0$ the 0th, 8th and 9th entries carry the greatest magnitude. In case of $v_1$ it's the 9th and 15th entries. By looking back at the data we can tell which products these correspond to.

```python
print('Entries in v_0 with the greatest magnitude:\n')
print(prods[[0, 8, 9]], '/n')

print('\nEntries in v_1 with the greatest magnitude:\n')
print(prods[[9, 15]], '\n')
```

    Entries in v_0 with the greatest magnitude:

    0    Alcoholic drinks
    8         Fresh fruit
    9      Fresh potatoes
    Name: Product, dtype: object /n

    Entries in v_1 with the greatest magnitude:

    9     Fresh potatoes
    15       Soft drinks
    Name: Product, dtype: object

Finally, we can take the 17-dimensional data and project it onto the 2-dimensional subspace defined by $v_0$ and $v_1$ to see the difference between the countries.

```python
# Project into 2D
X_proj = X.dot(VT[0:2, :].T)

# Plot
plt.figure(figsize=(6, 6))
plt.scatter(X_proj[:, 0], X_proj[:, 1], color='c', s=200)
for x_0, x_1, c in zip(X_proj[:, 0], X_proj[:, 1], ctries):
    plt.annotate(c, xy=(x_0, x_1))
plt.xlabel('X projected onto v_0 (horizontal movement)')
plt.ylabel('X projected onto v_1 (vertical movement)')
```

**Outcome.** By plotting using the first two right singular vectors, $v_0$ and $v_1$, we can see clearly that Northern Ireland is quite different from the other three countries in the UK.

# Demo No. 2: Image compression

When we compress an image we end up with a lower-resolution equivalent of the original. If done well, i.e. with a low relative error, the compressed representation might be deceivingly hard to distinguish from the original - at least for the human eye. And, even though they are not identical, the low-resolution compression might be equally usable and, because it's considerably smaller, saves computational resources.

NB: This type of compression is known as _lossy_ compression since the resulting compressed image is a degraded version of the original. There is a degration or _distortion_ taking place.

**Using SVD for compression.** If we represent an image as a real-valued matrix $X \in \mathbb{R}^{n \times p}$, then we can approximate $X$ with its $k$-truncated SVD, i.e. $U_k \Sigma_k V_k^T$ for $0<k<s$. Recall that $s = \min(n, p)$. If the size of its representation, $\tilde{X}$, is lower than the original, then we have compressed $X$.

The squared error of the compressed representation, measured in the Frobenius norm, is given by the sum of the trailing singular values, namely

$$
  \|X - U_k \Sigma_k V_k^T\|_F^2 = \sigma_{k}^2 + \sigma_{k+1}^2 + \cdots + \sigma_{s-1}^2
  .
$$

Consequently, the _relative_ error, can be measured as

$$
  \dfrac{\|X - U_k \Sigma_k V_k^T\|_F}
        {\|X\|_F}
        .
$$

$\|X\|_F$ is the Frobenius norm of $X$ which is given by the sum of all singular values; the diagonal entries of $\Sigma$.

**Display original image.** I took one image from today's "NASA Image of the Day" of a solar solar eclipse visible in Antarctica on Dec. 4, 2021. https://www.nasa.gov/multimedia/imagegallery/iotd.html

```python
from PIL import Image
img_raw = Image.open('data/epic_1b_20211204075803.png', 'r').convert('L')
img = np.asarray(img_raw)
plt.figure(figsize=(6,6))
plt.imshow(img, interpolation='nearest', cmap='gray')
```

**Compress via SVD.** Say we want to compress using the 15 biggest singular values, i.e. $k=15$, then the relative reconstruction error is 16.4%.

```python
# Compute the SVD of the image
U, S, VT = svd(img, full_matrices=False)
```

```python
# Get the k-th truncation of U and VT
k = 15
Uk = U[:, :k]
VkT = VT[:k, :]
print('Uk:', Uk.shape)
print('S:', S.shape)
print('VkT:', VkT.shape)
```

    Uk: (2048, 15)
    S: (2048,)
    VkT: (15, 2048)

```python
# Compute relative reconstruction error
err = np.sqrt(np.sum(np.square(S[k:]))) # trailing k singular values
norm = np.sqrt(np.sum(np.square(S)))
rel_err = err / norm
print('% .3f' % rel_err)
```

     0.164

**Viz original v compressed representation.**

```python
# Reconstruct compressed image
img_lossy = Uk @ np.diag(S[:k]) @ VkT # first k singular values
```

```python
fig, ax = plt.subplots(1, 2, figsize=(12, 6))
ax[0].imshow(img, interpolation='nearest', cmap='gray')
ax[0].set_title('Original')
ax[1].imshow(img_lossy, interpolation='nearest', cmap='gray')
ax[1].set_title('Compressed Representation')
```

**Outcome.** Using the 15 biggest singular values, opposite to using all 2,048 of them, we lose only about 16% of the original image. Depending on the application we might want to go for a lower relative reconstruction error.


# Music machine learning - Introduction

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A [first definition](#definition) on the concept of machine learning
2. An introduction to a simple problem of [linear regression](#regression)
3. An explanation on [model capacity and overfitting](#capacity)
4. An introduction to the [audio datasets](#audio) that we will use

## Defining machine learning

In all natural process, there exists complex relations between sets $\mathcal{X} \mapsto \mathcal{Y}$. This can relate some objects with their names, or a cause to a consequence. In most cases, _we do not know the precise relations_ between these sets, all we have is _observations_ such as pairs $(x,y)$, composed of input data $x \in \mathcal{X}$, which have a corresponding expected output $y \in \mathcal{Y}$. The overarching goal of machine learning is to approximate such _unknown processes_ as a function $\mathcal{F}_{\theta}$, which _transforms_ input data $x$ into output data $y$.

Hence, machine learning aims to understand and model the relationship between some (usually complex and high-dimensional) inputs $\mathbf{x}\in\mathcal{X}\subset\mathbb{R}^{\mathcal{X}}$ and outputs $\mathbf{y}\in\mathcal{Y}\subset\mathbb{R}^{\mathcal{Y}}$, given by a set of data examples $\mathcal{D}=\left\{(x_1,y_1),\cdots,(x_N,y_N)\right\}$. This is achieved by defining a parametric model $f_{\mathbf{\theta}}\in\mathcal{F}$ inside a family of functions $\mathcal{F}$, which depends on parameters $\mathbf{\theta} \in \mathbf{\Theta}$ and that could approximate the underlying relationship. The _learning_ aspect refers to the adjustment of the parameters $\mathbf{\theta}$ in order to obtain the best approximation of the given task
$$
\begin{equation}
f_{\mathbf{\theta}}(\mathbf{x}) = \hat{\mathbf{y}}\approx \mathbf{y}.
\end{equation}
$$

To observe this idea in simple setups, we are going to use the `numpy` library and `matplotlib` for plotting. We also set some fixed random generator, to ensure reproducibility

```python
# Base imports
import numpy as np
import matplotlib.pyplot as plt
from helper_plot import prep_plots, finalize_plots, hdr_plot_style
rng = np.random.RandomState(1)
```

## Simple learning problem

Imagine that a certain process somewhere follows the form of a quadratic relationship

$$
 y = a x^{2} + bx + c
$$

In this case, all the **unknown parameters** are that of a polynomial model, therefore we have $\theta = \{a, b, c\}$. However, this is clearly an ideal (clean) case, whereas in natural observations, there might be some noise in our observations
$$
 y = a x^{2} + bx + c +\epsilon \quad \mbox{with} \quad \epsilon \in [-0.1, 0.1]
$$

An example of such noisy observations for different parameters is given below

```python
noise_level = 0.1
a, b, c = -2, 0, 1
# Generating
x = np.linspace(0, 1, 100)
poly = np.poly1d([a, b, c])
epsilon = np.random.uniform(-noise_level,noise_level,x.shape)
y = poly(x) + epsilon
ax1, = prep_plots(["Simple learning problem"], fig_size=(10,8), fig_num=1)
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
#plt.xlim([-1, 2])
```

Now our main problem is that this function can follow different types of parameters

```python
params = [[3, 0, 1], [-2, 1, 0], [0.1, 1, 1]]
ax = prep_plots(["Problem 1", "Problem 2", "Problem 3"], fig_size=(20,6), fig_num=1)
# Generating
x = np.linspace(0, 1, 100)
for p in range(len(params)):
    poly = np.poly1d(params[p])
    epsilon = np.random.uniform(-noise_level,noise_level,x.shape)
    y = poly(x) + epsilon
    ax[p].scatter(x, y, color='green', s=50, marker='o', edgecolor='k', label="training samples")
finalize_plots(ax, fig_title="Different observations")
```

In real-life settings, this function can also have different levels of noise, as exemplified in the following code.

```python
params = [3, 0, 1]
noise_levels = [0.1, 1.0, 8.0]
ax = prep_plots(["Problem 1", "Problem 2", "Problem 3"], fig_size=(20,6), fig_num=1)
# Generating
x = np.linspace(0, 1, 100)
for p in range(len(noise_levels)):
    poly = np.poly1d(params)
    epsilon = np.random.uniform(-noise_levels[p],noise_levels[p],x.shape)
    y = poly(x) + epsilon
    ax[p].scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots(ax, fig_title="Different amounts of noise")
```

To summarize, we will have some observations of a function, and we would like to optimize a function that gets as close as possible to the real function that generated this data. Here, we plot the real function and also _subsample_ our number of observations (having only a few points to find the corresponding function)

```python
# Generating the data and subsampling
x_all = np.linspace(0, 1, 100); x_plot = np.linspace(0, 1, 100)
rng.shuffle(x_all); x = np.sort(x_all[:10])
poly = np.poly1d([3,0,1])
# Adding some external
epsilon = np.random.uniform(-0.1,0.1,x.shape)
y = poly(x)+ epsilon
# We keep the standard deviation for later
nnstd = np.std(epsilon); lw = 2
ax1, = prep_plots(["Simple learning problem"], fig_size=(10,8), fig_num=1)
ax1.plot(x_plot, poly(x_plot), color='green', linewidth=4, label='true function')
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
```

To get a first grip on what machine learning does, we will rely on the `scikit-learn` library. This contains already coded models and learning procedure, that will allow us to _learn_ the parameters of this unknown function.

Here we already know that we want to use a `PolynomialFeatures` model to perfom `LinearRegression` and that this polynomial should be of degree 2.

```python
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
# Our data to fit
X = x[:, np.newaxis]
# Degree of our polynomial
degree = 2;
# Create our polynomial model for regression
model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
# Fit the parameters of this model
model.fit(X, y)
```

    Pipeline(memory=None,
         steps=[('polynomialfeatures', PolynomialFeatures(degree=2, include_bias=True, interaction_only=False)), ('linearregression', LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False))])

Now that we have trained the model, we can perform _predictions_ from it, meaning that we can infer the output of the function at values that we did not observe originally.

```python
# Inference points (not observed)
X_plot = x_plot[:, np.newaxis]
# Predict the values
y_plot = model.predict(X_plot)
# Compute the error of our model at observed points
Y_model_err = np.sqrt(np.mean(np.square(y-model.predict(X))))
# Plot the result
ax1, = prep_plots(["Trained model"], fig_size=(10,8), fig_num=1)
ax1.plot(x_plot, y_plot, color='red', linewidth=4, label='trained model')
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
print(f'Model error : {Y_model_err}')
```

## Understanding model capacity and selection

In real-life problem, we are aiming to find the parameters of a model, but we do not really know what is the _real_ function underlying this process. So what we can decide to select _any_ function of _any_ **capacity** (complexity of the function). One of the problem with that, is that if we have a too simple function, it will _underfit_ (it is not complex enough for our observations). On the opposite end, if we have a function which is too complex, it might be able to _fit through all training points exactly_ ... even though there is noise in our observations ! This is examplified in the following

We can observe this idea and play with it directly by trying to find a function approximating our previous observations with a polynomial function chosen to have a degree inside \([1,2,8]\).

```python
X_plot = x_plot[:, np.newaxis]
ax1, ax2 = prep_plots(["training error", "model generalization"], fig_size=(15,4), fig_num=1)
ax1.plot([1,8], [nnstd, nnstd], label="noise level")
ax2.scatter(x, y, color='orange', edgecolor='w', s=50, marker='o', label="training samples")
colors = ['blue', 'green', 'red']; labels = ['underfitting', 'good fit', 'overfitting']
axes_sub = prep_plots(labels, fig_size=(15,4), fig_num=2)
# Optimize different degree polynomials
for count, degree in enumerate([1,2,8]):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X, y)
    Y_model_err = np.sqrt(np.mean(np.square(y-model.predict(X))))
    y_plot = model.predict(X_plot)
    ax1.scatter(degree, Y_model_err, s=50, marker="o", edgecolor='w', label=labels[count])
    ax2.plot(x_plot, y_plot, color=colors[count], linewidth=2,label=labels[count])
    ax2.set_ylim((0,4))
    axes_sub[count].scatter(x, y, color='orange', edgecolor='w', s=50, marker='o', label="Training samples")
    axes_sub[count].plot(x_plot, y_plot, color=colors[count], linewidth=2,label=labels[count])
    axes_sub[count].set_ylim((0,4))
finalize_plots(axes_sub, fig_title="Linear regression")
finalize_plots([ax1,ax2], fig_title="Linear regression")
```

Depending on the _capacity_ of the model, what we can observe is that

- `capacity too low   -> underfitting   : prediction variance >  noise variance`
- `adequate capacity  -> good fit       : prediction variance == noise variance`
- `capacity too high  -> overfitting    : prediction variance <  noise variance`

A similar example can be given for a classification problem in two dimensions as follows

## Audio applications

In order to test our algorithms on audio and music data, we will work with several datasets that should be downloaded on your local computer first from this [link ](https://nuage.ircam.fr/index.php/s/FTsaaAMFV1jEwsk)

  |**Type**|*Origin*|
  |-------:|:---------|
  |**Classification**|[*MuscleFish*](http://knight.cis.temple.edu/~vasilis/Courses/CIS750/Papers/muscle_fish.pdf) dataset|
  |**Music-speech**|[*MIREX Recognition*](http://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection) set|
  |**Source separation**|[*SMC Mirum*](http://smc.inesctec.pt/research/data-2/) dataset|
  |**Speech recognition**|[*CMU Arctic*](http://festvox.org/cmu_arctic/) dataset|

**Unzip the file and place the `data` folder along with the other code folders**
For the first parts of the tutorial, we will mostly rely solely on the classification dataset. In order to facilitate the interactions, we provide the function `import_dataset` that will allow to import all audio datasets along the tutorials.

```Python
def importDataset(class_path, type):
    """
    Helper function to import datasets
    % class_path  : Path to the dataset (string)
    % type       : Type of dataset (string: 'classify', 'plain', 'metadata')
    """
    # Returns the data_struct structure with
    data_struct["filenames"]  # Cell containing the list of audio files
    data_struct["classes"]    # Vector of indexes assigning each file to a class
    data_struct["class_names"] % Cell of class names
    return data_struct
```

***

**_Exercise_**

  1. Launch the import procedure  and check the corresponding structure
  2. Code a count function that prints the name and number of examples for each classes

***

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from helper_data import import_dataset
# 0.1 - Import the classification dataset
class_path = 'data/classification';
data_struct = import_dataset(class_path, 'classification');
```

        - Importing dataset data/classification.

```python
#%% Q-0.1.2 - Count function to print the number of examples

######################
# YOUR CODE GOES HERE
######################

######################
# Solution:
n_classes = len(data_struct["class_names"]);
for ind1 in range(n_classes):
    temp = data_struct["classes"][data_struct["classes"] == ind1]
    print('%18s \t : %d files' % (data_struct["class_names"][ind1], len(temp)))
print('%18s \t : %d files' % ('total', len(data_struct["classes"])))
######################
```

                 bells 	 : 7 files
           speech.male 	 : 17 files
                 water 	 : 7 files
                  oboe 	 : 32 files
          altotrombone 	 : 13 files
         speech.female 	 : 35 files
             telephone 	 : 17 files
            percussion 	 : 99 files
            violinpizz 	 : 40 files
              laughter 	 : 7 files
              machines 	 : 11 files
          tubularbells 	 : 20 files
               animals 	 : 9 files
           violinbowed 	 : 45 files
            cellobowed 	 : 47 files
                crowds 	 : 4 files
                 total 	 : 410 files

### Preprocessing

We will rely on a set of spectral transforms that allow to obtain a more descriptive view over the audio information. As most of these are out of the scope of the machine learning course, we redirect you to a [signal processing course](https://ccrma.stanford.edu/~jos/sasp/) proposed by [Julius O. Smith](https://ccrma.stanford.edu/~jos/).

The following functions to compute various types of transforms are given as part of the basic package, in the `helper_data.py` file

  |**File**|*Transform*|
  |-------:|:----------|
  |`stft.m`       |[Short-term Fourier transform](https://en.wikipedia.org/wiki/Short-time_Fourier_transform)|
  |`fft2barkmx.m` |[Bark scale](https://en.wikipedia.org/wiki/Bark_scale) transform|
  |`fft2melmx.m`  |[Mel scale](https://en.wikipedia.org/wiki/Mel_scale) transform|
  |`fft2chromamx` |[Chromas vector](https://en.wikipedia.org/wiki/Harmonic_pitch_class_profiles)|
  |`spec2cep.m`   |[Cepstrum](https://en.wikipedia.org/wiki/Cepstrum) transform|
  |`cqt.m`        |[Constant-Q](https://en.wikipedia.org/wiki/Constant_Q_transform) transform|

In order to perform the various computations, we provide the following function, which performs the different transforms on a complete dataset.

``` Python
def computeTransforms(data_struct)
    """ data_struct   : Dataset structure with filenames """
    return data_struct

# Returns the data_struct structure with
data_struct["spectrum_power"]     # Power spectrum (STFT)
data_struct["spectrum_mel"]       # Spectrum in Mel scale
data_struct["spectrum_chroma"]    # Chroma vectors
data_struct["spectrum_CQT"]       # Constant-Q transform
```

***

**Exercise**

  1. Launch the transform computation procedure and check the corresponding structure
  2. For each class, select a random element and plot its various transforms on a single plot. You should obtain plots similar to those shown afterwards.
  3. For each transform, try to spot major pros and cons of their representation.

***

```python
# 0.2 - Pre-process the audio to obtain spectral transforms
# (may take around a minute)
from helper_data import compute_transforms
data_struct = compute_transforms(data_struct);
```

        - Performing transforms.

    /usr/local/lib/python3.7/site-packages/librosa/core/pitch.py:146: UserWarning: Trying to estimate tuning from empty frequency set.
      warnings.warn('Trying to estimate tuning from empty frequency set.')

```python
#%% Q-0.2.2 - Plot the various transforms

# Just a little helper to make your figures pretty
from helper_plot import hdr_plot_style
hdr_plot_style()

######################
# YOUR CODE GOES HERE
######################

######################
# Solution
for ind1 in range(2):
    temp = np.linspace(0, len(data_struct["classes"])-1, len(data_struct["classes"]))[data_struct["classes"] == ind1]
    transforms = ['power', 'mel', 'chroma', 'CQT'];
    fig = plt.figure(figsize=(12, 8))
    fig.subplots(nrows=2,ncols=2,gridspec_kw={'width_ratios':[1, 1], 'height_ratios':[1, 1]})
    for ind2 in range(len(transforms)):
        ax = plt.subplot(2, 2, ind2 + 1)
        ax.imshow(np.flipud(np.log1p(data_struct["spectrum_" + transforms[ind2]][int(temp[1])])), interpolation='nearest', aspect='auto', cmap='inferno')
        plt.title(transforms[ind2] + ' spectrum from class ' + data_struct["class_names"][ind1])
        plt.grid(False)
        #ax.set_adjustable('box-forced')
    fig.tight_layout()
######################
```

### Features

As you might have noted from the previous exercice, most spectral transforms have a very high dimensionality, and might not be suited to exhibit the relevant structure of different classes. To that end, we provide a set of functions for computing several spectral features in the `helper_data` folder, we redirect interested readers to this [exhaustive article](http://recherche.ircam.fr/anasyn/peeters/ARTICLES/Peeters_2003_cuidadoaudiofeatures.pdf) on spectral features computation.

  |**File**|*Transform*|
  |-------:|:----------|
  |`spectral_centroid`|Spectral centroid|
  |`spectral_bandwidth`|Spectral bandwidth|
  |`spectral_contrast`|Spectral contrast|
  |`spectral_flatness`|Spectral flatness|
  |`spectral_rolloff`|Spectral rolloff|

Once again, we provide a function to perform the computation of different features on a complete set. Note that for each feature, we compute the temporal evolution in a vector along with the mean and standard deviation of each feature. We only detail the resulting data structure for a single feature (`SpectralCentroid`).

``` Python
def data_struct = computeFeatures(data_struct)
     """ data_struct   : Dataset structure with filenames """
    return data_struct

% Returns the data_struct structure with
data_struct["spectral_centroid"]      # Temporal value of a feature
data_struct["spectral_centroid_mean"] # Mean value of that feature
data_struct["spectral_centroid_std"]  # Standard deviation
```

***

**Exercise**

  1. Launch the feature computation procedure and check the corresponding structure
  2. This time for each class, superimpose the plots of various features on a single plot, along with a boxplot of mean and standard deviations. You should obtain plots similar to those shown afterwards.
  3. What conclusions can you make on the discriminative power of each feature ?
  4. Perform scatter plots of the mean features for all the dataset, while coloring different classes.
  5. What conclusions can you make on the discriminative power of mean features ?

***

```python
#%%
# 0.3 - Compute a set of temporal and spectral features
# (may take around 1-2 minutes)
from helper_data import compute_features
data_struct = compute_features(data_struct);
```

        - Performing features.

```python
#%% Q-0.3.2 - Plot the various features

# Just a little helper to make your figures pretty
from helper_plot import hdr_plot_style
hdr_plot_style()
# Use these styles for boxplot
boxprops=dict(linewidth=3, color='white')
whiskerprops=dict(linewidth=3, color='white')
medianprops=dict(linewidth=2.5, color='firebrick')
flierprops = dict(markeredgecolor='white', markerfacecolor='firebrick')

######################
# YOUR CODE GOES HERE
######################

######################
# Solution
for ind1 in range(2):
    temp = np.linspace(0, len(data_struct["classes"])-1, len(data_struct["classes"]))[data_struct["classes"] == ind1]
    #features = ['SpectralVariation','SpectralFlux',
    #    'SpectralDecrease','SpectralFlatness','PerceptualSharpness',
    #    'SpectralRolloff','SpectralSlope']
    features = ['loudness', 'spectral_centroid', "spectral_bandwidth", 'spectral_contrast', 'spectral_flatness', 'spectral_rolloff']
    fig = plt.figure(figsize=(22, 18))
    fig.subplots(nrows=len(features),ncols=2,gridspec_kw={'width_ratios':[5, 1]})
    plt.subplots_adjust(hspace=.5)
    for ind2 in range(len(features)):
        plt.subplot(len(features), 2, (2 * ind2) + 1)
        for ind3 in range(len(temp)):
            plt.plot(data_struct[features[ind2]][int(temp[ind3])][0])
        plt.title(features[ind2].replace('_', ' ') + ' of class ' + data_struct["class_names"][ind1])
        featM = data_struct[features[ind2] + '_mean']
        featS = data_struct[features[ind2] + '_std']
        featCat = np.column_stack((featM, featS))
        plt.subplot(len(features), 2, (2 * ind2) + 2)
        plt.boxplot(featCat, patch_artist=True, boxprops=boxprops, medianprops=medianprops, flierprops=flierprops, whiskerprops=whiskerprops)
        plt.title('Boxplots of mean and std')
######################
```

```python
#%% Q-0.3.4 - Observe the distribution of classes for different features

# This allows to use 3D rendering in matplotlib
from mpl_toolkits.mplot3d import Axes3D
plt.figure(figsize=(12,8))
# Create a vector of random colors for each class
colorVect = np.zeros((3, len(data_struct["class_names"])));
for c in range(len(data_struct["class_names"])):
    colorVect[:,c] = np.random.rand(3);

######################
# YOUR CODE GOES HERE
######################

######################
# Solution
#features = ['SpectralCentroidMean','LoudnessMean', 'SpectralSlopeMean'];
features = ['loudness_mean', 'spectral_centroid_mean', "spectral_bandwidth_mean", 'spectral_contrast_mean', 'spectral_flatness_mean', 'spectral_rolloff_mean']
fig = plt.figure(figsize=(15, 15))
ax = fig.add_subplot(111, projection='3d')
for ind1 in range(n_classes):
    temp = (np.linspace(0, len(data_struct["classes"])-1, len(data_struct["classes"]), dtype=int)[data_struct["classes"] == ind1])
    X = [data_struct[features[0]][i] for i in temp]
    Y = [data_struct[features[1]][i] for i in temp]
    Z = [data_struct[features[2]][i] for i in temp]
    ax.scatter(X,Y,Z,s=50,c=colorVect[:,ind1], edgecolor='w')
ax.set_xlabel(features[0].replace('_', ' '))
ax.set_ylabel(features[1].replace('_', ' '))
ax.set_zlabel(features[2].replace('_', ' '))
plt.show()
######################
```

That's it for this tutorial, now remember that we can use any form of description (features) as a basis for learning algorithms. We will see in the next tutorial what we an do with these features.