# Normalizing flows in PyTorch

One of the key in modern generative models is to find ways of optimizing the probability distribution of a given set of data. The recent idea of *Normalizing Flows* [[1](#reference1),[2](#reference)] addresses this problem and be able to rely on richer probability distributions. The main idea is to start from a simple probability distribution and approximate a complex multimodal density by *transforming* the simpler density through a sequence of invertible nonlinear transforms. To fully understand this blazing tool, we will see in this tutorial

1. The new [PyTorch distributions](#distribs) module and how to use it
2. How transforming a distribution is expressed as a [change of variables](#change) leading to a flow
3. How we can [chain multiple transforms](#chaining) leading to the overall framework of normalizing flows
4. Understanding the original [planar flow](#planar), its parameters and how to implement it
5. Defining [learnable flows](#learning) and performing optimization on a target density

### PyTorch distributions

In this tutorial, we are going to rely on the novel [PyTorch distributions module](https://pytorch.org/docs/stable/_modules/torch/distributions/), which is defined in `torch.distributions`. Most notably, we are going to rely both on the `Distribution` and `Transform` objects.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.distributions as distrib
import torch.distributions.transforms as transform
# Imports for plotting
import numpy as np
import matplotlib.pyplot as plt
from helper_plot import hdr_plot_style
hdr_plot_style()
# Define grids of points (for later plots)
x = np.linspace(-4, 4, 1000)
z = np.array(np.meshgrid(x, x)).transpose(1, 2, 0)
z = np.reshape(z, [z.shape[0] * z.shape[1], -1])
```

Inside this toolbox, we can already find some of the major probability distributions that we are used to deal with

```python
p = distrib.Normal(loc=0, scale=1)
p = distrib.Bernoulli(probs=torch.tensor([0.5]))
p = distrib.Beta(concentration1=torch.tensor([0.5]), concentration0=torch.tensor([0.5]))
p = distrib.Gamma(concentration=torch.tensor([1.0]), rate=torch.tensor([1.0]))
p = distrib.Pareto(alpha=torch.tensor([1.0]), scale=torch.tensor([1.0]))
```

The interesting aspect of these `Distribution` objects is that we can both obtain some samples from it through the `sample` (or `sample_n`) function, but we can also obtain the analytical density at any given point through the `log_prob` function

```python
# Based on a normal
n = distrib.Normal(0, 1)
# Obtain some samples
samples = n.sample((1000, ))
# Evaluate true density at given points
density = torch.exp(n.log_prob(torch.Tensor(x))).numpy()
# Plot both samples and density
fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, figsize=(15, 4))
ax1.hist(samples, 50, alpha=0.8);
ax1.set_title('Empirical samples', fontsize=18);
ax2.plot(x, density); ax2.fill_between(x, density, 0, alpha=0.5)
ax2.set_title('True density', fontsize=18);
```

## Transforming distributions

### Change of variables and flow

In order to transform a probability distribution, we can perform a *change of variable*. As we are interested in probability distributions, we need to *scale* our transformed density so that the total probability still sums to one. This is directly measured with the determinant of our transform.

Let $\mathbf{z}\in\mathcal{R}^d$ be a random variable with distribution $q(\mathbf{z})$ and $f:\mathcal{R}^d\rightarrow\mathcal{R}^d$ an invertible smooth mapping (meaning that $f^{-1} = g$ and $g\circ f(\mathbf{z})=\mathbf{z}'$. We can use $f$ to transform $\mathbf{z}\sim q(\mathbf{z})$. The resulting random variable $\mathbf{z}'=f(\mathbf{z})$ has the following probability distribution

$$
q(\mathbf{z}')=q(\mathbf{z})\left|\text{ det}\frac{\delta f^{-1}}{\delta \mathbf{z}'}\right| = q(\mathbf{z})\left|\text{ det}\frac{\delta f}{\delta \mathbf{z}}\right|^{-1}
\tag{1}
$$

where the last equality is obtained through both the inverse function theorem [1] and the property of Jacobians of invertible functions. Therefore, we can transform probability distributions with this property.

Fortunately, this can be easily implemented in PyTorch with the `Transform` classes, that already defines some basic probability distribution transforms. For instance, if we define $\mathbf{z}\sim q_0(\mathbf{z})=\mathcal{N}(0, 1)$, we can apply the transform $\mathbf{z}'=exp(\mathbf{z})$ so that $\mathbf{z}'\sim q_1(\mathbf{z}')$

```python
q0 = distrib.Normal(0, 1)
exp_t = transform.ExpTransform()
q1 = distrib.TransformedDistribution(q0, exp_t)
samples_q0 = q0.sample((int(1e4),))
samples_q1 = q1.sample((int(1e4),))
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 4))
ax1.hist(samples_q0, 50, alpha=0.8);
ax1.set_title('$q_0 = \mathcal{N}(0,1)$', fontsize=18);
ax2.hist(samples_q1, 50, alpha=0.8, color='g');
ax2.set_title('$q_1=exp(q_0)$', fontsize=18);
```

But remember as the objects `q0` and `q1` are defined as `Distribution`, we can actually observe their true densities instead of just empirical samples

```python
hdr_plot_style()
x2 = np.linspace(-0.5, 7.5, 1000)
q0_density = torch.exp(q0.log_prob(torch.Tensor(x))).numpy()
q1_density = torch.exp(q1.log_prob(torch.Tensor(x2))).numpy()
fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, figsize=(15, 5))
ax1.plot(x, q0_density); ax1.fill_between(x, q0_density, 0, alpha=0.5)
ax1.set_title('$q_0 = \mathcal{N}(0,1)$', fontsize=18);
ax2.plot(x2, q1_density, color='g'); ax2.fill_between(x2, q1_density, 0, alpha=0.5, color='g')
ax2.set_title('$q_1=exp(q_0)$', fontsize=18);
fig.savefig('transform.pdf')
```

What we obtain here with `q1` is actually the `LogNormal` distribution. Interestingly, several distributions in the `torch.distributions` module are already defined based on `TransformedDistribution`. You can convince yourself of that by lurking in the code of the [`torch.distributions.LogNormal`](https://pytorch.org/docs/stable/_modules/torch/distributions/log_normal.html#LogNormal)

### Chaining transforms (normalizing flows)

Now, if we start with a random vector $\mathbf{z}_0$ with distribution $q_0$, we can apply a series of mappings $f_i$, $i \in 1,\cdots,k$ with $k\in\mathcal{N}^{+}$ and obtain a normalizing flow. Hence, if we apply $k$ normalizing flows, we obtain a chain of change of variables

$$
\mathbf{z}_k=f_k\circ f_{k-1}\circ...\circ f_1(\mathbf{z}_0)
\tag{2}
$$

Therefore the distribution of $\mathbf{z}_k\sim q_k(\mathbf{z}_k)$ will be given by

   $$
   \begin{align}
   q_k(\mathbf{z}_k) &= q_0(f_1^{-1} \circ f_{2}^{-1} \circ ... \circ f_k^{-1}(\mathbf{z}_k))\prod_{i=1}^k\left|\text{det}\frac{\delta f^{-1}_i}{\delta\mathbf{z}_{i}}\right|\\
   &= q_0(\mathbf{z_0})\prod_{i=1}^k\left|\text{det}\frac{\delta f_i}{\delta\mathbf{z}_{i-1}}\right|^{-1}
   \end{align}
   \tag{3}
   $$

where we compute the determinant of the Jacobian of each normalizing flow (as explained in the previous section). This series of transformations can transform a simple probability distribution (e.g. Gaussian) into a complicated multi-modal one. As usual, we will rely on log-probabilities to simplify the computation and obtain

$$
\text{log} q_K(\mathbf{z}_k) = \text{log} q_0(\mathbf{z}_0) - \sum_{i=1}^{k} \text{log} \left|\text{det}\frac{\delta f_i}{\delta\mathbf{z}_{i-1}}\right|
\tag{4}
$$

To be of practical use, however, we can consider only transformations whose determinants of Jacobians are easy to compute. Of course, we can perform any amount of combined transformations, and it also works with multivariate distributions. Here, this is demonstrated by transforming a `MultivariateNormal` successively with an `ExpTransform` and `AffineTransform`. (Note that the final distribution `q2` is defined as a `TransformedDistribution` directly with a *sequence* of transformations)

```python
q0 = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
# Define an affine transform
f1 = transform.ExpTransform()
q1 = distrib.TransformedDistribution(q0, f1)
# Define an additional transform
f2 = transform.AffineTransform(2, torch.Tensor([0.2, 1.5]))
# Here I define on purpose q2 as a sequence of transforms on q0
q2 = distrib.TransformedDistribution(q0, [f1, f2])
# Plot all these lads
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))
ax1.hexbin(z[:,0], z[:,1], C=torch.exp(q0.log_prob(torch.Tensor(z))), cmap='rainbow')
ax1.set_title('$q_0 = \mathcal{N}(\mathbf{0},\mathbb{I})$', fontsize=18);
ax2.hexbin(z[:,0], z[:,1], C=torch.exp(q1.log_prob(torch.Tensor(z))), cmap='rainbow')
ax2.set_title('$q_1=exp(q_0)$', fontsize=18);
ax3.hexbin(z[:,0], z[:,1], C=torch.exp(q2.log_prob(torch.Tensor(z))), cmap='rainbow')
ax3.set_title('$q_2=Affine(exp(q_0))$', fontsize=18);
```

## Normalizing flows

Now, we are interested in normalizing flows as we could define our own flows. And, most importantly, we could optimize the parameters of these flow in order to fit complex and richer probability distributions. We will see how this plays out by trying to implement the *planar flow* proposed in the original paper by Rezende [1].

### Planar flow

 A planar normalizing flow is defined as a function of the form

   $$
   f(\mathbf{z})=\mathbf{z}+\mathbf{u}h(\mathbf{w}^T\mathbf{z}+b)
   \tag{5}
   $$

   where $\mathbf{u}\in\mathbb{R}^D$ and $\mathbf{w}\in\mathbb{R}^D$ are vectors (called here scale and weight), $b\in\mathbb{R}$ is a scalar (bias) and $h$ is an activation function. These transform functions are chosen depending on the fact that
1. the determinant of their Jacobian can be computed in linear time
2. the transformation is invertible (under usually mild conditions only)

As shown in the paper, for the planar flow, the determinant of the Jacobian can be computed in $O(D)$ time by relying on the matrix determinant lemma

$$
\psi(\mathbf{z})=h'(\mathbf{w}^T\mathbf{z}+b)\mathbf{w}
\tag{6}
$$

$$
\left|\text{det}\frac{\delta f}{\delta\mathbf{z}}\right| = \left|\text{det}\left(\mathbf{I}+\mathbf{u}\psi(\mathbf{z})^{T}\right)\right|=\left|1+\mathbf{u}^T\psi(\mathbf{z})\right|
\tag{7}
$$

Therefore, we have all definitions that we need to implement this flow as a `Transform` object. Note that here the non-linear activation function $h$ is selected as a $tanh$. Therefore the derivative $h'$ is $1-tanh(x)^2$

```python
class PlanarFlow(transform.Transform):

    def __init__(self, weight, scale, bias):
        super(PlanarFlow, self).__init__()
        self.bijective = False
        self.weight = weight
        self.scale = scale
        self.bias = bias

    def _call(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return f_z

    def log_abs_det_jacobian(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return abs_log_det
```

As before, we can witness the effect of this transform on a given `MultivariateNormal` distribution. You should note here that I am using the density estimation for `q0`, but only display empirical samples from `q1`.

```python
w = torch.Tensor([[3., 0]])
u = torch.Tensor([[2, 0]])
b = torch.Tensor([0])
q0 = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
flow_0 = PlanarFlow(w, u, b)
q1 = distrib.TransformedDistribution(q0, flow_0)
q1_samples = q1.sample((int(1e6), ))
# Plot this
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))
ax1.hexbin(z[:,0], z[:,1], C=torch.exp(q0.log_prob(torch.Tensor(z))), cmap='rainbow')
ax1.set_title('$q_0 = \mathcal{N}(\mathbf{0},\mathbb{I})$', fontsize=18);
ax2.hexbin(q1_samples[:,0], q1_samples[:,1], cmap='rainbow')
ax2.set_title('$q_1=planar(q_0)$', fontsize=18);
```

The reason for this is that the `PlanarFlow` is not invertible in all regions of the space. However, if we recall the mathematical reasoning of the previous section, we can see how the change of variables plays out if we are able to compute the determinant of the Jacobian of this transform.

```python
q0_density = torch.exp(q0.log_prob(torch.Tensor(z)))
# Apply our transform on coordinates
f_z = flow_0(torch.Tensor(z))
# Obtain our density
q1_density = q0_density.squeeze() / np.exp(flow_0.log_abs_det_jacobian(torch.Tensor(z)).squeeze())
# Plot this
fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(15, 5))
ax1.hexbin(z[:,0], z[:,1], C=q0_density.numpy().squeeze(), cmap='rainbow')
ax1.set_title('$q_0 = \mathcal{N}(\mathbf{0},\mathbb{I})$', fontsize=18);
ax2.hexbin(f_z[:,0], f_z[:,1], C=q1_density.numpy().squeeze(), cmap='rainbow')
ax2.set_title('$q_1=planar(q_0)$', fontsize=18);
```

So we were able to "split" our distribution and transform a unimodal gaussian into a multimodal distribution ! Pretty neat

### Visualizing parameters effects

Here, we provide a little toy example so that you can play around with the parameters of the flow in order to get a better understanding of how it operates. As put forward by Rezende [1], this flow is related to the hyperplane defined by $\mathbf{w}^{T}\mathbf{z}+b=0$ and transforms the original density by applying a series of contractions and expansions in the direction perpendicular to this hyperplane.

```python
id_figure=1
plt.figure(figsize=(16, 18))
for i in np.arange(5):
    # Draw a random hyperplane
    w = torch.rand(1, 2) * 5
    b = torch.rand(1) * 5
    for j in np.arange(5):
        # Different effects of scaling factor u on the same hyperplane (row)
        u = torch.Tensor([[((j < 3) and (j / 2.0) or 0), ((j > 2) and ((j - 2) / 2.0) or 0)]])
        flow_0 = PlanarFlow(w, u, b)
        q1 = distrib.TransformedDistribution(q0, flow_0)
        q1_samples = q1.sample((int(1e6), ))
        plt.subplot(5,5,id_figure)
        plt.hexbin(q1_samples[:,0], q1_samples[:,1], cmap='rainbow')
        plt.title("u=(%.1f,%.1f)"%(u[0,0],u[0,1]) + " w=(%d,%d)"%(w[0,0],w[0,1]) + ", " + "b=%d"%b)
        plt.xlim([-3, 3])
        plt.ylim([-3, 3])
        id_figure += 1
```

## Optimizing normalizing flows

Now that we have this magnificent tool, we would like to apply this in order to learn richer distributions and perform *inference*. Now, we have to deal with the fact that the `Transform` object is not inherently parametric and cannot yet be optimized similarly to other modules.

To do so, we will start by defining our own `Flow` class which can be seen both as a `Transform` and also a `Module`that can be optmized

```python
class Flow(transform.Transform, nn.Module):

    def __init__(self):
        transform.Transform.__init__(self)
        nn.Module.__init__(self)

    # Init all parameters
    def init_parameters(self):
        for param in self.parameters():
            param.data.uniform_(-0.01, 0.01)

    # Hacky hash bypass
    def __hash__(self):
        return nn.Module.__hash__(self)
```

Thanks to this little trick, we can use the same `PlanarFlow` class as before, that we put back here just to show that the only change is that it now inherits from the `Flow` class (with the small added bonus that now parameters of this flow are also registered in the `Module` interface)

```python
class PlanarFlow(Flow):

    def __init__(self, dim):
        super(PlanarFlow, self).__init__()
        self.weight = nn.Parameter(torch.Tensor(1, dim))
        self.scale = nn.Parameter(torch.Tensor(1, dim))
        self.bias = nn.Parameter(torch.Tensor(1))
        self.init_parameters()

    def _call(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return f_z

    def log_abs_det_jacobian(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return abs_log_det
```

Now let's say that we have a given complex density that we aim to model through normalizing flows, such as the following one

```python
def density_ring(z):
    z1, z2 = torch.chunk(z, chunks=2, dim=1)
    norm = torch.sqrt(z1 ** 2 + z2 ** 2)
    exp1 = torch.exp(-0.5 * ((z1 - 2) / 0.8) ** 2)
    exp2 = torch.exp(-0.5 * ((z1 + 2) / 0.8) ** 2)
    u = 0.5 * ((norm - 4) / 0.4) ** 2 - torch.log(exp1 + exp2)
    return torch.exp(-u)

# Plot it
x = np.linspace(-5, 5, 1000)
z = np.array(np.meshgrid(x, x)).transpose(1, 2, 0)
z = np.reshape(z, [z.shape[0] * z.shape[1], -1])
plt.hexbin(z[:,0], z[:,1], C=density_ring(torch.Tensor(z)).numpy().squeeze(), cmap='rainbow')
plt.title('Target density', fontsize=18);
```

Now to approximate such a complicated density, we will need to chain multiple planar flows and optimize their parameters to find a suitable approximation. We can do exactly that like in the following (you can see that we start by a simple normal density and perform 16 successive planar flows)

```python
# Main class for normalizing flow
class NormalizingFlow(nn.Module):

    def __init__(self, dim, flow_length, density):
        super().__init__()
        biject = []
        for f in range(flow_length):
            biject.append(PlanarFlow(dim))
        self.transforms = transform.ComposeTransform(biject)
        self.bijectors = nn.ModuleList(biject)
        self.base_density = density
        self.final_density = distrib.TransformedDistribution(density, self.transforms)
        self.log_det = []

    def forward(self, z):
        self.log_det = []
        # Applies series of flows
        for b in range(len(self.bijectors)):
            self.log_det.append(self.bijectors[b].log_abs_det_jacobian(z))
            z = self.bijectors[b](z)
        return z, self.log_det

# Create normalizing flow
flow = NormalizingFlow(dim=2, flow_length=16, density=distrib.MultivariateNormal(torch.zeros(2), torch.eye(2)))
```

Now the only missing ingredient is the loss function that is simply defined as follows

```python
def loss(density, zk, log_jacobians):

    ######################
    # YOUR CODE GOES HERE
    ######################

    return loss
```

We can now perform optimization as usual by defining an optimizer, the parameters it will act on and eventually a learning rate scheduler

```python
import torch.optim as optim
# Create optimizer algorithm
optimizer = optim.Adam(flow.parameters(), lr=2e-3)
# Add learning rate scheduler
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, 0.9999)
```

And now we perform the loop by sampling a batch (here of 512) from the reference Normal distribution, and then evaluating our loss with respect to the density we want to approximate.

```python
ref_distrib = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
id_figure=2
plt.figure(figsize=(16, 18))
plt.subplot(3,4,1)
plt.hexbin(z[:,0], z[:,1], C=density_ring(torch.Tensor(z)).numpy().squeeze(), cmap='rainbow')
plt.title('Target density', fontsize=15);
# Main optimization loop
for it in range(10001):
    # Draw a sample batch from Normal
    samples = ref_distrib.sample((512, ))
    # Evaluate flow of transforms
    zk, log_jacobians = flow(samples)
    # Evaluate loss and backprop
    optimizer.zero_grad()
    loss_v = loss(density_ring, zk, log_jacobians)
    loss_v.backward()
    optimizer.step()
    scheduler.step()
    if (it % 1000 == 0):
        print('Loss (it. %i) : %f'%(it, loss_v.item()))
        # Draw random samples
        samples = ref_distrib.sample((int(1e5), ))
        # Evaluate flow and plot
        zk, _ = flow(samples)
        zk = zk.detach().numpy()
        plt.subplot(3,4,id_figure)
        plt.hexbin(zk[:,0], zk[:,1], cmap='rainbow')
        plt.title('Iter.%i'%(it), fontsize=15);
        id_figure += 1
```

That concludes this tutorial ! In the next one we will see how to implement more complicated flows and how this can fit in a global inference framework

### References

[1] Rezende, Danilo Jimenez, and Shakir Mohamed. "Variational inference with normalizing flows." _arXiv preprint arXiv:1505.05770_ (2015). [link](http://arxiv.org/pdf/1505.05770)

[2] Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving Variational Inference with Inverse Autoregressive Flow." _arXiv preprint arXiv:1606.04934_ (2016). [link](https://arxiv.org/abs/1606.04934)

[3] Germain, Mathieu, et al. "Made: masked autoencoder for distribution estimation." International Conference on Machine Learning. 2015.

### Inspirations and resources

https://blog.evjang.com/2018/01/nf1.html
https://github.com/ex4sperans/variational-inference-with-normalizing-flows
https://akosiorek.github.io/ml/2018/04/03/norm_flows.html
https://github.com/abdulfatir/normalizing-flows


```python
from scipy.stats import norm

import numpy as np
from math import cos, sin

import matplotlib.pyplot as plt
%matplotlib inline
import seaborn
from copy import copy
from sklearn.decomposition import PCA
```

    /home/preston/anaconda3/envs/py27/lib/python2.7/site-packages/IPython/html.py:14: ShimWarning: The `IPython.html` package has been deprecated. You should import from `notebook` instead. `IPython.html.widgets` has moved to `ipywidgets`.
      "`IPython.html.widgets` has moved to `ipywidgets`.", ShimWarning)

# 7.1  The Curse of Dimensionality

Consider a D-dimensional uniform distribution with coordinates between [r,-r] on each axis.
What is the probability of randomly drawing a point that lies within a distance $ r=\sqrt{x_1^2+\dots+x_{D}^2} $ of the origin?

The volume of a hypersphere with radius r is $$ V_D(r)=\frac{2 r^D \pi^{D/2}}{D~\Gamma(D/2)} $$ so this probability is given by $$ f_D=\frac{V_D(r)}{(2r)^D} = \frac{\pi^{D/2}}{D~2^{D-1}~\Gamma(D/2)} $$ The limit of which goes to zero as D approaches infinity: $$  \lim_   {D\to \infty} f_d = 0 $$

The larger the dimensionality of your data set, the more difficult it is to evenly sample its volume.

# 7.3 Principal Component Analysis
### Simple explanation:
#### PCA is a method for reducing the dimensionality of a data set while preserving as much of the valuable information in that set as possible.
#### Some reasons why PCA is useful:
- By reducing the dimensionality of our data, we can more easily visualize it (e.g., 3D->2D transformation)
- It reduces the amount of space needed to store the data
- It can speed up computation time in working with the data (less things to calculate)
- It can give us useful insights into the way our data's features relate to one another

#### The PCA algorithm essentially performs the following steps:
1. Given a set of data $\vec{x}$ in $k$-dimensions aligned with axes $\hat{e}_{i}$, find a new set of $k$ axes $\hat{e}_{i}'$ such that the covariances of the data set with respect to the new axes is zero: $\sigma_{\hat{e}_{i}'}\sigma_{\hat{e}_{j}'} = 0 \text{ for all } i\neq j$.
2. Drop the axes with the least variances

#### A very informative link:
http://setosa.io/ev/principal-component-analysis/

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Set parameters and draw the random sample
np.random.seed(42)
r = 0.9

sigma1 = 0.25
sigma2 = 0.08
rotation = np.pi / 6
s = np.sin(rotation)
c = np.cos(rotation)

X = np.random.normal(0, [sigma1, sigma2], size=(100, 2)).T
R = np.array([[c, -s],
              [s, c]])
X = np.dot(R, X)

#------------------------------------------------------------
# Plot the diagram
fig = plt.figure(figsize=(10, 10), facecolor='w')
ax = plt.axes((0, 0, 1, 1), xticks=[], yticks=[], frameon=False)

# draw axes
ax.annotate(r'$x$', (-r, 0), (r, 0),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))
ax.annotate(r'$y$', (0, -r), (0, r),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))

# draw rotated axes
ax.annotate(r'$x^\prime$', (-r * c, -r * s), (r * c, r * s),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))
ax.annotate(r'$y^\prime$', (r * s, -r * c), (-r * s, r * c),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))

# scatter points
ax.scatter(X[0], X[1], s=25, lw=0, c='k', zorder=2)

# draw lines
vnorm = np.array([s, -c])
for v in (X.T):
    d = np.dot(v, vnorm)
    v1 = v - d * vnorm
    ax.plot([v[0], v1[0]], [v[1], v1[1]], '-k')
    ax.scatter(v1[0], v1[1], color = (1,0,0), s = 50)

# draw ellipses
for sigma in (1, 2, 3):
    ax.add_patch(Ellipse((0, 0), 2 * sigma * sigma1, 2 * sigma * sigma2,
                         rotation * 180. / np.pi,
                         ec='k', fc='gray', alpha=0.2, zorder=1))

ax.set_xlim(-1, 1)
ax.set_ylim(-1, 1)

plt.show()
```

## 7.3.1 Derivation of Principal Component Analysis (page 293)

- Start with a set of data $\boldsymbol{X}$ comprising a series of $N$ observations (e.g., stars), each observation made up of $K$ features (e.g. magnitude, color, etc.). $X^{\left(j\right)}_{i}$ is feature $j$ of the $i^{\text{th}}$ observation.

- Subtract the mean of each feature from every data point: $X^{\left(j\right)}_{i}\rightarrow X^{\left(j\right)}_{i}-\bar{X}^{\left(j\right)}$
    - Optionally, one can scale all features by dividing data by the feature standard deviation

- Calculate the variance of the data along some arbitrary axis $\hat{r}_{1}$:
$$\begin{align}
\sigma_{\hat{r}_{1}}^{2} &= \sum_{i}^{N}\left(\vec{X_{i}}\cdot\hat{r}_{1}\right)^{2}\\
&=\sum_{i}^{N}\left(\hat{r}_{1}^{T}\cdot\vec{X_{i}}^{T}\right)\left(\vec{X_{i}}\cdot\hat{r}_{1}\right)\\
\end{align}
$$

- Choose $\hat{r}_{1}$ to be the axis of maximum variance. To find $\hat{r}_{1}$, maximize the variance subject to the constraint $\hat{r}_{1}^{T}\hat{r}_{1} = 1$ (the axis is a unit vector); this can be done by using Lagrange multipliers:

$$\phi\left(\hat{r}_{1},\lambda_{1}\right)=\
\sum_{i}^{N}\left(\hat{r}_{1}^{T}\cdot\vec{X_{i}}^{T}\right)\left(\vec{X_{i}}\cdot\hat{r}_{1}\right)\
-\lambda_{1}\left(\hat{r}_{1}^{T}\hat{r}_{1}-1\right)$$

- Setting $\frac{\text{d}\phi}{\text{d}\hat{r}_{1}} = 0$ yields

$$\left(\sum_{i}^{N}\vec{X}_{i}^{T}\vec{X}_{i}\right)\hat{r}_{1}-\lambda_{1}\hat{r}_{1}=0$$

- This is an eigenvalue equation for the maximum variance $\lambda_{1}$ and the corresponding axis of maximal variance $\hat{r}_{1}$.

- Notice that $\sum_{i}^{N}\vec{X}_{i}^{T}\vec{X}_{i} = \boldsymbol{X}^{T}\boldsymbol{X}=\boldsymbol{C}_{\boldsymbol{X}}$--the axis of maximum variance is an eigenvector of the covariance matrix

- To get the next axis $\hat{r}_{2}$, we solve the same Lagrange multiplier equation with the additional constraint that the covariance $\sigma_{\hat{r}_{1}\hat{r}_{2}} = 0$, and repeat

- Finally, we transform the data by taking $\boldsymbol{Y}=\boldsymbol{X}\boldsymbol{R}$ and dropping low-variance features (the right-most rows in the transformed data

### Singular value decomposition (SVD)

- The PCA can be computed through the data matrix $\boldsymbol{X}$ itself rather than through the eigenvalue decomposition of the covariance matrix; this method is known as **singular value decomposition**

- SVD:
$$\boldsymbol{U\Sigma V}^{T}=\frac{1}{\sqrt{N-1}}\boldsymbol{X}$$

- $\boldsymbol{U}$: *left-singular matrix*
- $\boldsymbol{\Sigma}$: matrix of singular values
- $\boldsymbol{V}$: *right-singular matrix*

$$\boldsymbol{C}_{\boldsymbol{X}}=\left[\frac{1}{\sqrt{N-1}}\right]$$

- Can show that
    - $\boldsymbol{V}=\boldsymbol{R}$
    - $\boldsymbol{\Sigma}^{2}=\boldsymbol{C}_{\boldsymbol{Y}}$
- SVD of the data directly yields the projection $\boldsymbol{R}$ and variances $\boldsymbol{C}_{\boldsymbol{Y}}$

### Three ways to calculate the principal components $\boldsymbol{R}$ and the eigenvalues $\boldsymbol{C}_{\boldsymbol{Y}}$:
1. Eigenvalue decomposition of covariance matrix $\boldsymbol{C}_{\boldsymbol{X}}$
    - Use if $N\gg K$
2. SVD of $\boldsymbol{X}$
    - Use if $N\sim K$
3. Eigenvalue decomposition of correlation matrix $\boldsymbol{M}_{\boldsymbol{X}}$
    - Use if $K\gg N$

## 7.3.2 The Application of PCA

```python
#We have ~100 carbon enhanced metal poor stars with 23 chemical abundances measured [Fe/H] and [X/Fe]

#importing and floatify data
infile=open('CEMPabundances.dat','r')
per_row = []
for line in infile:
    per_row.append(line.split('\t'))
elem_abundance_arr_str = zip(*per_row)

truncated_elem_abundance_arr=[]
tot_elem_abundance_arr=[]
for arr0 in elem_abundance_arr_str:
    tot_elem_abundance_arr.append([float(i) for i in arr0])

#remove unmeasured elements
for n in range(len(tot_elem_abundance_arr)):
    arr0=tot_elem_abundance_arr[n]
    truncated_elem_abundance_arr.append([x for x in arr0 if x!=-10000.])
#read labels for abundances
infile.close()
#for arr in elem_abundance_arr:
#    for i in range(len(arr)):
#        arr[i]=float(arr[i])
infile=open('elem_list.dat','r')
per_row = []
for line in infile:
    per_row.append(line.split('\t'))
elems=per_row[0]
infile.close()

#histogram distributions in abundances
fig = plt.figure(figsize=(20, 20))
for n in range(0,23):
    ax=fig.add_subplot(5,5,(n+1))
    ax.hist(truncated_elem_abundance_arr[n], 10, normed=1, facecolor='green', alpha=0.75)
    #ax.set_xlabel(elems[n])
    ax.annotate(elems[n] ,xy=(0.05,0.9), xycoords='axes fraction', size=14)

plt.show()
```

# searching for structure in a 23-dimensional data set

Without reducing the dimensionality, we can play around and try to search for some structure in the abundances. Looking for correlations between two dimensions is doable. If we were looking for structure in a plane defined by just two of these axes, we could make a triangle plot (no room here though) and visually inspect for any pairs of related abundances. We could do something similar to search for structure in the data that depended on 3 dimensions, but that would be hard to visualize and we'd have $\mathcal{O}(D^3)$ combinations to test.

```python
Dim1=10  #0-22    cheat: try combinations of 1,17,18,20,21,22
Dim2=17  #0-22

Nobjs=len(tot_elem_abundance_arr[0])

#reform this array since next cell is changing it for whatever reason
tot_elem_abundance_arr=[]
for arr0 in elem_abundance_arr_str:
    tot_elem_abundance_arr.append([float(i) for i in arr0])

#only use objects with measurements for both elements
arr1,arr2=[],[]
for obj in range(Nobjs):
    if tot_elem_abundance_arr[Dim1][obj]!=-10000. and tot_elem_abundance_arr[Dim2][obj]!=-10000.:
        arr1.append(tot_elem_abundance_arr[Dim1][obj])
        arr2.append(tot_elem_abundance_arr[Dim2][obj])

fig=plt.figure(figsize=(5, 5))
ax=fig.add_subplot(111)

ax.plot(arr1,arr2, 'o')
ax.annotate('N$_{obj}$='+repr(len(arr1)) ,xy=(0.05,0.9), xycoords='axes fraction', size=14)
ax.set_xlabel(elems[Dim1],fontsize=22)
ax.set_ylabel(elems[Dim2],fontsize=22)
```

# use PCA to find vectors in 23-space that account for most variance

```python
#unmeasured elements replaced with median of measurements
processed_arr=[]
copy_arr=list(copy(tot_elem_abundance_arr))

for n in range(len(copy_arr)):
    arr=copy_arr[n]
    processed_arr.append(arr)
    temp_arr=[]
    for i in range(Nobjs):
        if arr[i]!=-10000:
            temp_arr.append(arr[i])
    for i in range(Nobjs):
        if arr[i]==-10000:
            processed_arr[n][i]=np.median(temp_arr)
processed_arr=np.transpose(processed_arr)

pca = PCA(n_components=3)

pca.fit(processed_arr)

print 'Percent of variance in each PC'
print(pca.explained_variance_ratio_)

pca_score = pca.explained_variance_ratio_
V = pca.components_
PC1, PC2, PC3=V[0],V[1],V[2]

# plot orientations of principal components
fig=plt.figure(figsize=(20, 12))
ax=fig.add_subplot(311)

x=np.linspace(0,22,23)
zeroline=np.zeros(23)
ax.set_xlim([-1,24])
ax.set_ylim([-0.3,1.2])
ax.set_ylabel('PC 1',fontsize=20)
ax.plot(x,zeroline,'--',color='grey')

for n in range(len(elems)):
    ax.annotate(elems[n] ,xy=(n,1.0+((-1)**n)*0.05), xycoords='data',ha="center", size=14)
ax.plot(x,PC1,'o')

ax2=fig.add_subplot(312)
ax2.set_xlim([-1,24])
ax2.set_ylim([-0.6,1.0])
ax2.set_ylabel('PC 2',fontsize=20)
ax2.plot(x,zeroline,'--',color='grey')

for n in range(len(elems)):
    ax2.annotate(elems[n] ,xy=(n,0.7+((-1)**n)*0.05), xycoords='data',ha="center", size=14)
ax2.plot(x,PC2,'o')

ax3=fig.add_subplot(313)
ax3.set_xlim([-1,24])
ax3.set_ylim([-0.6,1.0])
ax3.set_ylabel('PC 3',fontsize=20)
ax3.plot(x,zeroline,'--',color='grey')

for n in range(len(elems)):
    ax3.annotate(elems[n] ,xy=(n,0.7+((-1)**n)*0.05), xycoords='data',ha="center", size=14)
ax3.plot(x,PC3,'o')

```

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt
from astroML import datasets

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# load data:
data = datasets.fetch_sdss_corrected_spectra()
spectra = datasets.sdss_corrected_spectra.reconstruct_spectra(data)

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]

#------------------------------------------------------------
# plot the eigenvalues
fig = plt.figure(figsize=(5, 3.75))
fig.subplots_adjust(hspace=0.05, bottom=0.12)

ax = fig.add_subplot(211, xscale='log', yscale='log')
ax.grid()
ax.plot(evals, c='k')
ax.set_ylabel('Normalized Eigenvalues')
ax.xaxis.set_major_formatter(plt.NullFormatter())
ax.set_ylim(5E-4, 100)

ax = fig.add_subplot(212, xscale='log')
ax.grid()
ax.semilogx(evals_cs, color='k')
ax.set_xlabel('Eigenvalue Number')
ax.set_ylabel('Cumulative Eigenvalues')
ax.set_ylim(0.65, 1.00)

plt.show()
```

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#------------------------------------------------------------
# Compute PCA components

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(5, 5))
fig.subplots_adjust(hspace=0, top=0.95, bottom=0.1, left=0.12, right=0.93)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.02, 0.93, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

### Reconstruction of data from the eigenbasis:

$$\vec{x}_{i}\left(k\right)=\vec{\mu}\left(k\right)+\sum_{j}^{R}\theta_{ij}\vec{e}_{j}\left(k\right)$$

$$\theta_{ij}=\sum_{k}\vec{e}_{j}\left(k\right)\left(\vec{x}_{i}\left(k\right)-\vec{\mu}\left(k\right)\right)$$

- Truncate the expansion to compress the data:
$\vec{x}_{i}\left(k\right)=\sum_{i}^{r<R}\theta_{i}\vec{e}_{i}\left(k\right)$

- How many terms to include in the expansion? One way is to define a bound $\alpha$ on the fraction of the variance we want to capture:

$$\frac{\sum_{i}^{i=r}\sigma_{i}}{\sum_{i}^{i=R}\sigma_{i}}<\alpha$$

- $\alpha$ typically taken from 0.70 to 0.95;

- Manually inspect the cumulative-eigenvalues plot, truncate at the 'knee'

## PCA with missing data

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import ticker

from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.datasets import sdss_corrected_spectra

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Get spectra and eigenvectors used to reconstruct them
data = fetch_sdss_corrected_spectra()
spec = sdss_corrected_spectra.reconstruct_spectra(data)
lam = sdss_corrected_spectra.compute_wavelengths(data)
evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((lam > 5750) & (lam < 6350))
lam = lam[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(5, 6.25))
fig.subplots_adjust(left=0.09, bottom=0.08, hspace=0, right=0.92, top=0.95)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spec_i_centered = spec[i] / norms[i] - mu
    coeffs = np.dot(spec_i_centered, evecs.T)

    # blank out masked regions
    spec_i = spec[i]
    mask_i = mask[i]
    spec_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(lam, spec_i[i_plot], '-', color='k',
            label='True spectrum', lw=1.5)

    # plot two levels of reconstruction
    for nev in [10]:
        if nev == 0:
            label = 'mean'
        else:
            label = 'reconstruction\n(nev=%i)' % nev
        spec_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(lam, spec_i_recons[i_plot], label=label, color='grey')

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([lam[:1], lam, lam[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(lam[0], lam[-1])
    ax.set_ylim(ylim)
    ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1)

    ax.set_xlabel('$\lambda\ (\AA)$')
    ax.set_ylabel('normalized flux')

plt.show()
```


# Mathematical programming with Pyomo

Author: Jérémie Decock (2019)

[](https://mybinder.org/v2/gh/jdhp-docs/notebooks/master?urlpath=apps%2Fnb_sci_ai%2Fai_ml_multilayer_perceptron_fr.ipynb)

## What is mathematical programming

Mathematical programming aims at finding the *solutions* (vectors) which minimize (or maximize) an *objective function*, the feasible solution space being defined by a set of *constraints* (equations and inequations).

Very basic example of *linear program* (i.e. a mathematical optimization problem with a linear objective function and linear constraints):

$$
\begin{align}
    \min_{x_1,x_2} & \quad -x_1 + 4 x_2 \\
    \text{s.t.}    & \quad -3 x_1 + x_2 \leq 6 \\
                   & \quad -x_1 - 2 x_2 \geq -4 \\
                   & \quad  x_1 \leq 10
\end{align}
$$

The first line is the objective function and the following inequations are the constraints that define the solution space.

The optimal solution of this problem is $\boldsymbol{x}^* = \begin{pmatrix}10 \\ -3\end{pmatrix}$ with a cost equals to -22.

```python
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
```

```python
sns.set_context('talk')
```

```python
def intersect(A, b):
    # Ax = b   <=>   x = A^{-1} b
    inv_A = np.linalg.inv(A)
    return np.dot(inv_A, b)

fig, ax = plt.subplots(figsize=(16, 10))

a1 = [-3,  1]
a2 = [-1, -2]
a3 = [ 1,  0]

a4 = [ 1,  0]
a5 = [ 1,  0]
a6 = [ 0,  1]
a7 = [ 0,  1]

b1 =  6
b2 = -4
b3 = 10

b4 = -3
b5 = 12
b6 = -10
b7 = 50

A = np.array([a1, a2])
b = np.array([[b1], [b2]])
p12 = intersect(A, b)

A = np.array([a1, a3])
b = np.array([[b1], [b3]])
p13 = intersect(A, b)

A = np.array([a2, a3])
b = np.array([[b2], [b3]])
p23 = intersect(A, b)

A = np.array([a1, a4])
b = np.array([[b1], [b4]])
p14 = intersect(A, b)

A = np.array([a1, a5])
b = np.array([[b1], [b5]])
p15 = intersect(A, b)

A = np.array([a2, a4])
b = np.array([[b2], [b4]])
p24 = intersect(A, b)

A = np.array([a2, a5])
b = np.array([[b2], [b5]])
p25 = intersect(A, b)

A = np.array([a3, a6])
b = np.array([[b3], [b6]])
p36 = intersect(A, b)

A = np.array([a3, a7])
b = np.array([[b3], [b7]])
p37 = intersect(A, b)

ax.plot([p14[0], p15[0]], [p14[1], p15[1]], ":k")
ax.plot([p24[0], p25[0]], [p24[1], p25[1]], ":k")
ax.plot([p36[0], p37[0]], [p36[1], p37[1]], ":k")

ax.plot([p12[0], p13[0]], [p12[1], p13[1]], "o-k")
ax.plot([p12[0], p23[0]], [p12[1], p23[1]], "o-k")
ax.plot([p13[0], p23[0]], [p13[1], p23[1]], "o-k")

plt.fill_between([float(p12[0]), b3], np.array([p12[1], p13[1]]).flatten(), np.array([p12[1], p23[1]]).flatten(), facecolor='red', alpha=0.5)

ax.plot([p23[0]], [p23[1]], "or")

# CONTOURS ###

delta = 0.1
x1 = np.arange(b4, b5, delta)
x2 = np.arange(b6, b7, delta)
X1, X2 = np.meshgrid(x1, x2)

Z = -X1 + 4*X2

cs = plt.contour(x1, x2, Z,
                 alpha=0.5,
                 label="TC")
ax.clabel(cs, inline=False, fontsize=12)

ax.set_xlabel("x1")
ax.set_ylabel("x2");
```

Here, the solution space defined by the three constraints is depicted by the red area.
Colored lines are isocontours of the objective function.
The red dot is the optimal solution (the point in the red area that minimize the objective function).

Note: a **linear** program can be fully defined with one matrix and two vectors (respectively $\color{\orange}{A}$, $\color{\green}{b}$ and $\color{\red}{c}$ in the following example):

$$
\begin{align}
    \min_{\boldsymbol{x}} & \quad \color{\red}{\boldsymbol{c}}^{\top} \boldsymbol{x} \\
    \text{s.t.}           & \quad \color{\orange}{\boldsymbol{A}} \boldsymbol{x} \leq \color{\green}{\boldsymbol{b}}
\end{align}
$$

$$
\color{\red}{
\boldsymbol{c} = \begin{pmatrix}
    c_1 \\
    c_2 \\
    \vdots
\end{pmatrix}
}
\quad
\color{\orange}{
\boldsymbol{A} = \begin{pmatrix}
    A_{1,1} & A_{1,2} & \cdots \\
    A_{2,1} & A_{2,2} & \cdots\\
    \vdots  & \vdots  & \ddots
\end{pmatrix}
}
\quad
\color{\green}{
\boldsymbol{b} = \begin{pmatrix}
    b_1 \\
    b_2 \\
    \vdots
\end{pmatrix}
}
$$

Kind of problems solved in mathematical programming:
- Linear programming (LP): linear objective function and constraints, continuous solution space
- Quadratic programming (QP): quadratic objective function and constraints, continuous solution space
- Nonlinear programming: nonlinear objective function and constraints, continuous solution space
- Mixed-integer linear programming (MILP): linear objective function and constraints, locally discrete or continuous solution space
- Mixed-integer quadratic programming: quadratic objective function and constraints, locally discrete or continuous solution space
- Mixed-integer nonlinear programming: nonlinear objective function and constraints, locally discrete or continuous solution space
- Stochastic programming
- Generalized disjunctive programming
- Differential algebraic equations
- Bilevel programming
- Mathematical programs with equilibrium constraints

The two main components involved in mathematical programming are:
- *Algebraic modeling tools* and *Algebraic modeling language*: languages or libraries used to easily *define* optimization problems (optional but very often used)
- *Solvers*: they implement algorithms used to find the optimal solution to optimization problems

Problems are defined with *Algebraic modeling tools* (in a specific *Algebraic modeling language*).
Then the *Algebraic modeling tools* submit this problem to a selected *solver* and get back the optimal solution(s).
*Algebraic modeling tools* are usually compatible with several solvers (thanks to common formats or libraries).

### Algebraic modeling language (AML)

See https://en.wikipedia.org/wiki/Algebraic_modeling_language

Examples of commercial AMLs:
- [AMPL](https://en.wikipedia.org/wiki/AMPL)
- [GAMS](https://en.wikipedia.org/wiki/General_Algebraic_Modeling_System)
- [AIMMS](https://en.wikipedia.org/wiki/AIMMS)

Examples of open source AMLs:
- [Pyomo](https://en.wikipedia.org/wiki/Pyomo)
- [PuLP](https://en.wikipedia.org/wiki/COIN-OR#PuLP)
- [SMI](https://en.wikipedia.org/wiki/COIN-OR#SMI): a stochastic programming modeler and solver
- [GNU MathProg (previously known as GMPL)](https://en.wikipedia.org/wiki/GNU_Linear_Programming_Kit)

### Solvers

Examples of commercial solvers:
- [CPLEX](https://en.wikipedia.org/wiki/CPLEX): linear, quadratic, second-order cone and mixed integer programming
- [Xpress]()
- [Gurobi]()
- [MOSEK](https://en.wikipedia.org/wiki/MOSEK)
- [Artelys Knitro](https://en.wikipedia.org/wiki/Artelys_Knitro): linear, quadratic and nonlinear programming

Examples of open source solvers:
- [GLPK](https://en.wikipedia.org/wiki/GNU_Linear_Programming_Kit): solve linear and mixed integer programming
- [COIN-OR CLP]()
- [COIN-OR CBC]()
- [IPOPT]()
- [Bonmin]()
- [Couenne]()
- [JaCoP](): a constraint solver for constraint satisfaction problems

## What is Pyomo

Pyomo is a Python-based open-source optimization *modeling language* for mathematical programming.

See http://www.pyomo.org/about

## Getting started with Pyomo

**TODO** Installation instructions (Pyomo + open source solvers) can be find on [this page](http://www.jdhp.org/docs/notebook/python_pyomo_getting_started_0_installation_instructions_pyomo_and_solvers.html) or [this one](https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.01-Installing-Pyomo.ipynb).

Installation instructions (Pyomo + open source solvers) can be find on [this page](https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.01-Installing-Pyomo.ipynb).

As an alternative, [Google Colaboratory](https://colab.research.google.com/) can be used to work directly in the cloud and avoid local installation (check for instance [this page](https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.02-Running-Pyomo-on-Google-Colab.ipynb)).

Pyomo's official documentation [is there](http://www.pyomo.org/documentation) and very nice tutorials are [available there](https://jckantor.github.io/ND-Pyomo-Cookbook/).

## References

- Pyomo's official guide: [*Pyomo — Optimization Modeling in Python*, William E. Hart et al., Springer International Publishing (2nd edition) ; 2017](https://focus.universite-paris-saclay.fr/primo-explore/fulldisplay?docid=TN_springer_series978-3-319-58821-6&context=PC&vid=33PUP_VU1&lang=fr_FR&search_scope=default_scope&adaptor=primo_central_multiple_fe&tab=default_tab&query=any,contains,978-3-319-58819-3)
- Modeling optimization problems: [*Optimisation discrète : de la modélisation à la résolution par des logiciels de programmation mathématique*, Alain Billionnet, Paris, Dunod, 2007](https://focus.universite-paris-saclay.fr/primo-explore/fulldisplay?docid=33IMT_ILS24771&context=L&vid=33PUP_VU1&lang=fr_FR&tab=default_tab&query=any,contains,optimisation%20discr%C3%A8te,AND&query=creator,contains,billionnet,AND&mode=advanced&offset=0) (in French)


#
EE 380L: Data Mining

#
Assignment 2

##
Total points: 72

##
Due: Tuesday, February 22nd, submitted via Canvas by 11:59 pm

Your homework should be written in a **Jupyter notebook**. You may work in groups of two if you wish. Only one student per team needs to submit the assignment on Canvas.  But be sure to include name and UTID for both students.  Homework groups will be created and managed through Canvas, so please do not arbitrarily change your homework group.

Also, please make sure your code runs and the graphics (and anything else) are displayed in your notebook before submitting. (%matplotlib inline)

# Question 1: Ridge and Lasso Regression (20 points)
Use the following code to import the dow_jones dataset.

```python
% matplotlib inline

import numpy as np
import pandas as pd
from sklearn import datasets, linear_model
from sklearn.metrics import mean_squared_error

from matplotlib import pyplot as plt
```

```python
df = pd.read_csv('dow_jones_index.data')
df = df.dropna()
X = df[[u'open', u'high', u'low', u'close',
       u'volume', u'percent_change_price',
       u'percent_change_volume_over_last_wk', u'previous_weeks_volume', u'next_weeks_close',
       u'percent_change_next_weeks_price', u'days_to_next_dividend',
       u'percent_return_next_dividend']]
X = X.as_matrix()

for feat in range(X.shape[1]):
    try:
        if "$" in X[0, feat]:
            X[:,feat] = [i.split("$")[1] for i in X[:,feat]]
    except:
        pass
X = X.astype('float')
X = (X - np.mean(X, axis=0))/np.std(X, axis=0)

y = df[[u'next_weeks_open']]
y = y.as_matrix()
y = [i.split("$")[1] for i in y[:, 0]]
y = np.array(y)
y = y.astype('float')
```

You will reuse the same dataset that you used in HW1. In this HW, you will add additional second-order polynomial and interaction variables to the features matrix. Note that this code adds all combinations of the features with degree less than or equal to two; in practice one may introduce only a few based on domain knowledge or experimentation.

```python
from sklearn.preprocessing import PolynomialFeatures
X = PolynomialFeatures(2, include_bias=False).fit_transform(X)
```

In this question, you will explore the application of Lasso and Ridge regression using sklearn package in Python. The following code will split the data into training and test set using [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) with **random state 20** and **test_size = 0.33**.  Note: lambda is called alpha in sklearn.

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.33, random_state=20)
```

1) Use sklearn.linear_model.Lasso and sklearn.linear_model.Ridge classes to do a [5-fold cross validation](http://scikit-learn.org/stable/auto_examples/exercises/plot_cv_diabetes.html#example-exercises-plot-cv-diabetes-py) using sklearn's [KFold](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold). For the sweep of the regularization parameter, we will look at a grid of values ranging from $\lambda = 10^{6}$ to $\lambda = 10^{-2}$. In Python, you can consider this range of values as follows:

      import numpy as np

      alphas =  10**np.linspace(6,-2,100)*0.5

  Report the best chosen $\lambda$ based on cross validation on each model. The cross validation should happen on your training data using  average MSE as the scoring metric. (5pts)

2) Run ridge and lasso for all of the alphas specified above (on training data), and plot the coefficients learned for each of them - there should be one plot each for lasso and ridge, so a total of two plots; the plots for different features for a method should be on the same plot (e.g. Fig 6.6 of JW). What do you qualitatively observe when value of the regularization parameter is changed? What do you observe about the coefficients returned from the ridge and lasso models? (5pts)

3) What are the top 10 most important features for the lasso and ridge models for the hyperparameters that got the best MSE? (2pts)

4) SKLearn can perform this cross validation for us and choose the best regularization hyper-parameter. It will automatically determine the range of $\lambda$ s to search over. Run lasso again with cross validation using [sklearn.linear_model.LassoCV](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html). Set the cross validation parameters as follows:

    LassoCV(alphas=None, cv=10, max_iter=10000)

    Report the best $\lambda$ based on cross validation. (3 pts)

5)
    i) Run Ridge, lasso with the best hyperparameters and report the MSE on test data.
    ii) Run OLS on the training data with all the features and then with the features selected through lasso (features with non zero weights). Report the MSE on test data. (5 pts)

## Answer

```python
from sklearn.cross_validation import KFold
from sklearn.linear_model import Ridge, Lasso

import numpy as np
import matplotlib.pyplot as plt

n_folds = 5
k_fold = KFold(len(X_train), n_folds)
alphas =  10**np.linspace(6,-2,100)*0.5
lasso_avg_rmse = {}
ridge_avg_rmse = {}

for alpha in alphas:
    lasso = Lasso(alpha=alpha, max_iter=1000)
    avg_mse = 0
    for k, (train, test) in enumerate(k_fold):
        lasso.fit(X_train[train], y_train[train])
        avg_mse = avg_mse + mean_squared_error(y_train[test], lasso.predict(X_train[test]))
    lasso_avg_rmse[alpha] = avg_mse / n_folds
best_alpha_lasso = min(lasso_avg_rmse, key=lasso_avg_rmse.get)
print("Best lasso λ: {0}, Best MSE: {1}".format(best_alpha_lasso, min(lasso_avg_rmse)))

for alpha in alphas:
    ridge = Ridge(alpha=alpha, max_iter=1000)
    avg_mse = 0
    for k, (train, test) in enumerate(k_fold):
        ridge.fit(X_train[train], y_train[train])
        avg_mse = avg_mse + mean_squared_error(y_train[test], ridge.predict(X_train[test]))
    ridge_avg_rmse[alpha] = avg_mse / n_folds
best_alpha_ridge = min(ridge_avg_rmse, key=ridge_avg_rmse.get)
print("Best ridge λ: {0}, Best MSE: {1}".format(best_alpha_ridge, min(ridge_avg_rmse)))
```

    Best lasso λ: 0.008737642000038414, Best MSE: 0.005
    Best ridge λ: 0.005, Best MSE: 0.005

As we can see above, the optimal alpha for Lasso is 0.005; and the optimal alpha for Ridge is 0.005.

```python
#2)
cof_lasso, cof_ridge = [], []

for a in alphas:
    regression_lasson = Lasso(alpha = a)
    regression_lasson.fit(X_train, y_train)
    cof_lasso.append(regression_lasson.coef_)

    regression_ridge = Ridge(alpha = a)
    regression_ridge.fit(X_train, y_train)
    cof_ridge.append(regression_ridge.coef_)
# semi-column surpress the unwanted output
# https://stackoverflow.com/questions/25790062/how-to-hide-matplotlib-lines-line2d-in-ipython-notebook
plt.figure().set_size_inches(8, 6);
plt.semilogx(alphas, cof_lasso);
plt.figure().set_size_inches(8, 6);
plt.semilogx(alphas, cof_ridge);
```

For both the Lasso and Ridge methods, the coefficients will become 0 when lambda is large enough. For the Lasso method, the coefficient becomes 0 earlier than the Ridge and can generate different models when $\lambda$ is diffrent. But for the Ridge method, the change is more smooth and all the coefficients will become 0 when $\lambda$ reaches a same value.

```python
#3)

#
df = pd.read_csv('dow_jones_index.data')
df = df.dropna()
X = df[[u'open', u'high', u'low', u'close',
       u'volume', u'percent_change_price',
       u'percent_change_volume_over_last_wk', u'previous_weeks_volume', u'next_weeks_close',
       u'percent_change_next_weeks_price', u'days_to_next_dividend',
       u'percent_return_next_dividend']]
X = X.as_matrix()

names = [u'open', u'high', u'low', u'close',
       u'volume', u'percent_change_price',
       u'percent_change_volume_over_last_wk', u'previous_weeks_volume', u'next_weeks_close',
       u'percent_change_next_weeks_price', u'days_to_next_dividend',
       u'percent_return_next_dividend']

for feat in range(X.shape[1]):
    try:
        if "$" in X[0, feat]:
            X[:,feat] = [i.split("$")[1] for i in X[:,feat]]
    except:
        pass
X = X.astype('float')
X = (X - np.mean(X, axis=0))/np.std(X, axis=0)

y = df[[u'next_weeks_open']]
y = y.as_matrix()
y = [i.split("$")[1] for i in y[:, 0]]
y = np.array(y)
y = y.astype('float')

from sklearn.preprocessing import PolynomialFeatures
feature_names_lasso = PolynomialFeatures(2, include_bias=False).fit(X).get_feature_names(names)
X = PolynomialFeatures(2, include_bias=False).fit_transform(X)

lasso = Lasso(alpha = 0.008)
lasso.fit(X_train, y_train)
# https://stackoverflow.com/questions/6910641/how-to-get-indices-of-n-maximum-values-in-a-numpy-array
max_index = lasso.coef_.argsort()[-10:][::-1]

print(" -------------- Lasso -----------")
for i in max_index:
    print('index: ', i, '   feature name: ', feature_names_lasso[i], '    weight:  ',lasso.coef_[i])
#     print('index {0},   {1:.30},   {2:.10}'.format(i, feature_names_lasso[i], lasso.coef_[i]))

# ####################

print('#' * 60)

df = pd.read_csv('dow_jones_index.data')
df = df.dropna()
X = df[[u'open', u'high', u'low', u'close',
       u'volume', u'percent_change_price',
       u'percent_change_volume_over_last_wk', u'previous_weeks_volume', u'next_weeks_close',
       u'percent_change_next_weeks_price', u'days_to_next_dividend',
       u'percent_return_next_dividend']]
X = X.as_matrix()

for feat in range(X.shape[1]):
    try:
        if "$" in X[0, feat]:
            X[:,feat] = [i.split("$")[1] for i in X[:,feat]]
    except:
        pass
X = X.astype('float')
X = (X - np.mean(X, axis=0))/np.std(X, axis=0)

y = df[[u'next_weeks_open']]
y = y.as_matrix()
y = [i.split("$")[1] for i in y[:, 0]]
y = np.array(y)
y = y.astype('float')

from sklearn.preprocessing import PolynomialFeatures
feature_names_ridge = PolynomialFeatures(2, include_bias=False).fit(X).get_feature_names(names)
X = PolynomialFeatures(2, include_bias=False).fit_transform(X)

ridge = Ridge(alpha= 0.005)
ridge.fit(X_train, y_train)
max_index = ridge.coef_.argsort()[-10:][::-1]
print(" -------------- Ridge -----------")
for i in max_index:
    print('index: ', i, 'feature name: ', feature_names_ridge[i], 'weight:  ', ridge.coef_[i])
```

     -------------- Lasso -----------
    index:  0    feature name:  open     weight:   28.4012191303
    index:  8    feature name:  next_weeks_close     weight:   3.85441096634
    index:  5    feature name:  percent_change_price     weight:   1.18539532599
    index:  28    feature name:  high percent_change_price     weight:   0.611858802042
    index:  1    feature name:  high     weight:   0.436928103707
    index:  17    feature name:  open percent_change_price     weight:   0.136023948367
    index:  3    feature name:  close     weight:   0.0823935323743
    index:  6    feature name:  percent_change_volume_over_last_wk     weight:   0.0230343353713
    index:  11    feature name:  percent_return_next_dividend     weight:   0.00981566193473
    index:  63    feature name:  percent_change_price percent_change_volume_over_last_wk     weight:   0.00237593840013
    ############################################################
     -------------- Ridge -----------
    index:  8 feature name:  next_weeks_close weight:   29.6440509209
    index:  0 feature name:  open weight:   1.72760991947
    index:  3 feature name:  close weight:   1.72621161747
    index:  46 feature name:  close volume weight:   0.96458453443
    index:  49 feature name:  close previous_weeks_volume weight:   0.741990301318
    index:  53 feature name:  close percent_return_next_dividend weight:   0.37841323959
    index:  16 feature name:  open volume weight:   0.348841876175
    index:  18 feature name:  open percent_change_volume_over_last_wk weight:   0.272912264264
    index:  32 feature name:  high percent_change_next_weeks_price weight:   0.22193644543
    index:  34 feature name:  high percent_return_next_dividend weight:   0.21418947485

```python
# 4)
from sklearn.linear_model import LassoCV
lasso_cv = LassoCV(alphas=None, cv=10, max_iter=10000)
regression_lasso = lasso_cv.fit(X_train, y_train)
lambda_optimal = regression_lasso.alpha_
coef = regression_lasso.coef_

print("Best lambda: ", lambda_optimal)
print("Coefficient: ", coef)
```

    Best lambda:  0.0466112165952
    Coefficient:  [  0.00000000e+00   2.53525574e+00   2.16556032e+00   1.91278452e+01
      -0.00000000e+00   5.39972624e-02   0.00000000e+00  -0.00000000e+00
       9.06155486e+00  -3.72584773e-01  -0.00000000e+00   0.00000000e+00
       0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
      -0.00000000e+00   0.00000000e+00   0.00000000e+00  -0.00000000e+00
       0.00000000e+00  -0.00000000e+00   0.00000000e+00  -0.00000000e+00
       0.00000000e+00   0.00000000e+00   0.00000000e+00  -0.00000000e+00
       2.72372354e-02   0.00000000e+00  -0.00000000e+00   0.00000000e+00
      -2.25561293e-01  -0.00000000e+00  -0.00000000e+00   0.00000000e+00
       0.00000000e+00  -0.00000000e+00   0.00000000e+00   0.00000000e+00
      -0.00000000e+00   0.00000000e+00  -0.00000000e+00  -0.00000000e+00
      -0.00000000e+00   7.74405274e-04  -0.00000000e+00   0.00000000e+00
       0.00000000e+00  -0.00000000e+00   0.00000000e+00  -1.06866344e-03
      -0.00000000e+00  -0.00000000e+00  -0.00000000e+00  -0.00000000e+00
      -0.00000000e+00  -0.00000000e+00  -0.00000000e+00   0.00000000e+00
       0.00000000e+00   0.00000000e+00  -0.00000000e+00   0.00000000e+00
      -0.00000000e+00   0.00000000e+00  -0.00000000e+00  -0.00000000e+00
      -0.00000000e+00  -0.00000000e+00  -0.00000000e+00   0.00000000e+00
      -0.00000000e+00   0.00000000e+00   0.00000000e+00  -0.00000000e+00
      -0.00000000e+00   0.00000000e+00  -0.00000000e+00   0.00000000e+00
       0.00000000e+00  -0.00000000e+00  -0.00000000e+00  -0.00000000e+00
      -0.00000000e+00  -0.00000000e+00   0.00000000e+00  -0.00000000e+00
      -0.00000000e+00  -0.00000000e+00]

```python
# 5 - (1))

regression_lasso = Lasso(alpha= 0.0087)
regression_lasso.fit(X_train, y_train)
mse_lasso = np.mean((regression_lasso.predict(X_test) - y_test) ** 2)

regression_ridge = Ridge(alpha= 0.005)
regression_ridge.fit(X_train, y_train)
mse_ridge = np.mean((regression_ridge.predict(X_test) - y_test) ** 2)

print("Lasso MSE:", mse_lasso)
print("Ridge MSE:", mse_ridge)
```

    Lasso MSE: 0.159849159821
    Ridge MSE: 0.00193173303189

```python
from sklearn.linear_model import LinearRegression

X_train_selected = X_train[:,lasso.coef_!=0]
X_test_selected = X_test[:,lasso.coef_!=0]

linear = LinearRegression()
linear.fit(X_train, y_train)
mse = mean_squared_error(y_test, linear.predict(X_test))
print('MSE for linear with all features: {}'.format(mse))

linear.fit(X_train_selected, y_train)
mse = mean_squared_error(y_test, linear.predict(X_test_selected))
print('MSE for linear with selected features: {}'.format(mse))
```

    MSE for linear with all features: 0.0002554088434953395
    MSE for linear with selected features: 0.15441347946418985

# Question 2: Bias-Variance Tradeoff (5 points)

1. Describe the relationship between Ordinary Least Squares and Ridge Regression (the problem analyzed in the previous question) with reference to Bias-Variance trade-off.
3. Which is a higher variance model, kNN with k = 1 or with k = 10?

1.When the number of samples n is far larger than the number of dimensionality p, least square yields lower bias. However, else if n is not sufficiently larger than p, least square can also bring higher variance, which will cause overfitting and poor predictions. Also considering the case when p > n, least square method cannot be used at all since the prediction is no longer reliable.

2.kNN with k = 1 is a higher variance model, because it is only taking the nearest 1 point for clustering; on the other hand, k = 10 means that averaging over 10 nearest points, which will lead to more robust results.

# Question 3 - Principal Component Analysis (15 points)

Import the same dataset with the interaction variables as in Q1.

```python
% matplotlib inline

import numpy as np
import pandas as pd
from sklearn import datasets, linear_model
from sklearn.metrics import mean_squared_error

from matplotlib import pyplot as plt
```

```python
df = pd.read_csv('dow_jones_index.data')
df = df.dropna()
X = df[[u'open', u'high', u'low', u'close',
       u'volume', u'percent_change_price',
       u'percent_change_volume_over_last_wk', u'previous_weeks_volume', u'next_weeks_close',
       u'percent_change_next_weeks_price', u'days_to_next_dividend',
       u'percent_return_next_dividend']]
X = X.as_matrix()

for feat in range(X.shape[1]):
    try:
        if "$" in X[0, feat]:
            X[:,feat] = [i.split("$")[1] for i in X[:,feat]]
    except:
        pass
X = X.astype('float')
X = (X - np.mean(X, axis=0))/np.std(X, axis=0)

y = df[[u'next_weeks_open']]
y = y.as_matrix()
y = [i.split("$")[1] for i in y[:, 0]]
y = np.array(y)
y = y.astype('float')
```

```python
from sklearn.preprocessing import PolynomialFeatures
X = PolynomialFeatures(2, include_bias=False).fit_transform(X)
```

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.33, random_state=20)

from sklearn.decomposition import PCA
pca = PCA()
pca.fit(X_train)
variance = pca.explained_variance_ratio_
variance_plot, = plt.plot(variance, 'r', label='variance')

cumulative_variance = []
N1 = 0
N2 = 0
N3 = 0
count = 0
for item in variance:
    count = count + 1
    if len(cumulative_variance) == 0:
        cumulative_variance.append(item)
        continue
    else:
        cumulative_variance.append(item + cumulative_variance[-1])
    if cumulative_variance[-1] >= 0.9 and cumulative_variance[-2] < 0.9:
        N1 = count
    if cumulative_variance[-1] >= 0.95 and cumulative_variance[-2] < 0.95:
        N2 = count
    if cumulative_variance[-1] >= 0.99 and cumulative_variance[-2] < 0.99:
        N3 = count

cumulative_variance_plot, = plt.plot(cumulative_variance, 'b', label = 'cumulative variance')
plt.legend(handles=[variance_plot, cumulative_variance_plot])
plt.xlabel('Principal Component')
plt.ylabel('Proportion of Variance')
plt.show()

print("Number of principal components to explain cumulative variance of 90%: " + str(N1))
print("Number of principal components to explain cumulative variance of 95%: " + str(N2))
print("Number of principal components to explain cumulative variance of 99%: " + str(N3))
```

```python
import statsmodels.api as sm

X_train_pca = pca.transform(X_train)

X_train_90 = sm.add_constant(X_train_pca[:,:12])
X_train_95 = sm.add_constant(X_train_pca[:,:18])
X_train_99 = sm.add_constant(X_train_pca[:,:29])
X_train_pca = sm.add_constant(X_train_pca)

model_ols = sm.OLS(y_train, X_train_pca).fit()
model_ols_90 = sm.OLS(y_train, X_train_90).fit()
model_ols_95 = sm.OLS(y_train, X_train_95).fit()
model_ols_99 = sm.OLS(y_train, X_train_99).fit()

X_test_pca = pca.transform(X_test)
X_test_90 = sm.add_constant(X_test_pca[:,:12])
X_test_95 = sm.add_constant(X_test_pca[:,:18])
X_test_99 = sm.add_constant(X_test_pca[:,:29])
X_test_pca = sm.add_constant(X_test_pca)

mse = np.mean((model_ols.predict(X_test_pca) - y_test) ** 2)
mse_90 = np.mean((model_ols_90.predict(X_test_90) - y_test) ** 2)
mse_95 = np.mean((model_ols_95.predict(X_test_95) - y_test) ** 2)
mse_99 = np.mean((model_ols_99.predict(X_test_99) - y_test) ** 2)

print("MSE of X_text using N1 features from PCA: " + str(mse_90))
print("MSE of X_text using N2 features from PCA: " + str(mse_95))
print("MSE of X_text using N3 features from PCA: " + str(mse_99))
print("MSE of X_text using all features from PCA: " + str(mse))
```

    MSE of X_text using N1 features from PCA: 145.469202378
    MSE of X_text using N2 features from PCA: 27.7258861025
    MSE of X_text using N3 features from PCA: 3.19815659743
    MSE of X_text using all features from PCA: 0.000255408950307

By comparison, it is easy to find that with more principal components, the MSE of test data will be smaller.

(a) Perform PCA using the sklearn [PCA](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) package.  Create i) a scree plot depicting the proportion of variance and ii) a cumulative proportion of variance explained by the principal components of the data (X_train matrix).  Refer to Figure 10.4 of JW for an example.  You may use the output attribute *explained variance ratio*. How many principal components (N1, N2, N3) are required to explain cumulative variance of 90%, 95%, and 99%, respectively? (6 pts)

(c) Fit an ordinary least squares linear regression using N1, N2, and N3 number of principal components, respectively. Fit PCA on X_train and predict on X_test. Compare the test MSE using N1, N2, N3 to using all features from PCA (6 pts)

(d) Compare this to the MSE reported in Q1.For this dataset, would you use PCA/Lasso as a feature elimination technique based on: (3 pts)
i) Interpretability of results
ii) MSE value
iii) Hyperparameter tuning?

## Answer
(a)
Answer is shown above.
(b)
Answer is shown above.
(c)
i) Based on interpretability of results, I would use Lasso as a feature elimination technique. PCA is actually doing projection, after which features will be hard to be interpreted and with no practical meanings.
ii) Based on MSE value, I would use PCA as a feature elimination technique, since the MSE of PCA is much smaller than that of Lasso.
iii) Based on Hyperparameter tuning, I would use PCA as a feature elimination technique. PCA does not need to find the best parameters for training model by cross validation. However, Lasso with KFolds does.

# Question 4 - Sampling (3+2=5pts)

1 . Your firm is conducting a customer survey for a new product. You are tasked with estimating whether the product will be liked or disliked by the entire market. Unfortunately, you cannot reach all the customers. So you randomly sample 400 participants and ask them "Will you like the product" as a question. The responses are evenly split. But you attended a Data Mining course, so you know the estimate is not simply 50%.

What are the lower and upper bounds of probability corresponding to a confidence of 99% ($\alpha = 0.99$)?

2 . Your manager asks you to halve the probability range you reported in part 1. This can be accomplished in two different ways:

    a) Reduce the confidence without going below 90%.
    OR
    b) Conduct a new survey to acquire more samples. In this case let us assume that the results are again evenly split.

Which option is viable? If a), what is the highest confidence you can use? And if b), how many samples do you need?

## Answer

1.
For each participant, his/her answer takes a Bernoulli Distribution. Its mean is $\hat{p}$, where $\hat{p}$ is $0.5$ here. Its variance is $\sigma^{2} = \frac{\hat{p}(1-\hat{p})}{m}$, where m is sample number, which is 400 here. And thus, $\sigma = \frac{1}{40}$.
When $\alpha = 0.99$, the upper bound of probability should be:
$$
\begin{align}
& 0.5 + 2.58\sigma \\
= \ & 0.5 + 0.0645 \\
= \ & 0.5645
\end{align}
$$
Similarly, the lower bound of probability should be:
$$
\begin{align}
& 0.5 - 2.58\sigma \\
= \ & 0.5 - 0.0645 \\
= \ & 0.4355
\end{align}
\\
$$

2.
Option b) is viable, since option a) actually does nothing and won't improve the accuracy of our estimation. For option b), the required sample number to halve the probability range should be:
$$
\begin{align}
& 0.5 \times (1-0.5) \times (\frac{2.58}{\frac{0.0645}{2}})^2 \\
= \ & 1600
\end{align}
$$

# Question 5: Ridge and Lasso Regression (15 points)
The ridge regression problem is characterized by the following loss function:
$$\mathcal{L}(\beta) = ||X \beta - y||_2^2 + \lambda ||\beta||^2_2$$

The ridge regression problem is the case where the penalty term ($\lambda ||\beta||$) uses the squared l-2 norm ($\lambda ||\beta||^2$).

a) Find the closed form solution to the ridge regresion problem
$$\underset{\beta}{\mathrm{argmin}} \mathcal{L}(\beta)$$
where $\mathcal{L}(\beta)$ is the loss function above.

b) Now given the SVD of $X$ as $U \Sigma V^T$, express $X \hat \beta_{ridge}$ in terms of $U$, $V$ and $\Sigma$

c) Explain what the expression for $X \hat \beta_{ridge}$ derived above tells us about how the addition of the regularization term affects the solution. (Hint: think about $\lambda ||\beta||^2$ as a penalty term)

# Answer
a)
The closed form solution to the ridge regression problem above is:
$$
\begin{align}
\underset{\beta}{\mathrm{argmin}} \mathcal{L}(\beta) & \ =  \ \hat \beta_{ridge}\\
& \ = \ (\lambda I + X^T X)^{-1}X^T y
\end{align}
\\
$$
b)
Given the SVD of $X$ as $U \Sigma V^T$,  $X \hat \beta_{ridge}$ can be expressed as below:
$$
\begin{align}
X \hat \beta_{ridge} & \ = \ U \Sigma V^T (\lambda I + V \Sigma^T U^T U \Sigma V^T)^{-1} V \Sigma^T U^T y \\
& \ = \ U \Sigma V^T (\lambda I + V \Sigma^2 V^T)^{-1} V \Sigma^T U^T y \\
& \ = \ U \Sigma V^T ([V (\lambda I + \Sigma^2) V^T])^{-1} V \Sigma^T U^T y \\
& \ = \ U \Sigma V^T V (\lambda I + \Sigma^2)^{-1} V^{-1} V \Sigma^T U^T y \\
& \ = \ U \Sigma (\lambda I + \Sigma^2)^{-1} \Sigma^T U^T y \\
& \ = \ U \Sigma (\lambda I + \Sigma^2)^{-1} (U \Sigma)^T y \\
\end{align}
\\
$$
c)
According to the expression above, $\hat{y} \ =  \sum_{j=1}^{p} u_j \frac{\sigma^2_j}{\sigma^2_j + \lambda} u_j^T y$, where $u_j$ is the $j^{th}$ normalized principal components of $X$, $\sigma_{j}$ is the $j^{th}$ singular value of $X$. The shrinkage factor given by ridge regression here is:
$$
\frac{\sigma^2_j}{\sigma^2_j + \lambda}
$$
Since for any $\lambda > 0$, the value of expression above will be smaller than that when $\lambda = 0$, the addition of the regularization term $\lambda$ will actually affect the solution by shrinking the singular values. So, when $\lambda$ grows, the regression curve will be compressed to some extent.

# Question 6 - Outliers and Huber Loss (12 pts)

In this problem, we will use the same data set from the previous problem set to fit a linear model to the data using a Huber loss function rather than the l-2 norm usually used in OLS. sklearn has a nice API you can use: [http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.HuberRegressor.html#sklearn-linear-model-huberregressor](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.HuberRegressor.html#sklearn-linear-model-huberregressor)

Below is a snippet from the previous problem set to help you get started. For this problem, the only independent variable will be the "percent\_change\_price" feature.

```python
import pandas as pd
import numpy as np
from sklearn import linear_model
from sklearn.metrics import mean_squared_error

df = pd.read_csv('dow_jones_index.data')
stock = df[["percent_change_price", "next_weeks_open"]]
stock = stock.dropna()
stock = stock.as_matrix()[1:]
stock[:, 1] = [i.split("$")[1] for i in stock[:, 1]]
stock = stock.astype('float')
stock[:, 0] = (stock[:, 0] - np.mean(stock[:, 0], axis=0))/np.std(stock[:, 0], axis=0)

X = stock[:,:1]
y = stock[:, 1]
X_train = X[:400,]
y_train = y[:400]

X_test = X[400:,]
y_test = y[400:,]

clf_train = linear_model.LinearRegression()
clf_train.fit(X_train, y_train)
predictions_train = clf_train.predict(X_train)
mse_train = mean_squared_error(y_train, predictions_train)
print('MSE train: ', mse_train)

predictions_test = clf_train.predict(X_test)
mse_test = mean_squared_error(y_test, predictions_test)
print('MSE test: ', mse_test)
```

    MSE train:  953.538042823
    MSE test:  1192.17336874

a) Now do the same for the Huber loss function and print the test and train MSE. Use a regularization coefficient of 0. (5 pts)

```python
# ANSWER

import pandas as pd
import numpy as np
from sklearn import linear_model
from sklearn.metrics import mean_squared_error

df = pd.read_csv('dow_jones_index.data')
stock = df[["percent_change_price", "next_weeks_open"]]
stock = stock.dropna()
stock = stock.as_matrix()[1:]
stock[:, 1] = [i.split("$")[1] for i in stock[:, 1]]
stock = stock.astype('float')
stock[:, 0] = (stock[:, 0] - np.mean(stock[:, 0], axis=0))/np.std(stock[:, 0], axis=0)

X = stock[:,:1]
y = stock[:, 1]
X_train = X[:400,]
y_train = y[:400]

X_test = X[400:,]
y_test = y[400:,]

clf_train = linear_model.HuberRegressor()
clf_train.fit(X_train, y_train)
predictions_train = clf_train.predict(X_train)
mse_train = mean_squared_error(y_train, predictions_train)
print('MSE train: ', mse_train)

predictions_test = clf_train.predict(X_test)
mse_test = mean_squared_error(y_test, predictions_test)
print('MSE test: ', mse_test)
```

    MSE train:  969.075119932
    MSE test:  1235.96980953

Now let's artificially introduce some errors intro the train set:

```python
y_train_outliers = np.copy(y_train)
y_train_outliers[0] = 10000.0
```

Note that we would never actually do this in a real application, this is simply for demo purposes.

b) Now create two models as before, but using the new y vector during training (one model using OLS and another using the Huber loss). Print the test and train MSE. Use a regularization coefficient of 0. (5 pts)

```python
# ANSWER

import pandas as pd
import numpy as np
from sklearn import linear_model
from sklearn.metrics import mean_squared_error

df = pd.read_csv('dow_jones_index.data')
stock = df[["percent_change_price", "next_weeks_open"]]
stock = stock.dropna()
stock = stock.as_matrix()[1:]
stock[:, 1] = [i.split("$")[1] for i in stock[:, 1]]
stock = stock.astype('float')
stock[:, 0] = (stock[:, 0] - np.mean(stock[:, 0], axis=0))/np.std(stock[:, 0], axis=0)

X = stock[:,:1]
y = stock[:, 1]
X_train = X[:400,]
y_train = y[:400]

X_test = X[400:,]
y_test = y[400:,]

y_train_outliers = np.copy(y_train)
y_train_outliers[0] = 10000.0

clf_train = linear_model.LinearRegression()
clf_train.fit(X_train, y_train_outliers)
predictions_train = clf_train.predict(X_train)
mse_train = mean_squared_error(y_train_outliers, predictions_train)
print('(OLS) MSE train: ', mse_train)

predictions_test = clf_train.predict(X_test)
mse_test = mean_squared_error(y_test, predictions_test)
print('(OLS) MSE test: ', mse_test)

clf_train_huber = linear_model.HuberRegressor()
clf_train_huber.fit(X_train, y_train_outliers)
predictions_train = clf_train_huber.predict(X_train)
mse_train = mean_squared_error(y_train_outliers, predictions_train)
print('(Huber loss) MSE train: ', mse_train)

predictions_test = clf_train_huber.predict(X_test)
mse_test = mean_squared_error(y_test, predictions_test)
print('(Huber loss) MSE test: ', mse_test)
```

    (OLS) MSE train:  246073.324779
    (OLS) MSE test:  3570.05063993
    (Huber loss) MSE train:  248826.748391
    (Huber loss) MSE test:  1232.54690548

c) Explain the difference in prediction performance of the two models in these two scenarios (one without significant outliers in the data, and the other with a single outlier). (2 pts)

Answer:
Both the MSE of training datasets of the two models are high. However, when using Huber loss, the MSE of test dataset is much less than that when using OLS. And it alomost keeps the same with the one when there is no significant outliers. The reason is that, Huber loss helps build robust regression. It adopts linear penalty for large residuals, and thus, when there is a significant outlier, its effects on the coefficients of prediction model will be much less than OLS's. In this way, the model can still be accurate.