# Machine Learning: Basic Principles 2018

# Feature Learning

### Learning goals

In this exercise you will learn how to use __principal component analysis (PCA)__ for __feature learning__. PCA allows us to optimally compress the information contained in high-dimensional raw feature vectors into a small set of new features, which are called __principal components__ (PC). These new features are then used for formulating and solving an overall ML problem  (regression or classification). Using the PCs as features (instead of the raw feature vectors) is beneficial in terms of computational complexity (using smaller feature vectors means less computation) and statistical properties of the learning method (smaller feature vectors reduces the risk of overfitting). Moreover, using the first two PCs allows to visualize high-dimensional data points conveniently using a scatter plot.

### Exercise Contents

1. [Data](#data)
2. Exercise
    - 2.1 [Implement Principal Component Analysis (PCA)](#Q1)
    - 2.2 [Compression vs.  Information Loss](#Q2)
    - 2.3 [PCA for Data Visualization](#Q3)

### Keywords
`principal component analysis (PCA)`, `eigenvalue decomposition`, `data visualization`, `overfitting`.

## 1. The Data

In this exercise we will work with a dataset containing fruits, more precisely apples and bananas.
The available dataset consists of $N=30$ pictures, saved in the folder named `fruits`:
* 15 images of apples (`image_1.jpg` to `image_15.jpg`)
* 15 images of bananas (`image_16.jpg` to `image_30.jpg`)

The images are color images, but for this image we convert them to grayscale. This means that each pixel has one value between 0 and 255, where 0 means white and 255 means black. The size of the images is $50x50$ pixels. We convert each image to an array of length 2500.

The following code block loads the images and stores them in the array `datamatrix`. We also plot six of the images, to get a feeling for the data.

```python
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np

samplesize = 30
dataset = np.zeros((samplesize,50,50),dtype=np.uint8)

for i in range(1,samplesize+1):
    # with convert('L') we convert the images to grayscale
    img = Image.open('./fruits/%s.jpg'%(str(i))).convert('L')
    dataset[i-1] = np.array(img,dtype=np.uint8)

# Reshaping the images as vectors of length 2500 and store them in the rows of "datamatrix"
datamatrix = np.reshape(dataset,(samplesize,-1))
print("The shape of the datamatrix is", datamatrix.shape)

fig,ax = plt.subplots(3,2,figsize=(10,10),gridspec_kw = {'wspace':0, 'hspace':0})

for i in range(3):
    for j in range(2):
        ax[i,j].imshow(dataset[12+i+j*3], cmap='gray')
        ax[i,j].axis('off')
plt.show()
```

## 2 Exercise
The actual exercise starts from here and it has been divided into 3 tasks:
* 2.1 Implement Principal Component Analysis (PCA)
* 2.2 Compression vs.  Information Loss
* 2.3 PCA for Data Visualization

## 2.1 Implement Principal Component Analysis (PCA)

Each fruit image is represented by the raw data vector  $\mathbf{z}^{(i)} \in \mathbb{R}^{D}$, which contains the grayscale values for each of the $D=2500$ pixels.
The aim of PCA is to find the (in a certain sense) optimal compression matrix $\mathbf{W} \in \mathbb{R}^{d \times D}$, with the number of features $d \ll D$, that transforms the raw data vector $\mathbf{z}^{(i)}$ to a short feature vector $\mathbf{x}^{(i)}$.

\begin{equation}
    \mathbf{x}^{(i)}=\mathbf{W} \mathbf{z}^{(i)}.
    \tag{1}
    \label{eq:1}
\end{equation}

The optimal compression matrix $\mathbf{W}$ is the matrix which gives the lowest loss of information. We quantify the loss of information when replacing the original raw data vectors $\mathbf{z}^{(i)}$ with the new feature vectors $\mathbf{x}^{(i)}$ by the smallest possible  error achieved by reconstructing (in a linear fashion) the raw data vector $\mathbf{z}^{(i)}$ from the new feature vector $\mathbf{x}^{(i)}$. The reconstruction error is given by:

\begin{align}
 \mathcal{E}(\mathbf{W}) & = \min_{\mathbf{R} \in \mathbb{R}^{D \times d}} \dfrac{1}{N} \sum_{i=1}^{N} \| \mathbf{z}^{(i)} - \mathbf{R} \mathbf{x}^{(i)} \|^{2}  \nonumber \\
 & \stackrel{\eqref{eq:1}}{=} \min_{\mathbf{R} \in \mathbb{R}^{D \times d}} \dfrac{1}{N} \sum_{i=1}^{N} \| \mathbf{z}^{(i)} - \mathbf{R} \mathbf{W} \mathbf{z}^{(i)} \|^{2}.
 \label{eq:2}
 \tag{2}
\end{align}

Note that the optimal reconstruction matrix $\mathbf{R}$ depends on the given compression matrix $\mathbf{W}$. However, we are not interested in the optimal reconstruction matrix itself but only in the associated minimum reconstruction error $\mathcal{E}(\mathbf{W})$. In order to construct the optimal compression matrix $\mathbf{W}$ which minimizes the reconstruction error $\mathcal{E}(\mathbf{W})$, we will use the eigenvalue decomposition of the sample covariance matrix $\mathbf{Q}$.

\begin{equation}
\mathbf{Q} = \dfrac{1}{N} \mathbf{Z}^{T} \mathbf{Z} \mbox{ with data matrix } \mathbf{Z}=\big(\mathbf{z}^{(1)},\ldots,\mathbf{z}^{(N)} \big)^{T} \in \mathbb{R}^{N \times D}.
\label{eq:3}
\tag{3}
\end{equation}

Since the matrix $\mathbf{Q}$ is **positive semi-definite**, it has an eigenvalue decomposition of the form

\begin{equation}
\mathbf{Q} = \big( \mathbf{u}^{(1)},\ldots,\mathbf{u}^{(D)}\big) {\rm diag} \{\lambda^{(1)},\ldots,\lambda^{(D)} \}
 \big( \mathbf{u}^{(1)},\ldots,\mathbf{u}^{(D)}\big)^{T}.
\end{equation}

Here, the orthonormal vectors $\mathbf{u}^{(r)}$ are eigenvectors of $\mathbf{Q}$, which correspond to the decreasingly ordered eigenvalues $\lambda^{(1)} \geq \lambda^{(2)} \geq \ldots \geq \lambda^{(D)}$ of $\mathbf{Q}$.

It can be shown that the reconstruction error $\mathcal{E}(\mathbf{W}_{\rm PCA}) = \min_{\mathbf{W} \in \mathbb{R}^{d \times D}} \mathcal{E}(\mathbf{W})$ (see \eqref{eq:2}) can only be minimal for the particular compression matrix

\begin{equation}
\label{eq:4}
\tag{4}
\mathbf{W}_{\rm PCA} = \big( \mathbf{u}^{(1)},\ldots,\mathbf{u}^{(d)} \big)^{T},\in \mathbb{R}^{d \times D}
\end{equation}

whose rows are the eigenvectors $\mathbf{u}^{(1)},\ldots,\mathbf{u}^{(d)}$ of $\mathbf{Q}$ corresponding to the
$d$ largest eigenvalues $\lambda^{(1)},\ldots,\lambda^{(d)}$ of $\mathbf{Q}$.

Moreover, it can be shown that the minimum reconstruction error $\mathcal{E}(\mathbf{W}_{\rm PCA})$ is achieved
for the reconstruction matrix $\mathbf{R} = \mathbf{W}_{\rm PCA}^{T}$. This simplifies the reconstruction error $\mathcal{E}(\mathbf{W}_{\rm PCA})$ to

\begin{equation}
\label{eq:5}
\tag{5}
\mathcal{E}(\mathbf{W}_{\rm PCA}) = \sum_{r = d+1}^{D} \lambda^{(r)}.
\end{equation}

For an intuitive explanation of PCA, this is an excellent StackExchange answer: https://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues

#### Tasks
- Implement a Python function `compute_pca`, which has two input arguments: the data matrix $\mathbf{Z}$ and the number $d$ of new features (principal components). In particular:

    - compute the sample covariance matrix $\mathbf{Q}=\dfrac{1}{N} \mathbf{Z}^{T} \mathbf{Z}$ (see \eqref{eq:3})
    - compute eigenvectors $(\mathbf{u}^{(1)},...,\mathbf{u}^{(D)})$ and corresponding eigenvalues $(\lambda^1,...,\lambda^D)$ of $\mathbf{Q}$. (see the Round 1 - Introduction notebook for a review of eigendecomposition in Python).
    - sort the eigenvalues in a decreasing order: $\lambda^{(1)} \geq \lambda^{(2)} \geq...\lambda^{(D)} \geq 0$.
    - choose $d$ eigenvectors corresponding to the $d$  largest eigenvalues to form the compression matrix $\mathbf{W}_{\rm PCA}=(\mathbf{u}^{(1)},...,\mathbf{u}^{(d)})^{T} \in \mathbb{R}^{d \times D}$
    - return the compresssion matrix $\mathbf{W}_{\rm PCA}$ (see \eqref{eq:4}) and all eigenvalues of $\mathbf{Q}$.

**Note:** You may get eigenvectors and eigen values in complex data type (a+bi format), this occurs due to small numerical errors in calculation. The imaginary term is usually zero and can be ignored. The real part can be extracted using `.real`.

```python
def compute_pca(Z, d):
    # Input: the N by D data matrix Z, the number of components d
    # Output: a d by D matrix W_pca, and all eigenvalues of Q

    ### STUDENT TASK ###

    # step1: compute the sample cov. matrix Q
    # YOUR CODE HERE
    raise NotImplementedError()

    #step2: compute the eigenvalues and eigenvectors (see introduction notebook)
    # YOUR CODE HERE
    raise NotImplementedError()

    #step3: Sort the eigenvectors by decreasing eigenvalues, choose the d largest eigenvalues, form W_pca
    # YOUR CODE HERE
    raise NotImplementedError()

    return W_pca.real,eigvalues
```

## 2.2. Compression vs Information Loss

We are going to use the PCA function you have implemented above to compute the new feature vectors $\mathbf{x}^{(i)}$ for the raw feature vectors $\mathbf{z}^{(i)}$ of the images ([see Section 1 “The Data”](#data)). We will try out using different values of $d$, the length of the new feature vectors, in order to see that there is a trade-off between compression (which requires small values of $d$) and having small loss of information and reconstruction error (which requires large values of $d$).

```python
# Compute the first 50 components of PCA and its eigenvectors
# We divide the datamatrix by 255 since PCA works better on normalized data
PCA,eigvalues = compute_pca(datamatrix/255.,50)
```

To get an insight of how the reconstruction error changes when $d$ increases, we make a plot of this.

```python
def plot_error(eigvalues,max_d):
    x=range(1,max_d+1)
    errors=[sum(eigvalues[d:]) for d in x]
    plt.plot(x,errors)
    plt.xlabel('Number of principal components $d$')
    plt.ylabel('Reconstruction error $\mathcal{E}$')
    plt.title('Number of principal components vs the reconstruction error')
    plt.show()

# plot the number of principal components vs the reconstruction error
plot_error(eigvalues,30)
```

Now let's try to understand what PCA is doing. For this we plot a few of the rows $\mathbf{u}^{(r)}$ of the optimal compression matrix $\mathbf{W}_{\rm PCA}$. These rows, which are eigenvectors of the sample covariance matrix $\mathbf{Q}$, are sometimes called "principal directions".

```python
def plot_princ_comp(PCA):
    fig,ax = plt.subplots(1,5,figsize=(15,15))
    # select the principal components we are plotting
    # You can change these to see what other components look like
    plot_pd = [0,4,9,14,19]

    for i in range(len(plot_pd)):
        ax[i].imshow(np.reshape(PCA[plot_pd[i]]*255,(50,50)),cmap='gray')
        ax[i].set_title("Principal Direction %d"%(plot_pd[i]+1))
    plt.show()

plot_princ_comp(PCA)
```

To get an even better understanding, we want to create the reconstructed images using $ d $ components, i.e. $\mathbf{R} \mathbf{x}^{(i)}$.

We plot the same images as at the start of the notebook, but now for $d=1,5,50$.

Can you see the differences? You can try yourself how a different number of $d$ changes the output!

```python
from matplotlib.lines import Line2D
import matplotlib.gridspec as gridspec
import warnings
warnings.filterwarnings("ignore")

### Input:
##  X: Dataset
##  d: number of dimensions
##  n_pics: number of pics per class (Apple, Banana). Min 1, Max 15
def plot_reconstruct(X,d, n_pics=5):
    # x=w*z
    X_pca = np.matmul(PCA[:d,:],X[:,:,None])
    # x_reversed=r*x
    X_reversed = np.matmul(PCA[:d,:].T, X_pca)[:,:,0]

    # Setup Figure size that scales with number of images
    fig = plt.figure(figsize = (4,max(8*np.log(n_pics),5)))

    # Setup a (n_pics,2) grid of plots (images)
    gs = gridspec.GridSpec(n_pics, 2)
    gs.update(wspace=0.0, hspace=0.0)
    for i in range(n_pics):
        for j in range(0,2):
            # Add a new subplot
            ax = plt.subplot(gs[i,j])

            # Insert image data into the subplot
            ax.imshow(np.reshape(X_reversed[i+(15*j)],(50,50)),cmap='gray',interpolation='nearest')

            # Remove x- and y-axis from each plot
            ax.set_axis_off()

    # Setup a visual divider between Apples and Bananas (i.e. a black line between plots)
    transFigure = fig.transFigure.inverted()
    column_divider = Line2D((0.5,0.5),(0.13,0.91),color='black', transform=fig.transFigure)
    fig.lines.extend([column_divider])

    # Setup column title for Apples and Bananas.
    plt.subplot(gs[0,0]).set_title('Apples', size='large', y=1.08)
    plt.subplot(gs[0,1]).set_title('Bananas', size='large', y=1.08)

    # Render the plot
    plt.show()
```

```python
# The values of d to plot for. You can change these to experiment
# If you want to print different amount of pictures, you can change the value of n_pics. (1-15)
num_com=[1,5,50]
for d in num_com:
    print("Reconstructed image using %d principal components:"%(d))
    plot_reconstruct(datamatrix,d,n_pics=5)
```

## 2.3 PCA for Data Visualization

One of the uses of PCA is visualization. We do PCA with $d=2$  in order to visualize the dataset $\mathbf{z}^{(1)},\ldots,\mathbf{z}^{(N)}$. Indeed, the resulting feature vectors $\mathbf{x}^{(1)} = W_{\rm PCA} \mathbf{z}^{(1)}, \ldots, \mathbf{x}^{(N)} = W_{\rm PCA} \mathbf{z}^{(N)} \in \mathbb{R}^{2}$
can be visualized as a scatter plot whose x-axis represents the first principal component $x_{1}^{(i)}$ and y-asix represents the second principal component $x_{2}^{(i)}$.

Recall that in Round 3 - Classification, we displayed similar scatter plots based on the manually chosen features (the colors). We now used PCA to find the features instead of manually choosing them.

```python
def plot_scatter(PCA,Z):
    # get x for d=2
    X_2d = np.matmul(PCA[:2,:],Z[:,:,None])[:,:,0]

    plt.scatter(X_2d[:15,0],X_2d[:15,1],c='r',marker='o',label='Apple')
    plt.scatter(X_2d[15:,0],X_2d[15:,1],c='y',marker='^',label='Banana')
    plt.legend()
    plt.xlabel('First principal component')
    plt.ylabel('Second principal component')
    plt.show()

plot_scatter(PCA,datamatrix)
```

```python

```


# An Introduction to the Black-Litterman in Python

## Introduction
### Background and Theory

The Black-Litterman asset allocation model \cite{black1992global}, \cite{he1999intuition} provides a methodical way of combining an investors subjective views of the future performance of a risky investment asset with the views implied by the market equilibrium. The method has seen wide acceptance amongst practitioners as well as academics in spite of the fact that it originated as an internal Goldman Sachs working paper, rather than as a piece of research from academia.

The Black Litterman procedure can be viewed as a bayesian shrinkage method, that shrinks the expected returns constructed from an investor's views on asset returns towards asset returns implied by the market equilibrium. The procedure computes a set of expected returns that uses the market equilibrium implied  as a prior. This is then combined with returns implied by subjective investor views to produce a set of posterior expected returns $\mu^{BL}$ and covariances $\Sigma^{BL}$.

Besides the obvious attraction of being able to incorporate subjective investor views, the Black-Litterman procedure has a second feature that makes it extremely attractive to portfolio optimization. It is well known that the Markowitz optimization procedure is highly sensitive to estimation errors in Expected Returns and Covariances, and this _error maximizing_ nature of the Markowitz procedure causes unstable portfolios with extreme weights that diverge rapidly from the market equilibrium portfolio even with minor changes to the inputs (e.g. \cite{chopra1993effect}, \cite{michaud1989markowitz}). However, the posterior parameters $\mu^{BL}, \Sigma^{BL}$ computed by the Black Litterman procedure are derived in part from the market portfolio, and therefore are much more pragmatic inputs for purposes of portfolio optimization. Specifically, when $\mu^{BL}, \Sigma^{BL}$ as used as as inputs to a Markowitz Optimizer, they produce optimized weights that diverge from the market portfolio in limited ways, and only to the extent of the confidence that the investor expresses in the views. Consequently the optimized portfolios are more stable portfolios than with pure Markowitz optimization with sample estimates. In the extreme, with appropriately set parameters, the Markowitz portfolio computed from the Black-Litterman parameters when there are no subjective investor views exactly coincides and is able to recover the market equilibrium portfolio.

### The Black Litterman Formulas

Assume that we have $N$ assets, and $K$ views. There are two sets of inputs to the procedure. The first set of inputs relate to market parameters and these are:

\begin{array}{ll}
w & \mbox{A Column Vector ($N \times 1$) of Equilibrium Market Weights of the Assets} \\
\Sigma & \mbox{A Covariance Matrix ($N \times N$) of the Assets} \\
R_f & \mbox{The Risk Free Rate} \\
\delta & \mbox{The investor's Risk Aversion parameter}  \\
\tau & \mbox{A scalar indicating the uncertainty of the prior (details below)}
\end{array}

Some of these parameters can be inferred from other parameters if they are not explicitly specified. For instance, the risk aversion parameter can be set arbitrarily. For instance, some authors use $\delta = 2.5$ while others use the value of $\delta = 2.14$ in order to be consistent with the value calculated in \cite{dimson2008triumph}.

\cite{beach2007application} suggest using $2.65$. Another common approach is to set $\delta$ to the Market Price of Risk (i.e. a measure of the risk aversion of the Representative Investor, which is computed as $\delta = \mu_M/\sigma^2_M$ where $\mu_M$ and $\sigma^2_M$ are estimates of the mean and variance of the returns of the market portfolio. Frequently, a broad market index such as the S\&P500 is taken as a proxy for the market in order to compute the market price of risk from $\mu_M$ and $\sigma^2_M$.

The treatment of $\tau$ is the source of some confusion. As we will explain in the following section, some implementors have done away with $\tau$ by setting it to $1$ or to calibrate the model to $tau$. In the original model, Black and Litterman suggest using a small number. A common technique is to set $\tau = 1/T$ where $T$ is the number of periods of data used. Thus, for $T=5$ you would use $1/(5 \times 12)$ which yields a value of approximately $\tau=.02$.

The second set of inputs that the procedure needs is a representation of the investors views. These are specified via:

\begin{array}{ll}
Q & \mbox{An $K \times 1$ ``Qualitative Views'' or simply, Views matrix} \\
P & \mbox{A $K \times N$ ``Projection'' or ``Pick'' matrix, linking each view to the assets} \\
\Omega & \mbox{A Covariance matrix representing the uncertainty of views}
\end{array}

Views are represented in $Q$ and $P$ as follows:

If the $k$-th view is an absolute view, it is represented by setting $Q_k$ to the expected return of asset $k$ and setting $P_{ki}$ to 1 and all other elements of row $k$ in $P$ to zero.

If the $k$-th view is an relative view, between assets $i$ and $j$ it is represented by setting $Q_k$ to the expected difference of returns between assets $i$ and $j$, and setting $P_{ki}$ to $-1$ for the underperforming asset, $P_{kj}$ to $+1$ and all other elements of row $k$ in $P$ to zero. $\Omega$ is either set to the specified uncertainty or is inferred from the user or from the data.

The uncertainty of the views $\Omega$ is either set by the user, or inferred (e.g. via statements of confidence, from market data, from the variance of residuals from a prediction model used to generate the views etc, we shall see examples in sections below). In particular, \cite{he1999intuition} suggest setting it to be the diagonal matrix obtained from the diagonal elements of $P \tau \Sigma P^T$, which is what we shall do for some of our initial tests. In my implementation the code accepts a matrix, but uses this assumption as the default if the user does not specify a matrix to use as $\Omega$.

#### The Master Formula

The first step of the procedure is a _reverse-optimization_ step that infers the implied returns vector $\pi$ that are implied by the equilibrium weights $w$ using the formula:

$$\pi = \delta\Sigma w$$

Next, the posterior returns and covariances are obtained from the _Black-Litterman Master Formula_ which is the following set of equations:

\begin{equation}
\label{eq:blMuOrig}
\mu^{BL} = [(\tau\Sigma)^{-1} + P \Omega^{-1} P]^{-1}[(\tau\Sigma)^{-1} \pi + P \Omega^{-1} Q]
\end{equation}

\begin{equation}
\label{eq:blSigmaOrig}
\Sigma^{BL} = \Sigma + [(\tau\Sigma)^{-1} + P \Omega^{-1} P]^{-1}
\end{equation}

#### Inverting $\Omega$

While the master formulas identified in Equation \ref{eq:blMuOrig} and Equation \ref{eq:blSigmaOrig} are frequently easy to implement, they do involve the term $\Omega^{-1}$. Unfortuantely, $\Omega$ is sometimes non-invertible, which poses difficulties to implement the equations as-is. Fortunately the equations are easily transformed to a form that does not require this troublesome inversion. Therefore, frequently, implementations use the following equivalent versions of these equations which are sometimes computationally more stable, since they do not involve inverting $\Omega$. Derivations of these alternate forms are provided in the appendices of \cite{walters2011black}:

\begin{equation}
\label{eq:blMu}
\mu^{BL} = \pi + \tau \Sigma P^T[(P \tau \Sigma P^T) + \Omega]^{-1}[Q - P \pi]
\end{equation}

\begin{equation}
\label{eq:blSigma}
\Sigma^{BL} = \Sigma + \tau \Sigma - \tau\Sigma P^T(P \tau \Sigma P^T + \Omega)^{-1} P \tau \Sigma
\end{equation}

### Flavors of Black-Litterman

The original method described above has also seen a number of modifications and extensions (e.g. see \cite{walters2011black} for an extensive and detailed summary) to the point where there is some confusion about exactly what comprises the true _Black-Litterman_ model.

I shall use a nomenclature that is consistent with \cite{walters2011black}. Walters classifies implementations in two broad categories. The first category was implemented by \cite{black1992global} and \cite{he1999intuition}, and Walters refers to these as the _Reference Model_. The second category consists of well known implementations described in \cite{satchell2000demystification} and a series of papers by Meucci (e.g. \cite{meucci2005beyond}, \cite{meucci2009enhancing}, \cite{meucci2012fully}). In these models, the $\tau$ parameter is eliminated, either by setting it to 1 or by incorporating it into the $\Omega$ matrix.

For the rest of this document, I shall be restricting myself to the _Reference Model_ as originally described in \cite{black1992global} and \cite{he1999intuition}, and I shall not be implementing the extensions of Meucci and others.

### Implementation Overview

The rest of this notebook proceeds as follows. In the following section, I shall implement the Black Litterman procedure in Python and annotate the code as I proceed, to illustrate each step. I then use the code to exactly reproduce the results in \cite{he1999intuition}.

Having established that the code accurately implements the Black Litterman procedure, I shall get down apply the procedure to the Fama French 6-portfolio allocation problem. Along the way, my tests will impose absolute views as well as relative views, and test the impact of the procedure on portfolios using a range of Seven different prediction strategies to obtain views. I also backtest these strategies over time and examine various portfolio metrics, while comparing the Black Litterman derived (BL) expected returns being supplied to an optimizer with weights obtained from Naive Mean-Variance optimization using expected returns and covariance matrixes directly from the prediction strategy. Finally, I conclude the section by examining the impact of these portfolios on transaction costs.

## Annotated Implementation of Black-Litterman
### The Code

The Black Litterman procedure is implemented in Python in the function `bl`. Before we implement the body of `bl`, let's build a few helper functions that will hopefully make the code a bit easier to understand and deal with.

numpy treats a column vector differently from a 1 dimensional array. In order to consistently use column vectors, the following helper function takes either a numpy array or a numpy one-column matrix (i.e. a column vector) and returns the data as a column vector. Let's call this function `as_colvec`

```python
import numpy as np
import pandas as pd

def as_colvec(x):
    if (x.ndim == 2):
        return x
    else:
        return np.expand_dims(x, axis=1)

```

```python
np.arange(4)
```

    array([0, 1, 2, 3])

```python
as_colvec(np.arange(4))
```

    array([[0],
           [1],
           [2],
           [3]])

Recall that the first step in the Black Litterman procedure was to reverse engineer the implied returns vector $\pi$ from a set of portfolio weights $w$.

$$\pi = \delta\Sigma w$$

This is performed by the following code:

```python
def implied_returns(delta, sigma, w):
    """
Obtain the implied expected returns by reverse engineering the weights
Inputs:
delta: Risk Aversion Coefficient (scalar)
sigma: Variance-Covariance Matrix (N x N) as DataFrame
    w: Portfolio weights (N x 1) as Series
Returns an N x 1 vector of Returns as Series
    """
    ir = delta * sigma.dot(w).squeeze() # to get a series from a 1-column dataframe
    ir.name = 'Implied Returns'
    return ir

```

As we noted previously, \cite{he1999intuition} suggest that if the investor does not have a specific way to explicitly quantify the uncertaintly associated with the view in the $\Omega$ matrix, one could make the simplifying assumption that $\Omega$ is proportional to the variance of the prior.

Specifically, they suggest that:

$$\Omega = diag(P (\tau \Sigma) P^T) $$

This is implemented in Python as:

```python
# Assumes that Omega is proportional to the variance of the prior
def proportional_prior(sigma, tau, p):
    """
    Returns the He-Litterman simplified Omega
    Inputs:
    sigma: N x N Covariance Matrix as DataFrame
    tau: a scalar
    p: a K x N DataFrame linking Q and Assets
    returns a P x P DataFrame, a Matrix representing Prior Uncertainties
    """
    helit_omega = p.dot(tau * sigma).dot(p.T)
    # Make a diag matrix from the diag elements of Omega
    return pd.DataFrame(np.diag(np.diag(helit_omega.values)),index=p.index, columns=p.index)

```

We use this function to compute the posterior expected returns as follows:

```python
from numpy.linalg import inv

def bl(w_prior, sigma_prior, p, q,
                omega=None,
                delta=2.5, tau=.02):
    """
# Computes the posterior expected returns based on
# the original black litterman reference model
#
# W.prior must be an N x 1 vector of weights, a Series
# Sigma.prior is an N x N covariance matrix, a DataFrame
# P must be a K x N matrix linking Q and the Assets, a DataFrame
# Q must be an K x 1 vector of views, a Series
# Omega must be a K x K matrix a DataFrame, or None
# if Omega is None, we assume it is
#    proportional to variance of the prior
# delta and tau are scalars
    """
    if omega is None:
        omega = proportional_prior(sigma_prior, tau, p)
    # Force w.prior and Q to be column vectors
    # How many assets do we have?
    N = w_prior.shape[0]
    # And how many views?
    K = q.shape[0]
    # First, reverse-engineer the weights to get pi
    pi = implied_returns(delta, sigma_prior,  w_prior)
    # Adjust (scale) Sigma by the uncertainty scaling factor
    sigma_prior_scaled = tau * sigma_prior
    # posterior estimate of the mean, use the "Master Formula"
    # we use the versions that do not require
    # Omega to be inverted (see previous section)
    # this is easier to read if we use '@' for matrixmult instead of .dot()
    #     mu_bl = pi + sigma_prior_scaled @ p.T @ inv(p @ sigma_prior_scaled @ p.T + omega) @ (q - p @ pi)
    mu_bl = pi + sigma_prior_scaled.dot(p.T).dot(inv(p.dot(sigma_prior_scaled).dot(p.T) + omega).dot(q - p.dot(pi).values))
    # posterior estimate of uncertainty of mu.bl
#     sigma_bl = sigma_prior + sigma_prior_scaled - sigma_prior_scaled @ p.T @ inv(p @ sigma_prior_scaled @ p.T + omega) @ p @ sigma_prior_scaled
    sigma_bl = sigma_prior + sigma_prior_scaled - sigma_prior_scaled.dot(p.T).dot(inv(p.dot(sigma_prior_scaled).dot(p.T) + omega)).dot(p).dot(sigma_prior_scaled)
    return (mu_bl, sigma_bl)

```

### A Simple Example: Absolute Views

We start with a simple 2-Asset example. Let's start with an example from _Statistical Models and Methods for Financial Markets (Springer Texts in Statistics) 2008th Edition, Tze Lai and Haipeng Xing_.

Consider the portfolio consisting of just two stocks: Intel (INTC) and Pfizer (PFE).

From Table 3.1 on page 72 of the book, we obtain the covariance matrix (multipled by $10^4$)

\begin{array}{lcc}
INTC & 46.0 & 1.06 \\
PFE   & 1.06 & 5.33
\end{array}

Assume that Intel has a market capitalization of approximately USD 80B and that of Pfizer is approximately USD 100B (this is not quite accurate, but works just fine as an example!).
Thus, if you held a market-cap weighted portfolio you would hold INTC and PFE with the following weights: $W_{INTC} = 80/180 = 44\%, W_{PFE} = 100/180 = 56\%$. These appear to be reasonable weights without an extreme allocation to either stock, even though Pfizer is slightly overweighted.

We can compute the equilibrium implied returns $\pi$ as follows:

```python
tickers = ['INTC', 'PFE']
s = pd.DataFrame([[46.0, 1.06], [1.06, 5.33]], index=tickers, columns=tickers) *  10E-4
pi = implied_returns(delta=2.5, sigma=s, w=pd.Series([.44, .56], index=tickers))
pi
```

    INTC    0.052084
    PFE     0.008628
    Name: Implied Returns, dtype: float64

Thus the equilibrium implied returns for INTC are a bit more than 5\% and a bit less than 1\% for PFE.

Assume that the investor thinks that Intel will return 2\% and that Pfizer is poised to rebounce, and will return 4\% . We can now examine the optimal weights according to the Markowitz procedure.
What would happen if we used these expected returns to compute the Optimal Max Sharpe Ratio portfolio?

The Max Sharpe Ratio (MSR) Portfolio weights are easily computed in explicit form if there are no constraints on the weights.
The weights are given by the expression (e.g. See  \cite{campbell1996econometrics} page 188 Equation 5.2.28):

$$ W_{MSR} = \frac{\Sigma^{-1}\mu_e}{\bf{1}^T \Sigma^{-1}\mu_e} $$

where $\mu_e$ is the vector of expected excess returns and $\Sigma$ is the variance-covariance matrix.

This is implemented as follows:

```python
# for convenience and readability, define the inverse of a dataframe
def inverse(d):
    """
    Invert the dataframe by inverting the underlying matrix
    """
    return pd.DataFrame(inv(d.values), index=d.columns, columns=d.index)

def w_msr(sigma, mu, scale=True):
    """
    Optimal (Tangent/Max Sharpe Ratio) Portfolio weights
    by using the Markowitz Optimization Procedure
    Mu is the vector of Excess expected Returns
    Sigma must be an N x N matrix as a DataFrame and Mu a column vector as a Series
    This implements page 188 Equation 5.2.28 of
    "The econometrics of financial markets" Campbell, Lo and Mackinlay.
    """
    w = inverse(sigma).dot(mu)
    if scale:
        w = w/sum(w) # fix: this assumes all w is +ve
    return w

```

Recall that the investor expects that Intel will return 2\% and Pfizer will return 4\% . We can now examine the optimal weights obtained by naively implementing the Markowitz procedure with these expected returns.

```python
mu_exp = pd.Series([.02, .04],index=tickers) # INTC and PFE
np.round(w_msr(s, mu_exp)*100, 2)

```

    INTC     3.41
    PFE     96.59
    dtype: float64

Consistent with the poor reputation of naive Markowitz optimization, the Markwitz procedure places an unrealistic weight of more than 96\% in Pfizer and less than 4\% in Intel. This is completely impractical and no reasonable investor would make such dramatic bets.

In contrast, let us now find the weights that the Black Litterman procedure would place. We allow $\Omega$ to be computed automatically, and are willing to use all the other defaults. We find the Black Litterman weights as follows:

```python
# Absolute view 1: INTC will return 2%
# Absolute view 2: PFE will return 4%
q = pd.Series({'INTC': 0.02, 'PFE': 0.04})

# The Pick Matrix
# For View 2, it is for PFE
p = pd.DataFrame([
# For View 1, this is for INTC
    {'INTC': 1, 'PFE': 0},
# For View 2, it is for PFE
    {'INTC': 0, 'PFE': 1}
    ])

# Find the Black Litterman Expected Returns
bl_mu, bl_sigma = bl(w_prior=pd.Series({'INTC':.44, 'PFE':.56}), sigma_prior=s, p=p, q=q)
# Black Litterman Implied Mu
bl_mu

```

    INTC    0.037622
    PFE     0.024111
    dtype: float64

The posterior returns returned by the procedure are clearly weighted between that of the equilibrium implied expected returns (in the range of 5\% and 1\%) and that of the investor (2\% and 4\%). The question is are these weights likely to yield more realistic portfolios? To answer that question we supply the Black Litterman expected returns and covariance matrix to the optimizer:

```python
# Use the Black Litterman expected returns to get the Optimal Markowitz weights
w_msr(bl_sigma, bl_mu)
```

    INTC    0.140692
    PFE     0.859308
    dtype: float64

We see that we get much more reasonable weights than we did with naive optimization. These weights are also much closer to the 45-55 mix in the cap weighted portfolio.
On the other hand, they respect the investor's view that expects Pfizer to rebound, and places a higher weight on Pfizer relative to the cap weighted portfolio.

### A Simple Example: Relative Views

In this example, we examine relative views. We stick with our simple 2-stock example. Recall that the Cap-Weighted implied expected returns are:

```python
# Expected returns inferred from the cap-weights
pi

```

    INTC    0.052084
    PFE     0.008628
    Name: Implied Returns, dtype: float64

Recall also that the cap-weighted portfolio is approximately a 45-55 mix of Intel and Pfizer.

Assume instead that the investor feels that the Intel will outperform Pfizer by only 2\%. This view is implemented as follows:

```python
q = pd.Series([
# Relative View 1: INTC will outperform PFE by 2%
  0.02
    ]
)
# The Pick Matrix
p = pd.DataFrame([
  # For View 1, this is for INTC outperforming PFE
  {'INTC': +1, 'PFE': -1}
])

# Find the Black Litterman Expected Returns
bl_mu, bl_sigma = bl(w_prior=pd.Series({'INTC': .44, 'PFE': .56}), sigma_prior=s, p=p, q=q)
# Black Litterman Implied Mu
bl_mu

```

    INTC    0.041374
    PFE     0.009646
    dtype: float64

Once again we see that the Black Litterman expected returns are a blend between the cap-weight implied weights and the investor view. The outperformance of Intel in the implied returns is:

```python
pi[0]-pi[1]
```

    0.043456

In contrast, the investor felt it only would be 2\%. The expected returns returned by the Black Litterman procedure show a spread that is a blend between the cap-weight implied returns and that of the investor:

```python
bl_mu[0]-bl_mu[1]
```

    0.031728

And, the weights in the Optimized portfolio when we use these expected returns are:

```python
# Use the Black Litterman expected returns and covariance matrix
w_msr(bl_sigma, bl_mu)
```

    INTC    0.347223
    PFE     0.652777
    dtype: float64

These seem like reasonable weights, and demonstrates the power of using the Black Litterman procedure. In contrast, consider the weights we would get if we implemented the same view without Black Litterman. We set the returns of Intel and Pfizer to be 3\% and 1\% respectively.

```python
w_msr(s, [.03, .01])
```

    INTC    0.258528
    PFE     0.741472
    dtype: float64

The weights are significantly more dramatic than one might be willing to implement, and are likely unwarranted given the relatively weak view. In fact, if the same view were implemented as Intel and Pfizer returning 2\% and 0\%, the results are even more extreme:

```python
w_msr(s, [.02, .0])
```

    INTC    1.248244
    PFE    -0.248244
    dtype: float64

In this case, the Markowitz recommends shorting Pfizer to the extent of nearly 25\% of the portfolio and leveraging Intel to 125\%. Clearly this is not a plausible allocation based on the simple view expressed above.

## Reproducing the He-Litterman (1999) Results

We now reproduce the results in the He-Litterman paper that first detailed the steps in the procedure. We obtained the data by typing it in from the He-Litterman tables, and used it to test the implementation.

The He-Litterman example involves an international allocation between 7 countries. The data is as follows:

```python
# The 7 countries ...
countries  = ['AU', 'CA', 'FR', 'DE', 'JP', 'UK', 'US']
# Table 1 of the He-Litterman paper
# Correlation Matrix
rho = pd.DataFrame([
    [1.000,0.488,0.478,0.515,0.439,0.512,0.491],
    [0.488,1.000,0.664,0.655,0.310,0.608,0.779],
    [0.478,0.664,1.000,0.861,0.355,0.783,0.668],
    [0.515,0.655,0.861,1.000,0.354,0.777,0.653],
    [0.439,0.310,0.355,0.354,1.000,0.405,0.306],
    [0.512,0.608,0.783,0.777,0.405,1.000,0.652],
    [0.491,0.779,0.668,0.653,0.306,0.652,1.000]
], index=countries, columns=countries)

# Table 2 of the He-Litterman paper: volatilities
vols = pd.DataFrame([0.160,0.203,0.248,0.271,0.210,0.200,0.187],index=countries, columns=["vol"])
# Table 2 of the He-Litterman paper: cap-weights
w_eq = pd.DataFrame([0.016,0.022,0.052,0.055,0.116,0.124,0.615], index=countries, columns=["CapWeight"])
# Compute the Covariance Matrix
sigma_prior = vols.dot(vols.T) * rho
# Compute Pi and compare:
pi = implied_returns(delta=2.5, sigma=sigma_prior, w=w_eq)
(pi*100).round(1)
```

    AU    3.9
    CA    6.9
    FR    8.4
    DE    9.0
    JP    4.3
    UK    6.8
    US    7.6
    Name: Implied Returns, dtype: float64

The values of $\pi$ computed by the Python code exactly matches column 3 of Table 2

### View 1: Germany vs Rest of Europe

Next, we impose the view that German equities will outperform the rest of European equities by 5\%.

The other European equities are France and the UK. We split the outperformance proportional to the Market Caps of France and the UK.

```python
# Germany will outperform other European Equities (i.e. FR and UK) by 5%
q = pd.Series([.05]) # just one view
# start with a single view, all zeros and overwrite the specific view
p = pd.DataFrame([0.]*len(countries), index=countries).T
# find the relative market caps of FR and UK to split the
# relative outperformance of DE ...
w_fr =  w_eq.loc["FR"]/(w_eq.loc["FR"]+w_eq.loc["UK"])
w_uk =  w_eq.loc["UK"]/(w_eq.loc["FR"]+w_eq.loc["UK"])
p.iloc[0]['DE'] = 1.
p.iloc[0]['FR'] = -w_fr
p.iloc[0]['UK'] = -w_uk
(p*100).round(1)

```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>AU</th>
      <th>CA</th>
      <th>FR</th>
      <th>DE</th>
      <th>JP</th>
      <th>UK</th>
      <th>US</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>-29.5</td>
      <td>100.0</td>
      <td>0.0</td>
      <td>-70.5</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
</div>

 The results of implementing this view appear in the He-Litterman paper in Table 4. This exactly reproduces column 1 of Table 4. Next, we examine the values of $\mu^{BL}$:

```python
delta = 2.5
tau = 0.05 # from Footnote 8
# Find the Black Litterman Expected Returns
bl_mu, bl_sigma = bl(w_eq, sigma_prior, p, q, tau = tau)
(bl_mu*100).round(1)

```

    AU     4.3
    CA     7.6
    FR     9.3
    DE    11.0
    JP     4.5
    UK     7.0
    US     8.1
    dtype: float64

The  Black Litterman expected returns computed by the code exactly reproduces column 2 of Table 4.

He-Litterman compute the optimal portfolio $w^*$ as follows (this is Equation (13) on page 6 of their paper)

```python
def w_star(delta, sigma, mu):
    return (inverse(sigma).dot(mu))/delta

wstar = w_star(delta=2.5, sigma=bl_sigma, mu=bl_mu)
# display w*
(wstar*100).round(1)

```

    AU     1.5
    CA     2.1
    FR    -4.0
    DE    35.4
    JP    11.0
    UK    -9.5
    US    58.6
    dtype: float64

The computed $w^*$ exactly replicates column 3 ($w^*$) of Table 4. Finally, they compute $w^* - \frac{w_{eq}}{1+\tau}$ which is the difference in weights between the optimal portfolio and the equilibrium portfolio (they use unscaled weights) in column 4. We replicate that column as follows:

```python
w_eq  = w_msr(delta*sigma_prior, pi, scale=False)
# Display the difference in Posterior and Prior weights
np.round(wstar - w_eq/(1+tau), 3)*100

```

    AU     0.0
    CA    -0.0
    FR    -8.9
    DE    30.2
    JP     0.0
    UK   -21.3
    US     0.0
    dtype: float64

which exactly matches Column 4 of Table 4. This completes our reproduction of the first view in He-Litterman (1999).

Note that this demonstrates the power of the approach. The weights for assets that do not involve the view remain unchanged. The two underperforming countries (according to the view) are underweighted, while the overperforming country is overweighted, but not to the extreme extent that a naive portfolio optimizer would have produced.

### View 2: Canada vs US

For their second case, He and Litterman implement the additional view that Canadian Equities will outperform US Equities by 3\%. The results are in (their) Table 5, which we shall now reproduce.

```python
view2 = pd.Series([.03], index=[1])
q = q.append(view2)
pick2 = pd.DataFrame([0.]*len(countries), index=countries, columns=[1]).T
p = p.append(pick2)
p.iloc[1]['CA']=+1
p.iloc[1]['US']=-1
np.round(p.T, 3)*100
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>AU</th>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>CA</th>
      <td>0.0</td>
      <td>100.0</td>
    </tr>
    <tr>
      <th>FR</th>
      <td>-29.5</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>DE</th>
      <td>100.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>JP</th>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>UK</th>
      <td>-70.5</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>US</th>
      <td>0.0</td>
      <td>-100.0</td>
    </tr>
  </tbody>
</table>
</div>

This matches columns 1 and 2 of Table 5. We now compute the Black Litterman weights as
before:

```python
bl_mu, bl_sigma = bl(w_eq, sigma_prior, p, q, tau = tau)
np.round(bl_mu*100, 1)
```

    AU     4.4
    CA     8.7
    FR     9.5
    DE    11.2
    JP     4.6
    UK     7.0
    US     7.5
    dtype: float64

The Black Litterman expected returns computed by the Python code exactly reproduces column 3 of
Table 5.
He-Litterman compute the optimal portfolio w ∗ as follows (this is Equation (13) on page 6 of
their paper)

```python
wstar = w_star(delta=2.5, sigma=bl_sigma, mu=bl_mu)
# display w*
(wstar*100).round(1)
```

    AU     1.5
    CA    41.9
    FR    -3.4
    DE    33.6
    JP    11.0
    UK    -8.2
    US    18.8
    dtype: float64

The computed $w^*$ exactly replicates column 4 ($w^*$) of Table 5. Finally, as in the previous case, they compute $w^* - \frac{w_{eq}}{1+\tau}$ in column 5. We replicate that column as follows:

```python
w_eq  = w_msr(delta*sigma_prior, pi, scale=False)
# Display the difference in Posterior and Prior weights
np.round(wstar - w_eq/(1+tau), 3)*100
```

    AU     0.0
    CA    39.8
    FR    -8.4
    DE    28.4
    JP    -0.0
    UK   -20.0
    US   -39.8
    dtype: float64

Which exactly reproduces the last column of Table 5 of their paper.

Once again, we see the power of the approach. The weights for assets that do not involve the view (AU, JP) remain unchanged. The two underperforming countries (FR, UK, US, according to the view) are underweighted, while the overperforming countries (CA, DE) are overweighted, but not to the extreme extent that a naive portfolio optimizer would have produced.

### View 3: More Bullish Canada vs US

For their third case, He and Litterman alter the second view that Canadian Equities will outperform US Equities by increasing the expected out-performance from the previously stated 3\% to 4\%. The results are in Table 6 of their paper, which we shall now reproduce.

```python
q[1] = .04
q
```

    0    0.05
    1    0.04
    dtype: float64

Note that P remains unchanged since we have only altered Q, not P

```python
np.round(p.T*100, 1)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>AU</th>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>CA</th>
      <td>0.0</td>
      <td>100.0</td>
    </tr>
    <tr>
      <th>FR</th>
      <td>-29.5</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>DE</th>
      <td>100.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>JP</th>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>UK</th>
      <td>-70.5</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>US</th>
      <td>0.0</td>
      <td>-100.0</td>
    </tr>
  </tbody>
</table>
</div>

This matches columns 1 and 2 of Table 6. We now compute the Black Litterman weights as before:

```python
bl_mu, bl_sigma = bl(w_eq, sigma_prior, p, q, tau = tau)
np.round(bl_mu, 3)*100
```

    AU     4.4
    CA     9.1
    FR     9.5
    DE    11.3
    JP     4.6
    UK     7.0
    US     7.3
    dtype: float64

The  Black Litterman expected returns computed by my code exactly reproduces column 3 of Table 6.

He-Litterman compute the optimal portfolio $w^*$ as follows (this is Equation (13) on page 6 of their paper)

```python
wstar = w_star(delta=2.5, sigma=bl_sigma, mu=bl_mu)
# display w*
(wstar*100).round(1)
```

    AU     1.5
    CA    53.3
    FR    -3.3
    DE    33.1
    JP    11.0
    UK    -7.8
    US     7.3
    dtype: float64

The computed $w^*$ exactly replicates column 4 ($w^*$) of Table 7. Finally, as in the previous case, they compute $w^* - \frac{w_{eq}}{1+\tau}$ in column 6. We replicate that column as follows:

```python
w_eq  = w_msr(delta*sigma_prior, pi, scale=False)
# Display the difference in Posterior and Prior weights
np.round(wstar - w_eq/(1+tau), 3)*100
```

    AU     0.0
    CA    51.3
    FR    -8.2
    DE    27.8
    JP    -0.0
    UK   -19.6
    US   -51.3
    dtype: float64

Which exactly reproduces the last column of Table 6 of their paper. Again, we see how the weights increase allocations consistent with the view, but keep allocations from getting extreme.

### View 4: Increasing View Uncertainty

As a final step, He and Litterman demonstrate the effect of $\Omega$. They increase the uncertainty associated with the first of the two views (i.e. the one that Germany will outperform the rest of Europe). First we compute the default value of $\Omega$ and then increase the uncertainty associated with the first view alone.

```python
# This is the default "Proportional to Prior" assumption
omega = proportional_prior(sigma_prior, tau, p)
# Now, double the uncertainty associated with View 1
omega.iloc[0,0] = 2*omega.iloc[0,0]
np.round(p.T*100, 1)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>AU</th>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>CA</th>
      <td>0.0</td>
      <td>100.0</td>
    </tr>
    <tr>
      <th>FR</th>
      <td>-29.5</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>DE</th>
      <td>100.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>JP</th>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>UK</th>
      <td>-70.5</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>US</th>
      <td>0.0</td>
      <td>-100.0</td>
    </tr>
  </tbody>
</table>
</div>

This matches columns 1 and 2 of Table 7 (which is if course, unchanged, since we have only altered $\Omega$, not Q or P). We now compute the Black Litterman weights as before, but supplying the value of $\Omega$ we just adjusted:

```python
bl_mu, bl_sigma = bl(w_eq, sigma_prior, p, q, tau = tau, omega=omega)
np.round(bl_mu, 3)*100
```

    AU     4.3
    CA     8.9
    FR     9.3
    DE    10.6
    JP     4.6
    UK     6.9
    US     7.2
    dtype: float64

The  Black Litterman expected returns computed by the code exactly reproduces column 3 of Table 7.

He-Litterman compute the optimal portfolio $w^*$ as follows (this is Equation (13) on page 6 of their paper)

```python
wstar = w_star(delta=2.5, sigma=bl_sigma, mu=bl_mu)
# display w*
(wstar*100).round(1)
```

    AU     1.5
    CA    53.9
    FR    -0.5
    DE    23.6
    JP    11.0
    UK    -1.1
    US     6.8
    dtype: float64

The computed $w^*$ exactly replicates column 4 ($w^*$) of Table 7. Finally, as in the previous case, they compute $w^* - \frac{w_{eq}}{1+\tau}$ in column 6. We replicate that column as follows:

```python
w_eq  = w_msr(delta*sigma_prior, pi, scale=False)
# Display the difference in Posterior and Prior weights
np.round(wstar - w_eq/(1+tau), 3)*100
```

    AU    -0.0
    CA    51.8
    FR    -5.4
    DE    18.4
    JP     0.0
    UK   -13.0
    US   -51.8
    dtype: float64

Which exactly reproduces the last column of Table 7 of their paper. Again, we see how the weights increase allocations consistent with the view, but keep allocations from getting extreme.

That concludes our reproduction of the paper. Note that He and Litterman also produce an extra table (Table 8) which demonstrates the value of adding a third view. However, the third view is identical to the values implied by the equilibrium and as a result, they produce exactly the same results as Table 7. I do not bother reproduce it here since the results are exactly the same as Table 7.

## Try it on Industry Data ...

Now that you've reproduced the results, you should be able to run the code on the Fama-French Industry Portfolios ...

Start out by loading the data as follows, and then play around!

```python
import edhec_risk_kit_206 as erk

ind49_rets = erk.get_ind_returns(weighting="vw", n_inds=49)["2014":]
ind49_mcap = erk.get_ind_market_caps(49, weights=True)["2014":]
```


# Universidade Federal do Amazonas
# Departamento de Engenharia de Produção

## - Projeto Super 2021 - Curso Python 2.0

### Aula 4 - Biblioteca Numpy

Biblioteca Numpy - numerial python

** Matemática computacional e ciência de dados

** Numpy - basicamente é uma estrutura de dados - array[ ]

    - estrutura de array;
    - manipular array;
    - criação de array;
    - aplicar métodos matemáticos em array.
    - Álgebra linear.

```python
# SOLVER - equações
# sp.solve() ou sp.solveset()

# Biblioteca sympy
import sympy as sp
sp.init_printing()
# f = x**3 + 5*x**2  +x
x = sp.symbols('x')
f_x = x**3 + 5*x**2 + x
f_x
```

```python

```

```python
f_x
```

```python
# Resolvendo a Equação f_x - método sp.solve(f,x)
sp.solve(f_x,x)
```

```python
# x = -4.791287
```

```python
# funcao.subs(x,x1)
f_x
x1 = -4.791287
f_x.subs(x,x1)
```

```python
# RESOLVENDO SISTEMAS LINEARES COM O SOLVE
A = sp.Matrix([[3,2,4],[1,1,2],[4,3,-2]])
A
x1,x2,x3 = sp.symbols(['x1','x2','x3'])
x = sp.Matrix([[x1,x2,x3]]).T
x
R = sp.Matrix([[11,22,33]]).T
R
# A * x = R
# A * x - R = 0
sistema = A * x - R
sistema
```

$\displaystyle \left[\begin{matrix}3 x_{1} + 2 x_{2} + 4 x_{3} - 11\\x_{1} + x_{2} + 2 x_{3} - 22\\4 x_{1} + 3 x_{2} - 2 x_{3} - 33\end{matrix}\right]$

```python
sp.solve(sistema,(x1,x2,x3))
```

```python
# RESOLVENDO SISTEMAS LINEARES COM O SOLVE
A = sp.Matrix([[3,2,4],[1,1,2],[4,3,-2]])
A
x1,x2,x3 = sp.symbols(['x1','x2','x3'])
x = sp.Matrix([[x1,x2,x3]])
x
```

$\displaystyle \left[\begin{matrix}x_{1} & x_{2} & x_{3}\end{matrix}\right]$

```python
from sympy.interactive import printing
printing.init_printing(use_latex=True)
from sympy import *
import sympy as sp
# criar um simbolo para uma variavel
x = sp.symbols('x')
f = sp.Function('f')(x)
f
sp.diff(f,x)
```

```python
# criar uma EDO 2º ordem
diffeq = Eq(f.diff(x,x)-5*f,0)
display(diffeq)
```

```python
# Resolver EDO 2ª ordem usando método sp.dsolve(f,x)
sp.dsolve(diffeq,f)
```

```python
# Atividade - resolver o exemplo anterior so condirar EDO de 1ª ordem.

```

```python

```

# O que é o Numpy ?

- biblioteca python - criação de estruturas de array

- ARRAY ?
    ** é uma estrutura de dados multidimensional

    Ex:
    vetor (unidimensional - uma linha ou uma coluna)
    matriz (duas dimensõess - linhas x colunas)

# **Porque usar Numpy ?**
- manipular grandes volumes de dados (Ex:dados da produção industrial);
#### - Pesquisa acadêmica (matemática numérica - cálculo numérico).

```python
# Importar biblioteca
import numpy as np
a_1 = np.array([1,2,3])
a_1
```

    array([1, 2, 3])

```python
type(a_1)
```

    numpy.ndarray

```python
# forma do array
a_1.shape
```

```python
a_1 = np.array([[1,1,1]])
a_1
```

    array([[1, 1, 1]])

```python
a_1.shape
```

```python
# saber a dimensao
a_1
a_1.ndim
```

```python
# criar um array 2x3
a_2 = np.array([[1,2,3],[3,4,5]])
a_2
```

    array([[1, 2, 3],
           [3, 4, 5]])

```python
a_2.shape
```

```python
# vamos ver o tipo da variavel
a_2
a_2.dtype
```

    dtype('int64')

```python
# Algumas funcionalidade do Numpy - matemática
import numpy as np
a = np.cos(0)
a
b = np.sin(0)
b
```

```python
# funçao linspace
#                 (inicio,fim,qte)
a_3 = np.linspace(0,20,20)
a_3
```

    array([ 0.        ,  1.05263158,  2.10526316,  3.15789474,  4.21052632,
            5.26315789,  6.31578947,  7.36842105,  8.42105263,  9.47368421,
           10.52631579, 11.57894737, 12.63157895, 13.68421053, 14.73684211,
           15.78947368, 16.84210526, 17.89473684, 18.94736842, 20.        ])

```python

```


Machine Learning with Python

Laboratory on Numpy / Matplotlib / Scikit-learn

***

***

## Introduction

In the past few years, Python has become the de-facto standard programming language for data analytics. Python's success is due to several factors, but one major reason has been the availability of powerful, open-source libraries for scientific computation such as Numpy, Scipy and Matplotlib. Python is also the most popular programming language for machine learning, thanks to libraries such as Scikit-learn and TensorFlow.

In this lecture we will explore the basics of Numpy, Matplotlib and Scikit-learn. The first is a library for data manipulation through the powerfull `numpy.ndarray` data structure; the second is useful for graphical visualization and plotting; the third is a general purpose library for machine learning, containing dozens of algorithms for classification, regression and clustering.

In this lecture we assume familiarity with the Python programming language. If you are not familiar with the language, we advise you to look it up before carrying over to the next sections. Here are some useful links to learn about Python:
- https://docs.python.org/3/tutorial/introduction.html
- https://www.learnpython.org/
- http://www.scipy-lectures.org/

If you have never seen a page like this, it is a **Jupyther Notebook**. Here one can easily embed Python code and run it on the fly. You can run the code in a cell by selecting the cell and clicking the *Run* button (top). You can do the same using the **SHIFT+Enter** shortcut. You can modify the existing cells, run them and finally save your changes.

## Requirements

1. Python (preferably version > 3.3): https://www.python.org/downloads/
2. Numpy, Scipy and Matplotlib: https://www.scipy.org/install.html
3. Scikit-learn: http://scikit-learn.org/stable/install.html

## References

- https://docs.scipy.org/doc/numpy/
- https://docs.scipy.org/doc/scipy/reference/
- https://matplotlib.org/users/index.html
- http://scikit-learn.org/stable/documentation.html

# Numpy

Numpy provides high-performance data structures for data manipulation and numeric computation. In particular, we will look at the `numpy.ndarray`, a data structure for manipulating vectors, matrices and tensors. Let's start by importing `numpy`:

```python
# the np alias is very common
import numpy as np
```

We can initialize a Numpy array from a Python list using the `numpy.array` function:

```python
# if the argument is a list of numbers, the array will be a 1-dimensional vector
a = np.array([1, 2, 3, 4, 5, 6])

a
```

    array([1, 2, 3, 4, 5, 6])

```python
# if the argument is a list of lists, the array will be a 2-dimensional matrix
M = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]])

M
```

    array([[ 1,  2,  3,  4],
           [ 5,  6,  7,  8],
           [ 9, 10, 11, 12],
           [13, 14, 15, 16]])

Given a Numpy array, we can check its `shape`, a tuple containing the number of elements for each dimension:

```python
a.shape
```

    (6,)

```python
M.shape
```

    (4, 4)

The size of an array is its total number of elements:

```python
a.size
```

    6

```python
M.size
```

    16

We can do quite some nice things with Numpy arrays that are not possible with standard Python lists.

### Indexing
Numpy array allow us to index arrays in quite advanced ways.

```python
# A 1d vector can be indexed in all the common ways
a[0]
```

    1

```python
a[1:3]
```

    array([2, 3])

```python
a[0:5:2]
```

    array([1, 3, 5])

```python
# Use a boolean mask
mask = [True, False, False, True, True, False]
a[mask]
```

    array([1, 4, 5])

```python
# Access specific elements by passing a list of index
a[[1, 4, 5]]
```

    array([2, 5, 6])

The power of Numpy indexing capabilities starts showing up with 2d arrays:

```python
# Access a single element of the matrix
M[0, 1]
```

    2

```python
# Access an entire row
M[1]
```

    array([5, 6, 7, 8])

```python
# Access an entire column
M[:,2]
```

    array([ 3,  7, 11, 15])

```python
# Extract a sub-matrix
M[1:3, 0:2]
```

    array([[ 5,  6],
           [ 9, 10]])

### Data manipulation
We can manipulate data in several ways.

```python
# Flatten a matrix
M.flatten()
```

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16])

```python
# Reshaping a matrix
M.reshape(2, 8)
```

    array([[ 1,  2,  3,  4,  5,  6,  7,  8],
           [ 9, 10, 11, 12, 13, 14, 15, 16]])

```python
# The last index can be automatically inferred using -1
M.reshape(2, -1)
```

    array([[ 1,  2,  3,  4,  5,  6,  7,  8],
           [ 9, 10, 11, 12, 13, 14, 15, 16]])

```python
# Computing the max and the min
M.max(), M.min()
```

    (16, 1)

```python
# Computing the mean and standard deviation
M.mean(), M.std()
```

    (8.5, 4.6097722286464435)

```python
# Computing the sum along the rows
M.sum(axis=1)
```

    array([10, 26, 42, 58])

### Linear algebra
Numpy is very useful to all sort of numeric computation, especially linear algebra:

```python
# Transpose
M.T
```

    array([[ 1,  5,  9, 13],
           [ 2,  6, 10, 14],
           [ 3,  7, 11, 15],
           [ 4,  8, 12, 16]])

```python
# Adding and multiplying a constant
10 * M + 5
```

    array([[ 15,  25,  35,  45],
           [ 55,  65,  75,  85],
           [ 95, 105, 115, 125],
           [135, 145, 155, 165]])

```python
# Element wise product
b = np.array([-1, -2, 4, 6, 8, -4])

a * b
```

    array([ -1,  -4,  12,  24,  40, -24])

```python
# Dot product
a.dot(b)
```

    47

```python
# More linear algebra in the package numpy.linalg
# Determinant
np.linalg.det(M)
```

    0.0

```python
# Eigenvalues
np.linalg.eigvals(M)
```

    array([ 3.62093727e+01, -2.20937271e+00, -3.36399340e-15, -2.27991821e-16])

### Vector generation and sampling
Numpy allows us to generate or randomly sample vectors:

```python
# Generate an array with 0.5 spacing
x = np.arange(-10, 10, 0.5)

x
```

    array([-10. ,  -9.5,  -9. ,  -8.5,  -8. ,  -7.5,  -7. ,  -6.5,  -6. ,
            -5.5,  -5. ,  -4.5,  -4. ,  -3.5,  -3. ,  -2.5,  -2. ,  -1.5,
            -1. ,  -0.5,   0. ,   0.5,   1. ,   1.5,   2. ,   2.5,   3. ,
             3.5,   4. ,   4.5,   5. ,   5.5,   6. ,   6.5,   7. ,   7.5,
             8. ,   8.5,   9. ,   9.5])

```python
# Generate an array with 20 equally spaced points
x = np.linspace(-10, 10, 20)

x
```

    array([-10.        ,  -8.94736842,  -7.89473684,  -6.84210526,
            -5.78947368,  -4.73684211,  -3.68421053,  -2.63157895,
            -1.57894737,  -0.52631579,   0.52631579,   1.57894737,
             2.63157895,   3.68421053,   4.73684211,   5.78947368,
             6.84210526,   7.89473684,   8.94736842,  10.        ])

```python
# Sample a vector from a standardize normal distribution
np.random.normal(size=(10,))
```

    array([ 1.44201973, -0.11330043,  0.08256323, -1.15813372,  0.15709646,
           -0.06512693,  0.90353507,  0.97059199,  0.57933141,  0.17082713])

### Functions
Numpy provides all sorts of mathematical functions we can apply to arrays

```python
# Exponential function
np.exp(x)
```

    array([4.53999298e-05, 1.30079023e-04, 3.72699966e-04, 1.06785292e-03,
           3.05959206e-03, 8.76628553e-03, 2.51169961e-02, 7.19647439e-02,
           2.06192028e-01, 5.90777514e-01, 1.69268460e+00, 4.84984802e+00,
           1.38956932e+01, 3.98136782e+01, 1.14073401e+02, 3.26840958e+02,
           9.36458553e+02, 2.68312340e+03, 7.68763460e+03, 2.20264658e+04])

```python
# Sine
np.sin(x)
```

    array([ 0.54402111, -0.4594799 , -0.99916962, -0.53027082,  0.47389753,
            0.99970104,  0.5163796 , -0.48818921, -0.99996678, -0.50235115,
            0.50235115,  0.99996678,  0.48818921, -0.5163796 , -0.99970104,
           -0.47389753,  0.53027082,  0.99916962,  0.4594799 , -0.54402111])

```python
# A gaussian function
y = np.exp(-(x ** 2)/2)

y
```

    array([1.92874985e-22, 4.13228632e-18, 2.92342653e-14, 6.82937941e-11,
           5.26814324e-08, 1.34190319e-05, 1.12868324e-03, 3.13480292e-02,
           2.87498569e-01, 8.70659634e-01, 8.70659634e-01, 2.87498569e-01,
           3.13480292e-02, 1.12868324e-03, 1.34190319e-05, 5.26814324e-08,
           6.82937941e-11, 2.92342653e-14, 4.13228632e-18, 1.92874985e-22])

# Matplotlib
The above matrices provide little insight without the possibility of visualizing them properly. Matplotlib is a powerful library for data visualization. Let's plot the above function.

```python
# the following line is only needed to show plots in the notebook
%matplotlib inline
import matplotlib.pyplot as plt

x = np.linspace(-10, 10, 200)    # get a sample of the x axis
y = np.exp(-(x**2)/(2*1))        # compute the function for all points in the sample
plt.plot(x, y)                   # add the curve to the plot
plt.ylim(-0.05,1.05)             # set bottom and top limits for y axis
plt.show()                       # show the plot
```

We can also plot more than one line in the same figure and add a grid to the plot.

```python
z = np.exp(-(x**2)/(2*10))
plt.grid()                 # add the grid under the curves
plt.plot(x, y)             # add the first curve to the plot
plt.plot(x, z)             # add the second curve to the plot
plt.ylim(-0.05,1.05)             # set bottom and top limits for y axis
plt.show()                 # show the plot
```

We can also set several properties of the plot in this way:

```python
plt.grid()
plt.xlabel('x')                     # add a label to the x axis
plt.ylabel('y')                     # add a label to the y axis

plt.xticks(np.arange(-10, 11, 2))   # specify in which point to place a tick on the x axis
plt.yticks(np.arange(0, 2.2, 0.2))  # and on the y axis

# rs- stands for red, squared markers, solid line
# yd-- stands for yellow, diamond markers, dashed line
plt.plot(x, y, 'rs-', markevery=10, label='sigma=1')    # add a style and a label and specify the gap
plt.plot(x, z, 'yd--', markevery=10, label='sigma=10')  # between markers for both curves

plt.ylim(-0.05,1.05)             # set bottom and top limits for y axis
plt.legend()  # add the legend (displays the labels of the curves)
plt.show()    # show the plot
```

Finally, we can save the plot into a png file in this way:

```python
plt.grid()
plt.xlabel('x')
plt.ylabel('y')
plt.xticks(np.arange(-10, 11, 2))
plt.yticks(np.arange(0, 2.2, 0.2))
plt.plot(x, y, 'rs-', markevery=10, label='sigma=1')
plt.plot(x, z, 'yd--', markevery=10, label='sigma=10')
plt.ylim(-0.05,1.05)             # set bottom and top limits for y axis
plt.legend()
plt.savefig('plot.png', dpi=300)  # saves the plot into the file plot.png with 300 dpi
```

# Scikit-learn
Let's now dive into the real **Machine Learning** part. *Scikit-learn* is perhaps the most wide-spread library for Machine Learning in use nowadays, and most of its fame is due to its extreme simplicity. With Scikit-learn it is possible to easily manage datasets, and train a wide range of classifiers out-of-the-box. It is also useful for several other Machine Learning tasks such as regression, clustering, dimensionality reduction, and model selection.

In the following we will see how to use Scikit-learn to load a dataset, train a classifier and perform validation and model selection.

Scikit-learn comes with a range of popular reference datasets. Let's load and use the *Digits* dataset:

```python
from sklearn.datasets import load_digits

digits = load_digits()

print(digits.DESCR)     # print a description of the digits dataset
```

    .. _digits_dataset:

    Optical recognition of handwritten digits dataset
    --------------------------------------------------

    **Data Set Characteristics:**

        :Number of Instances: 5620
        :Number of Attributes: 64
        :Attribute Information: 8x8 image of integer pixels in the range 0..16.
        :Missing Attribute Values: None
        :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)
        :Date: July; 1998

    This is a copy of the test set of the UCI ML hand-written digits datasets
    http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits

    The data set contains images of hand-written digits: 10 classes where
    each class refers to a digit.

    Preprocessing programs made available by NIST were used to extract
    normalized bitmaps of handwritten digits from a preprinted form. From a
    total of 43 people, 30 contributed to the training set and different 13
    to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of
    4x4 and the number of on pixels are counted in each block. This generates
    an input matrix of 8x8 where each element is an integer in the range
    0..16. This reduces dimensionality and gives invariance to small
    distortions.

    For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.
    T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.
    L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,
    1994.

    .. topic:: References

      - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their
        Applications to Handwritten Digit Recognition, MSc Thesis, Institute of
        Graduate Studies in Science and Engineering, Bogazici University.
      - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.
      - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.
        Linear dimensionalityreduction using relevance weighted LDA. School of
        Electrical and Electronic Engineering Nanyang Technological University.
        2005.
      - Claudio Gentile. A New Approximate Maximal Margin Classification
        Algorithm. NIPS. 2000.

Let's take a look at the data:

```python
X, y = digits.data, digits.target

# The attributes of the first instance (notice it is a Numpy array)
X[0]
```

    array([ 0.,  0.,  5., 13.,  9.,  1.,  0.,  0.,  0.,  0., 13., 15., 10.,
           15.,  5.,  0.,  0.,  3., 15.,  2.,  0., 11.,  8.,  0.,  0.,  4.,
           12.,  0.,  0.,  8.,  8.,  0.,  0.,  5.,  8.,  0.,  0.,  9.,  8.,
            0.,  0.,  4., 11.,  0.,  1., 12.,  7.,  0.,  0.,  2., 14.,  5.,
           10., 12.,  0.,  0.,  0.,  0.,  6., 13., 10.,  0.,  0.,  0.])

```python
# The label of the first instance
y[0]
```

    0

Being a Numpy array, we can actually take a look at this image. We first need to reshape it into an 8x8 matrix and then use matplotlib.

```python
x = X[0].reshape((8, 8))

plt.gray()      # use a grayscale
plt.matshow(x)  # display a matrix of values
plt.show()      # show the figure
```

Now we want to train a classifier to recognize the digits from the images and then we want to evaluate it. In order to make a proper evaluation, we first need to split the dataset in two sets, one for training and one for testing. Scikit-learn helps us with that:

```python
import sklearn
# the lab PCs have an outdated version of scikit-learn, with different names for packages
try:
    from sklearn.model_selection import train_test_split
except ImportError:
    from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Let's check the length of the two sets
len(X_train), len(X_test)
```

    (1437, 360)

Now we need a classifier. Let's use an **SVM**. A reminder:

\begin{align}
   \min_{\boldsymbol{w}} \quad & \frac{1}{2}\|\boldsymbol{w}\|^2 + C \sum_{i\in|\mathcal{D}|} \xi_i \\
   \forall (x_i, y_i) \in \mathcal{D} \quad & y_i ( \boldsymbol{w}^T x_i + b ) \ge 1 - \xi_i
\end{align}

```python
from sklearn.svm import SVC

# Specify the parameters in the constructor.
# C is the parameter of the primal problem of the SVM;
# The rbf kernel is the Gaussian kernel;
# The rbf kernel takes one parameter: gamma (gaussian width)
clf = SVC(C=10, kernel='rbf', gamma=0.02)
```

Now the classifier can be trained and then used to predict unseen instances.

```python
# Training
clf.fit(X_train, y_train)

# Prediction
y_pred = clf.predict(X_test)

y_pred
```

    array([3, 3, 3, 7, 3, 1, 3, 3, 3, 3, 3, 3, 3, 0, 3, 3, 3, 3, 8, 3, 3, 3,
           3, 3, 3, 3, 3, 3, 6, 3, 3, 9, 1, 3, 3, 6, 3, 4, 3, 6, 6, 3, 1, 3,
           3, 3, 3, 3, 6, 3, 3, 3, 3, 3, 3, 0, 3, 3, 0, 1, 3, 3, 3, 3, 3, 5,
           3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 0, 3, 4,
           3, 3, 3, 3, 8, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 3, 3, 3, 3, 3,
           7, 7, 3, 3, 3, 3, 3, 3, 3, 7, 2, 6, 3, 3, 3, 3, 3, 7, 3, 3, 3, 3,
           3, 3, 3, 6, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 6, 3, 3, 3, 3, 3, 6, 3,
           3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 4, 3, 1, 3,
           7, 3, 2, 2, 3, 3, 8, 3, 3, 2, 3, 3, 6, 9, 3, 3, 1, 3, 3, 2, 3, 3,
           3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 1, 3, 3, 3, 3, 6, 1, 3, 6, 0, 4, 3,
           2, 7, 3, 6, 3, 3, 3, 3, 3, 2, 3, 6, 3, 1, 3, 1, 3, 3, 3, 3, 3, 3,
           3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 3, 3, 3, 0, 3, 3,
           3, 0, 1, 3, 4, 3, 1, 3, 3, 6, 0, 3, 3, 0, 3, 3, 3, 3, 3, 3, 3, 3,
           3, 9, 3, 3, 3, 3, 3, 3, 0, 3, 3, 3, 3, 0, 3, 3, 6, 3, 3, 3, 3, 3,
           3, 2, 1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 3, 3, 7, 0, 6, 3, 3, 3,
           3, 1, 3, 4, 3, 3, 7, 3, 3, 3, 3, 3, 3, 0, 7, 3, 3, 3, 3, 2, 7, 3,
           1, 3, 7, 3, 3, 3, 3, 3])

Now we want to evaluate the performance of our classifier. A reminder:

\begin{align}
    \text{Accuracy } &= \frac{\text{true-positive} + \text{true-negative}}{\text{all examples}} \\
    \text{Precision } &= \frac{\text{true-positive}}{\text{true-positive} + \text{false-positive}} \\
    \text{Recall } &= \frac{\text{true-positive}}{\text{true-positive} + \text{false-negative}} \\
    F_1 &= \frac{2 \times \text{precision} \times \text{recall}}{\text{precision} + \text{recall}} \\
\end{align}

In a multiclass classification Precision, Recall and $F_1$ are computed per class, considering the given class as positive and all others as negative.

We can use Scikit-learn to compute and show these measures for all classes.

```python
from sklearn import metrics

report = metrics.classification_report(y_test, y_pred)

# the support is the number of instances having the given label in y_test
print(report)
```

                  precision    recall  f1-score   support

               0       1.00      0.39      0.57        33
               1       1.00      0.61      0.76        28
               2       1.00      0.36      0.53        33
               3       0.12      1.00      0.22        34
               4       1.00      0.20      0.33        46
               5       1.00      0.02      0.04        47
               6       1.00      0.49      0.65        35
               7       1.00      0.35      0.52        34
               8       1.00      0.10      0.18        30
               9       1.00      0.07      0.14        40

       micro avg       0.34      0.34      0.34       360
       macro avg       0.91      0.36      0.39       360
    weighted avg       0.92      0.34      0.37       360

Finally we can compute the accuracy of our classifier:

```python
metrics.accuracy_score(y_test, y_pred)
```

    0.33611111111111114

Apparently our classifier performs a bit poorly out-of-sample. This is probably due to the random choice of the parameters for the classifier. We can do much better! We need to perform model selection, that is we need to search for better parameters for our classifier.

In particular, we are going to perform a **cross-validation** on the training set and see how the classifier performs with different values of *gamma*.

A $k$-fold cross-validation works like this:
- Split the dataset $D$ in $k$ equally sized disjoint subsets $D_i$
- For $i \in [1, k]$
 - Train the classifier on $T_i = D \setminus D_i$
 - Compute the score (accuracy, precision, ...) on $D_i$
- Return the list of scores, one for each fold

Scikit-learn helps us with this as well. We compute the cross-validated accuracy for all the possible values of *gamma* and select the *gamma* with the best average accuracy.

```python
# Again, we have to add some code to have this run on the lab PCs
try:
    from sklearn.model_selection import KFold, cross_val_score
    legacy = False
except ImportError:
    from sklearn.cross_validation import KFold, cross_val_score
    legacy = True

# 3-fold cross-validation
# random_state ensures same split for each value of gamma
# KFold has a different syntax for legacy versions of scikit-learn
if legacy:
    kf = KFold(len(y_train),n_folds=3, shuffle=True, random_state=42)
else:
    kf = KFold(n_splits=3, shuffle=True, random_state=42)

gamma_values = [0.1, 0.05, 0.02, 0.01]
accuracy_scores = []

# Do model selection over all the possible values of gamma
for gamma in gamma_values:

    # Train a classifier with current gamma
    clf = SVC(C=10, kernel='rbf', gamma=gamma)

    # Compute cross-validated accuracy scores
    # So legacy....
    if legacy:
        scores = cross_val_score(clf, X_train, y_train, cv=kf, scoring='accuracy')
    else:
         scores = cross_val_score(clf, X_train, y_train, cv=kf.split(X_train), scoring='accuracy')

    # Compute the mean accuracy and keep track of it
    accuracy_score = scores.mean()
    accuracy_scores.append(accuracy_score)

# Get the gamma with highest mean accuracy
best_index = np.array(accuracy_scores).argmax()
best_gamma = gamma_values[best_index]

# Train over the full training set with the best gamma
clf = SVC(C=10, kernel='rbf', gamma=best_gamma)
clf.fit(X_train, y_train)

# Evaluate on the test set
y_pred = clf.predict(X_test)
accuracy = metrics.accuracy_score(y_test, y_pred)

accuracy
```

    0.8138888888888889

Much better! Model selection allows us to fine-tune the parameters of a lerning algorithm to get the best performance.

Let's now look at the **Learnig curve** of our classifier, in which we plot the training accuracy and the cross-validated accuracy for increasing number of examples.

```python
try:
    from sklearn.model_selection import learning_curve
except ImportError:
    from sklearn.learning_curve import learning_curve

plt.figure()
plt.title("Learning curve")
plt.xlabel("Training examples")
plt.ylabel("Score")
plt.grid()

clf = SVC(C=10, kernel='rbf', gamma=best_gamma)

# Compute the scores of the learning curve
# by default the (relative) dataset sizes are: 10%, 32.5%, 55%, 77.5%, 100%
# The function automatuically executes a Kfold cross validation for each dataset size
train_sizes, train_scores, val_scores = learning_curve(clf, X_train, y_train, scoring='accuracy', cv=3)

# Get the mean and std of train and validation scores over the cv folds along the varying dataset sizes
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
val_scores_mean = np.mean(val_scores, axis=1)
val_scores_std = np.std(val_scores, axis=1)

# Plot the mean  for the training scores
plt.plot(train_sizes, train_scores_mean, 'o-', color="r", label="Training score")

# Plot the  std for the training scores
plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                 train_scores_mean + train_scores_std, alpha=0.1, color="r")

# Plot the mean  for the validation scores
plt.plot(train_sizes, val_scores_mean, 'o-', color="g", label="Cross-validation score")

# Plot the std for the validation scores
plt.fill_between(train_sizes, val_scores_mean - val_scores_std,
                 val_scores_mean + val_scores_std, alpha=0.1, color="g")
plt.ylim(0.05,1.3)             # set bottom and top limits for y axis
plt.legend()
plt.show()
```

Now we want to go even further. We can perform the above model selection procedure considering the *C* parameter as well. In general, this process over several parameters is called **grid search**, and Scikit-learn has an automated procedure to perform cross-validated grid search for any classifier.

```python
try:
    from sklearn.model_selection import GridSearchCV
except ImportError:
    from sklearn.grid_search import GridSearchCV
possible_parameters = {
    'C': [1e0, 1e1, 1e2, 1e3],
    'gamma': [1e-1, 1e-2, 1e-3, 1e-4]
}

svc = SVC(kernel='rbf')

# The GridSearchCV is itself a classifier
# we fit the GridSearchCV with the training data
# and then we use it to predict on the test set
clf = GridSearchCV(svc, possible_parameters, n_jobs=4, cv=3) # n_jobs=4 means we parallelize the search over 4 threads
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
accuracy = metrics.accuracy_score(y_test, y_pred)

accuracy
```

    0.9888888888888889

Nice! Now we have a classifier with a quite competitive accuracy. The state-of-the-art (on a very similar task) has accuracy around $0.9979$, achieved by using Neural Networks, which we will see in the next Lab. Stay tuned!