Supervised Learning¶

Outline¶

Introduction

Regression methods

Linear Regression

Penalized Regressions: Lasso, Ridge and Elastic Net

Polynomial regression and kernel methods

Classification methods

Support Vector Machines

Bayesian Classification

K Nearest Neighbors

Logistic regression

Machine learning¶

Nearly all machine-learning problem can be posed as optimisation problems applied to data.

We define a loss function: $L: (\mathbf{x}, F_\mathbf{w}) \rightarrow \mathbb{R}$

where $\mathbf{x}$ is some data, and $F_\mathbf{w}(\mathbf{x})$ is an arbitrary function parameterised by numeric weights $\mathbf{w}$.

$F$ can be non-linear and/or stochastic.

We minimise the expected loss by choosing appropriate weights:

\begin{equation}
 \underset{\mathbf{w}}{\operatorname{argmin}} E[L(\mathbf{x}, F_\mathbf{w})]
\end{equation}

In statistical learning we sometimes use the word "coefficient" instead of "weight", but the meaning is identical.

Supervised versus unsupervised learning¶

In a supervised learning problem we have some additional training data $\mathbf{y}$.

The training data provides a finite subset of observed values $\mathbf{y}$ from an unknown function $F$, such that $\mathbf{y} = F(\mathbf{x}) + \epsilon$

where $\epsilon$ are iid. random noise.

We want to generalise from the training data to derive the complete mapping $F$.

This can then used to generalise to unseen cases where no training data is available.

In a supervised learning problem therefore, our loss function takes the form $L: (\mathbf{x}, \mathbf{y}, \hat{F}_\mathbf{w}) \rightarrow \mathbb{R}$.

Intuitively, we should define a loss function that tells us how good the fit is between the function $\hat{F}$ and the observations of $F$.

Statistical and machine-learning¶

Standard methods in statistics can also be viewed as optimisation problems.

For example, in ordinary least-squares estimation (OLS) the loss function is simply the sum of the squared residuals (RSS):

\begin{eqnarray}
L(\mathbf{x}, \mathbf{y}, \mathbf{w}) & = & \sum_{i=1}^n r_i^2 \\
r_i & = & y_i - F_\mathbf{w}(x_i) \\
F_\mathbf{w}(\mathbf{x}) & = & \mathbf{x}^T \mathbf{w}
\end{eqnarray}

In the above, the loss function is a simple linear function.

The OLS loss function is very straightforward to minimize using calculus.

In general, however, machine-learning methods allow us to work with arbitrary non-linear functions.

Practical machine learning¶

Machine learning has recently become very practical, because:

we now have very large data sets,

we have very good algorithms for performing automated differentiation of arbitrary functions,

we have heuristic optimization algorithms for minimizing arbitrary functions even when they are not well behaved,

Practical machine learning: GPUs¶

Graphical-Processing Units GPUs can perform parallel computation on multi-dimensional numerical data using thousands or tens of thousands of cores simultaneously.

Python for machine-learning¶

There are many powerful tools for machine-learning which are available in the Python programming language:

scikit-learn

TensorFlow

scipy

Optimizing a simple function in Python¶

```python
import numpy as np
import matplotlib.pylab as plt

def F(x):
    return x**2 - 2*x

domain = np.linspace(-19., +21.0)
plt.plot(domain, F(domain)); plt.show()
```

Using gradient descent¶

```python
import scipy.optimize as opt

initial_guess = [20.]
solution = opt.minimize(F, initial_guess)
solution.x
```

    array([1.00000009])

Generating some training data¶

Now suppose we don't simply want to the minimum of $F$, instead we want to find $F$ itself, given some finite noisy observations $\mathbf{y}$.

```python
data = np.array([-15, -12, -5, 1., 3., -10, +14, +15])
training_data = np.array([F(x) + np.random.normal(scale=30.) for x in data])
plt.plot(domain, F(domain))
plt.plot(data, training_data, 'x')
plt.show()
```

Optimising a simple loss function in Python¶

Let's assume that $F$ is a polynomial of order 2.

```python
def F_hat(x, w):
    (a, b, c) = w
    return a + b*x + c*x**2
```

Now we define our loss function, which in this case is simply the RSS:

```python
def L(w):
    predicted_data = np.array([F_hat(x, w) for x in data])
    return np.sum((training_data - predicted_data) ** 2)
```

Finally we optimize the loss function given some initial weights:

```python
initial_weights = [0., 0., 0.]
final_weights = opt.minimize(L, initial_weights)
final_weights.x
```

    array([-3.35123236, -2.5881134 ,  0.92571244])

Evaluating the model¶

```python
plt.plot(domain, F(domain))
plt.plot(domain, F_hat(domain, final_weights.x))
plt.plot(data, training_data, 'x')
plt.legend(['$F(x)$', '$\hat{F}(x)$'])
plt.show()
```

Statistical and machine learning in finance¶

Estimating the single-index model to derive alpha and beta.

Estimating default probabilities in credit-risk models.

Solving stochastic-control problems for algorithmic trading.

Time-series forecasting.

Predicting excess returns for portfolio allocation.

Asset rating models.

Regression methods¶

Simple linear regression¶

Consider a vector $\mathbf{x}$ containing samples from our predictor, and a vector of $\mathbf{y}$ of corresponding responses, both of size $n$.

We assume a stochastic linear relationship between response and predictor: $y = F(x) = \alpha + \beta x + \epsilon$,

Our weight vector simply consists of our estimated intercept and slope: $\mathbf{w} = (\hat{\alpha}, \hat{\beta})$

Our predictions are given by:

\begin{equation}
\hat{F}_{(\hat{\alpha}, \hat{\beta})} = \hat{\alpha} + \hat{\beta} x
\end{equation}

The residuals are the differences between our predictions and the observed training response:

\begin{equation}
r_i  =  y_i - F_\mathbf{w}(x_i)
\end{equation}

The loss function $L$ is the residual sum of squares (RSS):

\begin{equation}
L(\mathbf{x}, \mathbf{y}, \mathbf{w}) = \sum_{i=1}^n r_i^2
\end{equation}

Assumptions¶

$F(x)$ is linear.

The noise term is i.d.d. normally distributed with a mean 0 and constant variance:

$$\epsilon \sim N(0, \sigma^2_{\epsilon})$$

The independent variable is uncorrelated with the noise $E[x_{ij}\epsilon_i] = 0$.

Least squares estimates¶

We want to maximise the goodness-of-fit by minimising the expected loss:

\begin{equation}
 \underset{\mathbf{w}}{\operatorname{argmin}} E[L(\mathbf{x}, F_\mathbf{w})]
\end{equation}
where $\mathbf{w} = (\hat{\alpha}, \hat{\beta})$

In this case we can use calculus to obtain a closed-form solution without resorting to computational methods:

\begin{eqnarray}
\hat{\beta} & = & \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x}) ^ 2} \\
\hat{\alpha} & = & \bar{y} - \hat{\beta} \bar{x}
\end{eqnarray}
where $\bar{x}$ and $\bar{y}$ are the sample means.

Applications to Finance - The Single Index Model¶
$$r_{i,t} - r_f = \alpha_i + \beta_i ( r_{m,t} - r_f) + \epsilon_{i,t}$$$$\epsilon_{i, t} \sim N(0, \sigma_i)$$

$r_{i,t}$ is return to stock $i$ in period $t$.

$r_f$ is the risk-free rate.

$r_{m,t}$ is the return to the market portfolio.

Elton, E. J., & Gruber, M. J. (1997). Modern portfolio theory, 1950 to date. Journal of Banking and Finance, 21(11–12), 1743–1759. https://doi.org/10.1016/S0378-4266(97)00048-4

Loading data into a pandas dataframe¶

We will first obtain some data from Yahoo finance using the pandas library.

First we will import the functions and modules we need.

```python
import matplotlib.pyplot as plt
import datetime
import pandas as pd
import numpy as np
```

Downloading price data using as CSV¶

Here we obtain price data on Microsoft Corporation Common Stock, so we specify the symbol MSFT.

```python
def prices_from_csv(fname):
    df = pd.read_csv(fname)
    df.set_index(pd.to_datetime(df['Date']), inplace=True)
    return df
```

```python
msft = prices_from_csv('data/MSFT.csv')
msft.head()
```

Plotting the price of the stock¶

```python
msft['Adj Close'].plot()
plt.ylabel('MSFT price')
plt.show()
```

Converting to monthly data¶

We will resample the data at a frequency of one calendar month.

The code below takes the last price in every month.

```python
daily_prices = msft['Adj Close']
```

monthly_prices = daily_prices.resample('M').last()
monthly_prices.plot()
plt.ylabel('MSFT Price')
plt.show()

Calculating log returns¶

```python
stock_returns = pd.DataFrame({'MSFT monthly returns': np.log(monthly_prices).diff().dropna()})
stock_returns.plot()
plt.xlabel('t'); plt.ylabel('$r_t$')
plt.show()
```

Return histogram¶

```python
stock_returns.hist()
plt.show()
```

Descriptive statistics of the return distribution¶

```python
stock_returns.describe()
```

Summarising the distribution using a boxplot¶

```python
stock_returns.boxplot()
plt.show()
```

Q-Q plots¶

Quantile-Quantile (Q-Q) plots are a useful way to compare distributions.

We plot empirical quantiles against the quantiles computed the inverted c.d.f. of a specified theoretical distribution.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

stats.probplot(stock_returns.values[:,0], dist="norm", plot=plt)
plt.show()
```

The market index¶

We will first obtain data on the market index: in this case the NASDAQ:

```python
nasdaq_index = prices_from_csv('data/^NDX.csv')
nasdaq_index.head()
```

Converting to monthly data¶

As before, we can resample to obtain monthly data.

```python
nasdaq_monthly_prices = nasdaq_index['Adj Close'].resample('M').last()
nasdaq_monthly_prices.head()
```

Plotting monthly returns¶

```python
index_log_returns = pd.DataFrame({'NASDAQ monthly returns': np.log(nasdaq_monthly_prices).diff().dropna()})
index_log_returns.plot()
plt.show()
```

Converting to simple returns¶

```python
index_simple_returns = np.exp(index_log_returns) - 1.
index_simple_returns.plot()
plt.show()
```

```python
stock_simple_returns = np.exp(stock_returns) - 1.
```

Concatenating data into a single data frame¶

We will now concatenate the data into a single data fame.

We can use pd.concat(), specifying an axis of 1 to merge data along columns.

This is analogous to performing a zip() operation.

```python
comparison_df = pd.concat([index_simple_returns, stock_simple_returns], axis=1)
comparison_df.head()
```

Scatter plots¶

We can produce a scatter plot to see whether there is any relationship between the stock returns, and the index returns.

There are two ways to do this:

Use the function scatter() in matplotlib.pyplot

Invoke the plot() method on a data frame, passing kind='scatter'

Scatter plots using the plot() method of a data frame¶

In the example below, the x and y named arguments refer to column numbers of the data frame.

Notice that the plot() method is able to infer the labels automatically.

```python
comparison_df.plot(x=0, y=1, kind='scatter')
plt.show()
```

Computing the correlation matrix¶

For random variables $X$ and $Y$, the Pearson correlation coefficient is:

\begin{eqnarray}
\rho_{X,Y} & = & \frac{\operatorname{cov}(X, Y)}{\sigma_X \sigma_Y} \\
        & = & \frac{E[(X - \mu_x)(Y - \mu_Y)]}{\sigma_X \sigma_Y}\\
\end{eqnarray}

Covariance and correlation of a data frame¶

We can invoke the cov() and corr() methods on a data frame.

```python
comparison_df.cov()
```

```python
comparison_df.corr()
```

Comparing multiple attributes in a data frame¶

It is often useful to work with more than two variables.

We can add columns (attributes) to our data frame.

Many of the methods we are using will automatically incorporate the additional variables into the analysis.

Using a function to compute returns¶

The code below defines a function defines a function which will return a data frame containing a single series of returns for the specified symbol, and sampled over the specified frequency.

```python
def returns_df(symbol, frequency='M'):
    df = prices_from_csv('data/%s.csv' % symbol)
    prices = df['Adj Close'].resample(frequency).last()
    column_name = symbol + ' returns (' + frequency + ')'
    return pd.DataFrame({column_name: np.exp(np.log(prices).diff().dropna()) - 1.})
```

```python
apple_returns = returns_df('AAPL')
apple_returns.head()
```

Adding another stock to the portfolio¶

```python
comparison_df = pd.concat([comparison_df, apple_returns], axis=1)
comparison_df.head()
```

```python
comparison_df.plot()
plt.show()
```

```python
comparison_df.corr()
```

Scatter matrices¶

```python
pd.plotting.scatter_matrix(comparison_df, figsize=(8, 6))
plt.show()
```

Scatter matrices with Kernel-density plots¶

We can use Kernel density estimation (KDE) to plot an approximation of the pdf.

```python
pd.plotting.scatter_matrix(comparison_df, diagonal='kde', figsize=(8, 6))
plt.show()
```

Ordinary-least squares estimation in scikit-learn¶

First we import the required modules:

```python
from sklearn.preprocessing import scale
import sklearn.linear_model as skl_lm
from sklearn.metrics import mean_squared_error, r2_score
```

Now we prepare the data set:

```python
rr = 0.01 # risk-free rate
ydata = stock_simple_returns - rr
xdata = index_simple_returns - rr
```

Finally we fit the model

```python
regr = skl_lm.LinearRegression()
regr.fit(xdata, ydata)

alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
print(alpha);  print(beta)
```

Plotting the fitted model¶

```python
plt.scatter(x=xdata, y=ydata)
plt.plot(xdata, alpha + beta * xdata)
plt.xlabel('index return')
plt.ylabel('stock return')
plt.title('Single-index model fit ')
plt.show()
```

Plotting the residuals¶

```python
residuals = regr.predict(xdata) - ydata
plt.plot(residuals)
plt.title('residuals plot')
plt.show()
```

RSS contours and surface¶

```python
# Regression coefficients (Ordinary Least Squares)
regr = skl_lm.LinearRegression()

X = scale(xdata, with_mean=True, with_std=False).reshape(-1,1)
y = ydata['MSFT monthly returns']

regr.fit(X,y)

# Create grid coordinates for plotting
B0 = np.linspace(regr.intercept_-0.15, regr.intercept_+0.15, 50)
B1 = np.linspace(regr.coef_-2.5, regr.coef_+2.5, 50)
xx, yy = np.meshgrid(B0, B1, indexing='xy')
Z = np.zeros((B0.size,B1.size))

# Calculate Z-values (RSS) based on grid of coefficients
for (i,j),v in np.ndenumerate(Z):
    Z[i,j] =((y - (xx[i,j] + X.ravel() * yy[i,j]))**2).sum() / 52.5

# Minimized RSS
min_RSS = r'$\alpha$, $\beta$ for minimized RSS'
min_rss = np.sum((regr.intercept_ + regr.coef_ * X - y.values.reshape(-1,1))**2) / 52.5
```

```python
# %load ../standard_import.txt
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
import seaborn as sns

from sklearn.preprocessing import scale
import sklearn.linear_model as skl_lm
from sklearn.metrics import mean_squared_error, r2_score
import statsmodels.api as sm
import statsmodels.formula.api as smf

%matplotlib inline
plt.style.use('seaborn-white')

fig = plt.figure(figsize=(15,6))
#fig.suptitle('RSS - Regression coefficients', fontsize=20)

ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122, projection='3d')

# Left plot
CS = ax1.contour(xx, yy, Z, cmap=plt.cm.Set1, levels=[0.01, 0.02, 0.03, 0.04, 0.05])
ax1.scatter(regr.intercept_, regr.coef_[0], c='r', label=min_RSS)
ax1.clabel(CS, inline=True, fontsize=10, fmt='%1.2f')

# Right plot
ax2.plot_surface(xx, yy, Z, rstride=3, cstride=3, alpha=0.3)
ax2.contour(xx, yy, Z, zdir='z', offset=Z.min(), cmap=plt.cm.Set1,
            alpha=0.4, levels=[0.01, 0.02])
ax2.scatter3D(regr.intercept_, regr.coef_[0], min_rss, c='r', label=min_RSS)
ax2.set_zlabel('RSS')
ax2.set_zlim(Z.min(),Z.max())
ax2.set_ylim(-1, 3)

# settings common to both plots
for ax in fig.axes:
    ax.set_xlabel(r'$\alpha$', fontsize=17)
    ax.set_ylabel(r'$\beta$', fontsize=17)
   # ax.set_yticks([0.03,0.04,0.05,0.06])
    ax.legend()

plt.show()
```

Confidence intervals and standard error¶

Notice that the optimal values of $\alpha$ and $\beta$ can be expressed as functions of sample means.

What happens when we use different training data, i.e. different samples of the training data?

Estimation error¶

By the law of large numbers $\lim_{n \rightarrow \infty} \bar{\mathbf{x}} = E(X)$.

However, for finite values of $n$ we will have an estimation error.

Can we quantify the estimation error as a function of $n$?

Computing the error numerically¶

If we draw from a standard normal distribution, we know that $E(X) = 0$.

Therefore we can easily compute the estimation error in any given sample.

The error for a small random sample.¶

Here $X \sim N(0, 1)$, and we draw a random sample $\mathbf{x} = (x_1, x_2, \ldots, x_n)$ of size $n=5$.

We will compute $\epsilon_\mathbf{x} = | \bar{\mathbf{x}} - E(X) | = | \bar{\mathbf{x}} |$

```python
x = np.random.normal(size=5)
x
```

    array([ 0.30223606,  1.2483894 , -1.28832977,  1.11715834,  0.18664801])

```python
np.mean(x)
```

    0.31322040749786373

```python
estimation_error = np.sqrt(np.mean(x)**2)
estimation_error
```

    0.31322040749786373

Variance in the error¶

If we draw a different sample, will the error be different or the same?

```python
x = np.random.normal(size=5)
estimation_error = np.mean(x)**2
estimation_error
```

    0.01171702349311607

```python
x = np.random.normal(size=5)
estimation_error = np.mean(x)**2
estimation_error
```

    0.03023963175168937

```python
x = np.random.normal(size=5)
estimation_error = np.mean(x)**2
estimation_error
```

    0.19454420001429074

Expected error¶

The error $\epsilon_{\mathbf{x}}$ is itself a random variable.

How can we compute $E(\epsilon_{\mathbf{x}})$?

Monte-Carlo estimation of the sampling error¶

```python
def sampling_error(n):
    errors = [np.sqrt(np.mean(np.random.normal(size=n))**2) \
              for i in range(100000)]
    return np.mean(errors)

sampling_error(5)
```

    0.35656013206003007

Notice that this estimate is relatively stable:

```python
sampling_error(5)
```

    0.3557039282930304

```python
sampling_error(5)
```

    0.3557907867498346

Monte-Caro estimation of the standard error¶

We can now examine the relationship between sample size $n$ and the expected error using a Monte-Carlo method.

```python
import matplotlib.pyplot as plt
n = np.arange(5, 200, 10)
plt.plot(n, np.vectorize(sampling_error)(n))
plt.xlabel('$n$');  plt.ylabel('$e_\mathbf{x}$')
plt.show()
```

The sampling distribution of the mean¶

The variance in the error occurs because the sample mean is a random variable.

What is the distribution of the sample mean?

The sampling distribution of the mean¶

Let's fix the sample size at $n=30$, and look at the empirical distribution of the sample means.

```python
# Sample size
n = 30
# Number of repeated samples
N = 20000

means_30 = [np.mean(np.random.normal(size=n)) for i in range(N)]
ax = plt.hist(means_30, bins=50)
plt.show()
```

The sampling distribution of the mean¶

Now let's do this again for a variable sampled from a different distribution: $X \sim U(0, 1)$.

```python
# Sample size
n = 30
# Number of repeated samples
N = 20000
means_30_uniform = [np.mean(np.random.uniform(size=n)) for i in range(N)]
ax = plt.hist(means_30_uniform, bins=50)
plt.show()
```

Increasing the sample size¶

```python
# Sample size
n = 200

means_200 = [np.mean(np.random.normal(size=n)) for i in range(N)]
ax1 = plt.hist(means_30, bins=50)
ax2 = plt.hist(means_200, bins=50)
plt.show()
```

```python
# Sample size
n = 1000
means_1000 = [np.mean(np.random.normal(size=n)) for i in range(N)]
ax1 = plt.hist(means_30, bins=50)
ax2 = plt.hist(means_200, bins=50)
ax3 = plt.hist(means_1000, bins=50)
plt.show()
```

The sampling distribution of the mean¶

In general the sampling distribution of the mean approximates a normal distribution.

If $X \sim N(\mu, \sigma^2)$ then $\bar{\mathbf{x}_n} \sim N(\mu, \frac{\sigma^2}{n})$.

The standard error of the mean is $\sigma_{\bar{\mathbf{x}}} = \frac{\sigma}{\sqrt{n}}$.

Therefore sample size must be quadrupled to achieve half the error.

Errors in simple regression¶

Let's perform a similar simulation experiment for OLS.

We will generate some simulated data from the function $F(x) = 2 + 3x + \epsilon$.

```python
n = 20

def noise(size=n):
    return np.random.normal(size=(size, 1), scale=5.0)

def F(x):
    return 2 + 3*x

def F_noise(x):
    return F(x) + noise(len(x))
```

First sample¶

```python
xdata = np.random.uniform(size=(n, 1), low=-8., high=+8.)
ydata = F_noise(xdata)
regr = skl_lm.LinearRegression()
regr.fit(xdata, ydata)
alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
print(alpha);  print(beta)
```

    1.642420706251706
    3.0552559965405663

Second sample¶

```python
xdata = np.random.uniform(size=(n, 1), low=-8., high=+8.)
ydata = F_noise(xdata)
regr = skl_lm.LinearRegression()
regr.fit(xdata, ydata)
alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
print(alpha);  print(beta)
```

    2.3194865384370655
    2.965875701867936

Regression plots for different training sets¶

```python
plt.figure()
domain = np.linspace(-8, 8.)
plt.plot(domain, F(domain), color='red')
for i in range(5):
    xdata = np.random.uniform(size=(20, 1), low=-8., high=+8.)
    ydata = F_noise(xdata)
    plt.plot(xdata, ydata, 'x')
    regr = skl_lm.LinearRegression()
    regr.fit(xdata, ydata)
    alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
    plt.plot(domain, alpha + beta * domain, color='blue', alpha=0.3)
plt.show()
```

The distribution of the estimates¶

```python
domain = np.linspace(-8, 8.)
num_experiments = 10000; results = np.zeros((num_experiments, 2))
for i in range(num_experiments):
    xdata = np.random.uniform(size=(n, 1), low=-8., high=+8.)
    ydata = F_noise(xdata)
    regr = skl_lm.LinearRegression()
    regr.fit(xdata, ydata)
    alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
    results[i, 0] = alpha;  results[i, 1] = beta
```

```python
print (results)
```

    [[2.23715473 2.73423658]
     [2.84481862 3.33877566]
     [1.06434277 2.82663063]
     ...
     [3.69954616 3.02820171]
     [4.11258572 3.05502981]
     [0.6256662  3.12803704]]

```python
pd.DataFrame(results, columns=['$\\alpha$', '$\\beta$']).describe()
```

      $\alpha$
      $\beta$

      count
      10000.000000
      10000.000000

      mean
      2.003860
      3.004086

      std
      1.139815
      0.253460

      min
      -2.019178
      1.909462

      25%
      1.238947
      2.837807

      50%
      1.998437
      3.005890

      75%
      2.768708
      3.173130

      max
      6.551488
      3.953136

Quantifying the variance of a model¶

We can quantify the goodness-of-fit of our model when we train it on different data sets.

Consider some out-of-sample data $x_0 = 10$ and $y_0 = f(x_0) + \epsilon$.

Over the different training sets, we can estimate:

the variance of our forecasts: $\operatorname{Var}(\hat{f}(x_0))$,

the expected squared-error $E(y_0 - \hat{f}(x_0))^2$.

Estimating MSE and variance using Monte-Carlo¶

```python
domain = np.linspace(-8, 8.);
out_of_sample_data = np.array([10.])
num_experiments = 50000; results = np.zeros((num_experiments, 2))
for i in range(num_experiments):
    xdata = np.random.uniform(size=(n, 1), low=-8., high=+8.)
    ydata = F_noise(xdata)
    regr = skl_lm.LinearRegression()
    regr.fit(xdata, ydata)
    alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
    prediction = alpha + beta * out_of_sample_data
    squared_error = (F_noise(out_of_sample_data) - prediction) ** 2
    results[i] = [squared_error, prediction]
```

The relationship between error and variance¶

```python
mean_squared_error = np.mean(results[:,0])
mean_squared_error
```

    32.74892918802504

```python
prediction_variance = np.var(results[:,1])
prediction_variance
```

    7.833350861678941

```python
noise_variance = np.var(noise(size=50000))
noise_variance
```

    25.119853685100384

```python
noise_variance + prediction_variance
```

    32.953204546779325

Fitting a biased model¶

In the previous example we fitted a linear model to linear data.

Let's see what happens if we attempt to fit a linear model to non-linear data.

```python
def F(x):
    return 2 + 3*x + 0.2*x**2

def F_noise(x):
    return F(x) + noise(len(x))
```

Regression plots for different training sets¶

```python
plt.figure()
domain = np.linspace(-8, 8.)
plt.plot(domain, F(domain), color='red')
for i in range(5):
    xdata = np.random.uniform(size=(20, 1), low=-8., high=+8.)
    ydata = F_noise(xdata)
    plt.plot(xdata, ydata, 'x')
    regr = skl_lm.LinearRegression()
    regr.fit(xdata, ydata)
    alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
    plt.plot(domain, alpha + beta * domain, color='blue', alpha=0.3)
plt.show()
```

Estimating error and variance¶

```python
domain = np.linspace(-8, 8.)
out_of_sample_data = np.array([10.])
num_experiments = 50000; results = np.zeros((num_experiments, 2))
for i in range(num_experiments):
    xdata = np.random.uniform(size=(n, 1), low=-8., high=+8.)
    ydata = F_noise(xdata)
    regr = skl_lm.LinearRegression()
    regr.fit(xdata, ydata)
    alpha = regr.intercept_[0]; beta = regr.coef_[0][0]
    prediction = alpha + beta * out_of_sample_data
    squared_error = (F_noise(out_of_sample_data) - prediction) ** 2
    results[i] = [squared_error, prediction]
```

Results¶

```python
mean_squared_error = np.mean(results[:,0])
mean_squared_error
```

    295.29044416473397

```python
prediction_variance = np.var(results[:,1])
prediction_variance
```

    15.510077287021177

```python
noise_variance = np.var(noise(size=50000))
noise_variance
```

    25.14380260485566

```python
noise_variance + prediction_variance
```

    40.629930972121564

The bias-variance trade-off¶

The additional source of error in the previous example is called the bias.

Bias arises when our model is of the wrong form to be able to fully fit the data.

In general:

\begin{equation}
E[y_0 - \hat{f}(x)]^2 = \operatorname{Var}[\hat{f}(x)] + \operatorname{Bias}[\hat{f}(x)]^2 + \operatorname{Var}[\epsilon]
\end{equation}
where

\begin{equation}
\operatorname{Bias}[\hat{f}(x)] = E[\hat{f}(x)] - f(x)
\end{equation}

```python
bias = np.sqrt(mean_squared_error - (noise_variance + prediction_variance))
bias
```

    15.957335751085052

Improving the fit the model¶

We cannot eliminate the final term in the previous equation; $\operatorname{Var}(\epsilon)$ is irreducible error.

In general, if we reduce bias, we increase variance.

This is called the bias-variance trade-off.

We can reduce variance by reducing the number of weights in $\mathbf{w}$.

In multiple regression this can be done by reducing the number of predictors.

Correspondingly as we increase the number of predictors we increase variance, but reduce bias.

This can result in over-fitting to the training data.

Bias-variance tradeoff example: test function and data¶

Bias-variance trade-off example: spread=5¶

Bias-variance trade-off example: spread=1¶

Bias-variance trade-off example: spread=0.1¶

Bias-variance trade-off as multi-objective optimization¶

Basis Functions¶

We can use non-linear basis functions to decrease the bias of our model.

We transform a variable $X$ into $b_1(X), b_2(X), \ldots, b_K(X)$

We then fit the model:

\begin{equation}
y_i = \beta_0 + \beta_1 b_1(x_i) + \beta_2 b_2(x_i) + \beta_3 b_3(x_i) + \ldots + \beta_Kb_K(x_i) + \epsilon_i
\end{equation}

The basis functions are known and chosen.

The above equation is still linear in the basis functions.

Multiple linear-regression¶

Simple regression can be generalized to multiple predictors.

Consider $p$ distinct predictors with corresponding observations of size $n$: $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_p$, then:

\begin{equation}
\mathbf{y} = \beta_0 + \beta_1 \mathbf{x}_1 + \beta_2 \mathbf{x}_2 + \ldots + \beta_p \mathbf{x}_p + \epsilon
\end{equation}

In matrix notation:

\begin{equation}
\mathbf{y} = \mathbf{X} \mathbf{w} + \mathbf{\epsilon}
\end{equation}\begin{equation}
\begin{pmatrix}
Y_1 \\
Y_2 \\
\vdots \\
Y_n
\end{pmatrix} =
\begin{pmatrix}
1 & X_{11} & X_{21} & \ldots & X_{p1} \\
1 & X_{12} & X_{21} & \ldots & X_{p2} \\
\vdots & \vdots & \vdots & \ldots & \vdots \\
1 & X_{1n} & X_{2n} & \ldots & X_{pn} \\
\end{pmatrix}
\begin{pmatrix}
\beta_1 \\
\beta_2 \\
\vdots  \\
\beta_n \\
\end{pmatrix} +
\begin{pmatrix}
\epsilon_1 \\
\epsilon_2 \\
\vdots \\
\epsilon_n \\
\end{pmatrix}
\end{equation}

The closed-form solution to the loss-minimization problem is:

\begin{equation}
\hat{\mathbf{w}} = (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}
\end{equation}

Applications to Finance: factor models¶

Fama-French 3-factor model:

\begin{equation}
r_i - r_f = \beta_{0} + \beta_{1} ( r_m - r_f) + \beta_{2} s_i + \beta_{3} v_i + \epsilon_{i,t}
\end{equation}

$r_i$ is the return for stock $i$.

$s_i$ is the size of stock $i$.

$v_i$ is the value of stock $i$.

$r_f$ is the risk-free rate.

$r_m$ is the return to the market portfolio.

Fama, Eugene F., and Kenneth R. French. "Common risk factors in the returns on stocks and bonds." Journal of Financial Economics 33:1 (1993). https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.139.5892&rep=rep1&type=pdf

Polynomial regression¶

For polynomial regression we use:

\begin{equation}
b_j(x_i) = x_i^{j-1}
\end{equation}

For $K=2$ the simple linear regression model becomes:

\begin{equation}
y = \beta_0 + \beta_1 x + \beta_2 x^2
\end{equation}

Note that we have introduced an additional feature into our model.

Polynomial basis functions in scikit-learn¶

```python
from sklearn.preprocessing import PolynomialFeatures
x = np.array([2, 3, 4])
poly = PolynomialFeatures(3, include_bias=False)
poly.fit_transform(x[:, None])
```

    array([[ 2.,  4.,  8.],
           [ 3.,  9., 27.],
           [ 4., 16., 64.]])

```python
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression
poly_model = make_pipeline(PolynomialFeatures(7),
                           LinearRegression())
```

Plotting the fitted model¶

Here we use polynomial basis functions with $K=7$ to fit $y = \sin(x) + \epsilon$.

```python
rng = np.random.RandomState(1)
x = np.linspace(0, 10.)
y = np.sin(x) + 0.1 * rng.randn(50)

poly_model.fit(x[:, np.newaxis], y)
xfit = x
yfit = poly_model.predict(xfit[:, np.newaxis])

plt.scatter(x, y)
plt.plot(xfit, yfit);
```

Overfitting¶

Let's see what happens if we fit a polynomial model to linear data.

We see that the model has simply fitted to the noise; it has over-fitted to the training data.

```python
rng = np.random.RandomState(1)
x = np.linspace(0, 10., 20)
y = 0.1 + x + rng.randn(20)*3.
poly_model.fit(x[:, np.newaxis], y)
xfit = x
yfit = poly_model.predict(xfit[:, np.newaxis])
plt.scatter(x, y)
plt.plot(xfit, yfit);
```

Model validation¶

It is very important to test for over-fitting, particularly with high-variance models.

To test for over-fitting we hold back some data from the training.

We divide data into two subsets:

training data

validation data

We validate the model by estimating the MSE on the validation data.

This is also called out-of-sample validation.

Cross-validation¶

In cross-validation we create multiple partitions of the data

Each way of paritition data is called a round.

We take an average of the error across rounds.

Leave-one-out cross-validation¶

k-fold cross-validation¶

Validating the single-index model¶

Let's perform model validation with the data we used to train the single-index model:

```python
rr = 0.01 # risk-free rate
ydata = stock_simple_returns - rr
xdata = index_simple_returns - rr
```

We first partition the data into chunks of equal size $n$:

```python
def in_subset(data, i, n=10):
    return data[i*n:(i+1)*n]
```

The first subset¶

```python
in_subset(xdata, 0, n=5)
```

The second subset¶

```python
in_subset(xdata, 1, n=5)
```

The third subset¶

```python
in_subset(xdata, 2, n=5)
```

The remaining data¶

```python
def out_subset(data, i, size=10):
    return pd.concat([data[:i*size], data[(i+1)*size:]])
```

Model fitting¶

Here we fit the model to given data, returning a function that can
later be applied to make a prediction.

```python
def fit_model(xdata, ydata):
    regr = skl_lm.LinearRegression()
    regr.fit(xdata, ydata)
    alpha = regr.intercept_[0]
    beta = regr.coef_[0][0]
    return lambda x: alpha + beta*x
```

Leave-one-out cross-validation in Python¶

```python
n = 10; N = int(np.floor(len(xdata) / n)); mse = np.zeros(N)
for i in range(N):
    test_data_x = in_subset(xdata, i)
    test_data_y = in_subset(ydata, i)
    training_data_x = out_subset(xdata, i)
    training_data_y = out_subset(ydata, i)
    model = fit_model(training_data_x, training_data_y)
    predicted_ydata = model(test_data_x)
    mse[i] = np.mean((test_data_y.values - predicted_ydata.values)**2)
mse
```

```python
np.mean(mse)
```

Potential problems with linear-regression¶

Non-linearity of the response-predictor relationships.¶

If the actual relationship between response and predictor is non-linear then linear regression with give a very biased result.

Correlation or non-constant variance in error terms.¶

We assume that the $\epsilon$ variate is i.i.d.

If there are correlations in $\epsilon$, linear regression can give misleading results.

This often occurs in time-series data.

Similarly if the variance of the residuals is not constant.

As we saw with the single-index model fit, we can examine the plot of the residuals for patterns.

Outliers and high-leverage points¶

Extreme values of response variables are called outliers.

They can be analyzed using box-plots.

Outliers can cause over-fitting.

Extreme values of predictor variables are called high-leverage points.

Collinearity.¶

We assume there are no linear relationships between predictors.

If there are, then we are using redundant information, which can make the model hard to interpret.

Collinearity can be detected by looking at scatter matrices.

In general, combinations of variables can exhibit a linear relationship, which is called multi-collinearity.

Multi-collinearity can be detected by computing the Variable Inflation Factor (VIF).

If there are redundant variables, then typically we want to omit them from the regression.

Finding the correct set of predictor variables is called variable selection.

Variable-selection and regularization¶

We can address some of these issues using penalized regression techniques.

Penalized regression can be used to reduce the variance of a model through regularization.

It can also be used to eliminate redundant variables.

This is particularly important for "big-data" problems with large numbers of predictor variables $p$;

for an example in Finance, see the WRDS database.

Provided the number of observations is significantly greater that the number of preditors $n > p$, ordinary least squares estimation can give low-variance estimates.

If $n < p$ there is no single optimal solution to the OLS optimization problem.

Shrinkage using Ridge regression¶

Recall that in linear regression our loss function $L$
is simply the residual sum of squares (RSS):

\begin{equation}
\operatorname{RSS} = \sum_{i=1}^n r_i^2
\end{equation}
where

\begin{eqnarray}
r_i  & = &  y_i - F_\mathbf{w}(x_i) \\
    & = & y_i - w_0 - \sum_{j=1}^p w_j x_{ij}
\end{eqnarray}

In ridge regression we add an additional shrinkage penalty term:

\begin{equation}
L(\mathbf{x}, \mathbf{y}, \mathbf{w}) = \operatorname{RSS} + \lambda \sum_{j=1}^p w_j^2
\end{equation}
where $\lambda$ is a tunable hyper-parameter.

The optimal weights are:

\begin{equation}
w^* = (\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^T \mathbf{y}
\end{equation}

Ridge regression in Python¶

In scikit-learn you can perform ridge regression simply by using the Ridge() function instead of the LinearRegression() function:

```python
from sklearn.linear_model import Ridge
```

The $\mathscr{l_2}$ norm¶

The $\mathscr{l_2}$ norm of a vector gives its Euclidian distance from the origin:

\begin{equation}
||\mathbf{w}||_2 = \sqrt{\sum_{j=1}^p {w_j}^2}
\end{equation}

As $\lambda$ increases $||\hat{\mathbf{w}}_{\lambda}||_2$ decreases.

```python
# %load ../standard_import.txt
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

import glmnet_python
from glmnet import glmnet

from sklearn.preprocessing import scale
from sklearn import model_selection
from sklearn.linear_model import LinearRegression, Ridge, RidgeCV, Lasso, LassoCV
from sklearn.decomposition import PCA
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.metrics import mean_squared_error

%matplotlib inline
plt.style.use('seaborn-white')

df = pd.read_csv('data/Hitters.csv', index_col=0).dropna()
df.index.name = 'Player'
y = df.Salary
dummies = pd.get_dummies(df[['League', 'Division', 'NewLeague']])
# Drop the column with the independent variable (Salary), and columns for which we created dummy variables
X_ = df.drop(['Salary', 'League', 'Division', 'NewLeague'], axis=1).astype('float64')
# Define the feature set X.
X = pd.concat([X_, dummies[['League_N', 'Division_W', 'NewLeague_N']]], axis=1)
```

```python
X_train = pd.read_csv('data/Hitters_X_train.csv', index_col=0)
y_train = pd.read_csv('data/Hitters_y_train.csv', index_col=0)
X_test = pd.read_csv('data/Hitters_X_test.csv', index_col=0)
y_test = pd.read_csv('data/Hitters_y_test.csv', index_col=0)
```

```python
alphas = 10**np.linspace(6,-2,100)*0.5

ridge = Ridge()
coefs = []

for a in alphas:
    ridge.set_params(alpha=a)
    ridge.fit(scale(X), y)
    coefs.append(ridge.coef_)

ax = plt.gca()
ax.plot(alphas, coefs)
ax.set_xscale('log')
ax.set_xlim(ax.get_xlim()[::-1])  # reverse axis
plt.axis('tight')
plt.xlabel('$\lambda$')
plt.ylabel('$\mathbf{w}$')
plt.title('Ridge coefficients as a function of the regularization');
```

Scaling and standardization¶

The ridge penalty term is not scale equivariant.

Therefore when using penalised regression we must use standardized  predictors:

\begin{equation}
    \tilde{x}_{ij} = \frac{x_{ij}}{\sqrt{\frac{1}{n}\sum_{i=1}^n (x_ij - \bar{x}_j)^2}}
\end{equation}

In scikit-learn you can use the scale() function:

```python
from sklearn.preprocessing import scale
```

Shrinkage, bias and variance¶

Shrinkage reduces variance.

Therefore it increases bias.

Therefore there exist optimal values of $\lambda$ which minimize out-of-sample MSE.

This hyper-parameter can be tuned by using cross-validation methods to find the optimal value.

Lasso regression¶

The lasso regression uses an $\mathscr{l_1}$ penalty term:

\begin{equation}
L(\mathbf{x}, \mathbf{y}, \mathbf{w}) = \operatorname{RSS} + \lambda \sum_{j=1}^p |w_j|
\end{equation}

In contrast to ridge regression, the lasso does not shrink all coefficients to zero for the same $\lambda$.

This means that it can be used for variable selection.

```python
import warnings; warnings.simplefilter('ignore')

alphas = 10**np.linspace(6,-2,100)*0.5

lasso = Lasso()
coefs = []

for a in alphas:
    lasso.set_params(alpha=a)
    lasso.fit(scale(X), y)
    coefs.append(lasso.coef_)

ax = plt.gca()
ax.plot(alphas, coefs)
ax.set_xscale('log')
ax.set_xlim(ax.get_xlim()[::-1])  # reverse axis
plt.axis('tight')
plt.xlabel('$\lambda$')
plt.ylabel('$\mathbf{w}$')
plt.title('Lasso coefficients as a function of the regularization');
```

Penalized regression as constrained optimization¶

An alternative formulation of penalized regression is to view it as constrained optimization.

We minimize the standard OLS loss function:

$L(\mathbf{x}, \mathbf{y}, \mathbf{w}) = \operatorname{RSS}$

For ridge regression we use the constraint:

$$\sum_{j=1}^p w_{j}^2 \leq s$$

For lasso regression we use the constraint:

$$\sum_{j=1}^p |w_{j}| \leq s$$

For every $\lambda$ there is a corresponding constant budget $s$ for which solutions to the constrained and unconstrained problem are identical.

Error contours and contraints for the ridge penalty¶

Error contours and contraints for the lasso penalty¶

Lasso and Ridge comparision¶

Lasso is able to perform variable-selection when the data-set contains many redundant features that do not influence the response.

Ridge works when most predictors impact the response.

In practice we don't know the true model, so how should we select the penalization?

Elastic-net¶

We can combine both penalties into a single term by using a linear combination.

This transforms the problem into a multi-objective optimisation problem (MOO).

The parameter $\alpha$ specifies the weighting towards a given objective.

\begin{equation}
L(\mathbf{x}, \mathbf{y}, \mathbf{w}) = \operatorname{RSS} + \lambda \left( \frac{1 - \alpha}{2} \sum_{j=1}^p |w_j| + \alpha  \sum_{j=1}^p w_j^2 \right)
\end{equation}

Penalized regression in Finance¶

Bianchi and McAlinn (2020) attempt to predict monthly excess returns based on 70 predictors.

The predictors are financial ratios taken from the WRDS database.

Bianchi, D., & McAlinn, K. (2020). Divide and Conquer: Financial Ratios and Industry Returns Predictability (No. 3136368; SSRN). https://doi.org/10.2139/ssrn.3136368

Predictor categories¶

The predictors are grouped into seven categories:

Capitalization

Efficiency

Financial Soundness/Solvency

Liquidity

Profitability

Valuation

Other

Example predictors¶

Gross profitability as a fraction of total assets,

Net Income as a fraction of average of Common Equity based on most recent two periods,

$\ldots$

(Bianchi and McAlinn 2020, p. 70)

Annualized certainty-equivalent by method¶

Method
Durables
NonDurables
Manuf
Energy
HiTech
Health
Other
Shops
Telecomm
Utils

OLS
-1.660
-0.527
-1.072
-0.874
-0.856
-0.353
-0.898
-0.163
-0.411
-0.227
-0.729
-1.409

Lasso
-0.478
-0.425
-0.291
-0.734
-0.182
-0.081
-0.135
-0.753
-0.057
-0.248
-0.001
-0.567

Ridge
0.151
-0.256
-0.138
-1.027
-0.138
-0.328
-0.187
-0.097
-0.188
-0.202
-0.718
-0.455

Enet
0.013
-0.553
-0.346
0.048
-0.434
-0.499
-0.086
-0.582
-1.250
-0.092
0.125
-0.490

EW
-0.142
0.020
-0.202
-0.919
-0.606
0.058
0.050
-0.681
0.355
0.323
0.127
0.441

BMA
0.409
0.158
0.284
-0.211
0.467
0.337
0.042
0.018
0.335
0.126
0.146
0.670

Factor
-0.017
0.383
0.277
0.139
0.272
-0.113
0.041
0.292
0.391
0.571
0.144
0.001

Macro
0.050
0.262
0.422
-0.009
0.012
0.066
0.039
0.030
0.352
0.283
0.347
0.001

DRS
0.972
1.458
0.544
0.181
0.281
0.755
-0.031
0.581
0.712
0.807
0.622
1.564

(Bianchi and McAlinn 2020, p. 58)

Kernel methods¶

Recall that we can sometimes we can fit to non-linear data using basis functions.

We use a set of functions  that increases the number of the features, e.g.:

\begin{equation}
\{ 1, x_1, x_2, x_3, x_1x_2, x_1x_3, x_2 x_3, x_1^2, x_2^2, x_3^2 \}
\end{equation}

Our set of basis functions is $ \{ \phi_m(\mathbf{x})\}_{m=1}^{M_\phi} $

\begin{eqnarray*}
\phi_1(\mathbf{x}) & = & 1 \\
\phi_2(\mathbf{x}) & = & x_1 \\
\ldots \\
\phi_6(\mathbf{x}) & = & x_1 x_3 \\
\ldots \\
\end{eqnarray*}

Model¶
\begin{equation}
F_{\mathbf{w}}(\mathbf{x}) = \sum_{m=1}^{M_\phi} w_m \phi_m(\mathbf{x})
\end{equation}

Features¶
\begin{equation}
\mathbf{\Phi} =
\begin{pmatrix}
\phi_1(\mathbf{x}_1) & \cdots & \phi_{M_{\phi}}(\mathbf{x}_1) \\
\vdots & \vdots & \vdots \\
\phi_1(\mathbf{x}_n) & \cdots & \phi_{M_{\phi}}(\mathbf{x}_n) \\
\end{pmatrix}
\end{equation}

Basis vector¶
\begin{equation}
\phi(\mathbf{x}) = (\phi_1(\mathbf{x}), \phi_2(\mathbf{x}), \ldots , \phi_{M_{\phi}} )
\end{equation}

Vectorized model¶
\begin{equation}
F_{\mathbf{w}}(\mathbf{x}) = \mathbf{\Phi} \mathbf{w}
\end{equation}

Penalised least-squares with basis functions¶

Penalised loss function:

\begin{eqnarray*}
L(\mathbf{w}) & = & \sum_{i=1}^n \left( y_i - \sum_{m=1}^{M_\phi} w_m \phi_m( \mathbf{x}_i ) \right) ^ 2 + \frac{\lambda}{2} \sum_{m=1}^{M_\phi} w^2_m \\
              & = & (\mathbf{y} - \mathbf{\Phi} \mathbf{w})^T (\mathbf{y} - \mathbf{\Phi} \mathbf{w}) + \frac{\lambda}{2} \mathbf{w}^T \mathbf{w}
\end{eqnarray*}

Solution

\begin{equation}
\mathbf{w}^* = (\mathbf{\Phi}^T \mathbf{\Phi} + \lambda I)^{-1} \mathbf{\Phi}^T \mathbf{y}
\end{equation}

Kernelization¶
\begin{eqnarray}
(\mathbf{\Phi^T} \mathbf{\Phi} + \lambda I) \mathbf{w}^* & = &\mathbf{\Phi}^T \mathbf{y} \\
\lambda \mathbf{w}^* & = & \mathbf{\Phi}^T \mathbf{y} - \mathbf{\Phi}^T \mathbf{\Phi} \mathbf{w}^* \\
        & = & \mathbf{\Phi}^T ( y - \mathbf{\Phi} \mathbf{w}^* ) \\
\mathbf{w}^* & = & \lambda^{-1}\mathbf{\Phi}^T(y - \mathbf{\Phi} \mathbf{w}^* ) \\
    & = & \mathbf{\Phi}^T \mathbf{\alpha} \\
\end{eqnarray}
where $$\\\alpha_i = \lambda^{-1} \left[ y_i - \mathbf{w}^T \phi(\mathbf{x_i})\right]$$

\begin{eqnarray}
\lambda \mathbf{\alpha} & = & \mathbf{y} - \mathbf{\Phi} \mathbf{w}^* \\
    & = & \mathbf{y} - \mathbf{\Phi} \mathbf{\Phi}^T \alpha \\
\mathbf{\Phi} \mathbf{\Phi}^T \alpha + \lambda \alpha & = & \mathbf{y} \\
\alpha & = & (\mathbf{\Phi} \mathbf{\Phi}^T + \lambda I)^{-1} \mathbf{y} \\
    & = & (K + \lambda I) ^ {-1} \mathbf{y} \\
\end{eqnarray}

Kernel matrix¶
\begin{equation}
\mathbf{\Phi}  \mathbf{\Phi}^T = \mathbf{K} =
\begin{pmatrix}
\phi(\mathbf{x}_1)^T \phi(\mathbf{x}_1) & \phi(\mathbf{x}_1)^T \phi(\mathbf{x}_2) & \cdots & \phi(\mathbf{x}_1)^T \phi(\mathbf{x}_n) \\
\phi(\mathbf{x}_2)^T \phi(\mathbf{x}_1) & \phi(\mathbf{x}_2)^T \phi(\mathbf{x}_2) & \cdots & \phi(\mathbf{x}_2)^T \phi(\mathbf{x}_n) \\
\vdots & \vdots & \vdots & \vdots \\
\phi(\mathbf{x}_n)^T \phi(\mathbf{x}_n) & \phi(\mathbf{x}_n)^T \phi(\mathbf{x}_2) & \cdots & \phi(\mathbf{x}_n)^T \phi(\mathbf{x}_n) \\
\end{pmatrix}
\end{equation}

Kernelization¶

The non-kernelized optimal weight vector is:

\begin{equation}
\mathbf{w}^* = (\mathbf{\Phi}^T \mathbf{\Phi} + \lambda I)^{-1} \mathbf{\Phi}^T \mathbf{y}
\end{equation}

In contrast the kernelized version is:

\begin{equation}
\mathbf{w}^* = \mathbf{\Phi}^T \mathbf{\alpha} = \mathbf{\Phi}^T (\mathbf{K} + \lambda I)^{-1} \mathbf{y}
\end{equation}

Kernalized predictions¶
\begin{eqnarray}
\mathbf{F} & = & \mathbf{\Phi} \mathbf{w^*} \\
\mathbf{F}^T & = & \mathbf{w^*}^T \mathbf{\Phi}^T \\
                   & = & \mathbf{y}^T( \mathbf{\Phi} \mathbf{\Phi}^T + \lambda I)^{-1} \mathbf{\Phi^T} \mathbf{\Phi} \\
                   & = & \mathbf{y}^T(\mathbf{K} + \lambda I)^{-1} \mathbf{K}
\end{eqnarray}

We do not require $\mathbf{\Phi}$.

Predictions can be made using only the $n \times n$ kernel matrix K.

Kernel in terms of features¶
\begin{equation}
\mathbf{K}  =
\begin{pmatrix}
\phi(\mathbf{x}_1)^T \phi(\mathbf{x}_1) & \phi(\mathbf{x}_1)^T \phi(\mathbf{x}_2) & \cdots & \phi(\mathbf{x}_1)^T \phi(\mathbf{x}_n) \\
\phi(\mathbf{x}_2)^T \phi(\mathbf{x}_1) & \phi(\mathbf{x}_2)^T \phi(\mathbf{x}_2) & \cdots & \phi(\mathbf{x}_2)^T \phi(\mathbf{x}_n) \\
\vdots & \vdots & \vdots & \vdots \\
\phi(\mathbf{x}_n)^T \phi(\mathbf{x}_n) & \phi(\mathbf{x}_n)^T \phi(\mathbf{x}_2) & \cdots & \phi(\mathbf{x}_n)^T \phi(\mathbf{x}_n) \\
\end{pmatrix}
\end{equation}

Using a kernel function¶
\begin{equation}
\mathbf{K} =
\begin{pmatrix}
\operatorname{k}(\mathbf{x}_1, \mathbf{x}_1) & \operatorname{k}(\mathbf{x}_1, \mathbf{x}_2) & \cdots & \operatorname{k}(\mathbf{x}_1, \mathbf{x}_n) \\
\operatorname{k}(\mathbf{x}_2, \mathbf{x}_1) & \operatorname{k}(\mathbf{x}_2, \mathbf{x}_2) & \cdots & \operatorname{k}(\mathbf{x}_2, \mathbf{x}_n) \\
\vdots & \vdots & \vdots & \vdots \\
\operatorname{k}(\mathbf{x}_n, \mathbf{x}_1) & \operatorname{k}(\mathbf{x}_n, \mathbf{x}_2) & \cdots & \operatorname{k}(\mathbf{x}_n, \mathbf{x}_n) \\
\end{pmatrix}
\end{equation}

where $\operatorname{k}$ is a symmetric function such that $K$ is positive semi-definite $\mathbf{x}^T \mathbf{K} \mathbf{x} \geq 0 \; \forall \mathbf{x}$.

The kernel trick¶

The linear kernel is simply the inner product:

$\operatorname{k}(\mathbf{x}_i, \mathbf{x}_j) = \mathbf{x}_i ^T \cdot \mathbf{y}_j$

In this case our basis mapping is simply the identity function.

However, there are generalizations of the inner product called kernel functions.

Every kernel function defines an implicit set of basis functions.

Example¶

Consider an original feature space with $p=2$ dimensions, and with basis functions:

\begin{eqnarray*}
\phi_1(\mathbf{x}) & = & x_1^2 \\
\phi_2(\mathbf{x}) & = & \sqrt{2} x_1 x_2 \\
\phi_3(\mathbf{x}) & = & x_2^2 \\
\end{eqnarray*}\begin{equation*}
\phi(\mathbf{x}) = (x_1^2, \sqrt{2} x_1 x_2, x_2^2)
\end{equation*}

The polynomial kernel¶
\begin{eqnarray*}
\phi(\mathbf{x}_i)^T \phi(\mathbf{x}_j) & = & x_{i1}^2 x_{j1}^2 + 2x_{i1}^2 x_{j1}^2 + x_{i2}^2 x_{j2}^2 \\
 & = & (x_{i1}x_{j1} + x_{i2} x_{j2})^2 \\
 & = &  (\mathbf{x}^T_i \mathbf{x}_j)^2 \\
 & = & \operatorname{k}(\mathbf{x}_i, \mathbf{x}_j)
\end{eqnarray*}

Our kernel matrix is defined only in terms of $\operatorname{k}$.

Using this kernel function we can work in the original 2-dimensional feature space instead of the 3-dimensional mapped feature space.

This is an example of a Polynomial kernel:

\begin{equation}
\operatorname{k}(\mathbf{x}_i, \mathbf{x}_j) = (\mathbf{x}^T_i \mathbf{x}_j + r)^d
\end{equation}

For other kernels see Murphy (2012), ch. 14.

Out of sample forecasts¶

For an out of sample observation $\mathbf{x}_j$:

\begin{eqnarray}
\hat{F} & = & \mathbf{w}^{*^T} \phi(\mathbf{x}_j) \\
    & = & \sum_i^n \alpha_i \phi^T(\mathbf{x}_i) \phi(\mathbf{x}_j) \\
    & = & \sum_i^n \alpha_i k(\mathbf{x}_i, \mathbf{x}_j)
\end{eqnarray}

Classification methods¶

In a classification problem we attempt to map quantitative data onto categorical variables.

Categorical variables are also called qualitative variables.

They take one of a discrete set of categories.

Examples:

Given data on a banking transaction, is it a) fraudulent, or b) legitimate?

Given a credit-history, is the applicant a) high-risk, or b) low-risk?

Given the state of the order-book, will the mid-price move a) up, or b) down.

Classification methods in Finance¶

Business failure prediction

Credit risk assessment

Corporate mergers and acquisitions

Stock rating models

Bond rating models

Price movement prediction

Example features¶

$x_1$ Total assets

$x_2$ Total debt

$x_3$ Long-term debt / total invested capital

$x_4$ Debt ratio

$\ldots$

Example categories¶

Ratings: AAA, AA, A, BBB, BB, B, $\ldots$

Validating classifiers¶

Because data are qualitative as opposed to quantitative, we cannot use the same metrics, such as $R^2$ or out-of-sample $RSS$ that we used for regression methods.

We explicitly count false positives ($FP$), false negatives ($FN$), true positives ($TP$) and true negatives ($TN$).

For example, in a credit-risk model, a false positive would occur if we mis-classify an individual as high-risk, when in fact they were low-risk.

Accuracy¶
\begin{equation}
accuracy = \frac{correct}{correct + incorrect}
\end{equation}

Precision¶
\begin{equation}
precision = \frac{TP}{TP + FP}
\end{equation}

Recall (aka sensitivity)¶
\begin{equation}
recall = \frac{TP}{TP + FN}
\end{equation}

```python
# %load ../standard_import.txt
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

import sklearn.linear_model as skl_lm
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.metrics import confusion_matrix, classification_report, precision_score
from sklearn import preprocessing
from sklearn import neighbors

import statsmodels.api as sm
import statsmodels.formula.api as smf

%matplotlib inline
plt.style.use('seaborn-white')
```

Support Vector Machines¶

Support vector machines (SVMs) can be used to classify data into one of two classes (labels).

It is a supervised learning algorithm, so we start with known labels in our training data: for every training input $\mathbf{x}_i \in X$ we have a known label $y_i \in {+1, -1}$.

The task of the learning algorithm is to generalize to unseen cases not in our training set $X$.

The output from the classification is a definitive and non-probabilistic.

As in regression, we make use of a linear algebra to model the problem.

Non-linear problems can be solved using kernel functions.

scikit-learn¶

We will first illustrate the problem with some plots and solutions and then move to the math.

We begin with the standard imports:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
import sklearn

# use seaborn plotting defaults
import seaborn as sns; sns.set()
```

Example classification problem¶

We first consider some training data (in this case we use simulated data).

Each input $\mathbf{x}_i$ has two features $p=2$, so we can plot the inputs in a two-dimensional plane.

We color-code the associated class red for $y_i = 1$ and yellow for $y_i = -1$.

```python
from sklearn.datasets import samples_generator
X, y = samples_generator.make_blobs(n_samples=50, centers=2,
                  random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn');
```

Linear separability¶

We attempt to find a hyper-plane that cleanly separates the two classes.

\begin{equation}
\beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_p X_p = 0
\end{equation}

The hyper-plane has dimensions one less than the original feature space.

For two-dimensional data the hyper-plane is therefore simply a line:

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')

plt.plot(xfit, 0.5 * xfit + 1.6, '-k')

plt.xlim(-1, 3.5);
```

Classification using hyperplanes¶

We can classify the data according to which side of the hyper-plan a data-point lies on.

\begin{eqnarray}
\beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_p X_p & > & 0 \\
\beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_p X_p & < & 0 \\
\end{eqnarray}

That is, for a test observation $\mathbf{x}_j$ with $p$ features we can make predictions by examining the sign of:

\begin{equation}
f(\mathbf{x}) = \beta_0 + \beta_1 x_{j1} + \beta_2 x_{j2} + \ldots + \beta_p x_{jp}
\end{equation}

Model selection¶

The problem of finding a separating hyper-plane does not have a unique solution.

Going back to our previous example, there are three very different separators which perfectly discriminate between these samples.

Model selection illustrated¶

The point marked with a red cross represents out-of-sample data whose true label is $+1$.

However, depending on which model we choose, this observations will be assigned a different label.

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plt.plot([0.6], [2.1], 'x', color='red', markeredgewidth=2, markersize=10)

for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-k')

plt.xlim(-1, 3.5);
```

The Margin¶

Given a particular hyper-plane, we calculate a margin $M$ which is the distance from the hyper-plane to the nearest training input.

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')

for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-k')
    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',
                     color='#AAAAAA', alpha=0.4)

plt.xlim(-1, 3.5);
```

Maximising the margin¶

We can use $M$ to translate the classification into a loss-minimization problem with a unique solution.

First we will vectorize our equation for the hyperplane:

\begin{equation}
\mathbf{w} \cdot \mathbf{x} + b = 0
\end{equation}

where the vector $\mathbf{w}$ is perpendicular to the plane.

For an observation $\mathbf{x}_i$ with label $y_i$, the distance to the hyperplane is:

\begin{equation}
D_i =   \frac{|\mathbf{w} \cdot \mathbf{x}_i + b|}{||\mathbf{w}||_2}
\end{equation}

The margin is the distance to the closest observation:

\begin{equation}
M = \min_i D_i
\end{equation}

The optimal model is the hyper-plane that gives us the best separation, i.e. the maximum margin:

\begin{equation}
\mathbf{w}^* = \max_{\mathbf{w},b} M
\end{equation}

Predictions¶

We can make predictions by examining the sign of $f(\mathbf{x}) = \mathbf{w}^T \cdot \mathbf{x} + b$.

To test against training data $\mathbf{y}$ we can examine the sign of the product $y_i f(\mathbf{x}_i)$.

Therefore the constraint that the classification is correct can be expressed:

\begin{equation}
y_i[\mathbf{w}^T \cdot \mathbf{x} + b] \geq 0
\end{equation}

Using Scikit-learn to fit a linear support vector machine¶

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear', C=1E10)
model.fit(X, y)
```

    SVC(C=10000000000.0, kernel='linear')

The fitted model¶

```python
def plot_svc_decision_function(model, ax=None, plot_support=True):
    """Plot the decision function for a 2D SVC"""
    if ax is None:
        ax = plt.gca()
    xlim = ax.get_xlim()
    ylim = ax.get_ylim()

    # create grid to evaluate model
    x = np.linspace(xlim[0], xlim[1], 30)
    y = np.linspace(ylim[0], ylim[1], 30)
    Y, X = np.meshgrid(y, x)
    xy = np.vstack([X.ravel(), Y.ravel()]).T
    P = model.decision_function(xy).reshape(X.shape)

    # plot decision boundary and margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

    # plot support vectors
    if plot_support:
        ax.scatter(model.support_vectors_[:, 0],
                   model.support_vectors_[:, 1],
                   s=300, linewidth=1, facecolors='none');
    ax.set_xlim(xlim)
    ax.set_ylim(ylim)
```

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(model, plot_support=True);
```

This is the dividing line that maximizes the margin between the two sets of points.

Notice that a few of the training points just touch the margin.

These points are are known as the support vectors.

Support-vectors¶

In Scikit-Learn, the identity of these points are stored in the support_vectors_ attribute of the classifier:

```python
model.support_vectors_
```

    array([[0.44359863, 3.11530945],
           [2.33812285, 3.43116792],
           [2.06156753, 1.96918596]])

Only the position of the support vectors matter

Any points further from the margin which are on the correct side do not modify the fit.

These points do not contribute to the loss function used to fit the model, so their position and number do not matter so long as they do not cross the margin.

We can see this, for example, if we plot the model learned from the first 60 points and first 120 points of this dataset:

Support-vector machine fit with different training sets¶

```python
def plot_svm(N=10, ax=None):
    X, y = samples_generator.make_blobs(n_samples=200, centers=2,
                      random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    model = SVC(kernel='linear', C=1E10)
    model.fit(X, y)

    ax = ax or plt.gca()
    ax.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
    ax.set_xlim(-1, 4)
    ax.set_ylim(-1, 6)
    plot_svc_decision_function(model, ax)

fig, ax = plt.subplots(1, 2, figsize=(16, 6))
fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
for axi, N in zip(ax, [60, 120]):
    plot_svm(N, axi)
    axi.set_title('N = {0}'.format(N))
```

Overlapping data¶

Consider a data set that has some overlap, e.g.:

```python
X, y = samples_generator.make_blobs(n_samples=100, centers=2,
                  random_state=0, cluster_std=1.2)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn');
```

Soft margins¶

We introduce a non-negative hyper-parameter $C$ which "softens" the margin, and our constraint becomes:

\begin{equation}
y_i (B_0 + B_1 x_{i1} + \beta_2 x_{i2} + \ldots + \beta_p x_{ip})) \geq M(1 - \xi_i) \; \forall i \in \{1, \ldots, n\},
\end{equation}\begin{equation}
\xi_i \geq 0, \;  \sum_{i=1}^n \xi_i \leq C.
\end{equation}

Softening the margins¶

```python
X, y = samples_generator.make_blobs(n_samples=100, centers=2,
                  random_state=0, cluster_std=0.8)

fig, ax = plt.subplots(1, 2, figsize=(16, 6))
fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)

for axi, C in zip(ax, [10.0, 0.1]):
    model = SVC(kernel='linear', C=C).fit(X, y)
    axi.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
    plot_svc_decision_function(model, axi)
    axi.scatter(model.support_vectors_[:, 0],
                model.support_vectors_[:, 1],
                s=300, lw=1, facecolors='none');
    axi.set_title('C = {0:.1f}'.format(C), size=14)
```

SVM as constrained optimization¶

Define the functional margin:

\begin{equation}
M = \min_i  y_i \left( \frac{\mathbf{w}}{ ||\mathbf{w}|| } \cdot \mathbf{x}_i + \frac{b}{||\mathbf{w}||} \right)
\end{equation}

We want to solve

\begin{equation}
\max_{\mathbf{w}, b} M
\end{equation}
subject to:

\begin{equation}
y_i \left( \frac{\mathbf{w}}{ ||\mathbf{w}|| } \cdot \mathbf{x}_i + \frac{b}{||\mathbf{w}||} \right) \geq M \; \forall i
\end{equation}

Eliminating $M$¶

Define

\begin{equation}
F = \min_i y_i ( \mathbf{w} \cdot \mathbf{x}_i + b )
\end{equation}
Then the above constraint can be written:

\begin{equation}
y_i ( \mathbf{w} \cdot \mathbf{x}_i + b ) \geq F \; \forall i
\end{equation}

We can normalize by rescaling $\mathbf{w}$ so that $F=1$ which gives us $M = \frac{1}{||\mathbf{w}||}$.

Reformulating as a quadratic¶

Our optimization problem is now:

\begin{equation}
\max_{\mathbf{w},b} \frac{1}{||\mathbf{w}||}
\end{equation}
subject to:

\begin{equation}
y_i(\mathbf{w} \cdot x + b) - 1 \geq 0 \; \forall i
\end{equation}

This can be written as a quadratic optimization problem:

\begin{equation}
\min_{\mathbf{w},b} \frac{1}{2} ||\mathbf{w}||^2
\end{equation}
subject to the same constraint:

\begin{equation}
y_i(\mathbf{w} \cdot x + b) - 1 \geq 0 \; \forall i
\end{equation}

Softening the margins¶
\begin{equation}
\min_{\mathbf{w},b} \frac{1}{2} ||\mathbf{w}||^2 + \frac{C}{n} \sum_i^n \xi_i
\end{equation}
subject to constraints:

\begin{eqnarray*}
y_i(\mathbf{w} \cdot x + b) & \geq & 1 - \xi_i \; \forall i \\
\xi_i & \geq & 0 \; \forall i \\
\end{eqnarray*}

Standard form¶
\begin{equation}
\min_{\mathbf{w},b} \frac{1}{2} ||\mathbf{w}||^2 + C \sum_i^n \xi_i
\end{equation}
subject to constraints:

\begin{eqnarray*}
1 - \xi_i - y_i(\mathbf{w} \cdot x + b) & \leq & 0 \; \forall i \\
- \xi_i & \leq & 0 \; \forall i \\
\end{eqnarray*}

KKT conditions¶

\begin{equation}
\nabla f(\mathbf{x}^*) + \sum_{i=1}^m \lambda_i^* \nabla g_i(\mathbf{x}^*) + \sum_{i=1}^p v_i^* \nabla h_i(\mathbf{x}^*) = 0
\end{equation}

\begin{equation}
g_i(\mathbf{x}^*) \leq 0, \; i=1, \ldots, m
\end{equation}\begin{equation}
h_i(\mathbf{x}^*) = 0, \; \; i=1, \ldots, p
\end{equation}\begin{equation}
\lambda_i^* \geq 0, \; i=1, \ldots, m
\end{equation}\begin{equation}
\lambda_i^* g_i(\mathbf{x}^*) = 0, \; i=1, \ldots, m
\end{equation}

Lagrangian¶

Allocating Lagrange multipliers $\mathbf{\alpha}, \mathbf{\beta}$:

\begin{eqnarray}
L(\mathbf{w}, b, \mathbf{\xi}, \mathbf{\alpha}, \mathbf{\beta})
& = & \frac{1}{2}\mathbf{w}^T\mathbf{w} + \frac{C}{n} \sum_{i=1} \xi_i  + \sum_{i=1}^n \alpha_i (1 - \xi_i - y_i(\mathbf{w}^T x_i + b)) - \sum_{i=1}^n \beta_i \xi_i \\
\end{eqnarray}

Lagrangian dual¶
\begin{equation}
L_{D}(\mathbf{\alpha}, \mathbf{\beta}) = \min_{\mathbf{w}, b, \xi_i} L(\mathbf{w}, b, \mathbf{\xi}, \mathbf{\alpha}, \mathbf{\beta})
\end{equation}
The dual optimization problem is:

\begin{equation}
\max_{\mathbf{w}, b, \xi_i} L_D(\mathbf{\alpha}, \mathbf{\beta})
\end{equation}

KTT conditions¶
\begin{eqnarray}
L(\mathbf{w}, b, \mathbf{\xi}, \mathbf{\alpha}, \mathbf{\beta})
& = & \frac{1}{2}\mathbf{w}^T\mathbf{w} + \frac{C}{n} \sum_{i=1} \xi_i  + \sum_{i=1}^n \alpha_i (1 - \xi_i - y_i(\mathbf{w}^T x_i + b)) - \sum_{i=1}^n \beta_i \xi_i \\
\end{eqnarray}\begin{equation}
\nabla_\mathbf{w} L(\mathbf{w}, b, \mathbf{\xi}, \mathbf{\alpha}, \mathbf{\beta}) = \mathbf{w} - \sum_{i=1}^n \alpha_i y_i \mathbf{x}_i = 0 \\
\end{equation}\begin{equation}
\frac{\partial}{\partial b} L(\mathbf{w}, b, \mathbf{\xi}, \mathbf{\alpha}, \mathbf{\beta}) = - \sum_{i=1}^n \alpha_i y_i = 0
\end{equation}\begin{equation}
\frac{\partial}{\partial b} L(\mathbf{w}, b, \mathbf{\xi}, \mathbf{\alpha}, \mathbf{\beta}) = \frac{C}{n} - \alpha_i - \beta_i = 0
\end{equation}

Substituting into the dual¶
\begin{equation}
L_D(\mathbf{\alpha}, \mathbf{\beta}) = - \frac{1}{2} \sum_{i,j} \alpha_i \alpha_j y_i y_j \mathbf{x}_i ^ T \mathbf{x}_j + \sum_{i}^n \alpha_i
\end{equation}

We can now write the dual optimization problem as:

\begin{equation}
\max_{\mathbf{\alpha}, \mathbf{\beta}} - \frac{1}{2} \sum_{i, j} \alpha_i \alpha_j y_i y_j \mathbf{x}_i^T \mathbf{x}_j + \sum_{i}^n \alpha_i
\end{equation}
subject to:

\begin{eqnarray}
\sum_{i} \alpha_i y_i & = & 0 \\
\alpha_i + \beta_i & = & \frac{C}{n} \\
\alpha_i, \beta_i & \geq & 0 \; i=1, \ldots, n
\end{eqnarray}

As a quadratic programming problem¶

We an eliminate $\mathbf{\beta}$ to transform this into a quadratic programming problem:

\begin{equation}
\max_{\mathbf{\alpha}} - \frac{1}{2} \sum_{i,j} \alpha_i \alpha_j y_i y_j \mathbf{x}_i^T \mathbf{x}_j + \sum_{i} \alpha_i
\end{equation}
subject to:

\begin{eqnarray}
\sum_i \alpha_i y_i & = & 0 \\
0 \leq \alpha_i \leq \frac{C}{n} \; i=1,\ldots,n \\
\end{eqnarray}

This can be solved numerically using a quadratic optimizer to obtain $\mathbf{\alpha}^*$.

Optimal weights¶

From KKT condition 1:

\begin{equation}
\mathbf{w}* - \sum_{i=1}^n \alpha_i^* y_i \mathbf{x}_i = 0
\end{equation}
Thus:

\begin{equation}
\mathbf{w}* = \sum_{i=1} \alpha_i ^ * y_i \mathbf{x}_i
\end{equation}

Support vectors¶

From KKT condition 5:

\begin{equation}
\alpha_i ^ * ( 1 - \xi_i^* - y_i ( \mathbf{w}^{*^T} \mathbf{x}_i + b^*)) = 0
\end{equation}

The data that satisfy these constraints are the support vectors.

By the KKT conditions, $\alpha_i^* \neq 0$ i.f.f. $\mathbf{x}_i$ is a support vector.

In typical learning problems only a small fraction of the training inputs are support vectors $S$.

Our classifier has a compact representation:

\begin{equation}
F(\mathbf{x}_j) = \operatorname{sign}( \sum_i \alpha_i^* y_i \mathbf{x}_j ^ T \mathbf{x}_i + b^*)
\end{equation}

Non linearly-seperable data¶

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(clf, plot_support=False);
```

Radial basis functions¶

We can project the data into higher dimensions to obtain linear-separability.

For example, we can use a radial basis function:

```python
r = np.exp(-(X ** 2).sum(1))
```

Projection into three dimensions¶

```python
ax = plt.subplot(projection='3d')
ax.scatter3D(X[:, 0], X[:, 1], r, c=y, cmap='autumn')
plt.show()
```

Inner products¶

The solution to the linear support-vector classifier optimization problem involves only the inner products of the observations:

\begin{equation}
\langle x_i, x_{i'} \rangle = \sum_{j=1}^p x_{ij}x_{i'j}
\end{equation}

The linear SVC can thus be represented:

\begin{equation}
f(x) = \beta_0 + \sum_{i=1}^n \alpha_i \langle x, x_i \rangle
\end{equation}

The kernel trick¶

A kernel is a function $\operatorname{k}(x_i, x_{i'})$ which quantifies the similarity of two observations:

The linear kernel is:

\begin{equation}
\operatorname{k}(x_i, x_{i'}) = \sum_{j=1}^p x_{ij}{x_{i'j}}
\end{equation}

If we use centered and scaled data, it is equivalent to the Pearson correlation.

We can use non-linear kernels, e.g. polynomial of degree $d$:

\begin{equation}
\operatorname{k}(x_i, x_{i'}) = (1 + \sum_{j=1}^p x_{ij} x_{i'j})^d
\end{equation}

Kernel-trick illustration¶

Consider

$$\phi(x)  = (1, \sqrt{2}x_1, \sqrt{2}x_2, x_1^2, x_2 ^ 2, \sqrt{2}x_1x_2) \\
\phi(z) = (1, \sqrt{2}z_1, \sqrt{2} z_2, z_1^2, z_2 ^ 2, \sqrt{2}z_1 z_2)$$\begin{eqnarray}
\phi(x) \cdot \phi(z) & = & 1 + 2 x_1 z_1 + 2 x_2 z_2 + x_1^2 z_1^2 + x_2 ^ 2 z_2 ^ 2 +  2 x_1 x_2 z_1 z_2 \\
& = & (1 + x_1 z_1 + x_2 z_2)^2 \\
& = & (1 + x.z)^2
\end{eqnarray}

Non-linear kernels in scikit-learn¶

To use the radial basis function kernel (RBF):

\begin{equation}
\operatorname{k}(\mathbf{x}_i, \mathbf{x}_j) = \exp \left( -\frac{||\mathbf{x}_i - \mathbf{x}_j || ^ 2}{2\sigma^2} \right)
\end{equation}

```python
clf = SVC(kernel='rbf', C=1E6)
clf.fit(X, y)
```

    SVC(C=1000000.0)

Plotting the decision boundary¶

We can plot the decision boundary in the unmapped space:

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=300, lw=1, facecolors='none');
```

Modeling high-frequency limit-order book dynamics¶

In Kercheval & Zhang (2015) the feature vector consists of the prices and volumes on each side of the order-book.

The model is trained to predict the movement of the mid-price in the subsequent time period.

Kercheval, A. N., & Zhang, Y. (2015). Modelling high-frequency limit order book dynamics with support vector machines. Quantitative Finance, 15(8), 1315-1329.

Categories¶

Mid-price will move up

Mid-price will move down

Mid-price will stay the same

Features¶

(Kercheval and Zhang 2015)

Results¶

Validation¶

The Bayes Classifier¶

The Bayes Classifier is the theoretically-ideal classifier.

We know a-priori the probability distributions for each class $P(X|Y=j)$.

We assign each observation to the most likely class, given its predictor values.

For a given data point $\mathbf{x}_i$ we choose the label $j$ for which maximises:

\begin{equation}
Pr(Y = j | X = \mathbf{x}_i)
\end{equation}
Bayes Rule¶
\begin{equation}
P(A|B) = \frac{P(B|A)}{P(A)}
\end{equation}

For a binary classifier with mutually exclusive labels $Y=0$ or $Y=1$:

\begin{eqnarray}
Pr(Y = 1|X) & = & \frac{Pr(X|Y = 1) Pr(Y=1)}{Pr(X)} \\
\end{eqnarray}

Under the assumption that predictors are independent we can use:

\begin{equation}
P(Y|X) = P(x_1 | Y) \times P(x_2|Y) \times \ldots \times P(x_n | Y) \times P(Y)
\end{equation}

The above assumption is (sometimes) "naive", and gives us the name "Naive Bayes Classifier".

The Bayes Decision Boundary¶

For a binary classification problem with mutually exclusive labels $Y=0$ or $Y=1$ we use the Bayes decision boundary:

For a binary classification task with two labels $Pr(Y=1) = 1 - Pr(Y=0)$.

Therefore the decision boundary is $\{ \mathbf{x}_i \in \mathbf{X} : P(Y=1|X=\mathbf{x}_i) > 0.5 \}$.

\begin{equation}
Pr(Y = 1 | X) > 0.5
\end{equation}

To illustrate, we first generate data using known conditional probabilities:

```python
from sklearn.datasets import make_blobs
X, y = make_blobs(30, 2, centers=2, random_state=2, cluster_std=1.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu');
```

Plotting the decision boundary¶

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_blobs
X, y = make_blobs(30, 2, centers=2, random_state=2, cluster_std=1.5)

fig, ax = plt.subplots(figsize=(8,6))
#plt.gca().set_aspect('equal', adjustable='box')
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu')

xlim = (-8, 8)
ylim = (-15, 5)

xg = np.linspace(xlim[0], xlim[1], 600)
yg = np.linspace(ylim[0], ylim[1], 400)
xx, yy = np.meshgrid(xg, yg)
Xgrid = np.vstack([xx.ravel(), yy.ravel()]).T
PL = np.zeros((2, len(Xgrid)))
for label, color in enumerate(['red', 'blue']): #, 'blue']):
    mask = (y == label)
    mu, std = X[mask].mean(0), X[mask].std(0)
    P = np.exp(-0.5 * (Xgrid - mu) ** 2 / std ** 2).prod(1)
    PL[label, :] = P[:]
    Pm = np.ma.masked_array(P, P < 0.03)
    ax.pcolorfast(xg, yg, Pm.reshape(xx.shape), alpha=0.8,
                  cmap=color.title() + 's')
    ax.contour(xx, yy, P.reshape(xx.shape),
               levels=[0.01, 0.2, 0.5],
               colors=color, alpha=0.2)

conditional =  (PL[1, :] / PL[0, :])
plt.contour(xx, yy, conditional.reshape(xx.shape), levels=[0.5])

#ax.set(xlim=xlim, ylim=ylim)
plt.show()
```

Naive Gaussian-Bayes Classifier¶

We can make assumptions about the model for each label.

In a Gaussian Naive Bayes classification we assume that data for each feature are i.i.d. distributed from a Gaussian distribution:

\begin{equation}
P(x_i | y) = \frac{1}{\sqrt{2\pi\sigma_y^2}} \exp \left( - \frac{(x_i - \mu_y)^2}{2 \sigma^2{y}} \right)
\end{equation}

The mean and variance can be estimated from sample mean and sample variance.

In scikit-learn:

```python
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X, y);
```

Predictions¶

We can now predict the label for new observations:

```python
rng = np.random.RandomState(0)
Xnew = [-6, -14] + [14, 18] * rng.rand(2000, 2)
ynew = model.predict(Xnew)
```

Decision boundary¶

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu')
lim = plt.axis()
plt.scatter(Xnew[:, 0], Xnew[:, 1], c=ynew, s=20, cmap='RdBu', alpha=0.1)
plt.axis(lim);
```

Predicted posterior probabilities¶

```python
yprob = model.predict_proba(Xnew)
yprob[-8:].round(2)
```

    array([[1.  , 0.  ],
           [1.  , 0.  ],
           [1.  , 0.  ],
           [1.  , 0.  ],
           [1.  , 0.  ],
           [1.  , 0.  ],
           [0.  , 1.  ],
           [0.98, 0.02]])

K-Nearest Neighbors¶

In general we do not know a-priori the actual conditional probabilities.

Therefore we have to estimate them.

With K-nearest neighbors (KNN) we first identify the $K$ points in the training data that are closest to a given training data $x_0$, e.g. using an Euclidian distance metric.

We then estimate the conditional probability as a sample mean:

\begin{equation}
Pr(Y = j| X = x_0) = \frac{1}{K} \sum_{i \in N_0} I(y_i = j)
\end{equation}

KNN example for $K=3$¶

```python
fig, ax = plt.subplots(figsize=(6, 8))
ax.scatter(X[:, 0], X[:, 1], c=y, cmap='RdBu')
test_x = np.array([0, -5])
ax.plot(test_x[0], test_x[1], 'x', color='r')
plt.gca().set_aspect('equal', adjustable='box')
K = 3
distances = np.sum((test_x - X)**2, axis=1)
N = np.argsort(distances)[:K]
max_dist = np.sqrt(np.sort(distances)[K-1])
#plt.plot(X[N][:, 0], X[N][:, 1], 'x')
max_dist
ax.add_artist(plt.Circle(test_x, max_dist+0.2, alpha=0.5))
plt.show()
```

Logistic regression¶

So far we have modelled dependent variables as deterministic quantities.

In many applications we want to forecast probabilities, e.g. probability of default.

Credit data set¶

```python
df = pd.read_excel('data/Default.xlsx')

df['default2'] = df.default.factorize()[0]
df['student2'] = df.student.factorize()[0]
df[['default', 'student', 'balance', 'income']].head(5)
```

```python
pip install xgboost
```

    Collecting xgboost
      Downloading xgboost-1.2.1-py3-none-macosx_10_13_x86_64.macosx_10_14_x86_64.macosx_10_15_x86_64.whl (1.2 MB)
         |████████████████████████████████| 1.2 MB 5.7 MB/s eta 0:00:01
    Requirement already satisfied: scipy in /Users/aq/opt/anaconda3/lib/python3.7/site-packages (from xgboost) (1.5.2)
    Requirement already satisfied: numpy in /Users/aq/opt/anaconda3/lib/python3.7/site-packages (from xgboost) (1.19.1)
    Installing collected packages: xgboost
    Successfully installed xgboost-1.2.1
    Note: you may need to restart the kernel to use updated packages.

```python
pip install graphviz
```

    Collecting graphviz
      Downloading graphviz-0.14.2-py2.py3-none-any.whl (18 kB)
    Installing collected packages: graphviz
    Successfully installed graphviz-0.14.2
    Note: you may need to restart the kernel to use updated packages.

Credit data-set: visualisation¶

```python
fig = plt.figure(figsize=(12,5))
gs = mpl.gridspec.GridSpec(1, 4)
ax1 = plt.subplot(gs[0,:-2])
ax2 = plt.subplot(gs[0,-2])
ax3 = plt.subplot(gs[0,-1])

# Take a fraction of the samples where target value (default) is 'no'
df_no = df[df.default2 == 0].sample(frac=0.15)
# Take all samples  where target value is 'yes'
df_yes = df[df.default2 == 1]
df_ = df_no.append(df_yes)

ax1.scatter(df_[df_.default == 'Yes'].balance, df_[df_.default == 'Yes'].income, s=40, c='orange', marker='+',
            linewidths=1)
ax1.scatter(df_[df_.default == 'No'].balance, df_[df_.default == 'No'].income, s=40, marker='o', linewidths='1',
            edgecolors='lightblue', facecolors='white', alpha=.6)

ax1.set_ylim(ymin=0)
ax1.set_ylabel('Income')
ax1.set_xlim(xmin=-100)
ax1.set_xlabel('Balance')

c_palette = {'No':'lightblue', 'Yes':'orange'}
sns.boxplot('default', 'balance', data=df, orient='v', ax=ax2, palette=c_palette)
sns.boxplot('default', 'income', data=df, orient='v', ax=ax3, palette=c_palette)
gs.tight_layout(plt.gcf())
```

Dummy variables and probabilities¶

We could map labels onto integers, but we have to take great care as labels have no meaningful intrinsic ordering.

For a binary classifier with only two labels we can use dummy variables by transforming the labels onto 0 and 1.

We can straightforwardly use dummy-variables for predictors.

We often try to interpret the real-valued response as a probability.

Probabilities are not linear in predictors¶

Naively, we might attempt to fit a model of the form:

$$p(X) = \beta_0 + \beta_1 X$$

However, this will give nonsensical results.

```python
X_train = df.balance.values.reshape(-1,1)
y = df.default2

X_test = np.arange(df.balance.min(), df.balance.max()).reshape(-1,1)

clf = skl_lm.LogisticRegression(solver='newton-cg')
clf.fit(X_train,y)
prob = clf.predict_proba(X_test)

sns.regplot(df.balance, df.default2, order=1, ci=None,
            scatter_kws={'color':'orange'},
            line_kws={'color':'lightblue', 'lw':2}) #, ax=ax1)
plt.show()
```

Logistic regression assumptions¶

We use a sigmoid function:

\begin{equation}
\sigma(z) = \frac{1}{1 + e^{-z}}
\end{equation}

for a single training pair $(\mathbf{x}, y)$, we assume:

\begin{equation}
P(Y=1 | \mathbf{X}=\mathbf{x}) = \sigma(z)
\end{equation}
where:

\begin{equation}
z = w_0 + \sum_{i=1}^m w_i x_i
\end{equation}

Vectorized conditional probabilities¶
\begin{eqnarray}
P(Y=1 | \mathbf{X} = \mathbf{x}) & = & \sigma(\mathbf{w}^T \mathbf{x}) \\
P(Y=0 | \mathbf{X} = \mathbf{x}) & = & 1 - \sigma(\mathbf{w}^T \mathbf{x}) \\
\end{eqnarray}
If the labels are 0 and 1, then $Y$ is a Bernoulli random variable $Y \sim Ber(p)$ where $p = \sigma(\mathbf{w}^T \mathbf{x})$, therefore:

\begin{equation}
P(Y = y| X = \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x})^y \cdot \left[1 - \sigma(\mathbf{w}^T \mathbf{x}) \right] ^{(1 - y)}
\end{equation}

Likelihood function¶
\begin{eqnarray*}
L(\mathbf{w}) & = & \prod_{i=1}^n P(Y = y_i | X = \mathbf{x}_i) \\
L(\mathbf{w}) & = & \prod_{i=1}^n \sigma(\mathbf{w}^T \mathbf{x}_i)^y \cdot \left[1 - \sigma(\mathbf{w}^T \mathbf{x}_i) \right] ^{(1 - y)}
\end{eqnarray*}

Log-likelihood function¶
\begin{eqnarray*}
LL(\mathbf{w}) = \sum_{i=1}^n y_i \log \sigma(\mathbf{w}^T \mathbf{x}_i)^y  \log \left[1 - \sigma(\mathbf{w}^T \mathbf{x}_i) \right]
\end{eqnarray*}

To obtain optimal weights we maximise the above, which gives us the maximum likelihood estimate (MLE).

Provided that the data are not separated, this function is concave and can be solved numerically using gradient ascent.

Gradient function¶
\begin{equation}
\frac{\partial LL(\mathbf{w})}{\partial w_j} = \sum_{i=1}^n \left[ y_i - \sigma(\mathbf{w}^T \mathbf{x}_i) \right] x_{ij}
\end{equation}

Logistic regression in scikit-learn¶

```python
from sklearn.linear_model import LogisticRegression

logistic = skl_lm.LogisticRegression(solver='newton-cg')
logistic.fit(X_train, y)
print(clf)
print('classes: ',clf.classes_)
print('coefficients: ',clf.coef_)
print('intercept :', clf.intercept_)
```

Probability of default¶

```python
fig = plt.figure(figsize=(10,6))
plt.scatter(X_train, y, color='orange')
plt.plot(X_test, prob[:,1], color='lightblue')

for ax in fig.axes:
    ax.hlines(1, xmin=ax.xaxis.get_data_interval()[0],
              xmax=ax.xaxis.get_data_interval()[1], linestyles='dashed', lw=1)
    ax.hlines(0, xmin=ax.xaxis.get_data_interval()[0],
              xmax=ax.xaxis.get_data_interval()[1], linestyles='dashed', lw=1)
    ax.set_ylabel('Probability of default')
    ax.set_xlabel('Balance')
    ax.set_yticks([0, 0.25, 0.5, 0.75, 1.])
    ax.set_xlim(xmin=-100)

y = df.default2
X_train = df.balance.values.reshape(-1,1)

plt.show()
```

Bibliography¶

Bianchi, D., & McAlinn, K. (2020). Divide and Conquer: Financial Ratios and Industry Returns Predictability (No. 3136368; SSRN). https://doi.org/10.2139/ssrn.3136368

Bhattacharyya, S., Jha, S., Tharakunnel, K., & Westland, J. C. (2011). Data mining for credit card fraud: A comparative study. Decision Support Systems, 50(3), 602-613.

Doumpos, M., & Zopounidis, C. (2002). Classification problems in finance. Multicriteria Decision Aid Classification Methods, 159-224.

Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.

Gilli, M., Maringer, D., & Winker, P. (2008). Applications of heuristics in finance. In Handbook on information technology in finance (pp. 635-653). Springer, Berlin, Heidelberg.

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: Springer.

Kercheval, A. N., & Zhang, Y. (2015). Modelling high-frequency limit order book dynamics with support vector machines. Quantitative Finance, 15(8), 1315-1329.

Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press.

VanderPlas, J. (2016). Python data science handbook: Essential tools for working with data. " O'Reilly Media, Inc.".

Acknowledgements¶

Some of the materials from the above adapted for these notes under CC BY-SA 4.0.

supervised machine learning _ steve.ipynb


STAT 453: Deep Learning (Spring 2020)
Instructor: Sebastian Raschka (sraschka@wisc.edu)

Course website: http://pages.stat.wisc.edu/~sraschka/teaching/stat453-ss2020/
GitHub repository: https://github.com/rasbt/stat453-deep-learning-ss20

```python
%load_ext watermark
%watermark -a 'Sebastian Raschka' -v -p matplotlib,torch,pandas
```

# HW 2: A Single Layer Neuron with a Non-Linear Activation Function (50 pts)

# Problem Setting

Suppose you have a neuron model similar to ADALINE (discussed in class) but the activation function (which is an identity function in ADALINE) is replaced by a non-linear activation function. The figure below shows an arbitrary single-layer neuron.

- If $\sigma(z) = z$, the activation function is an identity function and the neuron represents ADALINE.
- If $\sigma(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$, the activation function is a hyperbolic tangent (tanh), which is a non-linear sigmoid function (https://en.wikipedia.org/wiki/Sigmoid_function).

As mentioned above, in this homework, we are working with a tanh activation function, which we defined as
$$
\sigma(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}},
$$

Here $z$ denotes the net input,  $z = \mathbf{w}^\top \mathbf{x} + b$ (for a single training example, we write $z^{[i]} = \mathbf{w}^\top \mathbf{x}^{[i]} + b$).

Assume now that we want learn the parameters of the neuron model for a binary classification task with class labels $y \in \{0, 1\}$ similar to ADALINE. We use the same loss function, mean squared error (MSE), as in ADALINE, during training:

$$
\mathcal{L}(\mathbf{\hat{y}}, \mathbf{y}) = \frac{1}{n} \sum_{i}^{} (\hat{y}^{[i]} - y^{[i]})^2.
$$

## TASK 1: Compute the Loss Gradients with respect to the weights and bias unit  -- 25 pts

Your task is to derive the gradient of the loss $\mathcal{L}$ with respect to the weight vector and the bias unit and formulate the learning rule.

Remember that the gradient of the loss is defined as

$$
\nabla_\mathbf{w} \mathcal{L}(\mathbf{w}) =
\begin{bmatrix}
\frac{\partial \mathcal{L}(\mathbf{w})}{\partial w_1}\\
\vdots \\
\frac{\partial \mathcal{L}(\mathbf{w})}{\partial w_m}\\
\end{bmatrix}
$$

**For simplicity, it is sufficient if you write down the partial derivative and learning rule for a single weight $w_j$ and the bias unit $b$**.

To provide you with a hint, recall that we computed the partial Loss derivatives for ADALINE as follows:

$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial w_j} &= \frac{\partial}{\partial w_j} \frac{1}{n} \sum_i (\hat{y}^{[i]} - y^{[i]} )^2\\
&= \frac{\partial}{\partial w_j}  \frac{1}{n} \sum_i (\sigma(\mathbf{w}^T\mathbf{x}^{[i]} + b) - y^{[i]})^2\\
\\
&= \quad ... \\
\\
&= \sum_i \frac{2}{n}  (\sigma(\mathbf{w}^T\mathbf{x}^{[i]} + b) - y^{[i]})   x_j^{[i]}
\end{align}
$$

$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial b} &= \frac{\partial}{\partial b} \frac{1}{n}\sum_i (\hat{y}^{[i]} - y^{[i]} )^2\\
&= \frac{\partial}{\partial b}  \frac{1}{n} \sum_i (\sigma(\mathbf{w}^T\mathbf{x}^{[i]} + b) - y^{[i]})^2\\
\\
&= \quad ... \\
\\
&= \sum_i \frac{2}{n}  (\sigma(\mathbf{w}^T\mathbf{x}^{[i]} + b) - y^{[i]})
\end{align}
$$

Again, your task is to derive

- a) $$\frac{\partial \mathcal{L}}{\partial w_j}$$

and

- b) $$\frac{\partial \mathcal{L}}{\partial b}$$

for the neuron model with the non-linear activation function

$$
\sigma(\mathbf{w}^\top \mathbf{x} + b) =  \frac{e^{\mathbf{w}^\top \mathbf{x} + b} - e^{- (\mathbf{w}^\top \mathbf{x} + b)}}{e^{\mathbf{w}^\top \mathbf{x}+b} + e^{- (\mathbf{w}^\top \mathbf{x}+ b)}}
$$

For partial credits in case of a wrong solution, also write down the individual steps in the cell below.

**!!!Fill in with your solution below!!!**

$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial w_j} &= \frac{\partial}{\partial w_j} \frac{1}{n}\sum_i (\hat{y}^{[i]} - y^{[i]} )^2\\
&= ...\\
&= ...\\
&= ...\\
\end{align}
$$

**!!!Fill in with your solution below!!!**
$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial b} &= \frac{\partial}{\partial b} \frac{1}{n} \sum_i (\hat{y}^{[i]} - y^{[i]} )^2\\
&= ...\\
&= ...\\
&= ...\\
\end{align}
$$

# Coding Section

Complete the code below, the missing parts are indicated via

    # <YOUR CODE HERE>

to implement the neuron model (it is very similar to the ADALINE model we discussed in class, except the derivatives).

## Imports (Don't modify this section, just execute)

**No modification required.** You should execute this code and are encouraged to explore it further, but it is recommended to  not make any alterations here.

```python
import matplotlib.pyplot as plt
import pandas as pd
import torch
%matplotlib inline
```

## Loading the dataset (Don't modify this section, just execute)

**No modification required.** You should execute this code and are encouraged to explore it further, but it is recommended to  not make any alterations here.

```python
df = pd.read_csv('./datasets/iris.data', index_col=None, header=None)
df.columns = ['x1', 'x2', 'x3', 'x4', 'y']
df = df.iloc[50:150]
df['y'] = df['y'].apply(lambda x: 0 if x == 'Iris-versicolor' else 1)
df.tail()
```

```python
# Assign features and target

X = torch.tensor(df[['x2', 'x3']].values, dtype=torch.float)
y = torch.tensor(df['y'].values, dtype=torch.int)

# Shuffling & train/test split

torch.manual_seed(123)
shuffle_idx = torch.randperm(y.size(0), dtype=torch.long)

X, y = X[shuffle_idx], y[shuffle_idx]

percent70 = int(shuffle_idx.size(0)*0.7)

X_train, X_test = X[shuffle_idx[:percent70]], X[shuffle_idx[percent70:]]
y_train, y_test = y[shuffle_idx[:percent70]], y[shuffle_idx[percent70:]]

# Normalize (mean zero, unit variance)

mu, sigma = X_train.mean(dim=0), X_train.std(dim=0)
X_train = (X_train - mu) / sigma
X_test = (X_test - mu) / sigma
```

```python
plt.scatter(X_train[y_train == 0, 0], X_train[y_train == 0, 1], label='class 0')
plt.scatter(X_train[y_train == 1, 0], X_train[y_train == 1, 1], label='class 1')
plt.legend()
plt.show()
```

```python
plt.scatter(X_test[y_test == 0, 0], X_test[y_test == 0, 1], label='class 0')
plt.scatter(X_test[y_test == 1, 0], X_test[y_test == 1, 1], label='class 1')
plt.legend()
plt.show()
```

## TASK 2: Implement the Neuron Model (Modifications required) -- 25 pts

Your task is to complete the `backward` method to compute the gradients based on the gradients you computed in TASK1.

```python
class NeuronModel():
    def __init__(self, num_features):
        self.num_features = num_features
        self.weights = torch.zeros(num_features, 1,
                                   dtype=torch.float)
        self.bias = torch.zeros(1, dtype=torch.float)

    def activation_func(self, x):
        return (torch.exp(x) - torch.exp(-x)) / (torch.exp(x) + torch.exp(-x))

    def netinput_func(self, x, w, b):
         return torch.add(torch.mm(x, w), b)

    def forward(self, x):
        netinputs = self.netinput_func(x, self.weights, self.bias)
        activations = self.activation_func(netinputs)
        return activations.view(-1)

    def backward(self, x, yhat, y):

        # note that here, "yhat" are the "activations"
        netinputs = self.netinput_func(x, self.weights, self.bias)

        ###############################################################################
        # YOU ONLY NEED TO EDIT IN THE BOX BELOW
        ###############################################################################
        grad_loss_yhat = # < YOUR CODE>
        grad_yhat_bias = # < YOUR CODE>
        grad_yhat_weights = # < YOUR CODE>

        grad_loss_weights = # < YOUR CODE> based on grad_yhat_weights & grad_loss_yhat

        grad_loss_bias = # < YOUR CODE> based on grad_yhat_weights & grad_loss_yhat
        ################################################################################

        return (-1)*grad_loss_weights, (-1)*grad_loss_bias
```

# No modifications required beyond this point

You do not need to modify anything below. However, you should run and analyze the code to verify that your implementation of the Neuron model is likely correct.

## Training the Neuron Model (Don't modify this section, just execute)

```python
####################################################
##### Training and evaluation wrappers
###################################################

def loss(yhat, y):
    return torch.mean((yhat - y)**2)

def train(model, x, y, num_epochs,
          learning_rate=0.01, seed=123, minibatch_size=10):
    cost = []

    torch.manual_seed(seed)
    for e in range(num_epochs):

        #### Shuffle epoch
        shuffle_idx = torch.randperm(y.size(0), dtype=torch.long)
        minibatches = torch.split(shuffle_idx, minibatch_size)

        for minibatch_idx in minibatches:

            #### Compute outputs ####
            yhat = model.forward(x[minibatch_idx])

            #### Compute gradients ####
            negative_grad_w, negative_grad_b = \
                model.backward(x[minibatch_idx], yhat, y[minibatch_idx])

            #### Update weights ####
            model.weights += learning_rate * negative_grad_w
            model.bias += learning_rate * negative_grad_b

            #### Logging ####
            #minibatch_loss = loss(yhat, y[minibatch_idx])
            #print('    Minibatch MSE: %.3f' % minibatch_loss)

        #### Logging ####
        yhat = model.forward(x)
        curr_loss = loss(yhat, y)
        print('Epoch: %03d' % (e+1), end="")
        print(' | MSE: %.5f' % curr_loss)
        cost.append(curr_loss)

    return cost
```

```python
model = NeuronModel(num_features=X_train.size(1))
cost = train(model,
             X_train, y_train.float(),
             num_epochs=150,
             learning_rate=0.2,
             seed=123,
             minibatch_size=10)
```

## Evaluate the Trained Model  (Don't modify this section, just execute)

```python
plt.plot(range(len(cost)), cost)
plt.ylabel('Mean Squared Error')
plt.xlabel('Epoch')
plt.show()
```

```python
print('Weights', model.weights)
print('Bias', model.bias)
```

```python
train_pred = model.forward(X_train)
train_acc = torch.mean(
    (torch.where(train_pred > 0.5,
                 torch.tensor(1),
                 torch.tensor(0)).int() == y_train).float())

test_pred = model.forward(X_test)
test_acc = torch.mean(
    (torch.where(train_pred > 0.5,
                 torch.tensor(1),
                 torch.tensor(0)).int() == y_train).float())

print('Training Accuracy: %.2f%%' % (train_acc*100))
print('Test Accuracy: %.2f%%' % (test_acc*100))
```

## Decision Boundary  (Don't modify this section, just execute)

```python
##########################
### 2D Decision Boundary
##########################

w, b = model.weights, model.bias

x_min = -3
y_min = ( (-(w[0] * x_min) - b[0])
          / w[1] )

x_max = 3
y_max = ( (-(w[0] * x_max) - b[0])
          / w[1] )

fig, ax = plt.subplots(1, 2, sharex=True, figsize=(7, 3))

ax[0].plot([x_min, x_max], [y_min, y_max])
ax[1].plot([x_min, x_max], [y_min, y_max])

ax[0].scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], label='class 0', marker='o')
ax[0].scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], label='class 1', marker='s')

ax[1].scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], label='class 0', marker='o')
ax[1].scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], label='class 1', marker='s')

ax[1].legend(loc='upper left')
plt.show()
```