Table of Contents

1  Logistic Regression

1.1  Logistic Function

1.2  Interpreting the Intercept

1.3  Defining The Cost Function

1.4  Gradient

1.5  Stochastic/Mini-batch Gradient

1.6  Implementation

1.7  Comparing Result and Convergence Behavior

1.8  Pros and Cons of Logistic Regression

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', 'notebook_format'))

from formats import load_style
load_style(plot_style = False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import metrics
from sklearn.linear_model import LogisticRegression

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    Ethen 2019-01-06 15:25:09

    CPython 3.6.4
    IPython 6.4.0

    numpy 1.14.2
    pandas 0.23.0
    matplotlib 2.2.2
    sklearn 0.19.1

# Logistic Regression

**Logistic regression** is an excellent tool to know for classification problems, which are problems where the output value that we wish to predict only takes on only a small number of discrete values. Here we'll focus on the binary classification problem, where the output can take on only two distinct classes. To make our examples more concrete, we will consider the Glass dataset.

```python
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/glass/glass.data'
col_names = ['id', 'ri', 'na', 'mg', 'al', 'si', 'k', 'ca', 'ba', 'fe', 'glass_type']
glass = pd.read_csv(url, names = col_names, index_col = 'id')
glass.sort_values('al', inplace = True)

# convert the glass type into binary outcome
# types 1, 2, 3 are window glass
# types 5, 6, 7 are household glass
glass['household'] = glass['glass_type'].map({1: 0, 2: 0, 3: 0, 5: 1, 6: 1, 7: 1})
glass.head()
```

      ri
      na
      mg
      al
      si
      k
      ca
      ba
      fe
      glass_type
      household

      id

      22
      1.51966
      14.77
      3.75
      0.29
      72.02
      0.03
      9.00
      0.0
      0.00
      1
      0

      185
      1.51115
      17.38
      0.00
      0.34
      75.41
      0.00
      6.65
      0.0
      0.00
      6
      1

      40
      1.52213
      14.21
      3.82
      0.47
      71.77
      0.11
      9.57
      0.0
      0.00
      1
      0

      39
      1.52213
      14.21
      3.82
      0.47
      71.77
      0.11
      9.57
      0.0
      0.00
      1
      0

      51
      1.52320
      13.72
      3.72
      0.51
      71.75
      0.09
      10.06
      0.0
      0.16
      1
      0

Our task is to predict the `household` column using the `al` column. Let's visualize the relationship between the input and output and also train the logsitic regression to see the outcome that it produces.

```python
logreg = LogisticRegression(C = 1e9)
X = glass['al'].values.reshape(-1, 1) # sklearn doesn't accept 1d-array, convert it to 2d
y = np.array(glass['household'])
logreg.fit(X, y)

# predict the probability that each observation belongs to class 1
# The first column indicates the predicted probability of class 0,
# and the second column indicates the predicted probability of class 1
glass['household_pred_prob'] = logreg.predict_proba(X)[:, 1]
```

```python
# plot the predicted probability (familiarize yourself with the S-shape)
# change default figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

plt.scatter(glass['al'], glass['household'])
plt.plot(glass['al'], glass['household_pred_prob'])
plt.xlabel('al')
plt.ylabel('household')
plt.show()
```

As we can see, logistic regression can output the probabilities of observation belonging to a specific class and these probabilities can be converted into class predictions by choosing a cutoff value (e.g. probability higher than 0.5 is classified as class 1).

## Logistic Function

In **Logistic Regression**, the log-odds of a categorical response being "true" (1) is modeled as a linear combination of the features:

\begin{align*}
    \log \left({p\over 1-p}\right) &= w_0 + w_1x_1, ..., w_jx_j \nonumber \\
    &= w^Tx \nonumber
\end{align*}

Where:

- $w_{0}$ is the intercept term, and $w_1$ to $w_j$ represents the parameters for all the other features (a total of j features).
- By convention of we can assume that $x_0 = 1$, so that we can re-write the whole thing using the matrix notation $w^Tx$.

This is called the **logit function**. The equation can be re-arranged into the **logistic function**:

$$p = \frac{e^{w^Tx}} {1 + e^{w^Tx}}$$

Or in the more commonly seen form:

$$h_w(x) = \frac{1}{ 1 + e^{-w^Tx} }$$

Let's take a look at the plot of the function:

```python
x_values = np.linspace(-5, 5, 100)
y_values = [1 / (1 + np.exp(-x)) for x in x_values]
plt.plot(x_values, y_values)
plt.title('Logsitic Function')
plt.show()
```

The **logistic function** has some nice properties. The y-value represents the probability and it is always bounded between 0 and 1, which is want we wanted for probabilities. For an x value of 0 you get a 0.5 probability. Also as you get more positive x value you get a higher probability, on the other hand, a more negative x value results in a lower probability.

Toy sample code of how to predict the probability given the data and the weight is provided below.

```python
def predict_probability(data, weights):
    """probability predicted by the logistic regression"""
    score = np.dot(data, weights)
    predictions = 1 / (1 + np.exp(-score))
    return predictions
```

## Interpreting the Intercept

We can check logistic regression's coefficient does in fact generate the log-odds.

```python
# compute predicted log-odds for al = 2 using the equation
# convert log-odds to odds
# convert odds to probability
logodds = logreg.intercept_ + logreg.coef_[0] * 2
odds = np.exp(logodds)
prob = odds / (1 + odds)
print(prob)

logreg.predict_proba(2)[:, 1]
```

    [0.65638445]

    array([0.65638445])

```python
# examine the coefficient for al
print('a1', logreg.coef_[0])
```

    a1 [4.18040386]

**Interpretation:** 1 unit increase in `al` is associated with a 4.18 unit increase in the log-odds of the observation being classified as `household 1`. We can confirm that again by doing the calculation ourselves.

```python
# increasing al by 1 (so that al now becomes 3)
# increases the log-odds by 4.18
logodds = logodds + logreg.coef_[0]
odds = np.exp(logodds)
prob = odds / (1 + odds)
print(prob)

logreg.predict_proba(3)[:, 1]
```

    [0.99205808]

    array([0.99205808])

## Defining The Cost Function

When utilizing logistic regression, we are trying to learn the $w$ values in order to maximize the probability of correctly classifying our glasses. Let's say someone did give us some $w$ values of the logistic regression model, how would we determine if they were good values or not? What we would hope is that for the household of class 1, the probability values are close to 1 and for the household of class 0 the probability is close to 0.

But we don't care about getting the correct probability for just one observation, we want to correctly classify all our observations. If we assume our data are independent and identically distributed (think of it as all of them are treated equally), we can just take the product of all our individually calculated probabilities and that becomes the objective function we want to maximize. So in math:

$$\prod_{class1}h_w(x)\prod_{class0}1 - h_w(x)$$

The $\prod$ symbol means take the product of the $h_w(x)$ for the observations that are classified as that class. You will notice that for observations that are labeled as class 0, we are taking 1 minus the logistic function. That is because we are trying to find a value to maximize, and since observations that are labeled as class 0 should have a probability close to zero, 1 minus the probability should be close to 1. This procedure is also known as the **maximum likelihood estimation** and the following link contains a nice discussion of maximum likelihood using linear regression as an example. [Blog: The Principle of Maximum Likelihood](http://suriyadeepan.github.io/2017-01-22-mle-linear-regression/)

Next we will re-write the original cost function as:

$$\ell(w) = \sum_{i=1}^{N}y_{i}log(h_w(x_{i})) + (1-y_{i})log(1-h_w(x_{i}))$$

Where:

- We define $y_{i}$ to be 1 when the $i_{th}$ observation is labeled class 1 and 0 when labeled as class 0, then we only compute $h_w(x_{i})$ for observations that are labeled class 1 and $1 - h_w(x_{i})$ for observations that are labeled class 0, which is still the same idea as the original function.
- Next we'll transform the original $h_w(x_{i})$ by taking the log. As we'll later see this logarithm transformation will make our cost function more convenient to work with, and because the logarithm is a monotonically increasing function, the logarithm of a function achieves its maximum value at the same points as the function itself. When we take the log, our product across all data points, it becomes a sum. See [log rules](http://www.mathwords.com/l/logarithm_rules.htm) for more details (Hint: log(ab) = log(a) + log(b)).
- The $N$ simply represents the total number of the data.

Often times you'll also see the notation above be simplified in the form of a maximum likelihood estimator:

$$ \ell(w) = \sum_{i=1}^{N} log \big( P( y_i \mid x_i, w ) \big) $$

The equation above simply denotes the idea that , $\mathbf{w}$ represents the parameters we would like to estimate the parameters $w$ by maximizing conditional probability of $y_i$ given $x_i$.

Now by definition of probability in the logistic regression model: $h_w(x_{i}) = 1 \big/ 1 + e^{-w^T x_i}$ and $1- h_w(x_{i}) = e^{ -w^T x_i } \big/ ( 1 + e^{ -w^T x_i } )$. By substituting these expressions into our $\ell(w)$ equation and simplifying it further we can obtain  a simpler expression.

$$
\begin{align}
\ell(w)
&= \sum_{i=1}^{N}y_{i}log(h_w(x_{i})) + (1-y_{i})log(1-h_w(x_{i})) \nonumber \\
&= \sum_{i=1}^{N} y_{i} log( \frac{1}{ 1 + e^{ -w^T x_i } } ) + ( 1 - y_{i} )
log( \frac{ e^{ -w^T x_i } }{ 1 + e^{ -w^T x_i } } ) \nonumber \\
&= \sum_{i=1}^{N} -y_{i} log( 1 + e^{ -w^T x_i } ) + ( 1 - y_{i} )
( -w^T x_i - log( 1 + e^{ -w^T x_i } ) ) \nonumber \\
&= \sum_{i=1}^{N} ( y_{i} - 1 ) ( w^T x_i ) - log( 1 + e^{ -w^T x_i } ) \nonumber
\end{align}
$$

We'll use the formula above to compute the log likelihood for the entire dataset, which is used to assess the convergence of the algorithm. Toy code provided below.

```python
def compute_avg_log_likelihood(data, label, weights):
    """
    the function uses a simple check to prevent overflow problem,
    where numbers gets too large to represent and is converted to inf
    an example of overflow is provided below, when this problem occurs,
    simply use the original score (without taking the exponential)

    scores = np.array( [ -10000, 200, 300 ] )
    logexp = np.log( 1 + np.exp(-scores) )
    logexp
    """
    scores = np.dot(data, weights)
    logexp = np.log(1 + np.exp(-scores))

    # simple check to prevent overflow
    mask = np.isinf(logexp)
    logexp[mask] = -scores[mask]

    log_likelihood = np.sum((label - 1) * scores - logexp) / data.shape[0]
    return log_likelihood
```

**Note:** We made one tiny modification to the log likelihood function We added a ${1/N}$ term which averages the log likelihood across all data points. The ${1/N}$ term will make it easier for us to compare stochastic gradient ascent with batch gradient ascent later.

## Gradient

Now that we obtain the formula to assess our algorithm, we'll dive into the meat of the algorithm, which is to derive the gradient for the formula (the derivative of the formula with respect to each coefficient):

$$\ell(w) = \sum_{i=1}^{N} ( y_{i} - 1 ) ( w^T x_i ) - log( 1 + e^{ -w^T x_i } )$$

And it turns out the derivative of log likelihood with respect to to a single coefficient $w_j$ is as follows (the form is the same for all coefficients):

$$
\frac{\partial\ell(w)}{\partial w_j} = \sum_{i=1}^N (x_{ij})\left( y_i - \frac{1}{ 1 + e^{-w^Tx_i} } \right )
$$

To compute it, you simply need the following two terms:

- $\left( y_i - \frac{1}{ 1 + e^{-w^Tx_i} } \right )$ is the vector containing the difference between the predicted probability and the original label.
- $x_{ij}$ is the vector containing the $j_{th}$ feature's value.

For a step by step derivation, consider going through the following link. [Blog: Maximum likelihood and gradient descent demonstration](https://zlatankr.github.io/posts/2017/03/06/mle-gradient-descent), it uses a slightly different notation, but the walkthrough should still be pretty clear.

## Stochastic/Mini-batch Gradient

The problem with computing the gradient (or so called batched gradient) is the term $\sum_{i=1}^{N}$. This means that we must sum the contributions over all the data points to calculate the gradient, and this can be problematic if the dataset we're studying is extremely large. Thus, in stochastic gradient, we can use a single point as an approximation to the gradient:

$$
\frac{\partial\ell_i(w)}{\partial w_j} = (x_{ij})\left( y_i - \frac{1}{ 1 + e^{-w^Tx_i} } \right )
$$

**Note1:** Because the **Stochastic Gradient** algorithm uses each row of data in turn to update the gradient, if our data has some sort of implicit ordering, this will negatively affect the convergence of the algorithm. At an extreme, what if we had the data sorted so that all positive reviews came before negative reviews?  In that case, even if most reviews are negative, we might converge on an answer of +1 because we never get to see the other data. To avoid this, one practical trick is to shuffle the data before we begin so the rows are in random order.

**Note2:** Stochastic gradient compute the gradient using only 1 data point to update the the parameters, while batch gradient uses all $N$ data points. An alternative to these two extremes is a simple change that allows us to use a **mini-batch** of $B \leq N$ data points to calculate the gradient. This simple approach is faster than batch gradient but less noisy than stochastic gradient that uses only 1 data point. Given a mini-batch (or a set of data points) $\mathbf{x}_{i}, \mathbf{x}_{i+1} \ldots \mathbf{x}_{i+B}$, the gradient function for this mini-batch of data points is given by:

$$
\sum_{s = i}^{i+B} \frac{\partial\ell_s(w)}{\partial w_j} = \frac{1}{B} \sum_{s = i}^{i+B} (x_{sj})\left( y_i - \frac{1}{ 1 + e^{-w^Tx_i} } \right )
$$

Here, the $\frac{1}{B}$ means that we are normalizing the gradient update rule by the batch size $B$. In other words, we update the coefficients using the **average gradient over data points** (instead of using a pure summation). By using the average gradient, we ensure that the magnitude of the gradient is approximately the same for all batch sizes. This way, we can more easily compare various batch sizes and study the effect it has on the algorithm.

## Implementation

Recall our task is to find the optimal value for each individual weight to lower the cost. This requires taking the partial derivative of the cost/error function with respect to a single weight, and then running gradient descent for each individual weight to update them. Thus, for any individual weight $w_j$, we'll compute the following:

$$ w_j^{(t + 1)} = w_j^{(t)} + \alpha * \sum_{s = i}^{i+B} \frac{\partial\ell_s(w)}{\partial w_j}$$

Where:

- $\alpha$ denotes the the learning rate or so called step size, in other places you'll see it denoted as $\eta$.
- $w_j^{(t)}$ denotes the weight of the $j_{th}$ feature at iteration $t$.

And we'll do this iteratively for each weight, many times, until the whole network's cost function is minimized.

```python
# put the code together into one cell

def predict_probability(data, weights):
    """probability predicted by the logistic regression"""
    score = np.dot(data, weights)
    predictions = 1 / (1 + np.exp(-score))
    return predictions

def compute_avg_log_likelihood(data, label, weights):
    """
    the function uses a simple check to prevent overflow problem,
    where numbers gets too large to represent and is converted to inf
    an example of overflow is provided below, when this problem occurs,
    simply use the original score (without taking the exponential)

    scores = np.array([-10000, 200, 300])
    logexp = np.log(1 + np.exp(-scores))
    logexp
    """
    scores = np.dot(data, weights)
    logexp = np.log(1 + np.exp(-scores))

    # simple check to prevent overflow
    mask = np.isinf(logexp)
    logexp[mask] = -scores[mask]

    log_likelihood = np.sum((label - 1) * scores - logexp) / data.shape[0]
    return log_likelihood
```

```python
def logistic_regression(data, label, step_size, batch_size, max_iter):

    # weights of the model are initialized as zero
    data_num = data.shape[0]
    feature_num = data.shape[1]
    weights = np.zeros(data.shape[1])

    # `i` keeps track of the starting index of current batch
    # and shuffle the data before starting
    i = 0
    permutation = np.random.permutation(data_num)
    data, label = data[permutation], label[permutation]

    # do a linear scan over data, for each iteration update the weight using
    # batches of data, and store the log likelihood record to visualize convergence
    log_likelihood_record = []
    for _ in range(max_iter):

        # extract the batched data and label use it to compute
        # the predicted probability using the current weight and the errors
        batch = slice(i, i + batch_size)
        batch_data, batch_label = data[batch], label[batch]
        predictions = predict_probability(batch_data, weights)
        errors = batch_label - predictions

        # loop over each coefficient to compute the derivative and update the weight
        for j in range(feature_num):
            derivative = np.dot(errors, batch_data[:, j])
            weights[j] += step_size * derivative / batch_size

        # track whether log likelihood is increasing after
        # each weight update
        log_likelihood = compute_avg_log_likelihood(
            data = batch_data,
            label = batch_label,
            weights = weights
        )
        log_likelihood_record.append(log_likelihood)

        # update starting index of for the batches
        # and if we made a complete pass over data, shuffle again
        # and refresh the index that keeps track of the batch
        i += batch_size
        if i + batch_size > data_num:
            permutation = np.random.permutation(data_num)
            data, label = data[permutation], label[permutation]
            i = 0

    # We return the list of log likelihoods for plotting purposes.
    return weights, log_likelihood_record
```

## Comparing Result and Convergence Behavior

We'll use the logistic regression code that we've implemented and compare the predicted auc score with scikit-learn's implementation. This only serves to check that the predicted results are similar and that our toy code is correctly implemented. Then we'll also explore the convergence difference between batch gradient descent and stochastic gradient descent.

```python
# manually append the coefficient term,
# every good open-source library does not
# require this additional step from the user
data = np.c_[np.ones(X.shape[0]), X]

# using our logistic regression code
weights_batch, log_likelihood_batch = logistic_regression(
    data = data,
    label = np.array(y),
    step_size = 5e-1,
    batch_size = X.shape[0],  # batch gradient descent
    max_iter = 200
)
```

```python
# compare both logistic regression's auc score
logreg = LogisticRegression(C = 1e9)
logreg.fit(X, y)
pred_prob = logreg.predict_proba(X)[:, 1]

proba = predict_probability(data, weights_batch)

# check that the auc score is similar
auc1 = metrics.roc_auc_score(y, pred_prob)
auc2 = metrics.roc_auc_score(y, proba)
print('auc', auc1, auc2)
```

    auc 0.8699025622518947 0.8699025622518947

```python
weights_sgd, log_likelihood_sgd = logistic_regression(
    data = data,
    label = y,
    step_size = 5e-1,
    batch_size = 30,  # stochastic gradient descent
    max_iter = 200
)
```

```python
weights_minibatch, log_likelihood_minibatch = logistic_regression(
    data = data,
    label = y,
    step_size = 5e-1,
    batch_size = 100,  # mini-batch gradient descent
    max_iter = 200
)
```

```python
plt.figure(figsize = (10, 7))
plt.plot(log_likelihood_sgd, label = 'stochastic gradient descent')
plt.plot(log_likelihood_batch, label = 'batch gradient descent')
plt.plot(log_likelihood_minibatch, label = 'mini-batch gradient descent')
plt.legend(loc = 'best')
plt.xlabel('# of iterations')
plt.ylabel('Average log likelihood')
plt.title('Convergence Plot')
plt.show()
```

Based on the convergence plot above, we can see that the it's a good idea to use mini-batch gradient descent since it strikes a good balance between batch gradient, which convergences steadily but can be computationly too expensive when the dataset is too large, and stochastic gradient, which is faster to train, but the result can be too noisy.

## Pros and Cons of Logistic Regression

We'll end this notebook listing out some pros and cons of this method.

**Pros:**

- Highly interpretable (if you remember how).
- Model training and prediction are fast. Thus can be desirable in large-scale applications when we're dealing with millions of parameters.
- Almost no parameter tuning is required (excluding regularization).
- Outputs well-calibrated predicted probabilities.

**Cons:**

- Presumes a linear relationship between the features
- Performance is (generally) not competitive with the best supervised learning methods.
- Can't automatically learn feature interactions.

# Reference

- [Notebook: Logistic Regression](http://nbviewer.jupyter.org/github/justmarkham/DAT8/blob/master/notebooks/12_logistic_regression.ipynb)
- [Coursersa: Washington Classification](https://www.coursera.org/learn/ml-classification)
- [Blog: Maximum likelihood and gradient descent demonstration](https://zlatankr.github.io/posts/2017/03/06/mle-gradient-descent)


# LINEAR MODELS FOR CLASSIFICATION

---

In this chapter we start to consider the problem of classification in which, differently from the previous chapter the target variables is discrete. For the moment just the basic algorithms will be presented, leaving more advanced methods for the next chapters.

## Linear classification

---

The problem is similar to the one in the previous notebook but in this case the outputs are classes and not continuous variables. In classification, given an input $x$ we want to classify it into one of the $C_k$ classes where $k=1,...,K$. In this case, there is no noise in the output, an apple is an apple no matter what uncertainty we have in the experiment. An example can be: given as input the pixels of an X-ray image we want to say if there is cancer or not. In principle we can use the same techniques used in the regression problem but, in the end, we will obtain really bad solutions.

In linear classification, the input space is divided into decision regions whose boundaries are called **decision boundaries**. In this chapter we will focus just in linear models for classification which are able to work with dataset whose classes can be separated by linear decision surfaces. Let's consider the linear regression model:

$$
\begin{align}
y(\textbf x, \textbf w) = w_0 + \sum_{j=1}^{D-1} w_j x_j = \textbf x^T \textbf w + w_0
\end{align}
$$

In classification we can work with categorical variables as *male* or *female*. We can convert these classes into numbers, for example, the classes *male* and *female* can be converted into $0$ and $1$. The problem is that the codomain of the linear model is not $0$ and $1$, but $-\infty$ and $+\infty$ causing incompatibility with our classes. In this case it is better to use the **generalized linear model** in which the output is not directly the weighted sum of the inputs but, is the result of a nonlinear activation function applied to this sum.

$$
\begin{align}
y(\textbf x, \textbf w) = f(\textbf x^T \textbf w + w_0)
\end{align}
$$

The model $y$ is no more linear in the parameters since they pass through a nonlinear function. Why we still call this model linear? We can answer this question by looking at the shape of the boundaries that separate the two classes into the input space. We can plot the **decision surface** by putting:

$$
\begin{align}
y(\textbf x, \textbf w) = const
\end{align}
$$

We see that these surfaces are linear functions of the inputs and the parameters since they correspond to hyperplanes. As we did for linear regression problem, also here, we can transform the input space with basis functions but we'll see this later.

In classification, there are different ways of representing the target values. In the two-class problem we have a binary target $t \in \{0, 1 \}$ and we can think at the output as a probability distribution in which $t=1$ means that the class is $\mathcal C_1$ and $t=0$ means that the class is $\mathcal C_2$. This is the point of view of probabilistic models. If the classes are more generally $K$, instead of just $2$, we can use the $1$-of-$K$ coding scheme on which $\textbf t$ is a vector of length $K$ that contains $1$ for the correct class and $0$ for the others. For example, if we have $5$ classes and the correct class is $\mathcal C_5$, then the vector $\textbf t$ will be $\{0, 0, 0, 0, 1\}$. In this, case we can interpret the vector $\textbf t$ as a probability distribution over classes.

As already said in a previous chapter, there are three different approaches to classification. The simplest approach makes use of a **discriminant function**, in which we build a function that directly maps each input to a specific class. In a **probabilistic** approach we model the conditional probability distribution $p(\mathcal C_k | \textbf x)$ to make optimal decisions. We have to alternatives in a probabilistic view.

* The *discriminative* approach models the conditional probability directly using for example parametric models and optimizing the parameters using a training set.

* The *generative* approach models first the class conditional probabilities $p(\textbf x | \mathcal C_k)$ together with the prior of the class and then applies the Bayes' rule:

$$
\begin{align}
p(\mathcal C_k | \textbf x) = \frac{p(\textbf x | \mathcal C_k)p(\mathcal C_k)}{p(\textbf x}
\end{align}
$$

## Discriminant functions

---

Let's start considering the simplest approach in which we build a function that directly takes an input vector $\textbf x$ and assigns it to one of the $K$ classes.

### Two classes

The simplest discriminant function is obtained by taking a linear combination of the inputs as follow:

$$
\begin{align}
y(\textbf x) = \textbf w^T \textbf x + w_0
\end{align}
$$

This model assigns an input $\textbf x$ to the class $\mathcal C_1$ if $y(\textbf x) \geq 0$ and to the class $\mathcal C_2$ otherwise. The decision boundary is therefore defined by $y(\textbf x) = 0$ which correspond to a $(D-1)$-dimensional hyperplane in the input space.

It is possible to demonstrate that the weight vector $\textbf w$ is orthogonal to the decision surface. Taking two different points onto the decision boundary we have :

$$
\begin{align}
y(\textbf x_A) = y(\textbf x_B) = 0
\end{align}
$$

$$
\begin{align}
\textbf w^T(\textbf x_A - \textbf x_B) = 0
\end{align}
$$

The last equation is satisfied only if the weights vector is orthogonal to the line connecting the two points. We can also prove that $w_0$ represents the location of the decision surface. If we take a point onto the decision surface we have:

$$
\begin{align}
\frac{\textbf w^T \textbf x}{\left\lVert \textbf w \right\rVert_2} = - \frac{w_0}{\left\lVert \textbf w \right\rVert_2}
\end{align}
$$

### Multiple classes

To extend our model to the case of $K>2$ classes we have different possibilities.

Consider the case of *one-versus-the-rest* classifiers in which we build $K-1$ classifiers, each of which solves a two-class problem separating the class $\mathcal C_k$ from the others. An alternative is to use *one-versus-one* classifiers in which the $K(K-1)/2$ binary discriminant functions separate pairs of classes. The resulting decision boundaries of the two methods are reported in the pictures below.

It is possible to see that in both the cases there are regions of ambiguity. We can solve this problem by considering a single $K$-class discriminant comprising $K$ linear functions of the form:

$$
\begin{align}
y_k(\textbf x) = \textbf w_k^T \textbf x + w_{k0}
\end{align}
$$

Then, we assign a point to the class $ \mathcal C_k $ if $y_k(\textbf x) > y_j(\textbf x) $ for all $j \neq k$. The decision boundary between class $k$ and $j$ is therefore given by $y_k(\textbf x) = y_j(\textbf x)$ and hence correspond to a $(D-1)$-dimensional hyperplane. Can be easily demonstrated that the decision boundaries are always singly connected and convex. Let's take two points inside the region $\mathcal R_k$, any point that lies on the line connecting these two points can be expressed as:

$$
\begin{align}
\hat{\textbf x}= \lambda \textbf x_A + (1- \lambda) \textbf x_B
\end{align}
$$

Thanks to the linearity of the discriminant :

$$
\begin{align}
y_k(\hat{\textbf x})= \lambda y_k(\textbf x_A) + (1- \lambda) y_k(\textbf x_B)
\end{align}
$$

It follows that $y_k(\textbf x_A) > y_j(\textbf x_A)$ and $y_k(\textbf x_B) > y_j(\textbf x_B)$, hence $y_k(\hat{\textbf x}) > y_j(\hat{\textbf x})$ for all $j \neq k$ and so $\hat{\textbf x}$ belongs to $\mathcal R_k$.

### Least squares for classification

Considering a general classification problem with $K$ classes using the $1$-of-$K$ encoding scheme we can use the least squares in order to approximate the conditional expectation $ \mathbb E[\textbf t| \textbf x] $. Each class is described by its own linear model and we can conveniently write the model in matrix notation:

$$
\begin{align}
\textbf{y(x)} = \widetilde{\textbf W}^T \widetilde{\textbf x}
\end{align}
$$

where $\widetilde{\textbf W}$ is a $(D+1) \times K$ matrix whose $k$-th column is $\widetilde{\textbf w}_k = (w_{k0}, \textbf w_k^T)^T $ and $\widetilde{\textbf x} = (1, \textbf x^T)^T$. We can determine the parameters matrix by minimizing the sum-of-squares error function. Given a dataset $\mathcal D=\{\textbf x_i, t_i\}$ where $n=1,...,N$ the least-squares solution is:

$$
\begin{align}
\widetilde{\textbf W} = (\widetilde{\textbf X}^T \widetilde{\textbf X})^{-1} \widetilde{\textbf X}^T \textbf T \end{align}
$$

where $\textbf{T}$ is a $N \times K$ matrix whose $n$-th row is the  vector $\textbf t_n^T$  and $ \widetilde{\textbf X}$ is an $N \times (D+1)$ matrix whose $i$-th row is $\widetilde{\textbf x}_i^T$. A new input is assigned to the class for which $t_k = \widetilde{\textbf x}^T \widetilde{\textbf w}_k$ is largest.

Despite the least squares method provides a closed form solution of the parameters it lacks of robustness to outliers as clearly visible from the pictures below. The green line is obtained with logistic regression (presented later) and the purple one is obtained with least-squares.

Another problem deriving from the least-squares method is that of non-Gaussian distributions. Clearly, the binary target vectors have a distribution that is far from Gaussian. In linear regression models we add the Gaussian distribution through the noise model. In the pictures below we have on the rleft the classification boundaries provided by least-squares and on the right the ones provided by logistic regression. In this case, despite the three classes can be easily separated by linear boundaries the method completely fails in do this.

### Fixed basis functions

As in the case of regression, we can work in the original input space or we can operate a nonlinear transformation of it using a vector of *basis functions* $\phi(\textbf x)$. The resulting decision boundaries will be linear in the feature space $\phi$ but will be non linear in the original input space. In this way, we can obtain a linear separation in the new space, also if in the original one would be impossible. An example of this approach is reported in the pictures below.

The red and blue dots correspond to two different classes. Obviously, it is impossible in the figure on the left to generate a linear boundary able to separate blue and red dots. If we take two *Gaussian basis function*, $\phi_1(\textbf x)$ and $\phi_2(\textbf x)$ we can separate the two classes easily in the figure on the right. Another approach could be to transform the original features into polar coordinates. In the new features space the horizontal axis will be the radius $\rho$ and the vertical one the angle $\theta$. Also in linear classification, we have seen that the big challenge is to find the best way of representing the input variables.

### The perceptron

Another discriminant (not probabilistic) model for linear classification is the *perceptron* of Rosemblat (1962). This is an online algorithm, so it is able to process one sample at time and can manage two-class problems. The first step of the perceptron is to transform the original input space with fixed nonlinear basis functions, then it uses these new features to construct a generalized linear model.

$$
\begin{align}
y(\textbf x) = f(\textbf w^T \phi(\textbf x))
\end{align}
$$

The nonlinear activation function $f(\cdot)$ is given by a sign function:

$$
\begin{align}
f(a) =
\begin{cases}
+1, \quad a \geq 0 \\
-1, \quad a < 0
\end{cases}
\end{align}
$$

We assign the value of $+1$ to the class $\mathcal C_1$ and the value $-1$ to the class $\mathcal C_2$. One possibility to learn the parameters of the model would be the minimization of the misclassified points. This approach doesn't lead to a good approach since the error function would be a piecewise constant function of $\textbf w$. To understand this just imagine moving the decision boundary without increase or decrease the number of misclassified points, the error function will remain constant. By considering the optimization of the parameters with this method through gradient descent we will have a lot of plateaux. A better choice of the error function is what is called the **perceptron criterion**. We are looking for a function such that in class $\mathcal C_1$ we have $\textbf w^t \phi(\textbf x) > 0$ and the opposite for the class $\mathcal C_2$. If the target is $t \in \{-1, +1 \}$ we would like that all the inputs satisfy $\textbf w^T \phi(\textbf x_n) t_n > 0$. The perceptron algorithm assigns zero error to correctly classified inputs. The perceptron criterion uses a loss function given by:

$$
\begin{align}
L_P(\textbf w) = -\sum_{n \in \mathcal M} \textbf w^T \phi_n t_n
\end{align}
$$

where $\mathcal M$ represents the set of all misclassified points. In this way, the error function is piecewise linear in $\textbf w$. The optimization of the parameters is obtained with Stochastic Gradient Descent (SGD) as follow:

$$
\begin{align}
\textbf w^{(k + 1)} = \textbf w^{(k)} - \alpha \nabla L_P(\textbf w) = \textbf w^{(k)} + \alpha \phi(\textbf x_n) t_n
\end{align}
$$

where $\alpha$ is the learning rate. Since multiplying $\textbf w$ by a constant term the perceptron function does not change we can set the learning rate equal to one. The pseudo code of the perceptron is:

An update example of the perceptron algorithm is illustrated in the figures below.

The black arrow represents the actual parameter vector $\textbf w$ while the point with a green circle around is the misclassified one. The feature vector of the misclassified point is added to the current weight vector giving the new decision boundary. Since the change in the weights vector may cause some previously correctly classified points to be misclassified, the perceptron rule doesn't guarantee to reduce the total error at each step. The effect of a single update is just to reduce the error due to the misclassified pattern.

$$
\begin{align}
- \textbf w^{(k+1)T} \phi_n t_n = - \textbf w^{(k)T} \phi_n t_n - (\phi_n t_n)^T\phi_n t_n < - \textbf w^{(k)T} \phi_n t_n
\end{align}
$$

The *perceptron convergence theorem* states that if there exists an exact solution, so if the training set is linearly separable in the feature space $ \bf \Phi $, then the perceptron learning algorithm is guaranteed to find the exact solution in a finite number of steps. However, the convergence can be really slow and we may not be able to distinguish between non-separable problems and slowly converging ones. If multiple solutions exist, the one found at convergence depends on the initialization of the parameters and on the order of presentation of the data points. For dataset that are not linearly separable the perceptron will never converge.

## Probabilistic discriminative approach

---

### Logistic regression

We will now take a step forward considering one of the most used algorithms for classification. Let's start considering the case of two classes and the posterior probability for the first class.

$$
\begin{align}
p(\mathcal C_1 | \textbf x) = \frac{p(\textbf x | \mathcal C_1) p(\mathcal C_1)}{p(\textbf x | \mathcal C_1) p(\mathcal C_1) + p(\textbf x | \mathcal C_2) p(\mathcal C_2)} =
\frac{1}{1 + exp(-a)} = \sigma(a)
\end{align}
$$

where $\sigma(a)$ is called *logistic sigmoid function*. We can thus write the posterior of class $\mathcal C_1$ as a logistic sigmoid acting on a linear function of the feature vector as:

$$
\begin{align}
p(\mathcal C_1 | \phi) = y(\phi) = \sigma(\textbf w^T \phi)
\end{align}
$$

Consequently $p(\mathcal C_2 | \phi) = 1 - p(\mathcal C_1 | \phi)$. This is a generalized linear model since, as in previous cases, the boundary is defined by $\sigma(\textbf w^T \phi) =$ const. Despite the logistic sigmoid function is used for classification this model is called *logistic regression*. The shape of the logistic sigmoid is depicted in the figure below.

The parameters of the logistic regression model are determined through maximum likelihood. Given a dataset $\mathcal D = \{\textbf x_n, t_n\}$ for $n=1,...,N$ and $t \in \{0, 1\}$, the likelihood function is given by:

$$
\begin{align}
p(\textbf t | \textbf X, \textbf w) = \prod_{n=1}^N y_n^{t_n}(1 - y_n)^{1-t_n}
\end{align}
$$

where $y_n = \sigma(\textbf w^T \phi_n)$. We can define the error function by taking the negative log of the likelihood, obtaining the **cross entropy error function**:

$$
\begin{align}
L(\textbf w) = - \ln p(\textbf t | \textbf X, \textbf w) = - \sum_{n=1}^N(t_n \ln y_n - (1 - t_n) \ln(1 - y_n)) = \sum_{n=1}^N L_n
\end{align}
$$

In this case there is no a closed form solution, due to the nonlinearity of the sigmoid, but we can compute the gradient of the loss function w.r.t. the parameters obtaining

$$
\begin{align}
\nabla L(\textbf w) = \sum_{n=1}^N(y_n - t_n) \phi_n
\end{align}
$$

We can notice that it takes exactly the same form as the gradient of the sum-of-squares error function for the linear regression model. The error function is convex, so a global optimum exists and can be found with gradient-based optimization and adopting online methods.

NOTE (Deriving logistic regression):
The idea is that we want to use the same linear equation to compute probability of two classes. First of all, in binary classification the output can be $0$ or $1$ but in a linear model it can goes from $-\infty$ to $+\infty$. Let's bound the linear equation between $0$ and $1$ in order to have a valid probability function.

$$ y(\textbf w) = \textbf w^T \textbf x $$

$$ p = \frac{e^y}{e^y + 1} $$

Let's take now the odds ratio, or rather the ratio between the probability of the positive event and the probability of the negative one.

$$ odds(p) = \frac{p}{1 - p} = e^y $$

By taking the log of the odds we have:

$$ \log\Big(\frac{p}{1-p}\Big) = y $$

### Multiclass logistic regression

In the multiclass case the posterior probabilities are represented through the generalization of the logistic sigmoid called **softmax function**. This name comes from the fact that the function is a smoothed version of the 'max' function. The posterior can be written as:

$$
\begin{align}
p(\mathcal C_k | \phi) = y_k(\phi) = \frac{exp(\textbf w_k^T \phi)}{\sum_j exp(\textbf w_j^T \phi)}
\end{align}
$$

The denominator guarantees that the probability will be bounded between $0$ and $1$. At this point we compute the value of the parameters directly with maximum likelihood. It is easy to write down the likelihood using the $1$-of-$K$ coding scheme where the target is binary:

$$
\begin{align}
p(\textbf T | \textbf w_1, ..., \textbf w_K) = \prod_{n=1}^N \prod_{k=1}^Kp(\mathcal C_k | \phi_n )^{t_{nk} } = \prod_{n=1}^N \prod_{k=1}^K y_{nk}^{t_{nk}}
\end{align}
$$

where $\textbf T$ is a $N \times K$ matrix of target variables and:

$$
\begin{align}
y_{nk} = p(\mathcal C_k | \phi_n) = \frac{exp(\textbf w_k^T \phi)}{\sum_j exp(\textbf w_j^T \phi)}
\end{align}
$$

Taking the negative logarithm gives the **cross-entropy function**:

$$
\begin{align}
L(\textbf w_1, ..., \textbf w_K) = - \ln p(\textbf T | \textbf w_1, ..., \textbf w_K) = - \sum_{n=1}^N \sum_{k=1}^K t_{nk} \ln y_{nk}
\end{align}
$$

Taking the gradient w.r.t. a generic parameter vector, for example $\textbf w_j$, we obtain:

$$
\begin{align}
\nabla _{\textbf w_j} E(\textbf w_1, ..., \textbf w_K) = \sum_{n=1}^N(y_{nj} - t_{nj}) \phi_n
\end{align}
$$

Once again, we found the same form that was found for the sum-of-squares error function with the linear model and the cross-entropy for the logistic regression. Also in this case we can apply a sequential algorithm for the learning. As one can observe from the plot of the logistic function, it is very similar to the step function:

The main difference is that the logistic is smoother and this allows to remove the instability related to nonlinearly separable variables in the input space that prevent the algorithm to converge. The logistic regression will converge also in the case of nonlinearly separable features in a point in which there will be some misclassified value.

> With this notebook we have finished all the *basic* tools of machine learning

References:

1. *Restelli M., Machine Learning - course slides*

2. *Bishop C.M., "Pattern Recognition and Machine Learning", chapter: 4*


# NANO106 - Symmetry Computations on $m\overline{3}m$ ($O_h$) Point Group
*by Shyue Ping Ong*

This notebook demonstrates the computation of orbits in the $m\overline{3}m$ ($O_h$) point group (more complex than the simple mmm example). It is part of course material for UCSD's NANO106 - Crystallography of Materials.

# Initializing the $m\overline{3}m$ point group.

Let's start by importing the numpy, sympy and other packages we need. Instead of going through all the steps one by one, we will use the symmetry.point_groups written by Prof Ong, which basically consolidates the information for all point groups in a single module.

```python
import numpy as np
from sympy import symbols, Mod
from symmetry.groups import PointGroup

#Create the point group.
oh = PointGroup("m-3m")

print "The generators for this point group are:"
for m in oh.generators:
    print m
print "The order of the point group is %d." % len(oh.symmetry_ops)
```

    The generators for this point group are:
    [[0, 0, 1], [1, 0, 0], [0, 1, 0]]
    [[0, -1, 0], [1, 0, 0], [0, 0, 1]]
    [[-1, 0, 0], [0, -1, 0], [0, 0, -1]]
    [[0, 1, 0], [1, 0, 0], [0, 0, 1]]
    The order of the point group is 48.

# Computing Orbits

Using sympy, we specify the symbolic symbols x, y, z to represent position coordinates. We also define a function to generate the orbit given a set of symmetry operations and a point p.

```python
x, y, z = symbols("x y z")

def pt_2_str(pt):
    return str([i.args[0] if isinstance(i, Mod) else i for i in pt])
```

## Orbit for General Position

```python
p = np.array([x, y, z])
orb = oh.get_orbit(p, tol=0)
print "For the general position %s on the two-fold axis, the orbit comprise %d points:" % (str(p), len(orb))
for o in orb:
    print pt_2_str(o),
```

    For the general position [x y z] on the two-fold axis, the orbit comprise 48 points:
    [z, x, y] [-y, x, z] [-x, -y, -z] [y, x, z] [y, z, x] [z, -y, x] [-z, -x, -y] [z, y, x] [-x, z, y] [-x, -y, z] [y, -x, -z] [-x, y, z] [x, y, z] [-y, -x, -z] [x, z, y] [x, -y, z] [x, z, -y] [-y, -z, -x] [z, y, -x] [-y, z, -x] [-x, -z, y] [y, z, -x] [-x, -z, -y] [-y, z, x] [y, -x, z] [z, -x, -y] [-z, y, -x] [z, -x, y] [x, -z, y] [-y, x, -z] [y, -z, -x] [-z, x, y] [z, -y, -x] [-z, x, -y] [y, -z, x] [-z, -y, -x] [-z, -x, y] [z, x, -y] [-z, y, x] [x, -z, -y] [-x, z, -y] [x, y, -z] [-y, -x, z] [-y, -z, x] [-z, -y, x] [y, x, -z] [x, -y, -z] [-x, y, -z]

## Orbit for Special Position on four-fold rotation axis

```python
p = np.array([0, 0, z])
orb = oh.get_orbit(p, tol=0)

print "For the special position %s on the two-fold axis, the orbit comprise %d points:" % (str(p), len(orb))
for o in orb:
    print pt_2_str(o),
```

    For the special position [0 0 z] on the two-fold axis, the orbit comprise 6 points:
    [z, 0, 0] [0, 0, z] [0, 0, -z] [0, z, 0] [-z, 0, 0] [0, -z, 0]

The orbit is similar for the other two-fold axes on the a and b axes are similar.

## Orbit for Special Position on three-fold rotation axis

The three-fold rotation axis are given by (x, x, x)

```python
p = np.array([x, x, x])
orb = oh.get_orbit(p, tol=0)
print "For the special position %s on the two-fold axis, the orbit comprise %d points:" % (str(p), len(orb))
for o in orb:
    print pt_2_str(o),
```

    For the special position [x x x] on the two-fold axis, the orbit comprise 8 points:
    [x, x, x] [-x, x, x] [-x, -x, -x] [x, -x, x] [-x, -x, x] [x, -x, -x] [x, x, -x] [-x, x, -x]

## Orbit for Special Position on two-fold rotation axis

The two-fold rotation axis are given by (x, x, 0).

```python
p = np.array([x, x, 0])
orb = oh.get_orbit(p, tol=0)
print "For the special position %s on the two-fold axis, the orbit comprise %d points:" % (str(p), len(orb))
for o in orb:
    print pt_2_str(o),
```

    For the special position [x x 0] on the two-fold axis, the orbit comprise 12 points:
    [0, x, x] [-x, x, 0] [-x, -x, 0] [x, x, 0] [x, 0, x] [0, -x, x] [0, -x, -x] [-x, 0, x] [x, -x, 0] [x, 0, -x] [-x, 0, -x] [0, x, -x]

## Orbit for Special Position on mirror planes

Positions on the mirror on the a-b plane have coordinates (x, y, 0).

```python
p = np.array([x, y, 0])
orb = oh.get_orbit(p, tol=0)
print "For the special position %s on the two-fold axis, the orbit comprise %d points:" % (str(p), len(orb))
for o in orb:
    print pt_2_str(o),
```

    For the special position [x y 0] on the two-fold axis, the orbit comprise 24 points:
    [0, x, y] [-y, x, 0] [-x, -y, 0] [y, x, 0] [y, 0, x] [0, -y, x] [0, -x, -y] [0, y, x] [-x, 0, y] [y, -x, 0] [-x, y, 0] [x, y, 0] [-y, -x, 0] [x, 0, y] [x, -y, 0] [x, 0, -y] [-y, 0, -x] [0, y, -x] [y, 0, -x] [-x, 0, -y] [-y, 0, x] [0, -x, y] [0, -y, -x] [0, x, -y]

The orbit is similar for the other mirror planes on the a-c and b-c planes are similar.

lectures/lecture_4_point_group_symmetry/Symmetry Computations on m-3m (O_h) Point Group.ipynb