```python
import pandas as pd
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
import random

from scipy.optimize import curve_fit

import matplotlib.ticker as plticker
%matplotlib inline

```

### Forecasting Random Walks?

* Recall that in a random walk is defined

 $$
 v_t = v_{t-1} + \epsilon_t,
 $$

* Where $\epsilon_t$ is  white noise
  * Recall that we defined white noise as having:
    * Constant mean
    * Constant variance
    * Values are independently and identically distributed

* A special but  common case is the Gaussian noise

$$
    e \sim \mathcal{N}(\mu, \sigma)
$$

```python
plt.figure(figsize=(20,6))

v = np.zeros(101)
for t in range(1, len(v)):
    v[t] = v[t-1] + 10 *  np.random.normal(0,1)

plt.plot(np.arange(101),v)
```

### Random Walks and White Noise

* In a random walk, the change in values is merely the white noise

$$
    v_t - v_{t-1}  = \epsilon_t
$$

* We know we cannot predict white noise
  * We don't know the distribution not its parameters

* In many cases, the best we can do is assume that the future will be like the immediate past
  * Causal factors will be the same in the short term

### Predicting Random Walks -- Cont'd

* The best guess for $v_t$ is $v_{t-1}$

$$
\hat{v}_{t} = v_{t-1}
$$

* This is the "state of the art" for handling data from random walks
  * Also, called the naïve method or random walk forecasts

* Our error is just the noise

```python
plt.figure(figsize=(20,6))
x_axis = np.arange(0,100)
v_t_subset = v[1:101]
v_tm1_subset = v[0:100]
plt.subplot(2,1,1)
plt.plot(x_axis[:-1], v_t_subset[:-1], label="$v_t$", linewidth=5)
plt.plot(x_axis[1:], v_tm1_subset[1:], label="$v_{t-1}$", linewidth=5)
#plt.fill_between(x_axis, v_tm1_subset, v_t_subset, color='g', alpha=0.5)
plt.legend()
plt.subplot(2,1,2)
plt.plot(x_axis[19:29], v_t_subset[19:29], label="$v_{t-1}$", linewidth=5)
plt.plot(x_axis[20:30], v_tm1_subset[20:30], label="$v_{t}$", linewidth=5)
#plt.fill_between(x_axis, v_tm1_subset, v_t_subset, color='g', alpha=0.5)

```

### Forecasting Using a Sliding Window

* Noise effect can be additive
  * Multiple sources of noise can be compounded onto the signal
  * E.g., in stock market, noise can be due to political instability, public health news, unforeseen weather events, etc..
  * E.g., IoT sensors can be subject to noise from ambient temperature, interference from external devices, natural interference, etc.

* In some cases, the effect of some sources of noise can be short-lived
    * Cause major spikes in the data than can bias predictions
    * Time series regresses to its original value after the noise subsides

```python
plt.figure(figsize=(20,8))

v = np.zeros(500)
other_noise = 0

for t in range(1, len(v)):
    v[t] = v[t-1] + 10 *  np.random.normal(0,1) - other_noise
    other_noise = 0
    if np.random.binomial(1,0.02):
        noise = np.random.normal(0,100)
        v[t] +=  noise

plt.plot(v)
```

### Forecasting Using a Sliding Window

* Outliers can significantly affect the prediction

### Forecasting Using a Sliding Window - Cont'd

* To know the past, we need to abstract the noise the data, i.e., smooth the data

* We can smooth th prediction by using the average of "some" $n$ previous values
$$
v_t = \frac{1}{n}\sum_{i=0}^{n-1}v_{t-i}
$$

```python
plt.figure(figsize=(12,6))

random_ts = pd.read_csv("data/rand_ts.csv", header=None, names=['val'])

random_ts
```

      val

      0
      -1.0

      1
      -1.2

      2
      -1.0

      3
      -1.2

      4
      -1.0

      ...
      ...

      318
      2.2

      319
      2.4

      320
      2.6

      321
      2.8

      322
      6.0

323 rows × 1 columns

```python
plt.figure(figsize=(20,6))
pred = random_ts[-10:].mean()
plt.subplot(1,2,1)
plt.plot(random_ts)
plt.scatter(len(random_ts)+1, random_ts.val.iloc[-1], color='r', s=100)
plt.title("Prediction based on the value at time $t-1$")
plt.subplot(1,2,2)
plt.plot(random_ts)
plt.scatter(len(random_ts)+1, pred, color='r', s=100)
plt.title("Prediction based on the average of the last 10 values")

```

### Smoothing data

- This strategy is the same as the one we used for computing the rolling average for the CO2 data

  * Smoothing the curve prior to finding the power law parameter that best fits it

    * Recall that we used `scipy.optimize.curve_fit`

* Ideal for subtracting the trend to explore the seasonality

### Smoothing Using a Running or Moving Average

* For every value $x_i$, we compute a new smoothed score value $s_i$
  * i.e., compute $x_i$ in light of its neighbors values
  * Uses one the sliding window strategies dicussed before

* Example 1:
For some window of size $2k+1$, with $k$ a positive value, we compute $s_i$ as:

$$
s_i = \frac{1}{2k+1} \sum_{j=-k}^{k}x_{i+j}
$$

* Ex. for $k=3$, the window size is $2k+1=7$ and computing the smoothed score $s_{11}$ at position $x_{11}$ amount to averaging the score for positions:

$$
x_{8}, x_{9}, x_{10}, x_{11}, x_{12}, x_{13}, x_{14}
$$

* Example 1:
For some window of size k, we compute $s_i$ as:

$$
s_i = \frac{1}{k} \sum_{j=0}^{k}x_{i+j}
$$

* Ex. for $k=4$,  computing the smoothed score $s_{11}$ at position $x_{11}$ amount to averaging the score for positions:

$$
x_{8}, x_{9}, x_{10}, x_{11}
$$

```python
plt.figure(figsize=(16,7))
co2 = pd.read_csv("data/carbon_dioxide.txt", header=None, names=['val'])
plt.plot(co2)
plt.plot(co2.rolling(12, center=True).mean(), linewidth=5)

```

```python
plt.figure(figsize=(12,10))
plt.xlim(0, len(random_ts))
plt.plot(random_ts)

plt.plot(random_ts['val'].rolling(window=5, center=True).mean(), linewidth=4)
```

### Effect of the Window Size

* The rolling window smooths the data, but the curve is still bumpy around large spikes

* Sudden jumps in the MA value when a relatively large observation (a spike) comes into or falls out of the moving average window.

* We can further smooth the curve by choosing a larger window size
  * More values to dampen the effect of larger values

```python
plt.figure(figsize=(20,8))
plt.xlim(0, len(random_ts))
plt.plot(random_ts)
plt.plot(random_ts['val'].rolling(window=10, center=True).mean(), linewidth=4)
```

### Handling Outliers

* Increasing the window to compensate for few spikes can lead to underfitting the data
  * Overly "flatten" the curve
  * Unduly biases toward sharp, even legitimate increases or decreases
    * Causes a "lag" in the signal

* Decreasing the window to compensate cna lead to overfitting the data
  * Lose the ability to smooth the data
    * Causes sudden fluctuations that are not reflective of the data

### Handling Outliers

* We can keep a large window size but, for instance, weigh the values in the window differentially

* The points close to window boundaries contribute less than the point in the center

### Question

* Which approach used a similar strategy for computing the contribution of points differentially across a window?

### Using a Gaussian for Computing the Contributions

* You can assign the same preset values to specific indices.

* We can use more sophisticated versions that assign custom weights. E.g.,

$ s_i = \sum_{j=-k}^{k}w_{i+j}x_{i+j} $

* Where weight $w_i$ for a point $x_i$ can be determined using, for example a Gaussian, such as:
$$
f(\mu_x, \sigma_x) = \frac{1}{\sqrt{2\pi\sigma_x^2}}exp(-\frac{1}{2}(\frac{\mu_x}{\sigma_x})^2)
$$

### Challenges of Using a Gaussian for Computing the Contributions

* The range of values in each window can vary substantially

* For each window, we need to dynamically scale $\sigma$ so that the likelihoods are not too small

### Challenges of Using a Gaussian for Computing the Contributions  - Cont'd

* Easy to implement but not ideal:

  * The weighted MA can be computationally intensive to evaluate
    * For each window, we need to determine a specific $\sigma$ and compute the likelihoods for all the points in the window

  * We lose data at the beginning and the end of data due to window constraints

* No easy way to incorporate information about the trend or seasonality

### Exponential Smoothing

* Exponential smoothing mitigates the shortcomings of the rolling window and weighted rolling window

* Intuition: Instead of using simply a subset of the data, we will use all past observations

    * The contributions (weights) associated with each point decays exponentially as the observations "get older"

* This simpleconcept is thee basis for some of the most successful forecasting methods.

  * One such power method is exponential smoothing

### Forms of Exponential Smoothing

* Different types of smoothing capture different types of signals

| Type of Smoothing            | Data type                     |
|:------------------------------|:-------------------------------|
| Single Exponential Smoothing | Neither trend nor seasonality |
| Double Exponential Smoothing | Trend but no seasonality      |
| Triple Exponential Smoothing | Trend and seasonality         |

### Single Exponential Smoothing

* Computes smoothed data $s_t$ at time $t$ using the observed value $v_t$ and smoothed value from previous time step ($v_{t-1}$)

$$
s_t = \alpha~v_t + (1-\alpha)~s_{t-1} ~~~~~~\text{where}~~~~~~ 0 \le \alpha \le 1
$$

$s_t$: smoothed value at timestep $t$
$v_t$: actual unsmoothed value at time step $t$
$s_{t-1}$: smoothed value at time step $t-1$

### Single Exponential Smoothing: Remarks

Note that at:

* $\alpha =0, s_t = (1-\alpha)~s_{t-1}$ we are only retaining the past
* $\alpha =1, s_t = \alpha v_t$ we are only retaining current values

How is this exponential?

It turns out that:

$$
\begin{align}
s_t &= \alpha~v_t + (1-\alpha)s_{t-1} \\
    &=  \alpha~v_t + (1-\alpha) (\alpha~v_{t-1} + (1-\alpha)s_{t-2})\\
    &=  \alpha~v_t + \alpha(1-\alpha)~v_{t-1} + (1-\alpha)^2s_{t-2}\\
    &=  \alpha[v_t + (1-\alpha)v_{t-1} + (1-\alpha)^2v_{t-2}+ \ldots + (1-\alpha)^{t-1}v_{1}]+(1-\alpha)^t v_{0}\\
\end{align}
$$

### Single Exponential Smoothing: Remarks -- Cont'd

$$
\begin{align}
s_t  &=  \alpha[v_t + (1-\alpha)v_{t-1} + (1-\alpha)^2v_{t-2}+ \ldots + (1-\alpha)^{t-1}v_{1}]+(1-\alpha)^t v_{0}\\
\end{align}
$$

* All the previous values contribute to smoothed value of $x_i$ but their contribution as increasingly small
* The contribution of past values is controlled by the decay of the parameter $\alpha$
  * This geometric progression (geometric series) is the discrete version of an exponential function
    * A geometric sequence is discrete while an exponential function is continuous
      * This is where the name for this smoothing method originated

### Single Exponential Smoothing: Remarks -- Cont'd

* Given that we have smoothed our signal, we can use $s_t$ in forecasting

* Our approach is going to be simple

$ F_{t+1} = s_t$

* Our data is more resilient to outliers
* The prediction takes into account weighted contributions of previous time points

### Single Exponential Smoothing as a Function of the forecasting error

* Note that the forecast at time $t+1$ ($F_{t+1}$) is similar to the random walk algorithm used earlier

$$
\begin{align}
s_t &= \alpha~v_t + (1-\alpha)s_{t-1} \\
    &= \alpha v_t + s_{t-1}-\alpha s_{t-1} \\
    &= s_{t-1} + \alpha (v_t - s_{t-1}) \\
\end{align}
$$

Since
$ F_{t+1} = s_t$,
and
$ F_{t} = s_{t-1}$

then

$$
\begin{align}
F_{t+1} &= F_{t} + \alpha (v_t - F_t)\\
F_{t+1} &= F_{t} + \alpha (E)\\
\end{align}
$$

Where epsilon is here the E in forecasting (the difference between forecasted and observed values)

### Single Exponential Smoothing as a Function of the forecasting error - Cont'd

$$
\begin{align}
F_{t+1} = &= F_{t-1} + \alpha (E)\\
\end{align}
$$

* $E$ in forecasting (the difference between forecasted and observed values)
* This means that to forcast $F_{t+1}$ is simply the forecast at $F_T$ padded with a fraction of the error in predicting $F_T$.

### Using Exponential Smoothing with  `Pandas.Series`

* The Series objects has a `ewm` (exponentially weighted method)

```
> random_ts['val'].ewm(alpha = 0.5)
ExponentialMovingWindow [com=0.5,min_periods=1,adjust=True,ignore_na=False,axis=0,method=single]
```

* Note that single exponential smoothing is simply a complex weighted average

* We compute the new values by using the mean function

```
random_ts['val'].ewm(alpha=0.5).mean()
```

```python
plt.figure(figsize=(18,6))
plt.plot(random_ts)
plt.plot(random_ts['val'].ewm(alpha=0.5).mean(), color='r', lw=5, label="Single Exp. Smoothing")
plt.plot(random_ts['val'].rolling(window=5, center=True).mean(), color='g', lw=5, label="Rollowing Window")
plt.legend()
```

### Impact of Param $\alpha$

```python
plt.figure(figsize=(18,6))
plt.plot(random_ts)
plt.plot(random_ts['val'].ewm(alpha=0.3).mean(), color='k', lw=2, alpha=0.5, label=r"$\alpha = 0.3$")
plt.plot(random_ts['val'].ewm(alpha=0.7).mean(), color='g', lw=2, alpha=0.5, label=r"$\alpha = 0.8$")
plt.legend()
```

### Double Exponential Smoothing

* It would unwise to discard the trend information if one exists.

* Double exponential smoothing implicitly accounts for information about the trend

   * A "non-parametric equivalent" for identifying the equation that best fits the trend

* We will include the trend in our forecast

$$
S_t = \mbox{S}_{t-1} + \mbox{r}_{t-1},
$$

* Where $r_{t-1}$ is the trend at time ${t-1}$

* $s_t$ is also called the level

### Accounting for the Trend

* It helps to think of the trend as the exponentially weighted increases since t=0
  * The most recent inrease gets weight of alpha

### Double Exponential Smoothing - Step 2

* We can infer the trend $r_t$ using the same approach used to compute $s_t$

$$
r_t = \beta(s_{t} - s_{t-1}) + (1-\beta)r_{t-1}
$$

- This equation uses a trend coefficient $\beta$ which acts in a similar as $\alpha$ on signal

  - Recall that the assumption here is that there is no seasonality

### Double Exponential Smoothing: Step 1

$$
s_t = \alpha \cdot v_t + (1-\alpha)(s_{t-1} + r_{t-1})
$$

- Similar to the equation used in single exponential smoothing

  - We add the effect of the trend to $s_t$

### Double Exponential Smoothing for Forecasting

- The double exponential smoothing can be used in forecasting by using the last smoothed value and adding, at each predicted timestep, the smoothed trend to it.

$$
F_{i+1} = s_i + r_{i}
$$

### Triple Exponential Smoothing

* The assumption here is that there is seasonality in the data
  * We need another "quantity" to capture the seasonality
  * Similar to how we direved a quantity to capture the trend

* The idea is to capture seasonality by using the data across past seasons at the same time in the period
  * For instance, in time series with a yearly seasonality, the seasonality components for March would be exponentially smoothed with only data points from March in previous years.

* A non-parametric way to accout for seasonality

### Triple Exponential Smoothing -- Cont'd

The new equation for the triple exponential component is:
$$
    p_t = \gamma(v_t - s_t) + (1-\gamma) p_{t-k}
$$

where:

* $k$: is the period’s length
  * This is a param of the model; cna be derived using ACF or by find the the value that best fits the data.
* $\gamma$: is the new smoothing factor for the seasonal component

### Triple Exponential Smoothing: Remarks

- The concept here is similar to what we have done in the past

- Seasonality component $p_i$ is weighted using the last previous observed point exactly one period, $k$, ago

* The function for predicting future values is computed as:

$$
F_{t+1} = s_t + r_t + p_t
$$

### Testing the Explonential Smoothing

* Does the approach pass muster with the CO2 data set we manually investigated

  * The data exhibits clear trend and seasonality

* Here we implement and test our owen version

  * If you decide to test this with the assignment data, you should use `statsmodel` package

```python
sum(co2_data["co2_val"][12*0:12*0+12])/float(12)

```

    315.8266666666666

```python
# function that returns the initial value of trend
# \beta_0
def initial_trend(series, slen):
    sum = 0.0
    for i in range(slen):
        sum += float(series["co2_val"][i+slen] - series["co2_val"][i]) / slen
    return sum / slen

# function that return the initial value of seasonality
# \gamma_0
def initial_seasonal_components(series, slen):
    seasonals = {}
    season_averages = []
    n_seasons = int(len(series)/slen)
    # compute season averages
    for j in range(n_seasons):

        season_averages.append(sum(series["co2_val"][slen*j:slen*j+slen])/float(slen))
    # compute initial values
    for i in range(slen):
        sum_of_vals_over_avg = 0.0
        for j in range(n_seasons):
            sum_of_vals_over_avg += series["co2_val"][slen*j+i]-season_averages[j]
        seasonals[i] = sum_of_vals_over_avg/n_seasons
    return seasonals

def triple_exponential_smoothing(series, slen, alpha, beta, gamma, n_preds):
    result = []
    seasonals = initial_seasonal_components(series, slen)
    for i in range(len(series)+n_preds):
        if i == 0: # initial values
            smooth = series["co2_val"][0]
            trend = initial_trend(series, slen)
            result.append(series["co2_val"][0])
            continue
        if i >= len(series): # we are forecasting
            m = i - len(series) + 1
            result.append((smooth + m*trend) + seasonals[i%slen])
        else:
            val = series["co2_val"][i]
            last_smooth, smooth = smooth, alpha*(val-seasonals[i%slen]) + (1-alpha)*(smooth+trend)
            trend = beta * (smooth-last_smooth) + (1-beta)*trend
            seasonals[i%slen] = gamma*(val-smooth) + (1-gamma)*seasonals[i%slen]
            result.append(smooth+trend+seasonals[i%slen])
    return result
```

```python
import pandas as pd
co2_data = pd.read_csv("https://www.dropbox.com/s/cegyok4vqmhikey/carbon_dioxide.txt?dl=1", names=["co2_val"])
co2_data.co2_val = co2_data.co2_val.astype('float64')
co2_data.head()
```

      co2_val

      0
      315.42

      1
      316.32

      2
      316.49

      3
      317.56

      4
      318.13

```python
# We use the exponential smoothing to forecast the next 24 months (2 periods)

plt.figure(figsize=(18,6))

#forecast = triple_exponential_smoothing(co2_data, 12, 0.005, 0.2, 0.00001, 24)

# pyplot.plot(forecast, 'r--', lw=4) # the smoothed/predicted data in red

plt.plot(co2_data, alpha=0.5, lw=4) # the true data in blue
```

```python
# We use the exponential smoothing to forecast the next 24 months (2 periods)

plt.figure(figsize=(18,6))

forecast = triple_exponential_smoothing(co2_data, 12, 0.005, 0.2, 0.00001, 24)

plt.plot(forecast, 'r--', lw=4) # the smoothed/predicted data in red

plt.plot(co2_data, alpha=0.5, lw=4) # the true data in blue
```

### Question

- In your opinion, how did I derive the values for $\alpha$, $\beta$ and $\gamma$?
  - Hint: how did we derive the data for the power law function used to model the trend or the value of $k$ when clustering the time series (assignment 3)?


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 7: Deterministic Time Series Models I

Time series models are at the foundatation of dynamic macroeconomic theory. A time series model is an equation or system of equations that describes how the variables in the model change with time. Here, we examine some theory about **deterministic**, i.e., non-random, time series models and we explore methods for simulating them. Later, we'll examine the properties of **stochastic** time series models by introducing random variables to the discrete time models covered below.

## Discrete Versus Continuous Time

To begin, suppose that we are interested in a variable $y$ that takes on the value $y_t$ at date $t$. The date index $t$ is a real number. We'll say that $y_t$ is a **discrete time** variable if $t$ takes on values from a countable sequence; e.g. $t = 1, 2, 3 \ldots$ and so on. Otherwise, if $t$ takes on values from an uncountable sequence; e.g. $t\in \mathbb{R}^+$, then we'll say that $y_t$ is a **continuous time** variable. Discrete and continuous time models both have important places in macroeconomic theory, but we're going to focus on understanding discrete time models.

## First-Order Difference Equations

Now, suppose that the variable $y_t$ is determined by a linear function of $y_{t-1}$ and some other exogenously given variable $w_t$

\begin{align}
    y_{t} & =  \rho y_{t-1} + w_t,  \tag{1}\\
\end{align}

where $\rho$ is a constant. Equation (1) is an example of a **linear first-order difference equation**. As a *difference equation*, it specifies how $y_t$ is related to past values of $y$. The equation is a *first-order* difference equation because it specifies $y_t$ as a function only of $y_{t-1}$ and not $y_{t-2}$ or $y_{t-3}$ and so on. As simple as the equation looks, it's used in a wide range of modeling applications.

### Example: Compounding Interest

Suppose that you have an initial balance of $b_0$ dollars in a savings account that pays an interest rate $i$ per compounding period. Then, after the first compounding, your account will have $b_1 = (1+i)b_0$ dollars in it. Assuming that you never withdraw funds from the account, then your account balance in any subsequent period $t$ is given by the following difference equation:

\begin{align}
b_{t} & =  \left(1+i\right) b_{t-1}. \tag{2}
\end{align}

Equation (2) is linear first-order difference equation in the same form as Equation (1). You can see this by setting $y_t = b_t$, $\rho=1+i$, and $w_t=0$ in Equation (1).

### Example: Capital Accumulation

Let $K_t$ denote the amount of physical capital in a country at date $t$, let $\delta$ denote the rate at which the capital stock depreciates each period, and let $I_t$ denote the country's investment in new capital in date $t$. Then the law of motion for the stock of physical capital is:

\begin{align}
K_{t+1} & =  I_t + (1-\delta)K_t. \tag{3}
\end{align}

This standard expression for the law of motion for the capital stock is a linear first-order difference equation. To reconcile Equation (3) with Equation (1), set $y_t = K_{t+1}$, $\rho=1-\delta$, and $w_t=I_t$.

*Note*: There is a potentially confusing way in which we identified the $t+1$-dated variable $K_{t+1}$ with the $t$-dated variable $y_t$ in this example. We can do this because the value of $K_{t+1}$ truly is determined at date $t$ even though the capital isn't used for production until the next period.

## Computation

From Equation (1), it's easy to compute the value of $y_t$ as long as you know the value of the constant $\rho$ and the variables $y_{t-1}$ and $w_t$. To begin, let's suppose that $\rho=0.5$. Then Equation (1) in our example looks like this:

\begin{align}
y_{t} & =  0.5 y_{t-1} + w_t. \tag{4}\\
\end{align}

Now, suppose that the initial value of $y$ is $y_0=0$ and that $w$ is equal to 1 in the first period and equal to zero in subsequent periods. That is: $w_1=1$ and $w_2=w_3=\cdots =0$. Now, with what we have, we can compute $y_1$. Here's how:

```python
# Initialize values for rho, y0, and w1
y0 = 0
rho = 0.5
w1 = 1

# Compute the value of y1
y1 = rho*y0 + w1

# Print the value of y1
print('y1 =',y1)
```

    y1 = 1.0

The variable `y1` in the preceding example stores the computed value for $y_1$. We can continue to *iterate* on Equation (4) to compute $y_2$, $y_3$, and so on. For example:

```python
# Initialize value for w2
w2=0

# Compute the value of y2
y2 = rho*y1 + w2

# Print the result
print('y2 =',y2)
```

    y2 = 0.5

We can do this as many times as necessary to reach the desired value of $t$. Note that iteration is necesary. There is no *vectorized* way to compute an array of values $\left[y_1, y_2, y_3, \ldots, y_{T}\right]$ from a difference equation simultaneously. The linear first-order difference equation is an example of a *recursive* model and in general iteration is necessary for computing recursive models.

Of course, there are smart ways to write code to simulate models. Rather than creating a bunch of variables `y1`, `y2`, etc., let's define a function called `diff1_example()`that takes as arguments $\rho$, $y_0$m and an array of values for $w$, and returns an array of simulated $y$ values . The function should:

1. Define a variable named `T` equal to the length of  `w`
2. Define an array of of length called `y` that is filled with zeros
3. Set the first vale of `y` to $y_0$
4. Fill-in the remaining values of `y` by iterating over:
        y[t+1]=rho*y[t]+w[t+1]
   for `t` in $[0,1,\ldots, T-1]$
5. Return `y`

```python
# Define a function that returns an array of y-values given rho, y0, and an array of w values.
def diff1_example(rho,w,y0):
    '''Function for computing the first-order difference equation y[t] = rho*y[t-1] + w[t]

    Args:
        rho (float):                Autoregressive coefficient on y[t-1]
        w (NumPy ndarray or list):  Array of exogenous values
        y0 (float):                 Initial value of the process

     Returns:
         NumPy ndarray
     '''

    T = len(w)
    y = np.zeros(T)
    y[0] = y0

    for t in range(T-1):
        y[t+1]=rho*y[t]+w[t+1]

    return y
```

### Example: Temporary Shock to $w_t$ with $\rho=0.5$

Now, use the function `diff1_example()` to simulate $y$ for 11 periods (i.e., for $t = 0, 1, \ldots 10$) with
1. $\rho=0.5$
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Initialize the variable 'T' to be equal to the number of periods to simulate
T = 11

# Create new variable 'rho' equal to the value for rho
rho = 0.5

# Create new variable 'y0' equal to the initial value for y
y0 = 0

# Initialize variable 'w' and set initial value
w = np.zeros(T)
w[1]=1

# Simulate y
y = diff1_example(rho,w,y0)

# Construct figure with two line graphs plotting the trajectories of y and w with:
#    1. Plot for w in left panel and plot for y in right panel
#    2. Clear title, x-, and y-axis labels
#    3. y-axis limits [0,1.2]

# Create figure
fig = plt.figure(figsize=(12,4))

# w trajectory
ax = fig.add_subplot(1,2,1)
ax.plot(w,'-',lw=5,alpha = 0.75)
ax.set_title('$w_t$')
ax.set_ylabel('w')
ax.set_xlabel('t')
ax.set_ylim([0,1.2])
ax.grid()

# y trajectory
ax = fig.add_subplot(1,2,2)
ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$y_t$ with $\\rho=0.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.set_ylim([0,1.2])
ax.grid()

# Use fig.tight_layout() to manage whitespace in the figure
fig.tight_layout()
```

In period 1, the *shock* to $w_t$ creates an exogenous increase in the process $y_t$. Since $y_{t+1}$ depends on the value of $y_{t}$, the transient or temporary shock to $w_t$ has a persistent effect on the series $y_t$. We call plots like those in the previous figure **impulse responses** because they depict the *responses* of variables to a one-time shock or *impulse* in the exogenous variable.

### Example: Temporary Shock to $w_t$ for Different Values of $\rho$

Next, use the function `diff1_example()` to simulate $y$ for 11 periods with:
1. $\rho=0.1,0.2, 0.3, \ldots, 0.9$
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Note: Since T, w, and y0 are already defined, there's no need to re-define them

# Construct a line graph plotting the trajectories of y with:
#    1. Clear title, x-, and y-axis labels
#    2. Y-axis limits [0,1.2]
#    3. Add a legend outside of the plot with arguments: loc='center left', bbox_to_anchor=(1, 0.5)

fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Iterate over rho in [0.1, 0.2, 0.3, ... , 0.9]. Simulate y for each rho and plot. Set plot label equal to rho value
for rho in np.arange(0.1,1,0.1):
    y = diff1_example(rho,w,y0)
    plt.plot(y,'-b',lw=5,alpha = rho,label=round(rho,1))

ax.set_title('$y_t$')
plt.ylabel('y')
plt.xlabel('t')
# ax.set_ylim([0,1.2])
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5),title='$\\rho$')
ax.grid()
```

Apparently increasing $\rho$ also increases the degree of persistence in the process $y$ following the shock to $w$. By $t=10$, the effects of the one-time increase in $w$ on $y$ have essentially dissipated for the simulations with $\rho= 0.1, 0.2, \ldots, 0.7$. But for $\rho=0.8$, the value of the $y$ is still above $0.15$ in period $10$ and for $\rho=0.9$, the process $y$ is nearly at $0.4$; i.e., nearly $40$ percent of the shock still remains. But regardless, in all cases it appears that $y$ approaches zero in the limit.

### Example: Temporary Shock to $w_t$ for $\rho=-0.5$

Use the function `diff1_example()` to simulate $y$ for 11 periods with:
1. $\rho=-0.5$
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Note: Since T, w, and y0 are already defined, there's no need to re-define them

# Create new variable 'rho' equal to the value for rho
rho = -0.5

# Simulate y
y = diff1_example(rho,w,y0)

# Construct a line graph plotting the trajectory of y with:
#    1. Plot of y
#    2. Clear title, x-, and y-axis labels
#    3. Y-axis limits [-0.8,1.2]

# Create figure
fig = plt.figure()

# y trajectory
ax = fig.add_subplot(1,1,1)
ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$y_t$ with $\\rho=-0.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.set_ylim([-0.8,1.2])
ax.grid()
```

With $\rho=-0.5$, the sign of the pricess $y$ oscilates between positive and negative values but the absolute value of $y$ smoothly converges to zero in the limit.

### Example: Temporary Shock to $w_t$ for $\rho=1.5$ and $\rho=-1.5$

So far all of the values that we hav used for $\rho$ have been less than 1 in absolute value and in all cases, the process $y$ has returned to zero in the limit. But this may not always be the case. Let's consider the cases where $\rho=1.5$ and $\rho=-1.5$.

Use the function `diff1_example()` to simulate $y$ for 11 periods with:
1. $\rho=1.5$ (left panel) and $\rho=-1.5$ (right panel)
2. $w_1 = 1$ and $w_0 = w_2 = \ldots = w_{10} = 0$
3. $y_0 = 0$

```python
# Note: Since T, w, and y0 are already defined, there's no need to re-define them

# Create figure
fig = plt.figure(figsize=(12,4))

# Set rho=1.5
rho = 1.5

# Simulate y with rho = 1.5
y = diff1_example(rho,w,0)

# plot trajectory of y with rho = 1.5
ax = fig.add_subplot(1,2,1)

ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$\\rho=1.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.grid()

# Set rho=-1.5
rho = -1.5

# Simulate y with rho = -1.5
y = diff1_example(rho,w,0)

# plot trajectory of y with rho = -1.5
ax = fig.add_subplot(1,2,2)

ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$\\rho=1.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.grid()
```

Unlike the earlier simulations, the ones in the previous example do not return to zero and instead either explode toward infinity ($\rho = 1.5$) or oscillate towards positive and negative infinity. These two simulations are examples of *explosive processes* while the ones prior were examples of *stable processes*. In the next section, I elaborate on the notion of stability.

### Example: Permanent Shock to $w_t$ with $\rho=0.5$

So far we hav eonly considered temporary shocks to to $wt$. But sometimes we want to know what effect does a permanent change in $w_t$ have on $y_t$. To answer this, we set $w_0=0$ and $w_1 = w_2 = \cdots = 1$.

Now, use the function `diff1_example()` to simulate $y$ for 11 periods (i.e., for $t = 0, 1, \ldots 10$) with
1. $\rho=0.5$
2. $w_1 = 0$ and $w_1 = w_2 = \ldots = w_{10} = 1$
3. $y_0 = 0$

```python
# Initialize the variable 'T' to be equal to the number of periods to simulate
T = 11

# Create new variable 'rho' equal to the value for rho
rho = 0.5

# Create new variable 'y0' equal to the initial value for y
y0 = 0

# Initialize variable 'w' as arrays of ones
w = np.ones(T-1)
w[0] = 0

# Simulate y
y = diff1_example(rho,w,y0)

# Construct figure with two line graphs plotting the trajectories of y and w with:
#    1. Plot for w in left panel and plot for y in right panel
#    2. Clear title, x-, and y-axis labels
#    3. y-axis limits [0,2.5]

# Create figure
fig = plt.figure(figsize=(12,4))

# w trajectory
ax = fig.add_subplot(1,2,1)
ax.plot(w,'-',lw=5,alpha = 0.75)
ax.set_title('$w_t$')
ax.set_ylabel('w')
ax.set_xlabel('t')
ax.set_ylim([0,2.5])
ax.grid()

# y trajectory
ax = fig.add_subplot(1,2,2)
ax.plot(y,'-',lw=5,alpha = 0.75)
ax.set_title('$y_t$ with $\\rho=0.5$')
ax.set_ylabel('y')
ax.set_xlabel('t')
ax.set_ylim([0,2.5])
ax.grid()

# Use fig.tight_layout() to manage whitespace in the figure
fig.tight_layout()
```

The cumulative effect on $y$ in period $T$ of the permanent change in $w_t$ is:

\begin{align}
y_T & = \rho^{T-1} + \rho^{T-2} + \rho^{T-3} + \cdots + \rho + 1\\
& = \frac{1}{1-\rho},
\end{align}

which, for the example above, is $2$ and that is confirmed by the simulation.

## Stability of First-Order Difference Equations

You might already see that the value of $\rho$ is important in determining the stability of the process $y$, but we can see clearly why. Suppose that $\mu=0$ (this just makes the argument more clear) and start with the equation for determining $y_1$:

\begin{align}
y_1 & = \rho y_0 + w_1
\end{align}

The value for $y_2$ is determined by an analogous equation:

\begin{align}
y_2 & = \rho y_1 + w_2
\end{align}

Then use the equation for $y_1$ to eliminate $y_1$ from the equation for $y_2$:

\begin{align}
y_2 & = \rho^2 y_0 + \rho w_1 + w_2
\end{align}

Continue this process to arrive at a formula that specifies the value of $y$ in period $T$ as a function of just $\rho$, $y_0$, and the values of $w$: $w_1, w_2, \ldots, w_T$:

\begin{align}
y_T & = \rho^T y_0 + \rho^{T-1} w_1 + \rho^{T-2} w_2 + \cdots + \rho w_{T-1} + w_T\\
\end{align}

Then, you can see that the marginal effect of $w_1$ on $y_T$ is:

\begin{align}
\frac{\partial y_T}{\partial w_1} & = \rho^{T-1}
\end{align}

From this, there are three facts:

1. If $|\rho|<1$, then the process $y$ is **stable** because $|\rho^{T-1}|\rightarrow 0$ as $T\rightarrow \infty$.
2. If $|\rho|>1$, then the process $y$ is **explosive** because $|\rho^{T-1}|\rightarrow \infty$ as $T\rightarrow \infty$.
2. If $|\rho|=1$, then the process $y$ is neither stable nor explosive because $|\rho^{T-1}|\rightarrow 1$ as $T\rightarrow \infty$.

The final case, with $|\rho^T|=1$, is closely related to the *random walk* stochastic process that is often used to describe finacial asset returns over short time horizons. We will encounter this process later.

Stability is a fundamental concept in economics and there are important roles for models with stable and explosive properties.

## Higher-order Difference Equations (Optional)

The $p$th-order linear difference equation is given by the following expression:

\begin{align}
y_t & = \rho_1 y_{t-1} + \rho_2 y_{t-2} + \cdots + \rho_p y_{t-p} + w_t,
\end{align}

where $\rho_1, \rho_2, \ldots, \rho_p$ are constants. In principle, computing the $p$th-order difference equation is the same as computing the first-order difference equation. The only difference real difference is that you need $p$ initial values of $y$ and not just 1. That is, in order to compute the period 1 value $y_1$, you need values for $y_0, y_{-1}, \ldots, y_{-(p-1)}$.

The function in the following cell simulates $p$-order difference equations so we can use it for the examples below.

```python
# Define a function that returns a DataFrame of y and w values given coefficients, y0, and an array of w values. CELL PROVIDED
def pth_order_diff(rho,y0,w):
    '''Function for computing the first-order difference equation:

            y[t] = C + rho[0]*y[t-1] + rho[1]*y[t-2] + ... + rho[p]*y[t-p] + w[t]

    Args:
        rho (NumPy ndarray or list):   Autoregressive coefficients on y[t-1], y[t-2], ... y[t-1]
        y0 (NumPy ndarray or list):    Initial value of the process (must be same length as rho)
        w (NumPy ndarray or list):     Array of exogenous values for t=0,..,T+1 where T is the end date

     Returns:
         NumPy ndarray with columns 'w' and 'y' and index: -(p-1), -(p-2), ..., T
     '''

    # Set order of the process. If rho is a list or array, set p to the length. Otherwise, p=1.
    try:
        p = len(rho)
    except:
        p = 1

    # Length of simulation
    try:
        T = len(w)
    except:
        T = 1

    # Construct series for y
    y = np.zeros(p+T-1)
    y[0:p] = y0

    # Construct series for w
    empty = np.empty(p-1)
    empty[:] = np.nan
    w = np.concatenate([empty,w[:T+1]])

    # Date index
    index = np.arange(-(p-1),T)

    # Initialize DataFrame
    df = pd.DataFrame({'w':w,'y':y},index=index)

    # Compute y
    for i in df.index[p:]:
        df.loc[i,'y'] = np.dot(rho,np.flip(df.loc[i-p:i-1,'y'])) + df.loc[i,'w']

    return df
```

### Example: A Second-Order Difference Equation

Consider the second order difference equation:

\begin{align}
y_t & = \rho_1 y_{t-1} + \rho_2 y_{t-2} + w_t,
\end{align}

As usual, take $w_1=1$ and $w_0=w_2=w_3 = \cdots$ and $y_0 = y_{-1} = 0$. Let's simulate the model with two different combinations of $\rho_1$ and $\rho_2$:

\begin{align}
y_t & = 0.5 y_{t-1} + 0.2 y_{t-2} + w_t,
\end{align}

and:

\begin{align}
y_t & = 0.6 y_{t-1} + -0.9 y_{t-2} + w_t,
\end{align}

Simulate the processes for 24 periods and, instead of using a line plot this time, use barcharts. Stack the axes for each vertically in a single figure.

```python
# Create figure that has dimensions 12x8
fig = plt.figure(figsize=(12,8))

# Initial y values
y0 = [0,0]

# End period of simulation
T = 24

# Create w array and set period 1 value
w = np.zeros(T+1)
w[1] = 1

# Specify coefficients
rho = [0.5,0.2]

# Simulate y and remove zeros from the simulated data
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(2,1,1)
ax.bar(df.index,df['y'],alpha = 0.65)
ax.set_title('$\\rho_1 ='+str(rho[0])+'$ and $\\rho_2 = '+str(rho[1])+'$',fontsize=15)
ax.grid()

# Specify coefficients
rho = [0.6,-.9]

# Simulate y
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(2,1,2)
ax.bar(df.index,df['y'],alpha = 0.65)
ax.set_title('$\\rho_1 ='+str(rho[0])+'$ and $\\rho_2 = '+str(rho[1])+'$',fontsize=15)
ax.grid()
```

Apparently under both parameterizations the second-order difference equation is stable. But change the sign of $\rho_2$ in the last simulation so that the model becomes:

\begin{align}
y_t & = 0.6 y_{t-1} + 0.9 y_{t-2} + w_t,
\end{align}

```python
# Create figure that has dimensions 12x4
fig = plt.figure(figsize=(12,4))

# Initial y values
y0 = [0,0]

# End period of simulation
T = 24

# Create w array and set period 1 value
w = np.zeros(T+1)
w[1] = 1

# Specify coefficients
rho = [0.6,0.9]

# Simulate y and remove zeros from the simulated data
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(1,1,1)
ax.bar(df.index,df['y'],alpha = 0.65)
title = ''
for n,r in enumerate(rho):
    title+='$\\rho_'+str(n+1)+' = '+str(round(r,4))+'$, '
ax.set_title(title,fontsize=15)
ax.grid()
```

Evidently reversing the sign on the second coefficient has caused the model to become explosive. Determining the stability of second-order and higher-order difference equations is not trivial and requires math that is beyond what most students in the course will have seen and I won't go into the topic here. See Chapter 1 of James Hamilton's book [*Time Series Analysis*](https://press.princeton.edu/titles/5386.html]).

However, as the examples suggest, simulating a process can give strong clues about whether a process is stable.

### A Fifth-Order Difference Equation

Now consider the following fifth-order difference equation difference equation:

\begin{align}
y_t & =  -0.3948 y_{t-1} - 0.3719 y_{t-2} - 0.4126 y_{t-3} - 0.3845y_{t-4} +  0.2477 y_{t-5} + w_t
\end{align}

The coefficients were actually chosen randomly using NumPy as you can see in the code below.

```python
# Create figure that has dimensions 12x4
fig = plt.figure(figsize=(12,4))

# Initial y values
y0 = [0,0,0,0,0]

# End period of simulation
T = 28

# Create w array and set period 1 value
w = np.zeros(T+1)
w[1] = 1

# Specify coefficients
np.random.seed(126)
rho = np.random.uniform(size=5)-0.5
# rho = [0.5,0.2]

# Simulate y and remove zeros from the simulated data
df = pth_order_diff(rho,y0,w)

# Remove zeros from the simulated data
df = df.replace(0,np.nan)

# Construct barchart of y values
ax = fig.add_subplot(1,1,1)
ax.bar(df.index,df['y'],alpha = 0.65)
title = ''
for n,r in enumerate(rho):
    title+='$\\rho_'+str(n+1)+' = '+str(round(r,4))+'$, '
ax.set_title(title,fontsize=15)
ax.grid()
```


# TOC

__Lab 07 - Classification__
1. [Import](#Import)
1. [Multi-class classification](#Multi-class-classification)
    1. [Example 1 - Distance from planes](#Distance-from-planes-Example1)
    1. [Example 2 - Iris classification w/ logistic regression](#Iris-classification-w/-logistic-regression-Example2)

# Import

```python
import numpy as np

np.set_printoptions(threshold=np.inf, suppress=True)
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline
import seaborn as sns
import warnings

warnings.simplefilter("ignore")
from IPython.core.display import display, HTML

display(HTML("<style>.container { width:78% !important; }</style>"))

import os
import itertools

from sklearn.neighbors import NearestNeighbors
from sklearn.datasets import load_iris, fetch_mldata
from sklearn.model_selection import train_test_split, KFold
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler

sns.set(style="whitegrid", font_scale=1.3)
```

# Multi-class classification

Tne method for classifying an observation in a multi-class problem is to calculate the perpendicular distance of the observation from each of the separating planes. The shortest distance is the class selection.

## Example 1 - Distance from planes
Consider a three-class classification problem with the following separating hyperplanes:

$$
\begin{align}
x_1 + x_2 - 2 = 0 & & 1^{\text{st}} \text{ class}\\
-4x_1 + 3x_2 - 11 = 0 & & 2^{\text{nd}} \text{ class}\\
-x_1 - 3x_2 - 1 = 0 & & 3^{\text{rd}} \text{ class}
\end{align}
$$

Calculate the score (unnormalized perpendicular distance) for each class for the test case $(x_1, x_2)=(0, 1)$ .

```python
# plot planes and observation
x = np.arange(-3, 5)
plt.plot(x, (2.0 - 1 * x) / 1.0, label="1st", lw=5)
plt.plot(x, (11.0 + 4 * x) / 3.0, label="2nd", lw=5)
plt.plot(x, (-1.0 - 1 * x) / 3.0, label="3rd", lw=5)

plt.gca().set_aspect("equal", adjustable="box")
plt.scatter([0], [1], c="k", s=100)
plt.xlim([-3, 4])
plt.ylim([-3, 4])
plt.grid()
plt.legend()
plt.show()
```

```python
# build arrays for weights, bias units and observations
W = np.transpose(np.array([[1.0, 1.0], [-4, 3.0], [-1.0, -3.0]])).reshape(2, 3)
b = np.array([-2.0, -11.0, -1.0])

# test point
x = np.array([0.0, 1.0])
```

```python
# review data
print(W)
print(b)
print(x)
```

    [[ 1. -4. -1.]
     [ 1.  3. -3.]]
    [ -2. -11.  -1.]
    [0. 1.]

```python
# determine distance from each plane
def scores(W, b, x):
    return np.dot(x, W) + b

scores(W, b, x)
```

    array([-1., -8., -4.])

> Observations is closest to class 1

```python
# determine probabilities associated with predictions
def probas(W, b, x):
    sc = scores(W, b, x)
    scExp = np.exp(sc)
    return scExp / np.sum(scExp)

np.round(probas(W, b, x), 5)
```

    array([0.95175, 0.00087, 0.04738])

> .95 probability that observatation belongs to class one

```python
# generate the answer
def predict_class(W, b, x):
    prob = probas(W, b, x)
    return np.argmax(prob) + 1

predict_class(W, b, x)
```

    1

> Class 1 is chosen

## Example 2 - Iris classification w/ logistic regression
Build a classifier to identify an iris sample as belonging to one of three iris classes based on four features.

```python
# load data
iris = load_iris()
```

```python
# create observation and label datasets
X = pd.DataFrame(iris.data, columns=iris.feature_names)
y = iris.target
```

```python
# nspect features
X.head()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)

      0
      5.1
      3.5
      1.4
      0.2

      1
      4.9
      3.0
      1.4
      0.2

      2
      4.7
      3.2
      1.3
      0.2

      3
      4.6
      3.1
      1.5
      0.2

      4
      5.0
      3.6
      1.4
      0.2

```python
# inspect feature summary statistics
X.describe()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)

      count
      150.000000
      150.000000
      150.000000
      150.000000

      mean
      5.843333
      3.057333
      3.758000
      1.199333

      std
      0.828066
      0.435866
      1.765298
      0.762238

      min
      4.300000
      2.000000
      1.000000
      0.100000

      25%
      5.100000
      2.800000
      1.600000
      0.300000

      50%
      5.800000
      3.000000
      4.350000
      1.300000

      75%
      6.400000
      3.300000
      5.100000
      1.800000

      max
      7.900000
      4.400000
      6.900000
      2.500000

>All the features here are number numberal features

__EDA__

```python
# correlation matrix
class_labels = ["setosa", "versicolour", "virginica"]

corr = X.corr()
mask = np.zeros_like(corr, dtype=bool)
mask[np.triu_indices_from(mask)] = True

f, ax = plt.subplots(figsize=(20, 20))
cmap = sns.diverging_palette(220, 10, as_cmap=True)
sns.heatmap(
    corr,
    mask=mask,
    cmap=cmap,
    vmin=-1.0,
    vmax=1.0,
    square=True,
    linewidths=0.5,
    cbar_kws={"shrink": 0.5},
    ax=ax,
)
```

>Petal width and petal length are strongly positively correlated. Petal length and width are somewhat negatively correlated with sepal width.

```python
# pair-wise plots
sns.pairplot(sns.load_dataset("iris"), hue="species", size=3)
```

>The correlations describes above are apparent. Additionaly the setosa class (identified by blue dots) is often completely separated from the versicolor and virginica classes.

```python
# train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

```python
# fitting
log_reg = LogisticRegression(C=1e6, multi_class="multinomial", solver="lbfgs")
log_reg.fit(X_train, y_train)
```

    LogisticRegression(C=1000000.0, class_weight=None, dual=False,
                       fit_intercept=True, intercept_scaling=1, l1_ratio=None,
                       max_iter=100, multi_class='multinomial', n_jobs=None,
                       penalty='l2', random_state=None, solver='lbfgs', tol=0.0001,
                       verbose=0, warm_start=False)

```python
# model evaluation
plt.figure(figsize=(12, 8))
plt.bar(
    np.arange(log_reg.coef_.shape[1]) - 0.2,
    log_reg.coef_[0],
    color="red",
    width=0.2,
    label="setosa",
)
plt.bar(
    np.arange(log_reg.coef_.shape[1]) - 0.0,
    log_reg.coef_[1],
    color="green",
    width=0.2,
    label="versicolour",
)
plt.bar(
    np.arange(log_reg.coef_.shape[1]) + 0.2,
    log_reg.coef_[2],
    color="blue",
    width=0.2,
    label="virginica",
)
plt.xticks(np.arange(log_reg.coef_.shape[1]), X.columns, rotation=90)
plt.xlim(-1, log_reg.coef_.shape[1])
plt.title("Model coefficients")
plt.legend(loc="lower right")
```

>The most pronounced features in terms of the coefficients are pedal length and pedal width. This is consistent with then pairwise plots above in the these features seem to have clear divisions between each class.

```python
# test set predictions
y_preds_train = log_reg.predict(X_train)
y_preds_test = log_reg.predict(X_test)
```

```python
# review accuracy
print("Train accuracy: ", np.round(accuracy_score(y_train, y_preds_train), 3))
print("Test accuracy: ", np.round(accuracy_score(y_test, y_preds_test), 3))
```

    Train accuracy:  0.983
    Test accuracy:  1.0

```python
# log results
results = pd.DataFrame(columns=["Accuracy"])
results = results.append(
    pd.DataFrame(
        [
            np.round(accuracy_score(y_train, y_preds_train), 3),
            np.round(accuracy_score(y_test, y_preds_test), 3),
        ],
        columns=["Accuracy"],
        index=["Sklearn Train", "Sklearn Test"],
    )
)
results
```

      Accuracy

      Sklearn Train
      0.983

      Sklearn Test
      1.000

__Custom function__

```python
# custom function
def plot_confusion_matrix(
    cm, classes, normalize=False, title="Confusion matrix", cmap=plt.cm.Blues
):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype("float") / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print("Confusion matrix, without normalization")

    plt.imshow(cm, interpolation="nearest", cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=90)
    plt.yticks(tick_marks, classes)

    fmt = ".2f" if normalize else "d"
    thresh = cm.max() / 2.0
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(
            j,
            i,
            format(cm[i, j], fmt),
            horizontalalignment="center",
            color="white" if cm[i, j] > thresh else "black",
        )

    plt.tight_layout()
    plt.ylabel("True label")
    plt.xlabel("Predicted label")
```

```python
# confusion matrix
np.set_printoptions(precision=2)

# plot normalized confusion matrix
plt.figure(figsize=(10, 10))
plot_confusion_matrix(
    cm=confusion_matrix(y_test, y_preds_test),
    classes=class_labels,
    normalize=True,
    title="Normalized confusion matrix - Test set",
)

plt.show()
```

> The test class is perfectly classified


```python
%matplotlib inline
%run ../setup/nb_setup
```

# Collections of Orbits 2: Self-consistent stellar systems

Author(s): Helmer Koppelman

## Introduction

In this notebook, we will generate initial conditions (positions and velocities) for a stellar system of $10^4$ stars. We will assume that the density of the system is that of a Plummer profile with mass $M = 10^6 {\rm M}_\odot$ and scale $b = 1$ kpc. Several methods exist to draw positions $(x,y,z)$ such that $r = \sqrt{x^2+y^2+z^2}$ is distributed according to $\rho(r)$. Below we will follow the method described in the Appendix of [Aarseth, Henon, Wielen (1974)](https://ui.adsabs.harvard.edu/abs/1974A%26A....37..183A/abstract).

In most cases, the difficulty with generating mock galaxies is not in the sampling of the spatial coordinates $\vec{x} = (x,y,z)$ but rather in sampling velocities $\vec{v} = (v_x,v_y,v_z)$ that are consistent with the spatial distribution. To ensure consistency we will need a distribution function $f(\vec{x}, \vec{v})$ such that $\rho(r) \propto \int f(\vec{x}, \vec{v})~{\rm d}^3v$ – such a system is called **self-consistent** because the distribution function $f$ generates the density, which in turn generates the gravitational potential.

### Notebook Setup and Package Imports

```python
# Essential - general
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import quad

# Essential - astro
import astropy.units as u
import gala.integrate as gi
import gala.dynamics as gd
import gala.potential as gp
from gala.units import galactic
from astropy.constants import G

# Style
import matplotlib as mpl
import matplotlib.cm as cm
```

## Sampling a system following a Plummer profile

For the Plummer profile, we can use a simple distribution function $f(\mathcal{E})$, known as a polytrope of order $n$, that is associated with this density profile for $(n=5)$ to generate the velocities.

### Useful equations

---
The distribution function of a polytrope of order $n$
$
\begin{equation}
f(\mathcal{E}) =  \begin{cases}
    F \mathcal{E}^{n-\frac{3}{2}} ~ &(\mathcal{E} \gt 0) \\
    0 ~ &(\mathcal{E}\leq 0)
    \end{cases}
\end{equation}
$

where $\mathcal{E}$ and $\Psi$ are the relative energy and relative potential, defined as
$
\begin{align}
    \Psi &\equiv \Phi_0 - \Phi \\
    \mathcal{E} &\equiv \Psi - \frac{1}{2}v^2
\end{align}
$

where $\Phi$ is the regular potential and $\Phi_0 = 0$ for an isolated system that extends to infinity. For such a system, $\Psi$ then is related to the density through Poisson's equation

$
\begin{equation}
\nabla^2\Psi = -4\pi G\rho
\end{equation}
$

---

The density distribution of a Plummer sphere is given by
$
\begin{align}
\rho(r) &= \frac{3M}{4\pi b^3} \bigg(1+\frac{r^2}{b^2}\bigg)^{-\frac{5}{2}}, \\
     &= c_n \Psi^n \quad (\Psi\gt 0)
\end{align}
$

where $c_n = \frac{(2\pi)^{\frac{3}{2}}(n-\frac{3}{2})!F}{n!}$

---

### Exercises (optional):
* Derive the expression for $\Psi(r)$ from the density.
* Given that $\Phi(r) = -\frac{GM}{\sqrt{r^2+b^2}}$, derive the expression for $F$

### Generating a stellar system from a Normal distribution
Before generating a self-consistent system we will run a test with a system that is not self-consistent to see what this means. We will simply sample initial conditions from a Gaussian distribution for simplicity.

```python
# Example generating a stellar system from a Gaussian distribution.

rng = np.random.default_rng(3)

N = 10_000
x, y, z = rng.normal(loc=0, scale=2, size=(3, N))
vx, vy, vz = rng.normal(loc=0, scale=1, size=(3, N))
```

### Now we will inspect whether the generated system is self-consistent with a Plummer model or not.
(probably not since it is a random Gaussian)

```python
# First we generate a `gala.dynamics.PhaseSpacePosition`
pos = [x, y, z] * u.kpc
vel = [vx, vy, vz] * u.km / u.s

w0 = gd.PhaseSpacePosition(pos=pos, vel=vel)
```

```python
# Next we integrate the distribution in a Plummer potential for 1 Gyr

M = 1e6 * u.Msun
b = 1 * u.kpc

potential = gp.PlummerPotential(m=M, b=b, units=galactic)
Hamiltonian = gp.Hamiltonian(potential)

orbits = Hamiltonian.integrate_orbit(
    w0, t=np.linspace(0, 1, 100) * u.Gyr, Integrator=gi.DOPRI853Integrator
)
```

```python
# Inspecting the evolution of the distribution:
# if the system is self-consistent it's shape should not evolve.
r = orbits.spherical.distance
r, r_unit = r.value, r.unit

plt.figure(figsize=(8, 5))
args = dict(histtype="step", lw=2, bins=100, range=(0, 10))
plt.hist(r[0], label="t = 0", **args)
plt.hist(r[-1], label="t = {}".format(orbits.t[-1].to("Gyr")), **args)
plt.legend(frameon=False)

plt.xlabel(f"r [{r_unit}]")
```

### Exercises

Inspect the system:
- Compute the velocity dispersion in each of the directions $(x,y,z)$. What is the shape of the velocity dispersions? Why?
- Can you think of a better test of self-consistency? You could look at the evolution of the size of the system – or at the amplitude of its velocity distribution $\sigma_v$.

### Generating a self-consistent Plummer

We will now generate a self-consistent initial conditions for a Plummer model using the relevant section in [Aarseth, Henon, Wielen (1974)](https://ui.adsabs.harvard.edu/abs/1974A%26A....37..183A/abstract)

For a spherical system we can obtain $M(r)$ from the density

$
\begin{equation}
    M(r) = 4\pi \int_0^r x^2 \rho(x)~{\rm d}x
\end{equation}
$

For the Plummer, this results in
$
\begin{equation}
     M(r) = \frac{M}{b^3} r^3 (1+\frac{r^2}{b^2})^{\frac{3}{2}},
\end{equation}
$
where $M$ is the total mass of the system.

The general expression for the escape velocity is (if $\Phi(r) \rightarrow 0$ for $r \rightarrow \infty$)
$
\begin{equation}
    v_e(r) \equiv \sqrt{2|\Phi(r)|}
\end{equation}
$

the potential of a Plummer profile is given by
$
\begin{equation}
    \Phi(r) = - \frac{GM}{\sqrt{r^2 + b^2}}
\end{equation}
$

```python
M = 1e6 * u.Msun
b = 1 * u.kpc
```

### Exercise:
* Modify the code below according to the steps described above. \
The solution is given below.

Hint: if it helps you can sample `q` from a beta distribution (and skip the von Neumann rejection technique) as follows
```
q = np.sqrt(rng.beta(1.5,4.5, size=N))
```

  Click to expand!

```Python
def selfconsistent_plummer(N, M, b):
    """This function generates a stellar system that is consistent
    with a Plummer, following (more or less)
    the steps in Aarseth, Henon, Wielen (1974)

    Try to code this up yourself - if you have time."""
    x1 = rng.uniform(0, 1, size=N*2)

    r = b * (x1**(-2/3) -1)**-0.5
    r = r[r<5*u.kpc][0:N]

    xyz = rng.normal(size=(3,N))
    pos = xyz * r.value / np.linalg.norm(xyz, axis=0) * r.unit

    def v_esc(r):
        r = r if isinstance(r, u.Quantity) else r*u.kpc
        pot_abs = G*M / np.sqrt(r**2+b**2)
        return np.sqrt(2 * pot_abs).to('km/s')

    q = np.sqrt(rng.beta(1.5,4.5, size=N))
    v = q * v_esc(r)

    v_xyz = rng.normal(size=(3,N))
    vel = v_xyz * v.value / np.linalg.norm(v_xyz, axis=0)*v.unit

    return pos, vel
```

```python
# Repeating the test of self-consistency
pos, vel = selfconsistent_plummer(N, M, b)
w0 = gd.PhaseSpacePosition(pos=pos, vel=vel)

orbits = Hamiltonian.integrate_orbit(
    w0, t=np.linspace(0, 1, 100) * u.Gyr, Integrator=gi.DOPRI853Integrator
)

r = orbits.spherical.distance
r, r_unit = r.value, r.unit

plt.figure(figsize=(8, 5))
plt.hist(r[0], histtype="step", lw=2, bins=100, label="t = 0", range=(0, 10))
plt.hist(
    r[-1],
    histtype="step",
    lw=2,
    bins=100,
    label="t = {}".format(orbits.t[-1].to("Gyr")),
    range=(0, 10),
)
plt.legend(frameon=False)

plt.xlabel(f"r [{r_unit}]")
```

### Exercises

Inspect the system:
- Inspect (by plotting) the distribution of the energies
- Compute the velocity dispersion in each of the directions $(x,y,z)$. What is the shape of the velocity dispersions? Why?
- Can you think of a better test of self-consistency? You could look at the evolution of the size of the system – or at the amplitude of its velocity distribution $\sigma_v$.

##  A small galaxy embedded in a dark matter halo
For the second part we will generate a two-component system: a spherical galaxy in a dark matter halo. For simplicity, assume that both components are described by a Plummer profile.
For the stellar component use $M_\star = 10^4 {\rm M}_\odot$ and scale $b_\star= 0.05$ kpc and for the dark matter component use $M_{\rm DM} = 10^6 {\rm M}_\odot$ and scale $b_{\rm DM} = 1$ kpc.

```python
M_star, b_star = 1e3 * u.Msun, 0.05 * u.kpc
M_dm, b_dm = 1e6 * u.Msun, 1 * u.kpc

p_star = gp.PlummerPotential(m=M_star, b=b_star, units=galactic)
p_dm = gp.PlummerPotential(m=M_dm, b=b_dm, units=galactic)
```

Use the same method as before to generate the spatial distribution of the stellar component.

To obtain the velocities of the stars we will have to solve the Jeans equations.
The Jeans equation in spherical coordinates are (Section 4.8.1 of Binney & Tremaine):
$ \large \frac{d\big(\rho_\star(r)\sigma^2_v(r)\big)}{dr} + \rho_\star(r) \frac{d\Phi(r)}{dr}= 0$

Use the equation above to find the velocity dispersion of the system as a function of radius $\sigma_v(r)$.
Assume that the potential is dominated by the dark matter component. For the density in the equation above use the density of the stellar component.

The equation above can be transformed to
$
\begin{equation}
\sigma_v^2(r) = \frac{1}{\rho_\star(r)} \int_r^\infty \rho_\star(r') \frac{{\rm d}\Phi(r')}{{\rm d}r'} ~ dr'
\end{equation}
$

Which can be solved with the code below

```python
# Solving the integral for sigma_v(r):
def integrand(x):
    b1, b2 = b_star.value, b_dm.value

    p1 = np.power(1 + (x / b1) ** 2, -5 / 2)
    p2 = np.power(x ** 2 + b2 ** 2, -3 / 2)
    return p1 * p2 * x

def sigma_v(r):
    # strip r from unit
    r, unit = (r.value, r.unit) if isinstance(r, u.Quantity) else (r, u.kpc)

    int_, err = quad(integrand, r, np.infty)
    int_ = int_ / unit

    sv_2 = int_ * G * M_dm * (1 + ((r * unit) / b_star) ** 2) ** (5 / 2)
    return np.sqrt(sv_2).to("km/s")
```

Finally, by assuming that the velocity distribution is well described by a Gaussian, generate velocities for the stellar particles using the velocity dispersion obtained from the Jeans equation.

### Generating a self-consistent system for a 2-component Plummer model

  Click to expand!

```Python
# Generating the IC
def generate_2comp_Plummer(N_star):
    xyz, _ = selfconsistent_plummer(N_star, M_star, b_star)
    r = np.linalg.norm(xyz, axis=0)

    s_v = np.array([sigma_v(ri).value for ri in r])
    v = rng.normal(scale=s_v, size=N_star)

    v_xyz = rng.normal(size=(3, N_star))
    v_xyz = v * v_xyz / np.linalg.norm(v_xyz, axis=0) * u.km/u.s
    return xyz, v_xyz
```

```python
xyz, v_xyz = generate_2comp_Plummer(N_star=10_000)
```

```python
# Repeating the test of self-consistency
w0 = gd.PhaseSpacePosition(pos=xyz, vel=v_xyz)

potential = gp.CompositePotential(star=p_star, dm=p_dm)
Hamiltonian = gp.Hamiltonian(potential)

orbits = Hamiltonian.integrate_orbit(
    w0, t=np.linspace(0, 1, 100) * u.Gyr, Integrator=gi.DOPRI853Integrator
)

r = orbits.spherical.distance
r, r_unit = r.value, r.unit

plt.figure(figsize=(8, 5))
plt.hist(r[0], histtype="step", lw=2, bins=100, label="t = 0", range=(0, 0.5))
plt.hist(
    r[-1],
    histtype="step",
    lw=2,
    bins=100,
    label="t = {}".format(orbits.t[-1].to("Gyr")),
    range=(0, 0.5),
)
plt.legend(frameon=False)

plt.xlabel(f"r [{r_unit}]")
```

### Exercise

As you can see, the system is not fully self-consistent. There appear to be two regimes with a transition at $\sim 0.05$ kpc. Do you know why? Can you confirm your suspicion?

NB: we only generated the stellar component of the two-component galaxy.

### Optional exercise:
- Generate the dark matter component using the code in the first part of this notebook and by assuming that it is not sensitive to the stellar part (i.e. just generating a Plummer with the dark matter parameters, ignoring the stars).
- Use the two component galaxy in the next notebook.


# Sequential Quadratic Programming

## Quadratic Programming

An optimization problem with a quadratic objective function and linear
constraints is called a _quadratic program_. Problems of this
type are important in their own right, and they also arise a subproblems
in methods for general constrained optimization such as sequential
quadratic programming (this chapter) and interior-point methods (lecture 10).

The general quadratic program can be stated as

\begin{align*}
\min_{\vec{x}}\, & f\left(\vec{x}\right)\overset{\vartriangle}{=}\frac{1}{2}\vec{x}^\mathsf{T}Q\vec{x}-\vec{c}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & \begin{cases}
A_{\textrm{eq}}\vec{x}=\vec{b}_{\textrm{eq}}\,,\\
A_{\textrm{in}}\vec{x}\leq\vec{b}_{\textrm{in}}\,,
\end{cases}
\end{align*}

where $Q$ is a symmetric $n\times n$ matrix, $\vec{c}\in\mathbb R^{n}$,
$A_{\textrm{eq}}$ is a $m\times n$ matrix, $\vec{b}_{\textrm{eq}}\in\mathbb R^{m}$,
$A_{\textrm{in}}$ is a $p\times n$ matrix and $\vec{b}_{\textrm{in}}\in\mathbb R^{p}$.
If the Hessian matrix $Q$ is positive semidefinite, we have a convex
quadratic program. Non-convex quadratic programs, in which $Q$ is
an indefinite matrix, can be more challenging because they have several
stationary points and local minima.

## Equality Constraints

We begin our discussion of algorithms for quadratic programming by considering
the case in which only equality constraints are present. We consider
to following equality-constrained quadratic problem
\begin{align}
\min_{\vec{x}}\, & f\left(\vec{x}\right)\overset{\vartriangle}{=}\frac{1}{2} \vec{x}^\mathsf{T}Q\vec{x}- \vec{c}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & A\vec{x}=\vec{b}\,,\nonumber
\end{align}
where $A$ is the $m\times n$ Jacobian of constraints and $\vec{b}\in\mathbb R^{m}$.
We assume that $A$ has rank $m$ so that the constraints are consistent.

The First Order Necessary Condition (FONC) uses the Langrangian function
\begin{equation}
\mathcal L\left(\vec x, \vec \lambda\right)=\vec{x}^\mathsf{T}Q\vec{x}- \vec{c}^\mathsf{T}\vec{x} + \vec\lambda^\mathsf{T} \left(A\vec{x}-\vec{b}\right)
\end{equation}
and gives the following condition
\begin{align}
\nabla_{\vec x}\mathcal L\left(\vec x ^\star, \vec \lambda ^\star\right)&=\vec 0\,,\\
\nabla_{\vec \lambda}\mathcal L\left(\vec x ^\star, \vec \lambda ^\star\right)&=\vec 0\,,
\end{align}
So the FONC for $\vec{x}^{\star}$ to be a solution of the quadratic
problem yields a vector $\vec{\lambda}^{\star}$ such
that the following system of equations is satisfied:
\begin{equation}
\begin{pmatrix}Q & A^\mathsf{T}\\
A & 0
\end{pmatrix}\begin{pmatrix}\vec{x}^{\star}\\
\vec{\lambda}^{\star}
\end{pmatrix}=\begin{pmatrix}\vec{c}\\
\vec{b}
\end{pmatrix}\,.\label{eq:equality_constraint}
\end{equation}
This system can be solved directly by factorization. An alternative
is to use an iterative method.

Consider the quadratic programming problem
\begin{align*}
\min_{\vec{x}}\, & \frac{1}{2} \vec{x}^\mathsf{T}\begin{pmatrix}6 & 2 & 1\\
2 & 5 & 2\\
1 & 2 & 4
\end{pmatrix}\vec{x}- \begin{pmatrix}8\\
3\\
3
\end{pmatrix}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & \begin{pmatrix}1 & 0 & 1\\
0 & 1 & 1
\end{pmatrix}\vec{x}=\begin{pmatrix}3\\
0
\end{pmatrix}\,.
\end{align*}

```julia
Q = [6 2 1
     2 5 2
     1 2 4]
c = [8, 3, 3]
A = [1 0 1
     0 1 1]
b = [3, 0]
sol = [Q transpose(A)
       A zeros(2,2)] \ [c; b]
```

    5-element Array{Float64,1}:
      2.0
     -1.0
      1.0
     -3.0
      1.9999999999999996

The solution $\vec{x}^{\star}$ and optimal Lagrange multiplier vector $\vec \lambda^\star$ are given by
\begin{align*}
\vec{x}^{\star} & = \begin{pmatrix}2 & -1 & 1\end{pmatrix}^\mathsf{T}\,,\\
\vec{\lambda}^{\star} & = \begin{pmatrix}-3 & 2\end{pmatrix}^\mathsf{T}\,.
\end{align*}

## Active Set Method

We now describe active-set methods for solving quadratic programs
containing both equality and inequality constraints. We consider only
the convex case, in which the matrix $Q$ is positive semidefinite.

If the contents of the optimal active set $J\left(\vec x^\star\right)$ were known in advance, we could find the solution $\vec{x}^{\star}$
by applying the technique for equality constrained quadratic programs
to the problem
\begin{align*}
\min_{\vec{x}}\, & f\left(\vec{x}\right)\overset{\vartriangle}{=}\frac{1}{2} \vec{x}^\mathsf{T}Q\vec{x}- \vec{c}^\mathsf{T}\vec{x}\\
\textrm{subject to}\, & \begin{cases}
A_{\textrm{eq}}\vec{x}=\vec{b}_{\textrm{eq}}\,,\\
 \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}=b_{\textrm{in},j}\,, & \forall j\in J\left(\vec{x}^{\star}\right)\,,
\end{cases}
\end{align*}
where $\vec{a}_{\textrm{in},j}$ is the $j$th row in the matrix $A_{\textrm{in}}$
and $b_{\textrm{in},j}$ is the $j$th element of the vector $\vec{b}_{\textrm{in}}$.
Of course, we usually do not have prior knowledge of $J\left(\vec{x}^{\star}\right)$
and determination of this set is the main challenge facing algorithms
for inequality-constrained quadratic programs.

Active-set methods find a step from one iterate to the next by solving
a quadratic subproblem in which some of the inequality constraints,
and all the equality constraints are imposed as equalities. This subset
is referred to as the _working set_ and is denoted at the $k$th
iterate by $W_{k}$. An important requirement we impose on $W_{k}$
is that the gradients $\vec{a}_{\textrm{eq},i}$, $i=1,\dots,m$ and
$\vec{a}_{\textrm{in},j}$, $j\in W_{k}$ are linearly independent,
even when the full set of active constraints at that point has linearly
dependent gradients.

Given an iterate $\vec{x}^{\left(k\right)}$ and the working set $W_{k}$,
we first check whether $\vec{x}^{\left(k\right)}$ minimizes the quadratic
function $f$ in the subspace defined by the working set. If not,
we compute a step $\vec{d}^{\left(k\right)}$ by solving an equality-constrained
quadratic subproblem in which the inequality constraints corresponding
to the working set $W_{k}$ are regarded as equalities and the other
inequality constraints are temporarily disregarded. To express this
subproblem in terms of the step $\vec{d}^{\left(k\right)}$, we define
 \begin{equation}
\vec{d}^{\left(k\right)}=\vec{x}^{\left(k+1\right)}-\vec{x}^{\left(k\right)}\,,\quad\vec{g}_{k}=Q\vec{x}^{\left(k\right)}-\vec{c}\,.
 \end{equation}
By substituting for $\vec{x}^{\left(k+1\right)}$ into the objective
function, we find that
 \begin{equation}
f\left(\vec{x}^{\left(k+1\right)}\right)=f\left(\vec{x}^{\left(k\right)}+\vec{d}^{\left(k\right)}\right)=\frac{1}{2}\left(\vec{d}^{\left(k\right)}\right)^\mathsf{T}Q\vec{d}^{\left(k\right)}+\vec{g}_{k}^\mathsf{T}\vec{d}^{\left(k\right)}+\rho_{k}\,,
 \end{equation}
where $\rho_{k}=\frac{1}{2} \left(\vec{x}^{\left(k\right)}\right)^\mathsf{T}Q\vec{x}^{\left(k\right)}- \vec{c}^\mathsf{T}\vec{x}^{\left(k\right)}$
is independent of $\vec{d}^{\left(k\right)}$. Since we can drop $\rho_{k}$
from the objective function without changing the solution of the problem,
we can write the subproblem to be solved at the $k$th iteration as
follows
\begin{align}
\min_{\vec{d}^{\left(k)\right)}}\, & \frac{1}{2} \left(\vec{d}^{\left(k\right)}\right)^\mathsf{T}Q\vec{d}^{\left(k\right)}+ \vec{g}_{k}^\mathsf{T}\vec{d}^{\left(k\right)}\label{eq:quadratic_subproblem}\\
\textrm{subject to}\, & \begin{cases}
A_{\textrm{eq}}\vec{d}^{\left(k\right)}=\vec{0}\,,\\
 \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}=0\,, & \forall j\in W_{k}\,.
\end{cases}\nonumber
\end{align}
Note that for each $j\in W_{k}$, the value of $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}$
does not change as we move along $\vec{d}^{\left(k\right)}$, since
we have $ \vec{a}_{\textrm{in},j}^\mathsf{T}\left(\vec{x}^{\left(k\right)}+\alpha\vec{d}^{\left(k\right)}\right)= \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}=b_{\textrm{in},j}$
for all $\alpha$. Since the constraints in $W_{k}$ were satisfied
at $\vec{x}^{\left(k\right)}$, they are also satisfied at $\vec{x}^{\left(k\right)}+\alpha\vec{d}^{\left(k\right)}$,
for any value of $\alpha$.

Supposing for the moment that the optimal $\vec{d}^{\left(k\right)}$
is nonzero, we need to decide how far to move along this direction.
If $\vec{x}^{\left(k\right)}+\vec{d}^{\left(k\right)}$ is feasible
with respect to all the constraints, we set $\vec{x}^{\left(k+1\right)}=\vec{x}^{\left(k\right)}+\vec{d}^{\left(k\right)}$.
Otherwise, we set
 \begin{equation}
\vec{x}^{\left(k+1\right)}=\vec{x}^{\left(k\right)}+\alpha_{k}\vec{d}^{\left(k\right)}\,,
 \end{equation}
where the step-length parameter $\alpha_{k}$ is chosen to be the
largest value in the range $\left[0,1\right]$ for which all constraints
are satisfied. We can derive an explicit definition of $\alpha_{k}$
by considering what happens to the constraints $j\notin W_{k}$, since
the constraints $j\in W_{k}$ will certainly be satisfied regardless
of the choice of $\alpha_{k}$. If $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}\leq0$
for some $j\notin W_{k}$, then for all $\alpha_{k}\geq0$, we have
$ \vec{a}_{\textrm{in},j}^\mathsf{T}\left(\vec{x}^{\left(k\right)}+\alpha_{k}\vec{d}^{\left(k\right)}\right)\leq \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}\leq b_{\textrm{in},j}$.
Hence, constraint $j$ will be satisfied for all nonnegative choices
of the step-length parameter. Whenever $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}>0$
for some $j\notin W_{k}$, however, we have that $ \vec{a}_{\textrm{in},j}^\mathsf{T}\left(\vec{x}^{\left(k\right)}+\alpha_{k}\vec{d}^{\left(k\right)}\right)\leq b_{\textrm{in},j}$
only if
 \begin{equation}
\alpha_{k}\leq\frac{b_{\textrm{in},j}- \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}}{ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}}\,.
 \end{equation}
To maximize the decrease in $f$, we want $\alpha_{k}$ to be as large
as possible in $\left[0,1\right]$ subject to retaining feasibility,
so we obtain the following definition
 \begin{equation}
\alpha_{k}\overset{\textrm{def}}{=}\min\left\{ 1,\min_{j\notin W_{k}, \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}>0}\frac{b_{\textrm{in},j}- \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{x}^{\left(k\right)}}{ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}}\right\} \,.
 \end{equation}
We call the constraints $j$ for which this minimum is achieved the
_blocking constraints_. Note that it is quite possible for $\alpha_{k}$
to be zero, because we could have $ \vec{a}_{\textrm{in},j}^\mathsf{T}\vec{d}^{\left(k\right)}>0$
for some constraint $j$ that is active at $\vec{x}^{\left(k\right)}$
but not a member of the current working set $W_{k}$.

If $\alpha_{k}<1$, that is, the step along $\vec{d}_{k}$ was blocked
by some constraint not in $W_{k}$, a new working set $W_{k+1}$ is
constructed by adding one of the blocking constraints to $W_{k}$.
We continue to iterate in this manner, adding constraints to the working
set until the subproblem has solution $\vec{d}^{\circ}=\vec{0}$.
Since $\vec{d}^{\circ}=\vec{0}$ satisfy the optimality condition, we have that
 \begin{equation}
\sum_{i=1}^{m}\lambda_{i}^{\circ}\vec{a}_{\textrm{eq},i}+\sum_{j\in W^{\circ}}\mu_{j}^{\circ}\vec{a}_{\textrm{in},j}=-\vec{g}^{\circ}=-Q\vec{x}^{\circ}+\vec{c}\,,
 \end{equation}
for some Lagrange multipliers $\lambda_{i}^{\circ}$, $i=1,\dots,m$
and $\mu_{j}^{\circ}$, $j\in W^{\circ}$. It follows that $\vec{x}^{\circ}$,
$\vec{\lambda}^{\circ}$ and $\vec{\mu}^{\circ}$ satisfy the second
KKT condition, if we define the multipliers corresponding to the inequality
constraints not in the working set to be zero. Because of the control
imposed on the step length, $\vec{x}^{\circ}$ is also feasible with
respect to all the constraints, so the third, fourth and fifth KKT
conditions are satisfied at this point.

We now examine the signs of the KKT multipliers in the working set,
that is, the indices $j\in W^{\circ}$. If these conditions are all
nonnegative, the first KKT condition is also satisfied, so we conclude
that $\vec{x}^{\circ}$ is a KKT point for the original problem. In
fact, since $Q$ is positive semidefinite, we have that $\vec{x}^{\circ}$
is a global minimum.

If, on the other hand, on or more of the multipliers $\mu_{j}^{\circ}$,
$j\in W^{\circ}$, are negative, the first KKT condition is not satisfied
and the objective function $f$ may be decreased by dropping one of
these constraints. Thus, we remove an index $j$ corresponding to
one of the negative multipliers from the working set and solve a new
subproblem for the next step. While any index $j$ for which $\mu_{j}^{\circ}<0$
usually will yield in a direction $\vec{d}$ along which the algorithm
can make progress, the most negative multiplier is often chosen in
practice. This choice is motived by a sensitivity analysis, which
shows that the rate of decrease in the objective function when one
constraint is removed, is proportional to the magnitude of the Lagrange
multiplier for that constraint.

## Example

Apply the active-set method to the following problem:
\begin{align*}
\min_{\vec{x}}\, & f\left(\vec{x}\right)=\left(x_{1}-1\right)^{2}+\left(x_{2}-2.5\right)^{2}\\
\textrm{subject to}\, & \begin{cases}
-x_{1}+2x_{2}-2\leq0\,,\\
x_{1}+2x_{2}-6\leq0\,.\\
x_{1}-2x_{2}-2\leq0\,,\\
-x_{1}\leq0\,,\\
-x_{2}\leq0\,.
\end{cases}
\end{align*}

```julia
using Plots
using LaTeXStrings

Q = Float64[2 0
     0 2]
c = Float64[2, 5]
A = Float64[-1  2
      1  2
      1 -2
     -1  0
      0 -1]
b = Float64[2, 6, 2, 0, 0]

x = -1:0.05:5
y = -1:0.05:5
z = Surface((x,y)->(0.5 .* [x y]*Q*[x;y] .- transpose(c)*[x;y] .+ 0.5 .* transpose(c)*Q*c)[1], x, y)
contour(x, y, z, levels=35)
plot!(x, (2 .+ x) ./ 2, linestyle=:dash, label=L"-x_1+2x_2-2\le 0")
plot!(x, (6 .- x) ./ 2, linestyle=:dash, label=L"x_1+2x_2-6\le 0")
plot!(x, (2 .- x) ./ -2, linestyle=:dash, label=L"x_1-2x_2-2\le 0")
plot!([0,2,4,2,0,0],[0,0,1,2,1,0], linewidth=2, label="domain")
```

We refer the constraints, in order, by indices $1$ through $5$.
For this problem it is easy to determine a feasible initial point;
say $\vec{x}^{\left(0\right)}=\begin{pmatrix}2 & 0\end{pmatrix}\mathsf{T}$.
Constraints $3$ and $5$ are active at this point, and we set $W_{0}=\left\{ 3,5\right\} $.
Note that we could just as validly have chosen $W_{0}=\left\{ 5\right\} $
or $W_{0}=\left\{ 3\right\} $ or even $W_{0}=\emptyset$; each choice
would lead the algorithm to perform somewhat differently.

```julia
x = [2, 0]
g = Q*x - c
ain = [reshape(A[3,:], 1, 2); reshape(A[5,:], 1, 2)]
sol = [Q transpose(ain)
       ain zeros(2,2)] \ [-g; 0; 0]
```

    4-element Array{Float64,1}:
      6.661338147750939e-16
      4.440892098500626e-16
     -2.0000000000000013
     -0.9999999999999967

Since $\vec{x}^{\left(0\right)}$ lies on a vertex of the feasible
region, it is obviously a minimizer of the objective function $f$
with respect to the working set $W_{0}$; that is, the solution of
the subproblem with $k=0$ is $\vec{d}^{\left(0\right)}=\vec{0}$.
We can then find the multipliers $\mu_{3}^{\circ}$ and $\mu_{5}^{\circ}$
associated with the active constraints. Substitution of the data from
our problem yields
\begin{equation}
\begin{pmatrix}-1\\
2
\end{pmatrix}\lambda_{3}^{\circ}+\begin{pmatrix}0\\
1
\end{pmatrix}\lambda_{5}^{\circ}=\begin{pmatrix}2\\
-5
\end{pmatrix}\,,
\end{equation}
which has solution $\lambda_{3}^{\circ}=-2$ and $\lambda_{5}^{\circ}=-1$.

We now remove constraint $3$ from the working set, because it has
the most negative multiplier, and set $W_{1}=\{5\}$.

```julia
x = [2, 0]
g = Q*x - c
ain = reshape(A[5,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

    3-element Array{Float64,1}:
     -1.0
      0.0
     -5.0

We begin iteration $1$ by finding the solution of the subproblem for $k=1$, which is
$\vec{d}^{\left(1\right)}= \begin{pmatrix}-1 & 0\end{pmatrix}^\mathsf{T}$.

```julia
d = sol[1:2]
α = min(1.0, [(reshape(A[j,:], 1, 2) * d)[1,1] ≤ 0 ? 1.0 : ((b[j] .- reshape(A[j,:], 1, 2) * x) / (reshape(A[j,:], 1, 2) * d))[1,1] for j in (1,2,3,4)]...)
```

    1.0

The step-length formula yields $\alpha_{1}=1$, and the new iterate
is $\vec{x}^{\left(2\right)}=\begin{pmatrix}1 & 0\end{pmatrix}^\mathsf{T}$. There are no blocking constraints, so that $W_{2}=W_{1}=\left\{ 5\right\} $

```julia
x = x + α .* d
g = Q*x - c
ain = reshape(A[5,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

    3-element Array{Float64,1}:
      0.0
      0.0
     -5.0

We find at the start of iteration $2$ that the solution of the
subproblem is $\vec{d}^{\left(2\right)}=\vec{0}$. We deduce that
the Lagrange multiplier for the lone working constraint is $\lambda_{5}^{\circ}=-5$,
so we drop the working set to obtain $W_{3}=\emptyset$.

```julia
g = Q*x - c
sol = Q \ (-g)
```

    2-element Array{Float64,1}:
     -0.0
      2.5

Iteration $3$ starts by solving the unconstrained problem, to obtain
the solution $\vec{d}^{\left(3\right)}=\begin{pmatrix}0 & 2.5\end{pmatrix}^\mathsf{T}$.

```julia
d = sol[1:2]
α = min(1.0, [(reshape(A[j,:], 1, 2) * d)[1,1] ≤ 0 ? 1.0 : ((b[j] .- reshape(A[j,:], 1, 2) * x) / (reshape(A[j,:], 1, 2) * d))[1,1] for j in (1,2,3,4,5)]...)
```

    0.6000000000000001

The step-length formula yields a step length of $\alpha_{3}=0.6$
and a new iterate $\vec{x}^{\left(4\right)}=\begin{pmatrix}1 & 1.5\end{pmatrix}^\mathsf{T}$. There
is a single blocking constraint (constraint $1$), so we obtain $W_{4}=\left\{ 1\right\} $.

```julia
x = x + α .* d
g = Q*x - c
ain = reshape(A[1,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

    3-element Array{Float64,1}:
     0.3999999999999999
     0.19999999999999996
     0.7999999999999998

The solution of the subproblem for $k=4$ is then $\vec{d}^{\left(4\right)}=\begin{pmatrix}0.4 & 0.2\end{pmatrix}^\mathsf{T}$.

```julia
d = sol[1:2]
α = min(1.0, [(reshape(A[j,:], 1, 2) * d)[1,1] ≤ 0 ? 1.0 : ((b[j] .- reshape(A[j,:], 1, 2) * x) / (reshape(A[j,:], 1, 2) * d))[1,1] for j in (2,3,4,5)]...)
```

    1.0

The new step-length is $1$. There are no blocking constraints
on this step, so the next working set in unchanged: $W_{5}=\left\{ 1\right\} $. The new iterate is $\vec{x}^{\left(5\right)}=\begin{pmatrix}1.4 & 1.7\end{pmatrix}^\mathsf{T}$.

```julia
x = x + α .* d
g = Q*x - c
ain = reshape(A[1,:], 1, 2)
sol = [Q transpose(ain)
       ain zeros(1,1)] \ [-g; 0]
```

    3-element Array{Float64,1}:
     0.0
     0.0
     0.7999999999999998

Finally, we solve the subproblem for $k=5$ to obtain a solution $\vec{d}^{\left(5\right)}=\vec{0}$.
We find a multiplier $\mu_{1}^{\circ}=0.8$, so we have found the
solution. Wet set $\vec{x}^{\star}=\begin{pmatrix}1.4 & 1.7\end{pmatrix}^\mathsf{T}$
and terminate.

```julia
x = -1:0.05:5
y = -1:0.05:5
z = Surface((x,y)->(0.5 .* [x y]*Q*[x;y] .- transpose(c)*[x;y] .+ 0.5 .* transpose(c)*Q*c)[1], x, y)
contour(x, y, z, levels=35)
plot!(x, (2 .+ x) ./ 2, linestyle=:dash, label=L"-x_1+2x_2-2\le 0")
plot!(x, (6 .- x) ./ 2, linestyle=:dash, label=L"x_1+2x_2-6\le 0")
plot!(x, (2 .- x) ./ -2, linestyle=:dash, label=L"x_1-2x_2-2\le 0")
plot!([0,2,4,2,0,0],[0,0,1,2,1,0], linewidth=2, label="domain")
plot!([2,2,1,1,1,1.4], [0,0,0,0,1.5,1.7], linewidth=2, label="iterates", markershape=:circle)
```

## Julia

The package `GeneralQP` solves the quadratic programming problems using the function `solve(Q, -c, A, b, x₀)`.

```julia
#using Pkg
#pkg"add https://github.com/oxfordcontrol/GeneralQP.jl"
using GeneralQP

sol = GeneralQP.solve(Q, -c, A, b, Float64[2,0])
```

## Sequential Quadratic Programming

We consider the general constrained problem
\begin{align*}
\min\, & f\left(\vec{x}\right)\\
\textrm{subject to} & \begin{cases}
\vec{h}\left(\vec{x}\right)=\vec{0}\\
\vec{g}\left(\vec{x}\right)\leq\vec{0}
\end{cases}
\end{align*}
where $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$, $\vec{h}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$,
$m\leq n$, and $\vec{g}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{p}$. The idea
behind the _sequential quadratic programming_ (SQP) approach
is to model the general problem at the current iterate $\vec{x}^{\left(k\right)}$
by a quadratic programming subproblem, then use the minimizer of this
subproblem to define a new iterate $\vec{x}^{\left(k+1\right)}$.
The challenge is to design the quadratic subproblem so that it yields
a good step for the general optimization problem.

We know that the extended Lagrangian function for this problem is

\begin{equation}
\mathcal{L}\left(\vec{x},\vec{\lambda},\vec{\mu}\right)=f\left(\vec{x}\right)+\vec{\lambda}^\mathsf{T}\vec{h}\left(\vec{x}\right)+\vec{\mu}^\mathsf{T}\vec g\left(\vec{x}\right)\,.
\end{equation}

Applying Newton's method to the Lagrangian function and linearizing
both the equality and the inequality constraints yields the following
subproblem
\begin{align*}
\min_{\vec{d}^{\left(k\right)}}\, & \frac{1}{2}\left(\vec{d}^{\left(k\right)}\right)^\mathsf{T} \mathsf{H} \mathcal{L}\left(\vec{x}^{\left(k\right)},\vec{\lambda}^{\left(k\right)},\vec{\mu}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}+ \mathsf{D} \mathcal{L}\left(\vec{x}^{\left(k\right)},\vec{\lambda}^{\left(k\right)},\vec{\mu}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}\\
\textrm{subject to}\, & \begin{cases}
 \mathsf{J}\vec{h}\left(\vec{x}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}=-\vec{h}\left(\vec{x}^{\left(k\right)}\right)\\
 \mathsf{D} g_{j}\left(\vec{x}^{\left(k\right)}\right)\vec{d}^{\left(k\right)}=-g_{j}\left(\vec{x}^{\left(k\right)}\right)\,, & j\in W_{k}\,,
\end{cases}
\end{align*}

where $\mu_{j}^{\left(k\right)}=0$, for all
$j\notin W_{k}$. We can use the active-set method for quadratic programming
to solve this subproblem. The new iterate is given by $\vec{x}^{\left(k+1\right)}$,
$\vec{\lambda}^{\left(k+1\right)}$, $\vec{\mu}^{\left(k+1\right)}$
and $W_{k+1}$.

If the SQP method is able to identify the optimal active set then
it will act like a Newton method for equality-constrained optimization
and will converge rapidly.

It is also remarkable that, far from the solution, the SQP approach
is usually able to improve the estimate of the active set and guide
the iterates towards a solution.

Non-quadratic objective functions, however, can impede progress of
the SQP algorithm, a phenomenon known as the Maratos effect. Steps
that make good progress toward a solution are rejected and the algorithm
fails to converge rapidly. These difficulties can be overcome by means
of a _second-order correction_.

```julia
#using Pkg
#pkg"add NLopt"

using NLopt

function myfunc(x::Vector, grad::Vector)
    if length(grad) > 0
        grad[1] = 0
        grad[2] = 0.5/sqrt(x[2])
    end
    return sqrt(x[2])
end

function myconstraint(x::Vector, grad::Vector, a, b)
    if length(grad) > 0
        grad[1] = 3a * (a*x[1] + b)^2
        grad[2] = -1
    end
    (a*x[1] + b)^3 - x[2]
end

opt = Opt(:LD_SLSQP, 2)
opt.lower_bounds = [-Inf, 0.]
opt.xtol_rel = 1e-4

opt.min_objective = myfunc
inequality_constraint!(opt, (x,g) -> myconstraint(x,g,2,0), 1e-8)
inequality_constraint!(opt, (x,g) -> myconstraint(x,g,-1,1), 1e-8)

(minf,minx,ret) = optimize(opt, [1.234, 5.678])
numevals = opt.numevals # the number of function evaluations
println("got $minf at $minx after $numevals iterations (returned $ret)")
```

```julia

```


# Logistic Regression from Scratch: An Immersion to Machine Learning Concepts

**Dimitrios Tanoglidis**

This is the first of a series of notebooks on basic Machine Learning algorithms, written (when relatively easy) from scratch. That means that I will try to avoid using the [scikit-learn](https://scikit-learn.org/stable/) library, at least not before writing my own functions first.

The reason for writing these notebooks is the same as the reason one teaches a class: to learn the material themselves. That being said, I hope that other people will find them useful, too.

The first algorithm I am going to present is the logistic regression classifier (yes, although it has the name "regression", it is a classifier). The reason for selecting LR as the first classifier to present is that:

- a) It is intuitive

- b) Often works "just fine" for practical applications (useful skill for industry-oriented people)

-  c) Can be easily coded from scratch

-  d) It is (usually) the last layer of a deep neural network

- e) Allow us to introduce concepts such as regularization, and selection of hyperparameters

- f) Allow us to nicely introduce and hard-code the loss function and its optimization methods

- g) and many more!!!

Before discussing the model, let's import all the necessary packages we are going to need.

```python
# Basic packages and plotting
import numpy as np
import scipy.io as sio
import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import rcParams
import pandas as pd
#import seaborn as sns
rcParams['font.family'] = 'serif'

# Adjust rc parameters to make plots pretty
def plot_pretty(dpi=200, fontsize=9):

    import matplotlib.pyplot as plt

    plt.rc("savefig", dpi=dpi)       # dpi resolution of saved image files
    plt.rc('text', usetex=True)      # use LaTeX to process labels
    plt.rc('font', size=fontsize)    # fontsize
    plt.rc('xtick', direction='in')  # make axes ticks point inward
    plt.rc('ytick', direction='in')
    plt.rc('xtick.major', pad=10)
    plt.rc('xtick.minor', pad=5)
    plt.rc('ytick.major', pad=10)
    plt.rc('ytick.minor', pad=5)
    plt.rc('lines', dotted_pattern = [0.5, 1.1]) # fix dotted lines

    return

plot_pretty()

# =============================================
# Scikit-learn etc
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import confusion_matrix
```

If you run this notebook on Colab, uncomment and run the following cell

```python
#! sudo apt-get install texlive-latex-recommended
#! sudo apt install texlive-latex-extra
#! sudo apt install dvipng
#!apt install cm-super
```

## Logistic Regression for binary classification

LR is a **supervised** machine learning classifier. We will consider a binary classification problem. Assume that we have $m$ training pairs $(\mathbf{x}^{(i)},y^{(i)})$, where $\mathbf{x}^{(i)} = [x_1^{(i)},x_1^{(i)},\dots,x_n^{(i)}]$ a feature vector and the labels $y^{(i)} \in \{0,1\}$, the probability to get $y=1$ is modeled as:

\begin{equation}\boxed{
p(y=1|\mathbf{x};\mathbf{w}) = \frac{1}{1+e^{-\mathbf{w}^T\mathbf{x}}},}
\end{equation}

where $\mathbf{w}^T\mathbf{x} = \sum_{i=0}^n w_i x_i = w_0 + w_1 x_1 + \dots + w_n x_n $.

The parameters $w$ are **weights to be learned**. Specifically the paramter $w_0$ is called **bias**.

The property that makes the logistic function (also known as **sigmoid**) $\sigma(z) = 1/(1+e^{-z})$ useful is that it can take number (here $\mathbf{w}^T\mathbf{x}$) and map it to the interval $(0,1)$, thus making it easy to be interpreted as probability.

Let's plot the logistic function in a range of $z$ values:

```python
def logist(z):
    g = 1. + np.exp(-z)
    return 1./g

z = np.linspace(-10,10,100)
logist_z = logist(z)

plt.figure(figsize = (8.0,8.0))
plt.plot(z,logist_z,c='mediumblue',linewidth=2.5)
plt.grid(ls='--',alpha=0.7)
plt.title('Logistic or Sigmoid function', fontsize=15)
plt.xlabel('$z$',fontsize=20)
plt.ylabel('$\sigma(z)$', fontsize=20)

plt.show()

```

### Learning the weights: An introduction to the loss function and the gradient descent

#### The loss function

If we find optimal weights, $w$, then the predicted label of a single observation, $\hat{y}$, will be:

\begin{equation}
\hat{y} = \begin{cases}
1, & \mbox{if}\,\, \sigma(\mathbf{w^Tx}) \geq 0.5 \\
0, & \mbox{otherwise}
\end{cases}
\end{equation}

How are we going to find the weights?

Since we are dealing with a binary problem, labels $Y \in \{0,1\}$ the probability to get label $y$ when we have an observation $\mathbf{x}$ is given by the **Bernoulli distribution**:

\begin{equation}
p(y|\mathbf{x};\mathbf{w}) \\= p(y=1|\mathbf{x};\mathbf{w})^y (1- p(y=1|\mathbf{x};\mathbf{w}))^{(1-y)}  \\=\sigma(\mathbf{w^Tx})^y (1 - \sigma(\mathbf{w^Tx}))^{(1-y)}
\end{equation}

Assuming that the $m$ training samples are independent, we can write down a **likelihood** of parameters given observations, whose **maximization** will yield the best set  of parameters:

\begin{equation}
L(\mathbf{w}|\mathbf{x},\mathbf{y}) = \prod_{i=1}^m p(y^{(i)}|\mathbf{x}^{(i)};\mathbf{w}) \\ = \prod_{i=1}^m \sigma(\mathbf{w^Tx}^{(i)})^{y^{(i)}} (1 - \sigma(\mathbf{w^Tx}^{(i)}))^{(1-y^{(i)})}
\end{equation}

Now to form the **loss function** (something to be **minimized**) we will get the *negative* of the the log of the above likelihood (the logarithm is for computational ease):

\begin{equation}
\boxed{
J(\mathbf{w}) = - \frac{1}{m}\sum_{i=1}^m y^{(i)}\log \sigma(\mathbf{w^Tx}^{(i)}) + (1-y^{(i)})\log (1 - \sigma(\mathbf{w^Tx}^{(i)}))}
\end{equation}

This is also known as the **binary cross-entropy** loss function (note that we have averaged over all samples).

(For the name cross-entropy check the relevant [wikipedia article](https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_loss_function_and_logistic_regression))

Let's write a function that implements gradient descent for logistic regression and apply it to the problem of star-galaxy separation from SDSS data

#### Gradient descent

It is not possible to get an expression for the parameters that minimize $J(\mathbf{w})$ in a closed form.

For that readon we will use the **gradient descent** algorithm, which is an iterative process to update the weights:

\begin{equation}
\boxed{
\mathbf{w} \leftarrow \mathbf{w} - \eta \nabla_\mathbf{w} J(\mathbf{w})}
\end{equation}

(till convergence or for a fixed number of steps). The (hyper)parameter $\eta$ is knows as the **learning rate**, and dictates how much the weights are being updated in each iteration of the algorithm.

The intuition behind this is that we are moving towards the region that the function $J$ changes more rapidly each time.

The above equation can be also written in terms of the components of the weights vector as:

\begin{equation}
w_j \leftarrow w_j - \eta \frac{\partial J}{\partial w_j}, \quad j=0, \dots, n
\end{equation}

$n$ being the number of features (remember $w_0$ is the bias factor).

For the case of the logistic loss function, one can show that (see appendix):

\begin{equation}
\frac{\partial J}{\partial w_j} = \frac{1}{m}\sum_{i=1}^m \left[ \sigma(\mathbf{w^Tx}^{(i)}) - y^{(i)}\right]x_j^{(i)}
\end{equation}

with $x_j^{(i)}$ the $j$-th component of the feature vector of the $i$-th example, and $x_0^{(i)} = 1$.

Summarizing, we can **learn** the optimal weights, given a training set $(x^{(i)},y^{(i)})$ of size $m$, in an iterative way, as:

\begin{equation}
\boxed{
w_j \leftarrow w_j - \eta \frac{1}{m}\sum_{i=1}^m \left[ \sigma(\mathbf{w^Tx}^{(i)}) - y^{(i)}\right]x_j^{(i)}}
\end{equation}

A catalog of SDSS instances

```python
# Import the data
SDSS_table = pd.read_csv("SDSS_file.csv")
SDSS_table.head()
```

      objid
      ra
      dec
      u
      g
      r
      i
      z
      specobjid
      class
      redshift

      0
      1237646798137787517
      121.802126
      0.855476
      25.02634
      21.90814
      20.17926
      19.22341
      18.68446
      5342652717419438080
      GALAXY
      0.504717

      1
      1237646798137787833
      121.748271
      0.891010
      23.74677
      22.75365
      21.10284
      20.06169
      19.50690
      5342652167663624192
      GALAXY
      0.579273

      2
      1237646798137788079
      121.708487
      0.874751
      23.15607
      22.95075
      20.89328
      19.71096
      19.10471
      5342651617907810304
      GALAXY
      0.772647

      3
      1237646798137852371
      121.820752
      0.931990
      19.37035
      17.34262
      16.35286
      15.92400
      15.58903
      5342663162779901952
      GALAXY
      0.101993

      4
      1237646798137852646
      121.910430
      0.896389
      23.42736
      22.81119
      20.27570
      19.14886
      18.67588
      5342662613024088064
      GALAXY
      0.585733

```python
# Get the redshift
redshift = SDSS_table["redshift"]
# Get the magnitudes
u = SDSS_table["u"];g = SDSS_table["g"];r = SDSS_table["r"];i = SDSS_table["i"];z =SDSS_table["z"]
class_obs = SDSS_table["class"]

class_label = np.zeros(len(class_obs))
```

Now change the class of objects to something that a machine can understand. Say "GALAXY" = 0, "STAR" = 1, "QSO"=2

```python
for i_l in range(len(class_label)):

    if (class_obs[i_l]=="GALAXY"):
        class_label[i_l] = 0
    elif (class_obs[i_l]=="STAR"):
        class_label[i_l] = 1
    else:
        class_label[i_l] = 2
```

```python
cuts = (u>-10.)&(g>-10.)&(r>-10.0)&(i>-10.0)&(z>-10.0)

u = u[cuts]
g = g[cuts]
r = r[cuts]
i = i[cuts]
z = z[cuts]
redshift = redshift[cuts]
class_label = class_label[cuts]
print(len(redshift))
```

    299908

Now let's create some extra features, beyond the raw magnitudes we imported above. Namely the colors and the magnitude ratios.

The colors are $u-g$, $u-r$, $u-i$, $u-z$, $g-r$, $g-i$, $g-z$, $r-i$, $r-z$

And the ratios, similarly: $u/g$, $u/r$, $u/i$, $u/z$, $g/r$, $g/i$, $g/z$, $r/i$, $r/z$.

```python
# Define colors first
u_g = u-g; u_r = u-r; u_i = u-i; u_z = u-z; g_r = g-r; g_i = g-i; g_z = g-z; r_i = r-i; r_z = r-z
# Define the ratios now
u_ov_g = u/g; u_ov_r = u/r; u_ov_i = u/i; u_ov_z = u/z; g_ov_r = g/r; g_ov_i = g/i; g_ov_z=g/z; r_ov_i = r/i
r_ov_z = r/z
```

Now create a feature matrix.

```python
n_inst = len(redshift) #number of examples

# Initialize feature matrix
X_feat = np.zeros([n_inst,23])
# ================================================================
# ================================================================
# Define feature matrix
X_feat[:,0] = u;X_feat[:,1] = g; X_feat[:,2] = r; X_feat[:,3] = i; X_feat[:,4] =z
X_feat[:,5] = u_g;X_feat[:,6] = u_r;X_feat[:,7] = u_i;X_feat[:,8] = u_z; X_feat[:,9] = g_r
X_feat[:,10] = g_i; X_feat[:,11] = g_z; X_feat[:,12] = r_i; X_feat[:,13] = r_z
X_feat[:,14] = u_ov_g; X_feat[:,15] = u_ov_r; X_feat[:,16] = u_ov_i; X_feat[:,17] = u_ov_z
X_feat[:,18] = g_ov_r;X_feat[:,19] = g_ov_i;X_feat[:,20] = g_ov_z
X_feat[:,21] = r_ov_i; X_feat[:,22] = r_ov_z
```

Now, keep only stars and galaxies

```python
X_ft_new = X_feat[(class_label==0.0)|(class_label==1.0),:]
y_label_new = class_label[(class_label==0.0)|(class_label==1.0)]

X_ft = X_ft_new[:3000,:]
y_true = y_label_new[:3000]

print(len(y_true[y_true==1]))

# Also reserve a test matrix
X_test = X_ft_new[4000:5000,:]
y_test = y_label_new[4000:5000]
```

    1148

Define function that fits the logistic regression

```python
def Logistic_fit(X, y, eta=0.001, n_iter=10000):
    """
    Fits data to a LR model and returns weights

    X -> feature matrix [n_samples,n_features]
    y -> true labels [n_samples]
    eta -> learning rate
    n_iter -> number of iterations
    """

    n_feats = np.shape(X)[1] # number of features
    m_samples = np.shape(X)[0] # number of samples (instances)

    # Initialize weights
    w = np.random.normal(0,0.5,n_feats+1)

    for i in range(n_iter):

        z_inp = np.dot(X,w[1:]) + w[0]
        # Predictions - output of the sigmoid function
        sigma_lr = 1./(1.+np.exp(-np.clip(z_inp,-200,200)))
        # Errors
        err = (sigma_lr - y)

        # Update weights
        w[1:] -= (eta/m_samples)*(np.dot(X.T,err))
        w[0] -= (eta/m_samples)*(err.sum())

    return w
```

```python
weights = Logistic_fit(X_ft,y_true)

def predict_LR(w,X):
    n_samples = np.shape(X)[0]

    z_inp = np.dot(X,w[1:]) + w[0]
    sig_out = 1./(1.+np.exp(-np.clip(z_inp,-200,200)))

    y_pr = np.zeros(n_samples)
    y_pr[sig_out>0.5] = 1

    return(y_pr)

y_pred = predict_LR(weights,X_test)
```

```python
print(accuracy_score(y_test,y_pred))
print(precision_score(y_test,y_pred))
print(recall_score(y_test,y_pred))
```

    0.899
    0.8181818181818182
    0.8773584905660378

```python
import seaborn as sns

con_mat = confusion_matrix(y_test, y_pred).T
con_mat = con_mat.astype(float)
class_names = ['Galaxy', 'Star']

plt.figure(figsize = (7.0,7.0))
plt.title('Confusion Matrix',fontsize=10)
g =sns.heatmap(con_mat.T, square = True, annot=True,fmt='.0f',annot_kws={"size": 25}, cmap='Blues', cbar=False,
           xticklabels= class_names,
           yticklabels= class_names)

g.set_xticklabels(class_names, fontsize = 19)
g.set_yticklabels(class_names, fontsize = 19)

plt.xlabel('True label',fontsize=19)
plt.ylabel('Predicted label',fontsize=19)

plt.show()
```

## References

```python

```