# `DSML Workshop 09` - Advanced Non-Linear Regression

In this workshop we continue with hands-on supervised learning (regression). We continue

We will cover the following:
1. Regularization: L1 (LASSO) and L2 (ridge) regression
1. General non-linear features: Radial Basis Functions
1. Other regression algorithms: Overview of some selected algorithms

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import datetime

%matplotlib inline
```

## Example: predicting peak electrical power

We continue with our electric power example from last week which we retieved from PJM from the following link [here](https://dataminer2.pjm.com/feed/hrl_load_metered/definition)). The files we are loading are the raw files we downloaded from this source. The final input data for our code is `Pittsburgh_load_data.csv`.

```python
df = pd.read_csv("Pittsburgh_load_data.csv")
df["Date"] = pd.to_datetime(df["Date"], format="%d.%m.%Y")
df["Month"] = df["Date"].dt.month
df.head()
```

      Date
      AVG
      MAX
      MIN
      Total
      High_temp
      Avg_temp
      Month

      0
      2013-01-01
      1.598524
      1.859947
      0.001599
      38.368031
      0.0
      -1.68
      1

      1
      2013-01-02
      1.809347
      2.054215
      0.001809
      43.428194
      -3.9
      -6.58
      1

      2
      2013-01-03
      1.832822
      2.049550
      0.001833
      43.991607
      0.6
      -6.12
      1

      3
      2013-01-04
      1.812699
      2.008168
      0.001813
      43.508609
      0.0
      -1.95
      1

      4
      2013-01-05
      1.662036
      1.838251
      0.001662
      39.892360
      1.7
      -1.47
      1

## `Recap`

**Recall from previous workshop**: We fitted a polynomial model to the full range of electricity data. To do so we used `scikit learn` and obtained the following result:

```python
xp = df["High_temp"].values
yp = df["MAX"].values
```

```python
plt.figure(figsize = (8,6))
plt.scatter(xp, yp, marker="x")
plt.xlabel("High Temperature (°C)")
plt.ylabel("Peak Demand (GW)")
plt.show()
```

```python
# x is the input variable
# y is the output vaiable
# d is degree pf polynomial regression

# Exercise: write a function to:
#   - scale the vector X to [-1, 1]
#   - use sklearn.preprocessing.PolynomialFeatures to generate the all polynoms from 1 to d
#   - use the sklearn.linear_model.LinearRegression to fit a regression
#   - plot the resulting Line and the input data in a figure

def plot_regression_poly(x, y, d):

    min_x, max_x = x.min(), x.max()

    # Rescale x

    # Create polynomial features

    # Fit the linear regression model

    # Artificial test data
    xt0 = np.arange(-15, 35, 0.01)
    # Transform the test data to the polynomial degree
    Xt = []

    # Predict the output of the artificial test data in yt
    yt = []

    # Get the coefficients and print the first 4 of them
    print()

    # Plot results
    plt.figure(figsize = (8,6))
    plt.scatter(x, y, marker="x")
    ylim = plt.ylim()
    plt.plot(xt0, yt, 'C1')
    plt.xlabel("Temperature (°C)")
    plt.ylabel("Demand (GW)")
    plt.xlim([min_x-2, max_x+2])
    plt.ylim(ylim)
```

```python
plot_regression_poly(xp, yp, d=100)
```

## `Regularization` (by hand)

We have so far seen that the degree of the polynomial we use for our nonlinear features acts as a nice characterization of the model complexity.  But there is another notion of model complexity that is also important to understand, the _magnitude_ of the model parameters.  To see why this is important, let's look again at our degree 100 polynomial fit to the data.

Let's also look at the actual weights involved with this fit.

```python
plot_regression_poly(xp, yp, d=100)
```

The way that we get the polynomial to exactly pass through the points requires extremely high values for the coefficients: we need to set the coefficients just so that the different polynomial terms largely “cancel” out precisely at the data points and fit the function exactly to the data.

This also suggests another method for controlling the complexity of the model class:
restircting the magnitude of the coefficients. This is the basis of the technique known as regularization.

Formally, regularization is often written as an additional term in the canonical machine learning problem.  Instead of simply minimizing the average loss, we minimize the average loss plus a term that penalizes the magnitude of the coefficients (usually some function of a norm of the weights, often just the sum of squared weights also called $\ell_2$ regularization, but other functions are possible as well).  For example, let's consider the following optimizaiton problem:
\begin{equation}
\min \theta \; \frac{1}{m}\sum_{i=1}^m \ell \left(h_\theta(x^{(i)}),y^{(i)} \right) + \lambda \sum_{i=1}^n \theta_i^2
\end{equation}
where $\lambda \in \mathbb{R}_+$ is what is called a _regularization parameter_.  $\lambda$ effectively trades off between minimizing the training loss (which naturally "wants" to use large weights), and keeping the weights small.  If $\lambda = 0$, we ignore the regularization term entirely, and just minimize training loss; but as $\lambda \rightarrow \infty$, the _only_ relevant term in the optimization problem becomes the sum of the squared weights, which is clearly minimized if we just pick $\theta = 0$.  Thus, by varying $\lambda$ between zero and some very large constant, we can "sweep out" different ranges of model complexity.

### Visualizing regularization
Let's see what this looks like on our 100 degree polynomial.  The figure above shows the situation with no regularization, i.e, $\lambda = 0$.  If we instead choose $\lambda = 1$, we get the following figure.

```python
# x is the input variable
# y is the output vaiable
# d is degree pf polynomial regression
# lam is lambda, the degree of regularization

def plot_regularized_polyregression (x, y, lam, d):
    # Hard task:
    # Implement the linear regression by hand (without using scikit learn)
    # also include the regularization parameter lambda
    pass
```

```python
plot_regularized_polyregression (xp, yp, lam=0.1, d=100)
```

What happens if we regularize further?

```python
plot_regularized_polyregression (xp, yp, lam=100, d=100)
```

We can also understand what is happening here by reference to the previous section when we discussed polynomial fittings as a function of the degree of the polynomial.  Remember that after degree 10 or so, there wasn't a huge benefit to including additional polynomial terms in the regression.  Yet, if include these terms within the context of a tradition least squares fit, we have seen that in order to use the polynomial degree to minimize the error (essentially done by "exactly" passing through some of the points), this is accomplished by a very high weight on the high-order coefficients.

So what happens when we apply a regularization penalty?  Because we know that we can achieve low error with a lower-degree polynomial of relatively small weights, if we add sufficient regularization to the optimization objective, this will have the effect of avoiding putting much weight on the high-order terms in the polynomial, and just putting the weight on the lower order terms.  We can view this by looking at the magnitude of the actual coefficients of $\theta$ before and after regularization (remember, $\theta$ is ordered in higher-to-lower degree polynomial terms, so we will flip the order to correspond to increasing degree of the polynomial terms.

```python
def ls_poly_reg(x, y, lam, degree):
    xs = 2*(x - min(x))/(max(x) - min(x)) - 1   # rescale to range [-1,1]
    X = np.array([xs**i for i in range(degree,-1,-1)]).T
    return np.linalg.solve(X.T @ X + lam*np.eye(X.shape[1]), X.T @ y)
```

```python
# Define inputs
x=xp
y=yp
lam=1
degree= 50

# Plot function
theta = ls_poly_reg(x, y, lam, degree)

plt.figure(figsize = (8,6))
plt.semilogy(range(degree+1), np.abs(theta[::-1]))
plt.xlabel("Degree of coefficient")
plt.ylabel("Coefficient weight")
plt.show()
```

**What do you observe as you change the degree of polynomial and the regularization parameter?**

## Regularization using `scikit learn`

### Ridge regression ($L_2$ Regularization)

Perhaps the most common form of regularization is known as ridge regression or $L_2$ regularization. This is the process we have implemented manually above. It proceeds by penalizing the sum of squares (L2-norm) of the model coefficients; in this case, the penalty on the model fit would be $P = \alpha\sum_{n=1}^N \theta_n^2$ where $\alpha$ is a free parameter that controls the strength of the penalty (note that this is equivalent to our $\lambda$ from above). This type of penalized model is built into Scikit-Learn with the Ridge estimator:

First we need to create polynomial features using the `PolynomialFeatures` module in `scikit learn`

```python
from sklearn.preprocessing import PolynomialFeatures

# initialize model
Poly = PolynomialFeatures(degree = 50)

# fit and transform xp
X_poly = Poly.fit_transform(xp.reshape(-1,1))
```

```python
#len(X_poly)
```

We then import the `Ridge` regression model

```python
from sklearn.linear_model import Ridge

model_L2 = Ridge(alpha = 0.01, normalize = True, solver = 'lsqr') # select least squares regression as solver

model_L2.fit(X_poly, yp)
prediction = model_L2.predict(X_poly)

print("Coefficients ", model_L2.coef_, "\nIntercept ", model_L2.intercept_ )
```

    Coefficients  [ 0.00000000e+00 -2.32640141e-02  5.84347924e-05  6.13898052e-06
      3.33662650e-07  1.07952528e-08  3.27215854e-10  8.81663013e-12
      2.23665684e-13  5.17885431e-15  1.07560117e-16  1.79689238e-18
      1.39128809e-20 -6.59554607e-22 -4.74124289e-23 -2.09838984e-24
     -7.84618536e-26 -2.67816111e-27 -8.60338073e-29 -2.63977107e-30
     -7.79437033e-32 -2.22231243e-33 -6.12133462e-35 -1.62513656e-36
     -4.13374005e-38 -9.95388249e-40 -2.21399631e-41 -4.29129615e-43
     -5.93554881e-45  2.09363482e-47  6.17553472e-48  3.35292651e-49
      1.39778610e-50  5.17393937e-52  1.78342400e-53  5.84705960e-55
      1.84300545e-56  5.61572341e-58  1.65785897e-59  4.74022713e-61
      1.30860068e-62  3.46361529e-64  8.66817836e-66  1.99233509e-67
      3.90866418e-69  4.88106818e-71 -7.53423952e-73 -9.32437350e-74
     -5.01593599e-75 -2.19332730e-76 -8.69503304e-78]
    Intercept  1.8846505592013083

```python
# function for plotting

def plot_scikit_output (x, y, fitted_model):

    min_x, max_x = x.min(), x.max()
    xt0 = np.linspace(min_x-1, max_x+1, 400)

    xt0_poly = Poly.fit_transform(xt0.reshape(-1,1))

    # plotting routine
    plt.figure(figsize = (8,6))
    plt.scatter(x, y, marker="x")
    ylim = plt.ylim()
    plt.plot(xt0, fitted_model.predict(xt0_poly), 'C1')
    plt.xlabel("Temperature (°C)")
    plt.ylabel("Demand (GW)")
    plt.xlim([min_x-2, max_x+2])
    plt.ylim(ylim)
```

```python
plot_scikit_output (xp, yp, model_L2)
```

### LASSO regression ($L_1$ regularization)

Another very common type of regularization is known as lasso, and involves penalizing the sum of absolute values (1-norms) of regression coefficients:
$$ P = \alpha\sum_{n=1}^N |\theta_n| $$
Though this is conceptually very similar to ridge regression, the results can differ surprisingly: for example, due to geometric reasons lasso regression tends to favor sparse models where possible: that is, it preferentially sets model coefficients to exactly zero. As a results Lasso can be readily used as an embedded-method in feature selection.

```python
from sklearn.linear_model import Lasso

model_L1 = Lasso(alpha = 1)

model_L1.fit(X_poly, yp)
predict = model_L1.predict(X_poly)

print("Coefficients ", model_L2.coef_, "\nIntercept ", model_L2.intercept_ )
```

    Coefficients  [ 0.00000000e+00 -2.32640141e-02  5.84347924e-05  6.13898052e-06
      3.33662650e-07  1.07952528e-08  3.27215854e-10  8.81663013e-12
      2.23665684e-13  5.17885431e-15  1.07560117e-16  1.79689238e-18
      1.39128809e-20 -6.59554607e-22 -4.74124289e-23 -2.09838984e-24
     -7.84618536e-26 -2.67816111e-27 -8.60338073e-29 -2.63977107e-30
     -7.79437033e-32 -2.22231243e-33 -6.12133462e-35 -1.62513656e-36
     -4.13374005e-38 -9.95388249e-40 -2.21399631e-41 -4.29129615e-43
     -5.93554881e-45  2.09363482e-47  6.17553472e-48  3.35292651e-49
      1.39778610e-50  5.17393937e-52  1.78342400e-53  5.84705960e-55
      1.84300545e-56  5.61572341e-58  1.65785897e-59  4.74022713e-61
      1.30860068e-62  3.46361529e-64  8.66817836e-66  1.99233509e-67
      3.90866418e-69  4.88106818e-71 -7.53423952e-73 -9.32437350e-74
     -5.01593599e-75 -2.19332730e-76 -8.69503304e-78]
    Intercept  1.8846505592013083

```python
plot_scikit_output (xp, yp, model_L1)
```

### Regularization and cross-validation performance

We can also illustrate the effects of regularization as they relate to training and validation performance.  Just as we did with the degree of the polynomial, we can consider the training and validation errors for different amounts of regularization.

```python
# Large Exercise

# Write one function for Ridge and one for Lasso that:
#   - take x, y, and the degree
#   - split the test data in train and validation set
#   - loop over different values for the regularization parameter
#   - for each iteration calculate the MSE on the train and test data
#   - plot the train and validation error
```

```python
def plot_L2_regression_performance (x, y, deg):

    err_train = []
    err_cv = []

    # train test split

    # np.logspace generates exponentially spaced values
    for alpha in np.logspace(-15,10,100):

        # create Polynomial Features

        # fit model

        # compute errors
        err_train.append()
        err_cv.append()

    plt.figure(figsize = (8,6))
    plt.loglog(np.logspace(-15,10,100), err_train, np.logspace(-15,10,100), err_cv)
    plt.legend(["Training", "Validation"])
    plt.xlabel("$\lambda$ (or alpha in scikit learn terms)")
    plt.ylabel("Mean squared error")
    plt.show()
```

```python
plot_L2_regression_performance(xp,yp,75)
```

A few points are worth emphasizing here.  First, the nature of the regularization term: lower $\lambda$ means _less_ regularization, whereas large $\lambda$ mean more regularization (eventually just essentially corresponding to all zero weights) results in the above shape.  Thus, larger $\lambda$ means _lower_ model complexity, so the x-axis of the figure works in the opposite direction as in the polynomial degree example.  Second, also note that we are using a _logarithmic_ scale on the x-axis (and the y-axis, as before, but the x-axis is the important part here).  This means that regularization typically works on a scale of _orders of magnitude_.  If you search over possible regularization terms, you'll want to do this search over a logarithmic space, because you need very large changes to the magnitude of $\lambda$ to really illustrate the full differences.  Third and last, just as was the case for the polynomial degree, we emphasize that the cross validation error is not a nice unimodal function of $\lambda$; there are multiple local optima owing to the pecularities of the particular polynomial, and it is not easy to globally optimize $\lambda$ by looking at cross validation error in some local region alone.  For this reason, techniques like grid searches are often more common in practice for finding model hyperparamters (including the $\lambda$ term), instead of techniques like gradient-based optimization.

**Exercise**: Write a small function to visualize how the $MSE$ improves as you increase $\alpha$ for a fixed choice of polynomial degree in LASSO regression.

```python
# YOUR CODE HERE

def plot_L1_regression_performance (x, y, deg):
    pass
```

```python
plot_L1_regression_performance (xp, yp, 50)
```

## General non-linear features

Using polynomials served as as good illustration of the basic principles of nonlinear features, generalization, and regularization, but they are far from the only such type of feature used in practice (and indeed, polynomials are probably a bit less common in most cases than other feature classes.  We also only covered polynomials for one dimensional "raw" inputs, where it was easy to enumerate all possible polynomials.  In this section we'll cover another type of common nonlinear feature, radial basis functions, and illustrate how to create both polynomials and radial basis functions over multi-dimensional raw inputs.

For the purposes of this section, we're going to adopt a slightly more explicit notation, though in general we're going to use it _only_ for this section.  Specifically, whereas before we used $x^{(i)}$ to generally refer to the input features to the algorithm, here we're going to use $x^{(i)} \in \mathbb{R}^n$ (or often just $x \in \mathbb{R}^n$, if we don't need to index over a data set), to refer to just the "raw" input features: i.e., in the case of our peak demand prediction problem $x$ would just refer to the high temperature
\begin{equation}
x^{(i)} \in \mathbb{R}^1 = \left [ \; \mathrm{HighTemperature}^{(i)} \; \right ]
\end{equation}
The raw inputs need not always be one dimensional, of course, for instance we previously used the example of including both the temperature and a day of the week flag as features
\begin{equation}
x^{(i)} \in \mathbb{R}^2 = \left [ \begin{array}{c} \mathrm{HighTemperature}^{(i)} \\ \mathrm{IsWeekday}^{(i)} \end{array} \right ]
\end{equation}

But note that here we don't include any of the polynomial features directly in $x$; instead, $x$ only captures the tru underlying inputs to the algorithm, the elements that we are providing that are not derived from the other quantities (and note that that it also doesn't include the constant feature, for instance).  Instead, we'll define a _feature mapping_
\begin{equation}
\phi : \mathbb{R}^n \rightarrow \mathbb{R}^k
\end{equation}
to be a function that maps $n$-dimensional inputs to $k$ dimensional _features_.  Everything else remains the same, except that we now consider the hypothesis function that is linear in these feature vectors, i.e.,
\begin{equation}
h_{\theta}(x) = \theta^T \phi(x)
\end{equation}
parameterized by $\theta \in \mathbb{R}^k$.

For example, for a degree-3 polynomial (in one input variable), we can define $\phi : \mathbb{R} \rightarrow \mathbb{R}^4$ as
\begin{equation}
\phi(x) = \left [ \begin{array}{c} x^3 \\ x^2 \\ x \\ 1 \end{array} \right ]
\end{equation}
and similarly for larger degree
polynomials.  Hopefully it is clear that this is just a notational definition, but it is useful for begin a bit more precise about these nonlinear features.

### `Radial basis function (RBF)`

If I were to make a completely anecdotal estimate, I would guess that the most frequently used type of nonlinear feature in not the polynomial, but something called the _radial basis function_ (this is actually the case both for explicit features and for the kernel methods we'll talk about shortly), often abreviated as RBF.  Radial basis functions are similar to polynomials in that they are non-linear functions of the input data, but they are notably different in that they are generally _local_ features: the value of any particular feature is close to zero for most of the input space, but non-zero in a small region around a "center" parameter.  Let's start with the definition, and we can then provide some illustrations that hopefully make this more concrete.  To keep this simple to start, we're only going to consider radial basis functions of one-dimensional raw inputs, though we'll shortly expand this to cover the general $n$-dimensional case.  A radial basis function feature vector is defined as the following:
\begin{equation}
\phi : \mathbb{R} \rightarrow \mathbb{R}^k = \left [ \begin{array}{c}
\exp \left(\frac{-(x - \mu^{(1)})^2}{2\sigma^2} \right) \\
\exp \left(\frac{-(x - \mu^{(2)})^2}{2\sigma^2} \right) \\
\vdots \\
\exp \left(\frac{-(x - \mu_{(k-1)})^2}{2\sigma^2} \right) \\
1 \end{array} \right ]
\end{equation}
where $\mu^{(1)},\ldots,\mu^{(k-1)} \in \mathbb{R}$ (called the means) and $\sigma \in \mathbb{R}$ (called the bandwidth) are the hyperparameters of this feature vector.

Let's look at a single one of these terms $\phi_j(x)$ (this is the $j$th element of the feature vector, because remember $\phi(x)$ outputs a $k$-dimensional vector.
\begin{equation}
\phi_j(x) = \exp \left(\frac{-(x - \mu^{(j)})^2}{2\sigma^2} \right)
\end{equation}
If you're familiar with the Gaussian distribution, you may recognize this as looking similar to the density function of the Gaussian (though without the normalizeng constant).  One single dimension of this feature (for varying inputs $x$, and here assuming mean $\mu^{(j)} = 1.5$ and $\sigma = 0.4$) looks like the following:

```python
x = np.linspace(-0.5,4.5,100)
mu = np.linspace(0,4,9)
sigma = 1
for mu_ in mu:
    plt.plot(x, np.exp(-(x-mu_)**2 / (2*sigma**2)))
plt.plot([-0.5,4.5], [1,1])
plt.xlim([-0.5,4.5])
plt.legend([r"$\phi_{" + str(j+1) + "}(x)$" for j in range(10)], bbox_to_anchor=(1.02,0.95))
plt.show()
```

The goal of nonlinear fitting with RBFs is to approximate the underlying function with a linear combination of these features.  By combining them in the proper manner, it is possible to approximate very general functions.

To see, this, let's go back again to the nonlinear version of the peak demand prediction problem.  We can construct a set of 10 dimensional RBFs spanning the minimum and maximum values of $x$.  For simplicity, we choose $\sigma$ to be equal to the distance between the means (this was done above, and seems reasonable, though we'll consider other ways of choosing $\sigma$ below).  Note also that there is no need to normalize the data, because the RBF features will always be scaled to be between zero and one (we could further normalize the generated features themselves, but this is typically not needed, as the features by definition will already be scaled to the range $[0,1]$).

```python
# create RBF features
def rbf_feat(x, mu, sig):
    return np.hstack([np.exp(-(x[:,None] - mu)**2/(2*sig**2)), np.ones((len(x),1))])

def plot_regression_rbf(theta, mu, sig):
    xt = np.linspace(-20,35, 400)
    yt = rbf_feat(xt, mu, sig) @ theta

    plt.figure(figsize = (8,6))
    plt.scatter(df["High_temp"], df["MAX"], marker="x")
    ylim = plt.ylim()
    plt.plot(xt, yt, 'C1')
    plt.xlabel("Temperature (C)")
    plt.ylabel("Peak Demand (GW)")
    plt.xlim([-18,35])
    plt.ylim(ylim)

def train_rbf(x, y, n_rbf):
    min_x, max_x = x.min(), x.max()
    sig = (max_x - min_x)/(n_rbf-1)
    mu = np.linspace(min_x, max_x, n_rbf-1)

    Phi = rbf_feat(x, mu, sig)
    theta = np.linalg.solve(Phi.T @ Phi, Phi.T @ y)
    return theta, mu, sig

x = df["High_temp"].values
plot_regression_rbf(*train_rbf(xp, yp, 10))
```

### Hyperparameters in RBFs
Unlike polynomials, where the only real hyperparameter relevant to the features themselves (not the regularization) is the degree of the polynomial, for RBF features there are a number of hyperparameter choices: the choice of centers themselves (and the number of centers as highlighted above), the bandwidth parameter $\sigma$, and the regularization parameter $\theta$.  It can be somewhat understand intuitively how we might trade off between all these different choices, and the good news is that there are some rules of thumb for choosing reasonable values for many of the hyperparameters without resorting to a grid search.  However, for the time being, we do want to briefly highlight the effect that the different hyperparameters have on the resulting performance.

**Effect of centers** We have already seen how the number of centers affects the fit of the data, so we will just briefly mention here that while the "obvious" choice for RBF centers on 1D data is to simply use an even grid over the input space, this doesn't work well for higher dimensions.

**Effect of regularization** Just like with polynomial features, we can add regularization to additionally smooth the function.  Unlike regularization for polynomial features, however, with a lot of narrow-peaked RBF functions, it is not trivial to fit the data with small weights.  This is precisely due to the local nature of the RBF features.  Because each features is only non-zero for a small part of the input space, we often cannot find a good fit to the data that has very low weights: there is no equivalent to the "just choosing low degree terms" as we did for the polynomial.

**Effect of bandwidth parameter** The effect of the bandwidth parameter, $\sigma$, can be somewhat less obvious.  At a high-level, though, the intuition that is important here is that larger $\sigma$ leads to _smoother_ feature RBF functions, which in turns leads to smoother final functions.

## Some other regression techniques

So far we have only talked about parameterized regression algorithms, i.e. algorithms that make relatively strong assumptions about the functional relationship between features and dependent variable (i.e. the target).

### `KNN regression`

Regression based on k-nearest neighbors. The target is predicted by local interpolation of the targets associated of the nearest neighbors in the training set.

Out of all the machine learning algorithms, KNN is easily the simplest to pick up. Despite it’s simplicity, it has proven to be effective at certain tasks. KNN can be used for both classification and regression problems. Although it is far more popular for classification problems it can perform well in regression tasks as well. One of the benefits of KNN is that it is an **unparametric** algorithm, i.e it does not make strong assumptions about the form of the mapping function. By not making assumptions, KNN is free to learn any functional form from the training data.

```python
x_train, x_test, y_train, y_test = train_test_split(xp, yp, test_size=0.3,random_state=10)
```

```python
from sklearn.neighbors import KNeighborsRegressor

#Fit model
KNN_reg = KNeighborsRegressor(n_neighbors=25)
KNN_model = KNN_reg.fit(x_train.reshape((-1,1)), y_train)

# Predict
y_hat_KNN = KNN_model.predict(x_test.reshape((-1,1)))
```

```python
print("Test set performance:")

print("MAE:",mean_absolute_error(y_hat_KNN, y_test), "GW")
print("RMSE:",(mean_squared_error(y_hat_KNN, y_test))**(0.5), "GW")
#print("R2:",r2_score(y_hat_KNN, y_test))
```

    Test set performance:
    MAE: 0.1276345969343066 GW
    RMSE: 0.1598484899732429 GW

Let us visualize the results...

```python
plt.figure(figsize = (8,6))
plt.scatter(x_train, y_train, marker="x")
plt.plot(np.arange(-18,40,1), KNN_reg.predict(np.arange(-18,40,1).reshape((-1,1))), marker="x", color='C1')
plt.xlabel("High Temperature (°C)")
plt.ylabel("Peak Demand (GW)")
plt.show()
```

**Exercise**: What is a good choice of the number of neighbors `n_neighbors`, the key hyperparameter in KNN regression? Back up your answer with numbers. To do so write a small loop to test different values for `n_neighbors` (i.e., perform a grid search over different choices for `n_neighbors`).

```python
# YOUR CODE HERE:

def find_knn():
    pass
```

```python
find_knn (xp,yp,max_k=50)
```

### `Tree-based regression`

```python
# Import the necessary modules and libraries

from sklearn.tree import DecisionTreeRegressor, plot_tree

# Fit regression model
Tree_reg = DecisionTreeRegressor(max_depth=5)
tree_model = Tree_reg.fit(x_train.reshape((-1,1)), y_train)

# Predict
y_hat_tree = tree_model.predict(x_test.reshape((-1,1)))
```

```python
print("Test set performance:")

print("MAE:",mean_absolute_error(y_hat_tree, y_test), "GW")
print("RMSE:",(mean_squared_error(y_hat_tree, y_test))**(0.5), "GW")
#print("R2:",r2_score(y_hat_tree, y_test))
```

    Test set performance:
    MAE: 0.12751480011005473 GW
    RMSE: 0.1595320383040373 GW

```python
plt.figure(figsize = (8,6))
plt.scatter(x_train, y_train, marker="x")
plt.plot(np.arange(-18,40,1), Tree_reg.predict(np.arange(-18,40,1).reshape((-1,1))), marker="x", color='C1')
plt.xlabel("High Temperature (°C)")
plt.ylabel("Peak Demand (GW)")
plt.show()
```

```python
plot_tree(tree_model)
```

Play around with the `max_depth` parameter. What do you observe? How do you think the underlying tree looks like?

**Exercise**: What is a good choice of the tree depths `max_depth`, the key hyperparameter in tree regression? Back up your answer with numbers. To do so, write a small loop to test different values for `n_neighbors` and the respective error metrics. Visualize your results.

```python
# YOUR CODE HERE:
def find_tree_depth():
    pass
```

```python
find_tree_depth (xp,yp)
```

The above exercises are what is commonly known as **grid searching an algorithm**. Grid searching is the practice of testing a large range of model hyperparameters via brute force. It is a key component of model selection and evaluation and should be carried out very thoroughly!

---


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## 5 Scalar and Vector Operators

Chapter 5 in [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

By the end of this session, you should be able to:
- Understand the concept of operators
- Apply some operators to functions
- Understand how to apply multiple operators in order (right to left, inside out)
- Understand that some operators do not commute, and we use the [A, B] "commutator" to test that

### Scalars vs. Vectors

Reminder...

**Scalars** are just numbers - they have a magnitude, a size. The mass of a molecule would be an example, e.g., 120 amu.

**Vectors** have both a magnitude and a direction:
- velocity $\mathbf{v}$
- acceleration $\mathbf{a}$
- force $\mathbf{F}$
- electric field $\mathbf{E}$

### Functions

A **function** takes in a number, a vector, etc. and returns a number:

$$
\sin 0 = 0
$$

### Operators

An operator takes in a **function** and returns a *new function*.

We've already worked with lots of operators, we just didn't call them that at the time.

Examples of mathematical operations include multiplying a function by a constant or taking the derivative of a function.

**Consider the operator $\hat{A}$**

$$
\hat{A} f(x)=2 f(x)
$$

We'd say this as "the operator $\hat{A}$ acts on the function $f (x)$. In this case, we can see that $\hat{A}$ is simply the ***doubling*** operator. Whatever $f(x)$ is, $\hat{A}$ doubles it by multiplying by 2.

**Another example might be $\hat{B}$**

$$
\hat{B} f(x, y)=(f(x, y))^{2}
$$

Here, $\hat{B}$ acting on $f(x)$ is the ***square*** operator - whatever $f(x)$ might be, the resulting function is the square of that function.

**We also have the differential operator $\hat{d_x}$**

$$
\hat{d_x} f(x,y) = \frac{\partial}{\partial x}[f(x,y)]
$$

**What's the point?** As the algebra and calculus get a bit more involved, operators give us a way to write more complex math in a shorter way.

Operators may seem abstract. It's often better to consider them acting on some function. But most mathematical operations can be written as operators:
- adding
- subtracting
- multiplying (by a scalar or vector)
- dividing
- raising a function to a power
- taking a derivative
- integrating
- translating coordinates
- rotating
- (etc.)

We'll see in quantum chemistry that certain physical properties can be written as operators, for example:

**Linear momentum operator $\hat{p_x}$**

$$
\hat{p_x} f(x,y) = -i\hbar \frac{\partial}{\partial x}[f(x,y)]
$$

where $\hbar$ is the [*reduced Planck constant*](https://en.wikipedia.org/wiki/Planck_constant) $h / 2\pi$.

There are meaningful consequences to that $i$ in the linear momentum operator...

**Kinetic energy operator $\hat{T}$**

Since kinetic energy of a particle can be written as:

$$
E = \frac{p^2}{2m}
$$

The kinetic energy operator is:
$$
\hat{T} = \frac{\hat{p}^2}{2m} = \frac{1}{2m} \hat{p}^2
$$

where $m$ is the mass of the particle.

**Math with operators** We can (carefully) combine and modify operators. Not all operators commute, so we have to take them in order - inside to out (e.g., right to left).

Normally, we write out multiple operations as if they are ***multiplied***.

**Example**:

$$
    x\hat{d_x}f(x) = x \cdot \frac{\partial}{\partial x}f(x)
$$

$$
    \hat{d_x}x f(x) = \frac{\partial}{\partial x}[x \cdot f(x)]
$$

**Linear Operators**

Linear operators can be distributed over multiple terms of a function.

For example, multiplying by 2 or taking a derivative are both linear functions.

$$
2\times(f(x) + g(x) = 2f(x) + 2g(x)
$$

Taking the derivative:

$$
\hat{d_x}(f(x) + g(x)) = \hat{d_x}f(x) + \hat{d_x}g(x) = \frac{\partial}{\partial x}f(x) + \frac{\partial}{\partial x}g(x)
$$

That "square" operator up above, $\hat{B}$ is *not* a linear function.

$$
(x^2 + 3x)^2 \neq (x^2)^2 + (3x)^2
$$

```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.6.2 (Python 3.6.11-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.6.2/

```python
(x**2)**2 + (3*x)**2
```

```python
(x**2 + 3*x)**2
```

Hmph. Sympy sometimes needs a nudge to expand or simplify expressions.

```python
expand((x**2 + 3*x)**2)
```

Obviously, taking the square of a function isn't a linear function - in our example, we have the $6x^3$ term there, which comes from multiplying $x^2$ and $3x$ together, twice.

Fortunately, in most cases operators we need for physics and chemistry are usually linear.

Let's consider "multiply by x" $\hat{x}$ and "take a derivative" $\hat{d_x}$ operators:

```python
# let's set up an example function to test out some of these operators
f = x**2 + 3*x
```

```python
x * diff(f, x)
```

```python
diff(x*f, x)
```

Okay, so that's a good reminder.

Operators do not necessarily commute. If I multiply by x and take a derivative, I end up with a very different result than if I take a derivative and multiply the result by x.

**Commuting Operators**

Quick reminder: [commuting](https://en.wikipedia.org/wiki/Commutative_property) means that you can add or multiply in any order:

$$
2\times 3 = 3 \times 2 = 6
$$

$$
101 + 52 = 52 + 101 = 153
$$

As we saw above with "multiply by x" and "take the partial derivative with respect to y" operators, operator order matters.

Sometimes we check this, with a concept called the "commutator":

$$
[\hat{A}, \hat{B}] = \hat{A}\hat{B} - \hat{B}\hat{A}
$$

If the two operators commute, then this will obviously equal zero - you can take them in any order. In some cases, this won't be zero.

We're going through all this because in quantum mechanics, all the properties of a system (e.g., an atom, a molecule, etc.) can be described by "wavefunctions" $\psi(x)$.

Any physical observable, like "find the position of" or "what is the energy of" corresponds to specific operators like the linear momentum operator.

If two operators commute, we can make both measurements successfully.

*If two operators do not commute, weird things happen (e.g., we can't know the exact location and momentum of an electron at the same time).*

Let's see this in action with "multiply by x" $\hat{x}$ (which is actually the 'measure the position' operator) and linear momentum $\hat{p} = -i\hbar\frac{\partial}{\partial x}$:

```python
# we set this above, but we can use any function
f = x**2 + 3*x
i, hbar = symbols("i hbar")
```

```python
# let's consider x * p(f)
x * (-i * hbar)*diff(f, x)
```

```python
# now let's see p(x*f)
(-i * hbar)*diff(x*f, x)
```

```python
# and we'll take the difference
part_one = x * (-i * hbar)*diff(f, x)
part_two = (-i * hbar)*diff(x*f, x)
part_one - part_two
```

Obviously, that can be simplified further - we need to give Sympy a nudge:

(Even computer programs can be lazy.)

```python
simplify(part_one - part_two)
```

We also mentioned the linear momentum and kinetic energy operators back there...

**Linear momentum:**

$$
$\hat{p} = -i\hbar\frac{\partial}{\partial x}$
$$

**Kinetic energy:**

$$
\hat{T} = \frac{\hat{p}^2}{2m} = \frac{1}{2m} \hat{p}^2
$$

```python
# here, we'll apply linear momentum operator to "measure" our function
# (e.g., f(x) describes some electron)
(-i * hbar)*diff(f, x)
```

```python
# Okay, if we want to "measure" the kinetic energy
# we want 1/2m * p**2
# p**2 = take the second derivative
# and then we have some constant terms
p_squared = (-i * hbar)**2 * diff(f, x, 2)
t = 0.5 * p_squared / m
```

```python
# print t with some nice formatting
t
```

As a human, we can see a little simplification (e.g. $i^2 = -1$) but otherwise it's fine:

$$
\hat{t} = \frac{-\hbar^2 y^2}{m}
$$

**One more thing...**

We had this concept of the Euler test to see if a total differential was an exact differential.

$$
\hat{d_x} = \frac{\partial}{\partial_x}
$$

$$
\hat{d_y} = \frac{\partial}{\partial_y}
$$

Since we can apply operators sequentially:

$$
\hat{d_y}\hat{d_x}f(x,y) = \frac{\partial}{\partial_y} [ \frac{\partial}{\partial_x}[f(x,y)] ]
$$

Then the Euler test would be:
$$
\hat{d_y}\hat{d_x}f(x,y) - \hat{d_x}\hat{d_y}f(x,y) = 0
$$

Or we can use the commutator notation:

$$
[\hat{d_x}, \hat{d_y}] = 0
$$

```python
f = x**2*y**2 + 3*x*y + 4
```

```python
diff(f, x, y)
```

```python
diff(f, y, x)
```

```python
diff(f, x, y) - diff(f, y, x)
```

So the big points for now:
- Operators are used in quantum mechanics to describe physical properties like energy and momentum
- Some operators commute. Those that don't mean that those physical properties can't be measured at the same time.
  - e.g. position and momentum in the [Heisenberg Uncertainty Principle](https://en.wikipedia.org/wiki/Uncertainty_principle)
- We like operators because they help us write complicated formulas in a shorter way
  - they're kind of like little mini programs (e.g. "take the derivative of this twice, then multiply by some constants")
- It's a bit abstract for now, but we'll have concrete, chemical examples soon

-------
This notebook is from Prof. Geoffrey Hutchison, University of Pittsburgh
https://github.com/ghutchis/chem1000


# Logistic Regression

## Introduction

In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable $y_i$ is based on some
independent variables $\hat{x}_i$. Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters $\hat{\beta}$ to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.

Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients' brains, figure out if there is a tumor or
not; or given a specific physical system, we'd like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.

The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.

Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters $\hat{\beta}$. The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
family of gradient descent methods. The latter are the working horses
of basically all modern machine learning algorithms.

We note also that many of the topics discussed here on logistic
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.

## Basics

We consider the case where the dependent variables, also called the
responses or the outcomes, $y_i$ are discrete and only take values
from $k=0,\dots,K-1$ (i.e. $K$ classes).

The goal is to predict the
output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
made of $n$ samples, each of which carries $p$ features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.

Let us specialize to the case of two classes only, with outputs
$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a
credit card user that could default or not on her/his credit card
debt. That is

$$
y_i = \begin{bmatrix} 0 & \mathrm{no}\\  1 & \mathrm{yes} \end{bmatrix}.
$$

Before moving to the logistic model, let us try to use our linear
regression model to classify these two outcomes. We could for example
fit a linear model to the default case if $y_i > 0.5$ and the no
default case $y_i \leq 0.5$.

We would then have our
weighted linear combination, namely

$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} +  \hat{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
$$

where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.

The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels $y_i$ are discrete variables. A typical example is the credit
card data discussed below here, where we can set the state of
defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons
in the data set (see the full example below).

One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values $\{0,1\}$,
$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks. Historically it is called the ``perceptron" model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft" classifier that outputs
the probability of a given category. This leads us to the logistic function.

The following example on data for coronary heart disease (CHD) as function of age may serve as an illustration. In the code here we read and plot whether a person has had CHD (output = 1) or not (output = 0). This ouput  is plotted the person's against age. Clearly, the figure shows that attempting to make a standard linear regression fit may not be very meaningful.

```python
%matplotlib inline

# Common imports
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.utils import resample
from sklearn.metrics import mean_squared_error
from IPython.display import display
from pylab import plt, mpl
plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

infile = open(data_path("chddata.csv"),'r')

# Read the chd data as  csv file and organize the data into arrays with age group, age, and chd
chd = pd.read_csv(infile, names=('ID', 'Age', 'Agegroup', 'CHD'))
chd.columns = ['ID', 'Age', 'Agegroup', 'CHD']
output = chd['CHD']
age = chd['Age']
agegroup = chd['Agegroup']
numberID  = chd['ID']
display(chd)

plt.scatter(age, output, marker='o')
plt.axis([18,70.0,-0.1, 1.2])
plt.xlabel(r'Age')
plt.ylabel(r'CHD')
plt.title(r'Age distribution and Coronary heart disease')
plt.show()
```

What we could attempt however is to plot the mean value for each group.

```python
agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])
group = np.array([1, 2, 3, 4, 5, 6, 7, 8])
plt.plot(group, agegroupmean, "r-")
plt.axis([0,9,0, 1.0])
plt.xlabel(r'Age group')
plt.ylabel(r'CHD mean values')
plt.title(r'Mean values for each age group')
plt.show()
```

We are now trying to find a function $f(y\vert x)$, that is a function which gives us an expected value for the output $y$ with a given input $x$.
In standard linear regression with a linear dependence on $x$, we would write this in terms of our model

$$
f(y_i\vert x_i)=\beta_0+\beta_1 x_i.
$$

This expression implies however that $f(y_i\vert x_i)$ could take any
value from minus infinity to plus infinity. If we however let
$f(y\vert y)$ be represented by the mean value, the above example
shows us that we can constrain the function to take values between
zero and one, that is we have $0 \le f(y_i\vert x_i) \le 1$. Looking
at our last curve we see also that it has an S-shaped form. This leads
us to a very popular model for the function $f$, namely the so-called
Sigmoid function or logistic model. We will consider this function as
representing the probability for finding a value of $y_i$ with a given
$x_i$.

## The logistic function

Another widely studied model, is the so-called
perceptron model, which is an example of a "hard classification" model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
$y_i=0$ or $y_i=1$). In many cases, and the coronary heart disease data forms one of many such examples, it is favorable to have a "soft"
classifier that outputs the probability of a given category rather
than a single value. For example, given $x_i$, the classifier
outputs the probability of being in a category $k$.  Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point $x_i$
belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,

$$
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$

Note that $1-p(t)= p(-t)$.

The following code plots the logistic function, the step function and other functions we will encounter from here and on.

```python
"""The sigmoid function (or the logistic curve) is a
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm."""

import numpy
import matplotlib.pyplot as plt
import math as mt

z = numpy.arange(-5, 5, .1)
sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
sigma = sigma_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, sigma)
ax.set_ylim([-0.1, 1.1])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('sigmoid function')

plt.show()

"""Step Function"""
z = numpy.arange(-5, 5, .02)
step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
step = step_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, step)
ax.set_ylim([-0.5, 1.5])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('step function')

plt.show()

"""tanh Function"""
z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
t = numpy.tanh(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, t)
ax.set_ylim([-1.0, 1.0])
ax.set_xlim([-2*mt.pi,2*mt.pi])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('tanh function')

plt.show()
```

## Two parameters

We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities

$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
\end{align*}
$$

where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.

Note that we used

$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
$$

In order to define the total likelihood for all possible outcomes from a
dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels
$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is

$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$

from which we obtain the log-likelihood and our **cost/loss** function

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
$$

Reordering the logarithms, we can rewrite the **cost/loss** function as

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that

$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.

The cross entropy is a convex function of the weights $\hat{\beta}$ and,
therefore, any local minimizer is a global minimizer.

Minimizing this
cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$

and

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$

Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
derivative of cost function as

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
$$

If we in addition define a diagonal matrix $\hat{W}$ with elements
$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as

$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
$$

### Extending to more predictors

Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors

$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$

Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to

$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$

### Including more classes

Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to $K$ classes.  Let us for the sake
of simplicity assume we have only two predictors. We have then following model

$$
\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
$$

and

$$
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
$$

and so on till the class $C=K-1$ class

$$
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
$$

and the model is specified in term of $K-1$ so-called log-odds or
**logit** transformations.

In our discussion of neural networks we will encounter the above again
in terms of a slightly modified function, the so-called **Softmax** function.

The softmax function is used in various multiclass classification
methods, such as multinomial logistic regression (also known as
softmax regression), multiclass linear discriminant analysis, naive
Bayes classifiers, and artificial neural networks.  Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of $K$ distinct linear functions,
and the predicted probability for the $k$-th class given a sample
vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two
predictors):

$$
p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
$$

It is easy to extend to more predictors. The final class is

$$
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
$$

and they sum to one. Our earlier discussions were all specialized to
the case with two classes only. It is easy to see from the above that
what we derived earlier is compatible with these equations.

## Wisconsin Cancer Data

We show here how we can use a simple regression case on the breast
cancer data using Logistic regression as our algorithm for
classification.

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import  train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression

# Load the data
cancer = load_breast_cancer()

X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
logreg.fit(X_train, y_train)
print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
#now scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Logistic Regression
logreg.fit(X_train_scaled, y_train)
print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
```

In addition to the above scores, we could also study the covariance (and the correlation matrix).
We use **Pandas** to compute the correlation matrix.

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import  train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
cancer = load_breast_cancer()
import pandas as pd
# Making a data frame
cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)

fig, axes = plt.subplots(15,2,figsize=(10,20))
malignant = cancer.data[cancer.target == 0]
benign = cancer.data[cancer.target == 1]
ax = axes.ravel()

for i in range(30):
    _, bins = np.histogram(cancer.data[:,i], bins =50)
    ax[i].hist(malignant[:,i], bins = bins, alpha = 0.5)
    ax[i].hist(benign[:,i], bins = bins, alpha = 0.5)
    ax[i].set_title(cancer.feature_names[i])
    ax[i].set_yticks(())
ax[0].set_xlabel("Feature magnitude")
ax[0].set_ylabel("Frequency")
ax[0].legend(["Malignant", "Benign"], loc ="best")
fig.tight_layout()
plt.show()

import seaborn as sns
correlation_matrix = cancerpd.corr().round(1)
# use the heatmap function from seaborn to plot the correlation matrix
# annot = True to print the values inside the square
plt.figure(figsize=(15,8))
sns.heatmap(data=correlation_matrix, annot=True)
plt.show()
```

In the above example we note two things. In the first plot we display
the overlap of benign and malignant tumors as functions of the various
features in the Wisconsing breast cancer data set. We see that for
some of the features we can distinguish clearly the benign and
malignant cases while for other features we cannot. This can point to
us which features may be of greater interest when we wish to classify
a benign or not benign tumour.

In the second figure we have computed the so-called correlation
matrix, which in our case with thirty features becomes a $30\times 30$
matrix.

We constructed this matrix using **pandas** via the statements

```python
cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)
```

and then

```python
correlation_matrix = cancerpd.corr().round(1)
```

Diagonalizing this matrix we can in turn say something about which
features are of relevance and which are not. This leads  us to
the classical Principal Component Analysis (PCA) theorem with
applications. This will be discussed later this semester ([week 43](https://compphysics.github.io/MachineLearning/doc/pub/week43/html/week43-bs.html)).

### Other measures in classification studies: Cancer Data  again

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import  train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression

# Load the data
cancer = load_breast_cancer()

X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
logreg.fit(X_train, y_train)
print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
#now scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Logistic Regression
logreg.fit(X_train_scaled, y_train)
print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))

from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import cross_validate
#Cross validation
accuracy = cross_validate(logreg,X_test_scaled,y_test,cv=10)['test_score']
print(accuracy)
print("Test set accuracy with Logistic Regression  and scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))

import scikitplot as skplt
y_pred = logreg.predict(X_test_scaled)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
plt.show()
y_probas = logreg.predict_proba(X_test_scaled)
skplt.metrics.plot_roc(y_test, y_probas)
plt.show()
skplt.metrics.plot_cumulative_gain(y_test, y_probas)
plt.show()
```


# "Interactive and visualize linear regression in python"
> "Interactive and visualize linear regression in python"

- toc:true
- branch: master
- badges: true
- comments: true
- categories: [ml]
- tags: [jupyter, linear regression, regression]

Linear regression assumes relation between input variable and output. It predicts the output as a linear function from input variables.

Various methods of Linear Regression
- **Simple Linear Regression**
 - Requires to calculate mean, std-deviation, co-variance, corelation
- **Ordinary Least Squares**
 - uses Linear Algebra to estimate best values for co-efficients
- **Gradient Descent**
 - Iteratively miniize the error of model this it gets to a minimum
   - Batch Gradient descent
   - Scochastic gradient descent
   - mini batch gradient descent
- **Regularized Linear Regression**
 - Lasso Regression
 - Ridge Regression

# Introduction and Terminology

Lets say we have below sample data.

| size in feet | price ($) |
| ------  | ----- |
| 2104 | 460 |
| 1416 | 232 |

$m$ = number of training example

$x$ = input variable/ feature/ independent variable/ predictor

$y$ = output variable/ target variable/ dependent variable

How do we represent h ?

**Hypothesis**:
$h_\theta(x) = \theta_0 + \theta_1 * x$

$h(x)$ is predictor of $y$

This is linear regression with one variable

Let's express this as Matrix and vectors

$h_\theta(x) = \begin{pmatrix} x \\ 1\end{pmatrix}^T * \begin{pmatrix} \theta_1 \\ \theta_0\end{pmatrix}$

Let $ X = \begin{pmatrix} x \\ 1\end{pmatrix}^T$ and $\theta = \begin{pmatrix} \theta_1 \\ \theta_0\end{pmatrix}$

we can write it as $ h_\theta(x) = X * \theta $

We need to choose $\theta_0$ and $\theta_1$ so it best fits our function

# Simple Linear Regression

- uses stats to estimate co-efficients
- need to calculate mean, std-devitation, co-variance and corelation.

Lets generate some data for simple linear regression. Lets take house price prediction example

Linear equation for our calculations is $y = \theta_1 * x + \theta_0$

**Steps**:

- **Estimate the slope $\theta_1$**

\begin{equation}
\\\theta_1 = \frac{\sum(x_i – mean(x)) (y_i – mean(y)} {\sum(x_i – mean(x))^2}
\end{equation}

where

\begin{equation}
mean(x) = \frac{1}{n}\sum_{i=1}^n x_i
\end{equation}

- **Estimate the intercept $\theta_0$**

\begin{equation}
\theta_0 = mean(y) – \theta_1 * mean(x)
\end{equation}

- **Estimating Error**

Root Mean Square Error

$RMSE = \sqrt{(\frac{1}{n})\sum_{i=1}^{n}(h_{i} - y_{i})^{2}}$

$h_i$ is predicted value and $y_i$ is actual value

```
# imports
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
```

### Generate Data

```
# Generate 'random' data
np.random.seed(0)
no_of_records=200
sq_ft = 50 * np.random.rand(no_of_records) + 500  # Array of 100 values with mean = 1000
res = 5 * np.random.randn(no_of_records)       # Generate 100 residual terms
price = 2 * sq_ft + 1000 + res                  # Actual values of Y

# create pandas dataframe to store size and price
df = pd.DataFrame(
    {'sq_ft': sq_ft,
     'price': price}
)

# first five rows of our dataframe
df.head()
```

      sq_ft
      price

      0
      527.440675
      2060.514530

      1
      535.759468
      2066.119279

      2
      530.138169
      2054.538994

      3
      527.244159
      2052.299218

      4
      521.182740
      2039.875318

### Visualize Data

```
X = df.iloc[:, 0]
Y = df.iloc[:, 1]
plt.title('House Price vs Sq ft')
plt.xlabel('Sq ft')
plt.ylabel('House Price')
plt.scatter(X,Y)
```

### Build the model

Lets calculate

\begin{equation}
\\m = \frac{\sum(x_i – mean(x)) (y_i – mean(y)} {\sum(x_i – mean(x))^2}
\end{equation}

\begin{equation}
c = mean(y) – \theta_1 * mean(x)
\end{equation}

```
# Building the model
X_mean = np.mean(X)
Y_mean = np.mean(Y)

num = 0
den = 0
for i in range(len(X)):
    num += (X[i] - X_mean)*(Y[i] - Y_mean) # sum of all differences in numerator
    den += (X[i] - X_mean)**2 # sum of all differences in denom
m = num / den # final slope \theta_1
c = Y_mean - m*X_mean # final intercept \theta_0

print (m, c)
```

    1.956195597832525 1022.4369105614524

Now our linear equation is $House Price (Y) = (1.956195597832525) * x + (1022.4369105614524)$

Lets plot and visualize it

```
m = 1.956195597832525
b = 1022.4369105614524
Y_pred = m*X+b # predicted values
plt.scatter(X,Y)
plt.plot(X,Y_pred,color='red')
plt.show()
```

### Check accuracy

Lets check how accurate is our model by estimating error.

$RMSE = \sqrt{(\frac{1}{n})\sum_{i=1}^{n}(h_{i} - y_{i})^{2}}$

```
Y = df.iloc[:, 1] # observed
Y_pred
summation = 0  #summation of differences
n = len(Y) #no of training examples
for i in range (0,n):
  difference = Y[i] - Y_pred[i]  #finding the difference between observed and predicted value
  squared_difference = difference**2  #square of the differene
  summation = summation + squared_difference  #sum of all the differences
MSE = summation/n  #dividing summation by no of training examples
print("ROOT Mean Square Error is: {}".format(MSE))
```

    ROOT Mean Square Error is: 22.826601317067663

This mean 22% of time our estimation is not accurate.

# Gradient Descent

### Cost function & Gradient Descents

Lets see how can we choose parameters $\theta_0$ and $\theta_1$

Let y be result after substitution

**Cost Function**:

**Loss Function**
$J(\theta_0, \theta_1) = \sum\limits_{i=1}^m (y_{predicted} - y_{actual})^2$

**MSE**
$J(\theta_0, \theta_1) = \dfrac{1}{2m}\sum\limits_{i=1}^m (h_\theta(x^i) - y^i)^2$

**Goal** : minimize $J(\theta_0, \theta_1)$

Linear regression uses `Gradient Descent` to minimize the cost Function

**Update rules**
\begin{aligned}
&\theta_{0}:=\theta_{0}-\alpha \frac{\partial}{\partial \theta_{0}} J\left(\theta_{0}, \theta_{1}\right)\\
&\theta_{1}:=\theta_{1}-\alpha \frac{\partial}{\partial \theta_{1}} J\left(\theta_{0}, \theta_{1}\right)
\end{aligned}

**Partial derivatives**

\begin{array}{l}
\frac{\partial}{\partial \theta_{0}} J\left(\theta_{0}, \theta_{1}\right)=\frac{1}{m} \sum_{i=1}^{m}\left(h_{\theta}\left(x^{(i)}\right)-y^{(i)}\right) \\
\frac{\partial}{\partial \theta_{1}} J\left(\theta_{0}, \theta_{1}\right)=\frac{1}{m} \sum_{i=1}^{m}\left(h_{\theta}\left(x^{(i)}\right)-y^{(i)}\right) \cdot x^{(i)}
\end{array}

### Gradient Descent procedure

- start with initial values for coefficients

> coefficient = 0.0

- calculate cost of coefficient

> cost = f(coefficient)

- calculate derivative of cost function

> delta = derivative(cost)

- update coefficients

> coefficient = coefficient - (alpha * delta)

- This process is updated until cost function is minimum

### Batch Gradient descent

**Pseudo Code**
```
while True:
	Wgradient = evaluate_gradient(loss, data, W)
	W += -alpha * Wgradient
```

```
#populate x with input variable (sq_ft) and y with actual output variable (price)
x = df['sq_ft']
y = df['price']

x = (x - x.mean()) / x.std()
x = np.c_[np.ones(x.shape[0]), x]

#Gradient Descent Algo

learning_rate = 0.01 #learning rate (or) step_size
iterations = 1000 #no of iterations
m = y.size #no of training examples
np.random.seed(123) #Set the seed
theta = np.random.rand(2) #Pick some random values to start with

def  cal_cost(theta,x,y):

    m = len(y)

    predictions = x.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

def cal_co_efficients(theta, x, y):
    predictions = x.dot(theta)
    error = predictions-y
    theta = theta - (learning_rate * (1/m) * np.dot(x.T, error))
    return theta

#gradient descent function
def gradient_descent(x, y, theta, iterations, learning_rate):
    prev_costs = []
    prev_thetas = [theta]
    for i in range(iterations):
       # calculating predicted output y_predicted = m * x + c
        cost = cal_cost(theta, x, y)
        prev_costs.append(cost)
        theta = cal_co_efficients(theta, x, y)
        prev_thetas.append(theta)

    return prev_thetas, prev_costs

#Pass the relevant variables to the function and get the new values back...
prev_thetas, prev_costs = gradient_descent(x, y, theta, iterations, learning_rate)
theta = prev_thetas[-1]

#Print the results...
print("Gradient Descent: {:.2f}, {:.2f}".format(theta[0], theta[1]))
```

    Gradient Descent: 2049.39, 27.84

```
#Plot the cost function...
plt.title('Cost Function J')
plt.xlabel('No. of iterations')
plt.ylabel('Cost')
plt.plot(past_costs)
plt.show()
```

#### Stochastic Gradient descent pseudo code

[]

**Pseudo Code**
```
while True:
	batch = next_training_batch(data, 256)
	Wgradient = evaluate_gradient(loss, batch, W)
	W += -alpha * Wgradient
```

```
#populate x with input variable (sq_ft) and y with actual output variable (price)
x = df['sq_ft']
y = df['price']

x = (x - x.mean()) / x.std()
x = np.c_[np.ones(x.shape[0]), x]

#Gradient Descent Algo

learning_rate = 0.01 #learning rate (or) step_size
iterations = 50 #no of iterations
m = y.size #no of training examples
np.random.seed(123) #Set the seed
theta = np.random.rand(2) #Pick some random values to start with

def  cal_cost(theta,x,y):

    m = len(y)

    predictions = x.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

def cal_co_efficients(theta, x, y):
    predictions = x.dot(theta)
    error = predictions-y
    theta = theta - (learning_rate * (1/m) * np.dot(x.T, error))
    return theta

#gradient descent function
def stocashtic_gradient_descent(x, y, theta, iterations, learning_rate):
    prev_costs = []
    prev_thetas = [theta]
    cost =0.0
    for i in range(iterations):
        rand_ind = np.random.randint(0,iterations)
       # calculating predicted output y_predicted = m * x + c
        cost = cal_cost(theta, x, y)
        prev_costs.append(cost)
        theta = cal_co_efficients(theta, x, y)
        prev_thetas.append(theta)

    return prev_thetas, prev_costs

#Pass the relevant variables to the function and get the new values back...
prev_thetas, prev_costs = stocashtic_gradient_descent(x, y, theta, iterations, learning_rate)
theta = prev_thetas[-1]

#Print the results...
print("Stocashtic Gradient Descent: {:.2f}, {:.2f}".format(theta[0], theta[1]))
```

    Stocashtic Gradient Descent: 809.95, 11.13

```
#Plot the cost function...
plt.title('Cost Function J')
plt.xlabel('No. of iterations')
plt.ylabel('Cost')
plt.plot(prev_costs)
plt.show()
```

### Batch Gradient descent

```
#populate x with input variable (sq_ft) and y with actual output variable (price)
x = df['sq_ft']
y = df['price']

x = (x - x.mean()) / x.std()

#Gradient Descent Algo

learning_rate = 0.01 #learning rate (or) step_size
iterations = 10 #no of iterations
batch_size = 20
m = y.size #no of training examples
np.random.seed(123) #Set the seed
theta = np.random.randn(2) #Pick some random values to start with

def  cal_cost(theta,x,y):

    m = len(y)

    predictions = x.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

def cal_co_efficients(theta, x, y):
    predictions = x.dot(theta)
    error = predictions-y
    theta = theta - (learning_rate * (1/m) * np.dot(x.T, error))
    return theta

#gradient descent function
def minibatch_gradient_descent(x, y, theta, iterations, learning_rate, batch_size):
    m = len(y)
    n_batches = int(m/batch_size)
    prev_costs = []
    prev_thetas = [theta]
    for i in range(iterations):
        cost =0.0
        indices = np.random.permutation(m)
        x = x[indices]
        y = y[indices]
        for i in range(0,m,batch_size):
            x_i = x[i:i+batch_size]
            y_i = y[i:i+batch_size]
            x_i = np.c_[np.ones(len(x_i)),x_i]
            cost += cal_cost(theta,x_i,y_i)
            theta = cal_co_efficients(theta, x_i, y_i)
        prev_costs.append(cost)
        prev_thetas.append(theta)
    return prev_thetas, prev_costs

#Pass the relevant variables to the function and get the new values back...
prev_thetas, prev_costs = minibatch_gradient_descent(x, y, theta, iterations, learning_rate, batch_size)
theta = prev_thetas[-1]

#Print the results...
print("Minibatch Gradient Descent: {:.2f}, {:.2f}".format(theta[0], theta[1]))
```

    Minibatch Gradient Descent: 194.15, 3.54

```
#Plot the cost function...
plt.title('Cost Function J')
plt.xlabel('No. of iterations')
plt.ylabel('Cost')
plt.plot(prev_costs)
plt.show()
```


---
 # With great growth come great errors
 ---

```python
import numpy as np
np.version.version
```

    '1.14.3'

Lets compute derivate based on the stencil method [link](http://web.media.mit.edu/~crtaylor/calculator.html)
I have chosen locations of sampled points equal to 0,1,2,3,4,5 since I do not want use negative locations with limited data.

```python
def df_1(f,h):
    f = np.float64(f)
    h = np.float64(h)
    df = lambda f, h:((-137*f[0]+300*f[1]-300*f[2]+200*f[3]-75*f[4]+12*f[5])/(60*1.0*h))
    return [df(f[j:j+6], 1) for j in range(0,f.size-6)]
```

Lets take strongly monotonicly growing function:

```python
TestFun  = np.asarray([np.power(2,x)+1 for x in range(50)])
```

We know that we know that f is monotonicly growing function, so we can present it as $f(x)=e^{g(x)}$, if we differentiate it we will get:

$\frac{d}{dx}f(x)=\frac{d}{dx}(e^{g(x)})=e^{g(x)}\frac{d}{dx}g(x)$ so:

\begin{equation}\tag{1} \frac{d}{dx}g(x) = \frac{\frac{d}{dx}f(x)}{f(x)}\end{equation}

```python
def d_g(f,h):
    df1 = df_1(f,h)
    return [(df1[j]/f[j]) for j in range(0,f.size-6)]

dg1 = d_g(TestFun,100)
np.transpose(dg1)
```

    array([0.39166667, 0.52222222, 0.62666667, 0.6962963 , 0.7372549 ,
           0.75959596, 0.77128205, 0.77726098, 0.78028534, 0.78180637,
           0.78256911, 0.78295103, 0.78314214, 0.78323772, 0.78328553,
           0.78330943, 0.78332138, 0.78332736, 0.78333035, 0.78333184,
           0.78333259, 0.78333296, 0.78333315, 0.78333324, 0.78333329,
           0.78333331, 0.78333332, 0.78333333, 0.78333333, 0.78333333,
           0.78333333, 0.78333333, 0.78333333, 0.78333333, 0.78333333,
           0.78333333, 0.78333333, 0.78333333, 0.78333333, 0.78333333,
           0.78333333, 0.78333333, 0.78333333, 0.78333333])

But since $f(x)=e^{g(x)}$ then we also know that:
\begin{equation}\nonumber\ln(f(x))=\ln(e^{g(x)})=g(x)\end{equation} so:
\begin{equation}\tag{2}\frac{d}{dx}\ln(f(x))=\frac{d}{dx}(\ln(e^{g(x)}))=\frac{d}{dx}g(x)\end{equation}

```python
def d_ln(f,h):
    return df_1(np.log(f),h)

dg2 = d_ln(TestFun,100)
np.transpose(dg2)
```

    array([0.34505258, 0.4608561 , 0.55417653, 0.61620674, 0.65251573,
           0.67224712, 0.68254505, 0.6878073 , 0.69046744, 0.69180484,
           0.6924754 , 0.69281113, 0.69297912, 0.69306314, 0.69310516,
           0.69312617, 0.69313667, 0.69314193, 0.69314455, 0.69314587,
           0.69314652, 0.69314685, 0.69314702, 0.6931471 , 0.69314714,
           0.69314716, 0.69314717, 0.69314718, 0.69314718, 0.69314718,
           0.69314718, 0.69314718, 0.69314718, 0.69314718, 0.69314718,
           0.69314718, 0.69314718, 0.69314718, 0.69314718, 0.69314718,
           0.69314718, 0.69314718, 0.69314718, 0.69314718])

since [(1)](#chain_identity) and [(2)](#log_identity) we get:

\begin{equation}\nonumber\frac{d}{dx}\ln(f(x))=\frac{\frac{d}{dx}f(x)}{f(x)}\end{equation}

```python
np.subtract(dg1,dg2)
```

    array([0.04661409, 0.06136612, 0.07249014, 0.08008956, 0.08473917,
           0.08734884, 0.088737  , 0.08945368, 0.08981791, 0.09000152,
           0.09009371, 0.0901399 , 0.09016302, 0.09017458, 0.09018037,
           0.09018326, 0.09018471, 0.09018543, 0.09018579, 0.09018597,
           0.09018606, 0.09018611, 0.09018613, 0.09018614, 0.09018615,
           0.09018615, 0.09018615, 0.09018615, 0.09018615, 0.09018615,
           0.09018615, 0.09018615, 0.09018615, 0.09018615, 0.09018615,
           0.09018615, 0.09018615, 0.09018615, 0.09018615, 0.09018615,
           0.09018615, 0.09018615, 0.09018615, 0.09018615])

Not true, there is significant difference. Lets check which solution is closer to the truth.

\begin{equation}\nonumber \frac{d}{dx}g(x) = \frac{d}{dx}(\log(2^x + 1)) \end{equation}

```python
from sympy import *
x = symbols('x')
init_printing(use_latex='mathjax')
diff(log(pow(2,x)+1), x)
```

$$\frac{2^{x} \log{\left (2 \right )}}{2^{x} + 1}$$

\begin{equation}\nonumber \frac{d}{dx}g(x) = \frac{2^x \log(2)}{2^x + 1}\end{equation}

```python
def d_direct():
    f_size = 50
    return np.asarray([(np.power(2,x)*np.log(2)/(np.power(2,x)+1)) for x in range(0,f_size-6)])
dg3 = d_direct()
np.transpose(dg3)
```

    array([0.34657359, 0.46209812, 0.55451774, 0.61613083, 0.65237382,
           0.67214272, 0.68248338, 0.68777395, 0.69045011, 0.69179602,
           0.69247094, 0.69280889, 0.692978  , 0.69306258, 0.69310488,
           0.69312603, 0.6931366 , 0.69314189, 0.69314454, 0.69314586,
           0.69314652, 0.69314685, 0.69314702, 0.6931471 , 0.69314714,
           0.69314716, 0.69314717, 0.69314718, 0.69314718, 0.69314718,
           0.69314718, 0.69314718, 0.69314718, 0.69314718, 0.69314718,
           0.69314718, 0.69314718, 0.69314718, 0.69314718, 0.69314718,
           0.69314718, 0.69314718, 0.69314718, 0.69314718])

```python
np.linalg.norm(np.subtract(dg3,dg2))
```

$$0.00200365957032477$$

When you play with exponentially growing functions, be careful with calculus. In general:

From rules of [uncertanity propagation](https://en.wikipedia.org/wiki/Propagation_of_uncertainty) we can take:

\begin{equation} f(x) = a\cdot e^{b\cdot g(x)}  \implies  \sigma_{f(x)} \approx \lvert f(x)\cdot b\cdot\sigma_{g(x)}\rvert\end{equation}

Which means the $\sigma_{f(x)}$ grows with growing $f(x)$ even if $\sigma_{g(x)}$ stabilizes at some point.

### Conclusion:
### [With great power comes great responsibility.](https://en.wikipedia.org/wiki/Uncle_Ben) [With great growth come great errors.](https://flmath.github.io/subpages/cv/my_cv.html)


# TODO:

* It is incorrect to state that the x axis of the left column is "iteration" because those are only the accepted points in both algorithms. I don't know what else to call it. The assignment states to plot the estimate of the maximum as a function of the iteration number. I'm filtering my domain, so my plot will look like it converges much faster.

  Mention this in the report, but don't worry about fixing it.

```python
%config InlineBackend.figure_format = 'svg'
%matplotlib inline

import itertools
import multiprocessing
from multiprocessing.dummy import Pool as ThreadPool
from pathlib import Path

import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
# Apparently, SNS stands for "Samuel Norman Seaborn", a fictional
# character from The West Wing
import seaborn as sns
import sympy

sns.set()
sympy.init_printing()
# Make the figures directory if it doesn't exist.
Path('figures/').mkdir(exist_ok=True)
```

```python
def f(x):
    """The function to evaluate.

    This function returns a sympy symbolic function, float, or np.ndarray
    depending on the type of the input.
    """
    # Use symbolic sine, pi if necessary.
    sin, pi = (sympy.sin, sympy.pi) if isinstance(x, sympy.Symbol) else (np.sin, np.pi)

    return 2 ** (-2 * ((x - 0.1) / 0.9)**2) * sin(5 * pi * x) ** 6
```

```python
x = np.linspace(0, 1, 200)
plt.plot(x, f(x))
plt.title('The objective function $f(x)$')
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.savefig('figures/prob1-function.pdf')
plt.show()
```

We can visually pick out the periodic extremals occuring at $\frac{n}{5} - \frac{1}{10}$, with the global optimum over $[0, 1]$ occuring at $\frac{1}{10}$.

# Symbolic and Gradient Methods

yuck

```python
x_ = sympy.Symbol('x')
f(x_)
```

```python
fp = sympy.diff(f(x_), x_)
fp
```

```python
# solve f'(x) = 0
sympy.solveset(fp, x_)
```

```python
fp = sympy.lambdify([x_], fp, 'numpy')
```

```python
plt.plot(x, fp(x))
plt.title('The derivative of $f(x)$')
plt.xlabel('$x$')
plt.ylabel('$f\'(x)$')
plt.savefig('figures/prob1-derivative.pdf')
plt.show()
```

```python
result = sp.optimize.minimize(lambda x: -f(x), 0.19, bounds=[(0, 1)])
result
```

```python
result = sp.optimize.minimize(lambda x: -f(x), 0.2, bounds=[(0, 1)])
result
```

```python
result = sp.optimize.minimize(lambda x: -f(x), 0.21, bounds=[(0, 1)])
result
```

Observations:

The (default) gradient method is just as susceptible to local solutions as the hill climbing algorithm. This makes sense.

# Hill Climbing

```python
def perturb(x, bounds, sigma):
    """Perturb the given value by adding zero-mean white noise.

    :param x: The value to perturb.
    :type x: float
    :param bounds: The lower and upper bounds on the feasible region.
    :type bounds: a (lower, upper) tuple.
    :param sigma: The standard deviation to use when adding white noise.
    """
    m, M = bounds
    xp = x + np.random.normal(scale=sigma)
    while xp >= M or xp <= m:
        xp = x + np.random.normal(scale=sigma)
    return xp
```

```python
def hill_climbing(func, bounds, sigma, iters):
    """Minimize the given function using the Hill Climbing algorithm.

    Each run generates a random initial guess.

    :param func: The function to minimize.
    :type func: a function f: float -> float
    :param bounds: The lower and upper bounds on the feasible region.
    :type bounds: a (lower, upper) tuple.
    :param sigma: The standard deviation to use when perturbing the current guess.
    :param iters: The number of iterations to run the hill climbing algorithm for.
    :returns: The path of points visited from the initial guess to the final solution.
    :rtype: An np.ndarray with shape (iters,)
    """
    path = np.zeros(iters)
    x0 = np.random.uniform(*bounds)
    for i in range(iters):
        xp = perturb(x0, bounds, sigma)
        if func(xp) < func(x0):
            x0 = xp
        path[i] = x0
    return path
```

```python
rows = 3
fig, axes = plt.subplots(rows, 2, figsize=(9, 9), sharey=True)
axes = iter(axes.flatten())
for _ in range(rows):
    path = hill_climbing(lambda x: -f(x), bounds=(0, 1), sigma=0.1, iters=100)

    ax = next(axes)

    ax.plot(path)
    ax.set_title('Hill Climbing Path')
    ax.set_xlabel('iteration')
    ax.set_ylabel('$x$')

    ax = next(axes)

    # Remove duplicate entries
    path = np.unique(path)
    ax.plot(x, f(x), label='$f(x)$')
    ax.plot(path, f(path), 'o', label='Points accepted')

    ax.set_title('Hill Climbing Path')
    ax.set_xlabel('$x$')
    ax.set_ylabel('$f(x)$')
    ax.legend()

plt.tight_layout()
plt.savefig('figures/prob1-hill-climbing-results.pdf')
plt.show()
```

```python
def iterated_hill_climbing(func, bounds, sigma, inner_iters, iters):
    """Repeatedly climb the hill to find the less-local extremum via PTSD.

    Runs the iterations in parallel with as many threads as there are processors.

    :param func: The function to minimize.
    :type func: a function f: float -> float
    :param bounds: The lower and upper bounds on the feasible region.
    :type bounds: a (lower, upper) tuple.
    :param sigma: The standard deviation to use when perturbing the current guess.
    :param inner_iters: The number of iterations to use for each run of the algorithm.
    :param iters: The number of times to run the algorithm.
    :returns: An array of solutions from each run, sorted by their fitness.
    """
    pool = ThreadPool(multiprocessing.cpu_count())
    # starmap consumes the given iterable in parallel until it is exhausted, collecting the results.
    results = pool.starmap(hill_climbing, itertools.repeat((func, bounds, sigma, inner_iters), times=iters))
    # Each result is a full path, not the optimal value
    optimums = [r[-1] for r in results]
    # Sort the optimums by their fitness.
    optimums.sort(key=func)
    return np.array(optimums)
```

```python
optimums = iterated_hill_climbing(lambda x: -f(x), bounds=(0, 1), sigma=0.1, inner_iters=50, iters=20)
solution = optimums[0]
print(solution)

plt.plot(x, f(x), label='$f(x)$')
plt.plot(solution, f(solution), 'go', label='Best solution')
plt.plot(optimums, f(optimums), '.', label='Found optimums')

plt.title('Iterated Hill Climbing Optimums')
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.legend()
plt.savefig('figures/prob1-hill-climbing-solution.pdf')
plt.show()
```

# Simulated Annealing

```python
def simulated_annealing(func, bounds, sigma, temp, cooling_factor):
    """Use simulated annealing to optimize the given function.

    :param func: The function to minimize.
    :type func: a function f: float -> float
    :param bounds: The lower and upper bounds on the feasible region.
    :type bounds: a (lower, upper) tuple.
    :param sigma: The standard deviation to use when perturbing the current guess.
    :param temp: The initial temperature of the system.
    :param cooling_factor: How quickly the system should cool.
    :returns: An array of points *accepted* by the random condition.
    :rtype: a 1D np.ndarray
    """
    # Pick a random starting point somewhere in the domain.
    x0 = np.random.uniform(*bounds)
    current = func(x0)
    path = []
    while temp > 0.001:
        xp = perturb(x0, bounds, sigma)
        potential = func(xp)
        if potential < current or np.random.random() < np.exp((current - potential) / temp):
            x0 = xp
            path.append(x0)
            current = potential
        temp *= 1 - cooling_factor

    return np.array(path)
```

```python
rows = 3
fig, axes = plt.subplots(rows, 2, figsize=(9, 9), sharey=True)
axes = iter(axes.flatten())
for _ in range(rows):
    path = simulated_annealing(lambda x: -f(x), bounds=(0, 1), sigma=0.1, temp=0.01, cooling_factor=0.001)
    solution = path[-1]

    ax = next(axes)

    ax.plot(path)
    ax.set_title('Simulated Annealing Path')
    ax.set_xlabel('iteration')
    ax.set_ylabel('$x$')

    ax = next(axes)

    ax.plot(x, f(x), label='$f(x)$')
    ax.plot(path, f(path), '.', label='Points accepted')
    ax.plot(solution, f(solution), 'o', label='Optimum')

    ax.set_title('Simulated Annealing Path')
    ax.set_xlabel('$x$')
    ax.set_ylabel('$f(x)$')
    ax.legend()

plt.tight_layout()
plt.savefig('figures/prob1-simulated-annealing-results.pdf')
plt.show()
```

```python
def iterated_simulated_annealing(func, bounds, sigma, temp, cooling_factor, iters):
    """Repeatedly run simulated annealing to improve the quality of the results.

    Runs the iterations in parallel with as many threads as there are processors.

    :param func: The function to minimize.
    :type func: a function f: float -> float
    :param bounds: The lower and upper bounds on the feasible region.
    :type bounds: a (lower, upper) tuple.
    :param sigma: The standard deviation to use when perturbing the current guess.
    :param temp: The initial temperature of the system.
    :param cooling_factor: How quickly the system should cool.
    :param iters: The number of times to run the algorithm.
    :returns: An array of solutions from each run, sorted by their fitness.
    """
    pool = ThreadPool(multiprocessing.cpu_count())
    # starmap consumes the given iterable in parallel until it is exhausted, collecting the results.
    results = pool.starmap(simulated_annealing, itertools.repeat((func, bounds, sigma, temp, cooling_factor), times=iters))
    # Each result is a full path, not the optimal value
    optimums = [r[-1] for r in results]
    # Sort the optimums by their fitness.
    optimums.sort(key=func)
    return np.array(optimums)
```

```python
optimums = iterated_simulated_annealing(lambda x: -f(x), bounds=(0, 1), sigma=0.1, temp=0.01, cooling_factor=0.001, iters=20)
solution = optimums[0]
print(solution)

plt.plot(x, f(x), label='$f(x)$')
plt.plot(solution, f(solution), 'go', label='Best solution')
plt.plot(optimums, f(optimums), '.', alpha=0.5, label='Found optimums')

plt.title('Iterated Simulated Annealing Optimums')
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.legend()
plt.savefig('figures/prob1-simulated-annealing-solutions.pdf')
plt.show()
```

Observations:

* When the simulated annealing algorithm converges, it converges closer to one of the actual peaks. The hill climbing algorithm has more noise in its convergence.
* Simulated annealing required a lot of tuning to work, and even more tuning to work well.
* The tunable parameters that affected the correctness of the solution the most were the standard deviation of the random noise and the cooling factor. The parameter that affected speed of convergence the most was the initial temperature.
* The solution clusters for the simulated annealing algorithm are much tighter than those for the hill climbing algorithm.

You want enough noise to explore, yet not so much that you jump all over the place. Temperatures ranging from 1000 to 0.002 (because 0.001 is the convergence criterion) worked well, but smaller temperatures converged much faster.