```python
from sympy import *
```

```python
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
```

```python
eq = Eq(x**2 - x + 2)
eq
```

$\displaystyle x^{2} - x + 2 = 0$

```python
solveset(eq, x)
```

$\displaystyle \left\{\frac{1}{2} - \frac{\sqrt{7} i}{2}, \frac{1}{2} + \frac{\sqrt{7} i}{2}\right\}$

```python
inte = Integral(sin(x)**6 * cos(x) ** 2, (x, pi / 3,  pi / 2))
inte
```

$\displaystyle \int\limits_{\frac{\pi}{3}}^{\frac{\pi}{2}} \sin^{6}{\left(x \right)} \cos^{2}{\left(x \right)}\, dx$

```python
inte.doit()
```

$\displaystyle \frac{9 \sqrt{3}}{2048} + \frac{5 \pi}{768}$

```python
inte2 = Integral(exp(-4 * x)* cos(3 * x), (x, 0, pi / 4))
inte2
```

$\displaystyle \int\limits_{0}^{\frac{\pi}{4}} e^{- 4 x} \cos{\left(3 x \right)}\, dx$

```python
inte2.doit()
```

$\displaystyle \frac{7 \sqrt{2}}{50 e^{\pi}} + \frac{4}{25}$

```python
inte3 = Integral(1/ (x**2 * (x - 4)), (x, 1, 3))
inte3
```

$\displaystyle \int\limits_{1}^{3} \frac{1}{x^{2} \left(x - 4\right)}\, dx$

```python
inte3.doit()
```

$\displaystyle - \frac{1}{6} - \frac{\log{\left(3 \right)}}{8}$

```python
inte4 = Integral(sqrt(1-y**2), (y, 0, x))
inte4 = Integral(inte4, (x, 0, 1))
inte4
```

$\displaystyle \int\limits_{0}^{1}\int\limits_{0}^{x} \sqrt{1 - y^{2}}\, dy\, dx$

```python
inte4.doit()
```

$\displaystyle - \frac{1}{3} + \frac{\pi}{4}$

```python
inte5 = Integral(1 / (3*sin(x) + 4*cos(x)), (x, 0, pi / 2))
inte5
```

$\displaystyle \int\limits_{0}^{\frac{\pi}{2}} \frac{1}{3 \sin{\left(x \right)} + 4 \cos{\left(x \right)}}\, dx$

```python
simplify(inte5.doit())
```

$\displaystyle \frac{\log{\left(6 \right)}}{5}$

```python
inte6 = Integral(1 / (sin(x) * cos(x) + cos(x)**2), (x, 0, pi / 4))
inte6
```

$\displaystyle \int\limits_{0}^{\frac{\pi}{4}} \frac{1}{\sin{\left(x \right)} \cos{\left(x \right)} + \cos^{2}{\left(x \right)}}\, dx$

```python
inte6.doit()
```

$\displaystyle - \log{\left(\sqrt{2} \right)} + \log{\left(-1 - \left(-1 + \sqrt{2}\right)^{2} + 2 \sqrt{2} \right)} - \log{\left(2 - \sqrt{2} \right)}$

```python
simplify(inte6.doit())
```

$\displaystyle \log{\left(\frac{2 \sqrt{2} \left(-1 + \sqrt{2}\right)}{2 - \sqrt{2}} \right)}$

```python
# numerilical
nsimplify(inte6.doit())
```

$\displaystyle \log{\left(2 \right)}$

```python
sum1 = Sum(k**-2, (k, 1, oo))
sum1
```

$\displaystyle \sum_{k=1}^{\infty} \frac{1}{k^{2}}$

```python
sum1.doit()
```

$\displaystyle \frac{\pi^{2}}{6}$

```python

```


# Analyzing real-valued functions

```
from sympy import *
init_printing()
```

```
var('x z')
```

```
f = 1 / (1 + x**2)
```

```
f.subs(x, 1)
```

```
diff(f, x)
```

```
limit(f, x, oo)
```

```
series(f, x0=0, n=9)
```

```
integrate(f, (x, -oo, oo))
```

```
integrate(f, x)
```

```
fourier_transform(f, x, z)
```


# TSSL Lab 1 - Autoregressive models

We load a few packages that are useful for solvign this lab assignment.

```python
import pandas  # Loading data / handling data frames
import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model as lm  # Used for solving linear regression problems
from sklearn.neural_network import MLPRegressor # Used for NAR model

from tssltools_lab1 import acf, acfplot # Module available in LISAM - Used for plotting ACF
```

## 1.1 Loading, plotting and detrending data

In this lab we will build autoregressive models for a data set corresponding to the Global Mean Sea Level (GMSL) over the past few decades. The data is taken from https://climate.nasa.gov/vital-signs/sea-level/ and is available on LISAM in the file `sealevel.csv`.

**Q1**: Load the data and plot the GMSL versus time. How many observations are there in total in this data set?

_Hint:_ With pandas you can use the function `pandas.read_csv` to read the csv file into a data frame. Plotting the time series can be done using `pyplot`. Note that the sea level data is stored in the 'GMSL' column and the time when each data point was recorded is stored in the column 'Year'.

**A1**:

```python
# Load GMSL data
sealevel = pandas.read_csv("sealevel.csv")

# Number of samples in the data set
n_samples = sealevel.shape[0]
print("Total number of observations in the sealevel data set : ", n_samples)

# Plot sea level data against time
plt.figure(figsize=(12,7))
plt.plot(sealevel['Year'], sealevel['GMSL'])
plt.title('GMSL vs Year')
plt.xlabel('Year')
plt.ylabel('GMSL')
```

**Q2**: The data has a clear upward trend. Before fitting an AR model to this data need to remove this trend. Explain, using one or two sentences, why this is necessary.

**A2:** Dertrending is essential to identify other potential sub-trends and patterns (ex: seasonal patterns) in the data. Removing the trend aspect from the data helps to focus on the smaller fluctuations in the data and analyze them. The main focus of AR model is to use the temporal dependencies between the data pointsat each time step for forecasting, filtering and smoothing. To achieve that, trend has to be removed and the data should attain a constant mean for better results.

**Q3** Detrend the data following these steps:
1. Fit a straight line, $\mu_t=\theta_0 + \theta_1 u_t$ to the data based on the method of least squares. Here, $u_t$ is the time point when obervation $t$ was recorded.

    _Hint:_ You can use `lm.LinearRegression().fit(...)` from scikit-learn. Note that the inputs need to be passed as a 2D array.

    Before going on to the next step, plot your fitted line and the data in one figure.

2. Subtract the fitted line from $y_t$ for the whole data series and plot the deviations from the straight line.

**From now, we will use the detrended data in all parts of the lab.**

_Note:_ The GMSL data is recorded at regular time intervals, so that $u_{t+1} - u_t =$ const. Therefore, you can just as well use $t$ directly in the linear regression function if you prefer, $\mu_t=\theta_0 + \theta_1 t$.

**A3:**

```python
# Reshape the data to pass to LinearRegression().fit()
X = sealevel['Year'].values.reshape(n_samples, 1)
y = sealevel['GMSL'].values

# Fit a linear regression model to the data
sealevel_lm = lm.LinearRegression().fit(X, y)
y_pred = sealevel_lm.predict(X)

# Plot the fitted line over the actual data
plt.figure(figsize=(12,7))
plt.plot(X, y)
plt.plot(X, y_pred, color='red')
plt.title('LinearRegression Model fitted on GMSL data')
plt.xlabel('Year')
plt.ylabel('GMSL')
```

```python
# Detrend the data by subtracting the fitted line from yt for the whole data series
y_detrend = y - y_pred

# Plot the fitted line over the data
plt.figure(figsize=(12,7))
plt.plot(X, y_detrend)
plt.title('Detrended GMSL data')
plt.xlabel('Year')
plt.ylabel('GMSL')
```

**Q4:** Split the (detrended) time series into training and validation sets. Use the values from the beginning up to the 700th time point (i.e. $y_t$ for $t=1$ to $t=700$) as your training data, and the rest of the values as your validation data. Plot the two data sets.

_Note:_ In the above, we have allowed ourselves to use all the available data (train + validation) when detrending. An alternative would be to use only the training data also when detrending the model. The latter approach is more suitable if, either:
* we view the linear detrending as part of the model choice. Perhaps we wish to compare different polynomial trend models, and evaluate their performance on the validation data, or
* we wish to use the second chunk of observations to estimate the performance of the final model on unseen data (in that case it is often referred to as "test data" instead of "validation data"), in which case we should not use these observations when fitting the model, including the detrending step.

In this laboration we consider the linear detrending as a predetermined preprocessing step and therefore allow ourselves to use the validation data when computing the linear trend.

**A4:**

```python
# Split the detrended data into train and validation sets
X_train = X[0:700]
y_train = y_detrend[0:700]

X_valid = X[700:]
y_valid = y_detrend[700:]

# Plot the two data sets
plt.figure(figsize=(12,7))
plt.plot(X_train, y_train, label='Train set')
plt.plot(X_valid, y_valid, label='Validation set')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')
```

## 1.2 Fit an autoregressive model
We will now fit an AR$(p)$ model to the training data for a given value of the model order $p$.

**Q5**: Create a function that fits an AR$(p)$ model for an arbitrary value of p. Use this function to fit a model of order $p=10$ to the training data and write out (or plot) the coefficients.

_Hint:_ Since fitting an AR model is essentially just a standard linear regression we can make use of `lm.LinearRegression().fit(...)` similarly to above. You may use the template below and simply fill in the missing code.

**A5:**

```python
def fit_ar(y, p):
    """Fits an AR(p) model. The loss function is the sum of squared errors from t=p+1 to t=n.

    :param y: array (n,), training data points
    :param p: int, AR model order
    :return theta: array (p,), learnt AR coefficients
    """

    # Number of training data points
    n = len(y)

    # Construct the regression matrix
    Phi = np.zeros(shape=(n-p, p))
    for j in range(p):
        Phi[:,j] = y[p-1-j:n-1-j]

        # Same as above but gives flipped theta
        #Phi[:,j] = y[j:n-p+j]

    # Drop the first p values from the target vector y
    yy = y[p:]  # yy = (y_{p+1}, ..., y_n)

    # Here we use fit_intercept=False since we do not want to include an intercept term in the AR model
    regr = lm.LinearRegression(fit_intercept=False)
    regr.fit(Phi,yy)

    return regr.coef_
```

```python
# Fit an AR(10) model using simple linear regression
p = 10
theta = fit_ar(y_train, p) # theta = (theta_1, theta_2, ..., theta_p)
print("The AR coefficients : \n\n", theta)
```

    The AR coefficients :

     [ 0.62156052  0.10763277  0.15104657  0.1745703  -0.02184709 -0.05955406
     -0.09578106  0.07585221 -0.11175939  0.02305208]

**Q6:** Next, write a function that computes the one-step-ahead prediction of your fitted model. 'One-step-ahead' here means that in order to predict $y_t$ at $t=t_0$, we use the actual values of $y_t$ for $t<t_0$ from the data. Use your function to compute the predictions for both *training data* and *validation data*. Plot the predictions together with the data (you can plot both training and validation data in the same figure). Also plot the *residuals*.

_Hint:_ It is enought to call the predict function once, for both training and validation data at the same time.

**A6:**

```python
def predict_ar_1step(theta, y_target):
    """Predicts the value y_t for t = p+1, ..., n, for an AR(p) model, based on the data in y_target using
    one-step-ahead prediction.

    :param theta: array (p,), AR coefficients, theta=(a1,a2,...,ap).
    :param y_target: array (n,), the data points used to compute the predictions.
    :return y_pred: array (n-p,), the one-step predictions (\hat y_{p+1}, ...., \hat y_n)
    """

    n = len(y_target)
    p = len(theta)

    # Number of steps in prediction
    m = n-p
    y_pred = np.zeros(m)

    # Flip the theta because y is reversed (looking ahead instead of looking back)
    theta = np.flip(theta.copy()) # theta = (theta_p, ..., theta_2, theta_1)
    for i in range(m):
        y_pred[i] = np.sum(theta * y_target[i:p+i]) # (y_{i+1}, y_{i+2}, ..., y_{i+p})

    return y_pred
```

```python
# Make one-step-ahead predictions from the data and compute the residuals
y = np.concatenate([y_train, y_valid])
y_pred = predict_ar_1step(theta, y)
residuals = y[p:]-y_pred

# Plot the predictions for training and validation sets
plt.figure(figsize=(12,7))
plt.plot(X_train, y_train, label='Training data')
plt.plot(X_valid, y_valid, label='Validation data')
plt.plot(X_train[p:], y_pred[0:700-p], label='Training predictions')
plt.plot(X_valid, y_pred[700-p:], label='Validation predictions')
plt.title('Predictions')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')

# Residuals plot
plt.figure(figsize=(12,7))
plt.plot(X_train[p:], residuals[0:700-p], label='Training residuals')
plt.plot(X_valid, residuals[700-p:], label='Validation residuals')
plt.title('Residuals')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')
```

**Q7:** Compute and plot the autocorrelation function (ACF) of the *residuals* only for the *validation data*. What conclusions can you draw from the ACF plot?

_Hint:_ You can use the function `acfplot` from the `tssltools` module, available on the course web page.

**A7:**

```python
# Plot the residuals for validation data
acfplot(residuals[700-p:])
```

If the AR(10) model is accurate he ACF plot of residuals should resemble the ACF plot of white gaussian noise i.e. the empirical ACF at lag h (h>0) should be asymptotically zero for large n. But, it can be seen from the graph that several ACF values for the AR(10) model deviate significantly from zero and lie outside the 95% coverage of the asymptotic distribution of ACF for white gaussian noise. So we conclude that the AR(10) model is not the best fit for given data and that more model orders need to be evaluated to get a better fit.

## 1.3 Model validation and order selection
Above we set the model order $p=10$ quite arbitrarily. In this section we will try to find an appropriate order by validation.

**Q8**: Write a loop in which AR-models of orders from $p=2$ to $p=150$ are fitted to the data above. Plot the training and validation mean-squared errors for the one-step-ahead predictions versus the model order.

Based on your results:
- What is the main difference between the changes in training error and validation error as the order increases?
- Based on these results, which model order would you suggest to use and why?

_Note:_ There is no obvious "correct answer" to the second question, but you still need to pick an order an motivate your choice!

**A8:**

```python
mse_train = list()
mse_valid = list()

for p in range(2, 151):
    # Fit an AR model of order p using simple linear regression
    theta = fit_ar(y_train, p)

    # Make one-step-ahead predictions from the data
    y = np.concatenate([y_train, y_valid])
    y_pred = predict_ar_1step(theta, y)

    # Compute the residuals
    residuals = y[p:]-y_pred

    # Compute the validation mean-squared errors
    mse_train.append(np.sum(residuals[0:700-p] ** 2) / len(residuals[0:700-p])) # ntrain-p terms
    mse_valid.append(np.sum(residuals[700-p:] ** 2) / len(residuals[700-p:])) # nvalid

# Plot the training and validation MSE for different model orders
fig,ax = plt.subplots(nrows=1, ncols=2, sharex='all', figsize=(18,7))
fig.suptitle('Training and Validation MSE')
ax[0].plot(range(2,151), mse_train)
ax[0].set(xlabel='Model Order p', ylabel='Training MSE')
ax[1].plot(range(2,151), mse_valid)
ax[1].set(xlabel='Model Order p', ylabel='Validation MSE')
```

For one-step ahead predictions, we see that the training MSE decreases consistently as the model order p increases whereas the validation MSE plot is not so smooth. We see a dip in the validation MSE for model order between p=45 and p=75 and then increases again for higher model orders which could suggest overfitting. The training MSE for the model orders between p=45 and p=75 is higher than the validation MSE. But we chose the model order p=70 as the validation MSE is low as well as the training MSE is reasonably low and the model is not too complicated.

**Q9:** Based on the chosen model order, compute the residuals of the one-step-ahead predictions on the *validation data*. Plot the autocorrelation function of the residuals. What conclusions can you draw? Compare to the ACF plot generated above for p=10.

```python
# Fit an AR(70) model using simple linear regression
p = 70
theta = fit_ar(y_train, p)

# Make one-step-ahead predictions on the validation data and compute the residuals
y = np.concatenate([y_train[700-p:], y_valid])
y_pred = predict_ar_1step(theta, y)
residuals_valid = y_valid-y_pred

# Plot the residuals for validation data
acfplot(residuals_valid)
```

The ACF plot for AR(70) model looks better than the one for AR(10) model. As can be seen from the graph now that fewer ACF values deviate from zero or lie outside the 95% coverage of white gaussian noise ACF. It is still not the best fit but is better than the simpler AR models.

## 1.4 Long-range predictions
So far we have only considered one-step-ahead predictions. However, in many practical applications it is of interest to use the model to predict further into the future. For intance, for the sea level data studied in this laboration, it is more interesting to predict the level one year from now, and not just 10 days ahead (10 days = 1 time step in this data).

**Q10**:
Write a function that simulates the value of an AR($p$) model $m$ steps into the future, conditionally on an initial sequence of data points. Specifically, given $y_{1:n}$ with $n\geq p$ the function/code should predict the values

\begin{align}
    \hat y_{t|n} &= \mathbb{E}[y_{t} | y_{1:n}], &t &=n+1,\dots,n+m.
\end{align}

Use this to predict the values for the validation data ($y_{701:997}$) conditionally on the training data ($y_{1:700}$) and plot the result.

_Hint:_ Use the pseudo-code derived at the first pen-and-paper session.

**A10:**

```python
def simulate_ar(y, theta, m):
    """Simulates an AR(p) model for m steps, with initial condition given by the last p values of y

    :param y: array (n,) with n>=p. The last p values are used to initialize the simulation.
    :param theta: array (p,). AR model parameters,
    :param m: int, number of time steps to simulate the model for.
    """

    p = len(theta)
    y_sim = np.zeros(m)
    phi = np.flip(y[-p:].copy()) # (y_{n-1}, ..., y_{n-p})^T - note that y[ntrain-1] is the last training data point

    for i in range(m):
        # Make the one-step ahead prediction
        y_sim[i] = np.dot(theta, phi) # add np.random.normal(0,1)

        # Add the prediction to last p observations and remove the
        phi = np.concatenate([y_sim[i:i+1], phi[:-1]])

    return y_sim
```

```python
# Determine the optimum model order p for m-step predictions
mse_valid = list()

for p in range(2, 151):
    # Fit an AR model of order p using simple linear regression
    theta = fit_ar(y_train, p)

    # Make m-step predictions from the data
    y_pred = simulate_ar(y_train, theta, m=len(y_valid))

    # Compute the residuals
    residuals = y_valid-y_pred

    # Compute the validation mean-squared errors
    mse_valid.append(np.sum(residuals ** 2) / len(residuals)) # nvalid

# Plot the training and validation MSE for different model orders
plt.figure(figsize=(12,7))
plt.title('Validation MSE using m-step predictions')
plt.plot(range(2,151), mse_valid)
plt.xlabel('Model Order p')
plt.ylabel('Validation MSE')
```

For m-step ahead predictions, we plot the validation MSE to re-evaluate the optimum model order p and concluded that model order p=50 is a better choice for m-step-ahead predictions.

```python
# Fit an AR(50) model using simple linear regression
p = 50
theta = fit_ar(y_train, p)

# Make m-step predictions on the validation data
m = len(y_valid)
y_valid_pred = simulate_ar(y_train, theta, m)

# Plot the validation data and its predictions
plt.figure(figsize=(12,7))
plt.plot(X_valid, y_valid, label='Validation data')
plt.plot(X_valid, y_valid_pred, label='Validation predictions')
plt.title('Validation predictions')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')
```

We see that the m-step predictions become poorer as the time grows i.e. the GMSL values predicted at initial time steps are close the actuals and the predictions too far into the future are not close to the actual values. This may be explained by the fact that the AR model makes the initial predictions using observed data whereas with growing time the far-future predictions are based on more simulated values than actual observations in the data and the quality of the predictions drops.

**Q11:** Using the same function as above, try to simulate the process for a large number of time steps (say, $m=2000$). You should see that the predicted values eventually converge to a constant prediction of zero. Is this something that you would expect to see in general? Explain the result.

**A11:**

```python
# Make m-step predictions on the validation data
m = 2000
y_pred = simulate_ar(y_train, theta, m)

# Plot the validation data and its predictions
plt.figure(figsize=(12,7))
plt.plot(y_pred, label='Validation predictions')
plt.title('Validation predictions')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')
```

Yes the predicted values eventually converged to a constant prediction of zero. The result was a bit surprising at first. But this may be explained by the absence of the random noise component from the model when used for predictions. Due to which every new prediction becomes more and more similar to its previous prediction and eventually all prdictions converge to the mean of the detrended dataset which is zero. We verified that this behaviour changes when we add the random white gaussian noise component in the prediction step in simulate_ar() method.

## 1.5 Nonlinear AR model
 In this part, we switch to a nonlinear autoregressive (NAR) model, which is based on a feedforward neural network. This means that in this model the recursive equation for making predictions is still in the form $\hat y_t=f_\theta(y_{t-1},...,y_{t-p})$, but this time $f$ is a nonlinear function learned by the neural network. Fortunately almost all of the work for implementing the neural network and training it is handled by the `scikit-learn` package with a few lines of code, and we just need to choose the right structure, and prepare the input-output data.

**Q12**: Construct a NAR($p$) model with a feedforward (MLP) network, by using the `MLPRegressor` class from `scikit-learn`. Set $p$ to the same value as you chose for the linear AR model above. Initially, you can use an MLP with a single hidden layer consisting of 10 hidden neurons.
Train it using the same training data as above and plot the one-step-ahead predictions as well as the residuals, on both the training and validation data.

_Hint:_ You will need the methods `fit` and `predict` of `MLPRegressor`. Read the user guide of `scikit-learn` for more details. Recall that a NAR model is conceptuall very similar to an AR model, so you can reuse part of the code from above.

**A12:**

```python
def simulate_nar(y_train, y_valid, p, hidden_size=(20,), act='relu', solver='adam', max_iter=200):
    """Predicts the value y_t for t = p+1, ..., n, for an NAR(p) model, based on the training data in y_train using
    one-step-ahead prediction.

    :param y_train: array (ntrain,) with n>=p. the data points used to learn the NAR model parameters and make predictions.
    :param y_valid: array (nvalid,) the data points to make predictions on
    :param p: model order p
    :return y_pred: array (n-p,), the one-step predictions (\hat y_{p+1}, ...., \hat y_n)
    """

    # Build the nonlinear autoregressive (NAR) model
    mlp = MLPRegressor(hidden_layer_sizes=hidden_size, activation=act, solver=solver, max_iter=max_iter)

    # Number of training data points
    ntrain = len(y_train)

    # Construct the regression matrix to train and fit the model
    Phi_train = np.zeros(shape=(ntrain-p, p))
    for j in range(p):
        Phi_train[:,j] = y_train[j:ntrain-p+j]

    # Drop the first p values from the target vector y
    yy_train = y_train[p:]  # yy = (y_{p+1}, ..., y_n)

    # Train the model using train data
    mlp.fit(X=Phi_train, y=yy_train)

    # Make one-step-ahead predictions on training data
    y_pred_train = mlp.predict(Phi_train)

    # Number of training data points
    nvalid = len(y_valid)

    # Take the last p observations from the training data to start predicting from the first validation data point
    y = np.concatenate([y_train[ntrain-p:], y_valid])

    # Construct the input matrix to predict on validation data
    Phi_valid = np.zeros(shape=(nvalid, p))
    for j in range(p):
        Phi_valid[:,j] = y[j:nvalid+j]

    # Make one-step-ahead predictions on training data
    y_pred_valid = mlp.predict(Phi_valid)

    # Calculate residuals for training and validation data
    residuals_train = y_train[p:] - y_pred_train
    residuals_valid = y_valid - y_pred_valid

    return y_pred_train, y_pred_valid, residuals_train, residuals_valid
```

```python
p = 50
y_pred_train, y_pred_valid, res_train, res_valid = simulate_nar(y_train, y_valid, p)

# Plot the predictions for training and validation sets
plt.figure(figsize=(12,7))
plt.plot(X_train, y_train, label='Training data')
plt.plot(X_valid, y_valid, label='Validation data')
plt.plot(X_train[p:], y_pred_train, label='Training predictions')
plt.plot(X_valid, y_pred_valid, label='Validation predictions')
plt.title('Predictions')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')

# Residuals plot
plt.figure(figsize=(12,7))
plt.plot(X_train[p:], res_train, label='Training residuals')
plt.plot(X_valid, res_valid, label='Validation residuals')
plt.title('Residuals')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')
```

**Q13:** Try to expirement with different choices for the hyperparameters of the network (e.g. number of hidden layers and units per layer, activation function, etc.) and the optimizer (e.g. `solver` and `max_iter`).

Are you satisfied with the results? Why/why not? Discuss what the limitations of this approach might be.

**A13:**

```python
# Different hyperparameters
p = 50
y_pred_train, y_pred_valid, res_train, res_valid = simulate_nar(
    y_train, y_valid, p, hidden_size=(20, 30), act='tanh', solver='adam', max_iter=200)

# Residuals plot
plt.figure(figsize=(12,7))
plt.plot(X_train[p:], res_train, label='Training residuals')
plt.plot(X_valid, res_valid, label='Validation residuals')
plt.title('Residuals')
plt.legend()
plt.xlabel('Year')
plt.ylabel('GMSL')
```

Increasing the number of hidden layers/units per layer with tanh activation improved the predictions to some extent and the residuals became smaller in magnitude. Increasing the number of layers and layer size beyond a certain point caused the NAR(p) model to overfit to the training data and perform poorly on the validation dataset. The training residuals decreased in magnitude while the validation residuals increased. Increasing the number of iterations had a similar effect of overfitting the model to the training data.

The results are not very satisfying because we see that the magnitude of the residuals in most cases are almost in the same order as that of the data which indicates that the model is not consistant in making good predictions. This could be due to the fact that even though the NAR(p) model captures the more complex temporal dependencies in the data compared to the linear AR(p) models, its receptive field is still limited to p time steps i.e. only takes into account the last p steps while making predictions.

Linear and Non Linear AR Modelling/Code.ipynb


# Linear Regression Tutorial

```python
import numpy as np
import scipy.linalg
import matplotlib.pyplot as plt
%matplotlib inline
```

We consider a linear regression problem of the form
$$
y = \boldsymbol x^T\boldsymbol\theta + \epsilon\,,\quad \epsilon \sim \mathcal N(0, \sigma^2)
$$
where $\boldsymbol x\in\mathbb{R}^D$ are inputs and $y\in\mathbb{R}$ are noisy observations. The parameter vector $\boldsymbol\theta\in\mathbb{R}^D$ parametrizes the function.

We assume we have a training set $(\boldsymbol x_n, y_n)$, $n=1,\ldots, N$. We summarize the sets of training inputs in $\mathcal X = \{\boldsymbol x_1, \ldots, \boldsymbol x_N\}$ and corresponding training targets $\mathcal Y = \{y_1, \ldots, y_N\}$, respectively.

In this tutorial, we are interested in finding good parameters $\boldsymbol\theta$.

```python
# Define training set
X = np.array([-3, -1, 0, 1, 3]).reshape(-1,1) # 5x1 vector, N=5, D=1
y = np.array([-1.2, -0.7, 0.14, 0.67, 1.67]).reshape(-1,1) # 5x1 vector

# Plot the training set
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

## 1. Maximum Likelihood
We will start with maximum likelihood estimation of the parameters $\boldsymbol\theta$. In maximum likelihood estimation, we find the parameters $\boldsymbol\theta^{\mathrm{ML}}$ that maximize the likelihood
$$
p(\mathcal Y | \mathcal X, \boldsymbol\theta) = \prod_{n=1}^N p(y_n | \boldsymbol x_n, \boldsymbol\theta)\,.
$$
From the lecture we know that the maximum likelihood estimator is given by
$$
\boldsymbol\theta^{\text{ML}} = (\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\in\mathbb{R}^D\,,
$$
where
$$
\boldsymbol X = [\boldsymbol x_1, \ldots, \boldsymbol x_N]^T\in\mathbb{R}^{N\times D}\,,\quad \boldsymbol y = [y_1, \ldots, y_N]^T \in\mathbb{R}^N\,.
$$

Let us compute the maximum likelihood estimate for a given training set

```python
## EDIT THIS FUNCTION
def max_lik_estimate(X, y):

    # X: N x D matrix of training inputs
    # y: N x 1 vector of training targets/observations
    # returns: maximum likelihood parameters (D x 1)

    N, D = X.shape
    theta_ml = np.zeros((D,1))
    theta_ml = (np.linalg.inv(X.T @ X)@ X.T)@ y ## <-- EDIT THIS LINE
    return theta_ml
```

```python
# get maximum likelihood estimate
theta_ml = max_lik_estimate(X,y)

#@test
assert np.any(theta_ml[0]), "max_lik_estimate is not implemented"
```

Now, make a prediction using the maximum likelihood estimate that we just found

```python
## EDIT THIS FUNCTION
def predict_with_estimate(Xtest, theta):

    # Xtest: K x D matrix of test inputs
    # theta: D x 1 vector of parameters
    # returns: prediction of f(Xtest); K x 1 vector

    prediction = Xtest @ theta ## <-- EDIT THIS LINE

    return prediction
```

Now, let's see whether we got something useful:

```python
# define a test set
Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest, theta_ml)

#@test
assert not np.array_equal(ml_prediction, Xtest), "predict_with_estimate not implemented"

# plot
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Questions
1. Does the solution above look reasonable?
2. Play around with different values of $\theta$. How do the corresponding functions change?
3. Modify the training targets $\mathcal Y$ and re-run your computation. What changes?

Let us now look at a different training set, where we add 2.0 to every $y$-value, and compute the maximum likelihood estimate

```python
ynew = y + 2.0

plt.figure()
plt.plot(X, ynew, '+', markersize=10)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

```python
# get maximum likelihood estimate
theta_ml = max_lik_estimate(X, ynew)
print(theta_ml)

# define a test set
Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest, theta_ml)

# plot
plt.figure()
plt.plot(X, ynew, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Question:
1. This maximum likelihood estimate doesn't look too good: The orange line is too far away from the observations although we just shifted them by 2. Why is this the case?
2. How can we fix this problem?

Let us now define a linear regression model that is slightly more flexible:
$$
y = \theta_0 + \boldsymbol x^T \boldsymbol\theta_1 + \epsilon\,,\quad \epsilon\sim\mathcal N(0,\sigma^2)
$$
Here, we added an offset (bias) parameter $\theta_0$ to our original model.

#### Question:
1. What is the effect of this bias parameter, i.e., what additional flexibility does it offer?

If we now define the inputs to be the augmented vector $\boldsymbol x_{\text{aug}} = \begin{bmatrix}1\\\boldsymbol x\end{bmatrix}$, we can write the new linear regression model as
$$
y = \boldsymbol x_{\text{aug}}^T\boldsymbol\theta_{\text{aug}} + \epsilon\,,\quad \boldsymbol\theta_{\text{aug}} = \begin{bmatrix}
\theta_0\\
\boldsymbol\theta_1
\end{bmatrix}\,.
$$

```python
N, D = X.shape
X_aug = np.hstack([np.ones((N,1)), X]) # augmented training inputs of size N x (D+1)
theta_aug = np.zeros((D+1, 1)) # new theta vector of size (D+1) x 1
```

Let us now compute the maximum likelihood estimator for this setting.
_Hint:_ If possible, re-use code that you have already written

```python
## EDIT THIS FUNCTION
def max_lik_estimate_aug(X_aug, y):

    theta_aug_ml = np.zeros((D+1,1))
    theta_aug_ml = max_lik_estimate(X_aug, y)## <-- EDIT THIS LINE

    return theta_aug_ml
```

```python
theta_aug_ml = max_lik_estimate_aug(X_aug, y)

#@test
assert np.any(theta_aug_ml), "Max_lik_estimate_aug not implemented"
```

Now, we can make predictions again:

```python
# define a test set (we also need to augment the test inputs with ones)
Xtest_aug = np.hstack([np.ones((Xtest.shape[0],1)), Xtest]) # 100 x (D + 1) vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest_aug, theta_aug_ml)

# plot
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

It seems this has solved our problem!
#### Question:
1. Play around with the first parameter of $\boldsymbol\theta_{\text{aug}}$ and see how the fit of the function changes.
2. Play around with the second parameter of $\boldsymbol\theta_{\text{aug}}$ and see how the fit of the function changes.

### Nonlinear Features
So far, we have looked at linear regression with linear features. This allowed us to fit straight lines. However, linear regression also allows us to fit functions that are nonlinear in the inputs $\boldsymbol x$, as long as the parameters $\boldsymbol\theta$ appear linearly. This means, we can learn functions of the form
$$
f(\boldsymbol x, \boldsymbol\theta) = \sum_{k = 1}^K \theta_k \phi_k(\boldsymbol x)\,,
$$
where the features $\phi_k(\boldsymbol x)$ are (possibly nonlinear) transformations of the inputs $\boldsymbol x$.

Let us have a look at an example where the observations clearly do not lie on a straight line:

```python
y = np.array([10.05, 1.5, -1.234, 0.02, 8.03]).reshape(-1,1)
plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Polynomial Regression
One class of functions that is covered by linear regression is the family of polynomials because we can write a polynomial of degree $K$ as
$$
\sum_{k=0}^K \theta_k x^k = \boldsymbol \phi(x)^T\boldsymbol\theta\,,\quad
\boldsymbol\phi(x)=
\begin{bmatrix}
x^0\\
x^1\\
\vdots\\
x^K
\end{bmatrix}\in\mathbb{R}^{K+1}\,.
$$
Here, $\boldsymbol\phi(x)$ is a nonlinear feature transformation of the inputs $x\in\mathbb{R}$.

Similar to the earlier case we can define a matrix that collects all the feature transformations of the training inputs:
$$
\boldsymbol\Phi = \begin{bmatrix}
\boldsymbol\phi(x_1) & \boldsymbol\phi(x_2) & \cdots & \boldsymbol\phi(x_n)
\end{bmatrix}^T \in\mathbb{R}^{N\times K+1}
$$

Let us start by computing the feature matrix $\boldsymbol \Phi$

```python
## EDIT THIS FUNCTION
def poly_features(X, K):

    #X: inputs of size N x 1
    #K: degree of the polynomial
    # computes the feature matrix Phi (N x (K+1))

    X = X.flatten()
    N = X.shape[0]

    #initialize Phi
    Phi = np.zeros((N, K+1))

    # Compute the feature matrix in stages
    Phi = np.zeros((N, K+1))
    for i in range(len(X)):
        for j in range(K+1):
            Phi[i,j] = X[i]**(j)  ## <-- EDIT THIS LINE
    return Phi
```

With this feature matrix we get the maximum likelihood estimator as
$$
\boldsymbol \theta^\text{ML} = (\boldsymbol\Phi^T\boldsymbol\Phi)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$
For reasons of numerical stability, we often add a small diagonal "jitter" $\kappa>0$ to $\boldsymbol\Phi^T\boldsymbol\Phi$ so that we can invert the matrix without significant problems so that the maximum likelihood estimate becomes
$$
\boldsymbol \theta^\text{ML} = (\boldsymbol\Phi^T\boldsymbol\Phi + \kappa\boldsymbol I)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$

```python
## EDIT THIS FUNCTION
def nonlinear_features_maximum_likelihood(Phi, y):
    # Phi: features matrix for training inputs. Size of N x D
    # y: training targets. Size of N by 1
    # returns: maximum likelihood estimator theta_ml. Size of D x 1

    jitter = 1e-08 # good for numerical stability

    K = Phi.shape[1]

    # maximum likelihood estimate
    theta_ml = np.zeros((K,1))
    theta_ml = np.linalg.inv(Phi.T @ Phi + jitter * np.eye(K,K)) @ (Phi.T @ y)   ## <-- EDIT THIS LINE

    return theta_ml
```

Now we have all the ingredients together: The computation of the feature matrix and the computation of the maximum likelihood estimator for polynomial regression. Let's see how this works.

To make predictions at test inputs $\boldsymbol X_{\text{test}}\in\mathbb{R}$, we need to compute the features (nonlinear transformations) $\boldsymbol\Phi_{\text{test}}= \boldsymbol\phi(\boldsymbol X_{\text{test}})$ of $\boldsymbol X_{\text{test}}$ to give us the predicted mean
$$
\mathbb{E}[\boldsymbol y_{\text{test}}] = \boldsymbol \Phi_{\text{test}}\boldsymbol\theta^{\text{ML}}
$$

```python
K = 5 # Define the degree of the polynomial we wish to fit
Phi = poly_features(X, K) # N x (K+1) feature matrix

#@Test
assert np.any(Phi), "Poly_features not implemented"

theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator

#@Test
assert np.any(theta_ml), "nonlinear_features_maximum_likelihood"

# test inputs
Xtest = np.linspace(-4,4,100).reshape(-1,1)

# feature matrix for test inputs
Phi_test = poly_features(Xtest, K)

#@Test
assert np.any(theta_ml), "nonlinear_features_maximum_likelihood"

y_pred = Phi_test @ theta_ml # predicted y-values

plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

Experiment with different polynomial degrees in the code above.
#### Questions:
1. What do you observe?
2. What is a good fit?

## Evaluating the Quality of the Model

Let us have a look at a more interesting data set

```python
def f(x):
    return np.cos(x) + 0.2*np.random.normal(size=(x.shape))

X = np.linspace(-4,4,20).reshape(-1,1)
y = f(X)

plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

Now, let us use the work from above and fit polynomials to this dataset.

```python
## EDIT THIS CELL
K = 2 # Define the degree of the polynomial we wish to fit

Phi = poly_features(X, K) # N x (K+1) feature matrix

#@Test
assert np.any(Phi), "Poly_features not implemented"

theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator

#@Test
assert np.any(theta_ml), "nonlinear_features_maximum_likelihood"

# test inputs
Xtest = np.linspace(-5,5,100).reshape(-1,1)
ytest = f(Xtest) # ground-truth y-values

# feature matrix for test inputs
Phi_test = poly_features(Xtest, K)

#@Test
assert np.any(Phi), "Poly_features not implemented"

#y_pred = Xtest*0
y_pred = Phi_test @ theta_ml# <-- EDIT THIS LINE

#@Test
assert np.any(y_pred), "Prediction not implemented"

# plot
plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred)
plt.plot(Xtest, ytest)
plt.legend(["data", "prediction", "ground truth observations"])
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Questions:
1. Try out different degrees of polynomials.
2. Based on visual inspection, what looks like the best fit?

Let us now look at a more systematic way to assess the quality of the polynomial that we are trying to fit. For this, we compute the root-mean-squared-error (RMSE) between the $y$-values predicted by our polynomial and the ground-truth $y$-values. The RMSE is then defined as
$$
\text{RMSE} = \sqrt{\frac{1}{N}\sum_{n=1}^N(y_n - y_n^\text{pred})^2}
$$
Write a function that computes the RMSE.

```python
## EDIT THIS FUNCTION
def RMSE(y, ypred):
    rmse = np.sqrt(sum((y - ypred)**2)/y.shape[0]) ## <-- EDIT THIS LINE
    return rmse
```

Now compute the RMSE for different degrees of the polynomial we want to fit.

```python
## EDIT THIS CELL
K_max = 20
rmse_train = np.zeros((K_max+1,))

for k in range(K_max+1):

    Phi = poly_features(X, k)
    theta_ml = nonlinear_features_maximum_likelihood(Phi, y)
    Phi_test = poly_features(Xtest, k)
    y_pred = Phi @ theta_ml

    rmse_train[k] = RMSE(y, y_pred) # <-- EDIT THIS LINE

    assert rmse_train[k] != -1, "RMSE not implemented"

plt.figure()
plt.plot(rmse_train)
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE");
```

#### Question:
1. What do you observe?
2. What is the best polynomial fit according to this plot?
3. Write some code that plots the function that uses the best polynomial degree (use the test set for this plot). What do you observe now?

```python
# WRITE THE PLOTTING CODE HERE
plt.figure()
plt.plot(X, y, '+')

k = 5
Phi = poly_features(X, k)
theta_ml = nonlinear_features_maximum_likelihood(Phi, y)
Phi_test = poly_features(Xtest, k)
ypred_test = Phi_test @ theta_ml

#ypred_test = Xtest*0 ## <--- EDIT THIS LINE (hint: you may require a few lines to do the computation)

#@Test
assert any(ypred_test), "Prediction not implemented"

plt.plot(Xtest, ypred_test)
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.legend(["data", "maximum likelihood fit"]);
```

The RMSE on the training data is somewhat misleading, because we are interested in the generalization performance of the model. Therefore, we are going to compute the RMSE on the test set and use this to choose a good polynomial degree.

```python
## EDIT THIS CELL
K_max = 20
rmse_train = np.zeros((K_max+1,))
rmse_test = np.zeros((K_max+1,))

for k in range(K_max+1):

    # feature matrix
    Phi = poly_features(X, k) ## <--- EDIT THIS LINE

    # maximum likelihood estimate
    theta_ml = nonlinear_features_maximum_likelihood(Phi, y) ## <--- EDIT THIS LINE

    # predict y-values of training set
    ypred_train = Phi @ theta_ml ## <--- EDIT THIS LINE

    # RMSE on training set
    rmse_train[k] = RMSE(y, ypred_train) ## <--- EDIT THIS LINE

    # feature matrix for test inputs
    Phi_test = poly_features(Xtest, k) ## <--- EDIT THIS LINE

    # prediction (test set)
    ypred_test = Phi_test @ theta_ml ## <--- EDIT THIS LINE

    # RMSE on test set
    rmse_test[k] = RMSE(ytest, ypred_test) ## <--- EDIT THIS LINE

plt.figure()
plt.semilogy(rmse_train) # this plots the RMSE on a logarithmic scale
plt.semilogy(rmse_test) # this plots the RMSE on a logarithmic scale
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE")
plt.legend(["training set", "test set"]);
```

#### Questions:
1. What do you observe now?
2. Why does the RMSE for the test set not always go down?
3. Which polynomial degree would you choose now?
4. Plot the fit for the "best" polynomial degree.

```python
# WRITE THE PLOTTING CODE HERE
plt.figure()
plt.plot(X, y, '+')

k = 5
Phi = poly_features(X, k)
theta_ml = nonlinear_features_maximum_likelihood(Phi, y)
Phi_test = poly_features(Xtest, k)
ypred_test = Phi_test @ theta_ml

#ypred_test = Xtest*0 ## <--- EDIT THIS LINE (hint: you may require a few lines to do the computation)

#@Test
assert any(ypred_test), "Prediction not implemented"

plt.plot(Xtest, ypred_test)
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.legend(["data", "maximum likelihood fit"]);
```

#### Question
If you did not have a designated test set, what could you do to estimate the generalization error (purely using the training set)?

## 2. Maximum A Posteriori Estimation

We are still considering the model
$$
y = \boldsymbol\phi(\boldsymbol x)^T\boldsymbol\theta + \epsilon\,,\quad \epsilon\sim\mathcal N(0,\sigma^2)\,.
$$
We assume that the noise variance $\sigma^2$ is known.

Instead of maximizing the likelihood, we can look at the maximum of the posterior distribution on the parameters $\boldsymbol\theta$, which is given as
$$
p(\boldsymbol\theta|\mathcal X, \mathcal Y) = \frac{\overbrace{p(\mathcal Y|\mathcal X, \boldsymbol\theta)}^{\text{likelihood}}\overbrace{p(\boldsymbol\theta)}^{\text{prior}}}{\underbrace{p(\mathcal Y|\mathcal X)}_{\text{evidence}}}
$$
The purpose of the parameter prior $p(\boldsymbol\theta)$ is to discourage the parameters to attain extreme values, a sign that the model overfits. The prior allows us to specify a "reasonable" range of parameter values. Typically, we choose a Gaussian prior $\mathcal N(\boldsymbol 0, \alpha^2\boldsymbol I)$, centered at $\boldsymbol 0$ with variance $\alpha^2$ along each parameter dimension.

The MAP estimate of the parameters is
$$
\boldsymbol\theta^{\text{MAP}} = (\boldsymbol\Phi^T\boldsymbol\Phi + \frac{\sigma^2}{\alpha^2}\boldsymbol I)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$
where $\sigma^2$ is the variance of the noise.

```python
## EDIT THIS FUNCTION
def map_estimate_poly(Phi, y, K, sigma, alpha):
    # Phi: training inputs, Size of N x (K+1)
    # y: training targets, Size of D x 1
    # K: degree of the polynomial
    # sigma: standard deviation of the noise
    # alpha: standard deviation of the prior on the parameters
    # returns: MAP estimate theta_map, Size of (K+1) x 1

    theta_map = np.zeros((K+1,1)) ## <-- EDIT THIS LINE

    return theta_map
```

```python
# define the function we wish to estimate later
def g(x, sigma):
    p = np.hstack([x**0, x**1, np.sin(x)])
    w = np.array([-1.0, 0.1, 1.0]).reshape(-1,1)
    return p @ w + sigma*np.random.normal(size=x.shape)
```

```python
# Generate some data
sigma = 1.0 # noise standard deviation
alpha = 1.0 # standard deviation of the parameter prior
N = 20

np.random.seed(42)

X = (np.random.rand(N)*10.0 - 5.0).reshape(-1,1)
y = g(X, sigma) # training targets

plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

```python
# get the MAP estimate
K = 8 # polynomial degree

# feature matrix
Phi = poly_features(X, K)

theta_map = map_estimate_poly(Phi, y, K, sigma, alpha)

#@Test
assert any(theta_map), "map_estimate_poly not implemented"

# maximum likelihood estimate
theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

Xtest = np.linspace(-5,5,100).reshape(-1,1)
ytest = g(Xtest, sigma)

Phi_test = poly_features(Xtest, K)
y_pred_map = Phi_test @ theta_map

y_pred_mle = Phi_test @ theta_ml

plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred_map)
plt.plot(Xtest, g(Xtest, 0))
plt.plot(Xtest, y_pred_mle)

plt.legend(["data", "map prediction", "ground truth function", "maximum likelihood"]);
```

```python
print(np.hstack([theta_ml, theta_map]))
```

Now, let us compute the RMSE for different polynomial degrees and see whether the MAP estimate addresses the overfitting issue we encountered with the maximum likelihood estimate.

```python
## EDIT THIS CELL

K_max = 12 # this is the maximum degree of polynomial we will consider
assert(K_max < N) # this is the latest point when we'll run into numerical problems

rmse_mle = np.zeros((K_max+1,))
rmse_map = np.zeros((K_max+1,))

for k in range(K_max+1):

    rmse_mle[k] = -1 ## Compute the maximum likelihood estimator, compute the test-set predicitons, compute the RMSE
    rmse_map[k] = -1 ## Compute the MAP estimator, compute the test-set predicitons, compute the RMSE

    #@Test
    assert rmse_mle[k] != -1, "mle prediction not implemented"
    assert rmse_map[k] != -1, "map prediction not implemented"

plt.figure()
plt.semilogy(rmse_mle) # this plots the RMSE on a logarithmic scale
plt.semilogy(rmse_map) # this plots the RMSE on a logarithmic scale
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE")
plt.legend(["Maximum likelihood", "MAP"])
```

#### Questions:
1. What do you observe?
2. What is the influence of the prior variance on the parameters ($\alpha^2$)? Change the parameter and describe what happens.

## 3. Bayesian Linear Regression

```python
# Test inputs
Ntest = 200
Xtest = np.linspace(-5, 5, Ntest).reshape(-1,1) # test inputs

prior_var = 2.0 # variance of the parameter prior (alpha^2). We assume this is known.
noise_var = 1.0 # noise variance (sigma^2). We assume this is known.

pol_deg = 3 # degree of the polynomial we consider at the moment
```

Assume a parameter prior $p(\boldsymbol\theta) = \mathcal N (\boldsymbol 0, \alpha^2\boldsymbol I)$. For every test input $\boldsymbol x_*$ we obtain the
prior mean
$$
E[f(\boldsymbol x_*)] = 0
$$
and the prior (marginal) variance (ignoring the noise contribution)
$$
V[f(\boldsymbol x_*)] = \alpha^2\boldsymbol\phi(\boldsymbol x_*) \boldsymbol\phi(\boldsymbol x_*)^\top
$$
where $\boldsymbol\phi(\cdot)$ is the feature map.

```python
## EDIT THIS CELL

# compute the feature matrix for the test inputs
Phi_test = np.zeros((Ntest, pol_deg+1))  # N x (pol_deg+1) feature matrix <--- EDIT THIS LINE
raise NotImplementedError

# compute the (marginal) prior at the test input locations
# prior mean
prior_mean = np.ones((Ntest,1))  # prior mean at test inputs (size: (Ntest,1)) <-- EDIT THIS LINE
raise NotImplementedError

# prior variance
full_covariance = np.zeros((Ntest, Ntest)) # N x N covariance matrix of all function values <-- EDIT THIS LINE
prior_marginal_var = 0 # marginal of size (N, )
raise NotImplementedError

# Let us visualize the prior over functions
plt.figure()
plt.plot(Xtest, prior_mean, color="k")

conf_bound1 = np.sqrt(prior_marginal_var).flatten()
conf_bound2 = 2.0*np.sqrt(prior_marginal_var).flatten()
conf_bound3 = 2.0*np.sqrt(prior_marginal_var).flatten() + 2.0*np.sqrt(noise_var)
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound1,
             prior_mean.flatten() - conf_bound1, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound2,
                 prior_mean.flatten() - conf_bound2, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound3,
                 prior_mean.flatten() - conf_bound3, alpha = 0.1, color="k")

plt.xlabel('$x$')
plt.ylabel('$y$')
plt.title("Prior over functions");
```

Now, we will use this prior distribution and sample functions from it.

```python
## EDIT THIS CELL

# samples from the prior
num_samples = 10

# We first need to generate random weights theta_i, which we sample from the parameter prior
random_weights = np.random.normal(size=(pol_deg+1,num_samples), scale=np.sqrt(prior_var))

# Now, we compute the induced random functions, evaluated at the test input locations
# Every function sample is given as f_i = Phi * theta_i,
# where theta_i is a sample from the parameter prior

sample_function = np.zeros((Ntest,)) # <-- EDIT THIS LINE
raise NotImplementedError

plt.figure()
plt.plot(Xtest, sample_function, color="r")
plt.title("Plausible functions under the prior")
print("Every sampled function is a polynomial of degree "+str(pol_deg));
```

Now we are given some training inputs $\boldsymbol x_1, \dotsc, \boldsymbol x_N$, which we collect in a matrix $\boldsymbol X = [\boldsymbol x_1, \dotsc, \boldsymbol x_N]^\top\in\mathbb{R}^{N\times D}$

```python
N = 10
X = np.random.uniform(high=5, low=-5, size=(N,1)) # training inputs, size Nx1
y = g(X, np.sqrt(noise_var)) # training targets, size Nx1
```

Now, let us compute the posterior

```python
## EDIT THIS FUNCTION

def polyfit(X, y, K, prior_var, noise_var):
    # X: training inputs, size NxD
    # y: training targets, size Nx1
    # K: degree of polynomial we consider
    # prior_var: prior variance of the parameter distribution
    # sigma: noise variance

    jitter = 1e-08

    Phi = poly_features(X, K) # N x (K+1) feature matrix

    # Compute maximum likelihood estimate
    theta_ml = np.zeros((K+1,1)) # <-- EDIT THIS LINE

    # MAP estimate
    theta_map = np.zeros((K+1,1)) # <-- EDIT THIS LINE

    # Parameter posterior
    SN = np.zeros(K+1) # covariance matrix of the parameter posterior # <-- EDIT THIS LINE
    mN = np.zeros((K+1,1)) # mean vector of the parameter posterior   # <-- EDIT THIS LINE

    raise NotImplementedError

    return (theta_ml, theta_map, mN, SN)
```

```python
theta_ml, theta_map, theta_mean, theta_var = polyfit(X, y, pol_deg, alpha, sigma)
```

Now, let's make predictions (ignoring the measurement noise). We obtain three predictors:
\begin{align}
&\text{Maximum likelihood: }E[f(\boldsymbol X_{\text{test}})] = \boldsymbol \phi(X_{\text{test}})\boldsymbol \theta_{ml}\\
&\text{Maximum a posteriori: } E[f(\boldsymbol X_{\text{test}})] = \boldsymbol \phi(X_{\text{test}})\boldsymbol \theta_{map}\\
&\text{Bayesian: } p(f(\boldsymbol X_{\text{test}})) = \mathcal N(f(\boldsymbol X_{\text{test}}) \,|\, \boldsymbol \phi(X_{\text{test}}) \boldsymbol\theta_{\text{mean}},\, \boldsymbol\phi(X_{\text{test}}) \boldsymbol\theta_{\text{var}}  \boldsymbol\phi(X_{\text{test}})^\top)
\end{align}
We already computed all quantities. Write some code that implements all three predictors.

```python
## EDIT THIS CELL

# predictions (ignoring the measurement/observations noise)

# maximum likelihood predictions (just the mean)
m_mle_test = np.zeros((Ntest,1)) # <-- EDIT THIS LINE

# MAP predictions (just the mean)
m_map_test = np.zeros((Ntest,1)) # <-- EDIT THIS LINE

# predictive distribution (Bayesian linear regression)
# mean prediction
mean_blr = np.zeros((Ntest,1)) # <-- EDIT THIS LINE
# variance prediction
cov_blr =  np.ones((Ntest,Ntest)) # <-- EDIT THIS LINE

raise NotImplementedError
```

```python
# plot the posterior
plt.figure()
plt.plot(X, y, "+")
plt.plot(Xtest, m_mle_test)
plt.plot(Xtest, m_map_test)
var_blr = np.diag(cov_blr)
conf_bound1 = np.sqrt(var_blr).flatten()
conf_bound2 = 2.0*np.sqrt(var_blr).flatten()
conf_bound3 = 2.0*np.sqrt(var_blr).flatten() + 2.0*sigma

plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound1,
                 mean_blr.flatten() - conf_bound1, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound2,
                 mean_blr.flatten() - conf_bound2, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound3,
                 mean_blr.flatten() - conf_bound3, alpha = 0.1, color="k")
plt.legend(["Training data", "MLE", "MAP", "BLR"])
plt.xlabel('$x$');
plt.ylabel('$y$');
```

```python
li = np.array([[1],[2],[3],[4]])
li
```

    array([[1],
           [2],
           [3],
           [4]])

```python
li.shape
```

    (4, 1)

```python
li.flatten()
```

    array([1, 2, 3, 4])

```python
K = 5
Phi = np.zeros((len(li), K+1))
for i in range(len(li)):
    for j in range(K+1):
        Phi[i,j] = li[i]**(j)

```

```python
np.eye(3,3)
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 1.]])

```python
Phi.shape
```

    (4, 6)

```python

```


# Working with numerical features

We have to prepare our data to work with ML algorithms. In the case of numerical values we have some methods that we should apply before we start working with ML algorithms. Some of those methosts are:

    Imputation
    Handling Outliers
    Feature Scaling
    Feature Transformation
    Binning
    Log Transform

We saw how to work with outliers, and null values, and the techniques for imputation of NaN values. In this lesson we are going to focus on scaling, bining and log transformation.

We discussed previously that the scale of the features is an important consideration when building machine learning models. Briefly:
Feature magnitude matters because:

    The regression coefficients of linear models are directly influenced by the scale of the variable.
    Variables with bigger magnitude / larger value range dominate over those with smaller magnitude / value range
    Gradient descent converges faster when features are on similar scales
    Feature scaling helps decrease the time to find support vectors for SVMs
    Euclidean distances are sensitive to feature magnitude.
    Some algorithms, like PCA require the features to be centered at 0.

The machine learning models affected by the feature scale are:

    Linear and Logistic Regression
    Neural Networks
    Support Vector Machines
    KNN
    K-means clustering
    Linear Discriminant Analysis (LDA)
    Principal Component Analysis (PCA)

## Feature Scaling

Feature scaling refers to the methods or techniques used to normalize the range of independent variables in our data, or in other words, the methods to set the feature value range within a similar scale. Feature scaling is generally the last step in the data preprocessing pipeline, performed just before training the machine learning algorithms.

## Feature Scaling: Z-Score Standardization and Min-Max Scaling

- [About standardization](#About-standardization)
- [About Min-Max scaling / "normalization"](#About-Min-Max-scaling-normalization)
- [Standardization or Min-Max scaling?](#Standardization-or-Min-Max-scaling?)
- [Standardizing and normalizing - how it can be done using scikit-learn](#Standardizing-and-normalizing---how-it-can-be-done-using-scikit-learn)
- [Bottom-up approaches](#Bottom-up-approaches)
- [The effect of standardization on PCA in a pattern classification task](#The-effect-of-standardization-on-PCA-in-a-pattern-classification-task)

### About standardization

The result of **standardization** (or **Z-score normalization**) is that the features will be rescaled so that they'll have the properties of a standard normal distribution with

$\mu = 0$ and $\sigma = 1$

where $\mu$ is the mean (average) and $\sigma$ is the standard deviation from the mean; standard scores (also called ***z*** scores) of the samples are calculated as follows:

$z = \frac{x - \mu}{\sigma}$

Standardizing the features so that they are centered around 0 with a standard deviation of 1 is not only important if we are comparing measurements that have different units, but it is also a general requirement for many machine learning algorithms. Intuitively, we can think of gradient descent as a prominent example
(an optimization algorithm often used in logistic regression, SVMs, perceptrons, neural networks etc.); with features being on different scales, certain weights may update faster than others since the feature values x_j  play a role in the weight updates

$\Delta w_j = - \eta \frac{\partial J}{\partial w_j} = \eta \sum_i (t^{(i)} - o^{(i)})x^{(i)}_{j}$,

so that

$w_j := w_j + \Delta w_j$

where $\eta$ is the learning rate, $t$ the target class label, and $o$ the actual output.
Other intuitive examples include K-Nearest Neighbor algorithms and clustering algorithms that use, for example, Euclidean distance measures -- in fact, tree-based classifier are probably the only classifiers where feature scaling doesn't make a difference.

To quote from the [`scikit-learn`](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) documentation:

*"Standardization of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual feature do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance)."*

### About Min-Max scaling

[[back to top](#Sections)]

An alternative approach to Z-score normalization (or standardization) is the so-called **Min-Max scaling** (often also simply called "normalization" - a common cause for ambiguities).
In this approach, the data is scaled to a fixed range - usually 0 to 1.
The cost of having this bounded range - in contrast to standardization - is that we will end up with smaller standard deviations, which can suppress the effect of outliers.

A Min-Max scaling is typically done via the following equation:

\begin{equation} X_{norm} = \frac{X - X_{min}}{X_{max}-X_{min}} \end{equation}

### Z-score standardization or Min-Max scaling?

[[back to top](#Sections)]

*"Standardization or Min-Max scaling?"* - There is no obvious answer to this question: it really depends on the application.

For example, in clustering analyses, standardization may be especially crucial in order to compare similarities between features based on certain distance measures. Another prominent example is the Principal Component Analysis, where we usually prefer standardization over Min-Max scaling, since we are interested in the components that maximize the variance (depending on the question and if the PCA computes the components via the correlation matrix instead of the covariance matrix; [but more about PCA in my previous article](http://sebastianraschka.com/Articles/2014_pca_step_by_step.html)).

However, this doesn't mean that Min-Max scaling is not useful at all! A popular application is image processing, where pixel intensities have to be normalized to fit within a certain range (i.e., 0 to 255 for the RGB color range). Also, typical neural network algorithm require data that on a 0-1 scale.

## Standardizing and normalizing - how it can be done using scikit-learn

[[back to top](#Sections)]

Of course, we could make use of NumPy's vectorization capabilities to calculate the z-scores for standardization and to normalize the data using the equations that were mentioned in the previous sections. However, there is an even more convenient approach using the preprocessing module from one of Python's open-source machine learning library [scikit-learn](http://scikit-learn.org ).

For the following examples and discussion, we will have a look at the free "Wine" Dataset that is deposited on the UCI machine learning repository
(http://archive.ics.uci.edu/ml/datasets/Wine).

**Reference:**
Forina, M. et al, PARVUS - An Extendible Package for Data
Exploration, Classification and Correlation. Institute of Pharmaceutical
and Food Analysis and Technologies, Via Brigata Salerno,
16147 Genoa, Italy.

Bache, K. & Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

The Wine dataset consists of 3 different classes where each row correspond to a particular wine sample.

The class labels (1, 2, 3) are listed in the first column, and the columns 2-14 correspond to 13 different attributes (features):

1) Alcohol
2) Malic acid
...

#### Loading the wine dataset

```python

import pandas as pd
import numpy as np

df = pd.io.parsers.read_csv(
    'https://raw.githubusercontent.com/rasbt/pattern_classification/master/data/wine_data.csv',
     header=None,
     usecols=[0,1,2]
    )

df.columns=['Class label', 'Alcohol', 'Malic acid']

df.head()

```

      Class label
      Alcohol
      Malic acid

      0
      1
      14.23
      1.71

      1
      1
      13.20
      1.78

      2
      1
      13.16
      2.36

      3
      1
      14.37
      1.95

      4
      1
      13.24
      2.59

As we can see in the table above, the features **Alcohol** (percent/volumne) and **Malic acid** (g/l) are measured on different scales, so that ***Feature Scaling*** is necessary important prior to any comparison or combination of these data.

#### Standardization and Min-Max scaling

```python
from sklearn import preprocessing

std_scale = preprocessing.StandardScaler().fit(df[['Alcohol', 'Malic acid']])
np_std = std_scale.transform(df[['Alcohol', 'Malic acid']]) # estandarizado

minmax_scale = preprocessing.MinMaxScaler().fit(df[['Alcohol', 'Malic acid']])
np_minmax = minmax_scale.transform(df[['Alcohol', 'Malic acid']]) # escalado

```

```python
std_scale.mean_
```

    array([13.00061798,  2.33634831])

```python
type(std_scale)
```

    sklearn.preprocessing._data.StandardScaler

```python
type(np_std)
```

    numpy.ndarray

```python
type(minmax_scale)
```

    sklearn.preprocessing._data.MinMaxScaler

```python
type(np_minmax)
```

    numpy.ndarray

```python
np_std[0:5, :]
```

    array([[ 1.51861254, -0.5622498 ],
           [ 0.24628963, -0.49941338],
           [ 0.19687903,  0.02123125],
           [ 1.69154964, -0.34681064],
           [ 0.29570023,  0.22769377]])

```python
np_minmax[0:5, :]
```

    array([[0.84210526, 0.1916996 ],
           [0.57105263, 0.2055336 ],
           [0.56052632, 0.3201581 ],
           [0.87894737, 0.23913043],
           [0.58157895, 0.36561265]])

```python

print('Mean after standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(np_std[:,0].mean() , np_std[:,1].mean()))
print('\nStandard deviation after standardization:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format( np_std[:,0].std() , np_std[:,1].std()))

```

    Mean after standardization:
    Alcohol=-0.00, Malic acid=-0.00

    Standard deviation after standardization:
    Alcohol=1.00, Malic acid=1.00

```python

print('Min-value after min-max scaling:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(np_minmax[:,0].min(), np_minmax[:,1].min()))
print('\nMax-value after min-max scaling:\nAlcohol={:.2f}, Malic acid={:.2f}'
      .format(np_minmax[:,0].max(), np_minmax[:,1].max()))

```

    Min-value after min-max scaling:
    Alcohol=0.00, Malic acid=0.00

    Max-value after min-max scaling:
    Alcohol=1.00, Malic acid=1.00

#### Plotting

```python
%matplotlib inline
```

```python

from matplotlib import pyplot as plt

def plot():
    plt.figure(figsize=(8,6))

    plt.scatter(df['Alcohol'], df['Malic acid'],
            color='green', label='input scale', alpha=0.5)

    plt.scatter(np_std[:,0], np_std[:,1], color='red',
            label='Standardized [$N  (\mu=0, \; \sigma=1)$]', alpha=0.3)

    plt.scatter(np_minmax[:,0], np_minmax[:,1],
            color='blue', label='min-max scaled [min=0, max=1]', alpha=0.3)

    plt.title('Alcohol and Malic Acid content of the wine dataset')
    plt.xlabel('Alcohol')
    plt.ylabel('Malic Acid')
    plt.legend(loc='upper left')
    plt.grid() # cuadrícula

    plt.tight_layout() # te ajusta para que se vea bien

plot() #¡la acabo de definir yo!

```

The plot above includes the wine datapoints on all three different scales: the input scale where the alcohol content was measured in volume-percent (green), the standardized features (red), and the normalized features (blue).
In the following plot, we will zoom in into the three different axis-scales.

Recordatorio de numpy

```python
# en numpy un array UNIDIMENSIONAL tiene de dimensiones
np.array([0,1,2,3,4]).shape

# 5 filas y NINGUNA columna
```

    (5,)

```python
np.array([0,1,2,3,4])
```

    array([0, 1, 2, 3, 4])

```python
# un numpy un array 2D tiene de dimensiones
np.array([[0,1,2,3,4], [5,6,7,8,9]])

```

    array([[0, 1, 2, 3, 4],
           [5, 6, 7, 8, 9]])

```python
np.array([[0,1,2,3,4], [5,6,7,8,9]]).shape
```

    (2, 5)

```python
# ¡Importante!
# No es lo mismo un array UNIDIMENSIONAL que uno en 2D con solo una fila o solo una columna
```

```python
np.array([[0,1,2,3,4]]).shape
```

    (1, 5)

```python
np.array([[0,1,2,3,4]])
```

    array([[0, 1, 2, 3, 4]])

```python
np.array([[0],[1],[2],[3],[4]]).shape
```

    (5, 1)

```python
np.array([[0],[1],[2],[3],[4]])
```

    array([[0],
           [1],
           [2],
           [3],
           [4]])

```python
# un array UNIDIMENSIONAL transpuesto se queda igual
np.array([0,1,2,3,4]).T == np.array([0,1,2,3,4])
```

    array([ True,  True,  True,  True,  True])

```python
np.array([0,1,2,3,4])
```

    array([0, 1, 2, 3, 4])

```python
np.array([0,1,2,3,4]).T
```

    array([0, 1, 2, 3, 4])

```python
# un array 2D con una fila o una columna sí se transpone
```

```python
np.array([[0,1,2,3,4]]).T
```

    array([[0],
           [1],
           [2],
           [3],
           [4]])

```python
np.array([[0], [1], [2], [3], [4]]).T
```

    array([[0, 1, 2, 3, 4]])

```python
# veamos el método zip de Python
# zip itera sobre tuplas y crea tuplas nuevas

for x in zip(('azul', 'rojo', 'verde'), ('perro', 'gato')):
    print(x)
```

    ('azul', 'perro')
    ('rojo', 'gato')

```python
# si zip tiene una sola tupla de argumento
for x in zip(('azul', 'rojo', 'verde')):
    print(x)

```

    ('azul',)
    ('rojo',)
    ('verde',)

```python

```

Seguimos con el tratamiento de datos numéricos

```python

fig, ax = plt.subplots(3, figsize=(6,14))

for a,d,l in zip(range(len(ax)),
               (df[['Alcohol', 'Malic acid']].values, np_std, np_minmax), # values devuelve los valores en tipo numpy
               ('Input scale',
                'Standardized [$N  (\mu=0, \; \sigma=1)$]',
                'min-max scaled [min=0, max=1]')
                ):
    # a es 0, 1, 2
    # d es df[['Alcohol', 'Malic acid']].values, np_std, np_minmax
    # l es 'Input scale', 'Standardized [$N  (\mu=0, \; \sigma=1)$]', 'min-max scaled [min=0, max=1]'

    for i,c in zip(range(1,4), ('red', 'blue', 'green')):
        ax[a].scatter(d[df['Class label'].values == i, 0],
                  d[df['Class label'].values == i, 1],
                  alpha=0.5,
                  color=c,
                  label='Class %s' %i
                  )
        # i es 1, 2, 3
        # c es 'red', 'blue', 'green'

    ax[a].set_title(l)
    ax[a].set_xlabel('Alcohol')
    ax[a].set_ylabel('Malic Acid')
    ax[a].legend(loc='upper left')
    ax[a].grid()

plt.tight_layout()

```

```python
# aplicar values a un DataFrame
df_ejemplo = pd.DataFrame({'col1': [1,2,3], 'col2': [4,5,6]})
df_ejemplo.values
# values devuelve un DataFrame en formato numpy
```

    array([[1, 4],
           [2, 5],
           [3, 6]], dtype=int64)

## Bottom-up approaches

Of course, we can also code the equations for standardization and 0-1 Min-Max scaling "manually". However, the scikit-learn methods are still useful if you are working with test and training data sets and want to scale them equally.

E.g.,

std_scale = preprocessing.StandardScaler().fit(X_train)
X_train = std_scale.transform(X_train)
X_test = std_scale.transform(X_test)

Below, we will perform the calculations using "pure" Python code, and an more convenient NumPy solution, which is especially useful if we attempt to transform a whole matrix.

Just to recall the equations that we are using:

Standardization: \begin{equation} z = \frac{x - \mu}{\sigma} \end{equation}

with mean:

\begin{equation}\mu = \frac{1}{N} \sum_{i=1}^N (x_i)\end{equation}

and standard deviation:

\begin{equation}\sigma = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \mu)^2}\end{equation}

Min-Max scaling: \begin{equation} X_{norm} = \frac{X - X_{min}}{X_{max}-X_{min}} \end{equation}

```python
# lógica del bucle for

suelto = ['caracter. ' + p for p in 'hola']
print(suelto)
```

    ['caracter. h', 'caracter. o', 'caracter. l', 'caracter. a']

### Pure Python

```python

# Standardization
x = [1,4,5,6,6,2,3]
mean = sum(x)/len(x)
std_dev = (1/len(x) * sum([ (x_i - mean) ** 2 for x_i in x]))**0.5
z_scores = [(x_i - mean)/std_dev for x_i in x]

# Min-Max scaling

minmax = [(x_i - min(x))/ (max(x) - min(x)) for x_i in x]

```

### NumPy

```python
import numpy as np

# Standardization
x_np = np.asarray(x) # as array lo convierte en un array de numpy
z_scores_np = (x_np - x_np.mean()) / x_np.std()

# Min-Max scaling
np_minmax = (x_np - x_np.min()) / (x_np.max() - x_np.min())

```

### Visualization

Just to make sure that our code works correctly, let us plot the results via matplotlib.

```python
%matplotlib inline
```

```python

from matplotlib import pyplot as plt

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(nrows=2, ncols=2, figsize=(10,5)) #devuelve fig, ax con 4 subplots

y_pos = [0 for i in range(len(x))]

ax1.scatter(z_scores, y_pos, color='g')
ax1.set_title('Python standardization', color='g')

ax2.scatter(minmax, y_pos, color='g')
ax2.set_title('Python Min-Max scaling', color='g')

ax3.scatter(z_scores_np, y_pos, color='b')
ax3.set_title('Python NumPy standardization', color='b')

ax4.scatter(np_minmax, y_pos, color='b')
ax4.set_title('Python NumPy Min-Max scaling', color='b')

plt.tight_layout() # para ajustar la visualización

for ax in (ax1, ax2, ax3, ax4):
    ax.get_yaxis().set_visible(False)
    ax.grid()

plt.show()

```

```python
# vemos lo mismo, más fácil usando librerías
# estandarizado con valores con media cero y desviación típica 1
# escalado con valores entre 0 y 1
```

## The effect of standardization on PCA in a pattern classification task

[[back to top](#Sections)]

Earlier, I mentioned the Principal Component Analysis (PCA) as an example where standardization is crucial, since it is "analyzing" the variances of the different features.
Now, let us see how the standardization affects PCA and a following supervised classification on the whole wine dataset.

In the following section, we will go through the following steps:

- Reading in the dataset
- Dividing the dataset into a separate training and test dataset
- Standardization of the features
- Principal Component Analysis (PCA) to reduce the dimensionality
- Training a naive Bayes classifier
- Evaluating the classification accuracy with and without standardization

### Reading in the dataset

[[back to top](#Sections)]

```python

import pandas as pd

df = pd.io.parsers.read_csv(
    'https://raw.githubusercontent.com/rasbt/pattern_classification/master/data/wine_data.csv',
    header=None,
    )

```

### Dividing the dataset into a separate training and test dataset

[[back to top](#Sections)]

In this step, we will randomly divide the wine dataset into a training dataset and a test dataset where the training dataset will contain 70% of the samples and the test dataset will contain 30%, respectively.

```python
df.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13

      0
      1
      14.23
      1.71
      2.43
      15.6
      127
      2.80
      3.06
      0.28
      2.29
      5.64
      1.04
      3.92
      1065

      1
      1
      13.20
      1.78
      2.14
      11.2
      100
      2.65
      2.76
      0.26
      1.28
      4.38
      1.05
      3.40
      1050

      2
      1
      13.16
      2.36
      2.67
      18.6
      101
      2.80
      3.24
      0.30
      2.81
      5.68
      1.03
      3.17
      1185

      3
      1
      14.37
      1.95
      2.50
      16.8
      113
      3.85
      3.49
      0.24
      2.18
      7.80
      0.86
      3.45
      1480

      4
      1
      13.24
      2.59
      2.87
      21.0
      118
      2.80
      2.69
      0.39
      1.82
      4.32
      1.04
      2.93
      735

```python
from sklearn.model_selection import train_test_split

X_wine = df.values[:,1:]
y_wine = df.values[:,0]

X_train, X_test, y_train, y_test = train_test_split(X_wine, y_wine,
    test_size=0.30, random_state=12345)
```

```python

```

### Feature Scaling - Standardization

[[back to top](#Sections)]

```python
from sklearn import preprocessing

std_scale = preprocessing.StandardScaler().fit(X_train)
X_train_std = std_scale.transform(X_train)
X_test_std = std_scale.transform(X_test)
```

### Dimensionality reduction via Principal Component Analysis (PCA)

[[back to top](#Sections)]

Now, we perform a PCA on the standardized and the non-standardized datasets to transform the dataset onto a 2-dimensional feature subspace.
In a real application, a procedure like cross-validation would be done in order to find out what choice of features would yield a optimal balance between "preserving information" and "overfitting" for different classifiers. However, we will omit this step since we don't want to train a perfect classifier here, but merely compare the effects of standardization.

```python
from sklearn.decomposition import PCA

# on non-standardized data
pca = PCA(n_components=2).fit(X_train)
X_train = pca.transform(X_train)
X_test = pca.transform(X_test)

# on standardized data
pca_std = PCA(n_components = 2).fit(X_train_std)
X_train_std = pca_std.transform(X_train_std)
X_test_std = pca_std.transform(X_test_std)

```

Let us quickly visualize how our new feature subspace looks like (note that class labels are not considered in a PCA - in contrast to a Linear Discriminant Analysis - but I will add them in the plot for clarity).

```python
%matplotlib inline
```

```python

from matplotlib import pyplot as plt

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10,4))

for l,c,m in zip(range(1,4), ('blue', 'red', 'green'), ('^', 's', 'o')):
    ax1.scatter(X_train[y_train==l, 0], X_train[y_train==l, 1],
        color=c,
        label='class %s' %l,
        alpha=0.5,
        marker=m
        )

for l,c,m in zip(range(1,4), ('blue', 'red', 'green'), ('^', 's', 'o')):
    ax2.scatter(X_train_std[y_train==l, 0], X_train_std[y_train==l, 1],
        color=c,
        label='class %s' %l,
        alpha=0.5,
        marker=m
        )

ax1.set_title('Transformed NON-standardized training dataset after PCA')
ax2.set_title('Transformed standardized training dataset after PCA')

for ax in (ax1, ax2):

    ax.set_xlabel('1st principal component')
    ax.set_ylabel('2nd principal component')
    ax.legend(loc='upper right')
    ax.grid()

plt.tight_layout()

plt.show()

```

### Training a naive Bayes classifier

[[back to top](#Sections)]

We will use a naive Bayes classifier for the classification task. If you are not familiar with it, the term "naive" comes from the assumption that all features are "independent".
All in all, it is a simple but robust classifier based on Bayes' rule

Bayes' Rule:

\begin{equation} P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)} \end{equation}

where

- ω:  class label
- *P(ω|x)*: the posterior probability
- *p(x|ω)*: prior probability (or likelihood)

and the **decsion rule:**

Decide $ \omega_1 $ if $ P(\omega_1|x) > P(\omega_2|x) $ else decide $ \omega_2 $.

\begin{equation}
\Rightarrow \frac{p(x|\omega_1) * P(\omega_1)}{p(x)} > \frac{p(x|\omega_2) * P(\omega_2)}{p(x)}
\end{equation}

I don't want to get into more detail about Bayes' rule in this article, but if you are interested in a more detailed collection of examples, please have a look at the [Statistical Patter Classification](https://github.com/rasbt/pattern_classification#statistical-pattern-recognition-examples) in my pattern classification repository.

```python
from sklearn.naive_bayes import GaussianNB

# on non-standardized data
gnb = GaussianNB()
fit = gnb.fit(X_train, y_train)

# on standardized data
gnb_std = GaussianNB()
fit_std = gnb_std.fit(X_train_std, y_train)
```

### Evaluating the classification accuracy with and without standardization

[[back to top](#Sections)]

```python

from sklearn import metrics

pred_train = gnb.predict(X_train)

print('\nPrediction accuracy for the training dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_train, pred_train)))

pred_test = gnb.predict(X_test)

print('\nPrediction accuracy for the test dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_test, pred_test)))

```

    Prediction accuracy for the training dataset
    81.45%

    Prediction accuracy for the test dataset
    64.81%

```python

pred_train_std = gnb_std.predict(X_train_std)

print('\nPrediction accuracy for the training dataset')
print('{:.2%}'.format(metrics.accuracy_score(y_train, pred_train_std)))

pred_test_std = gnb_std.predict(X_test_std)

print('\nPrediction accuracy for the test dataset')
print('{:.2%}\n'.format(metrics.accuracy_score(y_test, pred_test_std)))

```

    Prediction accuracy for the training dataset
    96.77%

    Prediction accuracy for the test dataset
    98.15%

As we can see, the standardization prior to the PCA definitely led to an decrease in the empirical error rate on classifying samples from test dataset.

# Feature transformations

### Normalization and changing distribution

Monotonic feature transformation is critical for some algorithms and has no effect on others. This is one of the reasons for the increased popularity of decision trees and all its derivative algorithms (random forest, gradient boosting). Not everyone can or want to tinker with transformations, and these algorithms are robust to unusual distributions.

There are also purely engineering reasons: `np.log` is a way of dealing with large numbers that do not fit in `np.float64`. This is an exception rather than a rule; often it's driven by the desire to adapt the dataset to the requirements of the algorithm. Parametric methods usually require a minimum of symmetric and unimodal distribution of data, which is not always given in real data. There may be more stringent requirements; recall [our earlier article about linear models](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-4-linear-classification-and-regression-44a41b9b5220).

However, data requirements are imposed not only by parametric methods; [K nearest neighbors](https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-3-classification-decision-trees-and-k-nearest-neighbors-8613c6b6d2cd) will predict complete nonsense if features are not normalized e.g. when one distribution is located in the vicinity of zero and does not go beyond (-1, 1) while the other’s range is on the order of hundreds of thousands.

A simple example: suppose that the task is to predict the cost of an apartment from two variables — the distance from city center and the number of rooms. The number of rooms rarely exceeds 5 whereas the distance from city center can easily be in the thousands of meters.

The simplest transformation is Standard Scaling (or Z-score normalization):

$$ \large z= \frac{x-\mu}{\sigma} $$

Note that Standard Scaling does not make the distribution normal in the strict sense.

```python
from sklearn.preprocessing import StandardScaler
from scipy.stats import beta # beta es un tipo de ditribución
from scipy.stats import shapiro
# shapiro es un test que te dice cómo es de probable
# que tus datos sean gaussianos
import numpy as np

data = beta(1,10).rvs(1000).reshape(-1,1)
shapiro(data)
```

    ShapiroResult(statistic=0.8493459820747375, pvalue=8.4536979627521e-30)

```python
# Value of the statistic, p-value
shapiro(StandardScaler().fit_transform(data))

# we reject H0, they are not Gaussian
```

    ShapiroResult(statistic=0.8825300931930542, pvalue=7.698277099526105e-27)

Hipótesis nula: $\sf{H_{0}}$
Es la hipótesis por defecto. En este caso: los datos vienen de una Gaussiana.
El p-valor es un estadístico que nos dice si es probable que la hipótesis nula sea cierta o sea falsa.
Si p-valor <= 0.05 entonces rechazo $\sf{H_{0}}$

But, to some extent, it protects against outliers:

```python
prueba = np.array([1,2,3,4,5]).reshape(-1,1)
prueba
```

    array([[1],
           [2],
           [3],
           [4],
           [5]])

```python
np.array([1,2,3,4,5]).shape
```

    (5,)

```python
prueba.shape
```

    (5, 1)

```python
np.array([1,2,3,4,5,6]).reshape(3,2)
```

    array([[1, 2],
           [3, 4],
           [5, 6]])

```python
np.array([1,2,3,4,5,6]).reshape(-3,2)
# poner un menos es decirle "no lo sé, hazlo tú"
```

    array([[1, 2],
           [3, 4],
           [5, 6]])

```python
np.array([1,2,3,4,5,6]).reshape(-7,2)
```

    array([[1, 2],
           [3, 4],
           [5, 6]])

```python
np.array([1,2,3,4,5,6]).reshape(1,6).reshape(2,-1)
```

    array([[1, 2, 3],
           [4, 5, 6]])

```python
data = np.array([1,1,0,-1,2,1,2,3,-2,4,100]).reshape(-1,1).astype(np.float64)
StandardScaler().fit_transform(data)
```

    array([[-0.31922662],
           [-0.31922662],
           [-0.35434155],
           [-0.38945648],
           [-0.28411169],
           [-0.31922662],
           [-0.28411169],
           [-0.24899676],
           [-0.42457141],
           [-0.21388184],
           [ 3.15715128]])

```python
(data - data.mean())/data.std()
```

    array([[-0.31922662],
           [-0.31922662],
           [-0.35434155],
           [-0.38945648],
           [-0.28411169],
           [-0.31922662],
           [-0.28411169],
           [-0.24899676],
           [-0.42457141],
           [-0.21388184],
           [ 3.15715128]])

Another fairly popular option is MinMax Scaling, which brings all the points within a predetermined interval (typically (0, 1)).

$$ \large X_{norm}=\frac{X-X_{min}}{X_{max}-X_{min}} $$

```python
from sklearn.preprocessing import MinMaxScaler

MinMaxScaler().fit_transform(data)
```

    array([[0.02941176],
           [0.02941176],
           [0.01960784],
           [0.00980392],
           [0.03921569],
           [0.02941176],
           [0.03921569],
           [0.04901961],
           [0.        ],
           [0.05882353],
           [1.        ]])

```python
(data - data.min()) / (data.max() - data.min())
```

    array([[0.02941176],
           [0.02941176],
           [0.01960784],
           [0.00980392],
           [0.03921569],
           [0.02941176],
           [0.03921569],
           [0.04901961],
           [0.        ],
           [0.05882353],
           [1.        ]])

StandardScaling and MinMax Scaling have similar applications and are often more or less interchangeable. However, if the algorithm involves the calculation of distances between points or vectors, the default choice is StandardScaling. But MinMax Scaling is useful for visualization by bringing features within the interval (0, 255).

If we assume that some data is not normally distributed but is described by the [log-normal distribution](https://en.wikipedia.org/wiki/Log-normal_distribution), it can easily be transformed to a normal distribution:

```python
from scipy.stats import lognorm

data = lognorm(s=1).rvs(1000) # no le estoy pasando la semilla
shapiro(data)
```

    ShapiroResult(statistic=0.6178099513053894, pvalue=2.5223372357846707e-42)

```python
shapiro(np.log(data))
```

    ShapiroResult(statistic=0.9986675977706909, pvalue=0.6666808724403381)

The lognormal distribution is suitable for describing salaries, price of securities, urban population, number of comments on articles on the internet, etc. However, to apply this procedure, the underlying distribution does not necessarily have to be lognormal; you can try to apply this transformation to any distribution with a heavy right tail. Furthermore, one can try to use other similar transformations, formulating their own hypotheses on how to approximate the available distribution to a normal. Examples of such transformations are [Box-Cox transformation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.boxcox.html) (logarithm is a special case of the Box-Cox transformation) or [Yeo-Johnson transformation](https://gist.github.com/mesgarpour/f24769cd186e2db853957b10ff6b7a95) (extends the range of applicability to negative numbers). In addition, you can also try adding a constant to the feature — `np.log (x + const)`.

# Binning

```python
import pandas as pd

fcc_survey_df = pd.read_csv('ficheros/fcc_2016_coder_survey_subset.csv', encoding='utf-8', sep=',')
fcc_survey_df[['ID.x', 'EmploymentField', 'Age', 'Income']].head()
```

      ID.x
      EmploymentField
      Age
      Income

      0
      cef35615d61b202f1dc794ef2746df14
      office and administrative support
      28.0
      32000.0

      1
      323e5a113644d18185c743c241407754
      food and beverage
      22.0
      15000.0

      2
      b29a1027e5cd062e654a63764157461d
      finance
      19.0
      48000.0

      3
      04a11e4bcb573a1261eb0d9948d32637
      arts, entertainment, sports, or media
      26.0
      43000.0

      4
      9368291c93d5d5f5c8cdb1a575e18bec
      education
      20.0
      6000.0

## Fixed-width binning

### Developer age distribution

```python
fig, ax = plt.subplots()
fcc_survey_df['Age'].hist(color='#C5A9D3')
ax.set_title('Developer Age Histogram', fontsize = 12)
ax.set_xlabel('Age', fontsize=12)
ax.set_ylabel('Frequency', fontsize=12)
```

### Binning based on rounding

```
Age Range: Bin
---------------
 0 -  9  : 0
10 - 19  : 1
20 - 29  : 2
30 - 39  : 3
40 - 49  : 4
50 - 59  : 5
60 - 69  : 6
  ... and so on
```

```python
#fcc_survey_df['Age_bin_round'] = np.array(np.floor(np.array(fcc_survey_df['Age']) / 10.))
#fcc_survey_df[['ID.x', 'Age', 'Age_bin_round']].iloc[1071:1076]
```

      ID.x
      Age
      Age_bin_round

      1071
      6a02aa4618c99fdb3e24de522a099431
      17.0
      1.0

      1072
      f0e5e47278c5f248fe861c5f7214c07a
      38.0
      3.0

      1073
      6e14f6d0779b7e424fa3fdd9e4bd3bf9
      21.0
      2.0

      1074
      c2654c07dc929cdf3dad4d1aec4ffbb3
      53.0
      5.0

      1075
      f07449fc9339b2e57703ec7886232523
      35.0
      3.0

```python
fcc_survey_df['Age_bin_round'] = (np.floor((fcc_survey_df['Age']) / 10.))
```

```python
fcc_survey_df['Age_bin_round']
```

    0        2.0
    1        2.0
    2        1.0
    3        2.0
    4        2.0
            ...
    15615    3.0
    15616    2.0
    15617    3.0
    15618    2.0
    15619    2.0
    Name: Age_bin_round, Length: 15620, dtype: float64

### Binning based on custom ranges

```
Age Range : Bin
---------------
 0 -  15  : 1
16 -  30  : 2
31 -  45  : 3
46 -  60  : 4
61 -  75  : 5
75 - 100  : 6
```

```python
# cut crea bins

bin_ranges = [0,15,30,45,60,75, 100]
bin_names = [1, 2, 3, 4, 5, 6]
fcc_survey_df['Age_bin_custom_range'] = pd.cut(fcc_survey_df['Age'],
                                              bins=bin_ranges)
fcc_survey_df['Age_bin_custom_label'] = pd.cut(fcc_survey_df['Age'], bins=bin_ranges, labels = bin_names)
fcc_survey_df[['ID.x', 'Age', 'Age_bin_round',
              'Age_bin_custom_range', 'Age_bin_custom_label']].iloc[1071:1076]
```

      ID.x
      Age
      Age_bin_round
      Age_bin_custom_range
      Age_bin_custom_label

      1071
      6a02aa4618c99fdb3e24de522a099431
      17.0
      1.0
      (15, 30]
      2

      1072
      f0e5e47278c5f248fe861c5f7214c07a
      38.0
      3.0
      (30, 45]
      3

      1073
      6e14f6d0779b7e424fa3fdd9e4bd3bf9
      21.0
      2.0
      (15, 30]
      2

      1074
      c2654c07dc929cdf3dad4d1aec4ffbb3
      53.0
      5.0
      (45, 60]
      4

      1075
      f07449fc9339b2e57703ec7886232523
      35.0
      3.0
      (30, 45]
      3

## Quantile based binning

```python
fcc_survey_df[['ID.x', 'Age', 'Income']].iloc[4:9]
```

      ID.x
      Age
      Income

      4
      9368291c93d5d5f5c8cdb1a575e18bec
      20.0
      6000.0

      5
      dd0e77eab9270e4b67c19b0d6bbf621b
      34.0
      40000.0

      6
      7599c0aa0419b59fd11ffede98a3665d
      23.0
      32000.0

      7
      6dff182db452487f07a47596f314bddc
      35.0
      40000.0

      8
      9dc233f8ed1c6eb2432672ab4bb39249
      33.0
      80000.0

```python
fig, ax = plt.subplots()
fcc_survey_df['Income'].hist(bins=30, color='#A9C5D3')
ax.set_title('Developer Income Histogram', fontsize=12)
ax.set_xlabel('Developer Income', fontsize=12)
ax.set_ylabel('Frequency', fontsize=12)
```

```python
quantile_list = [0, .25, .5, .75, 1.]
quantiles = fcc_survey_df['Income'].quantile(quantile_list)
quantiles
```

    0.00      6000.0
    0.25     20000.0
    0.50     37000.0
    0.75     60000.0
    1.00    200000.0
    Name: Income, dtype: float64

```python
fig, ax = plt.subplots()
fcc_survey_df['Income'].hist(bins=30, color='#A9C5D3')

for quantile in quantiles:
    qvl = plt.axvline(quantile, color='r')
ax.legend([qvl], ['Quantiles'], fontsize=10)

ax.set_title('Developer Income Histogram with quantiles', fontsize=12)
ax.set_xlabel('Developer Income', fontsize=12)
ax.set_ylabel('Frequency', fontsize=12)
```

```python
quantile_labels = ['0-25Q', '25-50Q', '50-75Q', '75-100Q']
fcc_survey_df['Income_quantile_range'] = pd.qcut(fcc_survey_df['Income'],
                                                q=quantile_list)
fcc_survey_df['Income_quantile_label'] = pd.qcut(fcc_survey_df['Income'],
                                                q=quantile_list,
                                                labels=quantile_labels)
fcc_survey_df[['ID.x', 'Age', 'Income',
              'Income_quantile_range', 'Income_quantile_label']].iloc[4:9]

```

      ID.x
      Age
      Income
      Income_quantile_range
      Income_quantile_label

      4
      9368291c93d5d5f5c8cdb1a575e18bec
      20.0
      6000.0
      (5999.999, 20000.0]
      0-25Q

      5
      dd0e77eab9270e4b67c19b0d6bbf621b
      34.0
      40000.0
      (37000.0, 60000.0]
      50-75Q

      6
      7599c0aa0419b59fd11ffede98a3665d
      23.0
      32000.0
      (20000.0, 37000.0]
      25-50Q

      7
      6dff182db452487f07a47596f314bddc
      35.0
      40000.0
      (37000.0, 60000.0]
      50-75Q

      8
      9dc233f8ed1c6eb2432672ab4bb39249
      33.0
      80000.0
      (60000.0, 200000.0]
      75-100Q

# Mathematical Transformations

## Log transform

```python
fcc_survey_df['Income_log'] = np.log((1+ fcc_survey_df['Income']))
fcc_survey_df[['ID.x', 'Age', 'Income', 'Income_log']].iloc[4:9]
```

      ID.x
      Age
      Income
      Income_log

      4
      9368291c93d5d5f5c8cdb1a575e18bec
      20.0
      6000.0
      8.699681

      5
      dd0e77eab9270e4b67c19b0d6bbf621b
      34.0
      40000.0
      10.596660

      6
      7599c0aa0419b59fd11ffede98a3665d
      23.0
      32000.0
      10.373522

      7
      6dff182db452487f07a47596f314bddc
      35.0
      40000.0
      10.596660

      8
      9dc233f8ed1c6eb2432672ab4bb39249
      33.0
      80000.0
      11.289794

```python
income_log_mean = np.round(np.mean(fcc_survey_df['Income_log']), 2)
fig, ax = plt.subplots()
fcc_survey_df['Income_log'].hist(bins=30, color='#A9C5D3')
plt.axvline(income_log_mean, color='r')
ax.set_title('Developer Income Histogram after Log Transform',
            fontsize=12)
ax.set_xlabel('Developer Income (log scale)', fontsize=12)
ax.set_ylabel('Frequency', fontsize=12)
ax.text(11.5, 450, r'$\mu$='+str(income_log_mean), fontsize=10)
```

## Box–Cox transform

```python
from scipy import stats

# get optimal lambda value from non null income values
income = np.array(fcc_survey_df['Income'])
income_clean = income[~np.isnan(income)] # los no nan
l, opt_lambda = stats.boxcox(income_clean)
print('Optimal lambda value:', opt_lambda)
```

    Optimal lambda value: 0.11799122497648248

```python
fcc_survey_df['Income_boxcox_lambda_0'] = stats.boxcox((1+fcc_survey_df['Income']),
                                                         lmbda=0)
fcc_survey_df['Income_boxcox_lambda_opt'] = stats.boxcox(fcc_survey_df['Income'],
                                                           lmbda=opt_lambda)
fcc_survey_df[['ID.x', 'Age', 'Income', 'Income_log',
               'Income_boxcox_lambda_0', 'Income_boxcox_lambda_opt']].iloc[4:9]
```

      ID.x
      Age
      Income
      Income_log
      Income_boxcox_lambda_0
      Income_boxcox_lambda_opt

      4
      9368291c93d5d5f5c8cdb1a575e18bec
      20.0
      6000.0
      8.699681
      8.699681
      15.180667

      5
      dd0e77eab9270e4b67c19b0d6bbf621b
      34.0
      40000.0
      10.596660
      10.596660
      21.115340

      6
      7599c0aa0419b59fd11ffede98a3665d
      23.0
      32000.0
      10.373522
      10.373522
      20.346418

      7
      6dff182db452487f07a47596f314bddc
      35.0
      40000.0
      10.596660
      10.596660
      21.115340

      8
      9dc233f8ed1c6eb2432672ab4bb39249
      33.0
      80000.0
      11.289794
      11.289794
      23.637128

```python
income_boxcox_mean = np.round(np.mean(fcc_survey_df['Income_boxcox_lambda_opt']), 2)

fig, ax = plt.subplots()
fcc_survey_df['Income_boxcox_lambda_opt'].hist(bins=30, color='#A9C5D3')
plt.axvline(income_boxcox_mean, color='r')
ax.set_title('Developer Income Histogram after Box–Cox Transform', fontsize=12)
ax.set_xlabel('Developer Income (Box–Cox transform)', fontsize=12)
ax.set_ylabel('Frequency', fontsize=12)
ax.text(24, 450, r'$\mu$='+str(income_boxcox_mean), fontsize=10)
```

```python

```

4-Machine_Learning/Feature Engineering/Numericas/Practica/1_Numerical_Features - solucion.ipynb