# TSSL Lab 1 - Autoregressive models

We load a few packages that are useful for solvign this lab assignment.

```python
import pandas as pd # Loading data / handling data frames
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
from sklearn import linear_model as lm  # Used for solving linear regression problems
from sklearn.neural_network import MLPRegressor # Used for NAR model

from tssltools_lab1 import acf, acfplot # Module available in LISAM - Used for plotting ACF
```

## 1.1 Loading, plotting and detrending data

In this lab we will build autoregressive models for a data set corresponding to the Global Mean Sea Level (GMSL) over the past few decades. The data is taken from https://climate.nasa.gov/vital-signs/sea-level/ and is available on LISAM in the file `sealevel.csv`.

**Q1**: Load the data and plot the GMSL versus time. How many observations are there in total in this data set?

_Hint:_ With pandas you can use the function `pandas.read_csv` to read the csv file into a data frame. Plotting the time series can be done using `pyplot`. Note that the sea level data is stored in the 'GMSL' column and the time when each data point was recorded is stored in the column 'Year'.

**A1**:

```python
seaLevelData = pd.read_csv("sealevel.csv")
```

```python
#plot of GMSL vs Time

plt.plot(seaLevelData['Year'] ,  seaLevelData['GMSL'])
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.suptitle("GMSL VS Time")
plt.show()
```

```python
#total number of observation

nObs = seaLevelData.shape[0]

print(f"Total number of obervation in given data is {nObs}")
```

    Total number of obervation in given data is 997

**Q2**: The data has a clear upward trend. Before fitting an AR model to this data need to remove this trend. Explain, using one or two sentences, why this is necessary.

**A2:** Since given data have a clear upward trend which introduces changing mean over time. Now to apply the AR model we need data with constant mean. Thus we have to detrend the data so stationary dataset can achieve and use for AR modeling and another benefit is that it enables discovering other aspects of the data like the autocorrelations and dependencies of the variables.

**Q3** Detrend the data following these steps:
1. Fit a straight line, $\mu_t=\theta_0 + \theta_1 u_t $ to the data based on the method of least squares. Here, $u_t$ is the time point when obervation $t$ was recorded.

    _Hint:_ You can use `lm.LinearRegression().fit(...)` from scikit-learn. Note that the inputs need to be passed as a 2D array.

    Before going on to the next step, plot your fitted line and the data in one figure.

2. Subtract the fitted line from $y_t$ for the whole data series and plot the deviations from the straight line.

**From now, we will use the detrended data in all parts of the lab.**

_Note:_ The GMSL data is recorded at regular time intervals, so that $u_{t+1} - u_t = $ const. Therefore, you can just as well use $t$ directly in the linear regression function if you prefer, $\mu_t=\theta_0 + \theta_1 t $.

**A3:**

```python
x = seaLevelData['Year'].to_numpy().reshape((-1 ,1))
y = seaLevelData['GMSL'].to_numpy()

model_lm = lm.LinearRegression().fit(x , y)

```

```python
theta0 = model_lm.intercept_
theta1 = model_lm.coef_
#Y_Pred = theta0 + theta1 * x
Y_Pred = model_lm.predict(x)

plt.plot(seaLevelData['Year'] ,  seaLevelData['GMSL'] , label = "Original Data")
plt.plot(x, Y_Pred, color='red' , label = "Fitted Line")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()
```

```python
detrededData = y - Y_Pred

plt.plot(x, Y_Pred, color='red' , label = "Fitted Line")
plt.plot(x, detrededData , color='blue' , label = "Detrended Data")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.suptitle("Detrended Data GMSL Vs Time")
plt.legend()
plt.show()
```

**Q4:** Split the (detrended) time series into training and validation sets. Use the values from the beginning up to the 700th time point (i.e. $y_t$ for $t=1$ to $t=700$) as your training data, and the rest of the values as your validation data. Plot the two data sets.

_Note:_ In the above, we have allowed ourselves to use all the available data (train + validation) when detrending. An alternative would be to use only the training data also when detrending the model. The latter approach is more suitable if, either:
* we view the linear detrending as part of the model choice. Perhaps we wish to compare different polynomial trend models, and evaluate their performance on the validation data, or
* we wish to use the second chunk of observations to estimate the performance of the final model on unseen data (in that case it is often referred to as "test data" instead of "validation data"), in which case we should not use these observations when fitting the model, including the detrending step.

In this laboration we consider the linear detrending as a predetermined preprocessing step and therefore allow ourselves to use the validation data when computing the linear trend.

**A4:**

```python
trainDataY = detrededData[0:700 , ]
ValidDataY = detrededData[700: , ]

trainDataX = seaLevelData['Year'][0:700]
ValidDataX = seaLevelData['Year'][700:]
```

```python
plt.plot(trainDataX, trainDataY, color='blue')
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.suptitle("Training Data")
plt.show()

plt.plot(ValidDataX, ValidDataY, color='blue')
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.suptitle("Validation Data")
plt.show()

```

## 1.2 Fit an autoregressive model
We will now fit an AR$(p)$ model to the training data for a given value of the model order $p$.

**Q5**: Create a function that fits an AR$(p)$ model for an arbitrary value of p. Use this function to fit a model of order $p=10$ to the training data and write out (or plot) the coefficients.

_Hint:_ Since fitting an AR model is essentially just a standard linear regression we can make use of `lm.LinearRegression().fit(...)` similarly to above. You may use the template below and simply fill in the missing code.

**A5:**

```python
def fit_ar(y, p):
    """
    Fits an AR(p) model. The loss function is the sum of squared errors from t=p+1 to t=n.

    :param y: array (n,), training data points
    :param p: int, AR model order
    :return theta: array (p,), learnt AR coefficients
    """

    # Number of training data points
    n = y.shape[0]

    # Construct the regression matrix
    Phi = np.zeros((n-p,p), dtype= int)

    for j in range(p):
        Phi[:,j] = y[j:n+j-p]

    # Drop the first p values from the target vector y
    yy = y[p:]  # yy = (y_{t+p+1}, ..., y_n)

    # Here we use fit_intercept=False since we do not want to include an intercept term in the AR model
    regr = lm.LinearRegression(fit_intercept=False)
    regr.fit(Phi,yy)

    return regr.coef_
```

```python
#Calculating theta_hat from training data
#set p = 10 as AR(10) model and detrained train data as training data
theta_hat = fit_ar( y = trainDataY , p = 10 )
plt.plot(range(len(theta_hat)), theta_hat, color='blue')
plt.ylabel("Theta")
plt.xlabel("Index")
plt.suptitle("Theta Coef")
plt.show()
```

**Q6:** Next, write a function that computes the one-step-ahead prediction of your fitted model. 'One-step-ahead' here means that in order to predict $y_t$ at $t=t_0$, we use the actual values of $y_t$ for $t<t_0$ from the data. Use your function to compute the predictions for both *training data* and *validation data*. Plot the predictions together with the data (you can plot both training and validation data in the same figure). Also plot the *residuals*.

_Hint:_ It is enought to call the predict function once, for both training and validation data at the same time.

**A6:**

```python
def predict_ar_1step(theta, y_target):
    """Predicts the value y_t for t = p+1, ..., n, for an AR(p) model, based on the data in y_target using
    one-step-ahead prediction.

    :param theta: array (p,), AR coefficients, theta=(a1,a2,...,ap).
    :param y_target: array (n,), the data points used to compute the predictions.
    :return y_pred: array (n-p,), the one-step predictions (\hat y_{p+1}, ...., \hat y_n)
    """

    n = len(y_target)
    p = len(theta)

    # Number of steps in prediction
    m = n-p
    y_pred = np.zeros(m)

    for i in range(m):
      # if i == 0 :
      #   prevY = y_target[i:p+i]
      # #print(i)
      # #print(prevY[i:p+i])
      # y_pred[i] = np.flip(prevY[i:p+i]).dot(theta)
      # #print(y_pred[i])
      # prevY = np.append(prevY , y_pred[i])
      # #print(prevY[i:p+i])

      y_pred[i] = y_target[i:p+i].dot(theta)
    return y_pred
```

```python
YPred_Train = predict_ar_1step(theta = theta_hat , y_target = trainDataY)
YPred_Valid = predict_ar_1step(theta = theta_hat , y_target = ValidDataY)

```

```python
#plot for predicted values for train and validation data
plt.plot(trainDataX , trainDataY , label = "Training Data")
plt.plot(trainDataX[len(theta_hat):,] , YPred_Train , label = "Prediction for Training Data")
plt.plot(ValidDataX , ValidDataY , label = "Validation Data")
plt.plot(ValidDataX[len(theta_hat):,] , YPred_Valid , label = "Prediction for Validation Data")
#figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()

```

**Q7:** Compute and plot the autocorrelation function (ACF) of the *residuals* only for the *validation data*. What conclusions can you draw from the ACF plot?

_Hint:_ You can use the function `acfplot` from the `tssltools` module, available on the course web page.

**A7:**

```python
help(acfplot)
```

    Help on function acfplot in module tssltools_lab1:

    acfplot(x, lags=None, conf=0.95)
        Plots the empirical autocorralation function.

        :param x: array (n,), sequence of data points
        :param lags: int, maximum lag to compute the ACF for. If None, this is set to n-1. Default is None.
        :param conf: float, number in the interval [0,1] which specifies the confidence level (based on a central limit
                     theorem under a white noise assumption) for two dashed lines drawn in the plot. Default is 0.95.
        :return:

The plot indicates that the sign of the auto-correlation changes depending on the value of the coefficients. Moreover, we can see that the decay in the ACF is relatively slow, meaning the process takes a long to forget its past.

```python
resiValidation = ValidDataY[10:] - YPred_Valid

AutoCoR = acfplot(x = resiValidation)

```

## 1.3 Model validation and order selection
Above we set the model order $p=10$ quite arbitrarily. In this section we will try to find an appropriate order by validation.

**Q8**: Write a loop in which AR-models of orders from $p=2$ to $p=150$ are fitted to the data above. Plot the training and validation mean-squared errors for the one-step-ahead predictions versus the model order.

Based on your results:
- What is the main difference between the changes in training error and validation error as the order increases?
- Based on these results, which model order would you suggest to use and why?

_Note:_ There is no obvious "correct answer" to the second question, but you still need to pick an order an motivate your choice!

```python
MSE_Train = []
MSE_Valid = []

# def ModelCompare(y,p):
#   theta_hat = fit_ar(y, p)
#   predY = predict_ar_1step(theta = theta_hat , y_target = y)
#   residuals  = y[len(theta_hat):] - predY
#   mse = sum(residuals**2) / len(residuals)
#   return mse

for i in range(2,151):
  #calculate theta_hat
  theta_hat = fit_ar(trainDataY, i)
  #pred for train Data
  predYtrain = predict_ar_1step(theta = theta_hat , y_target = trainDataY)
  residualsYtrain  = trainDataY[len(theta_hat):] - predYtrain
  MSE_Train.append(sum(residualsYtrain**2) / len(residualsYtrain))

  #pred for Valid Data
  predYValid = predict_ar_1step(theta = theta_hat , y_target = ValidDataY)
  residualsYvalid  = ValidDataY[len(theta_hat):] - predYValid
  MSE_Valid.append(sum(residualsYvalid**2) / len(residualsYvalid))

```

```python
plt.plot(range(2,151) , MSE_Train , label = "MSE Train Data")
plt.plot(range(2,151) , MSE_Valid , label = "MSE Valid Data")
plt.xlabel("p")
plt.ylabel("MSE")
plt.suptitle("MSE vs Model Order(P)")
plt.legend()
plt.show()
```

**A8:**

Looking at the above plot of MSE vs Model Order P we can see that MSE for training data decreases with increased model order p while for validation data it is marginally changing between 4.6 and 5.

While around model order 107 both values decrease and post that MSE for Validation Data starts to increase.

Based on the above result we can consider model order as 10.

**Q9:** Based on the chosen model order, compute the residuals of the one-step-ahead predictions on the *validation data*. Plot the autocorrelation function of the residuals. What conclusions can you draw? Compare to the ACF plot generated above for p=10.

```python
#at model order of 110

#calculate theta_hat
theta_hat = fit_ar(trainDataY, 107)
#pred for Valid Data
predYValid = predict_ar_1step(theta = theta_hat , y_target = ValidDataY)
residualsValid107  = ValidDataY[len(theta_hat):] - predYValid

AutoCoR110 = acfplot(x = residualsValid107)

```

The plot suggests a decrease in the ACF compared to when p=10. Because we are using a much higher order now which means we
consider 107 previous times when predicting a new value. This leads to a
lower correlation as the lags used are far apart from the predictions
point.

## 1.4 Long-range predictions
So far we have only considered one-step-ahead predictions. However, in many practical applications it is of interest to use the model to predict further into the future. For intance, for the sea level data studied in this laboration, it is more interesting to predict the level one year from now, and not just 10 days ahead (10 days = 1 time step in this data).

**Q10**:
Write a function that simulates the value of an AR($p$) model $m$ steps into the future, conditionally on an initial sequence of data points. Specifically, given $y_{1:n}$ with $n\geq p$ the function/code should predict the values

\begin{align}
    \hat y_{t|n} &= \mathbb{E}[y_{t} | y_{1:n}], & t&=n+1,\dots,n+m.
\end{align}

Use this to predict the values for the validation data ($y_{701:997}$) conditionally on the training data ($y_{1:700}$) and plot the result.

_Hint:_ Use the pseudo-code derived at the first pen-and-paper session.

**A10:**

```python
def simulate_ar(y, theta, m):
    """Simulates an AR(p) model for m steps, with initial condition given by the last p values of y

    :param y: array (n,) with n>=p. The last p values are used to initialize the simulation.
    :param theta: array (p,). AR model parameters,
    :param m: int, number of time steps to simulate the model for.
    """

    p = len(theta)
    y_sim = np.zeros(m)
    phi = np.flip(y[-p:].copy()) # (y_{n-1}, ..., y_{n-p})^T - note that y[ntrain-1] is the last training data point

    #Last Element Index which is to be removed
    #popOutIndex = len(phi) - 1

    theta = np.flip(theta.copy())

    for i in range(m):
        y_sim[i] = phi.dot(theta)
        #phi = np.append(y_sim[i] , np.delete(phi , popOutIndex))
        phi = np.append(y_sim[i] , np.delete(phi , -1))
    return y_sim

```

```python
theta_Gen = fit_ar(y = trainDataY , p = 27)
m = 997 - 700

ySimPredValid = simulate_ar( y = trainDataY , theta = theta_Gen , m = m)

plt.plot(ValidDataX , ValidDataY , label = "Original Validation Data")
plt.plot(ValidDataX , ySimPredValid , label = "Simulated Validation Data")

#figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()

```

**Q11:** Using the same function as above, try to simulate the process for a large number of time steps (say, $m=2000$). You should see that the predicted values eventually converge to a constant prediction of zero. Is this something that you would expect to see in general? Explain the result.

**A11**

```python
ySimLargeM = simulate_ar( y = trainDataY , theta = theta_Gen , m = 2000)

plt.plot(range(2000) , ySimLargeM , label = "Simulated Data with m = 2000")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()
```

```python
AutoCoR_YSimLarge = acfplot(x = ySimLargeM)
```

It can be seen that for such large prediction, predicted values end up being a function of Noise as after a point all past values considered are newly predicted values. Now Noise is distributed with mean = 0 thus it is expected to see a convergence of mean to 0.

It can be seen from the ACF plot that after a large lag like 1000, autocorrelation between values is very low and now they are only random walk generated by the distribution of noise function which has mean as 0, thus new predicted expected values are converging to 0.

## 1.5 Nonlinear AR model
 In this part, we switch to a nonlinear autoregressive (NAR) model, which is based on a feedforward neural network. This means that in this model the recursive equation for making predictions is still in the form $\hat y_t=f_\theta(y_{t-1},...,y_{t-p})$, but this time $f$ is a nonlinear function learned by the neural network. Fortunately almost all of the work for implementing the neural network and training it is handled by the `scikit-learn` package with a few lines of code, and we just need to choose the right structure, and prepare the input-output data.

**Q12**: Construct a NAR($p$) model with a feedforward (MLP) network, by using the `MLPRegressor` class from `scikit-learn`. Set $p$ to the same value as you chose for the linear AR model above. Initially, you can use an MLP with a single hidden layer consisting of 10 hidden neurons.
Train it using the same training data as above and plot the one-step-ahead predictions as well as the residuals, on both the training and validation data.

_Hint:_ You will need the methods `fit` and `predict` of `MLPRegressor`. Read the user guide of `scikit-learn` for more details. Recall that a NAR model is conceptuall very similar to an AR model, so you can reuse part of the code from above.

**A12:**

```python
def phiCal(y , p):
  n = y.shape[0]

  Phi = np.zeros((n-p,p), dtype= int)

  for j in range(p):
      Phi[:,j] = y[j:(n+j-p)]
  return Phi
p = 108
yModel = trainDataY[p:]

model = MLPRegressor(hidden_layer_sizes=(10,) , activation= "relu")
reg = model.fit(phiCal(y = trainDataY , p = 108), yModel)
n = trainDataY.shape[0]
dataValid_Y = np.append(trainDataY[(n - p): ] , ValidDataY)
predictedVal = reg.predict(phiCal(y = dataValid_Y , p = 108))
```

    /usr/local/lib/python3.6/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:571: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
      % self.max_iter, ConvergenceWarning)

```python
reg.coefs_[0].shape
```

    (108, 10)

```python
plt.plot(ValidDataX , predictedVal , label = "NAR Prediction with p = 108")
plt.plot(ValidDataX , ValidDataY , label = "Original Data")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()
```

```python
NARresidual = ValidDataY - predictedVal
plt.plot(ValidDataX , ValidDataY , label = "Original Data")
plt.plot(ValidDataX , NARresidual , label = "Residuals")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()
```

**Q13:** Try to expirement with different choices for the hyperparameters of the network (e.g. number of hidden layers and units per layer, activation function, etc.) and the optimizer (e.g. `solver` and `max_iter`).

Are you satisfied with the results? Why/why not? Discuss what the limitations of this approach might be.

**A13:**

We have tried with multiple settings, and with a higher number of hidden layers and other different parameters, we could see that residual for training data was significantly low while there was high variance for Validation data, which in turn suggested overfitting of the model.

The below model captures non-linearity in data with a comparatively lower residual, thus performs better comparatively.

If comparing the above two models below is satisfactory while it still has a high residual for validation data.

```python
model = MLPRegressor(hidden_layer_sizes=(30,10) , activation= "tanh" ,
                     solver="sgd" , learning_rate = "adaptive" , max_iter = 200)
reg = model.fit(phiCal(y = trainDataY , p = 108), yModel)
n = trainDataY.shape[0]
#dataValid_Y = np.append(trainDataY[(n - p): ] , ValidDataY)
predictTrainData = reg.predict(phiCal(y = trainDataY , p = 108))
predictTrainDataResidual = trainDataY[p:] - predictTrainData
predictedVal = reg.predict(phiCal(y = dataValid_Y , p = 108))
NARresidualOpt = ValidDataY - predictedVal
plt.plot(ValidDataX , NARresidual , label = "Pre Changes Residuals A12 ")
plt.plot(ValidDataX , NARresidualOpt , label = "Post Changes Residuals A13")
plt.plot(trainDataX[p:] , predictTrainDataResidual , label = "Train Pred Residuals")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()
```

```python
plt.plot(ValidDataX , NARresidual , label = "Pre Changes Residuals :: A12 ")
plt.plot(ValidDataX , NARresidualOpt , label = "Post Changes Residuals :: A13")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.legend()
plt.show()
```


# 2/ Exercise solutions

```python
# setup SymPy
from sympy import *
x, y, z, t = symbols('x y z t')
init_printing()
```

```python

```

## Definitions

```python

```

```python

```

### E2.1

```python

```

```python

```

### E2.2

Given the matrices $A=\begin{pmatrix}1 & 3 \\ 4 & 5\end{pmatrix}$ and $B=\begin{pmatrix} -1 & 0 \\ 3 & 3 \end{pmatrix}$,
and the vectors $\vec{v}=\begin{pmatrix}1 \\  2\end{pmatrix}$ and $\vec{w}=\begin{pmatrix}-3 \\ -4\end{pmatrix}$,
compute the following expressions.

- a) $A\vec{v}$
- b) $B\vec{v}$
- c) $A(B\vec{v})$
- d) $B(A\vec{v})$
- e) $A\vec{w}$
- f) $B\vec{w}$

```python
# define the matrices A and B, and the vecs v and w
A = Matrix([[1,3],
            [4,5]])
B = Matrix([[-1,0],
            [ 3,3]])
v = Matrix([[1,2]]).T     # the .T makes v a column vector
w = Matrix([[-3,-4]]).T
```

```python
# a)
A*v
```

```python
# b)
B*v
```

```python
# c)
A*B*v
```

```python
# d)
B*A*v
```

```python
# e)
A*w
```

```python
# f)
B*w
```

### E2.3

```python

```

```python

```

## Vector operations

```python

```

```python

```

### E2.4

```python

```

```python

```

### E2.5

```python

```

```python

```

### E2.6

```python

```

```python

```

## Matrix operations

```python

```

```python

```

### E2.7

```python

```

```python

```

### E2.8

```python

```

```python

```

### E2.9

```python

```

```python

```

## Linearity

```python

```

```python

```

### E2.10

```python

```

```python

```


```python
# 그래프, 수학 기능 추가
# Add graph and math features
import pylab as py

```

# 수치 적분
Numerical Integration

면적이 2인 원의 반지름을 구해 보자.
Let's find the radius of a circle with area of 2.

$$
\begin{align}
    \pi r^2 &= 2 \\
    r^2 &= \frac{2}{\pi} \\
    r &= \sqrt{\frac{2}{\pi}}
\end{align}
$$

```python
r = py.sqrt(2.0 / py.pi)

```

```python
r

```

이러한 원의 중심이 원점에 위치하고 있다고 생각해 보자.
Let's assume that a circle of such radius has its center at the origin.

$$
x^2 + y^2 = r^2 \\
y^2 = r^2 - x^2 \\
y = \pm \sqrt{r^2 - x^2}
$$

```python
x_array = py.linspace(-r, r)
y_plus = py.sqrt(r**2 - x_array ** 2)
y_minus = -py.sqrt(r**2 - x_array ** 2)

py.plot(x_array, y_plus)
py.plot(x_array, y_minus)

py.axis('equal')
py.grid(True)

```

$+$ 부분만 생각하기로 하자.
Let's just think about the $+$ side only.

$$
y = \sqrt{r^2 - x^2}
$$

```python
x_array = py.linspace(-r, r)
y_plus = py.sqrt(r**2 - x_array ** 2)

py.fill_between(x_array, y_plus)

py.axis('equal')
py.grid(True)

```

이 반원의 면적을 수치적으로 구해보기로 하자.  (반원의 정확한 값은 얼마이겠는가?)

Let's try to numerically find the area of this half-circle. (What would be the exact value?)

## 0차 적분
0th Order Integration

우선 $x$를 일정 간격으로 나누어 보자.
Let's divide the $x$ coordinates in a constant interval.

```python
d = r * 2.0

n = 10

x_interval = d / n

x_array_bar = py.arange(-r, r+x_interval*0.1, x_interval)
y_array_bar = py.sqrt(abs(r**2 - x_array_bar ** 2))
x_interval = x_array_bar[1]-x_array_bar[0]

py.fill_between(x_array, y_plus)
# TODO : 막대그래프 직사각형 안을 칠하지 않으려면 어떻게 하면 좋겠는가?
# TODO : How can we remove the color inside the rectangle?
py.bar(x_array_bar, y_array_bar, width=x_interval, alpha=0.5, align='edge')

py.axis('equal')
py.grid(True)

```

직사각형의 모양과 반원의 모양이 정확히 일치하지는 않는다는 점을 기억하자.

Let's remember that the areas of the rectangles and the half circle are not exactly the same.

각 직사각형의 면적을 구해서 더해 보자
Let's find the area of each rectangle and sum up.

$$
    Area = \sum_{k=0}^{n-1} F_k
$$

$$
    F_k = f(x_k)\cdot \Delta x
$$

```python
summation = 0

for i, y_i in enumerate(y_array_bar[:-1]):
    area_i = x_interval * y_i
    print('i = %2d, area_i = %g' % (i, area_i))
    summation += area_i

print('summation =', summation)

```

예상한 값 1에 더 비슷한 값을 얻기 위해 더 잘게 나누어 보자
To obtain the result closer to the expected value of 1, let's divide with a narrower interval.

```python
n = 100

delta_x = 2.0 * r / n

x_array_bar = py.arange(-r, r+delta_x*0.1, delta_x)
y_array_bar = py.sqrt(r**2 - x_array_bar ** 2)
x_interval = x_array_bar[1]-x_array_bar[0]

py.fill_between(x_array, y_plus)
# TODO : 막대그래프 직사각형 안을 칠하지 않으려면 어떻게 하면 좋겠는가?
# TODO : How can we remove the color inside the rectangle?
py.bar(x_array_bar, y_array_bar, width=x_interval, alpha=0.5, align='edge')

py.axis('equal')
py.grid(True)

summation = 0

for i, y_i in enumerate(y_array_bar[:-1]):
    area_i = x_interval * y_i
    summation += area_i

print('summation =', summation)

```

더 잘게 나눈 결과에 대한 의견은 어떠한가?

What is your opinion about using the narrower interval?

### 함수로 구현
Implement in a Function

다양한 경우에 더 편리하게 적용하기 위해 함수 형태로 만들어 보자.
To make it more convenient to apply to various cases, let's implement in a function

```python
def half_circle(x):
    return py.sqrt(r**2 - x ** 2)

```

```python
def num_int_0(f, xi, xe, delta_x):
    x_array = py.arange(xi, xe+delta_x*0.1, delta_x)

    integration_result = 0.0

    for x_i in x_array:
        y_i = f(x_i)
        area_i = delta_x * y_i
        integration_result += area_i

    return integration_result

```

```python
n = 100
result = num_int_0(half_circle, -r, r, 2*r/n)
print('result =', result)

```

도전 과제 1: $sin \left( cos x \right)$ 를 $0 \le x \le 1$ 구간에서 적분하시오.
Try this 1: Integrate $sin \left( cos x \right)$ over $0 \le x \le 1$ interval.

(ref : [Examples for
Numerical Integration](https://www.wolframalpha.com/examples/mathematics/applied-mathematics/numerical-analysis/numerical-integration/), Wolfram Alpha, Accessed Aug 28 2018)

도전 과제 2: 이전 단원으로부터 이분법 함수를 복사하여 면적이 2인 원의 반지름을 구하는 프로그램을 작성하고 실행해 보시오.
Try this 2: Duplicate the bisection method function, write a program finding radius of a circle with area of two, and run it.

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```


```python
import sympy as sp
sp.init_printing()
```

```python
f=(x**3-4)**2-x**6+2*x**5
sp.simplify(f)
```

```python
f=(3*x**6-11*x+2)/(6*x)
```

```python
sp.simplify(f)
```

```python
f
```

    1/6*(3*x^6 - 11*x + 2)/x

```python
sp.plot(f)
```

```python
f(x)=(5*x^2-2)^3-1/2*x^3
```

```python
f
```

    x |--> (5*x^2 - 2)^3 - 1/2*x^3

```python
sp.expand(f)
```

```python
sp.expand((1-2*x)^3)
```

```python
sp.expand(-3*(x+4)*(x-2)+2*(3*x-1))
```

```python
sp.expand(((2*x+1)^2)*(x-1)-x*(4*x^2-1))
```

```python
y=x^2+500000
```

```python
sp.plot(y)
```

```python

```


# Supervised Learning: Neural Networks

```python
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style='ticks')

import tensorflow as tf
from scipy import optimize
from ipywidgets import interact
from IPython.display import SVG
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import scale
```

## McCulloch and Pitts Neuron

In 1943, McCulloch and Pitts introduced a mathematical model of a neuron. It consisted of three components:

1. A set of **weights** $w_i$ corresponding to synapses (inputs)
2. An **adder** for summing input signals; analogous to cell membrane that collects charge
3. An **activation function** for determining when the neuron fires, based on accumulated input

The neuron model is shown schematically below. On the left are input nodes $\{x_i\}$, usually expressed as a vector. The strength with which the inputs are able to deliver the signal along the synapse is determined by their corresponding weights $\{w_i\}$. The adder then sums the inputs from all the synapses:

$$h = \sum_i w_i x_i$$

The parameter $\theta$ determines whether or not the neuron fires given a weighted input of $h$. If it fires, it returns a value $y=1$, otherwise $y=0$. For example, a simple **activation function** is using $\theta$ as a simple fixed threshold:

$$y = g(h) = \left\{ \begin{array}{l}
1, \text{if } h \gt \theta \\
0, \text{if } h \le \theta
\end{array} \right.$$

this activation function may take any of several forms, such as a logistic function.

A single neuron is not interesting, nor useful, from a learning perspective. It cannot learn; it simply receives inputs and either fires or not. Only when neurons are joined as a **network** can they perform useful work.

Learning takes place by changing the weights of the connections in a neural network, and by changing the parameters of the activation functions of neurons.

## Perceptron

A collection of McCullough and Pitts neurons, along with a set of input nodes connected to the inputs via weighted edges, is a perceptron, the simplest neural network.

Each neuron is independent of the others in the perceptron, in the sense that its behavior and performance depends only on its own weights and threshold values, and not of those for the other neurons. Though they share inputs, they operate independently.

The number of inputs and outputs are determined by the data. Weights are stored as a `N x K` matrix, with N observations and K neurons, with $w_{ij}$ specifying the weight on the *i*th observation on the *j*th neuron.

In order to use the perceptron for statistical learning, we compare the outputs $y_j$ from each neuron to the obervation target $t_j$, and adjust the input weights when they do not correspond (*e.g.* if a neuron fires when it should not have).

$$t_j - y_j$$

We use this difference to update the weight $w_{ij}$, based on the input and a desired **learning rate**. This results in an update rule:

$$w_{ij} \leftarrow w_{ij} + \eta (t_j - y_j) x_i$$

After an incremental improvement, the perceptron is shown the training data again, resulting in another update. This is repeated until the performance no longer improves. Having a learning rate less than one results in a more stable learning rate, though this stability is traded off against having to expose the network to the data multiple times. Typical learning rates are in the 0.1-0.4 range.

An additional input node is typically added to the perceptron model, which is a constant value (usually -1, 0, or 1) that acts analogously to an intercept in a regression model. This establishes a baseline input for the case when all inputs are zero.

## Learning with Perceptrons

1. Initialize weights $w_{ij}$ to small, random numbers.
2. For each t in T iterations
    * compute activation for each neuron *j* connected to each input vector *i*
    $$y_j = g\left( h=\sum_i w_{ij} x_i \right) = \left\{ \begin{array}{l}
1, \text{if } h \gt 0 \\
0, \text{if } h \le 0
\end{array} \right.$$
    * update weights
    $$w_{ij} \leftarrow w_{ij} + \eta (t_j - y_j) x_i$$

This algorithm is $\mathcal{O}(Tmn)$

### Example: Logical functions

Let's see how the perceptron learns by training it on a couple of of logical functions, AND and OR. For two variables `x1` and `x2`, the AND function returns 1 if both are true, or zero otherwise; the OR function returns 1 if either variable is true, or both. These functions can be expressed as simple lookup tables.

```python
AND = pd.DataFrame({'x1': (0,0,1,1), 'x2': (0,1,0,1), 'y': (0,0,0,1)})
AND
```

First, we need to initialize weights to small, random values (can be positive and negative).

```python
w = np.random.randn(3)*1e-4
```

Then, a simple activation function for calculating $g(h)$:

```python
g = lambda inputs, weights: np.where(np.dot(inputs, weights)>0, 1, 0)
```

Finally, a training function that iterates the learning algorithm, returning the adapted weights.

```python
def train(inputs, targets, weights, eta, n_iterations):

    # Add the inputs that match the bias node
    inputs = np.c_[inputs, -np.ones((len(inputs), 1))]

    for n in range(n_iterations):

        activations = g(inputs, weights);
        weights -= eta*np.dot(np.transpose(inputs), activations - targets)

    return(weights)
```

Let's test it first on the AND function.

```python
inputs = AND[['x1','x2']]
target = AND['y']

w = train(inputs, target, w, 0.25, 10)
```

Checking the performance:

```python
g(np.c_[inputs, -np.ones((len(inputs), 1))], w)
```

Thus, it has learned the function perfectly. Now for OR:

```python
OR = pd.DataFrame({'x1': (0,0,1,1), 'x2': (0,1,0,1), 'y': (0,1,1,1)})
OR
```

```python
w = np.random.randn(3)*1e-4
```

```python
inputs = OR[['x1','x2']]
target = OR['y']

w = train(inputs, target, w, 0.25, 20)
```

```python
g(np.c_[inputs, -np.ones((len(inputs), 1))], w)
```

Also 100% correct.

### Exercise: XOR

Now try running the model on the XOR function, where a one is returned for either `x1` or `x2` being true, but *not* both. What happens here?

```python
# Write your answer here
```

Let's explore the problem graphically:

```python
AND.plot(kind='scatter', x='x1', y='x2', c='y', s=50, colormap='winter')
plt.plot(np.linspace(0,1.4), 1.5 - 1*np.linspace(0,1.4), 'k--');
```

```python
OR.plot(kind='scatter', x='x1', y='x2', c='y', s=50, colormap='winter')
plt.plot(np.linspace(-.4,1), .5 - 1*np.linspace(-.4,1), 'k--');
```

```python
XOR = pd.DataFrame({'x1': (0,0,1,1), 'x2': (0,1,0,1), 'y': (0,1,1,0)})

XOR.plot(kind='scatter', x='x1', y='x2', c='y', s=50, colormap='winter');
```

The perceptron tries to find a separating hyperplane for the two response classes. Namely, a set of weights that satisfies:

$$\mathbf{x_1}\mathbf{w}^T=0$$

and:

$$\mathbf{x_2}\mathbf{w}^T=0$$

Hence,

$$\begin{aligned}
\mathbf{x}_1\mathbf{w}^T &= \mathbf{x}_2\mathbf{w}^T \\
\Rightarrow (\mathbf{x}_1 - \mathbf{x}_2) \mathbf{w}^T &= 0
\end{aligned}$$

This means that either the norms of $\mathbf{x}_1 - \mathbf{x}_2$ or $\mathbf{w}$ are zero, or the cosine of the angle between them is equal to zero, due to the identity:

$$\mathbf{a}\mathbf{b} = \|a\| \|b\| \cos \theta$$

Since there is no reason for the norms to be zero in general, we need the two vectors to be at right angles to one another. So, we need a weight vector that is perpendicular to the decision boundary.

Clearly, for the XOR function, the output classes are not linearly separable. So, the algorithm does not converge on an answer, but simply cycles through two incorrect solutions.

## Multi-layer Perceptron

The solution to fitting more complex (*i.e.* non-linear) models with neural networks is to use a more complex network that consists of more than just a single perceptron. The take-home message from the perceptron is that all of the learning happens by adapting the synapse weights until prediction is satisfactory. Hence, a reasonable guess at how to make a perceptron more complex is to simply **add more weights**.

There are two ways to add complexity:

1. Add backward connections, so that output neurons feed back to input nodes, resulting in a **recurrent network**
2. Add neurons between the input nodes and the outputs, creating an additional ("hidden") layer to the network, resulting in a **multi-layer perceptron**

The latter approach is more common in applications of neural networks.

How to train a multilayer network is not intuitive. Propagating the inputs forward over two layers is straightforward, since the outputs from the hidden layer can be used as inputs for the output layer. However, the process for updating the weights based on the prediction error is less clear, since it is difficult to know whether to change the weights on the input layer or on the hidden layer in order to improve the prediction.

Updating a multi-layer perceptron (MLP) is a matter of:

1. moving forward through the network, calculating outputs given inputs and current weight estimates
2. moving backward updating weights according to the resulting error from forward propagation.

In this sense, it is similar to a single-layer perceptron, except it has to be done twice, once for each layer (in principle, we can add additional hidden layers, but without sacrificing generality, I will keep it simple).

### Error back-propagation

We update the weights in a MLP using **back-propagation** of the prediction errors, which is essentially a form of gradient descent, as we have used previously for optimization.

First, for the multi-layer perceptron we need to modify the error function, which in the single-layer case was a simple difference between the predicted and observed outputs. Because we will be summing errors, we have to avoid having errors in different directions cancelling each other out, so a sum of squares error is more appropriate:

$$E(t,y) = \frac{1}{2} \sum_i (t_i - y_i)^2$$

It is on this function that we will perform gradient descent, since the goal is to minimize the error. Specificially, we will differentiate with respect to the weights, since it is the weights that we are manipulating in order to get better predictions.

Recall that the error is a function of the threshold function

$$E(\mathbf{w}) = \frac{1}{2} \sum_i (t_i - y_i)^2 = \frac{1}{2} \sum_i \left(t_i - g\left[ \sum_j w_{ij} a_j \right]\right)^2$$

So, we will also need to differentiate that. However, the threshold function we used in the single-layer perceptron was discontinuous, making it non-differentiable. Thus, we need to modify it as well. An alternative is to employ some type of sigmoid function, such as the logistic, which can be parameterized to resemble a threshold function, but varies smoothly across its range.

$$g(h) = \frac{1}{1 + \exp(-\beta h)}$$

```python
logistic = lambda h, beta: 1./(1 + np.exp(-beta * h))

@interact(beta=(-1, 25))
def logistic_plot(beta=5):
    hvals = np.linspace(-2, 2)
    plt.plot(hvals, logistic(hvals, beta))
```

This has the advantage of having a simple derivative:

$$\frac{dg}{dh} = \beta g(h)(1 - g(h))$$

Alternatively, the hyperbolic tangent function is also sigmoid:

$$g(h) = \tanh(h) = \frac{\exp(h) - \exp(-h)}{\exp(h) + \exp(-h)}$$

```python
hyperbolic_tangent = lambda h: (np.exp(h) - np.exp(-h)) / (np.exp(h) + np.exp(-h))

@interact(theta=(-1, 25))
def tanh_plot(theta=5):
    hvals = np.linspace(-2, 2)
    h = hvals*theta
    plt.plot(hvals, hyperbolic_tangent(h))
```

Notice that the hyperbolic tangent function asymptotes at -1 and 1, rather than 0 and 1, which is sometimes beneficial, and its derivative is simple:

$$\frac{d \tanh(x)}{dx} = 1 - \tanh^2(x)$$

Performing gradient descent will allow us to change the weights in the direction that optimially reduces the error. The next trick will be to employ the **chain rule** to decompose how the error changes as a function of the input weights into the change in error as a function of changes in the inputs to the weights, mutliplied by the changes in input values as a function of changes in the weights.

$$\frac{\partial E}{\partial w} = \frac{\partial E}{\partial h}\frac{\partial h}{\partial w}$$

This will allow us to write a function describing the activations of the output weights as a function of the activations of the hidden layer nodes and the output weights, which will allow us to propagate error backwards through the network.

The second term in the chain rule simplifies to:

$$\begin{align}
\frac{\partial h_k}{\partial w_{jk}} &= \frac{\partial \sum_l w_{lk} a_l}{\partial w_{jk}}  \\
&= \sum_l \frac{\partial w_{lk} a_l}{\partial w_{jk}} \\
& = a_j
\end{align}$$

where $a_j$ is the activation of the jth hidden layer neuron.

For the first term in the chain rule above, we decompose it as well:

$$\frac{\partial E}{\partial h_k} = \frac{\partial E}{\partial y_k}\frac{\partial y_k}{\partial h_k} = \frac{\partial E}{\partial g(h_k)}\frac{\partial g(h_k)}{\partial h_k}$$

The second term of this chain rule is just the derivative of the activation function, which we have chosen to have a conveneint form, while the first term simplifies to:

$$\frac{\partial E}{\partial g(h_k)} = \frac{\partial}{\partial g(h_k)}\left[\frac{1}{2} \sum_k (t_k - y_k)^2 \right] = t_k - y_k$$

Combining these, and assuming (for illustration) a logistic activiation function, we have the gradient:

$$\frac{\partial E}{\partial w} = (t_k - y_k) y_k (1-y_k) a_j$$

Which ends up getting plugged into the weight update formula that we saw in the single-layer perceptron:

$$w_{jk} \leftarrow w_{jk} - \eta (t_k - y_k) y_k (1-y_k) a_j$$

Note that here we are *subtracting* the second term, rather than adding, since we are doing gradient descent.

We can now outline the MLP learning algorithm:

1. Initialize all $w_{jk}$ to small random values
2. For each input vector, conduct forward propagation:
    * compute activation of each neuron $j$ in hidden layer (here, sigmoid):
    $$h_j = \sum_i x_i v_{ij}$$
    $$a_j = g(h_j) = \frac{1}{1 + \exp(-\beta h_j)}$$
    * when the output layer is reached, calculate outputs similarly:
    $$h_k = \sum_k a_j w_{jk}$$
    $$y_k = g(h_k) = \frac{1}{1 + \exp(-\beta h_k)}$$
3. Calculate loss for resulting predictions:
    * compute error at output:
    $$\delta_k = (t_k - y_k) y_k (1-y_k)$$
4. Conduct backpropagation to get partial derivatives of cost with respect to weights, and use these to update weights:
    * compute error of the hidden layers:
    $$\delta_{hj} = \left[\sum_k w_{jk} \delta_k \right] a_j(1-a_j)$$
    * update output layer weights:
    $$w_{jk} \leftarrow w_{jk} - \eta \delta_k a_j$$
    * update hidden layer weights:
    $$v_{ij} \leftarrow v_{ij} - \eta \delta_{hj} x_i$$

Return to (2) and iterate until learning completes. Best practice is to shuffle input vectors to avoid training in the same order.

Its important to be aware that because gradient descent is a hill-climbing (or descending) algorithm, it is liable to be caught in local minima with respect to starting values. Therefore, it is worthwhile training several networks using a range of starting values for the weights, so that you have a better chance of discovering a globally-competitive solution.

One useful performance enhancement for the MLP learning algorithm is the addition of **momentum** to the weight updates. This is just a coefficient on the previous weight update that increases the correlation between the current weight and the weight after the next update. This is particularly useful for complex models, where falling into local mimima is an issue; adding momentum will give some weight to the previous direction, making the resulting weights essentially a weighted average of the two directions. Adding momentum, along with a smaller learning rate, usually results in a more stable algorithm with quicker convergence. When we use momentum, we lose this guarantee, but this is generally seen as a small price to pay for the improvement momentum usually gives.

A weight update with momentum looks like this:

$$w_{jk} \leftarrow w_{jk} - \eta \delta_k a_j + \alpha \Delta w_{jk}^{t-1}$$

where $\alpha$ is the momentum (regularization) parameter and $\Delta w_{jk}^{t-1}$ the update from the previous iteration.

The multi-layer pereptron is implemented below in the `MLP` class. The implementation uses the scikit-learn interface, so it is uses in the same way as other supervised learning algorithms in that package.

```python
softmax = lambda a: a / np.sum(a, axis=1, keepdims=True)

class MLP:

    def __init__(self, alpha=0.01, eta=0.01, n_hidden_dim=25):

        self.alpha = alpha
        self.eta = eta
        self.n_hidden_dim = n_hidden_dim

    # Helper function to evaluate the total loss on the dataset
    def calculate_loss(self, X, y):

        # Forward propagation to calculate our predictions
        z1 = X.dot(self.w1) + self.b1
        a1 = np.tanh(z1)
        z2 = a1.dot(self.w2) + self.b2
        exp_scores = np.exp(z2)
        probs = softmax(exp_scores)

        # Calculating the loss
        data_loss = -np.log(probs[range(num_examples), y]).sum()

        # Add regulatization term to loss (optional)
        data_loss += self.alpha/2 * np.square(self.w1).sum() + np.square(self.w2).sum()

        return 1./num_examples * data_loss

    # Helper function to predict an output (0 or 1)
    def predict(self, x):

        # Forward propagation
        z1 = x.dot(self.w1) + self.b1
        a1 = np.tanh(z1)
        z2 = a1.dot(self.w2) + self.b2
        exp_scores = np.exp(z2)
        probs = softmax(exp_scores)

        return np.argmax(probs, axis=1)

    def fit(self, X, y, num_passes=20000, print_loss=False, seed=42):

        num_examples, nn_input_dim = X.shape
        nn_output_dim = len(set(y))

        # Initialize the parameters to random values. We need to learn these.
        np.random.seed(seed)
        self.w1 = np.random.randn(nn_input_dim, self.n_hidden_dim) / np.sqrt(nn_input_dim)
        self.b1 = np.zeros((1, self.n_hidden_dim))
        self.w2 = np.random.randn(self.n_hidden_dim, nn_output_dim) / np.sqrt(self.n_hidden_dim)
        self.b2 = np.zeros((1, nn_output_dim))

        # Gradient descent. For each batch...
        for i in range(num_passes):

            # Forward propagation
            z1 = X.dot(self.w1) + self.b1
            a1 = np.tanh(z1)
            z2 = a1.dot(self.w2) + self.b2
            exp_scores = np.exp(z2)

            # Backpropagation
            delta3 = softmax(exp_scores)
            delta3[range(num_examples), y] -= 1
            dw2 = (a1.T).dot(delta3)
            db2 = np.sum(delta3, axis=0, keepdims=True)
            delta2 = delta3.dot(self.w2.T) * (1 - np.power(a1, 2))
            dw1 = np.dot(X.T, delta2)
            db1 = np.sum(delta2, axis=0)

            # Add regularization terms (b1 and b2 don't have regularization terms)
            dw2 += self.alpha * self.w2
            dw1 += self.alpha * self.w1

            # Gradient descent parameter update
            self.w1 += -self.eta * dw1
            self.b1 += -self.eta * db1
            self.w2 += -self.eta * dw2
            self.b2 += -self.eta * db2

            # Optionally print the loss.
            # This is expensive because it uses the whole dataset, so we don't want to do it too often.
            if print_loss and i % 1000 == 0:
              print("Loss after iteration %i: %f" %(i, calculate_loss(X, y)))

```

Let's initialize a MLP classifier, specifying the conjugate gradient minimization method.

```python
mlp = MLP()
```

Now we can confirm that it solves a non-linear classification, using the simple XOR example

```python
X = XOR[['x1','x2']].values
y = XOR['y'].values
```

```python
mlp.fit(X, y, num_passes=100)
```

```python
mlp.predict(X)
```

For a somewhat more sophisiticated example, we can use scikit-learn to simulate some data with a non-linear boundary.

```python
# Generate a dataset and plot it
np.random.seed(0)
X, y = datasets.make_moons(200, noise=0.20)
X = X.astype(np.float32)
y = y.astype(np.int32)
plt.scatter(X[:,0], X[:,1], s=40, c=y, cmap=plt.cm.Set1)
```

```python
clf = MLP(n_hidden_dim=3)
clf.fit(X, y)
```

```python
def plot_decision_boundary(pred_func, X=X, y=y):
    # Set min and max values and give it some padding
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole gid
    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.summer_r)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Greens)
```

```python
plot_decision_boundary(lambda x: clf.predict(x))
plt.title("Decision Boundary for hidden layer size 3")
```

Since we used the scikit-learn interface, its easy to take advantage of the `metrics` module to evaluate the MLP's performance.

```python
X, y = datasets.make_moons(50, noise=0.20)
X = X.astype(np.float32)
y = y.astype(np.int32)
```

```python
from sklearn.metrics import accuracy_score, confusion_matrix

accuracy_score(y, clf.predict(X))
```

```python
confusion_matrix(y, clf.predict(X))
```

# Varying the hidden layer size

In the example above we picked a hidden layer size of 3. Let's now get a sense of how varying the hidden layer size affects the result.

```python
plt.figure(figsize=(16, 32))
n_hidden_dim = [1, 2, 3, 4, 5, 20, 50]
for i, h in enumerate(n_hidden_dim):
    plt.subplot(5, 2, i+1)
    plt.title('Hidden Layer size %d' % h)
    model = MLP(n_hidden_dim=h)
    model.fit(X, y)
    plot_decision_boundary(lambda x: model.predict(x))
```

### Neural network specification

The MLP implemented above uses a single hidden layer, though it allows a user-specified number of hidden layer nodes (defaults to 25). It is worth considering whether it is useful having **multiple hidden layers**, and whether more hidden nodes is desirable.

Unfortunately, there is no theory to guide the choice of hidden node number. As a result, we are left to experiment with this parameter, perhaps in some systematic fashion such as cross-validation.

Adding additional layers presents only additional "bookkeeping" overhead to the user, with the weight updating becoming more complicated as layers are added. So, we don't want to add more hidden layers if it does not pay off in performance. It turns out that two or three layers (including the output layer) can be shown to approximate almost any smooth function. Combining 3 sigmoid functions allows local responses to be approximated with arbitrary accuracy. This is sufficient for determining any decision boundary.

### Neural network validation

Just as with other supervised learning algorithms, neural networks can be under- or over-fit to a training dataset. The degree to which a network is trained to a particular dataset depends on how long we train it on that dataset. Every time we run the MLP learning algorithm over a dataset (an **epoch**), it reduces the prediction error for that dataset. Thus, the number of epochs should be tuned as a hyperparameter, stopping when the testing-training error gap begins to widen.

Note that though we can also use cross-validation to tune the number of hidden layers in the network, there is no risk of overfitting by having too many layers.

### Exercise: Epoch tuning for MLPs

The dataset `pima-indians-diabetes.data.txt` in your data folder contains eight measurements taken from a group of Pima Native Americans in Arizona, along with an indicator of the onset of diabetes. Use the MLP class to fit a neural network classifier to this dataset, and use cross-validation to examine the optimal number of epochs to use in training.

1. Number of times pregnant
2. Plasma glucose concentration a 2 hours in an oral glucose tolerance test
3. Diastolic blood pressure (mm Hg)
4. Triceps skin fold thickness (mm)
5. 2-Hour serum insulin (mu U/ml)
6. Body mass index (weight in kg/(height in m)^2)
7. Diabetes pedigree function
8. Age (years)
9. Class variable (0 or 1)

```python
pima = pd.read_csv('../data/pima-indians-diabetes.data.txt', header=None)
pima.head()
```

```python
# Write your answer here
```

## Example: Mulitilayer perceptron in TensorFlow

TensorFlow is designed to evaluate expressions efficiently, and one of its key features is that it automatically differentiates expressions. This saves us from having to code a gradient function by hand.

For this, we will use the MNIST dataset, which we will introduce more formally later.

```python
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
```

```python
# Parameters
learning_rate = 0.001
training_epochs = 15
batch_size = 100
display_step = 1

# Network Parameters
nn_hdim = 256 # hidden layer number of neurons
nn_input_dim = 784 # MNIST data input (img shape: 28*28)
nn_output_dim = 10 # MNIST total classes (0-9 digits)
```

## Defining the Computation Graph in TensorFlow

The first thing we need to is define our computations using TensorFlow. We start by defining our input data matrix `X` and our training labels `y`:

```python
# Our data vectors
X = tf.placeholder('float', [None, nn_input_dim], name='X')
Y = tf.placeholder('float', [None, nn_output_dim], name='y')
```

Remember, we have not assigned any values to `X` or `y`. All we have done is defined mathematical expressions for them. We can use these expressions in subsequent calculations. If we want to evaluate an expression we can call its `eval` method. For example, to evaluate the expression `X * 2` for a given value of `X` we could do the following:

```python
tf.InteractiveSession()
```

This creates a TensorFlow `Session` that can be used interactively. More on this later.

```python
(Y * 2).eval(feed_dict={Y : np.random.randn(10)[None, :]})
```

Recall also that in addition to tensor objects, Theano also has shared variables which have values associated with them. Their value that is kept in memory and can be shared by all functions that use them. Shared variables can also be updated, and Theano includes low-level optimizations that makes updating them very efficient, especially on GPUs. Our network parameters $W_1, b_1, W_2, b_2$ are constantly updated using gradient descent, so they can be represented by shared variables:

```python
# Store layers weight & bias
weights = {
    'W1': tf.Variable(tf.random_normal([nn_input_dim, nn_hdim])),
    'W2': tf.Variable(tf.random_normal([nn_hdim, nn_hdim])),
    'out': tf.Variable(tf.random_normal([nn_hdim, nn_output_dim]))
}
biases = {
    'b1': tf.Variable(tf.random_normal([nn_hdim])),
    'b2': tf.Variable(tf.random_normal([nn_hdim])),
    'out': tf.Variable(tf.random_normal([nn_output_dim]))
}
```

We can now build our forward propagation step for the perceptron, in TensorFlow.

Note that we don't need to explicitly do a forward propagation here. TensorFlow knows that our gradients depend on our predictions from the forward propagation and it will handle all the necessary calculations for us. It does everything it needs to update the values.

```python
def multilayer_perceptron(x):
    # Hidden fully connected layer with 256 neurons
    layer_1 = tf.add(tf.matmul(x, weights['W1']), biases['b1'])
    # Hidden fully connected layer with 256 neurons
    layer_2 = tf.add(tf.matmul(layer_1, weights['W2']), biases['b2'])
    # Output fully connected layer with a neuron for each class
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
    return out_layer
```

Rather than calling the `eval` method to evaluate our TF expressions, we can instead define operators for expressions we want to evaluate.

For example, to calculate the loss, we need to know the values for $X$ and $y$, and pass them to our loss function.

```python
# Construct model
logits = multilayer_perceptron(X)

# Define loss and optimizer
loss_op = tf.reduce_mean(tf.losses.softmax_cross_entropy(
    logits=logits, onehot_labels=Y))
optimizer = tf.train.AdamOptimizer(learning_rate=eta)
train_op = optimizer.minimize(loss_op)
# Initializing the variables
init = tf.global_variables_initializer()
```

```python
with tf.Session() as sess:
    sess.run(init)

    # Training cycle
    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = int(mnist.train.num_examples/batch_size)
        # Loop over all batches
        for i in range(total_batch):
            batch_x, batch_y = mnist.train.next_batch(batch_size)
            # Run optimization op (backprop) and cost op (to get loss value)
            _, c = sess.run([train_op, loss_op], feed_dict={X: batch_x,
                                                            Y: batch_y})
            # Compute average loss
            avg_cost += c / total_batch
        # Display logs per epoch step
        if epoch % display_step == 0:
            print("Epoch:", '%04d' % (epoch+1), "cost={:.9f}".format(avg_cost))
    print("Optimization Finished!")

    # Test model
    pred = tf.nn.softmax(logits)  # Apply softmax to logits
    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(Y, 1))
    # Calculate accuracy
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    print("Accuracy:", accuracy.eval({X: mnist.test.images, Y: mnist.test.labels}))
```

---
## References

[The TensorFlow Playground](https://playground.tensorflow.org/)

T. Hastie, R. Tibshirani and J. Friedman. (2009) [Elements of Statistical Learning: Data Mining, Inference, and Prediction](http://statweb.stanford.edu/~tibs/ElemStatLearn/), second edition. Springer.

X. Glorot, A. Bordes and Y. Bengio (2011). [Deep sparse rectifier neural networks (PDF)](http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf). AISTATS.

S. Marsland. (2009) [Machine Learning: An Algorithmic Perspective](Machine Learning: An Algorithmic Perspectivehttp://seat.massey.ac.nz/personal/s.r.marsland/MLBook.html). CRC Press.

D. Rodriguez. (2013) [Basic [1 hidden layer] neural network on Python](http://danielfrg.com/blog/2013/07/03/basic-neural-network-python/).

D. Britz. (2015) [Implementing a Neural Network from Scratch](http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/)


# M10. Amdahl's Law

The most useful corollaries to what is now known as *Amdahl's Law* are hardly profound. The notion of prioritizing the improvements *with the greatest bearing on the overall result* is almost common sense (which is maybe to suggest that it isn't common at all). Violations of the Law are prevalent (though not ubiquitous):

- Behavior characterized by the Britishism: "Penny wise, pound foolish"—people who are thrifty when it comes to small expenses but who are indulgent or forgetful of larger ones.
- Spending legislative energy on banning plastic straws when they constitute 0.02 *percent* (by mass) of annual plastic waste deposited in the oceans.

The original version of the Law, advanced by Gene Amdahl in 1967, was nowhere near as ambitious in its scope (nor did it pretend to be). It merely provided an upper bound on speedups afforded by one type of improvement, parallelization, and it was formulated as follows:

Suppose a fixed sequential workload of latency $W$ can be partitioned into a parallelizable fraction $p$ and a non-parallelizable fraction $1-p$. Then, the speedup in latency offered by running the workload on $n$ processors is given by

\begin{align}
S(n) &= \frac{W}{\frac{pW}{n} + (1-p)W}\\
&= \frac{1}{\frac{p}{n}+(1-p)}
\end{align}

and hence we obtain the upper bound

$$\lim_{n\to\infty} S(n) = \frac{1}{1-p}.$$

Amdahl's Law is a 'real law' (much as Moore's Law is *not*), but it is somewhat naive and primitive in the sense that its operative assumption is that workloads can be partitioned into different regimes, which is often easier said than done. In fact, Amdahl's Law is sometimes seen being used to *back-calculate* the fraction $p$ of a workload exposed to a given improvement.

One word of caution: the fraction $p$ referred to has nothing to do with the *frequency* of the improvable portion of a workload and everything to do with its *latency*. Things which occur infrequently may still be catastrophically time-intensive when they do occur. On the other hand, there are many things which *do* happen often and yet contribute little to overall execution time.


```python
import numpy as np
import random
import math
import pandas as pd
import numba as nb
import itertools
from scipy import optimize
from scipy import interpolate
from copy import copy
from types import SimpleNamespace
%load_ext autoreload
%autoreload 2
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
```

# The logit model

The function `DGP()` will create the $N$ observations of $(y_i,x_i)$:

```python
def DGP(mp):
    ''' The data generating process behind binary choice model

    Args:
        mp (SimpleNamespace): object containing parameters for data generation

    Returns:
        y_obs (ndarray): indicator for binary choices made by individuals
        x_obs (ndarray): independent variables

    '''

    # a. Exogenous variables
    x0 = np.tile(1.0, mp.N)
    x1 = np.random.normal(**mp.x1_distr)
    x2 = np.random.normal(**mp.x2_distr)
    x_obs = np.vstack((x0, x1, x2)).T

    # b. Probabilities of action choice
    y_prb = np.exp(x_obs @ mp.beta) / (1 + np.exp(x_obs @ mp.beta))

    # c. Dra binary choices from the binomial distribution based on individual probabilities
    y_obs = np.random.binomial(1, y_prb)
    return y_obs, x_obs
```

Create your data using the following parameterization:

```python
# Parameters
mp = SimpleNamespace()
mp.beta = np.array([0.15, 0.1, 0.2])
mp.N = 100_000
mp.x1_distr = {'loc': 4, 'scale': 3, 'size': mp.N}
mp.x2_distr = {'loc': 1, 'scale': 0.5, 'size': mp.N}

# Create data
np.random.seed(2021)
y_obs, x_obs = DGP(mp)
```

**Question 1:** Create a function that calculates the log-likelihood of your data based on a $\beta$. That is, the function must take as arguments an array `beta`, `y_obs` and `x_obs`

```python
def prb(beta, x):
    ''' Computes the probabilities P(y=1|x, beta)

    Args:

        beta (ndarray): coefficients to independent variables
        x (ndarray): independent variables

    Returns:

        (ndrarray): probability of each observation taking action y = 1 given coefficients in beta.

    '''
    return np.exp(x @ beta) / (1 + np.exp(x @ beta))

def log_likelihood(beta, y, x):
    ''' Computes the sum of log-likelihood contributions for observations given beta

    Args:

        beta (ndarray): coefficients to independent variables
        x (ndarray): independent variables
        y (ndarray): observed binary choices

    Returns:

        (float): sum of log-likelihood contributions

    '''
    # a. Calculate probabilities P(y=1|x, beta)
    prbs = prb(beta, x)

    # b. Avoiding numerical instabilities as prbs get very close to 0 or 1.
    # Done by adding or subtracting a small e
    e = 1e-5
    prbs += e*(prbs < e) - e*(prbs > 1.0 - e)

    # c. Return sum of log-likelihood contributions
    return np.sum(y*np.log(prbs) + (1-y)*np.log(1-prbs))

print(f'test of log_likelihood: {log_likelihood(mp.beta, y_obs, x_obs): .5f}')
```

    test of log_likelihood: -62236.59859

**Question 2:** Make a 3d-plot of the likelihood function where $\beta_1$ and $\beta_2$ are on the horizontal axes, and the log-likelihood is on the vertical axis. Visually confirm that it peaks at the data generating $\beta_1$ and $\beta_2$.

*Note:* You can let $\beta_0$=`mp.beta[0]`. Make sure that `mp.beta[1]` and `mp.beta[2]` are in the grids over $\beta_1$ and $\beta_2$.

```python
# a. Create grids over beta_1 and beta_2
Ngrid = 30
beta_1 = np.linspace(0, mp.beta[1]*2, Ngrid)
beta_2 = np.linspace(0,  mp.beta[2]*2, Ngrid)
beta_1_grid, beta_2_grid = np.meshgrid(beta_1, beta_2, indexing='ij')
ll_grid = np.empty((Ngrid, Ngrid))

# b. Initialize beta array with beta_0 set to the DGP-value
beta = np.array([mp.beta[0], np.nan, np.nan])
ll_opt = log_likelihood(mp.beta, y_obs, x_obs)

# c. Fill out grid
for i in range(Ngrid):
    for j in range(Ngrid):
        beta[1] = beta_1_grid[i,j]
        beta[2] = beta_2_grid[i,j]
        ll_grid[i,j] = log_likelihood(beta, y_obs, x_obs)

# d.i Create 3d-plot
fig = plt.figure(dpi = 120)
ax = fig.add_subplot(1,1,1,projection='3d')
ax.plot_surface(beta_1_grid,beta_2_grid,ll_grid, cmap=cm.jet, alpha = 1);
ax.set_xlabel(r'$\beta_1$')
ax.set_ylabel(r'$\beta_2$')
ax.set_zlabel('Log-likelihood', labelpad=13)
ax.tick_params(axis='z', which='major', pad=8)
ax.invert_xaxis()

# d.ii Indicate max
ax.plot(beta_1, np.repeat(mp.beta[2], Ngrid), np.repeat(ll_opt, Ngrid), '--', color = 'black')
ax.plot(np.repeat(mp.beta[1], Ngrid), beta_2, np.repeat(ll_opt, Ngrid), '--',  color = 'black')
ax.text(mp.beta[1]+0.01, mp.beta[2]+0.02, ll_opt, r'$\hat{\beta}$')

fig.tight_layout()
```

**Question 3:** Estimate $\beta$ by maximum likelihood. You may use a gradient-free approach or gradients if you will.

```python
# a. Objective function, depends on observations and coefficients
obj_func = lambda beta: -log_likelihood(beta, y_obs, x_obs)
beta_guess = np.array([2.2, 0.8, 0.01])

# b. Estimation using Nelder-Mead (derivative free)
res = optimize.minimize(obj_func, beta_guess, method='nelder-mead')
beta_hat = res.x

print(f'beta in data generating process:\n {np.around(mp.beta, 5)}\n')
print(f'initial guess of beta:\n {beta_guess}\n')
print(f'estimate of beta:\n {np.around(beta_hat, 3)}\n')
print(res.message)
print('Function evaluations: ', res.nfev)
```

    beta in data generating process:
     [0.15 0.1  0.2 ]

    initial guess of beta:
     [2.2  0.8  0.01]

    estimate of beta:
     [0.152 0.095 0.204]

    Optimization terminated successfully.
    Function evaluations:  236

**Using analytical derivatives**
The analytical derivative of the log-likelihood function can be calculated as
$$
\begin{equation}
	\frac{\partial}{\partial\beta}LL(\beta) = \frac{1}{N}\sum \limits_{i=1}^{N}(y_i - P(y_i = 1|x_i;\beta))x_i
\end{equation}
$$

```python
def ll_grad(beta, y, x):
    ''' Compute numerical derivative of log-likelihood

     Args:

        beta (ndarray): coefficients to independent variables
        x (ndarray): independent variables
        y (ndarray): observed binary choices

    Returns:

        (ndarray): array with derivatives of each element of beta

    '''
    err = y - prb(beta, x)
    grad = err[:,np.newaxis]*x
    return grad.mean(axis=0)
```

```python
# Estimate with derivative
grad_func = lambda beta: -ll_grad(beta, y_obs, x_obs)
res = optimize.minimize(obj_func, beta_guess, jac = grad_func, method='BFGS')
print(f'estimate of beta:\n {np.around(beta_hat, 3)}\n')
print(res.message)
print('Function evaluations: ', res.nfev)
```

    estimate of beta:
     [0.152 0.095 0.204]

    Optimization terminated successfully.
    Function evaluations:  14

Note that the same coefficients are obtained but in fewer function evaluations.

**Question 4:** Based on your estimated parameters, simulate a choice `y_sim` pr individual in `x_obs`. Create an output table that shows following 4 statistics:
The number of times where:
* `y_obs` = 1 and `y_sim` = 1
* `y_obs` = 1 and `y_sim` = 0
* `y_obs` = 0 and `y_sim` = 1
* `y_obs` = 0 and `y_sim` = 0

```python
# There are two ways to simulate choices.
# Either a) calculate y*, or b) draw from binomial distribution based on probabilities.
# For exposition, both are done here.

# a. Simulate y based on predicted y*
y_star = x_obs @ beta_hat
e = np.random.logistic(size=(mp.N,1))
y_star = y_star[:, np.newaxis] + e
y_sim = y_star > 0

# b. Simulate y based on draws from binomial distribution
y_sim = np.random.binomial(1, prb(beta_hat, x_obs))

# c. Create table. There are many ways of doing this.
# A quick and dirty is to use cross tabulation in pandas.
d = {'y_obs': y_obs, 'y_sim': y_sim}
df = pd.DataFrame(d)

print('Cross tabulation of choices (observations vs simulations).')
print('Simulation along rows, observations along columns.')
display(pd.crosstab(index = df.y_obs, columns = df.y_sim))

# Alternatively display share of observations in each group
display(pd.crosstab(index = df.y_obs, columns = df.y_sim, normalize = True))
```

    Cross tabulation of choices (observations vs simulations).
    Simulation along rows, observations along columns.

      y_sim
      0
      1

      y_obs

      0
      11056
      21661

      1
      21534
      45749

      y_sim
      0
      1

      y_obs

      0
      0.11056
      0.21661

      1
      0.21534
      0.45749

Observe that the off-diagonal elements are symetrically distributed - hence we have unbiased estimates.

**Question 5:** Test if your initial guess of $\beta$ will have an impact on the final estimate. Why do you think there is/is not an impact?

Theoretically, starting pooints should not matter as the log-likelihood function is globally concave. However, there may be numerical issues when trying to calculate the log of the likelihood at "extreme" values of $\beta$ that bring probabilities close to 0 or 1. You can check the `log_likelihood()` to see how this is dealt with.

```python
# a. Initialize containers and starting points beta0
Nguess = 10
obj_func = lambda beta: -log_likelihood(beta, y_obs, x_obs)
beta_hat = np.empty((Nguess,3))
beta0 = np.random.normal(loc=0.2, scale = 1.5, size = (Nguess,3))
converged = []
print('Starting points beta0:')
display(beta0)

# b. Estimate the logit model for each of the intial values
print('\nEstimating models')
for j in range(Nguess):
    res = optimize.minimize(obj_func, beta0[j,:], method='nelder-mead')
    beta_hat[j,:] = res.x
    converged.append(res.success)
    print(f'{int(100*(j+1)/Nguess)}% done')

# c. Check that all estimations converged and no estimate deviates significantly from mp.beta
print('\nNumber of models not converged')
print(sum(np.array(converged) == False))

print('\nNumber of estimates not within a 0.01 deviation of mp.beta, the DGP parameters')
print(sum(abs(beta_hat - mp.beta) > 0.01))
```

    Starting points beta0:

    array([[-2.86525731, -2.03609035,  0.38388402],
           [ 0.39462624, -0.26513596,  0.69645808],
           [-0.29099758, -2.49667252,  2.75097954],
           [ 1.73871813, -1.7501233 ,  1.52750451],
           [-0.02819087,  0.12570621,  0.38895426],
           [ 2.39126485,  0.80269743,  0.87043819],
           [-0.61952926, -0.73285285, -2.25049318],
           [-0.38585757, -0.6192172 , -0.02293941],
           [-1.55869982,  2.53268954,  1.18028118],
           [ 1.92900499,  0.01047725,  1.1124988 ]])

    Estimating models
    10% done
    20% done
    30% done
    40% done
    50% done
    60% done
    70% done
    80% done
    90% done
    100% done

    Number of models not converged
    0

    Number of estimates not within a 0.01 deviation of mp.beta, the DGP parameters
    [0 0 0]

# Consumption saving with borrowing

**Note**: checkout how **the borrowing constraint** is implemented in `solve_period_1()`.

```python
# Parameters
rho = 3
kappa = 0.5
nu = 0.1
r = 0.04
beta = 0.95
Delta = 0.5
```

**Question 1** Solve the model for each type of household. Plot the value functions $v_1(m_1)$ and $v_2(m_2)$ in one graph for each household type. Comment on the differences.

```python
# Set parameters
par = SimpleNamespace()
par.rho = rho
par.kappa = kappa
par.nu = nu
par.r = r
par.beta = beta
par.Delta = Delta
par.max_debt = (1-par.Delta)/(1+par.r)

# Type 1
par1 = copy(par)
par1.prb = {'low': 0.9, 'high': 0.1}

# Type 2
par2 = copy(par)
par2.prb = {'low': 0.1, 'high': 0.9}
```

```python
def utility(par, c):
    ''' Calculate flow utility of consumption level c

    Args:

        par (SimpleNamespace): model parameters
        c (ndarray): level of consumtion

    Returns:

        (ndarray): flow utility of consumption
    '''

    return c**(1-par.rho)/(1-par.rho)

def bequest(par, m, c):
    ''' Calculate flow utility of leaving bequest given residual consumption

    Args:

        par (SimpleNamespace): model parameters
        m (ndarray): cash-on-hand
        c (ndarray): level of consumtion

    Returns:

        (ndarray): utility of bequests
    '''

    return par.nu*(m-c+par.kappa)**(1-par.rho)/(1-par.rho)

def v2(par, c2, m2):
    ''' Compute state specific value of consumption choice and bequests in period 2

    Args:

        par (SimpleNamespace): model parameters
        m2 (ndarray): cash-on-hand in period 2
        c2 (ndarray): level of consumtion in period 2

    Returns:

        (ndarray): value of comsumption and bequests
    '''

    return utility(par, c2) + bequest(par, m2, c2)

def v1(par, c1, m1, v2_interp):
    ''' Compute state specific value of consumption choice in period 1

    Args:

        par (SimpleNamespace): model parameters
        m1 (ndarray): cash-on-hand in period 1
        c1 (ndarray): level of consumtion in period 1
        v2_interp (RegularGridInterpolator): interpolator between m in period 2 and value function

    Returns:

        (ndarray): state specific value of consumption choice in period 1

    '''

    # a. v2 if low income realization
    m2_low = (1+par.r)*(m1-c1) + 1-par.Delta
    v2_low = v2_interp([m2_low])[0]

    # b. v2 if high income realization
    m2_high = (1+par.r)*(m1-c1) + 1+par.Delta
    v2_high = v2_interp([m2_high])[0]

    # c. Expected v2 value
    expected_v2 = par.prb['low']*v2_low + par.prb['high']*v2_high

    # d. Total value
    return utility(par, c1) + par.beta*expected_v2
```

```python
def solve_period_2(par):
    ''' Solve the consumption problem of period 2

    Args:

        par (SimpleNamespace): model parameters

    Returns:

        m2s (ndarray): cash-on-hand levels in period 2
        v2s (ndarray): value function in period 2
        c2s (ndarray): consumption function in period 2 (ie policy function)

    '''

    # a. Initialize grids
    m2s = np.linspace(1e-4,5,500)
    v2s = np.empty(500)
    c2s = np.empty(500)

    # b. Solve consumption problem for each m2 in grid
    for i,m2 in enumerate(m2s):

        # i. Objective function
        obj = lambda x: -v2(par, x[0], m2)

        # ii. Initial guess (consume half of m2)
        x0 = m2/2

        # iii. Optimize the objective of allocating between consumption and bequests
        result = optimize.minimize(obj, [x0], method='L-BFGS-B', bounds=((1e-8, m2),))

        # iv. Save solution
        v2s[i] = -result.fun
        c2s[i] = result.x

    return m2s, v2s, c2s

def solve_period_1(par, v2_interp):
    ''' Solve the consumption problem of period 1

    Args:

        par (SimpleNamespace): model parameters
        v2_interp (RegularGridInterpolator): interpolator between m in period 2 and value function

    Returns:

        m1s (ndarray): cash-on-hand levels in period 1
        v1s (ndarray): value function in period 1
        c1s (ndarray): consumption function in period 1 (ie policy function)

    '''

    # a. Initialize grids
    m1s = np.linspace(1e-8, 5, 100)
    v1s = np.empty(100)
    c1s = np.empty(100)

    # b. Solve for each m1 in the grid
    for i, m1 in enumerate(m1s):

        # i. Objective function
        def obj(x): return -v1(par, x[0], m1, v2_interp)

        # ii. Initial guess (consume half of m1)
        x0 = m1/2

        # iii. Optimize the objective given debt constraint
        result = optimize.minimize(obj, [x0],
                                   method='L-BFGS-B', bounds=((1e-12, m1 + par.max_debt),))

        # iv. Save solution
        v1s[i] = -result.fun
        c1s[i] = result.x[0]

    return m1s, v1s, c1s

def solve(par):
    ''' Solve the consumption savings problem over all periods

    Args:

        par (SimpleNamespace): model parameters

    Returns:

        (dict): solutions for period 1 and period 2. Ie., cash-on-hand, value function and consumption function

    '''

    def solution_to_dict(sol):
        names = ['m','v','c']
        return dict((n, i) for n,i in zip(names, sol))

    # a. solve period 2
    sol2 = solve_period_2(par)
    sol2 = solution_to_dict(sol2)

    # b. construct interpolator
    v2_interp = interpolate.RegularGridInterpolator([sol2['m']], sol2['v'],
                                                    bounds_error=False, fill_value=None)

    # c. solve period 1
    sol1 = solve_period_1(par, v2_interp)
    sol1 = solution_to_dict(sol1)

    return {'t1': sol1, 't2': sol2}
```

```python
# a. Solve for both types of households
types = {1: par1, 2: par2}
solutions = {}

for i,p in types.items():
    solutions[i] = solve(p)

# b. Plot value functions for both types
fig = plt.figure(figsize=(14,5))
for i,sol in solutions.items():
    ax = fig.add_subplot(1,2,i)
    ax.plot(sol['t1']['m'], sol['t1']['v'], label=f'Period 1')
    ax.plot(sol['t2']['m'], sol['t2']['v'], label=f'Period 2')
    ax.legend(loc='lower right',facecolor='white',frameon=True)
    ax.set_xlabel('$m_t$')
    ax.set_ylabel('$v_t$')
    ax.set_title(f'Value functions type {i}');
    ax.set_xlim([0,4])
    ax.set_ylim([-20, 2]);
```

**Question 2** From the model solution, obtain the optimal consumption functions $c_1^*(m_1)$ and $c_2^*(m_2)$. Plot these in one graph for each type of household. Comment on the observed differences between household types.

```python
# Plot consumption functions for both types
fig = plt.figure(figsize=(14,5))
for i,sol in solutions.items():
    ax = fig.add_subplot(1,2,i)
    ax.plot(sol['t1']['m'], sol['t1']['c'], label=f'Period 1')
    ax.plot(sol['t2']['m'], sol['t2']['c'], label=f'Period 2')
    ax.legend(loc='lower right',facecolor='white',frameon=True)
    ax.set_xlabel('$m_t$')
    ax.set_ylabel('$c_t$')
    ax.set_title(f'Consumption functions type {i}');
    ax.set_xlim([0,5])
    ax.set_ylim([0, 4]);
```

Note that consumption functions in period 2 are identical across types, as they have identical preferences.

Period 1 consumption functions differ in that type 2 households will consume a higher fraction of $m$ due to higher expectations of $y_2$.

Furthermore, both types will borrow ($c_1 > m_1$) at sufficiently low levels of $m_1$, but the propensity is higher for type 2 households.

**Question 3** Simulate `simN` households of each type based on the distribution of $m_1$ below. You can use the same distribution for both household types. What is the fraction of households who *borrow* in period 1, $c_1 > m_1$, in each group?

**Note**: although it is not necessary for answering the question, both periods are simulated below. This is primarily to demonstrate how to draw realizations of $y_2$.

```python
np.random.seed(2021)
simN = 1000
sim_m1 = np.fmax(np.random.normal(1, 1, size = simN), 0) # No one gets negative m in first period
```

```python
def simulate(par):
    ''' Simulate consumption savings choices based on solution of problem over all periods

    Args:

        par (SimpleNamespace): model parameters

    Returns:

        (dict): consumption choices for period 1 and period 2 and period 2 cash.

    '''

    # a. Solve the model for both periods
    sol = solve(par)

    # b. Construct interpolaters between cash and consumption choices
    c1_interp = interpolate.RegularGridInterpolator([sol['t1']['m']], sol['t1']['c'],
                                                    bounds_error=False, fill_value=None)

    c2_interp = interpolate.RegularGridInterpolator([sol['t2']['m']], sol['t2']['c'],
                                                    bounds_error=False, fill_value=None)

    # c. Simulate period 1 based on array of m and solution
    sim_c1 = c1_interp(par.sim_m1)
    sim_a1 = par.sim_m1 - sim_c1

    # d. Transition to period 2 cash-on-hand based on random draws of income and period 1 choices
    y2_low = 1-par.Delta
    y2_high = 1+par.Delta
    y2 = np.random.choice([y2_low, y2_high],
                          p=[par.prb['low'], par.prb['high']],
                          size=(sim_c1.shape))

    sim_m2 = (1+par.r)*sim_a1 + y2

    # e. sim period 2 consumption choice based on model solution and sim_m2
    sim_c2 = c2_interp(sim_m2)

    return {'c1': sim_c1, 'c2': sim_c2, 'sim_m2': sim_m2}
```

```python
# a. Use the same distribution of cash-on-hand endowment for both types and simulate choices
for i,par in types.items():
    par.sim_m1 = copy(sim_m1)
    par.sim_c = simulate(par)

# b. Calculate the share of borrowers in period 1
for i,par in types.items():
    borrowers = np.sum(par.sim_c['c1'] > par.sim_m1)/simN
    print(f'Share of borrowers in type {i}: {borrowers: .3f}')
```

    Share of borrowers in type 1:  0.337
    Share of borrowers in type 2:  0.488

Note, there is a higher share of borrowers in period 1 among type 2 households. This is because they are more certain of high income in period 2

# Division by Newton's method

**Question 1**
Using $g(x)$ in $\mathcal{N}(x_k)$ allows us to reorganize the Newton iteration by
$$
x_{k+1} = x_{k} - \frac{g(x_{k})}{g^{\prime}(x_{k})} =  x_{k} + \frac{\frac{1}{x_k}-d}{\frac{1}{x^2_k}} = x_k + (1-x_k d)x_k = (2 - x_k d)x_k
$$
Thus, we can avoid using division while searching for the quotient $\frac{1}{d}$.

```python
def newton_division(n, d, d0, max_iter=500, tol=1e-8):
    ''' Calculate the quotient n/d using Newton's method

    Args:

        n (float): numerator
        d (float): denominator
        d0 (float): initial guess of 1/d
        max_iter (int): maximum number of iterations
        tol (float): convergence criterion

    Returns:

        (float): quotient n/d
        (ndarray): history of quotients up to convergence

    '''
    dts = []
    dt = d0
    k = 0

    while True:

        # Step 2: update x
        dt_new = dt*(2-d*dt)

        # Step 3: check convergence
        if abs(dt_new - dt) < tol:
            break

        # Step 4: update guess of d_tilde and store history
        dt = dt_new
        dts.append(dt_new)
        k += 1

        # If max iterations are exceeded, return nan
        if k >= max_iter:
            return np.nan, dts

    return dt*n, np.array(dts)*n
```

```python
n = 37.581
d = 5.9
d0 = 0.2
q, hist = newton_division(n, d, d0, max_iter=500, tol=1e-8)

print('q = ', q)
print('Is q close to result from native Python division algorithm?')
print(np.allclose(q, n/d))
print('Iterations:', len(hist))
```

    q =  6.369661016941419
    Is q close to result from native Python division algorithm?
    True
    Iterations: 4


Ridge Regression (L2 Regularization)

## Ridge Regression Solution
In the Ridge Regression (L2 Regularization) theory [video](https://youtu.be/skOcLw_fXDs) we derived the solution to be:

\begin{equation}
\hat{\theta} = (X^TX + \lambda I)^{-1}X^TY
\end{equation}

where
\begin{equation}
X = [\bar{x}^T_1, \bar{x}^T_2, ... , \bar{x}^T_n]^T
\end{equation}

\begin{equation}
Y = [y_1, y_2, ... , y_n]^T
\end{equation}

This solution minimizes the following cost function

\begin{equation}
J(x, \theta, y) = \sum_{i=1}^{m}(\theta^T\bar{x}_i - y_i)^2 + \lambda ||\theta||^2
\end{equation}

```python
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
```

```python
class Ridge:
    """Linear least squares with L2 regularization."""

    def __init__(self, lam):
        """Initialize a Ridge object.

        Args:
            lam: the regularization factor
        """
        self._lambda = lam

    @staticmethod
    def _x_bar(x):
        """Create the vector x_bar.

        Args:
            x: input vector
        """
        return np.hstack(([1.0], x, np.square(x)))

    def fit(self, x_train, y_train):
        """Generate a fit for the data.

        Args:
            x_train: the input values of the training data
            y_train: the output values of the training data
        """
        # stack the data
        X = np.vstack(([self._x_bar(x) for x in x_train]))
        Y = np.vstack(([y for y in y_train]))

        # compute the model coeff
        # theta = inv(xTx + lam*I) * xTy
        XT = np.transpose(X)
        XTX = np.matmul(XT, X) + self._lambda * np.identity(X.shape[1])
        self._coeff_hat = np.matmul(np.matmul(np.linalg.inv(XTX), XT), Y)

```

```python
"""Generate fake data"""
c2 = 0.01
c1 = 1.3
c0 = 3.456
x_in = np.linspace(-10.0, 20.2, 200.0)
y_out = c1 * x_in ** 2 + c1 * x_in + c0 + 500.0 * np.random.rand(len(x_in))

%matplotlib notebook
plt.figure()
plt.scatter(x_in, y_out)
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

```python
"""Train using the custom Ridge class"""
x_train, x_test, y_train, y_test = train_test_split(
    x_in, y_out, test_size=0.20)

lam = 0.1
ridge = Ridge(lam)
ridge.fit(x_train, y_train)
```

```python
"""Train using Sklearn ridge model"""

from sklearn import linear_model

reg = linear_model.Ridge(alpha=lam)
reg.fit([np.array([1.0, x, x**2]) for x in x_train], y_train)
```

```python
"""Plot test data and model predictions"""

plt.figure()
plt.scatter(x_test, y_test)
x_test_sorted = np.sort(x_test)
plt.plot(x_test_sorted,
         ridge._coeff_hat[0] + ridge._coeff_hat[1]*x_test_sorted + ridge._coeff_hat[2]*x_test_sorted**2,
         '-r', label='custom')
plt.plot(x_test_sorted,
         reg.intercept_ + reg.coef_[1]*x_test_sorted + reg.coef_[2]*x_test_sorted**2, '--g', label='sklearn')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()

# print the coeff
print(f'custom: {ridge._coeff_hat[0]}, {ridge._coeff_hat[1]}, {ridge._coeff_hat[2]}')
print(f'sklearn: {reg.intercept_ }, {reg.coef_[1]}, {reg.coef_[2]}')
```

```python
"""Effect of regularization factor"""

coeff_store = []
norm_store = []
factors = np.linspace(0.0, 1.0, 10)
for l in factors:
    ridge = Ridge(l)
    ridge.fit(x_train, y_train)

    coeff_store.append(ridge._coeff_hat)
    norm_store.append(np.linalg.norm(ridge._coeff_hat))
```

```python
plt.figure()
plt.subplot(411)
coeff_0 = [c[0] for c in coeff_store]
plt.plot(factors, coeff_0, 'or')
plt.ylabel('c0')
plt.subplot(412)
coeff_1 = [c[1] for c in coeff_store]
plt.plot(factors, coeff_1, 'og')
plt.ylabel('c1')
plt.subplot(413)
coeff_2 = [c[2] for c in coeff_store]
plt.plot(factors, coeff_2, 'ob')
plt.ylabel('c2')
plt.subplot(414)
plt.plot(factors, norm_store, 'o')
plt.xlabel('lambda')
plt.ylabel('Norm')
plt.show()
```

```python

```


### ¿Cómo funciona la suspensión de un auto?
> Una primer aproximación al modelo de la suspensión de un automovil es considerar el oscilador armónico amortiguado. El cual se representa a través de la siguiente ecuación diferencial.

\begin{equation}
m\ddot{x} + k x + B \dot{x} = 0
\end{equation}
donde $k$ es la constante del muelle, y $B$ la constante de amortiguación.

Referencia:
   - https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html
   - https://docs.scipy.org/doc/scipy/reference/index.html

___
<div>

 </div>

Esta es una ecuación diferencial ordinaria (EDO). En python existe una función llamada _odeint_ del paquete _integrate_ de la libreria _scipy_, que permite integrar sistemas del tipo
\begin{equation}
\frac{dy}{dt} = f(x,y)
\end{equation}
con condiciones iniciales $y(0) = y_{0}$. Ahora bien, si nos fijamos bien, la ecuación diferencial que tenemos es de segundo orden. No hay problema. La podemos simplificar como un sistema de ecuaciones de primer orden como sigue:

\begin{align}
\dot{x} & = y \\
\dot{y} & = -\frac{k}{m} x - \frac{B}{m} y
\end{align}

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint

%matplotlib inline
```

```python
from ipywidgets import *
import matplotlib as mpl
label_size = 14
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size
```

```python

```

```python

```

```python

plt.legend(loc = 'best', prop={'size': 14})
plt.xlabel('tiempo', fontsize = 14)
plt.show()
```

```python

```

```python
plt.figure(figsize = (6,6))

plt.show()
```

```python
def amortiguado(t = 0):
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)

    ax.text(4, .6, 'tiempo =    %s'%tiempo[t])
    ax.text(4, .5, 'posición =  %s'%xx[t])
    fig.canvas.draw()

t_f = len(tiempo)

```

### Casos

Teníamos
\begin{equation}
m\ddot{x} + k x + B \dot{x} = 0
\end{equation}

si recordamos que $\omega_0 ^2 = k/m$ y definimos $B/m\equiv 2\Gamma$, tendremos

\begin{equation}
\ddot{x} + 2\Gamma \dot{x}+ \omega_0^2 x = 0
\end{equation}

### Amortiguado

Si $\omega_0^2 > \Gamma^2$ se tiene movimiento oscilatorio amortiguado.

```python
omega0 = k/m
Gamma = B/(2*m)
```

```python
omega0**2, Gamma**2
```

```python
omega0**2 > Gamma**2
```

Entonces, el primer caso que ya habíamos presentado corresponde a movimiento amortiguado.

```python

plt.legend(loc = 'best', prop={'size': 14})
plt.xlabel('tiempo', fontsize = 14)
plt.show()
```

```python

```

### Sobreamortiguado

Si $\omega_0^2 < \Gamma^2$ se tiene movimiento oscilatorio amortiguado.

```python
k = .1 # Constante del muelle
m = 1.0   # Masa
B = .5 # Constante de amortiguación
```

```python
omega0 = k/m
Gamma = B/(2*m)
```

```python
omega0**2, Gamma**2
```

```python
omega0**2 < Gamma**2
```

```python

```

```python

```

```python

plt.legend(loc = 'best', prop={'size': 14})
plt.xlabel('tiempo', fontsize = 14)
plt.show()
```

```python
plt.plot(xxA,yyA, c = 'r', label="Posicion")

plt.legend(loc = 'best', prop={'size': 14})
plt.xlabel('tiempo', fontsize = 14)
plt.show()
```

### Amortiguamiento crítico

Si $\omega_0^2 = \Gamma^2$ se tiene movimiento aperiódico crítico (amortiguamiento crítico).

```python
k = np.sqrt(.0625) # Constante del muelle
m = 1.0   # Masa
B = .5 # Constante de amortiguación
```

```python
omega0 = k/m
Gamma = B/(2*m)
```

```python
omega0**2, Gamma**2
```

```python
omega0**2 == Gamma**2
```

```python

```

```python

```

```python

plt.legend(loc = 'best',prop={'size': 14})
plt.xlabel('tiempo', fontsize = 14)
plt.show()
```

```python

```

En resumen, se tiene entonces:

```python

```

> **Actividad**. ¿Cómo se ve el espacio fase para los diferentes casos así como para diferentes condiciones iniciales?

```python

```

```python

```