Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your name and collaborators below:

```python
NAME = ""
ID = ""
```

---

# Linear Regression

In this lab, we'll take a look at how to build and evaluate linear regression models. Linear regression works well when there is an (approximately) linear relationship between the features and the variable we're trying to predict.

Before we start, let's import the Python packages we'll need for the tutorial:

```python
import matplotlib.pyplot as plt
import numpy as np
```

## Univariate example
Here's an example from [Tim Niven's tutorial at Kaggle](https://www.kaggle.com/timniven/linear-regression-tutorial).

### Background
We would like to perform *univariate* linear regression using a single feature $x$, "Number of hours studied," to predict a single dependent variable, $y$, "Exam score."

We can say that we want to regress num_hours_studied onto exam_score in order to obtain a model to predict a student's exam score using the number of hours he or she studied.

In the standard setting, we assume that the dependent variable (the exam score) is a random variable that has a Gaussian distribution whose mean is a linear function of the independent variable(s) (the number of hours studied) and whose variance is unknown but constant:

\begin{equation}
y\sim\mathcal{N}(\theta_0+\theta_1x,\sigma^2)
\end{equation}

Our model or hypothesis, then, will be a function predicting $y$ based on $x$:
\begin{equation}
h_\theta(x)=\theta_0+\theta_1x
\end{equation}

Next we'll do something very typical in machine learning experiment: generate some synthetic data for which we know the "correct" model, then use those data to test our algorithm for finding the best model.

So let's generate some example data and examine the relationship between $x$ and $y$:

```python
# Independent variable
num_hours_studied = np.array([1, 3, 3, 4, 5, 6, 7, 7, 8, 8, 10])

# Dependent variable
exam_score = np.array([18, 26, 31, 40, 55, 62, 71, 70, 75, 85, 97])

# Plot the data
plt.scatter(num_hours_studied, exam_score)
plt.xlabel('num_hours_studied')
plt.ylabel('exam_score')
plt.title('Synthetic Exam Score Data Set')
plt.show()
```

### Design Matrix
The design matrix, usually written $\mathtt{X}$, contains our independent variables.

In general, with $m$ data points and $n$ features (independent variables), our design matrix will have $m$ rows and $n$ columns.

Note that we have a parameter $\theta_0$, which is the $y$-intercept term in our linear model. There is no independent variable to multiple $\theta_0$, so we will introduced a dummy variable always equal to 1 to represent the independent variable corresponding to $\theta_0$.

Putting the dummy variable and the number of hours studied together, we obtain the design matrix

\begin{equation}
\mathtt{X} = \begin{bmatrix}
    1 & 1\\
    1 & 3\\
    1 & 3\\
    1 & 4\\
    1 & 5\\
    1 & 6\\
    1 & 7\\
    1 & 7\\
    1 & 8\\
    1 & 8\\
    1 & 10\\
\end{bmatrix}
\end{equation} \
Notice that we do **not** include the dependent variable (exam score) in the design matrix.

```python
# Add dummy variable for intercept term to design matrix.
# Understand the numpy insert function by reading https://numpy.org/doc/stable/reference/generated/numpy.insert.html

X = np.array([num_hours_studied]).T
X = np.insert(X, 0, 1, axis=1)
y = exam_score
print(X.shape)
print(y.shape)
```

    (11, 2)
    (11,)

### Hypothesis

Let's rewrite the hypothesis function now that we have a dummy variable for the intercept term in the model. We can write the independent variables including the dummy variable as a vector

$$\mathbf{x} = \begin{bmatrix} x_0 \\ x_1 \end{bmatrix}, $$

where $x_0 = 1$ is our dummy variable and $x_1$ is the number of hours studied. We also write the parameters as a vector

$$\mathbf{\theta} = \begin{bmatrix} \theta_0 \\ \theta_1 \end{bmatrix} .$$

Now we can conveniently write the hypothesis as

$$ h_\mathbf{\theta}(\mathbf{x}) = \mathbf{\theta}^\top \mathbf{x} . $$

### Exercise 1 (2 points)

Write a Python code function to evaluate a hypothesis $\mathbf{\theta}$ for an entire design matrix:

**Hint**: Use numpy function of dot

```python
# Evaluate hypothesis over a design matrix

def h(X,theta):
    # YOUR CODE HERE
    raise NotImplementedError()
    return y_predicted
```

```python
print(h(X, np.array([0, 10])))
```

**Expected output**: [ 10,  30,  30,  40,  50,  60,  70,  70,  80,  80, 100]

### Cost function
How can we find the best value of $\mathbf{\theta}$? We need a cost function and an algorithm to minimize that cost function.

In a regression problem, we normally use squared error to measure the goodness of fit:

$$\begin{align}
\ J(\mathbf{\theta}) & = \frac{1}{2} \sum_{i=1}^{m}\left(h_\mathbf{\theta}\left(\mathbf{x}^{(i)}\right) - y^{(i)}\right)^2 \\
\                    & = \frac{1}{2} \left( \mathtt{X} \mathbf{\theta} - \mathbf{y} \right)^\top \left( \mathtt{X} \mathbf{\theta} - \mathbf{y} \right)
\end{align}$$
Here we've used $\mathtt{X}$ to denote the design matrix and $\mathbf{y}$ to denote the vector$$\begin{bmatrix} y_1 \\ \vdots \\ y_m \end{bmatrix} $$

We'll see in a moment how to minimize this cost function.

### Exercise 2 (2 points)

Let's implement **cost function** in Python by these steps:

 1. Calculate $dy = \hat{y} - y = \mathtt{X}\theta - y$
 2. Calcuate $cost = \frac{1}{2}{dy}^T{dy}$

```python
m = y.shape[0]

def cost(theta, X, y):
    # YOUR CODE HERE
    raise NotImplementedError()
    return J
```

```python
print(cost(np.array([0, 10]), X, y))
```

**Expected output**: 85.0

### Aside: minimizing a convex function using the gradient

To solve our linear regression problem, we want to minimize the cost function $J(\mathbf{\theta})$ above with respect to the parameters $\mathbf{\theta}$.

$J$ is convex (see [Wikipedia](https://en.wikipedia.org/wiki/Convex_function) for an explanation) so it has just one minimum for some specific value of $\mathbf{\theta}$.

To find this minimum, we will find the point at which the gradient is equal to the zero vector.

The gradient of a multivariate function at a particular point is a vector pointing in the direction of maximum slope with a magnitude indicating the slope of the tangent at that point.

To make this clear, let's consider an example in which we consider the function $f(x) = 4x^2 - 6x + 11$ on the interval $[-10, 10]$ and plot its tangent lines at regular intervals.

```python
# Define range for plotting x
x = np.arange(-10, 10, 1)

# Example function f(x)
def f(x):
    return 4 * x * x - 6 * x + 11

# Plot f(x)
plt.plot(x, f(x), 'g')

# First derivative of f(x)
def dfx(x):
    return 8 * x - 6

# Plot tangent lines for f(x)
for i in np.arange(-10,10,3):
    x_i = np.arange(i - 1.0, i + 1.0, .25)
    m_i = dfx(i)
    c =  f(i) - m_i*i
    y_i = m_i*(x_i)  +  c
    plt.plot(x_i,y_i,'b')

# Plot tangent line at the minimum of f(x)
minimum = 0.75

for i in [minimum]:
    x_i = np.arange(i - 1, i + 1, .5)
    m_i = dfx(i)
    c = f(i) - m_i * i
    y_i = m_i * (x_i) + c
    plt.plot(x_i, y_i, 'r-', label='Local minimum')

# Decorate the plot
plt.xlabel('x')
plt.ylabel('f(x)')
plt.title('Minimization example')
plt.grid(axis='both',color='c', alpha=0.25)
plt.legend();
plt.show()
```

### Minimizing the cost function

Based on the previous example, we can see that to minimize our cost function, we just need to take the gradient with respect to $\mathbf{\theta}$ and determine where that gradient is equal to $\mathbf{0}$.

We have$$ J(\mathbf{\theta}) = \frac{1}{2} \sum_{i=1}^{m} \left(h_\mathbf{\theta}(\mathbf{x}^{(i)}) - y^{(i)}\right)^2 .$$This is a convex function of two variables ($\theta_0$ and $\theta_1$), so it has a single minimum where the gradient $\nabla_J(\mathbf{\theta})$ is $\mathbf{0}$.

Depending on the specific data, the cost function will look something like the surface plotted by the following code. Regardless of where we begin, the gradient always points "uphill," away from the global minimum.

```python
# Plot a sample 2D squared error cost function

from mpl_toolkits.mplot3d import Axes3D

x1 = np.linspace(-5.0, 15.0, 100)
x2 = np.linspace(-12.0, 8.0, 100)
X1, X2 = np.meshgrid(x1, x2)
Y = (np.square(X1 - np.mean(X1)) + np.square(X2 - np.mean(X2))) + 10

fig = plt.figure()
ax = Axes3D(fig)
ax.set_xlabel('theta_0')
ax.set_ylabel('theta_1')
ax.set_zlabel('J')
ax.set_title('Sample cost function for linear regression')
cm = plt.cm.get_cmap('viridis')
ax.plot_surface(X1, X2, Y, cmap=cm)
ax.view_init(elev=25, azim=55)
plt.show()
```

Take a look at the lecture notes. If you obtain the partial derivatives of the cost function $J$ with respect to $\mathbf{\theta}$, you get

$$ \nabla_J(\mathbf{\theta}) = \mathtt{X}^\top (\mathtt{X}\mathbf{\theta}-\mathbf{y}).$$

### Exercise 3 (2 points)

Write the gradient calculation in the equation above as a Python function:

```python
# Gradient of cost function

def gradient(X, y, theta):
    # YOUR CODE HERE
    raise NotImplementedError()
    return grad
```

```python
print(gradient(X, y, np.array([0, 10])))
```

**Expected output**: [-10, -13]

This means that if we currently had the parameter vector [0, 10] (where the cost is 85) and wanted to increase the cost, we could move in the direction [-10, -13]. On the other hand, if we wanted to decrease the cost (which of course we do), we should move in the opposite direction, i.e., [10, 13].

### Exercise 4 (2 points)

Implement this idea of gradient descent:

1. Calculate gradient from $X$, $y$ and $\theta$ using function gradient
2. Update $\theta_{new} = \theta + {\alpha}*grad$

```python
def gradient_descent(X, y, theta_initial, alpha, num_iters):
    J_per_iter = np.zeros(num_iters)
    gradient_per_iter = np.zeros((num_iters,len(theta_initial)))
    # initialize theta
    theta = theta_initial
    for iter in np.arange(num_iters):
        # YOUR CODE HERE
        raise NotImplementedError()
        J_per_iter[iter] = cost(theta, X, y)
        gradient_per_iter[iter] = grad.T
    return (theta, J_per_iter, gradient_per_iter)
```

```python
(theta, J_per_iter, gradient_per_iter) = gradient_descent(X, y, np.array([0, 10]), 0.001, 10)
print("theta:", theta)
print("J_per_iter:", J_per_iter)
print("gradient_per_iter", gradient_per_iter)
```

**Expected output**: \
theta: [ 0.08327017 10.02116759]\
J_per_iter: [84.775269   84.65958757 84.5793525  84.51074587 84.44605981 84.38279953\
 84.32015717 84.25787073 84.19585485 84.13408132]\
gradient_per_iter [[-10.         -13.        ]\
 [ -9.084       -6.894     ]\
 [ -8.556648    -3.421524  ]\
 [ -8.25039038  -1.4471287 ]\
 [ -8.06991411  -0.32491618]\
 [ -7.96100025   0.31253312]\
 [ -7.8928063    0.67422616]\
 [ -7.84778746   0.87905671]\
 [ -7.81596331   0.9946576 ]\
 [ -7.79165648   1.05950182]]

```python
# Optimize parameters theta on dataset X, y

theta_initial = np.array([0, 0])
alpha = 0.0001
iterations = 3000
theta, costs, grad = gradient_descent(X, y, theta_initial, alpha, iterations)
print('Optimal parameters: theta_0 %f theta_1 %f' % (theta[0], theta[1]))
```

```python
# Visualize the results

plt.scatter(num_hours_studied, exam_score)

x = np.linspace(0,10,20)
y_predicted = theta[0] + theta[1] * x
plt.plot(x, y_predicted, 'g', label='Prediction')

plt.xlabel('num_hours_studied')
plt.ylabel('exam_score')
plt.legend();
plt.title('Linear regression result')
plt.show()
```

```python
# Visualize the loss

x_loss = np.arange(0, iterations, 1)

plt.plot(x_loss, costs, 'b-')
plt.xlabel('iterations')
plt.ylabel('loss')
plt.show()
```

## Excercise 5 (2 points)

Instead of repeating the code to plot the loss graph, we would like to encapsulate the code in a function.
Complete the loss plotting function below:

```python
def cost_plot(iterations, costs):
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
cost_plot(iterations, costs)
```

We can conclude from the loss curve that we have achieved convergence (the loss has stopped improving), and we can conclude that 3000 iterations is overkill! The loss is stable after 100 iterations or so.

### Goodness of fit
$R^2$ is a statistic that will give some information about the goodness of fit of a regression model. The $R^2$ coefficient of determination is 1 when the regression predictions perfectly fit the data. When $R^2$ is less than 1, it indicates the percentage of the variance in the target that is accounted for by the prediction.

$$\begin{align}
\ R^2 = 1 - \frac{\sum_{i=1}^{m} \left( y^{\left(i\right)}-\hat{y}^\left(i\right) \right)^2}
{\sum_{i=1}^{m} \left( y^{\left(i\right)}-\bar{y}^\left(i\right) \right)^2}
\end{align}$$

### Exercise 6 (2 points)

Complete the `goodnees_of_fit` function implementing the equation for $R^2$ above:

```python
def goodness_of_fit(y, y_predicted):
    # YOUR CODE HERE
    raise NotImplementedError()
    return r_square
```

```python
y_predicted =  h(X, theta)
r_square = goodness_of_fit(y, y_predicted)
print(r_square)
```

**Expected output**: 0.9786239731773175

An $R^2$ of 0.98 indicates an extremely good (outrageously good, in fact) fit to the data.

## Multivariate linear regression example

Next, we extend our model to multiple variables. We'll use a data set from Andrew Ng's class. The data include two independent variables,
"Square Feet" and "Number of Bedrooms," and the dependent variable is "Price."

Let's load the data:

```python
# We use numpy's genfromtxt function to load the data from the text file.

raw_data = np.genfromtxt('Housing_data.txt',delimiter = ',', dtype=str);

raw_data
```

Next, we split the raw data (currently strings) into headers and the data themselves:

```python
# Extract headers and data
headers = raw_data[0,:];
print(headers)
data = np.array(raw_data[1:,:], dtype=float);
print(data)
```

```python
# Visualise the distribution of independent and dependent variables

# Make three subplots, in one row and three columns
fig, ax = plt.subplots(1,3)
fig.set_figheight(5)
fig.set_figwidth(20)
fig.subplots_adjust(left=.2, bottom=None, right=None, top=None, wspace=.2, hspace=.2)
plt1 = plt.subplot(1,3,1)
plt2 = plt.subplot(1,3,2)
plt3 = plt.subplot(1,3,3)

# Variable 1: square footage
plt1.hist(data[:,0], label='Sq. feet', edgecolor='black')
plt1.set_title('House Size')
plt1.set_xlabel('units')
plt1.set_ylabel('Frequency')
plt1.grid(axis='both', alpha=.25)

# Variable 2: number of bedrooms
plt2.hist(data[:,1], label='Bedroom', edgecolor='black')
plt2.set_title('Bedrooms')
plt2.set_xlabel('units')
plt2.set_ylabel('Frequency')
plt2.grid(axis='both', alpha=.25)

# Variable 3: home price
plt3.hist(data[:,2], label='Price', edgecolor='black')
plt3.set_title('Price')
plt3.set_xlabel('units')
plt3.set_ylabel('Frequency')
plt3.grid(axis='both', alpha=.25)
```

### Normalization

We can see from the charts above that the independent variables and the dependent variables have very large differences in their ranges. If you try to use the gradient descent method on these data directly, you may have difficulty in finding a learning rate that is small enough that the costs will not grow out of control but is large enough that the number of iterations is not excessive.

Normalization of the independent and dependent variables can help with this.
One type of normalization, sometimes called "standardization" or "z-scaling,"
involves subtracting a variable's mean then dividing by its standard deviation,
calculated over the training samples. The result is a set of standardized variables,
each with a mean of 0 and a variance of 1 over the training set.

```python
# Normalize the data

means = np.mean(data, axis=0)
stds = np.std(data, axis=0)
data_norm = (data - means) / stds
```

```python
# Extract y from the normalized dataset

y_label = 'Price'
y_index = np.where(headers == y_label)[0][0]
y = np.array([data_norm[:,y_index]]).T

# Extract X from normalized dataset

X = data_norm[:,0:y_index]

# Insert column of 1's for intercept term

X = np.insert(X, 0, 1, axis=1)
```

```python
# Get number of examples (m) and number of parameters (n)
m = X.shape[0]
n = X.shape[1]
print(m, n)
```

### Excercise 7 (5 points)

Optimize the parameters using gradient descent:

```python
theta_initial = np.zeros((X.shape[1],1))
alpha = 0.01
iterations = 1000
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print('Theta values ', theta)
```

**Expected output**:\
Theta values  [[-9.15933995e-17]\
 [ 8.84765988e-01]\
 [-5.31788197e-02]]

```python
# Visualize the loss over the optimization
plt.title('Multivariate linear regression by gradient descent')
cost_plot(iterations, costs)
```

Transforming parameters back to the original scale
Now that we've got optimal parameters for our original data, we need to undo the normalization.

We have

$$\hat{y}^{\text{norm}} = \theta^\text{norm} \textbf{x}^\text{norm}$$

## Excercise 8 (3 points)

Modify the code to compute goodness of fit

```python
# Goodness of fit
y_predicted = h(X,theta)
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print(r_square)
```

### Transform standardized data back to original scale
We can transform standardized predicted values, y_predicted into the orginal data scale using$$y_{\text{norm}} = \sigma_y y + \mu_y$$

```python
# Compute mean and standard deviation of data

sigma = np.array(np.std(data,axis=0))
mu = np.array(np.mean(data,axis=0))

# De-normalize y

y_predicted =  np.round(h(X, theta) * sigma[2] + mu[2])

# Print first five values of y_predicted

print(y_predicted[0:5,:])
```

```python
# 3D plot of standardized data

from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = Axes3D(fig)
p = ax.scatter(X[:,1],X[:,2],y,edgecolors='black',c=data_norm[:,2],alpha=1)
ax.set_xlabel('x1')
ax.set_ylabel('x2')
ax.set_zlabel('y')

X1 = np.linspace(min(X[:,1]), max(X[:,1]), len(y))
X2 = np.linspace(min(X[:,2]), max(X[:,2]), len(y))

xx1,xx2 = np.meshgrid(X1,X2)

yy = (theta[0] + theta[1]*xx1.T + theta[2]*xx2)
ax.plot_surface(xx1,xx2,yy, alpha=0.5)
ax.view_init(elev=25, azim=10)
plt.colorbar(p)
plt.show()
```

## In-class exercises
Now that you're familiar with minimizing a cost function using its gradient and gradient descent, refer to the lecture notes to find the analytical solution (the normal equations) to the linear regression problem.

Implement the normal equation approach for the synthetic univariate data set and the housing price data set. Demonstrate your solution in the lab.

```python
# just remove all parameters
%reset
```

```python
import matplotlib.pyplot as plt
import numpy as np
```

### Exercise 2.1 (5 points)
Download raw_data and setup data

```python
# Download raw_data and setup data
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print(data[:5])
```

**Expected result**:\
[[2.104e+03 3.000e+00 3.999e+05]\
 [1.600e+03 3.000e+00 3.299e+05]\
 [2.400e+03 3.000e+00 3.690e+05]\
 [1.416e+03 2.000e+00 2.320e+05]\
 [3.000e+03 4.000e+00 5.399e+05]]

### Exercise 2.2 (5 points)
Normalized data

```python
# Normalized data
def normalized_data(data):
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
data_norm = normalized_data(data)
print(data_norm[:5])
```

**Expected result**:\
[[ 0.13141542 -0.22609337  0.48089023]\
 [-0.5096407  -0.22609337 -0.08498338]\
 [ 0.5079087  -0.22609337  0.23109745]\
 [-0.74367706 -1.5543919  -0.87639804]\
 [ 1.27107075  1.10220517  1.61263744]]

### Exercise 2.3 (5 points)
Extract X and y from data

```python
# Extract y from data
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print(y[:5])
```

**Expected result**: [ 0.48089023 -0.08498338  0.23109745 -0.87639804  1.61263744]

```python
# Extract X from data
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print(X[:5,:])
```

**Expected result**:\
[[ 1.          0.13141542 -0.22609337]\
 [ 1.         -0.5096407  -0.22609337]\
 [ 1.          0.5079087  -0.22609337]\
 [ 1.         -0.74367706 -1.5543919 ]\
 [ 1.          1.27107075  1.10220517]]

### Exercise 2.4 (8 points)
Create h, cost, gradient, and gradient_descent

```python
# create h function
def h(X,theta):
    # YOUR CODE HERE
    raise NotImplementedError()
    return y_predicted
```

```python
print(h(X, np.array([1, 2, 4]))[:5])
```

**Expected result**: [ 0.35845737 -0.92365487  1.11144393 -6.70492173  7.95096216]

```python
def cost(theta, X, y):
    # YOUR CODE HERE
    raise NotImplementedError()
    return J
```

```python
print(cost(np.array([1, 8, 10]), X, y))
```

**Expected result**: 5477.138628374691

```python
# Gradient of cost function
def gradient(X, y, theta):
    # YOUR CODE HERE
    raise NotImplementedError()
    return grad
```

```python
print(gradient(X, y, np.array([1, 8, 10])))
```

**Expected result**: [ 47.         599.00016917 659.76139633]

```python
def gradient_descent(X, y, theta_initial, alpha, num_iters):
    J_per_iter = np.zeros(num_iters)
    gradient_per_iter = np.zeros((num_iters,len(theta_initial)))
    # initialize theta
    theta = theta_initial
    for iter in np.arange(num_iters):
        # YOUR CODE HERE
        raise NotImplementedError()
        J_per_iter[iter] = cost(theta, X, y)
        gradient_per_iter[iter] = grad.T
    return (theta, J_per_iter, gradient_per_iter)
```

```python
(theta, J_per_iter, gradient_per_iter) = gradient_descent(X, y, np.array([0, 1, 10]), 0.001, 10)
print("theta:", theta)
print("J_per_iter:", J_per_iter)
print("gradient_per_iter", gradient_per_iter)
```

**Expected result**:
theta: [-8.20787882e-16 -7.72838948e-01  6.35294636e+00]\
J_per_iter: [2123.51284628 1873.56259758 1656.90935568 1468.93187452 1305.65834104\
 1163.67477334 1040.04635308  932.24986509  838.11567544  755.77790087]\
gradient_per_iter [[1.31450406e-13 2.70000169e+02 4.75532186e+02]\
 [9.68114477e-14 2.44794887e+02 4.46076185e+02]\
 [9.63673585e-14 2.21549490e+02 4.18667980e+02]\
 [8.92619312e-14 2.00117968e+02 3.93159744e+02]\
 [1.11022302e-13 1.80365065e+02 3.69414440e+02]\
 [7.40518757e-14 1.62165488e+02 3.47305031e+02]\
 [5.05151476e-14 1.45403177e+02 3.26713748e+02]\
 [6.09512441e-14 1.29970626e+02 3.07531415e+02]\
 [6.29496455e-14 1.15768253e+02 2.89656812e+02]\
 [4.74065232e-14 1.02703825e+02 2.72996100e+02]]\

### Exercise 2.5 (5 points)

Do optimization using gradient descent with $\alpha = 0.003$ and 30,000 iterations

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
print("theta:", theta)
print("cost_per_iter:", costs[-5:])
print("gradient_per_iter", grad[-5:])
```

**Expected result**:\
theta: [-1.05832010e-16  8.84765988e-01 -5.31788197e-02]\
J_per_iter: [6.27579208 6.27579208 6.27579208 6.27579208 6.27579208]\
gradient_per_iter [[ 0.00000000e+00 -1.72082220e-14  8.75724041e-16]\
 [ 0.00000000e+00 -1.72082220e-14  8.75724041e-16]\
 [ 0.00000000e+00 -1.72082220e-14  8.75724041e-16]\
 [ 0.00000000e+00 -1.72082220e-14  8.75724041e-16]\
 [ 0.00000000e+00 -1.72082220e-14  8.75724041e-16]]

### Exercise 2.6 (2 points)

Calculate goodness of fit

```python
def goodness_of_fit(y, y_predicted):
    # YOUR CODE HERE
    raise NotImplementedError()
    return r_square
```

```python
y_predicted =  h(X, theta)
r_square = goodness_of_fit(y, y_predicted)
print(r_square)
```

**Expected result**: 0.7329450180289143

## Excercise 2.7 (2 point)

Plot graph of cost results

```python
def cost_plot(iterations, costs):
    # YOUR CODE HERE
    raise NotImplementedError()

cost_plot(iterations, costs)
```

### Exercise 2.8 (8 points)

Write a function implementing the normal equations for linear equation:

```python
# Function to use the normal equations to find the optimal
# parameters for a linear regression model

def normal_equation(X, y):
    # YOUR CODE HERE
    raise NotImplementedError()
    return theta
```

```python
theta_norm = normal_equation(X,np.array([y]).T)
print("theta from normal equation:", theta_norm.T)
y_norm_predicted =  h(X, theta_norm)
r_norm_square = goodness_of_fit(y, y_norm_predicted)
print("r_square:", r_norm_square)
```

**Expected result**:\
theta from normal equation: [[-7.90434550e-17  8.84765988e-01 -5.31788197e-02]]\
r_square: 0.7329450180289143

### Take-home exercise (40 points)
Find an interesting dataset for linear regression on Kaggle. Implement the normal equations and gradient descent then evaluate your model's performance.

Write a brief report on your experiments and results in the form of a Jupyter notebook.

Explain the dataset which you get and which rows which you use.
How many data in your dataset?

YOUR ANSWER HERE

Write down your all code at below.
Show the results, goodness of fit and plot cost graph

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python

```


# Import sympy parametric plot modules

```python
%reset -f
from sympy import *
from sympy.plotting import plot_parametric
from sympy.plotting import plot3d_parametric_line
import math

init_printing()        # for nice math output

# uncomment for plot in separate window
%matplotlib qt

# uncomment for plot inline --- DEFAULT
#%matplotlib inline
```

## Declare symbolic variables

```python
t = symbols('t')
```

## Example: Plot the polar curve $r = 2\cos(t)$

```python
# 2d - Example 1
r = 2*cos(t)

%matplotlib inline
plot_parametric( r*cos(t), r*sin(t), (t, 0, 2*pi))
```

## Example: Plot the polar curve $r = 1 + 2 \sin(t)$

```python
r = 1 + 2*sin(t)

%matplotlib inline
plot_parametric( r*cos(t), r*sin(t), (t, 0, 2*pi))
```

## Example: Plot the polar curve $r = 3 + 2\cos(t)$

```python
r = 3 + 2*cos(t)

%matplotlib inline
plot_parametric( r*cos(t), r*sin(t), (t, 0, 2*pi))
```

## Example: Plot the polar curve $r = \sin(3t)$

```python
r = sin(2*t)

%matplotlib inline
plot_parametric( r*cos(t), r*sin(t), (t, 0, 2*pi))
```

## Example: Plot the family of curve curve $r = a + b \sin(t)$ with slidable $a$ and $b$.

```python
%reset -f
%matplotlib qt

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.widgets import Slider

plt.subplots_adjust(bottom=0.25)
t = np.arange(0.0, 2*np.pi, 0.001)

a = 1
b = 1
r = a + b*np.sin(t)
x = r*np.cos(t)
y = r*np.sin(t)

l, = plt.plot(x,y, lw=4, color='black')
plt.axis([-3, 3, -3, 3])

axcolor = 'lightgoldenrodyellow'
axb  = plt.axes([0.25, 0.10, 0.65, 0.03], axisbg=axcolor)
axa = plt.axes([0.25, 0.15, 0.65, 0.03], axisbg=axcolor)

sa = Slider(axa, 'a', 0, 2, valinit=a)
sb = Slider(axb, 'b', 0, 2, valinit=b)

def update(val):
    a = sa.val
    b = sb.val

    r = a + b*np.sin(t)
    x = r*np.cos(t)
    y = r*np.sin(t)

    l.set_ydata(y)
    l.set_xdata(x)
    fig.canvas.draw_idle()

sa.on_changed(update)
sb.on_changed(update)
plt.title('r = a + b cos(t)')

plt.show()
```

## Example: Plot the family of curve curve $r =  \sin[a(t+b)]$ with slidable $a$ and $b$.

```python
%reset -f
%matplotlib qt

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.widgets import Slider

plt.subplots_adjust(bottom=0.25)
t = np.arange(0.0, 2*np.pi, 0.001)

a = 2
b = 0
r = np.cos(a*(t + b))
x = r*np.cos(t)
y = r*np.sin(t)

l, = plt.plot(x,y, lw=4, color='black')
plt.axis([-3, 3, -3, 3])

axcolor = 'lightgoldenrodyellow'
axb  = plt.axes([0.25, 0.10, 0.65, 0.03], axisbg=axcolor)
axa = plt.axes([0.25, 0.15, 0.65, 0.03], axisbg=axcolor)

sa = Slider(axa, 'a', 0, 10, valinit=a)
sb = Slider(axb, 'b', -np.pi, np.pi, valinit=b)

def update(val):
    a = (sa.val)
    b = sb.val

    r = np.cos(a*(t + b))
    x = r*np.cos(t)
    y = r*np.sin(t)

    l.set_ydata(y)
    l.set_xdata(x)
    fig.canvas.draw_idle()

sa.on_changed(update)
sb.on_changed(update)
plt.title('r = cos(a(t + b))')

plt.show()
```

```python

```


```python
import sympy
from sympy import *
init_printing()
from IPython.display import display
```

# Beam Segment Equations

Loading along a beam is not always mathematically continuous.  Concentrated forces and moments, as well as starting and ending points of distributed loads interupt the mathematical continuity of the beam.  For this reason, it is convenient to segment the beam between discontinuities to form a series of mathematically continuous segments. This allows for the direct calculation of maximum/minimum forces, rotations, and deflections in each segment.

## Linear Distributed Load Equation

The equation for a linear distributed load is expressed as follows:

```python
w, w1, w2, x, L = symbols('omega, omega_1, omega_2, x, L')
w = w1+(w2-w1)/L*x
display(Eq(symbols('\\omega(x)'), w))
```

Where:

$\omega_1=$ magnitude of distributed load at starting point

$\omega_2=$ magnitude of distributed laod at ending point

$x_1=$ distributed load starting point

$x_2=$ distributed load ending point

## Shear Force Equation
The change in the shear force is equal to the area under (or integral of) the load diagram. If the shear force at the start of the segment, $V_1$, is known, the shear force at any point, $x$, relative to the start of the segment can be expressed as:

```python
V1 = symbols('V_1')
V = V1 + integrate(w, (x, 0, x))
display(Eq(symbols('V(x)'), V1 + Integral(symbols('\\omega(x)'), (x, 0, x))))
display(Eq(symbols('V(x)'), V1 + Integral(w, (x, 0, x))))
display(Eq(symbols('V(x)'), V))
print(V)
```

## Moment Equation

Similarly, the change in the moment diagram is equal to the area under the shear diagram. If the moment at the start of the segment, $M_1$, is known, the moment at any point, $x$, relative to the start of the segment can be expressed as:

```python
M1 = symbols('M_1')
M = M1 + integrate(V, (x, 0, x))
display(Eq(symbols('M(x)'), M1 + Integral(symbols('V(x)'), (x, 0, x))))
display(Eq(symbols('M(x)'), M1 + Integral(V, (x, 0, x))))
display(Eq(symbols('M(x)'), M))
print(M)
```

## Slope Equation
The slope of the elastic curve, $\theta$, can be obtained by integrating the moment diagram and dividing by the flexural rigidity, $EI$.

$\theta_1=$ slope (or rotation) at the start of the segment

```python
theta1, E, I = symbols('theta_1, E, I')
theta = theta1 - 1/(E*I)*integrate(M, (x, 0, x))
display(Eq(symbols('\\theta(x)'), theta1 + Integral(symbols('M(x)'), (x, 0, x))))
display(Eq(symbols('\\theta(x)'), theta1 + Integral(M, (x, 0, x))/(E*I)))
display(Eq(symbols('\\theta(x)'), theta))
print(theta)
```

## Deflection Equation
Integrating one more time gives us the equation for the deflection along the segment.

$\Delta_1=$ Deflection at the start of the segment

```python
Delta1 = symbols('Delta_1')
Delta = Delta1 - integrate(theta, (x, 0, x))
display(Eq(symbols('\\Delta(x)'), Delta1 - Integral(symbols('\\theta(x)'), (x, 0, x))))
display(Eq(symbols('\\Delta(x)'), Delta1 - Integral(theta, (x, 0, x))))
display(Eq(symbols('\\Delta(x)'), Delta))
print(Delta)
```

## Using the Equations
Each of the equations derived above requires knowledge of the initial conditions at the start of the segment ($V_1$, $M_1$, $\theta_1$, or $\Delta_1$).  An analysis will need to be performed to determine these values at the start of the beam.

The initial conditions at the start of the beam are the same as those at the start of the first segment.  If the initial conditions at the start of the first segment of the beam are known, the conditions at the start of the second segment can be found by calculating the conditions at the end of the first segment (from the equations with $x=L$) and adding any concentrated loads that may be at the junction between segments.  Likewise, the initial conditions for the third segment can be derived from the second segment, and so on until the last segment.  Any concentrated loads at the end of the last segment (including reactions) will need to be added to get shear and moment diagrams to close.

## Calculating $\theta_1$

For finite element solutions, the rotation $\theta_1$ at the start of the beam might be known. However, if end releases have been applied to the member (e.g. members with hinges at one or both ends), then either $\theta_1$ or $\theta_2$ or both will not be known. These terms will be lost when the stiffness matrix is condensed to simulate the end releases. In this situation the slope-deflection equation may be used to solve for $\theta_1$ as follows (since $M_1$, $M_2$, $\Delta_1$, $\Delta_2$, $FEM_1$, & $FEM_2$ will be known). The basic slope-deflection equation is:

$M_{nf}=\frac{2EI}{L}(2\theta_n+\theta_f-3\psi)+FEM_{nf}$

Substituting in appropriate values for each end of the segment yields the following equations:

```python
M1, M2, E, I, L, theta1, theta2, Delta1, Delta2 = symbols('M_1, M_2, E, I, L, theta_1, theta_2, Delta_1, Delta_2')
FEM1, FEM2 = symbols('FEM_1, FEM_2')
M1 = 2*E*I/L*(2*theta1 + theta2 - 3*(Delta2-Delta1)/L) + FEM1
M2 = 2*E*I/L*(2*theta2 + theta1 - 3*(Delta2-Delta1)/L) + FEM2
display(Eq(symbols('M_1'), M1))
display(Eq(symbols('M_2'), M2))
```

Solving the equation $M_2$ slope-deflection equation for $\theta_2$ gives:

```python
theta2 = solve(M2 - symbols('M_2'), theta2)[0]
display(Eq(symbols('\\theta_2'), expand(theta2)))
```

Substituting $\theta_2$ back into the slope-deflection equation for $M_1$ gives:

```python
M1 = M1.subs('theta_2', theta2)
display(Eq(symbols('M_1'), expand(M1)))
```

Solving this new equation for $\theta_1$ gives:

```python
theta1 = solve(M1 - symbols('M_1'), theta1)[0]
display(Eq(symbols('\\theta_1'), expand(theta1)))
print(theta1)
```

## Finding Max and Min Values

The points of max/min shear can be found by setting the derivative of the equation for shear equal to zero. Ther derivative of the shear force is:

```python
display(Eq(Derivative(symbols('V'), x), diff(V, x)))
display(Eq(symbols('x'), solve(diff(V, x), symbols('x'))[0]))
print(solve(diff(V, x), symbols('x'))[0])
```

The points of max/min moment can be found by setting the derivative of the equation for moment equal to zero. The derivative of the moment is:

```python
display(Eq(Derivative(symbols('M'), x), diff(M, x)))
display(Eq(symbols('x_1'), solve(diff(M, x), symbols('x'))[0]))
display(Eq(symbols('x_2'), solve(diff(M, x), symbols('x'))[1]))
print(solve(diff(M, x), symbols('x'))[0])
print(solve(diff(M, x), symbols('x'))[1])
```

```python

```

Derivations/Beam Segment Equations (y-Axis Bending).ipynb


# Neural Network - with multiple inputs and neurons in hidden layer

In this note, I am going to train a neural network model with `gradient decent` estimation. This model will only have 2 direct inputs and 1 hidden layer with 3 neurons and logistic transform. The output layer will be activated with softmax.

# Overview

This note will covere:
* Prepare the data
* Model and Cost function
* Backward propagation
* Momentum
* Code implementation

# Parepare the data

The dataset is generated by the scikit-learn make_circles method. It will contain 2 labels, blue $(t=1)$ and red $(t=0)$, where the red class is a circular distribution that surrounds the distribution of the blue class.

```python
# Python imports
import numpy as np  # Matrix and vector computation package
import sklearn.datasets # To generate the dataset
import matplotlib.pyplot as plt  # Plotting library
from matplotlib.colors import colorConverter, ListedColormap  # Some plotting functions
from mpl_toolkits.mplot3d import Axes3D  # 3D plots
from matplotlib import cm  # Colormaps
# Allow matplotlib to plot inside this notebook
%matplotlib inline

# Set the seed of the numpy random number generator so that the tutorial is reproducable
np.random.seed(seed=1)
# Generate the dataset
X, t = sklearn.datasets.make_circles(n_samples=100, shuffle=False, factor=0.4, noise=0.1)
T = np.zeros((100,2)) # Define target matrix
T[t==1,1] = 1
T[t==0,0] = 1
# Separate the red and blue points for plotting
x_red = X[t==0]
x_blue = X[t==1]

print('shape of X: {}'.format(X.shape))
print('shape of T: {}'.format(T.shape))
```

    shape of X: (100, 2)
    shape of T: (100, 2)

As you can see the plot below, it's not linear separatable.

```python
# Plot both classes on the x1, x2 plane
plt.plot(x_red[:,0], x_red[:,1], 'ro', label='class red')
plt.plot(x_blue[:,0], x_blue[:,1], 'bo', label='class blue')
plt.grid()
plt.legend(loc=1)
plt.xlabel('$x_1$', fontsize=15)
plt.ylabel('$x_2$', fontsize=15)
plt.axis([-1.5, 1.5, -1.5, 1.5])
plt.title('red vs blue classes in the input space')
plt.show()
```

# Model and Cost Function

Model can be visualized as below:
<p align="center">
</p>

### Input and Label
So for the input layer, we have 2 dimention inputs from N data points (Nx2):
$$X =
\begin{bmatrix}
x_{11} & x_{12} \\
\vdots & \vdots \\
x_{N1} & x_{N2}
\end{bmatrix}$$
And for output, we have 2 class (Nx2):
$$T = \begin{bmatrix}
t_{11} & t_{12}\\
\vdots & \vdots \\
t_{N1} & t_{N2}
\end{bmatrix}$$

Where $t_{ij}=1$ if and only if the $i$-th input sample belongs to class $j$. So blue points are labelled T = [0 1] and red points are labelled T = [1 0].

Note in the model we added a bias term $b$ to $X$ that has value +1 for all rows.

### Hidden Layer

The hidden layer has two set of parameters, $W_h$ (2x3) and bias term $b_h$ (1x3):

$$\begin{align}
W_h =
\begin{bmatrix}
w_{h11} & w_{h12} & w_{h13} \\
w_{h21} & w_{h22} & w_{h23}
\end{bmatrix}
&& \mathbf{b}_h =
\begin{bmatrix}
b_{h1} & b_{h2} & b_{h3}
\end{bmatrix}
\end{align}$$

And we use logistic function for activation, hence $H$ (Nx3):

$$H = \sigma(z_h) =\sigma(X \cdot W_h + \mathbf{b}_h) = \frac{1}{1+e^{-(X \cdot W_h + \mathbf{b}_h)}} = \begin{bmatrix}
h_{11} & h_{12} & h_{13} \\
\vdots & \vdots & \vdots \\
h_{N1} & h_{N2} & h_{N3}
\end{bmatrix}$$

#### Derivative of Logistic function

we will use the derivative of Logistic function in backward propagation. So writing down here in advance.

**(1)**
$$\frac{\partial h_i}{\partial z_{hi}} = \frac{\partial \sigma(z_{hi})}{\partial z_{hi}} = \frac{\partial \frac{1}{1+e^{-z_{hi}}}}{\partial z_{hi}} = \frac{-1}{(1+e^{-z_{hi}})^2} *e^{-z_{hi}}*-1 = \frac{1}{1+e^{-z_{hi}}} \frac{e^{-z_{hi}}}{1+e^{-z_{hi}}} = \sigma(z_{hi}) * (1- \sigma(z_{hi})) =  h_i (1-h_i)$$

### Output layer

The output layer has two set of parameters, $W_o $ (3x2) and bias term $b_o$ (2x1):

$$\begin{align}
W_o =
\begin{bmatrix}
w_{o11} & w_{o12} \\
w_{o21} & w_{o22} \\
w_{o31} & w_{o32}
\end{bmatrix}
&& \mathbf{b}_o =
\begin{bmatrix}
b_{o1} & b_{o2}
\end{bmatrix}
\end{align}$$

And we use `softmax function` for activation, hence $Y$ (Nx2):

$$Y = \varsigma(H \cdot W_o + \mathbf{b}_o)
= \frac{e^{Z_o}}{\sum_{d=1}^C e^{\mathbf{z}_{od}}}
= \frac{e^{H \cdot W_o + \mathbf{b}_o}}{\sum_{d=1}^C e^{H \cdot \mathbf{w}_{od} + b_{od}}}
= \begin{bmatrix}
y_{11} & y_{12}\\
\vdots & \vdots \\
y_{n1} & y_{n2}
\end{bmatrix}$$

with $\varsigma$ the softmax function. In our example data, $C=2$ as we have 2 class red and blue.

#### Derivative of Softmax function

we will use the derivative of Softmax function in backward propagation. So writing down here in advance.

For $\Sigma_C = \sum_{d=1}^C e^{z_d} \, \text{for} \; c = 1 \cdots C$ so that $y_c = e^{z_c} / \Sigma_C$, ${\partial y_i}/{\partial z_j}$ of the output $y$ of the softmax function with respect to its input $z$ can be calculated as:

**(2)**
$$
\begin{split}
\text{if} \; i = j :& \frac{\partial y_i}{\partial z_i} = \frac{\partial \frac{e^{z_i}}{\Sigma_C}}{\partial z_i} = \frac{e^{z_i}\Sigma_C - e^{z_i}e^{z_i}}{\Sigma_C^2} = \frac{e^{z_i}}{\Sigma_C}\frac{\Sigma_C - e^{z_i}}{\Sigma_C} = \frac{e^{z_i}}{\Sigma_C}(1-\frac{e^{z_i}}{\Sigma_C}) =  y_i (1 - y_i)\\
\text{if} \; i \neq j :& \frac{\partial y_i}{\partial z_j} = \frac{\partial \frac{e^{z_i}}{\Sigma_C}}{\partial z_j} = \frac{0 - e^{z_i}e^{z_j}}{\Sigma_C^2} = -\frac{e^{z_i}}{\Sigma_C} \frac{e^{z_j}}{\Sigma_C} = -y_i y_j
\end{split}
$$

### Code for forward step

We can at first code the functions for forward step.

```python
# Define the logistic function. - for hidden layer activation.
def logistic(z):
    return 1 / (1 + np.exp(-z))

# Define the softmax function
def softmax(z):
    return np.exp(z) / np.sum(np.exp(z), axis=1, keepdims=True)

# Function to compute the hidden activations
def hidden_activations(X, Wh, bh):
    return logistic(X.dot(Wh) + bh)

# Define output layer feedforward
def output_activations(H, Wo, bo):
    return softmax(H.dot(Wo) + bo)

# Define the neural network function
def nn(X, Wh, bh, Wo, bo):
    return output_activations(hidden_activations(X, Wh, bh), Wo, bo)

# Define the neural network prediction function that only returns
#  1 or 0 depending on the predicted class
def nn_predict(X, Wh, bh, Wo, bo):
    return np.around(nn(X, Wh, bh, Wo, bo))
```

### Cost Function

The parameter set $w$ can be optimized  by maximizing the `likelihood`:

$$\underset{\theta}{\text{argmax}}\; \mathcal{L}(\theta|\mathbf{t},\mathbf{z})$$

The likelihood can be described as join distribution of $t\;and\;z\;$given $\theta$:

$$P(\mathbf{t},\mathbf{z}|\theta) = P(\mathbf{t}|\mathbf{z},\theta)P(\mathbf{z}|\theta)$$

We don't care the probability of $\mathbf{z}$ so

$$\mathcal{L}(\theta|\mathbf{t},\mathbf{z}) = P(\mathbf{t}|\mathbf{z},\theta)$$

It can be written as $P(\mathbf{t}|\mathbf{z})$ for fixed $\theta$. Since each $t_i$ is dependent on the full $\mathbf{z}$, and only 1 class can be activated in the tt we can write:

$$P(\mathbf{t}|\mathbf{z}) = \prod_{i=c}^{C} P(t_c|\mathbf{z})^{t_c} = \prod_{i=c}^{C} \varsigma(\mathbf{z})_c^{t_c} = \prod_{i=c}^{C} y_c^{t_c}$$

Instead of maximizing this likelihood, it can also be done by minimizing the `negative log-likelihood`:

$$- log \mathcal{L}(\theta|\mathbf{t},\mathbf{z}) = \xi(\mathbf{t},\mathbf{z}) = - log \prod_{i=c}^{C} y_c^{t_c} = - \sum_{i=c}^{C} t_c \cdot log(y_c)$$

The cross-entropy error function $\xi$ for multiple class of sample size $n$ can be defined as:

$$\xi(T,Y) = \sum_{i=1}^n \xi(\mathbf{t}_i,\mathbf{y}_i) = - \sum_{i=1}^n \sum_{i=c}^{C} t_{ic} \cdot log( y_{ic})$$

Note $Y$ will be activated by `softmax` in output layer as we defined above.

#### Derivative of the cross-entropy cost function for the softmax function

The derivative ${\partial \xi}/{\partial z_i}$ of the cost function with respect to the softmax input $z_i$ can be calculated as **(Note below use (2) the derivative of softmax function)**:

**(3)**
$$
\begin{split}
\frac{\partial \xi}{\partial z_i} & = - \sum_{j=1}^C \frac{\partial t_j log(y_j)}{\partial z_i}{} = - \sum_{j=1}^C t_j \frac{\partial log(y_j)}{\partial z_i} = - \sum_{j=1}^C t_j \frac{1}{y_j} \frac{\partial y_j}{\partial z_i} \\
& = - \frac{t_i}{y_i} \frac{\partial y_i}{\partial z_i} - \sum_{j \neq i}^C \frac{t_j}{y_j} \frac{\partial y_j}{\partial z_i}
= - \frac{t_i}{y_i} y_i (1-y_i) - \sum_{j \neq i}^C \frac{t_j}{y_j} (-y_j y_i) \\
& = - t_i + t_i y_i + \sum_{j \neq i}^C t_j y_i = - t_i + \sum_{j = 1}^C t_j y_i
= -t_i + y_i \sum_{j = 1}^C t_j \\
& = y_i - t_i
\end{split}
$$

# Backward propagation

During the backward step, what's important is to get the error gradient in each layer.

### Gradient of output layer

The error gradient $\delta_{o}$ of this cost function at the softmax output layer is simply (from **(3)**):

$$\delta_{o} = \frac{\partial \xi}{\partial Z_o} = Y - T$$

As $Z_o = H \cdot W_o + \mathbf{b}_o$, so the output grandient over all $N$ samples is computed :

$$\frac{\partial \xi}{\partial \mathbf{w}_{oj}} =
\frac{\partial Z_{o}}{\partial \mathbf{w}_{oj}} \frac{\partial Y}{\partial Z_{o}} \frac{\partial \xi}{\partial Y} =
\frac{\partial Z_{o}}{\partial w_{oji}} \frac{\partial \xi}{\partial Z_o} =
\sum_{i=1}^N h_{ij} (\mathbf{y}_i - \mathbf{t}_i) =
\sum_{i=1}^N h_{ij} \delta_{oi}$$

In matrix form:

$$\frac{\partial \xi}{\partial W_o} = H^T \cdot (Y-T) = H^T \cdot \delta_{o}$$

For bias term $b_o$:

$$\frac{\partial \xi}{\partial \mathbf{b}_{o}} = \frac{\partial Z_{o}}{\partial \mathbf{b}_{o}} \frac{\partial Y}{\partial Z_{o}} \frac{\partial \xi}{\partial Y} = \sum_{i=1}^N 1 * (\mathbf{y}_i - \mathbf{t}_i) = \sum_{i=1}^N \delta_{oi}$$

### Gradient of hidden layer

The error gradient $\delta_{h}$ of this cost function at the hidden layer can be defined as:

$$\delta_{h} = \frac{\partial \xi}{\partial Z_h} = \frac{\partial H}{\partial Z_h} \frac{\partial \xi}{\partial H} = \frac{\partial H}{\partial Z_h} \frac{\partial Z_o}{\partial H} \frac{\partial \xi}{\partial Z_o}$$

As $Z_h = X \cdot W_h + \mathbf{b}_h$, $\delta_{h}$  will also result in a $N×3$ matrix. We use **(1)** and **(2)** to get result below.
Note the gradients that backpropagate from the previous layer via the weighted connections are summed for each $h_{ij}$

$$\delta_{hij} = \frac{\partial \xi}{\partial z_{hij}}
= \frac{\partial h_{ij}}{\partial z_{hij}} \frac{\partial \mathbf{z}_{oi}}{\partial h_{ij}} \frac{\partial \xi}{\partial \mathbf{z}_{oi}}
= h_{ij} (1-h_{ij}) \sum_{k=1}^2 w_{ojk} (y_{ik}-t_{ik})
= h_{ij} (1-h_{ij}) [\delta_{oi} \cdot \mathbf{w}_{oj}^T]$$

In matrix form with notation $\circ$ for `elementwise product`:

$$\delta_{h} = \frac{\partial \xi}{\partial Z_h} = H \circ (1 - H) \circ [\delta_{o} \cdot W_o^T]$$

The hidden layer gradient then can be defined as:

$$\frac{\partial \xi}{\partial \mathbf{w}_{hj}} =
\frac{\partial Z_{h}}{\partial \mathbf{w}_{hj}} \frac{\partial H}{\partial Z_{h}} \frac{\partial \xi}{\partial H} =
\frac{\partial Z_{h}}{\partial \mathbf{w}_{hj}} \frac{\partial \xi}{\partial Z_h} =
\sum_{i=1}^N x_{ij} \delta_{hi}$$

For bias term $b_h$:

$$\frac{\partial \xi}{\partial \mathbf{b}_{h}} = \frac{\partial Z_{h}}{\partial \mathbf{b}_{h}} \frac{\partial H}{\partial Z_{h}} \frac{\partial \xi}{\partial H}
= \sum_{j=1}^N \delta_{hj}$$

### Code for cost and backward gradient

```python
# Define the cost function
def cost(Y, T):
    return - np.multiply(T, np.log(Y)).sum()

# Define the error function at the output
def error_output(Y, T):
    return Y - T

# Define the gradient function for the weight parameters at the output layer
def gradient_weight_out(H, Eo):
    return  H.T.dot(Eo)

# Define the gradient function for the bias parameters at the output layer
def gradient_bias_out(Eo):
    return  np.sum(Eo, axis=0, keepdims=True)

# Define the error function at the hidden layer
def error_hidden(H, Wo, Eo):
    # H * (1-H) * (E . Wo^T)
    return np.multiply(np.multiply(H,(1 - H)), Eo.dot(Wo.T))

# Define the gradient function for the weight parameters at the hidden layer
def gradient_weight_hidden(X, Eh):
    return X.T.dot(Eh)

# Define the gradient function for the bias parameters at the output layer
def gradient_bias_hidden(Eh):
    return  np.sum(Eh, axis=0, keepdims=True)
```

# Momentum

Model like this is highly unlikely to have convex cost functions and we might easily get a local minimum with gradient decent. Momentum is created to solve this. It's probably the most popular extension of the backprop algorithm.

Momentum can be defined:

$$\begin{split}
V(i+1) & = \lambda V(i) - \mu \frac{\partial \xi}{\partial \theta(i)} \\
\theta(i+1) & = \theta(i) + V(i+1)
\end{split}
$$

Where $V(i)$ is the velocity of the parameters at iteration $i$ and 0< $\lambda$ <1 shows how much the velocity decreases due to 'resistance' and $\mu$ the learning rate.

Note here we are using previous $V(i)$ to update $V(i+1)$. The initial $V(0)$ should be nearly 0 so $V(1)$ is $\lambda$ (say, 0.9)*0 $-\mu \frac{\partial \xi}{\partial \theta(0)}$ (say, -1) = -1 and in iteration=2 the $V(i)$ is the same direction (say, -0.5) then $V(2)$ will become bigger (-0.9 - 0.5 = -1.4) and smaller if opposite direction (say. 0.5 -> -0.9 + 0.5 = -0.4).

Essentially, it's just like we push a ball down a hill. The ball accumulates momentum as it rolls downhill, becoming faster and faster on the way. The same thing happens to our parameter updates: The momentum term increases for dimensions whose gradients point in the same directions and reduces updates for dimensions whose gradients change directions. As a result, we gain faster convergence and reduced oscillation.

We can code backprop with momentum as below.

```python
# Define the update function to update the network parameters over 1 iteration
def backprop_gradients(X, T, Wh, bh, Wo, bo):
    # Compute the output of the network
    # Compute the activations of the layers
    H = hidden_activations(X, Wh, bh)
    Y = output_activations(H, Wo, bo)
    # Compute the gradients of the output layer
    Eo = error_output(Y, T)
    JWo = gradient_weight_out(H, Eo)
    Jbo = gradient_bias_out(Eo)
    # Compute the gradients of the hidden layer
    Eh = error_hidden(H, Wo, Eo)
    JWh = gradient_weight_hidden(X, Eh)
    Jbh = gradient_bias_hidden(Eh)
    return [JWh, Jbh, JWo, Jbo]

def update_velocity(X, T, ls_of_params, Vs, momentum_term, learning_rate):
    # ls_of_params = [Wh, bh, Wo, bo]
    # Js = [JWh, Jbh, JWo, Jbo]
    Js = backprop_gradients(X, T, *ls_of_params)
    return [momentum_term * V - learning_rate * J for V,J in zip(Vs, Js)]

def update_params(ls_of_params, Vs):
    # ls_of_params = [Wh, bh, Wo, bo]
    # Vs = [VWh, Vbh, VWo, Vbo]
    return [P + V for P,V in zip(ls_of_params, Vs)]
```

# Code Implementaion

```python
# Run backpropagation
# Initialize weights and biases
init_var = 0.1
# Initialize hidden layer parameters
bh = np.random.randn(1, 3) * init_var
Wh = np.random.randn(2, 3) * init_var
# Initialize output layer parameters
bo = np.random.randn(1, 2) * init_var
Wo = np.random.randn(3, 2) * init_var
# Parameters are already initilized randomly with the gradient checking
# Set the learning rate
learning_rate = 0.02
momentum_term = 0.9

# define the velocities Vs = [VWh, Vbh, VWo, Vbo]
Vs = [np.zeros_like(M) for M in [Wh, bh, Wo, bo]]

# Start the gradient descent updates and plot the iterations
nb_of_iterations = 300  # number of gradient descent updates
lr_update = learning_rate / nb_of_iterations # learning rate update rule
ls_costs = [cost(nn(X, Wh, bh, Wo, bo), T)]  # list of cost over the iterations
for i in range(nb_of_iterations):
    # Update the velocities and the parameters
    Vs = update_velocity(X, T, [Wh, bh, Wo, bo], Vs, momentum_term, learning_rate)
    Wh, bh, Wo, bo = update_params([Wh, bh, Wo, bo], Vs)
    ls_costs.append(cost(nn(X, Wh, bh, Wo, bo), T))
```

```python
# Plot the cost over the iterations
plt.plot(ls_costs, 'b-')
plt.xlabel('iteration')
plt.ylabel('$\\xi$', fontsize=15)
plt.title('Decrease of cost over backprop iteration')
plt.grid()
plt.show()
```

### Visualization of the trained classifier

The classifier we just trianed is circled around and between the blue and red class. It's non-linear and hence be able to correctly classify red and blue.

```python
# Plot the resulting decision boundary
# Generate a grid over the input space to plot the color of the
#  classification at that grid point
nb_of_xs = 200
xs1 = np.linspace(-2, 2, num=nb_of_xs)
xs2 = np.linspace(-2, 2, num=nb_of_xs)
xx, yy = np.meshgrid(xs1, xs2) # create the grid
# Initialize and fill the classification plane
classification_plane = np.zeros((nb_of_xs, nb_of_xs))
for i in range(nb_of_xs):
    for j in range(nb_of_xs):
        pred = nn_predict(np.asmatrix([xx[i,j], yy[i,j]]), Wh, bh, Wo, bo)
        classification_plane[i,j] = pred[0,0]
# Create a color map to show the classification colors of each grid point
cmap = ListedColormap([
        colorConverter.to_rgba('w', alpha=0.30),
        colorConverter.to_rgba('g', alpha=0.30)])

# Plot the classification plane with decision boundary and input samples
plt.contourf(xx, yy, classification_plane, cmap=cmap)
# Plot both classes on the x1, x2 plane
plt.plot(x_red[:,0], x_red[:,1], 'ro', label='class red')
plt.plot(x_blue[:,0], x_blue[:,1], 'bo', label='class blue')
plt.grid()
plt.legend(loc=1)
plt.xlabel('$x_1$', fontsize=15)
plt.ylabel('$x_2$', fontsize=15)
plt.axis([-1.5, 1.5, -1.5, 1.5])
plt.title('red vs blue classification boundary')
plt.show()
```

### Transformation of the input domain

You can see from the plot below. the 2-dimentions input have been project into 3-dimension space (hidden layer) and become liner separable.

```python
# Plot the projection of the input onto the hidden layer

# Define the projections of the blue and red classes
H_blue = hidden_activations(x_blue, Wh, bh)
H_red = hidden_activations(x_red, Wh, bh)
# Plot the error surface
fig = plt.figure()
ax = Axes3D(fig)
ax.plot(np.ravel(H_blue[:,2]), np.ravel(H_blue[:,1]), np.ravel(H_blue[:,0]), 'bo')
ax.plot(np.ravel(H_red[:,2]), np.ravel(H_red[:,1]), np.ravel(H_red[:,0]), 'ro')
ax.set_xlabel('$h_1$', fontsize=15)
ax.set_ylabel('$h_2$', fontsize=15)
ax.set_zlabel('$h_3$', fontsize=15)
ax.view_init(elev=10, azim=-40)
plt.title('Projection of the input X onto the hidden layer H')
plt.grid()
plt.show()
```

The sample codes in this note come from [peterroelants.github.io](http://peterroelants.github.io/) where providing more details on neural netwrok and deep learning. It's very informative and highly recommanded. Here is more like my personal memo.