# Problem 1: Basics of Neural Networks
* Learning Objective: In the entrance exam, we asked you to implement a K-NN classifier to classify some tiny images extracted from CIFAR-10 dataset. Probably many of you noticed that the performances were quite bad. In this problem, you are going to implement a basic multi-layer fully connected neural network to perform the same classification task.
* Provided Code: We provide the skeletons of classes you need to complete. Forward checking and gradient checkings are provided for verifying your implementation as well.
* TODOs: You are asked to implement the forward passes and backward passes for standard layers and loss functions, various widely-used optimizers, and part of the training procedure. And finally we want you to train a network from scratch on your own.

```python
from __future__ import division
from lib.fully_conn import *
from lib.layer_utils import *
from lib.grad_check import *
from lib.datasets import *
from lib.optim import *
from lib.train import *
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2
```

## Loading the data (CIFAR-10)
Run the following code block to load in the properly splitted CIFAR-10 data.

```python
data = CIFAR10_data()
for k, v in data.iteritems():
    print "Name: {} Shape: {}".format(k, v.shape)
```

    Name: data_train Shape: (49000, 3, 32, 32)
    Name: data_val Shape: (1000, 3, 32, 32)
    Name: data_test Shape: (1000, 3, 32, 32)
    Name: labels_train Shape: (49000,)
    Name: labels_val Shape: (1000,)
    Name: labels_test Shape: (1000,)

## Implement Standard Layers
You will now implement all the following standard layers commonly seen in a fully connected neural network. Please refer to the file layer_utils.py under the directory lib. Take a look at each class skeleton, and we will walk you through the network layer by layer. We provide results of some examples we pre-computed for you for checking the forward pass, and also the gradient checking for the backward pass.

## FC Forward
In the class skeleton "fc", please complete the forward pass in function "forward", the input to the fc layer may not be of dimension (batch size, features size), it could be an image or any higher dimensional data. Make sure that you handle this dimensionality issue.

```python
# Test the fc forward function
input_bz = 3
input_dim = (6, 5, 4)
output_dim = 4

input_size = input_bz * np.prod(input_dim)
weight_size = output_dim * np.prod(input_dim)

single_fc = fc(np.prod(input_dim), output_dim, init_scale=0.02, name="fc_test")

x = np.linspace(-0.1, 0.5, num=input_size).reshape(input_bz, *input_dim)
w = np.linspace(-0.2, 0.3, num=weight_size).reshape(np.prod(input_dim), output_dim)
b = np.linspace(-0.3, 0.1, num=output_dim)

single_fc.params[single_fc.w_name] = w
single_fc.params[single_fc.b_name] = b

out = single_fc.forward(x)

correct_out = np.array([[0.70157129, 0.83483484, 0.96809839, 1.10136194],
                        [1.86723094, 2.02561647, 2.18400199, 2.34238752],
                        [3.0328906,  3.2163981,  3.3999056,  3.5834131]])

# Compare your output with the above pre-computed ones.
# The difference should not be larger than 1e-8
print "Difference: ", rel_error(out, correct_out)
```

    Difference:  2.48539291792e-09

## FC Backward
Please complete the function "backward" as the backward pass of the fc layer. Follow the instructions in the comments to store gradients into the predefined dictionaries in the attributes of the class. Parameters of the layer are also stored in the predefined dictionary.

```python
# Test the fc backward function
x = np.random.randn(10, 2, 2, 3)
w = np.random.randn(12, 10)
b = np.random.randn(10)
dout = np.random.randn(10, 10)

single_fc = fc(np.prod(x.shape[1:]), 10, init_scale=5e-2, name="fc_test")
single_fc.params[single_fc.w_name] = w
single_fc.params[single_fc.b_name] = b

dx_num = eval_numerical_gradient_array(lambda x: single_fc.forward(x), x, dout)
dw_num = eval_numerical_gradient_array(lambda w: single_fc.forward(x), w, dout)
db_num = eval_numerical_gradient_array(lambda b: single_fc.forward(x), b, dout)

out = single_fc.forward(x)
dx = single_fc.backward(dout)
dw = single_fc.grads[single_fc.w_name]
db = single_fc.grads[single_fc.b_name]

# The error should be around 1e-10
print "dx Error: ", rel_error(dx_num, dx)
print "dw Error: ", rel_error(dw_num, dw)
print "db Error: ", rel_error(db_num, db)
```

    dx Error:  1.76725537891e-09
    dw Error:  4.59418034689e-10
    db Error:  5.33117814788e-11

## ReLU Forward
In the class skeleton "relu", please complete the forward pass.

```python
# Test the relu forward function
x = np.linspace(-1.0, 1.0, num=12).reshape(3, 4)
relu_f = relu(name="relu_f")

out = relu_f.forward(x)
correct_out = np.array([[0.,          0.,        0.,         0.        ],
                        [0.,          0.,        0.09090909, 0.27272727],
                        [0.45454545, 0.63636364, 0.81818182, 1.        ]])

# Compare your output with the above pre-computed ones.
# The difference should not be larger than 1e-8
print "Difference: ", rel_error(out, correct_out)
```

    Difference:  5.00000005012e-09

## ReLU Backward
Please complete the backward pass of the class relu.

```python
# Test the relu backward function
x = np.random.randn(10, 10)
dout = np.random.randn(*x.shape)
relu_b = relu(name="relu_b")

dx_num = eval_numerical_gradient_array(lambda x: relu_b.forward(x), x, dout)

out = relu_b.forward(x)
dx = relu_b.backward(dout)

# The error should not be larger than 1e-10
print "dx Error: ", rel_error(dx_num, dx)
```

    dx Error:  3.27562609906e-12

## Dropout Forward
In the class "dropout", please complete the forward pass. Remember that the dropout is only applied during training phase, you should pay attention to this while implementing the function.

```python
x = np.random.randn(100, 100) + 5.0

print "----------------------------------------------------------------"
for p in [0.25, 0.50, 0.75]:
    dropout_f = dropout(p)
    out = dropout_f.forward(x, True)
    out_test = dropout_f.forward(x, False)

    print "Dropout p = ", p
    print "Mean of input: ", x.mean()
    print "Mean of output during training time: ", out.mean()
    print "Mean of output during testing time: ", out_test.mean()
    print "Fraction of output set to zero during training time: ", (out == 0).mean()
    print "Fraction of output set to zero during testing time: ", (out_test == 0).mean()
    print "----------------------------------------------------------------"
```

    ----------------------------------------------------------------
    Dropout p =  0.25
    Mean of input:  4.9905029386
    Mean of output during training time:  4.97881954708
    Mean of output during testing time:  4.9905029386
    Fraction of output set to zero during training time:  0.2518
    Fraction of output set to zero during testing time:  0.0
    ----------------------------------------------------------------
    Dropout p =  0.5
    Mean of input:  4.9905029386
    Mean of output during training time:  4.87684799023
    Mean of output during testing time:  4.9905029386
    Fraction of output set to zero during training time:  0.5096
    Fraction of output set to zero during testing time:  0.0
    ----------------------------------------------------------------
    Dropout p =  0.75
    Mean of input:  4.9905029386
    Mean of output during training time:  5.03028699758
    Mean of output during testing time:  4.9905029386
    Fraction of output set to zero during training time:  0.7481
    Fraction of output set to zero during testing time:  0.0
    ----------------------------------------------------------------

## Dropout Backward
Please complete the backward pass. Again remember that the dropout is only applied during training phase, handle this in the backward pass as well.

```python
x = np.random.randn(5, 5) + 5
dout = np.random.randn(*x.shape)

p = 0.75
dropout_b = dropout(p, seed=100)
out = dropout_b.forward(x, True)
dx = dropout_b.backward(dout)
dx_num = eval_numerical_gradient_array(lambda xx: dropout_b.forward(xx, True), x, dout)

# The error should not be larger than 1e-9
print 'dx relative error: ', rel_error(dx, dx_num)
```

    dx relative error:  1.89289560828e-11

## Testing cascaded layers: FC + ReLU
Please find the TestFCReLU function in fully_conn.py under lib directory.

You only need to complete few lines of code in the TODO block.

Please design an FC --> ReLU two-layer-mini-network where the parameters of them match the given x, w, and b

Please insert the corresponding names you defined for each layer to param_name_w, and param_name_b respectively.

Here you only modify the param_name part, the _w, and _b are automatically assigned during network setup

```python
x = np.random.randn(2, 3, 4)  # the input features
w = np.random.randn(12, 10)   # the weight of fc layer
b = np.random.randn(10)       # the bias of fc layer
dout = np.random.randn(2, 10) # the gradients to the output, notice the shape

tiny_net = TestFCReLU()

tiny_net.net.assign("my_fc_w", w)
tiny_net.net.assign("my_fc_b", b)

out = tiny_net.forward(x)
dx = tiny_net.backward(dout)

dw = tiny_net.net.get_grads("my_fc_w")
db = tiny_net.net.get_grads("my_fc_b")

dx_num = eval_numerical_gradient_array(lambda x: tiny_net.forward(x), x, dout)
dw_num = eval_numerical_gradient_array(lambda w: tiny_net.forward(x), w, dout)
db_num = eval_numerical_gradient_array(lambda b: tiny_net.forward(x), b, dout)

# The errors should not be larger than 1e-7
print "dx error: ", rel_error(dx_num, dx)
print "dw error: ", rel_error(dw_num, dw)
print "db error: ", rel_error(db_num, db)
```

    dx error:  3.85642924037e-10
    dw error:  1.30141409408e-10
    db error:  3.27561702898e-12

## SoftMax Function and Loss Layer
In the layer_utils.py, please first complete the function softmax, which will be use in the function cross_entropy. Please refer to the lecture slides of the mathematical expressions of the cross entropy loss function, and complete its forward pass and backward pass.

```python
num_classes, num_inputs = 5, 50
x = 0.001 * np.random.randn(num_inputs, num_classes)
y = np.random.randint(num_classes, size=num_inputs)
test_loss = cross_entropy()

dx_num = eval_numerical_gradient(lambda x: test_loss.forward(x, y), x, verbose=False)

loss = test_loss.forward(x, y)
dx = test_loss.backward()

# Test softmax_loss function. Loss should be around 1.609
# and dx error should be at the scale of 1e-8 (or smaller)
print "Cross Entropy Loss: ", loss
print "dx error: ", rel_error(dx_num, dx)
```

    Cross Entropy Loss:  1.60945846468
    dx error:  3.28599046685e-09

## Test a Small Fully Connected Network
Please find the SmallFullyConnectedNetwork function in fully_conn.py under lib directory.

Again you only need to complete few lines of code in the TODO block.

Please design an FC --> ReLU --> FC --> ReLU network where the shapes of parameters match the given shapes

Please insert the corresponding names you defined for each layer to param_name_w, and param_name_b respectively.

Here you only modify the param_name part, the _w, and _b are automatically assigned during network setup

```python
model = SmallFullyConnectedNetwork()
loss_func = cross_entropy()

N, D, = 4, 4  # N: batch size, D: input dimension
H, C  = 30, 7 # H: hidden dimension, C: output dimension
std = 0.02
x = np.random.randn(N, D)
y = np.random.randint(C, size=N)

print "Testing initialization ... "
w1_std = abs(model.net.get_params("fc1_w").std() - std)
b1 = model.net.get_params("fc1_b").std()
w2_std = abs(model.net.get_params("fc2_w").std() - std)
b2 = model.net.get_params("fc2_b").std()

assert w1_std < std / 10, "First layer weights do not seem right"
assert np.all(b1 == 0), "First layer biases do not seem right"
assert w2_std < std / 10, "Second layer weights do not seem right"
assert np.all(b2 == 0), "Second layer biases do not seem right"
print "Passed!"

print "Testing test-time forward pass ... "
w1 = np.linspace(-0.7, 0.3, num=D*H).reshape(D, H)
w2 = np.linspace(-0.3, 0.4, num=H*C).reshape(H, C)
b1 = np.linspace(-0.1, 0.9, num=H)
b2 = np.linspace(-0.9, 0.1, num=C)

model.net.assign("fc1_w", w1)
model.net.assign("fc1_b", b1)
model.net.assign("fc2_w", w2)
model.net.assign("fc2_b", b2)

feats = np.linspace(-5.5, 4.5, num=N*D).reshape(D, N).T
scores = model.forward(feats)
correct_scores = np.asarray([[4.20670862, 4.87188359, 5.53705856, 6.20223352, 6.86740849, 7.53258346, 8.19775843],
                             [4.74826036, 5.35984681, 5.97143326, 6.58301972, 7.19460617, 7.80619262, 8.41777907],
                             [5.2898121,  5.84781003, 6.40580797, 6.96380591, 7.52180384, 8.07980178, 8.63779971],
                             [5.83136384, 6.33577326, 6.84018268, 7.3445921,  7.84900151, 8.35341093, 8.85782035]])
scores_diff = np.sum(np.abs(scores - correct_scores))
assert scores_diff < 1e-6, "Your implementation might went wrong!"
print "Passed!"

print "Testing the loss ...",
y = np.asarray([0, 5, 1, 4])
loss = loss_func.forward(scores, y)
dLoss = loss_func.backward()
correct_loss = 2.90181552716
assert abs(loss - correct_loss) < 1e-10, "Your implementation might went wrong!"
print "Passed!"

print "Testing the gradients (error should be no larger than 1e-7) ..."
din = model.backward(dLoss)
for layer in model.net.layers:
    if not layer.params:
        continue
    for name in sorted(layer.grads):
        f = lambda _: loss_func.forward(model.forward(feats), y)
        grad_num = eval_numerical_gradient(f, layer.params[name], verbose=False)
        print '%s relative error: %.2e' % (name, rel_error(grad_num, layer.grads[name]))
```

    Testing initialization ...
    Passed!
    Testing test-time forward pass ...
    Passed!
    Testing the loss ... Passed!
    Testing the gradients (error should be no larger than 1e-7) ...
    fc1_b relative error: 4.20e-09
    fc1_w relative error: 6.00e-09
    fc2_b relative error: 6.71e-08
    fc2_w relative error: 2.59e-09

## Test a Fully Connected Network regularized with Dropout
Please find the DropoutNet function in fully_conn.py under lib directory.

For this part you don't need to design a new network, just simply run the following test code

If something goes wrong, you might want to double check your dropout implementation

```python
N, D, C = 3, 15, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))
seed = 123

for dropout_p in [0., 0.25, 0.5]:
    print "Dropout p =", dropout_p
    model = DropoutNet(dropout_p=dropout_p, seed=seed)
    loss_func = cross_entropy()
    output = model.forward(X, True)
    loss = loss_func.forward(output, y)
    dLoss = loss_func.backward()
    dX = model.backward(dLoss)
    grads = model.net.grads
    print "Loss (should be ~2.30) : ", loss

    print "Error of gradients should be no larger than 1e-5"
    for name in sorted(model.net.params):
        f = lambda _: loss_func.forward(model.forward(X, True), y)
        grad_num = eval_numerical_gradient(f, model.net.params[name], verbose=False, h=1e-5)
        print "{} relative error: {}".format(name, rel_error(grad_num, grads[name]))
    print
```

    Dropout p = 0.0
    Loss (should be ~2.30) :  2.30285163514
    Error of gradients should be no larger than 1e-5
    fc1_b relative error: 3.86706741969e-08
    fc1_w relative error: 1.68428082848e-06
    fc2_b relative error: 3.81209866093e-09
    fc2_w relative error: 2.26785963522e-06
    fc3_b relative error: 1.52481732925e-10
    fc3_w relative error: 1.16962221954e-07

    Dropout p = 0.25
    Loss (should be ~2.30) :  2.30204156857
    Error of gradients should be no larger than 1e-5
    fc1_b relative error: 2.47220774157e-08
    fc1_w relative error: 5.90463480511e-07
    fc2_b relative error: 2.74009908736e-08
    fc2_w relative error: 2.11636623939e-05
    fc3_b relative error: 8.40508736712e-11
    fc3_w relative error: 3.09642935217e-07

    Dropout p = 0.5
    Loss (should be ~2.30) :  2.30443780379
    Error of gradients should be no larger than 1e-5
    fc1_b relative error: 2.22118085121e-08
    fc1_w relative error: 7.91512428221e-07
    fc2_b relative error: 9.43469443542e-07
    fc2_w relative error: 4.22495438947e-05
    fc3_b relative error: 8.85507369309e-11
    fc3_w relative error: 2.83107525583e-07

## Training a Network
In this section, we defined a TinyNet class for you to fill in the TODO block in fully_conn.py.
* Here please design a two layer fully connected network for this part.
* Please read the train.py under lib directory carefully and complete the TODO blocks in the train_net function first.
* In addition, read how the SGD function is implemented in optim.py, you will be asked to complete three other optimization methods in the later sections.

```python
# Arrange the data
data_dict = {
    "data_train": (data["data_train"], data["labels_train"]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}
```

```python
data["data_train"].shape
```

    (49000, 3, 32, 32)

```python
set(data["labels_train"])
```

    {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}

```python
model = TinyNet()
loss_f = cross_entropy()
optimizer = SGD(model.net, 1e-4)
```

### Now train the network to achieve at least 50% validation accuracy

```python
results = None
#############################################################################
# TODO: Use the train_net function you completed to train a network         #
#############################################################################
opt_params, loss_hist, train_acc_hist, val_acc_hist = train_net(data_dict, model, loss_f, optimizer, 128, 10)
print (val_acc_hist[-1])
#############################################################################
#                             END OF YOUR CODE                              #
#############################################################################
#opt_params, loss_hist, train_acc_hist, val_acc_hist = results
```

    0.504

```python
# Take a look at what names of params were stored
print opt_params.keys()
```

    ['fc1_w', 'fc2_b', 'fc1_b', 'fc2_w']

```python
# Demo: How to load the parameters to a newly defined network
model = TinyNet()
model.net.load(opt_params)
val_acc = compute_acc(model, data["data_val"], data["labels_val"])
print "Validation Accuracy: {}%".format(val_acc*100)
test_acc = compute_acc(model, data["data_test"], data["labels_test"])
print "Testing Accuracy: {}%".format(test_acc*100)
```

    Loading Params: fc1_w Shape: (3072, 512)
    Loading Params: fc1_b Shape: (512,)
    Loading Params: fc2_b Shape: (10,)
    Loading Params: fc2_w Shape: (512, 10)
    Validation Accuracy: 50.4%
    Testing Accuracy: 49.9%

```python
# Plot the learning curves
plt.subplot(2, 1, 1)
plt.title('Training loss')
loss_hist_ = loss_hist[1::100] # sparse the curve a bit
plt.plot(loss_hist_, '-o')
plt.xlabel('Iteration')

plt.subplot(2, 1, 2)
plt.title('Accuracy')
plt.plot(train_acc_hist, '-o', label='Training')
plt.plot(val_acc_hist, '-o', label='Validation')
plt.plot([0.5] * len(val_acc_hist), 'k--')
plt.xlabel('Epoch')
plt.legend(loc='lower right')
plt.gcf().set_size_inches(15, 12)
plt.show()
```

## Different Optimizers
There are several more advanced optimizers than vanilla SGD, you will implement three more sophisticated and widely-used methods in this section. Please complete the TODOs in the optim.py under lib directory.

## SGD + Momentum
The update rule of SGD plus momentum is as shown below:

\begin{equation}
v_t: velocity \\
\gamma: momentum \\
\eta: learning\ rate \\
v_t = \gamma v_{t-1} + \eta \nabla_{\theta}J(\theta) \\
\theta = \theta - v_t
\end{equation}
Complete the SGDM() function in optim.py

```python
# SGD with momentum
model = TinyNet()
loss_f = cross_entropy()
optimizer = SGD(model.net, 1e-4)
```

```python
# Test the implementation of SGD with Momentum
N, D = 4, 5
test_sgd = sequential(fc(N, D, name="sgd_fc"))

w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
v = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)

test_sgd.layers[0].params = {"sgd_fc_w": w}
test_sgd.layers[0].grads = {"sgd_fc_w": dw}

test_sgd_momentum = SGDM(test_sgd, 1e-3, 0.9)
test_sgd_momentum.velocity = {"sgd_fc_w": v}
test_sgd_momentum.step()

updated_w = test_sgd.layers[0].params["sgd_fc_w"]
velocity = test_sgd_momentum.velocity["sgd_fc_w"]

expected_updated_w = np.asarray([
  [ 0.1406,      0.20738947,  0.27417895,  0.34096842,  0.40775789],
  [ 0.47454737,  0.54133684,  0.60812632,  0.67491579,  0.74170526],
  [ 0.80849474,  0.87528421,  0.94207368,  1.00886316,  1.07565263],
  [ 1.14244211,  1.20923158,  1.27602105,  1.34281053,  1.4096    ]])
expected_velocity = np.asarray([
  [ 0.5406,      0.55475789,  0.56891579, 0.58307368,  0.59723158],
  [ 0.61138947,  0.62554737,  0.63970526,  0.65386316,  0.66802105],
  [ 0.68217895,  0.69633684,  0.71049474,  0.72465263,  0.73881053],
  [ 0.75296842,  0.76712632,  0.78128421,  0.79544211,  0.8096    ]])

print 'updated_w error: ', rel_error(updated_w, expected_updated_w)
print 'velocity error: ', rel_error(expected_velocity, velocity)
```

    updated_w error:  8.88234703351e-09
    velocity error:  4.26928774328e-09

Run the following code block to train a multi-layer fully connected network with both SGD and SGD plus Momentum. The network trained with SGDM optimizer should converge faster.

```python
# Arrange a small data
num_train = 4000
small_data_dict = {
    "data_train": (data["data_train"][:num_train], data["labels_train"][:num_train]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}

model_sgd      = FullyConnectedNetwork()
model_sgdm     = FullyConnectedNetwork()
loss_f_sgd     = cross_entropy()
loss_f_sgdm    = cross_entropy()
optimizer_sgd  = SGD(model_sgd.net, 1e-2)
optimizer_sgdm = SGDM(model_sgdm.net, 1e-2, 0.9)

print "Training with Vanilla SGD..."
results_sgd = train_net(small_data_dict, model_sgd, loss_f_sgd, optimizer_sgd, batch_size=100,
                        max_epochs=5, show_every=100, verbose=True)

print "\nTraining with SGD plus Momentum..."
results_sgdm = train_net(small_data_dict, model_sgdm, loss_f_sgdm, optimizer_sgdm, batch_size=100,
                         max_epochs=5, show_every=100, verbose=True)

opt_params_sgd,  loss_hist_sgd,  train_acc_hist_sgd,  val_acc_hist_sgd  = results_sgd
opt_params_sgdm, loss_hist_sgdm, train_acc_hist_sgdm, val_acc_hist_sgdm = results_sgdm

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgd, 'o', label="Vanilla SGD")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgd, '-o', label="Vanilla SGD")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgd, '-o', label="Vanilla SGD")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgdm, 'o', label="SGD with Momentum")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgdm, '-o', label="SGD with Momentum")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgdm, '-o', label="SGD with Momentum")

for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()
```

## RMSProp
The update rule of RMSProp is as shown below:

\begin{equation}
\gamma: decay\ rate \\
\epsilon: small\ number \\
g_t^2: squared\ gradients \\
\eta: learning\ rate \\
E[g^2]_t: decaying\ average\ of\ past\ squared\ gradients\ at\ update\ step\ t \\
E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g_t^2 \\
\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t+\epsilon}}
\end{equation}
Complete the RMSProp() function in optim.py

```python
# Test RMSProp implementation; you should see errors less than 1e-7
N, D = 4, 5
test_rms = sequential(fc(N, D, name="rms_fc"))

w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
cache = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)

test_rms.layers[0].params = {"rms_fc_w": w}
test_rms.layers[0].grads = {"rms_fc_w": dw}

opt_rms = RMSProp(test_rms, 1e-2, 0.99)
opt_rms.cache = {"rms_fc_w": cache}
opt_rms.step()

updated_w = test_rms.layers[0].params["rms_fc_w"]
cache = opt_rms.cache["rms_fc_w"]

expected_updated_w = np.asarray([
  [-0.39223849, -0.34037513, -0.28849239, -0.23659121, -0.18467247],
  [-0.132737,   -0.08078555, -0.02881884,  0.02316247,  0.07515774],
  [ 0.12716641,  0.17918792,  0.23122175,  0.28326742,  0.33532447],
  [ 0.38739248,  0.43947102,  0.49155973,  0.54365823,  0.59576619]])
expected_cache = np.asarray([
  [ 0.5976,      0.6126277,   0.6277108,   0.64284931,  0.65804321],
  [ 0.67329252,  0.68859723,  0.70395734,  0.71937285,  0.73484377],
  [ 0.75037008,  0.7659518,   0.78158892,  0.79728144,  0.81302936],
  [ 0.82883269,  0.84469141,  0.86060554,  0.87657507,  0.8926    ]])

print 'updated_w error: ', rel_error(expected_updated_w, updated_w)
print 'cache error: ', rel_error(expected_cache, opt_rms.cache["rms_fc_w"])
```

    updated_w error:  9.50264522989e-08
    cache error:  2.64779558072e-09

## Adam
The update rule of Adam is as shown below:

\begin{equation}
g_t: gradients\ at\ update\ step\ t \\
m_t = \beta_1m_{t-1} + (1-\beta_1)g_t \\
v_t = \beta_2v_{t-1} + (1-\beta_1)g_t^2 \\
\hat{m_t}: bias\ corrected\ m_t \\
\hat{v_t}: bias\ corrected\ v_t \\
\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v_t}}+\epsilon}
\end{equation}
Complete the Adam() function in optim.py

```python
# Test Adam implementation; you should see errors around 1e-7 or less
N, D = 4, 5
test_adam = sequential(fc(N, D, name="adam_fc"))

w = np.linspace(-0.4, 0.6, num=N*D).reshape(N, D)
dw = np.linspace(-0.6, 0.4, num=N*D).reshape(N, D)
m = np.linspace(0.6, 0.9, num=N*D).reshape(N, D)
v = np.linspace(0.7, 0.5, num=N*D).reshape(N, D)

test_adam.layers[0].params = {"adam_fc_w": w}
test_adam.layers[0].grads = {"adam_fc_w": dw}

opt_adam = Adam(test_adam, 1e-2, 0.9, 0.999, t=5)
opt_adam.mt = {"adam_fc_w": m}
opt_adam.vt = {"adam_fc_w": v}
opt_adam.step()

updated_w = test_adam.layers[0].params["adam_fc_w"]
mt = opt_adam.mt["adam_fc_w"]
vt = opt_adam.vt["adam_fc_w"]

expected_updated_w = np.asarray([
  [-0.40094747, -0.34836187, -0.29577703, -0.24319299, -0.19060977],
  [-0.1380274,  -0.08544591, -0.03286534,  0.01971428,  0.0722929],
  [ 0.1248705,   0.17744702,  0.23002243,  0.28259667,  0.33516969],
  [ 0.38774145,  0.44031188,  0.49288093,  0.54544852,  0.59801459]])
expected_v = np.asarray([
  [ 0.69966,     0.68908382,  0.67851319,  0.66794809,  0.65738853,],
  [ 0.64683452,  0.63628604,  0.6257431,   0.61520571,  0.60467385,],
  [ 0.59414753,  0.58362676,  0.57311152,  0.56260183,  0.55209767,],
  [ 0.54159906,  0.53110598,  0.52061845,  0.51013645,  0.49966,   ]])
expected_m = np.asarray([
  [ 0.48,        0.49947368,  0.51894737,  0.53842105,  0.55789474],
  [ 0.57736842,  0.59684211,  0.61631579,  0.63578947,  0.65526316],
  [ 0.67473684,  0.69421053,  0.71368421,  0.73315789,  0.75263158],
  [ 0.77210526,  0.79157895,  0.81105263,  0.83052632,  0.85      ]])

print 'updated_w error: ', rel_error(expected_updated_w, updated_w)
print 'mt error: ', rel_error(expected_m, mt)
print 'vt error: ', rel_error(expected_v, vt)
```

    updated_w error:  1.13956917985e-07
    mt error:  4.21496319311e-09
    vt error:  4.20831403811e-09

## Comparing the optimizers
Run the following code block to compare the plotted results among all the above optimizers

```python
model_rms      = FullyConnectedNetwork()
model_adam     = FullyConnectedNetwork()
loss_f_rms     = cross_entropy()
loss_f_adam    = cross_entropy()
optimizer_rms  = RMSProp(model_rms.net, 5e-4)
optimizer_adam = Adam(model_adam.net, 5e-4)

print "Training with RMSProp..."
results_rms = train_net(small_data_dict, model_rms, loss_f_rms, optimizer_rms, batch_size=100,
                        max_epochs=5, show_every=100, verbose=True)

print "\nTraining with Adam..."
results_adam = train_net(small_data_dict, model_adam, loss_f_adam, optimizer_adam, batch_size=100,
                         max_epochs=5, show_every=100, verbose=True)

opt_params_rms,  loss_hist_rms,  train_acc_hist_rms,  val_acc_hist_rms  = results_rms
opt_params_adam, loss_hist_adam, train_acc_hist_adam, val_acc_hist_adam = results_adam

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgd, 'o', label="Vanilla SGD")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgd, '-o', label="Vanilla SGD")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgd, '-o', label="Vanilla SGD")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgdm, 'o', label="SGD with Momentum")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgdm, '-o', label="SGD with Momentum")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgdm, '-o', label="SGD with Momentum")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_rms, 'o', label="RMSProp")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_rms, '-o', label="RMSProp")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_rms, '-o', label="RMSProp")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_adam, 'o', label="Adam")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_adam, '-o', label="Adam")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_adam, '-o', label="Adam")

for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Training a Network with Dropout
Run the following code blocks to compare the results with and without dropout

```python
# Train two identical nets, one with dropout and one without
num_train = 500
data_dict_500 = {
    "data_train": (data["data_train"][:num_train], data["labels_train"][:num_train]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}

solvers = {}
dropout_ps = [0, 0.25, 0.5, 0.75]  # you can try some dropout prob yourself

results_dict = {}
for dropout_p in dropout_ps:
    results_dict[dropout_p] = {}

for dropout_p in dropout_ps:
    print "Dropout =", dropout_p
    model = DropoutNetTest(dropout_p=dropout_p)
    loss_f = cross_entropy()
    optimizer = SGDM(model.net, 1e-4)
    results = train_net(data_dict_500, model, loss_f, optimizer, batch_size=100,
                        max_epochs=20, show_every=100, verbose=True)
    opt_params, loss_hist, train_acc_hist, val_acc_hist = results
    results_dict[dropout_p] = {
        "opt_params": opt_params,
        "loss_hist": loss_hist,
        "train_acc_hist": train_acc_hist,
        "val_acc_hist": val_acc_hist
    }
```

    Dropout = 0
    (Iteration 1 / 100) loss: 2.69008930553
    (Epoch 1 / 20) Training Accuracy: 0.088, Validation Accuracy: 0.1
    (Epoch 2 / 20) Training Accuracy: 0.094, Validation Accuracy: 0.096
    (Epoch 3 / 20) Training Accuracy: 0.096, Validation Accuracy: 0.099
    (Epoch 4 / 20) Training Accuracy: 0.104, Validation Accuracy: 0.104
    (Epoch 5 / 20) Training Accuracy: 0.114, Validation Accuracy: 0.111
    (Epoch 6 / 20) Training Accuracy: 0.132, Validation Accuracy: 0.115
    (Epoch 7 / 20) Training Accuracy: 0.158, Validation Accuracy: 0.122
    (Epoch 8 / 20) Training Accuracy: 0.178, Validation Accuracy: 0.136
    (Epoch 9 / 20) Training Accuracy: 0.196, Validation Accuracy: 0.14
    (Epoch 10 / 20) Training Accuracy: 0.206, Validation Accuracy: 0.146
    (Epoch 11 / 20) Training Accuracy: 0.214, Validation Accuracy: 0.151
    (Epoch 12 / 20) Training Accuracy: 0.224, Validation Accuracy: 0.159
    (Epoch 13 / 20) Training Accuracy: 0.242, Validation Accuracy: 0.173
    (Epoch 14 / 20) Training Accuracy: 0.252, Validation Accuracy: 0.174
    (Epoch 15 / 20) Training Accuracy: 0.262, Validation Accuracy: 0.176
    (Epoch 16 / 20) Training Accuracy: 0.276, Validation Accuracy: 0.176
    (Epoch 17 / 20) Training Accuracy: 0.286, Validation Accuracy: 0.182
    (Epoch 18 / 20) Training Accuracy: 0.294, Validation Accuracy: 0.182
    (Epoch 19 / 20) Training Accuracy: 0.304, Validation Accuracy: 0.186
    (Epoch 20 / 20) Training Accuracy: 0.318, Validation Accuracy: 0.191
    Dropout = 0.25
    (Iteration 1 / 100) loss: 2.81810487586
    (Epoch 1 / 20) Training Accuracy: 0.098, Validation Accuracy: 0.083
    (Epoch 2 / 20) Training Accuracy: 0.114, Validation Accuracy: 0.091
    (Epoch 3 / 20) Training Accuracy: 0.132, Validation Accuracy: 0.1
    (Epoch 4 / 20) Training Accuracy: 0.154, Validation Accuracy: 0.113
    (Epoch 5 / 20) Training Accuracy: 0.17, Validation Accuracy: 0.125
    (Epoch 6 / 20) Training Accuracy: 0.178, Validation Accuracy: 0.134
    (Epoch 7 / 20) Training Accuracy: 0.182, Validation Accuracy: 0.14
    (Epoch 8 / 20) Training Accuracy: 0.202, Validation Accuracy: 0.146
    (Epoch 9 / 20) Training Accuracy: 0.224, Validation Accuracy: 0.159
    (Epoch 10 / 20) Training Accuracy: 0.234, Validation Accuracy: 0.164
    (Epoch 11 / 20) Training Accuracy: 0.248, Validation Accuracy: 0.17
    (Epoch 12 / 20) Training Accuracy: 0.264, Validation Accuracy: 0.178
    (Epoch 13 / 20) Training Accuracy: 0.27, Validation Accuracy: 0.176
    (Epoch 14 / 20) Training Accuracy: 0.284, Validation Accuracy: 0.183
    (Epoch 15 / 20) Training Accuracy: 0.29, Validation Accuracy: 0.185
    (Epoch 16 / 20) Training Accuracy: 0.298, Validation Accuracy: 0.193
    (Epoch 17 / 20) Training Accuracy: 0.312, Validation Accuracy: 0.197
    (Epoch 18 / 20) Training Accuracy: 0.314, Validation Accuracy: 0.2
    (Epoch 19 / 20) Training Accuracy: 0.324, Validation Accuracy: 0.21
    (Epoch 20 / 20) Training Accuracy: 0.326, Validation Accuracy: 0.209
    Dropout = 0.5
    (Iteration 1 / 100) loss: 3.06663843572
    (Epoch 1 / 20) Training Accuracy: 0.09, Validation Accuracy: 0.083
    (Epoch 2 / 20) Training Accuracy: 0.104, Validation Accuracy: 0.094
    (Epoch 3 / 20) Training Accuracy: 0.128, Validation Accuracy: 0.103
    (Epoch 4 / 20) Training Accuracy: 0.15, Validation Accuracy: 0.12
    (Epoch 5 / 20) Training Accuracy: 0.17, Validation Accuracy: 0.123
    (Epoch 6 / 20) Training Accuracy: 0.176, Validation Accuracy: 0.132
    (Epoch 7 / 20) Training Accuracy: 0.182, Validation Accuracy: 0.141
    (Epoch 8 / 20) Training Accuracy: 0.21, Validation Accuracy: 0.148
    (Epoch 9 / 20) Training Accuracy: 0.214, Validation Accuracy: 0.153
    (Epoch 10 / 20) Training Accuracy: 0.23, Validation Accuracy: 0.153
    (Epoch 11 / 20) Training Accuracy: 0.244, Validation Accuracy: 0.166
    (Epoch 12 / 20) Training Accuracy: 0.258, Validation Accuracy: 0.173
    (Epoch 13 / 20) Training Accuracy: 0.276, Validation Accuracy: 0.177
    (Epoch 14 / 20) Training Accuracy: 0.29, Validation Accuracy: 0.174
    (Epoch 15 / 20) Training Accuracy: 0.284, Validation Accuracy: 0.18
    (Epoch 16 / 20) Training Accuracy: 0.292, Validation Accuracy: 0.19
    (Epoch 17 / 20) Training Accuracy: 0.304, Validation Accuracy: 0.196
    (Epoch 18 / 20) Training Accuracy: 0.31, Validation Accuracy: 0.196
    (Epoch 19 / 20) Training Accuracy: 0.334, Validation Accuracy: 0.205
    (Epoch 20 / 20) Training Accuracy: 0.342, Validation Accuracy: 0.206
    Dropout = 0.75
    (Iteration 1 / 100) loss: 3.60627243956
    (Epoch 1 / 20) Training Accuracy: 0.138, Validation Accuracy: 0.124
    (Epoch 2 / 20) Training Accuracy: 0.166, Validation Accuracy: 0.149
    (Epoch 3 / 20) Training Accuracy: 0.168, Validation Accuracy: 0.148
    (Epoch 4 / 20) Training Accuracy: 0.18, Validation Accuracy: 0.163
    (Epoch 5 / 20) Training Accuracy: 0.198, Validation Accuracy: 0.178
    (Epoch 6 / 20) Training Accuracy: 0.22, Validation Accuracy: 0.177
    (Epoch 7 / 20) Training Accuracy: 0.24, Validation Accuracy: 0.188
    (Epoch 8 / 20) Training Accuracy: 0.25, Validation Accuracy: 0.184
    (Epoch 9 / 20) Training Accuracy: 0.268, Validation Accuracy: 0.194
    (Epoch 10 / 20) Training Accuracy: 0.268, Validation Accuracy: 0.189
    (Epoch 11 / 20) Training Accuracy: 0.268, Validation Accuracy: 0.207
    (Epoch 12 / 20) Training Accuracy: 0.286, Validation Accuracy: 0.206
    (Epoch 13 / 20) Training Accuracy: 0.282, Validation Accuracy: 0.209
    (Epoch 14 / 20) Training Accuracy: 0.306, Validation Accuracy: 0.202
    (Epoch 15 / 20) Training Accuracy: 0.322, Validation Accuracy: 0.204
    (Epoch 16 / 20) Training Accuracy: 0.338, Validation Accuracy: 0.2
    (Epoch 17 / 20) Training Accuracy: 0.362, Validation Accuracy: 0.21
    (Epoch 18 / 20) Training Accuracy: 0.356, Validation Accuracy: 0.21
    (Epoch 19 / 20) Training Accuracy: 0.368, Validation Accuracy: 0.206
    (Epoch 20 / 20) Training Accuracy: 0.382, Validation Accuracy: 0.206

```python
# Plot train and validation accuracies of the two models
train_accs = []
val_accs = []
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    train_accs.append(curr_dict["train_acc_hist"][-1])
    val_accs.append(curr_dict["val_acc_hist"][-1])

plt.subplot(3, 1, 1)
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    plt.plot(curr_dict["train_acc_hist"], 'o', label='%.2f dropout' % dropout_p)
plt.title('Train accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    plt.plot(curr_dict["val_acc_hist"], 'o', label='%.2f dropout' % dropout_p)
plt.title('Val accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

### Inline Question: Describe what you observe from the above results and graphs
#### Ans:

From the last graph: A higher dropout prevents overtraining and hence the validation accuracy is consitently higher for higher dropout.
From the penultimate graph: Rate of convergence of adam optimizer appears to be higher than all other optimizers with a higher training accuracy per epoch. This is followed by RMSprop optimizer. SGD with momentum has higher convergence rate than vanilla SGD.

## Plot the Activation Functions
In each of the activation function, use the given lambda function template to plot their corresponding curves.

```python
left, right = -10, 10
X  = np.linspace(left, right, 100)
XS = np.linspace(-5, 5, 10)
lw = 4
alpha = 0.1
elu_alpha = 0.5
selu_alpha = 1.6732
selu_scale = 1.0507

#########################
####### YOUR CODE #######
#########################
sigmoid = lambda x: 1/(1+np.exp(-x))
leaky_relu = lambda x: np.maximum(alpha*x, x)
relu = lambda x: np.maximum(0, x)
elu = lambda x: np.where(x >=0, x , elu_alpha*(np.exp(x)-1))
selu = lambda x: np.where(x >=0, selu_scale*x , selu_scale*selu_alpha*(np.exp(x)-1))
tanh = lambda x: (np.exp(x)-np.exp(-x))/(np.exp(x)+np.exp(-x))
#########################
### END OF YOUR CODE ####
#########################

activations = {
    "Sigmoid": sigmoid,
    "LeakyReLU": leaky_relu,
    "ReLU": relu,
    "ELU": elu,
    "SeLU": selu,
    "Tanh": tanh
}

# Ground Truth activations
GT_Act = {
    "Sigmoid": [0.00669285092428, 0.0200575365379, 0.0585369028744, 0.158869104881, 0.364576440742,
                0.635423559258, 0.841130895119, 0.941463097126, 0.979942463462, 0.993307149076],
    "LeakyReLU": [-0.5, -0.388888888889, -0.277777777778, -0.166666666667, -0.0555555555556,
                  0.555555555556, 1.66666666667, 2.77777777778, 3.88888888889, 5.0],
    "ReLU": [-0.0, -0.0, -0.0, -0.0, -0.0, 0.555555555556, 1.66666666667, 2.77777777778, 3.88888888889, 5.0],
    "ELU": [-0.4966310265, -0.489765962143, -0.468911737989, -0.405562198581, -0.213123289631,
            0.555555555556, 1.66666666667, 2.77777777778, 3.88888888889, 5.0],
    "SeLU": [-1.74618571868, -1.72204772347, -1.64872296837, -1.42598202974, -0.749354802287,
             0.583722222222, 1.75116666667, 2.91861111111, 4.08605555556, 5.2535],
    "Tanh": [-0.999909204263, -0.999162466631, -0.992297935288, -0.931109608668, -0.504672397722,
             0.504672397722, 0.931109608668, 0.992297935288, 0.999162466631, 0.999909204263]
}

for label in activations:
    fig = plt.figure(figsize=(4,4))
    ax = fig.add_subplot(1, 1, 1)
    ax.plot(X, activations[label](X), color='darkorchid', lw=lw, label=label)
    assert rel_error(activations[label](XS), GT_Act[label]) < 1e-9, \
           "Your implementation of {} might be wrong".format(label)
    ax.legend(loc="lower right")
    ax.axhline(0, color='black')
    ax.axvline(0, color='black')
    ax.set_title('{}'.format(label), fontsize=14)
    plt.xlabel(r"X")
    plt.ylabel(r"Y")
    plt.show()
```

# Phew! You're done for problem 1 now, but 3 more to go... LOL


# 2022-02-16 QR Stability

* Sahitya's office hours: Friday 11-12:30

## Last time

* Gram-Schmidt process
* QR factorization

## Today
* Stability and ill conditioning
* Intro to performance modeling
* Classical vs Modified Gram-Schmidt
* Right vs left-looking algorithms

```julia
using LinearAlgebra
using Plots
using Polynomials
default(linewidth=4, legendfontsize=12)

function vander(x, k=nothing)
    if isnothing(k)
        k = length(x)
    end
    m = length(x)
    V = ones(m, k)
    for j in 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end
```

    vander (generic function with 2 methods)

# Gram-Schmidt orthogonalization

Suppose we're given some vectors and want to find an orthogonal basis for their span.

$$ \Bigg[ a_1 \Bigg| a_2 \Bigg] = \Bigg[ q_1 \Bigg| q_2 \Bigg] \begin{bmatrix} r_{11} & r_{12} \\ 0 & r_{22} \end{bmatrix} $$

# A naive algorithm

```julia
function gram_schmidt_naive(A)
    m, n = size(A)
    Q = zeros(m, n)
    R = zeros(n, n)
    for j in 1:n
        v = A[:,j]
        for k in 1:j-1
            r = Q[:,k]' * v
            v -= Q[:,k] * r
            R[k,j] = r
        end
        R[j,j] = norm(v)
        Q[:,j] = v / R[j,j]
    end
    Q, R
end
```

    gram_schmidt_naive (generic function with 1 method)

```julia
x = LinRange(-1, 1, 20)
A = vander(x, 20)
Q, R = gram_schmidt_naive(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A);
```

    norm(Q' * Q - I) = 1.073721107832196e-8
    norm(Q * R - A) = 8.268821431611631e-16

# What do orthogonal polynomials look like?

```julia
x = LinRange(-1, 1, 50)
A = vander(x, 6)
Q, R = gram_schmidt_naive(A)
plot(x, Q)
```

What happens if we use more than 50 values of $x$? Is there a continuous limit?

# Solving equations using $QR = A$

If $A x = b$ then $Rx = Q^T b$. (Why is it easy to solve with $R$?)

```julia
x1 = [-0.9, 0.1, 0.5, 0.8] # points where we know values
y1 = [1, 2.4, -0.2, 1.3]
scatter(x1, y1)
A = vander(x1, 3)
Q, R = gram_schmidt_naive(A)
p = R \ (Q' * y1)
p = A \ y1
plot!(x, vander(x, 3) * p)
```

# How accurate is it?

```julia
m = 10
x = LinRange(-1, 1, m)
A = vander(x)
Q, R = gram_schmidt_naive(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A)
```

    norm(Q' * Q - I) = 2.2794113434933815e-13
    norm(Q * R - A) = 3.6437542961698333e-16

    3.6437542961698333e-16

# A variant with more parallelism

\begin{align}
(I - q_2 q_2^T) (I - q_1 q_1^T) v &= (I - q_1 q_1^T - q_2 q_2^T + q_2 q_2^T q_1 q_1^T) v \\
&= \Bigg( I - \Big[ q_1 \Big| q_2 \Big] \begin{bmatrix} q_1^T \\ q_2^T \end{bmatrix} \Bigg) v
\end{align}

```julia
function gram_schmidt_classical(A)
    m, n = size(A)
    Q = zeros(m, n)
    R = zeros(n, n)
    for j in 1:n
        v = A[:,j]
        R[1:j-1,j] = Q[:,1:j-1]' * v
        v -= Q[:,1:j-1] * R[1:j-1,j]
        R[j,j] = norm(v)
        Q[:,j] = v / R[j,j]
    end
    Q, R
end
```

    gram_schmidt_classical (generic function with 1 method)

```julia
norm([0 1; 1 0])
```

    1.4142135623730951

```julia
m = 10
x = LinRange(-1, 1, m)
A = vander(x, m)
Q, R = gram_schmidt_classical(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A)
```

    norm(Q' * Q - I) = 6.339875256299394e-11
    norm(Q * R - A) = 1.217027619812654e-16

    1.217027619812654e-16

```julia

```

```julia

```

# Why does order of operations matter?

\begin{align}
(I - q_2 q_2^T) (I - q_1 q_1^T) v &= (I - q_1 q_1^T - q_2 q_2^T + q_2 q_2^T q_1 q_1^T) v \\
&= \Bigg( I - \Big[ q_1 \Big| q_2 \Big] \begin{bmatrix} q_1^T \\ q_2^T \end{bmatrix} \Bigg) v
\end{align}
is not exact in finite arithmetic.

# We can look at the size of what's left over

We project out the components of our vectors in the directions of each $q_j$.

```julia
x = LinRange(-1, 1, 20)
A = vander(x)
Q, R = gram_schmidt_classical(A)
scatter(diag(R), yscale=:log10)
```

# The next vector is almost linearly dependent

```julia
x = LinRange(-1, 1, 20)
A = vander(x)
Q, _ = gram_schmidt_classical(A)
#Q, _ = qr(A)
v = A[:,end]
@show norm(v)
scatter(abs.(Q[:,1:end-1]' * v), yscale=:log10)
```

    norm(v) = 1.4245900685395503

# Cost of Gram-Schmidt?

* We'll count flops (addition, multiplication, division*)
* Inner product $\sum_{i=1}^m x_i y_i$?
* Vector "axpy": $y_i = a x_i + y_i$, $i \in [1, 2, \dotsc, m]$.
* Look at the inner loop:
```julia
for k in 1:j-1
    r = Q[:,k]' * v
    v -= Q[:,k] * r
    R[k,j] = r
end
```

# Counting flops is a bad model

* We load a single entry (8 bytes) and do 2 flops (add + multiply). That's an **arithmetic intensity** of 0.25 flops/byte.
* Current hardware can do about 10 flops per byte, so our best algorithms will run at about 2% efficiency.
* Need to focus on memory bandwidth, not flops.

## Dense matrix-matrix mulitply

* [BLIS project](https://github.com/flame/blis/)
* [Analytic modeling](https://www.cs.utexas.edu/users/flame/pubs/TOMS-BLIS-Analytical.pdf)

# Inherent data dependencies

$$ \Bigg[ q_1 \Bigg| q_2 \Bigg| q_3 \Bigg| q_4 \Bigg| q_5 \Bigg] \begin{bmatrix} r_{11} & r_{12} & r_{13} & r_{14} & r_{15} \\
 & r_{22} & r_{23} & r_{24} & r_{25} \\
& & r_{33} & r_{34} & r_{35} \\
&&& r_{44} & r_{45} \\
&&& & r_{55}
\end{bmatrix}
$$

# Right-looking modified Gram-Schmidt

```julia
function gram_schmidt_modified(A)
    m, n = size(A)
    Q = copy(A)
    R = zeros(n, n)
    for j in 1:n
        R[j,j] = norm(Q[:,j])
        Q[:,j] /= R[j,j]
        R[j,j+1:end] = Q[:,j]'*Q[:,j+1:end]
        Q[:,j+1:end] -= Q[:,j]*R[j,j+1:end]'
    end
    Q, R
end
```

    gram_schmidt_modified (generic function with 1 method)

```julia
m = 20
x = LinRange(-1, 1, m)
A = vander(x, m)
Q, R = gram_schmidt_modified(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A)
```

    norm(Q' * Q - I) = 8.486718528276085e-9
    norm(Q * R - A) = 8.709998074379606e-16

    8.709998074379606e-16

# Classical versus modified?

* Classical
  * Really unstable, orthogonality error of size $1 \gg \epsilon_{\text{machine}}$
  * Don't need to know all the vectors in advance
* Modified
  * Needs to be right-looking for efficiency
  * Less unstable, but orthogonality error $10^{-9} \gg \epsilon_{\text{machine}}$

```julia
m = 10
x = LinRange(-1, 1, m)
A = vander(x, m)
Q, R = qr(A)
@show norm(Q' * Q - I)
```

    norm(Q' * Q - I) = 2.1776697623015113e-15

    2.1776697623015113e-15


>>> Work in Progress (Following are the lecture notes of Prof Andrew Ng - CS229 - Stanford. This is my interpretation of his excellent teaching and I take full responsibility of any misinterpretation/misinformation provided herein.)

# Lecture Notes

#### Outline:
- Naive Bayes
  - Laplace smoothing
  - Element models
- Comments on applying ML
- SVM

## Naive Bayes
- is a generative type of algorithm
- To build generative model, P(x|y) and P(y) needs to be modeled. Gaussian discriminative model uses Gaussian and Bernoulli respectively to model this. Naive Bayes uses a different model.
> - $ x_{j} = \mathbb 1$ {indicator - word j appears in email or not}
> - $ P(x|y) = \prod\limits_{j=1}^{n}P(x_{j}|y)$ - NB uses the product of conditional probabilities of individual features given in the class label y
- Parameters of NB model are:
  > - $P(y=1) = \phi_{y} $ - the class prior for y=1
  > - $P(x_{j}=1|y=0) = \phi_{j|y=0}$ - chances of the word appearing in non-spam email.
  > - $P(x_{j}=1|y=1) = \phi_{j|y=1}$ - chances of the word appearing in spam email.

### Maximum likelihood estimate
> $\phi_{y} = \frac{\sum\limits_{i=1}^{m}\mathbb 1 \{y^{(i)}=1\}}{m}$
> $\phi_{j|y=0} = \frac{\sum\limits_{i=1}^{m}\mathbb 1 \{x_{j}^{(i)}=1, y^{(i)}=0\}}{\sum\limits_{i=1}^{m}\mathbb 1 \{y^{(i)}=0\}}$

### Laplace smoothing
- ML conference papers - NIPS - Neural Information Processing Systems

- __NB breaks__ - because the probability of some event that you have not seen is trained as 0, which is statistically wrong
- the classifier gives wrong result, when it gets such event for the first time
  - instead using __Laplace smoothing__ helps this problem
    - add 1 each to pass and fail scenario

#### Maximum likelihood estimate
> $\phi_{x=i} = \frac{\sum\limits_{j=1}^{m}\mathbb 1 \{x^{(i)}=j\} + 1} {m + k}$
  - k is the size of dictionary
> $\phi_{i|y=0} = \frac{\sum\limits_{i=1}^{m}\mathbb 1 \{x_{j}^{(i)}=1, y^{(i)}=0\} + 1}{\sum\limits_{i=1}^{m}\mathbb 1 \{y^{(i)}=0\} + 2}$

## Event models for text classification
- The functionality can be generalized from binary to multinomial features instead
  - Multivariate Bernoulli event
  - Multinomial event

- We can discretize the size of house and transform a Bernoulli event to Multinomial event
  - a rule of thumb is to discretize variables into 10 buckets

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

## NB variation
- for text classification
  - if the email contains text "Drugs! Buy drugs now",
    - the feature vector will make binary vector of all words appearing in the email
    - it will lose the information that the word "drug" appeared twice
    - each featture only stores information $x_{j} \in \{0,1\}$
    - instead of making a feature vector of 10000 words of dictionary, we can make a feature vector of 4 words(as above) holding word index and $x_{j} \in \{1, 2, ... ,10000\}$
      - algorithm containing feature vector of 10000 is called __"Multivariate Bernoulli algorithm"__
      - algorithm containing feature vector of 4 is called __"Multinomial algorithm"__
        - Andrew McCallum used these 2 names

### NB advantage
- quick to implement
- computationally efficient
- no need to implement gradient descent
- easy to do a quick and dirty type of work
- SVM or logistic regression does better work in classification problems
- NB or GDA does not result in very good classification, but is very quick to implement, it is quick to train, it is non-iterative

## Support Vector Machines (SVM)
- find classification in the form of non linear decision boundaries
- SVM does not have many parameters to fiddle with
  - they have very robust packages
  - and does not have parameters like learning rate, etc to fiddle with

<br>

- Let the non-linear feature variables be mapped in vector form as below. This non-linear function can be viewed as a linear function over the variables $\phi(x)$
- derive an algorithm that can take input features of the $x_{1}, x_{2}$ and map them to much higher dimensional set of feature and then apply linear classifier, similar to logistic regression but different in details. This allows us to learn very non-linear decision boundaries.

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

### Outline for SVM
- Optimal margin classifier (separable case)

- Kernel
  - Kernels will allow us to fit in a large set of features
  - How to take feature vector say $\mathbb R^{2}$ and map it to $\mathbb R^{5}$ or $\mathbb R^{10000}$ or $\mathbb R^{\inf}$, and train the algorithm to this higher feature. Map 2-dimensional feature space to infinite dimensional set of features. What it helps us is relieve us from lot of burden of manually picking up features. ($x^{2}, \space x^{3}, \space \sqrt{x}, \space x_{1}x_{2}^{2/3}, ...$)

- Inseparable case

### Optimal margin classifier (separable case)
- Separable case means data is separable

#### Functional margin of classifier

- How confidently and accurately do you classify an example
- Using binary classification and logistic regression
- In logistic classifier:
> $h_{\theta}(x) = g(\theta^{T}x)$
- If you turn this into binary classification, if you have this algorithm predict not a probability but predict 0 or 1,
> - Predict "1", if $\theta^{T}x > 0 \implies h_{\theta}(x) = g(\theta^{T}x) \ge 0.5$)
> - Predict "0", otherwise
  - If $y^{(i)} = 1$, hope that $\theta^{T}x^{(i)} \gg 0$
  - If $y^{(i)} = 0$, hope that $\theta^{T}x^{(i)} \ll 0$
    - this implies that the prediction is very correct and accurate

#### Geometric margin
- assuming data is linearly separable,
- and there are two lines that separates the true and false variables
- the line that has a bigger separation or geometric margin meaning a physical separation from the training examples is a better choice

- We need to prove the optimal classifier is the algorithm that tries to maximize the geometric margin
- what SVM and low dimensional classifier do is pose an optimization problem to try and find the line that classify these examples to find bigger separation margin

#### Notation
- Labels: $y \in \{-1, +1\} $ to denote class labels
- SVM will generate hypothesis output value as {-1, +1} instead of probability as in logistic regression
- g(z) = $\begin{equation}
    \begin{cases}
      +1 & \text{if z $\ge$ 0}\\
      -1 & \text{otherwise}
    \end{cases}
\end{equation}$
  - instead of smooth transition, here we have a hard transition

Previously in logistic regression:
> $h_{\theta}(x) = g(\theta^{T}x)$
- where x is $\mathbb R^{n+1} $ and $x_{0} = 1$

In SVM:
> $h_{W,b}(x) = g(W^{T}x + b)$
- where x is $\mathbb R^{n} $ and b is $\mathbb R $ and drop $x_{0}=1$
- The term

Other way to think about is:
> $
\begin{bmatrix}
\theta_{0}\\
\theta_{1}\\
\theta_{2}\\
\theta_{3}
\end{bmatrix}
$
- where $\theta_{0}$ corresponds to b and $\theta_{1}, \theta_{2}, \theta_{3}$ corresponds to W

#### Functional margin of hyperplane (cont)
- Functional margin __wrt single training example__
  - Functional margin of hyperplane defined by $(w, b)$ wrt a single training example $(x^{(i)}, y^{(i)})$ is
  > $\hat{\gamma}^{(i)} = y^{(i)} (w^{T}x^{(i)} + b)$
    - If $y^{(i)} = +1$, hope that $(w^{T}x^{(i)} + b) \gg 0$
    - If $y^{(i)} = -1$, hope that $(w^{T}x^{(i)} + b) \ll 0$
  - Combining these two statements:
    - we hope that $\hat{\gamma}^{(i)} \gg 0$
  - If $\hat{\gamma}^{(i)} \gt 0$,
    - implies $h(x^{(i)}) = y^{(i)}$
  - In logistic regression
    - $\gt 0$ means the prediction is atleast a little bit above 0.5 or little bit below 0.5
    - $\gg 0$ means the prediction is either very close to 1 or very close to 0
- Functional margin __wrt entire training set__ (how well are you doing on the worst example in your training set)
  > $\hat{\gamma} = \min\limits_{i} \hat{\gamma}^{(i)}$
  - where i = 1,..,m - all the training examples
  - here the assumption is training set is linearly separable
    - we can assume this kind of worst-case notion because we are assuming that the boundary  is linearly separable
  - we can normalize (w, b), which does not change the classification boundary, it simply rescales the parameters
  > $(w, b) \rightarrow (\frac{w}{\Vert w \Vert}, \frac{b}{\Vert b \Vert})$
  - classification remains the same, with any rescale number

#### Geometric margin of hyperplane (cont)
- Geometric margin __wrt single training example__
  - Upper half of hyperplane has positive cases and the lower half negative cases
  - Let $(x^{(i)}, y^{(i)})$ be a training example which the classifier is classifying correctly, by predicting it as $h_{w,b}(x) = +1$. It predicts the lower half as $h_{w,b}(x) = -1$.
  - Let the distance from the plane to the training example be the geometric margin
  - Geometric margin of hyperplane (w,b) wrt $(x^{(i)}, y^{(i)})$ be
  > $\hat{\gamma} = \frac{(y^{(i)})(w^{T}x^{(i)} + b)}{\Vert w \Vert}$
    - This is the Euclidean distance between training example and decision boundary
    - For positive examples, $y^{(i)}$ is +1 and the equation reduces to
    > $\hat{\gamma} = \frac{(w^{T}x^{(i)} + b)}{\Vert w \Vert}$

$\tiny{\text{YouTube-Stanford-CS229-Andrew Ng}}$

#### Relationship between functional and geometric margin
> $\gamma^{(i)} = \frac{\hat{\gamma}^{(i)}}{\Vert w \Vert}$
> i.e., $\text{Geometric Margin} = \frac{\text{Functional Margin}}{\text{Norm of w}}$

#### Geometric margin of hyperplane (cont)
- Geometric margin __wrt entire training example__
  > $\gamma = \min\limits_{i} \gamma^{(i)}$

#### Optimal margin classifier (cont)
- what the optimal margin classifier does is to choose w,b to maximize $\gamma$
> $\max_{\gamma, w, b} \text{s.t.} \frac{(y^{(i)})(w^{T}x^{(i)} + b)}{\Vert w \Vert} \ge \gamma$ for i=1,...,m
  - subject to for every single training example  must have the geometric margin greater than or equal to gamma
  - this causes to maximize the worst-case geometric margin
  - this is not a convex optimization problem and cannot be solved using gradient descent or local optima
  - but this can be re-written/reformulated into a equivalent problem which is minimizing norm of w subject to the geometric margin
> $\min_{w,b} \Vert w \Vert ^{2}   \\
\text{s.t. } y^{(i)}(w^{T}x^{(i)} + b) \ge 1$
    - This is a convex optimization problem and can be solved using optimization packages
- All this study is for linear separable case only
  - this is a baby SVM
- Once we learn kernels and apply kernels with optimal margin classifier, we get solution for SVM


```python
%matplotlib inline
```

```python
from IPython.display import display

import numpy
from numpy import linspace
from sympy import lambdify, init_printing
from sympy import symbols, pi, sqrt, exp

from matplotlib import pyplot

init_printing()
```

# Logistic function

$$ S(x) = \frac{1}{1 + e^{-x}} = \frac{e^x}{e^x + 1} $$

# Normal distribution

$$ f(x | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$

- $\mu$ is the mean or expectation of the distribution
- $\sigma$ is the standard deviation
- $\sigma^2$ is the variance

# Likelihood ratio of two Normals with same variance

$$ f(x | \mu_0, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(x-\mu_0)^2}{2\sigma^2}} $$
$$ f(x | \mu_1, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(x-\mu_1)^2}{2\sigma^2}} $$

The likelihood ratio is

$$
\begin{align}
LR(x) =& \frac{f(x|\mu_0, \sigma^2)}{f(x|\mu_1, \sigma^2)} \\
      =& \frac{C exp[-\frac{(x-\mu_0)^2}{2\sigma^2}]}
      {C exp\left[-\frac{(x-\mu_1)^2}{2\sigma^2}\right]} \\
      =& exp\left[\frac{-(x-\mu_0)^2 + (x-\mu_1)^2}{2\sigma^2}\right] \\
      =& exp\left[\frac{-(x^2 + \mu_0^2 - 2\mu_0x) + (x^2 + \mu_1^2 - 2\mu_1x)}{2\sigma^2}\right] \\
      =& exp\left[\frac{-\mu_0^2 + 2\mu_0x + \mu_1^2 - 2\mu_1x}{2\sigma^2}\right] \\
      =& exp\left[\frac{2(\mu_0 - \mu_1)x  + \mu_1^2 - \mu_0^2}{2\sigma^2}\right] \\
      =& exp\left[\frac{2(\mu_0 - \mu_1)x - (\mu_0^2 - \mu_1^2)}{2\sigma^2}\right] \\
      =& exp\left[\frac{2(\mu_0 - \mu_1)x + (\mu_1 + \mu_0)(\mu_0 - \mu_1)}{2\sigma^2}\right] \\
      =& exp\left[\frac{(\mu_0 - \mu_1)(2x - (\mu_0 + \mu_1))}{2\sigma^2}\right] \\
      =& exp\left[\frac{(\mu_0 - \mu_1)}{\sigma^2}\left(x - \frac{(\mu_0 + \mu_1)}{2}\right)\right] \\
      =& exp\left[\gamma(x - m)\right] \\
\end{align}
$$

where

- $\gamma = \frac{(\mu_0 - \mu_1)}{\sigma^2}$ and
- $m = \frac{(\mu_0 + \mu_1)}{2}$

```python
from sympy import simplify

x, mu0, mu1, sigma = symbols('x mu0 mu1 sigma')

pdf_normal_0 = (1/sqrt(2*pi*sigma**2))*exp(-((x-mu0)**2)/(2*sigma**2))
pdf_normal_1 = (1/sqrt(2*pi*sigma**2))*exp(-((x-mu1)**2)/(2*sigma**2))

display(simplify(pdf_normal_0))
display(simplify(pdf_normal_0/(pdf_normal_0 + pdf_normal_1)))
```

```python
sigmoid = 1 / (1 + exp(-x))

display(sigmoid)
```

```python
from sympy import simplify

x, mu_0, sigma_0, mu_1, sigma_1 = symbols('x mu_0 sigma_0 mu_1 sigma_1')

y_0 = (1/sqrt(2*pi*sigma_0**2))*exp(-((x-mu_0)**2)/(2*sigma_0**2))
y_1 = (1/sqrt(2*pi*sigma_1**2))*exp(-((x-mu_1)**2)/(2*sigma_1**2))

display(simplify(y_0))
display(simplify(y_0/(y_0 + y_1)))
```

```python

```

```python
pdf_normal_0 = lambdify((x, mu_0, sigma_0), y_0, "numpy")
pdf_normal_1 = lambdify((x, mu_1, sigma_1), y_1, "numpy")

a = linspace(-5,5,100)
m0 = 0
s0 = 1
m1 = 2
s1 = 1

pyplot.plot(a, pdf_normal_0(a, m0, s0))
pyplot.plot(a, pdf_normal_1(a, m1, s1))
pyplot.plot(a, pdf_normal_0(a, m1, s1)/(pdf_normal_0(a, m0, s0) + pdf_normal_1(a, m1, s1)))
pyplot.grid(True)
```

# Expected brier score

Let's visualise the error space of a logistic regression with only one parameter $w$

$$  \frac{1}{1 + exp(-xw)} $$

```python
from numpy import exp as e

def brier_score(x1, x2):
    return numpy.mean((x1 - x2)**2)

w_list = linspace(-5, 5)
errors = []
for w in w_list:
    errors.append(brier_score(1/(1 + e(-a*w)),
                              pdf_normal_0(a, m1, s1)/(pdf_normal_0(a, m0, s0) + pdf_normal_1(a, m1, s1))))

pyplot.plot(w_list, errors)
min_idx = numpy.argmin(errors)
print((w_list[min_idx], errors[min_idx]))
pyplot.annotate("w = {:.2f}, BS = {:.2f}".format(w_list[min_idx], errors[min_idx]), (w_list[min_idx], errors[min_idx]),
                (1.5, 0.1), arrowprops={'arrowstyle': '->'})
```

```python

```

```python

```


# 15 - Symbolic Computations

```python
from sympy import *
init_printing()
```

## What are symbolic calculations?

```python
x = symbols('x')
```

```python
f = Lambda(x, 1/(x**2 + x + 1))
```

```python
integrate(f(x),x)
```

```python
pf = Lambda(x, integrate(f(x),x))
```

```python
diff(pf(x),x)
```

```python
simplify(diff(pf(x),x))
```

```python
simplify(pf(4)-pf(0))
```

```python
(pf(4)-pf(0)).evalf()
```

```python
from scipy.integrate import quad
```

```python
quad(lambda x : 1/(x**2+x+1),a=0, b=4)
```

## Basic elements of SymPy

```python
x,y,mass,torque, row = symbols('x y mass torque i')
```

```python
symbol_list=[symbols(l) for l in 'x y mass torque'.split()]
x, y, mass, torque = symbol_list
```

```python
row_index=symbols('i',integer=True)
row_index**2 # returns i**2
```

```python
x, y, mass, torque=symbols('x y mass torque')
symbol_list=[symbols(l) for l in 'x y mass torque'.split()]
x, y, mass, torque = symbol_list
```

```python
integervariables = symbols('i:l', integer=True)
dimensions = symbols('m:n', integer=True)
realvariables = symbols('x:z', real=True)
```

```python
integervariables
```

```python
A = symbols('A0:3(0:4)')
A
```

### Numbers

```python
1/3  # returns 0.3333333333333333
sympify(1)/sympify(3)  # returns '1/3'
```

```python
Rational(1,3)
```

### Functions

```python
f, g = symbols('f g', cls=Function)
```

```python
f = Function('f')
g = Function('g')
```

```python
x = symbols('x')
f, g = symbols('f g', cls=Function)
diff(f(x*g(x)),x)
```

```python
x = symbols('x:3')
f(*x)
```

```python
[diff(f(*x),xx) for xx in x]
```

```python
[f(*x).diff(xx) for xx in x]
```

```python
x = symbols('x')
f(x).series(x,0,n=4)
```

### Elementary Functions

```python
x = symbols('x')
simplify(cos(x)**2 + sin(x)**2)  # returns 1
```

```python
atan(x).diff(x) - 1./(x**2+1)  # returns 0
```

```python
import scipy as sp
import sympy as sym
# working with numbers
x=3
y=sp.sin(x)
# working with symbols
x=sym.symbols('x')
y=sym.sin(x)
```

###  Lambda functions

```python
C,rho,A,v=symbols('C rho A v')
# C drag coefficient, A coss-sectional area, rho density
# v speed
f_drag = Lambda(v,-Rational(1,2)*C*rho*A*v**2)
f_drag
```

```python
x = symbols('x')
f_drag(2)
f_drag(x/3)
```

```python
y = symbols('y')
t=Lambda((x,y),sin(x) + cos(2*y))
```

```python
t(pi,pi/2)  # returns -1
```

```python
p=(pi,pi/2)
t(*p)   # returns -1
```

```python
F=Lambda((x,y),Matrix([sin(x) + cos(2*y), sin(x)*cos(y)]))
```

```python
F(x,y).jacobian((x,y))
```

```python
x=symbols('x:2')
F=Lambda(x,Matrix([sin(x[0]) + cos(2*x[1]),sin(x[0])*cos(x[1])]))
F(*x).jacobian(x)
```

# Symbolic Linear Algebra

```python
phi=symbols('phi')
rotation=Matrix([[cos(phi), -sin(phi)],
                 [sin(phi), cos(phi)]])
```

```python
phi=symbols('phi')
rotation=Matrix([[cos(phi), -sin(phi)],
                 [sin(phi), cos(phi)]])

```

```python
simplify(rotation.T - rotation.inv())
```

```python
M = Matrix(3,3, symbols('M:3(:3)'))
M
```

```python
def toeplitz(n):
    a = symbols('a:'+str(2*n))
    f = lambda i,j: a[i-j+n-1]
    return Matrix(n,n,f)
```

```python
toeplitz(5)
```

```python
a=symbols('a')
M[0,2]=0  # changes one element
M[1,:]=Matrix(1,3,[1,2,3]) # changes an entire row
M
```

```python
---------------------so far-------------------------------------------
```

```python
t=Lambda((x,y),sin(x) + cos(2*y))
t
```

```python
t(pi,pi/2)
```

```python
p=(pi,pi/2)
t(*p)
```

```python
F=Lambda((x,y),Matrix([sin(x) + cos(2*y), sin(x)*cos(y)]))
F
```

```python
F(x,y).jacobian((x,y))
```

```python
x=symbols('x:2')
F=Lambda(x,Matrix([sin(x[0]) + cos(2*x[1]),sin(x[0])*cos(x[1])]))
F
```

```python
F(*x)
```

```python
F(*x).jacobian(x)
```

```python
phi=symbols('phi')
rotation=Matrix([[cos(phi), -sin(phi)],[sin(phi), cos(phi)]])
rotation
```

```python
simplify(rotation.transpose()*rotation)
```

```python
simplify(rotation.transpose()*rotation -eye(2))
```

```python
simplify(rotation.T - rotation.inv())
```

```python
M = Matrix(3,3, symbols('M:3(:3)'))
M
```

```python
M.subs({M[2,1]:0})
```

```python
M[0,2]=0  # changes one element
M[1,:]=Matrix(1,3,[1,2,3]) # changes an entire row
M
```

```python
A = Matrix(3,3,symbols('A1:4(1:4)'))
b = Matrix(3,1,symbols('b1:4'))
x = A.LUsolve(b)
x
```

```python
print(latex(simplify(x)))
```

```python
A*x-b
```

```python
simplify(A*x-b)
```

```python
A.det()
```

```python
simplify(x*A.det())
```

```python
x, a = symbols('x a')
b = x + a
```

```python
b
```

```python
x, a = symbols('x a')
b = x + a
c=b.subs(x,0)
d=c.subs(a,2*a)
```

```python
c, d
```

```python
b.subs(x,0)
```

```python
x, a = symbols('x a')
b = x + a
b.subs(x,0)
b.subs({x:0})
```

```python
d=b.subs({x:0,a:2*a})
```

```python
d
```

```python
b.subs({x:2*x, a:a*x})
```

```python
b.subs({a:a*x, x:2*x})
```

```python
y = symbols('y')
```

```python
b.subs({a:a*y, x:2*x, y:a/y})
```

```python
print(latex(b.subs({y:a/y, a:a*y, x:2*x})))
```

```python
c=b.subs({y:a/y})
c.subs({a:a/y, x:2*x})
```

```python
x, y, n = symbols('x y n')
b = acos(n*cos(x))
b.subs(cos(x),y)
```

```python
x y n = symbols('x y n')
b = acos(n*cos(x))
b.subs(cos(x),y)
```

```python
Tn = cos(n*acos(t))
```

```python
alpha = symbols('alpha',real=True)
Tn=Tn.subs(t,cos(alpha))
simplify(Tn)
```

```python
simplify(Tn)
```

```python
n, alpha = symbols('n alpha')
b = (2*cos(n*alpha)*cos(alpha*(n-1))-cos(alpha*(n-2)))**2
b.expand().subs(2*cos(alpha)*cos((n-1)*alpha)-cos((n-2)*alpha),cos(n*alpha))
```

```python
b.expand()
```

```python
b
```

```python
x, a = symbols('x a')
b = x + a
b.subs([(y,a/y), (a,a*y), (x,2*x)])
```

```python
T=toeplitz(6)
```

```python
t=symbols('t')
x=[0,t,1]
# The Vandermone Matrix
V = Matrix([[0, 0, 1], [t**2, t, 1], [1, 1,1]])
y = Matrix([0,1,-1])  # the data vcetor
a=V.LUsolve(y) # the coefficients
# the leading coefficient as a function of the parameter
a2 = Lambda(t,a[0](t))
```

##

```python
a2
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('apw-notebook.mplstyle')
%matplotlib inline

from scipy.integrate import quad
from scipy.interpolate import interp1d
```

Computers can generate (pseudo)-random, uniformly distributed random numbers (using, e.g., the [Mersenne Twister](https://en.wikipedia.org/wiki/Mersenne_Twister)). We often want to generate random samples from distributions, either with analytic forms or complex posterior probabilities. For some distributions, there are specific algorithms that efficiently generate samples (e.g., Box-Mueller for the Gaussian) -- for named distributions, you should almost always use [scipy.stats](https://docs.scipy.org/doc/scipy-0.18.1/reference/stats.html)! -- but in other cases, you'll need to generate samples numerically. Here we'll go through a few methods for doing that.

---

# Inverse transform sampling

Only works for 1D or separable ND distributions.

Given some (probability, number, mass) density function $p(x)$ compute the cumulative distribution:
$$
F(x) = \int_{-\infty}^x \, p(z) \, {\rm d}z
$$

To generate samples from $x \sim p(x)$, generate uniform random samples $u$ and invert the cdf:
$$
x = F^{-1}(u)
$$

Let's see this in action.

### Example 1: points uniformly distributed on the surface of a unit sphere

Generate 1000 angular positions $(\phi, \theta)$ on the surface of the unit sphere, uniformly distributed over the surface.

Do this by analytically inverting the cdf, generating uniform random samples, and evaluating:

```python
phi = np.random.uniform(0, 2*np.pi, size=10000)
theta = np.arccos(2*np.random.uniform(size=10000) - 1)
```

```python
plt.figure(figsize=(5,5))
plt.scatter(np.cos(phi)*np.sin(theta),
            np.sin(phi)*np.sin(theta),
            alpha=0.1, marker='.')
```

### Example 2: points drawn from a power-law distribution

$$
\begin{align}
p(x) &= C \, x^n\\
\quad x &\in (a, b)\\
n&<0\\
C &= \frac{1 + n}{b^{1+n} - a^{1+n}}
\end{align}
$$

Generate 1000 points drawn from a power-law distribution with the following parameters:
$$
n = -2\\
a = 0.1\\
b = 10.
$$

Do this by _numerically_ computing the cdf along a grid of values over the domain $(0.1,10.)$.

Hint: use scipy's integration function `quad()` and use an interpolating function, scipy's `interp1d()`, to invert the cdf. Both are already imported

```python
a = 0.1
b = 10.
n = -2.
```

```python
def power_law_pdf(x):
    C = (1+n) / (b**(1+n) - a**(1+n))
    return C*x**n
```

```python
quad(power_law_pdf, a, b)
```

    (0.9999999999999999, 3.038479026730819e-09)

```python
x_grid = np.linspace(a, b, 1024)
power_law_cdf = [quad(power_law_pdf, a, x)[0] for x in x_grid]
```

```python
plt.plot(x_grid, power_law_cdf, marker='');
plt.xlabel(r'$x$')
plt.ylabel(r'$F(x)$')
```

```python
func = interp1d(power_law_cdf, x_grid)
```

```python
xs = func(np.random.uniform(size=10000))
```

```python
plt.hist(np.log10(xs), bins='auto');
plt.yscale('log')
```

### Example 3: radii drawn from a Hernquist profile

$$
\begin{align}
\rho(r) &= \frac{M_{tot}}{2\pi\,a^3} \, \left[\frac{r}{a} \, \left(1+\frac{r}{a}\right)^3\right] \\
M(<r) &= 2\,M_{tot} \, \frac{\left(r/a\right)^2}{2\left(1 + r/a\right)^2}
\end{align}
$$

Use whatever method you prefer to sample 1000 radii from a Hernquist profile.

```python

```

---

# Rejection sampling

Only practical for low dimensionality.

### Example 1: sampling from a mixture of two Gaussians

Use rejection sampling to generate approximate samples from the mixture of Gaussians $\mathcal(N)(\mu,\sigma^2)$ defined by (and below):
$$
p(x) = 0.5\,\mathcal{N}(1,0.25) + 0.5\,\mathcal{N}(2.5,1)
$$

```python
def gaussian(x, mu, var):
    return 1/np.sqrt(2*np.pi*var) * np.exp(-0.5*(x-mu)**2/var)

def gaussian_mixture(x):
    return 0.5*gaussian(x, 1., 0.25) + 0.5*gaussian(x, 2.5, 1)
```

First plot the pdf of the mixture

```python
x_grid = np.linspace(-2, 10, 1024)
plt.plot(x_grid, gaussian_mixture(x_grid), marker='')
```

```python
import numpy.random as rand

n_sample = 500000
x_sample = 6.*rand.random(n_sample)
y_sample = 0.5*rand.random(n_sample)

x_to_keep=[]
for i in range(n_sample):
    if y_sample[i] < gaussian_mixture(x_sample[i]):
        x_to_keep.append(x_sample[i])

plt.hist(x_to_keep,bins=120,normed=True);
```

```python
N = 500000
x_sample = np.random.uniform(-2, 6, size=N)
y_sample = np.random.uniform(0, 0.5, size=N)

idx = y_sample < gaussian_mixture(x_sample)
x = x_sample[idx]

n,_,_ = plt.hist(x, bins='auto')
```

```python

```

(Comment on adaptive rejection sampling methods)

___

# Markov Chain Monte Carlo (MCMC): Metropolis-hastings

Lose independent samples, gain in scalability to higher dimensions

The simplest MCMC algorithm is "Metropolis-Hastings". I'm not going to explain it in detail, but in pseudocode, it looks like this:

- Start from some position in the space of variables you are sampling over, $\theta_0$ with probability $\pi_0$
- Iterate from 1 to $N_{\rm steps}$:
    - Sample an offset from $\delta\theta_0$ from some proposal distribution
    - Compute a new parameter value using this offset, $\theta_{\rm new} = \theta_0 + \delta\theta_0$
    - Evaluate the probability at the new parameter vector, $\pi_{\rm new}$
    - Sample a uniform random number, $r \sim \mathcal{U}(0,1)$
    - if $\pi_{\rm new}/\pi_0 > 1$ or $\pi_{\rm new}/\pi_0 > r$:
        - store $\theta_{\rm new}$
        - replace $\theta_0,\pi_0$ with $\theta_{\rm new},\pi_{\rm new}$
    - else:
        - store $\theta_0$ again

The proposal distribution has to be chosen and tuned by hand. We'll use a spherical / uncorrelated Gaussian distribution with root-variances set by hand.

### Example 1: implement and use Metropolis-Hastings MCMC to sample from the above mixture of Gaussians

```python
from scipy.misc import logsumexp
```

```python
# First, here's a log version of the mixture of Gaussians above that
#    is less sensitive to numerical issues

def ln_gaussian(x, mu, var):
    return -0.5*np.log(2*np.pi*var) - 0.5*(x-mu)**2/var

def ln_gaussian_mixture(x):
    x = np.atleast_1d(x)

    # weights
    w = np.array([0.5, 0.5])[:,np.newaxis]

    X = np.vstack((ln_gaussian(x, 1., 0.25), ln_gaussian(x, 2.5, 1)))
    return logsumexp(X, b=w, axis=0)
```

```python
ln_gaussian_mixture(1.)
```

    array([-0.76851516])

```python
# We're sampling from a 1D distribution, so we only need to propose in 1 dimension
def sample_proposal(sigma):
    return np.random.normal(0., sigma)
```

```python
def run_metropolis_hastings_1d(p0, n_steps, ln_prob_func, proposal_sigma):
    """
    Run a Metropolis-Hastings MCMC sampler to generate samples from the input
    log-posterior function, starting from some initial parameter vector.

    Parameters
    ----------
    p0 : numeric
        Initial value.
    n_steps : int
        Number of steps to run the sampler for.
    ln_prob_func : function
        A callable object that computes the log takes a parameter vector and computes
        the log of the posterior pdf.
    proposal_sigmas : numeric
        Standard-deviation passed to the sample_proposal function.
    """

    # the objects we'll fill and return:
    chain = np.zeros(n_steps) #  value at each step
    ln_probs = np.zeros(n_steps) # log-probability value at each step

    # we'll keep track of how many steps we accept to compute the acceptance fraction
    n_accept = 0

    # evaluate the log-posterior at the initial position and store starting position in chain
    ln_probs[0] = ln_prob_func(p0)
    chain[0] = p0

    # loop through the number of steps requested and run MCMC
    for i in range(1,n_steps):
        # proposed new parameters
        step = sample_proposal(proposal_sigma)
        new_p = chain[i-1] + step

        # compute log-posterior at new parameter values
        new_ln_prob = ln_prob_func(new_p)

        # log of the ratio of the new log-posterior to the previous log-posterior value
        ln_prob_ratio = new_ln_prob - ln_probs[i-1]

        if (ln_prob_ratio > 0) or (ln_prob_ratio > np.log(np.random.uniform())):
            chain[i] = new_p
            ln_probs[i] = new_ln_prob
            n_accept += 1

        else:
            chain[i] = chain[i-1]
            ln_probs[i] = ln_probs[i-1]

    acc_frac = n_accept / n_steps
    return chain, ln_probs, acc_frac
```

```python
# Run the MCMC sampler to generate samples from the mixture of Gaussians
```

```python
chain,ln_probs,acc_frac = run_metropolis_hastings_1d(-5.,
                                                     10000,
                                                     ln_gaussian_mixture,
                                                     2.5)
```

```python
acc_frac
```

    0.4278

```python
plt.plot(chain[1000:], marker='', drawstyle='steps-mid')
```

```python
plt.hist(chain[1000:], bins='auto');
```

```python

```

notebooks/Sampling from probability distributions.ipynb


# Integral Exercise # 1

```python
import sympy as sp
x = sp.Symbol('x')
sp.integrate(3.0*x**2 +1, x)
```

$\displaystyle 1.0 x^{3} + 1.0 x$

```python
# Doing numerically
from scipy.integrate import quad
#lets define a function

def f(y):
    return 3.0*y**2 +1
#print(f(2)) -> test it if it work

i = quad(f,0,2)
print(i)
print(i[0])

```

    (10.000000000000002, 1.1102230246251568e-13)
    10.000000000000002

```python
i, error = quad(f, 0, 2)
print(i)
print(error)
```

    10.000000000000002
    1.1102230246251568e-13

```python
# try yourself and plot this with you ipynb
# play around with http://wolframalpha.com, cool site for mathematics
from IPython.display import display, Math, Latex
display(Math(r'f(x) = \int_{0}^{2\pi}e^{-x}{sin(3x)} dx'))
```

$\displaystyle f(x) = \int_{0}^{2\pi}e^{-x}{sin(3x)} dx$

```python
#Solution
from scipy.integrate import quad
import numpy as np
# import sympy as sp - this wont really work, you can and should try throwing in numbers to see why its deosn't!

def f(x):
    return np.exp(-x)*np.sin(3.0*x)
i = quad(f, 0, 2*np.pi)
print(i)
```

    (0.29943976718048754, 5.05015300411582e-13)

## Exercise 2:  Simple derivatives or as I'd prefer to call differentials

```python
# Finding derivative the simple math way

def f(x):
    return x**2

def differential(x):
    h =  1.0/1000.0
    rise = f(x + h) - f(x)
    run = h
    slope = rise / run
    return slope

print(differential(1))
print(differential(2))
print(differential(3))
print(differential(4))
```

    2.0009999999996975
    4.000999999999699
    6.000999999999479
    8.0010000000037

```python
# Integrals are essentially the area under the sinusoidal curve tha covers
# all reactangular boxes undr the curve covered
def integral(startx, endx, numberofrectangles):
#     width = ((endx) - (startx)) /numberofrectangles = Makes no difference in Py 3.6, with or without float
    width = (float(endx) - float(startx)) /numberofrectangles
    runningSum = 0
    for i in range(numberofrectangles):
        height = f(startx + i*width)
        area = height*width
        runningSum += area
    return runningSum
print('Area is: ', integral(0, 1, 10))
print('Area is: ', integral(0, 1, 1000))
print('Area is: ', integral(0, 1, 100000))
```

    Area is:  0.2850000000000001
    Area is:  0.33283350000000034
    Area is:  0.33332833334999745

## Exercise 3: More integration, interpolation and curve fitting

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.integrate import *
```

#### You've seen one example of integra before, but lets try another one

Try solving this:
$f(x) = \int_{0}^{1}3x^2 + {1} dx$
#### Also BTW just use dollar sign before and after your formula as this $your formula$
to display formulas in your iPython Notebook use the inline function such as
$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$

```python
def f(x):
    return 3.0*x**2 + 1.0
xlow = 0
xhigh = 1
i, err = quad(f, xlow, xhigh)
print("Integral is: ", i)
print("Error is :", err)
# Use help(quad) to see what more it can do
```

    Integral is:  2.0
    Error is : 2.220446049250313e-14

#### Let's try interpolation

```python
from scipy.interpolate import * # We want interpid function
plt.xkcd()

# Here basically you have a series of data points.
# You want to estimate data points that are between given points
# you want to *linearly* interpolate between the given function (meaning drawing a straight line)
# Or, you can use higher order, called polynomial interpolation to approximate the curve between the data points
# Cubic spline is a popular example of this

x_givenpoint = np.linspace(0, 10, 10)
y_givenpoint = np.cos(x_givenpoint**2.0/8.0)

# Now we create an array
xx = np.linspace(0, 10, 1000)
yy = np.cos(xx**2.0/8.0)

plt.plot(x_givenpoint, y_givenpoint, 'x', label = 'given data')
plt.plot(xx, yy, ':', label='Perfect')
plt.ylabel('x')
plt.xlabel('y')
plt.legend(loc='best')

```

```python
x_i = np.linspace(0, 10, 100) #

# do linear interpolation
f_linear = interp1d(x_givenpoint, y_givenpoint)
y_il = f_linear(x_i)

# Cubic spline interpolation

f_spline = interp1d(x_givenpoint, y_givenpoint, kind='cubic')
y_is = f_linear(x_i)

# plot these results
plt.plot(x_givenpoint, y_givenpoint, 'x')
plt.plot(x_i, y_il, 'o')
plt.plot(x_i, y_is, '-')
plt.plot(xx, yy, ':')
plt.legend(['data', 'linear', 'perfect'], loc='best')
```

#### Curve fitting - how does it look like?

```python
x_given = np.array([0., 1., 2., 3., 4., 5.])
y_given = np.array([0, 0.8, 0.9, 0.1, -0.8, -1.0])

x_p = np.linspace(-2.0, 6.0, 100)    # Data for plotting the pply fit
p3 = np.polyfit(x_given, y_given, 3) # 3rd order polynomial object
y_p = np.polyval(p3, x_p)
plt.plot(x_given, y_given, 'o')
plt.plot(x_p, y_p, '-')
plt.legend(['data', 'polyfit'], loc='best')
plt.ylim(-2, 2)
```

```python
print(p3)
```

    [ 0.08703704 -0.81349206  1.69312169 -0.03968254]

#### Lets do some curve fitting
This function is available from scipy.optimizer import *
Fit function from scipy.org
$f(x) = a^-bx + c$

```python
from scipy.optimize import *
```

```python
def f(x, a, b, c):
    return a*np.exp(-b*x) + c

# Setting some given data

x_given = np.linspace(0,4,75)
y_given = f(x_given,2.5,1.5,0.8) + 0.2*np.random.normal(size=len(x_given))

# do some curve fitting

params, extras = curve_fit(f, x_given, y_given)

# output / plotting results

print("a=%g, b=%g, c=%g" %(params[0], params[1], params[2]))

plt.plot(x_given, y_given, 'o')
plt.plot(x_given, f(x_given, params[0], params[1], params[2]))
plt.legend(['data', 'curve-fit'], loc='best')
```

# Some more simple calculus stuff
Go to https://sympy.org for more basics

```python
from sympy import *
init_printing() # prints cool output
```

```python
x, y, z = symbols('x,y,z')
my_ex = x*cos(x)
my_ex
```

```python
# substituting this function
my_ex.subs(x, 1)
```

```python
diff(cos(x), x) #diff is the derivative function in sympy
```

```python
diff(exp(x**2), x)
```

```python
diff(x**4, x, x, x) #can take multiple derivatives at once
diff(x**4, x, 3)
```

#### Below will compute the following

$\frac{\partial 7}{\partial x \partial y^2 \partial z^4}e^{xyz}$

```python
expr = exp(x*y*z)
diff(expr, x,y,y,z,z,z,z)
```

```python
diff(expr, x,y,y,z,4)
```

```python
diff(expr, x,y,2,z,4)
```

```python
# It can be called as a method
expr.diff(x,y,y,z,4)
```

```python
# Create unevaluated rerivative by using Derivative class
deriv = Derivative(expr, x,y,y,z,4)
deriv
```

```python
type(Derivative)
```

    sympy.core.assumptions.ManagedProperties

```python
# Evaluate an unevaluated derivaive using doit method
deriv.doit()
```

## A bit more basics of Integrals

$f(x) = \int_{0}^{∞}e^{-x} dx$

##### alt+5 = infinity symbol

```python
integrate(exp(-x), (x,0, oo))
```

### Integrate this
$\int_{0}^{∞}\int_{-∞}^{∞}e^{-x^2-y^2} dxdy$

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

```python
# if integrate isnt able to compute an integral, it returns an unevaluated Integral object
expr = integrate(x**x, x)
print(expr)
```

    Integral(x**x, x)

```python
expr
```

```python
expr = Integral(log(x)**2, x)
expr
```

#### Itegrate uses powerful algorithms including heuristic pattern matching, Risch algorithm and Meijer G-functions
integrate uses powerful algorithms that are always improving to compute both definite and indefinite integrals. Here's a cool example.

```python
integ = Integral((x**4 + x**2*exp(x) - x**2 - 2*x*exp(x) - 2*x - \
                  exp(x))*exp(x)/((x - 1)**2*(x + 1)**2*(exp(x) + 1)), x)
integ
```

```python
integ.doit()
```

```python
integ = Integral(sin(x**2), x)
integ
```

```python
integ.doit()
```

```python
integ = Integral(x**y*exp(-x), (x,0,oo))
integ
```

```python
integ.doit()
```

$$\begin{cases} \Gamma{\left(y + 1 \right)} & \text{for}\: - \Re{\left(y\right)} < 1 \\\int_{0}^{\infty} x^{y} e^{- x}\, dx & \text{otherwise} \end{cases}$$

```python
help(Integral.doit)
```

    Help on function doit in module sympy.integrals.integrals:

    doit(self, **hints)
        Perform the integration using any hints given.

        Examples
        ========

        >>> from sympy import Integral
        >>> from sympy.abc import x, i
        >>> Integral(x**i, (i, 1, 3)).doit()
        Piecewise((2, Eq(log(x), 0)), (x**3/log(x) - x/log(x), True))

        See Also
        ========

        sympy.integrals.trigonometry.trigintegrate
        sympy.integrals.risch.heurisch
        sympy.integrals.rationaltools.ratint
        as_sum : Approximate the integral using a sum

## Coming up Limits...

$\lim_{a \rightarrow b}{f(x)}$

```python
limit(sin(x)/x, x, 0)
```

```python
expr = x**2/exp(x)
expr.subs(x,oo)
```

```python

```