# 521153S, Deep Learning assignment 2: Backpropagation, gradient check using linear approximation

## Outline
#### In this assignment, you will learn:
* **Backpropagation**.
* Building a one layer NN using numpy.
* Gradient check using finite-difference approximation.
* Stochatic Gradient Descent (SGD).
* Simple hype-parameters tuning methods to improve your NN performance.

#### Grading (15 points)
In this assignment, we are going to learn about **backpropagation**. This is a very importance algorithm you need to know when building and training your neural network.
* **Part 1.** Import libraries, loading and preprocessing the training and testing data.
* **Part 2.** Backpropagation. (10 points)
  * 2.1. Construct model in **Fig. 2** (2 points)

       * 2.1.1 Declare W1 and W2 (0.5 point)

       * 2.1.2 Implement sigmoid (0.5 point)

       * 2.1.3 Implement softmax (0.5 point)

       * 2.1.4 Implement the cross-entropy loss (0.5 point)

  * 2.2. Implement the forward-pass (1.5 point)

       * 2.2.1 Foward-pass in training (0.5 point)

       * 2.2.2 Call cross entropy loss in training (0.5 point)

       * 2.2.3 Forward-pass in testing (0.5 point)

  * 2.3. Implement SGD (1 point)

       * 2.3.1 Velocity (0.5 point)

       * 2.3.2 Update weights (0.5 point)

  * 2.4. Derive the derivatives (3.0 points)

       * Derive equation 6->12, which Eq. 6+7 (0.5 point), and from Eq. 8-12 (0.5 point) each

  * 2.5. Implement the derivatives (2.5 points)

       * 2.5.1 Implement equation 8 (0.5 point)

       * 2.5.2 Implement equation 6 (0.5 point)

       * 2.5.3 Implement equation 10 (0.5 point)

       * 2.5.4 Implement equation 10+11 (0.5 point)

       * 2.5.5 Implement equation 7 (0.5 point)

* **Part 3.** Gradient check using finite-difference approximation. (3 points)

  * 3.1. Implement gradient check for the model in **Fig. 2** (2.5 points)

       * 3.1.1 Implement reshape_params function (0.5 point)

       * 3.1.2 Adding the epsilon to params then reshape and split flatten_params to hidden weights (W1) and output weights (W2) (0.5 point)

       * 3.1.3 Implement the forward-pass (0.125 point)

       * 3.1.4 Call cross entropy loss in gradient check (0.125 point)

       * 3.1.5 Substracting the epsilon and do the similar calculations as above (0.5 point)

       * 3.1.6 Implement the forward-pass (0.125 point)

       * 3.1.7 Call cross entropy loss in gradient check (0.125 point)

       * 3.1.8 Compute the numerical gradient using the idea in the toy example (0.5 point)

  * 3.2. Question: why don't we use FDA to calculate the gradient to update our model? (0.5 point)

* **Part 4.** Regularization and NN simple tunning. (2 points)

  * 4.1. Applying weight decay. (1 point)

       * There are 5 spots you need to fill, (0.2 point) each spot.
  * 4.2. Change the number of neurons in the hidden layer and report the performance (1 point)

#### Environment
Python 3, Numpy, matplotlib, sklearn

#### Dataset
* [**Fashion-MNIST**](https://github.com/zalandoresearch/fashion-mnist)
is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. We intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. Using the Fashion-MNIST give you more room to wiggle your experiments.

#### Hints
* To find the place where you have to insert your solution, hit Crtl + F and search for **TODO:** . You are NOT suppose to modify the codes from other parts.
* **Be careful with the shape** of the weights, gradient, .. of your tensor in your implementation. Double check and make sure the shapes are fit for computation, especially matrix multiplication.

## Part 1. Import libraries, loading and preprocessing the training and testing data
**You don't need to change the code from this part.**

```python
# You will mainly use numpy to construct your NN
import numpy as np
import matplotlib, time, copy, os, requests, zipfile, sys
# Matplotlib to plot the image
import matplotlib.pyplot as plt
# Off-the-shelf evaluation functions provided by sklearn
from sklearn.metrics import confusion_matrix, accuracy_score
# Matplotlib predefined 'magic function'. It will include your graphs in your notebook, next to the code
%matplotlib inline
```

### Functions use to download the dataset from google drive
The code snipet was taken from [this thread](https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url).

```python
def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

```

### Functions use to pre-process your training/testing data

```python
def reshape_train_data(X):
    ''' Input training data has shape (60000, 28, 28)
        Input testing data has shape (10000, 28, 28)
        where:
        60000 is the numbers of input training samples
        10000 is the numbers of input testing samples
        similar to MNIST, resolution of each sample is 28 x 28
    '''
    samples, H, W = X.shape
    # Reshape input volume to (sample, 784), this mean, your NN input layer will have 784 placeholders
    # we scale the RGB values by divide them by 255, this will help improve the training performance
    return X.reshape(samples, H * W).T / 255

def one_hot_vector(x, num_classes):
    # By now, I think you already heard about this so many times
    return np.eye(num_classes)[x].T
```

### We took care of download the data for you.
The fashion-MNIST data will be download and store in your **work_dir/data/fashion_mnist_npy**

```python

PATH = './data'
if not os.path.exists(PATH):
    os.makedirs(PATH)

    file_id = '1DQ2Nf2rY467kyZKOf_CG3Kib5FLv0xQu'
    destination = os.path.join(PATH, 'fashion_mnist_npy.zip')
    download_file_from_google_drive(file_id, destination)

    with zipfile.ZipFile(destination, 'r') as zip_ref:
        zip_ref.extractall(PATH)

    print("Data downloaded and extracted!")

    os.remove(destination)

else:
    print("Data was already downloaded and extracted!")

PATH = os.path.join(PATH, 'fashion_mnist_npy')

# The actual meaning of the label of your classes.
# E.g. if a output one-hot vector is [0. 0. 0. 1. 0. 0. 0. 0. 0. 0.], it used to prepresent a Dress
label_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
```

    Data was already downloaded and extracted!

### Finally, we load and preprocess the data

```python
# Load the training input
X_train = np.load(os.path.join(PATH, 'train_data.npy'))
# Load the training labels
X_test = np.load(os.path.join(PATH, 'test_data.npy'))
# Load the testing input
Y_train = np.load(os.path.join(PATH, 'train_labels.npy'))
# Load the testing labels
Y_test = np.load(os.path.join(PATH, 'test_labels.npy'))

# Get the number of classes
num_classes = len(label_names)
# Get the number of training samples and their resolution for reshape
num_trains, HEIGHT, WIDTH = X_train.shape

# Reshape the training and testing inputs
X_train, X_test = reshape_train_data(X_train), reshape_train_data(X_test)

# Create one-hot vector for the training and testing labels
Y_train, Y_test = one_hot_vector(Y_train, num_classes), one_hot_vector(Y_test, num_classes)

# This part use to randomly load some of the training and testing image and the one-hot vectors for checking
fig_train, ax_train = plt.subplots(figsize=(16, 8), nrows=1, ncols=5)
fig_train.suptitle("Random image from the TRAINING set", y=0.73, fontsize=16, fontweight='bold')

fig_test, ax_test = plt.subplots(figsize=(16, 8), nrows=1, ncols=5)
fig_test.suptitle("Random image from the TESTING set", y=0.73, fontsize=16, fontweight='bold')

for idx in range(5):
    i, j = np.random.randint(num_trains), np.random.randint(X_test.shape[0])

    ax_train[idx].imshow(X_train[:,i].reshape(HEIGHT, WIDTH), cmap = matplotlib.cm.binary)
    ax_train[idx].set_title(label_names[np.argmax(Y_train[:,i])] + "\n" + str(Y_train[:,i]))
    ax_train[idx].axis('off')

    ax_test[idx].imshow(X_test[:,j].reshape(HEIGHT, WIDTH), cmap = matplotlib.cm.binary)
    ax_test[idx].set_title(label_names[np.argmax(Y_test[:,j])] + "\n" + str(Y_test[:,j]))
    ax_test[idx].axis('off')

```

## Part 2. Backpropagation (10 points)
#### The fun part start from here.
*Definition:* backpropagation is an algorithm used to **calculate the gradient of a loss function** (sometime called cost function), by applying the chain rule, so that we can update the weights (and bias) of our neural network using optimizer such as gradient descent.

It could be interpreted as the backward propagation (oppose to the forward propagation) of the network’s layers error, in which we calculate the error of the current layer, then pass the weighted error back to the previous layer, and recursively doing this until we travel back to the first hidden layer. At each layer, we update the weights using the derivative of the cost for each weight.

Let illustrate this concept using a simple toy example.

   Figure 1. Toy example of backpropagation

In **Fig. 1** we have a loss function $L$:
\begin{align}
L = c \times d
\end{align}

where

\begin{align}
c = a + b -5
\end{align}

and,

\begin{align}
d = b^2 + b -1
\end{align}

As illustrated in **Fig. 1**, the equations in the edge show the partial derivation of some functions with respect to their direct variables. For example, $\frac{\partial L}{\partial c}$ is the partial derivative of $L$ with respect to $c$.

However, we are actually interested in calculate the derivative of $L$ with respect to $a$ and $b$, which don't directly connected with each other. So, how can we do this? As mentioned earlier, we can do this using the chain-rule so that we can calculate $\frac{\partial L}{\partial a}$ and $\frac{\partial L}{\partial b}$ as:

\begin{align}
\frac{\partial L}{\partial a} = \frac{\partial L}{\partial c} \cdot \frac{\partial c}{\partial a}
\end{align}

and,

\begin{align}
\frac{\partial L}{\partial b} = \frac{\partial L}{\partial c} \cdot \frac{\partial c}{\partial b} + \frac{\partial L}{\partial d} \cdot \frac{\partial d}{\partial b}
\end{align}

More importantly, **we can see that $a$ affect $L$ through $c$, and so on**. This concept hold no matter how many hidden layers you have or how complicated your loss will be as long as you use backpropagation to calculate the derivative to update your weights.

Now, let's get your hand dirty. Let's take a look at the model in **Fig. 2** below. This is the model we are gonna use in this assignment.

   Figure 2. Structure of our neural network

### Construct model base on **Fig. 2** above (2 points)
As shown in the **Fig. 2**, our neural network contains one hidden layer (we use 5 neurons as default setting) and an output layer. To further simply things, we will not include biases in our model. Be aware that, while the number of neuron in the hidden layer can be abitrary, there must be 10 neurons in the output layer because we want our model to classify images from 10 classes. Based on the **Fig. 2**, you have to construct your network. Don't worry, it's easy and you already learnt how to do this in assignment 1

```python
import torch
import torch.nn as nn
# Numbers of input units
num_input = X_train.shape[0]
# Number of neural in your hidden layer
# TODO: modify the number of neurons in the hidden layer
num_hidden = 5
num_classes = 10

# Construct your neural network from Fig. 2 (0.5 point)
# TODO: Random initialize the hidden layer weights (w/o bias)
W1 = np.random.randn(num_hidden, num_input)
# TODO: Random initialize the output layer weights (w/o bias)
W2 = np.random.randn(num_classes, num_hidden)

### Define CrossEntropyLoss in nn
criterion = nn.CrossEntropyLoss()

def sigmoid(X):
    # TODO: implement Eq. 2 (0.5 point)
    return 1 / (1 + np.exp(-X))

def softmax(X):
    # TODO: implement Eq. 4 (0.5 point)
    e_x = np.exp(X)
    return e_x / np.sum(e_x, 0)

def cross_entropy_loss(Y, Y_pred):
    # TODO: implement Eq. 5 (0.5 point)
    m = Y.shape[1]
    log_likelihood = -np.log(Y_pred)*Y
    loss = np.sum(log_likelihood)/m
    return loss
```

According to the model in **Fig. 2**, our **forward-propagation** is going to start with the calculation of the output volume $A_1$ for the hidden layer:

\begin{align}
A_1 = \sigma(Z_1) = \sigma(W_1 X ) \tag{1}
\end{align}
where $W_1$ is the weights of the hidden layer, $X$ is the input, and $\sigma$ is the sigmoid activation where:

\begin{align}
\sigma(x) = \frac{1}{1 + e^{-x}} \tag{2}
\end{align}

Then, we calculate the output volume of the output layer $A_2$ by:

\begin{align}
A_2 = S(Z_2) = S(W_2 A_1) \tag{3}
\end{align}

In **Eq. 3**, $W_2$ is the weights of the output layer, $A_1$ is the output volume of the hidden layer, $S$ stands for softmax and defines by:

\begin{align}
S(x_i) = \frac{e^{x_i}}{\sum_{j=0}^{k} e^{x_j} } \tag{4}
\end{align}
where $i=0,1,..,k$. We use $k$ to represent classes, and $k=9$ in our case.

Finally, we compute the loss function using:

\begin{align}
L(Y, A_2) = - \frac{1}{M} \sum_{k=0}^{M} \sum_{i=0}^{N} Y_i^{k} log({A_2}_i^{k}) \tag{5}
\end{align}

where $Y$ is the ground truth labels, $N$ is the number of classes, $M$ is the number of samples in the training batch.

### Derive the derivative and fill the equation 6->12 below (3.0 points)

Using what you learnt in undergraduate school, let's calculate the **backward-propagation** for our model in **Fig. 2**.

As mentioned earlier, we are interested in $\frac{\partial L}{\partial W_1}$ and $\frac{\partial L}{\partial W_2}$, where: (0.5 point)

\begin{align}
\frac{\partial L}{\partial W_2} = \frac {\partial L}{\partial Z_2}\cdot\frac {\partial Z_2}{\partial W_2} = (A_2 - Y)\cdot A_1  \tag{6}
\end{align}

and,

\begin{align}
\frac{\partial L}{\partial W_1} = \frac {\partial L}{\partial A_1}\cdot \frac {\partial A_1}{\partial Z_1}\cdot \frac {\partial Z_1}{\partial W_1}= (A_2 - Y)\cdot {W_2}\cdot A_1\cdot(1-A_1)\cdot X\tag{7}
\end{align}

From **Eq. 6** we have:

The derivative of $L$ with respect to $Z_2$: (0.5 point)
\begin{align}
\frac{\partial L}{\partial Z_2} = \frac{\partial L}{\partial A_2} \cdot \frac{\partial A_2}{\partial Z_2} =  A_2 - Y\tag{8}
\end{align}

The derivative of $Z_2$ with respect to $W_2$: (0.5 point)
\begin{align}
\frac{\partial Z_2}{\partial W_2} = A_1 \tag{9}
\end{align}

From **Eq. 7** we have:

The derivative of $L$ with respect to $A_1$: (0.5 point)
\begin{align}
\frac{\partial L}{\partial A_1} = \frac{\partial L}{\partial Z_2} \cdot \frac{\partial Z_2}{\partial A_1} =  (A_2 - Y)\cdot {W_2}\tag{10}
\end{align}

The derivative of $A_1$ with respect to $Z_1$: (0.5 point)
\begin{align}
\frac{\partial A_1}{\partial Z_1} = A_1\cdot(1-A_1) \tag{11}
\end{align}

The derivative of $Z_1$ with respect to $W_1$: (0.5 point)
\begin{align}
\frac{\partial Z_1}{\partial W_1} =  X\tag{12}
\end{align}

## Part. 3 Gradient check using finite-difference approximation. (3 points)

When training deep neural network, there are many things that can go wrong. Until this point, you probably notice that we have been going on and on about the gradient. Why? Because the gradient is very important. Hence, you must making sure that the calculation of your gradient is correct. A correct gradient calculation won't promise your model will converge, but if the calculation was wrong your model will perform very weird. This type of error is hard to debug, so we better prevent it beforehand.

To perform the gradient check, we can calculate gradient using the [finite-difference approximation](https://en.wikipedia.org/wiki/Finite_difference) (FDA), and let's call the output of FDA numerical gradients. Then we compare this numerical gradients with the gradient we calculate from taking the derivative. If the differences between them are small enough, we can assume that the gradient was calculated correctly.

You probably learnt about FDA in your undergraduate, but to refresh your mind, let's have a simple example to see how FDA works. Assume that we have a function $f(x)$ which

\begin{align}
f(x) = \frac{1}{3} x^3 - \frac{1}{2} x^2 + 1 \tag{13}
\end{align}

Then, the derivative $\Delta f$ will be:

\begin{align}
\Delta f = x^2 - x \tag{14}
\end{align}

At $x=2.125$, using **Eq. 14** we have $\Delta f = 2.390625$

If we calculate the numerical gradient using FDA we have:
\begin{align}
\Delta_{num\_grad} f = \frac{f(x + \epsilon) - f(x - \epsilon)}{2 * \epsilon}  \tag{15}
\end{align}

where $\epsilon$ is a very small value (E.g. $\epsilon = 1e-{04}$)

At the same point $x=2.125$, using **Eq. 15** we have $\Delta_{num\_grad} f = 2.3906250033389753$

We can see that, the calculated values of $\Delta f$ and $\Delta_{num\_grad} f$ are very close to each other.

With the same idea, we can check the gradient calculation of our network using FDA. A simple way to do this is:

1) We wiggle (by a very small $\epsilon$ values) the value of our weight for all of the parameters in our model. By all parameters, I mean all of the weights of $W_1$ and $W_2$. E.g. if we use 5 neurons in the hidden layer, the number of parameters in our network is :

$num\_params(net) = num\_params(W_1) + num\_params(W_2) = 28*28*5 + 5*10 = 3970$

so we have to repeat the "wiggling" and calculate the numerical gradient 3970 times. At the end, we have a $num\_grad$ vector that have shape (3970,)

2) Calculate the gradient by taking the derivative. Similarly, we will have a $grad$ vector that also have shape (3970,)

3) Compare $num\_grad$ and $grad$ vectors by:

\begin{align}
grad\_diff = \frac{|grad - num\_grad|_2}{|grad + num\_grad|_2}  \tag{16}
\end{align}

If **grad_diff** is smaller than $1e-{08}$ than we assume that our gradient calculation is correct.

### Question: Why don't we use FDA to calculate the gradient to update our model? (0.5 points)

**Answer:**

On the one hand, it is an estimate value for gradient, and there will be a loss of accuracy. On the other hand, extra time spent when we calculating gradients by this method.

### Implement gradient check (2.5 points)

```python
def reshape_params(params, num_input, num_hid, num_classes):
    '''Reshape and split flatten_params to hidden weights (W1) and output weights (W2)
        Input:
            + params: the flatten parameters of W1 and W2
            + num_input: the number of input units
            + num_hid: the number of hidden neurons
            + num_classes: the number of output class
        Output:
            + W1: weights of the hidden layer in shape (num_hid, num_input)
            + W2: weights of the output layer in shape (num_class, num_hid)
    '''

    # TODO: implement this function (0.5 point)
    W1 = params[0:num_hid * num_input].reshape((num_hid, num_input))
    W2 = params[(num_hid * num_input):].reshape((num_classes, num_hid))
    return W1, W2

def calculate_numerical_gradient(_W1, _W2, X, y, is_weight_decay):
    params_flatten = np.concatenate( (copy.deepcopy(_W1).ravel(), copy.deepcopy(_W2).ravel()) )
    numgrad = np.zeros( len(params_flatten) )
    wiggle = np.zeros( len(params_flatten) )
    num_hidden, num_input = _W1.shape
    num_classes = _W2.shape[0]
    e = 1e-4
    batchsize = X.shape[1]

    # Loop through each parameter in our model
    for p in range(len(params_flatten)):
        wiggle[p] = e
        # TODO: adding the epsilon to params then reshape and split flatten_params to hidden weights (W1) and output weights (W2)
        # (0.5 point)
        params_flatten += wiggle
        W1, W2 = reshape_params(params_flatten, num_input, num_hidden, num_classes)

        # TODO: implement the forward-pass (0.125 point)
        y_pre1 = softmax(np.dot(W2,sigmoid(np.dot(W1,X))))

        if is_weight_decay:
            # TODO: call cross entropy loss with weight decay regularization
            # (0.2 point)

            loss1 = cross_entropy_loss(y, y_pre1) + lmda/2*sum(params_flatten*params_flatten)

        else:
            # TODO: call cross entropy loss (0.125 point)
            loss1 = cross_entropy_loss(y, y_pre1)

        # reset the params values
        params_flatten -= wiggle

        # TODO: substract the epsilon and do the similar calculations as above
        # (0.5 point)
        params_flatten -= wiggle
        W1, W2 = reshape_params(params_flatten, num_input, num_hidden, num_classes)

        # TODO: implement the forward-pass (0.125 point)
        y_pre2 = softmax(np.dot(W2,sigmoid(np.dot(W1,X))))

        if is_weight_decay:
            # TODO: call cross entropy loss with weight decay regularization
            # (0.2 point)
            loss2 = cross_entropy_loss(y, y_pre2) + lmda/2*sum(params_flatten*params_flatten)
        else:
            # TODO: call cross entropy loss (0.125 point)
            loss2 = cross_entropy_loss(y, y_pre2)

        # TODO: Compute the numerical gradient using the idea in the toy example (0.5 point)
        numgrad[p] = (loss1 - loss2)/(2*e)
        # reset the params values
        params_flatten += wiggle

        # Return the value we changed to zero:
        wiggle[p] = 0

    return numgrad
```

### Hyper-parameters in your training

```python
# Flag use to enable/disable gradient check and weight decay regularization
is_gradient_check = True
is_weight_decay = True

if is_weight_decay:
    # Setting lambda coefficient for weight decay
    lmda = np.exp(-7)

# Seting learning rate and momentum for SGD
learning_rate = 0.25
beta = 0.5

# Modify Learning rate and beta
# learning_rate = 0.0025
# beta = 0.005

# Seting the number of training epochs
epoch = 5
# Choose your batch size
batch_size = 100
# Calculate the number of training iterations base on the number of training samples and your batch size
num_batchs = num_trains // batch_size

print("Num_trains: {}, num_batchs: {}".format(num_trains, num_batchs))

```

    Num_trains: 60000, num_batchs: 600

### Training your network (4.5 points)

In this assignment, we will train our model using mini-batch stochatic gradient descent with momentum. To know more about this optimization algorithm, please check out this great [video](https://www.youtube.com/watch?v=k8fTYJPd3_I) from Dr. Andrew Ng.

For this assignment, we will use the implementation from **Eq. 17** and **Eq. 18**

\begin{align}
v_{dW} = \beta v_{dW} + (1 - \beta) dW \tag{17}
\end{align}
\begin{align}
W = W - \alpha v_{dW} \tag{18}
\end{align}

If you are curious, you can modify the hyper-params in the above section at your will.

```python
# Define an interactive ipython figure to display the training loss
%matplotlib notebook
fig = plt.figure("Training loss")
ax = fig.add_subplot(1, 1, 1)
plt.ion()
fig.show()
fig.canvas.draw()

# Log the training loss
loss_log = []

# Zeros initialize the momentum for SGD
V_dW1 = np.zeros(W1.shape)
V_dW2 = np.zeros(W2.shape)

# Training
for i in range(epoch):
    start_t = time.time()

    # Random shuffle training data every training epoch
    np.random.seed(np.random.randint(num_trains))
    indices = np.random.permutation(num_trains)
    X_train_shuffled, Y_train_shuffled = X_train[:, indices], Y_train[:, indices]

    for j in range(num_batchs):

        # Get mini-batch samples for training
        start_idx = j * batch_size
        end_idx = min(j * batch_size + batch_size, X_train.shape[1] - 1)
        X, Y = X_train_shuffled[:, start_idx : end_idx], Y_train_shuffled[:, start_idx : end_idx]
        # Size of actual mini-batch, it could be smaller than batch_size
        mini_batch = end_idx - start_idx

        # TODO: implement the forward-pass (0.5 point)

        ### Define A1 = 5×100 = 5×784 * 784×100
        A1 = sigmoid(np.dot(W1,X))

        ### A2 = 10×100 =10×5 * 5×100
        A2 = softmax(np.dot(W2,A1))

        params_flatten = np.concatenate((W1.ravel(), W2.ravel()))

        if is_weight_decay:
            # TODO: call cross entropy loss with weight decay regularization
            # (0.2 point)
            L = cross_entropy_loss(Y, A2) + lmda/2*sum(params_flatten*params_flatten)

        else:
            # TODO: call cross entropy loss (0.5 point)
            L = cross_entropy_loss(Y, A2)

        # Log the training loss during training
        loss_log.append(L)

        # Gradient check
        if is_gradient_check and j % 200 == 0:
            numgrad = calculate_numerical_gradient(W1, W2, X, Y, is_weight_decay)

        # TODO: calculate the derivative of 𝐿 with respect to 𝑍2 using eq. 8 (0.5 point)
        ### we got 10×100
        LresZ2 = A2 - Y

        if is_weight_decay:
            # TODO: calculate the derivative of 𝐿 with respect to W2 using eq. 6 with weight decay regularization
            # (0.2 point)
            ### we got 10×5 = 10×100 * 100×5 lmda*
            dW2 = np.dot(LresZ2,A1.T)/mini_batch + lmda*W2
        else:
            # TODO: calculate the derivative of 𝐿 with respect to W2 using eq. 6 (0.5 point)
            dW2 = np.dot(LresZ2,A1.T)/mini_batch

        # TODO: calculate the derivative of 𝐿 with respect to A1 using eq. 10 (0.5 point)
        ### we got 100×5 = 10×100 * 10×5
        LresA1 = np.dot (LresZ2.T, W2)

        # TODO: calculate the derivative of 𝐿 with respect to Z1 using eq. 10 and eq. 11 (0.5 point)
        LresA1Z1 = A1 * (1 - A1)
        LresZ1 = LresA1 * (LresA1Z1.T)

        if is_weight_decay:
            # TODO: calculate the derivative of 𝐿 with respect to W1 using eq. 7 with weight decay regularization
            # (0.2 point)
            ### we got 5×784 = 100×5 * 784×100/(784*mini_batch) lmda*
            dW1 = np.dot(LresZ1.T, X.T)/mini_batch + lmda*W1
        else:
            # TODO: calculate the derivative of 𝐿 with respect to W1 using eq. 7 (0.5 point)
            dW1 = np.dot(LresZ1.T, X.T)/mini_batch

        # Gradient check
        if is_gradient_check and j % 200 == 0:
            # Flatten and combine grad vector
            grad = np.concatenate((dW1.ravel(), dW2.ravel()))
            # Compare num_grad and grad
            grad_diff = np.linalg.norm(grad - numgrad) / np.linalg.norm(grad + numgrad)
            # print("Grad_diff: {}".format(grad_diff))
            assert grad_diff < 1e-8, "Gradient check fail, please carefully inspect your gradient calculations again!"

        # TODO: Update the learning velocity using Eq. 17 (0.5 point)

        V_dW1 = beta * V_dW1 + (1-beta) * dW1
        V_dW2 = beta * V_dW2 + (1-beta) * dW2

        # TODO: Update the model weights using Eq. 18 (0.5 point)
        W1 -= learning_rate * V_dW1
        W2 -= learning_rate * V_dW2

        if (j % 100 == 0):
            print("[Epoch/Iterations]:[{}/{}], loss: {}".format(i, j, L))

    ax.clear()
    ax.plot(loss_log)
    fig.canvas.draw()
    print("=> Elapsed time epoch #{} : {:.2f} seconds".format(i, time.time() - start_t))

```

    <IPython.core.display.Javascript object>

    [Epoch/Iterations]:[0/0], loss: 2.952348938107534
    [Epoch/Iterations]:[0/100], loss: 2.8041613349052685
    [Epoch/Iterations]:[0/200], loss: 2.7445247213054627
    [Epoch/Iterations]:[0/300], loss: 2.571711247203578
    [Epoch/Iterations]:[0/400], loss: 2.4787185435737342
    [Epoch/Iterations]:[0/500], loss: 2.5751586726850224
    => Elapsed time epoch #0 : 40.36 seconds
    [Epoch/Iterations]:[1/0], loss: 2.45321996226024
    [Epoch/Iterations]:[1/100], loss: 2.2764702251559084
    [Epoch/Iterations]:[1/200], loss: 2.2730896501693305
    [Epoch/Iterations]:[1/300], loss: 2.216651649694056
    [Epoch/Iterations]:[1/400], loss: 2.157715825627559
    [Epoch/Iterations]:[1/500], loss: 2.133759111106264
    => Elapsed time epoch #1 : 38.03 seconds
    [Epoch/Iterations]:[2/0], loss: 2.1318351118327756
    [Epoch/Iterations]:[2/100], loss: 1.9069691983695325
    [Epoch/Iterations]:[2/200], loss: 1.8840800433390132
    [Epoch/Iterations]:[2/300], loss: 1.8271604795714573
    [Epoch/Iterations]:[2/400], loss: 1.870875102426006
    [Epoch/Iterations]:[2/500], loss: 1.7527420393129143
    => Elapsed time epoch #2 : 37.63 seconds
    [Epoch/Iterations]:[3/0], loss: 1.6495257645109629
    [Epoch/Iterations]:[3/100], loss: 1.5820127654559029
    [Epoch/Iterations]:[3/200], loss: 1.590374297328685
    [Epoch/Iterations]:[3/300], loss: 1.5201645733525715
    [Epoch/Iterations]:[3/400], loss: 1.4051365284528803
    [Epoch/Iterations]:[3/500], loss: 1.4676890456962723
    => Elapsed time epoch #3 : 38.35 seconds
    [Epoch/Iterations]:[4/0], loss: 1.3741354370110754
    [Epoch/Iterations]:[4/100], loss: 1.2895320337011282
    [Epoch/Iterations]:[4/200], loss: 1.2680432215467659
    [Epoch/Iterations]:[4/300], loss: 1.2019760756462026
    [Epoch/Iterations]:[4/400], loss: 1.3893842521511515
    [Epoch/Iterations]:[4/500], loss: 1.1668449189399106
    => Elapsed time epoch #4 : 40.05 seconds

```python
print(X.shape)
print(W1.shape)
print(V_dW2.shape)
print(Y.shape)
```

    (784, 99)
    (5, 784)
    (10, 5)
    (10, 99)

### Evaluate the performance of your model (0.5 points)

```python
# TODO: implement the forward-pass (0.5 point)
A2 = softmax(np.dot(W2,sigmoid(np.dot(W1,X_test))))

# Evaluate the performance of your NN
predictions = np.argmax(A2, axis=0)
labels = np.argmax(Y_test, axis=0)
print("Confusion matrix:\n{}".format(confusion_matrix(labels, predictions)))
print("Testing accuracy: {}".format(accuracy_score(labels, predictions)))
```

    Confusion matrix:
    [[880   7  16  59   5   1  12   0  20   0]
     [ 14 939  15  25   3   0   0   1   2   1]
     [ 44   3 861  10  32   0  34   1  15   0]
     [150  22  15 772  15   0  20   1   4   1]
     [ 60   2 834  19  33   0  33   1  18   0]
     [  1   1   0   2   0 801   0 104  11  80]
     [335   4 540  21  37   0  29   0  34   0]
     [  0   0   0   0   0  40   0 880   0  80]
     [  7   2  63  11   1   8   4   5 899   0]
     [  0   0   0   3   0  18   0  45   1 933]]
    Testing accuracy: 0.7027

## Part 4. Regularization and NN simple tunning (2 points)

1. Applying weight decay (1 point)
  * Using what you learnt from assignment 1 to add the code at neccesary parts in **Training your network** and **Implement gradient check** above.
  * There are 5 spots, (0.2 point) each spot.
2. Change the number of neurons in the hidden layer and report the performance (1 point)

**Answer of question 2:**

When I changed number of neurons in the hidden layer to 10, time consuming increased a lot. And I even got a worse loss for each epoch training process.I think the number of neurons in the hidden layer is related to the data distribution, it does not mean that the larger the number of neurons, the better the result. And the time spent increases a lot.


# Symbolic Computation: The Pitfalls

This collection of notebooks is mostly numerical, with not a lot of exact or symbolic computation, with the exception of polynomial computation done by computing with vectors of monomial basis coefficients, so it's really numerical anyway.  Why not do symbolic computation?  And, for that matter, why is numerical computing (even with all the unexpected behaviour of floating-point arithmetic) so much more popular than symbolic or exact computing?

This section explores symbolic computation and its pitfalls.  We do so from the point of view of experience and with some authority: we have used symbolic computation (usually in Maple, but also in other symbolic languages) for years (decades!) and know it and its benefits well. Caveat: We do not know SymPy so well, and so if we say that SymPy can't do something, we may well be wrong. One of us knows Sage quite well, but we're not using Sage here (yet).

One of RMC's earliest co-authors, Honglin Ye, put it well when he suggested that not everyone needs numerical methods but that everyone could use symbolic computation.

Wait.  Isn't that contradictory?  If everyone could use it, why aren't they?

There are, we think, a few main obstacles.

1. Symbolic computation systems are hard to learn how to use well, because there's a lot to learn (indeed, you kind of have to know the math first, too).  Look at the SymPy Tutorial for example. It has ten sections, one labeled "Gotchas". The SAGEMATH system, which also works with Python, is both more powerful and more complicated: See the SAGEMATH Tutorial  to get started there.
2. Some mathematical problems are inherently too expensive to solve in human lifetimes, even with today's computers, and people unfairly blame symbolic computation systems for this.
3. Even if you can solve a problem exactly, with extra effort, that effort might be wasted because the approximate answers are also the "exact" answers to similar problems, and those similar problems might be just as good a model of whatever system you were trying to understand.  This is especially true if the data is only known approximately.
4. "Symbolic Computation" and "Computer Algebra" are related terms—about as close as "Numerical Analysis" and "Computational Science" if that comparison means anything—but the differences are remarkably important, because what gets implemented is usually a Computer Algebra system, whereas what people actually want to use is a symbolic computation system.  We'll show you what that means.
5. Symbolic computation systems are hard to implement well.  The major systems (Maple, Mathematica, and Matlab) charge money for their products, and get what they ask for; this is because their systems are better than the free ones in many respects, because they have invested significant programmer time to address the inherent difficulties.  Free systems, such as SymPy, will do the easy things for you; and we will see that they can be useful.  But in reality there's no comparison (although we admit that the SAGEMATH people may well disagree with our opinion).

All that said, symbolic computation can be extremely useful (and interesting), and is sometimes worth all the bother. Let's look first at what Python and SymPy can do.  Later we'll look at what the difficulties are.

```python
n = 100
p = 1
for i in range(n):
    p = p*(i+1)
print( n, ' factorial is ', p)
print( 'The floating point value of p is ', 1.0*p )
```

    100  factorial is  93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000
    The floating point value of p is  9.332621544394415e+157

The first thing we see is that Python has, built-in, arbitrary precision integer arithmetic.  Yay?

```python
n = 720
p = 1
for i in range(n):
    p = p*(i+1)
print( n, ' factorial is ', p)
# print( 'The floating point value of p is ', 1.0*p )  # Causes OverflowError
```

    720  factorial is  2601218943565795100204903227081043611191521875016945785727541837850835631156947382240678577958130457082619920575892247259536641565162052015873791984587740832529105244690388811884123764341191951045505346658616243271940197113909845536727278537099345629855586719369774070003700430783758997420676784016967207846280629229032107161669867260548988445514257193985499448939594496064045132362140265986193073249369770477606067680670176491669403034819961881455625195592566918830825514942947596537274845624628824234526597789737740896466553992435928786212515967483220976029505696699927284670563747137533019248313587076125412683415860129447566011455420749589952563543068288634631084965650682771552996256790845235702552186222358130016700834523443236821935793184701956510729781804354173890560727428048583995919729021726612291298420516067579036232337699453964191475175567557695392233803056825308599977441675784352815913461340394604901269542028838347101363733824484506660093348484440711931292537694657354337375724772230181534032647177531984537341478674327048457983786618703257405938924215709695994630557521063203263493209220738320923356309923267504401701760572026010829288042335606643089888710297380797578013056049576342838683057190662205291174822510536697756603029574043387983471518552602805333866357139101046336419769097397432285994219837046979109956303389604675889865795711176566670039156748153115943980043625399399731203066490601325311304719028898491856203766669164468791125249193754425845895000311561682974304641142538074897281723375955380661719801404677935614793635266265683339509760000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Large integers cost more to manipulate---the above number is pretty long.  But SymPy will do it if you ask.  One thing you might want to do is factor those numbers.  Or one might just want to know the prime factors.

```python
from sympy import primefactors

primefactors_n = primefactors(n)
print("The prime factors of {} : {}".format(n, primefactors_n))

primefactors_p = primefactors(p)
print("The prime factors of {} : {}".format(p, primefactors_p))

```

    The prime factors of 720 : [2, 3, 5]
    The prime factors of 2601218943565795100204903227081043611191521875016945785727541837850835631156947382240678577958130457082619920575892247259536641565162052015873791984587740832529105244690388811884123764341191951045505346658616243271940197113909845536727278537099345629855586719369774070003700430783758997420676784016967207846280629229032107161669867260548988445514257193985499448939594496064045132362140265986193073249369770477606067680670176491669403034819961881455625195592566918830825514942947596537274845624628824234526597789737740896466553992435928786212515967483220976029505696699927284670563747137533019248313587076125412683415860129447566011455420749589952563543068288634631084965650682771552996256790845235702552186222358130016700834523443236821935793184701956510729781804354173890560727428048583995919729021726612291298420516067579036232337699453964191475175567557695392233803056825308599977441675784352815913461340394604901269542028838347101363733824484506660093348484440711931292537694657354337375724772230181534032647177531984537341478674327048457983786618703257405938924215709695994630557521063203263493209220738320923356309923267504401701760572026010829288042335606643089888710297380797578013056049576342838683057190662205291174822510536697756603029574043387983471518552602805333866357139101046336419769097397432285994219837046979109956303389604675889865795711176566670039156748153115943980043625399399731203066490601325311304719028898491856203766669164468791125249193754425845895000311561682974304641142538074897281723375955380661719801404677935614793635266265683339509760000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 : [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677, 683, 691, 701, 709, 719]

Factoring seems like such a simple problem, and it's so natural to have it implemented in a symbolic computation system.  The number 720! is 1747 digits long.  Maybe all 1700--odd digits long integers are so easy to factor?

Um, no.  See the discussion [here](https://en.wikipedia.org/wiki/Integer_factorization) to get started.  Let's take a modest problem and time it here.

```python
funny = 3000000000238000000004719
#notfunny = 45000000000000000057990000000000000024761900000000000003506217
from sympy import factorint
import time
start_time = time.time()
factordict = factorint(funny)
print("The prime factors of {} : {}".format(funny, factordict))
print("--- %s seconds ---" % (time.time() - start_time))

```

    The prime factors of 3000000000238000000004719 : {1000000000039: 1, 3000000000121: 1}
    --- 0.9057760238647461 seconds ---

That factoring of $3000000000238000000004719$ took between 8 and 11 seconds on this machine (different times if executed more than once); on this very same machine, Maple's "ifactor" command succeeded so quickly that it registered no time taken at all, possibly because it was using a very specialized method; factoring integers is an important feature of symbolic computation and Maple's procedures for it have been a subject of serious research for a long time.  Maple's help pages cite three important papers, and tell you that it uses an algorithm called the quadratic sieve.  Maple can factor $45000000000000000057990000000000000024761900000000000003506217$ into its three prime factors in about 7.5 seconds on this machine; in contrast, after fifty minutes running trying to factor that with factorint as above, RMC had to hard-restart to get Python's attention.

That SymPy takes so long to factor integers, in comparsion, suggests that it isn't using the best methods (the documentation says that it switches between three methods, trial division, Pollard rho, and Pollard p-1) ; and because factoring is such a basic algorithm (an even more basic one is GCD or Greatest Common Divisor) this will have important knock-on effects.

But factoring, as old an idea as it is, is complicated enough to be used as a basic idea in modern cryptography.  The slowness of SymPy is not completely its fault: the problem is hard.

Let's move on to computing with functions.  As previously stated, most supposedly "symbolic" systems are really "algebra" systems: this means that they work well with polynomials (even multivariate polynomials).  A polynomial considered as an algebraic object is isomorphic to a polynomial considered as a function---but the difference in viewpoint can alter the affordances.  An "affordance" is a word meaning "something can happen with it": for instance, you can pick out a lowest-degree term; or you can add it to another polynomial; or you can square it; and so on.  As a function, you can evaluate it at a particular value for the symbols (variables).

```python
from sympy import *
x = symbols('x')

solveset(Eq(x**2, 3), x)

```

$\displaystyle \left\{- \sqrt{3}, \sqrt{3}\right\}$

```python
solveset(Eq(x**3+x-1, 0), x)
```

$\displaystyle \left\{- \frac{1}{3 \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}} + \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}, - \frac{\sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}}{2} + \frac{1}{6 \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}} + i \left(\frac{\sqrt{3}}{6 \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}} + \frac{\sqrt{3} \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}}{2}\right), - \frac{\sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}}{2} + \frac{1}{6 \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}} + i \left(- \frac{\sqrt{3} \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}}{2} - \frac{\sqrt{3}}{6 \sqrt[3]{\frac{1}{2} + \frac{\sqrt{93}}{18}}}\right)\right\}$

```python
start_time = time.time()
#solveset(Eq(x**4+x-1, 0), x) # Interrupted after about two hours: the code did not succeed
print("--- %s seconds ---" % (time.time() - start_time))
```

    --- 0.0 seconds ---

In those two hours, RMC went and had his dinner; then downloaded a paper by Dave Auckly from the American Mathematical Monthly 2007  which talks about solving the quartic with a pencil (an algebraic geometer's pencil!), read it, and solved the problem by hand, including solving the resolvent cubic by hand, which he already knew how to do.  And got it right, too. So there.

In contrast, Maple (nearly instantaneously) returns---if you force it to by saying you want the explicit solution---the answer

$$
\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}+\frac{\mathrm{I} \sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}-48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
,
\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}-\frac{\mathrm{I} \sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}-48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
,
-\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}+\frac{\sqrt{6}\, \sqrt{\frac{-\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}+48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
,
-\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}-\frac{\sqrt{6}\, \sqrt{\frac{-\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}+48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
$$

This is an example of what Velvel Kahan calls a "wallpaper expression."  He also famously said, "Have you ever asked a computer algebra system a question, and then, as the screensful of answer whizzed past your eyes, said "I wish I hadn't asked?""

The use of that exact answer (quickly obtained or not) is questionable.  Then, of course, the Abel-Ruffini theorem says that there is no general formula for solving polynomials of degree $5$ or higher in terms of radicals.  For degree $5$ polynomials, there is a solution in terms of elliptic functions; again, it's complicated enough that it's of questionable use.  Then there is Galois theory which describes the algebraic structures of polynomials.  See the interesting historical essay by Nick Trefethen on  What we learned from Galois .

The lesson here is that even when you can solve something exactly, maybe you shouldn't.

There are some interesting things you can do with univariate polynomials of high degree, including with the algebraic numbers that are their roots.  But computation with them isn't so easy.  SymPy actually has some quite advanced features for polynomials, including multivariate polynomials.

## Symbolic computation with functions

Let's try some calculus-like things.

```python
y = symbols('y')

solveset(Eq(exp(y), x), y)
```

$\displaystyle \left\{i \left(2 n \pi + \arg{\left(x \right)}\right) + \log{\left(\left|{x}\right| \right)}\; \middle|\; n \in \mathbb{Z}\right\}$

RMC really doesn't like that "solution"!  It has separated the real and imaginary parts without needing to.  A perfectly good answer would be $\ln_n(x)$, which looks a lot simpler.

$\ln_k(z)$, which might not look familiar to you, means $\ln(z) + 2\pi i k$.  Also, SymPy has chosen to emulate Maple and use I for the square root of $-1$, which made a kind of sense in the 1980's when Maple chose to do it that way, but given all the fonts we have nowadays that doesn't seem sensible at all.

Fine.  We will live with it.  The solution is not actually wrong.

```python
solveset(Eq( y*exp(y), x), y)
```

$\displaystyle \left\{y\; \middle|\; y \in \mathbb{C} \wedge - x + y e^{y} = 0 \right\}$

Oh, that's disappointing.  See  the Wikipedia article on Lambert W to see what that should have been.

```python
solve( Eq( y*exp(y), x ), y )
```

    [LambertW(x)]

That's better, but—like the logarithm above—there should be multiple branches.

### Integrals, and the difference between Computer Algebra and Symbolic Computation

We'll start with a nasty example.  You can find nice examples of SymPy and integration in many places, so we will assume that you have seen instances of computer implementations of the fundamental theorem of calculus: to find areas under curves by using antiderivatives.  The nasty integral that we will use is, if $x \ne 0$,

$$
f(x) = \frac{e^{-1/x}}{x^2(1+e^{-1/x})^2}
$$

with the removal of the discontinuity at $x=0$ by defining $f(0)=0$.

We will try to use both NumPy and SymPy to integrate this (infinitely smooth! Just check the derivatives at zero---the function is _infinitely flat_ there) function on various intervals in the real axis.  The function has $f(x)\ge 0$, and is zero only at $x=0$.  Therefore the integral of $f(x)$ from any $a$ to any $b > a$ will be positive.  Essentially positive functions have positive area underneath them, end of story.  See the figure below.  We do avoid computing the function at exactly zero; but that's just laziness on our part.

```python
from matplotlib import pyplot as plt
```

```python
# plt imported in a previous cell.
import numpy as np
n = 212 # Even, so we don't include zero
xi = np.linspace(-1, 1, n)
yi = np.zeros(n)
dx = 2/n
area = 0
for i in range(n):
    yi[i] = np.exp(-1.0/xi[i])/(xi[i]**2*(1+np.exp(-1/xi[i]))**2)
    area += yi[i]*dx

print( 'Using a Riemann sum, the area under the curve from -1 to 1 is approximately ', area )
plt.scatter(xi,yi,s=2)
plt.axis([-1,1 , 0, 0.5])
plt.show()
```

```python
f = exp(-1/x)/(x**2*(1+exp(-1/x))**2)
F = integrate(f, x )
print( 'The integral from SymPy is ', F )
FTOC = F.subs(x,1) - F.subs(x,-1)
print( 'The area under the curve from -1 to 1 is positive, not ', FTOC )
defint = integrate( f, (x,-1,1))
print( 'Using definite integration gets the same ', simplify(defint) )
E = np.exp(1.0)
answe = 1/(1+E) - 1/(1/E+1)
print( "That numerically evaluates to ", answe, "which is negative.  Oops.")
```

    The integral from SymPy is  -1/(1 + exp(-1/x))
    The area under the curve from -1 to 1 is positive, not  -1/(exp(-1) + 1) + 1/(1 + E)
    Using definite integration gets the same  (1 - E)/(1 + E)
    That numerically evaluates to  -0.4621171572600098 which is negative.  Oops.

```python
YI = np.zeros(n)
for i in range(n):
    YI[i] = -1/(1+np.exp(-1/xi[i]))
plt.scatter(xi,YI,s=2)
plt.axis([-1,1 , -1.25, 0.25])
plt.show()
```

Without calling "simplify," SymPy didn't recognize that `1/(1+E)-1/(exp(-1)+1) = (1-E)/(1+E)` (if by `E` Sympy means exp(1)).  But that's a smaller problem.  The computed answer has missed the discontinuity in the antiderivative at zero.

As of this writing, many computer algebra systems (not just SymPy) are broken by this example.  Maple's indefinite integral returns exactly the same wrong answer.  Matlab's Symbolic Toolbox gets it right!

There is something very important happening in that expression, though.  The computer algebra system has returned a discontinuous function as an antiderivative of a continuous function.

Anyone remember the tl;dr of the Fundamental Theorem of Calculus?  Likely not.  Even we have to look it up before we teach it, to be sure we have the fine details right.  The basic idea is, though, that integration smooths things out: integrate a continuous function, you get a continuously differentiable function, which is smoother.  Jump discontinuities spuriously introduced are right out.

So, what the computer algebra system is doing there is not integration.  Anyone get their knuckles rapped for not adding the "constant of integration" when you integrated on a test?  Lost a mark or two, maybe?

Turns out that's what's going on here.  By adding different constants in different intervals to this purported antiderivative, we can find a smooth antiderivative of $f(x)$.  The answer returned by SymPy (and Maple) is "correct" from the point of view of differential algebra, where constants are things that differentiate to zero and don't have "values" as such.

Many people who implement computer algebra systems (even some of our friends at Maple) argue that they are right and we are wrong, and don't see the point.  But they are wrong, and one day they'll get it.  Now, they do have a fix-up in their definite integration code: if you ask Maple int( f, x=-1..1) you will get the correct answer $\frac{2}{{\mathrm e}+1} \approx 0.53788\ldots$ after simplification.  But you have to ask Maple the right way, by using the syntax for definite integrals (as you see above, SymPy is different, and does not give the right answer that way).

Matlab gets it.  Matlab's Symbolic Toolbox works quite hard to get continuous antiderivatives.  So we have confidence that Maple will one day get it. But Wolfram Alpha gets a discontinuous antiderivative, too, at this time of writing, so there's only a little pressure from the competition.

These facts have been known for quite a while now.  See [The Importance of Being Continuous](https://www.tandfonline.com/doi/abs/10.1080/0025570X.1994.11996235) by David Jeffrey, a paper that was written in 1994 (long predating Python or SymPy).  See also [Rectifying Transformations for Trigonometric Integrals](https://www.sciencedirect.com/science/article/pii/S0747717197901528) by the same author.

The disagreement amongst mathematicians---algebra versus analysis---has been going on for much longer, and goes at least back to Cauchy.  We are on Cauchy's side, here.  But there are lots of people who just want formulas, and don't care if they are discontinuous.

RMC retweeted something apropos this morning: Prof Michael Kinyon said "I used to think it was weird when people disagreed with me, but now I understand that some people just enjoy the feeling of being wrong."


> [back](../README.md)

# gs-rs [ˈdʒiːzrs]

## Contact
Samuel Valenzuela, Florian Rohm , Dr. Daniel Pape, daniel.pape(at)tngtech.com

TNG Technology Consulting GmbH

## Abstract
**gs-rs** is a Rust library for the optimization of non-linear least squares problems embeddable
as a (hyper)graph.

It is suitable for the optimization of an error function with respect to a set of parameters as in
SLAM (simultaneous localization and mapping) or BA (bundle adjustment).

## Background Literature
The optimization algorithm used by **gs-rs** is based on parts of **g2o**. This project should suffice to understand the optimization's implementation in **gs-rs**.
The following papers by the developers of **g2o** are recommended if a deeper
understanding of the theory behind the algorithm is of interest:
* _g2o: A General Framework for Graph Optimization_, Kümmerle et al.: This paper documents the derivation of the algorithm's structure.
* _A Tutorial on Graph-Based SLAM_, Grisetti et al.: This paper contains additional comments on the calculations in 2D and 3D. Here it is presented how the least squares optimization works on a manifold.

## Hypergraph-Embeddable Optimization Problems
### Least Squares Optimization Problems
#### Definition
A least squares optimization problem with a set of constraints $\mathcal{C}$ can be generally formulated as:

\begin{align}
    F(\chi) &= \sum_{k\in\mathcal{C}}e_k(\chi_k, z_k)^{T}\Omega_k e_k(\chi_k, z_k)\\
    {\chi}^* &= \text{argmin}_{\chi} F(\chi),
\end{align}

where

\begin{align}
    \chi = (\chi_1^T,\dots,\chi_n^T)^T
\end{align}

is the structured vector of parameters subject to optimization and

\begin{align}
    \chi_k = (\chi_{k_1}^T,\dots,\chi_{k_q}^T)^T
\end{align}

is the structured vector of parameter blocks subject to the constraint $k\in\mathcal{C}$.

$z_k$ and $\Omega_k$ denote the mean and information matrix of constraint $k$, respectively.

Finally, $e_k(\chi_k, z_k)$ is a vector error function measuring how well the parameter blocks
$\chi_k$ satisfy the constraint $k$.

#### Linearization of the Error Function
Given an initial guess $\hat{\chi}$ of the parameter set $\chi$, we can expand the error function
by means of a first order _Taylor_ expansion:

\begin{align}
    e_k(\chi_k) &= e_k(\hat{\chi_k}+\Delta\chi_k)\\
    &\approx e_k(\hat{\chi_k})+J_{k}\Delta\chi_k,
\end{align}

where $J_{k}=\nabla e_k(\hat{\chi_k})$ is the
_Jacobian_ computed at $\hat{\chi}_k$.

Insertion into formulation of the optimization problem yields:

\begin{align}
    F_k(\hat{\chi_k}+\Delta\chi_k) &= e_k(\hat{\chi_k}+\Delta\chi_k)^{T}\Omega_k e_k(\hat{\chi_k}+\Delta\chi_k)\\
    &\approx \left(e_k(\hat{\chi_k})+J_{k}\Delta\chi_{k}\right)^T\Omega_k\left(e_k(\hat{\chi_k})+J_{k}\Delta\chi_{k}\right)\\
    &= \underbrace{e_k(\hat{\chi_k})^T\Omega_k e_k(\hat{\chi_k})}_{c_k}
    + 2\underbrace{e_k(\hat{\chi_k})^{T}\Omega_{k}J_k}_{b_k}\Delta \chi_k
    + \Delta\chi_k^T \underbrace{J_k^T\Omega_k J_k}_{H_k}\Delta \chi_k\\
    &= c_k + 2b_{k}\Delta \chi_k + \Delta \chi_k^T H_k \Delta \chi_k.
\end{align}

Generalizing this result to the set of all constraints yields:

\begin{align}
    F(\hat{\chi}+\Delta \chi) &= \sum_{k\in\mathcal{C}} F_k(\hat{\chi_k}+ \Delta\chi_k)\\
    &= \sum_{k\in\mathcal{C}} c_k + 2b_{k}\Delta \chi_k + \Delta \chi_k^T H_k \Delta \chi_k\\
    &= c + 2b^T\Delta\chi + \Delta \chi^T H \Delta \chi.
\end{align}

where

\begin{align}
    c &= \sum_{k\in\mathcal{C}}c_{k}\\
    b &= \bigoplus_{k\in\mathcal{C}}b_{k}\\
    \Delta \chi &= \bigoplus_{k\in\mathcal{C}}\Delta \chi_{k}\\
    H &= \bigoplus_{k\in\mathcal{C}} H_{k}.
\end{align}

Here, $\bigoplus$ denotes an addition of blocks for vectors and matrices, where the position is derived from the constraint index.

We need to minimize $F(\hat\chi+\Delta\chi)$ in $\Delta\chi$. It can be done by solving the linear system

\begin{align}
    H\Delta\chi^\star = -b
\end{align}

#### Example
Let us consider a system of two parameter blocks $\chi_0$, $\chi_1$ (e.g. poses in two dimensions)
and three constraints $z_0$, $z_1$ (e.g. position measurements), $z_{01}$ (e.g. an odometry measurement), where the first two constraints
control only one parameter block each and the
last constraint connects the two parameter blocks.

We will set $z_{0}=(0,1)^T, z_1 = (1,0)^T, z_{01} = (0.5, -0.5)^T$.
Thus, we have a system, where the position measurements are further apart
than the odometry requires.

The information matrices are defined as follows:

\begin{align}
    \Omega_{0} &= \begin{pmatrix}
                      10 & 0 \\ 0 & 10
    \end{pmatrix}\\
    \Omega_{1} &= \begin{pmatrix}
                      10 & 0 \\ 0 & 10
    \end{pmatrix}\\
    \Omega_{01} &=\begin{pmatrix}
                      1 & 0 \\ 0 & 1
    \end{pmatrix}
\end{align}

That is, the odometry measurement is considered ten times less precise than the position measurement.

We will define the error functions

\begin{align}
    e_0(\chi_0,z_0) &= z_0 - \chi_0 \\
    e_1(\chi_1,z_1) &= z_1 - \chi_1 \\
    e_{01}(\chi_{01},z_{01}) &= z_{01} - (\chi_1 - \chi_0).
\end{align}

Thus, for the _Jacobian_'s we find

\begin{align}
J_0 &= \begin{pmatrix}
           \frac{\partial (z_0 - \chi_0)_x}{\partial (\chi_0)_x} & \frac{\partial (z_0 - \chi_0)_x}{\partial (\chi_0)_y} \\
           \frac{\partial (z_0 - \chi_0)_y}{\partial (\chi_0)_x} & \frac{\partial (z_0 - \chi_0)_y}{\partial (\chi_0)_y}
\end{pmatrix}\newline
&= \begin{pmatrix}
       1 & 0 \\ 0 & 1
\end{pmatrix}\newline
J_1 &= \begin{pmatrix}
           1 & 0 \\ 0 & 1
\end{pmatrix}\newline
J_{01} &= \begin{pmatrix}
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_x}{\partial (\chi_0)_x} &
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_x}{\partial (\chi_0)_y} &
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_x}{\partial (\chi_1)_x} &
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_x}{\partial (\chi_1)_y} \\
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_y}{\partial (\chi_0)_x} &
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_y}{\partial (\chi_0)_y} &
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_y}{\partial (\chi_1)_x} &
              \frac{\partial (z_0 - (\chi_1 - \chi_0))_y}{\partial (\chi_1)_y}
\end{pmatrix}\\
&= \begin{pmatrix}
       -1 & 0 & 1 & 0 \\ 0 & -1 & 0 & 1
\end{pmatrix}
\end{align}

Let us apply the position constraints as initial guesses of the pose variables:

\begin{align}
    \hat{\chi_0} &= z_0 = (0,1)^{T}\\
    \hat{\chi_1} &= z_1 = (1,0)^{T}.
\end{align}

Now we can calculate the components of our target equation ?:

\begin{align}
    b_0 &= e_0^T \cdotp \Omega_0 J_{0}\\
    &= (0,0) \cdotp \begin{pmatrix}
                        10 & 0 \\ 0 & 10
    \end{pmatrix} \cdotp \begin{pmatrix}
                             1 & 0 \\ 0 & 1
    \end{pmatrix}\newline
    &= (0,0).\newline
    b_1 &= e_1^T \cdotp \Omega_{1} J_{1}\newline
    &= (0,0) \cdotp \begin{pmatrix}
                        10 & 0 \\ 0 & 10
    \end{pmatrix} \cdotp \begin{pmatrix}
                             1 & 0 \\ 0 & 1
    \end{pmatrix}\newline
    &= (0,0).\\
    b_{01} &= e_{01}^T \cdotp \Omega_{01} J_{01}\newline
    &= \left( (0.5, -0.5) - \left( (1,0) - (0,1) \right) \right)\cdotp \begin{pmatrix}
                                                                           1 & 0 \\ 0 & 1
    \end{pmatrix} \cdotp \begin{pmatrix}
                             1 & 0 & -1 & 0 \\ 0 & 1 & 0 & -1
    \end{pmatrix}\newline
    &= (-0.5, 0.5)\cdotp \begin{pmatrix}
                             1 & 0 \\ 0 & 1
    \end{pmatrix} \cdotp \begin{pmatrix}
                             1 & 0 & -1 & 0 \\ 0 & 1 & 0 & -1
    \end{pmatrix}\newline
    &= (-0.5, 0.5, 0.5, -0.5).\newline
    \Rightarrow b &= b_0 \oplus b_1 \oplus b_{01}\newline
    &= (0,0) \oplus (0,0) \oplus (-0.5, 0.5, 0.5, -0.5)\newline
    &= (-0.5, 0.5, 0.5, -0.5)\newline
    H_0 &= J_{0}^T \Omega_0 J_{0}\newline
    &= \begin{pmatrix}
           -1 & 0 \\ 0 & -1
    \end{pmatrix} \cdotp \begin{pmatrix}
                             10 & 0 \\ 0 & 10
    \end{pmatrix} \cdotp \begin{pmatrix}
                             -1 & 0 \\ 0 & -1
    \end{pmatrix}\newline
    &= \begin{pmatrix}
           10 & 0 \\ 0 & 10
    \end{pmatrix}.\newline
    H_1 &= J_{1}^T \Omega_1 J_{1}\newline
    &= \begin{pmatrix}
           10 & 0 \\ 0 & 10
    \end{pmatrix}.\newline
    H_{01} &= J_{01}^T \Omega_{01} J_{01}\newline
    &= \begin{pmatrix}
           1 & 0 \\ 0 & 1 \\ -1 & 0 \\ 0 & -1
    \end{pmatrix} \cdotp \begin{pmatrix}
                             1 & 0 \\ 0 & 1
    \end{pmatrix} \cdotp \begin{pmatrix}
                             1 & 0 & -1 & 0 \\ 0 & 1 & 0 & -1
    \end{pmatrix}\newline
    &= \begin{pmatrix}
           1 & 0 & -1 & 0\\ 0 & 1 & 0 & -1 \\ -1 & 0 & 1 & 0 \\ 0 & -1 & 0 & 1
    \end{pmatrix}.\newline
    \Rightarrow H &= H_0 \oplus H_1 \oplus H_{01}\\
    &= \begin{pmatrix}
           10 & 0 \\ 0 & 10
    \end{pmatrix} \oplus \begin{pmatrix}
                             10 & 0 \\ 0 & 10
    \end{pmatrix} \oplus \begin{pmatrix}
                             1 & 0 & -1 & 0\\ 0 & 1 & 0 & -1 \\ -1 & 0 & 1 & 0 \\ 0 & -1 & 0 & 1
    \end{pmatrix}\newline
    & = \begin{pmatrix}
            11 & 0 & -1 & 0\\ 0 & 11 & 0 & -1 \\ -1 & 0 & 11 & 0 \\ 0 & -1 & 0 & 11
    \end{pmatrix}
\end{align}

Now, the last step is to solve

\begin{align}
    H\Delta \chi^* &= -b\newline
    \begin{pmatrix}
        11 & 0 & -1 & 0\\ 0 & 11 & 0 & -1 \\ -1 & 0 & 11 & 0 \\ 0 & -1 & 0 & 11
    \end{pmatrix} \Delta \chi^* &=
    \begin{pmatrix}
        0.5 \\ -0.5 \\ -0.5 \\ 0.5
    \end{pmatrix} \newline
    \Rightarrow \Delta \chi^* &= \frac{1}{24} \begin{pmatrix}
                                                  1 \\ -1 \\ -1 \\ 1
    \end{pmatrix}.
\end{align}

and apply the update

\begin{align}
    \chi^{(1)} &= \hat{\chi} + \Delta \chi^{*}\\
    &= (0,1,1,0)^T + \frac{1}{24}(1, -1, -1, 1)^{T}\\
    &\approx (0.04, 0.96, 0.96, 0.04).
\end{align}

Thus, our pose variables moved a little closer together, driven by the odometry constraint.

## Optimization Algorithm
In the following sections, the optimization algorithm will be introduced. Firstly, the structure's main
iteration steps will be introduced. Most of the work happens in the first step, the calculation of
$H$ and $b$. Subsequently, the calculations which apply to all dimensions and factor types will be presented.

### Iteration Steps
Given a specific number of iterations $n$ and the initial guess $x_i^{(0)}$ for each variable, the optimizer algorithm will repeat the following steps $n$ times:

1. Calculate $H$ and $b$ by setting them to $\boldsymbol{0}$, then looping through all factors and updating their non-fixed variables' entries in $H$ and $b$.
2. Calculate $\Delta x$, the vector containing data about how much each current variable guess $x_i^{(k)}$ should be updated in this step, by solving the linear system
\begin{align}
	H \Delta x = -b^T.
\end{align}
3. Update the guesses for each non-fixed variable $x_i$ with
\begin{align}
	x_i^{(k+1)} = x_i^{(k)} + \Delta x_i.
\end{align}

In the case of 2D variables with a rotation, normalize the rotation to $[-\pi, \pi)$.
In the case of 3D variables with a rotation, normalize the rotation of $\Delta x_i$.
As explained in a later section, the quaternion data will only include the imaginary part.
Therefore, set the scalar part to $1$ before normalizing. The $+$ operator then resembles the concatenation of two isometries,
analogously as the $\circ$ operator later.

### Calculation of $\boldsymbol{H}$ and $\boldsymbol{b}$
How exactly the variables' entries in $H$ and $b$ are updated depends on the factor type. In all cases the factor's increments on
parts of $H$ and $b$, $H^{\text{fac}}$ and $b^{\text{fac}}$,
respectively, will be computed as follows:

\begin{align}
H^{\text{fac}} = J^T \cdot \Omega \cdot J,
\end{align}

\begin{align}
b^{\text{fac}} = e^T \cdot \Omega \cdot J,
\end{align}

where $\Omega$, $J$ and $e$ are the factor's information matrix, Jacobian matrix and error vector, respectively. While $\Omega$
is a given constant of the factor, $J$ and $e$ have to be
calculated for each factor in each iteration.

If the factor only involves the variable $x_i$, $H$ and $b$ are updated as follows:

\begin{align}
H_{ii} = H_{ii} + H^{\text{fac}},
\end{align}

\begin{align}
b_i = b_i + b^{\text{fac}},
\end{align}

where the subscripts of $H$ and $b$ denote the row and column index of the submatrix or subvector assigned to the respective variable.
If the factor involves two variables $x_i$ and
$x_j$, $H^{\text{fac}}$ and $b^{\text{fac}}$ will have the structure

\begin{align}
H^{fac} =
\begin{pmatrix}
\boldsymbol{H}^{\text{fac}}_{ii} & \boldsymbol{H}^{\text{fac}}_{ij}\\
\boldsymbol{H}^{\text{fac}}_{ji} & \boldsymbol{H}^{\text{fac}}_{jj}
\end{pmatrix}
\end{align}

and

\begin{align}
b^{fac} =
\begin{pmatrix}
\boldsymbol{b}^{fac}_{i} & \boldsymbol{b}^{fac}_{j}
\end{pmatrix},
\end{align}

respectively, such that $H_{mn}$ will be incremented by $H^{\text{fac}}_{mn}$ and $b_n$
will be incremented by $b^{\text{fac}}_n$.
Fixed variables are excluded from $H$ and $b$ and therefore do not have any submatrices or subvectors which would need to be updated.
In this case, these parts of $H^{\text{fac}}$ and $b^{\text{fac}}$
are simply ignored.

In the following sections, the calculation is described for all 2D and 3D factors supported by **gs-rs**.

## Optimization in 2D
In the following sections, the individual 2D factors' calculations of $J$ and $e$ are presented. The functions $\text{pos}(x)$ and $\text{rot}(x)$ will be used to refer to the 2D position vector and the rotation angle of a 2D pose $x$, respectively.
Similarly, the functions $\text{pos}_x(x)$ and $\text{pos}_y(x)$ will be used to refer to the single value within the respective dimension.

Furthermore, the rotation matrices are denoted such that

\begin{align}
	R_\alpha =
	\begin{pmatrix}
		\text{cos}(\alpha) & -\text{sin}(\alpha)\\
		\text{sin}(\alpha) &  \text{cos}(\alpha)
	\end{pmatrix}
\end{align}

equals the rotation matrix with the angle $\alpha$ in 2D, and

\begin{align}
	R_z(\alpha) =
	\begin{pmatrix}
		\text{cos}(\alpha) & -\text{sin}(\alpha) & 0\\
		\text{sin}(\alpha) &  \text{cos}(\alpha) & 0\\
		          0 &            0 & 1
	\end{pmatrix}
\end{align}

equals the rotation matrix for a rotation around the $z$ axis with the angle $\alpha$ in 3D.

### Position2D
The _Position2D_ factor involves one _VehicleVariable_ $x_v$.
The Jacobian matrix $J$ in this case is

\begin{align}
	J = R_z(-\text{rot}(x_v)).
\end{align}

Given the measurement $x_m$, the error vector

\begin{align}
	e = R_{-\text{rot}(x_m)} \circ (\text{pos}(x_v) - \text{pos}(x_m))
\end{align}

can be computed as well.

### Odometry2D

The _Odometry2D_ factor involves two _VehicleVariable_s $x_i$ and $x_j$.
Given the measurement $x_{ij}$, the Jacobian matrix $J$ is calculated as follows:

\begin{align}
	\Delta x_{ij} = x_j - x_i
\end{align}

\begin{align}
	\text{sin}_i = \text{sin}(\text{rot}(x_i))
\end{align}

\begin{align}
	\text{cos}_i = \text{cos}(\text{rot}(x_i))
\end{align}

\begin{align}
	@inproceedings{kummerle2011g2o,
      title={g2o: A general framework for graph optimization},
      author={Kümmerle, Rainer and Grisetti, Giorgio and Strasdat, Hauke and Konolige, Kurt and Burgard, Wolfram},
      booktitle={2011 IEEE International Conference on Robotics and Automation},
      pages={3607--3613},
      year={2011},
      organization={IEEE}
    }

    @article{grisetti2010tutorial,
      title={A tutorial on graph-based SLAM},
      author={Grisetti, Giorgio and Kummerle, Rainer and Stachniss, Cyrill and Burgard, Wolfram},
      journal={IEEE Intelligent Transportation Systems Magazine},
      volume={2},
      number={4},
      pages={31--43},
      year={2010},
      publisher={IEEE}
    }

	\begin{pmatrix}
		-\text{cos}_i & -\text{sin}_i & -\text{sin}_i\cdot\text{pos}_x(\Delta x_{ij}) + \text{cos}_i\cdot\text{pos}_y(\Delta x_{ij})\\
		 \text{sin}_i & -\text{cos}_i & -\text{cos}_i\cdot\text{pos}_x(\Delta x_{ij}) - \text{sin}_i\cdot\text{pos}_y(\Delta x_{ij})\\
		     0 &      0 &                                   -1
	\end{pmatrix}
\end{align}

\begin{align}
	J_j = R_z(-\text{rot}(x_{ij})) \cdot R_z(-\text{rot}(x_i))
\end{align}

\begin{align}
	J =
	\begin{pmatrix}
		\boldsymbol{J_i} & \boldsymbol{J_j}
	\end{pmatrix}
\end{align}

The error vector $e$ is computed as follows:

\begin{align}
	e_{pos} = R_{-\text{rot}(x_{ij})} \cdot (R_{-\text{rot}(x_i)} \cdot \text{pos}(\Delta x_{ij}) - \text{pos}(x_{ij}))
\end{align}

\begin{align}
	e_{rot} = \text{rot}(\Delta x_{ij}) - \text{rot}(x_{ij})
\end{align}

After normalizing $e_{rot}$ to $[-\pi, \pi)$ as $e_{rot}^0$ the full error vector can be constructed with

\begin{align}
	e =
	\begin{pmatrix}
		\boldsymbol{e_{\text{pos}}}\\
		e_{\text{rot}}^0
	\end{pmatrix}
	.
\end{align}

### Observation2D
The `Observation2D` factor involves one `VehicleVariable` $x_i$ and one `LandmarkVariable` $x_j$. The measurement is denoted as $x_{ij}$,
analogously to the previous section. Although $x_j$ and $x_{ij}$ are only positions rather than poses and therefore do not contain a rotation angle, the functions $\text{pos}(x)$, $\text{pos}_x(x)$ and $\text{pos}_y(x)$ will be used nevertheless to make the calculation path more understandable. Given the measurement $x_{ij}$, the Jacobian matrix $J$ is calculated as follows:

\begin{align}
	\text{pos}(\Delta x_{ij}) = \text{pos}(x_j) - \text{pos}(x_i)
\end{align}

\begin{align}
	\text{sin}_i = \text{sin}(\text{rot}(x_i))
\end{align}

\begin{align}
	\text{cos}_i = \text{cos}(\text{rot}(x_i))
\end{align}

\begin{align}
	J_i =
	\begin{pmatrix}
		-\text{cos}_i & -\text{sin}_i & -\text{sin}_i \cdot \text{pos}_x(\Delta x_{ij}) + \text{cos}_i \cdot \text{pos}_y(\Delta x_{ij})\\
		 \text{sin}_i & -\text{cos}_i & -\text{cos}_i \cdot \text{pos}_x(\Delta x_{ij}) - \text{sin}_i \cdot \text{pos}_y(\Delta x_{ij})
	\end{pmatrix}
\end{align}

\begin{align}
	J_j = R_{-\text{rot}(x_i)}
\end{align}

\begin{align}
	J =
	\begin{pmatrix}
		\boldsymbol{J_i} & \boldsymbol{J_j}
	\end{pmatrix}
\end{align}

The error vector $e$ is computed as follows:

\begin{align}
	e = R_{-\text{rot}(x_i)} \cdot \text{pos}(\Delta x_{ij}) - \text{pos}(x_{ij})
\end{align}

## Optimization in 3D

In 3D, rotations are often represented as quaternions.
To reduce these to a minimal representation, only the imaginary part of the unit quaternions is saved within the solution $\Delta x$,

In the following sections, the individual 3D factors' calculations of $J$ and $e$ are presented.
The functions $\text{pos}(x)$ and $\text{rot}_Q(x)$ will be used to refer to the 3D position vector and the rotation quaternion of a 3D pose or isometry $x$.
When referring to the rotation as a 3D rotation matrix, the function $\text{rot}_{RM}(x)$ will be used instead.
When using $x$ in calculations, it will refer to the pose's 3D isometry containing both the position vector as translation as well as the rotation quaternion.
The binary operator $\circ$ will be used to concatenate these isometries.
The inverse of an isometry will be denoted as $x^{-1}$.

Some functions will be used to calculate gradients of a 3D isometry $x$, such as $\frac{\text{d}q}{\text{d}R}(x)$, which is computed as follows using the trace function $\text{tr}(M)$ of 3x3 matrices:

\begin{align}
	s = \frac{1}{2}\sqrt{\text{tr}(\text{rot}_{RM}(x)) + 1}
\end{align}

\begin{align}
	a_1 = -\frac{\text{rot}_{RM}(x)_{21}-\text{rot}_{RM}(x)_{12}}{32s^3}
\end{align}

\begin{align}
	a_2 = -\frac{\text{rot}_{RM}(x)_{02}-\text{rot}_{RM}(x)_{20}}{32s^3}
\end{align}

\begin{align}
	a_3 = -\frac{\text{rot}_{RM}(x)_{10}-\text{rot}_{RM}(x)_{01}}{32s^3}
\end{align}

\begin{align}
	b = \frac{1}{4s}
\end{align}

\begin{align}
	\frac{\text{d}q}{\text{d}R}(x) =
	\begin{pmatrix}
		a_1 & 0 &  0 &  0 & a_1 & b & 0 & -b & a_1\\
		a_2 & 0 & -b &  0 & a_2 & 0 & b &  0 & a_2\\
		a_3 & b &  0 & -b & a_3 & 0 & 0 &  0 & a_3
	\end{pmatrix}
\end{align}

It will always be assumed that the condition $\text{tr}(\text{rot}_{RM}(x)) > 0$ holds, i.e. we are dealing with proper rotations.

Another function used in this context is $\text{skew}(x)$, computed as follows using a 3D vector or position $x$:

\begin{align}
	\text{skew}(x) =
	\begin{pmatrix}
		   0 &  x_z & -x_y\\
		-x_z &    0 &  x_x\\
		 x_y & -x_x &    0
	\end{pmatrix},
\end{align}

where $x_x$, $x_y$ and $x_z$ are the first, second and third vector components, respectively. The $\text{skew}(x)$ can be interpreted as the matrix
representation of the cross product with $x$.

### Position3D

The `Position3D` factor involves one `VehicleVariable` $x_v$.
Given the measurement $x_m$, the Jacobian matrix $J$ is calculated as follows:

\begin{align}
	E = x_m^{-1} \circ x_v
\end{align}

\begin{align}
	\frac{\text{d}te}{\text{d}qj} = \frac{\text{d}q}{\text{d}R}(E) \circ
	\begin{pmatrix}
		\text{skew}(\text{rot}_{RM}(E)_0)\\
		\text{skew}(\text{rot}_{RM}(E)_1)\\
		\text{skew}(\text{rot}_{RM}(E)_2)
	\end{pmatrix}
\end{align}

\begin{align}
	J =
	\begin{pmatrix}
		\boldsymbol{E} &               \boldsymbol{0}\\
		\boldsymbol{0} & \boldsymbol{\frac{\text{d}te}{\text{d}qj}}
	\end{pmatrix},
\end{align}

where $M_i$ refers to the column at index $i$ of the matrix $M$.

The error vector $e$ is computed as follows:

\begin{align}
	e = x_m^{-1} \circ x_v
\end{align}

As mentioned before, the scalar part of the quaternion is removed in $e$. $e$ is then interpreted as a six-dimensional
vector with the top three entries for its position and the bottom three entries for its rotation.

### Odometry3D
The `Odometry3D` factor involves two `VehicleVariable`s $x_i$ and $x_j$.
Given the measurement $x_{ij}$, the Jacobian matrix $J$ is calculated as follows:

\begin{align}
	A = x_{ij}^{-1}
\end{align}

\begin{align}
	B = x_i^{-1} \circ x_j
\end{align}

\begin{align}
	E = A \circ B
\end{align}

\begin{align}
	\frac{dte}{dqi} = \text{rot}_{RM}(A) \circ \text{skew}(\text{pos}(B))
\end{align}

\begin{align}
	\frac{dre}{dqi} = \frac{dq}{dR}(E) \circ
	\begin{pmatrix}
		\boldsymbol{\text{rot}_{RM}(A) \circ \text{skew}(\text{rot}_{RM}(B)_0)^T}\\
		\boldsymbol{\text{rot}_{RM}(A) \circ \text{skew}(\text{rot}_{RM}(B)_1)^T}\\
		\boldsymbol{\text{rot}_{RM}(A) \circ \text{skew}(\text{rot}_{RM}(B)_2)^T}
	\end{pmatrix}
\end{align}

\begin{align}
	J_i =
	\begin{pmatrix}
	\boldsymbol{-\text{rot}_{RM}(A)} & \boldsymbol{\frac{dte}{dqi}}\\
	\boldsymbol{0} & \boldsymbol{\frac{dre}{dqi}}
	\end{pmatrix}
\end{align}

\begin{align}
	\frac{dre}{dqj} = \frac{dq}{dR}(E) *
	\begin{pmatrix}
		\boldsymbol{\text{skew}(\text{rot}_{RM}(E)_0)}\\
		\boldsymbol{\text{skew}(\text{rot}_{RM}(E)_1)}\\
		\boldsymbol{\text{skew}(\text{rot}_{RM}(E)_2)}
	\end{pmatrix}
\end{align}

\begin{align}
	J_j =
	\begin{pmatrix}
		\boldsymbol{\text{rot}_{RM}(E)} & \boldsymbol{0}\\
		          \boldsymbol{0} & \boldsymbol{\frac{dre}{dqj}}
	\end{pmatrix}
\end{align}

\begin{align}
	J =
	\begin{pmatrix}
		\boldsymbol{J_i} & \boldsymbol{J_j}
	\end{pmatrix},
\end{align}

where $M_i$ refers to the column at index $i$ of the matrix $M$.

The error vector $e$ is computed as follows:

\begin{align}
	e = x_{ij}^{-1} \circ x_i^{-1} \circ x_j
\end{align}

As mentioned in section ?, the scalar part of the quaternion is removed in $e$. $e$ is then interpreted as a six-dimensional vector with the
top three entries for its position and the bottom three entries for its rotation.

### Observation3D
The `Observation3D` factor involves one `VehicleVariable` $x_i$ and one `LandmarkVariable` $x_j$.
In the following, the binary operator $\circ$ denotes the transformation of the position given by the right operand by the isometry given by the left operand.

With the measurement $x_{ij}$, the Jacobian matrix $J$ is calculated as follows:

\begin{align}
	J =
	\begin{pmatrix}
		\boldsymbol{-I_3} & \boldsymbol{\text{skew}(\text{pos}(x_i^{-1} \circ \text{pos}(x_j)))^T} & \boldsymbol{\text{rot}_{RM}(x_i^{-1})}
	\end{pmatrix}
\end{align}

Finally, the error vector $e$ is computed as follows:

\begin{align}
	e = \text{pos}(x_i^{-1} \circ \text{pos}(x_j)) - \text{pos}(x_{ij})
\end{align}

## References
[1] _g2o: A general framework for graph optimization_, Kümmerle, Grisetti, Strasdat, Konolige and Burgard,
2011 IEEE International Conference on Robotics and Automation, 3607 - 3613 (2011), IEEE.

[2] _A tutorial on graph-based SLAM_, Grisetti, Kummerle, Stachniss and Burgard,
  IEEE Intelligent Transportation Systems Magazine, 2(4), 31-43 (2010), IEEE


```python
%matplotlib widget
# %matplotlib qt
import matplotlib.pyplot as plt
import json
import numpy as np
import pandas as pd
import sympy as sp
from sympy import symbols
import bmcs_utils.api as bu
from bmcs_cross_section.api import MKappa, TShape, EC2, BarLayer, ReinfLayer
from bmcs_beam.api import BoundaryConditions, BoundaryConfig, DeflectionProfile, BeamDesign, BeamSLSCurve, LoadDeflectionParamsStudy
from os.path import join
from mpl_toolkits import mplot3d
sp.init_printing()
```

```python
def get_Fu_and_Fs(dp, n_i, rho_range=(0.0025, 0.025), slenderness_range=(10, 35), upper_reinforcement = False):
    rho_range = np.linspace(*rho_range, n_i)
    slenderness_range = np.linspace(*slenderness_range, n_i)

# def get_Fu_and_Fs(dp, rho_range=np.linspace(0.0025, 0.025), slenderness_range=np.linspace(10, 35), upper_reinforcement = False):
    if upper_reinforcement:
        d = dp.mc.cross_section_layout.items[0].z
    else:
        d = dp.mc.cross_section_shape_.H -  dp.mc.cross_section_layout.items[0].z
    b = dp.mc.cross_section_shape_.B
    area_g = b * d

    rho_grid, sl_grid = np.meshgrid(rho_range, slenderness_range)
    F_u_grid = np.zeros_like(rho_grid)
    F_s_grid = np.zeros_like(rho_grid)

    _ , ax = plt.subplots()
    ax.set_xlabel(r'$w$ [mm]')
    ax.set_ylabel(r'$F$ [KN]')

    for sl_idx in range(0,len(slenderness_range)):
        for rho_idx in range(0,len(rho_range)):
            rho = rho_grid[rho_idx, sl_idx]
            sl = sl_grid[rho_idx, sl_idx]

            print('parameter combination', rho, sl)

            # assigning the grid area (area_g) to the reinforcement area variable
            A_j_g = rho * area_g
            dp.mc.cross_section_layout.items[0].A = A_j_g

            # assigning the grid length (L_g) to the beam length variable
            L_g = sl * d
            dp.beam_design.L = L_g

            dp.mc.state_changed = True

            # running the deflection analysis
            F_data, w_data = dp.get_Fw()

            # plotting, post-processing & saving the data
            ax.plot(w_data, F_data/1000, label = "rho={}%-sl={} ".format(rho * 100, sl))

            w_s = dp.beam_design.L / 250
            F_u = max(F_data)
            F_s = np.interp(w_s, w_data, F_data, right=F_u*2)

            F_u_grid[rho_idx,sl_idx] = F_u
            F_s_grid[rho_idx,sl_idx] = F_s

    return F_u_grid, F_s_grid, rho_grid, sl_grid
```

### 1. EC2 - Steel

```python
# Concrete C30/37

f_ck = 30

b = 1000
h = 300
d = 0.9 * h

fctk_0_05 = EC2.get_f_ctk_0_05(f_ck)
f_ctd = fctk_0_05
# f_ctd = 0.85 * fctk_0_05 / 1.5
f_ctd_fl = max((1.6 - h /1000) * f_ctd, f_ctd)

f_ctm = EC2.get_f_ctm(f_ck)
print(f_ctm)
# Info: eps_cr =  0.000170 with default concrete law gives good slenderness curve

mc1 = MKappa(low_kappa = 0, high_kappa = 0.00003)

mc1.cs_design.matrix='EC2 with plateau'
mc1.cs_design.matrix_.trait_set(
#         eps_cr=2.5* f_ctd_fl /E, #0.000125 * f_ck/30, # f_ctm /E,
#         eps_tu=0.002,
#         mu=0.0,
    f_cm = EC2.get_f_cm(f_ck),
    factor = 0.85/1.5,
    )

mc1.cross_section_shape_.B = b
mc1.cross_section_shape_.H = h

rho = 0.01
A_s= rho * b * d

# NOTE THAT FOR STEEL STRENGTH IS CALLED f_sy

bl1 = ReinfLayer(name='layer1', z=h-d, A=A_s, matmod='steel')
bl1.matmod_.trait_set(E_s=200000, f_sy=500, factor = 1/1.15 )
mc1.cross_section_layout.add_layer(bl1)

dp1 = DeflectionProfile(mc=mc1)
dp1.beam_design.beam_conf_name = BoundaryConfig.SIMPLE_BEAM_DIST_LOAD
# dp1.beam_design.beam_conf_name = BoundaryConfig.FIXED_AND_ROLLER_SUPPORT_DIST_LOAD
# dp1.beam_design.beam_conf_name = BoundaryConfig.FIXED_SUPPORT_DIST_LOAD
# dp1.beam_design.beam_conf_name = BoundaryConfig.CANTILEVER_DIST_LOAD

dp1.F_scale = 1
dp1.beam_design.L = 8 * d # l/d = 10
```

    2.896468153816889

```python
dp1.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

```python
F_u_grid_s, F_s_grid_s, rho_grid_s, sl_grid_s = get_Fu_and_Fs(dp1, 6)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    parameter combination 0.0025 10.0
    parameter combination 0.0025 15.0
    parameter combination 0.0025 20.0
    parameter combination 0.0025 25.0
    parameter combination 0.0025 30.0
    parameter combination 0.0025 35.0
    parameter combination 0.007000000000000001 10.0
    parameter combination 0.007000000000000001 15.0
    parameter combination 0.007000000000000001 20.0
    parameter combination 0.007000000000000001 25.0
    parameter combination 0.007000000000000001 30.0
    parameter combination 0.007000000000000001 35.0
    parameter combination 0.011500000000000002 10.0
    parameter combination 0.011500000000000002 15.0
    parameter combination 0.011500000000000002 20.0
    parameter combination 0.011500000000000002 25.0
    parameter combination 0.011500000000000002 30.0
    parameter combination 0.011500000000000002 35.0
    parameter combination 0.016 10.0
    parameter combination 0.016 15.0
    parameter combination 0.016 20.0
    parameter combination 0.016 25.0
    parameter combination 0.016 30.0
    parameter combination 0.016 35.0
    parameter combination 0.0205 10.0
    parameter combination 0.0205 15.0
    parameter combination 0.0205 20.0
    parameter combination 0.0205 25.0
    parameter combination 0.0205 30.0
    parameter combination 0.0205 35.0
    parameter combination 0.025 10.0
    parameter combination 0.025 15.0
    parameter combination 0.025 20.0
    parameter combination 0.025 25.0
    parameter combination 0.025 30.0
    parameter combination 0.025 35.0

```python
np.save('F_u_grid_steel.npy', F_u_grid_s)
np.save('F_s_grid_steel.npy', F_s_grid_s)
np.save('rho_grid_steel.npy', rho_grid_s)
np.save('sl_grid_steel.npy', sl_grid_s)
```

### 2. EC2 - Carbon

```python
f_ck = 30

b = 1000
h = 300
d = 0.9 * h

E = EC2.get_E_cm(f_ck)

fctk_0_05 = EC2.get_f_ctk_0_05(f_ck)
f_ctd = fctk_0_05
# f_ctd = 0.85 * fctk_0_05 / 1.5
f_ctd_fl = max((1.6 - h /1000) * f_ctd, f_ctd)

mc2 = MKappa(low_kappa = 0,
            high_kappa = 0.00007,
             n_kappa= 200,
           )

mc2.cs_design.matrix='EC2 with plateau'
mc2.cs_design.matrix_.trait_set(
        eps_cr=f_ctd_fl /E,
    f_cm = EC2.get_f_cm(f_ck),
    factor = 0.85/1.5
    )

mc2.cross_section_shape_.B = b
mc2.cross_section_shape_.H = h

rho = 0.01
# A_s=6 * np.pi * (32/2)**2
A_s= rho * b * d

bl2 = ReinfLayer(name='layer1', z=h-d, A=A_s, matmod='carbon')
bl2.matmod_.trait_set(E=230000, f_t=3000, factor = 1./1.5)
mc2.cross_section_layout.add_layer(bl2)

dp2 = DeflectionProfile(mc=mc2)
dp2.beam_design.beam_conf_name = BoundaryConfig.SIMPLE_BEAM_DIST_LOAD

dp2.F_scale = 1
dp2.beam_design.L = 8 * d

# rho1 = np.linspace(0.0005, 0.007, 7)
# rho2 = np.linspace(0.0075, 0.025, 5)
# rho_range=np.concatenate((rho1, rho2))

# sl1 = np.linspace(3, 10, 7)
# sl2 = np.linspace(11, 50, 5)
# slenderness_range=np.concatenate((sl1, sl2))
```

```python
dp2.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

```python
for f_ck in [30, 50, 70, 90, 100]:

    b = 1000
    h = 300
    d = 0.9 * h

    E = EC2.get_E_cm(f_ck)

    fctk_0_05 = EC2.get_f_ctk_0_05(f_ck)
    f_ctd = fctk_0_05
    # f_ctd = 0.85 * fctk_0_05 / 1.5
    f_ctd_fl = max((1.6 - h /1000) * f_ctd, f_ctd)

    eps_cr = f_ctd_fl/E

    mc2 = MKappa(low_kappa = 0,
                high_kappa = 0.00007, # 0.00005,
               )
    mc2.cs_design.matrix='EC2 with plateau'
    mc2.cs_design.matrix_.trait_set(
            E_cc=E,
            E_ct=E,
            eps_cr=f_ctd_fl /E, #eps_cr, #0.000132 * f_ck/30,
            eps_cy=EC2.get_eps_c2(f_ck),
            eps_cu=EC2.get_eps_cu2(f_ck),
            eps_tu=0.002,
            mu=0.0,
            f_cd = f_ck*0.85/1.5,
    #     factor = 1
    #     factor = 0.85/1.5
        )

    mc2.cross_section_shape_.B = b
    mc2.cross_section_shape_.H = h

    rho = 0.01
    # A_s=6 * np.pi * (32/2)**2
    A_s= rho * b * d

    bl2 = ReinfLayer(name='layer1', z=h-d, A=A_s, matmod='carbon')
    bl2.matmod_.trait_set(E=230000, f_t=3000, factor = 1./1.5) #factor = 1.)
    mc2.cross_section_layout.add_layer(bl2)

    dp2 = DeflectionProfile(mc=mc2)
    dp2.beam_design.beam_conf_name = BoundaryConfig.SIMPLE_BEAM_DIST_LOAD

    dp2.F_scale = 1
    dp2.beam_design.L = 10 * d

    rho1 = np.linspace(0.0005, 0.007, 12)
    rho2 = np.linspace(0.0075, 0.025, 8)
    rho_range=np.concatenate((rho1, rho2))

    sl1 = np.linspace(3, 10, 12)
    sl2 = np.linspace(11, 50, 8)
    slenderness_range=np.concatenate((sl1, sl2))

    F_u_grid_c, F_s_grid_c, rho_grid_c, sl_grid_c = get_Fu_and_Fs(dp2, rho_range=rho_range, slenderness_range=slenderness_range)

    np.save('F_u_grid_carbon_c' + str(f_ck) + '.npy', F_u_grid_c)
    np.save('F_s_grid_carbon_c' + str(f_ck) + '.npy', F_s_grid_c)
    np.save('rho_grid_carbon_c' + str(f_ck) + '.npy', rho_grid_c)
    np.save('sl_grid_carbon_c' + str(f_ck) + '.npy', sl_grid_c)

```

```python
dp2.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

```python
F_u_grid_c, F_s_grid_c, rho_grid_c, sl_grid_c = get_Fu_and_Fs(dp2, 7, rho_range=(0.0002, 0.025), slenderness_range=(3, 50))
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    parameter combination 0.0002 3.0
    parameter combination 0.0002 10.833333333333332
    parameter combination 0.0002 18.666666666666664
    parameter combination 0.0002 26.5
    parameter combination 0.0002 34.33333333333333
    parameter combination 0.0002 42.166666666666664
    parameter combination 0.0002 50.0
    parameter combination 0.004333333333333333 3.0
    parameter combination 0.004333333333333333 10.833333333333332
    parameter combination 0.004333333333333333 18.666666666666664
    parameter combination 0.004333333333333333 26.5
    parameter combination 0.004333333333333333 34.33333333333333
    parameter combination 0.004333333333333333 42.166666666666664
    parameter combination 0.004333333333333333 50.0
    parameter combination 0.008466666666666667 3.0
    parameter combination 0.008466666666666667 10.833333333333332
    parameter combination 0.008466666666666667 18.666666666666664
    parameter combination 0.008466666666666667 26.5
    parameter combination 0.008466666666666667 34.33333333333333
    parameter combination 0.008466666666666667 42.166666666666664
    parameter combination 0.008466666666666667 50.0
    parameter combination 0.012600000000000002 3.0
    parameter combination 0.012600000000000002 10.833333333333332
    parameter combination 0.012600000000000002 18.666666666666664
    parameter combination 0.012600000000000002 26.5
    parameter combination 0.012600000000000002 34.33333333333333
    parameter combination 0.012600000000000002 42.166666666666664
    parameter combination 0.012600000000000002 50.0
    parameter combination 0.016733333333333333 3.0
    parameter combination 0.016733333333333333 10.833333333333332
    parameter combination 0.016733333333333333 18.666666666666664
    parameter combination 0.016733333333333333 26.5
    parameter combination 0.016733333333333333 34.33333333333333
    parameter combination 0.016733333333333333 42.166666666666664
    parameter combination 0.016733333333333333 50.0
    parameter combination 0.020866666666666665 3.0
    parameter combination 0.020866666666666665 10.833333333333332
    parameter combination 0.020866666666666665 18.666666666666664
    parameter combination 0.020866666666666665 26.5
    parameter combination 0.020866666666666665 34.33333333333333
    parameter combination 0.020866666666666665 42.166666666666664
    parameter combination 0.020866666666666665 50.0
    parameter combination 0.025 3.0
    parameter combination 0.025 10.833333333333332
    parameter combination 0.025 18.666666666666664
    parameter combination 0.025 26.5
    parameter combination 0.025 34.33333333333333
    parameter combination 0.025 42.166666666666664
    parameter combination 0.025 50.0

```python
np.save('F_u_grid_carbon.npy', F_u_grid_c)
np.save('F_s_grid_carbon.npy', F_s_grid_c)
np.save('rho_grid_carbon.npy', rho_grid_c)
np.save('sl_grid_carbon.npy', sl_grid_c)
```

```python
fig, ax = plt.subplots()

z = F_u_grid_s / F_s_grid_s - 1./0.59
CS = ax.contour(rho_grid_s, sl_grid_s, z, levels=[0])
ax.clabel(CS, inline=1, fontsize=10)

# z2 = F_u_grid_c / F_s_grid_c - 1./0.59
# CS2 = ax.contour(rho_grid_c, sl_grid_c, z2)
# ax.clabel(CS2, inline=1, fontsize=10)

ax.set_ylabel('L/d')
ax.set_xlabel(r'$\rho$ %')
ax.set_ylim(0, 35)
ax.set_xlim(0.0, 0.025)
ax.plot([0.005, 0.005],[0, 35], c='black', lw=0.8)
# Draw EC2 curve
BeamSLSCurve().plot_steel_sls_curves(ax=ax, f_cks=[30], axes_start_from_zero=True)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

# Checking steel and carbon graphs for reinforcement ratio rho = 0.5%

## Steel

```python
# set rho = 0.5%
d = 0.9 * mc1.cross_section_shape_.H
mc1.cross_section_layout.items[0].A = 0.005 * mc1.cross_section_shape_.B * d
mc1.state_changed = True

params_config = [
    ['L',   dp1.beam_design, [7*d, 20*d]],
]

LoadDeflectionParamsStudy(dp1, show_sls_deflection_limit = True).run(params_config)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Parametric study is running...
    L: (1890.0), (5400.0),
    Parametric study finished.

```python
mc1.cross_section_layout.items[0].A = 0.005 * mc1.cross_section_shape_.B * (0.9 * mc1.cross_section_shape_.H)
mc1.state_changed = True
dp1.beam_design.L = 7 * d

print('L = ' + str(dp1.beam_design.L/1000) + ' m')
print('w_limit = ' + str(dp1.beam_design.L/250) + ' mm')
dp1.interact()
```

    L = 1.89 m
    w_limit = 7.56 mm

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

## Carbon

```python
# set rho = 0.5%
d = 0.9 * mc2.cross_section_shape_.H
mc2.cross_section_layout.items[0].A = 0.005 * mc2.cross_section_shape_.B * d
mc2.state_changed = True

params_config = [
    ['L',   dp2.beam_design, [7*d, 20*d]],
#         ['L',   dp2.beam_design, [5*d, 7*d, 10*d, 15*d, 20*d, 25*d]],
]

LoadDeflectionParamsStudy(dp2, show_sls_deflection_limit = True).run(params_config)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Parametric study is running...
    L: (1890.0), (5400.0),
    Parametric study finished.

```python
mc2.cross_section_layout.items[0].A = 0.005* mc2.cross_section_shape_.B * (0.9 * mc2.cross_section_shape_.H)
mc2.state_changed = True
dp2.beam_design.L = 30 * d
dp2.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

## Plot all curves

```python
f_cks = [50]
F_u_grids = []
F_s_grids = []
rho_grids = []
sl_grids = []
reinforcement = 'carbon'
for f_ck in f_cks:
    f_ck = str(f_ck)

    F_u_grids.append(np.load('exported_data/F_u_grid_carbon_EC2_eq2_tension_E230_ft_3000_c' + str(f_ck) + '.npy'))
    F_s_grids.append(np.load('exported_data/F_s_grid_carbon_EC2_eq2_tension_E230_ft_3000_c' + str(f_ck) + '.npy'))
    rho_grids.append(np.load('exported_data/rho_grid_carbon_EC2_eq2_tension_E230_ft_3000_c' + str(f_ck) + '.npy'))
    sl_grids.append(np.load('exported_data/sl_grid_carbon_EC2_eq2_tension_E230_ft_3000_c' + str(f_ck) + '.npy'))
```

```python
_, ax = plt.subplots(1,1)

ax.set_ylabel('L/d')
ax.set_xlabel(r'$\rho$ %')
ax.set_ylim(0, 35)
ax.set_xlim(0.0, 0.025)

for f_ck, F_u_grid, F_s_grid, rho_grid, sl_grid in zip(f_cks, F_u_grids, F_s_grids, rho_grids, sl_grids):
    z = 0.5 * F_u_grid / F_s_grid - 1./0.59
    CS = ax.contour(rho_grid, sl_grid, z, colors=[np.random.rand(3,)], levels=[0])
    CS.collections[0].set_label('C' + str(f_ck))
#     ax.clabel(CS, inline=1, fontsize=10)

    BeamSLSCurve().plot_steel_sls_curves(ax=ax, f_cks=[50], axes_start_from_zero=True)

ax.legend()

```

    <matplotlib.legend.Legend at 0x1d8500ef220>