# Loss functions

Loss functions are used to train neural networks and to compute the difference between output and target variable. A critical component of training neural networks is the loss function. A loss function is a quantative measure of how bad the predictions of the network are when compared to ground truth labels. Given this score, a network can improve by iteratively updating its weights to minimise this loss. Some tasks use a combination of multiple loss functions, but often you'll just use one. MXNet Gluon provides a number of the most commonly used loss functions, and you'll choose certain loss functions depending on your network and task. Some common task and loss function pairs include:

- regression: [L1Loss](https://beta.mxnet.io/api/gluon/_autogen/mxnet.gluon.loss.L1Loss.html), [L2Loss](https://beta.mxnet.io/api/gluon/_autogen/mxnet.gluon.loss.L2Loss.html)
- classification: [SigmoidBinaryCrossEntropyLoss](https://beta.mxnet.io/api/gluon/_autogen/mxnet.gluon.loss.SigmoidBinaryCrossEntropyLoss.html), [SoftmaxBinaryCrossEntropyLoss](https://beta.mxnet.io/api/gluon/_autogen/mxnet.gluon.loss.SoftmaxBinaryCrossEntropyLoss.html)
- embeddings: [HingeLoss](https://beta.mxnet.io/api/gluon/_autogen/mxnet.gluon.loss.HingeLoss.html)

We'll first import the modules, where the `mxnet.gluon.loss` module is imported as `gloss` to avoid the commonly used name `loss`.

```python
from IPython import display
from matplotlib import pyplot as plt
import mxnet as mx
from mxnet import nd, autograd
from mxnet.gluon import nn, loss as gloss
```

## Basic Usages

Now let's create an instance of the $\ell_2$ loss, which is commonly used in regression tasks.

```python
loss = gloss.L2Loss()
```

And then feed two inputs to compute the elementwise loss values.

```python
x = nd.ones((2,))
y = nd.ones((2,)) * 2
loss(x, y)
```

These values should be equal to the math definition: $0.5\|x-y\|^2$.

```python
.5 * (x - y)**2
```

Next we show how to use a loss function to compute gradients.

```python
X = nd.random.uniform(shape=(2, 4))
net = nn.Dense(1)
net.initialize()
with autograd.record():
    l =  loss(net(X), y)
print(l)
```

We can compute the gradients w.r.t. the loss function.

```python
l.backward()
print(net.weight.grad())
```

## Loss functions

Most commonly used loss functions can be divided into 2 categories: regression and classification.

Let's first visualize several regression losses. We visualize the loss values versus the predicted values with label values fixed to be 0.

```python
def plot(x, y):
    display.set_matplotlib_formats('svg')
    plt.plot(x.asnumpy(), y.asnumpy())
    plt.xlabel('x')
    plt.ylabel('loss')
    plt.show()

def show_regression_loss(loss):
    x = nd.arange(-5, 5, .1)
    y = loss(x, nd.zeros_like(x))
    plot(x, y)

```

Then plot the classification losses with label values fixed to be 1.

```python
def show_classification_loss(loss):
    x = nd.arange(-5, 5, .1)
    y = loss(x, nd.ones_like(x))
    plot(x, y)
```

#### [L1 Loss](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.L1Loss)

L1 Loss, also called Mean Absolute Error, computes the sum of absolute distance between target values and the output of the neural network. It is defined as:

$$ L = \sum_i \vert {label}_i - {pred}_i \vert. $$

It is a non-smooth function that can lead to non-convergence. It creates the same gradient for small and large loss values, which can be problematic for the learning process.

```python
show_regression_loss(gloss.L1Loss())
```

#### [L2 Loss](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.L2Loss)

L2Loss, also called Mean Squared Error, is a regression loss function that computes the squared distances between the target values and the output of the neural network. It is defined as:

$$ L = \frac{1}{2} \sum_i \vert {label}_i - {pred}_i \vert^2. $$

Compared to L1, L2 loss it is a smooth function and it creates larger gradients for large loss values. However due to the squaring it puts high weight on outliers.

```python
show_regression_loss(gloss.L2Loss())
```

#### [Huber Loss](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.HuberLosss)

HuberLoss  combines advantages of L1 and L2 loss. It calculates a smoothed L1 loss that is equal to L1 if the absolute error exceeds a threshold $$\rho$$, otherwise it is equal to L2. It is defined as:
$$
\begin{split}L = \sum_i \begin{cases} \frac{1}{2 {rho}} ({label}_i - {pred}_i)^2 &
                   \text{ if } |{label}_i - {pred}_i| < {rho} \\
                   |{label}_i - {pred}_i| - \frac{{rho}}{2} &
                   \text{ otherwise }
    \end{cases}\end{split}
$$

```python
show_regression_loss(gloss.HuberLoss(rho=1))
```

An example of where Huber Loss is used can be found in [Deep Q Network](https://openai.com/blog/openai-baselines-dqn/).

#### [Cross Entropy Loss with Sigmoid](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.SigmoidBinaryCrossEntropyLoss)

Binary Cross Entropy is a loss function used for binary classification problems e.g. classifying images into 2 classes. Cross entropy measures the difference between two probability distributions and it is defined as:
$$\sum_i -{(y\log(p) + (1 - y)\log(1 - p))} $$
Before the loss is computed a sigmoid activation is applied per default. If your network has `sigmoid` activation as last layer, then you need set ```from_sigmoid``` to False, to avoid applying the sigmoid function twice.

```python
show_classification_loss(gloss.SigmoidBinaryCrossEntropyLoss())
```

#### [Cross Entropy Loss with Softmax](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.SoftmaxCrossEntropyLoss)

In classification, we often apply the
softmax operator to the predicted outputs to obtain prediction probabilities,
and then apply the cross entropy loss against the true labels:

$$ \begin{align}\begin{aligned}p = \softmax({pred})\\L = -\sum_i \sum_j {label}_j \log p_{ij}\end{aligned}\end{align}
$$

Running these two steps one-by-one, however, may lead to numerical instabilities. The `loss` module provides a single operators with softmax and cross entropy fused to avoid such problem.

```python
loss = gloss.SoftmaxCrossEntropyLoss()
x = nd.array([[1, 10], [8, 2]])
y = nd.array([0, 1])
loss(x, y)
```

#### [Hinge Loss](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.HingeLoss)

Commonly used in Support Vector Machines (SVMs), Hinge Loss is used to additionally penalize predictions that are correct but fall within a margin between classes (the region around a decision boundary). Unlike `SoftmaxCrossEntropyLoss`, it's rarely used for neural network training. It is defined as:

$$
L = \sum_i max(0, {margin} - {pred}_i \cdot {label}_i)
$$

```python
show_classification_loss(gloss.HingeLoss())
```

#### [Logistic Loss](https://mxnet.incubator.apache.org/versions/master/api/python/gluon/loss.html#mxnet.gluon.loss.LogisticLoss)

The Logistic Loss function computes the performance of binary classification models.
$$
L = \sum_i \log(1 + \exp(- {pred}_i \cdot {label}_i))
$$
The log loss decreases the closer the prediction is to the actual label. It is sensitive to outliers, because incorrectly classified points are penalized more.

```python
show_classification_loss(gloss.LogisticLoss())
```

#### [Kullback-Leibler Divergence Loss](https://mxnet.incubator.apache.org/versions/master/api/python/gluon/loss.html#mxnet.gluon.loss.KLDivLoss)

The Kullback-Leibler divergence loss measures the divergence between two probability distributions by calculating the difference between cross entropy and entropy. It takes as input the probability of predicted label and the probability of true label.

$$
L = \sum_i {label}_i * \big[\log({label}_i) - {pred}_i\big]
$$

The loss is large, if the predicted probability distribution is far from the ground truth probability distribution. KL divergence is an asymmetric measure. KL divergence loss can be used in Variational Autoencoders (VAEs), and reinforcement learning policy networks such as Trust Region Policy Optimization (TRPO)

For instance, in the following example we get a KL divergence of 0.02. We set ```from_logits=False```, so the loss functions will apply ```log_softmax``` on the network output, before computing the KL divergence.

```python
output = mx.nd.array([[0.39056206, 1.3068528, 0.39056206, -0.30258512]])
print('output.softmax(): {}'.format(output.softmax().asnumpy().tolist()))
target_dist = mx.nd.array([[0.3, 0.4, 0.1, 0.2]])
loss_fn = gloss.KLDivLoss(from_logits=False)
loss = loss_fn(output, target_dist)
print('loss (kl divergence): {}'.format(loss.asnumpy().tolist()))
```

#### [Triplet Loss](https://mxnet.incubator.apache.org/versions/master/api/python/gluon/loss.html#mxnet.gluon.loss.TripletLoss)

Triplet loss takes three input arrays and measures the relative similarity. It takes a positive and negative input and the anchor.

$$
L = \sum_i \max(\Vert {pos_i}_i - {pred} \Vert_2^2 -
    \Vert {neg_i}_i - {pred} \Vert_2^2 + {margin}, 0)
$$

The loss function minimizes the distance between similar inputs and maximizes the distance  between dissimilar ones.
In the case of learning embeddings for images of characters, the network may get as input the following 3 images:

The network would learn to minimize the distance between the two `A`'s and maximize the distance between `A` and `Z`.

#### [CTC Loss](https://mxnet.incubator.apache.org/versions/master/api/python/gluon/loss.html#mxnet.gluon.loss.CTCLoss)

CTC Loss is the [connectionist temporal classification loss](https://distill.pub/2017/ctc/) . It is used to train recurrent neural networks with variable time dimension. It learns the alignment and labelling of input sequences. It takes a sequence as input and gives probabilities for each timestep. For instance, in the following image the word is not well aligned with the 5 timesteps because of the different sizes of characters. CTC Loss finds for each timestep the highest probability e.g. `t1` presents with high probability a `C`. It combines the highest probapilities and returns the best path decoding. For an in-depth tutorial on how to use CTC-Loss in MXNet, check out this [example](https://github.com/apache/incubator-mxnet/tree/master/example/ctc).

#### [Cosine Embedding Loss](https://mxnet.incubator.apache.org/versions/master/api/python/gluon/loss.html#mxnet.gluon.loss.CosineEmbeddingLoss)
The cosine embedding loss computes the cosine distance between two input vectors.

$$
\begin{split}L = \sum_i \begin{cases} 1 - {cos\_sim({input1}_i, {input2}_i)} & \text{ if } {label}_i = 1\\
                 {cos\_sim({input1}_i, {input2}_i)} & \text{ if } {label}_i = -1 \end{cases}\\
cos\_sim(input1, input2) = \frac{{input1}_i.{input2}_i}{||{input1}_i||.||{input2}_i||}\end{split}
$$

Cosine distance measures the similarity between two arrays given a label and is typically used for learning nonlinear embeddings.
For instance, in the following code example we measure the similarity between the input vectors `x` and `y`. Since they are the same the label equals `1`. The loss function returns $$ \sum_i 1 - {cos\_sim({input1}_i, {input2}_i)} $$ which is equal `0`.

```python
x = mx.nd.array([1,0,1,0,1,0])
y = mx.nd.array([1,0,1,0,1,0])
label = mx.nd.array(1)
loss = gloss.CosineEmbeddingLoss()
print(loss(x,y,label))
```

Now let's make `y` the opposite of `x`, so we set the label `-1` and the function will return  $$ \sum_i cos\_sim(input1, input2) $$

```python
x = mx.nd.array([1,0,1,0,1,0])
y = mx.nd.array([0,1,0,1,0,1])
label = mx.nd.array(-1)
loss = gloss.CosineEmbeddingLoss()
print(loss(x,y,label))
```

#### [PoissonNLLLoss](https://mxnet.incubator.apache.org/versions/master/api/python/gluon/loss.html#mxnet.gluon.loss.PoissonNLLLoss)
Poisson distribution is widely used for modelling count data. It is defined as:

$$
f(x) = \frac{\mu ^ {\kern 0.08 em x} e ^ {-\mu}} {x!} \qquad \qquad x = 0,1,2 , \ldots \,.
$$

For instance, the count of cars in road traffic approximately follows a Poisson distribution. Using an ordinary least squares model for Poisson distributed data would not work well because of two reasons:
 - count data cannot be negative
 - variance may not be constant

Instead we can use a Poisson regression model, also known as log-linear model. Thereby the Poisson incident rate $$\mu$$ is
modelled by a linear combination of unknown parameters.
We can then use the PoissonNLLLoss which calculates the negative log likelihood for a target that follows a Poisson distribution.

$$ L = \text{pred} - \text{target} * \log(\text{pred}) +\log(\text{target!}) $$

## Advanced: Weighted Loss

Some examples in a batch may be more important than others. We can apply weights to individual examples during the forward pass of the loss function using the `sample_weight` argument. All examples are weighted equally by default.

```python
x = nd.ones((2,))
y = nd.ones((2,)) * 2
loss = gloss.L2Loss()
loss(x, y, nd.array([1, 2]))
```

## Conclusion

In this tutorial we saw an example of how to evaluate model performance using loss functions (during the forward pass). Crucially, we then saw how calculate parameter gradients (using `backward`) which would minimise this loss. You should now have a better understanding of when to apply different loss functions, especially for regression vs classification tasks.

## Recommended Next Steps

In addition to loss functions, which are used for explicit optimization, you might want to look at metrics that give useful evaluation feedback even if they're not explicitly optimized for in the same way as the loss. You might also want to learn more about the mechanics of the backpropagation stage in the autograd tutorial.


## The SIR model

The SIR model of epidemiology partitions the population into three compartments: susceptibles, S, who can catch the disease; infectives, I, who have already caught the disease and infect susceptibles; and removed individuals, R. Since the disease is assumed not to be fatal, the sum $N=S+I+R$ remains constant. The rate at which the susceptibles get infected is
$$
\lambda(t) = \frac{\beta I}{N}
$$
where the parameter $\beta$ is the probability of infection on contact. The infected individuals remove from the disease at a rate $\gamma$. Then, the ordinary differential equations of the SIR model are

\begin{align}
\dot S &= -\lambda(t)S \\
\dot I &= \lambda(t)I - \gamma I \\
\dot R &= \gamma I
\end{align}

This example integrates the above equations to obtain what is called the **epidemic curve**: a plot of the number of susceptibles and infectives as a function of time.

```python
%matplotlib inline
import numpy as np
import pyross
import matplotlib.pyplot as plt
#from matplotlib import rc; rc('text', usetex=True)
```

```python
M = 1                  # the SIR model has no age structure
Ni = 1000*np.ones(M)   # so there is only one age group
N = np.sum(Ni)         # and the total population is the size of this age group

beta  = 0.2            # infection rate
gIa   = 0.1            # removal rate of asymptomatic infectives
gIs   = 0.1            # removal rate of symptomatic infectives
alpha = 0              # fraction of asymptomatic infectives
fsa   = 1              # Fraction by which symptomatic individuals do not self isolate

Ia0 = np.array([0])     # the SIR model has only one kind of infective
Is0 = np.array([1])     # we take these to be symptomatic
R0  = np.array([0])     # and assume there are no removed individuals initially
S0  = N-(Ia0+Is0+R0)    # so that the initial susceptibles are obtained from S + Ia + Is + R = N

# there is no contact structure
def contactMatrix(t):
    return np.identity(M)

# duration of simulation and data file
Tf = 160;  Nt=160;

# instantiate model
parameters = {'alpha':alpha, 'beta':beta, 'gIa':gIa, 'gIs':gIs,'fsa':fsa}
model = pyross.deterministic.SIR(parameters, M, Ni)

# simulate model
#data = model.simulate(S0, Ia0, Is0, contactMatrix, Tf, Nt)
data = model.simulator(np.concatenate((S0, Ia0, Is0)), contactMatrix, Tf, Nt) # another possibility
```

```python
# plot the data and obtain the epidemic curve
S  = data['X'][:,0].flatten()
Ia = data['X'][:,1].flatten()
Is = data['X'][:,2].flatten()
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
plt.plot(t, S/N, '-', color="#348ABD", label='$S$', lw=4)

plt.fill_between(t, 0, Is/N, color='#A60628', alpha=0.3)
plt.plot(t, Is/N, '-', color='#A60628', label='$I$', lw=4)

R=N-S-Ia-Is; plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
plt.plot(t, R/N, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Fraction of compartment value')
plt.xlabel('Days');
```

## Plot the same epidemic curve using utils method

```python
fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

data = model.simulate(S0, Ia0, Is0, contactMatrix, Tf, Nt)
#data = model.simulator(np.concatenate((S0, Ia0, Is0)), contactMatrix, Tf, Nt) # another possibility
pyross.utils.plotSIR(data)
```

## Plot the effective basic reproductive ratio $\mathcal R_0^{\text{eff}}(t)$ as a function of time

```python
fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')

C=np.identity(M); contactMatrix = pyross.contactMatrix.SIR(0.25*C, 0.25*C, 0.25*C, 0.25*C)
r0de = contactMatrix.basicReproductiveRatio(data, state='dynamic')

plt.fill_between(t, 0, t*0+1, color="dimgrey", alpha=0.2); plt.ylabel('Basic reproductive ratio')
plt.plot(r0de, '*', color='#A60628')
plt.xlabel('Days'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True);
plt.yticks(np.arange(0, 3, step=0.5)); plt.ylim(0,2.2);
```

---

## The SIR model with two age groups

The partitioning of the population can be refined to include other attributes relevant to the disease. One of the most important of these is the age. Let us assume we partition the population into two age groups, children and adults, and label them by the index $i=1,2$. Children can catch the infection from other children or from adults; likewise, adults can catch the infection from other adults or from children. Calling their respective rates of infection $\lambda_1(t)$ and $\lambda_2(t)$ we get
\begin{align}
\lambda_1(t) = \beta(C_{11}\frac{I_1}{N_1} + C_{12}\frac{I_2}{N_2})S_1\\
\lambda_2(t) = \beta(C_{21}\frac{I_1}{N_1} + C_{22}\frac{I_2}{N_2})S_2
\end{align}
where $C_{ij}$ are contact matrices, quantifying how much each age group interacts with the other. The ordinary differential equations of this age-structured SIR model are

\begin{align}
\dot S_i &= -\lambda_i(t)S_i \\
\dot I_i &= \lambda(t)_iI_i - \gamma I_i \\
\dot R_i &= \gamma I_i
\end{align}

Again, for each $i$ the sum $N_i = S_i + I_i + R_i$ remains constant. What do we expect qualitatively ? The group that has a greater rate will catch the disease faster and catch more of it. This depends on how the entries of the contact matrix are distributed.

This example integrates the above equations to **epidemic curve** for both the children and the adults. We see that they have unequal rates of infection.

```python
M  = 2                  # the population has two age groups
N  =  1000000           # and this is the total population

beta  = 0.0131          # infection rate
gIa   = 1./7            # removal rate of asymptomatic infectives
gIs   = 1./7            # removal rate of asymptomatic infectives
alpha = 0               # fraction of asymptomatic infectives
fsa   = 1               # Fraction by which symptomatic individuals do not self isolate

Ni = np.zeros((M))      # population in each group
fi = np.zeros((M))      # fraction of population in age age group

# set the age structure
fi = np.array((0.25, 0.75))
for i in range(M):
    Ni[i] = fi[i]*N

# set the contact structure
C = np.array(([18., 9.], [3., 12.]))

Ia_0 = np.array((1,1))  # each age group has asymptomatic infectives
Is_0 = np.array((1,1))  # and also symptomatic infectives
R_0  = np.array((0,0))  # there are no removed individuals initially
S_0  = Ni - (Ia_0 + Is_0 + R_0)

# matrix for linearised dynamics
L = np.zeros((M, M))
for i in range(M):
    for j in range(M):
        L[i,j]=C[i,j]*Ni[i]/Ni[j]

L = (alpha*beta/gIs)*L

# the basic reproductive ratio
r0 = np.max(np.linalg.eigvals(L))

print("The basic reproductive ratio for these parameters is", r0)
```

    The basic reproductive ratio for these parameters is 0.0

```python
# duration of simulation and data file
Tf=200; Nf=2000; filename='this.mat'

# the contact structure is independent of time
def contactMatrix(t):
    return C

# instantiate model
parameters = {'alpha':alpha, 'beta':beta, 'gIa':gIa, 'gIs':gIs,'fsa':fsa}
model = pyross.deterministic.SIR(parameters, M, Ni)

# simulate model
data=model.simulate(S_0, Ia_0, Is_0, contactMatrix, Tf, Nf)
```

```python
IK = data.get('X')[:,2*M].flatten()
IA = data.get('X')[:,2*M+1].flatten()
t  = data.get('t')
```

```python
fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, IK/Ni[0], color="#348ABD", alpha=0.3)
plt.plot(t, IK/Ni[0], '-', color="#348ABD", label='$Children$', lw=4)

plt.fill_between(t, 0, IA/Ni[1], color='#A60628', alpha=0.3)
plt.plot(t, IA/Ni[1], '-', color='#A60628', label='$Adults$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
```


```python
import modern_robotics as mr
import sympy as sp
import numpy as np
from sympy.physics.mechanics import dynamicsymbols, mechanics_printing
mechanics_printing()
from Utilities.symbolicFunctions import*
from Utilities.kukaKinematics import*
from Utilities.RobotClass import*
```

    Jupyter environment detected. Enabling Open3D WebVisualizer.
    [Open3D INFO] WebRTC GUI backend enabled.
    [Open3D INFO] WebRTCWindowSystem: HTTP handshake server disabled.
    [Open3D INFO] Resetting default logger to print to terminal.

# TASK 2

### 1. Using the original Denavit-Hartenberg convention, determine the Denavit-Hartenberg parameters for the Agilus robot.

```python
th1, th2, th3, th4, th5, th6 = dynamicsymbols('theta_1, theta_2, theta_3, theta_4, theta_5, theta_6')
al1,al2,al3,al4,al5,al6,al7 = sp.symbols('alpha_1,alpha_2,alpha_3,alpha_4,alpha_5,alpha_6,alpha_7,')

sicConfig = sp.Matrix([[ 25, sp.pi/2 , -400,   th1],
                    [455, 0       ,    0 ,th2],
                    [ 35, sp.pi/2 ,    0,th3 -sp.pi/2],
                    [  0, -sp.pi/2, -420, th4],
                    [  0, sp.pi/2 ,    0,th5],
                    [  0, 0       ,    0,th6]])
sicConfig
```

#### Here, the first matrix represent the DH parameters in the same formate as in Modern Robotics. The second matrix represent the same table is illustrated as illustrated in Sicilliano. Theta are the free variables of each joint, where we add an additional pi/2 to the third joint to get a get the desired pose illustrated in the report.

### 2. Determine the end-effector zero position configuration M ∈ SE(3) for the Agilus robot

```python
config1 = sicConfig.subs({th1:0,th2:0,th3:0,th4:0,th5:0,th6:0})
T0i = T_from_sic(config1)             #List of T0i's, T01, T02, T03,...,T06
T0i.insert(0,sp.eye(4))               #insert T00 for visualization purpose
T0i.pop()

Tsi = [rotX(sp.pi)*T for T in T0i ]   #List of Tsi's, T1, T2,...,T6
```

```python
M_DH = Tsi[5] * Tne # Tsi[5] = Ts6 = Ts1*T16

M_DH, Me
```

####  Control: The M derived by visual inspection in Utilities/KukaKinematics and the Slist derived from the DH-formulation are in agreement.
#### To compansate for the additional rotation of pi/2 in the table, we rotate the third joint with -pi/2 to get the zero-state configuration. As M represent the transformation matrix from the {s} frame to the end effector, and since the table from task one only represent the parameters of each joint, we add an additional T from the n'th link to the end-effector.

### 3. Determine the space frame screw axes Si for the Agilus robot.

```python
sp.Matrix(Slist)
```

#### The Slist derived by visual inspection in Utilities/KukaKinematics

### 4. Determine the body-frame screw axes Bi for the Agilus robot

```python
Mb = M6*Tnb                     #Mb = Tsb for robot in zero-config
M_inv = mr.TransInv(Mb)         #Finding inverse of M
Ad_M_inv = mr.Adjoint(M_inv)    #Computing [Ad_M^-1]

#Using B_i = [Ad_M^-1]S_i
Blist2 = sp.zeros(6,6)
for i in range(6):
     Blist2[:, i] = Ad_M_inv @ Slist[:, i]

Blist, Blist2
```

####  Control: The Blist derived by visual inspection in Utilities/KukaKinematics and calculated Blist are in agreement.

### 5. Visualization
#### Visualization using custom class in Utilities module. The big coordinate system being the {s}-frame and joint-frames oriented with z-axis along joint screw-axis. Joint 6 is coloured red
#### Axis colours:
####    -Blue   -Z axis and joint screw-axis
####    -Red    -X axis
####    -Green  -Y axis

### Visualization of DH frames:

```python
#Finding M01, M02..., from List of Tsi developed in task 2.2
Mlist_DH = np.array(Tsi,dtype=float)

#Visualize frames
KukaDH = Robot(Mlist_DH, ['z','-z','x','y','-z','z','z'],Tne)
KukaDH.joints[5].set_colour([1,0,0])
KukaDH.draw_robot(method=2)
```

####

#### DH-frames shown as joint-frames. Note that all three wrist joints are co-located with {b}. {e} is translated 80mm along x-axis of {b} to show orientation clearly.

```python
KukaDH.transform(Slist,[0,0,np.pi/2,0,0,0])
KukaDH.draw_robot(method=2)
```

#### Robot shown in the config that was used for the DH-analysis

### Visualization of Kuka robot in zero-config. Showing Me, Si and Bi:
#### Me is the chains last frame (without grey joint), screw axes as the joint frames' z-axis (both Bi and Si).
#### {4} is drawn outside of the wrist to show functionality better.
#### {b} is located inside the wrist(red/grey) and oriented as {e}

```python
Kuka = Robot(Mlist, ['z', '-z', 'x', 'x', '-z','x'], Tne)
Kuka.joints[5].set_colour([1,0,0])
Kuka.joints[4].set_colour([0.2,0.2,0.2])
Kuka.draw_robot(method=2)
```

```python
#Oppgave 2.6

joint_config1 = [0,0,0,0,0,0]
joint_config2 = [0,0,-np.pi/2,np.pi/2,0,0]

M_poe = np.array([[0,0,-1,900],
                [0,1,0,0],
                [1,0,0,435],
                [0,0,0,1]])

T_poe = mr.FKinSpace(M_poe, Slist, joint_config2)
T_poe = T_poe * Tne

print(T_poe)

```

    Matrix([[1.11022302462516e-16, 1.00000000000000, -1.11022302462516e-16, 445.000000000000], [0, 1.11022302462516e-16, 1.00000000000000, 0], [1.00000000000000, -1.11022302462516e-16, 1.23259516440783e-32, 900.000000000000], [0, 0, 0, 1.00000000000000]])

```python
config2 = sicConfig.subs({th1:0,th2:0,th3:-np.pi/2,th4:np.pi/2,th5:0,th6:0}) #DH-parameters with given thetas
T0i = T_from_sic(config2)               #List of T0i's, T01, T02, T03,...,T06
Tsi = [rotX(sp.pi)*T for T in T0i ]     #List of Tsi's, T1, T2,...,T6
Tse = Tsi[5] * Tne
print(Tse)
```

    Matrix([[6.12323399573677e-17, 1.00000000000000, -6.12323399573677e-17, 445.000000000000], [0, 6.12323399573677e-17, 1.00000000000000, 0], [1.00000000000000, -6.12323399573677e-17, 3.74939945665464e-33, 900.000000000000], [0, 0, 0, 1]])

```python
def PoeDHCompare(thetaList):
    config2 = sicConfig.subs({th1:thetaList[0],th2:thetaList[1],th3:thetaList[2],th4:thetaList[3],th5:thetaList[4],th6:thetaList[5]}) #DH-parameters with given thetas
    T0i = T_from_sic(config2)               #List of T0i's, T01, T02, T03,...,T06
    Tsi = [rotX(sp.pi)*T for T in T0i ]     #List of Tsi's, T1, T2,...,T6
    Tse = Tsi[5] * Tne

    T_poe = mr.FKinSpace(M_poe, Slist, thetaList) #PoE
    T_poe = T_poe * Tne #From joint 5 to end-effector

    Tse = np.array(Tse).astype(np.float64) #Convert from Matrix to Numpy Array
    T_poe = np.array(T_poe).astype(np.float64)
    res = np.allclose(Tse, T_poe, 0.0001) #Compares the arrays with a given resoulution

    np.set_printoptions(precision=3)
    np.set_printoptions(suppress=True)

    #print("Theta List: ",thetaList)
    print("T_DH: \n", Tse)
    print("T_poe: \n", T_poe)
    return res

print(PoeDHCompare([0,0,0,0,0,0]))
```

    T_DH:
     [[  1.   0.   0. 980.]
     [  0.   1.   0.   0.]
     [  0.   0.   1. 435.]
     [  0.   0.   0.   1.]]
    T_poe:
     [[  1.   0.   0. 980.]
     [  0.   1.   0.   0.]
     [  0.   0.   1. 435.]
     [  0.   0.   0.   1.]]
    True

```python
PoeDHCompare([0,0,-np.pi/2,np.pi/2,0,0])
```

    T_DH:
     [[  0.   1.  -0. 445.]
     [  0.   0.   1.   0.]
     [  1.  -0.   0. 900.]
     [  0.   0.   0.   1.]]
    T_poe:
     [[  0.   1.  -0. 445.]
     [  0.   0.   1.   0.]
     [  1.  -0.   0. 900.]
     [  0.   0.   0.   1.]]

    True

```python
PoeDHCompare([1,1,1,1,1,1])
```

    T_DH:
     [[  -0.941    0.041    0.337   -6.159]
     [   0.155    0.936    0.317  -95.248]
     [  -0.302    0.35    -0.887 -403.507]
     [   0.       0.       0.       1.   ]]
    T_poe:
     [[  -0.941   -0.261    0.217   -6.159]
     [   0.155    0.239    0.959  -95.248]
     [  -0.302    0.935   -0.184 -403.507]
     [   0.       0.       0.       1.   ]]

    False

```python
Slist
```

    array([[   0.,    0.,    0.,   -1.,    0.,   -1.],
           [   0.,    1.,    1.,    0.,    1.,    0.],
           [  -1.,    0.,    0.,    0.,    0.,    0.],
           [   0., -400., -400.,    0., -435.,    0.],
           [   0.,    0.,    0., -435.,    0., -435.],
           [   0.,   25.,  480.,    0.,  900.,    0.]])

```python
Blist
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & -1 & 0 & -1\\0 & 1 & 1 & 0 & 1 & 0\\-1 & 0 & 0 & 0 & 0 & 0\\0 & 35 & 35 & 0 & 0 & 0\\-900 & 0 & 0 & 0 & 0 & 0\\0 & -875 & -420 & 0 & 0 & 0\end{matrix}\right]$

```python
Ts_poe = (exp6(Slist[:,0], th1) * exp6(Slist[:,1], th2) * exp6(Slist[:,2], th3) * exp6(Slist[:,3], th4) * exp6(Slist[:,4], th5) * exp6(Slist[:,5], th6)) * M_poe

Tb_poe = M_poe * (exp6(Blist[:,0], th1) * exp6(Blist[:,1], th2) * exp6(Blist[:,2], th3) * exp6(Blist[:,3], th4) * exp6(Blist[:,4], th5) * exp6(Blist[:,5], th6))

Ts_poe.simplify()
Tb_poe.simplify()
```

```python
Ts_poe
```

$\displaystyle \left[\begin{matrix}1.0 \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{cos}\left(\theta_{6}\right) - 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right) - \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{sin}\left(\theta_{6}\right) & - 1.0 \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{sin}\left(\theta_{6}\right) - 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right) - \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{6}\right) & 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right)\right) \operatorname{sin}\left(\theta_{5}\right) - 1.0 \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{5}\right) & \left(35.0 \operatorname{sin}\left(\theta_{2} + \theta_{3}\right) + 420.0 \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) + 455.0 \operatorname{cos}\left(\theta_{2}\right) + 25.0\right) \operatorname{cos}\left(\theta_{1}\right)\\- 1.0 \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) - \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right)\right) \operatorname{cos}\left(\theta_{6}\right) + 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right) + \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{sin}\left(\theta_{6}\right) & 1.0 \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) - \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right)\right) \operatorname{sin}\left(\theta_{6}\right) + 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right) + \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{6}\right) & - 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) - \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{sin}\left(\theta_{5}\right) + 1.0 \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{5}\right) & - \left(35.0 \operatorname{sin}\left(\theta_{2} + \theta_{3}\right) + 420.0 \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) + 455.0 \operatorname{cos}\left(\theta_{2}\right) + 25.0\right) \operatorname{sin}\left(\theta_{1}\right)\\- 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{5}\right) - \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{5}\right)\right) \operatorname{cos}\left(\theta_{6}\right) - 1.0 \operatorname{sin}\left(\theta_{4}\right) \operatorname{sin}\left(\theta_{6}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) & 1.0 \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{5}\right) - \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{5}\right)\right) \operatorname{sin}\left(\theta_{6}\right) - 1.0 \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{6}\right) & 1.0 \operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{5}\right) + 1.0 \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{4}\right) & - 420.0 \operatorname{sin}\left(\theta_{2} + \theta_{3}\right) - 455.0 \operatorname{sin}\left(\theta_{2}\right) + 35.0 \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) + 400.0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
Tb_poe
```

$\displaystyle \left[\begin{matrix}\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{4}\right) & - \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{5}\right) - \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{5}\right)\right) \operatorname{sin}\left(\theta_{6}\right) + \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{6}\right) & \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{5}\right) - \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{5}\right)\right) \operatorname{cos}\left(\theta_{6}\right) + \operatorname{sin}\left(\theta_{4}\right) \operatorname{sin}\left(\theta_{6}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) & 420 \operatorname{sin}\left(\theta_{2} + \theta_{3}\right) + 455 \operatorname{sin}\left(\theta_{2}\right) - 35 \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) + 935\\\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) - \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{sin}\left(\theta_{5}\right) - \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{5}\right) & \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) - \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right)\right) \operatorname{sin}\left(\theta_{6}\right) + \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right) + \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{6}\right) & - \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) - \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right)\right) \operatorname{cos}\left(\theta_{6}\right) + \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right) + \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{sin}\left(\theta_{6}\right) & - 5 \left(7 \operatorname{sin}\left(\theta_{2} + \theta_{3}\right) + 84 \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) + 91 \operatorname{cos}\left(\theta_{2}\right) + 5\right) \operatorname{sin}\left(\theta_{1}\right)\\- \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right)\right) \operatorname{sin}\left(\theta_{5}\right) + \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{5}\right) & - \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{sin}\left(\theta_{6}\right) - \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right) - \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{6}\right) & \left(\left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right) + \operatorname{sin}\left(\theta_{1}\right) \operatorname{sin}\left(\theta_{4}\right)\right) \operatorname{cos}\left(\theta_{5}\right) + \operatorname{sin}\left(\theta_{5}\right) \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right)\right) \operatorname{cos}\left(\theta_{6}\right) - \left(\operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{sin}\left(\theta_{4}\right) \operatorname{cos}\left(\theta_{1}\right) - \operatorname{sin}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{4}\right)\right) \operatorname{sin}\left(\theta_{6}\right) & 35 \operatorname{sin}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) + 420 \operatorname{cos}\left(\theta_{2} + \theta_{3}\right) \operatorname{cos}\left(\theta_{1}\right) + 455 \operatorname{cos}\left(\theta_{1}\right) \operatorname{cos}\left(\theta_{2}\right) + 25 \operatorname{cos}\left(\theta_{1}\right) - 465\\0 & 0 & 0 & 1\end{matrix}\right]$


# Lecture 4.3: Logistic Regression

This lecture, we are going to use logistic regression to predict the quality of wines.

**Learning goals:**

- solve a classification problem
- fit a logistic regression model
- predict using a logistic regression model
- visualize the decision boundary of a logistic regression model

## 1. Introduction

Humans have been making wine for more than [8000 years](https://www.nationalgeographic.com/travel/destinations/asia/georgia/sponsor-content-secret-birthplace-of-wine/),  and that's a lot of bottles. 🍷 There's too many options! Let's use data science to help choose the _tastiest_ red wines. Or at least, to classify high rated wines vs low rated wines based on some chemical features.

## 2. Data Munging

Let's load the dataset into a `DataFrame`:

```python
import pandas as pd

df = pd.read_csv('wine_quality.csv')
df.head()
```

```python
df.describe()
```

Each row of our dataset is a wine.

The `tasty` column indicates if a wine is "tasty" or not. i.e if it has a high rating. Even if it is represented by an integer, it is a _categorical_ column, as values are either 0 or 1. This will be the _label_ we are aiming to _predict_ in this _classification_ task.

The other columns are numerical features. Their means are close to 0, which suggests that this dataset was normalized, like for lecture 3.5.

To keep things simple and easy to visualize, let's start by using only two features to predict the "tastiness" of these wines: `sulphates` and `alcohol`.

```python
X = df[['sulphates', 'alcohol']]
y = df['tasty']
X.shape
```

Since our features are already a _matrix_, we do not need to convert our `X` and `y` to numpy arrays and reshape them. Which means we're ready for training!

## 3. Training

We'll use sklearn's [`LogisticRegression`](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression) model which, as always, will make things simple with the `.fit()` and `.predict()` methods.

```python
from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(random_state=0).fit(X, y)
```

🧠 Can you recall all the optimization steps that sklearn takes care of in the `.fit()` method? Refer to lectures 4.1 & 4.2 if this isn't clear.

Just like for linear regression, we can investigate the _parameters_ of our model, $\theta$.
They won't be _interpreted_ in the same way however, since our hypothesis uses the logistic function. Nevertheless, sklearn still calls these parameters `intercept_` and `coef_`.

```python
import numpy as np

theta = np.append(clf.intercept_, clf.coef_)
theta
```

This means that the optimization procedure found that the logistic hypothesis which best predits the `tasty` labels is:

$$\begin{align}
h_{\theta}(x_{1}, x_{2}) &= \frac{1}{1 + e^{-(\theta_{0} + \theta_{1} x_{1} + \theta_{2} x_{2})}} \\
h_{\theta}(x_{1}, x_{2}) &= \frac{1}{1 + e^{-(0.244 + 0.451 x_{1} + 1.099 x_{2})}}
\end{align}$$

where $x_{1}$ is our first feature, the `sulphates` column, and $x_{2}$ is our second feature, the `alcohol` column.

This is a function with two variables that we can plot in 3D:

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import seaborn as sns
from mpl_toolkits import mplot3d
import math

sns.set()

def hypothesis(x1, x2):
    linear_term = theta[0] + theta[1] * x1 + theta[2] * x2
    return 1 / (1 + math.exp(-linear_term))

v_hypothesis = np.vectorize(hypothesis)

def create_mesh(theta):
    x_values = np.linspace(-8, 8, 40)
    y_values = np.linspace(-8, 8, 40)
    x_mesh, y_mesh = np.meshgrid(x_values, y_values)
    z_mesh = v_hypothesis(x_mesh, y_mesh)
    return x_mesh, y_mesh, z_mesh

x_mesh, y_mesh, z_mesh = create_mesh(theta)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.plot_wireframe(x_mesh, y_mesh, z_mesh, rstride=5, cstride=5, alpha=0.4, color='g')

ax.set_title('Logistic Regression Model of Wine Tastiness')
ax.set_xlabel('Sulphates (norm)')
ax.set_ylabel('Alcohol (norm)')
ax.set_zlabel('Probability of Being Tasty');

```

🧠 Is this the shape you expected? Can you relate this to the 1D plots of the logistic function we saw in the slides?

We could add our data points to this 3D graph. However, things would get crowded, and _encoding_ a binary label on the z-axis of a 3D plot isn't very clear. We end up with some points near the "roof" of the volume, and others near the "floor", without a good way of comparing them.

Instead, we can use _color_ to encode the labels of the data points. We'll visualize the wines with the `tasty=1` label in red, and the non `tasty` wines in blue. In short, we are using a _2D graph_, where:
- on the x-axis, the `sulphates` feature
- on the y-axis, the `alcohol` feature
- as color, the `tasty` binary label

```python
%matplotlib inline

colors = sns.color_palette('husl').as_hex()

fig = plt.figure(dpi=120)
ax = fig.add_subplot(111)

# plot points
positives = X[y == 1]
negatives = X[y == 0]
ax.scatter(positives.iloc[:, 0], positives.iloc[:, 1], alpha=0.4, s=8,c=colors[0], label='tasty')
ax.scatter(negatives.iloc[:, 0], negatives.iloc[:, 1], alpha=0.4, s=8, c=colors[4], label='not tasty')

# formatting
ax.set_ylim((-2, 4))
ax.set_xlim((-2, 4))
ax.set_xlabel('Sulphates (norm)')
ax.set_ylabel('Alcohol (norm)')
ax.legend()
ax.set_title('Wine Tastiness vs Sulphates & Alcohol Contents');

```

😨 This data is messy! This is a good example of what _real world data_ looks like. No clean clusters, no clean cut classes. How easy it is to split the dataset into its categorical labels is often called the _separability_ of the data. A dataset that is _separable_ has clear optimal decision boundaries that would lead to few prediction errors. In this case, it's not even clear to a human where to draw the line between `tasty` and not `tasty`!

Let's look at our logistic regression model's attempt at separating the labels... maybe it has tasted more bottles than us 😏. Here, the prediction probabilities we visualized in the 3D graph isn't really what we wish to see. We would like to compare the _binary predictions_ against the binary labels.

For this, we have to show the _decision boundary_ of our model. But all we have are the model parameters, $\theta$ ! How do we go from the shape of the _hypothesis_ to the shape of the _decision boundary_ ? 🤨

## 4. Decision Boundary

Remember we defined how to go from prediction probability to binary prediction in the slides. We choose a threshold such that:

if $h_{\theta}(\textbf{x}) >= 0.5$ :
    _predict 1_
else:
    _predict 0_

The decision boundary is the set of points where predictions "switch" from 0 to 1. This means we can represent the it as all the points where:

$$
h_{\theta}(x_{1}, x_{2}) = 0.5 \label{eq:threshold} \tag{1}
$$

However, this doesn't help us much with plotting it in matplotlib... Ideally, we expect a line equation of the form:

$$
y = b + ax
$$

And since we're using a 2D plot where our feature `sulphates` ($x_{1}$) is on the x-axis, and our feature `alcohol` ($x_{2}$) is on the y-axis, we can rewrite the above as:

$$
x_{2} = b + ax_{1}
$$

So this is our goal. Now, let's use a bit of mathematical trickery to get there. 🧙♀
We can substitute the logistic hypothesis in equation $\eqref{eq:threshold}$:

$$
\begin{align}
\frac{1}{1 + e^{-(\theta_{0} + \theta_{1} x_{1} + \theta_{2} x_{2})}} &= 0.5 \label{eq:logistic} \tag{2} \\
\end{align}
$$

You might recall from the [plot](https://upload.wikimedia.org/wikipedia/commons/8/88/Logistic-curve.svg) of the logistic function that it is equal to $0.5$ when its input is $0$.
So we can rewrite equation $\eqref{eq:logistic}$ as:

$$
\begin{align}
\theta_{0} + \theta_{1} x_{1} + \theta_{2} x_{2} &= 0 \\
\end{align}
$$

Finally, we can re-arrange the terms to get the line equation:

$$
\begin{align}
\theta_{0} + \theta_{1} x_{1} + \theta_{2} x_{2} &= 0 \\
x_{2} &= \frac{-(\theta_{0} + \theta_{1} x_{1})}{\theta_{2}} \\
x_{2} &= \frac{-\theta_{0}}{\theta_{2}} + \frac{ \theta_{1}}{\theta_{2}}x_{1} \\
\end{align}
$$

This is the equation of a _line_ with intercept $\frac{-\theta_{0}}{\theta_{2}}$ and slope $\frac{ \theta_{1}}{\theta_{2}}$. We can plot this in our 2D graph! 🎊

We'll add our logistic regression model's decision boundary as a green line:

```python
%matplotlib inline

colors = sns.color_palette('husl').as_hex()

fig = plt.figure(dpi=120)
ax = fig.add_subplot(111)

# plot points
positives = X[y == 1]
negatives = X[y == 0]
ax.scatter(positives.iloc[:, 0], positives.iloc[:, 1], alpha=0.4, s=8,c=colors[0], label='tasty')
ax.scatter(negatives.iloc[:, 0], negatives.iloc[:, 1], alpha=0.4, s=8, c=colors[4], label='not tasty')

def decision_boundary(x1, theta):
    return -(x1 * theta[1] + theta[0])/theta[2]

# plot decision boundary
x1 = np.array(ax.get_xlim())
x2 = decision_boundary(x1, theta)
ax.plot(x1, x2, '--', c=colors[2]);

# formatting
ax.set_ylim((-2, 4))
ax.set_xlim((-2, 4))
ax.set_xlabel('Sulphates (norm)')
ax.set_ylabel('Alcohol (norm)')
ax.legend()
ax.set_title('Wine Tastiness vs Sulphates & Alcohol Contents');

```

The green line is all the points where $h_{\theta}(x_{1}, x_{2}) = 0.5$. The points _above_ the line are the model's _positive_ predictions, i.e it thinks those wines are `tasty`, whilst the points _below_ the line are the model's _negative_ predictions, i.e it thinks those wines are not so `tasty`. Take you time to understand this graph, as it is typical of classification tasks.

ℹ Notice that the predictions don't always match the true _label_ of the data. For example, take the blue point at $(-1.75, 2.4)$. The blue color shows that the label is 0, i.e a _not_ `tasty` wine. However, it is located _above_ the decision boundary, so the model think it is `tasty`. It's a classification error! Based on the _separability_ of the dataset, we can understand that the model makes a few mistakes... the goal of supervised learning algorithms isn't to perfectly predict a variable, but to find the model parameters that lead to the best _possible_ predictions.

🧠 Can you match this 2D graph with the 3D representation of the prediction probabilities of section 3? Where would the decision boundary be on the 3D graph?

## 5. Prediction

We are happy with our model, and want to try it out as soon as possible. During a visit to the supermarket, we find a strange looking wine bottle. It piques our curiosity, so we read the label: `sulphates=1.942` and `alcohol=-0.343` (in normalized units). Let's figure out if our model thinks this is a tasty bottle!

🧠 Can you graphically read the model prediction for this strange wine using the graph above?

We know by now that sklearn's `.predict()` method expects a matrix. Since we have only one example with two features, $x_{1}$ and $x_{2}$, it will be automatically instantiated as a 2D vector. Instead, let's reshape it to a ($1 \times 2$) `ndarray` matrix:

```python
sulphates = 1.942
alcohol = -0.343
x_strange_wine = np.asarray([sulphates, alcohol]).reshape(1, 2)
```

We can then predict the wine's tastiness using the logistic regression model that we trained earlier:

```python
y_strange_wine = clf.predict(x_strange_wine)
y_strange_wine
```

The prediction is positive, $1$, meaning our model thinks that this wine is `tasty`. Hopefully this means it doesn't taste as strange as it looks!

We can compare this bottle to the rest of our dataset, by plotting it on the 2D graph from section 4:

```python
%matplotlib inline

colors = sns.color_palette('husl').as_hex()

fig = plt.figure(dpi=120)
ax = fig.add_subplot(111)

# plot points
positives = X[y == 1]
negatives = X[y == 0]
ax.scatter(positives.iloc[:, 0], positives.iloc[:, 1], alpha=0.4, s=8,c=colors[0], label='tasty')
ax.scatter(negatives.iloc[:, 0], negatives.iloc[:, 1], alpha=0.4, s=8, c=colors[4], label='not tasty')

def decision_boundary(x1, theta):
    return -(x1 * theta[1] + theta[0])/theta[2]

# plot decision boundary
x1 = np.array(ax.get_xlim())
x2 = decision_boundary(x1, theta)
ax.plot(x1, x2, '--', c=colors[2]);

# plot strange wine
ax.plot(x_strange_wine[:, 0], x_strange_wine[:, 1], marker='x', markersize=20, markeredgewidth=4, color='r')

# formatting
ax.set_ylim((-2, 4))
ax.set_xlim((-2, 4))
ax.set_xlabel('Sulphates (norm)')
ax.set_ylabel('Alcohol (norm)')
ax.legend()
ax.set_title('Wine Tastiness vs Sulphates & Alcohol Contents');

```

We can indeed see that this strange looking wine is placed _above_ the decision boundary, which is why the `.predict()` method returned a value of $1$. 🙌

You might recall that we had many more features available in our dataset `df`. Going beyond two features means that we won't be able to plot the data or the decision boundary in a meaningful manner. However, the predictions are likely to be more accurate!

💪💪 Train a new logistic regression model named `full_clf` using _all_ the features available in our dataset `df`. Then predict the tastiness of the `mystery_wine` 🕵♀, and store it in a variable called `y_mystery`.
Pro-tip: don't use the `tasty` column in your feature matrix X! That should be reserved as label vector, y.
Pro-tip 2: please use the `random_state=0` argument when constructing your sklearn class, to make sure your results are reproducible. Else, the unit test won't pass!
Pro-tip 3: If you also don't like typing out a big list of column names, you might want to consider [`.drop()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html)

```python
mystery_wine = np.array([-0.12, 0.82, 1.43, -0.29, 0.03, -0.90, -0.55, 2.19, -0.62, 0.43, 1.85]).reshape(1, -1)

# INSERT CODE

def test_mystery_wine_prediction():
    assert y_mystery == 1, 'The mystery wine is actually tasty! But your prediction says otherwise 😢'
    prob_mystery = full_clf.predict_proba(mystery_wine)[:, 1]
    assert math.isclose(prob_mystery, 0.7615, rel_tol=1e-3), f'Your prediction probability for the mystery wine is wrong'
    print('Success! 🎉')

test_mystery_wine_prediction()

```

## 6. Summary

Today, we learned about our first classification method, **logistic regression**. We restated that classification tasks aim to predict **categorical** labels, and visualized this for one feature in a two dimensional plot. Linear models were a bad fit binary labels, so we discovered the **logistic function** that could predict **probabilities** between 0 & 1. We described how **thresholds** can turn these probabilities into a binary prediction. The logistic function can be **parameterized** by feeding a **linear model** as its argument, like a set of russian dolls. This means that we can **optimize** our logistic model using the **cross-entropy** cost function and **gradient descent**. We explained what **decision boundaries** are, and how important they are for **classifiers**. We also extended our logistic regression to the **multi-class** setting, where it could predict **multiple labels** with a **softmax** hypothesis. Finally, we put this knowledge to practice by predicting if wines were **tasty**, and visualizing the **decision boundary** of our trained **logistic regression** model.

# Resources
### Core Resources

- [Coursera Machine Learning - Logistic Regression](https://www.coursera.org/lecture/machine-learning/classification-wlPeP)
Andrew Ng's always excellent course goes into detail for this section on Logistic Regression.
- [Kaggle dataset - red wine quality](https://www.kaggle.com/uciml/red-wine-quality-cortez-et-al-2009)
The dataset used in this notebook.

### Additional Resources

- [ML cheatsheet cost functions](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html)
A handy and mathematical summary of common cost functions, including MSE and cross-entropy.
- [Cross-entropy loss a visual explanation](https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a)
A clear visual yet formal explanation of the cross-entropy cost function.
- [Binary vs multi-class logistic regression](https://chrisyeh96.github.io/2018/06/11/logistic-regression.html#multinomial-logistic-regression-via-cross-entropy)
Comparison of logistic vs softmax functions for logistic regression models.
- [Understanding the softmax function in minutes](https://medium.com/data-science-bootcamp/understand-the-softmax-function-in-minutes-f3a59641e86d)
Comprehensive blogpost defining the softmax function and its uses in Machine Learning models.


- - - -
# Mechpy Tutorials
a mechanical engineering toolbox

source code - https://github.com/nagordon/mechpy
documentation - https://nagordon.github.io/mechpy/web/

- - - -

Neal Gordon
2017-02-20

- - - -

## Finite Element Method
- - - -
This a collection of notes, examples and boilerplate code to understand fem.

[Finite Element Method](#Finite-Element-Method)
* [solids FEM example](#FEM-Example-1)

- - - -

## Python Initilaization with module imports

```python
# setup
import numpy as np
import sympy as sp
import scipy
from pprint import pprint
sp.init_printing(use_latex='mathjax')

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (12, 8)  # (width, height)
plt.rcParams['font.size'] = 14
plt.rcParams['legend.fontsize'] = 16
from matplotlib import patches

get_ipython().magic('matplotlib')  # seperate window
get_ipython().magic('matplotlib inline') # inline plotting
```

    Using matplotlib backend: Qt4Agg

```python
import os ; os.chdir('..')  # change to root from the examples folder
from mechpy.fem import cst_fem
```

## Finite-Element-Method
[index](#Mechpy)

The element connectivty is used to assemble the global stiffness matrix, the nodal force matrix, and the displacement matrix

The minimization of the potentail energy is used to solve the global equation once the boundary conditions are applied to prevent rigid body motion

$ \{F\} = [K]\{U\} $

where

$ \{F\}=nodal\ force\ matrix $
$ [K] = global\ stiffness\ matrix $
$ \{U\} = nodal\ displacement\ matrix $

Once the displacements, U are computed, the strain, $\bar{\varepsilon}$ is calcualted

with $\{\varepsilon\}=[B]\{U\}$

where

$[B]=strain-displacement\ matrix$

and stresses, $\bar{\sigma}$ are determined via Hookes Law and

$\{\sigma\}=[C]\{\varepsilon\}$

where

$[C] = compliance\ matrix$

### FEM-Example-1

```python
cst_fem(structure='9node')
```