# Empoderando o Python

# Empoderando o Python

* Importando bibliotecas
* Espaços de nomes
* NumPy
* Matplolib

# Importando bibliotecas

## Sintaxe

`import BIBLIOTECA`

`import BIBLIOTECA as BIB`

`from BIBLIOTECA import FUNCAO1, FUNCAO2, ...`

Após importar os comandos ficam imediatamente disponíveis. É fortemente recomendado que os comandos de importação fiquem no início do arquivo. Quando importando de uma biblioteca utilizando o `import` as funções disponíveis ficam associadas a um espaço de nomes, que é uma forma de evitar conflito de nomes e facilmente identificar a origem do objeto.

## Exemplos de importação

```python
import numpy
```

```python
numpy.sqrt(4)
```

    2.0

```python
import numpy as np
```

```python
np.sqrt(4)
```

    2.0

```python
sqrt(4)
```

```python
from cmath import sqrt, sin, cos
```

```python
sqrt(4)
```

    (2+0j)

# A biblioteca NumPy

* Parte do projeto SciPy;
* Implementa uma estrutura númerica n-dimensional muito poderosa;
* Computacionalmente muito eficiente;
* Possui uma vasta coleção de funções e métodos;
* Funções de álgebra linear (`numpy.linalg`)
* É provavelmente o motivo pelo qual o Python atingiu sua fama.

Inicialmente lançada como `numeric` em 1995, a biblioteca [NumPy](https://www.numpy.org) renomeada em 2005 é provavelmente a responsável pela fama que a linguagem Python possui. É a biblioteca fundamental na computação científica com Python, sendo parte do projeto [SciPy](https://www.scipy.org/). Em sua essência implementa o objeto **numpy.array** que é uma estrutura n-dimensional com suporte a operações vetoriais, matriciais e tensoriais, computacionalmente eficiente em termos de performance e precisão.

Apesar do Python ser considerada uma lingagem lenta, quando o numpy.array é utilizado de forma correta o speedup obtido é na casa de 3 a 4 ordens de magnitude, muitas vezes ultrapassando a performance de linguagens famosas por computação numérica eficiente como Fortran e Júlia. Isso é possível pois internamente toda computação é realizada de forma extremamente otimizada, quando possível utilizando instruções otimizadas do hardware disponível. Mostraremos exemplos comparando a performance entre uma implementação regular em Python e uma implementada utilizando corretamente o NumPy onde o ganho de performance chega a ser 8000x maior, mas apesar deste exemplo ilustrativo, nosso objetivo com esta aula não é otimização de performance.

A biblioteca NumPy possui uma [vasta coleção de funções](https://docs.scipy.org/doc/numpy/reference/routines.html), aqui trataremos somente de uma parte bem pequena. Dentro da NumPy existem os submódulos: `FFT` (transformada de Fourier), `linalg` (Algebra Linear), `matlib` (operações com matrizes) e `random` (números pseudo aleatórios).

## Operações com Numpy Array

### Criando Numpy Arrays

```python
M = np.array([1, 2, 3, 4, 5])
```

```python
N_lista = [1, 2, 3, 4, 5, 6, 7]
N = np.array(N_lista)
```

```python
I = np.ones(10)
I
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

```python
Z = np.zeros_like(I)
Z
```

    array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

```python
M.dtype
```

    dtype('int64')

```python
I.dtype
```

    dtype('float64')

Como os *numpy arrays*, são otimizados para performance, mudar o seu tamanho não é uma tarefa simples como acontece com as listas Python. Por esta razão esta operação não será explicada durante este curso. Sempre crie os arrays numpy com o tamanho correto para a aplicação.

Os elementos de um array numpy devem pertencer todos ao mesmo tipo de objeto (`int64`, `float64`, `object`, etc), quando adicionado tipos diferentes a um array, ele será convertido para o tipo do array (por exemplo adicionar um número  com casas decimais a um array do tipo `int64` converterá o número para o tipo inteiro. Este tipo de operação deve ser evitada a todo custo. Os tipos de dados válidos para um numpy array podem ser vistos em [https://docs.scipy.org/doc/numpy/reference/arrays.scalars.html]

Caso a biblioteca não consiga identificar o melhor tipo para um array ele irá configurar o tipo `object` que tem o comportamento semelhante as listas python (ou seja todos os tipos podem ser inseridos), mas que possui a pior performance, evite a todo custo.

## Aritmética com Numpy Arrays

```python
u = np.array([2,1,2,0])
v = np.array([1,1,2,3])
```

```python
u + v
```

    array([3, 2, 4, 3])

```python
u - v
```

    array([ 1,  0,  0, -3])

```python
u * v
```

    array([2, 1, 4, 0])

```python
u / v
```

    array([2., 1., 1., 0.])

```python
abs(u-v)
```

    array([1, 0, 0, 3])

```python
u @ v
```

    7

Arrays numpy se comportam na maioria dos casos como vetores matemáticos desta forma as operações ($+$, $-$, $*$, $/$) são aplicadas elemento a elemento. Isso exige que os vetores possuam as mesmas dimensões.

Além disso os arrays numpy também suportam a indexação e o fatiamento similar as listas.

## Exercicio 1

A diferença progressiva de primeira ordem de uma série, é dada pela diferença entre o elemento seguinte da série e o elemento atual. Se consideramos um elementos unitariamente espaçados de uma série, podemos considerar a diferença progressiva como um análogo discreto ao conceito de derivada.

Seja a série $x = \{ x_i \} \; 1 \le i \le \infty$. A $i$-ésima diferença progressiva de primeira ordem é dada por

\begin{equation}
\Delta x_i = x_{i+1} - x_i
\end{equation}

Utilizando o que aprendeu até o momento, crie um algoritmo para calcular a diferença progressiva dos 30 primeiros numeros de Fibonacci armazenados na variável `fibo`

```python
# Armazena os 30 primeiros Fibonacci na lista fibo
n = 30
a, b, i = 0, 1, 2
fibo = [a, b]
while i < n :
    i = i + 1
    a, b = b, a+b
    fibo.append(b)
```

```python
# Sua resposta aqui
```

```python
# Gabarito 1
# %load .resps/difprog.py
```

```python
# Gabarito 2
# %load .resps/difprog_o.py
```

# Gráficos

* Biblioteca `matplotlib`

* Imita o comportamento do Matlab

* Suporte a orientação a objetos

* Suporte a $\LaTeX$

* [Multiplos formatos de gráficos](https://matplotlib.org/gallery/index.html)

* Vamos ver:

  * Função

  * Imagens e Matrizes

  * Espalhamento

  * Barra

  * Setores (Pizza/Torta)

  * Histogramas

## Iniciando a MatplotLib

```python
import matplotlib.pyplot as plt
```

```python
x = np.arange(5)
```

```python
plt.plot(x, x**2);
```

## Definindo o domínio

`np.linspace(a, b, n)`

Onde `a` é o limite inferior, `b` o limite superior e `n` o número de intervalos entre os limites.

```python
x = np.linspace(-4, 3, 101)
```

```python
a, b, c = 1, 1, -2
```

```python
y = a*x**2 + b*x + c
```

```python
plt.plot(x, y);
plt.plot(x, .5*x+3);
```

## Exercício

Converta os primeiros 30 números de Fibonacci em um numpy array e crie um gráfico contendo tanto os numeros Fibonacci como as suas diferenças progressivas. Não se esqueça que quando gerando gráficos com mais de uma função é necessário que o número de pontos no domínio seja o mesmo em todas as funções.

```python
# Sua resposta aqui
```

```python
# Gabarito
# %load .resps/fibo_graf.py
```

# Tente

`import antigravity`

```python

```


# PC lab 10: Artificial Neural Networks
---

## Introduction

Although artificial neural networks enjoy a lot of progress today, they were first described by [Warren McCulloch and Walter Pitts](https://link.springer.com/article/10.1007%2FBF02478259) in 1943. Early progress in training competitive neural networks was stalled by a multitude of reasons, such as the limited computer resources, sub-optimal network architectures and the use of smaller datasets. In this PC-lab we will implement a custom neural network on a step-by-step basis, allowing an in-depth comprehension of the essential elements of deep learning

### Artificial Neuron

The core unit of every (artificial) neural network is considered the neuron. The neuron can be observed as a switch. It receives **one or more inputs** $\mathbf{x}$, processes a **weighted sum** $z$ (adding **bias** $b$) that is sent through the **sigmoid activation function $\sigma()$**, outputing a **single response**  $a$:

$$
z = \sum\limits_{i=1}^{n}(w_ix_i) + b = \sum\limits_{i=0}^{n}(w_ix_i)$$ with $$ x_0 = 1 \tag{1}\\
$$

$$ a = \sigma(z) $$

The default recommended activation function is the **Rectified Linear Unit**, or **ReLU**.

$$ ReLU(z) = max\{0,z\} $$

The ReLU function has many properties that make optimization easy using gradient-based methods. It can be seen as a switch giving no response for $z < 0$ and giving a response $z$ for $z > 0$.

### Artificial Neural Networks

The most basic artificial neural network is the **feedforward neural network**. There are no feedback connections such as can be found in **recurrent neural networks**. A feedforward neural network is called a network as it is composed out of many inheriting functions making up the model, e.g. $f(\textbf{x})= f^{(3)}(f^{(2)}(f^{(1)}(\textbf{x})))$. Neural networks typically are constructed in different layers of neurons in which every neuron is connected with all the neurons of the previous layer, eventually resulting in a set of **output neurons** $\mathbf{\hat{\textbf{y}}}$.

To train the network, samples are processed in batches. This allows for faster training and improved convergence of the loss during gradient descent. Advantages of stochastic gradient descent or other optimization algorithms for loss calculation are not discussed in this PC-lab, but have been [extensively discussed](https://ruder.io/optimizing-gradient-descent/) before.
Practically, the first fully-connected layer of the network using batch size $B$ is computed by matrix combination of the input $X \in \mathbb{R}^{B, D}$ with a set of weights $W^{(12)} \in \mathbb{R}^{D, M}$.

\begin{equation}
XW^{(12)} =
\begin{bmatrix}
1 & x_{0,1} & ...  & x_{0,D-1} & x_{0,D} \\
1 & x_{1,1} & ... & x_{1,D-1} & x_{1,D} \\
... & ... & ... & ... & ...\\
1 & x_{B-1,1} & ... & x_{B-1,D-1} & x_{B-1,D} \\
1 & x_{B,1} & ...  & x_{B,D-1} & x_{B,D} \\
\end{bmatrix}
\begin{bmatrix}
W_{0,0} & W_{0,1} & ...  & W_{0,M-1} & W_{0,M} \\
W_{1,0} & W_{1,1} & ... & W_{1,M-1} & W_{1,M} \\
... & ... & ... & ... & ...\\
W_{D-1,0} & W_{D-1,1} & ... & W_{D-1,M-1} & W_{D-1,M} \\
W_{D,0} & W_{D,1} & ...  & W_{D,M-1} & W_{D,M} \\
\end{bmatrix}
\end{equation}

Structure of the exercise

The idea of this PC-Lab will be to construct our very own neural network from scratch. Every exercize will introduce a new feature necessary to train a working model.

We define the class Neural_Network for which the structural architecture is given at initialization. Let's start by creating a neural network with one hidden layer. To include the bias term, we add a scalar to the weights vector for every layer.

    class Neural_Network(object):
    def __init__(self, input_nodes, output_nodes, hlayer_nodes):
        self.input_nodes = input_nodes
        self.output_nodes = output_nodes
        self.hlayer_nodes = hlayer_nodes
        # initialize weights + bias in the first layer
        self.W_12 = np.random.randn(self.input_nodes+1, self.hlayer_nodes))
        # initialize weights + bias in the second layer
        self.W_23 = append(np.random.randn(self.hlayer_nodes+1, self.output_nodes))

EXERCISE

Complete the attribute function Neural_Net.forward() and Neural_Net.relu()

|Code Symbol | Math Symbol | Definition | Dimensions
| :-: | :-: | :-: | :-: |
|X|$$X$$|Input Data| (batch_size, input_nodes) |
|y|$$y$$|label| (batch_size, output_nodes) |
|y_hat| $$\hat{y}$$| Output Data | (batch_size, output_nodes) |
|W_12 | $$W^{(12)}$$ | Layer 1 weights | (input_nodes + 1, hlayer_nodes) |
|W_23 | $$W^{(23)}$$ | Layer 2 weights | (hlayer_nodes + 1, output_nodes) |
|a_1 | $$a^{(1)}$$ | Layer 1 activity (inputs) | (batch_size, input_nodes + 1) |
|z_2 | $$z^{(2)}$$ | Layer 2 linear combination | (batch_size, hlayer_nodes) |
|a_2 | $$a^{(2)}$$ | Layer 2 activity | (batch_size, hlayer_nodes + 1) |
| y_hat | $\hat{y}$ | Layer 3 linear combination | (batch_size, output_nodes) |

$$
\textbf{z}^{(2)} = \textbf{a}^{(1)}\textbf{W}^{(12)}  \tag{2}\\
$$
$$
\textbf{a}^{(2)} = ReLU(\textbf{z}^{(2)}) \tag{3}\\
$$
$$
\hat{\textbf{y}} = \textbf{a}^{(2)}\textbf{W}^{(23)} \tag{4}\\
$$

```python
import numpy as np
```

```python
class Neural_Network(object):
    def __init__(self, input_nodes, output_nodes, hlayer_nodes):
        self.input_nodes = input_nodes
        self.output_nodes = output_nodes
        self.hlayer_nodes = hlayer_nodes

        # initialize weights + bias in the first layer
        self.W_12 = np.random.randn(self.input_nodes+1, self.hlayer_nodes)
        # initialize weights + bias in the second layer
        self.W_23 = np.random.randn(self.hlayer_nodes+1, self.output_nodes)

    def forward(self, X):
        # add bias to input samples
        self.a_1 = np.hstack((np.ones((len(X),1)), X)) # add one acting as a bias to all samples

        #//solution
        self.z_2 = np.matmul(self.a_1, self.W_12)
        self.a_2 = np.hstack((np.ones((len(X),1)), self.relu(self.z_2)))
        self.y_hat = np.matmul(self.a_2, self.W_23)
        #//solution
        return np.clip(self.y_hat, -10e6, 10e6)

    def relu(self, z):
        #//solution
        return  np.maximum(0., z)
        #//solution

    def relu_prime(self, z):

        return (z>0)*1
```

```python
np.random.seed(1)
# Initialize neural network with 1 input, 1 output and 2 nodes in hidden layer
NN = Neural_Network(1,1,3)
# Input two samples, each with one input
NN.forward([[2],[3]])
```

    array([[2.10183529],
           [2.37793415]])

## Backpropagation

Let's construct a dataset that follows the function $\mathbf{y = sin(x)}$. To train the neural network we have to evaluate how the prediction $\hat{y}$ compares to the true label $y$. This is done through the loss function $L$. The formula used for L is dependent upon the problem we are trying to solve. In line with previous methods fitting a **regression** problem, we consider the **MSE** :

$$
L = \frac{1}{n}\sum_{i=1}^{n}{(y_i-\hat{y_i})^2}\tag{6}
$$

```python
import matplotlib.pyplot as plt
%matplotlib inline

def squared_error(y, y_hat):
    return (y-y_hat)**2

X = np.random.uniform(-3, 3, size=(1000,1)) # sample random points for x=[0, pi]
y = -26*X + 145*X**2 + 28*X**3 - 26*X**4 - 2*X**5 + X**6 - 80
#y = np.sum(np.sin(X),axis=1).reshape(-1,1) # sinus transform
NN = Neural_Network(1,1,40)
y_hat = NN.forward(X) # predict values
loss = squared_error(y, y_hat)

fig, (ax_1, ax_2, ax_3) = plt.subplots(1,3, figsize=(15,4))
ax_1.set_title("sample data")
ax_1.scatter(X, y)
ax_2.set_title("predictions on sample data")
ax_2.scatter(X, y_hat)
ax_3.set_title("loss function at each sample")
ax_3.scatter(X, loss)
```

To train our model, we must find a set of weights that minimizes the overall loss function. An exhaustive method would be to try all possible combinations of weights, a solution that blows up with increasing dimensionality. In order to reduce processing times to obtain the optimal set of weights, we evaluate the influence of each weight on the loss function. If we consider the different equations to obtain our loss:

$$
\textbf{z}^{(2)} = \textbf{a}^{(1)}\textbf{W}^{(12)} \tag{2}\\
$$
$$
\textbf{a}^{(2)} = ReLU(\textbf{z}^{(2)}) \tag{3}\\
$$
$$
\hat{\textbf{y}} = \textbf{a}^{(2)}\textbf{W}^{(23)} \tag{4}\\
$$
$$
L = \frac{1}{n} \sum_{i=0}^{n}{(y_{i}-\hat{y_{i}})^2} \tag{6}\\
$$

To evaluate the influence of $W^{(23)}$ on the loss $L$, we can find the **partial derivative** of the loss in function of $W^{(23)}$. Using the **chain rule**, this can be calculated using a step-by-step approach.
$$ \frac{\delta L}{\delta W^{(23)}} = \frac{\delta \textbf{L}}{\delta \hat{\textbf{y}}} \cdot \frac{\delta \hat{\textbf{y}}}{\delta \textbf{W}^{(23)}} = \delta_{1} \cdot \frac{\delta \hat{\textbf{y}}}{\delta \textbf{W}^{(23)}}  \tag{7}$$

Backpropagation is the step-by-step evaluation of the partial derivatives of the loss in function of the network weights, performed by backward iteration over the different transformations performed by the network. The values obtained when iterating down from the loss function are recyclable when calculating the derivative of weights further down in the chain rule, making **a substantial difference** in the amount of processing power needed to obtain all derivatives. The use of backpropagation is a major element in making training of deep neural networks possible. For example, to obtain the derivatives of the weights in the first layer ($\textbf{W}^{(12)}$) we apply:

$$ \frac{\delta L}{\delta W^{(12)}} = \delta_{1} \cdot \frac{\delta \hat{\textbf{y}}}{\delta \textbf{a}^{(2)}} \cdot \frac{\delta \textbf{a}^{(2)}}{\delta \textbf{z}^{(2)}} \cdot \frac{\delta \textbf{z}^{(2)}}{\delta \textbf{W}^{(12)}} = \delta_{2} \cdot \frac{\delta \textbf{z}^{(2)}}{\delta \textbf{W}^{(12)}} \tag{8} $$

$\delta_{1}$ and $\delta_{2}$ are solely introduced as variables to which the intermediate values can be assigned. Using these variables when implementing backpropagation into the neural network will make things easier.

EXERCISE:

Write out the following partial derivatives as found in the chain rule.

$$\frac{\delta L}{\delta \hat{y}}$$

$$\frac{\delta \hat{y}}{\delta a^{(2)}}$$

$$\frac{\delta a^{(2)}}{\delta z^{(2)}}$$

$$\frac{\delta z^{(2)}}{\delta W^{(12)}}$$

EXERCISE:

Write out the following partial derivatives as found in the chain rule.

$$\frac{\delta L}{\delta \hat{y}} = \frac{1}{n}\sum_{i=0}^{n}{2*(\hat{y_{i}} - y_{i})}$$

$$\frac{\delta \hat{y}}{\delta a^{(2)}} = W^{(23)}$$

$$\frac{\delta a^{(2)}}{\delta z^{(2)}} =
\begin{cases}
0 & \text{if  }  (z^{(2)}) < 0 \\
1 & \text{if  }  (z^{(2)}) > 0 \\
\end{cases}$$

$$\frac{\delta z^{(2)}}{\delta W^{(12)}} = a^{(1)}$$

EXERCISE

Complete the attribute functions NN.backpropagate() and NN.relu_prime(), and the function loss_prime. Use the partial derivatives of the previous exercise to get an overview of the sequential steps executed in backpropagation.

```python
class Neural_Network(object):
    def __init__(self, input_nodes, output_nodes, hlayer_nodes):
        self.input_nodes = input_nodes
        self.output_nodes = output_nodes
        self.hlayer_nodes = hlayer_nodes

        # initialize weights + bias in the first layer
        self.W_12 = np.random.randn(self.input_nodes+1, self.hlayer_nodes)
        # initialize weights + bias in the second layer
        self.W_23 = np.random.randn(self.hlayer_nodes+1, self.output_nodes)

    def forward(self, X):
        # add bias to input samples
        self.a_1 = np.hstack((np.ones((len(X),1)), X)) # add one acting as a bias to all samples

        #//solution
        self.z_2 = np.matmul(self.a_1, self.W_12)
        self.a_2 = np.hstack((np.ones((len(X),1)), self.relu(self.z_2)))
        self.y_hat = np.matmul(self.a_2, self.W_23)
        #//solution
        return np.clip(self.y_hat, -10e6, 10e6) # stability

    def relu(self, z):
        #//solution
        return  np.maximum(0., z)
        #//solution

    def relu_prime(self, z):

        return (z>0)*1

    def backpropagate(self, X, y, get_loss_prime):
        self.y_hat = self.forward(X)
        self.delta_1 = get_loss_prime(self.y_hat,y) # derivative of cost function

        self.dLdW_23 = np.matmul(self.delta_1.T, self.a_2)
        self.dLda_2 = np.matmul(self.delta_1, self.W_23[1:].T)
        self.delta_2 = self.dLda_2*self.relu_prime(self.z_2)
        self.dLdW_12 = np.matmul(self.a_1.T, self.delta_2)

        # join two flattened arrays in one vector
        return np.concatenate((self.dLdW_12.ravel(), self.dLdW_23.ravel()))
```

```python
NN.a_2.shape
```

    (2, 4)

```python

```

```python
from pdb import set_trace
```

```python
NN.delta_1
```

    225.88805523890605

```python
NN.a_2
```

    array([[1.        , 4.66693506, 0.        , 5.99824271],
           [1.        , 0.52690763, 0.27338689, 0.        ]])

```python
def get_loss_prime(y_hat, y):
    return 2*(y_hat-y) # Cost function over all batches, divided by # batches

np.random.seed(1)
NN = Neural_Network(1,1,3)
NN.backpropagate(X[:2], y[:2], get_loss_prime)
```

    array([-343.89509296,  -28.12439719, -134.64243737, 1234.09013299,
            -28.76577578,  381.8021183 ,  451.77611048, 2473.36747371,
            -24.09999765, 3238.62855083])

EXERCISE:

Evaluate the values and shapes of dLdW_12 and dLdW_23 separately, and compare the size of vectors. Can you differentiate the updates for weights from the updates for the biases? Can you predict the shapes of these values after changing the amount of nodes of the different layers within the neural network? Don't forget to run NN.backpropagation() with updated arguments when changing the hyperparameters of the network!

```python
NN = Neural_Network(1,1,2)
NN.backpropagate([[1],[2]], [[np.sin(1)],[np.sin(2)]], get_loss_prime)
print("dLdW_12:\n {}".format(NN.dLdW_12))
print("dLdW_23:\n {}".format(NN.dLdW_23))

```

    dLdW_12:
     [[ 0.         -2.89823566]
     [ 0.         -4.44545308]]
    dLdW_23:
     [[-4.97282087  0.         -4.32282731]]

## Gradient Descent

Our simple neural network is close to completion now. One more important element is the ability to train this network. To adjust the weights based on the partial derivatives we need to apply the obtained gradient to perform an update to the actual weights. As we are dealing with a **non-convex** and **non-linear optimization problem**, updates are performed using the well-known **gradient descent** algorithm. Small variations on the gradient descent method have shown to drastically increase the speed at which deep neural networks can be trained (i.e. [momentum, adam, ...](http://ruder.io/optimizing-gradient-descent/index.html)).

### A brief recap: Gradient Descent

$$W_{t+1} = W_{t} - \eta \nabla L(\theta)$$

where

$\eta$: learning rate (default: 1e-4)

>**input** starting point $\textbf{x}\in$ **dom** $f$.
>
>**repeat**
>
>>    1. $\Delta \textbf{x} := -\nabla f(\textbf{x})$.
>>    3. *Update*. $\textbf{x}:=\textbf{x}+\eta\Delta \textbf{x}$.
>
>**until** stopping criterion is satisfied.

>**output** $x$

The stopping criterion is usually expressed as the amount of times the dataset is iterated. An **epoch** is one iteration over the whole dataset.

```python
from sklearn.utils import shuffle
```

```python
class Neural_Network(object):
    def __init__(self, input_nodes, output_nodes, hlayer_nodes):
        self.input_nodes = input_nodes
        self.output_nodes = output_nodes
        self.hlayer_nodes = hlayer_nodes

        # initialize weights + bias in the first layer
        self.W_12 = np.random.randn(self.input_nodes+1, self.hlayer_nodes)
        # initialize weights + bias in the second layer
        self.W_23 = np.random.randn(self.hlayer_nodes+1, self.output_nodes)

    def forward(self, X):
        # add bias to input samples
        self.a_1 = np.hstack((np.ones((len(X),1)), X)) # add one acting as a bias to all samples

        #//solution
        self.z_2 = np.matmul(self.a_1, self.W_12)
        self.a_2 = np.hstack((np.ones((len(X),1)), self.relu(self.z_2)))
        self.y_hat = np.matmul(self.a_2, self.W_23)

        #//solution
        return np.clip(self.y_hat, -10e6, 10e6)

    def relu(self, z):
        #//solution
        return  np.maximum(0., z)
        #//solution

    def relu_prime(self, z):

        return (z>0)*1

    def backpropagate(self, X, y, get_loss_prime):
        self.y_hat = self.forward(X)
        self.delta_1 = get_loss_prime(self.y_hat,y) # derivative of cost function

        self.dLdW_23 = np.matmul(self.delta_1.T, self.a_2)
        self.dLda_2 = np.matmul(self.delta_1, self.W_23[1:].T)
        self.delta_2 = self.dLda_2*self.relu_prime(self.z_2)
        self.dLdW_12 = np.matmul(self.a_1.T, self.delta_2)

        # join two flattened arrays in one vector
        return np.concatenate((self.dLdW_12.ravel(), self.dLdW_23.ravel()))

    def set_weights(self, weights):
        self.W_12 = weights[:len(self.W_12.ravel())].reshape(self.W_12.shape)
        self.W_23 = weights[len(self.W_12.ravel()):].reshape(self.W_23.shape)

    def get_weights(self):
        return np.concatenate((self.W_12.ravel(), self.W_23.ravel()))

    def compile(self, optimizer_f, loss_f, loss_prime_f):
        self.optimizer =  optimizer_f
        self.get_loss = loss_f
        self.get_loss_prime = loss_prime_f

    def fit(self, X, y, batch_size=8, epochs=50, eta=1e-5):
        epoch = 0   # set starting epoch
        pool = np.arange(len(X)//batch_size)   # setup batch pool
        avg_loss_all = []   # initialize loss vector
        while epoch<epochs:
            print("\repoch: {}".format(epoch), end="")
            epoch +=1
            # shuffle X, y
            X_scrambled, y_scrambled = shuffle(X,y)
            avg_loss = 0
            for i in shuffle(pool):   # select random batch
                # select batch data
                X_batch = X_scrambled[i*batch_size:(i+1)*batch_size]
                y_batch = y_scrambled[i*batch_size:(i+1)*batch_size]
                # get prediction
                y_hat = self.forward(X_batch)
                # get loss on prediction
                avg_loss += self.get_loss(y_hat, y_batch)
                # get gradients on weights
                dLdW = self.backpropagate(X_batch, y_batch, self.get_loss_prime)
                # get update values with weight gradients
                dLdW_update = self.optimizer(dLdW, eta)
                self.set_weights(self.get_weights()+dLdW_update)   # set new weights with update values
            avg_loss_all.append(avg_loss)

        return avg_loss_all
```

EXERCISE

Complete the function gradient_descent(). The different steps as explained above have already been partially implemented in the Neural_Network.fit() function, make sure to analyze it before completing the code for the gradient_descent function. gradient_descent is one way to optimize your weights given a loss, and is stored in Neural_Network.optimizer when running Neural_Network.compile()

```python
def MSE_loss(y_hat, y):
    return np.mean((y-y_hat)**2)/(len(y))

def MSE_loss_prime(y_hat, y):
    return np.minimum(2*(y_hat-y), 10e6)

def gradient_descent(dLdW, eta):
    return dLdW*-eta
```

EXERCISE

Run the code below and evaluate how the network trains when tweaking with specific functions such as the amount of hidden nodes, epochs and step size of the gradient update.

```python
np.random.seed(1)
NN = Neural_Network(1,1,160)
NN.compile(gradient_descent, MSE_loss, MSE_loss_prime)

fig, ax = plt.subplots(2,2, figsize=(10,10))
ax_1, ax_2, ax_3, ax_4 = ax.ravel()
ax_1.scatter(X , NN.forward(X))
ax_1.set_title("Predicted y values BEFORE GD")
loss = NN.fit(X, y, epochs=200, batch_size=5, eta=1e-5)
ax_2.scatter(X, NN.forward(X))
ax_2.set_title("Predicted y values AFTER GD")
ax_3.scatter(X,y)
ax_3.set_title("True y values")
ax_4.set_title("Loss function throughout training")
ax_4.plot(range(len(loss)),loss, 'r')

```

```python
np.random.seed(1)
NN = Neural_Network(1,1,320)
NN.compile(gradient_descent, MSE_loss, MSE_loss_prime)

fig, ax = plt.subplots(2,2, figsize=(10,10))
ax_1, ax_2, ax_3, ax_4 = ax.ravel()
ax_1.scatter(X , NN.forward(X))
ax_1.set_title("Predicted y values BEFORE GD")
loss = NN.fit(X, y, epochs=200, batch_size=5, eta=1e-5)
ax_2.scatter(X, NN.forward(X))
ax_2.set_title("Predicted y values AFTER GD")
ax_3.scatter(X,y)
ax_3.set_title("True y values")
ax_4.set_title("Loss function throughout training")
ax_4.plot(range(len(loss)),loss, 'r')

```

Optional EXERCISE

Tweak the network to allow for classification purposes. For this, a sigmoid layer has to be added as the final step in
    NN.forward(). Make sure to adjust the backpropagate step and the loss function accordingly.

```python

```


## Model Predictive Control Example

Consider the following model predictive control [1], [2] example. To determine the optimal control input $u \in \mathbb{R}^m$ applied to the linear time-invariant (LTI) system $\left( A, B\right) \in \left( \mathbb{R}^{n \times n}, \mathbb{R}^{n \times m} \right)$ with state $x \in \mathbb{R}^n$, we solve the optimization problem

\begin{equation}
\begin{array}{ll}
\text{minimize} \quad & x_H^T P x_H + \sum_{i=0}^{H-1} x_i^T Q x_i + u_i^T R u_i\\
\text{subject to} \quad &x_{i+1} = A x_i + B u_i \quad \forall i \in \{0, ..., H-1\} \\
& \Vert u_i \Vert_\infty \leq 1 \quad \forall i \in \{0, ..., H-1\} \\
& x_0 = x_\mathrm{init}, \\
\end{array}
\end{equation}

where $x_i \in \mathbb{R}^n$ and $u_i \in \mathbb{R}^m$ are the variables. The prediction horizon is $H$, the control input is constrained within a box of size $1$ and the cost matrices are positive definite: $P, Q, R \in \mathbb{S}_{++}^n, \mathbb{S}_{++}^n, \mathbb{S}_{++}^m$. Usually, $P$ is chosen as the solution to the discrete-time algebraic Riccati equation for the given LTI system and cost matrices $Q, R$. The initial measured state is $x_\mathrm{init}$. We arrange the state and input variables to matrices $X$ and $U$ with $X_{:,i} = x_i$ and $U_{:,i} = u_i$, respectively, and reformulate the problem to be [DPP-compliant](https://www.cvxpy.org/tutorial/advanced/index.html#disciplined-parametrized-programming), i.e.,

\begin{equation}
\begin{array}{ll}
\text{minimize} \quad & \Vert P^{1/2} X_{:,H} \Vert_2^2 + \Vert Q^{1/2} X_{:,0:H-1} \Vert_F^2 + \Vert R^{1/2} U \Vert_F^2\\
\text{subject to} \quad &X_{:,1:H} = A X_{:,0:H-1} + B U \\
& | U | \leq \mathbb{1} \\
& X_{:,0} = x_\mathrm{init}, \\
\end{array}
\end{equation}

with variables $X \in \mathbb{R}^{n \times H+1}$ and $U \in \mathbb{R}^{m \times H}$. Note that $|U|$ denotes the element-wise absolute value of $U$.

Let's define the corresponding CVXPY problem.

```python
import cvxpy as cp

# define dimensions
H, n, m = 10, 6, 3

# define variables
U = cp.Variable((m, H), name='U')
X = cp.Variable((n, H+1), name='X')

# define parameters
Psqrt = cp.Parameter((n, n), name='Psqrt')
Qsqrt = cp.Parameter((n, n), name='Qsqrt')
Rsqrt = cp.Parameter((m, m), name='Rsqrt')
A = cp.Parameter((n, n), name='A')
B = cp.Parameter((n, m), name='B')
x_init = cp.Parameter(n, name='x_init')

# define objective
objective = cp.Minimize(cp.sum_squares(Psqrt@X[:,H-1]) + cp.sum_squares(Qsqrt@X[:,:H]) + cp.sum_squares(Rsqrt@U))

# define constraints
constraints = [X[:,1:] == A@X[:,:H]+B@U,
               cp.abs(U) <= 1,
               X[:,0] == x_init]

# define problem
problem = cp.Problem(objective, constraints)
```

Assign parameter values and solve the problem. In this case, the state $x = \left[p^T v^T\right]^T$ consists of position $p \in \mathbb{R}^3$ and velocity $v \in \mathbb{R}^3$ of some rigid body in three-dimensional space. The control input $u$ is the force vector that acts on the body's center of mass and its rotational dynamics is not considered. The discretization step is denoted by $t_d \in \mathbb{R}_{++}$.

```python
import numpy as np

# continuous-time dynmaics
A_cont = np.concatenate((np.array([[0, 0, 0, 1, 0, 0],
                                   [0, 0, 0, 0, 1, 0],
                                   [0, 0, 0, 0, 0, 1]]),
                         np.zeros((3,6))), axis=0)
mass = 1
B_cont = np.concatenate((np.zeros((3,3)),
                         (1/mass)*np.diag(np.ones(3))), axis=0)

# discrete-time dynamics
td = 0.1
A.value = np.eye(n)+td*A_cont
B.value = td*B_cont

# cost
Psqrt.value = np.eye(n)
Qsqrt.value = np.eye(n)
Rsqrt.value = np.sqrt(0.1)*np.eye(m)

# measurement
x_init.value = np.array([2, 2, 2, -1, -1, 1])

val = problem.solve()
```

Generating C source for the problem is as easy as:

```python
from cvxpygen import cpg

cpg.generate_code(problem, code_dir='MPC_code')
```

Now, you can use a python wrapper around the generated code as a custom CVXPY solve method.

```python
from MPC_code.cpg_solver import cpg_solve
import numpy as np
import pickle
import time

# load the serialized problem formulation
with open('MPC_code/problem.pickle', 'rb') as f:
    prob = pickle.load(f)

# assign parameter values
prob.param_dict['A'].value = np.eye(n)+td*A_cont
prob.param_dict['B'].value = td*B_cont
prob.param_dict['Psqrt'].value = np.eye(n)
prob.param_dict['Qsqrt'].value = np.eye(n)
prob.param_dict['Rsqrt'].value = np.sqrt(0.1)*np.eye(m)
prob.param_dict['x_init'].value = np.array([2, 2, 2, -1, -1, 1])

# solve problem conventionally
t0 = time.time()
# CVXPY chooses eps_abs=eps_rel=1e-5, max_iter=10000, polish=True by default,
# however, we choose the OSQP default values here, as they are used for code generation as well
val = prob.solve(eps_abs=1e-3, eps_rel=1e-3, max_iter=4000, polish=False)
t1 = time.time()
print('\nCVXPY\nSolve time: %.3f ms' % (1000 * (t1 - t0)))
print('Objective function value: %.6f\n' % val)

# solve problem with C code via python wrapper
prob.register_solve('CPG', cpg_solve)
t0 = time.time()
val = prob.solve(method='CPG')
t1 = time.time()
print('\nCVXPYgen\nSolve time: %.3f ms' % (1000 * (t1 - t0)))
print('Objective function value: %.6f\n' % val)
```

\[1\] Wang, Yang, and Stephen Boyd. "Fast model predictive control using online optimization." IEEE Transactions on control systems technology 18.2 (2009): 267-278.

\[2\] Hovgaard, T. G., Larsen, L. F., Jørgensen, J. B., and Boyd, S.  "MPC for wind power gradients—utilizing forecasts, rotor inertia, and central energy storage." 2013 European Control Conference (ECC). IEEE, 2013.


##  Построение 3Д модели Ленты Мебиусы

### Подключенине необходимых библиотек

```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
```

### Производим вычисления для построеня графика

Использованная формула:

\begin{equation}
x = \bigg(1 + \frac{v}{2}\cos\Big(\frac{u}{2}\Big)\bigg)\cos(u) \\
y = \bigg(1 + \frac{v}{2}\cos\Big(\frac{u}{2}\Big)\bigg)\sin(u) \\
z = \frac{v}{2}\sin\Big(\frac{u}{2}\Big) \\
u \in [0; 2\pi], \quad
v \in [{-1}; 1]
\end{equation}

```python
u, v = np.mgrid[0:2*np.pi:50j, -1:1:50j]
x = (1 + (v / 2) * np.cos(u / 2)) * np.cos(u)
y = (1 + (v / 2) * np.cos(u / 2)) * np.sin(u)
z = (v / 2) * np.sin(u / 2)
```

### Построение фигуры:

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(x, y, z, cmap="plasma")
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')
ax.text2D(0.25, 0.95, "Mobius Ribbon", transform=ax.transAxes)
plt.show()
```


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section 18-2, diffusion binaire dans un milieu stagnant

Voir les hypothèses dans le livre de Transport Phenomena.

```python
#
# Pierre Proulx
#
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
%matplotlib inline
```

```python
# Paramètres, variables et fonctions

z_1,z_2,z,D_AB,X_A1,X_A2,S,delta_z,C1,C2=sp.symbols('z_1,z_2,z,D_AB,X_A1,X_A2,S,delta_z,C1,C2')
N_A=sp.symbols('N_A')
X_A=sp.Function('X_A')(z)
```

```python
#
# Bilan sur la colonne de diffusion
#
eq=(S*N_A(z)-S*N_A(z+delta_z))/(S*delta_z)
display(eq)
```

```python
eq=sp.limit(eq,delta_z,0).doit()
display(eq)
```

```python
fick=-D_AB/(1-X_A)*sp.diff(X_A,z)
eq=eq.subs(N_A(z),fick)
display(eq)
```

```python
X_A=sp.dsolve(eq,X_A)
display(X_A)
```

```python
X_A=X_A.rhs
# Pose et solution des 2 équations de conditions aux limites pour C1 et C2
condition_1=sp.Eq(X_A.subs(z,z_1)-X_A1,0)
condition_2=sp.Eq(X_A.subs(z,z_2)-X_A2,0)
constantes=sp.solve([condition_1,condition_2],sp.symbols('C1,C2'),dict=True) # C1 et C2
constantes=constantes[0]         # ramener la liste contenant le dictionnaire des constantes
display(constantes)              # sous forme de dictionnaire
```

```python
X_A=X_A.subs(constantes)             #
display(X_A.simplify())
#
# La forme de l'équation est lourde, voir plus bas pour formuler de façon plus simple.
#
```

```python
#
# Tracons avec des valeurs réalistes
#
dico={'z_1':0.00,'z_2':0.2,'X_A1':0.5,'X_A2':0.1}
X_Aplot=X_A.subs(dico)
display(X_Aplot)
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize']=10,8
sp.plot(X_Aplot,(z,0.0,0.2),ylabel='X_A',ylim=[0,1],adaptive=False)
```

```python
### Ici on peut regarder le profil de concentration en choisissant une géométrie
### un peu simplifiée, par exemple z1=0 et z2=L. Regardons si le profil serait plus simple
### à lire
X_As=X_A.subs(sp.symbols('z_1'),0)
X_As=X_As.subs(sp.symbols('z_2'),sp.symbols('L'))
display(X_As)
### De plus, on pourrait utiliser XB=XA-1
X_As=X_As.subs(sp.symbols('X_A1'),1-sp.symbols('X_B1'))
X_As=X_As.subs(sp.symbols('X_A2'),1-sp.symbols('X_B2'))
display(X_As.simplify())
### en effet, on aurait intérêt à utiliser une notation simplifiée ou z2-z1=L et z1=0.
```


# The Harmonic Oscillator Strikes Back

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week we continue our adventures with the harmonic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2} m \omega_0^2 {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

If the equilibrium value of the harmonic oscillator is not zero, then

$$ F=- \omega_0^2 m (x-x_{eq}), \text{ }  V(x) = {1 \over 2} m \omega_0^2 (x-x_{eq})^2$$

## 1. Harmonic oscillator from last time (with some better defined conditions)

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= -m \omega_0^2 (x-x_{eq}) $$

$$ a = - \omega_0^2 (x-x_{eq}) $$

$$ x(t)'' = - \omega_0^2 (x-x_{eq}) $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

This is already solved to remind you how we found these values

```python
import sympy as sym
sym.init_printing()
```

**Note** that this time we define some of the properties of the symbols. Namely, that the frequency is always positive and real and that the positions are always real

```python
omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)
x(t),omega0
```

```python
dfeq=sym.Derivative(x(t),t,2)+omega0**2*(x(t)-xeq)
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

```python
sol,sol.args[0],sol.args[1]
```

**Note** this time we define the initial positions and velocities as real

```python
x0,v0=sym.symbols("x_0,v_0",real=True)
ics=[sym.Eq(sol.args[1].subs(t, 0), x0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

```python
solved_ics=sym.solve(ics)
solved_ics
```

### 1.1 Equation of motion for $x(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

### 1.2 Equation of motion for $p(t)$

```python
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
```

## 2. Time average values for a harmonic oscillator

If we want to understand the average value of a time dependent observable, we need to solve the following integral

$${\left\langle {A(t)} \right\rangle}_t  = \begin{array}{*{20}{c}}
{\lim }\\
{\tau  \to 0}
\end{array}\frac{1}{\tau }\int\limits_0^\tau  {A(t)dt} $$

### 2.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
xfunc=full_sol.args[1]
xavet=(xfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
xavet
```

The computer does not always make the best choices the first time. If you treat each sum individually this is not a hard limit to do by hand. The computer is not smart. We can help it by inseting an `expand()` function in the statement

```python
xavet=(xfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
xavet
```

### 2.2 Excercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

```python
import sympy as sym
sym.init_printing()
```

```python
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
```

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
pfunc=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t)).args[1]
pavet=(pfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
pavet
```

### 2.3 Exercise: Calculate the average kinetic energy of a harmonic oscillator

```python
kefunc=(sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t)).args[1])**2/(2*m)
keavet=(kefunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
keavet
```

```python
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t)).args[1]**2/(2*m)
```

```python
((m*sol.args[1].subs(solved_ics[0]).diff(t)**2/(2*m)))
```

```python
m*sol.args[1].subs(solved_ics[0]).diff(t)**2/(2*m)
```

## 3.  Ensemble (Thermodynamic) Average values for a harmonic oscillator

If we want to understand the thermodynamics ensemble average value of an observable, we need to solve the following integral.

$${\left\langle {A(t)} \right\rangle}_{T}  = \frac{\int{A e^{-\beta H}dqdp}}{\int{e^{-\beta H}dqdp} } $$

You can think of this as a Temperature average instead of a time average.

Here $\beta=\frac{1}{k_B T}$ and the classical Hamiltonian, $H$ is

$$ H = \frac{p^2}{2 m} + V(q)$$

**Note** that the factors of $1/h$ found in the classical partition function cancel out when calculating average values

### 3.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

For a harmonic oscillator with equilibrium value $x_{eq}$, the Hamiltonian is
$$ H = \frac{p^2}{2 m} + \frac{1}{2} m \omega_0 (x-x_{eq})^2 $$

First we will calculate the partition function $\int{e^{-\beta H}dqdp}$

```python
k,T=sym.symbols("k,T",positive=True,nonnegative=True,real=True)
xT,pT=sym.symbols("x_T,p_T",real=True)
ham=sym.Rational(1,2)*(pT)**2/m + sym.Rational(1,2)*m*omega0**2*(xT-xeq)**2
beta=1/(k*T)
bolz=sym.exp(-beta*ham)
z=sym.integrate(bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
z
```

Then we can calculate the numerator $\int{A e^{-\beta H}dqdp}$

```python
numx=sym.integrate(xT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
numx
```

And now the average value

```python
xaveT=numx/z
xaveT
```

### 3.2 Exercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

After calculating the value, explain why you think you got this number

```python
nump=sym.integrate(pT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
nump
```

```python
paveT=nump/z
paveT
```

### 3.3 Exercise: Calculate the average kinetic energy

The answer you get here is a well known result related to the energy equipartition theorem

```python
keaveT=sym.integrate(pT**2/(2*m)*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))/z
keaveT
```

# Back to the lecture

## 4. Exercise Verlet integrators

In this exercise we will write a routine to solve for the equations of motion for a hamonic oscillator.

Plot the positions and momenta (seprate plots) of the harmonic oscillator as a functions of time.

Calculaate trajectories using the following methods:
1. Exact solution
2. Simple taylor series expansion
3. Predictor-corrector method
4. Verlet algorithm
5. Leapfrog algorithm
6. Velocity Verlet algorithm

```python
# Your code here
```

```python
%matplotlib inline
```

```python
omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)
full_sol1 = sym.simplify(full_sol.subs({x0:10, xeq:0 , v0:10, omega0:1}))
sym.plot(full_sol1.rhs,(t,-10,10))
```

```python
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
momentum=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
```

```python
momentum1=sym.simplify(momentum.subs({x0:10, xeq:0, v0:10, omega0:1, m:1}))
```

```python
sym.plot(momentum1.rhs,(t,-10,10))
```

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
x_t0=0
t=0.25
v_t0=1
xlist=[]
for i in range(0,500):
    v_t=v_t0+(1/2)*t**2*(-(x_t0))
    v_t0=v_t
    x_t= x_t0+(v_t0)*(t)
    x_t0=x_t
    xlist.append(x_t)
plt.plot(xlist)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.title('Taylor series approximation')
```

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
x_t0=0
x_t1=1
t=0.4
a=1
xlist=[]
for i in range(0,100):
    x_t2=2*x_t1-x_t0+t**2*(-(x_t1))
    x_t0=x_t1
    x_t1=x_t2
    xlist.append(x_t2)
plt.plot(xlist)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.title('Verlet Algorithm-Position')
```

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
x_t0=0
x_t2=1
t=2
xlist=[]
for i in range(0,100):
    v_t=(x_t2-x_t0)/2*t
    x_t0=x_t2
    x_t2=v_t
    xlist.append(v_t)
plt.plot(xlist)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.title('Verlet Algorithm-Velocity')
```

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
x_t0=1
v_minushalft=0
t=0.2
xlist=[]
for i in range(0,100):
    v_halft=v_minushalft+(t)*(-(x_t0))
    v_minushalft=v_halft
    x_t1=x_t0+(t)*(v_halft)
    x_t0=x_t1
    xlist.append(x_t1)
plt.plot(xlist)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.title('Leapfrog Algorithm-Position')
```

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
x_t0=1
t=0.3
v_minushalft=1
v_halft=2
xlist=[]
for i in range(0,100):
    v_t=(1/2)*((v_halft)+(v_minushalft))
    v_minushalft=v_t
    v_halft=v_minushalft+(t)*(-(x_t0))
    v_minushalft=v_halft
    x_t1=x_t0+(t)*(v_halft)
    x_t0=x_t1
    xlist.append(v_t)
plt.plot(xlist)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.title('Leapfrog Algorithm-Velocity')
```

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
dt=0.1
x_0=1
v_0=1
xlist=[]
for i in range(0,100):
    x_1 = x_0 + v_0 * dt + 0.5 * -x_0 * dt * dt
    x_0 = x_1
    v_1 = v_0 + 0.5 * (-x_0 + -x_1) * dt
    v_0 = v_1
    xlist.append(x_1)
    #print(xlist)
plt.plot(xlist)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.title('Velocity Verlet-Position Approx')
```

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
x_t0=1
t=2
v_t0=1
xlist=[]
for i in range(0,50):
    v_halft=v_t0+(1/2)*t*(-x_t0)
    x_t0=v_halft
    xlist.append(v_halft)
plt.plot(xlist)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.title('Velocity Verlet-Velocity Approximation')
```

```python
#i officially quit and resign
```

```python

```


```
#@title Imports
%matplotlib inline

import numpy as np
import matplotlib.colors
import matplotlib.pyplot as plt

import sklearn.linear_model
import sklearn.discriminant_analysis

import seaborn as sns
sns.set_style('whitegrid')
```

# Generative and discriminative classifiers

Suppose you have a dataset of $x^{(i)}, y^{(i)}$ pairs, where $x^{(i)}\in\mathbb{R}^D$ and $y^{(i)}\in\{0, 1, ..., C\}$.

A *classifier* is tasked with finding the label (or class) $y$ of a givan sample $x$. In probabilistic terms, the classifier should find
$$
p(y|x).
$$

The classification task can be oslved in two ways:
1. Directly, by modeling $p(y|x)$. We call such classifiers **discriminative**.

   E.g. logistic regression, with the model fit to data using the maximum likelihood principle.

2. Indirectly, by first modeling $p(y)$ and $p(x|y)$, then using the Bayes theorem to get $p(y|x)$. We call such models **generative**

   E.g. the Naive Bayes and the Gaussian Discriminant Analysis, defined below, are both generative. They are fit by modeling separately points belinging to the same class.

Generatively and discriminatively trained models approach the classification problem from two directions:
- generative models first learn to describe samples belongign to each class. Then, classification of a new sample amounts to finding a class that makes the observation the most probable.
- on the other hand, discriminative models are trained to find the few features that can set the two classes apart. Ideally, they will only focus on the few discriminating traits, and ignore everything else.

# Gaussian Discriminant Analysis

The GDA models a joint distribution $p(x,y) = p(x|y)p(y)$ with:
$$
\begin{align}
y &\sim \text{Bernoulli}(\phi) \\
x|y=0 & \sim\mathcal{N}(\mu_0, \Sigma_0) \\
x|y=1 & \sim\mathcal{N}(\mu_1, \Sigma_1)
\end{align}
$$

Which corresponds to the following expressions of probality:
$$
\begin{align}
p(y)     &= \phi^y(1-\phi)^{1-y}\\
p(x|y=0) &= \frac{1}{(2\pi)^{d/2}|\Sigma_0|^{1/2}}\exp\left(-\frac{1}{2}(x-\mu_0)^T\Sigma_0^{-1}(x-\mu_0)\right) \\
p(x|y=1) &= \frac{1}{(2\pi)^{d/2}|\Sigma_1|^{1/2}}\exp\left(-\frac{1}{2}(x-\mu_1)^T\Sigma_1^{-1}(x-\mu_1)\right)
\end{align}
$$

Using the Bayes theorem, we recover the decision boundary:
$$
\begin{split}
p(y=1|x) &= \frac{p(x|y=1)p(y=1)}{p(x|y=1)p(y=1) + p(x|y=0)p(y=0)}
         &= \frac{1}{1+\frac{p(x|y=0)p(y=0)}{p(x|y=1)p(y=1)}}
\end{split}
$$

Let's expand the term in the denominator:
$$
\begin{split}
&\frac{p(x|y=0)p(y=0)}{p(x|y=1)p(y=1)} = \\
&\frac{\frac{1}{(2\pi)^{d/2}|\Sigma_0|^{1/2}}\exp\left(-\frac{1}{2}(x-\mu_0)^T\Sigma_0^{-1}(x-\mu_0)\right) (1-\phi)}{\frac{1}{(2\pi)^{d/2}|\Sigma_1|^{1/2}}\exp\left(-\frac{1}{2}(x-\mu_1)^T\Sigma_1^{-1}(x-\mu_1)\right) \phi}
\end{split}
$$

This leads to a quadratic boundary function and the resulting model is ofetn called "Quadratic disrciminant Analysis".

An interesintg simplification happens when $\Sigma_0=\Sigma_1=\Sigma$, i.e. when we asume that samles belonging to all classes have the same covariance matrix. Then:

$$
\begin{split}
&\frac{p(x|y=0)p(y=0)}{p(x|y=1)p(y=1)} = \\
&\frac{1-\phi}{\phi}exp\left(-\frac{1}{2}(x-\mu_0)^T\Sigma^{-1}(x-\mu_0) +\frac{1}{2}(x-\mu_1)^T\Sigma^{-1}(x-\mu_1) \right) =\\
&= \exp(-\Theta^T x)
\end{split}
$$
For a certain value of $\Theta$. Thus, the GDA's decision boundary is liear and the resulting model is called "Linear Discriminant Analysis" (LDA).

We note that while it computes the decision boundry using a firmula similar to logistic regression, i.e.
$$
p(y=1|x) = \frac{1}{1+\exp(-\Theta^T x)}
$$
it is trained in a very different fashion.

## LDA training
We fit the LDA model by maximizing the joint loglikelihood equal to
$$
\begin{split}
\mathcal{l}(\phi,\mu_0,\mu_1, \Sigma) &= \log\prod_{i=1}^{N}p(x^{(i)}, y^{(i)}; \phi,\mu_0,\mu_1, \Sigma) \\
&= \log\prod_{i=1}^{N}p(x^{(i)} | y^{(i)}; \mu_0,\mu_1, \Sigma)p(y^{(i)}; \phi)
\end{split}
$$

Maximizing the likelihood leads to intuitive formulas for the parameters:

$$
\begin{align}
\phi &= \frac{1}{N}\sum_{i=1}^{N}[y^{(i)}=1] \\
\mu_0 &= \frac{\sum_{i=1}^{N}[y^{(i)}=0]x^{(i)}}{\sum_{i=1}^{N}[y^{(i)}=0]} \\
\mu_1 &= \frac{\sum_{i=1}^{N}[y^{(i)}=1]x^{(i)}}{\sum_{i=1}^{N}[y^{(i)}=1]} \\
\Sigma &= \frac{1}{N}\sum_{i=1}^{N}(x^{(i)} - \mu_{y^{(i)}})(x^{(i)} - \mu_{y^{(i)}})^T \\
&= (1-\phi) \frac{\sum_{i: y^{(i)}=0}(x^{(i)} - \mu_{0})(x^{(i)} - \mu_{0})^T}{\sum_{i: y^{(i)}=0}1} + \phi \frac{\sum_{i: y^{(i)}=1}(x^{(i)} - \mu_{1})(x^{(i)} - \mu_{1})^T}{\sum_{i:y^{(i)}=1}1}
\end{align}
$$

We see that $\phi$ simply equals to the fration of samples belonging to class $1$, that the means $\mu_0$ and $\mu_1$ correspond to the means of samples in both classes and finally that the covariance matrix is just the weighted mean of the covariance matrices for each class.

## LDA vs Logistic regression

Once trained, LDA and logistic regression's predictions can be computed using the same formula. However, the classifiers are fitted using a different objective and thus will be different. Let's plot them on some exemplary data.

```
## Helper function
def plot_dec_boundary(X, Y, model, class_threshold=0.5):
    plt.scatter(X[0, :], X[1,:], c=Y, cmap='spring', edgecolors='black')
    nx, ny = 200, 100
    x_min, x_max = plt.xlim()
    y_min, y_max = plt.ylim()
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, nx),
                            np.linspace(y_min, y_max, ny))
    Z = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])
    Z = Z[:, 1].reshape(xx.shape)
    plt.pcolormesh(xx, yy, Z, cmap='spring',
                    norm=matplotlib.colors.Normalize(0., 1.), zorder=0)
    plt.contour(xx, yy, Z, [class_threshold], linewidths=1., colors='white')
```

```
## Gen some data
Sigma = np.array([
    [1.0, 0.8],
    [0.8, 2.0]])
Mu0 = np.array([[-2], [-2]])  # Try also -1 , -1
Mu1 = np.array([[2], [2]])
N0 = 50
N1 = 30

X = np.concatenate([
    Mu0 + Sigma @ np.random.randn(2, N0),
    Mu1 + Sigma @ np.random.randn(2, N1)], axis=-1)
Y = np.concatenate([
    np.zeros(N0), np.ones(N1)], axis=0)

for model in [
    sklearn.linear_model.LogisticRegression(solver='lbfgs'),
    sklearn.discriminant_analysis.LinearDiscriminantAnalysis(),
]:
    model.fit(X.T, Y)

    plt.figure()
    plot_dec_boundary(X, Y, model)
    plt.title(model.__class__.__name__)
```

When the data is generated according to the LDA model the two classifiers act similarly. However, differences start to show up when the data is generated form a distribution that deviates from the normality assumtions.

```
## Gen some data
Sigma = np.array([
    [1.0, -0.6],
    [-0.6, 2.0]])
Mu0 = np.array([[-2], [-2]])  # Try also -1 , -1
Mu11 = np.array([[2], [2]])
Mu12 = np.array([[10], [10]])
N0 = 50
N1 = 30

X = np.concatenate([
    Mu0 + Sigma @ np.random.randn(2, N0),
    Mu11 + Sigma @ np.random.randn(2, N1//2),
    Mu12 + Sigma @ np.random.randn(2, N1//2)], axis=-1)
Y = np.concatenate([
    np.zeros(N0), np.ones(N1)], axis=0)

for model in [
    sklearn.linear_model.LogisticRegression(solver='lbfgs'),
    sklearn.discriminant_analysis.LinearDiscriminantAnalysis(),
]:
    model.fit(X.T, Y)

    plt.figure()
    plot_dec_boundary(X, Y, model)
    plt.title(model.__class__.__name__)
```

# The relationship between LDA and Logistic Regression

While both classifiers separate the two classes using a hyperplane, they make different assumptions about the data:
- GDA (and in turn LDA and QGA) assume that the class-conditional data densities are normally distributed. Whtn this is satisfied, they have a better smaple efficiency than logistic regresion
- Logistic Regression makes no assumptions about the class conditional densities, and instead directly models the probability $p(y|x)$. This makes it more robust, but less sample efficient when the data is indeed normal.

In practice, Logistic Regression often works better!

# Naive Bayes and Logistic Regression

It turns out, that Naive Bayes and Logistic Regression are simiar as well.

Assume that each sample is described using $F$ discrete features, with $x_f$ denoting the value of the $f$-th feature.

Assume that the data is discrete. For Naive bayes we can model it directly, while for Logistic Regression we will need to encode each feature using a 1-hot encoding, in which we replace each discrete feature with a binary vector which is all zeros, except for a single one at the location corresponding to the discrete value.

For both classifiers the decision boundary takes the form:
$$
\sum_f \theta_{f,x_f} > c.
$$
In naive Bayes, $\theta_{f,x_f} = \log p(f=x_f|y=1) - \log p(f=x_f|y=0)$, while in the case of Logistic Regression $\Theta_{f,x_f}$ is the coefficeint of the $x_f$'s value of feature $f$ determined using optimization.

The two classifiers behave in a similar way:
- Naive Bayes has a stronger inductive bias, often working better on very small datasets and saturating to a worse error rates
- Logistic Regression needs more data to work well, but oftensaturates at better error rates than naive Bayes.

## Empirical comparision of generative and discriminative classifiers

Andrew Ng made an extensive comparision of performance of generative and discriminative classifiers, with main results summarised in [Figure 1](https://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf).

# Further reading
- CS229 lecture notes on discriminative classifiers http://cs229.stanford.edu/notes2019fall/cs229-notes2.pdf
- Andrew Ng, Michael Jordan, "On discriminative vs. Generative classifiers: A comparison of logistic regression and naive bayes" https://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf
- Bradley Efron "The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.3459&rep=rep1&type=pdf

```

```


```python
import numpy as np
import matplotlib.pyplot as plt
import imageio

plt.style.use('fivethirtyeight')
```

# Introduction to Engineering Dynamics
## Newtonian Mechanics

There are three laws that define _Newtonian Mechanics_:
1. An object at rest will stay at rest and an object in motion will remain in motion unless acted upon by an external force $\sum\mathbf{F}=\mathbf{0}$ and $\sum\mathbf{M} = \mathbf{0}$
2. The change in momentum of a body is proportional to the force applied, $\sum \mathbf{F} = \frac{d}{dt}\left(m\mathbf{v}\right)$ and $\sum\mathbf{M} = \frac{d}{dt}\left(\mathbf{h}\right)$
3. For every applied force, there is an equal and opposite reaction force

```python
import sympy
```

```python

sympy.var('m, g, P_x, P_y, R_y, W_P, L, x')

Fx = P_x
Fy = P_y + R_y - W_P - m*g
Mz = -L/2*m*g + L*R_y - x*W_P

eqns = sympy.Matrix([Fx, Fy, Mz])
A = sympy.solve((eqns[0], eqns[1], eqns[2]), (P_x, P_y, R_y))
Py_function = sympy.lambdify((x, L, W_P, m, g), A[P_y], 'numpy')
Ry_function = sympy.lambdify((x, L, W_P, m, g), A[R_y], 'numpy')
```

## Bridge example: live load no acceleration

Consider a 50-kg person travelling from the left-to-right edges of the bridge at a constant velocity. The acceleration of the person is 0 and the bridge has 0 acceleration.

- $v_{person} = 10~\frac{m}{s}$
- $x_{person} = \int_0^tv_{person}dt = 10t~m$

The first law defines the study of static structures and objects that are moving slowly or have negligible momentum, $m\mathbf{v}$. In order to keep the bridge from moving, the total applied force $\mathbf{F}$ and moments $\mathbf{M}$ must be equal to $\mathbf{0}$.

$\left[\begin{matrix}
\sum F_x\\
\sum F_y\\
\sum M_z
\end{matrix}\right] =
\left[\begin{matrix}
P_{x}\\
P_{y} + R_{y} - W_{P} - g m\\
L R_{y} - \frac{L g m}{2} - W_{P} x
\end{matrix}\right] =
\left[\begin{matrix}
0\\ 0\\ 0\\
\end{matrix}\right]$

> __Note__: If you set up the equations by separating the unknown variables, $[P_x,~P_y,~R_y]$, from the known variables, you can set up a _linear algebra_ problem. For more information, check out the excellent video series [3Blue 1Brown - Essence of Linear Algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)

>$\left[\begin{matrix}
~1 & 0 & 0\\
0 & 1 & 1 \\
0 & 0 &L
\end{matrix}\right]
\left[\begin{array}
~P_x\\
P_y\\
R_y\end{array}\right]=
\left[\begin{matrix}
0\\ W_{P} + mg \\ \frac{mgL}{2}+W_Px\\
\end{matrix}\right]$

Solving for the reaction forces, you have three equations,

1. $P_x = 0$
2. $P_y = W_P +mg - \frac{mg}{2}-W_P\frac{x}{L}$
3. $R_y = \frac{mg}{2}+W_P\frac{x}{L}$

```python
g = 9.81
L = 10
m = 100
Wp = 50*g

Px = lambda x: 0*x
Py = lambda x: Wp + m*g/2 - Wp*x/L
Ry = lambda x: m*g/2 +Wp*x/L

t = np.linspace(0, 1)
v = 10
x = v*t

plt.plot(x, Px(x), label = r'$P_x$')
plt.plot(x, Py(x), label = r'$P_y$')
plt.plot(x, Ry(x), label = r'$R_y$')
plt.legend()
plt.xlabel('walker position (m)')
plt.ylabel('reaction force (N)')

```

Here you can see the reaction forces at each end of the bridge as a function of the person moving from left, $x=0~m$ to right, $x=10~m$. The reaction force increases as the walker moves away from the support.

You can watch the reaction forces change in the animation below.

```python
from matplotlib import animation
from IPython.display import HTML
fig, ax = plt.subplots(1,1);
ax.plot(x, x*0, '-', linewidth = 10, label = 'bridge')
Q = ax.quiver([0, 10],
           [0, 0],
           [0, 0],
           [Py(x[1]), Ry(x[1])],
          scale = 3500, label = 'reaction forces')
mark, = ax.plot(x[1], 0.01, 'v', markersize = 30, label = 'walker position')

ax.legend(loc = 'lower center')
ax.set_xlim(-0.5, 10.5)
ax.set_ylim(-0.1, 0.1)

def animate(i):
    """updates the horizontal and vertical vector components by a
    fixed increment on each frame
    """
    Q.set_UVC(np.array([0,  0]),
              np.array([Py(x[i]), Ry(x[i])]))

    mark.set_data(x[i], 0.05)

    return Q, mark

# you need to set blit=False, or the first set of arrows never gets
# cleared on subsequent frames
anim = animation.FuncAnimation(fig, animate,
                               frames = range(len(t)-1), blit=False)
HTML(anim.to_html5_video())
```

## An example with $a\neq 0$

## Acceleration of a falling ball $a\neq 0$

In the previous bridge example, the person and bridge had 0 acceleration. Consider an falling object. Suppose you want to use video capture of a falling ball to _compute_ the acceleration of gravity.

Watch the video of the ball falling.

```python
from IPython.display import YouTubeVideo
vid = YouTubeVideo("xQ4znShlK5A")
display(vid)
```

We learn from the video that the marks on the panel are every $0.25\rm{m}$, and on the [website](http://techtv.mit.edu/collections/physicsdemos/videos/831-strobe-of-a-falling-ball) they say that the strobe light flashes at about 15 Hz (that's 15 times per second). The final [image on Flickr](https://www.flickr.com/photos/physicsdemos/3174207211) notes that the strobe fired 16.8 times per second.

The ball is freefalling, so the acceleration is constant, $a = g = 9.81~\frac{m}{s^2}$. Its not moving left or right, so you can describe its position and velocity as

- $x(t) = x_0$
- $y(t) = y_0 + \dot{y}_0t +\frac{g}{2}t^2$

where the initial position is $(x_0,~y_0)$,  the initial velocity is $(0,~\dot{y}_0)$, and $t$ is elapsed time. Take a look at the final frame of the compiled positions below.

```python
reader = imageio.get_reader('https://go.gwu.edu/engcomp3vidmit', format='mp4')
image = reader.get_data(1100)

plt.imshow(image, interpolation='nearest');
```

The first position after releasing the ball, is just below the first white line at 125 pixels. The average distance between markers is 495 pixels/meter, so the camera would measure an acceleration

$g = 9.81~\frac{m}{s^2}\cdot\frac{495~pixels}{1~meter}=4850~\frac{pixels}{s^2}$.

With a little guess-and-check, the initial velocity just below the first white marker is $\dot{y}_0=800~\frac{pixels}{s}$. The position on the image is then,

$y(t) = 125 + 800 t + \frac{4850}{2}t^2~pixels$.

The frames are plotted on top of the image below.

```python
x0 = 720
y0 = 125
bars = np.array([115, 980])
scale = np.diff(bars)/1.75
print(scale, 'px/m')

g = 9.81*scale # m/s^2*d*0.25 px/m = px/s^2

t = np.arange(0, 8, 1)*1/16.8
x = np.ones(len(t))*x0
y = y0 + 800*t + g*t**2/2
plt.clf()
plt.imshow(image)
plt.scatter(x, y, s = 80, facecolor='none', edgecolors='b')
```

## Wrapping up

In this notebook, you looked at two situations

1. forces and motion when acceleration is 0
2. motion when acceleration is $a\neq0$

Next, you will define properties of kinetics (forces, energy, and impulse) and kinematics (motion described by position, velocity, acceleration).


# Lecture 5

This lecture covers linear difference equations (recurrence equations). Below are some examples solved using SymPy.

We first import the necessary modules:

```python
from sympy import *
init_printing()
from IPython.display import display

from fractions import Fraction

# Support for interactive plots
from ipywidgets import interact

%matplotlib inline
```

## Student grant example

This example is covered in the lecture (solved by hand). Here it is solved using SymPy.

The difference equation (non-homegeneous case) for the student grant example from the lecture notes has the form

$$
y_{n} - 2.95 y_{n-1} + 2 y_{n-2} = −(63.685)(1.07^{n})
$$

We first define $n$ as an integer symbol and $y$ as a function:

```python
n = Symbol("n", integer=True)
y = Function("y")
```

Now, we define the difference equation:

```python
f = y(n) - Fraction(295, 100)*y(n - 1) + 2*y(n - 2)
display(f)
```

We'll solve the homogeneous version of the equation using `rsolve`. To compare to the solution in the lecture notes, we'll also evaluate the the solution in floating point:

```python
eqn = Eq(f, 0)
soln = rsolve(eqn, y(n))
display(soln)
soln.evalf()
```

The solution is the same as in the lecture notes.

We now consider the non-homogeneous case:

```python
# Create non-homogeneous equation
eqn = Eq(f, -63.685*(1.07**n))
display(eqn)
```

Solving the non-homogeneous equation, on this occasion providing the initial conditions $y(0) = 2000$ and $y(1) = 2200$,

```python
soln = rsolve(eqn, y(n), init={y(0) : 2000, y(1) : 2200})
display(soln)
soln.evalf()
```

```python
# Plot
plot(soln, (n, 0, 12), xlabel="year (n)", ylabel="fee")

# Evaluate fee at 10 years
print("Fee after 10 years (n=10): {}".format(soln.subs(n, 10).evalf()))
```

## General case

Below is the general case for a constant coefficient, second-order difference equation.

```python
n = Symbol("n", integer=True)
y = Function("y")

a, b, c = symbols("a b c")

f = a*y(n) - b*y(n - 1) + c*y(n - 2)
display(f)
```

Solving for the homogeneous case,

```python
eqn = Eq(f, 0)
soln = rsolve(eqn, y(n))
display(soln)
```

The above is a general solution. Solving for some initial conditions,

```python
soln = rsolve(eqn, y(n), init={y(0) : 2, y(1) : 1})
display(soln)
```

If we now specify the values for the constants, the solution can be plotted:

```python
eqn1 = eqn.subs('a', 4).subs('b', 1).subs('c', 2)
soln = rsolve(eqn1, y(n), init={y(0) : 1, y(1) : 2})
display(soln)

plot(soln, (n, 0, 12), xlabel="$n$", ylabel="$y_{n}$");
```

## Interactive solution

With an interactive plot, we can see how the form of the solution changes with changing constants in the difference equation.

We first define the difference equation:

```python
f = a*y(n) - b*y(n - 1) + c*y(n - 2)
eqn = Eq(f, 0)
display(eqn)
```

Solving and plotting:

```python
def plot_solution(a=6, b=1, c=8):
    # Substitute in parameters
    eqn1 = eqn.subs('a', a).subs('b', b).subs('c', c)

    # Check roots
    roots = ((-b - sqrt(b**2 - 4*a*c))/(2*a), (-b + sqrt(b**2 - 4*a*c))/(2*a))
    print("Roots of characteristic eqn:")
    display(roots)

    # Solve
    soln = rsolve(eqn1, y(n), init={y(0) : 1, y(1) : 2})
    print("Solution:")
    display(soln)

    # Plot
    plot(soln, (n, 0, 20), xlabel="$n$", ylabel="$y_n$", axis_center="auto", adaptive=False, nb_of_points=21)

interact(plot_solution, a=(-10, 11, 2), b=(-2, 20, 1), c=(1, 10, 1), continuous_update=False);
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.


# Linear Models

There's a few linear models out there that we felt were generally useful. This
document will highlight some of them.

## LOWESS

Lowess stands for LOcally WEighted regreSSion and has historically
been used for smoothing but you can also use it
for machine learning where you're interested in interpolating. Here's
a demo;

```python
import numpy as np
import matplotlib.pylab as plt
from sklego.linear_model import LowessRegression

n = 100
xs = np.linspace(0, np.pi, n)
ys = 1 + np.sin(xs) + np.cos(xs**2) + np.random.normal(0, 0.1, n)

mod = LowessRegression(sigma=0.1, span=0.5).fit(xs.reshape(-1, 1), ys)

xs_new = np.linspace(-1, np.pi + 1, n*2)
preds = mod.predict(xs_new.reshape(-1, 1))

plt.figure(figsize=(12, 4))
plt.scatter(xs, ys)
plt.plot(xs_new, preds, color='orange')
plt.title("Be careful with extrapolation here.");
```

The line does not look linear but that's because internally, during prediction,
many weighted linear regressions are happening. The gif below demonstrates how
the data is being weighted when we would make a prediction.

### Details on `sigma`.

We'll also show two different prediction outcomes depending on the hyperparameter
`sigma`.

You may be tempted now to think that a lower sigma always has a better fit, but you
need to be careful here. The data might have gaps and larger sigma values will be
able to properly regularize.

Note that this regression also works in higher dimensions but the main downside of
this approach is that it is really slow when making predictions.

If you really want to get advanced there's also a hyperparameter `span` but you'll
really need to know what you're doing. It was added for completeness but the authors
of this package have yet to find a proper usecase for it.

### Details on `span`.

The `span` parameter can be used to force that you'll only include a certain
  percentage of your dataset. Technically without a `span` you are still using
  all the data in your dataset, albeit with small weights if they are far away.

The effect of the `span` parameter on the weights can be seen below.

This will also effect the predictions.

You may need to squint your eyes a bit to see it, but lower spans cause more
  jiggles and less smooth curves.

## ProbWeightRegression

> Note, this is a somewhat experimental feature. We found this feature to be plausibly useful but we've not seen it cause a big "win" yet.

Let's say that you're interested in combining a few models for a regression task.
You could put them together in an ensemble. Say we've got predictions $y_1, y_2, y_3$, each of which come from respectable models, then you may want to combine these together in another linear regression. This way the new, hopefully best, prediction $y_*$ is defined via;

$$y^* = w_1 y_1 + w_2 y_2 + w_3 y_3$$

This can be a valid way of reweighing. But there's one issue:
technically the weights $w_1, w_2, w_3$ can sum to a number that isn't one. Since that's
numerically totally possible we need to be aware that we can end up in a strange situation.

The `ProbWeightRegression` adresses this. It assumes that every input it receives is
the output of a model and it will ensure that they are reweighed with a constraint in mind. For this usecase, it would optimise;

\begin{equation} \label{eq1}
\begin{split}
\text{min} & \sum_i (y_i - (w_1 y_{1,i} + w_2 y_{2,i} + w_3 y_{3,i}))^2 \\
\text{subject to} & \\
& \text{ } w_1 + w_2 + w_3 = 1 \\
& \text{ } w_1 \geq 0, w_2 \geq 0, w_3 \geq 0
\end{split}
\end{equation}

The final positivity constraint is optional in our model.

Here's an example usage.

```python
from sklearn.datasets import make_regression
import pandas as pd

X, y = make_regression(n_samples=1000, n_features=10, random_state=42)
df = pd.DataFrame(X)
```

We've turned the array into a dataframe so that we can apply the `ColumnSelector`.

```python
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.linear_model import LinearRegression
from sklearn.decomposition import PCA
from sklearn.model_selection import GridSearchCV

from sklego.meta import EstimatorTransformer
from sklego.linear_model import ProbWeightRegression
from sklego.preprocessing import ColumnSelector

pipe = Pipeline([
    ('models', FeatureUnion([
        ('path1', Pipeline([
            ('select1', ColumnSelector([0,1,2,3,4])),
            ('pca', PCA(n_components=3)),
            ('linear', EstimatorTransformer(LinearRegression()))
        ])),
        ('path2', Pipeline([
            ('select2', ColumnSelector([5,6,7,8,9])),
            ('pca', PCA(n_components=2)),
            ('linear', EstimatorTransformer(LinearRegression()))
        ]))
    ])),
    ('prob_weight', ProbWeightRegression())
])

grid = GridSearchCV(estimator=pipe, param_grid={}, cv=3).fit(df, y)
```

```python
from sklearn import set_config
set_config(display='diagram')
grid
```

<input class="sk-toggleable__control sk-hidden--visually" id="1f1cc5db-07d1-470d-aa5a-253d7d316602" type="checkbox" >GridSearchCV

GridSearchCV(cv=3,
             estimator=Pipeline(steps=[('models',
                                        FeatureUnion(transformer_list=[('path1',
                                                                        Pipeline(steps=[('select1',
                                                                                ColumnSelector(columns=[0,
                                                                                1,
                                                                                2,
                                                                                3,
                                                                                4])),
                                                                                ('pca',
                                                                                PCA(n_components=3)),
                                                                                ('linear',
                                                                                EstimatorTransformer(estimator=LinearRegression()))])),
                                                                       ('path2',
                                                                        Pipeline(steps=[('select2',
                                                                                ColumnSelector(columns=[5,
                                                                                6,
                                                                                7,
                                                                                8,
                                                                                9])),
                                                                                ('pca',
                                                                                PCA(n_components=2)),
                                                                                ('linear',
                                                                                EstimatorTransformer(estimator=LinearRegression()))]))])),
                                       ('prob_weight',
                                        ProbWeightRegression())]),
             param_grid={})

<input class="sk-toggleable__control sk-hidden--visually" id="722adf51-b4eb-4aea-acce-ff4fd0835955" type="checkbox" >models: FeatureUnion

FeatureUnion(transformer_list=[('path1',
                                Pipeline(steps=[('select1',
                                                 ColumnSelector(columns=[0, 1,
                                                                         2, 3,
                                                                         4])),
                                                ('pca', PCA(n_components=3)),
                                                ('linear',
                                                 EstimatorTransformer(estimator=LinearRegression()))])),
                               ('path2',
                                Pipeline(steps=[('select2',
                                                 ColumnSelector(columns=[5, 6,
                                                                         7, 8,
                                                                         9])),
                                                ('pca', PCA(n_components=2)),
                                                ('linear',
                                                 EstimatorTransformer(estimator=LinearRegression()))]))])

path1

<input class="sk-toggleable__control sk-hidden--visually" id="ee9ceae5-59b7-4c88-9fbf-9505349fa887" type="checkbox" >ColumnSelector

ColumnSelector(columns=[0, 1, 2, 3, 4])

<input class="sk-toggleable__control sk-hidden--visually" id="e0129048-6eb8-47bc-a8c0-ae9d96d63cd1" type="checkbox" >PCA

PCA(n_components=3)

<input class="sk-toggleable__control sk-hidden--visually" id="fe8293b5-d914-4a59-8c39-e2b4e18851c8" type="checkbox" >linear: EstimatorTransformer

EstimatorTransformer(estimator=LinearRegression())

<input class="sk-toggleable__control sk-hidden--visually" id="41c53c2a-d2c2-4241-b549-8f7653a0029f" type="checkbox" >LinearRegression

LinearRegression()

path2

<input class="sk-toggleable__control sk-hidden--visually" id="86b9c93e-12be-4fa6-865f-bb874ff99129" type="checkbox" >ColumnSelector

ColumnSelector(columns=[5, 6, 7, 8, 9])

<input class="sk-toggleable__control sk-hidden--visually" id="ec46eb32-4807-466f-a47b-dba193ff5315" type="checkbox" >PCA

PCA(n_components=2)

<input class="sk-toggleable__control sk-hidden--visually" id="5efe2360-3f3b-4c67-bc8f-e2619f06d229" type="checkbox" >linear: EstimatorTransformer

EstimatorTransformer(estimator=LinearRegression())

<input class="sk-toggleable__control sk-hidden--visually" id="6ceb127c-c870-4944-b6c3-5f47c0b5af1b" type="checkbox" >LinearRegression

LinearRegression()

<input class="sk-toggleable__control sk-hidden--visually" id="8d1aaabb-dda3-4fe0-99dc-c4b60597b332" type="checkbox" >ProbWeightRegression

ProbWeightRegression()

You can see that the `ProbWeightRegression` indeeds sums to one.

```python
grid.best_estimator_[1].coefs_
```

    array([0.03102466, 0.96897535])

## Least Absolute Deviation Regression

Imagine that you have a dataset with some outliers.

```python
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(0)
X = np.linspace(0, 1, 20)
y = 3*X + 1 + 0.5*np.random.randn(20)
X = X.reshape(-1, 1)

y[10] = 8
y[15] = 15

plt.figure(figsize=(16, 4))
plt.scatter(X, y)
```

A simple linear regression will not do a good job since it is distracted by the outliers. That is because it optimizes the mean squared error
$$ \sum_i \left(y_i-\textrm{model}(x_i)\right)^2 $$

which penalizes a few large errors more than many tiny errors. For example, if y-model(x) = 4 for some single observation, the MSE here is 16. If there are two observations with y_1 - model(x_1) = 2 and y_2 - model(x_2) = 2, the MSE is 8 in total, which is less than for one larger error. Note that the sum of the errors is the same in both cases.

Hence, linear regression does the following:

```python
from sklearn.linear_model import LinearRegression

x = np.array([0, 1]).reshape(-1, 1)
plt.figure(figsize=(16, 4))
plt.scatter(X, y)
plt.plot(x, LinearRegression().fit(X, y).predict(x), 'r')
```

By changing the loss function to the mean absolute deviation
$$ \sum_i \left|y_i-\textrm{model}(x_i)\right|\enspace, $$

we can let the model put the same focus on each error. This yields the least absolute deviation (LAD) regression that tries to agree with the majority of the points.

```python
from sklearn.linear_model import LinearRegression
from sklego.linear_model import LADRegression

x = np.array([0, 1]).reshape(-1, 1)
plt.figure(figsize=(16, 4))
plt.scatter(X, y)
plt.plot(x, LinearRegression().fit(X, y).predict(x), 'r')
plt.plot(x, LADRegression().fit(X, y).predict(x), 'g')
```

### See also
scikit-learn tackles this problem by offering a variety of [robust regressors](https://scikit-learn.org/stable/auto_examples/linear_model/plot_robust_fit.html). Many of them use an indirect approach to reduce the effect of outliers. [RANSAC](https://en.wikipedia.org/wiki/Random_sample_consensus), for example, samples random points from the dataset until it consists of all inliers.
The closest thing to LADRegression that scikit-learn offers is the [HuberRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.HuberRegressor.html) with a loss function that is partly a squared and partly an absolute error. However, it is more complicated and requires hyperparameter tuning to unleash its full potential.

## QuantileRegression

This is an extension of the LADRegression (see above). While the LADRegression outputs a line that over- and underestimates around 50% of the data, the Quantileregression yields lines with different over- and underestimation shares. This can be used for creating simple confidence intervals around predictions. As an example, consider the following:
1. Create a QuantileRegression with quantile=0.1,
2. create a QuantileRegression with quantile=0.9,

and around 80% of the data is between these two lines.

```python
from skbonus.linear_model import QuantileRegression
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(123)
X = np.arange(100).reshape(-1, 1)
y = 2*X.ravel() + X.ravel()*np.random.standard_cauchy(100)

q_10 = QuantileRegression(quantile=0.1).fit(X, y)
q_90 = QuantileRegression(quantile=0.9).fit(X, y)
lad = QuantileRegression().fit(X, y)

plt.plot(X, y)
plt.plot(X, lad.predict(X))
plt.fill_between(X.ravel(), q_10.predict(X), q_90.predict(X), alpha=0.33, color='orange')
```

```python

```


### Computing for Mathematics - Example individual coursework

This jupyter notebook contains questions that will resemble the questions in your individual coursework.

**Important** Do not delete the cells containing:

```
### BEGIN SOLUTION

### END SOLUTION
```

write your solution attempts in those cells.

Note that this notebook also includes some cells that check your answers.

#### Question 1

For each of the following functions **output** \\(\frac{dy}{dx}\\).

a. \\(y= 2 x (x - 3) (x - 5)\\)

```python
### BEGIN SOLUTION
import sympy as sym
x = sym.Symbol("x")
y = 2 * x * (x - 3) * (x - 5)
sym.diff(y, x)
### END SOLUTION
```

$\displaystyle 2 x \left(x - 5\right) + 2 x \left(x - 3\right) + \left(x - 3\right) \left(2 x - 10\right)$

```python
q1_a_answer = _
feedback_text = """Your output is not a symbolic expression.

You are expected to use sympy for this question.
"""
try:
    assert q1_a_answer.is_algebraic_expr(), feedback_text
except AttributeError:
    print(feedback_text)
```

```python
x = sym.Symbol("x")
expected_answer = 2 * x * (x - 5) + 2 * x * (x - 3) + (x - 3) * (2 * x - 10)
feedback_text = f"""Your answer is not correct.

The expected answer is {expected_answer}."""
assert sym.simplify(q1_a_answer - expected_answer) == 0, feedback_text
```

b. y =\\(\frac{3x ^ 3 + 6 \sqrt{x} + 3) }{ (3  x ^{(1 / 4)})}\\)

```python
### BEGIN SOLUTION
y = (3 * x ** 3 + 6 * sym.sqrt(x) + 3) / (3 * x ** (sym.S(1) / 4))
sym.diff(y, x)
### END SOLUTION
```

$\displaystyle \frac{9 x^{2} + \frac{3}{\sqrt{x}}}{3 \sqrt[4]{x}} - \frac{6 \sqrt{x} + 3 x^{3} + 3}{12 x^{\frac{5}{4}}}$

```python
q1_b_answer = _
feedback_text = """Your output is not a symbolic expression.

You are expected to use sympy for this question.
"""
try:
    assert q1_b_answer.is_algebraic_expr(), feedback_text
except AttributeError:
    print(feedback_text)
```

```python
x = sym.Symbol("x")
expected_answer = sym.diff((3 * x ** 3 + 6 * sym.sqrt(x) + 3) / (3 * x ** (sym.S(1) / 4)), x)
feedback_text = f"""Your answer is not correct.

The expected answer is {expected_answer}."""
assert sym.simplify(q1_b_answer - expected_answer) == 0, feedback_text
```

3. \\(y=2 x (x - 3) (x - 5)\\)

```python
### BEGIN SOLUTION
y = 2 * x * (x - 3) * (x - 5)
sym.diff(y, x)
### END SOLUTION
```

$\displaystyle 2 x \left(x - 5\right) + 2 x \left(x - 3\right) + \left(x - 3\right) \left(2 x - 10\right)$

```python
q1_c_answer = _
feedback_text = """Your output is not a symbolic expression.

You are expected to use sympy for this question.
"""
try:
    assert q1_c_answer.is_algebraic_expr(), feedback_text
except AttributeError:
    print(feedback_text)
```

```python
x = sym.Symbol("x")
expected_answer = sym.diff(2 * x * (x - 3) * (x - 5), x)
feedback_text = f"""Your answer is not correct.

The expected answer is {expected_answer}."""
assert sym.simplify(q1_c_answer - expected_answer) == 0, feedback_text
```

### Question 2

Consider the functions \\(f(x)=-x^3+2x^2+3x\\) and \\(g(x)=-x^3+3x^2-x+3\\).

a. Create a variable `turning_points` which has value the turning points of \\(f(x)\\).

```python
### BEGIN SOLUTION
f = - x ** 3 + 2 * x ** 2 + 3 * x
f_dash = sym.diff(f, x)
turning_points = sym.solveset(f_dash, x)
### END SOLUTION
```

```python
feedback_text = """
Your output is not a sympy Finite Set which is what is expected here.
"""
try:
    assert type(turning_points) is sym.FiniteSet
except NameError:
    print("You did not create a variable called `turning_points`")
```

```python
expected_answer = sym.solveset(- 3 * x ** 2 + 4 * x + 3, x)
feedback_text = f"""Your answer is not correct.

The expected answer is obtained by equating the derivative of f to 0 which gives:

{expected_answer}
"""
assert set(turning_points) == set(expected_answer), feedback_text
```

b. Create a variable `intersection_points` which has value the points where \\(f(x)\\) and \\(g(x)\\) intersect.

```python
### BEGIN SOLUTION
g = - x ** 3 + 3 * x ** 2 - x + 3
intersection_points = sym.solveset(sym.Eq(f, g), x)
### END SOLUTION
```

```python
feedback_text = """
Your output is not a sympy Finite Set which is what is expected here.
"""
try:
    assert type(intersection_points) is sym.FiniteSet
except NameError:
    print("You did not create a variable called `turning_points`")
```

```python
expected_answer = sym.FiniteSet(1, 3)
feedback_text = f"""Your answer is not correct.

The expected answer is obtained by equating the f and g:

{expected_answer}
"""
assert set(intersection_points) == set(expected_answer), feedback_text
```

c. Using your answers to parts b., calculate the area of the region between \\(f\\) and \\(g\\). Assign this value to a variable `area_of_shaded_region`.

```python
### BEGIN SOLUTION
area_of_shaded_region = abs(sym.integrate(f, (x, 1, 3)) - sym.integrate(g, (x, 1, 3)))
area_of_shaded_region
### END SOLUTION
```

$\displaystyle \frac{4}{3}$

```python
feedback_text = "Your output is not a sympy rational which is expected here"
try:
    assert type(area_of_shaded_region) is sym.Rational, feedback_text
except NameError:
    print("You did not create a variable called `area_of_shaded_region`")
```

```python
expected_answer = sym.S(4) / 3
feedback_text = f"""The expected answer is {expected_answer} which is obtained by
calculating the definite integral of |f - g| between 1 and 3.

"""
assert float(area_of_shaded_region) == float(expected_answer), expected_answer
```

### Question 3

Three letters are selected at random from the 8 letters of the word `MONGOLIA`, without regard to order.

a. Create a variable `number_of_selections` with value the number of possible selections of 4 letters.

```python
import itertools
### BEGIN SOLUTION
letters = "MONGOLIA"
words = list(itertools.combinations(letters, 4))
number_of_selections = len(words)
### END SOLUTION
```

```python
expected_answer = 70
feedback_text = f"""The expected number of selections is {expected_answer}.

This is obtained using the `itertools.combinations` function."""
try:
    assert number_of_selections == expected_answer, feedback_text
except NameError:
    print("You did not create a variable `number_of_selections`")
```

b. Create a variable `probability_of_selecting_N` with value the probability that the letter `N` is included in the selection.

```python
### BEGIN SOLUTION
probability_of_selecting_N = sum('N' in word for word in words) / len(words)
### END SOLUTION
```

```python
expected_answer = 35 / 70
feedback_text = f"""The expected probability is {expected_answer}.

This is obtained using a conditional summation over the generated words."""
try:
    assert float(probability_of_selecting_N) == float(expected_answer), feedback_text
except NameError:
    print("You did not create a variable `probability_of_selecting_P`")
```

c. letters are now selected at random, one at a time, from the 8 letters of the word `MONGOLIA`, and are placed in order in a line. Create a variable `probability_of_selecting_GOAL` with value the probability that the 4 letters can form the word `GOAL`.

```python
### BEGIN SOLUTION
words = list(itertools.permutations(letters, 4))
probability_of_selecting_GOAL = sum((["A", "G", "L", "O"] == sorted(word)) for word in words) / len(words)
### END SOLUTION
```

```python
expected_answer = 48 / 1680
feedback_text = f"""The expected probability is {expected_answer}.

This is obtained using a conditional summation over the words generated using `itertools.permutations`."""
try:
    assert np.isclose(probability_of_selecting_GOAL, expected_answer), feedback_text
except NameError:
    print("You did not create a variable `probability_of_selecting_TOP`")
```

    You did not create a variable `probability_of_selecting_TOP`

### Question 4

A sequence is given by:

\\[
    x_1 = 1\\
    x_{n + 1}= x_n(p + x_n)
\\]

for \\(p\ne0\\).

a. Define a python function `generate_x(n, p)` that returns the terms of that sequence.

```python
def generate_x(n, p):
    ### BEGIN SOLUTION
    """
    Gives the nth term of the sequence for a given value of p
    """
    if n == 1:
        return 1
    previous_x = generate_x(n - 1, p)
    return previous_x * (p + previous_x)
    ### END SOLUTION
```

```python
feedback_text = """You did not include a docstring. This is important to help document your code.

It is done  using triple quotation marks. For example:

def get_remainder(m, n):
    \"\"\"
    This function returns the remainder of m when dividing by n
    \"\"\"
    ...

Using that it's possible to access the docstring,
one way to do this is to type: `get_remainder?`
(which only works in Jupyter) or help(get_remainder).

We can also comment code using `#` but this is completely
ignored by Python so cannot be accessed in the same way.

"""
try:
    assert  generate_x.__doc__ is not None, feedback_text
except NameError:
    print("You did not create a variable called `area_of_shaded_region`")
```

```python
try:
    assert generate_x(n=1, p=2) == 1, f"Your function did not give the expected answer for n=1, p=2"
    assert generate_x(n=1, p=1) == 1, f"Your function did not give the expected answer for n=1, p=2"
    assert generate_x(n=5, p=1) == 1806, f"Your function did not give the expected answer for n=5, p=1"
    assert generate_x(n=5, p=2) == 65535, f"Your function did not give the expected answer for n=5, p=2"
except NameError:
    print("You did not create a variable called `area_of_shaded_region`")
```

b. **Output** an expression for \\(x_2\\) in terms of \\(p\\).

```python
### BEGIN SOLUTION
p = sym.Symbol("p")
generate_x(n=2, p=p)
### END SOLUTION
```

$\displaystyle p + 1$

```python
q4_a_answer = _
feedback_text = """Your output is not a symbolic expression.

You are expected to use sympy for this question.
"""
try:
    assert q4_a_answer.is_algebraic_expr(), feedback_text
except AttributeError:
    print(feedback_text)
```

```python
p = sym.Symbol("p")
expected_answer = p + 1
feedback_text = f"""Your answer is not correct.

The expected answer is {expected_answer}."""
assert sym.simplify(q4_a_answer - expected_answer) == 0, feedback_text
```

b. **Output** an expression for \\(x_3\\) in terms of \\(p\\)

```python
### BEGIN SOLUTION
generate_x(3, p=p)
### END SOLUTION
```

$\displaystyle \left(p + 1\right) \left(2 p + 1\right)$

```python
q4_b_answer = _
feedback_text = """Your output is not a symbolic expression.

You are expected to use sympy for this question.
"""
try:
    assert q4_b_answer.is_algebraic_expr(), feedback_text
except AttributeError:
    print(feedback_text)
```

```python
p = sym.Symbol("p")
expected_answer = 2 * p ** 2 + 3 * p + 1
feedback_text = f"""Your answer is not correct.

The expected answer is {expected_answer}."""
assert sym.simplify(q4_b_answer - expected_answer) == 0, feedback_text
```

c. **Output** the values of \\(p\\) for which \\(x_3=1\\).

```python
### BEGIN SOLUTION
sym.solveset(sym.Eq(generate_x(3, p=p), 1), p)
### END SOLUTION
```

$\displaystyle \left\{- \frac{3}{2}, 0\right\}$

```python
q4_c_answer = _
feedback_text = """
Your output is not a sympy Finite Set which is what is expected here.
"""
assert type(q4_c_answer) is sym.FiniteSet, feedback_text
```

d. Using the non zero value of \\(p\\) calculated in the previous question calculate \\(x_n\\) for \\(n\in\{1, 2, 3, 4, 5, 6, 7, 8, 9, 100, 200, 300\}\\) and create a variable `range_of_values` with minimum and maximum value or \\(x_n\\).

```python
values_of_n = [1, 2, 3, 4, 5, 6, 7, 8, 9, 100, 200, 300]
### BEGIN SOLUTION
values = [generate_x(n=n, p=-sym.S(3) / 2) for n in values_of_n]
range_of_values = min(values), max(values)
### END SOLUTION
```

```python
import numpy as np
expected_answer = (- sym.S(1) / 2, 1)
expected_answer = np.array(sorted(expected_answer), dtype=np.float64)
range_of_values = np.array(sorted(range_of_values), dtype=np.float64)
feedback_text = f"""Your answer is not correct, the expected answer is:

{expected_answer}
"""
try:
    assert np.allclose(expected_answer, range_of_values), feedback_text
except NameError:
    print("You did not create a variable called `all_values_of_n`")
```


# Differentiation
This notebook will take you through mathematical differentiation, and the way that it relates the previously discussed Position, Velocity and Acceleration.

```python
from sympy import *
from sympy.plotting import plot
```

## Calculus

Differentiation is one half of the mathematical practice of calculus. This is all to do with rates of change, and areas covered. The two are fundamentally related, which will become apparent as we move through the notebook.

To start with, lets have our customary stolen graph to talk around.

Most of the above graph should be familiar. An _x_ and _y_ axis, a line on the graph, a region of space outlined by the _x_ axis, vertical lines at _a_ and _b_ and the aformentioned line. And finally an example line drawn to represent the slope.

## Part 1: Differentiations, or _Fancy Slopes_

One of the new symbols is the dy/dx part. This is basically saying, [The Rate of change of y with respect to x at that point in x]. As you can see there is a straight line which can touch a point on the line, and show the gradient of the line _at that point_. The key difference here from the gradients you've worked with before is that these can change. As you move along x, there is a different straight line which can describe the gradient at that point.

So dy/dx is a variable with x, just like y is. You could draw out the line for dy/dx manually, if you were to put a line at every point on the graph and work out the gradient at that point.

However that sounds like a lot of work. And we are lazy. so we can just get python to do it for us.

Start by considering a simple example. the quadratic graph.

```python
x = symbols('x')
# This is to tell python that x is no longer just an object, or a name for something.
# It is an algebraic symbol, like we use in equations

y = (x*x - 1)
p1 = plot(y, ylabel = "y")
```

    <Figure size 640x480 with 1 Axes>

```python
p2 = plot(diff(y, x), ylabel = "dy/dx")
```

Hopefully you can see that there is a sector of the quadratic graph where the gradient is positive, and increasing. and another section which is negative and decreasing. Those gradients are then mirrored in the 'derivative' of the quadratic function.

### Exercise 1.1

Play around with the provided quadratic equation, to prove to yourself that the intercept term is irrelevant.

## Example 1.2: dydx as a function

In the above, we have used `diff(y,x)` as a way of saying dy/dx. This then gives us the rate of change of y with respect to x. we jumped ahead to a graph, since that made the discussion easier. But the actual output for the function is a bit simpler.

```python
diff(y, x)
```

$\displaystyle 2 x$

`diff(y,x)` actually provides us with the function to calculate the gradient at every point on the graph. In the case of `2x` then the gradient at the point `x = 1` is 2. If the gradient was `4.5x` then the gradient at the point `x = 1` is 4.5

### Exercise 1.2  Gradient of an arbitrary function.

Take the example of the function `y = x^3 - x^2`. What does it look like. and what is the equation of its gradient

```python
## Defining the function
y_test = x*x*x - 7*x*x
```

```python
## Plotting the function
test_plot = plot(y_test, xlim = [-5, 10], ylim = [-200, 200])
```

```python
## Calculating the derivative function
diff(y_test, x)
```

$\displaystyle 3 x^{2} - 14 x$

```python
## plotting it to prove it to ourselves
test_diff_plot = plot(diff(y_test, x), xlim = [-5, 10])
```

## Example 1.2

How does this relate to our examples before.

Critically, the x axis here does not have to be the position in x. it could also be time.

if you had an equation. y = 2t. where y is the position along a line, and t is the time. then you have an equation which can be differentiated.

```python
t = symbols('t')
# like with x. we need to tell python that t is no longer a normal name for something
```

```python
example_y = 2*t
example_y_plot = plot(example_y)
```

In this example, we have a plot of the position. but we are able to plot the velocity. by simply differentiating the equation for position

```python
plot(diff(example_y, t), ylim = [0, 4], axis_center = (0, 0))
```

we could also try to get the acceleration by differentiating again. However....

```python
diff(example_y, t, 2)
```

$\displaystyle 0$

The acceleration here is zero. Which is OK, since we can tell that the velocity is constant, which would imply no accelerations

### Playground

Now we want to play around with plotting functions and their differentials

```python
y_test = 3*x + 20
y_plot = plot(y_test, axis_center = (0, 0))
```

```python
dydx_plot = plot(diff(y_test, x), axis_center = (0, 0))
```

```python
d2ydx2_plot = plot(diff(y_test, x, 2), axis_center = (0, 0))
```

```python

```

```python

```