This is a notebook explaining the mathematics of Linear and Logistic Regression. It is adapted from by [blog](http://ataspinar.com/2016/03/28/regression-logistic-regression-and-maximum-entropy). All of the code is available on [GitHub](https://github.com/taspinar/siml).

 1. Regression Analysis

Regression Analysis is the field of mathematics where the goal is to find a function which best [correlates](http://guessthecorrelation.com/) with a dataset. Let's say we have a dataset containing $ n $ datapoints;

$ X = ( x^{(1)}, x^{(2)}, .., x^{(n)} ) $.

For each of these (input) datapoints there is a corresponding (output) $ y^{(i)}$-value.

Here, the $ x$-datapoints are called the [independent variables](https://en.wikipedia.org/wiki/Dependent_and_independent_variables) and $ y$ the dependent variable;
the value of $ y^{(i)} $ depends on the value of $ x^{(i)} $, while the value of $ x^{(i)}$ may be freely chosen without any restriction imposed on it by any other variable.

The goal of Regression analysis is to find a function $ f(X) $ which can best describe the correlation between $ X $ and $ Y $. In the field of Machine Learning, this function is called the hypothesis function and is denoted as $ h_{\theta}(x) $. If we can find such a function, we can say we have successfully built a Regression model.

If the input-data lives in a 2D-space, this boils down to finding a curve which fits through the data points. In the 3D case we have to find a plane and in higher dimensions a hyperplane.

To give an example, let's say that we are trying to find a predictive model for the success of students in a course called Machine Learning. We have a dataset $ Y $ which contains the final grade of $ n $ students. Dataset $ X $ contains the values of the independent variables. Our initial assumption is that the final grade only depends on the studying time. The variable $ x^{(i)} $ therefore indicates how many hours student $ i $ has studied. The first thing we would do is visualize this data:

If the results looks like the figure on the left, then we are out of luck. It looks like the points are distributed randomly and there is no correlation between $ Y$ and $ X$ at all. However, if it looks like the figure on the right, there is probably a strong correlation and we can start looking for the function which describes this correlation.

This function could for example be:

$ h_{\theta}(X) =  \theta_0+ \theta_1 \cdot x $

or

$ h_{\theta}(X) = \theta_0 + \theta_1 \cdot x^2 $

where $ \theta $ are the dependent parameters of our model.

 1.2 Multivariate Regression

In evaluating the results from the previous section, we may find the results unsatisfying; the function does not correlate with the datapoints strongly enough. Our initial assumption is probably not complete. Taking only the studying time into account is not enough.
The final grade does not only depend on the studying time, but also on how much the students have slept the night before the exam. Now the dataset contains an additional variable which represents the sleeping time.

Our dataset is then given by $ X = ( (x_1^{(1)}, x_2^{(1)}), (x_1^{(2)}, x_2^{(2)}), .., (x_1^{(n)}, x_2^{(n)}) ) $. In this dataset $ x_1^{(i)} $ indicates how many hours student $ i $ has studied and $ x_2^{(i)} $ indicates how many hours he has slept.

This is an example of [Multivariate Regression](https://en.wikipedia.org/wiki/Linear_regression#Simple_and_multiple_regression). The function has to include both variables. For example:

$ h_{\theta}(x) = \theta_0 + \theta_1 \cdot x_1 + \theta_2 \cdot x_2$

or

$ h_{\theta}(x) = \theta_0 + \theta_1 \cdot x_1 + \theta_2 \cdot x_2^3 $.

 1.3 Linear vs Non-Linear Regression

All of the above examples are examples of linear regression. We have seen that in some cases $ y^{(i)} $ depends on a linear form of $ x^{(i)} $, but it can also depend on some power of $ x^{(i)}$, or on the log or any other form of $ x^{(i)}$. However, in all cases its dependence on the parameters $\theta$ was linear.

So, what makes linear regression linear is not that $ Y$ depends in a linear way on $ X $, but that it depends in a linear way on $ \theta$.
$ Y $ needs to be [linear](http://www.investopedia.com/terms/l/linearrelationship.asp) with respect to the model-parameters $ \theta $. Mathematically speaking it needs to satisfy the [superposition principle](http://www.cut-the-knot.org/do_you_know/superposition.shtml).

Examples of nonlinear regression would be:

$ h_{\theta}(x) = \theta_0 + x_1^{\theta_1} $

or

$ h_{\theta}(x) = \theta_0 + \theta_1 / x_1 $

The reason why the distinction is made between linear and nonlinear regression is that nonlinear regression problems are more difficult to solve and therefore more computational intensive algorithms are needed.

Linear regression models can be written as a linear system of equations, which can be solved by finding the closed-form solution $ \theta = ( X^TX )^{-1}X^TY $ with Linear Algebra. See these statistics notes for more on solving linear models with linear algebra.

As discussed before, such a closed-form solution can only be found for linear regression problems. However, even when the problem is linear in nature, we need to take into account that calculating the inverse of a $ n $ by $ n $ matrix has a time-complexity of $ O(n^3) $. This means that for large datasets ($ n \gt 10.000 $ ) finding the closed-form solution will take more time than solving it iteratively (gradient descent method) as is done for nonlinear problems. So solving it iteratively is usually preferred for larger datasets, even if it is a linear problem.

 1.4 Gradient Descent

The Gradient Descent method is a general optimization technique in which we try to find the value of the parameters $ \theta $ with an iterative approach.

First, we construct a cost function (also known as loss function or error function) which gives the difference between the values of the hypothesis function $h_{\theta}(x)$ (the values you expect $ Y$ to have with the current values of $ \theta$ ) and the actual values of $ Y $. The better your estimation of $ \theta $ is, the better the values of $ h_{\theta}(x) $ will approach the values of $ Y$ and the smaller the cost function will be.

Usually, the cost function is expressed as the squared error of the difference between these functions:

$ J(x) = \frac{1}{2n} \sum_i^n ( h_{\theta}(x^{(i)}) - y^{(i)} )^2 $

At each iteration we choose new values for the parameters $ \theta$, and move towards the 'true' values of these parameters, i.e. the values which make this cost function as small as possible. The direction in which we have to move is the negative gradient direction;

 $ \Delta\theta = - \alpha \frac{d}{d\theta} J(x) $.

The reason for this is that  a function's value decreases fastest if we move towards the direction of the negative gradient (the directional derivative is maximal in the direction of the gradient).

Taking all this into account, this is how gradient descent works:

Make an initial but intelligent guess for the values of the parameters $ \theta $.

Keep iterating while the value of the cost function has not met your criteria:

With the current values of $ \theta $, calculate the gradient of the cost function J  ( $ \Delta \theta = - \alpha \frac{d}{d\theta} J(x)$ ).

Update the values for the parameters $ \theta := \theta + \alpha \Delta \theta $

Fill in these new values in the hypothesis function and calculate again the value of the cost function;

Just as important as the initial guess of the parameters is the value you choose for the learning rate $ \alpha $. This learning rate determines how fast you move along the slope of the gradient. If the selected value of this learning rate is too small, it will take too many iterations before you reach your convergence criteria. If this value is too large, you might overshoot and not converge.

 2. Logistic Regression

Logistic Regression is similar to (linear) regression, but adapted for the purpose of classification. The difference is small; for Logistic Regression we also have to iteratively apply the gradient descent method to estimate the values of the parameter $ \theta$. And again, during the iteration, the values are estimated by taking the gradient of the cost function. And also, the cost function is given by the squared error of the difference between the hypothesis function $ h_{\theta}(x)$ and $ Y$. The major difference however, is the hypothesis function itself.

In order to understand the hypothesis function of Logistic Regression, we must first understand the idea behind classification.

When you want to classify something, there are a limited number of classes it can belong to. And for each of these possible classes there can only be two states for $ y^{(i)} $;
either $ y^{(i)} $ belongs to the specified class and $ y = 1 $, or it does not belong to the class and $ y = 0 $. Even though the output values $ Y $ are binary, the independent variables $ X $ are still continuous. So, we need a function which has as input a large set of continuous variables $ X $ and for each of these variables produces a binary output. This function, the hypothesis function, has the following form:

$ h_{\theta} = \frac{1}{1 + \exp(-z)} = \frac{1}{1 + \exp(-\theta x)} $.

This function is also known as the logistic function, which is a part of the sigmoid function family. These functions are widely used in the natural sciences because they provide the simplest model for population growth. However, the reason why the logistic function is used for classification in Machine Learning is its 'S-shape'.

As you can see this function is bounded in the y-direction by 0 and 1. If the variable $ z$ is very negative, the output function will go to zero (it does not belong to the class). If the variable $ z$ is very positive, the output will be one and it does belong to the class.
(Such a function is called an indicator function.)

The question then is, what will happen to input values which are neither very positive nor very negative, but somewhere 'in the middle'. We have to define a decision boundary, which separates the positive from the negative class. Usually this decision boundary is chosen at the middle of the logistic function, namely at $ z = 0 $ where the output value $ y$ is $ 0.5$.

\begin{equation}
y =\begin{cases}
1, \text{if $z \gt  0 $}.\\
0, \text{if $z \lt 0 $}.
\end{cases}
\end{equation}

As we can see in the formula of the logistic function, $ z = \theta \cdot x $. Meaning, the dependent parameter $ \theta$ (also known as the feature), maps the input variable $ x$  to a position on the $ z$-axis. With its $ z$-value,  we can use the logistic function to calculate the $ y$ -value. If this $ y$-value $ \gt 0.5 $ we assume it does belong in this class and vice versa.

So the feature $ \theta $ should be chosen such that it predicts the class membership correctly. It is therefore essential to know which features are useful for the classification task. Once the appropriate features are selected , gradient descent can be used to find the optimal value of these features.

How can we do gradient descent with this logistic function? Except for the hypothesis function having a different form, the gradient descent method is exactly the same. We again have a cost function, of which we have to iteratively take the gradient w.r.t. the feature $ \theta $ and update the feature value at each iteration.

This cost function is given by

\begin{equation}
\begin{split}
J(x) = -\frac{1}{2n} \sum_i^n \left(  y^{(i)} log( h_{\theta}(x^{(i)})) + (1-y^{(i)})log(1-h_{\theta}(x^{(i)})) \right) \\
 = -\frac{1}{2n} \sum_i^n \left( y^{(i)} log(\frac{1}{1+exp(-\theta x)}) + (1-y^{(i)})log(1-\frac{1}{1+exp(-\theta x)}) \right)
\end{split}
\end{equation}

We know that:
$ log(\frac{1}{1+exp(-\theta x)}) = log(1) - log(1+exp(-\theta x)) = - log(1+exp(-\theta x))$

and

$ log(1-\frac{1}{1+exp(-\theta x)}) = log( \frac{exp(-\theta x)}{1+exp(-\theta x)}) $

$ = log(exp(-\theta x)) - log(1+exp(-\theta x)) $

$ = -\theta x^{(i)} -  log(1+exp(-\theta x)) $

Plugging these two equations back into the cost function gives us:

\begin{equation}
\begin{split}
J(x)  = - \frac{1}{2n} \sum_i^n \left( - y^{(i)} log(1+exp(-\theta x)) - (1-y^{(i)})(\theta x^{(i)} +  log(1+exp(-\theta x))) \right) \\
 = - \frac{1}{2n} \sum_i^n \left(  y^{(i)} \theta x^{(i)} -\theta x^{(i)} -log(1+exp(-\theta x)) \right)
\end{split}
\end{equation}

The gradient of the cost function with respect to $ \theta $ is given by

\begin{align}
\frac{d}{d\theta} J(x) = - \frac{1}{2n} \sum_i^n \left(  y^{(i)} x^{(i)} - x^{(i)} + x^{(i)} \frac{ exp(-\theta x)}{1+exp(-\theta x)} \right) \\
 = - \frac{1}{2n} \sum_i^n \left( x^{(i)} ( y^{(i)} - 1 +\frac{exp(-\theta x)}{1+exp(-\theta x)} ) \right) \\
 = - \frac{1}{2n} \sum_i^n  \left( x^{(i)} ( y^{(i)} - \frac{1}{1+exp(-\theta x)} ) \right) \\
 = - \frac{1}{2n} \sum_i^n  \left( x^{(i)} ( y^{(i)} - h_{\theta}(x^{(i)}) )\right)
\end{align}

So the gradient of the seemingly difficult cost function, turns out to be a much simpler equation. And with this simple equation, gradient descent for Logistic Regression is again performed in the same way:

Make an initial but intelligent guess for the values of the parameters $ \theta $.

Keep iterating while the value of the cost function has not met your criteria:

With the current values of $ \theta $, calculate the gradient of the cost function J  ( $ \Delta \theta = - \alpha \frac{d}{d\theta} J(x)$ ).

Update the values for the parameters $ \theta := \theta + \alpha \Delta \theta $

Fill in these new values in the hypothesis function and calculate again the value of the cost function;

 2.2 Logistic Regression - Example 1

Now that we have looked at the theory, let's lets have a look at an example of Logistic Regression. We will start out with the example of students passing a course or not.

Let's generate some data points. There are $n = 300$ students participating in the course Machine Learning and whether a student $ i $ passes ( $ y_i = 1 $) or not ( $ y_i = 0$ ) depends on two variables;

$x_i^{(1)}$ : how many hours student $ i $ has studied for the exam.

$x_i^{(2)}$ : how many hours student $ i $ has slept the day before the exam.

In our example, the results are pretty binary; everyone who has studied less than 4 hours fails the course, as well as everyone whose studying time + sleeping time is less than or equal to 13 hours ($x_i^{(1)} + x_i^{(2)} \leq 13$).

```python
import random
import numpy as np

def func2(x_i):
    if x_i[1]+x_i[2] >= 13:
        return 0
    else:
        return 1

def generate_data2(no_points):
    X = np.zeros(shape=(no_points, 3))
    Y = np.zeros(shape=no_points)
    for ii in range(no_points):
        X[ii][0] = 1
        X[ii][1] = random.random()*9+0.5
        X[ii][2] = random.random()*9+0.5
        Y[ii] = func2(X[ii])
    return X, Y

X, Y = generate_data2(300)

```

The results looks like this (the green dots indicate a pass and the red dots a fail)

```python
import math
import numpy as np

def z_to_y(z_i):
    return 1 if z_i > 0.5 else 0

def determine_correct_guesses(X, Y, theta, m):
    determined_z = [np.dot(theta, X[ii]) for ii in range(m)]
    determined_Y = [z_to_y(elem) for elem in determined_z]
    correct_guesses = [y_i - det_y_i for y_i, det_y_i in zip(Y, determined_Y)]
    return correct_guesses

def hypothesis(theta, x_i):
    z = np.dot(theta, x_i)
    sigmoid = 1.0 / (1.0 + math.exp(-1.0*z))
    return sigmoid

def gradient_descent(X, Y, theta, alpha, m, number_of_iterations):
    for iter in range(0,number_of_iterations):
        cost = (-1.0/m)*sum([Y[ii]*math.log(hypothesis(theta, X[ii]))+(1-Y[ii])*math.log(1-hypothesis(theta, X[ii])) for ii in range(m)])
        grad = (-1.0/m)*sum([X[ii]*(Y[ii]-hypothesis(theta, X[ii])) for ii in range(m)])
        theta = theta - alpha * grad
        correct_guesses = determine_correct_guesses(X, Y, theta, m).count(0)
        if iter % 100 == 0: print("iteration %s : cost %s : correct_guesses %s" % (iter, cost, correct_guesses))
    return theta

numIterations = 1500
alpha = 0.6
m,n = np.shape(X)
theta = np.ones(n)
theta = gradient_descent(X, Y, theta, alpha, m, numIterations)

```

    iteration 0 : cost 3.12146529928 : correct_guesses 241
    iteration 100 : cost 0.852070281571 : correct_guesses 107
    iteration 200 : cost 0.852743840457 : correct_guesses 241
    iteration 300 : cost 0.168394744623 : correct_guesses 225
    iteration 400 : cost 1.84940208932 : correct_guesses 257
    iteration 500 : cost 1.26800878408 : correct_guesses 222
    iteration 600 : cost 0.97047975629 : correct_guesses 214
    iteration 700 : cost 0.780941221589 : correct_guesses 217
    iteration 800 : cost 0.571828461512 : correct_guesses 227
    iteration 900 : cost 0.353647771742 : correct_guesses 247
    iteration 1000 : cost 0.203740088896 : correct_guesses 261
    iteration 1100 : cost 0.140443855068 : correct_guesses 268
    iteration 1200 : cost 0.109774575449 : correct_guesses 275
    iteration 1300 : cost 0.0910747686516 : correct_guesses 280
    iteration 1400 : cost 0.078624219286 : correct_guesses 285

 2.3 Logistic Regression - Example 2: Medical Data

Now  that the concept of Logistic Regression is a bit more clear, let's classify real-world data!
The University of Massachusetts provides some datasets which are ideal to perform Logistic Regression on. They are small (so my small laptop can also perform it in a reasonable amount of time) and there are various datasets with different (amount of) features.

The dataset "myopia.dat" contains the medical data of 618 subjects, and has 15 features describing the characteristics of each subject. We can read this data in as follows:

```python
datafile = '../datasets/myopia.dat'
file = open(datafile, 'r')
no_points = 618
X = np.zeros(shape=(no_points, 16))
Y = np.zeros(shape=no_points)
rownum = 0

for line in file:
    line = line.split()
    Y[rownum] = int(line[2])
    X[rownum][0] = 1
    X[rownum][1] = int(line[3])/10.0
    X[rownum][2] = int(line[4])
    X[rownum][3] = float(line[5])
    X[rownum][4] = float(line[6])/10.0
    X[rownum][5] = float(line[7])/10.0
    X[rownum][6] = float(line[8])/10.0
    X[rownum][7] = float(line[9])/10.0
    X[rownum][8] = int(line[10])/10.0
    X[rownum][9] = int(line[11])/10.0
    X[rownum][10] = int(line[12])/10.0
    X[rownum][11] = int(line[13])/10.0
    X[rownum][12] = int(line[14])/10.0
    X[rownum][13] = int(line[15])/10.0
    X[rownum][14] = int(line[16])
    X[rownum][15] = int(line[17])
    rownum+=1

numIterations = 1500
alpha = 0.5
m,n = np.shape(X)
theta = np.ones(n)
theta = gradient_descent(X, Y, theta, alpha, m, numIterations)

```

    iteration 0 : cost 11.9698579441 : correct_guesses 81
    iteration 100 : cost 0.271623687304 : correct_guesses 543
    iteration 200 : cost 0.255773830584 : correct_guesses 550
    iteration 300 : cost 0.251054357413 : correct_guesses 551
    iteration 400 : cost 0.249151329807 : correct_guesses 552
    iteration 500 : cost 0.2482092657 : correct_guesses 553
    iteration 600 : cost 0.247665724849 : correct_guesses 553
    iteration 700 : cost 0.247315253506 : correct_guesses 553
    iteration 800 : cost 0.247071379562 : correct_guesses 553
    iteration 900 : cost 0.24689286954 : correct_guesses 553
    iteration 1000 : cost 0.246757673012 : correct_guesses 553
    iteration 1100 : cost 0.246652753284 : correct_guesses 553
    iteration 1200 : cost 0.246569757092 : correct_guesses 553
    iteration 1300 : cost 0.246503005543 : correct_guesses 553
    iteration 1400 : cost 0.24644847617 : correct_guesses 553

While reading in, all values are normalized to a value around 1. This is done in order to speed up the calculations and to ensure that you never take the logarithm of a zero value (log functions don't really like that).

Once the data has been read into the $X$ and $Y$ matrices, logistic regression can be applied. This simple algorithm for logistic regression correctly classifies ~550 of the 618 subjects, giving it an accuracy of ~90%.

```python

```

notebooks/Linear Regression, Logistic Regression.ipynb


```
%matplotlib inline
```

#Artigo: Morphology and metallicity of the Small Magellanic Cloud using RRab stars - Sukanta Deb

link: http://arxiv.org/pdf/1502.05824v1.pdf

# Estrutura da SMC

Para determinar a estrutura, o Sukanta converteu a ascensão reta ($\alpha$), declinação ($\delta$) e distância ($D$) nas coordenadas cartesianas. $(\alpha , \delta , D) \to (x,y,z)$.

O sistema cartesiano $(x,y,z)$ é centrado no centro da SMC $(\alpha_0, \delta_0 , D_0)$. O eixo $z$ aponta na direção do observador, o eixo $x$ é antiparalelo ao eixo $\alpha$ e o eixo $y$ é paralelo ao eixo $\delta$.

Possuindo os valores do centro da SMC (o sukanta pegou de um artigo), as coordenadas $x, y$ e $z$ são obtidas a partir das seguintes transformações:
\begin{equation}
x = -D \sin (\alpha - \alpha_0) \cos(\delta) \\
y = D \sin (\delta) \cos (\delta_0) − D \sin (\delta_0) \cos (\alpha - \alpha_0) \cos( \delta) \\
z = D_0 - D \sin(\delta)\sin(\delta_0) - D\cos(\delta_0)\cos(\alpha) - \alpha_0 \cos (\delta)
\end{equation}

essas transformações foram obtidas do artigo neste link http://arxiv.org/pdf/astro-ph/0105339v2.pdf

```

```

notebooks/Resumo - metalicidade e morfologia da SMC, sukanta deb.ipynb


# Linear Support Vector Machine (SVM)

We've now seen how to optimise analytic functions using PyTorch's optimisers, and in the previous labs and exercises we played with training simple machine learning models with hand-coded gradient descent. Let's put everything together and implement a Soft-Margin Linear Support Vector Machine, which we'll train on some artifically generated data using a range of optimisers.

```python
# We're going to use a library called celluloid to make animations that work on colab
try:
    from celluloid import Camera
except:
    !pip install celluloid

from IPython.display import HTML
import torch
import torch.optim as optim
```

## SVM Recap

Recall that an SVM tries to find the maximum margin hyperplane which separates the data classes. For a soft margin SVM
where $\textbf{x}$ is our data, we minimize:

\begin{equation}
\left[\frac 1 n \sum_{i=1}^n \max\left(0, 1 - y_i(\textbf{w}\cdot \textbf{x}_i - b)\right) \right] + \lambda\lVert \textbf{w} \rVert^2
\end{equation}

We can formulate this as an optimization over our weights $\textbf{w}$ and bias $b$, where we minimize the
hinge loss subject to a level 2 weight decay term. The hinge loss for some model outputs
$z = \textbf{w}\textbf{x} + b$ with targets $y$ is given by:

\begin{equation}
\ell(y,z) = \max\left(0, 1 - yz \right)
\end{equation}

First, complete the following function to implement the hinge loss for batches of predictions `y_pred` and targets `y_true`. You should return the mean of the hinge loss across the batch. Note that this is a binary problem with labels are chosen to be $\{-1,1\}$.

```python
def hinge_loss(y_pred, y_true):
    return torch.mean(torch.clamp(1 - y_pred * y_true, min=0))
    raise NotImplementedError()
```

## Defining the SVM

Defining the SVM is pretty simple - it's just a basic linear classifier like a Perceptron; what distinguishes it is the loss.  We'll wrap it up in a function:

```python
def svm(x, w, b):
    h = (w*x).sum(1) + b
    return h
```

Creating Synthetic Data
-----------------------------------------------

Now for some data, 1024 samples should do the trick. We normalise here so that our random init is in the same space as
the data:

```python
import numpy as np
from sklearn.datasets.samples_generator import make_blobs

X, Y = make_blobs(n_samples=1024, centers=2, cluster_std=1.2, random_state=1)
X = (X - X.mean()) / X.std()
Y[np.where(Y == 0)] = -1
X, Y = torch.FloatTensor(X), torch.FloatTensor(Y)
print(X.shape)
print(Y.shape)
```

    torch.Size([1024, 2])
    torch.Size([1024])

For the first time, we're going to do proper mini-batch gradient descent. As such, we actually need to be able to produce batches of data. PyTorch has the concept of datasets (which represent entire collections of data) and data loaders (which allow us to iterate batches of data from a dataset). This allows the framework to do all the hard work for us:

```python
from torch.utils import data

dataset = data.TensorDataset(X,Y) # create your datset
dataloader = data.DataLoader(dataset, batch_size=32, shuffle=True) # create your dataloader
```

Visualizing the Training
----------------------------------------

We now aim to create a nice visualisation, such as the one below, that shows what happens as our SVM learns.

The code for the visualisation (using [pyplot](https://matplotlib.org/api/pyplot_api.html)) is a bit ugly but we'll
try to explain it to some degree. First, we need a mesh grid `xy` over the range of our data:

```python
delta = 0.01
x = np.arange(X[:, 0].min(), X[:, 0].max(), delta)
y = np.arange(X[:, 1].min(), X[:, 1].max(), delta)
x, y = np.meshgrid(x, y)
xy = list(map(np.ravel, [x, y]))
```

Now things get a little strange. We start by evaluating our model over the mesh grid from earlier.

For our outputs $z \in \textbf{Z}$, we can make some observations about the decision boundary. First, that we are
outside the margin if $z \lt -1$ or $z \gt 1$. Conversely, we are inside the margine where $z \gt -1$
or $z \lt 1$.

This whole process is shown in the function below, which we can call at the end of every epoch. The `camera` takes snapshots of the current plot and is used later to render a video.

```python
import matplotlib
import matplotlib.pyplot as plt

def draw_margin(w, b, camera):
    w = w.data.numpy()
    b = b.data.numpy()

    z = (w.dot(xy) + b).reshape(x.shape)
    z[np.where(z > 1.)] = 4
    z[np.where((z > 0.) & (z <= 1.))] = 3
    z[np.where((z > -1.) & (z <= 0.))] = 2
    z[np.where(z <= -1.)] = 1

    plt.scatter(x=X[:, 0], y=X[:, 1], c="black", s=10)
    plt.contourf(x, y, z, cmap=plt.cm.jet, alpha=0.5)
    camera.snap()
```

Since we don't know that our data is linearly separable, we would like to use a soft-margin SVM. That is, an SVM for
which the data does not all have to be outside of the margin. This takes the form of a weight decay term,
$\lambda\lVert \textbf{w} \rVert^2$ in the above equation. This term is called weight decay because the gradient
corresponds to subtracting some amount ($2\lambda\textbf{w}$) from our weights at each step.

Most PyTorch optimisers actually have weight decay built in to them as an option (`weight_decay=...`), so its trivial to incorporate this.

At this point we are ready to create and train our model. We've written most of the code, but you'll need to implement the forward and backward pass:

```python
# Set up drawing
fig = plt.figure(figsize=(5, 5))
camera = Camera(fig)

w = torch.randn(1, 2, requires_grad=True)
b = torch.randn(1, requires_grad=True)

opt = optim.SGD([w,b], lr=0.1, weight_decay=0.01)

for epoch in range(50):
    for batch in dataloader:
        opt.zero_grad()
        output = svm(batch[0],w,b)
        loss = hinge_loss(output, batch[1])
        loss.backward()
       # raise NotImplementedError()
        opt.step()
    draw_margin(w, b, camera)

# create the animation and display it
anim = camera.animate()
plt.close()
HTML(anim.to_html5_video())
```

Now do some further experiments. What optimiser and parameters gets you to a good solution the quickest? Do you notice that when the model is near a solution it jitters around upon each step? Can you add some kind of learning rate decay or schedule from the `torch.optim.lr_scheduler` package to reduce the learning rate over time?

```python
# YOUR CODE HERE
raise NotImplementedError()
```


# Assignment - 1 Bayesian Decision Theory

In this assignment we apply discriminant analysis to recognize the digits in the MNIST datas set. We are using 60000 digit images as the training set while there are 10000 digit images as the testing set which are of 28 x 28 vector. We consider this vector as the features of the digit images.

```python
from mlxtend.data import loadlocal_mnist
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import cv2 as cv
import math
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from numpy.linalg import inv
```

## Part 1 - Mean Digits and Standard Deviation Digits

First, we calculate the mean digits of the digits (0-9). We do this by first seperating the training data set into its corresponding digit label. Then add all the corresponding features together and divide it by the number of images of that digit class.

\begin{align}
\mu_i = \frac{\sum_{N_i}(  x_{ij})}{N_i}
\end{align}

Second, we calculate the Standard Deviation by subtracting the Mean of the digit classes from each of the images of that class and squaring the value. We then sum up the values of the corresponding features together and divide by the number of images of that digit class. This gives us the Variance of the digit class. By taking the square root of the value, we arrive at the standard deviation.

\begin{align}
    \sigma_i = \sqrt{\frac{ \sum( x_{ij} - \mu_i)}{N_i}}
\end{align}

```python
trainImgs, trainLabels = loadlocal_mnist(
        images_path='train-images-idx3-ubyte',
        labels_path='train-labels-idx1-ubyte')
testImgs, testLabels = loadlocal_mnist(
        images_path='t10k-images-idx3-ubyte',
        labels_path='t10k-labels-idx1-ubyte')

imgDF = pd.DataFrame(trainImgs, trainLabels)

imgTestDF = pd.DataFrame(testImgs)

```

```python
def calVarianceStd(digitMean, imgClass):
    digitMean = digitMean.reshape(1,784)
    imgClass = imgClass.values
    diff_sq = np.square(imgClass - digitMean)
    variance = diff_sq.sum(axis=0)/len(imgClass)
    std_deviation = np.sqrt(variance)
    return std_deviation.reshape(28,28)

```

```python
def printMeanStd():
    for i in range(10):
        fileNameMean = "Class " + str(i)+" Mean.png"
        cv.imwrite(fileNameMean, imgDigitMean[i])
        fileNameStd = "Class " + str(i)+" Std_Dev.png"
        cv.imwrite(fileNameStd, imgDigitStd[i])
```

```python
imgDigitMean = []
imgDigitStd = []

for i in range(10):
    imgClass = imgDF.loc[i]
    imgDigit =  imgClass.sum()/len(imgClass)
    imgDigitMean.append(imgDigit.values.reshape(28,28))
    imgDigitStd.append(calVarianceStd(imgDigitMean[i], imgClass))

printMeanStd()
```

These are the Mean Digit Images

The Standard Deviation Images are :

# Part 2 - Classify using Discriminant Analysis

According to Bayesian Decision Rule, we try to minimize the Conditional Risk for the images. Instead of minimizing the Risk, we can maximize the posterior probability to find the corresponding class the image belongs to. Hence we can use Discriminant Analysis to classify the test images.

Since the images are in Multivariate Normal Distribution, we derive the discriminant function by taking the log likelihood of the Guassian Distribution.

### Guassian Distribution

\begin{align}
p(x)=\frac{1}{(2π)^{d/2}|\sum|^{1/2}}exp(−\frac{1}{2}(x−\mu)^{′}\sum^{−1}(x−\mu))
\end{align}

### Log Likelihood

\begin{align}
g_{i}(x) = \log p(x|\omega_i) + \log P(\omega_i)
\end{align}
\begin{align}
= -\frac{1}{2}(x - \mu_{i})^{-1} \sum^{-1}(x - \mu_{i}) - \frac{d}{2} \log (2 \pi) -\frac{1}{2} \log |\sum_i| + \log P(\omega_i)
\end{align}

While calculating the Mean and Covariance, we see that the Covariance matrices are different for each class, hence we shall apply the Case 3 equation. This is the Quadratic Discriminant Function which is expressed as the following:

### Quadratic Discriminant Functions
\begin{align}
g_i = x^{t} W_i x + N_{i}^{t} x + B_{i0}
\end{align}

\begin{align}
where, W_i = - \frac{1}{2} \sum_{i}^{-1} , N_{i} = \sum_{i}^{-1} \mu_{i} and B_{i0} = - \frac{1}{2} \mu_{i}^{t} \sum_{i}^{-1} \mu_{i} + ln P(w_i) - \frac{1}{2} ln |\sum_{i}|
\end{align}

After finding the values of G(i) for each image, we then maximize G to find the corresponding class the test image belongs to.

```python
def getCovarianceMat():
    epsilon = 0.1
    covarianceMat = []
    invCovarianceMat = []

    for i in range(len(imgDigitStd)):
        var = np.asarray(imgDigitStd[i]).reshape(28*28) + epsilon
        coVar = np.diag(np.square(var))
        covarianceMat.append(coVar)

        invCovarianceMat.append(inv(coVar))
    return covarianceMat, invCovarianceMat
```

```python
def quadDiscrFunc(x):
    g = []
    for i in range(10):
        W = -1/2*invCovarianceMat[i]
        N = np.matmul(invCovarianceMat[i],imgDigitMean[i].reshape(28*28,1))
        s, det = np.linalg.slogdet(covarianceMat[i])
        prior = np.log(1/10)
        prod = np.matmul(-1/2*imgDigitMean[i].reshape(1,28*28),(np.matmul(invCovarianceMat[i],imgDigitMean[i].reshape(28*28,1))))
        B0 = prod + prior
        tempG = np.matmul(np.matmul(x.reshape(1,784),W),x.reshape(784,1)) + np.matmul(np.asarray(N).T,x)+ B0
        g.append(tempG)
    return np.argmax(g)
```

### 0-1 Error Function

This loss function is used to determine how many of the test images are of the correct label and how much is the error rate. This tells us the accuracy and the error rate. It is given by :

\begin{align}
  \lambda =
\begin{cases}
     0, i=j\\
    \newline 1,i \neq j\\
\end{cases}
\end{align}

```python
def calAccuracy():
    predicted = np.asarray(pred)
    accuracy = (predicted == testLabels).sum()/len(testLabels)*100
    return accuracy, 100-accuracy

```

```python
covarianceMat, invCovarianceMat = getCovarianceMat()
```

```python
pred = []
for i in range(imgTestDF.shape[0]):
    pred.append(quadDiscrFunc(imgTestDF.loc[i].values))
#pred
```

```python
acc , err = calAccuracy()
print ("Accuracy : " + str(acc))
print ("Error Rate : " + str(err))
```

## Why it doesn't perform as good as many other methods on LeCuns web page?

In this process, we are assuming that the images are in Gaussian Distribution. This may not actually be the case for the images. Since we are finding the discriminant function by utilizing the Gaussian Distribution, the whole process might be wrong and we will get a lower accuracy. The other methods such as SVM, NN all find an optimal hyperplance to seperate the classes to accurately predict the classes. This is not possible by using this method.

# Part 3 - Fisher Digits

The scatter matrices are pair wise plot of several variables presented in a matrix format.

The Scatter W (Sw) is the scatter plot within the classes, while the Scatter B (Sb) is between the classes.

```python
def scatterW():
    Sw = np.empty([784,784])
    for i in range(len(imgDigitMean)):
        s = np.matmul((imgDF.loc[i] - imgDigitMean[i].reshape(784)).T,(imgDF.loc[i]- imgDigitMean[i].reshape(784)))
        #print (s.shape)
        Sw = Sw + s
    return Sw
```

```python
def totalMean():
    Mu = np.empty([784])
    for i in range(len(imgDigitMean)):
        Mu = Mu + np.asarray(imgDigitMean[i]).reshape(784) * imgDF.loc[i].shape[0]
        #print (Mu)

    return Mu/imgDF.shape[0]
```

```python
def scatterB():
    Sb = np.empty([784,784])
    for i in range(len(imgDigitMean)):
        Sb = Sb + (imgDF.loc[i].shape[0] * (np.matmul((imgDigitMean[i].reshape(784) - Mu),(imgDigitMean[i].reshape(784) - Mu).T)))
    return Sb

```

```python
Sw = scatterW()
Mu = totalMean()
Sb = scatterB()
```

Here we find the eigen values and vectors by taking the product of inv(Sw) and Sb
Then we take the eigen vectors which corresponds to eigen values which are high and of the desired Dimensionality. This will be your new feature vector.

```python
mat = np.dot(np.linalg.pinv(Sw), Sb)
eigvals, eigvecs = np.linalg.eig(mat)

np.argmax(eigvals)
eigvals[:300]
W = eigvecs[:3]
W.shape
#eigvecs.shape
y = np.matmul( W,testImgs.T)

y.T.shape
z = y.T
```

```python

```

```python

```


# Introduction to PDEs

- toc:false
- branch: master
- badges: true
- comments: false
- categories: [mathematics, numerical recipes]
- hide: true

-----
Questions:
- Which physical systems can be described using a PDE?
- What is the Laplacian operator?
-  When to I need boundary conditions and initial conditions?

Objectives:
- Recognise common classes of PDE: the diffusion equation, Poisson's equation and the wave equation
- Express the Laplacian as a differential operator
- Identify boundary value problems and initial value problems

-----

### Partial differential equations have multiple inputs

In the previous section of the course we studied <bold>ordinary differential equations</bold>. ODEs have a single input (also known as independent variable) - for example, time.

Partial differential equations (PDEs) have multiple inputs (independent variables). For example, think about a sheet of metal that has been heated unevenly across the surface. Over time, heat will diffuse through the 2-dimensional sheet. The temperature depends on both time *and* position - there are two inputs.

> youtube: https://youtu.be/twBcpxrWm5E

Because PDEs have multiple inputs they are generally much more difficult to solve analytically than ODEs. However, there are a range of numerical methods that can be used to find approximate solutions.

### PDEs have application across a wide variety of topics

The same type of PDE often appears in different contexts. For example, the diffusion equation takes the form:

\begin{equation}
\nabla^2T = \alpha \frac{\partial T}{\partial t}
\end{equation}

When used to describe heat diffusion, this PDE is known as the heat equation. This same PDE however can be used to model other seemingly unrelated processes such as brownian motion, or used in financial modelling via the Black-Sholes equation.

Another type of PDE is known as Poisson's equation:

\begin{equation}
\nabla^2\phi = f(x,y,z)
\end{equation}

Poisson's equation can be used to describe electrostatic forces, where $\phi$ is the electric potential. It can also be applied to mechanics (where $\phi$ is the gravitational potential) or thermodynamics (where $\phi$ is the temperature). When $f(x,y,z)=0$ this equation is known as Laplace's equation.

The third common type of PDE is the wave equation:

\begin{equation}
\nabla^2r = \alpha \frac{\partial^2 r}{\partial t^2}
\end{equation}

This describes mechanical processes such as the vibration of a string or the motion of a pendulum. It can also be used in electrodynamics to describe the exchange of energy between the electric and magnetic fields.

In this course we will look at techniques for solving the diffusion equation and Poisson's equation, but many of the topics we will discuss - such as boundary conditions, and finite difference methods - can be transferred to PDEs more generally.

### The Laplacian operator corresponds to an average rate of change

*But what is the operator $\nabla^2$?*. This is the Laplacian operator. When applied to $\phi$ and written in full for a three dimensional cartesian coordinate system with dependent variables $x$, $y$ and $z$ it takes the following form:

\begin{equation}
\nabla^2\phi = \frac{\partial^2\phi}{\partial x^2} + \frac{\partial^2\phi}{\partial y^2} + \frac{\partial^2\phi}{\partial z^2}.
\end{equation}

We can think of the laplacian as encoding an average rate of change. To develop an intuition for how the laplacian can be interpreted physically, we need to understand two related operators - div and curl. We will not explore these operators further in this lesson, but a related video is below:

> youtube: https://youtu.be/EW08rD-GFh0

### PDEs can have boundary conditions *and* initial conditions
In the previous section of the course we learnt that ODEs have either initial values or boundary values.

#### Boundary value problems

PDEs can also be separated in a similar manner. Boundary value problems describe the behaviour of a variable in a space and we are given some constraints on the variable around the boundary of that space. For example, consider the 2-dimensional problem of a thin rectangular sheet with one side at voltage $V$ and all others at voltage zero.

The specification that one side is at voltage $V$ and all others are at voltage zero are the boundary conditions. We could then calculate the electrostatic potential $\phi$ at all points within the sheet using the two-dimensional Laplace's equation:

\begin{equation}
\nabla^2\phi = \frac{\partial^2\phi}{\partial x^2} + \frac{\partial^2\phi}{\partial y^2} = 0
\end{equation}

#### Initial value problems

Initial value problems are where the field - or other variable of interest - is varying in both space and time. We now require boundary conditions *and* initial values. This is a more difficult type of PDE to solve.

For example, consider heat diffusion in a two-dimensional sheet. Here we could specify that there is no heat flow in or out of the sheet - this is the boundary condition.

We could also specify that at time $t=0$ the centre of the sheet is at temperature $T_1$, whilst surrounding areas are at temperature  $T_0$. This is the initial condition. It differs from a boundary condition in that we are told what the temperature is at the start of our time grid (at $t=0$) but not at the end of our time grid (when the simulation finishes).

We could then calculate the temperature at time $t$ at all points $[x,y]$ within the sheet using the two-dimensional Diffusion equation:

\begin{equation}
\nabla^2T = \frac{\partial^2 T}{\partial x^2} + \frac{\partial^2 T}{\partial y^2}=\alpha \frac{\partial T}{\partial t}
\end{equation}

---

Keypoints:

- PDEs have multiple inputs (independent variables)
- PDEs have application across a wide variety of topics
- The Laplacian operator corresponds to an average rate of change
- PDEs can have boundary values and initial values

---

Do [the quick-test](https://nu-cem.github.io/CompPhys/2021/08/02/PDE-Intro-Qs.html).

Back to [Modelling with Partial Differential Equations](https://nu-cem.github.io/CompPhys/2021/08/02/PDEs.html).

---


This notebook explains our design decisions for simulating stochastic dynamical systems.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from kalman import LDSNet
```

# Weiner Process

Describes a stochastic process $W$ which is driven by noise $w$ and governed by

$$dW/dt=v$$

The process is characterized $W(t+\Delta t)-W(t)\sim\mathcal{N}(0, \Delta t)$ (i.e., normally-distributed with mean 0 and variance $\Delta t$.

Discretize the stochastic differential equation SDE by chopping up time into $\Delta t$-sized bins and apply the Euler step

\begin{align}
\frac{W_{t+\Delta t}-W_t}{\Delta t} &= v_t \\
W_{t+\Delta t} &= W_t + \Delta tv_t \\
\end{align}

So that $W_{t+\Delta t}-W_t~\sim\mathcal{N}(0, \Delta t)$, we'll need to set $v_t~\sim\mathcal{N}\left(0, \frac{1}{\Delta t}\right)$

As an alternative, we could use the update $W_{t+\Delta t} = W_t + \sqrt{\Delta t}\tilde{v}_t$ where $\tilde{v}_t\sim\mathcal{N}(0, 1)$

```python
T = 1
Q = 1 # Target variance per unit time
dts = [1, 0.5, 0.1, 0.05, 0.01][::-1]
runs = 100000
data = {}
for dt in dts:
    N = int(np.round(T/dt))
    dW = np.random.randn(runs, N)*np.sqrt(dt)*np.sqrt(Q)
    W = np.zeros((runs, N+1))
    W[:, 1:] = np.cumsum(dW, axis=1)
    data[dt] = W
```

Showing a sample paths for each dt

```python
# plot trajectories
N_PLOT = 2
for dt in dts:
    N = int(np.round(T/dt))
    trange = np.arange(N+1)*dt
    color = plt.plot(trange, data[dt][0, :], label=dt)[0].get_color()
    for plt_idx in range(N_PLOT)[1:]:
        plt.plot(trange, data[dt][plt_idx+1, :], color=color)
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5), title="dt")
plt.xlabel("Time")
plt.ylabel("W")
```

Note how regardless of time discretization, the variance grows at the same rate

```python
# plot variance over time
plt.figure()
for dt in dts:
    N = int(np.round(T/dt))
    trange = np.arange(N+1)*dt
    var = np.var(data[dt], axis=0)
    plt.plot(trange, var, label=dt)
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5), title="dt")
plt.xlabel("Time")
plt.ylabel("Variance")

plt.figure()
for dt in dts:
    plt.hist(data[dt][:, -1], bins=100, density=True, histtype="step", label=str(dt))
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5), title="dt")
plt.title("Distribution of W at T=1")
plt.xlabel("W(1)")
```

## A Note of Caution

If we naively used $W_{t+\Delta t} = W_t + \Delta t\tilde{v}_t$ where $\tilde{v}_t\sim\mathcal{N}(0, 1)$, then the rate of variance growth over time is linear with $\Delta t$.

```python
for dt in dts:
    N = int(np.round(T/dt))
    dW = np.random.randn(runs, N)*dt
    W = np.zeros((runs, N+1))
    W[:, 1:] = np.cumsum(dW, axis=1)
    data[dt] = W

# plot trajectories
plt.figure()
N_PLOT = 2
for dt in dts:
    N = int(np.round(T/dt))
    trange = np.arange(N+1)*dt
    color = plt.plot(trange, data[dt][0, :], label=dt)[0].get_color()
    for plt_idx in range(N_PLOT)[1:]:
        plt.plot(trange, data[dt][plt_idx+1, :], color=color)
plt.xlabel("Time")
plt.ylabel("W")
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5), title="dt")

# plot variance over time
plt.figure()
for dt in dts:
    N = int(np.round(T/dt))
    trange = np.arange(N+1)*dt
    var = np.var(data[dt], axis=0)
    plt.plot(trange, var, label=dt)
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5), title="dt")
plt.xlabel("Time")
plt.ylabel("Variance")

plt.figure()
for dt in dts:
    plt.hist(data[dt][:, -1], bins=100, density=True, histtype="step", label=str(dt))
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5), title="dt")
plt.title("Distribution of W at T=1")
plt.xlabel("W(1)")
```

```python

```


```python
import numpy as np
import sympy as sy
```

```python
p   = 3
Nel = 5

t, dt = sy.symbols('t, dt')
knots = (np.arange(Nel + 1 + 2*p) - p)*dt
```

```python
knots
```

    array([-3*dt, -2*dt, -dt, 0, dt, 2*dt, 3*dt, 4*dt, 5*dt, 6*dt, 7*dt, 8*dt],
          dtype=object)

```python
N3 = [[[]], [[], []], [[], [], []], [[], [], [], []]]

# initialization
for i in range(len(knots) - 1):
    N3[0][0].append(1)

# recursion
for i in range(1, p + 1):

    pk = i

    for j in range(len(knots) - pk - 1):
        for k in range(pk + 1):

            if k == 0:

                N3[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*N3[i - 1][k][j] + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*0))

            elif k == pk:

                N3[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*0 + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*N3[i - 1][k - 1][j + 1]))

            else:

                N3[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*N3[i - 1][k][j] + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*N3[i - 1][k - 1][j + 1]))
```

```python
N3[3][3][0]
```

    1/6 - t/(2*dt) + t**2/(2*dt**2) - t**3/(6*dt**3)

```python
p   = 2
Nel = 5

t, dt = sy.symbols('t, dt')
knots = (np.arange(Nel + 1 + 2*p) - p)*dt
```

```python
knots
```

    array([-2*dt, -dt, 0, dt, 2*dt, 3*dt, 4*dt, 5*dt, 6*dt, 7*dt],
          dtype=object)

```python
N2 = [[[]], [[], []], [[], [], []]]

# initialization
for i in range(len(knots) - 1):
    N2[0][0].append(1)

# recursion
for i in range(1, p + 1):

    pk = i

    for j in range(len(knots) - pk - 1):
        for k in range(pk + 1):

            if k == 0:

                N2[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*N2[i - 1][k][j] + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*0))

            elif k == pk:

                N2[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*0 + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*N2[i - 1][k - 1][j + 1]))

            else:

                N2[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*N2[i - 1][k][j] + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*N2[i - 1][k - 1][j + 1]))
```

```python
N2[2][0][2]
```

    t**2/(2*dt**2)

```python
p   = 1
Nel = 5

t, dt = sy.symbols('t, dt')
knots = (np.arange(Nel + 1 + 2*p) - p)*dt
```

```python
knots
```

    array([-dt, 0, dt, 2*dt, 3*dt, 4*dt, 5*dt, 6*dt], dtype=object)

```python
N1 = [[[]], [[], []]]

# initialization
for i in range(len(knots) - 1):
    N1[0][0].append(1)

# recursion
for i in range(1, p + 1):

    pk = i

    for j in range(len(knots) - pk - 1):
        for k in range(pk + 1):

            if k == 0:

                N1[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*N1[i - 1][k][j] + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*0))

            elif k == pk:

                N1[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*0 + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*N1[i - 1][k - 1][j + 1]))

            else:

                N1[i][k].append(sy.expand((t - knots[j])/(knots[j + pk] - knots[j])*N1[i - 1][k][j] + (knots[j + pk + 1] - t)/(knots[j + pk + 1] - knots[j + 1])*N1[i - 1][k - 1][j + 1]))
```

```python
N1[1][0][1]
```

    t/dt

```python

```


Rechneranwendungen in der Physik - Übung N.4 Umlaufbahnen

Santiago.R

```python
import numpy as np
import sympy as sy
from scipy import optimize
import matplotlib.pyplot as plt
from scipy import integrate
```

# Simulation der Erdumlaufbahn $U_E(t)$

Zuerst werden die wichtigen Konstanten und Startparameter definiert;

```python
#Konstanten
G = 6.67430e-11 #in m3/kg*s2
m_sonne = 1.989e30 #in kg
m_erde = 5.972e24 #in kg (wird in dieser Lösung vernachlässigt, da die Sonne als stationär im Punkt (x,y)=(0,0) angenohmen wird)
r_ES = 1.496e11 #in m
v_E = 29.78e3 #in m/s
#Startparameter der Erde
y0_erde = r_ES
x0_erde = 0
v_x0_erde=-v_E #negativ um in Richtung gegen den Uhrzeigersinn zu zeigen
v_y0_erde=0
```

Um einen Plot $U_E(t)$ für die Umlaufbahn der Erde um die Sonne zu erstellen müssen zuerst die Bewegungsgleichungen formuliert werden. Diese hängen im wesetlichen von der einzigen in diesem System relevanten Kraft, die Gravitationskraft $\vec{F_g}=-G \cdot \frac{M \cdot m}{\vec{r}^2}$ ab. Diese kann dann mit $\vec{r}=\frac{\vec{r_e}}{|r|}$ und $|r|=\sqrt{x^2+y^2}$ umgeschrieben werden als $\vec{F_g}=-G \cdot \frac{M \cdot m}{\sqrt{x^2+y^2}^{3/2}} \cdot \vec{r_e}$. Die $x$- und $y$-Komponenten der angreifenden Gravitationskraft können dann mit $F_x=-G \cdot \frac{M \cdot m}{\sqrt{x^2+y^2}^{3}} \cdot x$ und $F_y=-G \cdot \frac{M \cdot m}{\sqrt{x^2+y^2}^{3}} \cdot y$ parametrisiert und anschließend auf die gekoppelten Differentialgleichungen $\frac{d^2x}{dt^2}=-\frac{GM}{\sqrt{x^2+y^2}^{3}} \cdot x$ und $\frac{d^2y}{dt^2}=- \frac{GM}{\sqrt{x^2+y^2}^{3}} \cdot y$ überführt werden. Nummerisches Integrieren dieser Differentialgleichungen nach der Zeit $t$ liefert dann die $x$- und $y$-Werte der Umlaufbahn $U_E(t)$ an jedem integrierten Zeitpunkt $t$

```python
def dgl_erde_sonne(i, t):
    x, y, v_x, v_y = i #Input für die Startparameter
    g = G*m_sonne/np.sqrt(x**2+y**2)**3; #der angreifende Parameter an jedem Punkt x,y
    return [v_x, v_y, -x*g, -y*g];

t = np.linspace(0, 31536000, 50000) #in SI-Einheiten ist die Angabe für ein Jahr auf Sekunden umgestellt
startparameter = [x0_erde, y0_erde, v_x0_erde, v_y0_erde]
s_t = integrate.odeint(dgl_erde_sonne, startparameter, t)

x,y,_,_ = s_t.T

plt.plot(0,0,'oy', ms=8, label = 'Sonne')
plt.plot(x,y, label = 'Erdumlaufbahn'); plt.axis('equal');
plt.xlabel("x-Achse in 10^11 Meter")
plt.ylabel("y-Achse in 10^11 Meter")
plt.legend(loc='upper right')
plt.show()
```

# Simulation für unterschiedliche Toleranzen [e-1,e-5]

```python
t = np.linspace(0, 31536000, 50000) #in SI-Einheiten ist die Angabe für ein Jahr auf Sekunden umgestellt
startparameter = [x0_erde, y0_erde, v_x0_erde, v_y0_erde]
for i in range(1,6,1):
    s_t = integrate.odeint(dgl_erde_sonne, startparameter, t, rtol=10**(-i))
    x,y,_,_ = s_t.T
    plt.plot(x,y, label = 10**(-i)); plt.axis('equal');
plt.plot(0,0,'oy', ms=8, label = 'Sonne')
plt.xlabel("x-Achse in 10^11 Meter")
plt.ylabel("y-Achse in 10^11 Meter")
plt.legend(title='Plot-Toleranz',loc='upper right')
plt.show()
```

# Simulation für unterschiedliche Startwerte $r_0$

```python
#Neue Startwerte
y1_erde=0.6*r_ES
y2_erde=1.4*r_ES
#Integration für y1
t = np.linspace(0, 4*31536000, 50000) #in SI-Einheiten ist die Angabe für ein Jahr auf Sekunden umgestellt
startparameter1 = [x0_erde, y1_erde, v_x0_erde, v_y0_erde]
s_t1 = integrate.odeint(dgl_erde_sonne, startparameter1, t)
#Integration für y2
startparameter2 = [x0_erde, y2_erde, v_x0_erde, v_y0_erde]
s_t2 = integrate.odeint(dgl_erde_sonne, startparameter2, t)
#Plots
x1,y1,_,_ = s_t1.T
x2,y2,_,_ = s_t2.T
plt.plot(0,0,'oy', ms=8, label = 'Sonne')
plt.plot(x1,y1, label = '0.6*r_ES'); plt.axis('equal');
plt.plot(x2,y2, label = '1.4*r_ES'); plt.axis('equal');
plt.xlabel("x-Achse in 10^11 Meter")
plt.ylabel("y-Achse in 10^11 Meter")
plt.legend(title='Startparameter',loc='upper right')
plt.show()
```


## The SIR model

The SIR model of epidemiology partitions the population into three compartments: susceptibles, S, who can catch the disease; infectives, I, who have already caught the disease and infect susceptibles; and recovered individuals, R. Since the disease is assumed not to be fatal, the sum $N=S+I+R$ remains constant. The rate at which the susceptibles get infected is
$$
\lambda(t) = \frac{\beta I}{N}
$$
where the parameter $\beta$ is the probability of infection on contact. The infected individuals recover from the disease at a rate $\gamma$. Then, the ordinary differential equations of the SIR model are

\begin{align}
\dot S &= -\lambda(t)S \\
\dot I &= \lambda(t)I - \gamma I \\
\dot R &= \gamma I
\end{align}

This example integrates the above equations to obtain what is called the **epidemic curve**: a plot of the number of susceptibles and infectives as a function of time.

```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../')
%run setup.py install
os.chdir(owd)
```

```python
%matplotlib inline
import numpy as np
import pyross
import matplotlib.pyplot as plt
#from matplotlib import rc; rc('text', usetex=True)
```

```python
M = 1                  # the SIR model has no age structure
Ni = 1000*np.ones(M)   # so there is only one age group
N = np.sum(Ni)         # and the total population is the size of this age group

beta  = 0.2            # infection rate
gIa   = 0.1            # recovery rate of asymptomatic infectives
gIs   = 0.1            # recovery rate of symptomatic infectives
alpha = 0              # fraction of asymptomatic infectives
fsa   = 1              # the self-isolation parameter

Ia0 = np.array([0])     # the SIR model has only one kind of infective
Is0 = np.array([1])     # we take these to be symptomatic
R0  = np.array([0])     # and assume there are no recovered individuals initially
S0  = N-(Ia0+Is0+R0)    # so that the initial susceptibles are obtained from S + Ia + Is + R = N

# there is no contact structure
def contactMatrix(t):
    return np.identity(M)

# duration of simulation and data file
Tf = 160;  Nt=160;

# instantiate model
parameters = {'alpha':alpha, 'beta':beta, 'gIa':gIa, 'gIs':gIs,'fsa':fsa}
model = pyross.models.SIR(parameters, M, Ni)

# simulate model
data = model.simulate(S0, Ia0, Is0, contactMatrix, Tf, Nt)
```

```python
# plot the data and obtain the epidemic curve
S  = data['X'][:,0].flatten()
Ia = data['X'][:,1].flatten()
Is = data['X'][:,2].flatten()
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
plt.plot(t, S/N, '-', color="#348ABD", label='$S$', lw=4)

plt.fill_between(t, 0, Is/N, color='#A60628', alpha=0.3)
plt.plot(t, Is/N, '-', color='#A60628', label='$I$', lw=4)

R=N-S-Ia-Is; plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
plt.plot(t, R/N, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
```


# Scalar wave equation seismic amplitude derivations

## Green's functions
The wavefield at $(\mathbf{x}, t)$ due to source term $f(\mathbf{x}, t)$ is given by
$$ u(\mathbf{x}, t) = \int \int G(\mathbf{x}, t; \mathbf{x'}, t')f(\mathbf{x'}, t') d\mathbf{x'}dt', $$
where $G(\mathbf{x}, t; \mathbf{x'}, t')$ is the Green's function for the wave equation between $(\mathbf{x'}, t')$ and $(\mathbf{x}, t)$.

The scalar wave equation is
$$ \frac{1}{c^2(\mathbf{x})}\partial_{tt}u(\mathbf{x}, t) - \nabla u(\mathbf{x}, t) = f(\mathbf{x}, t) $$

For the scalar wave equation when the wave speed is constant, the Green's functions are given by

Dimension | Green's function
--- | ---
**1D** | $\frac{c}{2}\Theta(\tau - r/c)$
**2D** | $\frac{c}{2\pi \sqrt{c^2\tau^2 - r^2}}\Theta(\tau - r/c)$
**3D** | $\frac{\delta(\tau-\frac{r}{c})}{4 \pi r}$

where $r = ||\mathbf{x}-\mathbf{x'}||_2$, $\tau=t-t'$ for the causal Green's function or $\tau=t'-t$ for the anticausal Green's function, and $\Theta(x)$ is a step function.

## Direct wave

The expected amplitude of the direct, unscattered wave in a constant wave speed model can thus be computed using the equations above.

For a discretized simulation, with a source that is only active in a single cell of the model (at position $\mathbf{x_s}$), so $f(\mathbf{x}, t) = f(t)\delta(\mathbf{x} - \mathbf{x_s})v$ (where $v$ is the volume of the cell) and $r = ||\mathbf{x}-\mathbf{x_s}||_2$, the amplitudes are thus:

\begin{align}
u_{1D}(\mathbf{x}, t) &= \frac{\Delta x\Delta tc}{2}\sum_{t'=0}^{t - r/c} f(t') \\
u_{2D}(\mathbf{x}, t) &= \frac{\Delta x^2\Delta tc}{2\pi}\sum_{t'=0}^{t - r/c}\frac{f(t')}{\sqrt{c^2(t-t')^2 - r^2}} \approx \mathcal{F^{-1}} \left( \frac{i}{4}H^{(1)}_0\left(\frac{-2\pi \omega r}{c}\right)f(\omega)\Delta x^2 \right) \\
u_{3D}(\mathbf{x}, t) &= \frac{\Delta x^3f(t-\frac{r}{c})}{4 \pi r}
\end{align}

## Scattered wave

To calculate the expected amplitude of a wave that has been scattered by a wave speed anomaly in a single cell (a "point scatterer"), we need to derive the scattering amplitude.

Separating the wave speed into a smooth background ($c_0$, which we will assume varies slowly enough to be considered locally constant) and short wavelength changes ($\Delta c$), and performing a linear expansion around $\Delta c(\mathbf{x})=0$,
$$\frac{1}{c^2(\mathbf{x})} = \frac{1}{(c_0+\Delta c)^2(\mathbf{x})} \approx \frac{1}{c_0^2(\mathbf{x})} - 2 \frac{\Delta c(\mathbf{x})}{c_0^3(\mathbf{x})}.$$
We also assume that the wavefield can be split into a term that only senses the smooth model, and a scattered term,
$$u(\mathbf{x}, t) = u_0(\mathbf{x}, t) + u_{sc}(\mathbf{x}, t).$$
For the scalar wave equation, this gives
$$ \left(\frac{1}{c_0^2(\mathbf{x})}-2\frac{\Delta c(\mathbf{x})}{c_0^3(\mathbf{x})}\right)\partial_{tt}(u_0(\mathbf{x}, t)+u_{sc}(\mathbf{x},t)) - \nabla (u_0(\mathbf{x}, t)+u_{sc}(\mathbf{x},t)) = f(\mathbf{x}, t), $$
and
$$ \frac{1}{c_0^2(\mathbf{x})}\partial_{tt}u_0(\mathbf{x}, t) - \nabla u_0(\mathbf{x}, t) = f(\mathbf{x}, t). $$
Subtracting the second of these from the first gives
$$ \frac{1}{c_0^2(\mathbf{x})}\partial_{tt}u_{sc}(\mathbf{x}, t) - \nabla u_{sc}(\mathbf{x}, t) = 2\frac{\Delta c(\mathbf{x})}{c_0^3(\mathbf{x})}\partial_{tt}u(\mathbf{x}, t). $$

Making the Born approximation, we assume that multiply scattered waves have negligible amplitude. That is, the scattered wavefield $u_{sc}$ is approximated as being only generated by interactions of $u_0$ with scatterers, and further interactions of $u_{sc}$ with the scatterers are ignored. This is expressed by changing $u$ to $u_0$ in the previous equation,
$$ \frac{1}{c_0^2(\mathbf{x})}\partial_{tt}u_{sc}(\mathbf{x}, t) - \nabla u_{sc}(\mathbf{x}, t) = 2\frac{\Delta c(\mathbf{x})}{c_0^3(\mathbf{x})}\partial_{tt}u_0(\mathbf{x}, t). $$
Note that this is in the form of the wave equation, with the right-hand side as the source term. We can use this to calculate the scattered wavefield. In terms of Green's functions,
$$ u_0(\mathbf{x}, t) = \int \int G(\mathbf{x}, t; \mathbf{x'}, t') f(\mathbf{x'}, t') d\mathbf{x'} dt', $$
$$ u_{sc}(\mathbf{x}, t) = 2\int \int G(\mathbf{x}, t; \mathbf{x'}, t') \frac{\Delta c(\mathbf{x'})}{c_0^3(\mathbf{x'})}\partial_{t't'}u_0(\mathbf{x'}, t') d\mathbf{x'} dt'. $$

For a discretized simulation with a single cell source at $\mathbf{x_s}$ as above, and a wave speed model that is the constant value $c$ everywhere except at a single cell ($\mathbf{x_p}$) where it is $c+\Delta c$ (a point scatterer),  with $r=||\mathbf{x}-\mathbf{x_p}||_2$, and with $u_{1D}$, $u_{2D}$, and $u_{3D}$ from the direct wave calculation above, and using the direct wave amplitudes derived above, with $u_{p}(\mathbf{x}, t)$ being the direct wave that originates from the scatterer location with an amplitude that is the second time derivative of the direct wave that arrives at that location from the real source, the expected amplitudes of the scattered wavefields are:
\begin{equation}
u^{sc}(\mathbf{x}, t) = \frac{2 \Delta c}{c^3} u_{p}(\mathbf{x}, t).
\end{equation}

## Scatterer image/Gradient

Let's assume we only know the constant background wave speed $c$, and we want to find the location and amplitude of a point scatterer. The only information we have to help us find it is the wavefield recorded on the surface of the model that contains the scatterer, due to sources on the surface, which I will call $d(t)$. This is the typical situation when applying RTM, LSRTM, or FWI.

We'll define a mean squared error cost function that compares the wavefield at the surface when using the constant model $c$ with the data that was provided to us, $d(t)$,
$$ J = \frac{1}{T}||u_{sc}(\mathbf{x_{s}}, t) - d(t)||_2^2, $$
where $\mathbf{x_s}$ are the coordinates of the observation surface.

The derivative of this cost function with respect to changes in the model, where $T$ is the maximum recorded time, is
$$ \frac{\partial J}{\partial \Delta c(\mathbf{x})} = \frac{1}{T}\int_0^T2(u_{sc}(\mathbf{x_{s}}, t) - d(t))\frac{\partial u_{sc}}{\partial \Delta c(\mathbf{x})}dt. $$

Using our result from earlier, we get
$$ \frac{\partial u_{sc}(\mathbf{x_{s}}, t)}{\partial \Delta c(\mathbf{x})} = \frac{2v}{c_0^3(\mathbf{x})}\int_0^t G(\mathbf{x_s}, t; \mathbf{x}, t') \partial_{t't'}u_0(\mathbf{x}, t') dt', $$
where $u_0$ is, as before, the wavefield propagated in the background model, and $v$ is the volume of one cell. So,
$$ \frac{\partial J}{\partial \Delta c(\mathbf{x})} = \frac{1}{T}\int_0^T\frac{4v}{c_0^3(\mathbf{x})}(u_{sc}(\mathbf{x_{s}}, t) - d(t)) \int_0^t G(\mathbf{x_s}, t; \mathbf{x}, t') \partial_{t't'}u_0(\mathbf{x}, t') dt'dt. $$

Since $G(\mathbf{x_s}, t; \mathbf{x}, t') = 0 \forall t'>t$ when using the causal Green's function, the upper limit of the inner integral can be increased to $\infty$ (so it no longer depends on $t$ and thus the order of integration can be switched), and since the model is time invariant and due to source-receiver reciprocity, $G^+(\mathbf{x_s}, t; \mathbf{x}, t') = G^-(\mathbf{x}, t'; \mathbf{x_s}, t)$, we can write this as
$$ \frac{\partial J}{\partial \Delta c(\mathbf{x})} = \frac{1}{T}\int_0^\infty\int_T^0\frac{4v}{c_0^3(\mathbf{x})}(u_{sc}(\mathbf{x_{s}}, t) - d(t)) G^-(\mathbf{x}, t'; \mathbf{x_s}, t) \partial_{t't'}u_0(\mathbf{x}, t') dtdt'. $$

This is a product involving $u_0$ and a new wavefield I will call $u_r$, obtained by propagating the source $u_{sc}(\mathbf{x_{s}}, t) - d(t)$ backward in time:
$$ \frac{\partial J}{\partial\Delta c(\mathbf{x})} = \frac{4}{Tc_0^3(\mathbf{x})}\int u_r(\mathbf{x}, t)\partial_{tt}u_0(\mathbf{x}, t) dt. $$

This is known as an adjoint-state method means of calculating the gradient.


```python
import sympy as sp
import scipy as sc
import matplotlib as mpl
import matplotlib.pyplot as plt

sp.init_printing(use_unicode = True)
```

Kiindulási adatok megadása

```python
L, F, IE = sp.symbols("L, F, IE")
adat = [(L, 3.0), (F,7000.0), (IE, 200e3)]
```

# 1. Reakcióerők számítása

A lehetséges reakcióerők a megtámasztásokból adódóan: Ax, Ay, By

```python
Ax, Ay, By = sp.symbols("Ax, Ay, By")
```

A 3 statikai egyenlet ( két erő-egyensúlyi egyenlet és 1 nyomatéki egyenlet például az A pontra felírva):

```python
stat1 = Ax
stat2 = Ay + By - F
stat3 = By*L/2 - F*L
```

Hajlítónyomatéki függvények mindkét szakaszon:

```python
x = sp.Symbol('x')

Mh1 = -Ay*x
Mh2 = -Ay*x - By*(x-L/2)
```

# 2. Rugalmas szál differenciálegyenlete

A rugalmas szál diffegyenlete a két szakaszon: $$y_1{}''\left(x\right)=-\dfrac{M_{\mathrm{h1}}(x)}{IE}$$ és $$y_2{}''(x)=-\dfrac{M_{\mathrm{h2}}(x)}{IE}$$. Itt most $y_1(x)$ és $y_2(x)$ függvények jelölik a lehajlásokat.

A lehajlásfüggvények számításához kétszer kell integrálni nem elfeledkezve az integrálási konstansokról!
Legyen $\varphi_1(x)=y_1{}'(x)$ függvénnyel jelölve a
szögelfordulás függvény.

```python
c1, c2, c3, c4 = sp.symbols("c1 c2 c3 c4")

φ1 = sp.integrate(-Mh1/IE,x) + c1
y1 = sp.integrate(φ1,x) + c2

φ2 = sp.integrate(-Mh2/IE,x) + c3
y2 = sp.integrate(φ2,x) + c4
```

Az ismeretlen $c_1,c_2,c_3,c_4$ kiszámításához szükség van az illesztési és peremfeltételek megadására.

Jelen esetben van két peremfeltételünk, miszerint az A és B helyeken a
lehajlás zérus.
Valamint van két illesztési feltételünk, hogy az $y_1$ és $y_2$ függvények értékei és első deriváltjai azonosak a B helyen.

```python
pf1 = y1.subs([(x,0)])
pf2 = y1.subs([(x,L/2)])

if1 = (y1 - y2).subs([(x,L/2)])
if2 = (φ1 - φ2).subs([(x,L/2)])
```

Megoldás számítása

```python
ismeretlenek = [Ax,Ay,By,c1,c2,c3,c4]
mego = sp.solve([stat1,stat2,stat3,pf1,pf2,if1,if2],ismeretlenek)
mego
```

```python
print('y1=')
display(y1.subs(mego).simplify())
print('\n\ny2=')
display(y2.subs(mego).simplify())
print('\n\nφ1=')
display(φ1.subs(mego).simplify())
print('\n\nφ2=')
display(φ2.subs(mego).simplify())
```

@Tomi: ez kell?
[Mi legyen, hogy legyen?](https://docs.sympy.org/latest/modules/numeric-computation.html)

A numerikus vizsgálatokhoz helyettesítsünk vissza, és készítsünk a szimbolikus függvényből numerikus függvényt:

```python
y1num = sp.lambdify(x,y1.subs(mego).subs(adat))
y1num(x)
```

```python
y2num = sp.lambdify(x,y2.subs(mego).subs(adat))
y2num(x)
```

```python
φ1num = sp.lambdify(x,φ1.subs(mego).subs(adat))
φ1num(x)
```

```python
φ2num = sp.lambdify(x,φ2.subs(mego).subs(adat))
φ2num(x)
```

# 3. Ábrázolás

Ábrázoljuk mindkét lehajlás függvényt

```python
xs = sc.linspace(0,3,20)
plt.figure(num = 1, figsize=(20/2.54,20/2.54))

plt.plot(xs,y1num(xs), label = r"$y_1$")
plt.plot(xs,y2num(xs), label = r"$y_2$")

plt.plot([L.subs(adat)/2,L.subs(adat)/2],[-0.12,0.01], color = 'k')

plt.xlabel(r"$x \, (\mathrm{m})$")
plt.ylabel(r"$y \, (\mathrm{m})$")

plt.legend()
plt.grid()
plt.show()
```

```python
plt.figure(num = 2, figsize=(16/2.54,10/2.54))

plt.plot(xs,φ1num(xs), label = r"$\varphi_1$")
plt.plot(xs,φ2num(xs), label = r"$\varphi_2$")

plt.plot([L.subs(adat)/2,L.subs(adat)/2],[-0.15,0.1], color = 'k')

plt.xlabel(r"$x \, (\mathrm{m})$")
plt.ylabel(r"$\varphi \, (\mathrm{rad})$")

plt.grid()
plt.legend()
plt.show()
```

Látható, hogy a B helyen az értékük és deriváltjaiknak értéke is azonos. Ábrázoljuk a függvényeket csak a rájuk vonatkozó tartományokon:

```python
def ykozos(x):
    if x < L.subs(adat)/2:
        return y1num(x)
    else:
        return y2num(x)
ykozoslist = [ykozos(x) for x in xs]
```

```python
plt.figure(num = 1, figsize=(16/2.54,10/2.54))
plt.plot(xs,ykozoslist)

plt.ylabel(r"$y \, (\mathrm{m})$")
plt.xlabel(r"$x \, (\mathrm{m})$")
plt.grid()
plt.show()
```

Nézzük meg hasonlóképpen a szögelfordulás függvényt:

```python
def φkozos(x):
    if x < L.subs(adat)/2:
        return φ1num(x)
    else:
        return φ2num(x)
φkozoslist = [φkozos(x) for x in xs]
```

```python
plt.figure(figsize=(16/2.54,10/2.54))
plt.plot(xs,φkozoslist)

plt.ylabel(r"$\varphi \, (\mathrm{rad})$")
plt.xlabel(r"$x \, (\mathrm{m})$")
plt.grid()
plt.show()
```

# 4. Szélsőérték számítása az AB szakaszon

Látható, hogy az AB szakaszon szélsőértéke van egy ismeretlen $x_0$ helyen a lehajlásfüggvénynek. Ezen a helyen a szögelfordulás zérus. Tehát $x_0$ könnyen
számítható:

```python
x0mego = sp.solve(φ1.subs(mego),x)
x0mego
```

```python
sp.N(x0mego[1].subs(adat))
```

A kapott megoldásokból a pozitív érték adja x0 értékét. Ezt akár numerikusan is számíthatjuk gyökkereséssel:

```python
import scipy.optimize as sco
```

```python
x0 = sco.brentq(φ1num,0,L.subs(adat)/2)
x0
```

$x_0$ helyen a lehajlás értéke:

```python
y1.subs(mego).subs([(x,x0mego[1])])
```

```python
sp.N(y1.subs(mego).subs([(x,x0mego[1])]).subs(adat))
```

vagy:

```python
y1num(x0)
```


# ETS models

The ETS models are a family of time series models with an underlying state space model consisting of a level component, a trend component (T), a seasonal component (S), and an error term (E).

This notebook shows how they can be used with `statsmodels`. For a more thorough treatment we refer to [1], chapter 8 (free online resource), on which the implementation in statsmodels and the examples used in this notebook are based.

`statsmodels` implements all combinations of:
- additive and multiplicative error model
- additive and multiplicative trend, possibly dampened
- additive and multiplicative seasonality

However, not all of these methods are stable. Refer to [1] and references therein for more info about model stability.

[1] Hyndman, Rob J., and George Athanasopoulos. *Forecasting: principles and practice*, 3rd edition, OTexts, 2019. https://www.otexts.org/fpp3/7

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline
from statsmodels.tsa.exponential_smoothing.ets import ETSModel
```

```python
plt.rcParams['figure.figsize'] = (12, 8)
```

## Simple exponential smoothing

The simplest of the ETS models is also known as *simple exponential smoothing*. In ETS terms, it corresponds to the (A, N, N) model, that is, a model with additive errors, no trend, and no seasonality. The state space formulation of Holt's method is:

\begin{align}
y_{t} &= y_{t-1} + e_t\\
l_{t} &= l_{t-1} + \alpha e_t\\
\end{align}

This state space formulation can be turned into a different formulation, a forecast and a smoothing equation (as can be done with all ETS models):

\begin{align}
\hat{y}_{t|t-1} &= l_{t-1}\\
l_{t} &= \alpha y_{t-1} + (1 - \alpha) l_{t-1}
\end{align}

Here, $\hat{y}_{t|t-1}$ is the forecast/expectation of $y_t$ given the information of the previous step. In the simple exponential smoothing model, the forecast corresponds to the previous level. The second equation (smoothing equation) calculates the next level as weighted average of the previous level and the previous observation.

```python
oildata = [
    111.0091, 130.8284, 141.2871, 154.2278,
    162.7409, 192.1665, 240.7997, 304.2174,
    384.0046, 429.6622, 359.3169, 437.2519,
    468.4008, 424.4353, 487.9794, 509.8284,
    506.3473, 340.1842, 240.2589, 219.0328,
    172.0747, 252.5901, 221.0711, 276.5188,
    271.1480, 342.6186, 428.3558, 442.3946,
    432.7851, 437.2497, 437.2092, 445.3641,
    453.1950, 454.4096, 422.3789, 456.0371,
    440.3866, 425.1944, 486.2052, 500.4291,
    521.2759, 508.9476, 488.8889, 509.8706,
    456.7229, 473.8166, 525.9509, 549.8338,
    542.3405
]
oil = pd.Series(oildata, index=pd.date_range('1965', '2013', freq='AS'))
oil.plot()
plt.ylabel("Annual oil production in Saudi Arabia (Mt)");
```

The plot above shows annual oil production in Saudi Arabia in million tonnes. The data are taken from the R package `fpp2` (companion package to prior version [1]).
Below you can see how to fit a simple exponential smoothing model using statsmodels's ETS implementation to this data. Additionally, the fit using `forecast` in R is shown as comparison.

```python
model = ETSModel(oil, error='add', trend='add', damped_trend=True)
fit = model.fit(maxiter=10000)
oil.plot(label='data')
fit.fittedvalues.plot(label='statsmodels fit')
plt.ylabel("Annual oil production in Saudi Arabia (Mt)");

# obtained from R
params_R = [0.99989969, 0.11888177503085334, 0.80000197, 36.46466837, 34.72584983]
yhat = model.smooth(params_R).fittedvalues
yhat.plot(label='R fit', linestyle='--')

plt.legend();
```

By default the initial states are considered to be fitting parameters and are estimated by maximizing log-likelihood. Additionally it is possible to only use a heuristic for the initial values. In this case this leads to better agreement with the R implementation.

```python
model_heuristic = ETSModel(oil, error='add', trend='add', damped_trend=True,
                          initialization_method='heuristic')
fit_heuristic = model_heuristic.fit()
oil.plot(label='data')
fit.fittedvalues.plot(label='estimated')
fit_heuristic.fittedvalues.plot(label='heuristic', linestyle='--')
plt.ylabel("Annual oil production in Saudi Arabia (Mt)");

# obtained from R
params = [0.99989969, 0.11888177503085334, 0.80000197, 36.46466837, 34.72584983]
yhat = model.smooth(params).fittedvalues
yhat.plot(label='with R params', linestyle=':')

plt.legend();
```

The fitted parameters and some other measures are shown using `fit.summary()`. Here we can see that the log-likelihood of the model using fitted initial states is a bit lower than the one using a heuristic for the initial states.
Additionally, we see that $\beta$ (`smoothing_trend`) is at the boundary of the default parameter bounds, and therefore it's not possible to estimate confidence intervals for $\beta$.

```python
print(fit.summary())
```

                                     ETS Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                   49
    Model:                      ETS(AAdN)   Log Likelihood                -258.108
    Date:                Tue, 02 Feb 2021   AIC                            528.216
    Time:                        07:02:07   BIC                            539.567
    Sample:                    01-01-1965   HQIC                           532.523
                             - 01-01-2013   Scale                         2202.048
    Covariance Type:               approx
    ===================================================================================
                          coef    std err          z      P>|z|      [0.025      0.975]
    -----------------------------------------------------------------------------------
    smoothing_level     0.9999      0.136      7.326      0.000       0.732       1.267
    smoothing_trend  9.999e-05        nan        nan        nan         nan         nan
    damping_trend       0.8798      0.059     14.901      0.000       0.764       0.996
    initial_level      54.2331     55.750      0.973      0.331     -55.034     163.500
    initial_trend      52.9975     34.392      1.541      0.123     -14.409     120.404
    ===================================================================================
    Ljung-Box (Q):                        1.42   Jarque-Bera (JB):                20.73
    Prob(Q):                              0.49   Prob(JB):                         0.00
    Heteroskedasticity (H):               0.52   Skew:                            -1.02
    Prob(H) (two-sided):                  0.20   Kurtosis:                         5.45
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using numerical (complex-step) differentiation.

```python
print(fit_heuristic.summary())
```

                                     ETS Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                   49
    Model:                      ETS(AAdN)   Log Likelihood                -258.881
    Date:                Tue, 02 Feb 2021   AIC                            525.761
    Time:                        07:02:08   BIC                            533.328
    Sample:                    01-01-1965   HQIC                           528.632
                             - 01-01-2013   Scale                         2272.587
    Covariance Type:               approx
    ===================================================================================
                          coef    std err          z      P>|z|      [0.025      0.975]
    -----------------------------------------------------------------------------------
    smoothing_level     0.9999      0.097     10.289      0.000       0.809       1.190
    smoothing_trend     0.1183        nan        nan        nan         nan         nan
    damping_trend       0.8000        nan        nan        nan         nan         nan
    ==============================================
                  initialization method: heuristic
    ----------------------------------------------
    initial_level                          33.6309
    initial_trend                          34.8115
    ===================================================================================
    Ljung-Box (Q):                        0.27   Jarque-Bera (JB):                19.93
    Prob(Q):                              0.88   Prob(JB):                         0.00
    Heteroskedasticity (H):               0.50   Skew:                            -0.95
    Prob(H) (two-sided):                  0.18   Kurtosis:                         5.48
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using numerical (complex-step) differentiation.

## Holt-Winters' seasonal method

The exponential smoothing method can be modified to incorporate a trend and a seasonal component. In the additive Holt-Winters' method, the seasonal component is added to the rest. This model corresponds to the ETS(A, A, A) model, and has the following state space formulation:

\begin{align}
y_t &= l_{t-1} + b_{t-1} + s_{t-m} + e_t\\
l_{t} &= l_{t-1} + b_{t-1} + \alpha e_t\\
b_{t} &= b_{t-1} + \beta e_t\\
s_{t} &= s_{t-m} + \gamma e_t
\end{align}

```python
austourists_data = [
    30.05251300, 19.14849600, 25.31769200, 27.59143700,
    32.07645600, 23.48796100, 28.47594000, 35.12375300,
    36.83848500, 25.00701700, 30.72223000, 28.69375900,
    36.64098600, 23.82460900, 29.31168300, 31.77030900,
    35.17787700, 19.77524400, 29.60175000, 34.53884200,
    41.27359900, 26.65586200, 28.27985900, 35.19115300,
    42.20566386, 24.64917133, 32.66733514, 37.25735401,
    45.24246027, 29.35048127, 36.34420728, 41.78208136,
    49.27659843, 31.27540139, 37.85062549, 38.83704413,
    51.23690034, 31.83855162, 41.32342126, 42.79900337,
    55.70835836, 33.40714492, 42.31663797, 45.15712257,
    59.57607996, 34.83733016, 44.84168072, 46.97124960,
    60.01903094, 38.37117851, 46.97586413, 50.73379646,
    61.64687319, 39.29956937, 52.67120908, 54.33231689,
    66.83435838, 40.87118847, 51.82853579, 57.49190993,
    65.25146985, 43.06120822, 54.76075713, 59.83447494,
    73.25702747, 47.69662373, 61.09776802, 66.05576122,
]
index = pd.date_range("1999-03-01", "2015-12-01", freq="3MS")
austourists = pd.Series(austourists_data, index=index)
austourists.plot()
plt.ylabel('Australian Tourists');
```

```python
# fit in statsmodels
model = ETSModel(austourists, error="add", trend="add", seasonal="add",
                damped_trend=True, seasonal_periods=4)
fit = model.fit()

# fit with R params
params_R = [
    0.35445427, 0.03200749, 0.39993387, 0.97999997, 24.01278357,
    0.97770147, 1.76951063, -0.50735902, -6.61171798, 5.34956637
]
fit_R = model.smooth(params_R)

austourists.plot(label='data')
plt.ylabel('Australian Tourists')

fit.fittedvalues.plot(label='statsmodels fit')
fit_R.fittedvalues.plot(label='R fit', linestyle='--')
plt.legend();
```

```python
print(fit.summary())
```

                                     ETS Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                   68
    Model:                      ETS(AAdA)   Log Likelihood                -152.627
    Date:                Tue, 02 Feb 2021   AIC                            327.254
    Time:                        07:02:10   BIC                            351.668
    Sample:                    03-01-1999   HQIC                           336.928
                             - 12-01-2015   Scale                            5.213
    Covariance Type:               approx
    ======================================================================================
                             coef    std err          z      P>|z|      [0.025      0.975]
    --------------------------------------------------------------------------------------
    smoothing_level        0.3398      0.111      3.070      0.002       0.123       0.557
    smoothing_trend        0.0259      0.008      3.158      0.002       0.010       0.042
    smoothing_seasonal     0.4010      0.080      5.041      0.000       0.245       0.557
    damping_trend          0.9800        nan        nan        nan         nan         nan
    initial_level         29.4475   2.92e+04      0.001      0.999   -5.72e+04    5.72e+04
    initial_trend          0.6147      0.392      1.567      0.117      -0.154       1.383
    initial_seasonal.0    -3.4379   2.92e+04     -0.000      1.000   -5.72e+04    5.72e+04
    initial_seasonal.1    -5.9571   2.92e+04     -0.000      1.000   -5.72e+04    5.72e+04
    initial_seasonal.2   -11.4881   2.92e+04     -0.000      1.000   -5.72e+04    5.72e+04
    initial_seasonal.3          0   2.92e+04          0      1.000   -5.72e+04    5.72e+04
    ===================================================================================
    Ljung-Box (Q):                        5.76   Jarque-Bera (JB):                 7.69
    Prob(Q):                              0.67   Prob(JB):                         0.02
    Heteroskedasticity (H):               0.46   Skew:                            -0.63
    Prob(H) (two-sided):                  0.07   Kurtosis:                         4.05
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using numerical (complex-step) differentiation.

## Predictions

The ETS model can also be used for predicting. There are several different methods available:
- `forecast`: makes out of sample predictions
- `predict`: in sample and out of sample predictions
- `simulate`: runs simulations of the statespace model
- `get_prediction`: in sample and out of sample predictions, as well as prediction intervals

We can use them on our previously fitted model to predict from 2014 to 2020.

```python
pred = fit.get_prediction(start='2014', end='2020')
```

```python
df = pred.summary_frame(alpha=0.05)
df
```

      mean
      pi_lower
      pi_upper

      2014-03-01
      67.611048
      63.136076
      72.086020

      2014-06-01
      42.814811
      38.339839
      47.289783

      2014-09-01
      54.106462
      49.631490
      58.581434

      2014-12-01
      57.928276
      53.453304
      62.403248

      2015-03-01
      68.422079
      63.947107
      72.897051

      2015-06-01
      47.277849
      42.802877
      51.752821

      2015-09-01
      58.954740
      54.479768
      63.429712

      2015-12-01
      63.982155
      59.507183
      68.457127

      2016-03-01
      75.905268
      71.430295
      80.380240

      2016-06-01
      51.418122
      46.654016
      56.182228

      2016-09-01
      63.703292
      58.629393
      68.777191

      2016-12-01
      67.978107
      62.575778
      73.380436

      2017-03-01
      78.315764
      71.735800
      84.895727

      2017-06-01
      53.780408
      46.883520
      60.677296

      2017-09-01
      66.018333
      58.788303
      73.248362

      2017-12-01
      70.246847
      62.668821
      77.824872

      2018-03-01
      80.539128
      71.892785
      89.185471

      2018-06-01
      55.959305
      46.968347
      64.950264

      2018-09-01
      68.153652
      58.805308
      77.501996

      2018-12-01
      72.339460
      62.621963
      82.056957

      2019-03-01
      82.589889
      71.864783
      93.314995

      2019-06-01
      57.969051
      46.876058
      69.062044

      2019-09-01
      70.123203
      58.652201
      81.594204

      2019-12-01
      74.269620
      62.411255
      86.127984

      2020-03-01
      84.481446
      71.656450
      97.306441

In this case the prediction intervals were calculated using an analytical formula. This is not available for all models. For these other models, prediction intervals are calculated by performing multiple simulations (1000 by default) and using the percentiles of the simulation results. This is done internally by the `get_prediction` method.

We can also manually run simulations, e.g. to plot them. Since the data ranges until end of 2015, we have to simulate from the first quarter of 2016 to the first quarter of 2020, which means 17 steps.

```python
simulated = fit.simulate(anchor="end", nsimulations=17, repetitions=100)
```

```python
for i in range(simulated.shape[1]):
    simulated.iloc[:,i].plot(label='_', color='gray', alpha=0.1)
df["mean"].plot(label='mean prediction')
df["pi_lower"].plot(linestyle='--', color='tab:blue', label='95% interval')
df["pi_upper"].plot(linestyle='--', color='tab:blue', label='_')
pred.endog.plot(label='data')
plt.legend()
```

In this case, we chose "end" as simulation anchor, which means that the first simulated value will be the first out of sample value. It is also possible to choose other anchor inside the sample.

```python

```


# Extended Kalman filter with Parameter Identification for Nomoto model
An Extended Kalman filter with a Nomoto model as the predictor will be developed.
The filter should also estimate the parameters in the Nomoto model. This is done by setting up a system where the parameters are defined as some of the states in the model
The filter is run on simulated data as well as real model test data.

```python
%load_ext autoreload
%autoreload 2

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv
import sympy as sp

import src.visualization.book_format as book_format
book_format.set_style()
from src.substitute_dynamic_symbols import lambdify
from sympy import Matrix
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify
from sympy.physics.vector.printing import vpprint, vlatex
from src.data import mdl
from src.kalman_filter import extended_kalman_filter
```

## Nomoto model for ship manoeuvring dynamics
The Nomoto model can be written as:

```python
r,r1d,r2d = sp.symbols('r \dot{r} \ddot{r}')
psi,psi1d = sp.symbols('psi \dot{\psi}')
h,u = sp.symbols('h u')
x, x1d = sp.symbols('x \dot{x}')
A,B,C,D,E, Phi = sp.symbols('A B C D E Phi')
w = sp.symbols('w')

K, delta, T_1, T_2 = sp.symbols('K delta T_1 T_2')

eq_nomoto = sp.Eq(K*delta,
                 r + T_1*r1d + T_2*r2d)
Math(vlatex(eq_nomoto))
```

where $r$ is yaw rate with its time derivatives and $\delta$ is the rudder angle. $K$, $T_{1}$
 and $T_{1}$ are the coefficients describing the hydrodynamics of the ship.

For slow manoeuvres this equation can be further simplified by removing the $\ddot{r}$ term into a first order Nomoto model:

```python
eq_nomoto_simple = eq_nomoto.subs(r2d,0)
Math(vlatex(eq_nomoto_simple))
```

### Simulation model

```python
f_hat = sp.Function('\hat{f}')(x,u,w)
eq_system = sp.Eq(x1d, f_hat)
eq_system
```

Where the state vector $x$:

```python
eq_x = sp.Eq(x, sp.UnevaluatedExpr(Matrix([psi,r])))
eq_x
```

and input vector $u$:

and $w$ is zero mean Gausian process noise

For the nomoto model the time derivatives for the states can be expressed as:

```python
eq_psi1d = sp.Eq(psi1d,r)
eq_psi1d
```

```python
eq_r1d = sp.Eq(r1d,sp.solve(eq_nomoto_simple,r1d)[0])
eq_r1d
```

```python
def lambda_f_constructor(K, T_1):
    def lambda_f(x, u):
        delta = u
        f = np.array([[x[1], (K*delta-x[1])/T_1]]).T
        return f

    return lambda_f
```

## Simulation
Simulation with this model where rudder angle shifting between port and starboard

```python
T_1_ = 1.8962353076056344
K_ = 0.17950970687951323
h_ = 0.01

lambda_f = lambda_f_constructor(K=K_, T_1=T_1_)
```

```python
def simulate(E, ws, t, us):

    simdata = []
    x_=np.deg2rad(np.array([[0,0]]).T)

    for u_,w_ in zip(us,ws):

        x_=x_ + h_*lambda_f(x=x_.flatten(), u=u_)

        simdata.append(x_.flatten())

    simdata = np.array(simdata)
    df = pd.DataFrame(simdata, columns=["psi","r"], index=t)
    df['delta'] = us

    return df
```

```python
N_ = 8000

t_ = np.arange(0,N_*h_,h_)

us = np.deg2rad(np.concatenate((-10*np.ones(int(N_/4)),
                                10*np.ones(int(N_/4)),
                                -10*np.ones(int(N_/4)),
                                10*np.ones(int(N_/4)))))

np.random.seed(42)
E = np.array([[0, 1]]).T
process_noise = np.deg2rad(0.01)
ws = process_noise*np.random.normal(size=N_)
df = simulate(E=E, ws=ws, t=t_, us=us)

measurement_noise = np.deg2rad(0.5)
df['epsilon'] = measurement_noise*np.random.normal(size=N_)
df['psi_measure'] = df['psi'] + df['epsilon']
df['psi_deg'] = np.rad2deg(df['psi'])
df['psi_measure_deg'] = np.rad2deg(df['psi_measure'])
df['delta_deg'] = np.rad2deg(df['delta'])

```

```python
fig,ax=plt.subplots()
df.plot(y='psi_deg', ax=ax)
df.plot(y='psi_measure_deg', ax=ax, zorder=-1)
df.plot(y='delta_deg', ax=ax, zorder=-1)
df.plot(y='r')
ax.set_title('Simulation with measurement and process noise')
ax.set_xlabel('Time [s]');

```

## Kalman filter
Implementation of the Kalman filter. The code is inspired of this Matlab implementation: [ExEKF.m](https://github.com/cybergalactic/MSS/blob/master/mssExamples/ExEKF.m).

```python
jac = sp.eye(3,3) + Matrix([r,eq_r1d.rhs,0]).jacobian([psi,r,T_1])*h
jac
```

```python
def lambda_f_constructor2(K):
    def lambda_f(x, u):
        delta = u
        T_1 = x[2]  # Note! T_1 is the third state now!
        r = x[1]

        f = np.array([[r, (K*delta-r)/T_1, 0]]).T
        return f

    return lambda_f
```

```python
def lambda_jacobian_constructor(h, K):
    def lambda_jacobian(x, u):

        T_1 = x[2]  # Note! T_1 is the third state now!
        delta = u
        r = x[1]

        jac = np.array(
            [
                [1, h, 0],
                [0, 1 - h / T_1, -h * (K * delta - r) / T_1 ** 2],
                [0, 0, 1],
            ]
        )
        return jac

    return lambda_jacobian
```

```python
lambda_jacobian = lambda_jacobian_constructor(h=h_, K=K_)
lambda_f = lambda_f_constructor2(K=K_)
```

```python
lambda_jacobian(x=[0,0,0.1], u=0)
```

```python
lambda_f(x=[0,0,0.1], u=0)
```

```python
x0=np.deg2rad(np.array([[0,0,3]]).T)
P_prd = np.diag([np.deg2rad(1), np.deg2rad(0.1), 0.1])

Qd = np.diag([np.deg2rad(0), 2])

Rd = np.deg2rad(1)

ys = df['psi_measure'].values

E_ = np.array(
    [
     [0,0],
     [1,0],
     [0,1]
    ],
)

C_ = np.array([[1, 0, 0]])

Cd_ = C_
Ed_ = h_ * E_

time_steps = extended_kalman_filter(x0=x0, P_prd=P_prd, lambda_f=lambda_f,
                                    lambda_jacobian=lambda_jacobian,h=h_, us=us, ys=ys, E=E_, Qd=Qd, Rd=Rd, Cd=Cd_)
x_hats = np.array([time_step["x_hat"] for time_step in time_steps]).T
time = np.array([time_step["time"] for time_step in time_steps]).T
Ks = np.array([time_step["K"] for time_step in time_steps]).T
stds = np.sqrt(np.array([[time_step["P_hat"][0,0],
                          time_step["P_hat"][1,1],
                          time_step["P_hat"][2,2]] for time_step in time_steps]).T)

```

```python
n=len(P_prd)
fig,axes=plt.subplots(nrows=n)

df['T_1'] = T_1_
keys = ['psi','r','T_1']
for i,key in enumerate(keys):

    ax=axes[i]
    df.plot(y=key, ax=ax, label="True")
    if key=='psi':
        df.plot(y='psi_measure', ax=ax, label="Measured", zorder=-1)

    ax.plot(time, x_hats[i, :], "-", label="kalman")

    std_top = x_hats[i, :] + stds[i, :]
    std_btm = x_hats[i, :] - stds[i, :]
    ax.plot(time, std_top, linestyle=':', color='k', lw=1, alpha=0.4)
    ax.plot(time, std_btm, linestyle=':', color='k', lw=1, alpha=0.4)
    ax.fill_between(time, std_top, std_btm,
                     facecolor='yellow', alpha=0.2, interpolate=True, label='+/- std')

    ax.set_ylabel(key)
    ax.legend()

```

```python
fig,ax=plt.subplots()
for i,key in enumerate(keys):
    ax.plot(time,Ks[i,:],label=key)
ax.set_title('Kalman gains')
ax.legend();
ax.set_ylim(-1,1);

```

```python
fig,axes=plt.subplots(nrows=2)

ax=axes[0]
for i,key in enumerate(keys):
    ax.plot(time,stds[i,:]**2,label=key)
ax.set_title('Variances')
ax.legend();

ax=axes[1]
df.plot(y='delta',ax=ax)

```

# Real data
Using the developed Kalman filter on some real model test data

## Load test

```python
id=22773
df, units, meta_data = mdl.load(dir_path = '../data/raw', id=id)
df.index = df.index.total_seconds()
df.index-=df.index[0]
```

```python
from src.visualization.plot import track_plot
fig,ax=plt.subplots()
fig.set_size_inches(10,10)
track_plot(df=df, lpp=meta_data.lpp, x_dataset='x0', y_dataset='y0',  psi_dataset='psi', beam=meta_data.beam, ax=ax);
```

```python
ys = df['psi'].values
h_m=h_ = df.index[1]-df.index[0]
us = df['delta'].values

x0=np.deg2rad(np.array([[0,0,T_1_]]).T)
P_prd = np.diag([np.deg2rad(1), np.deg2rad(0.1), 100])

Qd = np.diag([np.deg2rad(5), 10])

Rd = np.deg2rad(0.1)

E_ = np.array(
    [
     [0,0],
     [1,0],
     [0,1]
    ],
)

C_ = np.array([[1, 0, 0]])

Cd_ = C_
Ed_ = h_ * E_

time_steps = extended_kalman_filter(x0=x0, P_prd=P_prd, lambda_f=lambda_f,
                                    lambda_jacobian=lambda_jacobian,h=h_, us=us, ys=ys, E=E_, Qd=Qd, Rd=Rd, Cd=Cd_)
x_hats = np.array([time_step["x_hat"] for time_step in time_steps]).T
time = np.array([time_step["time"] for time_step in time_steps]).T
Ks = np.array([time_step["K"] for time_step in time_steps]).T
stds = np.sqrt(np.array([[time_step["P_hat"][0,0],
                          time_step["P_hat"][1,1],
                          time_step["P_hat"][2,2]] for time_step in time_steps]).T)
```

```python
n=len(P_prd)
fig,axes=plt.subplots(nrows=n)

ax = axes[0]
df.plot(y='psi', label='Measured', ax=ax)

df['T_1'] = T_1_
keys = ['psi','r','T_1']
for i,key in enumerate(keys):

    ax=axes[i]

    ax.plot(time, x_hats[i, :], "-", label="kalman")

    std_top = x_hats[i, :] + stds[i, :]
    std_btm = x_hats[i, :] - stds[i, :]
    ax.plot(time, std_top, linestyle=':', color='k', lw=1, alpha=0.4)
    ax.plot(time, std_btm, linestyle=':', color='k', lw=1, alpha=0.4)
    ax.fill_between(time, std_top, std_btm,
                     facecolor='yellow', alpha=0.2, interpolate=True, label='+/- std')

    ax.set_ylabel(key)
    ax.legend()
```

```python
fig,ax=plt.subplots()
for i,key in enumerate(keys):
    ax.plot(time,Ks[i,:],label=key)
ax.set_title('Kalman gains')
ax.legend();
ax.set_ylim(-1,1);
```

```python

```